Tomas Rosicky's first goal in nine months gave the North London side a 38th-minute advantage but the hosts came storming back with second-half strikes by Giannis Maniatis and substitute Kostas Mitroglou.
Schalke 04 finished top of the group with 12 points after drawing 1-1 at Montpellier, ahead of Arsenal, who were already assured of a place in the 16, on 10 points.
Greek side Olympiakos, who had qualified for the Europa League in third position before the match, ended with nine points from their six games.
Arsenal rested several first-team players and handed 20-year-old Jernade Meade his Champions League debut at left back.
It was part of a much-changed side selected by Arsene Wenger for his 150th Champions League match, with midfielder Rosicky and striker Marouane Chamakh named in the starting lineup for the first time in the competition this season.
"We had a good first half, but I believe we dropped physically in the second half because many of the players didn't have the match experience at that level," Wenger told reporters before venting his frustration at referee Alberto Mallenco.
The Spaniard awarded Olympiakos a corner that led to their first goal after Arsenal goalkeeper Wojciech Szczesny was ruled to have touched the ball.
"It was a very, very bad decision against us," Wenger said. "I don't like to say anything about the referee, but I would like at least for them to concentrate on the job... I don't want to make too much fuss about it so we have to take it on the chin."
LIVELY START
After a lively beginning by the visitors with Gervinho trying his luck with an early snapshot and then playing in Chamakh with the Moroccan losing his footing, it was Olympiakos who sought to assert their dominance.
Vassilis Torossidis went close with a header, which was cleared off the line by Carl Jenkinson, before their Algerian midfielder Djamel Abdoun became the focal point in attack, drawing a fine save from Szczesny after a fierce shot.
Rafik Djebbour then headed straight at the Arsenal keeper from the resulting corner before Abdoun created an opening for Torossidis, only to see the latter's tame shot sail over.
Arsenal held firm, though, and took the lead with halftime approaching when Gervinho broke down the right and picked out the run of Rosicky, who cliniclaly side-footed home for his first goal since the start of March against AC Milan.
Andrey Arshavin replaced Rosicky at the break and the diminutive Russian was busy early on as he firstly headed narrowly over the bar from a Chamakh cross before forcing a fine save by Olympiakos' Northern Irish keeper Roy Carroll.
At the other end, Szczesny made a sprawling save to keep out Paulo Machado but he could do little about Maniatis' close-range finish after 64 minutes with the midfielder equalising after a knock-down from Kostas Manolas.
Mitroglou replaced Abdoun in the 72nd and won the game a minute later with a superb curling strike from the edge of the penalty area.
The Greeks were already destined for the Europa League,
