The Brazilian raced clear to slot through the legs of Arsenal keeper Lukasz Fabianski just two minutes into the second half.
Greek titleholders Olympiakos went through as the second-placed team from Group H with 10 points, three behind Arsenal who had already qualified before the final group match.
Arsenal manager Arsene Wenger fielded a starting lineup with an average age of just 21 and his young team were unlucky not to snatch a draw with Carlos Vela guilty of passing up a couple of good opportunities.
The visitors, with Theo Walcott full of running, controlled much of the early play as the hosts began nervously.
It took Arsenal just nine minutes to register the first attempt on goal with England forward Walcott stinging the palms of Antonis Nikopolidis with a snapped shot from inside the penalty area.
Kostas Mitroglou responded for Olympiacos after 13 minutes, slicing an effort over the crossbar when he should have at least forced Fabianski to make a save.
The home side then broke on the counter and a decent opportunity went begging when Luciano Galletti volleyed Leonardo's cross into the side-netting. Arsenal continued to command the midfield and Walcott was proving a constant threat in and around the penalty area with his quick feet and darting runs.
LEONARDO STRIKES
Arsenal kept the hosts at arms length, and it took Mitroglou 43 minutes to register Olympiakos' first shot on target with a speculative shot.
The second half began with a flourish when Olympiakos took the lead after just two minutes.
Arsenal were caught cold when the ball broke kindly for Leonardo and he surged into the penalty area before slotting through the legs of Fabianski and into the bottom right corner.
Frenzied celebrations ensued as the home fans sensed the first knockout round was near and they were thankful just four minutes later when Nikopolidis kept his team ahead with a brilliant save from Vela after the Arsenal striker was clean through on goal.
Vela was guilty of profligacy in front of goal when he prodded over with only Nikopolidis to beat in the 70th minute as Arsenal looked to respond.
At the other end, Leonardo then spurned a chance to put Olympiakos 2-0 up just two minutes later when he was denied by Fabianski's sprawling save.
Arsenal pressed forward in the closing stages as they sought to grab an equaliser but Nikopolidis put in several good blocks to keep a clean sheet for the home side.
The Greek club's Brazilian coach Zico said his team had gone all out for victory from the opening minute.
"It's a major achievement for this team when you consider we have suffered major injuries along the way so we have exceeded expectations by qualifying for the next round," he said.
"I have no preference as to which team we get in the next round and I'm sure the other teams will not want to be drawn against Olympiakos either. If they do, they are making a mistake."
Wenger added: "We were caught cold at the start of the second half. We had two clear cut chances to come back b
