Not even a sixth-minute opener from Javier Hernandez could inspire United as Aras Ozbiliz levelled eight minutes before the break and Toby Aldeweireld's late header set up a nerve-wracking finale.
United manager Sir Alex Ferguson took the blame for fielding too many young defenders after his side slunk through 3-2 on aggregate in only the second meeting of two of Europe's most successful clubs.
They will face Atheltic Bilbao in the last 16 after the side led by eccentric former Argentina and Chile coach Marcelo Bielsa beat Lokomotiv Moscow 1-0, despite being reduced to 10-man, to qualify on away goals.
Atletico Madrid beat Lazio 1-0 to complete a 4-1 aggregate win and Valencia, Spain's other representatives, beat Stoke City 1-0 to complete a 2-0 aggregate win.
Steaua Bucharest's dream of reaching the final in their home city ended at Twente Enschede who were joined in the last sixteen by fellow Dutch side PSV Eindhoven.
"I'm disappointed, I'm responsible for picking the team and there were too many young players in the back four, that was a big problem in terms of experience," said Ferguson in a television interview. "It was a big risk I took."
"Ajax are a fantastic football team in terms of keeping the ball, they never looked like scoring [at first] but they got the goal and it gave them that encouragement."
Although Ajax have not won a European trophy since 1995, they still have a more impressive tally than United with four European Cup titles, one Cup Winners' Cup and one UEFA Cup.
Missing Wayne Rooney due to a throat infection, United appeared to be cruising when Hernandez put them ahead after six minutes, cutting inside his marker to score with a clinical finish after being set up by Dimitr Berbatov.
LOUD FANS
Ajax's possession was pleasing to witness but ineffective until the 37th minute when Aras Ozbiliz thumped home a first-time shot from 20 metres despite a suspected handball by Nicolas Lodeiro in the build-up.
Ajax, the first Dutch team to win at Old Trafford, took control of the game and sounded a warning just before the hour when Siem de Jong's header was brilliantly turned over the bar by David de Gea.
Although United eased Ajax's grip, defender Aldeweireld stole ahead of hesitant defenders to head home from an inswinging free-kick in the 87th minute.
It was hard to fathom how Ajax, whose fans outsang the home crowd throughout the match, had been knocked out in the group stages of the Champions League.
Athletic struggled to claw back a 2-1 first-leg deficit against Lokomotiv and their cause appeared to be lost when Fernando Amorebieta was given a second yellow card for a trip of Roman Shishkin on the hour.
Yet, two minutes later, teenager Iker Muniain scored at the far post to send Athletic through on away goals.
Former European champions Steaua were already trailing 1-0 as they travelled to Twente Enschede and their hopes ended just before the half hour following a blunder by Ciprian Tatarusanu, who lost his footing as he tried to control a back-pass.
Luuk de Jong pounced before squaring to Nacer Chadli, who calmly rolled into the unguarded net to give Twente another 1-0 win.
Braga, last season's runners-up, followed fellow Portuguese and defending champions Porto out of the tournament as they lost 2-1 on aggregate to Besiktas, despite winning the second leg 1-0 away with a Lima goal.
Uruguayan defender Diego Godin headed the winner for Atletico Madri
