A 19th-minute goal from in-form striker Jermaine Beckford gave the League One leaders victory as Manchester United, who have won the Cup a record 11 times, went out at this stage for the first time since 1984.
NEWS: Leeds handed Tottenham tie
Holders Chelsea made sure there was no slip-up at Stamford Bridge where they crushed Championship Watford 5-0 after streaking into a 3-0 lead in 22 minutes.
Arsenal, the second most successful club in the cup's history with 10 wins, left it late to beat West Ham United in a pulsating all-London tie at Upton Park.
The biggest upset of the round, and the competition, for many years came at Old Trafford where Beckford's solitary strike was fashioned by a 50-metre punt from Johnny Howson.
The ball went over the champions' defence, allowing Beckford to get a touch and angle a shot wide of keeper Tomas Kuszczak.
United's first loss to lower league opposition since 1984 also gave Leeds a memorable victory at a ground where they regularly used to duel with United for national supremacy before their slide down the leagues.
BIG SHOCK
"I must admit I didn't expect that performance," Ferguson told MUTV. "If you don't start and the other team does, you're always up against it.
"Leeds had a far bigger appetite than us for the game. You need a bit of luck and they got it but they deserved it because they played really well."
"It was a good long diagonal ball from Johnny Howson and nine times out of 10 Jermaine finishes them off and that's what he did," he told ITV.
"We gave ourselves an opportunity to go and win the game from there and with a bit of luck here and there we've hung on.
"I am sure they will be celebrating in Leeds for the next few days. The players will have tomorrow off then we will start preparing for the match against Wycombe."
Grayson's men thoroughly deserved their victory which would have been more comfortable if Beckford had scored after 78 minutes instead of dragging his shot just wide.
