Ryan Flynn's 81st-minute strike on Saturday resulted in Villa exiting the competition at this stage for the first time since 2007-08 and piles the pressure on Paul Lambert at the end of a week in which he admitted he would rather avoid the distraction of the FA Cup to ensure Premier League survival.
A draw looked likely after Nicklas Helenius scored his first Villa goal to cancel out Jamie Murphy's opener, but Flynn sealed victory for Nigel Clough's side, who are 18th in League One.
Scott Hogan's fifth goal in four games put the League Two side ahead just before half-time and Ian Henderson added a second six minutes from time.
Arsenal defeated fierce local rivals Tottenham 2-0 at the Emirates Stadium to progress. Santi Cazorla and Tomas Rosicky confirmed the victory with goals in either half to extend the side's unbeaten home run against their north London neighbours to four games in all competitions.
Stoke City saw off the threat of Championship leaders Leicester City 2-1, Southampton survived a late scare to beat Burnley 4-3 and Tony Pulis' Crystal Palace defeated West Brom 2-0 thanks to goals from Dwight Gayle and Marouane Chamakh.
Two of the three non-league sides in the third round will face replays after earning draws at home. Macclesfield Town had Steve Williams to thank as he cancelled out Reda Johnson's opener for Sheffield Wednesday to seal a 1-1 draw at Moss Rose, while Kidderminster held Peterborough United 0-0 at Aggborough.
Grimsby Town, the third non-league outfit, were denied a memorable victory over Huddersfield Town as Mark Robins' side scored twice in the last five minutes to win 3-2. Grimsby had led thanks to Ross Hannah and Craig Disley either side of Oliver Norwood netting for Huddersfield, but Martin Paterson's fourth goal in as many games and an Aswad Thomas own goal saw them exit the competition.
Elsewhere, Bristol City held Championship side Watford 1-1 at Ashton Gate, Kevin Davies earned Preston North End a 1-1 draw and a replay against Ipswich Town and Robert Snodgrass cancelled out Darren Bent's 40th-minute strike as Norwich drew 1-1 with Fulham.
