In other replays, which also featured Premier League sides beating second tier teams, Wolverhampton Wanderers thrashed Doncaster Rovers 5-0 while Stoke City needed extra-time to beat Cardiff City 2-0.
Two goals in just over a minute towards the end of the first half tipped the game in Manchester City's favour after Carlos Tevez's 15th-minute opener for the home side was cancelled out by Paul Gallagher's penalty four minutes later.
Patrick Vieira and Adam Johnson struck for City in quick succession but Lloyd Dyer's 83rd-minute strike set nerves jangling before Aleksandar Kolarov made sure of the win for the Premier League high-fliers with a last-minute goal.
Tevez had a 59th-minute penalty saved but his side were not made to pay for it as they set up a trip to League One Notts County in the fourth round.
"I think we made it hard for ourselves," City winger Johnson told ESPN television. "I think when we got a two-goal lead we shouldn't really have been put under pressure."
City manager Roberto Mancini sported a red and black scarf over the top of his usual blue and white one as he joined thousands of home fans in honouring terminally ill ex-City forward Neil Young, who scored the winner in their 1969 FA Cup final triumph over Leicester.
STRONG SIDE
A strong City side took the lead when Tevez muscled his way through the Leicester defence to unleash a fierce right-footed shot after a quarter of an hour.
Four minutes later, a careless mistake by Vieira, when his outstretched leg brought down Dyer in the box, gave the visitors a penalty which Gallagher netted with a neat strike that sent keeper Joe Hart the wrong way.
Vieira then made up for his error when he tapped in on 37 minutes after David Silva's shot was not cleared properly and Johnson quickly added another when he beat the offside trap.
City should have wrapped the match up when Tevez was brought down by Jack Hobbs in the box but keeper Chris Weale did well to save the resulting spot-kick and keep his side in the match.
That lifted Leicester and Dyer gave them some late hope when he slotted home despite looking offside, only for Kolarov to deal the final blow with a low left-footed shot into the bottom corner.
"They have better players than we have but I think we should be proud of the performance we had today," Eriksson, who managed City in 2007/08, told ESPN.
In a goal fest at Molineux, Premier League strugglers Wolves went ahead thanks to a fifth-minute strike from Steven Fletcher.
Geoffrey Mujangi Bia added a second just after the hour mark, opening the floodgates as Kevin Doyle, Matt Jarvis and David Jones added their names to the scoresheet.
Stoke striker Jon Walters struck twice in extra-time to lift his side to victory over Cardiff and set up a fourth-round clash with Wolves.
