Solskjaer's men looked to be on their way to defeat in an uninspiring encounter, which only sprung into life when Newcastle took the lead through Papiss Cisse just after the hour mark.
However, the Welsh club displayed excellent fighting spirit, and they levelled matters shortly afterwards thanks to Craig Noone's excellent strike from distance.
And the fightback was completed 10 minutes from time as ex-Sunderland man Fraizer Campbell turned home Peter Whittingham's corner to get Solskjaer's reign off to the perfect start.
Newcastle were without the suspended Mathieu Debuchy, while Cisse started in attack in a side captained by Cheick Tiote in the absence of Fabricio Coloccini.
Solskjaer, meanwhile, made five changes to Cardiff's starting XI from the 2-0 Premier League defeat to Arsenal, with Andreas Cornelius and Peter Odemwingie paired up front for the visitors.
Cardiff made the better start to proceedings and looked to have taken the lead when Mark Hudson headed home Whittingham's corner after nine minutes, only for the goal to be ruled out for a push in the area.
Kevin McNaughton and Kim Bo-Kyung then went close from distance as Newcastle struggled to gain a foothold in the game.
However, the hosts did finally threaten when Hatem Ben Arfa crashed an effort against the left-hand post after 24 minutes.
Yoan Gouffran then shot wide on the half-hour mark, but Newcastle generally struggled to test goalkeeper David Marshall.
Marshall was called into action twice in the early stages of the second half as he denied both Ben Arfa and Moussa Sissoko.
Newcastle were rewarded for their positive second-half showing in the 61st minute, Cisse converting from point-blank range for only his third goal of the season after Sissoko's initial shot had only been partially cleared.
Cardiff's response was a strong one and Solskjaer's men came close to levelling when Campbell struck the post with an effort from 18 yards.
It was Noone's introduction that would prove pivotal, though - the midfielder's long-range piledriver flying past Rob Elliot and into the top corner.
The tie appeared poised for a replay at that point, with both sides pushing for a winner late on.
And it was the Welsh side who ultimately got it - as with 10 minutes left Campbell rose highest to head Cardiff into round four.
‘After Manchester City’s recent form, maybe they’re the underdogs against Manchester United!’ Former Red Devils defender on this weekend’s derby
‘Arteta, Alonso, Emery, me… none of us were physical players – we needed the understanding of the game. That probably helped us move into management’: Premier League boss reveals reasons for natural career progression
