Pardew has won all four matches in charge since he took over from Neil Warnock earlier this month, and Palace have come from behind in each of their last three outings.
In an exciting first half which provided all of the game's goals, Southampton twice took the lead through Graziano Pelle and Scott Dann's own goal.
Marouane Chamakh and Yaya Sanogo found the net to cancel out those two goals, before Chamakh netted his second to give Palace the lead for the first time.
The visitors held on in the second period to book their place in the fifth round for the first time since 2010, and Pardew was thrilled with the display put on by his side.
"The players have been phenomenal," he told BBC Sport.
"I did have a chuckle this morning when I saw the think-tank in the paper when it said we had 0.02 per cent chance to win the FA Cup, that's probably gone up a bit.
"I thought both teams were great. There was some really good football played. We are disappointed with the goals we conceded but the third goal would have graced any pitch in the world.
"Yaya Sanogo was good, Marouane Chamakh was better so it gives me a welcome selection headache."
‘After Manchester City’s recent form, maybe they’re the underdogs against Manchester United!’ Former Red Devils defender on this weekend’s derby
‘Arteta, Alonso, Emery, me… none of us were physical players – we needed the understanding of the game. That probably helped us move into management’: Premier League boss reveals reasons for natural career progression
