The Dutch striker's earlier penalty miss with his side trailing 2-1 looked to have condemned United to a second defeat in their opening three league games but he made amends with two poacher's goals in the dying minutes to steal the points.
Van Persie's thumping 92nd-minute header was his 100th Premier League goal and his fourth in three appearances for United since he joined from Arsenal for 24 million pounds in the transfer window.
Earlier, Lukas Podolski, the man tasked with filling the prolific Dutchman's boots for Arsenal, opened his account in English football in a 2-0 victory at Liverpool.
It was Arsenal's first win, and first goals, of the season after two 0-0 draws, while Liverpool's solitary point from their opening three fixtures is their worst start for 50 years.
In the day's other match, Newcastle United were held to a 1-1 draw at home by Aston Villa, leaving Southampton as the only top-flight side still without a point.
Van Persie scored 37 times for Arsenal last season to clinch the Premier League's Player of the Year award and has carried that prolific form to United.
"He's got four goals in two starts which is a great statistic and he'll get better," United manager Sir Alex Ferguson, whose side climbed to fifth with six points, told Sky Sports.
Until the 87th minute, however, it was Van Persie's fluffed penalty which was set to make the headlines.
Shortly after Patrice Evra's slip allowed Morgan Schneiderlin to restore the home side's lead, Van Persie was felled in the area by Jos Hooiveld.
After placing the ball on the spot he dispensed with his usual penalty-taking style, electing instead to try and dink the ball past Kelvin Davies but his execution was poor and the keeper was able to claw the ball away.
Newly-promoted Southampton had taken the lead after 16 minutes when Rickie Lambert climbed high to head past Anders Lindegaard after United had lost possession in midfield.
Van Persie levelled eight minutes later, chesting down Antonio Valencia's pass and smashing left-footed past Davis.
United never looked secure at the back, though, and Schneiderlin headed Lambert's cross past Lindegaard after 56 minutes, finding himself unmarked following Evra's slip.
After Van Persie's penalty miss the home fans were preparing to celebrate a famous win, only for Van Persie to ruin the party. First he prodded home from close range after Rio Ferdinand's header rebounded off the post.
Then, deep into added time, he made sublime contact with a corner to flash a header past Davis.
"We were a bit lucky with the 2-2 and the dramatic end," Van Persie told Sky Sports.
"I have to say a big thank you to Paul Scholes. When he came on, every single pass he hit was the right one. With him you are always on your toes. For me, he was the man of the match."
Of his penalty blunder, he said: "I don't know what I was thinking with the penalty. I was going to hit it hard but at the last second I changed my mind."
‘After Manchester City’s recent form, maybe they’re the underdogs against Manchester United!’ Former Red Devils defender on this weekend’s derby
‘Arteta, Alonso, Emery, me… none of us were physical players – we needed the understanding of the game. That probably helped us move into management’: Premier League boss reveals reasons for natural career progression
