The League One side looked lively right from the start and, after seeing numerous chances go begging in the first hour, netted the all-important winner through Mark McNulty with 27 minutes to go at Bramall Lane.
Southampton, who have now lost five matches in succession, saw their misery compounded by the late dismissal of Florin Gardos.
Forty-four places separate the two teams in the English football pyramid, but United looked more than a match for their opponents in the first half, with Jay McEveley and Florent Cuvelier both going close.
The visitors saw penalty appeals turned down early in the second period and United duly capitalised, McNulty making the most of unconvincing goalkeeping from the previously impressive Fraser Forster, who spilled a powerful free-kick from Bob Harris.
Southampton poured men forward after falling behind and forced Mark Howard into action through Sadio Mane with just over 20 minutes to go, while Ryan Flynn and Jamie Murphy each spurned opportunities to net a second on the break for United.
Those misses did not matter in the end as United, who knocked out another Premier League side in West Ham in round two, held firm in the face of intense pressure to secure a famous triumph.
Ronald Koeman once again resisted the urge to make wholesale changes to his team for a League Cup tie, while James Ward-Prowse was given his first start since September 20 after an injury lay-off.
However, even with Southampton naming a strong line-up, United kept the visitors under significant pressure.
And the underdogs went close to taking the lead after just three minutes, as Forster dropped a cross while seemingly being fouled by McEveley, who subsequently looped an effort over his own head and on to the top of the crossbar.
United caused problems again when centre-back Chris Basham produced a surging run up the right flank and, after linking well with Jamal Campbell-Ryce, crossed for Cuvelier, who sliced wide from eight yards.
But the opening half's best chance fell to the visitors just before the break, as Mane steered narrowly off target following Shane Long's left-wing delivery.
Both sides threatened at the start of the second half, as Southampton saw penalty claims rejected after Gardos' header appeared to strike a hand and Murphy inexplicably sliced wide at the other end.
United were well on top again by the 55th minute, with Forster first producing a fine save to keep out Che Adams' cross and then palming Michael Doyle's effort over the crossbar.
Yet the England international made a costly mistake in the 63rd minute when he failed to hold on to Harris' free-kick, enabling McNulty to tap in from inside a yard.
The United goal seemingly woke Southampton up and they almost pulled level as Mane broke free and saw his effort well saved by Howard.
Koeman's men were given a let-off when Flynn could only fire straight at Forster when one-on-one, but one goal was to prove enough for United and Southampton's frustrating evening was summed up at the death as Gardos brought down McNulty as the last man to receive a red card.
