Enner Valencia made his first start for West Ham, but the World Cup star missed the crucial penalty in the shootout, allowing Michael Doyle to seal a 5-4 win following a 1-1 draw after 120 minutes.
Nigel Clough led his League One side to wins over two Premier League teams on their run to the FA Cup semi-finals last term and they showed their giant-killing instincts again on Tuesday to doggedly take the game the distance.
Valencia started brightly, but it was striker partner Diafra Sakho who opened the scoring with his opening goal for the club after a spell of sustained West Ham pressure in the first half.
Sam Allardyce made nine changes to his side from their weekend win over Crystal Palace and a lack of cohesion was perhaps to blame as the visitors roared out for the second half and hit back as Winston Reid scored an own-goal.
The Yorkshire club showed tremendous resolve to repel everything that West Ham threw at them, and grabbed a hard-fought spot in the third round.
In front of a packed Upton Park crowd, Valencia looked a threat from the off through the right channel, stinging Mark Howard's palms from 25 yards 20 minutes in before the former Arsenal trainee stood tall to deny the Ecuador striker in a one-on-one.
The visitors were again indebted to Howard when he tipped Ravel Morrison's free-kick around the post and then impressively kept out teenage debutant Reece Burke's acrobatic effort from the resulting corner.
However, Howard was beaten five minutes before the interval as neat play between Morrison and Burke led to the latter swinging a cross in for Sakho, who glanced a header over the goalkeeper and into the net.
Allardyce's side looked slack immediately after the break, however, and Sheffield United took advantage as Bob Harris zipped up the left flank and his cross was flicked on by Mark McNulty, giving a hapless Reid no chance to react, shinning into his own net.
The West Ham boss reacted by throwing Stewart Downing on for goalscorer Sakho and the winger had an instant impact on the left, crossing for Reid and Valencia, who were unable to convert.
Mauro Zarate was introduced as the hosts grew desperate, the Argentine attacker bringing a scrambled save out of Howard after the keeper had kept out Downing's powerful effort from range, but it was not enough and the game drifted into extra time.
The away side tenaciously defended their goal in the extra periods, blocking a number of shots from West Ham.
Howard again made a stunning save to deny Mark Noble's curler and the hosts failed to craft a telling chance in the final 10 minutes, taking the game to penalties.
After eight perfect spot-kicks, Valencia stepped up for West Ham's fifth, only for Howard to save to his right.
And Doyle took full advantage, sending Jussi Jaaskelainen the wrong way to take Sheffield United to the third round for the first time since 2008.
