Milan Jovanovic grabbed the winner in the 38th minute with Germany in disarray moments after striker Klose was sent off for a foul that brought him a second yellow card in an ill-tempered Group D match at Nelson Mandela Bay Stadium.
Germany should have equalised in the 61st when Nemanja Vidic needlessly handled in the area but goalkeeper Vladimir Stojkovic dived to his left to save Podolski's poor spot kick.
"This victory is more than just a victory. This victory has a much greater impact on us psychologically," said Serbia coach Radomir Antic. "With this victory we can decide our own fate. This is a victory for our people who'll know how to celebrate."
Until Klose's dismissal, the Germans had been in control of fiercely contested match, playing with attacking flair, fast copunter-attacks and a fluid movement off the ball that left Serbia on the back foot and staring at an early exit.
The three-times World Cup winners' spurned chance after chance, though, with Sami Khedira volleying over after a neat cross by Thomas Mueller before rattling the crossbar with a ferocious long-range drive in the second half.
The Serbs had only a few chances but went close early on when Milos Ninkovic found himself with time and space but blasted Milos Krasic's inch-perfect cross over the bar while Nikola Zigic's header clipped the upright.
Klose's frustrations mounted after his missed chances and a disallowed goal and he was given his marching orders after a receiving a second booking for bringing down Milos Krasic.
With the Germans stunned, Serbia took full advantage and snatched the lead a minute later with a superb team effort that culminated in Jovanovic's close-range strike.
"We are strong, we were a bit lucky but we deserved to win," said Ser
