Germany, who beat second-placed Russia 1-0 on Saturday to seal top spot in Group Four, went behind after 11 minutes.
The hosts failed to clear a header by Roman Eremenko and the ball landed at the feet of Jonathan Johansson, who netted on his 100th international appearance.
Having made six changes to the lineup which beat Russia, Germany looked uninspired, failing to create a single scoring chance in the first half.
Lacking stability in their backline and clinical finishing in attack despite playing with three strikers, the Germans were facing what looked like their first defeat by Finland in 86 years.
Finnish striker Jari Litmanen put the ball into the net early in the second half but his effort was ruled off side.
The Germans gradually upped the tempo and the arrivals on the pitch of midfielder Mesut Ozil and striker Miroslav Klose added much-needed pace.
With fans jeering throughout the second half, it was Ozil who lifted the ball into the box, the Finnish defence scrambled to clear and Klose found Podolski to slot home with seconds left.
"This is understandable," Philipp Lahm told reporters of the fans' whistles. "They paid money to see goals and we failed to do that tonight.
"We have found it hard going against defensive teams lately. We did not pressure them and did not get the ball into the box often enough."
