The hosts, backed by a passionate home crowd which clearly believed in the possibility of an upset and even jeered the German national anthem, huffed and puffed and had their opponents on the back foot for most of the time before slumping to another heroic defeat.
Marco Reus and Mesut Ozil, from a penalty, scored either side of half-time to set Germany on the way to an eighth successive win against their smaller neighbours and leave them with six points from their opening two Group C games.
The Germans, who extended their unbeaten run in qualifying matches to 24 games, still had to cling on through a difficult last half-hour after Zlatko Junuzovic pulled one back for the hosts and they had a huge let-off with four minutes to go.
Substitute Jakob Jantscher burst down the left and delivered a low cross which fiery striker Arnautovic scuffed wide from only three metres.
"That was a head in your hands moment," Austria coach Marcel Koller, making his competitive debut with the team on a balmy September evening, told reporters.
"It was same story as in the last few years when Germany and Austria played, We played well, we created lots of chances but Germany won the game,.
"In the first half, we pressed them but we lacked precision, we didn't score a goal although we had five or six good chances and Germany had two.
"But that is what makes a team world class, when their opponents make mistakes, they pounce and that is the difference."
Coach Joachim Low complained that Germany allowed Austria back into the game and singled out Philipp Lahm for criticism after the Germany captain played a back-pass to Austrian substitute Guido Burgstaller, who was foiled when goalkeeper Manuel Neuer grabbed the ball at his feet.
"We had some good spells when we controlled the game, but not for the whole match," he said. "We had fewer chances and Austria should have equalised at the end."
The win left Germany with six points from their opening two games in Group C while Austria, who began with nine Bundesliga players in the starting line-up, have yet to get off the mark.
MAJOR TOURNAMENT
The hosts have not qualified for a major tournament since the 1998 World Cup but were convinced that this time, with an improving team, they could end their dismal run against their neighbours.
Holger Badstuber rescued Germany after only four minutes with a last-ditch tackle to foil Martin Harnik after he had a clear run on goal.
Both goalkeepers had let-offs from botched clearances.
Austria's Robert Almer made a complete hash of a back-pass from Christian Fuchs but the ball went out for a corner while Neuer hit one effort straight at Andreas Ivanschitz but the ball ricocheted wide of his goal.
Harnik and Andreas Ivanschitz, mysteriously overlooked throughout the two-year reign of Koller's predecessor Didi Constantini, also sent long-range shots fizzing wide of the goal as Austria's unrefined and aggressive approach unnerved the visitors.
