Austria's disappointing Euro 2012 qualifying campaign resulted in Constantini's departure in September.
The 50-year-old Koller, a former Swiss international, coached St Gallen to the Swiss championship in 2000 and had a four-year spell until 2009 at German club VfL Bochum.
He takes over with the aim of taking Austria to the 2014 World Cup, the Austrian Football Federation said.
Koller will not be in charge for the Euro 2012 qualifiers against Azerbaijan and Kazakhstan this week and next week and will officially take over on November 1.
"Austria actually play good football. There are many young players who have great potential," Koller told reporters during his official presentation. "I cam convinced we can go down a good path.
"I am a coach who wants to win games, who wants to be successful. Bringing in two or three young players, that is not success."
Austria are fourth in their Euro 2012 Group A on eight points, six behind second-placed Turkey with Germany already through. They travel to Azerbaijan on Friday before going on to Kazakhstan on Tuesday.
Their goalless draw against Turkey in September ended their chances of playing at Euro 2012 and maintained their record of having never reached the tournament finals through the qualifying competition.
