Levein, a former international who became manager in December 2009, has departed with Scotland bottom of their 2014 World Cup qualifying group.
Scotland have collected two points from four matches after opening draws with Serbia and Macedonia before defeats in Wales and Belgium last month.
"The Board of the Scottish FA has taken the decision primarily due to the disappointing results in the FIFA World Cup Brazil 2014 Qualifiers," the Scottish FA said in a statement.
Scottish FA chief executive Stewart Regan said Levein had paid the price for the increasing importance of results in modern-day football and Scotland's drop in the world rankings.
"We are taking this decision with real sadness. Craig has worked hard to bring success, and has been thoroughly professional in his approach to the job," said Regan.
"However, he would be the first to agree that football is a results-driven business. The challenge for the squad now is to improve our position in Group A and, subsequently, improve our position in the FIFA rankings."
Regan said the Scottish FA would honour Levein's contract, due to run until the end of the 2014 World Cup qualifying campaign next October.
