Zambia, who will face Nigeria in the last eight, finished in a three-way tie in the group but came out on top by virtue of the number of goals scored against the other three teams.
They were level on points with Cameroon and Gabon and also tied on goal difference but in a mini-league between the trio had scored four goals to Cameroon's three and Gabon's two.
Gabon were eliminated despite upsetting Cameroon in the group's opening game last week and also drawing with Tunisia.
But on Thursday they were outplayed by quick-fire Zambia, who had several early chances before a through pass from Felix Katongo saw Rainford Kalaba slide the ball past onrushing goalkeeper Didier Evono to open the scoring in the 28th minute.
James Chamanga'ssimple tap-in saw Zambia extend their lead just after the hour when he was left unmarked to receive Jacob Mulenga's square pass with Gabon's defence out of position.
Mulenga squandered several other chances while captain Chris Katongo was also wide of the target with his shooting.
A goal seven minutes from the end by Gabon substitute Fabrice do Marcelino set up an exciting finish.
Gabon needed a draw to win the group but wasted two late long-range freekicks as Zambia held on to reach the last eight for the first time in 14 years.
"People were laughing at our chances and all the betting had us as the worst team here but today we proved we are good enough to reach the quarter-finals," said Zambia's youthful French coach Herve Renard.
Gabon celebrated at the final whistle after being told by officials they had qualified but were later crestfallen when informed of the final placings.
"We thought we had qualified but then they told us no. It is heartbreaking but this is what sport is all about," said captain Daniel Cousin.
Cameroon and Tunisia drew 2-2 in the group's other match in Lubango on Thursday.
