Seydou Keita scored the decisive kick after Gabon's talisman Pierre-Emerick Aubameyang had his effort saved, the only player to fail in the nail-biting shootout.
The shock loss for Gabon, who were just six minutes from success, left some 40,000 at the new stadium in the capital silent while Mali celebrated unlikely progress.
They now meet the highly-fancied Ivory Coast in the semi-final in Libreville on Wednesday.
"It was not easy but we have reached the semi-finals. It was not Barcelona or Brazil but we worked within our capabilities," Keita, who plays at the Spanish club, told reporters.
Mali had snatched an equaliser in the 85th minute through substitute Cheick Tidiane Diabate to force extra time and go on and win the game.
He squeezed the ball under the body of the Gabon goalkeeper Didier Ovono after a smart turn inside the penalty area.
Gabon had looked to be on their way to a fourth successive win at the tournament they are co-hosting with Equatorial Guinea when Eric Mouloungui scored from close range in the 55th minute after a pull-back from Aubameyang.
NERVY START
Gabon, who overcame a nervy start to impose themselves on the game, also twice hit the woodwork.
Aubameyang was unable to keep up his run of a goal per match at the tournament but struck the upright in the 29th minute after beating the offside trap and running onto a through ball from Daniel Cousin.
Cousin's effort four minutes after Mouloungui's goal also bounced back off the post.
"Had those two chances gone in, it might have been a different result. It's very disappointing to lose in this way," said Gabon coach Gernot Rohr.
Mali, whose defence looked disorganised for most of the match and for whom talisman Keita proved largely ineffective in midfield, created few chances.
They might have equalised with 20 minutes to go had Modibo Maiga not stumbled as he shot with just the goalkeeper to beat, minutes before Diabate made it 1-1.
The match kicked off with the stadium only half full with thousands still streaming in up until half-time, a reflection of the draconian security arrangements around the venue.
