Avram Grant's men looked destined to crash out of the tournament after Mandla Masango gave South Africa the lead in the 17th minute, but mounted a late fightback and remain in the hunt for the title.
John Boye restored parity 17 minutes from time to give Ghana hope, but they were still on course to be sent home as the clock ticked down.
However, Ayew latched onto the end of a Baba Rahman cross to put the ball into the back of the net in the 83rd minute, and see Ghana into the next round.
Grant's side finish top thanks to their head-to-head record against Algeria, who beat Senegal 2-0. South Africa are eliminated as the group's bottom side.
The South African backline was given a stern examination of its capabilities in the early stages.
After Jonathan Mensah had headed just wide from a free-kick in the fifth minute, Eric Mathoho had to block a Jordan Ayew strike on the line, with goalkeeper Nhlanhla Khuzwayo – one of four changes to the side that drew 1-1 with Senegal - having spilled the ball into his path from an earlier effort.
Ghana, who brought Wakaso Mubarak in for Emmanuel Agyemang-Badu in the middle of the park, were having the better of the early possession, but a moment of inspiration handed South Africa the lead out of the blue in the 17th minute.
Having lost his footing when initially attempting to control a long ball forward with his chest, Masango climbed to his feet before sending a sublime looping volley over the head of Brimah Razak when a failed headed clearance fell his way.
There was concern for South Africa when Khuzwayo went down with an apparent hamstring strain on the half-hour mark, but the goalkeeper was in the right place at the right time to block a rifled Rahman effort from 20 yards soon after.
Mubarak could maybe count himself lucky to be on the pitch after a crunching challenge on Andile Jali deep inside his own half, and South Africa were not even placated with a free-kick as the referee pointed for a corner.
Asamoah Gyan sent a free-kick whistling over the crossbar from just outside the penalty area at the beginning of the second half, before Jordan Ayew saw a speculative long-range effort tipped wide.
John Boye tucked under Khuzwayo after the ball was drilled into the penalty area to give Ghana hope in the 73rd minute, before Kwesi Appiah saw an effort tipped onto the base of the post.
Ghana were throwing everything at finding a winner, and finally got their wish when Andre Ayew headed home seven minutes from time.
Watch every match of the Africa Cup of Nations LIVE on British Eurosport HD or sign up to Eurosport Player and follow the action on mobile, tablet and other devices: www.eurosportplayer.com
