Substitute Gouffran struck 17 minutes from time to secure victory for Alan Pardew's men after Christian Benteke had cancelled out Hatem Ben Arfa's early opener.
Newcastle headed into the contest after an international break that has seen supporters voice their anger at the lack of signings by owner Mike Ashley and director of football Joe Kinnear.
But the old guard produced the goods at Villa Park as the visitors extended Villa's streak without a Premier League clean sheet to a staggering 27 matches.
Loic Remy made his first start for Newcastle after coming off the bench in the 1-0 win over Fulham prior to the international break, while Yohan Cabaye, who had been linked with a move to Arsenal in the window, was named in the starting XI for the first time this campaign.
And the visitors were rewarded for their early dominance after 18 minutes when Ben Arfa slammed home from point-blank range after Cisse had flicked Remy's left-wing cross into his path.
Both sides continued to push forward following the opener, with Remy at the fore for Pardew's men as his pace caused plenty of problems for the Villa defence.
Still, it was Pardew’s side who should have scored the game’s second goal on the stroke of half-time when Cabaye lashed a volley just past the post after collecting a superb pass from Moussa Sissoko.
The pace of Remy and Ben Arfa was continuing to trouble Lambert's side, the latter denied a second with a curling effort by Brad Guzan’s strong save.
Remy was brought off shortly afterwards - Gouffran coming on in his place - and Newcastle soon conceded the initiative as Benteke levelled matters with a bullet header. The Belgian international met Ashley Westwood’s excellent delivery to fire home.
However, Newcastle regained the advantage six minutes later – and Ben Arfa was heavily involved once again. He produced a fierce left-footed shot that Guzan could only parry and Gouffran was on hand to slot into the empty net.
