Arsenal looked down and out after Andy Najar's 71st-minute header gave Anderlecht a deserved lead in the Group D UEFA Champions League encounter.
But Anthony Vanden Borre hit the post and Anderlecht wasted other late chances, for which they were made to pay.
Gibbs first drew Arsenal level in the 88th minute, volleying home at the back post before Podolski rifled in from close range in the 91st minute.
Chancel Mbemba Mangulu could not clear Alexis Sanchez's cut-back and the rebound fell invitingly to the Germany international, who sent the away fans wild when he fired home from four yards.
And the win means Arsenal are in a strong position to qualify from Group D, with their last-gasp triumph putting them five points clear of both Anderlecht and Galatasaray - who were beaten 4-0 by Borussia Dortmund - in second.
With Wojciech Szczesny suspended and back-up David Ospina sidelined with a thigh injury, Arsene Wenger handed a start in goal to former Argentina youth international Emiliano Martinez.
Martinez conceded five times in his last Arsenal appearance, an incredible 7-5 win over Reading after extra time in the League Cup's last 16 in October 2012.
But a repeat never looked likely in an uninspiring first half, which saw Arsenal shade the contest but fail to create many meaningful chances.
The closest Arsenal came was in the 16th minute, when Danny Welbeck headed Alexis' cross over after a quick break, while the Chilean, Santi Cazorla and Per Mertesacker all missed the target with opportunities.
Anderlecht were far from threatening, either, and a series of blocks and defensive headers from Arsenal meant it took until the 39th minute before Martinez was tested, as he rushed off his line to deny to Dennis Praet from an acute angle.
Alexis had a good chance to give Arsenal the lead five minutes into the second half, but with his back to goal, his flick - which came off his shoulder - could not beat goalkeeper Silvio Proto from Cazorla's free-kick.
Aaron Ramsey then fired wide after Alexis' cut-back, before Gohi Cyriac shot at Martinez at the other end.
Anderlecht were growing into the game, as Praet fired over, and shortly after Cazorla's close call - which saw Proto deny him with an excellent save - the hosts took the lead.
Najar inspired the sweeping counter-attack, bursting down the right before feeding Praet, who composed himself and swung in an excellent cross.
And Najar continued his run, nodding a free header past Martinez and into the bottom-right corner.
Vanden Borre could have wrapped up the win for Anderlecht with 11 minutes left, but his mis-hit effort struck the post after good work from Najar and Praet, and Arsenal made them pay - Gibbs volleying a side-footed finish past Proto from Calum Chambers' cross.
The goal lifted Arsenal and they surged forward in search of a winner - one that came from the boot of Podolski.
Thank you for reading 5 articles this month* Join now for unlimited access
Enjoy your first month for just £1 / $1 / €1
*Read 5 free articles per month without a subscription
Join now for unlimited access
Try first month for just £1 / $1 / €1