The France international had seen numerous opportunities come and go as the match looked to be petering out to a drab goalless draw.
However, substitute Jefferson Montero found Gomis with a low cross from the left in the 87th minute - the striker applying the simple finish to earn Swansea a first win in three.
Villa had won their last three matches in all competitions, but this weekend's clash was in stark contrast to the 4-0 win at Sunderland last time out.
That fixture saw Villa net four goals in the first half, but chances were few and far between for the home side in the opening 45 minutes against Swansea.
In fact, Sherwood's men were booed off by some at half-time.
Things did not get any better for Villa after the break, and Gomis finally took a chance to see Swansea climb to eighth in the top flight.
Both sides made one change at the back, with Alan Hutton replacing Matthew Lowton for Villa and Federico Fernandez coming back into the Swansea side in place of Jordi Amat, but it was the hosts' defensive line that came under most pressure early on.
After Kyle Naughton had seen an effort blocked in the Villa box, Gomis should have done better with his first real opening from six yards – Alan Hutton sliding in to make an excellent intervention.
Gomis was causing problems in the Villa box again in the 22nd minute as he fired against goalkeeper Brad Guzan after being played in by Ki Sung-yueng, before the hosts were forced into a change when Tom Cleverley left the field through injury, with Carlos Sanchez introduced in his place.
It was not until the 33rd minute that Swansea stopper Lukasz Fabianski was called into action for the first time, with Febian Delph seeing his low drive kept out by the Poland international after being teed up by Scott Sinclair – playing against his former club.
Ciaran Clark was in the six-yard box to keep out Routledge after Guzan flapped at a free-kick, and the American was lucky to get away with another fluffed clearance when he kicked the ball straight at Gomis.
Sinclair saw the ball run through his legs as he unleashed an air shot at the beginning of the second half, before Guzan tipped over from a Sigurdsson free-kick.
Gomis endured a largely difficult day at the office and cut a dejected figure after heading well over in the 54th minute.
Gabriel Agbonlahor squandered a glorious opportunity to go ahead soon after – his poke from six yards deflected wide by Neil Taylor - and Gomis' problems in front of goal continued as he saw an effort diverted over by Clark.
However, the 29-year-old eventually found the net three minutes from time to seemingly put the result beyond doubt.
Sinclair came close to snatching a spectacular leveller with a volley soon after, before Routledge should have made it 2-0 when sent through on goal with the match nearing a conclusion, but saw his effort smothered.
