The Frenchman has had to make do with coming off the bench in Swansea's two Premier League fixtures so far, but was given a chance from the start in the second-round clash.
And Gomis certainly gave manager Garry Monk some food for thought as his 22nd-minute strike from the edge of the penalty area proved decisive.
The triumph is Swansea's first in the competition since they beat Bradford City 5-0 at Wembley to win the 2012 final.
In among Swansea's 10 changes from their 1-0 Premier League win over Burnley were debuts for new arrivals Federico Fernandez and Tom Carroll, while Steve Evans made five alterations to his Rotherham side.
And it was the Championship outfit who got the first sight of goal – Michael Tidser volleying wide after the Swansea defence failed to deal with a Ben Pringle free-kick.
At the other end, Dwight Tiendalli's 12th-minute cross just evaded the outstretched boot of Gomis, before stand-in captain Nathan Dyer saw a shot blocked by Craig Morgan.
Swansea started to apply pressure as the half progressed, with Jefferson Montero testing goalkeeper Scott Loach, while Josh Sheehan had an effort deflected wide.
And that dominance told after 22 minutes, when Gomis netted his first goal since arriving from Lyon during the close-season, turning well on the edge of the penalty area after receiving the ball from Dyer before firing home.
Alex Revell headed just over from a Lee Frecklington corner as the visitors went in search of an equaliser, before the latter cleared the crossbar with an effort from the centre of the box.
Kyle Bartley denied Revell another goalscoring opportunity with a last-ditch challenge just after the break, but Swansea continued to threaten in attack - substitute Jonjo Shelvey drawing a save from Loach with a rifled effort from 25 yards.
A clash of heads between Dyer and Mat Sadler caused concern in both dugouts, with the latter subsequently having to be replaced by Joe Skarz.
Gylfi Sigurdsson and Wilfried Bony both came on for Swansea during the second period.
And although the hosts' star names were unable to inspire further goals, boss Monk will be pleased to have safely booked a place in the third-round draw.
