Half-time substitute Cork nodded in Dusan Tadic's 53rd-minute corner for his first Southampton goal before Pelle wrapped up the win in stoppage time.
Pelle had endured a frustrating night in front of goal until scoring, as he spurned countless opportunities.
But the Italian forward – signed from Koeman's ex-club Feyenoord last month – finally converted in the 93rd minute, coolly slotting past Millwall goalkeeper David Forde.
Koeman - drafted in to replace Mauricio Pochettino who left for Tottenham - has been given the tough task of rebuilding Southampton after a spate of close-season sales.
The Dutch manager has also brought several players in, though, and it was one of his new arrivals, Tadic, who stole the show in front of a small crowd at The Den.
Koeman showed he was taking the competition seriously by making just three changes, as Shane Long – who joined for a reported £12million from Hull City earlier this month – started for the visitors, alongside debutant left-back Matt Targett.
Pelle had two early half-chances for Southampton, but failed to hit the target, while Millwall, who made nine changes themselves, saw Byron Webster flash a header wide from Jack Powell's near-post corner.
Lloyd Isgrove hobbled off for Southampton prior to the half-hour mark, replaced by the influential Tadic, before Millwall's Magaye Gueye wasted a great chance to give his side the lead in the 33rd minute.
Forde was again called into action as Pelle's shot on the turn was turned behind for a corner, but from the resulting set-piece, Southampton did score.
Cork, who replaced Victor Wanyama at the break, timed his run to perfection and powered a header beyond Forde to delight the travelling fans.
Southampton did not let up after the goal, either, as Pelle headed a Nathaniel Clyne cross against the woodwork before James Ward-Prowse curled just wide from 17 yards.
Pelle's night of frustration continued as he controlled under pressure, only to see a back-heel diverted wide by Millwall's Shaun Williams.
Shortly after shooting wide again, he then thrashed an effort at Forde that required a smart stop.
Steven Davis played in Pelle and the forward finally got a reward for his persistence, as he fired his new club into the competition's third round.
