Half-time substitute Cork nodded in Dusan Tadic's 53rd-minute corner for his first Southampton goal before Pelle wrapped up the win in stoppage time.
Pelle had endured a frustrating night in front of goal until scoring, as he spurned countless opportunities.
But the Italian forward – signed from Koeman's ex-club Feyenoord last month – finally converted in the 93rd minute, coolly slotting past Millwall goalkeeper David Forde.
Koeman - drafted in to replace Mauricio Pochettino who left for Tottenham - has been given the tough task of rebuilding Southampton after a spate of close-season sales.
The Dutch manager has also brought several players in, though, and it was one of his new arrivals, Tadic, who stole the show in front of a small crowd at The Den.
Koeman showed he was taking the competition seriously by making just three changes, as Shane Long – who joined for a reported £12million from Hull City earlier this month – started for the visitors, alongside debutant left-back Matt Targett.
Pelle had two early half-chances for Southampton, but failed to hit the target, while Millwall, who made nine changes themselves, saw Byron Webster flash a header wide from Jack Powell's near-post corner.
Lloyd Isgrove hobbled off for Southampton prior to the half-hour mark, replaced by the influential Tadic, before Millwall's Magaye Gueye wasted a great chance to give his side the lead in the 33rd minute.
A goal-kick from Forde caught Southampton napping and allowed Gueye through on goal, but the former Everton man could not compose himself and fired horribly wide with his left foot.
Jermaine Easter went close just after the break for Millwall, just missing the target after cutting inside, before Forde made a stunning save to tip a powerful Tadic effort over the bar.
Forde was again called into action as Pelle's shot on the turn was turned behind for a corner, but from the resulting set-piece, Southampton did score.
Cork, who replaced Victor Wanyama at the break, timed his run to perfection and powered a header beyond Forde to delight the travelling fans.
Southampton did not let up after the goal, either, as Pelle headed a Nathaniel Clyne cross against the woodwork before James Ward-Prowse curled just wide from 17 yards.
Pelle's night of frustration continued as he controlled under pressure, only to see a back-heel diverted wide by Millwall's Shaun Williams.
Shortly after shooting wide again, he then thrashed an effort at Forde that required a smart stop.
Williams fired at Southampton goalkeeper Fraser Forster at the other end but the visitors were still on top, with Tadic just curling wide, before they eventually got their second goal.
Steven Davis played in Pelle and the forward finally got a reward for his persistence, as he fired his new club into the competition's third round.
