Following a quiet first half, Juan Falcon opened the scoring with a curling free-kick on the hour-mark, before Cheick Doukoure powered home a second two minutes later.
Oswaldo Vizcarrondo met a corner at the back post to reduce the deficit with a header in the 71st minute, and Remi Gomis looked as though he would draw Nantes level when he pulled the trigger inside the penalty area nine minutes from time.
His effort was stopped on the line by the hand of Gaetan Bussman, who was swiftly dismissed, with Serge Gakpe converting the subsequent spot-kick to force extra-time.
The match became a fiery encounter thereafter, but it was Nantes who drew first blood in additional time - Johan Audel sending a low drive beyond goalkeeper Anthony Mfa Mezui from the centre of the penalty area.
Metz had a penalty shout turned down four minutes from time, and Ismael Bangoura's sublime lob from outside the penalty area two minutes from the end put the game to bed.
In the day's other last-16 clash, Benjamin Mokulu scored an extra-time winner to earn Bastia a 3-2 win over Caen.
Caen sit bottom of Ligue 1, but took a 12th-minute lead when Sloan Privat headed home from a left-wing cross.
Guillaume Gillet's thunderbolt three minutes before the break restored parity, but Jose Saez appeared to have snatched the win for the visitors when he found the net from similar distance 18 minutes from time.
Famoussa Kone sent the home fans into raptures with a driven effort in the 89th minute to take the match to extra time, and Mokulu scrambled the ball over the line with eight minutes to go.
Dennis Appiah headed against the left-hand post in the closing stages, as Caen agonisingly missed out on a place in the quarter-finals.
