The 10-times Cup winners struggled after the Ligue 2 team took an early lead before putting together a combination of tough defence and lightning counter-attacks.
Earlier, Olympique Lyon were also made to work hard and needed two late goals to seal a last-16 top-flight clash with Girondins Bordeaux with a 2-0 win at fourth division Lucon.
France forward Bafetimbi Gomis gave Lyon the lead with 15 minutes left and Lisandro Lopez made sure of victory when he scored from close range after initiating a quick counter-attack with one minute to go.
Lyon struggled to create clear-cut chances throughout the first half but had many opportunities after the interval although they had luck on their side when Lucon struck the woodwork five minutes after Gomis's opener.
In Marseille, Le Havre offered even tougher opposition and took the early lead in the fifth minute when forward Ryan Mendes was set up from long range and ran through to shoot past goalkeeper Steve Mandanda.
Didier Deschamps' squad had their chances before the interval and finally came back when Brazilian striker Brandao headed in a Morgan Amalfitano pass which Le Havre goalkeeper Johny Placide failed to catch.
Le Havre, France's oldest professional team, could have stunned their hosts in the last minute but midfielder Alexandre Bonnet missed an opportunity into an empty net.
Midfielder Morgan Amalfitano finally gave Marseille the lead during extra-time with a powerful shot from outside the box and France Loic Remy scored a last-minute goal.
Marseille will travel to fourth division Bourg-Peronnas, who dumped out Ligue 1's bottom side Ajaccio, in the last 16.
Also on Sunday, third division GFCO Ajaccio, who have already beaten Ligue 1 Toulouse, knocked out Ligue 2 Troyes thanks to a second-half goal by Mickael Colloredo.
Algerian Foued Kadir scored either side of the interval to help top-flight Valenciennes recover from a goal down to beat second division side Bastia who were down to 10 men.
