Lorient and second division Angers were the other two clubs to qualify for the quarter-finals.
Rennes were hoping to bounce back from their 5-1 Ligue 1 thrashing by Sochaux at the weekend but instead suffered further humiliation against a team languishing in 17th place in Ligue 2.
Reims goalkeeper Kossi Agassa proved to be the hero of the evening after Rennes has fought back from a 3-1 down to 3-3 thanks to two second-half goals by substitute Jerome Leroy.
With a few seconds of normal time to go, Rennes striker Julien Toudic hit the crossbar before Agassa stopped Yacine Brahimi's penalty and forced the match into extra-time.
In the opening seconds of extra-time, Rennes were awarded another penalty but Leroy failed to convert the spot-kick.
Three minutes later, Maxime Thonnel netted the winner while an acrobatic Agassa produced more scintillating saves to help his team through to the next round.
"I can only thank Kossi for his amazing match," Toudic told Eurosport. I had predicted it. We have two legs and two arms just like them. Now we will go all the way."
Lorient breezed past second-division side Metz 3-0 thanks to two goals from in-form striker Kevin Gameiro.
Angers initially struggled against third-division Strasbourg but prevailed 2-0 with late goals from strikers Gaetan Charbonnier and Fode Dore.
