Centre-back Rami found the back of the net with three minutes left after Franck Ribery and Mathieu Debuchy had cancelled out Birkir Bjarnason and Kolbeinn Sigthorsson's first-half goals for the visitors.
France, however, produced a terrible defensive display with left-back Patrice Evra at fault for both goals two days before coach Laurent Blanc has to trim his Euro squad from 25 to 23 players.
Les Bleus, in Group D at Euro 2012 with Ukraine, England and Sweden, host Serbia in Reims on Thursday before taking on Estonia in Le Mans on June 5.
"We cannot pretend the first half did not happen," Blanc told French TV channel TF1 after France extended their unbeaten run to 19 games.
"It is hard to find the balance between the attack and the defence," said the former world champion, who fielded Yohan Cabaye as the only holding midfielder.
It was the first time Karim Benzema, Jeremy Menez, Samir Nasri and Hatem Ben Arfa started a game together since they won the Under-17 European championship in 2004.
Iceland went ahead in the 29th minute when Bjarnason fired an angled shot past Steve Mandanda.
Sigthorsson made it 2-0 six minutes later when he latched on to a fine cross from the right at the end of a sharp counter-attack that left Evra trailing in Rurik Gislason's slipstream.
Seven minutes into the second half, right-back Debuchy pulled one back, scoring from close range after being set up by Benzema.
Yoann Gourcuff, one of a handful players not guaranteed a spot in the final squad, produced an unconvincing performance and was replaced with 15 minutes remaining by Ribery who had an almost immediate impact.
Ribery lobbed Hannes Halldorsson in the 85th after a fine one-two with substitute Olivier Giroud who also had a hand on the winner.
Giroud headed the ball into the path of Rami who fired past Halldorsson to give France a flattering victory.
