Both second-half goals came from France left-back Patrice Evra's side of the defence as Germany fought back after Mathieu Valbuena had opened the scoring on the stroke of half-time.
Valbuena headed home from close range in the 44th minute after Karim Benzema's free-kick bounced off the bar, but Muller slotted past Hugo Lloris early the second half and Khedira wrapped it up from Mesut Ozil's pass with 17 minutes left.
France, reeling following a poor run that lasted from 2008-12, drew 1-1 with Spain in a World Cup qualifier and beat Italy 1-0 in a friendly in their last two outings but the world's second-ranked team proved a bridge too far.
They became the first German team to beat France since East Germany's 1-0 win at the Parc des Princes in a Euro 88 qualifier in November 1987. That year, West Germany also beat Les Bleus, 2-1 in a friendly in Berlin with goals by Rudi Voller.
"We put a good tempo into the game, we had good organisation which helped us win a lot of the ball," German coach Joachim Low told a news conference.
"We were the best team tonight."
His counterpart Didier Deschamps said the teams had been "blow-for-blow" for most of the encounter, the Frenchman left to rue his side's errors and Germany's killer instinct.
"We showed some interesting things offensively but we made some mistakes and Germany showed they have great qualities. They are fast on the break."
Khedira's low shot from just outside the box went just wide in the fourth minute as Germany got off to a confident start.
Per Mertesacker's header was tipped on to the bar and over by Lloris in the 17th minute.
