Praising Kawashima's performance in Japan's surprise 1-0 victory over France in a friendly in Paris last Friday, Laurent Ruquier's wisecrack drew laughter and applause in the studio.
However, Japan's Chief Cabinet Secretary Osamu Fujimura on Tuesday called Ruquier's remark - a reference to last year's nuclear crisis in Fukushima - "inappropriate".
The letter said the remark "hurts the feelings of people affected by the disaster and hinders efforts for reconstruction," Fujimura added.
Japan's education minister Makiko Tanaka, whose portfolio also includes sports, was far from amused.
"If it's true [he said it] it is behaviour which lacks sensitivity," she said. "Many people were hurt in the nuclear crisis. I don't know how people can make fun of it."
"It's very sad something like this happens just when people's lives are finally starting to get back to normal," JFA vice president Kozo Tashima said.
"This cannot be dismissed simply as a joke."
Belgian-based keeper Kawashima suffered sick abuse by fans during a game last year when opposition fans chanted "Kawashima, Fukushima!"
