Klinsmann was told after a morning meeting with board members Uli Hoeness, Rummenigge and Karl Hopfner. Assistant coaches Martin Vasquez and Nick Theslof were also sacked.
Hoeness said the coach had been surprised and "even shocked."
"Naturally, I am very disappointed at the moment," Klinsmann said in a statement. "We have set the foundation for the future. I still believe the team can win the championship."
The defeat by Schalke piled the pressure on the former Germany coach, with thousands of fans chanting "Klinsmann out".
"We needed this solution," said Rummenigge. "We trust Jupp, with his calmness and experience to steer things in the right direction."
"I am doing this for Bayern because they were the stepping stone for my international career and because of my friendship with Uli Hoeness," Heynckes told the online edition of the Rheinische Post newspaper.
"I want to make every player a bit better after every game," he said.
Bayern's defence has also been a far cry from last season's rock-solid backline.
The job was Klinsmann's first club assignment after the former international striker, who also played for Bayern, steered Germany to third place in the 2006 World Cup on home soil.
