Ramalho, who was hospitalized in January with diverticulitis - a digestive disease, met with Sao Paulo president Carlos Miguel Aidar and vice president of football Ataide Gil Guerreiro after expressing concerns about whether he could lead the team.
Last year, Ramalho spent time in hospital with tachycardia - a faster than normal heart rate.
While Guerreiro claimed Sao Paulo were happy with the job Ramalho was doing, he told the club's website they had to let the 59-year-old go.
"Aidar and I had a long meeting with Muricy," Guerreiro said.
"I wanted the whole way for him to continue with us. He was doing a good job, it was good for us, but we could see he was debilitated."
Interim technical coordinator Milton Cruz will coach Sao Paulo for "three, four games" until the club can find a replacement for Ramalho, according to Guerreiro.
Ramalho led Sao Paulo to three straight Brazilian Serie A titles starting in 2006 - during his second time at the club - but has not won a trophy since returning in 2013.
Sao Paulo sit second in their Copa Libertadores group after four matches, while they lead their group in the Campeonato Paulista.
"He [Ramalho] leaves now, we agree [he is] to leave, because the doors open are always here," Guerreiro said.
"He did excellent work."
Ramalho first coached Sao Paulo from 1994-1996, rejoined the club from 2006-2009 and has been back since September 2013.
The Sao Paulo native also played for the club in the 1970s but is better known for his coaching career, having won the 2010 Brazilian Serie A with Fluminense and the 2011 Copa Libertadores with Santos.
