Rangnick, who said he was suffering from exhaustion, replaced Felix Magath in March and led Schalke to the Champions League semi-finals and German Cup victory last season.
"He has stepped down after discussions this morning," a club official told Reuters.
"This decision comes completely out of the blue and I am very surprised," said club president Clemens Toennies, who will discuss the news in a board meeting later on Thursday.
"As sad as this is, obviously his health is above everything. We must now find the best solution [to replace Rangnick] and not just the quickest," he told reporters.
Schalke, who are ninth in the league, and Rangnick said in a statement the 53-year-old coach had stepped down because of "exhaustion syndrome."
"After a long and mature consideration I decided that I needed a break," Rangnick said. "This decision is very hard but my current energy level is not sufficient to be successful and especially to further develop the team."
"His decision deserves respect," said Schalke sports director Horst Heldt. "We also believe that health should be put above any professional targets and challenges."
