Rangnick, who said he was suffering from exhaustion, replaced Felix Magath in March and led Schalke to the Champions League semi-finals and German Cup victory last season.
"He has stepped down after discussions this morning," a club official told Reuters.
Schalke said while it was not clear how long Rangnick would be out, they would immediately start searching for a replacement.
"This decision comes completely out of the blue and I am very surprised," said club president Clemens Toennies, who will discuss the news in a board meeting later on Thursday.
"As sad as this is, obviously his health is above everything. We must now find the best solution [to replace Rangnick] and not just the quickest," he told reporters.
Schalke, who are ninth in the league, and Rangnick said in a statement the 53-year-old coach had stepped down because of "exhaustion syndrome."
"After a long and mature consideration I decided that I needed a break," Rangnick said. "This decision is very hard but my current energy level is not sufficient to be successful and especially to further develop the team."
Rangnick, who joined Schalke after leaving Hoffenheim in January, will be replaced by interim coach Seppo Eichkorn.
"His decision deserves respect," said Schalke sports director Horst Heldt. "We also believe that health should be put above any professional targets and challenges."
Schalke become the second Bundesliga team to be searching for a coach after bottom-placed Hamburg SV sacked Michael Oenning days ago.
‘After Manchester City’s recent form, maybe they’re the underdogs against Manchester United!’ Former Red Devils defender on this weekend’s derby
‘Arteta, Alonso, Emery, me… none of us were physical players – we needed the understanding of the game. That probably helped us move into management’: Premier League boss reveals reasons for natural career progression
