Schalke, lying 10th in the Bundesliga, will face second division Duisburg in Berlin in May after reaching the final for the first time since 2005.
The defeat left last season's domestic double winners Bayern with only the Champions League to fight for following a 3-1 Bundesliga defeat by Borussia Dortmund on Saturday that effectively ended their league title hopes.
"We knew that we would get a chance," said Schalke coach Felix Magath. "We could have added another goal from a set piece but overall I think we deserved to go through even though we did not do much in the second half."
In a rematch of last year's Cup semi-final, visitors Schalke, missing injured Klaas-Jan Huntelaar, made the most of Bayern's sloppy passing game in the first half to dominate in midfield.
The 24-year-old Germany goalkeeper was in stellar form throughout Bayern's second-half charge, ignoring hundreds of Bayern fans opposed to his transfer holding up banners reading "No Neuer".
"This is a big disappointment and my players can still not quite understand how it happened. But that is football, a team defending all the time they get a break and score," Van Gaal said.
