Former Real forward Klaas-Jan Huntelaar threatened a famous upset with a stunning 85th-minute strike that meant Schalke needed to score just once more to progress on away goals.
The holders' 2-0 victory in Gelsenkirchen in the first leg ultimately proved enough to see them through 5-4 on aggregate - but the Bernaneu faithful were less than impressed, as they greeted the final whistle with a deafening chorus of jeers as Real's relative slump in form continued.
Cristiano Ronaldo had twice saved Real in a poor first-half showing as his headers cancelled out goals from Christian Fuchs and Huntelaar - the Portuguese's double making him the highest goalscorer in European competition.
Karim Benzema put Real ahead on the night shortly after half-time, only for Leroy Sane to curl home an exquisite equaliser for Schalke, before Huntelaar's superb finish set up a grandstand ending to the game.
Real just about held on to deny Roberto Di Matteo's impressive Schalke side, but the pressure continues to mount on Carlo Ancelotti after another below-par display.
Ancelotti's men - who slipped off the top of La Liga at the weekend - were poor from the off, and Schalke capitalised to take a deserved lead in the 20th minute.
Tranquillo Barnetta's curling cross from the right found Fuchs at the back post and his shot was too powerful for Iker Casillas - who may feel he should have made the save having got two hands on the ball.
The visitors' joy proved short lived as Real struck back five minutes later, with Toni Kroos' outswinging corner met by Ronaldo's bullet header that flew into the far corner.
Schalke were unfazed, though, and Huntelaar was desperately unlucky to see a stunning 20-yard half-volley rebound off the crossbar before the Dutchman deservedly restored the Bundesliga outfit's advantage soon after.
Max Meyer's original shot was parried by Casillas and Huntelaar pounced to slam home the rebound.
But once again Ronaldo saved Real, their talisman nodding home Fabio Coentrao's exquisite left-wing delivery just before the break.
Real moved ahead for the first time on the night eight minutes after the break, Benzema showing great composure to beat a couple of defenders and rookie goalkeeper Timon Wellenreuther before tucking into the unguarded goal.
Schalke refused to lie down, though, as teenager Sane marked his European debut with a sublime left-footed effort that curled past a despairing Casillas.
Real responded well, but were made to endure a nervy ending when the returning Luka Modric's misplaced pass was picked up by Huntelaar, who slammed his shot off the underside of the bar from the edge of the penalty area.
The Bernabeu's discontent was in full evidence by this point and Casillas prevented a full-on mutiny when he denied Benedikt Howedes in the closing stages, as Real just about survived an almighty scare.
