Austria international Fuchs scored with his smooth left foot, taking a pot shot in the 77th minute to put Schalke top of Group B on 11 points and through to the next round.
Arsenal beat Montpellier 2-0 in the other Group B game to move up to 10 points and also make sure of advancing. Olympiakos were left in third place on six points and, with one game remaining, beyond any hope of surviving in the competition.
"I do get angry if I see a game like this with these chances," Schalke coach Huub Stevens told reporters.
"But at least we did not allow them many chances. We also had lots of shots towards goal though few between the posts and the crossbar.
"At the end it is all about winning and when you reach the knockout stage with a home win you have to be satisfied."
"We failed to get what we wanted," said Olympiakos coach Leonardo Jardim. "It was balanced though we were not good at the start and then we conceded a goal from a long way out."
Huntelaar's sizzling low drive narrowly missed the target in the 31st. The Greek champions tried to catch Schalke on the break and one effort nearly paid off but Djamel Abdoun fired the ball over the bar.
Fuchs's strike settled their nerves and both Julian Draxler and Huntelaar came close to adding another goal late in the game but were again denied by Carroll.
"That was my most beautiful one [goal]," Fuchs told reporters. "We had a lot of chances in the first half so I am just happy mine went in."
