A one-minute period at the end of the first half turned the match in Malmo's favour as Konstantinos Mitroglou had a goal rightly ruled out for offside before the hosts took the lead from the resulting free-kick.
Olympiacos failed to clear the routine ball forward and Rosenberg took advantage to put Malmo 1-0 up after 42 minutes, having been second best up to that point.
Rosenberg's opener completely transformed the pattern of the match that Olympiacos entered off the back of a shock win over Atletico Madrid in their Group A opener.
Malmo looked far more comfortable after the break and Olympiacos left it too late to push men forward in search of an equalizer that then created the gaps for Rosenberg to seal the result late on.
As you would expect from Malmo's first home game back in Europe's elite club competition since 1990, they were welcomed by a raucous atmosphere.
That was soon punctured, however, as a series of well-anticipated interceptions were all that kept the visitors from taking the lead in a one-sided opening 15 minutes following good work from Ibrahim Afellay and Delvin Ndinga down the Olympiacos left.
However, that defence could do little as Luka Milivojevic powered at goal from a free-kick in the 21st minute, but luckily for Malmo, goalkeeper Robin Olsen was equal to the strike and Giannis Maniatis' volley from the resulting corner.
Despite that dominance it was Malmo who created the first clear opening three minutes later when Magnus Eriksson's diagonal, left-footed cross from the right was met by a Rosenberg diving header five yards from goal, only to be wonderfully saved by Roberto Jimenez.
Mitroglou had the ball in the net for the visitors after 42 minutes, but it was ruled out for offside.
And Malmo opened the scoring from the free-kick that followed.
Isaac Thelin flicked on Olsen's long punt that Alberto Botia looked to have dealt with before a breakdown in communication with his goalkeeper allowed Rosenberg to nip in and roll into an empty net.
Thelin had a fantastic opportunity to double the lead on the hour mark when the rebound from his own close-range header fell at his feet four yards from goal, but he miscued his half-volley, squandering a shot at an open net.
Olympiacos showed little sign of leveling until Alejandro Dominguez's low right-wing cross reached Mitroglou at the back post in the 68th minute. His shot was saved before the rebound sat up nicely for Maniatis, whose follow-up was cleared off the line.
Gaps began to appear but Olympiacos failed to fashion any chances of note and were undone by a long ball once more in the 82nd minute.
Thelin was involved again, flicking on for substitute Pawel Cibicki to occupy a defender, and the goalkeeper, before poking across goal for Rosenberg to finish and seal a historic win.
Malmo travel to Atletico Madrid in their next group match, while Olympiacos host leaders Juventus.
‘Maybe I’ll be signed by Wrexham, who knows? English football would suit me – if an offer came, I wouldn’t think twice, I’d go immediately’ Euro 2024 cult hero reveals his UK dream
‘Managing Leeds? It was an option that appeared, but it wasn’t the right timing. I decided it wasn’t a good idea to leave the club I was at mid-season’: Premier League boss admits to turning down opportunity to replace Jesse Marsch in 2023
