The Qatar-owned club were leading the quarter-final, second leg 2-1 as the game went into stoppage time before Dortmund, who were held to a 0-0 draw in last week's first leg in Spain, first equalised and then grabbed a 93rd-minute winner.
Television replays showed several players from the Bundesliga side in offside positions in the build-up to the third goal and normally restrained Malaga coach Manuel Pellegrini slammed the match officials.
"On this occasion we could not, or they did not want us to, get through," Pellegrini told a news conference.
"After we went 2-1 ahead there was no refereeing," the Chilean added.
"They forced us back with elbows and shoves. There were two sendings off that were not given, a double offside in the third goal which should not have counted.
"They said this was the best team in Europe and by the end they were hoofing long balls forward. We are leaving with very bitter feelings."
Winger Joaquin, who put Malaga ahead in the 25th minute, and club owner Sheikh Abdullah Al Thani, a member of the Qatar royal family, were more outspoken.
In a series of tweets on his official Twitter feed soon after the final whistle, Al Thani blamed the defeat on "racism" and called on governing body UEFA to launch an inquiry.
Joaquin suggested UEFA President Michel Platini may have had something to do with the defeat.
Platini is deeply unpopular in Malaga after UEFA slapped a ban on the club from competing in continental competition from next season due to delays in payments to creditors.
"We suspect Platini and all the rest of them who are involved," Joaquin told Spanish radio.
"Because we are Malaga and not Real Madrid it's easier to do this to us," he added.
Whatever conclusions are drawn about Tuesday's game, Malaga outperformed on their first foray into Europe's elite club competition but now need to put their disappointment behind them and focus on qualifying again for next season.
They have appealed the UEFA ban at the Court of Arbitration for Sport (CAS) and are currently sixth in La Liga, four points behind Real Sociedad who occupy Spain's fourth Champions League qualification berth.
"We all made a super-human effort and going out of the competition because of a mistake like that is painful," forward Roque Santa Cruz told Spanish television.
"We were four minutes away from semi-finals and it was snatched away from us," the Paraguayan added.
"Now we have to lift our spirits and tackle the rest of the season in the best possible way."
