A drone carrying a flag featuring Albanian national symbols flew over the ground in the first half of the clash, which was abandoned with the score at 0-0.
Serbia defender Stefan Mitrovic jumped to bring the flag down and an unsavoury melee between players from both sides soon escalated as a number of supporters entered the field of play.
Following an initial suspension of play and the deployment of riot police around the pitch, the match was abandoned.
Ivanovic praised his team for being supportive of their opponents, but found the situation upsetting.
"For us it is most important that we as a team were at all times with Albania's team and we offered them support," he said.
"It's sad football finished in second place."
Ivanovic said Albania felt they were in no position to continue the match after an initial postponement.
"A delegate wanted the game to continue, but Albanians footballers said physically they weren't ready for a restart," he said.
"Details that happened, we cannot now observe.
"We wanted to continue the game, but after speaking to the delegate and people in charge of that, a decision was reached for the match to be abandoned."
UEFA has announced it is investigating.
