The result left Serbia, who missed a penalty in the match, third in the group on 15 points, one behind Estonia who finished their campaign on Friday with a 2-1 win in Northern Ireland.
Italy, who had won the group and qualified automatically with two games to spare, finished top with 26 points after a 3-0 home win over Northern Ireland.
Needing a win to leapfrog Estonia, the Serbs threw men forward from the kick off but were downed by a freak goal from Dare Vrsic on the stroke of half-time.
The midfielder floated a free-kick from 40 metres into the penalty area and the ball bounced over the line off the underside of the bar after it caught keeper Bojan Jorgacevic stranded, too far off his line.
Both sides had missed several good chances in a lively first half, with Vrsic firing just over the bar from 20 metres while Milos Ninkovic skied a close-range shot over the bar after a sharp low cross from Aleksandar Kolarov.
Defender Nemanja Vidic missed Serbia's best chance to get back into the match in the 65th minute, allowing keeper Jasmin Handanovic to easily smother his feeble penalty after Kolarov was hauled down by centre-back Marko Suler.
Substitute Marko Pantelic had the ball in the net for Serbia in the closing stages but his effort was rightly disallowed for marginal offside after Handanovic parried a fierce shot by Dragan Mrdja.
The Slovenia keeper produced another fine save when he kept out a Milos Krasic volley, while Tim Matavz shot just wide after a good solo run at the other end.
