The 23-year-old Blackburn Rovers striker, who came through the ranks at Hajduk Split, completed Croatia's recovery with a 78th-minute effort that sent the home crowd into raptures.
"The winning goal will come as a tremendous confidence boost after I was starved of first-team football at Blackburn for the best part of this season," a delighted Kalinic told Croatian television.
"The most important thing, though, is we fought back after falling behind to earn three priceless points."
The victory lifted Croatia to the top of Group F with 13 points from six matches, ahead of Greece who have 11 from five and Israel who are on 10 points from six games.
The Croatians had drawn six and lost three matches at Poljud since first playing there as an independent nation in 1995 and the jinx seemed set to continue when they fell behind against the run of play.
Georgia winger David Siradze was given too much room to operate on the right flank and Jaba Kankava ghosted in at the near post to head his pinpoint cross past Croatia keeper Vedran Runje after 17 minutes.
NEAT FINISH
The home side, repeatedly frustrated by a packed Goergian defence, hit the woodwork three times before Mario Mandzukic equalised in the 76th minute with a neat finish from the edge of the six-yard box.
Second-half substitutes Ivan Klasnic and Tomislav Dujmovic carved Georgia open with a delightful one-two and the latter squared the ball for Mandzukic to steer it past keeper Giorgi Loria with a deft touch.
Roared on by 35,000 fans who lit flares and set off fireworks, the Croatians went marauding down the right flank almost straight from the restart and got their reward through some superb individual skill from their forwards.
Former Arsenal striker Eduardo teased and taunted three defenders before threading a superb low cross for Kalinic to slide in and drill the ball past Loria.
"I have to congratulate my players for showing resilience and character after a poor first half to keep their composure and claw out a very important win from the jaws of defeat," Croatia coach Slaven Bilic said.
"It's difficult to break down teams like Georgia who defend with 10 men behind the ball but the substitutions paid off and so did the second-half strategy to play the ball wide and make deep runs into the penalty box whenever possible.
"We were shaken after falling behind and the players were too impatient in the first half but I believe we are now in a good position to qualify for the finals," added Bilic.
