After the hosts had enjoyed the better of a tight encounter, the turning point came when Kazaishvili was introduced with 13 minutes left in place of Mike Havenaar.
He ultimately made the decisive contribution, running at the home defence and beating Stefano Denswil before curling a shot past Jasper Cillessen in injury time to snatch the win for Vitesse.
That put Peter Bosz's in second, a point behind AZ Alkmaar, who moved top of the table thanks to a 2-0 home win over strugglers ADO Den Haag.
Nemanja Gudelj broke the deadlock in the 42nd minute before Johann Gudmundsson completed the scoring 25 minutes from time.
Elsewhere, PSV Eindhoven failed to win for the third successive league match as this season's surprise package, Zwolle, battled to a 1-1 draw at Philips Stadion.
Jeffrey Bruma gave the hosts a third-minute lead but it did not last long, as Fred Benson netted his fifth league goal of the season just before the quarter hour.
The 29-year-old spoiled his day by earning a second yellow card in the final 10 minutes, although Zwolle held on nonetheless as they ended the day in sixth place, while PSV sit fourth on the same points.
In the day's other game, Twente let a two-goal lead slip at home to NEC Nijmegen as they also dropped points for the third league game in a row.
Shadrach Eghan opened the scoring 23 minutes in, with Quincy Promes adding the second nine minutes later.
However, NEC fought back after the break and they were level by the hour mark as Soren Rieks and Jakob Jantscher found the net for the visitors, who remain bottom despite their point.
‘Arteta, Alonso, Emery, me… none of us were physical players – we needed the understanding of the game. That probably helped us move into management’: Premier League boss reveals reasons for natural career progression
‘England have the players to win the World Cup – it’ll be tough for Thomas Tuchel to do a bad job, with the squad he has at his disposal’ Former Three Lions winger backs new boss after gentle qualifying draw
