Moritz Leitner handed Stuttgart a fifth-minute lead before Kalou levelled from the penalty spot midway through the opening period.
Stuttgart were twice made to pay for losing possession in midfield after half-time, with Kalou and Roy Beerens the beneficiaries before Hertha substitute Sandro Wagner headed into his own net to unwittingly set up a tense finish.
Huub Stevens' Stuttgart broke their Bundesliga duck with a 1-0 win over Hannover last time out and they picked up where they left off when Vedad Ibisevic flicked Florian Klein's cross towards the onrushing Leitner to finish.
Hertha had the chance to level when Carlos Gruezo bundled full debutant Valentin Stocker to the ground inside the box and Kalou made no mistake from 12 yards.
Thorsten Kirschbaum made a fine save from Beerens on the volley to ensure the sides went in at the break all square.
Timo Werner, on as a first-half replacement for Gruezo, passed up a glorious chance to give Stuttgart a 50th-minute lead and it was a wasteful moment they would regret.
The visitors gave up possession through Antonio Rudiger in the 63rd minute, prompting Stocker to burst forward a feed Kalou.
The former Chelsea man benefitted from a fortunate ricochet when he entered the box but fired home in assured fashion.
A similarly clinical attack followed 11 minutes later, with Stocker involved once more before Anis Ben-Hatira crossed for Beerens to volley his first Bundesliga goal.
Wagner replaced goal hero Kalou and offered an unfortunate imitation when he nodded Moritz Leitner's free-kick through his own goal, but the hosts held on to sweeten the capital's Unity Day celebrations.
