The Highlands club took the lead after just six minutes as Richie Foran headed home from point-blank range after good work from Billy McKay and Aaron Doran.
Winless Kilmarnock levelled in bizarre fashion just six minutes into the second half, Graeme Shinnie's clearance cannoning off Barry Nicholson and into the net.
But parity was shortlived for the hosts, though, with McKay lobbing home the winner after 56 minutes to secure Inverness' fifth win of the campaign.
Kris Commons gave Neil Lennon's side the advantage from the penalty spot after 19 minutes, but Gary Locke's men responded early in the second half as Jason Holt beat Fraser Forster with a low drive.
Anthony Stokes restored Celtic's lead, before recent signing Teemu Pukki wrapped things up with a goal on his debut for the club.
Motherwell continued their strong start to the season with a 1-0 triumph at St Mirren, John Sutton's strike seven minutes before the interval giving Stuart McCall's men all three points.
Elsewhere, Hibernian came from behind to beat St Johnstone 2-1, Paul Heffernan and James Collins turning things around for Pat Fenlon's men, who extended their unbeaten run to four matches after Stevie May had given the hosts the lead.
