The Kazakhstan outfit have already beaten BATE Borisov and Skenderbeu Korce to reach the play-off round, and put one foot in the group stages after their success over the 1966-67 European Cup winners at the Astana Arena.
The Scottish champions were guilty of some poor defending and wasting chances and will have a huge task to overturn the deficit in the second leg.
Andrey Finonchenko grabbed the opening goal when he pounced on a loose ball from a long throw in and Sergei Khizhnichenko doubled the advantage in the second half with a close-range header.
Kris Commons came closest for Celtic when he hit the crossbar from 25 yards in the first half.
Celtic manager Neil Lennon made three changes to the side that defeated Aberdeen on Saturday, with Virgil van Dijk making his first start since his close-season move from Eredivisie side Groningen. Mikael Lustig and James Forrest also came into the starting XI.
Viktor Kumykov made 10 changes to the Karagandy team that beat Astana 2-1 on Saturday, and was clearly keen to focus on the team's Champions League aspirations.
It was Celtic who made the brighter start, pressurising goalkeeper Alexandr Mokin at corners, although he did make an excellent point-blank save from a Georgios Samaras header.
Fraser Forster had been a bystander in the Celtic goal until Karagandy opened the scoring with 12 minutes gone.
The visitors failed to clear their lines from a Nikola Vasiljevic long throw and Finonchenko ghosted in between Joe Ledley and Emile Izaguirre to poke home from close range.
Commons came close to producing a swift response when he unleashed a fierce 25-yard effort that struck the crossbar and bounced on the line before spinning safely into the hands of Mokin.
Karagandy posed a considerable threat in the air and missed a good opportunity to double their advantage when an Aleksandar Simcevic centre was headed off target by Sergei Maliy.
And they were nearly punished on the half-hour mark when Samaras glanced a Charlie Mulgrew free-kick from the left-hand side just wide of the far post.
Celtic knew they needed to try and work harder to get a foothold in the game, and should have equalised in the 52nd minute.
Izaguirre did well to cut in from the left-hand side, but blazed well over the crossbar on his weaker right foot.
And Celtic should have been level with just under 20 minutes remaining when Forrest timed his run into the box to perfection to meet Izaguirre's curling cross, but he could only head straight at the goalkeeper.
That missed proved costly as Karagandy grabbed a second when more poor defending from Gediminas Vicius' effort led to Khizhnichenko having the simplest of headers from six yards to send the home fans into raptures.
Celtic's hopes now hang by a thread and they will look to overturn the deficit at Celtic Park on August 28.
