The home side were awarded the spot-kick in the 94th minute when Steven Reid was deemed to have fouled Ramires in the area.
Eden Hazard converted to salvage a 2-2 draw for Jose Mourinho's side, with Clarke claiming referee Andre Marriner was swayed by the home support.
"It's the kind of penalty that can be given when you’ve got a home crowd that shout for everything and put pressure on the referee," he said.
"Referees are human; they make mistakes like everyone else but for us today, the most galling fact is it should have been three points from Stamford Bridge which would have been a fantastic result for us.
"Instead of that it’s just a good result, it’s only a point but the team were tremendous today and I’m really proud of the way they responded in the second half."
Clarke was angered by the nature of the penalty as well as its timing, as his team were denied back to back Premier League wins
"The referee has to be 100 per cent sure it’s a penalty," he continued.
"How he can be 100 per cent sure in that instance is beyond me. It’s not correct – it was a bad decision. I think he went down early before the contact.
"The onus is on the referee to make the right decision –and he didn’t make it."
