After taking the lead at the Britannia Stadium on Saturday through Andy Carroll in the fifth minute, West Ham were undone as Peter Odemwingie's brace and one from Marko Arnautovic sealed all three points for the hosts.
It is the fifth time this season that West Ham have fallen to defeat after taking the lead in the Premier League, more than any other side, but Allardyce felt his players deserved at least a point.
"I think that in the cold light of day we have to blame ourselves, there is a big part to blame the referee for," Allardyce said.
"Of that there is no doubt in my mind. I have reviewed the situation and watched a blatant handball (from Muniesa) that stops Carroll chesting a ball down and trying to volley it.
"The ball is diverted away by his left arm. It's arm to ball, not ball to arm. The referee is in a fantastic position to see it. I'm totally and utterly baffled."
"It's our fault for the poor defending has cost us," he added. "On two occasions we just have to defend correctly.
"We were in the ascendancy, in control looking for a goal to get in front, then we fall asleep and I don't know where our defending went."
