Right-back Jose Bosingwa and striker Didier Drogba were dismissed in the first half after the visitors had fallen behind to a 10th-minute penalty by Heidar Helguson awarded after the Rangers striker was bundled over by David Luiz.
The defeat was Chelsea's second in the Premier League this season and left them in third spot, six points behind leaders Manchester City who thrashed Manchester United 6-1 at Old Trafford earlier in the day.
"I just cannot understand the difference of judgement from the referee, not only the major decisions but the fouls, the throw-ins," Villas-Boas told a news conference.
"I was aggressive to him [after the game] but I think I have to be. I don't care if he is OK or not. Everybody can have a bad day but this was not a bad day for us, it was a good day for us and a bad day for the referee.
"It is not a conspiracy theory... we are not cry babies, we are tough people like we showed today.
"On a normal situation with a normal referee with good judgement towards both sides we win the game," added the Portuguese as he praised his nine-man team for the way they dominated possession in the second half.
"Penalties can be given or not given but if he gives a soft penalty like that he has to give soft penalties in the box for our chances on the opposite side as well," said Villas-Boas.
"There was a difference of criteria there which doesn't [make] us happy. The judgement used was not the same.
"I have nothing to say about Didier's sending-off because it seems fair to me but it doesn't seem fair to me that Bosingwa was sent off because JT [John Terry] was in a position of cover and I think a yellow card should have been applied."
