Right-back Jose Bosingwa and striker Didier Drogba were dismissed in the first half after the visitors had fallen behind to a 10th-minute penalty by Heidar Helguson awarded after the Rangers striker was bundled over by David Luiz.
The defeat was Chelsea's second in the Premier League this season and left them in third spot, six points behind leaders Manchester City who thrashed Manchester United 6-1 at Old Trafford earlier in the day.
"I just cannot understand the difference of judgement from the referee, not only the major decisions but the fouls, the throw-ins," Villas-Boas told a news conference.
"I was aggressive to him [after the game] but I think I have to be. I don't care if he is OK or not. Everybody can have a bad day but this was not a bad day for us, it was a good day for us and a bad day for the referee.
"It is not a conspiracy theory... we are not cry babies, we are tough people like we showed today.
"On a normal situation with a normal referee with good judgement towards both sides we win the game," added the Portuguese as he praised his nine-man team for the way they dominated possession in the second half.
However, poor concentration and a lack of discipline led to Luiz barging into Helguson to give away the penalty.
SOFT PENALTIES
"Penalties can be given or not given but if he gives a soft penalty like that he has to give soft penalties in the box for our chances on the opposite side as well," said Villas-Boas.
"There was a difference of criteria there which doesn't [make] us happy. The judgement used was not the same.
"I have nothing to say about Didier's sending-off because it seems fair to me but it doesn't seem fair to me that Bosingwa was sent off because JT [John Terry] was in a position of cover and I think a yellow card should have been applied."
Luiz was one of seven Chelsea players and two from Rangers who were booked as the home team battled to hang on to all three points after the red cards for Bosingwa, after 33 minutes, and Drogba, eight minutes later.
Portuguese Bosingwa went for pulling down Shaun Wright-Phillips as the pair tussled in pursuit of the ball outside the penalty box while Drogba was dismissed for a two-footed tackle on Adel Taarabt.
Chelsea created several chances - Nicolas Anelka wasting the best of them when he headed straight at keeper Paddy Kenny from six metres in the second period - before the home crowd erupted to celebrate their first home win of the season.
Promoted Rangers climbed to 10th in the table with 12 points.
