Samuel Eto'o's hat-trick ensured David Moyes' men were beaten for the seventh time in the top flight this season - a result that leaves them 14 points adrift of league leaders Arsenal.
While Moyes refused to give up on the title after the game, captain Vidic - who was sent off in the closing stages at Stamford Bridge - believes the gap is too great.
"We are too far behind and we have now to focus on the top three or four," he told Sky Sports.
"We have to be positive, (on Sunday) we didn't play a bad game. They had three shots and scored three goals and that's the difference in the game, we have been punished for every mistake we made.
"It's not going to be easy, we have to now fight to achieve the Champions League. We have to stay positive and stick together and work hard. With a bit of luck hopefully we can achieve that."
Vidic was dismissed for a reckless challenge on Eden Hazard to compound a difficult day for Moyes, whose side will look to overturn a 2-1 deficit in Wednesday's League Cup semi-final second leg against Sunderland.
In what has been an uncomfortable first campaign at the club for Moyes, Vidic said his team-mates are fed up of seeing the Scot blamed for United's underwhelming season.
"We cannot always speak about managers, the players are tired of speaking about that because the manager is trying to do the best he can," he added.
"Tactically we did well (on Sunday)."
