Chelsea made at least three attempts to lure Modric to Stamford Bridge in the summer window, but even a reported bid of £40 million on the final day of trading last week failed to persuade Spurs chairman Daniel Levy to sell their creative playmaker.
Asked if Chelsea would renew their chase in the New Year, Villas Boas told reporters at the club's training ground at Cobham on Thursday: "No, I don't think so. The market is closed now and I won't speculate on what happens in January."
Instead of signing Modric, Chelsea switched their attention to Liverpool's Portuguese midfielder Raul Meireles, signing him on the last day of the window for £12 million.
Villas-Boas said he was delighted with the deal.
"We wanted to fill in a space in midfield and bring one more player in and Raul has that ability, he can play in the channels or behind the strikers, and with what we wanted to achieve in terms of the playing style, Raul fits the requirements," Villas Boas said.
"Let me remind you he was voted Player of the Season by the fans of Liverpool last season so this was an extremely good deal for us and we are very happy to have him here."
Chelsea, who are away to Sunderland on Saturday, have made an unbeaten start to the new season with wins over West Bromwich Albion and Norwich City after an opening day draw at Stoke City.
Villas-Boas is optimistic goalkeeper Petr Cech will return for Saturday's match after missing the last two games with a knee injury and said that defender David Luiz, who has missed the start of the season, had also recovered from his thigh problem.
However, striker Didier Drogba, who suffered a bad head injury and concussion against Norwich on August 27, faces a race against time to be fit for Chelsea's Premier League visit to Manchester United on September 18.
The Ivorian will miss the Sunderland match and next Tuesday's opening Champions League game against Bayer Leverkusen.
The manager said: "He has to be authorised medically to make his full return to training. He has been doing light sessions, of course, with our physios. Let's push it to the limit to try for him to be available for Man United.
"I wouldn't say he would be available for Leverkusen.
"The first days were, of course, tremendous pain, and he had his family supporting him.
"He had complete support from the medical staff, who were with him 24-seven. And we've been getting this support to him.
"We just have to make sure that he's fully recovered before he comes back to training because that's the requirement you have to do neurologically."
