Tottenham Hotspur looked to be on course for their first win of their Europa League campaign after Michael Dawson netted a first half header.
The North Londoners squandered a number of chances to double their advantage before Toche equalised for the hosts with 13 minutes to go.
Spurs now sit third in Group J, two points behind Lazio, but Villas-Boas has quashed suggestions his team believed they had won the game before the final whistle.
"I don't think we were complacent," he said.
"We have to give credit to Panathinaikos, they pressed much more and started to gain belief from that and that led to the goal.
"I think the whole team felt a bit disappointed because the first half was so positive and we were in control of situation.
"We were a bit sloppy in the second half. I understand the disappointment at not putting the game to bed, but in the end we congratulated the players because if it had been other way round, the whole perspective would have been different.
"Panathinaikos came at us very strong in the second half, they made a change with Ibrahim Sissoko. He improved the quality of their game and could not stay in control of game so much in second half."
Villas-Boas is refusing to panic following the stalemate in Greece, should they fail to overcome Slovenian side Maribor in their next Europa League encounter
"In the end it leaves us with two points. Maribor have three and Lazio have four so everything is open as we go into the double-headers," he continued.
"It's only after the fourth game that you tend to know who will go through or whether there is any competition for that second place.
"In a group like this, normally 10 or 11 points to qualify."
The Spurs head coach lifted the trophy with FC Porto in 2011 and the 34-year-old has re-iterated his desire to win the competition again this season.
"The ambition we set for this competition is to go to the final, to win the trophy," said Villas-Boas.
"It is a trophy this club have won in the past so we embrace this tradition. We will try to qualify from this group and then take it from there."
STATS ZONE Our award-winning app extends into Europa League
