Winger Andros Townsend put Spurs ahead by half-time at White Hart Lane as he kept his cool to score from the penalty spot after Roberto Soldado was fouled in the penalty area by Liam Moore.
As the clock wound down it appeared as though that would be decisive, but Leonardo Ulloa equalised in the 83rd minute when Spurs failed to properly clear their lines from a corner.
There was more drama to come in stoppage time as Jeffrey Schlupp side-footed home the winner to spark celebrations in the away end.
A disappointed Pochettino told BBC Sport: "We made two mistakes at the end of the game and we paid by now being out of the FA Cup.
"The game was always under our control. We scored in the first half and created chances to add to it.
"It is football. We can make a mistake but I am very disappointed because I think we expect to be through to the next round. So I'm not happy.
"Now we need to focus on the next game against Sheffield United.
"In football you need to look forward. It is not easy playing so many games but on the day it was about us making a mistake and that cost us the game in the last seven minutes."
