Spurs made the short journey to Upton Park on Saturday and, despite failing to impress for long spells, claimed a last-gasp winner thanks to Eric Dier's 93rd-minute effort.
West Ham were boosted in the first half when Kyle Naughton was given a straight red card for handball but Mark Noble fired the resulting penalty wide.
Allardyce's men were unable to capitalise on their numerical advantage and then saw it wiped when James Collins was given his marching orders for a second yellow card with 27 minutes remaining.
And heartbreak was to follow for West Ham as Dier's late effort saw Tottenham claim all three points.
After the match, the goalscorer labelled West Ham's style as "direct" in a television interview but Allardyce did not agree and felt that they outplayed their more-fancied opponents, only for their finishing to let them down.
"I listened to the Tottenham lad who scored the goal [Dier] saying we played direct football," he told reporters.
"He must have been at a different match to the one I saw. We outplayed and out-created Tottenham, who are supposed to be finishing in the top four, but we didn't finish them off.
"We haven't been clinical enough. I can't complain about the performance, or how many chances we made.
"I thought it was a fantastic performance by the players, [but] the game was there for us to take, we did not take it and we have ended up with nothing.
"It is difficult to swallow as we dominated and played some excellent football but there were three really big factors that decided it.
"First, we did not take our chances, then there was Mark Noble's missed penalty and finally we allowed ourselves to go to 10 v 10.
"It was the first Noble had missed [from the penalty spot] in three years. It would happen at some time but it is just unfortunate it happened today.
"We created enough chances to win this game, but this is a big league and it punishes you and it punished us."
Allardyce was impressed with the competitive debut of Senegal midfielder Cheikhou Kouyate and added that further signings should be expected before the transfer window closes.
"I thought Kouyate was the man of the match for me," he added.
"I think he is a terrific young man who is only going to get better.
"Maybe if there is a nice player who becomes available who we think will contribute to the first team in a big way [then signings can be made]. Yes, that might be possible."
‘After Manchester City’s recent form, maybe they’re the underdogs against Manchester United!’ Former Red Devils defender on this weekend’s derby
‘Arteta, Alonso, Emery, me… none of us were physical players – we needed the understanding of the game. That probably helped us move into management’: Premier League boss reveals reasons for natural career progression
