The St James' Park hosts appeared to be on course for victory when Papiss Cisse scored just after the hour mark, but goals from substitutes Craig Noone and Fraizer Campbell turned things around in Ole Gunnar Solskjaer's first game in charge of Cardiff.
Pardew bemoaned Newcastle's inability to hold on to a lead after seeing his team knocked out in the third round for the third time in four seasons.
"Our aim is to win a trophy and we won't do that by going out in the third round of the FA Cup," he said.
"For the first 20 minutes of the second half we were in control and scored our goal. Then in the last period of the match we let things slip and paid the penalty."
Pardew pinpointed defensive errors as the primary reason for Newcastle's defeat.
"We had to defend better from set-plays," he added. "We selected a whole new back five today and apart from set plays they did OK.
"Before Cardiff equalised the referee (Anthony Taylor) had the whistle to his lips and looked set to give us a free-kick. But he didn't and they smashed the ball into our goal.
"The second was nothing to do with the referee. We should have defended it better and looked vulnerable whenever Cardiff had a free-kick or corner.
"Everybody in our dressing room is disappointed. We wanted a cup run and now we are out.
"We put out a team more than strong enough to win that game. We had a good shape and were in control. Then it got away from us.
"Our performance deserved better, but we let ourselves down in those closing stages."
