Pulis' men looked to be heading for a useful point at St James' Park, with a string of highly impressive saves from Julian Speroni keeping the game level.
Palace also hit the bar through Yannick Bolasie early in the second half, but were forced to leave with nothing as Papiss Cisse headed home the winner for Newcastle in the fourth minute of injury time.
The fourth official had initially signalled for a minimum of three minutes added time and Pulis, who felt his side should have had a free-kick in the lead up to Cisse's goal, was quick to criticise Probert's decision to play over the allotted time.
"I thought for an hour we did very, very well. We tired at the end, they had a couple of chances at the end, but I thought we'd really seen it out," the Palace boss said.
"The disappointing thing is, if you look at the video, we think we should have had a free-kick outside their box with 20 seconds to go.
"Then they (Newcastle) come down and they score, which is over time. First half he (Probert) played a minute and 40 seconds over time.
"And in the second half I think it's 3.40 (into injury time) when they score the goal.
"We need a break, we need one or two things going for us. We missed a great chance at Sunderland last minute and today we've got done in the 48th minute, which was 30 seconds over time."
