The visitors moved ahead in just the fifth minute on the opening day of the Premier League season at The Hawthorns thanks to Lee Cattermole's 25-yard thunderbolt.
However, Sunderland felt strongly aggrieved three minutes before half-time when Swarbrick adjudged Valentin Roberge to have tugged down Victor Anichebe in the area and Saido Berahino confidently levelled with the resulting spot-kick.
Sunderland were staring down the barrel when Berahino put the hosts ahead in the 74th minute, but Sebastian Larsson earned Sunderland a point in the closing stages.
Despite seeing his side claim a point, Poyet was left baffled by the penalty decision.
"With the incident I was quite disappointed," he told Sky Sports.
"I was not expecting to come here and even try to speak about a referee's decision.
"[It was] soft. We are not playing football to give soft decisions. Everybody in the stadium thought he was going to give a yellow card to give Anichebe [for simulation].
"After that I don't know what happened, but it is not the way I wanted to start the Premier League.
"I don't want to talk about referees. I want to talk about football."
Though disappointed not to win, Poyet credited his team for their second-half showing.
"We went for it and had nothing to lose. We were much better in the second half and let us see if we can maintain it." he said.
