The visitors moved ahead in just the fifth minute on the opening day of the Premier League season at The Hawthorns thanks to Lee Cattermole's 25-yard thunderbolt.
However, Sunderland felt strongly aggrieved three minutes before half-time when Swarbrick adjudged Valentin Roberge to have tugged down Victor Anichebe in the area and Saido Berahino confidently levelled with the resulting spot-kick.
Sunderland were staring down the barrel when Berahino put the hosts ahead in the 74th minute, but Sebastian Larsson earned Sunderland a point in the closing stages.
Despite seeing his side claim a point, Poyet was left baffled by the penalty decision.
"With the incident I was quite disappointed," he told Sky Sports.
"I was not expecting to come here and even try to speak about a referee's decision.
"[It was] soft. We are not playing football to give soft decisions. Everybody in the stadium thought he was going to give a yellow card to give Anichebe [for simulation].
"After that I don't know what happened, but it is not the way I wanted to start the Premier League.
"I don't want to talk about referees. I want to talk about football."
Though disappointed not to win, Poyet credited his team for their second-half showing.
"We went for it and had nothing to lose. We were much better in the second half and let us see if we can maintain it." he said.
Get the best features, fun and footballing frolics straight to your inbox every week.