The Welsh club fought back from 2-0 down to lead 3-2 at the Liberty Stadium but were denied three points courtesy of a 96th-minute Charlie Adam spot-kick.
Referee Robert Madley awarded the penalty after the ball appeared to strike Wayne Routledge on the arm in the area, although Williams was perplexed by the decision.
"It seemed like none of their players appealed for a penalty - maybe one of them was appealing for a corner," the defender told the South Wales Evening Post.
"It was a shock to everyone. I didn't see a handball from Wayne or Leon (Britton). I don't know.
"All I can say is that I didn't see a handball and I think both sides were surprised."
Adam's penalty ensured Swansea were denied a first victory in five matches and the chance to get back on track following defeat to rivals Cardiff City in their previous Premier League outing.
"It's a sickener because to get three points going into the mini-international break would have been good," Williams added.
"It (a win) is what we deserved and the table would have looked a bit better, but we can be pleased with what we did in the second half."
Get the best features, fun and footballing frolics straight to your inbox every week.
Thank you for signing up to Four Four Two. You will receive a verification email shortly.
There was a problem. Please refresh the page and try again.