Garry Monk's men are seven points clear of the bottom three following a 3-0 win over Norwich City last weekend and can move closer to safety with victory at Hull City on Saturday.
But captain Williams feels that the Welsh club should be aiming higher and has challenged his team-mates to claim maximum points in all of their final six games.
"What we don't want to focus on is getting to 37 or 38 points and then stopping," Williams told The South Wales Evening Post.
"We're trying not to lose again this season, to win as many as we can and to just keep going. It's in our hands.
"We're not saying: 'Let's go for two more wins'.
"Forget that. We've got six games left so why can't we try to win them all? If we can't win them, let's not lose.
"That's the way we're looking at it.
"It's a bit bullish to say that after we've just drawn one game and won one game, but that's the way we feel.
"We've got to set our standards high and get back to where we have been in the last two years.
"We want to finish the season strong and take the momentum into next season."
‘After Manchester City’s recent form, maybe they’re the underdogs against Manchester United!’ Former Red Devils defender on this weekend’s derby
‘Arteta, Alonso, Emery, me… none of us were physical players – we needed the understanding of the game. That probably helped us move into management’: Premier League boss reveals reasons for natural career progression
