Christian Benteke and Gabriel Agbonlahor both scored first-half braces as the hosts capitulated in the opening period at the Stadium of Light.
The defeat further piles the pressure on Poyet, with Sunderland now just one place above the relegation zone after Villa leapfrogged them with Saturday's victory.
Asked if he expects to speak to Sunderland owner Ellis Short on the back of such a heavy defeat, Poyet replied: "I don't know. I don't speak to the owner after every game, we speak when we need to speak.
"I'm just trying to accept something that I've got no explanation for. I prefer not to go into details, we're going to analyse the first half very carefully.
"I'm extremely disappointed. I did not expect this to happen but that's football I suppose.
"I'm honest, when things go right I tell you I'm responsible and when things go wrong I'm responsible.
"I hate people who change and have too many faces. I'm responsible and I'm the one who needs to take it."
Poyet also took time to explain the absence of Sebastian Larsson at the start of the second half, after the Sweden international emerged late to the action following the interval.
"Seb Larsson is one of the biggest professionals at the club, he had a massive cut in his leg and they tried to stitch it," Poyet said.
"So we waited for him, and he came out, sometimes we need to think twice before we write [false information] because Seb Larsson doesn't deserve that."
