"The crowd wanted to see an Azerbaijan win but Germany play in a different league to us and the fans don't understand that," the former Germany coach and player told ARD television.
"It's normal here, the coach is always to blame," he added.
Pressure has been mounting on Vogts since Friday's 2-1 defeat to Kazakhstan left his side with only three points in Euro 2012 qualifying Group A.
The three men who attacked him on Sunday were not journalists and were stopped by security guards.
"If you look at the clubs here, the players only train for one-and-a-half days here. We must work harder," said Vogts, who said he would see out his contract which runs to the end of the year before deciding what to do next.
Azerbaijan mustered only five points in 10 games in the qualifiers for the 2010 World Cup and are a lowly 108th in the FIFA world rankings.
"The whole sport here needs to be re-structured," said Vogts, adding there was too much pressure on his players.
"The players must be allowed to make mistakes, otherwise then cannot progress to another level."
He added: "There are idiots all over the world. You can't tar the whole country with the same brush because of three idiots.
"It's a beautiful country and I'm very happy here."
