Holt, who is on loan at Villa Park from Championship outfit Wigan Athletic, received a tirade of criticism after opting to join ex-boss Paul Lambert during the January transfer window.
Many Norwich supporters have not forgiven Lambert for leaving the club in 2012 and bombarded Holt's Twitter account with abuse following his decision to move.
But the striker, who was a hero throughout Norwich's successive promotions from League One to the Premier League, insists any negative reception he receives on Sunday will not bother him.
"I think anyone can tell you, they can come to me as much as they want," he told the Birmingham Mail.
"If any fan wants to abuse me all game, I'm not bothered as long as at the end I can walk off smiling having won the game.
"There's always one, but that's football. Everyone's got an opinion.
"I'm sure back in the day when I was on the Carlisle terraces I slagged a few players off as well.
"I think a lot of people disapproved about the move on my Twitter account, but that's football.
"The thing I love about my Twitter is it's so diverse. I get a lot of good messages and do lot of good work on it by tweeting charities and this and that. I enjoy it anyway."
‘Scoring in a World Cup is like winning the title – can you imagine millions of people celebrating something you did? It’s insane and made me very proud’: Brazil legend explains how much 2002 goal meant to him
'He already had the ability but didn’t use it because he was afraid to shoot – I said, "You’re a player who has to decide games – you have to take risks, mate"': Liverpool star was forced to become confident by team-mates
