Holt, who is on loan at Villa Park from Championship outfit Wigan Athletic, received a tirade of criticism after opting to join ex-boss Paul Lambert during the January transfer window.
Many Norwich supporters have not forgiven Lambert for leaving the club in 2012 and bombarded Holt's Twitter account with abuse following his decision to move.
But the striker, who was a hero throughout Norwich's successive promotions from League One to the Premier League, insists any negative reception he receives on Sunday will not bother him.
"I think anyone can tell you, they can come to me as much as they want," he told the Birmingham Mail.
"If any fan wants to abuse me all game, I'm not bothered as long as at the end I can walk off smiling having won the game.
"There's always one, but that's football. Everyone's got an opinion.
"I'm sure back in the day when I was on the Carlisle terraces I slagged a few players off as well.
"I think a lot of people disapproved about the move on my Twitter account, but that's football.
"The thing I love about my Twitter is it's so diverse. I get a lot of good messages and do lot of good work on it by tweeting charities and this and that. I enjoy it anyway."
Get the best features, fun and footballing frolics straight to your inbox every week.
Thank you for signing up to Four Four Two. You will receive a verification email shortly.
There was a problem. Please refresh the page and try again.