There are currently just three black managers employed in England's Football League, with the issue reportedly discussed at a recent Professional Footballers Association summit.
Palmer - who had stints in charge of Stockport County and Mansfield Town - admits there are barriers for black managers in English football, and believes he could have been overlooked for roles due to his skin colour, but feels chances will arise if people prove they are capable of doing the job.
"If black managers and black coaches want to get jobs they have to make themselves the best and if they do that they'll get employed regardless," Palmer told Perform.
"Don't get me wrong the colour issue is a problem but I've never made it an issue in my life. I try not to make it an issue about colour.
"It's not going to change by people talking about it, it will change by more black footballers wanting to do it and wanting to be the best.
"Is it more difficult for black managers to get jobs? It's difficult full stop if you're black I think, but it's not something I make an excuse, it's something that's always been there, you've got to be good enough.
"I'm sure that's been the case (that I have been overlooked). But I live my life as a human being, not that I'm black or I'm white, I go about my business in the way that if I'm good at what I'm doing then I deserve to do what I want to do."
The lack of black managers in English football is a recurring issue, with FIFA's anti-discrimination chief Jeffrey Webb calling for more to be done to get black players into coaching roles.
But Palmer says it is difficult to gauge the extent of the issue without knowing how many black managers are applying for vacant positions.
He added: "Until it comes out how many black managers are applying for jobs and how many black managers are actually getting jobs then how can you make an informed argument?
"I know black players who have applied for jobs but I don't particularly hear that a lot of black players are applying for manager's jobs and getting turned down.
"When I started playing there wasn't particularly a lot of black players around, but now most clubs have black players and a lot of the top players are black.
"Chris Hughton has done a great job at Norwich, Incy (Paul Ince) has done well apart from that short stint at Blackburn Rovers."
