Barry Hearn, whose club is close to the £500 million stadium in a previously rundown part of east London, has long been against a bigger side like second tier West Ham United moving into the venue and his patch after 2012.
A football tenant for the stadium, which will retain its athletics track after the Games and will host the 2017 World Championships, seems the most likely.
"The more I look at the stadium, the more and more convinced I am is that it is not fit for football purposes," the businessman told Britain's Sky Sports News.
"I have to say after my tour of the stadium, wonderful stadium though it is, I don't think it is good for football."
"No one has actually taken responsibility or admitted they have built the "wrong" stadium. They didn't listen to advice five years ago when everyone was telling them what kind of stadium can work," he added.
"We have to face these problems and be honest. Let's use some common sense otherwise it is going to be a horrendous white elephant. We have seen that happen around the world, let's hope it does not happen to London."
