Barry Hearn, whose club is close to the £500 million stadium in a previously rundown part of east London, has long been against a bigger side like second tier West Ham United moving into the venue and his patch after 2012.
A football tenant for the stadium, which will retain its athletics track after the Games and will host the 2017 World Championships, seems the most likely.
"The more I look at the stadium, the more and more convinced I am is that it is not fit for football purposes," the businessman told Britain's Sky Sports News.
"I have to say after my tour of the stadium, wonderful stadium though it is, I don't think it is good for football."
His club were involved in a legal challenge against West Ham taking over the stadium and although an initial deal collapsed, the Hammers are still widely expected to eventually call the arena their home once capacity is cut from 80,000 to 60,000.
A new process has started to find tenants for the stadium.
Hearn, whose small club are still interested despite his reservations, said he hoped the goalposts would not change again before the bid process concludes next month.
"No one has actually taken responsibility or admitted they have built the "wrong" stadium. They didn't listen to advice five years ago when everyone was telling them what kind of stadium can work," he added.
"We have to face these problems and be honest. Let's use some common sense otherwise it is going to be a horrendous white elephant. We have seen that happen around the world, let's hope it does not happen to London."
‘After Manchester City’s recent form, maybe they’re the underdogs against Manchester United!’ Former Red Devils defender on this weekend’s derby
‘Arteta, Alonso, Emery, me… none of us were physical players – we needed the understanding of the game. That probably helped us move into management’: Premier League boss reveals reasons for natural career progression
