Spurs, bidding with entertainment giants AEG, have been short-listed along with a rival West Ham United consortium as preferred tenants for the 516 million pound venue in east London.
Tim Framp, who has gathered more than 2,100 signatories including local parliamentarian David Lammy to an online petition "Say No to 'Stratford Hotspur'", told Reuters on Wednesday his 'We are N17' group were determined to keep the Premier League club at the north London post code.
He said they aimed to hand out 20,000 flyers before Sunday's home match against Liverpool and organise a march down Tottenham High Road ahead of the game with champions Chelsea on December 12.
"We don't want to go anywhere else, it's all about staying in Tottenham," said Framp, whose group have also prepared posters and a website.
The 80,000 seater Olympic stadium was designed to be stripped down to a 25,000-capacity athletics venue after the Games. However, there are fears that it would become unsustainable without a high-profile tenant.
ATHLETICS LEGACY
West Ham, partnered by the local Newham Council, have pledged to leave a running track and are supported by UK Athletics but Tottenham have been less enthusiastic about retaining one and could instead pay for a facility elsewhere.
"West Ham's proposals are to leave the running track in place and if it stacks up economically I am sure the OPLC will award it to them," Keith Mills, deputy chairman of London 2012 and a Tottenham director, told the Evening Standard on Tuesday.
"If it doesn't stack up they have an alternative in Tottenham with an athletics legacy elsewhere.
"We promised in our bid to leave an athletics legacy. The IOC president (Jacques Rogge) has made it clear that it is more important not to leave a white elephant in London than whether that legacy is in the stadium or elsewhere in London," he added.
Tottenham, founded in 1882, currently have a capacity of 36,000.
They are planning to develop White Hart Lane into a new 56,000 seater stadium but have said the Olympic bid keeps their options open.
Framp, a 26-year-old who has supported Tottenham since he was six, feared it could become the preferred option and said the focus had to be on staying at White Hart Lane.
"Spurs is really embedded in the community. Tottenham is quite a deprived area and by pulling out the long-term outlook for that area is relatively bleak. It's quite bleak anyway," he said.
"If the club stays, it encourages people into the area. It's about keeping the club where it belongs really. Football clubs represent the area where they were formed.
"By moving you disconnect the club from the history that it has built up until now."
