Cameron said Britain should be proud of the achievements of the campaign to rid the sport of racism over the past two decades, adding that other countries had failed to take similar action.
"But we have some problems still today," Cameron said at the opening of an anti-discrimination summit with former players and football officials. "We need to act quickly to make sure those problems do not creep back in.
"If everyone plays their role, then we can easily crush and deal with this problem.
"We want to make sure football is all about a power to do good, rather than anything else."
Cameron said he often took his young son to local football matches and had seen how bad behaviour on the pitch could influence children and other spectators.
"What happens on the field influences what happens off the field. You see children as young as six imitating the behaviour they see on the field," he told a panel including former England internationals John Barnes and Graeme Le Saux.
"So this is not just important for football, it's important for the whole country."
FA Chairman David Bernstein said his organisation was working at all levels to deal with the issue and was determined not to allow any complacency because of the past success.
Earlier this month, Liverpool's Uruguayan striker Luis Suarez was forced to apologise for refusing to shake hands with Manchester United's French defender Patrice Evra before a Premier League match.
Suarez had only recently returned to action for Liverpool following an eight-match ban for racially abusing Evra during a game in October.
England international John Terry has been stripped of the captaincy pending his trial in July on charges of racially abusing Queens Park Rangers defender Anton Ferdinand. Terry denies the charges.
