"This harassment has included police raids, arbitrary arrests, ill-treatment and extortion, as well as destruction of informal housing," the London-based human rights group said in a statement.
In particular, vagrants and informal street vendors were being booted out of "controlled access sites" and exclusion zones demanded around World Cup venues by the sport's world governing body FIFA, Amnesty said.
"The requirements under the 'FIFA by-laws' which create extensive exclusion zones for informal economic activity are seen as particularly prejudicial," Amnesty said.
Police spokesman Vish Naidoo defended the action by police and local authorities, saying Amnesty was trying to link the World Cup to a perennial problem in South Africa, where millions live in grinding poverty and shanty towns.
"Just because the World Cup is happening, must we ignore laws and by-laws that are being broken?" he said.
Amnesty also said the deployment of huge numbers of police at World Cup-related sites would mean fewer officers in poorer neighbourhoods, where the largest part of South Africa's notoriously high levels of crime occur.
It added that it was worried about police getting trigger-happy in their zeal to protect the 350,000 foreign visitors who are expected for the month-long tournament, which kicks off on June 11.
Security and crime have been one of the biggest issues ahead of the first World Cup on African soil. South Africa has 50 murders a day, almost the same rate as the United States, which has six times as many people.
