Homeless people claim they have been forced off the streets and taken there in an attempt to "clean up" before kick-off next month.
The city denies this, saying it is part of general policy.
"There has not and will not be a concerted clean-up campaign," said city spokesperson Kylie Hatton.
As the continent's biggest economy, South Africa's streets are a magnet for immigrant beggars, who see the country as a step out of poverty.
In the last two months, police in Johannesburg swooped on mainly Zimbabwean blind beggars who roam the busy streets and women who sit with babies at road intersections - much to the anger of rights groups.
"Their presence violates the city bylaws and we arrest them. In many cases those in need, like women with children and disabled people, are referred to places of safety, where they can access welfare services," said Edna Mamonyane, spokeswoman for the Johannesburg Metro Police.
The country's three major cities - Johannesburg, Cape Town and Durban - have already started relocation programs, while six remaining host cities are set to follow suit.
Reports allege that more than 800 people have been taken from the derelict area surrounding Ellis Park in Johannesburg, the venue of seven matches including a quarter-final, and moved to shelters away from the inner city. There are also reported plans in place to see another 1,500 people, mostly street children, moved to alternate housing.
This is Tin Can Town, or Blikkiesdorp, described by the mayor of Cape Town as a "temporary relocation area" (TRA), but by its residents as a concentration camp. Many say they were forcibly evicted from their former homes and moved here against their will. And for this they blame one thing: the football World Cup.
"It's a dumping place," said Jane Roberts, who lives in the sparsely furnished structure known as M49. "They took people from the streets because they don't want them in the city for the World Cup. Now we are living in a concentration camp."