Families say the problem in Pleck is worse than ever despite a crackdown being launched last year.
Kerb crawlers are approaching prostitutes outside people's houses, while needles and used condoms are being discarded in the gutter.
Now the neighbourhood watch group is calling police and councillors to a meeting on September 4 to tackle the issue.
Operation Scarlet was launched last year, with sex workers and kerb crawlers being hauled before the courts.
And though police insist the crackdown is continuing, residents say the problem is getting worse. And some are even considering moving out of the area.
Darlaston Road and Gower Street are particular hotspots.
Retired police officer and Pleck Neighbourhood Watch chairman David Baker, aged 66, said: "I live in Gower Street and these women are operating under our noses, and enough is enough.
"The property value round here is dropping and we have had all we can take. I think the number of women and punters has trebled in the last couple of months.
"Women are frightened of being accosted as they walk down the street. It's unacceptable."
He has called a meeting on September 4 at James Bridge Copper Social Club in Reservoir Place at 7.30pm and expects local police officers to attend.
Police say powers under Operation Scarlet include issuing street cautions and in some cases if they repeatedly re-offend, applying to the courts for a criminal anti-social behaviour order (CRASBO) which bans them from entering a specific area of Walsall.
Sergeant Mo Shaid, from Pleck neighbourhood policing team, said: "People understandably do not want this type of behaviour on their doorstep and we will always work with the local community to tackle the issues that concern them the most.
"Historically we are aware of street workers operating in the area and along with the Walsall South Area Partnership a joint operation to look at the problem of street sex workers and kerb crawlers has been put in place.
"The joint operation aims to reduce on street sex working, reduce vulnerability to sex workers and to improve the quality of life for the community and local residents.
"My team of officers regularly patrol the area and I would encourage local residents to call and report any anti-social behaviour relating to sex workers and kerb crawlers."