Residents say tenants of J9 Accommodation properties are making their lives a misery, with some saying they fear leaving their homes due to the amount of anti-social behaviour in the area.
J9 owns and runs around 140 properties for 900 tenants in the Black Country and is run by magnate Jim Haliburton, who dubs himself the 'HMO Daddy'.
The title refers to his tactic of buying derelict buildings and converting them into five or six-bedroom homes known as Houses in Multiple Occupation (HMO).
But residents say rooms in the properties are being let out to drug addicts, alcoholics and people who have recently been released from prison.
Mr Haliburton has refuted the claims and insists he has done more than most to support the regeneration of Darlaston.
On Saturday, protesters stationed themselves outside the firm's head office in Pinfold Street demanding to speak with him.
The protest was organised via Facebook by Ryan Pressdee, aged 30, who lives in York Crescent. He said: "The situation has been getting gradually worse over the past five years and people have just about had enough.
"J9 owns a large number of properties and the vast majority of the rooms are being given to the worst types of people. We see people taking drugs in front of children and drunks arguing in the street.
"A lot of people are scared to come out of their houses. Police and ambulances are here all the time.
"It's become a cesspit of druggies and drunks. Mr Haliburton is getting filthy rich by housing scum. We want to sit down with him and talk to him about the tenants he is taking on and how they are destroying the area."
Mr Haliburton, a multi-millionaire who grew up in a children's home in Edinburgh, said: "As a landlord I get fed up of people accusing me of taking in druggies.
"It's not something I do. We want good tenants, not bad ones, but sometimes it is impossible to tell what people will be like. If someone comes in we can't always tell if they've been in prison or they are taking drugs.
"We work with the police and relevant authorities to do the checks necessary to find these things out. There are times when we have had to evict people, but the process can take over two months.
"For that you can blame the courts."
Mr Haliburton said he is willing to meet with protesters to discuss their grievances. "We're all on the same side," he added. "I will co-operate with anyone who wants to improve Darlaston."