The fire has taken hold of a building on Marsh Lane.
West Midlands Fire Service was first alerted to the blaze at 6.24pm with multiple 999 calls received.
Crews have deployed six fire engines and one hydraulic platform, say West Midlands Fire Service.
It is not clear how the fire started or if there are any injuries. The building is attached to a nearby block of flats.
West Midlands Police has evacuated residents nearby whilst some road closures are in place.
Residents have been asked to keep their windows closed.
Photographs on social media show smoke billowing into the sky from a large brick building.
Simon Woodward, the on-duty press officer for the incident, said: "We are dealing with a fire in a derelict four-storey factory on Marsh Lane.
"We have six fire engines and one hydraulic platform in use.
"We have asked nearby residents to keep windows closed whilst we deal with this incident.
"The building is attached to a nearby block of flats. We are ensuring all residents are safe."