The fire, which has now been extinguished, broke out in the early hours of Monday morning and tore through The Grasshopper Building in Whitmore Reans.
Thirteen fire engines and between 70 and 80 firefighters were sent to the building off Craddock Street and Gatis Street after the emergency call was made just before 1am.
Police and paramedics were also sent but nobody is thought to have been hurt.
The steel-framed warehouse, which also includes offices and belonged to the Grasshopper baby clothing company before becoming an Xercise4Less gym in 2014, was well alight when firefighters arrived.
Residents living nearby have described feeling the heat from the blaze in their homes before being evacuated.
A West Midlands Fire Service spokesman said: "It was challenging job for the firefighters, it is a large building and areas of it were dangerous as they were collapsing or at risk of collapsing.
"In spite of that we got on top of the fire and started to scale it down at 3.30am this morning.
"At around 7am this morning resources were reduced to one engine and one hydraulic platform.
"Remaining crews are keeping a watching brief with some small fire remaining.
"Investigators will be returning to the scene to find out how the fire started."
Videos from the scene show flames rising high into the sky, with smoke billowing out over nearby houses, while photos of the aftermath confirmed much of the building has been destroyed.
Two cars parked nearby also appeared to have been damaged.
Residents in the local area were warned to keep doors and windows closed to protect themselves from the smoke.