The blaze has engulfed the empty building at the junction of Princip Street and Lancaster Street with smoke billowing into sky and nearby roads.
West Midlands Fire Service has advised people nearby to keep windows shut and warned those heading to the Commonwealth Games through the city centre that their journeys may be disrupted.
In a statement the fire service said: "We have a number of fire engines at a large factory fire at Princip Street, Birmingham city centre.
"There is a large amount of smoke in the area-please close your windows to keep yourselves safe. If you're heading into Birmingham for #B2022 (Commonwealth Games) please allow extra travel time."
Fire crews were called at 6.47am, with nine fire engines and two aerial platforms - crewed by 50 firefighters - called to the scene.
Two crews from Aston were joined by firefighters from Hansdworth, Bournebrook, Wood End, Smethwick, Erdington, Perry Barr and Hay Mills.
Speaking shortly after 9.15am, a fire service spokeswoman said: "We are expected to be there for a while but good progress is being made at the moment.
"The fire is contained to just that building."
Shortly before 4pm a fire service statement said: "Our crews have made good progress and continue to scale back our operations.
"Due to the structural damage caused by the Blaze the northbound slipway connecting A4400 and B4114 will be closed for a considerable amount of time. All other roads have re-opened."