JMI Planning wanted to create new leisure and commercial facilities within a part five storey and part three storey building on Lower Forster Street.
But Walsall Council planning officers are recommending the application is thrown out amid fears over the size of the development and the impact it would have on neighbours.
Several objections to the plans were made by the owners of neighbouring properties, who also raised concerns about loss of light and privacy, increased noise and a loss of views of the nearby Arboretum.
The land is now vacant but once housed the renowned saddle-making business Jabez Cliff, which was established in the town in 1793.
It was severely damaged in a huge fire as a result of a suspected arson attack in 2011 with the shell of the building that remained demolished shortly after.
The company had relocated to a new base in Aldridge Road two years before the fire but went into administration and ceased trading in 2014.
In their application, the developers said the land, which is on a main route into the town centre, has been on the market for six years but has remained undeveloped.
If the development was given the go-ahead, they added it would create an estimated 85 full times jobs along with 12 apprenticeship roles.
They said: “The site consists of of brownfield land, which currently lies derelict and unused.
“The unoccupied nature of the site makes it vulnerable to misuse by way of arson and fly tipping. The proposal outlined in this document aims to bring this iconic site location back into use.
“The site was formerly used by historic Walsall saddle maker Jabez Cliff which went into administration. The company was one of the UK’s most respected saddle makers, widely known for its Barnsby brand.
“In 2011, the former factory suffered an arson attack and was burned down. The site has remained derelict since.
“Today the site is considered to occupy a ‘landmark’ location within Walsall.”
They added: “The proposed development makes use of underutilised brownfield land and redevelops the derelict landmark site into a mixed-use leisure and commercial facility.
“This development would act as a gateway building to a commercial corridor, leading to the town centre of Walsall and improving the general aesthetic and experience of the surrounding environment.
“The leisure component of the proposed development would operate into the early-evening hours, ensuring “eyes on the street” and a sense of surveillance, assisting in crime prevention in the area.
“Significant regeneration benefits and will contribute to meeting the council’s vision to generate economic and employment growth objections, as well as enhancing the built environment.
“In its current, derelict state, the site does not contribute to the townscape and requires redevelopment in order to ensure no further misuse and degradation.”
But in a report going to Walsall Council’s planning committee on November 4, officers are recommending the proposal be turned down.
They said: “The proposed development by reason of its scale, mass, bulk, height, design, and materials represents poor design that visually detracts from the character, architectural and historical significance of the street scene and the character and setting of Lichfield Conservation Area that neither enhances nor preserves (it) and the setting and significance of the Grade II listed building Queen Mary’s School.
“The proposal would also have a detrimental impact to the adjoining premises on Lichfield Street due to additional shading and loss of light to rear amenity areas and rear windows within first floors of residential premises within flats.”