Developers McCarthy and Stone Retirement Lifestyles Ltd and Kimberley Developments Ltd had hoped to build 48 apartments and convert Aldridge Manor House into seven flats and three homes on the land off Little Aston Road in Aldridge.
But Walsall Council planners felt the design and size of the scheme would have a negative impact on the listed building.
Officers also raised concerns about a lack of affordable housing provision as well as drainage and tree protection issues.
The news of a refusal has been welcomed by ward councillors Tim Wilson and John Murray, who said residents were against the plans.
Grade II Listed Aldridge Manor House, which is more than 100 years old, and as a hospital for wounded soldiers during the First World War and has a blue plaque on the front of the building.
In more recent years, it was used as a library and then a youth centre by Walsall Council and is currently used to provide temporary accommodation.
The proposal had been looking to create 32 one-bedroom apartments, 29 two-bedroom apartments, one three-bedroom flat while the four homes would also have had two bedrooms each.
The redevelopment also included improvements to the Masonic Hall car park while 15 jobs would have been created as part of the scheme.
In their application, Inspire Design said: “The proposals are practical, honest, sustainable and sensitive to the surrounding area and improves the appearance and function of the site.
"The Manor House has had many occupants throughout its history and recently it has been owned by Walsall Council and used as Youth Centre.
“It is currently in use as temporary bedsit residential use. The proposal to convert to residential use has the advantage of returning the building back closer to its original use.
“The requirement for cellular rooms means that the layout of the building is already well suited and minimal work is required to form new openings in walls.”
Councillor Wilson said: “I am pleased planning officers have listened to the objections of local residents, Councillor John Murray and I and refused the planning application for Aldridge Manor House and land behind.
“In my opinion, the proposed development was too large for the site and the issues of noise, parking and additional traffic pressures at an already busy location were not mitigated against by the applicants.”