The site on Compton Road, which has been abandoned for almost 10 years, had been one of a number of plots in the city highlighted by McCarthy and Stone as having the potential for new development.
But bosses at the retirement housing developer now say they will not be bidding for the land.
Owners the Royal Wolverhampton NHS Trust had been desperately trying to offload the dilapidated site amid spiralling security costs.
A spokesperson for McCarthy and Stone said: "There is a high demand for specialist homes for older people within Wolverhampton.
"However, I can confirm that McCarthy & Stone does not have an interest in the Wolverhampton Eye Infirmary site."
Councillor Craig Collingswood, who represents Park ward on Wolverhampton council, said: "This is extremely disappointing news as the site has been crying out for development for too long.
"It is on one of the main gateways to the city and sticks out like a sore thumb when you consider all of the other developments that have happened in recent years.
"Residents are desperate to see the site brought up to speed with the rest of the Chapel Ash area."
The eye infirmary opened in 1888 and closed in 2007 when services were moved to New Cross Hospital.
It was originally put on the market for £6 million, but numerous plans for development have fallen through.
Security has been a major issue on the site, with the Royal Wolverhampton NHS Trust spending around £40,000 a year.
It has been the scene of two serious arson attacks and is regularly targeted by vandals.
A spokesman for the trust confirmed there was currently no deal on the table to buy the plot.