NHS chiefs confirmed the discovery after routine water testing at Penn Hospital, in Wolverhampton, and will make an assessment next week whether it can reopen.
The Penn Road site offers 24-hour care and support to older people with mental health needs. Staff are believed to have been alerted to the issue on Sunday.
Legionella is a bacteria which can give rise to a mild flu-like illness known as Pontiac fever, and a serious type of lung infection known as Legionnaires' disease.
A spokeswoman for the Black Country Healthcare NHS Trust, which runs Penn Hospital, said: "Urgent works are ongoing at Penn Hospital and we will be making an assessment next week as to re-opening.
Patients have been transferred to sites within and outside of the trust. We are keeping patients, service users and their families/carers informed and will carefully manage their return to Penn Hospital to minimise any disruption."
The NHS trust has over 3,000 staff based at 65 sites – including community bases and hospitals in the Black Country, including Bloxwich Hospital and Dorothy Pattison Hospital in Walsall, Bushey Fields Hospital in Dudley, Edward Street Hospital and Hallam Street Hospital in West Bromwich, alongside the location in Penn and others.
A spokesman for the organisation previously confirmed there were "no confirmed cases" but bosses "continued to monitor the situation closely".
It marks the second time there has been Legionella bacteria detected at the site, with health chiefs previously confirming it had been detected in 2014 in the water.
During that occasion access to some sinks, showers and toilets were blocked as a precautionary measure and temporary toilets were installed for patients to use.
Specialist filters were also fitted on some showers so patients could continue to shower and, where necessary, bottled drinking water was provided.