Wheaton Aston Old Hall on the A5 at Ivetsey Bank used to host Utopia, a sex club for couples who would swap partners and use the venue's swimming pool, jacuzzi, sauna, and steam room facilities.
Under plans submitted to South Staffordshire District Council, the 18th century hall at the near-one acre site will be demolished and replaced with 10 homes.
In a statement supporting the scheme, MTC Planning and Design said demolition was "the only viable option", noting that if the site was converted potential buyers would be "put off" due to its notoriety.
The statement says: "The site is, unfortunately, well known both locally and nationally due to its former use, having been previously a country club then more notoriously known as an adults leisure club, or more commonly known as an adult swingers club."
The club shut down in 2007 and the site has remained vacant ever since.
Plans to put it forward for listed status were rejected by councillors in 2016, then in 2018, according to the statement, a suspected arson attack left the building with "significant damage" to the first and second floor areas.
The statement adds that due to the building's past use as a sex club, it has "attracted numerous inquisitive visitors and the site has been subjected to numerous break-ins".
"A number of the intruders have taken videos and images of the interior, which had been left fully intact, and posted these online," it added.
"As such, the site has claimed national notoriety for its previous use and, even today, website searches for A5 and Wheaton Aston will provide numerous links to national newspaper articles featuring the property."
The proposed scheme features six three-bedroom homes and four two-bedroom homes, one of which is a bungalow, and a new courtyard. The existing boundary wall along the A5 will be demolished and replaced.