But charity bosses have said the fight continues to rejuvenate Old Stafford Library.
The Grade II-listed library was designed by Briggs, Wolstenholme and Thorneley in 1913 and partially funded by philanthropist Andrew Carnegie.
It was described last year by the Society as a 'small but impressive classical building in the centre of Stafford' which had not be used for nearly 20 years since the library closed in 1998.
Hopes for its future emerged earlier this year, with a business plan drawn up the Stafford Old Library Trust.
This set out proposals for how different areas of the Carnegie Library at The Green, next to the cinema, could be used as a theatre, a performance cafe, gallery space, and workshop space.
The Old Library Trust's campaign is 'well into its second year' but continuing, according to a post on its website.
A statement on the Victorian Society website says: "The Stafford Old Library Trust are fighting for acquisition of the Grade II-listed building.
"Last week, a reasonable offer of £400,000 was sadly rejected by the owner.
"The sale notice has since been withdrawn, so the hope is the Carnegie Library will re-emerge for sale at a much more reasonable price so the trust can raise the money for purchase to keep it as a community space."
Partly funded by philanthropist Andrew Carnegie and opened in 1903, the building, off Lichfield Road, served as Stafford’s library until it closed in 1998.
It was sold to a private investment fund and an investor by Staffordshire County Council in 2012.
The trust, which gained charity status in 2015, has been raising awareness about the plight of the Old Library for two years now.
The charity is now now taking pledges from people in an attempt to raise essential funds to start the project rolling, and also demonstrate the commitment of the town to the reclamation and restoration of this important heritage building.
SOLT believes that the building should be bought back into the public sector, restored, and developed as a cultural and arts centre for Stafford.
The building once housed Clement Lindley Wragge's collection of ethnographic, zoological and geological material but it is unclear where the collection is now.
A planning application for conversion to an Indian restaurant was approved in 2013 but nothing has happened since.