The cost-cutting measure has been touted by Walsall Council which is looking to save £86 million in the next four years.
But more than 30 people, including former leather tradespeople, took to the Littleton Street West venue to show their anger at the idea.
The have argued the museum, which opened in 1988 having a leather factory itself once upon a time, will lose its attraction if it is moved.
Mother-of-three Sam Hale, aged 51, who co-ordinated the demonstration has been a visiting artist in the museum before.
She said: "The building is part of the experience. I have been there when people who used to work in the trade have been there and when they walk into the place they are just transported back.
"You walk in and you can smell the leather.
"When you talk to the demonstrators they tell you leather gets in your blood.
"The museum is an important part of our heritage. It is sad to say but if our leather factories weren't to survive this would be our main reminder of our town's trade.
"It is something for us to be proud of. This is the story of Walsall men and women. Leather was the lifeblood of the town. I'm astounded they want to get rid of it.
"If you move it it becomes a totally different experience and it will lose its uniqueness."
More than 2,400 people have signed a petition to 'save' the museum.
The move is part of the plans to reduce the amount of libraries in the borough placing more emphasis on the Lichfield Street Central Library which would incorporate the leather museum and the Local History Centre.
The move would save £2.9 million a year but see 130 jobs lost.
Council consultation documents state: "The proposal to consolidate three separate sites into a single town centre location, close to other amenities, transport links and parking, offers the opportunity to redesign the service, make improvements and efficiencies whilst also protecting service delivery overall in the longer term."