Currently libraries in Aldridge, Bloxwich, Brownhills, Darlaston and Willenhall open for three hours every Sunday – from 1pm to 4pm.
But Walsall Council want to stop opening them on Sundays in a bid to save on costs.
The issue will be decided by the council’s cabinet at the end of October.
Sunday opening at Streetly community library will also be stopped.
Councillor Garry Perry said it was a decision the council “would not like to be making” but that Sunday opening was just not viable any more.
He said: “The main reason behind it all is to plug a gap in funding.
“We trialled Sunday openings but sadly it has not worked. The number of visits and the cost of staff, it just was not worth it. So we have taken the decision not to do it. But it will mean we can try and put those hours back in the working week.”
Councillor Perry also said that the reason for the closure was down to problems with Open Plus, a self-secure facility that increases the hours people can use libraries.
Walsall Council had originally joined up with the group to allow locals to enjoy the libraries, while saving money on staff.
But due to significant delays and problems with the system, Open Plus pulled out which left the council with a “funding gap”, Councillor Perry explained.
He added: "Libraries across Walsall have bucked the national trend in that there has been an increase in usage. But Sundays were not proving to be very successful."
Libraries across the region are facing a crisis, with low usage and depleted staffing levels pushing services close to the brink.
Across the country footfall has tumbled by 10 million since 2016-17, while local authority spending has continued to drop since 2010 as councils battle to balance the books in a time of severely restricted budgets.
Last year 130 libraries closed their doors for good.
The matter of Sunday opening will be put to Walsall council cabinet on October 23.