Bosses said they could not continue to 'trade viably' from the Wolverhampton Street branch. It will shut in September.
Shopkeepers and civic leaders have spoken of their disappointment over the announcement, which comes just days after Thomas Cook closed its Dudley branch.
Alan Caswell, who owns the Arcade Toyshop in the town's Fountain Arcade, said the closure was a sign that the town's centre was heading in the wrong direction.
"Dudley is fast becoming a ghost town. I can't believe WH Smith is going as well," he said. "Somebody has got to stop the demise of the High Street and the council needs to get their finger out and do something.
"They need to listen to what people want and not just do what they want. People don't want a new market place in Dudley. They don't want the Hippodrome knocked down.
"We desperately need a multi-storey car-park," he added.
Thomas Cook, which was based on the High Street, closed with bosses blaming its decision on making sure it had 'the right stores in the right places for our customers'.
The nearest Thomas Cook is in Merry Hill shopping centre.
It also comes just days after a last-minute change of heart stopped the closure of WH Smith's store in Wolverhampton's Mander Centre, with a new lease agreement saving the day. That store had been chosen to close because it had been 'unable to trade viably', according to the company's bosses. There has been no talk of such an agreement for the Dudley store, however.
Dudley Council's cabinet member for regeneration, Councillor Judy Foster, said: "We're obviously very sad and disappointed that this decision has been made. WH Smith has been there for a long time and is a reputable retailer.
"We are disappointed but we as a local authority have plans for the town, such as an improved market place and a number of supermarkets interested in Dudley."