The 20-metre-tall pole has been installed on a strip of grass between Lawnwood Road and Saltwells Road, in Dudley Wood.
Richard Lowe, 50, who lives on Lawnwood Road, claims not all residents received notification letters about the plans.
Mr Lowe, who works at a foodbank, said: "We weren't sent out many letters - only 41. I didn't receive one myself. There are 64 residents who haven't had responses."
The planning application said the mast would be built on "land near a play area between Orchard Road and Saltwells Road".
However the mast has actually been built just over the road. Dudley Council says the location is still correct but Mr Lowe disputes this.
Mr Lowe says residents have been left angered by the mast and has accused Dudley Council of acting in a "cloak and dagger" manner.
He claims he smelt a "burning or sulphur" smell coming from the mast but said he had "no qualms about 5G".
Doreen Fox, who lives in a flat right opposite the pole, is furious about the phone mast.
Quarry Bank and Dudley Wood councillor Bryan Cotterill also shares concerns about the mast.
He said: "They should have been a lot more sensitive where they put it because where they have put it is totally unacceptable.
"I certainly wouldn't want it in front of my windows nor any windows of the people that I represent on the council."
In response to the Express & Star's story, Councillor Ian Kettle, cabinet member for regeneration and enterprise, said: "We understand that some residents are concerned that the location of the proposed mast has altered but this is not the case and the location where it is to be erected remains exactly the same as shown on the approved plan.
“Dudley Council wrote to all residents who live in close proximity to the development earlier this year to give them opportunity to comment.”