The 53ft high structure is in the early stages of its build, but angry residents said they were not informed about the development by food distributor Kilo which was approved in April.
One neighbour, Malcolm Pearson, accused the authority of 'incompetence' and residents have been told to look at taking legal action by a local councillor.
However, Councillor Khurshid Ahmed, cabinet member of economic development and planning, said letters were sent out to almost 50 homes and notices put up at the site in Ham Lane.
Mr Pearson, of nearby Hawkeswell Drive, said he was astounded by Dudley Council, accusing it of 'incompetence' regarding the development.
"We have not been consulted and neither have the residents most affected in Keyes Drive," he said.
Councillor Natalie Neale said she was astonished by the 'awful' structure.
"The building is absolutely awful and it has appeared out of nowhere.
"I'm devastated for the local residents - I've never seen anything as big as this right next to somebody's back garden.
"It's not the fault of the applicant in any way, as they just went through the planning process.
"The council should have known how close it would be to the residential areas."
Dudley South MP Mike Wood to their cause.
Mr Wood said: "This is a complete and utter mistake.
"How on earth was this building recommended for approval without a full consultation exercise involving residents whose properties this structure now towers over?
"Having contacted the council it is clear this is their mistake and we look to them to now do what they can along with the developer to see what can be done to help those residents most affected by this appalling blunder," he said.
Councillor Ahmed said the council would work with concerned residents to do what they could to resolve the problems.
He said: "We consulted with residents who were affected by the development, which was granted planning approval earlier this year.
"Letters were sent out to almost 50 properties and a notice was put up at the proposed site, in line with national planning guidelines.
"We did not receive any objections from residents, but given that some are now concerned by the development, we will be visiting them to see what can be done to help them."
Kilo declined to comment.