The dog, believed to be a central Asian shepherd dog with cropped ears, was discovered behind Kingshill Park in Wednesbury, by a member of the public.
Officers from the RSPCA say it is not known how long the dog had been there, but it is thought to have been buried not long before being found on March 20.
RSPCA animal rescuer Vicki Taylor attended and she had to call for assistance due to the size of the dog.
She said: "It was a really upsetting discovery to think someone has treated an animal this way, just leaving him in a shallow grave in the dirt.
“We don’t know how he died or how he came to be here so really we are relying on local people to come forward with any evidence of what happened.
“He’s a big dog and there was a builder’s refuse sack nearby so that may have been how he was moved.
"We are urging anyone who noticed anything unusual to get in touch with us on our appeal line.”
The dog was chipped suggesting he had been owned, but unfortunately the details are out of date.
Anyone with information about the dog is asked to call the RSPCA's appeal line on 0300 123 8018.