The scandal at Walsall Manor Hospital came to light following an internal review and it has been blamed on an 'administrative error'.
Bosses at the hospital today apologised and a full independent investigation has been launched.
Walsall North MP David Winnick today described the news as 'puzzling and distressing.'
It has emerged there were 86 occasions where remains from miscarriages or terminations were kept for longer than they should have been.
After advice from clinicians, the Trust chaplaincy and relevant charities, hospital bosses have decided to only contact five women, where records indicate they would want to be informed, rather than cause 'widespread distress' to others.
Two of the cases date back to 2010, a further two in 2011, a total of 30 in 2012 and then 52 up to December 2013.
Around 700 foetal remains are kept at Walsall Manor every year until the appropriate paperwork has been completed. The hospital said it then 'appropriately disposes' of foetal remains through the crematorium in line with 'usual processes'.
A dedicated hotline has also been set up for anyone with concerns on 0345 835 7626.
Reacting to the news Mr Winnick said today: "I find this very puzzling and distressing.
"There needs to be a full explanation and if necessarily a full inquiry.
"It will come as a shock to the people in the borough and understandably so."
MP for Walsall South Valerie Vaz said she did not want to comment until she had seen the full details surrounding the case
Such remains are usually stored in the hospital's mortuary for between one and three months. Then in the vast majority of occasions they are sent to be cremated.
However, an error in getting documents signed off to allow this to happen led to the foetal remains in these cases being kept for up to four years. In 73 cases they belonged to foetuses under 12 weeks with the remaining 13 being under 21 weeks.
Trust chief executive Richard Kirby told the Express & Star: "It is clear that in some cases the trust has kept remains following terminations and miscarriages for longer than we should have done.
"I would like to apologise to each and every patient who has been affected by this and would like to re-assure anyone who might be using our services now or in the future of our dedication in ensuring that this situation does not arise again.
"We are confident we have contacted everyone we need to in relation to this issue."
Mr Kirby said: "There are five of the 86 who we think we should contact.
"For the others, the feel of all the advice we have taken and all the work we have done, we risk causing more distress by making contact.
"We will now be proceeding to arrange the cremations in line with our standard policy.
Mr Kirby said that without the relevant certification the hospital was unable to release the remains for cremation.
The full reasons behind what went wrong in the 86 cases which have been discovered are now being looked at.
"We have asked for an independent review.
"We want a really thorough understanding of what went wrong," he said.
The hospital has received advice from the likes of the Miscarriage Association and bereavement teams on how to contact and support patients.
Mr Kirby said the results of the independent review are expected at the end of April .
The hospital said it will 'appropriately dispose' of the foetal remains through the crematorium in line with 'usual processes'.
If a pregnancy ends before the 24th week, it is known as a miscarriage as the baby has not developed sufficiently for it to survive outside the mother's womb.
Under UK law, a termination can usually only be carried out during the first 24 weeks of pregnancy as long as certain criteria are met.