Dr David Drew claimed he was sacked as senior paediatric consultant for Walsall Healthcare NHS Trust in December 2010 after emailing a prayer to colleagues and being told to 'refrain from using religious references' at work.
But the practising Christian failed to prove religious discrimination at an employment appeal tribunal in London earlier this year, after his case was thrown out at a tribunal in Birmingham.
And he said following a meeting with his legal advisers he has taken the decision not to pursue the matter any further.
The doctor had also raised concerns over a case where a baby was sent home and died and alleged there was bullying and understaffed wards.
But the trust had argued that Dr Drew, of Anchorage Road, Sutton Coldfield, had failed to accept recommendations made following an independent review sparked by him lodging a grievance against a colleague.
He was dismissed for gross misconduct and insubordination, for failing to accept recommendations from the review and for also disclosing confidential information.
The 65-year-old said today: "I am not going to take it any further. We could have gone up to the Court of Appeal but if I lost then I would be liable for the trust's legal costs as well as my own and I just cannot do that."
He added: "It may seem like defeat for me, but I know I have done the right thing. I have lost the last three years of my life due to fighting this and have lost hundreds of thousands of pounds in legal costs and lost income."
The doctor, of Anchorage Road, Sutton Coldfield, thanked his family and he said the many parents of children he has cared for over the years who had sent messages of support.
After the employment tribunal in Birmingham in March and April 2012, Judge David Kearsley ruled if complaints were made about Muslim or Hindu doctors who had quoted from holy texts, they too would be asked to refrain from such behaviour.
In a ruling following the appeal tribunal in February, Judge Jeffrey Burke QC said: "Much of the appeal was or amounted to arguments of perversity; but the employment tribunal had reached factual conclusions which it was open to them to reach."
The doctor, who had worked at the Manor Hospital since 1993, now plans to publish a book about his experiences as a whistleblower.