Qamar Aslam was handed the ban, and a 20-month jail term, at Birmingham Crown Court in March 2011.
That was after he admitted dangerous driving and driving whilst disqualified after leading police on a high-speed chase in October 2010.
He was also told he must pass another driving test before he is allowed back behind the wheel of a car.
The 31-year-old, of Earlsbury Gardens, Perry Barr, was already banned when he committed the offences, having previously been prosecuted for dangerous driving.
He told judges at the Court of Appeal, in London, he was 'silly and naive' at the time but is now a 'changed man'.
Aslam said the ban is having a damaging effect on his job prospects.
He told the court: "I understand that the law is in place to protect people from harm.
"I didn't realise that at the time, I was young, silly and naive. I would just like to apologise.
"What I did was silly, but I have grown up now. I was in a dark place at the time."
The court heard he got married in 2015 and is now a family man, with a child on the way and a poorly mother who lives 10 miles away and relies on him for support.
He said he had been told by one firm he could start work for them immediately, riding a moped on a provisional licence, if he could get the ban lifted.
However he said he was struggling to support his family both practically and financially without being able to drive.
Lord Justice Davis told Aslam: "I'm sure you may be growing up and out of what you used to be doing, hopefully you are."
But he said the ban could only be lifted or reduced if it was unjust for the crown court judge to have imposed it in 2011.
Sitting with Sir David Calvert-Smith and Judge Simon Bourne-Arton QC, the judge encouraged Aslam to speak to a solicitor and apply to the crown court to get the driving ban reduced.