Ibrar Shah, aged 38, was allegedly involved in a plot in 2006 to smuggle a lorry load of cigarettes over the border from Spain.
He was convicted in his absence by a French court and sentenced to a two-year jail term and 600,000 euro fine.
Shah, who is said to have links to Bell Flower Drive, Walsall, has since fought all attempts to get him to serve his term.
But after a hearing at the High Court in London, a top judge has finally now rejected his case and ordered his extradition.
Mr Justice Sweeney said it would not be a violation of his human rights to send him to serve his punishment.
The court heard Shah was alleged to be one of the organisers of the importation of Lambert and Butler cigarettes in September 2006.
The lorry was intercepted in the Pyrenees and the driver arrested.
Prosecutors said Shah's phone was found in the lorry and that he had been in Gibraltar days earlier.
There was also evidence of telephone contact between him and others involved.
Shah claimed he had an alibi but was convicted in his absence in Bordeaux in March 2011.
Last December his case finally reached a British court when District Judge Robin McPhee ordered his extradition.
At the High Court, his barrister Nicholas Hearn argued that the decision was wrong and should be overturned.
At the time of the hearing before Judge McPhee, he had three children of school age and has since had a fourth, the judge was told.
But giving judgment on Monday, Mr Justice Sweeney said that had to be balanced against the serious nature of the offence.
It involved international crime, sophisticated planning and multiple offenders, he said.
"In my judgment, the district judge did not make the wrong decision when he decided that the interference with the private and family life of Mr Shah was outweighed by the public interest in extradition," he said. "On the contrary, he clearly made the right decision."
He dismissed Shah's appeal against the extradition order.