The High Court rejected an emergency injunction to halt "enabling" works, brought by conservationist Chris Packham, with judges declaring there was "no real prospect of success" for a judicial review.
The decision gives HS2 the green light to start "translocation" work – where trees are felled and soil removed – at five Midlands woodlands in the coming weeks.
The sites include Fulfen Wood, an ancient woodland between Huddleford and Lichfield, prompting campaigners to warn that the "irreplaceable" woodland will soon be lost forever.
They also say the work will now begin at the worst possible time, disturbing nesting birds and other species, as well as reducing the chances of survival for any replanted woodland flowers.
The hearing last week heard it would cost £20m–£25m to "press the pause button" on HS2, while a full judicial review of the government’s backing for the budget-busting project could be considered.
It comes after HS2 workers at some sites were accused of flouting Covid-19 social distancing rules, while Stone MP Sir Bill Cash has called for the entire project to be scrapped in order to offset the huge financial strain on The Treasury caused by the current lockdown.
Staffordshire Wildlife Trust's senior conservation manager, David Cadman, said it was "an act of betrayal" by the Government that the work was taking place during the coronavirus crisis.
"The injunction being placed and the work being halted would have been absolutely the right thing to do, so it is devastating for wildlife, breeding birds, bats and butterflies that it hasn’t, and is completely short sighted," he said.
"It completely goes against best practice and conservation principles and professional standards.
"There’s a human element too. People are being urged to stay at home due to coronavirus and the safety of the construction workers, the police and protestors, are being put at severe risk.
"Once we get to the other side of the coronavirus crisis, it is massively important our Government address the issues of biodiversity loss and climate change as an immediate priority.
"This crisis really calls into question whether we should be proceeding with this costly and unnecessary project at all."
Suzanne Savage, of Extinction Rebellion Midlands, said it was "utterly reprehensible" that ministers were pressing ahead with HS2 despite its spiralling costs at a time when the NHS was under unprecedented strain.
The first stage of HS2 runs from London to Birmingham, with Phase 2a ploughing through 45 miles of Staffordshire countryside on its way to Crewe.
At a cost of £106bn it is already almost triple its original budget, although critics believe it will end up costing far more.
The Department for Transport said the court's decision would allow the Government to continue its work to "level up the country".
"HS2 will be crucial to that ambition, boosting capacity and connectivity across our rail network," a spokesperson said.
"We have tasked HS2 Ltd with delivering one of the UK’s most environmentally responsible infrastructure projects."