In its summer industry forecast, industry body The Construction Products Association cited 'concerns regarding planning and financing' as reasons why it does not think main civil engineering works will start for five years.
And experts say people affected by the £50 billion rail line are also potentially causing 'delays through appeals'.
The Government announced in June that construction of the high speed link, which will connect London, Birmingham, Leeds and Manchester with 225mph trains, would start in 2017.
And the hybrid bill, the legislation permitting HS2 to go ahead, is on course to be signed off with Royal Assent by the end of 2016.
The CPA, a group that represents around 30,000 companies across product manufacturing, said work is unlikely to begin until 2020.
CPA's economics director Dr Noble Francis said: "When you look at major rail infrastructure, there is usually a significant period of time before projects get off the ground.
"Thameslink was originally the Thameslink 2000, Crossrail was given the green light in 1990. You won't see those sort of delays to this project, but when we are talking about major infrastructure, we as forecasters have to forecast a degree later than officially stated."
Campaigners fighting HS2, which would would cut through swathes of Staffordshire countryside, have hailed Dr Francis's views as the latest evidence that it should be scrapped.
Stop HS2 Campaign Manager Joe Rukin said: "There have been concerns since the start that HS2 would be late and over budget, but despite the fact it is already late and over budget, HS2 Ltd always dismiss anyone who criticises their competence, no matter how expert and impartial they are. The thing is, it always turns out those impartial experts were right and HS2 Ltd were wrong."
HS2 bosses insist they do not forecast any delays. HS2 Ltd spokesman Richard Pain said: "Successful planning is the key to most major projects remaining on schedule. Following a resounding backing from parliament at the second reading of the hybrid bill for HS2 Phase 1, we remain confident that the main construction of HS2 will start in 2017, with civil engineering work to begin in 2018."
He added: "Substantive design work, surveying and ground investigations along the entire route of HS2 Phase 1 is already underway and, following on from extensive industry communication and supply chain engagement work, we plan to start the procurement process for civils contracts this autumn."