One in four new fathers don't take any paternity leave and few go beyond the statutory two week break, according to a new study.
Research among almost 1,000 employees and 800 managers found that a lack of support from employers was to blame for limiting time off among men after the birth of their child.
The Institute of Leadership & Management (ILM) said its survey also found that fewer than one in 10 new fathers take more than two weeks of paternity leave, falling to just 2% among managers.
The ILM said plans for shared parental leave from April 2015 will have little impact if "ingrained" attitudes were not tackled.
The report also warned that low levels of paternity pay affected take up of leave, with just 9% of fathers receiving anything longer than two weeks at full pay.
Chief executive Charles Elvin said: "The introduction of shared parental leave is a crucial step towards enabling more women to progress into senior roles, yet our research revealed cultural barriers are impeding the uptake of both two weeks statutory paternity leave and additional paternity leave.
"The paternity pay gap not only creates practical financial barriers to shared parental leave, it also reinforces a cultural expectation within organisations that women will be the ones taking extended periods away from the workplace, which may halt their career progression.
"The new shared leave proposals create a real opportunity for employers to address these issues around enhanced pay and retain the flow of female talent into senior roles."
TUC general secretary Frances O'Grady said: "Britain is the worst country in Europe for decently-paid parental leave. Many dads simply can't afford to take time off, particularly as employers rarely top up their statutory pay.
"Government plans to encourage dads to spend time with their new children are very welcome, but they are being completely undermined by the Chancellor's continued cutting away at paternity and maternity pay."
A spokesperson from the Department for Business, Innovation and Skills said: "We want the introduction of shared parental leave to drive a real cultural shift and help working dads play a greater role in their child's early months.
"Getting the detail right on this is crucial if we are to see an increase in take-up of dads taking leave. We will be providing guidance and an online tool for parents and employers to help them work out entitlement and to make the system easy to administer.
"Shared parental leave will be available to new parents after April 2015. Employers too can gain from a system which allows them to keep talented women in the workforce and have more motivated and productive staff. The new system will give us a great opportunity to make our workforce even more flexible, help working families and boost economic growth."
Labour leader Ed Miliband told ITV's The Agenda: "We need to do more. We are way behind other countries. We are way behind Scandinavia, the countries that are the world's best on this.
"We made strides, the last Labour government, but we have a lot further to go."
The programme also heard a less-than-flattering account of Education Secretary Michael Gove's contribution to family life during his paternity leave.
His wife, Sarah Vine, told the programme: "He hung around the house, eating biscuits and getting cups of tea made for him and looking at the baby and then reading another book, then looking a bit more at the baby a bit more.
"The thing is, when you've just had a baby, you sort of don't really want your husband around. Well, I didn't really want anyone else around."
Men could become "quite needy" when there was a cute baby competing for the attention of their partners, she suggested.
"Having the father around, sort of hovering in the first two weeks, well it was just a bit annoying. Maybe that's just my experience."