Sir Michael Lyons, former chairman of the BBC Trust and former Wolverhampton Council chief executive, made the remarks in an interview with the Express & Star as he introduced a meeting in the city to discuss housing policies across the West Midlands.
It is directly opposite to speculation that the Conservatives plan to expand Margaret Thatcher's flagship right to buy to cover 2.5 million housing association properties as well as council homes, with some people offered a stake in their home for free if they hold down a job for a year.
David Cameron has said a Conservative government would guarantee the right to buy council homes.
Labour's shadow housing minister, Wolverhampton North East MP Emma Reynolds, also stressed the party would not be scrapping right to buy.
Sir Michael said: "We will see an end to the right to buy within the next five years and a recognition that it was a failure. It was introduced in the belief that it would lead to more home ownership and more people, through owning their own homes, investing in their communities.
"But homes sold under right to buy often ended up as rented accommodation."
He also criticised the rate of council homes replaced after being sold under right to buy, saying significantly more were bought than were built to replace them.
And he warned of the speculation on taking it further: "Extending this to housing associations would be disastrous." Sir Michael said it would deter housing associations from building homes because their assets would be 'taken away' from them.
Sir Michael has set out how he believes Labour could build 200,000 homes a year by 2020 but told the Wolverhampton City Council-backed meeting at Molineux he believed the country needed 240,000 to 250,000 a year.
Council leaders across the West Midlands are going to work together to tackle the housing shortage but said they would not be coming up with region-wide plans that would impose more housing in some areas than others.
They also said the answer was not just giving approval for more homes to be built amid calls for government support.
Sir Michael was chief executive of Wolverhampton Council when it bought Molineux and the surrounding land after the club entered receivership at the end of the 1986. It was then sold to Sir Jack Hayward who invested in it during his his 17 years as owner.
Sir Michael was chairman of the BBC Trust from 2007 to 2011. He is now on the board of Wolves owner Steve Morgan's house building company, Redrow.
He told the meeting of the West Midlands Local Government Association of council leaders, along with housing executives, that Britain had not built 200,000 homes a year since 1988 and now faced a 'housing crisis'.
"We need a government willing to say we will build 200,000 homes a year," he said.
Sir Michael called for stronger powers for councils to compulsorily purchase land and for an increase in all forms of house building, including for private ownership and affordable rent.
He also said the soaring housing benefit bill was not down to rising numbers of 'feckless' people but because the supply of new homes was so low it was driving up prices and rents, leaving them unaffordable.
Miss Reynolds said it was not Labour's policy to scrap right to buy.
She said: "We support people who want the security of owning their own home. In government our priority will be to ensure councils are able to replace every home sold on a one for one basis. The government promised such a replacement but have failed to do so in practice. For every 21 homes sold through right to buy, only one home has been built."