The former Dudley North MP, who quit Labour in 2019 in opposition to Jeremy Corbyn's leadership, said his old party had "ignored" the concerns of people outside the capital and was now "too weak to win an election".
Lord Austin urged Sir Keir to "get out of London" and listen to voters, and to set out a "vision people can follow".
He also called on him to "boot out the hard left loonies that poisoned the party" - starting with Mr Corbyn.
Ahead of the last general election Lord Austin urged people to vote Tory in his constituency to stop Mr Corbyn from getting into power.
He said today: "The voters warned Labour enough times before switching to the Tories and Thursday’s results show they are in no mood to think again.
"The country needs a decent alternative to the Tories but Labour limps on — strong enough to prevent a new party breaking through but too weak to win an election.
"Sir Keir Starmer needs to boot out the hard left loonies that poisoned the party and he should start with Corbyn himself. That would show he means business."
Outlining what Sir Keir should do to start winning back voters, Lord Austin added: "He should apologise properly for the way Labour let them down, ignored their concerns and asked them to vote for Corbyn.
"Show them he finally gets it and prove Labour can be trusted on defence, benefits and borders, as well as having a plan to bring good new jobs to the areas of Britain left behind for the past 20 years.
"And show some passion, for God’s sake. Tell us what you want to do with the country.
"Set out a vision people can follow."