The 36-year-old, son of the West Bromwich lorry dealership Keltruck's owner Chris Kelly, has not revealed his reasons.
But it comes just days after the Tories were left reeling by the defection of Eurosceptic MP Douglas Carswell to UKIP. He resigned as an MP triggering a crunch by-election where he will stand for his new party.
Wolverhampton Grammar School educated Mr Kelly, who was elected in 2010, was among 79 Tories who rebelled against their own party in 2011 to demand a referendum on Britain's membership of the European Union.
However, Mr Kelly has said he has no plans to join UKIP, although he wished Mr Carswell 'all the best' and praised his 'honourable decision' to stand for re-election.
In a statement released last night Mr Kelly said: "I would like to thank all those who have supported me since I was selected for Dudley South in September 2007.
"I would especially like to thank all those who voted for me at the General Election in May 2010 and all of the friends and supporters who helped me for that election, as well as those who supported my efforts since.
"I am proud to have served the good people of Dudley South for the past five years."
Asked prior to his announcement whether he was planning to follow Mr Carswell to side with Nigel Farage's UKIP, Mr Kelly said: "I won't be joining UKIP, though I wish Douglas Carswell all the best. He has been a superb MP and it's honourable to stand for re-election."
Mr Kelly is not the only West Midlands Tory elected in 2010 to leave after just one term.
Mr Kelly worked as a marketing director in his father's West Bromwich-based firm. He has chaired Parliamentary committees on family business and combatting metal theft.
He and fellow Tory MP Justin Tomlinson placed £50 bets while they were at Oxford Brookes University that one of them would become Prime Minister before 2038.
Had either of them done so bookmaker William Hill would have to pay out £500,000.