The South Staffordshire MP spoke to the Express & Star after bringing an end to his latest Cabinet post last night, saying claims over his conduct had become a "distraction" to the Government's work.
His role as a Cabinet Office Minister without portfolio lasted just two weeks and followed spells as Chief Whip, Defence Secretary and Education Secretary which he held under two different Prime Ministers.
He has vowed to clear his name of any wrongdoing following claims over bullying and intimidation, including that he told a senior civil servant to "slit your throat".
Sir Gavin told the Star he would be returning to the backbenches to do his "very best" for the people of South Staffordshire.
He said: "I am very much focused on my constituency work, which is the thing that has always given me the greatest pleasure and satisfaction."
He also confirmed that he will not be taking any severance payment from his time in office, saying it should go instead "toward the Government‘s priorities like reducing the NHS’s waiting lists".
His resignation came after a series of claims against his behaviour towards MPs and civil servants over the last six years.
They included texts he sent to Aldridge-Brownhills MP Wendy Morton when she was Chief Whip complaining about not receiving an invitation to the Queen's funeral.
Mrs Morton submitted complaints to Conservative Party headquarters and the parliamentary bullying watchdog regarding the texts, which were leaked to the press over the weekend.
Speaking in the Commons today in response to a question from Labour leader Sir Keir Starmer, Prime Minister Rishi Sunak said Sir Gavin's behaviour was unacceptable and that it was "absolutely right" that he resigned.
Mr Sunak appointed Sir Gavin to his Cabinet last month after the South Staffordshire MP had assisted in his parliamentary campaign to reach Number 10 following the resignation of Liz Truss.
Labour's Shadow Chief Secretary to the Treasury, Pat McFadden, highlighted Sir Gavin's sacking as Defence Secretary under Theresa May in 2019 over claims he had leaked information from a National Security Council meeting.
The Wolverhampton South East MP, said: "The appointment of Gavin Williamson following the serious national security concerns when he was last in government exposes Rishi Sunak's lack of judgement. The same is true of the Home Secretary.
"The Prime Minister promised us integrity on the day he was appointed, but these appointments have shown the opposite and reappointing Gavin Williamson to government exposes a serious lack of judgement on the Prime Minister's part."