The South Staffordshire MP was the first ministerial departure as the Prime Minister today made changes to his top team, losing the position of Education Secretary he had held since July 2019.
In an interview with the Express & Star he said he had faced a series of major challenges in the role due to the pandemic, but believes he has left a "lasting legacy" through positive developments in post-16 education.
It came as the Prime Minister launched a cull of the West Midlands in Westminster, with Cannock Chase MP Amanda Milling and Wolverhampton-born Robert Jenrick also losing their Cabinet posts.
Mr Williamson said he had left the Department for Education with a positive future ahead under his successor, who has been named as Nadhim Zahawi.
He said: "I think education is in a good position. The reforms that we have brought in in terms of skills and technical education are so important, and we have seen record investment in schools over the last two years, with more money coming into education than ever before.
"One of the key things for me is we have always needed to change the debate about what youngsters do between 16 and 18, which is why so much of what I have done has been focused on trying to do just that.
"That's why legislation on skills and on improving options for young people – so it is not just about going on to university – has been so important to me.
"I think the impact of having options like apprenticeships and higher technical qualifications has been very positive, and along with being able to deliver investment in terms of new school buildings in places including South Staffordshire and Wolverhampton, I hope it will be a lasting legacy.
"Education is in the best possible place."
Mr Williamson met with a barrage of criticism over his handling of schools and exams during the pandemic. He was forced into a U-turn after thousands of results were downgraded in the 2020 GCSEs, and also faced calls to resign after changing his mind over this year's exams.
Asked if he had any regrets over his handling of schools, he said: "All the way through the pandemic we were put in a situation where we had to deal with things we had never had to deal with in the past.
"Undoubtedly, there were things we learned from, such as the awarding of grades in 2020 and people saw how we changed our approach in 2021.
"We learned from what happened and for me, that was the right thing to do."
Mr Williamson added that his departure from government would give him more time to devote to constituency matters.
"I'm certainly looking forward to throwing myself into local issues from the backbenches," he said.
"People right across South Staffordshire have been amazingly supportive and kind, and I can't say how much I appreciate their support.
"It's great to be able to have even more time to dedicate to the causes and the issues that are close to their hearts. I'll be using all that time to deliver the best for Staffordshire and the Black Country."
Meanwhile Mr Jenrick was sacked as Housing Secretary just days after visiting Wolverhampton to open a new government office at the i9 building.
He said it had been "a huge privilege" to serve in the role and that he was "deeply proud" of the department's achievements.
"I will continue to support the Prime Minister and the Government in every way I can," he added.
Former Dudley North MP Lord Austin was among those to pay tribute to his work, saying: "He’s done a good job for the Black Country, bringing MHCLG jobs to Wolverhampton and Dudley’s new hi-tech university campus was funded by him through the Towns Fund and will help many local people get better jobs for years to come."
Mrs Milling saw the curtain brought down on her 19-month reign as Conservative Party chair. She said on Twitter: "It’s been a privilege and an honour to be the co-chairman of the Conservative Party.
Thank you to the voluntary party and the team at CCHQ for their support. Thank you to Boris Johnson for this opportunity. I will continue working to deliver on our plans to level up the UK."