He also declared he would 'fight tooth and nail' to protect the Armed Forces, and spoke about the 'genuine threats' from Russia and North Korea.
Speaking less than two weeks since his appointment, the South Staffordshire MP said he would not hesitate in authorising strikes on British jihadists abroad.
He said: "They are going to inflict more and more harm on our country, so does that mean eliminating that threat completely? Yes it does.
"Do you want some of those foreign fighters potentially in this country? Do your readers? They certainly don't, so the job isn't done."
Hundreds of British-born Islamic State fighters are believed to have travelled to Syria, including at least a dozen from the Black Country.
Mr Williamson also warned that Russia was more dangerous than at any time since the Cold War.
And he branded North Korean dictator Kim Jong-un a 'madman'.
He said: "Let's be under no illusion that North Korea isn't a threat to us, they are a massive threat to us.
"This is a madman that cares little about human life and we need to do everything we can to get to a situation where he becomes less of a threat to this country and the world."
And he insisted he was not bound by the decisions of the past, when questioned about cuts to services including the Royal Marines.
He said: "I want to make sure the Armed Services are going to be best able to deliver a strong Britain.
"We've just got to make sure they have the equipment they need to do their job.
"We cannot let our Armed Services be the victim of people who don't have the imagination or the passion or the understanding of the important and vital role that they do."
He also said it was 'disgusting' that Northern Ireland veterans faced being hounded over alleged crimes decades ago.
And he hit out at Jeremy Corbyn as a pacifist who does not believe in Britain or the Armed Forces.