The Chancellor admitted he had concerns about immigration but said he believes the numbers coming from the EU to the UK will fall 'quite sharply' in the coming years as other European economies grow stronger.
Speaking during a visit to the Express & Star's Queen Street offices in Wolverhampton, Mr Osborne said: "You can control immigration from inside the EU, but to do so you need three things.
"You need to control non-EU immigration, which is nothing to do with our membership of the EU. Secondly you want to limit access to benefits, which we have done.
"Then crucially we have had this situation over the last couple of years where the British economy is stronger than the European economy.
"The good news now is that these other European economies are growing, so I think we could see immigration drop quite sharply."
The Chancellor added that there was 'absolutely no evidence' that immigration would fall should Britain vote to leave the union on Thursday.
"We are going to want to have access to the European single market so that factories here in the West Midlands can flourish," he said. "A condition of that is agreeing to free movement.
"British people are going to want to work in the EU, so we are going to have to accept French, Germans and Italians coming here."
Of the Vote Leave campaign he said: "I don't think they've got a plan to reduce immigration if we leave the EU.
"But if we listen to them we risk sleepwalking out of it and waking up on Friday to find that our country is poorer, not just for this generation but the next generation."
Mr Osborne insisted the Government was seeking to reduce the number of migrants coming to Britain, despite numbers of migrants rising steeply over the last year.
"Of course people have concerns about immigration, that's why we are seeking to bring down the numbers and control people who are coming from outside the EU," he said.
"And to make sure those who come from the EU put in before they take out in terms of welfare, but you don''t help anyone in this part of the world or help the immigration issue by doing real damage to the economy.
"I don't see how you address anyone's concerns about immigration by plunging the country into recession.
"You can address migration, but you don't do that by quitting the EU."
Last week Mr Osborne released details of an emergency Budget claiming Brexit would lead to £30 billion of spending cuts.
He said the EU made Britain 'stronger and more powerful' as a country and warned that leaving would mean 'less money' for the NHS and other services.
He also described the assertion that Turkey was set to join the EU as 'absolute nonsense' and laughed off suggestions of a European army.
"If people want to quit the EU I respect that opinion, but let's not do it on a fools prospectus," he said.
During his visit to the Black Country Mr Osborne stopped off at UTC Aerospace Systems in Stafford Road, where he met engineers and warned that Brexit would have a negative impact on trade for manufacturing firms.
The American firm, which employs around 1,000 people in the city, makes components for the world's biggest aerospace firms including Airbus and Boeing and is backing British membership of the EU.
Mr Osborne said: "There are parts made here that go in the Airbus engine and the Airbus wings.
"Now that is a plane that is assembled across Europe, the parts come from across Europe and the workforce here is concerned that if we cut ourselves off from Europe that we will not be able to do as much business here in the West Midlands.
"I do not want to see that happen, jobs put at risk here, I don't want to see families' security put at risk here, and I do not want to see prices go up here."