Iain Duncan Smith's comments came as he visited Midland Chilled Foods in Willenhall.
After touring the company's Stringes Lane site and tucking into a Black Country faggot and paes pie, he said: "This is a company that was started by someone with drive and energy from nothing. It is now an incredible success story but the message coming across is that more and more EU regulation is making it more expensive and becoming an overbearing burden.
"Despite that the business is success but just imagine how more successful it could be. The only way you can get rid of that regulation is to get out of the EU. For small businesses it is an absolute must."
The company was founded Peter Shirley in 1976 and today he employs 220 people on the site and the company supports many more jobs across the Black Country.
Mr Shirley, who supports leaving the EU, said: "I just cannot see the benefit of us staying in. I have suppliers in France and Spain and if we were to leave the EU they are telling me they are still desperate for our business." Mr Duncan Smith also said immigration was no longer a 'taboo' subject and a 'proper' discussion was taking place.
It comes after the Archbishop of Canterbury declared it wrong to label people as 'racist' for fearing immigration.
He said: "Hallelujah. For years we were told to talk about immigration was racist. It shut the debate down and what happens if you shut down the debate is that then it goes to the margins. If you don't get a proper debate you can't have a reasoned answer. If you take control of the borders you do not get migration, you get controlled migration.
"If there are companies who need certain skills they can talk to the government and they can let certain people in.
"There is nothing wrong about worrying for your job because of immigration.
"These are not extremists or racists. They are ordinary people going about their business and their work.
"We should have had this debate years ago."
Talking about the row with the Queen after it was reported she favoured leaving the EU, Mr Duncan Smith said: "I think everyone should calm down sit in a dark room somewhere and continue to take their tablets because this is going nowhere."