Some staff were in tears as they took part in today's protest, blaming the 'culture of fear' that has developed at the college.
It comes after bosses announced last month that jobs could be at risk due to funding cuts, with a review under way.
Staff walked out at 3pm, gathering outside the main entrance for around an hour. Earlier in the day, around 300 employees were called to a meeting in the lecture theatre when they voted twice for Ms Smith to step down.
Marina Bowler, course leader for sports and public service at the East Street college, was one of those who was protesting.
She said: "Staff feel they are operating in a culture of fear and they feel like they are being bullied. Workloads have gone up, stress levels have gone up. Staff are not being listened to and skills are not being utilised.
"Good staff have left the organisation and been made redundant.
"We know the further education sector is operating in strange times, we know it has been hit by the Government.
"But the problems at Stafford College are an aside to that. The situation here is not helping the college to grow and prosper. It is just getting worse."
Ms Bowler, who is also a University and College Union (UCU) representative, claimed the issues have been ongoing since Christmas and said staff had now reached the point where desperate action was needed.
"I think initially we were viewed as a small militant faction of lefties but it is totally not the case as proven by events today," she continued.
"This is a lobby with a view to further action. We have been trying for six months to negotiate with the college. We don't want to go down the route of industrial action. We are passionate about the college it is an amazing place to be.
"But that care is not being supported by a positive culture and it is not being supported by principal Beverley Smith."
Some protesters were ex-employees who have left their jobs in recent months.
Rebecca Stewart, regional official at UCU, said: "I am just astonished after such a mass protest and all of see votes of no confidence that the principal has not considered her position.
"It is humiliating. It is becoming a soap opera about Beverley Smith and not about the college but it is causing long-term damage. As we know the funding cuts in further education have been pretty brutal. But this is not about the money it is about the way the staff are being treated.
"They are not being listened to. It is the increased work loads, unrealistic target setting and the simple way people are spoken to."
A statement from the college said: "The leadership team are working closely with UCU and Unison about recent developments at the college. We are waiting on further direction from our board of governors. We remain committed to our students and thank staff for their hard work and professionalism during this time."