Mr Anderson said abuse and threats against his family and staff had "taken a heavy toll" and left him with no choice but to stand down at the next general election.
He said the "toxic political environment" in Wolverhampton had gradually worn him down until his position became "unsustainable".
"It's been a terrible time for my family and I have to put them first," he told the Express & Star.
Former soldier Mr Anderson was elected as the Conservative MP for Wolverhampton South West in 2019, winning the seat off Labour with a majority of 1,661.
But over the past 18 months he says he has faced a series of threats to himself, his family and office staff. The incidents have included 19 that have gone to the police, including death threats and vandalism.
Four cases have gone before the courts, including a threat to blow up his office which saw Abdul Alam, 38, handed a suspended sentence.
Explaining his decision to stand down, Mr Anderson said: "Winning the election and taking the seat from Labour to become a member of parliament in Wolverhampton, I still pinch myself, it's such a huge honour.
"My family are a major part of my life and when I first moved here I put them front and centre. The majority of people in the city have welcomed us with open arms, but there is such a toxic political environment in Wolverhampton where you are either blue or red, and it can spill over into abuse.
"Sadly my family – who are not political at all – became a target for that hatred, as did my staff. It has taken a heavy toll on all of us.
"For me it has been a constant worry whether they are okay and eventually the "drip, drip" of abuse has worn us down.
"I've looked at the impact on my team and my family and the position has become unsustainable."
Mr Anderson moved to the city from Herefordshire where he was a local councillor. As an MP he has served on the defence select committee and is a government whip.
He said he hopes his political journey will not end here and plans to apply to stand elsewhere.
The city's other Conservative MP, Jane Stevenson, who represents Wolverhampton North East, said: "I know some of the abuse Stuart and his family have had targeted at them and it's entirely unacceptable.
"I'm not surprised a family man with young children and a wife is thinking twice about standing as a candidate. It's a terrible state of affairs when a public servant feels unable to stand again because of these circumstances."
Another chapter in seat that is no stranger to controversy
It has long been a constituency scarred by controversy.
Wolverhampton South West was, for almost a quarter of a century, the seat of Enoch Powell, who achieved notoriety for the ‘Rivers of Blood’ speech that still sparks debate to this day.
More recently Labour MP Eleanor Smith became embroiled in a race row over comments she made about the Black Country flag.
Now another chapter in its tempestuous history has been written.
Stuart Anderson has announced he will not be standing at the next general election in the constituency, having gained the seat from Labour in 2019.
The father-of-five puts his decision down to the amount of abuse he and his family have faced since moving to the city, which he says has taken a heavy toll and left him with no choice but to stand down.
He believes the sheer number of times he has been targeted to be extreme, even in an age when the abuse of MPs, particularly online, has become commonplace.
In just over a year his office reported 19 incidents to West Midlands Police, including death threats, harassment, abuse and intimidation.
Four cases have gone before the courts, including a threat to blow up his office which saw 38-year-old Abdul Alam handed a suspended sentence.
Mr Anderson said the “toxic political environment” in the city had been a major factor in his decision to depart. He says the abuse sometimes followed Commons votes, such as when he joined other Tory MPs in opposing a Labour motion on free school meals, or over lockdown measures.
Speaking about his family, he told the Express & Star: "In hindsight it may have been easier for them to live outside the city.
“As an MP you effectively have two jobs, one in the constituency and one in London, where I spend four days a week. My team who are tasked with managing the day-to-day in Wolverhampton have increasingly been dealing with threats and assaults.
“Wolverhampton is one of the most politically charged environments I have ever seen. People on the streets will tell you straight if they are Labour or Conservative.
“There is no middle ground, and my family have been branded the same as me and have had to deal with a lot of hatred.
“It has been an agonising decision to go. Nobody can say I haven’t given my all for Wolverhampton and together with my team we have achieved so many successes, particularly when it comes to bringing in investment.
“It has been a terrible time for my family and I have to put them first. I know my decision is the right one.”
Mr Anderson insisted it was possible for the Conservatives to retain the seat, which is a classic marginal that swings between red and blue.
It has been earmarked as a key target for Labour, although as the city’s Momentum stronghold it is posing its own set of problems for Sir Keir Starmer.
Before Christmas it emerged Labour had delayed its selection process over concerns about the quality of the prospective candidates.
The other two city seats have had Labour candidates in place for several months.