The 84-year-old convicted paedophile was transferred from HMP Bullingdon, in Oxfordshire after being repeatedly targeted by other inmates, sources said.
It comes as police say they are investigating after 10 further women claimed they had been sexually abused by the children's presenter.
He was convicted of 12 indecent assaults on June 30 at Southwark Crown Court by Mr Justice Sweeney – one on an eight-year-old autograph hunter, two on girls in their early teens and a catalogue of abuse of his daughter's friend over 16 years. He was jailed for five years and nine months for the sex abuse, meaning that he is due to serve just under three years for the crimes, which spanned between 1968 and 1986.
In July it emerged the former TV personality was spat at by an inmate during a prison church service which sparked a mass brawl. Other prisoners verbally abused Harris, who painted an official portrait of the Queen, from the pews of the chapel.
Elsewhere, it is claimed Harris faced threats from prisoners on his wing for hogging the prison phones.
A source said: 'The other prisoners were keen to take him down a peg or two. He was moved to another prison for his own safety about two weeks ago.'
Stafford Prison completed its transition to a centre for sex offenders only last month, with all 741 inmates at the 200-year-old town centre category C jail now having been convicted of sex crimes.
Stafford MP Jeremy Lefroy said today: "I'm not going to comment on anyone who may or may not be at the prison. When people enter into prison, it's entirely up to the prison service to move them. I supported the move from having 50 per cent of inmates who were sex offenders up to 100 per cent. I have not changed my mind on that since."