Speaking to Sky News Mr McFadden said despite many supporters backing the Government's pandemic policies and job furlong schemes, the furore over reports of parties being held in Downing Street in the height of lockdown meant fuel and cost of living issues were not being tackled.
He said although some people were pleased with Mr Johnson's track record so far, his behaviour mattered.
Mr Mc Fadden, also the Shadow Chief Secretary to the Treasury, said: "It matters for two reasons. First of all, the country made great sacrifices people couldn't see loved ones, they tried to do the right thing and then they find there's not just one transgression in Number 10 there's a whole series of them.
"Obviously there is a culture there of one rule for the public and a different rule inside Number 10.
"I used to work at Number 10 before I became a MP. I know that culture at Number 10 comes from the top. It comes from the PM of the day.
"But it matters for another reason too. This has become so dominant it is undermining the governance of the country. You have to ask yourself whose concerns is the PM thinking about of the moment. Is he thinking of high energy bills, is he thinking about the cost of living crisis facing the people in Wolverhampton or around the country. Or is he just thinking about his own survival.
"What we learn is that he's really thinking about is what's being called the 'red meat' strategy which is a private negotiations between him and his back benchers."
He said policies over troops being sent to the English Channel and debates over the BBC licence fee were essentially a distraction and were not addressing issues including potential tax rises and fuel prices which were expected to add £1,200 to household budgets in the coming financial year.
"It matters because we should have trusted the PM about the rules, but it also we need a government that is going to address the real issues facing the country."
"He may or may not ride it out. But the view of how the public see this. Doesn't really matter. It's not healthy for democracy," he added.