The Shadow City Minister said financial services were "largely absent" from the trade deal with the EU, and accused Boris Johnson of ignoring the sector and dismissing concerns about its future.
The Labour MP for Wolverhampton South East said a properly regulated financial services sector could "help the UK succeed", but that anyone raising concerns about it was accused of "backing the bankers".
Mr McFadden said: "Financial services are largely absent from the deal the Prime Minister negotiated beyond the usual Free Trade Agreement statements about working together.
"The Prime Minister himself admitted that he had fallen short on what he wanted to achieve in this area. But the truth is he didn’t really try.
"Much now hinges on the separate memorandum of understanding (MOU) to be agreed by the end of March.
"Labour hopes that MOU agrees an equivalence regime, though even that will be a significant downgrade from where we were as a member state.
"The story of how financial services have been handled and in the end disdained by the Government speaks volumes about what the Brexit process has done to the Conservative Party. It can no longer claim to be a pro-business party.
"The approach to this issue demands more maturity than 'backing the bankers'.
"It ought to revolve around support for wealth creation and fair wealth distribution so that the country can secure both prosperity and the good society. That’s the approach Labour will take."
He added: "The bottom line is this is a sector which employs a huge number of people, is a successful exporter and generates significant tax revenue for the exchequer.
"Not only that but as the challenges of meeting our net zero targets or shaping the great acceleration generated by Covid emerge, it is a sector which, properly regulated, can help the UK succeed in the future."
Mr Johnson said the Brexit deal "perhaps does not go as far as we would like" over the financial services sector’s access to EU markets