Work has started on the facility which will house Carters Green Medical Centre and Lyndon Health Centre. Both will move to the hospital site from their current surgeries to serve more than 15,000 patients.
It is hoped having GP services on site will help provide patients with the best possible care and ease pressure on hospital staff.
Many people who arrive at A&E could have had their complaints dealt with by a GP.
Attending a sod cutting ceremony to mark the start of construction work, Toby Lewis, chief executive of the Sandwell and West Birmingham Trust, which runs the hospital, said the creation of the medical centre was an important step.
Sandwell Hospital's A&E department will be downgraded to an urgent care centre when the £475m Midland Metropolitan Hospital open in nearby Smethwick in 2022.
Mr Lewis said: "It's really important that we're opening a new GP surgery towards the start of 2021.
"The sense was when things change at Sandwell Hospital and we have an urgent care centre and all the outpatients people will stay but inpatients services go to Midland Metropolitan it will be important to get the closest possible link between primary care and hospital services.
"Our friends and colleagues at Your Health Partnership are working with us to open this large two-storey practice and it's going to be fantastic.
"It's a really obvious symbol of the way local general practice and hospital medicine are working more closely together."
Mr Lewis added: "It must make sense for people to be able to go to an urgent care centre at Sandwell, there's adjacent GP practice.
"A lot of people who go to things like walk-in centres will get a better deal going to their own GP because that means the person that knows you, has your records is seeing you so it really is a manifestation of what we're trying to do in Sandwell with health services."