It is understood that Councillor David Fisher agreed to step down from his role as group leader at a meeting last week.
Deputy leader Councillor Liam Abrahams has been appointed as interim leader.
Councillor Fisher, who represents Charlemont with Grove Vale, apologised earlier this month over a 2016 tweet in which he called for more "white representation within the medical profession".
He conceded that "some of my comments can be viewed as offensive".
It comes after the ruling Labour group elected its sixth leader in three years after Kerrie Carmichael took over from Rajbir Singh, who stood down for personal reasons and also quit as a councillor.
The Conservative group is said to be considering whether to hold a leadership election in the new year or leave it until after May's local elections, where they are aiming to double the number of seats held.
In this year's local elections the Conservatives took seats in the borough for the first time in six years. The party now has 10 councillors, having picked up an extra seat in July's Tividale by-election.
Councillor Jay Anandou is understood to have resigned as the group's deputy leader earlier this year. He did not respond to requests for a comment.
Labour meanwhile is dealing with another bout of internal strife over candidate selections, as speculation grows that up to 12 serving councillors could quit the party.
Last month protesters led by former MP Adrian Bailey chanted "no racism in Sandwell" outside Labour's regional headquarters after it emerged that four Asian councillors had been deselected.
Sandwell Conservatives did not respond to requests for a comment.