Councillor Steve Trow defeated Yvonne Davies 37-27 in a leadership election among group members at Sandwell Council House in Oldbury last night.
The election was insisted upon by Labour's National Executive Committee (NEC), which suspended Councillor Eling as part of an investigation into a number of allegations.
There was controversy before the meeting after a post on the Warley constituency Labour party's Facebook page branded the Labour group "snakes". The post was later deleted.
Mr Trow, who is the authority's cabinet member for culture and core council services and has represented Old Warley since 2010.
He served as Labour group secretary under Mr Eling from 2016-18, and could now lead the group until May, when a permanent leader is set to be elected following the borough's local elections.
Councillor Eling has vowed to stay on as leader of the council until a decision from the NEC, although in the coming weeks members may call a vote of no confidence in his leadership.
This could force a council leadership vote of all 72 councillors – including those who quit the Labour party and now serve as independents.
The authority is now in the unusual position of having a leader who is effectively not the chief of the party in control.
Sources inside last night's meeting told the Express & Star that the general atmosphere was "amicable", although a post on the Warley Labour party Facebook page – apparently in support of Mr Eling – is said to have caused "a degree of quiet anger".
The post read: "So the Labour Group has finally lost its marbles and is choosing a new leader this evening. Snakes, every, last, one!"
Tipton green councillor Syeda Khatun is staying on as deputy leader.