The hosts had three new signings in their line-up – Dan Sweeney, Joel Taylor and Callum Mendez-Jones, who joined from Solihull Moors, Kidderminster and Hednesford respectively.
Another arrival from Kidderminster, Harry Higginson, was on the bench.
And Yeltz had the perfect start in the Southern Premier Central clash when Lewis Reilly fired them ahead after 10 minutes.
But Joshua March struck in the dying seconds to deny Rob Smith’s men. Rushall Olympic progressed past Chasetown in the Staffordshire Senior Cup, winning 5-4 on penalties after the game finished 2-2.
Jazz Luckie and Tom Hill netted for the Scholars but Danny Waldron struck a second-half brace to haul Olympic level.
Stafford Rangers beat Marine 4-3 on penalties to reach the last eight of the Integro League Cup.
The game finished 1-1, Luke Jones netting for Rangers.