In front of 2.451 fans at the Banks’s Stadium, goals from Felipe Morais and Tom Dallison put the visitors 2-0 up.
Lavery pulled one back from the penalty spot but Ashley Nadesan restored Crawley’s two-goal cushion just 60 seconds later.
Lavery again netted to give the Saddlers hope. They then piled on the pressure in the closing stages with Cameron Norman coming closest to securing a dramatic equaliser when he rattled the crossbar.
Clarke named a strong line-up for the visit of Gabriele Cioffi's side with the boss opting to make just three changes from the team that started against Forest Green Rovers.
Zak Jules made his Saddlers debut in defence while there were also starts for Callum Cockerilm-Mollett and Alfie Bates.
It meant the Saddlers continued in a 3-5-2 system with Josh Gordon and Elijah Adebayo the front two.
Crawley lined up in a 4-2-3-1 system with Ollie Palmer the lone striker, Morais on the right, Ashley Nathaniel-George on the left and Nadesan in the number 10 role.
It was the Saddlers who started the game on the front foot and they came close to opening the scoring early on when Adebayo did well to burst past his marker and to the byline.
The former Fulham man then cut the ball back for Gordon but six-yards out the striker saw a deflected effort sail wide.
As the clock ticked past the 20 minute mark, Clarke's side had enjoyed more than 60 per cent possession.
But they found themselves behind when Nadesan collected the ball wide on the left and sent in a cross that was begging to be tapped home. Morais, inside the six-yard-box, duly obliged.
Walsall hit-back and almost pulled level when Cameron Norman's corner was met by Adebayo.
The striker nodded the ball down and into the danger area with Sadler latching onto it.
But his half-volley was expertly tipped over by Michael Luyambula.
The game was now proving an even affair. But in first-half stoppage time, Crawley should have doubled their advantage.
After Jules lost the ball, a cross from the right was met by Panutche Camara.
And he then saw a fierce low drive brilliantly stopped by Liam Roberts.
Three minutes after restart, though, the visitors did double their advantage with Dallsion powerfully heading in a corner from Nathaniel-George.
In response Walsall made a triple change with Wes McDonald, James Hardy and Lavery replacing Jules, Bates and Gordon.
And just 60 seconds later the Saddlers pulled a goal back when Luyambula hauled down Sadler in the box.
Lavery took the penalty and squeezed the ball past the keeper to score with his first chance.
Walsall now should have had the momentum. But the switched off and allowed Crawley to score immediately from the restart with Nadesan firing in a cross from Josh Doherty.
To the Saddlers credit, they didn’t let the set-back affect them.
And Lavery gave them renewed hope when he netted his second of the night in the 72nd minute.
The goal came from Norman’s clever thinking with the defender quickly taking a free-kick to find the forward in the box. And he then emphatically fired past the keeper.
From that point on Clarke’s men pressed for an equaliser.
And they thought they had one with a little under 10 minutes remaining when - after a spell of head tennis - the ball fell for Norman.
The full-back then smashed a volley towards goal but his effort clipped the top of the bar and sailed over.
Walsall: Roberts, Sadler, Clarke, Jules (McDonald 53), Norman, Cockerill-Mollett, Sinclair, Kinsella, Bates (Hardy 53), Gordon (Lavery 53), Adebayo
Subs not used: Rose, Pring, Facey, Kiersey.
Crawley: Luyambula, Docherty, Payne, Dallison, Palmer, Nadesan (Grego-Cox 59), Francomb, Nathaniel-George (Allarakhia 81), Tunicliffe, Morais (Ferguson 69), Camara.
Subs not used: Morris, Smith, Sesay, Galach.