A dramatic equaliser from substitute Andy Williams two minutes from time followed by a winner five minutes later from Douglas James Taylor provoked scenes of celebration at The Bescot.
Walsall had met Carlisle two weeks ago in the league when they fought out a dour 0-0 draw.
Both clubs enjoyed league wins last weekend so this game was expected to be close, with a place in the third round of the FA Cup at stake.
With Cambridge United still being involved in the cup, Liam Bennett wasn't available for selection at the request of his parent club.
Forward Conor Wilkinson started on the bench for the first time this season for Walsall.
The early exchanges were scrappy but Carlisle managed to find their feet first and Owen Evans produced a great save from Taylor Charters low to his left after the striker had been set up by Jordan Gibson
went behind on 28 minutes when Jack Armer crossed low from the left and the ball wasn't cleared by the defence, it eventually came out to Ryan Edmondson to score.
But four minutes later Donervon Daniels very nearly equalised when he ran into the box at the Carlisle defence and unleashed a shot that was only just cleared off the line.
Isaac Hutchinson then headed just wide from a free kick but these were fleeting moments from Walsall and it was their opponents who looked most dangerous throughout the first half.
Walsall took two players off at half time to try and turn things around and one of the substitutes - Taylor Allen - almost made an impact on 57 minutes with a shot in the box from close range which was instinctively saved by Tomas Holy.
But Carlisle were looking fairly comfortable in midfield, though Walsall were breaking well, Liam Gordon producing a good cross just before he was taken off, with nobody there to convert it.
But Walsall were chipping away at The Cumbrians and got their reward two minutes from time when Holy made hash of controlling a clearance and Williams chased the loose ball, beating the defenders and bundling it home.
Three minutes into injury time James-Taylor produced a winner that the Saddler's pressure deserved to take them into the third round.
28 GOAL-Walsall's defence didn't clear a cross from the left by Jack Armer and Ryan Edmondson scored from the edge of the area.
88 GOAL- Thomas Holy couldn't control the ball and Andy Williams bore down on the loose ball, bundling it home.
93 GOAL - Douglas James Taylor bundles the ball home for the win
Walsall (3-4-1-2) Evans; White; Gordon (Williams 71); Daniels; Monthe; Kinsella (Cashman 76); Knowles (James-Taylor 58); Comley (Allen 45); Maddox (Earing 45); Hutchinson; Johnson.
Subs not used: Barrett (GK); Wilkinson; McEntee;
Carlisle (5-3-2) Holy; Armer; Moxon; Huntington; Gibson (Harris 76); Edmondson (Sho-Silva 81); Dennis (Stretton 62); Charters; Whelan (Barclay 81); Ellis; Mellish.
Subs not used: Kelly (GK); Hilton; Devitt; Kelly; Idehen