The Saddlers made the perfect start after 14 seconds when Caolan Lavery notched his second goal in as many games to give them the lead.
After a first half in which Liam Roberts made several superb saves, only 10 minutes into the second half and Kinsella was shown a straight red card for a robust tackle on Jack Aitchison.
Walsall were fighting to keep hold of their lead but eventually Elliot List found a way past Roberts.
Both sides had chances in the closing stages but the Saddlers, despite having one less man, held firm. They left Hertfordshire with a point.
Brian Dutton made one change to his side for his first away game as head coach, starting with a 5-2-3 formation.
Emmanuel Osadebe came in for his first start since Boxing Day as Josh Gordon dropped to the bench.
Tyreik Wright, Stuart Sinclair, James Clarke and Rory Holden were still missing through injury. Youngsters Tom Leak and Sam Perry travelled with the team but were not involved in the squad.
Alex Revell made three changes to his Stevenage side, starting in a 4-4-2 formation.
Loanee goalkeeper David Stockdale made a late return to parent club Wycombe, meaning Jamie Cumming took his place in the starting XI. The 21-year-old is on-loan from Premier League club Chelsea.
Aitchison and Elliot Osborne also came into the team.
All it took was 14 seconds for Walsall to take the lead at the Lamex Stadium. Lavery chased down a back pass and as goalkeeper Cumming cleared it the striker slid in, blocked it, and the ball rebounded into the net.
Both sides started the game brightly and the Saddlers were keen to get Cameron Norman and Max Melbourne on the ball. Although Osadebe and Wes McDonald started wide they were regularly cutting inside to make space for the wing-backs.
Boro were not without their moments in the early minutes of the game, though, as Luke Norris was found in space but his shot was straight at Roberts.
List should have equalised after 14 minutes. A free-kick was lifted in and headed across goal. List arrived at the far post but sliding in he couldn’t reach the ball.
McDonald turned on the style and almost turned provider when he reached the byline and cut inside Luther James-Wildin. The winger pulled the ball back to Frank Vincent on the edge of the box but his deflected strike was saved.
Roberts kept Walsall’s lead intact with an excellent save when he clawed away List’s header that was destined for the top corner. The striker, who was heavily involved in the first half, was then played in on the right but shot into the side netting.
Walsall’s goalkeeper was again called into action to turn Osborne’s curled effort around the post. The hosts had grown into the game tremendously by this point, after 30 minutes.
With a barrage of attacks on the Saddlers goal, Roberts again had to tip over an Osborne effort. The chance came when McDonald stopped playing after taking a knock to the lip.
Despite the hosts having plenty of opportunities Walsall entered the break winning 1-0.
As the second half got underway Norman moved into defensive midfield, meaning Walsall changed to a 4-1-4-1 formation. Danny Newton replaced Norris for the hosts.
Just over 10 minutes into the half and the Saddlers were dealt a major blow. Kinsella went in for a heavy challenge on Aitchison and was shown a straight red card. Although it was a robust challenge it seemed a fair one on first viewing and perhaps Kinsella was unfortunate to receive his marching orders.
The incident happened right in front of the Stevenage bench, who reacted angrily. Left-back Ben Coker grabbed Kinsella by the scruff of his collar but was not disciplined for his actions.
The Saddlers were sitting back and trying to run down the clock but the hosts finally scored in the 69th minute. List was played in by substitute Joe Martin and finished beyond Roberts.
It was a scrappy half from that point but while Walsall stood firm defensively, they also showed fight to take home a point.
Walsall: Roberts, Norman, White, Scarr, Sadler, Melbourne, Kinsella, Vincent, Osadebe (Cockerill-Mollett, 86), McDonald (Gordon, 60), Lavery (Bates, 69).
Subs not used: Rose, Nolan, Osei Yaw, Reid.
Stevenage: Cumming, James-Wildin, Vancooten, Prosser (Martin, 62), Coker, Pett, Lines, Osborne, Aitchison, List (Stevens, 86), Norris (Newton, 45).
Subs not used: Johnson, Vincelot, Cuthbert, Read.