In an entertaining clash at the Banks’s Stadium, the Silkmen grabbed a two goal lead though Jamie Grimes and Harry Smith - with both goals coming against the run of play.
Walsall though managed to then turn the tie on its head with Kieron Morris, Andy Cook and Josh Gordon putting them into the ascendancy.
That had looked to be enough to get them into the third round but Tyrone Marsh equalised with a cross-come-shot in stoppage time.
And the Slikmen then went on to triumph on penalties with on-loan Manchester United keeper Kieran O’Hara saving spot-kicks from Cook, Luke Leahy and Morris.
“We are disappointed we have been knocked out,” Keates said.
“But there were a lot of positives to take out of the game. We’ve come from 2-0 down to take the lead at 3-2.
“We’ve had a lot of possession and have dominated for long periods.
“But we need to make better decisions in the final third. We have to be better with our shot selection.
"And we have to improve our crossing. Our final ball was not up to scratch.
“We had some great phases of play in the first-half especially. But the end product wasn’t there.”
Keates was pleased to see Gordon net on what was his full Saddlers debut since joining the club from Leicester in the summer.
“I thought he (Gordon) worked hard,” the boss continued.
“I think there were a few times in the first half he was a little bit too enthusiastic. He was trying to get involved in everything.
“We had a chat at half-time just to tell him to calm down and get himself in the 18-yard box and he scored his goal from there.”
Keates also confirmed he expects to make a signing before the transfer window for loan arrivals closes on Friday.
“I’d like to think there will be somebody in,” Keates added.
“I’ve had a conversation with someone today and hopefully we’ll get that done.”