The Saddlers suffered their third straight League Two loss, which leaves them 17th in the table and closer in points to the relegation zone than the play-offs.
Tim Dieng and Cheick Diabate both headed home from corners in a 2-0 triumph for the Grecians, while Walsall had Rollin Menayese sent off for two bookings on another abject afternoon.
Saddlers chief Taylor said: "We started really well, on the front foot.
"We had an opportunity within the first couple of minutes, but we didn't build on that.
"In the middle part of the pitch, in between both boxes, we were good at 11v11.
"When you look at the scoreboard and you see it's 44 minutes, you think 'right, get to half-time at 0-0 and regroup'.
"But when you concede a goal, especially from a corner, the players that were given jobs to mark didn't do it.
"That's why we lost the game of football.
"That's not down to tactics. That's not down to anything other than players (not) doing their jobs.
"But everything that could go against us probably did so.
"We lost Stephen (Ward) to a bad calf injury, and then a centre-half gets sent off in Rollin.
"When we were in offensive areas, we probably weren't good enough. It's just about decision-making."
Menayese was sent off for two bookings and will miss Tuesday's visit of Bradford City, along with the injured Ward.
"It's decision-making. Rollin is extremely disappointed and it's another blow for us as we lose him for Tuesday night now," said Taylor.
"The group are extremely down having lost three in a row – there's no getting away from the facts.
"Part of my job is to pick the players up because in three days time, we've got another game.
"The players selected on Tuesday night need to give everything for the football club."
Goalkeeper Carl Rushworth, meanwhile, says he takes 'full responsibility' for Menayese's dismissal.
He rushed off his line before the centre-half hacked down Jevani Brown for a second yellow, having also been cautioned for a foul on Brown in the first half.
"With Rollin's red card, I take full responsibility," added Rushworth.
"It was like a fluke incident, a one-off, but I take full responsibility for him getting sent off. That killed the game, really.
"As soon as I got in, I apologised to Rollin.
"I feel bad for him because it was my mistake that led to him getting sent off, and now we miss him for the next game."