The Saddlers went ahead through Kieran Phillips but fell apart in the final half an hour as Connor Taylor and Sion Spence struck for the visitors.
Failing to deal with deliveries into the box, Rovers were handed their first away victory since last December.
Walsall, meanwhile, now sit 22nd in League Two, with eight of the 13 goals they have conceded this season coming from set plays and Taylor said: "It's unacceptable.
"We've conceded 13 goals and eight of them have been from set plays.
"From my perspective, you always look at yourself and your coaches first.
"We spent two days last week working on set plays, so we need to make sure we're better for the next game.
"We weren't cut open by the opposition, but players have not done their jobs from set plays and it's cost us.
"It's quite obvious it's not good enough, and it's costing us points."
The Saddlers were slow out the blocks at the Banks's Stadium but were in front at half-time thanks to Phillips' cool finish.
They then allowed Rovers into it after the hour amid a series of changes.
Brendan Kiernan went off for Emmanuel Osadebe, who later went off injured.
Phillips was also substituted before Taylor and Spence got the visitors' goals.
"For the first 15 minutes, we probably, again, gave the opposition too much respect," said Taylor.
"But we sorted that and came on strong in terms of our influence on the game.
"We went in at 1-0 and I was expecting the players to feel confident.
"We started again, though, a little on the back foot – I don't know why.
"Brendan rolled his ankle and Kieran had a tight hamstring.
"Osadebe has been tackled and I can't believe it wasn't at least a foul. It looks a nasty one for him.
"There was a lot of disruption, but that shouldn't have caused the issues it did, where players are not doing their jobs from set plays.
"You can't concede from two set plays in the way we did.
"It's an issue which we had identified and, obviously, we need to practice more."
Taylor added: "The result wasn't good enough. In any league, set plays are hugely important.
"You've got to do your job for and against, and we didn't do that as a group.
"If we continue to make errors like we did, we're going to win less matches."