A Liam Kinsella own goal in the 83rd minute, under pressure from veteran striker James Vaughan, was the difference between the two sides at the Banks's Stadium.
It leaves the Saddlers in 21st in the league, winless in nine games and is their fourth loss in eight league games.
In all competitions they have scored 5 goals in 10 games.
And Clarke says everyone is failing, but urges the fans to stick with his side.
“At the minute we’re failing miserably, I’m failing miserably, I lead the troops and we have to do a lot, lot, lot better than we have been," he said.
“I know it’s boring and fans will be sick of hearing it but you have to keep working with it and working with the players and changing the mindset of the players in that final third, you keep doing that and eventually it’ll turn.
“You’re not going to score a goal if you’re outside the box, get in the box, commit players into the box, then it has to be a better cross, crossing was poor today.
“Options in and around the box weren’t there, players not taking a shot when they can have a shot, there’s so much work on."
Despite the poor start, the 41-year-old is adamant that the tide will turn, with the final phase of play still yet to click for his young side.
He added: “I’m sick of saying it, five, six weeks we’ve hammered our forward play, we’ve hammered our work and in training the ball is hitting the back of the net but in matches it’s not going anywhere near it, so we’ve got to keep working and drilling it.
“Making sure players don’t lose confidence, making sure that next time we step on the pitch on Tuesday night at Morecambe that we’re ready to go again.
“We’re in a disappointing position at the minute, but let’s see where we go, you can comfortably lose 15 games in this league and still finish in the top half of the table.
“So we’ll be working away, chipping away, and building a football club that I think needs building, that’s why I came in and took the job."