The visitors took the lead through Leon Legge, who nodded in from a cross after a short corner, before Elijah Adebayo equalised.
Before half-time John Askey's side took the lead after Scott Burgess' shot was deflected but despite seeing his side fight back to earn a point - after a Josh Gordon equaliser - Clarke was unhappy with his defence.
“I’m not happy," he said.
“In the first-half for me, there’s not a lot in the game.
“Defensively we were very poor and I wasn’t happy.
“Second-half I thought we played some outstanding football.
“You won’t get too many pats on the back for a home draw from me.
“I thought we weren’t aggressive enough in the first half and paid them too much respect - switched off at a set piece again and they score a deflection, where we can do better.
“All of a sudden you're chasing a game again at home and then the response is emphatic and we were really unlucky not to win the game in the end.
“But you can’t be sloppy like that.
“I think we’re moving in the right direction, don’t get me wrong.
“But you won’t see me too happy with a home point, certainly with the way we defended in the first-half.
“It was probably a good game to watch for the neutral. Port Vale are a good team, obviously that’s why they’re up there at the minute."