Sam Hall gave the visitors the perfect start at Coles Lane when he headed them in front after just six minute.
But the hosts were leading 3-1 at the break and Lye’s challenge got steeper when Hall was shown a straight red card for a challenge on Luke Shearer, two minutes into the second half.
Joshua’s men refused to buckle and hit back to snatch an improbable point with goals from Josh Hesson and Maxwell Ngbeken.
Joshua told Switch Radio: “We never have good results here. At half-time there was a real sense of déjà vu. We are delighted with the point.
“The red card throws everything out of the window. At that point it is damage limitation.
“We had to really dig in, so credit to all of the lads. To win the second half 2-0 win 10 men is testament to how far they have come.”
Lye are 12th in the Midland Premier Division table but just six points behind second-placed Darlaston with two matches in hand and are very much in the running to be best of the rest behind runaway leaders Walsall Wood.
The latter continue to hold a 13-point lead at the summit after second half goals from Jai Verma and Mitch Tolley, the latter from the penalty spot, earned a 2-0 win at Highgate United.
Darlaston were winners by the same scoreline at home to Uttoxeter Town thanks to goals after the break from Ben Podmore and Joshua Jones.
Boss Dean Gill believes it will require 15 wins from the remaining 20 matches to secure a top-five finish and a place in the play-offs.
He said: “It was another week of strange results in our league, where everyone from second to 15th will still think they can win promotion.
“I think we are now more or less safe in the division for another season. Now it is about finding those wins to make the play-offs.”
AFC Wulfrunians are very much in the mix after climbing to fourth courtesy of a 2-1 win at home to Whitchurch Alport. Goals from Aaron Bishop and Craig Gregg put Wulfs in command before a late goal from Ollie Pope ensured a nervy finish.
Stourport Swifts are one of the teams they leapfrogged in the table after the Worcestershire team were beaten 1-0 away at Studley.
Lichfield City sit sixth following a 3-0 home win over Market Drayton Town but Tividale lost ground as they were beaten 2-0 away at Shifnal Town. Wolves Casuals’ home match with Worcester City was postponed due to a waterlogged pitch.
Owen Massey bagged a first half hat-trick as Division One leaders Dudley Town thumped Paget Rangers 5-1.
They now hold an eight-point advantage at the top, with Bilston Town having dropped to third following a 3-0 defeat at Hinckley.
A double from Zaqib Hussain helped OJM Black Country beat Coton Green 4-1, while Wolverhampton Sporting were 2-0 winners at home to Haughmond.
Cradley Town remain bottom despite picking up just their third win of the season as they edged a five-goal thriller at Wednesfield 3-2.
Brocton bowed out of the Macron Cup as they were beaten 2-1 at home by Congleton Town.
The visitors play in the Premier Division of the North West Counties League one level above the hosts and eventually their superior class told.
Darren Chadwick headed home from a corner to give Congleton the lead but Brocton stayed in the game and levelled on the hour mark through a Connor Haddaway penalty. Parity lasted only 60 seconds as former Port Vale striker Tom Pope set up Ruchard Duffy for what proved to be the winner.