Tigers had the majority of the possession for long spells, and probably shaded the first half.
But Albion took their chances with a masterclass in counter attacking football, and clinical finishing.
Worksop midfielder Harrison said: “All the lads coming in at half-time were buzzing, playing really well, passing the ball well.
“We had a few chances I think we could have done better with, but it was 1-1 at half-time.
“We came out second half, started really brightly and then just before they score we have a chance from a corner, but they’ve gone down the other end.
“It’s one ball over the top.
“Four shots, four goals for them. On Wednesday we were so ruthless in front of goal, and that’s what they (Tadcaster) are like.
“It’s a tough one to take.
“We played really well the other night and won 8-1, and we’ve just got to take the positives from this one and bounce back.”
Harrison admits Tigers can’t afford to slip up many more times if they’re to remain title contenders, but he says the players will take great heart from the support they’ve been shown by the club’s fanbase.
“It’s great support. You would never think we’d lost the game, listening to them, even after the game.
“All the lads, we were so disappointed with how the game finished, we were halfway to the dressing rooms but realised we should go to the fans and clap them, because we do appreciate it.
“That might be the difference towards the end of the season, the encouragement from the supporters, they come home and away. They’re brilliant.”
Harrison is enjoying an improvement in both fitness and form, having come to the club lacking match sharpness.
And he feels right at home with the Sandy Lane outfit.
“I’m enjoying it. The manager has a good philosophy, how he wants to play.
“When you get down to this level you can get pulled into dogfights, but he doesn’t want us to play like that, he wants us to pass it and create chances.
“When I was at Parramore I was in and out of the team.
“The manager had a philosophy that if you missed a game for whatever reason you’d come back on the bench, and if the team were winning you wouldn’t get back in.
“That’s fair enough, they win most of there games.
“I wasn’t playing much so I was lacking match fitness, but as the games are going on now with Worksop I feel a lot more confident, fitter.
“When I first signed I was dead on my feet towards the end of matches, but now I can keep going.”