I thought the building I was thinking of was visible in Grain Street, near Mill Street end. I accept you are correct...I must admit I didn't pay too much attention to old buildings or their precise locations, when I was younger. I think I'll have to invest in some maps.
I passed the area in the 1970s when there was some work going on (between Harlow Street and Grain Street). I used to have to walk in the road to get past the work that was going on. I must be confusing the two.