What I understand is that you are actually clicking on the Label on the Bar. If you check on the sheet Top list and you click on the bar at the right of the label you get the correct behavior. If you click on the label then you get the arrows allowing you to move the label. And when you click on the label you can also see a tiny arrow appearing on the top center of the label. If you click on it then you will clear the selection. I know, it doesn't completely resolve your problem, but I don't know if there is a better way to resolve this.
I though about maybe creating a dual axis with a transparent bar without any label. By moving this bar to front, when clicking on the bar it highlights the front bar, but with the effect of hiding the label on the bar at the back (
Huh. I see it now. I have added a blank line both above and below the text of the label in the label editor. (Those are remnants from when there was more stuff being displayed in the label.) And that makes the label completely occupy the space where the bar resides. When I remove the blank lines, I have some "unlabeled" bar area where I can now click and de-select the mark.