In your scenario would it work to use parameters instead of filters? For example you could create 1 Red Parameter and 1 Blue Parameter. Then create a calculated field instead that takes into account the selection in those parameters. This would ensure you only can choose 1 or All of each color.
I think I see what you're getting at. I'm not sure that'll work for me though. I'm trying to create a deck building tool for a game in Tableau. So that'll mean a lot of parameters that I'd have to manually create.
I'm going to take a slightly different approach that's not quite ideal, but will still work fine.