Are you able to post the workbook?
Is the quick filter set to Only Relevant Values or All Values in Database?
I can't with that particular workbook. But here is one, it is an example of Year over Year filtering that I believe Joe Mako published. It has the exact same thing happening. To see it in action change the selection on the Select Date Level parameter.
Currently, there is no way to have the default always go to All when changing the view based on a parameter. You could instruct the user to always click All before changing the parameter--though that's not ideal. Might make a good one for the Ideas section!