Unfortunately I do not have a solution to this problem either but I have the same issue.
Therefore, I notice that my Set Actions didn't work with "date" as well, I didn't have any problem with string.
I opened a case with Tableau customer support, I will share on this thread if I learn anything.
Alright, I found a solution for my case which might help you.
It seems that with your date filters, you might have to put everyone on discreet or everyone in continuous in the sheet which is the source of your Set action...
As soon as I did this, my problem disappeared
Not great because there are some visuals which can only appear with one or another... but it does the job until someone finds something more elegant.
Thank you for your suggestions.
Yesterday I figure out why it's happened.
It is because I directly create set from [DATE]--date and when I do customer date first and it becomes number.
Then it works.
And I think the key is set can be string, number but no a date.