You should try to use DATEADD as [Date]-1 doesn't always work as expected. Try DATEADD('day',-1,[Date]) instead.
Thanks for the answer. I put that in and it doesn't seem to make a difference. I think the issue is that it's not actually operating as a filter for the count, just for the table as a whole. What I mean is that if my first date on the table is 6/27, and select 6/28 as "today" (which would make 6/27 "yesterday"), it returns the same number for each count. If I select 6/27 as "today", it shows "today's" count, but "yesterday's" shows as zero. So it's filtering correctly for the date, just not for the count. I tried adding my "yesterday" filter to the filter's shelf, but that doesn't make a difference.
Can you share a sample workbook and a mock-up of what you want it to look like? I think a solution for this is going to depend heavily on what type of viz you're attempting to create.