I think you already have all the pieces in place in order to accomplish the desired filtering. All we need to do is have only the [Date toggle] field on the Filters shelf, then expose the Quick Filter for that calculated field.
In the screenshot below I also displayed the worksheet title which will update to show which date value the view is being filtered on. When the parameter value is changed, both the Title and Quick Filter values will change appropriately.
Hope this helps!
Thank you! Good to know that I was close to the solution at least!