The method I typically use is to duplicate the field you are using for the quick filter. Then, place the duplicate on the Filters shelf and choose to Exclude the values you don't want showing (do not display this filter in the view). Then, place the original field name on the Filters shelf, set it to "Only Relevant Values", and display it in the view -- the options you excluded from the duplicate won't be displayed in the quick filter you display in the view.
More info here, if you need some more clarification: Custom Filter with Excluded Values
There are other options, but this one is quick and easy.