I've attached a workbook that shows a possible way of solving these types of problems. The basic idea is to use a table calculation as the filter. The reason this works is that the average calculation is done at the data source, but the table calculation filter is performed after the data is returned to the data engine. So the average reflects everything, even if they are subsequently filtered.
Please take a look and let me know if you have any questions. Also feel free to upload a packaged workbook of your own if you'd like someone to look at your specific case.