Let's start with basics - what version of Desktop and Server are you using?
Also, you just mixed together question about different filters and question about live vs published - which is it? Or both? Performance differences are most likely coming from live vs published differences (there are few and most do affect performance).
Without seeing concrete example (ie workbook) that is "fast" and example (i.e. workbook and published datasource it uses) it will be hard to tell exactly what's happening in your case.