It is very probable that the size of your Excel file is not the problem here. I personally deal with a dataset with 13M rows and it can provide me results in 2-3s. Having several filters can create problems in special if you select "Only Relevant Values" as option for your filters.
I suggest you to record the performance of your workbook to find the bottlenecks.
There you will find lots of valuable information.
Along with that, please take a read in the Tableau Best Practices for Designing Efficient Workbooks.
I am sure with these two resources you will be able to drill down the causes of the slowness and take actions to improve the performance.
I hope these helps
Thanks for getting in touch. I did try to record the performance and I found it is the computing layout which takes a long time. From the links that you have shared it says simplify your workbook when the computing layout takes a long time, but I am not sure how to. As per the requirement I have, the end user wants as many data panes. so I am pretty much close to the limit i.e 16. I have tried to remove a couple of filters and I still face the same issue. I don't see a problem when I try to search something using the filter. It loads up pretty quickly but the problem is really when I clear the filter and that it when the load is very slow as it is trying to repopulate all the data.
Is there anyway you could share the workbook so we can take a better look?