Vikash- You're right, unless you have calculations on the view Tableau won't process it. So keeping unused parameters/calcs wouldn't impact performance. However, there are plenty of other things to look for, a good place to start is "Start Performance Recording" to see what's impacting the performance significantly. From there you could start your investigation. Here are some useful links,
Hope this helps. Good luck!