Several things to think about:
1. Are you using any context filters? Those will often cause Tableau to create temp tables which can improve overall performance, but you'll get an initial performance hit. (And another hit every time the context changes -- so be careful when using them).
2. Do you have any "relevant value only" filters? They can cause an initial performance hit.
3. Have you checked out the logs to see what queries are being executed on the initial load? That would likely give you some good insight into what's going on.
4. Are you planning to upgrade to Tableau 8? (it's going to production later this month!) First, it has a lot of performance improvements. Second, it has a performance recorder to help you analyze what is happening.