Have you made progress improving the performance? You don't mention which version of Tableau Server is in use with this question. Please let us know the version of Tableau Server in use and consider updating to the most recent version to verify there is not a a performance increase with a more recent version to assist (especially if this is pre-hyper).
Did you get thru to this one? we are facing a similar behavior in Tableau 10.5.3 Server. Unable to boil down to the exact root cause. Which version are you on?