I moved this to the Server Admin forum because it's more about what goes on in the server environment. I also don't think a workbook is necessary as this is more of a thought exercise and could be reproduced from any one of us if necessary.
Thanks for testing the workbook on the actual Tableau Server, that helps. I don't necessarily know what could be happening but in order to help, what are your server specs? OS, hardware, Tableau version, proxy/firewalls, etc. Details about your server environment would be helpful.
Here is the info.
Tableau Server Version: 10.5.0 (10500.18.0109.1050)
Tableau Desktop Version: 10.5.0 (10500.17.1226.1925)
Windows Server 2012 64-bit Windows
VMWare version 10
RAM 32 GB