A few questions to start ....
Tableau Server or Tableau Online ?
Live connection to Excel or Extracts ?
Are the Excel files really 2-4GB ?
1.we are using tableau server
2.They were extracts not an live connection
3.Each file size will cross 1gb after extracting an zip file
How many extracts and what sizes are they ? (each & in total ?)
It sounds like you have several large extracts and if the network to the server is a bit slow then it will take a while to do.