Adding a little more information as I come across stuff, just in case it'll help someone else.
Our environment is setup with a single primary and two workers. Each worker has 32g of ram but seems to now be struggle when trying to refresh over 500 extracts on a Sunday. Some of these extracts can take a long time to run and return a fair amount of information/records. At a glance, it seems that some of the extracts that are taking a long time to return their data are starting to 'clog' up the memory available and not release it soon enough before another extract starts on one of the backgrounder processes. We are going to start staggering our extract refreshes throughout the day on Sunday and also look at upping the memory if we have it available.