Hi Ryan -
Use Hyper instead of TDE. It is much faster and compresses the file more efficiently.
Good to know.
What are your thoughts on making each month it's own hyper file and then have a separate flow that combines the hyper files?
In addition to use Hyper instead of TDE, consider the version of Tableau Prep you are using. In version 2019.3 there were performance improvements, some related to unions: New in Tableau Prep: Performance improvements, reuse steps across flows, and usability enhancements | Tableau Software. As mentioned in that article, if you are using wildcard unions (which sound ideal for your use case) you may see a big performance increase:
In regards to creating different Hyper files per month and then union them together, I'm not sure if that'll get you a big performance boost, but you may be onto something. What if you store the resulting hyper file of this month in another folder, let's call it 'Last Month's Hyper'. Your Prep flow can then search for the file in the 'Last Month's Hyper' and union it with next month's new csv file, creating a refreshed Hyper file which you can use for analysis but also use it to replace the old file in 'Last Month's Hyper' folder. In other words, you would be doing an incremental union: historical hyper file + new month's csv file.
Hope this helps!