When do you encounter the error?
Prep has 2 modes of execution:
1. Authoring: When you are building out the flow, you are usually processing a sample of data from your input datasources. This data is ingested into Hyper and transformation results are processed against that so your see immediate results in the profiles.
2. Batch: When you run the flow to produce an output datasource, we do what you described in 1 - consolidate all the transformations into what seems like a large query that we push down to your source DBs if that's the most efficient execution policy determined by Tableau's federation engine.
If you see the out of memory error in Authoring mode, I'd recommend limiting the data in the sample (in the Input Step you can adjust Sample settings) and give it another try. If you see an error in Batch mode, I'd recommend filing a Support case (Support Case | Tableau Software ) for further investigation/trouble-shooting.
Senior Product Manager, Tableau Prep
Thanks for replying.
We encountered this error during Batch execution. I'll file a support case for further troubleshooting.