I need a little help with understanding the RAM usage of Tableau Public when getting data using the WDC. I can successfully connect to my data source and bring records in when I have say 10k rows. However, when I change the dataset to have 100k records the WDC connector takes up all of the RAM on my laptop. I tried on a machine with 32GB RAM and it eventually would use it all if it didn't fail first. The dataset retrieves relatively fast as JSON and I perform a bit of a pivot on it to put it in a clean array and then append all records to the table.
Anyone have any ideas why its using this much RAM? Would it be better to page the response and append the records after each call vs trying to pull them all in and append all at once?
The pivot sounds like a problem since that may be causing the data to be loaded in memory. Could you expand a bit on what is happening with the pivot?