I'm a fairly new Tableau user (most of our current analysts use it, but as a developer it's fairly new territory for me), but I've run into some issues that I was hoping someone had the answers to. Currently we have massive datasets sitting in file format on various layers of our application and I'm trying to get that connected to Tableau for Visualizations. I've also created a custom Elasticsearch-Tableau WDC which seems to work pretty well for datasets less than 10K records, but anything over that takes an abhorrently long time to process. Is there a size limitation for data extracts using the WDC?
1.) What is the maximum amount of records that a WDC supports? Say for example I had a datastore (ES) with an Index/Type with about 10 million records of 20 fields per record. Is this a number that's way out of the league of the Tableau WDC's scope for creating a TDE?
2.) Using Elasticsearch and the Scroll API's, I'm able to get fairly responsive rest calls for batched requests of around 5000 - 10000 records per call, but when I debug through my custom Tableau WDC using the Simulator (which is very helpful, thanks btw), it seems like the rest calls return fairly quckly, but after I do the tableau.datacallback it takes about 20-30seconds before another rest call is made (and this gets progressively longer as more queries are sent out). Is this the expected behavior?
Does anyone have tips as to what the best approach would be to visualize such a large dataset (preferably from Elasticsearch); I'm willing to make concessions and use other data stores, but would like to keep that as for a last resort type situation.
---Additionally, any chance we see a native Tableau Connector for Elasticsearch anytime soon? Would be nice to be able to do a live connection to Elasticsearch and have Tableau execute ES Queries to return data back. (couldn't find any sources stating if this was going to be done anytime soon, or if it's in the works?)
-Using Tableau 9.3 (wdc 1.1.1)
-Using Elasticsearch 2.2.x+
-Dataset consists of millions of records (< 10 million), 20 columns across.