4 Replies Latest reply on Feb 15, 2013 5:06 AM by Russell Christopher

    Can I initially build a TDE from a database source incrementally, and then when all historical data is loaded, allow normal incremental updates to occur?

    david.millrod

      The data source is a large mySQL table that is partitioned by time for performance.  Querying all partitions at once takes much too long.

      The goal is to seed a Tableau Data Extract with all historical data, and then allow the standard incremental refresh option to update it daily.

      The issue is that the very first db query would run too long if it tried to get all historical data at once.  Ideally we would use a where clause to limit it to one monthly partition at a time, thereby incrementally building up the TDE until it has all historical data before allowing it to update daily.

       

      In Tableau v7 I tried to do this by using Custom SQL to just point at the oldest historical monthly partition and setting the TDE to incrementally update by date.  My plan was to then edit the Custom SQL to expand the where clause to include the next historical month and then click on "Refresh (Incremental)".  The problem is that Tableau noticed I changed the SQL and insisted on regenerating the whole TDE from scratch.

       

      Here are the very imperfect alternatives I've thought of - any suggestions?

       

      In v7:

      - Point Tableau at a database view instead and edit the view multiple times to reveal more and more historical data until all historical data has been incrementally loaded into the TDE.

      - Edit the .tds file by hand to trick Tableau into not purging the TDE even though the SQL has been changed

      - Somehow use the "Add data from File..." option to load the historical data, but I'm not sure how to create the data source with the ongoing query defined, but prevent it from immediately running so that I can first add the data from file.

       

      Waiting for v8:

      - Possibly do something with data source filters on date

      - Possibly do something with the new TDE API

       

      Thanks