1 Reply Latest reply on Aug 28, 2012 1:09 PM by Russell Christopher

    Incremental extract refresh and imported data

    Bryan Jones

      I want to make sure I understand what happens with incremental extracts before going down the path of re-building everything so thought I'd toss the topic in here.

       

      We have an extract created (using Tableau Desktop Professional) as follows:

       

      1. A full extract from a Teradata warehouse (say about 100,000 rows) of sales aggregated by day by product type since January 2010

      2. Data added from an Excel file (15,000 rows) for a few of those product types - again aggregated by day - that were sold using a side system and aren't in the Teradata warehouse. That data ranges from January 2010 to February 2012 when they were finally transferred into the main sales system - data for those products beyond February 2010 can be pulled via #1 above.

       

      Everything is fine until we start to do incremental refreshes - we have to push out a daily update. Every time we attempt this, it seems to wipe out the data brought in from #2. The original full extract was through July 30, 2012 (and then the data from #2 was added). When we go to add August 1, the data from the warehouse updates just fine but what happens to #2? The incremental extract/all rows boxes are checked and we're using sale date as the column to check for new rows.

       

      Maybe I'm confused about the word "incremental" - if it is really incremental why would prior data be removed? Or are we just doing something wrong?

        • 1. Re: Incremental extract refresh and imported data
          Russell Christopher

          Bryan -

           

          I just attempted to reproduce this in a simple way, and things seemed to work for me. Can you confirm these steps for me? (I'm using SQL Server instead of Teradata - don't have one of those lying around on my laptop!)

           

          1. In a SQL table, insert a single row with two columns: Date and FieldOne...with values 12/1/2011 and "Record One"
          2. Connect to this table with Tableau, create an incremental extract using the Date column to identify new rows
          3. Use "Add Data From File" against the exact and point at an Excel workbook with a single row of data. Values: 1/1/2012, "Record Two".
          4. A report based on the extract now shows two rows, as expected
          5. Add a second row to the SQL Server table from step 1: Values: 1/2/1012, "Record Three"
          6. In Tableau, perform an Incremental Refresh
          7. The report based on the extact now shows three rows (2 from SQL, 1 from Excel), as expected

           

          Based on what you said, I should be missing my 1/1/2012, "Record Two" row from Excel, right?