2 Replies Latest reply on Aug 21, 2012 2:05 PM by Erik Ford

    Incremental extract focusing on Data Deltas

    Erik Ford

      Our data set updates daily, where the last days data is added incrementally, but also, the last 30 days worth of data can potentially change due to rolling 30 day conversion windows. Is there a way to set an incremental extract to not only pull in the latest day's data but also Overwrite any value changes that it comes across in the rolling 30 day set as well?


      For example, new day starts, additional rows of daily data come in, but also conversion data and metrics change for last 30 days of data within the data set.


      As it appears there may not be a direct way to do this, I've proposed this as an idea to tableau. Please vote it up here to see it come to life!

        • 1. Re: Incremental extract focusing on Data Deltas
          Russell Christopher

          Hey Eric -


          I'm afraid there isn't a good solution for this scenario. We don't support overwriting existing rows during an incremental refresh.


          I can see a couple ways you might approach this:


          • Do a full refresh. Yes, it'll take (much) longer, but your "old" rows will get thrown out.
          • Figure out a way to write "delta" rows into your data source. For example: I'm doing a SUM([Sales]) by Product ID. On 8/19, you insert $10 of sales for Product 1. On 8/20, you get a corrected number of $9....Write a new row for $-1 on 8/20 - When we SUM these two rows together in your report, the answer will be $9.


          There are probably one or two other ways (data blending?) you could try to attack this, but none of them are really clean.

          1 of 1 people found this helpful
          • 2. Re: Incremental extract focusing on Data Deltas
            Erik Ford

            Thanks Russell, appreciate those callouts. The calculations/ views of the data would make it far to complex to try to do delta records and sum them in our instance. I'll stick to a full refresh, and I've also suggested it as an idea and would greatly appreciate the support / voting up of the solution here. Also if you think there's anything I should add or call out in that idea, please feel free to add to it!




            1 of 1 people found this helpful