I'm looking for a way to have an extract do an incremental refresh if a value changes in the results, not just rows.
Basically, my extract is a custom SQL which Counts records with the same transaction_date.
SELECT transaction_date, count(*) as my_count
GROUP BY transaction_date
It seems, when the incremental refresh runs Daily at 3:00am, I have a new 'row' for that day with 5 records. However, by the end of the day, I should have 10+ records for that row.
Since the 'rows' have not increased, the incremental refresh is missing updates to my count.
My results look like this;
When they should look like this:
There are actually billions of records in my database each day, so a Full Refresh takes a very long time.
The data is also a rolling 30 days, so if I do Full Refresh, I lose anything from the extract older than 30 days, which I prefer to keep the counts in tableau.
I have the same question but would like to add to it. Can the refresh also delete records if they are deleted in the source data?