Skip navigation

Allow Incremental Refresh to Modify Rows

score 361
You have not voted. Active

Currently the incremental refresh adds new rows but does not alter rows that have changed since the last update. This doesn't work well for many situations where a full refresh is expensive but some of the most recent rows may be altered. I would like to see an option for refreshing altered rows in an incremental extract if there is a Date Modified-type field in the data source.

An example use case is a table in Salesforce or some other cloud-based system that has a million records with some new additions and a small number of changes to existing records each day. You can't do a full refresh very often because of API call limits, but an incremental refresh would leave a number of records outdated. This could make the incremental refresh effectively useless since the data isn't all current. A refresh done as above would solve the issue.

Comments

Vote history