So I set my data to refresh based on timestamp (day, hour, minute, second, millisecond) every 15 minutes. The table has about 1 billion rows, so in order to cut down the work for Tableau and to not slow down the system, I date_trunc'd the timestamp to the 'minute'. (date_trunc('minute',timestamp) as a Custom Query at the data source.
What is happening is that the incremental refresh is just looking at timestamp, seeing if it is a new minute or not, and updating for every minute. In the process, it is missing some entries that happened in the minute previous and the current minute when it is being refreshed .
So for example, since I have my refresh at 15 minute intervals, starting at 3:00 pm, every 15th minute is missing almost all the entries, along with some from the 14th minute. I suspect it is due to how Tableau is looking at "new" entries. Is there any way to overcome this without having to just use and pull in all 1 billion rows? Thanks so much
incremental refresh will bring new records only and it will not bring older update and delete things here. please check the db with any modification happens or not. if your db getting any update or delete then go with full load.