1 Reply Latest reply on Oct 2, 2018 11:02 PM by Sankarmagesh Rajan

    Not All Data Being Refreshed In Incremental Refresh - I Suspect It Has To Do With Date Truncate

    Jeffrey Sigler

      Hi All,

       

      So I set my data to refresh based on timestamp (day, hour, minute, second, millisecond) every 15 minutes.  The table has about 1 billion rows, so in order to cut down the work for Tableau and to not slow down the system, I date_trunc'd the timestamp to the 'minute'. (date_trunc('minute',timestamp) as a Custom Query at the data source.

       

      What is happening is that the incremental refresh is just looking at timestamp, seeing if it is a new minute or not, and updating for every minute.  In the process, it is missing some entries that happened in the minute previous and the current minute when it is being refreshed .

       

      So for example, since I have my refresh at 15 minute intervals, starting at 3:00 pm, every 15th minute is missing almost all the entries, along with some from the 14th minute.  I suspect it is due to how Tableau is looking at "new" entries.  Is there any way to overcome this without having to just use and pull in all 1 billion rows?  Thanks so much