2 Replies Latest reply on Jul 10, 2017 11:10 AM by Russell Christopher

    Best way to handle daily huge volume of data? Say, Daily 1M rows

    Bodhisattva Dasgupta



      What is the best way to handle a daily flow of huge volume of data into Tableau?


      Say, for instance i have client account data in a Table, which is going to be about 500K-1M rows per day.


      In Tableau, I will have 3-4 dashboards, couple of them at a summarized level, and couple at detail level with filters available for filtering.


      My questions are...

      - what would be the best way to manage this data? The database would be Hadoop, with an Impala connector.

      - Is it okay to just do a daily incremental refresh into Tableau? If so, how to handle cases where historical data gets updated? (might happen a handful of times in the year)

      - would using another tool in front of Tableau, like Alteryx, help?

      - is there any way in which we can segregate out a data source into a moving 1 month data that always FULL refreshes + another that INCREMENTALLY refreshes daily? And UNION them?


      Any help/thoughts in this regard will be appreciated.