2 Replies Latest reply on Aug 17, 2018 9:52 AM by Scott Carman

    How to create extract with giant dataset

    Scott Carman

      Morning all!

       

      Ok so the current project that I am working is requiring me to create pretty substantial extract.  The database has contract information that goes back in excess of 10 years.  I am thinking that when all is said and done there is going to be in excess of a million rows but I am not totally sure as an extract of this size is new to me and uncharted territory.

       

      I am working in 10.5 and have seen presentations on how to add a blank extract to the server then remove the condition causing it to be blank, letting the server do the heavy lifting.  The problem is that I have tried this and it's failed.  I guess in the update for 10.5 and beyond, Tableau removed this capability.

       

      So I configured my extract as normal in the data source pane.  I tried this connection with just a few records, then 10,000 records, then 100,000 records and it worked every time.

       

      I am not trying to create the "Full Monty" extract.  When I left work yesterday afternoon the dang thing had been running for 4 hours.  Some where in the middle of the night the process stopped.

       

      Just wondering if anyone else has advice on how to do this?

       

      Thanks,

       

      Scott Carman

        • 1. Re: How to create extract with giant dataset
          Monica Van Loon

          Hi Scott,

          It seems like 1 million rows should not be a problem.

          Have you tried hide all unused fields to make it smaller?  And did you see why it failed.  I assume this is a hyper extract and you have Tableau Server?

          If so Tableau server has some stats on failures if you hover over the marks under Background Tasks for Extracts. You can see the errors

          This is 2018.2 Tableau Online,  hopefully screens are similar in 10.5

          You may need to look at Tableau Server Performance too we did a webinar on this

          https://www.senturus.com/resources/revving-tableau-server-performance/

          And there is some good info on Hyper here. At the post below there are even more resources. 

          Too much to read too little time

          Extracts at Hyper speed: What you need to know | Tableau Software

          1 of 1 people found this helpful
          • 2. Re: How to create extract with giant dataset
            Scott Carman

            Monica,

             

            I can't say for sure what is the issue is.  That said part of the issue is that the tables are not aggregated at the same level.  While I was writing a custom SQL statement to help with this issue, it just won't work.

             

            That said I had a conversation with my boss and we are going to build one view from one table as a single source.  The other view from the other table as a single source.  Both views will have a common hierarchy that will allow us to filter.  While this does reduce some of Tableau's functionality at this point, hopefully in the future, I can have the DBA go back and create a custom view for me that will accomplish what I was originally trying to do.

             

            Tableau is good and wonderful but poor data structure is hard to work around.

             

            Thanks,

             

            Scott