4 Replies Latest reply on May 13, 2016 9:05 AM by Mahfooj Khan

    First extraction - Loading a huge amount of past data

    Hannah Williams

      Hi Everyone

       

      I am creating an extract on a table with a huge amount of past data, so creating the initial extract is going to take ages to run.

       

      I am wondering if there is a more efficient, or less impacting way to do this initial extraction. Any ideas?

       

       

      Thanks

       

      Hannah

        • 1. Re: First extraction - Loading a huge amount of past data
          Alastair Young

          Hannah,

           

          Are you running the first extract so you can incrementally update this in the future?

           

          The speed of building an extract is determined by a number of factors and not just the size of the data as a whole. Tableau will optimise the extract and materialise any calculations to ensure future refreshes are quicker and more efficient.

           

          Tableau is not a database and is only as fast as the resources you provide it and the database it connects too.

           

          Generally, Hiding unused fields, Optimising the extract, and optimising the database are the key ones that have the most impact if you are not running incremental refreshes.

           

          One other option is to aggregate the data if this is possible.

           

          Regards

          Alastair

          • 2. Re: First extraction - Loading a huge amount of past data
            Hannah Williams

            Yes it will incrementally update moving forward, there's no issue here. The initial extraction in this case is simply huge due to past data.

             

            I was wondering if there was a way that I could somehow load the first X rows when I create the datasource in tab desktop, then let the first incremental refresh on server pull everything in.

            • 3. Re: First extraction - Loading a huge amount of past data
              Alastair Young

              Hannah, this is possible,

               

              However it would require some kind of calculation to be applied to the dataset. For example - create a calculated field using the following

               

              If NOW() > DATETIME("May 13, 2016 17:01:08") then 'true' else 'false' END

               

              Have this a few minutes in the future, put this on as a data source filter = true.

               

              Run the extract, the extract should then be blank.  Post this up to Server, And run an incremental refresh or FULL refresh after the time stated.

               

               

              This should achieve what you are after.

               

              Alastair

              • 4. Re: First extraction - Loading a huge amount of past data
                Mahfooj Khan

                Hi,

                You can do one thing.

                First at database level create a view of your table where put condition on rownum<100.

                Then connect that view in tableau. Instead of live go for extract data connection. Save the .tde file locally and publish that .tde in server from data menu Data>your View> Publish to Server. After publishing your tde will have only 99 records in your extract.

                Once you publish the metadata then go to your database alter the view just remove the where condition and create and compile the view once again. Now Login to your tableau web server create a schedule for full refresh and run the job. Let finish the job. Once done go to your tableau desktop and try. I hope in this way you will be able to extract all the data in tableau engine.

                 

                I've already given this solution in below thread and it worked.

                Tableau space error

                Let me know if you've any query.

                 

                Mahfooj