4 Replies Latest reply on Jan 22, 2013 5:40 PM by Budy Sugianto

    Re-publishing datasource metadata without overwriting incremental extracted tde history

    Craig Dewar

      I created a SQL query datasource - that connects to a live logfile with one million+ rows.

      The initial extract took some time to download to my PC, and the tde file was 74MB or so.

      I then published the datasource to server, setting up an incremental extract update to the source logfile data, which occurs weekly (on weekends so that I don't impact system users with database performance issues)

      The weekly incremental updates take a very long time, the last one took over 90 minutes to insert around 27000 rows - (which sounds pretty ordinary to me as Tableau Server lives in the same datacenter as my source database!)

       

      While using the datasource to do some log file analysis - I found I needed to tweak a calculation, and as I could only 'Edit Copy', went back to the MASTER twb - from where I published the original datasource from, fixed the tableau calculation, and re-published DataSource.

       

      When I refreshed my log file analysis to update the calculation, TO MY HORROR, the dataset went back to my original 23-Jun data, and wiped out all of the incremental updates that have occurred since.  I have realised (too late) that each time you publish a datasource (which has an extract), the local TDE overwrites the Server TDE. 

       

      I am now faced with waiting for next weekends incremental update, which will probably take several hours to process! - or refreshing my local PC tde - which will probably impact system users.

       

      I only have a datasource with one million rows - this must be a showstopper for customers with really massive datasets.  Surely there is a better way of editing metadata assocated with a extracted datasource, without having to reset all the data back to a local TDE on a users harddrive.  Am I missing something here?  Is there a better way to manage and re-publish only the metadata to large datasources that already exists on Tableau Server?

       

      Please help

       

      cheers

       

      Craig

        • 1. Re: Re-publishing datasource metadata without overwriting incremental extracted tde history
          Vusi Nkomo

          Inserting the number of records that you are inserting should not take much time, unless you have a very large number of columns. We have incremental refreshes that add more data than that in seconds. You will need to check what other services are running at the same time and if you are maxing out your CPU etc when the refreshes are running in order to pick up what the issue could be.

           

          On the wiping out of data, to the best of my knowledge, overwriting a tde with incremental extracts is the equivalent of a performing a full refresh, and you will therefore lose the prior data. If you want to keep the data, then save a copy of the server tde locally before republishing. You can then use it as a separate data source while the other one re-builds. Depending on the data size, you could even export the data and then use your "Add data from file" option. There may be a better and more elegant way, but this is what i think may work for you. Also, remember to back up your server data sources often...

          • 2. Re: Re-publishing datasource metadata without overwriting incremental extracted tde history
            thierryjakircevic

            I have a similar question: I work with extracts of a Tableau Data Server (TDS) connection. When I change the metadata (alias mostly) or create new calculated field, I'd like to publish these changes to the server and overwrite the TDS so that it can be reflected to all workbooks using the same connection.

             

            However, the option "Publish to Server" is not available from my extracts. I need to create a local copy then publish that copy to the server. Then close it and refresh my extract.

             

            Isn't it an easier way?

            1 of 1 people found this helpful
            • 3. Re: Re-publishing datasource metadata without overwriting incremental extracted tde history
              Craig Dewar

              Thanks for the tip Ndabezinhle

               

              Have checked with our DBA, and it turns out that one of the joins was using a foreign key with no index.

               

              After implementing the fix, the same incremental extract now takes 30s to insert 10000 rows.

               

              However, the core problem still remains.  Every time I want to republish the datasource with changes to metadata, as I have no access to the file system on the Tableau Server,  I have to reset back my last manual TDE extract on my hard disk, and re-upload 74MB to server, and kick-off an incremental update from 23-Jun to current date.  Not a very elegant solution.

              1 of 1 people found this helpful
              • 4. Re: Re-publishing datasource metadata without overwriting incremental extracted tde history
                Budy Sugianto

                I think this use case is very common and Tableau should have a solution: Republishing workbook without overwrites existing the server TDE.