3 Replies Latest reply on Nov 6, 2013 11:40 AM by Jean-Christophe Rodrigue

    Combine extracts from same data source for 2+ periods

    Lloyd Bloom

      I typically analyze a single week's worth of data from a Cloudera Hive datastore, about 2.5M records.  So, I end up identically-structured extracts named for each week.  I also have the need to analyze results of a month's worth of data and would prefer to combine or merge four weekly extracts rather than re-query to create another one, as the query could take many hours.

       

      I thought I was shown how to do this during the Tableau sales process, but can't find the documentation or discover the solution myself.  I have tried Extract/Add Data from File, but this gives me a file format error when I point it to another .tde file.

       

      Any clues?

       

      Thanks,

      lloyd

        • 1. Re: Combine extracts from same data source for 2+ periods
          Dimitri.B

          I believe this functionality is for adding data from files like .csv, .txt, Excel, etc.. I don't think you can add from .tde, because it is compressed and modified for size and speed.

          If you don't have your weekly flat files, then you can create them from your extracts by viewing extract's data and then copying it into Excel (if there are less than 1M rows), or using more tedious way of building a view that just shows all the data and exporting it into Access database.

          • 2. Re: Combine extracts from same data source for 2+ periods
            Lloyd Bloom

            Hi Dimitri,

             

            Your solution worked for a sample data set of 300K+ rows, so I thank you for that.  My problem is that each week of my data set is 2.5 million rows and growing.  Seems like I will need to requery for the larger, combined datasets at about a half-hour per week of data.  Not the end of the world.  In fact, the copy/paste from a data view may not be much if any faster.  I'll have to grin and bear it, I suppose.

             

            Thanks for the pointers.

            • 3. Re: Combine extracts from same data source for 2+ periods
              Jean-Christophe Rodrigue

              I am in the same painful boat as Lloyd Bloom. I have different extracts and when trying to combine them using the "add data from file" and pointing to the tableau extract it fails. Tableau seems to process the file but tells me in the end that zero rows were added. Yet if I open the extract independently I see that the data is there.

               

              I have several Millions rows to append and exporting to text file sounds like a very tedious and dissapointing option.

               

              has there been any progress made on this issue since then?