2 Replies Latest reply on Mar 5, 2015 1:08 AM by Shashank Gaurav

    Data Extract API in C# app - performance issue

    Srinivas Nallamothu

      Hi,

       

      I am using data extract api in our C# based application to generate .tde files.  It exports the data from SQl server table to the .tde file.

      I am able to generate the files with no issues. We are seeing few performance issues when exporting data. Our aplication  is taking 8 hours to export the sql server table which is having 1 million records.

       

      Below are the steps I am following to export the data to tde file.

       

      1. Generate the .tde file using api

      2. Connect to SQL server and get the table schema and construct a tableau table

      3. Then add the tableau table to the .tde file (using TabExtractAddTable method in the api)

      4. Then read the data from SQL server

      5. Add the data to the tableau table row by row(adding row by using 'TabTableInsert' method) ( looping 1 milloin times to add 1 million rows to the tableau file)

       

      Is there a way to export bulk data using tableau api?   Can someone plese help me how increase the performance of the application which is using data extract api ?

        • 1. Re: Data Extract API in C# app - performance issue
          Russell Christopher

          Hi Srinivas -

           

          How many columns are you exporting? Keep in mind that we are column-based and that row count isn't quite as relevant.

           

          I just ran a quick test with Python creating a 1M row extract with 25 columns and it took me just under 1.5 minutes. Most of that time was not the API doing it's work, but my code generating random values for fields like "sales",  "profit", and quantity to stick in the extract. 

           

           

          A few months ago I also  generated a 1+ billion row extract (just a couple fields) and it took about 4.5 hours. I just tried it again with 1M rows, and it took 10.2 seconds.

           

          You said you're writing your code in C# - Are you sure the (non-supported) C# wrapper itself is not the problem as it marshals calls back and forth? In your shoes I'd want to rule the C# wrapper out as the cause of the problem before I did anything else.

           

          I'd recommend you dump a bunch of your SQL rows to CSV and then use the "csv2tde" sample python code to see how long it takes to create an extract out of them. That will give you a feel for the potential speed of the API minus outside factors like the amount of time it takes to get your SQL Server rows local and the C# wrapper marshaling calls.


           


          • 2. Re: Data Extract API in C# app - performance issue
            Shashank Gaurav

            Hi Srinivas,

            I've to create .tde files using c# .net as required by our stakeholder.

            I'm using the same C# wrapper and MakeOrder client to generate .tde files. However, I'm receiving exception "Invalid Extract handle" at line "Table table = extract.AddTable( "Extract", tableDef );".

             

            After going through this topic, we would like to know, How you implemented this part (Generation of .tde files) before running into performance issue.

             

            Please revert for clarifications,

             

            Regards

            Shashank