4 Replies Latest reply on May 22, 2015 8:02 AM by Toby Erkson

    Data Extract API Performance

    Salman Bashir

      Hi there,

       

      I am using Data Extrect API with Java and trying to create an Extract with about 350 columns and 7500 rows but facing following issues:

       

      1- While inserting rows in Extract table, it eats up all the system memory, so I have to call system.gc to free up the ram after every few inserts (like 200 rows)

      2- When all is done and I call Close method on extract object, it takes too much time in freeing up the resources.

       

      Any help regarding tuning up the Data Extract API will be appreciated.

       

      regards,

      Salman

        • 1. Re: Data Extract API Performance
          Toby Erkson

          350 columns?  That sounds like a problem right there.

           

          How much RAM is on the machine you're using?

           

          In the meantime, close every application that you absolutely don't need open.  I often help users who never seem to close anything they open (and save every file on their Desktop ) and all those open applications take up space and use resources.

          • 2. Re: Data Extract API Performance
            Tom W

            Toby Erkson wrote:

             

            350 columns?  That sounds like a problem right there.

             

            Do you really need 350 columns? Are you using them all? Can you split into multiple tables / facts? Or have you considered a 'deep' vertical data structure instead of the wide horizontal?

            • 3. Re: Data Extract API Performance
              Salman Bashir

              Dear Toby,

               

              Thank you for such a quick reply. I agree that # of columns in my case is huge but unfortunately it is necessary, I can reduce it by 50 once I clean up my script but it will still remain a lot. I have 8 gig RAM installed on my machine and almost 1/4th of it is consumed by other applications. In the same situation, if I create same extract using tableau desktop, it only takes less than half amount of time than what TDE API is consuming.

               

              You caught me there , I am the one who keeps everything on Desktop, however, I don't open any unnecessary application, especially when I am going to perform any such task.

               

              Can I find something to understand what exactly the "Close" method of "Extract" object does and if it could be optimized somehow?

               

              Thank you once again,

              Salman

              • 4. Re: Data Extract API Performance
                Toby Erkson

                Salman,

                The only thing I can think of is that you need more RAM, however, if you are watching your Task Manager and you're NOT maxing out your RAM then I really don't know what's going on