4 Replies Latest reply on Oct 10, 2015 5:50 AM by Jeff Strauss

    Extract fetch size

    Jeff Strauss

      Our ultimate goal is to improve the throughput speed of extract refreshes.  So I've been doing some digging and found the below stuff. Has anybody successfully adjusted the buffer settings?  I have not yet, but was thinking about at least testing it.

       

       

      1. Within a few of the tdeserver logs, there are many lines that indicate 1 meg chunks of data being imported

          2015-10-07 01:46:54.907 (16208): Session288: UploadSend:  upload_guid=1 num_bytes=1048576

       

      2. When doing some searches, I see that the following settings within httpd.conf

      ProxyIOBufferSize 1048576

      ProxyReceiveBufferSize 1048576

        • 1. Re: Extract fetch size
          John Kuo

          Jeff - This is interesting...do post your findings if you plan to test it

          • 2. Re: Extract fetch size
            Jeff Strauss

            I tested it yesterday and found no positive effect to changing the httpd.conf settings, I also tested changing a setting in tabsvc.xml that had a buffer size of 1024, but this too had no positive effect.  My theory continues that there is a buffer setting somewhere (perhaps internal within tdeserver64.exe) that can take better advantage of fetching the data and using more of the server memory (256 gig) to speed up the import / processing.

             

            I do plan to  keep looking a bit more, my next step is to modify my python script that uses the dataextract api to do a fetchmany instead of fetchone to see what happens when I up the buffer size here, I just need to figure out how to do this as I am trying to learn python at the same time as applying it.  do you know python by chance?

            • 3. Re: Extract fetch size
              Matt Coles

              Are these extracts based in workbooks, or datasources? If they're workbooks, make sure that the unused fields are hidden. If authors don't know to use the "hide unused fields" feature, the extracts include unnecessary data and thus, cause refreshes to take longer.

              • 4. Re: Extract fetch size
                Jeff Strauss

                these are datasource extracts.