6 Replies Latest reply on Feb 25, 2015 5:59 AM by michael.galouzis

    TDE file size for appended data using data extract API

    michael.galouzis

      I have written a python script that runs once a day and gets statistics from a website from the day before. Using the DataExtract API those data are appended to the data extract table every time the script runs.

      Everything works fine, I am able to open the tde and all the data are there, but the "problem" is that the tde-file is getting extremely big after a few days.

      For a tde that has run for two weeks (14 times) and contains about 1.500 records (not much at all) the size of the tde-file is 3.323 KB. When I create an extract with the exact same data using a "full history load" (i.e. loading all the data before creating the extract, so no appending) the size of the tde file is 541 KB.

       

      So by appending the new data every day, my tde is 6 times bigger within 14 days (and this with only 1.500 records).

       

      Furthermore, when the tde-file (3.323 KB) is opened in Tableau and I then choose to "Extract Data" the new tde-file is again 540 KB.

       

      My question is now:

      Why is the tde-file getting so much bigger when appending and is a there a way to "optimise" the tde so it has the same size as when the tde is created with all the data in one go?

       

      Message was edited by: Michael Galouzis Spelling error in the title.