0 Replies Latest reply on Nov 5, 2018 3:05 PM by Mark Dunlap

    TDE Size Discrepancies

    Mark Dunlap

      Hi everyone,


      I could use some help understanding if there are other steps I can take to optimize a TDE.  I'm sourcing data via a connection to Cloudera Hadoop.


      The problem I'm facing is that I'm using a smaller set of data and the same extract creation process as a colleague, but my TDEs are coming out about 8x the size of hers.


      My data:

      75 M records

      31 columns

      Data size = 19.5GB

      Format = Text

      TDE size = ~3.9GB


      Colleague's data:

      56 M records

      48 columns

      Data size = 26.7GB

      Format = Text

      TDE size = ~475MB


      We're both connecting directly to HDFS directories using Cloudera Hadoop.  Even when I've attempted to optimize using ORC and Parquet file types, I can't shrink mine below 2GB.


      Currently using Tableau 10.4.


      Have attached some screenshots of the table analyses and field types, in the event it helps.  Was wondering if it was that I have "Double" types, but all the fields in the TDE seem to show up as Double for both of us.


      If anyone has better knowledge of how TDEs are compressed and if any of my data types are causing problems (or other advice), I'd really appreciate it!


      Currently using v10.4, but we should be getting upgraded to 2018.2 at some point in the next few weeks.