3 Replies Latest reply on May 6, 2013 9:11 AM by Cristian Vasile

    How do I create a tde from large txt files

    Brian McClain
    Brian McClain May 3, 2013 3:32 PM

      I have two TXT files (7.5 GB and 1.6 GB) that I'd like to join and create a TDE but it keeps failing due to lack of memory.  I have 12 GB RAM on my machine so I don't know why it's an issue.  Anyone else having this problem or have tips on how to get it done?

      • 233 Views
      • Tags:
        • 1. Re: How do I create a tde from large txt files
          Cristian Vasile
          Cristian Vasile May 3, 2013 3:47 PM (in response to Brian McClain)

          Brian,

           

          You will never be able to import a text file (i suppose is .csv .tsv or something similar) larger than 4Gbytes, due to Microsoft JET Engine constrains.

           

          I 'll suggest you a two stages approach

          Stage 1

          a. split the biggest file in 2 or maybe 3 to be safe small files.

          b. import text files in a ms access database; after import do some minimal checks.

           

          Stage 2

          a. import ms access table or tables in Tableau

           

          Be aware that the time frame for entire operations could take a couple of hours - highly dependable of your hardware configuration.

           

          Regards,

          Cristian.

          1 of 1 people found this helpful
          • 2. Re: How do I create a tde from large txt files
            Brian McClain
            Brian McClain May 6, 2013 6:37 AM (in response to Cristian Vasile)

            I've actually been working with the files within Tableau for the past week after tricking them into the software, linked live.  As is always the problem with live data, however, it's painfully slow.  That fact is obviously worsened by the tremendous size.

            • 3. Re: How do I create a tde from large txt files
              Cristian Vasile
              Cristian Vasile May 6, 2013 9:11 AM (in response to Brian McClain)

              Brian,

               

              Well, with huge text files without any indexes obviously you cannot drive on fast lane. Try to import your data in Tableau Data Engine (TDE) and then you could do a fair comparison.

               

              If you run V8 then enable performance recording, and application will create a nice graph (Gantt type) displaying all major steps in order to achieve the tasks.

               

               

              I did play with 110 Milion records imported from csv on TDE and creation of graphs is running very fast on my old laptop.

               

              If from various reasons your are forced to keep data in flat files, linked live, i suggest you to create a RAM disk, with compression enabled at OS level, and then copy your files on RAM disk; the time period to create the graphs should be extremely short.

               

              Other option is to install your working environment (flat files, twb, etc) on a brand new SSD disk, maybe on external cage connected via USB 3.0, prices are very competitive now.

               

              Regards,

              Cristian.