-
1. Re: How do I create a tde from large txt files
Cristian Vasile May 3, 2013 3:47 PM (in response to Brian McClain)1 of 1 people found this helpfulBrian,
You will never be able to import a text file (i suppose is .csv .tsv or something similar) larger than 4Gbytes, due to Microsoft JET Engine constrains.
I 'll suggest you a two stages approach
Stage 1
a. split the biggest file in 2 or maybe 3 to be safe small files.
b. import text files in a ms access database; after import do some minimal checks.
Stage 2
a. import ms access table or tables in Tableau
Be aware that the time frame for entire operations could take a couple of hours - highly dependable of your hardware configuration.
Regards,
Cristian.
-
2. Re: How do I create a tde from large txt files
Brian McClain May 6, 2013 6:37 AM (in response to Cristian Vasile)I've actually been working with the files within Tableau for the past week after tricking them into the software, linked live. As is always the problem with live data, however, it's painfully slow. That fact is obviously worsened by the tremendous size.
-
3. Re: How do I create a tde from large txt files
Cristian Vasile May 6, 2013 9:11 AM (in response to Brian McClain)Brian,
Well, with huge text files without any indexes obviously you cannot drive on fast lane. Try to import your data in Tableau Data Engine (TDE) and then you could do a fair comparison.
If you run V8 then enable performance recording, and application will create a nice graph (Gantt type) displaying all major steps in order to achieve the tasks.
I did play with 110 Milion records imported from csv on TDE and creation of graphs is running very fast on my old laptop.
If from various reasons your are forced to keep data in flat files, linked live, i suggest you to create a RAM disk, with compression enabled at OS level, and then copy your files on RAM disk; the time period to create the graphs should be extremely short.
Other option is to install your working environment (flat files, twb, etc) on a brand new SSD disk, maybe on external cage connected via USB 3.0, prices are very competitive now.
Regards,
Cristian.