6 Replies Latest reply on Aug 5, 2013 8:45 AM by Allan Walker

    How to handle large amount of data

    Josh Price

      Good Afternoon,

      I have a file containing over 60K lat/long points that I need to plot in Tableau.  It was a shapefile containing all US railroads in linestrings, which I used QGIS to converted to a usable CSV file for Tableau (each line now has an Id and a numbered sequence).   The problem is this turned into a 71 mg file!  I can put the data into Tableau but as you would imagine, it's so slow it's not even useful.

       

      As I said, it is a file containing every rail line in America - Each point is a station, yard, switch, etc...

       

      Is there is another way to get this layer on Tableau? 

       

      My goal is map all rail lines and color code based on the rail owner. Is there a way I can convert this CSV to a format usable to Tableau making it smaller?  I extracted the data and created a tde file, but it didn't help.

       

      I also thought about limiting the user to specific regions, limiting the data Tableau has to process, but this would mean I would have to create many different versions for each region.  And I REALLY want to see all the data together. 

       

      The only option I can think of is to create/host a WMS server with the data.  Then just link the WMS to Tableau, but I was really hoping to avoid this option.

        • 1. Re: How to handle large amount of data
          Nathan Schneider

          Afternoon,

           

          I had this issue myself. My had a 500MB Excel file, and as you can imagine it was too big to manipulate in a reasonable amount of time.

           

          What I had to do is put the data purge into Access, and Tableau connected to it instantly.

           

          So that's my only suggestion: is there anyway you can get your data into Access? That is, is your data Access friendly? It would allow you to connect much faster.

           

          Regards,

           

          Nate

          • 2. Re: How to handle large amount of data
            Dan Huff

            Josh--

             

            Two questions:

             

            Are you using Tableau Desktop 7?

             

            If so, have you considered upgrading to Tableau Desktop 8?

             

            There have been a number of significant improvements in rendering performance that have landed in 8.0 that may help you out in this situation. I say "may" as I do not know if you hardware or what you are doing with your data will fall into the categories that benefit from these changes but I would wager it is worth a shot.

             

            If this is not an option, here is what I would suggest trying:

             

            1. Make an extract (as you have done)
            2. Build your map
            3. Add Region to the filters shelf, right click it, and select Add to Context
            4. Now change the filter into a single select filter

             

            This may help performance you are seeing by limiting the records to a much smaller subset than the whole. The Add to Context options creates a temp table that can improve performance in situations where it removes a large portion of the data. This will also allow you to use filters in addition to region within the vizzes. These filters will now wait for the temp table to be made thus possibly resulting in faster response times.

             

            Sorry for the ambiguity in my response. It is just hard to know without having the workbook in hand along with some other facts.

             

            Hope this helps,

             

            Dan

            1 of 1 people found this helpful
            • 3. Re: How to handle large amount of data
              Josh Price

              Thanks for the response Dan.  I actually thought just adding filters wouldn't really help the performance, since the source data doesn't change.  But I will give it a try!  I think I can still use the data by only making one region selectable at a time.  This way I can avoid someone selecting the entire data set. 

               

              I wanted to be able to email this file but at 85MB I've given up on that pipe dream!

               

              I just upgraded to Tableau 8, so here is to hoping this helps!

              • 4. Re: How to handle large amount of data
                Allan Walker

                Josh,

                 

                Another way is to create your own WMS server (Geoserver), put your shapefile in that, and then link/add your WMS server to Tableau.  If you are used to GIS systems (I see you are using QGIS) then installing your own WMS server will be a snap.

                 

                Best Regards,

                 

                Allan

                • 5. Re: How to handle large amount of data
                  Frank Benavides

                  Hi Josh,

                   

                  I need to build something just like what you are describing. Did you get your map to work with filters that show specific rail lines?

                  • 6. Re: How to handle large amount of data
                    Allan Walker

                    Frank,

                     

                    If you are looking to convert linestring to point, the best way I have found is to use either ET tools or X Tools Pro in/for ESRI.

                     

                    The other question is whether you want to analyse them.  If you want to just use the linestring, then import that into PostgreSQL and connect that table into Geoserver, and use WMS.  If you want a foreground layer, convert to Points using the above, use a personal geodatabase, and connect to the mdb with the Access connection (if less than 2GB), or import the point table into PostgreSQL.  If you want to use WMS, you can style the attributes using AtlasStyler.

                     

                    Best Regards,

                     

                    Allan