1 2 Previous Next 15 Replies Latest reply on Jul 1, 2014 5:20 AM by Ian Pytlarz

    Pushing the boundaries: Displaying an infinite hierarchy of data

    Ian Pytlarz

      So, I saw a project that a graphics programmer was working on yesterday and ever since my head has been buzzing to see if I can figure out how to mimic it in Tableau. Essentially, he used an ever-unfolding graph to display hierarchical dimensions until you got down to the lowest child elements and could look at the data: http://youtu.be/KN3tKT0E0t8?t=14m25s

       

      As far as exact mimicry, obviously even his simplest display is impossible in Tableau. However, in theory, I can imagine setting up a viz that lets you navigate through your data by spawning a new set of nodes in the perpendicular direction when you click on a node (so down, right, down, left, etc). It feels like you would need to use the custom mapping and pathing to essentially map all of your data onto a giant coordinate plane, following pathing graphs from the top data elements to all of their eventual children. Since you can precisely control the XY points you put things on, you might even be able to make it look pretty cool and have patterns other than branching straight lines after you figure out the initial math. You could also (in theory) use the zooming on maps to perhaps get some of the intuitive zooming you see on the video.

       

      Has anyone tried anything like this in Tableau? I'm currently trying to understand how I would have to organize the data in order to use the paths appropriately, and how it might look in tableau. I'll probably have to create a trivially small sample to test on. I've been thinking about how to code something like this for any dataset, and the algorithm seems like it will end up being fairly complex. Probably many days of work, so I wanted to get the community's thoughts now as I likely will only have spots of time here and there to work on this and any input that helps speed this up would be great.

       

      End goal: Create a SAS (or any other appropriate language) program to use hierarchical data to create a pathing dataset to load into tableau and display in such a way that you can dig into even very complex hierarchical data intuitively and get a sense of its structure. Perhaps displaying data in a side panel that summarizes the satellite data of everything underneath the currently selected node (assuming, unlike in the video, that all of your endpoints contain the same types of satellite data). For my purposes, I'm envisioning course data where you drill from college to department to course to section to individual students.

       

      Assuming I end up concluding that this is indeed possible in some form, this will probably be a project I work on intermittently until I feel I have something satisfying. Any thoughts or suggestions would be very appreciated.

        • 1. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
          Allan Walker

          Hello Ian,

           

          The first thing I'd take a look at is this blog post.  Upshot - network graphs take a lot of effort.

           

          I'm interested too but from a very different angle: my use case is GIS, and representing OpenStreetMap nodes, pathways and relationships natively as a foreground map in Tableau.  I get around this at the moment by using shapefile hierarchies, merging polygons to point, lines to point, and then all the point files.  The OSM schema is fairly difficult to work with but can be handled using PostgresSQL/PostGIS.  I suspect you'll have to work out a way of using BLOBs or do some sort of XML transform.

           

          We do have a method for "dummy" (i.e. not using the built in) hierarchies but it involves a lot of calculated fields, basically concatenating every branch and leaf, use a boolean T/F and contains at each level and then stripping back out. What I can't do is programmatically create a loop to re-create the Calculated Fields at each run time.

           

          Best Regards,

           

          Allan

          • 2. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
            Ian Pytlarz

            This is a great starting point for sure. I could use the data I have to generate point-to-point data elements for every node in the graph just as he has his data organized. Since I'm doing this on a fairly straightforward organizational hierarchy (for now) that actually shouldn't be too bad. I don't think storage would be much of an issue, I'm sure I can store it as a text file since the number of records shouldn't exceed ~1.5M for what I'm looking at. The hard part will be doing the math as to where to put the various nodes so that the end display looks readable, and ensuring that I aggregate the satellite data for child elements properly on each parent node, especially if I map down to the individual student level (which is where the real size comes from, without them we're looking at a hundred thousand nodes or so). I worry about Tableau freaking out trying to display so much at once.

             

            I wonder if there is a way that I could have the path branch out based on user interaction. For instance if they just saw some plain nodes, one for each college at the start, and had a line of nodes appear underneath a college after clicking on it while also having the other nodes fade out. I guess I have to get the dataset working before I can test that, but for now I have something to work on. Thanks!

            • 3. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
              Allan Walker

              Ian,

               

              I've successfully displayed 60,000,000 marks in a Tableau Viz before (circle mark type).  I suspect when you get hold of the 64-bit version of desktop you'll be very happy.

               

              Best Regards,

               

              Allan

              • 4. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
                Ian Pytlarz

                That's right, I forgot Tableau was only 32 bit. I'm downloading the 64 bit now, hope to find time soon to make this and see how well it works, maybe I could even leverage R to do something interesting. I'll try to come up with ways to put it through its paces. We're a huge school and have an incredible amount of data, I'm looking forward to experimenting ways to visualize everything at once.

                • 5. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
                  Ian Pytlarz

                  Alright, I've made some progress on this. I made a visual basic program in excel to take pathing data, organize it into a network graph, and spit it out in a format tableau can read. The program also adds all satellite data (anything to the right of your path columns) to each node it passes through on the way to its last node. It skips blank entries, and builds a network of node objects in memory. It then uses a visualization method to assign XY coordinates and spits out the data in a format tableau can read using the tutorial you provided.

                   

                  I've currently only got one method in the Visualization class to create XY coordinates, and it assumes you have an acyclical graph where each node has at most one parent. It creates a "bloom" tree pattern. It isn't perfect (you can see that in the display, there is still some overlapping) but it is pretty good.

                   

                  Here's what it produced:

                   

                  Workbook: Network Graph

                   

                  Now I'm wondering how I can improve this. I'd like it if clicking a node filtered (or at least highlighted) to all things that connected to that node further down the tree. I'd also like it if I could have clicking a node pass the name of that node to a filter on another dataset, so I could have a sub-view that contained more detailed data on things in that node. Maybe I just need to jam everything in the tooltip though.

                   

                  Would love to hear suggestions for improvement. Right now I'd say it has 'potential' but isn't quite usable yet.

                   

                  EDIT: Here is the excel program: Network Graph Tableau Data Generator.xlsm - Google Drive

                  1 of 1 people found this helpful
                  • 6. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
                    Allan Walker

                    Ian


                    That's stunning.

                     

                    I've used NodeXL in the past, but your script looks really promising! I'd love to see Tableau build Network Graphs natively.

                     

                    If you have a Google Drive account you could share the XLS on there?

                     

                    Best Regards,

                     

                    Allan

                    • 7. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
                      Ian Pytlarz

                      Edited my last post to include a google drive link. I programmed it in a totally object oriented style, so if you had an idea for how to visualize certain graph types, you could just add methods to the Visualization class and call them instead of Visualize.createBloom.

                       

                      Make sure you have an acyclical graph when you use this, otherwise the createBloom method will get stuck in an infinite recursive loop .

                      • 8. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
                        Allan Walker

                        Ian,  it would be great if you could put a copy of the packaged workbook in the G Drive; maybe create a folder "Network Charts for Tableau" - oh, and for users, a little tutorial in MS Word/PDF: The PDF could be uploaded to the Tab-Wiki.

                         

                        BTW: simply awesome.  We have virtually the same hierarchy and measures here at USU: this is just a great tool.

                        • 9. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
                          Ian Pytlarz

                          Unfortunately I built that in 8.1, so I can't package it for anyone that doesn't have the beta. I can try to find some time to re-create it in 8.0 and package it up. I'll also try to find time to flesh out the description of how to use the program, things that may be good to edit, etc. I have another project that came up and demands my attention, so all that will probably have to wait till next week.

                           

                          Thanks for the interest, and let me know if you make a viz with more usability. What I've done so far seems more proof-of-concept than anything else right now. Needs more bells and whistles.

                          • 10. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
                            Anusha M

                            Hey Ian,

                             

                            Could you please share the Workbook recreated in Tableau 8.0? It would be of great help.

                             

                            Thanks,

                            Anusha

                            • 11. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
                              Ian Pytlarz

                              I no longer have 8.0 installed anywhere, and unfortunately I won't have the time to re-create it anytime soon. If your organization isn't past 8.0, I'd encourage you to grab 8.2 for your desktop and just open this in there. You should be able to use what you find to re-create it in 8.0.

                              • 12. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
                                Anusha M

                                Hi Ian,

                                 

                                Well i did that and i couldnt thank you enough for coming up with this. It

                                was very crucial. I havent worked on these ever. Have a couple of doubts:

                                How you calculate the values for line x and line y. I might be sounding

                                too stupid but this is my first time ever Have always stuck to "Show me"

                                options. Could me please help me out with this? I have to use this

                                representation for a very huge data set, wherein i have around 1200 highest

                                parent nodes. If you could please help me figure how you have calculated

                                them, itd be of great help. Also has there been any custom sql youve

                                written for this?

                                 

                                All your help is appreciated. Thanks a lot

                                 

                                 

                                 

                                On Thu, Jun 26, 2014 at 6:03 PM, Ian Pytlarz <

                                • 13. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
                                  Cristian Vasile

                                  Anusha,

                                   

                                  Maybe Tulip could help you better.

                                  Take a look here: http://tulip.labri.fr/TulipDrupal/

                                   

                                  Regards,

                                  Cristian.

                                  • 14. Re: Pushing the boundaries: Displaying an infinite hierarchy of data
                                    Anusha M

                                    Cristian,

                                     

                                    I have to build this tool in tableau only and thats a major constraint.

                                     

                                    Regards,

                                    Anusha

                                     

                                     

                                    On Sun, Jun 29, 2014 at 10:05 PM, Cristian Vasile <

                                    1 2 Previous Next