10 Replies Latest reply on Jul 2, 2015 11:53 PM by Russell Christopher

    Tableau 9 extracts slower than 8.x?

    ravi.kulkarni

      We are struggling big time with extracts, connecting especially to our hadoop cluster. Extracts started taking more than 50% -100%, compared to 8.2 environment. We changed 2 variables at same time. For our 9 environment, we got VMs, we had 8.2 running on physical server. So unable to nail down, whether it’s because of hardware or tableau release. Tableau support wasn’t able to help us figure out, what’s causing this.


      Our IT folks, think it’s because of newer version tableau than because of hardware.


      Just to have little more details on extract in discussion, we run server on 8 core server with 64G RAM and extract in discussion, pulls around 100Million records, used to finish between 90-120 minutes. Physical server had Intel chipset, with Hyperthreading enabled. We tested disabling hyper threading and 8.3 extract on physical server runs way faster. On other hand VMs use AMD chip set with higher clock speed. We also managed to reserve CPU in VM environment, with no luck.


      Why we have 100Mil+ records in extract?


      Well, the promise was to provide all data, all the time with super fast response, right? I know, no body can use all of those 100Mil records at the same time to visualize it in meaningful way, unless, they aggregate at low cardinality, but in order to provide 360 degree view of our data, there is need to provide data at this grain. Live connection is out of question, because of response time of Hadoop, so ......


      Tableau support wasn’t able to provide details on how to understand the log file to figure out “how much time was spent” on various logical steps in extracts, like executing the query, when was last record was received, time spent to actually build tde, to build indexes etc.

       

      Anybody with similar experience?

      Thanks in advance

        • 1. Re: Tableau 9 extracts slower than 8.x?
          Rody Zakovich

          Hello Ravi,

           

          Unfortunately, I am not very familiar with these situations, but I wanted to get in a ping Russell Christopher

           

          Anytime I have struggles with Server related issues, I find his blog to be absolutely amazing!

           

          Tableau Love

           

          I also suggest posting this in the Server Admin Community. That forum has less activity, and more dedicated Tableau Server gurus on it to help out.

           

          Best of luck!

           

          Regards,

          Rody

          • 2. Re: Tableau 9 extracts slower than 8.x?
            vishwanath Pendyala

            Which version of 9.X are you running. ?

             

            Are there any activity going on while executing extracts .Monitoring ram usage and  memory did it give any insights on the server ?

             

            Thanks

            Vishwa

            • 3. Re: Tableau 9 extracts slower than 8.x?
              ravi.kulkarni

              Thanks Rody,

              I will post it on Server Admin community and tableau love.

              • 4. Re: Tableau 9 extracts slower than 8.x?
                Russell Christopher

                When creating extracts (especially big ones) , lots of stuff gets landed to the disk. IT is notoriously stingy on disk when they provision a VM - unless one specifically asks for high throughput (which is what Tableau needs), you tend to get "shared dregs"...This could substantially slow your extract prep down. Check to see what sort of disk throughput you have.

                 

                There are really no changes to 9 that will substantially impact the speed at which extracts are created, so I doubt that's it.

                 

                The easiest way for you to prove this to yourself is to simply install 8.2 desktop on a machine and extract...then replace with 9.03 and try again. Yeah, I know you're running server (and you could do the same test with Server, too) , but the same stuff is essentially happening under the covers. Heck, your problem could be a simple as networking -- since you're on VMs, your virtual boxes could likely be on a different subnet than Server used to be...and now your requests may be getting bounced around a lot more on the network than they were before.

                 

                If there were my problem to solve, I'd rule Tableau out first with the test above. Once you do so, you'll want to use 3rd party tools to benchmark CPU, IO, etc. on the "old" and "new" boxes to see what's different. You might need to use wireshark or other tools to understand how network requests are being routed, too.

                • 5. Re: Tableau 9 extracts slower than 8.x?
                  ravi.kulkarni

                  Russell,

                  Thanks for the reply.

                  I think we did some tests using 8.2 vs 9, but not back to back to provide comparison, so that makes sense and I will run 2 extracts using desktops.

                   

                  On network layer, we did run some diagnostics to check the throughput and both servers pumps more or less same.

                  I haven't compared disk I/O, on physical server, storage is local and on VM it comes from SAN. As it comes from SAN, we tried using SSDs in VM environment.

                   

                  There is one more variable in the equation.

                  8.2 was installed on one server with 8 cores and 9 is installed on 2 servers 4 cores each. Worker node is configured with 4 backgrounders. 2 data servers and 1 data engine on Primary. (See image below)

                   

                  So another reason or followup question could be, if you have 4 backgrounder processes on 8 core machine vs 4 backgrounders on 4 core machine, do they behave differently, means during morning hours, we have to run multiple extracts and if you have 4 backgrounders on 4 core machine, processing power might not be enough? I do see CPU maxing out on working when this particular extract runs.

                   

                  6-30-2015 9-34-28 AM.png

                  • 6. Re: Tableau 9 extracts slower than 8.x?
                    Russell Christopher

                    Yes, this.

                     

                    in nearly all scenarios, splitting a single 8 core is bad. While process isolation can be a good thing, very few are happy with the results of you end up with when there are less than 8 cores to handle either VizQL or Engine/Backgrounder work...you end up with a distributed installation that doesn't do anything especially well...

                     

                    Sorry to be the one to say this, but just keep it simple. stick everything back on an 8 core vm and see how things go...obviously, you'll want less backgrounders in that scenario, as well.  Probably no more than 2.

                    • 7. Re: Tableau 9 extracts slower than 8.x?
                      ravi.kulkarni

                      Russell Christopher

                      We decided to distribute it, because while these big extracts would run, we had situations, when desktop users won't be able to connect to server deployed data source, you can imagine what would happen with server users. With distributing processes, most of the time (over 95%) user complains were gone.

                      I was planning to have one 8 core server, but planning to have 4 backgrounders, as I have seen in past 2 backgrounders not enough to finish all extracts.

                      I will update after we try these options, reconfiguration is supposed to happen tonight.

                       

                      Also at this time, not considering to buy any additional core licenses.

                       

                      Thanks for your help.

                      • 8. Re: Tableau 9 extracts slower than 8.x?
                        Russell Christopher

                        Understood - you needed to isolate in order to protect your interactive users. But, you have to realize (and I think you do) that you're attempting to do the same amount of "extract refreshing work" that you used to with half the resources.

                         

                        If I take in half the calories I used to, I'll start dropping pounds fast - and probably some muscle too. Your 4-core box just can't do as much as the eight core in regards to this processor-intensive work. You simply have less raw horsepower to bring to bear on the task.

                         

                        Have you read the topics in help on the # of backgrounders to use based on what other services are on the box? You generally do Cores/4 when have "other services" on the machine. When it's an extract-only box, Cores/2 or Cores/4 is generally appropriate...If you have lots of small extracts you might be able to get away with "Cores":

                         

                        Improve Server Performance

                         

                        (Skip to about 50-70% of the way down) You're being pretty aggressive by putting as many backgrounders on the machine as there are cores - especially since we already know that your extract building activities are not trivial. You'll also notice in the topic above there is no mention / guidance of how many backgrounders to put on a split 8 core - because we know that scenario generally ends in tears

                         

                        I suspect you've thought over much of this already and you're in the best place you can be without more hardware to throw at the problem. FYI, you're not alone in this - adoption of Tableau can be viral at companies, and that 8 core you thought would last you a couple of years can get over-utilized FAST.

                        • 9. Re: Tableau 9 extracts slower than 8.x?
                          ravi.kulkarni

                          So in other words you are telling, although backgrounders are single threaded processes, can still run on more than one core? Which also means, in 8.2 setup on 1 physical scenario, 4 backgrounders were using all 8 cores when needed? I don't see that from CPU utilization chart below.

                          1server.png

                          On the other hand when we went live on 4 backgrounders on VM with tableau 9, I see this on worker node where we have backgrounder. See how many times we hit 100% CPU

                          worker.png

                          It's more dramatic when you look at individual day

                           

                          worker-today.png

                          Thanks for all valuable information.

                          • 10. Re: Tableau 9 extracts slower than 8.x?
                            Russell Christopher

                            The backgrounders are indeed single-threaded, but they don't do the actual work - They're just a scheduler. If you use a tool like SysInternals Process Explorer, you'll see that all the Backgrounder does is launch a copy of tdeserver64.exe which does the extract generation. tdeserver64 is multi-threaded and will eat up as much CPU (on as many cores) as you can give it.

                             

                            Don't know why you're not seeing this in 8.X, but I'd assume it's because the 8.x machine has twice the processing power as your VM.

                            1 of 1 people found this helpful