5 Replies Latest reply on Dec 4, 2017 10:25 AM by Nick Heigerick

    Performance improvement - rendering and extracts - big find

    Jeff Strauss

      This is a massive speed lift for us.  Perhaps your IT windows admin already has this covered.  But if not, read on...

       

      If you have sub-optimal performance, there could be many many contributing factors (i.e. workbook design best practices).  But still, there may be a few "easy buttons" out there.  The one that we came across (within our on-premise Tableau Server deployment) has to do with an underlying server BIOS setting.

       

      We run Dell physical server hardware.  Mind you that each BIOS version seems to have different default settings and options, our BIOS default was DAPC (Dell Active Power Controller), which essentially indicates a variable speed CPU frequency based on demand.  So the more demand, the higher the clock speed.  This is quite beneficial from an overall datacenter energy footprint and perhaps an electrical engineer out there won an award for this, however it's not so handy for optimal Tableau performance.  We recently changed to a system profile of "Performance".  For more info about the internals, see here:  Dell PowerEdge R730 Owner's Manual

       

      Before the change, the clock speed is usually in the 1Ghz range and it fluctuates based on CPU demand.  Here's a screenprint from one of our dev boxes (1.57 GHz).

       

      After the change, the clock speed speed stays at the maximum (with turbo-boost turned on).

       

      So, with today's processors, for many transaction based applications, variable speed is great, it saves power.  But for computational intensive platforms (i.e. Tableau), it's not so great.

       

      We wondered what the power delta was, so we went and captured it via Perfmon / a Tableau dashboard.  It's not so much.  ~ 100 watts.

       

       

       

       

      And now for the rendering impact / extract impact.

       

      RENDERING times - adjusted BIOS setting late on 4/17

      - Tableau Server users are giving high fives!!!!

       

      - Decisions are being more swift!!!

       

      - It's getting closer to me being able to grab a well deserved drink!!!

       

       

       

      EXTRACT times - adjusted BIOS setting prior to 4/14 run,

       

       

       

      Conclusion

      - Maximize your investment

       

      - This by no means is a blanket endorsement to go in and adjust your BIOS setting

       

      - Go have a look at your BIOS settings, and share what you find out

       

      - For those wondering what's up with the above motorcycle picture, this is Evel Knievel.  if you've never heard of him, go watch some cool videos on Youtube

        • 1. Re: Performance improvement - rendering and extracts - big finding
          Matt Coles

          Great find, Jeff! For what it's worth, we recently switched from the Dell PowerEdge servers to Cisco UCS machines, and found that the new ones were underperforming fairly drastically. After much troubleshooting we found this power-optimization setting was enabled, and after disabling it, saw a marked decrease in computation time (we were running HammerDB for an apples-to-apples comparison between two vanilla Dell/Cisco machines). I imagine it must be the same exact thing you encountered as well.

           

          Here's what the Dell settings were:

           

          Screen Shot 2016-12-14 at 11.16.38 AM.png

           

          Screen Shot 2016-12-14 at 11.17.02 AM.png

           

          Here's what happened to our CPU utilization stats when we moved to the new hosts:

           

          2017-04-19_9-07-47.png

          Here's what happened to median viz load times, week over week for Wendesdays, broken out by hour of the day (I love this viz for looking at perf differences):

           

          2017-04-19_9-08-00.png

           

          We ran Hammer DB using the TPC-C test on a local postgreSQL server created on a vanilla version of each machine type (Dell vs Cisco) just to have some kind of benchmark (we'd previously identified that queries to Postgres within Tableau Server were running way more slowly than usual, so we figured we'd be able to repro the issue this way):

           

          2017-04-19_9-10-30.png

           

           

           

          Fixing the Cisco BIOS settings brought the benchmark back up to reasonable transactions per minute, so we figured we'd licked the problem. Made the change on the prod machines, and we were back in business!

          2 of 2 people found this helpful
          • 2. Re: Performance improvement - rendering and extracts - big finding
            Jeff Strauss

            thanks for sharing your comparisons.  They're quite insightful.  So I'm not the only one that's treaded down this troubleshooting path.  How I found this issue was a comparison of 2 dev vanilla environments with one of our prod dashboards for benchmarking.

             

            Environment 1: VM - 8 seconds render

             

            Environment 2: 7 year old dusty physical box - 16 seconds render

             

             

            There were a plethora of internal architecture differences (too many to name), and I was at first being persuaded that the physical box was really old, but I pressed on.  It was a labor of love looking through the logs and comparing milliseconds for instructions when I came to the conclusion that CPU frequency is a big factor when it comes to computational heavy platforms.

            • 3. Re: Performance improvement - rendering and extracts - big find
              brian.zimmerman.4

              Hello Jeff,

               

              We are facing a very similar issue. We run on VMware. Any insight on how this setting is applied to the VM tier when virtually hosted? We see we can apply a similar system to the Host but wanted to see if you had any ideas or leads?

               

              Thanks in advance-

               

              Brian

              • 4. Re: Performance improvement - rendering and extracts - big find
                Jeff Strauss

                We did not have to adjust our VM instance, it was already set properly, so unfortunately I don't have much insight as to whether this will help or not.

                • 5. Re: Performance improvement - rendering and extracts - big find
                  Nick Heigerick

                  Hey Brian,

                   

                  Just came across your question, hopefully you've got this resolved but in case you haven't I suggest checking out this thread.

                   

                  Tableau VM Tuning and Best Practices

                   

                  The attached pdf has a bunch of valuable information for those of us running virtual deployments. I image you would find the first link on page 5 useful. It links to a VMware document on how to change the BIOS settings in vSphere.

                   

                  Hope this helps!

                  Nick H.