3 Replies Latest reply on Oct 13, 2015 8:37 AM by Emma Whyte

    Testing a HA environment

    Emma Whyte

      Hello,

       

      I just wanted to start a discussion for any members that might be responsible for maintaining a HA Tableau Server environment (i.e. a Primary, Backup Primary & Workers).

       

      What sort of testing do you do before you make your environment available to end users?

       

      Do you do any load testing with TabJolt?

       

      Do you do failover testing - i.e. bring a worker or primary down and see how it failsover.

       

      Testing on individual processes - e.g. bring down the backgrounder and see what happens & how to fix.

       

      Thanks!

        • 1. Re: Testing a HA environment
          Michael Perillo

          My Environment Config

          All Prod and Non-Prod environments are of the same configuration using 8-core.

          • Primary
          • Worker 1 (4-core)
          • Worker 2 (4-core)
          • Backup Primary

           

          1. Production / Disaster Recovery (Production Environment)
            • Active/Passive (We're not consuming two-licenses, only one. We do keep drivers and patches in sync.)
            • Licenses are deactivated/activated during testing periods
          2. Development (Non-Prod Environment #1)
          3. Sandox (Non-Prod Environment #2)
          4. Pilot (Single Server for Alpha/Beta Testing)

           

          What sort of testing do you do before you make your environment available to end users?

          • Connectivity Testing on all types of connections (firewall, io, network stuff)
          • Random Workbook Testing on various connection types
          • Soak test for a few days to see how the environment responds with no user activity
            • Look for CPU spikes, or anything else that's been mentioned over the years in the forums or release notes.

           

          Do you do any load testing with TabJolt?

          • Yes, we started using TabJolt. It's helpful for capacity planning

           

          Do you do failover testing - i.e. bring a worker or primary down and see how it failsover.

          • All apps/services in our organization are required to have a DR environment
          • I do test twice per year our failover capabilities. We'll bring down individual processes, nodes and entire environments.
            • This is an effort to test all forms on business continuity.

           

          Testing on individual processes - e.g. bring down the backgrounder and see what happens & how to fix.

          • These types of tests are conducted twice per year as mentioned above, or unless it really happens.
          • The tests are performed in our Sandbox environment and in our our Dev enviornment. We do not "test" in the Production environment; except when DR is performed in Production and notify users in advance.
          3 of 3 people found this helpful
          • 2. Re: Testing a HA environment
            Jeff Strauss

            What sort of testing do you do before you make your environment available to end users?

            We have a test plan of right now 86 cases that we run through during each upgrade.  These cases are mostly a combination of critical functionality, performance benchmarking, cuonnectivity, a few unique reports and testing of custom scripts / custom portal.

             

            Do you do any load testing with TabJolt?

            Yes, we've run it once so far on dev, and once over a weekend on prod with 9.0.2 with a combination of simple/complex workbooks mostly against extracts.  The results were a bit shockingly low in terms of concurrent users.  I would love to review it at the conference to ensure that we are looking at the numbers correctly for our 24-core prod deployment.  We will do more load testing when going to 9.1

             

            Do you do failover testing - i.e. bring a worker or primary down and see how it failsover.

            I believe we tested failover functionality of the repository and data engine with 8.3, this is when we went to a distributed H.A. environment.  We haven't tested it yet with 9.0, but its probably worthwhile doing again once going to 9.1.  Also, the only time I was able to test the primary failing was when the VM box that the prod primary lives on failed live.  I tried running the manual procedures and I think ran into licensing issues that the failover box didn't have sufficient licensing.  And it was a time critical thing and we were able to get the active prod VM back up and running.  Since this time, we have updated our manual procedures.  It would be awesome to have an automated process for the primary failover!!!!!!!!

             

            Testing on individual processes - e.g. bring down the backgrounder and see what happens & how to fix.

            We have prod tested the killing of background processes, and it seems that the cluster controller successfully revives the processes as expected.

            2 of 2 people found this helpful
            • 3. Re: Testing a HA environment
              Emma Whyte

              That's great thanks very much! I'm putting together a testing plan at the moment so all your input has been really valuable.

               

              Thanks