3 Replies Latest reply on Aug 29, 2017 1:53 PM by Glen Robinson

    Changing configuration of 3 node setup - Any concerns in a Production environment?

    Greg Pie

      Hello, I am currently on Tableau 9.3 and soon I would like to upgrade to 10.3.1. Before I do this I was advised that my Production environment should be configured differently because having processes such as the data server/vizQL...etc will count towards cores against my license agreement.

       

      I'd like the change the current setup per below to match the proposed setup. I have 1 primary gateway, 1 backup gateway, and 2 worker nodes. Does anyone have any tips for things to watch out for doing this in a Production environment with 140 extracts that kick off based on a timed schedule. I know I will need to coordinate an outage with the users from my company but I had concerns about how long of an outage I could face with stopping services and syncing servers. Does anyone have a high level overview what would need to be done to reconfigure the bottom left to match the bottom right. Would it be like the following;

       

      1. Log into Primary server. Use Tabadmin to stop services(How long could this take?)

      2. When services are stopped go to the Tableau configuration page and remove 'Application Service, VizQL Server, Cache Server, Backgrounder, Data Server, Data Engine, File Store, Repository' from the Primary server and add them to the workers.

      3. Use Tabadmin to start the services back up(how long could this take?)

      tableau_931setup.jpg

        • 1. Re: Changing configuration of 3 node setup - Any concerns in a Production environment?
          Glen Robinson

          Hi Greg

          With your current setup, you have 3 nodes (1 primary and 2 workers) which all have licensed processes running on them, so are all contributing to your core license cost.

          Whilst the new environment, only has licensed products on the 2 workers, so only they are contributing to license cost.

          Does this mean that you are increasing the number of cores on these 2 workers, or are you planning to not use those core license?

          Therefore, I am not sure why you have been advised to change to a different configuration because of version upgrade.

          Upgrading to 10.x with the current configuration isn't going to change how you are currently licensed.

           

          Changing the configuration should be done, because there is a particular issue you are looking to resolve. ie You want to have a High Availability environment, or you are looking to use your resources more efficiently (or resolve a bottleneck)

           

          The process itself to migrate is very straight forward. Firstly, you should decide whether to upgrade to 10.x first and then change configuration, or the other way around.

          However, the configuration change process is as you describe. It should take no more than an hour. ( i would think, but have some contingency just in case)

           

          Additionally, the new configuration is not showing the same number of processes on each of the workers. Is there a reason for this?

          And, with only 2 background processes (the previous configuration shows 4, and all being actively used), means that there will probably be delays in extract refreshes etc, if you move to this configuration.

           

          Personally, if this was my environment, my approach would be to do the following.

          1. Build a brand new cluster on version 10.x, with the configuration desired.

          2. Restore current data to it (dashboards, extracts), and test. Test the dashboards, test the performance, test load (using Tabjolt), Test Everything.....Ensure that the environment does what you expect it to do. And get everyone who is testing to sign off that they have tested and are happy with the new environment.

          3. Pick a Go Live Date.

          4. On Go Live Date, perform final backup of prod 9.3 , and restore to new prod 10.3

          5. Point everyone to new Tableau Cluster.

          6. If there is a problem with the new cluster, then reverting to the old cluster is very straight forward, as it is still there. Just point end users to it.

          7. Once everyone is happy with new cluster, decommission old cluster.

           

          Hope that this helps

          Glen

          • 2. Re: Changing configuration of 3 node setup - Any concerns in a Production environment?
            Greg Pie

            Hi Glen,

             

                           Whilst the new environment, only has licensed products on the 2 workers, so only they are contributing to license cost.

                           Does this mean that you are increasing the number of cores on these 2 workers, or are you planning to not use those core license?

             

            So right now I have the following to be specific I am planning on increasing the total cores on all of my VMs. So right now I have 4 cores across all machines. Because of this I am only using 12 cores due to the way the services are configured. I was told I could increase the number of cores to 8 cores on all machines and still be in my license agreement if I move the processes to the worker nodes and like the Primary nodes as just a gateway.

             

            Current environment

            Server

            Cores

            Primary Gateway

            4

            Backup Gateway

            4

            Worker 1

            4

            Worker 2

            4

             

            Future environment

            Server

            Cores

            Primary Gateway

            8

            Backup Gateway

            8

            Worker 1

            8

            Worker 2

            8

             

             

                           Changing the configuration should be done, because there is a particular issue you are looking to resolve. ie You want to have a High Availability environment, or you are looking to use your resources more efficiently (or resolve a bottleneck)

             

            I’ve been recently given the role as Tableau admin(I have a MicroStrategy background) and the person who originally setup the environment is no longer with my company and there is no documentation as to why the environment was setup the way it was. There currently is a lot of performance issues with Tableau and I have seen evidence of the cpu being 100% at times on worker 2 due to the amount of extracts. The users want to upgrade to v10.3.1 in order to take advantage of new Vis but before I do that I would like to address the issue of the performance. The solution I came up with was to fully utilize the amount of cores that my company is licensed for. I found out that my Tableau environment is only using 12 cores out of 16 cores(seen in the attached JPEG). I would like to increase the core count from 4 cores to 8 cores on all Windows VMs that Tableau is on and then reconfigure the services so the two Worker nodes are using all of the licenses  (8 cores + 8 cores = 16 cores). I will then leave the Primary and Backup Gateway as having 8 cores each but they will not count towards my license count. I listed my setup in the attached JPEG.

             

                        The process itself to migrate is very straight forward. Firstly, you should decide whether to upgrade to 10.x first and then change configuration, or the other way around.

             

            Yes I would like to upgrade my Windows VMs with more resources first, then change the configuration on 9.3 to use these resources. Then a couple weeks late I would then like to perform a upgrade from 9.3 to 10.3.1. I want to address the resources first because my company just purchased an additional 160 licenses which is going to really start adding a load to Tableau since I am only using 100 licenses right now.

             

             

                            Additionally, the new configuration is not showing the same number of processes on each of the workers. Is there a reason for this?

                            And, with only 2 background processes (the previous configuration shows 4, and all being actively used), means that there will probably be delays in extract refreshes etc, if you move to this configuration.

             

             

            I should clarify, the proposed setup I showed was a sample sent from my Tableau rep for a typical 3 node setup showing that the Primary node should only be a gateway so you can setup the worker nodes with a higher capacity of cores to utilize your core license better. I attached a screenshot of my current setup for ‘Tableau Server Configuration’. If I move all of the services off of the Primary (except Gateway and Search and Browse) typically how should this be setup based on what I have now? Do you have any suggestions as a guideline? So for example would I just move the VizQL 2 over to the Worker 2 node? I have no documentation whatsoever on why the services we’re configured the way they were. The only thing I do know is that I currently am having performances issues(seems like worker 2 is always running high on the cpu). Also I will be adding a lot of new users to my environment within the next 2 months so I would like this configured much more robust.

            Thank you for the steps you listed. Unfortunately I will not be able to have a separate cluster to point users to. I only have a testing cluster and I will have to perform an in place upgrade on the current Production cluster

             

             

            tableau_config.jpg

            tableauenviromentcores.jpg

            • 3. Re: Changing configuration of 3 node setup - Any concerns in a Production environment?
              Glen Robinson

              Hi Greg

              Thanks for response, and additional information.

              1. Regarding licensing. I am assuming that when you say Primary Gateway and Backup Gateway, you are referring to a Backup Primary Server

              Use a Backup Primary

              If this is the case, then having this server doesn't count towards your licensing, as it isn't an active part of the cluster. Therefore, under the current arrangement, you are 'wasting' 4 cores, which you could legitimately add to the current configuration.

              Additionally, in your new configuration, as the Primary (and Backup Primary) are only handling licensing and gateway processes, then 4 cores should be ample (I would say)

              2. You say that Worker 2 regularly hits 100% CPU utilization. Considering it is only 4 cores, and is running 4 backgrounder processes, this isn't surprising if you have lots of extracts running.

              3. For your new configuration, I would start with the standard High Availability Cluster (for servers with 8 core). Once up and running, regularly check on server performance (CPU / memory utilization) to ensure that there are no further bottlenecks. Migration to this configuration is as described earlier.

              4. Additionally, do you have an external Load-Balancer in your environment, so that in the event of a server failure, end users are redirected to a working gateway process.

               

               

               

              Hope this helps

              Glen