Not much to say I got my additional secondary nodes -- a total of 4 additional ones, bringing my environment up to 6 servers -- so I got those up and going. Through trial and several errors I found out that Tableau Server is particular about how the Coordination Service Ensemble is made.
I worked with another Tableau employee, Kevin, who is a Principal Solutions Engineer. He collected my TS health check data and gave me a document that had recommendations for setting up my new environment to handle high extract and high view usage. I will use this to set up my environment. With so many ways to set up a TS his help was invaluable for me. His recommendation is also built to easily accommodate current demands -- the environment will also be expandable vertically and horizontally for future demands when needed.
I found that if I follow the instructions on the Example: Install and Configure a Three-Node HA Cluster - Tableau page the process went smoothly. I already had a primary and secondary environment going so this was just installing the additional nodes. I followed the "Use the TSM web interface" instructions, skipping Step 1 of course since the initial node was already established. On Step 2 I saved the bootstrap file on a network drive so it was easy to access on the other four nodes. Briefly, here's what I did on each node:
- I performed Step 3 (Step 4 is the same instructions). You NEED to follow the process as given!
- Don't configure the node, just leave it be. Configuration will happen later. Be patient.
- Close the browser as it won't be needed any more.
- Repeat #1 for the next node.
Perform the installs sequentially, meaning, don't try to install all of them in parallel. You can install the software (Run As Administrator) on all the additional nodes in parallel (do them all at once) but when you get to the "Node Configuration" screen asking for the bootstrap file and credentials you need to stop! This is where you now work serially: Perform the node configuration on your first new node and perform each step until the end. Then go to the next node and perform the node configuration on that one. Repeat until all are completed.
Now move on to Step 5 on the primary node and follow the instructions.
Move on to Step 6 and follow the instructions. Since I have 6 nodes I used the recommendation of deploying a Coordination Service to 5 nodes. Below is my screen shot of the steps for deploying and then cleaning the coordination service:
Step 7 & Step 8 are the same and didn't apply to me so I skipped them, making the configuration changes as recommended to me.
I did run into an issue that stopped me from successfully completing and that was my #3 Gateway was failing to work. After a search I found the problem and fixed it. See the document here, Tableau Server Gateway Services fail to start or TSM becomes inaccessible on Windows Server 2016 (by Matthew Brimer) , and my comment in it (it's the first Comment). Once I fixed the Windows registry value in four of my offending servers everything worked perfectly! I then ran my server setup script (custom settings, banner images, etc.), proxy/gateway setup script, and data source drivers on all of the new servers since they require it.
Here's how my environment looks at the time of this writing: