2 Replies Latest reply on Oct 3, 2018 4:41 PM by Eswar Poosarla

    Tableau Server 2018.2 - Distributed environment - Failed to deploy a new Coordination Service ensemble id '1'

    Eswar Poosarla

      Hi All,

      I am expanding Tableau environment after upgrading a single server environment from 10.5 to 2018.2. I am facing a challenge in deploying coordination service in additional nodes.

       

      On the initial node, i had installed Tableau server on G drive and ran upgrade-tsm.cmd script. The environment started fine.

       

      Next I added additional two nodes (NOTE: I had to install tableau on F drive). I added a repository and also had File store on all three nodes.

       

      Now adding coordination services on all three nodes is causing an issue.

      Servers have windows 2012 R2 64 bit OS and are running with AD service account.

       

      Steps followed to resolve:

      - Activated the license

      - Cleanup coordination services

      - Rebooted servers

       

      Here is the error message I am receiving:

      C:\Windows\system32>tsm topology deploy-coordination-service -n node1,node2,node3 --request-timeout 3600

      Deploying Coordination Service ensemble on nodes [node1, node2, node3].

      Job id is '46', timeout is 20 minutes.

      25% - Validating that there are no pending changes.

      50% - Failed to deploy a new Coordination Service ensemble id '1'.

      An error occurred while deploying Coordination Service ensemble on nodes [node1, node2, node3]. Use 'topology cleanup-coordination-service' to remove non-production ensemble.

       

       

      See 'G:\Tableau\Tableau Server\data\tabsvc\logs\tabadmincontroller\tabadmincontroller-*.log' on Tableau Server nodes running the Administration Controller process for server log information.

       

       

      2018-09-04 11:11:22.363 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated hosts: [server1, server2, server3]

      2018-09-04 11:11:22.366 -0700 14920 main : INFO  com.tableausoftware.config.ServiceRegistrationInfoFile - no registration file found at

      2018-09-04 11:11:22.366 -0700 14920 main : WARN  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Unable to flatten service registration info, because there is no registration file.

      2018-09-04 11:11:22.366 -0700 14920 main : INFO  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Generating workgroup settings: G:\Tableau\Tableau Server\data\tabsvc\config\tabadmincontroller_0.20182.18.0627.2230\workgroup.yml

      2018-09-04 11:11:22.405 -0700 14920 main : INFO  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Generating connection settings: G:\Tableau\Tableau Server\data\tabsvc\services\connections.yml, G:\Tableau\Tableau Server\data\tabsvc\services\connections.properties

      2018-09-04 11:11:22.405 -0700 14920 main : INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting

      2018-09-04 11:11:22.406 -0700 14920 main : INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=server1:8151 sessionTimeout=120000 watcher=org.apache.curator.ConnectionState@3a5f0ed5

      2018-09-04 11:11:22.496 -0700 14920 main-SendThread(server1.cloud.org:8151) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server server1.cloud.org/fe80:0:0:0:5e8:5d7:96b1:4d5e%12:8151. Will not attempt to authenticate using SASL (unknown error)

      2018-09-04 11:11:22.509 -0700 14920 main-SendThread(server1.cloud.org:8151) : INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to server1.cloud.org/fe80:0:0:0:5e8:5d7:96b1:4d5e%12:8151, initiating session

      2018-09-04 11:11:22.637 -0700 14920 main-SendThread(server1.cloud.org:8151) : INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server server1.cloud.org/fe80:0:0:0:5e8:5d7:96b1:4d5e%12:8151, sessionid = 0x100000072820f69, negotiated timeout = 120000

      2018-09-04 11:11:22.638 -0700 14920 main-EventThread : INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED

      2018-09-04 11:11:22.678 -0700 14920 Curator-Framework-0 : INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting

      2018-09-04 11:11:22.687 -0700 14920 main : INFO  org.apache.zookeeper.ZooKeeper - Session: 0x100000072820f69 closed

      2018-09-04 11:11:22.687 -0700 14920 main-EventThread : INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100000072820f69

      2018-09-04 11:11:22.965 -0700 14920 main : INFO  com.tableausoftware.service.helpers.JavaTrustStoreDeployer - Deploying Java trust store from G:/Tableau/Tableau Server/data/tabsvc/services/tabadmincontroller_0.20182.18.0627.2230/repository/jre/lib/security/cacerts to G:/Tableau/Tableau Server/data/tabsvc/config/tabadmincontroller_0.20182.18.0627.2230/cacerts

      2018-09-04 11:11:22.980 -0700 14920 main : INFO  com.tableausoftware.tabadmin.webapp.TabadminController - Generating tabadmincontroller runjavaservice configuration file

      2018-09-04 11:11:22.987 -0700 14920 main : INFO  com.tableausoftware.service.helpers.JavaServiceConfiguration - saving configuration to G:\Tableau\Tableau Server\data\tabsvc\config\tabadmincontroller_0.20182.18.0627.2230\tabadmincontroller.runjavaservice.json

      2018-09-04 11:11:23.017 -0700 14920 main : INFO  com.tableausoftware.tabadmin.configuration.AppConfigUtils - No templates found because G:\Tableau\Tableau Server\data\tabsvc\services\tabadmincontroller_0.20182.18.0627.2230\templates\tabadmincontroller is not a directory.

      2018-09-04 11:11:23.614 -0700 14920 main : INFO  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Loading port assignment from G:\Tableau\Tableau Server\data\tabsvc\config\tabadmincontroller_0.20182.18.0627.2230\ports.yml

      2018-09-04 11:11:23.616 -0700 14920 main : INFO  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Loading user settings from G:\Tableau\Tableau Server\data\tabsvc\config\tabadmincontroller_0.20182.18.0627.2230\tabsvc.yml

      2018-09-04 11:11:23.627 -0700 14920 main : INFO  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Loading topology settings from G:\Tableau\Tableau Server\data\tabsvc\config\tabadmincontroller_0.20182.18.0627.2230\topology.yml

      2018-09-04 11:11:23.629 -0700 14920 main : INFO  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Merging topology settings into user settings.

      2018-09-04 11:11:23.629 -0700 14920 main : INFO  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Setting config.worker_id to 0 in the bootstrap configuration

      2018-09-04 11:11:23.629 -0700 14920 main : INFO  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Merging default, feature, and user settings and registered ports.

      2018-09-04 11:11:24.136 -0700 14920 main : INFO  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Creating PassthroughServiceRegistry

      2018-09-04 11:11:24.136 -0700 14920 main : INFO  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Using ServiceRegistryInfoFile at baseDeployDir G:/Tableau/Tableau Server/data/tabsvc

      2018-09-04 11:11:24.136 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceInfoCache - registered com.tableausoftware.config.PassthroughServiceRegistry@40f9f97e as default registry

      2018-09-04 11:11:24.136 -0700 14920 main : INFO  com.tableau.core.config.FileWatchingAppConfig - Properties with crypto config is not being watched, will not use crypto while reading properties.

      2018-09-04 11:11:24.136 -0700 14920 main : INFO  com.tableau.core.config.FileWatchingAppConfig - Properties URL null for system property: connections.properties

      2018-09-04 11:11:24.136 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceInfoCache - registered com.tableausoftware.config.LegacyPgsqlRegistry@6fc277e4 as registry for pgsql

      2018-09-04 11:11:24.136 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - refresh; forceUpdate

      2018-09-04 11:11:24.157 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated services: [ServiceInfo[m_name=authentication], ServiceInfo[m_name=authorization], ServiceInfo[m_name=backgrounder], ServiceInfo[m_name=backuprestore], ServiceInfo[m_name=cacheserver], ServiceInfo[m_name=clientfileservice], ServiceInfo[m_name=clustercontroller], ServiceInfo[m_name=databasemaintenance], ServiceInfo[m_name=dataserver], ServiceInfo[m_name=extensions], ServiceInfo[m_name=gateway], ServiceInfo[m_name=hyper], ServiceInfo[m_name=keychain], ServiceInfo[m_name=licenseservice], ServiceInfo[m_name=maintenance], ServiceInfo[m_name=monolith_grpc], ServiceInfo[m_name=pgsql], ServiceInfo[m_name=publishing], ServiceInfo[m_name=recommendations], ServiceInfo[m_name=samlservice], ServiceInfo[m_name=searchserver], ServiceInfo[m_name=siteimportexport], ServiceInfo[m_name=tabadminagent], ServiceInfo[m_name=tabadmincontroller], ServiceInfo[m_name=vizportal], ServiceInfo[m_name=vizqlserver], ServiceInfo[m_name=zookeeper]]

      2018-09-04 11:11:24.158 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'gateway' instances: [ServiceInstanceInfo[m_id=1,m_service=gateway,m_host=server1,m_typeToPort={staticPrimary=80, status=80}]]

      2018-09-04 11:11:24.158 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'monolith_grpc' instances: [ServiceInstanceInfo[m_id=2:0:vizportal,m_service=monolith_grpc,m_host=server2,m_typeToPort={grpc=8722}], ServiceInstanceInfo[m_id=3:0:vizportal,m_service=monolith_grpc,m_host=server3,m_typeToPort={grpc=8653}], ServiceInstanceInfo[m_id=1:0:vizportal,m_service=monolith_grpc,m_host=server1,m_typeToPort={grpc=8691}]]

      2018-09-04 11:11:24.158 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'publishing' instances: [ServiceInstanceInfo[m_id=2:0:vizportal,m_service=publishing,m_host=server2,m_typeToPort={grpc=8247}], ServiceInstanceInfo[m_id=3:0:vizportal,m_service=publishing,m_host=server3,m_typeToPort={grpc=8153}], ServiceInstanceInfo[m_id=1:0:vizportal,m_service=publishing,m_host=server1,m_typeToPort={grpc=8240}]]

      2018-09-04 11:11:24.159 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'backgrounder' instances: [ServiceInstanceInfo[m_id=2:0,m_service=backgrounder,m_host=server2,m_typeToPort={staticPrimary=8364, status=8364}], ServiceInstanceInfo[m_id=2:1,m_service=backgrounder,m_host=server2,m_typeToPort={staticPrimary=8384, status=8384}], ServiceInstanceInfo[m_id=3:0,m_service=backgrounder,m_host=server3,m_typeToPort={staticPrimary=8804, status=8804}], ServiceInstanceInfo[m_id=3:1,m_service=backgrounder,m_host=server3,m_typeToPort={staticPrimary=8911, status=8911}], ServiceInstanceInfo[m_id=1:0,m_service=backgrounder,m_host=server1,m_typeToPort={staticPrimary=8418, status=8418}], ServiceInstanceInfo[m_id=1:1,m_service=backgrounder,m_host=server1,m_typeToPort={staticPrimary=8468, status=8468}]]

      2018-09-04 11:11:24.159 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'vizportal' instances: [ServiceInstanceInfo[m_id=2:0,m_service=vizportal,m_host=server2,m_typeToPort={staticPrimary=8262, status=8262}], ServiceInstanceInfo[m_id=3:0,m_service=vizportal,m_host=server3,m_typeToPort={staticPrimary=8689, status=8689}], ServiceInstanceInfo[m_id=1:0,m_service=vizportal,m_host=server1,m_typeToPort={staticPrimary=8999, status=8999}]]

      2018-09-04 11:11:24.159 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'authorization' instances: [ServiceInstanceInfo[m_id=2:0:vizportal,m_service=authorization,m_host=server2,m_typeToPort={grpc=8721}], ServiceInstanceInfo[m_id=3:0:vizportal,m_service=authorization,m_host=server3,m_typeToPort={grpc=8124}], ServiceInstanceInfo[m_id=1:0:vizportal,m_service=authorization,m_host=server1,m_typeToPort={grpc=8716}]]

      2018-09-04 11:11:24.159 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'dataserver' instances: [ServiceInstanceInfo[m_id=1:0,m_service=dataserver,m_host=server1,m_typeToPort={staticPrimary=8563, status=8563}], ServiceInstanceInfo[m_id=1:1,m_service=dataserver,m_host=server1,m_typeToPort={staticPrimary=8455, status=8455}]]

      2018-09-04 11:11:24.159 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'authentication' instances: [ServiceInstanceInfo[m_id=2:0:vizportal,m_service=authentication,m_host=server2,m_typeToPort={grpc=8642}], ServiceInstanceInfo[m_id=3:0:vizportal,m_service=authentication,m_host=server3,m_typeToPort={grpc=8982}], ServiceInstanceInfo[m_id=1:0:vizportal,m_service=authentication,m_host=server1,m_typeToPort={grpc=8616}]]

      2018-09-04 11:11:24.160 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'backuprestore' instances: [ServiceInstanceInfo[m_id=1,m_service=backuprestore,m_host=server1,m_typeToPort={staticPrimary=8152, status=8152}], ServiceInstanceInfo[m_id=2,m_service=backuprestore,m_host=server2,m_typeToPort={staticPrimary=8750, status=8750}], ServiceInstanceInfo[m_id=3,m_service=backuprestore,m_host=server3,m_typeToPort={staticPrimary=8068, status=8068}]]

      2018-09-04 11:11:24.160 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'searchserver' instances: [ServiceInstanceInfo[m_id=1,m_service=searchserver,m_host=server1,m_typeToPort={staticPrimary=8162, status=8162}], ServiceInstanceInfo[m_id=2,m_service=searchserver,m_host=server2,m_typeToPort={staticPrimary=8026, status=8026}], ServiceInstanceInfo[m_id=3,m_service=searchserver,m_host=server3,m_typeToPort={staticPrimary=8869, status=8869}]]

      2018-09-04 11:11:24.160 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'hyper' instances: [ServiceInstanceInfo[m_id=1,m_service=hyper,m_host=server1,m_typeToPort={staticPrimary=8923, status=8923}], ServiceInstanceInfo[m_id=2,m_service=hyper,m_host=server2,m_typeToPort={staticPrimary=8025, status=8025}], ServiceInstanceInfo[m_id=3,m_service=hyper,m_host=server3,m_typeToPort={staticPrimary=8062, status=8062}]]

      2018-09-04 11:11:24.160 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'vizqlserver' instances: [ServiceInstanceInfo[m_id=2:1,m_service=vizqlserver,m_host=server2,m_typeToPort={staticPrimary=8431, status=8431}], ServiceInstanceInfo[m_id=2:0,m_service=vizqlserver,m_host=server2,m_typeToPort={staticPrimary=8874, status=8874}], ServiceInstanceInfo[m_id=3:1,m_service=vizqlserver,m_host=server3,m_typeToPort={staticPrimary=8610, status=8610}], ServiceInstanceInfo[m_id=3:0,m_service=vizqlserver,m_host=server3,m_typeToPort={staticPrimary=8230, status=8230}], ServiceInstanceInfo[m_id=1:0,m_service=vizqlserver,m_host=server1,m_typeToPort={staticPrimary=8784, status=8784}], ServiceInstanceInfo[m_id=1:1,m_service=vizqlserver,m_host=server1,m_typeToPort={staticPrimary=8845, status=8845}]]

      2018-09-04 11:11:24.160 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'tabadmincontroller' instances: [ServiceInstanceInfo[m_id=1,m_service=tabadmincontroller,m_host=server1,m_typeToPort={staticPrimary=8850, status=8850}]]

      2018-09-04 11:11:24.161 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'databasemaintenance' instances: [ServiceInstanceInfo[m_id=1,m_service=databasemaintenance,m_host=server1,m_typeToPort={staticPrimary=8320, status=8320}], ServiceInstanceInfo[m_id=2,m_service=databasemaintenance,m_host=server2,m_typeToPort={staticPrimary=8510, status=8510}], ServiceInstanceInfo[m_id=3,m_service=databasemaintenance,m_host=server3,m_typeToPort={staticPrimary=8226, status=8226}]]

      2018-09-04 11:11:24.161 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'siteimportexport' instances: [ServiceInstanceInfo[m_id=1,m_service=siteimportexport,m_host=server1,m_typeToPort={staticPrimary=8672, status=8672}], ServiceInstanceInfo[m_id=2,m_service=siteimportexport,m_host=server2,m_typeToPort={staticPrimary=8119, status=8119}], ServiceInstanceInfo[m_id=3,m_service=siteimportexport,m_host=server3,m_typeToPort={staticPrimary=8569, status=8569}]]

      2018-09-04 11:11:24.161 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'recommendations' instances: [ServiceInstanceInfo[m_id=2:0:vizportal,m_service=recommendations,m_host=server2,m_typeToPort={grpc=8691}], ServiceInstanceInfo[m_id=3:0:vizportal,m_service=recommendations,m_host=server3,m_typeToPort={grpc=8756}], ServiceInstanceInfo[m_id=1:0:vizportal,m_service=recommendations,m_host=server1,m_typeToPort={grpc=8948}]]

      2018-09-04 11:11:24.162 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'clustercontroller' instances: [ServiceInstanceInfo[m_id=1,m_service=clustercontroller,m_host=server1,m_typeToPort={status=8334}], ServiceInstanceInfo[m_id=2,m_service=clustercontroller,m_host=server2,m_typeToPort={status=8405}], ServiceInstanceInfo[m_id=3,m_service=clustercontroller,m_host=server3,m_typeToPort={status=8548}]]

      2018-09-04 11:11:24.162 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'keychain' instances: [ServiceInstanceInfo[m_id=2:0:vizportal,m_service=keychain,m_host=server2,m_typeToPort={grpc=8023}], ServiceInstanceInfo[m_id=3:0:vizportal,m_service=keychain,m_host=server3,m_typeToPort={grpc=8066}], ServiceInstanceInfo[m_id=1:0:vizportal,m_service=keychain,m_host=server1,m_typeToPort={grpc=8385}]]

      2018-09-04 11:11:24.163 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'maintenance' instances: [ServiceInstanceInfo[m_id=2:0:vizportal,m_service=maintenance,m_host=server2,m_typeToPort={grpc=8584}], ServiceInstanceInfo[m_id=3:0:vizportal,m_service=maintenance,m_host=server3,m_typeToPort={grpc=8601}], ServiceInstanceInfo[m_id=1:0:vizportal,m_service=maintenance,m_host=server1,m_typeToPort={grpc=8055}]]

      2018-09-04 11:11:24.163 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'zookeeper' instances: [ServiceInstanceInfo[m_id=1:0,m_service=zookeeper,m_host=server1,m_typeToPort={primary=8678, staticPrimary=8678, status=8678}]]

      2018-09-04 11:11:24.163 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'licenseservice' instances: [ServiceInstanceInfo[m_id=1:0,m_service=licenseservice,m_host=server1,m_typeToPort={primary=8092, staticPrimary=8092, status=8092}]]

      2018-09-04 11:11:24.163 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'samlservice' instances: [ServiceInstanceInfo[m_id=2,m_service=samlservice,m_host=server2,m_typeToPort={staticPrimary=8411, status=8411}], ServiceInstanceInfo[m_id=3,m_service=samlservice,m_host=server3,m_typeToPort={staticPrimary=8768, status=8768}], ServiceInstanceInfo[m_id=1,m_service=samlservice,m_host=server1,m_typeToPort={staticPrimary=8003, status=8003}]]

      2018-09-04 11:11:24.164 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'extensions' instances: [ServiceInstanceInfo[m_id=2:0:vizportal,m_service=extensions,m_host=server2,m_typeToPort={grpc=8756}], ServiceInstanceInfo[m_id=3:0:vizportal,m_service=extensions,m_host=server3,m_typeToPort={grpc=8625}], ServiceInstanceInfo[m_id=1:0:vizportal,m_service=extensions,m_host=server1,m_typeToPort={grpc=8103}]]

      2018-09-04 11:11:24.164 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'tabadminagent' instances: [ServiceInstanceInfo[m_id=1,m_service=tabadminagent,m_host=server1,m_typeToPort={primary=8558, staticPrimary=8558, status=8558}], ServiceInstanceInfo[m_id=2,m_service=tabadminagent,m_host=server2,m_typeToPort={staticPrimary=8558, status=8558}], ServiceInstanceInfo[m_id=3,m_service=tabadminagent,m_host=server3,m_typeToPort={staticPrimary=8558, status=8558}]]

      2018-09-04 11:11:24.164 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'pgsql' instances: [ServiceInstanceInfo[m_id=routingWorkgroup,m_service=pgsql,m_host=server1,m_typeToPort={primary=8060, status=8060}]]

      2018-09-04 11:11:24.165 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'cacheserver' instances: [ServiceInstanceInfo[m_id=2:0,m_service=cacheserver,m_host=server2,m_typeToPort={staticPrimary=8251, status=8251}], ServiceInstanceInfo[m_id=2:1,m_service=cacheserver,m_host=server2,m_typeToPort={staticPrimary=8705, status=8705}], ServiceInstanceInfo[m_id=3:0,m_service=cacheserver,m_host=server3,m_typeToPort={staticPrimary=8065, status=8065}], ServiceInstanceInfo[m_id=3:1,m_service=cacheserver,m_host=server3,m_typeToPort={staticPrimary=8914, status=8914}], ServiceInstanceInfo[m_id=1:0,m_service=cacheserver,m_host=server1,m_typeToPort={staticPrimary=8251, status=8251}], ServiceInstanceInfo[m_id=1:1,m_service=cacheserver,m_host=server1,m_typeToPort={staticPrimary=8657, status=8657}]]

      2018-09-04 11:11:24.165 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated 'clientfileservice' instances: [ServiceInstanceInfo[m_id=1,m_service=clientfileservice,m_host=server1,m_typeToPort={primary=8625, staticPrimary=8625, status=8034}]]

      2018-09-04 11:11:24.166 -0700 14920 main : INFO  com.tableausoftware.service.discovery.ServiceDiscoveryClient - updated hosts: [server1, server2, server3]

      2018-09-04 11:11:24.169 -0700 14920 main : INFO  com.tableausoftware.config.ServiceRegistrationInfoFile - no registration file found at

      2018-09-04 11:11:24.169 -0700 14920 main : WARN  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Unable to flatten service registration info, because there is no registration file.

      2018-09-04 11:11:24.169 -0700 14920 main : INFO  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Generating workgroup settings: G:\Tableau\Tableau Server\data\tabsvc\config\tabadmincontroller_0.20182.18.0627.2230\workgroup.yml

      2018-09-04 11:11:24.241 -0700 14920 main : INFO  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Generating connection settings: G:\Tableau\Tableau Server\data\tabsvc\services\connections.yml, G:\Tableau\Tableau Server\data\tabsvc\services\connections.properties

      2018-09-04 11:11:24.241 -0700 14920 main : INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting

      2018-09-04 11:11:24.242 -0700 14920 main : INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=server1:8151 sessionTimeout=120000 watcher=org.apache.curator.ConnectionState@64231d5d

      2018-09-04 11:11:24.307 -0700 14920 main-SendThread(server1.cloud.org:8151) : INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server server1.cloud.org/172.27.163.106:8151. Will not attempt to authenticate using SASL (unknown error)

      2018-09-04 11:11:24.308 -0700 14920 main-SendThread(server1.cloud.org:8151) : INFO  org.apache.zookeeper.ClientCnxn - Socket connection established to server1.cloud.org/172.27.163.106:8151, initiating session

      2018-09-04 11:11:24.387 -0700 14920 main-SendThread(server1.cloud.org:8151) : INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server server1.cloud.org/172.27.163.106:8151, sessionid = 0x100000072820f6a, negotiated timeout = 120000

      2018-09-04 11:11:24.392 -0700 14920 main-EventThread : INFO  org.apache.curator.framework.state.ConnectionStateManager - State change: CONNECTED

      2018-09-04 11:11:24.448 -0700 14920 Curator-Framework-0 : INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting

      2018-09-04 11:11:24.471 -0700 14920 main : INFO  org.apache.zookeeper.ZooKeeper - Session: 0x100000072820f6a closed

      2018-09-04 11:11:24.475 -0700 14920 main-EventThread : INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100000072820f6a

      2018-09-04 11:11:24.860 -0700 14920 main : INFO  com.tableausoftware.tabadmin.webapp.TabadminController - The configuration for tabadmincontroller has been updated

      2018-09-04 11:11:24.860 -0700 14920 main : INFO  com.tableausoftware.tabadmin.agent.zookeeper.WriteThroughTreeCacheWorker - Cleanup called

      2018-09-04 11:11:24.865 -0700 14920 main : INFO  com.tableausoftware.tabadmin.agent.zookeeper.ConnectionHolder - Closing Zookeeper client instance org.apache.curator.framework.imps.CuratorFrameworkImpl@216c303d in state STARTED.

      2018-09-04 11:11:24.865 -0700 14920 Curator-Framework-0 : INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting

      2018-09-04 11:11:24.873 -0700 14920 main : INFO  org.apache.zookeeper.ZooKeeper - Session: 0x100000072820f67 closed

      2018-09-04 11:11:24.874 -0700 14920 ZKWorker-ScheduledTask-0-EventThread : INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100000072820f67

      2018-09-04 11:11:24.874 -0700 14920 Curator-Framework-0 : INFO  org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting

      2018-09-04 11:11:24.878 -0700 14920 main : INFO  org.apache.zookeeper.ZooKeeper - Session: 0x100000072820f64 closed

      2018-09-04 11:11:24.879 -0700 14920 main-EventThread : INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100000072820f64

      2018-09-04 11:11:34.895 -0700 15396 main : INFO  com.tableausoftware.tabadmin.webapp.TabadminController - Service directory is G:\Tableau\Tableau Server\data\tabsvc\services\tabadmincontroller_0.20182.18.0627.2230