8 Replies Latest reply on Mar 22, 2019 5:08 AM by Ciara Brennan

    Tableau Upgrade from 2018.2.6 to 2018.3.2 failing at 33%

    swathi kadem

      Hi ,

       

      I am working on tableau upgrade . I was able to successfully upgrade from 10.5.11 to 2018.2.6.

      But when i try to do an upgrade from 2018.2.6 to 2018.3.2 , its failing at 33% (Installing backup services failed)

      I thought its happening because i had the backup file from my previous upgrade in files folder.

      So I removed the backup file and tried again, its still  the same error.

       

       

      Please let me know if anyone faced this issue and how you solved it.

      Looking forward for your reply.

       

      Configs:

      - Windows server 2016 (AWS )

      - Multi-node ( 2 clusters)

      - Both the nodes in same subnet

      - Created a admin user called tableau and gave permission to D and C drive both drives

      - Have set Hostname (ip-10-xx-x-xx)

       

      Error: (logs from Tabadmincontrollernode1 file)

       

      2019-02-12 11:09:01.029 -0500  qtp987487474-75 : INFO  com.tableausoftware.service.discovery.ServiceInfoCache - registered com.tableausoftware.config.LegacyPgsqlRegistry@15adafea as registry for pgsql

      2019-02-12 11:09:01.035 -0500  qtp987487474-75 : INFO  com.tableausoftware.config.ServiceRegistrationInfoFile - no registration file found at

      2019-02-12 11:09:01.035 -0500  qtp987487474-75 : WARN  com.tableausoftware.tabadmin.configuration.builder.AppConfigurationBuilder - Unable to flatten service registration info, because there is no registration file.

      2019-02-12 11:09:01.062 -0500  Curator-LeaderSelector-0 : INFO  com.tableausoftware.tabadmin.webapp.asyncjobs.AsyncJobService - Running job 16 of type PostUpgradeJob

      2019-02-12 11:09:01.068 -0500  Curator-LeaderSelector-0 : INFO  com.tableausoftware.tabadmin.webapp.asyncjobs.AsyncJobService - Waiting up to 1200 seconds for job 16 to finish.

      2019-02-12 11:09:01.081 -0500  pool-11-thread-1 : INFO  com.tableausoftware.tabadmin.webapp.asyncjobs.PostUpgradeJob - Running post-upgrade on the following nodes: node1, node2

      2019-02-12 11:09:01.115 -0500  pool-11-thread-1 : INFO  com.tableausoftware.tabadmin.webapp.restore.RestoreServiceInstalls - Waiting for {node1=[backuprestore_0.20183.18.1214.0808:ENABLED], node2=[backuprestore_0.20183.18.1214.0808:ENABLED]} services to start.

      2019-02-12 11:09:01.116 -0500  pool-11-thread-1 : INFO  com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher - Waiting for 2 services to start with timeout: 600 seconds

      2019-02-12 11:09:01.117 -0500  pool-11-thread-1 : INFO  com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher - There are 1 service instances on node node1 not in the desired state: backuprestore_0 with status STATUS_UNAVAILABLE and current deployment state DISABLED

      2019-02-12 11:09:02.143 -0500  pool-11-thread-1 : INFO  com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher - There are 1 service instances on node node1 not in the desired state: backuprestore_0 with status STATUS_UNAVAILABLE and current deployment state ENABLING

      2019-02-12 11:09:10.150 -0500  pool-11-thread-1 : INFO  com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher - There are 1 service instances on node node1 not in the desired state: backuprestore_0 with status STATUS_UNAVAILABLE and current deployment state ENABLED

      2019-02-12 11:19:01.117 -0500  pool-11-thread-1 : ERROR com.tableausoftware.tabadmin.webapp.impl.status.ServiceStatusWatcherIterationHelper - Timed out waiting for statusStore notification

      2019-02-12 11:19:01.117 -0500  pool-11-thread-1 : ERROR com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher - Not all requested services have started

      2019-02-12 11:19:01.117 -0500  pool-11-thread-1 : ERROR com.tableausoftware.tabadmin.webapp.asyncjobs.JobStepRunner - Running step EnableServices failed

      com.tableausoftware.tabadmin.webapp.exceptions.ServiceOperationTimeoutException

      at com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher.waitForServicesToStartInternal(ServiceStatusWatcher.java:471)

      at com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher.waitForServiceInstancesToStart(ServiceStatusWatcher.java:174)

      at com.tableausoftware.tabadmin.webapp.impl.ServiceStatusWatcher.waitForServiceInstancesToStart(ServiceStatusWatcher.java:162)

      at com.tableausoftware.tabadmin.webapp.restore.RestoreServiceInstalls.enableMaintenanceServices(RestoreServiceInstalls.java:254)

      at com.tableausoftware.tabadmin.webapp.asyncjobs.PostUpgradeJob.enableServicesStep(PostUpgradeJob.java:159)

      at com.tableausoftware.tabadmin.webapp.asyncjobs.PostUpgradeJob.lambda$enableServices$1(PostUpgradeJob.java:152)

      at com.tableausoftware.tabadmin.webapp.asyncjobs.JobStepRunner.runStep(JobStepRunner.java:266)

      at com.tableausoftware.tabadmin.webapp.asyncjobs.JobStepRunner.runStep(JobStepRunner.java:92)

      at com.tableausoftware.tabadmin.webapp.asyncjobs.JobStepRunner.runStep(JobStepRunner.java:136)

      at com.tableausoftware.tabadmin.webapp.asyncjobs.JobStepRunner.runStep(JobStepRunner.java:67)

      at com.tableausoftware.tabadmin.webapp.asyncjobs.PostUpgradeJob.enableServices(PostUpgradeJob.java:146)

      at com.tableausoftware.tabadmin.webapp.asyncjobs.PostUpgradeJob.run(PostUpgradeJob.java:122)

      at com.tableausoftware.tabadmin.webapp.asyncjobs.AsyncJobService.lambda$runJob$1(AsyncJobService.java:202)

      at java.util.concurrent.FutureTask.run(FutureTask.java:266)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

      2019-02-12 11:19:01.120 -0500  Curator-LeaderSelector-0 : ERROR com.tableausoftware.tabadmin.webapp.asyncjobs.AsyncJobService - Error running job 16 of type PostUpgradeJob

      java.util.concurrent.ExecutionException: com.tableausoftware.tabadmin.webapp.exceptions.ServiceOperationTimeoutException

      at java.util.concurrent.FutureTask.report(FutureTask.java:122)

      at java.util.concurrent.FutureTask.get(FutureTask.java:206)

      at com.tableausoftware.tabadmin.webapp.asyncjobs.AsyncJobService.runJob(AsyncJobService.java:210)

      at com.tableausoftware.tabadmin.webapp.asyncjobs.AsyncJobService.leaderCallback(AsyncJobService.java:113)

      at com.tableausoftware.tabadmin.agent.zookeeper.LeaderWorker$LeaderListener.takeLeadership(LeaderWorker.java:94)

      at org.apache.curator.framework.recipes.leader.LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:559)

      at org.apache.curator.framework.recipes.leader.LeaderSelector.doWork(LeaderSelector.java:421)

      at org.apache.curator.framework.recipes.leader.LeaderSelector.doWorkLoop(LeaderSelector.java:466)

      at org.apache.curator.framework.recipes.leader.LeaderSelector.access$100(LeaderSelector.java:65)

      at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:246)

      at org.apache.curator.framework.recipes.leader.LeaderSelector$2.call(LeaderSelector.java:240)

      at java.util.concurrent.FutureTask.run(FutureTask.java:266)

      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

      at java.util.concurrent.FutureTask.run(FutureTask.java:266)

      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

      at java.lang.Thread.run(Thread.java:748)

       

      Upgrade:

      D:\Tableau Server\packages\scripts.20183.18.1214.0808>upgrade-tsm.cmd -u tableau

      Upgrading Tableau Server to version 20183.18.1214.0808. See log file at D:\Tableau Server\logs\app-upgrade.log for progress.

      Install directory D:\Tableau Server

      Config file: "D:\Tableau Server\data\tabsvc\config\workgroup.yml"

      Password:

      Operation failed.

      An error occurred during installation.

      Unable to login

      Password:

      Validating Tableau Server version.

      Verifying that the new packages have been installed on all cluster nodes.

      Stopping service...

      Verifying Tableau Server is stopped.

      Verifying that there is only one deployed Tableau Server Coordination Service ensemble.

      Verifying that Tableau Server has been initialized.

      Verifying licensing state.

      Reconnecting to asynchronous job...

      Job id is '7', timeout is 60 minutes.

      6% - Retrieving the topology to deploy.

      13% - Retrieving the configuration to deploy.

      20% - Validating the new topology.

      26% - Determining if server needs to be started.

      33% - Disabling all services.

      40% - Waiting for the services to stop.

      46% - Updating nodes to new topology.

      53% - Waiting for topology to be applied.

      60% - Updating nodes to new configuration.

      66% - Disabling all services.

      73% - Waiting for the services to stop.

      80% - Reconfiguring services.

      86% - Waiting for services to reconfigure.

      93% - Enabling all services.

      100% - Waiting for the services to start.

      Backing up repository.

      Upgrading TSM services.

      Waiting for upgraded TSM services to start.

      Updating configuration.

      Reconnecting to asynchronous job...

      Job id is '9', timeout is 10 minutes.

      Updating Tableau Server Coordination Service.

      Updating Tableau Server Client File Service.

      Generating new credentials.

      Reconnecting to asynchronous job...

      Job id is '11', timeout is 10 minutes.

      33% - Generating passwords.

      66% - Promoting configuration.

      100% - job.reconfigure.step.wait_for_configure_and_start

      Updating remaining services.

      Restoring repository.

      Updating repository version in Tableau Server Coordination Service.

      Updating repository.

      Reconnecting to asynchronous job...

      Job id is '12', timeout is 120 minutes.

      11% - Enabling the database services.

      22% - Waiting for the database services to enable.

      33% - Putting the repository into local trust mode.

      44% - Running migrations against the primary database.

      55% - Setting the next active repository.

      66% - Taking the repository out of local trust mode.

      77% - Disabling database services.

      88% - Waiting for database services to disable.

      100% - Upgrading database

      Updating asset keys.

      Reconnecting to asynchronous job...

      Job id is '13', timeout is 120 minutes.

      100% - Checking if asset key upgrade is needed.

      Rebuilding search index.

      Reconnecting to asynchronous job...

      Job id is '14', timeout is 120 minutes.

      Reconnecting to asynchronous job...

      Job id is '15', timeout is 120 minutes.

      25% - Enabling the services required for indexing.

      50% - Connecting to Vizportal Maintenance.

      75% - Rebuilding the search index.

      100% - Disabling the services used for indexing.

      Updating data directory version.

      Running post-upgrade steps on all cluster nodes.

      Reconnecting to asynchronous job...

      Job id is '16', timeout is 20 minutes.

      33% - Installing backup services failed.

      Failed a post-upgrade step.

      An error occurred during installation.

      Failed to run async job during installer operation.