1 2 Previous Next 20 Replies Latest reply on Feb 15, 2016 2:49 PM by Jeff Strauss

    Tableau 9.1.2 upgrade - error with distributed environment

    Jeff Strauss

      Have you run into this and what have you done about it?  It's a fresh install.

      worker.png

       

       

      Upon digging into the tabadmin log on the worker I find the following:

      worker2.png

       

       

      Within task scheduler there's the task that is queued, and if I kick it off at the right time, then I get away with successfully installing.

       

      restart.png

        • 1. Re: Tableau 9.1.2 upgrade - error with distributed environment
          John Kuo

          Jeff,

           

          Try the following steps with an admin user on the machine where Tableau Server is installed:

           

          1. Stop Tableau Server

          2. Create a Backup

          3. Uninstall Tableau Server

          4. Reboot the Machine

          5. Install the new version of Tableau Server

           

          The key step is to reboot the machine after uninstalling Tableau Server. This allow the system to stop all processes used by Tableau Server, before installing the new version.


          Best,


          John

          • 2. Re: Tableau 9.1.2 upgrade - error with distributed environment
            Jeff Strauss

            too late.  I've already installed 9.1.2 by a workaround where I kickoff the task on the workers manually.  The problem isn't with processes in use.  The issue seems to be the start task time.  In the screenprint above "WARNING: Task may not run because /ST is earlier than current time."

            • 3. Re: Tableau 9.1.2 upgrade - error with distributed environment
              Jeff Strauss

              server admin experts, any other thoughts?  It seems like a bug to me as the task is getting queued because the time is already past.  But it also seems easily fixable.

              • 4. Re: Tableau 9.1.2 upgrade - error with distributed environment
                Jeff Strauss

                this occurs again with 9.2.0, for now I am using the workaround approach.

                • 5. Re: Tableau 9.1.2 upgrade - error with distributed environment
                  Jeff Strauss

                  this also occurs with 9.2.3

                   

                   

                  Ramesh Gowrishankar  any thoughts?

                  • 6. Re: Tableau 9.1.2 upgrade - error with distributed environment
                    Ramesh Gowrishankar

                    Thanks for bringing this to my attention Jeffrey. Let me follow up with you offline and we can update this thread.

                    • 7. Re: Tableau 9.1.2 upgrade - error with distributed environment
                      Joshua Swickard

                      Hi Jeffrey,

                       

                      Blake and i dug into this a bit more and this is what we found:

                       

                      Worker1 calls schtasks to create a task:

                       

                      schtasks /Create /SC ONCE /ST 00:00 /TN "Restart tabadmsvc" /F /RU "SYSTEM" /TR "cmd.exe /c call \"<InstallDIrectory>r<templocation>/restart.cmd\" 2>&1 > \"<install directory>/logs/restart.log\""

                      WARNING: Task may not run because /ST is earlier than current time.

                       

                      This warning turns out to be expected.  That is because schtasks doesn't have the concept of 'run now'.  Instead you create a run once task and then call /run to run it.

                       

                      We call /run to run the task:

                      'cmd.exe /c call schtasks /Run /TN "Restart tabadmsvc"' from C:/Windows/System32

                       

                      We then start checking to see if the service restarted:

                       

                      2016-01-28 09:50:41.238 -0600_DEBUG_:_:_pid=___user=__request=__ ====>> <script> 9.2: restarted starting at Thu Jan 28 09:50:41 -0600 2016 <<====

                      2016-01-28 09:50:41.238 -0600_DEBUG_:_:_pid=___user=__request=__ restarted = false

                      2016-01-28 09:50:41.238 -0600_DEBUG_:_:_pid=___user=__request=__ restarted finished at Thu Jan 28 09:50:41 -0600 2016 <<====

                      2016-01-28 09:50:44.254 -0600_DEBUG_:_:_pid=___user=__request=__ ====>> <script> 9.2: restarted starting at Thu Jan 28 09:50:44 -0600 2016 <<====

                       

                      We do this check six times before we log:

                      2016-01-28 09:51:20.275 -0600_DEBUG_:_:_pid=___user=__request=__ restarted finished at Thu Jan 28 09:51:20 -0600 2016 <<===

                       

                      Comparing this to the other worker, we create the task:

                       

                       

                      schtasks /Create /SC ONCE /ST 00:00 /TN "Restart tabadmsvc" /F /RU "SYSTEM" /TR "cmd.exe /c call \"<InstallDirectory>\<templocation>/restart.cmd\" 2>&1 > \"<InstallDirectory>/logs/restart.log\""

                       

                      WARNING: Task may not run because /ST is earlier than current time.

                       

                      SUCCESS: The scheduled task "Restart tabadmsvc" has successfully been created.

                       

                      We call /run to run the task:

                       

                      2016-01-28 09:50:41.910 -0600_DEBUG_:_:_pid=___user=__request=__ running command: 'cmd.exe /c call schtasks /Run /TN "Restart tabadmsvc"' from C:/Windows/System32

                      2016-01-28 09:50:41.910 -0600_DEBUG_:_:_pid=___user=__request=__ restart finished at Thu Jan 28 09:50:41 -0600 2016 <<====

                       

                      We immediately finish without logging restarted = false at all because the restart is already done.

                       

                       

                      Based on this, the logic to create the task and then run it with /run is likely correct. 

                       

                       

                      I have a few questions and ideas for next steps:

                       

                      Questions:

                      1. Is it always the same worker you have to run the task on?
                      2. When you open taskschd.msc, before running the task
                        • Are there any errors in "Last Run Result" ? (Like access is denied?)?
                        • Is there a Last Run time?

                       

                       

                      At this time I think it is appropriate to create a support case if we want to continue to troubleshoot this issue.  Please let me know if you'd like to do that and I can create one for you.

                       

                      Josh

                      • 8. Re: Tableau 9.1.2 upgrade - error with distributed environment
                        Jeff Strauss

                        yes, please create a support case.  You already have the logs, so please submit.  And even though the warning may be expected, it's more than a "trivial superfluous warning" when it causes the installation to fail because the primary never received a sufficient response back and shows the worker initialization failed.

                         

                        worker (1).png

                         

                        Here are answers to your questions.

                         

                        1. So far, I have run into this issue on dev (two workers) upon upgrading to 9.1.x and 9.2.x.  We will be upgrading prod this weekend and I can report back as to whether it occurred here.

                         

                        2. No errors, the task is queued.  I do not know if there is a last runtime, but if I manually run the task by right clicking within task scheduler and clicking on run, then it runs.  I need to do this twice on each worker during each upgrade cycle.

                        • 9. Re: Tableau 9.1.2 upgrade - error with distributed environment
                          Joshua Swickard

                          Hi Jeffrey,

                           

                          I definitely wasn't saying that the result of a failed install was trivial and superfluous .  Just explaining why this specifically "WARNING: Task may not run because /ST is earlier than current time." is expected because of how schtasks works, and, is not itself the cause of the initialization failure. 

                           

                          That cause appears to be the next call to schtasks /run not completing the execution of the task.

                           

                          I'll create a support case with the existing logs and reach out on next steps.

                           

                          Thanks,

                          Josh

                          • 10. Re: Tableau 9.1.2 upgrade - error with distributed environment
                            Jeff Strauss

                            understood.  (to help out) I just ran a test on worker1 (and also on my own pc) and got the result of the task being created but not running.  What happens when you try to do the same?

                             

                            schtasks /Create /SC ONCE /ST 00:00 /TN "test2" /F /RU "SYSTEM" /TR "cmd.exe /c \"echo hello\" 2>&1"

                             

                             

                            • 11. Re: Tableau 9.1.2 upgrade - error with distributed environment
                              Joshua Swickard

                              Hi Jeffrey,

                               

                              I ran an upgrade yesterday to validate this, these are the results of the task correctly running:

                               

                               

                              If you read down the tabadmin log a bit more you'll see that is because we call this to run the task:

                               

                              'cmd.exe /c call schtasks /Run /TN "Restart tabadmsvc"' from C:/Windows/System32

                               

                              Can you try that ('cmd.exe /c call schtasks /Run /TN "Test2"') from c:\windows\system32 logged in as the run as user after creating the task?

                              • 12. Re: Tableau 9.1.2 upgrade - error with distributed environment

                                Hi Jeffrey, I ran a similar test as you and saw the same warning on task creation, but as @Josh S described above, that's expected. It's expected because we create the task with the '/ST 00:00' start time which won't run, but it isn't executed until the 'cmd.exe /c call schtasks /Run /TN <taskname>' call.

                                 

                                Using your example, it would happen like this:

                                 

                                1. The scheduled task is created: schtasks /Create /SC ONCE /ST 00:00 /TN "test2" /F /RU "SYSTEM" /TR "cmd.exe /c \"echo hello\" 2>&1"
                                  • This will successfully create the task, and will naturally throw the same warning we've both seen in our tests due to the time being earlier than the current time, and it won't run yet.
                                  • The /ST parameter is only added here because it's required for the /SC ONCE value.
                                  • FWIW, Josh looked into healthy, successful upgrades and saw this same behavior as well.
                                2. Task is executed: schtasks /Run /TN "test2"
                                  • Task executes successfully.

                                 

                                Does that help? It seems like the problem isn't step #1 like we originally thought, but that for some reason step #2 isn't succeeding and is causing the worker initialization failure. Josh and our Support team will help you get that sorted!

                                • 13. Re: Tableau 9.1.2 upgrade - error with distributed environment
                                  Jeff Strauss

                                  do you run your test on windows server 2008 or windows server 2012 R2?

                                   

                                   

                                  here's what I ran and the results with Windows 2012 R2.

                                  schtasks /Create /SC ONCE /ST 00:00 /TN "test100" /F /RU "SYSTEM" /TR "cmd.exe /c \"echo hello\" 2>&1 > \"G:\Scripts\ServerAdmin\Logs\test.log\""

                                  schtasks /Run /TN "test100"

                                   

                                   

                                   

                                  It never actually runs.  the task is created at 9:47:20, it's triggered at 9:47:36, but immediately is queued.

                                   

                                  • 14. Re: Tableau 9.1.2 upgrade - error with distributed environment
                                    Joshua Swickard

                                    Launch request queued/Task triggered by user  appears to be the key here. Task scheduler is for some reason not creating the task process for you.

                                     

                                    This is what i get:

                                     

                                    C:\test2>type restart.cmd

                                    echo "restart.cmd has executed" >> results.txt

                                     

                                     

                                    C:\test2>schtasks /Create /SC ONCE /ST 00:00 /TN "runme2" /F /RU "SYSTEM" /TR "cmd.exe /c call \"c:\test2\restart.cmd\" 2>&1 > \"c:\test2\results3.txt\""

                                    WARNING: Task may not run because /ST is earlier than current time.

                                    SUCCESS: The scheduled task "runme2" has successfully been created.

                                     

                                     

                                    C:\test2>schtasks /run /TN "runme2"

                                    SUCCESS: Attempted to run the scheduled task "runme2".

                                     

                                     

                                    C:\test2>type results3.txt

                                     

                                     

                                    C:\Windows\system32>echo "restart.cmd has executed"  1>>results.txt

                                     

                                    The task triggered by user (the 4th event / 2nd from the top) should have spawned a create task process like this (event id 129):

                                     

                                     

                                    Can you try changing the task owner to yourself / only run when user is logged in or yourself and bind a password before calling /run? I'm wondering if a policy is blocking the task from running in the system context or possibly a higher UAC level?

                                    1 2 Previous Next