6 Replies Latest reply on Aug 13, 2018 2:39 AM by Iwona Wiktorowska

    How to drop backgrounder.exe if it is stuck

    Egor Larin

      Hello Server admins.

       

       

      First of all, i want to say thank you to Toby Erkson for published guides:

      Stopping a 8.x Tableau Server Background Task (Windows)

      Stopping a 9.x Tableau Server Background Task (Windows)

       

      Please vote for the Idea here (also voted):

      https://community.tableau.com/ideas/1108

       

      I want to share my little story.

      We have Tableau Server 10.0.1.

      4 nodes, separate node for 9 backgrounder processes.

       

      We had spotted Tableau Server status page showed that one backgrounder was down (backgrounder 8:8258)

      It was in runtime for 38K sec (10 hours). Don't know why (we have limit to 7200 sec). Based on Tableau Server admin view (Background Tasks for Extracts) that was last activity on that backgrounder.

      Not sure about description error - that backgrounder was still hanging and didn't go live

      ts_report.png

      On Tableau Server i found in Task Manager that one of backgrounder process had no activity:

      • Columns Threads, IO Reads, IO Writes, IO Read Bytes IO Write Bytes were motionless.

      Our assumption was after that long running task (extract update) it was frozen and stuck.

       

      We were trying to identify which backgrounder.exe is for which TS process is. Main idea was just to kill it.

      I explored your docs and did the same, but:

      • Version 10 has another structure
      • Task Manager cannot show full line for Command Line column
      • We didn't receive any notification from Tableau Server that it is down (probably because it wasn't really down, it just stuck)

      task_manager.png

       

      Logs from Splunk analysis had that error for vizportal:

      2016-12-28 09:20:48.927 +0000 (Production,*username*,90ukkyOzIJUeHLR1b2Q7ybl0Jb5oPFXo,WGODlgohEQsAAC4kBcEAAAPL) catalina-exec-489 : WARN  com.tableausoftware.domain.admin.BackgrounderWorkerStatusHelper - Connection error to url: http:/node1:80/admin/status_backgrounder/3/8258 with exception: Read timed out

       

      2016-12-28 10:03:49.792 +0000 (Production,*username*,Idkxx2oJJEfSzUQVa0ejbB4GEzoJECks,WGON2wohEQsAAC4kXbUAAAOX) catalina-exec-475 : WARN  com.tableausoftware.domain.admin.BackgrounderWorkerStatusHelper - Error response code '503' from status url: http://*backgrounder_node*:80/admin/status_backgrounder/3/8258

       

      After manual killing - background process was automatically picked up and started refreshing extracts.

       

      Ihpe that will help somebody to solve the same problem.

       

      Cheers.

        • 1. Re: How to drop backgrounder.exe if it is stuck

          Hey Egor,

           

          Thanks for sharing. This definitely highlights the need to be able to kill individual refresh tasks. I've voted up the idea and hopefully others will as well :-)

           

          -Diego

          • 2. Re: How to drop backgrounder.exe if it is stuck
            Toby Erkson

            Yep, I voted a long time ago and we're still waiting...

             

            What I don't understand is who in their right mind would down-vote this Idea?  Would be really nice if such folks provided a reason why (hellooo!).

            • 3. Re: How to drop backgrounder.exe if it is stuck
              Jackson Kibler

              I have a similar but different variation on the issue. I have a user who manually kicked off an extract refresh while that same extract was in process for a scheduled refresh. So I have two refreshes of the same extract going on simultaneously. That alone I'm uncertain of the impact, but to bring it to this topic, I wanted to end the latter job that was manually performed. We have a 2 Worker with 2 Backgrounder each config. I followed the task manager instructions and successfully ended the backgrounder task on Worker 2. Well Worker 1 just picked it up and I still have both refreshes in process.

               

              In your case, automatically restarting the killed refresh was good, but I'm trying to end it

              • 4. Re: How to drop backgrounder.exe if it is stuck
                Chris Ellis

                Did anyone find a resolution for Tableau 10?

                We are on 10.5 and are unable to kill Subscription notifications.

                 

                 

                They just hang till the runtime hits 90000

                • 5. Re: How to drop backgrounder.exe if it is stuck
                  Paul Furman

                  This is what you can do to kill a rogue/stuck/dead process:

                   

                  Find the ProcessID of the backgrounder in Tableau Server Status.

                   

                  Run the following command in the Command Prompt as Administrator:

                   

                  netstat -a -b -n -o > netstat.txt

                  Search netstat.txt for the ProcessID in question, the last column will contain the PID that you need to kill in Windows.

                  [backgrounder.exe]

                    TCP    127.0.0.1:8252         127.0.0.1:51509        CLOSE_WAIT      28468

                  Alternatively, run the following command in the Command Prompt, searching for the rogue ProcessID

                  netstat -ano | findstr 8252

                    TCP    0.0.0.0:8252           0.0.0.0:0              LISTENING       14344

                    TCP    127.0.0.1:8252         127.0.0.1:57174        ESTABLISHED     14344

                    TCP    127.0.0.1:8252         127.0.0.1:57196        ESTABLISHED     14344

                  Voila!

                  • 6. Re: How to drop backgrounder.exe if it is stuck
                    Iwona Wiktorowska

                    For the backgounder, I want to kill one long running data source refresh, but in the "Background tasks for extracts", when I click on the tooltip with information on that job, I have information like this: localhost:0:2 or localhost:0.4. How do I map this to the port numbers on the status page? Is there any connection?