7 Replies Latest reply on Mar 17, 2019 10:49 PM by Anuj Bansal

    Tableau Server restarting on its own

    Jae Park

      Hello,

       

      Our Tableau Server Cluster randomly crashes and restarts on its own after certain error logs appear.

       

      -0500 pool-22-thread-1   WARN : com.tableausoftware.tdfs.common.thriftauth.ClientIPWhitelistProcessor -

      • org.apache.thrift.TApplicationException: Unauthorized client IP /xxxx

                      at com.tableausoftware.tdfs.common.thriftauth.ClientIPWhitelistProcessor.process(ClientIPWhitelistProcessor.java:62)

                      at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                      at java.lang.Thread.run(Thread.java:745)

      -0500 pool-22-thread-1   ERROR : org.apache.thrift.server.TThreadPoolServer - Thrift error occurred during processing of message.

      • org.apache.thrift.TApplicationException: Unauthorized client IP /xxxx

                      at com.tableausoftware.tdfs.common.thriftauth.ClientIPWhitelistProcessor.process(ClientIPWhitelistProcessor.java:62)

                      at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)

                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

                      at java.lang.Thread.run(Thread.java:745)

       

      Then we get

      -0500 Curator-PathChildrenCache-3 ERROR : com.tableausoftware.cluster.postgres.PostgresController - Current Postgres master node disconnected

      and

      Restarting dead component 'Tableau Server Cluster Controller'.

       

      I am wondering how this unauthorized client is making the server reboot.

       

      Thank you in advance

        • 1. Re: Tableau Server restarting on its own
          Carisa Chang

          Hi Jae,

           

          Please open a case with Tableau Technical Support to investigate - they will need to take a deeper look at your logs to determine what the cause of ClusterController restarting is, and whether the error message you've posted above is related. Thank you!

          • 2. Re: Tableau Server restarting on its own
            Mark McGhee

            Hi Jae,

             

            It's probably a good idea to open a support case as Carisa suggested.  I have seen at a few customer locations where the security team is doing a scan of Tableau Server typically by trying to add in an unauthorized node.  So in addition to opening a case it may not be a bad idea to reach out to your IT/Security team to see if your Tableau cluster is part of a scan.

            • 3. Re: Tableau Server restarting on its own
              prasanth.avirnene.0

              Hi,

               

              Any update on this ?  we are having the same issue.

               

              Thanks.

              • 4. Re: Tableau Server restarting on its own
                Justin D'Cruze

                I raised a case about this a few months ago.

                 

                As Mark mentioned, the issue can occur during specific vulnerability scans/tests that your security team is running periodically.

                 

                Long story short, there was an enhancement request raised to fix this but you'll need to chase your account manager to get it prioritised. I ended up just asking our security team to put exceptions on our Tableau Server machines (which is not ideal from a security standpoint).

                • 5. Re: Tableau Server restarting on its own
                  Mark McGhee

                  Not to say it will be everyone's case but what we've seen is if the security scan continues to try and add in an authorized node repeatedly this will result in a number of exceptions and this will eat into the Java Heap size and eventually the process runs out of memory which kills it and hence it restarts.  But if you're IT team says they're not scanning Tableau Server and/or you would just like confirmation it's best to open a support case.

                  • 6. Re: Tableau Server restarting on its own
                    Jeffrey Lutomski

                    What log are you displaying fragments from? clustercontroller.log?

                     

                    We had some random restarts last night too:

                     

                     

                    2018-12-11 18:28:42.213 -0500 pool-19-thread-2   ERROR : com.tableausoftware.cluster.http.HttpServiceMonitor - IOException connecting to HTTP server at http://localhost:80/favicon.ico

                    java.net.SocketTimeoutException: Read timed out

                    at java.net.SocketInputStream.socketRead0(Native Method)

                    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)

                    at java.net.SocketInputStream.read(SocketInputStream.java:171)

                    at java.net.SocketInputStream.read(SocketInputStream.java:141)

                    at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:139)

                    at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:155)

                    at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:284)

                    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)

                    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)

                    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)

                    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)

                    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)

                    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)

                    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)

                    at org.apache.http.impl.execchain.MinimalClientExec.execute(MinimalClientExec.java:213)

                    at org.apache.http.impl.client.MinimalHttpClient.doExecute(MinimalHttpClient.java:107)

                    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)

                    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)

                    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)

                    at com.tableausoftware.tabadmin.status.HttpStatusMonitor.getStatus(HttpStatusMonitor.java:90)

                    at com.tableausoftware.cluster.http.HttpServiceMonitor.getState(HttpServiceMonitor.java:97)

                    at com.tableausoftware.cluster.notification.PolledNotificationSource.run(PolledNotificationSource.java:105)

                    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

                    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)

                    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)

                    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)

                    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

                    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

                    at java.lang.Thread.run(Thread.java:748)

                    • 7. Re: Tableau Server restarting on its own
                      Anuj Bansal

                      so how we can avoid this issue if there is mandatory requirement by IT team to scan all the window host present in the Firm?