2 Replies Latest reply on Jun 17, 2019 12:42 AM by Andrzej Prusinowski

    Zookeeper ConnectionLossException

    Andrzej Prusinowski

      Hi,

       

      I have an AWS instance running Tableau 2019.1.3 and an AMI created out of it. I'm trying to launch a new instance with that AMI (to test the performance on other instance type) and I'm getting a following error when running `tsm status`:

       

      Could not connect to TSM Controller at '<ip>:8850'

       

      When I open https://<ip>:8850 in the browser, here's what I see:

       

      Whitelabel Error Page

      This application has no explicit mapping for /error, so you are seeing this as a fallback.

       

      Fri May 24 10:39:18 UTC 2019

      There was an unexpected error (type=Internal Server Error, status=500).

      com.tableausoftware.tabadmin.webapp.exceptions.RestException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss

       

      Any ideas what it's trying to connect to and why it can't?

       

      The log file in /var/opt/tableau/tableau_server/data/tabsvc/logs/tabadmincontroller gives me the following:

       

      2019-05-24 10:39:18.287 +0000 1536 qtp20039101-31 : WARN  org.eclipse.jetty.servlet.ServletHandler - /

      com.tableausoftware.tabadmin.webapp.exceptions.RestException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss

              at com.tableausoftware.tabadmin.webapp.impl.ClusterPropertiesService.getClusterProperties(ClusterPropertiesService.java:240)

              at com.tableausoftware.tabadmin.webapp.impl.ClusterPropertiesService.getInitializationState(ClusterPropertiesService.java:84)

              at com.tableausoftware.tabadmin.webapp.InitializeAPIFilter.isAllowed(InitializeAPIFilter.java:149)

              at com.tableausoftware.tabadmin.webapp.InitializeAPIFilter.doFilterInternal(InitializeAPIFilter.java:165)

              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)

              at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)

              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)

              at com.tableausoftware.tabadmin.webapp.HttpMethodOverrideFilter.doFilterInternal(HttpMethodOverrideFilter.java:28)

              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)

              at org.springframework.web.multipart.support.MultipartFilter.doFilterInternal(MultipartFilter.java:122)

              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)

              at org.springframework.boot.actuate.trace.WebRequestTraceFilter.doFilterInternal(WebRequestTraceFilter.java:105)

              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)

              at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)

              at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

              at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

              at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

              at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

              at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

              at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

              at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

              at com.tableausoftware.tabadmin.webapp.auth.CsrfFilter.doFilter(CsrfFilter.java:75)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

              at com.tableausoftware.tabadmin.webapp.auth.RestAuthenticationFilter.doFilter(RestAuthenticationFilter.java:68)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

              at com.tableausoftware.tabadmin.webapp.auth.CookieAuthenticationFilter.doFilter(CookieAuthenticationFilter.java:68)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

              at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57)

              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

              at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

              at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)

              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

              at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)

              at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)

              at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)

              at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)

              at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)

              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)

              at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)

              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)

              at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)

              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)

              at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)

              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)

              at org.springframework.boot.actuate.autoconfigure.MetricsFilter.doFilterInternal(MetricsFilter.java:106)

              at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

              at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)

              at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)

              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

              at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

              at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)

              at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)

              at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)

              at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

              at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)

              at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

              at org.eclipse.jetty.server.Server.handle(Server.java:534)

              at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)

              at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)

              at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)

              at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)

              at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:220)

              at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)

              at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)

              at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)

              at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)

              at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

              at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)

              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

              at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

              at java.lang.Thread.run(Thread.java:748)

      Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss

              at com.tableausoftware.tabadmin.agent.zookeeper.ConnectionHolder.lambda$runWithConnectedClient$2(ConnectionHolder.java:179)

              at com.tableausoftware.tabadmin.agent.zookeeper.ConnectionHolder.runWithConnectedClient(ConnectionHolder.java:199)

              at com.tableausoftware.tabadmin.agent.zookeeper.ConnectionHolder.runWithConnectedClient(ConnectionHolder.java:178)

              at com.tableausoftware.tabadmin.agent.zookeeper.LifecycleAwareConnection.runWithSynchronizedConnectedClient(LifecycleAwareConnection.java:308)

              at com.tableausoftware.tabadmin.agent.zookeeper.ZooKeeperWorker.getDataForPath(ZooKeeperWorker.java:343)

              at com.tableausoftware.tabadmin.cluster.properties.ClusterPropertiesStore.getClusterPropertiesZNode(ClusterPropertiesStore.java:119)

              at com.tableausoftware.tabadmin.cluster.properties.ClusterPropertiesStore.getClusterProperties(ClusterPropertiesStore.java:132)

              at com.tableausoftware.tabadmin.webapp.impl.ClusterPropertiesService.getClusterProperties(ClusterPropertiesService.java:232)

              ... 89 more