6 Replies Latest reply on Mar 3, 2017 7:27 AM by Toby Erkson

    Connection Errors Contacting Workers

    chris.mccord

      I'veOur test/staging Tableau 8 system starting having issues over the weekend, and I've been digging to try to find out why.

      Here is what our status says...

       

      D:\Tableau>cd "Tableau Server"

      D:\Tableau\Tableau Server>cd 8.0

      D:\Tableau\Tableau Server\8.0>cd bin

      D:\Tableau\Tableau Server\8.0\bin>tabadmin status -v
      10.9.105.126:
              Status: STOPPED
      10.9.105.128:
              Connection error contacting worker 1
      10.9.105.129:
              Connection error contacting worker 2
      10.9.105.127:
              Status: STOPPED

      D:\Tableau\Tableau Server\8.0\bin>

       

       

      This is from the log on one of the workers with issues...

      2013-11-04 13:25:08.414 -0600_DEBUG_10.9.105.128:OKMTST3268_:_pid=2012_0x1311ad__user=__request=__ Checking Windows Firewall for service app permissions for D:/Tableau/Tableau Server/worker/bin/tabadmwrk.exe

      2013-11-04 13:25:08.414 -0600_DEBUG_10.9.105.128:OKMTST3268_:_pid=2012_0x1311ad__user=__request=__ D:/Tableau/Tableau Server/worker/bin/tabadmwrk.exe does not exist

       

      More weird stuff from the Event viewer on one of the servers in error...

      Log Name:      Application
      Source:        Tableau Server (Backgrounder)
      Date:          11/5/2013 3:14:18 PM
      Event ID:      4096
      Task Category: (5)
      Level:         Error
      Keywords:      Classic
      User:          N/A
      Description:
      The description for Event ID 4096 from source Tableau Server (Backgrounder) cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

      If the event originated on another computer, the display information had to be saved with the event.

      The following information was included with the event:

      2013-11-05 15:14:18.926 -0600 main   ERROR : org.hibernate.engine.jdbc.spi.SqlExceptionHelper - Cannot create PoolableConnectionFactory (Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.)

       

       

      Log Name:      Application
      Source:        Tableau Server (Backgrounder)
      Date:          11/5/2013 3:14:18 PM
      Event ID:      4096
      Task Category: (5)
      Level:         Error
      Keywords:      Classic
      User:          N/A

      Description:
      The description for Event ID 4096 from source Tableau Server (Backgrounder) cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

      If the event originated on another computer, the display information had to be saved with the event.

      The following information was included with the event:

      2013-11-05 15:14:18.926 -0600 main   ERROR : com.tableausoftware.model.workgroup.service.ServerLicenseServiceImpl - Error getting coreLicense allocated cores
      javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1377) at org.hibernate.ejb.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1300) at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:318) at com.tableausoftware.model.workgroup.jpa.CoreLicenseDaoJpa.findTotalAllocatedCores(CoreLicenseDaoJpa.java:77) at com.tableausoftware.model.workgroup.service.CoreLicenseServiceImpl.totalAllocatedCores(CoreLicenseServiceImpl.java:160) at com.tableausoftware.model.workgroup.service.ServerLicenseServiceImpl.totalAllocatedCores(ServerLicenseServiceImpl.java:557) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:508) at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:368) at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:50) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:133) at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:63) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216) at org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120) at org.jruby.ast.RescueNode.interpret(RescueNode.java:110) at org.jruby.ast.BeginNode.interpret(BeginNode.java:83) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:147) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:133) at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:63) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216) at org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120) at org.jruby.ast.RescueNode.interpret(RescueNode.java:110) at org.jruby.ast.BeginNode.interpret(BeginNode.java:83) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:111) at org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:374) at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:295) at org.jruby.runtime.InterpretedBlock.yieldSpecific(InterpretedBlock.java:229) at org.jruby.runtime.Block.yieldSpecific(Block.java:99) at org.jruby.ast.ZYieldNode.interpret(ZYieldNode.java:25) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216) at org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120) at org.jruby.ast.RescueNode.interpret(RescueNode.java:110) at org.jruby.ast.BeginNode.interpret(BeginNode.java:83) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:212) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:322) at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:178) at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:187) at org.jruby.ast.FCallOneArgBlockNode.interpret(FCallOneArgBlockNode.java:34) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.IfNode.interpret(IfNode.java:117) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:147) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:292) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:135) at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:63) at org.jruby.ast.LocalAsgnNode.interpret(LocalAsgnNode.java:123) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) at org.jruby.ast.IfNode.interpret(IfNode.java:119) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) at org.jruby.ast.RootNode.interpret(RootNode.java:129) at org.jruby.evaluator.ASTInterpreter.INTERPRET_ROOT(ASTInterpreter.java:119) at org.jruby.Ruby.runInterpreter(Ruby.java:724) at org.jruby.Ruby.loadFile(Ruby.java:2487) at org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:66) at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:745) at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:332) at org.jruby.runtime.load.LoadService.require(LoadService.java:379) at org.jruby.runtime.load.LoadService.lockAndRequire(LoadService.java:304) at org.jruby.RubyKernel.requireCommon(RubyKernel.java:1050) at org.jruby.RubyKernel.require(RubyKernel.java:1033) at org.jruby.RubyKernel$s$1$0$require.call(RubyKernel$s$1$0$require.gen:65535) at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:86) at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:86) at org.jruby.runtime.callsite.SuperCallSite.callBlock(SuperCallSite.java:190) at org.jruby.runtime.callsite.SuperCallSite.call(SuperCallSite.java:197) at org.jruby.runtime.callsite.SuperCallSite.callVarargs(SuperCallSite.java:108) at org.jruby.ast.ZSuperNode.interpret(ZSuperNode.java:102) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.evaluator.ASTInterpreter.INTERPRET_BLOCK(ASTInterpreter.java:111) at org.jruby.runtime.InterpretedBlock.evalBlockBody(InterpretedBlock.java:374) at org.jruby.runtime.InterpretedBlock.yield(InterpretedBlock.java:295) at org.jruby.runtime.InterpretedBlock.yieldSpecific(InterpretedBlock.java:229) at org.jruby.runtime.Block.yieldSpecific(Block.java:99) at org.jruby.ast.ZYieldNode.interpret(ZYieldNode.java:25) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.IfNode.interpret(IfNode.java:119) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216) at org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120) at org.jruby.ast.RescueNode.interpret(RescueNode.java:110) at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:212) at org.jruby.runtime.callsite.CachingCallSite.callBlock(CachingCallSite.java:176) at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:187) at org.jruby.ast.FCallOneArgBlockNode.interpret(FCallOneArgBlockNode.java:34) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:190) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169) at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) at org.jruby.ast.RootNode.interpret(RootNode.java:129) at org.jruby.evaluator.ASTInterpreter.INTERPRET_ROOT(ASTInterpreter.java:119) at org.jruby.Ruby.runInterpreter(Ruby.java:724) at org.jruby.Ruby.loadFile(Ruby.java:2487) at org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:66) at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:745) at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:332) at org.jruby.runtime.load.LoadService.require(LoadService.java:379) at org.jruby.runtime.load.LoadService.lockAndRequire(LoadService.java:304) at org.jruby.RubyKernel.requireCommon(RubyKernel.java:1050) at org.jruby.RubyKernel.require(RubyKernel.java:1033) at org.jruby.RubyKernel$s$1$0$require.call(RubyKernel$s$1$0$require.gen:65535) at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:319) at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169) at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.IfNode.interpret(IfNode.java:117) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216) at org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120) at org.jruby.ast.RescueNode.interpret(RescueNode.java:110) at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:190) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169) at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216) at org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120) at org.jruby.ast.RescueNode.interpret(RescueNode.java:110) at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:147) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:292) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:135) at org.jruby.ast.CallNoArgNode.interpret(CallNoArgNode.java:63) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) at org.jruby.ast.RootNode.interpret(RootNode.java:129) at org.jruby.evaluator.ASTInterpreter.INTERPRET_ROOT(ASTInterpreter.java:119) at org.jruby.Ruby.runInterpreter(Ruby.java:724) at org.jruby.Ruby.loadFile(Ruby.java:2487) at org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:66) at org.jruby.runtime.load.LoadService.load(LoadService.java:270) at org.jruby.RubyKernel.loadCommon(RubyKernel.java:1076) at org.jruby.RubyKernel.load(RubyKernel.java:1058) at org.jruby.RubyKernel$s$0$1$load.call(RubyKernel$s$0$1$load.gen:65535) at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:211) at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:207) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169) at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) at org.jruby.ast.RootNode.interpret(RootNode.java:129) at org.jruby.evaluator.ASTInterpreter.INTERPRET_ROOT(ASTInterpreter.java:119) at org.jruby.Ruby.runInterpreter(Ruby.java:724) at org.jruby.Ruby.loadFile(Ruby.java:2487) at org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:66) at org.jruby.runtime.load.LoadService.load(LoadService.java:270) at org.jruby.RubyKernel.loadCommon(RubyKernel.java:1076) at org.jruby.RubyKernel.load(RubyKernel.java:1058) at org.jruby.RubyKernel$s$0$1$load.call(RubyKernel$s$0$1$load.gen:65535) at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:211) at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:207) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169) at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.RootNode.interpret(RootNode.java:129) at org.jruby.evaluator.ASTInterpreter.INTERPRET_ROOT(ASTInterpreter.java:119) at org.jruby.Ruby.runInterpreter(Ruby.java:724) at org.jruby.Ruby.loadFile(Ruby.java:2487) at org.jruby.runtime.load.ExternalScript.load(ExternalScript.java:66) at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:745) at org.jruby.runtime.load.LoadService.smartLoad(LoadService.java:332) at org.jruby.runtime.load.LoadService.require(LoadService.java:379) at org.jruby.runtime.load.LoadService.lockAndRequire(LoadService.java:304) at org.jruby.RubyKernel.requireCommon(RubyKernel.java:1050) at org.jruby.RubyKernel.require(RubyKernel.java:1033) at org.jruby.RubyKernel$s$1$0$require.call(RubyKernel$s$1$0$require.gen:65535) at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:319) at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169) at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.IfNode.interpret(IfNode.java:117) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216) at org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120) at org.jruby.ast.RescueNode.interpret(RescueNode.java:110) at org.jruby.evaluator.ASTInterpreter.INTERPRET_METHOD(ASTInterpreter.java:74) at org.jruby.internal.runtime.methods.InterpretedMethod.call(InterpretedMethod.java:190) at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:312) at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:169) at org.jruby.ast.FCallOneArgNode.interpret(FCallOneArgNode.java:36) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.BlockNode.interpret(BlockNode.java:71) at org.jruby.ast.RescueNode.executeBody(RescueNode.java:216) at org.jruby.ast.RescueNode.interpretWithJavaExceptions(RescueNode.java:120) at org.jruby.ast.RescueNode.interpret(RescueNode.java:110) at org.jruby.ast.BeginNode.interpret(BeginNode.java:83) at org.jruby.ast.NewlineNode.interpret(NewlineNode.java:104) at org.jruby.ast.RootNode.interpret(RootNode.java:129) at org.jruby.evaluator.ASTInterpreter.INTERPRET_ROOT(ASTInterpreter.java:119) at org.jruby.Ruby.runInterpreter(Ruby.java:724) at org.jruby.Ruby.runInterpreter(Ruby.java:732) at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:120) at org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1263) at org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1256) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at JarMain.launchJRuby(JarMain.java:98) at JarMain.start(JarMain.java:111) at JarMain.main(JarMain.java:136)Caused by: org.hibernate.exception.GenericJDBCException: Could not open connection at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:54) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:125) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:110) at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:304) at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.getConnection(LogicalConnectionImpl.java:169) at org.hibernate.engine.jdbc.internal.proxy.ConnectionProxyHandler.extractPhysicalConnection(ConnectionProxyHandler.java:82) at org.hibernate.engine.jdbc.internal.proxy.ConnectionProxyHandler.continueInvocation(ConnectionProxyHandler.java:138) at org.hibernate.engine.jdbc.internal.proxy.AbstractProxyHandler.invoke(AbstractProxyHandler.java:81) at sun.proxy.$Proxy67.prepareStatement(Unknown Source) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$5.doPrepare(StatementPreparerImpl.java:147) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl$StatementPreparationTemplate.prepareStatement(StatementPreparerImpl.java:166) at org.hibernate.engine.jdbc.internal.StatementPreparerImpl.prepareQueryStatement(StatementPreparerImpl.java:145) at org.hibernate.loader.Loader.prepareQueryStatement(Loader.java:1711) at org.hibernate.loader.Loader.doQuery(Loader.java:828) at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:289) at org.hibernate.loader.Loader.doList(Loader.java:2438) at org.hibernate.loader.Loader.doList(Loader.java:2424) at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2254) at org.hibernate.loader.Loader.list(Loader.java:2249) at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:470) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:355) at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:195) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1248) at org.hibernate.internal.QueryImpl.list(QueryImpl.java:101) at org.hibernate.ejb.QueryImpl.getSingleResult(QueryImpl.java:285) ... 227 moreCaused by: org.apache.commons.dbcp.SQLNestedException: Cannot create PoolableConnectionFactory (Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.) at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1549) at org.apache.commons.dbcp.BasicDataSource.createDataSource(BasicDataSource.java:1388) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) at org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider.getConnection(InjectedDataSourceConnectionProvider.java:70) at org.hibernate.internal.AbstractSessionImpl$NonContextualJdbcConnectionAccess.obtainConnection(AbstractSessionImpl.java:281) at org.hibernate.engine.jdbc.internal.LogicalConnectionImpl.obtainConnection(LogicalConnectionImpl.java:297) ... 248 moreCaused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:123) at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:28) at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:20) at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30) at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:22) at org.postgresql.Driver.makeConnection(Driver.java:391) at org.postgresql.Driver.connect(Driver.java:265) at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38) at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582) at org.apache.commons.dbcp.BasicDataSource.validateConnectionFactory(BasicDataSource.java:1556) at org.apache.commons.dbcp.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:1545) ... 253 moreCaused by: java.net.ConnectException: Connection refused: connect at java.net.DualStackPlainSocketImpl.connect0(Native Method) at

       

      I've double checked the firewalls on all the machines in the HA cluster, I can ping all servers from all servers.

      I am not familar with the file noted in bold, but I checked out our future Production install of Tableau 8, its working fine, this file does't exist on that server either. So this log is complaining about this file not being there and its not, but yet its not in a totally separate environment either that is working fine.

       

      The big problem trying to fix this is that both of these servers with issue, are where my repositories are located. I can't remove them from the HA setup, because it requires you move the repository and have an active one first, our active and standby repositories are both in error with no contact. So I can't remove them. My only other option, I guess is to try to uninstall the worker software on those two servers, and reinstall, but not sure what that will do to the system? I'm afraid I'm hosed, and might have to rebuild from scratch.

       

      So any clues what I should chase next?