Service stops working randomly
Matt Neumann Oct 9, 2015 5:06 AMHi,
After migrating our server to the cloud recently, we're experiencing intermittent issues with our reports. I exported everything from the old box and imported it to the one with the --no-config option. Then one our data guys re-published his workbooks and things seemed to be working.
But now we're experiencing 'Service Unavailable' issues when running reports every now and then. The only way around this is 'tabadmin restart' then things go back to normal for a few hours.
I've looks through some of the logs and came across the following in the tomcat logs;
Oct 09, 2015 11:43:27 AM org.apache.catalina.startup.ContextConfig processAnnotationsJar
SEVERE: Unable to process Jar entry [org/apache/hadoop/hdfs/server/datanode/browseBlock_jsp.class] from Jar [jar:jndi:/localhost/solr/WEB-INF/lib/hadoop-hdfs-2.2.0.jar!/] for annotations
java.io.EOFException
at org.apache.tomcat.util.bcel.classfile.FastDataInputStream.readInt(FastDataInputStream.java:145)
at org.apache.tomcat.util.bcel.classfile.ClassParser.readID(ClassParser.java:200)
at org.apache.tomcat.util.bcel.classfile.ClassParser.parse(ClassParser.java:78)
at org.apache.catalina.startup.ContextConfig.processAnnotationsStream(ContextConfig.java:2071)
at org.apache.catalina.startup.ContextConfig.processAnnotationsJar(ContextConfig.java:1947)
at org.apache.catalina.startup.ContextConfig.processAnnotationsUrl(ContextConfig.java:1913)
at org.apache.catalina.startup.ContextConfig.processAnnotations(ContextConfig.java:1898)
at org.apache.catalina.startup.ContextConfig.webConfig(ContextConfig.java:1330)
at org.apache.catalina.startup.ContextConfig.configureStart(ContextConfig.java:889)
at org.apache.catalina.startup.ContextConfig.lifecycleEvent(ContextConfig.java:386)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:117)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:90)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5380)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:649)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1081)
at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1877)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Has anyone seen this/know how to resolve it?
Cheers,
Matt