2

I'm honestly not sure, what is going on. I have tomcat 8.0.9 package installed (tomcat8) and it works fine once I start the service. However, at some point, it shuts down and I'm not sure why. The catalina.out does not say why either. Does anyone have any ideas on how I can find out what is causing the shutdown?

Here's a snippet of catalina.out showing I last started the server Jan 13. It has no shutdown messages and shows me restarting it this morning (sudo service tomcat8 start):

INFO: Starting ProtocolHandler ["http-nio-8080"]
Jan 13, 2015 11:44:37 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-nio-8009"]
Jan 13, 2015 11:44:37 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 7942 ms
Tue Jan 13 23:45:58 EST 2015-25 java.lang.instrument.Instrumentation is used to reload class files
Jan 15, 2015 9:53:52 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8080"]
Jan 15, 2015 9:53:52 AM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
INFO: Using a shared selector for servlet write/read
Jan 15, 2015 9:53:52 AM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-nio-8009"]
Jan 15, 2015 9:53:52 AM org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
INFO: Using a shared selector for servlet write/read
Jan 15, 2015 9:53:52 AM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 2925 ms
Jan 15, 2015 9:53:52 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Jan 15, 2015 9:53:52 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/8.0.9 (Ubuntu)
Jan 15, 2015 9:53:52 AM org.apache.catalina.startup.HostConfig deployDescriptor

Unless this line 'java.lang.instrument.Instrumentation is used to reload class files' is telling me about a shutdown, I don't see any information about why it shutdown.

I thought maybe someone was telneting into to it blindly and shutting it down by issueing SHUTDOWN, but my iptables is setup to now allow any connections but my home computer.

iptables -L --line-numbers
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    ACCEPT     all  --  anywhere             anywhere
2    ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
3    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
4    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
5    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ftp
6    ACCEPT     tcp  --  removed.my.ip.here   anywhere             tcp
7    DROP       all  --  anywhere             anywhere

Chain FORWARD (policy ACCEPT)
num  target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination

Is there a way to enable additional logging in tomcat8 so that I might see what is causing the shutdown?

Edit: 1/23/2015 - I've ugprade to Tomcat 8.0.17 (latest version), and it's still happening and still without any clues in the log file...

Requested Additional Information:

  • Ubuntu 14.10
    • Hosted by Luna Node Dynamic
    • Linux dev 3.16.0-29-generic #39-Ubuntu SMP Mon Dec 15 22:27:29 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
    • 512MB RAM
    • 256MB SWAP
    • 1 CPU
  • Tomcat 8.0.9
    • Installed via apt-get
    • Upgraded to 8.0.17 by downloading binaries
    • Started via 'service tomcat8 start'
    • Tomcat stops itself (which is the issue) I've never stopped it but would use 'service tomcat8 stop' if I wanted to stop it
  • Java Version Info:
    • openjdk version "1.8.0_40-internal"
    • OpenJDK Runtime Environment (build 1.8.0_40-internal-b09)
    • OpenJDK 64-Bit Server VM (build 25.40-b13, mixed mode)
    • Install via apt-get

/etc/default/tomcat8:

TOMCAT8_USER=tomcat8
TOMCAT8_GROUP=tomcat8
JAVA_OPTS="-Djava.awt.headless=true -Xms256m -Xmx256m -XX:+UseConcMarkSweepGC -javaagent:/usr/share/tomcat8/railo/railo-inst.jar"

/etc/tomcat8/logging.properties:

handlers = 1catalina.org.apache.juli.FileHandler, 2localhost.org.apache.juli.FileHandler, java.util.logging.ConsoleHandler
.handlers = 1catalina.org.apache.juli.FileHandler, java.util.logging.ConsoleHandler

1catalina.org.apache.juli.FileHandler.level = FINE
1catalina.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
1catalina.org.apache.juli.FileHandler.prefix = catalina.

2localhost.org.apache.juli.FileHandler.level = FINE
2localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs
2localhost.org.apache.juli.FileHandler.prefix = localhost.

java.util.logging.ConsoleHandler.level = ALL
java.util.logging.ConsoleHandler.formatter = java.util.logging.SimpleFormatter

org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = FINE
org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.org.apache.juli.FileHandler

org.apache.catalina.session.level=ALL
7
  • Added a bit of information... Hopefully helpful... If it becomes starved for memory, would it just drop itself? That's the only logical explanation I can see but I would still expect something in the log.
    – Brad
    Jan 28, 2015 at 18:02
  • 2
    Also, what is your VPS provider? It is written here, that hosting company killing CPU consuming processes.
    – c0rp
    Jan 28, 2015 at 18:04
  • Added, it's Luna Node Dynamic instance. Here's a chart of my CPU usage over the last month, not even hitting 20%. This chart is from their VM control panel. imgur.com/FVEUrgb
    – Brad
    Jan 28, 2015 at 18:07
  • There is a lot interesting answers here. Did you read them all? May be something fits your problem
    – c0rp
    Jan 28, 2015 at 18:12
  • For example, try to undeploy all your applications and see if problems is persists
    – c0rp
    Jan 28, 2015 at 18:14

1 Answer 1

1

The KVM virtualization kernel was killing the process because the VM was running out of memory. This is validated by checking out /var/log/kern.log:

Jan 28 11:30:07 dfrewind-dev kernel: [970762.688259] Out of memory: Kill process 11884 (java) score 536 or sacrifice child
Jan 28 11:30:07 dfrewind-dev kernel: [970762.690756] Killed process 11884 (java) total-vm:2317312kB, anon-rss:257188kB, file-rss:0kB

Based on the question here regarding Tomcat restarting, I decided to email my hosting provider to ask if they were killing my process despite my CPU utilization being low. They stated that it was impossible for them to kill a process on my server without remoting into it due to the virtualization technology they use (KVM) and their privacy policy.

They elaborated that the KVM virtualization kernel will kill the process using the most memory whenever the VM runs out of memory and I could verify if this was happening by viewing /var/log/kern.log.

2
  • 1
    Could you also expand you answer. Tell how did you find culprit, put link to q/w that inspires you, etc... This will help a lot to other users
    – c0rp
    Jan 28, 2015 at 18:21
  • 1
    My host actually gave me the clue when I asked about them killing processes. I will expand this answer.
    – Brad
    Jan 28, 2015 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.