Can't get to bash in 9.10 Server VM, zombie process and hung tomcat

Question

I have a simple 9.10 Ubuntu server hosted in VirtualBox on a Windows 7 machine.

Overview of issues

• Tomcat crashes
• Something about a zombie process
• Can't get to the shell

---

This VM's been runing fine for almost a year, as have two other VMs that are nearly identical to it.

Recently the physical disk ran out of space, but I feel this is unrelated.

On boot:

 fsck from util-linux-ng 2.16
 /dev/sda1: clean, 142228/2514944 files, 759405/10042625 blocks
 [      3.001120] piix4_xmbus 0000:00:07.0: SMBus base address uninitalized - upgrade BIOS or use force_addr=0xaddr

I don't believe these are directly related to my issue however.

It proceeds to start various daemons, and reaches Tomcat, where it times out entirely.

* Starting Tomcat serverlet engine tomcat6
......... normally, it would throw warning about 
Tomcat not having started correctly, and crashing after 120 seconds,
For some reason, it didn't do it this time.

The system then gets to a login prompt, where I can log in successfully with my username/password combination. It shows all the standard "welcome" information, and also previously displayed a message about a zombie process, but now I can't replicate that after hard rebooting the VM.

Then the terminal acts like it died on me, and I can't reach the $ shell to type something. All my keys can be used, it just doesn't take anything to the shell. If I slam ctrl+c a few times, it logs me out back to the authentication screen, but nothing past that.

If I wait long enough:

[480.450049] INFO: task cron:1256 blocked for more than 120 seconds.
[480.450798] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message

Okay, remember here, I can't get to the shell. So I've got no idea what to do.

This is basically what I'm looking at:

You can see where I'm slamming on enter out of frustration.

Update

Okay, so it looks like all those times I slammed on enter, shell showed up all at once after another message about something being blocked, so it looks like bash is just really laggy? I'm not sure. Also, the Apache HTTPD seems to send data, again, just sending the response back after a considerable delay.

Update2

I was unable to SSH or otherwise into the server, due to the response delay.

After several of these task cron blocked messages came through, I am now able to use console, but SSH will not let me in. The delay in apache seems to have dissapated however.

Because of the ability to log into shell, I removed Tomcat6, and am presently trying to get into crontab -e to remove some tasks.

Update3

Okay, so I've removed a secondary VHD, that way I only have the host drive on the system, I've also run sudo apt-get remove tomcat6. I'm still seeing the blocking issue, and I'm also now seeing * Invalid CATALINA_BASE in /var/lib/tomcat6 on startup.

Answer 1

I ran TOP and saw that winbindd was eating up a ton of resources, and attempting a kill-9 on it resulted in the process simply jumping to a new ID, somehow.

I removed winbindd and tomcat, this has resolved the issue.

I suspect the AD/LDAP has changed somehow without my knowledge and impacted the way winbindd was interacting with it.

Answer 2

My first guess is that you are seeing the VM trashing, probably because of a virtual disk running out of space. [I've seen similar strangeness when I have had my / volume filled and nothing can be written to /tmp.]

If you can, reboot the VM and select "recovery mode" instead of a normal boot. When prompted for recovery action, ask to drop into a root shell prompt. From there, you should be able to do more diagnosis and resolve the issue.

The things I would look for are

• free disk space
• are /, /var, /tmp writable? (Did they get mounted ro?, If so, do you need to fsck?)