High System Load, Low CPU/RAM Utilization on Ubuntu 15.04

Question

not really a system administrator here but really trying to just set up a server (a rented VDS, really) for some friends.

I recently transferred basically game servers/MySQL/web sites over from one VPS to another - while there hasn't been any issues on the new one I keep seeing my system load spike and take up both processors; previous server system load averaged at about .3-.5. Previous server was on Ubuntu 14, I exported a list of packages I installed from there and apt-get installed them on the new server; I also rsync'd most of the files from the old server over as well (I'm thinking I copied over something bad that's messing with my kernel...)

Anyway, here is the results of my uname -a:

 Linux ophq 3.19.0-18-generic #18-Ubuntu SMP Tue May 19 18:31:35 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

And the results of landscape-sysinfo/logging in screen:

  Welcome to Ubuntu 15.04 (GNU/Linux 3.19.0-18-generic x86_64)
  System load:  2.13                Processes:           11
  Usage of /:   22.6% of 196.64GB   Users logged in:     1
  Memory usage: 32%                 IP address for eth0: 123.123.123.123
  Swap usage:   0%

(currently one game server is in use hence the memory usage - I have to reduce how much RAM is allocated to Minecraft from the default values)

Result of top: http://ericbarber.me/serverproblem/top.png

To add to this - if I hit F and then hit S on 'Process Status' and resort the top lists I have 2 commands listed under 'D'... kworker/u30:0 and kworker/u30:1 which leads me to my kernel assumption...

I'm totally stumped on why load average is so high - I had my users test on both MC and the CS:GO servers and they aren't experiencing any lag - I also tested the web servers and they're delivering pages extremely fast (in comparison to the old server.)

I thought it may be an interrupt issue, so here's the results of cat /proc/interrupts:

http://ericbarber.me/serverproblem/interrupts2.png

Along with this, another question suggested running grep . -r /sys/firmware/acpi/interrupts/ and disabling any values above 0...although all my values are 0 unfortunately.

same url as above serverproblem/interrupts.png

I installed perf and did a quick 30 second report - but I don't understand this output too much:

same url as above serverproblem/perf.png

I'll omit CPU info, but it is an Intel Xeon CPU E5-2690, 2 cores, 2gb RAM, and I believe about 500gb harddrive. My apologies if this is a dumb question or has been asked before - I've been working on this for a few hours now and I'm running into dead-ends with Google past just starting over from scratch...which preferably I would like to avoid.

Apologies on the links..new user limitations.

Edit: To add, the results of mpstat:

Linux 3.19.0-18-generic (ophq)  06/05/2015  _x86_64_    (2 CPU)

02:10:35 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal       %guest  %gnice   %idle
02:10:35 PM  all    7.28    0.00    1.72   47.13    0.00    0.09    0.53        0.00    0.00   43.24

Answer 1

This ended up being what I believe to be a kernel bug. Upon updating to 4.0.0-040000-generic #201504121935 my CPU wait has been normal and system load under .10 in most cases unless something is happening on the hosted servers.

Anyway, I used the following link to help : http://ubuntuhandbook.org/index.php/2015/04/upgrade-to-linux-kernel-4-0-in-ubuntu/

and just to keep in compliance with the rules, I did the following as root and then rebooted the machine:

wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0-vivid/linux-headers-4.0.0-040000_4.0.0-040000.201504121935_all.deb
wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0-vivid/linux-image-4.0.0-040000-generic_4.0.0-040000.201504121935_amd64.deb
wget http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.0-vivid/linux-headers-4.0.0-040000-generic_4.0.0-040000.201504121935_amd64.deb
dpkg -i linux-headers-4.0.0*.deb linux-image-4.0.0*.deb
update-grub

As far as how I came to this - after reading through countless forums and newsgroups/mailing lists and getting nowhere (tried messing with BIOs, boot options, commit=60, disabling services, changing physical server location, etc.) I decided to either downgrade or update the kernel...being that 15.04 is new I updated. Still unsure the root cause as I haven't seen any other reports of this issue, my assumption is when I used rsync from my old 14.10 system a faulty driver was copied over or a faulty kernel file - why 4.0.0 fixes this is beyond me...but at least no more kworker writing every 5 seconds to kern.log and my harddrives.

Answer 2

Sometimes the hard disk could be the bottleneck and cause high system load so you might want to look into that.

You can check these links for various suggestions about tools you can use to find out whether this is the problem in your case.

https://serverfault.com/questions/9428/how-can-i-monitor-hard-disk-load-on-linux

https://softwarerecs.stackexchange.com/questions/460/command-line-tool-on-ubuntu-server-to-see-disk-io-stats

Answer 3

I recently ran into a similar issue with 14.10 and 15.04 server, traced it to a cheap display adapter (pny geforce 210 pci-E) causing nouveau to freak out anytime a display was not attached to the card. may be unrelated to your issue but after removing the card from my box it cleared up for me