KVM guest performance impacted by very long virtblk disk wait times

Question

I have several KVM guests running on Ubuntu 14.04 using libvirt. Their performance is very poor. The intermittent nature of the poor performance with an order of magnitude variability in simple benchmark tests and apparent influence of activity on other VMs suggests strongly that the problem is contention for resources in the host. Sar monitoring on the guests reveals a problem with virtblk disks having to wait a very large time for I/O requests to be serviced. E.g.

00:00:01          DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
00:05:01     dev253-0      0.10      0.00      1.39     14.34      0.00 682539.72      0.14      0.00

The await time is consistently extremely high and is similar across all guests.

A typical disk configuration is

<disk type='file' device='disk'>
  <driver name='qemu' type='raw'/>
  <source file='/var/lib/libvirt/images/Base-dev-0002-clone.img'/>
  <target dev='vda' bus='virtio'/>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</disk>

A number of things have been tried on the guests including cache settings and io=native in the driver line, cloning from raw img file storage to a logical volume, elevator=deadline, virtio_blk.use_bio=1 but without any significant impact.

I understand that a virtual disk must take a performance hit but this is extreme.

What is typical await time for a KVM guest?

Bearing in mind my resource contention claim, what resources, ulimits etc might need to be allocated on the host? The host has plenty of resources and shows no obvious signs of problems.

Answer 1

I recently have the same issue

# iostat -dx 60
Linux 3.13.0-32-generic (hostname)  Wednesday, 04 March, 2015   _x86_64_    (6 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     2.46   51.49    4.12  1742.66   325.36    74.38     0.05  770.46  487.83 4300.67   0.15   0.81
scd0              0.00     0.00    0.08    0.00     0.31     0.00     8.00     0.00    0.73    0.73    0.00   0.73   0.01

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.03    0.02    0.20     0.13     0.93     9.85     0.00 35787.38    4.00 38769.33   0.31   0.01
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Upgrading the kernel and installing linux-headers and linux-headers-generic on the KVM guests fix my I/O issue. Reboot after installing.

# aptitude install linux-headers-3.13.0-46 linux-headers-3.13.0-46-generic linux-image-3.13.0-46-generic

After installation:

# iostat -dx 60
Linux 3.13.0-46-generic (hostname)  Wednesday, 04 March, 2015   _x86_64_    (6 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.36   19.36    0.42   227.23     4.51    23.44     0.01    0.50    0.45    2.92   0.14   0.28
scd0              0.00     0.00    0.04    0.00     0.18     0.00     8.00     0.00    0.36    0.36    0.00   0.36   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
scd0              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00