0

I'm using Ubuntu 15.04 with MD RAID1, LVM and LXC on top of them as a webserver. I have normal load average about 1-2, and all is working well until I perfrom any massive IO operation. The strange thing is that I'm doing these operations on unused disks. So, for example, sdc and sdd are in RAID1 used by webserver and I'm doing dd if=/dev/zero of=/dev/sdb, or just simply running du. After few seconds, load average jumps to 10 and more and the system became almost unusable. Nmon shows that all disks became overloaded, and if I would not stop the operation, overloading will continue to grow.

I'm attaching screenshots of nmon. Overload with dd if=/dev/zero of=/dev/sdb.

Output of uname:

... 3.19.0-39-generic #44-Ubuntu SMP Tue Dec 1 14:39:05 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

UPDATE:

I have changed disk configuration, replaced system disks with more robust HGST. System disk aliases have been changed after the system rebooting.

Output of lsblk:

NAME                     FSTYPE              SIZE MOUNTPOINT
sda                                          1,8T 
├─sda1                                         1M 
├─sda2                   linux_raid_member    50G 
│ └─md0                  LVM2_member          50G 
│   ├─system-swap        swap                 10G [SWAP]
│   └─system-root        ext4                 40G /
└─sda3                   linux_raid_member 881,5G 
  └─md1                  LVM2_member       881,4G 
    ├─lxc-hosting        ext3                450G 
    ├─lxc-ns1-real                             1G 
    │ ├─lxc-ns1          ext3                  1G 
    │ └─lxc-ns1--snap6   ext3                  1G 
    └─lxc-ns1--snap6-cow                       1G 
      └─lxc-ns1--snap6   ext3                  1G 
sdb                                          1,8T 
├─sdb1                                         1M 
├─sdb2                   linux_raid_member    50G 
│ └─md0                  LVM2_member          50G 
│   ├─system-swap        swap                 10G [SWAP]
│   └─system-root        ext4                 40G /
├─sdb3                   linux_raid_member 881,5G 
│ └─md1                  LVM2_member       881,4G 
│   ├─lxc-hosting        ext3                450G 
│   ├─lxc-ns1-real                             1G 
│   │ ├─lxc-ns1          ext3                  1G 
│   │ └─lxc-ns1--snap6   ext3                  1G 
│   └─lxc-ns1--snap6-cow                       1G 
│     └─lxc-ns1--snap6   ext3                  1G 
└─sdb4                   ext4              931,5G 
sdc                      linux_raid_member 931,5G 
└─md2                    LVM2_member       931,4G 
  └─reserve-backups      ext4              931,4G /var/backups/mounted
sdd                      linux_raid_member 931,5G 
└─md2                    LVM2_member       931,4G 
  └─reserve-backups      ext4              931,4G /var/backups/mounted

Output of lsscsi:

[0:0:0:0]    disk    ATA      HGST HUS726020AL W517  /dev/sda 
[1:0:0:0]    disk    ATA      HGST HUS726020AL W517  /dev/sdb 
[2:0:0:0]    disk    ATA      WDC WD1002FBYS-0 0C06  /dev/sdc 
[3:0:0:0]    disk    ATA      WDC WD1002FBYS-0 0C06  /dev/sdd

I have performed new test with pv (see Load with pv < /dev/zero > /dev/md2). Now the situation is better. Operation on disks sdc and sdd still affects performance of sda, but load average keeps on 6-7 and the system is not overloaded. And I think now I understand what may be the reason of the issue. Since sda and sdb (as parts of RAID) used for webserver, they have constant stream of parallel requests. Hard disks have to constanly move their heads thereby reducing IO performance, but the system IO cache (I believe linux uses all free memory for this purpose) helps to minimize the number of disk accesses. When I begin to intensively use other disks, the cache becames filled with new data, that is why this increases the load on the main disks. If I'm right, increasing the amount of RAM will help. Now it's only 16 GB.

Improve this question

2 Answers 2

Reset to default
1

Most likely the disks are all on the same controller and the controller is the bottleneck. That is to say, there is a limit to the throughput of the sum total of io to all of the disks, so when you start using one heavily, there is less available to the others.

Improve this answer
11
  • I was thinking that controller may be a bottleneck, but I don't know how to diagnoze it. We use ICH10, it's a quite good server controller. I think there was not to much IO to make it a bottleneck. On the screenshot it's only 150 MBs on sdb and this makes sdc overloaded with only 10 MBs? When I tested RAID1, it was 300 Mbs. And I don't believe that du generates so much traffic that it can overload this controller (I haven't posted the screenshot with du, but it was amost the same as with dd). Maybe there is a limit not on the volume of transfered data, but on the quantity of IO queries?
    – stant
    Dec 24, 2015 at 11:27
  • @stant, it is a combined 200 mb/s which isn't bad... but yes, I would expect more like 300 from an ICH10. What is your exact lvm and raid configuration? If you have a logical volume spread between two disks and you start doing a lot of activity to another partition on one of those disks, then that IO is competing for access to the disk with your IO to your logical volume. Try adding the output of lsblk to your question.
    – psusi
    Dec 24, 2015 at 17:56
  • I have updated the question. I think now I know the reason of the issue. Can you confirm or disprove it?
    – stant
    Dec 25, 2015 at 14:07
  • @stant, your system and lxc volume groups are on sda and sdb, so if you ran dd if=/dev/zero of=/dev/sdb, you corrupted your raid array. At this point you need to immediately fail the two partitions on sdb out of their respective raid arrays, shut down, and fsck the volumes on them and hope they can be recovered before re-adding the sdb partitions to the arrays and letting them resync. To make sure the data is fully resynced, you need to mdadm --zero-superblocks on the sdb partitions before adding them back to the arrays.
    – psusi
    Dec 26, 2015 at 1:14
  • @stant, also /dev/md2 contains your reserve-backups volume group, which you totally destroyed by running pv < /dev/zero > /dev/md2
    – psusi
    Dec 26, 2015 at 1:17
0

You could try a different disk scheduler. Check to see what is currently in use on your disks (i.e. for sda):

cat /sys/block/sda/queue/scheduler

You might see:

noop anticipatory deadline [cfq]

Then you could decide what scheduler to use for each disk, depending on what type they are (ssd, hdd) or what controller they are on. Simply set it to a different scheduler by typing:

echo {SCHEDULER-NAME} > /sys/block/{DEVICE-NAME}/queue/scheduler

So if you want to use deadline for sda it would be:

echo deadline > /sys/block/sda/queue/scheduler

You might find using deadline on a server will give you better responsiveness while under heavy load.

If you want to change the scheduler for ALL disks:

Edit /etc/default/grub:

sudo nano /etc/default/grub

You need to add elevator=deadline. Change GRUB_CMDLINE_LINUX_DEFAULT="quiet splash" to

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash elevator=deadline"

Then run

sudo update-grub2

and restart.

Improve this answer
2
  • Thank you for your answer, but I already tried all schedulers. There is almost no difference. The best results was with deadline, it's default scheduler in 15.04 and I used it while doing these screenshots.
    – stant
    Dec 24, 2015 at 11:02
  • Well, it was worth a try. Check your disks with smartctl to make sure it is not the controller having to wait constantly for a bad disk. Also, if you have a couple spare SATA cables kicking around then swap them out one by one to see if there is improvement.
    – G Trawo
    Dec 24, 2015 at 14:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .