scheduler: how to tune cfq to favor interactive processes

Question

Problem: scheduler doesn't seem to favor interactive processes:

On a desktop system with automatic cron-scheduled backups from one (btrfs) disk to another (ext4). The backup process mounts the idle disk (/dev/sda<X>), backups to it, and finally unmounts it.

Every time the backup process kicks in, the system becomes unusable. The scheduler seems to be failing to do its most basic job of favoring interactive processes over batch ones. While the backup processes run, there's a lot of IO going on and everything else freezes. The keyboard and mouse pointer stop responding. Echo when keys are pressed in any terminal/shell is delayed by several seconds.

As soon as the backup completes, interactive response goes back to normal.

More details on the setup and configs:

The backup process uses rsnapshot (which calls rsync and cp -al) and runs at a lower priority (the backup job is preceded by nice), like so:

nice /usr/bin/rsnapshot -VD -c /etc/my-rsnapshot.conf daily

Running the backup under nice doesn't seem to help. During backups, all interactive processes seem to be starved by the heavy CPU and IO of the rsync and cp processes.

This is a IA-64, iCore-7 system, which should be able to run 8-processes in parallel. Memory is 16GB and some of it is free. Trimmed down mount output (when additional disk is mounted) is:

/dev/sdb2 on / type btrfs (rw,relatime,subvol=@,thread_pool=4)
/dev/sdb3 on /home type btrfs (rw,relatime,subvol=@home,thread_pool=4)

/dev/sda2 on /media/idisk/root ext4 (rw,relatime)
/dev/sda3 on /media/idisk/home ext4 (rw,relatime)

none on /sys/fs/cgroup type tmpfs (rw)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,relatime,cpuset,release_agent=/run/cgmanager/agents/cgm-release-agent.cpuset,clone_children)
cgroup on /sys/fs/cgroup/cpu type cgroup (rw,relatime,cpu,release_agent=/run/cgmanager/agents/cgm-release-agent.cpu)
cgroup on /sys/fs/cgroup/cpuacct type cgroup (rw,relatime,cpuacct,release_agent=/run/cgmanager/agents/cgm-release-agent.cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,relatime,memory,release_agent=/run/cgmanager/agents/cgm-release-agent.memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices,release_agent=/run/cgmanager/agents/cgm-release-agent.devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,relatime,freezer,release_agent=/run/cgmanager/agents/cgm-release-agent.freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,relatime,blkio,release_agent=/run/cgmanager/agents/cgm-release-agent.blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,relatime,perf_event,release_agent=/run/cgmanager/agents/cgm-release-agent.perf_event)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,relatime,hugetlb,release_agent=/run/cgmanager/agents/cgm-release-agent.hugetlb)

This is on an up-to-date 14.04 LTS system. The scheduler, by default is set to completely-fair-queue (cfq):

# cat /sys/block/sda/queue/scheduler
noop deadline [cfq]
# cat /sys/block/sdb/queue/scheduler
noop deadline [cfq]

I was able to find one related question. scheduler starves processes which suggests to use nice, but I'm already doing this.

Another related question with relevant information is: How do I change the noop scheduler

How can I make keyboard, mouse, and interactive-shells more responsive when the backup is running?

Thanks in advance.

Answer 1

Just a partial answer, done more research and experiments since asking which have solved my problem, and seeing there are no responses

There are known issues/bugs in the Linux kernel schedulers as of early 2016.

The short summary is that under different circumstances, cores remain idle even though there are runnable processes in the process queue.

References:

• The Linux Scheduler, a decade of wasted cores
• github repository with experimental fixes to scheduling

A switch from btrfs to ext4 can alleviate these issues:

I personally switched back from btrfs to ext4. I/O performance has noticeably improved.

A switch to SSD can further alleviate IO performance

SSDs have dropped significantly in price and reliability. A 2TB Samsung SSD (EVO 850) now costs a little over $600. Switching the system (root and home) to SSD now makes the intensive backup activity completely unnoticeable (system SSD is snappy while doing heavy writing to a regular ext4-formatted disk on the same system).

Finally: with SSD, the benefit of complex schedulers in the kernel seems to becoming questionable. I changed my default to noop with no noticeable degradation whatsoever in performance. I fact, with a noop scheduler, I see a reduction in system load, lower CPU scaling numbers, and lower hardware temperatures.

$ cat /sys/block/sda/queue/scheduler
[noop] deadline cfq

$ cat /proc/cpuinfo | grep  Hz
model name      : Intel(R) Core(TM) i7-4771 CPU @ 3.50GHz
cpu MHz         : 836.308
model name      : Intel(R) Core(TM) i7-4771 CPU @ 3.50GHz
cpu MHz         : 990.253
... similar low actual frequency scaling for all cores ...

Dec 2021 update

Michael Larabel @ Phoronix has run some benchmarks confirming the bottom line:

While some Linux distributions are still using MQ-Deadline or Kyber by default for NVMe SSD storage, using no I/O scheduler still tends to perform the best overall ...

• 20.04 NOTE: the noop scheduler has changed name in Linux kernel 5.x. It is now called none so if your system has no rotational disks and you want to switch to the noop scheduler, you may use this (must be run in a root shell, and must make sure you have the device correct):

# NOTE: 'sda' in the path below is just an example
# Your SSD may be on a different device
echo none > /sys/block/sda/queue/scheduler