What are the rcuos/rcuob processes I'm seeing in top?

Question

I ran top, and I see lots of rcuos/rcuob processes. Here's a sample of top output:

top - 21:41:15 up 22 days, 19:20,  1 user,  load average: 0.00, 0.01, 0.05
Tasks: 332 total,   1 running, 331 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni, 99.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  32910904k total,  2232224k used, 30678680k free,   246636k buffers
Swap: 33517564k total,        0k used, 33517564k free,  1048244k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 3098 root      20   0 11.8g 113m 9.9m S    1  0.4  67:57.98 asterisk
   12 root      20   0     0    0    0 S    0  0.0   0:53.61 rcuos/3
 1130 root      39  19     0    0    0 S    0  0.0  78:26.78 kipmi0
 1735 root      20   0 15988  740  540 S    0  0.0  17:34.69 irqbalance
 1875 zabbix    20   0 89416 2472 1828 S    0  0.0  12:55.47 zabbix_agentd
    1 root      20   0 24316 2288 1336 S    0  0.0   0:05.06 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.25 kthreadd
    3 root      20   0     0    0    0 S    0  0.0   0:02.40 ksoftirqd/0
    5 root       0 -20     0    0    0 S    0  0.0   0:00.00 kworker/0:0H
    6 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/u48:0
    8 root      20   0     0    0    0 S    0  0.0   4:55.92 rcu_sched
    9 root      20   0     0    0    0 S    0  0.0   1:31.03 rcuos/0
   10 root      20   0     0    0    0 S    0  0.0   0:54.04 rcuos/1
   11 root      20   0     0    0    0 S    0  0.0   0:53.16 rcuos/2
   13 root      20   0     0    0    0 S    0  0.0   0:45.96 rcuos/4
   14 root      20   0     0    0    0 S    0  0.0   0:41.84 rcuos/5
   15 root      20   0     0    0    0 S    0  0.0   0:02.56 rcuos/6
   16 root      20   0     0    0    0 S    0  0.0   0:07.25 rcuos/7
   17 root      20   0     0    0    0 S    0  0.0   0:12.06 rcuos/8
   18 root      20   0     0    0    0 S    0  0.0   0:12.14 rcuos/9
   19 root      20   0     0    0    0 S    0  0.0   0:11.21 rcuos/10
   20 root      20   0     0    0    0 S    0  0.0   0:05.66 rcuos/11
   21 root      20   0     0    0    0 S    0  0.0   1:39.42 rcuos/12
   22 root      20   0     0    0    0 S    0  0.0   0:35.66 rcuos/13
   23 root      20   0     0    0    0 S    0  0.0   0:26.46 rcuos/14
   24 root      20   0     0    0    0 S    0  0.0   0:00.88 rcuos/15
   25 root      20   0     0    0    0 S    0  0.0   0:00.30 rcuos/16
   26 root      20   0     0    0    0 S    0  0.0   0:31.93 rcuos/17
   27 root      20   0     0    0    0 S    0  0.0   0:18.58 rcuos/18
   28 root      20   0     0    0    0 S    0  0.0   0:20.53 rcuos/19
   29 root      20   0     0    0    0 S    0  0.0   0:44.95 rcuos/20
   30 root      20   0     0    0    0 S    0  0.0   0:01.92 rcuos/21
   31 root      20   0     0    0    0 S    0  0.0   0:00.24 rcuos/22
   32 root      20   0     0    0    0 S    0  0.0   0:05.01 rcuos/23
   33 root      20   0     0    0    0 S    0  0.0   0:00.00 rcu_bh
   34 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/0
   35 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/1
   36 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/2
   37 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/3
   38 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/4
   39 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/5
   40 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/6
   41 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/7
   42 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/8
   43 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/9
   44 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/10
   45 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/11
   46 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/12
   47 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/13
   48 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/14
   49 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/15
   50 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/16
   51 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/17
   52 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/18
   53 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/19
   54 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/20
   55 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/21
   56 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/22
   57 root      20   0     0    0    0 S    0  0.0   0:00.00 rcuob/23
   58 root      RT   0     0    0    0 S    0  0.0   0:00.87 migration/0
   59 root      RT   0     0    0    0 S    0  0.0   0:08.47 watchdog/0
   60 root      RT   0     0    0    0 S    0  0.0   0:08.41 watchdog/1

What are these processes?

Answer 1

The key article explaining this is here: https://lwn.net/Articles/522262/

For faster synchronization, many structures in the kernel are "RCU", or read-copy-update. Many threads can read simultaneously without blocking, and threads that want to alter the structure can do so quickly by copying the relevant data to a new location and changing the reference ("removal"). However, the old data must live for some time longer so that reading threads can finish their work. When the old data no longer has readers working on it, it can finally be deleted ("reclamation").

The rcuos, rcuob threads are there to handle the reclamation part of this process. It used to be handled as part of software IRQ, which meant it could take processor power away from critical user processes at inopportune times. By moving this to a bunch of dedicated rcu** threads, the system can be tuned to make this reclamation process less intrusive.

Reducing OS jitter is not the purpose of these threads, though it may be the main reason they exist.

Answer 2

According to the ubuntuforums they are coming from the "Reduce OS Jitter" which is turned on by the following kernel flags

CONFIG_RCU_NOCB_CPU
CONFIG_RCU_NOCB_CPU_ALL
CONFIG_RCU_STALL_COMMON
CONFIG_RCU_USER_QS

Since 13.10 (Saucy) they are enabled by default.

See:

• http://ubuntuforums.org/showthread.php?t=2225218
• https://www.kernel.org/doc/Documentation/kernel-per-CPU-kthreads.txt