How to get isolcpus kernel parameter working with Precise 12.04 amd64?

Question

I've been testing this stuff trying to reserve some CPUs in a host with two Intel E5645 but I can't get it working for some reason. Steps I followed:

1. Edit /etc/default/grub and added isolcpus=0,1 to GRUB_CMDLINE_LINUX_DEFAULT

   GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=0,1"

2. Run update-grub

3. Reboot

After that, cat /proc/cmdline reveals:

BOOT_IMAGE=/boot/vmlinuz-3.2.0-26-generic root=UUID=52cfedad-40be-41b9-9f88-c282a7ae3f24 ro quiet splash isolcpus=0,1 vt.handoff=7

Tested using stress:

apt-get install stress && stress -c 24

Monitored using top and pressing 1 to display individual CPU stats. So far no CPUs are isolated from the scheduler and all of them are busted by stress.

Cpu0  : 99.7%us,  0.3%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu1  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu2  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu3  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu4  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu5  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu6  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu7  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu8  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu9  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu10 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu11 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu12 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu13 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu14 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu15 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu16 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu17 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu18 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu19 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu20 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu21 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu22 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu23 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa

Tested the same stuff in CentOS 6 x86_64 and it works as expected.

Searched ubuntu and linux bugs database in launchpad but did not find anything so far.

Is it me being dumb or am I missing something? Hints?

Thanks!

Refs:

http://www.kernel.org/doc/Documentation/kernel-parameters.txt

Answer 1

Disabled SMT (Hyper-Threading) in BIOS, power cycled, tested again with HT off and isolcpus works as expected.

Re-enabled SMT in BIOS, power cycled, still works as expected.

Definitely not the best answer you could expect, but it worked for me nevertheless.

Answer 2

isolcpus only isloate from ceratin user-space activity and will not isolate the CPU from the kernel activity (watchdogs, kworkers, linux stack ...). Moreover, you should not isolate cpu 0 (which has numerous dedicated activities like usb discovery, acpi timer setup, wrmsr/rdmsr dispatching .... which tend to amount to 2% with standard distro where all drivers are enabled by default)

You can verify which processes are running on which cores, by displaying all the threads sorted by core number.

ps -aFeL | cut -c 48- | sort -n

You also need to verify and modify the affinities in /proc/irq/* , in order to try to change the interrupt affinities.

You might dynamically spare cpus and remove "most" of linux kernel task using cpuset tool.

instead of

taskset -c 3-7,11-15 program args

try this

sudo apt install cpuset
cset set --list
cset shield -c 3-7,11-15
cset set --list
cset shield -e program -- args  

And then, you can appreciate the difference

ps -aFeL | cut -c 48- | sort -n

When running these 2 commands

cset shield -e stress -- -c 16
stress -c 16

Answer 3

Actually, isolcpus seems (by my testing) to behave differently on AMD and Intel CPUs.

Kernel: 4.10.0-38-generic x86_64 (64 bit) on both

/etc/default/grub added isolcpus=2 to the GRUB_CMDLINE_LINUX_DEFAULT= line, similar to the OP. Then sudo update-grub and reboot.

• On Intel core i7 dual core with hyperthreading, ps -eF | grep " 2 " shows only a very few root-owned processes get started on hyperthread 2 (and they all seem to have /2 after the CMD).

• On AMD quad core without hyperthreading, many root-owned processes get started on core 2 (and most don't have /2 after the CMD).

Is this a bug that will probably get fixed, or a feature?