6

I've been testing this stuff trying to reserve some CPUs in a host with two Intel E5645 but I can't get it working for some reason. Steps I followed:

  1. Edit /etc/default/grub and added isolcpus=0,1 to GRUB_CMDLINE_LINUX_DEFAULT

    GRUB_CMDLINE_LINUX_DEFAULT="quiet splash isolcpus=0,1"

  2. Run update-grub

  3. Reboot

After that, cat /proc/cmdline reveals:

BOOT_IMAGE=/boot/vmlinuz-3.2.0-26-generic root=UUID=52cfedad-40be-41b9-9f88-c282a7ae3f24 ro quiet splash isolcpus=0,1 vt.handoff=7

Tested using stress:

apt-get install stress && stress -c 24

Monitored using top and pressing 1 to display individual CPU stats. So far no CPUs are isolated from the scheduler and all of them are busted by stress.

Cpu0  : 99.7%us,  0.3%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu1  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu2  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu3  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu4  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu5  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu6  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu7  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu8  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu9  :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu10 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu11 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu12 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu13 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu14 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu15 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu16 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu17 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu18 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu19 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu20 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu21 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu22 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa
Cpu23 :100.0%us,  0.0%sy,  0.0%ni,  0.0%id,  0.0%wa

Tested the same stuff in CentOS 6 x86_64 and it works as expected.

Searched ubuntu and linux bugs database in launchpad but did not find anything so far.

Is it me being dumb or am I missing something? Hints?

Thanks!

Refs:

http://www.kernel.org/doc/Documentation/kernel-parameters.txt

3 Answers 3

4

Disabled SMT (Hyper-Threading) in BIOS, power cycled, tested again with HT off and isolcpus works as expected.

Re-enabled SMT in BIOS, power cycled, still works as expected.

Definitely not the best answer you could expect, but it worked for me nevertheless.

1
  • 2
    Are you saying that both reboots gave the same result regardless of the Hyper-Threading settings? In which case, maybe rebooting was the work around(?) Sep 15, 2014 at 9:45
3

isolcpus only isloate from ceratin user-space activity and will not isolate the CPU from the kernel activity (watchdogs, kworkers, linux stack ...). Moreover, you should not isolate cpu 0 (which has numerous dedicated activities like usb discovery, acpi timer setup, wrmsr/rdmsr dispatching .... which tend to amount to 2% with standard distro where all drivers are enabled by default)

You can verify which processes are running on which cores, by displaying all the threads sorted by core number.

ps -aFeL | cut -c 48- | sort -n

You also need to verify and modify the affinities in /proc/irq/* , in order to try to change the interrupt affinities.

You might dynamically spare cpus and remove "most" of linux kernel task using cpuset tool.

instead of

taskset -c 3-7,11-15 program args

try this

sudo apt install cpuset
cset set --list
cset shield -c 3-7,11-15
cset set --list
cset shield -e program -- args  

And then, you can appreciate the difference

ps -aFeL | cut -c 48- | sort -n

When running these 2 commands

cset shield -e stress -- -c 16
stress -c 16
0

Actually, isolcpus seems (by my testing) to behave differently on AMD and Intel CPUs.

Kernel: 4.10.0-38-generic x86_64 (64 bit) on both

/etc/default/grub added isolcpus=2 to the GRUB_CMDLINE_LINUX_DEFAULT= line, similar to the OP. Then sudo update-grub and reboot.

  • On Intel core i7 dual core with hyperthreading, ps -eF | grep " 2 " shows only a very few root-owned processes get started on hyperthread 2 (and they all seem to have /2 after the CMD).

  • On AMD quad core without hyperthreading, many root-owned processes get started on core 2 (and most don't have /2 after the CMD).

Is this a bug that will probably get fixed, or a feature?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.