1

I've been searching for the solution to this problem for days, but I couldn't find the exact information I need. I'd be grateful if anyone could help.

Aug 28 14:23:06 AMS kernel: [1499509.120893] watchdog: BUG: soft lockup - CPU#111 stuck for 23s! [swapper/111:0]
Aug 28 14:23:06 AMS kernel: [1499509.121452] Modules linked in: amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass mlx5_core(OE) mlxdevm(OE) auxiliary(OE) mlx_compat(OE) mlxfw(OE) nls_iso8859_1 ipmi_ssif rndis_host cdc_ether usbnet joydev input_leds mii k10temp shpchp ipmi_si ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 multipath linear ast i2c_algo_bit ttm raid0 drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect crc32_pclmul sysimgblt ghash_clmulni_intel fb_sys_fops pcbc ixgbe(OE) aesni_intel aes_x86_64 crypto_simd glue_helper hid_generic cryptd dca bnxt_en usbhid ahci ptp nvme drm hid
Aug 28 14:23:06 AMS kernel: [1499509.121485]  devlink libahci i2c_piix4 nvme_core pps_core
Aug 28 14:23:06 AMS kernel: [1499509.121489] CPU: 111 PID: 0 Comm: swapper/111 Tainted: G        W  OEL   4.15.0-189-generic #200-Ubuntu
Aug 28 14:23:06 AMS kernel: [1499509.121490] Hardware name: Supermicro AS -1114S-WN10RT/H12SSW-NTR, BIOS 2.4 04/13/2022
Aug 28 14:23:06 AMS kernel: [1499509.121495] RIP: 0010:_raw_spin_unlock_irqrestore+0x15/0x20
Aug 28 14:23:06 AMS kernel: [1499509.121495] RSP: 0018:ffff9f198e1c3e48 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff11
Aug 28 14:23:06 AMS kernel: [1499509.121496] RAX: 0000000000000074 RBX: ffffcbe03e240000 RCX: 000000018040003c
Aug 28 14:23:06 AMS kernel: [1499509.121497] RDX: 000000018040003d RSI: 0000000000000286 RDI: 0000000000000286
Aug 28 14:23:06 AMS kernel: [1499509.121497] RBP: ffff9f198e1c3e48 R08: ffff9f193e758a00 R09: 000000018040003c
Aug 28 14:23:06 AMS kernel: [1499509.121498] R10: ffff9f198e1c3d38 R11: 0000000000000000 R12: ffff9f1979d0c858
Aug 28 14:23:06 AMS kernel: [1499509.121498] R13: ffffcbe03e242008 R14: 0000000000000286 R15: 0000000000000011
Aug 28 14:23:06 AMS kernel: [1499509.121499] FS:  0000000000000000(0000) GS:ffff9f198e1c0000(0000) knlGS:0000000000000000
Aug 28 14:23:06 AMS kernel: [1499509.121500] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 28 14:23:06 AMS kernel: [1499509.121501] CR2: 00005575716aa000 CR3: 0000007be8e0a000 CR4: 0000000000340ee0
Aug 28 14:23:06 AMS kernel: [1499509.121501] Call Trace:
Aug 28 14:23:06 AMS kernel: [1499509.121503]  <IRQ>
Aug 28 14:23:06 AMS kernel: [1499509.121507]  fq_flush_timeout+0x77/0xa0
Aug 28 14:23:06 AMS kernel: [1499509.121508]  ? fq_ring_free+0xd0/0xd0
Aug 28 14:23:06 AMS kernel: [1499509.121510]  call_timer_fn+0x30/0x130
Aug 28 14:23:06 AMS kernel: [1499509.121511]  run_timer_softirq+0x3f3/0x430
Aug 28 14:23:06 AMS kernel: [1499509.121514]  ? timerqueue_add+0x5c/0x70
Aug 28 14:23:06 AMS kernel: [1499509.121515]  ? ktime_get+0x43/0xb0
Aug 28 14:23:06 AMS kernel: [1499509.121517]  ? lapic_next_event+0x20/0x30
Aug 28 14:23:06 AMS kernel: [1499509.121518]  __do_softirq+0xe4/0x2d4
Aug 28 14:23:06 AMS kernel: [1499509.121521]  irq_exit+0xc5/0xd0
Aug 28 14:23:06 AMS kernel: [1499509.121522]  smp_apic_timer_interrupt+0x79/0x140
Aug 28 14:23:06 AMS kernel: [1499509.121523]  apic_timer_interrupt+0x90/0xa0
Aug 28 14:23:06 AMS kernel: [1499509.121524]  </IRQ>
Aug 28 14:23:06 AMS kernel: [1499509.121526] RIP: 0010:cpuidle_enter_state+0xa7/0x2f0
Aug 28 14:23:06 AMS kernel: [1499509.121526] RSP: 0018:ffffabe04cb33e68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11
Aug 28 14:23:06 AMS kernel: [1499509.121527] RAX: ffff9f198e1e6940 RBX: 000553c5ef1f8a5d RCX: 000000000000001f
Aug 28 14:23:06 AMS kernel: [1499509.121528] RDX: 000553c5ef1f8a5d RSI: ffffffe6eb5eefe4 RDI: 0000000000000000
Aug 28 14:23:06 AMS kernel: [1499509.121528] RBP: ffffabe04cb33ea8 R08: 0000000000000004 R09: 0000000000026180
Aug 28 14:23:06 AMS kernel: [1499509.121529] R10: ffffabe04cb33e38 R11: 000aa7b70999fc24 R12: ffff9f195d9caa00
Aug 28 14:23:06 AMS kernel: [1499509.121529] R13: 0000000000000002 R14: ffffffffacf81d58 R15: 0000000000000000
Aug 28 14:23:06 AMS kernel: [1499509.121531]  ? cpuidle_enter_state+0x97/0x2f0
Aug 28 14:23:06 AMS kernel: [1499509.121532]  cpuidle_enter+0x17/0x20
Aug 28 14:23:06 AMS kernel: [1499509.121534]  call_cpuidle+0x23/0x40
Aug 28 14:23:06 AMS kernel: [1499509.121535]  do_idle+0x18c/0x1f0
Aug 28 14:23:06 AMS kernel: [1499509.121536]  cpu_startup_entry+0x73/0x80
Aug 28 14:23:06 AMS kernel: [1499509.121537]  start_secondary+0x1ab/0x200
Aug 28 14:23:06 AMS kernel: [1499509.121539]  secondary_startup_64+0xa5/0xb0
Aug 28 14:23:06 AMS kernel: [1499509.121540] Code: c4 53 ff ff ff 7f 89 d0 5d c3 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 c6
Aug 28 14:23:35 AMS kernel: [1499537.640930] watchdog: BUG: soft lockup - CPU#111 stuck for 23s! [swapper/111:0]
Aug 28 14:23:35 AMS kernel: [1499537.641473] Modules linked in: amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass mlx5_core(OE) mlxdevm(OE) auxiliary(OE) mlx_compat(OE) mlxfw(OE) nls_iso8859_1 ipmi_ssif rndis_host cdc_ether usbnet joydev input_leds mii k10temp shpchp ipmi_si ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 multipath linear ast i2c_algo_bit ttm raid0 drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect crc32_pclmul sysimgblt ghash_clmulni_intel fb_sys_fops pcbc ixgbe(OE) aesni_intel aes_x86_64 crypto_simd glue_helper hid_generic cryptd dca bnxt_en usbhid ahci ptp nvme drm hid
Aug 28 14:23:35 AMS kernel: [1499537.641506]  devlink libahci i2c_piix4 nvme_core pps_core
Aug 28 14:23:35 AMS kernel: [1499537.641511] CPU: 111 PID: 0 Comm: swapper/111 Tainted: G        W  OEL   4.15.0-189-generic #200-Ubuntu
Aug 28 14:23:35 AMS kernel: [1499537.641511] Hardware name: Supermicro AS -1114S-WN10RT/H12SSW-NTR, BIOS 2.4 04/13/2022
Aug 28 14:23:35 AMS kernel: [1499537.641516] RIP: 0010:_raw_spin_unlock_irqrestore+0x15/0x20
Aug 28 14:23:35 AMS kernel: [1499537.641517] RSP: 0018:ffff9f198e1c3e48 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff11
Aug 28 14:23:35 AMS kernel: [1499537.641518] RAX: 000000000000008f RBX: ffffcbe03e700000 RCX: 0000000100400038
Aug 28 14:23:35 AMS kernel: [1499537.641519] RDX: 0000000100400039 RSI: 0000000000000286 RDI: 0000000000000286
Aug 28 14:23:35 AMS kernel: [1499537.641519] RBP: ffff9f198e1c3e48 R08: ffff9f187980d400 R09: 0000000100400038
Aug 28 14:23:35 AMS kernel: [1499537.641520] R10: ffff9f198e1c3d38 R11: 0000000000000000 R12: ffff9f1979d0c858
Aug 28 14:23:35 AMS kernel: [1499537.641520] R13: ffffcbe03e702008 R14: 0000000000000286 R15: 0000000000000024
Aug 28 14:23:35 AMS kernel: [1499537.641521] FS:  0000000000000000(0000) GS:ffff9f198e1c0000(0000) knlGS:0000000000000000
Aug 28 14:23:35 AMS kernel: [1499537.641522] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 28 14:23:35 AMS kernel: [1499537.641522] CR2: 00005575716aa000 CR3: 0000007be8e0a000 CR4: 0000000000340ee0
Aug 28 14:23:35 AMS kernel: [1499537.641523] Call Trace:
Aug 28 14:23:35 AMS kernel: [1499537.641525]  <IRQ>
Aug 28 14:23:35 AMS kernel: [1499537.641529]  fq_flush_timeout+0x77/0xa0
Aug 28 14:23:35 AMS kernel: [1499537.641530]  ? fq_ring_free+0xd0/0xd0
Aug 28 14:23:35 AMS kernel: [1499537.641533]  call_timer_fn+0x30/0x130
Aug 28 14:23:35 AMS kernel: [1499537.641534]  run_timer_softirq+0x3f3/0x430
Aug 28 14:23:35 AMS kernel: [1499537.641536]  ? timerqueue_add+0x5c/0x70
Aug 28 14:23:35 AMS kernel: [1499537.641537]  ? ktime_get+0x43/0xb0
Aug 28 14:23:35 AMS kernel: [1499537.641539]  ? lapic_next_event+0x20/0x30
Aug 28 14:23:35 AMS kernel: [1499537.641541]  __do_softirq+0xe4/0x2d4
Aug 28 14:23:35 AMS kernel: [1499537.641543]  irq_exit+0xc5/0xd0
Aug 28 14:23:35 AMS kernel: [1499537.641544]  smp_apic_timer_interrupt+0x79/0x140
Aug 28 14:23:35 AMS kernel: [1499537.641545]  apic_timer_interrupt+0x90/0xa0
Aug 28 14:23:35 AMS kernel: [1499537.641546]  </IRQ>
Aug 28 14:23:35 AMS kernel: [1499537.641548] RIP: 0010:cpuidle_enter_state+0xa7/0x2f0
Aug 28 14:23:35 AMS kernel: [1499537.641549] RSP: 0018:ffffabe04cb33e68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11
Aug 28 14:23:35 AMS kernel: [1499537.641550] RAX: ffff9f198e1e6940 RBX: 000553c5ef1f8a5d RCX: 000000000000001f
Aug 28 14:23:35 AMS kernel: [1499537.641550] RDX: 000553c5ef1f8a5d RSI: ffffffe6eb5eefe4 RDI: 0000000000000000
Aug 28 14:23:35 AMS kernel: [1499537.641551] RBP: ffffabe04cb33ea8 R08: 0000000000000004 R09: 0000000000026180
Aug 28 14:23:35 AMS kernel: [1499537.641551] R10: ffffabe04cb33e38 R11: 000aa7b70999fc24 R12: ffff9f195d9caa00
Aug 28 14:23:35 AMS kernel: [1499537.641552] R13: 0000000000000002 R14: ffffffffacf81d58 R15: 0000000000000000
Aug 28 14:23:35 AMS kernel: [1499537.641553]  ? cpuidle_enter_state+0x97/0x2f0
Aug 28 14:23:35 AMS kernel: [1499537.641555]  cpuidle_enter+0x17/0x20
Aug 28 14:23:35 AMS kernel: [1499537.641557]  call_cpuidle+0x23/0x40
Aug 28 14:23:35 AMS kernel: [1499537.641558]  do_idle+0x18c/0x1f0
Aug 28 14:23:35 AMS kernel: [1499537.641559]  cpu_startup_entry+0x73/0x80
Aug 28 14:23:35 AMS kernel: [1499537.641560]  start_secondary+0x1ab/0x200
Aug 28 14:23:35 AMS kernel: [1499537.641562]  secondary_startup_64+0xa5/0xb0
Aug 28 14:23:35 AMS kernel: [1499537.641563] Code: c4 53 ff ff ff 7f 89 d0 5d c3 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 c6
Aug 28 14:23:41 AMS kernel: [1499544.522463] INFO: rcu_sched detected stalls on CPUs/tasks:
Aug 28 14:23:41 AMS kernel: [1499544.523009]    111-....: (1 GPs behind) idle=b76/1/0 softirq=28818911/28818912 fqs=6142
Aug 28 14:23:41 AMS kernel: [1499544.523432]    (detected by 77, t=15002 jiffies, g=79801827, c=79801826, q=266623)
Aug 28 14:23:41 AMS kernel: [1499544.523850] Sending NMI from CPU 77 to CPUs 111:
Aug 28 14:23:41 AMS kernel: [1499544.524898] NMI backtrace for cpu 111
Aug 28 14:23:41 AMS kernel: [1499544.524899] CPU: 111 PID: 0 Comm: swapper/111 Tainted: G        W  OEL   4.15.0-189-generic #200-Ubuntu
Aug 28 14:23:41 AMS kernel: [1499544.524899] Hardware name: Supermicro AS -1114S-WN10RT/H12SSW-NTR, BIOS 2.4 04/13/2022
Aug 28 14:23:41 AMS kernel: [1499544.524899] RIP: 0010:native_queued_spin_lock_slowpath+0x180/0x1c0
Aug 28 14:23:41 AMS kernel: [1499544.524900] RSP: 0018:ffff9f198e1c3d68 EFLAGS: 00000046
Aug 28 14:23:41 AMS kernel: [1499544.524900] RAX: 0000000000000000 RBX: 0000000000000082 RCX: 0000000000000070
Aug 28 14:23:41 AMS kernel: [1499544.524901] RDX: ffff9f198e1e7500 RSI: 0000000001c00000 RDI: ffff9f1979d0c140
Aug 28 14:23:41 AMS kernel: [1499544.524901] RBP: ffff9f198e1c3d68 R08: ffff9f198cee7500 R09: 0000000100400032
Aug 28 14:23:41 AMS kernel: [1499544.524901] R10: ffff9f198e1c3d38 R11: 0000000000000000 R12: 0000000000000007
Aug 28 14:23:41 AMS kernel: [1499544.524902] R13: ffff9f0de72e9080 R14: ffff9f1979d0c140 R15: 000ffffffff57080
Aug 28 14:23:41 AMS kernel: [1499544.524902] FS:  0000000000000000(0000) GS:ffff9f198e1c0000(0000) knlGS:0000000000000000
Aug 28 14:23:41 AMS kernel: [1499544.524902] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 28 14:23:41 AMS kernel: [1499544.524902] CR2: 00005575716aa000 CR3: 0000007be8e0a000 CR4: 0000000000340ee0
Aug 28 14:23:41 AMS kernel: [1499544.524903] Call Trace:
Aug 28 14:23:41 AMS kernel: [1499544.524903]  <IRQ>
Aug 28 14:23:41 AMS kernel: [1499544.524903]  _raw_spin_lock_irqsave+0x37/0x40
Aug 28 14:23:41 AMS kernel: [1499544.524903]  __free_iova+0x19/0x40
Aug 28 14:23:41 AMS kernel: [1499544.524903]  free_iova+0x22/0x30
Aug 28 14:23:41 AMS kernel: [1499544.524904]  free_iova_fast+0x49/0x250
Aug 28 14:23:41 AMS kernel: [1499544.524904]  fq_ring_free+0xa1/0xd0
Aug 28 14:23:41 AMS kernel: [1499544.524904]  fq_flush_timeout+0x6c/0xa0
Aug 28 14:23:41 AMS kernel: [1499544.524904]  ? fq_ring_free+0xd0/0xd0
Aug 28 14:23:41 AMS kernel: [1499544.524905]  call_timer_fn+0x30/0x130
Aug 28 14:23:41 AMS kernel: [1499544.524905]  run_timer_softirq+0x3f3/0x430
Aug 28 14:23:41 AMS kernel: [1499544.524905]  ? timerqueue_add+0x5c/0x70
Aug 28 14:23:41 AMS kernel: [1499544.524905]  ? ktime_get+0x43/0xb0
Aug 28 14:23:41 AMS kernel: [1499544.524905]  ? lapic_next_event+0x20/0x30
Aug 28 14:23:41 AMS kernel: [1499544.524906]  __do_softirq+0xe4/0x2d4
Aug 28 14:23:41 AMS kernel: [1499544.524906]  irq_exit+0xc5/0xd0
Aug 28 14:23:41 AMS kernel: [1499544.524906]  smp_apic_timer_interrupt+0x79/0x140
Aug 28 14:23:41 AMS kernel: [1499544.524906]  apic_timer_interrupt+0x90/0xa0
Aug 28 14:23:41 AMS kernel: [1499544.524906]  </IRQ>
Aug 28 14:23:41 AMS kernel: [1499544.524907] RIP: 0010:cpuidle_enter_state+0xa7/0x2f0
Aug 28 14:23:41 kernel: [1499544.524907] RSP: 0018:ffffabe04cb33e68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11
Aug 28 14:23:41 kernel: [1499544.524907] RAX: ffff9f198e1e6940 RBX: 000553c5ef1f8a5d RCX: 000000000000001f
Aug 28 14:23:41 kernel: [1499544.524908] RDX: 000553c5ef1f8a5d RSI: ffffffe6eb5eefe4 RDI: 0000000000000000
Aug 28 14:23:41 kernel: [1499544.524908] RBP: ffffabe04cb33ea8 R08: 0000000000000004 R09: 0000000000026180
Aug 28 14:23:41 kernel: [1499544.524908] R10: ffffabe04cb33e38 R11: 000aa7b70999fc24 R12: ffff9f195d9caa00
Aug 28 14:23:41 kernel: [1499544.524909] R13: 0000000000000002 R14: ffffffffacf81d58 R15: 0000000000000000
Aug 28 14:23:41 kernel: [1499544.524909]  ? cpuidle_enter_state+0x97/0x2f0
Aug 28 14:23:41 kernel: [1499544.524909]  cpuidle_enter+0x17/0x20
Aug 28 14:23:41 kernel: [1499544.524909]  call_cpuidle+0x23/0x40
Aug 28 14:23:41 kernel: [1499544.524909]  do_idle+0x18c/0x1f0
Aug 28 14:23:41 kernel: [1499544.524910]  cpu_startup_entry+0x73/0x80
Aug 28 14:23:41 kernel: [1499544.524910]  start_secondary+0x1ab/0x200
Aug 28 14:23:41 kernel: [1499544.524910]  secondary_startup_64+0xa5/0xb0
Aug 28 14:23:41 kernel: [1499544.524910] Code: be c1 e9 12 83 e0 03 83 e9 01 48 c1 e0 04 48 63 c9 48 05 00 75 02 00 48 03 04 cd e0 16 bb ac 48 89 10 8b 42 08 85 c0 75 09 f3 90 <8b> 42 08 85 c0 74 f7 4c 8b 02 4d 85 c0 0f 84 5e ff ff ff 41 0f
Aug 28 14:24:06 kernel: [1499568.946118] watchdog: BUG: soft lockup - CPU#111 stuck for 23s! [swapper/111:0]
Aug 28 14:24:06 kernel: [1499568.947040] Modules linked in: amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass mlx5_core(OE) mlxdevm(OE) auxiliary(OE) mlx_compat(OE) mlxfw(OE) nls_iso8859_1 ipmi_ssif rndis_host cdc_ether usbnet joydev input_leds mii k10temp shpchp ipmi_si ipmi_devintf ipmi_msghandler mac_hid sch_fq_codel ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables x_tables autofs4 btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 multipath linear ast i2c_algo_bit ttm raid0 drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect crc32_pclmul sysimgblt ghash_clmulni_intel fb_sys_fops pcbc ixgbe(OE) aesni_intel aes_x86_64 crypto_simd glue_helper hid_generic cryptd dca bnxt_en usbhid ahci ptp nvme drm hid
Aug 28 14:24:06 kernel: [1499568.947096]  devlink libahci i2c_piix4 nvme_core pps_core
Aug 28 14:24:06 kernel: [1499568.947104] CPU: 111 PID: 0 Comm: swapper/111 Tainted: G        W  OEL   4.15.0-189-generic #200-Ubuntu
Aug 28 14:24:06 kernel: [1499568.947106] Hardware name: Supermicro AS -1114S-WN10RT/H12SSW-NTR, BIOS 2.4 04/13/2022
Aug 28 14:24:06 kernel: [1499568.947113] RIP: 0010:_raw_spin_unlock_irqrestore+0x15/0x20
Aug 28 14:24:06 kernel: [1499568.947115] RSP: 0018:ffff9f198e1c3e48 EFLAGS: 00000286 ORIG_RAX: ffffffffffffff11
Aug 28 14:24:06 kernel: [1499568.947117] RAX: 00000000000009c0 RBX: ffffcbe03ea00000 RCX: 0000000000000030
Aug 28 14:24:06 kernel: [1499568.947118] RDX: 0000000000000001 RSI: 0000000000000286 RDI: 0000000000000286
Aug 28 14:24:06 kernel: [1499568.947119] RBP: ffff9f198e1c3e48 R08: ffff000000000000 R09: ffffffffffffffff
Aug 28 14:24:06 kernel: [1499568.947120] R10: ffff9f198e1c3d38 R11: 0000000000000000 R12: ffff9f1979d0c858
Aug 28 14:24:06 kernel: [1499568.947121] R13: ffffcbe03ea02008 R14: 0000000000000286 R15: 0000000000000030
Aug 28 14:24:06 kernel: [1499568.947124] FS:  0000000000000000(0000) GS:ffff9f198e1c0000(0000) knlGS:0000000000000000
Aug 28 14:24:06 kernel: [1499568.947125] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 28 14:24:06 kernel: [1499568.947126] CR2: 00005575716aa000 CR3: 0000007be8e0a000 CR4: 0000000000340ee0
Aug 28 14:24:06 kernel: [1499568.947127] Call Trace:
Aug 28 14:24:06 kernel: [1499568.947130]  <IRQ>
Aug 28 14:24:06 kernel: [1499568.947135]  fq_flush_timeout+0x77/0xa0
Aug 28 14:24:06 kernel: [1499568.947137]  ? fq_ring_free+0xd0/0xd0
Aug 28 14:24:06 kernel: [1499568.947141]  call_timer_fn+0x30/0x130
Aug 28 14:24:06 kernel: [1499568.947142]  run_timer_softirq+0x3f3/0x430
Aug 28 14:24:06 kernel: [1499568.947146]  ? timerqueue_add+0x5c/0x70
Aug 28 14:24:06 kernel: [1499568.947148]  ? ktime_get+0x43/0xb0
Aug 28 14:24:06 kernel: [1499568.947151]  ? lapic_next_event+0x20/0x30
Aug 28 14:24:06 kernel: [1499568.947154]  __do_softirq+0xe4/0x2d4
Aug 28 14:24:06 kernel: [1499568.947158]  irq_exit+0xc5/0xd0
Aug 28 14:24:06 kernel: [1499568.947160]  smp_apic_timer_interrupt+0x79/0x140
Aug 28 14:24:06 kernel: [1499568.947162]  apic_timer_interrupt+0x90/0xa0
Aug 28 14:24:06 kernel: [1499568.947164]  </IRQ>
Aug 28 14:24:06 kernel: [1499568.947168] RIP: 0010:cpuidle_enter_state+0xa7/0x2f0
Aug 28 14:24:06 kernel: [1499568.947169] RSP: 0018:ffffabe04cb33e68 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11
Aug 28 14:24:06 kernel: [1499568.947170] RAX: ffff9f198e1e6940 RBX: 000553c5ef1f8a5d RCX: 000000000000001f
Aug 28 14:24:06 kernel: [1499568.947171] RDX: 000553c5ef1f8a5d RSI: ffffffe6eb5eefe4 RDI: 0000000000000000
Aug 28 14:24:06 kernel: [1499568.947172] RBP: ffffabe04cb33ea8 R08: 0000000000000004 R09: 0000000000026180
Aug 28 14:24:06 kernel: [1499568.947173] R10: ffffabe04cb33e38 R11: 000aa7b70999fc24 R12: ffff9f195d9caa00
Aug 28 14:24:06 kernel: [1499568.947174] R13: 0000000000000002 R14: ffffffffacf81d58 R15: 0000000000000000
Aug 28 14:24:06 kernel: [1499568.947177]  ? cpuidle_enter_state+0x97/0x2f0
Aug 28 14:24:06 kernel: [1499568.947180]  cpuidle_enter+0x17/0x20
Aug 28 14:24:06 kernel: [1499568.947183]  call_cpuidle+0x23/0x40
Aug 28 14:24:06 kernel: [1499568.947185]  do_idle+0x18c/0x1f0
Aug 28 14:24:06 kernel: [1499568.947187]  cpu_startup_entry+0x73/0x80
Aug 28 14:24:06 kernel: [1499568.947190]  start_secondary+0x1ab/0x200
Aug 28 14:24:06 kernel: [1499568.947192]  secondary_startup_64+0xa5/0xb0
Aug 28 14:24:06 kernel: [1499568.947194] Code: c4 53 ff ff ff 7f 89 d0 5d c3 90 90 90 90 90 90 90 90 90 90 90 90 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 c6
4
  • 2
    Your kernel is tainted, with an external unsigned module. Please explain the need for such. Also please provide some context as to what your system is doing when this occurs. Is this issue new with this kernel? Does going back to the previous kernel make a difference? What version of Ubuntu? Looks like 18.04, based on kernel version. Aug 28, 2022 at 14:06
  • Sorry for giving incomplete information. # cat /etc/os-release NAME="Ubuntu" VERSION="18.04.6 LTS (Bionic Beaver)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 18.04.6 LTS" VERSION_ID="18.04" HOME_URL="ubuntu.com" SUPPORT_URL="help.ubuntu.com" BUG_REPORT_URL="bugs.launchpad.net/ubuntu" PRIVACY_POLICY_URL="ubuntu.com/legal/terms-and-policies/privacy-policy" VERSION_CODENAME=bionic UBUNTU_CODENAME=bionic 4.15.0-189-generic
    – Ahmet
    Aug 28, 2022 at 14:46
  • This problem has been going on for about 2 months. I've tried 4 different kernel versions so far. none of them worked. I usually see this problem after kworker is using excessive cpu. I/O jumps to 99% and the server hangs for a short time. I can see this over ssh as well.
    – Ahmet
    Aug 28, 2022 at 14:50
  • If it is always CPU #111, then I would try taking that CPU offline. echo 0 | sudo tee /sys/devices/system/cpu/cpu111/online. Sep 1, 2022 at 0:46

0

You must log in to answer this question.

Browse other questions tagged .