NMI watchdog: BUG: soft lockup - CPU#22 stuck for 23s! [slurmstepd:188444]
Modules linked in: nvidia_uvm(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 beegfs(OE) dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm i40iw mlx5_ib ib_core dm_mirror dm_region_hash dm_log dm_mod vfat fat iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw
gf128mul glue_helper ablk_helper cryptd pcspkr joydev ipmi_ssif sg mei_me mei shpchp lpc_ich i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) raid1 sd_mod crc_t10dif crct10dif_generic nvidia(POE) ast i2c_algo_bit mlx5_core crct10dif_pclmul drm_kms_helper crct10dif_common crc32c_intel i40e ttm syscopyarea sysfillrect sysimgblt fb_sys_fops drm devlink ahci libahci ptp pps_core libata i2c_core wmi
CPU: 22 PID: 188444 Comm: slurmstepd Tainted: P           OEL ------------   3.10.0-693.11.6.el7.x86_64 #1
Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 3.3 02/21/2020
task: ffff887f77b19fa0 ti: ffff8800537fc000 task.ti: ffff8800537fc000
RIP: 0010:[<ffffffff816ade84>]  [<ffffffff816ade84>] _raw_spin_lock_irqsave+0x14/0x40
RSP: 0018:ffff8800537ffcc8  EFLAGS: 00000246
RAX: 0000000000000246 RBX: ffff8800537ffc38 RCX: ffff883f78360800
RDX: ffff883f78360800 RSI: 0000000000000001 RDI: ffff883f783609e4
RBP: ffff8800537ffcd0 R08: ffff883f77b35800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8800537ffc28
R13: ffff8800537ffc28 R14: ffff887f7da95b60 R15: 0000000000000001
FS:  00002ac69d906440(0000) GS:ffff887f7da80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002ac6a1397ff8 CR3: 0000007af76d2000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff811f3c38>] mem_cgroup_move_account+0xa8/0x160
[<ffffffff811f5338>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff811f56b4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110f467>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110f712>] cgroup_rmdir+0x22/0x40
[<ffffffff812114f6>] vfs_rmdir+0xd6/0x150
[<ffffffff81214191>] do_rmdir+0x1f1/0x220
[<ffffffff816b3801>] ? __do_page_fault+0x171/0x450
[<ffffffff81215396>] SyS_rmdir+0x16/0x20
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 89 c6 48 89 e5 e8 a1 28 ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 9c 58 0f 1f 44 00 00 48 89 c3 <fa> 66 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 85 c0 75 

NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [slurmstepd:134221]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev skx_edac iTCO_wdt edac_core iTCO_vendor_support intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass mgag200 crc32_pclmul ttm crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sr_mod cdrom mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge
stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) megaraid_sas aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil tg3(OE) e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci libata igb(OE) i2c_algo_bit i2c_core dca i40e(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 11 PID: 134221 Comm: slurmstepd Tainted: G        W  OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: FUJITSU PRIMERGY RX2540 M5/D3384-B1, BIOS V5.0.0.14 R1.15.0 for D3384-B1x                    10/17/2019
task: ffff882a927daf70 ti: ffff881d47370000 task.ti: ffff881d47370000
RIP: 0010:[<ffffffff811f1ca7>]  [<ffffffff811f1ca7>] mem_cgroup_move_account+0x57/0x160
RSP: 0018:ffff881d47373d10  EFLAGS: 00000202
RAX: 0000000000000246 RBX: ffff882fdf111ae0 RCX: 0000000000000001
RDX: 000060a020806768 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff881d47373d50 R08: ffff8817a3f5d000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 00000000d72006a4 R14: ffff881d47373c68 R15: ffff881d47373c68
FS:  00002aaaaaafd180(0000) GS:ffff882fdf100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aab580008f0 CR3: 0000001910633000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000246 00000001811959a2 ffff8817a3f5d000 ffffea01390b0ec0
ffff885a6d079440 ffff88603ffd7540 ffff8848b9cac3b0 ffffea01390b0ee0
ffff881d47373dd0 ffffffff811f33f8 0000000200000003 ffff8828ab755100
Call Trace:
[<ffffffff811f33f8>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 8b 07 f6 c4 40 0f 84 09 01 00 00 f0 0f ba 2b 00 19 c0 85 c0 0f 85 05 01 00 00 48 8b 03 41 bf ea ff ff ff a8 02 75 2d f0 80 23 fe <4c> 89 e6 4c 89 c7 e8 8e fd ff ff 4c 89 e6 4c 89 ef e8 83 fd ff 

NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [slurmstepd:167274]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache beegfs(OE) rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm i40iw mlx5_ib ib_core dm_mirror dm_region_hash dm_log dm_mod nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) vfat fat iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp nvidia(POE) coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul
ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev ipmi_ssif sg mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper mlx5_core crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt i40e fb_sys_fops ttm drm ahci devlink libahci ptp pps_core libata i2c_core wmi
CPU: 7 PID: 167274 Comm: slurmstepd Tainted: P           OE  ------------   3.10.0-693.11.6.el7.x86_64 #1
Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 3.3 02/21/2020
task: ffff887f73f28fd0 ti: ffff887bea2e0000 task.ti: ffff887bea2e0000
RIP: 0010:[<ffffffff811f3b9e>]  [<ffffffff811f3b9e>] mem_cgroup_move_account+0xe/0x160
RSP: 0018:ffff887bea2e3d18  EFLAGS: 00000246
RAX: ffff883ed6e7c840 RBX: ffffffff811922b0 RCX: ffff883ed6e7c400
RDX: ffff8840d0cb5750 RSI: 0000000000000001 RDI: ffffea01be0d5d40
RBP: ffff887bea2e3d20 R08: ffff883ed6e7c800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81192bcc
R13: ffff887bea2e3cc8 R14: ffff88807ffd7000 R15: ffff88807ffd7000
FS:  00002b3e95bf5340(0000) GS:ffff883f7e7c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007efde0dc14f0 CR3: 0000007ddcaf2000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff811f5338>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff811f56b4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110f467>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110f712>] cgroup_rmdir+0x22/0x40
[<ffffffff812114f6>] vfs_rmdir+0xd6/0x150
[<ffffffff81214191>] do_rmdir+0x1f1/0x220
[<ffffffff816b3801>] ? __do_page_fault+0x171/0x450
[<ffffffff81215396>] SyS_rmdir+0x16/0x20
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 00 00 00 41 b8 04 00 00 00 e9 33 ff ff ff 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 41 89 f1 48 89 e5 41 57 <41> 56 41 55 49 89 cd 41 54 49 89 fc 53 48 89 d3 48 83 ec 18 83 

NMI watchdog: BUG: soft lockup - CPU#27 stuck for 22s! [slurmstepd:68493]
Modules linked in: squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic nfs_layout_nfsv41_files crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log ses enclosure sd_mod sg dm_mod intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd serio_raw pcspkr uas usb_storage smartpqi nvme scsi_transport_sas hpilo hpwdt lpc_ich shpchp wmi acpi_power_meter rpcrdma ib_isert iscsi_target_mod target_core_mod crc_t10dif crct10dif_generic ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm
iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ipmi_devintf hfi1(OE) i2c_algo_bit i2c_core rdmavt(OE) crct10dif_pclmul crct10dif_common bnxt_en crc32c_intel ib_core ipmi_si fjes ipmi_msghandler
CPU: 27 PID: 68493 Comm: slurmstepd Tainted: G        W  OE  ------------ T 3.10.0-514.10.2.el7.x86_64 #1
Hardware name: HPE ProLiant DL560 Gen10/ProLiant DL560 Gen10, BIOS U34 03/16/2023
task: ffff8bace765ce70 ti: ffff8bb2a8640000 task.ti: ffff8bb2a8640000
RIP: 0010:[<ffffffff8168decb>]  [<ffffffff8168decb>] _raw_spin_lock_irqsave+0xb/0x60
RSP: 0018:ffff8bb2a8643d00  EFLAGS: 00000246
RAX: 0000000000000246 RBX: 000000000000000e RCX: ffff898222a31c00
RDX: ffff898222a31c00 RSI: 0000000000000001 RDI: ffff898222a31de4
RBP: ffff8bb2a8643d00 R08: ffff88fbce3a2c00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8bb2a8643c68
R13: ffff8bb2a8643c68 R14: ffff8bb2a8643c58 R15: ffff8bb2a8643c58
FS:  00002b84f06cf300(0000) GS:ffff89fc3fa40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001be13e0 CR3: 000002984aa1e000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8bb2a8643d50 ffffffff811efb58 ffff88fbce3a2c00 0000000181194873
ffff898222a31c00 ffffea082cde7700 ffff8a91eb750430 ffff8b003ffda540
ffff8a02353f9dc0 ffffea082cde7720 ffff8bb2a8643dd0 ffffffff811f12d8
Call Trace:
[<ffffffff811efb58>] mem_cgroup_move_account+0xa8/0x160
[<ffffffff811f12d8>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a278>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f315>] do_rmdir+0x1a5/0x200
[<ffffffff81692305>] ? do_page_fault+0x35/0x90
[<ffffffff812104c6>] SyS_rmdir+0x16/0x20
[<ffffffff81696b09>] system_call_fastpath+0x16/0x1b
Code: 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 83 e8 01 75 f1 48 89 df 0f 1f 80 00 00 00 00 eb e0 66 90 0f 1f 44 00 00 55 48 89 e5 9c 58 <0f> 1f 44 00 00 49 89 c0 fa 66 0f 1f 44 00 00 ba 00 00 02 00 f0 

NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [slurmstepd:27554]
Modules linked in: binfmt_misc nvidia_uvm(OE) nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 devlink tun overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi ext4 kvm_intel kvm mbcache jbd2 irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
ablk_helper cryptd snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq ipmi_ssif joydev snd_seq_device pcspkr snd_pcm sg snd_timer snd mei_me mei lpc_ich i2c_i801 soundcore shpchp ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_cpufreq acpi_pad acpi_power_meter ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul ttm crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ahci megaraid_sas drm ixgbe libahci libata mdio ptp i2c_core pps_core dca wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 10 PID: 27554 Comm: slurmstepd Kdump: loaded Tainted: P           OEL ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: Supermicro SYS-7049GP-TRT/X11DPG-QT, BIOS 3.1 05/23/2019
task: ffff8e39450f0000 ti: ffff8e39410a0000 task.ti: ffff8e39410a0000
RIP: 0010:[<ffffffffa2e0b917>]  [<ffffffffa2e0b917>] mem_cgroup_move_account+0x57/0x160
RSP: 0018:ffff8e39410a3ce0  EFLAGS: 00000202
RAX: 0000000000000246 RBX: ffff8e383cc95b80 RCX: 0000000000000001
RDX: 0000475cc0815050 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff8e39410a3d20 R08: ffff8e755dbafc00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 00000000fc4026c9 R14: ffff8e39410a3c38 R15: ffff8e39410a3c38
FS:  00002b4f2c631580(0000) GS:ffff8e383cc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0002f1010 CR3: 0000003f4d254000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa2e0d058>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffffa2e0d3d4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa2d1c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa2d1c482>] cgroup_rmdir+0x22/0x40
[<ffffffffa2e295ac>] vfs_rmdir+0xdc/0x150
[<ffffffffa2e2cb31>] do_rmdir+0x1f1/0x220
[<ffffffffa2e2dd66>] SyS_rmdir+0x16/0x20
[<ffffffffa331f7d5>] system_call_fastpath+0x1c/0x21
Code: 8b 07 f6 c4 40 0f 84 09 01 00 00 f0 0f ba 2b 00 19 c0 85 c0 0f 85 05 01 00 00 48 8b 03 41 bf ea ff ff ff a8 02 75 2d f0 80 23 fe <4c> 89 e6 4c 89 c7 e8 8e fd ff ff 4c 89 e6 4c 89 ef e8 83 fd ff 

NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [slurmstepd:341166]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt irqbypass iTCO_vendor_support crc32_pclmul crc32c_intel dell_smbios dcdbas sparse_keymap ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs
isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci libahci megaraid_sas tg3(OE) libata ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 7 PID: 341166 Comm: slurmstepd Tainted: G        W  OEL ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.8.1 06/30/2020
task: ffff882f7ac7bf40 ti: ffff884ad47a4000 task.ti: ffff884ad47a4000
RIP: 0010:[<ffffffff816ab865>]  [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff884ad47a7d00  EFLAGS: 00000246
RAX: 0000000000000246 RBX: 000000000000000e RCX: 0000000000000001
RDX: 000060a07281e480 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff884ad47a7d00 R08: ffff880cabe9a800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff884ad47a7c68
R13: ffff884ad47a7c68 R14: ffff884ad47a7c58 R15: ffff884ad47a7c58
FS:  00002aaaaaafd180(0000) GS:ffff885f7bec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aab421ad1f0 CR3: 000000528c28d000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff884ad47a7d50 ffffffff811f1d80 0000000000000246 00000001811959a2
ffff880cabe9a800 ffffea017c9747c0 ffff884d8d1a8440 ffff88603ffd7540
ffff8830912dd1f0 ffffea017c9747e0 ffff884ad47a7dd0 ffffffff811f33f8
Call Trace:
[<ffffffff811f1d80>] mem_cgroup_move_account+0x130/0x160
[<ffffffff811f33f8>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 

NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [slurmstepd:34377]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge 8021q garp mrp stp llc overlay(T) ip6_tables iTCO_wdt iTCO_vendor_support iptable_filter coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure cdc_ether scsi_transport_sas usbnet joydev mii sg mei_me mei shpchp i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace
binfmt_misc sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw ahci libahci mlx5_core(OE) libata mlxfw(OE) ptp megaraid_sas pps_core auxiliary(OE) mlx_compat(OE) i2c_core devlink [last unloaded: tracedev]
CPU: 17 PID: 34377 Comm: slurmstepd Tainted: G        W  OEL ------------ T 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW, BIOS AFE112A-1.02 05/10/2021
task: ffff89672f4a8000 ti: ffff89f2f3fe4000 task.ti: ffff89f2f3fe4000
RIP: 0010:[<ffffffff816ab905>]  [<ffffffff816ab905>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff89f2f3fe7d00  EFLAGS: 00000246
RAX: 0000000000000246 RBX: 000000000000000e RCX: 0000000000000001
RDX: 00005f07c04082b0 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff89f2f3fe7d00 R08: ffff89efed2b5c00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff89f2f3fe7c68
R13: ffff89f2f3fe7c68 R14: ffff89f2f3fe7c58 R15: ffff89f2f3fe7c58
FS:  00002ab626029fc0(0000) GS:ffff89f83de40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000c6d010 CR3: 00000131456ce000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff89f2f3fe7d50 ffffffff811f1cf0 0000000000000246 00000001811959d2
ffff89efed2b5c00 ffffea01c193f600 ffff89f0b267a430 ffff8a003ffd8540
ffff88048fa4fd80 ffffea01c193f620 ffff89f2f3fe7dd0 ffffffff811f3368
Call Trace:
[<ffffffff811f1cf0>] mem_cgroup_move_account+0x130/0x160
[<ffffffff811f3368>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff81213446>] SyS_rmdir+0x16/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 

NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [slurmstepd:99748]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) ip6_udp_tunnel udp_tunnel rdma_ucm ib_uverbs rdma_cm iw_cm ib_cm ib_core mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge 8021q garp mrp stp llc overlay(T) ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses joydev enclosure cdc_ether scsi_transport_sas usbnet mii sg mei_me mei shpchp i2c_i801 wmi ipmi_si ipmi_devintf
ipmi_msghandler acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw ahci libahci mlx5_core(OE) libata megaraid_sas mlxfw(OE) ptp pps_core auxiliary(OE) mlx_compat(OE) i2c_core devlink [last unloaded: tracedev]
CPU: 19 PID: 99748 Comm: slurmstepd Tainted: G        W  OEL ------------ T 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW, BIOS AFE112A-1.02 05/10/2021
task: ffff89f1ddeb2f70 ti: ffff892bb4174000 task.ti: ffff892bb4174000
RIP: 0010:[<ffffffff811f1c3d>]  [<ffffffff811f1c3d>] mem_cgroup_move_account+0x7d/0x160
RSP: 0018:ffff892bb4177d50  EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffffea070b63ed80 RCX: 0000000000000001
RDX: 0000000004ae518b RSI: ffffea070b63ee40 RDI: 0000000004ae518f
RBP: ffff892bb4177d50 R08: ffff89c353f5e400 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff811905c0
R13: 0000000000000000 R14: ffffffff81190edc R15: ffff892bb4177cf8
FS:  00002ab9e58c3fc0(0000) GS:ffff89f83dec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f851420f000 CR3: 00000029e53bc000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff892bb4177dd0 ffffffff811f3368 0000000200000000 ffff8812d7b66100
0000000000000000 0000000000001000 ffff8812d7b66040 0000000000000001
0000000100000002 ffff8812d7b66000 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f3368>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff81213446>] SyS_rmdir+0x16/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 2d f0 80 23 fe 4c 89 e6 4c 89 c7 e8 8e fd ff ff 4c 89 e6 4c 89 ef e8 83 fd ff ff 48 83 c4 18 44 89 f8 5b 41 5c 41 5d 41 5e 41 5f <5d> c3 90 48 8b 53 08 4c 39 ea 75 ca 4d 8d bd e4 01 00 00 41 83 

NMI watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [slurmstepd:120559]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul crc32c_intel dell_smbios dcdbas sparse_keymap ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs
isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci libahci megaraid_sas tg3(OE) ptp libata pps_core [last unloaded: ipmi_msghandler]
CPU: 16 PID: 120559 Comm: slurmstepd Tainted: G           OE  ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.5.4 01/14/2020
task: ffff8807c00a5ee0 ti: ffff882f3f25c000 task.ti: ffff882f3f25c000
RIP: 0010:[<ffffffff811f1ca7>]  [<ffffffff811f1ca7>] mem_cgroup_move_account+0x57/0x160
RSP: 0018:ffff882f3f25fd10  EFLAGS: 00000202
RAX: 0000000000000246 RBX: ffff882f7dc11ae0 RCX: 0000000000000001
RDX: 000060a080806e40 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff882f3f25fd50 R08: ffff885f397af800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 00000000f62b8296 R14: ffff882f3f25fc68 R15: ffff882f3f25fc68
FS:  00002aaaaaafd180(0000) GS:ffff882f7dc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aacc48039e0 CR3: 0000005f1e781000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000246 00000001811959a2 ffff885f397af800 ffffea00f31f2980
ffff884e7ce22440 ffff88603ffd7540 ffff8830b34fca60 ffffea00f31f29a0
ffff882f3f25fdd0 ffffffff811f33f8 0000000200000001 ffff880eb46f6100
Call Trace:
[<ffffffff811f33f8>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 8b 07 f6 c4 40 0f 84 09 01 00 00 f0 0f ba 2b 00 19 c0 85 c0 0f 85 05 01 00 00 48 8b 03 41 bf ea ff ff ff a8 02 75 2d f0 80 23 fe <4c> 89 e6 4c 89 c7 e8 8e fd ff ff 4c 89 e6 4c 89 ef e8 83 fd ff 

NMI watchdog: BUG: soft lockup - CPU#63 stuck for 22s! [slurmstepd:458449]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass dcdbas joydev iTCO_wdt iTCO_vendor_support crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables fuse btrfs raid6_pq xor msdos bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp
llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) megaraid_sas ahci tg3(OE) ptp libahci pps_core libata [last unloaded: ipmi_msghandler]
CPU: 63 PID: 458449 Comm: slurmstepd Tainted: G           OE  ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018
task: ffff88569f4ddee0 ti: ffff88310fb58000 task.ti: ffff88310fb58000
RIP: 0010:[<ffffffff816ab865>]  [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0000:ffff88310fb5bd00  EFLAGS: 00000246
RAX: 0000000000000246 RBX: 000000000000000e RCX: 0000000000000001
RDX: 000060a06801cd50 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff88310fb5bd00 R08: ffff880066807000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff88310fb5bc68
R13: ffff88310fb5bc68 R14: ffff88310fb5bc58 R15: ffff88310fb5bc58
FS:  00002aaaaaafd180(0000) GS:ffff885f7c5c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaaeac0d000 CR3: 000000305ff44000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff88310fb5bd50 ffffffff811f1d80 0000000000000246 00000001811959a2
ffff880066807000 ffffea00c64679c0 ffff883070229430 ffff88603ffd7540
ffff8830be999e70 ffffea00c64679e0 ffff88310fb5bdd0 ffffffff811f33f8
Call Trace:
[<ffffffff811f1d80>] mem_cgroup_move_account+0x130/0x160
[<ffffffff811f33f8>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 

NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [slurmstepd:169876]
Modules linked in: nvidia_uvm(OE) nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) vfio_iommu_type1 vfio squashfs loop overlay(T) 8021q garp mrp stp llc nfsv3 nfs_acl hfi1 rdmavt beegfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ucm rpcrdma rdma_ucm ib_uverbs ib_iser rdma_cm opa_vnic iw_cm ib_umad libiscsi ib_ipoib scsi_transport_iscsi ib_cm sunrpc iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm ast ttm ipmi_ssif irqbypass crc32_pclmul crc32c_intel drm_kms_helper ghash_clmulni_intel aesni_intel syscopyarea sysfillrect sysimgblt lrw fb_sys_fops gf128mul glue_helper ablk_helper drm cryptd ixgbe ib_core pcspkr i2c_algo_bit ptp i2c_i801
pps_core joydev mei_me mdio i2c_core lpc_ich sg dca mei shpchp ipmi_si ipmi_msghandler nfit wmi libnvdimm acpi_cpufreq acpi_pad acpi_power_meter binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ahci crct10dif_pclmul crct10dif_common libahci libata [last unloaded: ipmi_devintf]
CPU: 11 PID: 169876 Comm: slurmstepd Tainted: P        W  OEL ------------ T 3.10.0-862.3.3.el7.x86_64 #1
Hardware name: Supermicro SYS-1029GQ-TVRT/X11DGQ, BIOS 2.0b 02/28/2018
task: ffff9255ebbfaf70 ti: ffff925315c68000 task.ti: ffff925315c68000
RIP: 0010:[<ffffffff9bd16475>]  [<ffffffff9bd16475>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff925315c6bcd0  EFLAGS: 00000246
RAX: 0000000000000246 RBX: 000000000000000e RCX: 0000000000000001
RDX: 00002ecd9d022b60 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff925315c6bcd0 R08: ffff9246cc3ce800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff925315c6bc38
R13: ffff925315c6bc38 R14: ffff925315c6bc28 R15: ffff925315c6bc28
FS:  00002b06bc25d0c0(0000) GS:ffff92449f8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd1f6344010 CR3: 0000000a22440000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff9b80bd30>] mem_cgroup_move_account+0x130/0x160
[<ffffffff9b80d398>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff9b80d714>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff9b71c50a>] cgroup_destroy_locked+0xea/0x370
[<ffffffff9b71c7b2>] cgroup_rmdir+0x22/0x40
[<ffffffff9b8298ec>] vfs_rmdir+0xdc/0x150
[<ffffffff9b82ce71>] do_rmdir+0x1f1/0x220
[<ffffffff9b82e0a6>] SyS_rmdir+0x16/0x20
[<ffffffff9bd2082f>] system_call_fastpath+0x1c/0x21
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 

NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [slurmstepd:269298]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev dcdbas skx_edac edac_core intel_powerclamp coretemp intel_rapl iTCO_wdt iosf_mbi iTCO_vendor_support kvm_intel kvm irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx
hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) megaraid_sas ahci libahci tg3(OE) ptp pps_core libata [last unloaded: ipmi_msghandler]
CPU: 4 PID: 269298 Comm: slurmstepd Tainted: G           OE  ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018
task: ffff885f7733af70 ti: ffff88101adf0000 task.ti: ffff88101adf0000
RIP: 0010:[<ffffffff816ab865>]  [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0000:ffff88101adf3d00  EFLAGS: 00000246
RAX: 0000000000000246 RBX: 000000000000000e RCX: 0000000000000001
RDX: 000060a08080c2a0 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff88101adf3d00 R08: ffff885f430cc800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff88101adf3c68
R13: ffff88101adf3c68 R14: ffff88101adf3c58 R15: ffff88101adf3c58
FS:  00002aaaaaafd180(0000) GS:ffff882f7da80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000036ab000 CR3: 0000000d8ebd0000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff88101adf3d50 ffffffff811f1d80 0000000000000246 00000001811959a2
ffff885f430cc800 ffffea01037db740 ffff884d37e0e430 ffff88603ffd7540
ffff8830af676dd0 ffffea01037db760 ffff88101adf3dd0 ffffffff811f33f8
Call Trace:
[<ffffffff811f1d80>] mem_cgroup_move_account+0x130/0x160
[<ffffffff811f33f8>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 

NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [slurmstepd:130791]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp mgag200 coretemp ttm intel_rapl iosf_mbi kvm_intel drm_kms_helper kvm syscopyarea sysfillrect dell_smbios sysimgblt dcdbas irqbypass fb_sys_fops sparse_keymap crc32_pclmul drm crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs
isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci libahci megaraid_sas libata tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 17 PID: 130791 Comm: slurmstepd Tainted: G           OE  ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.9.3 09/23/2020
task: ffff882fa43faf70 ti: ffff880527204000 task.ti: ffff880527204000
RIP: 0010:[<ffffffff816abbcc>]  [<ffffffff816abbcc>] _raw_spin_lock_irqsave+0xc/0x40
RSP: 0018:ffff880527207cf8  EFLAGS: 00000246
RAX: 0000000000000246 RBX: ffff880527207c68 RCX: ffff885fb4c4f000
RDX: ffff885fb4c4f000 RSI: 0000000000000001 RDI: ffff885fb4c4f1e4
RBP: ffff880527207d00 R08: ffff885f7bc1c000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff880527207c58
R13: ffff880527207c58 R14: ffff885fb8011b60 R15: 0000000000000001
FS:  00002aaaaaafd180(0000) GS:ffff885fb8000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fffe7b7b9f0 CR3: 00000005e1348000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8830dc52a6c0 ffff880527207d50 ffffffff811f1cf8 ffff885f7bc1c000
00000001811959a2 ffff885fb4c4f000 ffffea01502a9b00 ffff883264793440
ffff88607bfd7540 ffff8830dc52a6c0 ffffea01502a9b20 ffff880527207dd0
Call Trace:
[<ffffffff811f1cf8>] mem_cgroup_move_account+0xa8/0x160
[<ffffffff811f33f8>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 61 2a ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 9c 58 <0f> 1f 44 00 00 48 89 c3 fa 66 0f 1f 44 00 00 31 c0 ba 01 00 00 

NMI watchdog: BUG: soft lockup - CPU#35 stuck for 23s! [slurmstepd:33676]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) squashfs loop overlay(T) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp iTCO_wdt dell_smbios intel_rapl iTCO_vendor_support sparse_keymap iosf_mbi dcdbas kvm_intel kvm irqbypass crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper ghash_clmulni_intel syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter
bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci libahci megaraid_sas libata tg3(OE) ptp pps_core [last unloaded: tracedev]
CPU: 35 PID: 33676 Comm: slurmstepd Tainted: G    B      OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/08CYF7, BIOS 1.7.0 12/14/2018
task: ffff8820abda6eb0 ti: ffff880c08420000 task.ti: ffff880c08420000
RIP: 0010:[<ffffffff816ab865>]  [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff880c08423d00  EFLAGS: 00000246
RAX: 0000000000000246 RBX: 000000000000000e RCX: 0000000000000001
RDX: 000060a08080c298 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff880c08423d00 R08: ffff885f4f896c00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff880c08423c68
R13: ffff880c08423c68 R14: ffff880c08423c58 R15: ffff880c08423c58
FS:  00002aaaaaafd180(0000) GS:ffff885f7c240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaaaaad0000 CR3: 0000000107639000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff880c08423d50 ffffffff811f1d80 0000000000000246 00000001811959a2
ffff885f4f896c00 ffffea014412a5c0 ffff8842b87e9440 ffff88603ffd7540
ffff88309f0ca970 ffffea014412a5e0 ffff880c08423dd0 ffffffff811f33f8
Call Trace:
[<ffffffff811f1d80>] mem_cgroup_move_account+0x130/0x160
[<ffffffff811f33f8>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 

BUG: soft lockup - CPU#22 stuck for 23s! [slurmstepd:24478]
Modules linked in: 8021q garp stp mrp llc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod intel_powerclamp coretemp iTCO_wdt intel_rapl iTCO_vendor_support kvm_intel kvm crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_devintf sb_edac mei_me lpc_ich mfd_core mei sg i2c_i801 edac_core pcspkr shpchp wmi ipmi_si ipmi_msghandler acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt igb drm_kms_helper crct10dif_pclmul ahci ttm crct10dif_common ptp crc32c_intel libahci pps_core drm libata dca i2c_algo_bit i2c_core
CPU: 22 PID: 24478 Comm: slurmstepd Not tainted 3.10.0-327.el7.x86_64 #1
Hardware name: GIGABYTE H270-H70/MH70-HD1, BIOS R10 06/15/2018
task: ffff882fb8c6a280 ti: ffff882ce5ef4000 task.ti: ffff882ce5ef4000
RIP: 0010:[<ffffffff811d04f4>]  [<ffffffff811d04f4>] mem_cgroup_move_account+0x134/0x160
RSP: 0018:ffff882ce5ef7d08  EFLAGS: 00000246
RAX: 0000000000000246 RBX: ffffea00d251ec00 RCX: 0000000000000001
RDX: 000060c0008005a0 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff882ce5ef7d48 R08: ffffc90028c64000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 00000000834fc66d
R13: ffff882ce5ef7c60 R14: ffff882ce5ef7c60 R15: ffff882ce5ef7ca0
FS:  00002b5da093d6c0(0000) GS:ffff883fff300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc78fd92838 CR3: 0000001435b43000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000246 000000018117ca28 ffffc90028c64000 ffffea00d225c900
ffff882ce5ed7430 ffff88407ffd7540 ffff883fead17240 ffffea00d225c920
ffff882ce5ef7dd0 ffffffff811d1bf8 ffffc9004a95f140 0000000200000001
Call Trace:
[<ffffffff811d1bf8>] mem_cgroup_reparent_charges+0x238/0x440
[<ffffffff811d1f7f>] mem_cgroup_css_offline+0x4f/0x100
[<ffffffff810fa347>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff810fa5e2>] cgroup_rmdir+0x22/0x40
[<ffffffff811ea1c8>] vfs_rmdir+0xa8/0x100
[<ffffffff811ef165>] do_rmdir+0x1a5/0x200
[<ffffffff811f0316>] SyS_rmdir+0x16/0x20
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 89 e6 49 8d b8 10 02 00 00 4c 89 43 08 44 89 c9 e8 92 e2 ff ff 48 8b 45 c0 4c 89 ff 45 31 ff 48 89 c6 e8 40 c5 46 00 4c 8b 45 d0 <e9> 1a ff ff ff 0f 1f 80 00 00 00 00 41 bf f0 ff ff ff e9 22 ff 

NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [slurmstepd:197355]
Modules linked in: squashfs loop overlay(T) nvidia_uvm(OE) beegfs(OE) nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) ipmi_devintf vfio_iommu_type1 vfio nfsv3 nfs_acl opa_vnic hfi1 rdmavt 8021q garp mrp stp llc rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ixgbe cryptd ipmi_ssif ast pcspkr ttm ptp joydev pps_core mdio dca mei_me sg lpc_ich mei shpchp i2c_i801 wmi ipmi_si nfit libnvdimm acpi_cpufreq acpi_power_meter acpi_pad rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm
ib_uverbs ib_umad rdma_cm ib_cm iw_cm binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic drm_kms_helper drm i2c_algo_bit crct10dif_pclmul ahci crct10dif_common crc32c_intel ib_core syscopyarea libahci sysfillrect sysimgblt fb_sys_fops i2c_core libata ipmi_msghandler [last unloaded: beegfs]
CPU: 23 PID: 197355 Comm: slurmstepd Tainted: P           OEL ------------ T 3.10.0-862.3.3.el7.x86_64 #1
Hardware name: Supermicro SYS-1029GQ-TVRT/X11DGQ, BIOS 2.0b 02/28/2018
task: ffff90e53e6e9fa0 ti: ffff90f443ef8000 task.ti: ffff90f443ef8000
RIP: 0010:[<ffffffffb7b16475>]  [<ffffffffb7b16475>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff90f443efbcd0  EFLAGS: 00000246
RAX: 0000000000000246 RBX: 000000000000000e RCX: 0000000000000001
RDX: 000040719b406fe0 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff90f443efbcd0 R08: ffff90f54e019400 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff90f443efbc38
R13: ffff90f443efbc38 R14: ffff90f443efbc28 R15: ffff90f443efbc28
FS:  00002b8a988420c0(0000) GS:ffff90f55dac0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b1ead728090 CR3: 0000002ec3ef2000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffb760bd30>] mem_cgroup_move_account+0x130/0x160
[<ffffffffb760d398>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffffb760d714>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffb751c50a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffb751c7b2>] cgroup_rmdir+0x22/0x40
[<ffffffffb76298ec>] vfs_rmdir+0xdc/0x150
[<ffffffffb762ce71>] do_rmdir+0x1f1/0x220
[<ffffffffb762e0a6>] SyS_rmdir+0x16/0x20
[<ffffffffb7b2082f>] system_call_fastpath+0x1c/0x21
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 

NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [runc:22267]
Modules linked in: isofs loop xt_nat xt_addrtype nf_conntrack_netlink xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 iptable_filter xt_conntrack iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle ip_tables cmac arc4 md4 nls_utf8 cifs ccm veth ip_set rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) nfnetlink br_netfilter tun bridge stp llc nfsv3 nfs_acl nfs lockd grace fscache overlay(T) bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) libcrc32c sunrpc dm_mirror dm_region_hash
dm_log dm_mod iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr wdat_wdt joydev sg i2c_i801 mei_me lpc_ich mei ipmi_si ipmi_devintf ipmi_msghandler knem(OE) ext4 mbcache jbd2 mlx4_ib(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe drm ahci mdio libahci mlx4_core(OE) igb libata drm_panel_orientation_quirks megaraid_sas ptp devlink pps_core dca mlx_compat(OE) i2c_algo_bit [last unloaded: ip_tables]
CPU: 2 PID: 22267 Comm: runc Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Inspur TS860G3/TS860G3, BIOS TS860G3_4.0.07 10/09/2016
task: ffff93a83c38d140 ti: ffff93a770990000 task.ti: ffff93a770990000
RIP: 0010:[<ffffffffab36a0f5>]  [<ffffffffab36a0f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff93a770993cd0  EFLAGS: 00000246
RAX: 0000000000000246 RBX: 000000000000000e RCX: 0000000000000001
RDX: 00002eb280809a60 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff93a770993cd0 R08: ffff93d05a6c6000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff93a770993c38
R13: ffff93a770993c38 R14: ffff93a770993c28 R15: ffff93a770993c28
FS:  00007f2969438740(0000) GS:ffff92f0bfa80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000e80000 CR3: 0000001ffba3e000 CR4: 00000000001607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaae31910>] mem_cgroup_move_account+0x130/0x160
[<ffffffffaae32f78>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffffaae332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffaad2573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffaad259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffaae4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffaae530d1>] do_rmdir+0x1f1/0x220
[<ffffffffaae47034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffffaae54335>] SyS_unlinkat+0x25/0x40
[<ffffffffab374ddb>] system_call_fastpath+0x22/0x27
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 

NMI watchdog: BUG: soft lockup - CPU#34 stuck for 22s! [runc:82531]
Modules linked in: tcp_diag inet_diag rpcsec_gss_krb5 xt_nat veth nvidia_uvm(OE) binfmt_misc nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support vfat fat nvidia_drm(POE) intel_powerclamp nvidia_modeset(POE) coretemp kvm_intel
kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_service_time nvidia(POE) snd_hda_codec_hdmi joydev cdc_ether usbnet pcspkr snd_hda_intel mii snd_hda_codec snd_hda_core sg snd_hwdep mpt3sas snd_seq ipmi_ssif snd_seq_device raid_class scsi_transport_sas snd_pcm snd_timer snd dm_multipath i2c_i801 soundcore mei_me mei ipmi_si ipmi_devintf ipmi_msghandler pinctrl_lewisburg pinctrl_intel acpi_cpufreq acpi_pad acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod lpfc ast drm_kms_helper nvmet_fc syscopyarea sysfillrect nvmet sysimgblt fb_sys_fops crc_t10dif ttm crct10dif_generic nvme_fc crct10dif_pclmul crc32c_intel nvme_fabrics drm nvme_core scsi_transport_fc ahci igb scsi_tgt crct10dif_common
libahci megaraid_sas libata ptp pps_core drm_panel_orientation_quirks dca i2c_algo_bit wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 34 PID: 82531 Comm: runc Tainted: P        W  OE  ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Nettrix X640 G40/60WB32, BIOS NKMH041018 10/08/2021
task: ffff8afbb5e71040 ti: ffff8acc5a7b8000 task.ti: ffff8acc5a7b8000
RIP: 0010:[<ffffffff9a36a0fb>]  [<ffffffff9a36a0fb>] _raw_spin_unlock_irqrestore+0x1b/0x20
RSP: 0000:ffff8acc5a7bbcd8  EFLAGS: 00000246
RAX: 0000000000000246 RBX: 000000000000000e RCX: 0000000000000001
RDX: 00004c6900808a70 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff8acc5a7bbd20 R08: ffff8a7d651b1800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8acc5a7bbc38
R13: ffff8acc5a7bbc38 R14: ffff8acc5a7bbc28 R15: ffff8acc5a7bbc28
FS:  00007fdb0f442740(0000) GS:ffff8afbbe680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f55d97f53e0 CR3: 000000413c1fa000 CR4: 0000000000760fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff99e31910>] ? mem_cgroup_move_account+0x130/0x160
[<ffffffff99e32f78>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff99e332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff99d2573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffff99d259e2>] cgroup_rmdir+0x22/0x40
[<ffffffff99e4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffff99e530d1>] do_rmdir+0x1f1/0x220
[<ffffffff99e47034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffff99e54335>] SyS_unlinkat+0x25/0x40
[<ffffffff9a374ddb>] system_call_fastpath+0x22/0x27
Code: 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d 0f 1f 44 00 00 5d <c3> 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 c7 c2 ff ff ff ff 

NMI watchdog: BUG: soft lockup - CPU#17 stuck for 23s! [runc:24664]
Modules linked in: binfmt_misc xt_nat veth nvidia_uvm(OE) fuse xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_addrtype iptable_filter br_netfilter bridge stp llc mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) dm_mirror dm_region_hash dm_log dm_mod nvidia_drm(POE) nvidia_modeset(POE) iTCO_wdt iTCO_vendor_support nvidia(POE) sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel
lrw gf128mul glue_helper ablk_helper cryptd squashfs loop snd_hda_codec_hdmi snd_hda_intel snd_hda_codec pcspkr snd_hda_core snd_hwdep joydev snd_seq snd_seq_device snd_pcm snd_timer sg snd i2c_i801 soundcore lpc_ich mei_me mei ioatdma ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter knem(OE) auth_rpcgss sunrpc ip_tables xfs libcrc32c mlx4_ib(OE) mlx4_en(OE) sd_mod crc_t10dif crct10dif_generic mxm_wmi mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm igb mlx4_core(OE) crct10dif_pclmul crct10dif_common crc32c_intel drm ahci dca libahci i2c_algo_bit mlx5_core(OE) libata mlxfw(OE) drm_panel_orientation_quirks devlink mlx_compat(OE) ptp pps_core wmi overlay(T)
CPU: 17 PID: 24664 Comm: runc Kdump: loaded Tainted: P           OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Supermicro SYS-4028GR-TR2-1-EC028/X10DRG-O+-CPU, BIOS 3.1 07/13/2018
task: ffff8ec6c1b42080 ti: ffff8e96d4648000 task.ti: ffff8e96d4648000
RIP: 0010:[<ffffffffae631837>]  [<ffffffffae631837>] mem_cgroup_move_account+0x57/0x160
RSP: 0018:ffff8e96d464bce0  EFLAGS: 00000202
RAX: 0000000000000246 RBX: ffff8f13ee0d9000 RCX: 0000000000000001
RDX: 00004afd4080e500 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff8e96d464bd20 R08: ffff8f13ee0d9000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 000000005c903db2 R14: ffff8e96d464bc38 R15: ffff8e96d464bc38
FS:  00007f7094b79740(0000) GS:ffff8f943eec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0004a2ed0 CR3: 000000c6a0872000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffae632f78>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffffae6332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffae52573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffae5259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffae64dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffae6530d1>] do_rmdir+0x1f1/0x220
[<ffffffffae647034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffffae654335>] SyS_unlinkat+0x25/0x40
[<ffffffffaeb74ddb>] system_call_fastpath+0x22/0x27
Code: 8b 07 f6 c4 40 0f 84 09 01 00 00 f0 0f ba 2b 00 19 c0 85 c0 0f 85 05 01 00 00 48 8b 03 41 bf ea ff ff ff a8 02 75 2d f0 80 23 fe <4c> 89 e6 4c 89 c7 e8 8e fd ff ff 4c 89 e6 4c 89 ef e8 83 fd ff 

NMI watchdog: BUG: soft lockup - CPU#98 stuck for 22s! [runc:129921]
Modules linked in: isofs loop xt_nat xt_addrtype nf_conntrack_netlink xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 iptable_filter xt_conntrack iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle ip_tables cmac arc4 md4 nls_utf8 cifs ccm veth ip_set rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) nfnetlink br_netfilter tun bridge stp llc nfsv3 nfs_acl nfs lockd grace fscache overlay(T) bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) libcrc32c sunrpc dm_mirror dm_region_hash
dm_log dm_mod iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr wdat_wdt joydev sg i2c_i801 mei_me lpc_ich mei ipmi_si ipmi_devintf ipmi_msghandler knem(OE) ext4 mbcache jbd2 mlx4_ib(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe drm ahci mdio libahci mlx4_core(OE) igb libata drm_panel_orientation_quirks megaraid_sas ptp devlink pps_core dca mlx_compat(OE) i2c_algo_bit [last unloaded: ip_tables]
CPU: 98 PID: 129921 Comm: runc Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Inspur TS860G3/TS860G3, BIOS TS860G3_4.0.07 10/09/2016
task: ffff9378a3b430c0 ti: ffff9370bb398000 task.ti: ffff9370bb398000
RIP: 0010:[<ffffffffaae31837>]  [<ffffffffaae31837>] mem_cgroup_move_account+0x57/0x160
RSP: 0018:ffff9370bb39bce0  EFLAGS: 00000202
RAX: 0000000000000246 RBX: ffff93b0bfa97c80 RCX: 0000000000000001
RDX: 00002eb280809a60 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff9370bb39bd20 R08: ffff93d05a6c6000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 000000004dfec918 R14: ffff9370bb39bc38 R15: ffff9370bb39bc38
FS:  00007f8448057740(0000) GS:ffff93b0bfa80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5f4e27ad00 CR3: 000000cc4776e000 CR4: 00000000001607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaae32f78>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffffaae332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffaad2573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffaad259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffaae4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffaae530d1>] do_rmdir+0x1f1/0x220
[<ffffffffaae47034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffffaae54335>] SyS_unlinkat+0x25/0x40
[<ffffffffab374ddb>] system_call_fastpath+0x22/0x27
Code: 8b 07 f6 c4 40 0f 84 09 01 00 00 f0 0f ba 2b 00 19 c0 85 c0 0f 85 05 01 00 00 48 8b 03 41 bf ea ff ff ff a8 02 75 2d f0 80 23 fe <4c> 89 e6 4c 89 c7 e8 8e fd ff ff 4c 89 e6 4c 89 ef e8 83 fd ff 

NMI watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [runc:162626]
Modules linked in: xt_TPROXY veth tcp_diag udp_diag inet_diag squashfs loop ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter devlink binfmt_misc overlay(T) nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink mpt2sas ebtable_nat raid_class scsi_transport_sas ebtable_broute bridge stp llc mptctl mptbase ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu sunrpc ext4 mbcache jbd2 dcdbas vfat fat intel_powerclamp coretemp
intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_multipath iTCO_wdt iTCO_vendor_support pcspkr sb_edac edac_core ipmi_devintf i2c_i801 sg lpc_ich mei_me mei shpchp wmi ipmi_si ipmi_msghandler acpi_power_meter sch_fq_codel ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ixgbe qla2xxx libahci drm igb mdio libata ptp i2c_algo_bit scsi_transport_fc pps_core i2c_core megaraid_sas scsi_tgt dca fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 15 PID: 162626 Comm: runc Tainted: G        W    L ------------ T 3.10.0-514.21.1.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R930/0Y0V4F, BIOS 2.1.3 08/08/2016
task: ffff886b3525edd0 ti: ffff881dc31d0000 task.ti: ffff881dc31d0000
RIP: 0010:[<ffffffff8168e70b>]  [<ffffffff8168e70b>] _raw_spin_unlock_irqrestore+0x1b/0x40
RSP: 0018:ffff881dc31d3cf8  EFLAGS: 00000246
RAX: 0000000000000246 RBX: ffff881dc31d3c68 RCX: 0000000000000001
RDX: 00006080008124a8 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff881dc31d3d00 R08: ffff8820d2d56400 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff881dc31d3c58
R13: ffff881dc31d3c58 R14: ffff887ffead1aa0 R15: 0000000000000001
FS:  00007fee738da740(0000) GS:ffff887ffeac0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2441992a08 CR3: 0000006914d8c000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8860ef85a170 ffff881dc31d3d50 ffffffff811f00d0 0000000000000246
0000000181194d63 ffff8820d2d56400 ffffea01c2f685c0 ffff88807fd6e430
ffff88807ffd6540 ffff8860ef85a170 ffffea01c2f685e0 ffff881dc31d3dd0
Call Trace:
[<ffffffff811f00d0>] mem_cgroup_move_account+0x130/0x160
[<ffffffff811f17c8>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffff811f1b44>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d6e7>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d982>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a778>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f815>] do_rmdir+0x1a5/0x200
[<ffffffff81203ee4>] ? SYSC_newfstatat+0x24/0x60
[<ffffffff81692db5>] ? do_page_fault+0x35/0x90
[<ffffffff812109f5>] SyS_unlinkat+0x25/0x40
[<ffffffff816975c9>] system_call_fastpath+0x16/0x1b
Code: 08 e8 ba 07 a0 ff 5d c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 48 89 f3 0f 1f 44 00 00 66 83 07 02 48 89 df 57 9d <0f> 1f 44 00 00 5b 5d c3 0f 1f 44 00 00 8b 37 f0 66 83 07 02 f6 

NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [runc:88796]
Modules linked in: xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel veth xt_set ip_set_bitmap_port ip_set_hash_ipportnet ip_set_hash_ipport ip_set_hash_ipportip ip_set dummy nfsv3 rpcsec_gss_krb5 nfsv4 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 dns_resolver nfs fscache xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack cambricon_gdrdrv(OE) iTCO_wdt iTCO_vendor_support vfat fat intel_powerclamp coretemp kvm
irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg mei_me mei i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler pinctrl_lewisburg pinctrl_intel acpi_cpufreq acpi_power_meter cambricon_drv(OE) nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt serio_raw fb_sys_fops ttm i40e drm ahci libahci megaraid_sas libata ptp drm_panel_orientation_quirks pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 30 PID: 88796 Comm: runc Kdump: loaded Tainted: G        W  OE  ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Inspur NF5468M6/NF5468M6, BIOS 06.01.02 06/28/2022
task: ffffa21a73e79040 ti: ffffa292736d4000 task.ti: ffffa292736d4000
RIP: 0010:[<ffffffffbab6a45c>]  [<ffffffffbab6a45c>] _raw_spin_lock_irqsave+0xc/0x40
RSP: 0018:ffffa292736d7cc8  EFLAGS: 00000246
RAX: 0000000000000246 RBX: ffffa292736d7c38 RCX: ffffa298a16cb000
RDX: ffffa298a16cb000 RSI: 0000000000000001 RDI: ffffa298a16cb1ec
RBP: ffffa292736d7cd0 R08: ffffa28e84a0e800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffa292736d7c28
R13: ffffa292736d7c28 R14: ffffa21abf697d00 R15: 0000000000000001
FS:  00007fae9d488740(0000) GS:ffffa21abf680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00115a048 CR3: 0000017bfabd8000 CR4: 0000000000760fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffba631888>] mem_cgroup_move_account+0xa8/0x160
[<ffffffffba632f78>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffffba6332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffba52573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffba5259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffba64dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffba6530d1>] do_rmdir+0x1f1/0x220
[<ffffffffba647034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffffba654335>] SyS_unlinkat+0x25/0x40
[<ffffffffbab74ddb>] system_call_fastpath+0x22/0x27
Code: b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 fd 1a ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 9c 58 <0f> 1f 44 00 00 48 89 c3 fa 66 0f 1f 44 00 00 31 c0 ba 01 00 00 

NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [runc:82641]
Modules linked in: xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel veth xt_set ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipport ip_set_hash_ipportip ip_set dummy nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack cambricon_gdrdrv(OE) joydev iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm
irqbypass crc32_pclmul vfat fat ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cp210x sg mei_me i2c_i801 mei wmi ipmi_si ipmi_devintf ipmi_msghandler pinctrl_lewisburg pinctrl_intel acpi_cpufreq acpi_power_meter cambricon_drv(OE) nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect serio_raw sysimgblt fb_sys_fops ttm i40e drm ahci libahci megaraid_sas ptp libata drm_panel_orientation_quirks pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 18 PID: 82641 Comm: runc Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Inspur NF5468M6/NF5468M6, BIOS 06.01.02 06/28/2022
task: ffff9ca863a95140 ti: ffff9c280d7ec000 task.ti: ffff9c280d7ec000
RIP: 0010:[<ffffffffa816a0f5>]  [<ffffffffa816a0f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9c280d7efcd0  EFLAGS: 00000246
RAX: 0000000000000246 RBX: 000000000000000e RCX: 0000000000000001
RDX: 00002ed340804e10 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff9c280d7efcd0 R08: ffff9b9b50a15800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff9c280d7efc38
R13: ffff9c280d7efc38 R14: ffff9c280d7efc28 R15: ffff9c280d7efc28
FS:  00007f036ce2e740(0000) GS:ffff9ba87f700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb85d920540 CR3: 000001f4eca6a000 CR4: 0000000000760fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa7c31910>] mem_cgroup_move_account+0x130/0x160
[<ffffffffa7c32f78>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffffa7c332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa7b2573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa7b259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffa7c4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffa7c530d1>] do_rmdir+0x1f1/0x220
[<ffffffffa7c47034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffffa7c54335>] SyS_unlinkat+0x25/0x40
[<ffffffffa8174ddb>] system_call_fastpath+0x22/0x27
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 

NMI watchdog: BUG: soft lockup - CPU#45 stuck for 22s! [runc:49598]
Modules linked in: xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel veth xt_set ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipport ip_set_hash_ipportip ip_set dummy nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack cambricon_gdrdrv(OE) joydev iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm
irqbypass crc32_pclmul vfat fat ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cp210x sg mei_me i2c_i801 mei wmi ipmi_si ipmi_devintf ipmi_msghandler pinctrl_lewisburg pinctrl_intel acpi_cpufreq acpi_power_meter cambricon_drv(OE) nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect serio_raw sysimgblt fb_sys_fops ttm i40e drm ahci libahci megaraid_sas ptp libata drm_panel_orientation_quirks pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 45 PID: 49598 Comm: runc Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Inspur NF5468M6/NF5468M6, BIOS 06.01.02 06/28/2022
task: ffff9ca863a0b0c0 ti: ffff9c97be294000 task.ti: ffff9c97be294000
RIP: 0010:[<ffffffffa7c31837>]  [<ffffffffa7c31837>] mem_cgroup_move_account+0x57/0x160
RSP: 0018:ffff9c97be297ce0  EFLAGS: 00000202
RAX: 0000000000000246 RBX: ffff9ca87eed7c80 RCX: 0000000000000001
RDX: 00002ed340812998 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff9c97be297d20 R08: ffff9b1928ac5400 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 00000000d716bb28 R14: ffff9c97be297c38 R15: ffff9c97be297c38
FS:  00007f3e82430740(0000) GS:ffff9ca87eec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00141c1d0 CR3: 000001fcdd752000 CR4: 0000000000760fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa7c32f78>] mem_cgroup_reparent_charges+0x1f8/0x3c0
[<ffffffffa7c332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa7b2573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa7b259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffa7c4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffa7c530d1>] do_rmdir+0x1f1/0x220
[<ffffffffa7c47034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffffa7c54335>] SyS_unlinkat+0x25/0x40
[<ffffffffa8174ddb>] system_call_fastpath+0x22/0x27
Code: 8b 07 f6 c4 40 0f 84 09 01 00 00 f0 0f ba 2b 00 19 c0 85 c0 0f 85 05 01 00 00 48 8b 03 41 bf ea ff ff ff a8 02 75 2d f0 80 23 fe <4c> 89 e6 4c 89 c7 e8 8e fd ff ff 4c 89 e6 4c 89 ef e8 83 fd ff