NMI watchdog: BUG: soft lockup - CPU#59 stuck for 22s! [runc:89508]
Modules linked in: ses enclosure scsi_transport_sas xt_nat binfmt_misc nfsv4 nfs fscache rpcsec_gss_krb5 veth cmac arc4 md4 nls_utf8 cifs ccm dns_resolver fuse ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 iTCO_wdt iTCO_vendor_support dell_smbios dcdbas
dell_wmi_descriptor skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ixgbe sg mdio dca mei_me i2c_i801 lpc_ich mei wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm megaraid_sas tg3 drm_panel_orientation_quirks ptp pps_core ahci libahci libata nfit libnvdimm uas usb_storage dm_mirror dm_region_hash dm_log dm_mod
CPU: 59 PID: 89508 Comm: runc Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R940xa/08XR9M, BIOS 2.9.4 11/09/2020
task: ffff8c000bc2d140 ti: ffff8bc38b524000 task.ti: ffff8bc38b524000
RIP: 0010:[<ffffffffa39d61a8>]  [<ffffffffa39d61a8>] page_mapping+0x8/0x50
RSP: 0018:ffff8bc38b527ce8  EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffff8c81bdd97c80 RCX: 0000000000000001
RDX: 006fffff0000000c RSI: 0000000000000001 RDI: ffffec42284dc700
RBP: ffff8bc38b527cf8 R08: ffff8bc7aafc3400 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 000000009f37ea2a R14: ffff8bc38b527c38 R15: ffff8bc38b527c38
FS:  00007fdfc9863740(0000) GS:ffff8c81bdd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005630120bfa98 CR3: 000000c3a5740000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa39ca3c2>] ? page_evictable+0x12/0x40
[<ffffffffa39ca41e>] putback_lru_page+0x2e/0xb0
[<ffffffffa3a32faa>] mem_cgroup_reparent_charges+0x22a/0x3c0
[<ffffffffa3a332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa392573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa39259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffa3a4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffa3a530d1>] do_rmdir+0x1f1/0x220
[<ffffffffa3a47034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffffa3a54335>] SyS_unlinkat+0x25/0x40
[<ffffffffa3f74ddb>] system_call_fastpath+0x22/0x27
Code: 00 75 0f 55 49 c1 e9 0c 48 89 e5 e8 b3 fe ff ff 5d c3 48 c7 c0 ea ff ff ff c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 17 <55> 48 8b 47 08 48 89 e5 48 c1 ea 10 83 e2 01 75 08 a8 01 48 0f 

NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [slurmstepd:272602]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss nfsv4 dell_rbu dns_resolver dm_mirror dm_region_hash dm_log iTCO_wdt dell_smbios iTCO_vendor_support sparse_keymap dcdbas skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data
dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci libata i40e(OE) megaraid_sas tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 7 PID: 272602 Comm: slurmstepd Tainted: G           OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.21.1 03/07/2024
task: ffff885c53f49fa0 ti: ffff884ef33f0000 task.ti: ffff884ef33f0000
RIP: 0010:[<ffffffff811a1338>]  [<ffffffff811a1338>] page_mapping+0x8/0x40
RSP: 0018:ffff884ef33f3d18  EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffff885f7bed1ae0 RCX: 0000000000000001
RDX: 006fffff0000000c RSI: 0000000000000001 RDI: ffffea017814e980
RBP: ffff884ef33f3d28 R08: ffff88603fdfdc00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 0000000051c82966 R14: ffff884ef33f3c68 R15: ffff884ef33f3c68
FS:  00002aaaaaafd180(0000) GS:ffff885f7bec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaaaaad1000 CR3: 0000003281ce4000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff81195ca2 ffffea017814e980 ffff884ef33f3d50 ffffffff81195cfe
ffffea017814e980 ffff8832ba60d430 ffff88603ffd7540 ffff884ef33f3dd0
ffffffff811f342a 0000000200000001 ffff884ea3d62d00 0000000000000001
Call Trace:
[<ffffffff81195ca2>] ? page_evictable+0x12/0x40
[<ffffffff81195cfe>] putback_lru_page+0x2e/0xb0
[<ffffffff811f342a>] mem_cgroup_reparent_charges+0x22a/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816a8fad>] ? __schedule+0x39d/0x8b0
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 00 75 0f 55 49 c1 e9 0c 48 89 e5 e8 c3 fe ff ff 5d c3 48 c7 c0 ea ff ff ff c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 17 <55> 48 8b 47 08 48 89 e5 48 c1 ea 10 83 e2 01 75 08 a8 01 48 0f 

NMI watchdog: BUG: soft lockup - CPU#38 stuck for 22s! [runc:193796]
Modules linked in: binfmt_misc veth rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache cmac arc4 md4 nls_utf8 cifs ccm dns_resolver fuse uas usb_storage ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iTCO_wdt iTCO_vendor_support dell_smbios dcdbas dell_wmi_descriptor ext4 mbcache jbd2 skx_edac intel_powerclamp
coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg ipmi_ssif i2c_i801 lpc_ich mei_me mei wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm tg3 megaraid_sas drm_panel_orientation_quirks ptp pps_core ahci libahci nfit libata libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 38 PID: 193796 Comm: runc Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R940xa/08XR9M, BIOS 2.9.4 11/09/2020
task: ffff93b66bc530c0 ti: ffff93b67621c000 task.ti: ffff93b67621c000
RIP: 0010:[<ffffffffab5d61a8>]  [<ffffffffab5d61a8>] page_mapping+0x8/0x50
RSP: 0018:ffff93b67621fce8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff94367e457c80 RCX: 0000000000000001
RDX: 006fffff00020008 RSI: 0000000000000001 RDI: ffffe9865c193b00
RBP: ffff93b67621fcf8 R08: ffff942dc471d000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 00000000bf2b7b2d R14: ffff93b67621fc38 R15: ffff93b67621fc38
FS:  00007f820f169740(0000) GS:ffff94367e440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffc7365f070 CR3: 000000903a11e000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffab5ca3c2>] ? page_evictable+0x12/0x40
[<ffffffffab5ca41e>] putback_lru_page+0x2e/0xb0
[<ffffffffab632faa>] mem_cgroup_reparent_charges+0x22a/0x3c0
[<ffffffffab6332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffab52573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffab5259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffab64dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffab6530d1>] do_rmdir+0x1f1/0x220
[<ffffffffab647034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffffab654335>] SyS_unlinkat+0x25/0x40
[<ffffffffabb74ddb>] system_call_fastpath+0x22/0x27
Code: 00 75 0f 55 49 c1 e9 0c 48 89 e5 e8 b3 fe ff ff 5d c3 48 c7 c0 ea ff ff ff c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 17 <55> 48 8b 47 08 48 89 e5 48 c1 ea 10 83 e2 01 75 08 a8 01 48 0f 

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [runc:62503]
Modules linked in: xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel veth xt_set ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipport ip_set_hash_ipportip ip_set dummy nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack cambricon_gdrdrv(OE) joydev iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm
irqbypass crc32_pclmul vfat fat ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cp210x sg mei_me i2c_i801 mei wmi ipmi_si ipmi_devintf ipmi_msghandler pinctrl_lewisburg pinctrl_intel acpi_cpufreq acpi_power_meter cambricon_drv(OE) nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect serio_raw sysimgblt fb_sys_fops ttm i40e drm ahci libahci megaraid_sas ptp libata drm_panel_orientation_quirks pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 62503 Comm: runc Kdump: loaded Tainted: G        W  OE  ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Inspur NF5468M6/NF5468M6, BIOS 06.01.02 06/28/2022
task: ffff9aaf0cf1e180 ti: ffff9ba81ff60000 task.ti: ffff9ba81ff60000
RIP: 0010:[<ffffffffa7bd61a9>]  [<ffffffffa7bd61a9>] page_mapping+0x9/0x50
RSP: 0018:ffff9ba81ff63ce0  EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffff9b287fad7c80 RCX: 0000000000000001
RDX: 00efffff00000848 RSI: 0000000000000001 RDI: ffffeb241d322fc0
RBP: ffff9ba81ff63cf8 R08: ffff9b2654290c00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 0000000098fadaec R14: ffff9ba81ff63c38 R15: ffff9ba81ff63c38
FS:  00007f54b8509740(0000) GS:ffff9b287fac0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7c1f07af00 CR3: 0000006f7222c000 CR4: 0000000000760fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa7bca3c2>] ? page_evictable+0x12/0x40
[<ffffffffa7bca41e>] putback_lru_page+0x2e/0xb0
[<ffffffffa7c32faa>] mem_cgroup_reparent_charges+0x22a/0x3c0
[<ffffffffa7c332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa7b2573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa7b259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffa7c4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffa7c530d1>] do_rmdir+0x1f1/0x220
[<ffffffffa7c47034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffffa7c54335>] SyS_unlinkat+0x25/0x40
[<ffffffffa8174ddb>] system_call_fastpath+0x22/0x27
Code: 75 0f 55 49 c1 e9 0c 48 89 e5 e8 b3 fe ff ff 5d c3 48 c7 c0 ea ff ff ff c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 17 55 <48> 8b 47 08 48 89 e5 48 c1 ea 10 83 e2 01 75 08 a8 01 48 0f 45 

NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [slurmstepd:206978]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt dell_smbios iTCO_vendor_support sparse_keymap irqbypass dcdbas crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ghash_clmulni_intel drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter
bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci libata i40e(OE) megaraid_sas tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 13 PID: 206978 Comm: slurmstepd Tainted: G           OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.21.1 03/07/2024
task: ffff884024350fd0 ti: ffff884b53ba0000 task.ti: ffff884b53ba0000
RIP: 0010:[<ffffffff811a1338>]  [<ffffffff811a1338>] page_mapping+0x8/0x40
RSP: 0018:ffff884b53ba3d18  EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffff885fb7f91ae0 RCX: 0000000000000001
RDX: 006fffff0000000c RSI: 0000000000000001 RDI: ffffea00c25cfd00
RBP: ffff884b53ba3d28 R08: ffff885f4133b800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 00000000c0915e35 R14: ffff884b53ba3c68 R15: ffff884b53ba3c68
FS:  00002aaaaaafd180(0000) GS:ffff885fb7f80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005555557c58d8 CR3: 00000017052b8000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff81195ca2 ffffea00c25cfd00 ffff884b53ba3d50 ffffffff81195cfe
ffffea00c25cfd00 ffff884241033430 ffff88607bfd7540 ffff884b53ba3dd0
ffffffff811f342a 0000000200000001 ffff881e58310500 0000000000000001
Call Trace:
[<ffffffff81195ca2>] ? page_evictable+0x12/0x40
[<ffffffff81195cfe>] putback_lru_page+0x2e/0xb0
[<ffffffff811f342a>] mem_cgroup_reparent_charges+0x22a/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 00 75 0f 55 49 c1 e9 0c 48 89 e5 e8 c3 fe ff ff 5d c3 48 c7 c0 ea ff ff ff c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 17 <55> 48 8b 47 08 48 89 e5 48 c1 ea 10 83 e2 01 75 08 a8 01 48 0f 

NMI watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [systemd:1]
Modules linked in: gsch(OE) redirfs(OE) dsa_filter(POE) dinet(OE) difile(OE) binfmt_misc af_packet_diag netlink_diag fuse veth tcp_diag udp_diag inet_diag unix_diag ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) team_mode_loadbalance team sunrpc vfat fat iTCO_wdt iTCO_vendor_support dell_smbios dcdbas dell_wmi_descriptor skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ipmi_ssif ses enclosure joydev sg lpc_ich i2c_i801 mei_me mei wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter
ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx5_core drm mpt3sas ahci libahci raid_class mlxfw scsi_transport_sas ptp drm_panel_orientation_quirks libata pps_core devlink nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod [last unloaded: redirfs]
CPU: 20 PID: 1 Comm: systemd Kdump: loaded Tainted: P           OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R740xd/0WXD1Y, BIOS 2.6.4 04/09/2020
task: ffff8b2067700000 ti: ffff8ae0e736c000 task.ti: ffff8ae0e736c000
RIP: 0010:[<ffffffffb6dd61c0>]  [<ffffffffb6dd61c0>] page_mapping+0x20/0x50
RSP: 0018:ffff8ae0e736fcc8  EFLAGS: 00000246
RAX: ffff8b20521fc270 RBX: ffff8b1f3cc97c80 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000001 RDI: fffff459790db540
RBP: ffff8ae0e736fcd8 R08: ffff8b5f387d9800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 00000000359deef4 R14: ffff8ae0e736fc18 R15: ffff8ae0e736fc18
FS:  00007fea1f015940(0000) GS:ffff8b1f3cc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000ce150000 CR3: 0000003f7cdbe000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffb6dca3c2>] ? page_evictable+0x12/0x40
[<ffffffffb6dca41e>] putback_lru_page+0x2e/0xb0
[<ffffffffb6e32faa>] mem_cgroup_reparent_charges+0x22a/0x3c0
[<ffffffffb6e332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffb6d2573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffb6d259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffb6e4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffb6e530d1>] do_rmdir+0x1f1/0x220
[<ffffffffb6e54306>] SyS_rmdir+0x16/0x20
[<ffffffffc089aba9>] gsch_rmdir_hook_fn+0x49/0x70 [gsch]
[<ffffffffb7374ddb>] system_call_fastpath+0x22/0x27
Code: ff c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 17 55 48 8b 47 08 48 89 e5 48 c1 ea 10 83 e2 01 75 08 a8 01 48 0f 45 c2 5d <c3> 48 8b 47 30 5d 48 89 c2 48 c1 e0 07 48 c1 e8 15 48 c1 ea 39 

NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [slurmstepd:33481]
Modules linked in: nvidia_uvm(OE) binfmt_misc fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache nvidia_drm(POE) nvidia_modeset(POE) btrfs iTCO_wdt iTCO_vendor_support nvidia(POE) raid456 async_raid6_recov async_memcpy async_pq coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel vfat lrw gf128mul fat glue_helper ablk_helper cryptd raid6_pq async_xor xor async_tx pcspkr cdc_ether joydev usbnet ipmi_ssif mii i2c_i801 sg shpchp ipmi_si ipmi_devintf ipmi_msghandler pinctrl_lewisburg pinctrl_intel
mei_me acpi_cpufreq mei acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic uas usb_storage nouveau ast video mxm_wmi drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea nvme sysfillrect sysimgblt fb_sys_fops nvme_core ttm ahci drm igb libahci libata ptp pps_core dca i2c_algo_bit i2c_core wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 33481 Comm: slurmstepd Kdump: loaded Tainted: P        W  OEL ------------   3.10.0-862.el7.x86_64 #1
Hardware name: N/A X660 G45/60WB32, BIOS NKMH051061 05/12/2023
task: ffff8c28286a4f10 ti: ffff8bdf27648000 task.ti: ffff8bdf27648000
RIP: 0010:[<ffffffff9e5b3668>]  [<ffffffff9e5b3668>] page_mapping+0x8/0x50
RSP: 0018:ffff8bdf2764bce8  EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffff8b283fb95b80 RCX: 0000000000000001
RDX: 006fffff0000080c RSI: 0000000000000001 RDI: ffffe05e1ea38880
RBP: ffff8bdf2764bcf8 R08: ffff8b282a77a800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 00000000e9a0773b R14: ffff8bdf2764bc38 R15: ffff8bdf2764bc38
FS:  00002b7661057400(0000) GS:ffff8b283fb80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000825018 CR3: 000001f1e692a000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff9e5a7b92>] ? page_evictable+0x12/0x40
[<ffffffff9e5a7bee>] putback_lru_page+0x2e/0xb0
[<ffffffff9e60d08a>] mem_cgroup_reparent_charges+0x22a/0x3c0
[<ffffffff9e60d3d4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff9e51c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffff9e51c482>] cgroup_rmdir+0x22/0x40
[<ffffffff9e6295ac>] vfs_rmdir+0xdc/0x150
[<ffffffff9e62cb31>] do_rmdir+0x1f1/0x220
[<ffffffff9e62dd66>] SyS_rmdir+0x16/0x20
[<ffffffff9eb1f7d5>] system_call_fastpath+0x1c/0x21
Code: 00 75 0f 55 49 c1 e9 0c 48 89 e5 e8 b3 fe ff ff 5d c3 48 c7 c0 ea ff ff ff c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 17 <55> 48 8b 47 08 48 89 e5 48 c1 ea 10 83 e2 01 75 08 a8 01 48 0f 

NMI watchdog: BUG: soft lockup - CPU#63 stuck for 22s! [slurmstepd:204289]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt irqbypass iTCO_vendor_support crc32_pclmul crc32c_intel dell_smbios dcdbas sparse_keymap ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs
isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci libahci megaraid_sas tg3(OE) libata ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 63 PID: 204289 Comm: slurmstepd Tainted: G        W  OE  ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.8.1 06/30/2020
task: ffff8832c20f4f10 ti: ffff8829f8ea4000 task.ti: ffff8829f8ea4000
RIP: 0010:[<ffffffff811a1338>]  [<ffffffff811a1338>] page_mapping+0x8/0x40
RSP: 0018:ffff8829f8ea7d18  EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffff882872362c00 RCX: 0000000000000001
RDX: 006fffff0000004c RSI: 0000000000000001 RDI: ffffea011991ea80
RBP: ffff8829f8ea7d28 R08: ffff882872365400 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 000000003ede7655 R14: ffff8829f8ea7c68 R15: ffff8829f8ea7c68
FS:  00002aaaaaafd180(0000) GS:ffff885f7c5c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaaabe2e1a0 CR3: 0000000641f82000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff81195ca2 ffffea011991ea80 ffff8829f8ea7d50 ffffffff81195cfe
ffffea011991ea80 ffff883257fb4440 ffff88603ffd7540 ffff8829f8ea7dd0
ffffffff811f342a 0000000200000001 ffff882872362d00 0000000000000001
Call Trace:
[<ffffffff81195ca2>] ? page_evictable+0x12/0x40
[<ffffffff81195cfe>] putback_lru_page+0x2e/0xb0
[<ffffffff811f342a>] mem_cgroup_reparent_charges+0x22a/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 00 75 0f 55 49 c1 e9 0c 48 89 e5 e8 c3 fe ff ff 5d c3 48 c7 c0 ea ff ff ff c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 17 <55> 48 8b 47 08 48 89 e5 48 c1 ea 10 83 e2 01 75 08 a8 01 48 0f 

NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [systemd:1]
Modules linked in: nfsv3 xt_nat veth xt_addrtype br_netfilter bridge stp llc overlay(T) mfe_fileaccess_100709137(OE) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c iptable_filter dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel vfat kvm fat irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq joydev sg snd_seq_device snd_pcm pcspkr snd_timer snd lpc_ich i2c_i801 soundcore
mei_me mei shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_cpufreq acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel ahci ixgbe drm libahci megaraid_sas libata nvme mdio ptp i2c_core nvme_core pps_core dca [last unloaded: mfe_aac_100710239]
CPU: 18 PID: 1 Comm: systemd Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-862.2.3.el7.x86_64 #1
Hardware name: Supermicro SYS-7049GP-TRT/X11DPG-QT, BIOS 2.0 11/29/2017
task: ffff9423777a0000 ti: ffff94237779c000 task.ti: ffff94237779c000
RIP: 0010:[<ffffffffa51b3608>]  [<ffffffffa51b3608>] page_mapping+0x8/0x50
RSP: 0018:ffff94237779fce8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff9481bc015b80 RCX: 0000000000000001
RDX: 002fffff00020048 RSI: 0000000000000001 RDI: ffffefaf00a34ec0
RBP: ffff94237779fcf8 R08: ffff944297608000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 0000000061c3590a R14: ffff94237779fc38 R15: ffff94237779fc38
FS:  00007f808d397940(0000) GS:ffff9481bc000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb0786877b0 CR3: 0000003c85bb6000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa51a7b32>] ? page_evictable+0x12/0x40
[<ffffffffa51a7b8e>] putback_lru_page+0x2e/0xb0
[<ffffffffa520d02a>] mem_cgroup_reparent_charges+0x22a/0x3c0
[<ffffffffa520d374>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa511c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa511c482>] cgroup_rmdir+0x22/0x40
[<ffffffffa522954c>] vfs_rmdir+0xdc/0x150
[<ffffffffa522cad1>] do_rmdir+0x1f1/0x220
[<ffffffffa521d18e>] ? ____fput+0xe/0x10
[<ffffffffa50b7850>] ? task_work_run+0xc0/0xe0
[<ffffffffa522dd06>] SyS_rmdir+0x16/0x20
[<ffffffffa571f795>] system_call_fastpath+0x1c/0x21
Code: 00 75 0f 55 49 c1 e9 0c 48 89 e5 e8 b3 fe ff ff 5d c3 48 c7 c0 ea ff ff ff c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 17 <55> 48 8b 47 08 48 89 e5 48 c1 ea 10 83 e2 01 75 08 a8 01 48 0f 

NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [slurmstepd:93719]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge 8021q garp mrp stp llc overlay(T) iTCO_wdt iTCO_vendor_support ip6_tables iptable_filter coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses pcspkr joydev enclosure cdc_ether scsi_transport_sas usbnet mii sg mei_me mei i2c_i801 shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace
binfmt_misc sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul ttm crct10dif_common crc32c_intel drm serio_raw ahci libahci mlx5_core(OE) libata mlxfw(OE) ptp megaraid_sas pps_core auxiliary(OE) mlx_compat(OE) devlink i2c_core [last unloaded: tracedev]
CPU: 11 PID: 93719 Comm: slurmstepd Tainted: G        W  OE  ------------ T 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW, BIOS AFE112A-1.02 05/10/2021
task: ffff897e51d3cf10 ti: ffff88ef14724000 task.ti: ffff88ef14724000
RIP: 0010:[<ffffffff811a12b9>]  [<ffffffff811a12b9>] page_mapping+0x19/0x40
RSP: 0018:ffff88ef14727d10  EFLAGS: 00000246
RAX: ffff89cf498640e0 RBX: ffff89f83dcd1ae0 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffffea05cc1e67c0
RBP: ffff88ef14727d10 R08: ffff897a78290c00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 0000000041c0a068 R14: ffff88ef14727c68 R15: ffff88ef14727c68
FS:  00002adf66325fc0(0000) GS:ffff89f83dcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000008c0f68 CR3: 000001f7123ce000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff88ef14727d28 ffffffff81195cd2 ffffea05cc1e67c0 ffff88ef14727d50
ffffffff81195d2e ffffea05cc1e67c0 ffff8926ce144430 ffff8a003ffd8540
ffff88ef14727dd0 ffffffff811f339a 0000000200000000 ffff8802374bc500
Call Trace:
[<ffffffff81195cd2>] page_evictable+0x12/0x40
[<ffffffff81195d2e>] putback_lru_page+0x2e/0xb0
[<ffffffff811f339a>] mem_cgroup_reparent_charges+0x22a/0x3c0
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff81213446>] SyS_rmdir+0x16/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: c3 48 c7 c0 ea ff ff ff c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 17 55 48 8b 47 08 48 89 e5 48 c1 ea 10 83 e2 01 75 08 <a8> 01 48 0f 45 c2 5d c3 48 8b 47 30 5d 48 c1 e8 39 48 69 c0 b0