NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [slurmstepd:189137]
Modules linked in: binfmt_misc udp_diag nfsv3 nfs fscache xt_CHECKSUM tun devlink tcp_diag inet_diag xt_cgroup xt_NFLOG xt_comment nfnetlink_log veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter mpt2sas raid_class scsi_transport_sas mptctl mptbase overlay(T) nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ftp ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_dccp xt_sctp ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support mxm_wmi sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi dcdbas kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg ixgbe ipmi_si ipmi_devintf ipmi_msghandler mdio acpi_power_meter mei_me shpchp mei lpc_ich wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ahci i40e libahci igb libata megaraid_sas dca i2c_algo_bit ptp i2c_core pps_core
CPU: 4 PID: 189137 Comm: slurmstepd Tainted: G             L ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R730xd/0WCJNT, BIOS 2.7.1 001/22/2018
task: ffff8831089c0000 ti: ffff8802febc8000 task.ti: ffff8802febc8000
RIP: 0010:[<ffffffff81195906>]  [<ffffffff81195906>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff8802febcbd20  EFLAGS: 00000246
RAX: 000000000001a21c RBX: ffff88307ffda000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff88307ffda540
RBP: ffff8802febcbd50 R08: fffffffffffffffa R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff882fbe691ae0
R13: ffffea00b33e13c0 R14: 000000000000000e R15: 00000000db4dd8ff
FS:  00002b0d4c1a6480(0000) GS:ffff882fbe680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002dc00c8 CR3: 0000005fa1e39000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
 ffffffff811914ee ffffea00aeb6fe00 ffff88048064a430 ffff88307ffda540
 ffff880154b5bf80 ffffea00aeb6fe20 ffff8802febcbdd0 ffffffff811f335d
 0000000200000000 ffff883b9b9f0500 0000000000000000 0000000000001000
Call Trace:
 [<ffffffff811914ee>] ? lru_cache_add+0xe/0x10
 [<ffffffff811f335d>] mem_cgroup_reparent_charges+0x16d/0x3c0
 [<ffffffff811f3764>] mem_cgroup_css_offline+0x84/0x140
 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
 [<ffffffff8120f626>] vfs_rmdir+0xd6/0x150
 [<ffffffff81212251>] do_rmdir+0x1f1/0x220
 [<ffffffff816b02e5>] ? do_page_fault+0x35/0x90
 [<ffffffff81213456>] SyS_rmdir+0x16/0x20
 [<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 8b d3 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [slurmstepd:271668]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) squashfs loop overlay(T) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iTCO_wdt iosf_mbi iTCO_vendor_support kvm_intel kvm mgag200 ttm irqbypass crc32_pclmul crc32c_intel drm_kms_helper syscopyarea dell_smbios sysfillrect dcdbas sysimgblt fb_sys_fops sparse_keymap drm ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data
dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci libata i40e(OE) megaraid_sas tg3(OE) ptp pps_core [last unloaded: tracedev]
CPU: 8 PID: 271668 Comm: slurmstepd Tainted: G    B      OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.21.1 03/07/2024
task: ffff883167465ee0 ti: ffff885582384000 task.ti: ffff885582384000
RIP: 0010:[<ffffffff81195913>]  [<ffffffff81195913>] isolate_lru_page+0x93/0x1c0
RSP: 0018:ffff885582387d48  EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffffffff81190590 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff88607bfd7540
RBP: ffff885582387d50 R08: ffffffffffffffc5 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190eac
R13: ffff885582387cf8 R14: ffff88607bfd7000 R15: ffff88607bfd7000
FS:  00002aaaaaafd180(0000) GS:ffff882fb8b00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffff7ff5000 CR3: 0000004d297cd000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffea013a4e9f60 ffff885582387dd0 ffffffff811f336d 0000000200000001
ffff88038ba3f900 0000000000000001 0000000000001000 ffff88038ba3f840
0000000000000001 0000000100000003 ffff88038ba3f800 0000000000000000
Call Trace:
[<ffffffff811f336d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 48 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e <41> 5f 5d c3 66 0f 1f 84 00 00 00 00 00 49 c1 ec 36 4a 8b 04 e5 

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [runc:9235]
Modules linked in: isofs loop xt_nat xt_addrtype nf_conntrack_netlink xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 iptable_filter xt_conntrack iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle ip_tables cmac arc4 md4 nls_utf8 cifs ccm veth ip_set rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) nfnetlink br_netfilter tun bridge stp llc nfsv3 nfs_acl nfs lockd grace fscache overlay(T) bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) libcrc32c sunrpc dm_mirror dm_region_hash
dm_log dm_mod iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr wdat_wdt joydev sg i2c_i801 mei_me lpc_ich mei ipmi_si ipmi_devintf ipmi_msghandler knem(OE) ext4 mbcache jbd2 mlx4_ib(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe drm ahci mdio libahci mlx4_core(OE) igb libata drm_panel_orientation_quirks megaraid_sas ptp devlink pps_core dca mlx_compat(OE) i2c_algo_bit [last unloaded: ip_tables]
CPU: 1 PID: 9235 Comm: runc Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Inspur TS860G3/TS860G3, BIOS TS860G3_4.0.07 10/09/2016
task: ffff93797aff8000 ti: ffff93919375c000 task.ti: ffff93919375c000
RIP: 0010:[<ffffffffaadca026>]  [<ffffffffaadca026>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff93919375fcf0  EFLAGS: 00000246
RAX: 000000000001f7b4 RBX: ffff93113ffda000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff93113ffda540
RBP: ffff93919375fd20 R08: fffffffffffffff4 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff92f0bfa57c80
R13: ffffd118c69564c0 R14: 000000000000000e R15: 000000008e6805b0
FS:  00007ff0ae104740(0000) GS:ffff92f0bfa40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000563d8855d700 CR3: 0000001556ef8000 CR4: 00000000001607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaadc588e>] ? lru_cache_add+0xe/0x10
[<ffffffffaae32eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffffaae332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffaad2573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffaad259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffaae4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffaae530d1>] do_rmdir+0x1f1/0x220
[<ffffffffaae54335>] SyS_unlinkat+0x25/0x40
[<ffffffffab374ddb>] system_call_fastpath+0x22/0x27
Code: 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#47 stuck for 22s! [cgclear:70126]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace tun devlink ip_set nfnetlink bridge stp llc libcrc32c cachefiles fscache dm_mirror dm_region_hash dm_log dm_mod vfat fat sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr sg hpilo hpwdt ipmi_si lpc_ich i2c_i801 ipmi_devintf ipmi_msghandler ioatdma dca pcc_cpufreq wmi acpi_power_meter auth_rpcgss sunrpc ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit ahci crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel libahci syscopyarea libata sysfillrect sysimgblt fb_sys_fops tg3 ttm ptp drm pps_core drm_panel_orientation_quirks [last unloaded: ip_tables]
CPU: 47 PID: 70126 Comm: cgclear Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-957.el7.centos.plus.x86_64 #1
Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 01/12/2023
task: ffff8ad19072e180 ti: ffff8adfd3be4000 task.ti: ffff8adfd3be4000
RIP: 0010:[<ffffffffa2fcb026>]  [<ffffffffa2fcb026>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff8adfd3be7cf0  EFLAGS: 00000246
RAX: 000000000001f7b4 RBX: 0000000000000001 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8ae03ffd9540
RBP: ffff8adfd3be7d20 R08: fffffffffffffff3 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8acfffad7c80
R13: ffff8adfd3be7cf8 R14: 000000000000000e R15: 000000008d5b85a0
FS:  00007fd7b3ba0b80(0000) GS:ffff8acfffac0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1c24f8fb80 CR3: 0000001e80ea6000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa2fc688e>] ? lru_cache_add+0xe/0x10
[<ffffffffa3033eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffffa30342f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa2f2673a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa2f269e2>] cgroup_rmdir+0x22/0x40
[<ffffffffa304ec2c>] vfs_rmdir+0xdc/0x150
[<ffffffffa30540d1>] do_rmdir+0x1f1/0x220
[<ffffffffa3055306>] SyS_rmdir+0x16/0x20
[<ffffffffa3586ddb>] system_call_fastpath+0x22/0x27
Code: 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#35 stuck for 22s! [cgconfigparser:81491]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter cachefiles fscache dm_mirror dm_region_hash dm_log dm_mod vfat fat sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper
cryptd iTCO_wdt iTCO_vendor_support pcspkr sg i2c_i801 lpc_ich ipmi_si hpilo hpwdt ipmi_devintf ipmi_msghandler ioatdma pcc_cpufreq dca wmi acpi_power_meter auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect crct10dif_pclmul sysimgblt crct10dif_common ahci fb_sys_fops crc32c_intel ttm libahci libata drm tg3 ptp pps_core drm_panel_orientation_quirks
CPU: 35 PID: 81491 Comm: cgconfigparser Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.centos.plus.x86_64 #1
Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 04/29/2021
task: ffff8f56befc2080 ti: ffff8f5496890000 task.ti: ffff8f5496890000
RIP: 0010:[<ffffffff8f1cb026>]  [<ffffffff8f1cb026>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff8f5496893cf0  EFLAGS: 00000246
RAX: 000000000001f7b3 RBX: ffff8f67fffd9000 RCX: ffffffffffffffdd
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8f67fffd9540
RBP: ffff8f5496893d20 R08: fffffffffffffffe R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8f67befd7c80
R13: ffffd551f3b13880 R14: 000000000000000e R15: 000000004f8768e3
FS:  00007f059c3feb80(0000) GS:ffff8f67befc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f059b892150 CR3: 000000053faba000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8f1c688e>] ? lru_cache_add+0xe/0x10
[<ffffffff8f233eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff8f2342f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8f12673a>] cgroup_destroy_locked+0xea/0x370
[<ffffffff8f1269e2>] cgroup_rmdir+0x22/0x40
[<ffffffff8f24ec2c>] vfs_rmdir+0xdc/0x150
[<ffffffff8f2540d1>] do_rmdir+0x1f1/0x220
[<ffffffff8f255306>] SyS_rmdir+0x16/0x20
[<ffffffff8f78705b>] tracesys+0xa3/0xc9
Code: 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [systemd:1]
Modules linked in: nvidia_uvm(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs beegfs(OE) fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm i40iw mlx5_ib ib_core dm_mirror dm_region_hash dm_log dm_mod vfat fat iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw
gf128mul glue_helper ablk_helper cryptd pcspkr joydev ipmi_ssif sg mei_me mei i2c_i801 shpchp lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_cpufreq acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) raid1 sd_mod crc_t10dif crct10dif_generic nvidia(POE) ast i2c_algo_bit mlx5_core crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel i40e ttm syscopyarea sysfillrect sysimgblt fb_sys_fops drm devlink ahci ptp libahci pps_core libata i2c_core wmi
CPU: 7 PID: 1 Comm: systemd Tainted: P        W  OEL ------------   3.10.0-693.11.6.el7.x86_64 #1
Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 3.3 02/21/2020
task: ffff8840a7708000 ti: ffff880127320000 task.ti: ffff880127320000
RIP: 0010:[<ffffffff81197626>]  [<ffffffff81197626>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff880127323cf0  EFLAGS: 00000246
RAX: 000000000001bf5d RBX: ffff88017fd96800 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff88407ffda540
RBP: ffff880127323d20 R08: fffffffffffffff6 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff887f4f2d0000
R13: ffffea003ffc1100 R14: 000000000000000e R15: 00000000e9e8d96a
FS:  00007fb4a25b4940(0000) GS:ffff883f7e7c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000563789665000 CR3: 0000000032e8e000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8119320e>] ? lru_cache_add+0xe/0x10
[<ffffffff811f52ad>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f56b4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110f467>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110f712>] cgroup_rmdir+0x22/0x40
[<ffffffff812114f6>] vfs_rmdir+0xd6/0x150
[<ffffffff81214191>] do_rmdir+0x1f1/0x220
[<ffffffff81204fde>] ? ____fput+0xe/0x10
[<ffffffff810aedec>] ? task_work_run+0xac/0xf0
[<ffffffff81215396>] SyS_rmdir+0x16/0x20
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 bb d5 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [slurmstepd:106407]
Modules linked in: nvidia_uvm(OE) nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) vfio_iommu_type1 vfio nouveau mxm_wmi video xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache beegfs(OE) rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm i40iw mlx5_ib ib_core dm_mirror dm_region_hash dm_log dm_mod vfat fat iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl
iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev ipmi_ssif pcspkr sg mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper mlx5_core crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect i40e sysimgblt fb_sys_fops ttm drm devlink ahci libahci ptp pps_core libata i2c_core wmi [last unloaded: nvidia]
CPU: 4 PID: 106407 Comm: slurmstepd Tainted: P           OE  ------------   3.10.0-693.11.6.el7.x86_64 #1
Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 3.3 02/21/2020
task: ffff887d5f2d9fa0 ti: ffff8823c7868000 task.ti: ffff8823c7868000
RIP: 0010:[<ffffffff81197626>]  [<ffffffff81197626>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff8823c786bcf0  EFLAGS: 00000246
RAX: 000000000001bfc4 RBX: ffff88807ffd7000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff88807ffd7540
RBP: ffff8823c786bd20 R08: fffffffffffffff6 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff883f7e715ae0
R13: ffffea0144ddda40 R14: 000000000000000e R15: 0000000033726899
FS:  00002adcdeef6340(0000) GS:ffff883f7e700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002adcec401f48 CR3: 0000004b5b4f2000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8119320e>] ? lru_cache_add+0xe/0x10
[<ffffffff811f52ad>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f56b4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110f467>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110f712>] cgroup_rmdir+0x22/0x40
[<ffffffff812114f6>] vfs_rmdir+0xd6/0x150
[<ffffffff81214191>] do_rmdir+0x1f1/0x220
[<ffffffff811e0922>] ? kmem_cache_free+0x1e2/0x200
[<ffffffff81215396>] SyS_rmdir+0x16/0x20
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 bb d5 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#54 stuck for 22s! [slurmstepd:17417]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic nfs_layout_nfsv41_files crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod ses enclosure sd_mod sg intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd serio_raw pcspkr uas usb_storage smartpqi nvme scsi_transport_sas hpilo hpwdt lpc_ich shpchp wmi acpi_power_meter rpcrdma ib_isert iscsi_target_mod target_core_mod crc_t10dif crct10dif_generic ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm
ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables hfi1(OE) i2c_algo_bit i2c_core crct10dif_pclmul ipmi_devintf rdmavt(OE) crct10dif_common crc32c_intel ib_core bnxt_en ipmi_si fjes ipmi_msghandler
CPU: 54 PID: 17417 Comm: slurmstepd Tainted: G        W  OEL ------------ T 3.10.0-514.10.2.el7.x86_64 #1
Hardware name: HPE ProLiant DL560 Gen10/ProLiant DL560 Gen10, BIOS U34 04/21/2022
task: ffff8a73df2b0000 ti: ffff8af7cb8b4000 task.ti: ffff8af7cb8b4000
RIP: 0010:[<ffffffff811947a8>]  [<ffffffff811947a8>] isolate_lru_page+0x88/0x1f0
RSP: 0018:ffff8af7cb8b7d20  EFLAGS: 00000282
RAX: 000000000001a145 RBX: ffff8a003ffda000 RCX: ffffffffffffff83
RDX: fffffffffffffff9 RSI: 0000000000000005 RDI: ffff8a003ffda000
RBP: ffff8af7cb8b7d50 R08: fffffffffffffffa R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8bfc3ea11a20
R13: ffffea04170f87c0 R14: 000000000000000e R15: 00000000283c7511
FS:  00002ad9f9193300(0000) GS:ffff8bfc3ea00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001f07d20 CR3: 00000139ba349000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffffffffffff ffffea07ec2b00c0 ffff89e5582a3440 ffff8a003ffda540
ffff89014512c030 ffffea07ec2b00e0 ffff8af7cb8b7dd0 ffffffff811f124d
0000000200000001 ffff8946aee69900 0000000000000001 0000000000001000
Call Trace:
[<ffffffff811f124d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a278>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f315>] do_rmdir+0x1a5/0x200
[<ffffffff81692305>] ? do_page_fault+0x35/0x90
[<ffffffff812104c6>] SyS_rmdir+0x16/0x20
[<ffffffff81696b09>] system_call_fastpath+0x16/0x1b
Code: 97 4f 00 4c 89 e6 48 89 df e8 55 c3 05 00 48 8b 13 49 89 c7 83 e2 20 75 6a 0f 1f 44 00 00 66 41 83 45 00 02 fb 66 0f 1f 44 00 00 <48> 83 c4 08 44 89 f0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 

NMI watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [docker-runc:109134]
Modules linked in: veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c br_netfilter bridge overlay(T) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache 8021q garp mrp stp llc bonding rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ip6_tables ib_cm iw_cm i40iw ib_core skx_edac edac_core coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support sg joydev pcspkr mei_me mei lpc_ich i2c_i801 shpchp
ipmi_si ipmi_devintf ipmi_msghandler nfit acpi_cpufreq libnvdimm wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic uas usb_storage ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe igb drm crct10dif_pclmul i40e crct10dif_common mpt3sas crc32c_intel ahci libahci libata mdio dca raid_class i2c_algo_bit scsi_transport_sas ptp i2c_core pps_core
CPU: 12 PID: 109134 Comm: docker-runc Tainted: G        W      ------------ T 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Sugon CX50-G35/60P16B-TC4600T, BIOS 0KLSN218 05/22/2019
task: ffff880afaf02f70 ti: ffff882fd9db8000 task.ti: ffff882fd9db8000
RIP: 0010:[<ffffffff81195936>]  [<ffffffff81195936>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff882fd9dbbd20  EFLAGS: 00000246
RAX: 000000000001a284 RBX: ffff88303ffd7000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff88303ffd7540
RBP: ffff882fd9dbbd50 R08: fffffffffffffff6 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff882fdda91ae0
R13: ffffea0065628680 R14: 000000000000000e R15: 0000000042ea28df
FS:  0000000002278880(0063) GS:ffff882fdda80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001f35008 CR3: 000000279747f000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff8119151e ffffea00656283c0 ffff88268d439430 ffff88303ffd7540
ffff8818bee0a0f0 ffffea00656283e0 ffff882fd9dbbdd0 ffffffff811f32dd
0000000200000001 ffff88189a637d00 0000000000000001 0000000000001000
Call Trace:
[<ffffffff8119151e>] ? lru_cache_add+0xe/0x10
[<ffffffff811f32dd>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff81206a04>] ? SYSC_newfstatat+0x24/0x60
[<ffffffff81213475>] SyS_unlinkat+0x25/0x40
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 db d2 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [systemd:1]
Modules linked in: vhost_net vhost macvtap macvlan ebt_arp tun fuse devlink bnx2i libiscsi bnx2fc cnic uio fcoe libfcoe libfc scsi_transport_iscsi scsi_transport_fc scsi_tgt nf_conntrack_netbios_ns nf_conntrack_broadcast openvswitch ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 dm_mirror dm_region_hash dm_log dcdbas iTCO_wdt iTCO_vendor_support dm_multipath mxm_wmi sb_edac intel_powerclamp coretemp intel_rapl
iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_mod pcspkr sg mei_me lpc_ich mei shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm bnx2x ahci igb ixgbe libahci crct10dif_pclmul crct10dif_common crc32c_intel libata i2c_algo_bit megaraid_sas dca i2c_core mdio ptp pps_core libcrc32c 8021q garp mrp bridge stp llc bonding
CPU: 15 PID: 1 Comm: systemd Kdump: loaded Tainted: G        W      ------------   3.10.0-862.2.3.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.7.1 001/22/2018
task: ffff8f2d66c28000 ti: ffff8f2d66c24000 task.ti: ffff8f2d66c24000
RIP: 0010:[<ffffffff86915590>]  [<ffffffff86915590>] _raw_spin_lock_irq+0x0/0x30
RSP: 0018:ffff8f2d66c27ce8  EFLAGS: 00000286
RAX: 0000000000000001 RBX: ffff8f2dbfd8e400 RCX: 0000000000000002
RDX: 0000000000000002 RSI: ffffd4a890b8561c RDI: ffff8facbffd9540
RBP: ffff8f2d66c27d20 R08: ffff8f2dbfd8e400 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 00000000594c8803 R14: ffff8f2d66c27c38 R15: ffff8f2d66c27c38
FS:  00007f8f6180d940(0000) GS:ffff8fabbdfc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fade8663130 CR3: 0000003f7d6ae000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff863a776a>] ? isolate_lru_page+0x5a/0x1c0
[<ffffffff863a32fe>] ? lru_cache_add+0xe/0x10
[<ffffffff8640cf6d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff8640d374>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8631c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffff8631c482>] cgroup_rmdir+0x22/0x40
[<ffffffff8642954c>] vfs_rmdir+0xdc/0x150
[<ffffffff8642cad1>] do_rmdir+0x1f1/0x220
[<ffffffff8641d18e>] ? ____fput+0xe/0x10
[<ffffffff8642dd06>] SyS_rmdir+0x16/0x20
[<ffffffff8691fa11>] tracesys+0x9d/0xc3
Code: 00 e8 c5 47 98 ff 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 f0 ff 47 04 48 8b 7d 08 be 00 02 00 00 e8 a5 47 98 ff 5d c3 0f 1f 00 <0f> 1f 44 00 00 55 48 89 e5 fa 66 0f 1f 44 00 00 31 c0 ba 01 00 

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [slurmstepd:19443]
Modules linked in: nvidia_uvm(OE) binfmt_misc fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache nvidia_drm(POE) nvidia_modeset(POE) btrfs iTCO_wdt iTCO_vendor_support nvidia(POE) raid456 async_raid6_recov async_memcpy async_pq coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel vfat lrw gf128mul fat glue_helper ablk_helper cryptd raid6_pq async_xor xor async_tx pcspkr cdc_ether joydev usbnet ipmi_ssif mii i2c_i801 sg shpchp ipmi_si ipmi_devintf ipmi_msghandler pinctrl_lewisburg pinctrl_intel
mei_me acpi_cpufreq mei acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic uas usb_storage nouveau ast video mxm_wmi drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea nvme sysfillrect sysimgblt fb_sys_fops nvme_core ttm ahci drm igb libahci libata ptp pps_core dca i2c_algo_bit i2c_core wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 19443 Comm: slurmstepd Kdump: loaded Tainted: P        W  OE  ------------   3.10.0-862.el7.x86_64 #1
Hardware name: N/A X660 G45/60WB32, BIOS NKMH051061 05/12/2023
task: ffff8b6282ea0fd0 ti: ffff8c2822224000 task.ti: ffff8c2822224000
RIP: 0010:[<ffffffff9e5a77f6>]  [<ffffffff9e5a77f6>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff8c2822227cf0  EFLAGS: 00000246
RAX: 000000000001c09c RBX: ffff8b2c3ffda000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8b2c3ffda540
RBP: ffff8c2822227d20 R08: fffffffffffffffe R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8b283fa55b80
R13: ffffe0599305b8c0 R14: 000000000000000e R15: 000000000ad1c736
FS:  00002b0552e62400(0000) GS:ffff8b283fa40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7fce0808d0 CR3: 000001fc5a6b6000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff9e5a335e>] ? lru_cache_add+0xe/0x10
[<ffffffff9e60cfcd>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff9e60d3d4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff9e51c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffff9e51c482>] cgroup_rmdir+0x22/0x40
[<ffffffff9e6295ac>] vfs_rmdir+0xdc/0x150
[<ffffffff9e62cb31>] do_rmdir+0x1f1/0x220
[<ffffffff9eb1a5bc>] ? __do_page_fault+0x1bc/0x4f0
[<ffffffff9e62dd66>] SyS_rmdir+0x16/0x20
[<ffffffff9eb1f7d5>] system_call_fastpath+0x1c/0x21
Code: 89 e6 48 89 df e8 0b 51 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [systemd:1]
Modules linked in: nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) vfio_iommu_type1 vfio xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs beegfs(OE) fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm i40iw mlx5_ib ib_core dm_mirror dm_region_hash dm_log dm_mod vfat fat iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass
crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev ipmi_ssif sg mei_me mei lpc_ich shpchp i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper mlx5_core crct10dif_pclmul crct10dif_common syscopyarea crc32c_intel sysfillrect sysimgblt i40e fb_sys_fops ttm drm ahci libahci devlink ptp libata pps_core i2c_core wmi [last unloaded: video]
CPU: 6 PID: 1 Comm: systemd Tainted: P    B      OEL ------------   3.10.0-693.11.6.el7.x86_64 #1
Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 3.3 02/21/2020
task: ffff8840a7708000 ti: ffff880127320000 task.ti: ffff880127320000
RIP: 0010:[<ffffffff81197626>]  [<ffffffff81197626>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff880127323cf0  EFLAGS: 00000246
RAX: 000000000001bfc5 RBX: ffff88017fd96800 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff88807ffd7540
RBP: ffff880127323d20 R08: fffffffffffffff6 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff88017fd96800
R13: ffffea01837c75c0 R14: 000000000000000e R15: 0000000085764510
FS:  00007fc016ee6940(0000) GS:ffff883f7e780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055b00d97a2a8 CR3: 0000003f75ba6000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8119320e>] ? lru_cache_add+0xe/0x10
[<ffffffff811f52ad>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f56b4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110f467>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110f712>] cgroup_rmdir+0x22/0x40
[<ffffffff812114f6>] vfs_rmdir+0xd6/0x150
[<ffffffff81214191>] do_rmdir+0x1f1/0x220
[<ffffffff81204fde>] ? ____fput+0xe/0x10
[<ffffffff810aedec>] ? task_work_run+0xac/0xf0
[<ffffffff81215396>] SyS_rmdir+0x16/0x20
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 bb d5 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [slurmstepd:176738]
Modules linked in: nfs_layout_nfsv41_files squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel mgag200 ttm aesni_intel drm_kms_helper lrw gf128mul glue_helper syscopyarea ablk_helper sysfillrect iTCO_wdt sysimgblt fb_sys_fops iTCO_vendor_support mxm_wmi cryptd dcdbas cdc_ether drm usbnet mei_me sb_edac pcspkr mii mei edac_core shpchp lpc_ich wmi acpi_power_meter tpm_crb binfmt_misc rpcrdma ib_isert iscsi_target_mod target_core_mod crc_t10dif crct10dif_generic ib_iser
libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables hfi1(OE) ipmi_devintf rdmavt(OE) ixgbe ahci ib_core dca libahci mdio ipmi_si i2c_algo_bit crct10dif_pclmul ptp crct10dif_common libata i2c_core crc32c_intel pps_core fjes ipmi_msghandler
CPU: 17 PID: 176738 Comm: slurmstepd Tainted: G           OE  ------------ T 3.10.0-514.10.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge C6320/082F9M, BIOS 2.1.5 04/12/2016
task: ffff880f439b5e20 ti: ffff88067d814000 task.ti: ffff88067d814000
RIP: 0010:[<ffffffff811947a8>]  [<ffffffff811947a8>] isolate_lru_page+0x88/0x1f0
RSP: 0018:ffff88067d817d20  EFLAGS: 00000282
RAX: 000000000001a0dd RBX: ffff88107ffda000 RCX: ffffffffffffff9c
RDX: ffffffffffffffff RSI: 0000000000000005 RDI: ffff88107ffda000
RBP: ffff88067d817d50 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000001 R12: ffff88203e011a20
R13: ffffea00304ce4c0 R14: 000000000000000e R15: 0000000066f8c2b4
FS:  00002b2dbfd94300(0000) GS:ffff88203e000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff9ef911000 CR3: 0000001096995000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffffffffffff ffffea003d857f00 ffff880423b11440 ffff88107ffda540
ffff880170e95fc0 ffffea003d857f20 ffff88067d817dd0 ffffffff811f124d
0000000200000000 ffff880168ed4900 0000000000000000 0000000000001000
Call Trace:
[<ffffffff811f124d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a278>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f315>] do_rmdir+0x1a5/0x200
[<ffffffff8168b5b0>] ? __schedule+0x3b0/0x990
[<ffffffff812104c6>] SyS_rmdir+0x16/0x20
[<ffffffff81696b09>] system_call_fastpath+0x16/0x1b
Code: 97 4f 00 4c 89 e6 48 89 df e8 55 c3 05 00 48 8b 13 49 89 c7 83 e2 20 75 6a 0f 1f 44 00 00 66 41 83 45 00 02 fb 66 0f 1f 44 00 00 <48> 83 c4 08 44 89 f0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 

NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [slurmstepd:42510]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs beegfs(OE) fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm i40iw mlx5_ib ib_core dm_mirror dm_region_hash dm_log dm_mod nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) vfat fat iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp nvidia(POE) coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul
ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev ipmi_ssif sg mei_me mei lpc_ich shpchp i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper mlx5_core crct10dif_pclmul crct10dif_common syscopyarea crc32c_intel sysfillrect sysimgblt i40e fb_sys_fops ttm drm ahci libahci devlink ptp libata pps_core i2c_core wmi
CPU: 2 PID: 42510 Comm: slurmstepd Tainted: P    B      OE  ------------   3.10.0-693.11.6.el7.x86_64 #1
Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 3.3 02/21/2020
task: ffff886789d3eeb0 ti: ffff8856900ac000 task.ti: ffff8856900ac000
RIP: 0010:[<ffffffff81197626>]  [<ffffffff81197626>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff8856900afcf0  EFLAGS: 00000246
RAX: 000000000001bfc4 RBX: ffff88807ffd7000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff88807ffd7540
RBP: ffff8856900afd20 R08: ffffffffffffffff R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff883f7e695ae0
R13: ffffea017222a2c0 R14: 000000000000000e R15: 0000000069ae67d6
FS:  00002b1dfc279340(0000) GS:ffff883f7e680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000005abf000 CR3: 0000007b49758000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8119320e>] ? lru_cache_add+0xe/0x10
[<ffffffff811f52ad>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f56b4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110f467>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110f712>] cgroup_rmdir+0x22/0x40
[<ffffffff812114f6>] vfs_rmdir+0xd6/0x150
[<ffffffff81214191>] do_rmdir+0x1f1/0x220
[<ffffffff816b3801>] ? __do_page_fault+0x171/0x450
[<ffffffff81215396>] SyS_rmdir+0x16/0x20
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 bb d5 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [nvidia-containe:60676]
Modules linked in: fuse xt_multiport ipt_rpfilter ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel nvidia_uvm(OE) veth xt_set ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_bitmap_port ip_set_hash_ipport ip_set dummy nf_tables(T) iptable_raw xt_CT ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6table_mangle ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nvidia_drm(POE) nvidia_modeset(POE) iTCO_wdt iTCO_vendor_support
skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd nvidia(POE) snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hda_core snd_seq pcspkr joydev snd_seq_device snd_hwdep snd_pcm snd_timer ipmi_ssif snd soundcore sg i2c_i801 lpc_ich mei_me mei shpchp ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_cpufreq acpi_power_meter acpi_pad nfsd overlay(T) nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic nouveau ast video mxm_wmi i2c_algo_bit drm_kms_helper ixgbe crct10dif_pclmul crct10dif_common syscopyarea crc32c_intel sysfillrect sysimgblt i40e fb_sys_fops ttm ahci
mdio drm dca libahci libata ptp pps_core i2c_core wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 60676 Comm: nvidia-containe Tainted: P    B      OEL ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: Kunqian KI4224G/KQ11DPG-OT-CPU, BIOS 3.3 02/21/2020
task: ffff883f53db1fa0 ti: ffff883ea860c000 task.ti: ffff883ea860c000
RIP: 0010:[<ffffffff81195906>]  [<ffffffff81195906>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff883ea860fd20  EFLAGS: 00000246
RAX: 000000000001a284 RBX: ffff88807ffd7000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff88807ffd7540
RBP: ffff883ea860fd50 R08: ffffffffffffffff R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff883f7e691ae0
R13: ffffea019d973940 R14: 000000000000000e R15: 0000000067af3354
FS:  00007efe81c65740(0000) GS:ffff883f7e680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055d5126d346c CR3: 0000003ea2264000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff811914ee ffffea019d9739c0 ffff887c7edd3430 ffff88807ffd7540
ffff8840d92dce70 ffffea019d9739e0 ffff883ea860fdd0 ffffffff811f335d
0000000200000001 ffff883b07698900 0000000000000001 0000000000001000
Call Trace:
[<ffffffff811914ee>] ? lru_cache_add+0xe/0x10
[<ffffffff811f335d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3764>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f626>] vfs_rmdir+0xd6/0x150
[<ffffffff81212251>] do_rmdir+0x1f1/0x220
[<ffffffff81206a84>] ? SYSC_newfstatat+0x24/0x60
[<ffffffff81213485>] SyS_unlinkat+0x25/0x40
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 8b d3 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [systemd:1]
Modules linked in: vhost_net vhost macvtap macvlan ebt_arp tun fuse devlink bnx2fc cnic uio fcoe libfcoe scsi_transport_iscsi libfc scsi_transport_fc scsi_tgt openvswitch nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 dm_mirror dm_region_hash dm_log dcdbas dm_multipath sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul
ghash_clmulni_intel aesni_intel lrw iTCO_wdt gf128mul iTCO_vendor_support glue_helper ablk_helper cryptd mxm_wmi dm_mod pcspkr sg lpc_ich mei_me mei shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm i40e ixgbe ahci crct10dif_pclmul igb crct10dif_common crc32c_intel libahci libata megaraid_sas mdio i2c_algo_bit ptp i2c_core pps_core dca 8021q garp mrp bridge stp llc bonding
CPU: 12 PID: 1 Comm: systemd Kdump: loaded Tainted: G        W      ------------   3.10.0-862.2.3.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.7.1 001/22/2018
task: ffff94e9676d8000 ti: ffff94a9e73e0000 task.ti: ffff94a9e73e0000
RIP: 0010:[<ffffffff90da7796>]  [<ffffffff90da7796>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff94a9e73e3cf0  EFLAGS: 00000246
RAX: 000000000001c104 RBX: ffff95293ffd9000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff95293ffd9540
RBP: ffff94a9e73e3d20 R08: fffffffffffffff7 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff94e83e795b80
R13: ffffef36a5f90480 R14: 000000000000000e R15: 00000000f47ebb3e
FS:  00007f78771b8940(0000) GS:ffff94e83e780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6d837f9c88 CR3: 0000007f7dc66000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff90da32fe>] ? lru_cache_add+0xe/0x10
[<ffffffff90e0cf6d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff90e0d374>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff90d1c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffff90d1c482>] cgroup_rmdir+0x22/0x40
[<ffffffff90e2954c>] vfs_rmdir+0xdc/0x150
[<ffffffff90e2cad1>] do_rmdir+0x1f1/0x220
[<ffffffff90e1d18e>] ? ____fput+0xe/0x10
[<ffffffff90e2dd06>] SyS_rmdir+0x16/0x20
[<ffffffff9131fa11>] tracesys+0x9d/0xc3
Code: 89 e6 48 89 df e8 0b 51 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [systemd:1]
Modules linked in: joydev nf_conntrack_netlink xt_multiport ipt_rpfilter xt_set iptable_raw ip6t_MASQUERADE nf_nat_masquerade_ipv6 iptable_mangle ip6table_filter ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6_tables xt_statistic xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_mark xt_addrtype ipt_REJECT nf_reject_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_comment xt_conntrack nf_conntrack iptable_filter ip_tables binfmt_misc ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel veth tun rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip_set nfnetlink dm_mirror dm_region_hash dm_log dm_mod sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul
glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support snd_hda_codec_hdmi snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq pcspkr snd_seq_device snd_pcm ipmi_ssif snd_timer snd i2c_i801 soundcore mei_me sg mei shpchp lpc_ich ipmi_si ipmi_devintf ipmi_msghandler acpi_pad fuse nfsd br_netfilter bridge auth_rpcgss nfs_acl stp lockd llc grace overlay(T) sunrpc xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) sd_mod crc_t10dif crct10dif_generic nvidia(POE) ast mxm_wmi drm_kms_helper ahci ttm libahci syscopyarea sysfillrect sysimgblt fb_sys_fops libata crct10dif_pclmul crct10dif_common igb drm crc32c_intel ptp pps_core dca i2c_algo_bit i2c_core wmi [last unloaded: ip6_tables]
CPU: 3 PID: 1 Comm: systemd Tainted: P           OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: TYAN B7079F77CV10HR-N/S7079GM2NR-N, BIOS V2.03.B10 09/07/2016
task: ffff880137290000 ti: ffff880137284000 task.ti: ffff880137284000
RIP: 0010:[<ffffffff81195906>]  [<ffffffff81195906>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff880137287d20  EFLAGS: 00000246
RAX: 000000000001a284 RBX: ffff88607ffd7000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff88607ffd7540
RBP: ffff880137287d50 R08: fffffffffffffff3 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff882fbf8d1ae0
R13: ffffea00e0afbd80 R14: 000000000000000e R15: 000000009415c034
FS:  00007f7b484ef900(0000) GS:ffff882fbf8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c001da0000 CR3: 0000002fba7a3000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff811914ee ffffea014dbe5600 ffff885c39748430 ffff88607ffd7540
ffff8830dd379580 ffffea014dbe5620 ffff880137287dd0 ffffffff811f336d
0000000200000001 ffff882fb0bffd00 0000000000000001 0000000000001000
Call Trace:
[<ffffffff811914ee>] ? lru_cache_add+0xe/0x10
[<ffffffff811f336d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff810ad24c>] ? task_work_run+0xac/0xf0
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 9b d3 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [kubelet:13752]
Modules linked in: ipip tunnel4 ip_tunnel vxlan ip6_udp_tunnel udp_tunnel binfmt_misc xt_statistic nf_conntrack_netlink xt_multiport ipt_rpfilter iptable_raw xt_set ip_set_hash_ip ip_set_hash_net ip_set veth ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables xt_nat ipt_REJECT nf_reject_ipv4 nf_tables(T) nfnetlink ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) dm_mirror dm_region_hash dm_log dm_mod sg i2c_piix4 joydev edac_core ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel virtio_balloon parport_pc aesni_intel parport lrw gf128mul glue_helper ablk_helper
cryptd pcspkr ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata floppy crct10dif_pclmul crct10dif_common crc32c_intel virtio_pci virtio_ring i2c_core serio_raw virtio [last unloaded: tunnel4]
CPU: 12 PID: 13752 Comm: kubelet Tainted: G               ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: RDO OpenStack Compute, BIOS 1.11.0-2.el7 04/01/2014
task: ffff880bb5e86eb0 ti: ffff880bd6ff8000 task.ti: ffff880bd6ff8000
RIP: 0010:[<ffffffff81195906>]  [<ffffffff81195906>] isolate_lru_page+0x86/0x1c0
RSP: 0000:ffff880bd6ffbd20  EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffff880c3ffd8000 RCX: ffffffffffffff9c
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff880c3ffd8540
RBP: ffff880bd6ffbd50 R08: fffffffffffffff6 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff880c0f511ae0
R13: ffffea002cc5be40 R14: 000000000000000e R15: 00000000f0b7aeb4
FS:  00007effbcff9700(0000) GS:ffff880c0f500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00008dff8 CR3: 0000000be1b07000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff811914ee ffffea000f16c540 ffff880c0c20c440 ffff880c3ffd8540
ffff88017c0db150 ffffea000f16c560 ffff880bd6ffbdd0 ffffffff811f335d
0000000200000000 ffff8802e1645900 0000000000000000 0000000000001000
Call Trace:
[<ffffffff811914ee>] ? lru_cache_add+0xe/0x10
[<ffffffff811f335d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3764>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f626>] vfs_rmdir+0xd6/0x150
[<ffffffff81212251>] do_rmdir+0x1f1/0x220
[<ffffffff810ad24c>] ? task_work_run+0xac/0xf0
[<ffffffff81213485>] SyS_unlinkat+0x25/0x40
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 8b d3 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef ff 14 25 50 31 a1 81 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#77 stuck for 22s! [systemd:1]
Modules linked in: rpcsec_gss_krb5 uas usb_storage binfmt_misc tcp_lp fuse xt_nat veth vxlan ip6_udp_tunnel udp_tunnel xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter nfsd auth_rpcgss nfs_acl lockd grace nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack loop ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod sunrpc ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt iTCO_vendor_support vfat fat i40iw skx_edac ib_core intel_powerclamp
coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel joydev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif pcspkr ses enclosure mei_me scsi_transport_sas sg mei shpchp lpc_ich i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter sch_fq_codel ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul syscopyarea crct10dif_common sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm drm i40e ahci libahci libata megaraid_sas ptp pps_core i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 77 PID: 1 Comm: systemd Kdump: loaded Tainted: G   M    W    L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: Inspur NF8480M5/IIMS, BIOS 4.1.11 12/26/2019
task: ffff89752e8c8000 ti: ffff89752e8bc000 task.ti: ffff89752e8bc000
RIP: 0010:[<ffffffff8a9a77f6>]  [<ffffffff8a9a77f6>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff89752e8bfcf0  EFLAGS: 00000246
RAX: 000000000001c16c RBX: ffff8a943ffda000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8a943ffda540
RBP: ffff89752e8bfd20 R08: fffffffffffffff3 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8af2beb55b80
R13: ffffc75b840bc080 R14: 000000000000000e R15: 000000009f676bef
FS:  00007f871bec3940(0000) GS:ffff8af2beb40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055954c935590 CR3: 00000125d081c000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff8a9a335e>] ? lru_cache_add+0xe/0x10
[<ffffffff8aa0cfcd>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff8aa0d3d4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8a91c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffff8a91c482>] cgroup_rmdir+0x22/0x40
[<ffffffff8aa295ac>] vfs_rmdir+0xdc/0x150
[<ffffffff8aa2cb31>] do_rmdir+0x1f1/0x220
[<ffffffff8aa1d1ee>] ? ____fput+0xe/0x10
[<ffffffff8a8b7850>] ? task_work_run+0xc0/0xe0
[<ffffffff8aa2dd66>] SyS_rmdir+0x16/0x20
[<ffffffff8af1f7d5>] system_call_fastpath+0x1c/0x21
Code: 89 e6 48 89 df e8 0b 51 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [systemd:1]
Modules linked in: veth isofs nfsv4 dns_resolver nfs fscache rpcsec_gss_krb5 xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache fuse jbd2 ses enclosure scsi_transport_sas edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr vfat
fat joydev uas usb_storage sg ipmi_ssif i2c_piix4 ipmi_si ipmi_devintf ipmi_msghandler pinctrl_amd i2c_designware_platform i2c_designware_core acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci igb drm libahci ptp libata nvme pps_core dca nvme_core i2c_algo_bit drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 11 PID: 1 Comm: systemd Kdump: loaded Tainted: G        W      ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Supermicro Super Server/H11DSi, BIOS 2.1 02/21/2020
task: ffff97ca239a0000 ti: ffff97aad36d4000 task.ti: ffff97aad36d4000
RIP: 0010:[<ffffffffa6bca023>]  [<ffffffffa6bca023>] isolate_lru_page+0x83/0x1c0
RSP: 0018:ffff97aad36d7cf0  EFLAGS: 00000246
RAX: 000000000001f7b5 RBX: ffff97aaffd9e800 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff97e9cfe59540
RBP: ffff97aad36d7d20 R08: fffffffffffffff4 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000001 R12: ffff97abd216fc00
R13: ffffe2b9f3570dc0 R14: 000000000000000e R15: 000000008479d84f
FS:  00007f49d3f46940(0000) GS:ffff97c94d2c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f49d3f59000 CR3: 0000001fa17d8000 CR4: 0000000000340fe0
Call Trace:
[<ffffffffa6bc588e>] ? lru_cache_add+0xe/0x10
[<ffffffffa6c32eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffffa6c332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa6b2573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa6b259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffa6c4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffa6c530d1>] do_rmdir+0x1f1/0x220
[<ffffffffa6c4363e>] ? ____fput+0xe/0x10
[<ffffffffa6abe7a0>] ? task_work_run+0xc0/0xe0
[<ffffffffa6c54306>] SyS_rmdir+0x16/0x20
[<ffffffffa7174ddb>] system_call_fastpath+0x22/0x27
Code: 5a 00 4c 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 66 66 66 90 fb 66 66 90 <66> 66 90 48 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 

NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [slurmstepd:18086]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge 8021q garp mrp stp llc overlay(T) ip6_tables iTCO_wdt iTCO_vendor_support iptable_filter coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure cdc_ether scsi_transport_sas usbnet joydev mii sg mei_me mei shpchp i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace
binfmt_misc sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw ahci libahci mlx5_core(OE) libata mlxfw(OE) ptp megaraid_sas pps_core auxiliary(OE) mlx_compat(OE) i2c_core devlink [last unloaded: tracedev]
CPU: 9 PID: 18086 Comm: slurmstepd Tainted: G        W  OEL ------------ T 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW, BIOS AFE112A-1.02 05/10/2021
task: ffff89f7db341fa0 ti: ffff89983eeec000 task.ti: ffff89983eeec000
RIP: 0010:[<ffffffff81195943>]  [<ffffffff81195943>] isolate_lru_page+0x93/0x1c0
RSP: 0018:ffff89983eeefd48  EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffffffff811905c0 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8a003ffd8540
RBP: ffff89983eeefd50 R08: fffffffffffffff5 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190edc
R13: ffff89983eeefcf8 R14: ffff8a003ffd8000 R15: ffff8a003ffd8000
FS:  00002b9018d4dfc0(0000) GS:ffff89f83dc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b901ea2959c CR3: 000001edf9d30000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffea046057faa0 ffff89983eeefdd0 ffffffff811f32dd 0000000200000000
ffff89f831387100 0000000000000000 0000000000001000 ffff89f831387040
0000000000000001 0000000100000002 ffff89f831387000 0000000000000000
Call Trace:
[<ffffffff811f32dd>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff81213446>] SyS_rmdir+0x16/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 48 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e <41> 5f 5d c3 66 0f 1f 84 00 00 00 00 00 49 c1 ec 36 4a 8b 04 e5 

NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [slurmstepd:96419]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge 8021q garp mrp stp llc overlay(T) ip6_tables iTCO_wdt iTCO_vendor_support iptable_filter coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure cdc_ether scsi_transport_sas usbnet joydev mii sg mei_me mei shpchp i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace
binfmt_misc sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw ahci libahci mlx5_core(OE) libata mlxfw(OE) ptp megaraid_sas pps_core auxiliary(OE) mlx_compat(OE) i2c_core devlink [last unloaded: tracedev]
CPU: 9 PID: 96419 Comm: slurmstepd Tainted: G        W  OEL ------------ T 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW, BIOS AFE112A-1.02 05/10/2021
task: ffff8871ed87bf40 ti: ffff88e8ff70c000 task.ti: ffff88e8ff70c000
RIP: 0010:[<ffffffff81195936>]  [<ffffffff81195936>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff88e8ff70fd20  EFLAGS: 00000246
RAX: 000000000001a21c RBX: ffff8a003ffd8000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8a003ffd8540
RBP: ffff88e8ff70fd50 R08: fffffffffffffffe R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff89f83dc51ae0
R13: ffffea046f58ec40 R14: 000000000000000e R15: 000000005f09c44d
FS:  00002ac7c66b7fc0(0000) GS:ffff89f83dc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f431d93d000 CR3: 000001f7e9c71000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff8119151e ffffea066aa468c0 ffff88aa73d73430 ffff8a003ffd8540
ffff880365691a30 ffffea066aa468e0 ffff88e8ff70fdd0 ffffffff811f32dd
0000000200000000 ffff8873f68be500 0000000000000000 0000000000001000
Call Trace:
[<ffffffff8119151e>] ? lru_cache_add+0xe/0x10
[<ffffffff811f32dd>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff816a904d>] ? __schedule+0x39d/0x8b0
[<ffffffff81213446>] SyS_rmdir+0x16/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 db d2 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [slurmstepd:2457]
Modules linked in: nvidia_uvm(OE) nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 devlink tun overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ext4 mbcache jbd2 iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper
cryptd snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm pcspkr joydev ipmi_ssif snd_timer snd sg lpc_ich i2c_i801 soundcore shpchp mei_me mei ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_cpufreq acpi_pad acpi_power_meter ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul ttm crct10dif_common syscopyarea crc32c_intel sysfillrect sysimgblt fb_sys_fops ahci drm ixgbe megaraid_sas libahci libata mdio ptp i2c_core pps_core dca wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 2457 Comm: slurmstepd Kdump: loaded Tainted: P           OEL ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: Supermicro SYS-7049GP-TRT/X11DPG-QT, BIOS 3.1 05/23/2019
task: ffff96068925dee0 ti: ffff9607dc020000 task.ti: ffff9607dc020000
RIP: 0010:[<ffffffff8e1a77f6>]  [<ffffffff8e1a77f6>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff9607dc023cf0  EFLAGS: 00000246
RAX: 000000000001c104 RBX: ffff9647bffd9000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff9647bffd9540
RBP: ffff9607dc023d20 R08: fffffffffffffffa R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff9606bcbd5b80
R13: fffff706144d05c0 R14: 000000000000000e R15: 000000007e10cb85
FS:  00002b8b8189d580(0000) GS:ffff9606bcbc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fce39bfb000 CR3: 000000409c504000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff8e1a335e>] ? lru_cache_add+0xe/0x10
[<ffffffff8e20cfcd>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff8e20d3d4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8e11c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffff8e11c482>] cgroup_rmdir+0x22/0x40
[<ffffffff8e2295ac>] vfs_rmdir+0xdc/0x150
[<ffffffff8e22cb31>] do_rmdir+0x1f1/0x220
[<ffffffff8e22dd66>] SyS_rmdir+0x16/0x20
[<ffffffff8e71f7d5>] system_call_fastpath+0x1c/0x21
Code: 89 e6 48 89 df e8 0b 51 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [runc:85894]
Modules linked in: tcp_lp vfat fat isofs xfs uas usb_storage nvidia_uvm(OE) veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs overlay(T) lockd grace fscache ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ucm rpcrdma sunrpc rdma_ucm ib_uverbs ib_umad ib_iser ib_ipoib rdma_cm iw_cm ib_cm libiscsi scsi_transport_iscsi iTCO_wdt iTCO_vendor_support snd_hda_codec_hdmi skx_edac coretemp intel_rapl iosf_mbi
kvm_intel kvm snd_hda_intel snd_hda_codec irqbypass crc32_pclmul snd_hda_core ghash_clmulni_intel snd_hwdep aesni_intel snd_seq lrw gf128mul glue_helper ablk_helper cryptd snd_seq_device mlx5_ib snd_pcm ib_core snd_timer snd joydev pcspkr ses soundcore enclosure scsi_transport_sas sg mei_me mei lpc_ich i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler acpi_cpufreq sch_fq_codel ip_tables ext4 mbcache jbd2 nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper mlx5_core syscopyarea sysfillrect ttm sysimgblt fb_sys_fops ahci crct10dif_pclmul crct10dif_common crc32c_intel drm libahci igb mlxfw libata devlink megaraid_sas ptp pps_core dca drm_panel_orientation_quirks i2c_algo_bit wmi nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 85894 Comm: runc Tainted: P        W  OE  ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Nettrix X640 G30/60P16-U4, BIOS NQYS041A18 03/15/2021
task: ffff88dffbf39040 ti: ffff88a1f636c000 task.ti: ffff88a1f636c000
RIP: 0010:[<ffffffffa89ca026>]  [<ffffffffa89ca026>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff88a1f636fcf0  EFLAGS: 00000246
RAX: 000000000001f7b5 RBX: ffff88a33fd96400 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff8921fffd9540
RBP: ffff88a1f636fd20 R08: fffffffffffffff3 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff88e28b5b9000
R13: ffffe448dad0ff40 R14: 000000000000000e R15: 0000000084f7e04c
FS:  00007f3ad7688740(0000) GS:ffff88e0ff6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f55639a7000 CR3: 0000000033138000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa89c588e>] ? lru_cache_add+0xe/0x10
[<ffffffffa8a32eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffffa8a332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa892573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa89259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffa8a4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffa8a530d1>] do_rmdir+0x1f1/0x220
[<ffffffffa8a47034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffffa8a54335>] SyS_unlinkat+0x25/0x40
[<ffffffffa8f74ddb>] system_call_fastpath+0x22/0x27
Code: 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [slurmstepd:88763]
Modules linked in: beegfs(OE) squashfs loop overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm i40iw mlx5_ib ib_core dm_mirror dm_region_hash dm_log dm_mod nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) vfat fat iTCO_wdt iTCO_vendor_support nvidia(POE) skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi
kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev ipmi_ssif sg mei_me mei lpc_ich i2c_i801 shpchp ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper mlx5_core crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect i40e sysimgblt fb_sys_fops ttm drm ahci devlink libahci ptp pps_core libata i2c_core wmi [last unloaded: beegfs]
CPU: 20 PID: 88763 Comm: slurmstepd Tainted: P    B      OE  ------------ T 3.10.0-693.11.6.el7.x86_64 #1
Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 3.3 02/21/2020
task: ffff88755b20cf10 ti: ffff887471a60000 task.ti: ffff887471a60000
RIP: 0010:[<ffffffff81197626>]  [<ffffffff81197626>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff887471a63cf0  EFLAGS: 00000246
RAX: 000000000001bfc5 RBX: ffff88807ffd7000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff88807ffd7540
RBP: ffff887471a63d20 R08: fffffffffffffff5 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff887f7da15ae0
R13: ffffea013a611500 R14: 000000000000000e R15: 000000007c176e8c
FS:  00002b7d05bf55c0(0000) GS:ffff887f7da00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff774192000 CR3: 0000001ac5a0e000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8119320e>] ? lru_cache_add+0xe/0x10
[<ffffffff811f52ad>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f56b4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110f467>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110f712>] cgroup_rmdir+0x22/0x40
[<ffffffff812114f6>] vfs_rmdir+0xd6/0x150
[<ffffffff81214191>] do_rmdir+0x1f1/0x220
[<ffffffff81215396>] SyS_rmdir+0x16/0x20
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 bb d5 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#88 stuck for 22s! [slurmstepd:127155]
Modules linked in: veth joydev squashfs loop mmfs26(OE) mmfslinux(OE) tracedev(OE) nf_conntrack_netlink nfnetlink overlay(T) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) mlx4_en(OE) mlx4_ib(OE) ib_uverbs(OE) ib_core(OE) mlx4_core(OE) sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vfat fat pcspkr cdc_ether usbnet mii sg lpc_ich i2c_i801 mei_me mei wmi ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler acpi_cpufreq acpi_pad acpi_power_meter binfmt_misc knem(OE) br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr
ip_vs nf_conntrack ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea mlx5_core(OE) sysfillrect mlxfw(OE) sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common devlink crc32c_intel ttm i40e mlx_compat(OE) drm ptp pps_core drm_panel_orientation_quirks ahci megaraid_sas libahci libata nfit libnvdimm
CPU: 88 PID: 127155 Comm: slurmstepd Kdump: loaded Tainted: G        W  OE  ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Lenovo ThinkSystem SR950 -[7X12CTO1WW]-/-[7X12CTO1WW]-, BIOS -[PSE122R-1.53]- 07/03/2019
task: ffff91891e39c100 ti: ffff915bfb314000 task.ti: ffff915bfb314000
RIP: 0010:[<ffffffffb53ca026>]  [<ffffffffb53ca026>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff915bfb317cf0  EFLAGS: 00000246
RAX: 000000000001f955 RBX: ffff91f77ffda000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff91f77ffda540
RBP: ffff915bfb317d20 R08: ffffffffffffffff R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff9135fe617c80
R13: ffffd7cc49faed80 R14: 000000000000000e R15: 0000000038e91aa2
FS:  00002b5892bd8300(0000) GS:ffff9135fe600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000087a3c0 CR3: 0000029eb0b90000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffb53c588e>] ? lru_cache_add+0xe/0x10
[<ffffffffb5432eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffffb54332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffb532573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffb53259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffb544dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffb54530d1>] do_rmdir+0x1f1/0x220
[<ffffffffb5454306>] SyS_rmdir+0x16/0x20
[<ffffffffb5974ddb>] system_call_fastpath+0x22/0x27
Code: 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#34 stuck for 22s! [runc:76694]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) beegfs(OE) obdclass(OE) lnet(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc libcfs(OE) ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 nfs_acl nfs lockd grace fscache overlay(T) bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) sunrpc dm_mirror dm_region_hash
dm_log dm_mod iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr wdat_wdt joydev sg i2c_i801 lpc_ich mei_me mei ipmi_si ipmi_devintf ipmi_msghandler knem(OE) ip_tables ext4 mbcache jbd2 mlx4_ib(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe drm igb mdio ahci mlx4_core(OE) ptp drm_panel_orientation_quirks libahci pps_core dca devlink i2c_algo_bit libata mlx_compat(OE) megaraid_sas
CPU: 34 PID: 76694 Comm: runc Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Inspur TS860G3/TS860G3, BIOS TS860G3_4.0.07 10/09/2016
task: ffff893014ea6180 ti: ffff8932ad434000 task.ti: ffff8932ad434000
RIP: 0010:[<ffffffff971ca026>]  [<ffffffff971ca026>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff8932ad437cf0  EFLAGS: 00000246
RAX: 000000000001f7b5 RBX: ffff889fbffda000 RCX: fffffffffffffff3
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff889fbffda540
RBP: ffff8932ad437d20 R08: fffffffffffffffb R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff88bf3fa97c80
R13: ffffd5e827102e80 R14: 000000000000000e R15: 00000000ea4d5407
FS:  00007fa1dfb93740(0000) GS:ffff88bf3fa80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3926b53d70 CR3: 00000058631c6000 CR4: 00000000001607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff971c588e>] ? lru_cache_add+0xe/0x10
[<ffffffff97232eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff972332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff9712573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffff971259e2>] cgroup_rmdir+0x22/0x40
[<ffffffff9724dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffff972530d1>] do_rmdir+0x1f1/0x220
[<ffffffff97247034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffff97254335>] SyS_unlinkat+0x25/0x40
[<ffffffff97774ddb>] system_call_fastpath+0x22/0x27
Code: 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#72 stuck for 23s! [runc:60187]
Modules linked in: xt_physdev xt_set ip_set_hash_ip vxlan ip6_udp_tunnel udp_tunnel ip_set ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_tables(T) binfmt_misc snd_seq snd_seq_device snd_timer snd soundcore tcp_diag udp_diag inet_diag tcp_lp fuse xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) sunrpc iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw
gf128mul glue_helper ablk_helper cryptd wdat_wdt pcspkr sg ipmi_ssif i2c_i801 ipmi_si lpc_ich ipmi_devintf ipmi_msghandler mei_me mei ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic qla2xxx crct10dif_pclmul crct10dif_common crc32c_intel ast nvme_fc drm_kms_helper nvme_fabrics nvme_core scsi_transport_fc syscopyarea sysfillrect sysimgblt scsi_tgt fb_sys_fops ttm ahci megaraid_sas libahci drm libata drm_panel_orientation_quirks ixgbe igb mdio ptp pps_core i2c_algo_bit dca dm_mirror dm_region_hash dm_log dm_mod
CPU: 72 PID: 60187 Comm: runc Kdump: loaded Tainted: G               ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Inspur NF8460M4/NF8460M4, BIOS 4.2.15 11/27/2017
task: ffff92d2a8e9b0c0 ti: ffff93016930c000 task.ti: ffff93016930c000
RIP: 0010:[<ffffffffacfca026>]  [<ffffffffacfca026>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff93016930fcf0  EFLAGS: 00000246
RAX: 000000000001f81d RBX: ffff9301bffda000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff9301bffda540
RBP: ffff93016930fd20 R08: ffffffffffffffff R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff93117ed17c80
R13: ffffde968235b740 R14: 000000000000000e R15: 00000000a3324a87
FS:  00007f6c068c2740(0000) GS:ffff93117ed00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055a5e3a2c430 CR3: 0000003fb0a46000 CR4: 00000000001607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffacfc588e>] ? lru_cache_add+0xe/0x10
[<ffffffffad032eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffffad0332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffacf2573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffacf259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffad04dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffad0530d1>] do_rmdir+0x1f1/0x220
[<ffffffffad054335>] SyS_unlinkat+0x25/0x40
[<ffffffffad574ddb>] system_call_fastpath+0x22/0x27
Code: 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#18 stuck for 23s! [slurmstepd:58166]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul sg ghash_clmulni_intel aesni_intel
lrw gf128mul iTCO_wdt hpwdt iTCO_vendor_support hpilo glue_helper ablk_helper cryptd sb_edac shpchp ipmi_ssif pcspkr edac_core ioatdma lpc_ich acpi_power_meter ipmi_si nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c mlx4_en sd_mod crc_t10dif crct10dif_generic nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) nvidia(POE) crct10dif_pclmul crct10dif_common mgag200 ipmi_msghandler crc32c_intel drm_kms_helper syscopyarea sysfillrect serio_raw mlx4_core sysimgblt fb_sys_fops ttm ahci igb libahci hpsa drm devlink libata ptp pps_core mpt2sas dca i2c_algo_bit i2c_core raid_class scsi_transport_sas wmi fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 18 PID: 58166 Comm: slurmstepd Tainted: P    B      OE  ------------   3.10.0-514.21.2.el7.x86_64 #1
Hardware name: HP ProLiant SL270s Gen8 SE/, BIOS P75 09/18/2013
task: ffff881fbee35e20 ti: ffff881fb4e04000 task.ti: ffff881fb4e04000
RIP: 0010:[<ffffffff81194c98>]  [<ffffffff81194c98>] isolate_lru_page+0x88/0x1f0
RSP: 0018:ffff881fb4e07d20  EFLAGS: 00000282
RAX: 000000000001a145 RBX: ffff88017fc4ec00 RCX: ffffffffffffff83
RDX: fffffffffffffff2 RSI: 0000000000000005 RDI: ffff88403ffd9000
RBP: ffff881fb4e07d50 R08: fffffffffffffff3 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffea0083343d80
R13: ffff8830eefb5800 R14: 000000000000000e R15: 00000000d8d92ba2
FS:  00007f07de337780(0000) GS:ffff88403f600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f01e4181000 CR3: 00000025f92f5000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffffffffffff ffffea008fe35f40 ffff883faa270440 ffff88403ffd9540
ffff8820bc00d7d0 ffffea008fe35f60 ffff881fb4e07dd0 ffffffff811f188d
0000000200000001 ffff8830eefb0900 0000000000000001 0000000000001000
Call Trace:
[<ffffffff811f188d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f1c94>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d6e7>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d982>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a8c8>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f965>] do_rmdir+0x1a5/0x200
[<ffffffff81210b16>] SyS_rmdir+0x16/0x20
[<ffffffff81697749>] system_call_fastpath+0x16/0x1b
Code: 9e 4f 00 4c 89 e6 48 89 df e8 a5 c4 05 00 48 8b 13 49 89 c7 83 e2 20 75 6a 0f 1f 44 00 00 66 41 83 45 00 02 fb 66 0f 1f 44 00 00 <48> 83 c4 08 44 89 f0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 

NMI watchdog: BUG: soft lockup - CPU#38 stuck for 23s! [slurmstepd:29254]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul
ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif pcspkr sg sb_edac edac_core lpc_ich hpwdt hpilo ioatdma shpchp ipmi_si acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c mlx4_en nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) sd_mod crc_t10dif crct10dif_generic nvidia(POE) crct10dif_pclmul mgag200 ipmi_msghandler crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect serio_raw sysimgblt mlx4_core fb_sys_fops ttm igb ahci hpsa libahci ptp drm devlink pps_core libata dca i2c_algo_bit mpt2sas i2c_core raid_class scsi_transport_sas wmi fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 38 PID: 29254 Comm: slurmstepd Tainted: P    B      OEL ------------   3.10.0-514.21.2.el7.x86_64 #1
Hardware name: HP ProLiant SL270s Gen8 SE/, BIOS P75 09/18/2013
task: ffff882ab5fd8fb0 ti: ffff8806303ec000 task.ti: ffff8806303ec000
RIP: 0010:[<ffffffff81194c98>]  [<ffffffff81194c98>] isolate_lru_page+0x88/0x1f0
RSP: 0018:ffff8806303efd20  EFLAGS: 00000282
RAX: 000000000001a144 RBX: ffff88403ffd9000 RCX: ffffffffffffff83
RDX: fffffffffffffff4 RSI: 0000000000000004 RDI: ffff88403ffd9000
RBP: ffff8806303efd50 R08: fffffffffffffff5 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff88403f891a20
R13: ffffea00b13d5380 R14: 000000000000000e R15: 00000000dc2cdd7a
FS:  00007fa956a43780(0000) GS:ffff88403f880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001c89710 CR3: 0000001fb32b2000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffffffffffff ffffea0097edcfc0 ffff8827d1a1b430 ffff88403ffd9540
ffff8820ba0373f0 ffffea0097edcfe0 ffff8806303efdd0 ffffffff811f188d
0000000200000001 ffff88234bf7f100 0000000000000001 0000000000001000
Call Trace:
[<ffffffff811f188d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f1c94>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d6e7>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d982>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a8c8>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f965>] do_rmdir+0x1a5/0x200
[<ffffffff81692f35>] ? do_page_fault+0x35/0x90
[<ffffffff81210b16>] SyS_rmdir+0x16/0x20
[<ffffffff81697749>] system_call_fastpath+0x16/0x1b
Code: 9e 4f 00 4c 89 e6 48 89 df e8 a5 c4 05 00 48 8b 13 49 89 c7 83 e2 20 75 6a 0f 1f 44 00 00 66 41 83 45 00 02 fb 66 0f 1f 44 00 00 <48> 83 c4 08 44 89 f0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 

NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [docker-runc:30146]
Modules linked in: vxlan ip6_udp_tunnel udp_tunnel xt_statistic ipt_REJECT nf_reject_ipv4 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack br_netfilter ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) bonding bridge stp llc iTCO_wdt iTCO_vendor_support sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev i2c_i801 sg mei_me lpc_ich mei ioatdma shpchp wmi ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad
nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm igb ahci drm libahci crct10dif_pclmul crct10dif_common crc32c_intel libata ptp megaraid_sas pps_core dca i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 13 PID: 30146 Comm: docker-runc Tainted: G               ------------ T 3.10.0-693.11.6.el7.x86_64 #1
Hardware name: Supermicro SYS-6028R-TR/X10DRi, BIOS 1.1 04/14/2015
task: ffff88166a3a0fd0 ti: ffff881fd664c000 task.ti: ffff881fd664c000
RIP: 0010:[<ffffffff81197626>]  [<ffffffff81197626>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff881fd664fcf0  EFLAGS: 00000246
RAX: 000000000001bfc4 RBX: ffff88407ffd7000 RCX: ffffffffffffff92
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff88407ffd7540
RBP: ffff881fd664fd20 R08: fffffffffffffff7 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff881fffbd5ae0
R13: ffffea00da7ff3c0 R14: 000000000000000e R15: 000000000bfa7de3
FS:  00007f062d99e740(0000) GS:ffff881fffbc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000559e0684ef60 CR3: 0000001b4a2a8000 CR4: 00000000001607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8119320e>] ? lru_cache_add+0xe/0x10
[<ffffffff811f52ad>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f56b4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110f467>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110f712>] cgroup_rmdir+0x22/0x40
[<ffffffff812114f6>] vfs_rmdir+0xd6/0x150
[<ffffffff81214191>] do_rmdir+0x1f1/0x220
[<ffffffff8120878e>] ? SYSC_newstat+0x2e/0x60
[<ffffffff816b3801>] ? __do_page_fault+0x171/0x450
[<ffffffff812153c5>] SyS_unlinkat+0x25/0x40
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 bb d5 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#50 stuck for 23s! [runc:98470]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) beegfs(OE) obdclass(OE) lnet(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc libcfs(OE) ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 nfs_acl nfs lockd grace fscache overlay(T) bonding rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) sunrpc dm_mirror dm_region_hash
dm_log dm_mod iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr wdat_wdt joydev sg i2c_i801 lpc_ich mei_me mei ipmi_si ipmi_devintf ipmi_msghandler knem(OE) ip_tables ext4 mbcache jbd2 mlx4_ib(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe drm igb mdio ahci mlx4_core(OE) ptp drm_panel_orientation_quirks libahci pps_core dca devlink i2c_algo_bit libata mlx_compat(OE) megaraid_sas
CPU: 50 PID: 98470 Comm: runc Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Inspur TS860G3/TS860G3, BIOS TS860G3_4.0.07 10/09/2016
task: ffff892112f0e180 ti: ffff8896b87c8000 task.ti: ffff8896b87c8000
RIP: 0010:[<ffffffff971ca026>]  [<ffffffff971ca026>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff8896b87cbcf0  EFLAGS: 00000246
RAX: 000000000001fa25 RBX: ffff895fbffd9000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff895fbffd9540
RBP: ffff8896b87cbd20 R08: fffffffffffffff8 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff88df3fa97c80
R13: ffffd5eb482fb680 R14: 000000000000000e R15: 0000000077d5f63a
FS:  00007f87f247f740(0000) GS:ffff88df3fa80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000c00bbc CR3: 0000007fd4a46000 CR4: 00000000001607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff971c588e>] ? lru_cache_add+0xe/0x10
[<ffffffff97232eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff972332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff9712573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffff971259e2>] cgroup_rmdir+0x22/0x40
[<ffffffff9724dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffff972530d1>] do_rmdir+0x1f1/0x220
[<ffffffff97247034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffff97254335>] SyS_unlinkat+0x25/0x40
[<ffffffff97774ddb>] system_call_fastpath+0x22/0x27
Code: 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [slurmstepd:59295]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel
lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr sg sb_edac edac_core ipmi_ssif lpc_ich hpilo hpwdt ioatdma shpchp ipmi_si acpi_power_meter pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c mlx4_en nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) sd_mod crc_t10dif crct10dif_generic nvidia(POE) crct10dif_pclmul crct10dif_common crc32c_intel mgag200 ipmi_msghandler serio_raw drm_kms_helper mlx4_core syscopyarea sysfillrect sysimgblt ttm fb_sys_fops ahci igb hpsa libahci devlink drm ptp pps_core libata dca i2c_algo_bit i2c_core mpt2sas raid_class scsi_transport_sas wmi fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 15 PID: 59295 Comm: slurmstepd Tainted: P           OE  ------------   3.10.0-514.21.1.el7.x86_64 #1
Hardware name: HP ProLiant SL270s Gen8 SE/, BIOS P75 09/18/2013
task: ffff883f9a2b8000 ti: ffff883f957d0000 task.ti: ffff883f957d0000
RIP: 0010:[<ffffffff81194c2c>]  [<ffffffff81194c2c>] isolate_lru_page+0x1c/0x1f0
RSP: 0018:ffff883f957d3d20  EFLAGS: 00000282
RAX: 006fffff0000002c RBX: ffff88403ffd9000 RCX: 0000000000000001
RDX: 0000000000000001 RSI: ffffea00972fd99c RDI: ffffea00972fd980
RBP: ffff883f957d3d50 R08: 0000000000000287 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff88403f551a20
R13: ffffea00972fd600 R14: 000000000000000e R15: 000000008d04e389
FS:  00007f9cea3d7780(0000) GS:ffff88403f540000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fed18884e00 CR3: 0000001fa9ff4000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffea00972fd940 ffffea00972fd980 ffff883fbc7ee430 ffff88403ffd9540
ffff8820ba53f660 ffffea00972fd9a0 ffff883f957d3dd0 ffffffff811f173d
0000000200000001 ffff883fa9bd6100 0000000000000001 0000000000001000
Call Trace:
[<ffffffff811f173d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f1b44>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d6e7>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d982>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a778>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f815>] do_rmdir+0x1a5/0x200
[<ffffffff812109c6>] SyS_rmdir+0x16/0x20
[<ffffffff816975c9>] system_call_fastpath+0x16/0x1b
Code: ff ff ff 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 89 fb 48 83 ec 08 48 8b 07 <a8> 20 0f 84 8c 01 00 00 4c 8b 27 41 f7 c4 00 00 00 80 75 70 4c 

NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [systemd:1]
Modules linked in: veth xt_addrtype br_netfilter overlay(T) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache devlink nls_utf8 isofs loop xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support dell_smbios sparse_keymap dcdbas vfat fat skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif joydev sg pcspkr ipmi_si ipmi_devintf ipmi_msghandler mei_me mei shpchp nfit i2c_i801 lpc_ich acpi_power_meter wmi libnvdimm acpi_pad tpm_crb
nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci tg3 libata megaraid_sas bnxt_en ptp i2c_core pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 10 PID: 1 Comm: systemd Tainted: G        W    L ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R740/0DY2X0, BIOS 2.8.2 08/27/2020
task: ffff8840a7618000 ti: ffff880127164000 task.ti: ffff880127164000
RIP: 0010:[<ffffffff81195906>]  [<ffffffff81195906>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff880127167d20  EFLAGS: 00000246
RAX: 000000000001a355 RBX: ffff88017fc6e400 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff89007ffd7540
RBP: ffff880127167d50 R08: fffffffffffffff5 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff882c68750800
R13: ffffea03e9d3cd80 R14: 000000000000000e R15: 00000000ae900d74
FS:  00007f0b0aa9c940(0000) GS:ffff887f7e280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000559c52261ac8 CR3: 000000ff7bbfa000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff811914ee ffffea03d8dbcac0 ffff88ff7ba9a440 ffff89007ffd7540
ffff88c0ca7ef2b0 ffffea03d8dbcae0 ffff880127167dd0 ffffffff811f335d
0000000200000003 ffff882c68750900 0000000000000003 0000000000001000
Call Trace:
[<ffffffff811914ee>] ? lru_cache_add+0xe/0x10
[<ffffffff811f335d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3764>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f626>] vfs_rmdir+0xd6/0x150
[<ffffffff81212251>] do_rmdir+0x1f1/0x220
[<ffffffff810ad24c>] ? task_work_run+0xac/0xf0
[<ffffffff81213456>] SyS_rmdir+0x16/0x20
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 8b d3 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [systemd:1]
Modules linked in: qax_tq_base(OE) veth ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc overlay(T) nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache bonding sunrpc dm_mirror dm_region_hash dm_log dm_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel pcspkr aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ipmi_si ipmi_devintf ipmi_msghandler i2c_designware_platform pinctrl_amd i2c_designware_core acpi_cpufreq i2c_piix4 sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm drm ahci drm_panel_orientation_quirks libahci libata megaraid_sas igb(OE) i2c_algo_bit nfit libnvdimm ixgbe mdio ptp pps_core dca [last unloaded: qax_tq_base]
CPU: 14 PID: 1 Comm: systemd Kdump: loaded Tainted: G        W  OE  ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: N/A 2U Rack Server/65N32-US, BIOS SZZQ042030 07/27/2020
task: ffff91c5e75c8000 ti: ffff9086e7358000 task.ti: ffff9086e7358000
RIP: 0010:[<ffffffff88bca023>]  [<ffffffff88bca023>] isolate_lru_page+0x83/0x1c0
RSP: 0018:ffff9086e735bcf0  EFLAGS: 00000246
RAX: 000000000001f74d RBX: ffff90c60feda000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff90c60feda540
RBP: ffff9086e735bd20 R08: fffffffffffffff5 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff90c50f997c80
R13: ffffe8378fdaa580 R14: 000000000000000e R15: 0000000080003444
FS:  00007f329c60e940(0000) GS:ffff90c50f980000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe69a7fca08 CR3: 0000023efbcea000 CR4: 0000000000340fe0
Call Trace:
[<ffffffff88bc588e>] ? lru_cache_add+0xe/0x10
[<ffffffff88c32eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff88c332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff88b2573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffff88b259e2>] cgroup_rmdir+0x22/0x40
[<ffffffff88c4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffff88c530d1>] do_rmdir+0x1f1/0x220
[<ffffffff88c4363e>] ? ____fput+0xe/0x10
[<ffffffff88abe7a0>] ? task_work_run+0xc0/0xe0
[<ffffffff88c54306>] SyS_rmdir+0x16/0x20
[<ffffffff89174ddb>] system_call_fastpath+0x22/0x27
Code: 5a 00 4c 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 66 66 66 90 fb 66 66 90 <66> 66 90 48 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 

NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd:1]
Modules linked in: vxlan ip6_udp_tunnel udp_tunnel iptable_mangle xt_mark veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) sb_edac edac_core coretemp iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr nfit ppdev vmw_balloon joydev libnvdimm sg parport_pc parport shpchp vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod vmwgfx crc_t10dif crct10dif_generic drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 i2c_core vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 1 Comm: systemd Tainted: G           OEL ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff880174a10000 ti: ffff880174a18000 task.ti: ffff880174a18000
RIP: 0010:[<ffffffff81195903>]  [<ffffffff81195903>] isolate_lru_page+0x83/0x1c0
RSP: 0018:ffff880174a1bd20  EFLAGS: 00000246
RAX: 000000000001a21d RBX: ffff88083ffda000 RCX: ffffffffffffffb8
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff88083ffda540
RBP: ffff880174a1bd50 R08: fffffffffffffff6 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff88081fc11ae0
R13: ffffea000916e680 R14: 000000000000000e R15: 00000000f363d1dd
FS:  00007f041c831940(0000) GS:ffff88081fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f041c842000 CR3: 000000081d2aa000 CR4: 00000000000407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffff811914ee ffffea0020044900 ffff880806a68440 ffff88083ffda540
ffff880177c91240 ffffea0020044920 ffff880174a1bdd0 ffffffff811f335d
0000000200000000 ffff88101f39e100 0000000000000000 0000000000001000
Call Trace:
[<ffffffff811914ee>] ? lru_cache_add+0xe/0x10
[<ffffffff811f335d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3764>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f626>] vfs_rmdir+0xd6/0x150
[<ffffffff81212251>] do_rmdir+0x1f1/0x220
[<ffffffff810ad24c>] ? task_work_run+0xac/0xf0
[<ffffffff81213456>] SyS_rmdir+0x16/0x20
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 51 00 4c 89 e6 48 89 df e8 8b d3 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 66 66 66 90 fb 66 66 90 <66> 66 90 48 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 

NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [runc:46890]
Modules linked in: binfmt_misc tcp_lp fuse xt_nat veth vxlan ip6_udp_tunnel udp_tunnel xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter nfsd auth_rpcgss nfs_acl lockd grace nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack loop ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod sunrpc ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt iTCO_vendor_support vfat fat i40iw skx_edac ib_core intel_powerclamp coretemp intel_rapl iosf_mbi
kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel joydev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif pcspkr ses enclosure mei_me scsi_transport_sas sg mei shpchp lpc_ich i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter sch_fq_codel ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul syscopyarea crct10dif_common sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm drm i40e ahci libahci libata megaraid_sas ptp pps_core i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 26 PID: 46890 Comm: runc Kdump: loaded Tainted: G   M           ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: Inspur NF8480M5/IIMS, BIOS 4.1.11 12/26/2019
task: ffff8a35862ceeb0 ti: ffff8a372ab84000 task.ti: ffff8a372ab84000
RIP: 0010:[<ffffffff8a9a77f6>]  [<ffffffff8a9a77f6>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff8a372ab87cf0  EFLAGS: 00000246
RAX: 000000000001c104 RBX: ffff8a343ffda000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8a343ffda540
RBP: ffff8a372ab87d20 R08: fffffffffffffff6 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8a32bf295b80
R13: ffffc75a0b4ab240 R14: 000000000000000e R15: 00000000006cffe5
FS:  00007ff510094740(0000) GS:ffff8a32bf280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f546545b8d0 CR3: 000000b025228000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff8a9a335e>] ? lru_cache_add+0xe/0x10
[<ffffffff8aa0cfcd>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff8aa0d3d4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8a91c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffff8a91c482>] cgroup_rmdir+0x22/0x40
[<ffffffff8aa295ac>] vfs_rmdir+0xdc/0x150
[<ffffffff8aa2cb31>] do_rmdir+0x1f1/0x220
[<ffffffff8aa20d84>] ? SYSC_newfstatat+0x24/0x60
[<ffffffff8aa2dd95>] SyS_unlinkat+0x25/0x40
[<ffffffff8af1f7d5>] system_call_fastpath+0x1c/0x21
Code: 89 e6 48 89 df e8 0b 51 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [slurmstepd:157352]
Modules linked in: 8021q garp mrp stp llc nfsv3 nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core dm_mirror dm_region_hash dm_log dm_mod skx_edac intel_powerclamp coretemp iTCO_wdt intel_rapl iTCO_vendor_support iosf_mbi kvm ast i2c_algo_bit ttm irqbypass crc32_pclmul ghash_clmulni_intel drm_kms_helper ixgbe syscopyarea aesni_intel sysfillrect sysimgblt fb_sys_fops lrw drm ptp gf128mul glue_helper i2c_i801 pps_core ablk_helper mdio mei_me cryptd dca joydev mei pcspkr i2c_core sg lpc_ich shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace
sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci libata
CPU: 2 PID: 157352 Comm: slurmstepd Kdump: loaded Not tainted 3.10.0-862.14.4.el7.x86_64 #1
Hardware name: Advanced Clustering S2600BPB/S2600BPB, BIOS SE5C620.86B.02.01.0008.C1.031920191559 03/19/2019
task: ffffa1163a7b2f70 ti: ffffa1162f6e4000 task.ti: ffffa1162f6e4000
RIP: 0010:[<ffffffff87daa556>]  [<ffffffff87daa556>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffffa1162f6e7cf0  EFLAGS: 00000246
RAX: 000000000001c144 RBX: ffffa0b83fdce400 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffffa116fffd0540
RBP: ffffa1162f6e7d20 R08: fffffffffffffffc R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffa0f6ccbe4000
R13: ffffe657dc8fef80 R14: 000000000000000e R15: 000000003aeca886
FS:  00002b4f74aeefc0(0000) GS:ffffa0e63da80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002ac33b4467c0 CR3: 0000005f7638a000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff87da60be>] ? lru_cache_add+0xe/0x10
[<ffffffff87e1144d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff87e11854>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff87d1f4aa>] cgroup_destroy_locked+0xea/0x370
[<ffffffff87d1f752>] cgroup_rmdir+0x22/0x40
[<ffffffff87e2db0c>] vfs_rmdir+0xdc/0x150
[<ffffffff87e31091>] do_rmdir+0x1f1/0x220
[<ffffffff87e322c6>] SyS_rmdir+0x16/0x20
[<ffffffff8832579b>] system_call_fastpath+0x22/0x27
Code: 89 e6 48 89 df e8 2b 68 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [slurmstepd:399371]
Modules linked in: squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod dcdbas iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel xfs aesni_intel lrw libcrc32c gf128mul glue_helper ablk_helper cryptd pcspkr sg mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm cdc_ether usbnet mii i2c_i801 lpc_ich nfit shpchp wmi libnvdimm acpi_power_meter acpi_pad rpcrdma ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm
ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables sd_mod crc_t10dif crct10dif_generic hfi1(OE) ipmi_devintf rdmavt(OE) igb ahci ib_core crct10dif_pclmul dca crct10dif_common libahci i2c_algo_bit ptp crc32c_intel megaraid_sas(OE) i2c_core pps_core libata ipmi_si ipmi_msghandler fjes
CPU: 1 PID: 399371 Comm: slurmstepd Tainted: G        W  OEL ------------ T 3.10.0-514.10.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R940/0V0267, BIOS 2.6.4 04/09/2020
task: ffff89295ef5de20 ti: ffff883395478000 task.ti: ffff883395478000
RIP: 0010:[<ffffffff811947a8>]  [<ffffffff811947a8>] isolate_lru_page+0x88/0x1f0
RSP: 0018:ffff88339547bd20  EFLAGS: 00000296
RAX: 000000000001a34c RBX: ffff8a403ffda000 RCX: ffffffffffffff83
RDX: fffffffffffffff6 RSI: 0000000000000004 RDI: ffff8a403ffda000
RBP: ffff88339547bd50 R08: fffffffffffffff7 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff891ebf011a20
R13: ffffea086b72d040 R14: 000000000000000e R15: 00000000bff6b694
FS:  00002b75cb20c5c0(0000) GS:ffff891ebf000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001351760 CR3: 00000175c5779000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffffffffffff ffffea08bad7d800 ffff8a3d17a6a430 ffff8a403ffda540
ffff89e0b1bdf600 ffffea08bad7d820 ffff88339547bdd0 ffffffff811f124d
0000000200000006 ffff886513f86100 0000000000000006 0000000000001000
Call Trace:
[<ffffffff811f124d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a278>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f315>] do_rmdir+0x1a5/0x200
[<ffffffff8168b5b0>] ? __schedule+0x3b0/0x990
[<ffffffff812104c6>] SyS_rmdir+0x16/0x20
[<ffffffff81696b09>] system_call_fastpath+0x16/0x1b
Code: 97 4f 00 4c 89 e6 48 89 df e8 55 c3 05 00 48 8b 13 49 89 c7 83 e2 20 75 6a 0f 1f 44 00 00 66 41 83 45 00 02 fb 66 0f 1f 44 00 00 <48> 83 c4 08 44 89 f0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 

NMI watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [runc:40441]
Modules linked in: fuse xt_nat veth vxlan ip6_udp_tunnel udp_tunnel xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) nfsd auth_rpcgss nfs_acl lockd grace nls_utf8 isofs loop xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcrdma sunrpc ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm iTCO_wdt iTCO_vendor_support vfat fat skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul
ghash_clmulni_intel aesni_intel lrw gf128mul i40iw glue_helper ablk_helper cryptd ib_core pcspkr ses enclosure scsi_transport_sas ipmi_ssif sg mei_me mei shpchp lpc_ich i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter sch_fq_codel ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci i40e drm libahci libata megaraid_sas ptp pps_core i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 25 PID: 40441 Comm: runc Kdump: loaded Tainted: G               ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: Inspur NF8480M5/IIMS, BIOS 4.1.11 12/26/2019
task: ffff9bf223530000 ti: ffff9ba565230000 task.ti: ffff9ba565230000
RIP: 0010:[<ffffffffa69a77f6>]  [<ffffffffa69a77f6>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff9ba565233cf0  EFLAGS: 00000246
RAX: 000000000001c16d RBX: ffff9c037ffda000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff9c037ffda540
RBP: ffff9ba565233d20 R08: fffffffffffffff8 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff9ba1ff655b80
R13: fffffd2dfa1dd480 R14: 000000000000000e R15: 0000000009de8dfd
FS:  00007f9d65d4e740(0000) GS:ffff9ba1ff640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0001df008 CR3: 00000063d8478000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa69a335e>] ? lru_cache_add+0xe/0x10
[<ffffffffa6a0cfcd>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffffa6a0d3d4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa691c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa691c482>] cgroup_rmdir+0x22/0x40
[<ffffffffa6a295ac>] vfs_rmdir+0xdc/0x150
[<ffffffffa6a2cb31>] do_rmdir+0x1f1/0x220
[<ffffffffa6f1a5bc>] ? __do_page_fault+0x1bc/0x4f0
[<ffffffffa6a2dd95>] SyS_unlinkat+0x25/0x40
[<ffffffffa6f1f7d5>] system_call_fastpath+0x1c/0x21
Code: 89 e6 48 89 df e8 0b 51 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [systemd:1]
Modules linked in: fuse vtsspp(OE) sep5(OE) socperf3(OE) pax(OE) nfsv3 nfs fscache nf_log_ipv4 nf_log_common xt_LOG xt_set xt_multiport xt_recent xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_set_hash_ip ip_set_hash_net ip_set nfnetlink dm_mirror dm_region_hash dm_log dm_mod opa_vnic ib_ipoib vfat fat zfs(POE) xfs libcrc32c zunicode(POE) zavl(POE) icp(POE) iTCO_wdt iTCO_vendor_support zcommon(POE) znvpair(POE) osst spl(OE) raid0 skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd hfi1 pcspkr st ch rdmavt joydev sg lpc_ich mei_me i2c_i801 mei wmi ipmi_si ipmi_devintf ipmi_msghandler
acpi_pad acpi_power_meter acpi_cpufreq rpcrdma ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm i40iw ib_core sch_fq_codel binfmt_misc nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables sd_mod crc_t10dif crct10dif_generic btrfs xor raid6_pq ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul ttm crct10dif_common crc32c_intel i40e mpt2sas ahci mpt3sas drm libahci libata raid_class nvme scsi_transport_sas ptp pps_core nvme_core drm_panel_orientation_quirks nfit libnvdimm
CPU: 19 PID: 1 Comm: systemd Kdump: loaded Tainted: P           OEL ------------   3.10.0-957.el7.x86_64 #1
Hardware name: Supermicro SSG-6019P-ACR12L/X11DDW-NT, BIOS 2.1 09/14/2018
task: ffff9e94ee8c8000 ti: ffff9e94ee8c4000 task.ti: ffff9e94ee8c4000
RIP: 0010:[<ffffffff8edca026>]  [<ffffffff8edca026>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff9e94ee8c7cf0  EFLAGS: 00000246
RAX: 000000000001f7b4 RBX: ffff9e957fdce000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff9f53fffd9540
RBP: ffff9e94ee8c7d20 R08: fffffffffffffff3 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff9ef27ae77000
R13: ffffe5d27053de40 R14: 000000000000000e R15: 00000000506c469e
FS:  00007f7a161ce940(0000) GS:ffff9f527bfc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555f996ad510 CR3: 000000606c954000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff8edc588e>] ? lru_cache_add+0xe/0x10
[<ffffffff8ee32eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff8ee332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8ed2573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffff8ed259e2>] cgroup_rmdir+0x22/0x40
[<ffffffff8ee4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffff8ee530d1>] do_rmdir+0x1f1/0x220
[<ffffffff8ee4363e>] ? ____fput+0xe/0x10
[<ffffffff8ecbe7a0>] ? task_work_run+0xc0/0xe0
[<ffffffff8ee54306>] SyS_rmdir+0x16/0x20
[<ffffffff8f374ddb>] system_call_fastpath+0x22/0x27
Code: 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [slurmstepd:157074]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log dcdbas skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e
ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci libahci megaraid_sas tg3(OE) ptp libata pps_core [last unloaded: ipmi_msghandler]
CPU: 0 PID: 157074 Comm: slurmstepd Tainted: G           OE  ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018
task: ffff8858077a2f70 ti: ffff8825c2ba4000 task.ti: ffff8825c2ba4000
RIP: 0010:[<ffffffff81195889>]  [<ffffffff81195889>] isolate_lru_page+0x9/0x1c0
RSP: 0018:ffff8825c2ba7d50  EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffffea0123162ac0 RCX: 0000000000000001
RDX: 0000000000000001 RSI: ffffea00cf87c71c RDI: ffffea00cf87c700
RBP: ffff8825c2ba7d50 R08: ffff88603f906000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590
R13: 0000000000000000 R14: ffffffff81190eac R15: ffff8825c2ba7cf8
FS:  00002aaaaaafd180(0000) GS:ffff882f7da00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffff7ff5000 CR3: 000000050be11000 CR4: 00000000003407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8825c2ba7dd0 ffffffff811f336d 0000000200000001 ffff885b15a89100
0000000000000001 0000000000001000 ffff885b15a89040 0000000000000001
0000000100000003 ffff885b15a89000 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f336d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 89 14 24 4c 89 ea 41 ff d2 4d 8b 17 4d 85 d2 75 d1 e9 76 ff ff ff 66 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 <41> 57 41 56 41 55 41 54 53 48 89 fb 48 83 ec 08 48 8b 07 a8 20 

NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [slurmstepd:82901]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi joydev iTCO_wdt iTCO_vendor_support dell_smbios dcdbas kvm_intel sparse_keymap kvm irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt aesni_intel lrw fb_sys_fops gf128mul glue_helper ablk_helper cryptd drm pcspkr mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp
llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci megaraid_sas tg3(OE) libahci ptp pps_core libata [last unloaded: tracedev]
CPU: 10 PID: 82901 Comm: slurmstepd Tainted: G    B      OE  ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.5.4 01/14/2020
task: ffff8828dd594f10 ti: ffff8830fa1e0000 task.ti: ffff8830fa1e0000
RIP: 0010:[<ffffffff81195906>]  [<ffffffff81195906>] isolate_lru_page+0x86/0x1c0
RSP: 0000:ffff8830fa1e3d20  EFLAGS: 00000246
RAX: 000000000001a285 RBX: ffff88017fc8e800 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff88603ffd7540
RBP: ffff8830fa1e3d50 R08: fffffffffffffff5 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000001 R12: ffff881dccccd000
R13: ffffea01566a9d40 R14: 000000000000000e R15: 00000000c772682a
FS:  00002aaaaaafd180(0000) GS:ffff882f7db40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aacd726f000 CR3: 0000001235248000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff811914ee ffffea00fb19a8c0 ffff8834aa176440 ffff88603ffd7540
ffff8830b14e6a30 ffffea00fb19a8e0 ffff8830fa1e3dd0 ffffffff811f336d
0000000200000001 ffff881dccccd100 0000000000000001 0000000000001000
Call Trace:
[<ffffffff811914ee>] ? lru_cache_add+0xe/0x10
[<ffffffff811f336d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 9b d3 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [systemd:1]
Modules linked in: binfmt_misc ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr nfit ppdev vmxnet3 sg vmw_balloon libnvdimm parport_pc parport shpchp vmw_vmci i2c_piix4 ip_tables
xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix scsi_transport_spi mptscsih mptbase drm crct10dif_pclmul crct10dif_common libata e1000 crc32c_intel serio_raw i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 1 Comm: systemd Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.14.4.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff8fe6f4ba8000 ti: ffff8fe6f4bb0000 task.ti: ffff8fe6f4bb0000
RIP: 0010:[<ffffffff8c7aa556>]  [<ffffffff8c7aa556>] isolate_lru_page+0x86/0x1c0
RSP: 0000:ffff8fe6f4bb3cf0  EFLAGS: 00000246
RAX: 000000000001c0db RBX: ffff8fedbffda000 RCX: ffffffffffffffca
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8fedbffda540
RBP: ffff8fe6f4bb3d20 R08: 0000000000000003 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8fedbfd15bc0
R13: fffff520cd095980 R14: 000000000000000e R15: 00000000ef961e13
FS:  00007f5657251940(0000) GS:ffff8fedbfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000035f9a4001000 CR3: 00000005f0198000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8c7a60be>] ? lru_cache_add+0xe/0x10
[<ffffffff8c81144d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff8c811854>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8c71f4aa>] cgroup_destroy_locked+0xea/0x370
[<ffffffff8c71f752>] cgroup_rmdir+0x22/0x40
[<ffffffff8c82db0c>] vfs_rmdir+0xdc/0x150
[<ffffffff8c831091>] do_rmdir+0x1f1/0x220
[<ffffffff8c82175e>] ? ____fput+0xe/0x10
[<ffffffff8c6bab90>] ? task_work_run+0xc0/0xe0
[<ffffffff8c8322c6>] SyS_rmdir+0x16/0x20
[<ffffffff8cd2579b>] system_call_fastpath+0x22/0x27
Code: 89 e6 48 89 df e8 2b 68 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [systemd:1]
Modules linked in: veth xt_nat xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) overlay(T) sunrpc snd_seq_midi snd_seq_midi_event coretemp iosf_mbi snd_ens1371 crc32_pclmul ghash_clmulni_intel snd_rawmidi snd_ac97_codec aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ac97_bus snd_seq ppdev snd_seq_device snd_pcm sg snd_timer snd vmw_balloon joydev soundcore pcspkr parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic
pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw e1000 vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 1 Comm: systemd Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffffa0553bb58000 ti: ffffa0553bbd0000 task.ti: ffffa0553bbd0000
RIP: 0010:[<ffffffffa05ca026>]  [<ffffffffa05ca026>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffffa0553bbd3cf0  EFLAGS: 00000246
RAX: 000000000001f74a RBX: ffffa0567ffda000 RCX: ffffffffffffffd6
RDX: 0000000000000000 RSI: 0000000000000002 RDI: ffffa0567ffda540
RBP: ffffa0553bbd3d20 R08: fffffffffffffffc R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffa05673697c80
R13: ffffe9eac40cf980 R14: 000000000000000e R15: 00000000f6f20b79
FS:  00007f921210f940(0000) GS:ffffa05673680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd58d235918 CR3: 00000002af5a0000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa05c588e>] ? lru_cache_add+0xe/0x10
[<ffffffffa0632eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffffa06332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa052573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa05259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffa064dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffa06530d1>] do_rmdir+0x1f1/0x220
[<ffffffffa064363e>] ? ____fput+0xe/0x10
[<ffffffffa04be7a0>] ? task_work_run+0xc0/0xe0
[<ffffffffa0654306>] SyS_rmdir+0x16/0x20
[<ffffffffa0b74ddb>] system_call_fastpath+0x22/0x27
Code: 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [slurmstepd:205732]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp iTCO_wdt intel_rapl iTCO_vendor_support iosf_mbi dcdbas kvm_intel kvm irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables fuse btrfs raid6_pq xor msdos bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter
bridge stp llc xfs isci libsas qla3xxx hpsa sg e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix sd_mod crc_t10dif mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun crct10dif_pclmul crct10dif_common i40e(OE) ahci tg3(OE) megaraid_sas ptp libahci pps_core libata [last unloaded: ipmi_msghandler]
CPU: 25 PID: 205732 Comm: slurmstepd Tainted: G           OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018
task: ffff884c01656eb0 ti: ffff8840fc6a8000 task.ti: ffff8840fc6a8000
RIP: 0010:[<ffffffff81195906>]  [<ffffffff81195906>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff8840fc6abd20  EFLAGS: 00000246
RAX: 000000000001a284 RBX: ffff88603ffd7000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff88603ffd7540
RBP: ffff8840fc6abd50 R08: fffffffffffffff9 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff885f7c111ae0
R13: ffffea01063f6680 R14: 000000000000000e R15: 0000000008959119
FS:  00002aaaaaafd180(0000) GS:ffff885f7c100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaaabe2e1a0 CR3: 00000036fbde2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff811914ee ffffea011f63f080 ffff88462fb5f430 ffff88603ffd7540
ffff8830a860fc20 ffffea011f63f0a0 ffff8840fc6abdd0 ffffffff811f336d
0000000200000001 ffff88446f069500 0000000000000001 0000000000001000
Call Trace:
[<ffffffff811914ee>] ? lru_cache_add+0xe/0x10
[<ffffffff811f336d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff810c13e6>] ? __cond_resched+0x26/0x30
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 9b d3 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#16 stuck for 23s! [systemd:1]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs 8021q garp mrp stp fscache llc ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd xfs libcrc32c pcspkr joydev i2c_i801 sg lpc_ich shpchp mei_me mei wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad acpi_cpufreq rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ast
drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect crct10dif_common sysimgblt crc32c_intel fb_sys_fops ttm drm igb ahci libahci ptp libata nvme pps_core dca nvme_core i2c_algo_bit i2c_core
CPU: 16 PID: 1 Comm: systemd Not tainted 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Supermicro SYS-1029U-TR4/X11DPU, BIOS 3.3a 07/21/2020
task: ffff8980fb138000 ti: ffff8801bb19c000 task.ti: ffff8801bb19c000
RIP: 0010:[<ffffffff81195936>]  [<ffffffff81195936>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff8801bb19fd20  EFLAGS: 00000246
RAX: 000000000001a21c RBX: ffff8803ffc8e400 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff89803ffda540
RBP: ffff8801bb19fd50 R08: fffffffffffffff5 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff897a329ef800
R13: ffffea0136061040 R14: 000000000000000e R15: 000000005df48a25
FS:  00007f292785d940(0000) GS:ffff8afa3bf80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f98e3e5e2dc CR3: 000002fa350c7000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff8119151e ffffea0136061200 ffff88e0c6a5a430 ffff89803ffda540
ffff8803b2818480 ffffea0136061220 ffff8801bb19fdd0 ffffffff811f32dd
0000000200000000 ffff897a329ef900 0000000000000000 0000000000001000
Call Trace:
[<ffffffff8119151e>] ? lru_cache_add+0xe/0x10
[<ffffffff811f32dd>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff810ad25c>] ? task_work_run+0xac/0xf0
[<ffffffff81213446>] SyS_rmdir+0x16/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 db d2 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#241 stuck for 23s! [slurmstepd:175013]
Modules linked in: iscsi_target_mod target_core_mod macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ip_vs nf_conntrack libcrc32c binfmt_misc vfat fat uas usb_storage tcp_lp fuse nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache bonding sunrpc dm_mirror dm_region_hash dm_log dm_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg ipmi_si ipmi_devintf ipmi_msghandler pinctrl_amd i2c_designware_platform i2c_designware_core acpi_cpufreq i2c_piix4 sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci
drm_panel_orientation_quirks libata megaraid_sas ixgbe igb(OE) mdio ptp pps_core i2c_algo_bit dca nfit libnvdimm [last unloaded: amifldrv_mod]
CPU: 241 PID: 175013 Comm: slurmstepd Kdump: loaded Tainted: P        W  OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: N/A 2U Rack Server/65N32-US, BIOS SZZQ042030 07/27/2020
task: ffff9f8cf56ed140 ti: ffff9f8cf5950000 task.ti: ffff9f8cf5950000
RIP: 0010:[<ffffffffba9ca023>]  [<ffffffffba9ca023>] isolate_lru_page+0x83/0x1c0
RSP: 0018:ffff9f8cf5953cf0  EFLAGS: 00000246
RAX: 000000000001f74d RBX: ffff9fcd0feda000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff9fcd0feda540
RBP: ffff9f8cf5953d20 R08: fffffffffffffffc R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffa1ccbf857c80
R13: ffffe9e66bac1540 R14: 000000000000000e R15: 0000000093b368f4
FS:  00002ae13f5a7fc0(0000) GS:ffffa1ccbf840000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002ae143b87720 CR3: 0000002445e26000 CR4: 0000000000340fe0
Call Trace:
[<ffffffffba9c588e>] ? lru_cache_add+0xe/0x10
[<ffffffffbaa32eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffffbaa332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffba92573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffba9259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffbaa4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffbaa530d1>] do_rmdir+0x1f1/0x220
[<ffffffffbaa54306>] SyS_rmdir+0x16/0x20
[<ffffffffbaf74ddb>] system_call_fastpath+0x22/0x27
Code: 5a 00 4c 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 66 66 66 90 fb 66 66 90 <66> 66 90 48 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 

NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [slurmstepd:41333]
Modules linked in: fuse rpcsec_gss_krb5 nfsv4 dns_resolver 8021q garp mrp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 nfs fscache dm_mirror dm_region_hash dm_log dm_mod sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel mxm_wmi iTCO_wdt iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr mei_me i2c_i801 sg mei ioatdma lpc_ich shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter acpi_pad acpi_cpufreq rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi
ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm nfsd ib_cm iw_cm auth_rpcgss ib_core nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt igb fb_sys_fops ttm crct10dif_pclmul ahci ptp crct10dif_common drm crc32c_intel libahci pps_core libata dca i2c_algo_bit i2c_core
CPU: 5 PID: 41333 Comm: slurmstepd Tainted: G             L ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Advanced Clustering S2600TPR/S2600TPR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016
task: ffff880f89ec4f10 ti: ffff882d3992c000 task.ti: ffff882d3992c000
RIP: 0010:[<ffffffff816aba59>]  [<ffffffff816aba59>] _raw_spin_lock_irq+0x9/0x30
RSP: 0018:ffff882d3992fd10  EFLAGS: 00000286
RAX: 0000000000000001 RBX: ffff881ffe751ae0 RCX: 0000000000000001
RDX: 0000000000000001 RSI: ffffea008d92085c RDI: ffff88407ffd7540
RBP: ffff882d3992fd10 R08: ffff8810ec9f5c00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: 000000000000000e
R13: 00000000943a5929 R14: ffff882d3992fc68 R15: ffff882d3992fc68
FS:  00002b702a3fed00(0000) GS:ffff881ffe740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000d87980 CR3: 000000319f9ad000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff882d3992fd50 ffffffff811958da ffffffff811914ee ffffea008d920840
ffff88208f3ca430 ffff88407ffd7540 ffff8820fd2c8210 ffffea008d920860
ffff882d3992fdd0 ffffffff811f336d 0000000200000001 ffff88300f654d00
Call Trace:
[<ffffffff811958da>] isolate_lru_page+0x5a/0x1c0
[<ffffffff811914ee>] ? lru_cache_add+0xe/0x10
[<ffffffff811f336d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816a8fad>] ? __schedule+0x39d/0x8b0
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 1f 00 0f 1f 44 00 00 55 48 89 e5 f0 ff 47 04 48 8b 7d 08 be 00 02 00 00 e8 f5 44 9e ff 5d c3 0f 1f 00 0f 1f 44 00 00 55 48 89 e5 <fa> 66 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 85 c0 75 

NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [slurmstepd:153803]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log dcdbas skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul crc32c_intel mgag200 ghash_clmulni_intel ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci
libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) megaraid_sas ahci tg3(OE) libahci ptp pps_core libata [last unloaded: ipmi_msghandler]
CPU: 11 PID: 153803 Comm: slurmstepd Tainted: G           OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018
task: ffff88444a28eeb0 ti: ffff8831f1d5c000 task.ti: ffff8831f1d5c000
RIP: 0010:[<ffffffff81195906>]  [<ffffffff81195906>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff8831f1d5fd20  EFLAGS: 00000246
RAX: 000000000001a285 RBX: ffff88603ffd7000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff88603ffd7540
RBP: ffff8831f1d5fd50 R08: ffffffffffffffff R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff885f7bf51ae0
R13: ffffea00ea7e9e80 R14: 000000000000000e R15: 00000000af2528a0
FS:  00002aaaaaafd180(0000) GS:ffff885f7bf40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffff4e855f0 CR3: 0000000584e11000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff811914ee ffffea0116359d00 ffff88370c207440 ffff88603ffd7540
ffff8830aa956740 ffffea0116359d20 ffff8831f1d5fdd0 ffffffff811f336d
0000000200000001 ffff8801e7c17500 0000000000000001 0000000000001000
Call Trace:
[<ffffffff811914ee>] ? lru_cache_add+0xe/0x10
[<ffffffff811f336d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 9b d3 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#45 stuck for 22s! [slurmstepd:441903]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) squashfs loop overlay(T) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl dell_smbios iosf_mbi dcdbas sparse_keymap kvm_intel kvm irqbypass crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt ghash_clmulni_intel fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat
br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) megaraid_sas tg3(OE) ptp ahci pps_core libahci libata [last unloaded: tracedev]
CPU: 45 PID: 441903 Comm: slurmstepd Tainted: G    B   W  OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.5.4 01/14/2020
task: ffff884b5a13cf10 ti: ffff883143f84000 task.ti: ffff883143f84000
RIP: 0010:[<ffffffff81195906>]  [<ffffffff81195906>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff883143f87d20  EFLAGS: 00000246
RAX: 000000000001a285 RBX: ffff88603ffd7000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff88603ffd7540
RBP: ffff883143f87d50 R08: fffffffffffffff8 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff885f7c391ae0
R13: ffffea0143217700 R14: 000000000000000e R15: 0000000022306969
FS:  00002aaaaaafd180(0000) GS:ffff885f7c380000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaab400c09c CR3: 000000108c665000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff811914ee ffffea0117b66100 ffff883128d5e440 ffff88603ffd7540
ffff8830aa759840 ffffea0117b66120 ffff883143f87dd0 ffffffff811f336d
0000000200000001 ffff8801b3e64100 0000000000000001 0000000000001000
Call Trace:
[<ffffffff811914ee>] ? lru_cache_add+0xe/0x10
[<ffffffff811f336d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 9b d3 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [docker-runc:28941]
Modules linked in: ip6t_REJECT nf_reject_ipv6 xt_owner xt_REDIRECT nf_nat_redirect vxlan ip6_udp_tunnel udp_tunnel xt_statistic ipt_REJECT nf_reject_ipv4 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack br_netfilter overlay(T) ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bonding bridge stp llc iTCO_wdt iTCO_vendor_support sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev i2c_i801 lpc_ich mei_me sg mei ioatdma shpchp wmi ipmi_ssif
ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm igb ahci crct10dif_pclmul crct10dif_common libahci crc32c_intel megaraid_sas libata ptp pps_core dca i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 13 PID: 28941 Comm: docker-runc Tainted: G        W      ------------ T 3.10.0-693.11.6.el7.x86_64 #1
Hardware name: Supermicro SYS-6028R-TR/X10DRi, BIOS 1.1 04/14/2015
task: ffff8815f5a1eeb0 ti: ffff88174d3e8000 task.ti: ffff88174d3e8000
RIP: 0010:[<ffffffff81197626>]  [<ffffffff81197626>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff88174d3ebcf0  EFLAGS: 00000246
RAX: 000000000001bfc4 RBX: ffff88407ffd7000 RCX: ffffffffffffff92
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff88407ffd7540
RBP: ffff88174d3ebd20 R08: fffffffffffffff4 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff881fffbd5ae0
R13: ffffea00e62a00c0 R14: 000000000000000e R15: 00000000a30eb280
FS:  00007fb5f167f740(0000) GS:ffff881fffbc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055c7ddc90f60 CR3: 000000169d62e000 CR4: 00000000001607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8119320e>] ? lru_cache_add+0xe/0x10
[<ffffffff811f52ad>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f56b4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110f467>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110f712>] cgroup_rmdir+0x22/0x40
[<ffffffff812114f6>] vfs_rmdir+0xd6/0x150
[<ffffffff81214191>] do_rmdir+0x1f1/0x220
[<ffffffff8120878e>] ? SYSC_newstat+0x2e/0x60
[<ffffffff816b3801>] ? __do_page_fault+0x171/0x450
[<ffffffff812153c5>] SyS_unlinkat+0x25/0x40
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 bb d5 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [slurmstepd:37609]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel
lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr sg sb_edac edac_core hpwdt hpilo ipmi_ssif lpc_ich ioatdma shpchp ipmi_si acpi_power_meter pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c mlx4_en sd_mod crc_t10dif crct10dif_generic nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) nvidia(POE) crct10dif_pclmul crct10dif_common crc32c_intel ipmi_msghandler mgag200 serio_raw drm_kms_helper mlx4_core syscopyarea sysfillrect sysimgblt ahci fb_sys_fops ttm igb hpsa libahci devlink drm ptp libata pps_core mpt2sas dca i2c_algo_bit i2c_core raid_class scsi_transport_sas wmi fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 37609 Comm: slurmstepd Tainted: P           OE  ------------   3.10.0-514.21.2.el7.x86_64 #1
Hardware name: HP ProLiant SL270s Gen8 SE/, BIOS P75 09/18/2013
task: ffff8818dcbf0fb0 ti: ffff881fb0648000 task.ti: ffff881fb0648000
RIP: 0010:[<ffffffff81194c98>]  [<ffffffff81194c98>] isolate_lru_page+0x88/0x1f0
RSP: 0018:ffff881fb064bd20  EFLAGS: 00000286
RAX: 000000000001a144 RBX: ffff88403ffd9000 RCX: ffffffffffffff83
RDX: fffffffffffffff6 RSI: 0000000000000004 RDI: ffff88403ffd9000
RBP: ffff881fb064bd50 R08: fffffffffffffff7 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff881fbf611a20
R13: ffffea008b5c74c0 R14: 000000000000000e R15: 00000000503b44ba
FS:  00007fcc3f74b780(0000) GS:ffff881fbf600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4be71d53e4 CR3: 0000001fb0345000 CR4: 00000000001407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffffffffffff ffffea00d6eecb40 ffff883fb4aae430 ffff88403ffd9540
ffff8820aa43b2d0 ffffea00d6eecb60 ffff881fb064bdd0 ffffffff811f188d
0000000200000001 ffff883f9b685500 0000000000000001 0000000000001000
Call Trace:
[<ffffffff811f188d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f1c94>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d6e7>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d982>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a8c8>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f965>] do_rmdir+0x1a5/0x200
[<ffffffff8168c130>] ? __schedule+0x3b0/0x990
[<ffffffff81210b16>] SyS_rmdir+0x16/0x20
[<ffffffff81697749>] system_call_fastpath+0x16/0x1b
Code: 9e 4f 00 4c 89 e6 48 89 df e8 a5 c4 05 00 48 8b 13 49 89 c7 83 e2 20 75 6a 0f 1f 44 00 00 66 41 83 45 00 02 fb 66 0f 1f 44 00 00 <48> 83 c4 08 44 89 f0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 

NMI watchdog: BUG: soft lockup - CPU#35 stuck for 23s! [slurmstepd:180490]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm mgag200 ttm irqbypass drm_kms_helper crc32_pclmul crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops drm ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sr_mod cdrom mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge
stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) megaraid_sas aacraid aic79xx ata_piix sd_mod crc_t10dif mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil tg3(OE) e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sg crct10dif_pclmul crct10dif_common igb(OE) i2c_algo_bit i2c_core ahci libahci dca i40e(OE) ptp libata pps_core [last unloaded: ipmi_msghandler]
CPU: 35 PID: 180490 Comm: slurmstepd Tainted: G        W  OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: FUJITSU PRIMERGY RX2540 M5/D3384-B1, BIOS V5.0.0.14 R1.15.0 for D3384-B1x                    10/17/2019
task: ffff8844244a9fa0 ti: ffff880143670000 task.ti: ffff880143670000
RIP: 0010:[<ffffffff81195906>]  [<ffffffff81195906>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff880143673d20  EFLAGS: 00000246
RAX: 000000000001a355 RBX: ffff88603ffd7000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff88603ffd7540
RBP: ffff880143673d50 R08: fffffffffffffff7 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff885fdea11ae0
R13: ffffea01371bb640 R14: 000000000000000e R15: 0000000093a715dc
FS:  00002aaaaaafd180(0000) GS:ffff885fdea00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaab47934c0 CR3: 00000001250c9000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff811914ee ffffea0153730c40 ffff88485a40f440 ffff88603ffd7540
ffff8848b364c310 ffffea0153730c60 ffff880143673dd0 ffffffff811f336d
0000000200000003 ffff8830d8725100 0000000000000003 0000000000001000
Call Trace:
[<ffffffff811914ee>] ? lru_cache_add+0xe/0x10
[<ffffffff811f336d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 9b d3 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [slurmstepd:320837]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core dcdbas joydev intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass iTCO_wdt crc32_pclmul iTCO_vendor_support crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc
xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci tg3(OE) megaraid_sas ptp libahci pps_core libata [last unloaded: ipmi_msghandler]
CPU: 7 PID: 320837 Comm: slurmstepd Tainted: G           OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018
task: ffff881f7ff66eb0 ti: ffff88037e650000 task.ti: ffff88037e650000
RIP: 0010:[<ffffffff81195906>]  [<ffffffff81195906>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffff88037e653d20  EFLAGS: 00000246
RAX: 000000000001a285 RBX: ffff88603ffd7000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff88603ffd7540
RBP: ffff88037e653d50 R08: fffffffffffffff7 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff885f7bed1ae0
R13: ffffea0108572280 R14: 000000000000000e R15: 00000000b26ed96a
FS:  00002aaaaaafd180(0000) GS:ffff885f7bec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaab025c2a8 CR3: 00000001f9eeb000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff811914ee ffffea00c9a67240 ffff88550f49f440 ffff88603ffd7540
ffff8830bdf19c90 ffffea00c9a67260 ffff88037e653dd0 ffffffff811f336d
0000000200000001 ffff880adf311d00 0000000000000001 0000000000001000
Call Trace:
[<ffffffff811914ee>] ? lru_cache_add+0xe/0x10
[<ffffffff811f336d>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 89 e6 48 89 df e8 9b d3 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [docker-runc:2725]
Modules linked in: veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack br_netfilter bridge overlay(T) 8021q garp mrp stp llc nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6_tables iptable_filter rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm amd64_edac_mod edac_mce_amd edac_core kvm_amd ib_uverbs kvm ib_umad irqbypass crc32_pclmul sg rdma_cm ghash_clmulni_intel hpilo hpwdt aesni_intel shpchp ipmi_si lrw gf128mul fam15h_power glue_helper sp5100_tco ipmi_devintf ablk_helper k10temp i2c_piix4 ib_cm cryptd ipmi_msghandler iw_cm ib_core pcspkr acpi_power_meter
pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper bnx2x syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_generic pata_acpi drm ahci libahci pata_atiixp libata crct10dif_pclmul hpsa crct10dif_common crc32c_intel mdio libcrc32c serio_raw ptp i2c_core pps_core scsi_transport_sas
CPU: 25 PID: 2725 Comm: docker-runc Tainted: G             L ------------ T 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: HP ProLiant BL465c Gen8, BIOS A26 11/02/2014
task: ffff882fffa9bf40 ti: ffff882678fc0000 task.ti: ffff882678fc0000
RIP: 0010:[<ffffffff81195933>]  [<ffffffff81195933>] isolate_lru_page+0x83/0x1c0
RSP: 0018:ffff882678fc3d20  EFLAGS: 00000246
RAX: 000000000001a355 RBX: ffff88403efd8000 RCX: ffffffffffffff88
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff88403efd8540
RBP: ffff882678fc3d50 R08: fffffffffffffff7 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff88403eb11ae0
R13: ffffea00cf534800 R14: 000000000000000e R15: 000000002e5afdd2
FS:  00007f37e6e08740(0000) GS:ffff88403eb00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c4203dd000 CR3: 00000035f8321000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffff8119151e ffffea00f9971bc0 ffff883c93060440 ffff88403efd8540
ffff8830b1edc6f0 ffffea00f9971be0 ffff882678fc3dd0 ffffffff811f32dd
0000000200000003 ffff8815f5d23d00 0000000000000003 0000000000001000
Call Trace:
[<ffffffff8119151e>] ? lru_cache_add+0xe/0x10
[<ffffffff811f32dd>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff81206a04>] ? SYSC_newfstatat+0x24/0x60
[<ffffffff81213475>] SyS_unlinkat+0x25/0x40
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 51 00 4c 89 e6 48 89 df e8 db d2 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 66 66 66 90 fb 66 66 90 <66> 66 90 48 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 

NMI watchdog: BUG: soft lockup - CPU#45 stuck for 22s! [runc:88088]
Modules linked in: isofs ses enclosure nvidia_uvm(OE) veth tcp_lp vfat fat fuse nf_conntrack_netlink xt_addrtype br_netfilter mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support ext4 mbcache jbd2 sb_edac intel_powerclamp
coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) snd_hda_codec_hdmi nouveau uas pcspkr usb_storage snd_hda_intel joydev snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd mxm_wmi video soundcore ixgbe sg ptp ipmi_ssif pps_core lpc_ich mdio i2c_i801 megaraid_sas mei_me mei ioatdma dca wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic i2c_algo_bit crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci libata drm_panel_orientation_quirks
CPU: 45 PID: 88088 Comm: runc Tainted: P           OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Supermicro SYS-8028B-C0R4FT/X10QBL-4CT, BIOS 3.1 07/20/2018
task: ffffa099867130c0 ti: ffffa142c8780000 task.ti: ffffa142c8780000
RIP: 0010:[<ffffffff9d7ca026>]  [<ffffffff9d7ca026>] isolate_lru_page+0x86/0x1c0
RSP: 0018:ffffa142c8783cf0  EFLAGS: 00000246
RAX: 000000000001f74d RBX: ffffa1757ffd9000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffffa1757ffd9540
RBP: ffffa142c8783d20 R08: fffffffffffffffe R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffa1717e957c80
R13: fffff3c02df79ac0 R14: 000000000000000e R15: 00000000accb2fa6
FS:  00007fd922b43740(0000) GS:ffffa1717e940000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f315f794dc9 CR3: 000000b047d9c000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9d7c588e>] ? lru_cache_add+0xe/0x10
[<ffffffff9d832eed>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff9d8332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff9d72573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffff9d7259e2>] cgroup_rmdir+0x22/0x40
[<ffffffff9d84dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffff9d8530d1>] do_rmdir+0x1f1/0x220
[<ffffffff9d847034>] ? SYSC_newfstatat+0x24/0x60
[<ffffffff9d854335>] SyS_unlinkat+0x25/0x40
[<ffffffff9dd74ddb>] system_call_fastpath+0x22/0x27
Code: 89 e6 48 89 df e8 fb 87 06 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <48> 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 

NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [slurmstepd:54545]
Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) 8021q garp mrp stp llc ip6_tables rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi amd64_edac_mod scsi_transport_iscsi edac_mce_amd edac_core kvm_amd ib_srpt kvm target_core_mod dcdbas irqbypass crc32_pclmul ghash_clmulni_intel ib_srp scsi_transport_srp aesni_intel scsi_tgt lrw gf128mul glue_helper ablk_helper cryptd ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm sg pcspkr ib_cm ccp shpchp i2c_piix4 iw_cm ipmi_si ipmi_devintf ib_core ipmi_msghandler acpi_power_meter tpm_crb nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200
i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci crct10dif_pclmul crct10dif_common tg3 libata crc32c_intel megaraid_sas bnxt_en ptp i2c_core pps_core
CPU: 8 PID: 54545 Comm: slurmstepd Tainted: G           OE  ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R7415/07YXFK, BIOS 1.8.7 04/02/2019
task: ffff881501fb8000 ti: ffff881962218000 task.ti: ffff881962218000
RIP: 0010:[<ffffffff81195933>]  [<ffffffff81195933>] isolate_lru_page+0x83/0x1c0
RSP: 0018:ffff88196221bd20  EFLAGS: 00000246
RAX: 000000000001a21d RBX: ffff88407f1d8000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000005 RDI: ffff88407f1d8540
RBP: ffff88196221bd50 R08: fffffffffffffff6 R09: ffff883f7d8d7880
R10: 0000000000000001 R11: 0000000000000001 R12: ffff883f7ce11ae0
R13: ffffea00cd355640 R14: 000000000000000e R15: 00000000bf69cd52
FS:  00002b5b376cdfc0(0000) GS:ffff883f7ce00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f22e2024140 CR3: 0000002e5174f000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffff8119151e ffffea00cd37f100 ffff8828efb61440 ffff88407f1d8540
ffff88014ccdfc40 ffffea00cd37f120 ffff88196221bdd0 ffffffff811f32dd
0000000200000000 ffff883b13fecd00 0000000000000000 0000000000001000
Call Trace:
[<ffffffff8119151e>] ? lru_cache_add+0xe/0x10
[<ffffffff811f32dd>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff81213446>] SyS_rmdir+0x16/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 51 00 4c 89 e6 48 89 df e8 db d2 05 00 48 8b 0b 49 89 c4 ba f0 ff ff ff 83 e1 20 75 43 4c 89 ef c6 07 00 66 66 66 90 fb 66 66 90 <66> 66 90 48 83 c4 08 89 d0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 

NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [runc:23945]
Modules linked in: fuse veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 nfs fscache vfat fat dm_round_robin iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp intel_rapl iosf_mbi kvm ipmi_devintf dm_service_time irqbypass ses enclosure pcspkr scsi_transport_sas sg i2c_i801 shpchp lpc_ich ipmi_si ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad dm_multipath nfsd auth_rpcgss nfs_acl lockd binfmt_misc grace ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic 8021q garp stp llc mrp
crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ixgbe igb dca ptp pps_core mdio qla2xxx ast i2c_algo_bit megaraid_sas drm_kms_helper scsi_transport_fc syscopyarea scsi_tgt sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci libata i2c_core scsi_transport_iscsi wmi fjes uas usb_storage sunrpc dm_mirror dm_region_hash dm_log dm_mod
CPU: 23 PID: 23945 Comm: runc Tainted: G               ------------ T 3.10.0-514.26.1.el7.x86_64 #1
Hardware name: Inspur NF5280M5/YZMB-00882-102, BIOS 3.0.8 11/24/2017
task: ffff884101166dd0 ti: ffff88010c92c000 task.ti: ffff88010c92c000
RIP: 0010:[<ffffffff81194ca8>]  [<ffffffff81194ca8>] isolate_lru_page+0x88/0x1f0
RSP: 0018:ffff88010c92fd20  EFLAGS: 00000206
RAX: 000000000001a144 RBX: ffff88017fc6e800 RCX: ffffffffffffff83
RDX: 0000000000000004 RSI: 0000000000000004 RDI: ffff88806ffd5000
RBP: ffff88010c92fd50 R08: 0000000000000005 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffea01d8dbf340
R13: ffff88411d8c5800 R14: 000000000000000e R15: 000000000be771a9
FS:  00007f34cc548740(0000) GS:ffff887f6dac0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f34ccd49120 CR3: 00000047ab506000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffffffffffff ffffea01d8dbb040 ffff8842a784f430 ffff88806ffd5540
ffff8840ba0eec10 ffffea01d8dbb060 ffff88010c92fdd0 ffffffff811f19bd
0000000200000001 ffff88411d8c5900 0000000000000001 0000000000001000
Call Trace:
[<ffffffff811f19bd>] mem_cgroup_reparent_charges+0x16d/0x3c0
[<ffffffff811f1dc4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d6f7>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d992>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a9f8>] vfs_rmdir+0xa8/0x100
[<ffffffff8120fa95>] do_rmdir+0x1a5/0x200
[<ffffffff81204164>] ? SYSC_newfstatat+0x24/0x60
[<ffffffff81210c75>] SyS_unlinkat+0x25/0x40
[<ffffffff81697909>] system_call_fastpath+0x16/0x1b
Code: 9f 4f 00 4c 89 e6 48 89 df e8 c5 c5 05 00 48 8b 13 49 89 c7 83 e2 20 75 6a 0f 1f 44 00 00 66 41 83 45 00 02 fb 66 0f 1f 44 00 00 <48> 83 c4 08 44 89 f0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f