| Operating system | Count |
|---|---|
| CentOS 7 | 210 |
| Architecture | Count |
|---|---|
| x86_64 | 210 |
| Related packages | Count |
|---|---|
| kernel | 209 |
| kernel-plus | 1 |
Packages names with count less than the total count of reports are most probably not the cause of the problem.
No reports in the last 20 days.
No reports in the last 20 weeks.
NMI watchdog: BUG: soft lockup - CPU#18 stuck for 23s! [systemd:1] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache snapapi26(OE) iscsi_target_mod target_core_mod macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc 8021q garp mrp veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) overlay(T) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support dcdbas skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether usbnet mii joydev sg mei_me mei lpc_ich i2c_i801 ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad knem(OE) auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 mlx4_ib(OE) mlx4_en(OE) mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common mgag200 crc32c_intel drm_kms_helper mlx4_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx5_core(OE) drm mlxfw(OE) vfio_mdev(OE) vfio_iommu_type1 i40e ahci vfio igb mdev(OE) devlink libahci mlx_compat(OE) megaraid_sas libata drm_panel_orientation_quirks ptp dca i2c_algo_bit pps_core nfit libnvdimm [last unloaded: file_protector] CPU: 18 PID: 1 Comm: systemd Tainted: G OE ------------ T 3.10.0-957.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R640/0CRT1G, BIOS 1.2.11 10/19/2017 task: ffff8ce7d3518000 ti: ffff8ce7d3514000 task.ti: ffff8ce7d3514000 RIP: 0010:[<ffffffffadf6a0f5>] [<ffffffffadf6a0f5>] _raw_spin_unlock_irqrestore+0x15/0x20 RSP: 0018:ffff8ce7d3517d20 EFLAGS: 00000283 RAX: 0000000000000283 RBX: ffffbc915a1f4b00 RCX: 0000000000000001 RDX: 0000000000000001 RSI: 0000000000000283 RDI: 0000000000000283 RBP: ffff8ce7d3517d20 R08: ffff8d266ac35400 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffad9c4930 R13: 0000000000000000 R14: ffffffffad9c524c R15: ffff8ce7d3517cc8 FS: 00007f150db84940(0000) GS:ffff8d066d040000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f753dd49830 CR3: 0000003fee17c000 CR4: 00000000007607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: [<ffffffffada32eb9>] mem_cgroup_reparent_charges+0x139/0x3c0 [<ffffffffada332f4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffffad92573a>] cgroup_destroy_locked+0xea/0x370 [<ffffffffad9259e2>] cgroup_rmdir+0x22/0x40 [<ffffffffada4dc2c>] vfs_rmdir+0xdc/0x150 [<ffffffffada530d1>] do_rmdir+0x1f1/0x220 [<ffffffffada4363e>] ? ____fput+0xe/0x10 [<ffffffffad8be7a0>] ? task_work_run+0xc0/0xe0 [<ffffffffada54306>] SyS_rmdir+0x16/0x20 [<ffffffffadf74ddb>] system_call_fastpath+0x22/0x27 Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [systemd:1] Modules linked in: fuse binfmt_misc uas usb_storage 8021q garp mrp vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge stp llc bonding ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ucm i40iw rpcrdma sunrpc rdma_ucm ib_uverbs ib_iser rdma_cm iw_cm libiscsi ib_umad ib_ipoib scsi_transport_iscsi ib_cm iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mlx4_ib mlx5_ib ib_core pcspkr sg mei_me joydev i2c_i801 mei lpc_ich shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi nfit acpi_cpufreq libnvdimm acpi_power_meter acpi_pad ip_tables xfs libcrc32c mlx4_en sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core fb_sys_fops ttm mlx4_core crct10dif_pclmul i40e crct10dif_common ahci drm crc32c_intel libahci mlxfw megaraid_sas libata devlink ptp pps_core i2c_core dm_mirror dm_region_hash dm_log dm_mod CPU: 3 PID: 1 Comm: systemd Kdump: loaded Not tainted 3.10.0-862.6.3.el7.x86_64 #1 Hardware name: Supermicro X11DPi-N(T)/X11DPi-N, BIOS 2.0b 02/28/2018 task: ffff975027668000 ti: ffff9710e76dc000 task.ti: ffff9710e76dc000 RIP: 0010:[<ffffffffa1ba2989>] [<ffffffffa1ba2989>] put_page+0x29/0x50 RSP: 0018:ffff9710e76dfd20 EFLAGS: 00000206 RAX: 006fffff00000848 RBX: fffffe3cc11aee80 RCX: 0000000000000001 RDX: 0000000000000008 RSI: 0000000000000001 RDI: fffffe3cc11a90c0 RBP: ffff9710e76dfd20 R08: ffff978e5c76d400 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa1ba2bc0 R13: 0000000000000000 R14: ffffffffa1ba34dc R15: ffff9710e76dfcc8 FS: 00007fc75db69940(0000) GS:ffff974eff6c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055a8fd1071d8 CR3: 0000003f35ca8000 CR4: 00000000007627e0 PKRU: 55555554 Call Trace: [<ffffffffa1c0d852>] mem_cgroup_reparent_charges+0x232/0x3c0 [<ffffffffa1c0db94>] mem_cgroup_css_offline+0x84/0x140 [<ffffffffa1b1c9ca>] cgroup_destroy_locked+0xea/0x370 [<ffffffffa1b1cc72>] cgroup_rmdir+0x22/0x40 [<ffffffffa1c29d6c>] vfs_rmdir+0xdc/0x150 [<ffffffffa1c2d2f1>] do_rmdir+0x1f1/0x220 [<ffffffffa1c1d9ae>] ? ____fput+0xe/0x10 [<ffffffffa1ab8040>] ? task_work_run+0xc0/0xe0 [<ffffffffa1c2e526>] SyS_rmdir+0x16/0x20 [<ffffffffa2120795>] system_call_fastpath+0x1c/0x21 Code: 00 00 0f 1f 44 00 00 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 58 ff fe ff 5d c3 66 0f 1f 44 00 00 f6 c4 c0 75 14 f0 ff 4f 1c <0f> 94 c0 84 c0 66 90 74 e6 e8 99 ff ff ff 5d c3 0f 1f 80 00 00
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 3354s! [systemd:1] Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev coretemp crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi snd_ac97_codec aesni_intel ac97_bus snd_seq lrw gf128mul glue_helper ablk_helper cryptd snd_seq_device vmw_balloon snd_pcm pcspkr joydev snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix nfit crct10dif_pclmul crct10dif_common drm crc32c_intel libnvdimm libata mptspi serio_raw scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks fuse CPU: 0 PID: 1 Comm: systemd Kdump: loaded Not tainted 3.10.0-1160.36.2.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015 task: ffff9bdb39e30000 ti: ffff9bdb39dbc000 task.ti: ffff9bdb39dbc000 RIP: 0010:[<ffffffff8638b7b5>] [<ffffffff8638b7b5>] _raw_spin_unlock_irqrestore+0x15/0x20 RSP: 0018:ffff9bdb39dbfd18 EFLAGS: 00000246 RAX: 0000000000000246 RBX: ffff9bdb39dbfcf0 RCX: ffff9bdb39dbffd8 RDX: 0000000000000001 RSI: 0000000000000246 RDI: 0000000000000246 RBP: ffff9bdb39dbfd18 R08: ffffffff86960ec8 R09: 0000000000000002 R10: ffffffff8695e000 R11: ffff9bdac3497f00 R12: ffff9bdaf45212c0 R13: ffff9bdb3b61acc0 R14: ffff9bdaf45212c0 R15: ffff9bdaf45212c0 FS: 00007f1252acd940(0000) GS:ffff9bdb3b600000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000014c6bcc CR3: 00000000785b0000 CR4: 00000000003607f0 Call Trace: [<ffffffff85e3f39b>] mem_cgroup_reparent_charges+0x30b/0x3c0 [<ffffffff85e3f604>] mem_cgroup_css_offline+0x84/0x170 [<ffffffff85d2b33a>] cgroup_destroy_locked+0xea/0x370 [<ffffffff85d2b5e2>] cgroup_rmdir+0x22/0x40 [<ffffffff85e5bc8c>] vfs_rmdir+0xdc/0x150 [<ffffffff85e60911>] do_rmdir+0x1f1/0x220 [<ffffffff85e50a6e>] ? ____fput+0xe/0x10 [<ffffffff85cc2980>] ? task_work_run+0xc0/0xe0 [<ffffffff85e61b46>] SyS_rmdir+0x16/0x20 [<ffffffff86395f92>] system_call_fastpath+0x25/0x2a Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
NMI watchdog: BUG: soft lockup - CPU#48 stuck for 22s! [slurmstepd:84268] Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver dm_mirror dm_region_hash dm_log joydev iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ghash_clmulni_intel drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sr_mod cdrom mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) megaraid_sas aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil tg3(OE) e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci igb(OE) i2c_algo_bit libata i2c_core dca i40e(OE) ptp pps_core [last unloaded: ipmi_msghandler] CPU: 48 PID: 84268 Comm: slurmstepd Tainted: G W OE ------------ T 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: FUJITSU PRIMERGY RX2540 M5/D3384-B1, BIOS V5.0.0.14 R1.15.0 for D3384-B1x 10/17/2019 task: ffff88307508eeb0 ti: ffff883a18694000 task.ti: ffff883a18694000 RIP: 0010:[<ffffffff81190359>] [<ffffffff81190359>] put_page+0x29/0x50 RSP: 0018:ffff883a18697d50 EFLAGS: 00000202 RAX: 00efffff0000004c RBX: ffffea015b2ac000 RCX: 0000000000000001 RDX: 000000000000000b RSI: 0000000000000001 RDI: ffffea013b720500 RBP: ffff883a18697d50 R08: ffff885fdc297c00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590 R13: 0000000000000000 R14: ffffffff81190eac R15: ffff883a18697cf8 FS: 00002aaaaaafd180(0000) GS:ffff882fdf380000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002ab03ccef000 CR3: 0000001847b2e000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff883a18697dd0 ffffffff811f3432 0000000200000003 ffff885c77765d00 0000000000000003 0000000000001000 ffff885c77765c40 0000000000000001 0000000100000003 ffff885c77765c00 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f3432>] mem_cgroup_reparent_charges+0x232/0x3c0 [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff816b0325>] ? do_page_fault+0x35/0x90 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: 00 00 0f 1f 44 00 00 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 18 0c ff ff 5d c3 66 0f 1f 44 00 00 f6 c4 c0 75 14 f0 ff 4f 1c <0f> 94 c0 84 c0 66 90 74 e6 e8 99 ff ff ff 5d c3 0f 1f 80 00 00
NMI watchdog: BUG: soft lockup - CPU#63 stuck for 23s! [slurmstepd:392445] Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crc32_pclmul dell_smbios crc32c_intel dcdbas sparse_keymap ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops aesni_intel lrw gf128mul drm glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa sg e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix sd_mod crc_t10dif mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun crct10dif_pclmul crct10dif_common ahci libahci libata megaraid_sas i40e(OE) tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler] CPU: 63 PID: 392445 Comm: slurmstepd Tainted: G OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.21.1 03/07/2024 task: ffff884b0d060fd0 ti: ffff8801b1a00000 task.ti: ffff8801b1a00000 RIP: 0010:[<ffffffff81190359>] [<ffffffff81190359>] put_page+0x29/0x50 RSP: 0018:ffff8801b1a03d50 EFLAGS: 00000202 RAX: 006fffff0000006c RBX: ffffea00dd311300 RCX: ffff8801b1a03c58 RDX: 0000000000000000 RSI: 000000000000000e RDI: ffffea00dd311300 RBP: ffff8801b1a03d50 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590 R13: 0000000000000000 R14: ffffffff81190eac R15: ffff8801b1a03cf8 FS: 00002aaaaaafd180(0000) GS:ffff885f7c5c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000069c3260 CR3: 0000000bc1d8e000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff8801b1a03dd0 ffffffff811f3432 0000000200000001 ffff880f1f945900 0000000000000001 0000000000001000 ffff880f1f945840 0000000000000001 0000000100000003 ffff880f1f945800 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f3432>] mem_cgroup_reparent_charges+0x232/0x3c0 [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: 00 00 0f 1f 44 00 00 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 18 0c ff ff 5d c3 66 0f 1f 44 00 00 f6 c4 c0 75 14 f0 ff 4f 1c <0f> 94 c0 84 c0 66 90 74 e6 e8 99 ff ff ff 5d c3 0f 1f 80 00 00
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [slurmstepd:119171] Modules linked in: squashfs loop overlay(T) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache panfs(POE) 8021q garp mrp stp llc nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) nvidia(POE) mlx5_fpga_tools(OE) ip6_tables iptable_filter dell_smbios sparse_keymap dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr shpchp i2c_piix4 ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter tpm_crb mlx5_ib(OE) mlx4_ib(OE) ib_uverbs(OE) mlx4_en(OE) mlx5_core(OE) nfsd mlxfw(OE) vfio_mdev(OE) binfmt_misc vfio_iommu_type1 vfio auth_rpcgss mdev(OE) nfs_acl ptp lockd pps_core grace mlx4_core(OE) sunrpc devlink knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci crct10dif_pclmul drm crct10dif_common libahci crc32c_intel libata bnxt_en i2c_core megaraid_sas wmi ib_ipoib(OE) ib_cm(OE) ib_umad(OE) ib_core(OE) mlx_compat(OE) CPU: 0 PID: 119171 Comm: slurmstepd Tainted: P W OE ------------ T 3.10.0-693.5.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R7525/0H3K7P, BIOS 2.8.4 06/23/2022 task: ffff893590ad8000 ti: ffff897679e24000 task.ti: ffff897679e24000 RIP: 0010:[<ffffffff81190389>] [<ffffffff81190389>] put_page+0x29/0x50 RSP: 0018:ffff897679e27d50 EFLAGS: 00000202 RAX: 006fffff0000000c RBX: ffffea051fc8dd80 RCX: 0000000000000001 RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffea051fc8de00 RBP: ffff897679e27d50 R08: ffff88763ff98c00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff811905c0 R13: 0000000000000000 R14: ffffffff81190edc R15: ffff897679e27cf8 FS: 00002af80d423300(0000) GS:ffff887e7e200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000425bd0 CR3: 0000017dbae5c000 CR4: 00000000003406f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Stack: ffff897679e27dd0 ffffffff811f33a2 0000000200000001 ffff887dff7bc900 0000000000000001 0000000000001000 ffff887dff7bc840 0000000000000001 0000000100000002 ffff887dff7bc800 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f33a2>] mem_cgroup_reparent_charges+0x232/0x3c0 [<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da12>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150 [<ffffffff81212241>] do_rmdir+0x1f1/0x220 [<ffffffff81213446>] SyS_rmdir+0x16/0x20 [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b Code: 00 00 66 66 66 66 90 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 f8 0b ff ff 5d c3 66 0f 1f 44 00 00 f6 c4 c0 75 14 f0 ff 4f 1c <0f> 94 c0 84 c0 66 90 74 e6 e8 99 ff ff ff 5d c3 0f 1f 80 00 00
NMI watchdog: BUG: soft lockup - CPU#39 stuck for 23s! [slurmstepd:273709] Modules linked in: binfmt_misc nvidia_uvm(OE) nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 devlink tun overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi ext4 kvm_intel kvm mbcache jbd2 irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq ipmi_ssif joydev snd_seq_device pcspkr snd_pcm sg snd_timer snd mei_me mei lpc_ich i2c_i801 soundcore shpchp ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_cpufreq acpi_pad acpi_power_meter ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul ttm crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ahci megaraid_sas drm ixgbe libahci libata mdio ptp i2c_core pps_core dca wmi dm_mirror dm_region_hash dm_log dm_mod CPU: 39 PID: 273709 Comm: slurmstepd Kdump: loaded Tainted: P OEL ------------ T 3.10.0-862.el7.x86_64 #1 Hardware name: Supermicro SYS-7049GP-TRT/X11DPG-QT, BIOS 3.1 05/23/2019 task: ffff8e3276a5eeb0 ti: ffff8e2951388000 task.ti: ffff8e2951388000 RIP: 0010:[<ffffffffa2da21c9>] [<ffffffffa2da21c9>] put_page+0x29/0x50 RSP: 0018:ffff8e295138bd20 EFLAGS: 00000202 RAX: 006fffff0002000c RBX: ffffd80ca555c040 RCX: 0000000000000001 RDX: 000000000000000a RSI: 0000000000000001 RDI: ffffd80cc7c8c480 RBP: ffff8e295138bd20 R08: ffff8e3abe3e6c00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa2da2400 R13: 0000000000000000 R14: ffffffffa2da2d1c R15: ffff8e295138bcc8 FS: 00002b3c18355580(0000) GS:ffff8e783c2c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f6a51465000 CR3: 00000070045c4000 CR4: 00000000007607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: [<ffffffffa2e0d092>] mem_cgroup_reparent_charges+0x232/0x3c0 [<ffffffffa2e0d3d4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffffa2d1c1da>] cgroup_destroy_locked+0xea/0x370 [<ffffffffa2d1c482>] cgroup_rmdir+0x22/0x40 [<ffffffffa2e295ac>] vfs_rmdir+0xdc/0x150 [<ffffffffa2e2cb31>] do_rmdir+0x1f1/0x220 [<ffffffffa2e2dd66>] SyS_rmdir+0x16/0x20 [<ffffffffa331f7d5>] system_call_fastpath+0x1c/0x21 Code: 00 00 0f 1f 44 00 00 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 58 ff fe ff 5d c3 66 0f 1f 44 00 00 f6 c4 c0 75 14 f0 ff 4f 1c <0f> 94 c0 84 c0 66 90 74 e6 e8 99 ff ff ff 5d c3 0f 1f 80 00 00
NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [slurmstepd:204817] Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) squashfs loop overlay(T) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev dcdbas skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel iTCO_wdt iTCO_vendor_support ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) megaraid_sas ahci libahci tg3(OE) libata ptp pps_core [last unloaded: tracedev] CPU: 17 PID: 204817 Comm: slurmstepd Tainted: G B OE ------------ T 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018 task: ffff884d85208fd0 ti: ffff8801ef720000 task.ti: ffff8801ef720000 RIP: 0010:[<ffffffff81190359>] [<ffffffff81190359>] put_page+0x29/0x50 RSP: 0000:ffff8801ef723d50 EFLAGS: 00000202 RAX: 006fffff0000000c RBX: ffffea014f41bf80 RCX: 0000000000000001 RDX: 0000000000000009 RSI: 0000000000000001 RDI: ffffea00c9f5e780 RBP: ffff8801ef723d50 R08: ffff885ed173dc00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590 R13: 0000000000000000 R14: ffffffff81190eac R15: ffff8801ef723cf8 FS: 00002aaaaaafd180(0000) GS:ffff885f7c000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002aaabe75a000 CR3: 00000001dddb7000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff8801ef723dd0 ffffffff811f3432 0000000200000001 ffff8857f6fcd900 0000000000000001 0000000000001000 ffff8857f6fcd840 0000000000000001 0000000100000002 ffff8857f6fcd800 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f3432>] mem_cgroup_reparent_charges+0x232/0x3c0 [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff816b0325>] ? do_page_fault+0x35/0x90 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: 00 00 0f 1f 44 00 00 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 18 0c ff ff 5d c3 66 0f 1f 44 00 00 f6 c4 c0 75 14 f0 ff 4f 1c <0f> 94 c0 84 c0 66 90 74 e6 e8 99 ff ff ff 5d c3 0f 1f 80 00 00
NMI watchdog: BUG: soft lockup - CPU#39 stuck for 22s! [slurmstepd:60168] Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log dcdbas skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci libahci megaraid_sas libata tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler] CPU: 39 PID: 60168 Comm: slurmstepd Tainted: G OE ------------ T 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018 task: ffff88316a61bf40 ti: ffff88310f804000 task.ti: ffff88310f804000 RIP: 0010:[<ffffffff81190350>] [<ffffffff81190350>] put_page+0x20/0x50 RSP: 0018:ffff88310f807d50 EFLAGS: 00000246 RAX: 006fffff00000068 RBX: ffffea0161ef8440 RCX: ffff88310f807c58 RDX: 0000000000000000 RSI: 000000000000000e RDI: ffffea0161ef8440 RBP: ffff88310f807d50 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590 R13: 0000000000000000 R14: ffffffff81190eac R15: ffff88310f807cf8 FS: 00002aaaaaafd180(0000) GS:ffff885f7c2c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffff7ff5000 CR3: 000000012fb12000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff88310f807dd0 ffffffff811f3432 0000000200000001 ffff884a01437500 0000000000000001 0000000000001000 ffff884a01437440 0000000000000001 0000000100000003 ffff884a01437400 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f3432>] mem_cgroup_reparent_charges+0x232/0x3c0 [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff816b0325>] ? do_page_fault+0x35/0x90 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 18 0c ff ff 5d c3 66 0f 1f 44 00 00 <f6> c4 c0 75 14 f0 ff 4f 1c 0f 94 c0 84 c0 66 90 74 e6 e8 99 ff
NMI watchdog: BUG: soft lockup - CPU#28 stuck for 23s! [systemd:1] Modules linked in: veth nvidia_uvm(OE) tcp_diag inet_diag xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc fuse overlay(T) dm_mirror dm_region_hash dm_log dm_mod ib_isert iscsi_target_mod nvidia_drm(POE) nvidia_modeset(POE) ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt iTCO_wdt iTCO_vendor_support ib_ucm rpcrdma rdma_ucm ib_uverbs skx_edac edac_core ib_iser coretemp intel_rapl ib_umad rdma_cm iosf_mbi ib_ipoib kvm_intel iw_cm i40iw kvm irqbypass crc32_pclmul ghash_clmulni_intel libiscsi aesni_intel lrw scsi_transport_iscsi gf128mul ib_cm glue_helper ablk_helper cryptd nvidia(POE) pcspkr joydev sg mei_me mei i2c_i801 shpchp lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm mlx5_ib acpi_power_meter ib_core auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea crc32c_intel mlx5_core sysfillrect sysimgblt fb_sys_fops ttm i40e nvme drm nvme_core ahci libahci devlink libata ptp pps_core i2c_core wmi CPU: 28 PID: 1 Comm: systemd Tainted: P OE ------------ T 3.10.0-693.el7.x86_64 #1 NMI watchdog: BUG: soft lockup - CPU#28 stuck for 23s! [systemd:1] Hardware name: Inspur NF5468M5/YZMB-01130-10C, BIOS 4.1.14 12/23/2020 task: ffff88012ef30000 ti: ffff88012ef38000 task.ti: ffff88012ef38000 RIP: 0010:[<ffffffff811f334b>] [<ffffffff811f334b>] mem_cgroup_reparent_charges+0x15b/0x3c0 RSP: 0018:ffff88012ef3bd60 EFLAGS: 00000246 RAX: 0000000000000002 RBX: ffffea01c4e6e640 RCX: 0000000000000003 RDX: 0000000000000002 RSI: ffffea01f4368e5c RDI: 0000000007d0da39 RBP: ffff88012ef3bdd0 R08: ffff885eb6e62000 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffea01e5bf9200 R13: 0000000000000246 R14: ffffffff81190590 R15: 0000000000000000 FS: 00007fac5fcac940(0000) GS:ffff88bebdc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000c00112a000 CR3: 000000bea7a16000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: 0000000200000001 ffff885eafb3f900 0000000000000001 0000000000001000 ffff885eafb3f840 0000000000000001 0000000100000002 ffff885eafb3f800 0000000000000000 0000000000000000 ffff885e99ecd000 ffff885eafb3f800 Call Trace: [<ffffffff811f3764>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f626>] vfs_rmdir+0xd6/0x150 [<ffffffff81212251>] do_rmdir+0x1f1/0x220 [<ffffffff810ad24c>] ? task_work_run+0xac/0xf0 [<ffffffff81213456>] SyS_rmdir+0x16/0x20 [<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b Code: c6 4c 89 ef e8 07 85 4b 00 48 89 df e8 4f 3b 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 23 25 fa ff 85 c0 0f
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [slurmstepd:342365] Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss dell_rbu nfsv4 dns_resolver dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt irqbypass iTCO_vendor_support dell_smbios dcdbas sparse_keymap crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci libata i40e(OE) megaraid_sas tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler] CPU: 3 PID: 342365 Comm: slurmstepd Tainted: G OE ------------ T 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.21.1 03/07/2024 task: ffff881d2af78fd0 ti: ffff883f953d4000 task.ti: ffff883f953d4000 RIP: 0010:[<ffffffff811f34f6>] [<ffffffff811f34f6>] mem_cgroup_reparent_charges+0x2f6/0x3c0 RSP: 0018:ffff883f953d7d60 EFLAGS: 00000206 RAX: 0000000000000001 RBX: ffffea015a5f9f00 RCX: 0000000000000001 RDX: 00000000148a9849 RSI: 0000000000000001 RDI: ffff883b64084d00 RBP: ffff883f953d7dd0 R08: ffff880bcc8ecc00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffea013d90aa80 R13: 0000000000000246 R14: ffffffff81190590 R15: 0000000000000000 FS: 00002aaaaaafd180(0000) GS:ffff885f7be40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002aaab400c09c CR3: 0000005142ec8000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: 0000000200000001 ffff883b64084d00 0000000000000001 0000000000001000 ffff883b64084c40 0000000000000001 0000000100000003 ffff883b64084c00 0000000000000000 0000000000000000 ffff884938674c00 ffff883b64084c00 Call Trace: [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: 89 45 b8 e8 7d bc ff ff 8b 05 c7 06 93 00 44 8b 55 c4 85 c0 0f 84 26 ff ff ff 48 8b 75 b8 48 8b 7d 98 e8 5e bc ff ff 44 8b 55 c4 <e9> 10 ff ff ff 0f 1f 44 00 00 48 89 c6 4c 89 ef e8 45 83 4b 00
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [systemd:1] Modules linked in: joydev vhost_net vhost macvtap macvlan ebt_arp tun fuse geneve ip6_udp_tunnel udp_tunnel bnx2fc cnic uio fcoe libfcoe libfc scsi_transport_fc scsi_tgt scsi_transport_iscsi nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack openvswitch ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 dm_mirror dm_region_hash dm_log dm_service_time iTCO_wdt iTCO_vendor_support dm_multipath skx_edac dell_smbios intel_powerclamp dcdbas sparse_keymap coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_mod pcspkr sg mei_me mei lpc_ich i2c_i801 shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect crct10dif_pclmul sysimgblt crct10dif_common fb_sys_fops crc32c_intel ttm ahci drm tg3 libahci bnxt_en libata megaraid_sas ptp devlink i2c_core pps_core 8021q garp mrp bridge stp llc bonding CPU: 13 PID: 1 Comm: systemd Kdump: loaded Tainted: G W ------------ 3.10.0-862.2.3.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R640/06NR82, BIOS 2.4.8 11/26/2019 task: ffff8b2867718000 ti: ffff8ae8e7288000 task.ti: ffff8ae8e7288000 RIP: 0010:[<ffffffff86e0cf5b>] [<ffffffff86e0cf5b>] mem_cgroup_reparent_charges+0x15b/0x3c0 RSP: 0018:ffff8ae8e728bd30 EFLAGS: 00000246 RAX: 0000000000000002 RBX: fffffbe2815f3740 RCX: 0000000000000003 RDX: 0000000000000002 RSI: fffffbe323c925dc RDI: 00000000078f2497 RBP: ffff8ae8e728bda0 R08: ffff8ae93fdce400 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: fffffbe322b06300 R13: 0000000000000246 R14: ffffffff86da23a0 R15: 0000000000000000 FS: 00007f6e5ca5a940(0000) GS:ffff8b673bf80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055b6d3410438 CR3: 0000007f7d130000 CR4: 00000000007607e0 PKRU: 55555554 Call Trace: [<ffffffff86e0d374>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff86d1c1da>] cgroup_destroy_locked+0xea/0x370 [<ffffffff86d1c482>] cgroup_rmdir+0x22/0x40 [<ffffffff86e2954c>] vfs_rmdir+0xdc/0x150 [<ffffffff86e2cad1>] do_rmdir+0x1f1/0x220 [<ffffffff86e1d18e>] ? ____fput+0xe/0x10 [<ffffffff86e2dd06>] SyS_rmdir+0x16/0x20 [<ffffffff8731fa11>] tracesys+0x9d/0xc3 Code: c6 4c 89 ef e8 57 84 50 00 48 89 df e8 9f 3c 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 a3 a7 f9 ff 85 c0 0f
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [slurmstepd:371919] Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss nfsv4 dell_rbu dns_resolver dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul crc32c_intel dell_smbios dcdbas sparse_keymap ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect aesni_intel sysimgblt lrw gf128mul fb_sys_fops glue_helper drm ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci libata i40e(OE) megaraid_sas tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler] CPU: 8 PID: 371919 Comm: slurmstepd Tainted: G W OE ------------ T 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.21.1 03/07/2024 task: ffff883e8cf41fa0 ti: ffff88321bb24000 task.ti: ffff88321bb24000 RIP: 0010:[<ffffffff811f335b>] [<ffffffff811f335b>] mem_cgroup_reparent_charges+0x15b/0x3c0 RSP: 0018:ffff88321bb27d60 EFLAGS: 00000246 RAX: 0000000000000001 RBX: ffffea00ec6dbb00 RCX: 0000000000000002 RDX: 0000000000000001 RSI: ffffea0100dc4f1c RDI: 000000000403713c RBP: ffff88321bb27dd0 R08: ffff885f4ef2ac00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffea00db8b4040 R13: 0000000000000246 R14: ffffffff81190590 R15: 0000000000000000 FS: 00002aaaaaafd180(0000) GS:ffff882f7db00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffff7ff8000 CR3: 0000003219b11000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: 0000000200000001 ffff8801e1ed8100 0000000000000001 0000000000001000 ffff8801e1ed8040 0000000000000001 0000000100000002 ffff8801e1ed8000 0000000000000000 0000000000000000 ffff88041a196c00 ffff8801e1ed8000 Call Trace: [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff816b0325>] ? do_page_fault+0x35/0x90 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: c6 4c 89 ef e8 17 85 4b 00 48 89 df e8 4f 3b 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 13 25 fa ff 85 c0 0f
NMI watchdog: BUG: soft lockup - CPU#27 stuck for 23s! [cgclear:12485] Modules linked in: binfmt_misc nfsv3 nfs_acl rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace tun devlink ip_set nfnetlink bridge stp llc libcrc32c cachefiles fscache dm_mirror dm_region_hash dm_log dm_mod vfat fat iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg i2c_i801 hpilo hpwdt ipmi_si ipmi_devintf lpc_ich ipmi_msghandler pcc_cpufreq ioatdma wmi acpi_power_meter dca auth_rpcgss sunrpc ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 i2c_algo_bit drm_kms_helper syscopyarea tg3 sysfillrect sysimgblt fb_sys_fops ttm ptp ahci pps_core drm libahci libata drm_panel_orientation_quirks [last unloaded: ip_tables] CPU: 27 PID: 12485 Comm: cgclear Kdump: loaded Tainted: G ------------ T 3.10.0-957.el7.centos.plus.x86_64 #1 Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 04/29/2021 task: ffff9bd362815140 ti: ffff9bdda4bc4000 task.ti: ffff9bdda4bc4000 RIP: 0010:[<ffffffffbd57c0f5>] [<ffffffffbd57c0f5>] _raw_spin_unlock_irqrestore+0x15/0x20 RSP: 0018:ffff9bdda4bc7d20 EFLAGS: 00000283 RAX: 0000000000000283 RBX: fffff8e857d6b080 RCX: ffff9bdda4bc7c28 RDX: 0000000000000001 RSI: 0000000000000283 RDI: 0000000000000283 RBP: ffff9bdda4bc7d20 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffbcfc5930 R13: 0000000000000000 R14: ffffffffbcfc624c R15: ffff9bdda4bc7cc8 FS: 00007fa26dc0bb80(0000) GS:ffff9bddbedc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000006dee20 CR3: 00000020377e0000 CR4: 00000000003607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: [<ffffffffbd033eb9>] mem_cgroup_reparent_charges+0x139/0x3c0 [<ffffffffbd0342f4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffffbcf2673a>] cgroup_destroy_locked+0xea/0x370 [<ffffffffbcf269e2>] cgroup_rmdir+0x22/0x40 [<ffffffffbd04ec2c>] vfs_rmdir+0xdc/0x150 [<ffffffffbd0540d1>] do_rmdir+0x1f1/0x220 [<ffffffffbd055306>] SyS_rmdir+0x16/0x20 [<ffffffffbd586ddb>] system_call_fastpath+0x22/0x27 Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [slurmstepd:85646] Modules linked in: beegfs(OE) nvidia_uvm(OE) nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) vfio_iommu_type1 vfio nouveau mxm_wmi video xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm i40iw mlx5_ib ib_core dm_mirror dm_region_hash dm_log dm_mod vfat fat iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev ipmi_ssif pcspkr sg mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper mlx5_core crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect i40e sysimgblt fb_sys_fops ttm drm devlink ahci libahci ptp pps_core libata i2c_core wmi [last unloaded: beegfs] CPU: 23 PID: 85646 Comm: slurmstepd Tainted: P OEL ------------ 3.10.0-693.11.6.el7.x86_64 #1 Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 3.3 02/21/2020 task: ffff88004aa7bf40 ti: ffff88020097c000 task.ti: ffff88020097c000 RIP: 0010:[<ffffffff811f529b>] [<ffffffff811f529b>] mem_cgroup_reparent_charges+0x15b/0x3c0 RSP: 0018:ffff88020097fd30 EFLAGS: 00000246 RAX: 0000000000000001 RBX: ffffea013f4deec0 RCX: 0000000000000002 RDX: 0000000000000001 RSI: ffffea01893f059c RDI: 000000000624fc16 RBP: ffff88020097fda0 R08: ffff884df025b000 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffea01f7d29540 R13: 0000000000000246 R14: ffffffff811922b0 R15: 0000000000000000 FS: 00002ada079a5440(0000) GS:ffff887f7dac0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002ada0b437ff8 CR3: 0000004148638000 CR4: 00000000003607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: [<ffffffff811f56b4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110f467>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110f712>] cgroup_rmdir+0x22/0x40 [<ffffffff812114f6>] vfs_rmdir+0xd6/0x150 [<ffffffff81214191>] do_rmdir+0x1f1/0x220 [<ffffffff816b3801>] ? __do_page_fault+0x171/0x450 [<ffffffff81215396>] SyS_rmdir+0x16/0x20 [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b Code: c6 4c 89 ef e8 87 88 4b 00 48 89 df e8 4f 3b 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 f3 22 fa ff 85 c0 0f
NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [slurmstepd:215944] Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) 8021q garp mrp stp llc panfs(POE) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ib_isert iscsi_target_mod iTCO_wdt iTCO_vendor_support ib_srpt target_core_mod rpcrdma ib_srp scsi_transport_srp ib_iser scsi_tgt libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm ib_ucm skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm ib_uverbs ib_umad irqbypass crc32_pclmul ghash_clmulni_intel rdma_cm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ib_cm iw_cm pcspkr joydev sg i2c_i801 lpc_ich mei_me mei shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler mlx5_ib ib_core nfit libnvdimm acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ast crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea sysfillrect mlx5_core sysimgblt fb_sys_fops ttm drm igb ahci libahci nvme devlink libata nvme_core dca ptp i2c_algo_bit pps_core i2c_core dm_mirror dm_region_hash dm_log dm_mod CPU: 30 PID: 215944 Comm: slurmstepd Tainted: P OE ------------ 3.10.0-693.el7.x86_64 #1 Hardware name: Penguin Computing Relion XE1112/MR91-FS0-ZB, BIOS R03 06/03/2019 task: ffff88bcfa870fd0 ti: ffff88bc79e70000 task.ti: ffff88bc79e70000 RIP: 0010:[<ffffffff811f34c9>] [<ffffffff811f34c9>] mem_cgroup_reparent_charges+0x2d9/0x3c0 RSP: 0018:ffff88bc79e73d60 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffffea0484c7e400 RCX: 0000000000000001 RDX: 00000000083465a8 RSI: 0000000000000001 RDI: ffff897d247bcc40 RBP: ffff88bc79e73dd0 R08: ffff88b7a573dc00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffea0484c7e340 R13: 0000000000000246 R14: ffffffff81190590 R15: 0000000000000000 FS: 00002b238472b340(0000) GS:ffff897d3c180000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f09de87e000 CR3: 000000b9b12fb000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: 0000000200000001 ffff897d247bcd00 0000000000000001 0000000000001000 ffff897d247bcc40 0000000000000001 0000000100000002 ffff897d247bcc00 0000000000000000 0000000000000000 ffff8977eeb7ba00 ffff897d247bcc00 Call Trace: [<ffffffff811f3764>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f626>] vfs_rmdir+0xd6/0x150 [<ffffffff81212251>] do_rmdir+0x1f1/0x220 [<ffffffff816b02e5>] ? do_page_fault+0x35/0x90 [<ffffffff81213456>] SyS_rmdir+0x16/0x20 [<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b Code: 89 c6 e8 8b 83 4b 00 e9 46 fe ff ff 66 0f 1f 44 00 00 48 8b 7d b0 49 63 c3 48 89 c6 48 89 45 b8 e8 7d bc ff ff 8b 05 d7 06 93 00 <44> 8b 55 c4 85 c0 0f 84 26 ff ff ff 48 8b 75 b8 48 8b 7d 98 e8
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [slurmstepd:37077] Modules linked in: squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic nfs_layout_nfsv41_files crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log ses enclosure sd_mod sg dm_mod intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd uas serio_raw pcspkr usb_storage smartpqi scsi_transport_sas hpilo hpwdt nvme shpchp lpc_ich wmi acpi_power_meter rpcrdma ib_isert iscsi_target_mod target_core_mod crc_t10dif crct10dif_generic ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ipmi_devintf hfi1(OE) i2c_algo_bit i2c_core bnxt_en crct10dif_pclmul crct10dif_common rdmavt(OE) crc32c_intel ib_core ipmi_si fjes ipmi_msghandler CPU: 11 PID: 37077 Comm: slurmstepd Tainted: G W OEL ------------ T 3.10.0-514.10.2.el7.x86_64 #1 Hardware name: HPE ProLiant DL560 Gen10/ProLiant DL560 Gen10, BIOS U34 03/16/2023 task: ffff88856e3bce70 ti: ffff896d09a14000 task.ti: ffff896d09a14000 RIP: 0010:[<ffffffff811f123b>] [<ffffffff811f123b>] mem_cgroup_reparent_charges+0x15b/0x3c0 RSP: 0018:ffff896d09a17d60 EFLAGS: 00000246 RAX: 0000000000000002 RBX: ffffea0e393a66c0 RCX: 0000000000000003 RDX: 0000000000000002 RSI: ffffea0e393a67dc RDI: ffffea0e393a67c0 RBP: ffff896d09a17dd0 R08: 0000000000000287 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffea0e393a6540 R13: 0000000000000246 R14: ffffffff8118eb20 R15: 0000000000000000 FS: 00002ada5c8d0300(0000) GS:ffff88fc3fec0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ff18e219000 CR3: 000003255122e000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: 0000000200000003 ffff8b00df5b5100 0000000000000003 0000000000001000 ffff8b00df5b5040 0000000000000001 0000000100000003 ffff8b00df5b5000 0000000000000000 0000000000000000 ffff8b26efbd7a00 ffff8b00df5b5000 Call Trace: [<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360 [<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40 [<ffffffff8120a278>] vfs_rmdir+0xa8/0x100 [<ffffffff8120f315>] do_rmdir+0x1a5/0x200 [<ffffffff81692305>] ? do_page_fault+0x35/0x90 [<ffffffff812104c6>] SyS_rmdir+0x16/0x20
NMI watchdog: BUG: soft lockup - CPU#69 stuck for 22s! [slurmstepd:105299] Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic nfs_layout_nfsv41_files crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod ses enclosure sd_mod sg intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd serio_raw pcspkr uas usb_storage smartpqi nvme scsi_transport_sas hpilo hpwdt lpc_ich shpchp wmi acpi_power_meter rpcrdma ib_isert iscsi_target_mod target_core_mod crc_t10dif crct10dif_generic ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables hfi1(OE) i2c_algo_bit i2c_core crct10dif_pclmul ipmi_devintf rdmavt(OE) crct10dif_common crc32c_intel ib_core bnxt_en ipmi_si fjes ipmi_msghandler CPU: 69 PID: 105299 Comm: slurmstepd Tainted: G W OEL ------------ T 3.10.0-514.10.2.el7.x86_64 #1 Hardware name: HPE ProLiant DL560 Gen10/ProLiant DL560 Gen10, BIOS U34 04/21/2022 task: ffff8ae84aa36dd0 ti: ffff8af378f5c000 task.ti: ffff8af378f5c000 RIP: 0010:[<ffffffff8168decb>] [<ffffffff8168decb>] _raw_spin_lock_irqsave+0xb/0x60 RSP: 0018:ffff8af378f5fd50 EFLAGS: 00000287 RAX: 0000000000000287 RBX: ffffea0a4510e340 RCX: 0000000000000001 RDX: 0000000000000009 RSI: 0000000000000001 RDI: ffff8b003ffda540 RBP: ffff8af378f5fd50 R08: ffff89fc3df7e000 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8118eb20 R13: 0000000000000000 R14: ffffffff8118f44c R15: ffff8af378f5fcf8 FS: 00002b7c2921e300(0000) GS:ffff8bfc3edc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000025d2390 CR3: 000003f5ec361000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff8af378f5fdd0 ffffffff811f11f2 0000000200000002 ffff8966ada02d00 0000000000000002 0000000000001000 ffff8966ada02c40 0000000000000001 0000000100000002 ffff8966ada02c00 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f11f2>] mem_cgroup_reparent_charges+0x112/0x3c0 [<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360 [<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40 [<ffffffff8120a278>] vfs_rmdir+0xa8/0x100 [<ffffffff8120f315>] do_rmdir+0x1a5/0x200 [<ffffffff81692305>] ? do_page_fault+0x35/0x90 [<ffffffff812104c6>] SyS_rmdir+0x16/0x20 [<ffffffff81696b09>] system_call_fastpath+0x16/0x1b Code: 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 83 e8 01 75 f1 48 89 df 0f 1f 80 00 00 00 00 eb e0 66 90 0f 1f 44 00 00 55 48 89 e5 9c 58 <0f> 1f 44 00 00 49 89 c0 fa 66 0f 1f 44 00 00 ba 00 00 02 00 f0
NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [slurmstepd:98377] Modules linked in: 8021q garp mrp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm dm_mirror dm_region_hash dm_log dm_mod skx_edac edac_core intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel hfi1 aesni_intel lrw gf128mul glue_helper ablk_helper cryptd rdmavt ib_core mei_me joydev mei sg pcspkr i2c_i801 shpchp lpc_ich ipmi_si wmi ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ixgbe ttm crc32c_intel ahci mdio libahci drm ptp libata pps_core i2c_core dca CPU: 23 PID: 98377 Comm: slurmstepd Tainted: G W L ------------ 3.10.0-693.5.2.el7.x86_64 #1 Hardware name: Advanced Clustering S2600BPB/S2600BPB, BIOS SE5C620.86B.00.01.0009.C2.101920170742 10/19/2017 task: ffff885f62a40000 ti: ffff885f67cb4000 task.ti: ffff885f67cb4000 RIP: 0010:[<ffffffff811f32e5>] [<ffffffff811f32e5>] mem_cgroup_reparent_charges+0x175/0x3c0 RSP: 0018:ffff885f67cb7d60 EFLAGS: 00000246 RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffffffffff83 RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff88603ffd6540 RBP: ffff885f67cb7dd0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000001 R12: ffffea0157a08480 R13: 0000000000000246 R14: ffffffff811905c0 R15: 0000000000000000 FS: 00002ad2510cad00(0000) GS:ffff885f7bec0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f458d4c7000 CR3: 0000002f65168000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: 0000000200000001 ffff882f6d3cf500 0000000000000001 0000000000001000 ffff882f6d3cf440 0000000000000001 0000000100000002 ffff882f6d3cf400 0000000000000000 0000000000000000 ffff882f66eb8c00 ffff882f6d3cf400 Call Trace: [<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da12>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150 [<ffffffff81212241>] do_rmdir+0x1f1/0x220 [<ffffffff816a904d>] ? __schedule+0x39d/0x8b0 [<ffffffff81213446>] SyS_rmdir+0x16/0x20 [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b Code: 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc 39 c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 d3 25 fa ff 85 c0 0f 85 7b ff ff ff <49> 8b 47 e0 48 c7 45 d0 00 00 00 00 25 00 40 00 00 48 83 f8 01
NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [slurmstepd:264116] Modules linked in: 8021q garp mrp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 nfs fscache ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ucm rpcrdma rdma_ucm ib_uverbs ib_iser dm_mirror rdma_cm opa_vnic dm_region_hash iw_cm dm_log ib_umad libiscsi dm_mod ib_ipoib scsi_transport_iscsi ib_cm skx_edac intel_powerclamp iTCO_wdt coretemp iTCO_vendor_support intel_rapl iosf_mbi kvm hfi1 irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel rdmavt lrw gf128mul glue_helper ablk_helper cryptd ib_core pcspkr mei_me sg joydev mei ipmi_si i2c_i801 shpchp lpc_ich wmi ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect crct10dif_pclmul sysimgblt crct10dif_common fb_sys_fops ixgbe crc32c_intel ttm ahci drm mdio libahci ptp libata pps_core i2c_core dca nfit libnvdimm fuse CPU: 22 PID: 264116 Comm: slurmstepd Kdump: loaded Tainted: G L ------------ 3.10.0-862.9.1.el7.x86_64 #1 Hardware name: Advanced Clustering S2600BPB/S2600BPB, BIOS SE5C620.86B.00.01.0013.C1.030920180427 03/09/2018 task: ffffa0efcc312f70 ti: ffffa0f7fafac000 task.ti: ffffa0f7fafac000 RIP: 0010:[<ffffffffa6b16890>] [<ffffffffa6b16890>] _raw_spin_lock_irqsave+0x0/0x40 RSP: 0018:ffffa0f7fafafd28 EFLAGS: 00000283 RAX: 006fffff00000000 RBX: fffffdd443dd9a80 RCX: ffffa0f7fafafc28 RDX: 0000000000000000 RSI: 000000000000000e RDI: ffffa120fffd4540 RBP: ffffa0f7fafafda0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa65a2bc0 R13: 0000000000000000 R14: ffffffffa65a34dc R15: ffffa0f7fafafcc8 FS: 00002b1c38745d40(0000) GS:ffffa1203be80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002ad8d630b0a0 CR3: 000000029f2c6000 CR4: 00000000007607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: [<ffffffffa660d732>] ? mem_cgroup_reparent_charges+0x112/0x3c0 [<ffffffffa660db94>] mem_cgroup_css_offline+0x84/0x140 [<ffffffffa651c9ca>] cgroup_destroy_locked+0xea/0x370 [<ffffffffa651cc72>] cgroup_rmdir+0x22/0x40 [<ffffffffa6629d6c>] vfs_rmdir+0xdc/0x150 [<ffffffffa662d2f1>] do_rmdir+0x1f1/0x220 [<ffffffffa662e526>] SyS_rmdir+0x16/0x20 [<ffffffffa6b20795>] system_call_fastpath+0x1c/0x21 Code: 02 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 82 2c ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 <0f> 1f 44 00 00 55 48 89 e5 53 9c 58 0f 1f 44 00 00 48 89 c3 fa
NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [slurmstepd:49637] Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge 8021q garp mrp stp llc overlay(T) ip6_tables iTCO_wdt iTCO_vendor_support iptable_filter coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure cdc_ether scsi_transport_sas usbnet joydev mii sg mei_me mei shpchp i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace binfmt_misc sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw ahci libahci mlx5_core(OE) libata mlxfw(OE) ptp megaraid_sas pps_core auxiliary(OE) mlx_compat(OE) i2c_core devlink [last unloaded: tracedev] CPU: 24 PID: 49637 Comm: slurmstepd Tainted: G W OEL ------------ T 3.10.0-693.5.2.el7.x86_64 #1 Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW, BIOS AFE112A-1.02 05/10/2021 task: ffff88cf781f0000 ti: ffff88c545d64000 task.ti: ffff88c545d64000 RIP: 0010:[<ffffffff816abc6b>] [<ffffffff816abc6b>] _raw_spin_lock_irqsave+0xb/0x40 RSP: 0018:ffff88c545d67d40 EFLAGS: 00000283 RAX: 002fffff00000000 RBX: ffffffff811905c0 RCX: 0000000000000001 RDX: 0000000000000006 RSI: 0000000000000001 RDI: ffff8a003ffd8540 RBP: ffff88c545d67d50 R08: ffff89efdc63d800 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190edc R13: ffff88c545d67cf8 R14: ffff8a003ffd8000 R15: ffff8a003ffd8000 FS: 00002af6b12d1fc0(0000) GS:ffff89f83e000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3a35e3e000 CR3: 000000c4fe833000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: 0000000000000283 ffffea0774ab6340 ffff88c545d67dd0 ffffffff811f3282 0000000200000000 ffff89efdc639d00 0000000000000000 0000000000001000 ffff89efdc639c40 0000000000000001 0000000100000002 ffff89efdc639c00 Call Trace: [<ffffffff811f3282>] mem_cgroup_reparent_charges+0x112/0x3c0 [<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da12>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150 [<ffffffff81212241>] do_rmdir+0x1f1/0x220 [<ffffffff81213446>] SyS_rmdir+0x16/0x20 [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b Code: 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 61 2a ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 9c <58> 0f 1f 44 00 00 48 89 c3 fa 66 0f 1f 44 00 00 31 c0 ba 01 00
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [slurmstepd:222001] Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sr_mod cdrom mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) megaraid_sas aacraid aic79xx ata_piix sd_mod crc_t10dif mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil tg3(OE) e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sg crct10dif_pclmul crct10dif_common ahci libahci igb(OE) i2c_algo_bit i2c_core i40e(OE) dca ptp libata pps_core [last unloaded: tracedev] CPU: 4 PID: 222001 Comm: slurmstepd Tainted: G B W OE ------------ 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: FUJITSU PRIMERGY RX2540 M5/D3384-B1, BIOS V5.0.0.14 R1.15.0 for D3384-B1x 10/17/2019 task: ffff88487c232f70 ti: ffff885fb4718000 task.ti: ffff885fb4718000 RIP: 0010:[<ffffffff816ab865>] [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20 RSP: 0018:ffff885fb471bd50 EFLAGS: 00000287 RAX: 0000000000000287 RBX: ffffea0176795e40 RCX: ffff885fb471bc58 RDX: 0000000000000001 RSI: 0000000000000287 RDI: 0000000000000287 RBP: ffff885fb471bd50 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590 R13: 0000000000000000 R14: ffffffff81190eac R15: ffff885fb471bcf8 FS: 00002aaaaaafd180(0000) GS:ffff882fdf040000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002aac1f2c6120 CR3: 000000184dbcf000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff885fb471bdd0 ffffffff811f3339 0000000200000003 ffff885dd9c6b100 0000000000000003 0000000000001000 ffff885dd9c6b040 0000000000000001 0000000100000003 ffff885dd9c6b000 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f3339>] mem_cgroup_reparent_charges+0x139/0x3c0 [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff816b0325>] ? do_page_fault+0x35/0x90 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
NMI watchdog: BUG: soft lockup - CPU#23 stuck for 23s! [slurmstepd:126513] Modules linked in: nvidia_uvm(OE) nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 devlink tun overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel ext4 lrw gf128mul glue_helper ablk_helper cryptd mbcache jbd2 snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq ipmi_ssif snd_seq_device pcspkr joydev snd_pcm snd_timer sg snd mei_me mei soundcore i2c_i801 lpc_ich shpchp ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_cpufreq acpi_power_meter acpi_pad ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper ttm crct10dif_pclmul syscopyarea crct10dif_common sysfillrect sysimgblt crc32c_intel fb_sys_fops ixgbe ahci drm megaraid_sas libahci libata mdio ptp i2c_core pps_core dca wmi dm_mirror dm_region_hash dm_log dm_mod CPU: 23 PID: 126513 Comm: slurmstepd Kdump: loaded Tainted: P OEL ------------ T 3.10.0-862.el7.x86_64 #1 Hardware name: Supermicro SYS-7049GP-TRT/X11DPG-QT, BIOS 3.1 05/23/2019 task: ffff88dcc6adcf10 ti: ffff894ceda8c000 task.ti: ffff894ceda8c000 RIP: 0010:[<ffffffffa5515375>] [<ffffffffa5515375>] _raw_spin_unlock_irqrestore+0x15/0x20 RSP: 0018:ffff894ceda8fd20 EFLAGS: 00000283 RAX: 0000000000000283 RBX: ffffee473280d840 RCX: 0000000000000001 RDX: 0000000000000001 RSI: 0000000000000283 RDI: 0000000000000283 RBP: ffff894ceda8fd20 R08: ffff8952b7b34c00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa4fa2400 R13: 0000000000000000 R14: ffffffffa4fa2d1c R15: ffff894ceda8fcc8 FS: 00002b865a64b580(0000) GS:ffff8952fbec0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000001c34930 CR3: 0000007f4bbf6000 CR4: 00000000007607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: [<ffffffffa500cf99>] mem_cgroup_reparent_charges+0x139/0x3c0 [<ffffffffa500d3d4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffffa4f1c1da>] cgroup_destroy_locked+0xea/0x370 [<ffffffffa4f1c482>] cgroup_rmdir+0x22/0x40 [<ffffffffa50295ac>] vfs_rmdir+0xdc/0x150 [<ffffffffa502cb31>] do_rmdir+0x1f1/0x220 [<ffffffffa502dd66>] SyS_rmdir+0x16/0x20 [<ffffffffa551f7d5>] system_call_fastpath+0x1c/0x21 Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [slurmstepd:34377] Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge 8021q garp mrp stp llc overlay(T) ip6_tables iTCO_wdt iTCO_vendor_support iptable_filter coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure cdc_ether scsi_transport_sas usbnet joydev mii sg mei_me mei shpchp i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace binfmt_misc sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw ahci libahci mlx5_core(OE) libata mlxfw(OE) ptp megaraid_sas pps_core auxiliary(OE) mlx_compat(OE) i2c_core devlink [last unloaded: tracedev] CPU: 10 PID: 34377 Comm: slurmstepd Tainted: G W OEL ------------ T 3.10.0-693.5.2.el7.x86_64 #1 Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW, BIOS AFE112A-1.02 05/10/2021 task: ffff89672f4a8000 ti: ffff89f2f3fe4000 task.ti: ffff89f2f3fe4000 RIP: 0010:[<ffffffff811f32cb>] [<ffffffff811f32cb>] mem_cgroup_reparent_charges+0x15b/0x3c0 RSP: 0018:ffff89f2f3fe7d60 EFLAGS: 00000246 RAX: 0000000000000002 RBX: ffffea008fe16e40 RCX: 0000000000000003 RDX: 0000000000000002 RSI: ffffea00764a6b9c RDI: 0000000001d929ae RBP: ffff89f2f3fe7dd0 R08: ffff89efed2b3000 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffea062dba8140 R13: 0000000000000246 R14: ffffffff811905c0 R15: 0000000000000000 FS: 00002ab626029fc0(0000) GS:ffff89f83dc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002ab62bf08620 CR3: 00000131456ce000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: 0000000200000000 ffff89efed2b7d00 0000000000000000 0000000000001000 ffff89efed2b7c40 0000000000000001 0000000100000002 ffff89efed2b7c00 0000000000000000 0000000000000000 ffff891a47532200 ffff89efed2b7c00 Call Trace: [<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da12>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150 [<ffffffff81212241>] do_rmdir+0x1f1/0x220 [<ffffffff81213446>] SyS_rmdir+0x16/0x20 [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b Code: c6 4c 89 ef e8 47 86 4b 00 48 89 df e8 4f 3b 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 d3 25 fa ff 85 c0 0f
NMI watchdog: BUG: soft lockup - CPU#38 stuck for 23s! [slurmstepd:30810] Modules linked in: nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) nvidia(POE) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif pcspkr sg sb_edac edac_core hpilo hpwdt lpc_ich ioatdma shpchp ipmi_si acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c mlx4_en sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul mgag200 crct10dif_common ipmi_msghandler crc32c_intel drm_kms_helper mlx4_core syscopyarea sysfillrect serio_raw sysimgblt fb_sys_fops ttm igb ahci hpsa libahci drm scsi_transport_sas ptp devlink pps_core libata dca i2c_algo_bit i2c_core wmi fjes dm_mirror dm_region_hash dm_log dm_mod [last unloaded: nvidia] CPU: 38 PID: 30810 Comm: slurmstepd Tainted: P B OEL ------------ 3.10.0-514.21.2.el7.x86_64 #1 Hardware name: HP ProLiant SL270s Gen8 SE/, BIOS P75 09/18/2013 task: ffff883f26fa8000 ti: ffff883f4ad3c000 task.ti: ffff883f4ad3c000 RIP: 0010:[<ffffffff8168ea4b>] [<ffffffff8168ea4b>] _raw_spin_lock_irqsave+0xb/0x60 RSP: 0018:ffff883f4ad3fd50 EFLAGS: 00000287 RAX: 0000000000000287 RBX: ffffea00a083ccc0 RCX: 0000000000000001 RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88403ffd9540 RBP: ffff883f4ad3fd50 R08: ffff883ef52b9800 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8118f010 R13: 0000000000000000 R14: ffffffff8118f93c R15: ffff883f4ad3fcf8 FS: 00007f6ac1a79780(0000) GS:ffff88403f880000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fc648970000 CR3: 00000007a42af000 CR4: 00000000001407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Stack: ffff883f4ad3fdd0 ffffffff811f1832 0000000200000001 ffff882093a64900 0000000000000001 0000000000001000 ffff882093a64840 0000000000000001 0000000100000002 ffff882093a64800 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f1832>] mem_cgroup_reparent_charges+0x112/0x3c0 [<ffffffff811f1c94>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d6e7>] cgroup_destroy_locked+0xe7/0x360 [<ffffffff8110d982>] cgroup_rmdir+0x22/0x40 [<ffffffff8120a8c8>] vfs_rmdir+0xa8/0x100 [<ffffffff8120f965>] do_rmdir+0x1a5/0x200 [<ffffffff8168c130>] ? __schedule+0x3b0/0x990 [<ffffffff81210b16>] SyS_rmdir+0x16/0x20 [<ffffffff81697749>] system_call_fastpath+0x16/0x1b Code: 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 83 e8 01 75 f1 48 89 df 0f 1f 80 00 00 00 00 eb e0 66 90 0f 1f 44 00 00 55 48 89 e5 9c 58 <0f> 1f 44 00 00 49 89 c0 fa 66 0f 1f 44 00 00 ba 00 00 02 00 f0
NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [slurmstepd:136229] Modules linked in: tcp_lp fuse nf_conntrack_netlink xt_addrtype xt_CHECKSUM br_netfilter ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iTCO_wdt iTCO_vendor_support ext4 skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel mbcache jbd2 kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep joydev snd_seq ipmi_ssif snd_seq_device pcspkr snd_pcm sg snd_timer mei_me snd lpc_ich mei i2c_i801 shpchp soundcore ipmi_si ipmi_devintf ipmi_msghandler nfit acpi_cpufreq libnvdimm acpi_power_meter acpi_pad ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common ttm syscopyarea crc32c_intel sysfillrect ixgbe sysimgblt fb_sys_fops ahci drm megaraid_sas libahci libata mdio ptp i2c_core pps_core dca wmi dm_mirror dm_region_hash dm_log dm_mod CPU: 22 PID: 136229 Comm: slurmstepd Kdump: loaded Tainted: P OE ------------ T 3.10.0-862.el7.x86_64 #1 Hardware name: Supermicro SYS-7049GP-TRT/X11DPG-QT, BIOS 3.1 05/23/2019 task: ffff92d401b18fd0 ti: ffff92d40062c000 task.ti: ffff92d40062c000 RIP: 0010:[<ffffffffabf156db>] [<ffffffffabf156db>] _raw_spin_lock_irqsave+0xb/0x40 RSP: 0018:ffff92d40062fd10 EFLAGS: 00000287 RAX: 006fffff00020000 RBX: ffffffffab9a2400 RCX: 0000000000000001 RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff92d53ffd9540 RBP: ffff92d40062fd20 R08: ffff92d4389a6800 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffab9a2d1c R13: ffff92d40062fcc8 R14: ffff92d53ffd9000 R15: ffff92d53ffd9000 FS: 00002ac2b58c1580(0000) GS:ffff92d43be80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f1355bec160 CR3: 0000007f60bce000 CR4: 00000000007607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: [<ffffffffaba0cf72>] mem_cgroup_reparent_charges+0x112/0x3c0 [<ffffffffaba0d3d4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffffab91c1da>] cgroup_destroy_locked+0xea/0x370 [<ffffffffab91c482>] cgroup_rmdir+0x22/0x40 [<ffffffffaba295ac>] vfs_rmdir+0xdc/0x150 [<ffffffffaba2cb31>] do_rmdir+0x1f1/0x220 [<ffffffffaba2dd66>] SyS_rmdir+0x16/0x20 [<ffffffffabf1f7d5>] system_call_fastpath+0x1c/0x21 Code: 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 5c 2d ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 9c <58> 0f 1f 44 00 00 48 89 c3 fa 66 0f 1f 44 00 00 31 c0 ba 01 00
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [systemd:1] Modules linked in: veth xt_addrtype br_netfilter overlay(T) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache devlink nls_utf8 isofs loop xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support dell_smbios sparse_keymap dcdbas vfat fat skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif joydev sg pcspkr ipmi_si ipmi_devintf ipmi_msghandler mei_me mei shpchp nfit i2c_i801 lpc_ich acpi_power_meter wmi libnvdimm acpi_pad tpm_crb nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci tg3 libata megaraid_sas bnxt_en ptp i2c_core pps_core dm_mirror dm_region_hash dm_log dm_mod CPU: 13 PID: 1 Comm: systemd Tainted: G W ------------ T 3.10.0-693.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R740/0DY2X0, BIOS 2.8.2 08/27/2020 task: ffff8840a7618000 ti: ffff880127164000 task.ti: ffff880127164000 RIP: 0010:[<ffffffff816abbb1>] [<ffffffff816abbb1>] _raw_spin_lock_irqsave+0x11/0x40 RSP: 0018:ffff880127167d48 EFLAGS: 00000283 RAX: 0000000000000283 RBX: ffffffff81190590 RCX: 0000000000000001 RDX: 0000000000000002 RSI: 0000000000000001 RDI: ffff89007ffd7540 RBP: ffff880127167d50 R08: ffff884208ae3c00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190eac R13: ffff880127167cf8 R14: ffff89007ffd7000 R15: ffff89007ffd7000 FS: 00007f0b0aa9c940(0000) GS:ffff88bf7e2c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007faf575c4e20 CR3: 000000ff7bbfa000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffffea033963b880 ffff880127167dd0 ffffffff811f3302 0000000200000003 ffff888a407f2500 0000000000000003 0000000000001000 ffff888a407f2440 0000000000000001 0000000100000002 ffff888a407f2400 0000000000000000 Call Trace: [<ffffffff811f3302>] mem_cgroup_reparent_charges+0x112/0x3c0 [<ffffffff811f3764>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f626>] vfs_rmdir+0xd6/0x150 [<ffffffff81212251>] do_rmdir+0x1f1/0x220 [<ffffffff810ad24c>] ? task_work_run+0xac/0xf0 [<ffffffff81213456>] SyS_rmdir+0x16/0x20 [<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b Code: 01 c3 55 89 c6 48 89 e5 e8 61 2a ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 9c 58 0f 1f 44 00 00 <48> 89 c3 fa 66 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17
NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [slurmstepd:434263] Modules linked in: 8021q garp mrp stp llc nfsv3 nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core dm_mirror dm_region_hash dm_log dm_mod skx_edac intel_powerclamp coretemp iTCO_wdt intel_rapl iTCO_vendor_support iosf_mbi kvm ast i2c_algo_bit ttm irqbypass crc32_pclmul ghash_clmulni_intel drm_kms_helper ixgbe syscopyarea aesni_intel sysfillrect sysimgblt fb_sys_fops lrw drm ptp gf128mul glue_helper i2c_i801 pps_core ablk_helper mdio mei_me cryptd dca joydev mei pcspkr i2c_core sg lpc_ich shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci libata CPU: 19 PID: 434263 Comm: slurmstepd Kdump: loaded Tainted: G L ------------ 3.10.0-862.14.4.el7.x86_64 #1 Hardware name: Advanced Clustering S2600BPB/S2600BPB, BIOS SE5C620.86B.02.01.0008.C1.031920191559 03/19/2019 task: ffffa1163a716eb0 ti: ffffa0ef213ec000 task.ti: ffffa0ef213ec000 RIP: 0010:[<ffffffff8831b445>] [<ffffffff8831b445>] _raw_spin_unlock_irqrestore+0x15/0x20 RSP: 0018:ffffa0ef213efd20 EFLAGS: 00000287 RAX: 0000000000000287 RBX: ffffe656eaec2880 RCX: 0000000000000001 RDX: 0000000000000001 RSI: 0000000000000287 RDI: 0000000000000287 RBP: ffffa0ef213efd20 R08: ffffa0ced2339000 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff87da5160 R13: 0000000000000000 R14: ffffffff87da5a7c R15: ffffa0ef213efcc8 FS: 00002b2e6988bfc0(0000) GS:ffffa1163be40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002b6e30007058 CR3: 000000319d382000 CR4: 00000000007607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: [<ffffffff87e11419>] mem_cgroup_reparent_charges+0x139/0x3c0 [<ffffffff87e11854>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff87d1f4aa>] cgroup_destroy_locked+0xea/0x370 [<ffffffff87d1f752>] cgroup_rmdir+0x22/0x40 [<ffffffff87e2db0c>] vfs_rmdir+0xdc/0x150 [<ffffffff87e31091>] do_rmdir+0x1f1/0x220 [<ffffffff87dfa282>] ? kmem_cache_free+0x1e2/0x200 [<ffffffff87e322c6>] SyS_rmdir+0x16/0x20 [<ffffffff8832579b>] system_call_fastpath+0x22/0x27 Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [slurmstepd:165465] Modules linked in: squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt intel_powerclamp iTCO_vendor_support mxm_wmi coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul dcdbas ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm cdc_ether usbnet mii pcspkr sb_edac mei_me lpc_ich mei edac_core shpchp wmi tpm_crb acpi_power_meter rpcrdma ib_isert iscsi_target_mod target_core_mod crc_t10dif crct10dif_generic ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables hfi1(OE) ipmi_devintf rdmavt(OE) ixgbe ahci ib_core libahci dca crct10dif_pclmul mdio crct10dif_common ptp crc32c_intel i2c_algo_bit libata ipmi_si pps_core i2c_core fjes ipmi_msghandler CPU: 18 PID: 165465 Comm: slurmstepd Tainted: G OE ------------ T 3.10.0-514.10.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge C6320/082F9M, BIOS 2.1.5 04/12/2016 task: ffff88129b202f10 ti: ffff8810b5b60000 task.ti: ffff8810b5b60000 RIP: 0010:[<ffffffff811f123b>] [<ffffffff811f123b>] mem_cgroup_reparent_charges+0x15b/0x3c0 RSP: 0018:ffff8810b5b63d60 EFLAGS: 00000246 RAX: 0000000000000002 RBX: ffffea001a28c500 RCX: 0000000000000003 RDX: 0000000000000002 RSI: ffffea001a28c4dc RDI: ffffea001a28c4c0 RBP: ffff8810b5b63dd0 R08: 0000000000000283 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffea001a2874c0 R13: 0000000000000246 R14: ffffffff8118eb20 R15: 0000000000000000 FS: 00002b17143da5c0(0000) GS:ffff88103e840000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000099bad0 CR3: 0000001095b8f000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: 0000000200000000 ffff88013f13f900 0000000000000000 0000000000001000 ffff88013f13f840 0000000000000001 0000000100000003 ffff88013f13f800 0000000000000000 0000000000000000 ffff8802c8cb6c00 ffff88013f13f800 Call Trace: [<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360 [<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40 [<ffffffff8120a278>] vfs_rmdir+0xa8/0x100 [<ffffffff8120f315>] do_rmdir+0x1a5/0x200 [<ffffffff8168b5b0>] ? __schedule+0x3b0/0x990 [<ffffffff812104c6>] SyS_rmdir+0x16/0x20 [<ffffffff81696b09>] system_call_fastpath+0x16/0x1b Code: c6 4c 89 ef e8 b7 ca 49 00 48 89 df e8 9f 3b 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 d3 34 fa ff 85 c0 0f
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [slurmstepd:201567] Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev dcdbas iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) megaraid_sas ahci tg3(OE) ptp libahci pps_core libata [last unloaded: ipmi_msghandler] CPU: 9 PID: 201567 Comm: slurmstepd Tainted: G OE ------------ T 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018 task: ffff8830d85c9fa0 ti: ffff88315ac44000 task.ti: ffff88315ac44000 RIP: 0010:[<ffffffff811f34d9>] [<ffffffff811f34d9>] mem_cgroup_reparent_charges+0x2d9/0x3c0 RSP: 0018:ffff88315ac47d60 EFLAGS: 00000202 RAX: 0000000000000001 RBX: ffffea00e94e1100 RCX: 0000000000000001 RDX: 0000000005e12040 RSI: 0000000000000001 RDI: ffff88313e951c40 RBP: ffff88315ac47dd0 R08: ffff885f6e548800 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffea00df9a08c0 R13: 0000000000000246 R14: ffffffff81190590 R15: 0000000000000000 FS: 00002aaaaaafd180(0000) GS:ffff885f7bf00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002aaaab0fc0a0 CR3: 000000022a42e000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: 0000000200000001 ffff88313e951d00 0000000000000001 0000000000001000 ffff88313e951c40 0000000000000001 0000000100000003 ffff88313e951c00 0000000000000000 0000000000000000 ffff8844af6b5e00 ffff88313e951c00 Call Trace: [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff816b0325>] ? do_page_fault+0x35/0x90 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: 89 c6 e8 9b 83 4b 00 e9 46 fe ff ff 66 0f 1f 44 00 00 48 8b 7d b0 49 63 c3 48 89 c6 48 89 45 b8 e8 7d bc ff ff 8b 05 c7 06 93 00 <44> 8b 55 c4 85 c0 0f 84 26 ff ff ff 48 8b 75 b8 48 8b 7d 98 e8
NMI watchdog: BUG: soft lockup - CPU#81 stuck for 22s! [slurmstepd:127949] Modules linked in: iscsi_target_mod target_core_mod macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ip_vs nf_conntrack libcrc32c binfmt_misc vfat fat uas usb_storage tcp_lp fuse nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache bonding sunrpc dm_mirror dm_region_hash dm_log dm_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg ipmi_si ipmi_devintf ipmi_msghandler pinctrl_amd i2c_designware_platform i2c_designware_core acpi_cpufreq i2c_piix4 sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci drm_panel_orientation_quirks libata megaraid_sas ixgbe igb(OE) mdio ptp pps_core i2c_algo_bit dca nfit libnvdimm [last unloaded: amifldrv_mod] CPU: 81 PID: 127949 Comm: slurmstepd Kdump: loaded Tainted: P W OEL ------------ T 3.10.0-957.el7.x86_64 #1 Hardware name: N/A 2U Rack Server/65N32-US, BIOS SZZQ042030 07/27/2020 task: ffffa14fb7edd140 ti: ffffa17d5deec000 task.ti: ffffa17d5deec000 RIP: 0010:[<ffffffffbaf6a0f5>] [<ffffffffbaf6a0f5>] _raw_spin_unlock_irqrestore+0x15/0x20 RSP: 0018:ffffa17d5deefd20 EFLAGS: 00000287 RAX: 0000000000000287 RBX: ffffe9ecb26bd140 RCX: 0000000000000001 RDX: 0000000000000001 RSI: 0000000000000287 RDI: 0000000000000287 RBP: ffffa17d5deefd20 R08: ffffa195ee71d000 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffba9c4930 R13: 0000000000000000 R14: ffffffffba9c524c R15: ffffa17d5deefcc8 FS: 00002ae07001afc0(0000) GS:ffffa14bbf640000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000002c057e000 CR3: 00000203c7702000 CR4: 0000000000340fe0 Call Trace: [<ffffffffbaa32eb9>] mem_cgroup_reparent_charges+0x139/0x3c0 [<ffffffffbaa332f4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffffba92573a>] cgroup_destroy_locked+0xea/0x370 [<ffffffffba9259e2>] cgroup_rmdir+0x22/0x40 [<ffffffffbaa4dc2c>] vfs_rmdir+0xdc/0x150 [<ffffffffbaa530d1>] do_rmdir+0x1f1/0x220 [<ffffffffbaa54306>] SyS_rmdir+0x16/0x20 [<ffffffffbaf74ddb>] system_call_fastpath+0x22/0x27 Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48
NMI watchdog: BUG: soft lockup - CPU#96 stuck for 22s! [slurmstepd:64066] Modules linked in: squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod dcdbas intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel xfs aesni_intel lrw gf128mul glue_helper libcrc32c ablk_helper cryptd iTCO_wdt iTCO_vendor_support mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops pcspkr sg drm cdc_ether usbnet mii nfit lpc_ich wmi i2c_i801 shpchp libnvdimm acpi_pad acpi_power_meter rpcrdma ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables sd_mod crc_t10dif crct10dif_generic ipmi_devintf hfi1(OE) igb rdmavt(OE) ahci dca crct10dif_pclmul ib_core crct10dif_common ptp libahci i2c_algo_bit crc32c_intel megaraid_sas(OE) i2c_core pps_core libata ipmi_si ipmi_msghandler fjes CPU: 96 PID: 64066 Comm: slurmstepd Tainted: G W OE ------------ T 3.10.0-514.10.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R940/0V0267, BIOS 2.6.4 04/09/2020 task: ffff88be83364e70 ti: ffff88bc2e6a4000 task.ti: ffff88bc2e6a4000 RIP: 0010:[<ffffffff8168dceb>] [<ffffffff8168dceb>] _raw_spin_unlock_irqrestore+0x1b/0x40 RSP: 0018:ffff88bc2e6a7d48 EFLAGS: 00000287 RAX: 0000000000000287 RBX: ffffffff8118eb20 RCX: 000000000000c144 RDX: 00000000c144c144 RSI: 0000000000000287 RDI: 0000000000000287 RBP: ffff88bc2e6a7d50 R08: 0000000000000287 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8118f44c R13: ffff88bc2e6a7cf8 R14: ffff8a403ffda000 R15: ffff8a403ffda000 FS: 00002b8d4d1ee5c0(0000) GS:ffff885ec0900000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000013030f0 CR3: 000000a8b0bbe000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffffea0803935f00 ffff88bc2e6a7dd0 ffffffff811f1219 0000000200000006 ffff88509cb4b900 0000000000000006 0000000000001000 ffff88509cb4b840 0000000000000001 0000000100000002 ffff88509cb4b800 0000000000000000 Call Trace: [<ffffffff811f1219>] mem_cgroup_reparent_charges+0x139/0x3c0 [<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360 [<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40 [<ffffffff8120a278>] vfs_rmdir+0xa8/0x100 [<ffffffff8120f315>] do_rmdir+0x1a5/0x200 [<ffffffff8168b5b0>] ? __schedule+0x3b0/0x990 [<ffffffff812104c6>] SyS_rmdir+0x16/0x20 [<ffffffff81696b09>] system_call_fastpath+0x16/0x1b Code: 08 e8 6a 0e a0 ff 5d c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 48 89 f3 0f 1f 44 00 00 66 83 07 02 48 89 df 57 9d <0f> 1f 44 00 00 5b 5d c3 0f 1f 44 00 00 8b 37 f0 66 83 07 02 f6
NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [slurmstepd:193578] Modules linked in: mmfs26(OE) mmfslinux(OE) squashfs loop overlay(T) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core iTCO_wdt intel_powerclamp iTCO_vendor_support coretemp dcdbas intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ghash_clmulni_intel drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf nfit ipmi_msghandler libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci megaraid_sas tg3(OE) libahci ptp pps_core libata [last unloaded: mmfslinux] CPU: 30 PID: 193578 Comm: slurmstepd Tainted: G B OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018 task: ffff88022974cf10 ti: ffff8823c9c0c000 task.ti: ffff8823c9c0c000 RIP: 0010:[<ffffffff811f335b>] [<ffffffff811f335b>] mem_cgroup_reparent_charges+0x15b/0x3c0 RSP: 0018:ffff8823c9c0fd60 EFLAGS: 00000246 RAX: 0000000000000001 RBX: ffffea017a9e7640 RCX: 0000000000000002 RDX: 0000000000000001 RSI: ffffea017506b6dc RDI: 0000000005d41adb RBP: ffff8823c9c0fdd0 R08: ffff885f445fbc00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffea0177754c40 R13: 0000000000000246 R14: ffffffff81190590 R15: 0000000000000000 FS: 00002aaaaaafd180(0000) GS:ffff882f7ddc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002aaaabe2e1a0 CR3: 0000004f0e757000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: 0000000200000001 ffff883163cff500 0000000000000001 0000000000001000 ffff883163cff440 0000000000000001 0000000100000003 ffff883163cff400 0000000000000000 0000000000000000 ffff8831041a8e00 ffff883163cff400 Call Trace: [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff816b0325>] ? do_page_fault+0x35/0x90 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: c6 4c 89 ef e8 17 85 4b 00 48 89 df e8 4f 3b 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 13 25 fa ff 85 c0 0f
NMI watchdog: BUG: soft lockup - CPU#31 stuck for 22s! [slurmstepd:80311] Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass dell_smbios dcdbas sparse_keymap crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci megaraid_sas libahci tg3(OE) ptp libata pps_core [last unloaded: ipmi_msghandler] CPU: 31 PID: 80311 Comm: slurmstepd Tainted: G OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.9.3 09/23/2020 task: ffff8839afe06eb0 ti: ffff8831c479c000 task.ti: ffff8831c479c000 RIP: 0010:[<ffffffff816abbcb>] [<ffffffff816abbcb>] _raw_spin_lock_irqsave+0xb/0x40 RSP: 0018:ffff8831c479fd40 EFLAGS: 00000287 RAX: 006fffff00000000 RBX: ffffffff81190590 RCX: 0000000000000001 RDX: 0000000000000003 RSI: 0000000000000001 RDI: ffff88607bfd7540 RBP: ffff8831c479fd50 R08: ffff882fa8ae3400 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190eac R13: ffff8831c479fcf8 R14: ffff88607bfd7000 R15: ffff88607bfd7000 FS: 00002aaaaaafd180(0000) GS:ffff885fb81c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002aaaab7296d0 CR3: 0000000278f65000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: 0000000000000287 ffffea01105d3f40 ffff8831c479fdd0 ffffffff811f3312 0000000200000001 ffff883225c05900 0000000000000001 0000000000001000 ffff883225c05840 0000000000000001 0000000100000003 ffff883225c05800 Call Trace: [<ffffffff811f3312>] mem_cgroup_reparent_charges+0x112/0x3c0 [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff816b0325>] ? do_page_fault+0x35/0x90 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 61 2a ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 9c <58> 0f 1f 44 00 00 48 89 c3 fa 66 0f 1f 44 00 00 31 c0 ba 01 00
NMI watchdog: BUG: soft lockup - CPU#48 stuck for 22s! [slurmstepd:265028] Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp dell_smbios coretemp dcdbas intel_rapl sparse_keymap iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf nfit ipmi_msghandler libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) megaraid_sas ahci libahci tg3(OE) libata ptp pps_core [last unloaded: tracedev] CPU: 48 PID: 265028 Comm: slurmstepd Tainted: G B OE ------------ 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.8.2 08/31/2020 task: ffff8830b5410000 ti: ffff882172a48000 task.ti: ffff882172a48000 RIP: 0010:[<ffffffff816ab865>] [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20 RSP: 0000:ffff882172a4bd50 EFLAGS: 00000283 RAX: 0000000000000283 RBX: ffffea012a49de80 RCX: 0000000000000001 RDX: 0000000000000101 RSI: 0000000000000283 RDI: 0000000000000283 RBP: ffff882172a4bd50 R08: 0000000000000101 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590 R13: 0000000000000000 R14: ffffffff81190eac R15: ffff882172a4bcf8 FS: 00002aaaaaafd180(0000) GS:ffff882fb9000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002aaac485c118 CR3: 00000010ef72e000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff882172a4bdd0 ffffffff811f3339 0000000200000001 ffff8801b92ec100 0000000000000001 0000000000001000 ffff8801b92ec040 0000000000000001 0000000100000002 ffff8801b92ec000 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f3339>] mem_cgroup_reparent_charges+0x139/0x3c0 [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff816b0325>] ? do_page_fault+0x35/0x90 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
BUG: soft lockup - CPU#1 stuck for 22s! [slurmstepd:17937] Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver intel_powerclamp coretemp intel_rapl kvm_intel kvm ast syscopyarea sysfillrect sysimgblt crc32_pclmul ttm crc32c_intel iTCO_wdt ghash_clmulni_intel iTCO_vendor_support cryptd drm_kms_helper drm mei_me mei shpchp pcspkr lpc_ich ipmi_devintf i2c_i801 wmi mfd_core ipmi_si ipmi_msghandler acpi_pad acpi_power_meter xprtrdma ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss binfmt_misc ip_tables i40e raid1 raid0 bridge stp llc hpilo xfs isci libsas qla3xxx hpsa sg e1000e ixgbe megaraid_sas aacraid aic79xx ata_piix sd_mod crc_t10dif mpt2sas mptspi mptscsih mptbase scsi_transport_spi arcmsr sata_nv sata_svw bnx2x mdio libcrc32c bnx2 ext4 mbcache jbd2 sata_sil tg3 e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun mlx4_en mlx4_ib vxlan ib_sa ip6_udp_tunnel udp_tunnel ib_mad ib_core ib_addr crct10dif_pclmul mlx4_core crct10dif_common igb ahci libahci i2c_algo_bit mpt3sas i2c_core libata dca raid_class ptp scsi_transport_sas pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler] CPU: 1 PID: 17937 Comm: slurmstepd Not tainted 3.10.0-327.3.1.el7.x86_64 #1 Hardware name: SGI.COM CH-C1104-GP2/X10DRU-i+, BIOS 2.0 02/18/2016 task: ffff88712ddca280 ti: ffff883c585c0000 task.ti: ffff883c585c0000 RIP: 0010:[<ffffffff8163cbaa>] [<ffffffff8163cbaa>] _raw_spin_lock_irqsave+0xa/0x60 RSP: 0018:ffff883c585c3d40 EFLAGS: 00000287 RAX: 0000000000000000 RBX: 0000000000000246 RCX: 0000000000000001 RDX: 0000000000000002 RSI: 0000000000001000 RDI: ffff88807ffd7540 RBP: ffff883c585c3d48 R08: ffffc90040d76000 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000001000 R12: 0000000000000000 R13: ffffffff811773b8 R14: ffff883c585c3cf0 R15: ffff88807ffd7000 FS: 00002aaaaaafe540(0000) GS:ffff883f7fc20000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007ffff7ff6000 CR3: 0000007f01fdf000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: 0000000000000287 ffff883c585c3dd0 ffffffff811d1aa2 ffffc9004aacd140 0000000200000001 ffffc9004aacd108 ffffc9004aacd040 0000000000000001 0000000000001000 0000000100000003 0000000000000001 ffffc9004aacd000 Call Trace: [<ffffffff811d1aa2>] mem_cgroup_reparent_charges+0x152/0x440 [<ffffffff811d1f0f>] mem_cgroup_css_offline+0x4f/0x100 [<ffffffff810fa347>] cgroup_destroy_locked+0xe7/0x360 [<ffffffff810fa5e2>] cgroup_rmdir+0x22/0x40 [<ffffffff811ea158>] vfs_rmdir+0xa8/0x100 [<ffffffff811ef0f5>] do_rmdir+0x1a5/0x200 [<ffffffff8163a238>] ? __schedule+0x2d8/0x900 [<ffffffff811f02a6>] SyS_rmdir+0x16/0x20 [<ffffffff816458c9>] system_call_fastpath+0x16/0x1b Code: 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 83 e8 01 75 f1 48 89 df 0f 1f 80 00 00 00 00 eb e0 66 90 0f 1f 44 00 00 55 48 89 e5 9c <58> 0f 1f 44 00 00 49 89 c0 fa 66 0f 1f 44 00 00 ba 00 00 02 00
NMI watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [slurmstepd:371639] Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev dcdbas skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci megaraid_sas libahci libata tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler] CPU: 25 PID: 371639 Comm: slurmstepd Tainted: G W OE ------------ T 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018 task: ffff8855ff3bcf10 ti: ffff88024e1bc000 task.ti: ffff88024e1bc000 RIP: 0010:[<ffffffff816ab865>] [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20 RSP: 0018:ffff88024e1bfd50 EFLAGS: 00000287 RAX: 0000000000000287 RBX: ffffea00dbb30640 RCX: 0000000000000001 RDX: 0000000000000001 RSI: 0000000000000287 RDI: 0000000000000287 RBP: ffff88024e1bfd50 R08: ffff882f71403c00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590 R13: 0000000000000000 R14: ffffffff81190eac R15: ffff88024e1bfcf8 FS: 00002aaaaaafd180(0000) GS:ffff885f7c100000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002aaaac36a0f0 CR3: 000000126f0eb000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff88024e1bfdd0 ffffffff811f3339 0000000200000001 ffff88258af3ed00 0000000000000001 0000000000001000 ffff88258af3ec40 0000000000000001 0000000100000003 ffff88258af3ec00 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f3339>] mem_cgroup_reparent_charges+0x139/0x3c0 [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff816b0325>] ? do_page_fault+0x35/0x90 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
NMI watchdog: BUG: soft lockup - CPU#53 stuck for 23s! [slurmstepd:349322] Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp dcdbas intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm aesni_intel drm_kms_helper lrw gf128mul glue_helper ablk_helper cryptd syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr mei_me mei i2c_i801 shpchp lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) megaraid_sas ahci libahci libata tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler] CPU: 53 PID: 349322 Comm: slurmstepd Tainted: G OE ------------ T 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018 task: ffff88012c2fbf40 ti: ffff8803b0b78000 task.ti: ffff8803b0b78000 RIP: 0010:[<ffffffff816ab865>] [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20 RSP: 0018:ffff8803b0b7bd50 EFLAGS: 00000287 RAX: 0000000000000287 RBX: ffffea0136d07540 RCX: 0000000000000001 RDX: 0000000000000001 RSI: 0000000000000287 RDI: 0000000000000287 RBP: ffff8803b0b7bd50 R08: ffff882f143aac00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590 R13: 0000000000000000 R14: ffffffff81190eac R15: ffff8803b0b7bcf8 FS: 00002aaaaaafd180(0000) GS:ffff885f7c480000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055555557e160 CR3: 0000004de43b8000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff8803b0b7bdd0 ffffffff811f3339 0000000200000001 ffff88024a2c1d00 0000000000000001 0000000000001000 ffff88024a2c1c40 0000000000000001 0000000100000003 ffff88024a2c1c00 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f3339>] mem_cgroup_reparent_charges+0x139/0x3c0 [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff816b0325>] ? do_page_fault+0x35/0x90 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
NMI watchdog: BUG: soft lockup - CPU#45 stuck for 23s! [slurmstepd:95670] Modules linked in: osc(OE) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) 8021q garp mrp stp llc ip6_tables rpcrdma ib_isert iscsi_target_mod amd64_edac_mod edac_mce_amd edac_core kvm_amd ib_iser kvm libiscsi scsi_transport_iscsi ib_srpt irqbypass crc32_pclmul target_core_mod ghash_clmulni_intel aesni_intel lrw dcdbas gf128mul glue_helper ablk_helper ib_srp cryptd scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm pcspkr sg ib_cm ipmi_si ipmi_devintf iw_cm ipmi_msghandler ccp tpm_crb acpi_power_meter ib_core shpchp i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas bnxt_en ptp i2c_core pps_core CPU: 45 PID: 95670 Comm: slurmstepd Tainted: G OEL ------------ 3.10.0-693.5.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R7415/07YXFK, BIOS 1.8.7 04/02/2019 task: ffff8803bcc65ee0 ti: ffff880e5d560000 task.ti: ffff880e5d560000 RIP: 0010:[<ffffffff816ab905>] [<ffffffff816ab905>] _raw_spin_unlock_irqrestore+0x15/0x20 RSP: 0018:ffff880e5d563d50 EFLAGS: 00000287 RAX: 0000000000000287 RBX: ffffea00b48059c0 RCX: 0000000001690000 RDX: 0000000000f90000 RSI: 0000000000000287 RDI: 0000000000000287 RBP: ffff880e5d563d50 R08: ffff883f7d757880 R09: ffff883f7d397880 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff811905c0 R13: 0000000000000000 R14: ffffffff81190edc R15: ffff880e5d563cf8 FS: 00002ac93b0b7fc0(0000) GS:ffff883f7d740000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fcffaa0cfb8 CR3: 0000001d49f3d000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Stack: ffff880e5d563dd0 ffffffff811f32a9 0000000200000000 ffff88267bf04900 0000000000000000 0000000000001000 ffff88267bf04840 0000000000000001 0000000100000002 ffff88267bf04800 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f32a9>] mem_cgroup_reparent_charges+0x139/0x3c0 [<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da12>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150 [<ffffffff81212241>] do_rmdir+0x1f1/0x220 [<ffffffff816a904d>] ? __schedule+0x39d/0x8b0 [<ffffffff81213446>] SyS_rmdir+0x16/0x20 [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48