Info

Permalink
Function
_raw_spin_unlock_irqrestore
First occurrence
2020-09-25
Last occurrence
2020-10-15
State
NEW
Components
kernel
Quality
-17

Statistics

Operating system Count
CentOS 7 210
Architecture Count
x86_64 210
Related packages Count
kernel 209
 0:3.10.0-693.2.2.el7 87
 0:3.10.0-514.10.2.el7 56
 0:3.10.0-862.el7 16
 0:3.10.0-693.5.2.el7 12
 0:3.10.0-957.el7 12
 0:3.10.0-693.el7 10
 0:3.10.0-862.9.1.el7 8
 0:3.10.0-862.6.3.el7 2
 0:3.10.0-327.3.1.el7 1
 0:3.10.0-1160.36.2.el7 1
 0:3.10.0-862.14.4.el7 1
 0:3.10.0-514.21.2.el7 1
 0:3.10.0-693.11.6.el7 1
 0:3.10.0-862.2.3.el7 1
kernel-plus 1
 0:3.10.0-957.el7.centos.plus 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

No reports in the last 20 weeks.

Report backtrace


Complete report #1297839
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffadf6a0f5
Build id: 3.10.0-957.el7.x86_64
0x15
2
mem_cgroup_reparent_charges
vmlinux 0xffffffffada32eb9
Build id: 3.10.0-957.el7.x86_64
0x139
3
mem_cgroup_css_offline
vmlinux 0xffffffffada332f4
Build id: 3.10.0-957.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffffad92573a
Build id: 3.10.0-957.el7.x86_64
0xea
5
cgroup_rmdir
vmlinux 0xffffffffad9259e2
Build id: 3.10.0-957.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffffada4dc2c
Build id: 3.10.0-957.el7.x86_64
0xdc
7
do_rmdir
vmlinux 0xffffffffada530d1
Build id: 3.10.0-957.el7.x86_64
0x1f1
8
? ____fput
vmlinux 0xffffffffada4363e
Build id: 3.10.0-957.el7.x86_64
0xe
9
? task_work_run
vmlinux 0xffffffffad8be7a0
Build id: 3.10.0-957.el7.x86_64
0xc0
10
SyS_rmdir
vmlinux 0xffffffffada54306
Build id: 3.10.0-957.el7.x86_64
0x16
11
system_call_fastpath
vmlinux 0xffffffffadf74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#18 stuck for 23s! [systemd:1]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache snapapi26(OE) iscsi_target_mod target_core_mod macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag binfmt_misc 8021q garp mrp veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) overlay(T) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support dcdbas skx_edac intel_powerclamp
coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr cdc_ether usbnet mii joydev sg mei_me mei lpc_ich i2c_i801 ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad knem(OE) auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 mlx4_ib(OE) mlx4_en(OE) mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common mgag200 crc32c_intel drm_kms_helper mlx4_core(OE) syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx5_core(OE) drm mlxfw(OE) vfio_mdev(OE) vfio_iommu_type1 i40e ahci vfio igb mdev(OE) devlink libahci mlx_compat(OE) megaraid_sas libata drm_panel_orientation_quirks ptp dca i2c_algo_bit pps_core nfit libnvdimm [last unloaded: file_protector]
CPU: 18 PID: 1 Comm: systemd Tainted: G           OE  ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R640/0CRT1G, BIOS 1.2.11 10/19/2017
task: ffff8ce7d3518000 ti: ffff8ce7d3514000 task.ti: ffff8ce7d3514000
RIP: 0010:[<ffffffffadf6a0f5>]  [<ffffffffadf6a0f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8ce7d3517d20  EFLAGS: 00000283
RAX: 0000000000000283 RBX: ffffbc915a1f4b00 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff8ce7d3517d20 R08: ffff8d266ac35400 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffad9c4930
R13: 0000000000000000 R14: ffffffffad9c524c R15: ffff8ce7d3517cc8
FS:  00007f150db84940(0000) GS:ffff8d066d040000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f753dd49830 CR3: 0000003fee17c000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffada32eb9>] mem_cgroup_reparent_charges+0x139/0x3c0
[<ffffffffada332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffad92573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffad9259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffada4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffada530d1>] do_rmdir+0x1f1/0x220
[<ffffffffada4363e>] ? ____fput+0xe/0x10
[<ffffffffad8be7a0>] ? task_work_run+0xc0/0xe0
[<ffffffffada54306>] SyS_rmdir+0x16/0x20
[<ffffffffadf74ddb>] system_call_fastpath+0x22/0x27
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #961754
Frame # Function Binary Source or offset Line
1
put_page
vmlinux 0xffffffffa1ba2989
Build id: 3.10.0-862.6.3.el7.x86_64
0x29
2
mem_cgroup_reparent_charges
vmlinux 0xffffffffa1c0d852
Build id: 3.10.0-862.6.3.el7.x86_64
0x232
3
mem_cgroup_css_offline
vmlinux 0xffffffffa1c0db94
Build id: 3.10.0-862.6.3.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffffa1b1c9ca
Build id: 3.10.0-862.6.3.el7.x86_64
0xea
5
cgroup_rmdir
vmlinux 0xffffffffa1b1cc72
Build id: 3.10.0-862.6.3.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffffa1c29d6c
Build id: 3.10.0-862.6.3.el7.x86_64
0xdc
7
do_rmdir
vmlinux 0xffffffffa1c2d2f1
Build id: 3.10.0-862.6.3.el7.x86_64
0x1f1
8
? ____fput
vmlinux 0xffffffffa1c1d9ae
Build id: 3.10.0-862.6.3.el7.x86_64
0xe
9
? task_work_run
vmlinux 0xffffffffa1ab8040
Build id: 3.10.0-862.6.3.el7.x86_64
0xc0
10
SyS_rmdir
vmlinux 0xffffffffa1c2e526
Build id: 3.10.0-862.6.3.el7.x86_64
0x16
11
system_call_fastpath
vmlinux 0xffffffffa2120795
Build id: 3.10.0-862.6.3.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [systemd:1]
Modules linked in: fuse binfmt_misc uas usb_storage 8021q garp mrp vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bridge stp llc bonding ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ucm i40iw rpcrdma sunrpc rdma_ucm ib_uverbs ib_iser rdma_cm iw_cm libiscsi ib_umad ib_ipoib scsi_transport_iscsi ib_cm iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mlx4_ib mlx5_ib ib_core
pcspkr sg mei_me joydev i2c_i801 mei lpc_ich shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi nfit acpi_cpufreq libnvdimm acpi_power_meter acpi_pad ip_tables xfs libcrc32c mlx4_en sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt mlx5_core fb_sys_fops ttm mlx4_core crct10dif_pclmul i40e crct10dif_common ahci drm crc32c_intel libahci mlxfw megaraid_sas libata devlink ptp pps_core i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 1 Comm: systemd Kdump: loaded Not tainted 3.10.0-862.6.3.el7.x86_64 #1
Hardware name: Supermicro X11DPi-N(T)/X11DPi-N, BIOS 2.0b 02/28/2018
task: ffff975027668000 ti: ffff9710e76dc000 task.ti: ffff9710e76dc000
RIP: 0010:[<ffffffffa1ba2989>]  [<ffffffffa1ba2989>] put_page+0x29/0x50
RSP: 0018:ffff9710e76dfd20  EFLAGS: 00000206
RAX: 006fffff00000848 RBX: fffffe3cc11aee80 RCX: 0000000000000001
RDX: 0000000000000008 RSI: 0000000000000001 RDI: fffffe3cc11a90c0
RBP: ffff9710e76dfd20 R08: ffff978e5c76d400 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa1ba2bc0
R13: 0000000000000000 R14: ffffffffa1ba34dc R15: ffff9710e76dfcc8
FS:  00007fc75db69940(0000) GS:ffff974eff6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055a8fd1071d8 CR3: 0000003f35ca8000 CR4: 00000000007627e0
PKRU: 55555554
Call Trace:
[<ffffffffa1c0d852>] mem_cgroup_reparent_charges+0x232/0x3c0
[<ffffffffa1c0db94>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa1b1c9ca>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa1b1cc72>] cgroup_rmdir+0x22/0x40
[<ffffffffa1c29d6c>] vfs_rmdir+0xdc/0x150
[<ffffffffa1c2d2f1>] do_rmdir+0x1f1/0x220
[<ffffffffa1c1d9ae>] ? ____fput+0xe/0x10
[<ffffffffa1ab8040>] ? task_work_run+0xc0/0xe0
[<ffffffffa1c2e526>] SyS_rmdir+0x16/0x20
[<ffffffffa2120795>] system_call_fastpath+0x1c/0x21
Code: 00 00 0f 1f 44 00 00 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 58 ff fe ff 5d c3 66 0f 1f 44 00 00 f6 c4 c0 75 14 f0 ff 4f 1c <0f> 94 c0 84 c0 66 90 74 e6 e8 99 ff ff ff 5d c3 0f 1f 80 00 00 
Complete report #238971
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff8638b7b5
Build id: 3.10.0-1160.36.2.el7.x86_64
0x15
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff85e3f39b
Build id: 3.10.0-1160.36.2.el7.x86_64
0x30b
3
mem_cgroup_css_offline
vmlinux 0xffffffff85e3f604
Build id: 3.10.0-1160.36.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff85d2b33a
Build id: 3.10.0-1160.36.2.el7.x86_64
0xea
5
cgroup_rmdir
vmlinux 0xffffffff85d2b5e2
Build id: 3.10.0-1160.36.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff85e5bc8c
Build id: 3.10.0-1160.36.2.el7.x86_64
0xdc
7
do_rmdir
vmlinux 0xffffffff85e60911
Build id: 3.10.0-1160.36.2.el7.x86_64
0x1f1
8
? ____fput
vmlinux 0xffffffff85e50a6e
Build id: 3.10.0-1160.36.2.el7.x86_64
0xe
9
? task_work_run
vmlinux 0xffffffff85cc2980
Build id: 3.10.0-1160.36.2.el7.x86_64
0xc0
10
SyS_rmdir
vmlinux 0xffffffff85e61b46
Build id: 3.10.0-1160.36.2.el7.x86_64
0x16
11
system_call_fastpath
vmlinux 0xffffffff86395f92
Build id: 3.10.0-1160.36.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 3354s! [systemd:1]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev coretemp crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi snd_ac97_codec aesni_intel ac97_bus snd_seq lrw gf128mul glue_helper ablk_helper cryptd snd_seq_device vmw_balloon snd_pcm pcspkr joydev snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport
nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix nfit crct10dif_pclmul crct10dif_common drm crc32c_intel libnvdimm libata mptspi serio_raw scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 0 PID: 1 Comm: systemd Kdump: loaded Not tainted 3.10.0-1160.36.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9bdb39e30000 ti: ffff9bdb39dbc000 task.ti: ffff9bdb39dbc000
RIP: 0010:[<ffffffff8638b7b5>]  [<ffffffff8638b7b5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9bdb39dbfd18  EFLAGS: 00000246
RAX: 0000000000000246 RBX: ffff9bdb39dbfcf0 RCX: ffff9bdb39dbffd8
RDX: 0000000000000001 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff9bdb39dbfd18 R08: ffffffff86960ec8 R09: 0000000000000002
R10: ffffffff8695e000 R11: ffff9bdac3497f00 R12: ffff9bdaf45212c0
R13: ffff9bdb3b61acc0 R14: ffff9bdaf45212c0 R15: ffff9bdaf45212c0
FS:  00007f1252acd940(0000) GS:ffff9bdb3b600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000014c6bcc CR3: 00000000785b0000 CR4: 00000000003607f0
Call Trace:
[<ffffffff85e3f39b>] mem_cgroup_reparent_charges+0x30b/0x3c0
[<ffffffff85e3f604>] mem_cgroup_css_offline+0x84/0x170
[<ffffffff85d2b33a>] cgroup_destroy_locked+0xea/0x370
[<ffffffff85d2b5e2>] cgroup_rmdir+0x22/0x40
[<ffffffff85e5bc8c>] vfs_rmdir+0xdc/0x150
[<ffffffff85e60911>] do_rmdir+0x1f1/0x220
[<ffffffff85e50a6e>] ? ____fput+0xe/0x10
[<ffffffff85cc2980>] ? task_work_run+0xc0/0xe0
[<ffffffff85e61b46>] SyS_rmdir+0x16/0x20
[<ffffffff86395f92>] system_call_fastpath+0x25/0x2a
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #1415770
Frame # Function Binary Source or offset Line
1
put_page
vmlinux 0xffffffff81190359
Build id: 3.10.0-693.2.2.el7.x86_64
0x29
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f3432
Build id: 3.10.0-693.2.2.el7.x86_64
0x232
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f3774
Build id: 3.10.0-693.2.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.2.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120f636
Build id: 3.10.0-693.2.2.el7.x86_64
0xd6
7
do_rmdir
vmlinux 0xffffffff812122d1
Build id: 3.10.0-693.2.2.el7.x86_64
0x1f1
8
? do_page_fault
vmlinux 0xffffffff816b0325
Build id: 3.10.0-693.2.2.el7.x86_64
0x35
9
SyS_rmdir
vmlinux 0xffffffff812134d6
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#48 stuck for 22s! [slurmstepd:84268]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver dm_mirror dm_region_hash dm_log joydev iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ghash_clmulni_intel drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sr_mod cdrom mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio
dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) megaraid_sas aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil tg3(OE) e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci igb(OE) i2c_algo_bit libata i2c_core dca i40e(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 48 PID: 84268 Comm: slurmstepd Tainted: G        W  OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: FUJITSU PRIMERGY RX2540 M5/D3384-B1, BIOS V5.0.0.14 R1.15.0 for D3384-B1x                    10/17/2019
task: ffff88307508eeb0 ti: ffff883a18694000 task.ti: ffff883a18694000
RIP: 0010:[<ffffffff81190359>]  [<ffffffff81190359>] put_page+0x29/0x50
RSP: 0018:ffff883a18697d50  EFLAGS: 00000202
RAX: 00efffff0000004c RBX: ffffea015b2ac000 RCX: 0000000000000001
RDX: 000000000000000b RSI: 0000000000000001 RDI: ffffea013b720500
RBP: ffff883a18697d50 R08: ffff885fdc297c00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590
R13: 0000000000000000 R14: ffffffff81190eac R15: ffff883a18697cf8
FS:  00002aaaaaafd180(0000) GS:ffff882fdf380000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002ab03ccef000 CR3: 0000001847b2e000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff883a18697dd0 ffffffff811f3432 0000000200000003 ffff885c77765d00
0000000000000003 0000000000001000 ffff885c77765c40 0000000000000001
0000000100000003 ffff885c77765c00 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f3432>] mem_cgroup_reparent_charges+0x232/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 00 00 0f 1f 44 00 00 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 18 0c ff ff 5d c3 66 0f 1f 44 00 00 f6 c4 c0 75 14 f0 ff 4f 1c <0f> 94 c0 84 c0 66 90 74 e6 e8 99 ff ff ff 5d c3 0f 1f 80 00 00 
Complete report #1148785
Frame # Function Binary Source or offset Line
1
put_page
vmlinux 0xffffffff81190359
Build id: 3.10.0-693.2.2.el7.x86_64
0x29
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f3432
Build id: 3.10.0-693.2.2.el7.x86_64
0x232
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f3774
Build id: 3.10.0-693.2.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.2.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120f636
Build id: 3.10.0-693.2.2.el7.x86_64
0xd6
7
do_rmdir
vmlinux 0xffffffff812122d1
Build id: 3.10.0-693.2.2.el7.x86_64
0x1f1
8
SyS_rmdir
vmlinux 0xffffffff812134d6
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#63 stuck for 23s! [slurmstepd:392445]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crc32_pclmul dell_smbios crc32c_intel dcdbas sparse_keymap ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops aesni_intel lrw gf128mul drm glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data
dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa sg e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix sd_mod crc_t10dif mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun crct10dif_pclmul crct10dif_common ahci libahci libata megaraid_sas i40e(OE) tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 63 PID: 392445 Comm: slurmstepd Tainted: G           OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.21.1 03/07/2024
task: ffff884b0d060fd0 ti: ffff8801b1a00000 task.ti: ffff8801b1a00000
RIP: 0010:[<ffffffff81190359>]  [<ffffffff81190359>] put_page+0x29/0x50
RSP: 0018:ffff8801b1a03d50  EFLAGS: 00000202
RAX: 006fffff0000006c RBX: ffffea00dd311300 RCX: ffff8801b1a03c58
RDX: 0000000000000000 RSI: 000000000000000e RDI: ffffea00dd311300
RBP: ffff8801b1a03d50 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590
R13: 0000000000000000 R14: ffffffff81190eac R15: ffff8801b1a03cf8
FS:  00002aaaaaafd180(0000) GS:ffff885f7c5c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000069c3260 CR3: 0000000bc1d8e000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8801b1a03dd0 ffffffff811f3432 0000000200000001 ffff880f1f945900
0000000000000001 0000000000001000 ffff880f1f945840 0000000000000001
0000000100000003 ffff880f1f945800 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f3432>] mem_cgroup_reparent_charges+0x232/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 00 00 0f 1f 44 00 00 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 18 0c ff ff 5d c3 66 0f 1f 44 00 00 f6 c4 c0 75 14 f0 ff 4f 1c <0f> 94 c0 84 c0 66 90 74 e6 e8 99 ff ff ff 5d c3 0f 1f 80 00 00 
Complete report #1063040
Frame # Function Binary Source or offset Line
1
put_page
vmlinux 0xffffffff81190389
Build id: 3.10.0-693.5.2.el7.x86_64
0x29
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f33a2
Build id: 3.10.0-693.5.2.el7.x86_64
0x232
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f36e4
Build id: 3.10.0-693.5.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d767
Build id: 3.10.0-693.5.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110da12
Build id: 3.10.0-693.5.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120f5a6
Build id: 3.10.0-693.5.2.el7.x86_64
0xd6
7
do_rmdir
vmlinux 0xffffffff81212241
Build id: 3.10.0-693.5.2.el7.x86_64
0x1f1
8
SyS_rmdir
vmlinux 0xffffffff81213446
Build id: 3.10.0-693.5.2.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffff816b5089
Build id: 3.10.0-693.5.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [slurmstepd:119171]
Modules linked in: squashfs loop overlay(T) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache panfs(POE) 8021q garp mrp stp llc nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) nvidia(POE) mlx5_fpga_tools(OE) ip6_tables iptable_filter dell_smbios sparse_keymap dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr shpchp i2c_piix4 ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter tpm_crb mlx5_ib(OE) mlx4_ib(OE) ib_uverbs(OE) mlx4_en(OE) mlx5_core(OE) nfsd mlxfw(OE) vfio_mdev(OE) binfmt_misc vfio_iommu_type1 vfio auth_rpcgss mdev(OE) nfs_acl ptp lockd pps_core grace mlx4_core(OE) sunrpc devlink knem(OE) ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200
i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci crct10dif_pclmul drm crct10dif_common libahci crc32c_intel libata bnxt_en i2c_core megaraid_sas wmi ib_ipoib(OE) ib_cm(OE) ib_umad(OE) ib_core(OE) mlx_compat(OE)
CPU: 0 PID: 119171 Comm: slurmstepd Tainted: P        W  OE  ------------ T 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R7525/0H3K7P, BIOS 2.8.4 06/23/2022
task: ffff893590ad8000 ti: ffff897679e24000 task.ti: ffff897679e24000
RIP: 0010:[<ffffffff81190389>]  [<ffffffff81190389>] put_page+0x29/0x50
RSP: 0018:ffff897679e27d50  EFLAGS: 00000202
RAX: 006fffff0000000c RBX: ffffea051fc8dd80 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffea051fc8de00
RBP: ffff897679e27d50 R08: ffff88763ff98c00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff811905c0
R13: 0000000000000000 R14: ffffffff81190edc R15: ffff897679e27cf8
FS:  00002af80d423300(0000) GS:ffff887e7e200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000425bd0 CR3: 0000017dbae5c000 CR4: 00000000003406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff897679e27dd0 ffffffff811f33a2 0000000200000001 ffff887dff7bc900
0000000000000001 0000000000001000 ffff887dff7bc840 0000000000000001
0000000100000002 ffff887dff7bc800 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f33a2>] mem_cgroup_reparent_charges+0x232/0x3c0
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff81213446>] SyS_rmdir+0x16/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 00 00 66 66 66 66 90 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 f8 0b ff ff 5d c3 66 0f 1f 44 00 00 f6 c4 c0 75 14 f0 ff 4f 1c <0f> 94 c0 84 c0 66 90 74 e6 e8 99 ff ff ff 5d c3 0f 1f 80 00 00 
Complete report #686057
Frame # Function Binary Source or offset Line
1
put_page
vmlinux 0xffffffffa2da21c9
Build id: 3.10.0-862.el7.x86_64
0x29
2
mem_cgroup_reparent_charges
vmlinux 0xffffffffa2e0d092
Build id: 3.10.0-862.el7.x86_64
0x232
3
mem_cgroup_css_offline
vmlinux 0xffffffffa2e0d3d4
Build id: 3.10.0-862.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffffa2d1c1da
Build id: 3.10.0-862.el7.x86_64
0xea
5
cgroup_rmdir
vmlinux 0xffffffffa2d1c482
Build id: 3.10.0-862.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffffa2e295ac
Build id: 3.10.0-862.el7.x86_64
0xdc
7
do_rmdir
vmlinux 0xffffffffa2e2cb31
Build id: 3.10.0-862.el7.x86_64
0x1f1
8
SyS_rmdir
vmlinux 0xffffffffa2e2dd66
Build id: 3.10.0-862.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffffa331f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#39 stuck for 23s! [slurmstepd:273709]
Modules linked in: binfmt_misc nvidia_uvm(OE) nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 devlink tun overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi ext4 kvm_intel kvm mbcache jbd2 irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
ablk_helper cryptd snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq ipmi_ssif joydev snd_seq_device pcspkr snd_pcm sg snd_timer snd mei_me mei lpc_ich i2c_i801 soundcore shpchp ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_cpufreq acpi_pad acpi_power_meter ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul ttm crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ahci megaraid_sas drm ixgbe libahci libata mdio ptp i2c_core pps_core dca wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 39 PID: 273709 Comm: slurmstepd Kdump: loaded Tainted: P           OEL ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: Supermicro SYS-7049GP-TRT/X11DPG-QT, BIOS 3.1 05/23/2019
task: ffff8e3276a5eeb0 ti: ffff8e2951388000 task.ti: ffff8e2951388000
RIP: 0010:[<ffffffffa2da21c9>]  [<ffffffffa2da21c9>] put_page+0x29/0x50
RSP: 0018:ffff8e295138bd20  EFLAGS: 00000202
RAX: 006fffff0002000c RBX: ffffd80ca555c040 RCX: 0000000000000001
RDX: 000000000000000a RSI: 0000000000000001 RDI: ffffd80cc7c8c480
RBP: ffff8e295138bd20 R08: ffff8e3abe3e6c00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa2da2400
R13: 0000000000000000 R14: ffffffffa2da2d1c R15: ffff8e295138bcc8
FS:  00002b3c18355580(0000) GS:ffff8e783c2c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6a51465000 CR3: 00000070045c4000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa2e0d092>] mem_cgroup_reparent_charges+0x232/0x3c0
[<ffffffffa2e0d3d4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa2d1c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa2d1c482>] cgroup_rmdir+0x22/0x40
[<ffffffffa2e295ac>] vfs_rmdir+0xdc/0x150
[<ffffffffa2e2cb31>] do_rmdir+0x1f1/0x220
[<ffffffffa2e2dd66>] SyS_rmdir+0x16/0x20
[<ffffffffa331f7d5>] system_call_fastpath+0x1c/0x21
Code: 00 00 0f 1f 44 00 00 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 58 ff fe ff 5d c3 66 0f 1f 44 00 00 f6 c4 c0 75 14 f0 ff 4f 1c <0f> 94 c0 84 c0 66 90 74 e6 e8 99 ff ff ff 5d c3 0f 1f 80 00 00 
Complete report #162385
Frame # Function Binary Source or offset Line
1
put_page
vmlinux 0xffffffff81190359
Build id: 3.10.0-693.2.2.el7.x86_64
0x29
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f3432
Build id: 3.10.0-693.2.2.el7.x86_64
0x232
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f3774
Build id: 3.10.0-693.2.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.2.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120f636
Build id: 3.10.0-693.2.2.el7.x86_64
0xd6
7
do_rmdir
vmlinux 0xffffffff812122d1
Build id: 3.10.0-693.2.2.el7.x86_64
0x1f1
8
? do_page_fault
vmlinux 0xffffffff816b0325
Build id: 3.10.0-693.2.2.el7.x86_64
0x35
9
SyS_rmdir
vmlinux 0xffffffff812134d6
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [slurmstepd:204817]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) squashfs loop overlay(T) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev dcdbas skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel iTCO_wdt iTCO_vendor_support ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc
xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) megaraid_sas ahci libahci tg3(OE) libata ptp pps_core [last unloaded: tracedev]
CPU: 17 PID: 204817 Comm: slurmstepd Tainted: G    B      OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018
task: ffff884d85208fd0 ti: ffff8801ef720000 task.ti: ffff8801ef720000
RIP: 0010:[<ffffffff81190359>]  [<ffffffff81190359>] put_page+0x29/0x50
RSP: 0000:ffff8801ef723d50  EFLAGS: 00000202
RAX: 006fffff0000000c RBX: ffffea014f41bf80 RCX: 0000000000000001
RDX: 0000000000000009 RSI: 0000000000000001 RDI: ffffea00c9f5e780
RBP: ffff8801ef723d50 R08: ffff885ed173dc00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590
R13: 0000000000000000 R14: ffffffff81190eac R15: ffff8801ef723cf8
FS:  00002aaaaaafd180(0000) GS:ffff885f7c000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaabe75a000 CR3: 00000001dddb7000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8801ef723dd0 ffffffff811f3432 0000000200000001 ffff8857f6fcd900
0000000000000001 0000000000001000 ffff8857f6fcd840 0000000000000001
0000000100000002 ffff8857f6fcd800 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f3432>] mem_cgroup_reparent_charges+0x232/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 00 00 0f 1f 44 00 00 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 18 0c ff ff 5d c3 66 0f 1f 44 00 00 f6 c4 c0 75 14 f0 ff 4f 1c <0f> 94 c0 84 c0 66 90 74 e6 e8 99 ff ff ff 5d c3 0f 1f 80 00 00 
Complete report #73712
Frame # Function Binary Source or offset Line
1
put_page
vmlinux 0xffffffff81190350
Build id: 3.10.0-693.2.2.el7.x86_64
0x20
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f3432
Build id: 3.10.0-693.2.2.el7.x86_64
0x232
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f3774
Build id: 3.10.0-693.2.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.2.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120f636
Build id: 3.10.0-693.2.2.el7.x86_64
0xd6
7
do_rmdir
vmlinux 0xffffffff812122d1
Build id: 3.10.0-693.2.2.el7.x86_64
0x1f1
8
? do_page_fault
vmlinux 0xffffffff816b0325
Build id: 3.10.0-693.2.2.el7.x86_64
0x35
9
SyS_rmdir
vmlinux 0xffffffff812134d6
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#39 stuck for 22s! [slurmstepd:60168]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log dcdbas skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi iTCO_wdt iTCO_vendor_support kvm_intel kvm irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci
libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci libahci megaraid_sas libata tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 39 PID: 60168 Comm: slurmstepd Tainted: G           OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018
task: ffff88316a61bf40 ti: ffff88310f804000 task.ti: ffff88310f804000
RIP: 0010:[<ffffffff81190350>]  [<ffffffff81190350>] put_page+0x20/0x50
RSP: 0018:ffff88310f807d50  EFLAGS: 00000246
RAX: 006fffff00000068 RBX: ffffea0161ef8440 RCX: ffff88310f807c58
RDX: 0000000000000000 RSI: 000000000000000e RDI: ffffea0161ef8440
RBP: ffff88310f807d50 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590
R13: 0000000000000000 R14: ffffffff81190eac R15: ffff88310f807cf8
FS:  00002aaaaaafd180(0000) GS:ffff885f7c2c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffff7ff5000 CR3: 000000012fb12000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff88310f807dd0 ffffffff811f3432 0000000200000001 ffff884a01437500
0000000000000001 0000000000001000 ffff884a01437440 0000000000000001
0000000100000003 ffff884a01437400 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f3432>] mem_cgroup_reparent_charges+0x232/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 07 55 48 89 e5 a9 00 00 00 80 74 0d e8 18 0c ff ff 5d c3 66 0f 1f 44 00 00 <f6> c4 c0 75 14 f0 ff 4f 1c 0f 94 c0 84 c0 66 90 74 e6 e8 99 ff 
Complete report #1413180
Frame # Function Binary Source or offset Line
1
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f334b
Build id: 3.10.0-693.el7.x86_64
0x15b
2
mem_cgroup_css_offline
vmlinux 0xffffffff811f3764
Build id: 3.10.0-693.el7.x86_64
0x84
3
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.el7.x86_64
0xe7
4
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.el7.x86_64
0x22
5
vfs_rmdir
vmlinux 0xffffffff8120f626
Build id: 3.10.0-693.el7.x86_64
0xd6
6
do_rmdir
vmlinux 0xffffffff81212251
Build id: 3.10.0-693.el7.x86_64
0x1f1
7
? task_work_run
vmlinux 0xffffffff810ad24c
Build id: 3.10.0-693.el7.x86_64
0xac
8
SyS_rmdir
vmlinux 0xffffffff81213456
Build id: 3.10.0-693.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#28 stuck for 23s! [systemd:1]
Modules linked in: veth nvidia_uvm(OE) tcp_diag inet_diag xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc fuse overlay(T) dm_mirror dm_region_hash dm_log dm_mod ib_isert iscsi_target_mod nvidia_drm(POE) nvidia_modeset(POE) ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt iTCO_wdt iTCO_vendor_support ib_ucm rpcrdma rdma_ucm ib_uverbs skx_edac edac_core ib_iser coretemp intel_rapl ib_umad rdma_cm iosf_mbi ib_ipoib kvm_intel iw_cm i40iw kvm irqbypass crc32_pclmul ghash_clmulni_intel libiscsi aesni_intel lrw scsi_transport_iscsi gf128mul ib_cm glue_helper ablk_helper cryptd nvidia(POE) pcspkr joydev sg mei_me mei i2c_i801 shpchp lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm mlx5_ib acpi_power_meter ib_core auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea crc32c_intel mlx5_core sysfillrect sysimgblt fb_sys_fops ttm i40e nvme drm nvme_core ahci libahci devlink libata ptp pps_core i2c_core wmi
CPU: 28 PID: 1 Comm: systemd Tainted: P           OE  ------------ T 3.10.0-693.el7.x86_64 #1
NMI watchdog: BUG: soft lockup - CPU#28 stuck for 23s! [systemd:1]
Hardware name: Inspur NF5468M5/YZMB-01130-10C, BIOS 4.1.14 12/23/2020
task: ffff88012ef30000 ti: ffff88012ef38000 task.ti: ffff88012ef38000
RIP: 0010:[<ffffffff811f334b>]  [<ffffffff811f334b>] mem_cgroup_reparent_charges+0x15b/0x3c0
RSP: 0018:ffff88012ef3bd60  EFLAGS: 00000246
RAX: 0000000000000002 RBX: ffffea01c4e6e640 RCX: 0000000000000003
RDX: 0000000000000002 RSI: ffffea01f4368e5c RDI: 0000000007d0da39
RBP: ffff88012ef3bdd0 R08: ffff885eb6e62000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffea01e5bf9200
R13: 0000000000000246 R14: ffffffff81190590 R15: 0000000000000000
FS:  00007fac5fcac940(0000) GS:ffff88bebdc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00112a000 CR3: 000000bea7a16000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
 0000000200000001 ffff885eafb3f900 0000000000000001 0000000000001000
 ffff885eafb3f840 0000000000000001 0000000100000002 ffff885eafb3f800
 0000000000000000 0000000000000000 ffff885e99ecd000 ffff885eafb3f800
Call Trace:
 [<ffffffff811f3764>] mem_cgroup_css_offline+0x84/0x140
 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
 [<ffffffff8120f626>] vfs_rmdir+0xd6/0x150
 [<ffffffff81212251>] do_rmdir+0x1f1/0x220
 [<ffffffff810ad24c>] ? task_work_run+0xac/0xf0
 [<ffffffff81213456>] SyS_rmdir+0x16/0x20
 [<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: c6 4c 89 ef e8 07 85 4b 00 48 89 df e8 4f 3b 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 23 25 fa ff 85 c0 0f 
Complete report #1403748
Frame # Function Binary Source or offset Line
1
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f34f6
Build id: 3.10.0-693.2.2.el7.x86_64
0x2f6
2
mem_cgroup_css_offline
vmlinux 0xffffffff811f3774
Build id: 3.10.0-693.2.2.el7.x86_64
0x84
3
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.2.2.el7.x86_64
0xe7
4
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
5
vfs_rmdir
vmlinux 0xffffffff8120f636
Build id: 3.10.0-693.2.2.el7.x86_64
0xd6
6
do_rmdir
vmlinux 0xffffffff812122d1
Build id: 3.10.0-693.2.2.el7.x86_64
0x1f1
7
SyS_rmdir
vmlinux 0xffffffff812134d6
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
8
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [slurmstepd:342365]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss dell_rbu nfsv4 dns_resolver dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt irqbypass iTCO_vendor_support dell_smbios dcdbas sparse_keymap crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data
dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci libata i40e(OE) megaraid_sas tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 3 PID: 342365 Comm: slurmstepd Tainted: G           OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.21.1 03/07/2024
task: ffff881d2af78fd0 ti: ffff883f953d4000 task.ti: ffff883f953d4000
RIP: 0010:[<ffffffff811f34f6>]  [<ffffffff811f34f6>] mem_cgroup_reparent_charges+0x2f6/0x3c0
RSP: 0018:ffff883f953d7d60  EFLAGS: 00000206
RAX: 0000000000000001 RBX: ffffea015a5f9f00 RCX: 0000000000000001
RDX: 00000000148a9849 RSI: 0000000000000001 RDI: ffff883b64084d00
RBP: ffff883f953d7dd0 R08: ffff880bcc8ecc00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffea013d90aa80
R13: 0000000000000246 R14: ffffffff81190590 R15: 0000000000000000
FS:  00002aaaaaafd180(0000) GS:ffff885f7be40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaab400c09c CR3: 0000005142ec8000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000200000001 ffff883b64084d00 0000000000000001 0000000000001000
ffff883b64084c40 0000000000000001 0000000100000003 ffff883b64084c00
0000000000000000 0000000000000000 ffff884938674c00 ffff883b64084c00
Call Trace:
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 89 45 b8 e8 7d bc ff ff 8b 05 c7 06 93 00 44 8b 55 c4 85 c0 0f 84 26 ff ff ff 48 8b 75 b8 48 8b 7d 98 e8 5e bc ff ff 44 8b 55 c4 <e9> 10 ff ff ff 0f 1f 44 00 00 48 89 c6 4c 89 ef e8 45 83 4b 00 
Complete report #1147679
Frame # Function Binary Source or offset Line
1
mem_cgroup_reparent_charges
vmlinux 0xffffffff86e0cf5b
Build id: 3.10.0-862.2.3.el7.x86_64
0x15b
2
mem_cgroup_css_offline
vmlinux 0xffffffff86e0d374
Build id: 3.10.0-862.2.3.el7.x86_64
0x84
3
cgroup_destroy_locked
vmlinux 0xffffffff86d1c1da
Build id: 3.10.0-862.2.3.el7.x86_64
0xea
4
cgroup_rmdir
vmlinux 0xffffffff86d1c482
Build id: 3.10.0-862.2.3.el7.x86_64
0x22
5
vfs_rmdir
vmlinux 0xffffffff86e2954c
Build id: 3.10.0-862.2.3.el7.x86_64
0xdc
6
do_rmdir
vmlinux 0xffffffff86e2cad1
Build id: 3.10.0-862.2.3.el7.x86_64
0x1f1
7
? ____fput
vmlinux 0xffffffff86e1d18e
Build id: 3.10.0-862.2.3.el7.x86_64
0xe
8
SyS_rmdir
vmlinux 0xffffffff86e2dd06
Build id: 3.10.0-862.2.3.el7.x86_64
0x16
9
tracesys
vmlinux 0xffffffff8731fa11
Build id: 3.10.0-862.2.3.el7.x86_64
0x9d
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [systemd:1]
Modules linked in: joydev vhost_net vhost macvtap macvlan ebt_arp tun fuse geneve ip6_udp_tunnel udp_tunnel bnx2fc cnic uio fcoe libfcoe libfc scsi_transport_fc scsi_tgt scsi_transport_iscsi nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack openvswitch ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 dm_mirror dm_region_hash dm_log dm_service_time iTCO_wdt iTCO_vendor_support dm_multipath skx_edac dell_smbios
intel_powerclamp dcdbas sparse_keymap coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_mod pcspkr sg mei_me mei lpc_ich i2c_i801 shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect crct10dif_pclmul sysimgblt crct10dif_common fb_sys_fops crc32c_intel ttm ahci drm tg3 libahci bnxt_en libata megaraid_sas ptp devlink i2c_core pps_core 8021q garp mrp bridge stp llc bonding
CPU: 13 PID: 1 Comm: systemd Kdump: loaded Tainted: G        W      ------------   3.10.0-862.2.3.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R640/06NR82, BIOS 2.4.8 11/26/2019
task: ffff8b2867718000 ti: ffff8ae8e7288000 task.ti: ffff8ae8e7288000
RIP: 0010:[<ffffffff86e0cf5b>]  [<ffffffff86e0cf5b>] mem_cgroup_reparent_charges+0x15b/0x3c0
RSP: 0018:ffff8ae8e728bd30  EFLAGS: 00000246
RAX: 0000000000000002 RBX: fffffbe2815f3740 RCX: 0000000000000003
RDX: 0000000000000002 RSI: fffffbe323c925dc RDI: 00000000078f2497
RBP: ffff8ae8e728bda0 R08: ffff8ae93fdce400 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: fffffbe322b06300
R13: 0000000000000246 R14: ffffffff86da23a0 R15: 0000000000000000
FS:  00007f6e5ca5a940(0000) GS:ffff8b673bf80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055b6d3410438 CR3: 0000007f7d130000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff86e0d374>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff86d1c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffff86d1c482>] cgroup_rmdir+0x22/0x40
[<ffffffff86e2954c>] vfs_rmdir+0xdc/0x150
[<ffffffff86e2cad1>] do_rmdir+0x1f1/0x220
[<ffffffff86e1d18e>] ? ____fput+0xe/0x10
[<ffffffff86e2dd06>] SyS_rmdir+0x16/0x20
[<ffffffff8731fa11>] tracesys+0x9d/0xc3
Code: c6 4c 89 ef e8 57 84 50 00 48 89 df e8 9f 3c 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 a3 a7 f9 ff 85 c0 0f 
Complete report #1353002
Frame # Function Binary Source or offset Line
1
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f335b
Build id: 3.10.0-693.2.2.el7.x86_64
0x15b
2
mem_cgroup_css_offline
vmlinux 0xffffffff811f3774
Build id: 3.10.0-693.2.2.el7.x86_64
0x84
3
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.2.2.el7.x86_64
0xe7
4
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
5
vfs_rmdir
vmlinux 0xffffffff8120f636
Build id: 3.10.0-693.2.2.el7.x86_64
0xd6
6
do_rmdir
vmlinux 0xffffffff812122d1
Build id: 3.10.0-693.2.2.el7.x86_64
0x1f1
7
? do_page_fault
vmlinux 0xffffffff816b0325
Build id: 3.10.0-693.2.2.el7.x86_64
0x35
8
SyS_rmdir
vmlinux 0xffffffff812134d6
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [slurmstepd:371919]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss nfsv4 dell_rbu dns_resolver dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul crc32c_intel dell_smbios dcdbas sparse_keymap ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect aesni_intel sysimgblt lrw gf128mul fb_sys_fops glue_helper drm ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data
dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci libata i40e(OE) megaraid_sas tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 8 PID: 371919 Comm: slurmstepd Tainted: G        W  OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.21.1 03/07/2024
task: ffff883e8cf41fa0 ti: ffff88321bb24000 task.ti: ffff88321bb24000
RIP: 0010:[<ffffffff811f335b>]  [<ffffffff811f335b>] mem_cgroup_reparent_charges+0x15b/0x3c0
RSP: 0018:ffff88321bb27d60  EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffffea00ec6dbb00 RCX: 0000000000000002
RDX: 0000000000000001 RSI: ffffea0100dc4f1c RDI: 000000000403713c
RBP: ffff88321bb27dd0 R08: ffff885f4ef2ac00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffea00db8b4040
R13: 0000000000000246 R14: ffffffff81190590 R15: 0000000000000000
FS:  00002aaaaaafd180(0000) GS:ffff882f7db00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffff7ff8000 CR3: 0000003219b11000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000200000001 ffff8801e1ed8100 0000000000000001 0000000000001000
ffff8801e1ed8040 0000000000000001 0000000100000002 ffff8801e1ed8000
0000000000000000 0000000000000000 ffff88041a196c00 ffff8801e1ed8000
Call Trace:
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: c6 4c 89 ef e8 17 85 4b 00 48 89 df e8 4f 3b 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 13 25 fa ff 85 c0 0f 
Complete report #1255644
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffbd57c0f5
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x15
2
mem_cgroup_reparent_charges
vmlinux 0xffffffffbd033eb9
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x139
3
mem_cgroup_css_offline
vmlinux 0xffffffffbd0342f4
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffffbcf2673a
Build id: 3.10.0-957.el7.centos.plus.x86_64
0xea
5
cgroup_rmdir
vmlinux 0xffffffffbcf269e2
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffffbd04ec2c
Build id: 3.10.0-957.el7.centos.plus.x86_64
0xdc
7
do_rmdir
vmlinux 0xffffffffbd0540d1
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x1f1
8
SyS_rmdir
vmlinux 0xffffffffbd055306
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffffbd586ddb
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#27 stuck for 23s! [cgclear:12485]
Modules linked in: binfmt_misc nfsv3 nfs_acl rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace tun devlink ip_set nfnetlink bridge stp llc libcrc32c cachefiles fscache dm_mirror dm_region_hash dm_log dm_mod vfat fat iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg i2c_i801 hpilo hpwdt ipmi_si ipmi_devintf lpc_ich ipmi_msghandler pcc_cpufreq ioatdma wmi acpi_power_meter dca auth_rpcgss sunrpc ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 i2c_algo_bit drm_kms_helper syscopyarea tg3 sysfillrect sysimgblt fb_sys_fops ttm ptp ahci pps_core drm libahci libata drm_panel_orientation_quirks
[last unloaded: ip_tables]
CPU: 27 PID: 12485 Comm: cgclear Kdump: loaded Tainted: G               ------------ T 3.10.0-957.el7.centos.plus.x86_64 #1
Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 04/29/2021
task: ffff9bd362815140 ti: ffff9bdda4bc4000 task.ti: ffff9bdda4bc4000
RIP: 0010:[<ffffffffbd57c0f5>]  [<ffffffffbd57c0f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9bdda4bc7d20  EFLAGS: 00000283
RAX: 0000000000000283 RBX: fffff8e857d6b080 RCX: ffff9bdda4bc7c28
RDX: 0000000000000001 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff9bdda4bc7d20 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffbcfc5930
R13: 0000000000000000 R14: ffffffffbcfc624c R15: ffff9bdda4bc7cc8
FS:  00007fa26dc0bb80(0000) GS:ffff9bddbedc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006dee20 CR3: 00000020377e0000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffbd033eb9>] mem_cgroup_reparent_charges+0x139/0x3c0
[<ffffffffbd0342f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffbcf2673a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffbcf269e2>] cgroup_rmdir+0x22/0x40
[<ffffffffbd04ec2c>] vfs_rmdir+0xdc/0x150
[<ffffffffbd0540d1>] do_rmdir+0x1f1/0x220
[<ffffffffbd055306>] SyS_rmdir+0x16/0x20
[<ffffffffbd586ddb>] system_call_fastpath+0x22/0x27
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #1066923
Frame # Function Binary Source or offset Line
1
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f529b
Build id: 3.10.0-693.11.6.el7.x86_64
0x15b
2
mem_cgroup_css_offline
vmlinux 0xffffffff811f56b4
Build id: 3.10.0-693.11.6.el7.x86_64
0x84
3
cgroup_destroy_locked
vmlinux 0xffffffff8110f467
Build id: 3.10.0-693.11.6.el7.x86_64
0xe7
4
cgroup_rmdir
vmlinux 0xffffffff8110f712
Build id: 3.10.0-693.11.6.el7.x86_64
0x22
5
vfs_rmdir
vmlinux 0xffffffff812114f6
Build id: 3.10.0-693.11.6.el7.x86_64
0xd6
6
do_rmdir
vmlinux 0xffffffff81214191
Build id: 3.10.0-693.11.6.el7.x86_64
0x1f1
7
? __do_page_fault
vmlinux 0xffffffff816b3801
Build id: 3.10.0-693.11.6.el7.x86_64
0x171
8
SyS_rmdir
vmlinux 0xffffffff81215396
Build id: 3.10.0-693.11.6.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffff816b89fd
Build id: 3.10.0-693.11.6.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [slurmstepd:85646]
Modules linked in: beegfs(OE) nvidia_uvm(OE) nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) vfio_iommu_type1 vfio nouveau mxm_wmi video xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm i40iw mlx5_ib ib_core dm_mirror dm_region_hash dm_log dm_mod vfat fat iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl
iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev ipmi_ssif pcspkr sg mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper mlx5_core crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect i40e sysimgblt fb_sys_fops ttm drm devlink ahci libahci ptp pps_core libata i2c_core wmi [last unloaded: beegfs]
CPU: 23 PID: 85646 Comm: slurmstepd Tainted: P           OEL ------------   3.10.0-693.11.6.el7.x86_64 #1
Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 3.3 02/21/2020
task: ffff88004aa7bf40 ti: ffff88020097c000 task.ti: ffff88020097c000
RIP: 0010:[<ffffffff811f529b>]  [<ffffffff811f529b>] mem_cgroup_reparent_charges+0x15b/0x3c0
RSP: 0018:ffff88020097fd30  EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffffea013f4deec0 RCX: 0000000000000002
RDX: 0000000000000001 RSI: ffffea01893f059c RDI: 000000000624fc16
RBP: ffff88020097fda0 R08: ffff884df025b000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffea01f7d29540
R13: 0000000000000246 R14: ffffffff811922b0 R15: 0000000000000000
FS:  00002ada079a5440(0000) GS:ffff887f7dac0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002ada0b437ff8 CR3: 0000004148638000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff811f56b4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110f467>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110f712>] cgroup_rmdir+0x22/0x40
[<ffffffff812114f6>] vfs_rmdir+0xd6/0x150
[<ffffffff81214191>] do_rmdir+0x1f1/0x220
[<ffffffff816b3801>] ? __do_page_fault+0x171/0x450
[<ffffffff81215396>] SyS_rmdir+0x16/0x20
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: c6 4c 89 ef e8 87 88 4b 00 48 89 df e8 4f 3b 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 f3 22 fa ff 85 c0 0f 
Complete report #1059716
Frame # Function Binary Source or offset Line
1
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f34c9
Build id: 3.10.0-693.el7.x86_64
0x2d9
2
mem_cgroup_css_offline
vmlinux 0xffffffff811f3764
Build id: 3.10.0-693.el7.x86_64
0x84
3
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.el7.x86_64
0xe7
4
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.el7.x86_64
0x22
5
vfs_rmdir
vmlinux 0xffffffff8120f626
Build id: 3.10.0-693.el7.x86_64
0xd6
6
do_rmdir
vmlinux 0xffffffff81212251
Build id: 3.10.0-693.el7.x86_64
0x1f1
7
? do_page_fault
vmlinux 0xffffffff816b02e5
Build id: 3.10.0-693.el7.x86_64
0x35
8
SyS_rmdir
vmlinux 0xffffffff81213456
Build id: 3.10.0-693.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [slurmstepd:215944]
Modules linked in: mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) 8021q garp mrp stp llc panfs(POE) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ib_isert iscsi_target_mod iTCO_wdt iTCO_vendor_support ib_srpt target_core_mod rpcrdma ib_srp scsi_transport_srp ib_iser scsi_tgt libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm ib_ucm skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm ib_uverbs ib_umad irqbypass crc32_pclmul ghash_clmulni_intel rdma_cm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ib_cm iw_cm pcspkr joydev sg i2c_i801 lpc_ich mei_me mei shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler mlx5_ib ib_core nfit libnvdimm acpi_power_meter acpi_pad nfsd auth_rpcgss
nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ast crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea sysfillrect mlx5_core sysimgblt fb_sys_fops ttm drm igb ahci libahci nvme devlink libata nvme_core dca ptp i2c_algo_bit pps_core i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 30 PID: 215944 Comm: slurmstepd Tainted: P           OE  ------------   3.10.0-693.el7.x86_64 #1
Hardware name: Penguin Computing Relion XE1112/MR91-FS0-ZB, BIOS R03 06/03/2019
task: ffff88bcfa870fd0 ti: ffff88bc79e70000 task.ti: ffff88bc79e70000
RIP: 0010:[<ffffffff811f34c9>]  [<ffffffff811f34c9>] mem_cgroup_reparent_charges+0x2d9/0x3c0
RSP: 0018:ffff88bc79e73d60  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffea0484c7e400 RCX: 0000000000000001
RDX: 00000000083465a8 RSI: 0000000000000001 RDI: ffff897d247bcc40
RBP: ffff88bc79e73dd0 R08: ffff88b7a573dc00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffea0484c7e340
R13: 0000000000000246 R14: ffffffff81190590 R15: 0000000000000000
FS:  00002b238472b340(0000) GS:ffff897d3c180000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f09de87e000 CR3: 000000b9b12fb000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000200000001 ffff897d247bcd00 0000000000000001 0000000000001000
ffff897d247bcc40 0000000000000001 0000000100000002 ffff897d247bcc00
0000000000000000 0000000000000000 ffff8977eeb7ba00 ffff897d247bcc00
Call Trace:
[<ffffffff811f3764>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f626>] vfs_rmdir+0xd6/0x150
[<ffffffff81212251>] do_rmdir+0x1f1/0x220
[<ffffffff816b02e5>] ? do_page_fault+0x35/0x90
[<ffffffff81213456>] SyS_rmdir+0x16/0x20
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 89 c6 e8 8b 83 4b 00 e9 46 fe ff ff 66 0f 1f 44 00 00 48 8b 7d b0 49 63 c3 48 89 c6 48 89 45 b8 e8 7d bc ff ff 8b 05 d7 06 93 00 <44> 8b 55 c4 85 c0 0f 84 26 ff ff ff 48 8b 75 b8 48 8b 7d 98 e8 
Complete report #1057340
Frame # Function Binary Source or offset Line
1
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f123b
Build id: 3.10.0-514.10.2.el7.x86_64
0x15b
2
mem_cgroup_css_offline
vmlinux 0xffffffff811f1654
Build id: 3.10.0-514.10.2.el7.x86_64
0x84
3
cgroup_destroy_locked
vmlinux 0xffffffff8110d257
Build id: 3.10.0-514.10.2.el7.x86_64
0xe7
4
cgroup_rmdir
vmlinux 0xffffffff8110d4f2
Build id: 3.10.0-514.10.2.el7.x86_64
0x22
5
vfs_rmdir
vmlinux 0xffffffff8120a278
Build id: 3.10.0-514.10.2.el7.x86_64
0xa8
6
do_rmdir
vmlinux 0xffffffff8120f315
Build id: 3.10.0-514.10.2.el7.x86_64
0x1a5
7
? do_page_fault
vmlinux 0xffffffff81692305
Build id: 3.10.0-514.10.2.el7.x86_64
0x35
8
SyS_rmdir
vmlinux 0xffffffff812104c6
Build id: 3.10.0-514.10.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [slurmstepd:37077]
Modules linked in: squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic nfs_layout_nfsv41_files crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log ses enclosure sd_mod sg dm_mod intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd uas serio_raw pcspkr usb_storage smartpqi scsi_transport_sas hpilo hpwdt nvme shpchp lpc_ich wmi acpi_power_meter rpcrdma ib_isert iscsi_target_mod target_core_mod crc_t10dif crct10dif_generic ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm
iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ipmi_devintf hfi1(OE) i2c_algo_bit i2c_core bnxt_en crct10dif_pclmul crct10dif_common rdmavt(OE) crc32c_intel ib_core ipmi_si fjes ipmi_msghandler
CPU: 11 PID: 37077 Comm: slurmstepd Tainted: G        W  OEL ------------ T 3.10.0-514.10.2.el7.x86_64 #1
Hardware name: HPE ProLiant DL560 Gen10/ProLiant DL560 Gen10, BIOS U34 03/16/2023
task: ffff88856e3bce70 ti: ffff896d09a14000 task.ti: ffff896d09a14000
RIP: 0010:[<ffffffff811f123b>]  [<ffffffff811f123b>] mem_cgroup_reparent_charges+0x15b/0x3c0
RSP: 0018:ffff896d09a17d60  EFLAGS: 00000246
RAX: 0000000000000002 RBX: ffffea0e393a66c0 RCX: 0000000000000003
RDX: 0000000000000002 RSI: ffffea0e393a67dc RDI: ffffea0e393a67c0
RBP: ffff896d09a17dd0 R08: 0000000000000287 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffea0e393a6540
R13: 0000000000000246 R14: ffffffff8118eb20 R15: 0000000000000000
FS:  00002ada5c8d0300(0000) GS:ffff88fc3fec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff18e219000 CR3: 000003255122e000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000200000003 ffff8b00df5b5100 0000000000000003 0000000000001000
ffff8b00df5b5040 0000000000000001 0000000100000003 ffff8b00df5b5000
0000000000000000 0000000000000000 ffff8b26efbd7a00 ffff8b00df5b5000
Call Trace:
[<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a278>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f315>] do_rmdir+0x1a5/0x200
[<ffffffff81692305>] ? do_page_fault+0x35/0x90
[<ffffffff812104c6>] SyS_rmdir+0x16/0x20
Complete report #922801
Frame # Function Binary Source or offset Line
1
_raw_spin_lock_irqsave
vmlinux 0xffffffff8168decb
Build id: 3.10.0-514.10.2.el7.x86_64
0xb
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f11f2
Build id: 3.10.0-514.10.2.el7.x86_64
0x112
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f1654
Build id: 3.10.0-514.10.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d257
Build id: 3.10.0-514.10.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110d4f2
Build id: 3.10.0-514.10.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120a278
Build id: 3.10.0-514.10.2.el7.x86_64
0xa8
7
do_rmdir
vmlinux 0xffffffff8120f315
Build id: 3.10.0-514.10.2.el7.x86_64
0x1a5
8
? do_page_fault
vmlinux 0xffffffff81692305
Build id: 3.10.0-514.10.2.el7.x86_64
0x35
9
SyS_rmdir
vmlinux 0xffffffff812104c6
Build id: 3.10.0-514.10.2.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff81696b09
Build id: 3.10.0-514.10.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#69 stuck for 22s! [slurmstepd:105299]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic nfs_layout_nfsv41_files crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod ses enclosure sd_mod sg intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd serio_raw pcspkr uas usb_storage smartpqi nvme scsi_transport_sas hpilo hpwdt lpc_ich shpchp wmi acpi_power_meter rpcrdma ib_isert iscsi_target_mod target_core_mod crc_t10dif crct10dif_generic ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm
ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables hfi1(OE) i2c_algo_bit i2c_core crct10dif_pclmul ipmi_devintf rdmavt(OE) crct10dif_common crc32c_intel ib_core bnxt_en ipmi_si fjes ipmi_msghandler
CPU: 69 PID: 105299 Comm: slurmstepd Tainted: G        W  OEL ------------ T 3.10.0-514.10.2.el7.x86_64 #1
Hardware name: HPE ProLiant DL560 Gen10/ProLiant DL560 Gen10, BIOS U34 04/21/2022
task: ffff8ae84aa36dd0 ti: ffff8af378f5c000 task.ti: ffff8af378f5c000
RIP: 0010:[<ffffffff8168decb>]  [<ffffffff8168decb>] _raw_spin_lock_irqsave+0xb/0x60
RSP: 0018:ffff8af378f5fd50  EFLAGS: 00000287
RAX: 0000000000000287 RBX: ffffea0a4510e340 RCX: 0000000000000001
RDX: 0000000000000009 RSI: 0000000000000001 RDI: ffff8b003ffda540
RBP: ffff8af378f5fd50 R08: ffff89fc3df7e000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8118eb20
R13: 0000000000000000 R14: ffffffff8118f44c R15: ffff8af378f5fcf8
FS:  00002b7c2921e300(0000) GS:ffff8bfc3edc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000025d2390 CR3: 000003f5ec361000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8af378f5fdd0 ffffffff811f11f2 0000000200000002 ffff8966ada02d00
0000000000000002 0000000000001000 ffff8966ada02c40 0000000000000001
0000000100000002 ffff8966ada02c00 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f11f2>] mem_cgroup_reparent_charges+0x112/0x3c0
[<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a278>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f315>] do_rmdir+0x1a5/0x200
[<ffffffff81692305>] ? do_page_fault+0x35/0x90
[<ffffffff812104c6>] SyS_rmdir+0x16/0x20
[<ffffffff81696b09>] system_call_fastpath+0x16/0x1b
Code: 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 83 e8 01 75 f1 48 89 df 0f 1f 80 00 00 00 00 eb e0 66 90 0f 1f 44 00 00 55 48 89 e5 9c 58 <0f> 1f 44 00 00 49 89 c0 fa 66 0f 1f 44 00 00 ba 00 00 02 00 f0 
Complete report #729304
Frame # Function Binary Source or offset Line
1
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f32e5
Build id: 3.10.0-693.5.2.el7.x86_64
0x175
2
mem_cgroup_css_offline
vmlinux 0xffffffff811f36e4
Build id: 3.10.0-693.5.2.el7.x86_64
0x84
3
cgroup_destroy_locked
vmlinux 0xffffffff8110d767
Build id: 3.10.0-693.5.2.el7.x86_64
0xe7
4
cgroup_rmdir
vmlinux 0xffffffff8110da12
Build id: 3.10.0-693.5.2.el7.x86_64
0x22
5
vfs_rmdir
vmlinux 0xffffffff8120f5a6
Build id: 3.10.0-693.5.2.el7.x86_64
0xd6
6
do_rmdir
vmlinux 0xffffffff81212241
Build id: 3.10.0-693.5.2.el7.x86_64
0x1f1
7
? __schedule
vmlinux 0xffffffff816a904d
Build id: 3.10.0-693.5.2.el7.x86_64
0x39d
8
SyS_rmdir
vmlinux 0xffffffff81213446
Build id: 3.10.0-693.5.2.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffff816b5089
Build id: 3.10.0-693.5.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#23 stuck for 22s! [slurmstepd:98377]
Modules linked in: 8021q garp mrp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm dm_mirror dm_region_hash dm_log dm_mod skx_edac edac_core intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel hfi1 aesni_intel lrw gf128mul glue_helper ablk_helper cryptd rdmavt ib_core mei_me joydev mei sg pcspkr
i2c_i801 shpchp lpc_ich ipmi_si wmi ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ixgbe ttm crc32c_intel ahci mdio libahci drm ptp libata pps_core i2c_core dca
CPU: 23 PID: 98377 Comm: slurmstepd Tainted: G        W    L ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Advanced Clustering S2600BPB/S2600BPB, BIOS SE5C620.86B.00.01.0009.C2.101920170742 10/19/2017
task: ffff885f62a40000 ti: ffff885f67cb4000 task.ti: ffff885f67cb4000
RIP: 0010:[<ffffffff811f32e5>]  [<ffffffff811f32e5>] mem_cgroup_reparent_charges+0x175/0x3c0
RSP: 0018:ffff885f67cb7d60  EFLAGS: 00000246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffffffffffff83
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff88603ffd6540
RBP: ffff885f67cb7dd0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000001 R12: ffffea0157a08480
R13: 0000000000000246 R14: ffffffff811905c0 R15: 0000000000000000
FS:  00002ad2510cad00(0000) GS:ffff885f7bec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f458d4c7000 CR3: 0000002f65168000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000200000001 ffff882f6d3cf500 0000000000000001 0000000000001000
ffff882f6d3cf440 0000000000000001 0000000100000002 ffff882f6d3cf400
0000000000000000 0000000000000000 ffff882f66eb8c00 ffff882f6d3cf400
Call Trace:
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff816a904d>] ? __schedule+0x39d/0x8b0
[<ffffffff81213446>] SyS_rmdir+0x16/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc 39 c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 d3 25 fa ff 85 c0 0f 85 7b ff ff ff <49> 8b 47 e0 48 c7 45 d0 00 00 00 00 25 00 40 00 00 48 83 f8 01 
Complete report #725600
Frame # Function Binary Source or offset Line
1
_raw_spin_lock_irqsave
vmlinux 0xffffffffa6b16890
Build id: 3.10.0-862.9.1.el7.x86_64
-
2
? mem_cgroup_reparent_charges
vmlinux 0xffffffffa660d732
Build id: 3.10.0-862.9.1.el7.x86_64
0x112
3
mem_cgroup_css_offline
vmlinux 0xffffffffa660db94
Build id: 3.10.0-862.9.1.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffffa651c9ca
Build id: 3.10.0-862.9.1.el7.x86_64
0xea
5
cgroup_rmdir
vmlinux 0xffffffffa651cc72
Build id: 3.10.0-862.9.1.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffffa6629d6c
Build id: 3.10.0-862.9.1.el7.x86_64
0xdc
7
do_rmdir
vmlinux 0xffffffffa662d2f1
Build id: 3.10.0-862.9.1.el7.x86_64
0x1f1
8
SyS_rmdir
vmlinux 0xffffffffa662e526
Build id: 3.10.0-862.9.1.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffffa6b20795
Build id: 3.10.0-862.9.1.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [slurmstepd:264116]
Modules linked in: 8021q garp mrp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 nfs fscache ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ucm rpcrdma rdma_ucm ib_uverbs ib_iser dm_mirror rdma_cm opa_vnic dm_region_hash iw_cm dm_log ib_umad libiscsi dm_mod ib_ipoib scsi_transport_iscsi ib_cm skx_edac intel_powerclamp iTCO_wdt coretemp iTCO_vendor_support intel_rapl iosf_mbi kvm hfi1 irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel rdmavt lrw gf128mul glue_helper ablk_helper cryptd ib_core pcspkr mei_me sg joydev
mei ipmi_si i2c_i801 shpchp lpc_ich wmi ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect crct10dif_pclmul sysimgblt crct10dif_common fb_sys_fops ixgbe crc32c_intel ttm ahci drm mdio libahci ptp libata pps_core i2c_core dca nfit libnvdimm fuse
CPU: 22 PID: 264116 Comm: slurmstepd Kdump: loaded Tainted: G             L ------------   3.10.0-862.9.1.el7.x86_64 #1
Hardware name: Advanced Clustering S2600BPB/S2600BPB, BIOS SE5C620.86B.00.01.0013.C1.030920180427 03/09/2018
task: ffffa0efcc312f70 ti: ffffa0f7fafac000 task.ti: ffffa0f7fafac000
RIP: 0010:[<ffffffffa6b16890>]  [<ffffffffa6b16890>] _raw_spin_lock_irqsave+0x0/0x40
RSP: 0018:ffffa0f7fafafd28  EFLAGS: 00000283
RAX: 006fffff00000000 RBX: fffffdd443dd9a80 RCX: ffffa0f7fafafc28
RDX: 0000000000000000 RSI: 000000000000000e RDI: ffffa120fffd4540
RBP: ffffa0f7fafafda0 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa65a2bc0
R13: 0000000000000000 R14: ffffffffa65a34dc R15: ffffa0f7fafafcc8
FS:  00002b1c38745d40(0000) GS:ffffa1203be80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002ad8d630b0a0 CR3: 000000029f2c6000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa660d732>] ? mem_cgroup_reparent_charges+0x112/0x3c0
[<ffffffffa660db94>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa651c9ca>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa651cc72>] cgroup_rmdir+0x22/0x40
[<ffffffffa6629d6c>] vfs_rmdir+0xdc/0x150
[<ffffffffa662d2f1>] do_rmdir+0x1f1/0x220
[<ffffffffa662e526>] SyS_rmdir+0x16/0x20
[<ffffffffa6b20795>] system_call_fastpath+0x1c/0x21
Code: 02 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 82 2c ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 <0f> 1f 44 00 00 55 48 89 e5 53 9c 58 0f 1f 44 00 00 48 89 c3 fa 
Complete report #646914
Frame # Function Binary Source or offset Line
1
_raw_spin_lock_irqsave
vmlinux 0xffffffff816abc6b
Build id: 3.10.0-693.5.2.el7.x86_64
0xb
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f3282
Build id: 3.10.0-693.5.2.el7.x86_64
0x112
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f36e4
Build id: 3.10.0-693.5.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d767
Build id: 3.10.0-693.5.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110da12
Build id: 3.10.0-693.5.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120f5a6
Build id: 3.10.0-693.5.2.el7.x86_64
0xd6
7
do_rmdir
vmlinux 0xffffffff81212241
Build id: 3.10.0-693.5.2.el7.x86_64
0x1f1
8
SyS_rmdir
vmlinux 0xffffffff81213446
Build id: 3.10.0-693.5.2.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffff816b5089
Build id: 3.10.0-693.5.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [slurmstepd:49637]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge 8021q garp mrp stp llc overlay(T) ip6_tables iTCO_wdt iTCO_vendor_support iptable_filter coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure cdc_ether scsi_transport_sas usbnet joydev mii sg mei_me mei shpchp i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace
binfmt_misc sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw ahci libahci mlx5_core(OE) libata mlxfw(OE) ptp megaraid_sas pps_core auxiliary(OE) mlx_compat(OE) i2c_core devlink [last unloaded: tracedev]
CPU: 24 PID: 49637 Comm: slurmstepd Tainted: G        W  OEL ------------ T 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW, BIOS AFE112A-1.02 05/10/2021
task: ffff88cf781f0000 ti: ffff88c545d64000 task.ti: ffff88c545d64000
RIP: 0010:[<ffffffff816abc6b>]  [<ffffffff816abc6b>] _raw_spin_lock_irqsave+0xb/0x40
RSP: 0018:ffff88c545d67d40  EFLAGS: 00000283
RAX: 002fffff00000000 RBX: ffffffff811905c0 RCX: 0000000000000001
RDX: 0000000000000006 RSI: 0000000000000001 RDI: ffff8a003ffd8540
RBP: ffff88c545d67d50 R08: ffff89efdc63d800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190edc
R13: ffff88c545d67cf8 R14: ffff8a003ffd8000 R15: ffff8a003ffd8000
FS:  00002af6b12d1fc0(0000) GS:ffff89f83e000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3a35e3e000 CR3: 000000c4fe833000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000283 ffffea0774ab6340 ffff88c545d67dd0 ffffffff811f3282
0000000200000000 ffff89efdc639d00 0000000000000000 0000000000001000
ffff89efdc639c40 0000000000000001 0000000100000002 ffff89efdc639c00
Call Trace:
[<ffffffff811f3282>] mem_cgroup_reparent_charges+0x112/0x3c0
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff81213446>] SyS_rmdir+0x16/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 61 2a ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 9c <58> 0f 1f 44 00 00 48 89 c3 fa 66 0f 1f 44 00 00 31 c0 ba 01 00 
Complete report #622335
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff816ab865
Build id: 3.10.0-693.2.2.el7.x86_64
0x15
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f3339
Build id: 3.10.0-693.2.2.el7.x86_64
0x139
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f3774
Build id: 3.10.0-693.2.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.2.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120f636
Build id: 3.10.0-693.2.2.el7.x86_64
0xd6
7
do_rmdir
vmlinux 0xffffffff812122d1
Build id: 3.10.0-693.2.2.el7.x86_64
0x1f1
8
? do_page_fault
vmlinux 0xffffffff816b0325
Build id: 3.10.0-693.2.2.el7.x86_64
0x35
9
SyS_rmdir
vmlinux 0xffffffff812134d6
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [slurmstepd:222001]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sr_mod cdrom mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas
qla3xxx hpsa e1000e ixgbe(OE) megaraid_sas aacraid aic79xx ata_piix sd_mod crc_t10dif mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil tg3(OE) e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sg crct10dif_pclmul crct10dif_common ahci libahci igb(OE) i2c_algo_bit i2c_core i40e(OE) dca ptp libata pps_core [last unloaded: tracedev]
CPU: 4 PID: 222001 Comm: slurmstepd Tainted: G    B   W  OE  ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: FUJITSU PRIMERGY RX2540 M5/D3384-B1, BIOS V5.0.0.14 R1.15.0 for D3384-B1x                    10/17/2019
task: ffff88487c232f70 ti: ffff885fb4718000 task.ti: ffff885fb4718000
RIP: 0010:[<ffffffff816ab865>]  [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff885fb471bd50  EFLAGS: 00000287
RAX: 0000000000000287 RBX: ffffea0176795e40 RCX: ffff885fb471bc58
RDX: 0000000000000001 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff885fb471bd50 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590
R13: 0000000000000000 R14: ffffffff81190eac R15: ffff885fb471bcf8
FS:  00002aaaaaafd180(0000) GS:ffff882fdf040000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aac1f2c6120 CR3: 000000184dbcf000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff885fb471bdd0 ffffffff811f3339 0000000200000003 ffff885dd9c6b100
0000000000000003 0000000000001000 ffff885dd9c6b040 0000000000000001
0000000100000003 ffff885dd9c6b000 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f3339>] mem_cgroup_reparent_charges+0x139/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #558044
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffa5515375
Build id: 3.10.0-862.el7.x86_64
0x15
2
mem_cgroup_reparent_charges
vmlinux 0xffffffffa500cf99
Build id: 3.10.0-862.el7.x86_64
0x139
3
mem_cgroup_css_offline
vmlinux 0xffffffffa500d3d4
Build id: 3.10.0-862.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffffa4f1c1da
Build id: 3.10.0-862.el7.x86_64
0xea
5
cgroup_rmdir
vmlinux 0xffffffffa4f1c482
Build id: 3.10.0-862.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffffa50295ac
Build id: 3.10.0-862.el7.x86_64
0xdc
7
do_rmdir
vmlinux 0xffffffffa502cb31
Build id: 3.10.0-862.el7.x86_64
0x1f1
8
SyS_rmdir
vmlinux 0xffffffffa502dd66
Build id: 3.10.0-862.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffffa551f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#23 stuck for 23s! [slurmstepd:126513]
Modules linked in: nvidia_uvm(OE) nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 devlink tun overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel ext4 lrw gf128mul glue_helper ablk_helper cryptd mbcache
jbd2 snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq ipmi_ssif snd_seq_device pcspkr joydev snd_pcm snd_timer sg snd mei_me mei soundcore i2c_i801 lpc_ich shpchp ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_cpufreq acpi_power_meter acpi_pad ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper ttm crct10dif_pclmul syscopyarea crct10dif_common sysfillrect sysimgblt crc32c_intel fb_sys_fops ixgbe ahci drm megaraid_sas libahci libata mdio ptp i2c_core pps_core dca wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 23 PID: 126513 Comm: slurmstepd Kdump: loaded Tainted: P           OEL ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: Supermicro SYS-7049GP-TRT/X11DPG-QT, BIOS 3.1 05/23/2019
task: ffff88dcc6adcf10 ti: ffff894ceda8c000 task.ti: ffff894ceda8c000
RIP: 0010:[<ffffffffa5515375>]  [<ffffffffa5515375>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff894ceda8fd20  EFLAGS: 00000283
RAX: 0000000000000283 RBX: ffffee473280d840 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff894ceda8fd20 R08: ffff8952b7b34c00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffa4fa2400
R13: 0000000000000000 R14: ffffffffa4fa2d1c R15: ffff894ceda8fcc8
FS:  00002b865a64b580(0000) GS:ffff8952fbec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001c34930 CR3: 0000007f4bbf6000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa500cf99>] mem_cgroup_reparent_charges+0x139/0x3c0
[<ffffffffa500d3d4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffa4f1c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffffa4f1c482>] cgroup_rmdir+0x22/0x40
[<ffffffffa50295ac>] vfs_rmdir+0xdc/0x150
[<ffffffffa502cb31>] do_rmdir+0x1f1/0x220
[<ffffffffa502dd66>] SyS_rmdir+0x16/0x20
[<ffffffffa551f7d5>] system_call_fastpath+0x1c/0x21
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #529081
Frame # Function Binary Source or offset Line
1
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f32cb
Build id: 3.10.0-693.5.2.el7.x86_64
0x15b
2
mem_cgroup_css_offline
vmlinux 0xffffffff811f36e4
Build id: 3.10.0-693.5.2.el7.x86_64
0x84
3
cgroup_destroy_locked
vmlinux 0xffffffff8110d767
Build id: 3.10.0-693.5.2.el7.x86_64
0xe7
4
cgroup_rmdir
vmlinux 0xffffffff8110da12
Build id: 3.10.0-693.5.2.el7.x86_64
0x22
5
vfs_rmdir
vmlinux 0xffffffff8120f5a6
Build id: 3.10.0-693.5.2.el7.x86_64
0xd6
6
do_rmdir
vmlinux 0xffffffff81212241
Build id: 3.10.0-693.5.2.el7.x86_64
0x1f1
7
SyS_rmdir
vmlinux 0xffffffff81213446
Build id: 3.10.0-693.5.2.el7.x86_64
0x16
8
system_call_fastpath
vmlinux 0xffffffff816b5089
Build id: 3.10.0-693.5.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [slurmstepd:34377]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge 8021q garp mrp stp llc overlay(T) ip6_tables iTCO_wdt iTCO_vendor_support iptable_filter coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure cdc_ether scsi_transport_sas usbnet joydev mii sg mei_me mei shpchp i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace
binfmt_misc sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw ahci libahci mlx5_core(OE) libata mlxfw(OE) ptp megaraid_sas pps_core auxiliary(OE) mlx_compat(OE) i2c_core devlink [last unloaded: tracedev]
CPU: 10 PID: 34377 Comm: slurmstepd Tainted: G        W  OEL ------------ T 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW, BIOS AFE112A-1.02 05/10/2021
task: ffff89672f4a8000 ti: ffff89f2f3fe4000 task.ti: ffff89f2f3fe4000
RIP: 0010:[<ffffffff811f32cb>]  [<ffffffff811f32cb>] mem_cgroup_reparent_charges+0x15b/0x3c0
RSP: 0018:ffff89f2f3fe7d60  EFLAGS: 00000246
RAX: 0000000000000002 RBX: ffffea008fe16e40 RCX: 0000000000000003
RDX: 0000000000000002 RSI: ffffea00764a6b9c RDI: 0000000001d929ae
RBP: ffff89f2f3fe7dd0 R08: ffff89efed2b3000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffea062dba8140
R13: 0000000000000246 R14: ffffffff811905c0 R15: 0000000000000000
FS:  00002ab626029fc0(0000) GS:ffff89f83dc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002ab62bf08620 CR3: 00000131456ce000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000200000000 ffff89efed2b7d00 0000000000000000 0000000000001000
ffff89efed2b7c40 0000000000000001 0000000100000002 ffff89efed2b7c00
0000000000000000 0000000000000000 ffff891a47532200 ffff89efed2b7c00
Call Trace:
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff81213446>] SyS_rmdir+0x16/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: c6 4c 89 ef e8 47 86 4b 00 48 89 df e8 4f 3b 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 d3 25 fa ff 85 c0 0f 
Complete report #408874
Frame # Function Binary Source or offset Line
1
_raw_spin_lock_irqsave
vmlinux 0xffffffff8168ea4b
Build id: 3.10.0-514.21.2.el7.x86_64
0xb
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f1832
Build id: 3.10.0-514.21.2.el7.x86_64
0x112
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f1c94
Build id: 3.10.0-514.21.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d6e7
Build id: 3.10.0-514.21.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110d982
Build id: 3.10.0-514.21.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120a8c8
Build id: 3.10.0-514.21.2.el7.x86_64
0xa8
7
do_rmdir
vmlinux 0xffffffff8120f965
Build id: 3.10.0-514.21.2.el7.x86_64
0x1a5
8
? __schedule
vmlinux 0xffffffff8168c130
Build id: 3.10.0-514.21.2.el7.x86_64
0x3b0
9
SyS_rmdir
vmlinux 0xffffffff81210b16
Build id: 3.10.0-514.21.2.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff81697749
Build id: 3.10.0-514.21.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#38 stuck for 23s! [slurmstepd:30810]
Modules linked in: nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) nvidia(POE) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core iTCO_wdt iTCO_vendor_support
intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif pcspkr sg sb_edac edac_core hpilo hpwdt lpc_ich ioatdma shpchp ipmi_si acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c mlx4_en sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul mgag200 crct10dif_common ipmi_msghandler crc32c_intel drm_kms_helper mlx4_core syscopyarea sysfillrect serio_raw sysimgblt fb_sys_fops ttm igb ahci hpsa libahci drm scsi_transport_sas ptp devlink pps_core libata dca i2c_algo_bit i2c_core wmi fjes dm_mirror dm_region_hash dm_log dm_mod [last unloaded: nvidia]
CPU: 38 PID: 30810 Comm: slurmstepd Tainted: P    B      OEL ------------   3.10.0-514.21.2.el7.x86_64 #1
Hardware name: HP ProLiant SL270s Gen8 SE/, BIOS P75 09/18/2013
task: ffff883f26fa8000 ti: ffff883f4ad3c000 task.ti: ffff883f4ad3c000
RIP: 0010:[<ffffffff8168ea4b>]  [<ffffffff8168ea4b>] _raw_spin_lock_irqsave+0xb/0x60
RSP: 0018:ffff883f4ad3fd50  EFLAGS: 00000287
RAX: 0000000000000287 RBX: ffffea00a083ccc0 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000001 RDI: ffff88403ffd9540
RBP: ffff883f4ad3fd50 R08: ffff883ef52b9800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8118f010
R13: 0000000000000000 R14: ffffffff8118f93c R15: ffff883f4ad3fcf8
FS:  00007f6ac1a79780(0000) GS:ffff88403f880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc648970000 CR3: 00000007a42af000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff883f4ad3fdd0 ffffffff811f1832 0000000200000001 ffff882093a64900
0000000000000001 0000000000001000 ffff882093a64840 0000000000000001
0000000100000002 ffff882093a64800 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f1832>] mem_cgroup_reparent_charges+0x112/0x3c0
[<ffffffff811f1c94>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d6e7>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d982>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a8c8>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f965>] do_rmdir+0x1a5/0x200
[<ffffffff8168c130>] ? __schedule+0x3b0/0x990
[<ffffffff81210b16>] SyS_rmdir+0x16/0x20
[<ffffffff81697749>] system_call_fastpath+0x16/0x1b
Code: 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 83 e8 01 75 f1 48 89 df 0f 1f 80 00 00 00 00 eb e0 66 90 0f 1f 44 00 00 55 48 89 e5 9c 58 <0f> 1f 44 00 00 49 89 c0 fa 66 0f 1f 44 00 00 ba 00 00 02 00 f0 
Complete report #363815
Frame # Function Binary Source or offset Line
1
_raw_spin_lock_irqsave
vmlinux 0xffffffffabf156db
Build id: 3.10.0-862.el7.x86_64
0xb
2
mem_cgroup_reparent_charges
vmlinux 0xffffffffaba0cf72
Build id: 3.10.0-862.el7.x86_64
0x112
3
mem_cgroup_css_offline
vmlinux 0xffffffffaba0d3d4
Build id: 3.10.0-862.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffffab91c1da
Build id: 3.10.0-862.el7.x86_64
0xea
5
cgroup_rmdir
vmlinux 0xffffffffab91c482
Build id: 3.10.0-862.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffffaba295ac
Build id: 3.10.0-862.el7.x86_64
0xdc
7
do_rmdir
vmlinux 0xffffffffaba2cb31
Build id: 3.10.0-862.el7.x86_64
0x1f1
8
SyS_rmdir
vmlinux 0xffffffffaba2dd66
Build id: 3.10.0-862.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffffabf1f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [slurmstepd:136229]
Modules linked in: tcp_lp fuse nf_conntrack_netlink xt_addrtype xt_CHECKSUM br_netfilter ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iTCO_wdt iTCO_vendor_support ext4 skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel mbcache jbd2 kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper
cryptd snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep joydev snd_seq ipmi_ssif snd_seq_device pcspkr snd_pcm sg snd_timer mei_me snd lpc_ich mei i2c_i801 shpchp soundcore ipmi_si ipmi_devintf ipmi_msghandler nfit acpi_cpufreq libnvdimm acpi_power_meter acpi_pad ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common ttm syscopyarea crc32c_intel sysfillrect ixgbe sysimgblt fb_sys_fops ahci drm megaraid_sas libahci libata mdio ptp i2c_core pps_core dca wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 22 PID: 136229 Comm: slurmstepd Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: Supermicro SYS-7049GP-TRT/X11DPG-QT, BIOS 3.1 05/23/2019
task: ffff92d401b18fd0 ti: ffff92d40062c000 task.ti: ffff92d40062c000
RIP: 0010:[<ffffffffabf156db>]  [<ffffffffabf156db>] _raw_spin_lock_irqsave+0xb/0x40
RSP: 0018:ffff92d40062fd10  EFLAGS: 00000287
RAX: 006fffff00020000 RBX: ffffffffab9a2400 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff92d53ffd9540
RBP: ffff92d40062fd20 R08: ffff92d4389a6800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffab9a2d1c
R13: ffff92d40062fcc8 R14: ffff92d53ffd9000 R15: ffff92d53ffd9000
FS:  00002ac2b58c1580(0000) GS:ffff92d43be80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1355bec160 CR3: 0000007f60bce000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffaba0cf72>] mem_cgroup_reparent_charges+0x112/0x3c0
[<ffffffffaba0d3d4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffab91c1da>] cgroup_destroy_locked+0xea/0x370
[<ffffffffab91c482>] cgroup_rmdir+0x22/0x40
[<ffffffffaba295ac>] vfs_rmdir+0xdc/0x150
[<ffffffffaba2cb31>] do_rmdir+0x1f1/0x220
[<ffffffffaba2dd66>] SyS_rmdir+0x16/0x20
[<ffffffffabf1f7d5>] system_call_fastpath+0x1c/0x21
Code: 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 5c 2d ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 9c <58> 0f 1f 44 00 00 48 89 c3 fa 66 0f 1f 44 00 00 31 c0 ba 01 00 
Complete report #342035
Frame # Function Binary Source or offset Line
1
_raw_spin_lock_irqsave
vmlinux 0xffffffff816abbb1
Build id: 3.10.0-693.el7.x86_64
0x11
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f3302
Build id: 3.10.0-693.el7.x86_64
0x112
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f3764
Build id: 3.10.0-693.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120f626
Build id: 3.10.0-693.el7.x86_64
0xd6
7
do_rmdir
vmlinux 0xffffffff81212251
Build id: 3.10.0-693.el7.x86_64
0x1f1
8
? task_work_run
vmlinux 0xffffffff810ad24c
Build id: 3.10.0-693.el7.x86_64
0xac
9
SyS_rmdir
vmlinux 0xffffffff81213456
Build id: 3.10.0-693.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [systemd:1]
Modules linked in: veth xt_addrtype br_netfilter overlay(T) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache devlink nls_utf8 isofs loop xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support dell_smbios sparse_keymap dcdbas vfat fat skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif joydev sg pcspkr ipmi_si ipmi_devintf ipmi_msghandler mei_me mei shpchp nfit i2c_i801 lpc_ich acpi_power_meter wmi libnvdimm acpi_pad tpm_crb
nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci tg3 libata megaraid_sas bnxt_en ptp i2c_core pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 13 PID: 1 Comm: systemd Tainted: G        W      ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R740/0DY2X0, BIOS 2.8.2 08/27/2020
task: ffff8840a7618000 ti: ffff880127164000 task.ti: ffff880127164000
RIP: 0010:[<ffffffff816abbb1>]  [<ffffffff816abbb1>] _raw_spin_lock_irqsave+0x11/0x40
RSP: 0018:ffff880127167d48  EFLAGS: 00000283
RAX: 0000000000000283 RBX: ffffffff81190590 RCX: 0000000000000001
RDX: 0000000000000002 RSI: 0000000000000001 RDI: ffff89007ffd7540
RBP: ffff880127167d50 R08: ffff884208ae3c00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190eac
R13: ffff880127167cf8 R14: ffff89007ffd7000 R15: ffff89007ffd7000
FS:  00007f0b0aa9c940(0000) GS:ffff88bf7e2c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007faf575c4e20 CR3: 000000ff7bbfa000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffea033963b880 ffff880127167dd0 ffffffff811f3302 0000000200000003
ffff888a407f2500 0000000000000003 0000000000001000 ffff888a407f2440
0000000000000001 0000000100000002 ffff888a407f2400 0000000000000000
Call Trace:
[<ffffffff811f3302>] mem_cgroup_reparent_charges+0x112/0x3c0
[<ffffffff811f3764>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f626>] vfs_rmdir+0xd6/0x150
[<ffffffff81212251>] do_rmdir+0x1f1/0x220
[<ffffffff810ad24c>] ? task_work_run+0xac/0xf0
[<ffffffff81213456>] SyS_rmdir+0x16/0x20
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 01 c3 55 89 c6 48 89 e5 e8 61 2a ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 9c 58 0f 1f 44 00 00 <48> 89 c3 fa 66 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 
Complete report #256074
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff8831b445
Build id: 3.10.0-862.14.4.el7.x86_64
0x15
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff87e11419
Build id: 3.10.0-862.14.4.el7.x86_64
0x139
3
mem_cgroup_css_offline
vmlinux 0xffffffff87e11854
Build id: 3.10.0-862.14.4.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff87d1f4aa
Build id: 3.10.0-862.14.4.el7.x86_64
0xea
5
cgroup_rmdir
vmlinux 0xffffffff87d1f752
Build id: 3.10.0-862.14.4.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff87e2db0c
Build id: 3.10.0-862.14.4.el7.x86_64
0xdc
7
do_rmdir
vmlinux 0xffffffff87e31091
Build id: 3.10.0-862.14.4.el7.x86_64
0x1f1
8
? kmem_cache_free
vmlinux 0xffffffff87dfa282
Build id: 3.10.0-862.14.4.el7.x86_64
0x1e2
9
SyS_rmdir
vmlinux 0xffffffff87e322c6
Build id: 3.10.0-862.14.4.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff8832579b
Build id: 3.10.0-862.14.4.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [slurmstepd:434263]
Modules linked in: 8021q garp mrp stp llc nfsv3 nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core dm_mirror dm_region_hash dm_log dm_mod skx_edac intel_powerclamp coretemp iTCO_wdt intel_rapl iTCO_vendor_support iosf_mbi kvm ast i2c_algo_bit ttm irqbypass crc32_pclmul ghash_clmulni_intel drm_kms_helper ixgbe syscopyarea aesni_intel sysfillrect sysimgblt fb_sys_fops lrw drm ptp gf128mul glue_helper i2c_i801 pps_core ablk_helper mdio mei_me cryptd dca joydev mei pcspkr i2c_core sg lpc_ich shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace
sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci libata
CPU: 19 PID: 434263 Comm: slurmstepd Kdump: loaded Tainted: G             L ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: Advanced Clustering S2600BPB/S2600BPB, BIOS SE5C620.86B.02.01.0008.C1.031920191559 03/19/2019
task: ffffa1163a716eb0 ti: ffffa0ef213ec000 task.ti: ffffa0ef213ec000
RIP: 0010:[<ffffffff8831b445>]  [<ffffffff8831b445>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffffa0ef213efd20  EFLAGS: 00000287
RAX: 0000000000000287 RBX: ffffe656eaec2880 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffffa0ef213efd20 R08: ffffa0ced2339000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff87da5160
R13: 0000000000000000 R14: ffffffff87da5a7c R15: ffffa0ef213efcc8
FS:  00002b2e6988bfc0(0000) GS:ffffa1163be40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b6e30007058 CR3: 000000319d382000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff87e11419>] mem_cgroup_reparent_charges+0x139/0x3c0
[<ffffffff87e11854>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff87d1f4aa>] cgroup_destroy_locked+0xea/0x370
[<ffffffff87d1f752>] cgroup_rmdir+0x22/0x40
[<ffffffff87e2db0c>] vfs_rmdir+0xdc/0x150
[<ffffffff87e31091>] do_rmdir+0x1f1/0x220
[<ffffffff87dfa282>] ? kmem_cache_free+0x1e2/0x200
[<ffffffff87e322c6>] SyS_rmdir+0x16/0x20
[<ffffffff8832579b>] system_call_fastpath+0x22/0x27
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #221294
Frame # Function Binary Source or offset Line
1
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f123b
Build id: 3.10.0-514.10.2.el7.x86_64
0x15b
2
mem_cgroup_css_offline
vmlinux 0xffffffff811f1654
Build id: 3.10.0-514.10.2.el7.x86_64
0x84
3
cgroup_destroy_locked
vmlinux 0xffffffff8110d257
Build id: 3.10.0-514.10.2.el7.x86_64
0xe7
4
cgroup_rmdir
vmlinux 0xffffffff8110d4f2
Build id: 3.10.0-514.10.2.el7.x86_64
0x22
5
vfs_rmdir
vmlinux 0xffffffff8120a278
Build id: 3.10.0-514.10.2.el7.x86_64
0xa8
6
do_rmdir
vmlinux 0xffffffff8120f315
Build id: 3.10.0-514.10.2.el7.x86_64
0x1a5
7
? __schedule
vmlinux 0xffffffff8168b5b0
Build id: 3.10.0-514.10.2.el7.x86_64
0x3b0
8
SyS_rmdir
vmlinux 0xffffffff812104c6
Build id: 3.10.0-514.10.2.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffff81696b09
Build id: 3.10.0-514.10.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [slurmstepd:165465]
Modules linked in: squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt intel_powerclamp iTCO_vendor_support mxm_wmi coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul dcdbas ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm cdc_ether usbnet mii pcspkr sb_edac mei_me lpc_ich mei edac_core shpchp wmi tpm_crb acpi_power_meter rpcrdma ib_isert iscsi_target_mod target_core_mod crc_t10dif crct10dif_generic ib_iser libiscsi scsi_transport_iscsi ib_ipoib
rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables hfi1(OE) ipmi_devintf rdmavt(OE) ixgbe ahci ib_core libahci dca crct10dif_pclmul mdio crct10dif_common ptp crc32c_intel i2c_algo_bit libata ipmi_si pps_core i2c_core fjes ipmi_msghandler
CPU: 18 PID: 165465 Comm: slurmstepd Tainted: G           OE  ------------ T 3.10.0-514.10.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge C6320/082F9M, BIOS 2.1.5 04/12/2016
task: ffff88129b202f10 ti: ffff8810b5b60000 task.ti: ffff8810b5b60000
RIP: 0010:[<ffffffff811f123b>]  [<ffffffff811f123b>] mem_cgroup_reparent_charges+0x15b/0x3c0
RSP: 0018:ffff8810b5b63d60  EFLAGS: 00000246
RAX: 0000000000000002 RBX: ffffea001a28c500 RCX: 0000000000000003
RDX: 0000000000000002 RSI: ffffea001a28c4dc RDI: ffffea001a28c4c0
RBP: ffff8810b5b63dd0 R08: 0000000000000283 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffea001a2874c0
R13: 0000000000000246 R14: ffffffff8118eb20 R15: 0000000000000000
FS:  00002b17143da5c0(0000) GS:ffff88103e840000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000099bad0 CR3: 0000001095b8f000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000200000000 ffff88013f13f900 0000000000000000 0000000000001000
ffff88013f13f840 0000000000000001 0000000100000003 ffff88013f13f800
0000000000000000 0000000000000000 ffff8802c8cb6c00 ffff88013f13f800
Call Trace:
[<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a278>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f315>] do_rmdir+0x1a5/0x200
[<ffffffff8168b5b0>] ? __schedule+0x3b0/0x990
[<ffffffff812104c6>] SyS_rmdir+0x16/0x20
[<ffffffff81696b09>] system_call_fastpath+0x16/0x1b
Code: c6 4c 89 ef e8 b7 ca 49 00 48 89 df e8 9f 3b 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 d3 34 fa ff 85 c0 0f 
Complete report #212206
Frame # Function Binary Source or offset Line
1
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f34d9
Build id: 3.10.0-693.2.2.el7.x86_64
0x2d9
2
mem_cgroup_css_offline
vmlinux 0xffffffff811f3774
Build id: 3.10.0-693.2.2.el7.x86_64
0x84
3
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.2.2.el7.x86_64
0xe7
4
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
5
vfs_rmdir
vmlinux 0xffffffff8120f636
Build id: 3.10.0-693.2.2.el7.x86_64
0xd6
6
do_rmdir
vmlinux 0xffffffff812122d1
Build id: 3.10.0-693.2.2.el7.x86_64
0x1f1
7
? do_page_fault
vmlinux 0xffffffff816b0325
Build id: 3.10.0-693.2.2.el7.x86_64
0x35
8
SyS_rmdir
vmlinux 0xffffffff812134d6
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [slurmstepd:201567]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev dcdbas iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc
xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) megaraid_sas ahci tg3(OE) ptp libahci pps_core libata [last unloaded: ipmi_msghandler]
CPU: 9 PID: 201567 Comm: slurmstepd Tainted: G           OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018
task: ffff8830d85c9fa0 ti: ffff88315ac44000 task.ti: ffff88315ac44000
RIP: 0010:[<ffffffff811f34d9>]  [<ffffffff811f34d9>] mem_cgroup_reparent_charges+0x2d9/0x3c0
RSP: 0018:ffff88315ac47d60  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffea00e94e1100 RCX: 0000000000000001
RDX: 0000000005e12040 RSI: 0000000000000001 RDI: ffff88313e951c40
RBP: ffff88315ac47dd0 R08: ffff885f6e548800 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffea00df9a08c0
R13: 0000000000000246 R14: ffffffff81190590 R15: 0000000000000000
FS:  00002aaaaaafd180(0000) GS:ffff885f7bf00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaaab0fc0a0 CR3: 000000022a42e000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000200000001 ffff88313e951d00 0000000000000001 0000000000001000
ffff88313e951c40 0000000000000001 0000000100000003 ffff88313e951c00
0000000000000000 0000000000000000 ffff8844af6b5e00 ffff88313e951c00
Call Trace:
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 89 c6 e8 9b 83 4b 00 e9 46 fe ff ff 66 0f 1f 44 00 00 48 8b 7d b0 49 63 c3 48 89 c6 48 89 45 b8 e8 7d bc ff ff 8b 05 c7 06 93 00 <44> 8b 55 c4 85 c0 0f 84 26 ff ff ff 48 8b 75 b8 48 8b 7d 98 e8 
Complete report #202303
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffbaf6a0f5
Build id: 3.10.0-957.el7.x86_64
0x15
2
mem_cgroup_reparent_charges
vmlinux 0xffffffffbaa32eb9
Build id: 3.10.0-957.el7.x86_64
0x139
3
mem_cgroup_css_offline
vmlinux 0xffffffffbaa332f4
Build id: 3.10.0-957.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffffba92573a
Build id: 3.10.0-957.el7.x86_64
0xea
5
cgroup_rmdir
vmlinux 0xffffffffba9259e2
Build id: 3.10.0-957.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffffbaa4dc2c
Build id: 3.10.0-957.el7.x86_64
0xdc
7
do_rmdir
vmlinux 0xffffffffbaa530d1
Build id: 3.10.0-957.el7.x86_64
0x1f1
8
SyS_rmdir
vmlinux 0xffffffffbaa54306
Build id: 3.10.0-957.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffffbaf74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#81 stuck for 22s! [slurmstepd:127949]
Modules linked in: iscsi_target_mod target_core_mod macsec tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag ip_vs nf_conntrack libcrc32c binfmt_misc vfat fat uas usb_storage tcp_lp fuse nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache bonding sunrpc dm_mirror dm_region_hash dm_log dm_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg ipmi_si ipmi_devintf ipmi_msghandler pinctrl_amd i2c_designware_platform i2c_designware_core acpi_cpufreq i2c_piix4 sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci
drm_panel_orientation_quirks libata megaraid_sas ixgbe igb(OE) mdio ptp pps_core i2c_algo_bit dca nfit libnvdimm [last unloaded: amifldrv_mod]
CPU: 81 PID: 127949 Comm: slurmstepd Kdump: loaded Tainted: P        W  OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: N/A 2U Rack Server/65N32-US, BIOS SZZQ042030 07/27/2020
task: ffffa14fb7edd140 ti: ffffa17d5deec000 task.ti: ffffa17d5deec000
RIP: 0010:[<ffffffffbaf6a0f5>]  [<ffffffffbaf6a0f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffffa17d5deefd20  EFLAGS: 00000287
RAX: 0000000000000287 RBX: ffffe9ecb26bd140 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffffa17d5deefd20 R08: ffffa195ee71d000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffba9c4930
R13: 0000000000000000 R14: ffffffffba9c524c R15: ffffa17d5deefcc8
FS:  00002ae07001afc0(0000) GS:ffffa14bbf640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000002c057e000 CR3: 00000203c7702000 CR4: 0000000000340fe0
Call Trace:
[<ffffffffbaa32eb9>] mem_cgroup_reparent_charges+0x139/0x3c0
[<ffffffffbaa332f4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffffba92573a>] cgroup_destroy_locked+0xea/0x370
[<ffffffffba9259e2>] cgroup_rmdir+0x22/0x40
[<ffffffffbaa4dc2c>] vfs_rmdir+0xdc/0x150
[<ffffffffbaa530d1>] do_rmdir+0x1f1/0x220
[<ffffffffbaa54306>] SyS_rmdir+0x16/0x20
[<ffffffffbaf74ddb>] system_call_fastpath+0x22/0x27
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48 
Complete report #199654
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff8168dceb
Build id: 3.10.0-514.10.2.el7.x86_64
0x1b
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f1219
Build id: 3.10.0-514.10.2.el7.x86_64
0x139
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f1654
Build id: 3.10.0-514.10.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d257
Build id: 3.10.0-514.10.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110d4f2
Build id: 3.10.0-514.10.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120a278
Build id: 3.10.0-514.10.2.el7.x86_64
0xa8
7
do_rmdir
vmlinux 0xffffffff8120f315
Build id: 3.10.0-514.10.2.el7.x86_64
0x1a5
8
? __schedule
vmlinux 0xffffffff8168b5b0
Build id: 3.10.0-514.10.2.el7.x86_64
0x3b0
9
SyS_rmdir
vmlinux 0xffffffff812104c6
Build id: 3.10.0-514.10.2.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff81696b09
Build id: 3.10.0-514.10.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#96 stuck for 22s! [slurmstepd:64066]
Modules linked in: squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod dcdbas intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel xfs aesni_intel lrw gf128mul glue_helper libcrc32c ablk_helper cryptd iTCO_wdt iTCO_vendor_support mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops pcspkr sg drm cdc_ether usbnet mii nfit lpc_ich wmi i2c_i801 shpchp libnvdimm acpi_pad acpi_power_meter rpcrdma ib_isert iscsi_target_mod target_core_mod ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm
ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables sd_mod crc_t10dif crct10dif_generic ipmi_devintf hfi1(OE) igb rdmavt(OE) ahci dca crct10dif_pclmul ib_core crct10dif_common ptp libahci i2c_algo_bit crc32c_intel megaraid_sas(OE) i2c_core pps_core libata ipmi_si ipmi_msghandler fjes
CPU: 96 PID: 64066 Comm: slurmstepd Tainted: G        W  OE  ------------ T 3.10.0-514.10.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R940/0V0267, BIOS 2.6.4 04/09/2020
task: ffff88be83364e70 ti: ffff88bc2e6a4000 task.ti: ffff88bc2e6a4000
RIP: 0010:[<ffffffff8168dceb>]  [<ffffffff8168dceb>] _raw_spin_unlock_irqrestore+0x1b/0x40
RSP: 0018:ffff88bc2e6a7d48  EFLAGS: 00000287
RAX: 0000000000000287 RBX: ffffffff8118eb20 RCX: 000000000000c144
RDX: 00000000c144c144 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff88bc2e6a7d50 R08: 0000000000000287 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8118f44c
R13: ffff88bc2e6a7cf8 R14: ffff8a403ffda000 R15: ffff8a403ffda000
FS:  00002b8d4d1ee5c0(0000) GS:ffff885ec0900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000013030f0 CR3: 000000a8b0bbe000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffea0803935f00 ffff88bc2e6a7dd0 ffffffff811f1219 0000000200000006
ffff88509cb4b900 0000000000000006 0000000000001000 ffff88509cb4b840
0000000000000001 0000000100000002 ffff88509cb4b800 0000000000000000
Call Trace:
[<ffffffff811f1219>] mem_cgroup_reparent_charges+0x139/0x3c0
[<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40
[<ffffffff8120a278>] vfs_rmdir+0xa8/0x100
[<ffffffff8120f315>] do_rmdir+0x1a5/0x200
[<ffffffff8168b5b0>] ? __schedule+0x3b0/0x990
[<ffffffff812104c6>] SyS_rmdir+0x16/0x20
[<ffffffff81696b09>] system_call_fastpath+0x16/0x1b
Code: 08 e8 6a 0e a0 ff 5d c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 48 89 f3 0f 1f 44 00 00 66 83 07 02 48 89 df 57 9d <0f> 1f 44 00 00 5b 5d c3 0f 1f 44 00 00 8b 37 f0 66 83 07 02 f6 
Complete report #192561
Frame # Function Binary Source or offset Line
1
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f335b
Build id: 3.10.0-693.2.2.el7.x86_64
0x15b
2
mem_cgroup_css_offline
vmlinux 0xffffffff811f3774
Build id: 3.10.0-693.2.2.el7.x86_64
0x84
3
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.2.2.el7.x86_64
0xe7
4
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
5
vfs_rmdir
vmlinux 0xffffffff8120f636
Build id: 3.10.0-693.2.2.el7.x86_64
0xd6
6
do_rmdir
vmlinux 0xffffffff812122d1
Build id: 3.10.0-693.2.2.el7.x86_64
0x1f1
7
? do_page_fault
vmlinux 0xffffffff816b0325
Build id: 3.10.0-693.2.2.el7.x86_64
0x35
8
SyS_rmdir
vmlinux 0xffffffff812134d6
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
9
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [slurmstepd:193578]
Modules linked in: mmfs26(OE) mmfslinux(OE) squashfs loop overlay(T) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core iTCO_wdt intel_powerclamp iTCO_vendor_support coretemp dcdbas intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ghash_clmulni_intel drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf nfit ipmi_msghandler libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci
libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci megaraid_sas tg3(OE) libahci ptp pps_core libata [last unloaded: mmfslinux]
CPU: 30 PID: 193578 Comm: slurmstepd Tainted: G    B      OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018
task: ffff88022974cf10 ti: ffff8823c9c0c000 task.ti: ffff8823c9c0c000
RIP: 0010:[<ffffffff811f335b>]  [<ffffffff811f335b>] mem_cgroup_reparent_charges+0x15b/0x3c0
RSP: 0018:ffff8823c9c0fd60  EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffffea017a9e7640 RCX: 0000000000000002
RDX: 0000000000000001 RSI: ffffea017506b6dc RDI: 0000000005d41adb
RBP: ffff8823c9c0fdd0 R08: ffff885f445fbc00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffea0177754c40
R13: 0000000000000246 R14: ffffffff81190590 R15: 0000000000000000
FS:  00002aaaaaafd180(0000) GS:ffff882f7ddc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaaabe2e1a0 CR3: 0000004f0e757000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000200000001 ffff883163cff500 0000000000000001 0000000000001000
ffff883163cff440 0000000000000001 0000000100000003 ffff883163cff400
0000000000000000 0000000000000000 ffff8831041a8e00 ffff883163cff400
Call Trace:
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: c6 4c 89 ef e8 17 85 4b 00 48 89 df e8 4f 3b 00 00 41 8b 57 fc 49 89 c6 85 d2 74 ac 8d 4a 01 49 8d 77 fc 89 d0 f0 41 0f b1 4f fc <39> c2 89 c1 0f 85 bb 01 00 00 48 89 df e8 13 25 fa ff 85 c0 0f 
Complete report #190652
Frame # Function Binary Source or offset Line
1
_raw_spin_lock_irqsave
vmlinux 0xffffffff816abbcb
Build id: 3.10.0-693.2.2.el7.x86_64
0xb
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f3312
Build id: 3.10.0-693.2.2.el7.x86_64
0x112
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f3774
Build id: 3.10.0-693.2.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.2.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120f636
Build id: 3.10.0-693.2.2.el7.x86_64
0xd6
7
do_rmdir
vmlinux 0xffffffff812122d1
Build id: 3.10.0-693.2.2.el7.x86_64
0x1f1
8
? do_page_fault
vmlinux 0xffffffff816b0325
Build id: 3.10.0-693.2.2.el7.x86_64
0x35
9
SyS_rmdir
vmlinux 0xffffffff812134d6
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#31 stuck for 22s! [slurmstepd:80311]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass dell_smbios dcdbas sparse_keymap crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter
bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci megaraid_sas libahci tg3(OE) ptp libata pps_core [last unloaded: ipmi_msghandler]
CPU: 31 PID: 80311 Comm: slurmstepd Tainted: G           OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.9.3 09/23/2020
task: ffff8839afe06eb0 ti: ffff8831c479c000 task.ti: ffff8831c479c000
RIP: 0010:[<ffffffff816abbcb>]  [<ffffffff816abbcb>] _raw_spin_lock_irqsave+0xb/0x40
RSP: 0018:ffff8831c479fd40  EFLAGS: 00000287
RAX: 006fffff00000000 RBX: ffffffff81190590 RCX: 0000000000000001
RDX: 0000000000000003 RSI: 0000000000000001 RDI: ffff88607bfd7540
RBP: ffff8831c479fd50 R08: ffff882fa8ae3400 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190eac
R13: ffff8831c479fcf8 R14: ffff88607bfd7000 R15: ffff88607bfd7000
FS:  00002aaaaaafd180(0000) GS:ffff885fb81c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaaab7296d0 CR3: 0000000278f65000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000287 ffffea01105d3f40 ffff8831c479fdd0 ffffffff811f3312
0000000200000001 ffff883225c05900 0000000000000001 0000000000001000
ffff883225c05840 0000000000000001 0000000100000003 ffff883225c05800
Call Trace:
[<ffffffff811f3312>] mem_cgroup_reparent_charges+0x112/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 61 2a ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 9c <58> 0f 1f 44 00 00 48 89 c3 fa 66 0f 1f 44 00 00 31 c0 ba 01 00 
Complete report #159959
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff816ab865
Build id: 3.10.0-693.2.2.el7.x86_64
0x15
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f3339
Build id: 3.10.0-693.2.2.el7.x86_64
0x139
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f3774
Build id: 3.10.0-693.2.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.2.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120f636
Build id: 3.10.0-693.2.2.el7.x86_64
0xd6
7
do_rmdir
vmlinux 0xffffffff812122d1
Build id: 3.10.0-693.2.2.el7.x86_64
0x1f1
8
? do_page_fault
vmlinux 0xffffffff816b0325
Build id: 3.10.0-693.2.2.el7.x86_64
0x35
9
SyS_rmdir
vmlinux 0xffffffff812134d6
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#48 stuck for 22s! [slurmstepd:265028]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp dell_smbios coretemp dcdbas intel_rapl sparse_keymap iosf_mbi kvm_intel kvm irqbypass crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf nfit ipmi_msghandler libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp
llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) megaraid_sas ahci libahci tg3(OE) libata ptp pps_core [last unloaded: tracedev]
CPU: 48 PID: 265028 Comm: slurmstepd Tainted: G    B      OE  ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.8.2 08/31/2020
task: ffff8830b5410000 ti: ffff882172a48000 task.ti: ffff882172a48000
RIP: 0010:[<ffffffff816ab865>]  [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0000:ffff882172a4bd50  EFLAGS: 00000283
RAX: 0000000000000283 RBX: ffffea012a49de80 RCX: 0000000000000001
RDX: 0000000000000101 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff882172a4bd50 R08: 0000000000000101 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590
R13: 0000000000000000 R14: ffffffff81190eac R15: ffff882172a4bcf8
FS:  00002aaaaaafd180(0000) GS:ffff882fb9000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaac485c118 CR3: 00000010ef72e000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff882172a4bdd0 ffffffff811f3339 0000000200000001 ffff8801b92ec100
0000000000000001 0000000000001000 ffff8801b92ec040 0000000000000001
0000000100000002 ffff8801b92ec000 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f3339>] mem_cgroup_reparent_charges+0x139/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #127384
Frame # Function Binary Source or offset Line
1
_raw_spin_lock_irqsave
vmlinux 0xffffffff8163cbaa
Build id: 3.10.0-327.3.1.el7.x86_64
0xa
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811d1aa2
Build id: 3.10.0-327.3.1.el7.x86_64
0x152
3
mem_cgroup_css_offline
vmlinux 0xffffffff811d1f0f
Build id: 3.10.0-327.3.1.el7.x86_64
0x4f
4
cgroup_destroy_locked
vmlinux 0xffffffff810fa347
Build id: 3.10.0-327.3.1.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff810fa5e2
Build id: 3.10.0-327.3.1.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff811ea158
Build id: 3.10.0-327.3.1.el7.x86_64
0xa8
7
do_rmdir
vmlinux 0xffffffff811ef0f5
Build id: 3.10.0-327.3.1.el7.x86_64
0x1a5
8
? __schedule
vmlinux 0xffffffff8163a238
Build id: 3.10.0-327.3.1.el7.x86_64
0x2d8
9
SyS_rmdir
vmlinux 0xffffffff811f02a6
Build id: 3.10.0-327.3.1.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff816458c9
Build id: 3.10.0-327.3.1.el7.x86_64
0x16
BUG: soft lockup - CPU#1 stuck for 22s! [slurmstepd:17937]
Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver intel_powerclamp coretemp intel_rapl kvm_intel kvm ast syscopyarea sysfillrect sysimgblt crc32_pclmul ttm crc32c_intel iTCO_wdt ghash_clmulni_intel iTCO_vendor_support cryptd drm_kms_helper drm mei_me mei shpchp pcspkr lpc_ich ipmi_devintf i2c_i801 wmi mfd_core ipmi_si ipmi_msghandler acpi_pad acpi_power_meter xprtrdma ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss binfmt_misc ip_tables i40e raid1 raid0 bridge stp llc hpilo xfs isci libsas qla3xxx hpsa sg e1000e ixgbe megaraid_sas aacraid aic79xx ata_piix sd_mod crc_t10dif mpt2sas mptspi mptscsih mptbase scsi_transport_spi arcmsr sata_nv sata_svw bnx2x mdio libcrc32c bnx2 ext4 mbcache
jbd2 sata_sil tg3 e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun mlx4_en mlx4_ib vxlan ib_sa ip6_udp_tunnel udp_tunnel ib_mad ib_core ib_addr crct10dif_pclmul mlx4_core crct10dif_common igb ahci libahci i2c_algo_bit mpt3sas i2c_core libata dca raid_class ptp scsi_transport_sas pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]
CPU: 1 PID: 17937 Comm: slurmstepd Not tainted 3.10.0-327.3.1.el7.x86_64 #1
Hardware name: SGI.COM CH-C1104-GP2/X10DRU-i+, BIOS 2.0 02/18/2016
task: ffff88712ddca280 ti: ffff883c585c0000 task.ti: ffff883c585c0000
RIP: 0010:[<ffffffff8163cbaa>]  [<ffffffff8163cbaa>] _raw_spin_lock_irqsave+0xa/0x60
RSP: 0018:ffff883c585c3d40  EFLAGS: 00000287
RAX: 0000000000000000 RBX: 0000000000000246 RCX: 0000000000000001
RDX: 0000000000000002 RSI: 0000000000001000 RDI: ffff88807ffd7540
RBP: ffff883c585c3d48 R08: ffffc90040d76000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000001000 R12: 0000000000000000
R13: ffffffff811773b8 R14: ffff883c585c3cf0 R15: ffff88807ffd7000
FS:  00002aaaaaafe540(0000) GS:ffff883f7fc20000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffff7ff6000 CR3: 0000007f01fdf000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000287 ffff883c585c3dd0 ffffffff811d1aa2 ffffc9004aacd140
0000000200000001 ffffc9004aacd108 ffffc9004aacd040 0000000000000001
0000000000001000 0000000100000003 0000000000000001 ffffc9004aacd000
Call Trace:
[<ffffffff811d1aa2>] mem_cgroup_reparent_charges+0x152/0x440
[<ffffffff811d1f0f>] mem_cgroup_css_offline+0x4f/0x100
[<ffffffff810fa347>] cgroup_destroy_locked+0xe7/0x360
[<ffffffff810fa5e2>] cgroup_rmdir+0x22/0x40
[<ffffffff811ea158>] vfs_rmdir+0xa8/0x100
[<ffffffff811ef0f5>] do_rmdir+0x1a5/0x200
[<ffffffff8163a238>] ? __schedule+0x2d8/0x900
[<ffffffff811f02a6>] SyS_rmdir+0x16/0x20
[<ffffffff816458c9>] system_call_fastpath+0x16/0x1b
Code: 00 80 00 00 0f b7 0b 66 39 ca 74 ea f3 90 83 e8 01 75 f1 48 89 df 0f 1f 80 00 00 00 00 eb e0 66 90 0f 1f 44 00 00 55 48 89 e5 9c <58> 0f 1f 44 00 00 49 89 c0 fa 66 0f 1f 44 00 00 ba 00 00 02 00 
Complete report #72730
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff816ab865
Build id: 3.10.0-693.2.2.el7.x86_64
0x15
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f3339
Build id: 3.10.0-693.2.2.el7.x86_64
0x139
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f3774
Build id: 3.10.0-693.2.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.2.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120f636
Build id: 3.10.0-693.2.2.el7.x86_64
0xd6
7
do_rmdir
vmlinux 0xffffffff812122d1
Build id: 3.10.0-693.2.2.el7.x86_64
0x1f1
8
? do_page_fault
vmlinux 0xffffffff816b0325
Build id: 3.10.0-693.2.2.el7.x86_64
0x35
9
SyS_rmdir
vmlinux 0xffffffff812134d6
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [slurmstepd:371639]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev dcdbas skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc
xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci megaraid_sas libahci libata tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 25 PID: 371639 Comm: slurmstepd Tainted: G        W  OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018
task: ffff8855ff3bcf10 ti: ffff88024e1bc000 task.ti: ffff88024e1bc000
RIP: 0010:[<ffffffff816ab865>]  [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff88024e1bfd50  EFLAGS: 00000287
RAX: 0000000000000287 RBX: ffffea00dbb30640 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff88024e1bfd50 R08: ffff882f71403c00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590
R13: 0000000000000000 R14: ffffffff81190eac R15: ffff88024e1bfcf8
FS:  00002aaaaaafd180(0000) GS:ffff885f7c100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaaac36a0f0 CR3: 000000126f0eb000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff88024e1bfdd0 ffffffff811f3339 0000000200000001 ffff88258af3ed00
0000000000000001 0000000000001000 ffff88258af3ec40 0000000000000001
0000000100000003 ffff88258af3ec00 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f3339>] mem_cgroup_reparent_charges+0x139/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #12803
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff816ab865
Build id: 3.10.0-693.2.2.el7.x86_64
0x15
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f3339
Build id: 3.10.0-693.2.2.el7.x86_64
0x139
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f3774
Build id: 3.10.0-693.2.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d757
Build id: 3.10.0-693.2.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110da02
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120f636
Build id: 3.10.0-693.2.2.el7.x86_64
0xd6
7
do_rmdir
vmlinux 0xffffffff812122d1
Build id: 3.10.0-693.2.2.el7.x86_64
0x1f1
8
? do_page_fault
vmlinux 0xffffffff816b0325
Build id: 3.10.0-693.2.2.el7.x86_64
0x35
9
SyS_rmdir
vmlinux 0xffffffff812134d6
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#53 stuck for 23s! [slurmstepd:349322]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp dcdbas intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm aesni_intel drm_kms_helper lrw gf128mul glue_helper ablk_helper cryptd syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr mei_me mei i2c_i801 shpchp lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci
libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) megaraid_sas ahci libahci libata tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 53 PID: 349322 Comm: slurmstepd Tainted: G           OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018
task: ffff88012c2fbf40 ti: ffff8803b0b78000 task.ti: ffff8803b0b78000
RIP: 0010:[<ffffffff816ab865>]  [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8803b0b7bd50  EFLAGS: 00000287
RAX: 0000000000000287 RBX: ffffea0136d07540 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff8803b0b7bd50 R08: ffff882f143aac00 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590
R13: 0000000000000000 R14: ffffffff81190eac R15: ffff8803b0b7bcf8
FS:  00002aaaaaafd180(0000) GS:ffff885f7c480000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055555557e160 CR3: 0000004de43b8000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8803b0b7bdd0 ffffffff811f3339 0000000200000001 ffff88024a2c1d00
0000000000000001 0000000000001000 ffff88024a2c1c40 0000000000000001
0000000100000003 ffff88024a2c1c00 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f3339>] mem_cgroup_reparent_charges+0x139/0x3c0
[<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da02>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f636>] vfs_rmdir+0xd6/0x150
[<ffffffff812122d1>] do_rmdir+0x1f1/0x220
[<ffffffff816b0325>] ? do_page_fault+0x35/0x90
[<ffffffff812134d6>] SyS_rmdir+0x16/0x20
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #2010
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff816ab905
Build id: 3.10.0-693.5.2.el7.x86_64
0x15
2
mem_cgroup_reparent_charges
vmlinux 0xffffffff811f32a9
Build id: 3.10.0-693.5.2.el7.x86_64
0x139
3
mem_cgroup_css_offline
vmlinux 0xffffffff811f36e4
Build id: 3.10.0-693.5.2.el7.x86_64
0x84
4
cgroup_destroy_locked
vmlinux 0xffffffff8110d767
Build id: 3.10.0-693.5.2.el7.x86_64
0xe7
5
cgroup_rmdir
vmlinux 0xffffffff8110da12
Build id: 3.10.0-693.5.2.el7.x86_64
0x22
6
vfs_rmdir
vmlinux 0xffffffff8120f5a6
Build id: 3.10.0-693.5.2.el7.x86_64
0xd6
7
do_rmdir
vmlinux 0xffffffff81212241
Build id: 3.10.0-693.5.2.el7.x86_64
0x1f1
8
? __schedule
vmlinux 0xffffffff816a904d
Build id: 3.10.0-693.5.2.el7.x86_64
0x39d
9
SyS_rmdir
vmlinux 0xffffffff81213446
Build id: 3.10.0-693.5.2.el7.x86_64
0x16
10
system_call_fastpath
vmlinux 0xffffffff816b5089
Build id: 3.10.0-693.5.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#45 stuck for 23s! [slurmstepd:95670]
Modules linked in: osc(OE) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) 8021q garp mrp stp llc ip6_tables rpcrdma ib_isert iscsi_target_mod amd64_edac_mod edac_mce_amd edac_core kvm_amd ib_iser kvm libiscsi scsi_transport_iscsi ib_srpt irqbypass crc32_pclmul target_core_mod ghash_clmulni_intel aesni_intel lrw dcdbas gf128mul glue_helper ablk_helper ib_srp cryptd scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm pcspkr sg ib_cm ipmi_si ipmi_devintf iw_cm ipmi_msghandler ccp tpm_crb acpi_power_meter ib_core shpchp i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200
i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci crct10dif_pclmul crct10dif_common tg3 crc32c_intel libata megaraid_sas bnxt_en ptp i2c_core pps_core
CPU: 45 PID: 95670 Comm: slurmstepd Tainted: G           OEL ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R7415/07YXFK, BIOS 1.8.7 04/02/2019
task: ffff8803bcc65ee0 ti: ffff880e5d560000 task.ti: ffff880e5d560000
RIP: 0010:[<ffffffff816ab905>]  [<ffffffff816ab905>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff880e5d563d50  EFLAGS: 00000287
RAX: 0000000000000287 RBX: ffffea00b48059c0 RCX: 0000000001690000
RDX: 0000000000f90000 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff880e5d563d50 R08: ffff883f7d757880 R09: ffff883f7d397880
R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff811905c0
R13: 0000000000000000 R14: ffffffff81190edc R15: ffff880e5d563cf8
FS:  00002ac93b0b7fc0(0000) GS:ffff883f7d740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcffaa0cfb8 CR3: 0000001d49f3d000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff880e5d563dd0 ffffffff811f32a9 0000000200000000 ffff88267bf04900
0000000000000000 0000000000001000 ffff88267bf04840 0000000000000001
0000000100000002 ffff88267bf04800 0000000000000000 0000000000000000
Call Trace:
[<ffffffff811f32a9>] mem_cgroup_reparent_charges+0x139/0x3c0
[<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140
[<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370
[<ffffffff8110da12>] cgroup_rmdir+0x22/0x40
[<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150
[<ffffffff81212241>] do_rmdir+0x1f1/0x220
[<ffffffff816a904d>] ? __schedule+0x39d/0x8b0
[<ffffffff81213446>] SyS_rmdir+0x16/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48