| Operating system | Count |
|---|---|
| CentOS 7 | 21 |
| Architecture | Count |
|---|---|
| x86_64 | 21 |
| Related packages | Count |
|---|---|
| kernel | 21 |
Packages names with count less than the total count of reports are most probably not the cause of the problem.
No reports in the last 20 days.
No reports in the last 20 weeks.
NMI watchdog: BUG: soft lockup - CPU#58 stuck for 22s! [slurmstepd:33024] Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge 8021q garp mrp stp llc overlay(T) ip6_tables iTCO_wdt iTCO_vendor_support iptable_filter coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure cdc_ether scsi_transport_sas usbnet joydev mii sg mei_me mei shpchp i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace binfmt_misc sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw ahci libahci mlx5_core(OE) libata mlxfw(OE) ptp megaraid_sas pps_core auxiliary(OE) mlx_compat(OE) i2c_core devlink [last unloaded: tracedev] CPU: 58 PID: 33024 Comm: slurmstepd Tainted: G W OE ------------ T 3.10.0-693.5.2.el7.x86_64 #1 Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW, BIOS AFE112A-1.02 05/10/2021 task: ffff899cf4bb4f10 ti: ffff88f053c80000 task.ti: ffff88f053c80000 RIP: 0010:[<ffffffff811ef0c0>] [<ffffffff811ef0c0>] vma_adjust_trans_huge+0xf0/0xf0 RSP: 0018:ffff88f053c83d58 EFLAGS: 00000246 RAX: 0000000000000001 RBX: ffffea02a7cec500 RCX: 0000000000000001 RDX: 0000000004326f8a RSI: 0000000000000001 RDI: ffff884bc2edd440 RBP: ffff88f053c83dd0 R08: ffff89f821785800 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff811905c0 R13: 0000000000000000 R14: ffffffff81190edc R15: ffff88f053c83cf8 FS: 00002b7e18589fc0(0000) GS:ffff89f83e880000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f439af05000 CR3: 00000103931bb000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffffffff811f3443 0000000200000000 ffff884bc2edd500 0000000000000000 0000000000001000 ffff884bc2edd440 0000000000000001 0000000100000002 ffff884bc2edd400 0000000000000000 0000000000000000 ffff88e6e8af0400 Call Trace: [<ffffffff811f3443>] ? mem_cgroup_reparent_charges+0x2d3/0x3c0 [<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da12>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150 [<ffffffff81212241>] do_rmdir+0x1f1/0x220 [<ffffffff81213446>] SyS_rmdir+0x16/0x20 [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b Code: 97 fe ff ff 48 8b 4d e8 48 8b 55 f0 e9 55 ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 48 8b 7a 40 e8 77 fe ff ff 48 83 c4 10 5b 5d c3 <0f> 1f 44 00 00 55 48 89 f0 48 f7 d8 48 89 e5 53 f0 48 0f c1 07
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [slurmstepd:14140] Modules linked in: squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic nfs_layout_nfsv41_files crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log ses enclosure sd_mod sg dm_mod intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd uas serio_raw pcspkr usb_storage smartpqi scsi_transport_sas hpilo hpwdt nvme shpchp lpc_ich wmi acpi_power_meter rpcrdma ib_isert iscsi_target_mod target_core_mod crc_t10dif crct10dif_generic ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ipmi_devintf hfi1(OE) i2c_algo_bit i2c_core bnxt_en crct10dif_pclmul crct10dif_common rdmavt(OE) crc32c_intel ib_core ipmi_si fjes ipmi_msghandler CPU: 1 PID: 14140 Comm: slurmstepd Tainted: G W OEL ------------ T 3.10.0-514.10.2.el7.x86_64 #1 Hardware name: HPE ProLiant DL560 Gen10/ProLiant DL560 Gen10, BIOS U34 03/16/2023 task: ffff897bdb1cedd0 ti: ffff8906888e4000 task.ti: ffff8906888e4000 RIP: 0010:[<ffffffff811f4e07>] [<ffffffff811f4e07>] lookup_page_cgroup+0x47/0x50 RSP: 0018:ffff8906888e7d50 EFLAGS: 00000286 RAX: ffff88029744c2a0 RBX: ffffea01435e7d00 RCX: 0000000000000220 RDX: ffff89003ffbe220 RSI: 0000000000000287 RDI: ffffea01a2330a80 RBP: ffff8906888e7d50 R08: 0000000000000287 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8118eb20 R13: 0000000000000000 R14: ffffffff8118f44c R15: ffff8906888e7cf8 FS: 00002b236dd5b300(0000) GS:ffff88fc3fc40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002b753eca31e0 CR3: 0000033e4bbd6000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff8906888e7dd0 ffffffff811f1221 0000000200000000 ffff89fc3e67e500 0000000000000000 0000000000001000 ffff89fc3e67e440 0000000000000001 0000000100000003 ffff89fc3e67e400 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f1221>] mem_cgroup_reparent_charges+0x141/0x3c0 [<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360 [<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40 [<ffffffff8120a278>] vfs_rmdir+0xa8/0x100 [<ffffffff8120f315>] do_rmdir+0x1a5/0x200 [<ffffffff81692305>] ? do_page_fault+0x35/0x90 [<ffffffff812104c6>] SyS_rmdir+0x16/0x20 [<ffffffff81696b09>] system_call_fastpath+0x16/0x1b Code: c2 48 89 c1 48 c1 ea 16 48 c1 e9 0f 48 8b 14 d5 00 85 f4 81 48 85 d2 74 0a 83 e1 7f 48 c1 e1 05 48 01 ca 48 c1 e0 04 48 03 42 10 <5d> c3 0f 1f 80 00 00 00 00 0f 1f 44 00 00 48 89 fa 48 89 f8 48
NMI watchdog: BUG: soft lockup - CPU#55 stuck for 22s! [slurmstepd:47442] Modules linked in: squashfs loop overlay(T) osc(OE) mgc(OE) lustre(OE) lmv(OE) fld(OE) mdc(OE) fid(OE) lov(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic nfs_layout_nfsv41_files crypto_null libcfs(OE) dell_rbu rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log ses enclosure sd_mod sg dm_mod intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd uas serio_raw pcspkr usb_storage smartpqi scsi_transport_sas hpilo hpwdt nvme shpchp lpc_ich wmi acpi_power_meter rpcrdma ib_isert iscsi_target_mod target_core_mod crc_t10dif crct10dif_generic ib_iser libiscsi scsi_transport_iscsi ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ipmi_devintf hfi1(OE) i2c_algo_bit i2c_core bnxt_en crct10dif_pclmul crct10dif_common rdmavt(OE) crc32c_intel ib_core ipmi_si fjes ipmi_msghandler CPU: 55 PID: 47442 Comm: slurmstepd Tainted: G W OEL ------------ T 3.10.0-514.10.2.el7.x86_64 #1 Hardware name: HPE ProLiant DL560 Gen10/ProLiant DL560 Gen10, BIOS U34 03/16/2023 task: ffff8b3514441f60 ti: ffff8b15baa80000 task.ti: ffff8b15baa80000 RIP: 0010:[<ffffffff811f4e07>] [<ffffffff811f4e07>] lookup_page_cgroup+0x47/0x50 RSP: 0018:ffff8b15baa83d50 EFLAGS: 00000286 RAX: ffff8b01ff9de630 RBX: ffffea0f85773940 RCX: 0000000000000240 RDX: ffff8c003ffc7240 RSI: 0000000000000283 RDI: ffffea0d025798c0 RBP: ffff8b15baa83d50 R08: 0000000000000283 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8118eb20 R13: 0000000000000000 R14: ffffffff8118f44c R15: ffff8b15baa83cf8 FS: 00002b832e349300(0000) GS:ffff8c003f640000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f2792588258 CR3: 0000022727172000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff8b15baa83dd0 ffffffff811f1221 0000000200000003 ffff89f67eb05500 0000000000000003 0000000000001000 ffff89f67eb05440 0000000000000001 0000000100000002 ffff89f67eb05400 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f1221>] mem_cgroup_reparent_charges+0x141/0x3c0 [<ffffffff811f1654>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d257>] cgroup_destroy_locked+0xe7/0x360 [<ffffffff8110d4f2>] cgroup_rmdir+0x22/0x40 [<ffffffff8120a278>] vfs_rmdir+0xa8/0x100 [<ffffffff8120f315>] do_rmdir+0x1a5/0x200 [<ffffffff81254d11>] ? SyS_flock+0x71/0x1e0 [<ffffffff812104c6>] SyS_rmdir+0x16/0x20 [<ffffffff81696b09>] system_call_fastpath+0x16/0x1b Code: c2 48 89 c1 48 c1 ea 16 48 c1 e9 0f 48 8b 14 d5 00 85 f4 81 48 85 d2 74 0a 83 e1 7f 48 c1 e1 05 48 01 ca 48 c1 e0 04 48 03 42 10 <5d> c3 0f 1f 80 00 00 00 00 0f 1f 44 00 00 48 89 fa 48 89 f8 48
NMI watchdog: BUG: soft lockup - CPU#43 stuck for 22s! [runc:21043] Modules linked in: xt_nat veth nvidia_uvm(OE) fuse xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_addrtype iptable_filter br_netfilter bridge stp llc mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) dm_mirror dm_region_hash dm_log dm_mod nvidia_drm(POE) nvidia_modeset(POE) iTCO_wdt iTCO_vendor_support nvidia(POE) sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd squashfs loop snd_hda_codec_hdmi snd_hda_intel snd_hda_codec pcspkr snd_hda_core snd_hwdep joydev snd_seq snd_seq_device snd_pcm snd_timer sg snd i2c_i801 soundcore lpc_ich mei_me mei ioatdma ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter knem(OE) auth_rpcgss sunrpc ip_tables xfs libcrc32c mlx4_ib(OE) mlx4_en(OE) sd_mod crc_t10dif crct10dif_generic mxm_wmi mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm igb mlx4_core(OE) crct10dif_pclmul crct10dif_common crc32c_intel drm ahci dca libahci i2c_algo_bit mlx5_core(OE) libata mlxfw(OE) drm_panel_orientation_quirks devlink mlx_compat(OE) ptp pps_core wmi overlay(T) CPU: 43 PID: 21043 Comm: runc Kdump: loaded Tainted: P OEL ------------ T 3.10.0-957.el7.x86_64 #1 Hardware name: Supermicro SYS-4028GR-TR2-1-EC028/X10DRG-O+-CPU, BIOS 3.1 07/13/2018 task: ffff8efbc1a18000 ti: ffff8f1c3995c000 task.ti: ffff8f1c3995c000 RIP: 0010:[<ffffffffae636c1a>] [<ffffffffae636c1a>] lookup_page_cgroup+0x1a/0x50 RSP: 0018:ffff8f1c3995fd20 EFLAGS: 00000202 RAX: 000000000a7c78f2 RBX: ffffdfaba2513300 RCX: 0000000000000001 RDX: 000000000a7c78f2 RSI: 0000000000000283 RDI: 000000000a7c78f2 RBP: ffff8f1c3995fd20 R08: ffff8f13ee0d9000 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffae5c4930 R13: 0000000000000000 R14: ffffffffae5c524c R15: ffff8f1c3995fcc8 FS: 00007fca3fdfd740(0000) GS:ffff8f943f1c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000056067e79a680 CR3: 000000c668f1a000 CR4: 00000000003607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: [<ffffffffae632ec1>] mem_cgroup_reparent_charges+0x141/0x3c0 [<ffffffffae6332f4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffffae52573a>] cgroup_destroy_locked+0xea/0x370 [<ffffffffae5259e2>] cgroup_rmdir+0x22/0x40 [<ffffffffae64dc2c>] vfs_rmdir+0xdc/0x150 [<ffffffffae6530d1>] do_rmdir+0x1f1/0x220 [<ffffffffae647034>] ? SYSC_newfstatat+0x24/0x60 [<ffffffffae654335>] SyS_unlinkat+0x25/0x40 [<ffffffffaeb74ddb>] system_call_fastpath+0x22/0x27 Code: 00 48 8d 04 78 c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 2b 3d 64 e5 a0 00 55 48 89 e5 48 c1 ff 06 48 89 fa 48 89 f8 <48> c1 ea 16 48 c1 e8 0f 48 8b 14 d5 00 d5 5f af 48 85 d2 74 0a
NMI watchdog: BUG: soft lockup - CPU#46 stuck for 22s! [systemd:1] Modules linked in: binfmt_misc gsch(OE) redirfs(OE) dsa_filter(POE) vfat fat tcp_lp drbg ansi_cprng rmd160 crypto_null ip_vti ip_tunnel af_key ah6 ah4 esp6 esp4 xfrm4_mode_beet xfrm4_tunnel tunnel4 xfrm4_mode_tunnel xfrm4_mode_transport xfrm6_mode_transport xfrm6_mode_ro xfrm6_mode_beet xfrm6_mode_tunnel ipcomp ipcomp6 xfrm6_tunnel tunnel6 xfrm_ipcomp cmac camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 cast6_avx_x86_64 cast6_generic cast5_avx_x86_64 cast5_generic cast_common deflate cts gcm ccm serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic blowfish_generic blowfish_x86_64 blowfish_common twofish_generic twofish_avx_x86_64 twofish_x86_64_3way xts twofish_x86_64 twofish_common xcbc sha256_mb sha512_ssse3 sha512_generic sha512_mb mcryptd des_generic veth xt_addrtype br_netfilter overlay(T) fuse uas usb_storage tcp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel ext4 kvm mbcache jbd2 irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel i40iw snd_hda_codec ixgbe ib_core snd_hda_core joydev snd_hwdep ipmi_ssif mdio snd_seq dca pcspkr snd_seq_device snd_pcm sg snd_timer mei_me snd mei i2c_i801 lpc_ich shpchp soundcore wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_cpufreq acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common i40e ahci crc32c_intel drm libahci megaraid_sas libata ptp pps_core i2c_core CPU: 46 PID: 1 Comm: systemd Kdump: loaded Tainted: P OEL ------------ T 3.10.0-862.el7.x86_64 #1 Hardware name: Supermicro X11DAi-N/X11DAi-N, BIOS 2.1 07/03/2018 task: ffff925c676b0000 ti: ffff921d27698000 task.ti: ffff921d27698000 RIP: 0010:[<ffffffff99a10c84>] [<ffffffff99a10c84>] lookup_page_cgroup+0x44/0x50 RSP: 0018:ffff921d2769bd00 EFLAGS: 00000286 RAX: ffff925c914de8a0 RBX: ffffee2798f4bdc0 RCX: 0000000000000001 RDX: ffff929c3ffcdb00 RSI: 0000000000000287 RDI: 0000000006ec5e8a RBP: ffff921d2769bd00 R08: ffff929a86faac00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff999a2400 R13: 0000000000000000 R14: ffffffff999a2d1c R15: ffff921d2769bca8 FS: 00007fa8acac5940(0000) GS:ffff929b3de80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000008dea0d CR3: 0000004035ee0000 CR4: 00000000007607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace: [<ffffffff99a0cfa1>] mem_cgroup_reparent_charges+0x141/0x3c0 [<ffffffff99a0d3d4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff9991c1da>] cgroup_destroy_locked+0xea/0x370 [<ffffffff9991c482>] cgroup_rmdir+0x22/0x40 [<ffffffff99a295ac>] vfs_rmdir+0xdc/0x150 [<ffffffff99a2cb31>] do_rmdir+0x1f1/0x220 [<ffffffff99a2dd66>] SyS_rmdir+0x16/0x20 [<ffffffffc0fd0aa9>] gsch_rmdir_hook_fn+0x49/0x70 [gsch] [<ffffffff99f1f7d5>] system_call_fastpath+0x1c/0x21 Code: f8 48 c1 ea 16 48 c1 e8 0f 48 8b 14 d5 c0 a5 9d 9a 48 85 d2 74 0a 83 e0 7f 48 c1 e0 05 48 01 c2 48 89 f8 48 c1 e0 04 48 03 42 10 <5d> c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 fa 48
NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [runc:84831] Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop ipt_REJECT nf_reject_ipv4 devlink xt_multiport iptable_filter binfmt_misc overlay sunrpc iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif pcspkr ipmi_si ipmi_devintf sg ipmi_msghandler wmi acpi_power_meter hpilo hpwdt ioatdma lpc_ich shpchp dca pcc_cpufreq dm_multipath ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw tg3 hpsa i2c_core ptp scsi_transport_sas pps_core dm_mirror dm_region_hash dm_log dm_mod CPU: 26 PID: 84831 Comm: runc Kdump: loaded Tainted: G L ------------ 3.10.0-862.14.4.el7.x86_64 #1 Hardware name: HP ProLiant DL560 Gen8, BIOS P77 08/03/2014 task: ffff98302ba9dee0 ti: ffff98144b16c000 task.ti: ffff98144b16c000 RIP: 0010:[<ffffffffbb615137>] [<ffffffffbb615137>] lookup_page_cgroup+0x57/0x60 RSP: 0018:ffff98144b16fd20 EFLAGS: 00000282 RAX: ffff98143c744dd0 RBX: fffffde71b999cc0 RCX: 000000000000023d RDX: ffff9821bffd87a0 RSI: 0000000000000287 RDI: 00000000011ec4dd RBP: ffff98144b16fd20 R08: ffff9806ea88d400 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffffbb5a5160 R13: 0000000000000000 R14: ffffffffbb5a5a7c R15: ffff98144b16fcc8 FS: 00007f5be89fa740(0000) GS:ffff983d86e80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000055c27a987147 CR3: 0000003800696000 CR4: 00000000001607e0 Call Trace: [<ffffffffbb611421>] mem_cgroup_reparent_charges+0x141/0x3c0 [<ffffffffbb611854>] mem_cgroup_css_offline+0x84/0x140 [<ffffffffbb51f4aa>] cgroup_destroy_locked+0xea/0x370 [<ffffffffbb51f752>] cgroup_rmdir+0x22/0x40 [<ffffffffbb62db0c>] vfs_rmdir+0xdc/0x150 [<ffffffffbb631091>] do_rmdir+0x1f1/0x220 [<ffffffffbb6252f4>] ? SYSC_newfstatat+0x24/0x60 [<ffffffffbb6322f5>] SyS_unlinkat+0x25/0x40 [<ffffffffbbb2579b>] system_call_fastpath+0x22/0x27 Code: c0 48 21 d0 48 98 48 8b 14 c5 c0 d5 5d bc 48 85 d2 74 0d 48 89 c8 83 e0 7f 48 c1 e0 05 48 01 c2 48 89 f8 48 c1 e0 04 48 03 42 10 <5d> c3 0f 1f 80 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f9 ba
NMI watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [slurmstepd:142796] Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) ebtable_filter devlink ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass dcdbas joydev iTCO_wdt iTCO_vendor_support crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common i40e(OE) ahci megaraid_sas libahci tg3(OE) ptp libata pps_core [last unloaded: ipmi_msghandler] CPU: 29 PID: 142796 Comm: slurmstepd Tainted: G OE ------------ 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 1.3.7 02/09/2018 task: ffff885ecc5b6eb0 ti: ffff884cbad68000 task.ti: ffff884cbad68000 RIP: 0010:[<ffffffff811f6ed4>] [<ffffffff811f6ed4>] lookup_page_cgroup+0x44/0x50 RSP: 0018:ffff884cbad6bd50 EFLAGS: 00000282 RAX: ffff88309e64beb0 RBX: ffffea0122f68100 RCX: 0000000000000001 RDX: ffff88603ffce760 RSI: 0000000000000283 RDI: 00000000051dcbeb RBP: ffff884cbad6bd50 R08: ffff882f44a40400 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590 R13: 0000000000000000 R14: ffffffff81190eac R15: ffff884cbad6bcf8 FS: 00002aaaaaafd180(0000) GS:ffff885f7c180000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000bb6928 CR3: 0000004c9753f000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff884cbad6bdd0 ffffffff811f3341 0000000200000001 ffff885f0fb0ad00 0000000000000001 0000000000001000 ffff885f0fb0ac40 0000000000000001 0000000100000003 ffff885f0fb0ac00 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f3341>] mem_cgroup_reparent_charges+0x141/0x3c0 [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff816b0325>] ? do_page_fault+0x35/0x90 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: f8 48 c1 ea 16 48 c1 e8 0f 48 8b 14 d5 00 9a f9 81 48 85 d2 74 0a 83 e0 7f 48 c1 e0 05 48 01 c2 48 89 f8 48 c1 e0 04 48 03 42 10 <5d> c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 fa 48
NMI watchdog: BUG: soft lockup - CPU#24 stuck for 23s! [slurmstepd:103270] Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge 8021q garp mrp stp llc overlay(T) iTCO_wdt iTCO_vendor_support ip6_tables iptable_filter coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses pcspkr joydev enclosure cdc_ether scsi_transport_sas usbnet mii sg mei_me mei i2c_i801 shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace binfmt_misc sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul ttm crct10dif_common crc32c_intel drm serio_raw ahci libahci mlx5_core(OE) libata mlxfw(OE) ptp megaraid_sas pps_core auxiliary(OE) mlx_compat(OE) devlink i2c_core [last unloaded: tracedev] CPU: 24 PID: 103270 Comm: slurmstepd Tainted: G W OEL ------------ T 3.10.0-693.5.2.el7.x86_64 #1 Hardware name: Lenovo ThinkSystem SR650 V2/7Z73CTO1WW, BIOS AFE112A-1.02 05/10/2021 task: ffff89f7c83a0000 ti: ffff8835f0514000 task.ti: ffff8835f0514000 RIP: 0010:[<ffffffff811f6e2f>] [<ffffffff811f6e2f>] lookup_page_cgroup+0x2f/0x50 RSP: 0018:ffff8835f0517d50 EFLAGS: 00000286 RAX: 0000000000003e82 RBX: ffffea06bbf52640 RCX: 0000000000000001 RDX: ffff8a003ff59000 RSI: 0000000000000287 RDI: 000000001f412a02 RBP: ffff8835f0517d50 R08: ffff89f8237cb000 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff811905c0 R13: 0000000000000000 R14: ffffffff81190edc R15: ffff8835f0517cf8 FS: 00002ab6a78c3fc0(0000) GS:ffff89f83e000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f2651513000 CR3: 0000017a2fd05000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff8835f0517dd0 ffffffff811f32b1 0000000200000000 ffff8967852e5100 0000000000000000 0000000000001000 ffff8967852e5040 0000000000000001 0000000100000003 ffff8967852e5000 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f32b1>] mem_cgroup_reparent_charges+0x141/0x3c0 [<ffffffff811f36e4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d767>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da12>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f5a6>] vfs_rmdir+0xd6/0x150 [<ffffffff81212241>] do_rmdir+0x1f1/0x220 [<ffffffff81213446>] SyS_rmdir+0x16/0x20 [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b Code: 00 48 2b 3d dc cd 82 00 55 48 89 e5 48 c1 ff 06 48 89 fa 48 89 f8 48 c1 ea 16 48 c1 e8 0f 48 8b 14 d5 00 9a f9 81 48 85 d2 74 0a <83> e0 7f 48 c1 e0 05 48 01 c2 48 89 f8 48 c1 e0 04 48 03 42 10
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [slurmstepd:79745] Modules linked in: squashfs loop overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs beegfs(OE) fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm i40iw mlx5_ib ib_core dm_mirror dm_region_hash dm_log dm_mod vfat fat nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) iTCO_wdt iTCO_vendor_support skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm nvidia(POE) irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev ipmi_ssif sg mei_me mei shpchp i2c_i801 lpc_ich ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_cpufreq acpi_pad acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper mlx5_core crct10dif_pclmul crct10dif_common syscopyarea crc32c_intel sysfillrect sysimgblt fb_sys_fops i40e ttm drm ahci libahci devlink ptp libata pps_core i2c_core wmi CPU: 6 PID: 79745 Comm: slurmstepd Tainted: P B OEL ------------ T 3.10.0-693.11.6.el7.x86_64 #1 Hardware name: Supermicro SYS-4029GP-TRT2/X11DPG-OT-CPU, BIOS 3.3 02/21/2020 task: ffff883944e7af70 ti: ffff8802edb9c000 task.ti: ffff8802edb9c000 RIP: 0010:[<ffffffff811f8de0>] [<ffffffff811f8de0>] lookup_page_cgroup+0x10/0x50 RSP: 0018:ffff8802edb9fd20 EFLAGS: 00000202 RAX: 0000000000000287 RBX: ffffea00e4819780 RCX: 0000000000000001 RDX: 0000000000000001 RSI: 0000000000000287 RDI: 00000000a47d7b80 RBP: ffff8802edb9fd20 R08: ffff883f77b16800 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff811922b0 R13: 0000000000000000 R14: ffffffff81192bcc R15: ffff8802edb9fcc8 FS: 00002b48a74b95c0(0000) GS:ffff883f7e780000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f175309c9e0 CR3: 0000006af42c6000 CR4: 00000000003607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: [<ffffffff811f5281>] mem_cgroup_reparent_charges+0x141/0x3c0 [<ffffffff811f56b4>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110f467>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110f712>] cgroup_rmdir+0x22/0x40 [<ffffffff812114f6>] vfs_rmdir+0xd6/0x150 [<ffffffff81214191>] do_rmdir+0x1f1/0x220 [<ffffffff81215396>] SyS_rmdir+0x16/0x20 [<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b Code: 48 c1 e0 0c 48 03 05 78 3e 83 00 48 8d 04 78 c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 2b 3d 4c 3e 83 00 55 48 89 e5 <48> c1 ff 06 48 89 fa 48 89 f8 48 c1 ea 16 48 c1 e8 0f 48 8b 14
NMI watchdog: BUG: soft lockup - CPU#33 stuck for 22s! [slurmstepd:36575] Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx4_ib ib_core intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support pcspkr sg sb_edac edac_core lpc_ich hpilo ipmi_ssif hpwdt ioatdma shpchp ipmi_si acpi_power_meter pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mlx4_en nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) nvidia(POE) crct10dif_pclmul crct10dif_common crc32c_intel mgag200 ipmi_msghandler mlx4_core drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt ttm fb_sys_fops ahci igb hpsa devlink libahci drm ptp pps_core libata dca i2c_algo_bit mpt2sas i2c_core raid_class scsi_transport_sas wmi fjes dm_mirror dm_region_hash dm_log dm_mod CPU: 33 PID: 36575 Comm: slurmstepd Tainted: P OE ------------ 3.10.0-514.21.1.el7.x86_64 #1 Hardware name: HP ProLiant SL270s Gen8 SE/, BIOS P75 09/18/2013 task: ffff881f59f10fb0 ti: ffff881d03ac0000 task.ti: ffff881d03ac0000 RIP: 0010:[<ffffffff811f52f5>] [<ffffffff811f52f5>] lookup_page_cgroup+0x35/0x50 RSP: 0018:ffff881d03ac3d50 EFLAGS: 00000286 RAX: 0000000002d4f4ff RBX: ffffea00c78e7a80 RCX: 00000000000005a9 RDX: ffff88403ffd5000 RSI: 0000000000000287 RDI: ffffea00b53d3fc0 RBP: ffff881d03ac3d50 R08: 0000000000000287 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff8118f010 R13: 0000000000000000 R14: ffffffff8118f93c R15: ffff881d03ac3cf8 FS: 00007f7b964c2780(0000) GS:ffff88403f740000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f8bf5c20000 CR3: 0000001fa8afa000 CR4: 00000000001407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Stack: ffff881d03ac3dd0 ffffffff811f1711 0000000200000001 ffff880df7b4fd00 0000000000000001 0000000000001000 ffff880df7b4fc40 0000000000000001 0000000100000002 ffff880df7b4fc00 0000000000000000 0000000000000000 Call Trace: [<ffffffff811f1711>] mem_cgroup_reparent_charges+0x141/0x3c0 [<ffffffff811f1b44>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d6e7>] cgroup_destroy_locked+0xe7/0x360 [<ffffffff8110d982>] cgroup_rmdir+0x22/0x40 [<ffffffff8120a778>] vfs_rmdir+0xa8/0x100 [<ffffffff8120f815>] do_rmdir+0x1a5/0x200 [<ffffffff812109c6>] SyS_rmdir+0x16/0x20 [<ffffffff816975c9>] system_call_fastpath+0x16/0x1b Code: 00 00 16 00 00 55 48 01 f8 48 c1 f8 06 48 89 e5 48 89 c2 48 89 c1 48 c1 ea 16 48 c1 e9 0f 48 8b 14 d5 80 c5 f4 81 48 85 d2 74 0a <83> e1 7f 48 c1 e1 05 48 01 ca 48 c1 e0 04 48 03 42 10 5d c3 0f
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [slurmstepd:93175] Modules linked in: fuse rpcsec_gss_krb5 nfsv4 dns_resolver 8021q garp mrp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 nfs fscache dm_mirror dm_region_hash dm_log dm_mod sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel mxm_wmi iTCO_wdt iTCO_vendor_support aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr mei_me i2c_i801 sg mei ioatdma lpc_ich shpchp ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter acpi_pad acpi_cpufreq rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm nfsd ib_cm iw_cm auth_rpcgss ib_core nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt igb fb_sys_fops ttm crct10dif_pclmul ahci ptp crct10dif_common drm crc32c_intel libahci pps_core libata dca i2c_algo_bit i2c_core CPU: 1 PID: 93175 Comm: slurmstepd Tainted: G L ------------ 3.10.0-693.2.2.el7.x86_64 #1 Hardware name: Advanced Clustering S2600TPR/S2600TPR, BIOS SE5C610.86B.01.01.0016.C1.033120161139 03/31/2016 task: ffff883ffafb0fd0 ti: ffff8824e59ec000 task.ti: ffff8824e59ec000 RIP: 0010:[<ffffffff811f6e9c>] [<ffffffff811f6e9c>] lookup_page_cgroup+0xc/0x50 RSP: 0018:ffff8824e59efd58 EFLAGS: 00000206 RAX: 0000000000000283 RBX: ffffea00c23fc540 RCX: 0000000000000001 RDX: 0000000000000001 RSI: 0000000000000283 RDI: 00000000c23fc6c0 RBP: ffff8824e59efdd0 R08: ffff8810ec9f5c00 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000000001 R12: ffffffff81190590 R13: 0000000000000000 R14: ffffffff81190eac R15: ffff8824e59efcf8 FS: 00002af8691b9d00(0000) GS:ffff881ffe640000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000056246f197f62 CR3: 0000001e8aa0e000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffffffff811f3341 0000000200000001 ffff8825dce36d00 0000000000000001 0000000000001000 ffff8825dce36c40 0000000000000001 0000000100000002 ffff8825dce36c00 0000000000000000 0000000000000000 ffff880f8b2bde00 Call Trace: [<ffffffff811f3341>] ? mem_cgroup_reparent_charges+0x141/0x3c0 [<ffffffff811f3774>] mem_cgroup_css_offline+0x84/0x140 [<ffffffff8110d757>] cgroup_destroy_locked+0xe7/0x370 [<ffffffff8110da02>] cgroup_rmdir+0x22/0x40 [<ffffffff8120f636>] vfs_rmdir+0xd6/0x150 [<ffffffff812122d1>] do_rmdir+0x1f1/0x220 [<ffffffff816a8fad>] ? __schedule+0x39d/0x8b0 [<ffffffff812134d6>] SyS_rmdir+0x16/0x20 [<ffffffff816b5009>] system_call_fastpath+0x16/0x1b Code: 48 c1 f8 06 48 c1 e0 0c 48 03 05 78 cd 82 00 48 8d 04 78 c3 90 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 2b 3d 4c cd 82 00 <55> 48 89 e5 48 c1 ff 06 48 89 fa 48 89 f8 48 c1 ea 16 48 c1 e8
BUG: soft lockup - CPU#0 stuck for 22s! [slurmstepd:27354] Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver intel_powerclamp coretemp intel_rapl kvm_intel kvm crc32_pclmul crc32c_intel ast ghash_clmulni_intel syscopyarea aesni_intel sysfillrect lrw sysimgblt ttm gf128mul glue_helper ablk_helper cryptd drm_kms_helper drm iTCO_wdt iTCO_vendor_support pcspkr mei_me mei lpc_ich i2c_i801 shpchp mfd_core ipmi_devintf wmi ipmi_si ipmi_msghandler acpi_pad acpi_power_meter xprtrdma ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm nfsd auth_rpcgss binfmt_misc ip_tables raid1 raid0 bridge stp llc hpilo xfs isci libsas qla3xxx hpsa sg e1000e ixgbe megaraid_sas aacraid aic79xx ata_piix sd_mod crc_t10dif mpt2sas mptspi mptscsih mptbase scsi_transport_spi arcmsr sata_nv sata_svw bnx2x mdio libcrc32c bnx2 ext4 mbcache jbd2 sata_sil tg3 e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun mlx4_en mlx4_ib vxlan ib_sa ip6_udp_tunnel udp_tunnel ib_mad ib_core ib_addr crct10dif_pclmul crct10dif_common igb ahci mlx4_core i2c_algo_bit libahci i2c_core mpt3sas libata dca ptp raid_class pps_core scsi_transport_sas dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler] CPU: 0 PID: 27354 Comm: slurmstepd Not tainted 3.10.0-327.3.1.el7.x86_64 #1 Hardware name: SGI.COM C2112-4GP3/X10DRT-P-Series, BIOS 2.0a 05/09/2016 task: ffff887923e25c00 ti: ffff887f1b1c0000 task.ti: ffff887f1b1c0000 RIP: 0010:[<ffffffff811d57e3>] [<ffffffff811d57e3>] lookup_page_cgroup+0x43/0x50 RSP: 0018:ffff887f1b1c3d48 EFLAGS: 00000202 RAX: 000000004a706680 RBX: 0000000000000246 RCX: 00000000000009c0 RDX: ffff88807ffd49c0 RSI: 0000000000000287 RDI: ffffea0129c19a00 RBP: ffff887f1b1c3d48 R08: 0000000000000287 R09: 0000000000000001 R10: 0000000000000001 R11: 0000000000001000 R12: 0000000000000000 R13: ffffffff811773b8 R14: ffff887f1b1c3cf0 R15: ffff88807ffd7000 FS: 00002aaaaaafe540(0000) GS:ffff883f7fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00002aaaaf073580 CR3: 00000017e231a000 CR4: 00000000003407f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff887f1b1c3dd0 ffffffff811d1ad1 ffffc9004b539140 0000000200000001 ffffc9004b539108 ffffc9004b539040 0000000000000001 0000000000001000 0000000100000003 0000000000000001 ffffc9004b539000 0000000000000000 Call Trace: [<ffffffff811d1ad1>] mem_cgroup_reparent_charges+0x181/0x440 [<ffffffff811d1f0f>] mem_cgroup_css_offline+0x4f/0x100 [<ffffffff810fa347>] cgroup_destroy_locked+0xe7/0x360 [<ffffffff810fa5e2>] cgroup_rmdir+0x22/0x40 [<ffffffff811ea158>] vfs_rmdir+0xa8/0x100 [<ffffffff811ef0f5>] do_rmdir+0x1a5/0x200 [<ffffffff812331d1>] ? SyS_flock+0x71/0x1e0 [<ffffffff811f02a6>] SyS_rmdir+0x16/0x20 [<ffffffff816458c9>] system_call_fastpath+0x16/0x1b Code: 89 e5 48 89 c2 48 89 c1 48 c1 ea 16 48 c1 e9 0f 48 8b 14 d5 c0 b7 ed 81 48 85 d2 74 0a 83 e1 7f 48 c1 e1 05 48 01 ca 48 c1 e0 04 <48> 03 42 10 5d c3 0f 1f 80 00 00 00 00 0f 1f 44 00 00 48 89 fa