Info

Permalink
Function
native_queued_spin_lock_slowpath
First occurrence
2020-10-08
Last occurrence
2025-01-10
State
NEW
Components
kernel
Quality
-32

Statistics

Operating system Count
CentOS 7 642
Architecture Count
x86_64 642
Related packages Count
kernel 642
 0:3.10.0-957.27.2.el7 590
 0:3.10.0-862.14.4.el7 37
 0:3.10.0-1062.el7 4
 0:3.10.0-862.el7 4
 0:3.10.0-957.21.3.el7 2
 0:3.10.0-862.3.2.el7 1
 0:3.10.0-862.9.1.el7 1
 0:3.10.0-1062.4.1.el7 1
 0:3.10.0-693.el7 1
 0:3.10.0-693.11.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1115139
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff93508802
Build id: 3.10.0-862.el7.x86_64
0x122
2
queued_spin_lock_slowpath
vmlinux 0xffffffff93b0842a
Build id: 3.10.0-862.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff93b15680
Build id: 3.10.0-862.el7.x86_64
0x20
4
ext4_es_lru_del
ext4 0xffffffffc0de94a2
Build id: 3.10.0-862.el7.x86_64
0x32
5
ext4_clear_inode
ext4 0xffffffffc0dc7701
Build id: 3.10.0-862.el7.x86_64
0x41
6
ext4_free_inode
ext4 0xffffffffc0da1516
Build id: 3.10.0-862.el7.x86_64
0x116
7
? ext4_evict_inode
ext4 0xffffffffc0dab2e7
Build id: 3.10.0-862.el7.x86_64
0x337
8
? ext4_mark_inode_dirty
ext4 0xffffffffc0da8faf
Build id: 3.10.0-862.el7.x86_64
0x6f
9
ext4_evict_inode
ext4 0xffffffffc0dab3a7
Build id: 3.10.0-862.el7.x86_64
0x3f7
10
evict
vmlinux 0xffffffff93638054
Build id: 3.10.0-862.el7.x86_64
0xb4
11
iput
vmlinux 0xffffffff9363895c
Build id: 3.10.0-862.el7.x86_64
0xfc
12
do_unlinkat
vmlinux 0xffffffff9362cd0e
Build id: 3.10.0-862.el7.x86_64
0x1ae
13
SyS_unlink
vmlinux 0xffffffff9362ddc6
Build id: 3.10.0-862.el7.x86_64
0x16
14
system_call_fastpath
vmlinux 0xffffffff93b1f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [sftp-server:183459]
Modules linked in: fuse nvidia_uvm(OE) mst_pciconf(OE) mst_pci(OE) mmfs26(OE) sep4_1(OE) mmfslinux(OE) tracedev(OE) socperf2_0(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 tun bridge stp llc mpt3sas mpt2sas pax(OE) raid_class scsi_transport_sas mptctl mptbase ebtable_filter ebtables ip6table_filter ip6_tables rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack dm_mirror dm_region_hash dm_log dm_mod nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel
ext4 kvm irqbypass mbcache crc32_pclmul jbd2 ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg mei_me mei lpc_ich shpchp i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_cpufreq acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc knem(OE) binfmt_misc ip_tables xfs libcrc32c mlx4_ib(OE) ib_uverbs(OE) ib_core(OE) sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx4_core(OE) crct10dif_pclmul crct10dif_common crc32c_intel i40e ahci drm devlink libahci mlx_compat(OE) megaraid_sas libata ptp pps_core i2c_core wmi
CPU: 5 PID: 183459 Comm: sftp-server Kdump: loaded Tainted: P           OEL ------------   3.10.0-862.el7.x86_64 #1
Hardware name: Supermicro SYS-4029GP-TRT/X11DPG-OT-CPU, BIOS 3.3 02/21/2020
task: ffff95fa7ccbeeb0 ti: ffff96040ccb4000 task.ti: ffff96040ccb4000
RIP: 0010:[<ffffffff93508802>]  [<ffffffff93508802>] native_queued_spin_lock_slowpath+0x122/0x200
RSP: 0018:ffff96040ccb7cd8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff95ca32bd6bca RCX: 0000000000290000
RDX: ffff95f63f7d9700 RSI: 0000000000390101 RDI: ffff95f73676a480
RBP: ffff96040ccb7cd8 R08: ffff95f63f759700 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000678116fd R12: ffff95ca32bd6bc8
R13: ffffffff934bbde5 R14: ffff96040ccb7c50 R15: 0000000051611329
FS:  00002ae5397ec740(0000) GS:ffff95f63f740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055d29a908038 CR3: 000000438d2f0000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff93b0842a>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff93b15680>] _raw_spin_lock+0x20/0x30
[<ffffffffc0de94a2>] ext4_es_lru_del+0x32/0x70 [ext4]
[<ffffffffc0dc7701>] ext4_clear_inode+0x41/0x90 [ext4]
[<ffffffffc0da1516>] ext4_free_inode+0x116/0x610 [ext4]
[<ffffffffc0dab2e7>] ? ext4_evict_inode+0x337/0x4b0 [ext4]
[<ffffffffc0da8faf>] ? ext4_mark_inode_dirty+0x6f/0x210 [ext4]
[<ffffffffc0dab3a7>] ext4_evict_inode+0x3f7/0x4b0 [ext4]
[<ffffffff93638054>] evict+0xb4/0x180
[<ffffffff9363895c>] iput+0xfc/0x190
[<ffffffff9362cd0e>] do_unlinkat+0x1ae/0x2d0
[<ffffffff9362ddc6>] SyS_unlink+0x16/0x20
[<ffffffff93b1f7d5>] system_call_fastpath+0x1c/0x21
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 00 97 01 00 48 03 14 c5 a0 53 13 94 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 
Complete report #1092510
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffffb2d12452
Build id: 3.10.0-957.21.3.el7.x86_64
0x122
2
queued_spin_lock_slowpath
vmlinux 0xffffffffb335d28b
Build id: 3.10.0-957.21.3.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffb336b760
Build id: 3.10.0-957.21.3.el7.x86_64
0x20
4
ext4_es_lru_del
ext4 0xffffffffc0b31c32
Build id: 3.10.0-957.21.3.el7.x86_64
0x32
5
ext4_clear_inode
ext4 0xffffffffc0b0f951
Build id: 3.10.0-957.21.3.el7.x86_64
0x41
6
ext4_free_inode
ext4 0xffffffffc0ae85e6
Build id: 3.10.0-957.21.3.el7.x86_64
0x116
7
? ext4_evict_inode
ext4 0xffffffffc0af25f7
Build id: 3.10.0-957.21.3.el7.x86_64
0x317
8
? ext4_mark_inode_dirty
ext4 0xffffffffc0af033f
Build id: 3.10.0-957.21.3.el7.x86_64
0x6f
9
ext4_evict_inode
ext4 0xffffffffc0af26b7
Build id: 3.10.0-957.21.3.el7.x86_64
0x3d7
10
evict
vmlinux 0xffffffffb2e5f0b4
Build id: 3.10.0-957.21.3.el7.x86_64
0xb4
11
iput
vmlinux 0xffffffffb2e5f9bc
Build id: 3.10.0-957.21.3.el7.x86_64
0xfc
12
do_unlinkat
vmlinux 0xffffffffb2e53a6e
Build id: 3.10.0-957.21.3.el7.x86_64
0x1ae
13
? SYSC_newstat
vmlinux 0xffffffffb2e4777e
Build id: 3.10.0-957.21.3.el7.x86_64
0x3e
14
SyS_unlink
vmlinux 0xffffffffb2e54b26
Build id: 3.10.0-957.21.3.el7.x86_64
0x16
15
system_call_fastpath
vmlinux 0xffffffffb3375ddb
Build id: 3.10.0-957.21.3.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#28 stuck for 23s! [python:156553]
Modules linked in: binfmt_misc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log iTCO_wdt iTCO_vendor_support ext4 mbcache jbd2 dm_mod skx_edac intel_powerclamp coretemp snd_hda_codec_hdmi intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel snd_hda_codec_realtek lrw gf128mul glue_helper snd_hda_codec_generic ablk_helper cryptd snd_hda_intel pcspkr snd_hda_codec snd_hda_core joydev snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd i2c_i801 sg lpc_ich mei_me soundcore mei wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_cpufreq acpi_pad ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm crct10dif_pclmul ahci crct10dif_common drm crc32c_intel libahci libata i40e(OE) ptp pps_core drm_panel_orientation_quirks nfit libnvdimm fuse
CPU: 28 PID: 156553 Comm: python Tainted: G           OEL ------------   3.10.0-957.21.3.el7.x86_64 #1
Hardware name: Supermicro SYS-7039A-I/X11DAi-N, BIOS 3.1a 06/11/2019
task: ffff9d906f315140 ti: ffff9d9050be0000 task.ti: ffff9d9050be0000
RIP: 0010:[<ffffffffb2d12452>]  [<ffffffffb2d12452>] native_queued_spin_lock_slowpath+0x122/0x200
RSP: 0018:ffff9d9050be3cd8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff9daae04a39c2 RCX: 0000000000e10000
RDX: ffff9dc07dd5b780 RSI: 0000000001090101 RDI: ffff9dc072fc5c80
RBP: ffff9d9050be3cd8 R08: ffff9dc07dc1b780 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000675b0541 R12: ffff9daae04a39c0
R13: ffffffffb2cc2e35 R14: ffff9d9050be3c50 R15: 0000000030796b37
FS:  00002ae83f3d18c0(0000) GS:ffff9dc07dc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055e3a0b02f58 CR3: 0000001ed423e000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffb335d28b>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffffb336b760>] _raw_spin_lock+0x20/0x30
[<ffffffffc0b31c32>] ext4_es_lru_del+0x32/0x70 [ext4]
[<ffffffffc0b0f951>] ext4_clear_inode+0x41/0x90 [ext4]
[<ffffffffc0ae85e6>] ext4_free_inode+0x116/0x610 [ext4]
[<ffffffffc0af25f7>] ? ext4_evict_inode+0x317/0x470 [ext4]
[<ffffffffc0af033f>] ? ext4_mark_inode_dirty+0x6f/0x210 [ext4]
[<ffffffffc0af26b7>] ext4_evict_inode+0x3d7/0x470 [ext4]
[<ffffffffb2e5f0b4>] evict+0xb4/0x180
[<ffffffffb2e5f9bc>] iput+0xfc/0x190
[<ffffffffb2e53a6e>] do_unlinkat+0x1ae/0x2d0
[<ffffffffb2e4777e>] ? SYSC_newstat+0x3e/0x60
[<ffffffffb2e54b26>] SyS_unlink+0x16/0x20
[<ffffffffb3375ddb>] system_call_fastpath+0x22/0x27
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 a0 bc 94 b3 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 
Complete report #995456
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff82908806
Build id: 3.10.0-862.el7.x86_64
0x126
2
queued_spin_lock_slowpath
vmlinux 0xffffffff82f0842a
Build id: 3.10.0-862.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff82f15680
Build id: 3.10.0-862.el7.x86_64
0x20
4
ext4_es_lru_del
ext4 0xffffffffc05fc4a2
Build id: 3.10.0-862.el7.x86_64
0x32
5
ext4_clear_inode
ext4 0xffffffffc05da701
Build id: 3.10.0-862.el7.x86_64
0x41
6
ext4_free_inode
ext4 0xffffffffc05b4516
Build id: 3.10.0-862.el7.x86_64
0x116
7
? ext4_evict_inode
ext4 0xffffffffc05be2e7
Build id: 3.10.0-862.el7.x86_64
0x337
8
? ext4_mark_inode_dirty
ext4 0xffffffffc05bbfaf
Build id: 3.10.0-862.el7.x86_64
0x6f
9
ext4_evict_inode
ext4 0xffffffffc05be3a7
Build id: 3.10.0-862.el7.x86_64
0x3f7
10
evict
vmlinux 0xffffffff82a38054
Build id: 3.10.0-862.el7.x86_64
0xb4
11
iput
vmlinux 0xffffffff82a3895c
Build id: 3.10.0-862.el7.x86_64
0xfc
12
do_unlinkat
vmlinux 0xffffffff82a2cd0e
Build id: 3.10.0-862.el7.x86_64
0x1ae
13
? SYSC_newstat
vmlinux 0xffffffff82a20cce
Build id: 3.10.0-862.el7.x86_64
0x2e
14
? vfs_write
vmlinux 0xffffffff82a1ad92
Build id: 3.10.0-862.el7.x86_64
0x182
15
SyS_unlink
vmlinux 0xffffffff82a2ddc6
Build id: 3.10.0-862.el7.x86_64
0x16
16
system_call_fastpath
vmlinux 0xffffffff82f1f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [ChunkParser:45202]
Modules linked in: nfnetlink_queue nfnetlink_log bluetooth rfkill ip_set binfmt_misc nfnetlink br_netfilter bridge stp llc overlay(T) vfat fat ext4 mbcache jbd2 skx_edac coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support mei_me mei sg i2c_i801 joydev megaraid_sas pcspkr i2c_core shpchp lpc_ich nfit acpi_power_meter libnvdimm acpi_cpufreq xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mlx5_core crct10dif_pclmul crct10dif_common crc32c_intel ahci i40e libahci libata mlxfw devlink ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 28 PID: 45202 Comm: ChunkParser Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: XFUSION 2288H V5/BC11SPSCB10, BIOS 8.27 03/08/2022
task: ffff8cabb12c3f40 ti: ffff8ccc32a90000 task.ti: ffff8ccc32a90000
RIP: 0010:[<ffffffff82908806>]  [<ffffffff82908806>] native_queued_spin_lock_slowpath+0x126/0x200
RSP: 0018:ffff8ccc32a93cd8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8cab5a63c412 RCX: 0000000000e10000
RDX: ffff8ccb6fc59700 RSI: 0000000000490001 RDI: ffff8cf38ba00c80
RBP: ffff8ccc32a93cd8 R08: ffff8cf38eb19700 R09: 0000000000000000
R10: 0000000000000000 R11: 000000006678cc49 R12: ffff8cab5a63c410
R13: ffffffff828bbde5 R14: ffff8ccc32a93c50 R15: 000000003fa1bb42
FS:  00007f0d2249c700(0000) GS:ffff8cf38eb00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2c97e602e0 CR3: 0000004a12b6a000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff82f0842a>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff82f15680>] _raw_spin_lock+0x20/0x30
[<ffffffffc05fc4a2>] ext4_es_lru_del+0x32/0x70 [ext4]
[<ffffffffc05da701>] ext4_clear_inode+0x41/0x90 [ext4]
[<ffffffffc05b4516>] ext4_free_inode+0x116/0x610 [ext4]
[<ffffffffc05be2e7>] ? ext4_evict_inode+0x337/0x4b0 [ext4]
[<ffffffffc05bbfaf>] ? ext4_mark_inode_dirty+0x6f/0x210 [ext4]
[<ffffffffc05be3a7>] ext4_evict_inode+0x3f7/0x4b0 [ext4]
[<ffffffff82a38054>] evict+0xb4/0x180
[<ffffffff82a3895c>] iput+0xfc/0x190
[<ffffffff82a2cd0e>] do_unlinkat+0x1ae/0x2d0
[<ffffffff82a20cce>] ? SYSC_newstat+0x2e/0x60
[<ffffffff82a1ad92>] ? vfs_write+0x182/0x1f0
[<ffffffff82a2ddc6>] SyS_unlink+0x16/0x20
[<ffffffff82f1f7d5>] system_call_fastpath+0x1c/0x21
Code: 0d 48 98 83 e2 30 48 81 c2 00 97 01 00 48 03 14 c5 a0 53 53 83 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 
Complete report #946998
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff810fa332
Build id: 3.10.0-693.11.1.el7.x86_64
0x112
2
queued_spin_lock_slowpath
vmlinux 0xffffffff8169e6cf
Build id: 3.10.0-693.11.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff816abc20
Build id: 3.10.0-693.11.1.el7.x86_64
0x20
4
ext4_es_lru_del
ext4 0xffffffffc0294932
Build id: 3.10.0-693.11.1.el7.x86_64
0x32
5
ext4_clear_inode
ext4 0xffffffffc0272ac1
Build id: 3.10.0-693.11.1.el7.x86_64
0x41
6
ext4_free_inode
ext4 0xffffffffc024cc16
Build id: 3.10.0-693.11.1.el7.x86_64
0x116
7
? ext4_evict_inode
ext4 0xffffffffc0256857
Build id: 3.10.0-693.11.1.el7.x86_64
0x337
8
? ext4_mark_inode_dirty
ext4 0xffffffffc025453f
Build id: 3.10.0-693.11.1.el7.x86_64
0x6f
9
ext4_evict_inode
ext4 0xffffffffc0256917
Build id: 3.10.0-693.11.1.el7.x86_64
0x3f7
10
evict
vmlinux 0xffffffff8121d5a9
Build id: 3.10.0-693.11.1.el7.x86_64
0xa9
11
iput
vmlinux 0xffffffff8121deb9
Build id: 3.10.0-693.11.1.el7.x86_64
0xf9
12
do_unlinkat
vmlinux 0xffffffff8121229e
Build id: 3.10.0-693.11.1.el7.x86_64
0x1ae
13
? task_work_run
vmlinux 0xffffffff810ad25c
Build id: 3.10.0-693.11.1.el7.x86_64
0xac
14
SyS_unlink
vmlinux 0xffffffff81213326
Build id: 3.10.0-693.11.1.el7.x86_64
0x16
15
system_call_fastpath
vmlinux 0xffffffff816b5089
Build id: 3.10.0-693.11.1.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [afl-showmap:49427]
Modules linked in: veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) nf_conntrack_ftp udp_diag tcp_diag inet_diag mpt3sas mpt2sas raid_class xt_CHECKSUM scsi_transport_sas iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 mptctl nf_nat mptbase nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod nvidia_drm(POE) sb_edac edac_core nvidia_modeset(POE) intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd nvidia(POE) snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel
snd_hda_codec snd_hda_core snd_hwdep snd_seq joydev snd_seq_device snd_pcm pcspkr sg snd_timer mei_me snd mei ioatdma i2c_i801 shpchp lpc_ich soundcore ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm igb drm ahci libahci mxm_wmi libata crct10dif_pclmul crct10dif_common ptp crc32c_intel pps_core megaraid_sas dca i2c_algo_bit i2c_core wmi
CPU: 14 PID: 49427 Comm: afl-showmap Tainted: P           OEL ------------ T 3.10.0-693.11.1.el7.x86_64 #1
Hardware name: Powerleader PR4764GW/X10DRG-Q, BIOS 2.0a 08/29/2016
task: ffff88258f3b6eb0 ti: ffff8830303d4000 task.ti: ffff8830303d4000
RIP: 0010:[<ffffffff810fa332>]  [<ffffffff810fa332>] native_queued_spin_lock_slowpath+0x112/0x1e0
RSP: 0018:ffff8830303d7d08  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8830303d7c98 RCX: 0000000000710000
RDX: ffff881fff957880 RSI: 0000000000690101 RDI: ffff883ffaa14480
RBP: ffff8830303d7d08 R08: ffff883ffee17880 R09: 0000000000000000
R10: ffffe8ffff600c50 R11: ffffea00863edc40 R12: ffff880201914558
R13: ffff88218fb71028 R14: ffff8830303d7c80 R15: 00000000ebe51637
FS:  00007f2e7a6500c0(0000) GS:ffff883ffee00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000004b7450 CR3: 0000003df9f7b000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8830303d7d18 ffffffff8169e6cf ffff8830303d7d28 ffffffff816abc20
ffff8830303d7d50 ffffffffc0294932 ffff880201914558 ffff883fe6b04450
0000000001f50013 ffff8830303d7d68 ffffffffc0272ac1 ffff883ffaa14800
Call Trace:
[<ffffffff8169e6cf>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff816abc20>] _raw_spin_lock+0x20/0x30
[<ffffffffc0294932>] ext4_es_lru_del+0x32/0x70 [ext4]
[<ffffffffc0272ac1>] ext4_clear_inode+0x41/0x90 [ext4]
[<ffffffffc024cc16>] ext4_free_inode+0x116/0x610 [ext4]
[<ffffffffc0256857>] ? ext4_evict_inode+0x337/0x4b0 [ext4]
[<ffffffffc025453f>] ? ext4_mark_inode_dirty+0x6f/0x210 [ext4]
[<ffffffffc0256917>] ext4_evict_inode+0x3f7/0x4b0 [ext4]
[<ffffffff8121d5a9>] evict+0xa9/0x180
[<ffffffff8121deb9>] iput+0xf9/0x190
[<ffffffff8121229e>] do_unlinkat+0x1ae/0x2d0
[<ffffffff810ad25c>] ? task_work_run+0xac/0xf0
[<ffffffff81213326>] SyS_unlink+0x16/0x20
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 78 01 00 48 03 14 c5 e0 fd b0 81 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 
Complete report #401716
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff810fa322
Build id: 3.10.0-693.el7.x86_64
0x112
2
queued_spin_lock_slowpath
vmlinux 0xffffffff8169e5ff
Build id: 3.10.0-693.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff816abb50
Build id: 3.10.0-693.el7.x86_64
0x20
4
ext4_es_lru_del
ext4 0xffffffffc07b2922
Build id: 3.10.0-693.el7.x86_64
0x32
5
ext4_clear_inode
ext4 0xffffffffc0790ab1
Build id: 3.10.0-693.el7.x86_64
0x41
6
ext4_free_inode
ext4 0xffffffffc076ac06
Build id: 3.10.0-693.el7.x86_64
0x116
7
? ext4_evict_inode
ext4 0xffffffffc0774847
Build id: 3.10.0-693.el7.x86_64
0x337
8
? ext4_mark_inode_dirty
ext4 0xffffffffc077252f
Build id: 3.10.0-693.el7.x86_64
0x6f
9
ext4_evict_inode
ext4 0xffffffffc0774907
Build id: 3.10.0-693.el7.x86_64
0x3f7
10
evict
vmlinux 0xffffffff8121d619
Build id: 3.10.0-693.el7.x86_64
0xa9
11
iput
vmlinux 0xffffffff8121df29
Build id: 3.10.0-693.el7.x86_64
0xf9
12
do_unlinkat
vmlinux 0xffffffff8121242e
Build id: 3.10.0-693.el7.x86_64
0x1ae
13
? task_work_run
vmlinux 0xffffffff810ad24c
Build id: 3.10.0-693.el7.x86_64
0xac
14
SyS_unlink
vmlinux 0xffffffff812134b6
Build id: 3.10.0-693.el7.x86_64
0x16
15
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#55 stuck for 22s! [vim:327867]
Modules linked in: binfmt_misc l2tp_ppp l2tp_netlink l2tp_core pppox ppp_generic slhc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 dell_smbios sparse_keymap iTCO_wdt iTCO_vendor_support dcdbas skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi nfit libnvdimm kvm_intel kvm ipmi_ssif irqbypass joydev i2c_i801 pcspkr sg lpc_ich
shpchp ipmi_si ipmi_devintf ipmi_msghandler mei_me mei acpi_power_meter acpi_pad fuse ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd 8021q scsi_transport_iscsi mgag200 garp stp llc drm_kms_helper mrp syscopyarea sysfillrect sysimgblt fb_sys_fops ttm igb drm ahci dca libahci ptp pps_core libata i2c_algo_bit megaraid_sas bnxt_en i2c_core wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod
CPU: 55 PID: 327867 Comm: vim Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge T640/07978V, BIOS 2.9.4 11/09/2020
task: ffff887eec476eb0 ti: ffff883aedfd0000 task.ti: ffff883aedfd0000
RIP: 0010:[<ffffffff810fa322>]  [<ffffffff810fa322>] native_queued_spin_lock_slowpath+0x112/0x1e0
RSP: 0018:ffff883aedfd3d08  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff887d08b6cf0a RCX: 0000000001b90000
RDX: ffff887f6be57880 RSI: 0000000000190101 RDI: ffff887f5e3b1480
RBP: ffff883aedfd3d08 R08: ffff887f6c4d7880 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000062614eef R12: ffff887d08b6cf08
R13: ffffffff810b18d5 R14: ffff883aedfd3c80 R15: 000000004a7f0101
FS:  00007f73f99be600(0000) GS:ffff887f6c4c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000004bf310 CR3: 0000004f79bb5000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff883aedfd3d18 ffffffff8169e5ff ffff883aedfd3d28 ffffffff816abb50
ffff883aedfd3d50 ffffffffc07b2922 ffff886b9cebf5b8 ffff887f35520210
000000001f71030e ffff883aedfd3d68 ffffffffc0790ab1 ffff887f5e3b2000
Call Trace:
[<ffffffff8169e5ff>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff816abb50>] _raw_spin_lock+0x20/0x30
[<ffffffffc07b2922>] ext4_es_lru_del+0x32/0x70 [ext4]
[<ffffffffc0790ab1>] ext4_clear_inode+0x41/0x90 [ext4]
[<ffffffffc076ac06>] ext4_free_inode+0x116/0x610 [ext4]
[<ffffffffc0774847>] ? ext4_evict_inode+0x337/0x4b0 [ext4]
[<ffffffffc077252f>] ? ext4_mark_inode_dirty+0x6f/0x210 [ext4]
[<ffffffffc0774907>] ext4_evict_inode+0x3f7/0x4b0 [ext4]
[<ffffffff8121d619>] evict+0xa9/0x180
[<ffffffff8121df29>] iput+0xf9/0x190
[<ffffffff8121242e>] do_unlinkat+0x1ae/0x2d0
[<ffffffff810ad24c>] ? task_work_run+0xac/0xf0
[<ffffffff812134b6>] SyS_unlink+0x16/0x20
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 78 01 00 48 03 14 c5 e0 fd b0 81 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 
Complete report #398029
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff8e115b98
Build id: 3.10.0-1062.4.1.el7.x86_64
0x158
2
queued_spin_lock_slowpath
vmlinux 0xffffffff8e7733cb
Build id: 3.10.0-1062.4.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff8e781430
Build id: 3.10.0-1062.4.1.el7.x86_64
0x20
4
ext4_es_lru_del
ext4 0xffffffffc0ba61f2
Build id: 3.10.0-1062.4.1.el7.x86_64
0x32
5
ext4_clear_inode
ext4 0xffffffffc0b83cd1
Build id: 3.10.0-1062.4.1.el7.x86_64
0x41
6
ext4_free_inode
ext4 0xffffffffc0b5c616
Build id: 3.10.0-1062.4.1.el7.x86_64
0x116
7
? ext4_evict_inode
ext4 0xffffffffc0b66797
Build id: 3.10.0-1062.4.1.el7.x86_64
0x317
8
? ext4_mark_inode_dirty
ext4 0xffffffffc0b6447f
Build id: 3.10.0-1062.4.1.el7.x86_64
0x6f
9
ext4_evict_inode
ext4 0xffffffffc0b66857
Build id: 3.10.0-1062.4.1.el7.x86_64
0x3d7
10
evict
vmlinux 0xffffffff8e266c74
Build id: 3.10.0-1062.4.1.el7.x86_64
0xb4
11
iput
vmlinux 0xffffffff8e26709c
Build id: 3.10.0-1062.4.1.el7.x86_64
0xfc
12
do_unlinkat
vmlinux 0xffffffff8e25b30e
Build id: 3.10.0-1062.4.1.el7.x86_64
0x1ae
13
? vfs_read
vmlinux 0xffffffff8e248de8
Build id: 3.10.0-1062.4.1.el7.x86_64
0xf8
14
SyS_unlink
vmlinux 0xffffffff8e25c3c6
Build id: 3.10.0-1062.4.1.el7.x86_64
0x16
15
system_call_fastpath
vmlinux 0xffffffff8e78bede
Build id: 3.10.0-1062.4.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#100 stuck for 23s! [sftp-server:30636]
Modules linked in: tcp_diag udp_diag inet_diag nf_conntrack_ftp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi sunrpc iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm qla2xxx irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd nvme_fc nvme_fabrics nvme_core pcspkr scsi_transport_fc scsi_tgt joydev i2c_i801 ses enclosure scsi_transport_sas mei_me lpc_ich mei sg ipmi_si ipmi_devintf ipmi_msghandler acpi_pad pcc_cpufreq
binfmt_misc ip_tables xfs libcrc32c raid1 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ixgbe ttm ahci drm libahci igb libata aacraid mdio ptp pps_core dca drm_panel_orientation_quirks i2c_algo_bit wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 100 PID: 30636 Comm: sftp-server Tainted: G             L ------------   3.10.0-1062.4.1.el7.x86_64 #1
Hardware name: POWERLEADER DaPengWan/Default string, BIOS 5.11 12/05/2017
task: ffff9149fd80e2a0 ti: ffff910f26314000 task.ti: ffff910f26314000
RIP: 0010:[<ffffffff8e115b98>]  [<ffffffff8e115b98>] native_queued_spin_lock_slowpath+0x158/0x200
RSP: 0000:ffff910f26317cd8  EFLAGS: 00000202
RAX: 0000000000000101 RBX: ffff91600776806a RCX: 0000000003210000
RDX: 0000000003210101 RSI: 0000000000000101 RDI: ffff91e9fa794480
RBP: ffff910f26317cd8 R08: ffff91aa3f81b800 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000625a4ca3 R12: ffff916007768068
R13: ffffffff8e0c6265 R14: ffff910f26317c50 R15: 00000000991e22d9
FS:  00007f59ce57c840(0000) GS:ffff91aa3f800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f68a3ffec90 CR3: 0000002b4fb56000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8e7733cb>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff8e781430>] _raw_spin_lock+0x20/0x30
[<ffffffffc0ba61f2>] ext4_es_lru_del+0x32/0x70 [ext4]
[<ffffffffc0b83cd1>] ext4_clear_inode+0x41/0x90 [ext4]
[<ffffffffc0b5c616>] ext4_free_inode+0x116/0x610 [ext4]
[<ffffffffc0b66797>] ? ext4_evict_inode+0x317/0x470 [ext4]
[<ffffffffc0b6447f>] ? ext4_mark_inode_dirty+0x6f/0x210 [ext4]
[<ffffffffc0b66857>] ext4_evict_inode+0x3d7/0x470 [ext4]
[<ffffffff8e266c74>] evict+0xb4/0x180
[<ffffffff8e26709c>] iput+0xfc/0x190
[<ffffffff8e25b30e>] do_unlinkat+0x1ae/0x2d0
[<ffffffff8e248de8>] ? vfs_read+0xf8/0x170
[<ffffffff8e25c3c6>] SyS_unlink+0x16/0x20
[<ffffffff8e78bede>] system_call_fastpath+0x25/0x2a
Code: 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 8b 17 <0f> b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 00 00 
Complete report #344397
Frame # Function Binary Source or offset Line
1
ext4_inode_touch_time_cmp
ext4 0xffffffffc04cc02d
Build id: 3.10.0-862.14.4.el7.x86_64
0xd
2
queued_spin_lock_slowpath
vmlinux 0xffffffffba50dc5f
Build id: 3.10.0-862.14.4.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffba51b750
Build id: 3.10.0-862.14.4.el7.x86_64
0x20
4
ext4_es_lru_del
ext4 0xffffffffc04cd4f2
Build id: 3.10.0-862.14.4.el7.x86_64
0x32
5
ext4_clear_inode
ext4 0xffffffffc04ab751
Build id: 3.10.0-862.14.4.el7.x86_64
0x41
6
ext4_free_inode
ext4 0xffffffffc0485516
Build id: 3.10.0-862.14.4.el7.x86_64
0x116
7
? ext4_evict_inode
ext4 0xffffffffc048f2e7
Build id: 3.10.0-862.14.4.el7.x86_64
0x337
8
? ext4_mark_inode_dirty
ext4 0xffffffffc048cfaf
Build id: 3.10.0-862.14.4.el7.x86_64
0x6f
9
ext4_evict_inode
ext4 0xffffffffc048f3a7
Build id: 3.10.0-862.14.4.el7.x86_64
0x3f7
10
evict
vmlinux 0xffffffffba03c504
Build id: 3.10.0-862.14.4.el7.x86_64
0xb4
11
iput
vmlinux 0xffffffffba03ce0c
Build id: 3.10.0-862.14.4.el7.x86_64
0xfc
12
do_unlinkat
vmlinux 0xffffffffba03126e
Build id: 3.10.0-862.14.4.el7.x86_64
0x1ae
13
? __do_page_fault
vmlinux 0xffffffffba52056c
Build id: 3.10.0-862.14.4.el7.x86_64
0x1bc
14
SyS_unlinkat
vmlinux 0xffffffffba0322eb
Build id: 3.10.0-862.14.4.el7.x86_64
0x1b
15
system_call_fastpath
vmlinux 0xffffffffba52579b
Build id: 3.10.0-862.14.4.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#22 stuck for 23s! [kswapd0:229]
Modules linked in: binfmt_misc tcp_diag inet_diag bonding sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt dell_smbios iTCO_vendor_support sparse_keymap skx_edac dcdbas intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr mei_me mei lpc_ich i2c_i801 shpchp wmi nfit libnvdimm acpi_pad acpi_power_meter ipmi_si ipmi_devintf ipmi_msghandler ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel ahci drm libahci igb nvme libata megaraid_sas nvme_core i2c_algo_bit i2c_core ixgbe mdio dca i40e mlx5_core mlxfw devlink ptp pps_core
CPU: 22 PID: 229 Comm: kswapd0 Tainted: G             L ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R640/0PHYDR, BIOS 1.6.11 11/20/2018
task: ffff9379e04f4f10 ti: ffff9379dd108000 task.ti: ffff9379dd108000
RIP: 0010:[<ffffffffc04cc02d>]  [<ffffffffc04cc02d>] ext4_inode_touch_time_cmp+0xd/0x90 [ext4]
RSP: 0018:ffff9379dd10ba98  EFLAGS: 00000282
RAX: 0000000000080000 RBX: ffffffffb9ff8572 RCX: 000000024b3f6d12
RDX: ffff937cea4f4bf8 RSI: ffff936fae57cbf8 RDI: 0000000000000000
RBP: ffff9379dd10bae8 R08: ffff9379dd10bc18 R09: 0000000000000000
R10: 0000000015594a01 R11: ffffee4bdc556500 R12: 0000000000000000
R13: ffff9379df0ed300 R14: ffff9391d7a50400 R15: 0000000000000000
NMI watchdog: BUG: soft lockup - CPU#23 stuck for 23s! [rm:364795]
Modules linked in: binfmt_misc tcp_diag inet_diag bonding sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt dell_smbios iTCO_vendor_support sparse_keymap skx_edac dcdbas intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr mei_me mei lpc_ich i2c_i801 shpchp wmi nfit libnvdimm acpi_pad acpi_power_meter ipmi_si ipmi_devintf ipmi_msghandler ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel ahci drm libahci igb nvme libata megaraid_sas nvme_core i2c_algo_bit i2c_core ixgbe mdio dca i40e mlx5_core mlxfw devlink ptp pps_core
CPU: 23 PID: 364795 Comm: rm Tainted: G             L ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R640/0PHYDR, BIOS 1.6.11 11/20/2018
task: ffff93780b6d4f10 ti: ffff938135f6c000 task.ti: ffff938135f6c000
RIP: 0010:[<ffffffffb9f0bb3e>]  [<ffffffffb9f0bb3e>] native_queued_spin_lock_slowpath+0x1ce/0x200
RSP: 0018:ffff938135f6fcd8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff939083722752 RCX: 0000000000000001
RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff9391deffe480
RBP: ffff938135f6fcd8 R08: 0000000000000101 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000061f566b7 R12: ffff939083722750
R13: ffffffffb9ebeed5 R14: ffff938135f6fc50 R15: 0000000081d7af99
FS:  00007ff2fe6ee740(0000) GS:ffff9391dc0c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff2fe268c0e CR3: 000000186c366000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
 [<ffffffffba50dc5f>] queued_spin_lock_slowpath+0xb/0xf
 [<ffffffffba51b750>] _raw_spin_lock+0x20/0x30
 [<ffffffffc04cd4f2>] ext4_es_lru_del+0x32/0x70 [ext4]
 [<ffffffffc04ab751>] ext4_clear_inode+0x41/0x90 [ext4]
 [<ffffffffc0485516>] ext4_free_inode+0x116/0x610 [ext4]
 [<ffffffffc048f2e7>] ? ext4_evict_inode+0x337/0x4b0 [ext4]
 [<ffffffffc048cfaf>] ? ext4_mark_inode_dirty+0x6f/0x210 [ext4]
 [<ffffffffc048f3a7>] ext4_evict_inode+0x3f7/0x4b0 [ext4]
 [<ffffffffba03c504>] evict+0xb4/0x180
 [<ffffffffba03ce0c>] iput+0xfc/0x190
 [<ffffffffba03126e>] do_unlinkat+0x1ae/0x2d0
 [<ffffffffba52056c>] ? __do_page_fault+0x1bc/0x4f0
 [<ffffffffba0322eb>] SyS_unlinkat+0x1b/0x40
 [<ffffffffba52579b>] system_call_fastpath+0x22/0x27
Code: 37 81 fe 00 01 00 00 74 f4 e9 93 fe ff ff 0f 1f 80 00 00 00 00 83 fa 01 75 11 0f 1f 00 e9 68 fe ff ff 0f 1f 00 85 c0 74 0c f3 90 <8b> 07 0f b6 c0 83 f8 03 75 f0 b8 01 00 00 00 66 89 07 5d c3 66 
Complete report #64899
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffffba313536
Build id: 3.10.0-957.27.2.el7.x86_64
0x126
2
queued_spin_lock_slowpath
vmlinux 0xffffffffba95e2cb
Build id: 3.10.0-957.27.2.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffba96c7a0
Build id: 3.10.0-957.27.2.el7.x86_64
0x20
4
ext4_es_lru_del
ext4 0xffffffffc030fc32
Build id: 3.10.0-957.27.2.el7.x86_64
0x32
5
ext4_clear_inode
ext4 0xffffffffc02ed951
Build id: 3.10.0-957.27.2.el7.x86_64
0x41
6
ext4_free_inode
ext4 0xffffffffc02c65e6
Build id: 3.10.0-957.27.2.el7.x86_64
0x116
7
? ext4_evict_inode
ext4 0xffffffffc02d05f7
Build id: 3.10.0-957.27.2.el7.x86_64
0x317
8
? ext4_mark_inode_dirty
ext4 0xffffffffc02ce33f
Build id: 3.10.0-957.27.2.el7.x86_64
0x6f
9
ext4_evict_inode
ext4 0xffffffffc02d06b7
Build id: 3.10.0-957.27.2.el7.x86_64
0x3d7
10
evict
vmlinux 0xffffffffba45ff24
Build id: 3.10.0-957.27.2.el7.x86_64
0xb4
11
iput
vmlinux 0xffffffffba46082c
Build id: 3.10.0-957.27.2.el7.x86_64
0xfc
12
do_unlinkat
vmlinux 0xffffffffba4548de
Build id: 3.10.0-957.27.2.el7.x86_64
0x1ae
13
? SYSC_newstat
vmlinux 0xffffffffba4485de
Build id: 3.10.0-957.27.2.el7.x86_64
0x2e
14
SyS_unlink
vmlinux 0xffffffffba455996
Build id: 3.10.0-957.27.2.el7.x86_64
0x16
15
system_call_fastpath
vmlinux 0xffffffffba976ddb
Build id: 3.10.0-957.27.2.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [BackgrProcPool:10848]
Modules linked in: binfmt_misc nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c xt_comment iptable_filter ip_gre gre ip_tunnel bonding dcdbas iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm ipmi_ssif irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg lpc_ich i2c_i801 mei_me mei ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter tcp_yeah tcp_vegas ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm crc32c_intel drm ahci ixgbe libahci megaraid_sas igb libata mdio ptp pps_core dca i2c_algo_bit drm_panel_orientation_quirks nfit
libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 30 PID: 10848 Comm: BackgrProcPool Kdump: loaded Tainted: G             L ------------   3.10.0-957.27.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R740xd/08D89F, BIOS 1.4.8 05/21/2018
task: ffff92995f662080 ti: ffff92995f4f8000 task.ti: ffff92995f4f8000
RIP: 0010:[<ffffffffba313536>]  [<ffffffffba313536>] native_queued_spin_lock_slowpath+0x126/0x200
RSP: 0018:ffff92995f4fbcd8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff926f704d9862 RCX: 0000000000f10000
RDX: ffff92997bf5b780 RSI: 0000000000590101 RDI: ffff92797d1f4c80
RBP: ffff92995f4fbcd8 R08: ffff92797cddb780 R09: 0000000000000000
R10: 0000000000000000 R11: 00000000601bdeba R12: ffff926f704d9860
R13: ffffffffba2c3f15 R14: ffff92995f4fbc50 R15: 000000001af29561
FS:  00007fe966ff1700(0000) GS:ffff92797cdc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa366a00000 CR3: 0000001fd5ae0000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffba95e2cb>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffffba96c7a0>] _raw_spin_lock+0x20/0x30
[<ffffffffc030fc32>] ext4_es_lru_del+0x32/0x70 [ext4]
[<ffffffffc02ed951>] ext4_clear_inode+0x41/0x90 [ext4]
[<ffffffffc02c65e6>] ext4_free_inode+0x116/0x610 [ext4]
[<ffffffffc02d05f7>] ? ext4_evict_inode+0x317/0x470 [ext4]
[<ffffffffc02ce33f>] ? ext4_mark_inode_dirty+0x6f/0x210 [ext4]
[<ffffffffc02d06b7>] ext4_evict_inode+0x3d7/0x470 [ext4]
[<ffffffffba45ff24>] evict+0xb4/0x180
[<ffffffffba46082c>] iput+0xfc/0x190
[<ffffffffba4548de>] do_unlinkat+0x1ae/0x2d0
[<ffffffffba4485de>] ? SYSC_newstat+0x2e/0x60
[<ffffffffba455996>] SyS_unlink+0x16/0x20
[<ffffffffba976ddb>] system_call_fastpath+0x22/0x27
Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 a0 bf f4 ba 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 
Complete report #6732
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffffa190ba92
Build id: 3.10.0-862.14.4.el7.x86_64
0x122
2
queued_spin_lock_slowpath
vmlinux 0xffffffffa1f0dc5f
Build id: 3.10.0-862.14.4.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffa1f1b750
Build id: 3.10.0-862.14.4.el7.x86_64
0x20
4
ext4_es_lru_del
ext4 0xffffffffc0a374f2
Build id: 3.10.0-862.14.4.el7.x86_64
0x32
5
ext4_clear_inode
ext4 0xffffffffc0a15751
Build id: 3.10.0-862.14.4.el7.x86_64
0x41
6
ext4_free_inode
ext4 0xffffffffc09ef516
Build id: 3.10.0-862.14.4.el7.x86_64
0x116
7
? ext4_evict_inode
ext4 0xffffffffc09f92e7
Build id: 3.10.0-862.14.4.el7.x86_64
0x337
8
? ext4_mark_inode_dirty
ext4 0xffffffffc09f6faf
Build id: 3.10.0-862.14.4.el7.x86_64
0x6f
9
ext4_evict_inode
ext4 0xffffffffc09f93a7
Build id: 3.10.0-862.14.4.el7.x86_64
0x3f7
10
evict
vmlinux 0xffffffffa1a3c504
Build id: 3.10.0-862.14.4.el7.x86_64
0xb4
11
iput
vmlinux 0xffffffffa1a3ce0c
Build id: 3.10.0-862.14.4.el7.x86_64
0xfc
12
do_unlinkat
vmlinux 0xffffffffa1a3126e
Build id: 3.10.0-862.14.4.el7.x86_64
0x1ae
13
? ____fput
vmlinux 0xffffffffa1a2175e
Build id: 3.10.0-862.14.4.el7.x86_64
0xe
14
? task_work_run
vmlinux 0xffffffffa18bab90
Build id: 3.10.0-862.14.4.el7.x86_64
0xc0
15
SyS_unlink
vmlinux 0xffffffffa1a32326
Build id: 3.10.0-862.14.4.el7.x86_64
0x16
16
system_call_fastpath
vmlinux 0xffffffffa1f2579b
Build id: 3.10.0-862.14.4.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [foremost_files.:81765]
Modules linked in: fuse nfsd nfsv3 nfs_acl nfnetlink_queue nfnetlink_log xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc vfat fat ext4 mbcache jbd2 iTCO_wdt iTCO_vendor_support dcdbas skx_edac intel_powerclamp coretemp intel_rapl
iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif i2c_i801 pcspkr ipmi_si nfit sg ipmi_devintf ipmi_msghandler lpc_ich mei_me libnvdimm mei shpchp acpi_power_meter acpi_pad ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul ttm crct10dif_common crc32c_intel drm ahci i40e libahci libata megaraid_sas ptp i2c_core pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 21 PID: 81765 Comm: foremost_files. Kdump: loaded Tainted: G             L ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R640/008R9M, BIOS 1.2.71 12/04/2017
task: ffff979ac430eeb0 ti: ffff97983a7bc000 task.ti: ffff97983a7bc000
RIP: 0010:[<ffffffffa190ba92>]  [<ffffffffa190ba92>] native_queued_spin_lock_slowpath+0x122/0x200
RSP: 0018:ffff97983a7bfcd8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff97983a7bfc68 RCX: 0000000000a90000
RDX: ffff98533bfd9740 RSI: 0000000000790101 RDI: ffff97f4ecf64c80
RBP: ffff97983a7bfcd8 R08: ffff98533c099740 R09: 0000000000000000
R10: 0000000091cfc301 R11: ffffdbb6c4473f00 R12: ffff979bbe693d48
R13: ffff97f591cfc960 R14: ffffffffa19fa282 R15: ffff97983a7bfc50
FS:  00007f5a4eea9740(0000) GS:ffff98533c080000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000019ef768 CR3: 00000061de8f8000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa1f0dc5f>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffffa1f1b750>] _raw_spin_lock+0x20/0x30
[<ffffffffc0a374f2>] ext4_es_lru_del+0x32/0x70 [ext4]
[<ffffffffc0a15751>] ext4_clear_inode+0x41/0x90 [ext4]
[<ffffffffc09ef516>] ext4_free_inode+0x116/0x610 [ext4]
[<ffffffffc09f92e7>] ? ext4_evict_inode+0x337/0x4b0 [ext4]
[<ffffffffc09f6faf>] ? ext4_mark_inode_dirty+0x6f/0x210 [ext4]
[<ffffffffc09f93a7>] ext4_evict_inode+0x3f7/0x4b0 [ext4]
[<ffffffffa1a3c504>] evict+0xb4/0x180
[<ffffffffa1a3ce0c>] iput+0xfc/0x190
[<ffffffffa1a3126e>] do_unlinkat+0x1ae/0x2d0
[<ffffffffa1a2175e>] ? ____fput+0xe/0x10
[<ffffffffa18bab90>] ? task_work_run+0xc0/0xe0
[<ffffffffa1a32326>] SyS_unlink+0x16/0x20
[<ffffffffa1f2579b>] system_call_fastpath+0x22/0x27
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 40 97 01 00 48 03 14 c5 a0 54 53 a2 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b