Info

Permalink
Function
native_queued_spin_lock_slowpath
First occurrence
2021-04-27
Last occurrence
2022-04-22
State
NEW
Components
kernel
Quality
-53

Statistics

Operating system Count
CentOS 7 14
Architecture Count
x86_64 14
Related packages Count
kernel 14
 0:3.10.0-693.el7 8
 0:3.10.0-693.21.1.el7 5
 0:3.10.0-693.11.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1415693
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff810fdfb2
Build id: 3.10.0-693.21.1.el7.x86_64 		0x122
2
queued_spin_lock_slowpath
vmlinux 0xffffffff816a8ff4
Build id: 3.10.0-693.21.1.el7.x86_64 		0xb
3
_raw_spin_lock
vmlinux 0xffffffff816b6850
Build id: 3.10.0-693.21.1.el7.x86_64 		0x20
4
ext4_es_lru_add
ext4 0xffffffffc02729d7
Build id: 3.10.0-693.21.1.el7.x86_64 		0x57
5
ext4_ext_map_blocks
ext4 0xffffffffc025e8d5
Build id: 3.10.0-693.21.1.el7.x86_64 		0x7a5
6
? kmem_cache_alloc
vmlinux 0xffffffff811e4092
Build id: 3.10.0-693.21.1.el7.x86_64 		0x1c2
7
ext4_da_get_block_prep
ext4 0xffffffffc022de30
Build id: 3.10.0-693.21.1.el7.x86_64 		0x370
8
? alloc_page_buffers
vmlinux 0xffffffff8123b243
Build id: 3.10.0-693.21.1.el7.x86_64 		0x63
9
__block_write_begin
vmlinux 0xffffffff8123d33d
Build id: 3.10.0-693.21.1.el7.x86_64 		0x17d
10
? ext4_da_invalidatepage
ext4 0xffffffffc022dac0
Build id: 3.10.0-693.21.1.el7.x86_64 		0x320
11
? __ext4_journal_start_sb
ext4 0xffffffffc0260bc9
Build id: 3.10.0-693.21.1.el7.x86_64 		0x69
12
ext4_da_write_begin
ext4 0xffffffffc0234387
Build id: 3.10.0-693.21.1.el7.x86_64 		0x167
13
generic_file_buffered_write
vmlinux 0xffffffff81186944
Build id: 3.10.0-693.21.1.el7.x86_64 		0x124
14
__generic_file_aio_write
vmlinux 0xffffffff81188db2
Build id: 3.10.0-693.21.1.el7.x86_64 		0x1e2
15
? ima_file_check
vmlinux 0xffffffff812d98b3
Build id: 3.10.0-693.21.1.el7.x86_64 		0x63
16
generic_file_aio_write
vmlinux 0xffffffff81189029
Build id: 3.10.0-693.21.1.el7.x86_64 		0x59
17
ext4_file_write
ext4 0xffffffffc0228eeb
Build id: 3.10.0-693.21.1.el7.x86_64 		0xdb
18
? path_openat
vmlinux 0xffffffff81215652
Build id: 3.10.0-693.21.1.el7.x86_64 		0xc2
19
? try_to_wake_up
vmlinux 0xffffffff810c79ec
Build id: 3.10.0-693.21.1.el7.x86_64 		0x18c
20
do_sync_write
vmlinux 0xffffffff81204ba3
Build id: 3.10.0-693.21.1.el7.x86_64 		0x93
21
vfs_write
vmlinux 0xffffffff81205680
Build id: 3.10.0-693.21.1.el7.x86_64 		0xc0
22
? putname
vmlinux 0xffffffff812168ed
Build id: 3.10.0-693.21.1.el7.x86_64 		0x3d
23
SyS_write
vmlinux 0xffffffff8120649f
Build id: 3.10.0-693.21.1.el7.x86_64 		0x7f
24
system_call_fastpath
vmlinux 0xffffffff816c0715
Build id: 3.10.0-693.21.1.el7.x86_64 		0x1c 
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [sfavsrv:400358]
Modules linked in: udp_diag tcp_diag inet_diag binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bonding dm_mirror dm_region_hash dm_log dm_mod skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support vfat fat pcspkr cdc_ether usbnet mii uas sg usb_storage i2c_i801 shpchp lpc_ich mei_me mei wmi ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter nfsd auth_rpcgss nfs_acl
lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops i40e ttm igb drm ahci libahci dca ptp i2c_algo_bit pps_core libata megaraid_sas(OE) i2c_core
CPU: 13 PID: 400358 Comm: sfavsrv Tainted: G           OEL ------------   3.10.0-693.21.1.el7.x86_64 #1
Hardware name: Lenovo ThinkSystem SR850 -[7X19CTO1WW]-/-[7X19CTO1WW]-, BIOS -[TEE164L-2.80]- 10/23/2020
task: ffff881f51dc2f70 ti: ffff881b93f48000 task.ti: ffff881b93f48000
RIP: 0010:[<ffffffff810fdfb2>]  [<ffffffff810fdfb2>] native_queued_spin_lock_slowpath+0x122/0x200
RSP: 0018:ffff881b93f4b898  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8838f877cc98 RCX: 0000000000690000
RDX: ffff8817de7d9600 RSI: 0000000000390101 RDI: ffff8840a2477c80
RBP: ffff881b93f4b898 R08: ffff8817de959600 R09: 0000000000000000
R10: ffff8817de95b960 R11: ffffea005d4b8800 R12: ffff881b93f4b838
R13: ffff8838f877cd68 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00007f99167dc700(0000) GS:ffff8817de940000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f99167d8920 CR3: 00000019a0400000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff816a8ff4>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff816b6850>] _raw_spin_lock+0x20/0x30
[<ffffffffc02729d7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc025e8d5>] ext4_ext_map_blocks+0x7a5/0xf50 [ext4]
[<ffffffff811e4092>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc022de30>] ext4_da_get_block_prep+0x370/0x4c0 [ext4]
[<ffffffff8123b243>] ? alloc_page_buffers+0x63/0xc0
[<ffffffff8123d33d>] __block_write_begin+0x17d/0x4b0
[<ffffffffc022dac0>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc0260bc9>] ? __ext4_journal_start_sb+0x69/0xe0 [ext4]
[<ffffffffc0234387>] ext4_da_write_begin+0x167/0x360 [ext4]
[<ffffffff81186944>] generic_file_buffered_write+0x124/0x2c0
[<ffffffff81188db2>] __generic_file_aio_write+0x1e2/0x400
[<ffffffff812d98b3>] ? ima_file_check+0x63/0x1b0
[<ffffffff81189029>] generic_file_aio_write+0x59/0xa0
[<ffffffffc0228eeb>] ext4_file_write+0xdb/0x470 [ext4]
[<ffffffff81215652>] ? path_openat+0xc2/0x490
[<ffffffff810c79ec>] ? try_to_wake_up+0x18c/0x350
[<ffffffff81204ba3>] do_sync_write+0x93/0xe0
[<ffffffff81205680>] vfs_write+0xc0/0x1f0
[<ffffffff812168ed>] ? putname+0x3d/0x60
[<ffffffff8120649f>] SyS_write+0x7f/0xe0
[<ffffffff816c0715>] system_call_fastpath+0x1c/0x21
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 00 96 01 00 48 03 14 c5 a0 10 b2 81 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b
Complete report #140687
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff810fa336
Build id: 3.10.0-693.11.1.el7.x86_64 		0x116
2
queued_spin_lock_slowpath
vmlinux 0xffffffff8169e6cf
Build id: 3.10.0-693.11.1.el7.x86_64 		0xb
3
_raw_spin_lock
vmlinux 0xffffffff816abc20
Build id: 3.10.0-693.11.1.el7.x86_64 		0x20
4
ext4_es_lru_add
ext4 0xffffffffc08038c7
Build id: 3.10.0-693.11.1.el7.x86_64 		0x57
5
ext4_ext_map_blocks
ext4 0xffffffffc07ef7f5
Build id: 3.10.0-693.11.1.el7.x86_64 		0x795
6
? kmem_cache_alloc
vmlinux 0xffffffff811df65a
Build id: 3.10.0-693.11.1.el7.x86_64 		0x1ba
7
ext4_da_get_block_prep
ext4 0xffffffffc07bee10
Build id: 3.10.0-693.11.1.el7.x86_64 		0x370
8
? alloc_page_buffers
vmlinux 0xffffffff81236213
Build id: 3.10.0-693.11.1.el7.x86_64 		0x63
9
__block_write_begin
vmlinux 0xffffffff8123830a
Build id: 3.10.0-693.11.1.el7.x86_64 		0x17a
10
? ext4_da_invalidatepage
ext4 0xffffffffc07beaa0
Build id: 3.10.0-693.11.1.el7.x86_64 		0x320
11
? __ext4_journal_start_sb
ext4 0xffffffffc07f1ae9
Build id: 3.10.0-693.11.1.el7.x86_64 		0x69
12
ext4_da_write_begin
ext4 0xffffffffc07c5337
Build id: 3.10.0-693.11.1.el7.x86_64 		0x167
13
generic_file_buffered_write
vmlinux 0xffffffff811824de
Build id: 3.10.0-693.11.1.el7.x86_64 		0x11e
14
__generic_file_aio_write
vmlinux 0xffffffff81184902
Build id: 3.10.0-693.11.1.el7.x86_64 		0x1e2
15
? ima_file_check
vmlinux 0xffffffff812d3d43
Build id: 3.10.0-693.11.1.el7.x86_64 		0x63
16
generic_file_aio_write
vmlinux 0xffffffff81184b79
Build id: 3.10.0-693.11.1.el7.x86_64 		0x59
17
ext4_file_write
ext4 0xffffffffc07b9eeb
Build id: 3.10.0-693.11.1.el7.x86_64 		0xdb
18
? unlock_page
vmlinux 0xffffffff8118296b
Build id: 3.10.0-693.11.1.el7.x86_64 		0x2b
19
? do_read_fault.isra.44
vmlinux 0xffffffff811ad616
Build id: 3.10.0-693.11.1.el7.x86_64 		0xe6
20
do_sync_write
vmlinux 0xffffffff8120006d
Build id: 3.10.0-693.11.1.el7.x86_64 		0x8d
21
vfs_write
vmlinux 0xffffffff81200b2d
Build id: 3.10.0-693.11.1.el7.x86_64 		0xbd
22
SyS_write
vmlinux 0xffffffff8120193f
Build id: 3.10.0-693.11.1.el7.x86_64 		0x7f
23
system_call_fastpath
vmlinux 0xffffffff816b5089
Build id: 3.10.0-693.11.1.el7.x86_64 		0x16 
NMI watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [spades:62868]
Modules linked in: binfmt_misc fuse ext4 mbcache jbd2 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ipmi_si vfat fat uas usb_storage mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase dell_rbu nf_conntrack_ftp ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support gpio_ich intel_powerclamp dcdbas coretemp joydev kvm_intel kvm irqbypass bnx2 serio_raw pcspkr sg ipmi_ssif ipmi_devintf ipmi_msghandler i7core_edac edac_core ata_generic pata_acpi acpi_power_meter lpc_ich
shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_common mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crc32c_intel libata megaraid_sas i2c_core [last unloaded: ipmi_si]
CPU: 14 PID: 62868 Comm: spades Tainted: G             L ------------   3.10.0-693.11.1.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R910/0P658H, BIOS 1.2.0 06/22/2010
task: ffff8840a7c0dee0 ti: ffff883ff87e4000 task.ti: ffff883ff87e4000
RIP: 0010:[<ffffffff810fa336>]  [<ffffffff810fa336>] native_queued_spin_lock_slowpath+0x116/0x1e0
RSP: 0018:ffff883ff87e78c8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88564dafe4c8 RCX: 0000000000710000
RDX: ffff881fff8d7880 RSI: 0000000000610101 RDI: ffff887ffa751480
RBP: ffff883ff87e78c8 R08: ffff885fff8d7880 R09: 0000000000000000
R10: ffff885fff8d9be0 R11: ffffea016010ed00 R12: ffff883ff87e7868
R13: ffff88564dafe598 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  0000000000d422d0(0063) GS:ffff885fff8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000004c2000 CR3: 0000002aae325000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff883ff87e78d8 ffffffff8169e6cf ffff883ff87e78e8 ffffffff816abc20
ffff883ff87e7918 ffffffffc08038c7 ffff88564dafe598 0000000000000000
ffff8858043b4c00 0000000000000000 ffff883ff87e7a00 ffffffffc07ef7f5
Call Trace:
[<ffffffff8169e6cf>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff816abc20>] _raw_spin_lock+0x20/0x30
[<ffffffffc08038c7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc07ef7f5>] ext4_ext_map_blocks+0x795/0xf40 [ext4]
[<ffffffff811df65a>] ? kmem_cache_alloc+0x1ba/0x1e0
[<ffffffffc07bee10>] ext4_da_get_block_prep+0x370/0x4b0 [ext4]
[<ffffffff81236213>] ? alloc_page_buffers+0x63/0xc0
[<ffffffff8123830a>] __block_write_begin+0x17a/0x4b0
[<ffffffffc07beaa0>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc07f1ae9>] ? __ext4_journal_start_sb+0x69/0xe0 [ext4]
[<ffffffffc07c5337>] ext4_da_write_begin+0x167/0x350 [ext4]
[<ffffffff811824de>] generic_file_buffered_write+0x11e/0x2a0
[<ffffffff81184902>] __generic_file_aio_write+0x1e2/0x400
[<ffffffff812d3d43>] ? ima_file_check+0x63/0x1b0
[<ffffffff81184b79>] generic_file_aio_write+0x59/0xa0
[<ffffffffc07b9eeb>] ext4_file_write+0xdb/0x470 [ext4]
[<ffffffff8118296b>] ? unlock_page+0x2b/0x30
[<ffffffff811ad616>] ? do_read_fault.isra.44+0xe6/0x130
[<ffffffff8120006d>] do_sync_write+0x8d/0xd0
[<ffffffff81200b2d>] vfs_write+0xbd/0x1e0
[<ffffffff8120193f>] SyS_write+0x7f/0xe0
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 0d 48 98 83 e2 30 48 81 c2 80 78 01 00 48 03 14 c5 e0 fd b0 81 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2