Info

Permalink
Function
native_queued_spin_lock_slowpath
First occurrence
2020-10-22
Last occurrence
2025-11-27
State
NEW
Components
kernel
Quality
-52

Statistics

Operating system Count
CentOS 7 384
Architecture Count
x86_64 384
Related packages Count
kernel 384
 0:3.10.0-957.27.2.el7 305
 0:3.10.0-957.21.3.el7 25
 0:3.10.0-1062.1.2.el7 15
 0:3.10.0-862.3.2.el7 13
 0:3.10.0-1062.el7 8
 0:3.10.0-957.12.2.el7 4
 0:3.10.0-1062.4.1.el7 3
 0:3.10.0-862.el7 3
 0:3.10.0-862.9.1.el7 3
 0:3.10.0-957.el7 2
 0:3.10.0-957.5.1.el7 1
 0:3.10.0-957.12.1.el7 1
 0:3.10.0-1127.10.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1379254
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffffa4517f92
Build id: 3.10.0-1127.10.1.el7.x86_64
0x122
2
queued_spin_lock_slowpath
vmlinux 0xffffffffa4b7a024
Build id: 3.10.0-1127.10.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffa4b886d0
Build id: 3.10.0-1127.10.1.el7.x86_64
0x20
4
ext4_es_lru_add
ext4 0xffffffffc0ca88a7
Build id: 3.10.0-1127.10.1.el7.x86_64
0x57
5
ext4_ext_map_blocks
ext4 0xffffffffc0c94305
Build id: 3.10.0-1127.10.1.el7.x86_64
0x7d5
6
? do_get_write_access
jbd2 0xffffffffc0bd0dc7
Build id: 3.10.0-1127.10.1.el7.x86_64
0x357
7
? kmem_cache_alloc
vmlinux 0xffffffffa4628852
Build id: 3.10.0-1127.10.1.el7.x86_64
0x1c2
8
ext4_da_get_block_prep
ext4 0xffffffffc0c61d70
Build id: 3.10.0-1127.10.1.el7.x86_64
0x370
9
? __mem_cgroup_commit_charge
vmlinux 0xffffffffa463e262
Build id: 3.10.0-1127.10.1.el7.x86_64
0xe2
10
? alloc_page_buffers
vmlinux 0xffffffffa4685dd3
Build id: 3.10.0-1127.10.1.el7.x86_64
0x63
11
__block_write_begin_int
vmlinux 0xffffffffa4688008
Build id: 3.10.0-1127.10.1.el7.x86_64
0x198
12
? kmem_cache_alloc
vmlinux 0xffffffffa4628852
Build id: 3.10.0-1127.10.1.el7.x86_64
0x1c2
13
? ext4_da_invalidatepage
ext4 0xffffffffc0c61a00
Build id: 3.10.0-1127.10.1.el7.x86_64
0x320
14
? ext4_da_write_begin
ext4 0xffffffffc0c686b9
Build id: 3.10.0-1127.10.1.el7.x86_64
0x119
15
__block_write_begin
vmlinux 0xffffffffa46884d1
Build id: 3.10.0-1127.10.1.el7.x86_64
0x11
16
ext4_da_write_begin
ext4 0xffffffffc0c68707
Build id: 3.10.0-1127.10.1.el7.x86_64
0x167
17
generic_file_buffered_write
vmlinux 0xffffffffa45be04f
Build id: 3.10.0-1127.10.1.el7.x86_64
0x10f
18
__generic_file_aio_write
vmlinux 0xffffffffa45c08d2
Build id: 3.10.0-1127.10.1.el7.x86_64
0x1e2
19
generic_file_aio_write
vmlinux 0xffffffffa45c0b49
Build id: 3.10.0-1127.10.1.el7.x86_64
0x59
20
ext4_file_write
ext4 0xffffffffc0c5d5c8
Build id: 3.10.0-1127.10.1.el7.x86_64
0x348
21
? path_openat
vmlinux 0xffffffffa465e21d
Build id: 3.10.0-1127.10.1.el7.x86_64
0xcd
22
do_sync_write
vmlinux 0xffffffffa464c6c3
Build id: 3.10.0-1127.10.1.el7.x86_64
0x93
23
vfs_write
vmlinux 0xffffffffa464d1b0
Build id: 3.10.0-1127.10.1.el7.x86_64
0xc0
24
SyS_write
vmlinux 0xffffffffa464df7f
Build id: 3.10.0-1127.10.1.el7.x86_64
0x7f
25
system_call_fastpath
vmlinux 0xffffffffa4b92ed2
Build id: 3.10.0-1127.10.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [genEOS:32990]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm vfat ext4 fat irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul mbcache glue_helper ablk_helper cryptd jbd2 pcspkr cdc_ether usbnet sg mii lpc_ich i2c_i801 mei_me
mei wmi ipmi_ssif ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad dm_multipath ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea crc32c_intel sysfillrect sysimgblt fb_sys_fops ttm drm ahci i40e libahci megaraid_sas libata ptp pps_core drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 8 PID: 32990 Comm: genEOS Kdump: loaded Tainted: G             L ------------   3.10.0-1127.10.1.el7.x86_64 #1
Hardware name: Lenovo ThinkSystem SR850 -[7X19CTO1WW]-/-[7X19CTO1WW]-, BIOS -[TEE152L-2.51]- 01/13/2020
task: ffff8982528b41c0 ti: ffff89a1502e0000 task.ti: ffff89a1502e0000
RIP: 0010:[<ffffffffa4517f92>]  [<ffffffffa4517f92>] native_queued_spin_lock_slowpath+0x122/0x200
RSP: 0018:ffff89a1502e3870  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8929dad4d400 RCX: 0000000000410000
RDX: ffff89617efdb8c0 RSI: 0000000001790101 RDI: ffff89a17bfffc80
RBP: ffff89a1502e3870 R08: ffff89419f01b8c0 R09: 0000000000000000
R10: ffff89419f01f0c0 R11: fffff0e8e4836080 R12: 0000000000000000
R13: ffff89a1502e3810 R14: ffff8929dad4d170 R15: 0000000000000000
FS:  00007fb1e997cb80(0000) GS:ffff89419f000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb016c4e550 CR3: 00000015b9664000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa4b7a024>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffffa4b886d0>] _raw_spin_lock+0x20/0x30
[<ffffffffc0ca88a7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc0c94305>] ext4_ext_map_blocks+0x7d5/0xfa0 [ext4]
[<ffffffffc0bd0dc7>] ? do_get_write_access+0x357/0x4f0 [jbd2]
[<ffffffffa4628852>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc0c61d70>] ext4_da_get_block_prep+0x370/0x4c0 [ext4]
[<ffffffffa463e262>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
[<ffffffffa4685dd3>] ? alloc_page_buffers+0x63/0xc0
[<ffffffffa4688008>] __block_write_begin_int+0x198/0x650
[<ffffffffa4628852>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc0c61a00>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc0c686b9>] ? ext4_da_write_begin+0x119/0x360 [ext4]
[<ffffffffa46884d1>] __block_write_begin+0x11/0x20
[<ffffffffc0c68707>] ext4_da_write_begin+0x167/0x360 [ext4]
[<ffffffffa45be04f>] generic_file_buffered_write+0x10f/0x270
[<ffffffffa45c08d2>] __generic_file_aio_write+0x1e2/0x400
[<ffffffffa45c0b49>] generic_file_aio_write+0x59/0xa0
[<ffffffffc0c5d5c8>] ext4_file_write+0x348/0x600 [ext4]
[<ffffffffa465e21d>] ? path_openat+0xcd/0x5a0
[<ffffffffa464c6c3>] do_sync_write+0x93/0xe0
[<ffffffffa464d1b0>] vfs_write+0xc0/0x1f0
[<ffffffffa464df7f>] SyS_write+0x7f/0xf0
[<ffffffffa4b92ed2>] system_call_fastpath+0x25/0x2a
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 20 10 15 a5 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 
Complete report #1239561
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffffa7312186
Build id: 3.10.0-957.el7.x86_64
0x126
2
queued_spin_lock_slowpath
vmlinux 0xffffffffa795bf73
Build id: 3.10.0-957.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffa796a420
Build id: 3.10.0-957.el7.x86_64
0x20
4
ext4_es_lru_add
ext4 0xffffffffc0472bb7
Build id: 3.10.0-957.el7.x86_64
0x57
5
ext4_ext_map_blocks
ext4 0xffffffffc045e8f5
Build id: 3.10.0-957.el7.x86_64
0x7b5
6
? drain_stock.isra.23
vmlinux 0xffffffffa742fe40
Build id: 3.10.0-957.el7.x86_64
0x70
7
? __mem_cgroup_try_charge
vmlinux 0xffffffffa74336d2
Build id: 3.10.0-957.el7.x86_64
0x182
8
? kmem_cache_alloc
vmlinux 0xffffffffa741bec2
Build id: 3.10.0-957.el7.x86_64
0x1c2
9
ext4_da_get_block_prep
ext4 0xffffffffc042c8d0
Build id: 3.10.0-957.el7.x86_64
0x370
10
? alloc_page_buffers
vmlinux 0xffffffffa7478b73
Build id: 3.10.0-957.el7.x86_64
0x63
11
__block_write_begin_int
vmlinux 0xffffffffa747ac88
Build id: 3.10.0-957.el7.x86_64
0x198
12
? kmem_cache_alloc
vmlinux 0xffffffffa741bec2
Build id: 3.10.0-957.el7.x86_64
0x1c2
13
? ext4_da_invalidatepage
ext4 0xffffffffc042c560
Build id: 3.10.0-957.el7.x86_64
0x320
14
? ext4_da_write_begin
ext4 0xffffffffc0433099
Build id: 3.10.0-957.el7.x86_64
0x119
15
__block_write_begin
vmlinux 0xffffffffa747b0f1
Build id: 3.10.0-957.el7.x86_64
0x11
16
ext4_da_write_begin
ext4 0xffffffffc04330e7
Build id: 3.10.0-957.el7.x86_64
0x167
17
generic_file_buffered_write
vmlinux 0xffffffffa73b5d94
Build id: 3.10.0-957.el7.x86_64
0x124
18
__generic_file_aio_write
vmlinux 0xffffffffa73b86d2
Build id: 3.10.0-957.el7.x86_64
0x1e2
19
? do_last
vmlinux 0xffffffffa744fa3d
Build id: 3.10.0-957.el7.x86_64
0x66d
20
generic_file_aio_write
vmlinux 0xffffffffa73b8949
Build id: 3.10.0-957.el7.x86_64
0x59
21
ext4_file_write
ext4 0xffffffffc0428322
Build id: 3.10.0-957.el7.x86_64
0xd2
22
do_sync_write
vmlinux 0xffffffffa7440633
Build id: 3.10.0-957.el7.x86_64
0x93
23
vfs_write
vmlinux 0xffffffffa7441120
Build id: 3.10.0-957.el7.x86_64
0xc0
24
SyS_write
vmlinux 0xffffffffa7441f3f
Build id: 3.10.0-957.el7.x86_64
0x7f
25
tracesys
vmlinux 0xffffffffa797505b
Build id: 3.10.0-957.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#103 stuck for 22s! [java:150141]
Modules linked in: tcp_diag inet_diag binfmt_misc rpcrdma ib_isert iscsi_target_mod ib_iser ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core tcp_lp nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache fuse xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 tun ebtable_filter devlink ebtables ip6table_filter ip6_tables xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c br_netfilter bridge stp llc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi overlay(T) sunrpc dm_mirror dm_region_hash dm_log edac_mce_amd kvm_amd
kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr dm_mod ixgbe joydev sg mdio i2c_piix4 ipmi_si ipmi_devintf ipmi_msghandler pinctrl_amd i2c_designware_platform i2c_designware_core acpi_cpufreq ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci drm_panel_orientation_quirks libata megaraid_sas igb ptp pps_core nfit dca i2c_algo_bit libnvdimm
CPU: 103 PID: 150141 Comm: java Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: N/A 2U Rack Server/65N32-US, BIOS SZZQ042035 06/09/2021
task: ffff98f9846e8000 ti: ffff98a66771c000 task.ti: ffff98a66771c000
RIP: 0010:[<ffffffffa7312186>]  [<ffffffffa7312186>] native_queued_spin_lock_slowpath+0x126/0x200
RSP: 0018:ffff98a66771f8c8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff98b720708408 RCX: 0000000003390000
RDX: ffff98d98f61b780 RSI: 0000000001410001 RDI: ffff99392b14e480
RBP: ffff98a66771f8c8 R08: ffff98b98f9db780 R09: 0000000000000000
R10: ffff98b98f9df0c0 R11: fffffe4cff079400 R12: ffff98a66771f868
R13: ffff98b7207084d8 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00002b6ef71b7700(0000) GS:ffff98b98f9c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b6f86404000 CR3: 0000009066a90000 CR4: 0000000000340fe0
Call Trace:
[<ffffffffa795bf73>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffffa796a420>] _raw_spin_lock+0x20/0x30
[<ffffffffc0472bb7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc045e8f5>] ext4_ext_map_blocks+0x7b5/0xf60 [ext4]
[<ffffffffa742fe40>] ? drain_stock.isra.23+0x70/0x80
[<ffffffffa74336d2>] ? __mem_cgroup_try_charge+0x182/0x670
[<ffffffffa741bec2>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc042c8d0>] ext4_da_get_block_prep+0x370/0x4c0 [ext4]
[<ffffffffa7478b73>] ? alloc_page_buffers+0x63/0xc0
[<ffffffffa747ac88>] __block_write_begin_int+0x198/0x5f0
[<ffffffffa741bec2>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc042c560>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc0433099>] ? ext4_da_write_begin+0x119/0x360 [ext4]
[<ffffffffa747b0f1>] __block_write_begin+0x11/0x20
[<ffffffffc04330e7>] ext4_da_write_begin+0x167/0x360 [ext4]
[<ffffffffa73b5d94>] generic_file_buffered_write+0x124/0x2c0
[<ffffffffa73b86d2>] __generic_file_aio_write+0x1e2/0x400
[<ffffffffa744fa3d>] ? do_last+0x66d/0x12a0
[<ffffffffa73b8949>] generic_file_aio_write+0x59/0xa0
[<ffffffffc0428322>] ext4_file_write+0xd2/0x1e0 [ext4]
[<ffffffffa7440633>] do_sync_write+0x93/0xe0
[<ffffffffa7441120>] vfs_write+0xc0/0x1f0
[<ffffffffa7441f3f>] SyS_write+0x7f/0xf0
[<ffffffffa797505b>] tracesys+0xa3/0xc9
Code: 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 a0 b9 f4 a7 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 
Complete report #821070
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff97f08b36
Build id: 3.10.0-862.3.2.el7.x86_64
0x126
2
queued_spin_lock_slowpath
vmlinux 0xffffffff98509450
Build id: 3.10.0-862.3.2.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff98516780
Build id: 3.10.0-862.3.2.el7.x86_64
0x20
4
ext4_es_lru_add
ext4 0xffffffffc0742437
Build id: 3.10.0-862.3.2.el7.x86_64
0x57
5
ext4_ext_map_blocks
ext4 0xffffffffc072e365
Build id: 3.10.0-862.3.2.el7.x86_64
0x7b5
6
? __alloc_pages_nodemask
vmlinux 0xffffffff97f9e497
Build id: 3.10.0-862.3.2.el7.x86_64
0x197
7
? __brelse
vmlinux 0xffffffff9805199d
Build id: 3.10.0-862.3.2.el7.x86_64
0x3d
8
? kmem_cache_alloc
vmlinux 0xffffffff97ff7742
Build id: 3.10.0-862.3.2.el7.x86_64
0x1c2
9
ext4_da_get_block_prep
ext4 0xffffffffc06fd720
Build id: 3.10.0-862.3.2.el7.x86_64
0x370
10
? alloc_page_buffers
vmlinux 0xffffffff98051ed3
Build id: 3.10.0-862.3.2.el7.x86_64
0x63
11
__block_write_begin_int
vmlinux 0xffffffff98053fe8
Build id: 3.10.0-862.3.2.el7.x86_64
0x198
12
? kmem_cache_alloc
vmlinux 0xffffffff97ff7742
Build id: 3.10.0-862.3.2.el7.x86_64
0x1c2
13
? ext4_da_invalidatepage
ext4 0xffffffffc06fd3b0
Build id: 3.10.0-862.3.2.el7.x86_64
0x320
14
? ext4_da_write_begin
ext4 0xffffffffc0703d69
Build id: 3.10.0-862.3.2.el7.x86_64
0x119
15
__block_write_begin
vmlinux 0xffffffff98054451
Build id: 3.10.0-862.3.2.el7.x86_64
0x11
16
ext4_da_write_begin
ext4 0xffffffffc0703db7
Build id: 3.10.0-862.3.2.el7.x86_64
0x167
17
generic_file_buffered_write
vmlinux 0xffffffff97f93a74
Build id: 3.10.0-862.3.2.el7.x86_64
0x124
18
__generic_file_aio_write
vmlinux 0xffffffff97f96052
Build id: 3.10.0-862.3.2.el7.x86_64
0x1e2
19
generic_file_aio_write
vmlinux 0xffffffff97f962c9
Build id: 3.10.0-862.3.2.el7.x86_64
0x59
20
ext4_file_write
ext4 0xffffffffc06f91e2
Build id: 3.10.0-862.3.2.el7.x86_64
0xd2
21
? ext4_file_read
ext4 0xffffffffc06f8e53
Build id: 3.10.0-862.3.2.el7.x86_64
0x43
22
do_sync_write
vmlinux 0xffffffff9801a533
Build id: 3.10.0-862.3.2.el7.x86_64
0x93
23
vfs_write
vmlinux 0xffffffff9801b010
Build id: 3.10.0-862.3.2.el7.x86_64
0xc0
24
SyS_write
vmlinux 0xffffffff9801be3f
Build id: 3.10.0-862.3.2.el7.x86_64
0x7f
25
system_call_fastpath
vmlinux 0xffffffff9852082f
Build id: 3.10.0-862.3.2.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#37 stuck for 22s! [java:35869]
Modules linked in: ip_set binfmt_misc veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter tcp_diag inet_diag dm_round_robin iscsi_tcp libiscsi_tcp bonding ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ucm rpcrdma sunrpc dm_mirror dm_region_hash dm_log rdma_ucm ib_uverbs ib_iser rdma_cm iw_cm ib_umad libiscsi ib_ipoib scsi_transport_iscsi ib_cm iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm vfat fat irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul
glue_helper ablk_helper cryptd mlx4_ib pcspkr joydev ib_core dm_multipath dm_mod i2c_i801 lpc_ich ioatdma hpilo hpwdt sg dca ipmi_si ipmi_devintf ipmi_msghandler pcc_cpufreq shpchp acpi_power_meter wmi ip_tables ext4 mbcache jbd2 mlx4_en sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops bnx2x ttm mlx4_core drm crct10dif_pclmul crct10dif_common crc32c_intel tg3 hpsa i2c_core mdio libcrc32c ptp devlink scsi_transport_sas pps_core
CPU: 37 PID: 35869 Comm: java Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.3.2.el7.x86_64 #1
Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 12/27/2015
task: ffff8d07bfc73f40 ti: ffff8d4766368000 task.ti: ffff8d4766368000
RIP: 0010:[<ffffffff97f08b36>]  [<ffffffff97f08b36>] native_queued_spin_lock_slowpath+0x126/0x200
RSP: 0018:ffff8d476636b8c8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8d42f9fed0a0 RCX: 0000000001290000
RDX: ffff8d5eff0d9700 RSI: 0000000000690001 RDI: ffff8d5efb386c80
RBP: ffff8d476636b8c8 R08: ffff8d5eff459700 R09: 0000000000000000
R10: ffff8d5eff45ba60 R11: ffffee62e1e35480 R12: ffff8d476636b868
R13: ffff8d42f9fed170 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00007fb9337cc700(0000) GS:ffff8d5eff440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb92d226ff0 CR3: 0000005fa6f84000 CR4: 00000000001607e0
Call Trace:
[<ffffffff98509450>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff98516780>] _raw_spin_lock+0x20/0x30
[<ffffffffc0742437>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc072e365>] ext4_ext_map_blocks+0x7b5/0xf60 [ext4]
[<ffffffff97f9e497>] ? __alloc_pages_nodemask+0x197/0xbb0
[<ffffffff9805199d>] ? __brelse+0x3d/0x50
[<ffffffff97ff7742>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc06fd720>] ext4_da_get_block_prep+0x370/0x4c0 [ext4]
[<ffffffff98051ed3>] ? alloc_page_buffers+0x63/0xc0
[<ffffffff98053fe8>] __block_write_begin_int+0x198/0x5f0
[<ffffffff97ff7742>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc06fd3b0>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc0703d69>] ? ext4_da_write_begin+0x119/0x360 [ext4]
[<ffffffff98054451>] __block_write_begin+0x11/0x20
[<ffffffffc0703db7>] ext4_da_write_begin+0x167/0x360 [ext4]
[<ffffffff97f93a74>] generic_file_buffered_write+0x124/0x2c0
[<ffffffff97f96052>] __generic_file_aio_write+0x1e2/0x400
[<ffffffff97f962c9>] generic_file_aio_write+0x59/0xa0
[<ffffffffc06f91e2>] ext4_file_write+0xd2/0x1e0 [ext4]
[<ffffffffc06f8e53>] ? ext4_file_read+0x43/0x130 [ext4]
[<ffffffff9801a533>] do_sync_write+0x93/0xe0
[<ffffffff9801b010>] vfs_write+0xc0/0x1f0
[<ffffffff9801be3f>] SyS_write+0x7f/0xf0
[<ffffffff9852082f>] system_call_fastpath+0x1c/0x21
Code: 0d 48 98 83 e2 30 48 81 c2 00 97 01 00 48 03 14 c5 e0 52 b3 98 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 
Complete report #713755
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff91312286
Build id: 3.10.0-957.12.1.el7.x86_64
0x156
2
queued_spin_lock_slowpath
vmlinux 0xffffffff9195d1ab
Build id: 3.10.0-957.12.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff9196b660
Build id: 3.10.0-957.12.1.el7.x86_64
0x20
4
ext4_es_lru_add
ext4 0xffffffffc0473bb7
Build id: 3.10.0-957.12.1.el7.x86_64
0x57
5
ext4_ext_map_blocks
ext4 0xffffffffc045f8f5
Build id: 3.10.0-957.12.1.el7.x86_64
0x7b5
6
? do_select
vmlinux 0xffffffff9145803e
Build id: 3.10.0-957.12.1.el7.x86_64
0x73e
7
? kmem_cache_alloc
vmlinux 0xffffffff9141c312
Build id: 3.10.0-957.12.1.el7.x86_64
0x1c2
8
ext4_da_get_block_prep
ext4 0xffffffffc042d8d0
Build id: 3.10.0-957.12.1.el7.x86_64
0x370
9
? alloc_page_buffers
vmlinux 0xffffffff91479123
Build id: 3.10.0-957.12.1.el7.x86_64
0x63
10
__block_write_begin_int
vmlinux 0xffffffff9147b238
Build id: 3.10.0-957.12.1.el7.x86_64
0x198
11
? kmem_cache_alloc
vmlinux 0xffffffff9141c312
Build id: 3.10.0-957.12.1.el7.x86_64
0x1c2
12
? ext4_da_invalidatepage
ext4 0xffffffffc042d560
Build id: 3.10.0-957.12.1.el7.x86_64
0x320
13
? ext4_da_write_begin
ext4 0xffffffffc0434099
Build id: 3.10.0-957.12.1.el7.x86_64
0x119
14
__block_write_begin
vmlinux 0xffffffff9147b6a1
Build id: 3.10.0-957.12.1.el7.x86_64
0x11
15
ext4_da_write_begin
ext4 0xffffffffc04340e7
Build id: 3.10.0-957.12.1.el7.x86_64
0x167
16
generic_file_buffered_write
vmlinux 0xffffffff913b60d4
Build id: 3.10.0-957.12.1.el7.x86_64
0x124
17
__generic_file_aio_write
vmlinux 0xffffffff913b8a12
Build id: 3.10.0-957.12.1.el7.x86_64
0x1e2
18
generic_file_aio_write
vmlinux 0xffffffff913b8c89
Build id: 3.10.0-957.12.1.el7.x86_64
0x59
19
ext4_file_write
ext4 0xffffffffc0429322
Build id: 3.10.0-957.12.1.el7.x86_64
0xd2
20
do_sync_write
vmlinux 0xffffffff91440be3
Build id: 3.10.0-957.12.1.el7.x86_64
0x93
21
vfs_write
vmlinux 0xffffffff914416d0
Build id: 3.10.0-957.12.1.el7.x86_64
0xc0
22
SyS_write
vmlinux 0xffffffff914424ef
Build id: 3.10.0-957.12.1.el7.x86_64
0x7f
23
system_call_fastpath
vmlinux 0xffffffff91975d9b
Build id: 3.10.0-957.12.1.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [rsync:94789]
Modules linked in: target_core_user target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod ebt_arp vhost_net vhost macvtap macvlan tun iscsi_tcp libiscsi_tcp libiscsi drbd_transport_tcp(OE) drbd(OE) fuse udp_diag tcp_diag inet_diag nf_conntrack_netlink geneve binfmt_misc devlink bnx2fc cnic uio fcoe libfcoe libfc scsi_transport_fc scsi_tgt scsi_transport_iscsi ip6t_MASQUERADE nf_nat_masquerade_ipv6 ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack openvswitch ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables vxlan ip6_udp_tunnel udp_tunnel iptable_filter sunrpc vfat fat xfs libcrc32c dm_service_time skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm dell_smbios iTCO_wdt iTCO_vendor_support dcdbas dell_wmi_descriptor irqbypass crc32_pclmul ghash_clmulni_intel dm_multipath aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg mei_me i2c_i801 lpc_ich mei wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter tpm_crb acpi_pad ip_tables sr_mod ext4 cdrom mbcache jbd2 uas usb_storage sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm
ahci drm i40e nvme igb libahci nvme_core megaraid_sas libata drm_panel_orientation_quirks ptp dca pps_core i2c_algo_bit nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod 8021q garp mrp bridge stp llc bonding dummy
CPU: 19 PID: 94789 Comm: rsync Kdump: loaded Tainted: G           OE  ------------   3.10.0-957.12.1.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R640/0PHYDR, BIOS 2.13.3 12/13/2021
task: ffff9f36cbfb0000 ti: ffff9f29602f0000 task.ti: ffff9f29602f0000
RIP: 0010:[<ffffffff91312286>]  [<ffffffff91312286>] native_queued_spin_lock_slowpath+0x156/0x200
RSP: 0018:ffff9f29602f38c8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff9f2f4f011c38 RCX: 0000000000990000
RDX: 0000000000290001 RSI: 0000000000000101 RDI: ffff9f537564c480
RBP: ffff9f29602f38c8 R08: ffff9f827c05b780 R09: 0000000000000000
R10: ffff9f827c05f0c0 R11: ffffedf050ad0800 R12: ffff9f29602f3868
R13: ffff9f2f4f011d08 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00007fe7cf158740(0000) GS:ffff9f827c040000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe7c78cd000 CR3: 0000001cfbfbe000 CR4: 00000000007627e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff9195d1ab>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff9196b660>] _raw_spin_lock+0x20/0x30
[<ffffffffc0473bb7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc045f8f5>] ext4_ext_map_blocks+0x7b5/0xf60 [ext4]
[<ffffffff9145803e>] ? do_select+0x73e/0x7c0
[<ffffffff9141c312>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc042d8d0>] ext4_da_get_block_prep+0x370/0x4c0 [ext4]
[<ffffffff91479123>] ? alloc_page_buffers+0x63/0xc0
[<ffffffff9147b238>] __block_write_begin_int+0x198/0x5f0
[<ffffffff9141c312>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc042d560>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc0434099>] ? ext4_da_write_begin+0x119/0x360 [ext4]
[<ffffffff9147b6a1>] __block_write_begin+0x11/0x20
[<ffffffffc04340e7>] ext4_da_write_begin+0x167/0x360 [ext4]
[<ffffffff913b60d4>] generic_file_buffered_write+0x124/0x2c0
[<ffffffff913b8a12>] __generic_file_aio_write+0x1e2/0x400
[<ffffffff913b8c89>] generic_file_aio_write+0x59/0xa0
[<ffffffffc0429322>] ext4_file_write+0xd2/0x1e0 [ext4]
[<ffffffff91440be3>] do_sync_write+0x93/0xe0
[<ffffffff914416d0>] vfs_write+0xc0/0x1f0
[<ffffffff914424ef>] SyS_write+0x7f/0xf0
[<ffffffff91975d9b>] system_call_fastpath+0x22/0x27
Code: 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 <8b> 17 0f b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 
Complete report #650160
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffffa0708b36
Build id: 3.10.0-862.3.2.el7.x86_64
0x126
2
queued_spin_lock_slowpath
vmlinux 0xffffffffa0d09450
Build id: 3.10.0-862.3.2.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffa0d16780
Build id: 3.10.0-862.3.2.el7.x86_64
0x20
4
ext4_es_lru_add
ext4 0xffffffffc0530437
Build id: 3.10.0-862.3.2.el7.x86_64
0x57
5
ext4_ext_map_blocks
ext4 0xffffffffc051c365
Build id: 3.10.0-862.3.2.el7.x86_64
0x7b5
6
? drain_stock.isra.22
vmlinux 0xffffffffa080a220
Build id: 3.10.0-862.3.2.el7.x86_64
0x70
7
? __mem_cgroup_try_charge
vmlinux 0xffffffffa080da72
Build id: 3.10.0-862.3.2.el7.x86_64
0x182
8
? kmem_cache_alloc
vmlinux 0xffffffffa07f7742
Build id: 3.10.0-862.3.2.el7.x86_64
0x1c2
9
ext4_da_get_block_prep
ext4 0xffffffffc04eb720
Build id: 3.10.0-862.3.2.el7.x86_64
0x370
10
? alloc_page_buffers
vmlinux 0xffffffffa0851ed3
Build id: 3.10.0-862.3.2.el7.x86_64
0x63
11
__block_write_begin_int
vmlinux 0xffffffffa0853fe8
Build id: 3.10.0-862.3.2.el7.x86_64
0x198
12
? kmem_cache_alloc
vmlinux 0xffffffffa07f7742
Build id: 3.10.0-862.3.2.el7.x86_64
0x1c2
13
? ext4_da_invalidatepage
ext4 0xffffffffc04eb3b0
Build id: 3.10.0-862.3.2.el7.x86_64
0x320
14
? ext4_da_write_begin
ext4 0xffffffffc04f1d69
Build id: 3.10.0-862.3.2.el7.x86_64
0x119
15
__block_write_begin
vmlinux 0xffffffffa0854451
Build id: 3.10.0-862.3.2.el7.x86_64
0x11
16
ext4_da_write_begin
ext4 0xffffffffc04f1db7
Build id: 3.10.0-862.3.2.el7.x86_64
0x167
17
generic_file_buffered_write
vmlinux 0xffffffffa0793a74
Build id: 3.10.0-862.3.2.el7.x86_64
0x124
18
__generic_file_aio_write
vmlinux 0xffffffffa0796052
Build id: 3.10.0-862.3.2.el7.x86_64
0x1e2
19
? do_last
vmlinux 0xffffffffa082b2dd
Build id: 3.10.0-862.3.2.el7.x86_64
0x66d
20
generic_file_aio_write
vmlinux 0xffffffffa07962c9
Build id: 3.10.0-862.3.2.el7.x86_64
0x59
21
ext4_file_write
ext4 0xffffffffc04e71e2
Build id: 3.10.0-862.3.2.el7.x86_64
0xd2
22
? user_path_at_empty
vmlinux 0xffffffffa082da72
Build id: 3.10.0-862.3.2.el7.x86_64
0x72
23
do_sync_write
vmlinux 0xffffffffa081a533
Build id: 3.10.0-862.3.2.el7.x86_64
0x93
24
vfs_write
vmlinux 0xffffffffa081b010
Build id: 3.10.0-862.3.2.el7.x86_64
0xc0
25
SyS_write
vmlinux 0xffffffffa081be3f
Build id: 3.10.0-862.3.2.el7.x86_64
0x7f
26
system_call_fastpath
vmlinux 0xffffffffa0d2082f
Build id: 3.10.0-862.3.2.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [java:27971]
Modules linked in: binfmt_misc vsock_diag vsock sctp_diag sctp unix_diag af_packet_diag netlink_diag tcp_diag udp_diag inet_diag veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_round_robin iscsi_tcp libiscsi_tcp bonding ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ucm rpcrdma dm_mirror dm_region_hash dm_log sunrpc rdma_ucm ib_uverbs ib_iser ib_umad rdma_cm ib_ipoib iw_cm libiscsi scsi_transport_iscsi ib_cm iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm vfat
fat irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mlx4_ib ib_core pcspkr dm_multipath sg dm_mod hpilo ioatdma hpwdt lpc_ich i2c_i801 shpchp dca wmi ipmi_si ipmi_devintf ipmi_msghandler pcc_cpufreq acpi_power_meter ip_tables ext4 mbcache jbd2 mlx4_en sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops bnx2x ttm drm mlx4_core crct10dif_pclmul crct10dif_common crc32c_intel tg3 hpsa i2c_core mdio libcrc32c ptp devlink scsi_transport_sas pps_core
CPU: 19 PID: 27971 Comm: java Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.3.2.el7.x86_64 #1
Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 12/27/2015
task: ffff8e5e98fe9fa0 ti: ffff8e9e65f54000 task.ti: ffff8e9e65f54000
RIP: 0010:[<ffffffffa0708b36>]  [<ffffffffa0708b36>] native_queued_spin_lock_slowpath+0x126/0x200
RSP: 0018:ffff8e9e65f578c8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8eb411cd1020 RCX: 0000000000990000
RDX: ffff8e89bfb99700 RSI: 0000000000c10001 RDI: ffff8e8ab6e9e480
RBP: ffff8e9e65f578c8 R08: ffff8eb9bf259700 R09: 0000000000000000
R10: ffff8eb9bf25ba60 R11: ffffee22ad49b080 R12: ffff8e9e65f57868
R13: ffff8eb411cd10f0 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00007fe9334f4700(0000) GS:ffff8eb9bf240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe95cc2aed8 CR3: 0000002dfe31c000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa0d09450>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffffa0d16780>] _raw_spin_lock+0x20/0x30
[<ffffffffc0530437>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc051c365>] ext4_ext_map_blocks+0x7b5/0xf60 [ext4]
[<ffffffffa080a220>] ? drain_stock.isra.22+0x70/0x80
[<ffffffffa080da72>] ? __mem_cgroup_try_charge+0x182/0x670
[<ffffffffa07f7742>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc04eb720>] ext4_da_get_block_prep+0x370/0x4c0 [ext4]
[<ffffffffa0851ed3>] ? alloc_page_buffers+0x63/0xc0
[<ffffffffa0853fe8>] __block_write_begin_int+0x198/0x5f0
[<ffffffffa07f7742>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc04eb3b0>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc04f1d69>] ? ext4_da_write_begin+0x119/0x360 [ext4]
[<ffffffffa0854451>] __block_write_begin+0x11/0x20
[<ffffffffc04f1db7>] ext4_da_write_begin+0x167/0x360 [ext4]
[<ffffffffa0793a74>] generic_file_buffered_write+0x124/0x2c0
[<ffffffffa0796052>] __generic_file_aio_write+0x1e2/0x400
[<ffffffffa082b2dd>] ? do_last+0x66d/0x12c0
[<ffffffffa07962c9>] generic_file_aio_write+0x59/0xa0
[<ffffffffc04e71e2>] ext4_file_write+0xd2/0x1e0 [ext4]
[<ffffffffa082da72>] ? user_path_at_empty+0x72/0xc0
[<ffffffffa081a533>] do_sync_write+0x93/0xe0
[<ffffffffa081b010>] vfs_write+0xc0/0x1f0
[<ffffffffa081be3f>] SyS_write+0x7f/0xf0
[<ffffffffa0d2082f>] system_call_fastpath+0x1c/0x21
Code: 0d 48 98 83 e2 30 48 81 c2 00 97 01 00 48 03 14 c5 e0 52 33 a1 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 
Complete report #518227
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffffa9315b3e
Build id: 3.10.0-1062.el7.x86_64
0x1ce
2
queued_spin_lock_slowpath
vmlinux 0xffffffffa9973365
Build id: 3.10.0-1062.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffa9981ad0
Build id: 3.10.0-1062.el7.x86_64
0x20
4
ext4_es_lru_add
ext4 0xffffffffc0e7f187
Build id: 3.10.0-1062.el7.x86_64
0x57
5
ext4_ext_map_blocks
ext4 0xffffffffc0e6ad45
Build id: 3.10.0-1062.el7.x86_64
0x7b5
6
? kmem_cache_alloc
vmlinux 0xffffffffa9423622
Build id: 3.10.0-1062.el7.x86_64
0x1c2
7
ext4_da_get_block_prep
ext4 0xffffffffc0e38940
Build id: 3.10.0-1062.el7.x86_64
0x370
8
? alloc_page_buffers
vmlinux 0xffffffffa9480863
Build id: 3.10.0-1062.el7.x86_64
0x63
9
__block_write_begin_int
vmlinux 0xffffffffa9482ad8
Build id: 3.10.0-1062.el7.x86_64
0x198
10
? kmem_cache_alloc
vmlinux 0xffffffffa9423622
Build id: 3.10.0-1062.el7.x86_64
0x1c2
11
? ext4_da_invalidatepage
ext4 0xffffffffc0e385d0
Build id: 3.10.0-1062.el7.x86_64
0x320
12
? ext4_da_write_begin
ext4 0xffffffffc0e3f239
Build id: 3.10.0-1062.el7.x86_64
0x119
13
__block_write_begin
vmlinux 0xffffffffa9482fa1
Build id: 3.10.0-1062.el7.x86_64
0x11
14
ext4_da_write_begin
ext4 0xffffffffc0e3f287
Build id: 3.10.0-1062.el7.x86_64
0x167
15
generic_file_buffered_write
vmlinux 0xffffffffa93bb8ef
Build id: 3.10.0-1062.el7.x86_64
0x10f
16
__generic_file_aio_write
vmlinux 0xffffffffa93be152
Build id: 3.10.0-1062.el7.x86_64
0x1e2
17
? do_last
vmlinux 0xffffffffa945723d
Build id: 3.10.0-1062.el7.x86_64
0x66d
18
generic_file_aio_write
vmlinux 0xffffffffa93be3c9
Build id: 3.10.0-1062.el7.x86_64
0x59
19
ext4_file_write
ext4 0xffffffffc0e34352
Build id: 3.10.0-1062.el7.x86_64
0xd2
20
do_sync_write
vmlinux 0xffffffffa9447ba3
Build id: 3.10.0-1062.el7.x86_64
0x93
21
vfs_write
vmlinux 0xffffffffa9448690
Build id: 3.10.0-1062.el7.x86_64
0xc0
22
SyS_write
vmlinux 0xffffffffa94494af
Build id: 3.10.0-1062.el7.x86_64
0x7f
23
system_call_fastpath
vmlinux 0xffffffffa998bede
Build id: 3.10.0-1062.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#67 stuck for 23s! [tar:46277]
Modules linked in: veth tcp_lp fuse sep4_1(OE) socperf2_0(OE) tun pax(OE) devlink ebtable_filter ebtables rpcsec_gss_krb5 mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter bridge stp llc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi overlay(T) ip6table_nat nf_nat_ipv6 ip6table_mangle ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_CHECKSUM iptable_mangle dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support ext4 mbcache jbd2 skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel
kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ixgbe glue_helper ablk_helper cryptd pcspkr mdio dca joydev lpc_ich i2c_i801 sg mei_me mei wmi ipmi_si ipmi_devintf ipmi_msghandler pcc_cpufreq acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm i40e libahci crct10dif_pclmul crct10dif_common crc32c_intel libata megaraid_sas ptp pps_core nfit drm_panel_orientation_quirks libnvdimm
CPU: 67 PID: 46277 Comm: tar Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: Supermicro X11DPi-N(T)/X11DPi-N, BIOS 3.3 02/24/2020
task: ffff914556d7d230 ti: ffff913b31f70000 task.ti: ffff913b31f70000
RIP: 0010:[<ffffffffa9315b3e>]  [<ffffffffa9315b3e>] native_queued_spin_lock_slowpath+0x1ce/0x200
RSP: 0018:ffff913b31f738d8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff9145d288d8b0 RCX: 0000000000000001
RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff9173f805b480
RBP: ffff913b31f738d8 R08: 0000000000000101 R09: ffffffffc0e6ad2a
R10: ffff9153ff2df0c0 R11: fffff9fdff85f4c0 R12: ffff913b31f73878
R13: ffff9145d288d980 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00007f9ff7aa0840(0000) GS:ffff9153ff2c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa52e415c60 CR3: 0000001ff03d4000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa9973365>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffffa9981ad0>] _raw_spin_lock+0x20/0x30
[<ffffffffc0e7f187>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc0e6ad45>] ext4_ext_map_blocks+0x7b5/0xf60 [ext4]
[<ffffffffa9423622>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc0e38940>] ext4_da_get_block_prep+0x370/0x4c0 [ext4]
[<ffffffffa9480863>] ? alloc_page_buffers+0x63/0xc0
[<ffffffffa9482ad8>] __block_write_begin_int+0x198/0x650
[<ffffffffa9423622>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc0e385d0>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc0e3f239>] ? ext4_da_write_begin+0x119/0x360 [ext4]
[<ffffffffa9482fa1>] __block_write_begin+0x11/0x20
[<ffffffffc0e3f287>] ext4_da_write_begin+0x167/0x360 [ext4]
[<ffffffffa93bb8ef>] generic_file_buffered_write+0x10f/0x270
[<ffffffffa93be152>] __generic_file_aio_write+0x1e2/0x400
[<ffffffffa945723d>] ? do_last+0x66d/0x1290
[<ffffffffa93be3c9>] generic_file_aio_write+0x59/0xa0
[<ffffffffc0e34352>] ext4_file_write+0xd2/0x1e0 [ext4]
[<ffffffffa9447ba3>] do_sync_write+0x93/0xe0
[<ffffffffa9448690>] vfs_write+0xc0/0x1f0
[<ffffffffa94494af>] SyS_write+0x7f/0xf0
[<ffffffffa998bede>] system_call_fastpath+0x25/0x2a
Code: 37 81 fe 00 01 00 00 74 f4 e9 93 fe ff ff 0f 1f 80 00 00 00 00 83 fa 01 75 11 0f 1f 00 e9 68 fe ff ff 0f 1f 00 85 c0 74 0c f3 90 <8b> 07 0f b6 c0 83 f8 03 75 f0 b8 01 00 00 00 66 89 07 5d c3 66 
Complete report #291996
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff9ef15b66
Build id: 3.10.0-1062.4.1.el7.x86_64
0x126
2
queued_spin_lock_slowpath
vmlinux 0xffffffff9f5733cb
Build id: 3.10.0-1062.4.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff9f581430
Build id: 3.10.0-1062.4.1.el7.x86_64
0x20
4
ext4_es_lru_add
ext4 0xffffffffc0612187
Build id: 3.10.0-1062.4.1.el7.x86_64
0x57
5
ext4_ext_map_blocks
ext4 0xffffffffc05fdd45
Build id: 3.10.0-1062.4.1.el7.x86_64
0x7b5
6
? __alloc_pages_nodemask
vmlinux 0xffffffff9efc6854
Build id: 3.10.0-1062.4.1.el7.x86_64
0x184
7
? zone_statistics
vmlinux 0xffffffff9efde648
Build id: 3.10.0-1062.4.1.el7.x86_64
0x88
8
? kmem_cache_alloc
vmlinux 0xffffffff9f0237b2
Build id: 3.10.0-1062.4.1.el7.x86_64
0x1c2
9
ext4_da_get_block_prep
ext4 0xffffffffc05cb940
Build id: 3.10.0-1062.4.1.el7.x86_64
0x370
10
? alloc_page_buffers
vmlinux 0xffffffff9f081113
Build id: 3.10.0-1062.4.1.el7.x86_64
0x63
11
__block_write_begin_int
vmlinux 0xffffffff9f083348
Build id: 3.10.0-1062.4.1.el7.x86_64
0x198
12
? kmem_cache_alloc
vmlinux 0xffffffff9f0237b2
Build id: 3.10.0-1062.4.1.el7.x86_64
0x1c2
13
? ext4_da_invalidatepage
ext4 0xffffffffc05cb5d0
Build id: 3.10.0-1062.4.1.el7.x86_64
0x320
14
? ext4_da_write_begin
ext4 0xffffffffc05d2239
Build id: 3.10.0-1062.4.1.el7.x86_64
0x119
15
__block_write_begin
vmlinux 0xffffffff9f083811
Build id: 3.10.0-1062.4.1.el7.x86_64
0x11
16
ext4_da_write_begin
ext4 0xffffffffc05d2287
Build id: 3.10.0-1062.4.1.el7.x86_64
0x167
17
generic_file_buffered_write
vmlinux 0xffffffff9efbb9bf
Build id: 3.10.0-1062.4.1.el7.x86_64
0x10f
18
__generic_file_aio_write
vmlinux 0xffffffff9efbe222
Build id: 3.10.0-1062.4.1.el7.x86_64
0x1e2
19
generic_file_aio_write
vmlinux 0xffffffff9efbe499
Build id: 3.10.0-1062.4.1.el7.x86_64
0x59
20
ext4_file_write
ext4 0xffffffffc05c7352
Build id: 3.10.0-1062.4.1.el7.x86_64
0xd2
21
do_sync_write
vmlinux 0xffffffff9f048433
Build id: 3.10.0-1062.4.1.el7.x86_64
0x93
22
vfs_write
vmlinux 0xffffffff9f048f20
Build id: 3.10.0-1062.4.1.el7.x86_64
0xc0
23
SyS_write
vmlinux 0xffffffff9f049d3f
Build id: 3.10.0-1062.4.1.el7.x86_64
0x7f
24
system_call_fastpath
vmlinux 0xffffffff9f58bede
Build id: 3.10.0-1062.4.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#137 stuck for 22s! [hmmsearch:137119]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter devlink ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi sunrpc iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev i2c_i801 ses lpc_ich enclosure mei_me scsi_transport_sas mei sg ipmi_si ipmi_devintf ipmi_msghandler acpi_pad pcc_cpufreq binfmt_misc ip_tables xfs libcrc32c raid1 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul
crct10dif_common crc32c_intel mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ixgbe drm libahci igb libata aacraid mdio ptp pps_core dca drm_panel_orientation_quirks i2c_algo_bit wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 137 PID: 137119 Comm: hmmsearch Tainted: G             L ------------   3.10.0-1062.4.1.el7.x86_64 #1
Hardware name: POWERLEADER DaPengWan/Default string, BIOS 5.11 12/05/2017
task: ffff91dd03ed3150 ti: ffff9284197d0000 task.ti: ffff9284197d0000
RIP: 0010:[<ffffffff9ef15b66>]  [<ffffffff9ef15b66>] native_queued_spin_lock_slowpath+0x126/0x200
RSP: 0018:ffff9284197d38d8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff927feaf2b060 RCX: 0000000004490000
RDX: ffff923d7f91b800 RSI: 0000000002010101 RDI: ffff91de53741c80
RBP: ffff9284197d38d8 R08: ffff929d7fc5b800 R09: 0000000000000000
R10: ffff929d7fc5f0c0 R11: fffff3d481b23640 R12: ffff9284197d3878
R13: ffff927feaf2b130 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00007ffad29c3740(0000) GS:ffff929d7fc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001df5ff0 CR3: 000000c2ce640000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9f5733cb>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff9f581430>] _raw_spin_lock+0x20/0x30
[<ffffffffc0612187>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc05fdd45>] ext4_ext_map_blocks+0x7b5/0xf60 [ext4]
[<ffffffff9efc6854>] ? __alloc_pages_nodemask+0x184/0xbe0
[<ffffffff9efde648>] ? zone_statistics+0x88/0xa0
[<ffffffff9f0237b2>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc05cb940>] ext4_da_get_block_prep+0x370/0x4c0 [ext4]
[<ffffffff9f081113>] ? alloc_page_buffers+0x63/0xc0
[<ffffffff9f083348>] __block_write_begin_int+0x198/0x650
[<ffffffff9f0237b2>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc05cb5d0>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc05d2239>] ? ext4_da_write_begin+0x119/0x360 [ext4]
[<ffffffff9f083811>] __block_write_begin+0x11/0x20
[<ffffffffc05d2287>] ext4_da_write_begin+0x167/0x360 [ext4]
[<ffffffff9efbb9bf>] generic_file_buffered_write+0x10f/0x270
[<ffffffff9efbe222>] __generic_file_aio_write+0x1e2/0x400
[<ffffffff9efbe499>] generic_file_aio_write+0x59/0xa0
[<ffffffffc05c7352>] ext4_file_write+0xd2/0x1e0 [ext4]
[<ffffffff9f048433>] do_sync_write+0x93/0xe0
[<ffffffff9f048f20>] vfs_write+0xc0/0x1f0
[<ffffffff9f049d3f>] SyS_write+0x7f/0xf0
[<ffffffff9f58bede>] system_call_fastpath+0x25/0x2a
Code: 0d 48 98 83 e2 30 48 81 c2 00 b8 01 00 48 03 14 c5 e0 d6 b4 9f 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 
Complete report #167214
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffffa2112222
Build id: 3.10.0-957.5.1.el7.x86_64
0x122
2
queued_spin_lock_slowpath
vmlinux 0xffffffffa275bfcb
Build id: 3.10.0-957.5.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffa276a480
Build id: 3.10.0-957.5.1.el7.x86_64
0x20
4
ext4_es_lru_add
ext4 0xffffffffc0397bb7
Build id: 3.10.0-957.5.1.el7.x86_64
0x57
5
ext4_ext_map_blocks
ext4 0xffffffffc03838f5
Build id: 3.10.0-957.5.1.el7.x86_64
0x7b5
6
? __brelse
vmlinux 0xffffffffa2278b6d
Build id: 3.10.0-957.5.1.el7.x86_64
0x3d
7
? kmem_cache_alloc
vmlinux 0xffffffffa221c272
Build id: 3.10.0-957.5.1.el7.x86_64
0x1c2
8
ext4_da_get_block_prep
ext4 0xffffffffc03518d0
Build id: 3.10.0-957.5.1.el7.x86_64
0x370
9
? alloc_page_buffers
vmlinux 0xffffffffa22790a3
Build id: 3.10.0-957.5.1.el7.x86_64
0x63
10
__block_write_begin_int
vmlinux 0xffffffffa227b1b8
Build id: 3.10.0-957.5.1.el7.x86_64
0x198
11
? kmem_cache_alloc
vmlinux 0xffffffffa221c272
Build id: 3.10.0-957.5.1.el7.x86_64
0x1c2
12
? ext4_da_invalidatepage
ext4 0xffffffffc0351560
Build id: 3.10.0-957.5.1.el7.x86_64
0x320
13
? ext4_da_write_begin
ext4 0xffffffffc0358099
Build id: 3.10.0-957.5.1.el7.x86_64
0x119
14
__block_write_begin
vmlinux 0xffffffffa227b621
Build id: 3.10.0-957.5.1.el7.x86_64
0x11
15
ext4_da_write_begin
ext4 0xffffffffc03580e7
Build id: 3.10.0-957.5.1.el7.x86_64
0x167
16
generic_file_buffered_write
vmlinux 0xffffffffa21b5fc4
Build id: 3.10.0-957.5.1.el7.x86_64
0x124
17
__generic_file_aio_write
vmlinux 0xffffffffa21b8902
Build id: 3.10.0-957.5.1.el7.x86_64
0x1e2
18
generic_file_aio_write
vmlinux 0xffffffffa21b8b79
Build id: 3.10.0-957.5.1.el7.x86_64
0x59
19
ext4_file_write
ext4 0xffffffffc034d322
Build id: 3.10.0-957.5.1.el7.x86_64
0xd2
20
? tlb_finish_mmu
vmlinux 0xffffffffa21e5d25
Build id: 3.10.0-957.5.1.el7.x86_64
0x55
21
do_sync_write
vmlinux 0xffffffffa2240b63
Build id: 3.10.0-957.5.1.el7.x86_64
0x93
22
vfs_write
vmlinux 0xffffffffa2241650
Build id: 3.10.0-957.5.1.el7.x86_64
0xc0
23
SyS_write
vmlinux 0xffffffffa224246f
Build id: 3.10.0-957.5.1.el7.x86_64
0x7f
24
system_call_fastpath
vmlinux 0xffffffffa2774ddb
Build id: 3.10.0-957.5.1.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [pdbextract:431332]
Modules linked in: veth nvidia_uvm(OE) nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel sr_mod iTCO_wdt iTCO_vendor_support snd_hda_codec_hdmi
kvm cdrom irqbypass joydev snd_hda_intel snd_hda_codec crc32_pclmul ghash_clmulni_intel snd_hda_core snd_hwdep aesni_intel snd_seq lrw gf128mul snd_seq_device glue_helper snd_pcm snd_timer snd soundcore ablk_helper cryptd i2c_i801 mei_me mei sg pcspkr lpc_ich ipmi_si ipmi_devintf acpi_pad ipmi_msghandler acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic uas usb_storage nouveau crct10dif_pclmul crct10dif_common crc32c_intel video mxm_wmi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm serio_raw ahci igb libahci libata ptp pps_core drm_panel_orientation_quirks dca i2c_algo_bit wmi nfit libnvdimm
CPU: 18 PID: 431332 Comm: pdbextract Kdump: loaded Tainted: P           OEL ------------ T 3.10.0-957.5.1.el7.x86_64 #1
Hardware name: ASUSTeK COMPUTER INC. ESC4000 G4/Z11PG-D16 Series, BIOS 0601 03/01/2018
task: ffff8c671c004100 ti: ffff8c66eda3c000 task.ti: ffff8c66eda3c000
RIP: 0010:[<ffffffffa2112222>]  [<ffffffffa2112222>] native_queued_spin_lock_slowpath+0x122/0x200
RSP: 0018:ffff8c66eda3f8c8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8c66ce45a850 RCX: 0000000000910000
RDX: ffff8c5c1da9b780 RSI: 0000000000110101 RDI: ffff8c7404fdbc80
RBP: ffff8c66eda3f8c8 R08: ffff8c741bf9b780 R09: 0000000000000000
R10: ffff8c741bf9f0c0 R11: ffffe2d094a99b80 R12: ffff8c66eda3f868
R13: ffff8c66ce45a920 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00002b192e1ab9c0(0000) GS:ffff8c741bf80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b192e18d000 CR3: 00000021ed0c4000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa275bfcb>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffffa276a480>] _raw_spin_lock+0x20/0x30
[<ffffffffc0397bb7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc03838f5>] ext4_ext_map_blocks+0x7b5/0xf60 [ext4]
[<ffffffffa2278b6d>] ? __brelse+0x3d/0x50
[<ffffffffa221c272>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc03518d0>] ext4_da_get_block_prep+0x370/0x4c0 [ext4]
[<ffffffffa22790a3>] ? alloc_page_buffers+0x63/0xc0
[<ffffffffa227b1b8>] __block_write_begin_int+0x198/0x5f0
[<ffffffffa221c272>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc0351560>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc0358099>] ? ext4_da_write_begin+0x119/0x360 [ext4]
[<ffffffffa227b621>] __block_write_begin+0x11/0x20
[<ffffffffc03580e7>] ext4_da_write_begin+0x167/0x360 [ext4]
[<ffffffffa21b5fc4>] generic_file_buffered_write+0x124/0x2c0
[<ffffffffa21b8902>] __generic_file_aio_write+0x1e2/0x400
[<ffffffffa21b8b79>] generic_file_aio_write+0x59/0xa0
[<ffffffffc034d322>] ext4_file_write+0xd2/0x1e0 [ext4]
[<ffffffffa21e5d25>] ? tlb_finish_mmu+0x55/0x60
[<ffffffffa2240b63>] do_sync_write+0x93/0xe0
[<ffffffffa2241650>] vfs_write+0xc0/0x1f0
[<ffffffffa224246f>] SyS_write+0x7f/0xf0
[<ffffffffa2774ddb>] system_call_fastpath+0x22/0x27
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 60 b9 d4 a2 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 
Complete report #64913
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffffba313532
Build id: 3.10.0-957.27.2.el7.x86_64
0x122
2
queued_spin_lock_slowpath
vmlinux 0xffffffffba95e2cb
Build id: 3.10.0-957.27.2.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffba96c7a0
Build id: 3.10.0-957.27.2.el7.x86_64
0x20
4
ext4_es_lru_add
ext4 0xffffffffc030fbc7
Build id: 3.10.0-957.27.2.el7.x86_64
0x57
5
ext4_ext_map_blocks
ext4 0xffffffffc02fb905
Build id: 3.10.0-957.27.2.el7.x86_64
0x7b5
6
? kmem_cache_alloc
vmlinux 0xffffffffba41d7f2
Build id: 3.10.0-957.27.2.el7.x86_64
0x1c2
7
ext4_da_get_block_prep
ext4 0xffffffffc02c98d0
Build id: 3.10.0-957.27.2.el7.x86_64
0x370
8
? alloc_page_buffers
vmlinux 0xffffffffba47a1e3
Build id: 3.10.0-957.27.2.el7.x86_64
0x63
9
__block_write_begin_int
vmlinux 0xffffffffba47c2f8
Build id: 3.10.0-957.27.2.el7.x86_64
0x198
10
? kmem_cache_alloc
vmlinux 0xffffffffba41d7f2
Build id: 3.10.0-957.27.2.el7.x86_64
0x1c2
11
? ext4_da_invalidatepage
ext4 0xffffffffc02c9560
Build id: 3.10.0-957.27.2.el7.x86_64
0x320
12
? ext4_da_write_begin
ext4 0xffffffffc02d0099
Build id: 3.10.0-957.27.2.el7.x86_64
0x119
13
__block_write_begin
vmlinux 0xffffffffba47c761
Build id: 3.10.0-957.27.2.el7.x86_64
0x11
14
ext4_da_write_begin
ext4 0xffffffffc02d00e7
Build id: 3.10.0-957.27.2.el7.x86_64
0x167
15
generic_file_buffered_write
vmlinux 0xffffffffba3b73a4
Build id: 3.10.0-957.27.2.el7.x86_64
0x124
16
__generic_file_aio_write
vmlinux 0xffffffffba3b9ce2
Build id: 3.10.0-957.27.2.el7.x86_64
0x1e2
17
? __mem_cgroup_commit_charge
vmlinux 0xffffffffba4342b2
Build id: 3.10.0-957.27.2.el7.x86_64
0xe2
18
generic_file_aio_write
vmlinux 0xffffffffba3b9f59
Build id: 3.10.0-957.27.2.el7.x86_64
0x59
19
ext4_file_write
ext4 0xffffffffc02c5322
Build id: 3.10.0-957.27.2.el7.x86_64
0xd2
20
? ext4_file_read
ext4 0xffffffffc02c4d93
Build id: 3.10.0-957.27.2.el7.x86_64
0x43
21
do_sync_write
vmlinux 0xffffffffba441c13
Build id: 3.10.0-957.27.2.el7.x86_64
0x93
22
vfs_write
vmlinux 0xffffffffba442700
Build id: 3.10.0-957.27.2.el7.x86_64
0xc0
23
SyS_write
vmlinux 0xffffffffba44351f
Build id: 3.10.0-957.27.2.el7.x86_64
0x7f
24
system_call_fastpath
vmlinux 0xffffffffba976ddb
Build id: 3.10.0-957.27.2.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [BackgrProcPool:10842]
Modules linked in: binfmt_misc nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c xt_comment iptable_filter ip_gre gre ip_tunnel bonding dcdbas iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm ipmi_ssif irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg lpc_ich i2c_i801 mei_me mei ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter tcp_yeah tcp_vegas ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm crc32c_intel drm ahci ixgbe libahci megaraid_sas igb libata mdio ptp pps_core dca i2c_algo_bit drm_panel_orientation_quirks nfit
libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 15 PID: 10842 Comm: BackgrProcPool Kdump: loaded Tainted: G             L ------------   3.10.0-957.27.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R740xd/08D89F, BIOS 1.4.8 05/21/2018
task: ffff9299646b30c0 ti: ffff92997744c000 task.ti: ffff92997744c000
RIP: 0010:[<ffffffffba313532>]  [<ffffffffba313532>] native_queued_spin_lock_slowpath+0x122/0x200
RSP: 0018:ffff92997744f8c8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff927cd3eec488 RCX: 0000000000790000
RDX: ffff92997c11b780 RSI: 0000000000c90101 RDI: ffff92797d1f4c80
RBP: ffff92997744f8c8 R08: ffff92997bfdb780 R09: 0000000000000000
R10: ffff92997bfdf0c0 R11: ffffd85f8e11b4c0 R12: ffff92997744f868
R13: ffff927cd3eec558 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00007fe969ff7700(0000) GS:ffff92997bfc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdf7db2e000 CR3: 0000001fd5ae0000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffba95e2cb>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffffba96c7a0>] _raw_spin_lock+0x20/0x30
[<ffffffffc030fbc7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc02fb905>] ext4_ext_map_blocks+0x7b5/0xf60 [ext4]
[<ffffffffba41d7f2>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc02c98d0>] ext4_da_get_block_prep+0x370/0x4c0 [ext4]
[<ffffffffba47a1e3>] ? alloc_page_buffers+0x63/0xc0
[<ffffffffba47c2f8>] __block_write_begin_int+0x198/0x5f0
[<ffffffffba41d7f2>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc02c9560>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc02d0099>] ? ext4_da_write_begin+0x119/0x360 [ext4]
[<ffffffffba47c761>] __block_write_begin+0x11/0x20
[<ffffffffc02d00e7>] ext4_da_write_begin+0x167/0x360 [ext4]
[<ffffffffba3b73a4>] generic_file_buffered_write+0x124/0x2c0
[<ffffffffba3b9ce2>] __generic_file_aio_write+0x1e2/0x400
[<ffffffffba4342b2>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
[<ffffffffba3b9f59>] generic_file_aio_write+0x59/0xa0
[<ffffffffc02c5322>] ext4_file_write+0xd2/0x1e0 [ext4]
[<ffffffffc02c4d93>] ? ext4_file_read+0x43/0x130 [ext4]
[<ffffffffba441c13>] do_sync_write+0x93/0xe0
[<ffffffffba442700>] vfs_write+0xc0/0x1f0
[<ffffffffba44351f>] SyS_write+0x7f/0xf0
[<ffffffffba976ddb>] system_call_fastpath+0x22/0x27
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 a0 bf f4 ba 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 
Complete report #10381
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffffa9f15a92
Build id: 3.10.0-1062.1.2.el7.x86_64
0x122
2
queued_spin_lock_slowpath
vmlinux 0xffffffffaa5733c5
Build id: 3.10.0-1062.1.2.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffaa581b30
Build id: 3.10.0-1062.1.2.el7.x86_64
0x20
4
ext4_es_lru_add
ext4 0xffffffffc0a9b187
Build id: 3.10.0-1062.1.2.el7.x86_64
0x57
5
ext4_ext_map_blocks
ext4 0xffffffffc0a86d45
Build id: 3.10.0-1062.1.2.el7.x86_64
0x7b5
6
? kmem_cache_alloc
vmlinux 0xffffffffaa023622
Build id: 3.10.0-1062.1.2.el7.x86_64
0x1c2
7
ext4_da_get_block_prep
ext4 0xffffffffc0a54940
Build id: 3.10.0-1062.1.2.el7.x86_64
0x370
8
? alloc_page_buffers
vmlinux 0xffffffffaa080863
Build id: 3.10.0-1062.1.2.el7.x86_64
0x63
9
__block_write_begin_int
vmlinux 0xffffffffaa082ad8
Build id: 3.10.0-1062.1.2.el7.x86_64
0x198
10
? kmem_cache_alloc
vmlinux 0xffffffffaa023622
Build id: 3.10.0-1062.1.2.el7.x86_64
0x1c2
11
? ext4_da_invalidatepage
ext4 0xffffffffc0a545d0
Build id: 3.10.0-1062.1.2.el7.x86_64
0x320
12
? ext4_da_write_begin
ext4 0xffffffffc0a5b239
Build id: 3.10.0-1062.1.2.el7.x86_64
0x119
13
__block_write_begin
vmlinux 0xffffffffaa082fa1
Build id: 3.10.0-1062.1.2.el7.x86_64
0x11
14
ext4_da_write_begin
ext4 0xffffffffc0a5b287
Build id: 3.10.0-1062.1.2.el7.x86_64
0x167
15
generic_file_buffered_write
vmlinux 0xffffffffa9fbb8ef
Build id: 3.10.0-1062.1.2.el7.x86_64
0x10f
16
? down_write
vmlinux 0xffffffffaa57e3e2
Build id: 3.10.0-1062.1.2.el7.x86_64
0x12
17
__generic_file_aio_write
vmlinux 0xffffffffa9fbe152
Build id: 3.10.0-1062.1.2.el7.x86_64
0x1e2
18
generic_file_aio_write
vmlinux 0xffffffffa9fbe3c9
Build id: 3.10.0-1062.1.2.el7.x86_64
0x59
19
ext4_file_write
ext4 0xffffffffc0a50352
Build id: 3.10.0-1062.1.2.el7.x86_64
0xd2
20
do_sync_write
vmlinux 0xffffffffaa047ba3
Build id: 3.10.0-1062.1.2.el7.x86_64
0x93
21
vfs_write
vmlinux 0xffffffffaa048690
Build id: 3.10.0-1062.1.2.el7.x86_64
0xc0
22
SyS_write
vmlinux 0xffffffffaa0494af
Build id: 3.10.0-1062.1.2.el7.x86_64
0x7f
23
system_call_fastpath
vmlinux 0xffffffffaa58cede
Build id: 3.10.0-1062.1.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [mapproxy-seed:13644]
Modules linked in: ext4 mbcache jbd2 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw mpt2sas raid_class scsi_transport_sas mptctl mptbase iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu iTCO_wdt iTCO_vendor_support mxm_wmi dcdbas sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ipmi_ssif cryptd pcspkr lpc_ich sg mei_me mei ipmi_si ipmi_devintf ipmi_msghandler wmi pcc_cpufreq
acpi_power_meter tpm_crb ip_tables xfs sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper ahci syscopyarea sysfillrect sysimgblt fb_sys_fops bnx2x ttm libahci libata drm megaraid_sas crct10dif_pclmul crct10dif_common crc32c_intel mdio ptp drm_panel_orientation_quirks pps_core libcrc32c dm_mirror dm_region_hash dm_log dm_mod
CPU: 8 PID: 13644 Comm: mapproxy-seed Kdump: loaded Tainted: G             L ------------   3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R630/02C2CP, BIOS 2.9.1 12/04/2018
task: ffff98b4c37de2a0 ti: ffff9896fa094000 task.ti: ffff9896fa094000
RIP: 0010:[<ffffffffa9f15a92>]  [<ffffffffa9f15a92>] native_queued_spin_lock_slowpath+0x122/0x200
RSP: 0018:ffff9896fa0978d8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff98b3b538f0e0 RCX: 0000000000410000
RDX: ffff98b4ff45b840 RSI: 0000000000490101 RDI: ffff98b4faf46480
RBP: ffff9896fa0978d8 R08: ffff98b4ff41b840 R09: 0000000000000000
R10: ffff98b4ff41f0c0 R11: fffff8dd324eaa00 R12: ffff9896fa097878
R13: ffff98b3b538f1b0 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00007f04fff99740(0000) GS:ffff98b4ff400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f04fff9e000 CR3: 0000000f91138000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaa5733c5>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffffaa581b30>] _raw_spin_lock+0x20/0x30
[<ffffffffc0a9b187>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc0a86d45>] ext4_ext_map_blocks+0x7b5/0xf60 [ext4]
[<ffffffffaa023622>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc0a54940>] ext4_da_get_block_prep+0x370/0x4c0 [ext4]
[<ffffffffaa080863>] ? alloc_page_buffers+0x63/0xc0
[<ffffffffaa082ad8>] __block_write_begin_int+0x198/0x650
[<ffffffffaa023622>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc0a545d0>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc0a5b239>] ? ext4_da_write_begin+0x119/0x360 [ext4]
[<ffffffffaa082fa1>] __block_write_begin+0x11/0x20
[<ffffffffc0a5b287>] ext4_da_write_begin+0x167/0x360 [ext4]
[<ffffffffa9fbb8ef>] generic_file_buffered_write+0x10f/0x270
[<ffffffffaa57e3e2>] ? down_write+0x12/0x3d
[<ffffffffa9fbe152>] __generic_file_aio_write+0x1e2/0x400
[<ffffffffa9fbe3c9>] generic_file_aio_write+0x59/0xa0
[<ffffffffc0a50352>] ext4_file_write+0xd2/0x1e0 [ext4]
[<ffffffffaa047ba3>] do_sync_write+0x93/0xe0
[<ffffffffaa048690>] vfs_write+0xc0/0x1f0
[<ffffffffaa0494af>] SyS_write+0x7f/0xf0
[<ffffffffaa58cede>] system_call_fastpath+0x25/0x2a
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 40 b8 01 00 48 03 14 c5 e0 d7 b4 aa 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b