Info

Permalink
Function
native_queued_spin_lock_slowpath
First occurrence
2020-11-02
Last occurrence
2024-03-14
State
NEW
Components
kernel
Quality
-16

Statistics

Operating system Count
CentOS 7 43
Architecture Count
x86_64 43
Related packages Count
kernel 43
 0:3.10.0-1160.81.1.el7 14
 0:3.10.0-1062.el7 8
 0:3.10.0-1160.118.1.el7 6
 0:3.10.0-1160.114.2.el7 4
 0:3.10.0-957.el7 4
 0:3.10.0-1062.12.1.el7 3
 0:3.10.0-1062.4.1.el7 1
 0:3.10.0-1062.1.1.el7 1
 0:3.10.0-1160.11.1.el7 1
 0:3.10.0-1160.88.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #997190
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff9a91733d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x1d
2
queued_spin_lock_slowpath
vmlinux 0xffffffff9af7546a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff9af83350
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20
4
do_huge_pmd_numa_page
vmlinux 0xffffffff9aa340f5
Build id: 3.10.0-1062.12.1.el7.x86_64
0x85
5
handle_mm_fault
vmlinux 0xffffffff9a9f4417
Build id: 3.10.0-1062.12.1.el7.x86_64
0x877
6
__do_page_fault
vmlinux 0xffffffff9af88653
Build id: 3.10.0-1062.12.1.el7.x86_64
0x213
7
do_page_fault
vmlinux 0xffffffff9af88975
Build id: 3.10.0-1062.12.1.el7.x86_64
0x35
8
? error_swapgs
vmlinux 0xffffffff9af84ac9
Build id: 3.10.0-1062.12.1.el7.x86_64
0xaa
9
page_fault
vmlinux 0xffffffff9af84778
Build id: 3.10.0-1062.12.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [javamelody siie:30751]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache fuse sunrpc ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg nfit pcspkr joydev libnvdimm i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix scsi_transport_spi mptscsih libata crct10dif_pclmul crct10dif_common crc32c_intel mptbase vmxnet3 serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 30751 Comm: javamelody siie Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffff8e483640b150 ti: ffff8e5068ea8000 task.ti: ffff8e5068ea8000
RIP: 0010:[<ffffffff9a91733d>]  [<ffffffff9a91733d>] native_queued_spin_lock_slowpath+0x1d/0x200
RSP: 0000:ffff8e5068eabd60  EFLAGS: 00000293
RAX: 0000000000000001 RBX: 00000000ffffffff RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffcf0d2635c0b0
RBP: ffff8e5068eabd60 R08: ffff8e5190ad4b00 R09: 0000000000000000
R10: 0000000000000080 R11: 00000007a0a6c290 R12: 0000000000000000
R13: 0000000000000000 R14: ffffcf0d0b169030 R15: 0020021f0008007c
FS:  00007ff4f02f4700(0000) GS:ffff8e4b53480000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff6d0cbf740 CR3: 00000009928b8000 CR4: 00000000000407e0
Call Trace:
[<ffffffff9af7546a>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff9af83350>] _raw_spin_lock+0x20/0x30
[<ffffffff9aa340f5>] do_huge_pmd_numa_page+0x85/0x4b0
[<ffffffff9a9f4417>] handle_mm_fault+0x877/0x9b0
[<ffffffff9af88653>] __do_page_fault+0x213/0x500
[<ffffffff9af88975>] do_page_fault+0x35/0x90
[<ffffffff9af84ac9>] ? error_swapgs+0xaa/0xc0
[<ffffffff9af84778>] page_fault+0x28/0x30
Code: 47 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 <8b> 17 85 d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 0f 
Complete report #956378
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff9371ec96
Build id: 3.10.0-1160.114.2.el7.x86_64
0x156
2
queued_spin_lock_slowpath
vmlinux 0xffffffff93dac21a
Build id: 3.10.0-1160.114.2.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff93dba734
Build id: 3.10.0-1160.114.2.el7.x86_64
0x24
4
do_huge_pmd_numa_page
vmlinux 0xffffffff93843f69
Build id: 3.10.0-1160.114.2.el7.x86_64
0x1f9
5
handle_mm_fault
vmlinux 0xffffffff93802242
Build id: 3.10.0-1160.114.2.el7.x86_64
0xad2
6
__do_page_fault
vmlinux 0xffffffff93dbf7d3
Build id: 3.10.0-1160.114.2.el7.x86_64
0x213
7
do_page_fault
vmlinux 0xffffffff93dbfb05
Build id: 3.10.0-1160.114.2.el7.x86_64
0x35
8
page_fault
vmlinux 0xffffffff93dbb7b8
Build id: 3.10.0-1160.114.2.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 24s! [python3:41675]
Modules linked in: binfmt_misc tcp_lp nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul vmw_balloon glue_helper ablk_helper cryptd pcspkr sg joydev vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea crc_t10dif sysfillrect crct10dif_generic sysimgblt fb_sys_fops ttm ahci drm nfit ata_piix libahci crct10dif_pclmul libnvdimm crct10dif_common crc32c_intel libata
serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 15 PID: 41675 Comm: python3 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.114.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff950a31bbc200 ti: ffff950e2b88c000 task.ti: ffff950e2b88c000
RIP: 0010:[<ffffffff9371ec96>]  [<ffffffff9371ec96>] native_queued_spin_lock_slowpath+0x156/0x200
RSP: 0000:ffff950e2b88fd40  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff950e2b88fe20 RCX: 0000000000790000
RDX: 0000000001090001 RSI: 0000000000610001 RDI: ffffdc60773a22f0
RBP: ffff950e2b88fd40 R08: ffff950a3f9db8c0 R09: ffff950a3fa5b8c0
R10: ffff950e2b88fe20 R11: 000000000013b0fb R12: ffff950a31bbc200
R13: ffff950e2b88fd38 R14: ffff950a7ffdb038 R15: ffff950a31bbc200
FS:  00002b59fc001800(0000) GS:ffff950a3f9c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b5e1d200000 CR3: 0000000f585b6000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffff93dac21a>] queued_spin_lock_slowpath+0xb/0x13
[<ffffffff93dba734>] _raw_spin_lock+0x24/0x30
[<ffffffff93843f69>] do_huge_pmd_numa_page+0x1f9/0x4c0
[<ffffffff93802242>] handle_mm_fault+0xad2/0x1190
[<ffffffff93dbf7d3>] __do_page_fault+0x213/0x510
[<ffffffff93dbfb05>] do_page_fault+0x35/0x90
[<ffffffff93dbb7b8>] page_fault+0x28/0x30
Code: 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 <8b> 17 0f b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84 
Complete report #715081
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffffa191eacd
Build id: 3.10.0-1160.88.1.el7.x86_64
0x1d
2
queued_spin_lock_slowpath
vmlinux 0xffffffffa1fac21a
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffa1fba734
Build id: 3.10.0-1160.88.1.el7.x86_64
0x24
4
do_huge_pmd_numa_page
vmlinux 0xffffffffa1a43d65
Build id: 3.10.0-1160.88.1.el7.x86_64
0x85
5
? do_numa_page
vmlinux 0xffffffffa19ff7fe
Build id: 3.10.0-1160.88.1.el7.x86_64
0x1be
6
handle_mm_fault
vmlinux 0xffffffffa1a021b2
Build id: 3.10.0-1160.88.1.el7.x86_64
0xad2
7
__do_page_fault
vmlinux 0xffffffffa1fbf7d3
Build id: 3.10.0-1160.88.1.el7.x86_64
0x213
8
do_page_fault
vmlinux 0xffffffffa1fbfb05
Build id: 3.10.0-1160.88.1.el7.x86_64
0x35
9
page_fault
vmlinux 0xffffffffa1fbb7b8
Build id: 3.10.0-1160.88.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [splunkd:9735]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_15003(E) falcon_lsm_pinned_14812(E) falcon_lsm_pinned_14713(E) binfmt_misc ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter xt_nat vmw_vsock_vmci_transport vsock iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc i2c_piix4 parport vmw_vmci auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea crc_t10dif sysfillrect crct10dif_generic sysimgblt fb_sys_fops ttm ahci ata_piix libahci drm libata nfit crct10dif_pclmul crct10dif_common crc32c_intel serio_raw
libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal]
CPU: 5 PID: 9735 Comm: splunkd Kdump: loaded Tainted: P            EL ------------   3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff97022b02c200 ti: ffff970212048000 task.ti: ffff970212048000
RIP: 0010:[<ffffffffa191eacd>]  [<ffffffffa191eacd>] native_queued_spin_lock_slowpath+0x1d/0x200
RSP: 0000:ffff97021204bd40  EFLAGS: 00000293
RAX: 0000000000000001 RBX: ffffffffa1cdc5d7 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000001 RDI: fffffb80101a4eb0
RBP: ffff97021204bd40 R08: ffff97062e4d6400 R09: 0000000000000000
R10: ffff97021204be20 R11: 00007f2af39ae590 R12: ffffffffa1bb1ab9
R13: ffff97021204bcb8 R14: ffff97034f47dec0 R15: ffff97021204be18
FS:  00007f2ad53fe700(0000) GS:ffff97022fb40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2aa555e5a8 CR3: 000000081f13e000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffa1fac21a>] queued_spin_lock_slowpath+0xb/0x13
[<ffffffffa1fba734>] _raw_spin_lock+0x24/0x30
[<ffffffffa1a43d65>] do_huge_pmd_numa_page+0x85/0x4c0
[<ffffffffa19ff7fe>] ? do_numa_page+0x1be/0x260
[<ffffffffa1a021b2>] handle_mm_fault+0xad2/0x1190
[<ffffffffa1fbf7d3>] __do_page_fault+0x213/0x510
[<ffffffffa1fbfb05>] do_page_fault+0x35/0x90
[<ffffffffa1fbb7b8>] page_fault+0x28/0x30
Code: 47 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 <8b> 17 85 d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 cc 
Complete report #657769
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff9571e97d
Build id: 3.10.0-1160.81.1.el7.x86_64
0x1d
2
queued_spin_lock_slowpath
vmlinux 0xffffffff95dac21a
Build id: 3.10.0-1160.81.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff95dba734
Build id: 3.10.0-1160.81.1.el7.x86_64
0x24
4
do_huge_pmd_numa_page
vmlinux 0xffffffff95843a05
Build id: 3.10.0-1160.81.1.el7.x86_64
0x85
5
handle_mm_fault
vmlinux 0xffffffff9580205a
Build id: 3.10.0-1160.81.1.el7.x86_64
0xaba
6
? fput
vmlinux 0xffffffff9585e0d3
Build id: 3.10.0-1160.81.1.el7.x86_64
0x13
7
__do_page_fault
vmlinux 0xffffffff95dbf7d3
Build id: 3.10.0-1160.81.1.el7.x86_64
0x213
8
do_page_fault
vmlinux 0xffffffff95dbfb05
Build id: 3.10.0-1160.81.1.el7.x86_64
0x35
9
page_fault
vmlinux 0xffffffff95dbb7b8
Build id: 3.10.0-1160.81.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 21s! [http-192.168.97:43923]
Modules linked in: tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon edac_mce_amd joydev pcspkr sg i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix scsi_transport_spi drm mptscsih libata serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 15 PID: 43923 Comm: http-192.168.97 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff923d2ba04200 ti: ffff923cb1f44000 task.ti: ffff923cb1f44000
RIP: 0010:[<ffffffff9571e97d>]  [<ffffffff9571e97d>] native_queued_spin_lock_slowpath+0x1d/0x200
RSP: 0000:ffff923cb1f47d40  EFLAGS: 00000293
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000001 RDI: fffff3dccccb7ef0
RBP: ffff923cb1f47d40 R08: ffff923e23a7a580 R09: 0000000000000000
R10: ffff923cb1f47e20 R11: 0000000040002798 R12: 0000000000000010
R13: ffffffff9571e97d R14: ffffffffffffff10 R15: fffff3dccccb7ef0
FS:  00007f2ebb2f4700(0000) GS:ffff923e24dc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000004f3b0d068 CR3: 00000002629bc000 CR4: 00000000000007e0
Call Trace:
[<ffffffff95dac21a>] queued_spin_lock_slowpath+0xb/0x13
[<ffffffff95dba734>] _raw_spin_lock+0x24/0x30
[<ffffffff95843a05>] do_huge_pmd_numa_page+0x85/0x4c0
[<ffffffff9580205a>] handle_mm_fault+0xaba/0xfb0
[<ffffffff9585e0d3>] ? fput+0x13/0x20
[<ffffffff95dbf7d3>] __do_page_fault+0x213/0x510
[<ffffffff95dbfb05>] do_page_fault+0x35/0x90
[<ffffffff95dbb7b8>] page_fault+0x28/0x30
Code: 47 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 <8b> 17 85 d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 cc 
Complete report #609423
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffffb9f1e97d
Build id: 3.10.0-1160.81.1.el7.x86_64
0x1d
2
queued_spin_lock_slowpath
vmlinux 0xffffffffba5ac21a
Build id: 3.10.0-1160.81.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffba5ba734
Build id: 3.10.0-1160.81.1.el7.x86_64
0x24
4
do_huge_pmd_numa_page
vmlinux 0xffffffffba043a05
Build id: 3.10.0-1160.81.1.el7.x86_64
0x85
5
? do_numa_page
vmlinux 0xffffffffb9fff6be
Build id: 3.10.0-1160.81.1.el7.x86_64
0x1be
6
handle_mm_fault
vmlinux 0xffffffffba00205a
Build id: 3.10.0-1160.81.1.el7.x86_64
0xaba
7
__do_page_fault
vmlinux 0xffffffffba5bf7d3
Build id: 3.10.0-1160.81.1.el7.x86_64
0x213
8
do_page_fault
vmlinux 0xffffffffba5bfb05
Build id: 3.10.0-1160.81.1.el7.x86_64
0x35
9
page_fault
vmlinux 0xffffffffba5bb7b8
Build id: 3.10.0-1160.81.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [java:4021]
Modules linked in: tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon edac_mce_amd joydev sg pcspkr parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic vmwgfx crct10dif_common ata_generic pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm mptsas ata_piix libahci drm scsi_transport_sas libata mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 12 PID: 4021 Comm: java Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff88a8e8576300 ti: ffff88a9179f4000 task.ti: ffff88a9179f4000
RIP: 0010:[<ffffffffb9f1e97d>]  [<ffffffffb9f1e97d>] native_queued_spin_lock_slowpath+0x1d/0x200
RSP: 0000:ffff88a9179f7d40  EFLAGS: 00000293
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000001 RDI: fffffa721651a030
RBP: ffff88a9179f7d40 R08: ffff88a92258d140 R09: 0000000000000000
R10: ffff88a9179f7e20 R11: 0000000000000206 R12: 0000000000000002
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000001a7a
FS:  00007f50bfb3b700(0000) GS:ffff88a924d00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000030fa0c8c0 CR3: 00000002ad1a0000 CR4: 00000000000007e0
Call Trace:
[<ffffffffba5ac21a>] queued_spin_lock_slowpath+0xb/0x13
[<ffffffffba5ba734>] _raw_spin_lock+0x24/0x30
[<ffffffffba043a05>] do_huge_pmd_numa_page+0x85/0x4c0
[<ffffffffb9fff6be>] ? do_numa_page+0x1be/0x260
[<ffffffffba00205a>] handle_mm_fault+0xaba/0xfb0
[<ffffffffba5bf7d3>] __do_page_fault+0x213/0x510
[<ffffffffba5bfb05>] do_page_fault+0x35/0x90
[<ffffffffba5bb7b8>] page_fault+0x28/0x30
Code: 47 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 <8b> 17 85 d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 cc 
Complete report #142941
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff9691776d
Build id: 3.10.0-1160.11.1.el7.x86_64
0x1d
2
queued_spin_lock_slowpath
vmlinux 0xffffffff96f7afe5
Build id: 3.10.0-1160.11.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff96f896f0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x20
4
do_huge_pmd_numa_page
vmlinux 0xffffffff96a366d5
Build id: 3.10.0-1160.11.1.el7.x86_64
0x85
5
? do_numa_page
vmlinux 0xffffffff969f357e
Build id: 3.10.0-1160.11.1.el7.x86_64
0x1be
6
handle_mm_fault
vmlinux 0xffffffff969f5eaa
Build id: 3.10.0-1160.11.1.el7.x86_64
0xaba
7
__do_page_fault
vmlinux 0xffffffff96f8e653
Build id: 3.10.0-1160.11.1.el7.x86_64
0x213
8
do_page_fault
vmlinux 0xffffffff96f8e975
Build id: 3.10.0-1160.11.1.el7.x86_64
0x35
9
? error_swapgs
vmlinux 0xffffffff96f8aac9
Build id: 3.10.0-1160.11.1.el7.x86_64
0xaa
10
page_fault
vmlinux 0xffffffff96f8a778
Build id: 3.10.0-1160.11.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [java:13903]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_conntrack tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack libcrc32c ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr parport_pc vmw_vmci sg parport i2c_piix4 sch_fq_codel ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix scsi_transport_spi mptscsih crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy fuse
CPU: 6 PID: 13903 Comm: java Kdump: loaded Tainted: G             L ------------   3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff91c5ebbb3180 ti: ffff91c5f37fc000 task.ti: ffff91c5f37fc000
RIP: 0010:[<ffffffff9691776d>]  [<ffffffff9691776d>] native_queued_spin_lock_slowpath+0x1d/0x200
RSP: 0000:ffff91c5f37ffd40  EFLAGS: 00000293
RAX: 0000000000000001 RBX: 00000000dc57315f RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffdf87c1ccf830
RBP: ffff91c5f37ffd40 R08: ffff91c5f97dd780 R09: 0000000000000001
R10: ffff91c5f37ffe20 R11: 0000000000004480 R12: ffffffff968ca3f9
R13: ffff91c5f37ffce0 R14: ffff9d46ccdda140 R15: ffff91c5f37ffdb8
FS:  00007fecbf7bd700(0000) GS:ffff91d60ec00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000065a208284 CR3: 0000001079f94000 CR4: 00000000000607e0
Call Trace:
[<ffffffff96f7afe5>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff96f896f0>] _raw_spin_lock+0x20/0x30
[<ffffffff96a366d5>] do_huge_pmd_numa_page+0x85/0x4c0
[<ffffffff969f357e>] ? do_numa_page+0x1be/0x250
[<ffffffff969f5eaa>] handle_mm_fault+0xaba/0xfb0
[<ffffffff96f8e653>] __do_page_fault+0x213/0x500
[<ffffffff96f8e975>] do_page_fault+0x35/0x90
[<ffffffff96f8aac9>] ? error_swapgs+0xaa/0xc0
[<ffffffff96f8a778>] page_fault+0x28/0x30
Code: 47 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 <8b> 17 85 d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 0f 
Complete report #121633
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffffb3115a98
Build id: 3.10.0-1062.1.1.el7.x86_64
0x128
2
queued_spin_lock_slowpath
vmlinux 0xffffffffb37733c5
Build id: 3.10.0-1062.1.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffffb3781b30
Build id: 3.10.0-1062.1.1.el7.x86_64
0x20
4
do_huge_pmd_numa_page
vmlinux 0xffffffffb3231fa5
Build id: 3.10.0-1062.1.1.el7.x86_64
0x85
5
handle_mm_fault
vmlinux 0xffffffffb31f24e7
Build id: 3.10.0-1062.1.1.el7.x86_64
0x877
6
__do_page_fault
vmlinux 0xffffffffb3787653
Build id: 3.10.0-1062.1.1.el7.x86_64
0x213
7
do_page_fault
vmlinux 0xffffffffb3787975
Build id: 3.10.0-1062.1.1.el7.x86_64
0x35
8
page_fault
vmlinux 0xffffffffb3783778
Build id: 3.10.0-1062.1.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#90 stuck for 22s! [llvmpipe-13:23135]
Modules linked in: binfmt_misc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache tcp_lp fuse xt_CHECKSUM tun devlink nf_conntrack_netlink xt_addrtype br_netfilter bridge stp llc overlay(T) nf_conntrack_netbios_ns nf_conntrack_broadcast nf_nat_tftp nf_conntrack_tftp ip6t_MASQUERADE nf_nat_masquerade_ipv6 ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dcdbas vfat fat amd64_edac_mod edac_mce_amd kvm_amd kvm
irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg k10temp i2c_piix4 ccp ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter pcc_cpufreq acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci crct10dif_pclmul crct10dif_common drm crc32c_intel libahci tg3 megaraid_sas libata ptp drm_panel_orientation_quirks pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 90 PID: 23135 Comm: llvmpipe-13 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1062.1.1.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R7425/08V001, BIOS 1.6.7 10/29/2018
task: ffffa0bd580fd230 ti: ffffa0bc4460c000 task.ti: ffffa0bc4460c000
RIP: 0010:[<ffffffffb3115a98>]  [<ffffffffb3115a98>] native_queued_spin_lock_slowpath+0x128/0x200
RSP: 0000:ffffa0bc4460fd60  EFLAGS: 00000246
RAX: 0000000000000000 RBX: 00000000ffffffff RCX: 0000000002d10000
RDX: ffffa0ad5fd9b840 RSI: 0000000003b10101 RDI: ffffe00841608230
RBP: ffffa0bc4460fd60 R08: ffffa09d5fcdb840 R09: 0000000000000000
R10: 0000000000000080 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: ffffe007841bc9f0 R15: 00e0065600080078
FS:  00007fd2c8d66700(0000) GS:ffffa09d5fcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000038458cc0 CR3: 0000003057bd2000 CR4: 00000000003407e0
Call Trace:
[<ffffffffb37733c5>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffffb3781b30>] _raw_spin_lock+0x20/0x30
[<ffffffffb3231fa5>] do_huge_pmd_numa_page+0x85/0x4b0
[<ffffffffb31f24e7>] handle_mm_fault+0x877/0x9b0
[<ffffffffb3787653>] __do_page_fault+0x213/0x500
[<ffffffffb3787975>] do_page_fault+0x35/0x90
[<ffffffffb3783778>] page_fault+0x28/0x30
Code: 98 83 e2 30 48 81 c2 40 b8 01 00 48 03 14 c5 e0 d7 d4 b3 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 85 c0 <74> f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 
Complete report #12734
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff94b15a5d
Build id: 3.10.0-1062.4.1.el7.x86_64
0x1d
2
queued_spin_lock_slowpath
vmlinux 0xffffffff951733cb
Build id: 3.10.0-1062.4.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff95181430
Build id: 3.10.0-1062.4.1.el7.x86_64
0x20
4
do_huge_pmd_numa_page
vmlinux 0xffffffff94c327a5
Build id: 3.10.0-1062.4.1.el7.x86_64
0x85
5
? schedule_hrtimeout_range_clock
vmlinux 0xffffffff9517dba6
Build id: 3.10.0-1062.4.1.el7.x86_64
0xc6
6
handle_mm_fault
vmlinux 0xffffffff94bf2ad7
Build id: 3.10.0-1062.4.1.el7.x86_64
0x877
7
__do_page_fault
vmlinux 0xffffffff95186653
Build id: 3.10.0-1062.4.1.el7.x86_64
0x213
8
do_page_fault
vmlinux 0xffffffff95186975
Build id: 3.10.0-1062.4.1.el7.x86_64
0x35
9
? error_swapgs
vmlinux 0xffffffff95182ac9
Build id: 3.10.0-1062.4.1.el7.x86_64
0xaa
10
page_fault
vmlinux 0xffffffff95182778
Build id: 3.10.0-1062.4.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [I/O dispatcher :12681]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ext4 mbcache jbd2 bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc ppdev pcspkr vmw_balloon joydev sg vmw_vmci i2c_piix4 pcc_cpufreq parport_pc parport xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_common ttm drm ahci ata_piix libahci crc32c_intel libata nfit serio_raw libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 10 PID: 12681 Comm: I/O dispatcher  Kdump: loaded Tainted: G             L ------------   3.10.0-1062.4.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8fb235388000 ti: ffff8fb45a384000 task.ti: ffff8fb45a384000
RIP: 0010:[<ffffffff94b15a5d>]  [<ffffffff94b15a5d>] native_queued_spin_lock_slowpath+0x1d/0x200
RSP: 0000:ffff8fb45a387d60  EFLAGS: 00000293
RAX: 0000000000000001 RBX: ffff8fb235553840 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffeb21f12ccbf0
RBP: ffff8fb45a387d60 R08: ffff8fb235553840 R09: 0000000000000000
R10: 0000000000000080 R11: 00000000000003e8 R12: ffff8fb47fd1ac00
R13: 0000000000000000 R14: ffff8fb235388698 R15: 0000000194ae8dbd
FS:  00007f921b4f4700(0000) GS:ffff8fb47fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000011cf33170 CR3: 0000001016b52000 CR4: 00000000000007e0
Call Trace:
[<ffffffff951733cb>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff95181430>] _raw_spin_lock+0x20/0x30
[<ffffffff94c327a5>] do_huge_pmd_numa_page+0x85/0x4b0
[<ffffffff9517dba6>] ? schedule_hrtimeout_range_clock+0xc6/0x150
[<ffffffff94bf2ad7>] handle_mm_fault+0x877/0x9b0
[<ffffffff95186653>] __do_page_fault+0x213/0x500
[<ffffffff95186975>] do_page_fault+0x35/0x90
[<ffffffff95182ac9>] ? error_swapgs+0xaa/0xc0
[<ffffffff95182778>] page_fault+0x28/0x30
Code: 47 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 <8b> 17 85 d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 0f 
Complete report #518787
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff8851598d
Build id: 3.10.0-1062.el7.x86_64
0x1d
2
queued_spin_lock_slowpath
vmlinux 0xffffffff88b73365
Build id: 3.10.0-1062.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff88b81ad0
Build id: 3.10.0-1062.el7.x86_64
0x20
4
do_numa_page
vmlinux 0xffffffff885eef1f
Build id: 3.10.0-1062.el7.x86_64
0x9f
5
handle_pte_fault
vmlinux 0xffffffff885ef389
Build id: 3.10.0-1062.el7.x86_64
0x2b9
6
? do_huge_pmd_numa_page
vmlinux 0xffffffff886321d9
Build id: 3.10.0-1062.el7.x86_64
0x2b9
7
handle_mm_fault
vmlinux 0xffffffff885f200d
Build id: 3.10.0-1062.el7.x86_64
0x39d
8
__do_page_fault
vmlinux 0xffffffff88b86633
Build id: 3.10.0-1062.el7.x86_64
0x213
9
do_page_fault
vmlinux 0xffffffff88b86955
Build id: 3.10.0-1062.el7.x86_64
0x35
10
page_fault
vmlinux 0xffffffff88b82768
Build id: 3.10.0-1062.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [containerd:24906]
Modules linked in: xt_recent veth nf_tables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache xt_CHECKSUM tun ebtable_filter ebtables ip6table_filter devlink ip6_tables nf_conntrack_netlink xt_statistic xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 xt_mark xt_addrtype xt_comment xt_conntrack iptable_filter iptable_nat nf_nat_ipv4 nf_nat iptable_mangle ip_set nfnetlink nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack vmw_vsock_vmci_transport vsock dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio sb_edac ppdev iosf_mbi vmw_balloon crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper parport_pc pcspkr joydev sg i2c_piix4 cryptd parport
vmw_vmci br_netfilter auth_rpcgss bridge stp sunrpc llc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper ata_piix syscopyarea sysfillrect sysimgblt fb_sys_fops ttm libata mptspi crct10dif_pclmul crct10dif_common scsi_transport_spi drm serio_raw crc32c_intel mptscsih vmxnet3 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 14 PID: 24906 Comm: containerd Kdump: loaded Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff95622b265230 ti: ffff9562067a4000 task.ti: ffff9562067a4000
RIP: 0010:[<ffffffff8851598d>]  [<ffffffff8851598d>] native_queued_spin_lock_slowpath+0x1d/0x200
RSP: 0000:ffff9562067a7ce8  EFLAGS: 00000293
RAX: 0000000000000001 RBX: ffffc6ed017a6740 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffc6ed1085c770
RBP: ffff9562067a7ce8 R08: ffff955e6171d1d0 R09: ffff955e4fa45058
R10: 0000000000000000 R11: 00003ffffffff000 R12: ffffffff895d9648
R13: ffffffff884e215b R14: ffff9562067a7c78 R15: fffffffe00000001
FS:  00007efc89ffb700(0000) GS:ffff95627fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00163a808 CR3: 0000000405e44000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff88b73365>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff88b81ad0>] _raw_spin_lock+0x20/0x30
[<ffffffff885eef1f>] do_numa_page+0x9f/0x250
[<ffffffff885ef389>] handle_pte_fault+0x2b9/0xe20
[<ffffffff886321d9>] ? do_huge_pmd_numa_page+0x2b9/0x4b0
[<ffffffff885f200d>] handle_mm_fault+0x39d/0x9b0
[<ffffffff88b86633>] __do_page_fault+0x213/0x500
[<ffffffff88b86955>] do_page_fault+0x35/0x90
[<ffffffff88b82768>] page_fault+0x28/0x30
Code: 47 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 <8b> 17 85 d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 0f