Info

Permalink
Function
smp_call_function_many
First occurrence
2020-09-26
Last occurrence
2026-01-15
State
NEW
Components
kernel
Quality
-28

Statistics

Operating system Count
CentOS 7 961
Architecture Count
x86_64 961
Related packages Count
kernel 961
 0:3.10.0-1062.12.1.el7 700
 0:3.10.0-1160.el7 175
 0:3.10.0-1160.25.1.el7 19
 0:3.10.0-1160.42.2.el7 15
 0:3.10.0-1160.6.1.el7 12
 0:3.10.0-1062.el7 9
 0:3.10.0-1160.71.1.el7 9
 0:3.10.0-1160.62.1.el7 5
 0:3.10.0-1160.59.1.el7 5
 0:3.10.0-1160.66.1.el7 4
 0:3.10.0-1062.9.1.el7 3
 0:3.10.0-1160.76.1.el7 2
 0:3.10.0-1160.11.1.el7 1
 0:3.10.0-1160.2.2.el7 1
 0:3.10.0-1160.53.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1109867
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbc516c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbc47e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbc47e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbc5edd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbc5ef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbc5ef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffbc5f2dba
Build id: 3.10.0-1160.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffbc5ed2ad
Build id: 3.10.0-1160.el7.x86_64
0x49d
9
? vm_mmap_pgoff
vmlinux 0xffffffffbc5de1bb
Build id: 3.10.0-1160.el7.x86_64
0xeb
10
? system_call_after_swapgs
vmlinux 0xffffffffbcb93ed5
Build id: 3.10.0-1160.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffffbcb93ec9
Build id: 3.10.0-1160.el7.x86_64
0x96
12
system_call_fastpath
vmlinux 0xffffffffbcb93f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [java:75317]
Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter uio_ivshmem(OE) uio gsch(OE) redirfs(OE) sunrpc ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr virtio_balloon sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console net_failover virtio_scsi virtio_blk failover cirrus drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci virtio_ring drm_panel_orientation_quirks virtio floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 6 PID: 75317 Comm: java Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
task: ffff9e9211bbc200 ti: ffff9e97e3750000 task.ti: ffff9e97e3750000
RIP: 0010:[<ffffffffbc516c0a>]  [<ffffffffbc516c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9e97e3753c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000bc47e6a0 RCX: ffff9e98dce204c8
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9e97e3753ca8 R08: ffff9e9232354400 R09: ffffffffbc785f99
R10: ffff9e98dcf9f160 R11: ffffd76ac5ffa800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9e97e3753c20 R15: 0000000000000001
FS:  00007f835dfe1700(0000) GS:ffff9e98dcf80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5efdeee000 CR3: 0000000172394000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffbc47e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbc47e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbc5edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbc5ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbc5ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffbc5f2dba>] zap_page_range+0xea/0x130
[<ffffffffbc5ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffffbc5de1bb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffbcb93ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffbcb93ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffbcb93f92>] system_call_fastpath+0x25/0x2a
Complete report #904440
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb791718a
Build id: 3.10.0-1160.76.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb787e258
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb787e2cb
Build id: 3.10.0-1160.76.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb79ee6a6
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb79f016f
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb79f0293
Build id: 3.10.0-1160.76.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb79f375a
Build id: 3.10.0-1160.76.1.el7.x86_64
0xea
8
? vm_mmap_pgoff
vmlinux 0xffffffffb79deddb
Build id: 3.10.0-1160.76.1.el7.x86_64
0xeb
9
SyS_madvise
vmlinux 0xffffffffb79edc4d
Build id: 3.10.0-1160.76.1.el7.x86_64
0x49d
10
? system_call_after_swapgs
vmlinux 0xffffffffb7f99ed5
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffffb7f99ec9
Build id: 3.10.0-1160.76.1.el7.x86_64
0x96
12
tracesys
vmlinux 0xffffffffb7f9a226
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [java:16097]
Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sisap(POE) sisevt(PE) sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon ppdev vmw_vmci sg joydev i2c_piix4 pcspkr parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm nfit ata_piix libnvdimm crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 7 PID: 16097 Comm: java Kdump: loaded Tainted: P           OEL ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9eb57722c200 ti: ffff9eb1550d8000 task.ti: ffff9eb1550d8000
RIP: 0010:[<ffffffffb791718a>]  [<ffffffffb791718a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9eb1550dbc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffb787ddd0 RCX: ffff9eb743a206f0
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff9eb1550dbca8 R08: ffff9eb2b0990800 R09: ffffffffb7b869f9
R10: ffff9eb743bdf160 R11: fffff0be8400aa00 R12: 0000000000000286
R13: ffff9eb1550dbc20 R14: ffff9eb2bfd5c400 R15: 0000000000000001
FS:  00007fe0acf6f700(0000) GS:ffff9eb743bc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd5e7d7d000 CR3: 00000004ecd88000 CR4: 00000000000407e0
Call Trace:
[<ffffffffb787e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb787e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb79ee6a6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb79f016f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb79f0293>] tlb_finish_mmu+0x23/0x30
[<ffffffffb79f375a>] zap_page_range+0xea/0x130
[<ffffffffb79deddb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb79edc4d>] SyS_madvise+0x49d/0xac0
[<ffffffffb7f99ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb7f99ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb7f9a226>] tracesys+0xa6/0xcc
Code: 48 63 35 be 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 55 b8 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #572091
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb8116c5a
Build id: 3.10.0-1160.6.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb807e868
Build id: 3.10.0-1160.6.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb807e8db
Build id: 3.10.0-1160.6.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb81edd46
Build id: 3.10.0-1160.6.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb81ef80f
Build id: 3.10.0-1160.6.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb81ef933
Build id: 3.10.0-1160.6.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb81f2dfa
Build id: 3.10.0-1160.6.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffb81ed2ed
Build id: 3.10.0-1160.6.1.el7.x86_64
0x49d
9
? vm_mmap_pgoff
vmlinux 0xffffffffb81de1fb
Build id: 3.10.0-1160.6.1.el7.x86_64
0xeb
10
? system_call_after_swapgs
vmlinux 0xffffffffb8794ed5
Build id: 3.10.0-1160.6.1.el7.x86_64
0xa2
11
? system_call_after_swapgs
vmlinux 0xffffffffb8794ec9
Build id: 3.10.0-1160.6.1.el7.x86_64
0x96
12
system_call_fastpath
vmlinux 0xffffffffb8794f92
Build id: 3.10.0-1160.6.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffb8794ed5
Build id: 3.10.0-1160.6.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [qtp366590980-19:7026]
Modules linked in: iptable_filter xt_multiport sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper sg ppdev ablk_helper cryptd vmw_balloon pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm ata_piix crc32c_intel mptspi scsi_transport_spi serio_raw e1000 libata mptscsih vmxnet3 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 7026 Comm: qtp366590980-19 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff8d40f9244200 ti: ffff8d41a59ec000 task.ti: ffff8d41a59ec000
RIP: 0010:[<ffffffffb8116c5a>]  [<ffffffffb8116c5a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8d41a59efc70  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 00080000b807e6a0 RCX: ffff8d42bfda0600
RDX: 0000000000000006 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8d41a59efca8 R08: ffff8d41fd361000 R09: ffffffffb8385fd9
R10: ffff8d42bfc5f160 R11: ffffede80286fa00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8d41a59efc20 R15: 0000000000000001
FS:  00007f4f673ad700(0000) GS:ffff8d42bfc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff9437b4000 CR3: 00000000b5940000 CR4: 00000000000607e0
Call Trace:
[<ffffffffb807e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb807e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb81edd46>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb81ef80f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb81ef933>] tlb_finish_mmu+0x23/0x30
[<ffffffffb81f2dfa>] zap_page_range+0xea/0x130
[<ffffffffb81ed2ed>] SyS_madvise+0x49d/0xac0
[<ffffffffb81de1fb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb8794ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffb8794ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb8794f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb8794ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 d5 b8 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #361372
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff87316a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8727e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8727e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff873ec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff873edb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff873edc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff873f098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff873eb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x49d
9
? vm_mmap_pgoff
vmlinux 0xffffffff873ddffb
Build id: 3.10.0-1062.12.1.el7.x86_64
0xeb
10
? system_call_after_swapgs
vmlinux 0xffffffff8798de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
11
? system_call_after_swapgs
vmlinux 0xffffffff8798de15
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa2
12
system_call_fastpath
vmlinux 0xffffffff8798dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff8798de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [java:149601]
Modules linked in: tcp_lp fuse kbase(OE) xt_NFQUEUE nfnetlink_log nfnetlink nfring(OE) vmsecmod(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc ppdev sg parport_pc pcspkr parport virtio_balloon joydev i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci virtio_ring serio_raw floppy virtio
drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vmsec_nfq]
CPU: 11 PID: 149601 Comm: java Kdump: loaded Tainted: GF          OEL ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9ea84a7fe2a0 ti: ffff9ea7f9780000 task.ti: ffff9ea7f9780000
RIP: 0010:[<ffffffff87316a7a>]  [<ffffffff87316a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9ea7f9783c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008727e2b0 RCX: ffff9eaf7ce20590
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9ea7f9783ca8 R08: ffff9ea0e95dd400 R09: ffffffff87581839
R10: ffff9eaf7d0df160 R11: ffffdd7de275ba00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9ea7f9783c20 R15: 0000000000000001
FS:  00007f2de1c72700(0000) GS:ffff9eaf7d0c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2de1c5e688 CR3: 000000070ab94000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8727e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8727e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff873ec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff873edb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff873edc83>] tlb_finish_mmu+0x23/0x30
[<ffffffff873f098a>] zap_page_range+0xea/0x130
[<ffffffff873eb69d>] SyS_madvise+0x49d/0xac0
[<ffffffff873ddffb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff8798de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff8798de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff8798dede>] system_call_fastpath+0x25/0x2a
[<ffffffff8798de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 f4 87 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #42711
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8c716a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8c67e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8c67e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8c7ec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8c7edb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8c7edc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff8c7f098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
8
? futex_wake
vmlinux 0xffffffff8c711ed0
Build id: 3.10.0-1062.12.1.el7.x86_64
0x90
9
SyS_madvise
vmlinux 0xffffffff8c7eb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x49d
10
? vm_mmap_pgoff
vmlinux 0xffffffff8c7ddffb
Build id: 3.10.0-1062.12.1.el7.x86_64
0xeb
11
? system_call_after_swapgs
vmlinux 0xffffffff8cd8de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
12
? system_call_after_swapgs
vmlinux 0xffffffff8cd8de15
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa2
13
system_call_fastpath
vmlinux 0xffffffff8cd8dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffff8cd8de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [Messages View G:31136]
Modules linked in: tcp_diag udp_diag inet_diag tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack fuse nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_set ip_set nfnetlink iptable_filter sunrpc snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore ppdev sg i2c_piix4 virtio_balloon pcspkr joydev parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common virtio_net virtio_blk ata_generic pata_acpi cirrus drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata virtio_pci floppy virtio_ring serio_raw virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 31136 Comm: Messages View G Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff9a13ce361070 ti: ffff9a13d6378000 task.ti: ffff9a13d6378000
RIP: 0010:[<ffffffff8c716a7a>]  [<ffffffff8c716a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9a13d637bc70  EFLAGS: 00000202
RAX: 0000000000000007 RBX: ffffffff8c67e2b0 RCX: ffff9a144b9e0450
RDX: 0000000000000007 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff9a13d637bca8 R08: ffff9a03e1a1f400 R09: ffffffff8c981839
R10: ffff9a144b8df160 R11: ffffdac41a89ac00 R12: ffff9a13d637bc40
R13: ffffffff8c98178f R14: ffff9a03ffd56000 R15: 0000000000000001
FS:  00007fdf410ff700(0000) GS:ffff9a144b8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000013eddf000 CR3: 00000008342cc000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8c67e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8c67e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8c7ec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8c7edb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8c7edc83>] tlb_finish_mmu+0x23/0x30
[<ffffffff8c7f098a>] zap_page_range+0xea/0x130
[<ffffffff8c711ed0>] ? futex_wake+0x90/0x180
[<ffffffff8c7eb69d>] SyS_madvise+0x49d/0xac0
[<ffffffff8c7ddffb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff8cd8de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff8cd8de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff8cd8dede>] system_call_fastpath+0x25/0x2a
[<ffffffff8cd8de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 34 8d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #16602
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9f716c0a
Build id: 3.10.0-1160.2.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9f67e868
Build id: 3.10.0-1160.2.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9f67e8db
Build id: 3.10.0-1160.2.2.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9f7edd06
Build id: 3.10.0-1160.2.2.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9f7ef7cf
Build id: 3.10.0-1160.2.2.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9f7ef8f3
Build id: 3.10.0-1160.2.2.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff9f7f2dba
Build id: 3.10.0-1160.2.2.el7.x86_64
0xea
8
? do_mmap
vmlinux 0xffffffff9f7fc9d8
Build id: 3.10.0-1160.2.2.el7.x86_64
0x378
9
SyS_madvise
vmlinux 0xffffffff9f7ed2ad
Build id: 3.10.0-1160.2.2.el7.x86_64
0x49d
10
? vm_mmap_pgoff
vmlinux 0xffffffff9f7de1bb
Build id: 3.10.0-1160.2.2.el7.x86_64
0xeb
11
? system_call_after_swapgs
vmlinux 0xffffffff9fd93ed5
Build id: 3.10.0-1160.2.2.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffff9fd93ec9
Build id: 3.10.0-1160.2.2.el7.x86_64
0x96
13
system_call_fastpath
vmlinux 0xffffffff9fd93f92
Build id: 3.10.0-1160.2.2.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffff9fd93ed5
Build id: 3.10.0-1160.2.2.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 25s! [qtp366590980-12:26965]
Modules linked in: veeamsnap(OE) sunrpc dm_mirror dm_region_hash dm_log dm_mod pcspkr joydev i2c_piix4 sg virtio_balloon ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi virtio_net net_failover virtio_console failover virtio_scsi qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy
CPU: 3 PID: 26965 Comm: qtp366590980-12 Tainted: G           OEL ------------   3.10.0-1160.2.2.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-48-gd9c812dda519-prebuilt.qemu.org 04/01/2014
task: ffff95b4bdf60000 ti: ffff95b50cba4000 task.ti: ffff95b50cba4000
RIP: 0010:[<ffffffff9f716c0a>]  [<ffffffff9f716c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff95b50cba7c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009f67e6a0 RCX: ffff95b57fc20450
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff95b50cba7ca8 R08: ffff95b4bcb7e000 R09: ffffffff9f985f99
R10: ffff95b57fd9f160 R11: fffff449c8d53700 R12: 0000000000000282
R13: 0000000000000282 R14: ffff95b50cba7c20 R15: 0000000000000001
FS:  00007fdde4313700(0000) GS:ffff95b57fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdde42fea48 CR3: 00000000a7f6a000 CR4: 00000000000006e0
Call Trace:
[<ffffffff9f67e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9f67e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9f7edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9f7ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9f7ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff9f7f2dba>] zap_page_range+0xea/0x130
[<ffffffff9f7fc9d8>] ? do_mmap+0x378/0x530
[<ffffffff9f7ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffff9f7de1bb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff9fd93ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff9fd93ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff9fd93f92>] system_call_fastpath+0x25/0x2a
[<ffffffff9fd93ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 35 a0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #12271
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb7b16a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb7a7e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb7a7e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb7bec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb7bedb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb7bedc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffb7bf098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffffb7beb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x49d
9
? vm_mmap_pgoff
vmlinux 0xffffffffb7bddffb
Build id: 3.10.0-1062.12.1.el7.x86_64
0xeb
10
? system_call_after_swapgs
vmlinux 0xffffffffb818de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
11
? system_call_after_swapgs
vmlinux 0xffffffffb818de15
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa2
12
system_call_fastpath
vmlinux 0xffffffffb818dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffb818de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [java:42012]
Modules linked in: xt_NFQUEUE nfnetlink_log nfnetlink nfring(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc ppdev sg virtio_balloon joydev i2c_piix4 pcspkr parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata virtio_pci serio_raw virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 42012 Comm: java Kdump: loaded Tainted: G           OE  ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff98587fe162a0 ti: ffff98589864c000 task.ti: ffff98589864c000
RIP: 0010:[<ffffffffb7b16a7a>]  [<ffffffffb7b16a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff98589864fc70  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00080000b7a7e2b0 RCX: ffff985e5cf204f0
RDX: 0000000000000004 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff98589864fca8 R08: ffff9857b44acc00 R09: ffffffffb7d81839
R10: ffff985e5cfdf160 R11: ffffd77a46be1400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff98589864fc20 R15: 0000000000000001
FS:  00007f4354080700(0000) GS:ffff985e5cfc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f435406c798 CR3: 00000007edbc4000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb7a7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb7a7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb7bec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb7bedb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb7bedc83>] tlb_finish_mmu+0x23/0x30
[<ffffffffb7bf098a>] zap_page_range+0xea/0x130
[<ffffffffb7beb69d>] SyS_madvise+0x49d/0xac0
[<ffffffffb7bddffb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb818de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffb818de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffb818dede>] system_call_fastpath+0x25/0x2a
[<ffffffffb818de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 74 b8 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #5745
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9e116a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9e07e478
Build id: 3.10.0-1062.9.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9e07e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9e1ec026
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9e1edaef
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9e1edc13
Build id: 3.10.0-1062.9.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffff9e1f091a
Build id: 3.10.0-1062.9.1.el7.x86_64
0xea
8
SyS_madvise
vmlinux 0xffffffff9e1eb62d
Build id: 3.10.0-1062.9.1.el7.x86_64
0x49d
9
? vm_mmap_pgoff
vmlinux 0xffffffff9e1ddfdb
Build id: 3.10.0-1062.9.1.el7.x86_64
0xeb
10
? system_call_after_swapgs
vmlinux 0xffffffff9e78de21
Build id: 3.10.0-1062.9.1.el7.x86_64
0xae
11
? system_call_after_swapgs
vmlinux 0xffffffff9e78de15
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa2
12
system_call_fastpath
vmlinux 0xffffffff9e78dede
Build id: 3.10.0-1062.9.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffff9e78de21
Build id: 3.10.0-1062.9.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [qtp1225197672-2:1500]
Modules linked in: nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod ppdev aesni_intel lrw gf128mul sg glue_helper ablk_helper cryptd pcspkr joydev i2c_piix4 virtio_balloon parport_pc parport ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_common sr_mod cdrom virtio_net virtio_console virtio_scsi
ata_generic pata_acpi bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm ata_piix libata drm_panel_orientation_quirks virtio_pci serio_raw virtio_ring virtio floppy
CPU: 2 PID: 1500 Comm: qtp1225197672-2 Tainted: G             L ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014
task: ffff9706f161b150 ti: ffff970a9dcb0000 task.ti: ffff970a9dcb0000
RIP: 0010:[<ffffffff9e116a6a>]  [<ffffffff9e116a6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff970a9dcb3c70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009e07e2b0 RCX: ffff97055e020428
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff970a9dcb3ca8 R08: ffff97057a80b800 R09: ffffffff9e3817e9
R10: ffff97057fc1f160 R11: fffffd88cabc0d00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff970a9dcb3c20 R15: 0000000000000001
FS:  00007fb77ad91700(0000) GS:ffff97057fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000f1af2920 CR3: 00000002d669a000 CR4: 00000000000006e0
Call Trace:
[<ffffffff9e07e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9e07e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9e1ec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9e1edaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9e1edc13>] tlb_finish_mmu+0x23/0x30
[<ffffffff9e1f091a>] zap_page_range+0xea/0x130
[<ffffffff9e1eb62d>] SyS_madvise+0x49d/0xac0
[<ffffffff9e1ddfdb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff9e78de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff9e78de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffff9e78dede>] system_call_fastpath+0x25/0x2a
[<ffffffff9e78de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 fe 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 d4 9e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #2387
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbdb16a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbda7e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbda7e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbdbec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbdbedb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbdbedc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffbdbf098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
8
? do_mmap
vmlinux 0xffffffffbdbfad78
Build id: 3.10.0-1062.12.1.el7.x86_64
0x378
9
SyS_madvise
vmlinux 0xffffffffbdbeb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x49d
10
? vm_mmap_pgoff
vmlinux 0xffffffffbdbddffb
Build id: 3.10.0-1062.12.1.el7.x86_64
0xeb
11
? system_call_after_swapgs
vmlinux 0xffffffffbe18de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
12
? system_call_after_swapgs
vmlinux 0xffffffffbe18de15
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa2
13
system_call_fastpath
vmlinux 0xffffffffbe18dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffbe18de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [java:18290]
Modules linked in: arc4 md4 nls_utf8 cifs ccm dns_resolver binfmt_misc ext4 mbcache jbd2 ppdev sg i2c_piix4 parport_pc virtio_balloon joydev parport pcspkr ip_tables xfs libcrc32c sr_mod sd_mod crc_t10dif cdrom crct10dif_generic crct10dif_common ata_generic pata_acpi virtio_net virtio_console bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 18290 Comm: java Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.org 04/01/2014
task: ffff9f41f5fda0e0 ti: ffff9f45e263c000 task.ti: ffff9f45e263c000
RIP: 0010:[<ffffffffbdb16a7a>]  [<ffffffffbdb16a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9f45e263fc70  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000c0000bda7e2b0 RCX: ffff9f45ffda0400
RDX: 0000000000000003 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff9f45e263fca8 R08: ffff9f433a30cc00 R09: ffffffffbdd81839
R10: ffff9f45ffc9f160 R11: ffffead850058300 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9f45e263fc20 R15: 0000000000000001
FS:  00007f6a45da6700(0000) GS:ffff9f45ffc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcc7f5ad000 CR3: 000000041e64e000 CR4: 00000000000006e0
Call Trace:
[<ffffffffbda7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbda7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbdbec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbdbedb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbdbedc83>] tlb_finish_mmu+0x23/0x30
[<ffffffffbdbf098a>] zap_page_range+0xea/0x130
[<ffffffffbdbfad78>] ? do_mmap+0x378/0x530
[<ffffffffbdbeb69d>] SyS_madvise+0x49d/0xac0
[<ffffffffbdbddffb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffbe18de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffbe18de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffbe18dede>] system_call_fastpath+0x25/0x2a
[<ffffffffbe18de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 74 be f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #534898
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa5f170de
Build id: 3.10.0-1160.53.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffa5e7e258
Build id: 3.10.0-1160.53.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa5e7e2cb
Build id: 3.10.0-1160.53.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa5fee676
Build id: 3.10.0-1160.53.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffa5ff013f
Build id: 3.10.0-1160.53.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffa5ff0263
Build id: 3.10.0-1160.53.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffa5ff372a
Build id: 3.10.0-1160.53.1.el7.x86_64
0xea
8
? handle_mm_fault
vmlinux 0xffffffffa5ff66a2
Build id: 3.10.0-1160.53.1.el7.x86_64
0xb22
9
? do_mmap
vmlinux 0xffffffffa5ffd36c
Build id: 3.10.0-1160.53.1.el7.x86_64
0x39c
10
SyS_madvise
vmlinux 0xffffffffa5fedc1d
Build id: 3.10.0-1160.53.1.el7.x86_64
0x49d
11
? system_call_after_swapgs
vmlinux 0xffffffffa6595ed5
Build id: 3.10.0-1160.53.1.el7.x86_64
0xa2
12
? system_call_after_swapgs
vmlinux 0xffffffffa6595ec9
Build id: 3.10.0-1160.53.1.el7.x86_64
0x96
13
system_call_fastpath
vmlinux 0xffffffffa6595f92
Build id: 3.10.0-1160.53.1.el7.x86_64
0x25
14
? system_call_after_swapgs
vmlinux 0xffffffffa6595ed5
Build id: 3.10.0-1160.53.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 45s! [qtp2076287037-4:7949]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr vmw_balloon joydev sg parport_pc parport vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic
pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix nfit crct10dif_pclmul libata crct10dif_common libnvdimm mptspi e1000 crc32c_intel scsi_transport_spi mptscsih serio_raw mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 7949 Comm: qtp2076287037-4 Not tainted 3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8a5fb08f5280 ti: ffff8a60712fc000 task.ti: ffff8a60712fc000
RIP: 0010:[<ffffffffa5f170de>]  [<ffffffffa5f170de>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8a60712ffc70  EFLAGS: 00000202
RAX: 0000000000000006 RBX: ffffffffa5e7ddd0 RCX: ffff8a622f59fc10
RDX: 0000000000000006 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8a60712ffca8 R08: ffff8a60bfd55000 R09: ffffffffa6186bf9
R10: ffff8a622f41f160 R11: ffffef67c8453e00 R12: ffff8a60712ffc40
R13: ffffffffa6186b4f R14: ffff8a60bfd55c00 R15: 0000000000000001
FS:  00007fb954882700(0000) GS:ffff8a622f400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb9786d52b0 CR3: 00000000a74f0000 CR4: 00000000000407f0
Call Trace:
[<ffffffffa5e7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa5e7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa5fee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa5ff013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa5ff0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffa5ff372a>] zap_page_range+0xea/0x130
[<ffffffffa5ff66a2>] ? handle_mm_fault+0xb22/0xfb0
[<ffffffffa5ffd36c>] ? do_mmap+0x39c/0x590
[<ffffffffa5fedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffa6595ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffa6595ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffa6595f92>] system_call_fastpath+0x25/0x2a
[<ffffffffa6595ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 b5 a6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #97751
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbbf16a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbbe7e478
Build id: 3.10.0-1062.9.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbbe7e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbbfec026
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffbbfedaef
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffbbfedc13
Build id: 3.10.0-1062.9.1.el7.x86_64
0x23
7
zap_page_range
vmlinux 0xffffffffbbff091a
Build id: 3.10.0-1062.9.1.el7.x86_64
0xea
8
? futex_wake
vmlinux 0xffffffffbbf11ec0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x90
9
SyS_madvise
vmlinux 0xffffffffbbfeb62d
Build id: 3.10.0-1062.9.1.el7.x86_64
0x49d
10
? system_call_after_swapgs
vmlinux 0xffffffffbc58de21
Build id: 3.10.0-1062.9.1.el7.x86_64
0xae
11
? system_call_after_swapgs
vmlinux 0xffffffffbc58de15
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa2
12
system_call_fastpath
vmlinux 0xffffffffbc58dede
Build id: 3.10.0-1062.9.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffbc58de21
Build id: 3.10.0-1062.9.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 33s! [mysqld:10169]
Modules linked in: tun ppdev vmw_balloon sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi scsi_transport_spi libata crct10dif_pclmul crct10dif_common mptscsih crc32c_intel serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 10169 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9f395f7d0000 ti: ffff9f39bb424000 task.ti: ffff9f39bb424000
RIP: 0010:[<ffffffffbbf16a6a>]  [<ffffffffbbf16a6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9f39bb427c70  EFLAGS: 00000202
RAX: 000000000000000b RBX: ffffffffbbe7e2b0 RCX: ffff9f43d54dfc10
RDX: 000000000000000b RSI: 000000000000000c RDI: 0000000000000000
RBP: ffff9f39bb427ca8 R08: ffff9f3abfd58000 R09: ffffffffbc1817e9
R10: ffff9f43d521f160 R11: ffffe605c22b0c00 R12: 0000000000000286
R13: ffff9f39bb427c20 R14: ffff9f43d5200000 R15: ffffffffbbe2b621
FS:  00007f5ba7cc6700(0000) GS:ffff9f43d5200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5d00285030 CR3: 0000000a92e9c000 CR4: 00000000000407f0
Call Trace:
[<ffffffffbbe7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbbe7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbbfec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbbfedaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbbfedc13>] tlb_finish_mmu+0x23/0x30
[<ffffffffbbff091a>] zap_page_range+0xea/0x130
[<ffffffffbbf11ec0>] ? futex_wake+0x90/0x180
[<ffffffffbbfeb62d>] SyS_madvise+0x49d/0xac0
[<ffffffffbc58de21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffbc58de15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffbc58dede>] system_call_fastpath+0x25/0x2a
[<ffffffffbc58de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 fe 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 b4 bc f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89