Info

Permalink
Function
generic_exec_single
First occurrence
2020-09-22
Last occurrence
2026-03-01
State
NEW
Components
kernel
Quality
-37

Statistics

Operating system Count
CentOS 7 579
Architecture Count
x86_64 579
Related packages Count
kernel 579
 0:3.10.0-514.16.1.el7 353
 0:3.10.0-327.el7 103
 0:3.10.0-514.el7 53
 0:3.10.0-229.14.1.el7 17
 0:3.10.0-327.36.2.el7 7
 0:3.10.0-327.13.1.el7 6
 0:3.10.0-514.10.2.el7 5
 0:3.10.0-1160.62.1.el7 5
 0:3.10.0-229.20.1.el7 5
 0:3.10.0-327.22.2.el7 4
 0:3.10.0-1160.66.1.el7 4
 0:3.10.0-514.21.1.el7 3
 0:3.10.0-514.2.2.el7 2
 0:3.10.0-1160.15.2.el7 2
 0:3.10.0-514.6.2.el7 1
 0:3.10.0-1062.el7 1
 0:3.10.0-229.el7 1
 0:3.10.0-1160.36.2.el7 1
 0:3.10.0-1062.12.1.el7 1
 0:3.10.0-1062.9.1.el7 1
 0:3.10.0-1160.71.1.el7 1
 0:3.10.0-957.el7 1
 0:3.10.0-693.17.1.el7 1
 0:3.10.0-862.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1030607
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9930779a
Build id: 3.10.0-862.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff99276e50
Build id: 3.10.0-862.el7.x86_64 		0x110
3
? __get_free_pages
vmlinux 0xffffffff99398b8e
Build id: 3.10.0-862.el7.x86_64 		0xe
4
? leave_mm
vmlinux 0xffffffff99276e50
Build id: 3.10.0-862.el7.x86_64 		0x110
5
? leave_mm
vmlinux 0xffffffff99276e50
Build id: 3.10.0-862.el7.x86_64 		0x110
6
smp_call_function_single
vmlinux 0xffffffff993078af
Build id: 3.10.0-862.el7.x86_64 		0x5f
7
? cpumask_next_and
vmlinux 0xffffffff9954a275
Build id: 3.10.0-862.el7.x86_64 		0x35
8
smp_call_function_many
vmlinux 0xffffffff99307e5b
Build id: 3.10.0-862.el7.x86_64 		0x22b
9
native_flush_tlb_others
vmlinux 0xffffffff99277018
Build id: 3.10.0-862.el7.x86_64 		0xb8
10
flush_tlb_mm_range
vmlinux 0xffffffff99277088
Build id: 3.10.0-862.el7.x86_64 		0x68
11
tlb_flush_mmu.part.75
vmlinux 0xffffffff993c0497
Build id: 3.10.0-862.el7.x86_64 		0x37
12
tlb_finish_mmu
vmlinux 0xffffffff993c1c75
Build id: 3.10.0-862.el7.x86_64 		0x55
13
unmap_region
vmlinux 0xffffffff993ca624
Build id: 3.10.0-862.el7.x86_64 		0xf4
14
do_munmap
vmlinux 0xffffffff993ccbf5
Build id: 3.10.0-862.el7.x86_64 		0x295
15
SyS_brk
vmlinux 0xffffffff993cd488
Build id: 3.10.0-862.el7.x86_64 		0x178
16
system_call_fastpath
vmlinux 0xffffffff9991f7d5
Build id: 3.10.0-862.el7.x86_64 		0x1c
17
? system_call_after_swapgs
vmlinux 0xffffffff9991f721
Build id: 3.10.0-862.el7.x86_64 		0xae 
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [systemd-udevd:456]
Modules linked in: xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw vmxnet3 i2c_core vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 456 Comm: systemd-udevd Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff97019a779fa0 ti: ffff970196bec000 task.ti: ffff970196bec000
RIP: 0010:[<ffffffff9930779a>]  [<ffffffff9930779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff970196befc40  EFLAGS: 00000202
RAX: 00000000000008fd RBX: 00000000000000f5 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff970196befc90 R08: 0000000000000001 R09: ffffffff993d34c8
R10: ffff9701bfd1ba40 R11: ffffe37ae065dd00 R12: ffff9701bffbf000
R13: 0000014100000000 R14: ffff9701bffbf008 R15: ffff97019a779fa0
FS:  00007fd1966aa8c0(0000) GS:ffff9701bfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fd1952fe520 CR3: 0000000819778000 CR4: 00000000000407e0
Call Trace:
[<ffffffff99276e50>] ? leave_mm+0x110/0x110
[<ffffffff99398b8e>] ? __get_free_pages+0xe/0x40
[<ffffffff99276e50>] ? leave_mm+0x110/0x110
[<ffffffff99276e50>] ? leave_mm+0x110/0x110
[<ffffffff993078af>] smp_call_function_single+0x5f/0xa0
[<ffffffff9954a275>] ? cpumask_next_and+0x35/0x50
[<ffffffff99307e5b>] smp_call_function_many+0x22b/0x270
[<ffffffff99277018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff99277088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff993c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff993c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff993ca624>] unmap_region+0xf4/0x140
[<ffffffff993ccbf5>] do_munmap+0x295/0x470
[<ffffffff993cd488>] SyS_brk+0x178/0x200
[<ffffffff9991f7d5>] system_call_fastpath+0x1c/0x21
[<ffffffff9991f721>] ? system_call_after_swapgs+0xae/0x146
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 f3 99 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #1003714
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810fb21a
Build id: 3.10.0-693.17.1.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8106f340
Build id: 3.10.0-693.17.1.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffff8106f340
Build id: 3.10.0-693.17.1.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffff8106f340
Build id: 3.10.0-693.17.1.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffff810fb31f
Build id: 3.10.0-693.17.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81324bd5
Build id: 3.10.0-693.17.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810fb8b3
Build id: 3.10.0-693.17.1.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106f508
Build id: 3.10.0-693.17.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106f5d8
Build id: 3.10.0-693.17.1.el7.x86_64 		0x68
10
tlb_flush_mmu.part.63
vmlinux 0xffffffff811af0f3
Build id: 3.10.0-693.17.1.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811b0415
Build id: 3.10.0-693.17.1.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff811b74e4
Build id: 3.10.0-693.17.1.el7.x86_64 		0xf4
13
do_munmap
vmlinux 0xffffffff811b9aa2
Build id: 3.10.0-693.17.1.el7.x86_64 		0x292
14
SyS_brk
vmlinux 0xffffffff811ba328
Build id: 3.10.0-693.17.1.el7.x86_64 		0x178
15
system_call_fastpath
vmlinux 0xffffffff816b89fd
Build id: 3.10.0-693.17.1.el7.x86_64 		0x16
16
? system_call_after_swapgs
vmlinux 0xffffffff816b889d
Build id: 3.10.0-693.17.1.el7.x86_64 		0xca 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [pbs_mom:1349]
Modules linked in: 8021q garp mrp stp llc vmw_vsock_vmci_transport vsock ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ppdev pcspkr edac_mce_amd vmw_balloon joydev edac_core sg parport_pc i2c_piix4 parport shpchp vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm vmxnet3 ata_piix mptspi libata scsi_transport_spi mptscsih mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 1349 Comm: pbs_mom Not tainted 3.10.0-693.17.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8800badd3f40 ti: ffff880228718000 task.ti: ffff880228718000
RIP: 0010:[<ffffffff810fb21a>]  [<ffffffff810fb21a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88022871bc50  EFLAGS: 00000202
RAX: 00000000000008fd RBX: ffff88023ffd8000 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000282
RBP: ffff88022871bc98 R08: 0000000000000001 R09: ffffffff811bf808
R10: ffff88023fc1b900 R11: ffffea00082abd80 R12: ffffffff816b9bff
R13: ffffffff816b9c06 R14: ffffffff816b9c0d R15: ffffffff816b9c14
FS:  00007f032750a880(0000) GS:ffff88023fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0324b571a0 CR3: 0000000228726000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8106f340>] ? leave_mm+0x110/0x110
[<ffffffff8106f340>] ? leave_mm+0x110/0x110
[<ffffffff8106f340>] ? leave_mm+0x110/0x110
[<ffffffff810fb31f>] smp_call_function_single+0x5f/0xa0
[<ffffffff81324bd5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810fb8b3>] smp_call_function_many+0x223/0x260
[<ffffffff8106f508>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106f5d8>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811af0f3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811b0415>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b74e4>] unmap_region+0xf4/0x140
[<ffffffff811b9aa2>] do_munmap+0x292/0x470
[<ffffffff811ba328>] SyS_brk+0x178/0x200
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
[<ffffffff816b889d>] ? system_call_after_swapgs+0xca/0x214
Code: 40 95 01 00 48 89 de 48 03 14 c5 20 8e b1 81 48 89 df e8 5a e0 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #534745
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa7d1109a
Build id: 3.10.0-957.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffffa7c7a3d0
Build id: 3.10.0-957.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffffa7c7a3d0
Build id: 3.10.0-957.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffffa7c7a3d0
Build id: 3.10.0-957.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffffa7d111af
Build id: 3.10.0-957.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa7f75465
Build id: 3.10.0-957.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffa7d1175b
Build id: 3.10.0-957.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa7c7a598
Build id: 3.10.0-957.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa7c7a608
Build id: 3.10.0-957.el7.x86_64 		0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffa7de4077
Build id: 3.10.0-957.el7.x86_64 		0x37
11
tlb_finish_mmu
vmlinux 0xffffffffa7de5975
Build id: 3.10.0-957.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffffa7def014
Build id: 3.10.0-957.el7.x86_64 		0xf4
13
? __vma_rb_erase
vmlinux 0xffffffffa7def5d1
Build id: 3.10.0-957.el7.x86_64 		0x121
14
do_munmap
vmlinux 0xffffffffa7df1605
Build id: 3.10.0-957.el7.x86_64 		0x2a5
15
SyS_brk
vmlinux 0xffffffffa7df1e98
Build id: 3.10.0-957.el7.x86_64 		0x178
16
system_call_fastpath
vmlinux 0xffffffffa8374ddb
Build id: 3.10.0-957.el7.x86_64 		0x22 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [systemd-udevd:220]
Modules linked in: xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_pclmul crct10dif_common crc32c_intel serio_raw nfit libnvdimm e1000 ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks
CPU: 3 PID: 220 Comm: systemd-udevd Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8e0f35b20000 ti: ffff8e0f35b28000 task.ti: ffff8e0f35b28000
RIP: 0010:[<ffffffffa7d1109a>]  [<ffffffffa7d1109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8e0f35b2bc40  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffffffffa8412d88 RCX: 0000000000000001
RDX: ffffffffa8412d90 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8e0f35b2bc90 R08: ffffffffa8412d88 R09: ffffffffa7df8298
R10: ffff8e0f842df0a0 R11: ffffcb7f80d64dc0 R12: ffff8e0f842df0a0
R13: ffffcb7f80d64dc0 R14: ffffffffa8431d80 R15: ffff8e0f35b2bc20
FS:  00007f7973fbc8c0(0000) GS:ffff8e0f842c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7971c64484 CR3: 00000000359e4000 CR4: 00000000003607e0
Call Trace:
 [<ffffffffa7c7a3d0>] ? leave_mm+0x110/0x110
 [<ffffffffa7c7a3d0>] ? leave_mm+0x110/0x110
 [<ffffffffa7c7a3d0>] ? leave_mm+0x110/0x110
 [<ffffffffa7d111af>] smp_call_function_single+0x5f/0xa0
 [<ffffffffa7f75465>] ? cpumask_next_and+0x35/0x50
 [<ffffffffa7d1175b>] smp_call_function_many+0x22b/0x270
 [<ffffffffa7c7a598>] native_flush_tlb_others+0xb8/0xc0
 [<ffffffffa7c7a608>] flush_tlb_mm_range+0x68/0x140
 [<ffffffffa7de4077>] tlb_flush_mmu.part.76+0x37/0xe0
 [<ffffffffa7de5975>] tlb_finish_mmu+0x55/0x60
 [<ffffffffa7def014>] unmap_region+0xf4/0x140
 [<ffffffffa7def5d1>] ? __vma_rb_erase+0x121/0x220
 [<ffffffffa7df1605>] do_munmap+0x2a5/0x480
 [<ffffffffa7df1e98>] SyS_brk+0x178/0x200
 [<ffffffffa8374ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 94 a8 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #259117
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff90b163c6
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x106
2
? leave_mm
vmlinux 0xffffffff90a7e2b0
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x110
3
? leave_mm
vmlinux 0xffffffff90a7e2b0
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x110
4
? leave_mm
vmlinux 0xffffffff90a7e2b0
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x110
5
smp_call_function_single
vmlinux 0xffffffff90b164df
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff90d81635
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff90b16a8b
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff90a7e478
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff90a7e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff90bec026
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff90bedaef
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff90bedc13
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x23
13
unmap_region
vmlinux 0xffffffff90bf7404
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xf4
14
? __vma_rb_erase
vmlinux 0xffffffff90bf79c1
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x121
15
do_munmap
vmlinux 0xffffffff90bf9a05
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x2a5
16
SyS_brk
vmlinux 0xffffffff90bfa298
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x178
17
system_call_fastpath
vmlinux 0xffffffff9118dede
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [ossec-agentd:3245]
Modules linked in: uinput binfmt_misc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ppdev sg iosf_mbi crc32_pclmul virtio_rng ghash_clmulni_intel parport_pc i2c_piix4 parport aesni_intel pcspkr joydev nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables
xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic pata_acpi virtio_net virtio_console virtio_scsi qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul virtio_pci virtio_ring floppy serio_raw virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log authenc cmac zlib wp512 twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common tea sha512_ssse3 sha512_generic serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic seed salsa20_generic salsa20_x86_64 rmd320 rmd256 rmd160 rmd128 michael_mic md4 khazad ghash_generic gcm fcrypt drbg dm_crypt dm_mod des_generic deflate cts crypto_null crct10dif_generic crct10dif_common crc32c_intel ccm cast6_avx_x86_64 cast6_generic cast_common camellia_generic
camellia_aesni_avx2 camellia_aesni_avx_x86_64 ablk_helper cryptd camellia_x86_64 xts lrw gf128mul glue_helper blowfish_generic blowfish_x86_64 blowfish_common arc4 ansi_cprng [last unloaded: tcrypt]
CPU: 3 PID: 3245 Comm: ossec-agentd Kdump: loaded Tainted: G             L ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: oVirt oVirt Node, BIOS 1.11.0-2.el7 04/01/2014
task: ffffa17faf6e8000 ti: ffffa17fa8f6c000 task.ti: ffffa17fa8f6c000
RIP: 0010:[<ffffffff90b163c6>]  [<ffffffff90b163c6>] generic_exec_single+0x106/0x1c0
RSP: 0018:ffffa17fa8f6fc40  EFLAGS: 00000202
RAX: 0000000000000010 RBX: ffffa17fa8f6fc10 RCX: 0000000000000030
RDX: 000000000000ffff RSI: 0000000000000010 RDI: 0000000000000282
RBP: ffffa17fa8f6fc88 R08: ffffffff912155c0 R09: ffffffff90c006a8
R10: ffffa17fbd0df0a0 R11: ffffee1c9e22d240 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000002 R15: ffffa1817fe1e000
FS:  00007fda2aac5800(0000) GS:ffffa17fbd0c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001330b80 CR3: 0000006eeaafa000 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff90a7e2b0>] ? leave_mm+0x110/0x110
[<ffffffff90a7e2b0>] ? leave_mm+0x110/0x110
[<ffffffff90a7e2b0>] ? leave_mm+0x110/0x110
[<ffffffff90b164df>] smp_call_function_single+0x5f/0xa0
[<ffffffff90d81635>] ? cpumask_next_and+0x35/0x50
[<ffffffff90b16a8b>] smp_call_function_many+0x22b/0x270
[<ffffffff90a7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff90a7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff90bec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff90bedaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff90bedc13>] tlb_finish_mmu+0x23/0x30
[<ffffffff90bf7404>] unmap_region+0xf4/0x140
[<ffffffff90bf79c1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff90bf9a05>] do_munmap+0x2a5/0x480
[<ffffffff90bfa298>] SyS_brk+0x178/0x200
[<ffffffff9118dede>] system_call_fastpath+0x25/0x2a
Code: 14 c5 20 e8 74 91 48 89 df e8 f7 22 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 f6 43 20 01 <75> f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 00 0f 85 8f
Complete report #243512
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa5f16a12
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffffa5e7ddd0
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffffa5e7ddd0
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffffa5e7ddd0
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffffa5f16b2f
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa61869f5
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffffa5f170db
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa5e7e258
Build id: 3.10.0-1160.36.2.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa5e7e2cb
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa5fee656
Build id: 3.10.0-1160.36.2.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffa5ff011f
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffa5ff0243
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x23
13
unmap_region
vmlinux 0xffffffffa5ff9a34
Build id: 3.10.0-1160.36.2.el7.x86_64 		0xf4
14
do_munmap
vmlinux 0xffffffffa5ffc025
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x2a5
15
SyS_brk
vmlinux 0xffffffffa5ffc8b8
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x178
16
system_call_fastpath
vmlinux 0xffffffffa6595f92
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x25
17
? system_call_after_swapgs
vmlinux 0xffffffffa6595ed5
Build id: 3.10.0-1160.36.2.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 49s! [abrt-hook-ccpp:31575]
Modules linked in: ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat vmw_vsock_vmci_transport vsock nf_conntrack sunrpc coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd pcspkr vmw_balloon joydev sg parport_pc i2c_piix4 parport vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul ata_piix crct10dif_common drm mptspi e1000 crc32c_intel scsi_transport_spi mptscsih serio_raw libata mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 31575 Comm: abrt-hook-ccpp Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.36.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/30/2013
task: ffff88113f05b180 ti: ffff881137dc0000 task.ti: ffff881137dc0000
RIP: 0010:[<ffffffffa5f16a12>]  [<ffffffffa5f16a12>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff881137dc3c40  EFLAGS: 00000202
RAX: 00000000000008fd RBX: ffff88117ffd9800 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff881137dc3c88 R08: 0000000000000001 R09: ffffffffa6002f68
R10: ffff88117fd1f0a0 R11: ffffdc1901da8e00 R12: 0000000200000000
R13: ffffffffa603d7de R14: ffff881137dc3bf8 R15: ffff88117f952c00
FS:  00007fbdc8473900(0000) GS:ffff88117fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbdc6b87440 CR3: 000000003f1aa000 CR4: 00000000000407e0
Call Trace:
[<ffffffffa5e7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa5e7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa5e7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa5f16b2f>] smp_call_function_single+0x5f/0xa0
[<ffffffffa61869f5>] ? cpumask_next_and+0x35/0x50
[<ffffffffa5f170db>] smp_call_function_many+0x22b/0x270
[<ffffffffa5e7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa5e7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa5fee656>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa5ff011f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa5ff0243>] tlb_finish_mmu+0x23/0x30
[<ffffffffa5ff9a34>] unmap_region+0xf4/0x140
[<ffffffffa5ffc025>] do_munmap+0x2a5/0x480
[<ffffffffa5ffc8b8>] SyS_brk+0x178/0x200
[<ffffffffa6595f92>] system_call_fastpath+0x25/0x2a
[<ffffffffa6595ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 60 15 b5 a6 48 89 df e8 57 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #157286
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff993167a2
Build id: 3.10.0-1160.15.2.el7.x86_64 		0x102
2
? leave_mm
vmlinux 0xffffffff9927e6a0
Build id: 3.10.0-1160.15.2.el7.x86_64 		0x120
3
? leave_mm
vmlinux 0xffffffff9927e6a0
Build id: 3.10.0-1160.15.2.el7.x86_64 		0x120
4
? leave_mm
vmlinux 0xffffffff9927e6a0
Build id: 3.10.0-1160.15.2.el7.x86_64 		0x120
5
smp_call_function_single
vmlinux 0xffffffff993168bf
Build id: 3.10.0-1160.15.2.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff995862c5
Build id: 3.10.0-1160.15.2.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff99316e6b
Build id: 3.10.0-1160.15.2.el7.x86_64 		0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9927e868
Build id: 3.10.0-1160.15.2.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9927e8db
Build id: 3.10.0-1160.15.2.el7.x86_64 		0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff993ede86
Build id: 3.10.0-1160.15.2.el7.x86_64 		0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff993ef94f
Build id: 3.10.0-1160.15.2.el7.x86_64 		0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff993efa73
Build id: 3.10.0-1160.15.2.el7.x86_64 		0x23
13
unmap_region
vmlinux 0xffffffff993f9264
Build id: 3.10.0-1160.15.2.el7.x86_64 		0xf4
14
do_munmap
vmlinux 0xffffffff993fb855
Build id: 3.10.0-1160.15.2.el7.x86_64 		0x2a5
15
SyS_brk
vmlinux 0xffffffff993fc0e8
Build id: 3.10.0-1160.15.2.el7.x86_64 		0x178
16
system_call_fastpath
vmlinux 0xffffffff99994f92
Build id: 3.10.0-1160.15.2.el7.x86_64 		0x25
17
? system_call_after_swapgs
vmlinux 0xffffffff99994ed5
Build id: 3.10.0-1160.15.2.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [abrt-action-ure:69965]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink xenfs xen_privcmd nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ppdev joydev sg pcspkr i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom cirrus drm_kms_helper ata_generic syscopyarea sysfillrect pata_acpi sysimgblt fb_sys_fops
ttm xen_netfront xen_blkfront drm ata_piix libata serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 69965 Comm: abrt-action-ure Kdump: loaded Tainted: G             L ------------   3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: Xen HVM domU, BIOS 4.13 11/12/2020
task: ffff88ff59774200 ti: ffff88ff60480000 task.ti: ffff88ff60480000
RIP: 0010:[<ffffffff993167a2>]  [<ffffffff993167a2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff88ff60483c40  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88ff60483be0 RCX: 0000000000000080
RDX: ffffffff99a137d0 RSI: 0000000000000000 RDI: 0000000000000080
RBP: ffff88ff60483c88 R08: ffffffff99a137c8 R09: ffffffff99402798
R10: ffff89004925f0a0 R11: ffffd599026b98c0 R12: ffff88ff60483bc0
R13: ffffffff99352f6e R14: ffff88ff60483bb0 R15: ffffffff9934f577
FS:  00007f4cafef0740(0000) GS:ffff890049240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000b8e9f0 CR3: 0000000023064000 CR4: 00000000000006e0
Call Trace:
[<ffffffff9927e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9927e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9927e6a0>] ? leave_mm+0x120/0x120
[<ffffffff993168bf>] smp_call_function_single+0x5f/0xa0
[<ffffffff995862c5>] ? cpumask_next_and+0x35/0x50
[<ffffffff99316e6b>] smp_call_function_many+0x22b/0x270
[<ffffffff9927e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9927e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff993ede86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff993ef94f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff993efa73>] tlb_finish_mmu+0x23/0x30
[<ffffffff993f9264>] unmap_region+0xf4/0x140
[<ffffffff993fb855>] do_munmap+0x2a5/0x480
[<ffffffff993fc0e8>] SyS_brk+0x178/0x200
[<ffffffff99994f92>] system_call_fastpath+0x25/0x2a
[<ffffffff99994ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 e0 13 f5 99 48 89 df e8 97 6c 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #994225
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f93ba
Build id: 3.10.0-514.2.2.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.2.2.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.2.2.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.2.2.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810f94bf
Build id: 3.10.0-514.2.2.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81317b25
Build id: 3.10.0-514.2.2.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9a63
Build id: 3.10.0-514.2.2.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c5f8
Build id: 3.10.0-514.2.2.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c6c9
Build id: 3.10.0-514.2.2.el7.x86_64 		0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac1f3
Build id: 3.10.0-514.2.2.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad4e5
Build id: 3.10.0-514.2.2.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff811b43b4
Build id: 3.10.0-514.2.2.el7.x86_64 		0xf4
13
? __schedule
vmlinux 0xffffffff8168b290
Build id: 3.10.0-514.2.2.el7.x86_64 		0x3b0
14
? vma_rb_erase
vmlinux 0xffffffff811b4931
Build id: 3.10.0-514.2.2.el7.x86_64 		0x121
15
do_munmap
vmlinux 0xffffffff811b6765
Build id: 3.10.0-514.2.2.el7.x86_64 		0x245
16
vm_munmap
vmlinux 0xffffffff811b6981
Build id: 3.10.0-514.2.2.el7.x86_64 		0x41
17
SyS_munmap
vmlinux 0xffffffff811b7912
Build id: 3.10.0-514.2.2.el7.x86_64 		0x22
18
system_call_fastpath
vmlinux 0xffffffff816967c9
Build id: 3.10.0-514.2.2.el7.x86_64 		0x16 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [systemd-journal:534]
Modules linked in: tcp_lp isofs fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock intel_powerclamp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr sg vmw_balloon vmw_vmci parport_pc parport shpchp i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect crct10dif_common sysimgblt fb_sys_fops crc32c_intel serio_raw ttm mptspi scsi_transport_spi ata_piix drm mptscsih libata vmxnet3 mptbase i2c_core floppy fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 534 Comm: systemd-journal Tainted: G             L ------------   3.10.0-514.2.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff880232da4e70 ti: ffff88023fa04000 task.ti: ffff88023fa04000
RIP: 0010:[<ffffffff810f93ba>]  [<ffffffff810f93ba>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88023fa07c80  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff88023fa07c50 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff88023fa07cd0 R08: ffffffff816afea0 R09: 0000000180400015
R10: 0000000027e21c01 R11: ffffea00089f8840 R12: ffff880184ca0400
R13: 0000000000000001 R14: ffffea0001c72d00 R15: ffff88018779cb40
FS:  00007f18a456c880(0000) GS:ffff88023fcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000010f67c8 CR3: 0000000232a9f000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c430 ffff88023fa07d58
0000000000000003 000000008633dcbf 0000000100014c30 0000000000000002
ffffffff8106c430 ffffffff8106c430 ffff88023fa07d00 ffffffff810f94bf
Call Trace:
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff810f94bf>] smp_call_function_single+0x5f/0xa0
[<ffffffff81317b25>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9a63>] smp_call_function_many+0x223/0x260
[<ffffffff8106c5f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c6c9>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac1f3>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad4e5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b43b4>] unmap_region+0xf4/0x140
[<ffffffff8168b290>] ? __schedule+0x3b0/0x990
[<ffffffff811b4931>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6765>] do_munmap+0x245/0x420
[<ffffffff811b6981>] vm_munmap+0x41/0x60
[<ffffffff811b7912>] SyS_munmap+0x22/0x30
[<ffffffff816967c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 80 57 ad 81 48 89 df e8 0a 29 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #913896
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810d66ea
Build id: 3.10.0-229.20.1.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.20.1.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.20.1.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.20.1.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810d67ef
Build id: 3.10.0-229.20.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812d4745
Build id: 3.10.0-229.20.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810d6d83
Build id: 3.10.0-229.20.1.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8105fc78
Build id: 3.10.0-229.20.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8105fd3c
Build id: 3.10.0-229.20.1.el7.x86_64 		0x5c
10
tlb_flush_mmu.part.53
vmlinux 0xffffffff8117e063
Build id: 3.10.0-229.20.1.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff8117ebc5
Build id: 3.10.0-229.20.1.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff811866b2
Build id: 3.10.0-229.20.1.el7.x86_64 		0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811e67be
Build id: 3.10.0-229.20.1.el7.x86_64 		0x3e
14
? vma_rb_erase
vmlinux 0xffffffff81186c11
Build id: 3.10.0-229.20.1.el7.x86_64 		0x121
15
do_munmap
vmlinux 0xffffffff81188a42
Build id: 3.10.0-229.20.1.el7.x86_64 		0x242
16
vm_munmap
vmlinux 0xffffffff81188c51
Build id: 3.10.0-229.20.1.el7.x86_64 		0x41
17
SyS_munmap
vmlinux 0xffffffff81189b82
Build id: 3.10.0-229.20.1.el7.x86_64 		0x22
18
system_call_fastpath
vmlinux 0xffffffff81614409
Build id: 3.10.0-229.20.1.el7.x86_64 		0x16 
BUG: soft lockup - CPU#0 stuck for 23s! [systemd-journal:508]
BUG: soft lockup - CPU#1 stuck for 22s! [safe_timer:1423]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6table_filter ip6_tables iptable_filter tcp_diag udp_diag inet_diag onload(OF) sfc_char(OF) sfc_resource(OF) sfc_affinity(OF) fuse sfc(OF) mtd ptp pps_core mdio onload_cplane(POF) crc32_pclmul iTCO_wdt iTCO_vendor_support ghash_clmulni_intel aesni_intel lrw pcspkr sg gf128mul glue_helper ablk_helper cryptd lpc_ich mfd_core i2c_i801 shpchp nfsd nfs_acl lockd auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ahci cirrus crct10dif_pclmul crct10dif_common syscopyarea crc32c_intel libahci sysfillrect sysimgblt serio_raw drm_kms_helper libata virtio_pci virtio_ring virtio ttm e1000 drm i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1423 Comm: safe_timer Tainted: PF       W  O--------------   3.10.0-229.20.1.el7.x86_64 #1
Hardware name: RDO OpenStack Compute/RHEL-AV, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8804662e8000 ti: ffff8804662e4000 task.ti: ffff8804662e4000
RIP: 0010:[<ffffffff810d66ea>]  [<ffffffff810d66ea>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8804662e7c80  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff8804662e7c50 RCX: 0000000000000830
RDX: 0000000000000004 RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff8804662e7cd0 R08: ffffffff816303a0 R09: ffff88047fc963e0
R10: ffffffff8118deb5 R11: ffffea000c6eb340 R12: ffff8804662e7c10
R13: ffff8804671e79c4 R14: ffff88047fc93680 R15: ffff8804671e71c0
FS:  00007fe77d763700(0000) GS:ffff88047fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe7952082c0 CR3: 000000046695f000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8105fab0 ffff8804662e7d58
0000000000000003 00000000b151a82a 0000000000000000 0000000000000003
ffffffff8105fab0 ffffffff8105fab0 ffff8804662e7d00 ffffffff810d67ef
Call Trace:
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff810d67ef>] smp_call_function_single+0x5f/0xa0
[<ffffffff812d4745>] ? cpumask_next_and+0x35/0x50
[<ffffffff810d6d83>] smp_call_function_many+0x223/0x260
[<ffffffff8105fc78>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8105fd3c>] flush_tlb_mm_range+0x5c/0x180
[<ffffffff8117e063>] tlb_flush_mmu.part.53+0x33/0x90
[<ffffffff8117ebc5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811866b2>] unmap_region+0xe2/0x130
[<ffffffff811e67be>] ? mntput_no_expire+0x3e/0x120
[<ffffffff81186c11>] ? vma_rb_erase+0x121/0x220
[<ffffffff81188a42>] do_munmap+0x242/0x410
[<ffffffff81188c51>] vm_munmap+0x41/0x60
[<ffffffff81189b82>] SyS_munmap+0x22/0x30
[<ffffffff81614409>] system_call_fastpath+0x16/0x1b
Code: 40 41 01 00 48 89 de 48 03 14 c5 80 4d a1 81 48 89 df e8 7a 13 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #634864
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64 		0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff8119b2d2
Build id: 3.10.0-327.el7.x86_64 		0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811fe6fe
Build id: 3.10.0-327.el7.x86_64 		0x3e
14
? vma_rb_erase
vmlinux 0xffffffff8119b831
Build id: 3.10.0-327.el7.x86_64 		0x121
15
do_munmap
vmlinux 0xffffffff8119d612
Build id: 3.10.0-327.el7.x86_64 		0x242
16
vm_munmap
vmlinux 0xffffffff8119d821
Build id: 3.10.0-327.el7.x86_64 		0x41
17
SyS_munmap
vmlinux 0xffffffff8119e772
Build id: 3.10.0-327.el7.x86_64 		0x22
18
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64 		0x16 
BUG: soft lockup - CPU#3 stuck for 37s! [ftdc:2342]
Modules linked in: gsch(OE) redirfs(OE) dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop bridge stp llc ip_set nfnetlink dsa_filter(POE) vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr sg shpchp vmw_vmci parport_pc parport i2c_piix4 xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx serio_raw drm_kms_helper ttm mptsas scsi_transport_sas drm mptscsih mptbase ata_piix vmxnet3 libata i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 3 PID: 2342 Comm: ftdc Tainted: P           OEL ------------   3.10.0-327.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff88042703a280 ti: ffff8803fcad8000 task.ti: ffff8803fcad8000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8803fcadbc80  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff8803fcadbc50 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff8803fcadbcd0 R08: ffffffff8165ea60 R09: ffff88043fd97520
R10: ffffffff811a2bc5 R11: ffffea000e2f8500 R12: ffff880036acd780
R13: ffff88042703a280 R14: ffff8803fcadbc40 R15: ffff88042703a280
FS:  00007ff10d5e6700(0000) GS:ffff88043fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff1195d0000 CR3: 0000000406b57000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff8803fcadbd58
0000000000000002 0000000082570b9a 0000000000000000 0000000000000001
ffffffff81065c90 ffffffff81065c90 ffff8803fcadbd00 ffffffff810e685f
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b2d2>] unmap_region+0xe2/0x130
[<ffffffff811fe6fe>] ? mntput_no_expire+0x3e/0x120
[<ffffffff8119b831>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d612>] do_munmap+0x242/0x410
[<ffffffff8119d821>] vm_munmap+0x41/0x60
[<ffffffff8119e772>] SyS_munmap+0x22/0x30
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #355560
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810d66ea
Build id: 3.10.0-229.14.1.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.14.1.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.14.1.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.14.1.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810d67ef
Build id: 3.10.0-229.14.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812d4695
Build id: 3.10.0-229.14.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810d6d83
Build id: 3.10.0-229.14.1.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8105fc78
Build id: 3.10.0-229.14.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8105fd3c
Build id: 3.10.0-229.14.1.el7.x86_64 		0x5c
10
tlb_flush_mmu.part.53
vmlinux 0xffffffff8117e063
Build id: 3.10.0-229.14.1.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff8117ebc5
Build id: 3.10.0-229.14.1.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff811866b2
Build id: 3.10.0-229.14.1.el7.x86_64 		0xe2
13
? vma_rb_erase
vmlinux 0xffffffff81186c11
Build id: 3.10.0-229.14.1.el7.x86_64 		0x121
14
do_munmap
vmlinux 0xffffffff81188a42
Build id: 3.10.0-229.14.1.el7.x86_64 		0x242
15
vm_munmap
vmlinux 0xffffffff81188c51
Build id: 3.10.0-229.14.1.el7.x86_64 		0x41
16
SyS_munmap
vmlinux 0xffffffff81189b82
Build id: 3.10.0-229.14.1.el7.x86_64 		0x22
17
system_call_fastpath
vmlinux 0xffffffff81614389
Build id: 3.10.0-229.14.1.el7.x86_64 		0x16 
BUG: soft lockup - CPU#4 stuck for 22s! [litespeed:25780]
Modules linked in: binfmt_misc tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables dahdi(OF) crc_ccitt netatop(OF) xt_CHECKSUM iptable_mangle xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_nat_ipv4 nf_nat nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_recent xt_multiport xt_mark xt_conntrack nf_conntrack iptable_filter coretemp ext4 mbcache jbd2 snd_ens1371 snd_rawmidi snd_ac97_codec crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq snd_seq_device aesni_intel snd_pcm lrw gf128mul snd_timer ppdev glue_helper snd ablk_helper cryptd pcspkr vmw_balloon soundcore ahci libahci sg vmw_vmci i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif
crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel ata_piix ttm mptspi serio_raw scsi_transport_spi drm mptscsih libata e1000 mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 25780 Comm: litespeed Tainted: GF          O--------------   3.10.0-229.14.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff880271f60b60 ti: ffff88012153c000 task.ti: ffff88012153c000
RIP: 0010:[<ffffffff810d66ea>]  [<ffffffff810d66ea>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88012153fc80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88012153fc50 RCX: 0000000000000001
RDX: ffffffff8162fc30 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88012153fcd0 R08: ffffffff8162fc28 R09: ffff8803326963e0
R10: ffffffff8118de42 R11: ffffea00050aec80 R12: 0000000100000000
R13: 00000000814ecf31 R14: ffffffff81553b4d R15: ffff88012153fc40
FS:  00007f5a87f25640(0000) GS:ffff880332680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5a87f17ff0 CR3: 000000015826e000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8105fab0 ffff88012153fd58
0000000000000003 0000000038b9c549 0000000000000000 0000000000000000
ffffffff8105fab0 ffffffff8105fab0 ffff88012153fd00 ffffffff810d67ef
Call Trace:
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff810d67ef>] smp_call_function_single+0x5f/0xa0
[<ffffffff812d4695>] ? cpumask_next_and+0x35/0x50
[<ffffffff810d6d83>] smp_call_function_many+0x223/0x260
[<ffffffff8105fc78>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8105fd3c>] flush_tlb_mm_range+0x5c/0x180
[<ffffffff8117e063>] tlb_flush_mmu.part.53+0x33/0x90
[<ffffffff8117ebc5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811866b2>] unmap_region+0xe2/0x130
[<ffffffff81186c11>] ? vma_rb_erase+0x121/0x220
[<ffffffff81188a42>] do_munmap+0x242/0x410
[<ffffffff81188c51>] vm_munmap+0x41/0x60
[<ffffffff81189b82>] SyS_munmap+0x22/0x30
[<ffffffff81614389>] system_call_fastpath+0x16/0x1b
Code: 40 41 01 00 48 89 de 48 03 14 c5 00 4d a1 81 48 89 df e8 ba 12 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #319942
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e687a
Build id: 3.10.0-327.13.1.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff81065c80
Build id: 3.10.0-327.13.1.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff81065c80
Build id: 3.10.0-327.13.1.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff81065c80
Build id: 3.10.0-327.13.1.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810e697f
Build id: 3.10.0-327.13.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f28f5
Build id: 3.10.0-327.13.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6f23
Build id: 3.10.0-327.13.1.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e48
Build id: 3.10.0-327.13.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f16
Build id: 3.10.0-327.13.1.el7.x86_64 		0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192663
Build id: 3.10.0-327.13.1.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811931f5
Build id: 3.10.0-327.13.1.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff8119b3d2
Build id: 3.10.0-327.13.1.el7.x86_64 		0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811fe7ce
Build id: 3.10.0-327.13.1.el7.x86_64 		0x3e
14
? vma_rb_erase
vmlinux 0xffffffff8119b931
Build id: 3.10.0-327.13.1.el7.x86_64 		0x121
15
do_munmap
vmlinux 0xffffffff8119d712
Build id: 3.10.0-327.13.1.el7.x86_64 		0x242
16
vm_munmap
vmlinux 0xffffffff8119d921
Build id: 3.10.0-327.13.1.el7.x86_64 		0x41
17
SyS_munmap
vmlinux 0xffffffff8119e872
Build id: 3.10.0-327.13.1.el7.x86_64 		0x22
18
system_call_fastpath
vmlinux 0xffffffff81645ec9
Build id: 3.10.0-327.13.1.el7.x86_64 		0x16 
BUG: soft lockup - CPU#6 stuck for 22s! [ir_agent:22470]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_12904(E) falcon_lsm_pinned_12803(E) falcon_lsm_pinned_12704(E) falcon_lsm_pinned_12606(E) falcon_lsm_pinned_10807(E) dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag ip6t_REJECT nf_log_ipv6 nf_log_ipv4 nf_log_common ip6table_filter nf_nat_ftp xt_REDIRECT xt_conntrack iptable_mangle nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables ipt_REJECT iptable_filter dm_mirror dm_region_hash dm_log dm_mod crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr sg parport_pc parport shpchp vmw_vmci
i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper ttm mptspi serio_raw drm scsi_transport_spi ata_piix mptscsih e1000 libata mptbase i2c_core floppy [last unloaded: falcon_kal]
CPU: 6 PID: 22470 Comm: ir_agent Tainted: P        W   EL ------------   3.10.0-327.13.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8800b3632280 ti: ffff880035374000 task.ti: ffff880035374000
RIP: 0010:[<ffffffff810e687a>]  [<ffffffff810e687a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff880035377c80  EFLAGS: 00000202
RAX: 0000000000000020 RBX: ffff8800b3632280 RCX: 0000000000000020
RDX: 0000000000000020 RSI: 0000000000000020 RDI: 0000000000000286
RBP: ffff880035377cd0 R08: ffffffff8165e7e0 R09: ffff8807a12d7520
R10: ffffffff811a2cb5 R11: ffffea001e37ebc0 R12: 0000000000000000
R13: ffffffff816413dd R14: ffff880035377c40 R15: 0000000000000029
FS:  00007fa5cb7fe700(0000) GS:ffff8807a12c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa5f4cf2000 CR3: 000000078f424000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c80 ffff880035377d58
0000000000000003 0000000037c75518 0000000000000000 0000000000000000
ffffffff81065c80 ffffffff81065c80 ffff880035377d00 ffffffff810e697f
Call Trace:
[<ffffffff81065c80>] ? leave_mm+0x70/0x70
[<ffffffff81065c80>] ? leave_mm+0x70/0x70
[<ffffffff81065c80>] ? leave_mm+0x70/0x70
[<ffffffff810e697f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f28f5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6f23>] smp_call_function_many+0x223/0x260
[<ffffffff81065e48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f16>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192663>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811931f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b3d2>] unmap_region+0xe2/0x130
[<ffffffff811fe7ce>] ? mntput_no_expire+0x3e/0x120
[<ffffffff8119b931>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d712>] do_munmap+0x242/0x410
[<ffffffff8119d921>] vm_munmap+0x41/0x60
[<ffffffff8119e872>] SyS_munmap+0x22/0x30
[<ffffffff81645ec9>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 2a fd 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #209811
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f925a
Build id: 3.10.0-514.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810f935f
Build id: 3.10.0-514.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813179a5
Build id: 3.10.0-514.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9903
Build id: 3.10.0-514.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c4c8
Build id: 3.10.0-514.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c599
Build id: 3.10.0-514.el7.x86_64 		0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac093
Build id: 3.10.0-514.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad385
Build id: 3.10.0-514.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff811b4254
Build id: 3.10.0-514.el7.x86_64 		0xf4
13
? vma_rb_erase
vmlinux 0xffffffff811b47d1
Build id: 3.10.0-514.el7.x86_64 		0x121
14
do_munmap
vmlinux 0xffffffff811b6605
Build id: 3.10.0-514.el7.x86_64 		0x245
15
vm_munmap
vmlinux 0xffffffff811b6821
Build id: 3.10.0-514.el7.x86_64 		0x41
16
SyS_munmap
vmlinux 0xffffffff811b77b2
Build id: 3.10.0-514.el7.x86_64 		0x22
17
system_call_fastpath
vmlinux 0xffffffff816965c9
Build id: 3.10.0-514.el7.x86_64 		0x16 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [zmstat-cpu:10521]
Modules linked in: talpa_vfshook(OE) talpa_pedconnector(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) tcp_diag inet_diag binfmt_misc talpa_pedevice(OE) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock intel_powerclamp coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ppdev pcspkr vmw_balloon
parport_pc parport i2c_piix4 vmw_vmci shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mptspi ata_piix ttm crct10dif_pclmul scsi_transport_spi crct10dif_common libata mptscsih drm crc32c_intel serio_raw mptbase vmxnet3 i2c_core floppy fjes dm_mirror dm_region_hash dm_log dm_mod [last unloaded: talpa_syscallhook]
CPU: 2 PID: 10521 Comm: zmstat-cpu Tainted: G        W  OEL ------------   3.10.0-514.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff88041cbf8fb0 ti: ffff88000d9bc000 task.ti: ffff88000d9bc000
RIP: 0010:[<ffffffff810f925a>]  [<ffffffff810f925a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88000d9bfc80  EFLAGS: 00000202
RAX: 00000000000008fd RBX: ffff880036359140 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000282
RBP: ffff88000d9bfcd0 R08: 0000000000000001 R09: ffffffff811bbed8
R10: ffff88043fc99a80 R11: ffffea000dd3e140 R12: 0000000000000000
R13: ffffffffa02788a7 R14: ffff88000d9bfc58 R15: ffff88000d9bfce8
FS:  00007fd96940d740(0000) GS:ffff88043fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002450230 CR3: 0000000035959000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c300 ffff88000d9bfd58
0000000000000003 0000000044478a09 ffff88004bd870f0 0000000000000001
ffffffff8106c300 ffffffff8106c300 ffff88000d9bfd00 ffffffff810f935f
Call Trace:
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff810f935f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813179a5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9903>] smp_call_function_many+0x223/0x260
[<ffffffff8106c4c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c599>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac093>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad385>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b4254>] unmap_region+0xf4/0x140
[<ffffffff811b47d1>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6605>] do_munmap+0x245/0x420
[<ffffffff811b6821>] vm_munmap+0x41/0x60
[<ffffffff811b77b2>] SyS_munmap+0x22/0x30
[<ffffffff816965c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 40 57 ad 81 48 89 df e8 ea 28 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #193236
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f925a
Build id: 3.10.0-514.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810f935f
Build id: 3.10.0-514.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813179a5
Build id: 3.10.0-514.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9903
Build id: 3.10.0-514.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c4c8
Build id: 3.10.0-514.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c599
Build id: 3.10.0-514.el7.x86_64 		0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac093
Build id: 3.10.0-514.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad385
Build id: 3.10.0-514.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff811b4254
Build id: 3.10.0-514.el7.x86_64 		0xf4
13
? xfs_release
xfs 0xffffffffa024507e
Build id: 3.10.0-514.el7.x86_64 		0x9e
14
? mntput_no_expire
vmlinux 0xffffffff8121e7de
Build id: 3.10.0-514.el7.x86_64 		0x3e
15
? vma_rb_erase
vmlinux 0xffffffff811b47d1
Build id: 3.10.0-514.el7.x86_64 		0x121
16
do_munmap
vmlinux 0xffffffff811b6605
Build id: 3.10.0-514.el7.x86_64 		0x245
17
vm_munmap
vmlinux 0xffffffff811b6821
Build id: 3.10.0-514.el7.x86_64 		0x41
18
SyS_munmap
vmlinux 0xffffffff811b77b2
Build id: 3.10.0-514.el7.x86_64 		0x22
19
system_call_fastpath
vmlinux 0xffffffff816965c9
Build id: 3.10.0-514.el7.x86_64 		0x16 
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [I2Availability:2776]
Modules linked in: binfmt_misc iptable_filter fuse cfg80211 rfkill sfs(OE) sg ppdev parport_pc parport virtio_balloon i2c_piix4 intel_powerclamp pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_scsi virtio_net virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix serio_raw libata virtio_pci i2c_core virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 2776 Comm: I2Availability Tainted: G           OEL ------------   3.10.0-514.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff8800b9903ec0 ti: ffff88041ea70000 task.ti: ffff88041ea70000
RIP: 0010:[<ffffffff810f925a>]  [<ffffffff810f925a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88041ea73c80  EFLAGS: 00000202
RAX: 00000000000000a0 RBX: ffff88041ea73c50 RCX: 0000000000000020
RDX: 00000000ffffffff RSI: 00000000000000a0 RDI: 0000000000000282
RBP: ffff88041ea73cd0 R08: ffffffff816afc28 R09: ffffffff811bbfcc
R10: ffff88042cf99a80 R11: ffffea001069ca00 R12: 0000000000000001
R13: ffffffff8169151b R14: ffff88041ea73bf8 R15: 000000000000015e
FS:  00007f9d011738c0(0000) GS:ffff88042cf80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9d0119a000 CR3: 000000041d680000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c300 ffff88041ea73d58
0000000000000003 0000000013f831f6 ffff88041caf4cd0 0000000000000001
ffffffff8106c300 ffffffff8106c300 ffff88041ea73d00 ffffffff810f935f
Call Trace:
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff810f935f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813179a5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9903>] smp_call_function_many+0x223/0x260
[<ffffffff8106c4c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c599>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac093>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad385>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b4254>] unmap_region+0xf4/0x140
[<ffffffffa024507e>] ? xfs_release+0x9e/0x170 [xfs]
[<ffffffff8121e7de>] ? mntput_no_expire+0x3e/0x120
[<ffffffff811b47d1>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6605>] do_munmap+0x245/0x420
[<ffffffff811b6821>] vm_munmap+0x41/0x60
[<ffffffff811b77b2>] SyS_munmap+0x22/0x30
[<ffffffff816965c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 40 57 ad 81 48 89 df e8 ea 28 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #184794
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810d66ea
Build id: 3.10.0-229.14.1.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.14.1.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.14.1.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.14.1.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810d67ef
Build id: 3.10.0-229.14.1.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812d4695
Build id: 3.10.0-229.14.1.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810d6d83
Build id: 3.10.0-229.14.1.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8105fc78
Build id: 3.10.0-229.14.1.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8105fe15
Build id: 3.10.0-229.14.1.el7.x86_64 		0x135
10
tlb_flush_mmu.part.53
vmlinux 0xffffffff8117e063
Build id: 3.10.0-229.14.1.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff8117ebc5
Build id: 3.10.0-229.14.1.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff811866b2
Build id: 3.10.0-229.14.1.el7.x86_64 		0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811e670e
Build id: 3.10.0-229.14.1.el7.x86_64 		0x3e
14
? vma_rb_erase
vmlinux 0xffffffff81186c11
Build id: 3.10.0-229.14.1.el7.x86_64 		0x121
15
do_munmap
vmlinux 0xffffffff81188a42
Build id: 3.10.0-229.14.1.el7.x86_64 		0x242
16
vm_munmap
vmlinux 0xffffffff81188c51
Build id: 3.10.0-229.14.1.el7.x86_64 		0x41
17
SyS_munmap
vmlinux 0xffffffff81189b82
Build id: 3.10.0-229.14.1.el7.x86_64 		0x22
18
system_call_fastpath
vmlinux 0xffffffff81614389
Build id: 3.10.0-229.14.1.el7.x86_64 		0x16 
BUG: soft lockup - CPU#5 stuck for 23s! [ftdc:5059]
Modules linked in: binfmt_misc tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables dahdi(OF) crc_ccitt netatop(OF) xt_CHECKSUM iptable_mangle xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_nat_ipv4 nf_nat nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_recent xt_multiport xt_mark xt_conntrack vmw_vsock_vmci_transport nf_conntrack vsock iptable_filter ext4 mbcache jbd2 coretemp snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq crc32_pclmul snd_seq_device ghash_clmulni_intel snd_pcm ppdev aesni_intel snd_timer lrw gf128mul vmw_balloon snd glue_helper ablk_helper cryptd soundcore pcspkr ahci sg i2c_piix4 libahci vmw_vmci shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd sunrpc ip_tables xfs libcrc32c
sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common ttm crc32c_intel mptspi serio_raw drm scsi_transport_spi ata_piix mptscsih libata e1000 mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 5059 Comm: ftdc Tainted: GF          O--------------   3.10.0-229.14.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8800bafcc440 ti: ffff880328748000 task.ti: ffff880328748000
RIP: 0010:[<ffffffff810d66ea>]  [<ffffffff810d66ea>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88032874bc80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88032874bc50 RCX: 0000000000000001
RDX: ffffffff8162fc30 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88032874bcd0 R08: ffffffff8162fc28 R09: ffff8803326b63e0
R10: ffffffff8118deb5 R11: ffffea0007a58f00 R12: ffffffff8116520e
R13: ffff88032874bbf8 R14: ffff88032acc2798 R15: ffffea00013c8a80
FS:  00007f06b53eb700(0000) GS:ffff8803326a0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f06c1bd1000 CR3: 00000003285f8000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8105fab0 ffff88032874bd58
0000000000000003 000000003ff7507d 0000000000000000 0000000000000000
ffffffff8105fab0 ffffffff8105fab0 ffff88032874bd00 ffffffff810d67ef
Call Trace:
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff810d67ef>] smp_call_function_single+0x5f/0xa0
[<ffffffff812d4695>] ? cpumask_next_and+0x35/0x50
[<ffffffff810d6d83>] smp_call_function_many+0x223/0x260
[<ffffffff8105fc78>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8105fe15>] flush_tlb_mm_range+0x135/0x180
[<ffffffff8117e063>] tlb_flush_mmu.part.53+0x33/0x90
[<ffffffff8117ebc5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811866b2>] unmap_region+0xe2/0x130
[<ffffffff811e670e>] ? mntput_no_expire+0x3e/0x120
[<ffffffff81186c11>] ? vma_rb_erase+0x121/0x220
[<ffffffff81188a42>] do_munmap+0x242/0x410
[<ffffffff81188c51>] vm_munmap+0x41/0x60
[<ffffffff81189b82>] SyS_munmap+0x22/0x30
[<ffffffff81614389>] system_call_fastpath+0x16/0x1b
Code: 40 41 01 00 48 89 de 48 03 14 c5 00 4d a1 81 48 89 df e8 ba 12 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #174889
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810d677a
Build id: 3.10.0-229.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8105faf0
Build id: 3.10.0-229.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff8105faf0
Build id: 3.10.0-229.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff8105faf0
Build id: 3.10.0-229.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810d687f
Build id: 3.10.0-229.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812d4ae5
Build id: 3.10.0-229.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810d6e23
Build id: 3.10.0-229.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8105fcb8
Build id: 3.10.0-229.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8105fe55
Build id: 3.10.0-229.el7.x86_64 		0x135
10
tlb_flush_mmu.part.53
vmlinux 0xffffffff8117e083
Build id: 3.10.0-229.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff8117ec25
Build id: 3.10.0-229.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff81186732
Build id: 3.10.0-229.el7.x86_64 		0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811e68ae
Build id: 3.10.0-229.el7.x86_64 		0x3e
14
? vma_rb_erase
vmlinux 0xffffffff81186c91
Build id: 3.10.0-229.el7.x86_64 		0x121
15
do_munmap
vmlinux 0xffffffff81188ac2
Build id: 3.10.0-229.el7.x86_64 		0x242
16
vm_munmap
vmlinux 0xffffffff81188cd1
Build id: 3.10.0-229.el7.x86_64 		0x41
17
SyS_munmap
vmlinux 0xffffffff81189c02
Build id: 3.10.0-229.el7.x86_64 		0x22
18
system_call_fastpath
vmlinux 0xffffffff81614a29
Build id: 3.10.0-229.el7.x86_64 		0x16 
BUG: soft lockup - CPU#5 stuck for 23s! [systemd-journal:543]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd vmw_balloon pcspkr sg parport_pc parport shpchp vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic pata_acpi serio_raw vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel ttm ata_piix
mptspi scsi_transport_spi libata drm mptscsih mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 543 Comm: systemd-journal Not tainted 3.10.0-229.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff880fe3cdf1c0 ti: ffff880fe6f3c000 task.ti: ffff880fe6f3c000
RIP: 0010:[<ffffffff810d677a>]  [<ffffffff810d677a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff880fe6f3fc80  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff880fe6f3fc50 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff880fe6f3fcd0 R08: ffffffff81631620 R09: ffff88103fd563e0
R10: ffffffff8118df55 R11: ffffea003eaffe40 R12: ffffffff8116521e
R13: ffff880fe6f3fbf8 R14: ffff8807327d97a0 R15: ffffea00062c2680
FS:  00007fe9dd02a880(0000) GS:ffff88103fd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe9dd045000 CR3: 0000000fe4543000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8105faf0 ffff880fe6f3fd58
0000000000000003 00000000813dd2f0 ffff880fe4665228 0000000000000004
ffffffff8105faf0 ffffffff8105faf0 ffff880fe6f3fd00 ffffffff810d687f
Call Trace:
[<ffffffff8105faf0>] ? leave_mm+0x70/0x70
[<ffffffff8105faf0>] ? leave_mm+0x70/0x70
[<ffffffff8105faf0>] ? leave_mm+0x70/0x70
[<ffffffff810d687f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812d4ae5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810d6e23>] smp_call_function_many+0x223/0x260
[<ffffffff8105fcb8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8105fe55>] flush_tlb_mm_range+0x135/0x180
[<ffffffff8117e083>] tlb_flush_mmu.part.53+0x33/0x90
[<ffffffff8117ec25>] tlb_finish_mmu+0x55/0x60
[<ffffffff81186732>] unmap_region+0xe2/0x130
[<ffffffff811e68ae>] ? mntput_no_expire+0x3e/0x120
[<ffffffff81186c91>] ? vma_rb_erase+0x121/0x220
[<ffffffff81188ac2>] do_munmap+0x242/0x410
[<ffffffff81188cd1>] vm_munmap+0x41/0x60
[<ffffffff81189c02>] SyS_munmap+0x22/0x30
[<ffffffff81614a29>] system_call_fastpath+0x16/0x1b
Code: 40 41 01 00 48 89 de 48 03 14 c5 c0 4d a1 81 48 89 df e8 da 16 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #159035
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64 		0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff8119b2d2
Build id: 3.10.0-327.el7.x86_64 		0xe2
13
? vma_rb_erase
vmlinux 0xffffffff8119b831
Build id: 3.10.0-327.el7.x86_64 		0x121
14
do_munmap
vmlinux 0xffffffff8119d612
Build id: 3.10.0-327.el7.x86_64 		0x242
15
vm_munmap
vmlinux 0xffffffff8119d821
Build id: 3.10.0-327.el7.x86_64 		0x41
16
SyS_munmap
vmlinux 0xffffffff8119e772
Build id: 3.10.0-327.el7.x86_64 		0x22
17
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64 		0x16 
BUG: soft lockup - CPU#1 stuck for 22s! [mongod:7213]
Modules linked in: bnep bluetooth rfkill fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables vmhgfs(OE) ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device vmw_balloon snd_pcm snd_timer pcspkr snd soundcore vmw_vmci sg i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi crct10dif_pclmul crct10dif_common vmwgfx crc32c_intel drm_kms_helper ttm drm serio_raw i2c_core ata_piix mptspi scsi_transport_spi mptscsih e1000 libata mptbase
CPU: 1 PID: 7213 Comm: mongod Tainted: G           OEL ------------   3.10.0-327.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff88022e7fc500 ti: ffff88022f528000 task.ti: ffff88022f528000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88022f52bc80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88022f52bc50 RCX: 0000000000000004
RDX: ffffffff8165ef70 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88022f52bcd0 R08: ffffffff8165ef68 R09: ffff880236637520
R10: ffffffff811a2b52 R11: ffffea0008babf80 R12: 000000cf00000000
R13: ffffffff81641113 R14: ffff88022f52bc08 R15: 00000000000003a9
FS:  00007f59997a4700(0000) GS:ffff880236620000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f59a40f2000 CR3: 00000000b15af000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff88022f52bd58
0000000000000003 000000006ee8c942 0000000000000000 0000000000000003
ffffffff81065c90 ffffffff81065c90 ffff88022f52bd00 ffffffff810e685f
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b2d2>] unmap_region+0xe2/0x130
[<ffffffff8119b831>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d612>] do_munmap+0x242/0x410
[<ffffffff8119d821>] vm_munmap+0x41/0x60
[<ffffffff8119e772>] SyS_munmap+0x22/0x30
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #136866
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64 		0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff8119b2d2
Build id: 3.10.0-327.el7.x86_64 		0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811fe6fe
Build id: 3.10.0-327.el7.x86_64 		0x3e
14
? vma_rb_erase
vmlinux 0xffffffff8119b831
Build id: 3.10.0-327.el7.x86_64 		0x121
15
do_munmap
vmlinux 0xffffffff8119d612
Build id: 3.10.0-327.el7.x86_64 		0x242
16
vm_munmap
vmlinux 0xffffffff8119d821
Build id: 3.10.0-327.el7.x86_64 		0x41
17
SyS_munmap
vmlinux 0xffffffff8119e772
Build id: 3.10.0-327.el7.x86_64 		0x22
18
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64 		0x16 
BUG: soft lockup - CPU#3 stuck for 22s! [mongod:112987]
Modules linked in: bnep bluetooth rfkill fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables binfmt_misc vmhgfs(OE) ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 vmw_vsock_vmci_transport vsock xt_conntrack nf_conntrack iptable_filter dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event coretemp ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore i2c_piix4 sg shpchp vmw_vmci parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic
sr_mod cdrom crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic vmwgfx pata_acpi drm_kms_helper ttm e1000 drm mptspi scsi_transport_spi mptscsih i2c_core mptbase ata_piix libata
CPU: 3 PID: 112987 Comm: mongod Tainted: G           OEL ------------   3.10.0-327.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8800b49de780 ti: ffff8800b7100000 task.ti: ffff8800b7100000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8800b7103c80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8800b7103c50 RCX: 0000000000000001
RDX: ffffffff8165e7f0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8800b7103cd0 R08: ffffffff8165e7e8 R09: ffff88042e677520
R10: ffffffff811a2bc5 R11: ffffea0010555fc0 R12: ffff880407cc9f40
R13: ffff8800b49de780 R14: ffff8800b7103c40 R15: ffff8800b49de780
FS:  00007ff77fb5c700(0000) GS:ffff88042e660000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff792cbc000 CR3: 00000003f7be9000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff8800b7103d58
0000000000000003 000000001f01b5c6 0000000000000000 0000000000000000
ffffffff81065c90 ffffffff81065c90 ffff8800b7103d00 ffffffff810e685f
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b2d2>] unmap_region+0xe2/0x130
[<ffffffff811fe6fe>] ? mntput_no_expire+0x3e/0x120
[<ffffffff8119b831>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d612>] do_munmap+0x242/0x410
[<ffffffff8119d821>] vm_munmap+0x41/0x60
[<ffffffff8119e772>] SyS_munmap+0x22/0x30
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #136863
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64 		0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff8119b2d2
Build id: 3.10.0-327.el7.x86_64 		0xe2
13
? handle_mm_fault
vmlinux 0xffffffff81197088
Build id: 3.10.0-327.el7.x86_64 		0x5b8
14
? vma_rb_erase
vmlinux 0xffffffff8119b831
Build id: 3.10.0-327.el7.x86_64 		0x121
15
do_munmap
vmlinux 0xffffffff8119d612
Build id: 3.10.0-327.el7.x86_64 		0x242
16
vm_munmap
vmlinux 0xffffffff8119d821
Build id: 3.10.0-327.el7.x86_64 		0x41
17
SyS_munmap
vmlinux 0xffffffff8119e772
Build id: 3.10.0-327.el7.x86_64 		0x22
18
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64 		0x16 
BUG: soft lockup - CPU#1 stuck for 21s! [abrt-watch-log:835]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmhgfs(OE) vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec coretemp ac97_bus crc32_pclmul ghash_clmulni_intel snd_seq snd_seq_device ppdev snd_pcm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_timer snd vmw_balloon soundcore pcspkr sg i2c_piix4 vmw_vmci shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi
crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw ttm drm ata_piix mptspi i2c_core scsi_transport_spi e1000 libata mptscsih mptbase
CPU: 1 PID: 835 Comm: abrt-watch-log Tainted: G           OEL ------------   3.10.0-327.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff880035cab980 ti: ffff88003b1a0000 task.ti: ffff88003b1a0000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88003b1a3c80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88003b1a3c50 RCX: 0000000000000001
RDX: ffffffff8165e7f0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88003b1a3cd0 R08: ffffffff8165e7e8 R09: 00007ff29423e000
R10: ffff88003b68f498 R11: 0000000000000202 R12: ffffffffa024f328
R13: ffff88003b1a3c28 R14: ffff880039fe4e00 R15: ffff88003ac7ea18
FS:  00007ff2941f2880(0000) GS:ffff88003d620000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff29423d1a4 CR3: 000000003b2eb000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff88003b1a3d58
0000000000000003 000000007f952ec5 0000000000000000 0000000000000000
ffffffff81065c90 ffffffff81065c90 ffff88003b1a3d00 ffffffff810e685f
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b2d2>] unmap_region+0xe2/0x130
[<ffffffff81197088>] ? handle_mm_fault+0x5b8/0xf50
[<ffffffff8119b831>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d612>] do_munmap+0x242/0x410
[<ffffffff8119d821>] vm_munmap+0x41/0x60
[<ffffffff8119e772>] SyS_munmap+0x22/0x30
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #40319
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f93aa
Build id: 3.10.0-514.6.2.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.6.2.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.6.2.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.6.2.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810f94af
Build id: 3.10.0-514.6.2.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81317b15
Build id: 3.10.0-514.6.2.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9a53
Build id: 3.10.0-514.6.2.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c5f8
Build id: 3.10.0-514.6.2.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c6c9
Build id: 3.10.0-514.6.2.el7.x86_64 		0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac1e3
Build id: 3.10.0-514.6.2.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad4d5
Build id: 3.10.0-514.6.2.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff811b43a4
Build id: 3.10.0-514.6.2.el7.x86_64 		0xf4
13
? vma_rb_erase
vmlinux 0xffffffff811b4921
Build id: 3.10.0-514.6.2.el7.x86_64 		0x121
14
do_munmap
vmlinux 0xffffffff811b6755
Build id: 3.10.0-514.6.2.el7.x86_64 		0x245
15
vm_munmap
vmlinux 0xffffffff811b6971
Build id: 3.10.0-514.6.2.el7.x86_64 		0x41
16
SyS_munmap
vmlinux 0xffffffff811b7902
Build id: 3.10.0-514.6.2.el7.x86_64 		0x22
17
system_call_fastpath
vmlinux 0xffffffff816968c9
Build id: 3.10.0-514.6.2.el7.x86_64 		0x16 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [vmtoolsd:1094]
Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) ext4 mbcache jbd2 snd_seq_midi snd_seq_midi_event intel_powerclamp coretemp iosf_mbi snd_ens1371 crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul snd_rawmidi glue_helper snd_ac97_codec ablk_helper cryptd ppdev ac97_bus snd_seq snd_seq_device vmw_balloon snd_pcm snd_timer pcspkr snd nfit soundcore
sg libnvdimm parport_pc parport i2c_piix4 shpchp vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm mptspi crc32c_intel scsi_transport_spi mptscsih ata_piix mptbase serio_raw libata vmxnet3 i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1094 Comm: vmtoolsd Tainted: G           OEL ------------   3.10.0-514.6.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8800ba4fce70 ti: ffff8800ac7f4000 task.ti: ffff8800ac7f4000
RIP: 0010:[<ffffffff810f93aa>]  [<ffffffff810f93aa>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8800ac7f7c80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8800ac7f7c50 RCX: 0000000000000001
RDX: ffffffff816af9b0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8800ac7f7cd0 R08: ffffffff816af9a8 R09: ffffffff811bc028
R10: ffff8800ba459a80 R11: ffffea0002c5bbc0 R12: ffff8800ac7f7c48
R13: 0000000000000000 R14: ffff880087c76400 R15: 0000000000000001
FS:  00007f9456c67740(0000) GS:ffff8800ba440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9456c88000 CR3: 00000000abaf3000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c430 ffff8800ac7f7d58
0000000000000003 000000001f79e24b ffff8800b121e440 0000000000000000
ffffffff8106c430 ffffffff8106c430 ffff8800ac7f7d00 ffffffff810f94af
Call Trace:
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff810f94af>] smp_call_function_single+0x5f/0xa0
[<ffffffff81317b15>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9a53>] smp_call_function_many+0x223/0x260
[<ffffffff8106c5f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c6c9>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac1e3>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad4d5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b43a4>] unmap_region+0xf4/0x140
[<ffffffff811b4921>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6755>] do_munmap+0x245/0x420
[<ffffffff811b6971>] vm_munmap+0x41/0x60
[<ffffffff811b7902>] SyS_munmap+0x22/0x30
[<ffffffff816968c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 80 57 ad 81 48 89 df e8 0a 29 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #18165
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64 		0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff8119b2d2
Build id: 3.10.0-327.el7.x86_64 		0xe2
13
? vma_rb_erase
vmlinux 0xffffffff8119b831
Build id: 3.10.0-327.el7.x86_64 		0x121
14
do_munmap
vmlinux 0xffffffff8119d612
Build id: 3.10.0-327.el7.x86_64 		0x242
15
vm_munmap
vmlinux 0xffffffff8119d821
Build id: 3.10.0-327.el7.x86_64 		0x41
16
SyS_munmap
vmlinux 0xffffffff8119e772
Build id: 3.10.0-327.el7.x86_64 		0x22
17
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64 		0x16 
BUG: soft lockup - CPU#3 stuck for 22s! [python2.7:11172]
Modules linked in: binfmt_misc fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ppdev sg virtio_balloon i2c_piix4 pcspkr parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_blk virtio_scsi cirrus syscopyarea sysfillrect sysimgblt drm_kms_helper ttm ata_piix drm serio_raw libata virtio_pci i2c_core virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 11172 Comm: python2.7 Tainted: G             L ------------   3.10.0-327.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff88060d11d080 ti: ffff88069ee54000 task.ti: ffff88069ee54000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88069ee57c80  EFLAGS: 00000202
RAX: 00000000000000a0 RBX: ffff88069ee57c50 RCX: 0000000000000020
RDX: 00000000ffffffff RSI: 00000000000000a0 RDI: 0000000000000282
RBP: ffff88069ee57cd0 R08: ffffffff8165ece8 R09: ffff88081e277520
R10: ffffffff811a2b52 R11: ffffea001f899740 R12: ffffffff811d15c2
R13: ffff88069ee57c38 R14: 0000000000000000 R15: ffff88080c622bc0
FS:  00007f41e37fe700(0000) GS:ffff88081e260000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f41ff422000 CR3: 00000005e6c2d000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff88069ee57d58
0000000000000003 00000000b72abd96 0000000000000000 0000000000000002
ffffffff81065c90 ffffffff81065c90 ffff88069ee57d00 ffffffff810e685f
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b2d2>] unmap_region+0xe2/0x130
[<ffffffff8119b831>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d612>] do_munmap+0x242/0x410
[<ffffffff8119d821>] vm_munmap+0x41/0x60
[<ffffffff8119e772>] SyS_munmap+0x22/0x30
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #14306
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f925a
Build id: 3.10.0-514.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810f935f
Build id: 3.10.0-514.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813179a5
Build id: 3.10.0-514.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9903
Build id: 3.10.0-514.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c4c8
Build id: 3.10.0-514.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c599
Build id: 3.10.0-514.el7.x86_64 		0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac093
Build id: 3.10.0-514.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad385
Build id: 3.10.0-514.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff811b4254
Build id: 3.10.0-514.el7.x86_64 		0xf4
13
? vma_rb_erase
vmlinux 0xffffffff811b47d1
Build id: 3.10.0-514.el7.x86_64 		0x121
14
do_munmap
vmlinux 0xffffffff811b6605
Build id: 3.10.0-514.el7.x86_64 		0x245
15
vm_munmap
vmlinux 0xffffffff811b6821
Build id: 3.10.0-514.el7.x86_64 		0x41
16
SyS_munmap
vmlinux 0xffffffff811b77b2
Build id: 3.10.0-514.el7.x86_64 		0x22
17
system_call_fastpath
vmlinux 0xffffffff816965c9
Build id: 3.10.0-514.el7.x86_64 		0x16 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [systemd:1]
Modules linked in: xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle overlay ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc vmw_vsock_vmci_transport vsock sunrpc nfit libnvdimm snd_seq_midi snd_seq_midi_event ppdev vmw_balloon intel_powerclamp snd_ens1371 snd_rawmidi snd_ac97_codec coretemp ac97_bus iosf_mbi snd_seq crc32_pclmul ghash_clmulni_intel aesni_intel lrw sg snd_seq_device gf128mul snd_pcm snd_timer snd soundcore glue_helper ablk_helper cryptd pcspkr shpchp i2c_piix4 parport_pc parport vmw_vmci ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic
pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel mptspi vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops scsi_transport_spi mptscsih ttm ata_piix mptbase drm e1000 libata i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 1 Comm: systemd Tainted: G        W    L ------------   3.10.0-514.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff880184db0000 ti: ffff880184db8000 task.ti: ffff880184db8000
RIP: 0010:[<ffffffff810f925a>]  [<ffffffff810f925a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff880184dbbc80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff880184dbbc50 RCX: 0000000000000002
RDX: ffffffff816afc30 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff880184dbbcd0 R08: ffffffff816afc28 R09: ffffffff811bbed8
R10: ffff8802356d9a80 R11: ffffea0007908680 R12: ffffea000091ae00
R13: ffff880187f46b80 R14: ffff880231778c80 R15: ffff880184db0000
FS:  00007f3bf5cff940(0000) GS:ffff8802356c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3bf5d27000 CR3: 0000000230e08000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c300 ffff880184dbbd58
0000000000000003 00000000c1823def ffff880231ff4938 0000000000000001
ffffffff8106c300 ffffffff8106c300 ffff880184dbbd00 ffffffff810f935f
Call Trace:
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff810f935f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813179a5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9903>] smp_call_function_many+0x223/0x260
[<ffffffff8106c4c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c599>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac093>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad385>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b4254>] unmap_region+0xf4/0x140
[<ffffffff811b47d1>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6605>] do_munmap+0x245/0x420
[<ffffffff811b6821>] vm_munmap+0x41/0x60
[<ffffffff811b77b2>] SyS_munmap+0x22/0x30
[<ffffffff816965c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 40 57 ad 81 48 89 df e8 ea 28 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #13019
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f93ba
Build id: 3.10.0-514.2.2.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.2.2.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.2.2.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.2.2.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810f94bf
Build id: 3.10.0-514.2.2.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81317b25
Build id: 3.10.0-514.2.2.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9a63
Build id: 3.10.0-514.2.2.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c5f8
Build id: 3.10.0-514.2.2.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c6c9
Build id: 3.10.0-514.2.2.el7.x86_64 		0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac1f3
Build id: 3.10.0-514.2.2.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad4e5
Build id: 3.10.0-514.2.2.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff811b43b4
Build id: 3.10.0-514.2.2.el7.x86_64 		0xf4
13
? mntput_no_expire
vmlinux 0xffffffff8121e93e
Build id: 3.10.0-514.2.2.el7.x86_64 		0x3e
14
? vma_rb_erase
vmlinux 0xffffffff811b4931
Build id: 3.10.0-514.2.2.el7.x86_64 		0x121
15
do_munmap
vmlinux 0xffffffff811b6765
Build id: 3.10.0-514.2.2.el7.x86_64 		0x245
16
vm_munmap
vmlinux 0xffffffff811b6981
Build id: 3.10.0-514.2.2.el7.x86_64 		0x41
17
SyS_munmap
vmlinux 0xffffffff811b7912
Build id: 3.10.0-514.2.2.el7.x86_64 		0x22
18
system_call_fastpath
vmlinux 0xffffffff816967c9
Build id: 3.10.0-514.2.2.el7.x86_64 		0x16 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [java:6347]
Modules linked in: binfmt_misc ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock intel_powerclamp snd_seq_midi snd_seq_midi_event coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw snd_ens1371 gf128mul snd_rawmidi glue_helper snd_ac97_codec ablk_helper cryptd ac97_bus ppdev snd_seq snd_seq_device snd_pcm vmw_balloon snd_timer snd soundcore pcspkr sg parport_pc parport
i2c_piix4 shpchp vmw_vmci nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx serio_raw drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm e1000 mptspi ata_piix scsi_transport_spi i2c_core mptscsih libata mptbase fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 6347 Comm: java Not tainted 3.10.0-514.2.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff88081c2cce70 ti: ffff8807ee650000 task.ti: ffff8807ee650000
RIP: 0010:[<ffffffff810f93ba>]  [<ffffffff810f93ba>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8807ee653c80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8807ee653c50 RCX: 0000000000000004
RDX: ffffffff816b0130 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8807ee653cd0 R08: ffffffff816b0128 R09: ffffffff811bc12c
R10: ffff88081d659a80 R11: ffffea000717e800 R12: ffff8807ee653c48
R13: 0000000000000000 R14: ffff880187c76400 R15: 0000000000000001
FS:  00007f848af49700(0000) GS:ffff88081d640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f849805e000 CR3: 00000008053d0000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c430 ffff8807ee653d58
0000000000000002 000000002376ca59 ffff8807fee832f0 0000000000000003
ffffffff8106c430 ffffffff8106c430 ffff8807ee653d00 ffffffff810f94bf
Call Trace:
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff810f94bf>] smp_call_function_single+0x5f/0xa0
[<ffffffff81317b25>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9a63>] smp_call_function_many+0x223/0x260
[<ffffffff8106c5f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c6c9>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac1f3>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad4e5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b43b4>] unmap_region+0xf4/0x140
[<ffffffff8121e93e>] ? mntput_no_expire+0x3e/0x120
[<ffffffff811b4931>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6765>] do_munmap+0x245/0x420
[<ffffffff811b6981>] vm_munmap+0x41/0x60
[<ffffffff811b7912>] SyS_munmap+0x22/0x30
[<ffffffff816967c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 80 57 ad 81 48 89 df e8 0a 29 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #4497
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64 		0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff8119b2d2
Build id: 3.10.0-327.el7.x86_64 		0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811fe6fe
Build id: 3.10.0-327.el7.x86_64 		0x3e
14
? vma_rb_erase
vmlinux 0xffffffff8119b831
Build id: 3.10.0-327.el7.x86_64 		0x121
15
do_munmap
vmlinux 0xffffffff8119d612
Build id: 3.10.0-327.el7.x86_64 		0x242
16
vm_munmap
vmlinux 0xffffffff8119d821
Build id: 3.10.0-327.el7.x86_64 		0x41
17
SyS_munmap
vmlinux 0xffffffff8119e772
Build id: 3.10.0-327.el7.x86_64 		0x22
18
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64 		0x16 
BUG: soft lockup - CPU#3 stuck for 23s! [java:4789]
Modules linked in: 8021q garp mrp binfmt_misc bridge stp llc snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer ppdev snd sg soundcore i2c_piix4 parport_pc pcspkr parport virtio_balloon xfs libcrc32c sr_mod cdrom ata_generic pata_acpi qxl syscopyarea sysfillrect sysimgblt virtio_blk drm_kms_helper virtio_console ttm ata_piix virtio_net drm libata serio_raw virtio_pci i2c_core virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 3 PID: 4789 Comm: java Tainted: G             L ------------   3.10.0-327.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS Bochs 01/01/2011
task: ffff8801235d4500 ti: ffff88003639c000 task.ti: ffff88003639c000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88003639fc80  EFLAGS: 00000202
RAX: 00000000000400f6 RBX: ffffffff81e99780 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000282
RBP: ffff88003639fcd0 R08: 0000000000000001 R09: ffff88043fd97520
R10: ffffffff811a2b52 R11: ffffea000acca900 R12: ffffffff811d15c2
R13: ffff88003639fc38 R14: 0000000000000000 R15: ffff8801350512c0
FS:  00007fb3f27ef700(0000) GS:ffff88043fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb405c58000 CR3: 0000000427374000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff88003639fd58
0000000000000003 0000000034738f9d 0000000000000000 0000000000000001
ffffffff81065c90 ffffffff81065c90 ffff88003639fd00 ffffffff810e685f
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b2d2>] unmap_region+0xe2/0x130
[<ffffffff811fe6fe>] ? mntput_no_expire+0x3e/0x120
[<ffffffff8119b831>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d612>] do_munmap+0x242/0x410
[<ffffffff8119d821>] vm_munmap+0x41/0x60
[<ffffffff8119e772>] SyS_munmap+0x22/0x30
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00
Complete report #141
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f925a
Build id: 3.10.0-514.el7.x86_64 		0xfa
2
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64 		0x70
3
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64 		0x70
4
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64 		0x70
5
smp_call_function_single
vmlinux 0xffffffff810f935f
Build id: 3.10.0-514.el7.x86_64 		0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813179a5
Build id: 3.10.0-514.el7.x86_64 		0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9903
Build id: 3.10.0-514.el7.x86_64 		0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c4c8
Build id: 3.10.0-514.el7.x86_64 		0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c599
Build id: 3.10.0-514.el7.x86_64 		0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac093
Build id: 3.10.0-514.el7.x86_64 		0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad385
Build id: 3.10.0-514.el7.x86_64 		0x55
12
unmap_region
vmlinux 0xffffffff811b4254
Build id: 3.10.0-514.el7.x86_64 		0xf4
13
? vma_rb_erase
vmlinux 0xffffffff811b47d1
Build id: 3.10.0-514.el7.x86_64 		0x121
14
do_munmap
vmlinux 0xffffffff811b6605
Build id: 3.10.0-514.el7.x86_64 		0x245
15
vm_munmap
vmlinux 0xffffffff811b6821
Build id: 3.10.0-514.el7.x86_64 		0x41
16
SyS_munmap
vmlinux 0xffffffff811b77b2
Build id: 3.10.0-514.el7.x86_64 		0x22
17
system_call_fastpath
vmlinux 0xffffffff816965c9
Build id: 3.10.0-514.el7.x86_64 		0x16 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [3_scheduler:3036]
Modules linked in: fuse uinput xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_hda_codec_generic iosf_mbi crc32_pclmul snd_hda_intel ghash_clmulni_intel aesni_intel snd_hda_codec lrw gf128mul glue_helper ablk_helper snd_hda_core snd_hwdep snd_seq ppdev snd_seq_device cryptd snd_pcm snd_timer snd pcspkr parport_pc virtio_balloon parport sg i2c_piix4 soundcore nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_blk virtio_net qxl drm_kms_helper crct10dif_pclmul syscopyarea crct10dif_common sysfillrect
crc32c_intel sysimgblt ata_piix fb_sys_fops serio_raw ttm drm virtio_pci virtio_ring virtio libata i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 3036 Comm: 3_scheduler Tainted: G             L ------------   3.10.0-514.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff880221633ec0 ti: ffff8802216b8000 task.ti: ffff8802216b8000
RIP: 0010:[<ffffffff810f925a>]  [<ffffffff810f925a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8802216bbc80  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff8802216bbc50 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff8802216bbcd0 R08: ffffffff816afc20 R09: ffffffff811bbed8
R10: ffff880239d19a80 R11: ffffea00068f92c0 R12: 00000000813179a5
R13: ffff880239fd6008 R14: ffff880184e7a000 R15: ffffffff8118a67a
FS:  00007f7d3effd700(0000) GS:ffff880239d00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7d3cb02630 CR3: 00000000b8f44000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c300 ffff8802216bbd58
0000000000000003 0000000061b58962 ffff8800bb309bf8 0000000000000001
ffffffff8106c300 ffffffff8106c300 ffff8802216bbd00 ffffffff810f935f
Call Trace:
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff810f935f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813179a5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9903>] smp_call_function_many+0x223/0x260
[<ffffffff8106c4c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c599>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac093>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad385>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b4254>] unmap_region+0xf4/0x140
[<ffffffff811b47d1>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6605>] do_munmap+0x245/0x420
[<ffffffff811b6821>] vm_munmap+0x41/0x60
[<ffffffff811b77b2>] SyS_munmap+0x22/0x30
[<ffffffff816965c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 40 57 ad 81 48 89 df e8 ea 28 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00