Info

Permalink
Function
generic_exec_single
First occurrence
2020-09-22
Last occurrence
2026-08-28
State
NEW
Components
kernel
Quality
-37

Statistics

Operating system Count
CentOS 7 594
Architecture Count
x86_64 594
Related packages Count
kernel 594
 0:3.10.0-514.16.1.el7 354
 0:3.10.0-327.el7 105
 0:3.10.0-514.el7 58
 0:3.10.0-229.14.1.el7 17
 0:3.10.0-957.el7 17
 0:3.10.0-327.36.2.el7 7
 0:3.10.0-327.13.1.el7 6
 0:3.10.0-514.10.2.el7 5
 0:3.10.0-229.20.1.el7 5
 0:3.10.0-327.22.2.el7 4
 0:3.10.0-514.21.1.el7 3
 0:3.10.0-514.2.2.el7 2
 0:3.10.0-1160.11.1.el7 2
 0:3.10.0-1160.118.1.el7 2
 0:3.10.0-514.6.2.el7 1
 0:3.10.0-229.el7 1
 0:3.10.0-1160.45.1.el7 1
 0:3.10.0-1160.66.1.el7 1
 0:3.10.0-1160.76.1.el7 1
 0:3.10.0-1160.119.1.el7 1
 0:3.10.0-1160.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1421200
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff83d1109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff83c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff83c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff83c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff83d111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff83f75465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff83d1175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff83c7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff83c7a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff83de4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff83de5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff83def014
Build id: 3.10.0-957.el7.x86_64
0xf4
13
? __vma_rb_erase
vmlinux 0xffffffff83def5d1
Build id: 3.10.0-957.el7.x86_64
0x121
14
do_munmap
vmlinux 0xffffffff83df1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
15
SyS_mremap
vmlinux 0xffffffff83df5d39
Build id: 3.10.0-957.el7.x86_64
0x4e9
16
system_call_fastpath
vmlinux 0xffffffff84374ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [abrt-action-gen:76020]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer
snd soundcore sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi e1000 vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi scsi_transport_spi mptscsih nfit mptbase drm_panel_orientation_quirks libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 76020 Comm: abrt-action-gen Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/17/2026
task: ffff8ae36669c100 ti: ffff8ae3337f0000 task.ti: ffff8ae3337f0000
RIP: 0010:[<ffffffff83d1109a>]  [<ffffffff83d1109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8ae3337f3be0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8ae3337f3bb0 RCX: 0000000000000001
RDX: ffffffff84412d90 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8ae3337f3c28 R08: ffffffff84412d88 R09: ffffffff83df8298
R10: ffff8ae3f96df0a0 R11: fffff44582532680 R12: 0000000000000000
R13: 0000000300000002 R14: ffff8ae3fffda000 R15: ffffffff83dc0432
FS:  00007f3320919740(0000) GS:ffff8ae3f96c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd361bcc000 CR3: 0000000073084000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff83c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff83c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff83c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff83d111af>] smp_call_function_single+0x5f/0xa0
[<ffffffff83f75465>] ? cpumask_next_and+0x35/0x50
[<ffffffff83d1175b>] smp_call_function_many+0x22b/0x270
[<ffffffff83c7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff83c7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff83de4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff83de5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff83def014>] unmap_region+0xf4/0x140
[<ffffffff83def5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff83df1605>] do_munmap+0x2a5/0x480
[<ffffffff83df5d39>] SyS_mremap+0x4e9/0x740
[<ffffffff84374ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 94 84 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1407289
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa5b1109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffa5a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffa5a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffa5a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffa5b111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa5d75465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa5b1175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa5a7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa5a7a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffa5be4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffa5be5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffffa5bef014
Build id: 3.10.0-957.el7.x86_64
0xf4
13
? selinux_file_free_security
vmlinux 0xffffffffa5cfc023
Build id: 3.10.0-957.el7.x86_64
0x23
14
? __vma_rb_erase
vmlinux 0xffffffffa5bef5d1
Build id: 3.10.0-957.el7.x86_64
0x121
15
do_munmap
vmlinux 0xffffffffa5bf1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
16
vm_munmap
vmlinux 0xffffffffa5bf1845
Build id: 3.10.0-957.el7.x86_64
0x65
17
SyS_munmap
vmlinux 0xffffffffa5bf2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
18
system_call_fastpath
vmlinux 0xffffffffa6174ddb
Build id: 3.10.0-957.el7.x86_64
0x22
19
? system_call_after_swapgs
vmlinux 0xffffffffa6174d21
Build id: 3.10.0-957.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [systemd:1]
Modules linked in: xt_multiport ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel veth xt_addrtype xt_set ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_bitmap_port ip_set_hash_ipport ip_set dummy iptable_raw xt_CT rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter sunrpc ppdev sg virtio_balloon parport_pc parport i2c_piix4 joydev pcspkr br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack
ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_blk virtio_net virtio_console virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm floppy serio_raw drm ata_piix libata virtio_pci virtio_ring drm_panel_orientation_quirks virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 1 Comm: systemd Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9d88e9438000 ti: ffff9d88e94b0000 task.ti: ffff9d88e94b0000
RIP: 0010:[<ffffffffa5b1109a>]  [<ffffffffa5b1109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9d88e94b3c30  EFLAGS: 00000202
RAX: 0000000000000030 RBX: ffff9d88e94b3c00 RCX: 0000000000000010
RDX: 0000ffffffffffff RSI: 0000000000000030 RDI: 0000000000000286
RBP: ffff9d88e94b3c78 R08: ffffffffa6215300 R09: ffffffffa5bf838c
R10: ffff9d977ebdf0a0 R11: fffff610a2ef9a40 R12: 80000007f601a867
R13: 80000007f601a867 R14: 0000000400000000 R15: 000000010defc168
FS:  00007f5483c85940(0000) GS:ffff9d977ebc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5483c96000 CR3: 0000000ff7b3e000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa5a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa5a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa5a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa5b111af>] smp_call_function_single+0x5f/0xa0
[<ffffffffa5d75465>] ? cpumask_next_and+0x35/0x50
[<ffffffffa5b1175b>] smp_call_function_many+0x22b/0x270
[<ffffffffa5a7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa5a7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa5be4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffa5be5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffa5bef014>] unmap_region+0xf4/0x140
[<ffffffffa5cfc023>] ? selinux_file_free_security+0x23/0x30
[<ffffffffa5bef5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffa5bf1605>] do_munmap+0x2a5/0x480
[<ffffffffa5bf1845>] vm_munmap+0x65/0xb0
[<ffffffffa5bf2ad2>] SyS_munmap+0x22/0x30
[<ffffffffa6174ddb>] system_call_fastpath+0x22/0x27
[<ffffffffa6174d21>] ? system_call_after_swapgs+0xae/0x146
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 74 a6 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1330882
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9bd1dba6
Build id: 3.10.0-1160.118.1.el7.x86_64
0x106
2
? leave_mm
vmlinux 0xffffffff9bc81b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffff9bc81b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffff9bc81b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffff9bd1dcbf
Build id: 3.10.0-1160.118.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9bf9c615
Build id: 3.10.0-1160.118.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9bd1e27b
Build id: 3.10.0-1160.118.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9bc81fa8
Build id: 3.10.0-1160.118.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9bc8201b
Build id: 3.10.0-1160.118.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9bdfa0ce
Build id: 3.10.0-1160.118.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffff9bdfbc4f
Build id: 3.10.0-1160.118.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff9bdfbd83
Build id: 3.10.0-1160.118.1.el7.x86_64
0x23
13
unmap_region
vmlinux 0xffffffff9be05904
Build id: 3.10.0-1160.118.1.el7.x86_64
0xf4
14
? selinux_file_free_security
vmlinux 0xffffffff9bf1f423
Build id: 3.10.0-1160.118.1.el7.x86_64
0x23
15
? __vma_rb_erase
vmlinux 0xffffffff9be05f11
Build id: 3.10.0-1160.118.1.el7.x86_64
0x121
16
do_munmap
vmlinux 0xffffffff9be07f65
Build id: 3.10.0-1160.118.1.el7.x86_64
0x2a5
17
vm_munmap
vmlinux 0xffffffff9be081a5
Build id: 3.10.0-1160.118.1.el7.x86_64
0x65
18
SyS_munmap
vmlinux 0xffffffff9be094d2
Build id: 3.10.0-1160.118.1.el7.x86_64
0x22
19
system_call_fastpath
vmlinux 0xffffffff9c3c539a
Build id: 3.10.0-1160.118.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [falcon-sensor:21154]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(E) falcon_kal(E) falcon_lsm_pinned_18108(E) falcon_lsm_pinned_18003(E) cts 8021q garp mrp ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache falcon_lsm_pinned_17804(E) nf_nat_tftp nf_conntrack_tftp nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter
overlay(T) tun vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi nfit libata scsi_transport_spi mptscsih libnvdimm mptbase crct10dif_pclmul crct10dif_common vmxnet3 crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_kal]
CPU: 3 PID: 21154 Comm: falcon-sensor Kdump: loaded Tainted: P            EL ------------ T 3.10.0-1160.118.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff90db6212d280 ti: ffff90dc82b38000 task.ti: ffff90dc82b38000
RIP: 0010:[<ffffffff9bd1dba6>]  [<ffffffff9bd1dba6>] generic_exec_single+0x106/0x1c0
RSP: 0018:ffff90dc82b3bc20  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff90dc82b3bbf0 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff90dc82b3bc70 R08: ffffffff9c4139c0 R09: ffffffff9be0f065
R10: ffff90dcffd9f0a0 R11: ffffcaf081e7f380 R12: 0000000000000001
R13: 0000000000015bc0 R14: ffffffff9c3bfb05 R15: ffff90dc82b3bba8
FS:  00007fcd225fe700(0000) GS:ffff90dcffd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcd328e2000 CR3: 00000001dcfcc000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9bc81b10>] ? leave_mm+0x130/0x130
[<ffffffff9bc81b10>] ? leave_mm+0x130/0x130
[<ffffffff9bc81b10>] ? leave_mm+0x130/0x130
[<ffffffff9bd1dcbf>] smp_call_function_single+0x5f/0xa0
[<ffffffff9bf9c615>] ? cpumask_next_and+0x35/0x50
[<ffffffff9bd1e27b>] smp_call_function_many+0x22b/0x270
[<ffffffff9bc81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9bc8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9bdfa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff9bdfbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9bdfbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffff9be05904>] unmap_region+0xf4/0x150
[<ffffffff9bf1f423>] ? selinux_file_free_security+0x23/0x30
[<ffffffff9be05f11>] ? __vma_rb_erase+0x121/0x230
[<ffffffff9be07f65>] do_munmap+0x2a5/0x480
[<ffffffff9be081a5>] vm_munmap+0x65/0xb0
[<ffffffff9be094d2>] SyS_munmap+0x22/0x30
[<ffffffff9c3c539a>] system_call_fastpath+0x25/0x2a
Code: 14 c5 c0 19 95 9c 48 89 df e8 b7 67 29 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 f6 43 20 01 <75> f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 00 0f 85 8f 
Complete report #1090911
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa971dba2
Build id: 3.10.0-1160.119.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffa9681b10
Build id: 3.10.0-1160.119.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffffa9681b10
Build id: 3.10.0-1160.119.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffffa9681b10
Build id: 3.10.0-1160.119.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffffa971dcbf
Build id: 3.10.0-1160.119.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa999c615
Build id: 3.10.0-1160.119.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa971e27b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa9681fa8
Build id: 3.10.0-1160.119.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa968201b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa97fa0ce
Build id: 3.10.0-1160.119.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffffa97fbc4f
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffa97fbd83
Build id: 3.10.0-1160.119.1.el7.x86_64
0x23
13
unmap_region
vmlinux 0xffffffffa9805904
Build id: 3.10.0-1160.119.1.el7.x86_64
0xf4
14
? selinux_file_free_security
vmlinux 0xffffffffa991f423
Build id: 3.10.0-1160.119.1.el7.x86_64
0x23
15
? __vma_rb_erase
vmlinux 0xffffffffa9805f11
Build id: 3.10.0-1160.119.1.el7.x86_64
0x121
16
do_munmap
vmlinux 0xffffffffa9807f65
Build id: 3.10.0-1160.119.1.el7.x86_64
0x2a5
17
vm_munmap
vmlinux 0xffffffffa98081a5
Build id: 3.10.0-1160.119.1.el7.x86_64
0x65
18
SyS_munmap
vmlinux 0xffffffffa98094d2
Build id: 3.10.0-1160.119.1.el7.x86_64
0x22
19
system_call_fastpath
vmlinux 0xffffffffa9dc539a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd:1]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) nf_nat_ftp nf_conntrack_ftp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event kvm_intel kvm irqbypass crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi ppdev aesni_intel lrw snd_ac97_codec vmw_balloon
gf128mul ac97_bus glue_helper ablk_helper snd_seq cryptd raid456 async_raid6_recov ext4 async_memcpy async_pq mbcache raid6_pq joydev async_xor jbd2 pcspkr xor async_tx snd_seq_device btusb btrtl snd_pcm btbcm btintel bluetooth snd_timer rfkill snd sg soundcore i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic vmwgfx pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi scsi_transport_spi mptscsih ata_piix nfit mptbase libata libnvdimm crct10dif_pclmul crct10dif_common e1000 crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 1 Comm: systemd Tainted: G               ------------ T 3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9606765b0000 ti: ffff9606765b8000 task.ti: ffff9606765b8000
RIP: 0010:[<ffffffffa971dba2>]  [<ffffffffa971dba2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9606765bbc20  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9606765bbbf0 RCX: 0000000000000001
RDX: ffffffffa9e139d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9606765bbc70 R08: ffffffffa9e139c8 R09: 000000018040003f
R10: 0000000000000001 R11: ffff9605b97fef40 R12: 0000000000000001
R13: ffff96067ffd9000 R14: 0000000000000000 R15: 03ffffff00000000
FS:  00007efd7a1ef940(0000) GS:ffff960679640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbddd8c70a0 CR3: 0000000035412000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffffa9681b10>] ? leave_mm+0x130/0x130
[<ffffffffa9681b10>] ? leave_mm+0x130/0x130
[<ffffffffa9681b10>] ? leave_mm+0x130/0x130
[<ffffffffa971dcbf>] smp_call_function_single+0x5f/0xa0
[<ffffffffa999c615>] ? cpumask_next_and+0x35/0x50
[<ffffffffa971e27b>] smp_call_function_many+0x22b/0x270
[<ffffffffa9681fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa968201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa97fa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa97fbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa97fbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffffa9805904>] unmap_region+0xf4/0x150
[<ffffffffa991f423>] ? selinux_file_free_security+0x23/0x30
[<ffffffffa9805f11>] ? __vma_rb_erase+0x121/0x230
[<ffffffffa9807f65>] do_munmap+0x2a5/0x480
[<ffffffffa98081a5>] vm_munmap+0x65/0xb0
[<ffffffffa98094d2>] SyS_munmap+0x22/0x30
[<ffffffffa9dc539a>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 c0 19 35 aa 48 89 df e8 b7 67 29 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #609250
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb1f16ae2
Build id: 3.10.0-1160.76.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffb1e7ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffb1e7ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffb1e7ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffb1f16bff
Build id: 3.10.0-1160.76.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb2186845
Build id: 3.10.0-1160.76.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb1f171ab
Build id: 3.10.0-1160.76.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb1e7e258
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb1e7e2cb
Build id: 3.10.0-1160.76.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb1fee6a6
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb1ff016f
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb1ff0293
Build id: 3.10.0-1160.76.1.el7.x86_64
0x23
13
unmap_region
vmlinux 0xffffffffb1ff9a84
Build id: 3.10.0-1160.76.1.el7.x86_64
0xf4
14
? selinux_file_free_security
vmlinux 0xffffffffb210c303
Build id: 3.10.0-1160.76.1.el7.x86_64
0x23
15
? __vma_rb_erase
vmlinux 0xffffffffb1ffa041
Build id: 3.10.0-1160.76.1.el7.x86_64
0x121
16
do_munmap
vmlinux 0xffffffffb1ffc025
Build id: 3.10.0-1160.76.1.el7.x86_64
0x2a5
17
vm_munmap
vmlinux 0xffffffffb1ffc265
Build id: 3.10.0-1160.76.1.el7.x86_64
0x65
18
SyS_munmap
vmlinux 0xffffffffb1ffd562
Build id: 3.10.0-1160.76.1.el7.x86_64
0x22
19
system_call_fastpath
vmlinux 0xffffffffb2599f92
Build id: 3.10.0-1160.76.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [systemd-journal:504]
Modules linked in: tcp_diag udp_diag inet_diag veth xt_nat xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq crc32_pclmul snd_seq_device snd_pcm ghash_clmulni_intel snd_timer aesni_intel lrw gf128mul ppdev vmw_balloon glue_helper snd ablk_helper cryptd soundcore sg vmw_vmci pcspkr joydev i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif
crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix nfit libata libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel mptspi scsi_transport_spi mptscsih serio_raw mptbase e1000 drm_panel_orientation_quirks fuse
CPU: 10 PID: 504 Comm: systemd-journal Kdump: loaded Tainted: G        W      ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9de8dbc58000 ti: ffff9de7010fc000 task.ti: ffff9de7010fc000
RIP: 0010:[<ffffffffb1f16ae2>]  [<ffffffffb1f16ae2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9de7010ffc20  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9de7010ffbf0 RCX: 0000000000000007
RDX: ffffffffb26146d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9de7010ffc70 R08: ffffffffb26146c8 R09: 000000018040003f
R10: 0000000000000001 R11: ffff9de8b8a75480 R12: ffff9de23f1e1920
R13: 0000000000000001 R14: ffff9de8db011400 R15: ffffd9bf80fc3240
FS:  00007f679ce04880(0000) GS:ffff9de8dd880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6a4d200000 CR3: 0000000816c26000 CR4: 0000000000340fe0
Call Trace:
[<ffffffffb1e7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb1e7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb1e7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb1f16bff>] smp_call_function_single+0x5f/0xa0
[<ffffffffb2186845>] ? cpumask_next_and+0x35/0x50
[<ffffffffb1f171ab>] smp_call_function_many+0x22b/0x270
[<ffffffffb1e7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb1e7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb1fee6a6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb1ff016f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb1ff0293>] tlb_finish_mmu+0x23/0x30
[<ffffffffb1ff9a84>] unmap_region+0xf4/0x140
[<ffffffffb210c303>] ? selinux_file_free_security+0x23/0x30
[<ffffffffb1ffa041>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb1ffc025>] do_munmap+0x2a5/0x480
[<ffffffffb1ffc265>] vm_munmap+0x65/0xb0
[<ffffffffb1ffd562>] SyS_munmap+0x22/0x30
[<ffffffffb2599f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 00 19 b5 b2 48 89 df e8 d7 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #484113
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa9b16b62
Build id: 3.10.0-1160.66.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffa9a7ddd0
Build id: 3.10.0-1160.66.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffa9a7ddd0
Build id: 3.10.0-1160.66.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffa9a7ddd0
Build id: 3.10.0-1160.66.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffa9b16c7f
Build id: 3.10.0-1160.66.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa9d86835
Build id: 3.10.0-1160.66.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa9b1722b
Build id: 3.10.0-1160.66.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa9a7e258
Build id: 3.10.0-1160.66.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa9a7e2cb
Build id: 3.10.0-1160.66.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa9bee676
Build id: 3.10.0-1160.66.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffa9bf013f
Build id: 3.10.0-1160.66.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffa9bf0263
Build id: 3.10.0-1160.66.1.el7.x86_64
0x23
13
unmap_region
vmlinux 0xffffffffa9bf9a54
Build id: 3.10.0-1160.66.1.el7.x86_64
0xf4
14
? selinux_file_free_security
vmlinux 0xffffffffa9d0c2f3
Build id: 3.10.0-1160.66.1.el7.x86_64
0x23
15
? __vma_rb_erase
vmlinux 0xffffffffa9bfa011
Build id: 3.10.0-1160.66.1.el7.x86_64
0x121
16
do_munmap
vmlinux 0xffffffffa9bfc045
Build id: 3.10.0-1160.66.1.el7.x86_64
0x2a5
17
vm_munmap
vmlinux 0xffffffffa9bfc285
Build id: 3.10.0-1160.66.1.el7.x86_64
0x65
18
SyS_munmap
vmlinux 0xffffffffa9bfd582
Build id: 3.10.0-1160.66.1.el7.x86_64
0x22
19
system_call_fastpath
vmlinux 0xffffffffaa199f92
Build id: 3.10.0-1160.66.1.el7.x86_64
0x25
20
? system_call_after_swapgs
vmlinux 0xffffffffaa199ed5
Build id: 3.10.0-1160.66.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [Catalina-utilit:4219]
Modules linked in: nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) vboxsf(OE) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod ext4 mbcache jbd2 snd_intel8x0 vboxvideo(OE) snd_ac97_codec ac97_bus joydev ppdev snd_seq snd_seq_device ttm snd_pcm drm_kms_helper pcspkr syscopyarea snd_timer sysfillrect sg snd
sysimgblt fb_sys_fops vboxguest(OE) drm parport_pc soundcore i2c_piix4 drm_panel_orientation_quirks video parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi ahci libahci ata_piix crc32c_intel serio_raw e1000 libata fuse
CPU: 1 PID: 4219 Comm: Catalina-utilit Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-1160.66.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff890242e00000 ti: ffff890242fac000 task.ti: ffff890242fac000
RIP: 0010:[<ffffffffa9b16b62>]  [<ffffffffa9b16b62>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff890242fafc20  EFLAGS: 00010202
RAX: 00000000000008fb RBX: 0000000000000001 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000282
RBP: ffff890242fafc70 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff8902ccbbbdd8 R11: 0000000000000206 R12: ffff8902d6fd7000
R13: 0000014100000000 R14: ffff8902d6fd7008 R15: ffff890242e00000
FS:  00007fbe336f6700(0000) GS:ffff8902d6d00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbe49839858 CR3: 0000000209020000 CR4: 00000000000206e0
Call Trace:
[<ffffffffa9a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa9a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa9a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa9b16c7f>] smp_call_function_single+0x5f/0xa0
[<ffffffffa9d86835>] ? cpumask_next_and+0x35/0x50
[<ffffffffa9b1722b>] smp_call_function_many+0x22b/0x270
[<ffffffffa9a7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa9a7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa9bee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa9bf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa9bf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffa9bf9a54>] unmap_region+0xf4/0x140
[<ffffffffa9d0c2f3>] ? selinux_file_free_security+0x23/0x30
[<ffffffffa9bfa011>] ? __vma_rb_erase+0x121/0x220
[<ffffffffa9bfc045>] do_munmap+0x2a5/0x480
[<ffffffffa9bfc285>] vm_munmap+0x65/0xb0
[<ffffffffa9bfd582>] SyS_munmap+0x22/0x30
[<ffffffffaa199f92>] system_call_fastpath+0x25/0x2a
[<ffffffffaa199ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 60 18 75 aa 48 89 df e8 47 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #314250
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff81d16a32
Build id: 3.10.0-1160.45.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff81c7ddd0
Build id: 3.10.0-1160.45.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff81c7ddd0
Build id: 3.10.0-1160.45.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff81c7ddd0
Build id: 3.10.0-1160.45.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff81d16b4f
Build id: 3.10.0-1160.45.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81f86a45
Build id: 3.10.0-1160.45.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff81d170fb
Build id: 3.10.0-1160.45.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff81c7e258
Build id: 3.10.0-1160.45.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81c7e2cb
Build id: 3.10.0-1160.45.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff81dee676
Build id: 3.10.0-1160.45.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff81df013f
Build id: 3.10.0-1160.45.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff81df0263
Build id: 3.10.0-1160.45.1.el7.x86_64
0x23
13
unmap_region
vmlinux 0xffffffff81df9a54
Build id: 3.10.0-1160.45.1.el7.x86_64
0xf4
14
? __wake_up_common_lock
vmlinux 0xffffffff81cd31f0
Build id: 3.10.0-1160.45.1.el7.x86_64
0x90
15
? selinux_file_free_security
vmlinux 0xffffffff81f0c493
Build id: 3.10.0-1160.45.1.el7.x86_64
0x23
16
? __vma_rb_erase
vmlinux 0xffffffff81dfa011
Build id: 3.10.0-1160.45.1.el7.x86_64
0x121
17
do_munmap
vmlinux 0xffffffff81dfc045
Build id: 3.10.0-1160.45.1.el7.x86_64
0x2a5
18
vm_munmap
vmlinux 0xffffffff81dfc285
Build id: 3.10.0-1160.45.1.el7.x86_64
0x65
19
SyS_munmap
vmlinux 0xffffffff81dfd582
Build id: 3.10.0-1160.45.1.el7.x86_64
0x22
20
system_call_fastpath
vmlinux 0xffffffff82395f92
Build id: 3.10.0-1160.45.1.el7.x86_64
0x25
21
? system_call_after_swapgs
vmlinux 0xffffffff82395ed5
Build id: 3.10.0-1160.45.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [grep:35527]
Modules linked in: binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix serio_raw libata mptspi scsi_transport_spi mptscsih vmxnet3 mptbase nfit drm_panel_orientation_quirks libnvdimm floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 35527 Comm: grep Kdump: loaded Tainted: G             L ------------   3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8a85f809d280 ti: ffff8a88c189c000 task.ti: ffff8a88c189c000
RIP: 0010:[<ffffffff81d16a32>]  [<ffffffff81d16a32>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8a88c189fc20  EFLAGS: 00000202
RAX: 0000000000000040 RBX: ffffffff81cd7422 RCX: 0000000000000002
RDX: 0000000000000040 RSI: 0000000000000000 RDI: 0000000000000286
RBP: ffff8a88c189fc70 R08: ffffffff82413a48 R09: ffffffff81e0307c
R10: ffff8a896e69f0a0 R11: ffffc8fa4c693b40 R12: 0000000000000002
R13: ffff8a896e69acc0 R14: ffff8a896e69acc0 R15: ffff8a85f809e300
FS:  00007f9b52a23740(0000) GS:ffff8a896e680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9b5225bb30 CR3: 00000003ecbf6000 CR4: 00000000000607e0
Call Trace:
[<ffffffff81c7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff81c7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff81c7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff81d16b4f>] smp_call_function_single+0x5f/0xa0
[<ffffffff81f86a45>] ? cpumask_next_and+0x35/0x50
[<ffffffff81d170fb>] smp_call_function_many+0x22b/0x270
[<ffffffff81c7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81c7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff81dee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff81df013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff81df0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff81df9a54>] unmap_region+0xf4/0x140
[<ffffffff81cd31f0>] ? __wake_up_common_lock+0x90/0xc0
[<ffffffff81f0c493>] ? selinux_file_free_security+0x23/0x30
[<ffffffff81dfa011>] ? __vma_rb_erase+0x121/0x220
[<ffffffff81dfc045>] do_munmap+0x2a5/0x480
[<ffffffff81dfc285>] vm_munmap+0x65/0xb0
[<ffffffff81dfd582>] SyS_munmap+0x22/0x30
[<ffffffff82395f92>] system_call_fastpath+0x25/0x2a
[<ffffffff82395ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 60 15 95 82 48 89 df e8 87 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #84867
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff98116802
Build id: 3.10.0-1160.11.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff9807e6a0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9807e6a0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9807e6a0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff9811691f
Build id: 3.10.0-1160.11.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff98385fe5
Build id: 3.10.0-1160.11.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff98116ecb
Build id: 3.10.0-1160.11.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9807e868
Build id: 3.10.0-1160.11.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9807e8db
Build id: 3.10.0-1160.11.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff981edee6
Build id: 3.10.0-1160.11.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff981ef9af
Build id: 3.10.0-1160.11.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff981efad3
Build id: 3.10.0-1160.11.1.el7.x86_64
0x23
13
unmap_region
vmlinux 0xffffffff981f92c4
Build id: 3.10.0-1160.11.1.el7.x86_64
0xf4
14
? kfree
vmlinux 0xffffffff982262e6
Build id: 3.10.0-1160.11.1.el7.x86_64
0x106
15
? selinux_file_free_security
vmlinux 0xffffffff9830bcc3
Build id: 3.10.0-1160.11.1.el7.x86_64
0x23
16
? __vma_rb_erase
vmlinux 0xffffffff981f9881
Build id: 3.10.0-1160.11.1.el7.x86_64
0x121
17
do_munmap
vmlinux 0xffffffff981fb8b5
Build id: 3.10.0-1160.11.1.el7.x86_64
0x2a5
18
vm_munmap
vmlinux 0xffffffff981fbaf5
Build id: 3.10.0-1160.11.1.el7.x86_64
0x65
19
SyS_munmap
vmlinux 0xffffffff981fcd92
Build id: 3.10.0-1160.11.1.el7.x86_64
0x22
20
system_call_fastpath
vmlinux 0xffffffff98793f92
Build id: 3.10.0-1160.11.1.el7.x86_64
0x25
21
? system_call_after_swapgs
vmlinux 0xffffffff98793ed5
Build id: 3.10.0-1160.11.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [wazuh-modulesd:9893]
crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm crc32c_intel ata_piix serio_raw mptspi scsi_transport_spi libata mptscsih e1000 mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 9893 Comm: wazuh-modulesd Kdump: loaded Tainted: P        W  OEL ------------   3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff9cf0f52de300 ti: ffff9cf1c7dc8000 task.ti: ffff9cf1c7dc8000
RIP: 0010:[<ffffffff98116802>]  [<ffffffff98116802>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9cf1c7dcbc20  EFLAGS: 00000202
RAX: 00000000000008fd RBX: 0000000000000001 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000282
RBP: ffff9cf1c7dcbc70 R08: 0000000000000001 R09: ffffffff9820288c
R10: ffff9cf2ffc1f0a0 R11: fffff61d059fa080 R12: fffff61d07912040
R13: 0000000100000001 R14: ffff9cf0f52de300 R15: 0000000000000000
FS:  00007fe9da7fc700(0000) GS:ffff9cf2ffc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe9e2494000 CR3: 000000005fb4a000 CR4: 00000000000407f0
Call Trace:
[<ffffffff9807e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9807e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9807e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9811691f>] smp_call_function_single+0x5f/0xa0
[<ffffffff98385fe5>] ? cpumask_next_and+0x35/0x50
[<ffffffff98116ecb>] smp_call_function_many+0x22b/0x270
[<ffffffff9807e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9807e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff981edee6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff981ef9af>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff981efad3>] tlb_finish_mmu+0x23/0x30
[<ffffffff981f92c4>] unmap_region+0xf4/0x140
[<ffffffff982262e6>] ? kfree+0x106/0x140
[<ffffffff9830bcc3>] ? selinux_file_free_security+0x23/0x30
[<ffffffff981f9881>] ? __vma_rb_erase+0x121/0x220
[<ffffffff981fb8b5>] do_munmap+0x2a5/0x480
[<ffffffff981fbaf5>] vm_munmap+0x65/0xb0
[<ffffffff981fcd92>] SyS_munmap+0x22/0x30
[<ffffffff98793f92>] system_call_fastpath+0x25/0x2a
[<ffffffff98793ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 e0 13 d5 98 48 89 df e8 57 69 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #994225
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f93ba
Build id: 3.10.0-514.2.2.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.2.2.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.2.2.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.2.2.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f94bf
Build id: 3.10.0-514.2.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81317b25
Build id: 3.10.0-514.2.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9a63
Build id: 3.10.0-514.2.2.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c5f8
Build id: 3.10.0-514.2.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c6c9
Build id: 3.10.0-514.2.2.el7.x86_64
0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac1f3
Build id: 3.10.0-514.2.2.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad4e5
Build id: 3.10.0-514.2.2.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff811b43b4
Build id: 3.10.0-514.2.2.el7.x86_64
0xf4
13
? __schedule
vmlinux 0xffffffff8168b290
Build id: 3.10.0-514.2.2.el7.x86_64
0x3b0
14
? vma_rb_erase
vmlinux 0xffffffff811b4931
Build id: 3.10.0-514.2.2.el7.x86_64
0x121
15
do_munmap
vmlinux 0xffffffff811b6765
Build id: 3.10.0-514.2.2.el7.x86_64
0x245
16
vm_munmap
vmlinux 0xffffffff811b6981
Build id: 3.10.0-514.2.2.el7.x86_64
0x41
17
SyS_munmap
vmlinux 0xffffffff811b7912
Build id: 3.10.0-514.2.2.el7.x86_64
0x22
18
system_call_fastpath
vmlinux 0xffffffff816967c9
Build id: 3.10.0-514.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [systemd-journal:534]
Modules linked in: tcp_lp isofs fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock intel_powerclamp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr sg vmw_balloon vmw_vmci parport_pc parport shpchp i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect crct10dif_common sysimgblt fb_sys_fops crc32c_intel serio_raw ttm mptspi scsi_transport_spi ata_piix drm mptscsih libata vmxnet3 mptbase i2c_core floppy fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 534 Comm: systemd-journal Tainted: G             L ------------   3.10.0-514.2.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff880232da4e70 ti: ffff88023fa04000 task.ti: ffff88023fa04000
RIP: 0010:[<ffffffff810f93ba>]  [<ffffffff810f93ba>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88023fa07c80  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff88023fa07c50 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff88023fa07cd0 R08: ffffffff816afea0 R09: 0000000180400015
R10: 0000000027e21c01 R11: ffffea00089f8840 R12: ffff880184ca0400
R13: 0000000000000001 R14: ffffea0001c72d00 R15: ffff88018779cb40
FS:  00007f18a456c880(0000) GS:ffff88023fcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000010f67c8 CR3: 0000000232a9f000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c430 ffff88023fa07d58
0000000000000003 000000008633dcbf 0000000100014c30 0000000000000002
ffffffff8106c430 ffffffff8106c430 ffff88023fa07d00 ffffffff810f94bf
Call Trace:
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff810f94bf>] smp_call_function_single+0x5f/0xa0
[<ffffffff81317b25>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9a63>] smp_call_function_many+0x223/0x260
[<ffffffff8106c5f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c6c9>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac1f3>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad4e5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b43b4>] unmap_region+0xf4/0x140
[<ffffffff8168b290>] ? __schedule+0x3b0/0x990
[<ffffffff811b4931>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6765>] do_munmap+0x245/0x420
[<ffffffff811b6981>] vm_munmap+0x41/0x60
[<ffffffff811b7912>] SyS_munmap+0x22/0x30
[<ffffffff816967c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 80 57 ad 81 48 89 df e8 0a 29 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #913896
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810d66ea
Build id: 3.10.0-229.20.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.20.1.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.20.1.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.20.1.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810d67ef
Build id: 3.10.0-229.20.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812d4745
Build id: 3.10.0-229.20.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810d6d83
Build id: 3.10.0-229.20.1.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8105fc78
Build id: 3.10.0-229.20.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8105fd3c
Build id: 3.10.0-229.20.1.el7.x86_64
0x5c
10
tlb_flush_mmu.part.53
vmlinux 0xffffffff8117e063
Build id: 3.10.0-229.20.1.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff8117ebc5
Build id: 3.10.0-229.20.1.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff811866b2
Build id: 3.10.0-229.20.1.el7.x86_64
0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811e67be
Build id: 3.10.0-229.20.1.el7.x86_64
0x3e
14
? vma_rb_erase
vmlinux 0xffffffff81186c11
Build id: 3.10.0-229.20.1.el7.x86_64
0x121
15
do_munmap
vmlinux 0xffffffff81188a42
Build id: 3.10.0-229.20.1.el7.x86_64
0x242
16
vm_munmap
vmlinux 0xffffffff81188c51
Build id: 3.10.0-229.20.1.el7.x86_64
0x41
17
SyS_munmap
vmlinux 0xffffffff81189b82
Build id: 3.10.0-229.20.1.el7.x86_64
0x22
18
system_call_fastpath
vmlinux 0xffffffff81614409
Build id: 3.10.0-229.20.1.el7.x86_64
0x16
BUG: soft lockup - CPU#0 stuck for 23s! [systemd-journal:508]
BUG: soft lockup - CPU#1 stuck for 22s! [safe_timer:1423]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6table_filter ip6_tables iptable_filter tcp_diag udp_diag inet_diag onload(OF) sfc_char(OF) sfc_resource(OF) sfc_affinity(OF) fuse sfc(OF) mtd ptp pps_core mdio onload_cplane(POF) crc32_pclmul iTCO_wdt iTCO_vendor_support ghash_clmulni_intel aesni_intel lrw pcspkr sg gf128mul glue_helper ablk_helper cryptd lpc_ich mfd_core i2c_i801 shpchp nfsd nfs_acl lockd auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ahci cirrus crct10dif_pclmul crct10dif_common syscopyarea crc32c_intel libahci sysfillrect sysimgblt serio_raw drm_kms_helper libata virtio_pci virtio_ring virtio ttm e1000 drm i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1423 Comm: safe_timer Tainted: PF       W  O--------------   3.10.0-229.20.1.el7.x86_64 #1
Hardware name: RDO OpenStack Compute/RHEL-AV, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8804662e8000 ti: ffff8804662e4000 task.ti: ffff8804662e4000
RIP: 0010:[<ffffffff810d66ea>]  [<ffffffff810d66ea>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8804662e7c80  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff8804662e7c50 RCX: 0000000000000830
RDX: 0000000000000004 RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff8804662e7cd0 R08: ffffffff816303a0 R09: ffff88047fc963e0
R10: ffffffff8118deb5 R11: ffffea000c6eb340 R12: ffff8804662e7c10
R13: ffff8804671e79c4 R14: ffff88047fc93680 R15: ffff8804671e71c0
FS:  00007fe77d763700(0000) GS:ffff88047fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe7952082c0 CR3: 000000046695f000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8105fab0 ffff8804662e7d58
0000000000000003 00000000b151a82a 0000000000000000 0000000000000003
ffffffff8105fab0 ffffffff8105fab0 ffff8804662e7d00 ffffffff810d67ef
Call Trace:
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff810d67ef>] smp_call_function_single+0x5f/0xa0
[<ffffffff812d4745>] ? cpumask_next_and+0x35/0x50
[<ffffffff810d6d83>] smp_call_function_many+0x223/0x260
[<ffffffff8105fc78>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8105fd3c>] flush_tlb_mm_range+0x5c/0x180
[<ffffffff8117e063>] tlb_flush_mmu.part.53+0x33/0x90
[<ffffffff8117ebc5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811866b2>] unmap_region+0xe2/0x130
[<ffffffff811e67be>] ? mntput_no_expire+0x3e/0x120
[<ffffffff81186c11>] ? vma_rb_erase+0x121/0x220
[<ffffffff81188a42>] do_munmap+0x242/0x410
[<ffffffff81188c51>] vm_munmap+0x41/0x60
[<ffffffff81189b82>] SyS_munmap+0x22/0x30
[<ffffffff81614409>] system_call_fastpath+0x16/0x1b
Code: 40 41 01 00 48 89 de 48 03 14 c5 80 4d a1 81 48 89 df e8 7a 13 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #634864
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64
0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff8119b2d2
Build id: 3.10.0-327.el7.x86_64
0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811fe6fe
Build id: 3.10.0-327.el7.x86_64
0x3e
14
? vma_rb_erase
vmlinux 0xffffffff8119b831
Build id: 3.10.0-327.el7.x86_64
0x121
15
do_munmap
vmlinux 0xffffffff8119d612
Build id: 3.10.0-327.el7.x86_64
0x242
16
vm_munmap
vmlinux 0xffffffff8119d821
Build id: 3.10.0-327.el7.x86_64
0x41
17
SyS_munmap
vmlinux 0xffffffff8119e772
Build id: 3.10.0-327.el7.x86_64
0x22
18
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64
0x16
BUG: soft lockup - CPU#3 stuck for 37s! [ftdc:2342]
Modules linked in: gsch(OE) redirfs(OE) dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop bridge stp llc ip_set nfnetlink dsa_filter(POE) vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr sg shpchp vmw_vmci parport_pc parport i2c_piix4 xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx serio_raw drm_kms_helper ttm mptsas scsi_transport_sas drm mptscsih mptbase ata_piix vmxnet3 libata i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 3 PID: 2342 Comm: ftdc Tainted: P           OEL ------------   3.10.0-327.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff88042703a280 ti: ffff8803fcad8000 task.ti: ffff8803fcad8000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8803fcadbc80  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff8803fcadbc50 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff8803fcadbcd0 R08: ffffffff8165ea60 R09: ffff88043fd97520
R10: ffffffff811a2bc5 R11: ffffea000e2f8500 R12: ffff880036acd780
R13: ffff88042703a280 R14: ffff8803fcadbc40 R15: ffff88042703a280
FS:  00007ff10d5e6700(0000) GS:ffff88043fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff1195d0000 CR3: 0000000406b57000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff8803fcadbd58
0000000000000002 0000000082570b9a 0000000000000000 0000000000000001
ffffffff81065c90 ffffffff81065c90 ffff8803fcadbd00 ffffffff810e685f
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b2d2>] unmap_region+0xe2/0x130
[<ffffffff811fe6fe>] ? mntput_no_expire+0x3e/0x120
[<ffffffff8119b831>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d612>] do_munmap+0x242/0x410
[<ffffffff8119d821>] vm_munmap+0x41/0x60
[<ffffffff8119e772>] SyS_munmap+0x22/0x30
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #355560
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810d66ea
Build id: 3.10.0-229.14.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.14.1.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.14.1.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.14.1.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810d67ef
Build id: 3.10.0-229.14.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812d4695
Build id: 3.10.0-229.14.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810d6d83
Build id: 3.10.0-229.14.1.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8105fc78
Build id: 3.10.0-229.14.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8105fd3c
Build id: 3.10.0-229.14.1.el7.x86_64
0x5c
10
tlb_flush_mmu.part.53
vmlinux 0xffffffff8117e063
Build id: 3.10.0-229.14.1.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff8117ebc5
Build id: 3.10.0-229.14.1.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff811866b2
Build id: 3.10.0-229.14.1.el7.x86_64
0xe2
13
? vma_rb_erase
vmlinux 0xffffffff81186c11
Build id: 3.10.0-229.14.1.el7.x86_64
0x121
14
do_munmap
vmlinux 0xffffffff81188a42
Build id: 3.10.0-229.14.1.el7.x86_64
0x242
15
vm_munmap
vmlinux 0xffffffff81188c51
Build id: 3.10.0-229.14.1.el7.x86_64
0x41
16
SyS_munmap
vmlinux 0xffffffff81189b82
Build id: 3.10.0-229.14.1.el7.x86_64
0x22
17
system_call_fastpath
vmlinux 0xffffffff81614389
Build id: 3.10.0-229.14.1.el7.x86_64
0x16
BUG: soft lockup - CPU#4 stuck for 22s! [litespeed:25780]
Modules linked in: binfmt_misc tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables dahdi(OF) crc_ccitt netatop(OF) xt_CHECKSUM iptable_mangle xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_nat_ipv4 nf_nat nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_recent xt_multiport xt_mark xt_conntrack nf_conntrack iptable_filter coretemp ext4 mbcache jbd2 snd_ens1371 snd_rawmidi snd_ac97_codec crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq snd_seq_device aesni_intel snd_pcm lrw gf128mul snd_timer ppdev glue_helper snd ablk_helper cryptd pcspkr vmw_balloon soundcore ahci libahci sg vmw_vmci i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif
crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel ata_piix ttm mptspi serio_raw scsi_transport_spi drm mptscsih libata e1000 mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 25780 Comm: litespeed Tainted: GF          O--------------   3.10.0-229.14.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff880271f60b60 ti: ffff88012153c000 task.ti: ffff88012153c000
RIP: 0010:[<ffffffff810d66ea>]  [<ffffffff810d66ea>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88012153fc80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88012153fc50 RCX: 0000000000000001
RDX: ffffffff8162fc30 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88012153fcd0 R08: ffffffff8162fc28 R09: ffff8803326963e0
R10: ffffffff8118de42 R11: ffffea00050aec80 R12: 0000000100000000
R13: 00000000814ecf31 R14: ffffffff81553b4d R15: ffff88012153fc40
FS:  00007f5a87f25640(0000) GS:ffff880332680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5a87f17ff0 CR3: 000000015826e000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8105fab0 ffff88012153fd58
0000000000000003 0000000038b9c549 0000000000000000 0000000000000000
ffffffff8105fab0 ffffffff8105fab0 ffff88012153fd00 ffffffff810d67ef
Call Trace:
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff810d67ef>] smp_call_function_single+0x5f/0xa0
[<ffffffff812d4695>] ? cpumask_next_and+0x35/0x50
[<ffffffff810d6d83>] smp_call_function_many+0x223/0x260
[<ffffffff8105fc78>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8105fd3c>] flush_tlb_mm_range+0x5c/0x180
[<ffffffff8117e063>] tlb_flush_mmu.part.53+0x33/0x90
[<ffffffff8117ebc5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811866b2>] unmap_region+0xe2/0x130
[<ffffffff81186c11>] ? vma_rb_erase+0x121/0x220
[<ffffffff81188a42>] do_munmap+0x242/0x410
[<ffffffff81188c51>] vm_munmap+0x41/0x60
[<ffffffff81189b82>] SyS_munmap+0x22/0x30
[<ffffffff81614389>] system_call_fastpath+0x16/0x1b
Code: 40 41 01 00 48 89 de 48 03 14 c5 00 4d a1 81 48 89 df e8 ba 12 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #319942
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e687a
Build id: 3.10.0-327.13.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff81065c80
Build id: 3.10.0-327.13.1.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff81065c80
Build id: 3.10.0-327.13.1.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff81065c80
Build id: 3.10.0-327.13.1.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810e697f
Build id: 3.10.0-327.13.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f28f5
Build id: 3.10.0-327.13.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6f23
Build id: 3.10.0-327.13.1.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e48
Build id: 3.10.0-327.13.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f16
Build id: 3.10.0-327.13.1.el7.x86_64
0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192663
Build id: 3.10.0-327.13.1.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811931f5
Build id: 3.10.0-327.13.1.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff8119b3d2
Build id: 3.10.0-327.13.1.el7.x86_64
0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811fe7ce
Build id: 3.10.0-327.13.1.el7.x86_64
0x3e
14
? vma_rb_erase
vmlinux 0xffffffff8119b931
Build id: 3.10.0-327.13.1.el7.x86_64
0x121
15
do_munmap
vmlinux 0xffffffff8119d712
Build id: 3.10.0-327.13.1.el7.x86_64
0x242
16
vm_munmap
vmlinux 0xffffffff8119d921
Build id: 3.10.0-327.13.1.el7.x86_64
0x41
17
SyS_munmap
vmlinux 0xffffffff8119e872
Build id: 3.10.0-327.13.1.el7.x86_64
0x22
18
system_call_fastpath
vmlinux 0xffffffff81645ec9
Build id: 3.10.0-327.13.1.el7.x86_64
0x16
BUG: soft lockup - CPU#6 stuck for 22s! [ir_agent:22470]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_12904(E) falcon_lsm_pinned_12803(E) falcon_lsm_pinned_12704(E) falcon_lsm_pinned_12606(E) falcon_lsm_pinned_10807(E) dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag ip6t_REJECT nf_log_ipv6 nf_log_ipv4 nf_log_common ip6table_filter nf_nat_ftp xt_REDIRECT xt_conntrack iptable_mangle nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables ipt_REJECT iptable_filter dm_mirror dm_region_hash dm_log dm_mod crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr sg parport_pc parport shpchp vmw_vmci
i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper ttm mptspi serio_raw drm scsi_transport_spi ata_piix mptscsih e1000 libata mptbase i2c_core floppy [last unloaded: falcon_kal]
CPU: 6 PID: 22470 Comm: ir_agent Tainted: P        W   EL ------------   3.10.0-327.13.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8800b3632280 ti: ffff880035374000 task.ti: ffff880035374000
RIP: 0010:[<ffffffff810e687a>]  [<ffffffff810e687a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff880035377c80  EFLAGS: 00000202
RAX: 0000000000000020 RBX: ffff8800b3632280 RCX: 0000000000000020
RDX: 0000000000000020 RSI: 0000000000000020 RDI: 0000000000000286
RBP: ffff880035377cd0 R08: ffffffff8165e7e0 R09: ffff8807a12d7520
R10: ffffffff811a2cb5 R11: ffffea001e37ebc0 R12: 0000000000000000
R13: ffffffff816413dd R14: ffff880035377c40 R15: 0000000000000029
FS:  00007fa5cb7fe700(0000) GS:ffff8807a12c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa5f4cf2000 CR3: 000000078f424000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c80 ffff880035377d58
0000000000000003 0000000037c75518 0000000000000000 0000000000000000
ffffffff81065c80 ffffffff81065c80 ffff880035377d00 ffffffff810e697f
Call Trace:
[<ffffffff81065c80>] ? leave_mm+0x70/0x70
[<ffffffff81065c80>] ? leave_mm+0x70/0x70
[<ffffffff81065c80>] ? leave_mm+0x70/0x70
[<ffffffff810e697f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f28f5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6f23>] smp_call_function_many+0x223/0x260
[<ffffffff81065e48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f16>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192663>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811931f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b3d2>] unmap_region+0xe2/0x130
[<ffffffff811fe7ce>] ? mntput_no_expire+0x3e/0x120
[<ffffffff8119b931>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d712>] do_munmap+0x242/0x410
[<ffffffff8119d921>] vm_munmap+0x41/0x60
[<ffffffff8119e872>] SyS_munmap+0x22/0x30
[<ffffffff81645ec9>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 2a fd 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #209811
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f925a
Build id: 3.10.0-514.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f935f
Build id: 3.10.0-514.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813179a5
Build id: 3.10.0-514.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9903
Build id: 3.10.0-514.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c4c8
Build id: 3.10.0-514.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c599
Build id: 3.10.0-514.el7.x86_64
0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac093
Build id: 3.10.0-514.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad385
Build id: 3.10.0-514.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff811b4254
Build id: 3.10.0-514.el7.x86_64
0xf4
13
? vma_rb_erase
vmlinux 0xffffffff811b47d1
Build id: 3.10.0-514.el7.x86_64
0x121
14
do_munmap
vmlinux 0xffffffff811b6605
Build id: 3.10.0-514.el7.x86_64
0x245
15
vm_munmap
vmlinux 0xffffffff811b6821
Build id: 3.10.0-514.el7.x86_64
0x41
16
SyS_munmap
vmlinux 0xffffffff811b77b2
Build id: 3.10.0-514.el7.x86_64
0x22
17
system_call_fastpath
vmlinux 0xffffffff816965c9
Build id: 3.10.0-514.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [zmstat-cpu:10521]
Modules linked in: talpa_vfshook(OE) talpa_pedconnector(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) tcp_diag inet_diag binfmt_misc talpa_pedevice(OE) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock intel_powerclamp coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ppdev pcspkr vmw_balloon
parport_pc parport i2c_piix4 vmw_vmci shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mptspi ata_piix ttm crct10dif_pclmul scsi_transport_spi crct10dif_common libata mptscsih drm crc32c_intel serio_raw mptbase vmxnet3 i2c_core floppy fjes dm_mirror dm_region_hash dm_log dm_mod [last unloaded: talpa_syscallhook]
CPU: 2 PID: 10521 Comm: zmstat-cpu Tainted: G        W  OEL ------------   3.10.0-514.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff88041cbf8fb0 ti: ffff88000d9bc000 task.ti: ffff88000d9bc000
RIP: 0010:[<ffffffff810f925a>]  [<ffffffff810f925a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88000d9bfc80  EFLAGS: 00000202
RAX: 00000000000008fd RBX: ffff880036359140 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000282
RBP: ffff88000d9bfcd0 R08: 0000000000000001 R09: ffffffff811bbed8
R10: ffff88043fc99a80 R11: ffffea000dd3e140 R12: 0000000000000000
R13: ffffffffa02788a7 R14: ffff88000d9bfc58 R15: ffff88000d9bfce8
FS:  00007fd96940d740(0000) GS:ffff88043fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002450230 CR3: 0000000035959000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c300 ffff88000d9bfd58
0000000000000003 0000000044478a09 ffff88004bd870f0 0000000000000001
ffffffff8106c300 ffffffff8106c300 ffff88000d9bfd00 ffffffff810f935f
Call Trace:
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff810f935f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813179a5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9903>] smp_call_function_many+0x223/0x260
[<ffffffff8106c4c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c599>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac093>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad385>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b4254>] unmap_region+0xf4/0x140
[<ffffffff811b47d1>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6605>] do_munmap+0x245/0x420
[<ffffffff811b6821>] vm_munmap+0x41/0x60
[<ffffffff811b77b2>] SyS_munmap+0x22/0x30
[<ffffffff816965c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 40 57 ad 81 48 89 df e8 ea 28 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #193236
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f925a
Build id: 3.10.0-514.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f935f
Build id: 3.10.0-514.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813179a5
Build id: 3.10.0-514.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9903
Build id: 3.10.0-514.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c4c8
Build id: 3.10.0-514.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c599
Build id: 3.10.0-514.el7.x86_64
0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac093
Build id: 3.10.0-514.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad385
Build id: 3.10.0-514.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff811b4254
Build id: 3.10.0-514.el7.x86_64
0xf4
13
? xfs_release
xfs 0xffffffffa024507e
Build id: 3.10.0-514.el7.x86_64
0x9e
14
? mntput_no_expire
vmlinux 0xffffffff8121e7de
Build id: 3.10.0-514.el7.x86_64
0x3e
15
? vma_rb_erase
vmlinux 0xffffffff811b47d1
Build id: 3.10.0-514.el7.x86_64
0x121
16
do_munmap
vmlinux 0xffffffff811b6605
Build id: 3.10.0-514.el7.x86_64
0x245
17
vm_munmap
vmlinux 0xffffffff811b6821
Build id: 3.10.0-514.el7.x86_64
0x41
18
SyS_munmap
vmlinux 0xffffffff811b77b2
Build id: 3.10.0-514.el7.x86_64
0x22
19
system_call_fastpath
vmlinux 0xffffffff816965c9
Build id: 3.10.0-514.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [I2Availability:2776]
Modules linked in: binfmt_misc iptable_filter fuse cfg80211 rfkill sfs(OE) sg ppdev parport_pc parport virtio_balloon i2c_piix4 intel_powerclamp pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_scsi virtio_net virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix serio_raw libata virtio_pci i2c_core virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 2776 Comm: I2Availability Tainted: G           OEL ------------   3.10.0-514.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff8800b9903ec0 ti: ffff88041ea70000 task.ti: ffff88041ea70000
RIP: 0010:[<ffffffff810f925a>]  [<ffffffff810f925a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88041ea73c80  EFLAGS: 00000202
RAX: 00000000000000a0 RBX: ffff88041ea73c50 RCX: 0000000000000020
RDX: 00000000ffffffff RSI: 00000000000000a0 RDI: 0000000000000282
RBP: ffff88041ea73cd0 R08: ffffffff816afc28 R09: ffffffff811bbfcc
R10: ffff88042cf99a80 R11: ffffea001069ca00 R12: 0000000000000001
R13: ffffffff8169151b R14: ffff88041ea73bf8 R15: 000000000000015e
FS:  00007f9d011738c0(0000) GS:ffff88042cf80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9d0119a000 CR3: 000000041d680000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c300 ffff88041ea73d58
0000000000000003 0000000013f831f6 ffff88041caf4cd0 0000000000000001
ffffffff8106c300 ffffffff8106c300 ffff88041ea73d00 ffffffff810f935f
Call Trace:
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff810f935f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813179a5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9903>] smp_call_function_many+0x223/0x260
[<ffffffff8106c4c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c599>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac093>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad385>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b4254>] unmap_region+0xf4/0x140
[<ffffffffa024507e>] ? xfs_release+0x9e/0x170 [xfs]
[<ffffffff8121e7de>] ? mntput_no_expire+0x3e/0x120
[<ffffffff811b47d1>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6605>] do_munmap+0x245/0x420
[<ffffffff811b6821>] vm_munmap+0x41/0x60
[<ffffffff811b77b2>] SyS_munmap+0x22/0x30
[<ffffffff816965c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 40 57 ad 81 48 89 df e8 ea 28 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #184794
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810d66ea
Build id: 3.10.0-229.14.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.14.1.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.14.1.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8105fab0
Build id: 3.10.0-229.14.1.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810d67ef
Build id: 3.10.0-229.14.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812d4695
Build id: 3.10.0-229.14.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810d6d83
Build id: 3.10.0-229.14.1.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8105fc78
Build id: 3.10.0-229.14.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8105fe15
Build id: 3.10.0-229.14.1.el7.x86_64
0x135
10
tlb_flush_mmu.part.53
vmlinux 0xffffffff8117e063
Build id: 3.10.0-229.14.1.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff8117ebc5
Build id: 3.10.0-229.14.1.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff811866b2
Build id: 3.10.0-229.14.1.el7.x86_64
0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811e670e
Build id: 3.10.0-229.14.1.el7.x86_64
0x3e
14
? vma_rb_erase
vmlinux 0xffffffff81186c11
Build id: 3.10.0-229.14.1.el7.x86_64
0x121
15
do_munmap
vmlinux 0xffffffff81188a42
Build id: 3.10.0-229.14.1.el7.x86_64
0x242
16
vm_munmap
vmlinux 0xffffffff81188c51
Build id: 3.10.0-229.14.1.el7.x86_64
0x41
17
SyS_munmap
vmlinux 0xffffffff81189b82
Build id: 3.10.0-229.14.1.el7.x86_64
0x22
18
system_call_fastpath
vmlinux 0xffffffff81614389
Build id: 3.10.0-229.14.1.el7.x86_64
0x16
BUG: soft lockup - CPU#5 stuck for 23s! [ftdc:5059]
Modules linked in: binfmt_misc tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables dahdi(OF) crc_ccitt netatop(OF) xt_CHECKSUM iptable_mangle xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_nat_ipv4 nf_nat nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_recent xt_multiport xt_mark xt_conntrack vmw_vsock_vmci_transport nf_conntrack vsock iptable_filter ext4 mbcache jbd2 coretemp snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq crc32_pclmul snd_seq_device ghash_clmulni_intel snd_pcm ppdev aesni_intel snd_timer lrw gf128mul vmw_balloon snd glue_helper ablk_helper cryptd soundcore pcspkr ahci sg i2c_piix4 libahci vmw_vmci shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd sunrpc ip_tables xfs libcrc32c
sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common ttm crc32c_intel mptspi serio_raw drm scsi_transport_spi ata_piix mptscsih libata e1000 mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 5059 Comm: ftdc Tainted: GF          O--------------   3.10.0-229.14.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8800bafcc440 ti: ffff880328748000 task.ti: ffff880328748000
RIP: 0010:[<ffffffff810d66ea>]  [<ffffffff810d66ea>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88032874bc80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88032874bc50 RCX: 0000000000000001
RDX: ffffffff8162fc30 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88032874bcd0 R08: ffffffff8162fc28 R09: ffff8803326b63e0
R10: ffffffff8118deb5 R11: ffffea0007a58f00 R12: ffffffff8116520e
R13: ffff88032874bbf8 R14: ffff88032acc2798 R15: ffffea00013c8a80
FS:  00007f06b53eb700(0000) GS:ffff8803326a0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f06c1bd1000 CR3: 00000003285f8000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8105fab0 ffff88032874bd58
0000000000000003 000000003ff7507d 0000000000000000 0000000000000000
ffffffff8105fab0 ffffffff8105fab0 ffff88032874bd00 ffffffff810d67ef
Call Trace:
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff8105fab0>] ? leave_mm+0x70/0x70
[<ffffffff810d67ef>] smp_call_function_single+0x5f/0xa0
[<ffffffff812d4695>] ? cpumask_next_and+0x35/0x50
[<ffffffff810d6d83>] smp_call_function_many+0x223/0x260
[<ffffffff8105fc78>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8105fe15>] flush_tlb_mm_range+0x135/0x180
[<ffffffff8117e063>] tlb_flush_mmu.part.53+0x33/0x90
[<ffffffff8117ebc5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811866b2>] unmap_region+0xe2/0x130
[<ffffffff811e670e>] ? mntput_no_expire+0x3e/0x120
[<ffffffff81186c11>] ? vma_rb_erase+0x121/0x220
[<ffffffff81188a42>] do_munmap+0x242/0x410
[<ffffffff81188c51>] vm_munmap+0x41/0x60
[<ffffffff81189b82>] SyS_munmap+0x22/0x30
[<ffffffff81614389>] system_call_fastpath+0x16/0x1b
Code: 40 41 01 00 48 89 de 48 03 14 c5 00 4d a1 81 48 89 df e8 ba 12 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #174889
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810d677a
Build id: 3.10.0-229.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8105faf0
Build id: 3.10.0-229.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8105faf0
Build id: 3.10.0-229.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8105faf0
Build id: 3.10.0-229.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810d687f
Build id: 3.10.0-229.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812d4ae5
Build id: 3.10.0-229.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810d6e23
Build id: 3.10.0-229.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8105fcb8
Build id: 3.10.0-229.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8105fe55
Build id: 3.10.0-229.el7.x86_64
0x135
10
tlb_flush_mmu.part.53
vmlinux 0xffffffff8117e083
Build id: 3.10.0-229.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff8117ec25
Build id: 3.10.0-229.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff81186732
Build id: 3.10.0-229.el7.x86_64
0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811e68ae
Build id: 3.10.0-229.el7.x86_64
0x3e
14
? vma_rb_erase
vmlinux 0xffffffff81186c91
Build id: 3.10.0-229.el7.x86_64
0x121
15
do_munmap
vmlinux 0xffffffff81188ac2
Build id: 3.10.0-229.el7.x86_64
0x242
16
vm_munmap
vmlinux 0xffffffff81188cd1
Build id: 3.10.0-229.el7.x86_64
0x41
17
SyS_munmap
vmlinux 0xffffffff81189c02
Build id: 3.10.0-229.el7.x86_64
0x22
18
system_call_fastpath
vmlinux 0xffffffff81614a29
Build id: 3.10.0-229.el7.x86_64
0x16
BUG: soft lockup - CPU#5 stuck for 23s! [systemd-journal:543]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd vmw_balloon pcspkr sg parport_pc parport shpchp vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic pata_acpi serio_raw vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel ttm ata_piix
mptspi scsi_transport_spi libata drm mptscsih mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 543 Comm: systemd-journal Not tainted 3.10.0-229.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff880fe3cdf1c0 ti: ffff880fe6f3c000 task.ti: ffff880fe6f3c000
RIP: 0010:[<ffffffff810d677a>]  [<ffffffff810d677a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff880fe6f3fc80  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff880fe6f3fc50 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff880fe6f3fcd0 R08: ffffffff81631620 R09: ffff88103fd563e0
R10: ffffffff8118df55 R11: ffffea003eaffe40 R12: ffffffff8116521e
R13: ffff880fe6f3fbf8 R14: ffff8807327d97a0 R15: ffffea00062c2680
FS:  00007fe9dd02a880(0000) GS:ffff88103fd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe9dd045000 CR3: 0000000fe4543000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8105faf0 ffff880fe6f3fd58
0000000000000003 00000000813dd2f0 ffff880fe4665228 0000000000000004
ffffffff8105faf0 ffffffff8105faf0 ffff880fe6f3fd00 ffffffff810d687f
Call Trace:
[<ffffffff8105faf0>] ? leave_mm+0x70/0x70
[<ffffffff8105faf0>] ? leave_mm+0x70/0x70
[<ffffffff8105faf0>] ? leave_mm+0x70/0x70
[<ffffffff810d687f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812d4ae5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810d6e23>] smp_call_function_many+0x223/0x260
[<ffffffff8105fcb8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8105fe55>] flush_tlb_mm_range+0x135/0x180
[<ffffffff8117e083>] tlb_flush_mmu.part.53+0x33/0x90
[<ffffffff8117ec25>] tlb_finish_mmu+0x55/0x60
[<ffffffff81186732>] unmap_region+0xe2/0x130
[<ffffffff811e68ae>] ? mntput_no_expire+0x3e/0x120
[<ffffffff81186c91>] ? vma_rb_erase+0x121/0x220
[<ffffffff81188ac2>] do_munmap+0x242/0x410
[<ffffffff81188cd1>] vm_munmap+0x41/0x60
[<ffffffff81189c02>] SyS_munmap+0x22/0x30
[<ffffffff81614a29>] system_call_fastpath+0x16/0x1b
Code: 40 41 01 00 48 89 de 48 03 14 c5 c0 4d a1 81 48 89 df e8 da 16 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #159035
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64
0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff8119b2d2
Build id: 3.10.0-327.el7.x86_64
0xe2
13
? vma_rb_erase
vmlinux 0xffffffff8119b831
Build id: 3.10.0-327.el7.x86_64
0x121
14
do_munmap
vmlinux 0xffffffff8119d612
Build id: 3.10.0-327.el7.x86_64
0x242
15
vm_munmap
vmlinux 0xffffffff8119d821
Build id: 3.10.0-327.el7.x86_64
0x41
16
SyS_munmap
vmlinux 0xffffffff8119e772
Build id: 3.10.0-327.el7.x86_64
0x22
17
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64
0x16
BUG: soft lockup - CPU#1 stuck for 22s! [mongod:7213]
Modules linked in: bnep bluetooth rfkill fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables vmhgfs(OE) ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device vmw_balloon snd_pcm snd_timer pcspkr snd soundcore vmw_vmci sg i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi crct10dif_pclmul crct10dif_common vmwgfx crc32c_intel drm_kms_helper ttm drm serio_raw i2c_core ata_piix mptspi scsi_transport_spi mptscsih e1000 libata mptbase
CPU: 1 PID: 7213 Comm: mongod Tainted: G           OEL ------------   3.10.0-327.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff88022e7fc500 ti: ffff88022f528000 task.ti: ffff88022f528000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88022f52bc80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88022f52bc50 RCX: 0000000000000004
RDX: ffffffff8165ef70 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88022f52bcd0 R08: ffffffff8165ef68 R09: ffff880236637520
R10: ffffffff811a2b52 R11: ffffea0008babf80 R12: 000000cf00000000
R13: ffffffff81641113 R14: ffff88022f52bc08 R15: 00000000000003a9
FS:  00007f59997a4700(0000) GS:ffff880236620000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f59a40f2000 CR3: 00000000b15af000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff88022f52bd58
0000000000000003 000000006ee8c942 0000000000000000 0000000000000003
ffffffff81065c90 ffffffff81065c90 ffff88022f52bd00 ffffffff810e685f
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b2d2>] unmap_region+0xe2/0x130
[<ffffffff8119b831>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d612>] do_munmap+0x242/0x410
[<ffffffff8119d821>] vm_munmap+0x41/0x60
[<ffffffff8119e772>] SyS_munmap+0x22/0x30
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #136866
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64
0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff8119b2d2
Build id: 3.10.0-327.el7.x86_64
0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811fe6fe
Build id: 3.10.0-327.el7.x86_64
0x3e
14
? vma_rb_erase
vmlinux 0xffffffff8119b831
Build id: 3.10.0-327.el7.x86_64
0x121
15
do_munmap
vmlinux 0xffffffff8119d612
Build id: 3.10.0-327.el7.x86_64
0x242
16
vm_munmap
vmlinux 0xffffffff8119d821
Build id: 3.10.0-327.el7.x86_64
0x41
17
SyS_munmap
vmlinux 0xffffffff8119e772
Build id: 3.10.0-327.el7.x86_64
0x22
18
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64
0x16
BUG: soft lockup - CPU#3 stuck for 22s! [mongod:112987]
Modules linked in: bnep bluetooth rfkill fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables binfmt_misc vmhgfs(OE) ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 vmw_vsock_vmci_transport vsock xt_conntrack nf_conntrack iptable_filter dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event coretemp ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore i2c_piix4 sg shpchp vmw_vmci parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic
sr_mod cdrom crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic vmwgfx pata_acpi drm_kms_helper ttm e1000 drm mptspi scsi_transport_spi mptscsih i2c_core mptbase ata_piix libata
CPU: 3 PID: 112987 Comm: mongod Tainted: G           OEL ------------   3.10.0-327.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8800b49de780 ti: ffff8800b7100000 task.ti: ffff8800b7100000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8800b7103c80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8800b7103c50 RCX: 0000000000000001
RDX: ffffffff8165e7f0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8800b7103cd0 R08: ffffffff8165e7e8 R09: ffff88042e677520
R10: ffffffff811a2bc5 R11: ffffea0010555fc0 R12: ffff880407cc9f40
R13: ffff8800b49de780 R14: ffff8800b7103c40 R15: ffff8800b49de780
FS:  00007ff77fb5c700(0000) GS:ffff88042e660000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff792cbc000 CR3: 00000003f7be9000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff8800b7103d58
0000000000000003 000000001f01b5c6 0000000000000000 0000000000000000
ffffffff81065c90 ffffffff81065c90 ffff8800b7103d00 ffffffff810e685f
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b2d2>] unmap_region+0xe2/0x130
[<ffffffff811fe6fe>] ? mntput_no_expire+0x3e/0x120
[<ffffffff8119b831>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d612>] do_munmap+0x242/0x410
[<ffffffff8119d821>] vm_munmap+0x41/0x60
[<ffffffff8119e772>] SyS_munmap+0x22/0x30
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #136863
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64
0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff8119b2d2
Build id: 3.10.0-327.el7.x86_64
0xe2
13
? handle_mm_fault
vmlinux 0xffffffff81197088
Build id: 3.10.0-327.el7.x86_64
0x5b8
14
? vma_rb_erase
vmlinux 0xffffffff8119b831
Build id: 3.10.0-327.el7.x86_64
0x121
15
do_munmap
vmlinux 0xffffffff8119d612
Build id: 3.10.0-327.el7.x86_64
0x242
16
vm_munmap
vmlinux 0xffffffff8119d821
Build id: 3.10.0-327.el7.x86_64
0x41
17
SyS_munmap
vmlinux 0xffffffff8119e772
Build id: 3.10.0-327.el7.x86_64
0x22
18
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64
0x16
BUG: soft lockup - CPU#1 stuck for 21s! [abrt-watch-log:835]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmhgfs(OE) vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec coretemp ac97_bus crc32_pclmul ghash_clmulni_intel snd_seq snd_seq_device ppdev snd_pcm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_timer snd vmw_balloon soundcore pcspkr sg i2c_piix4 vmw_vmci shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi
crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw ttm drm ata_piix mptspi i2c_core scsi_transport_spi e1000 libata mptscsih mptbase
CPU: 1 PID: 835 Comm: abrt-watch-log Tainted: G           OEL ------------   3.10.0-327.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff880035cab980 ti: ffff88003b1a0000 task.ti: ffff88003b1a0000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88003b1a3c80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88003b1a3c50 RCX: 0000000000000001
RDX: ffffffff8165e7f0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88003b1a3cd0 R08: ffffffff8165e7e8 R09: 00007ff29423e000
R10: ffff88003b68f498 R11: 0000000000000202 R12: ffffffffa024f328
R13: ffff88003b1a3c28 R14: ffff880039fe4e00 R15: ffff88003ac7ea18
FS:  00007ff2941f2880(0000) GS:ffff88003d620000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff29423d1a4 CR3: 000000003b2eb000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff88003b1a3d58
0000000000000003 000000007f952ec5 0000000000000000 0000000000000000
ffffffff81065c90 ffffffff81065c90 ffff88003b1a3d00 ffffffff810e685f
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b2d2>] unmap_region+0xe2/0x130
[<ffffffff81197088>] ? handle_mm_fault+0x5b8/0xf50
[<ffffffff8119b831>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d612>] do_munmap+0x242/0x410
[<ffffffff8119d821>] vm_munmap+0x41/0x60
[<ffffffff8119e772>] SyS_munmap+0x22/0x30
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #40319
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f93aa
Build id: 3.10.0-514.6.2.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.6.2.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.6.2.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.6.2.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f94af
Build id: 3.10.0-514.6.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81317b15
Build id: 3.10.0-514.6.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9a53
Build id: 3.10.0-514.6.2.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c5f8
Build id: 3.10.0-514.6.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c6c9
Build id: 3.10.0-514.6.2.el7.x86_64
0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac1e3
Build id: 3.10.0-514.6.2.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad4d5
Build id: 3.10.0-514.6.2.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff811b43a4
Build id: 3.10.0-514.6.2.el7.x86_64
0xf4
13
? vma_rb_erase
vmlinux 0xffffffff811b4921
Build id: 3.10.0-514.6.2.el7.x86_64
0x121
14
do_munmap
vmlinux 0xffffffff811b6755
Build id: 3.10.0-514.6.2.el7.x86_64
0x245
15
vm_munmap
vmlinux 0xffffffff811b6971
Build id: 3.10.0-514.6.2.el7.x86_64
0x41
16
SyS_munmap
vmlinux 0xffffffff811b7902
Build id: 3.10.0-514.6.2.el7.x86_64
0x22
17
system_call_fastpath
vmlinux 0xffffffff816968c9
Build id: 3.10.0-514.6.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [vmtoolsd:1094]
Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) ext4 mbcache jbd2 snd_seq_midi snd_seq_midi_event intel_powerclamp coretemp iosf_mbi snd_ens1371 crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul snd_rawmidi glue_helper snd_ac97_codec ablk_helper cryptd ppdev ac97_bus snd_seq snd_seq_device vmw_balloon snd_pcm snd_timer pcspkr snd nfit soundcore
sg libnvdimm parport_pc parport i2c_piix4 shpchp vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm mptspi crc32c_intel scsi_transport_spi mptscsih ata_piix mptbase serio_raw libata vmxnet3 i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1094 Comm: vmtoolsd Tainted: G           OEL ------------   3.10.0-514.6.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8800ba4fce70 ti: ffff8800ac7f4000 task.ti: ffff8800ac7f4000
RIP: 0010:[<ffffffff810f93aa>]  [<ffffffff810f93aa>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8800ac7f7c80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8800ac7f7c50 RCX: 0000000000000001
RDX: ffffffff816af9b0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8800ac7f7cd0 R08: ffffffff816af9a8 R09: ffffffff811bc028
R10: ffff8800ba459a80 R11: ffffea0002c5bbc0 R12: ffff8800ac7f7c48
R13: 0000000000000000 R14: ffff880087c76400 R15: 0000000000000001
FS:  00007f9456c67740(0000) GS:ffff8800ba440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9456c88000 CR3: 00000000abaf3000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c430 ffff8800ac7f7d58
0000000000000003 000000001f79e24b ffff8800b121e440 0000000000000000
ffffffff8106c430 ffffffff8106c430 ffff8800ac7f7d00 ffffffff810f94af
Call Trace:
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff810f94af>] smp_call_function_single+0x5f/0xa0
[<ffffffff81317b15>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9a53>] smp_call_function_many+0x223/0x260
[<ffffffff8106c5f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c6c9>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac1e3>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad4d5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b43a4>] unmap_region+0xf4/0x140
[<ffffffff811b4921>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6755>] do_munmap+0x245/0x420
[<ffffffff811b6971>] vm_munmap+0x41/0x60
[<ffffffff811b7902>] SyS_munmap+0x22/0x30
[<ffffffff816968c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 80 57 ad 81 48 89 df e8 0a 29 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #18165
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64
0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff8119b2d2
Build id: 3.10.0-327.el7.x86_64
0xe2
13
? vma_rb_erase
vmlinux 0xffffffff8119b831
Build id: 3.10.0-327.el7.x86_64
0x121
14
do_munmap
vmlinux 0xffffffff8119d612
Build id: 3.10.0-327.el7.x86_64
0x242
15
vm_munmap
vmlinux 0xffffffff8119d821
Build id: 3.10.0-327.el7.x86_64
0x41
16
SyS_munmap
vmlinux 0xffffffff8119e772
Build id: 3.10.0-327.el7.x86_64
0x22
17
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64
0x16
BUG: soft lockup - CPU#3 stuck for 22s! [python2.7:11172]
Modules linked in: binfmt_misc fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ppdev sg virtio_balloon i2c_piix4 pcspkr parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_blk virtio_scsi cirrus syscopyarea sysfillrect sysimgblt drm_kms_helper ttm ata_piix drm serio_raw libata virtio_pci i2c_core virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 11172 Comm: python2.7 Tainted: G             L ------------   3.10.0-327.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff88060d11d080 ti: ffff88069ee54000 task.ti: ffff88069ee54000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88069ee57c80  EFLAGS: 00000202
RAX: 00000000000000a0 RBX: ffff88069ee57c50 RCX: 0000000000000020
RDX: 00000000ffffffff RSI: 00000000000000a0 RDI: 0000000000000282
RBP: ffff88069ee57cd0 R08: ffffffff8165ece8 R09: ffff88081e277520
R10: ffffffff811a2b52 R11: ffffea001f899740 R12: ffffffff811d15c2
R13: ffff88069ee57c38 R14: 0000000000000000 R15: ffff88080c622bc0
FS:  00007f41e37fe700(0000) GS:ffff88081e260000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f41ff422000 CR3: 00000005e6c2d000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff88069ee57d58
0000000000000003 00000000b72abd96 0000000000000000 0000000000000002
ffffffff81065c90 ffffffff81065c90 ffff88069ee57d00 ffffffff810e685f
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b2d2>] unmap_region+0xe2/0x130
[<ffffffff8119b831>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d612>] do_munmap+0x242/0x410
[<ffffffff8119d821>] vm_munmap+0x41/0x60
[<ffffffff8119e772>] SyS_munmap+0x22/0x30
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #14306
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f925a
Build id: 3.10.0-514.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f935f
Build id: 3.10.0-514.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813179a5
Build id: 3.10.0-514.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9903
Build id: 3.10.0-514.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c4c8
Build id: 3.10.0-514.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c599
Build id: 3.10.0-514.el7.x86_64
0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac093
Build id: 3.10.0-514.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad385
Build id: 3.10.0-514.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff811b4254
Build id: 3.10.0-514.el7.x86_64
0xf4
13
? vma_rb_erase
vmlinux 0xffffffff811b47d1
Build id: 3.10.0-514.el7.x86_64
0x121
14
do_munmap
vmlinux 0xffffffff811b6605
Build id: 3.10.0-514.el7.x86_64
0x245
15
vm_munmap
vmlinux 0xffffffff811b6821
Build id: 3.10.0-514.el7.x86_64
0x41
16
SyS_munmap
vmlinux 0xffffffff811b77b2
Build id: 3.10.0-514.el7.x86_64
0x22
17
system_call_fastpath
vmlinux 0xffffffff816965c9
Build id: 3.10.0-514.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [systemd:1]
Modules linked in: xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle overlay ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc vmw_vsock_vmci_transport vsock sunrpc nfit libnvdimm snd_seq_midi snd_seq_midi_event ppdev vmw_balloon intel_powerclamp snd_ens1371 snd_rawmidi snd_ac97_codec coretemp ac97_bus iosf_mbi snd_seq crc32_pclmul ghash_clmulni_intel aesni_intel lrw sg snd_seq_device gf128mul snd_pcm snd_timer snd soundcore glue_helper ablk_helper cryptd pcspkr shpchp i2c_piix4 parport_pc parport vmw_vmci ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic
pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel mptspi vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops scsi_transport_spi mptscsih ttm ata_piix mptbase drm e1000 libata i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 1 Comm: systemd Tainted: G        W    L ------------   3.10.0-514.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff880184db0000 ti: ffff880184db8000 task.ti: ffff880184db8000
RIP: 0010:[<ffffffff810f925a>]  [<ffffffff810f925a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff880184dbbc80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff880184dbbc50 RCX: 0000000000000002
RDX: ffffffff816afc30 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff880184dbbcd0 R08: ffffffff816afc28 R09: ffffffff811bbed8
R10: ffff8802356d9a80 R11: ffffea0007908680 R12: ffffea000091ae00
R13: ffff880187f46b80 R14: ffff880231778c80 R15: ffff880184db0000
FS:  00007f3bf5cff940(0000) GS:ffff8802356c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3bf5d27000 CR3: 0000000230e08000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c300 ffff880184dbbd58
0000000000000003 00000000c1823def ffff880231ff4938 0000000000000001
ffffffff8106c300 ffffffff8106c300 ffff880184dbbd00 ffffffff810f935f
Call Trace:
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff810f935f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813179a5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9903>] smp_call_function_many+0x223/0x260
[<ffffffff8106c4c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c599>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac093>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad385>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b4254>] unmap_region+0xf4/0x140
[<ffffffff811b47d1>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6605>] do_munmap+0x245/0x420
[<ffffffff811b6821>] vm_munmap+0x41/0x60
[<ffffffff811b77b2>] SyS_munmap+0x22/0x30
[<ffffffff816965c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 40 57 ad 81 48 89 df e8 ea 28 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #13019
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f93ba
Build id: 3.10.0-514.2.2.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.2.2.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.2.2.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.2.2.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f94bf
Build id: 3.10.0-514.2.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81317b25
Build id: 3.10.0-514.2.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9a63
Build id: 3.10.0-514.2.2.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c5f8
Build id: 3.10.0-514.2.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c6c9
Build id: 3.10.0-514.2.2.el7.x86_64
0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac1f3
Build id: 3.10.0-514.2.2.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad4e5
Build id: 3.10.0-514.2.2.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff811b43b4
Build id: 3.10.0-514.2.2.el7.x86_64
0xf4
13
? mntput_no_expire
vmlinux 0xffffffff8121e93e
Build id: 3.10.0-514.2.2.el7.x86_64
0x3e
14
? vma_rb_erase
vmlinux 0xffffffff811b4931
Build id: 3.10.0-514.2.2.el7.x86_64
0x121
15
do_munmap
vmlinux 0xffffffff811b6765
Build id: 3.10.0-514.2.2.el7.x86_64
0x245
16
vm_munmap
vmlinux 0xffffffff811b6981
Build id: 3.10.0-514.2.2.el7.x86_64
0x41
17
SyS_munmap
vmlinux 0xffffffff811b7912
Build id: 3.10.0-514.2.2.el7.x86_64
0x22
18
system_call_fastpath
vmlinux 0xffffffff816967c9
Build id: 3.10.0-514.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [java:6347]
Modules linked in: binfmt_misc ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock intel_powerclamp snd_seq_midi snd_seq_midi_event coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw snd_ens1371 gf128mul snd_rawmidi glue_helper snd_ac97_codec ablk_helper cryptd ac97_bus ppdev snd_seq snd_seq_device snd_pcm vmw_balloon snd_timer snd soundcore pcspkr sg parport_pc parport
i2c_piix4 shpchp vmw_vmci nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx serio_raw drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm e1000 mptspi ata_piix scsi_transport_spi i2c_core mptscsih libata mptbase fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 6347 Comm: java Not tainted 3.10.0-514.2.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff88081c2cce70 ti: ffff8807ee650000 task.ti: ffff8807ee650000
RIP: 0010:[<ffffffff810f93ba>]  [<ffffffff810f93ba>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8807ee653c80  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8807ee653c50 RCX: 0000000000000004
RDX: ffffffff816b0130 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8807ee653cd0 R08: ffffffff816b0128 R09: ffffffff811bc12c
R10: ffff88081d659a80 R11: ffffea000717e800 R12: ffff8807ee653c48
R13: 0000000000000000 R14: ffff880187c76400 R15: 0000000000000001
FS:  00007f848af49700(0000) GS:ffff88081d640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f849805e000 CR3: 00000008053d0000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c430 ffff8807ee653d58
0000000000000002 000000002376ca59 ffff8807fee832f0 0000000000000003
ffffffff8106c430 ffffffff8106c430 ffff8807ee653d00 ffffffff810f94bf
Call Trace:
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff810f94bf>] smp_call_function_single+0x5f/0xa0
[<ffffffff81317b25>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9a63>] smp_call_function_many+0x223/0x260
[<ffffffff8106c5f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c6c9>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac1f3>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad4e5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b43b4>] unmap_region+0xf4/0x140
[<ffffffff8121e93e>] ? mntput_no_expire+0x3e/0x120
[<ffffffff811b4931>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6765>] do_munmap+0x245/0x420
[<ffffffff811b6981>] vm_munmap+0x41/0x60
[<ffffffff811b7912>] SyS_munmap+0x22/0x30
[<ffffffff816967c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 80 57 ad 81 48 89 df e8 0a 29 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #4497
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64
0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff8119b2d2
Build id: 3.10.0-327.el7.x86_64
0xe2
13
? mntput_no_expire
vmlinux 0xffffffff811fe6fe
Build id: 3.10.0-327.el7.x86_64
0x3e
14
? vma_rb_erase
vmlinux 0xffffffff8119b831
Build id: 3.10.0-327.el7.x86_64
0x121
15
do_munmap
vmlinux 0xffffffff8119d612
Build id: 3.10.0-327.el7.x86_64
0x242
16
vm_munmap
vmlinux 0xffffffff8119d821
Build id: 3.10.0-327.el7.x86_64
0x41
17
SyS_munmap
vmlinux 0xffffffff8119e772
Build id: 3.10.0-327.el7.x86_64
0x22
18
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64
0x16
BUG: soft lockup - CPU#3 stuck for 23s! [java:4789]
Modules linked in: 8021q garp mrp binfmt_misc bridge stp llc snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer ppdev snd sg soundcore i2c_piix4 parport_pc pcspkr parport virtio_balloon xfs libcrc32c sr_mod cdrom ata_generic pata_acpi qxl syscopyarea sysfillrect sysimgblt virtio_blk drm_kms_helper virtio_console ttm ata_piix virtio_net drm libata serio_raw virtio_pci i2c_core virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 3 PID: 4789 Comm: java Tainted: G             L ------------   3.10.0-327.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS Bochs 01/01/2011
task: ffff8801235d4500 ti: ffff88003639c000 task.ti: ffff88003639c000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88003639fc80  EFLAGS: 00000202
RAX: 00000000000400f6 RBX: ffffffff81e99780 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000282
RBP: ffff88003639fcd0 R08: 0000000000000001 R09: ffff88043fd97520
R10: ffffffff811a2b52 R11: ffffea000acca900 R12: ffffffff811d15c2
R13: ffff88003639fc38 R14: 0000000000000000 R15: ffff8801350512c0
FS:  00007fb3f27ef700(0000) GS:ffff88043fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb405c58000 CR3: 0000000427374000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff88003639fd58
0000000000000003 0000000034738f9d 0000000000000000 0000000000000001
ffffffff81065c90 ffffffff81065c90 ffff88003639fd00 ffffffff810e685f
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119b2d2>] unmap_region+0xe2/0x130
[<ffffffff811fe6fe>] ? mntput_no_expire+0x3e/0x120
[<ffffffff8119b831>] ? vma_rb_erase+0x121/0x220
[<ffffffff8119d612>] do_munmap+0x242/0x410
[<ffffffff8119d821>] vm_munmap+0x41/0x60
[<ffffffff8119e772>] SyS_munmap+0x22/0x30
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #141
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f925a
Build id: 3.10.0-514.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f935f
Build id: 3.10.0-514.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813179a5
Build id: 3.10.0-514.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9903
Build id: 3.10.0-514.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c4c8
Build id: 3.10.0-514.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c599
Build id: 3.10.0-514.el7.x86_64
0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac093
Build id: 3.10.0-514.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad385
Build id: 3.10.0-514.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff811b4254
Build id: 3.10.0-514.el7.x86_64
0xf4
13
? vma_rb_erase
vmlinux 0xffffffff811b47d1
Build id: 3.10.0-514.el7.x86_64
0x121
14
do_munmap
vmlinux 0xffffffff811b6605
Build id: 3.10.0-514.el7.x86_64
0x245
15
vm_munmap
vmlinux 0xffffffff811b6821
Build id: 3.10.0-514.el7.x86_64
0x41
16
SyS_munmap
vmlinux 0xffffffff811b77b2
Build id: 3.10.0-514.el7.x86_64
0x22
17
system_call_fastpath
vmlinux 0xffffffff816965c9
Build id: 3.10.0-514.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [3_scheduler:3036]
Modules linked in: fuse uinput xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_hda_codec_generic iosf_mbi crc32_pclmul snd_hda_intel ghash_clmulni_intel aesni_intel snd_hda_codec lrw gf128mul glue_helper ablk_helper snd_hda_core snd_hwdep snd_seq ppdev snd_seq_device cryptd snd_pcm snd_timer snd pcspkr parport_pc virtio_balloon parport sg i2c_piix4 soundcore nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_blk virtio_net qxl drm_kms_helper crct10dif_pclmul syscopyarea crct10dif_common sysfillrect
crc32c_intel sysimgblt ata_piix fb_sys_fops serio_raw ttm drm virtio_pci virtio_ring virtio libata i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 3036 Comm: 3_scheduler Tainted: G             L ------------   3.10.0-514.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff880221633ec0 ti: ffff8802216b8000 task.ti: ffff8802216b8000
RIP: 0010:[<ffffffff810f925a>]  [<ffffffff810f925a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8802216bbc80  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff8802216bbc50 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff8802216bbcd0 R08: ffffffff816afc20 R09: ffffffff811bbed8
R10: ffff880239d19a80 R11: ffffea00068f92c0 R12: 00000000813179a5
R13: ffff880239fd6008 R14: ffff880184e7a000 R15: ffffffff8118a67a
FS:  00007f7d3effd700(0000) GS:ffff880239d00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7d3cb02630 CR3: 00000000b8f44000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c300 ffff8802216bbd58
0000000000000003 0000000061b58962 ffff8800bb309bf8 0000000000000001
ffffffff8106c300 ffffffff8106c300 ffff8802216bbd00 ffffffff810f935f
Call Trace:
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff810f935f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813179a5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9903>] smp_call_function_many+0x223/0x260
[<ffffffff8106c4c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c599>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac093>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad385>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b4254>] unmap_region+0xf4/0x140
[<ffffffff811b47d1>] ? vma_rb_erase+0x121/0x220
[<ffffffff811b6605>] do_munmap+0x245/0x420
[<ffffffff811b6821>] vm_munmap+0x41/0x60
[<ffffffff811b77b2>] SyS_munmap+0x22/0x30
[<ffffffff816965c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 40 57 ad 81 48 89 df e8 ea 28 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00