Info

Permalink
Function
generic_exec_single
First occurrence
2020-09-24
Last occurrence
2020-09-24
State
NEW
Components
kernel
Quality
-26

Statistics

Operating system Count
CentOS 7 1,599
Architecture Count
x86_64 1,599
Related packages Count
kernel 1,599
 0:3.10.0-1160.el7 973
 0:3.10.0-1160.83.1.el7 95
 0:3.10.0-957.el7 52
 0:3.10.0-862.el7 51
 0:3.10.0-1160.76.1.el7 41
 0:3.10.0-1160.49.1.el7 38
 0:3.10.0-1160.119.1.el7 35
 0:3.10.0-1160.81.1.el7 29
 0:3.10.0-1160.11.1.el7 26
 0:3.10.0-1062.el7 25
 0:3.10.0-1160.25.1.el7 21
 0:3.10.0-1160.105.1.el7 19
 0:3.10.0-1160.118.1.el7 15
 0:3.10.0-957.12.2.el7 15
 0:3.10.0-1062.1.2.el7 14
 0:3.10.0-1160.45.1.el7 14
 0:3.10.0-1160.59.1.el7 12
 0:3.10.0-1160.36.2.el7 10
 0:3.10.0-1160.92.1.el7 9
 0:3.10.0-693.21.1.el7 9
 0:3.10.0-1062.12.1.el7 7
 0:3.10.0-514.el7 7
 0:3.10.0-693.el7 7
 0:3.10.0-1160.15.2.el7 6
 0:3.10.0-1160.71.1.el7 6
 0:3.10.0-1160.6.1.el7 5
 0:3.10.0-862.11.6.el7 5
 0:3.10.0-693.5.2.el7 5
 0:3.10.0-1160.102.1.el7 4
 0:3.10.0-1160.31.1.el7 4
 0:3.10.0-957.10.1.el7 3
 0:3.10.0-1160.88.1.el7 3
 0:3.10.0-1160.90.1.el7 3
 0:3.10.0-327.el7 3
 0:3.10.0-1160.108.1.el7 3
 0:3.10.0-957.12.1.el7 2
 0:3.10.0-957.21.3.el7 2
 0:3.10.0-1160.42.2.el7 2
 0:3.10.0-514.16.1.el7 2
 0:3.10.0-1160.95.1.el7 2
 0:3.10.0-1127.18.2.el7 1
 0:3.10.0-327.28.3.el7 1
 0:3.10.0-514.10.2.el7 1
 0:3.10.0-1160.24.1.el7 1
 0:3.10.0-862.14.4.el7 1
 0:3.10.0-1062.9.1.el7 1
 0:3.10.0-957.27.2.el7 1
 0:3.10.0-957.5.1.el7 1
 0:3.10.0-1160.21.1.el7 1
 0:3.10.0-1062.4.1.el7 1
 0:3.10.0-1160.66.1.el7 1
 0:3.10.0-1160.80.1.el7 1
 0:3.10.0-957.1.3.el7 1
 0:3.10.0-1160.62.1.el7 1
 0:3.10.0-693.17.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1379880
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810fb21a
Build id: 3.10.0-693.17.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106f340
Build id: 3.10.0-693.17.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff8106f340
Build id: 3.10.0-693.17.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff8106f340
Build id: 3.10.0-693.17.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff810fb31f
Build id: 3.10.0-693.17.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81324bd5
Build id: 3.10.0-693.17.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810fb8b3
Build id: 3.10.0-693.17.1.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106f508
Build id: 3.10.0-693.17.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106f5d8
Build id: 3.10.0-693.17.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.63
vmlinux 0xffffffff811af0f3
Build id: 3.10.0-693.17.1.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811b0415
Build id: 3.10.0-693.17.1.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff811b299a
Build id: 3.10.0-693.17.1.el7.x86_64
0x13a
13
? kmem_cache_free
vmlinux 0xffffffff811e0b02
Build id: 3.10.0-693.17.1.el7.x86_64
0x1e2
14
SyS_madvise
vmlinux 0xffffffff811ae7bd
Build id: 3.10.0-693.17.1.el7.x86_64
0x3cd
15
system_call_fastpath
vmlinux 0xffffffff816b89fd
Build id: 3.10.0-693.17.1.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 21s! [mysqld:16967]
Modules linked in: tcp_lp ip6table_filter ip6_tables fuse ipt_REJECT nf_reject_ipv4 xt_multiport iptable_filter vmw_vsock_vmci_transport vsock vfat fat crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon nfit libnvdimm sg parport_pc i2c_piix4 tpm_crb parport joydev shpchp vmw_vmci pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi crct10dif_pclmul crct10dif_common scsi_transport_spi ata_piix libata e1000 mptscsih crc32c_intel mptbase i2c_core serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 16967 Comm: mysqld Tainted: G        W    L ------------   3.10.0-693.17.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware20,1/440BX Desktop Reference Platform, BIOS VMW201.00V.24504846.B64.2501180339 01/18/2025
task: ffff880176b84f10 ti: ffff880072700000 task.ti: ffff880072700000
RIP: 0010:[<ffffffff810fb21a>]  [<ffffffff810fb21a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff880072703bf0  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff880072703bc0 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff880072703c40 R08: ffffffff816d6ee0 R09: 00007f504db53000
R10: 00007f507555c240 R11: 00007f504db11000 R12: ffff880176b84f10
R13: ffffffff8163c5d2 R14: ffff880072703cd0 R15: 0000000000000004
FS:  00007f504db58700(0000) GS:ffff88063fd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3925ebe090 CR3: 0000000613eae000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8106f340>] ? leave_mm+0x110/0x110
[<ffffffff8106f340>] ? leave_mm+0x110/0x110
[<ffffffff8106f340>] ? leave_mm+0x110/0x110
[<ffffffff810fb31f>] smp_call_function_single+0x5f/0xa0
[<ffffffff81324bd5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810fb8b3>] smp_call_function_many+0x223/0x260
[<ffffffff8106f508>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106f5d8>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811af0f3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811b0415>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b299a>] zap_page_range+0x13a/0x180
[<ffffffff811e0b02>] ? kmem_cache_free+0x1e2/0x200
[<ffffffff811ae7bd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 40 95 01 00 48 89 de 48 03 14 c5 20 8e b1 81 48 89 df e8 5a e0 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1045631
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa6b1dba2
Build id: 3.10.0-1160.118.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffa6a81b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffffa6a81b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffffa6a81b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffffa6b1dcbf
Build id: 3.10.0-1160.118.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa6d9c615
Build id: 3.10.0-1160.118.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa6b1e27b
Build id: 3.10.0-1160.118.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa6a81fa8
Build id: 3.10.0-1160.118.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa6a8201b
Build id: 3.10.0-1160.118.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa6bfa0ce
Build id: 3.10.0-1160.118.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffffa6bfbc4f
Build id: 3.10.0-1160.118.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffa6bfbd83
Build id: 3.10.0-1160.118.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffa6bff2ca
Build id: 3.10.0-1160.118.1.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffffa6ae1118
Build id: 3.10.0-1160.118.1.el7.x86_64
0x58
15
SyS_madvise
vmlinux 0xffffffffa6bf96bd
Build id: 3.10.0-1160.118.1.el7.x86_64
0x49d
16
system_call_fastpath
vmlinux 0xffffffffa71c539a
Build id: 3.10.0-1160.118.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [docker-containe:15950]
Modules linked in: binfmt_misc cts 8021q garp mrp ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache falcon_lsm_serviceable(PE) falcon_nf_netcontain(E) falcon_kal(E) falcon_lsm_pinned_17005(E) nf_nat_tftp nf_conntrack_tftp nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) tun vmw_vsock_vmci_transport
vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper nfit ata_piix syscopyarea sysfillrect sysimgblt fb_sys_fops ttm libata libnvdimm drm crct10dif_pclmul crct10dif_common mptspi crc32c_intel serio_raw scsi_transport_spi mptscsih vmxnet3 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 15950 Comm: docker-containe Kdump: loaded Tainted: P            EL ------------ T 3.10.0-1160.118.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff93b1d4971080 ti: ffff93b1d4fac000 task.ti: ffff93b1d4fac000
RIP: 0010:[<ffffffffa6b1dba2>]  [<ffffffffa6b1dba2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff93b1d4fafbe0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff93b1d4fafbb0 RCX: 0000000000000830
RDX: 0000000000000004 RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff93b1d4fafc30 R08: ffffffffa7214140 R09: 00003ffffffff000
R10: ffff93b1d513d510 R11: 0000000000000246 R12: ffff93b1d4971080
R13: ffffffffa6aeab04 R14: ffff93b1d4fafb80 R15: ffff93b1d4fafc60
FS:  00007fd2a642a700(0000) GS:ffff93b1ffc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000db6578 CR3: 00000002148ac000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa6a81b10>] ? leave_mm+0x130/0x130
[<ffffffffa6a81b10>] ? leave_mm+0x130/0x130
[<ffffffffa6a81b10>] ? leave_mm+0x130/0x130
[<ffffffffa6b1dcbf>] smp_call_function_single+0x5f/0xa0
[<ffffffffa6d9c615>] ? cpumask_next_and+0x35/0x50
[<ffffffffa6b1e27b>] smp_call_function_many+0x22b/0x270
[<ffffffffa6a81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa6a8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa6bfa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa6bfbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa6bfbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffffa6bff2ca>] zap_page_range+0xea/0x140
[<ffffffffa6ae1118>] ? wake_up_q+0x58/0x90
[<ffffffffa6bf96bd>] SyS_madvise+0x49d/0xac0
[<ffffffffa71c539a>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 c0 19 75 a7 48 89 df e8 b7 67 29 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1026083
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff887113ea
Build id: 3.10.0-957.21.3.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8867a4e0
Build id: 3.10.0-957.21.3.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff8867a4e0
Build id: 3.10.0-957.21.3.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff8867a4e0
Build id: 3.10.0-957.21.3.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff887114ff
Build id: 3.10.0-957.21.3.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff88975cd5
Build id: 3.10.0-957.21.3.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff88711aab
Build id: 3.10.0-957.21.3.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8867a6a8
Build id: 3.10.0-957.21.3.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8867a718
Build id: 3.10.0-957.21.3.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff887e4687
Build id: 3.10.0-957.21.3.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff887e5f85
Build id: 3.10.0-957.21.3.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff887e8dd2
Build id: 3.10.0-957.21.3.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffff887e3c9d
Build id: 3.10.0-957.21.3.el7.x86_64
0x49d
14
? hrtimer_get_res
vmlinux 0xffffffff886c5950
Build id: 3.10.0-957.21.3.el7.x86_64
0x50
15
system_call_fastpath
vmlinux 0xffffffff88d75ddb
Build id: 3.10.0-957.21.3.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 27s! [node_exporter:14419]
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [node_exporter:4362]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev virtio_balloon i2c_piix4 pcspkr ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_net virtio_scsi crct10dif_pclmul crct10dif_common crc32c_intel bochs_drm drm_kms_helper ata_piix syscopyarea sysfillrect sysimgblt fb_sys_fops ttm floppy drm serio_raw libata virtio_pci drm_panel_orientation_quirks virtio_ring
virtio
CPU: 5 PID: 4362 Comm: node_exporter Kdump: loaded Tainted: G             L ------------   3.10.0-957.21.3.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
task: ffff8af096a130c0 ti: ffff8af090364000 task.ti: ffff8af090364000
RIP: 0010:[<ffffffff887113ea>]  [<ffffffff887113ea>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8af090367bf0  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff8af090367bc0 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff8af090367c38 R08: ffffffff88e13dc0 R09: ffffe2169d82fc40
R10: 0000000000004aac R11: 0000000000000246 R12: ffff8af0bffda000
R13: ffffffff887c11c6 R14: ffff8af090367c00 R15: 0000000000020200
FS:  000000c00004a090(0000) GS:ffff8af0bfd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000562f00dfa78c CR3: 0000000790298000 CR4: 0000000000760ee0
PKRU: 55555554
Call Trace:
[<ffffffff8867a4e0>] ? leave_mm+0x110/0x110
[<ffffffff8867a4e0>] ? leave_mm+0x110/0x110
[<ffffffff8867a4e0>] ? leave_mm+0x110/0x110
[<ffffffff887114ff>] smp_call_function_single+0x5f/0xa0
[<ffffffff88975cd5>] ? cpumask_next_and+0x35/0x50
[<ffffffff88711aab>] smp_call_function_many+0x22b/0x270
[<ffffffff8867a6a8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8867a718>] flush_tlb_mm_range+0x68/0x140
[<ffffffff887e4687>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff887e5f85>] tlb_finish_mmu+0x55/0x60
[<ffffffff887e8dd2>] zap_page_range+0x142/0x190
[<ffffffff887e3c9d>] SyS_madvise+0x49d/0xac0
[<ffffffff886c5950>] ? hrtimer_get_res+0x50/0x50
[<ffffffff88d75ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 a0 bc 34 89 48 89 df e8 4a b7 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1006791
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb191dba2
Build id: 3.10.0-1160.118.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffb1881b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffffb1881b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffffb1881b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffffb191dcbf
Build id: 3.10.0-1160.118.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb1b9c615
Build id: 3.10.0-1160.118.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb191e27b
Build id: 3.10.0-1160.118.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb1881fa8
Build id: 3.10.0-1160.118.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb188201b
Build id: 3.10.0-1160.118.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb19fa0ce
Build id: 3.10.0-1160.118.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb19fbc4f
Build id: 3.10.0-1160.118.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb19fbd83
Build id: 3.10.0-1160.118.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffb19ff2ca
Build id: 3.10.0-1160.118.1.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffffb18e1118
Build id: 3.10.0-1160.118.1.el7.x86_64
0x58
15
SyS_madvise
vmlinux 0xffffffffb19f96bd
Build id: 3.10.0-1160.118.1.el7.x86_64
0x49d
16
? pick_next_entity
vmlinux 0xffffffffb18e8699
Build id: 3.10.0-1160.118.1.el7.x86_64
0xa9
17
? set_next_entity
vmlinux 0xffffffffb18e918c
Build id: 3.10.0-1160.118.1.el7.x86_64
0x3c
18
system_call_fastpath
vmlinux 0xffffffffb1fc539a
Build id: 3.10.0-1160.118.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [containerd-shim:30877]
Modules linked in: veth xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_addrtype br_netfilter bridge stp llc iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon sg pcspkr joydev vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm nfit sd_mod libahci ata_piix crc_t10dif crct10dif_generic libnvdimm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash
dm_log dm_mod fuse
CPU: 3 PID: 30877 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.118.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9d13a3cc6300 ti: ffff9d13a1128000 task.ti: ffff9d13a1128000
RIP: 0010:[<ffffffffb191dba2>]  [<ffffffffb191dba2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9d13a112bbe0  EFLAGS: 00000202
RAX: 0000000000000006 RBX: ffff9d13a112bbb0 RCX: 0000000000000830
RDX: 0000000000000006 RSI: 0000000000000006 RDI: 0000000000000282
RBP: ffff9d13a112bc30 R08: ffffffffb2014640 R09: 00003ffffffff000
R10: ffff9d13ea810e58 R11: 0000000000000246 R12: ffff9d146a6730c0
R13: ffff9d146a6730c0 R14: 000000000000023a R15: ffff9d11c8faa000
FS:  000000c000200490(0000) GS:ffff9d147fcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00021b010 CR3: 00000004aa8c0000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffb1881b10>] ? leave_mm+0x130/0x130
[<ffffffffb1881b10>] ? leave_mm+0x130/0x130
[<ffffffffb1881b10>] ? leave_mm+0x130/0x130
[<ffffffffb191dcbf>] smp_call_function_single+0x5f/0xa0
[<ffffffffb1b9c615>] ? cpumask_next_and+0x35/0x50
[<ffffffffb191e27b>] smp_call_function_many+0x22b/0x270
[<ffffffffb1881fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb188201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb19fa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffb19fbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb19fbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffffb19ff2ca>] zap_page_range+0xea/0x140
[<ffffffffb18e1118>] ? wake_up_q+0x58/0x90
[<ffffffffb19f96bd>] SyS_madvise+0x49d/0xac0
[<ffffffffb18e8699>] ? pick_next_entity+0xa9/0x190
[<ffffffffb18e918c>] ? set_next_entity+0x3c/0x100
[<ffffffffb1fc539a>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 c0 19 55 b2 48 89 df e8 b7 67 29 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #920433
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8851109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8847a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff8847a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff8847a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff885111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff88775465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff8851175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8847a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8847a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff885e4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff885e5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff885e87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffff884d6768
Build id: 3.10.0-957.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffff885e368d
Build id: 3.10.0-957.el7.x86_64
0x49d
15
tracesys
vmlinux 0xffffffff88b7505b
Build id: 3.10.0-957.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [kube-apiserver:54766]
Modules linked in: ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_CHECKSUM tun ebtable_filter ebtables ip6table_filter devlink ip6_tables xt_multiport xt_nat veth nf_tables(T) vxlan ip6_udp_tunnel udp_tunnel ip_set_hash_ipportip ip_set_bitmap_port ip_set_hash_ipportnet ip_set_hash_ipport dummy iptable_mangle xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_nat_ipv4 ceph libceph dns_resolver overlay(T) toa(OE) vmw_vsock_vmci_transport vsock bnep sunrpc dm_mirror dm_region_hash dm_log dm_mod crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev btusb btrtl btbcm btintel bluetooth rfkill snd_seq_midi snd_seq_midi_event snd_ens1371
snd_rawmidi snd_ac97_codec ac97_bus parport_pc parport snd_seq snd_seq_device snd_pcm snd_timer snd soundcore sg vmw_vmci i2c_piix4 nf_conntrack_ipv4 nf_defrag_ipv4 ipip tunnel4 ip_tunnel ipt_REJECT nf_reject_ipv4 ipt_rpfilter xt_set ip_set nfnetlink ip_vs_ftp nf_nat ip_vs_sed ip_vs_nq ip_vs_dh ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs_lblcr ip_vs_lblc ip_vs_lc ip_vs nf_conntrack br_netfilter bridge stp llc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod cdrom crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw nfit libnvdimm ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix e1000 libata mptspi scsi_transport_spi drm_panel_orientation_quirks mptscsih mptbase
CPU: 3 PID: 54766 Comm: kube-apiserver Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff88caf3326180 ti: ffff88ca08b98000 task.ti: ffff88ca08b98000
RIP: 0010:[<ffffffff8851109a>]  [<ffffffff8851109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff88ca08b9bbf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88ca08b9bbc0 RCX: 0000000000000003
RDX: ffffffff88c13290 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88ca08b9bc38 R08: ffffffff88c13288 R09: 00003ffffffff000
R10: ffff88c9fe802948 R11: 0000000000000246 R12: ffff88caf965ab80
R13: ffff88ca616c90d0 R14: ffff88caf965ab80 R15: 00000000060b69cb
FS:  000000c000068c90(0000) GS:ffff88caf96c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000c4a020 CR3: 0000000044be0000 CR4: 0000000000360fe0
Call Trace:
[<ffffffff8847a3d0>] ? leave_mm+0x110/0x110
[<ffffffff8847a3d0>] ? leave_mm+0x110/0x110
[<ffffffff8847a3d0>] ? leave_mm+0x110/0x110
[<ffffffff885111af>] smp_call_function_single+0x5f/0xa0
[<ffffffff88775465>] ? cpumask_next_and+0x35/0x50
[<ffffffff8851175b>] smp_call_function_many+0x22b/0x270
[<ffffffff8847a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8847a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff885e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff885e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff885e87c2>] zap_page_range+0x142/0x190
[<ffffffff884d6768>] ? wake_up_q+0x58/0x80
[<ffffffff885e368d>] SyS_madvise+0x49d/0xac0
[<ffffffff88b7505b>] tracesys+0xa3/0xc9
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 14 89 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #919178
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff94d1109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff94c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff94c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff94c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff94d111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff94f75465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff94d1175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff94c7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff94c7a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff94de4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff94de5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff94de87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffff94cd6768
Build id: 3.10.0-957.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffff94de368d
Build id: 3.10.0-957.el7.x86_64
0x49d
15
? mntput
vmlinux 0xffffffff94e640a4
Build id: 3.10.0-957.el7.x86_64
0x24
16
system_call_fastpath
vmlinux 0xffffffff95374ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 39s! [kubelet:83601]
Modules linked in: xt_nat ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat nf_conntrack br_netfilter overlay(T) tun devlink ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock bnep sunrpc iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel snd_seq_midi snd_seq_midi_event aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr snd_ens1371 btusb btrtl btbcm btintel snd_rawmidi bluetooth snd_ac97_codec ac97_bus snd_seq rfkill snd_seq_device snd_pcm snd_timer snd soundcore sg parport_pc parport vmw_vmci i2c_piix4 xfs libcrc32c sd_mod crc_t10dif
crct10dif_generic sr_mod cdrom crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic vmwgfx drm_kms_helper syscopyarea pata_acpi sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi ata_piix drm e1000 mptscsih libata drm_panel_orientation_quirks nfit mptbase libnvdimm dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 0 PID: 83601 Comm: kubelet Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff980a0b36b0c0 ti: ffff980a0b388000 task.ti: ffff980a0b388000
RIP: 0010:[<ffffffff94d1109a>]  [<ffffffff94d1109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff980a0b38bbf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff980a0b38bbc0 RCX: 0000000000000002
RDX: ffffffff95413010 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff980a0b38bc38 R08: ffffffff95413008 R09: 00003ffffffff000
R10: ffff980af09ffcb0 R11: 0000000000000246 R12: ffffffff94cdd70b
R13: ffff980a0b38bbd0 R14: 0000000000000046 R15: ffff980a0b38bb58
FS:  00007fdca65e9700(0000) GS:ffff980af9600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f245a1a9000 CR3: 0000000135028000 CR4: 00000000003607f0
Call Trace:
[<ffffffff94c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff94c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff94c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff94d111af>] smp_call_function_single+0x5f/0xa0
[<ffffffff94f75465>] ? cpumask_next_and+0x35/0x50
[<ffffffff94d1175b>] smp_call_function_many+0x22b/0x270
[<ffffffff94c7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff94c7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff94de4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff94de5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff94de87c2>] zap_page_range+0x142/0x190
[<ffffffff94cd6768>] ? wake_up_q+0x58/0x80
[<ffffffff94de368d>] SyS_madvise+0x49d/0xac0
[<ffffffff94e640a4>] ? mntput+0x24/0x40
[<ffffffff95374ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 94 95 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #876476
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f925a
Build id: 3.10.0-514.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f935f
Build id: 3.10.0-514.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813179a5
Build id: 3.10.0-514.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9903
Build id: 3.10.0-514.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c4c8
Build id: 3.10.0-514.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c599
Build id: 3.10.0-514.el7.x86_64
0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac093
Build id: 3.10.0-514.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad385
Build id: 3.10.0-514.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff811af75a
Build id: 3.10.0-514.el7.x86_64
0x13a
13
? lru_cache_add
vmlinux 0xffffffff8118f898
Build id: 3.10.0-514.el7.x86_64
0x8
14
SyS_madvise
vmlinux 0xffffffff811ab7ec
Build id: 3.10.0-514.el7.x86_64
0x38c
15
system_call_fastpath
vmlinux 0xffffffff816965c9
Build id: 3.10.0-514.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [qemu-kvm:3750]
Modules linked in: vhost_net vhost macvtap macvlan nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter bridge stp llc vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event intel_powerclamp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel ppdev snd_ens1371 snd_rawmidi aesni_intel snd_ac97_codec lrw ac97_bus vmw_balloon gf128mul glue_helper snd_seq ablk_helper cryptd snd_seq_device snd_pcm snd_timer pcspkr sg snd soundcore i2c_piix4 vmw_vmci shpchp nfit libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c
sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw ttm drm ata_piix mptspi libata scsi_transport_spi i2c_core mptscsih e1000 mptbase fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3750 Comm: qemu-kvm Tainted: G             L ------------   3.10.0-514.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff88030bfc4e70 ti: ffff8802e1668000 task.ti: ffff8802e1668000
RIP: 0010:[<ffffffff810f925a>]  [<ffffffff810f925a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8802e166bc40  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8802e166bbd0 RCX: 0000000000000003
RDX: ffffffff816afeb0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8802e166bc88 R08: ffffffff816afea8 R09: ffffea000b63eb00
R10: 0000000000003ec4 R11: 0000000000000206 R12: 0000000000000010
R13: ffffffff81057b36 R14: ffffffffffffff10 R15: 0000000000000282
FS:  00007fc357bff700(0000) GS:ffff880331600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc3c8a76ff0 CR3: 00000002e17c5000 CR4: 00000000003427f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c300 ffff8802e166bd10
0000000000000003 0000000055ab89eb 0000000000000002 ffffffff8106c300
ffffffff8106c300 ffff8802e166bcb8 ffffffff810f935f ffff8802e166bcb8
Call Trace:
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff810f935f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813179a5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9903>] smp_call_function_many+0x223/0x260
[<ffffffff8106c4c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c599>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac093>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad385>] tlb_finish_mmu+0x55/0x60
[<ffffffff811af75a>] zap_page_range+0x13a/0x180
[<ffffffff8118f898>] ? lru_cache_add+0x8/0x10
[<ffffffff811ab7ec>] SyS_madvise+0x38c/0x8d0
[<ffffffff816965c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 40 57 ad 81 48 89 df e8 ea 28 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #875452
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff85b1109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff85a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff85a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff85a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff85b111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff85d75465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff85b1175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff85a7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff85a7a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff85be4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff85be5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff85be87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffff85ad6768
Build id: 3.10.0-957.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffff85be368d
Build id: 3.10.0-957.el7.x86_64
0x49d
15
? update_curr
vmlinux 0xffffffff85adfff6
Build id: 3.10.0-957.el7.x86_64
0x86
16
system_call_fastpath
vmlinux 0xffffffff86174ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [containerd-shim:20456]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon
pcspkr joydev snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer sg snd soundcore parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_pclmul crct10dif_common crc32c_intel ata_generic pata_acpi serio_raw vmwgfx e1000 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix scsi_transport_spi mptscsih libata mptbase nfit drm_panel_orientation_quirks libnvdimm
CPU: 1 PID: 20456 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9415f2b330c0 ti: ffff9415f5bb0000 task.ti: ffff9415f5bb0000
RIP: 0010:[<ffffffff85b1109a>]  [<ffffffff85b1109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9415f5bb3bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9415f5bb3bc0 RCX: 0000000000000001
RDX: ffffffff86212d90 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9415f5bb3c38 R08: ffffffff86212d88 R09: fffff88240a785c0
R10: 000000000000312b R11: 0000000000000202 R12: ffff94163ffd8000
R13: ffffffff85bc0c06 R14: ffff9415f5bb3c00 R15: 0000000000020200
FS:  000000c000080890(0000) GS:ffff941635640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f25e7e42a80 CR3: 00000001f9a12000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff85a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff85a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff85a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff85b111af>] smp_call_function_single+0x5f/0xa0
[<ffffffff85d75465>] ? cpumask_next_and+0x35/0x50
[<ffffffff85b1175b>] smp_call_function_many+0x22b/0x270
[<ffffffff85a7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff85a7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff85be4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff85be5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff85be87c2>] zap_page_range+0x142/0x190
[<ffffffff85ad6768>] ? wake_up_q+0x58/0x80
[<ffffffff85be368d>] SyS_madvise+0x49d/0xac0
[<ffffffff85adfff6>] ? update_curr+0x86/0x1e0
[<ffffffff86174ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 74 86 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #860205
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9e716532
Build id: 3.10.0-1160.31.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff9e67ddd0
Build id: 3.10.0-1160.31.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9e67ddd0
Build id: 3.10.0-1160.31.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9e67ddd0
Build id: 3.10.0-1160.31.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff9e71664f
Build id: 3.10.0-1160.31.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9e9864f5
Build id: 3.10.0-1160.31.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9e716bfb
Build id: 3.10.0-1160.31.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9e67e258
Build id: 3.10.0-1160.31.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9e67e2cb
Build id: 3.10.0-1160.31.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9e7ee176
Build id: 3.10.0-1160.31.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff9e7efc3f
Build id: 3.10.0-1160.31.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff9e7efd63
Build id: 3.10.0-1160.31.1.el7.x86_64
0x23
13
unmap_region
vmlinux 0xffffffff9e7f9554
Build id: 3.10.0-1160.31.1.el7.x86_64
0xf4
14
? xfs_iunlock
xfs 0xffffffffc03f03c4
Build id: 3.10.0-1160.31.1.el7.x86_64
0xe4
15
? __vma_rb_erase
vmlinux 0xffffffff9e7f9b11
Build id: 3.10.0-1160.31.1.el7.x86_64
0x121
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [commons-pool-Ev:2480]
Modules linked in: xt_CHECKSUM tun devlink xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_mark ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc squashfs loop sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper vmw_balloon cryptd joydev sg pcspkr vmw_vmci parport_pc i2c_piix4 parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic
pata_acpi vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common syscopyarea sysfillrect ahci sysimgblt fb_sys_fops crc32c_intel ttm serio_raw libahci ata_piix nfit drm vmxnet3 vmw_pvscsi libnvdimm libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 2480 Comm: commons-pool-Ev Kdump: loaded Tainted: G             L ------------   3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9ff5fd9f6300 ti: ffff9ff5fd9e4000 task.ti: ffff9ff5fd9e4000
RIP: 0010:[<ffffffff9e716532>]  [<ffffffff9e716532>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9ff5fd9e7c20  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff9ff5fd9e7bf0 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff9ff5fd9e7c70 R08: ffffffff9ee137c0 R09: ffffffff9e802b7c
R10: ffff9ff63fd1f0a0 R11: fffff3aa0cc48380 R12: ffffffff9ed8c778
R13: ffff9ff5fd9e7c70 R14: 0000000000001000 R15: 000000000000009e
FS:  00007faba5e8b700(0000) GS:ffff9ff63fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fac1a45a000 CR3: 000000032b4aa000 CR4: 00000000000607e0
Call Trace:
[<ffffffff9e67ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9e67ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9e67ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9e71664f>] smp_call_function_single+0x5f/0xa0
[<ffffffff9e9864f5>] ? cpumask_next_and+0x35/0x50
[<ffffffff9e716bfb>] smp_call_function_many+0x22b/0x270
[<ffffffff9e67e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9e67e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9e7ee176>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9e7efc3f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9e7efd63>] tlb_finish_mmu+0x23/0x30
[<ffffffff9e7f9554>] unmap_region+0xf4/0x140
[<ffffffffc03f03c4>] ? xfs_iunlock+0xe4/0x120 [xfs]
[<ffffffff9e7f9b11>] ? __vma_rb_erase+0x121/0x220
Complete report #851668
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb730779a
Build id: 3.10.0-862.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffb7276e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffb7276e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffb7276e50
Build id: 3.10.0-862.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffb73078af
Build id: 3.10.0-862.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb754a275
Build id: 3.10.0-862.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb7307e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb7277018
Build id: 3.10.0-862.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb7277088
Build id: 3.10.0-862.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffffb73c0497
Build id: 3.10.0-862.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffb73c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffb73c4a02
Build id: 3.10.0-862.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffffb72cee38
Build id: 3.10.0-862.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffffb73bfafd
Build id: 3.10.0-862.el7.x86_64
0x3cd
15
? update_curr
vmlinux 0xffffffffb72d80a6
Build id: 3.10.0-862.el7.x86_64
0x86
16
system_call_fastpath
vmlinux 0xffffffffb791f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 24s! [containerd-shim:3019]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event crc32_pclmul ghash_clmulni_intel snd_ens1371 aesni_intel snd_rawmidi lrw snd_ac97_codec gf128mul glue_helper ablk_helper ac97_bus cryptd snd_seq snd_seq_device snd_pcm snd_timer snd soundcore ppdev sg vmw_balloon pcspkr joydev nfit parport_pc libnvdimm parport i2c_piix4 vmw_vmci shpchp ip_tables xfs libcrc32c sr_mod
sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common mptspi libata crc32c_intel scsi_transport_spi e1000 mptscsih serio_raw mptbase i2c_core
CPU: 11 PID: 3019 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9d1138bc1fa0 ti: ffff9d185aa60000 task.ti: ffff9d185aa60000
RIP: 0010:[<ffffffffb730779a>]  [<ffffffffb730779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9d185aa63bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9d185aa63bc0 RCX: 0000000000000009
RDX: ffffffffb7a14570 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9d185aa63c38 R08: ffffffffb7a14568 R09: fffff6c72009f5c0
R10: fffff6c72009f5c0 R11: 0000000000000202 R12: ffff9d1138bc1fa0
R13: ffff9d1138bc1fa0 R14: ffff9d185aa63ba0 R15: 0000000000000001
FS:  000000c000200090(0000) GS:ffff9d18bd8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc1e24dc580 CR3: 00000007d0fea000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffb7276e50>] ? leave_mm+0x110/0x110
[<ffffffffb7276e50>] ? leave_mm+0x110/0x110
[<ffffffffb7276e50>] ? leave_mm+0x110/0x110
[<ffffffffb73078af>] smp_call_function_single+0x5f/0xa0
[<ffffffffb754a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffb7307e5b>] smp_call_function_many+0x22b/0x270
[<ffffffffb7277018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb7277088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb73c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffb73c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffb73c4a02>] zap_page_range+0x142/0x190
[<ffffffffb72cee38>] ? wake_up_q+0x58/0x80
[<ffffffffb73bfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffffb72d80a6>] ? update_curr+0x86/0x1e0
[<ffffffffb791f7d5>] system_call_fastpath+0x1c/0x21
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 f3 b7 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #826859
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64
0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff8119568a
Build id: 3.10.0-327.el7.x86_64
0x12a
13
SyS_madvise
vmlinux 0xffffffff81191db4
Build id: 3.10.0-327.el7.x86_64
0x394
14
? SyS_futex
vmlinux 0xffffffff810e55a0
Build id: 3.10.0-327.el7.x86_64
0x80
15
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64
0x16
BUG: soft lockup - CPU#5 stuck for 22s! [docker-containe:30479]
Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop ip6t_rpfilter ip6t_REJECT ipt_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev parport_pc virtio_rng sg parport virtio_balloon i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_blk virtio_console
virtio_net cirrus syscopyarea sysfillrect sysimgblt drm_kms_helper crct10dif_pclmul crct10dif_common ttm ata_piix drm virtio_pci crc32c_intel virtio_ring libata serio_raw i2c_core virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 30479 Comm: docker-containe Tainted: G        W    L ------------   3.10.0-327.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.11.0-2.el7 04/01/2014
task: ffff880428f4dc00 ti: ffff8803ea8f8000 task.ti: ffff8803ea8f8000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8803ea8fbc40  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff8803ea8fbc10 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff8803ea8fbc88 R08: ffffffff8165ece0 R09: ffffea000ca909c0
R10: 00000000000043cb R11: 0000000000000246 R12: ffff880428f4dc00
R13: ffff880428f4dc00 R14: 0000000000000001 R15: ffff880428f4dc00
FS:  000000c42003bc90(0000) GS:ffff88043fd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0e93e21000 CR3: 0000000362c41000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff8803ea8fbd10
0000000000000003 0000000059a1352f 0000000000000002 ffffffff81065c90
ffffffff81065c90 ffff8803ea8fbcb8 ffffffff810e685f ffff8803ea8fbcb8
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119568a>] zap_page_range+0x12a/0x170
[<ffffffff81191db4>] SyS_madvise+0x394/0x820
[<ffffffff810e55a0>] ? SyS_futex+0x80/0x180
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #820688
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb991109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffb987a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffb987a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffb987a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffb99111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb9b75465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb991175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb987a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb987a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffb99e4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffb99e5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffb99e87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffffb98d6768
Build id: 3.10.0-957.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffffb99e368d
Build id: 3.10.0-957.el7.x86_64
0x49d
15
tracesys
vmlinux 0xffffffffb9f7505b
Build id: 3.10.0-957.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [minio:112566]
Modules linked in: nfnetlink_queue nfnetlink_log bluetooth rfkill xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq joydev snd_seq_device snd_pcm pcspkr snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic
pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm e1000 ata_piix mptspi scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 112566 Comm: minio Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff8a6af8281040 ti: ffff8a6d7d774000 task.ti: ffff8a6d7d774000
RIP: 0010:[<ffffffffb991109a>]  [<ffffffffb991109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8a6d7d777bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8a6d7d777b80 RCX: 0000000000000004
RDX: ffffffffba013510 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8a6d7d777c38 R08: ffffffffba013508 R09: 00003ffffffff000
R10: ffff8a6a2d1540d8 R11: 0000000000000202 R12: 0000000000000010
R13: ffffffffb98611c6 R14: ffffffffffffff10 R15: 0000000000000282
FS:  000000c00137a890(0000) GS:ffff8a6dc5140000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fff735538c8 CR3: 00000003f9eb4000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb987a3d0>] ? leave_mm+0x110/0x110
[<ffffffffb987a3d0>] ? leave_mm+0x110/0x110
[<ffffffffb987a3d0>] ? leave_mm+0x110/0x110
[<ffffffffb99111af>] smp_call_function_single+0x5f/0xa0
[<ffffffffb9b75465>] ? cpumask_next_and+0x35/0x50
[<ffffffffb991175b>] smp_call_function_many+0x22b/0x270
[<ffffffffb987a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb987a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb99e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffb99e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffb99e87c2>] zap_page_range+0x142/0x190
[<ffffffffb98d6768>] ? wake_up_q+0x58/0x80
[<ffffffffb99e368d>] SyS_madvise+0x49d/0xac0
[<ffffffffb9f7505b>] tracesys+0xa3/0xc9
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 54 ba 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #813770
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff95b16ae2
Build id: 3.10.0-1160.76.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff95a7ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff95a7ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff95a7ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff95b16bff
Build id: 3.10.0-1160.76.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff95d86845
Build id: 3.10.0-1160.76.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff95b171ab
Build id: 3.10.0-1160.76.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff95a7e258
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff95a7e2cb
Build id: 3.10.0-1160.76.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff95bee6a6
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff95bf016f
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff95bf0293
Build id: 3.10.0-1160.76.1.el7.x86_64
0x23
13
unmap_region
vmlinux 0xffffffff95bf9a84
Build id: 3.10.0-1160.76.1.el7.x86_64
0xf4
14
? __vma_rb_erase
vmlinux 0xffffffff95bfa041
Build id: 3.10.0-1160.76.1.el7.x86_64
0x121
15
do_munmap
vmlinux 0xffffffff95bfc025
Build id: 3.10.0-1160.76.1.el7.x86_64
0x2a5
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [rpm:63088]
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [rpm:63088]
Modules linked in: nf_conntrack_netlink tcp_diag udp_diag inet_diag ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables ext4 mbcache jbd2 bridge stp llc ip_set binfmt_misc ppdev joydev sg virtio_balloon parport_pc parport i2c_piix4 pcspkr xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover virtio_console virtio_scsi virtio_blk failover cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix
libata virtio_pci floppy virtio_ring serio_raw virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 10 PID: 63088 Comm: rpm Kdump: loaded Tainted: G             L ------------   3.10.0-1160.76.1.el7.x86_64 #1
Modules linked in: nf_conntrack_netlink tcp_diag udp_diag inet_diag ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables ext4 mbcache jbd2 bridge stp llc ip_set binfmt_misc ppdev joydev sg virtio_balloon parport_pc parport i2c_piix4 pcspkr xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover virtio_console virtio_scsi virtio_blk failover cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix
libata virtio_pci floppy virtio_ring serio_raw virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 10 PID: 63088 Comm: rpm Kdump: loaded Tainted: G             L ------------   3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9af70d7b2100 ti: ffff9af3ed0a8000 task.ti: ffff9af3ed0a8000
RIP: 0010:[<ffffffff95b16ae2>]  [<ffffffff95b16ae2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9af3ed0abc20  EFLAGS: 00000202
RAX: 00000000000000a0 RBX: ffff9af3ed0abbf0 RCX: 0000000000000020
RDX: 00000000ffffffff RSI: 00000000000000a0 RDI: 0000000000000282
RBP: ffff9af3ed0abc70 R08: ffffffff96214e48 R09: ffffffff95c02f68
R10: ffff9afa5d09f0a0 R11: ffffe2c15b5bc0c0 R12: ffff9af357d8d4e8
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
FS:  00007fb593add840(0000) GS:ffff9afa5d080000(0000) knlGS:0000000000000000
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9af70d7b2100 ti: ffff9af3ed0a8000 task.ti: ffff9af3ed0a8000
RIP: 0010:[<ffffffff95b16ae2>]  [<ffffffff95b16ae2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9af3ed0abc20  EFLAGS: 00000202
RAX: 00000000000000a0 RBX: ffff9af3ed0abbf0 RCX: 0000000000000020
RDX: 00000000ffffffff RSI: 00000000000000a0 RDI: 0000000000000282
RBP: ffff9af3ed0abc70 R08: ffffffff96214e48 R09: ffffffff95c02f68
R10: ffff9afa5d09f0a0 R11: ffffe2c15b5bc0c0 R12: ffff9af357d8d4e8
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
FS:  00007fb593add840(0000) GS:ffff9afa5d080000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb593af6000 CR3: 0000000134792000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff95a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff95a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff95a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff95b16bff>] smp_call_function_single+0x5f/0xa0
[<ffffffff95d86845>] ? cpumask_next_and+0x35/0x50
[<ffffffff95b171ab>] smp_call_function_many+0x22b/0x270
[<ffffffff95a7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff95a7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff95bee6a6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff95bf016f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff95bf0293>] tlb_finish_mmu+0x23/0x30
[<ffffffff95bf9a84>] unmap_region+0xf4/0x140
[<ffffffff95bfa041>] ? __vma_rb_erase+0x121/0x220
[<ffffffff95bfc025>] do_munmap+0x2a5/0x480
Complete report #802214
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810fcf5a
Build id: 3.10.0-693.21.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff810709d0
Build id: 3.10.0-693.21.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff810709d0
Build id: 3.10.0-693.21.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff810709d0
Build id: 3.10.0-693.21.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff810fd06f
Build id: 3.10.0-693.21.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813288a5
Build id: 3.10.0-693.21.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810fd61b
Build id: 3.10.0-693.21.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff81070b98
Build id: 3.10.0-693.21.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81070c68
Build id: 3.10.0-693.21.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.63
vmlinux 0xffffffff811b18c3
Build id: 3.10.0-693.21.1.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811b2be5
Build id: 3.10.0-693.21.1.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff811b9cb4
Build id: 3.10.0-693.21.1.el7.x86_64
0xf4
13
? evict
vmlinux 0xffffffff812224b8
Build id: 3.10.0-693.21.1.el7.x86_64
0x108
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [vmtoolsd:903]
Modules linked in: binfmt_misc tcp_diag udp_diag inet_diag fuse nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac edac_core iosf_mbi crc32_pclmul snd_ens1371 snd_rawmidi ghash_clmulni_intel snd_ac97_codec aesni_intel lrw gf128mul glue_helper ablk_helper ac97_bus cryptd snd_seq snd_seq_device snd_pcm
ppdev vmw_balloon snd_timer snd pcspkr joydev vmw_vmci sg soundcore nfit i2c_piix4 shpchp libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi scsi_transport_spi mptscsih mptbase crct10dif_pclmul crct10dif_common e1000 crc32c_intel i2c_core serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 903 Comm: vmtoolsd Tainted: G             L ------------   3.10.0-693.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff880818a8af70 ti: ffff8800350b0000 task.ti: ffff8800350b0000
RIP: 0010:[<ffffffff810fcf5a>]  [<ffffffff810fcf5a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8800350b3c30  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8800350b3c00 RCX: 0000000000000003
RDX: ffffffff816dcdf0 RSI: 0000000000000080 RDI: 0000000000000286
RBP: ffff8800350b3c80 R08: ffffffff816dcde8 R09: ffffffff811c20dc
R10: ffff88081d75b940 R11: ffffea001d81b740 R12: ffff88017fd8ec00
R13: ffffea0008c06f40 R14: ffff88081788f080 R15: ffff880818a8af70
FS:  00007f75bab48880(0000) GS:ffff88081d740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f75bab69000 CR3: 00000008166fe000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff810709d0>] ? leave_mm+0x110/0x110
[<ffffffff810709d0>] ? leave_mm+0x110/0x110
[<ffffffff810709d0>] ? leave_mm+0x110/0x110
[<ffffffff810fd06f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813288a5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810fd61b>] smp_call_function_many+0x22b/0x270
[<ffffffff81070b98>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81070c68>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811b18c3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811b2be5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b9cb4>] unmap_region+0xf4/0x140
[<ffffffff812224b8>] ? evict+0x108/0x180
Complete report #777913
Frame # Function Binary Source or offset Line
1
? leave_mm
vmlinux 0xffffffffab276e50
Build id: 3.10.0-862.el7.x86_64
0x110
2
? leave_mm
vmlinux 0xffffffffab276e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffab276e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
smp_call_function_single
vmlinux 0xffffffffab3078af
Build id: 3.10.0-862.el7.x86_64
0x5f
5
? cpumask_next_and
vmlinux 0xffffffffab54a275
Build id: 3.10.0-862.el7.x86_64
0x35
6
smp_call_function_many
vmlinux 0xffffffffab307e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
7
native_flush_tlb_others
vmlinux 0xffffffffab277018
Build id: 3.10.0-862.el7.x86_64
0xb8
8
flush_tlb_mm_range
vmlinux 0xffffffffab277088
Build id: 3.10.0-862.el7.x86_64
0x68
9
tlb_flush_mmu.part.75
vmlinux 0xffffffffab3c0497
Build id: 3.10.0-862.el7.x86_64
0x37
10
tlb_finish_mmu
vmlinux 0xffffffffab3c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
11
unmap_region
vmlinux 0xffffffffab3ca624
Build id: 3.10.0-862.el7.x86_64
0xf4
12
? __vma_rb_erase
vmlinux 0xffffffffab3cabe1
Build id: 3.10.0-862.el7.x86_64
0x121
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 30s! [java:117922]
PKRU: 55555554
Call Trace:
[<ffffffffab276e50>] ? leave_mm+0x110/0x110
[<ffffffffab276e50>] ? leave_mm+0x110/0x110
[<ffffffffab276e50>] ? leave_mm+0x110/0x110
[<ffffffffab3078af>] smp_call_function_single+0x5f/0xa0
[<ffffffffab54a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffab307e5b>] smp_call_function_many+0x22b/0x270
[<ffffffffab277018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffab277088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffab3c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffab3c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffab3ca624>] unmap_region+0xf4/0x140
[<ffffffffab3cabe1>] ? __vma_rb_erase+0x121/0x220
Complete report #766884
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffac11117e
Build id: 3.10.0-957.1.3.el7.x86_64
0xfe
2
? leave_mm
vmlinux 0xffffffffac07a3d0
Build id: 3.10.0-957.1.3.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffac07a3d0
Build id: 3.10.0-957.1.3.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffac07a3d0
Build id: 3.10.0-957.1.3.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffac11128f
Build id: 3.10.0-957.1.3.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffac3756d5
Build id: 3.10.0-957.1.3.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffac11183b
Build id: 3.10.0-957.1.3.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffac07a598
Build id: 3.10.0-957.1.3.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffac07a608
Build id: 3.10.0-957.1.3.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffac1e42e7
Build id: 3.10.0-957.1.3.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffac1e5be5
Build id: 3.10.0-957.1.3.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffac1e8a32
Build id: 3.10.0-957.1.3.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffffac1e38fd
Build id: 3.10.0-957.1.3.el7.x86_64
0x49d
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mysqld:6847]
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [mysqld:6846]
Modules linked in: ip_set_hash_net xt_set nf_log_ipv4 nf_log_common xt_LOG xt_limit ip_set_hash_ip ip_set nfnetlink xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables nfsv3 nfs fscache vmw_vsock_vmci_transport vsock nf_conntrack_ipv4 nf_defrag_ipv4 xt_iprange xt_conntrack nf_conntrack libcrc32c iptable_filter dm_mirror dm_region_hash dm_log dm_mod ppdev vmw_balloon edac_mce_amd joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic sr_mod crct10dif_common cdrom serio_raw ata_generic pata_acpi vmwgfx floppy drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix vmxnet3 libata mptspi scsi_transport_spi drm_panel_orientation_quirks mptscsih mptbase
CPU: 3 PID: 6846 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-957.1.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff9c187ac30000 ti: ffff9c1a26af8000 task.ti: ffff9c1a26af8000
RIP: 0010:[<ffffffffac11117e>]  [<ffffffffac11117e>] generic_exec_single+0xfe/0x1b0
RSP: 0018:ffff9c1a26afbbf0  EFLAGS: 00000202
RAX: 00000000000008fd RBX: ffff9c187ac30000 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000282
RBP: ffff9c1a26afbc38 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff9c19f8eff438 R11: 00007f82941c8000 R12: ffff9c187ac30000
R13: ffffffffac6f5587 R14: ffff9c1a26afbcd0 R15: 0000000000000004
FS:  00007f8294210700(0000) GS:ffff9c1a3fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fccf3b6d000 CR3: 0000000321bf6000 CR4: 00000000000007e0
Call Trace:
[<ffffffffac07a3d0>] ? leave_mm+0x110/0x110
[<ffffffffac07a3d0>] ? leave_mm+0x110/0x110
[<ffffffffac07a3d0>] ? leave_mm+0x110/0x110
[<ffffffffac11128f>] smp_call_function_single+0x5f/0xa0
[<ffffffffac3756d5>] ? cpumask_next_and+0x35/0x50
[<ffffffffac11183b>] smp_call_function_many+0x22b/0x270
[<ffffffffac07a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffac07a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffac1e42e7>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffac1e5be5>] tlb_finish_mmu+0x55/0x60
[<ffffffffac1e8a32>] zap_page_range+0x142/0x190
[<ffffffffac1e38fd>] SyS_madvise+0x49d/0xac0
Complete report #715466
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa3116a32
Build id: 3.10.0-1160.49.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffa307ddd0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffa307ddd0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffa307ddd0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffa3116b4f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa3386a45
Build id: 3.10.0-1160.49.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa31170fb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa307e258
Build id: 3.10.0-1160.49.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa307e2cb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa31ee676
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffa31f013f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffa31f0263
Build id: 3.10.0-1160.49.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffa31f372a
Build id: 3.10.0-1160.49.1.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffffa30dada8
Build id: 3.10.0-1160.49.1.el7.x86_64
0x58
15
SyS_madvise
vmlinux 0xffffffffa31edc1d
Build id: 3.10.0-1160.49.1.el7.x86_64
0x49d
16
? __switch_to
vmlinux 0xffffffffa302b59e
Build id: 3.10.0-1160.49.1.el7.x86_64
0xce
17
system_call_fastpath
vmlinux 0xffffffffa3795f92
Build id: 3.10.0-1160.49.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [pushgateway:25938]
Modules linked in: xt_comment xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink tun vmw_vsock_vmci_transport vsock sunrpc vfat fat sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc vmw_vmci i2c_piix4 parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common mptspi libata crc32c_intel scsi_transport_spi mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ipmi_msghandler]
CPU: 1 PID: 25938 Comm: pushgateway Tainted: G        W    L ------------   3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.0.B64.1704120155 04/12/2017
task: ffff9d12b65ce300 ti: ffff9d0de2008000 task.ti: ffff9d0de2008000
RIP: 0010:[<ffffffffa3116a32>]  [<ffffffffa3116a32>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9d0de200bbe0  EFLAGS: 00000202
RAX: 0000000000000007 RBX: ffff9d0de200bbb0 RCX: 0000000000000039
RDX: 000000000000007f RSI: 0000000000000007 RDI: 0000000000000282
RBP: ffff9d0de200bc30 R08: ffffffffa3813cc0 R09: 00003ffffffff000
R10: ffff9d133ca29e58 R11: 0000000000000202 R12: ffff9d0de200bb60
R13: 0000000000000000 R14: ffff9d157fc9acc0 R15: ffffffffa30d3c1a
FS:  0000000001051b70(0063) GS:ffff9d157fc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa751568e50 CR3: 0000000095d24000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa307ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa307ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa307ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa3116b4f>] smp_call_function_single+0x5f/0xa0
[<ffffffffa3386a45>] ? cpumask_next_and+0x35/0x50
[<ffffffffa31170fb>] smp_call_function_many+0x22b/0x270
[<ffffffffa307e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa307e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa31ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa31f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa31f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffa31f372a>] zap_page_range+0xea/0x130
[<ffffffffa30dada8>] ? wake_up_q+0x58/0x80
[<ffffffffa31edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffa302b59e>] ? __switch_to+0xce/0x580
[<ffffffffa3795f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 60 15 d5 a3 48 89 df e8 87 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #686075
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffbe31109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffbe27a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffbe27a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffbe27a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffbe3111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffbe575465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffbe31175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffbe27a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffbe27a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffbe3e4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffbe3e5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffbe3e87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffffbe3e368d
Build id: 3.10.0-957.el7.x86_64
0x49d
14
tracesys
vmlinux 0xffffffffbe97505b
Build id: 3.10.0-957.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [java:142686]
Modules linked in: netfilter_edr(OE) tcp_lp fuse tun devlink ip_set nfnetlink bridge stp llc sunrpc ppdev sg virtio_balloon parport_pc parport i2c_piix4 joydev pcspkr binfmt_misc xfs libcrc32c sr_mod cdrom virtio_console virtio_blk virtio_scsi virtio_net ata_generic pata_acpi floppy cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix libata drm serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 0 PID: 142686 Comm: java Tainted: G           OEL ------------   3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff931196145140 ti: ffff931188b98000 task.ti: ffff931188b98000
RIP: 0010:[<ffffffffbe31109a>]  [<ffffffffbe31109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff931188b9bbf0  EFLAGS: 00000202
RAX: 00000000000000a0 RBX: ffff931188b9bbc0 RCX: 0000000000000020
RDX: 00000000ffffffff RSI: 00000000000000a0 RDI: 0000000000000282
RBP: ffff931188b9bc38 R08: ffffffffbea13a08 R09: 00003ffffffff000
R10: ffff93198eb8caf8 R11: 00007fe8389f2000 R12: ffff9319bffbe000
R13: ffffffffbe3c0c06 R14: ffff931188b9bc00 R15: 0000000000020200
FS:  00007fe838aef700(0000) GS:ffff93199ce00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe8c597ec80 CR3: 0000000002466000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffbe27a3d0>] ? leave_mm+0x110/0x110
[<ffffffffbe27a3d0>] ? leave_mm+0x110/0x110
[<ffffffffbe27a3d0>] ? leave_mm+0x110/0x110
[<ffffffffbe3111af>] smp_call_function_single+0x5f/0xa0
[<ffffffffbe575465>] ? cpumask_next_and+0x35/0x50
[<ffffffffbe31175b>] smp_call_function_many+0x22b/0x270
[<ffffffffbe27a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbe27a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffbe3e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffbe3e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffbe3e87c2>] zap_page_range+0x142/0x190
[<ffffffffbe3e368d>] SyS_madvise+0x49d/0xac0
[<ffffffffbe97505b>] tracesys+0xa3/0xc9
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 f4 be 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #639658
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f932a
Build id: 3.10.0-693.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106e4a0
Build id: 3.10.0-693.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106e4a0
Build id: 3.10.0-693.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106e4a0
Build id: 3.10.0-693.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f942f
Build id: 3.10.0-693.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813226f5
Build id: 3.10.0-693.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f99c3
Build id: 3.10.0-693.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
10
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff811b08fa
Build id: 3.10.0-693.el7.x86_64
0x13a
13
? wake_up_q
vmlinux 0xffffffff810c47c8
Build id: 3.10.0-693.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffff811ac71d
Build id: 3.10.0-693.el7.x86_64
0x3cd
15
? SyS_futex
vmlinux 0xffffffff810f8040
Build id: 3.10.0-693.el7.x86_64
0x80
16
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [containerd:8941]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) tcp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg nfit vmw_vmci libnvdimm i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm ata_piix drm mptspi libata scsi_transport_spi crct10dif_pclmul mptscsih crct10dif_common crc32c_intel mptbase e1000 serio_raw i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 8941 Comm: containerd Tainted: G               ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8801b0e33f40 ti: ffff8801bb278000 task.ti: ffff8801bb278000
RIP: 0010:[<ffffffff810f932a>]  [<ffffffff810f932a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8801bb27bc20  EFLAGS: 00000202
RAX: 0000000000000040 RBX: ffff88017c9e5000 RCX: 0000000000000005
RDX: 0000000000000040 RSI: 0000000000000000 RDI: 0000000000000286
RBP: ffff8801bb27bc70 R08: ffffffff816cee28 R09: 000000c000bce000
R10: 00000000000001e7 R11: 0000000000000202 R12: ffff88017fc19000
R13: 000000005076d999 R14: 0000000000000000 R15: 0000000000000000
FS:  00007fa40ffff700(0000) GS:ffff8802367c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa43c5b2098 CR3: 0000000206367000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106e4a0 ffff8801bb27bcf8
0000000000000003 000000005076d999 ffff8801bb27bc88 0000000000000004
ffffffff8106e4a0 ffffffff8106e4a0 ffff8801bb27bca0 ffffffff810f942f
Call Trace:
[<ffffffff8106e4a0>] ? leave_mm+0x70/0x70
[<ffffffff8106e4a0>] ? leave_mm+0x70/0x70
[<ffffffff8106e4a0>] ? leave_mm+0x70/0x70
[<ffffffff810f942f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813226f5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f99c3>] smp_call_function_many+0x223/0x260
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b08fa>] zap_page_range+0x13a/0x180
[<ffffffff810c47c8>] ? wake_up_q+0x58/0x80
[<ffffffff811ac71d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff810f8040>] ? SyS_futex+0x80/0x180
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 00 78 01 00 48 89 de 48 03 14 c5 e0 fd b0 81 48 89 df e8 7a da 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #588485
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810fcf5a
Build id: 3.10.0-693.21.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff810709d0
Build id: 3.10.0-693.21.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff810709d0
Build id: 3.10.0-693.21.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff810709d0
Build id: 3.10.0-693.21.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff810fd06f
Build id: 3.10.0-693.21.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813288a5
Build id: 3.10.0-693.21.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810fd61b
Build id: 3.10.0-693.21.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff81070b98
Build id: 3.10.0-693.21.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81070c68
Build id: 3.10.0-693.21.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.63
vmlinux 0xffffffff811b18c3
Build id: 3.10.0-693.21.1.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811b2be5
Build id: 3.10.0-693.21.1.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff811b516a
Build id: 3.10.0-693.21.1.el7.x86_64
0x13a
13
SyS_madvise
vmlinux 0xffffffff811b0f8d
Build id: 3.10.0-693.21.1.el7.x86_64
0x3cd
14
system_call_fastpath
vmlinux 0xffffffff816c0715
Build id: 3.10.0-693.21.1.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 33s! [kworker/0:3:7131]
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s! [mysqld:7155]
Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd pcspkr joydev sg parport_pc parport shpchp vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper
 syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul mptspi crct10dif_common crc32c_intel scsi_transport_spi serio_raw ata_piix mptscsih libata mptbase vmxnet3 i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 7155 Comm: mysqld Tainted: G             L ------------   3.10.0-693.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff881ef1f03f40 ti: ffff880501eb8000 task.ti: ffff880501eb8000
RIP: 0010:[<ffffffff810fcf5a>]  [<ffffffff810fcf5a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff880501ebbbf0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff880501ebbbc0 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff880501ebbc40 R08: ffffffff816dcb60 R09: 00007f2d2414d000
R10: 00007f2d28bd6240 R11: 00007f2d2410b000 R12: ffffffff00a2bd52
R13: 0000000000007530 R14: ffffffff8158f494 R15: ffff880501ebbb80
FS:  00007f2d24152700(0000) GS:ffff881f8fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fef0f10f000 CR3: 0000001efcbc8000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 [<ffffffff810709d0>] ? leave_mm+0x110/0x110
 [<ffffffff810709d0>] ? leave_mm+0x110/0x110
 [<ffffffff810709d0>] ? leave_mm+0x110/0x110
 [<ffffffff810fd06f>] smp_call_function_single+0x5f/0xa0
 [<ffffffff813288a5>] ? cpumask_next_and+0x35/0x50
 [<ffffffff810fd61b>] smp_call_function_many+0x22b/0x270
 [<ffffffff81070b98>] native_flush_tlb_others+0xb8/0xc0
 [<ffffffff81070c68>] flush_tlb_mm_range+0x68/0x140
 [<ffffffff811b18c3>] tlb_flush_mmu.part.63+0x33/0xc0
 [<ffffffff811b2be5>] tlb_finish_mmu+0x55/0x60
 [<ffffffff811b516a>] zap_page_range+0x13a/0x180
 [<ffffffff811b0f8d>] SyS_madvise+0x3cd/0x9c0
 [<ffffffff816c0715>] system_call_fastpath+0x1c/0x21
Code: 80 95 01 00 48 89 de 48 03 14 c5 a0 10 b2 81 48 89 df e8 7a 03 24 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #575849
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9d90779a
Build id: 3.10.0-862.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff9d876e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff9d876e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff9d876e50
Build id: 3.10.0-862.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff9d9078af
Build id: 3.10.0-862.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9db4a275
Build id: 3.10.0-862.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9d907e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9d877018
Build id: 3.10.0-862.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9d877088
Build id: 3.10.0-862.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffff9d9c0497
Build id: 3.10.0-862.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff9d9c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff9d9c4a02
Build id: 3.10.0-862.el7.x86_64
0x142
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [java:82027]
Modules linked in: af_packet_diag netlink_diag vsock_diag sctp_diag sctp tcp_diag udp_diag inet_diag unix_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink binfmt_misc bnep vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi snd_ac97_codec ppdev ac97_bus aesni_intel lrw snd_seq gf128mul btusb glue_helper ablk_helper cryptd btrtl btbcm btintel snd_seq_device bluetooth snd_pcm snd_timer joydev pcspkr snd sg rfkill soundcore nfit i2c_piix4 vmw_vmci
shpchp libnvdimm parport_pc parport ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi drm mptscsih ata_piix crct10dif_pclmul crct10dif_common e1000 crc32c_intel libata mptbase serio_raw i2c_core
CPU: 1 PID: 82027 Comm: java Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9b2247fb8fd0 ti: ffff9b22a6f44000 task.ti: ffff9b22a6f44000
RIP: 0010:[<ffffffff9d90779a>]  [<ffffffff9d90779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9b22a6f47bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9b22a6f47bc0 RCX: 0000000000000004
RDX: ffffffff9e0138f0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9b22a6f47c38 R08: ffffffff9e0138e8 R09: ffff9b1f32e5e438
R10: 0000000000000008 R11: 00007f864b8fc000 R12: ffff9b229a7e1d00
R13: 00000000bc1d6a46 R14: 00007f864b8fc000 R15: ffff9b2247fb8fd0
FS:  00007f864b9f9700(0000) GS:ffff9b22ad640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1d74962000 CR3: 00000003e32ee000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9d876e50>] ? leave_mm+0x110/0x110
[<ffffffff9d876e50>] ? leave_mm+0x110/0x110
[<ffffffff9d876e50>] ? leave_mm+0x110/0x110
[<ffffffff9d9078af>] smp_call_function_single+0x5f/0xa0
[<ffffffff9db4a275>] ? cpumask_next_and+0x35/0x50
[<ffffffff9d907e5b>] smp_call_function_many+0x22b/0x270
[<ffffffff9d877018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9d877088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff9d9c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff9d9c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff9d9c4a02>] zap_page_range+0x142/0x190
Complete report #506886
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f932a
Build id: 3.10.0-693.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106e4a0
Build id: 3.10.0-693.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106e4a0
Build id: 3.10.0-693.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106e4a0
Build id: 3.10.0-693.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f942f
Build id: 3.10.0-693.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813226f5
Build id: 3.10.0-693.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f99c3
Build id: 3.10.0-693.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
10
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff811b08fa
Build id: 3.10.0-693.el7.x86_64
0x13a
13
SyS_madvise
vmlinux 0xffffffff811ac71d
Build id: 3.10.0-693.el7.x86_64
0x3cd
14
? __audit_syscall_entry
vmlinux 0xffffffff8111f384
Build id: 3.10.0-693.el7.x86_64
0xb4
15
tracesys
vmlinux 0xffffffff816b51d2
Build id: 3.10.0-693.el7.x86_64
0xdd
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [java:110144]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371
snd_rawmidi snd_ac97_codec ac97_bus snd_seq ppdev snd_seq_device vmw_balloon snd_pcm pcspkr joydev snd_timer snd i2c_piix4 soundcore vmw_vmci sg shpchp nfit libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libata mptspi scsi_transport_spi mptscsih serio_raw mptbase e1000 i2c_core
CPU: 1 PID: 110144 Comm: java Tainted: G             L ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff880305b19fa0 ti: ffff8801f7b6c000 task.ti: ffff8801f7b6c000
RIP: 0010:[<ffffffff810f932a>]  [<ffffffff810f932a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8801f7b6fc20  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8801f7b6fbf0 RCX: 0000000000000004
RDX: ffffffff816cebb0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8801f7b6fc70 R08: ffffffff816ceba8 R09: ffffea0000a58480
R10: ffffea0000a58480 R11: 0000000000000246 R12: ffff88037fbd8000
R13: ffffffff8118caf6 R14: ffff8801f7b6fc30 R15: 0000000000020200
FS:  00007f0a8657f700(0000) GS:ffff880370240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0a557a8000 CR3: 00000001bddb0000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106e4a0 ffff8801f7b6fcf8
0000000000000003 00000000d64c985f ffff8801f7b6fc88 0000000000000003
ffffffff8106e4a0 ffffffff8106e4a0 ffff8801f7b6fca0 ffffffff810f942f
Call Trace:
[<ffffffff8106e4a0>] ? leave_mm+0x70/0x70
[<ffffffff8106e4a0>] ? leave_mm+0x70/0x70
[<ffffffff8106e4a0>] ? leave_mm+0x70/0x70
[<ffffffff810f942f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813226f5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f99c3>] smp_call_function_many+0x223/0x260
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b08fa>] zap_page_range+0x13a/0x180
[<ffffffff811ac71d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff8111f384>] ? __audit_syscall_entry+0xb4/0x110
[<ffffffff816b51d2>] tracesys+0xdd/0xe2
Code: 00 78 01 00 48 89 de 48 03 14 c5 e0 fd b0 81 48 89 df e8 7a da 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #499749
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675a
Build id: 3.10.0-327.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64
0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff8119568a
Build id: 3.10.0-327.el7.x86_64
0x12a
13
SyS_madvise
vmlinux 0xffffffff81191db4
Build id: 3.10.0-327.el7.x86_64
0x394
14
? __switch_to
vmlinux 0xffffffff8101360b
Build id: 3.10.0-327.el7.x86_64
0x17b
15
? __schedule
vmlinux 0xffffffff8163a2b8
Build id: 3.10.0-327.el7.x86_64
0x2d8
16
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64
0x16
BUG: soft lockup - CPU#1 stuck for 23s! [server.rb:286:57944]
Modules linked in: tcp_lp nls_utf8 isofs bnep bluetooth rfkill fuse tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables lp vmhgfs(OE) vmw_vsock_vmci_transport vsock ipt_REJECT xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_CHECKSUM iptable_mangle dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw snd_seq gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm snd_timer snd pcspkr vmw_vmci ppdev soundcore vmw_balloon sg i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif
crct10dif_generic sr_mod cdrom ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw ttm ata_piix libata drm mptspi scsi_transport_spi mptscsih e1000 mptbase i2c_core floppy
CPU: 1 PID: 57944 Comm: server.rb:286 Tainted: G           OEL ------------   3.10.0-327.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
task: ffff8801a78b1700 ti: ffff88012b8c8000 task.ti: ffff88012b8c8000
RIP: 0010:[<ffffffff810e675a>]  [<ffffffff810e675a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88012b8cbc40  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88012b8cbc10 RCX: 0000000000000001
RDX: ffffffff8165e7f0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88012b8cbc88 R08: ffffffff8165e7e8 R09: 00007f4be95fe000
R10: 0000000000001000 R11: 0000000000000202 R12: ffff88012b8cbbc0
R13: ffff88012b8cbd63 R14: ffffffff812fc122 R15: ffff88012b8cbbb8
FS:  00007f4beaeff700(0000) GS:ffff880236620000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f315b6b4750 CR3: 00000002080fc000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff88012b8cbd10
0000000000000003 000000008f112638 0000000000000000 ffffffff81065c90
ffffffff81065c90 ffff88012b8cbcb8 ffffffff810e685f ffff88012b8cbcb8
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119568a>] zap_page_range+0x12a/0x170
[<ffffffff81191db4>] SyS_madvise+0x394/0x820
[<ffffffff8101360b>] ? __switch_to+0x17b/0x4b0
[<ffffffff8163a2b8>] ? __schedule+0x2d8/0x900
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 40 52 01 00 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #490163
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffaed0779a
Build id: 3.10.0-862.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffaec76e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffaec76e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffaec76e50
Build id: 3.10.0-862.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffaed078af
Build id: 3.10.0-862.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffaef4a275
Build id: 3.10.0-862.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffaed07e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffaec77018
Build id: 3.10.0-862.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffaec77088
Build id: 3.10.0-862.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffffaedc0497
Build id: 3.10.0-862.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffaedc1c75
Build id: 3.10.0-862.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffaedc4a02
Build id: 3.10.0-862.el7.x86_64
0x142
13
? do_sync_write
vmlinux 0xffffffffaee1a1f3
Build id: 3.10.0-862.el7.x86_64
0x93
14
SyS_madvise
vmlinux 0xffffffffaedbfafd
Build id: 3.10.0-862.el7.x86_64
0x3cd
15
? vfs_write
vmlinux 0xffffffffaee1ad92
Build id: 3.10.0-862.el7.x86_64
0x182
16
system_call_fastpath
vmlinux 0xffffffffaf31f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [SystemLogFlush:1503]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 iosf_mbi ppdev crc32_pclmul snd_ens1371 btusb btrtl snd_rawmidi ghash_clmulni_intel btbcm btintel snd_ac97_codec bluetooth ac97_bus vmw_balloon snd_seq aesni_intel snd_seq_device lrw snd_pcm gf128mul glue_helper ablk_helper cryptd i2c_piix4 nfit rfkill joydev pcspkr snd_timer sg snd vmw_vmci soundcore libnvdimm shpchp parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ata_piix ttm mptspi crct10dif_pclmul crct10dif_common libata crc32c_intel drm scsi_transport_spi mptscsih serio_raw e1000 mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 1503 Comm: SystemLogFlush Not tainted 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff894232d3eeb0 ti: ffff894321b7c000 task.ti: ffff894321b7c000
RIP: 0010:[<ffffffffaed0779a>]  [<ffffffffaed0779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff894321b7fbf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff894321b7fbc0 RCX: 0000000000000001
RDX: ffffffffaf413170 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff894321b7fc38 R08: ffffffffaf413168 R09: ffff894236a69950
R10: ffff89433fbd5f40 R11: ffffffffffffffff R12: ffff894321b7fbe8
R13: ffff8943349f0ca0 R14: 0000000000000062 R15: 0000000001d2dd11
FS:  00007f21c2bfb700(0000) GS:ffff894337740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9f76d2c9e5 CR3: 00000000bb52c000 CR4: 00000000003607e0
Call Trace:
[<ffffffffaec76e50>] ? leave_mm+0x110/0x110
[<ffffffffaec76e50>] ? leave_mm+0x110/0x110
[<ffffffffaec76e50>] ? leave_mm+0x110/0x110
[<ffffffffaed078af>] smp_call_function_single+0x5f/0xa0
[<ffffffffaef4a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffaed07e5b>] smp_call_function_many+0x22b/0x270
[<ffffffffaec77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaec77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffaedc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffaedc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffaedc4a02>] zap_page_range+0x142/0x190
[<ffffffffaee1a1f3>] ? do_sync_write+0x93/0xe0
[<ffffffffaedbfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffffaee1ad92>] ? vfs_write+0x182/0x1f0
[<ffffffffaf31f7d5>] system_call_fastpath+0x1c/0x21
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 93 af 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #450023
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9f90779a
Build id: 3.10.0-862.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff9f876e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff9f876e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff9f876e50
Build id: 3.10.0-862.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff9f9078af
Build id: 3.10.0-862.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9fb4a275
Build id: 3.10.0-862.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9f907e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9f877018
Build id: 3.10.0-862.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9f877088
Build id: 3.10.0-862.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffff9f9c0497
Build id: 3.10.0-862.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff9f9c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffff9f9ca624
Build id: 3.10.0-862.el7.x86_64
0xf4
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [vmtoolsd:834]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache vmhgfs(OE) vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul snd_ens1371 snd_rawmidi ghash_clmulni_intel snd_ac97_codec ppdev ac97_bus snd_seq snd_seq_device aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon snd_pcm snd_timer pcspkr joydev snd sg soundcore nfit libnvdimm parport_pc parport shpchp i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm ata_piix libata mptspi scsi_transport_spi
e1000 mptscsih mptbase i2c_core
CPU: 1 PID: 834 Comm: vmtoolsd Kdump: loaded Tainted: G           OEL ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff89a1ae225ee0 ti: ffff89a03ac84000 task.ti: ffff89a03ac84000
RIP: 0010:[<ffffffff9f90779a>]  [<ffffffff9f90779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff89a03ac87c30  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff89a03ac87c00 RCX: 0000000000000001
RDX: ffffffffa0013170 RSI: 0000000000000080 RDI: 0000000000000286
RBP: ffff89a03ac87c78 R08: ffffffffa0013168 R09: ffffffff9f9d35bc
R10: ffff89a1b565ba40 R11: fffff58a88b49240 R12: fffff58a88acf770
R13: ffff89a02e1fa000 R14: ffffffff9f9d2058 R15: ffff89a03ac87bb0
FS:  00007fb12ae60880(0000) GS:ffff89a1b5640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb12ae57000 CR3: 000000022ca5e000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9f876e50>] ? leave_mm+0x110/0x110
[<ffffffff9f876e50>] ? leave_mm+0x110/0x110
[<ffffffff9f876e50>] ? leave_mm+0x110/0x110
[<ffffffff9f9078af>] smp_call_function_single+0x5f/0xa0
[<ffffffff9fb4a275>] ? cpumask_next_and+0x35/0x50
[<ffffffff9f907e5b>] smp_call_function_many+0x22b/0x270
[<ffffffff9f877018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9f877088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff9f9c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff9f9c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff9f9ca624>] unmap_region+0xf4/0x140
Complete report #444484
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa5914a22
Build id: 3.10.0-1062.1.2.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffa587d1d0
Build id: 3.10.0-1062.1.2.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffa587d1d0
Build id: 3.10.0-1062.1.2.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffa587d1d0
Build id: 3.10.0-1062.1.2.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffa5914b3f
Build id: 3.10.0-1062.1.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa5b7f475
Build id: 3.10.0-1062.1.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa59150eb
Build id: 3.10.0-1062.1.2.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa587d398
Build id: 3.10.0-1062.1.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa587d40b
Build id: 3.10.0-1062.1.2.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa59ea166
Build id: 3.10.0-1062.1.2.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffa59ebc2f
Build id: 3.10.0-1062.1.2.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffa59ebd53
Build id: 3.10.0-1062.1.2.el7.x86_64
0x23
13
unmap_region
vmlinux 0xffffffffa59f5544
Build id: 3.10.0-1062.1.2.el7.x86_64
0xf4
14
? __vma_rb_erase
vmlinux 0xffffffffa59f5b01
Build id: 3.10.0-1062.1.2.el7.x86_64
0x121
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 24s! [7_scheduler:19349]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun bridge stp llc devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc vmw_vmci i2c_piix4 parport ip_tables xfs libcrc32c
sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic ahci drm ata_piix libahci libata nfit libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 19349 Comm: 7_scheduler Tainted: G        W    L ------------ T 3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff96e9918ca0e0 ti: ffff96e977df4000 task.ti: ffff96e977df4000
RIP: 0010:[<ffffffffa5914a22>]  [<ffffffffa5914a22>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff96e977df7c20  EFLAGS: 00000202
RAX: 0000000000000010 RBX: ffff96eb3ffda000 RCX: 0000000000000030
RDX: 0000000000000010 RSI: 0000000000000010 RDI: 0000000000000286
RBP: ffff96e977df7c70 R08: ffffffffa6013a40 R09: ffffffffa59fe7e8
R10: ffff96ea3bd9f0a0 R11: fffffee8440b3440 R12: 0000000000000000
R13: ffff96ea3ffdb008 R14: 0000000000000008 R15: ffff96ea3ffdb000
FS:  00007f5985a2d700(0000) GS:ffff96ea3bd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f597e825000 CR3: 00000001c2720000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa587d1d0>] ? leave_mm+0x110/0x110
[<ffffffffa587d1d0>] ? leave_mm+0x110/0x110
[<ffffffffa587d1d0>] ? leave_mm+0x110/0x110
[<ffffffffa5914b3f>] smp_call_function_single+0x5f/0xa0
[<ffffffffa5b7f475>] ? cpumask_next_and+0x35/0x50
[<ffffffffa59150eb>] smp_call_function_many+0x22b/0x270
[<ffffffffa587d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa587d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa59ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa59ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa59ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffa59f5544>] unmap_region+0xf4/0x140
[<ffffffffa59f5b01>] ? __vma_rb_erase+0x121/0x220
Complete report #431550
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff89d1109e
Build id: 3.10.0-957.el7.x86_64
0xfe
2
? leave_mm
vmlinux 0xffffffff89c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff89c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff89c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff89d111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff89f75465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff89d1175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff89c7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff89c7a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff89de4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff89de5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff89de87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffff89de368d
Build id: 3.10.0-957.el7.x86_64
0x49d
14
system_call_fastpath
vmlinux 0xffffffff8a374ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [sdatad:305214]
Modules linked in: dattobd(OE) sfs(OE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc nfit libnvdimm iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr vmw_balloon joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom
crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm mptspi scsi_transport_spi mptscsih ahci vmxnet3 libahci mptbase drm_panel_orientation_quirks ata_piix libata dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dattobd]
CPU: 15 PID: 305214 Comm: sdatad Kdump: loaded Tainted: G        W  OEL ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff98692b74e180 ti: ffff986919f08000 task.ti: ffff986919f08000
RIP: 0010:[<ffffffff89d1109e>]  [<ffffffff89d1109e>] generic_exec_single+0xfe/0x1b0
RSP: 0018:ffff986919f0bbf0  EFLAGS: 00000202
RAX: 0000000000000014 RBX: ffff986919f0bbc0 RCX: 000000000000002c
RDX: 00000000000fffff RSI: 0000000000000014 RDI: 0000000000000282
RBP: ffff986919f0bc38 R08: ffffffff8a413f00 R09: 00003ffffffff000
R10: ffff986d27761bd8 R11: 0000000000000206 R12: ffff986919f0bbb0
R13: 0000000000000001 R14: ffff986d2cb1ac00 R15: ffff98692b74e1e8
FS:  00007f781ce7a700(0000) GS:ffff986d2cb00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc44caaa000 CR3: 0000000773a3c000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff89c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff89c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff89c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff89d111af>] smp_call_function_single+0x5f/0xa0
[<ffffffff89f75465>] ? cpumask_next_and+0x35/0x50
[<ffffffff89d1175b>] smp_call_function_many+0x22b/0x270
[<ffffffff89c7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff89c7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff89de4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff89de5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff89de87c2>] zap_page_range+0x142/0x190
[<ffffffff89de368d>] SyS_madvise+0x49d/0xac0
[<ffffffff8a374ddb>] system_call_fastpath+0x22/0x27
Code: 48 89 de 48 03 14 c5 60 b9 94 8a 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 f6 43 20 01 <75> f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 00 0f 85 87 
Complete report #386282
Frame # Function Binary Source or offset Line
1
? leave_mm
vmlinux 0xffffffff86876e50
Build id: 3.10.0-862.el7.x86_64
0x110
2
? leave_mm
vmlinux 0xffffffff86876e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff86876e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
smp_call_function_single
vmlinux 0xffffffff869078af
Build id: 3.10.0-862.el7.x86_64
0x5f
5
? cpumask_next_and
vmlinux 0xffffffff86b4a275
Build id: 3.10.0-862.el7.x86_64
0x35
6
smp_call_function_many
vmlinux 0xffffffff86907e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
7
native_flush_tlb_others
vmlinux 0xffffffff86877018
Build id: 3.10.0-862.el7.x86_64
0xb8
8
flush_tlb_mm_range
vmlinux 0xffffffff86877088
Build id: 3.10.0-862.el7.x86_64
0x68
9
tlb_flush_mmu.part.75
vmlinux 0xffffffff869c0497
Build id: 3.10.0-862.el7.x86_64
0x37
10
tlb_finish_mmu
vmlinux 0xffffffff869c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
11
zap_page_range
vmlinux 0xffffffff869c4a02
Build id: 3.10.0-862.el7.x86_64
0x142
12
SyS_madvise
vmlinux 0xffffffff869bfafd
Build id: 3.10.0-862.el7.x86_64
0x3cd
13
? ktime_get_ts64
vmlinux 0xffffffff868f76e2
Build id: 3.10.0-862.el7.x86_64
0x52
14
system_call_fastpath
vmlinux 0xffffffff86f1f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [go-jxhttp_idip:2677]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd joydev pcspkr nfit sg libnvdimm vmw_vmci i2c_piix4 shpchp parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom
crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common ata_piix crc32c_intel libata e1000 mptspi scsi_transport_spi i2c_core mptscsih mptbase serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 2677 Comm: go-jxhttp_idip Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9e1fdd366eb0 ti: ffff9e1fb2644000 task.ti: ffff9e1fb2644000
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9e1fdd366eb0 ti: ffff9e1fb2644000 task.ti: ffff9e1fb2644000
RSP: 0018:ffff9e1fb2647bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9e1fb2647bc0 RCX: 0000000000000003
RDX: ffffffff87013670 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9e1fb2647c38 R08: ffffffff87013668 R09: ffff9e1c7bbebe60
R10: 00007f549ca96db8 R11: 0000000000000246 R12: ffff9e1fed6d8b00
R13: 0000000300000000 R14: ffff9e1fed618b00 R15: ffff9e1d398a3800
FS:  00007f549ca97700(0000) GS:ffff9e1fed6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007efe4d5d6e70 CR3: 00000003f1fc2000 CR4: 00000000001607e0
Call Trace:
[<ffffffff86876e50>] ? leave_mm+0x110/0x110
[<ffffffff86876e50>] ? leave_mm+0x110/0x110
[<ffffffff86876e50>] ? leave_mm+0x110/0x110
[<ffffffff869078af>] smp_call_function_single+0x5f/0xa0
[<ffffffff86b4a275>] ? cpumask_next_and+0x35/0x50
[<ffffffff86907e5b>] smp_call_function_many+0x22b/0x270
[<ffffffff86877018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff86877088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff869c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff869c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff869c4a02>] zap_page_range+0x142/0x190
[<ffffffff869bfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff868f76e2>] ? ktime_get_ts64+0x52/0xf0
[<ffffffff86f1f7d5>] system_call_fastpath+0x1c/0x21
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 53 87 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #377973
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff93116a12
Build id: 3.10.0-1160.36.2.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff9307ddd0
Build id: 3.10.0-1160.36.2.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9307ddd0
Build id: 3.10.0-1160.36.2.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9307ddd0
Build id: 3.10.0-1160.36.2.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff93116b2f
Build id: 3.10.0-1160.36.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff933869f5
Build id: 3.10.0-1160.36.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff931170db
Build id: 3.10.0-1160.36.2.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9307e258
Build id: 3.10.0-1160.36.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9307e2cb
Build id: 3.10.0-1160.36.2.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff931ee656
Build id: 3.10.0-1160.36.2.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff931f011f
Build id: 3.10.0-1160.36.2.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff931f0243
Build id: 3.10.0-1160.36.2.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff931f370a
Build id: 3.10.0-1160.36.2.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffff930dad88
Build id: 3.10.0-1160.36.2.el7.x86_64
0x58
15
SyS_madvise
vmlinux 0xffffffff931edbfd
Build id: 3.10.0-1160.36.2.el7.x86_64
0x49d
16
system_call_fastpath
vmlinux 0xffffffff93795f92
Build id: 3.10.0-1160.36.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [docker-proxy:2758]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi ext4 snd_ac97_codec ac97_bus mbcache snd_seq jbd2 snd_seq_device snd_pcm snd_timer crc32_pclmul ghash_clmulni_intel snd aesni_intel
lrw gf128mul glue_helper ppdev soundcore ablk_helper cryptd sg vmw_balloon joydev pcspkr parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper nfit syscopyarea sysfillrect sysimgblt fb_sys_fops libnvdimm ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel mptspi scsi_transport_spi mptscsih mptbase libata serio_raw e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 2758 Comm: docker-proxy Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.36.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff8aebde783180 ti: ffff8ae7f9388000 task.ti: ffff8ae7f9388000
RIP: 0010:[<ffffffff93116a12>]  [<ffffffff93116a12>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8ae7f938bbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8ae7f938bbb0 RCX: 0000000000000001
RDX: ffffffff938137d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8ae7f938bc30 R08: ffffffff938137c8 R09: 00003ffffffff000
R10: ffff8aebdee495f0 R11: 0000000000000202 R12: ffff8aebf7ef8b40
R13: ffff8aebf7ef8b40 R14: 00000000000003ad R15: ffff8aec05650200
FS:  00007f64d8de0700(0000) GS:ffff8aec0b440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000141010 CR3: 000000049ecb4000 CR4: 0000000000360fe0
Call Trace:
[<ffffffff9307ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9307ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9307ddd0>] ? leave_mm+0x120/0x120
[<ffffffff93116b2f>] smp_call_function_single+0x5f/0xa0
[<ffffffff933869f5>] ? cpumask_next_and+0x35/0x50
[<ffffffff931170db>] smp_call_function_many+0x22b/0x270
[<ffffffff9307e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9307e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff931ee656>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff931f011f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff931f0243>] tlb_finish_mmu+0x23/0x30
[<ffffffff931f370a>] zap_page_range+0xea/0x130
[<ffffffff930dad88>] ? wake_up_q+0x58/0x80
[<ffffffff931edbfd>] SyS_madvise+0x49d/0xac0
[<ffffffff93795f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 60 15 d5 93 48 89 df e8 57 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #376794
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa291109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffa287a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffa287a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffa287a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffa29111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa2b75465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa291175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa287a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa287a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffa29e4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffa29e5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffa29e87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
? kmem_cache_free
vmlinux 0xffffffffa2a1b762
Build id: 3.10.0-957.el7.x86_64
0x1e2
14
SyS_madvise
vmlinux 0xffffffffa29e368d
Build id: 3.10.0-957.el7.x86_64
0x49d
15
system_call_fastpath
vmlinux 0xffffffffa2f74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [replexec-71:42505]
Modules linked in: ipt_rpfilter xt_set iptable_raw ip_set_hash_ip ip_set_hash_net ip_set ipip tunnel4 ip_tunnel xt_statistic xt_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc coretemp kvm_intel ppdev kvm irqbypass crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_vs_ftp nf_nat ip_vs_sed ip_vs_nq
ip_vs_sh ip_vs_dh ip_vs_lblcr ip_vs_lblc ip_vs_wrr ip_vs_rr ip_vs_wlc ip_vs_lc ip_vs nf_conntrack ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_pclmul crct10dif_common crc32c_intel ata_generic pata_acpi serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi e1000 libata mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 42505 Comm: replexec-71 Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff996e72b330c0 ti: ffff996e134c0000 task.ti: ffff996e134c0000
RIP: 0010:[<ffffffffa291109a>]  [<ffffffffa291109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff996e134c3bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff996e134c3bc0 RCX: 0000000000000001
RDX: ffffffffa3012d90 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff996e134c3c38 R08: ffffffffa3012d88 R09: 00003ffffffff000
R10: ffff996dde96cca8 R11: 00007f1ad5a39000 R12: ffff996e134c3bb0
R13: 0000000000000001 R14: ffff996df5de5c00 R15: ffff996e72b33128
FS:  00007f1ad6238700(0000) GS:ffff996ef7ac0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007efe4ce7d000 CR3: 00000000adf44000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa287a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa287a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa287a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa29111af>] smp_call_function_single+0x5f/0xa0
[<ffffffffa2b75465>] ? cpumask_next_and+0x35/0x50
[<ffffffffa291175b>] smp_call_function_many+0x22b/0x270
[<ffffffffa287a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa287a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa29e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffa29e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffa29e87c2>] zap_page_range+0x142/0x190
[<ffffffffa2a1b762>] ? kmem_cache_free+0x1e2/0x200
[<ffffffffa29e368d>] SyS_madvise+0x49d/0xac0
[<ffffffffa2f74ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 54 a3 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #358181
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810fcf5a
Build id: 3.10.0-693.21.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff810709d0
Build id: 3.10.0-693.21.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff810709d0
Build id: 3.10.0-693.21.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff810709d0
Build id: 3.10.0-693.21.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff810fd06f
Build id: 3.10.0-693.21.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813288a5
Build id: 3.10.0-693.21.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810fd61b
Build id: 3.10.0-693.21.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff81070b98
Build id: 3.10.0-693.21.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81070c68
Build id: 3.10.0-693.21.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.63
vmlinux 0xffffffff811b18c3
Build id: 3.10.0-693.21.1.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811b2be5
Build id: 3.10.0-693.21.1.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff811b516a
Build id: 3.10.0-693.21.1.el7.x86_64
0x13a
13
SyS_madvise
vmlinux 0xffffffff811b0f8d
Build id: 3.10.0-693.21.1.el7.x86_64
0x3cd
14
? __do_page_fault
vmlinux 0xffffffff816bb521
Build id: 3.10.0-693.21.1.el7.x86_64
0x171
15
? __do_page_fault
vmlinux 0xffffffff816bb54f
Build id: 3.10.0-693.21.1.el7.x86_64
0x19f
16
system_call_fastpath
vmlinux 0xffffffff816c0715
Build id: 3.10.0-693.21.1.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [DOM Worker:2915]
Modules linked in: fuse nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac edac_core iosf_mbi crc32_pclmul snd_ens1371 snd_rawmidi ghash_clmulni_intel snd_ac97_codec ppdev ac97_bus snd_seq aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm vmw_balloon pcspkr joydev snd_timer
sg snd vmw_vmci soundcore nfit i2c_piix4 shpchp libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul ttm crct10dif_common crc32c_intel drm ata_piix libata serio_raw mptspi scsi_transport_spi mptscsih mptbase e1000 i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 2915 Comm: DOM Worker Tainted: G             L ------------   3.10.0-693.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff8800bad89fa0 ti: ffff8801d96c8000 task.ti: ffff8801d96c8000
RIP: 0010:[<ffffffff810fcf5a>]  [<ffffffff810fcf5a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8801d96cbbf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8801d96cbbc0 RCX: 0000000000000001
RDX: ffffffff816dc8f0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8801d96cbc40 R08: ffffffff816dc8e8 R09: ffffea00088cf0c0
R10: ffffea00088cf0c0 R11: 0000000000000206 R12: ffff88023ffd8000
R13: ffffffff81191016 R14: ffff8801d96cbc00 R15: 0000000000020200
FS:  00007fdb50cff700(0000) GS:ffff880235640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdb4addb000 CR3: 00000000b163e000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff810709d0>] ? leave_mm+0x110/0x110
[<ffffffff810709d0>] ? leave_mm+0x110/0x110
[<ffffffff810709d0>] ? leave_mm+0x110/0x110
[<ffffffff810fd06f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813288a5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810fd61b>] smp_call_function_many+0x22b/0x270
[<ffffffff81070b98>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81070c68>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811b18c3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811b2be5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b516a>] zap_page_range+0x13a/0x180
[<ffffffff811b0f8d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff816bb521>] ? __do_page_fault+0x171/0x450
[<ffffffff816bb54f>] ? __do_page_fault+0x19f/0x450
[<ffffffff816c0715>] system_call_fastpath+0x1c/0x21
Code: 80 95 01 00 48 89 de 48 03 14 c5 a0 10 b2 81 48 89 df e8 7a 03 24 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #245259
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e675e
Build id: 3.10.0-327.el7.x86_64
0xfe
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810e685f
Build id: 3.10.0-327.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f2785
Build id: 3.10.0-327.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810e6e03
Build id: 3.10.0-327.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.el7.x86_64
0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff81192563
Build id: 3.10.0-327.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811930f5
Build id: 3.10.0-327.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff8119568a
Build id: 3.10.0-327.el7.x86_64
0x12a
13
SyS_madvise
vmlinux 0xffffffff81191db4
Build id: 3.10.0-327.el7.x86_64
0x394
14
? SyS_futex
vmlinux 0xffffffff810e55a0
Build id: 3.10.0-327.el7.x86_64
0x80
15
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64
0x16
BUG: soft lockup - CPU#5 stuck for 23s! [node_exporter:1065]
Modules linked in: binfmt_misc tcp_diag inet_diag xt_conntrack ipt_REJECT tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat vmw_vsock_vmci_transport nf_conntrack vsock xt_CHECKSUM iptable_mangle ext4 mbcache jbd2 crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr sg parport_pc vmw_vmci parport shpchp i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper ttm crct10dif_pclmul mptspi crct10dif_common crc32c_intel scsi_transport_spi ata_piix drm serio_raw mptscsih libata vmxnet3
mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 1065 Comm: node_exporter Tainted: G             L ------------   3.10.0-327.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff880421c7e780 ti: ffff8800356c4000 task.ti: ffff8800356c4000
RIP: 0010:[<ffffffff810e675e>]  [<ffffffff810e675e>] generic_exec_single+0xfe/0x1a0
RSP: 0018:ffff8800356c7c40  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff8800356c7c10 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff8800356c7c88 R08: ffffffff8165f1e0 R09: 000000c0006dc000
R10: 0000000000002000 R11: 0000000000000246 R12: 0000000000000000
R13: ffff88043fd54780 R14: 0000000500000003 R15: ffff88043fcd4780
FS:  0000000001116110(0063) GS:ffff88043fd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc585bac000 CR3: 0000000425eb4000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff8800356c7d10
0000000000000003 00000000f90fb311 0000000000000004 ffffffff81065c90
ffffffff81065c90 ffff8800356c7cb8 ffffffff810e685f ffff8800356c7cb8
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e685f>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f2785>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e6e03>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff81192563>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff811930f5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8119568a>] zap_page_range+0x12a/0x170
[<ffffffff81191db4>] SyS_madvise+0x394/0x820
[<ffffffff810e55a0>] ? SyS_futex+0x80/0x180
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 48 89 de 48 03 14 c5 e0 b8 a5 81 48 89 df e8 da fc 21 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 f6 43 20 01 <75> f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 00 0f 85 80 
Complete report #242040
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff91d1136a
Build id: 3.10.0-957.12.2.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff91c7a460
Build id: 3.10.0-957.12.2.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff91c7a460
Build id: 3.10.0-957.12.2.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff91c7a460
Build id: 3.10.0-957.12.2.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff91d1147f
Build id: 3.10.0-957.12.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff91f75c05
Build id: 3.10.0-957.12.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff91d11a2b
Build id: 3.10.0-957.12.2.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff91c7a628
Build id: 3.10.0-957.12.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff91c7a698
Build id: 3.10.0-957.12.2.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff91de4607
Build id: 3.10.0-957.12.2.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff91de5f05
Build id: 3.10.0-957.12.2.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff91de8d52
Build id: 3.10.0-957.12.2.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffff91de3c1d
Build id: 3.10.0-957.12.2.el7.x86_64
0x49d
14
? system_call_after_swapgs
vmlinux 0xffffffff92375d21
Build id: 3.10.0-957.12.2.el7.x86_64
0xae
15
tracesys
vmlinux 0xffffffff9237606b
Build id: 3.10.0-957.12.2.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [java:5506]
Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg virtio_balloon virtio_rng parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom virtio_console virtio_scsi virtio_net crct10dif_pclmul crct10dif_common crc32c_intel ata_generic serio_raw
pata_acpi floppy qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix virtio_pci libata virtio_ring drm_panel_orientation_quirks virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 5506 Comm: java Kdump: loaded Not tainted 3.10.0-957.12.2.el7.x86_64 #1
Hardware name: oVirt oVirt Node, BIOS 1.11.0-2.el7 04/01/2014
task: ffff9a4721251040 ti: ffff9a4731284000 task.ti: ffff9a4731284000
RIP: 0010:[<ffffffff91d1136a>]  [<ffffffff91d1136a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9a4731287bf0  EFLAGS: 00000202
RAX: 0000000000000010 RBX: ffff9a4731287bc0 RCX: 0000000000000030
RDX: 000000000000ffff RSI: 0000000000000010 RDI: 0000000000000282
RBP: ffff9a4731287c38 R08: ffffffff92413140 R09: 00003ffffffff000
R10: ffff9a46ffe0d878 R11: 00007fdec11b9000 R12: ffff9a477ffd6000
R13: ffffffff91dc1146 R14: ffff9a4731287c00 R15: 0000000000020200
FS:  00007fdec12b6700(0000) GS:ffff9a4777200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdf76a15530 CR3: 0000000223b52000 CR4: 00000000000206f0
Call Trace:
[<ffffffff91c7a460>] ? leave_mm+0x110/0x110
[<ffffffff91c7a460>] ? leave_mm+0x110/0x110
[<ffffffff91c7a460>] ? leave_mm+0x110/0x110
[<ffffffff91d1147f>] smp_call_function_single+0x5f/0xa0
[<ffffffff91f75c05>] ? cpumask_next_and+0x35/0x50
[<ffffffff91d11a2b>] smp_call_function_many+0x22b/0x270
[<ffffffff91c7a628>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff91c7a698>] flush_tlb_mm_range+0x68/0x140
[<ffffffff91de4607>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff91de5f05>] tlb_finish_mmu+0x55/0x60
[<ffffffff91de8d52>] zap_page_range+0x142/0x190
[<ffffffff91de3c1d>] SyS_madvise+0x49d/0xac0
[<ffffffff92375d21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff9237606b>] tracesys+0xa3/0xc9
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 bc 94 92 48 89 df e8 fa b6 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #223146
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa5b1109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffa5a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffa5a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffa5a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffa5b111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa5d75465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa5b1175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa5a7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa5a7a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffa5be4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffa5be5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffffa5bef014
Build id: 3.10.0-957.el7.x86_64
0xf4
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 67s! [systemd:1]
Modules linked in: tcp_diag inet_diag nfnetlink_queue nfnetlink_log nfnetlink bluetooth rfkill binfmt_misc ext4 mbcache jbd2 ppdev sg joydev virtio_balloon parport_pc parport i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw ttm floppy drm ata_piix libata virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1 Comm: systemd Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9da334bb0000 ti: ffff9da334bb8000 task.ti: ffff9da334bb8000
RIP: 0010:[<ffffffffa5b1109a>]  [<ffffffffa5b1109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9da334bbbc30  EFLAGS: 00000202
RAX: 00000000000000a0 RBX: ffff9da334bbbc00 RCX: 0000000000000020
RDX: 00000000ffffffff RSI: 00000000000000a0 RDI: 0000000000000286
RBP: ffff9da334bbbc78 R08: ffffffffa6212d88 R09: ffffffffa5bf8298
R10: ffff9da9dce5f0a0 R11: ffffe9990026a9c0 R12: ffffffffa5be93f8
R13: ffff9da334bbbc30 R14: 00007f34957fb000 R15: ffffe99905cf0af0
FS:  00007f34957ed940(0000) GS:ffff9da9dce40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f34957fb000 CR3: 000000081da72000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa5a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa5a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa5a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa5b111af>] smp_call_function_single+0x5f/0xa0
[<ffffffffa5d75465>] ? cpumask_next_and+0x35/0x50
[<ffffffffa5b1175b>] smp_call_function_many+0x22b/0x270
[<ffffffffa5a7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa5a7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa5be4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffa5be5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffa5bef014>] unmap_region+0xf4/0x140
Complete report #211064
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa2d0779a
Build id: 3.10.0-862.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffa2c76e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffa2c76e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffa2c76e50
Build id: 3.10.0-862.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffa2d078af
Build id: 3.10.0-862.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa2f4a275
Build id: 3.10.0-862.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa2d07e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa2c77018
Build id: 3.10.0-862.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa2c77088
Build id: 3.10.0-862.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffffa2dc0497
Build id: 3.10.0-862.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffa2dc1c75
Build id: 3.10.0-862.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffffa2dca624
Build id: 3.10.0-862.el7.x86_64
0xf4
13
? __vma_rb_erase
vmlinux 0xffffffffa2dcabe1
Build id: 3.10.0-862.el7.x86_64
0x121
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kubelet:4014]
Modules linked in: veth vxlan ip6_udp_tunnel udp_tunnel rpcsec_gss_krb5 nfsv4 dns_resolver nfs ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 fscache nf_nat_ipv6 xt_set xt_mark ip_set_hash_ipport ip_set_hash_ipportip ip_set_bitmap_port ip_set_hash_ipportnet ip_set dummy xt_comment nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack vmw_vsock_vmci_transport vsock crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev joydev cryptd parport_pc
vmw_balloon parport sg pcspkr vmw_vmci i2c_piix4 shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc sch_fq_codel ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci ata_piix mptspi scsi_transport_spi crct10dif_pclmul crct10dif_common i2c_core mptscsih libata crc32c_intel serio_raw mptbase vmxnet3 dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 4014 Comm: kubelet Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9e7c837d1fa0 ti: ffff9e79382d8000 task.ti: ffff9e79382d8000
RIP: 0010:[<ffffffffa2d0779a>]  [<ffffffffa2d0779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9e79382dbc30  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff9e79382dbc00 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000286
RBP: ffff9e79382dbc78 R08: ffffffffa3413160 R09: ffffffffa2dd34c8
R10: ffff9e7cbfd1ba40 R11: ffffee809087fd00 R12: 0000000000000001
R13: ffff9e79382dbbc8 R14: ffffffffa331a925 R15: ffff9e79382dbbb8
FS:  00007f6dd27fc700(0000) GS:ffff9e7cbfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6e0cd2d000 CR3: 000000041ae6a000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa2c76e50>] ? leave_mm+0x110/0x110
[<ffffffffa2c76e50>] ? leave_mm+0x110/0x110
[<ffffffffa2c76e50>] ? leave_mm+0x110/0x110
[<ffffffffa2d078af>] smp_call_function_single+0x5f/0xa0
[<ffffffffa2f4a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffa2d07e5b>] smp_call_function_many+0x22b/0x270
[<ffffffffa2c77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa2c77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa2dc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffa2dc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffa2dca624>] unmap_region+0xf4/0x140
[<ffffffffa2dcabe1>] ? __vma_rb_erase+0x121/0x220
Complete report #172168
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffae116802
Build id: 3.10.0-1160.11.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffae07e6a0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffae07e6a0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffae07e6a0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffae11691f
Build id: 3.10.0-1160.11.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffae385fe5
Build id: 3.10.0-1160.11.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffae116ecb
Build id: 3.10.0-1160.11.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffae07e868
Build id: 3.10.0-1160.11.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffae07e8db
Build id: 3.10.0-1160.11.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffae1edee6
Build id: 3.10.0-1160.11.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffae1ef9af
Build id: 3.10.0-1160.11.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffae1efad3
Build id: 3.10.0-1160.11.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffae1f2f9a
Build id: 3.10.0-1160.11.1.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffffae0db148
Build id: 3.10.0-1160.11.1.el7.x86_64
0x58
15
SyS_madvise
vmlinux 0xffffffffae1ed48d
Build id: 3.10.0-1160.11.1.el7.x86_64
0x49d
16
? SyS_futex
vmlinux 0xffffffffae1154c0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x80
17
system_call_fastpath
vmlinux 0xffffffffae793f92
Build id: 3.10.0-1160.11.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [containerd-shim:24728]
Modules linked in: binfmt_misc xt_multiport ipt_rpfilter iptable_raw ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel veth xt_statistic xt_recent xt_nat ipt_REJECT nf_reject_ipv4 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache overlay(T) xt_set ip_set nfnetlink xt_addrtype iptable_nat nf_nat_ipv4 nf_nat xt_comment xt_mark xt_conntrack vmw_vsock_vmci_transport vsock iptable_mangle dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd i2c_piix4 ppdev joydev pcspkr vmw_balloon sg vmw_vmci parport_pc parport nfsd nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh
ip_vs_wrr ip_vs_rr ip_vs auth_rpcgss nfs_acl lockd grace sunrpc nf_conntrack libcrc32c br_netfilter bridge stp llc ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops sd_mod ttm crc_t10dif crct10dif_generic drm ahci ata_piix libahci crct10dif_pclmul crct10dif_common crc32c_intel nfit libata serio_raw libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 5 PID: 24728 Comm: containerd-shim Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffff97cbbeb49080 ti: ffff97c8b55c0000 task.ti: ffff97c8b55c0000
RIP: 0010:[<ffffffffae116802>]  [<ffffffffae116802>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff97c8b55c3be0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff97c8b55c3bb0 RCX: 0000000000000001
RDX: ffffffffae8137d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff97c8b55c3c30 R08: ffffffffae8137c8 R09: ffffd391066359c0
R10: 0000000000004b29 R11: 0000000000000246 R12: ffffffffae1c8c76
R13: ffff97c8b55c3bf0 R14: 0000000000020200 R15: 0000000000000000
FS:  000000c000040b10(0000) GS:ffff97cc5d740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4cef4e9220 CR3: 000000069cf92000 CR4: 00000000001607e0
Call Trace:
[<ffffffffae07e6a0>] ? leave_mm+0x120/0x120
[<ffffffffae07e6a0>] ? leave_mm+0x120/0x120
[<ffffffffae07e6a0>] ? leave_mm+0x120/0x120
[<ffffffffae11691f>] smp_call_function_single+0x5f/0xa0
[<ffffffffae385fe5>] ? cpumask_next_and+0x35/0x50
[<ffffffffae116ecb>] smp_call_function_many+0x22b/0x270
[<ffffffffae07e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffae07e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffae1edee6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffae1ef9af>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffae1efad3>] tlb_finish_mmu+0x23/0x30
[<ffffffffae1f2f9a>] zap_page_range+0xea/0x130
[<ffffffffae0db148>] ? wake_up_q+0x58/0x80
[<ffffffffae1ed48d>] SyS_madvise+0x49d/0xac0
[<ffffffffae1154c0>] ? SyS_futex+0x80/0x190
[<ffffffffae793f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 13 d5 ae 48 89 df e8 57 69 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #153227
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9cf111be
Build id: 3.10.0-957.10.1.el7.x86_64
0xfe
2
? leave_mm
vmlinux 0xffffffff9ce7a400
Build id: 3.10.0-957.10.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff9ce7a400
Build id: 3.10.0-957.10.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff9ce7a400
Build id: 3.10.0-957.10.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff9cf112cf
Build id: 3.10.0-957.10.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9d175b35
Build id: 3.10.0-957.10.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9cf1187b
Build id: 3.10.0-957.10.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9ce7a5c8
Build id: 3.10.0-957.10.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9ce7a638
Build id: 3.10.0-957.10.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff9cfe4507
Build id: 3.10.0-957.10.1.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff9cfe5e05
Build id: 3.10.0-957.10.1.el7.x86_64
0x55
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [systemd-journal:1580]
Modules linked in: sunrpc vfat fat ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr virtio_balloon parport_pc parport i2c_piix4 ip_tables xfs libcrc32c virtio_blk virtio_console virtio_net crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic qxl pata_acpi drm_kms_helper floppy syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 1580 Comm: systemd-journal Tainted: G             L ------------   3.10.0-957.10.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.0.0 02/06/2015
task: ffff96c4b8e1c100 ti: ffff96c4b8e6c000 task.ti: ffff96c4b8e6c000
RIP: 0010:[<ffffffff9cf111be>]  [<ffffffff9cf111be>] generic_exec_single+0xfe/0x1b0
RSP: 0018:ffff96c4b8e6fc30  EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffff96c4b8e6fc30 RCX: 0000000000000830
RDX: 0000000000000002 RSI: 0000000000000002 RDI: 0000000000000286
RBP: ffff96c4b8e6fc78 R08: ffffffff9d613040 R09: ffffffff9cff881c
R10: ffff96c4be21f0a0 R11: fffff1b680a23b00 R12: 0000000000000001
R13: 0000000000000001 R14: ffff96c4b8e6fd00 R15: 0000000000000001
FS:  00007f43aa596880(0000) GS:ffff96c4be200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000006614f000 CR3: 000000007c1c2000 CR4: 00000000000406f0
Call Trace:
[<ffffffff9ce7a400>] ? leave_mm+0x110/0x110
[<ffffffff9ce7a400>] ? leave_mm+0x110/0x110
[<ffffffff9ce7a400>] ? leave_mm+0x110/0x110
[<ffffffff9cf112cf>] smp_call_function_single+0x5f/0xa0
[<ffffffff9d175b35>] ? cpumask_next_and+0x35/0x50
[<ffffffff9cf1187b>] smp_call_function_many+0x22b/0x270
[<ffffffff9ce7a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9ce7a638>] flush_tlb_mm_range+0x68/0x140
[<ffffffff9cfe4507>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff9cfe5e05>] tlb_finish_mmu+0x55/0x60
Complete report #95176
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9810779a
Build id: 3.10.0-862.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff98076e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff98076e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff98076e50
Build id: 3.10.0-862.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff981078af
Build id: 3.10.0-862.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9834a275
Build id: 3.10.0-862.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff98107e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff98077018
Build id: 3.10.0-862.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff98077088
Build id: 3.10.0-862.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffff981c0497
Build id: 3.10.0-862.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff981c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff981c4a02
Build id: 3.10.0-862.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffff981bfafd
Build id: 3.10.0-862.el7.x86_64
0x3cd
14
system_call_fastpath
vmlinux 0xffffffff9871f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 37s! [kubelet:85762]
Modules linked in: xt_recent tcp_diag udp_diag inet_diag binfmt_misc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache xt_set ipt_rpfilter xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ip_set ipip tunnel4 ip_tunnel xt_statistic xt_comment xt_mark ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs xt_nat veth nf_conntrack_netlink nfnetlink xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 br_netfilter tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel
lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg i2c_piix4 vmw_vmci shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi scsi_transport_spi crct10dif_pclmul crct10dif_common mptscsih crc32c_intel mptbase serio_raw i2c_core vmxnet3 floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 85762 Comm: kubelet Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff8cff229caf70 ti: ffff8cff05f7c000 task.ti: ffff8cff05f7c000
RIP: 0010:[<ffffffff9810779a>]  [<ffffffff9810779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8cff05f7fbf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8cff05f7fbc0 RCX: 0000000000000003
RDX: ffffffff98813670 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8cff05f7fc38 R08: ffffffff98813668 R09: ffff8d01bf7b80d8
R10: ffffd9ca4b17a880 R11: 0000000000000246 R12: ffff8cff229caf70
R13: ffff8cff229caf70 R14: ffff8cff05f7fb60 R15: 0000000000000081
FS:  00007f3333fff700(0000) GS:ffff8d032d740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00025f010 CR3: 000000041db86000 CR4: 00000000001607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff98076e50>] ? leave_mm+0x110/0x110
[<ffffffff98076e50>] ? leave_mm+0x110/0x110
[<ffffffff98076e50>] ? leave_mm+0x110/0x110
[<ffffffff981078af>] smp_call_function_single+0x5f/0xa0
[<ffffffff9834a275>] ? cpumask_next_and+0x35/0x50
[<ffffffff98107e5b>] smp_call_function_many+0x22b/0x270
[<ffffffff98077018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff98077088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff981c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff981c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff981c4a02>] zap_page_range+0x142/0x190
[<ffffffff981bfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff9871f7d5>] system_call_fastpath+0x1c/0x21
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 d3 98 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #79037
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb25111ba
Build id: 3.10.0-957.5.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffb247a400
Build id: 3.10.0-957.5.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffb247a400
Build id: 3.10.0-957.5.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffb247a400
Build id: 3.10.0-957.5.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffb25112cf
Build id: 3.10.0-957.5.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb2775a45
Build id: 3.10.0-957.5.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb251187b
Build id: 3.10.0-957.5.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb247a5c8
Build id: 3.10.0-957.5.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb247a638
Build id: 3.10.0-957.5.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffb25e4427
Build id: 3.10.0-957.5.1.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffb25e5d25
Build id: 3.10.0-957.5.1.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffb25e8b72
Build id: 3.10.0-957.5.1.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffffb25e3a3d
Build id: 3.10.0-957.5.1.el7.x86_64
0x49d
14
tracesys
vmlinux 0xffffffffb2b7505b
Build id: 3.10.0-957.5.1.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [filebeat:42866]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd joydev pcspkr sg snd_seq_midi snd_seq_midi_event parport_pc parport snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ata_generic pata_acpi serio_raw
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata e1000 mptspi scsi_transport_spi mptscsih nfit mptbase libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 42866 Comm: filebeat Not tainted 3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff9555b5906180 ti: ffff95551aef0000 task.ti: ffff95551aef0000
RIP: 0010:[<ffffffffb25111ba>]  [<ffffffffb25111ba>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff95551aef3bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff95551aef3bc0 RCX: 0000000000000002
RDX: ffffffffb2c13050 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff95551aef3c38 R08: ffffffffb2c13048 R09: 00003ffffffff000
R10: ffff955539953440 R11: 0000000000000246 R12: ffff9555bffd8000
R13: ffffffffb25c0eb6 R14: ffff95551aef3c00 R15: 0000000000020200
FS:  00007fb44ee25700(0000) GS:ffff9555b9680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f21e218a1a0 CR3: 0000000076de8000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb247a400>] ? leave_mm+0x110/0x110
[<ffffffffb247a400>] ? leave_mm+0x110/0x110
[<ffffffffb247a400>] ? leave_mm+0x110/0x110
[<ffffffffb25112cf>] smp_call_function_single+0x5f/0xa0
[<ffffffffb2775a45>] ? cpumask_next_and+0x35/0x50
[<ffffffffb251187b>] smp_call_function_many+0x22b/0x270
[<ffffffffb247a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb247a638>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb25e4427>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffb25e5d25>] tlb_finish_mmu+0x55/0x60
[<ffffffffb25e8b72>] zap_page_range+0x142/0x190
[<ffffffffb25e3a3d>] SyS_madvise+0x49d/0xac0
[<ffffffffb2b7505b>] tracesys+0xa3/0xc9
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 14 b3 48 89 df e8 da b6 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #59425
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff96b111ea
Build id: 3.10.0-957.12.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff96a7a400
Build id: 3.10.0-957.12.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff96a7a400
Build id: 3.10.0-957.12.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff96a7a400
Build id: 3.10.0-957.12.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff96b112ff
Build id: 3.10.0-957.12.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff96d75af5
Build id: 3.10.0-957.12.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff96b118ab
Build id: 3.10.0-957.12.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff96a7a5c8
Build id: 3.10.0-957.12.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff96a7a638
Build id: 3.10.0-957.12.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff96be44c7
Build id: 3.10.0-957.12.1.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff96be5dc5
Build id: 3.10.0-957.12.1.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff96be8c12
Build id: 3.10.0-957.12.1.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffff96ad67a8
Build id: 3.10.0-957.12.1.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffff96be3add
Build id: 3.10.0-957.12.1.el7.x86_64
0x49d
15
system_call_fastpath
vmlinux 0xffffffff97175d9b
Build id: 3.10.0-957.12.1.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 21s! [containerd-shim:16624]
Modules linked in: xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper serio_raw crc32c_intel ata_generic nfit libnvdimm syscopyarea pata_acpi sysfillrect sysimgblt
fb_sys_fops mptspi ttm ata_piix scsi_transport_spi drm mptscsih libata mptbase e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 16624 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8b7ac6f32080 ti: ffff8b7a83fd8000 task.ti: ffff8b7a83fd8000
RIP: 0010:[<ffffffff96b111ea>]  [<ffffffff96b111ea>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8b7a83fdbbf0  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff8b7a83fdbbc0 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff8b7a83fdbc38 R08: ffffffff97212dc0 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8b7a83fdbbb8
R13: ffff8b7a83fdbb80 R14: ffff8b7adb2e5b40 R15: 00000000f4e858cc
FS:  000000c4200d9490(0000) GS:ffff8b7afcd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c4202ad010 CR3: 00000006c6af0000 CR4: 00000000003607e0
Call Trace:
[<ffffffff96a7a400>] ? leave_mm+0x110/0x110
[<ffffffff96a7a400>] ? leave_mm+0x110/0x110
[<ffffffff96a7a400>] ? leave_mm+0x110/0x110
[<ffffffff96b112ff>] smp_call_function_single+0x5f/0xa0
[<ffffffff96d75af5>] ? cpumask_next_and+0x35/0x50
[<ffffffff96b118ab>] smp_call_function_many+0x22b/0x270
[<ffffffff96a7a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff96a7a638>] flush_tlb_mm_range+0x68/0x140
[<ffffffff96be44c7>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff96be5dc5>] tlb_finish_mmu+0x55/0x60
[<ffffffff96be8c12>] zap_page_range+0x142/0x190
[<ffffffff96ad67a8>] ? wake_up_q+0x58/0x80
[<ffffffff96be3add>] SyS_madvise+0x49d/0xac0
[<ffffffff97175d9b>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 a0 ba 74 97 48 89 df e8 6a b7 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #49071
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff973111ba
Build id: 3.10.0-957.10.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff9727a400
Build id: 3.10.0-957.10.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff9727a400
Build id: 3.10.0-957.10.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff9727a400
Build id: 3.10.0-957.10.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff973112cf
Build id: 3.10.0-957.10.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff97575b35
Build id: 3.10.0-957.10.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9731187b
Build id: 3.10.0-957.10.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9727a5c8
Build id: 3.10.0-957.10.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9727a638
Build id: 3.10.0-957.10.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff973e4507
Build id: 3.10.0-957.10.1.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff973e5e05
Build id: 3.10.0-957.10.1.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff973e8c52
Build id: 3.10.0-957.10.1.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffff973e3b1d
Build id: 3.10.0-957.10.1.el7.x86_64
0x49d
14
system_call_fastpath
vmlinux 0xffffffff97975ddb
Build id: 3.10.0-957.10.1.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [Chrome_ChildIOT:4051]
INFO: rcu_sched detected stalls on CPUs/tasks: { 1} 
Modules linked in:
tcp_lp
fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmhgfs(OE) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ext4 snd_ens1371 mbcache snd_rawmidi snd_ac97_codec iosf_mbi jbd2 kvm_intel ac97_bus kvm snd_seq irqbypass snd_seq_device snd_pcm snd_timer snd crc32_pclmul ghash_clmulni_intel soundcore aesni_intel lrw gf128mul glue_helper
ablk_helper cryptd nfit ppdev sg vmw_balloon libnvdimm pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix drm mptspi serio_raw libata scsi_transport_spi mptscsih e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 4051 Comm: Chrome_ChildIOT Kdump: loaded Tainted: G           OE  ------------   3.10.0-957.10.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9eb9e6336180 ti: ffff9eb9eb894000 task.ti: ffff9eb9eb894000
RIP: 0010:[<ffffffff973111ba>]  [<ffffffff973111ba>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9eb9eb897bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9eb9eb897bc0 RCX: 0000000000000003
RDX: ffffffff97a132d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9eb9eb897c38 R08: ffffffff97a132c8 R09: 00003ffffffff000
R10: ffff9eb966ff4ca8 R11: 0000000000000206 R12: ffff9eb9fffda000
R13: ffffffff973c0f96 R14: ffff9eb9eb897c00 R15: 0000000000020200
FS:  00007f7ca5c78700(0000) GS:ffff9eb9ed740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000024adb3637ff0 CR3: 00000003efaba000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9727a400>] ? leave_mm+0x110/0x110
[<ffffffff9727a400>] ? leave_mm+0x110/0x110
[<ffffffff9727a400>] ? leave_mm+0x110/0x110
[<ffffffff973112cf>] smp_call_function_single+0x5f/0xa0
[<ffffffff97575b35>] ? cpumask_next_and+0x35/0x50
[<ffffffff9731187b>] smp_call_function_many+0x22b/0x270
[<ffffffff9727a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9727a638>] flush_tlb_mm_range+0x68/0x140
[<ffffffff973e4507>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff973e5e05>] tlb_finish_mmu+0x55/0x60
[<ffffffff973e8c52>] zap_page_range+0x142/0x190
[<ffffffff973e3b1d>] SyS_madvise+0x49d/0xac0
[<ffffffff97975ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 a0 b9 f4 97 48 89 df e8 ca b7 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #45339
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff94d0a9fa
Build id: 3.10.0-862.11.6.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff94c79c90
Build id: 3.10.0-862.11.6.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff94c79c90
Build id: 3.10.0-862.11.6.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff94c79c90
Build id: 3.10.0-862.11.6.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff94d0ab0f
Build id: 3.10.0-862.11.6.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff94f4ec95
Build id: 3.10.0-862.11.6.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff94d0b0bb
Build id: 3.10.0-862.11.6.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff94c79e58
Build id: 3.10.0-862.11.6.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff94c79ec8
Build id: 3.10.0-862.11.6.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffff94dc33c7
Build id: 3.10.0-862.11.6.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff94dc4cc5
Build id: 3.10.0-862.11.6.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff94dc7b12
Build id: 3.10.0-862.11.6.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffff94cd1f98
Build id: 3.10.0-862.11.6.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffff94dc2a1d
Build id: 3.10.0-862.11.6.el7.x86_64
0x3cd
15
system_call_fastpath
vmlinux 0xffffffff9532579b
Build id: 3.10.0-862.11.6.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mysqld:13075]
Modules linked in: tcp_diag inet_diag talpa_vfshook(OE) talpa_pedconnector(OE) talpa_pedevice(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack xt_multiport iptable_filter vmw_vsock_vmci_transport vsock sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper nfit ablk_helper cryptd libnvdimm joydev pcspkr sg parport_pc parport vmw_vmci shpchp i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic ahci drm_kms_helper ata_piix libahci syscopyarea sysfillrect crct10dif_pclmul crct10dif_common sysimgblt fb_sys_fops libata ttm crc32c_intel serio_raw drm vmxnet3 vmw_pvscsi i2c_core dm_mirror dm_region_hash
dm_log dm_mod
CPU: 1 PID: 13075 Comm: mysqld Kdump: loaded Tainted: G           OE  ------------   3.10.0-862.11.6.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff919e2c361fa0 ti: ffff919da4ac0000 task.ti: ffff919da4ac0000
RIP: 0010:[<ffffffff94d0a9fa>]  [<ffffffff94d0a9fa>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff919da4ac3bf0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffff919da4ac3bc0 RCX: 000000000000003e
RDX: 0000000000000003 RSI: 0000000000000002 RDI: 0000000000000282
RBP: ffff919da4ac3c38 R08: ffffffff95413320 R09: 00003ffffffff000
R10: ffff919e2a7da5e8 R11: 00007f052307c000 R12: ffffffff95318978
R13: ffffffff95318984 R14: ffffffff95318978 R15: ffffffff95318984
FS:  00007f05230c4700(0000) GS:ffff919e3fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000559654b89000 CR3: 0000000035b70000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff94c79c90>] ? leave_mm+0x110/0x110
[<ffffffff94c79c90>] ? leave_mm+0x110/0x110
[<ffffffff94c79c90>] ? leave_mm+0x110/0x110
[<ffffffff94d0ab0f>] smp_call_function_single+0x5f/0xa0
[<ffffffff94f4ec95>] ? cpumask_next_and+0x35/0x50
[<ffffffff94d0b0bb>] smp_call_function_many+0x22b/0x270
[<ffffffff94c79e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff94c79ec8>] flush_tlb_mm_range+0x68/0x140
[<ffffffff94dc33c7>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff94dc4cc5>] tlb_finish_mmu+0x55/0x60
[<ffffffff94dc7b12>] zap_page_range+0x142/0x190
[<ffffffff94cd1f98>] ? wake_up_q+0x58/0x80
[<ffffffff94dc2a1d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff9532579b>] system_call_fastpath+0x22/0x27
Code: c0 96 01 00 48 89 de 48 03 14 c5 60 54 93 95 48 89 df e8 7a a0 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #23771
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa810aa2a
Build id: 3.10.0-862.14.4.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffa8079c90
Build id: 3.10.0-862.14.4.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffa8079c90
Build id: 3.10.0-862.14.4.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffa8079c90
Build id: 3.10.0-862.14.4.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffa810ab3f
Build id: 3.10.0-862.14.4.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa834edb5
Build id: 3.10.0-862.14.4.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa810b0eb
Build id: 3.10.0-862.14.4.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa8079e58
Build id: 3.10.0-862.14.4.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa8079ec8
Build id: 3.10.0-862.14.4.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffffa81c3487
Build id: 3.10.0-862.14.4.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffa81c4d85
Build id: 3.10.0-862.14.4.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffa81c7bd2
Build id: 3.10.0-862.14.4.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffffa80d1fc8
Build id: 3.10.0-862.14.4.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffffa81c2add
Build id: 3.10.0-862.14.4.el7.x86_64
0x3cd
15
? wake_up_state
vmlinux 0xffffffffa80d2010
Build id: 3.10.0-862.14.4.el7.x86_64
0x20
16
tracesys
vmlinux 0xffffffffa8725a1b
Build id: 3.10.0-862.14.4.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [prometheus:22200]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_lsm_pinned_10703(E) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache vmw_vsock_vmci_transport vsock falcon_lsm_pinned_10404(E) sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd vmw_balloon nfit pcspkr joydev sg libnvdimm vmw_vmci shpchp i2c_piix4 parport_pc parport ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common ata_piix crc32c_intel mptspi serio_raw libata scsi_transport_spi mptscsih mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_lsm_serviceable]
CPU: 5 PID: 22200 Comm: prometheus Kdump: loaded Tainted: P            EL ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9b7a3a322f70 ti: ffff9b7acba94000 task.ti: ffff9b7acba94000
RIP: 0010:[<ffffffffa810aa2a>]  [<ffffffffa810aa2a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9b7acba97bf0  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff9b7acba97bc0 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff9b7acba97c38 R08: ffffffffa88141a0 R09: 00003ffffffff000
R10: ffff9b7cda2ad878 R11: 0000000000000246 R12: ffffffffa834edb5
R13: ffff9b7acba97b78 R14: ffff9b7a39d29800 R15: 00000000a1cc8b98
FS:  00007f54df9c8700(0000) GS:ffff9b7cffd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f603834d000 CR3: 0000000421b8e000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa8079c90>] ? leave_mm+0x110/0x110
[<ffffffffa8079c90>] ? leave_mm+0x110/0x110
[<ffffffffa8079c90>] ? leave_mm+0x110/0x110
[<ffffffffa810ab3f>] smp_call_function_single+0x5f/0xa0
[<ffffffffa834edb5>] ? cpumask_next_and+0x35/0x50
[<ffffffffa810b0eb>] smp_call_function_many+0x22b/0x270
[<ffffffffa8079e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa8079ec8>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa81c3487>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffa81c4d85>] tlb_finish_mmu+0x55/0x60
[<ffffffffa81c7bd2>] zap_page_range+0x142/0x190
[<ffffffffa80d1fc8>] ? wake_up_q+0x58/0x80
[<ffffffffa81c2add>] SyS_madvise+0x3cd/0x9c0
[<ffffffffa80d2010>] ? wake_up_state+0x20/0x20
[<ffffffffa8725a1b>] tracesys+0xa3/0xc9
Code: c0 96 01 00 48 89 de 48 03 14 c5 a0 54 d3 a8 48 89 df e8 6a a1 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #22060
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f945a
Build id: 3.10.0-514.10.2.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.10.2.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.10.2.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106c430
Build id: 3.10.0-514.10.2.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f955f
Build id: 3.10.0-514.10.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81317bc5
Build id: 3.10.0-514.10.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9b03
Build id: 3.10.0-514.10.2.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c5f8
Build id: 3.10.0-514.10.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c6c9
Build id: 3.10.0-514.10.2.el7.x86_64
0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac293
Build id: 3.10.0-514.10.2.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad585
Build id: 3.10.0-514.10.2.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff811af95a
Build id: 3.10.0-514.10.2.el7.x86_64
0x13a
13
SyS_madvise
vmlinux 0xffffffff811ab9ec
Build id: 3.10.0-514.10.2.el7.x86_64
0x38c
14
system_call_fastpath
vmlinux 0xffffffff81696b09
Build id: 3.10.0-514.10.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 21s! [java:13628]
Modules linked in: binfmt_misc intel_powerclamp coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ppdev vmw_balloon pcspkr parport_pc parport i2c_piix4 vmw_vmci shpchp ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea mptspi sysfillrect sysimgblt scsi_transport_spi fb_sys_fops mptscsih ttm drm crct10dif_pclmul ata_piix crct10dif_common crc32c_intel serio_raw libata mptbase vmxnet3 i2c_core floppy fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 12 PID: 13628 Comm: java Tainted: G             L ------------   3.10.0-514.10.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff881119e74e70 ti: ffff8808d9d14000 task.ti: ffff8808d9d14000
RIP: 0010:[<ffffffff810f945a>]  [<ffffffff810f945a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8808d9d17c40  EFLAGS: 00000202
RAX: 0000000000000010 RBX: ffffffff810cbe9c RCX: 0000000000000030
RDX: 0000000000000010 RSI: 0000000000000010 RDI: 0000000000000286
RBP: ffff8808d9d17c88 R08: ffffffff816b1a20 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff881119e74ed8
R13: ffff8808d9d17c68 R14: ffff8811bffda000 R15: ffffffff8118aec6
FS:  00007f367e0f0700(0000) GS:ffff8811b5c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9160f9ded8 CR3: 00000005237ab000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c430 ffff8808d9d17d10
0000000000000003 000000009b3f17b9 000000000000000d ffffffff8106c430
ffffffff8106c430 ffff8808d9d17cb8 ffffffff810f955f ffff8808d9d17cb8
Call Trace:
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff8106c430>] ? leave_mm+0x70/0x70
[<ffffffff810f955f>] smp_call_function_single+0x5f/0xa0
[<ffffffff81317bc5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9b03>] smp_call_function_many+0x223/0x260
[<ffffffff8106c5f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c6c9>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac293>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad585>] tlb_finish_mmu+0x55/0x60
[<ffffffff811af95a>] zap_page_range+0x13a/0x180
[<ffffffff811ab9ec>] SyS_madvise+0x38c/0x8d0
[<ffffffff81696b09>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 40 57 ad 81 48 89 df e8 0a 29 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #21442
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f925a
Build id: 3.10.0-514.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106c300
Build id: 3.10.0-514.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f935f
Build id: 3.10.0-514.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813179a5
Build id: 3.10.0-514.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f9903
Build id: 3.10.0-514.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106c4c8
Build id: 3.10.0-514.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106c599
Build id: 3.10.0-514.el7.x86_64
0x69
10
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac093
Build id: 3.10.0-514.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ad385
Build id: 3.10.0-514.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff811af75a
Build id: 3.10.0-514.el7.x86_64
0x13a
13
SyS_madvise
vmlinux 0xffffffff811ab7ec
Build id: 3.10.0-514.el7.x86_64
0x38c
14
? SyS_futex
vmlinux 0xffffffff810f7f80
Build id: 3.10.0-514.el7.x86_64
0x80
15
system_call_fastpath
vmlinux 0xffffffff816965c9
Build id: 3.10.0-514.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [pool:2590]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event intel_powerclamp coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul btusb glue_helper btrtl btbcm ablk_helper btintel cryptd snd_ens1371 snd_rawmidi bluetooth snd_ac97_codec ac97_bus sg snd_seq ppdev rfkill snd_seq_device pcspkr snd_pcm vmw_balloon i2c_piix4 snd_timer snd soundcore vmw_vmci shpchp nfit libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
xfs libcrc32c sd_mod crc_t10dif sr_mod cdrom crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common vmwgfx crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt ata_piix fb_sys_fops ttm drm serio_raw mptspi libata scsi_transport_spi e1000 mptscsih mptbase i2c_core fjes
CPU: 3 PID: 2590 Comm: pool Not tainted 3.10.0-514.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8800a38a0fb0 ti: ffff8800a7364000 task.ti: ffff8800a7364000
RIP: 0010:[<ffffffff810f925a>]  [<ffffffff810f925a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8800a7367c40  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8800a7367c10 RCX: 0000000000000001
RDX: ffffffff816af9b0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8800a7367c88 R08: ffffffff816af9a8 R09: 00007f1a99e9e000
R10: 0000000003a5b7d0 R11: 00007f1a996a4000 R12: 0000000000000000
R13: ffff8801396d6c40 R14: 0000000300000000 R15: ffff880139616c40
FS:  00007f1a99ea3700(0000) GS:ffff8801396c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1a99690fc8 CR3: 0000000133252000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106c300 ffff8800a7367d10
0000000000000003 0000000022e1b0b1 0000000000000000 ffffffff8106c300
ffffffff8106c300 ffff8800a7367cb8 ffffffff810f935f ffff8800a7367cb8
Call Trace:
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff8106c300>] ? leave_mm+0x70/0x70
[<ffffffff810f935f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813179a5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f9903>] smp_call_function_many+0x223/0x260
[<ffffffff8106c4c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c599>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac093>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ad385>] tlb_finish_mmu+0x55/0x60
[<ffffffff811af75a>] zap_page_range+0x13a/0x180
[<ffffffff811ab7ec>] SyS_madvise+0x38c/0x8d0
[<ffffffff810f7f80>] ? SyS_futex+0x80/0x180
[<ffffffff816965c9>] system_call_fastpath+0x16/0x1b
Code: 80 77 01 00 48 89 de 48 03 14 c5 40 57 ad 81 48 89 df e8 ea 28 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #16765
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffad30779a
Build id: 3.10.0-862.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffad276e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffad276e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffad276e50
Build id: 3.10.0-862.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffad3078af
Build id: 3.10.0-862.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffad54a275
Build id: 3.10.0-862.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffad307e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffad277018
Build id: 3.10.0-862.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffad277088
Build id: 3.10.0-862.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffffad3c0497
Build id: 3.10.0-862.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffad3c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
12
unmap_region
vmlinux 0xffffffffad3ca624
Build id: 3.10.0-862.el7.x86_64
0xf4
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [java:27030]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ipt_MASQUERADE tun nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 ebtable_filter nf_defrag_ipv4 ebtables nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack ip6table_filter ip6_tables br_netfilter iptable_filter bridge stp llc devlink vmw_vsock_vmci_transport vsock overlay(T) sunrpc coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd vmw_balloon joydev pcspkr vmw_vmci sg i2c_piix4 floppy parport_pc parport shpchp ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix scsi_transport_spi drm crct10dif_pclmul
crct10dif_common libata mptscsih crc32c_intel e1000 serio_raw mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 27030 Comm: java Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8d7e8be43f40 ti: ffff8d7f6dfb0000 task.ti: ffff8d7f6dfb0000
RIP: 0010:[<ffffffffad30779a>]  [<ffffffffad30779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8d7f6dfb3c30  EFLAGS: 00000202
RAX: 0000000000000020 RBX: 0000000000001000 RCX: 0000000000000020
RDX: 0000000000000020 RSI: 0000000000000020 RDI: 0000000000000282
RBP: ffff8d7f6dfb3c78 R08: ffffffffada13660 R09: ffffffffad3d35bc
R10: ffff8d861ee5ba40 R11: ffffe0e3559d44c0 R12: 0000000000000001
R13: ffff8d7f6dfb3bc8 R14: ffffffffad91a925 R15: ffff8d7f6dfb3bb8
FS:  00007fc55a317700(0000) GS:ffff8d861ee40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc629c80000 CR3: 00000004aa324000 CR4: 00000000000007e0
Call Trace:
[<ffffffffad276e50>] ? leave_mm+0x110/0x110
[<ffffffffad276e50>] ? leave_mm+0x110/0x110
[<ffffffffad276e50>] ? leave_mm+0x110/0x110
[<ffffffffad3078af>] smp_call_function_single+0x5f/0xa0
[<ffffffffad54a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffad307e5b>] smp_call_function_many+0x22b/0x270
[<ffffffffad277018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffad277088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffad3c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffad3c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffad3ca624>] unmap_region+0xf4/0x140
Complete report #14680
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810e69de
Build id: 3.10.0-327.28.3.el7.x86_64
0xfe
2
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.28.3.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.28.3.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff81065c90
Build id: 3.10.0-327.28.3.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810e6adf
Build id: 3.10.0-327.28.3.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff812f3015
Build id: 3.10.0-327.28.3.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810e7083
Build id: 3.10.0-327.28.3.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.28.3.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81065f26
Build id: 3.10.0-327.28.3.el7.x86_64
0x66
10
tlb_flush_mmu.part.54
vmlinux 0xffffffff811929d3
Build id: 3.10.0-327.28.3.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff81193565
Build id: 3.10.0-327.28.3.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff81195afa
Build id: 3.10.0-327.28.3.el7.x86_64
0x12a
13
SyS_madvise
vmlinux 0xffffffff81192224
Build id: 3.10.0-327.28.3.el7.x86_64
0x394
14
? __audit_syscall_entry
vmlinux 0xffffffff8110b9f4
Build id: 3.10.0-327.28.3.el7.x86_64
0xb4
15
tracesys
vmlinux 0xffffffff81646d52
Build id: 3.10.0-327.28.3.el7.x86_64
0xdd
BUG: soft lockup - CPU#0 stuck for 81s! [s1-perf:31145]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc vmw_vsock_vmci_transport vsock crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr vmw_balloon sg parport_pc i2c_piix4 vmw_vmci parport shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod cdrom crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmwgfx drm_kms_helper vmxnet3 mptspi ttm scsi_transport_spi mptscsih mptbase drm ata_piix libata i2c_core dm_mirror dm_region_hash
dm_log dm_mod
CPU: 0 PID: 31145 Comm: s1-perf Tainted: G        W    L ------------   3.10.0-327.28.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff88002014ae00 ti: ffff8800b61b4000 task.ti: ffff8800b61b4000
RIP: 0010:[<ffffffff810e69de>]  [<ffffffff810e69de>] generic_exec_single+0xfe/0x1a0
RSP: 0018:ffff8800b61b7c40  EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffff8800b61b7c10 RCX: 0000000000000830
RDX: 0000000000000002 RSI: 0000000000000002 RDI: 0000000000000282
RBP: ffff8800b61b7c88 R08: ffffffff8165fbe0 R09: 00007f487aaa5000
R10: 00000000007fb000 R11: 0000000000000202 R12: 0000000000000000
R13: ffff88012dc167c0 R14: 0000000000000000 R15: 0000000000000000
FS:  00007f487aaaa700(0000) GS:ffff88012dc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f487d2db000 CR3: 0000000069ea6000 CR4: 00000000001407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81065c90 ffff8800b61b7d10
0000000000000003 00000000f639737f 0000000000000001 ffffffff81065c90
ffffffff81065c90 ffff8800b61b7cb8 ffffffff810e6adf ffff8800b61b7cb8
Call Trace:
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff81065c90>] ? leave_mm+0x70/0x70
[<ffffffff810e6adf>] smp_call_function_single+0x5f/0xa0
[<ffffffff812f3015>] ? cpumask_next_and+0x35/0x50
[<ffffffff810e7083>] smp_call_function_many+0x223/0x260
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81065f26>] flush_tlb_mm_range+0x66/0x140
[<ffffffff811929d3>] tlb_flush_mmu.part.54+0x33/0xc0
[<ffffffff81193565>] tlb_finish_mmu+0x55/0x60
[<ffffffff81195afa>] zap_page_range+0x12a/0x170
[<ffffffff81192224>] SyS_madvise+0x394/0x820
[<ffffffff8110b9f4>] ? __audit_syscall_entry+0xb4/0x110
[<ffffffff81646d52>] tracesys+0xdd/0xe2
Code: 48 89 de 48 03 14 c5 20 c9 a5 81 48 89 df e8 7a 03 22 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 f6 43 20 01 <75> f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 00 0f 85 80 
Complete report #5225
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9d31109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff9d27a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff9d27a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff9d27a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff9d3111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9d575465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9d31175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9d27a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9d27a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff9d3e4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff9d3e5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff9d3e87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffff9d3e368d
Build id: 3.10.0-957.el7.x86_64
0x49d
14
system_call_fastpath
vmlinux 0xffffffff9d974ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ActiveMQ Broker:44025]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev coretemp iosf_mbi vmw_balloon crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm sg snd_timer snd soundcore vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul
crct10dif_common crc32c_intel ata_generic pata_acpi serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm e1000 ata_piix libata mptspi scsi_transport_spi mptscsih drm_panel_orientation_quirks mptbase nfit libnvdimm
CPU: 3 PID: 44025 Comm: ActiveMQ Broker Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff95c263e5d140 ti: ffff95c236f7c000 task.ti: ffff95c236f7c000
RIP: 0010:[<ffffffff9d31109a>]  [<ffffffff9d31109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff95c236f7fbf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff95c236f7fbc0 RCX: 0000000000000003
RDX: ffffffff9da13290 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff95c236f7fc38 R08: ffffffff9da13288 R09: 00003ffffffff000
R10: ffff95c2a0ac9e60 R11: 00007fa54d2db000 R12: ffff95c2bffda000
R13: ffffffff9d3c0c06 R14: ffff95c236f7fc00 R15: 0000000000020200
FS:  00007fa54d3d7700(0000) GS:ffff95c2b56c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa5fe3ff2b0 CR3: 00000002292e0000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9d27a3d0>] ? leave_mm+0x110/0x110
[<ffffffff9d27a3d0>] ? leave_mm+0x110/0x110
[<ffffffff9d27a3d0>] ? leave_mm+0x110/0x110
[<ffffffff9d3111af>] smp_call_function_single+0x5f/0xa0
[<ffffffff9d575465>] ? cpumask_next_and+0x35/0x50
[<ffffffff9d31175b>] smp_call_function_many+0x22b/0x270
[<ffffffff9d27a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9d27a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff9d3e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff9d3e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff9d3e87c2>] zap_page_range+0x142/0x190
[<ffffffff9d3e368d>] SyS_madvise+0x49d/0xac0
[<ffffffff9d974ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 f4 9d 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1653
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffaa1111ba
Build id: 3.10.0-957.10.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffaa07a400
Build id: 3.10.0-957.10.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffaa07a400
Build id: 3.10.0-957.10.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffaa07a400
Build id: 3.10.0-957.10.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffaa1112cf
Build id: 3.10.0-957.10.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffaa375b35
Build id: 3.10.0-957.10.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffaa11187b
Build id: 3.10.0-957.10.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffaa07a5c8
Build id: 3.10.0-957.10.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffaa07a638
Build id: 3.10.0-957.10.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffaa1e4507
Build id: 3.10.0-957.10.1.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffaa1e5e05
Build id: 3.10.0-957.10.1.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffaa1e8c52
Build id: 3.10.0-957.10.1.el7.x86_64
0x142
13
? do_sync_write
vmlinux 0xffffffffaa240c23
Build id: 3.10.0-957.10.1.el7.x86_64
0x93
14
SyS_madvise
vmlinux 0xffffffffaa1e3b1d
Build id: 3.10.0-957.10.1.el7.x86_64
0x49d
15
? vfs_write
vmlinux 0xffffffffaa2417d2
Build id: 3.10.0-957.10.1.el7.x86_64
0x182
16
system_call_fastpath
vmlinux 0xffffffffaa775ddb
Build id: 3.10.0-957.10.1.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [SystemLogFlush:12278]
Modules linked in: binfmt_misc xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE dm_thin_pool nf_nat_masquerade_ipv4 iptable_nat dm_persistent_data dm_bio_prison nf_nat_ipv4 dm_bufio nf_nat loop nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event coretemp iosf_mbi snd_ens1371 crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul snd_rawmidi glue_helper ablk_helper snd_ac97_codec cryptd ac97_bus ppdev snd_seq snd_seq_device snd_pcm vmw_balloon snd_timer snd soundcore pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables
xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ata_generic pata_acpi serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 12278 Comm: SystemLogFlush Kdump: loaded Not tainted 3.10.0-957.10.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8fd834fa5140 ti: ffff8fd9a57e0000 task.ti: ffff8fd9a57e0000
RIP: 0010:[<ffffffffaa1111ba>]  [<ffffffffaa1111ba>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8fd9a57e3bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8fd9a57e3bc0 RCX: 0000000000000001
RDX: ffffffffaa812dd0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8fd9a57e3c38 R08: ffffffffaa812dc8 R09: 00003ffffffff000
R10: ffff8fd82da08948 R11: 0000000000000206 R12: ffff8fd9a57e3be8
R13: ffff8fd9b25b57a0 R14: 0000000000000077 R15: 00000000073199a2
FS:  00007fd4a9b62700(0000) GS:ffff8fd9b5680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd4aa923000 CR3: 00000000ad85c000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaa07a400>] ? leave_mm+0x110/0x110
[<ffffffffaa07a400>] ? leave_mm+0x110/0x110
[<ffffffffaa07a400>] ? leave_mm+0x110/0x110
[<ffffffffaa1112cf>] smp_call_function_single+0x5f/0xa0
[<ffffffffaa375b35>] ? cpumask_next_and+0x35/0x50
[<ffffffffaa11187b>] smp_call_function_many+0x22b/0x270
[<ffffffffaa07a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaa07a638>] flush_tlb_mm_range+0x68/0x140
[<ffffffffaa1e4507>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffaa1e5e05>] tlb_finish_mmu+0x55/0x60
[<ffffffffaa1e8c52>] zap_page_range+0x142/0x190
[<ffffffffaa240c23>] ? do_sync_write+0x93/0xe0
[<ffffffffaa1e3b1d>] SyS_madvise+0x49d/0xac0
[<ffffffffaa2417d2>] ? vfs_write+0x182/0x1f0
[<ffffffffaa775ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 a0 b9 d4 aa 48 89 df e8 ca b7 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1336655
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffaed1dba2
Build id: 3.10.0-1160.119.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffaec81b10
Build id: 3.10.0-1160.119.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffffaec81b10
Build id: 3.10.0-1160.119.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffffaec81b10
Build id: 3.10.0-1160.119.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffffaed1dcbf
Build id: 3.10.0-1160.119.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffaef9c615
Build id: 3.10.0-1160.119.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffaed1e27b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffaec81fa8
Build id: 3.10.0-1160.119.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffaec8201b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaedfa0ce
Build id: 3.10.0-1160.119.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffffaedfbc4f
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffaedfbd83
Build id: 3.10.0-1160.119.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffaedff2ca
Build id: 3.10.0-1160.119.1.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffffaedf96bd
Build id: 3.10.0-1160.119.1.el7.x86_64
0x49d
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [mysqld:3553]
Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev sg pcspkr parport_pc parport vmw_vmci i2c_piix4 ip_tables
xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea crct10dif_pclmul crct10dif_common sysfillrect sysimgblt crc32c_intel fb_sys_fops ttm ahci serio_raw libahci ata_piix drm vmxnet3 libata vmw_pvscsi nfit libnvdimm drm_panel_orientation_quirks fuse
CPU: 2 PID: 3553 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8f9f35171080 ti: ffff8f9f355e4000 task.ti: ffff8f9f355e4000
RIP: 0010:[<ffffffffaed1dba2>]  [<ffffffffaed1dba2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8f9f355e7be0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff8f9f355e7bb0 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff8f9f355e7c30 R08: ffffffffaf413c40 R09: 00003ffffffff000
R10: ffff8f9f34eec360 R11: 0000000000000206 R12: ffff8f9f35171080
R13: 0000000300000013 R14: ffff8f9f355e7cc0 R15: ffffffffaedd36bf
FS:  00007f4737ca6b00(0000) GS:ffff8f9f3fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055732aa4c000 CR3: 000000023577a000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffaec81b10>] ? leave_mm+0x130/0x130
[<ffffffffaec81b10>] ? leave_mm+0x130/0x130
[<ffffffffaec81b10>] ? leave_mm+0x130/0x130
[<ffffffffaed1dcbf>] smp_call_function_single+0x5f/0xa0
[<ffffffffaef9c615>] ? cpumask_next_and+0x35/0x50
[<ffffffffaed1e27b>] smp_call_function_many+0x22b/0x270
[<ffffffffaec81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaec8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaedfa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffaedfbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaedfbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffffaedff2ca>] zap_page_range+0xea/0x140
[<ffffffffaedf96bd>] SyS_madvise+0x49d/0xac0
Complete report #1183098
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8291dba6
Build id: 3.10.0-1160.118.1.el7.x86_64
0x106
2
? leave_mm
vmlinux 0xffffffff82881b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffff82881b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffff82881b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffff8291dcbf
Build id: 3.10.0-1160.118.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff82b9c615
Build id: 3.10.0-1160.118.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff8291e27b
Build id: 3.10.0-1160.118.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff82881fa8
Build id: 3.10.0-1160.118.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8288201b
Build id: 3.10.0-1160.118.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff829fa0ce
Build id: 3.10.0-1160.118.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffff829fbc4f
Build id: 3.10.0-1160.118.1.el7.x86_64
0x3f
12
arch_tlb_finish_mmu
vmlinux 0xffffffff829fbc4f
Build id: 3.10.0-1160.118.1.el7.x86_64
0x3f
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ping:6740]
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [java:6921]
Modules linked in: xt_statistic tcp_lp nls_utf8 isofs ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle vxlan ip6_udp_tunnel udp_tunnel xt_physdev xt_NFLOG nfnetlink_log xt_limit xt_mark xt_multiport xt_comment ip_set veth xt_nat nf_conntrack_netlink xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache overlay(T) br_netfilter bridge stp llc vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus iosf_mbi snd_seq
crc32_pclmul snd_seq_device ghash_clmulni_intel snd_pcm aesni_intel lrw gf128mul snd_timer snd soundcore glue_helper ablk_helper cryptd vmw_balloon ppdev pcspkr sg joydev parport_pc parport i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix nfit crct10dif_pclmul crct10dif_common crc32c_intel libnvdimm libata mptspi scsi_transport_spi e1000 mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 6921 Comm: java Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.118.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8c253262b180 ti: ffff8c2556518000 task.ti: ffff8c2556518000
RIP: 0010:[<ffffffff8291dba6>]  [<ffffffff8291dba6>] generic_exec_single+0x106/0x1c0
RSP: 0018:ffff8c255651bbe0  EFLAGS: 00000202
RAX: ffff8c2b9d620600 RBX: ffffffff828eaea9 RCX: ffff8c2b9d620628
RDX: ffff8c2b9d61b840 RSI: ffff8c255651bbe0 RDI: ffff8c255651bbe0
RBP: ffff8c255651bc30 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff8c261fcf10e0 R11: 00007f6e98e54000 R12: ffff8c2aa4268300
R13: ffff8c2aa4268300 R14: 0000000000000002 R15: ffff8c2af1cda400
FS:  00007f6e98f51700(0000) GS:ffff8c2b9d6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9e2baa6000 CR3: 0000000706564000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff82881b10>] ? leave_mm+0x130/0x130
[<ffffffff82881b10>] ? leave_mm+0x130/0x130
[<ffffffff82881b10>] ? leave_mm+0x130/0x130
[<ffffffff8291dcbf>] smp_call_function_single+0x5f/0xa0
[<ffffffff82b9c615>] ? cpumask_next_and+0x35/0x50
[<ffffffff8291e27b>] smp_call_function_many+0x22b/0x270
[<ffffffff82881fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8288201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff829fa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff829fbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff829fbc4f>] arch_tlb_finish_mmu+0x3f/0x80
Complete report #1088964
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9f1167a2
Build id: 3.10.0-1160.15.2.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff9f07e6a0
Build id: 3.10.0-1160.15.2.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9f07e6a0
Build id: 3.10.0-1160.15.2.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9f07e6a0
Build id: 3.10.0-1160.15.2.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff9f1168bf
Build id: 3.10.0-1160.15.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9f3862c5
Build id: 3.10.0-1160.15.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9f116e6b
Build id: 3.10.0-1160.15.2.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9f07e868
Build id: 3.10.0-1160.15.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9f07e8db
Build id: 3.10.0-1160.15.2.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9f1ede86
Build id: 3.10.0-1160.15.2.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff9f1ef94f
Build id: 3.10.0-1160.15.2.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff9f1efa73
Build id: 3.10.0-1160.15.2.el7.x86_64
0x23
13
unmap_region
vmlinux 0xffffffff9f1f9264
Build id: 3.10.0-1160.15.2.el7.x86_64
0xf4
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 27s! [sentinel-system:9577]
Modules linked in: binfmt_misc resguard_linux(OE) secmodel_linux(OE) syshook_linux(OE) secmodel3(OE) xt_set ip_set_hash_net xt_NFLOG nfnetlink_log ip_set xt_CHECKSUM iptable_raw ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock nf_conntrack_netlink nfnetlink nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack sunrpc ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev joydev vmw_balloon pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm crct10dif_pclmul crct10dif_common crc32c_intel mptspi libata scsi_transport_spi mptscsih serio_raw mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 10 PID: 9577 Comm: sentinel-system Kdump: loaded Tainted: G        W  OEL ------------   3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8d25c1ff4200 ti: ffff8d25e1fc8000 task.ti: ffff8d25e1fc8000
RIP: 0010:[<ffffffff9f1167a2>]  [<ffffffff9f1167a2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8d25e1fcbc20  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8d25e1fcbbf0 RCX: 0000000000000002
RDX: ffffffff9f813a50 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8d25e1fcbc70 R08: ffffffff9f813a48 R09: ffffffff9f20288c
R10: ffff8d347d89f0a0 R11: ffffe7c12da45380 R12: 00000000000280da
R13: ffffffff9f23e502 R14: ffff8d25e1fcbbd8 R15: 0000000000000000
FS:  00007f00988d2700(0000) GS:ffff8d347d880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f00e0004000 CR3: 000000017c382000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9f07e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9f07e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9f07e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9f1168bf>] smp_call_function_single+0x5f/0xa0
[<ffffffff9f3862c5>] ? cpumask_next_and+0x35/0x50
[<ffffffff9f116e6b>] smp_call_function_many+0x22b/0x270
[<ffffffff9f07e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9f07e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9f1ede86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9f1ef94f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9f1efa73>] tlb_finish_mmu+0x23/0x30
[<ffffffff9f1f9264>] unmap_region+0xf4/0x140
Complete report #967987
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8a716842
Build id: 3.10.0-1160.25.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff8a67e6a0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff8a67e6a0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff8a67e6a0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff8a71695f
Build id: 3.10.0-1160.25.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff8a986395
Build id: 3.10.0-1160.25.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff8a716f0b
Build id: 3.10.0-1160.25.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8a67e868
Build id: 3.10.0-1160.25.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8a67e8db
Build id: 3.10.0-1160.25.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8a7edf56
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff8a7efa1f
Build id: 3.10.0-1160.25.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff8a7efb43
Build id: 3.10.0-1160.25.1.el7.x86_64
0x23
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [Executor Servic:7947]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) mfe_aac_100717845(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc ext4 iosf_mbi mbcache jbd2 crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon pcspkr sg joydev parport_pc parport i2c_piix4 vmw_vmci binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt sd_mod fb_sys_fops ttm crc_t10dif crct10dif_generic
ahci drm nfit ata_piix libahci crct10dif_pclmul libnvdimm crct10dif_common crc32c_intel libata serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 7947 Comm: Executor Servic Tainted: G           OEL ------------ T 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9eb12d61a100 ti: ffff9eb2b0d60000 task.ti: ffff9eb2b0d60000
RIP: 0010:[<ffffffff8a716842>]  [<ffffffff8a716842>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9eb2b0d63c20  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9eb2b0d63bf0 RCX: 000000000000000f
RDX: ffffffff8ae15ad0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9eb2b0d63c70 R08: ffffffff8ae15ac8 R09: ffffffff8a80295c
R10: ffff9eb4956df0a0 R11: ffffee77dea9d240 R12: 00000000000280da
R13: ffffffff8a83e5d2 R14: ffff9eb2b0d63bd8 R15: 0000000000000000
FS:  00007fdf023e3700(0000) GS:ffff9eb4956c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdf3c09f000 CR3: 00000005f09a2000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff8a67e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8a67e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8a67e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8a71695f>] smp_call_function_single+0x5f/0xa0
[<ffffffff8a986395>] ? cpumask_next_and+0x35/0x50
[<ffffffff8a716f0b>] smp_call_function_many+0x22b/0x270
[<ffffffff8a67e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8a67e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8a7edf56>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8a7efa1f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8a7efb43>] tlb_finish_mmu+0x23/0x30
Complete report #924755
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8e116b82
Build id: 3.10.0-1160.62.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff8e07ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff8e07ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff8e07ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff8e116c9f
Build id: 3.10.0-1160.62.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff8e386825
Build id: 3.10.0-1160.62.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff8e11724b
Build id: 3.10.0-1160.62.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8e07e258
Build id: 3.10.0-1160.62.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8e07e2cb
Build id: 3.10.0-1160.62.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8e1ee666
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff8e1f012f
Build id: 3.10.0-1160.62.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff8e1f0253
Build id: 3.10.0-1160.62.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff8e1f371a
Build id: 3.10.0-1160.62.1.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffff8e1edc0d
Build id: 3.10.0-1160.62.1.el7.x86_64
0x49d
15
? __switch_to
vmlinux 0xffffffff8e02b621
Build id: 3.10.0-1160.62.1.el7.x86_64
0x151
16
system_call_fastpath
vmlinux 0xffffffff8e799f92
Build id: 3.10.0-1160.62.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 69s! [containerd-shim:3153]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi kvm_intel kvm snd_ens1371 irqbypass snd_rawmidi crc32_pclmul snd_ac97_codec ac97_bus ghash_clmulni_intel snd_seq aesni_intel lrw snd_seq_device gf128mul snd_pcm snd_timer ppdev glue_helper ablk_helper cryptd snd vmw_balloon soundcore joydev pcspkr sg parport_pc
parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mptspi ttm crct10dif_pclmul crct10dif_common scsi_transport_spi ata_piix crc32c_intel mptscsih drm nfit libata mptbase serio_raw libnvdimm e1000 drm_panel_orientation_quirks fuse
CPU: 5 PID: 3153 Comm: containerd-shim Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8a8188049080 ti: ffff8a80a16fc000 task.ti: ffff8a80a16fc000
RIP: 0010:[<ffffffff8e116b82>]  [<ffffffff8e116b82>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8a80a16ffbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8a80a16ffbb0 RCX: 0000000000000008
RDX: ffffffff8e814950 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8a80a16ffc30 R08: ffffffff8e814948 R09: ffffd8a582da5500
R10: 0000000000004190 R11: 0000000000000202 R12: ffff8a8188049080
R13: ffff8a82299c9680 R14: 00000000000000b5 R15: ffff8a7e357b1400
FS:  000000c000080090(0000) GS:ffff8a822d740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f05c112de00 CR3: 000000028e616000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8e07ddd0>] ? leave_mm+0x120/0x120
[<ffffffff8e07ddd0>] ? leave_mm+0x120/0x120
[<ffffffff8e07ddd0>] ? leave_mm+0x120/0x120
[<ffffffff8e116c9f>] smp_call_function_single+0x5f/0xa0
[<ffffffff8e386825>] ? cpumask_next_and+0x35/0x50
[<ffffffff8e11724b>] smp_call_function_many+0x22b/0x270
[<ffffffff8e07e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8e07e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8e1ee666>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8e1f012f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8e1f0253>] tlb_finish_mmu+0x23/0x30
[<ffffffff8e1f371a>] zap_page_range+0xea/0x130
[<ffffffff8e1edc0d>] SyS_madvise+0x49d/0xac0
[<ffffffff8e02b621>] ? __switch_to+0x151/0x580
[<ffffffff8e799f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 60 18 d5 8e 48 89 df e8 17 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #911210
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffab91db82
Build id: 3.10.0-1160.108.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffab881b10
Build id: 3.10.0-1160.108.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffffab881b10
Build id: 3.10.0-1160.108.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffffab881b10
Build id: 3.10.0-1160.108.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffffab91dc9f
Build id: 3.10.0-1160.108.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffabb9c5d5
Build id: 3.10.0-1160.108.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffab91e25b
Build id: 3.10.0-1160.108.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffab881fa8
Build id: 3.10.0-1160.108.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffab88201b
Build id: 3.10.0-1160.108.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffab9fa07e
Build id: 3.10.0-1160.108.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffffab9fbbff
Build id: 3.10.0-1160.108.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffab9fbd33
Build id: 3.10.0-1160.108.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffab9ff27a
Build id: 3.10.0-1160.108.1.el7.x86_64
0xea
14
? do_sync_write
vmlinux 0xffffffffaba5b263
Build id: 3.10.0-1160.108.1.el7.x86_64
0x93
15
SyS_madvise
vmlinux 0xffffffffab9f966d
Build id: 3.10.0-1160.108.1.el7.x86_64
0x49d
16
? vfs_write
vmlinux 0xffffffffaba5bdd8
Build id: 3.10.0-1160.108.1.el7.x86_64
0x168
17
system_call_fastpath
vmlinux 0xffffffffabfc539a
Build id: 3.10.0-1160.108.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [StreamTrans #93:8150]
Modules linked in: tcp_lp nls_utf8 isofs xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi snd_ens1371 crc32_pclmul snd_rawmidi snd_ac97_codec ghash_clmulni_intel ac97_bus ppdev snd_seq snd_seq_device aesni_intel vmw_balloon lrw gf128mul snd_pcm glue_helper ablk_helper cryptd snd_timer
sg joydev pcspkr snd vmw_vmci soundcore i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect ata_piix sysimgblt fb_sys_fops ttm nfit crct10dif_pclmul crct10dif_common drm mptspi libnvdimm libata crc32c_intel scsi_transport_spi mptscsih e1000 mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 8150 Comm: StreamTrans #93 Kdump: loaded Not tainted 3.10.0-1160.108.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8d9bd656b180 ti: ffff8d9c8d798000 task.ti: ffff8d9c8d798000
RIP: 0010:[<ffffffffab91db82>]  [<ffffffffab91db82>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8d9c8d79bbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8d9c8d79bbb0 RCX: 0000000000000002
RDX: ffffffffac013c50 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8d9c8d79bc30 R08: ffffffffac013c48 R09: 00003ffffffff000
R10: ffff8d9cd871d440 R11: 00007f598b037000 R12: ffffffffff768ca5
R13: 000000008d79bb98 R14: ffffffffab8ebe81 R15: ffff8d9c8d79bb80
FS:  00007f598b075700(0000) GS:ffff8d9cf7cc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f59657a8400 CR3: 000000019aff0000 CR4: 0000000000360fe0
Call Trace:
[<ffffffffab881b10>] ? leave_mm+0x130/0x130
[<ffffffffab881b10>] ? leave_mm+0x130/0x130
[<ffffffffab881b10>] ? leave_mm+0x130/0x130
[<ffffffffab91dc9f>] smp_call_function_single+0x5f/0xa0
[<ffffffffabb9c5d5>] ? cpumask_next_and+0x35/0x50
[<ffffffffab91e25b>] smp_call_function_many+0x22b/0x270
[<ffffffffab881fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffab88201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffab9fa07e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffab9fbbff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffab9fbd33>] tlb_finish_mmu+0x23/0x40
[<ffffffffab9ff27a>] zap_page_range+0xea/0x140
[<ffffffffaba5b263>] ? do_sync_write+0x93/0xe0
[<ffffffffab9f966d>] SyS_madvise+0x49d/0xac0
[<ffffffffaba5bdd8>] ? vfs_write+0x168/0x1f0
[<ffffffffabfc539a>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 80 19 55 ac 48 89 df e8 97 67 29 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #894356
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8931db12
Build id: 3.10.0-1160.95.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff89281b10
Build id: 3.10.0-1160.95.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffff89281b10
Build id: 3.10.0-1160.95.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffff89281b10
Build id: 3.10.0-1160.95.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffff8931dc2f
Build id: 3.10.0-1160.95.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff8959c565
Build id: 3.10.0-1160.95.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff8931e1eb
Build id: 3.10.0-1160.95.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff89281fa8
Build id: 3.10.0-1160.95.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8928201b
Build id: 3.10.0-1160.95.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff893fa00e
Build id: 3.10.0-1160.95.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffff893fbb8f
Build id: 3.10.0-1160.95.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff893fbcc3
Build id: 3.10.0-1160.95.1.el7.x86_64
0x23
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [synology-backup:38391]
Modules linked in: xt_multiport xt_set ipt_rpfilter iptable_raw ip_set_hash_ip ipt_REJECT nf_reject_ipv4 ip_set_hash_net ip_set vxlan ip6_udp_tunnel udp_tunnel ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_filter ip6table_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables xt_mark xt_comment iptable_mangle nf_tables veth xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat xt_addrtype iptable_filter br_netfilter bridge stp llc overlay(T) ip_vs nf_conntrack vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac snd_ens1371 snd_rawmidi snd_ac97_codec btusb btrtl ac97_bus btbcm btintel iosf_mbi ppdev snd_seq crc32_pclmul bluetooth ghash_clmulni_intel
snd_seq_device aesni_intel snd_pcm lrw gf128mul rfkill snd_timer pcspkr snd glue_helper ablk_helper vmw_balloon cryptd i2c_piix4 joydev soundcore sg vmw_vmci parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul nfit crct10dif_common crc32c_intel drm libata e1000 libnvdimm nvme serio_raw nvme_core drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod synosnap(OE)
CPU: 7 PID: 38391 Comm: synology-backup Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-1160.95.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9a12464bd280 ti: ffff9a12f7948000 task.ti: ffff9a12f7948000
RIP: 0010:[<ffffffff8931db12>]  [<ffffffff8931db12>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9a12f794bbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9a12f794bbb0 RCX: 0000000000000002
RDX: ffffffff89a13bd0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9a12f794bc30 R08: ffffffff89a13bc8 R09: 00003ffffffff000
R10: ffff9a14afef1290 R11: 0000000000000246 R12: ffff9a12464bd280
R13: ffffffff898ecace R14: ffff9a12f794bb58 R15: 0000000000000000
FS:  00007f944bfff700(0000) GS:ffff9a1841fc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000004d00a3 CR3: 00000005ffaf8000 CR4: 00000000003607e0
Call Trace:
[<ffffffff89281b10>] ? leave_mm+0x130/0x130
[<ffffffff89281b10>] ? leave_mm+0x130/0x130
[<ffffffff89281b10>] ? leave_mm+0x130/0x130
[<ffffffff8931dc2f>] smp_call_function_single+0x5f/0xa0
[<ffffffff8959c565>] ? cpumask_next_and+0x35/0x50
[<ffffffff8931e1eb>] smp_call_function_many+0x22b/0x270
[<ffffffff89281fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8928201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff893fa00e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff893fbb8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff893fbcc3>] tlb_finish_mmu+0x23/0x40
Complete report #876909
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb2d1db12
Build id: 3.10.0-1160.95.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffb2c81b10
Build id: 3.10.0-1160.95.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffffb2c81b10
Build id: 3.10.0-1160.95.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffffb2c81b10
Build id: 3.10.0-1160.95.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffffb2d1dc2f
Build id: 3.10.0-1160.95.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb2f9c565
Build id: 3.10.0-1160.95.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb2d1e1eb
Build id: 3.10.0-1160.95.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb2c81fa8
Build id: 3.10.0-1160.95.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb2c8201b
Build id: 3.10.0-1160.95.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb2dfa00e
Build id: 3.10.0-1160.95.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb2dfbb8f
Build id: 3.10.0-1160.95.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb2dfbcc3
Build id: 3.10.0-1160.95.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffb2dff20a
Build id: 3.10.0-1160.95.1.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffffb2df95fd
Build id: 3.10.0-1160.95.1.el7.x86_64
0x49d
15
tracesys
vmlinux 0xffffffffb33c562e
Build id: 3.10.0-1160.95.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [s1-perf:7110]
Modules linked in: devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg parport_pc vmw_vmci parport i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops nfit ttm crct10dif_pclmul crct10dif_common libnvdimm crc32c_intel serio_raw ata_piix drm libata e1000 mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 7110 Comm: s1-perf Kdump: loaded Tainted: G        W      ------------ T 3.10.0-1160.95.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8962000ac200 ti: ffff89633ae4c000 task.ti: ffff89633ae4c000
RIP: 0010:[<ffffffffb2d1db12>]  [<ffffffffb2d1db12>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff89633ae4fbe0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff89633ae4fbb0 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff89633ae4fc30 R08: ffffffffb3413e40 R09: 00003ffffffff000
R10: ffff8960b18cd1b0 R11: 00007f463150a000 R12: 0000000100000002
R13: ffff89653fd1acc0 R14: ffff8960704e7000 R15: 0000000000000087
FS:  00007f4631d09700(0000) GS:ffff89653fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006dee20 CR3: 00000006262a4000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb2c81b10>] ? leave_mm+0x130/0x130
[<ffffffffb2c81b10>] ? leave_mm+0x130/0x130
[<ffffffffb2c81b10>] ? leave_mm+0x130/0x130
[<ffffffffb2d1dc2f>] smp_call_function_single+0x5f/0xa0
[<ffffffffb2f9c565>] ? cpumask_next_and+0x35/0x50
[<ffffffffb2d1e1eb>] smp_call_function_many+0x22b/0x270
[<ffffffffb2c81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb2c8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb2dfa00e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffb2dfbb8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb2dfbcc3>] tlb_finish_mmu+0x23/0x40
[<ffffffffb2dff20a>] zap_page_range+0xea/0x140
[<ffffffffb2df95fd>] SyS_madvise+0x49d/0xac0
[<ffffffffb33c562e>] tracesys+0xa6/0xcc
Code: 89 de 48 03 14 c5 00 19 95 b3 48 89 df e8 87 67 29 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #855084
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb6d16b62
Build id: 3.10.0-1160.71.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffb6c7ddd0
Build id: 3.10.0-1160.71.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffb6c7ddd0
Build id: 3.10.0-1160.71.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffb6c7ddd0
Build id: 3.10.0-1160.71.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffb6d16c7f
Build id: 3.10.0-1160.71.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb6f86865
Build id: 3.10.0-1160.71.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb6d1722b
Build id: 3.10.0-1160.71.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb6c7e258
Build id: 3.10.0-1160.71.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb6c7e2cb
Build id: 3.10.0-1160.71.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb6dee6c6
Build id: 3.10.0-1160.71.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb6df018f
Build id: 3.10.0-1160.71.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb6df02b3
Build id: 3.10.0-1160.71.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffb6df377a
Build id: 3.10.0-1160.71.1.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffffb6dedc6d
Build id: 3.10.0-1160.71.1.el7.x86_64
0x49d
15
? switch_mm_irqs_off
vmlinux 0xffffffffb6c7dff5
Build id: 3.10.0-1160.71.1.el7.x86_64
0x115
16
? __switch_to
vmlinux 0xffffffffb6c2b621
Build id: 3.10.0-1160.71.1.el7.x86_64
0x151
17
? __schedule
vmlinux 0xffffffffb738c070
Build id: 3.10.0-1160.71.1.el7.x86_64
0x320
18
system_call_fastpath
vmlinux 0xffffffffb7399f92
Build id: 3.10.0-1160.71.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [Common:1911]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 crc32_pclmul ppdev ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec aesni_intel ac97_bus lrw gf128mul glue_helper ablk_helper snd_seq vmw_balloon snd_seq_device cryptd snd_pcm joydev pcspkr snd_timer sg snd soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi nfit libnvdimm vmwgfx drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix mptspi scsi_transport_spi mptscsih serio_raw libata e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 1911 Comm: Common Tainted: G             L ------------   3.10.0-1160.71.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffffa0846357a100 ti: ffffa0846358c000 task.ti: ffffa0846358c000
RIP: 0010:[<ffffffffb6d16b62>]  [<ffffffffb6d16b62>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffffa0846358fbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffffa0846358fbb0 RCX: 0000000000000003
RDX: ffffffffb7413cd0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffffa0846358fc30 R08: ffffffffb7413cc8 R09: 00003ffffffff000
R10: ffffa08619445a28 R11: 0000000000000206 R12: ffffffffb6dc760e
R13: ffffa0846358fbe0 R14: 0000000000000002 R15: ffffcff0472d5980
FS:  00007fbd3b8d4700(0000) GS:ffffa086f1640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5a7e660000 CR3: 000000032f1c2000 CR4: 0000000000360fe0
Call Trace:
[<ffffffffb6c7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb6c7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb6c7ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb6d16c7f>] smp_call_function_single+0x5f/0xa0
[<ffffffffb6f86865>] ? cpumask_next_and+0x35/0x50
[<ffffffffb6d1722b>] smp_call_function_many+0x22b/0x270
[<ffffffffb6c7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb6c7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb6dee6c6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb6df018f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb6df02b3>] tlb_finish_mmu+0x23/0x30
[<ffffffffb6df377a>] zap_page_range+0xea/0x130
[<ffffffffb6dedc6d>] SyS_madvise+0x49d/0xac0
[<ffffffffb6c7dff5>] ? switch_mm_irqs_off+0x115/0x290
[<ffffffffb6c2b621>] ? __switch_to+0x151/0x580
[<ffffffffb738c070>] ? __schedule+0x320/0x680
[<ffffffffb7399f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 60 18 95 b7 48 89 df e8 77 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #848331
Frame # Function Binary Source or offset Line
1
? leave_mm
vmlinux 0xffffffffa8a81b10
Build id: 3.10.0-1160.102.1.el7.x86_64
0x130
2
? leave_mm
vmlinux 0xffffffffa8a81b10
Build id: 3.10.0-1160.102.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffffa8a81b10
Build id: 3.10.0-1160.102.1.el7.x86_64
0x130
4
smp_call_function_single
vmlinux 0xffffffffa8b1dc9f
Build id: 3.10.0-1160.102.1.el7.x86_64
0x5f
5
? cpumask_next_and
vmlinux 0xffffffffa8d9c5d5
Build id: 3.10.0-1160.102.1.el7.x86_64
0x35
6
smp_call_function_many
vmlinux 0xffffffffa8b1e25b
Build id: 3.10.0-1160.102.1.el7.x86_64
0x22b
7
native_flush_tlb_others
vmlinux 0xffffffffa8a81fa8
Build id: 3.10.0-1160.102.1.el7.x86_64
0xb8
8
flush_tlb_mm_range
vmlinux 0xffffffffa8a8201b
Build id: 3.10.0-1160.102.1.el7.x86_64
0x6b
9
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa8bfa07e
Build id: 3.10.0-1160.102.1.el7.x86_64
0xae
10
arch_tlb_finish_mmu
vmlinux 0xffffffffa8bfbbff
Build id: 3.10.0-1160.102.1.el7.x86_64
0x3f
11
tlb_finish_mmu
vmlinux 0xffffffffa8bfbd33
Build id: 3.10.0-1160.102.1.el7.x86_64
0x23
12
zap_page_range
vmlinux 0xffffffffa8bff27a
Build id: 3.10.0-1160.102.1.el7.x86_64
0xea
13
? handle_mm_fault
vmlinux 0xffffffffa8c02503
Build id: 3.10.0-1160.102.1.el7.x86_64
0xdb3
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 93s! [kworker/u34:2:23897]
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa8a81b10>] ? leave_mm+0x130/0x130
[<ffffffffa8a81b10>] ? leave_mm+0x130/0x130
[<ffffffffa8a81b10>] ? leave_mm+0x130/0x130
[<ffffffffa8b1dc9f>] smp_call_function_single+0x5f/0xa0
[<ffffffffa8d9c5d5>] ? cpumask_next_and+0x35/0x50
[<ffffffffa8b1e25b>] smp_call_function_many+0x22b/0x270
[<ffffffffa8a81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa8a8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa8bfa07e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa8bfbbff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa8bfbd33>] tlb_finish_mmu+0x23/0x40
[<ffffffffa8bff27a>] zap_page_range+0xea/0x140
[<ffffffffa8c02503>] ? handle_mm_fault+0xdb3/0x1190
Complete report #784637
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffaa71d9c2
Build id: 3.10.0-1160.81.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffaa681ab0
Build id: 3.10.0-1160.81.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffffaa681ab0
Build id: 3.10.0-1160.81.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffffaa681ab0
Build id: 3.10.0-1160.81.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffffaa71dadf
Build id: 3.10.0-1160.81.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffaa99bf55
Build id: 3.10.0-1160.81.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffaa71e09b
Build id: 3.10.0-1160.81.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffaa681f48
Build id: 3.10.0-1160.81.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffaa681fbb
Build id: 3.10.0-1160.81.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaa7f9f0e
Build id: 3.10.0-1160.81.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffffaa7fba4f
Build id: 3.10.0-1160.81.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffaa7fbb83
Build id: 3.10.0-1160.81.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffaa7ff0ca
Build id: 3.10.0-1160.81.1.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffffaa7f949d
Build id: 3.10.0-1160.81.1.el7.x86_64
0x49d
15
system_call_fastpath
vmlinux 0xffffffffaadc539a
Build id: 3.10.0-1160.81.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [qtp192881625-17:73844]
Modules linked in: vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event vfat fat snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw soundcore gf128mul glue_helper ablk_helper cryptd ppdev sg joydev tpm_crb vmw_balloon pcspkr parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci ata_piix libahci e1000 libata mptsas crct10dif_pclmul crct10dif_common scsi_transport_sas crc32c_intel mptscsih nfit serio_raw mptbase libnvdimm drm_panel_orientation_quirks fuse
CPU: 2 PID: 73844 Comm: qtp192881625-17 Kdump: loaded Tainted: G        W      ------------   3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.14410784.B64.1908150010 08/15/2019
task: ffff95d9aa266300 ti: ffff95d759b98000 task.ti: ffff95d759b98000
RIP: 0010:[<ffffffffaa71d9c2>]  [<ffffffffaa71d9c2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff95d759b9bbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff95d759b9bbb0 RCX: 0000000000000001
RDX: ffffffffaae138d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff95d759b9bc30 R08: ffffffffaae138c8 R09: 00003ffffffff000
R10: ffff95d9a44b6f30 R11: 00007feff28dc000 R12: 0000000000000003
R13: ffffec09428b89b0 R14: 00007feff28dc000 R15: 0000000100400029
FS:  00007feff2918700(0000) GS:ffff95da6d680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000800df4408 CR3: 000000042bd5c000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffaa681ab0>] ? leave_mm+0x130/0x130
[<ffffffffaa681ab0>] ? leave_mm+0x130/0x130
[<ffffffffaa681ab0>] ? leave_mm+0x130/0x130
[<ffffffffaa71dadf>] smp_call_function_single+0x5f/0xa0
[<ffffffffaa99bf55>] ? cpumask_next_and+0x35/0x50
[<ffffffffaa71e09b>] smp_call_function_many+0x22b/0x270
[<ffffffffaa681f48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaa681fbb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaa7f9f0e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffaa7fba4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaa7fbb83>] tlb_finish_mmu+0x23/0x40
[<ffffffffaa7ff0ca>] zap_page_range+0xea/0x140
[<ffffffffaa7f949d>] SyS_madvise+0x49d/0xac0
[<ffffffffaadc539a>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 00 19 35 ab 48 89 df e8 d7 62 29 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #748679
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffaa116ae2
Build id: 3.10.0-1160.76.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffaa07ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffaa07ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffaa07ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffaa116bff
Build id: 3.10.0-1160.76.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffaa386845
Build id: 3.10.0-1160.76.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffaa1171ab
Build id: 3.10.0-1160.76.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffaa07e258
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffaa07e2cb
Build id: 3.10.0-1160.76.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaa1ee6a6
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffaa1f016f
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffaa1f0293
Build id: 3.10.0-1160.76.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffaa1f375a
Build id: 3.10.0-1160.76.1.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffffaa1edc4d
Build id: 3.10.0-1160.76.1.el7.x86_64
0x49d
15
system_call_fastpath
vmlinux 0xffffffffaa799f92
Build id: 3.10.0-1160.76.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [elasticsearch[a:12290]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus sb_edac snd_seq snd_seq_device iosf_mbi crc32_pclmul snd_pcm ghash_clmulni_intel
aesni_intel snd_timer btusb lrw snd btrtl btbcm btintel gf128mul bluetooth soundcore glue_helper rfkill ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix nfit mptspi libnvdimm libata scsi_transport_spi serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks synosnap(OE) fuse
CPU: 5 PID: 12290 Comm: elasticsearch[a Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffffa0ef7082b180 ti: ffffa0f11ce90000 task.ti: ffffa0f11ce90000
RIP: 0010:[<ffffffffaa116ae2>]  [<ffffffffaa116ae2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffffa0f11ce93be0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffffa0f11ce93bb0 RCX: 0000000000000007
RDX: ffffffffaa8146d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffffa0f11ce93c30 R08: ffffffffaa8146c8 R09: 00003ffffffff000
R10: ffffa0f10859eaf8 R11: 00007efa2084d000 R12: ffffa0ef7082b180
R13: ffffeafb88cd4370 R14: 00007efa2084d000 R15: 000000018040003f
FS:  00007efa20949700(0000) GS:ffffa0f11d740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdf0a8ef000 CR3: 0000000472c66000 CR4: 00000000003607e0
Call Trace:
[<ffffffffaa07ddd0>] ? leave_mm+0x120/0x120
[<ffffffffaa07ddd0>] ? leave_mm+0x120/0x120
[<ffffffffaa07ddd0>] ? leave_mm+0x120/0x120
[<ffffffffaa116bff>] smp_call_function_single+0x5f/0xa0
[<ffffffffaa386845>] ? cpumask_next_and+0x35/0x50
[<ffffffffaa1171ab>] smp_call_function_many+0x22b/0x270
[<ffffffffaa07e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaa07e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaa1ee6a6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaa1f016f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaa1f0293>] tlb_finish_mmu+0x23/0x30
[<ffffffffaa1f375a>] zap_page_range+0xea/0x130
[<ffffffffaa1edc4d>] SyS_madvise+0x49d/0xac0
[<ffffffffaa799f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 00 19 d5 aa 48 89 df e8 d7 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #674729
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9891db16
Build id: 3.10.0-1160.88.1.el7.x86_64
0x106
2
? leave_mm
vmlinux 0xffffffff98881b10
Build id: 3.10.0-1160.88.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffff98881b10
Build id: 3.10.0-1160.88.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffff98881b10
Build id: 3.10.0-1160.88.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffff9891dc2f
Build id: 3.10.0-1160.88.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff98b9c485
Build id: 3.10.0-1160.88.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9891e1eb
Build id: 3.10.0-1160.88.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff98881fa8
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9888201b
Build id: 3.10.0-1160.88.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff989fa00e
Build id: 3.10.0-1160.88.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffff989fbb8f
Build id: 3.10.0-1160.88.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff989fbcc3
Build id: 3.10.0-1160.88.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff989ff20a
Build id: 3.10.0-1160.88.1.el7.x86_64
0xea
14
? handle_mm_fault
vmlinux 0xffffffff98a02493
Build id: 3.10.0-1160.88.1.el7.x86_64
0xdb3
15
SyS_madvise
vmlinux 0xffffffff989f95fd
Build id: 3.10.0-1160.88.1.el7.x86_64
0x49d
16
system_call_fastpath
vmlinux 0xffffffff98fc539a
Build id: 3.10.0-1160.88.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [code:23982]
Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype mfe_fileaccess_100714877(OE) cbsensor_2_5_1230541(OE) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter mfe_aac_100714877(OE) vmw_vsock_vmci_transport vsock ppdev vmw_balloon joydev sg vmw_vmci i2c_piix4 pcspkr parport_pc parport nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c
sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic crct10dif_common pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit mptspi ata_piix scsi_transport_spi libata libnvdimm mptscsih mptbase serio_raw crc32c_intel vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse br_netfilter bridge stp llc overlay(T)
CPU: 9 PID: 23982 Comm: code Tainted: G           OEL ------------ T 3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffff8fc48e04a100 ti: ffff8fc35d558000 task.ti: ffff8fc35d558000
RIP: 0010:[<ffffffff9891db16>]  [<ffffffff9891db16>] generic_exec_single+0x106/0x1c0
RSP: 0018:ffff8fc35d55bbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8fc35d55bbb0 RCX: 0000000000000004
RDX: ffffffff99014050 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8fc35d55bc30 R08: ffffffff99014048 R09: 00003ffffffff000
R10: ffff8fc471177e60 R11: 0000000000000202 R12: ffff8fc300000000
R13: ffff8fc35fb85701 R14: 0000000000000000 R15: 000000010155bee0
FS:  00007fbd12c11bc0(0000) GS:ffff8fc493840000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000018fc012ca000 CR3: 0000000952404000 CR4: 00000000000007e0
Call Trace:
[<ffffffff98881b10>] ? leave_mm+0x130/0x130
[<ffffffff98881b10>] ? leave_mm+0x130/0x130
[<ffffffff98881b10>] ? leave_mm+0x130/0x130
[<ffffffff9891dc2f>] smp_call_function_single+0x5f/0xa0
[<ffffffff98b9c485>] ? cpumask_next_and+0x35/0x50
[<ffffffff9891e1eb>] smp_call_function_many+0x22b/0x270
[<ffffffff98881fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9888201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff989fa00e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff989fbb8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff989fbcc3>] tlb_finish_mmu+0x23/0x40
[<ffffffff989ff20a>] zap_page_range+0xea/0x140
[<ffffffff98a02493>] ? handle_mm_fault+0xdb3/0x1190
[<ffffffff989f95fd>] SyS_madvise+0x49d/0xac0
[<ffffffff98fc539a>] system_call_fastpath+0x25/0x2a
Code: 14 c5 00 19 55 99 48 89 df e8 c7 66 29 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 f6 43 20 01 <75> f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 00 0f 85 8f 
Complete report #628432
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffaab1da02
Build id: 3.10.0-1160.83.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffaaa81af0
Build id: 3.10.0-1160.83.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffffaaa81af0
Build id: 3.10.0-1160.83.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffffaaa81af0
Build id: 3.10.0-1160.83.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffffaab1db1f
Build id: 3.10.0-1160.83.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffaad9bf95
Build id: 3.10.0-1160.83.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffaab1e0db
Build id: 3.10.0-1160.83.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffaaa81f88
Build id: 3.10.0-1160.83.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffaaa81ffb
Build id: 3.10.0-1160.83.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaabf9f4e
Build id: 3.10.0-1160.83.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffffaabfba8f
Build id: 3.10.0-1160.83.1.el7.x86_64
0x3f
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [TaskCon~ller #0:107045]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 ppdev aesni_intel snd_rawmidi snd_ac97_codec lrw gf128mul pcspkr glue_helper ablk_helper cryptd joydev ac97_bus snd_seq snd_seq_device snd_pcm vmw_balloon snd_timer
snd soundcore sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix nfit libata libnvdimm mptspi e1000 crct10dif_pclmul crct10dif_common crc32c_intel scsi_transport_spi mptscsih mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 107045 Comm: TaskCon~ller #0 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.83.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9331b1d96300 ti: ffff933139288000 task.ti: ffff933139288000
RIP: 0010:[<ffffffffaab1da02>]  [<ffffffffaab1da02>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff93313928bbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff93313928bbb0 RCX: 0000000000000001
RDX: ffffffffab2138d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff93313928bc30 R08: ffffffffab2138c8 R09: 00003ffffffff000
R10: ffff932fcabb10e0 R11: 0000000000000246 R12: ffff9331b1d96300
R13: ffffffffaaae8bb4 R14: ffff93313928bb80 R15: ffff93313928bb80
FS:  00007f02d42ff700(0000) GS:ffff9331b56c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f02cb08dff0 CR3: 000000004abf0000 CR4: 00000000003607e0
Call Trace:
[<ffffffffaaa81af0>] ? leave_mm+0x130/0x130
[<ffffffffaaa81af0>] ? leave_mm+0x130/0x130
[<ffffffffaaa81af0>] ? leave_mm+0x130/0x130
[<ffffffffaab1db1f>] smp_call_function_single+0x5f/0xa0
[<ffffffffaad9bf95>] ? cpumask_next_and+0x35/0x50
[<ffffffffaab1e0db>] smp_call_function_many+0x22b/0x270
[<ffffffffaaa81f88>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaaa81ffb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaabf9f4e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffaabfba8f>] arch_tlb_finish_mmu+0x3f/0x80
Complete report #593767
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffaff163d6
Build id: 3.10.0-1062.12.1.el7.x86_64
0x106
2
? leave_mm
vmlinux 0xffffffffafe7e2b0
Build id: 3.10.0-1062.12.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffafe7e2b0
Build id: 3.10.0-1062.12.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffafe7e2b0
Build id: 3.10.0-1062.12.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffaff164ef
Build id: 3.10.0-1062.12.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb0181685
Build id: 3.10.0-1062.12.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffaff16a9b
Build id: 3.10.0-1062.12.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffafe7e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffafe7e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffaffec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffaffedb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffaffedc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffafff098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
14
? kmem_cache_free
vmlinux 0xffffffffb0024372
Build id: 3.10.0-1062.12.1.el7.x86_64
0x1e2
15
SyS_madvise
vmlinux 0xffffffffaffeb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x49d
16
system_call_fastpath
vmlinux 0xffffffffb058dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [splunkd:1772]
Modules linked in: sunrpc ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc i2c_piix4 parport vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit ata_piix crct10dif_pclmul crct10dif_common crc32c_intel serio_raw libnvdimm libata vmw_pvscsi vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 1772 Comm: splunkd Kdump: loaded Not tainted 3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9c6c773662a0 ti: ffff9c6bee98c000 task.ti: ffff9c6bee98c000
RIP: 0010:[<ffffffffaff163d6>]  [<ffffffffaff163d6>] generic_exec_single+0x106/0x1c0
RSP: 0018:ffff9c6bee98fbe0  EFLAGS: 00000202
RAX: 00000000000008fd RBX: 0000000000000000 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff9c6bee98fc30 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff9c6b7612b288 R11: 0000000000000202 R12: ffffffffb0452075
R13: ffff9c6bee98fbc0 R14: ffffffffafea4787 R15: ffff9c6bee98fb48
FS:  00007fe245bff700(0000) GS:ffff9c6c7fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe22f42f010 CR3: 0000000036666000 CR4: 00000000001607e0
Call Trace:
[<ffffffffafe7e2b0>] ? leave_mm+0x110/0x110
[<ffffffffafe7e2b0>] ? leave_mm+0x110/0x110
[<ffffffffafe7e2b0>] ? leave_mm+0x110/0x110
[<ffffffffaff164ef>] smp_call_function_single+0x5f/0xa0
[<ffffffffb0181685>] ? cpumask_next_and+0x35/0x50
[<ffffffffaff16a9b>] smp_call_function_many+0x22b/0x270
[<ffffffffafe7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffafe7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaffec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaffedb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaffedc83>] tlb_finish_mmu+0x23/0x30
[<ffffffffafff098a>] zap_page_range+0xea/0x130
[<ffffffffb0024372>] ? kmem_cache_free+0x1e2/0x200
[<ffffffffaffeb69d>] SyS_madvise+0x49d/0xac0
[<ffffffffb058dede>] system_call_fastpath+0x25/0x2a
Code: 14 c5 20 e8 b4 b0 48 89 df e8 37 23 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 f6 43 20 01 <75> f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 00 0f 85 8f 
Complete report #548180
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff95916562
Build id: 3.10.0-1160.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff9587e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9587e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9587e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff9591667f
Build id: 3.10.0-1160.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff95b85de5
Build id: 3.10.0-1160.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff95916c2b
Build id: 3.10.0-1160.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9587e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9587e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff959edd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff959ef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff959ef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff959f2dba
Build id: 3.10.0-1160.el7.x86_64
0xea
14
? __schedule
vmlinux 0xffffffff95f86e80
Build id: 3.10.0-1160.el7.x86_64
0x5a0
15
SyS_madvise
vmlinux 0xffffffff959ed2ad
Build id: 3.10.0-1160.el7.x86_64
0x49d
16
? schedule
vmlinux 0xffffffff95f87169
Build id: 3.10.0-1160.el7.x86_64
0x29
17
system_call_fastpath
vmlinux 0xffffffff95f93f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [sqlservr:13025]
Modules linked in: tcp_lp nls_utf8 isofs rfcomm xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq vmw_balloon crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_seq_device btusb
snd_pcm btrtl btbcm btintel bluetooth pcspkr snd_timer joydev snd sg rfkill soundcore parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix scsi_transport_spi serio_raw e1000 mptscsih libata mptbase nfit drm_panel_orientation_quirks libnvdimm dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 13025 Comm: sqlservr Kdump: loaded Tainted: G             L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9eed7af0c200 ti: ffff9eed99b3c000 task.ti: ffff9eed99b3c000
RIP: 0010:[<ffffffff95916562>]  [<ffffffff95916562>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9eed99b3fbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9eed99b3fbb0 RCX: 0000000000000001
RDX: ffffffff96013790 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9eed99b3fc30 R08: ffffffff96013788 R09: 00003ffffffff000
R10: ffff9eedd2576360 R11: 0000000000000202 R12: ffffffff958dfb48
R13: ffff9eed99b3fb58 R14: ffff9eedf4594268 R15: 0000000000000000
FS:  00007f4c22cf8700(0000) GS:ffff9eee77a40000(0000) knlGS:0000000804950000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4c23ca8ab0 CR3: 000000012da26000 CR4: 00000000003407e0
Call Trace:
[<ffffffff9587e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9587e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9587e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9591667f>] smp_call_function_single+0x5f/0xa0
[<ffffffff95b85de5>] ? cpumask_next_and+0x35/0x50
[<ffffffff95916c2b>] smp_call_function_many+0x22b/0x270
[<ffffffff9587e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9587e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff959edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff959ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff959ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff959f2dba>] zap_page_range+0xea/0x130
[<ffffffff95f86e80>] ? __schedule+0x5a0/0x860
[<ffffffff959ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffff95f87169>] ? schedule+0x29/0x70
[<ffffffff95f93f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 13 55 96 48 89 df e8 f7 69 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #506849
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa8516ae2
Build id: 3.10.0-1160.76.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffa847ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffa847ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffa847ddd0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffa8516bff
Build id: 3.10.0-1160.76.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa8786845
Build id: 3.10.0-1160.76.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa85171ab
Build id: 3.10.0-1160.76.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa847e258
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa847e2cb
Build id: 3.10.0-1160.76.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa85ee6a6
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffa85f016f
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffa85f0293
Build id: 3.10.0-1160.76.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffa85f375a
Build id: 3.10.0-1160.76.1.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffffa85edc4d
Build id: 3.10.0-1160.76.1.el7.x86_64
0x49d
15
? pick_next_entity
vmlinux 0xffffffffa84e2469
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa9
16
? set_next_entity
vmlinux 0xffffffffa84e283c
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3c
17
system_call_fastpath
vmlinux 0xffffffffa8b99f92
Build id: 3.10.0-1160.76.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [pbzip2:14467]
Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm dns_resolver file_protector(OE) snapapi26(OE) ipmi_devintf ipmi_msghandler vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev ses aesni_intel enclosure scsi_transport_sas lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c uas usb_storage sr_mod cdrom vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic nfit syscopyarea sysfillrect ata_generic sysimgblt pata_acpi fb_sys_fops ahci libnvdimm ttm libahci drm ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod tun fuse [last unloaded: edd]
CPU: 1 PID: 14467 Comm: pbzip2 Kdump: loaded Tainted: G           OE  ------------   3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8bd8d774c200 ti: ffff8bd242ff0000 task.ti: ffff8bd242ff0000
RIP: 0010:[<ffffffffa8516ae2>]  [<ffffffffa8516ae2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8bd242ff3be0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff8bd242ff3bb0 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff8bd242ff3c30 R08: ffffffffa8c13cc0 R09: 00003ffffffff000
R10: ffff8bd91ddbf5e8 R11: ffffffffffffffff R12: ffff8bd8d774c200
R13: ffff8bd900000000 R14: 0000000000000000 R15: ffffffffa85c8600
FS:  00007faccbebc700(0000) GS:ffff8bd93fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007faca8b14204 CR3: 0000000034720000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa847ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa847ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa847ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa8516bff>] smp_call_function_single+0x5f/0xa0
[<ffffffffa8786845>] ? cpumask_next_and+0x35/0x50
[<ffffffffa85171ab>] smp_call_function_many+0x22b/0x270
[<ffffffffa847e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa847e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa85ee6a6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa85f016f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa85f0293>] tlb_finish_mmu+0x23/0x30
[<ffffffffa85f375a>] zap_page_range+0xea/0x130
[<ffffffffa85edc4d>] SyS_madvise+0x49d/0xac0
[<ffffffffa84e2469>] ? pick_next_entity+0xa9/0x190
[<ffffffffa84e283c>] ? set_next_entity+0x3c/0xe0
[<ffffffffa8b99f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 00 19 15 a9 48 89 df e8 d7 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #444185
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb5916b62
Build id: 3.10.0-1160.66.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffb587ddd0
Build id: 3.10.0-1160.66.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffb587ddd0
Build id: 3.10.0-1160.66.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffb587ddd0
Build id: 3.10.0-1160.66.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffb5916c7f
Build id: 3.10.0-1160.66.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb5b86835
Build id: 3.10.0-1160.66.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb591722b
Build id: 3.10.0-1160.66.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb587e258
Build id: 3.10.0-1160.66.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb587e2cb
Build id: 3.10.0-1160.66.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb59ee676
Build id: 3.10.0-1160.66.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb59f013f
Build id: 3.10.0-1160.66.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb59f0263
Build id: 3.10.0-1160.66.1.el7.x86_64
0x23
13
unmap_region
vmlinux 0xffffffffb59f9a54
Build id: 3.10.0-1160.66.1.el7.x86_64
0xf4
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kesl:20520]
Modules linked in: nfnetlink_queue nfnetlink_log bluetooth rfkill binfmt_misc nf_conntrack_tftp nf_conntrack_ftp nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw mptspi scsi_transport_spi ata_piix mptscsih libata mptbase nfit vmxnet3 libnvdimm drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 20520 Comm: kesl Kdump: loaded Tainted: G             L ------------   3.10.0-1160.66.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9f87486fc200 ti: ffff9f875e9b4000 task.ti: ffff9f875e9b4000
RIP: 0010:[<ffffffffb5916b62>]  [<ffffffffb5916b62>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9f875e9b7c20  EFLAGS: 00000202
RAX: 00000000000008fd RBX: ffff9f875e9b7bf8 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000282
RBP: ffff9f875e9b7c70 R08: 0000000000000001 R09: ffffffffb5a02f88
R10: ffff9f88ffd1f0a0 R11: fffff76b840dbf00 R12: ffff9f883cba3420
R13: 0000000000000001 R14: ffff9f883fd52c00 R15: fffff76b87346840
FS:  00007f10b95fa700(0000) GS:ffff9f88ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f108188a008 CR3: 000000005b246000 CR4: 00000000000407e0
Call Trace:
[<ffffffffb587ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb587ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb587ddd0>] ? leave_mm+0x120/0x120
[<ffffffffb5916c7f>] smp_call_function_single+0x5f/0xa0
[<ffffffffb5b86835>] ? cpumask_next_and+0x35/0x50
[<ffffffffb591722b>] smp_call_function_many+0x22b/0x270
[<ffffffffb587e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb587e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb59ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb59f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb59f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffb59f9a54>] unmap_region+0xf4/0x140
Complete report #424034
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9ad16a32
Build id: 3.10.0-1160.45.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff9ac7ddd0
Build id: 3.10.0-1160.45.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9ac7ddd0
Build id: 3.10.0-1160.45.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9ac7ddd0
Build id: 3.10.0-1160.45.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff9ad16b4f
Build id: 3.10.0-1160.45.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9af86a45
Build id: 3.10.0-1160.45.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9ad170fb
Build id: 3.10.0-1160.45.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9ac7e258
Build id: 3.10.0-1160.45.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9ac7e2cb
Build id: 3.10.0-1160.45.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9adee676
Build id: 3.10.0-1160.45.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff9adf013f
Build id: 3.10.0-1160.45.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff9adf0263
Build id: 3.10.0-1160.45.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff9adf372a
Build id: 3.10.0-1160.45.1.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffff9adedc1d
Build id: 3.10.0-1160.45.1.el7.x86_64
0x49d
15
? put_prev_entity
vmlinux 0xffffffff9ace637b
Build id: 3.10.0-1160.45.1.el7.x86_64
0x8b
16
? __switch_to
vmlinux 0xffffffff9ac2b621
Build id: 3.10.0-1160.45.1.el7.x86_64
0x151
17
? __schedule
vmlinux 0xffffffff9b388df0
Build id: 3.10.0-1160.45.1.el7.x86_64
0x320
18
system_call_fastpath
vmlinux 0xffffffff9b395f92
Build id: 3.10.0-1160.45.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [firefox:12267]
Modules linked in: tcp_lp nls_utf8 isofs br_netfilter overlay(T) tun devlink bridge stp llc ip_set vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac snd_ens1371 iosf_mbi snd_rawmidi crc32_pclmul snd_ac97_codec ghash_clmulni_intel ac97_bus snd_seq ppdev snd_seq_device aesni_intel snd_pcm lrw gf128mul glue_helper ablk_helper snd_timer cryptd snd vmw_balloon soundcore joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common nfit crc32c_intel ata_piix libnvdimm libata mptspi scsi_transport_spi serio_raw e1000 mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash
dm_log dm_mod fuse [last unloaded: ip_tables]
CPU: 2 PID: 12267 Comm: firefox Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9c55d19d8000 ti: ffff9c55a456c000 task.ti: ffff9c55a456c000
RIP: 0010:[<ffffffff9ad16a32>]  [<ffffffff9ad16a32>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9c55a456fbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9c55a456fbb0 RCX: 0000000000000001
RDX: ffffffff9b4137d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9c55a456fc30 R08: ffffffff9b4137c8 R09: 00003ffffffff000
R10: ffff9c559dcba438 R11: 0000000000000206 R12: ffff9c55d19d8000
R13: 0000000300000000 R14: 0000000000000001 R15: ffffffff9adc862f
FS:  00007f96dbf40780(0000) GS:ffff9c562d680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f967e085000 CR3: 000000039dc54000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9ac7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9ac7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9ac7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9ad16b4f>] smp_call_function_single+0x5f/0xa0
[<ffffffff9af86a45>] ? cpumask_next_and+0x35/0x50
[<ffffffff9ad170fb>] smp_call_function_many+0x22b/0x270
[<ffffffff9ac7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9ac7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9adee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9adf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9adf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff9adf372a>] zap_page_range+0xea/0x130
[<ffffffff9adedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff9ace637b>] ? put_prev_entity+0x8b/0x400
[<ffffffff9ac2b621>] ? __switch_to+0x151/0x580
[<ffffffff9b388df0>] ? __schedule+0x320/0x680
[<ffffffff9b395f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 60 15 95 9b 48 89 df e8 87 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #371316
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff97316a32
Build id: 3.10.0-1160.59.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff9727ddd0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9727ddd0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9727ddd0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff97316b4f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff97586a55
Build id: 3.10.0-1160.59.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff973170fb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9727e258
Build id: 3.10.0-1160.59.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9727e2cb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff973ee676
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff973f013f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff973f0263
Build id: 3.10.0-1160.59.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff973f372a
Build id: 3.10.0-1160.59.1.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffff973edc1d
Build id: 3.10.0-1160.59.1.el7.x86_64
0x49d
15
? pick_next_entity
vmlinux 0xffffffff972e22d9
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa9
16
? set_next_entity
vmlinux 0xffffffff972e26ec
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3c
17
system_call_fastpath
vmlinux 0xffffffff97999f92
Build id: 3.10.0-1160.59.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [containerd-shim:4978]
Modules linked in: xt_multiport ipt_rpfilter xt_set iptable_raw ip_set_hash_ip ip_set_hash_net ip_set ipip tunnel4 ip_tunnel veth xt_nat xt_statistic ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs xt_comment xt_mark nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay(T) bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper sg btusb
btrtl ppdev btbcm ablk_helper btintel pcspkr bluetooth joydev cryptd rfkill vmw_balloon parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper ata_piix nfit syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops ttm crct10dif_common libnvdimm crc32c_intel libata drm mptspi scsi_transport_spi serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 4 PID: 4978 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9faed1144200 ti: ffff9faed3f20000 task.ti: ffff9faed3f20000
RIP: 0010:[<ffffffff97316a32>]  [<ffffffff97316a32>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9faed3f23be0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9faed3f23bb0 RCX: 0000000000000001
RDX: ffffffff97a137d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9faed3f23c30 R08: ffffffff97a137c8 R09: 00003ffffffff000
R10: ffff9fb0282765e8 R11: 0000000000000202 R12: 0000000000000006
R13: ffff9fb07579acc0 R14: ffff9fafba55a400 R15: ffff9fafba55f000
FS:  000000c00003e490(0000) GS:ffff9fb075700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa8272ab000 CR3: 0000000233722000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9727ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9727ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9727ddd0>] ? leave_mm+0x120/0x120
[<ffffffff97316b4f>] smp_call_function_single+0x5f/0xa0
[<ffffffff97586a55>] ? cpumask_next_and+0x35/0x50
[<ffffffff973170fb>] smp_call_function_many+0x22b/0x270
[<ffffffff9727e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9727e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff973ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff973f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff973f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff973f372a>] zap_page_range+0xea/0x130
[<ffffffff973edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff972e22d9>] ? pick_next_entity+0xa9/0x190
[<ffffffff972e26ec>] ? set_next_entity+0x3c/0xe0
[<ffffffff97999f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 60 18 f5 97 48 89 df e8 97 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #361764
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8ab16a32
Build id: 3.10.0-1160.49.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff8aa7ddd0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff8aa7ddd0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff8aa7ddd0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff8ab16b4f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff8ad86a45
Build id: 3.10.0-1160.49.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff8ab170fb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8aa7e258
Build id: 3.10.0-1160.49.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8aa7e2cb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8abee676
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff8abf013f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff8abf0263
Build id: 3.10.0-1160.49.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff8abf372a
Build id: 3.10.0-1160.49.1.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffff8abedc1d
Build id: 3.10.0-1160.49.1.el7.x86_64
0x49d
15
system_call_fastpath
vmlinux 0xffffffff8b195f92
Build id: 3.10.0-1160.49.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [qtp1131184547-5:1522]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13005(E) binfmt_misc xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables falcon_lsm_pinned_12905(E) devlink ip6table_filter ip6_tables iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_CHECKSUM iptable_mangle ext4 mbcache jbd2 snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core iosf_mbi snd_hwdep crc32_pclmul ghash_clmulni_intel snd_seq snd_seq_device snd_pcm iTCO_wdt iTCO_vendor_support ppdev joydev aesni_intel snd_timer lrw gf128mul glue_helper snd ablk_helper cryptd lpc_ich soundcore virtio_rng i2c_i801 parport_pc pcspkr virtio_balloon parport nfsd auth_rpcgss
nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c qxl drm_kms_helper ahci virtio_blk virtio_net virtio_console syscopyarea sysfillrect sysimgblt libahci fb_sys_fops net_failover ttm failover drm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal]
CPU: 2 PID: 1522 Comm: qtp1131184547-5 Kdump: loaded Tainted: P            EL ------------   3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.13.0-2.module+el8.3.0+7860+a7792d29 04/01/2014
task: ffff8e3cb2dc2100 ti: ffff8e3c8caf4000 task.ti: ffff8e3c8caf4000
RIP: 0010:[<ffffffff8ab16a32>]  [<ffffffff8ab16a32>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8e3c8caf7be0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff8e3c8caf7bb0 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff8e3c8caf7c30 R08: ffffffff8b213a40 R09: 00003ffffffff000
R10: ffff8e3e244940d8 R11: 00007f81c7868000 R12: ffff8e3cb2dc2100
R13: ffffffff8ac267ed R14: ffff8e3c8caf7be8 R15: ffff8e3df817caf8
FS:  00007f81c7965700(0000) GS:ffff8e3ebfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000b9 CR3: 0000000273ba2000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8aa7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff8aa7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff8aa7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff8ab16b4f>] smp_call_function_single+0x5f/0xa0
[<ffffffff8ad86a45>] ? cpumask_next_and+0x35/0x50
[<ffffffff8ab170fb>] smp_call_function_many+0x22b/0x270
[<ffffffff8aa7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8aa7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8abee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8abf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8abf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff8abf372a>] zap_page_range+0xea/0x130
[<ffffffff8abedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff8b195f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 60 15 75 8b 48 89 df e8 87 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #329227
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff81b16a32
Build id: 3.10.0-1160.49.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff81a7ddd0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff81a7ddd0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff81a7ddd0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff81b16b4f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81d86a45
Build id: 3.10.0-1160.49.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff81b170fb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff81a7e258
Build id: 3.10.0-1160.49.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81a7e2cb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff81bee676
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff81bf013f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff81bf0263
Build id: 3.10.0-1160.49.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff81bf372a
Build id: 3.10.0-1160.49.1.el7.x86_64
0xea
14
? handle_mm_fault
vmlinux 0xffffffff81bf677d
Build id: 3.10.0-1160.49.1.el7.x86_64
0xbfd
15
SyS_madvise
vmlinux 0xffffffff81bedc1d
Build id: 3.10.0-1160.49.1.el7.x86_64
0x49d
16
system_call_fastpath
vmlinux 0xffffffff82195f92
Build id: 3.10.0-1160.49.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [slack:54020]
Modules linked in: tcp_lp rfcomm xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc des_generic md4 nls_utf8 cifs ccm dns_resolver overlay(T) vmw_vsock_vmci_transport vsock bnep sunrpc snd_seq_midi snd_seq_midi_event squashfs loop snd_ens1371 snd_rawmidi ppdev snd_ac97_codec iosf_mbi crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq vmw_balloon btusb snd_seq_device btrtl btbcm btintel bluetooth aesni_intel snd_pcm lrw gf128mul glue_helper ablk_helper cryptd snd_timer snd joydev pcspkr rfkill sg soundcore
vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common nfit crc32c_intel libnvdimm ata_piix e1000 libata mptspi serio_raw scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 6 PID: 54020 Comm: slack Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff89398f1a3180 ti: ffff89373dd5c000 task.ti: ffff89373dd5c000
RIP: 0010:[<ffffffff81b16a32>]  [<ffffffff81b16a32>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff89373dd5fbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff89373dd5fbb0 RCX: 0000000000000008
RDX: ffffffff82214950 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff89373dd5fc30 R08: ffffffff82214948 R09: 00003ffffffff000
R10: ffff893f90d7f368 R11: 0000000000000206 R12: ffff89422d9f3380
R13: ffff89422d9f3380 R14: 0000000000000002 R15: ffff894233b9ad40
FS:  00007f99a156ebc0(0000) GS:ffff894233b80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000261200f39000 CR3: 0000000d90020000 CR4: 00000000003607e0
Call Trace:
[<ffffffff81a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff81a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff81a7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff81b16b4f>] smp_call_function_single+0x5f/0xa0
[<ffffffff81d86a45>] ? cpumask_next_and+0x35/0x50
[<ffffffff81b170fb>] smp_call_function_many+0x22b/0x270
[<ffffffff81a7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81a7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff81bee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff81bf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff81bf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff81bf372a>] zap_page_range+0xea/0x130
[<ffffffff81bf677d>] ? handle_mm_fault+0xbfd/0xfb0
[<ffffffff81bedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff82195f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 60 15 75 82 48 89 df e8 87 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #293035
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff955163d6
Build id: 3.10.0-1062.12.1.el7.x86_64
0x106
2
? leave_mm
vmlinux 0xffffffff9547e2b0
Build id: 3.10.0-1062.12.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff9547e2b0
Build id: 3.10.0-1062.12.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff9547e2b0
Build id: 3.10.0-1062.12.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff955164ef
Build id: 3.10.0-1062.12.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff95781685
Build id: 3.10.0-1062.12.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff95516a9b
Build id: 3.10.0-1062.12.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9547e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9547e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff955ec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff955edb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff955edc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff955f098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [java:18435]
Modules linked in: ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter sunrpc snd_hda_codec_generic snd_hda_intel ppdev snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd sg soundcore i2c_piix4 virtio_balloon parport_pc joydev parport pcspkr ip_tables xfs libcrc32c dm_crypt drbg ansi_cprng xts gf128mul sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix 8139too virtio_pci libata virtio_ring serio_raw 8139cp virtio mii drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 18435 Comm: java Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff9fd4729b41c0 ti: ffff9fd6c163c000 task.ti: ffff9fd6c163c000
RIP: 0010:[<ffffffff955163d6>]  [<ffffffff955163d6>] generic_exec_single+0x106/0x1c0
RSP: 0018:ffff9fd6c163fbe0  EFLAGS: 00000202
RAX: 00000000000008fd RBX: ffff9fd8aaa24000 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff9fd6c163fc30 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff9fd543fd1bd0 R11: 00007ff1a2665000 R12: ffffffff956b753a
R13: ffff9fd6c163fb90 R14: 0000000000001000 R15: ffffffff955c9512
FS:  00007ff1a2762700(0000) GS:ffff9fd8c1c80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff24b5bd530 CR3: 00000000b2b92000 CR4: 00000000000006e0
Call Trace:
[<ffffffff9547e2b0>] ? leave_mm+0x110/0x110
[<ffffffff9547e2b0>] ? leave_mm+0x110/0x110
[<ffffffff9547e2b0>] ? leave_mm+0x110/0x110
[<ffffffff955164ef>] smp_call_function_single+0x5f/0xa0
[<ffffffff95781685>] ? cpumask_next_and+0x35/0x50
[<ffffffff95516a9b>] smp_call_function_many+0x22b/0x270
[<ffffffff9547e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9547e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff955ec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff955edb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff955edc83>] tlb_finish_mmu+0x23/0x30
[<ffffffff955f098a>] zap_page_range+0xea/0x130
Complete report #210812
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa9516532
Build id: 3.10.0-1160.31.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffa947ddd0
Build id: 3.10.0-1160.31.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffa947ddd0
Build id: 3.10.0-1160.31.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffa947ddd0
Build id: 3.10.0-1160.31.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffa951664f
Build id: 3.10.0-1160.31.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa97864f5
Build id: 3.10.0-1160.31.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa9516bfb
Build id: 3.10.0-1160.31.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa947e258
Build id: 3.10.0-1160.31.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa947e2cb
Build id: 3.10.0-1160.31.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa95ee176
Build id: 3.10.0-1160.31.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffa95efc3f
Build id: 3.10.0-1160.31.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffa95efd63
Build id: 3.10.0-1160.31.1.el7.x86_64
0x23
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [qemu-kvm:3190]
Modules linked in: tcp_lp vhost_net vhost macvtap macvlan nls_utf8 isofs rfcomm xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter vmw_vsock_vmci_transport vsock bnep sunrpc snd_seq_midi snd_seq_midi_event vfat fat xfs iosf_mbi kvm_intel kvm irqbypass libcrc32c crc32_pclmul snd_ens1371 snd_rawmidi snd_ac97_codec ghash_clmulni_intel ac97_bus btusb snd_seq btrtl snd_seq_device btbcm btintel ppdev snd_pcm bluetooth aesni_intel lrw gf128mul glue_helper ablk_helper snd_timer vmw_balloon cryptd snd soundcore rfkill joydev sg pcspkr vmw_vmci i2c_piix4 tpm_crb parport_pc parport
ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect ata_piix sysimgblt fb_sys_fops ttm libata drm nfit libnvdimm crct10dif_pclmul crct10dif_common mptspi scsi_transport_spi crc32c_intel mptscsih mptbase e1000 serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 3190 Comm: qemu-kvm Kdump: loaded Not tainted 3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.16722896.B64.2008100651 08/10/2020
task: ffff898ae0172100 ti: ffff898ad69c0000 task.ti: ffff898ad69c0000
RIP: 0010:[<ffffffffa9516532>]  [<ffffffffa9516532>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff898ad69c3c20  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff898ad69c3bf0 RCX: 0000000000000003
RDX: ffffffffa9c13cd0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff898ad69c3c70 R08: ffffffffa9c13cc8 R09: 0000000180400023
R10: 00000000d6879301 R11: ffff898ad68791c0 R12: ffff898aed034e20
R13: ffffffffa94e4a1c R14: ffff898ad69c3bd0 R15: 0000000000000001
FS:  00007f095bdef700(0000) GS:ffff898af5640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f243f259000 CR3: 00000002108de000 CR4: 00000000003627e0
Call Trace:
[<ffffffffa947ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa947ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa947ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa951664f>] smp_call_function_single+0x5f/0xa0
[<ffffffffa97864f5>] ? cpumask_next_and+0x35/0x50
[<ffffffffa9516bfb>] smp_call_function_many+0x22b/0x270
[<ffffffffa947e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa947e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa95ee176>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa95efc3f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa95efd63>] tlb_finish_mmu+0x23/0x30
Complete report #148503
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff87516842
Build id: 3.10.0-1160.25.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff8747e6a0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff8747e6a0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff8747e6a0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff8751695f
Build id: 3.10.0-1160.25.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff87786395
Build id: 3.10.0-1160.25.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff87516f0b
Build id: 3.10.0-1160.25.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8747e868
Build id: 3.10.0-1160.25.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8747e8db
Build id: 3.10.0-1160.25.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff875edf56
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff875efa1f
Build id: 3.10.0-1160.25.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff875efb43
Build id: 3.10.0-1160.25.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff875f300a
Build id: 3.10.0-1160.25.1.el7.x86_64
0xea
14
? tlb_finish_mmu
vmlinux 0xffffffff875efb40
Build id: 3.10.0-1160.25.1.el7.x86_64
0x20
15
SyS_madvise
vmlinux 0xffffffff875ed4fd
Build id: 3.10.0-1160.25.1.el7.x86_64
0x49d
16
system_call_fastpath
vmlinux 0xffffffff87b95f92
Build id: 3.10.0-1160.25.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [chrome:75884]
Modules linked in: tcp_lp rfcomm xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter binfmt_misc des_generic md4 nls_utf8 cifs ccm dns_resolver overlay(T) bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul vmw_balloon snd_ens1371 ghash_clmulni_intel snd_rawmidi snd_ac97_codec ac97_bus btusb snd_seq aesni_intel btrtl btbcm btintel lrw gf128mul glue_helper ablk_helper cryptd bluetooth snd_seq_device snd_pcm joydev pcspkr snd_timer rfkill snd sg soundcore
vmw_vmci i2c_piix4 parport_pc parport squashfs loop ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect nfit sysimgblt fb_sys_fops ttm libnvdimm crct10dif_pclmul ata_piix crct10dif_common crc32c_intel drm mptspi scsi_transport_spi libata e1000 serio_raw mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 4 PID: 75884 Comm: chrome Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff904a4755d280 ti: ffff90480ea0c000 task.ti: ffff90480ea0c000
RIP: 0010:[<ffffffff87516842>]  [<ffffffff87516842>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff90480ea0fbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff90480ea0fbb0 RCX: 0000000000000004
RDX: ffffffff87c13f50 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff90480ea0fc30 R08: ffffffff87c13f48 R09: 00003ffffffff000
R10: ffff9049844bde60 R11: 0000000000000206 R12: 6666904a478f61d0
R13: 00000000cf1522c3 R14: ffffffff87790f10 R15: ffff90480ea0fb31
FS:  00007f91766b2b40(0000) GS:ffff9051b3b00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcb55077000 CR3: 000000044ea68000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8747e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8747e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8747e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8751695f>] smp_call_function_single+0x5f/0xa0
[<ffffffff87786395>] ? cpumask_next_and+0x35/0x50
[<ffffffff87516f0b>] smp_call_function_many+0x22b/0x270
[<ffffffff8747e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8747e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff875edf56>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff875efa1f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff875efb43>] tlb_finish_mmu+0x23/0x30
[<ffffffff875f300a>] zap_page_range+0xea/0x130
[<ffffffff875efb40>] ? tlb_finish_mmu+0x20/0x30
[<ffffffff875ed4fd>] SyS_madvise+0x49d/0xac0
[<ffffffff87b95f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 a0 15 15 88 48 89 df e8 c7 6c 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #127566
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff97116732
Build id: 3.10.0-1160.21.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff9707e6a0
Build id: 3.10.0-1160.21.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9707e6a0
Build id: 3.10.0-1160.21.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9707e6a0
Build id: 3.10.0-1160.21.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff9711684f
Build id: 3.10.0-1160.21.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff97386265
Build id: 3.10.0-1160.21.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff97116dfb
Build id: 3.10.0-1160.21.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9707e868
Build id: 3.10.0-1160.21.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9707e8db
Build id: 3.10.0-1160.21.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff971ede26
Build id: 3.10.0-1160.21.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff971ef8ef
Build id: 3.10.0-1160.21.1.el7.x86_64
0x3f
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [edr_agent:4069]
Modules linked in: binfmt_misc tcp_diag inet_diag nf_conntrack_ftp tq_base(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc nfit libnvdimm ppdev coretemp vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm ata_piix libata crct10dif_pclmul vmxnet3
crct10dif_common crc32c_intel serio_raw vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 4069 Comm: edr_agent Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8ec60b7a8000 ti: ffff8ec36c790000 task.ti: ffff8ec36c790000
RIP: 0010:[<ffffffff97116732>]  [<ffffffff97116732>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8ec36c793c20  EFLAGS: 00000202
RAX: 0000000000000012 RBX: ffff8ec36c793bf0 RCX: 000000000000002e
RDX: 000000000003ffff RSI: 0000000000000012 RDI: 0000000000000282
RBP: ffff8ec36c793c70 R08: ffffffff97814440 R09: ffffffff97202738
R10: ffff8ec2e50df0a0 R11: ffffd2bb0db9f680 R12: ffffffff9778c778
R13: ffff8ec36c793c70 R14: 0000000000001000 R15: 0000000000000044
FS:  00007f53c6363700(0000) GS:ffff8ec2e50c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f53ca0f1000 CR3: 000000035ff8e000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9707e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9707e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9707e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9711684f>] smp_call_function_single+0x5f/0xa0
[<ffffffff97386265>] ? cpumask_next_and+0x35/0x50
[<ffffffff97116dfb>] smp_call_function_many+0x22b/0x270
[<ffffffff9707e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9707e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff971ede26>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff971ef8ef>] arch_tlb_finish_mmu+0x3f/0x80
Complete report #111349
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8bf14a22
Build id: 3.10.0-1062.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff8be7d1d0
Build id: 3.10.0-1062.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff8be7d1d0
Build id: 3.10.0-1062.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff8be7d1d0
Build id: 3.10.0-1062.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff8bf14b3f
Build id: 3.10.0-1062.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff8c17f475
Build id: 3.10.0-1062.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff8bf150eb
Build id: 3.10.0-1062.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8be7d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8be7d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8bfea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff8bfebc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff8bfebd53
Build id: 3.10.0-1062.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff8bfeea5a
Build id: 3.10.0-1062.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffff8bfe976d
Build id: 3.10.0-1062.el7.x86_64
0x49d
15
tracesys
vmlinux 0xffffffff8c58c172
Build id: 3.10.0-1062.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [Web Content:6189]
Modules linked in: binfmt_misc uas usb_storage tcp_lp fuse tun bridge stp llc devlink vmhgfs(OE) ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq joydev pcspkr snd_seq_device snd_pcm sg snd_timer nfit snd libnvdimm soundcore i2c_piix4 vmw_vmci parport_pc parport xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm serio_raw mptspi ata_piix scsi_transport_spi mptscsih libata e1000 mptbase drm_panel_orientation_quirks dm_mirror
dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 1 PID: 6189 Comm: Web Content Kdump: loaded Tainted: G           OE  ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff8ecc114ed230 ti: ffff8ecc54970000 task.ti: ffff8ecc54970000
RIP: 0010:[<ffffffff8bf14a22>]  [<ffffffff8bf14a22>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8ecc54973be0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8ecc54973bb0 RCX: 0000000000000001
RDX: ffffffff8c613510 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8ecc54973c30 R08: ffffffff8c613508 R09: 00003ffffffff000
R10: ffff8ecdca75b290 R11: 0000000000000206 R12: 0000000000000002
R13: ffff8ece3ffdb008 R14: 0000000000000000 R15: 0000000300000000
FS:  00007f508db2c780(0000) GS:ffff8ece35640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5029341000 CR3: 0000000217308000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8be7d1d0>] ? leave_mm+0x110/0x110
[<ffffffff8be7d1d0>] ? leave_mm+0x110/0x110
[<ffffffff8be7d1d0>] ? leave_mm+0x110/0x110
[<ffffffff8bf14b3f>] smp_call_function_single+0x5f/0xa0
[<ffffffff8c17f475>] ? cpumask_next_and+0x35/0x50
[<ffffffff8bf150eb>] smp_call_function_many+0x22b/0x270
[<ffffffff8be7d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8be7d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8bfea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8bfebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8bfebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffff8bfeea5a>] zap_page_range+0xea/0x130
[<ffffffff8bfe976d>] SyS_madvise+0x49d/0xac0
[<ffffffff8c58c172>] tracesys+0xa6/0xcc
Code: 89 de 48 03 14 c5 e0 d7 b4 8c 48 89 df e8 d7 1a 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #68830
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8c5165b2
Build id: 3.10.0-1160.6.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff8c47e6a0
Build id: 3.10.0-1160.6.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff8c47e6a0
Build id: 3.10.0-1160.6.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff8c47e6a0
Build id: 3.10.0-1160.6.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff8c5166cf
Build id: 3.10.0-1160.6.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff8c785e25
Build id: 3.10.0-1160.6.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff8c516c7b
Build id: 3.10.0-1160.6.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8c47e868
Build id: 3.10.0-1160.6.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8c47e8db
Build id: 3.10.0-1160.6.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8c5edd46
Build id: 3.10.0-1160.6.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff8c5ef80f
Build id: 3.10.0-1160.6.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff8c5ef933
Build id: 3.10.0-1160.6.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff8c5f2dfa
Build id: 3.10.0-1160.6.1.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffff8c5ed2ed
Build id: 3.10.0-1160.6.1.el7.x86_64
0x49d
15
system_call_fastpath
vmlinux 0xffffffff8cb94f92
Build id: 3.10.0-1160.6.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [in:imjournal:1305]
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [mysqld:124640]
Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm dns_resolver bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper ata_piix syscopyarea ahci sysfillrect sysimgblt fb_sys_fops ttm libahci crct10dif_pclmul crct10dif_common crc32c_intel drm libata nfit libnvdimm serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: CyProtectDrv]
CPU: 6 PID: 124640 Comm: mysqld Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8e7cf2389080 ti: ffff8e801503c000 task.ti: ffff8e801503c000
RIP: 0010:[<ffffffff8c5165b2>]  [<ffffffff8c5165b2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8e801503fbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8e801503fbb0 RCX: 0000000000000003
RDX: ffffffff8cc13cd0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8e801503fc30 R08: ffffffff8cc13cc8 R09: 00003ffffffff000
R10: ffff8e82f7ea8bd0 R11: 00007f3837513000 R12: ffff8e801503fcd0
R13: 0000000000000004 R14: ffff8e7cd07948d0 R15: ffffffff8ca3dc64
FS:  00007f383755b700(0000) GS:ffff8e839d780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f38374bfc98 CR3: 00000003b2d70000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8c47e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8c47e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8c47e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8c5166cf>] smp_call_function_single+0x5f/0xa0
[<ffffffff8c785e25>] ? cpumask_next_and+0x35/0x50
[<ffffffff8c516c7b>] smp_call_function_many+0x22b/0x270
[<ffffffff8c47e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8c47e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8c5edd46>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8c5ef80f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8c5ef933>] tlb_finish_mmu+0x23/0x30
[<ffffffff8c5f2dfa>] zap_page_range+0xea/0x130
[<ffffffff8c5ed2ed>] SyS_madvise+0x49d/0xac0
[<ffffffff8cb94f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 13 15 8d 48 89 df e8 e7 69 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #23716
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb0f14a22
Build id: 3.10.0-1062.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffb0e7d1d0
Build id: 3.10.0-1062.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffb0e7d1d0
Build id: 3.10.0-1062.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffb0e7d1d0
Build id: 3.10.0-1062.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffb0f14b3f
Build id: 3.10.0-1062.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb117f475
Build id: 3.10.0-1062.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb0f150eb
Build id: 3.10.0-1062.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb0e7d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb0e7d40b
Build id: 3.10.0-1062.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb0fea166
Build id: 3.10.0-1062.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb0febc2f
Build id: 3.10.0-1062.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb0febd53
Build id: 3.10.0-1062.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffb0feea5a
Build id: 3.10.0-1062.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffffb0fe976d
Build id: 3.10.0-1062.el7.x86_64
0x49d
15
system_call_fastpath
vmlinux 0xffffffffb158bede
Build id: 3.10.0-1062.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [pgyvpn_svr:2148]
Modules linked in: tcp_lp xt_nat veth xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter fuse vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon snd_ens1371 aesni_intel snd_rawmidi lrw gf128mul snd_ac97_codec glue_helper ablk_helper cryptd ac97_bus snd_seq snd_seq_device snd_pcm nfit pcspkr joydev snd_timer libnvdimm snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic
pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel drm libata e1000 serio_raw mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 8 PID: 2148 Comm: pgyvpn_svr Kdump: loaded Tainted: G               ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff97dd70a7c1c0 ti: ffff97dbf428c000 task.ti: ffff97dbf428c000
RIP: 0010:[<ffffffffb0f14a22>]  [<ffffffffb0f14a22>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff97dbf428fbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: 0000000000000286 RCX: 0000000000000007
RDX: ffff97dcbc57d010 RSI: 0000000000000000 RDI: 0000000000000286
RBP: ffff97dbf428fc30 R08: ffff97dcbc57d008 R09: 0000000000000002
R10: 0000000000000040 R11: 0000000000000000 R12: ffff97dbf428fbb0
R13: ffff97dcbc57d000 R14: 0000000000000040 R15: ffffffffb117f475
FS:  00007f24bd431700(0000) GS:ffff97dd75800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f66223cc000 CR3: 00000000b2e96000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb0e7d1d0>] ? leave_mm+0x110/0x110
[<ffffffffb0e7d1d0>] ? leave_mm+0x110/0x110
[<ffffffffb0e7d1d0>] ? leave_mm+0x110/0x110
[<ffffffffb0f14b3f>] smp_call_function_single+0x5f/0xa0
[<ffffffffb117f475>] ? cpumask_next_and+0x35/0x50
[<ffffffffb0f150eb>] smp_call_function_many+0x22b/0x270
[<ffffffffb0e7d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb0e7d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb0fea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb0febc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb0febd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffb0feea5a>] zap_page_range+0xea/0x130
[<ffffffffb0fe976d>] SyS_madvise+0x49d/0xac0
[<ffffffffb158bede>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 d7 b4 b1 48 89 df e8 d7 1a 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #11092
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff97f14a22
Build id: 3.10.0-1062.1.2.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff97e7d1d0
Build id: 3.10.0-1062.1.2.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff97e7d1d0
Build id: 3.10.0-1062.1.2.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff97e7d1d0
Build id: 3.10.0-1062.1.2.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff97f14b3f
Build id: 3.10.0-1062.1.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9817f475
Build id: 3.10.0-1062.1.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff97f150eb
Build id: 3.10.0-1062.1.2.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff97e7d398
Build id: 3.10.0-1062.1.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff97e7d40b
Build id: 3.10.0-1062.1.2.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff97fea166
Build id: 3.10.0-1062.1.2.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff97febc2f
Build id: 3.10.0-1062.1.2.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff97febd53
Build id: 3.10.0-1062.1.2.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff97feea5a
Build id: 3.10.0-1062.1.2.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffff97fe976d
Build id: 3.10.0-1062.1.2.el7.x86_64
0x49d
15
tracesys
vmlinux 0xffffffff9858d172
Build id: 3.10.0-1062.1.2.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [elasticsearch[n:95159]
Modules linked in: tcp_lp nls_utf8 isofs fuse veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink binfmt_misc vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device coretemp snd_pcm snd_timer snd ppdev soundcore vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev pcspkr parport_pc parport vmw_vmci i2c_piix4 sch_fq_codel ip_tables xfs libcrc32c
sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx ata_piix drm_kms_helper syscopyarea libata sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm mptspi nfit crc32c_intel scsi_transport_spi libnvdimm mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]
CPU: 1 PID: 95159 Comm: elasticsearch[n Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9a3ac526d230 ti: ffff9a3ad3edc000 task.ti: ffff9a3ad3edc000
RIP: 0010:[<ffffffff97f14a22>]  [<ffffffff97f14a22>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9a3ad3edfbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9a3ad3edfbb0 RCX: 0000000000000001
RDX: ffffffff98613550 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9a3ad3edfc30 R08: ffffffff98613548 R09: 00003ffffffff000
R10: ffff9a3c81412360 R11: 00007fad0b8ef000 R12: ffff9a3cfffd8000
R13: ffffffff97fc68f6 R14: ffff9a3ad3edfbf0 R15: 0000000000020200
FS:  00007fad0b9ec700(0000) GS:ffff9a3cf5640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fadb4804530 CR3: 00000001e3790000 CR4: 00000000001607e0
Call Trace:
[<ffffffff97e7d1d0>] ? leave_mm+0x110/0x110
[<ffffffff97e7d1d0>] ? leave_mm+0x110/0x110
[<ffffffff97e7d1d0>] ? leave_mm+0x110/0x110
[<ffffffff97f14b3f>] smp_call_function_single+0x5f/0xa0
[<ffffffff9817f475>] ? cpumask_next_and+0x35/0x50
[<ffffffff97f150eb>] smp_call_function_many+0x22b/0x270
[<ffffffff97e7d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff97e7d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff97fea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff97febc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff97febd53>] tlb_finish_mmu+0x23/0x30
[<ffffffff97feea5a>] zap_page_range+0xea/0x130
[<ffffffff97fe976d>] SyS_madvise+0x49d/0xac0
[<ffffffff9858d172>] tracesys+0xa6/0xcc
Code: 89 de 48 03 14 c5 e0 d7 b4 98 48 89 df e8 d7 1a 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #7298
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb87163d2
Build id: 3.10.0-1062.12.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffb867e2b0
Build id: 3.10.0-1062.12.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffb867e2b0
Build id: 3.10.0-1062.12.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffb867e2b0
Build id: 3.10.0-1062.12.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffb87164ef
Build id: 3.10.0-1062.12.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb8981685
Build id: 3.10.0-1062.12.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb8716a9b
Build id: 3.10.0-1062.12.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb867e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb867e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb87ec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb87edb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb87edc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffb87f098a
Build id: 3.10.0-1062.12.1.el7.x86_64
0xea
14
? do_sync_write
vmlinux 0xffffffffb8849d83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x93
15
SyS_madvise
vmlinux 0xffffffffb87eb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
0x49d
16
system_call_fastpath
vmlinux 0xffffffffb8d8dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [splunkd:7291]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink xt_comment ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd pcspkr vmw_balloon sg joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul crct10dif_common nfit crc32c_intel libnvdimm serio_raw mptspi scsi_transport_spi mptscsih vmxnet3 mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 7291 Comm: splunkd Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
task: ffff967b886a20e0 ti: ffff967b88524000 task.ti: ffff967b88524000
RIP: 0010:[<ffffffffb87163d2>]  [<ffffffffb87163d2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff967b88527be0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff967b88527bb0 RCX: 0000000000000830
RDX: 0000000000000004 RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff967b88527c30 R08: ffffffffb8e13cc0 R09: 00003ffffffff000
R10: ffff967b94fea510 R11: 0000000000000202 R12: ffff967b88527bd0
R13: 0000000000000000 R14: ffff967bb860e6c0 R15: 00000000000007c4
FS:  00007f8eee3fd700(0000) GS:ffff967bbfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8ee8c94040 CR3: 0000000061186000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb867e2b0>] ? leave_mm+0x110/0x110
[<ffffffffb867e2b0>] ? leave_mm+0x110/0x110
[<ffffffffb867e2b0>] ? leave_mm+0x110/0x110
[<ffffffffb87164ef>] smp_call_function_single+0x5f/0xa0
[<ffffffffb8981685>] ? cpumask_next_and+0x35/0x50
[<ffffffffb8716a9b>] smp_call_function_many+0x22b/0x270
[<ffffffffb867e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb867e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb87ec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb87edb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb87edc83>] tlb_finish_mmu+0x23/0x30
[<ffffffffb87f098a>] zap_page_range+0xea/0x130
[<ffffffffb8849d83>] ? do_sync_write+0x93/0xe0
[<ffffffffb87eb69d>] SyS_madvise+0x49d/0xac0
[<ffffffffb8d8dede>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 20 e8 34 b9 48 89 df e8 37 23 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #12923
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff88716f76
Build id: 3.10.0-1127.18.2.el7.x86_64
0x106
2
? leave_mm
vmlinux 0xffffffff8867e730
Build id: 3.10.0-1127.18.2.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff8867e730
Build id: 3.10.0-1127.18.2.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff8867e730
Build id: 3.10.0-1127.18.2.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff8871708f
Build id: 3.10.0-1127.18.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff88984fc5
Build id: 3.10.0-1127.18.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff8871763b
Build id: 3.10.0-1127.18.2.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8867e8f8
Build id: 3.10.0-1127.18.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8867e96b
Build id: 3.10.0-1127.18.2.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff887ede36
Build id: 3.10.0-1127.18.2.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff887ef8ff
Build id: 3.10.0-1127.18.2.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff887efa23
Build id: 3.10.0-1127.18.2.el7.x86_64
0x23
13
unmap_region
vmlinux 0xffffffff887f9234
Build id: 3.10.0-1127.18.2.el7.x86_64
0xf4
14
? xfs_iunlock
xfs 0xffffffffc042e344
Build id: 3.10.0-1127.18.2.el7.x86_64
0xe4
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ManagementAgent:1272]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock fuse sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer sg snd coretemp ppdev soundcore vmw_balloon vmw_vmci joydev pcspkr i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel nfit ata_piix mptspi e1000 libnvdimm scsi_transport_spi
mptscsih libata serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 1272 Comm: ManagementAgent Kdump: loaded Tainted: G             L ------------   3.10.0-1127.18.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9776b5dec1c0 ti: ffff977aacdbc000 task.ti: ffff977aacdbc000
RIP: 0010:[<ffffffff88716f76>]  [<ffffffff88716f76>] generic_exec_single+0x106/0x1c0
RSP: 0018:ffff977aacdbfc20  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff977aacdbfbf0 RCX: 0000000000000001
RDX: ffffffff88e136d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff977aacdbfc70 R08: ffffffff88e136c8 R09: ffffffff8880280c
R10: ffff977aad69f0a0 R11: ffffde4a4cf044c0 R12: ffffffff88d8d975
R13: ffff977aacdbfbb8 R14: 00007fb7587a5000 R15: ffff977aa391b000
FS:  00007fb74ffff700(0000) GS:ffff977aad680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb7587a5000 CR3: 000000042cad4000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8867e730>] ? leave_mm+0x120/0x120
[<ffffffff8867e730>] ? leave_mm+0x120/0x120
[<ffffffff8867e730>] ? leave_mm+0x120/0x120
[<ffffffff8871708f>] smp_call_function_single+0x5f/0xa0
[<ffffffff88984fc5>] ? cpumask_next_and+0x35/0x50
[<ffffffff8871763b>] smp_call_function_many+0x22b/0x270
[<ffffffff8867e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8867e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff887ede36>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff887ef8ff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff887efa23>] tlb_finish_mmu+0x23/0x30
[<ffffffff887f9234>] unmap_region+0xf4/0x140
[<ffffffffc042e344>] ? xfs_iunlock+0xe4/0x120 [xfs]