Info

Permalink
Function
generic_exec_single
First occurrence
2020-10-13
Last occurrence
2020-10-13
State
NEW
Components
kernel
Quality
-31

Statistics

Operating system Count
CentOS 7 421
Architecture Count
x86_64 421
Related packages Count
kernel 421
 0:3.10.0-862.el7 72
 0:3.10.0-693.11.1.el7 68
 0:3.10.0-957.el7 57
 0:3.10.0-1160.49.1.el7 38
 0:3.10.0-1160.el7 28
 0:3.10.0-1160.11.1.el7 23
 0:3.10.0-1127.10.1.el7 14
 0:3.10.0-1160.108.1.el7 14
 0:3.10.0-693.el7 12
 0:3.10.0-1127.el7 8
 0:3.10.0-1160.119.1.el7 8
 0:3.10.0-957.12.1.el7 7
 0:3.10.0-1160.105.1.el7 7
 0:3.10.0-1160.53.1.el7 6
 0:3.10.0-1127.19.1.el7 5
 0:3.10.0-1160.24.1.el7 5
 0:3.10.0-862.11.6.el7 5
 0:3.10.0-957.5.1.el7 5
 0:3.10.0-1160.42.2.el7 4
 0:3.10.0-957.27.2.el7 4
 0:3.10.0-1160.92.1.el7 3
 0:3.10.0-1160.36.2.el7 3
 0:3.10.0-862.14.4.el7 2
 0:3.10.0-1160.31.1.el7 2
 0:3.10.0-1160.62.1.el7 2
 0:3.10.0-1160.15.2.el7 2
 0:3.10.0-1160.45.1.el7 2
 0:3.10.0-1160.59.1.el7 2
 0:3.10.0-1160.118.1.el7 2
 0:3.10.0-1062.1.2.el7 1
 0:3.10.0-957.10.1.el7 1
 0:3.10.0-1127.8.2.el7 1
 0:3.10.0-1160.21.1.el7 1
 0:3.10.0-1160.25.1.el7 1
 0:3.10.0-1160.95.1.el7 1
 0:3.10.0-1160.76.1.el7 1
 0:3.10.0-1062.9.1.el7 1
 0:3.10.0-1062.12.1.el7 1
 0:3.10.0-693.21.1.el7 1
 0:3.10.0-1062.1.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1400444
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffad51dba2
Build id: 3.10.0-1160.119.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffad481b10
Build id: 3.10.0-1160.119.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffffad481b10
Build id: 3.10.0-1160.119.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffffad481b10
Build id: 3.10.0-1160.119.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffffad51dcbf
Build id: 3.10.0-1160.119.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffad79c615
Build id: 3.10.0-1160.119.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffad51e27b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffad481fa8
Build id: 3.10.0-1160.119.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffad48201b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffad5fa0ce
Build id: 3.10.0-1160.119.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffffad5fbc4f
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffad5fbd83
Build id: 3.10.0-1160.119.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffad5ff2ca
Build id: 3.10.0-1160.119.1.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffffad4e1118
Build id: 3.10.0-1160.119.1.el7.x86_64
0x58
15
? futex_wake
vmlinux 0xffffffffad5196b0
Build id: 3.10.0-1160.119.1.el7.x86_64
0x90
16
SyS_madvise
vmlinux 0xffffffffad5f96bd
Build id: 3.10.0-1160.119.1.el7.x86_64
0x49d
17
? pick_next_entity
vmlinux 0xffffffffad4e8699
Build id: 3.10.0-1160.119.1.el7.x86_64
0xa9
18
? set_next_entity
vmlinux 0xffffffffad4e918c
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3c
19
system_call_fastpath
vmlinux 0xffffffffadbc539a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [containerd-shim:52254]
Modules linked in: tcp_lp veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device snd_pcm aesni_intel lrw gf128mul snd_timer glue_helper ablk_helper snd soundcore cryptd ppdev vmw_balloon sg pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm libnvdimm ata_piix mptspi crct10dif_pclmul crct10dif_common crc32c_intel libata scsi_transport_spi mptscsih e1000 mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 52254 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffffa01c55504200 ti: ffffa01c278d8000 task.ti: ffffa01c278d8000
RIP: 0010:[<ffffffffad51dba2>]  [<ffffffffad51dba2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffffa01c278dbbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffffa01c278dbbb0 RCX: 0000000000000002
RDX: ffffffffadc13c50 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffffa01c278dbc30 R08: ffffffffadc13c48 R09: 00003ffffffff000
R10: ffffa01c2403c1b0 R11: 0000000000000246 R12: ffffa01c4f1f75c0
R13: ffffa01c4f1f75c0 R14: 00000000000003f5 R15: ffffa01475ed9600
FS:  000000c000080090(0000) GS:ffffa01c5d6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00035b010 CR3: 00000007cf30c000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffffad481b10>] ? leave_mm+0x130/0x130
[<ffffffffad481b10>] ? leave_mm+0x130/0x130
[<ffffffffad481b10>] ? leave_mm+0x130/0x130
[<ffffffffad51dcbf>] smp_call_function_single+0x5f/0xa0
[<ffffffffad79c615>] ? cpumask_next_and+0x35/0x50
[<ffffffffad51e27b>] smp_call_function_many+0x22b/0x270
[<ffffffffad481fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffad48201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffad5fa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffad5fbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffad5fbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffffad5ff2ca>] zap_page_range+0xea/0x140
[<ffffffffad4e1118>] ? wake_up_q+0x58/0x90
[<ffffffffad5196b0>] ? futex_wake+0x90/0x190
[<ffffffffad5f96bd>] SyS_madvise+0x49d/0xac0
[<ffffffffad4e8699>] ? pick_next_entity+0xa9/0x190
[<ffffffffad4e918c>] ? set_next_entity+0x3c/0x100
[<ffffffffadbc539a>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 c0 19 15 ae 48 89 df e8 b7 67 29 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1189521
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa111109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffa107a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffa107a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffa107a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffa11111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa1375465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa111175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa107a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa107a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffa11e4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffa11e5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffa11e87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffffa10d6768
Build id: 3.10.0-957.el7.x86_64
0x58
14
? futex_wake
vmlinux 0xffffffffa110cca0
Build id: 3.10.0-957.el7.x86_64
0x90
15
SyS_madvise
vmlinux 0xffffffffa11e368d
Build id: 3.10.0-957.el7.x86_64
0x49d
16
tracesys
vmlinux 0xffffffffa177505b
Build id: 3.10.0-957.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [ActiveMQ Inacti:126546]
Modules linked in: nf_conntrack_netlink xt_nat veth nfnetlink ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_addrtype overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer pcspkr joydev snd soundcore sg parport_pc parport i2c_piix4 vmw_vmci sch_fq_codel ip_tables xfs libcrc32c
sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx serio_raw drm_kms_helper syscopyarea sysfillrect ata_piix sysimgblt fb_sys_fops ttm e1000 libata drm mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 126546 Comm: ActiveMQ Inacti Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff971c08f6a080 ti: ffff971c07e2c000 task.ti: ffff971c07e2c000
RIP: 0010:[<ffffffffa111109a>]  [<ffffffffa111109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff971c07e2fbf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff971c07e2fbc0 RCX: 0000000000000006
RDX: ffffffffa1813a10 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff971c07e2fc38 R08: ffffffffa1813a08 R09: 00003ffffffff000
R10: ffff971b056aed80 R11: 00007fe87b5ba000 R12: ffff971d320f9ba0
R13: ffffe63488515ac0 R14: 0000000000000000 R15: ffffe63488515ac0
FS:  00007fe87b6b6700(0000) GS:ffff971d35780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe9aa1152b0 CR3: 0000000210e3e000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffa107a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa107a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa107a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa11111af>] smp_call_function_single+0x5f/0xa0
[<ffffffffa1375465>] ? cpumask_next_and+0x35/0x50
[<ffffffffa111175b>] smp_call_function_many+0x22b/0x270
[<ffffffffa107a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa107a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa11e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffa11e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffa11e87c2>] zap_page_range+0x142/0x190
[<ffffffffa10d6768>] ? wake_up_q+0x58/0x80
[<ffffffffa110cca0>] ? futex_wake+0x90/0x180
[<ffffffffa11e368d>] SyS_madvise+0x49d/0xac0
[<ffffffffa177505b>] tracesys+0xa3/0xc9
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 d4 a1 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #987682
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9d30779a
Build id: 3.10.0-862.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff9d276e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff9d276e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff9d276e50
Build id: 3.10.0-862.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff9d3078af
Build id: 3.10.0-862.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9d54a275
Build id: 3.10.0-862.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9d307e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9d277018
Build id: 3.10.0-862.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9d277088
Build id: 3.10.0-862.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffff9d3c0497
Build id: 3.10.0-862.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff9d3c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff9d3c4a02
Build id: 3.10.0-862.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffff9d3bfafd
Build id: 3.10.0-862.el7.x86_64
0x3cd
14
? vm_mmap_pgoff
vmlinux 0xffffffff9d3b35eb
Build id: 3.10.0-862.el7.x86_64
0xeb
15
system_call_fastpath
vmlinux 0xffffffff9d91f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [java:9219]
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [java:508976]
Modules linked in: ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi snd_ens1371 snd_rawmidi snd_ac97_codec crc32_pclmul ghash_clmulni_intel ac97_bus aesni_intel lrw snd_seq gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm ppdev vmw_balloon snd_timer snd soundcore joydev pcspkr vmw_vmci sg i2c_piix4 shpchp nfit libnvdimm parport_pc parport binfmt_misc xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel mptspi libata scsi_transport_spi e1000 mptscsih serio_raw i2c_core mptbase [last unloaded: ip_tables]
CPU: 3 PID: 508976 Comm: java Tainted: G        W    L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8b7d506dcf10 ti: ffff8b7bbd630000 task.ti: ffff8b7bbd630000
RIP: 0010:[<ffffffff9d30779a>]  [<ffffffff9d30779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8b7bbd633bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8b7bbd633bc0 RCX: 000000000000000c
RDX: ffffffff9da14cf0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8b7bbd633c38 R08: ffffffff9da14ce8 R09: ffff8b7bf7f52948
R10: ffffd4020df4f340 R11: 00002b634b605000 R12: ffff8b7d506dcf10
R13: ffff8b7d506dcf10 R14: 00002b634b605000 R15: ffff8b7d506dcf10
FS:  00002b634b702700(0000) GS:ffff8b7fbd6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b6217fb2680 CR3: 0000000ff8f30000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff9d276e50>] ? leave_mm+0x110/0x110
[<ffffffff9d276e50>] ? leave_mm+0x110/0x110
[<ffffffff9d276e50>] ? leave_mm+0x110/0x110
[<ffffffff9d3078af>] smp_call_function_single+0x5f/0xa0
[<ffffffff9d54a275>] ? cpumask_next_and+0x35/0x50
[<ffffffff9d307e5b>] smp_call_function_many+0x22b/0x270
[<ffffffff9d277018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9d277088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff9d3c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff9d3c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff9d3c4a02>] zap_page_range+0x142/0x190
[<ffffffff9d3bfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff9d3b35eb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff9d91f7d5>] system_call_fastpath+0x1c/0x21
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 f3 9d 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #919178
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff94d1109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff94c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff94c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff94c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff94d111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff94f75465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff94d1175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff94c7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff94c7a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff94de4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff94de5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff94de87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffff94cd6768
Build id: 3.10.0-957.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffff94de368d
Build id: 3.10.0-957.el7.x86_64
0x49d
15
? mntput
vmlinux 0xffffffff94e640a4
Build id: 3.10.0-957.el7.x86_64
0x24
16
system_call_fastpath
vmlinux 0xffffffff95374ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 39s! [kubelet:83601]
Modules linked in: xt_nat ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_comment xt_mark ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat nf_conntrack br_netfilter overlay(T) tun devlink ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock bnep sunrpc iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel snd_seq_midi snd_seq_midi_event aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr snd_ens1371 btusb btrtl btbcm btintel snd_rawmidi bluetooth snd_ac97_codec ac97_bus snd_seq rfkill snd_seq_device snd_pcm snd_timer snd soundcore sg parport_pc parport vmw_vmci i2c_piix4 xfs libcrc32c sd_mod crc_t10dif
crct10dif_generic sr_mod cdrom crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic vmwgfx drm_kms_helper syscopyarea pata_acpi sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi ata_piix drm e1000 mptscsih libata drm_panel_orientation_quirks nfit mptbase libnvdimm dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 0 PID: 83601 Comm: kubelet Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff980a0b36b0c0 ti: ffff980a0b388000 task.ti: ffff980a0b388000
RIP: 0010:[<ffffffff94d1109a>]  [<ffffffff94d1109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff980a0b38bbf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff980a0b38bbc0 RCX: 0000000000000002
RDX: ffffffff95413010 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff980a0b38bc38 R08: ffffffff95413008 R09: 00003ffffffff000
R10: ffff980af09ffcb0 R11: 0000000000000246 R12: ffffffff94cdd70b
R13: ffff980a0b38bbd0 R14: 0000000000000046 R15: ffff980a0b38bb58
FS:  00007fdca65e9700(0000) GS:ffff980af9600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f245a1a9000 CR3: 0000000135028000 CR4: 00000000003607f0
Call Trace:
[<ffffffff94c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff94c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff94c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff94d111af>] smp_call_function_single+0x5f/0xa0
[<ffffffff94f75465>] ? cpumask_next_and+0x35/0x50
[<ffffffff94d1175b>] smp_call_function_many+0x22b/0x270
[<ffffffff94c7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff94c7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff94de4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff94de5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff94de87c2>] zap_page_range+0x142/0x190
[<ffffffff94cd6768>] ? wake_up_q+0x58/0x80
[<ffffffff94de368d>] SyS_madvise+0x49d/0xac0
[<ffffffff94e640a4>] ? mntput+0x24/0x40
[<ffffffff95374ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 94 95 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #880574
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffbe50779a
Build id: 3.10.0-862.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffbe476e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffbe476e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffbe476e50
Build id: 3.10.0-862.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffbe5078af
Build id: 3.10.0-862.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffbe74a275
Build id: 3.10.0-862.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffbe507e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffbe477018
Build id: 3.10.0-862.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffbe477088
Build id: 3.10.0-862.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffffbe5c0497
Build id: 3.10.0-862.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffbe5c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffbe5c4a02
Build id: 3.10.0-862.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffffbe4cee38
Build id: 3.10.0-862.el7.x86_64
0x58
14
? futex_wake
vmlinux 0xffffffffbe5033a0
Build id: 3.10.0-862.el7.x86_64
0x90
15
SyS_madvise
vmlinux 0xffffffffbe5bfafd
Build id: 3.10.0-862.el7.x86_64
0x3cd
16
system_call_fastpath
vmlinux 0xffffffffbeb1f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [ipc_listen:61078]
Modules linked in: binfmt_misc fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul snd_ens1371 snd_rawmidi snd_ac97_codec glue_helper ablk_helper cryptd ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore joydev
ppdev pcspkr sg vmw_balloon nfit libnvdimm parport_pc parport i2c_piix4 shpchp vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common mptspi crc32c_intel libata scsi_transport_spi e1000 mptscsih mptbase serio_raw i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 61078 Comm: ipc_listen Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8827d2f3dee0 ti: ffff88291566c000 task.ti: ffff88291566c000
RIP: 0010:[<ffffffffbe50779a>]  [<ffffffffbe50779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff88291566fbf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88291566fbc0 RCX: 0000000000000008
RDX: ffffffffbec142f0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88291566fc38 R08: ffffffffbec142e8 R09: ffff882602bd6ca8
R10: 0000000000000002 R11: 0000000000000246 R12: 000280da00000000
R13: ffffffffffffffff R14: 0000000000000002 R15: 0000000000000030
FS:  00007fe049ffb700(0000) GS:ffff882ad1a80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe018022558 CR3: 00000002ea602000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffbe476e50>] ? leave_mm+0x110/0x110
[<ffffffffbe476e50>] ? leave_mm+0x110/0x110
[<ffffffffbe476e50>] ? leave_mm+0x110/0x110
[<ffffffffbe5078af>] smp_call_function_single+0x5f/0xa0
[<ffffffffbe74a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffbe507e5b>] smp_call_function_many+0x22b/0x270
[<ffffffffbe477018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbe477088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffbe5c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffbe5c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffbe5c4a02>] zap_page_range+0x142/0x190
[<ffffffffbe4cee38>] ? wake_up_q+0x58/0x80
[<ffffffffbe5033a0>] ? futex_wake+0x90/0x180
[<ffffffffbe5bfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffffbeb1f7d5>] system_call_fastpath+0x1c/0x21
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 13 bf 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #861717
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff99f16b82
Build id: 3.10.0-1160.62.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff99e7ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff99e7ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff99e7ddd0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff99f16c9f
Build id: 3.10.0-1160.62.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9a186825
Build id: 3.10.0-1160.62.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff99f1724b
Build id: 3.10.0-1160.62.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff99e7e258
Build id: 3.10.0-1160.62.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff99e7e2cb
Build id: 3.10.0-1160.62.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff99fee666
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff99ff012f
Build id: 3.10.0-1160.62.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff99ff0253
Build id: 3.10.0-1160.62.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff99ff371a
Build id: 3.10.0-1160.62.1.el7.x86_64
0xea
14
? futex_wake
vmlinux 0xffffffff99f127c0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x90
15
SyS_madvise
vmlinux 0xffffffff99fedc0d
Build id: 3.10.0-1160.62.1.el7.x86_64
0x49d
16
? put_prev_entity
vmlinux 0xffffffff99ee64cb
Build id: 3.10.0-1160.62.1.el7.x86_64
0x8b
17
? switch_mm_irqs_off
vmlinux 0xffffffff99e7dfe9
Build id: 3.10.0-1160.62.1.el7.x86_64
0x109
18
? __switch_to
vmlinux 0xffffffff99e2b621
Build id: 3.10.0-1160.62.1.el7.x86_64
0x151
19
system_call_fastpath
vmlinux 0xffffffff9a599f92
Build id: 3.10.0-1160.62.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 21s! [threads_sampler:2483]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi vmw_balloon aesni_intel snd_ac97_codec lrw gf128mul glue_helper ac97_bus ablk_helper snd_seq cryptd snd_seq_device snd_pcm joydev pcspkr sg snd_timer snd soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm libnvdimm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata mptspi scsi_transport_spi e1000 mptscsih serio_raw mptbase drm_panel_orientation_quirks fuse
CPU: 4 PID: 2483 Comm: threads_sampler Kdump: loaded Tainted: G             L ------------   3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9c33aaeb4200 ti: ffff9c32b6834000 task.ti: ffff9c32b6834000
RIP: 0010:[<ffffffff99f16b82>]  [<ffffffff99f16b82>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9c32b6837be0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9c32b6837bb0 RCX: 0000000000000001
RDX: ffffffff9a6137d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9c32b6837c30 R08: ffffffff9a6137c8 R09: 00003ffffffff000
R10: ffff9c32aef22288 R11: 0000000000000206 R12: 00006ff9930a5717
R13: ffffffff99ee497c R14: ffff9c32b6837b80 R15: ffff9c30f59ee000
FS:  00007f5208dff700(0000) GS:ffff9c33ad700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff7db18c860 CR3: 00000000916e8000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff99e7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff99e7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff99e7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff99f16c9f>] smp_call_function_single+0x5f/0xa0
[<ffffffff9a186825>] ? cpumask_next_and+0x35/0x50
[<ffffffff99f1724b>] smp_call_function_many+0x22b/0x270
[<ffffffff99e7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff99e7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff99fee666>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff99ff012f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff99ff0253>] tlb_finish_mmu+0x23/0x30
[<ffffffff99ff371a>] zap_page_range+0xea/0x130
[<ffffffff99f127c0>] ? futex_wake+0x90/0x180
[<ffffffff99fedc0d>] SyS_madvise+0x49d/0xac0
[<ffffffff99ee64cb>] ? put_prev_entity+0x8b/0x400
[<ffffffff99e7dfe9>] ? switch_mm_irqs_off+0x109/0x290
[<ffffffff99e2b621>] ? __switch_to+0x151/0x580
[<ffffffff9a599f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 60 18 b5 9a 48 89 df e8 17 6e 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #762534
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff84b1109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff84a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff84a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff84a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff84b111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff84d75465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff84b1175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff84a7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff84a7a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff84be4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff84be5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff84be87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffff84be368d
Build id: 3.10.0-957.el7.x86_64
0x49d
14
? vm_mmap_pgoff
vmlinux 0xffffffff84bd612b
Build id: 3.10.0-957.el7.x86_64
0xeb
15
system_call_fastpath
vmlinux 0xffffffff85174ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [java:62583]
Modules linked in: tcp_lp rfcomm fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev vmw_balloon kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd btusb btrtl joydev btbcm btintel bluetooth pcspkr snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq rfkill snd_seq_device snd_pcm sg snd_timer snd soundcore i2c_piix4 vmw_vmci parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmwgfx drm_kms_helper ata_piix syscopyarea sysfillrect sysimgblt fb_sys_fops e1000 ttm libata drm mptspi scsi_transport_spi nfit mptscsih mptbase libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 62583 Comm: java Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff88cf772c5140 ti: ffff88cdc2784000 task.ti: ffff88cdc2784000
RIP: 0010:[<ffffffff84b1109a>]  [<ffffffff84b1109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff88cdc2787bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88cdc2787bc0 RCX: 0000000000000007
RDX: ffffffff85213c90 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88cdc2787c38 R08: ffffffff85213c88 R09: 00003ffffffff000
R10: ffff88cdf1e2be60 R11: 00007f6342df3000 R12: ffff88cffffda000
R13: ffffffff84bc0c06 R14: ffff88cdc2787c00 R15: 0000000000020200
FS:  00007f6342ef0700(0000) GS:ffff88cfcd680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc7f770e000 CR3: 0000000a31374000 CR4: 0000000000340fe0
Call Trace:
[<ffffffff84a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff84a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff84a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff84b111af>] smp_call_function_single+0x5f/0xa0
[<ffffffff84d75465>] ? cpumask_next_and+0x35/0x50
[<ffffffff84b1175b>] smp_call_function_many+0x22b/0x270
[<ffffffff84a7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff84a7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff84be4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff84be5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff84be87c2>] zap_page_range+0x142/0x190
[<ffffffff84be368d>] SyS_madvise+0x49d/0xac0
[<ffffffff84bd612b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff85174ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 74 85 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #657051
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffbb7124ca
Build id: 3.10.0-957.27.2.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffbb67b5c0
Build id: 3.10.0-957.27.2.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffbb67b5c0
Build id: 3.10.0-957.27.2.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffbb67b5c0
Build id: 3.10.0-957.27.2.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffbb7125df
Build id: 3.10.0-957.27.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffbb977345
Build id: 3.10.0-957.27.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffbb712b8b
Build id: 3.10.0-957.27.2.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffbb67b788
Build id: 3.10.0-957.27.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffbb67b7f8
Build id: 3.10.0-957.27.2.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffbb7e57d7
Build id: 3.10.0-957.27.2.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffbb7e70d5
Build id: 3.10.0-957.27.2.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffbb7e9f02
Build id: 3.10.0-957.27.2.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffffbb7e4ded
Build id: 3.10.0-957.27.2.el7.x86_64
0x49d
14
? vm_mmap_pgoff
vmlinux 0xffffffffbb7d784b
Build id: 3.10.0-957.27.2.el7.x86_64
0xeb
15
system_call_fastpath
vmlinux 0xffffffffbbd76ddb
Build id: 3.10.0-957.27.2.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [java:30282]
Modules linked in: fuse nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 iosf_mbi snd_rawmidi kvm_intel snd_ac97_codec kvm ac97_bus irqbypass crc32_pclmul snd_seq ghash_clmulni_intel aesni_intel snd_seq_device lrw gf128mul snd_pcm glue_helper snd_timer snd ablk_helper ppdev soundcore cryptd vmw_balloon pcspkr sg joydev parport_pc
parport i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi drm ata_piix scsi_transport_spi crct10dif_pclmul nfit mptscsih crct10dif_common libata crc32c_intel libnvdimm serio_raw e1000 drm_panel_orientation_quirks mptbase
CPU: 2 PID: 30282 Comm: java Kdump: loaded Not tainted 3.10.0-957.27.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff99cca5eb1040 ti: ffff99caaf600000 task.ti: ffff99caaf600000
RIP: 0010:[<ffffffffbb7124ca>]  [<ffffffffbb7124ca>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff99caaf603bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff99caaf603b80 RCX: 0000000000000001
RDX: ffffffffbbe13150 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff99caaf603c38 R08: ffffffffbbe13148 R09: 00003ffffffff000
R10: ffff99ca4bf5f5f0 R11: 00007f2a349fc000 R12: 0000000000000010
R13: ffffffffbb661516 R14: ffffffffffffff10 R15: 0000000000000282
FS:  00007f2a34af9700(0000) GS:ffff99ccab680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb47f02d000 CR3: 00000004bf680000 CR4: 00000000001607e0
Call Trace:
[<ffffffffbb67b5c0>] ? leave_mm+0x110/0x110
[<ffffffffbb67b5c0>] ? leave_mm+0x110/0x110
[<ffffffffbb67b5c0>] ? leave_mm+0x110/0x110
[<ffffffffbb7125df>] smp_call_function_single+0x5f/0xa0
[<ffffffffbb977345>] ? cpumask_next_and+0x35/0x50
[<ffffffffbb712b8b>] smp_call_function_many+0x22b/0x270
[<ffffffffbb67b788>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbb67b7f8>] flush_tlb_mm_range+0x68/0x140
[<ffffffffbb7e57d7>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffbb7e70d5>] tlb_finish_mmu+0x55/0x60
[<ffffffffbb7e9f02>] zap_page_range+0x142/0x190
[<ffffffffbb7e4ded>] SyS_madvise+0x49d/0xac0
[<ffffffffbb7d784b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffbbd76ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 a0 bf 34 bc 48 89 df e8 1a bd 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #600226
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffbb114a22
Build id: 3.10.0-1062.1.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffbb07d1d0
Build id: 3.10.0-1062.1.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffbb07d1d0
Build id: 3.10.0-1062.1.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffbb07d1d0
Build id: 3.10.0-1062.1.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffbb114b3f
Build id: 3.10.0-1062.1.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffbb37f475
Build id: 3.10.0-1062.1.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffbb1150eb
Build id: 3.10.0-1062.1.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffbb07d398
Build id: 3.10.0-1062.1.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffbb07d40b
Build id: 3.10.0-1062.1.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffbb1ea166
Build id: 3.10.0-1062.1.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffbb1ebc2f
Build id: 3.10.0-1062.1.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffbb1ebd53
Build id: 3.10.0-1062.1.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffbb1eea5a
Build id: 3.10.0-1062.1.1.el7.x86_64
0xea
14
? do_mmap
vmlinux 0xffffffffbb1f8e48
Build id: 3.10.0-1062.1.1.el7.x86_64
0x378
15
SyS_madvise
vmlinux 0xffffffffbb1e976d
Build id: 3.10.0-1062.1.1.el7.x86_64
0x49d
16
? vm_mmap_pgoff
vmlinux 0xffffffffbb1dc14b
Build id: 3.10.0-1062.1.1.el7.x86_64
0xeb
17
system_call_fastpath
vmlinux 0xffffffffbb78cede
Build id: 3.10.0-1062.1.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ActiveMQ Broker:50483]
Modules linked in: rfcomm bnep vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event squashfs loop iosf_mbi ppdev crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi snd_ac97_codec vmw_balloon ac97_bus snd_seq btusb btrtl btbcm btintel snd_seq_device aesni_intel bluetooth snd_pcm lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr snd_timer snd rfkill sg soundcore parport_pc parport vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit libnvdimm ata_piix drm e1000 mptspi crct10dif_pclmul crct10dif_common scsi_transport_spi libata crc32c_intel mptscsih mptbase serio_raw drm_panel_orientation_quirks
dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 50483 Comm: ActiveMQ Broker Kdump: loaded Tainted: G        W    L ------------   3.10.0-1062.1.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff95ca81f05230 ti: ffff95ca82164000 task.ti: ffff95ca82164000
RIP: 0010:[<ffffffffbb114a22>]  [<ffffffffbb114a22>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff95ca82167be0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff95ca82167bb0 RCX: 0000000000000005
RDX: ffffffffbb813f50 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff95ca82167c30 R08: ffffffffbb813f48 R09: 00003ffffffff000
R10: ffff95caa5098af8 R11: 00007fe9f76ba000 R12: 00000000b965acc0
R13: fffff2378023a030 R14: 00007fe9f76ba000 R15: ffff95ca81f05230
FS:  00007fe9f77b7700(0000) GS:ffff95cab9640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3af4342000 CR3: 00000001309d2000 CR4: 00000000003607e0
Call Trace:
[<ffffffffbb07d1d0>] ? leave_mm+0x110/0x110
[<ffffffffbb07d1d0>] ? leave_mm+0x110/0x110
[<ffffffffbb07d1d0>] ? leave_mm+0x110/0x110
[<ffffffffbb114b3f>] smp_call_function_single+0x5f/0xa0
[<ffffffffbb37f475>] ? cpumask_next_and+0x35/0x50
[<ffffffffbb1150eb>] smp_call_function_many+0x22b/0x270
[<ffffffffbb07d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbb07d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffbb1ea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffbb1ebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffbb1ebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffffbb1eea5a>] zap_page_range+0xea/0x130
[<ffffffffbb1f8e48>] ? do_mmap+0x378/0x530
[<ffffffffbb1e976d>] SyS_madvise+0x49d/0xac0
[<ffffffffbb1dc14b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffbb78cede>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 d7 d4 bb 48 89 df e8 d7 1a 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #586220
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8c916562
Build id: 3.10.0-1160.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff8c87e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff8c87e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff8c87e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff8c91667f
Build id: 3.10.0-1160.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff8cb85de5
Build id: 3.10.0-1160.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff8c916c2b
Build id: 3.10.0-1160.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8c87e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8c87e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8c9edd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff8c9ef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff8c9ef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff8c9f2dba
Build id: 3.10.0-1160.el7.x86_64
0xea
14
? futex_wake
vmlinux 0xffffffff8c912040
Build id: 3.10.0-1160.el7.x86_64
0x90
15
SyS_madvise
vmlinux 0xffffffff8c9ed2ad
Build id: 3.10.0-1160.el7.x86_64
0x49d
16
? __schedule
vmlinux 0xffffffff8cf86e82
Build id: 3.10.0-1160.el7.x86_64
0x5a2
17
? __schedule
vmlinux 0xffffffff8cf86e76
Build id: 3.10.0-1160.el7.x86_64
0x596
18
? __schedule
vmlinux 0xffffffff8cf86e82
Build id: 3.10.0-1160.el7.x86_64
0x5a2
19
system_call_fastpath
vmlinux 0xffffffff8cf93f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 68s! [JS Helper:59749]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi aesni_intel snd_ac97_codec lrw ac97_bus
gf128mul ppdev glue_helper ablk_helper snd_seq snd_seq_device cryptd vmw_balloon snd_pcm snd_timer snd joydev pcspkr soundcore sg i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom ata_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops pata_acpi mptspi ttm scsi_transport_spi crct10dif_pclmul mptscsih crct10dif_common ata_piix drm mptbase nfit e1000 crc32c_intel serio_raw drm_panel_orientation_quirks libata libnvdimm fuse
CPU: 3 PID: 59749 Comm: JS Helper Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff99a68398e300 ti: ffff99a678398000 task.ti: ffff99a678398000
RIP: 0010:[<ffffffff8c916562>]  [<ffffffff8c916562>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff99a67839bbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff99a67839bbb0 RCX: 0000000000000003
RDX: ffffffff8d013c90 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff99a67839bc30 R08: ffffffff8d013c88 R09: 00003ffffffff000
R10: ffff99a643702e58 R11: 0000000000000246 R12: ffff99a76f53e400
R13: ffff99a68398e368 R14: ffffffff8c8e4759 R15: ffff99a67839bb60
FS:  00007f5bd0f7c700(0000) GS:ffff99a7796c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f79a335242c CR3: 0000000046c48000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8c87e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8c87e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8c87e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8c91667f>] smp_call_function_single+0x5f/0xa0
[<ffffffff8cb85de5>] ? cpumask_next_and+0x35/0x50
[<ffffffff8c916c2b>] smp_call_function_many+0x22b/0x270
[<ffffffff8c87e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8c87e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8c9edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8c9ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8c9ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff8c9f2dba>] zap_page_range+0xea/0x130
[<ffffffff8c912040>] ? futex_wake+0x90/0x180
[<ffffffff8c9ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffff8cf86e82>] ? __schedule+0x5a2/0x860
[<ffffffff8cf86e76>] ? __schedule+0x596/0x860
[<ffffffff8cf86e82>] ? __schedule+0x5a2/0x860
[<ffffffff8cf93f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 13 55 8d 48 89 df e8 f7 69 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #583992
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa3d1109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffa3c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffa3c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffa3c7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffa3d111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa3f75465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa3d1175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa3c7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa3c7a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffa3de4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffa3de5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffa3de87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
? do_mmap
vmlinux 0xffffffffa3df28f8
Build id: 3.10.0-957.el7.x86_64
0x378
14
SyS_madvise
vmlinux 0xffffffffa3de368d
Build id: 3.10.0-957.el7.x86_64
0x49d
15
? vm_mmap_pgoff
vmlinux 0xffffffffa3dd612b
Build id: 3.10.0-957.el7.x86_64
0xeb
16
system_call_fastpath
vmlinux 0xffffffffa4374ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [java:21317]
Modules linked in: vmw_vsock_vmci_transport vsock vfat fat crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd pcspkr joydev sg parport_pc parport tpm_crb vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx e1000e crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm_kms_helper ptp pps_core syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci mptsas libahci drm ata_piix scsi_transport_sas nfit mptscsih libata libnvdimm mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 21317 Comm: java Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.17369862.B64.2012240522 12/24/2020
task: ffff889fff7a5140 ti: ffff889acb470000 task.ti: ffff889acb470000
RIP: 0010:[<ffffffffa3d1109a>]  [<ffffffffa3d1109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff889acb473bf0  EFLAGS: 00000202
RAX: 0000000000000010 RBX: ffff889acb473bc0 RCX: 0000000000000030
RDX: 000000000000ffff RSI: 0000000000000010 RDI: 0000000000000282
RBP: ffff889acb473c38 R08: ffffffffa4413500 R09: 00003ffffffff000
R10: ffff88a0302e57a0 R11: 00007f89c2bfb000 R12: ffff88a20fe82940
R13: 000000001469ad54 R14: 00007f89c2bfb000 R15: ffff889fff7a5140
FS:  00007f89c2cf8700(0000) GS:ffff88a2df380000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2212c90f80 CR3: 000000080ca7a000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffffa3c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa3c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa3c7a3d0>] ? leave_mm+0x110/0x110
[<ffffffffa3d111af>] smp_call_function_single+0x5f/0xa0
[<ffffffffa3f75465>] ? cpumask_next_and+0x35/0x50
[<ffffffffa3d1175b>] smp_call_function_many+0x22b/0x270
[<ffffffffa3c7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa3c7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa3de4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffa3de5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffa3de87c2>] zap_page_range+0x142/0x190
[<ffffffffa3df28f8>] ? do_mmap+0x378/0x530
[<ffffffffa3de368d>] SyS_madvise+0x49d/0xac0
[<ffffffffa3dd612b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffa4374ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 94 a4 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #511361
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8190779a
Build id: 3.10.0-862.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff81876e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff81876e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff81876e50
Build id: 3.10.0-862.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff819078af
Build id: 3.10.0-862.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81b4a275
Build id: 3.10.0-862.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff81907e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff81877018
Build id: 3.10.0-862.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81877088
Build id: 3.10.0-862.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffff819c0497
Build id: 3.10.0-862.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff819c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff819c4a02
Build id: 3.10.0-862.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffff819bfafd
Build id: 3.10.0-862.el7.x86_64
0x3cd
14
? vm_mmap_pgoff
vmlinux 0xffffffff819b35eb
Build id: 3.10.0-862.el7.x86_64
0xeb
15
system_call_fastpath
vmlinux 0xffffffff81f1f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [java:29695]
Modules linked in: ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 vmw_vsock_vmci_transport vsock xt_multiport xt_conntrack nf_conntrack iptable_filter sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr sg vmw_balloon nfit i2c_piix4 vmw_vmci parport_pc parport libnvdimm shpchp joydev sch_fq_codel ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm ata_piix libahci drm crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw vmxnet3 i2c_core vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod
CPU: 22 PID: 29695 Comm: java Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff88878cb99fa0 ti: ffff8883f5b7c000 task.ti: ffff8883f5b7c000
RIP: 0010:[<ffffffff8190779a>]  [<ffffffff8190779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8883f5b7fbf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8883f5b7fbc0 RCX: 0000000000000005
RDX: ffffffff82013b70 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8883f5b7fc38 R08: ffffffff82013b68 R09: ffff88801b3da288
R10: ffffe29f6038e500 R11: 00007fa5fee7c000 R12: ffff88878cb99fa0
R13: ffff88878cb99fa0 R14: 00007fa5fee7c000 R15: ffff88878cb99fa0
FS:  00007fa5fef79700(0000) GS:ffff88879db80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9798ac90c8 CR3: 0000000801298000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff81876e50>] ? leave_mm+0x110/0x110
[<ffffffff81876e50>] ? leave_mm+0x110/0x110
[<ffffffff81876e50>] ? leave_mm+0x110/0x110
[<ffffffff819078af>] smp_call_function_single+0x5f/0xa0
[<ffffffff81b4a275>] ? cpumask_next_and+0x35/0x50
[<ffffffff81907e5b>] smp_call_function_many+0x22b/0x270
[<ffffffff81877018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81877088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff819c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff819c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff819c4a02>] zap_page_range+0x142/0x190
[<ffffffff819bfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff819b35eb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff81f1f7d5>] system_call_fastpath+0x1c/0x21
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 53 82 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #506540
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb6d16a36
Build id: 3.10.0-1160.59.1.el7.x86_64
0x106
2
? leave_mm
vmlinux 0xffffffffb6c7ddd0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffb6c7ddd0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffb6c7ddd0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffb6d16b4f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb6f86a55
Build id: 3.10.0-1160.59.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb6d170fb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb6c7e258
Build id: 3.10.0-1160.59.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb6c7e2cb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb6dee676
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb6df013f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb6df0263
Build id: 3.10.0-1160.59.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffb6df372a
Build id: 3.10.0-1160.59.1.el7.x86_64
0xea
14
? futex_wake
vmlinux 0xffffffffb6d12670
Build id: 3.10.0-1160.59.1.el7.x86_64
0x90
15
SyS_madvise
vmlinux 0xffffffffb6dedc1d
Build id: 3.10.0-1160.59.1.el7.x86_64
0x49d
16
system_call_fastpath
vmlinux 0xffffffffb7399f92
Build id: 3.10.0-1160.59.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ganesha.nfsd:24380]
Modules linked in: tcp_diag inet_diag iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod xfs libcrc32c sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg i2c_piix4 parport_pc parport vmw_vmci ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci ata_piix drm nfit libata crct10dif_pclmul crct10dif_common crc32c_intel libnvdimm serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 5 PID: 24380 Comm: ganesha.nfsd Kdump: loaded Tainted: G        W      ------------   3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8ad6b04d9080 ti: ffff8addf888c000 task.ti: ffff8addf888c000
RIP: 0010:[<ffffffffb6d16a36>]  [<ffffffffb6d16a36>] generic_exec_single+0x106/0x1c0
RSP: 0018:ffff8addf888fbe0  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff8addf888fbb0 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff8addf888fc30 R08: ffffffffb74146c0 R09: 00007fd1771f5000
R10: 00007fd1772f2ba0 R11: 0000000000000206 R12: 000042957e50596b
R13: ffffffffb6ce482c R14: ffff8addf888fb80 R15: ffff8ade3ffd9008
FS:  00007fd1772f4700(0000) GS:ffff8ade3fd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000022a09d0 CR3: 000000080c7c6000 CR4: 00000000001607e0
Call Trace:
 [<ffffffffb6c7ddd0>] ? leave_mm+0x120/0x120
 [<ffffffffb6c7ddd0>] ? leave_mm+0x120/0x120
 [<ffffffffb6c7ddd0>] ? leave_mm+0x120/0x120
 [<ffffffffb6d16b4f>] smp_call_function_single+0x5f/0xa0
 [<ffffffffb6f86a55>] ? cpumask_next_and+0x35/0x50
 [<ffffffffb6d170fb>] smp_call_function_many+0x22b/0x270
 [<ffffffffb6c7e258>] native_flush_tlb_others+0xb8/0xc0
 [<ffffffffb6c7e2cb>] flush_tlb_mm_range+0x6b/0x140
 [<ffffffffb6dee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
 [<ffffffffb6df013f>] arch_tlb_finish_mmu+0x3f/0x80
 [<ffffffffb6df0263>] tlb_finish_mmu+0x23/0x30
 [<ffffffffb6df372a>] zap_page_range+0xea/0x130
 [<ffffffffb6d12670>] ? futex_wake+0x90/0x180
 [<ffffffffb6dedc1d>] SyS_madvise+0x49d/0xac0
 [<ffffffffb7399f92>] system_call_fastpath+0x25/0x2a
Code: 14 c5 60 18 95 b7 48 89 df e8 97 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 f6 43 20 01 <75> f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 00 0f 85 8f 
Complete report #500377
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810fcf5a
Build id: 3.10.0-693.21.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff810709d0
Build id: 3.10.0-693.21.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff810709d0
Build id: 3.10.0-693.21.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff810709d0
Build id: 3.10.0-693.21.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff810fd06f
Build id: 3.10.0-693.21.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813288a5
Build id: 3.10.0-693.21.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810fd61b
Build id: 3.10.0-693.21.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff81070b98
Build id: 3.10.0-693.21.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81070c68
Build id: 3.10.0-693.21.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.63
vmlinux 0xffffffff811b18c3
Build id: 3.10.0-693.21.1.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811b2be5
Build id: 3.10.0-693.21.1.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff811b516a
Build id: 3.10.0-693.21.1.el7.x86_64
0x13a
13
? futex_wake
vmlinux 0xffffffff810f8b80
Build id: 3.10.0-693.21.1.el7.x86_64
0x90
14
SyS_madvise
vmlinux 0xffffffff811b0f8d
Build id: 3.10.0-693.21.1.el7.x86_64
0x3cd
15
? vm_mmap_pgoff
vmlinux 0xffffffff811a57e2
Build id: 3.10.0-693.21.1.el7.x86_64
0xe2
16
system_call_fastpath
vmlinux 0xffffffff816c0715
Build id: 3.10.0-693.21.1.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [java:17389]
Modules linked in: hid_wacom tcp_lp veth rfcomm fuse xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul vmw_balloon btusb glue_helper ablk_helper cryptd btrtl btbcm btintel bluetooth joydev pcspkr nfit libnvdimm rfkill sg vmw_vmci i2c_piix4 parport_pc parport shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c vmwgfx sr_mod sd_mod crc_t10dif
drm_kms_helper cdrom crct10dif_generic syscopyarea ata_generic sysfillrect sysimgblt fb_sys_fops ttm pata_acpi drm crct10dif_pclmul crct10dif_common crc32c_intel mptspi scsi_transport_spi serio_raw e1000 mptscsih mptbase ata_piix i2c_core libata dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 17389 Comm: java Tainted: G             L ------------   3.10.0-693.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff88011aaa4f10 ti: ffff880023bc4000 task.ti: ffff880023bc4000
RIP: 0010:[<ffffffff810fcf5a>]  [<ffffffff810fcf5a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff880023bc7bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff880023bc7bc0 RCX: 0000000000000002
RDX: ffffffff816dcb70 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff880023bc7c40 R08: ffffffff816dcb68 R09: 00007f4148cde000
R10: 0000000000000008 R11: 00007f4148be6000 R12: 00007f4148be6000
R13: ffffffff810d16ac R14: ffff880023bc7ba0 R15: 0000000000000001
FS:  00007f4148ce3700(0000) GS:ffff880235600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4148cceab8 CR3: 000000009d9fe000 CR4: 00000000003607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff810709d0>] ? leave_mm+0x110/0x110
[<ffffffff810709d0>] ? leave_mm+0x110/0x110
[<ffffffff810709d0>] ? leave_mm+0x110/0x110
[<ffffffff810fd06f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813288a5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810fd61b>] smp_call_function_many+0x22b/0x270
[<ffffffff81070b98>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81070c68>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811b18c3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811b2be5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b516a>] zap_page_range+0x13a/0x180
[<ffffffff810f8b80>] ? futex_wake+0x90/0x180
[<ffffffff811b0f8d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff811a57e2>] ? vm_mmap_pgoff+0xe2/0x110
[<ffffffff816c0715>] system_call_fastpath+0x1c/0x21
Code: 80 95 01 00 48 89 de 48 03 14 c5 a0 10 b2 81 48 89 df e8 7a 03 24 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #432994
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9930779a
Build id: 3.10.0-862.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff99276e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff99276e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff99276e50
Build id: 3.10.0-862.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff993078af
Build id: 3.10.0-862.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9954a275
Build id: 3.10.0-862.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff99307e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff99277018
Build id: 3.10.0-862.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff99277088
Build id: 3.10.0-862.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffff993c0497
Build id: 3.10.0-862.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff993c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff993c4a02
Build id: 3.10.0-862.el7.x86_64
0x142
13
? do_mmap
vmlinux 0xffffffff993cdf0c
Build id: 3.10.0-862.el7.x86_64
0x3ac
14
SyS_madvise
vmlinux 0xffffffff993bfafd
Build id: 3.10.0-862.el7.x86_64
0x3cd
15
? vm_mmap_pgoff
vmlinux 0xffffffff993b35eb
Build id: 3.10.0-862.el7.x86_64
0xeb
16
system_call_fastpath
vmlinux 0xffffffff9991f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [java:109215]
Modules linked in: tcp_lp nls_utf8 isofs fuse veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec aesni_intel lrw ppdev ac97_bus gf128mul glue_helper snd_seq ablk_helper cryptd snd_seq_device vmw_balloon snd_pcm snd_timer snd pcspkr joydev nfit soundcore sg libnvdimm parport_pc vmw_vmci parport i2c_piix4 shpchp ip_tables xfs libcrc32c sr_mod cdrom ata_generic
pata_acpi sd_mod crc_t10dif crct10dif_generic ahci crct10dif_pclmul libahci crct10dif_common ata_piix vmwgfx drm_kms_helper mptspi scsi_transport_spi crc32c_intel syscopyarea mptscsih sysfillrect sysimgblt fb_sys_fops ttm drm e1000 libata serio_raw mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 109215 Comm: java Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff8dcb8d83bf40 ti: ffff8dcbbc808000 task.ti: ffff8dcbbc808000
RIP: 0010:[<ffffffff9930779a>]  [<ffffffff9930779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8dcbbc80bbf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8dcbbc80bbc0 RCX: 0000000000000001
RDX: ffffffff99a13170 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8dcbbc80bc38 R08: ffffffff99a13168 R09: ffff8dcbe389e5e8
R10: 0000000000000008 R11: 00007fdd3e305000 R12: ffff8dcd5bbc1ac0
R13: 000000001b6de301 R14: 00007fdd3e305000 R15: ffff8dcb8d83bf40
FS:  00007fdd3e402700(0000) GS:ffff8dcdf3680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb56aefdf68 CR3: 000000027f2d8000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff99276e50>] ? leave_mm+0x110/0x110
[<ffffffff99276e50>] ? leave_mm+0x110/0x110
[<ffffffff99276e50>] ? leave_mm+0x110/0x110
[<ffffffff993078af>] smp_call_function_single+0x5f/0xa0
[<ffffffff9954a275>] ? cpumask_next_and+0x35/0x50
[<ffffffff99307e5b>] smp_call_function_many+0x22b/0x270
[<ffffffff99277018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff99277088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff993c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff993c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff993c4a02>] zap_page_range+0x142/0x190
[<ffffffff993cdf0c>] ? do_mmap+0x3ac/0x4d0
[<ffffffff993bfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff993b35eb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff9991f7d5>] system_call_fastpath+0x1c/0x21
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 f3 99 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #425660
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffaa116f72
Build id: 3.10.0-1127.19.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffaa07e730
Build id: 3.10.0-1127.19.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffaa07e730
Build id: 3.10.0-1127.19.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffaa07e730
Build id: 3.10.0-1127.19.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffaa11708f
Build id: 3.10.0-1127.19.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffaa385005
Build id: 3.10.0-1127.19.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffaa11763b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffaa07e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffaa07e96b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6b
10
zap_page_range
vmlinux 0xffffffffaa1f2ed8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xd8
11
? futex_wake
vmlinux 0xffffffffaa112a70
Build id: 3.10.0-1127.19.1.el7.x86_64
0x90
12
SyS_madvise
vmlinux 0xffffffffaa1ed3ed
Build id: 3.10.0-1127.19.1.el7.x86_64
0x49d
13
system_call_fastpath
vmlinux 0xffffffffaa792ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [mariadbd:5480]
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [mariadbd:6029]
Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ppdev joydev pcspkr vmw_balloon i2c_piix4 parport_pc parport vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper ahci ata_piix syscopyarea sysfillrect sysimgblt libahci
fb_sys_fops ttm sd_mod crc_t10dif libata crct10dif_generic drm crct10dif_pclmul crct10dif_common serio_raw crc32c_intel nfit vmxnet3 libnvdimm vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 6029 Comm: mariadbd Kdump: loaded Tainted: G             L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8ebb02c00000 ti: ffff8ebc3896c000 task.ti: ffff8ebc3896c000
RIP: 0010:[<ffffffffaa116f72>]  [<ffffffffaa116f72>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8ebc3896fc30  EFLAGS: 00000202
RAX: 0000000000000006 RBX: ffff8ebc3896fc00 RCX: 000000000000003a
RDX: 000000000000003f RSI: 0000000000000006 RDI: 0000000000000286
RBP: ffff8ebc3896fc80 R08: ffffffffaa813e40 R09: 00007f5385f67000
R10: 00005613fff16760 R11: 0000000000000206 R12: ffff8ebc3fc5acc0
R13: ffffffffaa0e103c R14: ffff8ebc3896fbb8 R15: ffffffffaa0e2e89
FS:  00007f5386766700(0000) GS:ffff8ebc3fd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc8f4012000 CR3: 00000000b1e22000 CR4: 00000000001607e0
Call Trace:
[<ffffffffaa07e730>] ? leave_mm+0x120/0x120
[<ffffffffaa07e730>] ? leave_mm+0x120/0x120
[<ffffffffaa07e730>] ? leave_mm+0x120/0x120
[<ffffffffaa11708f>] smp_call_function_single+0x5f/0xa0
[<ffffffffaa385005>] ? cpumask_next_and+0x35/0x50
[<ffffffffaa11763b>] smp_call_function_many+0x22b/0x270
[<ffffffffaa07e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaa07e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaa1f2ed8>] zap_page_range+0xd8/0x150
[<ffffffffaa112a70>] ? futex_wake+0x90/0x180
[<ffffffffaa1ed3ed>] SyS_madvise+0x49d/0xab0
[<ffffffffaa792ed2>] system_call_fastpath+0x25/0x2a
Complete report #378311
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9eb16f76
Build id: 3.10.0-1127.19.1.el7.x86_64
0x106
2
? leave_mm
vmlinux 0xffffffff9ea7e730
Build id: 3.10.0-1127.19.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9ea7e730
Build id: 3.10.0-1127.19.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9ea7e730
Build id: 3.10.0-1127.19.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff9eb1708f
Build id: 3.10.0-1127.19.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9ed85005
Build id: 3.10.0-1127.19.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9eb1763b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9ea7e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9ea7e96b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6b
10
zap_page_range
vmlinux 0xffffffff9ebf2ed8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xd8
11
? do_mmap
vmlinux 0xffffffff9ebfcb28
Build id: 3.10.0-1127.19.1.el7.x86_64
0x378
12
SyS_madvise
vmlinux 0xffffffff9ebed3ed
Build id: 3.10.0-1127.19.1.el7.x86_64
0x49d
13
? vm_mmap_pgoff
vmlinux 0xffffffff9ebde63b
Build id: 3.10.0-1127.19.1.el7.x86_64
0xeb
14
system_call_fastpath
vmlinux 0xffffffff9f192ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [HikariPool-1 co:106961]
Modules linked in: ip_vs_rr xt_ipvs ip_vs xt_REDIRECT nf_nat_redirect xt_nat veth vxlan ip6_udp_tunnel udp_tunnel xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) ampnetworkflow(OE) ampfsm(OE) vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon joydev sg pcspkr parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm sd_mod drm crc_t10dif crct10dif_generic ahci ata_piix nfit libahci crct10dif_pclmul libnvdimm crct10dif_common crc32c_intel libata serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 106961 Comm: HikariPool-1 co Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8970e5b0a0e0 ti: ffff89730e854000 task.ti: ffff89730e854000
RIP: 0010:[<ffffffff9eb16f76>]  [<ffffffff9eb16f76>] generic_exec_single+0x106/0x1c0
RSP: 0018:ffff89730e857c30  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff89730e857c00 RCX: 0000000000000001
RDX: ffffffff9f2136d0 RSI: 0000000000000080 RDI: 0000000000000286
RBP: ffff89730e857c80 R08: ffffffff9f2136c8 R09: 00007f19a12ba000
R10: 00007f1968000078 R11: 0000000000000206 R12: ffffffff9ebcdd35
R13: ffff89730e857ba8 R14: ffff89730e857c88 R15: ffff89730e857c88
FS:  00007f19a13b7700(0000) GS:ffff89746d740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002f3b59c8f080 CR3: 00000000a39a8000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9ea7e730>] ? leave_mm+0x120/0x120
[<ffffffff9ea7e730>] ? leave_mm+0x120/0x120
[<ffffffff9ea7e730>] ? leave_mm+0x120/0x120
[<ffffffff9eb1708f>] smp_call_function_single+0x5f/0xa0
[<ffffffff9ed85005>] ? cpumask_next_and+0x35/0x50
[<ffffffff9eb1763b>] smp_call_function_many+0x22b/0x270
[<ffffffff9ea7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9ea7e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9ebf2ed8>] zap_page_range+0xd8/0x150
[<ffffffff9ebfcb28>] ? do_mmap+0x378/0x530
[<ffffffff9ebed3ed>] SyS_madvise+0x49d/0xab0
[<ffffffff9ebde63b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff9f192ed2>] system_call_fastpath+0x25/0x2a
Code: 14 c5 a0 10 75 9f 48 89 df e8 07 52 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 f6 43 20 01 <75> f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 00 0f 85 8f 
Complete report #309642
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8ad16a32
Build id: 3.10.0-1160.45.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff8ac7ddd0
Build id: 3.10.0-1160.45.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff8ac7ddd0
Build id: 3.10.0-1160.45.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff8ac7ddd0
Build id: 3.10.0-1160.45.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff8ad16b4f
Build id: 3.10.0-1160.45.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff8af86a45
Build id: 3.10.0-1160.45.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff8ad170fb
Build id: 3.10.0-1160.45.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8ac7e258
Build id: 3.10.0-1160.45.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8ac7e2cb
Build id: 3.10.0-1160.45.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8adee676
Build id: 3.10.0-1160.45.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff8adf013f
Build id: 3.10.0-1160.45.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff8adf0263
Build id: 3.10.0-1160.45.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff8adf372a
Build id: 3.10.0-1160.45.1.el7.x86_64
0xea
14
? futex_wake
vmlinux 0xffffffff8ad12670
Build id: 3.10.0-1160.45.1.el7.x86_64
0x90
15
SyS_madvise
vmlinux 0xffffffff8adedc1d
Build id: 3.10.0-1160.45.1.el7.x86_64
0x49d
16
system_call_fastpath
vmlinux 0xffffffff8b395f92
Build id: 3.10.0-1160.45.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 67s! [ThreadPoolForeg:40657]
Modules linked in: tcp_lp tun devlink nfsv3 nfs_acl cmac rpcsec_gss_krb5 auth_rpcgss arc4 md4 nfsv4 nls_utf8 cifs nfs lockd grace fscache ccm dns_resolver ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip_set ebtable_filter ebtables ip6table_filter ip6_tables ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_CHECKSUM iptable_mangle iptable_security iptable_raw ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul snd_ens1371
glue_helper ablk_helper cryptd snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device joydev snd_pcm pcspkr sg snd_timer snd soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix nfit crct10dif_pclmul crct10dif_common crc32c_intel libata libnvdimm e1000 nvme serio_raw nvme_core drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 8 PID: 40657 Comm: ThreadPoolForeg Not tainted 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9b8beba83180 ti: ffff9b8aed3f0000 task.ti: ffff9b8aed3f0000
RIP: 0010:[<ffffffff8ad16a32>]  [<ffffffff8ad16a32>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9b8aed3f3be0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9b8aed3f3bb0 RCX: 0000000000000004
RDX: ffffffff8b413f50 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9b8aed3f3c30 R08: ffffffff8b413f48 R09: 00003ffffffff000
R10: ffff9b8b9d0e0000 R11: 0000000000000206 R12: 0000005c0607eebb
R13: ffffffff8ace482c R14: ffff9b8aed3f3b80 R15: ffff9b8af8a8e000
FS:  00007fcf03aad700(0000) GS:ffff9b8c33800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb0e000bb98 CR3: 00000000a4bf8000 CR4: 00000000003407e0
Call Trace:
[<ffffffff8ac7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff8ac7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff8ac7ddd0>] ? leave_mm+0x120/0x120
[<ffffffff8ad16b4f>] smp_call_function_single+0x5f/0xa0
[<ffffffff8af86a45>] ? cpumask_next_and+0x35/0x50
[<ffffffff8ad170fb>] smp_call_function_many+0x22b/0x270
[<ffffffff8ac7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8ac7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8adee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8adf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8adf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff8adf372a>] zap_page_range+0xea/0x130
[<ffffffff8ad12670>] ? futex_wake+0x90/0x180
[<ffffffff8adedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff8b395f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 60 15 95 8b 48 89 df e8 87 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #281895
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff875163c2
Build id: 3.10.0-1062.9.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff8747e2b0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff8747e2b0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff8747e2b0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff875164df
Build id: 3.10.0-1062.9.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff87781635
Build id: 3.10.0-1062.9.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff87516a8b
Build id: 3.10.0-1062.9.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8747e478
Build id: 3.10.0-1062.9.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8747e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff875ec026
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff875edaef
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff875edc13
Build id: 3.10.0-1062.9.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff875f091a
Build id: 3.10.0-1062.9.1.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffff874db478
Build id: 3.10.0-1062.9.1.el7.x86_64
0x58
15
SyS_madvise
vmlinux 0xffffffff875eb62d
Build id: 3.10.0-1062.9.1.el7.x86_64
0x49d
16
? vm_mmap_pgoff
vmlinux 0xffffffff875ddfdb
Build id: 3.10.0-1062.9.1.el7.x86_64
0xeb
17
system_call_fastpath
vmlinux 0xffffffff87b8dede
Build id: 3.10.0-1062.9.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [d.JR Dispatcher:6380]
Modules linked in: pcc_cpufreq fuse tun ip6table_filter udp_diag netlink_diag af_packet_diag binfmt_misc unix_diag tcp_diag inet_diag ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables xt_statistic vxlan ip6_udp_tunnel udp_tunnel ipt_REJECT nf_reject_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs iptable_mangle xt_comment xt_mark xt_nat veth nf_conntrack_netlink nfnetlink ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) sunrpc dm_mirror dm_region_hash dm_log dm_mod ppdev sb_edac iosf_mbi cirrus kvm_intel ttm kvm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops irqbypass drm crc32_pclmul parport_pc joydev ghash_clmulni_intel
virtio_balloon parport pcspkr i2c_piix4 aesni_intel drm_panel_orientation_quirks lrw gf128mul glue_helper ablk_helper cryptd ip_tables xfs libcrc32c ata_generic pata_acpi virtio_net virtio_blk ata_piix libata crct10dif_pclmul crct10dif_common virtio_pci virtio_ring crc32c_intel floppy serio_raw virtio
CPU: 5 PID: 6380 Comm: d.JR Dispatcher Tainted: G             L ------------ T 3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: Red Hat OpenStack Compute, BIOS 1.11.0-2.el7 04/01/2014
task: ffff97a7b7601070 ti: ffff97a5fbad0000 task.ti: ffff97a5fbad0000
RIP: 0010:[<ffffffff875163c2>]  [<ffffffff875163c2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff97a5fbad3be0  EFLAGS: 00000202
RAX: ffff97a69763f700 RBX: ffff97a7bf340000 RCX: ffff97a69763f7a0
RDX: ffff97a7bf3db800 RSI: ffff97a5fbad3be0 RDI: ffff97a5fbad3be0
RBP: ffff97a5fbad3c30 R08: 0000000000000001 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000104 R12: ffff97a5fbad3ba8
R13: ffff97a7b7fcc6c0 R14: 0000000500000000 R15: 0000000000000000
FS:  00007eff7007d640(0000) GS:ffff97a7bf340000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000012b5099ea004 CR3: 000000048bac4000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8747e2b0>] ? leave_mm+0x110/0x110
[<ffffffff8747e2b0>] ? leave_mm+0x110/0x110
[<ffffffff8747e2b0>] ? leave_mm+0x110/0x110
[<ffffffff875164df>] smp_call_function_single+0x5f/0xa0
[<ffffffff87781635>] ? cpumask_next_and+0x35/0x50
[<ffffffff87516a8b>] smp_call_function_many+0x22b/0x270
[<ffffffff8747e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8747e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff875ec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff875edaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff875edc13>] tlb_finish_mmu+0x23/0x30
[<ffffffff875f091a>] zap_page_range+0xea/0x130
[<ffffffff874db478>] ? wake_up_q+0x58/0x80
[<ffffffff875eb62d>] SyS_madvise+0x49d/0xac0
[<ffffffff875ddfdb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff87b8dede>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 20 e8 14 88 48 89 df e8 f7 22 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #233100
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb3d111ba
Build id: 3.10.0-957.5.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffb3c7a400
Build id: 3.10.0-957.5.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffb3c7a400
Build id: 3.10.0-957.5.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffb3c7a400
Build id: 3.10.0-957.5.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffb3d112cf
Build id: 3.10.0-957.5.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb3f75a45
Build id: 3.10.0-957.5.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb3d1187b
Build id: 3.10.0-957.5.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb3c7a5c8
Build id: 3.10.0-957.5.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb3c7a638
Build id: 3.10.0-957.5.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffb3de4427
Build id: 3.10.0-957.5.1.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffb3de5d25
Build id: 3.10.0-957.5.1.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffb3de8b72
Build id: 3.10.0-957.5.1.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffffb3de3a3d
Build id: 3.10.0-957.5.1.el7.x86_64
0x49d
14
? vm_mmap_pgoff
vmlinux 0xffffffffb3dd64db
Build id: 3.10.0-957.5.1.el7.x86_64
0xeb
15
tracesys
vmlinux 0xffffffffb437505b
Build id: 3.10.0-957.5.1.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 67s! [nacos-grpc-exec:47488]
Modules linked in: tun veth br_netfilter cfg80211 rfkill binfmt_misc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop xt_CHECKSUM xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iptable_filter snd_seq_midi snd_seq_midi_event sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ext4 snd_ens1371 snd_rawmidi snd_ac97_codec
mbcache ac97_bus jbd2 snd_seq snd_seq_device snd_pcm snd_timer snd soundcore ppdev sg vmw_balloon pcspkr joydev parport_pc parport vmw_vmci i2c_piix4 uinput ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common ata_piix crc32c_intel libata mptspi scsi_transport_spi mptscsih mptbase serio_raw e1000 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 47488 Comm: nacos-grpc-exec Tainted: G             L ------------   3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2012
task: ffff8bf96670c100 ti: ffff8bfb2db9c000 task.ti: ffff8bfb2db9c000
RIP: 0010:[<ffffffffb3d111ba>]  [<ffffffffb3d111ba>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8bfb2db9fbf0  EFLAGS: 00000202
RAX: 0000000000000040 RBX: ffff8bfb63638810 RCX: 0000000000000001
RDX: 0000000000000040 RSI: 0000000000000000 RDI: 0000000000000286
RBP: ffff8bfb2db9fc38 R08: ffffffffb4412dc8 R09: 00003ffffffff000
R10: ffff8bfb63638288 R11: 00007f485f5f9000 R12: ffff8bfbbffda000
R13: ffffffffb3dc0eb6 R14: ffff8bfb2db9fc00 R15: 0000000000020200
FS:  00007f485f6f6700(0000) GS:ffff8bfbae640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9a06bbc180 CR3: 000000023fe60000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb3c7a400>] ? leave_mm+0x110/0x110
[<ffffffffb3c7a400>] ? leave_mm+0x110/0x110
[<ffffffffb3c7a400>] ? leave_mm+0x110/0x110
[<ffffffffb3d112cf>] smp_call_function_single+0x5f/0xa0
[<ffffffffb3f75a45>] ? cpumask_next_and+0x35/0x50
[<ffffffffb3d1187b>] smp_call_function_many+0x22b/0x270
[<ffffffffb3c7a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb3c7a638>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb3de4427>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffb3de5d25>] tlb_finish_mmu+0x55/0x60
[<ffffffffb3de8b72>] zap_page_range+0x142/0x190
[<ffffffffb3de3a3d>] SyS_madvise+0x49d/0xac0
[<ffffffffb3dd64db>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb437505b>] tracesys+0xa3/0xc9
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 94 b4 48 89 df e8 da b6 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #81852
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb39167a2
Build id: 3.10.0-1160.15.2.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffb387e6a0
Build id: 3.10.0-1160.15.2.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffb387e6a0
Build id: 3.10.0-1160.15.2.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffb387e6a0
Build id: 3.10.0-1160.15.2.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffb39168bf
Build id: 3.10.0-1160.15.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb3b862c5
Build id: 3.10.0-1160.15.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb3916e6b
Build id: 3.10.0-1160.15.2.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb387e868
Build id: 3.10.0-1160.15.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb387e8db
Build id: 3.10.0-1160.15.2.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb39ede86
Build id: 3.10.0-1160.15.2.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb39ef94f
Build id: 3.10.0-1160.15.2.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb39efa73
Build id: 3.10.0-1160.15.2.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffb39f2f3a
Build id: 3.10.0-1160.15.2.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffffb39ed42d
Build id: 3.10.0-1160.15.2.el7.x86_64
0x49d
15
? vm_mmap_pgoff
vmlinux 0xffffffffb39de32b
Build id: 3.10.0-1160.15.2.el7.x86_64
0xeb
16
system_call_fastpath
vmlinux 0xffffffffb3f94f92
Build id: 3.10.0-1160.15.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [Worker-66:25859]
Modules linked in: vfat fat uas usb_storage xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp nfsv3 llc nfs_acl nfs lockd grace fscache ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter mvfs(OE) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul
crct10dif_common nfit crc32c_intel libata libnvdimm mptspi e1000 scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks floppy fuse
CPU: 7 PID: 25859 Comm: Worker-66 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8e1eb420e300 ti: ffff8e21ec974000 task.ti: ffff8e21ec974000
RIP: 0010:[<ffffffffb39167a2>]  [<ffffffffb39167a2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8e21ec977be0  EFLAGS: 00000202
RAX: 0000000000000040 RBX: ffff8e2200000000 RCX: 0000000000000006
RDX: 0000000000000040 RSI: 0000000000000000 RDI: 0000000000000286
RBP: ffff8e21ec977c30 R08: ffffffffb4014448 R09: 00003ffffffff000
R10: ffff8e21d5c3b008 R11: 00007f6dfb0fe000 R12: ffffffffb39c8c16
R13: ffff8e21ec977bf0 R14: 0000000000020200 R15: 0000000000000000
FS:  00007f6dfb1fb700(0000) GS:ffff8e222e7c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f071680e8af CR3: 00000004137a8000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb387e6a0>] ? leave_mm+0x120/0x120
[<ffffffffb387e6a0>] ? leave_mm+0x120/0x120
[<ffffffffb387e6a0>] ? leave_mm+0x120/0x120
[<ffffffffb39168bf>] smp_call_function_single+0x5f/0xa0
[<ffffffffb3b862c5>] ? cpumask_next_and+0x35/0x50
[<ffffffffb3916e6b>] smp_call_function_many+0x22b/0x270
[<ffffffffb387e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb387e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb39ede86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb39ef94f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb39efa73>] tlb_finish_mmu+0x23/0x30
[<ffffffffb39f2f3a>] zap_page_range+0xea/0x130
[<ffffffffb39ed42d>] SyS_madvise+0x49d/0xac0
[<ffffffffb39de32b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb3f94f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 13 55 b4 48 89 df e8 97 6c 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #65332
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f933e
Build id: 3.10.0-693.11.1.el7.x86_64
0xfe
2
? leave_mm
vmlinux 0xffffffff8106e4a0
Build id: 3.10.0-693.11.1.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106e4a0
Build id: 3.10.0-693.11.1.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106e4a0
Build id: 3.10.0-693.11.1.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f943f
Build id: 3.10.0-693.11.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81322645
Build id: 3.10.0-693.11.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f99d3
Build id: 3.10.0-693.11.1.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.11.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.11.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.63
vmlinux 0xffffffff811acfc3
Build id: 3.10.0-693.11.1.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ae2e5
Build id: 3.10.0-693.11.1.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff811b086a
Build id: 3.10.0-693.11.1.el7.x86_64
0x13a
13
SyS_madvise
vmlinux 0xffffffff811ac68d
Build id: 3.10.0-693.11.1.el7.x86_64
0x3cd
14
? vm_mmap_pgoff
vmlinux 0xffffffff811a1232
Build id: 3.10.0-693.11.1.el7.x86_64
0xe2
15
system_call_fastpath
vmlinux 0xffffffff816b5089
Build id: 3.10.0-693.11.1.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [java:15652]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd nfit ppdev vmw_balloon libnvdimm sg joydev pcspkr parport_pc shpchp vmw_vmci parport i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common syscopyarea ahci crc32c_intel sysfillrect ata_piix
sysimgblt libahci fb_sys_fops ttm serio_raw libata drm vmxnet3 i2c_core vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 15652 Comm: java Tainted: G             L ------------   3.10.0-693.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff880941b09fa0 ti: ffff8809fc704000 task.ti: ffff8809fc704000
RIP: 0010:[<ffffffff810f933e>]  [<ffffffff810f933e>] generic_exec_single+0xfe/0x1a0
RSP: 0018:ffff8809fc707c20  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff8809fc707bf0 RCX: 0000000000000830
RDX: 0000000000000004 RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff8809fc707c70 R08: ffffffff816cebe0 R09: ffffea0023439200
R10: ffffea0023439200 R11: 00007fe5c192e000 R12: ffff880a2ffd8000
R13: ffffffff8118cb26 R14: ffff8809fc707c30 R15: 0000000000020200
FS:  00007fe5c196b700(0000) GS:ffff880a2fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000021fc000 CR3: 00000009d52e5000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106e4a0 ffff8809fc707cf8
0000000000000003 00000000f4cf833d 0000000000000000 0000000000000003
ffffffff8106e4a0 ffffffff8106e4a0 ffff8809fc707ca0 ffffffff810f943f
Call Trace:
[<ffffffff8106e4a0>] ? leave_mm+0x70/0x70
[<ffffffff8106e4a0>] ? leave_mm+0x70/0x70
[<ffffffff8106e4a0>] ? leave_mm+0x70/0x70
[<ffffffff810f943f>] smp_call_function_single+0x5f/0xa0
[<ffffffff81322645>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f99d3>] smp_call_function_many+0x223/0x260
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811acfc3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae2e5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b086a>] zap_page_range+0x13a/0x180
[<ffffffff811ac68d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff811a1232>] ? vm_mmap_pgoff+0xe2/0x110
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 48 89 de 48 03 14 c5 e0 fd b0 81 48 89 df e8 ba d9 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 f6 43 20 01 <75> f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 00 0f 85 7f 
Complete report #64559
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff88116802
Build id: 3.10.0-1160.11.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff8807e6a0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff8807e6a0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff8807e6a0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff8811691f
Build id: 3.10.0-1160.11.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff88385fe5
Build id: 3.10.0-1160.11.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff88116ecb
Build id: 3.10.0-1160.11.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8807e868
Build id: 3.10.0-1160.11.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8807e8db
Build id: 3.10.0-1160.11.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff881edee6
Build id: 3.10.0-1160.11.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff881ef9af
Build id: 3.10.0-1160.11.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff881efad3
Build id: 3.10.0-1160.11.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff881f2f9a
Build id: 3.10.0-1160.11.1.el7.x86_64
0xea
14
? handle_mm_fault
vmlinux 0xffffffff881f5f12
Build id: 3.10.0-1160.11.1.el7.x86_64
0xb22
15
? do_mmap
vmlinux 0xffffffff881fcbb8
Build id: 3.10.0-1160.11.1.el7.x86_64
0x378
16
SyS_madvise
vmlinux 0xffffffff881ed48d
Build id: 3.10.0-1160.11.1.el7.x86_64
0x49d
17
system_call_fastpath
vmlinux 0xffffffff88793f92
Build id: 3.10.0-1160.11.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ApplicationImpl:16866]
Modules linked in: xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) tcp_lp nls_utf8 isofs rfcomm xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event btusb btrtl snd_ens1371 btbcm snd_rawmidi snd_ac97_codec btintel coretemp bluetooth ac97_bus snd_seq iosf_mbi rfkill snd_seq_device crc32_pclmul ppdev snd_pcm ghash_clmulni_intel aesni_intel snd_timer snd lrw gf128mul soundcore glue_helper ablk_helper cryptd pcspkr sg joydev vmw_balloon parport_pc parport i2c_piix4 vmw_vmci
ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit libnvdimm ata_piix drm libata crct10dif_pclmul crct10dif_common crc32c_intel e1000 mptspi scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 16866 Comm: ApplicationImpl Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff900474382100 ti: ffff90057ef34000 task.ti: ffff90057ef34000
RIP: 0010:[<ffffffff88116802>]  [<ffffffff88116802>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff90057ef37be0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff90057ef37bb0 RCX: 0000000000000004
RDX: ffffffff88813f50 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff90057ef37c30 R08: ffffffff88813f48 R09: 00003ffffffff000
R10: ffff9006b4f73bd8 R11: 00007f08ce901000 R12: ffffffff882ba85c
R13: ffff90057ef37b68 R14: ffffffff8828631e R15: ffff90057ef37b48
FS:  00007f08ce9fd700(0000) GS:ffff9006df880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f09ca1452b0 CR3: 00000003b7a5e000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8807e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8807e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8807e6a0>] ? leave_mm+0x120/0x120
[<ffffffff8811691f>] smp_call_function_single+0x5f/0xa0
[<ffffffff88385fe5>] ? cpumask_next_and+0x35/0x50
[<ffffffff88116ecb>] smp_call_function_many+0x22b/0x270
[<ffffffff8807e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8807e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff881edee6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff881ef9af>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff881efad3>] tlb_finish_mmu+0x23/0x30
[<ffffffff881f2f9a>] zap_page_range+0xea/0x130
[<ffffffff881f5f12>] ? handle_mm_fault+0xb22/0xfb0
[<ffffffff881fcbb8>] ? do_mmap+0x378/0x530
[<ffffffff881ed48d>] SyS_madvise+0x49d/0xac0
[<ffffffff88793f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 13 d5 88 48 89 df e8 57 69 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #62098
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff95316f62
Build id: 3.10.0-1127.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff9527e730
Build id: 3.10.0-1127.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9527e730
Build id: 3.10.0-1127.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9527e730
Build id: 3.10.0-1127.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff9531707f
Build id: 3.10.0-1127.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff95584ee5
Build id: 3.10.0-1127.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9531762b
Build id: 3.10.0-1127.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9527e8f8
Build id: 3.10.0-1127.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9527e96b
Build id: 3.10.0-1127.el7.x86_64
0x6b
10
zap_page_range
vmlinux 0xffffffff953f2e58
Build id: 3.10.0-1127.el7.x86_64
0xd8
11
? wake_up_q
vmlinux 0xffffffff952db948
Build id: 3.10.0-1127.el7.x86_64
0x58
12
? futex_wake
vmlinux 0xffffffff95312a60
Build id: 3.10.0-1127.el7.x86_64
0x90
13
SyS_madvise
vmlinux 0xffffffff953ed36d
Build id: 3.10.0-1127.el7.x86_64
0x49d
14
system_call_fastpath
vmlinux 0xffffffff95992ed2
Build id: 3.10.0-1127.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 25s! [splunkd:1176]
Modules linked in: vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi kvm_intel kvm snd_ens1371 snd_rawmidi irqbypass snd_ac97_codec crc32_pclmul ac97_bus snd_seq ghash_clmulni_intel ppdev snd_seq_device btusb btrtl snd_pcm vmw_balloon aesni_intel btbcm lrw btintel gf128mul glue_helper ablk_helper bluetooth cryptd snd_timer joydev pcspkr rfkill snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common ahci drm ata_piix libahci crc32c_intel serio_raw nfit libnvdimm libata mptspi scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks
CPU: 4 PID: 1176 Comm: splunkd Kdump: loaded Not tainted 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff98f67bbaa0e0 ti: ffff98f6c73bc000 task.ti: ffff98f6c73bc000
RIP: 0010:[<ffffffff95316f62>]  [<ffffffff95316f62>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff98f6c73bfc30  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff98f6c73bfc00 RCX: 0000000000000004
RDX: ffffffff95a13d90 RSI: 0000000000000080 RDI: 0000000000000286
RBP: ffff98f6c73bfc80 R08: ffffffff95a13d88 R09: 00007f7e8e400000
R10: 0000000000000000 R11: 0000000000000206 R12: ffff98f7b371acc0
R13: ffffffff952e103c R14: ffff98f6c73bfbb8 R15: ffffffff952e2e89
FS:  00007f7e8e5ff700(0000) GS:ffff98f7b3700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000211e178 CR3: 00000001cd2ee000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9527e730>] ? leave_mm+0x120/0x120
[<ffffffff9527e730>] ? leave_mm+0x120/0x120
[<ffffffff9527e730>] ? leave_mm+0x120/0x120
[<ffffffff9531707f>] smp_call_function_single+0x5f/0xa0
[<ffffffff95584ee5>] ? cpumask_next_and+0x35/0x50
[<ffffffff9531762b>] smp_call_function_many+0x22b/0x270
[<ffffffff9527e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9527e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff953f2e58>] zap_page_range+0xd8/0x150
[<ffffffff952db948>] ? wake_up_q+0x58/0x80
[<ffffffff95312a60>] ? futex_wake+0x90/0x180
[<ffffffff953ed36d>] SyS_madvise+0x49d/0xab0
[<ffffffff95992ed2>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 0f f5 95 48 89 df e8 17 51 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #48494
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffbd50779a
Build id: 3.10.0-862.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffbd476e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffbd476e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffbd476e50
Build id: 3.10.0-862.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffbd5078af
Build id: 3.10.0-862.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffbd74a275
Build id: 3.10.0-862.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffbd507e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffbd477018
Build id: 3.10.0-862.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffbd477088
Build id: 3.10.0-862.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffffbd5c0497
Build id: 3.10.0-862.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffbd5c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffbd5c4a02
Build id: 3.10.0-862.el7.x86_64
0x142
13
? futex_wake
vmlinux 0xffffffffbd5033a0
Build id: 3.10.0-862.el7.x86_64
0x90
14
SyS_madvise
vmlinux 0xffffffffbd5bfafd
Build id: 3.10.0-862.el7.x86_64
0x3cd
15
system_call_fastpath
vmlinux 0xffffffffbdb1f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [ThreadPoolForeg:9745]
Modules linked in: tcp_lp rfcomm fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmhgfs(OE) bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev btusb btrtl vmw_balloon btbcm pcspkr btintel bluetooth vmw_vmci joydev snd_timer snd sg rfkill soundcore shpchp i2c_piix4 nfit libnvdimm parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul drm crct10dif_common crc32c_intel libata mptspi scsi_transport_spi serio_raw e1000 mptscsih mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 9745 Comm: ThreadPoolForeg Tainted: G           OE  ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff932ef3670000 ti: ffff932f3afa4000 task.ti: ffff932f3afa4000
RIP: 0010:[<ffffffffbd50779a>]  [<ffffffffbd50779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff932f3afa7bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff932f3afa7bc0 RCX: 0000000000000006
RDX: ffffffffbdc13df0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff932f3afa7c38 R08: ffffffffbdc13de8 R09: ffff932f10712948
R10: 00007fd9303ea8a0 R11: 0000000000000206 R12: ffff932f75718b00
R13: ffffffffbd4d865c R14: ffff932f3afa7ba0 R15: 0000000000000001
FS:  00007fd9303eb700(0000) GS:ffff932f757c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001e71c30 CR3: 00000001f8fba000 CR4: 00000000003607e0
Call Trace:
[<ffffffffbd476e50>] ? leave_mm+0x110/0x110
[<ffffffffbd476e50>] ? leave_mm+0x110/0x110
[<ffffffffbd476e50>] ? leave_mm+0x110/0x110
[<ffffffffbd5078af>] smp_call_function_single+0x5f/0xa0
[<ffffffffbd74a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffbd507e5b>] smp_call_function_many+0x22b/0x270
[<ffffffffbd477018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbd477088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffbd5c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffbd5c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffbd5c4a02>] zap_page_range+0x142/0x190
[<ffffffffbd5033a0>] ? futex_wake+0x90/0x180
[<ffffffffbd5bfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffffbdb1f7d5>] system_call_fastpath+0x1c/0x21
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 13 be 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #43078
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff96516562
Build id: 3.10.0-1160.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff9647e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9647e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9647e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff9651667f
Build id: 3.10.0-1160.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff96785de5
Build id: 3.10.0-1160.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff96516c2b
Build id: 3.10.0-1160.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9647e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9647e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff965edd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff965ef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff965ef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff965f2dba
Build id: 3.10.0-1160.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffff965ed2ad
Build id: 3.10.0-1160.el7.x86_64
0x49d
15
? vm_mmap_pgoff
vmlinux 0xffffffff965de1bb
Build id: 3.10.0-1160.el7.x86_64
0xeb
16
system_call_fastpath
vmlinux 0xffffffff96b93f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [java:12300]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod ext4 iosf_mbi crc32_pclmul mbcache ghash_clmulni_intel jbd2 aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon ppdev pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit libnvdimm ata_piix crct10dif_pclmul crct10dif_common e1000 mptspi
serio_raw crc32c_intel scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 3 PID: 12300 Comm: java Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8f4eceea1080 ti: ffff8f4ef5a7c000 task.ti: ffff8f4ef5a7c000
RIP: 0010:[<ffffffff96516562>]  [<ffffffff96516562>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8f4ef5a7fbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8f4ef5a7fbb0 RCX: 0000000000000002
RDX: ffffffff96c13a10 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8f4ef5a7fc30 R08: ffffffff96c13a08 R09: 00003ffffffff000
R10: ffff8f4e3cfbf878 R11: 00007f07fbb55000 R12: ffffffff965c8aa6
R13: ffff8f4ef5a7fbf0 R14: 0000000000020200 R15: 0000000000000000
FS:  00007f07fbc52700(0000) GS:ffff8f4ef56c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc822a06c00 CR3: 000000017e3d6000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9647e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9647e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9647e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9651667f>] smp_call_function_single+0x5f/0xa0
[<ffffffff96785de5>] ? cpumask_next_and+0x35/0x50
[<ffffffff96516c2b>] smp_call_function_many+0x22b/0x270
[<ffffffff9647e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9647e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff965edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff965ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff965ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff965f2dba>] zap_page_range+0xea/0x130
[<ffffffff965ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffff965de1bb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff96b93f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 13 15 97 48 89 df e8 f7 69 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #16713
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffbc31109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffbc27a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffbc27a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffbc27a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffbc3111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffbc575465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffbc31175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffbc27a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffbc27a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffbc3e4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffbc3e5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffbc3e87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
SyS_madvise
vmlinux 0xffffffffbc3e368d
Build id: 3.10.0-957.el7.x86_64
0x49d
14
? vm_mmap_pgoff
vmlinux 0xffffffffbc3d612b
Build id: 3.10.0-957.el7.x86_64
0xeb
15
system_call_fastpath
vmlinux 0xffffffffbc974ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [FelixResolver-1:20724]
Modules linked in: xt_nat veth xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) overlay(T) sunrpc snd_seq_midi snd_seq_midi_event coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_ens1371 lrw gf128mul ppdev glue_helper ablk_helper snd_rawmidi snd_ac97_codec ac97_bus cryptd snd_seq vmw_balloon snd_seq_device snd_pcm snd_timer snd soundcore pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic
pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ata_piix ttm mptspi libata scsi_transport_spi mptscsih mptbase drm e1000 drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 20724 Comm: FelixResolver-1 Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff944a2adb9040 ti: ffff944bbe724000 task.ti: ffff944bbe724000
RIP: 0010:[<ffffffffbc31109a>]  [<ffffffffbc31109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff944bbe727bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff944bbe727bc0 RCX: 0000000000000004
RDX: ffffffffbca13510 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff944bbe727c38 R08: ffffffffbca13508 R09: 00003ffffffff000
R10: ffff944c11a3b950 R11: 00007f6f744c4000 R12: ffff944c557d8000
R13: ffffffffbc3c0c06 R14: ffff944bbe727c00 R15: 0000000000020200
FS:  00007f6f745c1700(0000) GS:ffff944c48880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2267af2000 CR3: 00000000a8d64000 CR4: 00000000003607e0
Call Trace:
[<ffffffffbc27a3d0>] ? leave_mm+0x110/0x110
[<ffffffffbc27a3d0>] ? leave_mm+0x110/0x110
[<ffffffffbc27a3d0>] ? leave_mm+0x110/0x110
[<ffffffffbc3111af>] smp_call_function_single+0x5f/0xa0
[<ffffffffbc575465>] ? cpumask_next_and+0x35/0x50
[<ffffffffbc31175b>] smp_call_function_many+0x22b/0x270
[<ffffffffbc27a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbc27a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffbc3e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffbc3e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffbc3e87c2>] zap_page_range+0x142/0x190
[<ffffffffbc3e368d>] SyS_madvise+0x49d/0xac0
[<ffffffffbc3d612b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffbc974ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 f4 bc 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #10569
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb490aa2a
Build id: 3.10.0-862.14.4.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffb4879c90
Build id: 3.10.0-862.14.4.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffb4879c90
Build id: 3.10.0-862.14.4.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffb4879c90
Build id: 3.10.0-862.14.4.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffb490ab3f
Build id: 3.10.0-862.14.4.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb4b4edb5
Build id: 3.10.0-862.14.4.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb490b0eb
Build id: 3.10.0-862.14.4.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb4879e58
Build id: 3.10.0-862.14.4.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb4879ec8
Build id: 3.10.0-862.14.4.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffffb49c3487
Build id: 3.10.0-862.14.4.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffb49c4d85
Build id: 3.10.0-862.14.4.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffb49c7bd2
Build id: 3.10.0-862.14.4.el7.x86_64
0x142
13
? do_mmap
vmlinux 0xffffffffb49d145c
Build id: 3.10.0-862.14.4.el7.x86_64
0x3ac
14
SyS_madvise
vmlinux 0xffffffffb49c2add
Build id: 3.10.0-862.14.4.el7.x86_64
0x3cd
15
? vm_mmap_pgoff
vmlinux 0xffffffffb49b63ab
Build id: 3.10.0-862.14.4.el7.x86_64
0xeb
16
system_call_fastpath
vmlinux 0xffffffffb4f2579b
Build id: 3.10.0-862.14.4.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 65s! [java:49747]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_seq_midi snd_seq_midi_event lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer ppdev snd vmw_balloon soundcore pcspkr nfit sg joydev libnvdimm parport_pc parport vmw_vmci shpchp i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm mptspi serio_raw scsi_transport_spi e1000 mptscsih ata_piix libata i2c_core mptbase dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 49747 Comm: java Tainted: G        W    L ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9eb6aaa6cf10 ti: ffff9eb75fed4000 task.ti: ffff9eb75fed4000
RIP: 0010:[<ffffffffb490aa2a>]  [<ffffffffb490aa2a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9eb75fed7bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9eb75fed7bc0 RCX: 0000000000000001
RDX: ffffffffb50132b0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9eb75fed7c38 R08: ffffffffb50132a8 R09: 00003ffffffff000
R10: ffff9eb6baed9008 R11: 00007fd626f99000 R12: ffff9eb75fed7be8
R13: fffff1eed05abcb0 R14: 00007fd626f99000 R15: ffff9eb6aaa6cf10
FS:  00007fd627096700(0000) GS:ffff9eb76d640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fedacf99000 CR3: 00000003f97b6000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb4879c90>] ? leave_mm+0x110/0x110
[<ffffffffb4879c90>] ? leave_mm+0x110/0x110
[<ffffffffb4879c90>] ? leave_mm+0x110/0x110
[<ffffffffb490ab3f>] smp_call_function_single+0x5f/0xa0
[<ffffffffb4b4edb5>] ? cpumask_next_and+0x35/0x50
[<ffffffffb490b0eb>] smp_call_function_many+0x22b/0x270
[<ffffffffb4879e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb4879ec8>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb49c3487>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffb49c4d85>] tlb_finish_mmu+0x55/0x60
[<ffffffffb49c7bd2>] zap_page_range+0x142/0x190
[<ffffffffb49d145c>] ? do_mmap+0x3ac/0x4d0
[<ffffffffb49c2add>] SyS_madvise+0x3cd/0x9c0
[<ffffffffb49b63ab>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb4f2579b>] system_call_fastpath+0x22/0x27
Complete report #10440
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff81f16f72
Build id: 3.10.0-1127.19.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff81e7e730
Build id: 3.10.0-1127.19.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff81e7e730
Build id: 3.10.0-1127.19.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff81e7e730
Build id: 3.10.0-1127.19.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff81f1708f
Build id: 3.10.0-1127.19.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff82185005
Build id: 3.10.0-1127.19.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff81f1763b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff81e7e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff81e7e96b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6b
10
zap_page_range
vmlinux 0xffffffff81ff2ed8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xd8
11
? do_mmap
vmlinux 0xffffffff81ffcb28
Build id: 3.10.0-1127.19.1.el7.x86_64
0x378
12
SyS_madvise
vmlinux 0xffffffff81fed3ed
Build id: 3.10.0-1127.19.1.el7.x86_64
0x49d
13
? vm_mmap_pgoff
vmlinux 0xffffffff81fde63b
Build id: 3.10.0-1127.19.1.el7.x86_64
0xeb
14
system_call_fastpath
vmlinux 0xffffffff82592ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [ApplicationImpl:119897]
Modules linked in: tcp_lp rfcomm fuse veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter binfmt_misc des_generic md4 nls_utf8 cifs ccm dns_resolver overlay(T) bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev iosf_mbi vmw_balloon crc32_pclmul snd_ens1371 snd_rawmidi ghash_clmulni_intel btusb snd_ac97_codec btrtl btbcm btintel ac97_bus aesni_intel snd_seq snd_seq_device lrw bluetooth gf128mul glue_helper ablk_helper snd_pcm cryptd joydev pcspkr rfkill snd_timer snd sg vmw_vmci
soundcore i2c_piix4 parport_pc parport squashfs loop ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul nfit crct10dif_common crc32c_intel mptspi libnvdimm ata_piix scsi_transport_spi drm mptscsih mptbase libata serio_raw e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 119897 Comm: ApplicationImpl Kdump: loaded Tainted: G               ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffffa06b4c458000 ti: ffffa07097844000 task.ti: ffffa07097844000
RIP: 0010:[<ffffffff81f16f72>]  [<ffffffff81f16f72>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffffa07097847c30  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffffa07097847c00 RCX: 0000000000000001
RDX: ffffffff826136d0 RSI: 0000000000000080 RDI: 0000000000000286
RBP: ffffa07097847c80 R08: ffffffff826136c8 R09: 00007f54155d5000
R10: 0000000000000008 R11: 0000000000000206 R12: ffffffff81fcdd35
R13: ffffa07097847ba8 R14: ffffa07097847c88 R15: ffffa07097847c88
FS:  00007f54156d1700(0000) GS:ffffa072f3b40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1186c6c000 CR3: 00000008ad41a000 CR4: 00000000003607e0
Call Trace:
[<ffffffff81e7e730>] ? leave_mm+0x120/0x120
[<ffffffff81e7e730>] ? leave_mm+0x120/0x120
[<ffffffff81e7e730>] ? leave_mm+0x120/0x120
[<ffffffff81f1708f>] smp_call_function_single+0x5f/0xa0
[<ffffffff82185005>] ? cpumask_next_and+0x35/0x50
[<ffffffff81f1763b>] smp_call_function_many+0x22b/0x270
[<ffffffff81e7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81e7e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff81ff2ed8>] zap_page_range+0xd8/0x150
[<ffffffff81ffcb28>] ? do_mmap+0x378/0x530
[<ffffffff81fed3ed>] SyS_madvise+0x49d/0xab0
[<ffffffff81fde63b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff82592ed2>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 a0 10 b5 82 48 89 df e8 07 52 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #10004
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff84f111ba
Build id: 3.10.0-957.10.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff84e7a400
Build id: 3.10.0-957.10.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff84e7a400
Build id: 3.10.0-957.10.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff84e7a400
Build id: 3.10.0-957.10.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff84f112cf
Build id: 3.10.0-957.10.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff85175b35
Build id: 3.10.0-957.10.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff84f1187b
Build id: 3.10.0-957.10.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff84e7a5c8
Build id: 3.10.0-957.10.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff84e7a638
Build id: 3.10.0-957.10.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff84fe4507
Build id: 3.10.0-957.10.1.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff84fe5e05
Build id: 3.10.0-957.10.1.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff84fe8c52
Build id: 3.10.0-957.10.1.el7.x86_64
0x142
13
? do_mmap
vmlinux 0xffffffff84ff2d88
Build id: 3.10.0-957.10.1.el7.x86_64
0x378
14
SyS_madvise
vmlinux 0xffffffff84fe3b1d
Build id: 3.10.0-957.10.1.el7.x86_64
0x49d
15
? vm_mmap_pgoff
vmlinux 0xffffffff84fd65bb
Build id: 3.10.0-957.10.1.el7.x86_64
0xeb
16
system_call_fastpath
vmlinux 0xffffffff85575ddb
Build id: 3.10.0-957.10.1.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [coordinator-cos:2159]
Modules linked in: binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi vmwgfx floppy drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm vmxnet3 drm ata_piix mptspi scsi_transport_spi
mptscsih libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 2159 Comm: coordinator-cos Kdump: loaded Tainted: G             L ------------   3.10.0-957.10.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9fdd9b04b0c0 ti: ffff9fde22fe8000 task.ti: ffff9fde22fe8000
RIP: 0010:[<ffffffff84f111ba>]  [<ffffffff84f111ba>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9fde22febbf0  EFLAGS: 00000202
RAX: 00000000000008fb RBX: ffff9fdf856aa9c0 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000282
RBP: ffff9fde22febc38 R08: 0000000000000001 R09: 00003ffffffff000
R10: ffff9fdfa7e97948 R11: 00007fbfa2df7000 R12: ffff9fdf76b42cc0
R13: 0000000092bc4bf5 R14: 00007fbfa2df7000 R15: ffff9fdd9b04b0c0
FS:  00007fbfa2ef4700(0000) GS:ffff9fdfbfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbfa2edfaa8 CR3: 000000009f962000 CR4: 00000000001607f0
Call Trace:
[<ffffffff84e7a400>] ? leave_mm+0x110/0x110
[<ffffffff84e7a400>] ? leave_mm+0x110/0x110
[<ffffffff84e7a400>] ? leave_mm+0x110/0x110
[<ffffffff84f112cf>] smp_call_function_single+0x5f/0xa0
[<ffffffff85175b35>] ? cpumask_next_and+0x35/0x50
[<ffffffff84f1187b>] smp_call_function_many+0x22b/0x270
[<ffffffff84e7a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff84e7a638>] flush_tlb_mm_range+0x68/0x140
[<ffffffff84fe4507>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff84fe5e05>] tlb_finish_mmu+0x55/0x60
[<ffffffff84fe8c52>] zap_page_range+0x142/0x190
[<ffffffff84ff2d88>] ? do_mmap+0x378/0x530
[<ffffffff84fe3b1d>] SyS_madvise+0x49d/0xac0
[<ffffffff84fd65bb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff85575ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 a0 b9 b4 85 48 89 df e8 ca b7 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #8106
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff87b14a22
Build id: 3.10.0-1062.1.2.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff87a7d1d0
Build id: 3.10.0-1062.1.2.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff87a7d1d0
Build id: 3.10.0-1062.1.2.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff87a7d1d0
Build id: 3.10.0-1062.1.2.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff87b14b3f
Build id: 3.10.0-1062.1.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff87d7f475
Build id: 3.10.0-1062.1.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff87b150eb
Build id: 3.10.0-1062.1.2.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff87a7d398
Build id: 3.10.0-1062.1.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff87a7d40b
Build id: 3.10.0-1062.1.2.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff87bea166
Build id: 3.10.0-1062.1.2.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff87bebc2f
Build id: 3.10.0-1062.1.2.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff87bebd53
Build id: 3.10.0-1062.1.2.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff87beea5a
Build id: 3.10.0-1062.1.2.el7.x86_64
0xea
14
SyS_madvise
vmlinux 0xffffffff87be976d
Build id: 3.10.0-1062.1.2.el7.x86_64
0x49d
15
? vm_mmap_pgoff
vmlinux 0xffffffff87bdc14b
Build id: 3.10.0-1062.1.2.el7.x86_64
0xeb
16
system_call_fastpath
vmlinux 0xffffffff8818cede
Build id: 3.10.0-1062.1.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [java:26850]
Modules linked in: fuse cts rpcsec_gss_krb5 nfsv3 nfs fscache nfsd nfs_acl lockd grace xt_CHECKSUM tun devlink openafs(POE) xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter bridge stp llc overlay(T) nls_utf8 isofs ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ppdev vmw_balloon ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul
glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix mptspi scsi_transport_spi nfit libata drm libnvdimm crct10dif_pclmul crct10dif_common mptscsih mptbase serio_raw crc32c_intel vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 26850 Comm: java Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff98b04e1b9070 ti: ffff98b2f53b8000 task.ti: ffff98b2f53b8000
RIP: 0010:[<ffffffff87b14a22>]  [<ffffffff87b14a22>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff98b2f53bbbe0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff98b2f53bbbb0 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff98b2f53bbc30 R08: ffffffff88213540 R09: 00003ffffffff000
R10: ffff98b06f60daf8 R11: 00007f5795179000 R12: ffff98b43ffda000
R13: ffffffff87bc68f6 R14: ffff98b2f53bbbf0 R15: 0000000000020200
FS:  00007f5795276700(0000) GS:ffff98b43fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8ad2e43420 CR3: 00000000752dc000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff87a7d1d0>] ? leave_mm+0x110/0x110
[<ffffffff87a7d1d0>] ? leave_mm+0x110/0x110
[<ffffffff87a7d1d0>] ? leave_mm+0x110/0x110
[<ffffffff87b14b3f>] smp_call_function_single+0x5f/0xa0
[<ffffffff87d7f475>] ? cpumask_next_and+0x35/0x50
[<ffffffff87b150eb>] smp_call_function_many+0x22b/0x270
[<ffffffff87a7d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff87a7d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff87bea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff87bebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff87bebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffff87beea5a>] zap_page_range+0xea/0x130
[<ffffffff87be976d>] SyS_madvise+0x49d/0xac0
[<ffffffff87bdc14b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff8818cede>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 d7 74 88 48 89 df e8 d7 1a 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1417794
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa4b16562
Build id: 3.10.0-1160.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffa4a7e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffa4a7e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffa4a7e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffa4b1667f
Build id: 3.10.0-1160.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa4d85de5
Build id: 3.10.0-1160.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa4b16c2b
Build id: 3.10.0-1160.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa4a7e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa4a7e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa4bedd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffa4bef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffa4bef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffa4bf2dba
Build id: 3.10.0-1160.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffffa4adaaa8
Build id: 3.10.0-1160.el7.x86_64
0x58
15
SyS_madvise
vmlinux 0xffffffffa4bed2ad
Build id: 3.10.0-1160.el7.x86_64
0x49d
16
? __schedule
vmlinux 0xffffffffa5186e76
Build id: 3.10.0-1160.el7.x86_64
0x596
17
? __schedule
vmlinux 0xffffffffa5186e82
Build id: 3.10.0-1160.el7.x86_64
0x5a2
18
? __schedule
vmlinux 0xffffffffa5186e76
Build id: 3.10.0-1160.el7.x86_64
0x596
19
? __schedule
vmlinux 0xffffffffa5186e82
Build id: 3.10.0-1160.el7.x86_64
0x5a2
20
system_call_fastpath
vmlinux 0xffffffffa5193f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 60s! [containerd-shim:5009]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache veth vxlan ip6_udp_tunnel udp_tunnel ip6t_MASQUERADE nf_nat_masquerade_ipv6 xt_statistic xt_recent xt_nat xt_mark nf_tables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle xt_comment nf_conntrack_netlink nfnetlink xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev crc32_pclmul ghash_clmulni_intel aesni_intel snd_ens1371 snd_rawmidi snd_ac97_codec vmw_balloon ac97_bus lrw
snd_seq gf128mul glue_helper snd_seq_device ablk_helper cryptd snd_pcm joydev pcspkr snd_timer snd sg vmw_vmci i2c_piix4 soundcore parport_pc parport br_netfilter bridge stp llc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel ata_piix drm_kms_helper syscopyarea sysfillrect sysimgblt libata fb_sys_fops ttm drm mptspi e1000 scsi_transport_spi serio_raw mptscsih nfit libnvdimm mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 5009 Comm: containerd-shim Kdump: loaded Tainted: G               ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffffa04837af8000 ti: ffffa04830a1c000 task.ti: ffffa04830a1c000
RIP: 0010:[<ffffffffa4b16562>]  [<ffffffffa4b16562>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffffa04830a1fbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffffa04830a1fbb0 RCX: 0000000000000001
RDX: ffffffffa5213790 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffffa04830a1fc30 R08: ffffffffa5213788 R09: 00003ffffffff000
R10: ffffa04830960a20 R11: 0000000000000246 R12: ffffffffa4bc8aa6
R13: ffffa04830a1fbf0 R14: 0000000000020200 R15: 0000000000000000
FS:  000000c000288490(0000) GS:ffffa04875640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f71f10a8000 CR3: 00000000949b0000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffffa4a7e6a0>] ? leave_mm+0x120/0x120
[<ffffffffa4a7e6a0>] ? leave_mm+0x120/0x120
[<ffffffffa4a7e6a0>] ? leave_mm+0x120/0x120
[<ffffffffa4b1667f>] smp_call_function_single+0x5f/0xa0
[<ffffffffa4d85de5>] ? cpumask_next_and+0x35/0x50
[<ffffffffa4b16c2b>] smp_call_function_many+0x22b/0x270
[<ffffffffa4a7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa4a7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa4bedd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa4bef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa4bef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffa4bf2dba>] zap_page_range+0xea/0x130
[<ffffffffa4adaaa8>] ? wake_up_q+0x58/0x80
[<ffffffffa4bed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffffa5186e76>] ? __schedule+0x596/0x860
[<ffffffffa5186e82>] ? __schedule+0x5a2/0x860
[<ffffffffa5186e76>] ? __schedule+0x596/0x860
[<ffffffffa5186e82>] ? __schedule+0x5a2/0x860
[<ffffffffa5193f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 13 75 a5 48 89 df e8 f7 69 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1413653
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffae31db82
Build id: 3.10.0-1160.108.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffae281b10
Build id: 3.10.0-1160.108.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffffae281b10
Build id: 3.10.0-1160.108.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffffae281b10
Build id: 3.10.0-1160.108.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffffae31dc9f
Build id: 3.10.0-1160.108.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffae59c5d5
Build id: 3.10.0-1160.108.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffae31e25b
Build id: 3.10.0-1160.108.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffae281fa8
Build id: 3.10.0-1160.108.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffae28201b
Build id: 3.10.0-1160.108.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffae3fa07e
Build id: 3.10.0-1160.108.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffffae3fbbff
Build id: 3.10.0-1160.108.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffae3fbd33
Build id: 3.10.0-1160.108.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffae3ff27a
Build id: 3.10.0-1160.108.1.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffffae2e1118
Build id: 3.10.0-1160.108.1.el7.x86_64
0x58
15
? handle_mm_fault
vmlinux 0xffffffffae402503
Build id: 3.10.0-1160.108.1.el7.x86_64
0xdb3
16
SyS_madvise
vmlinux 0xffffffffae3f966d
Build id: 3.10.0-1160.108.1.el7.x86_64
0x49d
17
system_call_fastpath
vmlinux 0xffffffffae9c539a
Build id: 3.10.0-1160.108.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [code:3823]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event crc32_pclmul ghash_clmulni_intel ppdev snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper snd_seq_device cryptd snd_pcm joydev pcspkr snd_timer sg snd soundcore
vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common vmwgfx crc32c_intel ata_piix drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm e1000 mptspi scsi_transport_spi mptscsih mptbase nfit drm_panel_orientation_quirks libnvdimm libata dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 3823 Comm: code Kdump: loaded Not tainted 3.10.0-1160.108.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff99dbb97e4200 ti: ffff99dbbee58000 task.ti: ffff99dbbee58000
RIP: 0010:[<ffffffffae31db82>]  [<ffffffffae31db82>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff99dbbee5bbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff99dbbee5bbb0 RCX: 0000000000000001
RDX: ffffffffaea139d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff99dbbee5bc30 R08: ffffffffaea139c8 R09: 00003ffffffff000
R10: ffff99da6ffbd518 R11: 0000000000000202 R12: ffff99db00000000
R13: ffff99dbcaaa2e01 R14: 0000000000000000 R15: 0000000101ddce40
FS:  00007f1c0ebf4c00(0000) GS:ffff99dbf5640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000680009a5000 CR3: 00000001ebef0000 CR4: 00000000003407e0
Call Trace:
[<ffffffffae281b10>] ? leave_mm+0x130/0x130
[<ffffffffae281b10>] ? leave_mm+0x130/0x130
[<ffffffffae281b10>] ? leave_mm+0x130/0x130
[<ffffffffae31dc9f>] smp_call_function_single+0x5f/0xa0
[<ffffffffae59c5d5>] ? cpumask_next_and+0x35/0x50
[<ffffffffae31e25b>] smp_call_function_many+0x22b/0x270
[<ffffffffae281fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffae28201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffae3fa07e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffae3fbbff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffae3fbd33>] tlb_finish_mmu+0x23/0x40
[<ffffffffae3ff27a>] zap_page_range+0xea/0x140
[<ffffffffae2e1118>] ? wake_up_q+0x58/0x90
[<ffffffffae402503>] ? handle_mm_fault+0xdb3/0x1190
[<ffffffffae3f966d>] SyS_madvise+0x49d/0xac0
[<ffffffffae9c539a>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 80 19 f5 ae 48 89 df e8 97 67 29 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1199793
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8ed16f62
Build id: 3.10.0-1127.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff8ec7e730
Build id: 3.10.0-1127.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff8ec7e730
Build id: 3.10.0-1127.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff8ec7e730
Build id: 3.10.0-1127.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff8ed1707f
Build id: 3.10.0-1127.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff8ef84ee5
Build id: 3.10.0-1127.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff8ed1762b
Build id: 3.10.0-1127.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8ec7e8f8
Build id: 3.10.0-1127.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8ec7e96b
Build id: 3.10.0-1127.el7.x86_64
0x6b
10
zap_page_range
vmlinux 0xffffffff8edf2e58
Build id: 3.10.0-1127.el7.x86_64
0xd8
11
? wake_up_q
vmlinux 0xffffffff8ecdb948
Build id: 3.10.0-1127.el7.x86_64
0x58
12
SyS_madvise
vmlinux 0xffffffff8eded36d
Build id: 3.10.0-1127.el7.x86_64
0x49d
13
? wake_up_state
vmlinux 0xffffffff8ecdb990
Build id: 3.10.0-1127.el7.x86_64
0x20
14
system_call_fastpath
vmlinux 0xffffffff8f392ed2
Build id: 3.10.0-1127.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [node_exporter:1684]
Modules linked in: binfmt_misc vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd sb_edac soundcore iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd sg pcspkr vmw_balloon joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper ata_piix syscopyarea sysfillrect sysimgblt fb_sys_fops ttm libata nfit drm mptspi scsi_transport_spi mptscsih mptbase crct10dif_pclmul crct10dif_common libnvdimm crc32c_intel e1000 serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 1684 Comm: node_exporter Kdump: loaded Tainted: G        W    L ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff96ed3a94e2a0 ti: ffff96ed3a00c000 task.ti: ffff96ed3a00c000
RIP: 0010:[<ffffffff8ed16f62>]  [<ffffffff8ed16f62>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff96ed3a00fc30  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff96ed3a00fc00 RCX: 0000000000000001
RDX: ffffffff8f413610 RSI: 0000000000000080 RDI: 0000000000000286
RBP: ffff96ed3a00fc80 R08: ffffffff8f413608 R09: 00003ffffffff000
R10: ffff96edb4c7eaf8 R11: 0000000000000202 R12: 0000000000000001
R13: ffff96edb96dacc0 R14: ffff96edb96dad40 R15: ffff96ed3a94e308
FS:  000000c00004ac90(0000) GS:ffff96edb96c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0004ad000 CR3: 00000000b118c000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8ec7e730>] ? leave_mm+0x120/0x120
[<ffffffff8ec7e730>] ? leave_mm+0x120/0x120
[<ffffffff8ec7e730>] ? leave_mm+0x120/0x120
[<ffffffff8ed1707f>] smp_call_function_single+0x5f/0xa0
[<ffffffff8ef84ee5>] ? cpumask_next_and+0x35/0x50
[<ffffffff8ed1762b>] smp_call_function_many+0x22b/0x270
[<ffffffff8ec7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8ec7e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8edf2e58>] zap_page_range+0xd8/0x150
[<ffffffff8ecdb948>] ? wake_up_q+0x58/0x80
[<ffffffff8eded36d>] SyS_madvise+0x49d/0xab0
[<ffffffff8ecdb990>] ? wake_up_state+0x20/0x20
[<ffffffff8f392ed2>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 0f 95 8f 48 89 df e8 17 51 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1078820
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff90116562
Build id: 3.10.0-1160.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff9007e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9007e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9007e6a0
Build id: 3.10.0-1160.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff9011667f
Build id: 3.10.0-1160.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff90385de5
Build id: 3.10.0-1160.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff90116c2b
Build id: 3.10.0-1160.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9007e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9007e8db
Build id: 3.10.0-1160.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff901edd06
Build id: 3.10.0-1160.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff901ef7cf
Build id: 3.10.0-1160.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff901ef8f3
Build id: 3.10.0-1160.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff901f2dba
Build id: 3.10.0-1160.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffff900daaa8
Build id: 3.10.0-1160.el7.x86_64
0x58
15
SyS_madvise
vmlinux 0xffffffff901ed2ad
Build id: 3.10.0-1160.el7.x86_64
0x49d
16
? __schedule
vmlinux 0xffffffff90786e76
Build id: 3.10.0-1160.el7.x86_64
0x596
17
? __schedule
vmlinux 0xffffffff90786e82
Build id: 3.10.0-1160.el7.x86_64
0x5a2
18
? __schedule
vmlinux 0xffffffff90786e76
Build id: 3.10.0-1160.el7.x86_64
0x596
19
? __schedule
vmlinux 0xffffffff90786e82
Build id: 3.10.0-1160.el7.x86_64
0x5a2
20
system_call_fastpath
vmlinux 0xffffffff90793f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [containerd-shim:24953]
Modules linked in: veth xt_TCPMSS ip_set_hash_ip ip_set_hash_net nf_tables_ipv4 nf_tables nfnetlink_cttimeout xt_set ip_set_hash_ipportnet openvswitch ip_set_hash_ipportip ip_set_bitmap_port ip_set_hash_ipport dummy ip_set ip6table_mangle ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_conntrack ipt_REJECT nf_reject_ipv4 tun ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel vmw_balloon sg aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr parport_pc i2c_piix4 vmw_vmci parport
br_netfilter bridge stp llc ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs geneve ip6_udp_tunnel udp_tunnel iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod ahci crc_t10dif crct10dif_generic ata_piix nfit libahci drm libnvdimm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 24953 Comm: containerd-shim Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8893abf7d280 ti: ffff8890cf250000 task.ti: ffff8890cf250000
RIP: 0010:[<ffffffff90116562>]  [<ffffffff90116562>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8890cf253be0  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff8890cf253bb0 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff8890cf253c30 R08: ffffffff90813c80 R09: ffffed72442cf300
R10: 000000000000154c R11: 0000000000000202 R12: ffffffff901c8aa6
R13: ffff8890cf253bf0 R14: 0000000000020200 R15: 0000000000000000
FS:  000000c000040490(0000) GS:ffff8895ffc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000137010 CR3: 0000000111d68000 CR4: 00000000007607f0
PKRU: 55555554
Call Trace:
[<ffffffff9007e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9007e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9007e6a0>] ? leave_mm+0x120/0x120
[<ffffffff9011667f>] smp_call_function_single+0x5f/0xa0
[<ffffffff90385de5>] ? cpumask_next_and+0x35/0x50
[<ffffffff90116c2b>] smp_call_function_many+0x22b/0x270
[<ffffffff9007e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9007e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff901edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff901ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff901ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff901f2dba>] zap_page_range+0xea/0x130
[<ffffffff900daaa8>] ? wake_up_q+0x58/0x80
[<ffffffff901ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffff90786e76>] ? __schedule+0x596/0x860
[<ffffffff90786e82>] ? __schedule+0x5a2/0x860
[<ffffffff90786e76>] ? __schedule+0x596/0x860
[<ffffffff90786e82>] ? __schedule+0x5a2/0x860
[<ffffffff90793f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 13 d5 90 48 89 df e8 f7 69 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1045631
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa6b1dba2
Build id: 3.10.0-1160.118.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffa6a81b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffffa6a81b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffffa6a81b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffffa6b1dcbf
Build id: 3.10.0-1160.118.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa6d9c615
Build id: 3.10.0-1160.118.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa6b1e27b
Build id: 3.10.0-1160.118.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa6a81fa8
Build id: 3.10.0-1160.118.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa6a8201b
Build id: 3.10.0-1160.118.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa6bfa0ce
Build id: 3.10.0-1160.118.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffffa6bfbc4f
Build id: 3.10.0-1160.118.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffa6bfbd83
Build id: 3.10.0-1160.118.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffa6bff2ca
Build id: 3.10.0-1160.118.1.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffffa6ae1118
Build id: 3.10.0-1160.118.1.el7.x86_64
0x58
15
SyS_madvise
vmlinux 0xffffffffa6bf96bd
Build id: 3.10.0-1160.118.1.el7.x86_64
0x49d
16
system_call_fastpath
vmlinux 0xffffffffa71c539a
Build id: 3.10.0-1160.118.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [docker-containe:15950]
Modules linked in: binfmt_misc cts 8021q garp mrp ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache falcon_lsm_serviceable(PE) falcon_nf_netcontain(E) falcon_kal(E) falcon_lsm_pinned_17005(E) nf_nat_tftp nf_conntrack_tftp nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) tun vmw_vsock_vmci_transport
vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper nfit ata_piix syscopyarea sysfillrect sysimgblt fb_sys_fops ttm libata libnvdimm drm crct10dif_pclmul crct10dif_common mptspi crc32c_intel serio_raw scsi_transport_spi mptscsih vmxnet3 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 15950 Comm: docker-containe Kdump: loaded Tainted: P            EL ------------ T 3.10.0-1160.118.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff93b1d4971080 ti: ffff93b1d4fac000 task.ti: ffff93b1d4fac000
RIP: 0010:[<ffffffffa6b1dba2>]  [<ffffffffa6b1dba2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff93b1d4fafbe0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff93b1d4fafbb0 RCX: 0000000000000830
RDX: 0000000000000004 RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff93b1d4fafc30 R08: ffffffffa7214140 R09: 00003ffffffff000
R10: ffff93b1d513d510 R11: 0000000000000246 R12: ffff93b1d4971080
R13: ffffffffa6aeab04 R14: ffff93b1d4fafb80 R15: ffff93b1d4fafc60
FS:  00007fd2a642a700(0000) GS:ffff93b1ffc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000db6578 CR3: 00000002148ac000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa6a81b10>] ? leave_mm+0x130/0x130
[<ffffffffa6a81b10>] ? leave_mm+0x130/0x130
[<ffffffffa6a81b10>] ? leave_mm+0x130/0x130
[<ffffffffa6b1dcbf>] smp_call_function_single+0x5f/0xa0
[<ffffffffa6d9c615>] ? cpumask_next_and+0x35/0x50
[<ffffffffa6b1e27b>] smp_call_function_many+0x22b/0x270
[<ffffffffa6a81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa6a8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa6bfa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa6bfbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa6bfbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffffa6bff2ca>] zap_page_range+0xea/0x140
[<ffffffffa6ae1118>] ? wake_up_q+0x58/0x90
[<ffffffffa6bf96bd>] SyS_madvise+0x49d/0xac0
[<ffffffffa71c539a>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 c0 19 75 a7 48 89 df e8 b7 67 29 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #1006791
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb191dba2
Build id: 3.10.0-1160.118.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffb1881b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
3
? leave_mm
vmlinux 0xffffffffb1881b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
4
? leave_mm
vmlinux 0xffffffffb1881b10
Build id: 3.10.0-1160.118.1.el7.x86_64
0x130
5
smp_call_function_single
vmlinux 0xffffffffb191dcbf
Build id: 3.10.0-1160.118.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb1b9c615
Build id: 3.10.0-1160.118.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb191e27b
Build id: 3.10.0-1160.118.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb1881fa8
Build id: 3.10.0-1160.118.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb188201b
Build id: 3.10.0-1160.118.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb19fa0ce
Build id: 3.10.0-1160.118.1.el7.x86_64
0xae
11
arch_tlb_finish_mmu
vmlinux 0xffffffffb19fbc4f
Build id: 3.10.0-1160.118.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffb19fbd83
Build id: 3.10.0-1160.118.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffb19ff2ca
Build id: 3.10.0-1160.118.1.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffffb18e1118
Build id: 3.10.0-1160.118.1.el7.x86_64
0x58
15
SyS_madvise
vmlinux 0xffffffffb19f96bd
Build id: 3.10.0-1160.118.1.el7.x86_64
0x49d
16
? pick_next_entity
vmlinux 0xffffffffb18e8699
Build id: 3.10.0-1160.118.1.el7.x86_64
0xa9
17
? set_next_entity
vmlinux 0xffffffffb18e918c
Build id: 3.10.0-1160.118.1.el7.x86_64
0x3c
18
system_call_fastpath
vmlinux 0xffffffffb1fc539a
Build id: 3.10.0-1160.118.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [containerd-shim:30877]
Modules linked in: veth xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_addrtype br_netfilter bridge stp llc iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon sg pcspkr joydev vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm nfit sd_mod libahci ata_piix crc_t10dif crct10dif_generic libnvdimm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash
dm_log dm_mod fuse
CPU: 3 PID: 30877 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.118.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9d13a3cc6300 ti: ffff9d13a1128000 task.ti: ffff9d13a1128000
RIP: 0010:[<ffffffffb191dba2>]  [<ffffffffb191dba2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9d13a112bbe0  EFLAGS: 00000202
RAX: 0000000000000006 RBX: ffff9d13a112bbb0 RCX: 0000000000000830
RDX: 0000000000000006 RSI: 0000000000000006 RDI: 0000000000000282
RBP: ffff9d13a112bc30 R08: ffffffffb2014640 R09: 00003ffffffff000
R10: ffff9d13ea810e58 R11: 0000000000000246 R12: ffff9d146a6730c0
R13: ffff9d146a6730c0 R14: 000000000000023a R15: ffff9d11c8faa000
FS:  000000c000200490(0000) GS:ffff9d147fcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00021b010 CR3: 00000004aa8c0000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffb1881b10>] ? leave_mm+0x130/0x130
[<ffffffffb1881b10>] ? leave_mm+0x130/0x130
[<ffffffffb1881b10>] ? leave_mm+0x130/0x130
[<ffffffffb191dcbf>] smp_call_function_single+0x5f/0xa0
[<ffffffffb1b9c615>] ? cpumask_next_and+0x35/0x50
[<ffffffffb191e27b>] smp_call_function_many+0x22b/0x270
[<ffffffffb1881fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb188201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb19fa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffb19fbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb19fbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffffb19ff2ca>] zap_page_range+0xea/0x140
[<ffffffffb18e1118>] ? wake_up_q+0x58/0x90
[<ffffffffb19f96bd>] SyS_madvise+0x49d/0xac0
[<ffffffffb18e8699>] ? pick_next_entity+0xa9/0x190
[<ffffffffb18e918c>] ? set_next_entity+0x3c/0x100
[<ffffffffb1fc539a>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 c0 19 55 b2 48 89 df e8 b7 67 29 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #920433
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff8851109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8847a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff8847a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff8847a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff885111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff88775465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff8851175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff8847a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8847a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff885e4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff885e5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff885e87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffff884d6768
Build id: 3.10.0-957.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffff885e368d
Build id: 3.10.0-957.el7.x86_64
0x49d
15
tracesys
vmlinux 0xffffffff88b7505b
Build id: 3.10.0-957.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [kube-apiserver:54766]
Modules linked in: ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_CHECKSUM tun ebtable_filter ebtables ip6table_filter devlink ip6_tables xt_multiport xt_nat veth nf_tables(T) vxlan ip6_udp_tunnel udp_tunnel ip_set_hash_ipportip ip_set_bitmap_port ip_set_hash_ipportnet ip_set_hash_ipport dummy iptable_mangle xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_nat_ipv4 ceph libceph dns_resolver overlay(T) toa(OE) vmw_vsock_vmci_transport vsock bnep sunrpc dm_mirror dm_region_hash dm_log dm_mod crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev btusb btrtl btbcm btintel bluetooth rfkill snd_seq_midi snd_seq_midi_event snd_ens1371
snd_rawmidi snd_ac97_codec ac97_bus parport_pc parport snd_seq snd_seq_device snd_pcm snd_timer snd soundcore sg vmw_vmci i2c_piix4 nf_conntrack_ipv4 nf_defrag_ipv4 ipip tunnel4 ip_tunnel ipt_REJECT nf_reject_ipv4 ipt_rpfilter xt_set ip_set nfnetlink ip_vs_ftp nf_nat ip_vs_sed ip_vs_nq ip_vs_dh ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs_lblcr ip_vs_lblc ip_vs_lc ip_vs nf_conntrack br_netfilter bridge stp llc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod cdrom crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw nfit libnvdimm ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix e1000 libata mptspi scsi_transport_spi drm_panel_orientation_quirks mptscsih mptbase
CPU: 3 PID: 54766 Comm: kube-apiserver Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff88caf3326180 ti: ffff88ca08b98000 task.ti: ffff88ca08b98000
RIP: 0010:[<ffffffff8851109a>]  [<ffffffff8851109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff88ca08b9bbf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88ca08b9bbc0 RCX: 0000000000000003
RDX: ffffffff88c13290 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88ca08b9bc38 R08: ffffffff88c13288 R09: 00003ffffffff000
R10: ffff88c9fe802948 R11: 0000000000000246 R12: ffff88caf965ab80
R13: ffff88ca616c90d0 R14: ffff88caf965ab80 R15: 00000000060b69cb
FS:  000000c000068c90(0000) GS:ffff88caf96c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000c4a020 CR3: 0000000044be0000 CR4: 0000000000360fe0
Call Trace:
[<ffffffff8847a3d0>] ? leave_mm+0x110/0x110
[<ffffffff8847a3d0>] ? leave_mm+0x110/0x110
[<ffffffff8847a3d0>] ? leave_mm+0x110/0x110
[<ffffffff885111af>] smp_call_function_single+0x5f/0xa0
[<ffffffff88775465>] ? cpumask_next_and+0x35/0x50
[<ffffffff8851175b>] smp_call_function_many+0x22b/0x270
[<ffffffff8847a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8847a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff885e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff885e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff885e87c2>] zap_page_range+0x142/0x190
[<ffffffff884d6768>] ? wake_up_q+0x58/0x80
[<ffffffff885e368d>] SyS_madvise+0x49d/0xac0
[<ffffffff88b7505b>] tracesys+0xa3/0xc9
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 14 89 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #875452
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff85b1109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff85a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff85a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff85a7a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff85b111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff85d75465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff85b1175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff85a7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff85a7a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff85be4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff85be5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff85be87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffff85ad6768
Build id: 3.10.0-957.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffff85be368d
Build id: 3.10.0-957.el7.x86_64
0x49d
15
? update_curr
vmlinux 0xffffffff85adfff6
Build id: 3.10.0-957.el7.x86_64
0x86
16
system_call_fastpath
vmlinux 0xffffffff86174ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [containerd-shim:20456]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon
pcspkr joydev snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer sg snd soundcore parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_pclmul crct10dif_common crc32c_intel ata_generic pata_acpi serio_raw vmwgfx e1000 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix scsi_transport_spi mptscsih libata mptbase nfit drm_panel_orientation_quirks libnvdimm
CPU: 1 PID: 20456 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9415f2b330c0 ti: ffff9415f5bb0000 task.ti: ffff9415f5bb0000
RIP: 0010:[<ffffffff85b1109a>]  [<ffffffff85b1109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9415f5bb3bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9415f5bb3bc0 RCX: 0000000000000001
RDX: ffffffff86212d90 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9415f5bb3c38 R08: ffffffff86212d88 R09: fffff88240a785c0
R10: 000000000000312b R11: 0000000000000202 R12: ffff94163ffd8000
R13: ffffffff85bc0c06 R14: ffff9415f5bb3c00 R15: 0000000000020200
FS:  000000c000080890(0000) GS:ffff941635640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f25e7e42a80 CR3: 00000001f9a12000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff85a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff85a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff85a7a3d0>] ? leave_mm+0x110/0x110
[<ffffffff85b111af>] smp_call_function_single+0x5f/0xa0
[<ffffffff85d75465>] ? cpumask_next_and+0x35/0x50
[<ffffffff85b1175b>] smp_call_function_many+0x22b/0x270
[<ffffffff85a7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff85a7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff85be4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff85be5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff85be87c2>] zap_page_range+0x142/0x190
[<ffffffff85ad6768>] ? wake_up_q+0x58/0x80
[<ffffffff85be368d>] SyS_madvise+0x49d/0xac0
[<ffffffff85adfff6>] ? update_curr+0x86/0x1e0
[<ffffffff86174ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 74 86 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #851668
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb730779a
Build id: 3.10.0-862.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffb7276e50
Build id: 3.10.0-862.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffb7276e50
Build id: 3.10.0-862.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffb7276e50
Build id: 3.10.0-862.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffb73078af
Build id: 3.10.0-862.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb754a275
Build id: 3.10.0-862.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb7307e5b
Build id: 3.10.0-862.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb7277018
Build id: 3.10.0-862.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb7277088
Build id: 3.10.0-862.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffffb73c0497
Build id: 3.10.0-862.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffb73c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffb73c4a02
Build id: 3.10.0-862.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffffb72cee38
Build id: 3.10.0-862.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffffb73bfafd
Build id: 3.10.0-862.el7.x86_64
0x3cd
15
? update_curr
vmlinux 0xffffffffb72d80a6
Build id: 3.10.0-862.el7.x86_64
0x86
16
system_call_fastpath
vmlinux 0xffffffffb791f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 24s! [containerd-shim:3019]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event crc32_pclmul ghash_clmulni_intel snd_ens1371 aesni_intel snd_rawmidi lrw snd_ac97_codec gf128mul glue_helper ablk_helper ac97_bus cryptd snd_seq snd_seq_device snd_pcm snd_timer snd soundcore ppdev sg vmw_balloon pcspkr joydev nfit parport_pc libnvdimm parport i2c_piix4 vmw_vmci shpchp ip_tables xfs libcrc32c sr_mod
sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common mptspi libata crc32c_intel scsi_transport_spi e1000 mptscsih serio_raw mptbase i2c_core
CPU: 11 PID: 3019 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9d1138bc1fa0 ti: ffff9d185aa60000 task.ti: ffff9d185aa60000
RIP: 0010:[<ffffffffb730779a>]  [<ffffffffb730779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9d185aa63bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9d185aa63bc0 RCX: 0000000000000009
RDX: ffffffffb7a14570 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff9d185aa63c38 R08: ffffffffb7a14568 R09: fffff6c72009f5c0
R10: fffff6c72009f5c0 R11: 0000000000000202 R12: ffff9d1138bc1fa0
R13: ffff9d1138bc1fa0 R14: ffff9d185aa63ba0 R15: 0000000000000001
FS:  000000c000200090(0000) GS:ffff9d18bd8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc1e24dc580 CR3: 00000007d0fea000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffb7276e50>] ? leave_mm+0x110/0x110
[<ffffffffb7276e50>] ? leave_mm+0x110/0x110
[<ffffffffb7276e50>] ? leave_mm+0x110/0x110
[<ffffffffb73078af>] smp_call_function_single+0x5f/0xa0
[<ffffffffb754a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffb7307e5b>] smp_call_function_many+0x22b/0x270
[<ffffffffb7277018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb7277088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb73c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffb73c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffb73c4a02>] zap_page_range+0x142/0x190
[<ffffffffb72cee38>] ? wake_up_q+0x58/0x80
[<ffffffffb73bfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffffb72d80a6>] ? update_curr+0x86/0x1e0
[<ffffffffb791f7d5>] system_call_fastpath+0x1c/0x21
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 f3 b7 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #820688
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffb991109a
Build id: 3.10.0-957.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffb987a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffb987a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffb987a3d0
Build id: 3.10.0-957.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffb99111af
Build id: 3.10.0-957.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffb9b75465
Build id: 3.10.0-957.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffb991175b
Build id: 3.10.0-957.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffb987a598
Build id: 3.10.0-957.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffb987a608
Build id: 3.10.0-957.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffffb99e4077
Build id: 3.10.0-957.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffb99e5975
Build id: 3.10.0-957.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffb99e87c2
Build id: 3.10.0-957.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffffb98d6768
Build id: 3.10.0-957.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffffb99e368d
Build id: 3.10.0-957.el7.x86_64
0x49d
15
tracesys
vmlinux 0xffffffffb9f7505b
Build id: 3.10.0-957.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [minio:112566]
Modules linked in: nfnetlink_queue nfnetlink_log bluetooth rfkill xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq joydev snd_seq_device snd_pcm pcspkr snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic
pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm e1000 ata_piix mptspi scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 112566 Comm: minio Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff8a6af8281040 ti: ffff8a6d7d774000 task.ti: ffff8a6d7d774000
RIP: 0010:[<ffffffffb991109a>]  [<ffffffffb991109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8a6d7d777bf0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8a6d7d777b80 RCX: 0000000000000004
RDX: ffffffffba013510 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8a6d7d777c38 R08: ffffffffba013508 R09: 00003ffffffff000
R10: ffff8a6a2d1540d8 R11: 0000000000000202 R12: 0000000000000010
R13: ffffffffb98611c6 R14: ffffffffffffff10 R15: 0000000000000282
FS:  000000c00137a890(0000) GS:ffff8a6dc5140000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fff735538c8 CR3: 00000003f9eb4000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb987a3d0>] ? leave_mm+0x110/0x110
[<ffffffffb987a3d0>] ? leave_mm+0x110/0x110
[<ffffffffb987a3d0>] ? leave_mm+0x110/0x110
[<ffffffffb99111af>] smp_call_function_single+0x5f/0xa0
[<ffffffffb9b75465>] ? cpumask_next_and+0x35/0x50
[<ffffffffb991175b>] smp_call_function_many+0x22b/0x270
[<ffffffffb987a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb987a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb99e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffb99e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffb99e87c2>] zap_page_range+0x142/0x190
[<ffffffffb98d6768>] ? wake_up_q+0x58/0x80
[<ffffffffb99e368d>] SyS_madvise+0x49d/0xac0
[<ffffffffb9f7505b>] tracesys+0xa3/0xc9
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 54 ba 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #715466
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa3116a32
Build id: 3.10.0-1160.49.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffa307ddd0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffa307ddd0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffa307ddd0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffa3116b4f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa3386a45
Build id: 3.10.0-1160.49.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa31170fb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa307e258
Build id: 3.10.0-1160.49.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa307e2cb
Build id: 3.10.0-1160.49.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffa31ee676
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffa31f013f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffa31f0263
Build id: 3.10.0-1160.49.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffa31f372a
Build id: 3.10.0-1160.49.1.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffffa30dada8
Build id: 3.10.0-1160.49.1.el7.x86_64
0x58
15
SyS_madvise
vmlinux 0xffffffffa31edc1d
Build id: 3.10.0-1160.49.1.el7.x86_64
0x49d
16
? __switch_to
vmlinux 0xffffffffa302b59e
Build id: 3.10.0-1160.49.1.el7.x86_64
0xce
17
system_call_fastpath
vmlinux 0xffffffffa3795f92
Build id: 3.10.0-1160.49.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [pushgateway:25938]
Modules linked in: xt_comment xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink tun vmw_vsock_vmci_transport vsock sunrpc vfat fat sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc vmw_vmci i2c_piix4 parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common mptspi libata crc32c_intel scsi_transport_spi mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ipmi_msghandler]
CPU: 1 PID: 25938 Comm: pushgateway Tainted: G        W    L ------------   3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.0.B64.1704120155 04/12/2017
task: ffff9d12b65ce300 ti: ffff9d0de2008000 task.ti: ffff9d0de2008000
RIP: 0010:[<ffffffffa3116a32>]  [<ffffffffa3116a32>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9d0de200bbe0  EFLAGS: 00000202
RAX: 0000000000000007 RBX: ffff9d0de200bbb0 RCX: 0000000000000039
RDX: 000000000000007f RSI: 0000000000000007 RDI: 0000000000000282
RBP: ffff9d0de200bc30 R08: ffffffffa3813cc0 R09: 00003ffffffff000
R10: ffff9d133ca29e58 R11: 0000000000000202 R12: ffff9d0de200bb60
R13: 0000000000000000 R14: ffff9d157fc9acc0 R15: ffffffffa30d3c1a
FS:  0000000001051b70(0063) GS:ffff9d157fc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa751568e50 CR3: 0000000095d24000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa307ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa307ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa307ddd0>] ? leave_mm+0x120/0x120
[<ffffffffa3116b4f>] smp_call_function_single+0x5f/0xa0
[<ffffffffa3386a45>] ? cpumask_next_and+0x35/0x50
[<ffffffffa31170fb>] smp_call_function_many+0x22b/0x270
[<ffffffffa307e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa307e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa31ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa31f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa31f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffa31f372a>] zap_page_range+0xea/0x130
[<ffffffffa30dada8>] ? wake_up_q+0x58/0x80
[<ffffffffa31edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffa302b59e>] ? __switch_to+0xce/0x580
[<ffffffffa3795f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 60 15 d5 a3 48 89 df e8 87 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #639658
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f932a
Build id: 3.10.0-693.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff8106e4a0
Build id: 3.10.0-693.el7.x86_64
0x70
3
? leave_mm
vmlinux 0xffffffff8106e4a0
Build id: 3.10.0-693.el7.x86_64
0x70
4
? leave_mm
vmlinux 0xffffffff8106e4a0
Build id: 3.10.0-693.el7.x86_64
0x70
5
smp_call_function_single
vmlinux 0xffffffff810f942f
Build id: 3.10.0-693.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813226f5
Build id: 3.10.0-693.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f99c3
Build id: 3.10.0-693.el7.x86_64
0x223
8
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
10
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
11
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff811b08fa
Build id: 3.10.0-693.el7.x86_64
0x13a
13
? wake_up_q
vmlinux 0xffffffff810c47c8
Build id: 3.10.0-693.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffff811ac71d
Build id: 3.10.0-693.el7.x86_64
0x3cd
15
? SyS_futex
vmlinux 0xffffffff810f8040
Build id: 3.10.0-693.el7.x86_64
0x80
16
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [containerd:8941]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) tcp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg nfit vmw_vmci libnvdimm i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm ata_piix drm mptspi libata scsi_transport_spi crct10dif_pclmul mptscsih crct10dif_common crc32c_intel mptbase e1000 serio_raw i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 8941 Comm: containerd Tainted: G               ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8801b0e33f40 ti: ffff8801bb278000 task.ti: ffff8801bb278000
RIP: 0010:[<ffffffff810f932a>]  [<ffffffff810f932a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff8801bb27bc20  EFLAGS: 00000202
RAX: 0000000000000040 RBX: ffff88017c9e5000 RCX: 0000000000000005
RDX: 0000000000000040 RSI: 0000000000000000 RDI: 0000000000000286
RBP: ffff8801bb27bc70 R08: ffffffff816cee28 R09: 000000c000bce000
R10: 00000000000001e7 R11: 0000000000000202 R12: ffff88017fc19000
R13: 000000005076d999 R14: 0000000000000000 R15: 0000000000000000
FS:  00007fa40ffff700(0000) GS:ffff8802367c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa43c5b2098 CR3: 0000000206367000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff8106e4a0 ffff8801bb27bcf8
0000000000000003 000000005076d999 ffff8801bb27bc88 0000000000000004
ffffffff8106e4a0 ffffffff8106e4a0 ffff8801bb27bca0 ffffffff810f942f
Call Trace:
[<ffffffff8106e4a0>] ? leave_mm+0x70/0x70
[<ffffffff8106e4a0>] ? leave_mm+0x70/0x70
[<ffffffff8106e4a0>] ? leave_mm+0x70/0x70
[<ffffffff810f942f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813226f5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f99c3>] smp_call_function_many+0x223/0x260
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b08fa>] zap_page_range+0x13a/0x180
[<ffffffff810c47c8>] ? wake_up_q+0x58/0x80
[<ffffffff811ac71d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff810f8040>] ? SyS_futex+0x80/0x180
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 00 78 01 00 48 89 de 48 03 14 c5 e0 fd b0 81 48 89 df e8 7a da 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #377973
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff93116a12
Build id: 3.10.0-1160.36.2.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff9307ddd0
Build id: 3.10.0-1160.36.2.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9307ddd0
Build id: 3.10.0-1160.36.2.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9307ddd0
Build id: 3.10.0-1160.36.2.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff93116b2f
Build id: 3.10.0-1160.36.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff933869f5
Build id: 3.10.0-1160.36.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff931170db
Build id: 3.10.0-1160.36.2.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9307e258
Build id: 3.10.0-1160.36.2.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9307e2cb
Build id: 3.10.0-1160.36.2.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff931ee656
Build id: 3.10.0-1160.36.2.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffff931f011f
Build id: 3.10.0-1160.36.2.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffff931f0243
Build id: 3.10.0-1160.36.2.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffff931f370a
Build id: 3.10.0-1160.36.2.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffff930dad88
Build id: 3.10.0-1160.36.2.el7.x86_64
0x58
15
SyS_madvise
vmlinux 0xffffffff931edbfd
Build id: 3.10.0-1160.36.2.el7.x86_64
0x49d
16
system_call_fastpath
vmlinux 0xffffffff93795f92
Build id: 3.10.0-1160.36.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [docker-proxy:2758]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi ext4 snd_ac97_codec ac97_bus mbcache snd_seq jbd2 snd_seq_device snd_pcm snd_timer crc32_pclmul ghash_clmulni_intel snd aesni_intel
lrw gf128mul glue_helper ppdev soundcore ablk_helper cryptd sg vmw_balloon joydev pcspkr parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper nfit syscopyarea sysfillrect sysimgblt fb_sys_fops libnvdimm ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel mptspi scsi_transport_spi mptscsih mptbase libata serio_raw e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 2758 Comm: docker-proxy Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.36.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff8aebde783180 ti: ffff8ae7f9388000 task.ti: ffff8ae7f9388000
RIP: 0010:[<ffffffff93116a12>]  [<ffffffff93116a12>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8ae7f938bbe0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8ae7f938bbb0 RCX: 0000000000000001
RDX: ffffffff938137d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8ae7f938bc30 R08: ffffffff938137c8 R09: 00003ffffffff000
R10: ffff8aebdee495f0 R11: 0000000000000202 R12: ffff8aebf7ef8b40
R13: ffff8aebf7ef8b40 R14: 00000000000003ad R15: ffff8aec05650200
FS:  00007f64d8de0700(0000) GS:ffff8aec0b440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000141010 CR3: 000000049ecb4000 CR4: 0000000000360fe0
Call Trace:
[<ffffffff9307ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9307ddd0>] ? leave_mm+0x120/0x120
[<ffffffff9307ddd0>] ? leave_mm+0x120/0x120
[<ffffffff93116b2f>] smp_call_function_single+0x5f/0xa0
[<ffffffff933869f5>] ? cpumask_next_and+0x35/0x50
[<ffffffff931170db>] smp_call_function_many+0x22b/0x270
[<ffffffff9307e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9307e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff931ee656>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff931f011f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff931f0243>] tlb_finish_mmu+0x23/0x30
[<ffffffff931f370a>] zap_page_range+0xea/0x130
[<ffffffff930dad88>] ? wake_up_q+0x58/0x80
[<ffffffff931edbfd>] SyS_madvise+0x49d/0xac0
[<ffffffff93795f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 60 15 d5 93 48 89 df e8 57 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #172168
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffae116802
Build id: 3.10.0-1160.11.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffffae07e6a0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffffae07e6a0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffffae07e6a0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffffae11691f
Build id: 3.10.0-1160.11.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffae385fe5
Build id: 3.10.0-1160.11.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffae116ecb
Build id: 3.10.0-1160.11.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffae07e868
Build id: 3.10.0-1160.11.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffae07e8db
Build id: 3.10.0-1160.11.1.el7.x86_64
0x6b
10
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffae1edee6
Build id: 3.10.0-1160.11.1.el7.x86_64
0xa6
11
arch_tlb_finish_mmu
vmlinux 0xffffffffae1ef9af
Build id: 3.10.0-1160.11.1.el7.x86_64
0x3f
12
tlb_finish_mmu
vmlinux 0xffffffffae1efad3
Build id: 3.10.0-1160.11.1.el7.x86_64
0x23
13
zap_page_range
vmlinux 0xffffffffae1f2f9a
Build id: 3.10.0-1160.11.1.el7.x86_64
0xea
14
? wake_up_q
vmlinux 0xffffffffae0db148
Build id: 3.10.0-1160.11.1.el7.x86_64
0x58
15
SyS_madvise
vmlinux 0xffffffffae1ed48d
Build id: 3.10.0-1160.11.1.el7.x86_64
0x49d
16
? SyS_futex
vmlinux 0xffffffffae1154c0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x80
17
system_call_fastpath
vmlinux 0xffffffffae793f92
Build id: 3.10.0-1160.11.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [containerd-shim:24728]
Modules linked in: binfmt_misc xt_multiport ipt_rpfilter iptable_raw ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel veth xt_statistic xt_recent xt_nat ipt_REJECT nf_reject_ipv4 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache overlay(T) xt_set ip_set nfnetlink xt_addrtype iptable_nat nf_nat_ipv4 nf_nat xt_comment xt_mark xt_conntrack vmw_vsock_vmci_transport vsock iptable_mangle dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd i2c_piix4 ppdev joydev pcspkr vmw_balloon sg vmw_vmci parport_pc parport nfsd nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh
ip_vs_wrr ip_vs_rr ip_vs auth_rpcgss nfs_acl lockd grace sunrpc nf_conntrack libcrc32c br_netfilter bridge stp llc ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops sd_mod ttm crc_t10dif crct10dif_generic drm ahci ata_piix libahci crct10dif_pclmul crct10dif_common crc32c_intel nfit libata serio_raw libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 5 PID: 24728 Comm: containerd-shim Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffff97cbbeb49080 ti: ffff97c8b55c0000 task.ti: ffff97c8b55c0000
RIP: 0010:[<ffffffffae116802>]  [<ffffffffae116802>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff97c8b55c3be0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff97c8b55c3bb0 RCX: 0000000000000001
RDX: ffffffffae8137d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff97c8b55c3c30 R08: ffffffffae8137c8 R09: ffffd391066359c0
R10: 0000000000004b29 R11: 0000000000000246 R12: ffffffffae1c8c76
R13: ffff97c8b55c3bf0 R14: 0000000000020200 R15: 0000000000000000
FS:  000000c000040b10(0000) GS:ffff97cc5d740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4cef4e9220 CR3: 000000069cf92000 CR4: 00000000001607e0
Call Trace:
[<ffffffffae07e6a0>] ? leave_mm+0x120/0x120
[<ffffffffae07e6a0>] ? leave_mm+0x120/0x120
[<ffffffffae07e6a0>] ? leave_mm+0x120/0x120
[<ffffffffae11691f>] smp_call_function_single+0x5f/0xa0
[<ffffffffae385fe5>] ? cpumask_next_and+0x35/0x50
[<ffffffffae116ecb>] smp_call_function_many+0x22b/0x270
[<ffffffffae07e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffae07e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffae1edee6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffae1ef9af>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffae1efad3>] tlb_finish_mmu+0x23/0x30
[<ffffffffae1f2f9a>] zap_page_range+0xea/0x130
[<ffffffffae0db148>] ? wake_up_q+0x58/0x80
[<ffffffffae1ed48d>] SyS_madvise+0x49d/0xac0
[<ffffffffae1154c0>] ? SyS_futex+0x80/0x190
[<ffffffffae793f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 e0 13 d5 ae 48 89 df e8 57 69 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #59425
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff96b111ea
Build id: 3.10.0-957.12.1.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff96a7a400
Build id: 3.10.0-957.12.1.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff96a7a400
Build id: 3.10.0-957.12.1.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff96a7a400
Build id: 3.10.0-957.12.1.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff96b112ff
Build id: 3.10.0-957.12.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff96d75af5
Build id: 3.10.0-957.12.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff96b118ab
Build id: 3.10.0-957.12.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff96a7a5c8
Build id: 3.10.0-957.12.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff96a7a638
Build id: 3.10.0-957.12.1.el7.x86_64
0x68
10
tlb_flush_mmu.part.76
vmlinux 0xffffffff96be44c7
Build id: 3.10.0-957.12.1.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff96be5dc5
Build id: 3.10.0-957.12.1.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff96be8c12
Build id: 3.10.0-957.12.1.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffff96ad67a8
Build id: 3.10.0-957.12.1.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffff96be3add
Build id: 3.10.0-957.12.1.el7.x86_64
0x49d
15
system_call_fastpath
vmlinux 0xffffffff97175d9b
Build id: 3.10.0-957.12.1.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 21s! [containerd-shim:16624]
Modules linked in: xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper serio_raw crc32c_intel ata_generic nfit libnvdimm syscopyarea pata_acpi sysfillrect sysimgblt
fb_sys_fops mptspi ttm ata_piix scsi_transport_spi drm mptscsih libata mptbase e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 16624 Comm: containerd-shim Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8b7ac6f32080 ti: ffff8b7a83fd8000 task.ti: ffff8b7a83fd8000
RIP: 0010:[<ffffffff96b111ea>]  [<ffffffff96b111ea>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8b7a83fdbbf0  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff8b7a83fdbbc0 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff8b7a83fdbc38 R08: ffffffff97212dc0 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff8b7a83fdbbb8
R13: ffff8b7a83fdbb80 R14: ffff8b7adb2e5b40 R15: 00000000f4e858cc
FS:  000000c4200d9490(0000) GS:ffff8b7afcd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c4202ad010 CR3: 00000006c6af0000 CR4: 00000000003607e0
Call Trace:
[<ffffffff96a7a400>] ? leave_mm+0x110/0x110
[<ffffffff96a7a400>] ? leave_mm+0x110/0x110
[<ffffffff96a7a400>] ? leave_mm+0x110/0x110
[<ffffffff96b112ff>] smp_call_function_single+0x5f/0xa0
[<ffffffff96d75af5>] ? cpumask_next_and+0x35/0x50
[<ffffffff96b118ab>] smp_call_function_many+0x22b/0x270
[<ffffffff96a7a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff96a7a638>] flush_tlb_mm_range+0x68/0x140
[<ffffffff96be44c7>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff96be5dc5>] tlb_finish_mmu+0x55/0x60
[<ffffffff96be8c12>] zap_page_range+0x142/0x190
[<ffffffff96ad67a8>] ? wake_up_q+0x58/0x80
[<ffffffff96be3add>] SyS_madvise+0x49d/0xac0
[<ffffffff97175d9b>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 a0 ba 74 97 48 89 df e8 6a b7 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #45339
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff94d0a9fa
Build id: 3.10.0-862.11.6.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffff94c79c90
Build id: 3.10.0-862.11.6.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffff94c79c90
Build id: 3.10.0-862.11.6.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffff94c79c90
Build id: 3.10.0-862.11.6.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffff94d0ab0f
Build id: 3.10.0-862.11.6.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff94f4ec95
Build id: 3.10.0-862.11.6.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff94d0b0bb
Build id: 3.10.0-862.11.6.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff94c79e58
Build id: 3.10.0-862.11.6.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff94c79ec8
Build id: 3.10.0-862.11.6.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffff94dc33c7
Build id: 3.10.0-862.11.6.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffff94dc4cc5
Build id: 3.10.0-862.11.6.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffff94dc7b12
Build id: 3.10.0-862.11.6.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffff94cd1f98
Build id: 3.10.0-862.11.6.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffff94dc2a1d
Build id: 3.10.0-862.11.6.el7.x86_64
0x3cd
15
system_call_fastpath
vmlinux 0xffffffff9532579b
Build id: 3.10.0-862.11.6.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mysqld:13075]
Modules linked in: tcp_diag inet_diag talpa_vfshook(OE) talpa_pedconnector(OE) talpa_pedevice(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack xt_multiport iptable_filter vmw_vsock_vmci_transport vsock sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper nfit ablk_helper cryptd libnvdimm joydev pcspkr sg parport_pc parport vmw_vmci shpchp i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic ahci drm_kms_helper ata_piix libahci syscopyarea sysfillrect crct10dif_pclmul crct10dif_common sysimgblt fb_sys_fops libata ttm crc32c_intel serio_raw drm vmxnet3 vmw_pvscsi i2c_core dm_mirror dm_region_hash
dm_log dm_mod
CPU: 1 PID: 13075 Comm: mysqld Kdump: loaded Tainted: G           OE  ------------   3.10.0-862.11.6.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff919e2c361fa0 ti: ffff919da4ac0000 task.ti: ffff919da4ac0000
RIP: 0010:[<ffffffff94d0a9fa>]  [<ffffffff94d0a9fa>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff919da4ac3bf0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffff919da4ac3bc0 RCX: 000000000000003e
RDX: 0000000000000003 RSI: 0000000000000002 RDI: 0000000000000282
RBP: ffff919da4ac3c38 R08: ffffffff95413320 R09: 00003ffffffff000
R10: ffff919e2a7da5e8 R11: 00007f052307c000 R12: ffffffff95318978
R13: ffffffff95318984 R14: ffffffff95318978 R15: ffffffff95318984
FS:  00007f05230c4700(0000) GS:ffff919e3fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000559654b89000 CR3: 0000000035b70000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff94c79c90>] ? leave_mm+0x110/0x110
[<ffffffff94c79c90>] ? leave_mm+0x110/0x110
[<ffffffff94c79c90>] ? leave_mm+0x110/0x110
[<ffffffff94d0ab0f>] smp_call_function_single+0x5f/0xa0
[<ffffffff94f4ec95>] ? cpumask_next_and+0x35/0x50
[<ffffffff94d0b0bb>] smp_call_function_many+0x22b/0x270
[<ffffffff94c79e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff94c79ec8>] flush_tlb_mm_range+0x68/0x140
[<ffffffff94dc33c7>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff94dc4cc5>] tlb_finish_mmu+0x55/0x60
[<ffffffff94dc7b12>] zap_page_range+0x142/0x190
[<ffffffff94cd1f98>] ? wake_up_q+0x58/0x80
[<ffffffff94dc2a1d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff9532579b>] system_call_fastpath+0x22/0x27
Code: c0 96 01 00 48 89 de 48 03 14 c5 60 54 93 95 48 89 df e8 7a a0 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #23771
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffffa810aa2a
Build id: 3.10.0-862.14.4.el7.x86_64
0xfa
2
? leave_mm
vmlinux 0xffffffffa8079c90
Build id: 3.10.0-862.14.4.el7.x86_64
0x110
3
? leave_mm
vmlinux 0xffffffffa8079c90
Build id: 3.10.0-862.14.4.el7.x86_64
0x110
4
? leave_mm
vmlinux 0xffffffffa8079c90
Build id: 3.10.0-862.14.4.el7.x86_64
0x110
5
smp_call_function_single
vmlinux 0xffffffffa810ab3f
Build id: 3.10.0-862.14.4.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffffa834edb5
Build id: 3.10.0-862.14.4.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffffa810b0eb
Build id: 3.10.0-862.14.4.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffffa8079e58
Build id: 3.10.0-862.14.4.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffffa8079ec8
Build id: 3.10.0-862.14.4.el7.x86_64
0x68
10
tlb_flush_mmu.part.75
vmlinux 0xffffffffa81c3487
Build id: 3.10.0-862.14.4.el7.x86_64
0x37
11
tlb_finish_mmu
vmlinux 0xffffffffa81c4d85
Build id: 3.10.0-862.14.4.el7.x86_64
0x55
12
zap_page_range
vmlinux 0xffffffffa81c7bd2
Build id: 3.10.0-862.14.4.el7.x86_64
0x142
13
? wake_up_q
vmlinux 0xffffffffa80d1fc8
Build id: 3.10.0-862.14.4.el7.x86_64
0x58
14
SyS_madvise
vmlinux 0xffffffffa81c2add
Build id: 3.10.0-862.14.4.el7.x86_64
0x3cd
15
? wake_up_state
vmlinux 0xffffffffa80d2010
Build id: 3.10.0-862.14.4.el7.x86_64
0x20
16
tracesys
vmlinux 0xffffffffa8725a1b
Build id: 3.10.0-862.14.4.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [prometheus:22200]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_lsm_pinned_10703(E) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache vmw_vsock_vmci_transport vsock falcon_lsm_pinned_10404(E) sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd vmw_balloon nfit pcspkr joydev sg libnvdimm vmw_vmci shpchp i2c_piix4 parport_pc parport ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common ata_piix crc32c_intel mptspi serio_raw libata scsi_transport_spi mptscsih mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_lsm_serviceable]
CPU: 5 PID: 22200 Comm: prometheus Kdump: loaded Tainted: P            EL ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9b7a3a322f70 ti: ffff9b7acba94000 task.ti: ffff9b7acba94000
RIP: 0010:[<ffffffffa810aa2a>]  [<ffffffffa810aa2a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9b7acba97bf0  EFLAGS: 00000202
RAX: 0000000000000008 RBX: ffff9b7acba97bc0 RCX: 0000000000000038
RDX: 00000000000000ff RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffff9b7acba97c38 R08: ffffffffa88141a0 R09: 00003ffffffff000
R10: ffff9b7cda2ad878 R11: 0000000000000246 R12: ffffffffa834edb5
R13: ffff9b7acba97b78 R14: ffff9b7a39d29800 R15: 00000000a1cc8b98
FS:  00007f54df9c8700(0000) GS:ffff9b7cffd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f603834d000 CR3: 0000000421b8e000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa8079c90>] ? leave_mm+0x110/0x110
[<ffffffffa8079c90>] ? leave_mm+0x110/0x110
[<ffffffffa8079c90>] ? leave_mm+0x110/0x110
[<ffffffffa810ab3f>] smp_call_function_single+0x5f/0xa0
[<ffffffffa834edb5>] ? cpumask_next_and+0x35/0x50
[<ffffffffa810b0eb>] smp_call_function_many+0x22b/0x270
[<ffffffffa8079e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa8079ec8>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa81c3487>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffa81c4d85>] tlb_finish_mmu+0x55/0x60
[<ffffffffa81c7bd2>] zap_page_range+0x142/0x190
[<ffffffffa80d1fc8>] ? wake_up_q+0x58/0x80
[<ffffffffa81c2add>] SyS_madvise+0x3cd/0x9c0
[<ffffffffa80d2010>] ? wake_up_state+0x20/0x20
[<ffffffffa8725a1b>] tracesys+0xa3/0xc9
Code: c0 96 01 00 48 89 de 48 03 14 c5 a0 54 d3 a8 48 89 df e8 6a a1 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #16906
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff9a116f72
Build id: 3.10.0-1127.19.1.el7.x86_64
0x102
2
? leave_mm
vmlinux 0xffffffff9a07e730
Build id: 3.10.0-1127.19.1.el7.x86_64
0x120
3
? leave_mm
vmlinux 0xffffffff9a07e730
Build id: 3.10.0-1127.19.1.el7.x86_64
0x120
4
? leave_mm
vmlinux 0xffffffff9a07e730
Build id: 3.10.0-1127.19.1.el7.x86_64
0x120
5
smp_call_function_single
vmlinux 0xffffffff9a11708f
Build id: 3.10.0-1127.19.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff9a385005
Build id: 3.10.0-1127.19.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff9a11763b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x22b
8
native_flush_tlb_others
vmlinux 0xffffffff9a07e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
9
flush_tlb_mm_range
vmlinux 0xffffffff9a07e96b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6b
10
zap_page_range
vmlinux 0xffffffff9a1f2ed8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xd8
11
? wake_up_q
vmlinux 0xffffffff9a0db948
Build id: 3.10.0-1127.19.1.el7.x86_64
0x58
12
SyS_madvise
vmlinux 0xffffffff9a1ed3ed
Build id: 3.10.0-1127.19.1.el7.x86_64
0x49d
13
system_call_fastpath
vmlinux 0xffffffff9a792ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [containerd-shim:3538]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ppdev snd_ens1371 ghash_clmulni_intel snd_rawmidi snd_ac97_codec ac97_bus aesni_intel
vmw_balloon snd_seq lrw gf128mul snd_seq_device glue_helper ablk_helper cryptd snd_pcm snd_timer snd sg joydev pcspkr soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel nfit mptspi libnvdimm ata_piix scsi_transport_spi drm mptscsih e1000 libata serio_raw mptbase drm_panel_orientation_quirks
CPU: 1 PID: 3538 Comm: containerd-shim Tainted: G             L ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9b157412b150 ti: ffff9b18be730000 task.ti: ffff9b18be730000
RIP: 0010:[<ffffffff9a116f72>]  [<ffffffff9a116f72>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9b18be733c30  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9b18be733c00 RCX: 0000000000000001
RDX: ffffffff9a8136d0 RSI: 0000000000000080 RDI: 0000000000000286
RBP: ffff9b18be733c80 R08: ffffffff9a8136c8 R09: 00003ffffffff000
R10: ffff9b18c207cf30 R11: 0000000000000246 R12: ffff9b18ed65acc0
R13: ffffffff9a0e103c R14: ffff9b18be733bb8 R15: ffffffff9a0e2e89
FS:  000000c4200f8890(0000) GS:ffff9b18ed640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c42019b010 CR3: 00000003ff628000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9a07e730>] ? leave_mm+0x120/0x120
[<ffffffff9a07e730>] ? leave_mm+0x120/0x120
[<ffffffff9a07e730>] ? leave_mm+0x120/0x120
[<ffffffff9a11708f>] smp_call_function_single+0x5f/0xa0
[<ffffffff9a385005>] ? cpumask_next_and+0x35/0x50
[<ffffffff9a11763b>] smp_call_function_many+0x22b/0x270
[<ffffffff9a07e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9a07e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9a1f2ed8>] zap_page_range+0xd8/0x150
[<ffffffff9a0db948>] ? wake_up_q+0x58/0x80
[<ffffffff9a1ed3ed>] SyS_madvise+0x49d/0xab0
[<ffffffff9a792ed2>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 a0 10 d5 9a 48 89 df e8 07 52 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00