Complete report #1422151
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff86716c0a
Build id: 3.10.0-1160.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8667e868
Build id: 3.10.0-1160.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8667e8db
Build id: 3.10.0-1160.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff867edd06
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff867ef7cf
Build id: 3.10.0-1160.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff867ef8f3
Build id: 3.10.0-1160.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff867f2dba
Build id: 3.10.0-1160.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffff8684cf83
Build id: 3.10.0-1160.el7.x86_64
|
0x93
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffff867ed2ad
Build id: 3.10.0-1160.el7.x86_64
|
0x49d
|
|
10
|
system_call_fastpath
|
vmlinux |
0xffffffff86d93f92
Build id: 3.10.0-1160.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [server.rb:286:6444]
Modules linked in: tcp_diag inet_diag xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi snd_ens1371 crc32_pclmul snd_rawmidi snd_ac97_codec ghash_clmulni_intel ppdev ac97_bus snd_seq snd_seq_device snd_pcm vmw_balloon aesni_intel lrw
gf128mul glue_helper ablk_helper cryptd snd_timer sg snd joydev vmw_vmci pcspkr soundcore i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata nfit crct10dif_pclmul crct10dif_common libnvdimm mptspi crc32c_intel scsi_transport_spi e1000 mptscsih mptbase serio_raw drm_panel_orientation_quirks fuse
CPU: 4 PID: 6444 Comm: server.rb:286 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8f2548f49080 ti: ffff8f255f234000 task.ti: ffff8f255f234000
RIP: 0010:[<ffffffff86716c0a>] [<ffffffff86716c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8f255f237c70 EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800008667e6a0 RCX: ffff8f262d6a0678
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8f255f237ca8 R08: ffff8f237a74bc00 R09: ffffffff86985f99
R10: ffff8f262d71f160 R11: ffffd54245c9f000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8f255f237c20 R15: 0000000000000001
FS: 00007fd4e31fd700(0000) GS:ffff8f262d700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002378bb0 CR3: 0000000174f92000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff8667e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8667e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff867edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff867ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff867ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff867f2dba>] zap_page_range+0xea/0x130
[<ffffffff8684cf83>] ? do_sync_read+0x93/0xe0
[<ffffffff867ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffff86d93f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 35 87 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1198652
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8c71e25a
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8c681fa8
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8c68201b
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff8c7fa0ce
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xae
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff8c7fbc4f
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff8c7fbd83
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff8c7ff2ca
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffff8c9226b9
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xc9
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffff8c7f96bd
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffff8c75f95b
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x5b
|
|
11
|
tracesys
|
vmlinux |
0xffffffff8cdc562e
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [conn23:5105]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi ppdev snd_ac97_codec aesni_intel vmw_balloon
ac97_bus lrw snd_seq gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm joydev pcspkr snd_timer snd sg vmw_vmci soundcore i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit ata_piix libnvdimm libata crct10dif_pclmul crct10dif_common crc32c_intel mptspi e1000 scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 2 PID: 5105 Comm: conn23 Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8a6fd805e300 ti: ffff8a6fe1874000 task.ti: ffff8a6fe1874000
RIP: 0010:[<ffffffff8c71e25a>] [<ffffffff8c71e25a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8a6fe1877c70 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 000800008c681b10 RCX: ffff8a707a6604b0
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8a6fe1877ca8 R08: ffff8a70459e4000 R09: ffffffff8c99c7f9
R10: ffff8a707a69f160 R11: fffff15b8037ea00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8a6fe1877c20 R15: 0000000000000001
FS: 00007f0d54f20700(0000) GS:ffff8a707a680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f157622d2a0 CR3: 000000008a852000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8c681fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8c68201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8c7fa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff8c7fbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8c7fbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffff8c7ff2ca>] zap_page_range+0xea/0x140
[<ffffffff8c9226b9>] ? file_has_perm+0xc9/0x160
[<ffffffff8c7f96bd>] SyS_madvise+0x49d/0xac0
[<ffffffff8c75f95b>] ? __seccomp_filter+0x5b/0x310
[<ffffffff8cdc562e>] tracesys+0xa6/0xcc
Code: 48 63 35 ae 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 19 35 8d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 7d 18 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #882630
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb611722a
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb607e258
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffb607e2cb
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffb61ee666
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffb61f012f
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffb61f0253
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffb61f371a
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffb61edc0d
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffffb61573eb
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x5b
|
|
10
|
|
vmlinux |
0xffffffffb60daf40
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x20
|
|
11
|
tracesys
|
vmlinux |
0xffffffffb679a226
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [kube-apiserver:13049]
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [kube-apiserver:12602]
Modules linked in: xt_statistic vxlan ip6_udp_tunnel udp_tunnel xt_nat nf_tables ip_set ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_mangle ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter tun overlay(T) vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon sg pcspkr joydev vmw_vmci i2c_piix4 parport_pc parport
ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm nfit ata_piix libahci libnvdimm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 10 PID: 12602 Comm: kube-apiserver Tainted: G L ------------ T 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff90ae5cf20000 ti: ffff90b42dad8000 task.ti: ffff90b42dad8000
RIP: 0010:[<ffffffffb611722a>] [<ffffffffb611722a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff90b42dadbc70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000b607ddd0 RCX: ffff90ae67c207c8
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff90b42dadbca8 R08: ffff90aef0df6000 R09: ffffffffb63869d9
R10: ffff90b47fc9f160 R11: fffff61ceeffb200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff90b42dadbc20 R15: 0000000000000001
FS: 000000c00008c490(0000) GS:ffff90b47fc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c007cc2038 CR3: 0000000beda50000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb607e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb607e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb61ee666>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb61f012f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb61f0253>] tlb_finish_mmu+0x23/0x30
[<ffffffffb61f371a>] zap_page_range+0xea/0x130
[<ffffffffb61edc0d>] SyS_madvise+0x49d/0xac0
[<ffffffffb61573eb>] ? __seccomp_filter+0x5b/0x300
[<ffffffffb60daf40>] ? wake_up_state+0x20/0x20
[<ffffffffb679a226>] tracesys+0xa6/0xcc
Code: 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 d5 b6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #867366
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa971e23a
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa9681fa8
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa968201b
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffa97fa07e
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0xae
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffa97fbbff
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa97fbd33
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffa97ff27a
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffffa9802503
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0xdb3
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffffa97f966d
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffffa975f93b
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x5b
|
|
11
|
tracesys
|
vmlinux |
0xffffffffa9dc562e
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [popen.rb:41:4193]
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [memory_killer:4192]
Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq iosf_mbi snd_seq_device crc32_pclmul snd_pcm ghash_clmulni_intel aesni_intel snd_timer lrw gf128mul ppdev snd glue_helper ablk_helper
cryptd vmw_balloon soundcore sg pcspkr joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm e1000 crct10dif_pclmul drm crct10dif_common crc32c_intel ata_piix mptspi nfit scsi_transport_spi libata serio_raw mptscsih libnvdimm mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 7 PID: 4192 Comm: memory_killer Kdump: loaded Tainted: G ------------ T 3.10.0-1160.99.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8a8d6a5a1080 ti: ffff8a8d1748c000 task.ti: ffff8a8d1748c000
RIP: 0010:[<ffffffffa971e23a>] [<ffffffffa971e23a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8a8d1748fc70 EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00080000a9681b10 RCX: ffff8a8df57206f0
RDX: 0000000000000004 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8a8d1748fca8 R08: ffff8a8d3a88c800 R09: ffffffffa999c7b9
R10: ffff8a8df57df160 R11: ffffcb0645a91e00 R12: 0000000000000282
R13: 0000000000000018 R14: ffff8a8d1748fbe0 R15: 0000000000000282
FS: 00007f78c7260700(0000) GS:ffff8a8df57c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f79090f3338 CR3: 000000004d0d4000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa9681fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa968201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa97fa07e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa97fbbff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa97fbd33>] tlb_finish_mmu+0x23/0x40
[<ffffffffa97ff27a>] zap_page_range+0xea/0x140
[<ffffffffa9802503>] ? handle_mm_fault+0xdb3/0x1190
[<ffffffffa97f966d>] SyS_madvise+0x49d/0xac0
[<ffffffffa975f93b>] ? __seccomp_filter+0x5b/0x310
[<ffffffffa9dc562e>] tracesys+0xa6/0xcc
Code: 48 63 35 4e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 19 35 aa f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 18 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #812858
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9fd1720a
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9fc7e258
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9fc7e2cb
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff9fdee6c6
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff9fdf018f
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff9fdf02b3
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff9fdf377a
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffff9fdfd36c
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x39c
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffff9fdedc6d
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffff9fd573fb
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x5b
|
|
11
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffa0399ed5
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0xa2
|
|
12
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffa0399ec9
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x96
|
|
13
|
tracesys
|
vmlinux |
0xffffffffa039a226
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [nacos-grpc-clie:55956]
Modules linked in: veth binfmt_misc xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) ext4 ppdev mbcache jbd2 pcspkr joydev virtio_balloon sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover virtio_console virtio_scsi virtio_blk failover bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci floppy virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 55956 Comm: nacos-grpc-clie Kdump: loaded Tainted: G ------------ T 3.10.0-1160.71.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff8952722e6300 ti: ffff89511d084000 task.ti: ffff89511d084000
RIP: 0010:[<ffffffff9fd1720a>] [<ffffffff9fd1720a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff89511d087c70 EFLAGS: 00000202
RAX: 0000000000000005 RBX: 000c00009fc7ddd0 RCX: ffff8952acf60428
RDX: 0000000000000005 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff89511d087ca8 R08: ffff894ff5c42c00 R09: ffffffff9ff86a19
R10: ffff8952ace9f160 R11: ffffbf6287382200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff89511d087c20 R15: 0000000000000001
FS: 00007f70e0de5640(0000) GS:ffff8952ace80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1b3a1e3d70 CR3: 00000003f210c000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9fc7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9fc7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9fdee6c6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9fdf018f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9fdf02b3>] tlb_finish_mmu+0x23/0x30
[<ffffffff9fdf377a>] zap_page_range+0xea/0x130
[<ffffffff9fdfd36c>] ? do_mmap+0x39c/0x590
[<ffffffff9fdedc6d>] SyS_madvise+0x49d/0xac0
[<ffffffff9fd573fb>] ? __seccomp_filter+0x5b/0x300
[<ffffffffa0399ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffa0399ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffa039a226>] tracesys+0xa6/0xcc
Code: 48 63 35 9e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 95 a0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d 87 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #733799
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa0b170da
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa0a7e258
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa0a7e2cb
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffa0bee676
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffa0bf013f
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa0bf0263
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffa0bf372a
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffa0bedc1d
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffffa0b5729b
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x5b
|
|
10
|
tracesys
|
vmlinux |
0xffffffffa119a226
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [java:9490]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 nfit libnvdimm coretemp ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm pata_acpi mptspi drm scsi_transport_spi ata_piix mptscsih mptbase crct10dif_pclmul libata crct10dif_common crc32c_intel serio_raw vmxnet3 drm_panel_orientation_quirks
floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 8 PID: 9490 Comm: java Kdump: loaded Tainted: G W L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9057c66b6300 ti: ffff905251180000 task.ti: ffff905251180000
RIP: 0010:[<ffffffffa0b170da>] [<ffffffffa0b170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff905251183c70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a0a7ddd0 RCX: ffff9052f7c20850
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff905251183ca8 R08: ffff90557b0cfc00 R09: ffffffffa0d86c09
R10: ffff9056f7c1f160 R11: ffffe80d54635a00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff905251183c20 R15: 0000000000000001
FS: 00007fbe9b2d9700(0000) GS:ffff9056f7c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc0c000ad40 CR3: 0000000236dd2000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa0a7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa0a7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa0bee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa0bf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa0bf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffa0bf372a>] zap_page_range+0xea/0x130
[<ffffffffa0bedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffa0b5729b>] ? __seccomp_filter+0x5b/0x300
[<ffffffffa119a226>] tracesys+0xa6/0xcc
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 75 a1 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #639707
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffaab170da
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffaaa7e258
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffaaa7e2cb
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffaabee676
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffaabf013f
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffaabf0263
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffaabf372a
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffffaabf66a2
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xb22
|
|
9
|
|
vmlinux |
0xffffffffaabfd36c
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x39c
|
|
10
|
SyS_madvise
|
vmlinux |
0xffffffffaabedc1d
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x49d
|
|
11
|
|
vmlinux |
0xffffffffaab5729b
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x5b
|
|
12
|
tracesys
|
vmlinux |
0xffffffffab19a226
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [elasticsearch[s:26317]
Modules linked in: 8021q garp mrp stp llc ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter vmw_vsock_vmci_transport vsock iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr sg joydev parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel nfit serio_raw libnvdimm vmw_pvscsi vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 26317 Comm: elasticsearch[s Kdump: loaded Tainted: G L ------------ 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff98a1f8750000 ti: ffff98a283548000 task.ti: ffff98a283548000
RIP: 0010:[<ffffffffaab170da>] [<ffffffffaab170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff98a28354bc70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000aaa7ddd0 RCX: ffff98a57fc20628
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff98a28354bca8 R08: ffff98a2f6bc0c00 R09: ffffffffaad86c09
R10: ffff98a57fc9f160 R11: ffffe0decb05ca00 R12: 0000000000000282
R13: 0000000000000018 R14: ffff98a28354bbe0 R15: 0000000000000282
FS: 00007f806d2c7700(0000) GS:ffff98a57fc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f81a28472b0 CR3: 00000003d7942000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffaaa7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaaa7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaabee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaabf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaabf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffaabf372a>] zap_page_range+0xea/0x130
[<ffffffffaabf66a2>] ? handle_mm_fault+0xb22/0xfb0
[<ffffffffaabfd36c>] ? do_mmap+0x39c/0x590
[<ffffffffaabedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffaab5729b>] ? __seccomp_filter+0x5b/0x300
[<ffffffffab19a226>] tracesys+0xa6/0xcc
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 75 ab f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #619140
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff93b1e0ba
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff93a81f88
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff93a81ffb
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff93bf9f4e
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0xae
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff93bfba8f
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff93bfbbc3
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff93bff10a
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffff93ae1008
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x58
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffff93bf94dd
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffff93b5f7bb
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x5b
|
|
11
|
|
vmlinux |
0xffffffff93b1c620
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x80
|
|
12
|
tracesys
|
vmlinux |
0xffffffff941c562e
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [auditbeat:1151]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_14604(E) snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core iosf_mbi snd_hwdep crc32_pclmul snd_seq ghash_clmulni_intel aesni_intel snd_seq_device snd_pcm lrw gf128mul snd_timer glue_helper snd ablk_helper cryptd soundcore joydev virtio_rng virtio_balloon i2c_piix4 pcspkr ip_tables xfs libcrc32c ata_generic virtio_blk virtio_console pata_acpi qxl drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm ata_piix e1000 drm libata serio_raw crct10dif_pclmul crct10dif_common virtio_pci crc32c_intel virtio_ring floppy virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 1151 Comm: auditbeat Tainted: P E ------------ 3.10.0-1160.83.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.15.0-2.module+el8.6.0+20659+3dcf7c70 04/01/2014
task: ffff935c3102d280 ti: ffff935c328f8000 task.ti: ffff935c328f8000
RIP: 0010:[<ffffffff93b1e0ba>] [<ffffffff93b1e0ba>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff935c328fbc70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0008000093a81af0 RCX: ffff935c3fc203c0
RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff935c328fbca8 R08: ffff935af9312000 R09: ffffffff93d9c179
R10: ffff935c3fd5f160 R11: fffff46185c1d100 R12: 0000000000000282
R13: 0000000000000282 R14: ffff935c328fbc20 R15: 0000000000000001
FS: 00007fcca7fff700(0000) GS:ffff935c3fd40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c001b5f000 CR3: 00000000ba09a000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff93a81f88>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff93a81ffb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff93bf9f4e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff93bfba8f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff93bfbbc3>] tlb_finish_mmu+0x23/0x40
[<ffffffff93bff10a>] zap_page_range+0xea/0x140
[<ffffffff93ae1008>] ? wake_up_q+0x58/0x90
[<ffffffff93bf94dd>] SyS_madvise+0x49d/0xac0
[<ffffffff93b5f7bb>] ? __seccomp_filter+0x5b/0x310
[<ffffffff93b1c620>] ? SyS_futex+0x80/0x190
[<ffffffff941c562e>] tracesys+0xa6/0xcc
Code: 48 63 35 8e 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 75 94 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d 19 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #586493
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa8b16c0a
Build id: 3.10.0-1160.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa8a7e868
Build id: 3.10.0-1160.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa8a7e8db
Build id: 3.10.0-1160.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffa8bedd06
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffa8bef7cf
Build id: 3.10.0-1160.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa8bef8f3
Build id: 3.10.0-1160.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffa8bf2dba
Build id: 3.10.0-1160.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffa8bed2ad
Build id: 3.10.0-1160.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffffa8b56dfb
Build id: 3.10.0-1160.el7.x86_64
|
0x5b
|
|
10
|
|
vmlinux |
0xffffffffa8c4a6fa
Build id: 3.10.0-1160.el7.x86_64
|
0x1ca
|
|
11
|
tracesys
|
vmlinux |
0xffffffffa9194226
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [util.rb:23:101209]
Modules linked in: tcp_lp veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi ppdev snd_ac97_codec ac97_bus aesni_intel
snd_seq lrw gf128mul snd_seq_device glue_helper ablk_helper vmw_balloon snd_pcm cryptd snd_timer joydev pcspkr snd soundcore i2c_piix4 sg vmw_vmci parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt ata_generic pata_acpi crct10dif_pclmul mptspi fb_sys_fops crct10dif_common ttm scsi_transport_spi drm ata_piix e1000 mptscsih crc32c_intel serio_raw mptbase nfit libnvdimm libata drm_panel_orientation_quirks fuse
CPU: 1 PID: 101209 Comm: util.rb:23 Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff88958b05b180 ti: ffff88969f8fc000 task.ti: ffff88969f8fc000
RIP: 0010:[<ffffffffa8b16c0a>] [<ffffffffa8b16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff88969f8ffc70 EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00080000a8a7e6a0 RCX: ffff8897af2a0600
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff88969f8ffca8 R08: ffff8896fe34b000 R09: ffffffffa8d85f99
R10: ffff8897af25f160 R11: ffffe442c436ac00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff88969f8ffc20 R15: 0000000000000001
FS: 00007fe3a33ef700(0000) GS:ffff8897af240000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa515170000 CR3: 00000001391a4000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa8a7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa8a7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa8bedd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa8bef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa8bef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffa8bf2dba>] zap_page_range+0xea/0x130
[<ffffffffa8bed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffffa8b56dfb>] ? __seccomp_filter+0x5b/0x300
[<ffffffffa8c4a6fa>] ? __check_object_size+0x1ca/0x250
[<ffffffffa9194226>] tracesys+0xa6/0xcc
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 75 a9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #586400
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8c916c0a
Build id: 3.10.0-1160.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8c87e868
Build id: 3.10.0-1160.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8c87e8db
Build id: 3.10.0-1160.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff8c9edd06
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff8c9ef7cf
Build id: 3.10.0-1160.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff8c9ef8f3
Build id: 3.10.0-1160.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff8c9f2dba
Build id: 3.10.0-1160.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffff8c9f5d32
Build id: 3.10.0-1160.el7.x86_64
|
0xb22
|
|
9
|
|
vmlinux |
0xffffffff8c9fc9d8
Build id: 3.10.0-1160.el7.x86_64
|
0x378
|
|
10
|
SyS_madvise
|
vmlinux |
0xffffffff8c9ed2ad
Build id: 3.10.0-1160.el7.x86_64
|
0x49d
|
|
11
|
|
vmlinux |
0xffffffff8c956dfb
Build id: 3.10.0-1160.el7.x86_64
|
0x5b
|
|
12
|
tracesys
|
vmlinux |
0xffffffff8cf94226
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 62s! [FilePath.localP:6846]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi aesni_intel snd_ac97_codec lrw ac97_bus
gf128mul ppdev glue_helper ablk_helper snd_seq snd_seq_device cryptd vmw_balloon snd_pcm snd_timer snd joydev pcspkr soundcore sg i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom ata_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops pata_acpi mptspi ttm scsi_transport_spi crct10dif_pclmul mptscsih crct10dif_common ata_piix drm mptbase nfit e1000 crc32c_intel serio_raw drm_panel_orientation_quirks libata libnvdimm fuse
CPU: 1 PID: 6846 Comm: FilePath.localP Kdump: loaded Tainted: G W L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff99a752df6300 ti: ffff99a75297c000 task.ti: ffff99a75297c000
RIP: 0010:[<ffffffff8c916c0a>] [<ffffffff8c916c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff99a75297fc70 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000800008c87e6a0 RCX: ffff99a7796e0600
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff99a75297fca8 R08: ffff99a770fe3000 R09: ffffffff8cb85f99
R10: ffff99a77965f160 R11: fffff2ce82589600 R12: 0000000000000282
R13: 0000000000000282 R14: ffff99a75297fc20 R15: 0000000000000001
FS: 00007fce3ed70700(0000) GS:ffff99a779640000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fce5e86b530 CR3: 00000000996c6000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8c87e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8c87e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8c9edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8c9ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8c9ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff8c9f2dba>] zap_page_range+0xea/0x130
[<ffffffff8c9f5d32>] ? handle_mm_fault+0xb22/0xfb0
[<ffffffff8c9fc9d8>] ? do_mmap+0x378/0x530
[<ffffffff8c9ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffff8c956dfb>] ? __seccomp_filter+0x5b/0x300
[<ffffffff8cf94226>] tracesys+0xa6/0xcc
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 55 8d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #579523
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8b916bda
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8b87e258
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8b87e2cb
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff8b9ee176
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff8b9efc3f
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff8b9efd63
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff8b9f322a
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffff8b8dad78
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x58
|
|
9
|
|
vmlinux |
0xffffffff8b912170
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x90
|
|
10
|
SyS_madvise
|
vmlinux |
0xffffffff8b9ed71d
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x49d
|
|
11
|
|
vmlinux |
0xffffffff8b956d9b
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x5b
|
|
12
|
tracesys
|
vmlinux |
0xffffffff8bf96226
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [JS Helper:7567]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache macvlan dummy xt_nat veth tcp_lp nf_conntrack_netlink xt_addrtype br_netfilter xt_multiport xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) vmw_vsock_vmci_transport vsock
sunrpc sb_edac iosf_mbi kvm_intel ppdev kvm vmw_balloon irqbypass pcspkr joydev sg parport_pc parport vmw_vmci i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops sd_mod ttm crc32_pclmul crc32c_intel ahci crc_t10dif crct10dif_generic crct10dif_pclmul ghash_clmulni_intel crct10dif_common ata_piix libahci drm aesni_intel nfit libata glue_helper lrw gf128mul ablk_helper libnvdimm cryptd serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 4 PID: 7567 Comm: JS Helper Kdump: loaded Tainted: G OEL ------------ T 3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8c8ba2278000 ti: ffff8c8ba236c000 task.ti: ffff8c8ba236c000
RIP: 0010:[<ffffffff8b916bda>] [<ffffffff8b916bda>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8c8ba236fc70 EFLAGS: 00000202
RAX: 0000000000000007 RBX: 000800008b87ddd0 RCX: ffff8c8bfd7e0678
RDX: 0000000000000007 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8c8ba236fca8 R08: ffff8c7d59683c00 R09: ffffffff8bb866a9
R10: ffff8c8bfd71f160 R11: fffffe569e03d000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8c8ba236fc20 R15: 0000000000000001
FS: 00007fda72402700(0000) GS:ffff8c8bfd700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f71a7ffc318 CR3: 0000000f98ae8000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8b87e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8b87e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8b9ee176>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8b9efc3f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8b9efd63>] tlb_finish_mmu+0x23/0x30
[<ffffffff8b9f322a>] zap_page_range+0xea/0x130
[<ffffffff8b8dad78>] ? wake_up_q+0x58/0x80
[<ffffffff8b912170>] ? futex_wake+0x90/0x180
[<ffffffff8b9ed71d>] SyS_madvise+0x49d/0xac0
[<ffffffff8b956d9b>] ? __seccomp_filter+0x5b/0x300
[<ffffffff8bf96226>] tracesys+0xa6/0xcc
Code: 48 63 35 ce 8a c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 55 8c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 8a c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #554487
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa8916c0a
Build id: 3.10.0-1160.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa887e868
Build id: 3.10.0-1160.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa887e8db
Build id: 3.10.0-1160.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffa89edd06
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffa89ef7cf
Build id: 3.10.0-1160.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa89ef8f3
Build id: 3.10.0-1160.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffa89f2dba
Build id: 3.10.0-1160.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffffa89fc9d8
Build id: 3.10.0-1160.el7.x86_64
|
0x378
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffffa89ed2ad
Build id: 3.10.0-1160.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffffa8956dfb
Build id: 3.10.0-1160.el7.x86_64
|
0x5b
|
|
11
|
tracesys
|
vmlinux |
0xffffffffa8f94226
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 25s! [java:25014]
Modules linked in: veth xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock sg vmw_balloon ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel joydev lrw gf128mul glue_helper ablk_helper cryptd vmw_vmci parport_pc parport pcspkr i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea crc_t10dif sysfillrect crct10dif_generic sysimgblt fb_sys_fops ttm drm ahci crct10dif_pclmul nfit ata_piix libahci crct10dif_common crc32c_intel serio_raw libata libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash
dm_log dm_mod fuse
CPU: 1 PID: 25014 Comm: java Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8a6a9aea6300 ti: ffff8a6cbc8fc000 task.ti: ffff8a6cbc8fc000
RIP: 0010:[<ffffffffa8916c0a>] [<ffffffffa8916c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8a6cbc8ffc70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a887e6a0 RCX: ffff8a6a75a20558
RDX: 0000000000000000 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff8a6cbc8ffca8 R08: ffff8a692e7ac000 R09: ffffffffa8b85f99
R10: ffff8a6a75a5f160 R11: ffffd8a9ccd5e200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8a6cbc8ffc20 R15: 0000000000000001
FS: 00007f417cece700(0000) GS:ffff8a6a75a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f71167b3200 CR3: 00000004fc898000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffffa887e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa887e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa89edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa89ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa89ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffa89f2dba>] zap_page_range+0xea/0x130
[<ffffffffa89fc9d8>] ? do_mmap+0x378/0x530
[<ffffffffa89ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffffa8956dfb>] ? __seccomp_filter+0x5b/0x300
[<ffffffffa8f94226>] tracesys+0xa6/0xcc
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 55 a9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #443679
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb0f16c0a
Build id: 3.10.0-1160.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb0e7e868
Build id: 3.10.0-1160.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffb0e7e8db
Build id: 3.10.0-1160.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffb0fedd06
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffb0fef7cf
Build id: 3.10.0-1160.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffb0fef8f3
Build id: 3.10.0-1160.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffb0ff2dba
Build id: 3.10.0-1160.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffb0fed2ad
Build id: 3.10.0-1160.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffffb0f56dfb
Build id: 3.10.0-1160.el7.x86_64
|
0x5b
|
|
10
|
tracesys
|
vmlinux |
0xffffffffb1594226
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [popen.rb:42:14674]
Modules linked in: veth br_netfilter tun devlink overlay(T) bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec iosf_mbi crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq snd_seq_device aesni_intel snd_pcm lrw gf128mul ppdev glue_helper ablk_helper cryptd snd_timer vmw_balloon snd soundcore joydev sg pcspkr i2c_piix4 vmw_vmci parport_pc parport xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm nfit libnvdimm drm serio_raw mptspi e1000 scsi_transport_spi ata_piix mptscsih libata mptbase drm_panel_orientation_quirks
fuse [last unloaded: ip_tables]
CPU: 0 PID: 14674 Comm: popen.rb:42 Kdump: loaded Tainted: G B L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9c85837ea100 ti: ffff9c852e8f4000 task.ti: ffff9c852e8f4000
RIP: 0010:[<ffffffffb0f16c0a>] [<ffffffffb0f16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9c852e8f7c70 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000c0000b0e7e6a0 RCX: ffff9c85b96dfc10
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9c852e8f7ca8 R08: ffff9c85b91d7000 R09: ffffffffb1185f99
R10: ffff9c85b961f160 R11: ffffcfd502589e00 R12: 0000000000000282
R13: 0000000000000018 R14: ffff9c852e8f7be0 R15: 0000000000000282
FS: 00007f4cc8453700(0000) GS:ffff9c85b9600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4cbfdf7000 CR3: 000000003592e000 CR4: 00000000003607f0
Call Trace:
[<ffffffffb0e7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb0e7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb0fedd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb0fef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb0fef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffb0ff2dba>] zap_page_range+0xea/0x130
[<ffffffffb0fed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffffb0f56dfb>] ? __seccomp_filter+0x5b/0x300
[<ffffffffb1594226>] tracesys+0xa6/0xcc
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 b5 b1 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #407480
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8251722a
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8247e258
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8247e2cb
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff825ee666
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff825f012f
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff825f0253
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff825f371a
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffff824daef8
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x58
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffff825edc0d
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffff825573eb
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x5b
|
|
11
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff82b99ed5
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xa2
|
|
12
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff82b99ec9
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x96
|
|
13
|
tracesys
|
vmlinux |
0xffffffff82b9a226
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [portainer:23352]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc vfat fat snd_intel8x0 snd_ac97_codec ac97_bus iTCO_wdt iTCO_vendor_support snd_seq snd_seq_device snd_pcm snd_timer ppdev lpc_ich snd parport_pc soundcore sg video parport i2c_piix4 pcspkr ip_tables xfs libcrc32c sd_mod
crc_t10dif crct10dif_generic sr_mod cdrom ata_generic crct10dif_common pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci virtio_net ata_piix libahci serio_raw net_failover drm failover crc32c_intel libata virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 23352 Comm: portainer Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff933bf572b180 ti: ffff933bf5e44000 task.ti: ffff933bf5e44000
RIP: 0010:[<ffffffff8251722a>] [<ffffffff8251722a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff933bf5e47c70 EFLAGS: 00010202
RAX: 0000000000000000 RBX: ffff933bf5e47c30 RCX: ffff933c63c20400
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff933bf5e47ca8 R08: ffff9339b5c29800 R09: ffffffff827869d9
R10: ffff933c63c9f160 R11: ffffe2df8ed7f600 R12: ffff933bf5e47c40
R13: ffffffff8278692f R14: ffff933bf5d4cc00 R15: 0000000000000001
FS: 000000c000080090(0000) GS:ffff933c63c80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006e6bd4 CR3: 000000041139c000 CR4: 00000000000006e0
Call Trace:
[<ffffffff8247e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8247e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff825ee666>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff825f012f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff825f0253>] tlb_finish_mmu+0x23/0x30
[<ffffffff825f371a>] zap_page_range+0xea/0x130
[<ffffffff824daef8>] ? wake_up_q+0x58/0x80
[<ffffffff825edc0d>] SyS_madvise+0x49d/0xac0
[<ffffffff825573eb>] ? __seccomp_filter+0x5b/0x300
[<ffffffff82b99ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff82b99ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff82b9a226>] tracesys+0xa6/0xcc
Code: 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 15 83 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #379506
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffad9170da
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffad87e258
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffad87e2cb
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffad9ee676
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffad9f013f
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffad9f0263
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffad9f372a
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffffad8dada8
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x58
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffffad9edc1d
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffffad95729b
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x5b
|
|
11
|
|
vmlinux |
0xffffffffad8dadf0
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x20
|
|
12
|
|
vmlinux |
0xffffffffad9156f0
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x80
|
|
13
|
tracesys
|
vmlinux |
0xffffffffadf9a226
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [metricbeat:28762]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13109(E) snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq iosf_mbi snd_seq_device snd_pcm crc32_pclmul ghash_clmulni_intel aesni_intel snd_timer lrw gf128mul glue_helper ablk_helper snd cryptd soundcore ppdev pcspkr joydev virtio_rng parport_pc virtio_balloon parport i2c_piix4 ip_tables xfs libcrc32c ata_generic pata_acpi virtio_console virtio_blk qxl drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul ata_piix crct10dif_common crc32c_intel serio_raw libata e1000 virtio_pci drm_panel_orientation_quirks virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 28762 Comm: metricbeat Tainted: P EL ------------ 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module+el8.3.0+7860+a7792d29 04/01/2014
task: ffff88015e003180 ti: ffff8801bb770000 task.ti: ffff8801bb770000
RIP: 0010:[<ffffffffad9170da>] [<ffffffffad9170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8801bb773c70 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000ad87ddd0 RCX: ffff8802bfc9fc10
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8801bb773ca8 R08: ffff88027fd52000 R09: ffffffffadb86c09
R10: ffff8802bfc1f160 R11: ffffd952c6d7cb00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8801bb773c20 R15: 0000000000000001
FS: 00007fe4fdffb700(0000) GS:ffff8802bfc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd7880eed28 CR3: 00000000b79a8000 CR4: 00000000001606f0
Call Trace:
[<ffffffffad87e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffad87e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffad9ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffad9f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffad9f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffad9f372a>] zap_page_range+0xea/0x130
[<ffffffffad8dada8>] ? wake_up_q+0x58/0x80
[<ffffffffad9edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffad95729b>] ? __seccomp_filter+0x5b/0x300
[<ffffffffad8dadf0>] ? wake_up_state+0x20/0x20
[<ffffffffad9156f0>] ? SyS_futex+0x80/0x190
[<ffffffffadf9a226>] tracesys+0xa6/0xcc
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 55 ae f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #350868
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9b5170da
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9b47e258
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9b47e2cb
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff9b5ee676
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff9b5f013f
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff9b5f0263
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff9b5f372a
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffff9b4dada8
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x58
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffff9b5edc1d
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffff9b55729b
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x5b
|
|
11
|
|
vmlinux |
0xffffffff9b4dadf0
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x20
|
|
12
|
tracesys
|
vmlinux |
0xffffffff9bb96226
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [filebeat:17852]
Modules linked in: tcp_lp rfcomm veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter binfmt_misc ip6_tables iptable_filter des_generic md4 nls_utf8 cifs ccm dns_resolver overlay(T) vmw_vsock_vmci_transport vsock bnep sunrpc snd_seq_midi snd_seq_midi_event squashfs loop ppdev snd_ens1371 snd_rawmidi snd_ac97_codec iosf_mbi crc32_pclmul ac97_bus vmw_balloon snd_seq btusb ghash_clmulni_intel snd_seq_device btrtl btbcm btintel bluetooth snd_pcm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev snd_timer rfkill snd sg soundcore
vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi scsi_transport_spi libata mptscsih mptbase crct10dif_pclmul crct10dif_common e1000 nfit crc32c_intel libnvdimm serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 5 PID: 17852 Comm: filebeat Kdump: loaded Tainted: G ------------ T 3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffffa04e43014200 ti: ffffa04e44c98000 task.ti: ffffa04e44c98000
RIP: 0010:[<ffffffff9b5170da>] [<ffffffff9b5170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa04e44c9bc70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009b47ddd0 RCX: ffffa053f3a206a0
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffffa04e44c9bca8 R08: ffffa0475d4a4000 R09: ffffffff9b786bf9
R10: ffffa053f3b5f160 R11: ffffd38f414ea800 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa04e44c9bc20 R15: 0000000000000001
FS: 00007f94e37fe700(0000) GS:ffffa053f3b40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00003e8a0067f000 CR3: 0000000db8b16000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9b47e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9b47e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9b5ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9b5f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9b5f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff9b5f372a>] zap_page_range+0xea/0x130
[<ffffffff9b4dada8>] ? wake_up_q+0x58/0x80
[<ffffffff9b5edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff9b55729b>] ? __seccomp_filter+0x5b/0x300
[<ffffffff9b4dadf0>] ? wake_up_state+0x20/0x20
[<ffffffff9bb96226>] tracesys+0xa6/0xcc
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 15 9c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #349777
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff953170de
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9527e258
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9527e2cb
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff953ee676
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff953f013f
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff953f0263
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff953f372a
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff953edc1d
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffff9535729b
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x5b
|
|
10
|
?
poll_select_copy_remaining
|
vmlinux |
0xffffffff95464924
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x144
|
|
11
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff95995ed5
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xa2
|
|
12
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff95995ec9
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x96
|
|
13
|
tracesys
|
vmlinux |
0xffffffff95996226
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [grafana-server:5405]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix nfit libata drm libnvdimm crct10dif_pclmul
crct10dif_common crc32c_intel mptspi scsi_transport_spi mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 7 PID: 5405 Comm: grafana-server Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018
task: ffff9db2d4526300 ti: ffff9dafda9d4000 task.ti: ffff9dafda9d4000
RIP: 0010:[<ffffffff953170de>] [<ffffffff953170de>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff9dafda9d7c70 EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffffff9527ddd0 RCX: ffff9db2eee606f0
RDX: 0000000000000001 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff9dafda9d7ca8 R08: ffff9db035c04400 R09: ffffffff95586bf9
R10: ffff9db2eefdf160 R11: fffff954c272ca00 R12: 0000000000000286
R13: ffff9dafda9d7c20 R14: ffff9db2ed8e9800 R15: 0000000000000001
FS: 00007f201a96e700(0000) GS:ffff9db2eefc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3890401d70 CR3: 000000011a070000 CR4: 00000000000407e0
Call Trace:
[<ffffffff9527e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9527e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff953ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff953f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff953f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff953f372a>] zap_page_range+0xea/0x130
[<ffffffff953edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffff9535729b>] ? __seccomp_filter+0x5b/0x300
[<ffffffff95464924>] ? poll_select_copy_remaining+0x144/0x180
[<ffffffff95995ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff95995ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff95996226>] tracesys+0xa6/0xcc
Code: 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 f5 95 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #344951
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffaa3170da
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffaa27e258
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffaa27e2cb
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffaa3ee676
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffaa3f013f
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffaa3f0263
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffaa3f372a
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffffaa2dada8
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x58
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffffaa3edc1d
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffffaa35729b
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x5b
|
|
11
|
tracesys
|
vmlinux |
0xffffffffaa996226
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [metricbeat:26567]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_12905(E) xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay snd_hda_codec_generic snd_hda_intel iosf_mbi snd_hda_codec crc32_pclmul ppdev snd_hda_core ghash_clmulni_intel joydev aesni_intel snd_hwdep snd_seq lrw snd_seq_device gf128mul glue_helper snd_pcm ablk_helper cryptd virtio_balloon virtio_rng snd_timer snd parport_pc i2c_piix4 parport soundcore pcspkr ip_tables xfs libcrc32c virtio_blk virtio_console ata_generic pata_acpi qxl drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm e1000 ata_piix crct10dif_pclmul crct10dif_common virtio_pci libata crc32c_intel virtio_ring serio_raw virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 26567 Comm: metricbeat Tainted: P E ------------ 3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module+el8.3.0+7860+a7792d29 04/01/2014
task: ffff9f686ef74200 ti: ffff9f68254c4000 task.ti: ffff9f68254c4000
RIP: 0010:[<ffffffffaa3170da>] [<ffffffffaa3170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9f68254c7c70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000aa27ddd0 RCX: ffff9f693fc20370
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff9f68254c7ca8 R08: ffff9f687aec2000 R09: ffffffffaa586bf9
R10: ffff9f693fd9f160 R11: ffffe5a50733ad00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9f68254c7c20 R15: 0000000000000001
FS: 00007f542f445700(0000) GS:ffff9f693fd80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c001bff000 CR3: 00000000b8ce6000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaa27e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaa27e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffaa3ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffaa3f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffaa3f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffaa3f372a>] zap_page_range+0xea/0x130
[<ffffffffaa2dada8>] ? wake_up_q+0x58/0x80
[<ffffffffaa3edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffaa35729b>] ? __seccomp_filter+0x5b/0x300
[<ffffffffaa996226>] tracesys+0xa6/0xcc
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 f5 aa f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #266040
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa4b16bda
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa4a7e258
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa4a7e2cb
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffa4bee176
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffa4befc3f
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa4befd63
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffa4bf322a
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffa4bed71d
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffffa4b56d9b
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x5b
|
|
10
|
|
vmlinux |
0xffffffffa4ae22e9
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xa9
|
|
11
|
tracesys
|
vmlinux |
0xffffffffa5196226
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [java:22608]
Modules linked in: tcp_lp veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq iosf_mbi crc32_pclmul snd_seq_device ghash_clmulni_intel snd_pcm aesni_intel snd_timer snd lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon soundcore joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom
crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common nfit crc32c_intel drm libnvdimm e1000 ata_piix serio_raw mptspi scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 1 PID: 22608 Comm: java Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9d9cb4d1c200 ti: ffff9d9be6fbc000 task.ti: ffff9d9be6fbc000
RIP: 0010:[<ffffffffa4b16bda>] [<ffffffffa4b16bda>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9d9be6fbfc70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a4a7ddd0 RCX: ffff9d9e2d620600
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9d9be6fbfca8 R08: ffff9d9b75ccb000 R09: ffffffffa4d866a9
R10: ffff9d9e2d65f160 R11: ffffdbbe0cb6de00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9d9be6fbfc20 R15: 0000000000000001
FS: 00007feada30a700(0000) GS:ffff9d9e2d640000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6d15428810 CR3: 00000000091d2000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffa4a7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa4a7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa4bee176>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa4befc3f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa4befd63>] tlb_finish_mmu+0x23/0x30
[<ffffffffa4bf322a>] zap_page_range+0xea/0x130
[<ffffffffa4bed71d>] SyS_madvise+0x49d/0xac0
[<ffffffffa4b56d9b>] ? __seccomp_filter+0x5b/0x300
[<ffffffffa4ae22e9>] ? pick_next_entity+0xa9/0x190
[<ffffffffa5196226>] tracesys+0xa6/0xcc
Code: 48 63 35 ce 8a c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 75 a5 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 8a c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #171883
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff90116eea
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9007e868
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9007e8db
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff901edf56
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff901efa1f
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff901efb43
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff901f300a
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffff900dada8
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x58
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffff901ed4fd
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffff901570ab
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x5b
|
|
11
|
|
vmlinux |
0xffffffff900dadf0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x20
|
|
12
|
|
vmlinux |
0xffffffff90115500
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x80
|
|
13
|
tracesys
|
vmlinux |
0xffffffff90796226
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [metricbeat:1202]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) ext4 mbcache jbd2 snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq iosf_mbi crc32_pclmul snd_seq_device ghash_clmulni_intel ppdev snd_pcm joydev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd virtio_rng snd_timer virtio_balloon snd parport_pc soundcore i2c_piix4 parport pcspkr ip_tables xfs libcrc32c ata_generic pata_acpi virtio_blk virtio_console qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul crct10dif_common crc32c_intel e1000 floppy
virtio_pci virtio_ring virtio serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 1202 Comm: metricbeat Tainted: G ------------ T 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module_el8.3.0+555+a55c8938 04/01/2014
task: ffff92f1f8bf4200 ti: ffff92f1fa390000 task.ti: ffff92f1fa390000
RIP: 0010:[<ffffffff90116eea>] [<ffffffff90116eea>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff92f1fa393c70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009007e6a0 RCX: ffff92f23fc20348
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff92f1fa393ca8 R08: ffff92f1f43c1c00 R09: ffffffff90386549
R10: ffff92f23fd1f160 R11: fffff5e301253800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff92f1fa393c20 R15: 0000000000000001
FS: 00007f115bf7f700(0000) GS:ffff92f23fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1193067000 CR3: 00000000b8c16000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9007e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9007e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff901edf56>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff901efa1f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff901efb43>] tlb_finish_mmu+0x23/0x30
[<ffffffff901f300a>] zap_page_range+0xea/0x130
[<ffffffff900dada8>] ? wake_up_q+0x58/0x80
[<ffffffff901ed4fd>] SyS_madvise+0x49d/0xac0
[<ffffffff901570ab>] ? __seccomp_filter+0x5b/0x300
[<ffffffff900dadf0>] ? wake_up_state+0x20/0x20
[<ffffffff90115500>] ? SyS_futex+0x80/0x190
[<ffffffff90796226>] tracesys+0xa6/0xcc
Code: 48 63 35 fe 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 d5 90 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 87 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #171865
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9bb16dda
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9ba7e868
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9ba7e8db
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff9bbede26
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff9bbef8ef
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff9bbefa13
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff9bbf2eda
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffff9badb078
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x58
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffff9bbed3cd
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffff9bb56f9b
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x5b
|
|
11
|
tracesys
|
vmlinux |
0xffffffff9c196226
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [auditbeat:4275]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) ext4 mbcache jbd2 snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep iosf_mbi crc32_pclmul snd_seq snd_seq_device ghash_clmulni_intel snd_pcm aesni_intel ppdev lrw joydev parport_pc gf128mul glue_helper parport virtio_balloon snd_timer virtio_rng ablk_helper cryptd pcspkr snd i2c_piix4 soundcore ip_tables xfs libcrc32c ata_generic pata_acpi virtio_console virtio_blk qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul crct10dif_common drm libata e1000 crc32c_intel serio_raw
virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 4275 Comm: auditbeat Tainted: G ------------ T 3.10.0-1160.21.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.11.0-2.el7 04/01/2014
task: ffff89e27cbe4200 ti: ffff89e32ead0000 task.ti: ffff89e32ead0000
RIP: 0010:[<ffffffff9bb16dda>] [<ffffffff9bb16dda>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff89e32ead3c70 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 000800009ba7e6a0 RCX: ffff89e33fca0348
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff89e32ead3ca8 R08: ffff89e27cf41c00 R09: ffffffff9bd86419
R10: ffff89e33fd1f160 R11: ffffda29401f3200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff89e32ead3c20 R15: 0000000000000001
FS: 00007f3ced7fb700(0000) GS:ffff89e33fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000ea3000 CR3: 00000000029a6000 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9ba7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9ba7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9bbede26>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9bbef8ef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9bbefa13>] tlb_finish_mmu+0x23/0x30
[<ffffffff9bbf2eda>] zap_page_range+0xea/0x130
[<ffffffff9badb078>] ? wake_up_q+0x58/0x80
[<ffffffff9bbed3cd>] SyS_madvise+0x49d/0xac0
[<ffffffff9bb56f9b>] ? __seccomp_filter+0x5b/0x300
[<ffffffff9c196226>] tracesys+0xa6/0xcc
Code: 48 63 35 0e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 75 9c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 dd 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #93568
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa8b16e4e
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa8a7e868
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa8a7e8db
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffa8bede86
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffa8bef94f
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa8befa73
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffa8bf2f3a
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffa8bed42d
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffffa8b56ffb
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x5b
|
|
10
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffa9194ed5
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xa2
|
|
11
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffa9194ec9
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x96
|
|
12
|
tracesys
|
vmlinux |
0xffffffffa9195226
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [elasticsearch[n:86179]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev snd_timer snd soundcore vmw_balloon coretemp pcspkr joydev vmw_vmci sg i2c_piix4 parport_pc parport sch_fq ip_tables xfs libcrc32c
sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm ata_piix drm nfit libata mptspi serio_raw scsi_transport_spi libnvdimm e1000 mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 86179 Comm: elasticsearch[n Kdump: loaded Not tainted 3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8bce06be5280 ti: ffff8bd18ea9c000 task.ti: ffff8bd18ea9c000
RIP: 0010:[<ffffffffa8b16e4e>] [<ffffffffa8b16e4e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8bd18ea9fc70 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00080000a8a7e6a0 RCX: ffff8bd4dd6e0600
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8bd18ea9fca8 R08: ffff8bce32f53c00 R09: ffffffffa8d86479
R10: ffff8bd4dd69f160 R11: ffffee4e8586ec00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8bd18ea9fc20 R15: 0000000000000001
FS: 00007fb0ec7c5700(0000) GS:ffff8bd4dd680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f25de8ba000 CR3: 000000014dbf2000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa8a7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa8a7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa8bede86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa8bef94f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa8befa73>] tlb_finish_mmu+0x23/0x30
[<ffffffffa8bf2f3a>] zap_page_range+0xea/0x130
[<ffffffffa8bed42d>] SyS_madvise+0x49d/0xac0
[<ffffffffa8b56ffb>] ? __seccomp_filter+0x5b/0x300
[<ffffffffa9194ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffffa9194ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffa9195226>] tracesys+0xa6/0xcc
Code: 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 75 a9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 ad 86 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #24259
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa9b16c5a
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa9a7e868
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa9a7e8db
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffa9bedd46
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffa9bef80f
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa9bef933
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffa9bf2dfa
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffa9bed2ed
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffffa9b56e4b
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x5b
|
|
10
|
tracesys
|
vmlinux |
0xffffffffaa195226
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [java:32848]
Modules linked in: tcp_lp xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event xfs libcrc32c crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec aesni_intel lrw gf128mul ac97_bus pcspkr joydev ppdev snd_seq snd_seq_device snd_pcm glue_helper ablk_helper cryptd vmw_balloon snd_timer snd soundcore sg parport_pc parport vmw_vmci i2c_piix4 ip_tables ext4 mbcache jbd2
sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix mptspi drm nfit scsi_transport_spi mptscsih libata crct10dif_pclmul libnvdimm crct10dif_common crc32c_intel serio_raw mptbase e1000 drm_panel_orientation_quirks fuse
CPU: 13 PID: 32848 Comm: java Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff88d2e2cfe300 ti: ffff88d2d9c38000 task.ti: ffff88d2d9c38000
RIP: 0010:[<ffffffffa9b16c5a>] [<ffffffffa9b16c5a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff88d2d9c3bc70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a9a7e6a0 RCX: ffff88d5086207e0
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff88d2d9c3bca8 R08: ffff88c97c366000 R09: ffffffffa9d85fd9
R10: ffff88d50895f160 R11: ffffdd9768c19400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff88d2d9c3bc20 R15: 0000000000000001
FS: 00007f8f90ee1700(0000) GS:ffff88d508940000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb6e62cd000 CR3: 0000000712302000 CR4: 00000000003407e0
Call Trace:
[<ffffffffa9a7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa9a7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa9bedd46>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa9bef80f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa9bef933>] tlb_finish_mmu+0x23/0x30
[<ffffffffa9bf2dfa>] zap_page_range+0xea/0x130
[<ffffffffa9bed2ed>] SyS_madvise+0x49d/0xac0
[<ffffffffa9b56e4b>] ? __seccomp_filter+0x5b/0x300
[<ffffffffaa195226>] tracesys+0xa6/0xcc
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 75 aa f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89