Complete report #1409317
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa891173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa887a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa887a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffffa89e4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa89e5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffffa89e87c2
Build id: 3.10.0-957.el7.x86_64
|
0x142
|
|
7
|
SyS_madvise
|
vmlinux |
0xffffffffa89e368d
Build id: 3.10.0-957.el7.x86_64
|
0x49d
|
|
8
|
?
poll_select_copy_remaining
|
vmlinux |
0xffffffffa8a57044
Build id: 3.10.0-957.el7.x86_64
|
0x144
|
|
9
|
system_call_fastpath
|
vmlinux |
0xffffffffa8f74ddb
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [docker-containe:9966]
Modules linked in: tcp_lp ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmnet(OE) vmmon(OE) fuse vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 ppdev aesni_intel lrw gf128mul vmw_balloon snd_rawmidi glue_helper ablk_helper snd_ac97_codec cryptd ac97_bus snd_seq snd_seq_device snd_pcm snd_timer joydev snd pcspkr sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod vmwgfx crc_t10dif crct10dif_generic sr_mod cdrom drm_kms_helper ata_generic syscopyarea pata_acpi sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul
crct10dif_common crc32c_intel ttm mptspi scsi_transport_spi ata_piix drm mptscsih e1000 libata mptbase serio_raw drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 9966 Comm: docker-containe Tainted: G OE ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff95042a794100 ti: ffff950416e94000 task.ti: ffff950416e94000
RIP: 0010:[<ffffffffa891173a>] [<ffffffffa891173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff950416e97c78 EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00000000000000fc RCX: ffff9504303e0650
RDX: 0000000000000007 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff950416e97cb0 R08: ffff95023a8b4400 R09: ffffffffa8b755c9
R10: ffff95043035f160 R11: ffffdbdac0d39e00 R12: 0000000000000296
R13: ffff950416e97c28 R14: 0000000000000001 R15: ffffdbdac0a12380
FS: 00007f722335e700(0000) GS:ffff950430340000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002615094a6038 CR3: 00000000b9138000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa887a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa887a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa89e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffa89e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffa89e87c2>] zap_page_range+0x142/0x190
[<ffffffffa89e368d>] SyS_madvise+0x49d/0xac0
[<ffffffffa8a57044>] ? poll_select_copy_remaining+0x144/0x180
[<ffffffffa8f74ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 54 a9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #275918
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff86d07e3a
Build id: 3.10.0-862.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff86c77018
Build id: 3.10.0-862.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff86c77088
Build id: 3.10.0-862.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.75
|
vmlinux |
0xffffffff86dc0497
Build id: 3.10.0-862.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff86dc1c75
Build id: 3.10.0-862.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffff86dc4a02
Build id: 3.10.0-862.el7.x86_64
|
0x142
|
|
7
|
SyS_madvise
|
vmlinux |
0xffffffff86dbfafd
Build id: 3.10.0-862.el7.x86_64
|
0x3cd
|
|
8
|
?
poll_select_copy_remaining
|
vmlinux |
0xffffffff86e30801
Build id: 3.10.0-862.el7.x86_64
|
0x121
|
|
9
|
system_call_fastpath
|
vmlinux |
0xffffffff8731f7d5
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [dockerd-current:1415]
Modules linked in: fuse xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 sb_edac coretemp snd_rawmidi snd_ac97_codec ac97_bus snd_seq iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_seq_device
snd_pcm lrw gf128mul glue_helper snd_timer snd ppdev ablk_helper soundcore cryptd vmw_balloon sg nfit pcspkr joydev vmw_vmci i2c_piix4 shpchp libnvdimm parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix mptspi libata scsi_transport_spi serio_raw mptscsih e1000 mptbase i2c_core
CPU: 6 PID: 1415 Comm: dockerd-current Kdump: loaded Tainted: G L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9e4e371bbf40 ti: ffff9e4e38c6c000 task.ti: ffff9e4e38c6c000
RIP: 0010:[<ffffffff86d07e3a>] [<ffffffff86d07e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9e4e38c6fc78 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00000000000000fc RCX: ffff9e503365cf78
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9e4e38c6fcb0 R08: ffff9e4efb4a6400 R09: ffffffff86f4a3d9
R10: ffff9e503379bb00 R11: fffffa428ac83c00 R12: 0000000000000296
R13: ffff9e4e38c6fc28 R14: 0000000000000001 R15: fffffa4289a91e80
FS: 00007f6078a22700(0000) GS:ffff9e5033780000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f15b1ffafb8 CR3: 000000029feaa000 CR4: 00000000001607e0
Call Trace:
[<ffffffff86c77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff86c77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff86dc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff86dc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff86dc4a02>] zap_page_range+0x142/0x190
[<ffffffff86dbfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff86e30801>] ? poll_select_copy_remaining+0x121/0x150
[<ffffffff8731f7d5>] system_call_fastpath+0x1c/0x21
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 93 87 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89
Complete report #230755
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffaf707e3e
Build id: 3.10.0-862.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffaf677018
Build id: 3.10.0-862.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffaf677088
Build id: 3.10.0-862.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.75
|
vmlinux |
0xffffffffaf7c0497
Build id: 3.10.0-862.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffaf7c1c75
Build id: 3.10.0-862.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffffaf7c4a02
Build id: 3.10.0-862.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffffaf6cee38
Build id: 3.10.0-862.el7.x86_64
|
0x58
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffaf7bfafd
Build id: 3.10.0-862.el7.x86_64
|
0x3cd
|
|
9
|
?
poll_select_copy_remaining
|
vmlinux |
0xffffffffaf830801
Build id: 3.10.0-862.el7.x86_64
|
0x121
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 43s! [swapper/2:0]
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 36s! [etcd:2312]
Modules linked in: veth vxlan ip6_udp_tunnel udp_tunnel rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs ip6table_nat lockd nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 grace fscache xt_set xt_mark ip_set_hash_ipportnet ip_set_hash_ipport ip_set_bitmap_port ip_set_hash_ipportip ip_set dummy xt_comment nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 overlay(T) tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper
ppdev cryptd joydev pcspkr vmw_balloon sg parport_pc parport i2c_piix4 vmw_vmci shpchp sch_fq_codel ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ata_piix libahci drm libata crct10dif_pclmul crct10dif_common mptspi scsi_transport_spi mptscsih crc32c_intel serio_raw mptbase vmxnet3 i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 2312 Comm: etcd Kdump: loaded Tainted: G L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8ea63ba6bf40 ti: ffff8ea982f34000 task.ti: ffff8ea982f34000
RIP: 0010:[<ffffffffaf707e3e>] [<ffffffffaf707e3e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8ea982f37c78 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00000000000000fc RCX: ffff8ea9bfcdcf78
RDX: 0000000000000003 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8ea982f37cb0 R08: ffff8ea6f9cc8400 R09: ffffffffaf94a3d9
R10: ffff8ea9bfddbb00 R11: ffffede2cf41b600 R12: 0000000000000296
R13: ffff8ea982f37c28 R14: 0000000000000001 R15: ffffffffaf7f8139
FS: 000000c420062090(0000) GS:ffff8ea9bfdc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000c4200cd010 CR3: 0000000402ed4000 CR4: 00000000000007e0
Call Trace:
[<ffffffffaf677018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaf677088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffaf7c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffaf7c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffaf7c4a02>] zap_page_range+0x142/0x190
[<ffffffffaf6cee38>] ? wake_up_q+0x58/0x80
[<ffffffffaf7bfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffffaf830801>] ? poll_select_copy_remaining+0x121/0x150
Complete report #81285
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8a31188a
Build id: 3.10.0-957.12.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8a27a5c8
Build id: 3.10.0-957.12.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8a27a638
Build id: 3.10.0-957.12.1.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffff8a3e44c7
Build id: 3.10.0-957.12.1.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff8a3e5dc5
Build id: 3.10.0-957.12.1.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffff8a3e8c12
Build id: 3.10.0-957.12.1.el7.x86_64
|
0x142
|
|
7
|
SyS_madvise
|
vmlinux |
0xffffffff8a3e3add
Build id: 3.10.0-957.12.1.el7.x86_64
|
0x49d
|
|
8
|
?
poll_select_copy_remaining
|
vmlinux |
0xffffffff8a457674
Build id: 3.10.0-957.12.1.el7.x86_64
|
0x144
|
|
9
|
system_call_fastpath
|
vmlinux |
0xffffffff8a975d9b
Build id: 3.10.0-957.12.1.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [dockerd-current:1451]
Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc nfsv3 nfs_acl nfs lockd grace fscache overlay(T) sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd i2c_piix4 parport_pc parport pcspkr virtio_balloon joydev ip_tables xfs libcrc32c ata_generic pata_acpi virtio_net virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel serio_raw libata virtio_pci virtio_ring drm_panel_orientation_quirks virtio floppy
CPU: 1 PID: 1451 Comm: dockerd-current Tainted: G L ------------ T 3.10.0-957.12.1.el7.x86_64 #1
Hardware name: RDO OpenStack Compute, BIOS 1.11.0-2.el7 04/01/2014
task: ffff94512aa38000 ti: ffff945121394000 task.ti: ffff945121394000
RIP: 0010:[<ffffffff8a31188a>] [<ffffffff8a31188a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff945121397c78 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff94513fc20490
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff945121397cb0 R08: ffff944e7a2f6000 R09: ffffffff8a575c59
R10: ffff94513fc9f160 R11: fffff4e890bb6a00 R12: 0000000000000296
R13: ffff945121397c28 R14: 0000000000000001 R15: ffffffff8a41d51a
FS: 00007f80f2c4e700(0000) GS:ffff94513fc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1539490000 CR3: 0000000428fac000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8a27a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8a27a638>] flush_tlb_mm_range+0x68/0x140
[<ffffffff8a3e44c7>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff8a3e5dc5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8a3e8c12>] zap_page_range+0x142/0x190
[<ffffffff8a3e3add>] SyS_madvise+0x49d/0xac0
[<ffffffff8a457674>] ? poll_select_copy_remaining+0x144/0x180
[<ffffffff8a975d9b>] system_call_fastpath+0x22/0x27
Code: 48 63 35 3e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 ba f4 8a f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1384792
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa3b1173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa3a7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa3a7a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffffa3be4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa3be5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffffa3be87c2
Build id: 3.10.0-957.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffffa3ad6768
Build id: 3.10.0-957.el7.x86_64
|
0x58
|
|
8
|
|
vmlinux |
0xffffffffa3b0cca0
Build id: 3.10.0-957.el7.x86_64
|
0x90
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffffa3be368d
Build id: 3.10.0-957.el7.x86_64
|
0x49d
|
|
10
|
tracesys
|
vmlinux |
0xffffffffa417505b
Build id: 3.10.0-957.el7.x86_64
|
0xa3
|
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [ActiveMQ Broker:69108]
Modules linked in: nf_conntrack_netlink xt_nat veth nfnetlink ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_addrtype overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev snd_ens1371 pcspkr snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport sch_fq_codel ip_tables xfs libcrc32c
sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm e1000 libata mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 69108 Comm: ActiveMQ Broker Kdump: loaded Tainted: G L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffffa14bec33e180 ti: ffffa14baea90000 task.ti: ffffa14baea90000
RIP: 0010:[<ffffffffa3b1173a>] [<ffffffffa3b1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa14baea93c78 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00000000000000fc RCX: ffffa14c356606a0
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffffa14baea93cb0 R08: ffffa14b7c85cc00 R09: ffffffffa3d755c9
R10: ffffa14c357df160 R11: ffffd61408a9b200 R12: 0000000000000296
R13: ffffa14baea93c28 R14: 0000000000000001 R15: ffffd614050156c0
FS: 00007f8e795d5700(0000) GS:ffffa14c357c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8f8f0d12b0 CR3: 00000001b63b8000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffa3a7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa3a7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa3be4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffa3be5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffa3be87c2>] zap_page_range+0x142/0x190
[<ffffffffa3ad6768>] ? wake_up_q+0x58/0x80
[<ffffffffa3b0cca0>] ? futex_wake+0x90/0x180
[<ffffffffa3be368d>] SyS_madvise+0x49d/0xac0
[<ffffffffa417505b>] tracesys+0xa3/0xc9
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 74 a4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1048038
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
generic_exec_single
|
vmlinux |
0xffffffff9d30779a
Build id: 3.10.0-862.el7.x86_64
|
0xfa
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9d277018
Build id: 3.10.0-862.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9d277088
Build id: 3.10.0-862.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.75
|
vmlinux |
0xffffffff9d3c0497
Build id: 3.10.0-862.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff9d3c1c75
Build id: 3.10.0-862.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffff9d3c4a02
Build id: 3.10.0-862.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffff9d3cdf0c
Build id: 3.10.0-862.el7.x86_64
|
0x3ac
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff9d3bfafd
Build id: 3.10.0-862.el7.x86_64
|
0x3cd
|
|
9
|
|
vmlinux |
0xffffffff9d3b35eb
Build id: 3.10.0-862.el7.x86_64
|
0xeb
|
|
10
|
system_call_fastpath
|
vmlinux |
0xffffffff9d91f7d5
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [java:576282]
Modules linked in: ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi snd_ens1371 snd_rawmidi snd_ac97_codec crc32_pclmul ghash_clmulni_intel ac97_bus aesni_intel lrw snd_seq gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm ppdev vmw_balloon snd_timer snd soundcore joydev pcspkr vmw_vmci sg i2c_piix4 shpchp nfit libnvdimm parport_pc parport binfmt_misc xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel mptspi libata scsi_transport_spi e1000 mptscsih serio_raw i2c_core mptbase [last unloaded: ip_tables]
CPU: 1 PID: 576282 Comm: java Tainted: G W L ------------ 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8b7bfd23af70 ti: ffff8b7ea77ec000 task.ti: ffff8b7ea77ec000
RIP: 0010:[<ffffffff9d30779a>] [<ffffffff9d30779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff8b7ea77efc30 EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff8b7ea77efc00 RCX: 0000000000000008
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [java:576257]
Modules linked in: ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi snd_ens1371 snd_rawmidi snd_ac97_codec crc32_pclmul ghash_clmulni_intel ac97_bus aesni_intel lrw snd_seq gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm ppdev vmw_balloon snd_timer snd soundcore joydev pcspkr vmw_vmci sg i2c_piix4 shpchp nfit libnvdimm parport_pc parport binfmt_misc xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel mptspi libata scsi_transport_spi e1000 mptscsih serio_raw i2c_core mptbase [last unloaded: ip_tables]
CPU: 2 PID: 576257 Comm: java Tainted: G W L ------------ 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8b7cc1e75ee0 ti: ffff8b7309ffc000 task.ti: ffff8b7309ffc000
RIP: 0010:[<ffffffff9d307e3a>] [<ffffffff9d307e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b7309fffc78 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00000000000000fc RCX: ffff8b7fbd6dcdb8
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8b7309fffcb0 R08: ffff8b712966b800 R09: ffffffff9d54a3d9
R10: ffff8b7fbd69bb00 R11: ffffd4023b94e000 R12: 0000000000000296
R13: ffff8b7309fffc28 R14: 0000000000000001 R15: ffffd402359ddfc0
FS: 00002b339a8d6700(0000) GS:ffff8b7fbd680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b898d01e005 CR3: 000000004e150000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff9d277018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9d277088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff9d3c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff9d3c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff9d3c4a02>] zap_page_range+0x142/0x190
[<ffffffff9d3cdf0c>] ? do_mmap+0x3ac/0x4d0
[<ffffffff9d3bfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff9d3b35eb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff9d91f7d5>] system_call_fastpath+0x1c/0x21
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 f3 9d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89
Complete report #1001883
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
|
0x202
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.63
|
vmlinux |
0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
|
0x33
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffff811b08fa
Build id: 3.10.0-693.el7.x86_64
|
0x13a
|
|
7
|
SyS_madvise
|
vmlinux |
0xffffffff811ac71d
Build id: 3.10.0-693.el7.x86_64
|
0x3cd
|
|
8
|
|
vmlinux |
0xffffffff811a12c2
Build id: 3.10.0-693.el7.x86_64
|
0xe2
|
|
9
|
system_call_fastpath
|
vmlinux |
0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [LogStash::Runne:8098]
Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event crc32_pclmul ghash_clmulni_intel ppdev snd_ens1371 snd_rawmidi vmw_balloon aesni_intel lrw snd_ac97_codec gf128mul glue_helper ablk_helper cryptd ac97_bus snd_seq snd_seq_device snd_pcm pcspkr joydev snd_timer sg snd nfit soundcore libnvdimm vmw_vmci shpchp i2c_piix4 parport_pc parport
nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi drm scsi_transport_spi crct10dif_pclmul ata_piix crct10dif_common mptscsih mptbase libata crc32c_intel e1000 serio_raw i2c_core dm_mirror dm_region_hash dm_log dm_mod synosnap(OE)
CPU: 4 PID: 8098 Comm: LogStash::Runne Tainted: G OEL ------------ 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8800a22c8000 ti: ffff880628304000 task.ti: ffff880628304000
RIP: 0010:[<ffffffff810f99a2>] [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff880628307cb0 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008106e4a0 RCX: ffff88081d61af98
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff880628307ce8 R08: ffff880174424c00 R09: ffffffff813227d9
R10: ffff88081d719c80 R11: ffffea000288d600 R12: 0000000000000282
R13: 0000000000000282 R14: ffff880628307c60 R15: 0000000000000001
FS: 00007f85118a2700(0000) GS:ffff88081d700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000008 CR3: 0000000758289000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000000011189cfff ffff880174359608 ffff8801743592c0 0000000000000000
ffffffffffffffff ffff880174359608 ffff8801743592c0 ffff880628307d38
ffffffff8106e668 ffff8801743592c0 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b08fa>] zap_page_range+0x13a/0x180
[<ffffffff811ac71d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff811a12c2>] ? vm_mmap_pgoff+0xe2/0x110
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89
Complete report #842343
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa631173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa627a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa627a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffffa63e4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa63e5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffffa63e87c2
Build id: 3.10.0-957.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffffa63f28f8
Build id: 3.10.0-957.el7.x86_64
|
0x378
|
|
8
|
|
vmlinux |
0xffffffffa64404d3
Build id: 3.10.0-957.el7.x86_64
|
0x93
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffffa63e368d
Build id: 3.10.0-957.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffffa63d612b
Build id: 3.10.0-957.el7.x86_64
|
0xeb
|
|
11
|
system_call_fastpath
|
vmlinux |
0xffffffffa6974ddb
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [java:62469]
Modules linked in: vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev snd_hda_codec_generic snd_hda_intel snd_hda_codec vmw_balloon snd_hda_core snd_hwdep snd_seq sg snd_seq_device snd_pcm snd_timer snd joydev pcspkr vmw_vmci soundcore i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx serio_raw drm_kms_helper ahci syscopyarea sysfillrect sysimgblt fb_sys_fops libahci e1000e ttm drm ata_piix mptsas ptp pps_core libata scsi_transport_sas mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 62469 Comm: java Kdump: loaded Tainted: G L ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9b3ce2d18000 ti: ffff9b3d26084000 task.ti: ffff9b3d26084000
RIP: 0010:[<ffffffffa631173a>] [<ffffffffa631173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9b3d26087c78 EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00000000000000fc RCX: ffff9b40ad720678
RDX: 0000000000000004 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9b3d26087cb0 R08: ffff9b3df9cb6400 R09: ffffffffa65755c9
R10: ffff9b40ad79f160 R11: ffffe663404d3800 R12: 0000000000000296
R13: ffff9b3d26087c28 R14: 0000000000000001 R15: ffffe66340443080
FS: 00007f7ee298d700(0000) GS:ffff9b40ad780000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4b519e2f58 CR3: 000000042a276000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa627a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa627a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa63e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffa63e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffa63e87c2>] zap_page_range+0x142/0x190
[<ffffffffa63f28f8>] ? do_mmap+0x378/0x530
[<ffffffffa64404d3>] ? do_sync_read+0x93/0xe0
[<ffffffffa63e368d>] SyS_madvise+0x49d/0xac0
[<ffffffffa63d612b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffa6974ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 f4 a6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #820913
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb991173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb987a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffb987a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffffb99e4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffb99e5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffffb99e87c2
Build id: 3.10.0-957.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffffb99f28f8
Build id: 3.10.0-957.el7.x86_64
|
0x378
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffb99e368d
Build id: 3.10.0-957.el7.x86_64
|
0x49d
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [nacos-grpc-clie:74227]
Modules linked in: nfnetlink_queue nfnetlink_log bluetooth rfkill xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq joydev snd_seq_device snd_pcm pcspkr snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic
pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm e1000 ata_piix mptspi scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 74227 Comm: nacos-grpc-clie Kdump: loaded Tainted: G L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff8a6d9bee30c0 ti: ffff8a69ec5a8000 task.ti: ffff8a69ec5a8000
RIP: 0010:[<ffffffffb991173a>] [<ffffffffb991173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8a69ec5abc78 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8a6dc5020678
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8a69ec5abcb0 R08: ffff8a6af86cd000 R09: ffffffffb9b755c9
R10: ffff8a6dc519f160 R11: ffffe859010fbe00 R12: 0000000000000296
R13: ffff8a69ec5abc28 R14: 0000000000000001 R15: ffffe8590f95c400
FS: 00007f20167d2700(0000) GS:ffff8a6dc5180000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa43f636000 CR3: 00000000ad122000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb987a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb987a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb99e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffb99e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffb99e87c2>] zap_page_range+0x142/0x190
[<ffffffffb99f28f8>] ? do_mmap+0x378/0x530
[<ffffffffb99e368d>] SyS_madvise+0x49d/0xac0
Complete report #583996
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa3d1173e
Build id: 3.10.0-957.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa3c7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa3c7a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffffa3de4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa3de5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffffa3de87c2
Build id: 3.10.0-957.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffffa3df28f8
Build id: 3.10.0-957.el7.x86_64
|
0x378
|
|
8
|
|
vmlinux |
0xffffffffa3e404d3
Build id: 3.10.0-957.el7.x86_64
|
0x93
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffffa3de368d
Build id: 3.10.0-957.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffffa3dd612b
Build id: 3.10.0-957.el7.x86_64
|
0xeb
|
|
11
|
system_call_fastpath
|
vmlinux |
0xffffffffa4374ddb
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [java:5106]
Modules linked in: vmw_vsock_vmci_transport vsock vfat fat crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd pcspkr joydev sg parport_pc parport tpm_crb vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx e1000e crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm_kms_helper ptp pps_core syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci mptsas libahci drm ata_piix scsi_transport_sas nfit mptscsih libata libnvdimm mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 5106 Comm: java Kdump: loaded Tainted: G W L ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.17369862.B64.2012240522 12/24/2020
task: ffff889ac7320000 ti: ffff88a2ccba8000 task.ti: ffff88a2ccba8000
RIP: 0010:[<ffffffffa3d1173e>] [<ffffffffa3d1173e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff88a2ccbabc78 EFLAGS: 00000202
RAX: 000000000000000e RBX: 00000000000000fc RCX: ffff88a2df5a0600
RDX: 000000000000000e RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff88a2ccbabcb0 R08: ffff889c34bfbc00 R09: ffffffffa3f755c9
R10: ffff88a2df2df160 R11: fffffdf90750dc00 R12: 0000000000000296
R13: ffff88a2ccbabc28 R14: 0000000000000001 R15: fffffdf92018b8c0
FS: 00007f9278bdb700(0000) GS:ffff88a2df2c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2212c90700 CR3: 000000000da4e000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffffa3c7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa3c7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa3de4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffa3de5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffa3de87c2>] zap_page_range+0x142/0x190
[<ffffffffa3df28f8>] ? do_mmap+0x378/0x530
[<ffffffffa3e404d3>] ? do_sync_read+0x93/0xe0
[<ffffffffa3de368d>] SyS_madvise+0x49d/0xac0
[<ffffffffa3dd612b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffa4374ddb>] system_call_fastpath+0x22/0x27
Code: 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 94 a4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #523541
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff88f1185a
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff88e7a5c8
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff88e7a638
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffff88fe4427
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff88fe5d25
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffff88fe8b72
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffff88ff2ca8
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x378
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff88fe3a3d
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffff88fd64db
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xeb
|
|
10
|
system_call_fastpath
|
vmlinux |
0xffffffff89574ddb
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [java:7738]
Modules linked in: vfat fat uas usb_storage cmac arc4 md4 nls_utf8 cifs ccm dns_resolver fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun nfsv3 nfs_acl bridge nfs stp llc lockd grace fscache ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter mvfs(OE) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec joydev ac97_bus snd_seq pcspkr snd_seq_device snd_pcm snd_timer snd sg soundcore vmw_vmci i2c_piix4 nfit libnvdimm parport_pc
parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw ata_piix syscopyarea sysfillrect sysimgblt libata fb_sys_fops ttm mptspi scsi_transport_spi mptscsih drm e1000 mptbase drm_panel_orientation_quirks floppy
CPU: 6 PID: 7738 Comm: java Kdump: loaded Tainted: G W OE ------------ 3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8abe59690000 ti: ffff8abdfc210000 task.ti: ffff8abdfc210000
RIP: 0010:[<ffffffff88f1185a>] [<ffffffff88f1185a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8abdfc213c78 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000000000001b700 RCX: ffff8abe764e0678
RDX: 0000000000000003 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff8abdfc213cb0 R08: ffff8ab9b6644c00 R09: ffffffff89175ba9
R10: ffff8abe7659f160 R11: ffffd236c20e4800 R12: ffff8abdfc213c28
R13: 0000000000000001 R14: 0000000000000001 R15: ffffd236ca61ee00
FS: 00007fc666fef700(0000) GS:ffff8abe76580000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffdebc3ebc8 CR3: 00000006172fc000 CR4: 00000000001607e0
Call Trace:
[<ffffffff88e7a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff88e7a638>] flush_tlb_mm_range+0x68/0x140
[<ffffffff88fe4427>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff88fe5d25>] tlb_finish_mmu+0x55/0x60
[<ffffffff88fe8b72>] zap_page_range+0x142/0x190
[<ffffffff88ff2ca8>] ? do_mmap+0x378/0x530
[<ffffffff88fe3a3d>] SyS_madvise+0x49d/0xac0
[<ffffffff88fd64db>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff89574ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 2e 82 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 b4 89 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 fd 81 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #482067
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa8b1173e
Build id: 3.10.0-957.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa8a7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa8a7a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffffa8be4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa8be5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffffa8be87c2
Build id: 3.10.0-957.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffffa8bf28f8
Build id: 3.10.0-957.el7.x86_64
|
0x378
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffa8be368d
Build id: 3.10.0-957.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffffa8bd612b
Build id: 3.10.0-957.el7.x86_64
|
0xeb
|
|
10
|
system_call_fastpath
|
vmlinux |
0xffffffffa9174ddb
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [java:14909]
Modules linked in: vmw_vsock_vmci_transport vsock vfat fat crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper pcspkr cryptd ppdev vmw_balloon joydev sg vmw_vmci parport_pc parport i2c_piix4 tpm_crb ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx e1000e serio_raw ptp drm_kms_helper pps_core syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptsas scsi_transport_sas ahci mptscsih drm mptbase nfit libahci ata_piix libnvdimm libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 14909 Comm: java Kdump: loaded Tainted: G W L ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.17369862.B64.2012240522 12/24/2020
task: ffff91f33afc2080 ti: ffff91f3127f0000 task.ti: ffff91f3127f0000
RIP: 0010:[<ffffffffa8b1173e>] [<ffffffffa8b1173e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff91f3127f3c78 EFLAGS: 00000202
RAX: 000000000000000e RBX: 00000000000000fc RCX: ffff91fadf5a0678
RDX: 000000000000000e RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff91f3127f3cb0 R08: ffff91f434bfc800 R09: ffffffffa8d755c9
R10: ffff91fadf39f160 R11: ffffdbee82ba9e00 R12: 0000000000000296
R13: ffff91f3127f3c28 R14: 0000000000000001 R15: ffffdbee8b4434c0
FS: 00007f2a841c7700(0000) GS:ffff91fadf380000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc5313a1000 CR3: 000000034eb7c000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffffa8a7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa8a7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa8be4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffa8be5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffa8be87c2>] zap_page_range+0x142/0x190
[<ffffffffa8bf28f8>] ? do_mmap+0x378/0x530
[<ffffffffa8be368d>] SyS_madvise+0x49d/0xac0
[<ffffffffa8bd612b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffa9174ddb>] system_call_fastpath+0x22/0x27
Code: 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 74 a9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #436604
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff99307e3a
Build id: 3.10.0-862.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff99277018
Build id: 3.10.0-862.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff99277088
Build id: 3.10.0-862.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.75
|
vmlinux |
0xffffffff993c0497
Build id: 3.10.0-862.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff993c1c75
Build id: 3.10.0-862.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffff993c4a02
Build id: 3.10.0-862.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffff993cdf0c
Build id: 3.10.0-862.el7.x86_64
|
0x3ac
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff993bfafd
Build id: 3.10.0-862.el7.x86_64
|
0x3cd
|
|
9
|
|
vmlinux |
0xffffffff993b35eb
Build id: 3.10.0-862.el7.x86_64
|
0xeb
|
|
10
|
system_call_fastpath
|
vmlinux |
0xffffffff9991f7d5
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [java:44220]
Modules linked in: tcp_lp nls_utf8 isofs fuse veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec aesni_intel lrw ppdev ac97_bus gf128mul glue_helper snd_seq ablk_helper cryptd snd_seq_device vmw_balloon snd_pcm snd_timer snd pcspkr joydev nfit soundcore sg libnvdimm parport_pc vmw_vmci parport i2c_piix4 shpchp ip_tables xfs libcrc32c sr_mod cdrom ata_generic
pata_acpi sd_mod crc_t10dif crct10dif_generic ahci crct10dif_pclmul libahci crct10dif_common ata_piix vmwgfx drm_kms_helper mptspi scsi_transport_spi crc32c_intel syscopyarea mptscsih sysfillrect sysimgblt fb_sys_fops ttm drm e1000 libata serio_raw mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 44220 Comm: java Kdump: loaded Tainted: G L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff8dcdde3b2f70 ti: ffff8dcd4a79c000 task.ti: ffff8dcd4a79c000
RIP: 0010:[<ffffffff99307e3a>] [<ffffffff99307e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8dcd4a79fc78 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8dcdf361cd90
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8dcd4a79fcb0 R08: ffff8dccbb4ab400 R09: ffffffff9954a3d9
R10: ffff8dcdf365bb00 R11: ffffc2d6c927d600 R12: 0000000000000296
R13: ffff8dcd4a79fc28 R14: 0000000000000001 R15: ffffc2d6c41321c0
FS: 00007fdd3a194700(0000) GS:ffff8dcdf3640000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fec8df8e000 CR3: 000000027f2d8000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff99277018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff99277088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff993c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff993c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff993c4a02>] zap_page_range+0x142/0x190
[<ffffffff993cdf0c>] ? do_mmap+0x3ac/0x4d0
[<ffffffff993bfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff993b35eb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff9991f7d5>] system_call_fastpath+0x1c/0x21
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 f3 99 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89
Complete report #224151
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa4b12b6a
Build id: 3.10.0-957.27.2.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa4a7b788
Build id: 3.10.0-957.27.2.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa4a7b7f8
Build id: 3.10.0-957.27.2.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffffa4be57d7
Build id: 3.10.0-957.27.2.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa4be70d5
Build id: 3.10.0-957.27.2.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffffa4be9f02
Build id: 3.10.0-957.27.2.el7.x86_64
|
0x142
|
|
7
|
SyS_madvise
|
vmlinux |
0xffffffffa4be4ded
Build id: 3.10.0-957.27.2.el7.x86_64
|
0x49d
|
|
8
|
|
vmlinux |
0xffffffffa4bd784b
Build id: 3.10.0-957.27.2.el7.x86_64
|
0xeb
|
|
9
|
system_call_fastpath
|
vmlinux |
0xffffffffa5176ddb
Build id: 3.10.0-957.27.2.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [java:22084]
Modules linked in: nls_utf8 udf crc_itu_t fuse nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi snd_ens1371 kvm_intel kvm snd_rawmidi snd_ac97_codec ac97_bus irqbypass snd_seq crc32_pclmul snd_seq_device ghash_clmulni_intel aesni_intel snd_pcm ppdev snd_timer lrw gf128mul snd vmw_balloon glue_helper ablk_helper soundcore cryptd pcspkr
joydev sg parport_pc parport i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi mptscsih ata_piix crct10dif_pclmul crct10dif_common nfit drm crc32c_intel libata libnvdimm mptbase e1000 serio_raw drm_panel_orientation_quirks
CPU: 3 PID: 22084 Comm: java Kdump: loaded Tainted: G L ------------ 3.10.0-957.27.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff90f6f7b0d140 ti: ffff90f7ebb24000 task.ti: ffff90f7ebb24000
RIP: 0010:[<ffffffffa4b12b6a>] [<ffffffffa4b12b6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff90f7ebb27c78 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00000000000000fc RCX: ffff90f8ab660600
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff90f7ebb27cb0 R08: ffff90f577d3d400 R09: ffffffffa4d774a9
R10: ffff90f8ab6df160 R11: fffffb524fb2c600 R12: 0000000000000296
R13: ffff90f7ebb27c28 R14: 0000000000000001 R15: fffffb524b9c9c80
FS: 00007fdd9fa15700(0000) GS:ffff90f8ab6c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000771586120 CR3: 000000047d674000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa4a7b788>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa4a7b7f8>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa4be57d7>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffa4be70d5>] tlb_finish_mmu+0x55/0x60
[<ffffffffa4be9f02>] zap_page_range+0x142/0x190
[<ffffffffa4be4ded>] SyS_madvise+0x49d/0xac0
[<ffffffffa4bd784b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffa5176ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 5e 75 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 bf 74 a5 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 2d 75 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #65556
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f99b2
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x202
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8106e668
Build id: 3.10.0-693.11.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8106e738
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.63
|
vmlinux |
0xffffffff811acfc3
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x33
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff811ae2e5
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffff811b086a
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x13a
|
|
7
|
SyS_madvise
|
vmlinux |
0xffffffff811ac68d
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x3cd
|
|
8
|
|
vmlinux |
0xffffffff811a1232
Build id: 3.10.0-693.11.1.el7.x86_64
|
0xe2
|
|
9
|
system_call_fastpath
|
vmlinux |
0xffffffff816b5089
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [java:24092]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon nfit pcspkr joydev sg libnvdimm parport_pc vmw_vmci shpchp parport i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx ahci drm_kms_helper libahci syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common ata_piix crc32c_intel
drm serio_raw libata vmxnet3 vmw_pvscsi i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 24092 Comm: java Tainted: G L ------------ 3.10.0-693.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff88070530dee0 ti: ffff880703b6c000 task.ti: ffff880703b6c000
RIP: 0010:[<ffffffff810f99b2>] [<ffffffff810f99b2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff880703b6fcb0 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 000800008106e4a0 RCX: ffff880a2fc9a0e0
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff880703b6fce8 R08: ffff88017fc3a400 R09: ffffffff81322729
R10: ffff880a2fc19c80 R11: ffffea001c30df00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff880703b6fc60 R15: 0000000000000001
FS: 00007fda78f68700(0000) GS:ffff880a2fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007effea3a3000 CR3: 00000009ffa84000 CR4: 00000000003407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000178f62fff ffff8809d464a288 ffff8809d4649f40 0000000000000000
ffffffffffffffff ffff8809d464a288 ffff8809d4649f40 ffff880703b6fd38
ffffffff8106e668 ffff8809d4649f40 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811acfc3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae2e5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b086a>] zap_page_range+0x13a/0x180
[<ffffffff811ac68d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff811a1232>] ? vm_mmap_pgoff+0xe2/0x110
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 16 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 e5 44 a2 00 eb b7 0f b6 4d cc 4c 89
Complete report #18223
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8d70b0ce
Build id: 3.10.0-862.14.4.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8d679e58
Build id: 3.10.0-862.14.4.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8d679ec8
Build id: 3.10.0-862.14.4.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.75
|
vmlinux |
0xffffffff8d7c3487
Build id: 3.10.0-862.14.4.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff8d7c4d85
Build id: 3.10.0-862.14.4.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffff8d7c7bd2
Build id: 3.10.0-862.14.4.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffff8d7d145c
Build id: 3.10.0-862.14.4.el7.x86_64
|
0x3ac
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff8d7c2add
Build id: 3.10.0-862.14.4.el7.x86_64
|
0x3cd
|
|
9
|
|
vmlinux |
0xffffffff8d7b63ab
Build id: 3.10.0-862.14.4.el7.x86_64
|
0xeb
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [java:2440]
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [java:3416]
Modules linked in: binfmt_misc tcp_diag inet_diag vmw_vsock_vmci_transport vsock sunrpc ppdev sg nfit pcspkr joydev libnvdimm vmw_balloon parport_pc parport shpchp i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm crct10dif_common ahci ata_piix libahci libata serio_raw vmxnet3 vmw_pvscsi i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 3416 Comm: java Kdump: loaded Tainted: G L ------------ 3.10.0-862.14.4.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8c98aeef0000 ti: ffff8c9532ea8000 task.ti: ffff8c9532ea8000
RIP: 0010:[<ffffffff8d70b0ce>] [<ffffffff8d70b0ce>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8c9532eabc78 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000196c0 RCX: ffff8c98bfc1ce48
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8c9532eabcb0 R08: ffff8c95fa34fc00 R09: ffffffff8d94ef19
R10: ffff8c98bfd5bb40 R11: ffffd99a107c9e00 R12: ffffffff8d94ee6f
R13: ffff8c95ffc03500 R14: 0000000000000001 R15: ffffffff8d7fbb7a
FS: 00007f848bb9b700(0000) GS:ffff8c98bfd40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd16b5dbdb8 CR3: 0000000034ac0000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8d679e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8d679ec8>] flush_tlb_mm_range+0x68/0x140
[<ffffffff8d7c3487>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff8d7c4d85>] tlb_finish_mmu+0x55/0x60
[<ffffffff8d7c7bd2>] zap_page_range+0x142/0x190
[<ffffffff8d7d145c>] ? do_mmap+0x3ac/0x4d0
[<ffffffff8d7c2add>] SyS_madvise+0x3cd/0x9c0
[<ffffffff8d7b63ab>] ? vm_mmap_pgoff+0xeb/0x120
Complete report #13136
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff90f07e3a
Build id: 3.10.0-862.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff90e77018
Build id: 3.10.0-862.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff90e77088
Build id: 3.10.0-862.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.75
|
vmlinux |
0xffffffff90fc0497
Build id: 3.10.0-862.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff90fc1c75
Build id: 3.10.0-862.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffff90fc4a02
Build id: 3.10.0-862.el7.x86_64
|
0x142
|
|
7
|
SyS_madvise
|
vmlinux |
0xffffffff90fbfafd
Build id: 3.10.0-862.el7.x86_64
|
0x3cd
|
|
8
|
|
vmlinux |
0xffffffff90fb35eb
Build id: 3.10.0-862.el7.x86_64
|
0xeb
|
|
9
|
system_call_fastpath
|
vmlinux |
0xffffffff9151f7d5
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
NMI watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [java:97752]
Modules linked in: tcp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink binfmt_misc sunrpc ppdev sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel sg aesni_intel parport_pc lrw joydev virtio_balloon gf128mul parport glue_helper ablk_helper cryptd pcspkr i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom virtio_blk virtio_console virtio_scsi virtio_net ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul libata crct10dif_common crc32c_intel virtio_pci virtio_ring i2c_core floppy serio_raw
virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 29 PID: 97752 Comm: java Kdump: loaded Not tainted 3.10.0-862.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
task: ffff92a166728fd0 ti: ffff92a143f28000 task.ti: ffff92a143f28000
RIP: 0010:[<ffffffff90f07e3a>] [<ffffffff90f07e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff92a143f2bc78 EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00000000000000fc RCX: ffff92a95cfdd310
RDX: 0000000000000007 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff92a143f2bcb0 R08: ffff92a2b3d5a000 R09: ffffffff9114a3d9
R10: ffff92a95d55bb00 R11: ffffe80c801e6c00 R12: 0000000000000296
R13: 0000000000000018 R14: ffff92a143f2bbe8 R15: 0000000000000296
FS: 00007fb34048e700(0000) GS:ffff92a95d540000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb46d814738 CR3: 000000044df4c000 CR4: 00000000001607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff90e77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff90e77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff90fc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff90fc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff90fc4a02>] zap_page_range+0x142/0x190
[<ffffffff90fbfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffff90fb35eb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff9151f7d5>] system_call_fastpath+0x1c/0x21
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 b3 91 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89
Complete report #9159
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffab71173e
Build id: 3.10.0-957.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffab67a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffab67a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffffab7e4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffab7e5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffffab7e87c2
Build id: 3.10.0-957.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffffab7f28f8
Build id: 3.10.0-957.el7.x86_64
|
0x378
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffab7e368d
Build id: 3.10.0-957.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffffab7d612b
Build id: 3.10.0-957.el7.x86_64
|
0xeb
|
|
10
|
system_call_fastpath
|
vmlinux |
0xffffffffabd74ddb
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#19 stuck for 23s! [java:21128]
Modules linked in: binfmt_misc vmw_vsock_vmci_transport vsock sunrpc ppdev sb_edac coretemp iosf_mbi crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm libahci crct10dif_pclmul crct10dif_common drm ata_piix crc32c_intel serio_raw libata vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 19 PID: 21128 Comm: java Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9e96209c6180 ti: ffff9e96045f0000 task.ti: ffff9e96045f0000
RIP: 0010:[<ffffffffab71173e>] [<ffffffffab71173e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff9e96045f3c78 EFLAGS: 00000202
RAX: 000000000000000a RBX: 00000000000000fc RCX: ffff9e9fab0208e0
RDX: 000000000000000a RSI: 0000000000000014 RDI: 0000000000000000
RBP: ffff9e96045f3cb0 R08: ffff9e9b37e2d000 R09: ffffffffab9755c9
R10: ffff9e9fab49f160 R11: ffffeef8d384f600 R12: 0000000000000296
R13: ffff9e96045f3c28 R14: 0000000000000001 R15: ffffeef8d74ba7c0
FS: 00007f280a72f700(0000) GS:ffff9e9fab480000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f280a71a708 CR3: 0000000a04f3a000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffab67a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffab67a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffab7e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffab7e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffab7e87c2>] zap_page_range+0x142/0x190
[<ffffffffab7f28f8>] ? do_mmap+0x378/0x530
[<ffffffffab7e368d>] SyS_madvise+0x49d/0xac0
[<ffffffffab7d612b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffabd74ddb>] system_call_fastpath+0x22/0x27
Code: 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 34 ac f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #1253945
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9ed1173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9ec7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9ec7a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffff9ede4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff9ede5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffff9ede87c2
Build id: 3.10.0-957.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffff9ecd6768
Build id: 3.10.0-957.el7.x86_64
|
0x58
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff9ede368d
Build id: 3.10.0-957.el7.x86_64
|
0x49d
|
|
9
|
tracesys
|
vmlinux |
0xffffffff9f37505b
Build id: 3.10.0-957.el7.x86_64
|
0xa3
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [agent:8364]
Modules linked in: xt_REDIRECT nf_nat_redirect ip_vs_rr xt_ipvs ip_vs xt_nat veth vxlan ip6_udp_tunnel udp_tunnel xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter binfmt_misc overlay(T) vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi
crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmwgfx e1000 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm floppy ata_piix mptspi libata scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 8364 Comm: agent Kdump: loaded Tainted: G W L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8b2736c1b0c0 ti: ffff8b2e29b8c000 task.ti: ffff8b2e29b8c000
RIP: 0010:[<ffffffff9ed1173a>] [<ffffffff9ed1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b2e29b8fc78 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8b2a2fc1fdd8
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff8b2e29b8fcb0 R08: ffff8b277a2e3800 R09: ffffffff9ef755c9
R10: ffff8b2a2fc5f160 R11: ffffce848e60f400 R12: 0000000000000296
R13: ffff8b2e29b8fc28 R14: 0000000000000001 R15: ffffce848e454bc0
FS: 000000c000500090(0000) GS:ffff8b2a2fc40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0007ff000 CR3: 000000042cb52000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9ec7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9ec7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff9ede4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff9ede5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff9ede87c2>] zap_page_range+0x142/0x190
[<ffffffff9ecd6768>] ? wake_up_q+0x58/0x80
[<ffffffff9ede368d>] SyS_madvise+0x49d/0xac0
[<ffffffff9f37505b>] tracesys+0xa3/0xc9
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 94 9f f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1197549
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
|
0x202
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.63
|
vmlinux |
0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
|
0x33
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffff811b08fa
Build id: 3.10.0-693.el7.x86_64
|
0x13a
|
|
7
|
|
vmlinux |
0xffffffff810c47c8
Build id: 3.10.0-693.el7.x86_64
|
0x58
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff811ac71d
Build id: 3.10.0-693.el7.x86_64
|
0x3cd
|
|
9
|
|
vmlinux |
0xffffffff810cb62c
Build id: 3.10.0-693.el7.x86_64
|
0x3c
|
|
10
|
|
vmlinux |
0xffffffff810f8040
Build id: 3.10.0-693.el7.x86_64
|
0x80
|
|
11
|
system_call_fastpath
|
vmlinux |
0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [trantorAgent:17521]
Modules linked in: qax_tq_base(OE) uinput veth xt_nat xt_CHECKSUM iptable_mangle nf_conntrack_netlink nfnetlink ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat xt_addrtype nf_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack br_netfilter ipt_REJECT nf_reject_ipv4 overlay(T) tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter sunrpc cirrus ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm i2c_piix4 virtio_net virtio_console i2c_core sg virtio_scsi virtio_balloon snd_hda_codec_generic snd_hda_intel joydev snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer virtio_pci snd virtio_ring shpchp virtio soundcore ip_vs_ftp nf_nat ip_vs_sed ip_vs_nq ip_vs_sh ip_vs_dh ip_vs_lblcr ip_vs_lblc pcspkr ip_vs_wrr
ip_vs_rr ip_vs_wlc ip_vs_lc ip_vs nf_conntrack ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi ata_piix libata serio_raw floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: qax_tq_base]
CPU: 6 PID: 17521 Comm: trantorAgent Tainted: G OEL ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: inspur Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-2.module_kos5+70+5e037788 04/01/2014
task: ffff8803da2b2f70 ti: ffff880370ba4000 task.ti: ffff880370ba4000
RIP: 0010:[<ffffffff810f99a2>] [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff880370ba7cb0 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008106e4a0 RCX: ffff88043fc1ae30
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff880370ba7ce8 R08: ffff88017a360400 R09: ffffffff813227d9
R10: ffff88043fd99c80 R11: ffffea0002d92400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff880370ba7c60 R15: ffffffff81f579c0
FS: 00007f92bd7fa700(0000) GS:ffff88043fd80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c001e25010 CR3: 0000000356a9d000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000101e25fff ffff880356abed88 ffff880356abea40 0000000000000000
ffffffffffffffff ffff880356abed88 ffff880356abea40 ffff880370ba7d38
ffffffff8106e668 ffff880356abea40 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b08fa>] zap_page_range+0x13a/0x180
[<ffffffff810c47c8>] ? wake_up_q+0x58/0x80
[<ffffffff811ac71d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff810cb62c>] ? set_next_entity+0x3c/0xe0
[<ffffffff810f8040>] ? SyS_futex+0x80/0x180
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89
Complete report #1196963
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f99a6
Build id: 3.10.0-693.el7.x86_64
|
0x206
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.63
|
vmlinux |
0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
|
0x33
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffff811b08fa
Build id: 3.10.0-693.el7.x86_64
|
0x13a
|
|
7
|
|
vmlinux |
0xffffffff810c47c8
Build id: 3.10.0-693.el7.x86_64
|
0x58
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff811ac71d
Build id: 3.10.0-693.el7.x86_64
|
0x3cd
|
|
9
|
|
vmlinux |
0xffffffff810cb62c
Build id: 3.10.0-693.el7.x86_64
|
0x3c
|
|
10
|
|
vmlinux |
0xffffffff810f8040
Build id: 3.10.0-693.el7.x86_64
|
0x80
|
|
11
|
system_call_fastpath
|
vmlinux |
0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [trantorAgent:25249]
Modules linked in: qax_tq_base(OE) binfmt_misc xt_set ip_set_hash_ip cfg80211 rfkill uinput gsch(OE) redirfs(OE) nf_conntrack_netlink xt_addrtype br_netfilter dsa_filter(POE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ilocker(OE) overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter cirrus ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops snd_hda_codec_generic snd_hda_intel snd_hda_codec
drm virtio_net snd_hda_core snd_hwdep snd_seq snd_seq_device virtio_console pcspkr i2c_piix4 snd_pcm joydev snd_timer virtio_balloon virtio_scsi sg shpchp i2c_core snd soundcore virtio_pci virtio_ring virtio nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi ata_piix libata serio_raw floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: qax_tq_base]
CPU: 1 PID: 25249 Comm: trantorAgent Tainted: P OE ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: inspur Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-2.module_kos5+70+5e037788 04/01/2014
task: ffff8800b71d9fa0 ti: ffff8800877d0000 task.ti: ffff8800877d0000
RIP: 0010:[<ffffffff810f99a6>] [<ffffffff810f99a6>] smp_call_function_many+0x206/0x260
RSP: 0018:ffff8800877d3cb0 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008106e4a0 RCX: ffff88043fc1a6c8
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8800877d3ce8 R08: ffff88017a207400 R09: ffffffff813227d9
R10: ffff88043fc59c80 R11: ffffea000df4c000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8800877d3c60 R15: 0000000000000001
FS: 00007f40867fc700(0000) GS:ffff88043fc40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00167b010 CR3: 00000001e2755000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
000000010169dfff ffff8803e96d6108 ffff8803e96d5dc0 0000000000000000
ffffffffffffffff ffff8803e96d6108 ffff8803e96d5dc0 ffff8800877d3d38
ffffffff8106e668 ffff8803e96d5dc0 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b08fa>] zap_page_range+0x13a/0x180
[<ffffffff810c47c8>] ? wake_up_q+0x58/0x80
[<ffffffff811ac71d>] SyS_madvise+0x3cd/0x9c0
[<ffffffff810cb62c>] ? set_next_entity+0x3c/0xe0
[<ffffffff810f8040>] ? SyS_futex+0x80/0x180
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #1016176
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffaf11173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffaf07a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffaf07a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffffaf1e4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffaf1e5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffffaf1e87c2
Build id: 3.10.0-957.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffffaf0d6768
Build id: 3.10.0-957.el7.x86_64
|
0x58
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffaf1e368d
Build id: 3.10.0-957.el7.x86_64
|
0x49d
|
|
9
|
system_call_fastpath
|
vmlinux |
0xffffffffaf774ddb
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [serverLibDown:25860]
Modules linked in: tcp_lp nls_utf8 isofs fuse binfmt_misc tcp_diag inet_diag xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw
e1000 nfit libnvdimm ata_generic vmwgfx pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi ata_piix mptscsih drm libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 25860 Comm: serverLibDown Kdump: loaded Tainted: G W L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8f5455f68000 ti: ffff8f564cee4000 task.ti: ffff8f564cee4000
RIP: 0010:[<ffffffffaf11173a>] [<ffffffffaf11173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8f564cee7c78 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8f566d6204b8
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8f564cee7cb0 R08: ffff8f53b9cb3800 R09: ffffffffaf3755c9
R10: ffff8f566d69f160 R11: fffff8818f59a400 R12: 0000000000000296
R13: 0000000000000018 R14: ffff8f564cee7be8 R15: 0000000000000296
FS: 000000c000400090(0000) GS:ffff8f566d680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000b3e6f8 CR3: 00000003dabf0000 CR4: 00000000003607e0
Call Trace:
[<ffffffffaf07a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaf07a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffaf1e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffaf1e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffaf1e87c2>] zap_page_range+0x142/0x190
[<ffffffffaf0d6768>] ? wake_up_q+0x58/0x80
[<ffffffffaf1e368d>] SyS_madvise+0x49d/0xac0
[<ffffffffaf774ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 d4 af f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #820424
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb991173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb987a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffb987a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffffb99e4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffb99e5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffffb99e87c2
Build id: 3.10.0-957.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffffb98d6768
Build id: 3.10.0-957.el7.x86_64
|
0x58
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffb99e368d
Build id: 3.10.0-957.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffffb98d67b0
Build id: 3.10.0-957.el7.x86_64
|
0x20
|
|
10
|
system_call_fastpath
|
vmlinux |
0xffffffffb9f74ddb
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [process-exporte:9936]
Modules linked in: nfnetlink_queue nfnetlink_log bluetooth rfkill xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq joydev snd_seq_device snd_pcm pcspkr snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic
pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm e1000 ata_piix mptspi scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 9936 Comm: process-exporte Kdump: loaded Tainted: G L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff8a6dc1049040 ti: ffff8a6db0b3c000 task.ti: ffff8a6db0b3c000
RIP: 0010:[<ffffffffb991173a>] [<ffffffffb991173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8a6db0b3fc78 EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00000000000000fc RCX: ffff8a6dc51e0650
RDX: 0000000000000007 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8a6db0b3fcb0 R08: ffff8a6af86ccc00 R09: ffffffffb9b755c9
R10: ffff8a6dc515f160 R11: ffffe859006ab200 R12: 0000000000000296
R13: ffff8a6db0b3fc28 R14: 0000000000000001 R15: ffffe8590dff43c0
FS: 00007faf3166d700(0000) GS:ffff8a6dc5140000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6dac3d91bc CR3: 0000000430e40000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb987a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb987a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb99e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffb99e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffb99e87c2>] zap_page_range+0x142/0x190
[<ffffffffb98d6768>] ? wake_up_q+0x58/0x80
[<ffffffffb99e368d>] SyS_madvise+0x49d/0xac0
[<ffffffffb98d67b0>] ? wake_up_state+0x20/0x20
[<ffffffffb9f74ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 54 ba f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #705142
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8d71173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8d67a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8d67a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffff8d7e4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff8d7e5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffff8d7e87c2
Build id: 3.10.0-957.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffff8d6d6768
Build id: 3.10.0-957.el7.x86_64
|
0x58
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff8d7e368d
Build id: 3.10.0-957.el7.x86_64
|
0x49d
|
|
9
|
tracesys
|
vmlinux |
0xffffffff8dd7505b
Build id: 3.10.0-957.el7.x86_64
|
0xa3
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [minio:34157]
Modules linked in: binfmt_misc veth xt_nat xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) overlay(T) sunrpc snd_seq_midi snd_seq_midi_event sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_ens1371 lrw snd_rawmidi gf128mul ppdev glue_helper snd_ac97_codec ablk_helper cryptd ac97_bus vmw_balloon snd_seq snd_seq_device snd_pcm snd_timer snd joydev soundcore pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi scsi_transport_spi mptscsih e1000 mptbase nfit libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 34157 Comm: minio Kdump: loaded Tainted: G W OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8ce63331b0c0 ti: ffff8ce810b0c000 task.ti: ffff8ce810b0c000
RIP: 0010:[<ffffffff8d71173a>] [<ffffffff8d71173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8ce810b0fc78 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8ce9ad6204b8
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8ce810b0fcb0 R08: ffff8ce6f9cb3800 R09: ffffffff8d9755c9
R10: ffff8ce9ad69f160 R11: fffff56809f1c800 R12: 0000000000000296
R13: ffff8ce810b0fc28 R14: 0000000000000001 R15: fffff5680101bec0
FS: 000000c00008f890(0000) GS:ffff8ce9ad680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa75391f700 CR3: 00000000a95ca000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8d67a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8d67a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff8d7e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff8d7e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff8d7e87c2>] zap_page_range+0x142/0x190
[<ffffffff8d6d6768>] ? wake_up_q+0x58/0x80
[<ffffffff8d7e368d>] SyS_madvise+0x49d/0xac0
[<ffffffff8dd7505b>] tracesys+0xa3/0xc9
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 34 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #193502
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa5f07e3e
Build id: 3.10.0-862.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa5e77018
Build id: 3.10.0-862.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa5e77088
Build id: 3.10.0-862.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.75
|
vmlinux |
0xffffffffa5fc0497
Build id: 3.10.0-862.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa5fc1c75
Build id: 3.10.0-862.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffffa5fc4a02
Build id: 3.10.0-862.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffffa5ecee38
Build id: 3.10.0-862.el7.x86_64
|
0x58
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffa5fbfafd
Build id: 3.10.0-862.el7.x86_64
|
0x3cd
|
|
9
|
|
vmlinux |
0xffffffffa5ecee80
Build id: 3.10.0-862.el7.x86_64
|
0x20
|
|
10
|
system_call_fastpath
|
vmlinux |
0xffffffffa651f7d5
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 32s! [node_exporter:1561]
Modules linked in: iptable_filter ip_tables binfmt_misc ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg pcspkr joydev vmw_balloon parport_pc parport shpchp vmw_vmci i2c_piix4 xfs libcrc32c sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper sr_mod cdrom ata_generic syscopyarea pata_acpi sysfillrect sysimgblt fb_sys_fops ttm mptsas drm ata_piix scsi_transport_sas crct10dif_pclmul crct10dif_common libata mptscsih serio_raw crc32c_intel mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 3 PID: 1561 Comm: node_exporter Kdump: loaded Tainted: G L ------------ 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8a9a35e99fa0 ti: ffff8a9a0b590000 task.ti: ffff8a9a0b590000
RIP: 0010:[<ffffffffa5f07e3e>] [<ffffffffa5f07e3e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8a9a0b593c78 EFLAGS: 00000202
RAX: 0000000000000005 RBX: 00000000000000fc RCX: ffff8a9e3fd5cf00
RDX: 0000000000000005 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8a9a0b593cb0 R08: ffff8a9b7a357400 R09: ffffffffa614a3d9
R10: ffff8a9e3fcdbb00 R11: ffffda679089f400 R12: 0000000000000296
R13: ffff8a9a0b593c28 R14: 0000000000000001 R15: ffffda6781628700
FS: 000000c000054e90(0000) GS:ffff8a9e3fcc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00076f000 CR3: 00000003a49d6000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa5e77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa5e77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa5fc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffa5fc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffa5fc4a02>] zap_page_range+0x142/0x190
[<ffffffffa5ecee38>] ? wake_up_q+0x58/0x80
[<ffffffffa5fbfafd>] SyS_madvise+0x3cd/0x9c0
[<ffffffffa5ecee80>] ? wake_up_state+0x20/0x20
[<ffffffffa651f7d5>] system_call_fastpath+0x1c/0x21
Code: b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 b3 a6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #23323
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffaa911a0a
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffaa87a628
Build id: 3.10.0-957.12.2.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffaa87a698
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffffaa9e4607
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffaa9e5f05
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x55
|
|
6
|
zap_page_range
|
vmlinux |
0xffffffffaa9e8d52
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x142
|
|
7
|
|
vmlinux |
0xffffffffaa8d6a98
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x58
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffaa9e3c1d
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x49d
|
|
9
|
system_call_fastpath
|
vmlinux |
0xffffffffaaf75ddb
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [mysqld:73675]
Modules linked in: nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter binfmt_misc overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event coretemp iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 ppdev snd_rawmidi snd_ac97_codec ac97_bus snd_seq pcspkr snd_seq_device snd_pcm snd_timer snd soundcore vmw_balloon joydev sg parport_pc parport vmw_vmci i2c_piix4 ip_tables
xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw ttm drm mptspi scsi_transport_spi mptscsih mptbase e1000 ata_piix libata drm_panel_orientation_quirks nfit libnvdimm
CPU: 6 PID: 73675 Comm: mysqld Kdump: loaded Tainted: G L ------------ T 3.10.0-957.12.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9158d9caa080 ti: ffff9159f668c000 task.ti: ffff9159f668c000
RIP: 0010:[<ffffffffaa911a0a>] [<ffffffffaa911a0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9159f668fc78 EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00000000000000fc RCX: ffff915a2d720678
RDX: 0000000000000004 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9159f668fcb0 R08: ffff915779cb4800 R09: ffffffffaab75d69
R10: ffff915a2d79f160 R11: fffffd339076ec00 R12: 0000000000000296
R13: ffff9159f668fc28 R14: 0000000000000001 R15: fffffd3382ba1d80
FS: 00007f3bf6805700(0000) GS:ffff915a2d780000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3bf766df90 CR3: 00000003793d4000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaa87a628>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaa87a698>] flush_tlb_mm_range+0x68/0x140
[<ffffffffaa9e4607>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffaa9e5f05>] tlb_finish_mmu+0x55/0x60
[<ffffffffaa9e8d52>] zap_page_range+0x142/0x190
[<ffffffffaa8d6a98>] ? wake_up_q+0x58/0x80
[<ffffffffaa9e3c1d>] SyS_madvise+0x49d/0xac0
[<ffffffffaaf75ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 7e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 bc 54 ab f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 83 c4 00 eb b7 0f b6 4d cc 4c 89