Complete report #1375738
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa651e2b8
Build id: 3.10.0-1160.119.1.el7.tuxcare.els19.x86_64
|
0x208
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa6481fa8
Build id: 3.10.0-1160.119.1.el7.tuxcare.els19.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa648201b
Build id: 3.10.0-1160.119.1.el7.tuxcare.els19.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffa65fa12e
Build id: 3.10.0-1160.119.1.el7.tuxcare.els19.x86_64
|
0xae
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffa65fbcaf
Build id: 3.10.0-1160.119.1.el7.tuxcare.els19.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa65fbde3
Build id: 3.10.0-1160.119.1.el7.tuxcare.els19.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffa65ff32a
Build id: 3.10.0-1160.119.1.el7.tuxcare.els19.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffffa660931c
Build id: 3.10.0-1160.119.1.el7.tuxcare.els19.x86_64
|
0x39c
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffffa65f971d
Build id: 3.10.0-1160.119.1.el7.tuxcare.els19.x86_64
|
0x49d
|
|
10
|
system_call_fastpath
|
vmlinux |
0xffffffffa6bc539a
Build id: 3.10.0-1160.119.1.el7.tuxcare.els19.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [process reaper:46982]
Modules linked in: sunrpc vfat fat ppdev nfit libnvdimm iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev sg pcspkr parport_pc parport i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix xen_netfront xen_blkfront libata crct10dif_pclmul crct10dif_common crc32c_intel floppy serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod tun fuse
CPU: 7 PID: 46982 Comm: process reaper Kdump: loaded Tainted: G L ------------ 3.10.0-1160.119.1.el7.tuxcare.els19.x86_64 #1
Hardware name: Xen HVM domU, BIOS 4.13 09/09/2025
task: ffff9d22045a1080 ti: ffff9d2323914000 task.ti: ffff9d2323914000
RIP: 0010:[<ffffffffa651e2b8>] [<ffffffffa651e2b8>] smp_call_function_many+0x208/0x270
RSP: 0018:ffff9d2323917c70 EFLAGS: 00000202
RAX: 000000000000000d RBX: ffff9d2323917c10 RCX: ffff9d29ee5605b8
RDX: 000000000000000d RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff9d2323917ca8 R08: ffff9d236c79a800 R09: 0000000000000001
R10: ffff9d236c79a800 R11: 00007f07dad19000 R12: ffff9d2323917c00
R13: ffffffffa6877ff6 R14: ffff9d2323917be0 R15: ffffffffa655b78e
FS: 00007f07dad4e700(0000) GS:ffff9d29ee3c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc0b5c204b1 CR3: 000000018c920000 CR4: 00000000007606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffa6481fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa648201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa65fa12e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa65fbcaf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa65fbde3>] tlb_finish_mmu+0x23/0x40
[<ffffffffa65ff32a>] zap_page_range+0xea/0x140
[<ffffffffa660931c>] ? do_mmap+0x39c/0x590
[<ffffffffa65f971d>] SyS_madvise+0x49d/0xac0
[<ffffffffa6bc539a>] system_call_fastpath+0x25/0x2a
Code: 29 00 48 63 35 0e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 80 19 15 a7 f6 41 20 01 74 cd 0f 1f 44 00 00 <f3> 90 f6 41 20 01 75 f8 48 63 35 dd 17 c4 00 eb b7 0f b6 4d cc
Complete report #1363664
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb311e07a
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb3081f48
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffb3081fbb
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffb31f9f0e
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xae
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffb31fba4f
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffb31fbb83
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffb31ff0ca
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffffb31194d0
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x90
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffffb31f949d
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffffb31ea203
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xf3
|
|
11
|
system_call_fastpath
|
vmlinux |
0xffffffffb37c539a
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [java:55853]
Modules linked in: nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq sb_edac snd_seq_device snd_pcm iosf_mbi snd_timer crc32_pclmul ghash_clmulni_intel aesni_intel lrw snd gf128mul glue_helper ppdev ablk_helper cryptd soundcore vmw_balloon sg pcspkr joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common ata_piix nfit drm libnvdimm libata crc32c_intel e1000 mptspi serio_raw scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 55853 Comm: java Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff90c691722100 ti: ffff90c8e7028000 task.ti: ffff90c8e7028000
RIP: 0010:[<ffffffffb311e07a>] [<ffffffffb311e07a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff90c8e702bc70 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000b3081ab0 RCX: ffff90c8e9660650
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff90c8e702bca8 R08: ffff90c53255c000 R09: ffffffffb339c139
R10: ffff90c8e96df160 R11: fffffa350b455400 R12: 0000000000000282
R13: 0000000000000018 R14: ffff90c8e702bbe0 R15: 0000000000000282
FS: 00007f146bd17700(0000) GS:ffff90c8e96c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6b77484c38 CR3: 00000004f55de000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb3081f48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb3081fbb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb31f9f0e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffb31fba4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb31fbb83>] tlb_finish_mmu+0x23/0x40
[<ffffffffb31ff0ca>] zap_page_range+0xea/0x140
[<ffffffffb31194d0>] ? futex_wake+0x90/0x190
[<ffffffffb31f949d>] SyS_madvise+0x49d/0xac0
[<ffffffffb31ea203>] ? vm_mmap_pgoff+0xf3/0x120
[<ffffffffb37c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 d5 b3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 19 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1302727
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa5516c0a
Build id: 3.10.0-1160.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa547e868
Build id: 3.10.0-1160.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa547e8db
Build id: 3.10.0-1160.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffa55edd06
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffa55ef7cf
Build id: 3.10.0-1160.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa55ef8f3
Build id: 3.10.0-1160.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffa55f2dba
Build id: 3.10.0-1160.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffffa54daaa8
Build id: 3.10.0-1160.el7.x86_64
|
0x58
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffffa55ed2ad
Build id: 3.10.0-1160.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffffa55de1bb
Build id: 3.10.0-1160.el7.x86_64
|
0xeb
|
|
11
|
system_call_fastpath
|
vmlinux |
0xffffffffa5b93f92
Build id: 3.10.0-1160.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [Metrics-HealthC:23553]
Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) nfit libnvdimm snd_hda_codec_generic snd_hda_intel snd_hda_codec iosf_mbi crc32_pclmul ghash_clmulni_intel snd_hda_core snd_hwdep aesni_intel snd_seq lrw gf128mul snd_seq_device snd_pcm ppdev glue_helper ablk_helper cryptd snd_timer i2c_piix4 snd soundcore joydev parport_pc virtio_balloon parport pcspkr ip_tables xfs libcrc32c qxl drm_kms_helper ata_generic pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm virtio_console virtio_blk drm ata_piix 8139too libata crct10dif_pclmul crct10dif_common floppy crc32c_intel virtio_pci
serio_raw virtio_ring 8139cp virtio drm_panel_orientation_quirks mii dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 23553 Comm: Metrics-HealthC Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff8dd8f6cdd280 ti: ffff8dd661b74000 task.ti: ffff8dd661b74000
RIP: 0010:[<ffffffffa5516c0a>] [<ffffffffa5516c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8dd661b77c70 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000a547e6a0 RCX: ffff8dda7fc60398
RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8dd661b77ca8 R08: ffff8dd69cbb9c00 R09: ffffffffa5785f99
R10: ffff8dda7fd1f160 R11: fffff87190a1d600 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8dd661b77c20 R15: 0000000000000001
FS: 00007f8cfbdfd640(0000) GS:ffff8dda7fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055dd0e4e0330 CR3: 0000000449bc2000 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa547e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa547e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa55edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa55ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa55ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffffa55f2dba>] zap_page_range+0xea/0x130
[<ffffffffa54daaa8>] ? wake_up_q+0x58/0x80
[<ffffffffa55ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffffa55de1bb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffa5b93f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 15 a6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #913573
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff90516c0a
Build id: 3.10.0-1160.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9047e868
Build id: 3.10.0-1160.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9047e8db
Build id: 3.10.0-1160.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff905edd06
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff905ef7cf
Build id: 3.10.0-1160.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff905ef8f3
Build id: 3.10.0-1160.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff905f2dba
Build id: 3.10.0-1160.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff905ed2ad
Build id: 3.10.0-1160.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffff905de1bb
Build id: 3.10.0-1160.el7.x86_64
|
0xeb
|
|
10
|
system_call_fastpath
|
vmlinux |
0xffffffff90b93f92
Build id: 3.10.0-1160.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [java:39553]
Modules linked in: vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon pcspkr sg joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix mptspi libata scsi_transport_spi mptscsih serio_raw nfit e1000 mptbase libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 39553 Comm: java Kdump: loaded Tainted: G W L ------------ 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff97d75a6eb180 ti: ffff97d42e1c0000 task.ti: ffff97d42e1c0000
RIP: 0010:[<ffffffff90516c0a>] [<ffffffff90516c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff97d42e1c3c70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009047e6a0 RCX: ffff97d8a3620600
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff97d42e1c3ca8 R08: ffff97d37ef5b800 R09: ffffffff90785f99
R10: ffff97d8a365f160 R11: ffffee74157b5600 R12: 0000000000000282
R13: 0000000000000282 R14: ffff97d42e1c3c20 R15: 0000000000000001
FS: 00007f07744d0700(0000) GS:ffff97d8a3640000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000558b231a36f8 CR3: 0000000035f8c000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff9047e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9047e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff905edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff905ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff905ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff905f2dba>] zap_page_range+0xea/0x130
[<ffffffff905ed2ad>] SyS_madvise+0x49d/0xac0
[<ffffffff905de1bb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff90b93f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 15 91 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #807602
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa31170d8
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x208
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa307e258
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa307e2cb
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffa31ee676
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffa31f013f
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa31f0263
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffa31f372a
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffffa31fd36c
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x39c
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffffa31edc1d
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffffa31dedab
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xeb
|
|
11
|
system_call_fastpath
|
vmlinux |
0xffffffffa3795f92
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [ssh-agent: stdo:23537]
Modules linked in: xt_comment xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink tun vmw_vsock_vmci_transport vsock sunrpc vfat fat sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc vmw_vmci i2c_piix4 parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common mptspi libata crc32c_intel scsi_transport_spi mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ipmi_msghandler]
CPU: 4 PID: 23537 Comm: ssh-agent: stdo Tainted: G W L ------------ 3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.0.B64.1704120155 04/12/2017
task: ffff9d141f2fb180 ti: ffff9d0ea5fbc000 task.ti: ffff9d0ea5fbc000
RIP: 0010:[<ffffffffa31170d8>] [<ffffffffa31170d8>] smp_call_function_many+0x208/0x270
RSP: 0018:ffff9d0ea5fbfc70 EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00080000a307ddd0 RCX: ffff9d157fca0678
RDX: 0000000000000002 RSI: 0000000000000007 RDI: 0000000000000000
RBP: ffff9d0ea5fbfca8 R08: ffff9d0ca5624c00 R09: ffffffffa3386bf9
R10: ffff9d157fd1f160 R11: ffffed871bb1a400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9d0ea5fbfc20 R15: 0000000000000001
FS: 00007ff4d81c3700(0000) GS:ffff9d157fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff4d81c0908 CR3: 000000093e804000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa307e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa307e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa31ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa31f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa31f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffa31f372a>] zap_page_range+0xea/0x130
[<ffffffffa31fd36c>] ? do_mmap+0x39c/0x590
[<ffffffffa31edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffa31dedab>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffa3795f92>] system_call_fastpath+0x25/0x2a
Code: 28 00 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 d5 a3 f6 41 20 01 74 cd 0f 1f 44 00 00 <f3> 90 f6 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc
Complete report #589266
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa111718a
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa107e258
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa107e2cb
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffa11ee6a6
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffa11f016f
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa11f0293
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffa11f375a
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffffa11fd34c
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x39c
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffffa11edc4d
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffffa11deddb
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xeb
|
|
11
|
tracesys
|
vmlinux |
0xffffffffa179a226
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [RemoteInvocatio:30691]
Modules linked in: binfmt_misc unix_diag af_packet_diag netlink_diag udp_diag tcp_diag inet_diag uinput nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink cmac arc4 md4 nls_utf8 cifs ccm dns_resolver overlay(T) dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul sg glue_helper joydev pcspkr ablk_helper cryptd i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_net net_failover virtio_console
virtio_scsi failover qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy fuse
CPU: 0 PID: 30691 Comm: RemoteInvocatio Tainted: G L ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
task: ffff978a267f4200 ti: ffff978ad644c000 task.ti: ffff978ad644c000
RIP: 0010:[<ffffffffa111718a>] [<ffffffffa111718a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff978ad644fc70 EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00080000a107ddd0 RCX: ffff978dbfd1fc10
RDX: 0000000000000004 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff978ad644fca8 R08: ffff978affd4f800 R09: ffffffffa13869f9
R10: ffff978dbfc1f160 R11: fffff42684b13500 R12: 0000000000000282
R13: 0000000000000282 R14: ffff978ad644fc20 R15: 0000000000000001
FS: 00007fbf078f8700(0000) GS:ffff978dbfc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbf364a0768 CR3: 000000009fede000 CR4: 0000000000160ff0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa107e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa107e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa11ee6a6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa11f016f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa11f0293>] tlb_finish_mmu+0x23/0x30
[<ffffffffa11f375a>] zap_page_range+0xea/0x130
[<ffffffffa11fd34c>] ? do_mmap+0x39c/0x590
[<ffffffffa11edc4d>] SyS_madvise+0x49d/0xac0
[<ffffffffa11deddb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffa179a226>] tracesys+0xa6/0xcc
Code: 48 63 35 be 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 d5 a1 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #243730
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb4b16a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb4a7e478
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffb4a7e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffb4bec096
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffb4bedb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffb4bedc83
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffb4bf098a
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffffb4bfad78
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x378
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffffb4beb69d
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffffb4bddffb
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xeb
|
|
11
|
system_call_fastpath
|
vmlinux |
0xffffffffb518dede
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [java:5257]
Modules linked in: cts rpcsec_gss_krb5 veth nfsd nfsv3 nfs_acl nfs lockd grace fscache openafs(POE) xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) nls_utf8 isofs vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev pcspkr i2c_piix4 parport_pc vmw_vmci parport binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit crc32c_intel serio_raw libnvdimm vmxnet3 ata_generic drm_panel_orientation_quirks pata_acpi sd_mod
crc_t10dif crct10dif_generic sr_mod crct10dif_pclmul crct10dif_common cdrom mptspi scsi_transport_spi mptscsih mptbase ata_piix libata dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 5257 Comm: java Kdump: loaded Tainted: P OEL ------------ T 3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8c7ec2a2b150 ti: ffff8c76edfc8000 task.ti: ffff8c76edfc8000
RIP: 0010:[<ffffffffb4b16a7a>] [<ffffffffb4b16a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8c76edfcbc70 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000b4a7e2b0 RCX: ffff8c7effc60678
RDX: 0000000000000001 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff8c76edfcbca8 R08: ffff8c742d600800 R09: ffffffffb4d81839
R10: ffff8c7effd1f160 R11: fffffb7a8251ac00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8c76edfcbc20 R15: 0000000000000001
FS: 00007f8f1361f700(0000) GS:ffff8c7effd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8f13609eb8 CR3: 000000003653e000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffb4a7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb4a7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb4bec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb4bedb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb4bedc83>] tlb_finish_mmu+0x23/0x30
[<ffffffffb4bf098a>] zap_page_range+0xea/0x130
[<ffffffffb4bfad78>] ? do_mmap+0x378/0x530
[<ffffffffb4beb69d>] SyS_madvise+0x49d/0xac0
[<ffffffffb4bddffb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb518dede>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 74 b5 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #75681
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8bf16e4a
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8be7e868
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8be7e8db
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff8bfede86
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff8bfef94f
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff8bfefa73
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff8bff2f3a
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff8bfed42d
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffff8bfde32b
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xeb
|
|
10
|
system_call_fastpath
|
vmlinux |
0xffffffff8c594f92
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [Worker-97:27210]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc nfsv3 nfs_acl nfs lockd ebtable_filter grace ebtables fscache ip6table_filter devlink ip6_tables iptable_filter mvfs(OE) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul vmw_balloon glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul drm crct10dif_common crc32c_intel ata_piix
nfit libata serio_raw mptspi libnvdimm scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks floppy fuse
CPU: 4 PID: 27210 Comm: Worker-97 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9c4ccf1fd280 ti: ffff9c4b0c83c000 task.ti: ffff9c4b0c83c000
RIP: 0010:[<ffffffff8bf16e4a>] [<ffffffff8bf16e4a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9c4b0c83fc70 EFLAGS: 00000202
RAX: 0000000000000005 RBX: ffffffff8be7e6a0 RCX: ffff9c4dee760678
RDX: 0000000000000005 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff9c4b0c83fca8 R08: ffff9c4b3a343800 R09: ffffffff8c186479
R10: ffff9c4dee71f160 R11: ffffe2b146631a00 R12: 0000000000000286
R13: ffff9c4b0c83fc20 R14: ffff9c4b38740000 R15: 0000000000000001
FS: 00007fa65fafe700(0000) GS:ffff9c4dee700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000015cdd840e000 CR3: 0000000405532000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8be7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8be7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8bfede86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8bfef94f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8bfefa73>] tlb_finish_mmu+0x23/0x30
[<ffffffff8bff2f3a>] zap_page_range+0xea/0x130
[<ffffffff8bfed42d>] SyS_madvise+0x49d/0xac0
[<ffffffff8bfde32b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff8c594f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 de 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 b5 8c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ad 86 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #73650
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff91516e4a
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9147e868
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9147e8db
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff915ede86
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff915ef94f
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff915efa73
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff915f2f3a
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffff915fcb7c
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x39c
|
|
9
|
SyS_madvise
|
vmlinux |
0xffffffff915ed42d
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x49d
|
|
10
|
|
vmlinux |
0xffffffff915de32b
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xeb
|
|
11
|
system_call_fastpath
|
vmlinux |
0xffffffff91b94f92
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [systemd-journal:15084]
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [java:15334]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter sunrpc ext4 mbcache jbd2 dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev i2c_piix4 sg joydev pcspkr virtio_balloon parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi bochs_drm drm_kms_helper virtio_net net_failover failover syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm ahci ata_piix libahci crc32c_intel
libata serio_raw virtio_pci virtio_ring drm_panel_orientation_quirks virtio floppy fuse
CPU: 2 PID: 15334 Comm: java Tainted: G L ------------ 3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
task: ffff8e565ff9b180 ti: ffff8e5538f9c000 task.ti: ffff8e5538f9c000
RIP: 0010:[<ffffffff91516e4a>] [<ffffffff91516e4a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8e5538f9fc70 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 000800009147e6a0 RCX: ffff8e5937ca0628
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8e5538f9fca8 R08: ffff8e567937a800 R09: ffffffff91786479
R10: ffff8e5937d1f160 R11: ffffe44449aa9c00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8e5538f9fc20 R15: 0000000000000001
FS: 00007f9d6f206700(0000) GS:ffff8e5937d00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9d6f1f1a18 CR3: 00000003f2a4a000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9147e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9147e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff915ede86>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff915ef94f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff915efa73>] tlb_finish_mmu+0x23/0x30
[<ffffffff915f2f3a>] zap_page_range+0xea/0x130
[<ffffffff915fcb7c>] ? do_mmap+0x39c/0x590
[<ffffffff915ed42d>] SyS_madvise+0x49d/0xac0
[<ffffffff915de32b>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff91b94f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 de 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 15 92 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ad 86 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #56223
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9bd16a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9bc7e478
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9bc7e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff9bdec026
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff9bdedaef
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff9bdedc13
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff9bdf091a
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff9bdeb62d
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffff9bdddfdb
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xeb
|
|
10
|
tracesys
|
vmlinux |
0xffffffff9c38e172
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [elasticsearch[c:59626]
Modules linked in: tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter fuse vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac coretemp iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper snd_ens1371 vmw_balloon snd_rawmidi snd_ac97_codec nfit ablk_helper cryptd ac97_bus snd_seq libnvdimm snd_seq_device snd_pcm pcspkr snd_timer snd joydev soundcore vmw_vmci sg i2c_piix4 parport_pc parport ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi libata scsi_transport_spi mptscsih crct10dif_pclmul crct10dif_common mptbase crc32c_intel e1000 drm_panel_orientation_quirks serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 59626 Comm: elasticsearch[c Tainted: G L ------------ 3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff88a2cad30000 ti: ffff88a2ed32c000 task.ti: ffff88a2ed32c000
RIP: 0010:[<ffffffff9bd16a6a>] [<ffffffff9bd16a6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff88a2ed32fc70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009bc7e2b0 RCX: ffff88a4f5620600
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff88a2ed32fca8 R08: ffff88a43c84bc00 R09: ffffffff9bf817e9
R10: ffff88a4f565f160 R11: ffffbbce02e23200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff88a2ed32fc20 R15: 0000000000000001
FS: 00007f0694c6e700(0000) GS:ffff88a4f5640000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055af9bb36acc CR3: 00000002217e8000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9bc7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9bc7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9bdec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9bdedaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9bdedc13>] tlb_finish_mmu+0x23/0x30
[<ffffffff9bdf091a>] zap_page_range+0xea/0x130
[<ffffffff9bdeb62d>] SyS_madvise+0x49d/0xac0
[<ffffffff9bdddfdb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffff9c38e172>] tracesys+0xa6/0xcc
Code: 48 63 35 fe 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 94 9c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #9570
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb2716a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb267e478
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffb267e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffb27ec026
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffb27edaef
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffb27edc13
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffb27f091a
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffb27eb62d
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x49d
|
|
9
|
|
vmlinux |
0xffffffffb27ddfdb
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xeb
|
|
10
|
system_call_fastpath
|
vmlinux |
0xffffffffb2d8dede
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [RMI TCP Connect:17128]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter binfmt_misc sunrpc nfit libnvdimm ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper sg ablk_helper cryptd joydev pcspkr virtio_balloon parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom cirrus ata_generic drm_kms_helper pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm virtio_console virtio_blk ata_piix crct10dif_pclmul drm 8139too libata crct10dif_common crc32c_intel virtio_pci virtio_ring serio_raw 8139cp floppy drm_panel_orientation_quirks
mii virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 17128 Comm: RMI TCP Connect Kdump: loaded Tainted: G L ------------ 3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff8b31ef1541c0 ti: ffff8b30cfdc0000 task.ti: ffff8b30cfdc0000
RIP: 0010:[<ffffffffb2716a6a>] [<ffffffffb2716a6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b30cfdc3c70 EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00080000b267e2b0 RCX: ffff8b323a120320
RDX: 0000000000000002 RSI: 000000000000000a RDI: 0000000000000000
RBP: ffff8b30cfdc3ca8 R08: ffff8b3237ab9c00 R09: ffffffffb29817e9
R10: ffff8b323a09f160 R11: ffffd6dcc5d46a00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8b30cfdc3c20 R15: 0000000000000001
FS: 00007f2f0f4cc700(0000) GS:ffff8b323a080000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000013460001 CR3: 00000000a7120000 CR4: 00000000007606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffb267e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb267e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb27ec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb27edaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb27edc13>] tlb_finish_mmu+0x23/0x30
[<ffffffffb27f091a>] zap_page_range+0xea/0x130
[<ffffffffb27eb62d>] SyS_madvise+0x49d/0xac0
[<ffffffffb27ddfdb>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb2d8dede>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 fe 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 34 b3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #997488
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa931e23a
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa9281fa8
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa928201b
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffa93fa07e
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0xae
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffa93fbbff
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa93fbd33
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffa93ff27a
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0xea
|
|
8
|
|
vmlinux |
0xffffffffa940228a
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0xb3a
|
|
9
|
|
vmlinux |
0xffffffffa940926c
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0x39c
|
|
10
|
SyS_madvise
|
vmlinux |
0xffffffffa93f966d
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0x49d
|
|
11
|
system_call_fastpath
|
vmlinux |
0xffffffffa99c539a
Build id: 3.10.0-1160.108.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 25s! [InactivityMonit:7913]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter nfsv3 nfs_acl nfs lockd grace fscache vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 auth_rpcgss binfmt_misc sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops sd_mod ttm crc_t10dif crct10dif_generic ahci drm ata_piix libahci
nfit libata crct10dif_pclmul crct10dif_common libnvdimm crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 7 PID: 7913 Comm: InactivityMonit Kdump: loaded Tainted: G L ------------ 3.10.0-1160.108.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff97f95a6d0000 ti: ffff97f9c6f7c000 task.ti: ffff97f9c6f7c000
RIP: 0010:[<ffffffffa931e23a>] [<ffffffffa931e23a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff97f9c6f7fc70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a9281b10 RCX: ffff97f90da20750
RDX: 0000000000000000 RSI: 000000000000001e RDI: 0000000000000000
RBP: ffff97f9c6f7fca8 R08: ffff97ede0592000 R09: ffffffffa959c7b9
R10: ffff97f90dbdf160 R11: fffff947e3fbf800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff97f9c6f7fc20 R15: 0000000000000001
FS: 00007fc04fafa700(0000) GS:ffff97f90dbc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc1313062b0 CR3: 000000190c528000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffa9281fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa928201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa93fa07e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa93fbbff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa93fbd33>] tlb_finish_mmu+0x23/0x40
[<ffffffffa93ff27a>] zap_page_range+0xea/0x140
[<ffffffffa940228a>] ? handle_mm_fault+0xb3a/0x1190
[<ffffffffa940926c>] ? do_mmap+0x39c/0x590
[<ffffffffa93f966d>] SyS_madvise+0x49d/0xac0
[<ffffffffa99c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 8e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 80 19 f5 a9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d 18 c4 00 eb b7 0f b6 4d cc 4c 89