Complete report #534612
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f99a6
Build id: 3.10.0-693.el7.x86_64
|
0x206
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.63
|
vmlinux |
0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
|
0x33
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
|
0x55
|
|
6
|
unmap_region
|
vmlinux |
0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
|
0xf4
|
|
7
|
|
vmlinux |
0xffffffff81331f48
Build id: 3.10.0-693.el7.x86_64
|
0x48
|
|
8
|
|
vmlinux |
0xffffffff811b5a01
Build id: 3.10.0-693.el7.x86_64
|
0x121
|
|
9
|
do_munmap
|
vmlinux |
0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
|
0x292
|
|
10
|
vm_munmap
|
vmlinux |
0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
|
0x65
|
|
11
|
SyS_munmap
|
vmlinux |
0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
|
0x22
|
|
12
|
system_call_fastpath
|
vmlinux |
0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [java:116136]
Modules linked in: uio_ivshmem(OE) gsch(OE) redirfs(OE) binfmt_misc nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter uio ppdev parport_pc sg i2c_piix4 pcspkr parport virtio_balloon joydev nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci floppy virtio_ring i2c_core serio_raw virtio dm_mirror dm_region_hash dm_log dm_mod [last unloaded: redirfs]
CPU: 14 PID: 116136 Comm: java Tainted: G OEL ------------ 3.10.0-693.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff880813ecbf40 ti: ffff880801ee8000 task.ti: ffff880801ee8000
RIP: 0010:[<ffffffff810f99a6>] [<ffffffff810f99a6>] smp_call_function_many+0x206/0x260
RSP: 0018:ffff880801eebcf0 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008106e4a0 RCX: ffff88081ce1af70
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff880801eebd28 R08: ffff880174b8ec00 R09: ffffffff813227d9
R10: ffff88081d199c80 R11: ffffea00204df800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff880801eebca0 R15: 00007f0c340df000
FS: 00007f0c87fff700(0000) GS:ffff88081d180000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0c340df000 CR3: 000000080aae5000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000113ecbf40 ffff880819321c48 ffff880819321900 0000000000000000
ffffffffffffffff ffff880819321c48 ffff880034957290 ffff880801eebd78
ffffffff8106e668 ffff880819321900 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff81331f48>] ? lockref_put_or_lock+0x48/0x80
[<ffffffff811b5a01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #409300
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb2d07e3a
Build id: 3.10.0-862.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb2c77018
Build id: 3.10.0-862.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffb2c77088
Build id: 3.10.0-862.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.75
|
vmlinux |
0xffffffffb2dc0497
Build id: 3.10.0-862.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffb2dc1c75
Build id: 3.10.0-862.el7.x86_64
|
0x55
|
|
6
|
unmap_region
|
vmlinux |
0xffffffffb2dca624
Build id: 3.10.0-862.el7.x86_64
|
0xf4
|
|
7
|
|
vmlinux |
0xffffffffb2f683ed
Build id: 3.10.0-862.el7.x86_64
|
0xd
|
|
8
|
|
vmlinux |
0xffffffffb2cbba41
Build id: 3.10.0-862.el7.x86_64
|
0x31
|
|
9
|
|
vmlinux |
0xffffffffb2dcabe1
Build id: 3.10.0-862.el7.x86_64
|
0x121
|
|
10
|
do_munmap
|
vmlinux |
0xffffffffb2dccbf5
Build id: 3.10.0-862.el7.x86_64
|
0x295
|
|
11
|
vm_munmap
|
vmlinux |
0xffffffffb2dcce35
Build id: 3.10.0-862.el7.x86_64
|
0x65
|
|
12
|
SyS_munmap
|
vmlinux |
0xffffffffb2dce052
Build id: 3.10.0-862.el7.x86_64
|
0x22
|
|
13
|
system_call_fastpath
|
vmlinux |
0xffffffffb331f7d5
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [python2.7:55644]
Modules linked in: xt_iprange ip6table_filter ip6_tables iptable_filter binfmt_misc joydev bochs_drm ttm sb_edac drm_kms_helper iosf_mbi ppdev virtio_rng sg syscopyarea kvm_intel kvm sysfillrect sysimgblt irqbypass fb_sys_fops crc32_pclmul ghash_clmulni_intel i2c_piix4 drm aesni_intel lrw parport_pc i2c_core gf128mul parport glue_helper ablk_helper cryptd pcspkr ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_net virtio_console virtio_scsi crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libata serio_raw virtio_pci virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 55644 Comm: python2.7 Kdump: loaded Not tainted 3.10.0-862.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff99729e379fa0 ti: ffff99736a7cc000 task.ti: ffff99736a7cc000
RIP: 0010:[<ffffffffb2d07e3a>] [<ffffffffb2d07e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff99736a7cfcb8 EFLAGS: 00000202
RAX: 0000000000000002 RBX: 0000000000019680 RCX: ffff99737f69cf28
RDX: 0000000000000002 RSI: 00000000000000ff RDI: 0000000000000000
RBP: ffff99736a7cfcf0 R08: ffff9965ea6a2c00 R09: ffffffffb2f4a3d9
R10: ffff99737f71bb00 R11: fffffc19b895fe00 R12: ffff99736a7cfc68
R13: 00007f73a22de000 R14: 0000000000000001 R15: fffffc19b72a6d80
FS: 00007f738effd700(0000) GS:ffff99737f700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffb2dfe4080 CR3: 0000000e63366000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb2c77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb2c77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb2dc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffb2dc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffb2dca624>] unmap_region+0xf4/0x140
[<ffffffffb2f683ed>] ? list_del+0xd/0x30
[<ffffffffb2cbba41>] ? remove_wait_queue+0x31/0x40
[<ffffffffb2dcabe1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb2dccbf5>] do_munmap+0x295/0x470
[<ffffffffb2dcce35>] vm_munmap+0x65/0xb0
[<ffffffffb2dce052>] SyS_munmap+0x22/0x30
[<ffffffffb331f7d5>] system_call_fastpath+0x1c/0x21
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 93 b3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89
Complete report #254544
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
|
0x202
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.63
|
vmlinux |
0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
|
0x33
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
|
0x55
|
|
6
|
unmap_region
|
vmlinux |
0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
|
0xf4
|
|
7
|
|
vmlinux |
0xffffffff81331f48
Build id: 3.10.0-693.el7.x86_64
|
0x48
|
|
8
|
|
vmlinux |
0xffffffff811b5a01
Build id: 3.10.0-693.el7.x86_64
|
0x121
|
|
9
|
do_munmap
|
vmlinux |
0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
|
0x292
|
|
10
|
vm_munmap
|
vmlinux |
0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
|
0x65
|
|
11
|
SyS_munmap
|
vmlinux |
0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
|
0x22
|
|
12
|
system_call_fastpath
|
vmlinux |
0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 24s! [ftdc:1780]
Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac edac_core iosf_mbi crc32_pclmul snd_ens1371 snd_rawmidi ghash_clmulni_intel snd_ac97_codec ac97_bus aesni_intel lrw snd_seq gf128mul glue_helper ablk_helper snd_seq_device cryptd snd_pcm ppdev nfit vmw_balloon sg snd_timer pcspkr snd joydev vmw_vmci soundcore libnvdimm i2c_piix4
shpchp parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod ata_generic crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi crct10dif_pclmul scsi_transport_spi crct10dif_common mptscsih crc32c_intel libata e1000 mptbase serio_raw i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 1780 Comm: ftdc Tainted: G L ------------ 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff880232b95ee0 ti: ffff880221770000 task.ti: ffff880221770000
RIP: 0010:[<ffffffff810f99a2>] [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff880221773cf0 EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800008106e4a0 RCX: ffff88023569af98
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff880221773d28 R08: ffff88017cf35800 R09: ffffffff813227d9
R10: ffff880235719c80 R11: ffffea0008c85400 R12: 0000000000000286
R13: 0000000000000018 R14: ffff880221773c60 R15: 0000000000000286
FS: 00007f9ac5948700(0000) GS:ffff880235700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9ad1914000 CR3: 0000000035e88000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000132b95ee0 ffff880232b3a288 ffff880232b39f40 00007f9ad1914000
00007f9ad1915000 ffff880232b3a288 ffff880232247a28 ffff880221773d78
ffffffff8106e668 ffff880232b39f40 00007f9ad1914000 00007f9ad1915000
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff81331f48>] ? lockref_put_or_lock+0x48/0x80
[<ffffffff811b5a01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89
Complete report #138383
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff94d16eaa
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff94c7e868
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff94c7e8db
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff94dedee6
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff94def9af
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff94defad3
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x23
|
|
7
|
unmap_region
|
vmlinux |
0xffffffff94df92c4
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xf4
|
|
8
|
|
vmlinux |
0xffffffff94cc6b50
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x30
|
|
9
|
do_munmap
|
vmlinux |
0xffffffff94dfb8b5
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x2a5
|
|
10
|
vm_munmap
|
vmlinux |
0xffffffff94dfbaf5
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x65
|
|
11
|
SyS_munmap
|
vmlinux |
0xffffffff94dfcd92
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x22
|
|
12
|
system_call_fastpath
|
vmlinux |
0xffffffff95393f92
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x25
|
|
13
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff95393ed5
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xa2
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [f2b/f.webmin-au:1428]
Modules linked in: nfsv3 nfs_acl ppp_deflate ppp_async crc_ccitt ppp_generic slhc rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod pcspkr sg virtio_balloon joydev i2c_piix4 ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_common
ata_generic pata_acpi virtio_net net_failover failover bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata e1000 virtio_pci serio_raw virtio_ring virtio drm_panel_orientation_quirks floppy
CPU: 1 PID: 1428 Comm: f2b/f.webmin-au Kdump: loaded Tainted: G L ------------ 3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-48-gd9c812dda519-prebuilt.qemu.org 04/01/2014
task: ffffa0b4f7aae300 ti: ffffa0b4f7fd0000 task.ti: ffffa0b4f7fd0000
RIP: 0010:[<ffffffff94d16eaa>] [<ffffffff94d16eaa>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa0b4f7fd3cb0 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0008000094c7e6a0 RCX: ffffa0b4ffc20298
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffffa0b4f7fd3ce8 R08: ffffa0b4f8f89800 R09: ffffffff94f86199
R10: ffffa0b4ffc9f160 R11: fffff7fb01effd00 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa0b4f7fd3c60 R15: 0000000000000000
FS: 00007fe8a2d39700(0000) GS:ffffa0b4ffc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055c1db5ac7a0 CR3: 000000007761c000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff94c7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff94c7e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff94dedee6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff94def9af>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff94defad3>] tlb_finish_mmu+0x23/0x30
[<ffffffff94df92c4>] unmap_region+0xf4/0x140
[<ffffffff94cc6b50>] ? remove_wait_queue+0x30/0x40
[<ffffffff94dfb8b5>] do_munmap+0x2a5/0x480
[<ffffffff94dfbaf5>] vm_munmap+0x65/0xb0
[<ffffffff94dfcd92>] SyS_munmap+0x22/0x30
[<ffffffff95393f92>] system_call_fastpath+0x25/0x2a
[<ffffffff95393ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 7e 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 95 95 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 86 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #5832
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff825175ca
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8247e8f8
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8247e96b
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff825ede16
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff825ef8df
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff825efa03
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x23
|
|
7
|
unmap_region
|
vmlinux |
0xffffffff825f9214
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0xf4
|
|
8
|
|
vmlinux |
0xffffffff82665b30
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x50
|
|
9
|
|
vmlinux |
0xffffffff825f97d1
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x121
|
|
10
|
do_munmap
|
vmlinux |
0xffffffff825fb805
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x2a5
|
|
11
|
vm_munmap
|
vmlinux |
0xffffffff825fba45
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x65
|
|
12
|
SyS_munmap
|
vmlinux |
0xffffffff825fcce2
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x22
|
|
13
|
system_call_fastpath
|
vmlinux |
0xffffffff82b92ed2
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [PressureMonitor:4009]
<EOI>
Modules linked in:
binfmt_misc tcp_lp fuse xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq aesni_intel vmw_balloon snd_seq_device lrw gf128mul glue_helper ablk_helper cryptd snd_pcm joydev pcspkr snd_timer sg snd soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit libnvdimm ata_piix crct10dif_pclmul drm crct10dif_common libata crc32c_intel mptspi e1000 scsi_transport_spi mptscsih mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 4009 Comm: PressureMonitor Kdump: loaded Tainted: G L ------------ T 3.10.0-1127.8.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9c8b79e89070 ti: ffff9c8b35138000 task.ti: ffff9c8b35138000
RIP: 0010:[<ffffffff825175ca>] [<ffffffff825175ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9c8b3513bcb0 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000800008247e730 RCX: ffff9c8bf96e0600
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9c8b3513bce8 R08: ffff9c8bf7663000 R09: ffffffff82785119
R10: ffff9c8bf965f160 R11: ffffdcd582e7ea00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9c8b3513bc60 R15: 00007ff6a4169000
FS: 00007ff638ddd700(0000) GS:ffff9c8bf9640000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff6a4169000 CR3: 000000010d9cc000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8247e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8247e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff825ede16>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff825ef8df>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff825efa03>] tlb_finish_mmu+0x23/0x30
[<ffffffff825f9214>] unmap_region+0xf4/0x140
[<ffffffff82665b30>] ? d_free+0x50/0x70
[<ffffffff825f97d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff825fb805>] do_munmap+0x2a5/0x480
[<ffffffff825fba45>] vm_munmap+0x65/0xb0
[<ffffffff825fcce2>] SyS_munmap+0x22/0x30
[<ffffffff82b92ed2>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 9e 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 10 15 83 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d 7b c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1109922
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810fd5fa
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff81070b98
Build id: 3.10.0-693.21.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff81070c68
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.63
|
vmlinux |
0xffffffff811b18c3
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x33
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff811b2be5
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x55
|
|
6
|
unmap_region
|
vmlinux |
0xffffffff811b9cb4
Build id: 3.10.0-693.21.1.el7.x86_64
|
0xf4
|
|
7
|
|
vmlinux |
0xffffffff812224b8
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x108
|
|
8
|
|
vmlinux |
0xffffffff8113d14d
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x1d
|
|
9
|
|
vmlinux |
0xffffffff8121ce9f
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x4f
|
|
10
|
|
vmlinux |
0xffffffff811ba271
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x121
|
|
11
|
do_munmap
|
vmlinux |
0xffffffff811bc285
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x295
|
|
12
|
vm_munmap
|
vmlinux |
0xffffffff811bc4c5
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x65
|
|
13
|
SyS_munmap
|
vmlinux |
0xffffffff811bd5d2
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x22
|
|
14
|
system_call_fastpath
|
vmlinux |
0xffffffff816c0715
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x1c
|
|
15
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff816c0661
Build id: 3.10.0-693.21.1.el7.x86_64
|
0xae
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [java:28688]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun binfmt_misc ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev joydev vmw_balloon parport_pc parport pcspkr sg vmw_vmci shpchp i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc
ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common vmxnet3 mptspi crc32c_intel scsi_transport_spi serio_raw mptscsih mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 28688 Comm: java Tainted: G L ------------ 3.10.0-693.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffff880406711fa0 ti: ffff8801d26f4000 task.ti: ffff8801d26f4000
RIP: 0010:[<ffffffff810fd5fa>] [<ffffffff810fd5fa>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8801d26f7cc0 EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffff810709d0 RCX: ffff88043fc1cd08
RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff8801d26f7cf8 R08: ffff88017a365c00 R09: ffffffff81328989
R10: ffff88043fd1ba00 R11: ffffea00100cd300 R12: ffff8801d26f7c90
R13: ffffffff813288df R14: ffff88017fc01500 R15: 00007fe722d69000
FS: 00007fe72258f700(0000) GS:ffff88043fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe722d69000 CR3: 00000004102e4000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff81070b98>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81070c68>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811b18c3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811b2be5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b9cb4>] unmap_region+0xf4/0x140
[<ffffffff812224b8>] ? evict+0x108/0x180
[<ffffffff8113d14d>] ? call_rcu_sched+0x1d/0x20
[<ffffffff8121ce9f>] ? d_free+0x4f/0x70
[<ffffffff811ba271>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811bc285>] do_munmap+0x295/0x470
[<ffffffff811bc4c5>] vm_munmap+0x65/0xb0
[<ffffffff811bd5d2>] SyS_munmap+0x22/0x30
[<ffffffff816c0715>] system_call_fastpath+0x1c/0x21
[<ffffffff816c0661>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 8e 1b a3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 b2 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d 1b a3 00 eb b7 0f b6 4d cc 4c 89
Complete report #1026948
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8cb1173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8ca7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8ca7a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffff8cbe4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff8cbe5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
unmap_region
|
vmlinux |
0xffffffff8cbef014
Build id: 3.10.0-957.el7.x86_64
|
0xf4
|
|
7
|
|
vmlinux |
0xffffffff8cc593df
Build id: 3.10.0-957.el7.x86_64
|
0x4f
|
|
8
|
|
vmlinux |
0xffffffff8cbef5d1
Build id: 3.10.0-957.el7.x86_64
|
0x121
|
|
9
|
do_munmap
|
vmlinux |
0xffffffff8cbf1605
Build id: 3.10.0-957.el7.x86_64
|
0x2a5
|
|
10
|
vm_munmap
|
vmlinux |
0xffffffff8cbf1845
Build id: 3.10.0-957.el7.x86_64
|
0x65
|
|
11
|
SyS_munmap
|
vmlinux |
0xffffffff8cbf2ad2
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
|
12
|
system_call_fastpath
|
vmlinux |
0xffffffff8d174ddb
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [Wt-8:14154]
Modules linked in: tcp_lp fuse xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev crc32_pclmul ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper
cryptd joydev snd_ens1371 snd_rawmidi pcspkr snd_ac97_codec ac97_bus snd_seq snd_seq_device sg snd_pcm snd_timer snd soundcore parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm e1000 ata_piix mptspi scsi_transport_spi drm mptscsih libata mptbase drm_panel_orientation_quirks nfit libnvdimm
CPU: 6 PID: 14154 Comm: Wt-8 Kdump: loaded Tainted: G L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9e598689e180 ti: ffff9e59930c8000 task.ti: ffff9e59930c8000
RIP: 0010:[<ffffffff8cb1173a>] [<ffffffff8cb1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9e59930cbcb8 EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00000000000000fc RCX: ffff9e5b357e0678
RDX: 0000000000000007 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9e59930cbcf0 R08: ffff9e5a7c85c800 R09: ffffffff8cd755c9
R10: ffff9e5b3579f160 R11: fffffc04822d5e00 R12: 0000000000000296
R13: ffff9e59930cbc68 R14: 0000000000000001 R15: fffffc0480680700
FS: 00007f55d4e7e700(0000) GS:ffff9e5b35780000(0000) knlGS:00003fffb5d27000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f55d4f9f000 CR3: 00000001dcf26000 CR4: 0000000000360fe0
Call Trace:
[<ffffffff8ca7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8ca7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff8cbe4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff8cbe5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff8cbef014>] unmap_region+0xf4/0x140
[<ffffffff8cc593df>] ? d_free+0x4f/0x70
[<ffffffff8cbef5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff8cbf1605>] do_munmap+0x2a5/0x480
[<ffffffff8cbf1845>] vm_munmap+0x65/0xb0
[<ffffffff8cbf2ad2>] SyS_munmap+0x22/0x30
[<ffffffff8d174ddb>] system_call_fastpath+0x22/0x27
Complete report #820954
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffaed1173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffaec7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffaec7a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffffaede4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffaede5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
unmap_region
|
vmlinux |
0xffffffffaedef014
Build id: 3.10.0-957.el7.x86_64
|
0xf4
|
|
7
|
|
vmlinux |
0xffffffffaee593df
Build id: 3.10.0-957.el7.x86_64
|
0x4f
|
|
8
|
|
vmlinux |
0xffffffffaedef5d1
Build id: 3.10.0-957.el7.x86_64
|
0x121
|
|
9
|
do_munmap
|
vmlinux |
0xffffffffaedf1605
Build id: 3.10.0-957.el7.x86_64
|
0x2a5
|
|
10
|
vm_munmap
|
vmlinux |
0xffffffffaedf1845
Build id: 3.10.0-957.el7.x86_64
|
0x65
|
|
11
|
SyS_munmap
|
vmlinux |
0xffffffffaedf2ad2
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
|
12
|
system_call_fastpath
|
vmlinux |
0xffffffffaf374ddb
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
|
13
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffaf374d21
Build id: 3.10.0-957.el7.x86_64
|
0xae
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [java:8687]
Modules linked in: binfmt_misc sysmon_edr(OE) netfilter_edr(OE) ppdev sg i2c_piix4 joydev virtio_balloon parport_pc parport pcspkr ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_blk virtio_console virtio_scsi virtio_net serio_raw cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm floppy drm ata_piix libata virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 8687 Comm: java Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
task: ffff9cf93622d140 ti: ffff9cf93615c000 task.ti: ffff9cf93615c000
RIP: 0010:[<ffffffffaed1173a>] [<ffffffffaed1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9cf93615fcb8 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00000000000000fc RCX: ffff9d00dec60320
RDX: 0000000000000001 RSI: 0000000000000028 RDI: 0000000000000000
RBP: ffff9cf93615fcf0 R08: ffff9cfa34578800 R09: ffffffffaef755c9
R10: ffff9d00ded1f160 R11: ffffed43924bc400 R12: 0000000000000296
R13: ffff9cf93615fc68 R14: 0000000000000001 R15: ffffed4397ae1d80
FS: 00007f7b25674700(0000) GS:ffff9d00ded00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7c14001000 CR3: 00000005b42c2000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaec7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaec7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffaede4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffaede5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffaedef014>] unmap_region+0xf4/0x140
[<ffffffffaee593df>] ? d_free+0x4f/0x70
[<ffffffffaedef5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffaedf1605>] do_munmap+0x2a5/0x480
[<ffffffffaedf1845>] vm_munmap+0x65/0xb0
[<ffffffffaedf2ad2>] SyS_munmap+0x22/0x30
[<ffffffffaf374ddb>] system_call_fastpath+0x22/0x27
[<ffffffffaf374d21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 94 af f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #728984
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff96f0862a
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff96e77788
Build id: 3.10.0-862.9.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff96e777f8
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.75
|
vmlinux |
0xffffffff96fc0c57
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff96fc2435
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x55
|
|
6
|
unmap_region
|
vmlinux |
0xffffffff96fcade4
Build id: 3.10.0-862.9.1.el7.x86_64
|
0xf4
|
|
7
|
|
vmlinux |
0xffffffff970333cf
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x4f
|
|
8
|
|
vmlinux |
0xffffffff96fcb3a1
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x121
|
|
9
|
do_munmap
|
vmlinux |
0xffffffff96fcd3b5
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x295
|
|
10
|
vm_munmap
|
vmlinux |
0xffffffff96fcd5f5
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x65
|
|
11
|
SyS_munmap
|
vmlinux |
0xffffffff96fce812
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x22
|
|
12
|
system_call_fastpath
|
vmlinux |
0xffffffff97520795
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x1c
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ora_cjq0_pdbhdf:2336]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 ppdev snd_rawmidi aesni_intel vmw_balloon snd_ac97_codec lrw gf128mul ac97_bus glue_helper snd_seq ablk_helper cryptd snd_seq_device snd_pcm joydev pcspkr snd_timer snd soundcore vmw_vmci nfit sg shpchp i2c_piix4 libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic
sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea crct10dif_pclmul crct10dif_common crc32c_intel sysfillrect sysimgblt fb_sys_fops ttm drm serio_raw ata_piix libata e1000 mptspi scsi_transport_spi mptscsih mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 2336 Comm: ora_cjq0_pdbhdf Kdump: loaded Tainted: G L ------------ 3.10.0-862.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff9f338de10000 ti: ffff9f3270314000 task.ti: ffff9f3270314000
RIP: 0010:[<ffffffff96f0862a>] [<ffffffff96f0862a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9f3270317cb8 EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff9f33b309c6a0
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9f3270317cf0 R08: ffff9f2db52bc400 R09: ffffffff9714ae19
R10: ffff9f33b305bb40 R11: fffff95ec5cdc800 R12: 0000000000000296
R13: ffff9f3270317c68 R14: 0000000000000001 R15: fffff95ed51a2800
FS: 00007f2493fff700(0000) GS:ffff9f33b3040000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2521001000 CR3: 000000073e66e000 CR4: 00000000001607e0
Call Trace:
[<ffffffff96e77788>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff96e777f8>] flush_tlb_mm_range+0x68/0x140
[<ffffffff96fc0c57>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff96fc2435>] tlb_finish_mmu+0x55/0x60
[<ffffffff96fcade4>] unmap_region+0xf4/0x140
[<ffffffff970333cf>] ? d_free+0x4f/0x70
[<ffffffff96fcb3a1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff96fcd3b5>] do_munmap+0x295/0x470
[<ffffffff96fcd5f5>] vm_munmap+0x65/0xb0
[<ffffffff96fce812>] SyS_munmap+0x22/0x30
[<ffffffff97520795>] system_call_fastpath+0x1c/0x21
Code: 48 63 35 9e ae c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 b3 97 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d ae c3 00 eb b7 0f b6 4d cc 4c 89
Complete report #425905
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff89d1173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff89c7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff89c7a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffff89de4077
Build id: 3.10.0-957.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffff89de5975
Build id: 3.10.0-957.el7.x86_64
|
0x55
|
|
6
|
unmap_region
|
vmlinux |
0xffffffff89def014
Build id: 3.10.0-957.el7.x86_64
|
0xf4
|
|
7
|
|
vmlinux |
0xffffffff89e593df
Build id: 3.10.0-957.el7.x86_64
|
0x4f
|
|
8
|
|
vmlinux |
0xffffffff89def5d1
Build id: 3.10.0-957.el7.x86_64
|
0x121
|
|
9
|
do_munmap
|
vmlinux |
0xffffffff89df1605
Build id: 3.10.0-957.el7.x86_64
|
0x2a5
|
|
10
|
vm_munmap
|
vmlinux |
0xffffffff89df1845
Build id: 3.10.0-957.el7.x86_64
|
0x65
|
|
11
|
SyS_munmap
|
vmlinux |
0xffffffff89df2ad2
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
|
12
|
system_call_fastpath
|
vmlinux |
0xffffffff8a374ddb
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [sdatad:305180]
Modules linked in: dattobd(OE) sfs(OE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc nfit libnvdimm iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr vmw_balloon joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom
crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm mptspi scsi_transport_spi mptscsih ahci vmxnet3 libahci mptbase drm_panel_orientation_quirks ata_piix libata dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dattobd]
CPU: 12 PID: 305180 Comm: sdatad Kdump: loaded Tainted: G W OEL ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff986df4088000 ti: ffff986df31f0000 task.ti: ffff986df31f0000
RIP: 0010:[<ffffffff89d1173a>] [<ffffffff89d1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff986df31f3cb8 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff98678c820b18
RDX: 0000000000000000 RSI: 0000000000000014 RDI: 0000000000000000
RBP: ffff986df31f3cf0 R08: ffff986c1fc08800 R09: ffffffff89f755c9
R10: ffff986c64a1f160 R11: fffffb8520d5ba00 R12: 0000000000000296
R13: ffff986df31f3c68 R14: 0000000000000001 R15: fffffb851703e800
FS: 00007f7841c098c0(0000) GS:ffff986c64a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7841c00000 CR3: 0000000773a3c000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff89c7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff89c7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff89de4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff89de5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff89def014>] unmap_region+0xf4/0x140
[<ffffffff89e593df>] ? d_free+0x4f/0x70
[<ffffffff89def5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff89df1605>] do_munmap+0x2a5/0x480
[<ffffffff89df1845>] vm_munmap+0x65/0xb0
[<ffffffff89df2ad2>] SyS_munmap+0x22/0x30
[<ffffffff8a374ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 94 8a f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #188437
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffaf911a8a
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffaf87a6a8
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffaf87a718
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x68
|
|
4
|
tlb_flush_mmu.part.76
|
vmlinux |
0xffffffffaf9e4687
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x37
|
|
5
|
tlb_finish_mmu
|
vmlinux |
0xffffffffaf9e5f85
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x55
|
|
6
|
unmap_region
|
vmlinux |
0xffffffffaf9ef624
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xf4
|
|
7
|
|
vmlinux |
0xffffffffafa599ef
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x4f
|
|
8
|
|
vmlinux |
0xffffffffaf9efbe1
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x121
|
|
9
|
do_munmap
|
vmlinux |
0xffffffffaf9f1c15
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x2a5
|
|
10
|
vm_munmap
|
vmlinux |
0xffffffffaf9f1e55
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x65
|
|
11
|
SyS_munmap
|
vmlinux |
0xffffffffaf9f30e2
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x22
|
|
12
|
system_call_fastpath
|
vmlinux |
0xffffffffaff75ddb
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x22
|
|
13
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffaff75d21
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xae
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [nessusd:1638]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common ata_piix mptspi scsi_transport_spi mptscsih serio_raw crc32c_intel libata vmxnet3 mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 1638 Comm: nessusd Kdump: loaded Tainted: G L ------------ 3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff927db72d4100 ti: ffff927f27ad8000 task.ti: ffff927f27ad8000
RIP: 0010:[<ffffffffaf911a8a>] [<ffffffffaf911a8a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff927f27adbcb8 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00000000000000fc RCX: ffff927f3fca04b8
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff927f27adbcf0 R08: ffff927e7cf83c00 R09: ffffffffafb75e39
R10: ffff927f3fd1f160 R11: fffff08902cc7700 R12: 0000000000000296
R13: ffff927f27adbc68 R14: 0000000000000001 R15: fffff089058bd240
FS: 00007fb7e0b94780(0000) GS:ffff927f3fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb7e0ba6000 CR3: 0000000229662000 CR4: 00000000000607e0
Call Trace:
[<ffffffffaf87a6a8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaf87a718>] flush_tlb_mm_range+0x68/0x140
[<ffffffffaf9e4687>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffaf9e5f85>] tlb_finish_mmu+0x55/0x60
[<ffffffffaf9ef624>] unmap_region+0xf4/0x140
[<ffffffffafa599ef>] ? d_free+0x4f/0x70
[<ffffffffaf9efbe1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffaf9f1c15>] do_munmap+0x2a5/0x480
[<ffffffffaf9f1e55>] vm_munmap+0x65/0xb0
[<ffffffffaf9f30e2>] SyS_munmap+0x22/0x30
[<ffffffffaff75ddb>] system_call_fastpath+0x22/0x27
[<ffffffffaff75d21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 3e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 bc 54 b0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1086723
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_single
|
vmlinux |
0xffffffff8291dcbf
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x5f
|
|
2
|
|
vmlinux |
0xffffffff82b9c615
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x35
|
|
3
|
smp_call_function_many
|
vmlinux |
0xffffffff8291e27b
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x22b
|
|
4
|
native_flush_tlb_others
|
vmlinux |
0xffffffff82881fa8
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xb8
|
|
5
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8288201b
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x6b
|
|
6
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff829fa0ce
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xae
|
|
7
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff829fbc4f
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x3f
|
|
8
|
tlb_finish_mmu
|
vmlinux |
0xffffffff829fbd83
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x23
|
|
9
|
unmap_region
|
vmlinux |
0xffffffff82a05904
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xf4
|
|
10
|
|
vmlinux |
0xffffffff82a74d70
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x50
|
|
11
|
|
vmlinux |
0xffffffff82a05f11
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x121
|
|
12
|
do_munmap
|
vmlinux |
0xffffffff82a07f65
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x2a5
|
|
13
|
vm_munmap
|
vmlinux |
0xffffffff82a081a5
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x65
|
|
14
|
SyS_munmap
|
vmlinux |
0xffffffff82a094d2
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x22
|
|
15
|
system_call_fastpath
|
vmlinux |
0xffffffff82fc539a
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 33s! [ksoftirqd/7:44]
[<ffffffff8291dcbf>] smp_call_function_single+0x5f/0xa0
[<ffffffff82b9c615>] ? cpumask_next_and+0x35/0x50
[<ffffffff8291e27b>] smp_call_function_many+0x22b/0x270
[<ffffffff82881fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8288201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff829fa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff829fbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff829fbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffff82a05904>] unmap_region+0xf4/0x150
[<ffffffff82a74d70>] ? d_free+0x50/0x80
[<ffffffff82a05f11>] ? __vma_rb_erase+0x121/0x230
[<ffffffff82a07f65>] do_munmap+0x2a5/0x480
[<ffffffff82a081a5>] vm_munmap+0x65/0xb0
[<ffffffff82a094d2>] SyS_munmap+0x22/0x30
[<ffffffff82fc539a>] system_call_fastpath+0x25/0x2a
Complete report #556901
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff96716c0a
Build id: 3.10.0-1160.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9667e868
Build id: 3.10.0-1160.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9667e8db
Build id: 3.10.0-1160.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff967edd06
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff967ef7cf
Build id: 3.10.0-1160.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff967ef8f3
Build id: 3.10.0-1160.el7.x86_64
|
0x23
|
|
7
|
unmap_region
|
vmlinux |
0xffffffff967f90e4
Build id: 3.10.0-1160.el7.x86_64
|
0xf4
|
|
8
|
|
vmlinux |
0xffffffff968665a0
Build id: 3.10.0-1160.el7.x86_64
|
0x50
|
|
9
|
|
vmlinux |
0xffffffff967f96a1
Build id: 3.10.0-1160.el7.x86_64
|
0x121
|
|
10
|
do_munmap
|
vmlinux |
0xffffffff967fb6d5
Build id: 3.10.0-1160.el7.x86_64
|
0x2a5
|
|
11
|
vm_munmap
|
vmlinux |
0xffffffff967fb915
Build id: 3.10.0-1160.el7.x86_64
|
0x65
|
|
12
|
SyS_munmap
|
vmlinux |
0xffffffff967fcbb2
Build id: 3.10.0-1160.el7.x86_64
|
0x22
|
|
13
|
system_call_fastpath
|
vmlinux |
0xffffffff96d93f92
Build id: 3.10.0-1160.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 67s! [vmtoolsd:819]
Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm dns_resolver ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc kvm_intel kvm nfit libnvdimm irqbypass sg crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev ppdev vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul ata_piix crct10dif_common crc32c_intel libata mptspi scsi_transport_spi mptscsih e1000 mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 819 Comm: vmtoolsd Kdump: loaded Tainted: G W L ------------ 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffffa06027272100 ti: ffffa05c7a3b0000 task.ti: ffffa05c7a3b0000
RIP: 0010:[<ffffffff96716c0a>] [<ffffffff96716c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa05c7a3b3cb0 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009667e6a0 RCX: ffffa06029620600
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffffa05c7a3b3ce8 R08: ffffa05c5d3dcc00 R09: ffffffff96985f99
R10: ffffa0602965f160 R11: fffff1330039be00 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa05c7a3b3c60 R15: 00007fa15731e000
FS: 00007fa157309780(0000) GS:ffffa06029640000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa15731e000 CR3: 000000017dfd2000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9667e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9667e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff967edd06>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff967ef7cf>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff967ef8f3>] tlb_finish_mmu+0x23/0x30
[<ffffffff967f90e4>] unmap_region+0xf4/0x140
[<ffffffff968665a0>] ? d_free+0x50/0x70
[<ffffffff967f96a1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff967fb6d5>] do_munmap+0x2a5/0x480
[<ffffffff967fb915>] vm_munmap+0x65/0xb0
[<ffffffff967fcbb2>] SyS_munmap+0x22/0x30
[<ffffffff96d93f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 35 97 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #365747
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa1d170da
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa1c7e258
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffa1c7e2cb
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffa1dee676
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffa1df013f
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffa1df0263
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x23
|
|
7
|
unmap_region
|
vmlinux |
0xffffffffa1df9a54
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0xf4
|
|
8
|
|
vmlinux |
0xffffffffa1e66d90
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x50
|
|
9
|
|
vmlinux |
0xffffffffa1dfa011
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x121
|
|
10
|
do_munmap
|
vmlinux |
0xffffffffa1dfc045
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x2a5
|
|
11
|
vm_munmap
|
vmlinux |
0xffffffffa1dfc285
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x65
|
|
12
|
SyS_munmap
|
vmlinux |
0xffffffffa1dfd582
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x22
|
|
13
|
system_call_fastpath
|
vmlinux |
0xffffffffa2395f92
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [db2sysc:1080]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13005(E) snd_hda_codec_generic ext4 mbcache jbd2 iosf_mbi snd_hda_intel crc32_pclmul snd_hda_codec ghash_clmulni_intel snd_hda_core snd_hwdep snd_seq snd_seq_device ppdev snd_pcm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr snd_timer virtio_rng parport_pc virtio_balloon parport snd i2c_piix4 soundcore ip_tables xfs libcrc32c ata_generic pata_acpi virtio_console virtio_blk qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix libata drm e1000 crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1080 Comm: db2sysc Tainted: P E ------------ 3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module+el8.3.0+7860+a7792d29 04/01/2014
task: ffff8933d302b180 ti: ffff893477e50000 task.ti: ffff893477e50000
RIP: 0010:[<ffffffffa1d170da>] [<ffffffffa1d170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff893477e53cb0 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00080000a1c7ddd0 RCX: ffff89347fda0320
RDX: 0000000000000003 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff893477e53ce8 R08: ffff893478495800 R09: ffffffffa1f86bf9
R10: ffff89347fc9f160 R11: ffffc7c9410bcf00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff893477e53c60 R15: 00007f3a8aaf3000
FS: 00007f3a6f7fe700(0000) GS:ffff89347fc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3a8aaf3000 CR3: 00000000aa76a000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa1c7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa1c7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa1dee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa1df013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa1df0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffa1df9a54>] unmap_region+0xf4/0x140
[<ffffffffa1e66d90>] ? d_free+0x50/0x70
[<ffffffffa1dfa011>] ? __vma_rb_erase+0x121/0x220
[<ffffffffa1dfc045>] do_munmap+0x2a5/0x480
[<ffffffffa1dfc285>] vm_munmap+0x65/0xb0
[<ffffffffa1dfd582>] SyS_munmap+0x22/0x30
[<ffffffffa2395f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 95 a2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #318648
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8307e34e
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xee
|
|
2
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff831ee676
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xa6
|
|
3
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff831f013f
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x3f
|
|
4
|
tlb_finish_mmu
|
vmlinux |
0xffffffff831f0263
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x23
|
|
5
|
unmap_region
|
vmlinux |
0xffffffff831f9a54
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xf4
|
|
6
|
|
vmlinux |
0xffffffff83266d90
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x50
|
|
7
|
|
vmlinux |
0xffffffff831fa011
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x121
|
|
8
|
do_munmap
|
vmlinux |
0xffffffff831fc045
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x2a5
|
|
9
|
vm_munmap
|
vmlinux |
0xffffffff831fc285
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x65
|
|
10
|
SyS_munmap
|
vmlinux |
0xffffffff831fd582
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x22
|
|
11
|
system_call_fastpath
|
vmlinux |
0xffffffff83795f92
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 1702s! [gnome-shell:12731]
Modules linked in: tcp_lp rfcomm xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep sunrpc snd_seq_midi snd_seq_midi_event crc32_pclmul ppdev ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus aesni_intel snd_seq vmw_balloon lrw gf128mul glue_helper snd_seq_device ablk_helper cryptd btusb btrtl btbcm btintel
bluetooth snd_pcm joydev pcspkr snd_timer rfkill snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt nfit fb_sys_fops ttm libnvdimm drm ata_piix crct10dif_pclmul libata crct10dif_common e1000 crc32c_intel mptspi serio_raw scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 12731 Comm: gnome-shell Kdump: loaded Tainted: G L ------------ 3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8e55f60ed280 ti: ffff8e55851f4000 task.ti: ffff8e55851f4000
RIP: 0010:[<ffffffff8307e34e>] [<ffffffff8307e34e>] flush_tlb_mm_range+0xee/0x140
RSP: 0018:ffff8e55851f7d48 EFLAGS: 00000206
RAX: 0000000000000001 RBX: ffffd039001412b0 RCX: 0000000000000020
RDX: ffff8e559e01be80 RSI: 0000000000000020 RDI: 00007f2bc0b23000
RBP: ffff8e55851f7d68 R08: 00007f2bc0b24000 R09: ffffffff8320307c
R10: ffff8e55fb6df0a0 R11: ffffd0390031e940 R12: ffff8e55f60ed280
R13: ffff8e55f81e6580 R14: 00007f2bc0b24000 R15: 00007f2bc0b23fff
FS: 00007f2bc0bd4a80(0000) GS:ffff8e55fb6c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2bc0b23000 CR3: 00000000051c2000 CR4: 00000000003407e0
Call Trace:
[<ffffffff831ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff831f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff831f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff831f9a54>] unmap_region+0xf4/0x140
[<ffffffff83266d90>] ? d_free+0x50/0x70
[<ffffffff831fa011>] ? __vma_rb_erase+0x121/0x220
[<ffffffff831fc045>] do_munmap+0x2a5/0x480
[<ffffffff831fc285>] vm_munmap+0x65/0xb0
[<ffffffff831fd582>] SyS_munmap+0x22/0x30
[<ffffffff83795f92>] system_call_fastpath+0x25/0x2a
Code: 48 39 c1 48 0f 47 c8 4c 89 e0 4c 29 e8 48 c1 e8 0c 48 39 c1 72 2f 4d 39 ec 4d 89 ee 0f 86 56 ff ff ff 0f 1f 00 4c 89 f7 0f 01 3f <66> 66 66 90 49 81 c6 00 10 00 00 4d 39 f4 77 ea e9 38 ff ff ff
Complete report #154139
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8bb150ca
Build id: 3.10.0-1062.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8ba7d398
Build id: 3.10.0-1062.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8ba7d40b
Build id: 3.10.0-1062.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff8bbea166
Build id: 3.10.0-1062.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff8bbebc2f
Build id: 3.10.0-1062.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff8bbebd53
Build id: 3.10.0-1062.el7.x86_64
|
0x23
|
|
7
|
unmap_region
|
vmlinux |
0xffffffff8bbf5544
Build id: 3.10.0-1062.el7.x86_64
|
0xf4
|
|
8
|
|
vmlinux |
0xffffffff8bc60a80
Build id: 3.10.0-1062.el7.x86_64
|
0x50
|
|
9
|
|
vmlinux |
0xffffffff8bbf5b01
Build id: 3.10.0-1062.el7.x86_64
|
0x121
|
|
10
|
do_munmap
|
vmlinux |
0xffffffff8bbf7b45
Build id: 3.10.0-1062.el7.x86_64
|
0x2a5
|
|
11
|
vm_munmap
|
vmlinux |
0xffffffff8bbf7d85
Build id: 3.10.0-1062.el7.x86_64
|
0x65
|
|
12
|
SyS_munmap
|
vmlinux |
0xffffffff8bbf9022
Build id: 3.10.0-1062.el7.x86_64
|
0x22
|
|
13
|
system_call_fastpath
|
vmlinux |
0xffffffff8c18bede
Build id: 3.10.0-1062.el7.x86_64
|
0x25
|
|
14
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff8c18be21
Build id: 3.10.0-1062.el7.x86_64
|
0xae
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [pool-45-thread-:9488]
Modules linked in: fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun bridge stp llc devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec iosf_mbi ac97_bus snd_seq crc32_pclmul ppdev ghash_clmulni_intel snd_seq_device snd_pcm aesni_intel vmw_balloon snd_timer snd vmw_vmci lrw gf128mul glue_helper ablk_helper pcspkr soundcore
joydev cryptd sg i2c_piix4 nfit libnvdimm parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel mptspi libata scsi_transport_spi mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 9488 Comm: pool-45-thread- Kdump: loaded Not tainted 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff95a408f1e2a0 ti: ffff95a2c2ef0000 task.ti: ffff95a2c2ef0000
RIP: 0010:[<ffffffff8bb150ca>] [<ffffffff8bb150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff95a2c2ef3cb0 EFLAGS: 00000202
RAX: 0000000000000007 RBX: ffffffff8ba7d1d0 RCX: ffff95a4b6fe04b8
RDX: 0000000000000007 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff95a2c2ef3ce8 R08: ffff95a3fcfa3400 R09: ffffffff8bd7f629
R10: ffff95a4b6e9f160 R11: fffff6c747ddd800 R12: 0000000000000286
R13: 0000000000000018 R14: ffff95a2c2ef3c28 R15: 0000000000000286
FS: 00007f789d203700(0000) GS:ffff95a4b6e80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f78bc002000 CR3: 00000001c2214000 CR4: 00000000000407e0
Call Trace:
[<ffffffff8ba7d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8ba7d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8bbea166>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff8bbebc2f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8bbebd53>] tlb_finish_mmu+0x23/0x30
[<ffffffff8bbf5544>] unmap_region+0xf4/0x140
[<ffffffff8bc60a80>] ? d_free+0x50/0x70
[<ffffffff8bbf5b01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff8bbf7b45>] do_munmap+0x2a5/0x480
[<ffffffff8bbf7d85>] vm_munmap+0x65/0xb0
[<ffffffff8bbf9022>] SyS_munmap+0x22/0x30
[<ffffffff8c18bede>] system_call_fastpath+0x25/0x2a
[<ffffffff8c18be21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 74 8c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #125238
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff97516c5a
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9747e868
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9747e8db
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff975edd46
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff975ef80f
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff975ef933
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x23
|
|
7
|
unmap_region
|
vmlinux |
0xffffffff975f9124
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xf4
|
|
8
|
|
vmlinux |
0xffffffff976665e0
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x50
|
|
9
|
|
vmlinux |
0xffffffff975f96e1
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x121
|
|
10
|
do_munmap
|
vmlinux |
0xffffffff975fb715
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x2a5
|
|
11
|
vm_munmap
|
vmlinux |
0xffffffff975fb955
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x65
|
|
12
|
SyS_munmap
|
vmlinux |
0xffffffff975fcbf2
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x22
|
|
13
|
system_call_fastpath
|
vmlinux |
0xffffffff97b94f92
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x25
|
|
14
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff97b94ed5
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xa2
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [Execution worke:4266]
Modules linked in: tcp_lp vboxsf(OE) sunrpc snd_intel8x0 snd_ac97_codec ppdev ac97_bus snd_seq snd_seq_device iosf_mbi joydev crc32_pclmul snd_pcm ghash_clmulni_intel snd_timer snd sg aesni_intel parport_pc vboxguest(OE) lrw soundcore gf128mul video glue_helper parport i2c_piix4 ablk_helper cryptd pcspkr binfmt_misc ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel e1000 serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 4266 Comm: Execution worke Kdump: loaded Tainted: G OE ------------ 3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff960e84928000 ti: ffff960e88c38000 task.ti: ffff960e88c38000
RIP: 0010:[<ffffffff97516c5a>] [<ffffffff97516c5a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff960e88c3bcb0 EFLAGS: 00000202
RAX: 0000000000000005 RBX: 000800009747e6a0 RCX: ffff960fdfd5fc10
RDX: 0000000000000005 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff960e88c3bce8 R08: ffff96093fd4f800 R09: ffffffff97785fd9
R10: ffff960fdfc1f160 R11: ffffe32517661e00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff960e88c3bc60 R15: 00007f63940eb000
FS: 00007f636529d700(0000) GS:ffff960fdfc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f63940eb000 CR3: 00000006c4bec000 CR4: 00000000000606f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9747e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9747e8db>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff975edd46>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff975ef80f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff975ef933>] tlb_finish_mmu+0x23/0x30
[<ffffffff975f9124>] unmap_region+0xf4/0x140
[<ffffffff976665e0>] ? d_free+0x50/0x70
[<ffffffff975f96e1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff975fb715>] do_munmap+0x2a5/0x480
[<ffffffff975fb955>] vm_munmap+0x65/0xb0
[<ffffffff975fcbf2>] SyS_munmap+0x22/0x30
[<ffffffff97b94f92>] system_call_fastpath+0x25/0x2a
[<ffffffff97b94ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 15 98 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #111004
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb3b16a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb3a7e478
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffb3a7e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffb3bec026
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffb3bedaef
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffb3bedc13
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x23
|
|
7
|
unmap_region
|
vmlinux |
0xffffffffb3bf7404
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xf4
|
|
8
|
|
vmlinux |
0xffffffffb3c62be0
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x50
|
|
9
|
|
vmlinux |
0xffffffffb3bf79c1
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x121
|
|
10
|
do_munmap
|
vmlinux |
0xffffffffb3bf9a05
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x2a5
|
|
11
|
vm_munmap
|
vmlinux |
0xffffffffb3bf9c45
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x65
|
|
12
|
SyS_munmap
|
vmlinux |
0xffffffffb3bfaee2
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [PressureMonitor:12744]
Modules linked in: binfmt_misc xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) sunrpc snd_seq_midi snd_seq_midi_event coretemp nls_utf8 isofs iosf_mbi squashfs kvm_intel snd_ens1371 kvm loop snd_rawmidi snd_ac97_codec ppdev irqbypass crc32_pclmul joydev pcspkr ghash_clmulni_intel ac97_bus snd_seq aesni_intel vmw_balloon snd_seq_device snd_pcm lrw gf128mul glue_helper snd_timer snd ablk_helper soundcore
cryptd sg parport_pc parport pcc_cpufreq i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit libnvdimm ata_piix e1000 crct10dif_pclmul crct10dif_common crc32c_intel libata mptspi scsi_transport_spi mptscsih mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 12744 Comm: PressureMonitor Kdump: loaded Tainted: G OEL ------------ T 3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffffa07a1e75a0e0 ti: ffffa07a18b78000 task.ti: ffffa07a18b78000
RIP: 0010:[<ffffffffb3b16a6a>] [<ffffffffb3b16a6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa07a18b7bcb0 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000b3a7e2b0 RCX: ffffa07a3b420628
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffffa07a18b7bce8 R08: ffffa079fd0cb800 R09: ffffffffb3d817e9
R10: ffffa07a3b49f160 R11: fffff93a81e6f600 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa07a18b7bc60 R15: 00007f5d4801c000
FS: 00007f5cc184e700(0000) GS:ffffa07a3b480000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5d4801c000 CR3: 00000000a17d4000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb3a7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb3a7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb3bec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb3bedaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb3bedc13>] tlb_finish_mmu+0x23/0x30
[<ffffffffb3bf7404>] unmap_region+0xf4/0x140
[<ffffffffb3c62be0>] ? d_free+0x50/0x70
[<ffffffffb3bf79c1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb3bf9a05>] do_munmap+0x2a5/0x480
[<ffffffffb3bf9c45>] vm_munmap+0x65/0xb0
[<ffffffffb3bfaee2>] SyS_munmap+0x22/0x30
Complete report #19530
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff91b175ca
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff91a7e8f8
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff91a7e96b
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff91bede16
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff91bef8df
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff91befa03
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x23
|
|
7
|
unmap_region
|
vmlinux |
0xffffffff91bf9214
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0xf4
|
|
8
|
|
vmlinux |
0xffffffff91c65b30
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x50
|
|
9
|
|
vmlinux |
0xffffffff91bf97d1
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x121
|
|
10
|
do_munmap
|
vmlinux |
0xffffffff91bfb805
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x2a5
|
|
11
|
vm_munmap
|
vmlinux |
0xffffffff91bfba45
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x65
|
|
12
|
SyS_munmap
|
vmlinux |
0xffffffff91bfcce2
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x22
|
|
13
|
system_call_fastpath
|
vmlinux |
0xffffffff92192ed2
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [PressureMonitor:50061]
Modules linked in: tcp_lp xt_CHECKSUM tun devlink rfcomm ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter cifs ccm dns_resolver fuse ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat vmhgfs(OE) ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) bnep vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 snd_seq_midi snd_seq_midi_event btusb btrtl btbcm coretemp btintel bluetooth iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev snd_ens1371
snd_rawmidi aesni_intel snd_ac97_codec vmw_balloon ac97_bus snd_seq snd_seq_device snd_pcm lrw rfkill gf128mul glue_helper ablk_helper cryptd snd_timer sg snd joydev pcspkr i2c_piix4 soundcore vmw_vmci parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm ata_piix libnvdimm libata crct10dif_pclmul crct10dif_common crc32c_intel e1000 mptspi serio_raw scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 50061 Comm: PressureMonitor Kdump: loaded Tainted: G OE ------------ T 3.10.0-1127.8.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8c89cdb30000 ti: ffff8c8d3e178000 task.ti: ffff8c8d3e178000
RIP: 0010:[<ffffffff91b175ca>] [<ffffffff91b175ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8c8d3e17bcb0 EFLAGS: 00000202
RAX: 0000000000000005 RBX: ffffffff91a7e730 RCX: ffff8c8e97760678
RDX: 0000000000000005 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff8c8d3e17bce8 R08: ffff8c8af7bdc000 R09: ffffffff91d85119
R10: ffff8c8e9771f160 R11: ffffdea7c7dec600 R12: 0000000000000286
R13: ffff8c8d3e17bc60 R14: 00007f8fec061000 R15: 00007f8fec060000
FS: 00007f8f6d031700(0000) GS:ffff8c8e97700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8fec060000 CR3: 000000022648a000 CR4: 00000000001607e0
Call Trace:
[<ffffffff91a7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff91a7e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff91bede16>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff91bef8df>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff91befa03>] tlb_finish_mmu+0x23/0x30
[<ffffffff91bf9214>] unmap_region+0xf4/0x140
[<ffffffff91c65b30>] ? d_free+0x50/0x70
[<ffffffff91bf97d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff91bfb805>] do_munmap+0x2a5/0x480
[<ffffffff91bfba45>] vm_munmap+0x65/0xb0
[<ffffffff91bfcce2>] SyS_munmap+0x22/0x30
[<ffffffff92192ed2>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 9e 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 10 75 92 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d 7b c4 00 eb b7 0f b6 4d cc 4c 89