Info

Permalink
Function
smp_call_function_many
First occurrence
2020-10-11
Last occurrence
2026-08-29
State
NEW
Components
kernel
Quality
-19

Statistics

Operating system Count
CentOS 7 1,641
Architecture Count
x86_64 1,641
Related packages Count
kernel 1,641
 0:3.10.0-957.el7 1,119
 0:3.10.0-862.el7 173
 0:3.10.0-693.el7 159
 0:3.10.0-957.27.2.el7 48
 0:3.10.0-957.5.1.el7 29
 0:3.10.0-957.1.3.el7 22
 0:3.10.0-862.11.6.el7 18
 0:3.10.0-862.14.4.el7 16
 0:3.10.0-693.2.2.el7 12
 0:3.10.0-693.17.1.el7 8
 0:3.10.0-693.21.1.el7 7
 0:3.10.0-862.6.3.el7 6
 0:3.10.0-514.26.2.el7 5
 0:3.10.0-862.3.2.el7 5
 0:3.10.0-957.10.1.el7 4
 0:3.10.0-862.3.3.el7 3
 0:3.10.0-862.9.1.el7 2
 0:3.10.0-957.12.1.el7 1
 0:3.10.0-957.12.2.el7 1
 0:3.10.0-957.21.3.el7 1
 0:3.10.0-693.11.6.el7 1
 0:3.10.0-693.5.2.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1419302
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff811b5a01
Build id: 3.10.0-693.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
0x292
9
vm_munmap
vmlinux 0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
0x22
11
tracesys
vmlinux 0xffffffff816b51d2
Build id: 3.10.0-693.el7.x86_64
0xdd
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [java:54460]
Modules linked in: tcp_diag inet_diag veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd nfit ppdev pcspkr sg joydev vmw_balloon libnvdimm parport_pc parport vmw_vmci shpchp i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel e1000 libata mptspi scsi_transport_spi mptscsih serio_raw mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 54460 Comm: java Tainted: G             L ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff88023259dee0 ti: ffff8801cefa0000 task.ti: ffff8801cefa0000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8801cefa3cf0  EFLAGS: 00000202
RAX: 0000000000000003 RBX: ffffffff8106e4a0 RCX: ffff8802366daf98
RDX: 0000000000000003 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff8801cefa3d28 R08: ffff88017cecc000 R09: ffffffff813227d9
R10: ffff880236719c80 R11: ffffea000657b800 R12: 0000000000000282
R13: ffff8801cefa3ca0 R14: ffff88017fc56900 R15: 00007fade6dcf000
FS:  00007fadfdd11b38(0000) GS:ffff880236700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fade6dca000 CR3: 00000001b5b45000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000000013259dee0 ffff8800a15b28c8 ffff8800a15b2580 00007fade6dc7000
00007fade6dd0000 ffff8800a15b28c8 ffff880185fc7440 ffff8801cefa3d78
ffffffff8106e668 ffff8800a15b2580 00007fade6dc7000 00007fade6dd0000
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff811b5a01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b51d2>] tracesys+0xdd/0xe2
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1409264
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb4307e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb4277018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb4277088
Build id: 3.10.0-862.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffffb43c0497
Build id: 3.10.0-862.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffb43c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffb43ca624
Build id: 3.10.0-862.el7.x86_64
0xf4
7
? xfs_iunlock
xfs 0xffffffffc0501bec
Build id: 3.10.0-862.el7.x86_64
0xec
8
? __vma_rb_erase
vmlinux 0xffffffffb43cabe1
Build id: 3.10.0-862.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffb43ccbf5
Build id: 3.10.0-862.el7.x86_64
0x295
10
vm_munmap
vmlinux 0xffffffffb43cce35
Build id: 3.10.0-862.el7.x86_64
0x65
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [dm_sched_thd:2102]
Modules linked in: tcp_diag udp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi snd_ens1371 crc32_pclmul snd_rawmidi ghash_clmulni_intel snd_ac97_codec ac97_bus snd_seq aesni_intel lrw snd_seq_device gf128mul glue_helper snd_pcm ablk_helper cryptd ppdev snd_timer vmw_balloon snd soundcore pcspkr sg nfit joydev libnvdimm parport_pc parport shpchp vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx crct10dif_pclmul
drm_kms_helper crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix scsi_transport_spi serio_raw drm mptscsih e1000 i2c_core mptbase libata dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 2102 Comm: dm_sched_thd Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff895fd37e8000 ti: ffff895fce9d0000 task.ti: ffff895fce9d0000
RIP: 0010:[<ffffffffb4307e3a>]  [<ffffffffb4307e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff895fce9d3cb8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00000000000000fc RCX: ffff89608b85cdb8
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff895fce9d3cf0 R08: ffff8960891d3800 R09: ffffffffb454a3d9
R10: ffff89608b89bb00 R11: ffffeeae02717000 R12: 0000000000000296
R13: 0000000000000018 R14: ffff895fce9d3c28 R15: 0000000000000296
FS:  00007fb7689af700(0000) GS:ffff89608b880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb8377aa000 CR3: 0000000145bb4000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb4277018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb4277088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb43c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffb43c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffb43ca624>] unmap_region+0xf4/0x140
[<ffffffffc0501bec>] ? xfs_iunlock+0xec/0x130 [xfs]
[<ffffffffb43cabe1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb43ccbf5>] do_munmap+0x295/0x470
[<ffffffffb43cce35>] vm_munmap+0x65/0xb0
Complete report #1197547
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff811b5a01
Build id: 3.10.0-693.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
0x292
9
vm_munmap
vmlinux 0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [qaxsafed:7269]
Modules linked in: qax_tq_base(OE) udp_diag binfmt_misc tcp_diag inet_diag xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables ilocker(OE) nfsv3 nfs fscache gsch(OE) redirfs(OE) tun dsa_filter(POE) ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev
pcspkr joydev vmw_balloon sg i2c_piix4 parport_pc parport shpchp vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul mptspi crct10dif_common crc32c_intel scsi_transport_spi libata mptscsih serio_raw e1000 mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: qax_tq_base]
CPU: 4 PID: 7269 Comm: qaxsafed Tainted: P        W  OEL ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8804142a1fa0 ti: ffff88000f234000 task.ti: ffff88000f234000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff88000f237cf0  EFLAGS: 00000202
RAX: 0000000000000006 RBX: ffffffff8106e4a0 RCX: ffff88043fd9af98
RDX: 0000000000000006 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff88000f237d28 R08: ffff88017a311000 R09: ffffffff813227d9
R10: ffff88043fd19c80 R11: ffffea001008f000 R12: ffff88000f237cc0
R13: ffffffff8132272f R14: ffff88000f237e08 R15: 00007f13b850d000
FS:  00007f13d8731700(0000) GS:ffff88043fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe4ef35c000 CR3: 0000000222e75000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001142a1fa0 ffff88024bb3dac8 ffff88024bb3d780 0000000000000000
ffffffffffffffff ffff88024bb3dac8 ffff88041eb7e360 ffff88000f237d78
ffffffff8106e668 ffff88024bb3d780 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff811b5a01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1197541
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810fd5fa
Build id: 3.10.0-693.21.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff81070b98
Build id: 3.10.0-693.21.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff81070c68
Build id: 3.10.0-693.21.1.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811b18c3
Build id: 3.10.0-693.21.1.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811b2be5
Build id: 3.10.0-693.21.1.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b9cb4
Build id: 3.10.0-693.21.1.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff811ba271
Build id: 3.10.0-693.21.1.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff811bc285
Build id: 3.10.0-693.21.1.el7.x86_64
0x295
9
vm_munmap
vmlinux 0xffffffff811bc4c5
Build id: 3.10.0-693.21.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff811bd5d2
Build id: 3.10.0-693.21.1.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff816c0715
Build id: 3.10.0-693.21.1.el7.x86_64
0x1c
12
? system_call_after_swapgs
vmlinux 0xffffffff816c0661
Build id: 3.10.0-693.21.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [qaxsafed:23170]
Modules linked in: qax_tq_base(OE) cfg80211 rfkill gsch(OE) redirfs(OE) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dsa_filter(POE) snd_hda_codec_generic snd_hda_intel cirrus snd_hda_codec ttm snd_hda_core drm_kms_helper snd_hwdep snd_seq snd_seq_device syscopyarea sysfillrect sysimgblt fb_sys_fops drm virtio_net virtio_console pcspkr virtio_scsi snd_pcm joydev snd_timer snd virtio_balloon sg soundcore shpchp
i2c_piix4 i2c_core virtio_pci virtio_ring virtio ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi ata_piix libata serio_raw floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: qax_tq_base]
CPU: 1 PID: 23170 Comm: qaxsafed Tainted: P           OE  ------------   3.10.0-693.21.1.el7.x86_64 #1
Hardware name: inspur Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-2.module_kos5+70+5e037788 04/01/2014
task: ffff880137bd8fd0 ti: ffff8800a7c80000 task.ti: ffff8800a7c80000
RIP: 0010:[<ffffffff810fd5fa>]  [<ffffffff810fd5fa>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8800a7c83cc0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00080000810709d0 RCX: ffff88013fd1c3b8
RDX: 0000000000000002 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8800a7c83cf8 R08: ffff880139495000 R09: ffffffff81328989
R10: ffff88013fc9ba00 R11: ffffea00017f1800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8800a7c83c70 R15: 00007fa04bc53000
FS:  00007f9f82c18700(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa04bc53000 CR3: 00000000b0842000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff81070b98>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81070c68>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811b18c3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811b2be5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b9cb4>] unmap_region+0xf4/0x140
[<ffffffff811ba271>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811bc285>] do_munmap+0x295/0x470
[<ffffffff811bc4c5>] vm_munmap+0x65/0xb0
[<ffffffff811bd5d2>] SyS_munmap+0x22/0x30
[<ffffffff816c0715>] system_call_fastpath+0x1c/0x21
[<ffffffff816c0661>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 8e 1b a3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 b2 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d 1b a3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1179445
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb2507e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb2477018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb2477088
Build id: 3.10.0-862.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffffb25c0497
Build id: 3.10.0-862.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffb25c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffb25ca624
Build id: 3.10.0-862.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffffb25cabe1
Build id: 3.10.0-862.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffffb25ccbf5
Build id: 3.10.0-862.el7.x86_64
0x295
9
vm_munmap
vmlinux 0xffffffffb25cce35
Build id: 3.10.0-862.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffffb25ce052
Build id: 3.10.0-862.el7.x86_64
0x22
11
tracesys
vmlinux 0xffffffffb2b1fa51
Build id: 3.10.0-862.el7.x86_64
0x9d
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [1_aux:101835]
Modules linked in: tcp_lp veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) nls_utf8 isofs rfcomm fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmhgfs(OE) vmw_vsock_vmci_transport vsock bnep sunrpc snd_seq_midi snd_seq_midi_event crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi aesni_intel snd_ac97_codec ac97_bus lrw gf128mul glue_helper ppdev btusb ablk_helper cryptd snd_seq btrtl btbcm btintel snd_seq_device bluetooth snd_pcm joydev snd_timer pcspkr sg snd rfkill nfit soundcore i2c_piix4 vmw_vmci shpchp libnvdimm parport_pc
parport ip_tables xfs libcrc32c sr_mod sd_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel mptspi libata e1000 scsi_transport_spi mptscsih serio_raw mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 101835 Comm: 1_aux Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff924c1da2af70 ti: ffff924c777b4000 task.ti: ffff924c777b4000
RIP: 0010:[<ffffffffb2507e3a>]  [<ffffffffb2507e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff924c777b7cb8  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 00000000000000fc RCX: ffff924d7959cf28
RDX: 0000000000000006 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff924c777b7cf0 R08: ffff924bbad2c000 R09: ffffffffb274a3d9
R10: ffff924d7951bb00 R11: ffffee46cc66a400 R12: 0000000000000296
R13: ffff924c777b7c68 R14: 0000000000000001 R15: ffffee46c828fd00
FS:  00007f23f5dc66c0(0000) GS:ffff924d79500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005587f716d618 CR3: 00000002e6e8a000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb2477018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb2477088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb25c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffb25c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffb25ca624>] unmap_region+0xf4/0x140
[<ffffffffb25cabe1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb25ccbf5>] do_munmap+0x295/0x470
[<ffffffffb25cce35>] vm_munmap+0x65/0xb0
[<ffffffffb25ce052>] SyS_munmap+0x22/0x30
[<ffffffffb2b1fa51>] tracesys+0x9d/0xc3
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 13 b3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1154591
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff811b5a01
Build id: 3.10.0-693.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
0x292
9
vm_munmap
vmlinux 0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#28 stuck for 21s! [cap_process:4748]
Modules linked in: tcp_lp fuse udp_diag tcp_diag inet_diag xt_comment xt_multiport xt_set ip_set_hash_ip ip_set nf_conntrack_netlink nfnetlink xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack sfesp(OE) ipt_REJECT nf_reject_ipv4 tun bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr joydev sg nfit i2c_piix4 libnvdimm shpchp parport_pc vmw_vmci parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod vmwgfx crc_t10dif crct10dif_generic
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm e1000e ahci libahci crct10dif_pclmul ata_piix crct10dif_common crc32c_intel serio_raw ptp libata pps_core i2c_core vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod
CPU: 28 PID: 4748 Comm: cap_process Tainted: G        W  OEL ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/09/2019
task: ffff8807fc22cf10 ti: ffff8800b2470000 task.ti: ffff8800b2470000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8800b2473cf0  EFLAGS: 00000202
RAX: 0000000000000019 RBX: 000800008106e4a0 RCX: ffff88083f85b3c8
RDX: 0000000000000019 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff8800b2473d28 R08: ffff8804ba7a0800 R09: ffffffff813227d9
R10: ffff88083f919c80 R11: ffffea00002b3e00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8800b2473ca0 R15: 00007f236dd63000
FS:  00007f2314b5e700(0000) GS:ffff88083f900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f236dd63000 CR3: 00000000349db000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001fc22cf10 ffff88042b3fbb88 ffff88042b3fb840 00007f236dd63000
00007f236dd64000 ffff88042b3fbb88 ffff8803d768d950 ffff8800b2473d78
ffffffff8106e668 ffff88042b3fb840 00007f236dd63000 00007f236dd64000
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff811b5a01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #811398
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb6707e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb6677018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb6677088
Build id: 3.10.0-862.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffffb67c0497
Build id: 3.10.0-862.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffb67c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffb67ca624
Build id: 3.10.0-862.el7.x86_64
0xf4
7
? xfs_iunlock
xfs 0xffffffffc05c6bec
Build id: 3.10.0-862.el7.x86_64
0xec
8
? __vma_rb_erase
vmlinux 0xffffffffb67cabe1
Build id: 3.10.0-862.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffb67ccbf5
Build id: 3.10.0-862.el7.x86_64
0x295
10
vm_munmap
vmlinux 0xffffffffb67cce35
Build id: 3.10.0-862.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffb67ce052
Build id: 3.10.0-862.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [java:974735]
Modules linked in: ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec iosf_mbi ac97_bus crc32_pclmul ghash_clmulni_intel snd_seq aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm snd_timer snd ppdev soundcore vmw_balloon vmw_vmci i2c_piix4 joydev pcspkr sg shpchp nfit libnvdimm parport_pc parport binfmt_misc xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix crct10dif_pclmul scsi_transport_spi crct10dif_common drm libata crc32c_intel e1000 mptscsih mptbase serio_raw i2c_core [last unloaded: ip_tables]
CPU: 14 PID: 974735 Comm: java Tainted: G        W    L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8be87ef83f40 ti: ffff8be61d710000 task.ti: ffff8be61d710000
RIP: 0010:[<ffffffffb6707e3a>]  [<ffffffffb6707e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8be61d713cb8  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00000000000000fc RCX: ffff8beebd6dd0b8
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8be61d713cf0 R08: ffff8be02966e800 R09: ffffffffb694a3d9
R10: ffff8beebd99bb00 R11: fffff89f8e52b200 R12: 0000000000000296
R13: 0000000000000018 R14: ffff8be61d713c28 R15: 0000000000000296
FS:  00002b99c381b700(0000) GS:ffff8beebd980000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b98bffb0000 CR3: 0000000c512c0000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffb6677018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb6677088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb67c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffb67c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffb67ca624>] unmap_region+0xf4/0x140
[<ffffffffc05c6bec>] ? xfs_iunlock+0xec/0x130 [xfs]
[<ffffffffb67cabe1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb67ccbf5>] do_munmap+0x295/0x470
[<ffffffffb67cce35>] vm_munmap+0x65/0xb0
[<ffffffffb67ce052>] SyS_munmap+0x22/0x30
CPU: 13 PID: 974732 Comm: java Tainted: G        W    L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
Complete report #733921
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
0xf4
7
? copy_user_generic_unrolled
vmlinux 0xffffffff81330240
Build id: 3.10.0-693.el7.x86_64
0xa0
8
? __vma_rb_erase
vmlinux 0xffffffff811b5a01
Build id: 3.10.0-693.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
0x292
10
vm_munmap
vmlinux 0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 68s! [java:51709]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm sb_edac snd_timer edac_core snd nfit soundcore ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr vmw_balloon libnvdimm joydev sg shpchp i2c_piix4 parport_pc parport vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel mptspi ata_piix scsi_transport_spi libata mptscsih mptbase e1000 serio_raw i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 51709 Comm: java Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8804112ccf10 ti: ffff88023f744000 task.ti: ffff88023f744000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff88023f747cf0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800008106e4a0 RCX: ffff88042d69af70
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff88023f747d28 R08: ffff88017a395000 R09: ffffffff813227d9
R10: ffff88042d6d9c80 R11: ffffea000b28b000 R12: 0000000000000286
R13: 0000000000000286 R14: ffff88023f747ca0 R15: 00007fd8fbaa1000
FS:  00007fd8de07a700(0000) GS:ffff88042d6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd8fbaa1000 CR3: 000000038fa50000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001112ccf10 ffff88041e3d6748 ffff88041e3d6400 00007fd8fbaa1000
00007fd8fbaa2000 ffff88041e3d6748 ffff88018865be60 ffff88023f747d78
ffffffff8106e668 ffff88041e3d6400 00007fd8fbaa1000 00007fd8fbaa2000
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff81330240>] ? copy_user_generic_unrolled+0xa0/0xc0
[<ffffffff811b5a01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #604465
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810fd5fa
Build id: 3.10.0-693.21.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff81070b98
Build id: 3.10.0-693.21.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff81070c68
Build id: 3.10.0-693.21.1.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811b18c3
Build id: 3.10.0-693.21.1.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811b2be5
Build id: 3.10.0-693.21.1.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b9cb4
Build id: 3.10.0-693.21.1.el7.x86_64
0xf4
7
? handle_mm_fault
vmlinux 0xffffffff811b6781
Build id: 3.10.0-693.21.1.el7.x86_64
0x691
8
? __vma_rb_erase
vmlinux 0xffffffff811ba271
Build id: 3.10.0-693.21.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff811bc285
Build id: 3.10.0-693.21.1.el7.x86_64
0x295
10
vm_munmap
vmlinux 0xffffffff811bc4c5
Build id: 3.10.0-693.21.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff811bd5d2
Build id: 3.10.0-693.21.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff816c0715
Build id: 3.10.0-693.21.1.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [gnome-shell:1863]
Modules linked in: nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul snd_ens1371 glue_helper snd_rawmidi ablk_helper snd_ac97_codec cryptd ac97_bus snd_seq snd_seq_device snd_pcm ppdev vmw_balloon snd_timer snd joydev pcspkr
sg soundcore vmw_vmci nfit i2c_piix4 shpchp libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c vmwgfx drm_kms_helper sr_mod sd_mod cdrom crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ata_generic ttm pata_acpi drm mptspi scsi_transport_spi mptscsih ata_piix mptbase libata e1000 crct10dif_pclmul crct10dif_common crc32c_intel serio_raw i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 1863 Comm: gnome-shell Tainted: G             L ------------   3.10.0-693.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff8804288e3f40 ti: ffff88040073c000 task.ti: ffff88040073c000
RIP: 0010:[<ffffffff810fd5fa>]  [<ffffffff810fd5fa>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff88040073fcc0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000810709d0 RCX: ffff88042d61cd58
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff88040073fcf8 R08: ffff880179ca4800 R09: ffffffff81328989
R10: ffff88042d79ba00 R11: ffffea000c8cf800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff88040073fc70 R15: ffffffff812da063
FS:  00007fa180adeac0(0000) GS:ffff88042d780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa11ef35000 CR3: 000000041daa4000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff81070b98>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81070c68>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811b18c3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811b2be5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b9cb4>] unmap_region+0xf4/0x140
[<ffffffff811b6781>] ? handle_mm_fault+0x691/0xfa0
[<ffffffff811ba271>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811bc285>] do_munmap+0x295/0x470
[<ffffffff811bc4c5>] vm_munmap+0x65/0xb0
[<ffffffff811bd5d2>] SyS_munmap+0x22/0x30
[<ffffffff816c0715>] system_call_fastpath+0x1c/0x21
Code: 48 63 35 8e 1b a3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 b2 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d 1b a3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #534482
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbaf1173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbae7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbae7a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffffbafe4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffbafe5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffbafef014
Build id: 3.10.0-957.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffffbafef5d1
Build id: 3.10.0-957.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffffbaff1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffffbaff1845
Build id: 3.10.0-957.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffffbaff2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffffbb574ddb
Build id: 3.10.0-957.el7.x86_64
0x22
12
? system_call_after_swapgs
vmlinux 0xffffffffbb574d21
Build id: 3.10.0-957.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [mysqld:11996]
Modules linked in: udp_diag binfmt_misc resguard_linux(OE) secmodel_linux(OE) tcp_diag inet_diag ip6t_MASQUERADE nf_nat_masquerade_ipv6 ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_mark ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 nf_nat tun ipt_REJECT nf_reject_ipv4 xt_limit xt_conntrack iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod ppdev sg parport_pc virtio_balloon parport joydev i2c_piix4 pcspkr nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_scsi virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy
fuse
CPU: 8 PID: 11996 Comm: mysqld Kdump: loaded Tainted: G           OEL ------------   3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9de9f5656180 ti: ffff9decf77f0000 task.ti: ffff9decf77f0000
RIP: 0010:[<ffffffffbaf1173a>]  [<ffffffffbaf1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9decf77f3cb8  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00000000000000fc RCX: ffffd5acbd700c20
RDX: 0000000000000004 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9decf77f3cf0 R08: ffff9decd0f20400 R09: ffffffffbb1755c9
R10: ffff9deded01f160 R11: fffff7880a5c9c00 R12: 0000000000000296
R13: ffff9decf77f3c68 R14: 0000000000000001 R15: fffff7880508a3c0
FS:  00007fab4cb91700(0000) GS:ffff9deded000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa763ffe008 CR3: 000000040a6c4000 CR4: 00000000000006e0
Call Trace:
[<ffffffffbae7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbae7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffbafe4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffbafe5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffbafef014>] unmap_region+0xf4/0x140
[<ffffffffbafef5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffbaff1605>] do_munmap+0x2a5/0x480
[<ffffffffbaff1845>] vm_munmap+0x65/0xb0
[<ffffffffbaff2ad2>] SyS_munmap+0x22/0x30
[<ffffffffbb574ddb>] system_call_fastpath+0x22/0x27
[<ffffffffbb574d21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 b4 bb f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #509787
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffba907e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffba877018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffba877088
Build id: 3.10.0-862.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffffba9c0497
Build id: 3.10.0-862.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffba9c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffba9ca624
Build id: 3.10.0-862.el7.x86_64
0xf4
7
? do_sync_write
vmlinux 0xffffffffbaa1a1f0
Build id: 3.10.0-862.el7.x86_64
0x90
8
? __vma_rb_erase
vmlinux 0xffffffffba9cabe1
Build id: 3.10.0-862.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffba9ccbf5
Build id: 3.10.0-862.el7.x86_64
0x295
10
vm_munmap
vmlinux 0xffffffffba9cce35
Build id: 3.10.0-862.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffba9ce052
Build id: 3.10.0-862.el7.x86_64
0x22
12
tracesys
vmlinux 0xffffffffbaf1fa51
Build id: 3.10.0-862.el7.x86_64
0x9d
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [elasticsearch[n:18425]
Modules linked in: xt_statistic vxlan ip6_udp_tunnel udp_tunnel ipt_REJECT nf_reject_ipv4 nf_tables(T) ip_set ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_mangle ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_filter iptable_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables xt_comment xt_mark xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter bridge stp llc overlay(T) iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter ext4 mbcache jbd2 sunrpc sg 8139too virtio_balloon pcspkr joydev ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi ata_piix libata 8139cp virtio_pci virtio_ring mii serio_raw virtio dm_mirror
dm_region_hash dm_log dm_mod
CPU: 4 PID: 18425 Comm: elasticsearch[n Kdump: loaded Tainted: G               ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff88684b268000 ti: ffff886739ff8000 task.ti: ffff886739ff8000
RIP: 0010:[<ffffffffba907e3a>]  [<ffffffffba907e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff886739ffbcb8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0000000000019680 RCX: ffff886d6f21cd28
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff886739ffbcf0 R08: ffff885adf734c00 R09: ffffffffbab4a3d9
R10: ffff886d6f31bb00 R11: fffff3d98b88d200 R12: ffff886739ffbc30
R13: 0000000000000286 R14: 0000000000000010 R15: ffffffffba859653
FS:  00007ff0ce1fd700(0000) GS:ffff886d6f300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff0ccdd8000 CR3: 0000000db62a8000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffba877018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffba877088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffba9c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffba9c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffba9ca624>] unmap_region+0xf4/0x140
[<ffffffffbaa1a1f0>] ? do_sync_write+0x90/0xe0
[<ffffffffba9cabe1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffba9ccbf5>] do_munmap+0x295/0x470
[<ffffffffba9cce35>] vm_munmap+0x65/0xb0
[<ffffffffba9ce052>] SyS_munmap+0x22/0x30
[<ffffffffbaf1fa51>] tracesys+0x9d/0xc3
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 53 bb f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #496090
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9b11173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9b07a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9b07a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffff9b1e4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff9b1e5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff9b1ef014
Build id: 3.10.0-957.el7.x86_64
0xf4
7
? xfs_iunlock
xfs 0xffffffffc03db8bc
Build id: 3.10.0-957.el7.x86_64
0xec
8
? __vma_rb_erase
vmlinux 0xffffffff9b1ef5d1
Build id: 3.10.0-957.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff9b1f1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff9b1f1845
Build id: 3.10.0-957.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff9b1f2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff9b774ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [java:8478]
Modules linked in: vmw_vsock_vmci_transport vsock vfat fat crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg parport_pc vmw_vmci parport tpm_crb i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic e1000e crct10dif_pclmul crct10dif_common serio_raw crc32c_intel vmwgfx ptp pps_core drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci mptsas libahci drm ata_piix scsi_transport_sas nfit mptscsih libata libnvdimm mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 8478 Comm: java Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.17369862.B64.2012240522 12/24/2020
task: ffff9f623a59b0c0 ti: ffff9f6991300000 task.ti: ffff9f6991300000
RIP: 0010:[<ffffffff9b11173a>]  [<ffffffff9b11173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9f6991303cb8  EFLAGS: 00000202
RAX: 000000000000000e RBX: 00000000000000fc RCX: ffff9f699f5a04b8
RDX: 000000000000000e RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff9f6991303cf0 R08: ffff9f62f4bfb800 R09: ffffffff9b3755c9
R10: ffff9f699f29f160 R11: fffff1071e2c8400 R12: 0000000000000296
R13: ffff9f6991303c68 R14: 0000000000000001 R15: fffff1071a246c80
FS:  00007fdf151e4700(0000) GS:ffff9f699f280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdf117d5000 CR3: 0000000814b30000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffff9b07a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9b07a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff9b1e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff9b1e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff9b1ef014>] unmap_region+0xf4/0x140
[<ffffffffc03db8bc>] ? xfs_iunlock+0xec/0x130 [xfs]
[<ffffffff9b1ef5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff9b1f1605>] do_munmap+0x2a5/0x480
[<ffffffff9b1f1845>] vm_munmap+0x65/0xb0
[<ffffffff9b1f2ad2>] SyS_munmap+0x22/0x30
[<ffffffff9b774ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 d4 9b f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #493105
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810fd5fa
Build id: 3.10.0-693.21.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff81070b98
Build id: 3.10.0-693.21.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff81070c68
Build id: 3.10.0-693.21.1.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811b18c3
Build id: 3.10.0-693.21.1.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811b2be5
Build id: 3.10.0-693.21.1.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b9cb4
Build id: 3.10.0-693.21.1.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff811ba271
Build id: 3.10.0-693.21.1.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff811bc285
Build id: 3.10.0-693.21.1.el7.x86_64
0x295
9
vm_munmap
vmlinux 0xffffffff811bc4c5
Build id: 3.10.0-693.21.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff811bd5d2
Build id: 3.10.0-693.21.1.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff816c0715
Build id: 3.10.0-693.21.1.el7.x86_64
0x1c
12
? system_call_after_swapgs
vmlinux 0xffffffff816c0661
Build id: 3.10.0-693.21.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [libvirtd:1438]
Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter scsi_transport_iscsi softdog iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel dm_service_time aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg i2c_piix4 virtio_rng pcspkr joydev virtio_balloon nfsd auth_rpcgss nfs_acl dm_multipath lockd grace sunrpc 8021q garp mrp bridge stp llc bonding ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_net virtio_scsi virtio_console qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel i2c_core virtio_pci serio_raw virtio_ring floppy virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 1438 Comm: libvirtd Not tainted 3.10.0-693.21.1.el7.x86_64 #1
Hardware name: Red Hat RHEL/RHEL-AV, BIOS 1.14.0-1.module+el8.4.0+8855+a9e237a9 04/01/2014
task: ffff8800364c0fd0 ti: ffff880074900000 task.ti: ffff880074900000
RIP: 0010:[<ffffffff810fd5fa>]  [<ffffffff810fd5fa>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff880074903cc0  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00080000810709d0 RCX: ffff88007d2dcab8
RDX: 0000000000000003 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff880074903cf8 R08: ffff880077d9b400 R09: ffffffff81328989
R10: ffff88007d29ba00 R11: ffffea0001dba400 R12: 0000000000000286
R13: 0000000000000286 R14: ffff880074903c70 R15: 00007fe716fc5000
FS:  00007fe703f8f700(0000) GS:ffff88007d280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe716fc5000 CR3: 0000000077bd4000 CR4: 00000000000606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff81070b98>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81070c68>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811b18c3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811b2be5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b9cb4>] unmap_region+0xf4/0x140
[<ffffffff811ba271>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811bc285>] do_munmap+0x295/0x470
[<ffffffff811bc4c5>] vm_munmap+0x65/0xb0
[<ffffffff811bd5d2>] SyS_munmap+0x22/0x30
[<ffffffff816c0715>] system_call_fastpath+0x1c/0x21
[<ffffffff816c0661>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 8e 1b a3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 b2 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d 1b a3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #468702
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.2.2.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.2.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.2.2.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.2.2.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.2.2.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b5444
Build id: 3.10.0-693.2.2.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff811b5a01
Build id: 3.10.0-693.2.2.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff811b7a02
Build id: 3.10.0-693.2.2.el7.x86_64
0x292
9
vm_munmap
vmlinux 0xffffffff811b7c45
Build id: 3.10.0-693.2.2.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff811b8d52
Build id: 3.10.0-693.2.2.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff816b5009
Build id: 3.10.0-693.2.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [nbrmms:2277]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13707(E) ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ppdev vmw_balloon pcspkr joydev nfit sg libnvdimm vmw_vmci i2c_piix4 parport_pc shpchp parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix ahci libahci mptspi scsi_transport_spi i2c_core libata mptscsih serio_raw mptbase vmxnet3 dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 2277 Comm: nbrmms Tainted: P            EL ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffff880fe6e40000 ti: ffff880fb7fa0000 task.ti: ffff880fb7fa0000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff880fb7fa3cf0  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffffffff8106e4a0 RCX: ffff88103fd1af20
RDX: 0000000000000004 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff880fb7fa3d28 R08: ffff880169407400 R09: ffffffff813229e9
R10: ffff88103fc99c80 R11: ffffea003e71c200 R12: ffff880fb7fa3cc0
R13: ffffffff8132293f R14: ffff880fb7fa3e08 R15: 00007f7ada64b000
FS:  00007f7abdcda700(0000) GS:ffff88103fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7ada64b000 CR3: 0000000fcfb4b000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001e6e40000 ffff880ffaa7bb88 ffff880ffaa7b840 0000000000000000
ffffffffffffffff ffff880ffaa7bb88 ffff880fe97df518 ffff880fb7fa3d78
ffffffff8106e668 ffff880ffaa7b840 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff811b5a01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b5009>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #448717
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb991173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb987a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb987a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffffb99e4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffb99e5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffb99ef014
Build id: 3.10.0-957.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffffb99ef5d1
Build id: 3.10.0-957.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffffb99f1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffffb99f1845
Build id: 3.10.0-957.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffffb99f2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffffb9f74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [auth_server:19349]
Modules linked in: nvidia_uvm(OE) xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc nvidia_drm(POE) nvidia_modeset(POE) vfat fat ext4 skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel mbcache jbd2 kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul
glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support nvidia(POE) pcspkr joydev sg mei_me mei lpc_ich i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler acpi_cpufreq wmi acpi_power_meter acpi_pad sch_fq_codel ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit drm_kms_helper crct10dif_pclmul syscopyarea crct10dif_common sysfillrect crc32c_intel i40e sysimgblt fb_sys_fops ttm drm ptp pps_core ahci libahci libata drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 19 PID: 19349 Comm: auth_server Kdump: loaded Tainted: P           OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Supermicro X11DPG-OT/X11DPG-OT-CPU, BIOS 3.1 04/30/2019
task: ffff9e98b1232080 ti: ffff9e98a66b0000 task.ti: ffff9e98a66b0000
RIP: 0010:[<ffffffffb991173a>]  [<ffffffffb991173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9e98a66b3cb8  EFLAGS: 00000202
RAX: 0000000000000005 RBX: 000000e000000013 RCX: ffff9e98de760908
RDX: 0000000000000005 RSI: 00000000000000e0 RDI: 0000000000000000
RBP: ffff9e98a66b3cf0 R08: ffff9e91f31b3c00 R09: ffffffffb9b755c9
R10: ffff9e98deadf160 R11: fffff3ff2119b000 R12: 000000fcb9b7551f
R13: 0000000000000296 R14: 0000000000000296 R15: ffff9e98a66b3c68
FS:  00007fa72c625700(0000) GS:ffff9e98deac0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa72e342000 CR3: 0000001025b02000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffb987a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb987a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb99e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffb99e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffb99ef014>] unmap_region+0xf4/0x140
[<ffffffffb99ef5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb99f1605>] do_munmap+0x2a5/0x480
[<ffffffffb99f1845>] vm_munmap+0x65/0xb0
[<ffffffffb99f2ad2>] SyS_munmap+0x22/0x30
[<ffffffffb9f74ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 54 ba f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #436624
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff99307e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff99277018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff99277088
Build id: 3.10.0-862.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffff993c0497
Build id: 3.10.0-862.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff993c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff993ca624
Build id: 3.10.0-862.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff993cabff
Build id: 3.10.0-862.el7.x86_64
0x13f
8
do_munmap
vmlinux 0xffffffff993ccbf5
Build id: 3.10.0-862.el7.x86_64
0x295
9
vm_munmap
vmlinux 0xffffffff993cce35
Build id: 3.10.0-862.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff993ce052
Build id: 3.10.0-862.el7.x86_64
0x22
11
tracesys
vmlinux 0xffffffff9991fa51
Build id: 3.10.0-862.el7.x86_64
0x9d
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [java:13972]
Modules linked in: tcp_lp nls_utf8 isofs fuse veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec aesni_intel lrw ppdev ac97_bus gf128mul glue_helper snd_seq ablk_helper cryptd snd_seq_device vmw_balloon snd_pcm snd_timer snd pcspkr joydev nfit soundcore sg libnvdimm parport_pc vmw_vmci parport i2c_piix4 shpchp ip_tables xfs libcrc32c sr_mod cdrom ata_generic
pata_acpi sd_mod crc_t10dif crct10dif_generic ahci crct10dif_pclmul libahci crct10dif_common ata_piix vmwgfx drm_kms_helper mptspi scsi_transport_spi crc32c_intel syscopyarea mptscsih sysfillrect sysimgblt fb_sys_fops ttm drm e1000 libata serio_raw mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 13972 Comm: java Kdump: loaded Tainted: G               ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff8dcd7ca63f40 ti: ffff8dcd74fb4000 task.ti: ffff8dcd74fb4000
RIP: 0010:[<ffffffff99307e3a>]  [<ffffffff99307e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8dcd74fb7cb8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8dcdf361cdb8
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8dcd74fb7cf0 R08: ffff8dccbb4ab800 R09: ffffffff9954a3d9
R10: ffff8dcdf369bb00 R11: ffffc2d6c05ec400 R12: 0000000000000296
R13: ffff8dcd74fb7c68 R14: 0000000000000000 R15: 0000000000000200
FS:  00007fb56ae11b10(0000) GS:ffff8dcdf3680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb56adfd618 CR3: 0000000291732000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff99277018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff99277088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff993c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff993c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff993ca624>] unmap_region+0xf4/0x140
[<ffffffff993cabff>] ? __vma_rb_erase+0x13f/0x220
[<ffffffff993ccbf5>] do_munmap+0x295/0x470
[<ffffffff993cce35>] vm_munmap+0x65/0xb0
[<ffffffff993ce052>] SyS_munmap+0x22/0x30
[<ffffffff9991fa51>] tracesys+0x9d/0xc3
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 f3 99 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #406830
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb591173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb587a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb587a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffffb59e4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffb59e5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffb59ef014
Build id: 3.10.0-957.el7.x86_64
0xf4
7
? __schedule
vmlinux 0xffffffffb5f678e8
Build id: 3.10.0-957.el7.x86_64
0x5d8
8
? __vma_rb_erase
vmlinux 0xffffffffb59ef5d1
Build id: 3.10.0-957.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffb59f1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffb59f1845
Build id: 3.10.0-957.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffb59f2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
12
tracesys
vmlinux 0xffffffffb5f7505b
Build id: 3.10.0-957.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [elasticsearch[9:8854]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr btusb snd_ens1371 btrtl btbcm snd_rawmidi btintel bluetooth snd_ac97_codec ac97_bus snd_seq snd_seq_device rfkill snd_pcm snd_timer snd soundcore sg nfit parport_pc parport libnvdimm vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw e1000 ata_generic
pata_acpi floppy vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 8854 Comm: elasticsearch[9 Kdump: loaded Tainted: G               ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff9cb95ea62080 ti: ffff9cb95f15c000 task.ti: ffff9cb95f15c000
RIP: 0010:[<ffffffffb591173a>]  [<ffffffffb591173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9cb95f15fcb8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000000000001b700 RCX: ffff9cb97c620490
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff9cb95f15fcf0 R08: ffff9cb97b653800 R09: ffffffffb5b755c9
R10: ffff9cb97c6df160 R11: fffff6980083b600 R12: ffff9cb95f15fc68
R13: ffffffffc0312227 R14: ffffffffb5a1be42 R15: ffff9cb93678fd00
FS:  00007f82033ff700(0000) GS:ffff9cb97c6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f82a410b000 CR3: 0000000052ddc000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb587a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb587a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb59e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffb59e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffb59ef014>] unmap_region+0xf4/0x140
[<ffffffffb5f678e8>] ? __schedule+0x5d8/0x890
[<ffffffffb59ef5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb59f1605>] do_munmap+0x2a5/0x480
[<ffffffffb59f1845>] vm_munmap+0x65/0xb0
[<ffffffffb59f2ad2>] SyS_munmap+0x22/0x30
[<ffffffffb5f7505b>] tracesys+0xa3/0xc9
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 54 b6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #399824
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa3f0816a
Build id: 3.10.0-862.3.3.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa3e772d8
Build id: 3.10.0-862.3.3.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa3e77348
Build id: 3.10.0-862.3.3.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffffa3fc07d7
Build id: 3.10.0-862.3.3.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffa3fc1fb5
Build id: 3.10.0-862.3.3.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffa3fca964
Build id: 3.10.0-862.3.3.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffffa3fcaf21
Build id: 3.10.0-862.3.3.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffffa3fccf35
Build id: 3.10.0-862.3.3.el7.x86_64
0x295
9
vm_munmap
vmlinux 0xffffffffa3fcd175
Build id: 3.10.0-862.3.3.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffffa3fce392
Build id: 3.10.0-862.3.3.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffffa452082f
Build id: 3.10.0-862.3.3.el7.x86_64
0x1c
12
? system_call_after_swapgs
vmlinux 0xffffffffa452077b
Build id: 3.10.0-862.3.3.el7.x86_64
0xc8
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [spades-hammer:13600]
Modules linked in: fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vboxsf(OE) sunrpc vboxguest(OE) snd_intel8x0 vmwgfx snd_ac97_codec ttm ac97_bus ppdev drm_kms_helper snd_seq joydev syscopyarea snd_seq_device sysfillrect snd_pcm sysimgblt iosf_mbi fb_sys_fops snd_timer snd drm sg soundcore i2c_piix4 pcspkr parport_pc video parport i2c_core ip_tables
xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod crct10dif_common cdrom ata_generic pata_acpi ata_piix libata crc32c_intel serio_raw e1000 dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 13600 Comm: spades-hammer Kdump: loaded Tainted: G           OEL ------------   3.10.0-862.3.3.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff9576cc5ddee0 ti: ffff957757f90000 task.ti: ffff957757f90000
RIP: 0010:[<ffffffffa3f0816a>]  [<ffffffffa3f0816a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff957757f93cb8  EFLAGS: 00010202
RAX: 0000000000000001 RBX: 0000000000019680 RCX: ffff95775fc9cc20
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff957757f93cf0 R08: ffff95775949d400 R09: ffffffffa414a819
R10: ffff95775fd9bb00 R11: ffffd74602954200 R12: ffffffffa414a76f
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000200
FS:  00007f6312bfe700(0000) GS:ffff95775fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f631140f000 CR3: 00000000ac05a000 CR4: 00000000000206e0
Call Trace:
[<ffffffffa3e772d8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa3e77348>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa3fc07d7>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffa3fc1fb5>] tlb_finish_mmu+0x55/0x60
[<ffffffffa3fca964>] unmap_region+0xf4/0x140
[<ffffffffa3fcaf21>] ? __vma_rb_erase+0x121/0x220
[<ffffffffa3fccf35>] do_munmap+0x295/0x470
[<ffffffffa3fcd175>] vm_munmap+0x65/0xb0
[<ffffffffa3fce392>] SyS_munmap+0x22/0x30
[<ffffffffa452082f>] system_call_fastpath+0x1c/0x21
[<ffffffffa452077b>] ? system_call_after_swapgs+0xc8/0x160
Code: 48 63 35 9e b2 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 52 b3 a4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d b2 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #393264
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810fb892
Build id: 3.10.0-693.17.1.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106f508
Build id: 3.10.0-693.17.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106f5d8
Build id: 3.10.0-693.17.1.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811af0f3
Build id: 3.10.0-693.17.1.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811b0415
Build id: 3.10.0-693.17.1.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b74e4
Build id: 3.10.0-693.17.1.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff811b7aa1
Build id: 3.10.0-693.17.1.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff811b9aa2
Build id: 3.10.0-693.17.1.el7.x86_64
0x292
9
vm_munmap
vmlinux 0xffffffff811b9ce5
Build id: 3.10.0-693.17.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff811badf2
Build id: 3.10.0-693.17.1.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff816b89fd
Build id: 3.10.0-693.17.1.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [ds_am:2449]
Modules linked in: vfat fat isofs xfs libcrc32c binfmt_misc bmhook(OE) tmhook(OE) dsa_filter(POE) dsa_filter_hook(OE) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod btrfs sb_edac edac_core iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd raid6_pq xor joydev pcspkr sg nfit parport_pc parport libnvdimm vmw_vmci shpchp i2c_piix4 ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ahci ata_piix libahci libata serio_raw vmxnet3 i2c_core vmw_pvscsi floppy
CPU: 8 PID: 2449 Comm: ds_am Tainted: P           OEL ------------   3.10.0-693.17.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8805a4e1dee0 ti: ffff88058b65c000 task.ti: ffff88058b65c000
RIP: 0010:[<ffffffff810fb892>]  [<ffffffff810fb892>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff88058b65fcc0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800008106f340 RCX: ffff8805a769cd80
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff88058b65fcf8 R08: ffff880176d3d800 R09: ffffffff81324cb9
R10: ffff8805a781b9c0 R11: ffffea0000430800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff88058b65fc70 R15: 00007fbca92e5000
FS:  00007fbc8a051700(0000) GS:ffff8805a7800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbca92e5000 CR3: 0000000581a98000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8106f508>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106f5d8>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811af0f3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811b0415>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b74e4>] unmap_region+0xf4/0x140
[<ffffffff811b7aa1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b9aa2>] do_munmap+0x292/0x470
[<ffffffff811b9ce5>] vm_munmap+0x65/0xb0
[<ffffffff811badf2>] SyS_munmap+0x22/0x30
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 76 b6 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 8e b1 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 45 b6 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #385194
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa571173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa567a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa567a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffffa57e4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffa57e5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffa57ef014
Build id: 3.10.0-957.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffffa57ef5d1
Build id: 3.10.0-957.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffffa57f1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffffa57f1845
Build id: 3.10.0-957.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffffa57f2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
11
tracesys
vmlinux 0xffffffffa5d7505b
Build id: 3.10.0-957.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [com.alibaba.nac:20806]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype xt_CHECKSUM br_netfilter iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_seq_midi snd_seq_midi_event joydev pcspkr snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd sg soundcore parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul
crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi vmwgfx e1000 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 20806 Comm: com.alibaba.nac Kdump: loaded Tainted: G               ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff958d5a840000 ti: ffff958ce12e0000 task.ti: ffff958ce12e0000
RIP: 0010:[<ffffffffa571173a>]  [<ffffffffa571173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff958ce12e3cb8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff958d796204b8
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff958ce12e3cf0 R08: ffff958d7762b800 R09: ffffffffa59755c9
R10: ffff958d7969f160 R11: ffffec68405ffc00 R12: 0000000000000296
R13: ffff958ce12e3c68 R14: 0000000000000001 R15: ffffec6844785700
FS:  00007f1cc54c4700(0000) GS:ffff958d79680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1d5da40000 CR3: 000000011db2c000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa567a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa567a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa57e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffa57e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffa57ef014>] unmap_region+0xf4/0x140
[<ffffffffa57ef5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffa57f1605>] do_munmap+0x2a5/0x480
[<ffffffffa57f1845>] vm_munmap+0x65/0xb0
[<ffffffffa57f2ad2>] SyS_munmap+0x22/0x30
[<ffffffffa5d7505b>] tracesys+0xa3/0xc9
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 34 a6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #344835
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810fd5fa
Build id: 3.10.0-693.21.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff81070b98
Build id: 3.10.0-693.21.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff81070c68
Build id: 3.10.0-693.21.1.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811b18c3
Build id: 3.10.0-693.21.1.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811b2be5
Build id: 3.10.0-693.21.1.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b9cb4
Build id: 3.10.0-693.21.1.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff811ba271
Build id: 3.10.0-693.21.1.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff811bc285
Build id: 3.10.0-693.21.1.el7.x86_64
0x295
9
vm_munmap
vmlinux 0xffffffff811bc4c5
Build id: 3.10.0-693.21.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff811bd5d2
Build id: 3.10.0-693.21.1.el7.x86_64
0x22
11
tracesys
vmlinux 0xffffffff816c0991
Build id: 3.10.0-693.21.1.el7.x86_64
0x9d
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 106s! [java:5356]
Modules linked in: xt_nat veth fuse ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc vboxsf(OE) overlay(T) ext4 mbcache jbd2 joydev ppdev parport_pc vboxguest(OE) sg parport video pcspkr i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm serio_raw libata e1000 crc32c_intel i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 5356 Comm: java Tainted: G           OE  ------------ T 3.10.0-693.21.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff88049f220000 ti: ffff8804622ac000 task.ti: ffff8804622ac000
RIP: 0010:[<ffffffff810fd5fa>]  [<ffffffff810fd5fa>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8804622afcc0  EFLAGS: 00010202
RAX: 0000000000000001 RBX: 0000000000000010 RCX: ffff88051fc5be60
RDX: 0000000000000001 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff8804622afcf8 R08: ffff88017fd42c00 R09: ffffffff81328989
R10: ffff88051fc1ba00 R11: ffffea001379b100 R12: ffffffffffffff02
R13: 0000000000000286 R14: 00000000000000fc R15: 00000000000008fc
FS:  00007fd75388d700(0000) GS:ffff88051fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd73c000000 CR3: 00000004ce3c8000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff81070b98>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81070c68>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811b18c3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811b2be5>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b9cb4>] unmap_region+0xf4/0x140
[<ffffffff811ba271>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811bc285>] do_munmap+0x295/0x470
[<ffffffff811bc4c5>] vm_munmap+0x65/0xb0
[<ffffffff811bd5d2>] SyS_munmap+0x22/0x30
[<ffffffff816c0991>] tracesys+0x9d/0xc3
Code: 48 63 35 8e 1b a3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 b2 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d 1b a3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #294077
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a6
Build id: 3.10.0-693.el7.x86_64
0x206
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff811b5a01
Build id: 3.10.0-693.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
0x292
9
vm_munmap
vmlinux 0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [icsfilesec:55562]
Modules linked in: vmsec_nfq(OE) nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nfnetlink_queue vmsecmod(OE) tun ip_set nfnetlink bridge stp llc nf_conntrack sg i2c_piix4 ppdev pcspkr virtio_balloon joydev parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_scsi virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata virtio_pci floppy virtio_ring i2c_core serio_raw virtio dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vmsec_nfq]
CPU: 5 PID: 55562 Comm: icsfilesec Tainted: G           OEL ------------   3.10.0-693.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff88044ddeeeb0 ti: ffff8804cdaf8000 task.ti: ffff8804cdaf8000
RIP: 0010:[<ffffffff810f99a6>]  [<ffffffff810f99a6>] smp_call_function_many+0x206/0x260
RSP: 0018:ffff8804cdafbcf0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008106e4a0 RCX: ffff88081da1ae08
RDX: 0000000000000000 RSI: 0000000000000070 RDI: 0000000000000000
RBP: ffff8804cdafbd28 R08: ffff880174b4d000 R09: ffffffff813227d9
R10: ffff88081db59c80 R11: ffffea0015108200 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8804cdafbca0 R15: 00007fead309e000
FS:  00007feab9ffb700(0000) GS:ffff88081db40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fead309e000 CR3: 00000000030c4000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
000000014ddeeeb0 ffff880819f5e108 ffff880819f5ddc0 0000000000000000
ffffffffffffffff ffff880819f5e108 ffff8807e1380e58 ffff8804cdafbd78
ffffffff8106e668 ffff880819f5ddc0 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff811b5a01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #269238
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb250b09a
Build id: 3.10.0-862.11.6.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb2479e58
Build id: 3.10.0-862.11.6.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb2479ec8
Build id: 3.10.0-862.11.6.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffffb25c33c7
Build id: 3.10.0-862.11.6.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffb25c4cc5
Build id: 3.10.0-862.11.6.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffb25cdab4
Build id: 3.10.0-862.11.6.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffffb25ce071
Build id: 3.10.0-862.11.6.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffffb25d0085
Build id: 3.10.0-862.11.6.el7.x86_64
0x295
9
vm_munmap
vmlinux 0xffffffffb25d02c5
Build id: 3.10.0-862.11.6.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffffb25d14e2
Build id: 3.10.0-862.11.6.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffffb2b2579b
Build id: 3.10.0-862.11.6.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [systemd:1]
Modules linked in: binfmt_misc veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter gsch(OE) redirfs(OE) event_collector_1_9_8557(POE) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon
ppdev sg pcspkr joydev parport_pc parport vmw_vmci shpchp i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect crc_t10dif sysimgblt crct10dif_generic fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw i2c_core vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 1 Comm: systemd Kdump: loaded Tainted: P        W  OEL ------------   3.10.0-862.11.6.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9537ecc38000 ti: ffff9537ecc80000 task.ti: ffff9537ecc80000
RIP: 0010:[<ffffffffb250b09a>]  [<ffffffffb250b09a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9537ecc83cb8  EFLAGS: 00000202
RAX: 0000000000000005 RBX: 00000000000000fc RCX: ffff954f8d01d028
RDX: 0000000000000005 RSI: 000000000000000a RDI: 0000000000000000
RBP: ffff9537ecc83cf0 R08: ffff9543ad778c00 R09: ffffffffb274edf9
R10: ffff954f8d11bb40 R11: ffffe2c2b1f2b000 R12: 0000000000000296
R13: ffff9537ecc83c68 R14: 0000000000000001 R15: ffffe2c2b93a1e80
FS:  00007fd91b808940(0000) GS:ffff954f8d100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd91b819000 CR3: 0000000c8d6ce000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb2479e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb2479ec8>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb25c33c7>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffb25c4cc5>] tlb_finish_mmu+0x55/0x60
[<ffffffffb25cdab4>] unmap_region+0xf4/0x140
[<ffffffffb25ce071>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb25d0085>] do_munmap+0x295/0x470
[<ffffffffb25d02c5>] vm_munmap+0x65/0xb0
[<ffffffffb25d14e2>] SyS_munmap+0x22/0x30
[<ffffffffb2b2579b>] system_call_fastpath+0x22/0x27
Code: 48 63 35 ee 84 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 54 13 b3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 84 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #254554
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a6
Build id: 3.10.0-693.el7.x86_64
0x206
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff811b5a01
Build id: 3.10.0-693.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
0x292
9
vm_munmap
vmlinux 0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [java:21225]
Modules linked in: nfsv3 nfs fscache vmw_vsock_vmci_transport vsock coretemp ppdev vmw_balloon pcspkr joydev sg shpchp parport_pc vmw_vmci parport i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper crc32c_intel serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi drm scsi_transport_spi vmxnet3 mptscsih ata_piix mptbase libata i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 21225 Comm: java Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8804223c0000 ti: ffff88031cce0000 task.ti: ffff88031cce0000
RIP: 0010:[<ffffffff810f99a6>]  [<ffffffff810f99a6>] smp_call_function_many+0x206/0x260
RSP: 0018:ffff88031cce3cf0  EFLAGS: 00000202
RAX: 0000000000000003 RBX: ffffffff8106e4a0 RCX: ffff88043fd9a7c8
RDX: 0000000000000003 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff88031cce3d28 R08: ffff88017a25d000 R09: ffffffff813227d9
R10: ffff88043fc99c80 R11: ffffea0010adfe00 R12: ffff88031cce3cc0
R13: ffffffff8132272f R14: ffff88031cce3c70 R15: ffff88043fc96cc0
FS:  00007fdb60ad9700(0000) GS:ffff88043fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fdb60ba1010 CR3: 000000008d716000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001223c0000 ffff88034ad673c8 ffff88034ad67080 00007fdb60bfc000
00007fdb60c02000 ffff88034ad673c8 ffff8803c40bbd80 ffff88031cce3d78
ffffffff8106e668 ffff88034ad67080 00007fdb60bfc000 00007fdb60c02000
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff811b5a01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
Complete report #222195
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbe71185a
Build id: 3.10.0-957.5.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbe67a5c8
Build id: 3.10.0-957.5.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbe67a638
Build id: 3.10.0-957.5.1.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffffbe7e4427
Build id: 3.10.0-957.5.1.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffbe7e5d25
Build id: 3.10.0-957.5.1.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffbe7ef3c4
Build id: 3.10.0-957.5.1.el7.x86_64
0xf4
7
? xfs_iunlock
xfs 0xffffffffc03148bc
Build id: 3.10.0-957.5.1.el7.x86_64
0xec
8
? __vma_rb_erase
vmlinux 0xffffffffbe7ef981
Build id: 3.10.0-957.5.1.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffbe7f19b5
Build id: 3.10.0-957.5.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffbe7f1bf5
Build id: 3.10.0-957.5.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffbe7f2e82
Build id: 3.10.0-957.5.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffffbed74ddb
Build id: 3.10.0-957.5.1.el7.x86_64
0x22
13
? system_call_after_swapgs
vmlinux 0xffffffffbed74d21
Build id: 3.10.0-957.5.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 27s! [java:24353]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod ppdev vmw_balloon coretemp pcspkr joydev nfit sg libnvdimm parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common sr_mod cdrom crc32c_intel serio_raw ata_generic pata_acpi floppy vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm e1000e ata_piix drm mptspi scsi_transport_spi ptp pps_core mptscsih libata mptbase drm_panel_orientation_quirks
CPU: 1 PID: 24353 Comm: java Kdump: loaded Tainted: G        W      ------------   3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9631b0c8c100 ti: ffff963159ae0000 task.ti: ffff963159ae0000
RIP: 0010:[<ffffffffbe71185a>]  [<ffffffffbe71185a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff963159ae3cb8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000000000001b700 RCX: ffff9631bfc20490
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff963159ae3cf0 R08: ffff9630fcf2cc00 R09: ffffffffbe975ba9
R10: ffff9631bfc9f160 R11: ffffee4fc8a19700 R12: ffffffffbe975aff
R13: ffff9630ffc01500 R14: 0000000000000001 R15: ffffffffbe81d47a
FS:  00007f3de39f9700(0000) GS:ffff9631bfc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3e547f0000 CR3: 000000009ec4a000 CR4: 00000000000007e0
Call Trace:
[<ffffffffbe67a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbe67a638>] flush_tlb_mm_range+0x68/0x140
[<ffffffffbe7e4427>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffbe7e5d25>] tlb_finish_mmu+0x55/0x60
[<ffffffffbe7ef3c4>] unmap_region+0xf4/0x140
[<ffffffffc03148bc>] ? xfs_iunlock+0xec/0x130 [xfs]
[<ffffffffbe7ef981>] ? __vma_rb_erase+0x121/0x220
[<ffffffffbe7f19b5>] do_munmap+0x2a5/0x480
[<ffffffffbe7f1bf5>] vm_munmap+0x65/0xb0
[<ffffffffbe7f2e82>] SyS_munmap+0x22/0x30
[<ffffffffbed74ddb>] system_call_fastpath+0x22/0x27
[<ffffffffbed74d21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 2e 82 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 34 bf f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 fd 81 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #213219
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a6
Build id: 3.10.0-693.el7.x86_64
0x206
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff811b5a01
Build id: 3.10.0-693.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
0x292
9
vm_munmap
vmlinux 0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [sqlservr:2583]
Modules linked in: tcp_lp rfcomm tun lp ip_set nfnetlink bridge stp llc fuse bnep snd_seq_midi snd_seq_midi_event coretemp iosf_mbi kvm_intel kvm ppdev irqbypass crc32_pclmul snd_ens1371 snd_rawmidi btusb snd_ac97_codec btrtl btbcm btintel bluetooth ghash_clmulni_intel ac97_bus snd_seq aesni_intel lrw snd_seq_device gf128mul glue_helper snd_pcm ablk_helper vmw_balloon cryptd snd_timer rfkill joydev pcspkr snd sg soundcore parport_pc parport shpchp i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi scsi_transport_spi mptscsih mptbase crct10dif_pclmul crct10dif_common crc32c_intel e1000 serio_raw i2c_core floppy
dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 2 PID: 2583 Comm: sqlservr Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
task: ffff88003599bf40 ti: ffff88008d29c000 task.ti: ffff88008d29c000
RIP: 0010:[<ffffffff810f99a6>]  [<ffffffff810f99a6>] smp_call_function_many+0x206/0x260
RSP: 0018:ffff88008d29fcf0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffff8106e4a0 RCX: ffff8800bb61a7a8
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff88008d29fd28 R08: ffff880086453400 R09: ffffffff813227d9
R10: ffff8800bb699c80 R11: ffffea0002b4a800 R12: 0000000000000282
R13: 0000000000000018 R14: ffff88008d29fc68 R15: 0000000000000282
FS:  00007f211f536700(0000) GS:ffff8800bb680000(0000) knlGS:0000000181ca0000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f211ee21778 CR3: 000000008ebed000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000101301d80 ffff8800b3b5b548 ffff8800b3b5b200 0000000000000000
ffffffffffffffff ffff8800b3b5b548 ffff880090a896c8 ffff88008d29fd78
ffffffff8106e668 ffff8800b3b5b200 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff811b5a01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #200853
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa4b07e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa4a77018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa4a77088
Build id: 3.10.0-862.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffffa4bc0497
Build id: 3.10.0-862.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffa4bc1c75
Build id: 3.10.0-862.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffa4bca624
Build id: 3.10.0-862.el7.x86_64
0xf4
7
? xfs_iunlock
xfs 0xffffffffc0356bec
Build id: 3.10.0-862.el7.x86_64
0xec
8
? __vma_rb_erase
vmlinux 0xffffffffa4bcabe1
Build id: 3.10.0-862.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffa4bccbf5
Build id: 3.10.0-862.el7.x86_64
0x295
10
vm_munmap
vmlinux 0xffffffffa4bcce35
Build id: 3.10.0-862.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffa4bce052
Build id: 3.10.0-862.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffffa511f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [BgSchPool:2351]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev joydev nfit pcspkr sg vmw_balloon libnvdimm parport_pc parport vmw_vmci shpchp i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper ahci syscopyarea sysfillrect sd_mod sysimgblt fb_sys_fops ttm ata_piix crc_t10dif libahci crct10dif_generic drm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmw_pvscsi
vmxnet3 i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 2351 Comm: BgSchPool Kdump: loaded Not tainted 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff91588178af70 ti: ffff915885fe8000 task.ti: ffff915885fe8000
RIP: 0010:[<ffffffffa4b07e3a>]  [<ffffffffa4b07e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff915885febcb8  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff9158bfc9cf00
RDX: 0000000000000002 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff915885febcf0 R08: ffff9155fa357400 R09: ffffffffa4d4a3d9
R10: ffff9158bfcdbb00 R11: ffffca4c8e62ec00 R12: 0000000000000296
R13: ffff915885febc68 R14: 0000000000000001 R15: ffffca4c810f68c0
FS:  00007ff1153f6700(0000) GS:ffff9158bfcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff1431d2000 CR3: 000000041c336000 CR4: 00000000003407e0
Call Trace:
[<ffffffffa4a77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa4a77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffa4bc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffa4bc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffa4bca624>] unmap_region+0xf4/0x140
[<ffffffffc0356bec>] ? xfs_iunlock+0xec/0x130 [xfs]
[<ffffffffa4bcabe1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffa4bccbf5>] do_munmap+0x295/0x470
[<ffffffffa4bcce35>] vm_munmap+0x65/0xb0
[<ffffffffa4bce052>] SyS_munmap+0x22/0x30
[<ffffffffa511f7d5>] system_call_fastpath+0x1c/0x21
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 73 a5 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #191272
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb9707e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb9677018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb9677088
Build id: 3.10.0-862.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffffb97c0497
Build id: 3.10.0-862.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffb97c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffb97ca624
Build id: 3.10.0-862.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffffb97cabe1
Build id: 3.10.0-862.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffffb97ccbf5
Build id: 3.10.0-862.el7.x86_64
0x295
9
vm_munmap
vmlinux 0xffffffffb97cce35
Build id: 3.10.0-862.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffffb97ce052
Build id: 3.10.0-862.el7.x86_64
0x22
11
tracesys
vmlinux 0xffffffffb9d1fa51
Build id: 3.10.0-862.el7.x86_64
0x9d
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 21s! [java:8107]
Modules linked in: tcp_lp fuse tcp_diag inet_diag xt_set xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ip_set veth nf_conntrack_netlink nfnetlink xt_nat xt_statistic ip_vs_sh ip_vs_wrr ip_vs_rr cfg80211 rfkill ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ipt_REJECT nf_reject_ipv4 ip_vs nf_conntrack tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel snd_ens1371 ppdev snd_rawmidi snd_ac97_codec ac97_bus
snd_seq vmw_balloon aesni_intel snd_seq_device lrw gf128mul glue_helper ablk_helper cryptd snd_pcm snd_timer pcspkr joydev snd nfit vmw_vmci i2c_piix4 sg soundcore libnvdimm shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common ata_piix crc32c_intel mptspi libata scsi_transport_spi e1000 serio_raw mptscsih mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 8107 Comm: java Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff9c9589570fd0 ti: ffff9c9794eac000 task.ti: ffff9c9794eac000
RIP: 0010:[<ffffffffb9707e3a>]  [<ffffffffb9707e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9c9794eafcb8  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00000000000000fc RCX: ffff9c992571cf50
RDX: 0000000000000004 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9c9794eafcf0 R08: ffff9c9476544c00 R09: ffffffffb994a3d9
R10: ffff9c992575bb00 R11: ffffe2d04ff8d400 R12: 0000000000000296
R13: ffff9c9794eafc68 R14: ffffffff000000a8 R15: ffffffffb97f8139
FS:  00007f775d6a1700(0000) GS:ffff9c9925740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7745d7f000 CR3: 00000000b1242000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb9677018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb9677088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb97c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffb97c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffb97ca624>] unmap_region+0xf4/0x140
[<ffffffffb97cabe1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb97ccbf5>] do_munmap+0x295/0x470
[<ffffffffb97cce35>] vm_munmap+0x65/0xb0
[<ffffffffb97ce052>] SyS_munmap+0x22/0x30
[<ffffffffb9d1fa51>] tracesys+0x9d/0xc3
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 33 ba f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #183920
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9791173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9787a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9787a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffff979e4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff979e5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff979ef014
Build id: 3.10.0-957.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff979ef5d1
Build id: 3.10.0-957.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff979f1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffff979f1845
Build id: 3.10.0-957.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff979f2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
11
tracesys
vmlinux 0xffffffff97f7505b
Build id: 3.10.0-957.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [7_scheduler:20966]
Modules linked in: tcp_lp fuse xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec joydev ac97_bus pcspkr snd_seq snd_seq_device snd_pcm sg snd_timer snd soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables ext4 mbcache jbd2 sd_mod sr_mod cdrom crc_t10dif crct10dif_generic
ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm e1000 mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks ata_piix libata nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 20966 Comm: 7_scheduler Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9225a9a3b0c0 ti: ffff92255032c000 task.ti: ffff92255032c000
RIP: 0010:[<ffffffff9791173a>]  [<ffffffff9791173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff92255032fcb8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9225b3620600
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff92255032fcf0 R08: ffff92247b4b3c00 R09: ffffffff97b755c9
R10: ffff9225b36df160 R11: ffffe0a8c7fdbc00 R12: 0000000000000296
R13: ffff92255032fc68 R14: 0000000000000001 R15: ffffe0a8ca2de640
FS:  00007f6180d27b20(0000) GS:ffff9225b36c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7c965e45f0 CR3: 0000000249a04000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9787a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9787a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff979e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff979e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff979ef014>] unmap_region+0xf4/0x140
[<ffffffff979ef5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff979f1605>] do_munmap+0x2a5/0x480
[<ffffffff979f1845>] vm_munmap+0x65/0xb0
[<ffffffff979f2ad2>] SyS_munmap+0x22/0x30
[<ffffffff97f7505b>] tracesys+0xa3/0xc9
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 54 98 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #183416
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbad0b0ca
Build id: 3.10.0-862.14.4.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbac79e58
Build id: 3.10.0-862.14.4.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbac79ec8
Build id: 3.10.0-862.14.4.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffffbadc3487
Build id: 3.10.0-862.14.4.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffbadc4d85
Build id: 3.10.0-862.14.4.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffbadcdb74
Build id: 3.10.0-862.14.4.el7.x86_64
0xf4
7
? xfs_iunlock
xfs 0xffffffffc043dbdc
Build id: 3.10.0-862.14.4.el7.x86_64
0xec
8
? __vma_rb_erase
vmlinux 0xffffffffbadce131
Build id: 3.10.0-862.14.4.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffbadd0145
Build id: 3.10.0-862.14.4.el7.x86_64
0x295
10
vm_munmap
vmlinux 0xffffffffbadd0385
Build id: 3.10.0-862.14.4.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffbadd15a2
Build id: 3.10.0-862.14.4.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffffbb32579b
Build id: 3.10.0-862.14.4.el7.x86_64
0x22
13
? system_call_after_swapgs
vmlinux 0xffffffffbb3256e1
Build id: 3.10.0-862.14.4.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [java:16266]
Modules linked in: tcp_lp rfcomm fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel btusb btrtl btbcm btintel aesni_intel bluetooth lrw snd_ens1371 gf128mul glue_helper snd_rawmidi ablk_helper cryptd snd_ac97_codec ppdev ac97_bus snd_seq vmw_balloon snd_seq_device snd_pcm rfkill snd_timer pcspkr snd sg vmw_vmci joydev soundcore i2c_piix4 nfit shpchp libnvdimm parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif
crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul drm crct10dif_common libata crc32c_intel mptspi e1000 scsi_transport_spi serio_raw mptscsih mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 16266 Comm: java Kdump: loaded Tainted: G             L ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffffa1469e276eb0 ti: ffffa145d0be4000 task.ti: ffffa145d0be4000
RIP: 0010:[<ffffffffbad0b0ca>]  [<ffffffffbad0b0ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa145d0be7cb8  EFLAGS: 00000202
RAX: 0000000000000009 RBX: 00000000000000fc RCX: ffffa146ab85ce98
RDX: 0000000000000009 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffffa145d0be7cf0 R08: ffffa14377d3ec00 R09: ffffffffbaf4ef19
R10: ffffa146ab79bb40 R11: ffffe69391437600 R12: 0000000000000296
R13: ffffa145d0be7c68 R14: 0000000000000001 R15: ffffe6938fd54b80
FS:  00007f95da216700(0000) GS:ffffa146ab780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f95f40d0000 CR3: 00000003b4396000 CR4: 00000000000607e0
Call Trace:
[<ffffffffbac79e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbac79ec8>] flush_tlb_mm_range+0x68/0x140
[<ffffffffbadc3487>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffbadc4d85>] tlb_finish_mmu+0x55/0x60
[<ffffffffbadcdb74>] unmap_region+0xf4/0x140
[<ffffffffc043dbdc>] ? xfs_iunlock+0xec/0x130 [xfs]
[<ffffffffbadce131>] ? __vma_rb_erase+0x121/0x220
[<ffffffffbadd0145>] do_munmap+0x295/0x470
[<ffffffffbadd0385>] vm_munmap+0x65/0xb0
[<ffffffffbadd15a2>] SyS_munmap+0x22/0x30
[<ffffffffbb32579b>] system_call_fastpath+0x22/0x27
[<ffffffffbb3256e1>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 fe 84 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 54 93 bb f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 84 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #182538
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8cb12b6a
Build id: 3.10.0-957.27.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8ca7b788
Build id: 3.10.0-957.27.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8ca7b7f8
Build id: 3.10.0-957.27.2.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffff8cbe57d7
Build id: 3.10.0-957.27.2.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff8cbe70d5
Build id: 3.10.0-957.27.2.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff8cbf0754
Build id: 3.10.0-957.27.2.el7.x86_64
0xf4
7
? xfs_iunlock
xfs 0xffffffffc04fb67c
Build id: 3.10.0-957.27.2.el7.x86_64
0xec
8
? __vma_rb_erase
vmlinux 0xffffffff8cbf0d11
Build id: 3.10.0-957.27.2.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff8cbf2d45
Build id: 3.10.0-957.27.2.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffff8cbf2f85
Build id: 3.10.0-957.27.2.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff8cbf4212
Build id: 3.10.0-957.27.2.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff8d176ddb
Build id: 3.10.0-957.27.2.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [httpd:3298]
Modules linked in: binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter unix_diag vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 ppdev snd_rawmidi iosf_mbi snd_ac97_codec ac97_bus crc32_pclmul vmw_balloon snd_seq ghash_clmulni_intel aesni_intel snd_seq_device lrw gf128mul glue_helper snd_pcm ablk_helper cryptd snd_timer snd pcspkr joydev soundcore sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common
crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw ttm ata_piix mptspi scsi_transport_spi drm mptscsih libata mptbase e1000 nfit libnvdimm drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 3298 Comm: httpd Kdump: loaded Tainted: G             L ------------   3.10.0-957.27.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff88d0b16b4100 ti: ffff88d0203fc000 task.ti: ffff88d0203fc000
RIP: 0010:[<ffffffff8cb12b6a>]  [<ffffffff8cb12b6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff88d0203ffcb8  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000000000001b700 RCX: ffff88d0be6e04b8
RDX: 0000000000000003 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff88d0203ffcf0 R08: ffff88d0b71bb400 R09: ffffffff8cd774a9
R10: ffff88d0be69f160 R11: ffffdf2944c52000 R12: ffff88d0203ffc30
R13: 0000000000000286 R14: 0000000000000010 R15: ffffffff8ca5c493
FS:  00007f257f0a8700(0000) GS:ffff88d0be680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f25958b5000 CR3: 00000001327ce000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8ca7b788>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8ca7b7f8>] flush_tlb_mm_range+0x68/0x140
[<ffffffff8cbe57d7>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff8cbe70d5>] tlb_finish_mmu+0x55/0x60
[<ffffffff8cbf0754>] unmap_region+0xf4/0x140
[<ffffffffc04fb67c>] ? xfs_iunlock+0xec/0x130 [xfs]
[<ffffffff8cbf0d11>] ? __vma_rb_erase+0x121/0x220
[<ffffffff8cbf2d45>] do_munmap+0x2a5/0x480
[<ffffffff8cbf2f85>] vm_munmap+0x65/0xb0
[<ffffffff8cbf4212>] SyS_munmap+0x22/0x30
[<ffffffff8d176ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 5e 75 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 bf 74 8d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 2d 75 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #168330
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff82d1173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff82c7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff82c7a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffff82de4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff82de5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff82def014
Build id: 3.10.0-957.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff82def5d1
Build id: 3.10.0-957.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff82df1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffff82df1845
Build id: 3.10.0-957.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff82df2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff83374ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [360entclient:17174]
Modules linked in: kbase(OE) udp_diag tcp_diag inet_diag binfmt_misc fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd parport_pc parport virtio_balloon joydev i2c_piix4 pcspkr ip_tables ext4 mbcache jbd2 ata_generic pata_acpi
virtio_blk virtio_console virtio_net cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix floppy libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 17174 Comm: 360entclient Kdump: loaded Tainted: G           OEL ------------   3.10.0-957.el7.x86_64 #1
Hardware name: Fedora Project OpenStack Nova, BIOS 0.5.1 01/01/2011
task: ffff9c0e0aa75140 ti: ffff9c0dee370000 task.ti: ffff9c0dee370000
RIP: 0010:[<ffffffff82d1173a>]  [<ffffffff82d1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9c0dee373cb8  EFLAGS: 00000202
RAX: 0000000000000009 RBX: 00000000000000fc RCX: ffff9c0e3f460398
RDX: 0000000000000009 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff9c0dee373cf0 R08: ffff9bffa94f4400 R09: ffffffff82f755c9
R10: ffff9c0e3f39f160 R11: fffff81ed4acd600 R12: 0000000000000296
R13: ffff9c0dee373c68 R14: 0000000000000001 R15: fffff81ec4366c80
FS:  00007fad2379b700(0000) GS:ffff9c0e3f380000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007faceabcb000 CR3: 0000000fea2f0000 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff82c7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff82c7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff82de4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff82de5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff82def014>] unmap_region+0xf4/0x140
[<ffffffff82def5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff82df1605>] do_munmap+0x2a5/0x480
[<ffffffff82df1845>] vm_munmap+0x65/0xb0
[<ffffffff82df2ad2>] SyS_munmap+0x22/0x30
[<ffffffff83374ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 94 83 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #161556
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffadf07e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffade77018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffade77088
Build id: 3.10.0-862.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffffadfc0497
Build id: 3.10.0-862.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffadfc1c75
Build id: 3.10.0-862.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffadfca624
Build id: 3.10.0-862.el7.x86_64
0xf4
7
? xfs_iunlock
xfs 0xffffffffc03a5bec
Build id: 3.10.0-862.el7.x86_64
0xec
8
? __vma_rb_erase
vmlinux 0xffffffffadfcabe1
Build id: 3.10.0-862.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffffadfccbf5
Build id: 3.10.0-862.el7.x86_64
0x295
10
vm_munmap
vmlinux 0xffffffffadfcce35
Build id: 3.10.0-862.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffadfce052
Build id: 3.10.0-862.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffffae51f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [TaskSchedulerFo:32809]
Modules linked in: tcp_lp fuse xt_statistic nf_conntrack_netlink xt_set xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ip_set nfnetlink nfsv3 veth xt_nat ip_vs_sh ip_vs_wrr ip_vs_rr rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark ip_vs xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul snd_ens1371 snd_rawmidi snd_ac97_codec ghash_clmulni_intel ac97_bus snd_seq ppdev
snd_seq_device aesni_intel snd_pcm lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon snd_timer joydev pcspkr snd sg vmw_vmci soundcore nfit i2c_piix4 shpchp libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common ata_piix crc32c_intel mptspi scsi_transport_spi libata mptscsih serio_raw mptbase e1000 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 32809 Comm: TaskSchedulerFo Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff93525b25cf10 ti: ffff93508cd64000 task.ti: ffff93508cd64000
RIP: 0010:[<ffffffffadf07e3a>]  [<ffffffffadf07e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff93508cd67cb8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff93529cc1cd90
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff93508cd67cf0 R08: ffff9352813d8c00 R09: ffffffffae14a3d9
R10: ffff93529cc5bb00 R11: ffffcdeec2e00200 R12: 0000000000000296
R13: 0000000000000018 R14: ffff93508cd67c28 R15: 0000000000000296
FS:  00007f73f8e59700(0000) GS:ffff93529cc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f73fc656000 CR3: 000000004dfb8000 CR4: 00000000003607e0
Call Trace:
[<ffffffffade77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffade77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffadfc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffadfc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffadfca624>] unmap_region+0xf4/0x140
[<ffffffffc03a5bec>] ? xfs_iunlock+0xec/0x130 [xfs]
[<ffffffffadfcabe1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffadfccbf5>] do_munmap+0x295/0x470
[<ffffffffadfcce35>] vm_munmap+0x65/0xb0
[<ffffffffadfce052>] SyS_munmap+0x22/0x30
[<ffffffffae51f7d5>] system_call_fastpath+0x1c/0x21
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 b3 ae f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #15517
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff811b5a01
Build id: 3.10.0-693.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
0x292
9
vm_munmap
vmlinux 0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [in:imjournal:1215]
Modules linked in: binfmt_misc vmw_vsock_vmci_transport vsock sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg vmw_balloon joydev pcspkr parport_pc i2c_piix4 vmw_vmci parport shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix scsi_transport_spi crct10dif_pclmul mptscsih crct10dif_common libata drm serio_raw crc32c_intel mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 1215 Comm: in:imjournal Tainted: G        W    L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffff88022c7beeb0 ti: ffff8802357a8000 task.ti: ffff8802357a8000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8802357abcf0  EFLAGS: 00000202
RAX: 0000000000000009 RBX: ffffffff8106e4a0 RCX: ffff88023745b058
RDX: 0000000000000009 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff8802357abd28 R08: ffff88017cdc7800 R09: ffffffff813227d9
R10: ffff880237399c80 R11: ffffea000b81f400 R12: 0000000000000282
R13: ffff8802357abca0 R14: ffff8802357abe08 R15: 00007fefc22f6000
FS:  00007fefbdfcc700(0000) GS:ffff880237380000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fefc22f6000 CR3: 000000022ff9d000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000000012c7beeb0 ffff880235670fc8 ffff880235670c80 00007fefc22f6000
00007fefc22f7000 ffff880235670fc8 ffff880230ae8948 ffff8802357abd78
ffffffff8106e668 ffff880235670c80 00007fefc22f6000 00007fefc22f7000
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff811b5a01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #14410
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffac707e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffac677018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffac677088
Build id: 3.10.0-862.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffffac7c0497
Build id: 3.10.0-862.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffac7c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffac7ca624
Build id: 3.10.0-862.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffffac7cabe1
Build id: 3.10.0-862.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffffac7ccbf5
Build id: 3.10.0-862.el7.x86_64
0x295
9
vm_munmap
vmlinux 0xffffffffac7cce35
Build id: 3.10.0-862.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffffac7ce052
Build id: 3.10.0-862.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffffacd1f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [gnome-shell:2077]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac snd_ens1371 coretemp snd_rawmidi snd_ac97_codec iosf_mbi crc32_pclmul ghash_clmulni_intel ac97_bus aesni_intel snd_seq lrw gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm ppdev vmw_balloon i2c_piix4
snd_timer pcspkr snd sg soundcore joydev vmw_vmci shpchp nfit libnvdimm parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libata e1000 serio_raw mptspi scsi_transport_spi mptscsih mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 8 PID: 2077 Comm: gnome-shell Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9a4ef5748000 ti: ffff9a59fe260000 task.ti: ffff9a59fe260000
RIP: 0010:[<ffffffffac707e3a>]  [<ffffffffac707e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9a59fe263cb8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9a5a2041cfc8
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9a59fe263cf0 R08: ffff9a502e3ed000 R09: ffffffffac94a3d9
R10: ffff9a5a2061bb00 R11: ffffec74ed7b1400 R12: 0000000000000296
R13: 0000000000000018 R14: ffff9a59fe263c28 R15: 0000000000000296
FS:  00007f0a1d0b0ac0(0000) GS:ffff9a5a20600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f095f86a000 CR3: 0000000b45e92000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffac677018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffac677088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffac7c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffac7c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffac7ca624>] unmap_region+0xf4/0x140
[<ffffffffac7cabe1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffac7ccbf5>] do_munmap+0x295/0x470
[<ffffffffac7cce35>] vm_munmap+0x65/0xb0
[<ffffffffac7ce052>] SyS_munmap+0x22/0x30
[<ffffffffacd1f7d5>] system_call_fastpath+0x1c/0x21
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 33 ad f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #13279
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff98f0862a
Build id: 3.10.0-862.9.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff98e77788
Build id: 3.10.0-862.9.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff98e777f8
Build id: 3.10.0-862.9.1.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffff98fc0c57
Build id: 3.10.0-862.9.1.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff98fc2435
Build id: 3.10.0-862.9.1.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff98fcade4
Build id: 3.10.0-862.9.1.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff98fcb3bf
Build id: 3.10.0-862.9.1.el7.x86_64
0x13f
8
do_munmap
vmlinux 0xffffffff98fcd3b5
Build id: 3.10.0-862.9.1.el7.x86_64
0x295
9
vm_munmap
vmlinux 0xffffffff98fcd5f5
Build id: 3.10.0-862.9.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff98fce812
Build id: 3.10.0-862.9.1.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff99520795
Build id: 3.10.0-862.9.1.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [java:20256]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon sg pcspkr joydev nfit libnvdimm parport_pc i2c_piix4 vmw_vmci parport shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix
serio_raw mptspi scsi_transport_spi e1000 libata mptscsih mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 20256 Comm: java Kdump: loaded Tainted: G        W    L ------------   3.10.0-862.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9a9eb5160000 ti: ffff9aa04df4c000 task.ti: ffff9aa04df4c000
RIP: 0010:[<ffffffff98f0862a>]  [<ffffffff98f0862a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9aa04df4fcb8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00000000000000fc RCX: ffff9aa0bfc5ce70
RDX: 0000000000000001 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff9aa04df4fcf0 R08: ffff9a9ffcf36400 R09: ffffffff9914ae19
R10: ffff9aa0bfd5bb40 R11: ffffcbf486851600 R12: 0000000000000296
R13: ffff9aa04df4fc68 R14: ffffcbf482544b80 R15: ffffffff99028af1
FS:  00007f74f4149700(0000) GS:ffff9aa0bfd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1aa89949b1 CR3: 00000000b5b5c000 CR4: 00000000001607e0
Call Trace:
[<ffffffff98e77788>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff98e777f8>] flush_tlb_mm_range+0x68/0x140
[<ffffffff98fc0c57>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff98fc2435>] tlb_finish_mmu+0x55/0x60
[<ffffffff98fcade4>] unmap_region+0xf4/0x140
[<ffffffff98fcb3bf>] ? __vma_rb_erase+0x13f/0x220
[<ffffffff98fcd3b5>] do_munmap+0x295/0x470
[<ffffffff98fcd5f5>] vm_munmap+0x65/0xb0
[<ffffffff98fce812>] SyS_munmap+0x22/0x30
[<ffffffff99520795>] system_call_fastpath+0x1c/0x21
Code: 48 63 35 9e ae c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 b3 99 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d ae c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #12316
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff88907e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff88877018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff88877088
Build id: 3.10.0-862.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffff889c0497
Build id: 3.10.0-862.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff889c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff889ca624
Build id: 3.10.0-862.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff889cabe1
Build id: 3.10.0-862.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff889ccbf5
Build id: 3.10.0-862.el7.x86_64
0x295
9
vm_munmap
vmlinux 0xffffffff889cce35
Build id: 3.10.0-862.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff889ce052
Build id: 3.10.0-862.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff88f1f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
12
? system_call_after_swapgs
vmlinux 0xffffffff88f1f721
Build id: 3.10.0-862.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 37s! [ftdc:4048]
Modules linked in: binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype tun xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev sg pcspkr floppy parport_pc parport shpchp vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm e1000 ata_piix mptspi scsi_transport_spi
crct10dif_pclmul crct10dif_common crc32c_intel mptscsih serio_raw libata mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 4048 Comm: ftdc Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9704eea4eeb0 ti: ffff970282ae0000 task.ti: ffff970282ae0000
RIP: 0010:[<ffffffff88907e3a>]  [<ffffffff88907e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff970282ae3cb8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0000000000019680 RCX: ffff97059ee1cd90
RDX: 0000000000000000 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff970282ae3cf0 R08: ffff96fef4413c00 R09: ffffffff88b4a3d9
R10: ffff97059ee9bb00 R11: ffffce0554cbde00 R12: ffff970282ae3c68
R13: 00007f0014580000 R14: 0000000000000001 R15: ffffce0541ac2ac0
FS:  00007f000659c700(0000) GS:ffff97059ee80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0014580000 CR3: 00000004d27c6000 CR4: 00000000000007e0
Call Trace:
[<ffffffff88877018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff88877088>] flush_tlb_mm_range+0x68/0x140
[<ffffffff889c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffff889c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffff889ca624>] unmap_region+0xf4/0x140
[<ffffffff889cabe1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff889ccbf5>] do_munmap+0x295/0x470
[<ffffffff889cce35>] vm_munmap+0x65/0xb0
[<ffffffff889ce052>] SyS_munmap+0x22/0x30
[<ffffffff88f1f7d5>] system_call_fastpath+0x1c/0x21
[<ffffffff88f1f721>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 53 89 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #10858
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffab71173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffab67a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffab67a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffffab7e4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffab7e5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffab7ef014
Build id: 3.10.0-957.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffffab7ef5d1
Build id: 3.10.0-957.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffffab7f1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffffab7f1845
Build id: 3.10.0-957.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffffab7f2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffffabd74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [polkitd:2863]
Modules linked in: tcp_lp isofs ext4 mbcache jbd2 rfcomm fuse cmac xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ipmi_devintf ipmi_msghandler drbg ansi_cprng ccm ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep sunrpc vfat fat arc4 snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core rtsx_pci_ms nouveau snd_hwdep snd_seq ath10k_pci ath10k_core ath
mac80211 snd_seq_device edac_mce_amd kvm_amd snd_pcm kvm mxm_wmi video i2c_algo_bit ttm snd_timer drm_kms_helper snd cfg80211 irqbypass soundcore syscopyarea sysfillrect btusb btrtl btbcm dell_wmi btintel dell_smbios memstick bluetooth sysimgblt dcdbas sparse_keymap rfkill fb_sys_fops rtsx_pci_sdmmc dell_wmi_descriptor drm crc32_pclmul ghash_clmulni_intel drm_panel_orientation_quirks aesni_intel lrw gf128mul glue_helper ablk_helper wmi cryptd sg wdat_wdt rtsx_pci pcspkr sdhci_acpi r8169 sdhci mii ccp k10temp i2c_piix4 pinctrl_amd gpio_amdpt acpi_cpufreq mmc_core iosf_mbi i2c_designware_platform i2c_designware_core binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ahci libahci libata dm_mirror dm_region_hash
dm_log dm_mod
CPU: 6 PID: 2863 Comm: polkitd Tainted: G        W      ------------   3.10.0-957.el7.x86_64 #1
Hardware name: Dell Inc. Inspiron 5675/07PR60, BIOS 1.1.2 05/04/2017
task: ffff8b22beef0000 ti: ffff8b22a1f90000 task.ti: ffff8b22a1f90000
RIP: 0010:[<ffffffffab71173a>]  [<ffffffffab71173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b22a1f93cb8  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 000000000001b700 RCX: ffff8b22b6320678
RDX: 0000000000000004 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff8b22a1f93cf0 R08: ffff8b21bceec800 R09: ffffffffab9755c9
R10: ffff8b22b639f160 R11: ffffdb3388f8ae00 R12: ffff8b22a1f93c68
R13: ffff8b21bfc03500 R14: 0000000000000001 R15: ffffffffab81d04a
FS:  00007f0a399b4900(0000) GS:ffff8b22b6380000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0a399da000 CR3: 000000026af9c000 CR4: 00000000003407e0
Call Trace:
[<ffffffffab67a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffab67a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffab7e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffab7e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffab7ef014>] unmap_region+0xf4/0x140
[<ffffffffab7ef5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffab7f1605>] do_munmap+0x2a5/0x480
[<ffffffffab7f1845>] vm_munmap+0x65/0xb0
[<ffffffffab7f2ad2>] SyS_munmap+0x22/0x30
[<ffffffffabd74ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 34 ac f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #9654
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f9f72
Build id: 3.10.0-514.26.2.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106c5f8
Build id: 3.10.0-514.26.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106c6c9
Build id: 3.10.0-514.26.2.el7.x86_64
0x69
4
tlb_flush_mmu.part.61
vmlinux 0xffffffff811ac733
Build id: 3.10.0-514.26.2.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ada25
Build id: 3.10.0-514.26.2.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b4944
Build id: 3.10.0-514.26.2.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff811b4edf
Build id: 3.10.0-514.26.2.el7.x86_64
0x13f
8
do_munmap
vmlinux 0xffffffff811b6e75
Build id: 3.10.0-514.26.2.el7.x86_64
0x245
9
vm_munmap
vmlinux 0xffffffff811b7091
Build id: 3.10.0-514.26.2.el7.x86_64
0x41
10
SyS_munmap
vmlinux 0xffffffff811b8042
Build id: 3.10.0-514.26.2.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff81697809
Build id: 3.10.0-514.26.2.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [beam.smp:2831]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec intel_powerclamp ac97_bus snd_seq snd_seq_device coretemp snd_pcm ppdev snd_timer snd soundcore vmw_balloon pcspkr sg parport_pc parport i2c_piix4 shpchp vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crc32c_intel drm
mptspi scsi_transport_spi ata_piix mptscsih mptbase libata e1000 serio_raw i2c_core floppy fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 2831 Comm: beam.smp Tainted: G             L ------------   3.10.0-514.26.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
task: ffff88009535edd0 ti: ffff880098ef0000 task.ti: ffff880098ef0000
RIP: 0010:[<ffffffff810f9f72>]  [<ffffffff810f9f72>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff880098ef3d10  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffff8106c430 RCX: ffff8800bb41ad48
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff880098ef3d48 R08: ffff880086433400 R09: ffffffff81318499
R10: ffff8800bb499b40 R11: ffffea0002548000 R12: 0000000000000286
R13: ffff880098ef3cc0 R14: ffff880098ef3e28 R15: 00007f803d90a000
FS:  00007f803e23b700(0000) GS:ffff8800bb480000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f6eb65ce47e CR3: 000000009aeaf000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001fffff000 ffff8800358ac808 ffff8800358ac4c0 0000000000000000
ffffffffffffffff ffff8800358ac808 ffff8800358cc798 ffff880098ef3d98
ffffffff8106c5f8 ffff8800358ac4c0 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106c5f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c6c9>] flush_tlb_mm_range+0x69/0x140
[<ffffffff811ac733>] tlb_flush_mmu.part.61+0x33/0xc0
[<ffffffff811ada25>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b4944>] unmap_region+0xf4/0x140
[<ffffffff811b4edf>] ? __vma_rb_erase+0x13f/0x220
[<ffffffff811b6e75>] do_munmap+0x245/0x420
[<ffffffff811b7091>] vm_munmap+0x41/0x60
[<ffffffff811b8042>] SyS_munmap+0x22/0x30
[<ffffffff81697809>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 da 9e 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 80 99 ad 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 d9 9e 00 eb b7 0f b6 4d cc 4c 89 
Complete report #8954
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff84d1173e
Build id: 3.10.0-957.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffff84c7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff84c7a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffff84de4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff84de5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff84def014
Build id: 3.10.0-957.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff84def5d1
Build id: 3.10.0-957.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff84df1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffff84df1845
Build id: 3.10.0-957.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff84df2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff85374ddb
Build id: 3.10.0-957.el7.x86_64
0x22
12
? system_call_after_swapgs
vmlinux 0xffffffff85374d21
Build id: 3.10.0-957.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [Glowroot-Gauge-:2732]
Modules linked in: nfsv3 nfs_acl nfs lockd grace fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter qxl ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ppdev drm sg pcspkr drm_panel_orientation_quirks virtio_balloon joydev i2c_piix4 parport_pc parport auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi virtio_console
serio_raw floppy ata_piix libata e1000 virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 2732 Comm: Glowroot-Gauge- Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9d0ee0d5e180 ti: ffff9d0f2f138000 task.ti: ffff9d0f2f138000
RIP: 0010:[<ffffffff84d1173e>]  [<ffffffff84d1173e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff9d0f2f13bcb8  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff9d103fd1f610
RDX: 0000000000000002 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff9d0f2f13bcf0 R08: ffff9d0fffd42000 R09: ffffffff84f755c9
R10: ffff9d103fc1f160 R11: ffffe9a705037c00 R12: 0000000000000296
R13: ffff9d0f2f13bc68 R14: 0000000000000001 R15: ffffe9a70569b280
FS:  00007f7a4f5f5700(0000) GS:ffff9d103fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7ad4fd1000 CR3: 00000001a640a000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff84c7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff84c7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff84de4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff84de5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff84def014>] unmap_region+0xf4/0x140
[<ffffffff84def5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff84df1605>] do_munmap+0x2a5/0x480
[<ffffffff84df1845>] vm_munmap+0x65/0xb0
[<ffffffff84df2ad2>] SyS_munmap+0x22/0x30
[<ffffffff85374ddb>] system_call_fastpath+0x22/0x27
[<ffffffff85374d21>] ? system_call_after_swapgs+0xae/0x146
Code: 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 94 85 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #7450
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8631173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8627a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8627a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffff863e4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff863e5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff863ef014
Build id: 3.10.0-957.el7.x86_64
0xf4
7
? __vma_rb_erase
vmlinux 0xffffffff863ef5d1
Build id: 3.10.0-957.el7.x86_64
0x121
8
do_munmap
vmlinux 0xffffffff863f1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffff863f1845
Build id: 3.10.0-957.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff863f2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff86974ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [C1 CompilerThre:17930]
Modules linked in: fuse xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop binfmt_misc ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc coretemp ext4 iosf_mbi mbcache jbd2 crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr vmw_balloon
joydev parport_pc parport sg vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx serio_raw drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm e1000 ata_piix drm mptspi libata scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 17930 Comm: C1 CompilerThre Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9a1d5f68c100 ti: ffff9a1d7eb10000 task.ti: ffff9a1d7eb10000
RIP: 0010:[<ffffffff8631173a>]  [<ffffffff8631173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9a1d7eb13cb8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9a1ded6204b8
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9a1d7eb13cf0 R08: ffff9a1b39cb3800 R09: ffffffff865755c9
R10: ffff9a1ded69f160 R11: ffffd90c90b26a00 R12: 0000000000000296
R13: ffff9a1d7eb13c68 R14: 0000000000000001 R15: ffffd90c888e15c0
FS:  00007f63dd7ec700(0000) GS:ffff9a1ded680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f63dc9bd000 CR3: 00000001e77c0000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8627a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8627a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff863e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff863e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff863ef014>] unmap_region+0xf4/0x140
[<ffffffff863ef5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffff863f1605>] do_munmap+0x2a5/0x480
[<ffffffff863f1845>] vm_munmap+0x65/0xb0
[<ffffffff863f2ad2>] SyS_munmap+0x22/0x30
[<ffffffff86974ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 f4 86 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #930510
Frame # Function Binary Source or offset Line
1
flush_tlb_mm_range
vmlinux 0xffffffffb6877088
Build id: 3.10.0-862.el7.x86_64
0x68
2
tlb_flush_mmu.part.75
vmlinux 0xffffffffb69c0497
Build id: 3.10.0-862.el7.x86_64
0x37
3
tlb_finish_mmu
vmlinux 0xffffffffb69c1c75
Build id: 3.10.0-862.el7.x86_64
0x55
4
unmap_region
vmlinux 0xffffffffb69ca624
Build id: 3.10.0-862.el7.x86_64
0xf4
5
? __vma_rb_erase
vmlinux 0xffffffffb69cabe1
Build id: 3.10.0-862.el7.x86_64
0x121
6
do_munmap
vmlinux 0xffffffffb69ccbf5
Build id: 3.10.0-862.el7.x86_64
0x295
7
vm_munmap
vmlinux 0xffffffffb69cce35
Build id: 3.10.0-862.el7.x86_64
0x65
8
SyS_munmap
vmlinux 0xffffffffb69ce052
Build id: 3.10.0-862.el7.x86_64
0x22
9
system_call_fastpath
vmlinux 0xffffffffb6f1f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 41s! [java:6301]
[<ffffffffb6877088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb69c0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffb69c1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffb69ca624>] unmap_region+0xf4/0x140
[<ffffffffb69cabe1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb69ccbf5>] do_munmap+0x295/0x470
[<ffffffffb69cce35>] vm_munmap+0x65/0xb0
[<ffffffffb69ce052>] SyS_munmap+0x22/0x30
[<ffffffffb6f1f7d5>] system_call_fastpath+0x1c/0x21
Complete report #918421
Frame # Function Binary Source or offset Line
1
flush_tlb_mm_range
vmlinux 0xffffffffbd27a608
Build id: 3.10.0-957.el7.x86_64
0x68
2
tlb_flush_mmu.part.76
vmlinux 0xffffffffbd3e4077
Build id: 3.10.0-957.el7.x86_64
0x37
3
tlb_finish_mmu
vmlinux 0xffffffffbd3e5975
Build id: 3.10.0-957.el7.x86_64
0x55
4
unmap_region
vmlinux 0xffffffffbd3ef014
Build id: 3.10.0-957.el7.x86_64
0xf4
5
? __vma_rb_erase
vmlinux 0xffffffffbd3ef5d1
Build id: 3.10.0-957.el7.x86_64
0x121
6
do_munmap
vmlinux 0xffffffffbd3f1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
7
vm_munmap
vmlinux 0xffffffffbd3f1845
Build id: 3.10.0-957.el7.x86_64
0x65
8
SyS_munmap
vmlinux 0xffffffffbd3f2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
9
system_call_fastpath
vmlinux 0xffffffffbd974ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [rtkit-daemon:8867]
[<ffffffffbd27a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffbd3e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffbd3e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffbd3ef014>] unmap_region+0xf4/0x140
[<ffffffffbd3ef5d1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffbd3f1605>] do_munmap+0x2a5/0x480
[<ffffffffbd3f1845>] vm_munmap+0x65/0xb0
[<ffffffffbd3f2ad2>] SyS_munmap+0x22/0x30
[<ffffffffbd974ddb>] system_call_fastpath+0x22/0x27