Info

Permalink
Function
smp_call_function_many
First occurrence
2020-09-24
Last occurrence
2026-05-03
State
NEW
Components
kernel
Quality
-20

Statistics

Operating system Count
CentOS 7 392
Architecture Count
x86_64 392
Related packages Count
kernel 392
 0:3.10.0-957.el7 187
 0:3.10.0-957.1.3.el7 34
 0:3.10.0-693.el7 28
 0:3.10.0-1160.71.1.el7 21
 0:3.10.0-1062.12.1.el7 16
 0:3.10.0-1160.114.2.el7 15
 0:3.10.0-1160.62.1.el7 9
 0:3.10.0-1062.el7 9
 0:3.10.0-1160.15.2.el7 8
 0:3.10.0-957.12.2.el7 8
 0:3.10.0-1160.119.1.el7 5
 0:3.10.0-1160.el7 5
 0:3.10.0-1062.9.1.el7 4
 0:3.10.0-1062.18.1.el7 4
 0:3.10.0-1127.19.1.el7 4
 0:3.10.0-1160.80.1.el7 4
 0:3.10.0-1160.59.1.el7 3
 0:3.10.0-1160.81.1.el7 3
 0:3.10.0-1160.99.1.el7 3
 0:3.10.0-1160.66.1.el7 2
 0:3.10.0-1160.45.1.el7 2
 0:3.10.0-1160.53.1.el7 2
 0:3.10.0-1160.88.1.el7 2
 0:3.10.0-693.17.1.el7 2
 0:3.10.0-1062.1.1.el7 1
 0:3.10.0-1160.11.1.el7 1
 0:3.10.0-1160.42.2.el7 1
 0:3.10.0-1160.6.1.el7 1
 0:3.10.0-1160.90.1.el7 1
 0:3.10.0-1127.el7 1
 0:3.10.0-1160.118.1.el7 1
 0:3.10.0-1160.31.1.el7 1
 0:3.10.0-957.27.2.el7 1
 0:3.10.0-957.10.1.el7 1
 0:3.10.0-1160.76.1.el7 1
 0:3.10.0-1160.102.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #1402998
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffab31e07a
Build id: 3.10.0-1160.81.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffab281f48
Build id: 3.10.0-1160.81.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffab281fbb
Build id: 3.10.0-1160.81.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffab3f9f0e
Build id: 3.10.0-1160.81.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffab3fba4f
Build id: 3.10.0-1160.81.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffab3fbb83
Build id: 3.10.0-1160.81.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffab405524
Build id: 3.10.0-1160.81.1.el7.x86_64
0xf4
8
do_munmap
vmlinux 0xffffffffab407b75
Build id: 3.10.0-1160.81.1.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffffab407db5
Build id: 3.10.0-1160.81.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffffab4090e2
Build id: 3.10.0-1160.81.1.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffffab9c539a
Build id: 3.10.0-1160.81.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [9_scheduler:120578]
Modules linked in: mfe_aac_100716843(OE) mfe_fileaccess_100716843(OE) btrfs raid6_pq xor binfmt_misc veth xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) twnotify(OE) vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic ahci drm ata_piix libahci nfit crct10dif_pclmul libata crct10dif_common
libnvdimm crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 8 PID: 120578 Comm: 9_scheduler Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffffa00bfe798000 ti: ffffa005c4c94000 task.ti: ffffa005c4c94000
RIP: 0010:[<ffffffffab31e07a>]  [<ffffffffab31e07a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffffa005c4c97cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000ab281ab0 RCX: ffffa00cdd620718
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffffa005c4c97ce8 R08: ffffa0062c55d400 R09: ffffffffab59c139
R10: ffffa00cdd81f160 R11: ffffdbe5de977000 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa005c4c97c60 R15: 00007fa79a8df000
FS:  00007fa855481700(0000) GS:ffffa00cdd800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff2814ec240 CR3: 000000080a684000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffab281f48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffab281fbb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffab3f9f0e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffab3fba4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffab3fbb83>] tlb_finish_mmu+0x23/0x40
[<ffffffffab405524>] unmap_region+0xf4/0x150
[<ffffffffab407b75>] do_munmap+0x2a5/0x480
[<ffffffffab407db5>] vm_munmap+0x65/0xb0
[<ffffffffab4090e2>] SyS_munmap+0x22/0x30
[<ffffffffab9c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 f5 ab f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 19 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1251564
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8fb1e23a
Build id: 3.10.0-1160.102.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8fa81fa8
Build id: 3.10.0-1160.102.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8fa8201b
Build id: 3.10.0-1160.102.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff8fbfa07e
Build id: 3.10.0-1160.102.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffff8fbfbbff
Build id: 3.10.0-1160.102.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff8fbfbd33
Build id: 3.10.0-1160.102.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff8fc058b4
Build id: 3.10.0-1160.102.1.el7.x86_64
0xf4
8
do_munmap
vmlinux 0xffffffff8fc07f15
Build id: 3.10.0-1160.102.1.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffff8fc08155
Build id: 3.10.0-1160.102.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff8fc09482
Build id: 3.10.0-1160.102.1.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff901c539a
Build id: 3.10.0-1160.102.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 54s! [java:26795]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netbios_ns nf_conntrack_broadcast xt_nat xt_mark ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables binfmt_misc tcp_lp tun devlink bridge stp llc ip_set vmw_vsock_vmci_transport vsock sunrpc ppdev sb_edac vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul nfit glue_helper libnvdimm ablk_helper cryptd joydev sg pcspkr vmw_vmci
i2c_piix4 parport_pc parport xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi crct10dif_pclmul crct10dif_common libata scsi_transport_spi crc32c_intel serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ip_tables]
CPU: 6 PID: 26795 Comm: java Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.102.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8f2fee948000 ti: ffff8f2aa1010000 task.ti: ffff8f2aa1010000
RIP: 0010:[<ffffffff8fb1e23a>]  [<ffffffff8fb1e23a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8f2aa1013cb0  EFLAGS: 00000202
RAX: 0000000000000008 RBX: 000800008fa81b10 RCX: ffff8f329f4206c8
RDX: 0000000000000008 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff8f2aa1013ce8 R08: ffff8f2bec52a000 R09: ffffffff8fd9c7b9
R10: ffff8f329f39f160 R11: ffffeeee1a995a00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8f2aa1013c60 R15: ffff8f2fee948000
FS:  00007fa6451b7700(0000) GS:ffff8f329f380000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa6c6464280 CR3: 000000058db36000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8fa81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8fa8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8fbfa07e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff8fbfbbff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff8fbfbd33>] tlb_finish_mmu+0x23/0x40
[<ffffffff8fc058b4>] unmap_region+0xf4/0x150
[<ffffffff8fc07f15>] do_munmap+0x2a5/0x480
[<ffffffff8fc08155>] vm_munmap+0x65/0xb0
[<ffffffff8fc09482>] SyS_munmap+0x22/0x30
[<ffffffff901c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 4e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 19 75 90 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #887159
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb271e23a
Build id: 3.10.0-1160.99.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb2681fa8
Build id: 3.10.0-1160.99.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb268201b
Build id: 3.10.0-1160.99.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb27fa07e
Build id: 3.10.0-1160.99.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb27fbbff
Build id: 3.10.0-1160.99.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb27fbd33
Build id: 3.10.0-1160.99.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffb28058b4
Build id: 3.10.0-1160.99.1.el7.x86_64
0xf4
8
do_munmap
vmlinux 0xffffffffb2807f15
Build id: 3.10.0-1160.99.1.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffffb2808155
Build id: 3.10.0-1160.99.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffffb2809482
Build id: 3.10.0-1160.99.1.el7.x86_64
0x22
11
tracesys
vmlinux 0xffffffffb2dc562e
Build id: 3.10.0-1160.99.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [elasticsearch[s:9533]
Modules linked in: traps(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm ata_piix nfit crct10dif_pclmul crct10dif_common crc32c_intel libata libnvdimm serio_raw vmxnet3
vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: traps]
CPU: 4 PID: 9533 Comm: elasticsearch[s Kdump: loaded Tainted: G           OE  ------------   3.10.0-1160.99.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffff92c4ccdd1080 ti: ffff92c4cce34000 task.ti: ffff92c4cce34000
RIP: 0010:[<ffffffffb271e23a>]  [<ffffffffb271e23a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff92c4cce37cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffb2681b10 RCX: ffff92c9ffc20650
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff92c4cce37ce8 R08: ffff92bb19551400 R09: ffffffffb299c7b9
R10: ffff92c9ffd1f160 R11: fffff282b148ae00 R12: ffff92c4cce37c80
R13: ffffffffb299c6ef R14: ffff92c4cce37c30 R15: ffff92c4ccdd1080
FS:  00007fb29807b700(0000) GS:ffff92c9ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb8e100a000 CR3: 0000000fe3b20000 CR4: 00000000000007e0
Call Trace:
[<ffffffffb2681fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb268201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb27fa07e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffb27fbbff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb27fbd33>] tlb_finish_mmu+0x23/0x40
[<ffffffffb28058b4>] unmap_region+0xf4/0x150
[<ffffffffb2807f15>] do_munmap+0x2a5/0x480
[<ffffffffb2808155>] vm_munmap+0x65/0xb0
[<ffffffffb2809482>] SyS_munmap+0x22/0x30
[<ffffffffb2dc562e>] tracesys+0xa6/0xcc
Code: 48 63 35 4e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 19 35 b3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #826110
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9dd1173e
Build id: 3.10.0-957.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffff9dc7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9dc7a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffff9dde4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff9dde5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff9ddef014
Build id: 3.10.0-957.el7.x86_64
0xf4
7
do_munmap
vmlinux 0xffffffff9ddf1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
8
vm_munmap
vmlinux 0xffffffff9ddf1845
Build id: 3.10.0-957.el7.x86_64
0x65
9
SyS_munmap
vmlinux 0xffffffff9ddf2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
10
tracesys
vmlinux 0xffffffff9e37505b
Build id: 3.10.0-957.el7.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [java:106495]
Modules linked in: veth nfnetlink_queue nfnetlink_log bluetooth rfkill xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq joydev snd_seq_device pcspkr snd_pcm snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic
crct10dif_pclmul crct10dif_common crc32c_intel pata_acpi vmwgfx serio_raw drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix e1000 libata mptspi scsi_transport_spi nfit mptscsih libnvdimm mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 106495 Comm: java Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff8f15782e2080 ti: ffff8f168b278000 task.ti: ffff8f168b278000
RIP: 0010:[<ffffffff9dd1173e>]  [<ffffffff9dd1173e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8f168b27bcb8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8f1845020600
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8f168b27bcf0 R08: ffff8f15786cc400 R09: ffffffff9df755c9
R10: ffff8f18450df160 R11: fffff4f40db9b200 R12: 0000000000000296
R13: ffff8f168b27bc68 R14: 0000000000000001 R15: fffff4f40b231b80
FS:  00007fba091cab10(0000) GS:ffff8f18450c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2da8281508 CR3: 0000000094ad0000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9dc7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9dc7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff9dde4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff9dde5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff9ddef014>] unmap_region+0xf4/0x140
[<ffffffff9ddf1605>] do_munmap+0x2a5/0x480
[<ffffffff9ddf1845>] vm_munmap+0x65/0xb0
[<ffffffff9ddf2ad2>] SyS_munmap+0x22/0x30
[<ffffffff9e37505b>] tracesys+0xa3/0xc9
Code: 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 94 9e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #814616
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff82f1e23e
Build id: 3.10.0-1160.99.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffff82e81fa8
Build id: 3.10.0-1160.99.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff82e8201b
Build id: 3.10.0-1160.99.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff82ffa07e
Build id: 3.10.0-1160.99.1.el7.x86_64
0xae
5
arch_tlb_finish_mmu
vmlinux 0xffffffff82ffbbff
Build id: 3.10.0-1160.99.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff82ffbd33
Build id: 3.10.0-1160.99.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff830058b4
Build id: 3.10.0-1160.99.1.el7.x86_64
0xf4
8
do_munmap
vmlinux 0xffffffff83007f15
Build id: 3.10.0-1160.99.1.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffff83008155
Build id: 3.10.0-1160.99.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff83009482
Build id: 3.10.0-1160.99.1.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff835c539a
Build id: 3.10.0-1160.99.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [grafana:2525]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack vboxsf(OE) ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iosf_mbi kvm_intel kvm snd_intel8x0 irqbypass crc32_pclmul snd_ac97_codec ppdev ac97_bus snd_seq ghash_clmulni_intel snd_seq_device snd_pcm aesni_intel joydev lrw gf128mul glue_helper ablk_helper cryptd snd_timer vboxguest(OE) pcspkr i2c_piix4 parport_pc sg snd soundcore parport
video ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata e1000 crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 2525 Comm: grafana Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.99.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff8b8456d7c200 ti: ffff8b848d1f0000 task.ti: ffff8b848d1f0000
RIP: 0010:[<ffffffff82f1e23e>]  [<ffffffff82f1e23e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8b848d1f3cb0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0008000082e81b10 RCX: ffff8b849fc20348
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8b848d1f3ce8 R08: ffff8b81f62fdc00 R09: ffffffff8319c7b9
R10: ffff8b849fd1f160 R11: fffff5ed50336000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8b848d1f3c60 R15: 00007f4da0881000
FS:  00007f4d9f810f38(0000) GS:ffff8b849fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f02968f2000 CR3: 00000004075d2000 CR4: 00000000000606e0
Call Trace:
[<ffffffff82e81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff82e8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff82ffa07e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff82ffbbff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff82ffbd33>] tlb_finish_mmu+0x23/0x40
[<ffffffff830058b4>] unmap_region+0xf4/0x150
[<ffffffff83007f15>] do_munmap+0x2a5/0x480
[<ffffffff83008155>] vm_munmap+0x65/0xb0
[<ffffffff83009482>] SyS_munmap+0x22/0x30
[<ffffffff835c539a>] system_call_fastpath+0x25/0x2a
Code: 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 19 b5 83 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 1d 18 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #752569
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810fb892
Build id: 3.10.0-693.17.1.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106f508
Build id: 3.10.0-693.17.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106f5d8
Build id: 3.10.0-693.17.1.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811af0f3
Build id: 3.10.0-693.17.1.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811b0415
Build id: 3.10.0-693.17.1.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b74e4
Build id: 3.10.0-693.17.1.el7.x86_64
0xf4
7
do_munmap
vmlinux 0xffffffff811b9aa2
Build id: 3.10.0-693.17.1.el7.x86_64
0x292
8
vm_munmap
vmlinux 0xffffffff811b9ce5
Build id: 3.10.0-693.17.1.el7.x86_64
0x65
9
SyS_munmap
vmlinux 0xffffffff811badf2
Build id: 3.10.0-693.17.1.el7.x86_64
0x22
10
system_call_fastpath
vmlinux 0xffffffff816b89fd
Build id: 3.10.0-693.17.1.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 47s! [mono:116843]
Modules linked in: file_protector(OE) snapapi26(OE) tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_ftp ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel ext4 aesni_intel lrw gf128mul glue_helper ablk_helper mbcache cryptd jbd2 ppdev vmw_balloon joydev
pcspkr nfit sg libnvdimm parport_pc parport i2c_piix4 shpchp vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect crct10dif_pclmul crct10dif_common sysimgblt crc32c_intel fb_sys_fops ttm mptspi scsi_transport_spi drm mptscsih ata_piix mptbase serio_raw libata vmxnet3 i2c_core [last unloaded: edd]
CPU: 0 PID: 116843 Comm: mono Tainted: G           OEL ------------   3.10.0-693.17.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8801002bbf40 ti: ffff8803043f0000 task.ti: ffff8803043f0000
RIP: 0010:[<ffffffff810fb892>]  [<ffffffff810fb892>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8803043f3cc0  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000800008106f340 RCX: ffff8803316dbe20
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8803043f3cf8 R08: ffff88017fd8d400 R09: ffffffff81324cb9
R10: ffff88033161b9c0 R11: ffffea00000d0400 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8803043f3c70 R15: ffff8801002bbf40
FS:  00007f2be46e5780(0000) GS:ffff880331600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2bc3b4b3c6 CR3: 000000032c65e000 CR4: 00000000003607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8106f508>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106f5d8>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811af0f3>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811b0415>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b74e4>] unmap_region+0xf4/0x140
[<ffffffff811b9aa2>] do_munmap+0x292/0x470
[<ffffffff811b9ce5>] vm_munmap+0x65/0xb0
[<ffffffff811badf2>] SyS_munmap+0x22/0x30
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 76 b6 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 8e b1 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 45 b6 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #749898
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb691718a
Build id: 3.10.0-1160.76.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb687e258
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb687e2cb
Build id: 3.10.0-1160.76.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb69ee6a6
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb69f016f
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb69f0293
Build id: 3.10.0-1160.76.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffb69f9a84
Build id: 3.10.0-1160.76.1.el7.x86_64
0xf4
8
? do_sync_write
vmlinux 0xffffffffb6a4da20
Build id: 3.10.0-1160.76.1.el7.x86_64
0x90
9
do_munmap
vmlinux 0xffffffffb69fc025
Build id: 3.10.0-1160.76.1.el7.x86_64
0x2a5
10
vm_munmap
vmlinux 0xffffffffb69fc265
Build id: 3.10.0-1160.76.1.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffffb69fd562
Build id: 3.10.0-1160.76.1.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffffb6f99f92
Build id: 3.10.0-1160.76.1.el7.x86_64
0x25
13
? system_call_after_swapgs
vmlinux 0xffffffffb6f99ed5
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [java:72811]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) sunrpc ppdev sg joydev virtio_balloon parport_pc parport i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod cdrom virtio_net ata_generic pata_acpi net_failover virtio_console virtio_scsi failover virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ata_piix ttm libata drm serio_raw virtio_pci virtio_ring floppy virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 72811 Comm: java Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff956abb255280 ti: ffff956bb64f0000 task.ti: ffff956bb64f0000
RIP: 0010:[<ffffffffb691718a>]  [<ffffffffb691718a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff956bb64f3cb0  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00080000b687ddd0 RCX: ffff956d2dee0428
RDX: 0000000000000003 RSI: 0000000000000060 RDI: 0000000000000000
RBP: ffff956bb64f3ce8 R08: ffff956a763bb400 R09: ffffffffb6b869f9
R10: ffff956d2de9f160 R11: ffffdc6ccf49c000 R12: 0000000000000282
R13: 0000000000000282 R14: ffff956bb64f3c60 R15: 0000000000000000
FS:  00007fa9be97d700(0000) GS:ffff956d2de80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa960065fb0 CR3: 00000003d7e8e000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb687e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb687e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb69ee6a6>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb69f016f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb69f0293>] tlb_finish_mmu+0x23/0x30
[<ffffffffb69f9a84>] unmap_region+0xf4/0x140
[<ffffffffb6a4da20>] ? do_sync_write+0x90/0xe0
[<ffffffffb69fc025>] do_munmap+0x2a5/0x480
[<ffffffffb69fc265>] vm_munmap+0x65/0xb0
[<ffffffffb69fd562>] SyS_munmap+0x22/0x30
[<ffffffffb6f99f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb6f99ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 be 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 55 b7 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #506895
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
0xf4
7
do_munmap
vmlinux 0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
0x292
8
vm_munmap
vmlinux 0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
0x65
9
SyS_munmap
vmlinux 0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
0x22
10
tracesys
vmlinux 0xffffffff816b51d2
Build id: 3.10.0-693.el7.x86_64
0xdd
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [java:114787]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371
snd_rawmidi snd_ac97_codec ac97_bus snd_seq ppdev snd_seq_device vmw_balloon snd_pcm pcspkr joydev snd_timer snd i2c_piix4 soundcore vmw_vmci sg shpchp nfit libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libata mptspi scsi_transport_spi mptscsih serio_raw mptbase e1000 i2c_core
CPU: 2 PID: 114787 Comm: java Tainted: G             L ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff88034efa9fa0 ti: ffff88025ef68000 task.ti: ffff88025ef68000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff88025ef6bcf0  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000800008106e4a0 RCX: ffff8803702daf48
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff88025ef6bd28 R08: ffff88017afb3800 R09: ffffffff813227d9
R10: ffff880370299c80 R11: ffffea000c0dd000 R12: 0000000000000286
R13: 0000000000000286 R14: ffff88025ef6bca0 R15: 00007f5559340000
FS:  00007f5598269700(0000) GS:ffff880370280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f55443bd1b0 CR3: 000000019a66d000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000107553cc0 ffff88034d0be108 ffff88034d0bddc0 0000000000000000
ffffffffffffffff ffff88034d0be108 ffff88014ef49d88 ffff88025ef6bd78
ffffffff8106e668 ffff88034d0bddc0 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b51d2>] tracesys+0xdd/0xe2
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #413015
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9f711a0a
Build id: 3.10.0-957.12.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9f67a628
Build id: 3.10.0-957.12.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9f67a698
Build id: 3.10.0-957.12.2.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffff9f7e4607
Build id: 3.10.0-957.12.2.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff9f7e5f05
Build id: 3.10.0-957.12.2.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff9f7ef5a4
Build id: 3.10.0-957.12.2.el7.x86_64
0xf4
7
do_munmap
vmlinux 0xffffffff9f7f1b95
Build id: 3.10.0-957.12.2.el7.x86_64
0x2a5
8
vm_munmap
vmlinux 0xffffffff9f7f1dd5
Build id: 3.10.0-957.12.2.el7.x86_64
0x65
9
SyS_munmap
vmlinux 0xffffffff9f7f3062
Build id: 3.10.0-957.12.2.el7.x86_64
0x22
10
system_call_fastpath
vmlinux 0xffffffff9fd75ddb
Build id: 3.10.0-957.12.2.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [mysqld:368]
Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables vmw_vsock_vmci_transport vsock iptable_filter dm_mirror dm_region_hash dm_log dm_mod ppdev vmw_balloon sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc sch_fq_codel ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul
crct10dif_common crc32c_intel vmwgfx serio_raw drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm vmxnet3 drm ata_piix mptspi scsi_transport_spi mptscsih libata mptbase drm_panel_orientation_quirks floppy
CPU: 8 PID: 368 Comm: mysqld Tainted: G        W    L ------------   3.10.0-957.12.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9af268430000 ti: ffff9b048f460000 task.ti: ffff9b048f460000
RIP: 0010:[<ffffffff9f711a0a>]  [<ffffffff9f711a0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9b048f463cb8  EFLAGS: 00000202
RAX: 000000000000000a RBX: 000000000001b700 RCX: ffff9b059e8a06c8
RDX: 000000000000000a RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff9b048f463cf0 R08: ffff9aef1b057000 R09: ffffffff9f975d69
R10: ffff9b059e81f160 R11: ffffd2529e33c000 R12: ffff9b048f463c68
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000200
FS:  00007fceb437a700(0000) GS:ffff9b059e800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3e88c15000 CR3: 000000064c71a000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9f67a628>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9f67a698>] flush_tlb_mm_range+0x68/0x140
[<ffffffff9f7e4607>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff9f7e5f05>] tlb_finish_mmu+0x55/0x60
[<ffffffff9f7ef5a4>] unmap_region+0xf4/0x140
[<ffffffff9f7f1b95>] do_munmap+0x2a5/0x480
[<ffffffff9f7f1dd5>] vm_munmap+0x65/0xb0
[<ffffffff9f7f3062>] SyS_munmap+0x22/0x30
[<ffffffff9fd75ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 7e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 bc 34 a0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #398066
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb8d1722a
Build id: 3.10.0-1160.62.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb8c7e258
Build id: 3.10.0-1160.62.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb8c7e2cb
Build id: 3.10.0-1160.62.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffffb8dee666
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffffb8df012f
Build id: 3.10.0-1160.62.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffffb8df0253
Build id: 3.10.0-1160.62.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffffb8df9a44
Build id: 3.10.0-1160.62.1.el7.x86_64
0xf4
8
do_munmap
vmlinux 0xffffffffb8dfc035
Build id: 3.10.0-1160.62.1.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffffb8dfc275
Build id: 3.10.0-1160.62.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffffb8dfd572
Build id: 3.10.0-1160.62.1.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffffb9399f92
Build id: 3.10.0-1160.62.1.el7.x86_64
0x25
12
? system_call_after_swapgs
vmlinux 0xffffffffb9399ed5
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [Wt-780:4312]
Modules linked in: vboxsf(OE) veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ext4 mbcache jbd2 binfmt_misc overlay(T) nf_nat_ftp nf_conntrack_ftp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_intel8x0 snd_ac97_codec ac97_bus snd_seq ppdev snd_seq_device joydev snd_pcm snd_timer pcspkr snd vboxguest(OE) sg parport_pc i2c_piix4 soundcore video parport
sch_fq_codel ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common sr_mod vmwgfx cdrom ata_generic drm_kms_helper pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm libahci drm serio_raw ata_piix e1000 crc32c_intel libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 4312 Comm: Wt-780 Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff9ca28ab43180 ti: ffff9ca51d84c000 task.ti: ffff9ca51d84c000
RIP: 0010:[<ffffffffb8d1722a>]  [<ffffffffb8d1722a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9ca51d84fcb0  EFLAGS: 00010202
RAX: 0000000000000001 RBX: ffffffffb8c7ddd0 RCX: ffff9ca5dfc9fc10
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff9ca51d84fce8 R08: ffff9ca33fd52000 R09: ffffffffb8f869d9
R10: ffff9ca5dfc1f160 R11: ffffe8570d76ff00 R12: ffff9ca51d84fc80
R13: ffffffffb8f8692f R14: ffff9ca33fd52d08 R15: ffff9ca28ab43180
FS:  00007f0e061e2700(0000) GS:ffff9ca5dfc00000(0000) knlGS:0000000b226f0000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000b22b57006 CR3: 0000000407f9c000 CR4: 00000000000006f0
Call Trace:
[<ffffffffb8c7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb8c7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb8dee666>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb8df012f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb8df0253>] tlb_finish_mmu+0x23/0x30
[<ffffffffb8df9a44>] unmap_region+0xf4/0x140
[<ffffffffb8dfc035>] do_munmap+0x2a5/0x480
[<ffffffffb8dfc275>] vm_munmap+0x65/0xb0
[<ffffffffb8dfd572>] SyS_munmap+0x22/0x30
[<ffffffffb9399f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb9399ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 95 b9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #313774
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb8b1181a
Build id: 3.10.0-957.1.3.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb8a7a598
Build id: 3.10.0-957.1.3.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb8a7a608
Build id: 3.10.0-957.1.3.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffffb8be42e7
Build id: 3.10.0-957.1.3.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffb8be5be5
Build id: 3.10.0-957.1.3.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffb8bef284
Build id: 3.10.0-957.1.3.el7.x86_64
0xf4
7
do_munmap
vmlinux 0xffffffffb8bf1875
Build id: 3.10.0-957.1.3.el7.x86_64
0x2a5
8
vm_munmap
vmlinux 0xffffffffb8bf1ab5
Build id: 3.10.0-957.1.3.el7.x86_64
0x65
9
SyS_munmap
vmlinux 0xffffffffb8bf2d42
Build id: 3.10.0-957.1.3.el7.x86_64
0x22
10
system_call_fastpath
vmlinux 0xffffffffb9174ddb
Build id: 3.10.0-957.1.3.el7.x86_64
0x22
11
? system_call_after_swapgs
vmlinux 0xffffffffb9174d21
Build id: 3.10.0-957.1.3.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#38 stuck for 23s! [sanlock:52234]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver vhost_net vhost macvtap macvlan ebt_arp tun nfsv3 nfs_acl nfs lockd grace fscache bnx2i libiscsi devlink bnx2fc cnic uio fcoe libfcoe libfc scsi_transport_fc scsi_tgt ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack geneve ip6_udp_tunnel udp_tunnel openvswitch ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter softdog dcdbas dm_service_time dm_multipath amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass joydev
pcspkr sg sp5100_tco k10temp fam15h_power i2c_piix4 ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_cpufreq bridge bonding ip_tables ext4 mbcache jbd2 dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c uas usb_storage sd_mod sr_mod crc_t10dif crct10dif_generic cdrom mgag200 i2c_algo_bit drm_kms_helper syscopyarea crct10dif_pclmul crct10dif_common sysfillrect crc32_pclmul sysimgblt crc32c_intel fb_sys_fops ttm ghash_clmulni_intel aesni_intel lrw gf128mul drm glue_helper ablk_helper cryptd serio_raw scsi_transport_iscsi megaraid_sas drm_panel_orientation_quirks ahci libahci 8021q garp mrp libata stp llc bnx2 sunrpc dm_mirror dm_region_hash dm_log dm_mod
CPU: 38 PID: 52234 Comm: sanlock Kdump: loaded Tainted: G             L ------------   3.10.0-957.1.3.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R815/04Y8PT, BIOS 3.4.1 05/04/2018
task: ffff8a38bbda9040 ti: ffff8a38b3678000 task.ti: ffff8a38b3678000
RIP: 0010:[<ffffffffb8b1181a>]  [<ffffffffb8b1181a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8a38b367bcb8  EFLAGS: 00000202
RAX: 0000000000000023 RBX: 000000000001b700 RCX: ffff8a38bfce0e50
RDX: 0000000000000023 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff8a38b367bcf0 R08: ffff8a315fc0c000 R09: ffffffffb8d75839
R10: ffff8a38bfd9f160 R11: ffffeaa31fd1c000 R12: ffff8a38b367bc68
R13: ffffeaa31feb4d80 R14: 0000000000000001 R15: ffffeaa31fbf7b00
FS:  00007f658f441700(0000) GS:ffff8a38bfd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe9decee140 CR3: 0000002ff8b3e000 CR4: 00000000000407e0
Call Trace:
[<ffffffffb8a7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb8a7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb8be42e7>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffb8be5be5>] tlb_finish_mmu+0x55/0x60
[<ffffffffb8bef284>] unmap_region+0xf4/0x140
[<ffffffffb8bf1875>] do_munmap+0x2a5/0x480
[<ffffffffb8bf1ab5>] vm_munmap+0x65/0xb0
[<ffffffffb8bf2d42>] SyS_munmap+0x22/0x30
[<ffffffffb9174ddb>] system_call_fastpath+0x22/0x27
[<ffffffffb9174d21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 6e 82 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 74 b9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 3d 82 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #298467
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8691173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8687a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8687a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffff869e4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffff869e5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff869ef014
Build id: 3.10.0-957.el7.x86_64
0xf4
7
do_munmap
vmlinux 0xffffffff869f1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
8
vm_munmap
vmlinux 0xffffffff869f1845
Build id: 3.10.0-957.el7.x86_64
0x65
9
SyS_munmap
vmlinux 0xffffffff869f2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
10
system_call_fastpath
vmlinux 0xffffffff86f74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [360entclient:21862]
Modules linked in: kbase(OE) binfmt_misc udp_diag tcp_diag inet_diag fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel virtio_balloon aesni_intel parport_pc lrw parport gf128mul glue_helper i2c_piix4 ablk_helper cryptd joydev pcspkr ip_tables ext4 mbcache jbd2 virtio_console virtio_blk
virtio_net ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm floppy ata_piix drm crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw drm_panel_orientation_quirks virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod [last unloaded: kbase]
CPU: 7 PID: 21862 Comm: 360entclient Kdump: loaded Tainted: G           OEL ------------   3.10.0-957.el7.x86_64 #1
Hardware name: Fedora Project OpenStack Nova, BIOS 0.5.1 01/01/2011
task: ffff9b5fb861b0c0 ti: ffff9b5f897c0000 task.ti: ffff9b5f897c0000
RIP: 0010:[<ffffffff8691173a>]  [<ffffffff8691173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9b5f897c3cb8  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 00000000000000fc RCX: ffff9b5fbf3a03c0
RDX: 0000000000000006 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff9b5f897c3cf0 R08: ffff9b51294f4800 R09: ffffffff86b755c9
R10: ffff9b5fbf3df160 R11: fffffa0bfea4ac00 R12: 0000000000000296
R13: ffff9b5f897c3c68 R14: 0000000000000001 R15: fffffa0bd7a57180
FS:  00007f8205730700(0000) GS:ffff9b5fbf3c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8207fd7438 CR3: 0000000ff82f4000 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8687a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8687a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff869e4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffff869e5975>] tlb_finish_mmu+0x55/0x60
[<ffffffff869ef014>] unmap_region+0xf4/0x140
[<ffffffff869f1605>] do_munmap+0x2a5/0x480
[<ffffffff869f1845>] vm_munmap+0x65/0xb0
[<ffffffff869f2ad2>] SyS_munmap+0x22/0x30
[<ffffffff86f74ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 54 87 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #128414
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
0xf4
7
do_munmap
vmlinux 0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
0x292
8
vm_munmap
vmlinux 0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
0x65
9
SyS_munmap
vmlinux 0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
0x22
10
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [java:13244]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi coretemp snd_ac97_codec ac97_bus crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device aesni_intel snd_pcm lrw gf128mul glue_helper snd_timer ablk_helper snd cryptd soundcore pcspkr ppdev sg vmw_balloon joydev parport_pc parport shpchp i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libata mptspi serio_raw scsi_transport_spi e1000 mptscsih mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 13244 Comm: java Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2012
task: ffff8801bb2a8000 ti: ffff880044750000 task.ti: ffff880044750000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff880044753cf0  EFLAGS: 00000202
RAX: 0000000000000007 RBX: ffffffff8106e4a0 RCX: ffff8802367da7c8
RDX: 0000000000000007 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff880044753d28 R08: ffff88017ceab400 R09: ffffffff813227d9
R10: ffff880236659c80 R11: ffffea0001111a00 R12: 0000000000000282
R13: ffff880044753ca0 R14: ffff880044753c70 R15: ffff8801bb2a8000
FS:  00007f1a2cb1c700(0000) GS:ffff880236640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f592b1ba000 CR3: 0000000208f5f000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001bb2a8000 ffff88003541a288 ffff880035419f40 0000000000000000
ffffffffffffffff ffff88003541a288 ffff88020f6ce0d8 ffff880044753d78
ffffffff8106e668 ffff880035419f40 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #110971
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbab1173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbaa7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffbaa7a608
Build id: 3.10.0-957.el7.x86_64
0x68
4
tlb_flush_mmu.part.76
vmlinux 0xffffffffbabe4077
Build id: 3.10.0-957.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffbabe5975
Build id: 3.10.0-957.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffbabef014
Build id: 3.10.0-957.el7.x86_64
0xf4
7
do_munmap
vmlinux 0xffffffffbabf1605
Build id: 3.10.0-957.el7.x86_64
0x2a5
8
vm_munmap
vmlinux 0xffffffffbabf1845
Build id: 3.10.0-957.el7.x86_64
0x65
9
SyS_munmap
vmlinux 0xffffffffbabf2ad2
Build id: 3.10.0-957.el7.x86_64
0x22
10
system_call_fastpath
vmlinux 0xffffffffbb174ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [mysqld:21785]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event coretemp iosf_mbi kvm_intel ppdev kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul vmw_balloon glue_helper ablk_helper cryptd joydev pcspkr snd_ens1371 snd_rawmidi snd_ac97_codec
ac97_bus snd_seq sg snd_seq_device snd_pcm snd_timer snd soundcore parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ata_generic pata_acpi serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci ata_piix drm e1000 libata vmw_pvscsi drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 21785 Comm: mysqld Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff8bc2d6516180 ti: ffff8bc41da0c000 task.ti: ffff8bc41da0c000
RIP: 0010:[<ffffffffbab1173a>]  [<ffffffffbab1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8bc41da0fcb8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8bc4758204b8
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8bc41da0fcf0 R08: ffff8bc3bc8db800 R09: ffffffffbad755c9
R10: ffff8bc47589f160 R11: ffffef620824cc00 R12: 0000000000000296
R13: ffff8bc41da0fc68 R14: 0000000000000000 R15: 0000000000000200
FS:  00007f64b5e1b700(0000) GS:ffff8bc475880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f632699a000 CR3: 00000001baf74000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffbaa7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbaa7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffffbabe4077>] tlb_flush_mmu.part.76+0x37/0xe0
[<ffffffffbabe5975>] tlb_finish_mmu+0x55/0x60
[<ffffffffbabef014>] unmap_region+0xf4/0x140
[<ffffffffbabf1605>] do_munmap+0x2a5/0x480
[<ffffffffbabf1845>] vm_munmap+0x65/0xb0
[<ffffffffbabf2ad2>] SyS_munmap+0x22/0x30
[<ffffffffbb174ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 74 bb f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #24260
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff99716a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9967e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9967e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff997ec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff997edb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff997edc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff997f7474
Build id: 3.10.0-1062.12.1.el7.x86_64
0xf4
8
do_munmap
vmlinux 0xffffffff997f9a75
Build id: 3.10.0-1062.12.1.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffff997f9cb5
Build id: 3.10.0-1062.12.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff997faf52
Build id: 3.10.0-1062.12.1.el7.x86_64
0x22
11
tracesys
vmlinux 0xffffffff99d8e172
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [aux:33659]
Modules linked in: tcp_lp fuse veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event xfs ppdev vmw_balloon snd_ens1371 snd_rawmidi crc32_pclmul snd_ac97_codec libcrc32c ghash_clmulni_intel ac97_bus snd_seq aesni_intel snd_seq_device lrw gf128mul snd_pcm glue_helper snd_timer snd joydev ablk_helper pcspkr soundcore cryptd sg parport_pc parport vmw_vmci i2c_piix4 ip_tables ext4 mbcache
jbd2 sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix nfit drm crct10dif_pclmul crct10dif_common crc32c_intel mptspi libnvdimm libata serio_raw scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks
CPU: 11 PID: 33659 Comm: aux Kdump: loaded Tainted: G               ------------ T 3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff92cafbac62a0 ti: ffff92c11797c000 task.ti: ffff92c11797c000
RIP: 0010:[<ffffffff99716a7a>]  [<ffffffff99716a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff92c11797fcb0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800009967e2b0 RCX: ffff92cd886a0790
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff92c11797fce8 R08: ffff92c1ec665c00 R09: ffffffff99981839
R10: ffff92cd888df160 R11: ffffc3dae7d18400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff92c11797fc60 R15: 00007feba6634000
FS:  00007feba76cf700(0000) GS:ffff92cd888c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdbca606b80 CR3: 000000082b678000 CR4: 00000000003407e0
Call Trace:
[<ffffffff9967e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9967e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff997ec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff997edb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff997edc83>] tlb_finish_mmu+0x23/0x30
[<ffffffff997f7474>] unmap_region+0xf4/0x140
[<ffffffff997f9a75>] do_munmap+0x2a5/0x480
[<ffffffff997f9cb5>] vm_munmap+0x65/0xb0
[<ffffffff997faf52>] SyS_munmap+0x22/0x30
[<ffffffff99d8e172>] tracesys+0xa6/0xcc
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 34 9a f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #14524
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9e51761a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9e47e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9e47e96b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9e5ede36
Build id: 3.10.0-1127.19.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9e5ef8ff
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9e5efa23
Build id: 3.10.0-1127.19.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff9e5f9234
Build id: 3.10.0-1127.19.1.el7.x86_64
0xf4
8
do_munmap
vmlinux 0xffffffff9e5fb825
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffff9e5fba65
Build id: 3.10.0-1127.19.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff9e5fcd02
Build id: 3.10.0-1127.19.1.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff9eb92ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [5_scheduler:2809]
Modules linked in: xt_set ipt_rpfilter xt_multiport ip_set_hash_ip ip_set_hash_net vxlan ip6_udp_tunnel udp_tunnel ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs xt_comment xt_mark veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log
dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec iosf_mbi ac97_bus crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device snd_pcm ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon snd_timer snd joydev pcspkr sg soundcore i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel nfit ata_piix libnvdimm libata serio_raw mptspi scsi_transport_spi mptscsih e1000 mptbase drm_panel_orientation_quirks
CPU: 1 PID: 2809 Comm: 5_scheduler Tainted: G               ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff964cbe0062a0 ti: ffff964cbfa20000 task.ti: ffff964cbfa20000
RIP: 0010:[<ffffffff9e51761a>]  [<ffffffff9e51761a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff964cbfa23cb0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800009e47e730 RCX: ffff964cc02a0600
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff964cbfa23ce8 R08: ffff964bbb9cb000 R09: ffffffff9e7851b9
R10: ffff964cc025f160 R11: ffffccec02757800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff964cbfa23c60 R15: 00007f8ef99bf000
FS:  00007f8ffc433700(0000) GS:ffff964cc0240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8eff4b3000 CR3: 000000027f91a000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9e47e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9e47e96b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9e5ede36>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9e5ef8ff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9e5efa23>] tlb_finish_mmu+0x23/0x30
[<ffffffff9e5f9234>] unmap_region+0xf4/0x140
[<ffffffff9e5fb825>] do_munmap+0x2a5/0x480
[<ffffffff9e5fba65>] vm_munmap+0x65/0xb0
[<ffffffff9e5fcd02>] SyS_munmap+0x22/0x30
[<ffffffff9eb92ed2>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 15 9f f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #10220
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff84d16a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff84c7e478
Build id: 3.10.0-1062.9.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff84c7e4eb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff84dec026
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff84dedaef
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff84dedc13
Build id: 3.10.0-1062.9.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff84df7404
Build id: 3.10.0-1062.9.1.el7.x86_64
0xf4
8
do_munmap
vmlinux 0xffffffff84df9a05
Build id: 3.10.0-1062.9.1.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffff84df9c45
Build id: 3.10.0-1062.9.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff84dfaee2
Build id: 3.10.0-1062.9.1.el7.x86_64
0x22
11
tracesys
vmlinux 0xffffffff8538e172
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [C2 CompilerThre:22824]
Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon sb_edac iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel
lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nfit crct10dif_pclmul libata mptspi crct10dif_common crc32c_intel libnvdimm scsi_transport_spi serio_raw mptscsih mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 22824 Comm: C2 CompilerThre Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff97c6c23fa0e0 ti: ffff97c6c4e80000 task.ti: ffff97c6c4e80000
RIP: 0010:[<ffffffff84d16a6a>]  [<ffffffff84d16a6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff97c6c4e83cb0  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffffff84c7e2b0 RCX: ffff97c6ffc606c8
RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff97c6c4e83ce8 R08: ffff97c0344b2400 R09: ffffffff84f817e9
R10: ffff97c6ffd9f160 R11: ffffcd858db3f800 R12: ffff97c6c4e83c80
R13: ffffffff84f8173f R14: ffff97c6c23fa0e0 R15: 0000000000000000
FS:  00007f4e29c1e700(0000) GS:ffff97c6ffd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4dda3a6010 CR3: 00000007fbe3e000 CR4: 00000000001607e0
Call Trace:
[<ffffffff84c7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff84c7e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff84dec026>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff84dedaef>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff84dedc13>] tlb_finish_mmu+0x23/0x30
[<ffffffff84df7404>] unmap_region+0xf4/0x140
[<ffffffff84df9a05>] do_munmap+0x2a5/0x480
[<ffffffff84df9c45>] vm_munmap+0x65/0xb0
[<ffffffff84dfaee2>] SyS_munmap+0x22/0x30
[<ffffffff8538e172>] tracesys+0xa6/0xcc
Code: 48 63 35 fe 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 94 85 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1242
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9c116a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9c07e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9c07e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
tlb_flush_mmu_tlbonly
vmlinux 0xffffffff9c1ec096
Build id: 3.10.0-1062.12.1.el7.x86_64
0xa6
5
arch_tlb_finish_mmu
vmlinux 0xffffffff9c1edb5f
Build id: 3.10.0-1062.12.1.el7.x86_64
0x3f
6
tlb_finish_mmu
vmlinux 0xffffffff9c1edc83
Build id: 3.10.0-1062.12.1.el7.x86_64
0x23
7
unmap_region
vmlinux 0xffffffff9c1f7474
Build id: 3.10.0-1062.12.1.el7.x86_64
0xf4
8
do_munmap
vmlinux 0xffffffff9c1f9a75
Build id: 3.10.0-1062.12.1.el7.x86_64
0x2a5
9
vm_munmap
vmlinux 0xffffffff9c1f9cb5
Build id: 3.10.0-1062.12.1.el7.x86_64
0x65
10
SyS_munmap
vmlinux 0xffffffff9c1faf52
Build id: 3.10.0-1062.12.1.el7.x86_64
0x22
11
system_call_fastpath
vmlinux 0xffffffff9c78dede
Build id: 3.10.0-1062.12.1.el7.x86_64
0x25
12
? system_call_after_swapgs
vmlinux 0xffffffff9c78de21
Build id: 3.10.0-1062.12.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [java:11777]
Modules linked in: xt_nat veth dcdbas nf_conntrack_netlink nfnetlink br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay vmw_vsock_vmci_transport vsock xt_comment xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ext4 mbcache jbd2 sb_edac ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul parport_pc glue_helper ablk_helper sg vmw_vmci parport cryptd pcspkr i2c_piix4 joydev binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi crct10dif_pclmul libata
scsi_transport_spi crct10dif_common nfit crc32c_intel mptscsih e1000 serio_raw libnvdimm mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 11777 Comm: java Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8c32c177b150 ti: ffff8c2a24208000 task.ti: ffff8c2a24208000
RIP: 0010:[<ffffffff9c116a7a>]  [<ffffffff9c116a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8c2a2420bcb0  EFLAGS: 00000202
RAX: 0000000000000012 RBX: ffffffff9c07e2b0 RCX: ffff8c32cd4a06a0
RDX: 0000000000000012 RSI: 0000000000000014 RDI: 0000000000000000
RBP: ffff8c2a2420bce8 R08: ffff8c27acd52000 R09: ffffffff9c381839
R10: ffff8c32cd15f160 R11: ffffe39af1a9c800 R12: 0000000000000286
R13: ffff8c2a2420bc60 R14: 0000000000000000 R15: ffff8c32c177b150
FS:  00007f008cfa5700(0000) GS:ffff8c32cd140000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f000b584004 CR3: 00000008f37de000 CR4: 00000000000407e0
Call Trace:
[<ffffffff9c07e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9c07e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9c1ec096>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9c1edb5f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9c1edc83>] tlb_finish_mmu+0x23/0x30
[<ffffffff9c1f7474>] unmap_region+0xf4/0x140
[<ffffffff9c1f9a75>] do_munmap+0x2a5/0x480
[<ffffffff9c1f9cb5>] vm_munmap+0x65/0xb0
[<ffffffff9c1faf52>] SyS_munmap+0x22/0x30
[<ffffffff9c78dede>] system_call_fastpath+0x25/0x2a
[<ffffffff9c78de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 d4 9c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89