Info

Permalink
Function
smp_call_function_many
First occurrence
2022-07-30
Last occurrence
2026-07-07
State
NEW
Components
kernel
Quality
-16

Statistics

Operating system Count
CentOS 7 19
Architecture Count
x86_64 19
Related packages Count
kernel 19
 0:3.10.0-1062.12.1.el7 6
 0:3.10.0-1160.81.1.el7 4
 0:3.10.0-1160.59.1.el7 4
 0:3.10.0-1062.1.2.el7 1
 0:3.10.0-1160.90.1.el7 1
 0:3.10.0-1160.88.1.el7 1
 0:3.10.0-1062.el7 1
 0:3.10.0-1160.108.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1417438
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff915170da
Build id: 3.10.0-1160.59.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9147e258
Build id: 3.10.0-1160.59.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9147e2cb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x6b
4
do_huge_pmd_numa_page
vmlinux 0xffffffff916370a6
Build id: 3.10.0-1160.59.1.el7.x86_64
0x256
5
handle_mm_fault
vmlinux 0xffffffff915f663a
Build id: 3.10.0-1160.59.1.el7.x86_64
0xaba
6
? schedule_hrtimeout_range_clock
vmlinux 0xffffffff91b8b7a6
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa6
7
__do_page_fault
vmlinux 0xffffffff91b94653
Build id: 3.10.0-1160.59.1.el7.x86_64
0x213
8
do_page_fault
vmlinux 0xffffffff91b94975
Build id: 3.10.0-1160.59.1.el7.x86_64
0x35
9
page_fault
vmlinux 0xffffffff91b90778
Build id: 3.10.0-1160.59.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [java:8111]
Modules linked in: veth xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 nfit libnvdimm coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi scsi_transport_spi libata crct10dif_pclmul mptscsih crct10dif_common crc32c_intel serio_raw mptbase vmxnet3 drm_panel_orientation_quirks
floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 4 PID: 8111 Comm: java Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9ec6844c9080 ti: ffff9ec6a9fc0000 task.ti: ffff9ec6a9fc0000
RIP: 0010:[<ffffffff915170da>]  [<ffffffff915170da>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff9ec6a9fc3ca8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9ebf2fc206a0
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff9ec6a9fc3ce0 R08: ffff9ebfbfc09000 R09: ffffffff91786c09
R10: ffff9ec32fc1f160 R11: fffff89c17b62a00 R12: 0000000000000296
R13: ffff9ec6a9fc3c58 R14: 0000000100000004 R15: ffffffff91628ac9
FS:  00007fd5ee3e5700(0000) GS:ffff9ec32fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000100991ab0 CR3: 000000102f336000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9147e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9147e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff916370a6>] do_huge_pmd_numa_page+0x256/0x4c0
[<ffffffff915f663a>] handle_mm_fault+0xaba/0xfb0
[<ffffffff91b8b7a6>] ? schedule_hrtimeout_range_clock+0xa6/0x150
[<ffffffff91b94653>] __do_page_fault+0x213/0x500
[<ffffffff91b94975>] do_page_fault+0x35/0x90
[<ffffffff91b90778>] page_fault+0x28/0x30
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 15 92 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1013888
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb8d1e23e
Build id: 3.10.0-1160.108.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffb8c81fa8
Build id: 3.10.0-1160.108.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb8c8201b
Build id: 3.10.0-1160.108.1.el7.x86_64
0x6b
4
do_huge_pmd_numa_page
vmlinux 0xffffffffb8e43fa6
Build id: 3.10.0-1160.108.1.el7.x86_64
0x256
5
handle_mm_fault
vmlinux 0xffffffffb8e02222
Build id: 3.10.0-1160.108.1.el7.x86_64
0xad2
6
__do_page_fault
vmlinux 0xffffffffb93bf7d3
Build id: 3.10.0-1160.108.1.el7.x86_64
0x213
7
do_page_fault
vmlinux 0xffffffffb93bfb05
Build id: 3.10.0-1160.108.1.el7.x86_64
0x35
8
page_fault
vmlinux 0xffffffffb93bb7b8
Build id: 3.10.0-1160.108.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [Catalina-utilit:4845]
Modules linked in: bmsensor(OE) ip6table_filter ip6_tables iptable_filter tcp_diag inet_diag binfmt_misc tmhook(OE) vmw_vsock_vmci_transport vsock crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd ppdev pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit ata_piix crct10dif_pclmul drm crct10dif_common mptspi crc32c_intel libnvdimm scsi_transport_spi libata serio_raw mptscsih mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: bmsensor]
CPU: 15 PID: 4845 Comm: Catalina-utilit Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.108.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff91bf76dd0000 ti: ffff91bf76a88000 task.ti: ffff91bf76a88000
RIP: 0010:[<ffffffffb8d1e23e>]  [<ffffffffb8d1e23e>] smp_call_function_many+0x20e/0x270
RSP: 0000:ffff91bf76a8bca8  EFLAGS: 00000202
RAX: 000000000000000c RBX: 00000000000000fc RCX: ffff91bf7fc20968
RDX: 000000000000000c RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff91bf76a8bce0 R08: ffff91bdfb101400 R09: ffffffffb8f9c7b9
R10: ffff91bf7fd9f160 R11: ffffe4308f18dc00 R12: 0000000000000296
R13: ffff91bf76a8bc58 R14: 000000000000028e R15: ffffffffb8cecdc7
FS:  00007f95fffff700(0000) GS:ffff91bf7fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9484002580 CR3: 000000023750a000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffb8c81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb8c8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb8e43fa6>] do_huge_pmd_numa_page+0x256/0x4c0
[<ffffffffb8e02222>] handle_mm_fault+0xad2/0x1190
[<ffffffffb93bf7d3>] __do_page_fault+0x213/0x510
[<ffffffffb93bfb05>] do_page_fault+0x35/0x90
[<ffffffffb93bb7b8>] page_fault+0x28/0x30
Code: 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 80 19 95 b9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 5d 18 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #892945
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff94516a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9447e478
Build id: 3.10.0-1062.12.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9447e4eb
Build id: 3.10.0-1062.12.1.el7.x86_64
0x6b
4
do_huge_pmd_numa_page
vmlinux 0xffffffff946342c6
Build id: 3.10.0-1062.12.1.el7.x86_64
0x256
5
handle_mm_fault
vmlinux 0xffffffff945f4417
Build id: 3.10.0-1062.12.1.el7.x86_64
0x877
6
__do_page_fault
vmlinux 0xffffffff94b88653
Build id: 3.10.0-1062.12.1.el7.x86_64
0x213
7
do_page_fault
vmlinux 0xffffffff94b88975
Build id: 3.10.0-1062.12.1.el7.x86_64
0x35
8
? error_swapgs
vmlinux 0xffffffff94b84ac9
Build id: 3.10.0-1062.12.1.el7.x86_64
0xaa
9
page_fault
vmlinux 0xffffffff94b84778
Build id: 3.10.0-1062.12.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [javamelody siie:21385]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache fuse sunrpc ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper nfit pcspkr ablk_helper sg joydev cryptd libnvdimm parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi scsi_transport_spi crct10dif_pclmul mptscsih crct10dif_common crc32c_intel serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 21385 Comm: javamelody siie Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffff909af5ca1070 ti: ffff909e0b718000 task.ti: ffff909e0b718000
RIP: 0010:[<ffffffff94516a7a>]  [<ffffffff94516a7a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff909e0b71bcc8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000000000001b800 RCX: ffff909e13420800
RDX: 0000000000000000 RSI: 000000000000000e RDI: 0000000000000000
RBP: ffff909e0b71bd00 R08: ffff909e9fc0a000 R09: ffffffff94781839
R10: ffff90a13351f160 R11: ffffefb7a202bc00 R12: ffff909e0b71bc40
R13: 0000000000000282 R14: 0000000000000010 R15: ffffffff9445e493
FS:  00007fc70b9f9700(0000) GS:ffff90a133500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000d3b9f12c CR3: 000000035044a000 CR4: 00000000000407e0
Call Trace:
[<ffffffff9447e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9447e4eb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff946342c6>] do_huge_pmd_numa_page+0x256/0x4b0
[<ffffffff945f4417>] handle_mm_fault+0x877/0x9b0
[<ffffffff94b88653>] __do_page_fault+0x213/0x500
[<ffffffff94b88975>] do_page_fault+0x35/0x90
[<ffffffff94b84ac9>] ? error_swapgs+0xaa/0xc0
[<ffffffff94b84778>] page_fault+0x28/0x30
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 14 95 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #851720
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff957170da
Build id: 3.10.0-1160.59.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9567e258
Build id: 3.10.0-1160.59.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9567e2cb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x6b
4
do_huge_pmd_numa_page
vmlinux 0xffffffff958370a6
Build id: 3.10.0-1160.59.1.el7.x86_64
0x256
5
handle_mm_fault
vmlinux 0xffffffff957f663a
Build id: 3.10.0-1160.59.1.el7.x86_64
0xaba
6
? __check_object_size
vmlinux 0xffffffff9584b13a
Build id: 3.10.0-1160.59.1.el7.x86_64
0x1ca
7
__do_page_fault
vmlinux 0xffffffff95d94653
Build id: 3.10.0-1160.59.1.el7.x86_64
0x213
8
do_page_fault
vmlinux 0xffffffff95d94975
Build id: 3.10.0-1160.59.1.el7.x86_64
0x35
9
page_fault
vmlinux 0xffffffff95d90778
Build id: 3.10.0-1160.59.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#19 stuck for 24s! [geth:6239]
Modules linked in: tcp_diag udp_diag inet_diag xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper vmw_balloon cryptd sg pcspkr joydev vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci nfit ata_piix libahci drm
libnvdimm crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 19 PID: 6239 Comm: geth Tainted: G        W    L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
task: ffff8aeafeb02100 ti: ffff8aeafa52c000 task.ti: ffff8aeafa52c000
RIP: 0010:[<ffffffff957170da>]  [<ffffffff957170da>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8aeafa52fca8  EFLAGS: 00000202
RAX: 0000000000000010 RBX: 00000000000000fc RCX: ffff8aeaff220930
RDX: 0000000000000010 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff8aeafa52fce0 R08: ffff8adb99421400 R09: ffffffff95986c09
R10: ffff8aeaff2df160 R11: ffffebbb4148ba00 R12: 0000000000000296
R13: ffff8aeafa52fc58 R14: ffff8aeafa52fc68 R15: ffff8adb4c3131e8
FS:  00007f8a53fff700(0000) GS:ffff8aeaff2c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0139adc70 CR3: 0000000f9acba000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff9567e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9567e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff958370a6>] do_huge_pmd_numa_page+0x256/0x4c0
[<ffffffff957f663a>] handle_mm_fault+0xaba/0xfb0
[<ffffffff9584b13a>] ? __check_object_size+0x1ca/0x250
[<ffffffff95d94653>] __do_page_fault+0x213/0x500
[<ffffffff95d94975>] do_page_fault+0x35/0x90
[<ffffffff95d90778>] page_fault+0x28/0x30
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 35 96 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #818950
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9f51e07a
Build id: 3.10.0-1160.81.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9f481f48
Build id: 3.10.0-1160.81.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff9f481fbb
Build id: 3.10.0-1160.81.1.el7.x86_64
0x6b
4
do_huge_pmd_numa_page
vmlinux 0xffffffff9f643bd6
Build id: 3.10.0-1160.81.1.el7.x86_64
0x256
5
handle_mm_fault
vmlinux 0xffffffff9f60205a
Build id: 3.10.0-1160.81.1.el7.x86_64
0xaba
6
? pick_next_task_fair
vmlinux 0xffffffff9f4effae
Build id: 3.10.0-1160.81.1.el7.x86_64
0x62e
7
__do_page_fault
vmlinux 0xffffffff9fbbf7d3
Build id: 3.10.0-1160.81.1.el7.x86_64
0x213
8
do_page_fault
vmlinux 0xffffffff9fbbfb05
Build id: 3.10.0-1160.81.1.el7.x86_64
0x35
9
page_fault
vmlinux 0xffffffff9fbbb7b8
Build id: 3.10.0-1160.81.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#19 stuck for 23s! [java:2134]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon edac_mce_amd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect crct10dif_common sysimgblt ata_generic fb_sys_fops pata_acpi ttm ahci ata_piix drm libahci mptsas libata scsi_transport_sas mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 19 PID: 2134 Comm: java Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8f390b46b180 ti: ffff8f390f9f8000 task.ti: ffff8f390f9f8000
RIP: 0010:[<ffffffff9f51e07a>]  [<ffffffff9f51e07a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8f390f9fbca8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 000000000001b840 RCX: ffff8f3924a60aa8
RDX: 0000000000000001 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff8f390f9fbce0 R08: ffff8f37b92c3000 R09: ffffffff9f79c139
R10: ffff8f3924edf160 R11: ffffca714b007200 R12: ffff8f390f9fbc58
R13: 0000000000000282 R14: 0000000000000010 R15: ffffffff9f461a13
FS:  00007fdab093d700(0000) GS:ffff8f3924ec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000005a772aa50 CR3: 00000002dae1a000 CR4: 00000000000007e0
Call Trace:
[<ffffffff9f481f48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9f481fbb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9f643bd6>] do_huge_pmd_numa_page+0x256/0x4c0
[<ffffffff9f60205a>] handle_mm_fault+0xaba/0xfb0
[<ffffffff9f4effae>] ? pick_next_task_fair+0x62e/0x880
[<ffffffff9fbbf7d3>] __do_page_fault+0x213/0x510
[<ffffffff9fbbfb05>] do_page_fault+0x35/0x90
[<ffffffff9fbbb7b8>] page_fault+0x28/0x30
Code: 48 63 35 ce 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 15 a0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 19 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #715077
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa191e1ca
Build id: 3.10.0-1160.88.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa1881fa8
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa188201b
Build id: 3.10.0-1160.88.1.el7.x86_64
0x6b
4
do_huge_pmd_numa_page
vmlinux 0xffffffffa1a43f36
Build id: 3.10.0-1160.88.1.el7.x86_64
0x256
5
handle_mm_fault
vmlinux 0xffffffffa1a021b2
Build id: 3.10.0-1160.88.1.el7.x86_64
0xad2
6
? schedule_hrtimeout_range_clock
vmlinux 0xffffffffa1fb7006
Build id: 3.10.0-1160.88.1.el7.x86_64
0xa6
7
__do_page_fault
vmlinux 0xffffffffa1fbf7d3
Build id: 3.10.0-1160.88.1.el7.x86_64
0x213
8
? wake_up_state
vmlinux 0xffffffffa18e1170
Build id: 3.10.0-1160.88.1.el7.x86_64
0x20
9
do_page_fault
vmlinux 0xffffffffa1fbfb05
Build id: 3.10.0-1160.88.1.el7.x86_64
0x35
10
page_fault
vmlinux 0xffffffffa1fbb7b8
Build id: 3.10.0-1160.88.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [splunkd:9467]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_15003(E) falcon_lsm_pinned_14812(E) falcon_lsm_pinned_14713(E) binfmt_misc ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter xt_nat vmw_vsock_vmci_transport vsock iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc i2c_piix4 parport vmw_vmci auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea crc_t10dif sysfillrect crct10dif_generic sysimgblt fb_sys_fops ttm ahci ata_piix libahci drm libata nfit crct10dif_pclmul crct10dif_common crc32c_intel serio_raw
libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal]
CPU: 13 PID: 9467 Comm: splunkd Kdump: loaded Tainted: P            EL ------------   3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff970618881080 ti: ffff97062ddc0000 task.ti: ffff97062ddc0000
RIP: 0010:[<ffffffffa191e1ca>]  [<ffffffffa191e1ca>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff97062ddc3ca8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff97022fa20840
RDX: 0000000000000000 RSI: 0000000000000018 RDI: 0000000000000000
RBP: ffff97062ddc3ce0 R08: ffff9702b6557000 R09: ffffffffa1b9c669
R10: ffff97063f65f160 R11: fffffb80120a1a00 R12: 0000000000000296
R13: ffff97062ddc3c58 R14: ffff9702b6778800 R15: 000000020000000d
FS:  00007f2ae77ff700(0000) GS:ffff97063f640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2aa8893680 CR3: 000000081f13e000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffa1881fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa188201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa1a43f36>] do_huge_pmd_numa_page+0x256/0x4c0
[<ffffffffa1a021b2>] handle_mm_fault+0xad2/0x1190
[<ffffffffa1fb7006>] ? schedule_hrtimeout_range_clock+0xa6/0x160
[<ffffffffa1fbf7d3>] __do_page_fault+0x213/0x510
[<ffffffffa18e1170>] ? wake_up_state+0x20/0x20
[<ffffffffa1fbfb05>] do_page_fault+0x35/0x90
[<ffffffffa1fbb7b8>] page_fault+0x28/0x30
Code: 48 63 35 7e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 55 a2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #658505
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9571e07a
Build id: 3.10.0-1160.81.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff95681f48
Build id: 3.10.0-1160.81.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff95681fbb
Build id: 3.10.0-1160.81.1.el7.x86_64
0x6b
4
do_huge_pmd_numa_page
vmlinux 0xffffffff95843bd6
Build id: 3.10.0-1160.81.1.el7.x86_64
0x256
5
handle_mm_fault
vmlinux 0xffffffff9580205a
Build id: 3.10.0-1160.81.1.el7.x86_64
0xaba
6
? fput
vmlinux 0xffffffff9585e0d3
Build id: 3.10.0-1160.81.1.el7.x86_64
0x13
7
__do_page_fault
vmlinux 0xffffffff95dbf7d3
Build id: 3.10.0-1160.81.1.el7.x86_64
0x213
8
do_page_fault
vmlinux 0xffffffff95dbfb05
Build id: 3.10.0-1160.81.1.el7.x86_64
0x35
9
page_fault
vmlinux 0xffffffff95dbb7b8
Build id: 3.10.0-1160.81.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#20 stuck for 21s! [http-192.168.97:16434]
Modules linked in: tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon edac_mce_amd joydev pcspkr sg i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix scsi_transport_spi drm mptscsih libata serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 20 PID: 16434 Comm: http-192.168.97 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff924306e69080 ti: ffff92412d27c000 task.ti: ffff92412d27c000
RIP: 0010:[<ffffffff9571e07a>]  [<ffffffff9571e07a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff92412d27fca8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000000000001b840 RCX: ffff923e24a20ad0
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff92412d27fce0 R08: ffff923cb9165400 R09: ffffffff9599c139
R10: ffff923e24f1f160 R11: fffff3dcc98eb800 R12: ffff92412d27fc58
R13: ffff924306e69080 R14: ffff924306e696f8 R15: ffff924306e69080
FS:  00007f2de2e72700(0000) GS:ffff923e24f00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000003717ff6b0 CR3: 00000002629bc000 CR4: 00000000000007e0
Call Trace:
[<ffffffff95681f48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff95681fbb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff95843bd6>] do_huge_pmd_numa_page+0x256/0x4c0
[<ffffffff9580205a>] handle_mm_fault+0xaba/0xfb0
[<ffffffff9585e0d3>] ? fput+0x13/0x20
[<ffffffff95dbf7d3>] __do_page_fault+0x213/0x510
[<ffffffff95dbfb05>] do_page_fault+0x35/0x90
[<ffffffff95dbb7b8>] page_fault+0x28/0x30
Code: 48 63 35 ce 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 35 96 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 19 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #622372
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa1f1e07e
Build id: 3.10.0-1160.81.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffa1e81f48
Build id: 3.10.0-1160.81.1.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffa1e81fbb
Build id: 3.10.0-1160.81.1.el7.x86_64
0x6b
4
do_huge_pmd_numa_page
vmlinux 0xffffffffa2043bd6
Build id: 3.10.0-1160.81.1.el7.x86_64
0x256
5
handle_mm_fault
vmlinux 0xffffffffa200205a
Build id: 3.10.0-1160.81.1.el7.x86_64
0xaba
6
__do_page_fault
vmlinux 0xffffffffa25bf7d3
Build id: 3.10.0-1160.81.1.el7.x86_64
0x213
7
do_page_fault
vmlinux 0xffffffffa25bfb05
Build id: 3.10.0-1160.81.1.el7.x86_64
0x35
8
page_fault
vmlinux 0xffffffffa25bb7b8
Build id: 3.10.0-1160.81.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#18 stuck for 23s! [java:6880]
Modules linked in: tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon edac_mce_amd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic vmwgfx crct10dif_common drm_kms_helper syscopyarea ata_generic sysfillrect pata_acpi sysimgblt fb_sys_fops ttm ahci mptsas drm ata_piix libahci scsi_transport_sas mptscsih libata e1000 mptbase serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 18 PID: 6880 Comm: java Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8adaaf1ee300 ti: ffff8adaab188000 task.ti: ffff8adaab188000
RIP: 0010:[<ffffffffa1f1e07e>]  [<ffffffffa1f1e07e>] smp_call_function_many+0x20e/0x270
RSP: 0000:ffff8adaab18bca8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000000000001b840 RCX: ffff8adb24a20a80
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff8adaab18bce0 R08: ffff8ad9b9174c00 R09: ffffffffa219c139
R10: ffff8adb24e9f160 R11: ffffe3b60b264600 R12: ffff8adaab18bc58
R13: 0000000000000001 R14: ffff8adddffc96f0 R15: 0000000300000000
FS:  00007f1833fff700(0000) GS:ffff8adb24e80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000347400000 CR3: 000000027f866000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa1e81f48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa1e81fbb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa2043bd6>] do_huge_pmd_numa_page+0x256/0x4c0
[<ffffffffa200205a>] handle_mm_fault+0xaba/0xfb0
[<ffffffffa25bf7d3>] __do_page_fault+0x213/0x510
[<ffffffffa25bfb05>] do_page_fault+0x35/0x90
[<ffffffffa25bb7b8>] page_fault+0x28/0x30
Code: 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 b5 a2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 19 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #484407
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8f9150ca
Build id: 3.10.0-1062.1.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8f87d398
Build id: 3.10.0-1062.1.2.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8f87d40b
Build id: 3.10.0-1062.1.2.el7.x86_64
0x6b
4
do_huge_pmd_numa_page
vmlinux 0xffffffff8fa32176
Build id: 3.10.0-1062.1.2.el7.x86_64
0x256
5
handle_mm_fault
vmlinux 0xffffffff8f9f24e7
Build id: 3.10.0-1062.1.2.el7.x86_64
0x877
6
__do_page_fault
vmlinux 0xffffffff8ff87653
Build id: 3.10.0-1062.1.2.el7.x86_64
0x213
7
do_page_fault
vmlinux 0xffffffff8ff87975
Build id: 3.10.0-1062.1.2.el7.x86_64
0x35
8
page_fault
vmlinux 0xffffffff8ff83778
Build id: 3.10.0-1062.1.2.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 24s! [elasticsearch[j:63489]
Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter ip_tables xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd parport_pc pcspkr sg joydev parport i2c_piix4 vmw_vmci xfs libcrc32c sr_mod ata_generic cdrom pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea crc_t10dif sysfillrect crct10dif_generic sysimgblt fb_sys_fops ttm ahci drm libahci ata_piix nfit libata crct10dif_pclmul libnvdimm crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy
dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 7 PID: 63489 Comm: elasticsearch[j Tainted: G             L ------------ T 3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9aacd937e2a0 ti: ffff9aadc7b20000 task.ti: ffff9aadc7b20000
RIP: 0010:[<ffffffff8f9150ca>]  [<ffffffff8f9150ca>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff9aadc7b23cc8  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000000000001b7c0 RCX: ffff9aacf9ca06f0
RDX: 0000000000000002 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff9aadc7b23d00 R08: ffff9aacbd66d800 R09: ffffffff8fb7f629
R10: ffff9aacf9ddf160 R11: fffff75e84627800 R12: ffff9aadc7b23c78
R13: 0000000000000000 R14: ffff9aacf8f1f06d R15: 0000ffff00000009
FS:  00007f3a34bdf700(0000) GS:ffff9aacf9dc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000100000818 CR3: 00000001b9668000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8f87d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8f87d40b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff8fa32176>] do_huge_pmd_numa_page+0x256/0x4b0
[<ffffffff8f9f24e7>] handle_mm_fault+0x877/0x9b0
[<ffffffff8ff87653>] __do_page_fault+0x213/0x500
[<ffffffff8ff87975>] do_page_fault+0x35/0x90
[<ffffffff8ff83778>] page_fault+0x28/0x30
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 54 90 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89