Info

Permalink
Function
smp_call_function_many
First occurrence
2020-09-22
Last occurrence
2026-09-12
State
NEW
Components
kernel
Quality
-20

Statistics

Operating system Count
CentOS 7 96
Architecture Count
x86_64 96
Related packages Count
kernel 96
 0:3.10.0-1160.el7 16
 0:3.10.0-957.27.2.el7 16
 0:3.10.0-1160.71.1.el7 13
 0:3.10.0-693.el7 13
 0:3.10.0-514.el7 6
 0:3.10.0-1160.45.1.el7 4
 0:3.10.0-957.el7 4
 0:3.10.0-1160.41.1.el7 3
 0:3.10.0-1160.25.1.el7 2
 0:3.10.0-1160.53.1.el7 2
 0:3.10.0-1160.76.1.el7 2
 0:3.10.0-1160.88.1.el7 2
 0:3.10.0-1127.19.1.el7 1
 0:3.10.0-1160.108.1.el7 1
 0:3.10.0-514.6.1.el7 1
 0:3.10.0-514.26.2.el7 1
 0:3.10.0-1160.15.2.el7 1
 0:3.10.0-693.21.1.el7 1
 0:3.10.0-1160.62.1.el7 1
 0:3.10.0-957.1.3.el7 1
 0:3.10.0-327.el7 1
 0:3.10.0-1160.114.2.el7 1
 0:3.10.0-1160.80.1.el7 1
 0:3.10.0-1127.el7 1
 0:3.10.0-1127.18.2.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1351519
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffaa31720a
Build id: 3.10.0-1160.71.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaa27e258
Build id: 3.10.0-1160.71.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffaa27e3f4
Build id: 3.10.0-1160.71.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffaa408128
Build id: 3.10.0-1160.71.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffffaa3efc55
Build id: 3.10.0-1160.71.1.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffffaa3f207b
Build id: 3.10.0-1160.71.1.el7.x86_64
0xfb
7
handle_mm_fault
vmlinux 0xffffffffaa3f66f2
Build id: 3.10.0-1160.71.1.el7.x86_64
0xb22
8
? do_sync_read
vmlinux 0xffffffffaa44d963
Build id: 3.10.0-1160.71.1.el7.x86_64
0x93
9
__do_page_fault
vmlinux 0xffffffffaa994653
Build id: 3.10.0-1160.71.1.el7.x86_64
0x213
10
do_page_fault
vmlinux 0xffffffffaa994975
Build id: 3.10.0-1160.71.1.el7.x86_64
0x35
11
page_fault
vmlinux 0xffffffffaa990778
Build id: 3.10.0-1160.71.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [connection:27979]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc vfat fat iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev vmw_balloon ablk_helper cryptd pcspkr sg parport_pc parport joydev tpm_crb i2c_piix4 vmw_vmci
ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt crc_t10dif fb_sys_fops crct10dif_generic ttm ahci drm ata_piix libahci nfit libata crct10dif_pclmul libnvdimm crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod synosnap(OE) fuse
CPU: 7 PID: 27979 Comm: connection Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-1160.71.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.21100432.B64.2301110304 01/11/2023
task: ffff93be15a8a100 ti: ffff93bf1c5ec000 task.ti: ffff93bf1c5ec000
RIP: 0010:[<ffffffffaa31720a>]  [<ffffffffaa31720a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff93bf1c5efbe8  EFLAGS: 00000202
RAX: 0000000000000009 RBX: 00000000000000fc RCX: ffff93c31f4606f0
RDX: 0000000000000009 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff93bf1c5efc20 R08: ffff93bc6c598800 R09: ffffffffaa586a19
R10: ffff93c31f3df160 R11: ffffcb9860741000 R12: 0000000000000292
R13: ffff93bf1c5efb98 R14: ffff93c33ffda000 R15: ffffffffaa4289aa
FS:  00007f68287f8700(0000) GS:ffff93c31f3c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f66b8008000 CR3: 00000008046ea000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffaa27e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaa27e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffaa408128>] ptep_clear_flush+0x68/0xa0
[<ffffffffaa3efc55>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffffaa3f207b>] do_wp_page+0xfb/0x720
[<ffffffffaa3f66f2>] handle_mm_fault+0xb22/0xfb0
[<ffffffffaa44d963>] ? do_sync_read+0x93/0xe0
[<ffffffffaa994653>] __do_page_fault+0x213/0x500
[<ffffffffaa994975>] do_page_fault+0x35/0x90
[<ffffffffaa990778>] page_fault+0x28/0x30
Code: 48 63 35 9e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 f5 aa f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1384799
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8106e864
Build id: 3.10.0-693.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff811c2524
Build id: 3.10.0-693.el7.x86_64
0x54
5
wp_page_copy.isra.58
vmlinux 0xffffffff811adfe0
Build id: 3.10.0-693.el7.x86_64
0x240
6
do_wp_page
vmlinux 0xffffffff811afc1b
Build id: 3.10.0-693.el7.x86_64
0xeb
7
? do_read_fault.isra.44
vmlinux 0xffffffff811ad6a6
Build id: 3.10.0-693.el7.x86_64
0xe6
8
handle_mm_fault
vmlinux 0xffffffff811b1f8c
Build id: 3.10.0-693.el7.x86_64
0x70c
9
__do_page_fault
vmlinux 0xffffffff816affb4
Build id: 3.10.0-693.el7.x86_64
0x154
10
do_page_fault
vmlinux 0xffffffff816b02e5
Build id: 3.10.0-693.el7.x86_64
0x35
11
page_fault
vmlinux 0xffffffff816ac508
Build id: 3.10.0-693.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [prometheus:18881]
Modules linked in: tcp_lp xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) tcp_diag inet_diag fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon sg vmw_vmci joydev pcspkr i2c_piix4 nfit libnvdimm shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common libata e1000 mptspi scsi_transport_spi mptscsih crc32c_intel serio_raw mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 18881 Comm: prometheus Tainted: G               ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff880333653f40 ti: ffff8802b0e68000 task.ti: ffff8802b0e68000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0000:ffff8802b0e6bc50  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffffff8106e4a0 RCX: ffff88042e65afe8
RDX: 0000000000000001 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff8802b0e6bc88 R08: ffff88017a326400 R09: ffffffff813227d9
R10: ffff88042e799c80 R11: ffffea0001f0a000 R12: 0000000000000282
R13: ffff8802b0e6bc00 R14: 00000000000000b8 R15: 000280da00000000
FS:  00007f0e7220c740(0000) GS:ffff88042e780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0e6c9a31c0 CR3: 0000000329b54000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001b0e6bca8 ffff880389eb0348 ffff880389eb0000 00007f0e6c9a31c0
0000000000000000 ffff880389eb0348 ffff880278300d18 ffff8802b0e6bcd8
ffffffff8106e668 ffff880389eb0000 00007f0e6c9a31c0 0000000000000000
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e864>] flush_tlb_page+0x54/0xa0
[<ffffffff811c2524>] ptep_clear_flush+0x54/0x80
[<ffffffff811adfe0>] wp_page_copy.isra.58+0x240/0x470
[<ffffffff811afc1b>] do_wp_page+0xeb/0x5c0
[<ffffffff811ad6a6>] ? do_read_fault.isra.44+0xe6/0x130
[<ffffffff811b1f8c>] handle_mm_fault+0x70c/0xfa0
[<ffffffff816affb4>] __do_page_fault+0x154/0x450
[<ffffffff816b02e5>] do_page_fault+0x35/0x90
[<ffffffff816ac508>] page_fault+0x28/0x30
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1356790
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8ad16c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8ac7e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8ac7ea04
Build id: 3.10.0-1160.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff8ae07728
Build id: 3.10.0-1160.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff8adef295
Build id: 3.10.0-1160.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffff8adf16bb
Build id: 3.10.0-1160.el7.x86_64
0xfb
7
? kmem_cache_free
vmlinux 0xffffffff8ae26642
Build id: 3.10.0-1160.el7.x86_64
0x1e2
8
handle_mm_fault
vmlinux 0xffffffff8adf5d32
Build id: 3.10.0-1160.el7.x86_64
0xb22
9
__do_page_fault
vmlinux 0xffffffff8b38e653
Build id: 3.10.0-1160.el7.x86_64
0x213
10
do_page_fault
vmlinux 0xffffffff8b38e975
Build id: 3.10.0-1160.el7.x86_64
0x35
11
page_fault
vmlinux 0xffffffff8b38a778
Build id: 3.10.0-1160.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [npc:19618]
Modules linked in: tcp_lp bnep nfnetlink_queue nfnetlink_log nfnetlink bluetooth rfkill xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq iosf_mbi crc32_pclmul ghash_clmulni_intel snd_seq_device snd_pcm ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd snd_timer joydev sg pcspkr snd vmw_vmci soundcore i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif
crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit ata_piix libnvdimm crct10dif_pclmul crct10dif_common libata crc32c_intel e1000 mptspi scsi_transport_spi mptscsih serio_raw mptbase drm_panel_orientation_quirks fuse
CPU: 3 PID: 19618 Comm: npc Kdump: loaded Tainted: G             L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff88c0b4ed2100 ti: ffff88c319f34000 task.ti: ffff88c319f34000
RIP: 0010:[<ffffffff8ad16c0a>]  [<ffffffff8ad16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff88c319f37be8  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 00000000000000fc RCX: ffff88c3b17a0650
RDX: 0000000000000006 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff88c319f37c20 R08: ffff88c1e8fcb800 R09: ffffffff8af85f99
R10: ffff88c3b16df160 R11: ffffcd9fca726200 R12: 0000000000000292
R13: ffff88c319f37b98 R14: 0000000000000030 R15: 0000000019f37b60
FS:  000000c000054c90(0000) GS:ffff88c3b16c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f84113b5f90 CR3: 00000002aac26000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff8ac7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8ac7ea04>] flush_tlb_page+0x54/0xa0
[<ffffffff8ae07728>] ptep_clear_flush+0x68/0xa0
[<ffffffff8adef295>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffff8adf16bb>] do_wp_page+0xfb/0x720
[<ffffffff8ae26642>] ? kmem_cache_free+0x1e2/0x200
[<ffffffff8adf5d32>] handle_mm_fault+0xb22/0xfb0
[<ffffffff8b38e653>] __do_page_fault+0x213/0x500
[<ffffffff8b38e975>] do_page_fault+0x35/0x90
[<ffffffff8b38a778>] page_fault+0x28/0x30
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 95 8b f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1166019
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8e31761a
Build id: 3.10.0-1127.18.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8e27e8f8
Build id: 3.10.0-1127.18.2.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8e27ea94
Build id: 3.10.0-1127.18.2.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff8e407888
Build id: 3.10.0-1127.18.2.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff8e3ef3c5
Build id: 3.10.0-1127.18.2.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffff8e3f17eb
Build id: 3.10.0-1127.18.2.el7.x86_64
0xfb
7
? hrtimer_get_res
vmlinux 0xffffffff8e2ca2d0
Build id: 3.10.0-1127.18.2.el7.x86_64
0x50
8
handle_mm_fault
vmlinux 0xffffffff8e3f5e82
Build id: 3.10.0-1127.18.2.el7.x86_64
0xb22
9
__do_page_fault
vmlinux 0xffffffff8e98d653
Build id: 3.10.0-1127.18.2.el7.x86_64
0x213
10
do_page_fault
vmlinux 0xffffffff8e98d975
Build id: 3.10.0-1127.18.2.el7.x86_64
0x35
11
? error_swapgs
vmlinux 0xffffffff8e989ac9
Build id: 3.10.0-1127.18.2.el7.x86_64
0xaa
12
page_fault
vmlinux 0xffffffff8e989778
Build id: 3.10.0-1127.18.2.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 40s! [falcon-sensor:9638]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(E) falcon_kal(E) falcon_lsm_pinned_17405(E) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter falcon_lsm_pinned_17308(E) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg vmw_balloon ppdev pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci sch_fq_codel ip_tables ext4 mbcache jbd2 vmwgfx drm_kms_helper syscopyarea ata_generic sysfillrect nfit pata_acpi sysimgblt
fb_sys_fops libnvdimm ttm ahci sd_mod ata_piix libahci crc_t10dif drm crct10dif_generic crct10dif_pclmul vmxnet3 crct10dif_common libata crc32c_intel serio_raw vmw_pvscsi drm_panel_orientation_quirks [last unloaded: falcon_kal]
CPU: 4 PID: 9638 Comm: falcon-sensor Kdump: loaded Tainted: P            EL ------------   3.10.0-1127.18.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff8a4fdba6d230 ti: ffff8a4fd06f8000 task.ti: ffff8a4fd06f8000
RIP: 0010:[<ffffffff8e31761a>]  [<ffffffff8e31761a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8a4fd06fbbe8  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff8a4fffca0678
RDX: 0000000000000002 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff8a4fd06fbc20 R08: ffff8a4d3a373000 R09: ffffffff8e585179
R10: ffff8a4fffd1f160 R11: fffffd9650374700 R12: 0000000000000292
R13: ffff8a4fd06fbb98 R14: ffff8a4fd06fbb00 R15: ffff8a4ffffd3000
FS:  00007f5891057700(0000) GS:ffff8a4fffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055bd741a31e8 CR3: 00000004106ae000 CR4: 00000000000607e0
Call Trace:
[<ffffffff8e27e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8e27ea94>] flush_tlb_page+0x54/0xa0
[<ffffffff8e407888>] ptep_clear_flush+0x68/0xa0
[<ffffffff8e3ef3c5>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffff8e3f17eb>] do_wp_page+0xfb/0x720
[<ffffffff8e2ca2d0>] ? hrtimer_get_res+0x50/0x50
[<ffffffff8e3f5e82>] handle_mm_fault+0xb22/0xfb0
[<ffffffff8e98d653>] __do_page_fault+0x213/0x500
[<ffffffff8e98d975>] do_page_fault+0x35/0x90
[<ffffffff8e989ac9>] ? error_swapgs+0xaa/0xc0
[<ffffffff8e989778>] page_fault+0x28/0x30
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 f5 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1088035
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8791760a
Build id: 3.10.0-1127.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8787e8f8
Build id: 3.10.0-1127.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8787ea94
Build id: 3.10.0-1127.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff87a077f8
Build id: 3.10.0-1127.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff879ef345
Build id: 3.10.0-1127.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffff879f176b
Build id: 3.10.0-1127.el7.x86_64
0xfb
7
handle_mm_fault
vmlinux 0xffffffff879f5e02
Build id: 3.10.0-1127.el7.x86_64
0xb22
8
? do_sync_read
vmlinux 0xffffffff87a4c583
Build id: 3.10.0-1127.el7.x86_64
0x93
9
__do_page_fault
vmlinux 0xffffffff87f8d653
Build id: 3.10.0-1127.el7.x86_64
0x213
10
do_page_fault
vmlinux 0xffffffff87f8d975
Build id: 3.10.0-1127.el7.x86_64
0x35
11
page_fault
vmlinux 0xffffffff87f89778
Build id: 3.10.0-1127.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [redis-server:1543]
Modules linked in: binfmt_misc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper vmw_balloon ablk_helper sg cryptd joydev pcspkr vmw_vmci parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi ahci
vmwgfx libahci drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit sd_mod libnvdimm crc_t10dif crct10dif_generic drm ata_piix libata crct10dif_pclmul crct10dif_common vmxnet3 crc32c_intel serio_raw vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 1543 Comm: redis-server Kdump: loaded Not tainted 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff93300b3e20e0 ti: ffff933006f38000 task.ti: ffff933006f38000
RIP: 0010:[<ffffffff8791760a>]  [<ffffffff8791760a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff933006f3bbe8  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff93303fca06a0
RDX: 0000000000000002 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff933006f3bc20 R08: ffff932974412000 R09: ffffffff87b85099
R10: ffff93303fd5f160 R11: ffffd4fd1f056e00 R12: 0000000000000292
R13: ffff933006f3bb98 R14: 0000000000000030 R15: 0000000001119f00
FS:  00007fd8740c5f80(0000) GS:ffff93303fd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd823ddfa28 CR3: 000000080b278000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff8787e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8787ea94>] flush_tlb_page+0x54/0xa0
[<ffffffff87a077f8>] ptep_clear_flush+0x68/0xa0
[<ffffffff879ef345>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffff879f176b>] do_wp_page+0xfb/0x720
[<ffffffff879f5e02>] handle_mm_fault+0xb22/0xfb0
[<ffffffff87a4c583>] ? do_sync_read+0x93/0xe0
[<ffffffff87f8d653>] __do_page_fault+0x213/0x500
[<ffffffff87f8d975>] do_page_fault+0x35/0x90
[<ffffffff87f89778>] page_fault+0x28/0x30
Code: 48 63 35 1e 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 0f 55 88 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 7a c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1031363
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa09170da
Build id: 3.10.0-1160.41.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa087e258
Build id: 3.10.0-1160.41.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffa087e3f4
Build id: 3.10.0-1160.41.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffa0a08108
Build id: 3.10.0-1160.41.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffffa09efc05
Build id: 3.10.0-1160.41.1.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffffa09f202b
Build id: 3.10.0-1160.41.1.el7.x86_64
0xfb
7
? hrtimer_get_res
vmlinux 0xffffffffa08c9aa0
Build id: 3.10.0-1160.41.1.el7.x86_64
0x50
8
handle_mm_fault
vmlinux 0xffffffffa09f66a2
Build id: 3.10.0-1160.41.1.el7.x86_64
0xb22
9
__do_page_fault
vmlinux 0xffffffffa0f90653
Build id: 3.10.0-1160.41.1.el7.x86_64
0x213
10
do_page_fault
vmlinux 0xffffffffa0f90975
Build id: 3.10.0-1160.41.1.el7.x86_64
0x35
11
page_fault
vmlinux 0xffffffffa0f8c778
Build id: 3.10.0-1160.41.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [pool:17445]
Modules linked in: veeamsnap(OE) tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm iosf_mbi snd_timer crc32_pclmul snd ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
ablk_helper soundcore cryptd ppdev vmw_balloon joydev i2c_piix4 pcspkr vmw_vmci sg parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit libnvdimm drm ata_piix libata e1000 crct10dif_pclmul crct10dif_common crc32c_intel mptspi scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 1 PID: 17445 Comm: pool Kdump: loaded Tainted: G        W  OEL ------------   3.10.0-1160.41.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9f2875e28000 ti: ffff9f28f9890000 task.ti: ffff9f28f9890000
RIP: 0010:[<ffffffffa09170da>]  [<ffffffffa09170da>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff9f28f9893be8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9f29f7620600
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9f28f9893c20 R08: ffff9f29bb88b000 R09: ffffffffa0b86bd9
R10: ffff9f29f765f160 R11: ffffe28d05977e00 R12: 0000000000000292
R13: ffff9f28f9893b98 R14: ffff9f29fffd8000 R15: ffffffffa0a2898a
FS:  00007f2557fff700(0000) GS:ffff9f29f7640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2557ffeae8 CR3: 000000009c274000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa087e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa087e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffa0a08108>] ptep_clear_flush+0x68/0xa0
[<ffffffffa09efc05>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffffa09f202b>] do_wp_page+0xfb/0x720
[<ffffffffa08c9aa0>] ? hrtimer_get_res+0x50/0x50
[<ffffffffa09f66a2>] handle_mm_fault+0xb22/0xfb0
[<ffffffffa0f90653>] __do_page_fault+0x213/0x500
[<ffffffffa0f90975>] do_page_fault+0x35/0x90
[<ffffffffa0f8c778>] page_fault+0x28/0x30
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 55 a1 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #894917
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff94f1720a
Build id: 3.10.0-1160.71.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff94e7e258
Build id: 3.10.0-1160.71.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff94e7e3f4
Build id: 3.10.0-1160.71.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff95008128
Build id: 3.10.0-1160.71.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff94fefc55
Build id: 3.10.0-1160.71.1.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffff94ff207b
Build id: 3.10.0-1160.71.1.el7.x86_64
0xfb
7
? native_queued_spin_lock_slowpath
vmlinux 0xffffffff94f17acd
Build id: 3.10.0-1160.71.1.el7.x86_64
0x1d
8
handle_mm_fault
vmlinux 0xffffffff94ff66f2
Build id: 3.10.0-1160.71.1.el7.x86_64
0xb22
9
__do_page_fault
vmlinux 0xffffffff95594653
Build id: 3.10.0-1160.71.1.el7.x86_64
0x213
10
do_page_fault
vmlinux 0xffffffff95594975
Build id: 3.10.0-1160.71.1.el7.x86_64
0x35
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [gdbus:2763]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev vmw_balloon snd_ens1371 sb_edac snd_rawmidi iosf_mbi snd_ac97_codec crc32_pclmul ac97_bus snd_seq ghash_clmulni_intel snd_seq_device snd_pcm aesni_intel lrw gf128mul glue_helper snd_timer ablk_helper snd joydev cryptd
sg soundcore pcspkr vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit libnvdimm ata_piix libata mptspi crct10dif_pclmul scsi_transport_spi crct10dif_common mptscsih e1000 crc32c_intel mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 9 PID: 2763 Comm: gdbus Tainted: G             L ------------   3.10.0-1160.71.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff93ac0f90e300 ti: ffff93ac041d0000 task.ti: ffff93ac041d0000
RIP: 0010:[<ffffffff94f1720a>]  [<ffffffff94f1720a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff93ac041d3be8  EFLAGS: 00000202
RAX: 000000000000000b RBX: 00000000000000fc RCX: ffff93ac3d8e0740
RDX: 000000000000000b RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff93ac041d3c20 R08: ffff939d996a7800 R09: ffffffff95186a19
R10: ffff93ac3d85f160 R11: fffffc48bf54da00 R12: 0000000000000292
R13: ffff93ac041d3b98 R14: ffff93ac7ffda000 R15: ffffffff94fc8f64
FS:  00007f9b7082a700(0000) GS:ffff93ac3d840000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000010f6040 CR3: 0000000fe4bce000 CR4: 00000000001607e0
Call Trace:
[<ffffffff94e7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff94e7e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffff95008128>] ptep_clear_flush+0x68/0xa0
[<ffffffff94fefc55>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffff94ff207b>] do_wp_page+0xfb/0x720
[<ffffffff94f17acd>] ? native_queued_spin_lock_slowpath+0x1d/0x200
[<ffffffff94ff66f2>] handle_mm_fault+0xb22/0xfb0
[<ffffffff95594653>] __do_page_fault+0x213/0x500
[<ffffffff95594975>] do_page_fault+0x35/0x90
Complete report #884405
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff98716c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9867e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff9867ea04
Build id: 3.10.0-1160.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff98807728
Build id: 3.10.0-1160.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff987ef295
Build id: 3.10.0-1160.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffff987f16bb
Build id: 3.10.0-1160.el7.x86_64
0xfb
7
? __switch_to
vmlinux 0xffffffff9862b621
Build id: 3.10.0-1160.el7.x86_64
0x151
8
handle_mm_fault
vmlinux 0xffffffff987f5d32
Build id: 3.10.0-1160.el7.x86_64
0xb22
9
__do_page_fault
vmlinux 0xffffffff98d8e653
Build id: 3.10.0-1160.el7.x86_64
0x213
10
do_page_fault
vmlinux 0xffffffff98d8e975
Build id: 3.10.0-1160.el7.x86_64
0x35
11
page_fault
vmlinux 0xffffffff98d8a778
Build id: 3.10.0-1160.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 21s! [async:0:4361]
Modules linked in: xt_REDIRECT nf_nat_redirect tcp_lp bnep nfnetlink_queue nfnetlink_log bluetooth rfkill xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4
ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit ata_piix drm mptspi libata e1000 scsi_transport_spi libnvdimm mptscsih crct10dif_pclmul crct10dif_common mptbase drm_panel_orientation_quirks crc32c_intel serio_raw dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 7 PID: 4361 Comm: async:0 Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff976ed2106300 ti: ffff976ea04f8000 task.ti: ffff976ea04f8000
RIP: 0010:[<ffffffff98716c0a>]  [<ffffffff98716c0a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff976ea04fbbe8  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff9770698a06f0
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff976ea04fbc20 R08: ffff976fbe74c800 R09: ffffffff98985f99
R10: ffff9770699df160 R11: fffff812c78cc600 R12: 0000000000000292
R13: ffff976ea04fbb98 R14: 0000000000000030 R15: 0000000098716562
FS:  00007f7837fe2700(0000) GS:ffff9770699c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000607000115a77 CR3: 000000009236a000 CR4: 0000000000360fe0
Call Trace:
[<ffffffff9867e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9867ea04>] flush_tlb_page+0x54/0xa0
[<ffffffff98807728>] ptep_clear_flush+0x68/0xa0
[<ffffffff987ef295>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffff987f16bb>] do_wp_page+0xfb/0x720
[<ffffffff9862b621>] ? __switch_to+0x151/0x580
[<ffffffff987f5d32>] handle_mm_fault+0xb22/0xfb0
[<ffffffff98d8e653>] __do_page_fault+0x213/0x500
[<ffffffff98d8e975>] do_page_fault+0x35/0x90
[<ffffffff98d8a778>] page_fault+0x28/0x30
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 35 99 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #883010
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff97f1e1ca
Build id: 3.10.0-1160.88.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff97e81fa8
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff97e82144
Build id: 3.10.0-1160.88.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff980141b8
Build id: 3.10.0-1160.88.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff97ffb5ee
Build id: 3.10.0-1160.88.1.el7.x86_64
0x32e
6
do_wp_page
vmlinux 0xffffffff97ffdafb
Build id: 3.10.0-1160.88.1.el7.x86_64
0xfb
7
? try_to_wake_up
vmlinux 0xffffffff97ee0e92
Build id: 3.10.0-1160.88.1.el7.x86_64
0x192
8
handle_mm_fault
vmlinux 0xffffffff9800221a
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb3a
9
__do_page_fault
vmlinux 0xffffffff985bf7d3
Build id: 3.10.0-1160.88.1.el7.x86_64
0x213
10
do_page_fault
vmlinux 0xffffffff985bfb05
Build id: 3.10.0-1160.88.1.el7.x86_64
0x35
11
page_fault
vmlinux 0xffffffff985bb7b8
Build id: 3.10.0-1160.88.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [VM Thread:5909]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device iosf_mbi snd_pcm crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon aesni_intel lrw snd_timer gf128mul glue_helper snd ablk_helper cryptd sg pcspkr soundcore joydev vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm libnvdimm ata_piix libata crct10dif_pclmul crct10dif_common mptspi e1000 scsi_transport_spi crc32c_intel mptscsih mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 7 PID: 5909 Comm: VM Thread Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9c914eaf1080 ti: ffff9c91480a4000 task.ti: ffff9c91480a4000
RIP: 0010:[<ffffffff97f1e1ca>]  [<ffffffff97f1e1ca>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff9c91480a7be8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9c93dd6206f0
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9c91480a7c20 R08: ffff9c8d2c55d000 R09: ffffffff9819c669
R10: ffff9c93dd7df160 R11: ffffdb7dc82a8200 R12: 0000000000000292
R13: ffff9c91480a7b98 R14: ffff9c93fffda000 R15: ffffffff97fd3ee4
FS:  00007ff5fc1b7700(0000) GS:ffff9c93dd7c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff5e42b9018 CR3: 000000058eb74000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffff97e81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff97e82144>] flush_tlb_page+0x54/0xa0
[<ffffffff980141b8>] ptep_clear_flush+0x68/0xa0
[<ffffffff97ffb5ee>] wp_page_copy.isra.73+0x32e/0x5d0
[<ffffffff97ffdafb>] do_wp_page+0xfb/0x730
[<ffffffff97ee0e92>] ? try_to_wake_up+0x192/0x3a0
[<ffffffff9800221a>] handle_mm_fault+0xb3a/0x1190
[<ffffffff985bf7d3>] __do_page_fault+0x213/0x510
[<ffffffff985bfb05>] do_page_fault+0x35/0x90
[<ffffffff985bb7b8>] page_fault+0x28/0x30
Code: 48 63 35 7e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 b5 98 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #843283
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9f51e07a
Build id: 3.10.0-1160.80.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9f481f48
Build id: 3.10.0-1160.80.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff9f4820e4
Build id: 3.10.0-1160.80.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff9f613eb8
Build id: 3.10.0-1160.80.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff9f5fb4d5
Build id: 3.10.0-1160.80.1.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffff9f5fd9bb
Build id: 3.10.0-1160.80.1.el7.x86_64
0xfb
7
handle_mm_fault
vmlinux 0xffffffff9f6020c2
Build id: 3.10.0-1160.80.1.el7.x86_64
0xb22
8
__do_page_fault
vmlinux 0xffffffff9fbbf7d3
Build id: 3.10.0-1160.80.1.el7.x86_64
0x213
9
do_page_fault
vmlinux 0xffffffff9fbbfb05
Build id: 3.10.0-1160.80.1.el7.x86_64
0x35
10
page_fault
vmlinux 0xffffffff9fbbb7b8
Build id: 3.10.0-1160.80.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [mattermost:128407]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi snd_ens1371 crc32_pclmul snd_rawmidi ghash_clmulni_intel snd_ac97_codec ppdev ac97_bus aesni_intel lrw snd_seq gf128mul vmw_balloon glue_helper ablk_helper snd_seq_device cryptd snd_pcm joydev pcspkr snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
ttm nfit crct10dif_pclmul crct10dif_common drm crc32c_intel ata_piix libnvdimm serio_raw libata e1000 mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 128407 Comm: mattermost Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.80.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff95cc72ca0000 ti: ffff95cc16704000 task.ti: ffff95cc16704000
RIP: 0010:[<ffffffff9f51e07a>]  [<ffffffff9f51e07a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff95cc16707be8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00000000000000fc RCX: ffff95cc74a5fc10
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff95cc16707c20 R08: ffff95cbbfdd7000 R09: ffffffff9f79c139
R10: ffff95cc74a1f160 R11: ffffe06e4283ec00 R12: 0000000000000292
R13: ffff95cc16707b98 R14: ffff95cc7f3da000 R15: ffffffff9f5d3d84
FS:  00007fc0677fe700(0000) GS:ffff95cc74a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0122e5000 CR3: 00000001a4c18000 CR4: 00000000001607f0
Call Trace:
[<ffffffff9f481f48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9f4820e4>] flush_tlb_page+0x54/0xa0
[<ffffffff9f613eb8>] ptep_clear_flush+0x68/0xa0
[<ffffffff9f5fb4d5>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffff9f5fd9bb>] do_wp_page+0xfb/0x730
[<ffffffff9f6020c2>] handle_mm_fault+0xb22/0xfb0
[<ffffffff9fbbf7d3>] __do_page_fault+0x213/0x510
[<ffffffff9fbbfb05>] do_page_fault+0x35/0x90
[<ffffffff9fbbb7b8>] page_fault+0x28/0x30
Code: 48 63 35 ce 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 15 a0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 19 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #734035
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff92312b6a
Build id: 3.10.0-957.27.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9227b788
Build id: 3.10.0-957.27.2.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff9227b924
Build id: 3.10.0-957.27.2.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff923fecc7
Build id: 3.10.0-957.27.2.el7.x86_64
0x67
5
wp_page_copy.isra.71
vmlinux 0xffffffff923e6d1b
Build id: 3.10.0-957.27.2.el7.x86_64
0x30b
6
do_wp_page
vmlinux 0xffffffff923e8f6b
Build id: 3.10.0-957.27.2.el7.x86_64
0xfb
7
? futex_wait
vmlinux 0xffffffff9230ef85
Build id: 3.10.0-957.27.2.el7.x86_64
0x195
8
handle_pte_fault
vmlinux 0xffffffff923ea8fd
Build id: 3.10.0-957.27.2.el7.x86_64
0x37d
9
handle_mm_fault
vmlinux 0xffffffff923ed3ad
Build id: 3.10.0-957.27.2.el7.x86_64
0x39d
10
__do_page_fault
vmlinux 0xffffffff92971603
Build id: 3.10.0-957.27.2.el7.x86_64
0x203
11
do_page_fault
vmlinux 0xffffffff92971925
Build id: 3.10.0-957.27.2.el7.x86_64
0x35
12
? error_swapgs
vmlinux 0xffffffff9296dab6
Build id: 3.10.0-957.27.2.el7.x86_64
0xa7
13
page_fault
vmlinux 0xffffffff9296d768
Build id: 3.10.0-957.27.2.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [asterisk:1825]
Modules linked in: vmw_vsock_vmci_transport vsock crc32_pclmul ppdev ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common ata_piix crc32c_intel mptspi scsi_transport_spi serio_raw mptscsih mptbase libata drm_panel_orientation_quirks vmxnet3 floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 1825 Comm: asterisk Kdump: loaded Tainted: G             L ------------   3.10.0-957.27.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff907034adb0c0 ti: ffff907031d24000 task.ti: ffff907031d24000
RIP: 0010:[<ffffffff92312b6a>]  [<ffffffff92312b6a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff907031d27b70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffff9227b5c0 RCX: ffff9070396204b8
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff907031d27ba8 R08: ffff907036f83400 R09: ffffffff925774a9
R10: ffff90703969f160 R11: ffffee1a802f6a00 R12: 0000000000000286
R13: ffff907031d27b20 R14: ffff907031d27b30 R15: ffffffffffffffff
FS:  00007fae350d3700(0000) GS:ffff907039680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fae052a2d68 CR3: 00000000b1dc6000 CR4: 00000000000007e0
Call Trace:
[<ffffffff9227b788>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9227b924>] flush_tlb_page+0x54/0xa0
[<ffffffff923fecc7>] ptep_clear_flush+0x67/0xa0
[<ffffffff923e6d1b>] wp_page_copy.isra.71+0x30b/0x560
[<ffffffff923e8f6b>] do_wp_page+0xfb/0x720
[<ffffffff9230ef85>] ? futex_wait+0x195/0x280
[<ffffffff923ea8fd>] handle_pte_fault+0x37d/0xd10
[<ffffffff923ed3ad>] handle_mm_fault+0x39d/0x9b0
[<ffffffff92971603>] __do_page_fault+0x203/0x4f0
[<ffffffff92971925>] do_page_fault+0x35/0x90
[<ffffffff9296dab6>] ? error_swapgs+0xa7/0xbd
[<ffffffff9296d768>] page_fault+0x28/0x30
Code: 48 63 35 5e 75 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 bf f4 92 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 2d 75 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #642916
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffabd1173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffabc7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffabc7a734
Build id: 3.10.0-957.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffabdfd587
Build id: 3.10.0-957.el7.x86_64
0x67
5
wp_page_copy.isra.71
vmlinux 0xffffffffabde55bb
Build id: 3.10.0-957.el7.x86_64
0x30b
6
do_wp_page
vmlinux 0xffffffffabde782b
Build id: 3.10.0-957.el7.x86_64
0xfb
7
handle_pte_fault
vmlinux 0xffffffffabde91bd
Build id: 3.10.0-957.el7.x86_64
0x37d
8
? do_page_fault
vmlinux 0xffffffffac36f915
Build id: 3.10.0-957.el7.x86_64
0x35
9
handle_mm_fault
vmlinux 0xffffffffabdebc6d
Build id: 3.10.0-957.el7.x86_64
0x39d
10
__do_page_fault
vmlinux 0xffffffffac36f5e3
Build id: 3.10.0-957.el7.x86_64
0x203
11
do_page_fault
vmlinux 0xffffffffac36f915
Build id: 3.10.0-957.el7.x86_64
0x35
12
page_fault
vmlinux 0xffffffffac36b758
Build id: 3.10.0-957.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [python:11390]
Modules linked in: vmw_vsock_vmci_transport vsock vfat fat sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg tpm_crb parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw e1000e ahci mptsas drm scsi_transport_sas ata_piix libahci mptscsih libata mptbase ptp pps_core nfit libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 9 PID: 11390 Comm: python Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.17369862.B64.2012240522 12/24/2020
task: ffff981057a8d140 ti: ffff98105585c000 task.ti: ffff98105585c000
RIP: 0010:[<ffffffffabd1173a>]  [<ffffffffabd1173a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff98105585fb70  EFLAGS: 00000202
RAX: 000000000000000e RBX: 00080000abc7a3d0 RCX: ffff98105d9a06f0
RDX: 000000000000000e RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff98105585fba8 R08: ffff9809b4200400 R09: ffffffffabf755c9
R10: ffff98105d85f160 R11: fffffc1a143fb200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff98105585fb20 R15: ffffffffffffffff
FS:  00007f6bfd6fa700(0000) GS:ffff98105d840000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6bfd6faa08 CR3: 000000080c7a6000 CR4: 00000000001607e0
Call Trace:
[<ffffffffabc7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffabc7a734>] flush_tlb_page+0x54/0xa0
[<ffffffffabdfd587>] ptep_clear_flush+0x67/0xa0
[<ffffffffabde55bb>] wp_page_copy.isra.71+0x30b/0x560
[<ffffffffabde782b>] do_wp_page+0xfb/0x720
[<ffffffffabde91bd>] handle_pte_fault+0x37d/0xd10
[<ffffffffac36f915>] ? do_page_fault+0x35/0x90
[<ffffffffabdebc6d>] handle_mm_fault+0x39d/0x9b0
[<ffffffffac36f5e3>] __do_page_fault+0x203/0x500
[<ffffffffac36f915>] do_page_fault+0x35/0x90
[<ffffffffac36b758>] page_fault+0x28/0x30
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 94 ac f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #606581
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810e6de2
Build id: 3.10.0-327.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff81065e58
Build id: 3.10.0-327.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff81066055
Build id: 3.10.0-327.el7.x86_64
0x55
4
ptep_clear_flush
vmlinux 0xffffffff811a77c4
Build id: 3.10.0-327.el7.x86_64
0x54
5
do_wp_page
vmlinux 0xffffffff81194a41
Build id: 3.10.0-327.el7.x86_64
0x271
6
handle_mm_fault
vmlinux 0xffffffff8119712c
Build id: 3.10.0-327.el7.x86_64
0x65c
7
__do_page_fault
vmlinux 0xffffffff81640e22
Build id: 3.10.0-327.el7.x86_64
0x152
8
do_page_fault
vmlinux 0xffffffff81641113
Build id: 3.10.0-327.el7.x86_64
0x23
9
page_fault
vmlinux 0xffffffff8163d408
Build id: 3.10.0-327.el7.x86_64
0x28
BUG: soft lockup - CPU#1 stuck for 23s! [etcd:13512]
Modules linked in: gsch(OE) redirfs(OE) dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop bridge stp llc ip_set nfnetlink dsa_filter(POE) vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr sg shpchp vmw_vmci parport_pc parport i2c_piix4 xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx serio_raw drm_kms_helper ttm mptsas scsi_transport_sas drm mptscsih mptbase ata_piix vmxnet3 libata i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 1 PID: 13512 Comm: etcd Tainted: P           OEL ------------   3.10.0-327.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8803eb354500 ti: ffff88038be14000 task.ti: ffff88038be14000
RIP: 0010:[<ffffffff810e6de2>]  [<ffffffff810e6de2>] smp_call_function_many+0x202/0x260
RSP: 0000:ffff88038be17cc8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff88043fc182b0
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff88038be17d00 R08: ffff8804295ab800 R09: ffff88043fc975e0
R10: ffffea000f4fae00 R11: ffffffff812f2869 R12: 0000000000000292
R13: ffff88038be17c78 R14: 0000000000000286 R15: ffff88038be17c60
FS:  00007f2e89ffb700(0000) GS:ffff88043fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2e9024e8e0 CR3: 000000024a5c5000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001eb354500 ffff880428380fc8 ffff880428380c80 00007f2e9024e8e0
0000000000000000 ffff880428380fc8 ffff88024a581270 ffff88038be17d50
ffffffff81065e58 ffff880428380c80 00007f2e9024e8e0 0000000000000000
Call Trace:
[<ffffffff81065e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff81066055>] flush_tlb_page+0x55/0xa0
[<ffffffff811a77c4>] ptep_clear_flush+0x54/0x80
[<ffffffff81194a41>] do_wp_page+0x271/0x800
[<ffffffff8119712c>] handle_mm_fault+0x65c/0xf50
[<ffffffff81640e22>] __do_page_fault+0x152/0x420
[<ffffffff81641113>] do_page_fault+0x23/0x80
[<ffffffff8163d408>] page_fault+0x28/0x30
Code: 48 63 35 f6 2a 98 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 b8 a5 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 c5 2a 98 00 eb b7 0f b6 4d cc 4c 89 
Complete report #587662
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffabb1718a
Build id: 3.10.0-1160.76.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaba7e258
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffaba7e3f4
Build id: 3.10.0-1160.76.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffabc08108
Build id: 3.10.0-1160.76.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffffabbefc35
Build id: 3.10.0-1160.76.1.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffffabbf205b
Build id: 3.10.0-1160.76.1.el7.x86_64
0xfb
7
handle_mm_fault
vmlinux 0xffffffffabbf66d2
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb22
8
__do_page_fault
vmlinux 0xffffffffac194653
Build id: 3.10.0-1160.76.1.el7.x86_64
0x213
9
do_page_fault
vmlinux 0xffffffffac194975
Build id: 3.10.0-1160.76.1.el7.x86_64
0x35
10
? error_swapgs
vmlinux 0xffffffffac190ac9
Build id: 3.10.0-1160.76.1.el7.x86_64
0xaa
11
page_fault
vmlinux 0xffffffffac190778
Build id: 3.10.0-1160.76.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [HikariPool-1 ho:4528]
Modules linked in: tcp_lp veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter binfmt_misc overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac snd_ens1371 coretemp snd_rawmidi snd_ac97_codec ac97_bus snd_seq iosf_mbi crc32_pclmul snd_seq_device ghash_clmulni_intel snd_pcm aesni_intel snd_timer
lrw gf128mul snd glue_helper ablk_helper cryptd ppdev soundcore sg joydev vmw_balloon pcspkr vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix libata crct10dif_pclmul crct10dif_common drm crc32c_intel e1000 nfit mptspi serio_raw scsi_transport_spi libnvdimm mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 5 PID: 4528 Comm: HikariPool-1 ho Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff95c10a51c200 ti: ffff95be6c744000 task.ti: ffff95be6c744000
RIP: 0010:[<ffffffffabb1718a>]  [<ffffffffabb1718a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff95be6c747be8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff95c168e206a0
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff95be6c747c20 R08: ffff95bf774cd800 R09: ffffffffabd869f9
R10: ffff95c168f5f160 R11: ffffe68e89b18e00 R12: 0000000000000292
R13: ffff95be6c747b98 R14: ffff95c1787d2000 R15: ffffffffabbc8f44
FS:  00007fd2a2804b38(0000) GS:ffff95c168f40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd2a3e41000 CR3: 000000030173c000 CR4: 00000000000607e0
Call Trace:
[<ffffffffaba7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaba7e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffabc08108>] ptep_clear_flush+0x68/0xa0
[<ffffffffabbefc35>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffffabbf205b>] do_wp_page+0xfb/0x720
[<ffffffffabbf66d2>] handle_mm_fault+0xb22/0xfb0
[<ffffffffac194653>] __do_page_fault+0x213/0x500
[<ffffffffac194975>] do_page_fault+0x35/0x90
[<ffffffffac190ac9>] ? error_swapgs+0xaa/0xc0
[<ffffffffac190778>] page_fault+0x28/0x30
Code: 48 63 35 be 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 75 ac f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #538734
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9d316ee8
Build id: 3.10.0-1160.25.1.el7.x86_64
0x208
2
native_flush_tlb_others
vmlinux 0xffffffff9d27e868
Build id: 3.10.0-1160.25.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff9d27ea04
Build id: 3.10.0-1160.25.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff9d4079d8
Build id: 3.10.0-1160.25.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff9d3ef4e5
Build id: 3.10.0-1160.25.1.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffff9d3f190b
Build id: 3.10.0-1160.25.1.el7.x86_64
0xfb
7
handle_mm_fault
vmlinux 0xffffffff9d3f5f82
Build id: 3.10.0-1160.25.1.el7.x86_64
0xb22
8
__do_page_fault
vmlinux 0xffffffff9d990653
Build id: 3.10.0-1160.25.1.el7.x86_64
0x213
9
do_page_fault
vmlinux 0xffffffff9d990975
Build id: 3.10.0-1160.25.1.el7.x86_64
0x35
10
? error_swapgs
vmlinux 0xffffffff9d98cac9
Build id: 3.10.0-1160.25.1.el7.x86_64
0xaa
11
page_fault
vmlinux 0xffffffff9d98c778
Build id: 3.10.0-1160.25.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 265s! [gdbus:18071]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink vboxsf(OE) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc snd_intel8x0 vmwgfx snd_ac97_codec ac97_bus ttm snd_seq drm_kms_helper snd_seq_device syscopyarea snd_pcm sysfillrect joydev sysimgblt snd_timer fb_sys_fops ppdev vboxguest(OE) snd virtio_net parport_pc drm net_failover failover soundcore sg drm_panel_orientation_quirks
i2c_piix4 parport pcspkr video ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi ahci ata_piix libahci serio_raw virtio_pci virtio_ring libata e1000 crc32c_intel virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 6 PID: 18071 Comm: gdbus Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff9b71ce6ac200 ti: ffff9b7478c10000 task.ti: ffff9b7478c10000
RIP: 0010:[<ffffffff9d316ee8>]  [<ffffffff9d316ee8>] smp_call_function_many+0x208/0x270
RSP: 0000:ffff9b7478c13be8  EFLAGS: 00010202
RAX: 0000000000000001 RBX: 0000000000000010 RCX: ffff9b751fc604c8
RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff9b7478c13c20 R08: ffff9b7279359c00 R09: ffffffff9d586549
R10: ffff9b751fd9f160 R11: ffffe2e88f3ca200 R12: ffffffffffffff10
R13: 0000000000000286 R14: 00000000000000fc R15: 00000000000008fc
FS:  00007f9767976700(0000) GS:ffff9b751fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f975000e330 CR3: 00000003799da000 CR4: 00000000000206e0
Call Trace:
[<ffffffff9d27e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9d27ea04>] flush_tlb_page+0x54/0xa0
[<ffffffff9d4079d8>] ptep_clear_flush+0x68/0xa0
[<ffffffff9d3ef4e5>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffff9d3f190b>] do_wp_page+0xfb/0x720
[<ffffffff9d3f5f82>] handle_mm_fault+0xb22/0xfb0
[<ffffffff9d990653>] __do_page_fault+0x213/0x500
[<ffffffff9d990975>] do_page_fault+0x35/0x90
[<ffffffff9d98cac9>] ? error_swapgs+0xaa/0xc0
[<ffffffff9d98c778>] page_fault+0x28/0x30
Code: 28 00 48 63 35 fe 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 f5 9d f6 41 20 01 74 cd 0f 1f 44 00 00 <f3> 90 f6 41 20 01 75 f8 48 63 35 cd 87 c4 00 eb b7 0f b6 4d cc 
Complete report #516456
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9811181e
Build id: 3.10.0-957.1.3.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffff9807a598
Build id: 3.10.0-957.1.3.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff9807a734
Build id: 3.10.0-957.1.3.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff981fd7f7
Build id: 3.10.0-957.1.3.el7.x86_64
0x67
5
wp_page_copy.isra.71
vmlinux 0xffffffff981e582b
Build id: 3.10.0-957.1.3.el7.x86_64
0x30b
6
do_wp_page
vmlinux 0xffffffff981e7a9b
Build id: 3.10.0-957.1.3.el7.x86_64
0xfb
7
? sched_clock
vmlinux 0xffffffff98034919
Build id: 3.10.0-957.1.3.el7.x86_64
0x9
8
handle_pte_fault
vmlinux 0xffffffff981e942d
Build id: 3.10.0-957.1.3.el7.x86_64
0x37d
9
? __schedule
vmlinux 0xffffffff9876778f
Build id: 3.10.0-957.1.3.el7.x86_64
0x3ff
10
handle_mm_fault
vmlinux 0xffffffff981ebedd
Build id: 3.10.0-957.1.3.el7.x86_64
0x39d
11
__do_page_fault
vmlinux 0xffffffff9876f5e3
Build id: 3.10.0-957.1.3.el7.x86_64
0x203
12
do_page_fault
vmlinux 0xffffffff9876f915
Build id: 3.10.0-957.1.3.el7.x86_64
0x35
13
page_fault
vmlinux 0xffffffff9876b758
Build id: 3.10.0-957.1.3.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [sdsvrd:12625]
Modules linked in: fuse binfmt_misc xt_multiport xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac ext4 iosf_mbi crc32_pclmul mbcache ghash_clmulni_intel jbd2 aesni_intel lrw gf128mul glue_helper ablk_helper snd_ens1371 cryptd snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device
ppdev snd_pcm vmw_balloon snd_timer snd soundcore pcspkr joydev sg nfit libnvdimm parport_pc parport i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom vmwgfx crct10dif_pclmul crct10dif_common ata_generic pata_acpi crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm e1000 mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks ata_piix libata dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 12625 Comm: sdsvrd Kdump: loaded Tainted: G             L ------------   3.10.0-957.1.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9c9a4f7ea080 ti: ffff9c9a4aa38000 task.ti: ffff9c9a4aa38000
RIP: 0010:[<ffffffff9811181e>]  [<ffffffff9811181e>] smp_call_function_many+0x20e/0x270
RSP: 0000:ffff9c9a4aa3bb70  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 000800009807a3d0 RCX: ffff9c9a757a0628
RDX: 0000000000000006 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9c9a4aa3bba8 R08: ffff9c99bc85c000 R09: ffffffff98375839
R10: ffff9c9a7571f160 R11: ffffd81240923c00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9c9a4aa3bb20 R15: ffffffffffffffff
FS:  000000004453f940(0063) GS:ffff9c9a75700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000445235b8 CR3: 00000002183ae000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9807a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9807a734>] flush_tlb_page+0x54/0xa0
[<ffffffff981fd7f7>] ptep_clear_flush+0x67/0xa0
[<ffffffff981e582b>] wp_page_copy.isra.71+0x30b/0x560
[<ffffffff981e7a9b>] do_wp_page+0xfb/0x720
[<ffffffff98034919>] ? sched_clock+0x9/0x10
[<ffffffff981e942d>] handle_pte_fault+0x37d/0xd10
[<ffffffff9876778f>] ? __schedule+0x3ff/0x890
[<ffffffff981ebedd>] handle_mm_fault+0x39d/0x9b0
[<ffffffff9876f5e3>] __do_page_fault+0x203/0x500
[<ffffffff9876f915>] do_page_fault+0x35/0x90
[<ffffffff9876b758>] page_fault+0x28/0x30
Code: 82 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 d4 98 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 3d 82 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #441137
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb0f16c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb0e7e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffb0e7ea04
Build id: 3.10.0-1160.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffb1007728
Build id: 3.10.0-1160.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffffb0fef295
Build id: 3.10.0-1160.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffffb0ff16bb
Build id: 3.10.0-1160.el7.x86_64
0xfb
7
handle_mm_fault
vmlinux 0xffffffffb0ff5d32
Build id: 3.10.0-1160.el7.x86_64
0xb22
8
? __schedule
vmlinux 0xffffffffb1586e82
Build id: 3.10.0-1160.el7.x86_64
0x5a2
9
__do_page_fault
vmlinux 0xffffffffb158e653
Build id: 3.10.0-1160.el7.x86_64
0x213
10
do_page_fault
vmlinux 0xffffffffb158e975
Build id: 3.10.0-1160.el7.x86_64
0x35
11
page_fault
vmlinux 0xffffffffb158a778
Build id: 3.10.0-1160.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [server.rb:286:21315]
Modules linked in: veth br_netfilter tun devlink overlay(T) bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec iosf_mbi crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq snd_seq_device aesni_intel snd_pcm lrw gf128mul ppdev glue_helper ablk_helper cryptd snd_timer vmw_balloon snd soundcore joydev sg pcspkr i2c_piix4 vmw_vmci parport_pc parport xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm nfit libnvdimm drm serio_raw mptspi e1000 scsi_transport_spi ata_piix mptscsih libata mptbase drm_panel_orientation_quirks
fuse [last unloaded: ip_tables]
CPU: 1 PID: 21315 Comm: server.rb:286 Kdump: loaded Tainted: G    B        L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9c84dfe13180 ti: ffff9c84b5864000 task.ti: ffff9c84b5864000
RIP: 0010:[<ffffffffb0f16c0a>]  [<ffffffffb0f16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff9c84b5867be8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9c85b9620600
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9c84b5867c20 R08: ffff9c85b0fe3000 R09: ffffffffb1185f99
R10: ffff9c85b965f160 R11: ffffcfd502e59200 R12: 0000000000000292
R13: 0000000000000000 R14: ffff9c84b5867b58 R15: 0000000000000292
FS:  00007f4ccd7eb700(0000) GS:ffff9c85b9640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4cbfad9b00 CR3: 000000003592e000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb0e7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb0e7ea04>] flush_tlb_page+0x54/0xa0
[<ffffffffb1007728>] ptep_clear_flush+0x68/0xa0
[<ffffffffb0fef295>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffffb0ff16bb>] do_wp_page+0xfb/0x720
[<ffffffffb0ff5d32>] handle_mm_fault+0xb22/0xfb0
[<ffffffffb1586e82>] ? __schedule+0x5a2/0x860
[<ffffffffb158e653>] __do_page_fault+0x213/0x500
[<ffffffffb158e975>] do_page_fault+0x35/0x90
[<ffffffffb158a778>] page_fault+0x28/0x30
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 b5 b1 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #428490
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8731722a
Build id: 3.10.0-1160.62.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8727e258
Build id: 3.10.0-1160.62.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8727e3f4
Build id: 3.10.0-1160.62.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff874080f8
Build id: 3.10.0-1160.62.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff873efbf5
Build id: 3.10.0-1160.62.1.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffff873f201b
Build id: 3.10.0-1160.62.1.el7.x86_64
0xfb
7
handle_mm_fault
vmlinux 0xffffffff873f6692
Build id: 3.10.0-1160.62.1.el7.x86_64
0xb22
8
? update_curr
vmlinux 0xffffffff872e497c
Build id: 3.10.0-1160.62.1.el7.x86_64
0x14c
9
__do_page_fault
vmlinux 0xffffffff87994653
Build id: 3.10.0-1160.62.1.el7.x86_64
0x213
10
do_page_fault
vmlinux 0xffffffff87994975
Build id: 3.10.0-1160.62.1.el7.x86_64
0x35
11
? error_swapgs
vmlinux 0xffffffff87990ac9
Build id: 3.10.0-1160.62.1.el7.x86_64
0xaa
12
page_fault
vmlinux 0xffffffff87990778
Build id: 3.10.0-1160.62.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [gdbus:6470]
Modules linked in: tcp_lp nls_utf8 isofs veth vboxsf(OE) xt_addrtype br_netfilter overlay(T) xt_CHECKSUM tun devlink xt_REDIRECT nf_nat_redirect xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_mark ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device joydev snd_pcm ppdev snd_timer vboxguest(OE) pcspkr snd sg soundcore parport_pc
video parport i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci ata_piix serio_raw e1000 libata crc32c_intel drm_panel_orientation_quirks fuse
CPU: 3 PID: 6470 Comm: gdbus Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff8905987c1080 ti: ffff89059a0dc000 task.ti: ffff89059a0dc000
RIP: 0010:[<ffffffff8731722a>]  [<ffffffff8731722a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff89059a0dfbe8  EFLAGS: 00010202
RAX: 0000000000000000 RBX: 000000000001b840 RCX: ffff89069fc20450
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff89059a0dfc20 R08: ffff890698492000 R09: ffffffff875869d9
R10: ffff89069fd9f160 R11: ffffd41fc074e200 R12: ffffffff8758692f
R13: 0000000000000002 R14: ffff89069ffd6000 R15: ffffffff873c8f14
FS:  00007f43277fe700(0000) GS:ffff89069fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f43277fed20 CR3: 00000001141a4000 CR4: 00000000000206e0
Call Trace:
[<ffffffff8727e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8727e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffff874080f8>] ptep_clear_flush+0x68/0xa0
[<ffffffff873efbf5>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffff873f201b>] do_wp_page+0xfb/0x720
[<ffffffff873f6692>] handle_mm_fault+0xb22/0xfb0
[<ffffffff872e497c>] ? update_curr+0x14c/0x1e0
[<ffffffff87994653>] __do_page_fault+0x213/0x500
[<ffffffff87994975>] do_page_fault+0x35/0x90
[<ffffffff87990ac9>] ? error_swapgs+0xaa/0xc0
[<ffffffff87990778>] page_fault+0x28/0x30
Code: 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 f5 87 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #423360
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f98e0
Build id: 3.10.0-514.el7.x86_64
0x200
2
native_flush_tlb_others
vmlinux 0xffffffff8106c4c8
Build id: 3.10.0-514.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8106c6c5
Build id: 3.10.0-514.el7.x86_64
0x55
4
ptep_clear_flush
vmlinux 0xffffffff811c0d74
Build id: 3.10.0-514.el7.x86_64
0x54
5
wp_page_copy.isra.56
vmlinux 0xffffffff811acffb
Build id: 3.10.0-514.el7.x86_64
0x24b
6
do_wp_page
vmlinux 0xffffffff811aeba5
Build id: 3.10.0-514.el7.x86_64
0xe5
7
handle_mm_fault
vmlinux 0xffffffff811b0e3c
Build id: 3.10.0-514.el7.x86_64
0x72c
8
__do_page_fault
vmlinux 0xffffffff81691a94
Build id: 3.10.0-514.el7.x86_64
0x154
9
do_page_fault
vmlinux 0xffffffff81691dc5
Build id: 3.10.0-514.el7.x86_64
0x35
10
page_fault
vmlinux 0xffffffff8168e088
Build id: 3.10.0-514.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [systemd-journal:890]
Modules linked in: binfmt_misc xfs libcrc32c 8021q garp mrp stp llc bonding nls_utf8 isofs intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ixgbe mei_me mei iTCO_wdt ptp iTCO_vendor_support i2c_i801 pps_core i2c_core mdio lpc_ich sb_edac dca edac_core sg pcspkr fjes shpchp acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci mpt3sas libata raid_class scsi_transport_sas scsi_ep_front(OE) dm_mirror dm_region_hash dm_log dm_mod
CPU: 9 PID: 890 Comm: systemd-journal Tainted: G           OEL ------------   3.10.0-514.el7.x86_64 #1
Hardware name: Huawei RH2288H V3/BC11HGSA0, BIOS 5.03 07/25/2018
task: ffff881ff90c3ec0 ti: ffff881ffdc5c000 task.ti: ffff881ffdc5c000
RIP: 0010:[<ffffffff810f98e0>]  [<ffffffff810f98e0>] smp_call_function_many+0x200/0x260
RSP: 0000:ffff881ffdc5fc58  EFLAGS: 00000202
RAX: 0000000000000024 RBX: 0000000000017780 RCX: ffff881ffff9af60
RDX: 0000000000000024 RSI: 0000000000000038 RDI: 0000000000000000
RBP: ffff881ffdc5fc90 R08: ffff880173b32000 R09: ffffffff81317a89
R10: ffff881fffc59b40 R11: ffffea00009c1200 R12: ffff881ffdc5fc08
R13: 000280da00000000 R14: ffffffffffffffff R15: 0000000000000002
FS:  00007f2a4b83c880(0000) GS:ffff881fffc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2a4cb4de00 CR3: 0000003ffcfba000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
00000001fdc5fcb0 ffff883ffd100fc8 ffff883ffd100c80 00007f2a4cb4de00
0000000000000000 ffff883ffd100fc8 ffff880fd2b5fa68 ffff881ffdc5fce0
ffffffff8106c4c8 ffff883ffd100c80 00007f2a4cb4de00 0000000000000000
Call Trace:
[<ffffffff8106c4c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c6c5>] flush_tlb_page+0x55/0xa0
[<ffffffff811c0d74>] ptep_clear_flush+0x54/0x80
[<ffffffff811acffb>] wp_page_copy.isra.56+0x24b/0x470
[<ffffffff811aeba5>] do_wp_page+0xe5/0x530
[<ffffffff811b0e3c>] handle_mm_fault+0x72c/0xfe0
[<ffffffff81691a94>] __do_page_fault+0x154/0x450
[<ffffffff81691dc5>] do_page_fault+0x35/0x90
[<ffffffff8168e088>] page_fault+0x28/0x30
Code: 23 00 48 63 35 76 9e 9e 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 57 ad 81 f6 41 20 01 74 cd 0f 1f 44 00 00 <f3> 90 f6 41 20 01 75 f8 48 63 35 45 9e 9e 00 eb b7 0f b6 4d cc 
Complete report #367864
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff82f16c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff82e7e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff82e7ea04
Build id: 3.10.0-1160.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff83007728
Build id: 3.10.0-1160.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff82fef295
Build id: 3.10.0-1160.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffff82ff16bb
Build id: 3.10.0-1160.el7.x86_64
0xfb
7
handle_mm_fault
vmlinux 0xffffffff82ff5d32
Build id: 3.10.0-1160.el7.x86_64
0xb22
8
__do_page_fault
vmlinux 0xffffffff8358e653
Build id: 3.10.0-1160.el7.x86_64
0x213
9
do_page_fault
vmlinux 0xffffffff8358e975
Build id: 3.10.0-1160.el7.x86_64
0x35
10
? error_swapgs
vmlinux 0xffffffff8358aac9
Build id: 3.10.0-1160.el7.x86_64
0xaa
11
page_fault
vmlinux 0xffffffff8358a778
Build id: 3.10.0-1160.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 66s! [airflow schedul:16646]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) sunrpc snd_hda_codec_idt snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm ppdev snd_timer snd pcspkr parport_pc soundcore sg joydev i2c_piix4 parport video ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ahci libahci serio_raw libata e1000 crc32c_intel dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 16646 Comm: airflow schedul Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff96ce094d0000 ti: ffff96cecd87c000 task.ti: ffff96cecd87c000
RIP: 0010:[<ffffffff82f16c0a>]  [<ffffffff82f16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff96cecd87fbe8  EFLAGS: 00010202
RAX: 0000000000000001 RBX: 000000000001b840 RCX: ffff96cf5f49fc10
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff96cecd87fc20 R08: ffff96cf3fd52000 R09: ffffffff83185f99
R10: ffff96cf5f41f160 R11: ffffdf2d41256f00 R12: ffffffff83185eef
R13: 0000000000000002 R14: 0000000000000030 R15: 0000000082f16562
FS:  00007f6079c990c0(0000) GS:ffff96cf5f400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f60791079b0 CR3: 000000019748a000 CR4: 00000000000006f0
Call Trace:
[<ffffffff82e7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff82e7ea04>] flush_tlb_page+0x54/0xa0
[<ffffffff83007728>] ptep_clear_flush+0x68/0xa0
[<ffffffff82fef295>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffff82ff16bb>] do_wp_page+0xfb/0x720
[<ffffffff82ff5d32>] handle_mm_fault+0xb22/0xfb0
[<ffffffff8358e653>] __do_page_fault+0x213/0x500
[<ffffffff8358e975>] do_page_fault+0x35/0x90
[<ffffffff8358aac9>] ? error_swapgs+0xaa/0xc0
[<ffffffff8358a778>] page_fault+0x28/0x30
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 b5 83 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #286750
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbab1173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbaa7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffbaa7a734
Build id: 3.10.0-957.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffbabfd587
Build id: 3.10.0-957.el7.x86_64
0x67
5
wp_page_copy.isra.71
vmlinux 0xffffffffbabe55bb
Build id: 3.10.0-957.el7.x86_64
0x30b
6
do_wp_page
vmlinux 0xffffffffbabe782b
Build id: 3.10.0-957.el7.x86_64
0xfb
7
? futex_wait
vmlinux 0xffffffffbab0db55
Build id: 3.10.0-957.el7.x86_64
0x195
8
handle_pte_fault
vmlinux 0xffffffffbabe91bd
Build id: 3.10.0-957.el7.x86_64
0x37d
9
handle_mm_fault
vmlinux 0xffffffffbabebc6d
Build id: 3.10.0-957.el7.x86_64
0x39d
10
__do_page_fault
vmlinux 0xffffffffbb16f5e3
Build id: 3.10.0-957.el7.x86_64
0x203
11
do_page_fault
vmlinux 0xffffffffbb16f915
Build id: 3.10.0-957.el7.x86_64
0x35
12
? error_swapgs
vmlinux 0xffffffffbb16ba96
Build id: 3.10.0-957.el7.x86_64
0xa7
13
page_fault
vmlinux 0xffffffffbb16b758
Build id: 3.10.0-957.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kubectl:31046]
Modules linked in: vsock_diag tcp_diag udp_diag inet_diag unix_diag binfmt_misc nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ip_vs nfs nf_conntrack lockd grace libcrc32c fscache vmw_vsock_vmci_transport vsock sunrpc ppdev coretemp joydev pcspkr sg vmw_balloon parport_pc parport i2c_piix4 vmw_vmci ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix vmxnet3 mptspi scsi_transport_spi libata mptscsih drm_panel_orientation_quirks mptbase dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 31046 Comm: kubectl Tainted: G        W    L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff924b759ce180 ti: ffff924b411a0000 task.ti: ffff924b411a0000
RIP: 0010:[<ffffffffbab1173a>]  [<ffffffffbab1173a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff924b411a3b70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000baa7a3d0 RCX: ffff924b7fc20490
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff924b411a3ba8 R08: ffff924b7955cc00 R09: ffffffffbad755c9
R10: ffff924b7fc9f160 R11: fffff28942eace00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff924b411a3b20 R15: ffffffffffffffff
FS:  000000c000482090(0000) GS:ffff924b7fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0009b80b8 CR3: 0000000079d9e000 CR4: 00000000000007e0
Call Trace:
[<ffffffffbaa7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbaa7a734>] flush_tlb_page+0x54/0xa0
[<ffffffffbabfd587>] ptep_clear_flush+0x67/0xa0
[<ffffffffbabe55bb>] wp_page_copy.isra.71+0x30b/0x560
[<ffffffffbabe782b>] do_wp_page+0xfb/0x720
[<ffffffffbab0db55>] ? futex_wait+0x195/0x280
[<ffffffffbabe91bd>] handle_pte_fault+0x37d/0xd10
[<ffffffffbabebc6d>] handle_mm_fault+0x39d/0x9b0
[<ffffffffbb16f5e3>] __do_page_fault+0x203/0x500
[<ffffffffbb16f915>] do_page_fault+0x35/0x90
[<ffffffffbb16ba96>] ? error_swapgs+0xa7/0xbd
[<ffffffffbb16b758>] page_fault+0x28/0x30
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 74 bb f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #241324
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff93d1173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff93c7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff93c7a734
Build id: 3.10.0-957.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff93dfd587
Build id: 3.10.0-957.el7.x86_64
0x67
5
wp_page_copy.isra.71
vmlinux 0xffffffff93de55bb
Build id: 3.10.0-957.el7.x86_64
0x30b
6
do_wp_page
vmlinux 0xffffffff93de782b
Build id: 3.10.0-957.el7.x86_64
0xfb
7
? try_to_wake_up
vmlinux 0xffffffff93cd64f0
Build id: 3.10.0-957.el7.x86_64
0x190
8
handle_pte_fault
vmlinux 0xffffffff93de91bd
Build id: 3.10.0-957.el7.x86_64
0x37d
9
handle_mm_fault
vmlinux 0xffffffff93debc6d
Build id: 3.10.0-957.el7.x86_64
0x39d
10
__do_page_fault
vmlinux 0xffffffff9436f5e3
Build id: 3.10.0-957.el7.x86_64
0x203
11
do_page_fault
vmlinux 0xffffffff9436f915
Build id: 3.10.0-957.el7.x86_64
0x35
12
page_fault
vmlinux 0xffffffff9436b758
Build id: 3.10.0-957.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [ParGC Thread#3:57170]
Modules linked in: rpcsec_gss_krb5 nfsd auth_rpcgss nfs_acl lockd grace vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 iosf_mbi crc32_pclmul snd_rawmidi snd_ac97_codec ghash_clmulni_intel ac97_bus snd_seq aesni_intel snd_seq_device lrw snd_pcm gf128mul glue_helper ablk_helper snd_timer cryptd snd ppdev vmw_balloon soundcore sg joydev pcspkr parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel mptspi ata_piix drm scsi_transport_spi serio_raw mptscsih libata e1000 nfit mptbase libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 28 PID: 57170 Comm: ParGC Thread#3 Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff8d9310c7b0c0 ti: ffff8d8df070c000 task.ti: ffff8d8df070c000
RIP: 0010:[<ffffffff93d1173a>]  [<ffffffff93d1173a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8d8df070fb70  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 0008000093c7a3d0 RCX: ffff8d931eee09e8
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8d8df070fba8 R08: ffff8d84e9782000 R09: ffffffff93f755c9
R10: ffff8d931f51f160 R11: ffffc7c41d1a8c00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8d8df070fb20 R15: ffffffffffffffff
FS:  00007fb3d6b07700(0000) GS:ffff8d931f500000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb4053e12c8 CR3: 0000000b032ea000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff93c7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff93c7a734>] flush_tlb_page+0x54/0xa0
[<ffffffff93dfd587>] ptep_clear_flush+0x67/0xa0
[<ffffffff93de55bb>] wp_page_copy.isra.71+0x30b/0x560
[<ffffffff93de782b>] do_wp_page+0xfb/0x720
[<ffffffff93cd64f0>] ? try_to_wake_up+0x190/0x390
[<ffffffff93de91bd>] handle_pte_fault+0x37d/0xd10
[<ffffffff93debc6d>] handle_mm_fault+0x39d/0x9b0
[<ffffffff9436f5e3>] __do_page_fault+0x203/0x500
[<ffffffff9436f915>] do_page_fault+0x35/0x90
[<ffffffff9436b758>] page_fault+0x28/0x30
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 94 94 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #186809
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8106e864
Build id: 3.10.0-693.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff811c2524
Build id: 3.10.0-693.el7.x86_64
0x54
5
wp_page_copy.isra.58
vmlinux 0xffffffff811adfe0
Build id: 3.10.0-693.el7.x86_64
0x240
6
do_wp_page
vmlinux 0xffffffff811afc1b
Build id: 3.10.0-693.el7.x86_64
0xeb
7
handle_mm_fault
vmlinux 0xffffffff811b1f8c
Build id: 3.10.0-693.el7.x86_64
0x70c
8
__do_page_fault
vmlinux 0xffffffff816affb4
Build id: 3.10.0-693.el7.x86_64
0x154
9
do_page_fault
vmlinux 0xffffffff816b02e5
Build id: 3.10.0-693.el7.x86_64
0x35
10
page_fault
vmlinux 0xffffffff816ac508
Build id: 3.10.0-693.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [containerd:1858]
Modules linked in: overlay(T) vmw_vsock_vmci_transport vsock sunrpc edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg vmw_vmci vmw_balloon ppdev shpchp pcspkr i2c_piix4 joydev nfit libnvdimm parport_pc parport br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect crc_t10dif sysimgblt crct10dif_generic fb_sys_fops ttm drm ahci ata_piix libahci libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 i2c_core vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod
CPU: 18 PID: 1858 Comm: containerd Tainted: G               ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff88042debcf10 ti: ffff880428bbc000 task.ti: ffff880428bbc000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0000:ffff880428bbfc50  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000800008106e4a0 RCX: ffff88042fadb238
RDX: 0000000000000003 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff880428bbfc88 R08: ffff8804ba56ec00 R09: ffffffff813227d9
R10: ffff88083f699c80 R11: ffffea0002ee7600 R12: 0000000000000286
R13: 0000000000000286 R14: ffff880428bbfc00 R15: 000280da00000000
FS:  00007f79cffff700(0000) GS:ffff88083f680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f79cc7a1a78 CR3: 000000042832c000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000128bbfca8 ffff88042a8f6d88 ffff88042a8f6a40 00007f79cc7a1a78
0000000000000000 ffff88042a8f6d88 ffff880805258d08 ffff880428bbfcd8
ffffffff8106e668 ffff88042a8f6a40 00007f79cc7a1a78 0000000000000000
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e864>] flush_tlb_page+0x54/0xa0
[<ffffffff811c2524>] ptep_clear_flush+0x54/0x80
[<ffffffff811adfe0>] wp_page_copy.isra.58+0x240/0x470
[<ffffffff811afc1b>] do_wp_page+0xeb/0x5c0
[<ffffffff811b1f8c>] handle_mm_fault+0x70c/0xfa0
[<ffffffff816affb4>] __do_page_fault+0x154/0x450
[<ffffffff816b02e5>] do_page_fault+0x35/0x90
[<ffffffff816ac508>] page_fault+0x28/0x30
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #162425
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff84916e4a
Build id: 3.10.0-1160.15.2.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8487e868
Build id: 3.10.0-1160.15.2.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8487ea04
Build id: 3.10.0-1160.15.2.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff84a07908
Build id: 3.10.0-1160.15.2.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff849ef415
Build id: 3.10.0-1160.15.2.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffff849f183b
Build id: 3.10.0-1160.15.2.el7.x86_64
0xfb
7
? hrtimer_get_res
vmlinux 0xffffffff848c9a50
Build id: 3.10.0-1160.15.2.el7.x86_64
0x50
8
handle_mm_fault
vmlinux 0xffffffff849f5eb2
Build id: 3.10.0-1160.15.2.el7.x86_64
0xb22
9
__do_page_fault
vmlinux 0xffffffff84f8f653
Build id: 3.10.0-1160.15.2.el7.x86_64
0x213
10
do_page_fault
vmlinux 0xffffffff84f8f975
Build id: 3.10.0-1160.15.2.el7.x86_64
0x35
11
page_fault
vmlinux 0xffffffff84f8b778
Build id: 3.10.0-1160.15.2.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [node_exporter:4124]
Modules linked in: xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device snd_pcm aesni_intel lrw snd_timer gf128mul glue_helper ppdev ablk_helper sg snd vmw_balloon cryptd pcspkr vmw_vmci soundcore joydev i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi
vmwgfx e1000 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul drm crct10dif_common ata_piix crc32c_intel libata mptspi nfit scsi_transport_spi serio_raw mptscsih libnvdimm mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 4124 Comm: node_exporter Kdump: loaded Tainted: G               ------------ T 3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff9ea972ad5280 ti: ffff9ea972b7c000 task.ti: ffff9ea972b7c000
RIP: 0010:[<ffffffff84916e4a>]  [<ffffffff84916e4a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff9ea972b7fbe8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00000000000000fc RCX: ffff9eaa31660628
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9ea972b7fc20 R08: ffff9ea868fcb400 R09: ffffffff84b86479
R10: ffff9eaa3169f160 R11: ffffeba305b3f200 R12: 0000000000000292
R13: ffff9ea972b7fb98 R14: 0000000000000030 R15: 0000000072b7fb60
FS:  00000000011b32d0(0063) GS:ffff9eaa31680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb6a2c474f7 CR3: 000000026cd82000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff8487e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8487ea04>] flush_tlb_page+0x54/0xa0
[<ffffffff84a07908>] ptep_clear_flush+0x68/0xa0
[<ffffffff849ef415>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffff849f183b>] do_wp_page+0xfb/0x720
[<ffffffff848c9a50>] ? hrtimer_get_res+0x50/0x50
[<ffffffff849f5eb2>] handle_mm_fault+0xb22/0xfb0
[<ffffffff84f8f653>] __do_page_fault+0x213/0x500
[<ffffffff84f8f975>] do_page_fault+0x35/0x90
[<ffffffff84f8b778>] page_fault+0x28/0x30
Code: 48 63 35 de 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 55 85 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ad 86 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #135133
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f98e2
Build id: 3.10.0-514.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106c4c8
Build id: 3.10.0-514.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8106c6c5
Build id: 3.10.0-514.el7.x86_64
0x55
4
ptep_clear_flush
vmlinux 0xffffffff811c0d74
Build id: 3.10.0-514.el7.x86_64
0x54
5
wp_page_copy.isra.56
vmlinux 0xffffffff811acffb
Build id: 3.10.0-514.el7.x86_64
0x24b
6
do_wp_page
vmlinux 0xffffffff811aeba5
Build id: 3.10.0-514.el7.x86_64
0xe5
7
handle_mm_fault
vmlinux 0xffffffff811b0e3c
Build id: 3.10.0-514.el7.x86_64
0x72c
8
__do_page_fault
vmlinux 0xffffffff81691a94
Build id: 3.10.0-514.el7.x86_64
0x154
9
do_page_fault
vmlinux 0xffffffff81691dc5
Build id: 3.10.0-514.el7.x86_64
0x35
10
page_fault
vmlinux 0xffffffff8168e088
Build id: 3.10.0-514.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [docker:22671]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 dm_mirror dm_region_hash dm_log intel_powerclamp dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
ablk_helper cryptd ppdev vmw_balloon pcspkr sg shpchp vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod cdrom crct10dif_generic ata_generic pata_acpi vmwgfx ata_piix drm_kms_helper mptspi crct10dif_pclmul scsi_transport_spi crct10dif_common mptscsih crc32c_intel syscopyarea mptbase sysfillrect sysimgblt fb_sys_fops ttm serio_raw libata drm vmxnet3 i2c_core floppy fjes
CPU: 1 PID: 22671 Comm: docker Tainted: G               ------------ T 3.10.0-514.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8807dc305e20 ti: ffff8807c9f74000 task.ti: ffff8807c9f74000
RIP: 0010:[<ffffffff810f98e2>]  [<ffffffff810f98e2>] smp_call_function_many+0x202/0x260
RSP: 0000:ffff8807c9f77c58  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 0000000000017780 RCX: ffff88081e69a678
RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff8807c9f77c90 R08: ffff88017cab3800 R09: ffffffff81317a89
R10: ffff88081e659b40 R11: ffffea001bed8000 R12: ffff8807c9f77c08
R13: 000280da00000000 R14: ffffffffffffffff R15: 0000000000000002
FS:  00007f7900b34740(0000) GS:ffff88081e640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c4205d9860 CR3: 00000007c8260000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001c9f77cb0 ffff88081b4f0988 ffff88081b4f0640 000000c4205d9860
0000000000000000 ffff88081b4f0988 ffff8807cd28dec8 ffff8807c9f77ce0
ffffffff8106c4c8 ffff88081b4f0640 000000c4205d9860 0000000000000000
Call Trace:
[<ffffffff8106c4c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c6c5>] flush_tlb_page+0x55/0xa0
[<ffffffff811c0d74>] ptep_clear_flush+0x54/0x80
[<ffffffff811acffb>] wp_page_copy.isra.56+0x24b/0x470
[<ffffffff811aeba5>] do_wp_page+0xe5/0x530
[<ffffffff811b0e3c>] handle_mm_fault+0x72c/0xfe0
[<ffffffff81691a94>] __do_page_fault+0x154/0x450
[<ffffffff81691dc5>] do_page_fault+0x35/0x90
[<ffffffff8168e088>] page_fault+0x28/0x30
Code: 48 63 35 76 9e 9e 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 57 ad 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 45 9e 9e 00 eb b7 0f b6 4d cc 4c 89 
Complete report #65968
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa3d12b6e
Build id: 3.10.0-957.27.2.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffa3c7b788
Build id: 3.10.0-957.27.2.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffa3c7b924
Build id: 3.10.0-957.27.2.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffa3dfecc7
Build id: 3.10.0-957.27.2.el7.x86_64
0x67
5
wp_page_copy.isra.71
vmlinux 0xffffffffa3de6d1b
Build id: 3.10.0-957.27.2.el7.x86_64
0x30b
6
do_wp_page
vmlinux 0xffffffffa3de8f6b
Build id: 3.10.0-957.27.2.el7.x86_64
0xfb
7
? do_last
vmlinux 0xffffffffa3e50fed
Build id: 3.10.0-957.27.2.el7.x86_64
0x66d
8
handle_pte_fault
vmlinux 0xffffffffa3dea8fd
Build id: 3.10.0-957.27.2.el7.x86_64
0x37d
9
handle_mm_fault
vmlinux 0xffffffffa3ded3ad
Build id: 3.10.0-957.27.2.el7.x86_64
0x39d
10
__do_page_fault
vmlinux 0xffffffffa4371603
Build id: 3.10.0-957.27.2.el7.x86_64
0x203
11
do_page_fault
vmlinux 0xffffffffa4371925
Build id: 3.10.0-957.27.2.el7.x86_64
0x35
12
page_fault
vmlinux 0xffffffffa436d768
Build id: 3.10.0-957.27.2.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 198s! [kubelet:14949]
INFO: rcu_sched detected stalls on CPUs/tasks:
Modules linked in: xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 tun ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_conntrack ebtable_filter ebtables ip6table_filter ip6_tables devlink xt_addrtype xt_set xt_mark ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_bitmap_port ip_set_hash_ipport ip_set nfnetlink dummy xt_comment iptable_nat nf_nat_ipv4 nf_nat overlay(T) nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg i2c_piix4 parport_pc parport vmw_vmci br_netfilter
bridge stp llc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix crct10dif_pclmul crct10dif_common scsi_transport_spi crc32c_intel mptscsih libata serio_raw nfit mptbase libnvdimm vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 14949 Comm: kubelet Tainted: G             L ------------ T 3.10.0-957.27.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9647ea6830c0 ti: ffff9646d8f10000 task.ti: ffff9646d8f10000
RIP: 0010:[<ffffffffa3d12b6e>]  [<ffffffffa3d12b6e>] smp_call_function_many+0x20e/0x270
RSP: 0000:ffff9646d8f13b70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a3c7b5c0 RCX: ffff9647ffc204b8
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff9646d8f13ba8 R08: ffff96453a34f000 R09: ffffffffa3f774a9
R10: ffff9647ffc9f160 R11: ffffee0b8217f400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9646d8f13b20 R15: ffffffffffffffff
FS:  00007f67e667b700(0000) GS:ffff9647ffc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f67ea1b8838 CR3: 000000007fc82000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa3c7b788>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa3c7b924>] flush_tlb_page+0x54/0xa0
[<ffffffffa3dfecc7>] ptep_clear_flush+0x67/0xa0
[<ffffffffa3de6d1b>] wp_page_copy.isra.71+0x30b/0x560
[<ffffffffa3de8f6b>] do_wp_page+0xfb/0x720
[<ffffffffa3e50fed>] ? do_last+0x66d/0x12a0
[<ffffffffa3dea8fd>] handle_pte_fault+0x37d/0xd10
[<ffffffffa3ded3ad>] handle_mm_fault+0x39d/0x9b0
[<ffffffffa4371603>] __do_page_fault+0x203/0x4f0
[<ffffffffa4371925>] do_page_fault+0x35/0x90
[<ffffffffa436d768>] page_fault+0x28/0x30
Code: 75 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 bf 94 a4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 2d 75 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #42131
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f9a32
Build id: 3.10.0-514.6.1.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106c5f8
Build id: 3.10.0-514.6.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8106c7f5
Build id: 3.10.0-514.6.1.el7.x86_64
0x55
4
ptep_clear_flush
vmlinux 0xffffffff811c0ec4
Build id: 3.10.0-514.6.1.el7.x86_64
0x54
5
wp_page_copy.isra.56
vmlinux 0xffffffff811ad14b
Build id: 3.10.0-514.6.1.el7.x86_64
0x24b
6
do_wp_page
vmlinux 0xffffffff811aecf5
Build id: 3.10.0-514.6.1.el7.x86_64
0xe5
7
? do_read_fault.isra.42
vmlinux 0xffffffff811ac824
Build id: 3.10.0-514.6.1.el7.x86_64
0xe4
8
handle_mm_fault
vmlinux 0xffffffff811b0f8c
Build id: 3.10.0-514.6.1.el7.x86_64
0x72c
9
__do_page_fault
vmlinux 0xffffffff81691d94
Build id: 3.10.0-514.6.1.el7.x86_64
0x154
10
do_page_fault
vmlinux 0xffffffff816920c5
Build id: 3.10.0-514.6.1.el7.x86_64
0x35
11
page_fault
vmlinux 0xffffffff8168e388
Build id: 3.10.0-514.6.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#42 stuck for 23s! [postgres:123617]
Modules linked in: nfsv3 nfs fscache binfmt_misc arc4 md4 nls_utf8 cifs dns_resolver xt_time xt_recent nf_log_ipv4 nf_log_common xt_LOG xt_limit ipt_REJECT nf_reject_ipv4 vmw_vsock_vmci_transport nf_conntrack_ipv4 vsock nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter dm_mirror dm_region_hash dm_log dm_mod ppdev vmw_balloon intel_powerclamp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr i2c_piix4 shpchp vmw_vmci parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul ttm mptspi crct10dif_common crc32c_intel scsi_transport_spi mptscsih drm ata_piix serio_raw libata mptbase e1000 i2c_core fjes
CPU: 42 PID: 123617 Comm: postgres Tainted: G        W    L ------------   3.10.0-514.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff889dba393ec0 ti: ffff881416cec000 task.ti: ffff881416cec000
RIP: 0010:[<ffffffff810f9a32>]  [<ffffffff810f9a32>] smp_call_function_many+0x202/0x260
RSP: 0000:ffff881416cefc58  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff889dbd61b410
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff881416cefc90 R08: ffff88011efdd800 R09: ffffffff81317bf9
R10: ffff889dbe099b40 R11: ffffea005dcab400 R12: 0000000000000292
R13: ffff881416cefc08 R14: ffffffffffffffff R15: 0000000000000002
FS:  00007f150eec5740(0000) GS:ffff889dbe080000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f150ee7e438 CR3: 00000004961b0000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
 0000000116cefcb0 ffff8803e97cc1c8 ffff8803e97cbe80 00007f150ee7e438
 0000000000000000 ffff8803e97cc1c8 ffff881336d7a3f0 ffff881416cefce0
 ffffffff8106c5f8 ffff8803e97cbe80 00007f150ee7e438 0000000000000000
Call Trace:
 [<ffffffff8106c5f8>] native_flush_tlb_others+0xb8/0xc0
 [<ffffffff8106c7f5>] flush_tlb_page+0x55/0xa0
 [<ffffffff811c0ec4>] ptep_clear_flush+0x54/0x80
 [<ffffffff811ad14b>] wp_page_copy.isra.56+0x24b/0x470
 [<ffffffff811aecf5>] do_wp_page+0xe5/0x530
 [<ffffffff811ac824>] ? do_read_fault.isra.42+0xe4/0x130
 [<ffffffff811b0f8c>] handle_mm_fault+0x72c/0xfe0
 [<ffffffff81691d94>] __do_page_fault+0x154/0x450
 [<ffffffff816920c5>] do_page_fault+0x35/0x90
 [<ffffffff8168e388>] page_fault+0x28/0x30
Code: 48 63 35 66 9d 9e 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 80 57 ad 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 35 9d 9e 00 eb b7 0f b6 4d cc 4c 89 
Complete report #55
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff85b1761a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff85a7e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff85a7ea94
Build id: 3.10.0-1127.19.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff85c07888
Build id: 3.10.0-1127.19.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff85bef3c5
Build id: 3.10.0-1127.19.1.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffff85bf17eb
Build id: 3.10.0-1127.19.1.el7.x86_64
0xfb
7
handle_mm_fault
vmlinux 0xffffffff85bf5e82
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb22
8
__do_page_fault
vmlinux 0xffffffff8618d653
Build id: 3.10.0-1127.19.1.el7.x86_64
0x213
9
do_page_fault
vmlinux 0xffffffff8618d975
Build id: 3.10.0-1127.19.1.el7.x86_64
0x35
10
page_fault
vmlinux 0xffffffff86189778
Build id: 3.10.0-1127.19.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [/usr/libexec/we:4532]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink arc4 md4 nls_utf8 cifs ccm dns_resolver ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device snd_pcm aesni_intel lrw snd_timer gf128mul snd glue_helper ablk_helper
soundcore cryptd ppdev pcspkr joydev sg i2c_piix4 vmw_balloon vmw_vmci parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi scsi_transport_spi ahci mptscsih ata_piix nfit libahci crct10dif_pclmul crct10dif_common libnvdimm libata e1000 mptbase crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]
CPU: 15 PID: 4532 Comm: /usr/libexec/we Tainted: G             L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9e4de4e120e0 ti: ffff9e4d15a10000 task.ti: ffff9e4d15a10000
RIP: 0010:[<ffffffff85b1761a>]  [<ffffffff85b1761a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff9e4d15a13be8  EFLAGS: 00000202
RAX: 0000000000000009 RBX: 00000000000000fc RCX: ffff9e4e09e60830
RDX: 0000000000000009 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9e4d15a13c20 R08: ffff9e4dfd60e800 R09: ffffffff85d851b9
R10: ffff9e4e09fdf160 R11: ffffeea1c6216e00 R12: 0000000000000292
R13: 0000000000000000 R14: ffff9e4d15a13b58 R15: 0000000000000292
FS:  00007f28c11cd740(0000) GS:ffff9e4e09fc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000023ac2a0 CR3: 0000000094610000 CR4: 00000000003407e0
Call Trace:
[<ffffffff85a7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff85a7ea94>] flush_tlb_page+0x54/0xa0
[<ffffffff85c07888>] ptep_clear_flush+0x68/0xa0
[<ffffffff85bef3c5>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffff85bf17eb>] do_wp_page+0xfb/0x720
[<ffffffff85bf5e82>] handle_mm_fault+0xb22/0xfb0
[<ffffffff8618d653>] __do_page_fault+0x213/0x500
[<ffffffff8618d975>] do_page_fault+0x35/0x90
[<ffffffff86189778>] page_fault+0x28/0x30
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 75 86 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89