Info

Permalink
Function
smp_call_function_many
First occurrence
2020-11-12
Last occurrence
2026-04-24
State
NEW
Components
kernel
Quality
-23

Statistics

Operating system Count
CentOS 7 35
Architecture Count
x86_64 35
Related packages Count
kernel 35
 0:3.10.0-862.el7 10
 0:3.10.0-1160.62.1.el7 6
 0:3.10.0-957.el7 5
 0:3.10.0-1160.49.1.el7 3
 0:3.10.0-1160.el7 3
 0:3.10.0-1127.19.1.el7 2
 0:3.10.0-1062.18.1.el7 1
 0:3.10.0-1160.119.1.el7 1
 0:3.10.0-1160.92.1.el7 1
 0:3.10.0-1160.76.1.el7 1
 0:3.10.0-1160.88.1.el7 1
 0:3.10.0-1160.118.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #1423469
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa5716c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa567e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffa567ea04
Build id: 3.10.0-1160.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffa5807728
Build id: 3.10.0-1160.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffffa57ef295
Build id: 3.10.0-1160.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffffa57f16bb
Build id: 3.10.0-1160.el7.x86_64
0xfb
7
? poll_schedule_timeout
vmlinux 0xffffffffa5863f35
Build id: 3.10.0-1160.el7.x86_64
0x55
8
handle_mm_fault
vmlinux 0xffffffffa57f5d32
Build id: 3.10.0-1160.el7.x86_64
0xb22
9
__do_page_fault
vmlinux 0xffffffffa5d8e653
Build id: 3.10.0-1160.el7.x86_64
0x213
10
do_page_fault
vmlinux 0xffffffffa5d8e975
Build id: 3.10.0-1160.el7.x86_64
0x35
11
page_fault
vmlinux 0xffffffffa5d8a778
Build id: 3.10.0-1160.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [bt-task.run:3251]
Modules linked in: binfmt_misc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink xt_set ip_set_hash_net ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq ppdev snd_seq_device snd_pcm aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper
cryptd joydev snd_timer pcspkr snd soundcore sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw ttm mptspi ata_piix scsi_transport_spi drm e1000 nfit libata libnvdimm mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 3251 Comm: bt-task.run Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8d3effc66300 ti: ffff8d3effd10000 task.ti: ffff8d3effd10000
RIP: 0010:[<ffffffffa5716c0a>]  [<ffffffffa5716c0a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8d3effd13be8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8d3f3b620488
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8d3effd13c20 R08: ffff8d3f377db000 R09: ffffffffa5985f99
R10: ffff8d3f3b65f160 R11: ffffdced8035e200 R12: 0000000000000292
R13: ffff8d3effd13b98 R14: 0000000000000030 R15: 00000000ffd13b58
FS:  00001c63b2901090(0000) GS:ffff8d3f3b640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00001c63b2da2000 CR3: 000000003fc00000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa567e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa567ea04>] flush_tlb_page+0x54/0xa0
[<ffffffffa5807728>] ptep_clear_flush+0x68/0xa0
[<ffffffffa57ef295>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffffa57f16bb>] do_wp_page+0xfb/0x720
[<ffffffffa5863f35>] ? poll_schedule_timeout+0x55/0xc0
[<ffffffffa57f5d32>] handle_mm_fault+0xb22/0xfb0
[<ffffffffa5d8e653>] __do_page_fault+0x213/0x500
[<ffffffffa5d8e975>] do_page_fault+0x35/0x90
[<ffffffffa5d8a778>] page_fault+0x28/0x30
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 35 a6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1016252
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffaf11173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaf07a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffaf07a734
Build id: 3.10.0-957.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffaf1fd587
Build id: 3.10.0-957.el7.x86_64
0x67
5
wp_page_copy.isra.71
vmlinux 0xffffffffaf1e55bb
Build id: 3.10.0-957.el7.x86_64
0x30b
6
do_wp_page
vmlinux 0xffffffffaf1e782b
Build id: 3.10.0-957.el7.x86_64
0xfb
7
? inet_recvmsg
vmlinux 0xffffffffaf6c0060
Build id: 3.10.0-957.el7.x86_64
0x80
8
? sock_aio_read.part.9
vmlinux 0xffffffffaf6186ec
Build id: 3.10.0-957.el7.x86_64
0x14c
9
handle_pte_fault
vmlinux 0xffffffffaf1e91bd
Build id: 3.10.0-957.el7.x86_64
0x37d
10
handle_mm_fault
vmlinux 0xffffffffaf1ebc6d
Build id: 3.10.0-957.el7.x86_64
0x39d
11
__do_page_fault
vmlinux 0xffffffffaf76f5e3
Build id: 3.10.0-957.el7.x86_64
0x203
12
do_page_fault
vmlinux 0xffffffffaf76f915
Build id: 3.10.0-957.el7.x86_64
0x35
13
page_fault
vmlinux 0xffffffffaf76b758
Build id: 3.10.0-957.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [serverLibDown:123559]
Modules linked in: tcp_lp nls_utf8 isofs fuse binfmt_misc tcp_diag inet_diag xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw
e1000 nfit libnvdimm ata_generic vmwgfx pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi ata_piix mptscsih drm libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 123559 Comm: serverLibDown Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8f5242d430c0 ti: ffff8f524b0d4000 task.ti: ffff8f524b0d4000
RIP: 0010:[<ffffffffaf11173a>]  [<ffffffffaf11173a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8f524b0d7b70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000af07a3d0 RCX: ffff8f566d620650
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8f524b0d7ba8 R08: ffff8f53b9cb4400 R09: ffffffffaf3755c9
R10: ffff8f566d75f160 R11: fffff88184fc9400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8f524b0d7b20 R15: ffffffffffffffff
FS:  000000c000586490(0000) GS:ffff8f566d740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f34061269a0 CR3: 000000041a3b0000 CR4: 00000000003607e0
Call Trace:
[<ffffffffaf07a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaf07a734>] flush_tlb_page+0x54/0xa0
[<ffffffffaf1fd587>] ptep_clear_flush+0x67/0xa0
[<ffffffffaf1e55bb>] wp_page_copy.isra.71+0x30b/0x560
[<ffffffffaf1e782b>] do_wp_page+0xfb/0x720
[<ffffffffaf6c0060>] ? inet_recvmsg+0x80/0xb0
[<ffffffffaf6186ec>] ? sock_aio_read.part.9+0x14c/0x170
[<ffffffffaf1e91bd>] handle_pte_fault+0x37d/0xd10
[<ffffffffaf1ebc6d>] handle_mm_fault+0x39d/0x9b0
[<ffffffffaf76f5e3>] __do_page_fault+0x203/0x500
[<ffffffffaf76f915>] do_page_fault+0x35/0x90
[<ffffffffaf76b758>] page_fault+0x28/0x30
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 d4 af f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #19702
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb9d1173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb9c7a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffb9c7a734
Build id: 3.10.0-957.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffb9dfd587
Build id: 3.10.0-957.el7.x86_64
0x67
5
wp_page_copy.isra.71
vmlinux 0xffffffffb9de55bb
Build id: 3.10.0-957.el7.x86_64
0x30b
6
do_wp_page
vmlinux 0xffffffffb9de782b
Build id: 3.10.0-957.el7.x86_64
0xfb
7
? inet_recvmsg
vmlinux 0xffffffffba2c0060
Build id: 3.10.0-957.el7.x86_64
0x80
8
? sock_aio_read.part.9
vmlinux 0xffffffffba2186ec
Build id: 3.10.0-957.el7.x86_64
0x14c
9
handle_pte_fault
vmlinux 0xffffffffb9de91bd
Build id: 3.10.0-957.el7.x86_64
0x37d
10
handle_mm_fault
vmlinux 0xffffffffb9debc6d
Build id: 3.10.0-957.el7.x86_64
0x39d
11
__do_page_fault
vmlinux 0xffffffffba36f5e3
Build id: 3.10.0-957.el7.x86_64
0x203
12
do_page_fault
vmlinux 0xffffffffba36f915
Build id: 3.10.0-957.el7.x86_64
0x35
13
page_fault
vmlinux 0xffffffffba36b758
Build id: 3.10.0-957.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [plugin-manager:21287]
Modules linked in: xt_set ip_set_hash_net vxlan ip6_udp_tunnel udp_tunnel xt_nat xt_mark xfrm6_mode_tunnel xfrm4_mode_tunnel veth esp4 drbg ansi_cprng nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod coretemp iosf_mbi
crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev snd_seq_midi snd_seq_midi_event vmw_balloon joydev pcspkr sg snd_ens1371 snd_rawmidi snd_ac97_codec parport_pc parport ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper syscopyarea serio_raw sysfillrect sysimgblt fb_sys_fops ttm ata_piix mptspi libata scsi_transport_spi drm e1000 mptscsih mptbase drm_panel_orientation_quirks
CPU: 5 PID: 21287 Comm: plugin-manager Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
task: ffff90d4e3eb30c0 ti: ffff90d4e628c000 task.ti: ffff90d4e628c000
RIP: 0010:[<ffffffffb9d1173a>]  [<ffffffffb9d1173a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff90d4e628fb70  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000b9c7a3d0 RCX: ffff90d52d660650
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff90d4e628fba8 R08: ffff90d279c94400 R09: ffffffffb9f755c9
R10: ffff90d52d75f160 R11: fffffd280f058a00 R12: 0000000000000282
R13: 0000000000000000 R14: ffff90d4e628fae0 R15: 0000000000000282
FS:  00007f19037fe700(0000) GS:ffff90d52d740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f190b754ac8 CR3: 00000003e2f82000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb9c7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb9c7a734>] flush_tlb_page+0x54/0xa0
[<ffffffffb9dfd587>] ptep_clear_flush+0x67/0xa0
[<ffffffffb9de55bb>] wp_page_copy.isra.71+0x30b/0x560
[<ffffffffb9de782b>] do_wp_page+0xfb/0x720
[<ffffffffba2c0060>] ? inet_recvmsg+0x80/0xb0
[<ffffffffba2186ec>] ? sock_aio_read.part.9+0x14c/0x170
[<ffffffffb9de91bd>] handle_pte_fault+0x37d/0xd10
[<ffffffffb9debc6d>] handle_mm_fault+0x39d/0x9b0
[<ffffffffba36f5e3>] __do_page_fault+0x203/0x500
[<ffffffffba36f915>] do_page_fault+0x35/0x90
[<ffffffffba36b758>] page_fault+0x28/0x30
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 94 ba f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #830875
Frame # Function Binary Source or offset Line
1
native_flush_tlb_others
vmlinux 0xffffffffb4a81fa8
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb8
2
flush_tlb_page
vmlinux 0xffffffffb4a82144
Build id: 3.10.0-1160.88.1.el7.x86_64
0x54
3
ptep_clear_flush
vmlinux 0xffffffffb4c141b8
Build id: 3.10.0-1160.88.1.el7.x86_64
0x68
4
wp_page_copy.isra.73
vmlinux 0xffffffffb4bfb5ee
Build id: 3.10.0-1160.88.1.el7.x86_64
0x32e
5
do_wp_page
vmlinux 0xffffffffb4bfdafb
Build id: 3.10.0-1160.88.1.el7.x86_64
0xfb
6
? arch_tlb_finish_mmu
vmlinux 0xffffffffb4bfbb97
Build id: 3.10.0-1160.88.1.el7.x86_64
0x47
7
? tlb_finish_mmu
vmlinux 0xffffffffb4bfbcc3
Build id: 3.10.0-1160.88.1.el7.x86_64
0x23
8
? get_futex_key
vmlinux 0xffffffffb4b19398
Build id: 3.10.0-1160.88.1.el7.x86_64
0x1d8
9
handle_mm_fault
vmlinux 0xffffffffb4c0221a
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb3a
10
__do_page_fault
vmlinux 0xffffffffb51bf7d3
Build id: 3.10.0-1160.88.1.el7.x86_64
0x213
11
do_page_fault
vmlinux 0xffffffffb51bfb05
Build id: 3.10.0-1160.88.1.el7.x86_64
0x35
12
page_fault
vmlinux 0xffffffffb51bb7b8
Build id: 3.10.0-1160.88.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 3154s! [ThreadPoolForeg:19092]
[<ffffffffb4a81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb4a82144>] flush_tlb_page+0x54/0xa0
[<ffffffffb4c141b8>] ptep_clear_flush+0x68/0xa0
[<ffffffffb4bfb5ee>] wp_page_copy.isra.73+0x32e/0x5d0
[<ffffffffb4bfdafb>] do_wp_page+0xfb/0x730
[<ffffffffb4bfbb97>] ? arch_tlb_finish_mmu+0x47/0x80
[<ffffffffb4bfbcc3>] ? tlb_finish_mmu+0x23/0x40
[<ffffffffb4b19398>] ? get_futex_key+0x1d8/0x2d0
[<ffffffffb4c0221a>] handle_mm_fault+0xb3a/0x1190
[<ffffffffb51bf7d3>] __do_page_fault+0x213/0x510
[<ffffffffb51bfb05>] do_page_fault+0x35/0x90
[<ffffffffb51bb7b8>] page_fault+0x28/0x30
Complete report #1376142
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8351e25a
Build id: 3.10.0-1160.118.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff83481fa8
Build id: 3.10.0-1160.118.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff83482144
Build id: 3.10.0-1160.118.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff83614278
Build id: 3.10.0-1160.118.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff835fb6ae
Build id: 3.10.0-1160.118.1.el7.x86_64
0x32e
6
do_wp_page
vmlinux 0xffffffff835fdbbb
Build id: 3.10.0-1160.118.1.el7.x86_64
0xfb
7
? __mem_cgroup_commit_charge
vmlinux 0xffffffff8364bf62
Build id: 3.10.0-1160.118.1.el7.x86_64
0xe2
8
? get_futex_key
vmlinux 0xffffffff83519428
Build id: 3.10.0-1160.118.1.el7.x86_64
0x1d8
9
handle_mm_fault
vmlinux 0xffffffff836022da
Build id: 3.10.0-1160.118.1.el7.x86_64
0xb3a
10
__do_page_fault
vmlinux 0xffffffff83bbf7d3
Build id: 3.10.0-1160.118.1.el7.x86_64
0x213
11
do_page_fault
vmlinux 0xffffffff83bbfb05
Build id: 3.10.0-1160.118.1.el7.x86_64
0x35
12
page_fault
vmlinux 0xffffffff83bbb7b8
Build id: 3.10.0-1160.118.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [node:4650]
Modules linked in: nfnetlink_queue nfnetlink_log bluetooth rfkill veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM tun devlink overlay(T) ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq crc32_pclmul snd_seq_device ghash_clmulni_intel snd_pcm
aesni_intel lrw gf128mul snd_timer glue_helper snd ablk_helper soundcore ppdev sg cryptd pcspkr joydev vmw_balloon parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi crct10dif_pclmul drm crct10dif_common ata_piix nfit crc32c_intel scsi_transport_spi mptscsih libata libnvdimm e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 4650 Comm: node Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.118.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9d6f134ad280 ti: ffff9d6f2d290000 task.ti: ffff9d6f2d290000
RIP: 0010:[<ffffffff8351e25a>]  [<ffffffff8351e25a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff9d6f2d293be8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9d6ff9620600
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9d6f2d293c20 R08: ffff9d6ff6663000 R09: ffffffff8379c7f9
R10: ffff9d6ff965f160 R11: fffff2c604d3b200 R12: 0000000000000292
R13: 0000000000000000 R14: ffff9d6f2d293b58 R15: 0000000000000292
FS:  00007f1927bdab38(0000) GS:ffff9d6ff9640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f192571160e CR3: 0000000118594000 CR4: 00000000003607e0
Call Trace:
[<ffffffff83481fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff83482144>] flush_tlb_page+0x54/0xa0
[<ffffffff83614278>] ptep_clear_flush+0x68/0xa0
[<ffffffff835fb6ae>] wp_page_copy.isra.73+0x32e/0x5d0
[<ffffffff835fdbbb>] do_wp_page+0xfb/0x730
[<ffffffff8364bf62>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
[<ffffffff83519428>] ? get_futex_key+0x1d8/0x2d0
[<ffffffff836022da>] handle_mm_fault+0xb3a/0x1190
[<ffffffff83bbf7d3>] __do_page_fault+0x213/0x510
[<ffffffff83bbfb05>] do_page_fault+0x35/0x90
[<ffffffff83bbb7b8>] page_fault+0x28/0x30
Code: 48 63 35 ae 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 19 15 84 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 7d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1031723
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa9316c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa927e868
Build id: 3.10.0-1160.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffa927ea04
Build id: 3.10.0-1160.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffa9407728
Build id: 3.10.0-1160.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffffa93ef295
Build id: 3.10.0-1160.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffffa93f16bb
Build id: 3.10.0-1160.el7.x86_64
0xfb
7
? get_futex_key
vmlinux 0xffffffffa9311db8
Build id: 3.10.0-1160.el7.x86_64
0x1c8
8
? do_read_fault.isra.63
vmlinux 0xffffffffa93ee4d9
Build id: 3.10.0-1160.el7.x86_64
0x139
9
handle_mm_fault
vmlinux 0xffffffffa93f5d32
Build id: 3.10.0-1160.el7.x86_64
0xb22
10
__do_page_fault
vmlinux 0xffffffffa998e653
Build id: 3.10.0-1160.el7.x86_64
0x213
11
do_page_fault
vmlinux 0xffffffffa998e975
Build id: 3.10.0-1160.el7.x86_64
0x35
12
page_fault
vmlinux 0xffffffffa998a778
Build id: 3.10.0-1160.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [java:56555]
Modules linked in: tcp_lp nls_utf8 isofs rfcomm xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport bnep vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device crc32_pclmul snd_pcm ghash_clmulni_intel snd_timer aesni_intel btusb ppdev btrtl snd lrw btbcm gf128mul btintel glue_helper bluetooth vmw_balloon ablk_helper soundcore cryptd rfkill vmw_vmci joydev pcspkr sg i2c_piix4 parport_pc parport ip_tables ext4 mbcache jbd2 sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic
pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi crct10dif_pclmul scsi_transport_spi crct10dif_common ata_piix nfit crc32c_intel mptscsih libnvdimm serio_raw libata e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 7 PID: 56555 Comm: java Tainted: G        W    L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9b122cde5280 ti: ffff9b115de6c000 task.ti: ffff9b115de6c000
RIP: 0010:[<ffffffffa9316c0a>]  [<ffffffffa9316c0a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff9b115de6fbe8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9b12396206f0
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9b115de6fc20 R08: ffff9b1230fe4800 R09: ffffffffa9585f99
R10: ffff9b12397df160 R11: fffff1b984b36e00 R12: 0000000000000292
R13: ffff9b115de6fb98 R14: 0000000000000030 R15: 00000000397dacc0
FS:  00007f1a51735700(0000) GS:ffff9b12397c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1a52141000 CR3: 000000006e9b4000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffa927e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa927ea04>] flush_tlb_page+0x54/0xa0
[<ffffffffa9407728>] ptep_clear_flush+0x68/0xa0
[<ffffffffa93ef295>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffffa93f16bb>] do_wp_page+0xfb/0x720
[<ffffffffa9311db8>] ? get_futex_key+0x1c8/0x2c0
[<ffffffffa93ee4d9>] ? do_read_fault.isra.63+0x139/0x1b0
[<ffffffffa93f5d32>] handle_mm_fault+0xb22/0xfb0
[<ffffffffa998e653>] __do_page_fault+0x213/0x500
[<ffffffffa998e975>] do_page_fault+0x35/0x90
[<ffffffffa998a778>] page_fault+0x28/0x30
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 f5 a9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #586247
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa691718a
Build id: 3.10.0-1160.76.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa687e258
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffa687e3f4
Build id: 3.10.0-1160.76.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffa6a08108
Build id: 3.10.0-1160.76.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffffa69efc35
Build id: 3.10.0-1160.76.1.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffffa69f205b
Build id: 3.10.0-1160.76.1.el7.x86_64
0xfb
7
? hrtimer_get_res
vmlinux 0xffffffffa68c9bd0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x50
8
? get_futex_key
vmlinux 0xffffffffa6912498
Build id: 3.10.0-1160.76.1.el7.x86_64
0x1c8
9
handle_mm_fault
vmlinux 0xffffffffa69f66d2
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb22
10
__do_page_fault
vmlinux 0xffffffffa6f94653
Build id: 3.10.0-1160.76.1.el7.x86_64
0x213
11
do_page_fault
vmlinux 0xffffffffa6f94975
Build id: 3.10.0-1160.76.1.el7.x86_64
0x35
12
page_fault
vmlinux 0xffffffffa6f90778
Build id: 3.10.0-1160.76.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [10_scheduler:74862]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event sb_edac snd_ens1371 snd_rawmidi btusb iosf_mbi btrtl btbcm snd_ac97_codec ac97_bus btintel snd_seq bluetooth crc32_pclmul ghash_clmulni_intel aesni_intel snd_seq_device vmw_balloon
lrw gf128mul rfkill ppdev glue_helper pcspkr ablk_helper cryptd snd_pcm snd_timer joydev sg snd soundcore parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm crc32c_intel nfit ata_piix drm libnvdimm mptspi scsi_transport_spi libata serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks synosnap(OE) fuse
CPU: 5 PID: 74862 Comm: 10_scheduler Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff95e2d6fba100 ti: ffff95e263cf8000 task.ti: ffff95e263cf8000
RIP: 0010:[<ffffffffa691718a>]  [<ffffffffa691718a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff95e263cfbbe8  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff95e2dd6a06a0
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff95e263cfbc20 R08: ffff95dc2c57c800 R09: ffffffffa6b869f9
R10: ffff95e2dd75f160 R11: ffffd6665e778600 R12: 0000000000000292
R13: ffff95e263cfbb98 R14: ffff95e2fffda000 R15: ffffffffa69c8f44
FS:  00007f384608c700(0000) GS:ffff95e2dd740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f380c24b000 CR3: 00000007f4320000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa687e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa687e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffa6a08108>] ptep_clear_flush+0x68/0xa0
[<ffffffffa69efc35>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffffa69f205b>] do_wp_page+0xfb/0x720
[<ffffffffa68c9bd0>] ? hrtimer_get_res+0x50/0x50
[<ffffffffa6912498>] ? get_futex_key+0x1c8/0x2c0
[<ffffffffa69f66d2>] handle_mm_fault+0xb22/0xfb0
[<ffffffffa6f94653>] __do_page_fault+0x213/0x500
[<ffffffffa6f94975>] do_page_fault+0x35/0x90
[<ffffffffa6f90778>] page_fault+0x28/0x30
Code: 48 63 35 be 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 55 a7 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #499198
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8f11722a
Build id: 3.10.0-1160.62.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8f07e258
Build id: 3.10.0-1160.62.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8f07e3f4
Build id: 3.10.0-1160.62.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff8f2080f8
Build id: 3.10.0-1160.62.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff8f1efbf5
Build id: 3.10.0-1160.62.1.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffff8f1f201b
Build id: 3.10.0-1160.62.1.el7.x86_64
0xfb
7
? hrtimer_get_res
vmlinux 0xffffffff8f0c9bf0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x50
8
? get_futex_key
vmlinux 0xffffffff8f112538
Build id: 3.10.0-1160.62.1.el7.x86_64
0x1c8
9
handle_mm_fault
vmlinux 0xffffffff8f1f6692
Build id: 3.10.0-1160.62.1.el7.x86_64
0xb22
10
__do_page_fault
vmlinux 0xffffffff8f794653
Build id: 3.10.0-1160.62.1.el7.x86_64
0x213
11
do_page_fault
vmlinux 0xffffffff8f794975
Build id: 3.10.0-1160.62.1.el7.x86_64
0x35
12
page_fault
vmlinux 0xffffffff8f790778
Build id: 3.10.0-1160.62.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 24s! [15_scheduler:43550]
Modules linked in: xt_REDIRECT nf_nat_redirect ip_vs_rr xt_ipvs ip_vs vxlan ip6_udp_tunnel udp_tunnel xt_mark veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter tun vmw_vsock_vmci_transport vsock sunrpc sb_edac ppdev iosf_mbi crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev pcspkr vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm ahci drm ata_piix libahci crct10dif_pclmul crct10dif_common nfit crc32c_intel libata libnvdimm serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 9 PID: 43550 Comm: 15_scheduler Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8a257fba4200 ti: ffff8a23e4d84000 task.ti: ffff8a23e4d84000
RIP: 0010:[<ffffffff8f11722a>]  [<ffffffff8f11722a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8a23e4d87be8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8a259d620740
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8a23e4d87c20 R08: ffff8a1eec55d800 R09: ffffffff8f3869d9
R10: ffff8a259d85f160 R11: fffff5e795b20a00 R12: 0000000000000292
R13: ffff8a23e4d87b98 R14: ffff8a25bffda000 R15: ffffffff8f1c8f14
FS:  00007fa24f60eb38(0000) GS:ffff8a259d840000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa13d297a80 CR3: 0000000696f44000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8f07e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8f07e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffff8f2080f8>] ptep_clear_flush+0x68/0xa0
[<ffffffff8f1efbf5>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffff8f1f201b>] do_wp_page+0xfb/0x720
[<ffffffff8f0c9bf0>] ? hrtimer_get_res+0x50/0x50
[<ffffffff8f112538>] ? get_futex_key+0x1c8/0x2c0
[<ffffffff8f1f6692>] handle_mm_fault+0xb22/0xfb0
[<ffffffff8f794653>] __do_page_fault+0x213/0x500
[<ffffffff8f794975>] do_page_fault+0x35/0x90
[<ffffffff8f790778>] page_fault+0x28/0x30
Code: 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 d5 8f f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #268176
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9bb1761a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9ba7e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff9ba7ea94
Build id: 3.10.0-1127.19.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff9bc07888
Build id: 3.10.0-1127.19.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffff9bbef3c5
Build id: 3.10.0-1127.19.1.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffff9bbf17eb
Build id: 3.10.0-1127.19.1.el7.x86_64
0xfb
7
? try_to_wake_up
vmlinux 0xffffffff9badb6d0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x190
8
? get_futex_key
vmlinux 0xffffffff9bb127e8
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1c8
9
handle_mm_fault
vmlinux 0xffffffff9bbf5e82
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb22
10
__do_page_fault
vmlinux 0xffffffff9c18d653
Build id: 3.10.0-1127.19.1.el7.x86_64
0x213
11
do_page_fault
vmlinux 0xffffffff9c18d975
Build id: 3.10.0-1127.19.1.el7.x86_64
0x35
12
? error_swapgs
vmlinux 0xffffffff9c189ac9
Build id: 3.10.0-1127.19.1.el7.x86_64
0xaa
13
page_fault
vmlinux 0xffffffff9c189778
Build id: 3.10.0-1127.19.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [VM Thread:12959]
Modules linked in: dcdbas xt_nat veth nf_conntrack_netlink nfnetlink br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay vmw_vsock_vmci_transport vsock xt_comment xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ext4 mbcache jbd2 sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul vmw_balloon glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc vmw_vmci i2c_piix4 parport binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul ata_piix crct10dif_common
drm crc32c_intel libata mptspi nfit scsi_transport_spi serio_raw mptscsih e1000 libnvdimm mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 11 PID: 12959 Comm: VM Thread Kdump: loaded Tainted: G             L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8c541932d230 ti: ffff8c555d97c000 task.ti: ffff8c555d97c000
RIP: 0010:[<ffffffff9bb1761a>]  [<ffffffff9bb1761a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8c555d97fbe8  EFLAGS: 00000202
RAX: 0000000000000012 RBX: 000000000001b840 RCX: ffff8c63ff4a0790
RDX: 0000000000000012 RSI: 0000000000000014 RDI: 0000000000000000
RBP: ffff8c555d97fc20 R08: ffff8c55697b8400 R09: ffffffff9bd851b9
R10: ffff8c63ff2df160 R11: fffff114a3946200 R12: ffff8c555d97fb60
R13: 0000000000000286 R14: 0000000000000010 R15: ffffffff9ba5eb73
FS:  00007fc25861a700(0000) GS:ffff8c63ff2c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc2442180f0 CR3: 00000007357e4000 CR4: 00000000000407e0
Call Trace:
[<ffffffff9ba7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9ba7ea94>] flush_tlb_page+0x54/0xa0
[<ffffffff9bc07888>] ptep_clear_flush+0x68/0xa0
[<ffffffff9bbef3c5>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffff9bbf17eb>] do_wp_page+0xfb/0x720
[<ffffffff9badb6d0>] ? try_to_wake_up+0x190/0x390
[<ffffffff9bb127e8>] ? get_futex_key+0x1c8/0x2c0
[<ffffffff9bbf5e82>] handle_mm_fault+0xb22/0xfb0
[<ffffffff9c18d653>] __do_page_fault+0x213/0x500
[<ffffffff9c18d975>] do_page_fault+0x35/0x90
[<ffffffff9c189ac9>] ? error_swapgs+0xaa/0xc0
[<ffffffff9c189778>] page_fault+0x28/0x30
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 75 9c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #143641
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff93907e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff93877018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff938771b4
Build id: 3.10.0-862.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff939d8837
Build id: 3.10.0-862.el7.x86_64
0x67
5
wp_page_copy.isra.70
vmlinux 0xffffffff939c18be
Build id: 3.10.0-862.el7.x86_64
0x29e
6
do_wp_page
vmlinux 0xffffffff939c3abb
Build id: 3.10.0-862.el7.x86_64
0xfb
7
? unix_stream_recvmsg
vmlinux 0xffffffff93ea2f14
Build id: 3.10.0-862.el7.x86_64
0x54
8
? sock_aio_read.part.10
vmlinux 0xffffffff93dcb9bc
Build id: 3.10.0-862.el7.x86_64
0x14c
9
handle_pte_fault
vmlinux 0xffffffff939c536d
Build id: 3.10.0-862.el7.x86_64
0x36d
10
handle_mm_fault
vmlinux 0xffffffff939c747d
Build id: 3.10.0-862.el7.x86_64
0x39d
11
__do_page_fault
vmlinux 0xffffffff93f1a597
Build id: 3.10.0-862.el7.x86_64
0x197
12
do_page_fault
vmlinux 0xffffffff93f1a925
Build id: 3.10.0-862.el7.x86_64
0x35
13
page_fault
vmlinux 0xffffffff93f16768
Build id: 3.10.0-862.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [docker-containe:29002]
Modules linked in: xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_vs nf_conntrack devlink overlay(T) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq coretemp snd_seq_device iosf_mbi crc32_pclmul snd_pcm ghash_clmulni_intel snd_timer ppdev aesni_intel snd lrw gf128mul glue_helper soundcore vmw_vmci ablk_helper sg vmw_balloon joydev cryptd pcspkr i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif
crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mptspi ttm scsi_transport_spi ata_piix mptscsih crct10dif_pclmul drm crct10dif_common serio_raw e1000 mptbase libata crc32c_intel i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 29002 Comm: docker-containe Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
task: ffff8f1f21b35ee0 ti: ffff8f1f30864000 task.ti: ffff8f1f30864000
RIP: 0010:[<ffffffff93907e3a>]  [<ffffffff93907e3a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8f1f30867b78  EFLAGS: 00000202
RAX: 0000000000000005 RBX: 00000000000000fc RCX: ffff8f1f3975cfa0
RDX: 0000000000000005 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8f1f30867bb0 R08: ffff8f1f3762cc00 R09: ffffffff93b4a3d9
R10: ffff8f1f397dbb00 R11: ffffe6f502653600 R12: 0000000000000296
R13: ffff8f1f30867b28 R14: 0000000000000000 R15: ffffffff00000002
FS:  00007f3da1696700(0000) GS:ffff8f1f397c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c4204fc000 CR3: 000000007bd74000 CR4: 00000000003607e0
Call Trace:
[<ffffffff93877018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff938771b4>] flush_tlb_page+0x54/0xa0
[<ffffffff939d8837>] ptep_clear_flush+0x67/0xa0
[<ffffffff939c18be>] wp_page_copy.isra.70+0x29e/0x4f0
[<ffffffff939c3abb>] do_wp_page+0xfb/0x710
[<ffffffff93ea2f14>] ? unix_stream_recvmsg+0x54/0x70
[<ffffffff93dcb9bc>] ? sock_aio_read.part.10+0x14c/0x170
[<ffffffff939c536d>] handle_pte_fault+0x36d/0xc30
[<ffffffff939c747d>] handle_mm_fault+0x39d/0x9b0
[<ffffffff93f1a597>] __do_page_fault+0x197/0x4f0
[<ffffffff93f1a925>] do_page_fault+0x35/0x90
[<ffffffff93f16768>] page_fault+0x28/0x30
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 53 94 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #15114
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa6b1761a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa6a7e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffa6a7ea94
Build id: 3.10.0-1127.19.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffa6c07888
Build id: 3.10.0-1127.19.1.el7.x86_64
0x68
5
wp_page_copy.isra.73
vmlinux 0xffffffffa6bef3c5
Build id: 3.10.0-1127.19.1.el7.x86_64
0x335
6
do_wp_page
vmlinux 0xffffffffa6bf17eb
Build id: 3.10.0-1127.19.1.el7.x86_64
0xfb
7
? hrtimer_get_res
vmlinux 0xffffffffa6aca2d0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x50
8
? get_futex_key
vmlinux 0xffffffffa6b127e8
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1c8
9
handle_mm_fault
vmlinux 0xffffffffa6bf5e82
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb22
10
__do_page_fault
vmlinux 0xffffffffa718d653
Build id: 3.10.0-1127.19.1.el7.x86_64
0x213
11
do_page_fault
vmlinux 0xffffffffa718d975
Build id: 3.10.0-1127.19.1.el7.x86_64
0x35
12
page_fault
vmlinux 0xffffffffa7189778
Build id: 3.10.0-1127.19.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [7_scheduler:2116]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi iosf_mbi snd_ac97_codec crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq snd_seq_device snd_pcm aesni_intel lrw gf128mul glue_helper snd_timer snd sg
ppdev ablk_helper cryptd joydev pcspkr soundcore i2c_piix4 vmw_vmci vmw_balloon parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata mptspi scsi_transport_spi nfit mptscsih libnvdimm e1000 serio_raw mptbase drm_panel_orientation_quirks
CPU: 5 PID: 2116 Comm: 7_scheduler Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff88143d5de2a0 ti: ffff88143c560000 task.ti: ffff88143c560000
RIP: 0010:[<ffffffffa6b1761a>]  [<ffffffffa6b1761a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff88143c563be8  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 00000000000000fc RCX: ffff8814403a06a0
RDX: 0000000000000006 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff88143c563c20 R08: ffff88133b9cc000 R09: ffffffffa6d851b9
R10: ffff88144035f160 R11: fffff3d4c9fc6000 R12: 0000000000000292
R13: 0000000000000000 R14: ffff88143c563b58 R15: 0000000000000292
FS:  00007fc3f9139700(0000) GS:ffff881440340000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc2f7812000 CR3: 000000027c4b8000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa6a7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa6a7ea94>] flush_tlb_page+0x54/0xa0
[<ffffffffa6c07888>] ptep_clear_flush+0x68/0xa0
[<ffffffffa6bef3c5>] wp_page_copy.isra.73+0x335/0x5b0
[<ffffffffa6bf17eb>] do_wp_page+0xfb/0x720
[<ffffffffa6aca2d0>] ? hrtimer_get_res+0x50/0x50
[<ffffffffa6b127e8>] ? get_futex_key+0x1c8/0x2c0
[<ffffffffa6bf5e82>] handle_mm_fault+0xb22/0xfb0
[<ffffffffa718d653>] __do_page_fault+0x213/0x500
[<ffffffffa718d975>] do_page_fault+0x35/0x90
[<ffffffffa7189778>] page_fault+0x28/0x30
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 75 a7 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89