Info

Permalink
Function
memcpy
First occurrence
2020-09-29
Last occurrence
2026-08-24
State
NEW
Components
kernel
Quality
-20

Statistics

Operating system Count
CentOS 7 75
Architecture Count
x86_64 75
Related packages Count
kernel 75
 0:3.10.0-1160.el7 10
 0:3.10.0-957.el7 8
 0:3.10.0-1160.59.1.el7 7
 0:3.10.0-1160.71.1.el7 7
 0:3.10.0-1160.25.1.el7 7
 0:3.10.0-1160.92.1.el7 4
 0:3.10.0-1127.el7 3
 0:3.10.0-1160.49.1.el7 3
 0:3.10.0-1160.76.1.el7 3
 0:3.10.0-957.5.1.el7 3
 0:3.10.0-693.el7 3
 0:3.10.0-1160.81.1.el7 2
 0:3.10.0-1062.9.1.el7 2
 0:3.10.0-957.10.1.el7 2
 0:3.10.0-862.14.4.el7 2
 0:3.10.0-1062.1.2.el7 1
 0:3.10.0-1127.10.1.el7 1
 0:3.10.0-1127.19.1.el7 1
 0:3.10.0-1160.119.1.el7 1
 0:3.10.0-1160.15.2.el7 1
 0:3.10.0-1160.66.1.el7 1
 0:3.10.0-1160.21.1.el7 1
 0:3.10.0-1160.36.2.el7 1
 0:3.10.0-862.3.2.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1305416
Frame # Function Binary Source or offset Line
1
vmw_fb_dirty_flush
vmwgfx 0xffffffffc062fdbc
Build id: 3.10.0-862.3.2.el7.x86_64
0x1fc
2
process_one_work
vmlinux 0xffffffff858b312f
Build id: 3.10.0-862.3.2.el7.x86_64
0x17f
3
worker_thread
vmlinux 0xffffffff858b3df6
Build id: 3.10.0-862.3.2.el7.x86_64
0x126
4
? manage_workers.isra.24
vmlinux 0xffffffff858b3cd0
Build id: 3.10.0-862.3.2.el7.x86_64
0x2a0
5
kthread
vmlinux 0xffffffff858bb161
Build id: 3.10.0-862.3.2.el7.x86_64
0xd1
6
? insert_kthread_work
vmlinux 0xffffffff858bb090
Build id: 3.10.0-862.3.2.el7.x86_64
0x40
7
ret_from_fork_nospec_begin
vmlinux 0xffffffff85f20677
Build id: 3.10.0-862.3.2.el7.x86_64
0x21
8
? insert_kthread_work
vmlinux 0xffffffff858bb090
Build id: 3.10.0-862.3.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:5:6885]
Modules linked in: binfmt_misc veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop arc4 md4 nls_utf8 cifs ccm dns_resolver vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper nfit ablk_helper cryptd vmw_balloon libnvdimm joydev sg pcspkr parport_pc parport shpchp i2c_piix4 vmw_vmci fuse ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul ata_piix crct10dif_common
mptspi crc32c_intel libata scsi_transport_spi i2c_core mptscsih e1000 serio_raw mptbase dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 6885 Comm: kworker/1:5 Kdump: loaded Tainted: G        W    L ------------   3.10.0-862.3.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: events vmw_fb_dirty_flush [vmwgfx]
task: ffffa10cb80fdee0 ti: ffffa10f2c070000 task.ti: ffffa10f2c070000
RIP: 0010:[<ffffffffc062fdbc>]  [<ffffffffc062fdbc>] vmw_fb_dirty_flush+0x1fc/0x2c0 [vmwgfx]
RSP: 0000:ffffa10f2c073db0  EFLAGS: 00010206
RAX: ffffa1102a053e00 RBX: 0000000000000000 RCX: 0000000000000004
RDX: 00000000ffffffff RSI: ffffc0c7420e0b20 RDI: ffffc0c742404420
RBP: ffffa10f2c073e18 R08: 0000000000000000 R09: ffffc0c742404400
R10: 0000000000000000 R11: 0000000000000000 R12: ffffa10cb80fe578
R13: ffffa10f2c073e18 R14: ffffa1103fc40000 R15: ffffffff8582959e
FS:  0000000000000000(0000) GS:ffffa1103fc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000c00035f010 CR3: 00000003f0f14000 CR4: 00000000000607e0
Call Trace:
[<ffffffff858b312f>] process_one_work+0x17f/0x440
[<ffffffff858b3df6>] worker_thread+0x126/0x3c0
[<ffffffff858b3cd0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff858bb161>] kthread+0xd1/0xe0
[<ffffffff858bb090>] ? insert_kthread_work+0x40/0x40
[<ffffffff85f20677>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff858bb090>] ? insert_kthread_work+0x40/0x40
Code: 89 f7 4d 89 e6 49 89 dc 48 89 f3 0f 1f 40 00 48 89 da 4c 89 fe 4c 89 cf e8 f2 a6 52 c5 49 89 c1 49 8b 86 70 ff ff ff 41 83 ed 01 <8b> 50 48 49 01 d1 41 8b 94 24 30 01 00 00 49 01 d7 41 83 fd ff 
Complete report #963116
Frame # Function Binary Source or offset Line
1
? vmw_fb_dirty_flush
vmwgfx 0xffffffffc0625b3e
Build id: 3.10.0-1062.9.1.el7.x86_64
0x2fe
2
process_one_work
vmlinux 0xffffffff924be21f
Build id: 3.10.0-1062.9.1.el7.x86_64
0x17f
3
worker_thread
vmlinux 0xffffffff924bf336
Build id: 3.10.0-1062.9.1.el7.x86_64
0x126
4
? manage_workers.isra.26
vmlinux 0xffffffff924bf210
Build id: 3.10.0-1062.9.1.el7.x86_64
0x2a0
5
kthread
vmlinux 0xffffffff924c61f1
Build id: 3.10.0-1062.9.1.el7.x86_64
0xd1
6
? insert_kthread_work
vmlinux 0xffffffff924c6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
7
ret_from_fork_nospec_begin
vmlinux 0xffffffff92b8dd37
Build id: 3.10.0-1062.9.1.el7.x86_64
0x21
8
? insert_kthread_work
vmlinux 0xffffffff924c6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [kworker/u128:1:27814]
[<ffffffffc0625b3e>] ? vmw_fb_dirty_flush+0x2fe/0x3a0 [vmwgfx]
[<ffffffff924be21f>] process_one_work+0x17f/0x440
[<ffffffff924bf336>] worker_thread+0x126/0x3c0
[<ffffffff924bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff924c61f1>] kthread+0xd1/0xe0
[<ffffffff924c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffff92b8dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff924c6120>] ? insert_kthread_work+0x40/0x40
Complete report #467092
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffffa5d5ece4
Build id: 3.10.0-862.14.4.el7.x86_64
0x34
2
? vmw_fb_dirty_flush
vmwgfx 0xffffffffc05b4e1e
Build id: 3.10.0-862.14.4.el7.x86_64
0x1ee
3
process_one_work
vmlinux 0xffffffffa5ab613f
Build id: 3.10.0-862.14.4.el7.x86_64
0x17f
4
worker_thread
vmlinux 0xffffffffa5ab71d6
Build id: 3.10.0-862.14.4.el7.x86_64
0x126
5
? manage_workers.isra.24
vmlinux 0xffffffffa5ab70b0
Build id: 3.10.0-862.14.4.el7.x86_64
0x2a0
6
kthread
vmlinux 0xffffffffa5abdf21
Build id: 3.10.0-862.14.4.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffffa5abde50
Build id: 3.10.0-862.14.4.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffffa61255f7
Build id: 3.10.0-862.14.4.el7.x86_64
0x21
9
? insert_kthread_work
vmlinux 0xffffffffa5abde50
Build id: 3.10.0-862.14.4.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [kworker/2:0:20]
Modules linked in: vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr joydev sg nfit libnvdimm parport_pc parport shpchp i2c_piix4 vmw_vmci binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul mptspi crct10dif_common crc32c_intel scsi_transport_spi serio_raw mptscsih mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 20 Comm: kworker/2:0 Kdump: loaded Tainted: G             L ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: events vmw_fb_dirty_flush [vmwgfx]
task: ffffa03074499fa0 ti: ffffa030744b0000 task.ti: ffffa030744b0000
RIP: 0010:[<ffffffffa5d5ece4>]  [<ffffffffa5d5ece4>] memcpy+0x34/0x110
RSP: 0018:ffffa030744b3da8  EFLAGS: 00000202
RAX: ffffbc8cc4192580 RBX: ffffa0373fc95c60 RCX: 0000000000000004
RDX: 0000000000000380 RSI: ffffbc8cc3a12000 RDI: ffffbc8cc4192e40
RBP: ffffa030744b3e18 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa5bb66a6
R13: ffffa030744b3d18 R14: ffffa0373fc98b40 R15: ffffffffa5adb7ec
FS:  0000000000000000(0000) GS:ffffa0373fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1a6fa92000 CR3: 000000080c3f2000 CR4: 00000000000407e0
Call Trace:
[<ffffffffc05b4e1e>] ? vmw_fb_dirty_flush+0x1ee/0x2c0 [vmwgfx]
[<ffffffffa5ab613f>] process_one_work+0x17f/0x440
[<ffffffffa5ab71d6>] worker_thread+0x126/0x3c0
[<ffffffffa5ab70b0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffa5abdf21>] kthread+0xd1/0xe0
[<ffffffffa5abde50>] ? insert_kthread_work+0x40/0x40
[<ffffffffa61255f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa5abde50>] ? insert_kthread_work+0x40/0x40
Code: 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 4c 89 07 4c 89 4f 08 4c 89 57 10 <4c> 89 5f 18 48 8d 7f 20 73 d4 83 c2 20 eb 44 48 01 d6 48 01 d7 
Complete report #404666
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffffaf386348
Build id: 3.10.0-957.el7.x86_64
0x68
2
? vmw_fb_dirty_flush
vmwgfx 0xffffffffc05337ce
Build id: 3.10.0-957.el7.x86_64
0x2fe
3
process_one_work
vmlinux 0xffffffffaf0b9d4f
Build id: 3.10.0-957.el7.x86_64
0x17f
4
worker_thread
vmlinux 0xffffffffaf0bade6
Build id: 3.10.0-957.el7.x86_64
0x126
5
? manage_workers.isra.25
vmlinux 0xffffffffaf0bacc0
Build id: 3.10.0-957.el7.x86_64
0x2a0
6
kthread
vmlinux 0xffffffffaf0c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffffaf0c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffffaf774c37
Build id: 3.10.0-957.el7.x86_64
0x21
9
? insert_kthread_work
vmlinux 0xffffffffaf0c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [kworker/2:1:80331]
Modules linked in: binfmt_misc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables tcp_diag inet_diag veth br_netfilter overlay(T) ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd joydev pcspkr soundcore sg parport_pc parport vmw_vmci i2c_piix4 xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ata_generic pata_acpi serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm e1000 ata_piix drm libata
mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 2 PID: 80331 Comm: kworker/2:1 Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
Workqueue: events vmw_fb_dirty_flush [vmwgfx]
task: ffff88eceabc30c0 ti: ffff88ead2a50000 task.ti: ffff88ead2a50000
RIP: 0010:[<ffffffffaf386348>]  [<ffffffffaf386348>] memcpy+0x68/0x110
RSP: 0000:ffff88ead2a53d98  EFLAGS: 00010202
RAX: ffffb6bf4227b900 RBX: 0000000000000000 RCX: 0000000000000004
RDX: 0000000000000c40 RSI: ffffb6bf41f27920 RDI: ffffb6bf4227c580
RBP: ffff88ead2a53e18 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
R13: 0000000000000001 R14: 000000000001ab80 R15: ffff88ead2a53d90
FS:  0000000000000000(0000) GS:ffff88ecf9480000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fce4cf1cd90 CR3: 000000012b2da000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc05337ce>] ? vmw_fb_dirty_flush+0x2fe/0x3a0 [vmwgfx]
[<ffffffffaf0b9d4f>] process_one_work+0x17f/0x440
[<ffffffffaf0bade6>] worker_thread+0x126/0x3c0
[<ffffffffaf0bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffaf0c1c31>] kthread+0xd1/0xe0
[<ffffffffaf0c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffaf774c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffaf0c1b60>] ? insert_kthread_work+0x40/0x40
Code: d4 83 c2 20 eb 44 48 01 d6 48 01 d7 48 83 ea 20 0f 1f 00 48 83 ea 20 4c 8b 46 f8 4c 8b 4e f0 4c 8b 56 e8 4c 8b 5e e0 48 8d 76 e0 <4c> 89 47 f8 4c 89 4f f0 4c 89 57 e8 4c 89 5f e0 48 8d 7f e0 73 
Complete report #339543
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffffa9797a99
Build id: 3.10.0-1160.36.2.el7.x86_64
0x29
2
? vmw_fb_dirty_flush
vmwgfx 0xffffffffc0559b3e
Build id: 3.10.0-1160.36.2.el7.x86_64
0x2fe
3
process_one_work
vmlinux 0xffffffffa94bde6f
Build id: 3.10.0-1160.36.2.el7.x86_64
0x17f
4
worker_thread
vmlinux 0xffffffffa94bef86
Build id: 3.10.0-1160.36.2.el7.x86_64
0x126
5
? manage_workers.isra.26
vmlinux 0xffffffffa94bee60
Build id: 3.10.0-1160.36.2.el7.x86_64
0x2a0
6
kthread
vmlinux 0xffffffffa94c5e41
Build id: 3.10.0-1160.36.2.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffffa94c5d70
Build id: 3.10.0-1160.36.2.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffffa9b95df7
Build id: 3.10.0-1160.36.2.el7.x86_64
0x21
9
? insert_kthread_work
vmlinux 0xffffffffa94c5d70
Build id: 3.10.0-1160.36.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [kworker/5:2:2642]
Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic
pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt crc_t10dif fb_sys_fops crct10dif_generic ttm ata_piix ahci drm libahci e1000e crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 ptp pps_core vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ampnetworkflow]
CPU: 5 PID: 2642 Comm: kworker/5:2 Kdump: loaded Tainted: G        W  OEL ------------   3.10.0-1160.36.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
Workqueue: events vmw_fb_dirty_flush [vmwgfx]
task: ffffa0a0d792d280 ti: ffffa0a06fba4000 task.ti: ffffa0a06fba4000
RIP: 0010:[<ffffffffa9797a99>]  [<ffffffffa9797a99>] memcpy+0x29/0x110
RSP: 0018:ffffa0a06fba7d98  EFLAGS: 00010206
RAX: ffffc50e428d3000 RBX: ffffffffc05743cb RCX: 0000000000000004
RDX: 00000000000003c0 RSI: ffffc50e4217e820 RDI: ffffc50e428d4000
RBP: ffffa0a06fba7e18 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffa0a167dd3300
R13: ffffffffa97a67cd R14: ffffa0a06fba7d08 R15: ffffffffa9c1e720
FS:  0000000000000000(0000) GS:ffffa0a17fd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f757b6b9000 CR3: 000000042c73c000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc0559b3e>] ? vmw_fb_dirty_flush+0x2fe/0x3a0 [vmwgfx]
[<ffffffffa94bde6f>] process_one_work+0x17f/0x440
[<ffffffffa94bef86>] worker_thread+0x126/0x3c0
[<ffffffffa94bee60>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffa94c5e41>] kthread+0xd1/0xe0
[<ffffffffa94c5d70>] ? insert_kthread_work+0x40/0x40
[<ffffffffa9b95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa94c5d70>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 
Complete report #331314
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffffadb96e89
Build id: 3.10.0-1160.el7.x86_64
0x29
2
? vmw_fb_dirty_flush
vmwgfx 0xffffffffc047bb3e
Build id: 3.10.0-1160.el7.x86_64
0x2fe
3
process_one_work
vmlinux 0xffffffffad8bdc4f
Build id: 3.10.0-1160.el7.x86_64
0x17f
4
worker_thread
vmlinux 0xffffffffad8bed66
Build id: 3.10.0-1160.el7.x86_64
0x126
5
? manage_workers.isra.26
vmlinux 0xffffffffad8bec40
Build id: 3.10.0-1160.el7.x86_64
0x2a0
6
kthread
vmlinux 0xffffffffad8c5c21
Build id: 3.10.0-1160.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffffad8c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffffadf93df7
Build id: 3.10.0-1160.el7.x86_64
0x21
9
? insert_kthread_work
vmlinux 0xffffffffad8c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:2:50288]
Modules linked in: loop binfmt_misc rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace sunrpc fscache veth vport_vxlan vxlan ip6_udp_tunnel udp_tunnel xt_nat vsock_diag unix_diag ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) dtap(OE) openvswitch xt_multiport xt_comment udp_diag tcp_diag inet_diag ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport
vsock vfat fat sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg tpm_crb parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel vmxnet3 serio_raw drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm libahci drm ata_piix nfit libata vmw_pvscsi libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: dtap]
CPU: 2 PID: 50288 Comm: kworker/2:2 Tainted: G           OEL ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.18227214.B64.2106252220 06/25/2021
Workqueue: events vmw_fb_dirty_flush [vmwgfx]
task: ffff89f61b5a8000 ti: ffff89f713e48000 task.ti: ffff89f713e48000
RIP: 0010:[<ffffffffadb96e89>]  [<ffffffffadb96e89>] memcpy+0x29/0x110
RSP: 0000:ffff89f713e4bd98  EFLAGS: 00010206
RAX: ffffa2ad838d0000 RBX: ffffffffc04963cb RCX: 0000000000000004
RDX: 0000000000000fc0 RSI: ffffa2ad83c37160 RDI: ffffa2ad838d0000
RBP: ffff89f713e4be18 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff89f713e4bd10
R13: ffff89f61b5a8068 R14: ffff89fbffd1ad40 R15: ffff89f61b5a8068
FS:  0000000000000000(0000) GS:ffff89fbffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fda2ee39008 CR3: 0000000552ee4000 CR4: 00000000000607e0
Call Trace:
[<ffffffffc047bb3e>] ? vmw_fb_dirty_flush+0x2fe/0x3a0 [vmwgfx]
[<ffffffffad8bdc4f>] process_one_work+0x17f/0x440
[<ffffffffad8bed66>] worker_thread+0x126/0x3c0
[<ffffffffad8bec40>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffad8c5c21>] kthread+0xd1/0xe0
[<ffffffffad8c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffadf93df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffad8c5b50>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 
Complete report #288567
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff81331666
Build id: 3.10.0-693.el7.x86_64
0x16
2
? vmw_fb_dirty_flush
vmwgfx 0xffffffffc01ab24e
Build id: 3.10.0-693.el7.x86_64
0x1fe
3
process_one_work
vmlinux 0xffffffff810a881a
Build id: 3.10.0-693.el7.x86_64
0x17a
4
worker_thread
vmlinux 0xffffffff810a94e6
Build id: 3.10.0-693.el7.x86_64
0x126
5
? manage_workers.isra.24
vmlinux 0xffffffff810a93c0
Build id: 3.10.0-693.el7.x86_64
0x2a0
6
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
7
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
8
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
9
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [khugepaged:87]
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [kworker/6:1:5978]
Modules linked in: vxlan ip6_udp_tunnel udp_tunnel iptable_mangle xt_mark xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) vmw_vsock_vmci_transport vsock sb_edac edac_core coretemp iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr nfit vmw_balloon ppdev joydev sg libnvdimm vmw_vmci shpchp i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif vmwgfx crct10dif_generic drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul libata crct10dif_common crc32c_intel serio_raw vmxnet3 i2c_core vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 5978 Comm: kworker/6:1 Tainted: G           OEL ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Workqueue: events vmw_fb_dirty_flush [vmwgfx]
task: ffff88101ddeeeb0 ti: ffff880acdcf0000 task.ti: ffff880acdcf0000
RIP: 0010:[<ffffffff81331666>]  [<ffffffff81331666>] memcpy+0x16/0x110
RSP: 0000:ffff880acdcf3da8  EFLAGS: 00010206
RAX: ffffc90007140300 RBX: ffffffff810cdc7c RCX: 0000000000000004
RDX: 0000000000000760 RSI: ffffc90006df3000 RDI: ffffc900071406c0
RBP: ffff880acdcf3e18 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000001
R13: ffff88103fc16d40 R14: ffff88101ddeef18 R15: ffffffff810ca29e
FS:  0000000000000000(0000) GS:ffff88103fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000c41fdc2e00 CR3: 00000000019f2000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffffc01ab24e 0000025000000000 00000020000002d8 ffff88101ca40000
ffff88101d336100 0000000000000000 ffff88101bea0468 ffff88081d9d76c0
0000000019de6546 ffff88101bea0558 ffff88092dbfae80 ffff88103fc16540
Call Trace:
[<ffffffffc01ab24e>] ? vmw_fb_dirty_flush+0x1fe/0x2c0 [vmwgfx]
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 31 48 c1 e2 20 89 c0 48 09 c2 48 31 d3 e9 78 ff ff ff 90 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 <4c> 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 4c 89 
Complete report #237881
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffffacf97309
Build id: 3.10.0-1160.21.1.el7.x86_64
0x29
2
? vmw_fb_dirty_flush
vmwgfx 0xffffffffc031eb3e
Build id: 3.10.0-1160.21.1.el7.x86_64
0x2fe
3
process_one_work
vmlinux 0xffffffffaccbddcf
Build id: 3.10.0-1160.21.1.el7.x86_64
0x17f
4
worker_thread
vmlinux 0xffffffffaccbeee6
Build id: 3.10.0-1160.21.1.el7.x86_64
0x126
5
? manage_workers.isra.26
vmlinux 0xffffffffaccbedc0
Build id: 3.10.0-1160.21.1.el7.x86_64
0x2a0
6
kthread
vmlinux 0xffffffffaccc5da1
Build id: 3.10.0-1160.21.1.el7.x86_64
0xd1
7
? system_call_after_swapgs
vmlinux 0xffffffffad395ec9
Build id: 3.10.0-1160.21.1.el7.x86_64
0x96
8
? insert_kthread_work
vmlinux 0xffffffffaccc5cd0
Build id: 3.10.0-1160.21.1.el7.x86_64
0x40
9
ret_from_fork_nospec_begin
vmlinux 0xffffffffad395df7
Build id: 3.10.0-1160.21.1.el7.x86_64
0x21
10
? insert_kthread_work
vmlinux 0xffffffffaccc5cd0
Build id: 3.10.0-1160.21.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [kworker/0:1:13419]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables btrfs raid6_pq xor vfat msdos fat ext4 mbcache jbd2 cdp(OE) binfmt_misc tun devlink bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc coretemp crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev parport_pc sg parport vmw_vmci i2c_piix4 xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw ata_piix mptspi scsi_transport_spi libata mptscsih mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ip_tables]
CPU: 0 PID: 13419 Comm: kworker/0:1 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
Workqueue: events vmw_fb_dirty_flush [vmwgfx]
task: ffff8cef3c74a100 ti: ffff8cefbec74000 task.ti: ffff8cefbec74000
RIP: 0010:[<ffffffffacf97309>]  [<ffffffffacf97309>] memcpy+0x29/0x110
RSP: 0018:ffff8cefbec77d98  EFLAGS: 00010206
RAX: ffffb0e701760580 RBX: ffffffffc03393cb RCX: 0000000000000004
RDX: 00000000000001c0 RSI: ffffb0e7012c89c0 RDI: ffffb0e701761000
RBP: ffff8cefbec77e18 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffad393940
R13: ffff8cefbec77d38 R14: 0000000000000282 R15: 0000000000000282
FS:  0000000000000000(0000) GS:ffff8cefffc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000017a0878 CR3: 00000001c42ac000 CR4: 00000000000007f0
Call Trace:
 [<ffffffffc031eb3e>] ? vmw_fb_dirty_flush+0x2fe/0x3a0 [vmwgfx]
 [<ffffffffaccbddcf>] process_one_work+0x17f/0x440
 [<ffffffffaccbeee6>] worker_thread+0x126/0x3c0
 [<ffffffffaccbedc0>] ? manage_workers.isra.26+0x2a0/0x2a0
 [<ffffffffaccc5da1>] kthread+0xd1/0xe0
 [<ffffffffad395ec9>] ? system_call_after_swapgs+0x96/0x13a
 [<ffffffffaccc5cd0>] ? insert_kthread_work+0x40/0x40
 [<ffffffffad395df7>] ret_from_fork_nospec_begin+0x21/0x21
 [<ffffffffaccc5cd0>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 
Complete report #176888
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff81331666
Build id: 3.10.0-693.el7.x86_64
0x16
2
? vmw_fb_dirty_flush
vmwgfx 0xffffffffc018524e
Build id: 3.10.0-693.el7.x86_64
0x1fe
3
process_one_work
vmlinux 0xffffffff810a881a
Build id: 3.10.0-693.el7.x86_64
0x17a
4
worker_thread
vmlinux 0xffffffff810a94e6
Build id: 3.10.0-693.el7.x86_64
0x126
5
? manage_workers.isra.24
vmlinux 0xffffffff810a93c0
Build id: 3.10.0-693.el7.x86_64
0x2a0
6
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
7
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
8
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
9
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [kworker/2:1:110]
Modules linked in: nls_utf8 isofs loop ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod sb_edac edac_core iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg nfit libnvdimm vmw_vmci i2c_piix4 shpchp parport_pc parport xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci crct10dif_pclmul crct10dif_common crc32c_intel ata_piix vmxnet3 serio_raw libata vmw_pvscsi i2c_core [last unloaded: ip_tables]
CPU: 2 PID: 110 Comm: kworker/2:1 Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/09/2019
Workqueue: events vmw_fb_dirty_flush [vmwgfx]
task: ffff88042c0d8000 ti: ffff880036910000 task.ti: ffff880036910000
RIP: 0010:[<ffffffff81331666>]  [<ffffffff81331666>] memcpy+0x16/0x110
RSP: 0000:ffff880036913da8  EFLAGS: 00010202
RAX: ffffc90002c69400 RBX: ffff880036913fd8 RCX: 0000000000000004
RDX: 0000000000000a40 RSI: ffffc9000216d000 RDI: ffffc90002c69400
RBP: ffff880036913e18 R08: 0000000000000000 R09: ffffc90002c69400
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88042c0d8000
R13: ffffffff816a8f8d R14: ffff880036913d60 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff88043fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fc23c31ec60 CR3: 000000024f795000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffffc018524e 00000300f4c48d58 00000020000002a0 ffff88042a200000
ffff88043fee6d00 0000000000000000 ffff88042a248c68 6ac0000000000000
000000007a0472e0 ffff88042a248d58 ffff88042c002080 ffff88043fd16540
Call Trace:
[<ffffffffc018524e>] ? vmw_fb_dirty_flush+0x1fe/0x2c0 [vmwgfx]
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 31 48 c1 e2 20 89 c0 48 09 c2 48 31 d3 e9 78 ff ff ff 90 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 <4c> 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 4c 89 
Complete report #23436
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffff823868e9
Build id: 3.10.0-957.5.1.el7.x86_64
0x29
2
? vmw_fb_dirty_flush
vmwgfx 0xffffffffc02677ce
Build id: 3.10.0-957.5.1.el7.x86_64
0x2fe
3
process_one_work
vmlinux 0xffffffff820b9d8f
Build id: 3.10.0-957.5.1.el7.x86_64
0x17f
4
worker_thread
vmlinux 0xffffffff820bae26
Build id: 3.10.0-957.5.1.el7.x86_64
0x126
5
? manage_workers.isra.25
vmlinux 0xffffffff820bad00
Build id: 3.10.0-957.5.1.el7.x86_64
0x2a0
6
kthread
vmlinux 0xffffffff820c1c71
Build id: 3.10.0-957.5.1.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffff820c1ba0
Build id: 3.10.0-957.5.1.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffff82774c37
Build id: 3.10.0-957.5.1.el7.x86_64
0x21
9
? insert_kthread_work
vmlinux 0xffffffff820c1ba0
Build id: 3.10.0-957.5.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [kworker/0:0:19236]
Modules linked in: xt_set xt_multiport ip_set_hash_ip xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus
snd_seq snd_seq_device snd_pcm snd_timer snd soundcore sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi nfit scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 19236 Comm: kworker/0:0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
Workqueue: events vmw_fb_dirty_flush [vmwgfx]
task: ffff989875c930c0 ti: ffff9898f47e0000 task.ti: ffff9898f47e0000
RIP: 0010:[<ffffffff823868e9>]  [<ffffffff823868e9>] memcpy+0x29/0x110
RSP: 0018:ffff9898f47e3d98  EFLAGS: 00010206
RAX: ffffb3ff810c9a80 RBX: ffffffff820d2ef2 RCX: 0000000000000004
RDX: 00000000000006c0 RSI: ffffb3ff80d8ae80 RDI: ffffb3ff810ca000
RBP: ffff9898f47e3e18 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff9898f47e3d50
R13: 0000000000000001 R14: ffff98997741ac00 R15: ffff989875c93128
FS:  0000000000000000(0000) GS:ffff989977400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0b49ced2d0 CR3: 0000000134d24000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc02677ce>] ? vmw_fb_dirty_flush+0x2fe/0x3a0 [vmwgfx]
[<ffffffff820b9d8f>] process_one_work+0x17f/0x440
[<ffffffff820bae26>] worker_thread+0x126/0x3c0
[<ffffffff820bad00>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff820c1c71>] kthread+0xd1/0xe0
[<ffffffff820c1ba0>] ? insert_kthread_work+0x40/0x40
[<ffffffff82774c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff820c1ba0>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 
Complete report #21931
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffffb9d926e8
Build id: 3.10.0-1062.9.1.el7.x86_64
0x68
2
? vmw_fb_dirty_flush
vmwgfx 0xffffffffc02beb3e
Build id: 3.10.0-1062.9.1.el7.x86_64
0x2fe
3
process_one_work
vmlinux 0xffffffffb9abe21f
Build id: 3.10.0-1062.9.1.el7.x86_64
0x17f
4
worker_thread
vmlinux 0xffffffffb9abf336
Build id: 3.10.0-1062.9.1.el7.x86_64
0x126
5
? manage_workers.isra.26
vmlinux 0xffffffffb9abf210
Build id: 3.10.0-1062.9.1.el7.x86_64
0x2a0
6
kthread
vmlinux 0xffffffffb9ac61f1
Build id: 3.10.0-1062.9.1.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffffb9ac6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffffba18dd37
Build id: 3.10.0-1062.9.1.el7.x86_64
0x21
9
? insert_kthread_work
vmlinux 0xffffffffb9ac6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kworker/4:3:29716]
Modules linked in: vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper vmw_balloon ablk_helper e1000e cryptd pcspkr ptp sg joydev pps_core parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif ahci drm crct10dif_generic libahci ata_piix nfit libata libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 29716 Comm: kworker/4:3 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018
Workqueue: events vmw_fb_dirty_flush [vmwgfx]
task: ffff8cebfd2c41c0 ti: ffff8ceb83d04000 task.ti: ffff8ceb83d04000
RIP: 0010:[<ffffffffb9d926e8>]  [<ffffffffb9d926e8>] memcpy+0x68/0x110
RSP: 0018:ffff8ceb83d07d98  EFLAGS: 00010202
RAX: ffffb2b7415bf600 RBX: ffffffffc02d93cb RCX: 0000000000000004
RDX: 0000000000000c40 RSI: ffffb2b7412400e0 RDI: ffffb2b7415c0280
RBP: ffff8ceb83d07e18 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8ceb83d07d50
R13: 0000000000000001 R14: ffff8cec7fd1ad00 R15: ffff8cebfd2c4228
FS:  0000000000000000(0000) GS:ffff8cec7fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff3fa4fb70e CR3: 0000000034590000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc02beb3e>] ? vmw_fb_dirty_flush+0x2fe/0x3a0 [vmwgfx]
[<ffffffffb9abe21f>] process_one_work+0x17f/0x440
[<ffffffffb9abf336>] worker_thread+0x126/0x3c0
[<ffffffffb9abf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb9ac61f1>] kthread+0xd1/0xe0
[<ffffffffb9ac6120>] ? insert_kthread_work+0x40/0x40
[<ffffffffba18dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb9ac6120>] ? insert_kthread_work+0x40/0x40
Code: d4 83 c2 20 eb 44 48 01 d6 48 01 d7 48 83 ea 20 0f 1f 00 48 83 ea 20 4c 8b 46 f8 4c 8b 4e f0 4c 8b 56 e8 4c 8b 5e e0 48 8d 76 e0 <4c> 89 47 f8 4c 89 4f f0 4c 89 57 e8 4c 89 5f e0 48 8d 7f e0 73 
Complete report #3862
Frame # Function Binary Source or offset Line
1
memcpy
vmlinux 0xffffffffa4995fa9
Build id: 3.10.0-1127.el7.x86_64
0x29
2
? vmw_fb_dirty_flush
vmwgfx 0xffffffffc0470b3e
Build id: 3.10.0-1127.el7.x86_64
0x2fe
3
process_one_work
vmlinux 0xffffffffa46be6bf
Build id: 3.10.0-1127.el7.x86_64
0x17f
4
worker_thread
vmlinux 0xffffffffa46bf7d6
Build id: 3.10.0-1127.el7.x86_64
0x126
5
? manage_workers.isra.26
vmlinux 0xffffffffa46bf6b0
Build id: 3.10.0-1127.el7.x86_64
0x2a0
6
kthread
vmlinux 0xffffffffa46c6691
Build id: 3.10.0-1127.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffffa46c65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffffa4d92d37
Build id: 3.10.0-1127.el7.x86_64
0x21
9
? insert_kthread_work
vmlinux 0xffffffffa46c65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [kworker/1:1:2172]
Modules linked in: binfmt_misc vxlan ip6_udp_tunnel udp_tunnel xt_set ip_set_hash_ipportip ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipport ip_set dummy ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables iptable_mangle xt_comment xt_mark veth xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_nat_ipv4 nf_nat br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock iosf_mbi kvm_intel kvm btusb btrtl btbcm irqbypass ppdev btintel crc32_pclmul ghash_clmulni_intel aesni_intel bluetooth lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon rfkill pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack
ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ata_piix vmwgfx serio_raw e1000 libata drm_kms_helper mptspi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit scsi_transport_spi drm libnvdimm mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 2172 Comm: kworker/1:1 Tainted: G             L ------------ T 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
Workqueue: events vmw_fb_dirty_flush [vmwgfx]
task: ffff9fe19cf920e0 ti: ffff9fe128ba8000 task.ti: ffff9fe128ba8000
RIP: 0010:[<ffffffffa4995fa9>]  [<ffffffffa4995fa9>] memcpy+0x29/0x110
RSP: 0018:ffff9fe128babd98  EFLAGS: 00010202
RAX: ffffab378102ef00 RBX: ffffffffc048b3cb RCX: 0000000000000004
RDX: 0000000000000b40 RSI: ffffab3780c98260 RDI: ffffab378102f000
RBP: ffff9fe128babe18 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffa46af5ec
R13: ffff9fe128babd48 R14: 0000000000000082 R15: ffff9fe13672d380
FS:  0000000000000000(0000) GS:ffff9fe239640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055aab6011570 CR3: 0000000004486000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc0470b3e>] ? vmw_fb_dirty_flush+0x2fe/0x3a0 [vmwgfx]
[<ffffffffa46be6bf>] process_one_work+0x17f/0x440
[<ffffffffa46bf7d6>] worker_thread+0x126/0x3c0
[<ffffffffa46bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffa46c6691>] kthread+0xd1/0xe0
[<ffffffffa46c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa4d92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa46c65c0>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4