Info

Permalink
Function
manage_workers
First occurrence
2020-09-23
Last occurrence
2026-03-16
State
NEW
Components
kernel
Quality
-12

Statistics

Operating system Count
CentOS 7 165
Architecture Count
x86_64 165
Related packages Count
kernel 159
 0:3.10.0-1160.80.1.el7 31
 0:3.10.0-957.5.1.el7 15
 0:3.10.0-862.el7 9
 0:3.10.0-1160.59.1.el7 8
 0:3.10.0-1160.el7 8
 0:3.10.0-1160.49.1.el7 7
 0:3.10.0-1160.45.1.el7 7
 0:3.10.0-1160.119.1.el7 6
 0:3.10.0-862.14.4.el7 6
 0:3.10.0-1062.12.1.el7 4
 0:3.10.0-1160.36.2.el7 4
 0:3.10.0-1160.88.1.el7 4
 0:3.10.0-862.6.3.el7 4
 0:3.10.0-957.el7 4
 0:3.10.0-1160.90.1.el7 4
 0:3.10.0-1127.18.2.el7 3
 0:3.10.0-1127.19.1.el7 3
 0:3.10.0-1127.el7 3
 0:3.10.0-862.2.3.el7 3
 0:3.10.0-1062.el7 2
 0:3.10.0-1160.21.1.el7 2
 0:3.10.0-1160.76.1.el7 2
 0:3.10.0-862.11.6.el7 2
 0:3.10.0-1160.118.1.el7 2
 0:3.10.0-1127.13.1.el7 1
 0:3.10.0-1160.11.1.el7 1
 0:3.10.0-1160.2.2.el7 1
 0:3.10.0-1160.42.2.el7 1
 0:3.10.0-1160.66.1.el7 1
 0:3.10.0-1127.8.2.el7 1
 0:3.10.0-1160.95.1.el7 1
 0:3.10.0-1062.9.1.el7 1
 0:3.10.0-1160.24.1.el7 1
 0:3.10.0-1160.71.1.el7 1
 0:3.10.0-957.27.2.el7 1
 0:3.10.0-862.3.3.el7 1
 0:3.10.0-862.3.2.el7 1
 0:3.10.0-862.9.1.el7 1
 0:3.10.0-1160.62.1.el7 1
 0:3.10.0-957.12.2.el7 1
kernel-plus 6
 0:3.10.0-957.el7.centos.plus 6

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #851126
Frame # Function Binary Source or offset Line
1
process_one_work
vmlinux 0xffffffff88ab9e2d
Build id: 3.10.0-957.12.2.el7.x86_64
0x16d
2
worker_thread
vmlinux 0xffffffff88abaed6
Build id: 3.10.0-957.12.2.el7.x86_64
0x126
3
? manage_workers.isra.25
vmlinux 0xffffffff88abadb0
Build id: 3.10.0-957.12.2.el7.x86_64
0x2a0
4
kthread
vmlinux 0xffffffff88ac1d21
Build id: 3.10.0-957.12.2.el7.x86_64
0xd1
5
? insert_kthread_work
vmlinux 0xffffffff88ac1c50
Build id: 3.10.0-957.12.2.el7.x86_64
0x40
6
ret_from_fork_nospec_begin
vmlinux 0xffffffff89175c1d
Build id: 3.10.0-957.12.2.el7.x86_64
0x7
7
? insert_kthread_work
vmlinux 0xffffffff88ac1c50
Build id: 3.10.0-957.12.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 31s! [kworker/u8:2:31350]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vboxsf(OE) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod ppdev joydev nfit libnvdimm iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device pcspkr snd_pcm i2c_piix4 sg parport_pc snd_timer video parport snd vboxguest(OE) soundcore nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm libahci ata_piix serio_raw drm drm_panel_orientation_quirks e1000 libata
CPU: 3 PID: 31350 Comm: kworker/u8:2 Kdump: loaded Tainted: G           OEL ------------   3.10.0-957.12.2.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Workqueue: rpciod rpc_async_schedule [sunrpc]
task: ffff88a218bfe180 ti: ffff88a2173c4000 task.ti: ffff88a2173c4000
RIP: 0010:[<ffffffff88ab9e2d>]  [<ffffffff88ab9e2d>] process_one_work+0x16d/0x440
RSP: 0018:ffff88a2173c7e28  EFLAGS: 00010246
RAX: 0000000000000001 RBX: 0000000000000096 RCX: ffff88a1f6bac000
RDX: 0000000000000001 RSI: 0000000000000096 RDI: ffff88a1f6b73c00
RBP: ffff88a2173c7e60 R08: 0000000082e12034 R09: ffffffff89ae9c00
R10: 0000000000000000 R11: 0000000000000400 R12: 00000000000008fd
R13: ffff88a64aa64904 R14: ffff88a64aa64100 R15: ffff88a69fd9b680
FS:  0000000000000000(0000) GS:ffff88a69fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9d9e528000 CR3: 00000003f975c000 CR4: 00000000000206e0
Call Trace:
[<ffffffff88abaed6>] worker_thread+0x126/0x3c0
[<ffffffff88abadb0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff88ac1d21>] kthread+0xd1/0xe0
[<ffffffff88ac1c50>] ? insert_kthread_work+0x40/0x40
[<ffffffff89175c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff88ac1c50>] ? insert_kthread_work+0x40/0x40
Code: 49 63 45 0c 48 8b 13 49 89 c7 49 c1 e7 05 83 e2 01 0f 84 15 02 00 00 4c 89 3b 4c 89 ef ff 14 25 50 43 63 89 fb 66 0f 1f 44 00 00 <0f> 1f 44 00 00 49 8b 44 24 18 48 89 df e8 e1 d0 2c 00 0f 1f 44 
Complete report #804011
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffff96f0d768
Build id: 3.10.0-862.el7.x86_64
0x19
2
__warn
vmlinux 0xffffffff968916d8
Build id: 3.10.0-862.el7.x86_64
0xd8
3
warn_slowpath_null
vmlinux 0xffffffff9689181d
Build id: 3.10.0-862.el7.x86_64
0x1d
4
process_one_work
vmlinux 0xffffffff968b2f3f
Build id: 3.10.0-862.el7.x86_64
0x2bf
5
worker_thread
vmlinux 0xffffffff968b3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
6
? manage_workers.isra.24
vmlinux 0xffffffff968b39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
7
kthread
vmlinux 0xffffffff968bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
8
? insert_kthread_work
vmlinux 0xffffffff968bad60
Build id: 3.10.0-862.el7.x86_64
0x40
9
ret_from_fork_nospec_begin
vmlinux 0xffffffff96f1f637
Build id: 3.10.0-862.el7.x86_64
0x21
10
? insert_kthread_work
vmlinux 0xffffffff968bad60
Build id: 3.10.0-862.el7.x86_64
0x40
WARNING: CPU: 14 PID: 162094 at kernel/workqueue.c:2208 process_one_work+0x2bf/0x440
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ucm rpcrdma rdma_ucm ib_umad dm_mirror dm_region_hash dm_log dm_mod ib_uverbs ib_iser rdma_cm ib_ipoib iw_cm libiscsi scsi_transport_iscsi ib_cm mlx5_ib sb_edac intel_powerclamp coretemp ib_core intel_rapl iosf_mbi kvm irqbypass ipmi_si ipmi_devintf crc32_pclmul mlx5_core ghash_clmulni_intel aesni_intel mlxfw lrw gf128mul glue_helper iTCO_wdt iTCO_vendor_support devlink ablk_helper pcspkr ipmi_msghandler cryptd dcdbas mxm_wmi wmi mei_me acpi_power_meter mei sg lpc_ich shpchp binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm ahci ixgbe drm libahci libata tg3 crct10dif_pclmul crct10dif_common crc32c_intel megaraid_sas mdio dca i2c_core ptp pps_core
CPU: 14 PID: 162094 Comm: kworker/14:0 Kdump: loaded Tainted: G        W      ------------   3.10.0-862.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R430/0CN7X8, BIOS 2.5.4 08/17/2017
Call Trace:
[<ffffffff96f0d768>] dump_stack+0x19/0x1b
[<ffffffff968916d8>] __warn+0xd8/0x100
[<ffffffff9689181d>] warn_slowpath_null+0x1d/0x20
[<ffffffff968b2f3f>] process_one_work+0x2bf/0x440
[<ffffffff968b3ac6>] worker_thread+0x126/0x3c0
[<ffffffff968b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff968bae31>] kthread+0xd1/0xe0
[<ffffffff968bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff96f1f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff968bad60>] ? insert_kthread_work+0x40/0x40
Complete report #721746
Frame # Function Binary Source or offset Line
1
? manage_workers.isra.25
vmlinux 0xffffffff882bbcc0
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x2a0
2
kthread
vmlinux 0xffffffff882c2c31
Build id: 3.10.0-957.el7.centos.plus.x86_64
0xd1
3
? insert_kthread_work
vmlinux 0xffffffff882c2b60
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x40
4
ret_from_fork_nospec_begin
vmlinux 0xffffffff88986c37
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x21
5
? insert_kthread_work
vmlinux 0xffffffff882c2b60
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x40
WARNING: CPU: 47 PID: 64574 at drivers/cpufreq/cpufreq.c:2394 cpufreq_update_policy+0x1e6/0x1f0
[<ffffffff882bbcc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff882c2c31>] kthread+0xd1/0xe0
[<ffffffff882c2b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff88986c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff882c2b60>] ? insert_kthread_work+0x40/0x40
Complete report #595861
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffffac9865c9
Build id: 3.10.0-1160.76.1.el7.x86_64
0x19
2
__warn
vmlinux 0xffffffffac29b408
Build id: 3.10.0-1160.76.1.el7.x86_64
0xd8
3
warn_slowpath_null
vmlinux 0xffffffffac29b54d
Build id: 3.10.0-1160.76.1.el7.x86_64
0x1d
4
process_one_work
vmlinux 0xffffffffac2be0ff
Build id: 3.10.0-1160.76.1.el7.x86_64
0x2bf
5
worker_thread
vmlinux 0xffffffffac2bf0d6
Build id: 3.10.0-1160.76.1.el7.x86_64
0x126
6
? manage_workers.isra.26
vmlinux 0xffffffffac2befb0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x2a0
7
kthread
vmlinux 0xffffffffac2c5f91
Build id: 3.10.0-1160.76.1.el7.x86_64
0xd1
8
? insert_kthread_work
vmlinux 0xffffffffac2c5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
9
ret_from_fork_nospec_begin
vmlinux 0xffffffffac999de4
Build id: 3.10.0-1160.76.1.el7.x86_64
0xe
10
? insert_kthread_work
vmlinux 0xffffffffac2c5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
WARNING: CPU: 8 PID: 50 at kernel/workqueue.c:2244 process_one_work+0x2bf/0x440
Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm dns_resolver vfat fat snd_hda_codec_hdmi ppdev edac_mce_amd snd_hda_intel kvm snd_hda_codec snd_hda_core snd_hwdep snd_seq irqbypass crc32_pclmul snd_seq_device ghash_clmulni_intel snd_pcm joydev aesni_intel lrw gf128mul snd_timer glue_helper ablk_helper snd cryptd nvme sg soundcore pcspkr nvme_core ccp k10temp i2c_piix4 parport_pc parport i2c_designware_platform i2c_designware_core pinctrl_amd gpio_amdpt acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic nouveau video mxm_wmi i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci libata crct10dif_pclmul crct10dif_common sdhci_acpi crc32c_intel r8169 iosf_mbi serio_raw sdhci drm_panel_orientation_quirks mmc_core wmi dm_mirror
dm_region_hash dm_log dm_mod
CPU: 8 PID: 50 Comm: kworker/8:0 Kdump: loaded Not tainted 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: Gigabyte Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F5 01/25/2019
Call Trace:
[<ffffffffac9865c9>] dump_stack+0x19/0x1b
[<ffffffffac29b408>] __warn+0xd8/0x100
[<ffffffffac29b54d>] warn_slowpath_null+0x1d/0x20
[<ffffffffac2be0ff>] process_one_work+0x2bf/0x440
[<ffffffffac2bf0d6>] worker_thread+0x126/0x3c0
[<ffffffffac2befb0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffac2c5f91>] kthread+0xd1/0xe0
[<ffffffffac2c5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffffac999de4>] ret_from_fork_nospec_begin+0xe/0x21
[<ffffffffac2c5ec0>] ? insert_kthread_work+0x40/0x40
Complete report #556146
Frame # Function Binary Source or offset Line
1
process_one_work
vmlinux 0xffffffff918bde7d
Build id: 3.10.0-1160.45.1.el7.x86_64
0x16d
2
worker_thread
vmlinux 0xffffffff918befa6
Build id: 3.10.0-1160.45.1.el7.x86_64
0x126
3
? manage_workers.isra.26
vmlinux 0xffffffff918bee80
Build id: 3.10.0-1160.45.1.el7.x86_64
0x2a0
4
kthread
vmlinux 0xffffffff918c5e61
Build id: 3.10.0-1160.45.1.el7.x86_64
0xd1
5
? insert_kthread_work
vmlinux 0xffffffff918c5d90
Build id: 3.10.0-1160.45.1.el7.x86_64
0x40
6
ret_from_fork_nospec_begin
vmlinux 0xffffffff91f95df7
Build id: 3.10.0-1160.45.1.el7.x86_64
0x21
7
? insert_kthread_work
vmlinux 0xffffffff918c5d90
Build id: 3.10.0-1160.45.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u2:0:1094]
Modules linked in: ppp_deflate bsd_comp ppp_async crc_ccitt ppp_generic slhc tcp_lp nls_utf8 isofs vboxsf(OE) nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter binfmt_misc overlay(T) snd_intel8x0 snd_ac97_codec ac97_bus joydev snd_seq ppdev snd_seq_device vboxguest(OE) snd_pcm iosf_mbi sg snd_timer pcspkr snd i2c_piix4 soundcore parport_pc video parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod cdrom crct10dif_generic crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ata_piix libahci drm libata crc32c_intel serio_raw e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 1094 Comm: kworker/u2:0 Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Workqueue: writeback bdi_writeback_workfn
task: ffff9eb0a7571080 ti: ffff9eb0a9784000 task.ti: ffff9eb0a9784000
RIP: 0010:[<ffffffff918bde7d>]  [<ffffffff918bde7d>] process_one_work+0x16d/0x440
RSP: 0018:ffff9eb0a9787e28  EFLAGS: 00010202
RAX: 0000000000000002 RBX: 0000000000000096 RCX: ffff9eb0a9784000
RDX: 0000000000000001 RSI: 0000000000000096 RDI: ffff9eb107d2e000
RBP: ffff9eb0a9787e60 R08: 0000000000000001 R09: ffff9eb0a7571080
R10: 853ccce45a731680 R11: 0000000000000005 R12: ffff9eb100811884
R13: ffffffff918d757f R14: ffff9eb0a9787da8 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff9eb15f200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000b8 CR3: 00000000a6abe000 CR4: 00000000000206f0
Call Trace:
[<ffffffff918befa6>] worker_thread+0x126/0x3c0
[<ffffffff918bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff918c5e61>] kthread+0xd1/0xe0
[<ffffffff918c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffff91f95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff918c5d90>] ? insert_kthread_work+0x40/0x40
Code: 49 63 45 0c 48 8b 13 49 89 c7 49 c1 e7 05 83 e2 01 0f 84 15 02 00 00 4c 89 3b 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <0f> 1f 44 00 00 49 8b 44 24 18 48 89 df e8 c1 a0 2d 00 0f 1f 44 
Complete report #548333
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffff88fb1bec
Build id: 3.10.0-1160.80.1.el7.x86_64
0x19
2
process_one_work
vmlinux 0xffffffff888c3389
Build id: 3.10.0-1160.80.1.el7.x86_64
0x369
3
worker_thread
vmlinux 0xffffffff888c42e6
Build id: 3.10.0-1160.80.1.el7.x86_64
0x126
4
? manage_workers.isra.26
vmlinux 0xffffffff888c41c0
Build id: 3.10.0-1160.80.1.el7.x86_64
0x2b0
5
kthread
vmlinux 0xffffffff888cb4d1
Build id: 3.10.0-1160.80.1.el7.x86_64
0xd1
6
? insert_kthread_work
vmlinux 0xffffffff888cb400
Build id: 3.10.0-1160.80.1.el7.x86_64
0x40
7
ret_from_fork_nospec_begin
vmlinux 0xffffffff88fc51f7
Build id: 3.10.0-1160.80.1.el7.x86_64
0x21
8
? insert_kthread_work
vmlinux 0xffffffff888cb400
Build id: 3.10.0-1160.80.1.el7.x86_64
0x40
BUG: workqueue leaked lock or atomic: kworker/1:1/0x000000ff/4609#012     last function: vmstat_update
CPU: 1 PID: 4609 Comm: kworker/1:1 Kdump: loaded Tainted: G        W      ------------ T 3.10.0-1160.80.1.el7.x86_64 #1
Hardware name: System manufacturer System Product Name/P5KPL-AM IN/ROEM/SI, BIOS 0508    05/31/2010
Workqueue: events vmstat_update
Call Trace:
[<ffffffff88fb1bec>] dump_stack+0x19/0x1f
[<ffffffff888c3389>] process_one_work+0x369/0x440
[<ffffffff888c42e6>] worker_thread+0x126/0x3c0
[<ffffffff888c41c0>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff888cb4d1>] kthread+0xd1/0xe0
[<ffffffff888cb400>] ? insert_kthread_work+0x40/0x40
[<ffffffff88fc51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff888cb400>] ? insert_kthread_work+0x40/0x40
Complete report #452825
Frame # Function Binary Source or offset Line
1
worker_thread
vmlinux 0xffffffffb70b4686
Build id: 3.10.0-862.9.1.el7.x86_64
0x126
2
? manage_workers.isra.24
vmlinux 0xffffffffb70b4560
Build id: 3.10.0-862.9.1.el7.x86_64
0x2a0
3
kthread
vmlinux 0xffffffffb70bb621
Build id: 3.10.0-862.9.1.el7.x86_64
0xd1
4
? insert_kthread_work
vmlinux 0xffffffffb70bb550
Build id: 3.10.0-862.9.1.el7.x86_64
0x40
5
ret_from_fork_nospec_begin
vmlinux 0xffffffffb77205f7
Build id: 3.10.0-862.9.1.el7.x86_64
0x21
6
? insert_kthread_work
vmlinux 0xffffffffb70bb550
Build id: 3.10.0-862.9.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 40s! [oracle_68289_pd:68289]
[<ffffffffb70b4686>] worker_thread+0x126/0x3c0
[<ffffffffb70b4560>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffb70bb621>] kthread+0xd1/0xe0
[<ffffffffb70bb550>] ? insert_kthread_work+0x40/0x40
[<ffffffffb77205f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb70bb550>] ? insert_kthread_work+0x40/0x40
Complete report #372754
Frame # Function Binary Source or offset Line
1
worker_thread
vmlinux 0xffffffff980bf054
Build id: 3.10.0-1160.59.1.el7.x86_64
0x1d4
2
? manage_workers.isra.26
vmlinux 0xffffffff980bee80
Build id: 3.10.0-1160.59.1.el7.x86_64
0x2a0
3
kthread
vmlinux 0xffffffff980c5e61
Build id: 3.10.0-1160.59.1.el7.x86_64
0xd1
4
? insert_kthread_work
vmlinux 0xffffffff980c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
5
ret_from_fork_nospec_begin
vmlinux 0xffffffff98799df7
Build id: 3.10.0-1160.59.1.el7.x86_64
0x21
6
? insert_kthread_work
vmlinux 0xffffffff980c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 30s! [systemd-journal:2212]
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 30s! [kworker/1:2:15155]
Modules linked in: talpa_vfshook(OE) talpa_pedconnector(OE) talpa_pedevice(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_ipv4 nf_log_common ip6table_filter nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci tcp_htcp ip_tables xfs libcrc32c vmwgfx
sr_mod cdrom drm_kms_helper ata_generic pata_acpi syscopyarea sysfillrect sysimgblt sd_mod crc_t10dif fb_sys_fops crct10dif_generic ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix serio_raw libata vmw_pvscsi e1000 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 15155 Comm: kworker/1:2 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff99c4e0978000 ti: ffff99c43c440000 task.ti: ffff99c43c440000
RIP: 0010:[<ffffffff980bf054>]  [<ffffffff980bf054>] worker_thread+0x1d4/0x3c0
RSP: 0018:ffff99c43c443e70  EFLAGS: 00000202
RAX: ffff99c4e0978000 RBX: ffff99c43c443e50 RCX: dead000000000200
RDX: 0000000000000003 RSI: ffff99c63fc9a578 RDI: ffff99c63fc9a540
RBP: ffff99c43c443ec0 R08: ffff99c452c8f100 R09: 0000000000000001
R10: 0000000000000005 R11: fffff55f4b930880 R12: 0000000000000000
R13: ffff99c63fc9acc0 R14: ffff99c5a5059f40 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff99c63fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000b8 CR3: 00000002e040a000 CR4: 00000000000407e0
Call Trace:
[<ffffffff980bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff980c5e61>] kthread+0xd1/0xe0
[<ffffffff980c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffff98799df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff980c5d90>] ? insert_kthread_work+0x40/0x40
Code: ff 66 0f 1f 44 00 00 4c 89 ff e8 58 c3 ff ff 48 8b 45 c8 4c 89 f7 48 c7 00 01 00 00 00 c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <e8> 67 d3 6c 00 4c 89 f7 e8 8f fb 6c 00 41 8b 47 58 a8 02 0f 84 
Complete report #162304
Frame # Function Binary Source or offset Line
1
? manage_workers.isra.24
vmlinux 0xffffffff9eeb70b0
Build id: 3.10.0-862.11.6.el7.x86_64
0x2a0
2
kthread
vmlinux 0xffffffff9eebdf21
Build id: 3.10.0-862.11.6.el7.x86_64
0xd1
3
? insert_kthread_work
vmlinux 0xffffffff9eebde50
Build id: 3.10.0-862.11.6.el7.x86_64
0x40
4
ret_from_fork_nospec_begin
vmlinux 0xffffffff9f5255f7
Build id: 3.10.0-862.11.6.el7.x86_64
0x21
5
? insert_kthread_work
vmlinux 0xffffffff9eebde50
Build id: 3.10.0-862.11.6.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 161s! [kworker/0:1H:497]
[<ffffffff9eeb70b0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff9eebdf21>] kthread+0xd1/0xe0
[<ffffffff9eebde50>] ? insert_kthread_work+0x40/0x40
[<ffffffff9f5255f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9eebde50>] ? insert_kthread_work+0x40/0x40
Complete report #69492
Frame # Function Binary Source or offset Line
1
worker_thread
vmlinux 0xffffffffa3cbc036
Build id: 3.10.0-957.27.2.el7.x86_64
0x126
2
? manage_workers.isra.25
vmlinux 0xffffffffa3cbbf10
Build id: 3.10.0-957.27.2.el7.x86_64
0x2a0
3
kthread
vmlinux 0xffffffffa3cc2e81
Build id: 3.10.0-957.27.2.el7.x86_64
0xd1
4
? insert_kthread_work
vmlinux 0xffffffffa3cc2db0
Build id: 3.10.0-957.27.2.el7.x86_64
0x40
5
ret_from_fork_nospec_begin
vmlinux 0xffffffffa4376c37
Build id: 3.10.0-957.27.2.el7.x86_64
0x21
6
? insert_kthread_work
vmlinux 0xffffffffa3cc2db0
Build id: 3.10.0-957.27.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 226s! [kube-proxy:16596]
[<ffffffffa3cbc036>] worker_thread+0x126/0x3c0
[<ffffffffa3cbbf10>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffa3cc2e81>] kthread+0xd1/0xe0
[<ffffffffa3cc2db0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa4376c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa3cc2db0>] ? insert_kthread_work+0x40/0x40
Complete report #54464
Frame # Function Binary Source or offset Line
1
process_one_work
vmlinux 0xffffffffa62b9d7d
Build id: 3.10.0-957.5.1.el7.x86_64
0x16d
2
worker_thread
vmlinux 0xffffffffa62bae26
Build id: 3.10.0-957.5.1.el7.x86_64
0x126
3
? manage_workers.isra.25
vmlinux 0xffffffffa62bad00
Build id: 3.10.0-957.5.1.el7.x86_64
0x2a0
4
kthread
vmlinux 0xffffffffa62c1c71
Build id: 3.10.0-957.5.1.el7.x86_64
0xd1
5
? insert_kthread_work
vmlinux 0xffffffffa62c1ba0
Build id: 3.10.0-957.5.1.el7.x86_64
0x40
6
ret_from_fork_nospec_begin
vmlinux 0xffffffffa6974c37
Build id: 3.10.0-957.5.1.el7.x86_64
0x21
7
? insert_kthread_work
vmlinux 0xffffffffa62c1ba0
Build id: 3.10.0-957.5.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [kworker/0:1H:2408]
Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom
sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi serio_raw crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm vmw_pvscsi ata_piix drm libata vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 2408 Comm: kworker/0:1H Kdump: loaded Tainted: G             L ------------   3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
Workqueue: kblockd blk_timeout_work
task: ffff8d7c356a0000 ti: ffff8d7e33188000 task.ti: ffff8d7e33188000
RIP: 0010:[<ffffffffa62b9d7d>]  [<ffffffffa62b9d7d>] process_one_work+0x16d/0x440
RSP: 0018:ffff8d7e3318be28  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff8d7e34168c80 RCX: dead000000000200
RDX: 0000000000000001 RSI: 4140000000000000 RDI: ffff8d7e3fc1a7c0
RBP: ffff8d7e3318be60 R08: ffff8d7e31ea8830 R09: ca06c5d4114a8828
R10: ca06c5d4114a8828 R11: 0000000000000000 R12: ffff8d7e3fc1ab80
R13: 0000000000000000 R14: ffff8d7c356a0698 R15: 000000012ae751a8
FS:  0000000000000000(0000) GS:ffff8d7e3fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd69836c000 CR3: 0000000234144000 CR4: 00000000003607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa62bae26>] worker_thread+0x126/0x3c0
[<ffffffffa62bad00>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffa62c1c71>] kthread+0xd1/0xe0
[<ffffffffa62c1ba0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa6974c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa62c1ba0>] ? insert_kthread_work+0x40/0x40
Code: 49 63 45 0c 48 8b 13 49 89 c7 49 c1 e7 05 83 e2 01 0f 84 15 02 00 00 4c 89 3b 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <0f> 1f 44 00 00 49 8b 44 24 18 48 89 df e8 c1 cf 2c 00 0f 1f 44 
Complete report #36758
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffff9450e80e
Build id: 3.10.0-862.6.3.el7.x86_64
0x19
2
__warn
vmlinux 0xffffffff93e91e18
Build id: 3.10.0-862.6.3.el7.x86_64
0xd8
3
warn_slowpath_fmt
vmlinux 0xffffffff93e91e9f
Build id: 3.10.0-862.6.3.el7.x86_64
0x5f
4
cancel_userptr
i915 0xffffffffc05d3f43
Build id: 3.10.0-862.6.3.el7.x86_64
0xf3
5
process_one_work
vmlinux 0xffffffff93eb35ef
Build id: 3.10.0-862.6.3.el7.x86_64
0x17f
6
worker_thread
vmlinux 0xffffffff93eb4686
Build id: 3.10.0-862.6.3.el7.x86_64
0x126
7
? manage_workers.isra.24
vmlinux 0xffffffff93eb4560
Build id: 3.10.0-862.6.3.el7.x86_64
0x2a0
8
kthread
vmlinux 0xffffffff93ebb621
Build id: 3.10.0-862.6.3.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffff93ebb550
Build id: 3.10.0-862.6.3.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffff945205dd
Build id: 3.10.0-862.6.3.el7.x86_64
0x7
11
? insert_kthread_work
vmlinux 0xffffffff93ebb550
Build id: 3.10.0-862.6.3.el7.x86_64
0x40
WARNING: CPU: 3 PID: 316 at drivers/gpu/drm/i915/i915_gem_userptr.c:89 cancel_userptr+0xf3/0x100 [i915]
Failed to release pages: bind_count=1, pages_pin_count=1, pin_display=0
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink sunrpc vfat fat dm_mirror dm_region_hash dm_log dm_mod intel_powerclamp coretemp intel_rapl kvm_intel kvm snd_hda_codec_hdmi irqbypass snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel crc32_pclmul snd_hda_codec ghash_clmulni_intel aesni_intel lrw sg snd_hda_core gf128mul glue_helper snd_hwdep ablk_helper snd_seq cryptd mei_wdt wmi mei_me iTCO_wdt iTCO_vendor_support ppdev snd_seq_device snd_pcm pcspkr snd_timer shpchp mei parport_pc parport pinctrl_sunrisepoint snd pinctrl_intel i2c_i801
tpm_infineon soundcore acpi_pad sch_fq_codel ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i915 iosf_mbi i2c_algo_bit drm_kms_helper syscopyarea sysfillrect ahci sysimgblt fb_sys_fops libahci drm e1000e libata crct10dif_pclmul crct10dif_common crc32c_intel 8139too ptp 8139cp serio_raw pps_core mii video i2c_hid i2c_core
CPU: 3 PID: 316 Comm: kworker/u16:4 Kdump: loaded Not tainted 3.10.0-862.6.3.el7.x86_64 #1
Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./B150-HD3-CF, BIOS F5 03/11/2016
Workqueue: i915-userptr-release cancel_userptr [i915]
Call Trace:
[<ffffffff9450e80e>] dump_stack+0x19/0x1b
[<ffffffff93e91e18>] __warn+0xd8/0x100
[<ffffffff93e91e9f>] warn_slowpath_fmt+0x5f/0x80
[<ffffffffc05d3f43>] cancel_userptr+0xf3/0x100 [i915]
[<ffffffff93eb35ef>] process_one_work+0x17f/0x440
[<ffffffff93eb4686>] worker_thread+0x126/0x3c0
[<ffffffff93eb4560>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff93ebb621>] kthread+0xd1/0xe0
[<ffffffff93ebb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff945205dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff93ebb550>] ? insert_kthread_work+0x40/0x40
Complete report #22358
Frame # Function Binary Source or offset Line
1
? manage_workers.isra.26
vmlinux 0xffffffff820bf210
Build id: 3.10.0-1062.12.1.el7.x86_64
0x2a0
2
kthread
vmlinux 0xffffffff820c61f1
Build id: 3.10.0-1062.12.1.el7.x86_64
0xd1
3
? insert_kthread_work
vmlinux 0xffffffff820c6120
Build id: 3.10.0-1062.12.1.el7.x86_64
0x40
4
ret_from_fork_nospec_begin
vmlinux 0xffffffff8278dd37
Build id: 3.10.0-1062.12.1.el7.x86_64
0x21
5
? insert_kthread_work
vmlinux 0xffffffff820c6120
Build id: 3.10.0-1062.12.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/1:1:14671]
[<ffffffff820bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff820c61f1>] kthread+0xd1/0xe0
[<ffffffff820c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffff8278dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff820c6120>] ? insert_kthread_work+0x40/0x40
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48 
Complete report #15606
Frame # Function Binary Source or offset Line
1
process_one_work
vmlinux 0xffffffffad2b2dea
Build id: 3.10.0-862.el7.x86_64
0x16a
2
worker_thread
vmlinux 0xffffffffad2b3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
3
? manage_workers.isra.24
vmlinux 0xffffffffad2b39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
4
kthread
vmlinux 0xffffffffad2bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
5
? insert_kthread_work
vmlinux 0xffffffffad2bad60
Build id: 3.10.0-862.el7.x86_64
0x40
6
ret_from_fork_nospec_begin
vmlinux 0xffffffffad91f637
Build id: 3.10.0-862.el7.x86_64
0x21
7
? insert_kthread_work
vmlinux 0xffffffffad2bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 25s! [kworker/u64:1:6465]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 ipt_MASQUERADE tun nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 ebtable_filter nf_defrag_ipv4 ebtables nf_nat_ipv4 xt_addrtype xt_conntrack nf_nat nf_conntrack ip6table_filter ip6_tables br_netfilter iptable_filter bridge stp llc devlink vmw_vsock_vmci_transport vsock overlay(T) sunrpc coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd vmw_balloon joydev pcspkr vmw_vmci sg i2c_piix4 floppy parport_pc parport shpchp ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix scsi_transport_spi drm crct10dif_pclmul
crct10dif_common libata mptscsih crc32c_intel e1000 serio_raw mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 6465 Comm: kworker/u64:1 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
Workqueue: mpt_poll_0 mpt_fault_reset_work [mptbase]
task: ffff8d84082d5ee0 ti: ffff8d853d2f8000 task.ti: ffff8d853d2f8000
RIP: 0010:[<ffffffffad2b2dea>]  [<ffffffffad2b2dea>] process_one_work+0x16a/0x440
RSP: 0018:ffff8d853d2fbe28  EFLAGS: 00000202
RAX: 0000000000000040 RBX: 0000000000000046 RCX: 0000000000000000
RDX: 0000000000000001 RSI: 0000000000000046 RDI: ffff8d7f74412000
RBP: ffff8d853d2fbe60 R08: 0000000000000000 R09: ffff8d84082d5ee0
R10: 00000000bcd0c000 R11: 0000000000000400 R12: ffffffffad2cba9d
R13: ffff8d853d2fbda8 R14: 0000000000000004 R15: ffff8d861ef18b00
FS:  0000000000000000(0000) GS:ffff8d861ef00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000c420009fff CR3: 0000000283fb8000 CR4: 00000000000007e0
Call Trace:
[<ffffffffad2b3ac6>] worker_thread+0x126/0x3c0
[<ffffffffad2b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffad2bae31>] kthread+0xd1/0xe0
[<ffffffffad2bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffad91f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffad2bad60>] ? insert_kthread_work+0x40/0x40
Code: d5 ff ff 49 63 45 0c 48 8b 13 49 89 c7 49 c1 e7 05 83 e2 01 0f 84 15 02 00 00 4c 89 3b 4c 89 ef c6 07 00 66 66 66 90 fb 66 66 90 <66> 66 90 66 66 66 66 90 49 8b 44 24 18 48 89 df e8 51 77 2a 00 
Complete report #10667
Frame # Function Binary Source or offset Line
1
worker_thread
vmlinux 0xffffffffbb2bf884
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1d4
2
? manage_workers.isra.26
vmlinux 0xffffffffbb2bf6b0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2a0
3
kthread
vmlinux 0xffffffffbb2c6691
Build id: 3.10.0-1127.19.1.el7.x86_64
0xd1
4
? insert_kthread_work
vmlinux 0xffffffffbb2c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
5
ret_from_fork_nospec_begin
vmlinux 0xffffffffbb992d37
Build id: 3.10.0-1127.19.1.el7.x86_64
0x21
6
? insert_kthread_work
vmlinux 0xffffffffbb2c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 67s! [kworker/1:2:24229]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper joydev cryptd ppdev sg pcspkr vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif syscopyarea crct10dif_generic sysfillrect sysimgblt fb_sys_fops ttm nfit libnvdimm drm ata_piix ahci libahci crct10dif_pclmul libata crct10dif_common crc32c_intel serio_raw vmxnet3
vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 24229 Comm: kworker/1:2 Kdump: loaded Tainted: G             L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff8cb23ad820e0 ti: ffff8cb59c3f0000 task.ti: ffff8cb59c3f0000
RIP: 0010:[<ffffffffbb2bf884>]  [<ffffffffbb2bf884>] worker_thread+0x1d4/0x3c0
RSP: 0018:ffff8cb59c3f3e70  EFLAGS: 00000202
RAX: ffff8cb23ad820e0 RBX: 000000000001acc0 RCX: dead000000000200
RDX: 0000000000000001 RSI: ffff8cb7bfc9a578 RDI: ffff8cb7bfc9a540
RBP: ffff8cb59c3f3ec0 R08: ffff8cb699454380 R09: 0000000000000001
R10: ffff8cb7a1bb90e8 R11: ffff8cb737683f00 R12: ffff8cb59c3f3fd8
R13: ffff8cb59c3f3fd8 R14: ffffffffbb985942 R15: ffff8cb59c3f3e50
FS:  0000000000000000(0000) GS:ffff8cb7bfc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000000000b8 CR3: 0000000622dc2000 CR4: 00000000001607e0
Call Trace:
[<ffffffffbb2bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffbb2c6691>] kthread+0xd1/0xe0
[<ffffffffbb2c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffbb992d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffbb2c65c0>] ? insert_kthread_work+0x40/0x40
Code: ff 66 0f 1f 44 00 00 4c 89 ff e8 58 c3 ff ff 48 8b 45 c8 4c 89 f7 48 c7 00 01 00 00 00 c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <e8> f7 64 6c 00 4c 89 f7 e8 1f 8d 6c 00 41 8b 47 58 a8 02 0f 84 
Complete report #10604
Frame # Function Binary Source or offset Line
1
process_one_work
vmlinux 0xffffffff900b612d
Build id: 3.10.0-862.11.6.el7.x86_64
0x16d
2
worker_thread
vmlinux 0xffffffff900b71d6
Build id: 3.10.0-862.11.6.el7.x86_64
0x126
3
? manage_workers.isra.24
vmlinux 0xffffffff900b70b0
Build id: 3.10.0-862.11.6.el7.x86_64
0x2a0
4
kthread
vmlinux 0xffffffff900bdf21
Build id: 3.10.0-862.11.6.el7.x86_64
0xd1
5
? insert_kthread_work
vmlinux 0xffffffff900bde50
Build id: 3.10.0-862.11.6.el7.x86_64
0x40
6
ret_from_fork_nospec_begin
vmlinux 0xffffffff907255f7
Build id: 3.10.0-862.11.6.el7.x86_64
0x21
7
? insert_kthread_work
vmlinux 0xffffffff900bde50
Build id: 3.10.0-862.11.6.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [kworker/u2:0:14333]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr vmw_balloon joydev vmw_vmci sg i2c_piix4 shpchp parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel mptspi scsi_transport_spi serio_raw
mptscsih e1000 mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 14333 Comm: kworker/u2:0 Tainted: G             L ------------   3.10.0-862.11.6.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/30/2013
Workqueue: mpt_poll_0 mpt_fault_reset_work [mptbase]
task: ffff9d8d340caf70 ti: ffff9d8eb1480000 task.ti: ffff9d8eb1480000
RIP: 0010:[<ffffffff900b612d>]  [<ffffffff900b612d>] process_one_work+0x16d/0x440
RSP: 0018:ffff9d8eb1483e28  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 0000000000000046 RCX: 0000000000000000
RDX: 0000000000000001 RSI: 0000000000000046 RDI: ffff9d8dfcf47000
RBP: ffff9d8eb1483e60 R08: 0000000000000000 R09: ffff9d8d340caf70
R10: 89c5f97bb384ad60 R11: 0000000000000000 R12: ffffffff900cebfd
R13: ffff9d8eb1483da8 R14: 0000000000000000 R15: ffff9d8ec1418b40
FS:  0000000000000000(0000) GS:ffff9d8ec1400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbd58142000 CR3: 00000000b6352000 CR4: 00000000000407f0
Call Trace:
[<ffffffff900b71d6>] worker_thread+0x126/0x3c0
[<ffffffff900b70b0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff900bdf21>] kthread+0xd1/0xe0
[<ffffffff900bde50>] ? insert_kthread_work+0x40/0x40
[<ffffffff907255f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff900bde50>] ? insert_kthread_work+0x40/0x40
Code: 49 63 45 0c 48 8b 13 49 89 c7 49 c1 e7 05 83 e2 01 0f 84 15 02 00 00 4c 89 3b 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <0f> 1f 44 00 00 49 8b 44 24 18 48 89 df e8 41 8f 2a 00 0f 1f 44 
Complete report #1102
Frame # Function Binary Source or offset Line
1
process_one_work
vmlinux 0xffffffffa44be6ad
Build id: 3.10.0-1127.el7.x86_64
0x16d
2
worker_thread
vmlinux 0xffffffffa44bf7d6
Build id: 3.10.0-1127.el7.x86_64
0x126
3
? manage_workers.isra.26
vmlinux 0xffffffffa44bf6b0
Build id: 3.10.0-1127.el7.x86_64
0x2a0
4
kthread
vmlinux 0xffffffffa44c6691
Build id: 3.10.0-1127.el7.x86_64
0xd1
5
? insert_kthread_work
vmlinux 0xffffffffa44c65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
6
ret_from_fork_nospec_begin
vmlinux 0xffffffffa4b92d37
Build id: 3.10.0-1127.el7.x86_64
0x21
7
? insert_kthread_work
vmlinux 0xffffffffa44c65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 38s! [vmtoolsd:878]
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 38s! [kworker/u8:2:29460]
Modules linked in: xt_set xt_multiport ip_set_hash_ip binfmt_misc nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper pcspkr joydev cryptd sg vmw_balloon vmw_vmci parport_pc parport i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc
ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea ata_piix sysfillrect sysimgblt fb_sys_fops mptspi ttm crct10dif_pclmul crct10dif_common libata scsi_transport_spi crc32c_intel drm serio_raw mptscsih vmxnet3 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 29460 Comm: kworker/u8:2 Kdump: loaded Not tainted 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
Workqueue: mpt_poll_0 mpt_fault_reset_work [mptbase]
task: ffff950ab5209070 ti: ffff950ab5224000 task.ti: ffff950ab5224000
RIP: 0010:[<ffffffffa44be6ad>]  [<ffffffffa44be6ad>] process_one_work+0x16d/0x440
RSP: 0018:ffff950ab5227e28  EFLAGS: 00000202
RAX: 0000000000000008 RBX: 0000000000000096 RCX: ffff950ab5224000
RDX: 0000000000000001 RSI: 0000000000000096 RDI: ffff9509ffd3c400
RBP: ffff950ab5227e60 R08: 0000000000000001 R09: ffff950ab5209070
R10: 000000000000bc0b R11: 0000000000000400 R12: ffff9509fce4a8e4
R13: ffffffffa44d7daf R14: ffff950ab5227da8 R15: 0000000000000004
FS:  0000000000000000(0000) GS:ffff950abfd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbce9acd000 CR3: 0000000235c3c000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa44bf7d6>] worker_thread+0x126/0x3c0
[<ffffffffa44bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffa44c6691>] kthread+0xd1/0xe0
[<ffffffffa44c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa4b92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa44c65c0>] ? insert_kthread_work+0x40/0x40
Code: 49 63 45 0c 48 8b 13 49 89 c7 49 c1 e7 05 83 e2 01 0f 84 15 02 00 00 4c 89 3b 4c 89 ef c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <0f> 1f 44 00 00 49 8b 44 24 18 48 89 df e8 51 7d 2d 00 0f 1f 44 
Complete report #770549
Frame # Function Binary Source or offset Line
1
process_one_work
vmlinux 0xffffffff8b8bd0ed
Build id: 3.10.0-1062.el7.x86_64
0x16d
2
worker_thread
vmlinux 0xffffffff8b8be216
Build id: 3.10.0-1062.el7.x86_64
0x126
3
? manage_workers.isra.26
vmlinux 0xffffffff8b8be0f0
Build id: 3.10.0-1062.el7.x86_64
0x2a0
4
kthread
vmlinux 0xffffffff8b8c50d1
Build id: 3.10.0-1062.el7.x86_64
0xd1
5
? insert_kthread_work
vmlinux 0xffffffff8b8c5000
Build id: 3.10.0-1062.el7.x86_64
0x40
6
ret_from_fork_nospec_begin
vmlinux 0xffffffff8bf8bd1d
Build id: 3.10.0-1062.el7.x86_64
0x7
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 21s! [kworker/5:0:120236]
Modules linked in: binfmt_misc nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq ppdev snd_seq_device snd_pcm vmw_balloon snd_timer snd iosf_mbi crc32_pclmul ghash_clmulni_intel sg soundcore aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit libnvdimm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel mptspi libata scsi_transport_spi e1000 mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 120236 Comm: kworker/5:0 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
Workqueue: events vmstat_update
task: ffff93dc35418000 ti: ffff93dbace7c000 task.ti: ffff93dbace7c000
RIP: 0010:[<ffffffff8b8bd0ed>]  [<ffffffff8b8bd0ed>] process_one_work+0x16d/0x440
RSP: 0018:ffff93dbace7fe28  EFLAGS: 00000202
RAX: 000000000000000a RBX: ffff93dbdd685140 RCX: dead000000000200
RDX: 0000000000000001 RSI: ea00000000000000 RDI: ffff93dc3d75a4c0
RBP: ffff93dbace7fe60 R08: ffff93dc3d757d48 R09: b08e9e0648757d40
R10: b08e9e0648757d40 R11: 0000000000000001 R12: ffff93dc3d75ac40
R13: 0000000000000000 R14: ffff93dc35418698 R15: 00000001a92e6308
FS:  0000000000000000(0000) GS:ffff93dc3d740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00001dcbf571a000 CR3: 0000000f85ed4000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffff8b8be216>] worker_thread+0x126/0x3c0
[<ffffffff8b8be0f0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8b8c50d1>] kthread+0xd1/0xe0
[<ffffffff8b8c5000>] ? insert_kthread_work+0x40/0x40
[<ffffffff8bf8bd1d>] ret_from_fork_nospec_begin+0x7/0x21
Complete report #610459
Frame # Function Binary Source or offset Line
1
worker_thread
vmlinux 0xffffffff836bf0f6
Build id: 3.10.0-1160.62.1.el7.x86_64
0x126
2
? manage_workers.isra.26
vmlinux 0xffffffff836befd0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x2a0
3
kthread
vmlinux 0xffffffff836c5fb1
Build id: 3.10.0-1160.62.1.el7.x86_64
0xd1
4
? insert_kthread_work
vmlinux 0xffffffff836c5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x40
5
ret_from_fork_nospec_begin
vmlinux 0xffffffff83d99df7
Build id: 3.10.0-1160.62.1.el7.x86_64
0x21
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:1:84348]
[<ffffffff836bf0f6>] worker_thread+0x126/0x3c0
[<ffffffff836befd0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff836c5fb1>] kthread+0xd1/0xe0
[<ffffffff836c5ee0>] ? insert_kthread_work+0x40/0x40
[<ffffffff83d99df7>] ret_from_fork_nospec_begin+0x21/0x21