Info

Permalink
Function
sched_show_task
First occurrence
2020-12-28
Last occurrence
2026-07-06
State
NEW
Components
kernel
Quality
-41

Statistics

Operating system Count
CentOS 7 6
Architecture Count
x86_64 6
Related packages Count
kernel 6
 0:3.10.0-957.1.3.el7 3
 0:3.10.0-1062.12.1.el7 1
 0:3.10.0-1160.42.2.el7 1
 0:3.10.0-1160.119.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1417536
Frame # Function Binary Source or offset Line
1
dump_cpu_task
vmlinux 0xffffffffb08e4049
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x39
2
rcu_dump_cpu_stacks
vmlinux 0xffffffffb0960df0
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x90
3
rcu_check_callbacks
vmlinux 0xffffffffb0964602
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x442
4
? tick_sched_do_timer
vmlinux 0xffffffffb0917650
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x60
5
update_process_times
vmlinux 0xffffffffb08b3fb6
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x46
6
tick_sched_handle
vmlinux 0xffffffffb0917380
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x30
7
tick_sched_timer
vmlinux 0xffffffffb0917689
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x39
8
__hrtimer_run_queues
vmlinux 0xffffffffb08cfb1e
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x10e
9
hrtimer_interrupt
vmlinux 0xffffffffb08d00af
Build id: 3.10.0-1160.119.1.el7.x86_64 		0xaf
10
local_apic_timer_interrupt
vmlinux 0xffffffffb085fd7b
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x3b
11
smp_apic_timer_interrupt
vmlinux 0xffffffffb0fca053
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x43
12
apic_timer_interrupt
vmlinux 0xffffffffb0fc63f2
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x172
13
? __do_softirq
vmlinux 0xffffffffb08a953a
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x9a
14
? __do_softirq
vmlinux 0xffffffffb08a94c0
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x20
15
call_softirq
vmlinux 0xffffffffb0fc8aac
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x1c
16
do_softirq
vmlinux 0xffffffffb0830825
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x65
17
irq_exit
vmlinux 0xffffffffb08a9945
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x115
18
smp_call_function_single_interrupt
vmlinux 0xffffffffb085c829
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x39
19
call_function_single_interrupt
vmlinux 0xffffffffb0fc7772
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x172 
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 48s! [ksoftirqd/8:51]
[<ffffffffb08e4049>] dump_cpu_task+0x39/0x70
[<ffffffffb0960df0>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffffb0964602>] rcu_check_callbacks+0x442/0x730
[<ffffffffb0917650>] ? tick_sched_do_timer+0x60/0x60
[<ffffffffb08b3fb6>] update_process_times+0x46/0x80
[<ffffffffb0917380>] tick_sched_handle+0x30/0x80
[<ffffffffb0917689>] tick_sched_timer+0x39/0x80
[<ffffffffb08cfb1e>] __hrtimer_run_queues+0x10e/0x270
[<ffffffffb08d00af>] hrtimer_interrupt+0xaf/0x1e0
[<ffffffffb085fd7b>] local_apic_timer_interrupt+0x3b/0x70
[<ffffffffb0fca053>] smp_apic_timer_interrupt+0x43/0x60
[<ffffffffb0fc63f2>] apic_timer_interrupt+0x172/0x180
[<ffffffffb08a953a>] ? __do_softirq+0x9a/0x290
[<ffffffffb08a94c0>] ? __do_softirq+0x20/0x290
[<ffffffffb0fc8aac>] call_softirq+0x1c/0x30
[<ffffffffb0830825>] do_softirq+0x65/0xa0
[<ffffffffb08a9945>] irq_exit+0x115/0x120
[<ffffffffb085c829>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffffb0fc7772>] call_function_single_interrupt+0x172/0x180
Complete report #181066
Frame # Function Binary Source or offset Line
1
sched_show_task
vmlinux 0xffffffff814d5d88
Build id: 3.10.0-957.1.3.el7.x86_64 		0xa8
2
rcu_check_callbacks
vmlinux 0xffffffff8155624e
Build id: 3.10.0-957.1.3.el7.x86_64 		0x72e
3
? tick_sched_do_timer
vmlinux 0xffffffff8150ae40
Build id: 3.10.0-957.1.3.el7.x86_64 		0x50
4
update_process_times
vmlinux 0xffffffff814ab136
Build id: 3.10.0-957.1.3.el7.x86_64 		0x46
5
tick_sched_handle
vmlinux 0xffffffff8150abb0
Build id: 3.10.0-957.1.3.el7.x86_64 		0x30
6
tick_sched_timer
vmlinux 0xffffffff8150ae79
Build id: 3.10.0-957.1.3.el7.x86_64 		0x39
7
__hrtimer_run_queues
vmlinux 0xffffffff814c5f93
Build id: 3.10.0-957.1.3.el7.x86_64 		0xf3
8
hrtimer_interrupt
vmlinux 0xffffffff814c651f
Build id: 3.10.0-957.1.3.el7.x86_64 		0xaf
9
local_apic_timer_interrupt
vmlinux 0xffffffff8145a2cb
Build id: 3.10.0-957.1.3.el7.x86_64 		0x3b
10
smp_apic_timer_interrupt
vmlinux 0xffffffff81b796c3
Build id: 3.10.0-957.1.3.el7.x86_64 		0x43
11
apic_timer_interrupt
vmlinux 0xffffffff81b75df2
Build id: 3.10.0-957.1.3.el7.x86_64 		0x162
12
? __do_softirq
vmlinux 0xffffffff814a0ea7
Build id: 3.10.0-957.1.3.el7.x86_64 		0x97
13
? __do_softirq
vmlinux 0xffffffff814a0e30
Build id: 3.10.0-957.1.3.el7.x86_64 		0x20
14
call_softirq
vmlinux 0xffffffff81b7832c
Build id: 3.10.0-957.1.3.el7.x86_64 		0x1c
15
do_softirq
vmlinux 0xffffffff8142e675
Build id: 3.10.0-957.1.3.el7.x86_64 		0x65
16
irq_exit
vmlinux 0xffffffff814a1285
Build id: 3.10.0-957.1.3.el7.x86_64 		0x105
17
smp_call_function_single_interrupt
vmlinux 0xffffffff81457169
Build id: 3.10.0-957.1.3.el7.x86_64 		0x39
18
call_function_single_interrupt
vmlinux 0xffffffff81b770a2
Build id: 3.10.0-957.1.3.el7.x86_64 		0x162
19
? reschedule_interrupt
vmlinux 0xffffffff81b775eb
Build id: 3.10.0-957.1.3.el7.x86_64 		0xeb
20
? retint_careful
vmlinux 0xffffffff81b6b4ed
Build id: 3.10.0-957.1.3.el7.x86_64 		0xb 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 220s! [vmtoolsd:5896]
INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 0, t=240036 jiffies, g=448799032, c=448799031, q=1575)
All QSes seen, last rcu_sched kthread activity 240036 (7383371365-7383131329), jiffies_till_next_fqs=3
nagios          R  running task        0  4156   4153 0x00000088
Call Trace:
<IRQ>  [<ffffffff814d5d88>] sched_show_task+0xa8/0x110
[<ffffffff8155624e>] rcu_check_callbacks+0x72e/0x730
[<ffffffff8150ae40>] ? tick_sched_do_timer+0x50/0x50
[<ffffffff814ab136>] update_process_times+0x46/0x80
[<ffffffff8150abb0>] tick_sched_handle+0x30/0x70
[<ffffffff8150ae79>] tick_sched_timer+0x39/0x80
[<ffffffff814c5f93>] __hrtimer_run_queues+0xf3/0x270
[<ffffffff814c651f>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffff8145a2cb>] local_apic_timer_interrupt+0x3b/0x60
[<ffffffff81b796c3>] smp_apic_timer_interrupt+0x43/0x60
[<ffffffff81b75df2>] apic_timer_interrupt+0x162/0x170
[<ffffffff814a0ea7>] ? __do_softirq+0x97/0x280
[<ffffffff814a0e30>] ? __do_softirq+0x20/0x280
[<ffffffff81b7832c>] call_softirq+0x1c/0x30
[<ffffffff8142e675>] do_softirq+0x65/0xa0
[<ffffffff814a1285>] irq_exit+0x105/0x110
[<ffffffff81457169>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffff81b770a2>] call_function_single_interrupt+0x162/0x170
<EOI>  [<ffffffff81b775eb>] ? reschedule_interrupt+0xeb/0x170
[<ffffffff81b6b4ed>] ? retint_careful+0xb/0x32
Complete report #23421
Frame # Function Binary Source or offset Line
1
sched_show_task
vmlinux 0xffffffff8c8da658
Build id: 3.10.0-1062.12.1.el7.x86_64 		0xa8
2
rcu_check_callbacks
vmlinux 0xffffffff8c95b95e
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x72e
3
? tick_sched_do_timer
vmlinux 0xffffffff8c90ff50
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x50
4
update_process_times
vmlinux 0xffffffff8c8af536
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x46
5
tick_sched_handle
vmlinux 0xffffffff8c90fcc0
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x30
6
tick_sched_timer
vmlinux 0xffffffff8c90ff89
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x39
7
__hrtimer_run_queues
vmlinux 0xffffffff8c8ca5ee
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x10e
8
hrtimer_interrupt
vmlinux 0xffffffff8c8cab4f
Build id: 3.10.0-1062.12.1.el7.x86_64 		0xaf
9
local_apic_timer_interrupt
vmlinux 0xffffffff8c85c61b
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x3b
10
smp_apic_timer_interrupt
vmlinux 0xffffffff8cf929d3
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x43
11
apic_timer_interrupt
vmlinux 0xffffffff8cf8eefa
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x16a
12
? run_timer_softirq
vmlinux 0xffffffff8c8ae5b6
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x46
13
? __do_softirq
vmlinux 0xffffffff8c8a52a7
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x97
14
? __do_softirq
vmlinux 0xffffffff8c8a5230
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x20
15
call_softirq
vmlinux 0xffffffff8cf9142c
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x1c
16
do_softirq
vmlinux 0xffffffff8c82f715
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x65
17
irq_exit
vmlinux 0xffffffff8c8a5685
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x105
18
smp_call_function_single_interrupt
vmlinux 0xffffffff8c859479
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x39
19
call_function_single_interrupt
vmlinux 0xffffffff8cf901aa
Build id: 3.10.0-1062.12.1.el7.x86_64 		0x16a 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 80s! [kworker/u128:2:16657]
Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter ipmi_devintf ipmi_msghandler vmw_vsock_vmci_transport vsock ppdev vmw_balloon pcspkr sg joydev parport_pc parport vmw_vmci i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel ata_piix mptspi serio_raw scsi_transport_spi mptscsih libata mptbase vmxnet3 drm_panel_orientation_quirks floppy
dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 16657 Comm: kworker/u128:2 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
All QSes seen, last rcu_sched kthread activity 85071 (24201809317-24201724246), jiffies_till_next_fqs=3
Workqueue: writeback bdi_writeback_workfn
xfsaild/dm-0    R  running task        0   614      2 0x00000008
Call Trace:
<IRQ>  [<ffffffff8c8da658>] sched_show_task+0xa8/0x110
[<ffffffff8c95b95e>] rcu_check_callbacks+0x72e/0x730
[<ffffffff8c90ff50>] ? tick_sched_do_timer+0x50/0x50
[<ffffffff8c8af536>] update_process_times+0x46/0x80
[<ffffffff8c90fcc0>] tick_sched_handle+0x30/0x70
[<ffffffff8c90ff89>] tick_sched_timer+0x39/0x80
[<ffffffff8c8ca5ee>] __hrtimer_run_queues+0x10e/0x270
[<ffffffff8c8cab4f>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffff8c85c61b>] local_apic_timer_interrupt+0x3b/0x60
[<ffffffff8cf929d3>] smp_apic_timer_interrupt+0x43/0x60
[<ffffffff8cf8eefa>] apic_timer_interrupt+0x16a/0x170
[<ffffffff8c8ae5b6>] ? run_timer_softirq+0x46/0x300
[<ffffffff8c8a52a7>] ? __do_softirq+0x97/0x280
[<ffffffff8c8a5230>] ? __do_softirq+0x20/0x280
[<ffffffff8cf9142c>] call_softirq+0x1c/0x30
[<ffffffff8c82f715>] do_softirq+0x65/0xa0
[<ffffffff8c8a5685>] irq_exit+0x105/0x110
[<ffffffff8c859479>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffff8cf901aa>] call_function_single_interrupt+0x16a/0x170