Info

Permalink
Function
sched_show_task
First occurrence
2020-10-13
Last occurrence
2026-07-05
State
NEW
Components
kernel
Quality
-23

Statistics

Operating system Count
CentOS 7 110
Architecture Count
x86_64 110
Related packages Count
kernel 110
 0:3.10.0-1160.81.1.el7 14
 0:3.10.0-1160.62.1.el7 8
 0:3.10.0-957.el7 7
 0:3.10.0-1160.83.1.el7 6
 0:3.10.0-1160.119.1.el7 5
 0:3.10.0-1160.el7 5
 0:3.10.0-957.21.3.el7 4
 0:3.10.0-1062.18.1.el7 4
 0:3.10.0-1160.105.1.el7 4
 0:3.10.0-1160.36.2.el7 4
 0:3.10.0-1160.45.1.el7 4
 0:3.10.0-1062.1.1.el7 3
 0:3.10.0-1127.19.1.el7 3
 0:3.10.0-1160.25.1.el7 3
 0:3.10.0-1160.59.1.el7 3
 0:3.10.0-1160.66.1.el7 3
 0:3.10.0-1160.88.1.el7 3
 0:3.10.0-1160.90.1.el7 3
 0:3.10.0-862.el7 3
 0:3.10.0-1160.42.2.el7 2
 0:3.10.0-1160.76.1.el7 2
 0:3.10.0-693.el7 2
 0:3.10.0-1160.102.1.el7 1
 0:3.10.0-1160.114.2.el7 1
 0:3.10.0-1160.15.2.el7 1
 0:3.10.0-1160.49.1.el7 1
 0:3.10.0-1160.6.1.el7 1
 0:3.10.0-1160.95.1.el7 1
 0:3.10.0-693.11.6.el7 1
 0:3.10.0-862.14.4.el7 1
 0:3.10.0-957.27.2.el7 1
 0:3.10.0-693.11.1.el7 1
 0:3.10.0-1160.92.1.el7 1
 0:3.10.0-957.1.3.el7 1
 0:3.10.0-514.el7 1
 0:3.10.0-1160.31.1.el7 1
 0:3.10.0-1160.53.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

No reports in the last 20 weeks.

Report backtrace


Complete report #736297
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffbcbba385
Build id: 3.10.0-1160.90.1.el7.x86_64
0x15
2
sched_show_task
vmlinux 0xffffffffbc4e02b8
Build id: 3.10.0-1160.90.1.el7.x86_64
0xa8
3
dump_cpu_task
vmlinux 0xffffffffbc4e4049
Build id: 3.10.0-1160.90.1.el7.x86_64
0x39
4
rcu_dump_cpu_stacks
vmlinux 0xffffffffbc560d60
Build id: 3.10.0-1160.90.1.el7.x86_64
0x90
5
rcu_check_callbacks
vmlinux 0xffffffffbc564572
Build id: 3.10.0-1160.90.1.el7.x86_64
0x442
6
? tick_sched_do_timer
vmlinux 0xffffffffbc5175c0
Build id: 3.10.0-1160.90.1.el7.x86_64
0x60
7
update_process_times
vmlinux 0xffffffffbc4b3fb6
Build id: 3.10.0-1160.90.1.el7.x86_64
0x46
8
tick_sched_handle
vmlinux 0xffffffffbc5172f0
Build id: 3.10.0-1160.90.1.el7.x86_64
0x30
9
tick_sched_timer
vmlinux 0xffffffffbc5175f9
Build id: 3.10.0-1160.90.1.el7.x86_64
0x39
10
__hrtimer_run_queues
vmlinux 0xffffffffbc4cfb1e
Build id: 3.10.0-1160.90.1.el7.x86_64
0x10e
11
hrtimer_interrupt
vmlinux 0xffffffffbc4d00af
Build id: 3.10.0-1160.90.1.el7.x86_64
0xaf
12
local_apic_timer_interrupt
vmlinux 0xffffffffbc45fa3b
Build id: 3.10.0-1160.90.1.el7.x86_64
0x3b
13
smp_apic_timer_interrupt
vmlinux 0xffffffffbcbca053
Build id: 3.10.0-1160.90.1.el7.x86_64
0x43
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 143s! [snmpd:1251]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon coretemp joydev pcspkr sg parport_pc i2c_piix4 parport vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom vmwgfx ata_generic pata_acpi drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_common drm nfit ahci libnvdimm crc32c_intel serio_raw libahci ata_piix libata vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 1251 Comm: snmpd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.90.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff956ebacc6300 ti: ffff957036748000 task.ti: ffff957036748000
INFO: rcu_sched self-detected stall on CPU
RIP: 0010:[<ffffffffbcbba385>]  [<ffffffffbcbba385>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff95703674bc98  EFLAGS: 00000246
RAX: 0000000000000246 RBX: ffff95703ffd6800 RCX: ffffacbd00c68000
RDX: 00000000f00d0001 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff95703674bc98 R08: ffffffffc01900c0 R09: 00000000000000c8
{ 0}  (t=197747 jiffies g=754390 c=754389 q=72)
R10: 2020202020202065 R11: 7669656365522020 R12: ffff95703ffd6800
Task dump for CPU 0:
R13: ffffccf7c26ded00 R14: ffff957000000000 R15: ffffffffbc64aced
FS:  00007f2b9522d880(0000) GS:ffff95703fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
nagios          R  running task        0  1621      1 0x00000080
CR2: 00007f2b9525d000 CR3: 000000023677a000 CR4: 00000000000007e0
Call Trace:
<IRQ>  [<ffffffffbc4e02b8>] sched_show_task+0xa8/0x110
[<ffffffffbc4e4049>] dump_cpu_task+0x39/0x70
[<ffffffffbc560d60>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffffbc564572>] rcu_check_callbacks+0x442/0x730
[<ffffffffbc5175c0>] ? tick_sched_do_timer+0x60/0x60
[<ffffffffbc4b3fb6>] update_process_times+0x46/0x80
[<ffffffffbc5172f0>] tick_sched_handle+0x30/0x80
[<ffffffffbc5175f9>] tick_sched_timer+0x39/0x80
[<ffffffffbc4cfb1e>] __hrtimer_run_queues+0x10e/0x270
[<ffffffffbc4d00af>] hrtimer_interrupt+0xaf/0x1e0
[<ffffffffbc45fa3b>] local_apic_timer_interrupt+0x3b/0x70
[<ffffffffbcbca053>] smp_apic_timer_interrupt+0x43/0x60
Complete report #704301
Frame # Function Binary Source or offset Line
1
dump_cpu_task
vmlinux 0xffffffff8eae4049
Build id: 3.10.0-1160.88.1.el7.x86_64
0x39
2
rcu_dump_cpu_stacks
vmlinux 0xffffffff8eb60d60
Build id: 3.10.0-1160.88.1.el7.x86_64
0x90
3
rcu_check_callbacks
vmlinux 0xffffffff8eb64572
Build id: 3.10.0-1160.88.1.el7.x86_64
0x442
4
? ttwu_do_activate
vmlinux 0xffffffff8eadd37f
Build id: 3.10.0-1160.88.1.el7.x86_64
0x6f
5
? tick_sched_do_timer
vmlinux 0xffffffff8eb175c0
Build id: 3.10.0-1160.88.1.el7.x86_64
0x60
6
update_process_times
vmlinux 0xffffffff8eab3fb6
Build id: 3.10.0-1160.88.1.el7.x86_64
0x46
7
tick_sched_handle
vmlinux 0xffffffff8eb172f0
Build id: 3.10.0-1160.88.1.el7.x86_64
0x30
8
tick_sched_timer
vmlinux 0xffffffff8eb175f9
Build id: 3.10.0-1160.88.1.el7.x86_64
0x39
9
__hrtimer_run_queues
vmlinux 0xffffffff8eacfb1e
Build id: 3.10.0-1160.88.1.el7.x86_64
0x10e
10
hrtimer_interrupt
vmlinux 0xffffffff8ead00af
Build id: 3.10.0-1160.88.1.el7.x86_64
0xaf
11
local_apic_timer_interrupt
vmlinux 0xffffffff8ea5fa3b
Build id: 3.10.0-1160.88.1.el7.x86_64
0x3b
12
smp_apic_timer_interrupt
vmlinux 0xffffffff8f1ca053
Build id: 3.10.0-1160.88.1.el7.x86_64
0x43
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 78s! [HangDetector:953]
Modules linked in:
[<ffffffff8eae4049>] dump_cpu_task+0x39/0x70
[<ffffffff8eb60d60>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffff8eb64572>] rcu_check_callbacks+0x442/0x730
[<ffffffff8eadd37f>] ? ttwu_do_activate+0x6f/0x80
[<ffffffff8eb175c0>] ? tick_sched_do_timer+0x60/0x60
[<ffffffff8eab3fb6>] update_process_times+0x46/0x80
[<ffffffff8eb172f0>] tick_sched_handle+0x30/0x80
[<ffffffff8eb175f9>] tick_sched_timer+0x39/0x80
[<ffffffff8eacfb1e>] __hrtimer_run_queues+0x10e/0x270
[<ffffffff8ead00af>] hrtimer_interrupt+0xaf/0x1e0
[<ffffffff8ea5fa3b>] local_apic_timer_interrupt+0x3b/0x70
[<ffffffff8f1ca053>] smp_apic_timer_interrupt+0x43/0x60
Complete report #557232
Frame # Function Binary Source or offset Line
1
sched_show_task
vmlinux 0xffffffffb0eda0c8
Build id: 3.10.0-1160.el7.x86_64
0xa8
2
dump_cpu_task
vmlinux 0xffffffffb0eddeb9
Build id: 3.10.0-1160.el7.x86_64
0x39
3
rcu_dump_cpu_stacks
vmlinux 0xffffffffb0f58220
Build id: 3.10.0-1160.el7.x86_64
0x90
4
rcu_check_callbacks
vmlinux 0xffffffffb0f5b8e2
Build id: 3.10.0-1160.el7.x86_64
0x442
5
? tick_sched_do_timer
vmlinux 0xffffffffb0f100d0
Build id: 3.10.0-1160.el7.x86_64
0x50
6
update_process_times
vmlinux 0xffffffffb0eaef66
Build id: 3.10.0-1160.el7.x86_64
0x46
7
tick_sched_handle
vmlinux 0xffffffffb0f0fe40
Build id: 3.10.0-1160.el7.x86_64
0x30
8
tick_sched_timer
vmlinux 0xffffffffb0f10109
Build id: 3.10.0-1160.el7.x86_64
0x39
9
__hrtimer_run_queues
vmlinux 0xffffffffb0eca01e
Build id: 3.10.0-1160.el7.x86_64
0x10e
10
hrtimer_interrupt
vmlinux 0xffffffffb0eca57f
Build id: 3.10.0-1160.el7.x86_64
0xaf
11
? __d_alloc
vmlinux 0xffffffffb1069815
Build id: 3.10.0-1160.el7.x86_64
0x25
12
local_apic_timer_interrupt
vmlinux 0xffffffffb0e5cdcb
Build id: 3.10.0-1160.el7.x86_64
0x3b
13
smp_apic_timer_interrupt
vmlinux 0xffffffffb1598a83
Build id: 3.10.0-1160.el7.x86_64
0x43
14
apic_timer_interrupt
vmlinux 0xffffffffb1594fba
Build id: 3.10.0-1160.el7.x86_64
0x16a
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u256:2:5489]
INFO: rcu_sched detected stalls on CPUs/tasks: {
INFO: rcu_sched self-detected stall on CPU
6
{ 6
} 
}  (t=349450 jiffies g=50095 c=50094 q=1266)
(detected by 7, t=349450 jiffies, g=50095, c=50094, q=1266)
Task dump for CPU 6:
elasticsearch[b R  running task        0  4972   3500 0x000001a0
Call Trace:
<IRQ>  [<ffffffffb0eda0c8>] sched_show_task+0xa8/0x110
[<ffffffffb0eddeb9>] dump_cpu_task+0x39/0x70
[<ffffffffb0f58220>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffffb0f5b8e2>] rcu_check_callbacks+0x442/0x730
[<ffffffffb0f100d0>] ? tick_sched_do_timer+0x50/0x50
[<ffffffffb0eaef66>] update_process_times+0x46/0x80
[<ffffffffb0f0fe40>] tick_sched_handle+0x30/0x70
[<ffffffffb0f10109>] tick_sched_timer+0x39/0x80
[<ffffffffb0eca01e>] __hrtimer_run_queues+0x10e/0x270
[<ffffffffb0eca57f>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffffb1069815>] ? __d_alloc+0x25/0x180
[<ffffffffb0e5cdcb>] local_apic_timer_interrupt+0x3b/0x60
[<ffffffffb1598a83>] smp_apic_timer_interrupt+0x43/0x60
[<ffffffffb1594fba>] apic_timer_interrupt+0x16a/0x170
Complete report #294770
Frame # Function Binary Source or offset Line
1
sched_show_task
vmlinux 0xffffffff810c3a98
Build id: 3.10.0-693.el7.x86_64
0xa8
2
dump_cpu_task
vmlinux 0xffffffff810c7429
Build id: 3.10.0-693.el7.x86_64
0x39
3
rcu_dump_cpu_stacks
vmlinux 0xffffffff81137b10
Build id: 3.10.0-693.el7.x86_64
0x90
4
rcu_check_callbacks
vmlinux 0xffffffff8113b132
Build id: 3.10.0-693.el7.x86_64
0x442
5
? tick_sched_do_timer
vmlinux 0xffffffff810f2ff0
Build id: 3.10.0-693.el7.x86_64
0x50
6
update_process_times
vmlinux 0xffffffff8109a2f6
Build id: 3.10.0-693.el7.x86_64
0x46
7
tick_sched_handle
vmlinux 0xffffffff810f2df0
Build id: 3.10.0-693.el7.x86_64
0x30
8
tick_sched_timer
vmlinux 0xffffffff810f3029
Build id: 3.10.0-693.el7.x86_64
0x39
9
__hrtimer_run_queues
vmlinux 0xffffffff810b4ae4
Build id: 3.10.0-693.el7.x86_64
0xd4
10
hrtimer_interrupt
vmlinux 0xffffffff810b507f
Build id: 3.10.0-693.el7.x86_64
0xaf
11
local_apic_timer_interrupt
vmlinux 0xffffffff81053895
Build id: 3.10.0-693.el7.x86_64
0x35
12
smp_apic_timer_interrupt
vmlinux 0xffffffff816b76bd
Build id: 3.10.0-693.el7.x86_64
0x3d
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [flux_app:17275]
INFO: rcu_sched self-detected stall on CPU { 5}  (t=122214 jiffies g=896438526 c=896438525 q=1952)
Task dump for CPU 5:
xfsaild/dm-0    R  running task        0   520      2 0x00000008
ffff880434bd6eb0 00000000a1738cbc ffff880237d438e0 ffffffff810c3a98
0000000000000005 ffffffff81a550c0 ffff880237d438f8 ffffffff810c7429
0000000000000006 ffff880237d43928 ffffffff81137b10 ffff880237d50260
Call Trace:
<IRQ>  [<ffffffff810c3a98>] sched_show_task+0xa8/0x110
[<ffffffff810c7429>] dump_cpu_task+0x39/0x70
[<ffffffff81137b10>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffff8113b132>] rcu_check_callbacks+0x442/0x730
[<ffffffff810f2ff0>] ? tick_sched_do_timer+0x50/0x50
[<ffffffff8109a2f6>] update_process_times+0x46/0x80
[<ffffffff810f2df0>] tick_sched_handle+0x30/0x70
[<ffffffff810f3029>] tick_sched_timer+0x39/0x80
[<ffffffff810b4ae4>] __hrtimer_run_queues+0xd4/0x260
[<ffffffff810b507f>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffff81053895>] local_apic_timer_interrupt+0x35/0x60
[<ffffffff816b76bd>] smp_apic_timer_interrupt+0x3d/0x50
Complete report #237229
Frame # Function Binary Source or offset Line
1
copy_page
vmlinux 0xffffffffa61955b9
Build id: 3.10.0-1160.36.2.el7.x86_64
0x49
2
sched_show_task
vmlinux 0xffffffffa5eda3a8
Build id: 3.10.0-1160.36.2.el7.x86_64
0xa8
3
dump_cpu_task
vmlinux 0xffffffffa5ede019
Build id: 3.10.0-1160.36.2.el7.x86_64
0x39
4
rcu_dump_cpu_stacks
vmlinux 0xffffffffa5f586a0
Build id: 3.10.0-1160.36.2.el7.x86_64
0x90
5
rcu_check_callbacks
vmlinux 0xffffffffa5f5bd62
Build id: 3.10.0-1160.36.2.el7.x86_64
0x442
6
? tick_sched_do_timer
vmlinux 0xffffffffa5f106e0
Build id: 3.10.0-1160.36.2.el7.x86_64
0x50
7
update_process_times
vmlinux 0xffffffffa5eaf156
Build id: 3.10.0-1160.36.2.el7.x86_64
0x46
8
tick_sched_handle
vmlinux 0xffffffffa5f10450
Build id: 3.10.0-1160.36.2.el7.x86_64
0x30
9
tick_sched_timer
vmlinux 0xffffffffa5f10719
Build id: 3.10.0-1160.36.2.el7.x86_64
0x39
10
__hrtimer_run_queues
vmlinux 0xffffffffa5eca23e
Build id: 3.10.0-1160.36.2.el7.x86_64
0x10e
11
hrtimer_interrupt
vmlinux 0xffffffffa5eca79f
Build id: 3.10.0-1160.36.2.el7.x86_64
0xaf
12
local_apic_timer_interrupt
vmlinux 0xffffffffa5e5cdfb
Build id: 3.10.0-1160.36.2.el7.x86_64
0x3b
13
smp_apic_timer_interrupt
vmlinux 0xffffffffa659aa23
Build id: 3.10.0-1160.36.2.el7.x86_64
0x43
14
apic_timer_interrupt
vmlinux 0xffffffffa6596fba
Build id: 3.10.0-1160.36.2.el7.x86_64
0x16a
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 278s! [mrtg:5698]
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 278s! [khugepaged:48]
Modules linked in: ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat vmw_vsock_vmci_transport vsock nf_conntrack sunrpc coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd pcspkr vmw_balloon joydev sg parport_pc i2c_piix4 parport vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul ata_piix crct10dif_common drm mptspi e1000 crc32c_intel scsi_transport_spi mptscsih serio_raw libata mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 48 Comm: khugepaged Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.36.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/30/2013
task: ffff881176cf0000 ti: ffff881176cf8000 task.ti: ffff881176cf8000
RIP: 0010:[<ffffffffa61955b9>]  [<ffffffffa61955b9>] copy_page+0x49/0xe0
RSP: 0018:ffff881176cfbd70  EFLAGS: 00010206
RAX: 0000000000000000 RBX: ffff881176cfbd30 RCX: 000000000000003a
RDX: 0000000000000000 RSI: ffff88113a76b000 RDI: ffff88111b48e000
INFO: rcu_sched self-detected stall on CPU { 0}  (t=298605 jiffies g=7110467 c=7110466 q=529)
RBP: ffff881176cfbe38 R08: 0000000000000000 R09: 00007fbae66e4c20
Task dump for CPU 0:
R10: 000000000148aea0 R11: 00007fbae66e95d7 R12: 0000000000000001
R13: 0000000000000001 R14: ffffdc1901596640 R15: ffff88117c4cb320
FS:  0000000000000000(0000) GS:ffff88117fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
rngd            R  running task        0   729      1 0x00000080
CR2: 00007f8f03bf5000 CR3: 00000000242d2000 CR4: 00000000000407e0
Call Trace:
<IRQ>  [<ffffffffa5eda3a8>] sched_show_task+0xa8/0x110
[<ffffffffa5ede019>] dump_cpu_task+0x39/0x70
[<ffffffffa5f586a0>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffffa5f5bd62>] rcu_check_callbacks+0x442/0x730
[<ffffffffa5f106e0>] ? tick_sched_do_timer+0x50/0x50
[<ffffffffa5eaf156>] update_process_times+0x46/0x80
[<ffffffffa5f10450>] tick_sched_handle+0x30/0x70
[<ffffffffa5f10719>] tick_sched_timer+0x39/0x80
[<ffffffffa5eca23e>] __hrtimer_run_queues+0x10e/0x270
[<ffffffffa5eca79f>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffffa5e5cdfb>] local_apic_timer_interrupt+0x3b/0x60
[<ffffffffa659aa23>] smp_apic_timer_interrupt+0x43/0x60
[<ffffffffa6596fba>] apic_timer_interrupt+0x16a/0x170
Complete report #136899
Frame # Function Binary Source or offset Line
1
rcu_dump_cpu_stacks
vmlinux 0xffffffffbd758b30
Build id: 3.10.0-1127.19.1.el7.x86_64
0x90
2
rcu_check_callbacks
vmlinux 0xffffffffbd75c1f2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x442
3
? tick_sched_do_timer
vmlinux 0xffffffffbd710b00
Build id: 3.10.0-1127.19.1.el7.x86_64
0x50
4
update_process_times
vmlinux 0xffffffffbd6af9d6
Build id: 3.10.0-1127.19.1.el7.x86_64
0x46
5
tick_sched_handle
vmlinux 0xffffffffbd710870
Build id: 3.10.0-1127.19.1.el7.x86_64
0x30
6
tick_sched_timer
vmlinux 0xffffffffbd710b39
Build id: 3.10.0-1127.19.1.el7.x86_64
0x39
7
__hrtimer_run_queues
vmlinux 0xffffffffbd6caa8e
Build id: 3.10.0-1127.19.1.el7.x86_64
0x10e
8
hrtimer_interrupt
vmlinux 0xffffffffbd6cafef
Build id: 3.10.0-1127.19.1.el7.x86_64
0xaf
9
local_apic_timer_interrupt
vmlinux 0xffffffffbd65ccfb
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3b
10
smp_apic_timer_interrupt
vmlinux 0xffffffffbdd979c3
Build id: 3.10.0-1127.19.1.el7.x86_64
0x43
11
apic_timer_interrupt
vmlinux 0xffffffffbdd93efa
Build id: 3.10.0-1127.19.1.el7.x86_64
0x16a
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 2130s! [RequestThrottle:113452]
[<ffffffffbd758b30>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffffbd75c1f2>] rcu_check_callbacks+0x442/0x730
[<ffffffffbd710b00>] ? tick_sched_do_timer+0x50/0x50
[<ffffffffbd6af9d6>] update_process_times+0x46/0x80
[<ffffffffbd710870>] tick_sched_handle+0x30/0x70
[<ffffffffbd710b39>] tick_sched_timer+0x39/0x80
[<ffffffffbd6caa8e>] __hrtimer_run_queues+0x10e/0x270
[<ffffffffbd6cafef>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffffbd65ccfb>] local_apic_timer_interrupt+0x3b/0x60
[<ffffffffbdd979c3>] smp_apic_timer_interrupt+0x43/0x60
[<ffffffffbdd93efa>] apic_timer_interrupt+0x16a/0x170
Complete report #136240
Frame # Function Binary Source or offset Line
1
sched_show_task
vmlinux 0xffffffff810c41d8
Build id: 3.10.0-514.el7.x86_64
0xa8
2
dump_cpu_task
vmlinux 0xffffffff810c7a79
Build id: 3.10.0-514.el7.x86_64
0x39
3
rcu_dump_cpu_stacks
vmlinux 0xffffffff811372a0
Build id: 3.10.0-514.el7.x86_64
0x90
4
rcu_check_callbacks
vmlinux 0xffffffff8113a9f2
Build id: 3.10.0-514.el7.x86_64
0x442
5
? tick_sched_handle.isra.13
vmlinux 0xffffffff810f2f80
Build id: 3.10.0-514.el7.x86_64
0x60
6
update_process_times
vmlinux 0xffffffff81099177
Build id: 3.10.0-514.el7.x86_64
0x47
7
tick_sched_handle.isra.13
vmlinux 0xffffffff810f2f45
Build id: 3.10.0-514.el7.x86_64
0x25
8
tick_sched_timer
vmlinux 0xffffffff810f2fc1
Build id: 3.10.0-514.el7.x86_64
0x41
9
__hrtimer_run_queues
vmlinux 0xffffffff810b4862
Build id: 3.10.0-514.el7.x86_64
0xd2
10
hrtimer_interrupt
vmlinux 0xffffffff810b4e00
Build id: 3.10.0-514.el7.x86_64
0xb0
11
local_apic_timer_interrupt
vmlinux 0xffffffff810510d7
Build id: 3.10.0-514.el7.x86_64
0x37
12
smp_apic_timer_interrupt
vmlinux 0xffffffff81698ccf
Build id: 3.10.0-514.el7.x86_64
0x3f
13
apic_timer_interrupt
vmlinux 0xffffffff8169721d
Build id: 3.10.0-514.el7.x86_64
0x6d
14
? _raw_spin_unlock_irqrestore
vmlinux 0xffffffff8168d7ab
Build id: 3.10.0-514.el7.x86_64
0x1b
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 67s! [java:32522]
INFO: rcu_sched self-detected stall on CPU { 4}  (t=122920 jiffies g=713312 c=713311 q=352)
Task dump for CPU 4:
kworker/u16:31  R  running task        0   420      2 0x00000000
Workqueue: scsi_tmf_0 scmd_eh_abort_handler
ffff880230da3ec0 00000000f8dd8034 ffff88023fd03db0 ffffffff810c41d8
0000000000000004 ffffffff81a1a580 ffff88023fd03dc8 ffffffff810c7a79
0000000000000005 ffff88023fd03df8 ffffffff811372a0 ffff88023fd101c0
Call Trace:
<IRQ>  [<ffffffff810c41d8>] sched_show_task+0xa8/0x110
[<ffffffff810c7a79>] dump_cpu_task+0x39/0x70
[<ffffffff811372a0>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffff8113a9f2>] rcu_check_callbacks+0x442/0x720
[<ffffffff810f2f80>] ? tick_sched_handle.isra.13+0x60/0x60
[<ffffffff81099177>] update_process_times+0x47/0x80
[<ffffffff810f2f45>] tick_sched_handle.isra.13+0x25/0x60
[<ffffffff810f2fc1>] tick_sched_timer+0x41/0x70
[<ffffffff810b4862>] __hrtimer_run_queues+0xd2/0x260
[<ffffffff810b4e00>] hrtimer_interrupt+0xb0/0x1e0
[<ffffffff810510d7>] local_apic_timer_interrupt+0x37/0x60
[<ffffffff81698ccf>] smp_apic_timer_interrupt+0x3f/0x60
[<ffffffff8169721d>] apic_timer_interrupt+0x6d/0x80
<EOI>  [<ffffffff8168d7ab>] ? _raw_spin_unlock_irqrestore+0x1b/0x40
Complete report #135431
Frame # Function Binary Source or offset Line
1
dump_cpu_task
vmlinux 0xffffffffa98de919
Build id: 3.10.0-1127.19.1.el7.x86_64
0x39
2
rcu_dump_cpu_stacks
vmlinux 0xffffffffa9958b30
Build id: 3.10.0-1127.19.1.el7.x86_64
0x90
3
rcu_check_callbacks
vmlinux 0xffffffffa995c1f2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x442
4
? tick_sched_do_timer
vmlinux 0xffffffffa9910b00
Build id: 3.10.0-1127.19.1.el7.x86_64
0x50
5
update_process_times
vmlinux 0xffffffffa98af9d6
Build id: 3.10.0-1127.19.1.el7.x86_64
0x46
6
tick_sched_handle
vmlinux 0xffffffffa9910870
Build id: 3.10.0-1127.19.1.el7.x86_64
0x30
7
tick_sched_timer
vmlinux 0xffffffffa9910b39
Build id: 3.10.0-1127.19.1.el7.x86_64
0x39
8
__hrtimer_run_queues
vmlinux 0xffffffffa98caa8e
Build id: 3.10.0-1127.19.1.el7.x86_64
0x10e
9
hrtimer_interrupt
vmlinux 0xffffffffa98cafef
Build id: 3.10.0-1127.19.1.el7.x86_64
0xaf
10
local_apic_timer_interrupt
vmlinux 0xffffffffa985ccfb
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3b
11
smp_apic_timer_interrupt
vmlinux 0xffffffffa9f979c3
Build id: 3.10.0-1127.19.1.el7.x86_64
0x43
12
apic_timer_interrupt
vmlinux 0xffffffffa9f93efa
Build id: 3.10.0-1127.19.1.el7.x86_64
0x16a
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [gnome-shell:45206]
[<ffffffffa98de919>] dump_cpu_task+0x39/0x70
[<ffffffffa9958b30>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffffa995c1f2>] rcu_check_callbacks+0x442/0x730
[<ffffffffa9910b00>] ? tick_sched_do_timer+0x50/0x50
[<ffffffffa98af9d6>] update_process_times+0x46/0x80
[<ffffffffa9910870>] tick_sched_handle+0x30/0x70
[<ffffffffa9910b39>] tick_sched_timer+0x39/0x80
[<ffffffffa98caa8e>] __hrtimer_run_queues+0x10e/0x270
[<ffffffffa98cafef>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffffa985ccfb>] local_apic_timer_interrupt+0x3b/0x60
[<ffffffffa9f979c3>] smp_apic_timer_interrupt+0x43/0x60
[<ffffffffa9f93efa>] apic_timer_interrupt+0x16a/0x170
Complete report #68274
Frame # Function Binary Source or offset Line
1
sched_show_task
vmlinux 0xffffffff810c3aa8
Build id: 3.10.0-693.11.1.el7.x86_64
0xa8
2
dump_cpu_task
vmlinux 0xffffffff810c7439
Build id: 3.10.0-693.11.1.el7.x86_64
0x39
3
rcu_dump_cpu_stacks
vmlinux 0xffffffff81137b20
Build id: 3.10.0-693.11.1.el7.x86_64
0x90
4
rcu_check_callbacks
vmlinux 0xffffffff8113b142
Build id: 3.10.0-693.11.1.el7.x86_64
0x442
5
? tick_sched_do_timer
vmlinux 0xffffffff810f3000
Build id: 3.10.0-693.11.1.el7.x86_64
0x50
6
update_process_times
vmlinux 0xffffffff8109a306
Build id: 3.10.0-693.11.1.el7.x86_64
0x46
7
tick_sched_handle
vmlinux 0xffffffff810f2e00
Build id: 3.10.0-693.11.1.el7.x86_64
0x30
8
tick_sched_timer
vmlinux 0xffffffff810f3039
Build id: 3.10.0-693.11.1.el7.x86_64
0x39
9
__hrtimer_run_queues
vmlinux 0xffffffff810b4af4
Build id: 3.10.0-693.11.1.el7.x86_64
0xd4
10
hrtimer_interrupt
vmlinux 0xffffffff810b508f
Build id: 3.10.0-693.11.1.el7.x86_64
0xaf
11
local_apic_timer_interrupt
vmlinux 0xffffffff810538f5
Build id: 3.10.0-693.11.1.el7.x86_64
0x35
12
smp_apic_timer_interrupt
vmlinux 0xffffffff816b777d
Build id: 3.10.0-693.11.1.el7.x86_64
0x3d
13
apic_timer_interrupt
vmlinux 0xffffffff816b5cdd
Build id: 3.10.0-693.11.1.el7.x86_64
0x6d
14
? _raw_spin_unlock_irqrestore
vmlinux 0xffffffff816ab915
Build id: 3.10.0-693.11.1.el7.x86_64
0x15
15
pvscsi_queue
vmw_pvscsi 0xffffffffc0052632
Build id: 3.10.0-693.11.1.el7.x86_64
0x442
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [java:12223]
INFO: rcu_sched self-detected stall on CPU { 1}  (t=61448 jiffies g=24273748 c=24273747 q=1735)
Task dump for CPU 1:
kworker/u8:0    R  running task        0 32485      2 0x00000088
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
ffff88016fbf9fa0 00000000c5ca7baa ffff880a2fc83db8 ffffffff810c3aa8
0000000000000001 ffffffff81a550c0 ffff880a2fc83dd0 ffffffff810c7439
0000000000000002 ffff880a2fc83e00 ffffffff81137b20 ffff880a2fc90260
Call Trace:
<IRQ>  [<ffffffff810c3aa8>] sched_show_task+0xa8/0x110
[<ffffffff810c7439>] dump_cpu_task+0x39/0x70
[<ffffffff81137b20>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffff8113b142>] rcu_check_callbacks+0x442/0x730
[<ffffffff810f3000>] ? tick_sched_do_timer+0x50/0x50
[<ffffffff8109a306>] update_process_times+0x46/0x80
[<ffffffff810f2e00>] tick_sched_handle+0x30/0x70
[<ffffffff810f3039>] tick_sched_timer+0x39/0x80
[<ffffffff810b4af4>] __hrtimer_run_queues+0xd4/0x260
[<ffffffff810b508f>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffff810538f5>] local_apic_timer_interrupt+0x35/0x60
[<ffffffff816b777d>] smp_apic_timer_interrupt+0x3d/0x50
[<ffffffff816b5cdd>] apic_timer_interrupt+0x6d/0x80
<EOI>  [<ffffffff816ab915>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[<ffffffffc0052632>] pvscsi_queue+0x442/0x7e1 [vmw_pvscsi]
Complete report #67200
Frame # Function Binary Source or offset Line
1
sched_show_task
vmlinux 0xffffffffa3cd7218
Build id: 3.10.0-957.27.2.el7.x86_64
0xa8
2
dump_cpu_task
vmlinux 0xffffffffa3cdaf29
Build id: 3.10.0-957.27.2.el7.x86_64
0x39
3
rcu_dump_cpu_stacks
vmlinux 0xffffffffa3d53c90
Build id: 3.10.0-957.27.2.el7.x86_64
0x90
4
rcu_check_callbacks
vmlinux 0xffffffffa3d57322
Build id: 3.10.0-957.27.2.el7.x86_64
0x442
5
? tick_sched_do_timer
vmlinux 0xffffffffa3d0c160
Build id: 3.10.0-957.27.2.el7.x86_64
0x50
6
update_process_times
vmlinux 0xffffffffa3cac386
Build id: 3.10.0-957.27.2.el7.x86_64
0x46
7
tick_sched_handle
vmlinux 0xffffffffa3d0bed0
Build id: 3.10.0-957.27.2.el7.x86_64
0x30
8
tick_sched_timer
vmlinux 0xffffffffa3d0c199
Build id: 3.10.0-957.27.2.el7.x86_64
0x39
9
__hrtimer_run_queues
vmlinux 0xffffffffa3cc71e3
Build id: 3.10.0-957.27.2.el7.x86_64
0xf3
10
hrtimer_interrupt
vmlinux 0xffffffffa3cc776f
Build id: 3.10.0-957.27.2.el7.x86_64
0xaf
11
local_apic_timer_interrupt
vmlinux 0xffffffffa3c5a61b
Build id: 3.10.0-957.27.2.el7.x86_64
0x3b
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [etcd:1680]
INFO: rcu_sched self-detected stall on CPU
INFO: rcu_sched detected stalls on CPUs/tasks:
{
{
1
1
} 
(t=361925 jiffies g=6551209 c=6551208 q=80)
} (detected by 0, t=361925 jiffies, g=6551209, c=6551208, q=80)
Task dump for CPU 1:
kworker/u16:1   R  running task        0 30351      2 0x00000088
Workqueue: scsi_tmf_2 scmd_eh_abort_handler
Call Trace:
<IRQ>  [<ffffffffa3cd7218>] sched_show_task+0xa8/0x110
[<ffffffffa3cdaf29>] dump_cpu_task+0x39/0x70
[<ffffffffa3d53c90>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffffa3d57322>] rcu_check_callbacks+0x442/0x730
[<ffffffffa3d0c160>] ? tick_sched_do_timer+0x50/0x50
[<ffffffffa3cac386>] update_process_times+0x46/0x80
[<ffffffffa3d0bed0>] tick_sched_handle+0x30/0x70
[<ffffffffa3d0c199>] tick_sched_timer+0x39/0x80
[<ffffffffa3cc71e3>] __hrtimer_run_queues+0xf3/0x270
[<ffffffffa3cc776f>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffffa3c5a61b>] local_apic_timer_interrupt+0x3b/0x60
Complete report #58532
Frame # Function Binary Source or offset Line
1
dump_cpu_task
vmlinux 0xffffffff8d8d4e39
Build id: 3.10.0-862.14.4.el7.x86_64
0x39
2
rcu_dump_cpu_stacks
vmlinux 0xffffffff8d94bb00
Build id: 3.10.0-862.14.4.el7.x86_64
0x90
3
rcu_check_callbacks
vmlinux 0xffffffff8d94f1a2
Build id: 3.10.0-862.14.4.el7.x86_64
0x442
4
? tick_sched_do_timer
vmlinux 0xffffffff8d9046c0
Build id: 3.10.0-862.14.4.el7.x86_64
0x50
5
update_process_times
vmlinux 0xffffffff8d8a7a16
Build id: 3.10.0-862.14.4.el7.x86_64
0x46
6
tick_sched_handle
vmlinux 0xffffffff8d9044c0
Build id: 3.10.0-862.14.4.el7.x86_64
0x30
7
tick_sched_timer
vmlinux 0xffffffff8d9046f9
Build id: 3.10.0-862.14.4.el7.x86_64
0x39
8
__hrtimer_run_queues
vmlinux 0xffffffff8d8c2163
Build id: 3.10.0-862.14.4.el7.x86_64
0xf3
9
hrtimer_interrupt
vmlinux 0xffffffff8d8c26ef
Build id: 3.10.0-862.14.4.el7.x86_64
0xaf
10
local_apic_timer_interrupt
vmlinux 0xffffffff8d8596ab
Build id: 3.10.0-862.14.4.el7.x86_64
0x3b
11
smp_apic_timer_interrupt
vmlinux 0xffffffff8df2a083
Build id: 3.10.0-862.14.4.el7.x86_64
0x43
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 264s! [cc1plus:34065]
[<ffffffff8d8d4e39>] dump_cpu_task+0x39/0x70
[<ffffffff8d94bb00>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffff8d94f1a2>] rcu_check_callbacks+0x442/0x730
[<ffffffff8d9046c0>] ? tick_sched_do_timer+0x50/0x50
[<ffffffff8d8a7a16>] update_process_times+0x46/0x80
[<ffffffff8d9044c0>] tick_sched_handle+0x30/0x70
[<ffffffff8d9046f9>] tick_sched_timer+0x39/0x80
[<ffffffff8d8c2163>] __hrtimer_run_queues+0xf3/0x270
[<ffffffff8d8c26ef>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffff8d8596ab>] local_apic_timer_interrupt+0x3b/0x60
[<ffffffff8df2a083>] smp_apic_timer_interrupt+0x43/0x60
Complete report #41435
Frame # Function Binary Source or offset Line
1
sched_show_task
vmlinux 0xffffffffb8eda788
Build id: 3.10.0-1062.18.1.el7.x86_64
0xa8
2
dump_cpu_task
vmlinux 0xffffffffb8ede529
Build id: 3.10.0-1062.18.1.el7.x86_64
0x39
3
rcu_dump_cpu_stacks
vmlinux 0xffffffffb8f581f0
Build id: 3.10.0-1062.18.1.el7.x86_64
0x90
4
rcu_check_callbacks
vmlinux 0xffffffffb8f5b8b2
Build id: 3.10.0-1062.18.1.el7.x86_64
0x442
5
? tick_sched_do_timer
vmlinux 0xffffffffb8f10190
Build id: 3.10.0-1062.18.1.el7.x86_64
0x50
6
update_process_times
vmlinux 0xffffffffb8eaf666
Build id: 3.10.0-1062.18.1.el7.x86_64
0x46
7
tick_sched_handle
vmlinux 0xffffffffb8f0ff00
Build id: 3.10.0-1062.18.1.el7.x86_64
0x30
8
tick_sched_timer
vmlinux 0xffffffffb8f101c9
Build id: 3.10.0-1062.18.1.el7.x86_64
0x39
9
__hrtimer_run_queues
vmlinux 0xffffffffb8eca71e
Build id: 3.10.0-1062.18.1.el7.x86_64
0x10e
10
hrtimer_interrupt
vmlinux 0xffffffffb8ecac7f
Build id: 3.10.0-1062.18.1.el7.x86_64
0xaf
11
local_apic_timer_interrupt
vmlinux 0xffffffffb8e5c61b
Build id: 3.10.0-1062.18.1.el7.x86_64
0x3b
12
smp_apic_timer_interrupt
vmlinux 0xffffffffb95929d3
Build id: 3.10.0-1062.18.1.el7.x86_64
0x43
13
apic_timer_interrupt
vmlinux 0xffffffffb958eefa
Build id: 3.10.0-1062.18.1.el7.x86_64
0x16a
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 60s! [kworker/u256:2:45204]
<IRQ>  [<ffffffffb8eda788>] sched_show_task+0xa8/0x110
[<ffffffffb8ede529>] dump_cpu_task+0x39/0x70
[<ffffffffb8f581f0>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffffb8f5b8b2>] rcu_check_callbacks+0x442/0x730
[<ffffffffb8f10190>] ? tick_sched_do_timer+0x50/0x50
[<ffffffffb8eaf666>] update_process_times+0x46/0x80
[<ffffffffb8f0ff00>] tick_sched_handle+0x30/0x70
[<ffffffffb8f101c9>] tick_sched_timer+0x39/0x80
[<ffffffffb8eca71e>] __hrtimer_run_queues+0x10e/0x270
[<ffffffffb8ecac7f>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffffb8e5c61b>] local_apic_timer_interrupt+0x3b/0x60
[<ffffffffb95929d3>] smp_apic_timer_interrupt+0x43/0x60
[<ffffffffb958eefa>] apic_timer_interrupt+0x16a/0x170
Complete report #8129
Frame # Function Binary Source or offset Line
1
copy_user_generic_unrolled
vmlinux 0xffffffff86784831
Build id: 3.10.0-957.21.3.el7.x86_64
0x41
2
sched_show_task
vmlinux 0xffffffff864d6138
Build id: 3.10.0-957.21.3.el7.x86_64
0xa8
3
dump_cpu_task
vmlinux 0xffffffff864d9e49
Build id: 3.10.0-957.21.3.el7.x86_64
0x39
4
rcu_dump_cpu_stacks
vmlinux 0xffffffff86552bb0
Build id: 3.10.0-957.21.3.el7.x86_64
0x90
5
rcu_check_callbacks
vmlinux 0xffffffff86556252
Build id: 3.10.0-957.21.3.el7.x86_64
0x442
6
? tick_sched_do_timer
vmlinux 0xffffffff8650b080
Build id: 3.10.0-957.21.3.el7.x86_64
0x50
7
update_process_times
vmlinux 0xffffffff864ab2a6
Build id: 3.10.0-957.21.3.el7.x86_64
0x46
8
tick_sched_handle
vmlinux 0xffffffff8650adf0
Build id: 3.10.0-957.21.3.el7.x86_64
0x30
9
tick_sched_timer
vmlinux 0xffffffff8650b0b9
Build id: 3.10.0-957.21.3.el7.x86_64
0x39
10
__hrtimer_run_queues
vmlinux 0xffffffff864c6103
Build id: 3.10.0-957.21.3.el7.x86_64
0xf3
11
hrtimer_interrupt
vmlinux 0xffffffff864c668f
Build id: 3.10.0-957.21.3.el7.x86_64
0xaf
12
local_apic_timer_interrupt
vmlinux 0xffffffff8645a58b
Build id: 3.10.0-957.21.3.el7.x86_64
0x3b
13
smp_apic_timer_interrupt
vmlinux 0xffffffff86b7a6e3
Build id: 3.10.0-957.21.3.el7.x86_64
0x43
14
apic_timer_interrupt
vmlinux 0xffffffff86b76df2
Build id: 3.10.0-957.21.3.el7.x86_64
0x162
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 76s! [gnome-shell:9139]
Modules linked in: xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon joydev pcspkr sg vmw_vmci parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod
cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi floppy vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi libata scsi_transport_spi vmxnet3 mptscsih drm_panel_orientation_quirks mptbase dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 9139 Comm: gnome-shell Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff91550f5b2080 ti: ffff91550f50c000 task.ti: ffff91550f50c000
RIP: 0010:[<ffffffff86784831>]  [<ffffffff86784831>] copy_user_generic_unrolled+0x41/0xc0
RSP: 0018:ffff91550f50fb00  EFLAGS: 00010206
RAX: ffff91550f50c000 RBX: ffff91550f50fac8 RCX: 0000000000000200
RDX: 0000000000000000 RSI: 000000000397cd38 RDI: ffff9154f3aa0000
RBP: ffff91550f50fb38 R08: ff000000ff000000 R09: ff000000ff000000
R10: ff000000ff000000 R11: ff000000ff000000 R12: 0000000000000003
R13: 0000000000000000 R14: 00000000000052d0 R15: ffffffff875bb4c0
FS:  00007f5404b7ea80(0000) GS:ffff9155bfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2d4cd18000 CR3: 000000008f4be000 CR4: 00000000001607e0
INFO: rcu_sched self-detected stall on CPU { 0}  (t=92005 jiffies g=767266 c=767265 q=664)
Task dump for CPU 0:
gnome-shell     R  running task        0  3755   3711 0x00000088
Call Trace:
<IRQ>  [<ffffffff864d6138>] sched_show_task+0xa8/0x110
[<ffffffff864d9e49>] dump_cpu_task+0x39/0x70
[<ffffffff86552bb0>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffff86556252>] rcu_check_callbacks+0x442/0x730
[<ffffffff8650b080>] ? tick_sched_do_timer+0x50/0x50
[<ffffffff864ab2a6>] update_process_times+0x46/0x80
[<ffffffff8650adf0>] tick_sched_handle+0x30/0x70
[<ffffffff8650b0b9>] tick_sched_timer+0x39/0x80
[<ffffffff864c6103>] __hrtimer_run_queues+0xf3/0x270
[<ffffffff864c668f>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffff8645a58b>] local_apic_timer_interrupt+0x3b/0x60
[<ffffffff86b7a6e3>] smp_apic_timer_interrupt+0x43/0x60
[<ffffffff86b76df2>] apic_timer_interrupt+0x162/0x170
Complete report #688900
Frame # Function Binary Source or offset Line
1
sched_show_task
vmlinux 0xffffffff85cda3c8
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa8
2
rcu_check_callbacks
vmlinux 0xffffffff85d5c06e
Build id: 3.10.0-1160.59.1.el7.x86_64
0x72e
3
? tick_sched_do_timer
vmlinux 0xffffffff85d10700
Build id: 3.10.0-1160.59.1.el7.x86_64
0x50
4
update_process_times
vmlinux 0xffffffff85caf176
Build id: 3.10.0-1160.59.1.el7.x86_64
0x46
5
tick_sched_handle
vmlinux 0xffffffff85d10470
Build id: 3.10.0-1160.59.1.el7.x86_64
0x30
6
tick_sched_timer
vmlinux 0xffffffff85d10739
Build id: 3.10.0-1160.59.1.el7.x86_64
0x39
7
__hrtimer_run_queues
vmlinux 0xffffffff85cca25e
Build id: 3.10.0-1160.59.1.el7.x86_64
0x10e
8
hrtimer_interrupt
vmlinux 0xffffffff85cca7bf
Build id: 3.10.0-1160.59.1.el7.x86_64
0xaf
9
local_apic_timer_interrupt
vmlinux 0xffffffff85c5ce2b
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3b
10
smp_apic_timer_interrupt
vmlinux 0xffffffff8639ea23
Build id: 3.10.0-1160.59.1.el7.x86_64
0x43
11
apic_timer_interrupt
vmlinux 0xffffffff8639afba
Build id: 3.10.0-1160.59.1.el7.x86_64
0x16a
12
? __build_skb
vmlinux 0xffffffff86247d8b
Build id: 3.10.0-1160.59.1.el7.x86_64
0xdb
13
__netdev_alloc_skb
vmlinux 0xffffffff86247fc9
Build id: 3.10.0-1160.59.1.el7.x86_64
0xc9
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [khungtaskd:50]
INFO: rcu_sched detected stalls on CPUs/tasks: {} (detected by 0, t=60119 jiffies, g=20812439, c=20812438, q=2180)
All QSes seen, last rcu_sched kthread activity 22228 (4823365383-4823343155), jiffies_till_next_fqs=3
miniserv.pl     R  running task        0  2307      1 0x00000088
Call Trace:
<IRQ>  [<ffffffff85cda3c8>] sched_show_task+0xa8/0x110
[<ffffffff85d5c06e>] rcu_check_callbacks+0x72e/0x730
[<ffffffff85d10700>] ? tick_sched_do_timer+0x50/0x50
[<ffffffff85caf176>] update_process_times+0x46/0x80
[<ffffffff85d10470>] tick_sched_handle+0x30/0x70
[<ffffffff85d10739>] tick_sched_timer+0x39/0x80
[<ffffffff85cca25e>] __hrtimer_run_queues+0x10e/0x270
[<ffffffff85cca7bf>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffff85c5ce2b>] local_apic_timer_interrupt+0x3b/0x60
[<ffffffff8639ea23>] smp_apic_timer_interrupt+0x43/0x60
[<ffffffff8639afba>] apic_timer_interrupt+0x16a/0x170
[<ffffffff86247d8b>] ? __build_skb+0xdb/0x1b0
[<ffffffff86247fc9>] __netdev_alloc_skb+0xc9/0x110
Complete report #398015
Frame # Function Binary Source or offset Line
1
sched_show_task
vmlinux 0xffffffffafcda3c8
Build id: 3.10.0-1160.53.1.el7.x86_64
0xa8
2
dump_cpu_task
vmlinux 0xffffffffafcde039
Build id: 3.10.0-1160.53.1.el7.x86_64
0x39
3
rcu_dump_cpu_stacks
vmlinux 0xffffffffafd586c0
Build id: 3.10.0-1160.53.1.el7.x86_64
0x90
4
rcu_check_callbacks
vmlinux 0xffffffffafd5bd82
Build id: 3.10.0-1160.53.1.el7.x86_64
0x442
5
? tick_sched_do_timer
vmlinux 0xffffffffafd10700
Build id: 3.10.0-1160.53.1.el7.x86_64
0x50
6
update_process_times
vmlinux 0xffffffffafcaf176
Build id: 3.10.0-1160.53.1.el7.x86_64
0x46
7
tick_sched_handle
vmlinux 0xffffffffafd10470
Build id: 3.10.0-1160.53.1.el7.x86_64
0x30
8
tick_sched_timer
vmlinux 0xffffffffafd10739
Build id: 3.10.0-1160.53.1.el7.x86_64
0x39
9
__hrtimer_run_queues
vmlinux 0xffffffffafcca25e
Build id: 3.10.0-1160.53.1.el7.x86_64
0x10e
10
hrtimer_interrupt
vmlinux 0xffffffffafcca7bf
Build id: 3.10.0-1160.53.1.el7.x86_64
0xaf
11
local_apic_timer_interrupt
vmlinux 0xffffffffafc5ce2b
Build id: 3.10.0-1160.53.1.el7.x86_64
0x3b
12
smp_apic_timer_interrupt
vmlinux 0xffffffffb039aa23
Build id: 3.10.0-1160.53.1.el7.x86_64
0x43
13
apic_timer_interrupt
vmlinux 0xffffffffb0396fba
Build id: 3.10.0-1160.53.1.el7.x86_64
0x16a
14
? __build_skb
vmlinux 0xffffffffb0244d8b
Build id: 3.10.0-1160.53.1.el7.x86_64
0xdb
15
__netdev_alloc_skb
vmlinux 0xffffffffb0244fc9
Build id: 3.10.0-1160.53.1.el7.x86_64
0xc9
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 196s! [node /ql/build/:28953]
[<ffffffffafcda3c8>] sched_show_task+0xa8/0x110
[<ffffffffafcde039>] dump_cpu_task+0x39/0x70
[<ffffffffafd586c0>] rcu_dump_cpu_stacks+0x90/0xd0
[<ffffffffafd5bd82>] rcu_check_callbacks+0x442/0x730
[<ffffffffafd10700>] ? tick_sched_do_timer+0x50/0x50
[<ffffffffafcaf176>] update_process_times+0x46/0x80
[<ffffffffafd10470>] tick_sched_handle+0x30/0x70
[<ffffffffafd10739>] tick_sched_timer+0x39/0x80
[<ffffffffafcca25e>] __hrtimer_run_queues+0x10e/0x270
[<ffffffffafcca7bf>] hrtimer_interrupt+0xaf/0x1d0
[<ffffffffafc5ce2b>] local_apic_timer_interrupt+0x3b/0x60
[<ffffffffb039aa23>] smp_apic_timer_interrupt+0x43/0x60
[<ffffffffb0396fba>] apic_timer_interrupt+0x16a/0x170
[<ffffffffb0244d8b>] ? __build_skb+0xdb/0x1b0
[<ffffffffb0244fc9>] __netdev_alloc_skb+0xc9/0x110