Info

Permalink
Function
__schedule
First occurrence
2020-10-18
Last occurrence
2026-08-19
State
NEW
Components
kernel
Quality
-15

Statistics

Operating system Count
CentOS 7 441
Architecture Count
x86_64 441
Related packages Count
kernel 441
 0:3.10.0-1160.25.1.el7 379
 0:3.10.0-1062.4.1.el7 8
 0:3.10.0-1062.9.1.el7 6
 0:3.10.0-1062.12.1.el7 5
 0:3.10.0-1160.119.1.el7 5
 0:3.10.0-1160.108.1.el7 4
 0:3.10.0-1062.4.3.el7 2
 0:3.10.0-1127.el7 2
 0:3.10.0-862.14.4.el7 2
 0:3.10.0-957.1.3.el7 2
 0:3.10.0-957.el7 2
 0:3.10.0-1160.114.2.el7 2
 0:3.10.0-1160.88.1.el7 2
 0:3.10.0-1127.13.1.el7 1
 0:3.10.0-1127.18.2.el7 1
 0:3.10.0-1160.11.1.el7 1
 0:3.10.0-1160.31.1.el7 1
 0:3.10.0-1160.36.2.el7 1
 0:3.10.0-1160.45.1.el7 1
 0:3.10.0-1160.49.1.el7 1
 0:3.10.0-1160.53.1.el7 1
 0:3.10.0-1160.59.1.el7 1
 0:3.10.0-1160.6.1.el7 1
 0:3.10.0-1160.80.1.el7 1
 0:3.10.0-1160.81.1.el7 1
 0:3.10.0-1160.el7 1
 0:3.10.0-862.el7 1
 0:3.10.0-1160.102.1.el7 1
 0:3.10.0-1160.76.1.el7 1
 0:3.10.0-862.3.3.el7 1
 0:3.10.0-1160.71.1.el7 1
 0:3.10.0-1160.62.1.el7 1
 0:3.10.0-1160.99.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1028542
Frame # Function Binary Source or offset Line
1
? __schedule
vmlinux 0xffffffffb29b7b85
Build id: 3.10.0-1160.119.1.el7.x86_64
0x585
2
schedule
vmlinux 0xffffffffb29b7ca9
Build id: 3.10.0-1160.119.1.el7.x86_64
0x29
3
smpboot_thread_fn
vmlinux 0xffffffffb22d4392
Build id: 3.10.0-1160.119.1.el7.x86_64
0xe2
4
? lg_double_unlock
vmlinux 0xffffffffb22d42b0
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
5
kthread
vmlinux 0xffffffffb22cb621
Build id: 3.10.0-1160.119.1.el7.x86_64
0xd1
6
? insert_kthread_work
vmlinux 0xffffffffb22cb550
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
7
ret_from_fork_nospec_begin
vmlinux 0xffffffffb29c51f7
Build id: 3.10.0-1160.119.1.el7.x86_64
0x21
8
? insert_kthread_work
vmlinux 0xffffffffb22cb550
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 51s! [PassengerAgent:3110]
INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 0, t=69357 jiffies, g=1377520, c=1377519, q=71)
Task dump for CPU 2:
watchdog/2      S ffff8fcadb5947e8     0    17      2 0x00000000
Call Trace:
[<ffffffffb29b7b85>] ? __schedule+0x585/0x680
[<ffffffffb29b7ca9>] schedule+0x29/0x70
[<ffffffffb22d4392>] smpboot_thread_fn+0xe2/0x1b0
[<ffffffffb22d42b0>] ? lg_double_unlock+0x40/0x40
[<ffffffffb22cb621>] kthread+0xd1/0xe0
[<ffffffffb22cb550>] ? insert_kthread_work+0x40/0x40
[<ffffffffb29c51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb22cb550>] ? insert_kthread_work+0x40/0x40
Complete report #458055
Frame # Function Binary Source or offset Line
1
? __schedule
vmlinux 0xffffffff9b58c050
Build id: 3.10.0-1160.62.1.el7.x86_64
0x320
2
smpboot_thread_fn
vmlinux 0xffffffff9aecea04
Build id: 3.10.0-1160.62.1.el7.x86_64
0x144
3
? lg_double_unlock
vmlinux 0xffffffff9aece8c0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x40
4
kthread
vmlinux 0xffffffff9aec5fb1
Build id: 3.10.0-1160.62.1.el7.x86_64
0xd1
5
? insert_kthread_work
vmlinux 0xffffffff9aec5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x40
6
ret_from_fork_nospec_begin
vmlinux 0xffffffff9b599df7
Build id: 3.10.0-1160.62.1.el7.x86_64
0x21
7
? insert_kthread_work
vmlinux 0xffffffff9aec5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 32s! [mysqld:1773]
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [php:6240]
[<ffffffff9b58c050>] ? __schedule+0x320/0x680
[<ffffffff9aecea04>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff9aece8c0>] ? lg_double_unlock+0x40/0x40
[<ffffffff9aec5fb1>] kthread+0xd1/0xe0
[<ffffffff9aec5ee0>] ? insert_kthread_work+0x40/0x40
[<ffffffff9b599df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9aec5ee0>] ? insert_kthread_work+0x40/0x40
Complete report #361409
Frame # Function Binary Source or offset Line
1
smpboot_thread_fn
vmlinux 0xffffffffb7ecdb20
Build id: 3.10.0-1062.4.1.el7.x86_64
0x80
2
? lg_double_unlock
vmlinux 0xffffffffb7ecdaa0
Build id: 3.10.0-1062.4.1.el7.x86_64
0x40
3
kthread
vmlinux 0xffffffffb7ec51b1
Build id: 3.10.0-1062.4.1.el7.x86_64
0xd1
4
? insert_kthread_work
vmlinux 0xffffffffb7ec50e0
Build id: 3.10.0-1062.4.1.el7.x86_64
0x40
5
ret_from_fork_nospec_begin
vmlinux 0xffffffffb858bd37
Build id: 3.10.0-1062.4.1.el7.x86_64
0x21
6
? insert_kthread_work
vmlinux 0xffffffffb7ec50e0
Build id: 3.10.0-1062.4.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 60s! [watchdog/1:12]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13005(E) nls_utf8 isofs fuse rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache devlink binfmt_misc falcon_lsm_pinned_12905(E) ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_hda_codec_generic lrw gf128mul
glue_helper ablk_helper cryptd snd_hda_intel joydev snd_hda_codec virtio_balloon snd_hda_core sg snd_hwdep snd_seq parport_pc snd_seq_device parport snd_pcm snd_timer snd pcspkr i2c_piix4 soundcore auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw crct10dif_pclmul crct10dif_common virtio_pci crc32c_intel floppy virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_kal]
CPU: 1 PID: 12 Comm: watchdog/1 Kdump: loaded Tainted: P            EL ------------   3.10.0-1062.4.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffffa0a8f6cde2a0 ti: ffffa0a8f6d08000 task.ti: ffffa0a8f6d08000
RIP: 0010:[<ffffffffb7ecdb20>]  [<ffffffffb7ecdb20>] smpboot_thread_fn+0x80/0x1a0
RSP: 0018:ffffa0a8f6d0bea0  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffffa0a8f6cde2a0 RCX: 0000000000000000
RDX: ffffa0a8f6d09fd8 RSI: ffffa0a8f6cde2a0 RDI: ffffa0a8f6cde2a0
RBP: ffffa0a8f6d0bec0 R08: ffffa0a8f6d08000 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: ffffa0a93fc01b60
R13: ffffffffb8a63a60 R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffa0d71f040000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000a292b001 CR3: 0000002f4362a000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb7ecdaa0>] ? lg_double_unlock+0x40/0x40
[<ffffffffb7ec51b1>] kthread+0xd1/0xe0
[<ffffffffb7ec50e0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb858bd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb7ec50e0>] ? insert_kthread_work+0x40/0x40
Code: 48 c7 03 00 00 00 00 49 8b 45 40 48 85 c0 74 0c 41 83 7c 24 04 01 0f 84 97 00 00 00 e8 ba 73 ff ff eb b0 0f 1f 84 00 00 00 00 00 <41> 8b 3c 24 65 8b 05 f1 04 14 48 39 c7 0f 85 ff 00 00 00 41 8b 
Complete report #48334
Frame # Function Binary Source or offset Line
1
watchdog_should_run
vmlinux 0xffffffff8214e325
Build id: 3.10.0-1062.12.1.el7.x86_64
0x5
2
? smpboot_thread_fn
vmlinux 0xffffffff820cebb9
Build id: 3.10.0-1062.12.1.el7.x86_64
0xd9
3
? lg_double_unlock
vmlinux 0xffffffff820ceae0
Build id: 3.10.0-1062.12.1.el7.x86_64
0x40
4
kthread
vmlinux 0xffffffff820c61f1
Build id: 3.10.0-1062.12.1.el7.x86_64
0xd1
5
? insert_kthread_work
vmlinux 0xffffffff820c6120
Build id: 3.10.0-1062.12.1.el7.x86_64
0x40
6
ret_from_fork_nospec_begin
vmlinux 0xffffffff8278dd37
Build id: 3.10.0-1062.12.1.el7.x86_64
0x21
7
? insert_kthread_work
vmlinux 0xffffffff820c6120
Build id: 3.10.0-1062.12.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [watchdog/0:11]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables br_netfilter overlay(T) tun devlink bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev vmw_balloon glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc vmw_vmci parport i2c_piix4 binfmt_misc xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod vmwgfx crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci nfit crct10dif_pclmul crct10dif_common libnvdimm crc32c_intel ata_piix serio_raw libata vmxnet3 vmw_pvscsi
drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 0 PID: 11 Comm: watchdog/0 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff8be4e9523150 ti: ffff8be4e9534000 task.ti: ffff8be4e9534000
RIP: 0010:[<ffffffff8214e325>]  [<ffffffff8214e325>] watchdog_should_run+0x5/0x30
RSP: 0000:ffff8be4e9537e98  EFLAGS: 00000297
RAX: ffffffff8214e320 RBX: ffff8be4e9537fd8 RCX: 0000000000000000
RDX: ffff8be4e9535fd8 RSI: ffff8be4e9523150 RDI: 0000000000000000
RBP: ffff8be4e9537ec0 R08: ffff8be4e9534000 R09: 0000000000000001
R10: 000000000000001e R11: 00000000000001d0 R12: ffffffff827805c2
R13: ffff8be4e9537e80 R14: 0000000000000001 R15: ffff8bee37fc0000
FS:  0000000000000000(0000) GS:ffff8bf3bfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000232823c64000 CR3: 0000000acab68000 CR4: 00000000000607f0
Call Trace:
[<ffffffff820cebb9>] ? smpboot_thread_fn+0xd9/0x1a0
[<ffffffff820ceae0>] ? lg_double_unlock+0x40/0x40
[<ffffffff820c61f1>] kthread+0xd1/0xe0
[<ffffffff820c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffff8278dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff820c6120>] ? insert_kthread_work+0x40/0x40
Code: 60 01 00 48 89 e5 65 48 03 05 98 2e ec 7d c6 00 01 65 48 c7 05 71 7e ec 7d 00 00 00 00 5d c3 0f 1f 80 00 00 00 00 66 66 66 66 90 <55> 65 48 8b 05 c2 7d ec 7d 65 48 8b 15 c2 7d ec 7d 48 39 c2 48 
Complete report #21282
Frame # Function Binary Source or offset Line
1
schedule
vmlinux 0xffffffffae780a09
Build id: 3.10.0-1062.9.1.el7.x86_64
0x29
2
smpboot_thread_fn
vmlinux 0xffffffffae0cebc2
Build id: 3.10.0-1062.9.1.el7.x86_64
0xe2
3
? lg_double_unlock
vmlinux 0xffffffffae0ceae0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
4
kthread
vmlinux 0xffffffffae0c61f1
Build id: 3.10.0-1062.9.1.el7.x86_64
0xd1
5
? insert_kthread_work
vmlinux 0xffffffffae0c6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
6
ret_from_fork_nospec_begin
vmlinux 0xffffffffae78dd37
Build id: 3.10.0-1062.9.1.el7.x86_64
0x21
7
? insert_kthread_work
vmlinux 0xffffffffae0c6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [java:25355]
[<ffffffffae780a09>] schedule+0x29/0x70
[<ffffffffae0cebc2>] smpboot_thread_fn+0xe2/0x1a0
[<ffffffffae0ceae0>] ? lg_double_unlock+0x40/0x40
[<ffffffffae0c61f1>] kthread+0xd1/0xe0
[<ffffffffae0c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffffae78dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffae0c6120>] ? insert_kthread_work+0x40/0x40
Complete report #9258
Frame # Function Binary Source or offset Line
1
smpboot_thread_fn
vmlinux 0xffffffffb10cebc2
Build id: 3.10.0-1062.4.3.el7.x86_64
0xe2
2
? lg_double_unlock
vmlinux 0xffffffffb10ceae0
Build id: 3.10.0-1062.4.3.el7.x86_64
0x40
3
kthread
vmlinux 0xffffffffb10c61f1
Build id: 3.10.0-1062.4.3.el7.x86_64
0xd1
4
? insert_kthread_work
vmlinux 0xffffffffb10c6120
Build id: 3.10.0-1062.4.3.el7.x86_64
0x40
5
ret_from_fork_nospec_begin
vmlinux 0xffffffffb178cd37
Build id: 3.10.0-1062.4.3.el7.x86_64
0x21
6
? insert_kthread_work
vmlinux 0xffffffffb10c6120
Build id: 3.10.0-1062.4.3.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ksoftirqd/0:6]
Modules linked in: veth xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc cmac arc4 md4 nls_utf8 cifs ccm dns_resolver overlay(T) vmw_vsock_vmci_transport vsock sb_edac coretemp ppdev iosf_mbi crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr joydev vmw_vmci parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea ata_piix sysfillrect sysimgblt fb_sys_fops ttm mptspi libata crct10dif_pclmul crct10dif_common drm scsi_transport_spi crc32c_intel mptscsih serio_raw
mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.4.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff93a7fbaad230 ti: ffff93a7fbb54000 task.ti: ffff93a7fbb54000
RIP: 0010:[<ffffffffb10cebc2>]  [<ffffffffb10cebc2>] smpboot_thread_fn+0xe2/0x1a0
RSP: 0018:ffff93a7fbb57ea0  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff93a7fbb57fd8 RCX: 0000000000000000
RDX: ffff93a7fbb55fd8 RSI: ffff93a7fbaad230 RDI: ffff93a7fbaad230
RBP: ffff93a7fbb57ec0 R08: ffff93a7fbb54000 R09: 0000000000000001
R10: 000000000000b4f3 R11: 0000000000000001 R12: ffff93a7fbb57fd8
R13: ffff93a7fbaad230 R14: ffffffffb177f568 R15: ffff93a7fbb57e80
FS:  0000000000000000(0000) GS:ffff93a93fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f95fe046d40 CR3: 00000002a9248000 CR4: 00000000001607f0
Call Trace:
[<ffffffffb10ceae0>] ? lg_double_unlock+0x40/0x40
[<ffffffffb10c61f1>] kthread+0xd1/0xe0
[<ffffffffb10c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffffb178cd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb10c6120>] ? insert_kthread_work+0x40/0x40
Code: 36 2c 00 41 c7 44 24 04 01 00 00 00 e9 60 ff ff ff 0f 1f 84 00 00 00 00 00 49 8b 45 18 e8 47 36 2c 00 85 c0 75 53 e8 1e 0f 6b 00 <e9> 41 ff ff ff 66 0f 1f 84 00 00 00 00 00 48 c7 03 00 00 00 00