Complete report #1407887
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
finish_task_switch
|
vmlinux |
0xffffffff990da8a7
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x57
|
|
2
|
schedule_tail
|
vmlinux |
0xffffffff990de794
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x24
|
|
3
|
ret_from_fork
|
vmlinux |
0xffffffff997c518f
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xf
|
|
4
|
|
vmlinux |
0xffffffff990cb550
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 28s! [kthreadd:13442]
Modules linked in: binfmt_misc xt_CHECKSUM hids_driver(OE) tun devlink xt_set ip_set_hash_net ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack isofs ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sb_edac iosf_mbi kvm_intel kvm sg irqbypass joydev pcspkr virtio_balloon i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom 8021q garp mrp stp llc virtio_net net_failover virtio_scsi virtio_blk virtio_console failover ata_generic pata_acpi cirrus drm_kms_helper
syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common crc32_pclmul crc32c_intel ttm ata_piix ghash_clmulni_intel drm libata aesni_intel lrw gf128mul glue_helper ablk_helper scsi_transport_iscsi virtio_pci cryptd virtio_ring serio_raw floppy virtio drm_panel_orientation_quirks sunrpc dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 13442 Comm: kthreadd Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.0-1.module_el8.7.0+1140+ff0772f9 04/01/2014
task: ffff96582c329080 ti: ffff9655db7d4000 task.ti: ffff9655db7d4000
RIP: 0010:[<ffffffff990da8a7>] [<ffffffff990da8a7>] finish_task_switch+0x57/0x1c0
RSP: 0000:ffff9655db7d7f10 EFLAGS: 00000246
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
RDX: ffff9655db7d5fd8 RSI: ffff96582c329080 RDI: ffff96582f29acc0
RBP: ffff9655db7d7f30 R08: ffff9655db7d4000 R09: 0000000000000001
R10: 000000003b9aca00 R11: 00000000000001a6 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff96582f280000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000a3aed9c CR3: 0000000151410000 CR4: 0000000000362fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff990de794>] schedule_tail+0x24/0x70
[<ffffffff997c518f>] ret_from_fork+0xf/0x56
[<ffffffff990cb550>] ? insert_kthread_work+0x40/0x40
Code: 1f 44 00 00 65 48 8b 34 25 c0 0e 01 00 0f 1f 44 00 00 41 c7 45 28 00 00 00 00 48 89 df e8 6a 3d 04 00 66 90 fb 66 0f 1f 44 00 00 <65> 48 8b 04 25 c0 0e 01 00 48 8b 98 78 01 00 00 48 85 db 74 22
Complete report #886802
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
finish_task_switch
|
vmlinux |
0xffffffffbbed09f4
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x54
|
|
2
|
schedule_tail
|
vmlinux |
0xffffffffbbed46d6
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x26
|
|
3
|
ret_from_fork
|
vmlinux |
0xffffffffbc575bcf
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xf
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 40s! [run.sh:9231]
Modules linked in: arc4 md4 rpcsec_gss_krb5 nfsv4 nfs fscache xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter nls_utf8 cifs ccm dns_resolver vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 ppdev vmw_balloon coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr parport_pc parport sg vmw_vmci i2c_piix4 nfsd nfs_acl lockd grace auth_rpcgss binfmt_misc sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm vmxnet3 libata mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks involflt(POE) dm_mirror dm_region_hash dm_log dm_mod [last unloaded: floppy]
CPU: 2 PID: 9231 Comm: run.sh Kdump: loaded Tainted: P OEL ------------ 3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff9dd9a2239040 ti: ffff9dd97021c000 task.ti: ffff9dd97021c000
RIP: 0010:[<ffffffffbbed09f4>] [<ffffffffbbed09f4>] finish_task_switch+0x54/0x1c0
RSP: 0000:ffff9dd97021ff08 EFLAGS: 00000282
RAX: ffff9dd20b910000 RBX: ffff9dda1ae86984 RCX: 00000000c0000100
RDX: ffff9dd97021dfd8 RSI: ffff9dd9a2239040 RDI: ffff9dda3fd1ab80
RBP: ffff9dd97021ff28 R08: ffff9dd97021c000 R09: 0000000000000001
R10: 000000000000b570 R11: 0000000000000001 R12: ffff9dd97021ff08
R13: ffffffffbc568ed9 R14: ffff9dd97021fe70 R15: ffff9dda1ae86180
FS: 00007f06e4c36740(0000) GS:ffff9dda3fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f748328a9e5 CR3: 00000007a3bce000 CR4: 00000000000007e0
Call Trace:
[<ffffffffbbed46d6>] schedule_tail+0x26/0xa0
[<ffffffffbc575bcf>] ret_from_fork+0xf/0x56
Code: 8b 36 66 66 66 66 90 65 48 8b 34 25 80 0e 01 00 66 66 66 66 90 41 c7 45 28 00 00 00 00 48 89 df c6 07 00 66 66 66 90 fb 66 66 90 <66> 66 90 65 48 8b 04 25 80 0e 01 00 48 8b 98 78 01 00 00 48 85
Complete report #214127
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
finish_task_switch
|
vmlinux |
0xffffffffbd8d5407
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x57
|
|
2
|
schedule_tail
|
vmlinux |
0xffffffffbd8d90b4
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x24
|
|
3
|
ret_from_fork
|
vmlinux |
0xffffffffbdf92ccf
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xf
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ondot-sql-cmd:3624]
Modules linked in: vsock_diag tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink ipmi_devintf ipmi_msghandler vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev vmw_balloon pcspkr sg parport_pc parport vmw_vmci i2c_piix4 binfmt_misc ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx sd_mod crc_t10dif drm_kms_helper crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci
nfit drm libnvdimm crct10dif_pclmul crct10dif_common ata_piix e1000e crc32c_intel serio_raw libata ptp pps_core vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3624 Comm: ondot-sql-cmd Kdump: loaded Tainted: G W L ------------ 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
task: ffff9ecfd981b150 ti: ffff9ece0e5c8000 task.ti: ffff9ece0e5c8000
RIP: 0010:[<ffffffffbd8d5407>] [<ffffffffbd8d5407>] finish_task_switch+0x57/0x1c0
RSP: 0018:ffff9ece0e5cbf10 EFLAGS: 00000286
RAX: ffff9ecd881a5230 RBX: ffff9ecd15814688 RCX: 00000000c0000100
RDX: ffff9ece0e5c9fd8 RSI: ffff9ecfd981b150 RDI: ffff9ed13fc1acc0
RBP: ffff9ece0e5cbf30 R08: ffff9ece0e5c8000 R09: 0000000000000001
R10: ffff9ecd0eeb90e8 R11: 0000000000000246 R12: ffff9ecd158149c4
R13: ffffffffbd8a27d1 R14: ffff9ece0e5cbf08 R15: ffffffffbdf85da9
FS: 00007f2654564740(0000) GS:ffff9ed13fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f21eb023010 CR3: 000000012cd14000 CR4: 00000000003607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffbd8d90b4>] schedule_tail+0x24/0x70
[<ffffffffbdf92ccf>] ret_from_fork+0xf/0x56
Code: 1f 44 00 00 65 48 8b 34 25 c0 0e 01 00 0f 1f 44 00 00 41 c7 45 28 00 00 00 00 48 89 df c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <65> 48 8b 04 25 c0 0e 01 00 48 8b 98 78 01 00 00 48 85 db 74 22
Complete report #170015
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
finish_task_switch
|
vmlinux |
0xffffffffbaad5407
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x57
|
|
2
|
schedule_tail
|
vmlinux |
0xffffffffbaad90b4
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x24
|
|
3
|
ret_from_fork
|
vmlinux |
0xffffffffbb192ccf
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xf
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [ondot-sql-cmd:3270]
Modules linked in: vsock_diag tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter ipmi_devintf ipmi_msghandler vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd pcspkr joydev vmw_balloon sg parport_pc parport i2c_piix4 vmw_vmci binfmt_misc ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix ahci
drm libahci crct10dif_pclmul crct10dif_common e1000e nfit crc32c_intel libnvdimm libata serio_raw ptp pps_core vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 3270 Comm: ondot-sql-cmd Kdump: loaded Tainted: G L ------------ 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8e75e3a6d230 ti: ffff8e7527d04000 task.ti: ffff8e7527d04000
RIP: 0010:[<ffffffffbaad5407>] [<ffffffffbaad5407>] finish_task_switch+0x57/0x1c0
RSP: 0018:ffff8e7527d07f10 EFLAGS: 00000286
RAX: ffff8e78ee2a3150 RBX: ffff8e7558a79538 RCX: 00000000c0000100
RDX: ffff8e7527d05fd8 RSI: ffff8e75e3a6d230 RDI: ffff8e78ffd1acc0
RBP: ffff8e7527d07f30 R08: ffff8e7527d04000 R09: 0000000000000001
R10: ffff8e78350eb1c8 R11: 0000000000000246 R12: ffff8e7558a79874
R13: ffffffffbaaa27d1 R14: ffff8e7527d07f08 R15: ffffffffbb185da9
FS: 00007f3af358e740(0000) GS:ffff8e78ffd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc19ac02000 CR3: 000000005ad5a000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffbaad90b4>] schedule_tail+0x24/0x70
[<ffffffffbb192ccf>] ret_from_fork+0xf/0x56
Code: 1f 44 00 00 65 48 8b 34 25 c0 0e 01 00 0f 1f 44 00 00 41 c7 45 28 00 00 00 00 48 89 df c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <65> 48 8b 04 25 c0 0e 01 00 48 8b 98 78 01 00 00 48 85 db 74 22