Info

Permalink
Function
_raw_spin_unlock_irqrestore
First occurrence
2020-09-24
Last occurrence
2026-03-28
State
NEW
Components
kernel
Quality
-23

Statistics

Operating system Count
CentOS 7 480
Architecture Count
x86_64 480
Related packages Count
kernel 480
 0:3.10.0-957.el7 68
 0:3.10.0-1160.el7 56
 0:3.10.0-693.2.2.el7 40
 0:3.10.0-693.el7 25
 0:3.10.0-1160.15.2.el7 24
 0:3.10.0-1160.41.1.el7 22
 0:3.10.0-1160.53.1.el7 17
 0:3.10.0-1127.18.2.el7 12
 0:3.10.0-1127.19.1.el7 12
 0:3.10.0-1160.102.1.el7 12
 0:3.10.0-862.el7 12
 0:3.10.0-862.9.1.el7 11
 0:3.10.0-693.5.2.el7 11
 0:3.10.0-1160.76.1.el7 10
 0:3.10.0-1062.el7 9
 0:3.10.0-1127.el7 9
 0:3.10.0-1160.6.1.el7 8
 0:3.10.0-1127.10.1.el7 7
 0:3.10.0-1160.62.1.el7 7
 0:3.10.0-1160.80.1.el7 7
 0:3.10.0-1062.12.1.el7 6
 0:3.10.0-1062.9.1.el7 6
 0:3.10.0-1160.119.1.el7 6
 0:3.10.0-1160.92.1.el7 6
 0:3.10.0-1160.118.1.el7 5
 0:3.10.0-1160.31.1.el7 5
 0:3.10.0-1160.42.2.el7 5
 0:3.10.0-1160.71.1.el7 5
 0:3.10.0-693.21.1.el7 5
 0:3.10.0-1062.4.1.el7 4
 0:3.10.0-1160.88.1.el7 4
 0:3.10.0-1160.90.1.el7 4
 0:3.10.0-862.3.3.el7 4
 0:3.10.0-1160.105.1.el7 3
 0:3.10.0-1160.114.2.el7 3
 0:3.10.0-1160.81.1.el7 3
 0:3.10.0-1160.83.1.el7 3
 0:3.10.0-957.21.3.el7 3
 0:3.10.0-1062.18.1.el7 2
 0:3.10.0-1160.11.1.el7 2
 0:3.10.0-1160.25.1.el7 2
 0:3.10.0-1160.49.1.el7 2
 0:3.10.0-957.12.1.el7 2
 0:3.10.0-1160.24.1.el7 2
 0:3.10.0-1062.4.3.el7 1
 0:3.10.0-1160.21.1.el7 1
 0:3.10.0-1160.45.1.el7 1
 0:3.10.0-1160.59.1.el7 1
 0:3.10.0-957.27.2.el7 1
 0:3.10.0-693.17.1.el7 1
 0:3.10.0-1062.1.2.el7 1
 0:3.10.0-1160.36.2.el7 1
 0:3.10.0-957.1.3.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1389718
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff816ab845
Build id: 3.10.0-693.el7.x86_64 		0x15
2
ata_bmdma_error_handler
libata 0xffffffffc006e9e0
Build id: 3.10.0-693.el7.x86_64 		0x50
3
ata_scsi_port_error_handler
libata 0xffffffffc006b4e9
Build id: 3.10.0-693.el7.x86_64 		0x529
4
ata_scsi_error
libata 0xffffffffc006b940
Build id: 3.10.0-693.el7.x86_64 		0xa0
5
scsi_error_handler
vmlinux 0xffffffff8146a843
Build id: 3.10.0-693.el7.x86_64 		0xf3
6
? scsi_eh_get_sense
vmlinux 0xffffffff8146a750
Build id: 3.10.0-693.el7.x86_64 		0x250
7
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64 		0xcf
8
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64 		0x40
9
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64 		0x58
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 59s! [scsi_eh_0:282]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nls_utf8 isofs loop vmw_vsock_vmci_transport vsock edac_core coretemp iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd vmw_balloon e1000 pcspkr joydev sg parport_pc parport shpchp i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul
crct10dif_common crc32c_intel serio_raw mptspi scsi_transport_spi mptscsih mptbase i2c_core ata_piix libata dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 282 Comm: scsi_eh_0 Not tainted 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff880132634f10 ti: ffff880131c9c000 task.ti: ffff880131c9c000
RIP: 0010:[<ffffffff816ab845>]  [<ffffffff816ab845>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff880131c9fd78  EFLAGS: 00000202
RAX: 0000000000000050 RBX: ffff880131c9fd08 RCX: 0000000000000100
RDX: 00000000000003f6 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff880131c9fd78 R08: 7000000000000000 R09: 05c0000000000000
R10: dffac0dc2ee940b8 R11: 0000000000000001 R12: ffff880131c9fd30
R13: 0000000000000001 R14: ffffffff810a98bb R15: ffff880131c9fd40
FS:  0000000000000000(0000) GS:ffff880139600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f61cc5260f0 CR3: 00000000b12af000 CR4: 00000000003407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff880131c9fda0 ffffffffc006e9e0 ffff880131c94000 0000000000000001
0000000000000000 ffff880131c9fde8 ffffffffc006b4e9 ffff880131c97610
ffff880131c95e80 ffff880131c1a800 ffff880131c9fdf8 ffff880131c94000
Call Trace:
[<ffffffffc006e9e0>] ata_bmdma_error_handler+0x50/0x110 [libata]
[<ffffffffc006b4e9>] ata_scsi_port_error_handler+0x529/0x8e0 [libata]
[<ffffffffc006b940>] ata_scsi_error+0xa0/0xe0 [libata]
[<ffffffff8146a843>] scsi_error_handler+0xf3/0x8b0
[<ffffffff8146a750>] ? scsi_eh_get_sense+0x250/0x250
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #1319
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffae583815
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x15
2
ata_sff_error_handler
libata 0xffffffffc019f836
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x86
3
ata_bmdma_error_handler
libata 0xffffffffc019fe48
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x58
4
ata_scsi_port_error_handler
libata 0xffffffffc019c8c2
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x532
5
ata_scsi_error
libata 0xffffffffc019cd30
Build id: 3.10.0-1062.18.1.el7.x86_64 		0xa0
6
scsi_error_handler
vmlinux 0xffffffffae2e4396
Build id: 3.10.0-1062.18.1.el7.x86_64 		0xf6
7
? scsi_eh_get_sense
vmlinux 0xffffffffae2e42a0
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x250
8
kthread
vmlinux 0xffffffffadec6321
Build id: 3.10.0-1062.18.1.el7.x86_64 		0xd1
9
? insert_kthread_work
vmlinux 0xffffffffadec6250
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffffae58dd37
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x21
11
? insert_kthread_work
vmlinux 0xffffffffadec6250
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [scsi_eh_2:435]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev vmw_balloon sb_edac iosf_mbi snd_ens1371 crc32_pclmul snd_rawmidi snd_ac97_codec ghash_clmulni_intel ac97_bus snd_seq snd_seq_device aesni_intel lrw gf128mul snd_pcm glue_helper ablk_helper cryptd nfit libnvdimm pcspkr joydev snd_timer sg snd soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper ata_piix
syscopyarea sysfillrect libata sysimgblt fb_sys_fops ttm drm e1000 crct10dif_pclmul crct10dif_common mptspi crc32c_intel scsi_transport_spi mptscsih mptbase serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 435 Comm: scsi_eh_2 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9f68f55d8000 ti: ffff9f68f54ec000 task.ti: ffff9f68f54ec000
RIP: 0010:[<ffffffffae583815>]  [<ffffffffae583815>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9f68f54efd40  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffc01ae2a8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff9f68f54efd40 R08: 0000000000000051 R09: 0000000000000051
R10: 000000000000091f R11: 44207261656c6320 R12: ffff9f68f54efd20
R13: 0000000000000001 R14: ffff9f68f54ec000 R15: ffff9f68f54efcc0
FS:  0000000000000000(0000) GS:ffff9f6af6680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f878e7d1d78 CR3: 00000001b5634000 CR4: 00000000001607e0
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [systemd-journal:666]
Call Trace:
[<ffffffffc019f836>] ata_sff_error_handler+0x86/0xf0 [libata]
[<ffffffffc019fe48>] ata_bmdma_error_handler+0x58/0x130 [libata]
[<ffffffffc019c8c2>] ata_scsi_port_error_handler+0x532/0x900 [libata]
[<ffffffffc019cd30>] ata_scsi_error+0xa0/0xe0 [libata]
[<ffffffffae2e4396>] scsi_error_handler+0xf6/0x8b0
[<ffffffffae2e42a0>] ? scsi_eh_get_sense+0x250/0x250
[<ffffffffadec6321>] kthread+0xd1/0xe0
[<ffffffffadec6250>] ? insert_kthread_work+0x40/0x40
[<ffffffffae58dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffadec6250>] ? insert_kthread_work+0x40/0x40
Complete report #848600
Frame # Function Binary Source or offset Line
1
ata_sff_error_handler
libata 0xffffffffc04a77e5
Build id: 3.10.0-1160.el7.x86_64 		0xb5
2
ata_bmdma_error_handler
libata 0xffffffffc04a7dc8
Build id: 3.10.0-1160.el7.x86_64 		0x58
3
ata_scsi_port_error_handler
libata 0xffffffffc04a4842
Build id: 3.10.0-1160.el7.x86_64 		0x532
4
ata_scsi_error
libata 0xffffffffc04a4cb0
Build id: 3.10.0-1160.el7.x86_64 		0xa0
5
scsi_error_handler
vmlinux 0xffffffffbdae9036
Build id: 3.10.0-1160.el7.x86_64 		0xf6
6
? scsi_eh_get_sense
vmlinux 0xffffffffbdae8f40
Build id: 3.10.0-1160.el7.x86_64 		0x250
7
kthread
vmlinux 0xffffffffbd6c5c21
Build id: 3.10.0-1160.el7.x86_64 		0xd1
8
? insert_kthread_work
vmlinux 0xffffffffbd6c5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40
9
ret_from_fork_nospec_begin
vmlinux 0xffffffffbdd93ddd
Build id: 3.10.0-1160.el7.x86_64 		0x7
10
? insert_kthread_work
vmlinux 0xffffffffbd6c5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 78s! [irqbalance:877]
[<ffffffffc04a77e5>] ata_sff_error_handler+0xb5/0xf0 [libata]
[<ffffffffc04a7dc8>] ata_bmdma_error_handler+0x58/0x130 [libata]
[<ffffffffc04a4842>] ata_scsi_port_error_handler+0x532/0x900 [libata]
[<ffffffffc04a4cb0>] ata_scsi_error+0xa0/0xe0 [libata]
[<ffffffffbdae9036>] scsi_error_handler+0xf6/0x8b0
[<ffffffffbdae8f40>] ? scsi_eh_get_sense+0x250/0x250
[<ffffffffbd6c5c21>] kthread+0xd1/0xe0
[<ffffffffbd6c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffbdd93ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffbd6c5b50>] ? insert_kthread_work+0x40/0x40
Complete report #213428
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffb418b805
Build id: 3.10.0-1160.31.1.el7.x86_64 		0x15
2
ata_bmdma_error_handler
libata 0xffffffffc00b1dc0
Build id: 3.10.0-1160.31.1.el7.x86_64 		0x50
3
ata_scsi_port_error_handler
libata 0xffffffffc00ae842
Build id: 3.10.0-1160.31.1.el7.x86_64 		0x532
4
ata_scsi_error
libata 0xffffffffc00aecb0
Build id: 3.10.0-1160.31.1.el7.x86_64 		0xa0
5
scsi_error_handler
vmlinux 0xffffffffb3ee97d6
Build id: 3.10.0-1160.31.1.el7.x86_64 		0xf6
6
? scsi_eh_get_sense
vmlinux 0xffffffffb3ee96e0
Build id: 3.10.0-1160.31.1.el7.x86_64 		0x250
7
kthread
vmlinux 0xffffffffb3ac5e31
Build id: 3.10.0-1160.31.1.el7.x86_64 		0xd1
8
? insert_kthread_work
vmlinux 0xffffffffb3ac5d60
Build id: 3.10.0-1160.31.1.el7.x86_64 		0x40
9
ret_from_fork_nospec_begin
vmlinux 0xffffffffb4195df7
Build id: 3.10.0-1160.31.1.el7.x86_64 		0x21
10
? insert_kthread_work
vmlinux 0xffffffffb3ac5d60
Build id: 3.10.0-1160.31.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 44s! [scsi_eh_0:263]
Modules linked in: tcp_lp uinput xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_multiport tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ext4 mbcache jbd2 dm_mirror dm_region_hash dm_log dm_mod sg pcspkr joydev virtio_balloon i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi virtio_console qxl drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm drm ata_piix serio_raw virtio_pci e1000 virtio_ring libata virtio drm_panel_orientation_quirks floppy fuse
CPU: 0 PID: 263 Comm: scsi_eh_0 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
task: ffff908eb8032100 ti: ffff908eb3ffc000 task.ti: ffff908eb3ffc000
RIP: 0010:[<ffffffffb418b805>]  [<ffffffffb418b805>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff908eb3fffd78  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffff908eb3ff0000 RCX: 0000000000000100
RDX: 00000000000003f6 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff908eb3fffd78 R08: 0000000000000029 R09: 0000000000000029
R10: 0000000000000428 R11: 3578302073757461 R12: ffff908eb3fffcf8
R13: ffff908db5c909d8 R14: ffff908eb3fffce0 R15: ffff908eb3fffd20
FS:  0000000000000000(0000) GS:ffff908ebfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd2e54ac000 CR3: 00000000ac11c000 CR4: 00000000000006f0
Call Trace:
[<ffffffffc00b1dc0>] ata_bmdma_error_handler+0x50/0x130 [libata]
[<ffffffffc00ae842>] ata_scsi_port_error_handler+0x532/0x900 [libata]
[<ffffffffc00aecb0>] ata_scsi_error+0xa0/0xe0 [libata]
[<ffffffffb3ee97d6>] scsi_error_handler+0xf6/0x8b0
[<ffffffffb3ee96e0>] ? scsi_eh_get_sense+0x250/0x250
[<ffffffffb3ac5e31>] kthread+0xd1/0xe0
[<ffffffffb3ac5d60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb4195df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb3ac5d60>] ? insert_kthread_work+0x40/0x40
Code: 96 b7 98 ff 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 e8 76 b7 98 ff 66 90 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #568177
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff9bd6b455
Build id: 3.10.0-957.21.3.el7.x86_64 		0x15
2
ata_sff_error_handler
libata 0xffffffffc0419836
Build id: 3.10.0-957.21.3.el7.x86_64 		0x86
3
ata_bmdma_error_handler
libata 0xffffffffc0419e48
Build id: 3.10.0-957.21.3.el7.x86_64 		0x58
4
ata_scsi_port_error_handler
libata 0xffffffffc04168c2
Build id: 3.10.0-957.21.3.el7.x86_64 		0x532
5
ata_scsi_error
libata 0xffffffffc0416d30
Build id: 3.10.0-957.21.3.el7.x86_64 		0xa0
6
scsi_error_handler
vmlinux 0xffffffff9bad6d36
Build id: 3.10.0-957.21.3.el7.x86_64 		0xf6
7
? scsi_eh_get_sense
vmlinux 0xffffffff9bad6c40
Build id: 3.10.0-957.21.3.el7.x86_64 		0x250
8
kthread
vmlinux 0xffffffff9b6c1da1
Build id: 3.10.0-957.21.3.el7.x86_64 		0xd1
9
? insert_kthread_work
vmlinux 0xffffffff9b6c1cd0
Build id: 3.10.0-957.21.3.el7.x86_64 		0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffff9bd75c37
Build id: 3.10.0-957.21.3.el7.x86_64 		0x21
11
? insert_kthread_work
vmlinux 0xffffffff9b6c1cd0
Build id: 3.10.0-957.21.3.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 38s! [scsi_eh_2:949]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 nfs fscache cmac arc4 md4 nls_utf8 cifs ccm dns_resolver vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp crc32_pclmul ghash_clmulni_intel ext4 aesni_intel mbcache lrw gf128mul jbd2 glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci nfsd nfs_acl lockd auth_rpcgss grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 ata_generic
pata_acpi floppy vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi libata scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks involflt(POE) dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 949 Comm: scsi_eh_2 Kdump: loaded Tainted: P           OEL ------------   3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8af795da1040 ti: ffff8af795ca4000 task.ti: ffff8af795ca4000
RIP: 0010:[<ffffffff9bd6b455>]  [<ffffffff9bd6b455>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8af795ca7d40  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffc0428278 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff8af795ca7d40 R08: 0000000051524420 R09: 35362064656e6961
R10: 0000000000001c37 R11: 656c63206f742073 R12: ffff8af795ca7d20
R13: 0000000000000029 R14: 0000000000000029 R15: ffff8af795ca7cc0
FS:  0000000000000000(0000) GS:ffff8af7bfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f394c801000 CR3: 000000083fe92000 CR4: 00000000000007f0
Call Trace:
[<ffffffffc0419836>] ata_sff_error_handler+0x86/0xf0 [libata]
[<ffffffffc0419e48>] ata_bmdma_error_handler+0x58/0x130 [libata]
[<ffffffffc04168c2>] ata_scsi_port_error_handler+0x532/0x900 [libata]
[<ffffffffc0416d30>] ata_scsi_error+0xa0/0xe0 [libata]
[<ffffffff9bad6d36>] scsi_error_handler+0xf6/0x8b0
[<ffffffff9bad6c40>] ? scsi_eh_get_sense+0x250/0x250
[<ffffffff9b6c1da1>] kthread+0xd1/0xe0
[<ffffffff9b6c1cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffff9bd75c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9b6c1cd0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48
Complete report #559459
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffb83817c5
Build id: 3.10.0-1062.el7.x86_64 		0x15
2
ata_sff_error_handler
libata 0xffffffffc0471836
Build id: 3.10.0-1062.el7.x86_64 		0x86
3
ata_bmdma_error_handler
libata 0xffffffffc0471e48
Build id: 3.10.0-1062.el7.x86_64 		0x58
4
ata_scsi_port_error_handler
libata 0xffffffffc046e8c2
Build id: 3.10.0-1062.el7.x86_64 		0x532
5
ata_scsi_error
libata 0xffffffffc046ed30
Build id: 3.10.0-1062.el7.x86_64 		0xa0
6
scsi_error_handler
vmlinux 0xffffffffb80e1f56
Build id: 3.10.0-1062.el7.x86_64 		0xf6
7
? scsi_eh_get_sense
vmlinux 0xffffffffb80e1e60
Build id: 3.10.0-1062.el7.x86_64 		0x250
8
kthread
vmlinux 0xffffffffb7cc50d1
Build id: 3.10.0-1062.el7.x86_64 		0xd1
9
? finish_task_switch
vmlinux 0xffffffffb7cd3f07
Build id: 3.10.0-1062.el7.x86_64 		0x57
10
? insert_kthread_work
vmlinux 0xffffffffb7cc5000
Build id: 3.10.0-1062.el7.x86_64 		0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffffb838bd37
Build id: 3.10.0-1062.el7.x86_64 		0x21
12
? insert_kthread_work
vmlinux 0xffffffffb7cc5000
Build id: 3.10.0-1062.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 29s! [scsi_eh_2:524]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev i2c_piix4 vmw_balloon cryptd joydev vmw_vmci sg pcspkr parport_pc parport pcc_cpufreq ip_tables xfs libcrc32c nfit libnvdimm vmwgfx sr_mod sd_mod cdrom crc_t10dif crct10dif_generic drm_kms_helper ata_generic pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common e1000 mptspi
crc32c_intel scsi_transport_spi mptscsih mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 524 Comm: scsi_eh_2 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff91459555d230 ti: ffff914594ae8000 task.ti: ffff914594ae8000
RIP: 0010:[<ffffffffb83817c5>]  [<ffffffffb83817c5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff914594aebd40  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffc04802a8 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff914594aebd40 R08: 0000000000000051 R09: 0000000000000051
R10: 00000000000008be R11: 44207261656c6320 R12: ffff914594aebd20
R13: 0000000000000001 R14: ffff914594ae8000 R15: ffff914594aebcc0
FS:  0000000000000000(0000) GS:ffff91459d680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7fc7a03000 CR3: 000000081c996000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc0471836>] ata_sff_error_handler+0x86/0xf0 [libata]
[<ffffffffc0471e48>] ata_bmdma_error_handler+0x58/0x130 [libata]
[<ffffffffc046e8c2>] ata_scsi_port_error_handler+0x532/0x900 [libata]
[<ffffffffc046ed30>] ata_scsi_error+0xa0/0xe0 [libata]
[<ffffffffb80e1f56>] scsi_error_handler+0xf6/0x8b0
[<ffffffffb80e1e60>] ? scsi_eh_get_sense+0x250/0x250
[<ffffffffb7cc50d1>] kthread+0xd1/0xe0
[<ffffffffb7cd3f07>] ? finish_task_switch+0x57/0x1c0
[<ffffffffb7cc5000>] ? insert_kthread_work+0x40/0x40
[<ffffffffb838bd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb7cc5000>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #526753
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff87789785
Build id: 3.10.0-1160.el7.x86_64 		0x15
2
ata_sff_error_handler
libata 0xffffffffc02747b6
Build id: 3.10.0-1160.el7.x86_64 		0x86
3
ata_bmdma_error_handler
libata 0xffffffffc0274dc8
Build id: 3.10.0-1160.el7.x86_64 		0x58
4
ata_scsi_port_error_handler
libata 0xffffffffc0271842
Build id: 3.10.0-1160.el7.x86_64 		0x532
5
ata_scsi_error
libata 0xffffffffc0271cb0
Build id: 3.10.0-1160.el7.x86_64 		0xa0
6
scsi_error_handler
vmlinux 0xffffffff874e9036
Build id: 3.10.0-1160.el7.x86_64 		0xf6
7
? _raw_spin_unlock_irqrestore
vmlinux 0xffffffff87789785
Build id: 3.10.0-1160.el7.x86_64 		0x15
8
? scsi_eh_get_sense
vmlinux 0xffffffff874e8f40
Build id: 3.10.0-1160.el7.x86_64 		0x250
9
kthread
vmlinux 0xffffffff870c5c21
Build id: 3.10.0-1160.el7.x86_64 		0xd1
10
? insert_kthread_work
vmlinux 0xffffffff870c5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffff87793df7
Build id: 3.10.0-1160.el7.x86_64 		0x21
12
? insert_kthread_work
vmlinux 0xffffffff870c5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 41s! [scsi_eh_1:310]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter ip_vs nf_conntrack bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm iosf_mbi snd_timer crc32_pclmul ghash_clmulni_intel snd aesni_intel lrw gf128mul glue_helper btusb soundcore btrtl btbcm ablk_helper btintel cryptd bluetooth rfkill ppdev pcspkr joydev vmw_balloon sg parport_pc parport i2c_piix4 vmw_vmci binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic vmwgfx
drm_kms_helper pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common nfit mptspi ata_piix crc32c_intel scsi_transport_spi libnvdimm drm serio_raw libata mptscsih mptbase e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 310 Comm: scsi_eh_1 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff8aa335f92100 ti: ffff8aa3357e8000 task.ti: ffff8aa3357e8000
RIP: 0010:[<ffffffff87789785>]  [<ffffffff87789785>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8aa3357ebd40  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffc02832a8 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff8aa3357ebd40 R08: 0000000051524420 R09: 35362064656e6961
R10: 00000000000008ce R11: 656c63206f742073 R12: ffff8aa3357ebd20
R13: 0000000000000074 R14: 0000000000000074 R15: ffff8aa3357ebcc0
FS:  0000000000000000(0000) GS:ffff8aa439600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000ef4a4000 CR3: 00000000b9b02000 CR4: 00000000003607f0
Call Trace:
[<ffffffffc02747b6>] ata_sff_error_handler+0x86/0xf0 [libata]
[<ffffffffc0274dc8>] ata_bmdma_error_handler+0x58/0x130 [libata]
[<ffffffffc0271842>] ata_scsi_port_error_handler+0x532/0x900 [libata]
[<ffffffffc0271cb0>] ata_scsi_error+0xa0/0xe0 [libata]
[<ffffffff874e9036>] scsi_error_handler+0xf6/0x8b0
[<ffffffff87789785>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[<ffffffff874e8f40>] ? scsi_eh_get_sense+0x250/0x250
[<ffffffff870c5c21>] kthread+0xd1/0xe0
[<ffffffff870c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff87793df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff870c5b50>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #234576
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff816ab905
Build id: 3.10.0-693.5.2.el7.x86_64 		0x15
2
ata_sff_error_handler
libata 0xffffffffc00fe3d3
Build id: 3.10.0-693.5.2.el7.x86_64 		0x83
3
ata_bmdma_error_handler
libata 0xffffffffc00fe9e8
Build id: 3.10.0-693.5.2.el7.x86_64 		0x58
4
ata_scsi_port_error_handler
libata 0xffffffffc00fb4e9
Build id: 3.10.0-693.5.2.el7.x86_64 		0x529
5
ata_scsi_error
libata 0xffffffffc00fb940
Build id: 3.10.0-693.5.2.el7.x86_64 		0xa0
6
scsi_error_handler
vmlinux 0xffffffff8146a913
Build id: 3.10.0-693.5.2.el7.x86_64 		0xf3
7
? scsi_eh_get_sense
vmlinux 0xffffffff8146a820
Build id: 3.10.0-693.5.2.el7.x86_64 		0x250
8
kthread
vmlinux 0xffffffff810b099f
Build id: 3.10.0-693.5.2.el7.x86_64 		0xcf
9
? insert_kthread_work
vmlinux 0xffffffff810b08d0
Build id: 3.10.0-693.5.2.el7.x86_64 		0x40
10
ret_from_fork
vmlinux 0xffffffff816b4fd8
Build id: 3.10.0-693.5.2.el7.x86_64 		0x58
11
? insert_kthread_work
vmlinux 0xffffffff810b08d0
Build id: 3.10.0-693.5.2.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 34s! [scsi_eh_2:308]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ftp ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq ppdev aesni_intel lrw
snd_seq_device gf128mul snd_pcm glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev snd_timer snd sg soundcore nfit i2c_piix4 libnvdimm shpchp vmw_vmci parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi crct10dif_pclmul crct10dif_common crc32c_intel scsi_transport_spi e1000 mptscsih serio_raw mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 308 Comm: scsi_eh_2 Tainted: G        W    L ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff88017c940000 ti: ffff880231a24000 task.ti: ffff880231a24000
RIP: 0010:[<ffffffff816ab905>]  [<ffffffff816ab905>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff880231a27d40  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffc010d990 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff880231a27d40 R08: 00000000203a3261 R09: 63206f7420736574
R10: 00000000000009cd R11: 2064656e69617264 R12: ffff880231a27d20
R13: 0000000000000074 R14: 0000000000000074 R15: ffff880231a27cc0
FS:  0000000000000000(0000) GS:ffff880236640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fe81cf8b018 CR3: 00000000b7a89000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff880231a27d78 ffffffffc00fe3d3 ffff880230070000 ffff880230070158
0000000000000202 0000000000000246 0000000000000000 ffff880231a27da0
ffffffffc00fe9e8 ffff880230070000 0000000000000001 0000000000000000
Call Trace:
[<ffffffffc00fe3d3>] ata_sff_error_handler+0x83/0xf0 [libata]
[<ffffffffc00fe9e8>] ata_bmdma_error_handler+0x58/0x110 [libata]
[<ffffffffc00fb4e9>] ata_scsi_port_error_handler+0x529/0x8e0 [libata]
[<ffffffffc00fb940>] ata_scsi_error+0xa0/0xe0 [libata]
[<ffffffff8146a913>] scsi_error_handler+0xf3/0x8b0
[<ffffffff8146a820>] ? scsi_eh_get_sense+0x250/0x250
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #83958
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffb296a175
Build id: 3.10.0-957.1.3.el7.x86_64 		0x15
2
ata_sff_error_handler
libata 0xffffffffc025e836
Build id: 3.10.0-957.1.3.el7.x86_64 		0x86
3
ata_bmdma_error_handler
libata 0xffffffffc025ee48
Build id: 3.10.0-957.1.3.el7.x86_64 		0x58
4
ata_scsi_port_error_handler
libata 0xffffffffc025b8c2
Build id: 3.10.0-957.1.3.el7.x86_64 		0x532
5
ata_scsi_error
libata 0xffffffffc025bd30
Build id: 3.10.0-957.1.3.el7.x86_64 		0xa0
6
scsi_error_handler
vmlinux 0xffffffffb26d6206
Build id: 3.10.0-957.1.3.el7.x86_64 		0xf6
7
? scsi_eh_get_sense
vmlinux 0xffffffffb26d6110
Build id: 3.10.0-957.1.3.el7.x86_64 		0x250
8
kthread
vmlinux 0xffffffffb22c1c31
Build id: 3.10.0-957.1.3.el7.x86_64 		0xd1
9
? insert_kthread_work
vmlinux 0xffffffffb22c1b60
Build id: 3.10.0-957.1.3.el7.x86_64 		0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffffb2974c1d
Build id: 3.10.0-957.1.3.el7.x86_64 		0x7
11
? insert_kthread_work
vmlinux 0xffffffffb22c1b60
Build id: 3.10.0-957.1.3.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 64s! [scsi_eh_1:727]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmhgfs(OE) vmw_vsock_vmci_transport vsock sunrpc iosf_mbi snd_seq_midi snd_seq_midi_event crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd joydev pcspkr snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore sg parport_pc parport i2c_piix4 vmw_vmci binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod cdrom crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw nfit
libnvdimm ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi e1000 scsi_transport_spi ata_piix mptscsih libata drm_panel_orientation_quirks mptbase dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 727 Comm: scsi_eh_1 Kdump: loaded Tainted: G           OEL ------------   3.10.0-957.1.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff92f8f5bb2080 ti: ffff92f8f4aa8000 task.ti: ffff92f8f4aa8000
RIP: 0010:[<ffffffffb296a175>]  [<ffffffffb296a175>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff92f8f4aabd40  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffc026d278 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff92f8f4aabd40 R08: 00000000203a3261 R09: 63206f7420736574
R10: 0000000000000e46 R11: 2064656e69617264 R12: ffff92f8f4aabd20
R13: 0000000000000001 R14: ffff92f8f4aa8000 R15: ffff92f8f4aabcc0
FS:  0000000000000000(0000) GS:ffff92f97a640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000131bc50 CR3: 00000000ac6b4000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc025e836>] ata_sff_error_handler+0x86/0xf0 [libata]
[<ffffffffc025ee48>] ata_bmdma_error_handler+0x58/0x130 [libata]
[<ffffffffc025b8c2>] ata_scsi_port_error_handler+0x532/0x900 [libata]
[<ffffffffc025bd30>] ata_scsi_error+0xa0/0xe0 [libata]
[<ffffffffb26d6206>] scsi_error_handler+0xf6/0x8b0
[<ffffffffb26d6110>] ? scsi_eh_get_sense+0x250/0x250
[<ffffffffb22c1c31>] kthread+0xd1/0xe0
[<ffffffffb22c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb2974c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb22c1b60>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #52989
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff816ab865
Build id: 3.10.0-693.2.2.el7.x86_64 		0x15
2
ata_sff_error_handler
libata 0xffffffffc00ae3d3
Build id: 3.10.0-693.2.2.el7.x86_64 		0x83
3
ata_bmdma_error_handler
libata 0xffffffffc00ae9e8
Build id: 3.10.0-693.2.2.el7.x86_64 		0x58
4
ata_scsi_port_error_handler
libata 0xffffffffc00ab4e9
Build id: 3.10.0-693.2.2.el7.x86_64 		0x529
5
ata_scsi_error
libata 0xffffffffc00ab940
Build id: 3.10.0-693.2.2.el7.x86_64 		0xa0
6
scsi_error_handler
vmlinux 0xffffffff8146aa53
Build id: 3.10.0-693.2.2.el7.x86_64 		0xf3
7
? scsi_eh_get_sense
vmlinux 0xffffffff8146a960
Build id: 3.10.0-693.2.2.el7.x86_64 		0x250
8
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.2.2.el7.x86_64 		0xcf
9
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x40
10
ret_from_fork
vmlinux 0xffffffff816b4f58
Build id: 3.10.0-693.2.2.el7.x86_64 		0x58
11
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 29s! [scsi_eh_2:292]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter lp vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ppdev joydev snd_ens1371 vmw_balloon snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore nfit sg libnvdimm parport_pc parport shpchp vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi
sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common ata_piix crc32c_intel libata mptspi serio_raw scsi_transport_spi mptscsih mptbase vmxnet3 i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 292 Comm: scsi_eh_2 Tainted: G           OEL ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff880035298fd0 ti: ffff8800352a0000 task.ti: ffff8800352a0000
RIP: 0010:[<ffffffff816ab865>]  [<ffffffff816ab865>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8800352a3d40  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffc00bd990 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff8800352a3d40 R08: 0000000051524420 R09: 35362064656e6961
R10: 0000000000000a1b R11: 656c63206f742073 R12: ffff8800352a3d20
R13: 0000000000000074 R14: 0000000000000074 R15: ffff8800352a3cc0
FS:  0000000000000000(0000) GS:ffff88007b600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffb62cc6490 CR3: 0000000079082000 CR4: 00000000001407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff8800352a3d78 ffffffffc00ae3d3 ffff880035088000 ffff880035088158
0000000000000202 0000000000000246 0000000000000000 ffff8800352a3da0
ffffffffc00ae9e8 ffff880035088000 0000000000000001 0000000000000000
Call Trace:
[<ffffffffc00ae3d3>] ata_sff_error_handler+0x83/0xf0 [libata]
[<ffffffffc00ae9e8>] ata_bmdma_error_handler+0x58/0x110 [libata]
[<ffffffffc00ab4e9>] ata_scsi_port_error_handler+0x529/0x8e0 [libata]
[<ffffffffc00ab940>] ata_scsi_error+0xa0/0xe0 [libata]
[<ffffffff8146aa53>] scsi_error_handler+0xf3/0x8b0
[<ffffffff8146a960>] ? scsi_eh_get_sense+0x250/0x250
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f58>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #6405
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff8ad81825
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x15
2
ata_sff_error_handler
libata 0xffffffffc021d836
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x86
3
ata_bmdma_error_handler
libata 0xffffffffc021de48
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x58
4
ata_scsi_port_error_handler
libata 0xffffffffc021a8c2
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x532
5
ata_scsi_error
libata 0xffffffffc021ad30
Build id: 3.10.0-1062.1.2.el7.x86_64 		0xa0
6
scsi_error_handler
vmlinux 0xffffffff8aae1f56
Build id: 3.10.0-1062.1.2.el7.x86_64 		0xf6
7
? scsi_eh_get_sense
vmlinux 0xffffffff8aae1e60
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x250
8
kthread
vmlinux 0xffffffff8a6c50d1
Build id: 3.10.0-1062.1.2.el7.x86_64 		0xd1
9
? insert_kthread_work
vmlinux 0xffffffff8a6c5000
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffff8ad8cd37
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x21
11
? insert_kthread_work
vmlinux 0xffffffff8a6c5000
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 36s! [scsi_eh_1:306]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink binfmt_misc vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus coretemp snd_seq iosf_mbi crc32_pclmul snd_seq_device snd_pcm ghash_clmulni_intel aesni_intel ppdev snd_timer snd lrw gf128mul vmw_balloon glue_helper soundcore ablk_helper cryptd sg joydev pcspkr parport_pc parport vmw_vmci i2c_piix4 sch_fq_codel ip_tables xfs libcrc32c sr_mod cdrom sd_mod ata_generic
crc_t10dif pata_acpi crct10dif_generic vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nfit ttm mptspi libnvdimm serio_raw scsi_transport_spi drm mptscsih e1000 ata_piix libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]
CPU: 0 PID: 306 Comm: scsi_eh_1 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9da9f5ce20e0 ti: ffff9dabf00a8000 task.ti: ffff9dabf00a8000
RIP: 0010:[<ffffffff8ad81825>]  [<ffffffff8ad81825>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9dabf00abd40  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffc022c2a8 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff9dabf00abd40 R08: 0000000051524420 R09: 35362064656e6961
R10: 0000000000006abb R11: 656c63206f742073 R12: ffff9dabf00abd20
R13: 0000000000000029 R14: 0000000000000029 R15: ffff9dabf00abcc0
FS:  0000000000000000(0000) GS:ffff9dabf5600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe8d41df000 CR3: 00000001c66d0000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc021d836>] ata_sff_error_handler+0x86/0xf0 [libata]
[<ffffffffc021de48>] ata_bmdma_error_handler+0x58/0x130 [libata]
[<ffffffffc021a8c2>] ata_scsi_port_error_handler+0x532/0x900 [libata]
[<ffffffffc021ad30>] ata_scsi_error+0xa0/0xe0 [libata]
[<ffffffff8aae1f56>] scsi_error_handler+0xf6/0x8b0
[<ffffffff8aae1e60>] ? scsi_eh_get_sense+0x250/0x250
[<ffffffff8a6c50d1>] kthread+0xd1/0xe0
[<ffffffff8a6c5000>] ? insert_kthread_work+0x40/0x40
[<ffffffff8ad8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8a6c5000>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #3870
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff816ab905
Build id: 3.10.0-693.5.2.el7.x86_64 		0x15
2
ata_sff_error_handler
libata 0xffffffffc00e13d3
Build id: 3.10.0-693.5.2.el7.x86_64 		0x83
3
ata_bmdma_error_handler
libata 0xffffffffc00e19e8
Build id: 3.10.0-693.5.2.el7.x86_64 		0x58
4
ata_scsi_port_error_handler
libata 0xffffffffc00de4e9
Build id: 3.10.0-693.5.2.el7.x86_64 		0x529
5
ata_scsi_error
libata 0xffffffffc00de940
Build id: 3.10.0-693.5.2.el7.x86_64 		0xa0
6
scsi_error_handler
vmlinux 0xffffffff8146a913
Build id: 3.10.0-693.5.2.el7.x86_64 		0xf3
7
? scsi_eh_get_sense
vmlinux 0xffffffff8146a820
Build id: 3.10.0-693.5.2.el7.x86_64 		0x250
8
kthread
vmlinux 0xffffffff810b099f
Build id: 3.10.0-693.5.2.el7.x86_64 		0xcf
9
? insert_kthread_work
vmlinux 0xffffffff810b08d0
Build id: 3.10.0-693.5.2.el7.x86_64 		0x40
10
ret_from_fork
vmlinux 0xffffffff816b4fd8
Build id: 3.10.0-693.5.2.el7.x86_64 		0x58
11
? insert_kthread_work
vmlinux 0xffffffff810b08d0
Build id: 3.10.0-693.5.2.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 31s! [scsi_eh_1:285]
Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd hv_utils joydev ptp pcspkr sg pps_core i2c_piix4 i2c_core ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi hv_storvsc scsi_transport_fc hv_netvsc ata_piix hyperv_keyboard hid_hyperv scsi_tgt libata crct10dif_pclmul
hyperv_fb crct10dif_common crc32c_intel floppy hv_vmbus serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 285 Comm: scsi_eh_1 Tainted: G             L ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090006  05/23/2012
task: ffff8800366daf70 ti: ffff88003674c000 task.ti: ffff88003674c000
RIP: 0010:[<ffffffff816ab905>]  [<ffffffff816ab905>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff88003674fd40  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffc00f0990 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff88003674fd40 R08: 000000000000000a R09: 0000000000000001
R10: 00000000000007f1 R11: ffff88003674f9be R12: ffff88003674fd20
R13: 000000000000b344 R14: ffff88003674c000 R15: ffff88003674fcc0
FS:  0000000000000000(0000) GS:ffff880102600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f47355ae5f0 CR3: 00000000f2d72000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff88003674fd78 ffffffffc00e13d3 ffff8800366c0000 ffff8800366c0158
0000000000000202 0000000000000246 0000000000000000 ffff88003674fda0
ffffffffc00e19e8 ffff8800366c0000 0000000000000001 0000000000000000
Call Trace:
[<ffffffffc00e13d3>] ata_sff_error_handler+0x83/0xf0 [libata]
[<ffffffffc00e19e8>] ata_bmdma_error_handler+0x58/0x110 [libata]
[<ffffffffc00de4e9>] ata_scsi_port_error_handler+0x529/0x8e0 [libata]
[<ffffffffc00de940>] ata_scsi_error+0xa0/0xe0 [libata]
[<ffffffff8146a913>] scsi_error_handler+0xf3/0x8b0
[<ffffffff8146a820>] ? scsi_eh_get_sense+0x250/0x250
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48