Info

Permalink
Function
_raw_spin_unlock_irqrestore
First occurrence
2021-06-06
Last occurrence
2025-06-04
State
NEW
Components
kernel
Quality
-14

Statistics

Operating system Count
CentOS 7 61
Architecture Count
x86_64 61
Related packages Count
kernel 61
 0:3.10.0-1160.el7 36
 0:3.10.0-693.el7 6
 0:3.10.0-1160.36.2.el7 3
 0:3.10.0-1160.119.1.el7 2
 0:3.10.0-1160.83.1.el7 2
 0:3.10.0-1160.95.1.el7 2
 0:3.10.0-1127.19.1.el7 1
 0:3.10.0-1160.49.1.el7 1
 0:3.10.0-1160.59.1.el7 1
 0:3.10.0-514.el7 1
 0:3.10.0-514.21.1.el7 1
 0:3.10.0-1062.18.1.el7 1
 0:3.10.0-1160.118.1.el7 1
 0:3.10.0-1160.31.1.el7 1
 0:3.10.0-1160.88.1.el7 1
 0:3.10.0-1160.108.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1303062
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff8f5ba385
Build id: 3.10.0-1160.108.1.el7.x86_64
0x15
2
do_task_stat
vmlinux 0xffffffff8f0dee29
Build id: 3.10.0-1160.108.1.el7.x86_64
0x2a9
3
proc_tgid_stat
vmlinux 0xffffffff8f0e0274
Build id: 3.10.0-1160.108.1.el7.x86_64
0x14
4
proc_single_show
vmlinux 0xffffffff8f0da282
Build id: 3.10.0-1160.108.1.el7.x86_64
0x52
5
seq_read
vmlinux 0xffffffff8f085858
Build id: 3.10.0-1160.108.1.el7.x86_64
0x138
6
vfs_read
vmlinux 0xffffffff8f05bb9f
Build id: 3.10.0-1160.108.1.el7.x86_64
0x9f
7
SyS_read
vmlinux 0xffffffff8f05ca15
Build id: 3.10.0-1160.108.1.el7.x86_64
0x55
8
system_call_fastpath
vmlinux 0xffffffff8f5c539a
Build id: 3.10.0-1160.108.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 14336s! [top:20915]
<EOI> 
Modules linked in:
xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat xt_addrtype iptable_filter br_netfilter bridge stp llc nft_chain_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack nf_tables_ipv6 nf_tables_ipv4 nf_tables overlay(T) vboxsf(OE) snd_intel8x0 snd_ac97_codec crc32_pclmul joydev ac97_bus ghash_clmulni_intel ppdev snd_seq snd_seq_device snd_pcm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr snd_timer sg snd vboxguest(OE) parport_pc parport video i2c_piix4 soundcore sch_fq_codel binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic vmwgfx pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel
drm ahci libahci e1000 serio_raw ata_piix drm_panel_orientation_quirks libata dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 20915 Comm: top Tainted: G           OEL ------------ T 3.10.0-1160.108.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff942df594a100 ti: ffff942e27500000 task.ti: ffff942e27500000
RIP: 0010:[<ffffffff8f5ba385>]  [<ffffffff8f5ba385>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff942e27503ce0  EFLAGS: 00010246
RAX: 0000000000000001 RBX: ffff942e27503c98 RCX: ffffffff8fa55b70
RDX: ffffffff8fa55280 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff942e27503ce0 R08: 0000000000009e76 R09: 0000000000000000
R10: 0000000000009e76 R11: 0000000000009e76 R12: ffff942e27503c60
R13: 0000000000000001 R14: 0000000000000000 R15: ffff942df5b2df44
FS:  00007ff639268880(0000) GS:ffff942e81200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1c49cc30e0 CR3: 00000000b9c90000 CR4: 00000000000206f0
Call Trace:
[<ffffffff8f0dee29>] do_task_stat+0x2a9/0xbb0
[<ffffffff8f0e0274>] proc_tgid_stat+0x14/0x20
[<ffffffff8f0da282>] proc_single_show+0x52/0x90
[<ffffffff8f085858>] seq_read+0x138/0x460
[<ffffffff8f05bb9f>] vfs_read+0x9f/0x170
[<ffffffff8f05ca15>] SyS_read+0x55/0xd0
[<ffffffff8f5c539a>] system_call_fastpath+0x25/0x2a
Code: 8a 42 96 ff 66 90 5d c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 e8 6a 42 96 ff 66 90 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 cc cc cc cc 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #1234430
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff933ba385
Build id: 3.10.0-1160.88.1.el7.x86_64
0x15
2
__delayacct_blkio_ticks
vmlinux 0xffffffff92d66c46
Build id: 3.10.0-1160.88.1.el7.x86_64
0x46
3
do_task_stat
vmlinux 0xffffffff92edf155
Build id: 3.10.0-1160.88.1.el7.x86_64
0x725
4
proc_tgid_stat
vmlinux 0xffffffff92ee0124
Build id: 3.10.0-1160.88.1.el7.x86_64
0x14
5
proc_single_show
vmlinux 0xffffffff92eda132
Build id: 3.10.0-1160.88.1.el7.x86_64
0x52
6
seq_read
vmlinux 0xffffffff92e857e8
Build id: 3.10.0-1160.88.1.el7.x86_64
0x138
7
vfs_read
vmlinux 0xffffffff92e5bb2f
Build id: 3.10.0-1160.88.1.el7.x86_64
0x9f
8
SyS_read
vmlinux 0xffffffff92e5c9a5
Build id: 3.10.0-1160.88.1.el7.x86_64
0x55
9
system_call_fastpath
vmlinux 0xffffffff933c539a
Build id: 3.10.0-1160.88.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [pool:3776]
Modules linked in: ch osst st iscsit(OE) vtldev(OE) vtlitf(OE) qla2xxx(OE) nvme_fc nvme_fabrics nvme_core scsi_transport_fc scsi_tgt xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi vmw_vsock_vmci_transport vsock sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg vmw_balloon pcspkr joydev parport_pc parport vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c vmwgfx drm_kms_helper sr_mod cdrom syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm sd_mod ata_generic pata_acpi crc_t10dif crct10dif_generic drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw e1000 nfit libnvdimm vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 3776 Comm: pool Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff928ff9693180 ti: ffff9290f663c000 task.ti: ffff9290f663c000
RIP: 0010:[<ffffffff933ba385>]  [<ffffffff933ba385>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9290f663fcb8  EFLAGS: 00000282
RAX: 0000000000000004 RBX: fffffffdff0a0003 RCX: 00000000000000e4
RDX: 0000000000000004 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff9290f663fcb8 R08: 0000000000000025 R09: 0000000000000020
R10: 0000000000000000 R11: ffff9290f663fb86 R12: ffff9290411e9000
R13: 0000000000000002 R14: ffff928fefac4200 R15: ffff928ff07affff
FS:  00007f94c10c4700(0000) GS:ffff9290ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f94cf492000 CR3: 0000000232960000 CR4: 00000000001607e0
Call Trace:
[<ffffffff92d66c46>] __delayacct_blkio_ticks+0x46/0x60
[<ffffffff92edf155>] do_task_stat+0x725/0xbb0
[<ffffffff92ee0124>] proc_tgid_stat+0x14/0x20
[<ffffffff92eda132>] proc_single_show+0x52/0x90
[<ffffffff92e857e8>] seq_read+0x138/0x460
[<ffffffff92e5bb2f>] vfs_read+0x9f/0x170
[<ffffffff92e5c9a5>] SyS_read+0x55/0xd0
[<ffffffff933c539a>] system_call_fastpath+0x25/0x2a
Code: 07 00 0f 1f 40 00 5d c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 cc cc cc cc 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #1115158
Frame # Function Binary Source or offset Line
1
do_task_stat
vmlinux 0xffffffff81277639
Build id: 3.10.0-693.el7.x86_64
0x6c9
2
? rescuer_thread
vmlinux 0xffffffff810a8dfe
Build id: 3.10.0-693.el7.x86_64
0x31e
3
proc_tgid_stat
vmlinux 0xffffffff81278484
Build id: 3.10.0-693.el7.x86_64
0x14
4
proc_single_show
vmlinux 0xffffffff812729dd
Build id: 3.10.0-693.el7.x86_64
0x4d
5
seq_read
vmlinux 0xffffffff812262fa
Build id: 3.10.0-693.el7.x86_64
0x10a
6
vfs_read
vmlinux 0xffffffff81200b9c
Build id: 3.10.0-693.el7.x86_64
0x9c
7
SyS_read
vmlinux 0xffffffff81201a5f
Build id: 3.10.0-693.el7.x86_64
0x7f
8
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 49s! [nmon:16052]
Modules linked in: tcp_diag inet_diag nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache vmw_vsock_vmci_transport vsock xfs libcrc32c sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg parport_pc parport shpchp vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix mptspi drm libata scsi_transport_spi vmxnet3 mptscsih mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 16052 Comm: nmon Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff880088111fa0 ti: ffff880123d04000 task.ti: ffff880123d04000
RIP: 0010:[<ffffffff81277639>]  [<ffffffff81277639>] do_task_stat+0x6c9/0xb70
RSP: 0018:ffff880123d07d20  EFLAGS: 00010246
RAX: ffff8800357d8000 RBX: ffff880123d07ce8 RCX: 0000000000000002
RDX: 0000000000000037 RSI: 0000000000000020 RDI: ffff8800358f2600
RBP: ffff880123d07e48 R08: 0000000000000035 R09: 0000000000000020
R10: 0000000000000000 R11: ffff880123d07bb6 R12: 0000000000000f72
R13: ffffffff8132c312 R14: ffff880123d07ca0 R15: ffff8800357d0000
FS:  00007f595fec8740(0000) GS:ffff8801bfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f595feda000 CR3: 000000014a290000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
 ffffffffffffffff 0000000000000001 ffffffffffffffff 0000000000000000
 0000000000000000 0000000000000000 0000000000000000 0000000000000000
 0000000000000000 ffff8801b54c2380 0000000000000362 00000000ffffffec
Call Trace:
 [<ffffffff810a8dfe>] ? rescuer_thread+0x31e/0x3c0
 [<ffffffff81278484>] proc_tgid_stat+0x14/0x20
 [<ffffffff812729dd>] proc_single_show+0x4d/0x80
 [<ffffffff812262fa>] seq_read+0x10a/0x3b0
 [<ffffffff81200b9c>] vfs_read+0x9c/0x170
 [<ffffffff81201a5f>] SyS_read+0x7f/0xe0
 [<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 00 48 89 df e8 a9 f7 fa ff 49 63 94 24 90 04 00 00 be 20 00 00 00 48 89 df e8 24 f8 fa ff 49 8b 44 24 08 be 20 00 00 00 48 89 df <8b> 50 18 e8 0f f8 fa ff 41 8b 54 24 5c be 20 00 00 00 48 89 df 
Complete report #1112278
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff9418b805
Build id: 3.10.0-1160.31.1.el7.x86_64
0x15
2
__delayacct_blkio_ticks
vmlinux 0xffffffff93b5de56
Build id: 3.10.0-1160.31.1.el7.x86_64
0x46
3
do_task_stat
vmlinux 0xffffffff93cce885
Build id: 3.10.0-1160.31.1.el7.x86_64
0x725
4
? ep_poll
vmlinux 0xffffffff93c9d25e
Build id: 3.10.0-1160.31.1.el7.x86_64
0x23e
5
proc_tgid_stat
vmlinux 0xffffffff93ccf844
Build id: 3.10.0-1160.31.1.el7.x86_64
0x14
6
proc_single_show
vmlinux 0xffffffff93cc9a12
Build id: 3.10.0-1160.31.1.el7.x86_64
0x52
7
seq_read
vmlinux 0xffffffff93c767e0
Build id: 3.10.0-1160.31.1.el7.x86_64
0x130
8
vfs_read
vmlinux 0xffffffff93c4deef
Build id: 3.10.0-1160.31.1.el7.x86_64
0x9f
9
SyS_read
vmlinux 0xffffffff93c4ed6f
Build id: 3.10.0-1160.31.1.el7.x86_64
0x7f
10
system_call_fastpath
vmlinux 0xffffffff94195f92
Build id: 3.10.0-1160.31.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 43s! [pool:12511]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nf_nat_ftp nf_conntrack_ftp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom vmwgfx
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_generic pata_acpi ahci drm sd_mod crc_t10dif crct10dif_generic libahci nfit ata_piix libnvdimm libata crct10dif_pclmul crct10dif_common crc32c_intel vmxnet3 serio_raw vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 12511 Comm: pool Kdump: loaded Not tainted 3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffffa0d11b176300 ti: ffffa0d11b0d8000 task.ti: ffffa0d11b0d8000
RIP: 0010:[<ffffffff9418b805>]  [<ffffffff9418b805>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffffa0d11b0dbcb8  EFLAGS: 00000286
RAX: 0000000000000000 RBX: fffffffdff0a0003 RCX: 00000000000000f5
RDX: 0000000000000000 RSI: 0000000000000286 RDI: 0000000000000286
RBP: ffffa0d11b0dbcb8 R08: 000000000000001e R09: 0000000000000020
R10: 0000000000000000 R11: ffffa0d11b0dbb86 R12: ffffa0d132443000
R13: 0000000000000002 R14: ffffa0cfb3ad2100 R15: ffffa0d12468ffff
FS:  00007fc52e53b700(0000) GS:ffffa0d13fcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc53c908000 CR3: 000000023576e000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff93b5de56>] __delayacct_blkio_ticks+0x46/0x50
[<ffffffff93cce885>] do_task_stat+0x725/0xbb0
[<ffffffff93c9d25e>] ? ep_poll+0x23e/0x360
[<ffffffff93ccf844>] proc_tgid_stat+0x14/0x20
[<ffffffff93cc9a12>] proc_single_show+0x52/0x90
[<ffffffff93c767e0>] seq_read+0x130/0x440
[<ffffffff93c4deef>] vfs_read+0x9f/0x170
[<ffffffff93c4ed6f>] SyS_read+0x7f/0xf0
[<ffffffff94195f92>] system_call_fastpath+0x25/0x2a
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #1088708
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff9a989785
Build id: 3.10.0-1160.el7.x86_64
0x15
2
do_task_stat
vmlinux 0xffffffff9a4cdf99
Build id: 3.10.0-1160.el7.x86_64
0x2a9
3
proc_tgid_stat
vmlinux 0xffffffff9a4cf3d4
Build id: 3.10.0-1160.el7.x86_64
0x14
4
proc_single_show
vmlinux 0xffffffff9a4c95a2
Build id: 3.10.0-1160.el7.x86_64
0x52
5
seq_read
vmlinux 0xffffffff9a476500
Build id: 3.10.0-1160.el7.x86_64
0x130
6
vfs_read
vmlinux 0xffffffff9a44d9bf
Build id: 3.10.0-1160.el7.x86_64
0x9f
7
SyS_read
vmlinux 0xffffffff9a44e83f
Build id: 3.10.0-1160.el7.x86_64
0x7f
8
system_call_fastpath
vmlinux 0xffffffff9a993f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 224s! [ora_mman_orcl:2719]
Modules linked in: tcp_diag inet_diag vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus sb_edac snd_seq iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_seq_device snd_pcm lrw gf128mul snd_timer glue_helper ablk_helper snd ppdev joydev pcspkr cryptd soundcore sg vmw_balloon parport_pc parport vmw_vmci i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt nfit fb_sys_fops ttm libnvdimm ata_piix ahci drm crct10dif_pclmul crct10dif_common libahci crc32c_intel libata e1000 serio_raw vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 2719 Comm: ora_mman_orcl Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8e5b3399c200 ti: ffff8e5b289cc000 task.ti: ffff8e5b289cc000
RIP: 0010:[<ffffffff9a989785>]  [<ffffffff9a989785>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8e5b289cfce0  EFLAGS: 00000246
RAX: ffff8e5b435e6300 RBX: ffff8e5b289cfc98 RCX: ffff8e56b33894b0
RDX: ffffffff9ae55280 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff8e5b289cfce0 R08: 0000008c8e84874f R09: 000001749e342fa2
R10: 000002012cb8b6f1 R11: 0000008c8e84874f R12: 0000000000000086
R13: 0000000200000004 R14: 0000000000000000 R15: ffff8e5b3b853a44
FS:  00007f76c1005400(0000) GS:ffff8e5f55680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f13bc87e000 CR3: 00000005e8a02000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9a4cdf99>] do_task_stat+0x2a9/0xbb0
[<ffffffff9a4cf3d4>] proc_tgid_stat+0x14/0x20
[<ffffffff9a4c95a2>] proc_single_show+0x52/0x90
[<ffffffff9a476500>] seq_read+0x130/0x440
[<ffffffff9a44d9bf>] vfs_read+0x9f/0x170
[<ffffffff9a44e83f>] SyS_read+0x7f/0xf0
[<ffffffff9a993f92>] system_call_fastpath+0x25/0x2a
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #581923
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffa9189785
Build id: 3.10.0-1160.el7.x86_64
0x15
2
do_task_stat
vmlinux 0xffffffffa8ccdf99
Build id: 3.10.0-1160.el7.x86_64
0x2a9
3
proc_tgid_stat
vmlinux 0xffffffffa8ccf3d4
Build id: 3.10.0-1160.el7.x86_64
0x14
4
proc_single_show
vmlinux 0xffffffffa8cc95a2
Build id: 3.10.0-1160.el7.x86_64
0x52
5
seq_read
vmlinux 0xffffffffa8c76500
Build id: 3.10.0-1160.el7.x86_64
0x130
6
vfs_read
vmlinux 0xffffffffa8c4d9bf
Build id: 3.10.0-1160.el7.x86_64
0x9f
7
SyS_read
vmlinux 0xffffffffa8c4e83f
Build id: 3.10.0-1160.el7.x86_64
0x7f
8
tracesys
vmlinux 0xffffffffa9194226
Build id: 3.10.0-1160.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [kworker/3:0:470]
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 41s! [ps:3755]
Modules linked in: tcp_lp veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi ppdev snd_ac97_codec ac97_bus aesni_intel
snd_seq lrw gf128mul snd_seq_device glue_helper ablk_helper vmw_balloon snd_pcm cryptd snd_timer joydev pcspkr snd soundcore i2c_piix4 sg vmw_vmci parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt ata_generic pata_acpi crct10dif_pclmul mptspi fb_sys_fops crct10dif_common ttm scsi_transport_spi drm ata_piix e1000 mptscsih crc32c_intel serio_raw mptbase nfit libnvdimm libata drm_panel_orientation_quirks fuse
CPU: 1 PID: 3755 Comm: ps Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff88969e310000 ti: ffff889634f94000 task.ti: ffff889634f94000
RIP: 0010:[<ffffffffa9189785>]  [<ffffffffa9189785>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff889634f97ce0  EFLAGS: 00000246
RAX: ffff88968b396300 RBX: ffff889634f97c98 RCX: ffff8895e3bdb950
RDX: ffff8895b8370000 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff889634f97ce0 R08: 000000aa5c192a1b R09: 0000010148aaf156
R10: 000001abb6db48bc R11: 000000aa6e305766 R12: 0000000000000082
R13: 000001ac00000004 R14: 0000000000000000 R15: ffff8895bd3111c4
FS:  00007ffb6066b880(0000) GS:ffff8897af240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffb5fe50021 CR3: 000000000293c000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa8ccdf99>] do_task_stat+0x2a9/0xbb0
[<ffffffffa8ccf3d4>] proc_tgid_stat+0x14/0x20
[<ffffffffa8cc95a2>] proc_single_show+0x52/0x90
[<ffffffffa8c76500>] seq_read+0x130/0x440
[<ffffffffa8c4d9bf>] vfs_read+0x9f/0x170
[<ffffffffa8c4e83f>] SyS_read+0x7f/0xf0
[<ffffffffa9194226>] tracesys+0xa6/0xcc
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #1045218
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff861ba385
Build id: 3.10.0-1160.118.1.el7.x86_64
0x15
2
do_task_stat
vmlinux 0xffffffff85cdee79
Build id: 3.10.0-1160.118.1.el7.x86_64
0x2a9
3
? vma_merge
vmlinux 0xffffffff85c06c96
Build id: 3.10.0-1160.118.1.el7.x86_64
0x116
4
proc_tgid_stat
vmlinux 0xffffffff85ce02c4
Build id: 3.10.0-1160.118.1.el7.x86_64
0x14
5
proc_single_show
vmlinux 0xffffffff85cda2d2
Build id: 3.10.0-1160.118.1.el7.x86_64
0x52
6
seq_read
vmlinux 0xffffffff85c858a8
Build id: 3.10.0-1160.118.1.el7.x86_64
0x138
7
vfs_read
vmlinux 0xffffffff85c5bbef
Build id: 3.10.0-1160.118.1.el7.x86_64
0x9f
8
SyS_read
vmlinux 0xffffffff85c5ca65
Build id: 3.10.0-1160.118.1.el7.x86_64
0x55
9
system_call_fastpath
vmlinux 0xffffffff861c539a
Build id: 3.10.0-1160.118.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 38s! [nmon:23156]
Modules linked in: tcp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter hcpdriver(POE) cmac arc4 md4 nls_utf8 cifs ccm dns_resolver vmw_vsock_vmci_transport vsock sunrpc ppdev sb_edac vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg i2c_piix4 vmw_vmci parport_pc parport sch_fq_codel binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt crc_t10dif fb_sys_fops crct10dif_generic ttm ahci nfit drm
ata_piix libahci libnvdimm crct10dif_pclmul libata crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ipmi_msghandler]
CPU: 8 PID: 23156 Comm: nmon Kdump: loaded Tainted: P           OEL ------------   3.10.0-1160.118.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 38s! [rocksdb:low6:127192]
task: ffff91774fefd280 ti: ffff917784040000 task.ti: ffff917784040000
RIP: 0010:[<ffffffff861ba385>]  [<ffffffff861ba385>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff917784043ce0  EFLAGS: 00000246
RAX: ffff917788abd280 RBX: ffff917784043c98 RCX: ffff91817ddb5230
RDX: ffffffff86655280 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff917784043ce0 R08: 0000023b11f1e85d R09: 000000e293e071f6
R10: 000000e293e071f6 R11: 0000023b11f1e85d R12: 0000000000000086
R13: 0000016200000004 R14: 0000000000000000 R15: ffff917788bbcc44
FS:  00007fe222598740(0000) GS:ffff9181d3800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe2225af000 CR3: 000000008bc98000 CR4: 00000000003607e0
Call Trace:
[<ffffffff85cdee79>] do_task_stat+0x2a9/0xbb0
[<ffffffff85c06c96>] ? vma_merge+0x116/0x370
[<ffffffff85ce02c4>] proc_tgid_stat+0x14/0x20
[<ffffffff85cda2d2>] proc_single_show+0x52/0x90
[<ffffffff85c858a8>] seq_read+0x138/0x460
[<ffffffff85c5bbef>] vfs_read+0x9f/0x170
[<ffffffff85c5ca65>] SyS_read+0x55/0xd0
[<ffffffff861c539a>] system_call_fastpath+0x25/0x2a
Complete report #798151
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffa9dba385
Build id: 3.10.0-1160.95.1.el7.x86_64
0x15
2
do_task_stat
vmlinux 0xffffffffa98dedb9
Build id: 3.10.0-1160.95.1.el7.x86_64
0x2a9
3
? vma_merge
vmlinux 0xffffffffa9806bd6
Build id: 3.10.0-1160.95.1.el7.x86_64
0x116
4
proc_tgid_stat
vmlinux 0xffffffffa98e0204
Build id: 3.10.0-1160.95.1.el7.x86_64
0x14
5
proc_single_show
vmlinux 0xffffffffa98da212
Build id: 3.10.0-1160.95.1.el7.x86_64
0x52
6
seq_read
vmlinux 0xffffffffa98857e8
Build id: 3.10.0-1160.95.1.el7.x86_64
0x138
7
vfs_read
vmlinux 0xffffffffa985bb2f
Build id: 3.10.0-1160.95.1.el7.x86_64
0x9f
8
SyS_read
vmlinux 0xffffffffa985c9a5
Build id: 3.10.0-1160.95.1.el7.x86_64
0x55
9
system_call_fastpath
vmlinux 0xffffffffa9dc539a
Build id: 3.10.0-1160.95.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 37s! [systemd:1]
Modules linked in: nfnetlink_queue nfnetlink_log bluetooth rfkill xt_limit ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_hashlimit xt_multiport xt_conntrack nf_conntrack iptable_filter pcspkr sg virtio_balloon joydev i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_blk net_failover failover bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw floppy virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 1 Comm: systemd Tainted: G             L ------------   3.10.0-1160.95.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.1-0-g3208b098f51a-prebuilt.qemu.org 04/01/2014
task: ffffa1163b940000 ti: ffffa1163b8cc000 task.ti: ffffa1163b8cc000
RIP: 0010:[<ffffffffa9dba385>]  [<ffffffffa9dba385>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffffa1163b8cfce0  EFLAGS: 00000246
RAX: ffffa1163b5d8000 RBX: ffffa1163b8cfc98 RCX: ffffa1163814dbb0
RDX: ffffffffaa255280 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffffa1163b8cfce0 R08: 00000000000f09f2 R09: 000000000c1c241e
R10: 00000000132c8e85 R11: 0000000007106a67 R12: 0000000000000082
R13: 0000000000000004 R14: 0000000000000000 R15: ffffa116390035c4
FS:  00007f3a945f5940(0000) GS:ffffa1163fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3a94606000 CR3: 000000007b588000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa98dedb9>] do_task_stat+0x2a9/0xbb0
[<ffffffffa9806bd6>] ? vma_merge+0x116/0x370
[<ffffffffa98e0204>] proc_tgid_stat+0x14/0x20
[<ffffffffa98da212>] proc_single_show+0x52/0x90
[<ffffffffa98857e8>] seq_read+0x138/0x460
[<ffffffffa985bb2f>] vfs_read+0x9f/0x170
[<ffffffffa985c9a5>] SyS_read+0x55/0xd0
[<ffffffffa9dc539a>] system_call_fastpath+0x25/0x2a
Code: 07 00 0f 1f 40 00 5d c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 cc cc cc cc 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #371511
Frame # Function Binary Source or offset Line
1
do_task_stat
vmlinux 0xffffffff81cced31
Build id: 3.10.0-1160.36.2.el7.x86_64
0x6e1
2
proc_tgid_stat
vmlinux 0xffffffff81ccfd34
Build id: 3.10.0-1160.36.2.el7.x86_64
0x14
3
proc_single_show
vmlinux 0xffffffff81cc9f02
Build id: 3.10.0-1160.36.2.el7.x86_64
0x52
4
seq_read
vmlinux 0xffffffff81c76cc0
Build id: 3.10.0-1160.36.2.el7.x86_64
0x130
5
vfs_read
vmlinux 0xffffffff81c4e3af
Build id: 3.10.0-1160.36.2.el7.x86_64
0x9f
6
SyS_read
vmlinux 0xffffffff81c4f22f
Build id: 3.10.0-1160.36.2.el7.x86_64
0x7f
7
system_call_fastpath
vmlinux 0xffffffff82195f92
Build id: 3.10.0-1160.36.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [systemd-logind:9753]
Modules linked in: tcp_lp rfcomm xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 crc32_pclmul ppdev ext4 snd_rawmidi mbcache ghash_clmulni_intel jbd2 snd_ac97_codec btusb ac97_bus btrtl btbcm snd_seq btintel
snd_seq_device bluetooth vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_pcm joydev pcspkr snd_timer rfkill sg snd soundcore i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul crct10dif_common drm libata crc32c_intel nfit mptspi scsi_transport_spi serio_raw libnvdimm mptscsih e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 9753 Comm: systemd-logind Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.36.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff90c78fbbb180 ti: ffff90c76e040000 task.ti: ffff90c76e040000
RIP: 0010:[<ffffffff81cced31>]  [<ffffffff81cced31>] do_task_stat+0x6e1/0xbb0
RSP: 0018:ffff90c76e043cf0  EFLAGS: 00010246
RAX: ffff90c597954000 RBX: ffff90c76e043cb8 RCX: 0000000000000002
RDX: 0000000000000037 RSI: 0000000000000020 RDI: ffff90c7978c8a80
RBP: ffff90c76e043e18 R08: 0000000000000021 R09: 000000000000ffff
R10: 0000000000000000 R11: ffff90c76e043b86 R12: 0000000000000f58
R13: ffffffff81d91622 R14: ffff90c76e043c70 R15: ffff90c6b590c200
FS:  00007fdbe90aa900(0000) GS:ffff90c96d600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f14de3ca290 CR3: 00000001e298c000 CR4: 0000000000360ff0
Call Trace:
[<ffffffff81ccfd34>] proc_tgid_stat+0x14/0x20
[<ffffffff81cc9f02>] proc_single_show+0x52/0x90
[<ffffffff81c76cc0>] seq_read+0x130/0x450
[<ffffffff81c4e3af>] vfs_read+0x9f/0x170
[<ffffffff81c4f22f>] SyS_read+0x7f/0xf0
[<ffffffff82195f92>] system_call_fastpath+0x25/0x2a
Code: 00 48 89 df e8 41 7c fa ff 49 63 94 24 90 04 00 00 be 20 00 00 00 48 89 df e8 bc 7c fa ff 49 8b 44 24 08 be 20 00 00 00 48 89 df <8b> 50 18 e8 a7 7c fa ff 41 8b 54 24 5c be 20 00 00 00 48 89 df 
Complete report #175098
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffbb1883c5
Build id: 3.10.0-1127.19.1.el7.x86_64
0x15
2
do_task_stat
vmlinux 0xffffffffbaccd509
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2a9
3
proc_tgid_stat
vmlinux 0xffffffffbacce944
Build id: 3.10.0-1127.19.1.el7.x86_64
0x14
4
proc_single_show
vmlinux 0xffffffffbacc8b12
Build id: 3.10.0-1127.19.1.el7.x86_64
0x52
5
seq_read
vmlinux 0xffffffffbac75a50
Build id: 3.10.0-1127.19.1.el7.x86_64
0x130
6
vfs_read
vmlinux 0xffffffffbac4d06f
Build id: 3.10.0-1127.19.1.el7.x86_64
0x9f
7
SyS_read
vmlinux 0xffffffffbac4dedf
Build id: 3.10.0-1127.19.1.el7.x86_64
0x7f
8
system_call_fastpath
vmlinux 0xffffffffbb192ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ps:3407]
Modules linked in: vsock_diag tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter ipmi_devintf ipmi_msghandler vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd pcspkr joydev vmw_balloon sg parport_pc parport i2c_piix4 vmw_vmci binfmt_misc ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix ahci
drm libahci crct10dif_pclmul crct10dif_common e1000e nfit crc32c_intel libnvdimm libata serio_raw ptp pps_core vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 3407 Comm: ps Kdump: loaded Tainted: G             L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8e77179862a0 ti: ffff8e7681388000 task.ti: ffff8e7681388000
RIP: 0010:[<ffffffffbb1883c5>]  [<ffffffffbb1883c5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8e768138bce0  EFLAGS: 00000246
RAX: ffff8e78e97e18c0 RBX: ffff8e768138bdc0 RCX: ffffffffbb655a70
RDX: ffffffffbb655180 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff8e768138bce0 R08: ffff8e78e97f1608 R09: ffff8e768138bc00
R10: ffff8e763fc01300 R11: 0000000000000520 R12: ffff8e768138bc98
R13: 0000000000000086 R14: 0000000000000086 R15: ffff8e78e97c6844
FS:  00007f7ddc74a880(0000) GS:ffff8e78ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1b6f4a6000 CR3: 0000000185fb0000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffbaccd509>] do_task_stat+0x2a9/0xbb0
[<ffffffffbacce944>] proc_tgid_stat+0x14/0x20
[<ffffffffbacc8b12>] proc_single_show+0x52/0x90
[<ffffffffbac75a50>] seq_read+0x130/0x440
[<ffffffffbac4d06f>] vfs_read+0x9f/0x170
[<ffffffffbac4dedf>] SyS_read+0x7f/0xf0
[<ffffffffbb192ed2>] system_call_fastpath+0x25/0x2a
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #330462
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff8c783815
Build id: 3.10.0-1062.18.1.el7.x86_64
0x15
2
__delayacct_blkio_ticks
vmlinux 0xffffffff8c15de86
Build id: 3.10.0-1062.18.1.el7.x86_64
0x46
3
do_task_stat
vmlinux 0xffffffff8c2ca415
Build id: 3.10.0-1062.18.1.el7.x86_64
0x725
4
? scsi_error_handler
vmlinux 0xffffffff8c4e4343
Build id: 3.10.0-1062.18.1.el7.x86_64
0xa3
5
proc_tgid_stat
vmlinux 0xffffffff8c2cb3d4
Build id: 3.10.0-1062.18.1.el7.x86_64
0x14
6
proc_single_show
vmlinux 0xffffffff8c2c55a2
Build id: 3.10.0-1062.18.1.el7.x86_64
0x52
7
seq_read
vmlinux 0xffffffff8c272a30
Build id: 3.10.0-1062.18.1.el7.x86_64
0x130
8
vfs_read
vmlinux 0xffffffff8c24a8ff
Build id: 3.10.0-1062.18.1.el7.x86_64
0x9f
9
SyS_read
vmlinux 0xffffffff8c24b7bf
Build id: 3.10.0-1062.18.1.el7.x86_64
0x7f
10
system_call_fastpath
vmlinux 0xffffffff8c78dede
Build id: 3.10.0-1062.18.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 29s! [pidof:16017]
Modules linked in: xt_CHECKSUM devlink l2tp_ppp l2tp_netlink l2tp_core pppox ppp_generic slhc drbg ansi_cprng rmd160 crypto_null tun ip_vti ip_tunnel af_key ah6 ah4 esp6 esp4 xfrm4_mode_beet xfrm4_tunnel tunnel4 xfrm4_mode_tunnel xfrm4_mode_transport xfrm6_mode_transport xfrm6_mode_ro xfrm6_mode_beet xfrm6_mode_tunnel ipcomp ipcomp6 xfrm6_tunnel xfrm_ipcomp tunnel6 cmac camellia_generic camellia_aesni_avx2 camellia_aesni_avx_x86_64 camellia_x86_64 cast6_avx_x86_64 cast6_generic cast5_avx_x86_64 cast5_generic cast_common deflate cts gcm ccm serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic blowfish_generic blowfish_x86_64 blowfish_common twofish_generic twofish_avx_x86_64 twofish_x86_64_3way xts twofish_x86_64 twofish_common xcbc sha512_ssse3 sha512_generic des_generic
ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_mangle iptable_security iptable_raw ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables vmw_vsock_vmci_transport vsock iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper vmw_balloon ablk_helper sg cryptd joydev pcspkr parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod drm_kms_helper
crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit libnvdimm drm ahci libahci ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 drm_panel_orientation_quirks vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 16017 Comm: pidof Kdump: loaded Tainted: G             L ------------   3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
task: ffff91f6348f3150 ti: ffff91f735dc0000 task.ti: ffff91f735dc0000
RIP: 0010:[<ffffffff8c783815>]  [<ffffffff8c783815>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff91f735dc3cb8  EFLAGS: 00000286
RAX: 0000000000000000 RBX: fffffffdff0a0003 RCX: 0000000000000097
RDX: 0000000000000000 RSI: 0000000000000286 RDI: 0000000000000286
RBP: ffff91f735dc3cb8 R08: 000000000000001b R09: 0000000000000020
R10: 0000000000000000 R11: ffff91f735dc3b86 R12: ffff91f69ac69000
R13: 0000000000000002 R14: ffff91f634fac1c0 R15: ffff91f63492ffff
FS:  00007fd3e1f05740(0000) GS:ffff91f73fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd3e1f1b000 CR3: 00000001381be000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8c15de86>] __delayacct_blkio_ticks+0x46/0x50
[<ffffffff8c2ca415>] do_task_stat+0x725/0xbb0
[<ffffffff8c4e4343>] ? scsi_error_handler+0xa3/0x8b0
[<ffffffff8c2cb3d4>] proc_tgid_stat+0x14/0x20
[<ffffffff8c2c55a2>] proc_single_show+0x52/0x90
[<ffffffff8c272a30>] seq_read+0x130/0x440
[<ffffffff8c24a8ff>] vfs_read+0x9f/0x170
[<ffffffff8c24b7bf>] SyS_read+0x7f/0xf0
[<ffffffff8c78dede>] system_call_fastpath+0x25/0x2a
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #312978
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff8168e70b
Build id: 3.10.0-514.21.1.el7.x86_64
0x1b
2
__delayacct_blkio_ticks
vmlinux 0xffffffff8113d556
Build id: 3.10.0-514.21.1.el7.x86_64
0x46
3
do_task_stat
vmlinux 0xffffffff812731fd
Build id: 3.10.0-514.21.1.el7.x86_64
0x70d
4
? do_wait
vmlinux 0xffffffff8108baf3
Build id: 3.10.0-514.21.1.el7.x86_64
0x1f3
5
proc_tgid_stat
vmlinux 0xffffffff81273fc4
Build id: 3.10.0-514.21.1.el7.x86_64
0x14
6
proc_single_show
vmlinux 0xffffffff8126e4cd
Build id: 3.10.0-514.21.1.el7.x86_64
0x4d
7
seq_read
vmlinux 0xffffffff81222afa
Build id: 3.10.0-514.21.1.el7.x86_64
0x10a
8
vfs_read
vmlinux 0xffffffff811fe69e
Build id: 3.10.0-514.21.1.el7.x86_64
0x9e
9
SyS_read
vmlinux 0xffffffff811ff26f
Build id: 3.10.0-514.21.1.el7.x86_64
0x7f
10
system_call_fastpath
vmlinux 0xffffffff816975c9
Build id: 3.10.0-514.21.1.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 35s! [ps:7354]
Modules linked in: rfcomm veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter vmw_vsock_vmci_transport vsock vmhgfs(OE) fuse xt_CHECKSUM dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) bnep snd_seq_midi snd_seq_midi_event intel_powerclamp btusb btrtl
iosf_mbi btbcm crc32_pclmul btintel bluetooth ghash_clmulni_intel rfkill ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon snd_ens1371 snd_rawmidi pcspkr snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd nfit soundcore sg shpchp libnvdimm vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi crct10dif_pclmul crct10dif_common vmwgfx crc32c_intel drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi scsi_transport_spi ata_piix mptscsih e1000 mptbase i2c_core libata floppy fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 7354 Comm: ps Tainted: G           OE  ------------   3.10.0-514.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8800350b1f60 ti: ffff880029c4c000 task.ti: ffff880029c4c000
RIP: 0010:[<ffffffff8168e70b>]  [<ffffffff8168e70b>] _raw_spin_unlock_irqrestore+0x1b/0x40
RSP: 0018:ffff880029c4fcd8  EFLAGS: 00000286
RAX: 0000000000000001 RBX: 0000000000001000 RCX: 0000000000001016
RDX: 0000000000000001 RSI: 0000000000000286 RDI: 0000000000000286
RBP: ffff880029c4fce0 R08: 0000000000000286 R09: 0000000000000020
R10: 0000000000000000 R11: ffff880029c4fbae R12: 0000000000000002
R13: ffff88007958ce70 R14: ffff88007959ffff R15: ffffffff81324c48
FS:  00007f424d871840(0000) GS:ffff88007c640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f424d880000 CR3: 000000003593b000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff88007958ce70 ffff880029c4fd08 ffffffff8113d556 ffff880044aae500
ffff88007958ce70 ffff8800754fa580 ffff880029c4fe40 ffffffff812731fd
ffffffffffffffff 0000000000000001 ffffffffffffffff 00000000000003ce
Call Trace:
[<ffffffff8113d556>] __delayacct_blkio_ticks+0x46/0x50
[<ffffffff812731fd>] do_task_stat+0x70d/0xb70
[<ffffffff8108baf3>] ? do_wait+0x1f3/0x250
[<ffffffff81273fc4>] proc_tgid_stat+0x14/0x20
[<ffffffff8126e4cd>] proc_single_show+0x4d/0x80
[<ffffffff81222afa>] seq_read+0x10a/0x3b0
[<ffffffff811fe69e>] vfs_read+0x9e/0x170
[<ffffffff811ff26f>] SyS_read+0x7f/0xe0
[<ffffffff816975c9>] system_call_fastpath+0x16/0x1b
Code: 08 e8 ba 07 a0 ff 5d c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 53 48 89 f3 0f 1f 44 00 00 66 83 07 02 48 89 df 57 9d <0f> 1f 44 00 00 5b 5d c3 0f 1f 44 00 00 8b 37 f0 66 83 07 02 f6