Complete report #1323664
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc026244e
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc024574e
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc02172f0
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff9c103030
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff9c10cbdc
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff9bf687d9
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff9bf688aa
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff9bf6c2f8
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff9bf6c764
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc04b1979
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x1d9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc04b2200
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc04e1e70
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0xa0
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc04b2200
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc04e20fa
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x28a
|
|
15
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc04e1e70
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0xa0
|
|
16
|
kthread
|
vmlinux |
0xffffffff9bccb621
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0xd1
|
|
17
|
|
vmlinux |
0xffffffff9bccb550
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff9c3c51f7
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x21
|
|
19
|
|
vmlinux |
0xffffffff9bccb550
Build id: 3.10.0-1160.118.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 31s! [xfsaild/dm-0:456]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(E) falcon_kal(E) falcon_lsm_pinned_18108(E) falcon_lsm_pinned_18003(E) cts 8021q garp mrp ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache falcon_lsm_pinned_17804(E) nf_nat_tftp nf_conntrack_tftp nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter
overlay(T) tun vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi nfit libata scsi_transport_spi mptscsih libnvdimm mptbase crct10dif_pclmul crct10dif_common vmxnet3 crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_kal]
CPU: 2 PID: 456 Comm: xfsaild/dm-0 Kdump: loaded Tainted: P EL ------------ T 3.10.0-1160.118.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff90dcf23ee300 ti: ffff90dcf0fc0000 task.ti: ffff90dcf0fc0000
RIP: 0010:[<ffffffffc026244e>] [<ffffffffc026244e>] mpt_put_msg_frame+0x5e/0x90 [mptbase]
RSP: 0018:ffff90dcf0fc3b08 EFLAGS: 00000246
RAX: ffffa871c0e80000 RBX: ffff90db64f722b0 RCX: 0000000000000009
RDX: ffff90dcf2785a00 RSI: ffff90dcf2743000 RDI: 000000000000000e
RBP: ffff90dcf0fc3b18 R08: 0000000000000003 R09: 0000000235bcdfff
R10: ffff90daf5cddc00 R11: ffff90dcf38b88c0 R12: ffff90dc73667080
R13: ffff90dcf0fc3a7c R14: ffffffff9bf7158b R15: ffff90dcf0fc3ac0
FS: 0000000000000000(0000) GS:ffff90dcffd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000023445e000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc024574e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc02172f0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff9c103030>] scsi_dispatch_cmd+0xb0/0x250
[<ffffffff9c10cbdc>] scsi_request_fn+0x4ac/0x680
[<ffffffff9bf687d9>] __blk_run_queue+0x39/0x50
[<ffffffff9bf688aa>] queue_unplugged+0x2a/0xa0
[<ffffffff9bf6c2f8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff9bf6c764>] blk_finish_plug+0x14/0x50
[<ffffffffc04b1979>] xfs_buf_delwri_submit_buffers+0x1d9/0x250 [xfs]
[<ffffffffc04b2200>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc04e1e70>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [xfs]
[<ffffffffc04b2200>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc04e20fa>] xfsaild+0x28a/0x780 [xfs]
[<ffffffffc04e1e70>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [xfs]
[<ffffffff9bccb621>] kthread+0xd1/0xe0
[<ffffffff9bccb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff9c3c51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9bccb550>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 14 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 cc cc cc cc 48 8d 76 08 0f b7 c8 44 89 e2 48 c7
Complete report #1089701
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc01163be
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc014271e
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc01b02e0
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff9f4e3e70
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff9f4ed59c
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff9f353869
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff9f35392a
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff9f357215
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x185
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff9f3576d4
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc038b389
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x1d9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc038bbd0
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc03baa40
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc038bbd0
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc03bacca
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x28a
|
|
15
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc03baa40
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x90
|
|
16
|
kthread
|
vmlinux |
0xffffffff9f0c5e11
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xd1
|
|
17
|
|
vmlinux |
0xffffffff9f0c5d40
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff9f794df7
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x21
|
|
19
|
|
vmlinux |
0xffffffff9f0c5d40
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 29s! [xfsaild/dm-0:599]
Modules linked in: binfmt_misc resguard_linux(OE) secmodel_linux(OE) syshook_linux(OE) secmodel3(OE) xt_set ip_set_hash_net xt_NFLOG nfnetlink_log ip_set xt_CHECKSUM iptable_raw ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat iptable_mangle xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock nf_conntrack_netlink nfnetlink nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack sunrpc ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev joydev vmw_balloon pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm crct10dif_pclmul crct10dif_common crc32c_intel mptspi libata scsi_transport_spi mptscsih serio_raw mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 12 PID: 599 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G W OEL ------------ 3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8d25abbc5280 ti: ffff8d25acfb8000 task.ti: ffff8d25acfb8000
RIP: 0010:[<ffffffffc01163be>] [<ffffffffc01163be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff8d25acfbbb08 EFLAGS: 00000246
RAX: ffff9d74c6580000 RBX: ffff8d25b87a75a0 RCX: 0000000000000012
RDX: ffff8d259ffab200 RSI: ffff8d347f4f0000 RDI: 000000000000000e
RBP: ffff8d25acfbbb18 R08: 0000000000000002 R09: 0000000138fe4fff
R10: ffff8d259d79e000 R11: ffff8d24b5facfc0 R12: ffff8d27a48ded00
R13: ffff8d25acfbba7c R14: ffffffff9f35c28b R15: ffff8d25acfbbac0
FS: 0000000000000000(0000) GS:ffff8d347d900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3fd7245d80 CR3: 0000000530210000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc014271e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc01b02e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff9f4e3e70>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff9f4ed59c>] scsi_request_fn+0x4ac/0x680
[<ffffffff9f353869>] __blk_run_queue+0x39/0x50
[<ffffffff9f35392a>] queue_unplugged+0x2a/0xa0
[<ffffffff9f357215>] blk_flush_plug_list+0x185/0x230
[<ffffffff9f3576d4>] blk_finish_plug+0x14/0x40
[<ffffffffc038b389>] xfs_buf_delwri_submit_buffers+0x1d9/0x240 [xfs]
[<ffffffffc038bbd0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc03baa40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc038bbd0>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc03bacca>] xfsaild+0x28a/0x780 [xfs]
[<ffffffffc03baa40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff9f0c5e11>] kthread+0xd1/0xe0
[<ffffffff9f0c5d40>] ? insert_kthread_work+0x40/0x40
[<ffffffff9f794df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9f0c5d40>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 43 12
Complete report #649299
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_unlock_irqrestore
|
vmlinux |
0xffffffffb9b8b795
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x15
|
|
2
|
ata_scsi_queuecmd
|
libata |
0xffffffffc0435d45
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x155
|
|
3
|
|
libata |
0xffffffffc0432a20
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffb98e4860
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffffb98edf8c
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffffb9753f39
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffffb9753ffa
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffffb97578d8
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffffb9757d44
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc0662389
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x1d9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0662bd0
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0691a70
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0662bd0
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc0691cfa
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x28a
|
|
15
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0691a70
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x90
|
|
16
|
kthread
|
vmlinux |
0xffffffffb94c5e61
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0xd1
|
|
17
|
|
vmlinux |
0xffffffffb94c5d90
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb9b95df7
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x21
|
|
19
|
|
vmlinux |
0xffffffffb94c5d90
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [xfsaild/dm-0:413]
Modules linked in: tcp_lp nls_utf8 isofs nf_conntrack_netlink xt_addrtype br_netfilter vboxsf(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay(T) binfmt_misc snd_intel8x0 iosf_mbi crc32_pclmul snd_ac97_codec ghash_clmulni_intel ac97_bus joydev ppdev snd_seq snd_seq_device aesni_intel snd_pcm vboxguest(OE) lrw gf128mul glue_helper ablk_helper cryptd snd_timer pcspkr sg snd soundcore i2c_piix4 parport_pc video parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi
vmwgfx ahci drm_kms_helper syscopyarea libahci ata_piix sysfillrect sysimgblt fb_sys_fops ttm e1000 crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 413 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G W OEL ------------ T 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff93f55728e300 ti: ffff93f556b98000 task.ti: ffff93f556b98000
RIP: 0010:[<ffffffffb9b8b795>] [<ffffffffb9b8b795>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff93f556b9bba0 EFLAGS: 00010282
RAX: 0000000000000001 RBX: ffff93f4b5458000 RCX: 0000000000000000
RDX: ffff93f4b5a74018 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff93f556b9bba0 R08: ffffffffffffffff R09: 00000000d5d0cfff
R10: ffff93f5573bd400 R11: ffff93f52676e8c0 R12: ffff93f556b9bb48
R13: ffff93f4874ade00 R14: ffff93f4b5f98800 R15: ffff93f4b5fbd800
FS: 0000000000000000(0000) GS:ffff93f55f300000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002a308e02f908 CR3: 00000000cd226000 CR4: 00000000000206e0
Call Trace:
[<ffffffffc0435d45>] ata_scsi_queuecmd+0x155/0x450 [libata]
[<ffffffffc0432a20>] ? ata_scsi_invalid_field+0x50/0x50 [libata]
[<ffffffffb98e4860>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb98edf8c>] scsi_request_fn+0x4ac/0x680
[<ffffffffb9753f39>] __blk_run_queue+0x39/0x50
[<ffffffffb9753ffa>] queue_unplugged+0x2a/0xa0
[<ffffffffb97578d8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffb9757d44>] blk_finish_plug+0x14/0x40
[<ffffffffc0662389>] xfs_buf_delwri_submit_buffers+0x1d9/0x240 [xfs]
[<ffffffffc0662bd0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0691a70>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0662bd0>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0691cfa>] xfsaild+0x28a/0x780 [xfs]
[<ffffffffc0691a70>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffb94c5e61>] kthread+0xd1/0xe0
[<ffffffffb94c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffb9b95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb94c5d90>] ? insert_kthread_work+0x40/0x40
Code: 06 bd 98 ff 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 e8 e6 bc 98 ff 66 90 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #574536
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc03de3be
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc03ca71e
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc04562e0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff9bad1700
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff9badac9c
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x4cc
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff9b943849
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff9b94390a
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff9b947148
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff9b9475b4
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc062b0f9
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x1f9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc062be50
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc065aae0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc062be50
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc065ad3f
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x25f
|
|
15
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc065aae0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x90
|
|
16
|
kthread
|
vmlinux |
0xffffffff9b6c1da1
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xd1
|
|
17
|
|
vmlinux |
0xffffffff9b6c1cd0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff9bd75c37
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x21
|
|
19
|
|
vmlinux |
0xffffffff9b6c1cd0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 46s! [lftp:27895]
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 44s! [xfsaild/dm-7:10436]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 nfs fscache cmac arc4 md4 nls_utf8 cifs ccm dns_resolver vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp crc32_pclmul ghash_clmulni_intel ext4 aesni_intel mbcache lrw gf128mul jbd2 glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci nfsd nfs_acl lockd auth_rpcgss grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 ata_generic
pata_acpi floppy vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi libata scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks involflt(POE) dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 10436 Comm: xfsaild/dm-7 Kdump: loaded Tainted: P OEL ------------ 3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8af0333fe180 ti: ffff8af72e368000 task.ti: ffff8af72e368000
RIP: 0010:[<ffffffffc03de3be>] [<ffffffffc03de3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff8af72e36bb18 EFLAGS: 00000246
RAX: ffffa465436c0000 RBX: ffff8af3141afca0 RCX: 0000000000000038
RDX: ffff8af79671d000 RSI: ffff8af79ac8d000 RDI: 000000000000000e
RBP: ffff8af72e36bb28 R08: 0000000000000002 R09: 0000000114b1afff
R10: ffff8af79ac8e0d8 R11: ffff8af0bbf988c0 R12: ffff8af111650480
R13: ffff8af72e36ba8c R14: ffffffff9b94c09b R15: ffff8af72e36bad0
FS: 0000000000000000(0000) GS:ffff8af7bfd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb84653f000 CR3: 000000012d5f2000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc03ca71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc04562e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff9bad1700>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff9badac9c>] scsi_request_fn+0x4cc/0x680
[<ffffffff9b943849>] __blk_run_queue+0x39/0x50
[<ffffffff9b94390a>] queue_unplugged+0x2a/0xa0
[<ffffffff9b947148>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff9b9475b4>] blk_finish_plug+0x14/0x40
[<ffffffffc062b0f9>] xfs_buf_delwri_submit_buffers+0x1f9/0x240 [xfs]
[<ffffffffc062be50>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc065aae0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc062be50>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc065ad3f>] xfsaild+0x25f/0x6f0 [xfs]
[<ffffffffc065aae0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff9b6c1da1>] kthread+0xd1/0xe0
[<ffffffff9b6c1cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffff9bd75c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9b6c1cd0>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 c3 3e
Complete report #458522
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mptspi_qcmd
|
mptspi |
0xffffffffc00372e0
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x50
|
|
2
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffa24d1630
Build id: 3.10.0-957.12.2.el7.x86_64
|
0xb0
|
|
3
|
scsi_request_fn
|
vmlinux |
0xffffffffa24dabcc
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x4cc
|
|
4
|
__blk_run_queue
|
vmlinux |
0xffffffffa2343789
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x39
|
|
5
|
queue_unplugged
|
vmlinux |
0xffffffffa234384a
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x2a
|
|
6
|
blk_flush_plug_list
|
vmlinux |
0xffffffffa2347078
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x1d8
|
|
7
|
blk_finish_plug
|
vmlinux |
0xffffffffa23474e4
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x14
|
|
8
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc02d6369
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x1e9
|
|
9
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc02d70c0
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x10
|
|
10
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0305c60
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x90
|
|
11
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc02d70c0
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x10
|
|
12
|
xfsaild
|
xfs |
0xffffffffc0305ebf
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x25f
|
|
13
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0305c60
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x90
|
|
14
|
kthread
|
vmlinux |
0xffffffffa20c1d21
Build id: 3.10.0-957.12.2.el7.x86_64
|
0xd1
|
|
15
|
|
vmlinux |
0xffffffffa20c1c50
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x40
|
|
16
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffa2775c37
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x21
|
|
17
|
|
vmlinux |
0xffffffffa20c1c50
Build id: 3.10.0-957.12.2.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 29s! [kworker/u128:0:12459]
[<ffffffffc00372e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffa24d1630>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffa24dabcc>] scsi_request_fn+0x4cc/0x680
[<ffffffffa2343789>] __blk_run_queue+0x39/0x50
[<ffffffffa234384a>] queue_unplugged+0x2a/0xa0
[<ffffffffa2347078>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffa23474e4>] blk_finish_plug+0x14/0x40
[<ffffffffc02d6369>] xfs_buf_delwri_submit_buffers+0x1e9/0x230 [xfs]
[<ffffffffc02d70c0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0305c60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc02d70c0>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0305ebf>] xfsaild+0x25f/0x6f0 [xfs]
[<ffffffffc0305c60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffa20c1d21>] kthread+0xd1/0xe0
[<ffffffffa20c1c50>] ? insert_kthread_work+0x40/0x40
[<ffffffffa2775c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa20c1c50>] ? insert_kthread_work+0x40/0x40
Complete report #403418
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_unlock_irqrestore
|
vmlinux |
0xffffffff9e3893e5
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x15
|
|
2
|
pvscsi_queue
|
vmw_pvscsi |
0xffffffffc0080632
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x4a2
|
|
3
|
|
vmlinux |
0xffffffff9e0ec2a0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x170
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff9e0e3b90
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff9e0ed2bc
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff9df536d9
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff9df5379a
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff9df56fd8
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff9df57444
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc02f0389
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x1d9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc02f0bd0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc031fa40
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc02f0bd0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc031fcca
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x28a
|
|
15
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc031fa40
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x90
|
|
16
|
kthread
|
vmlinux |
0xffffffff9dcc5e71
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xd1
|
|
17
|
|
vmlinux |
0xffffffff9dcc5da0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff9e393ddd
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x7
|
|
19
|
|
vmlinux |
0xffffffff9dcc5da0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 25s! [xfsaild/dm-0:555]
Modules linked in: tcp_diag inet_diag rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_set nf_tables ip6t_MASQUERADE nf_nat_masquerade_ipv6 xt_comment xt_mark ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack_netlink xt_addrtype xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables veth vxlan ip6_udp_tunnel udp_tunnel ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport
ip_set_bitmap_port dummy br_netfilter tun devlink overlay(T) binfmt_misc bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev pcspkr vmw_vmci vmw_balloon parport_pc i2c_piix4 parport xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci nfit libnvdimm sd_mod libahci ata_piix crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ip_tables]
CPU: 5 PID: 555 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G W L ------------ T 3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff92f8b50db180 ti: ffff92f8b4c04000 task.ti: ffff92f8b4c04000
RIP: 0010:[<ffffffff9e3893e5>] [<ffffffff9e3893e5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff92f8b4c07b70 EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffff9dfb2523 RCX: ffff92f9f13cb060
RDX: ffffb56e00ea0000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff92f8b4c07b70 R08: 0000000000000000 R09: ffff92fa5d07d200
R10: ffff92f9efbafc00 R11: ffff92fab469d800 R12: 0000000000000000
R13: 0000000000000020 R14: ffff92f8e5430a80 R15: ffffffff9e0eaf59
FS: 0000000000000000(0000) GS:ffff92fabfd40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00017f010 CR3: 000000008be66000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffc0080632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffff9e0ec2a0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffff9e0e3b90>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff9e0ed2bc>] scsi_request_fn+0x4ac/0x680
[<ffffffff9df536d9>] __blk_run_queue+0x39/0x50
[<ffffffff9df5379a>] queue_unplugged+0x2a/0xa0
[<ffffffff9df56fd8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff9df57444>] blk_finish_plug+0x14/0x40
[<ffffffffc02f0389>] xfs_buf_delwri_submit_buffers+0x1d9/0x240 [xfs]
[<ffffffffc02f0bd0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc031fa40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc02f0bd0>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc031fcca>] xfsaild+0x28a/0x780 [xfs]
[<ffffffffc031fa40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff9dcc5e71>] kthread+0xd1/0xe0
[<ffffffff9dcc5da0>] ? insert_kthread_work+0x40/0x40
[<ffffffff9e393ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff9dcc5da0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #1216
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc01e23be
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc023171e
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc029d2e0
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff9dadeb10
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff9dae817c
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x4cc
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff9d94f0c9
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff9d94f18a
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff9d9529d8
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff9d952e44
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc0484d39
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1c9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0485930
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc04b43b0
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0485930
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc04b462a
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x27a
|
|
15
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc04b43b0
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x90
|
|
16
|
kthread
|
vmlinux |
0xffffffff9d6c61f1
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xd1
|
|
17
|
|
vmlinux |
0xffffffff9d6c6120
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff9dd8dd37
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x21
|
|
19
|
|
vmlinux |
0xffffffff9d6c6120
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [xfsaild/sda3:361]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_vs nf_conntrack vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon snd_ens1371 snd_rawmidi aesni_intel snd_ac97_codec ac97_bus lrw snd_seq gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm joydev pcspkr snd_timer sg snd vmw_vmci soundcore i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod ata_generic crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper
crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit libnvdimm mptspi serio_raw ata_piix scsi_transport_spi libata mptscsih e1000 mptbase drm_panel_orientation_quirks
CPU: 0 PID: 361 Comm: xfsaild/sda3 Kdump: loaded Tainted: G L ------------ 3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff9753b52e9070 ti: ffff9753b4b94000 task.ti: ffff9753b4b94000
RIP: 0010:[<ffffffffc01e23be>] [<ffffffffc01e23be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff9753b4b97b08 EFLAGS: 00000246
RAX: ffffae18006c0000 RBX: ffff97543b471a88 RCX: 0000000000000017
RDX: ffff9754b231b200 RSI: ffff9753b5f53000 RDI: 000000000000000e
RBP: ffff9753b4b97b18 R08: 0000000000000002 R09: 0000000131f69dff
R10: ffff9753b5f550d8 R11: ffff9753d915cfc0 R12: ffff975438b7e880
R13: ffff9753b4b97a7c R14: ffffffff9d95792b R15: ffff9753b4b97ac0
FS: 0000000000000000(0000) GS:ffff9754b9600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4c88cfa000 CR3: 000000004b97e000 CR4: 00000000000607f0
Call Trace:
[<ffffffffc023171e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc029d2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff9dadeb10>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff9dae817c>] scsi_request_fn+0x4cc/0x680
[<ffffffff9d94f0c9>] __blk_run_queue+0x39/0x50
[<ffffffff9d94f18a>] queue_unplugged+0x2a/0xa0
[<ffffffff9d9529d8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff9d952e44>] blk_finish_plug+0x14/0x40
[<ffffffffc0484d39>] xfs_buf_delwri_submit_buffers+0x1c9/0x240 [xfs]
[<ffffffffc0485930>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc04b43b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0485930>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc04b462a>] xfsaild+0x27a/0x760 [xfs]
[<ffffffffc04b43b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff9d6c61f1>] kthread+0xd1/0xe0
[<ffffffff9d6c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffff9dd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9d6c6120>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 03 1f
Complete report #322099
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
queue_work_on
|
vmlinux |
0xffffffffa96bc1ed
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x2d
|
|
2
|
pvscsi_queue
|
vmw_pvscsi |
0xffffffffc0218632
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x4a2
|
|
3
|
|
vmlinux |
0xffffffffa9aec580
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x170
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffa9ae3e70
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffffa9aed59c
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffffa9953809
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffffa99538ca
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffffa9957208
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffffa9957674
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc0450389
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x1d9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0450bd0
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc047fa40
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0450bd0
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc047fcca
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x28a
|
|
15
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc047fa40
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x90
|
|
16
|
kthread
|
vmlinux |
0xffffffffa96c5da1
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0xd1
|
|
17
|
|
vmlinux |
0xffffffffa96c5cd0
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffa9d95df7
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x21
|
|
19
|
|
vmlinux |
0xffffffffa96c5cd0
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 42s! [NetworkManager:740]
Modules linked in: talpa_vfshook(OE) talpa_pedconnector(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) talpa_pedevice(OE) tcp_diag inet_diag xt_connlimit ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 ip6table_filter nf_log_ipv4 nf_log_common nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd sg vmw_balloon pcspkr joydev parport_pc vmw_vmci parport i2c_piix4 tcp_htcp
ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw e1000 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: talpa_syscallhook]
CPU: 2 PID: 740 Comm: NetworkManager Kdump: loaded Tainted: G W OEL ------------ 3.10.0-1160.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8a51ee188000 ti: ffff8a4df66b4000 task.ti: ffff8a4df66b4000
RIP: 0010:[<ffffffffa96bc1ed>] [<ffffffffa96bc1ed>] queue_work_on+0x2d/0x50
RSP: 0000:ffff8a51ffd03e48 EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffffffffa96bbba5 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000092 RDI: 0000000000000246
RBP: ffff8a51ffd03e50 R08: ffffffffa9e1e640 R09: ffff8a51ee188000
R10: 0000000000003a2f R11: 0000000000000005 R12: ffff8a51ffd03db8
R13: ffffffffa9d96fba R14: ffff8a51ffd03e50 R15: 0000000000000246
FS: 00007f1783cc5940(0000) GS:ffff8a51ffd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0cbce309e5 CR3: 0000000035fe0000 CR4: 00000000000407e0
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 39s! [xfsaild/dm-1:434]
Modules linked in: talpa_vfshook(OE) talpa_pedconnector(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) talpa_pedevice(OE) tcp_diag inet_diag xt_connlimit ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 ip6table_filter nf_log_ipv4 nf_log_common nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd sg vmw_balloon pcspkr joydev parport_pc vmw_vmci parport i2c_piix4 tcp_htcp
ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw e1000 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: talpa_syscallhook]
CPU: 3 PID: 434 Comm: xfsaild/dm-1 Kdump: loaded Tainted: G W OEL ------------ 3.10.0-1160.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8a4f37fab180 ti: ffff8a4f3bbf0000 task.ti: ffff8a4f3bbf0000
RIP: 0010:[<ffffffffa9d8b495>] [<ffffffffa9d8b495>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8a4f3bbf3b70 EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffffa99b27a3 RCX: ffffffffaa234f40
RDX: ffffa89bc1b90000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff8a4f3bbf3b70 R08: 0000000000000001 R09: 0000000000000000
R10: ffff8a4f3b39c800 R11: ffff8a4f39380800 R12: 0000000000000000
R13: 0000000000000020 R14: ffff8a4ddef57e00 R15: ffffffffa9aeb239
FS: 0000000000000000(0000) GS:ffff8a51ffd80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fad80e9f000 CR3: 000000035564e000 CR4: 00000000000407e0
Call Trace:
[<ffffffffc0218632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffffa9aec580>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffffa9ae3e70>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffa9aed59c>] scsi_request_fn+0x4ac/0x680
[<ffffffffa9953809>] __blk_run_queue+0x39/0x50
[<ffffffffa99538ca>] queue_unplugged+0x2a/0xa0
[<ffffffffa9957208>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffa9957674>] blk_finish_plug+0x14/0x40
[<ffffffffc0450389>] xfs_buf_delwri_submit_buffers+0x1d9/0x240 [xfs]
[<ffffffffc0450bd0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc047fa40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0450bd0>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc047fcca>] xfsaild+0x28a/0x780 [xfs]
[<ffffffffc047fa40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffa96c5da1>] kthread+0xd1/0xe0
[<ffffffffa96c5cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa9d95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa96c5cd0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #273437
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_unlock_irqrestore
|
vmlinux |
0xffffffff88d8b565
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x15
|
|
2
|
pvscsi_queue
|
vmw_pvscsi |
0xffffffffc0279632
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x4a2
|
|
3
|
|
vmlinux |
0xffffffff88aec6b0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x170
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff88ae3fa0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff88aed6cc
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff88953939
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff889539fa
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff889572e5
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x185
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff889577a4
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc0545389
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x1d9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0545bd0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0574a40
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0545bd0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc0574cca
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x28a
|
|
15
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0574a40
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x90
|
|
16
|
kthread
|
vmlinux |
0xffffffff886c5da1
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xd1
|
|
17
|
|
vmlinux |
0xffffffff886c5cd0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff88d95df7
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x21
|
|
19
|
|
vmlinux |
0xffffffff886c5cd0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [xfsaild/dm-4:1125]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) vmw_vsock_vmci_transport falcon_lsm_pinned_11906(E) vsock sunrpc crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd pcspkr joydev sg parport_pc i2c_piix4 parport vmw_vmci binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix ahci nfit libahci libata libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 1125 Comm: xfsaild/dm-4 Tainted: P EL ------------ 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9f34db7a5280 ti: ffff9f34e038c000 task.ti: ffff9f34e038c000
RIP: 0010:[<ffffffff88d8b565>] [<ffffffff88d8b565>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9f34e038fb70 EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffff889b28d3 RCX: ffffffff89234f40
RDX: ffffbb05c3610000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff9f34e038fb70 R08: 0000000000000001 R09: 0000000000000000
R10: ffff9f34f03c6800 R11: ffff9f34f2798800 R12: 0000000000000000
R13: 0000000000000020 R14: ffff9f3393545080 R15: ffffffff88aeb369
FS: 0000000000000000(0000) GS:ffff9f39bfd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe9fa8fc000 CR3: 00000005c3396000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc0279632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffff88aec6b0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffff88ae3fa0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff88aed6cc>] scsi_request_fn+0x4ac/0x680
[<ffffffff88953939>] __blk_run_queue+0x39/0x50
[<ffffffff889539fa>] queue_unplugged+0x2a/0xa0
[<ffffffff889572e5>] blk_flush_plug_list+0x185/0x230
[<ffffffff889577a4>] blk_finish_plug+0x14/0x40
[<ffffffffc0545389>] xfs_buf_delwri_submit_buffers+0x1d9/0x240 [xfs]
[<ffffffffc0545bd0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0574a40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0545bd0>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0574cca>] xfsaild+0x28a/0x780 [xfs]
[<ffffffffc0574a40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff886c5da1>] kthread+0xd1/0xe0
[<ffffffff886c5cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffff88d95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff886c5cd0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48
Complete report #269253
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_unlock_irqrestore
|
vmlinux |
0xffffffffb2b1b2c5
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x15
|
|
2
|
pvscsi_queue
|
vmw_pvscsi |
0xffffffffc0309622
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x4a2
|
|
3
|
|
vmlinux |
0xffffffffb28a8500
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x170
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffb28a0010
Build id: 3.10.0-862.11.6.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffffb28a957c
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x4cc
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffffb271d4e9
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffffb271d5aa
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffffb2720d68
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffffb27211d4
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc0558e69
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x1f9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0559b20
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0587e70
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0559b20
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc05880cf
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x25f
|
|
15
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0587e70
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x90
|
|
16
|
kthread
|
vmlinux |
0xffffffffb24bdf21
Build id: 3.10.0-862.11.6.el7.x86_64
|
0xd1
|
|
17
|
|
vmlinux |
0xffffffffb24bde50
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb2b255dd
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x7
|
|
19
|
|
vmlinux |
0xffffffffb24bde50
Build id: 3.10.0-862.11.6.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 40s! [xfsaild/dm-14:7525]
Modules linked in: binfmt_misc veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter gsch(OE) redirfs(OE) event_collector_1_9_8557(POE) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon
ppdev sg pcspkr joydev parport_pc parport vmw_vmci shpchp i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect crc_t10dif sysimgblt crct10dif_generic fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw i2c_core vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 7525 Comm: xfsaild/dm-14 Kdump: loaded Tainted: P W OEL ------------ 3.10.0-862.11.6.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9536b56c8fd0 ti: ffff9536b5714000 task.ti: ffff9536b5714000
RIP: 0010:[<ffffffffb2b1b2c5>] [<ffffffffb2b1b2c5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9536b5717b80 EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffffffffb27795b3 RCX: ffff95433fe48060
RDX: ffffaf888ce20000 RSI: 0000000000000286 RDI: 0000000000000286
RBP: ffff9536b5717b80 R08: 0000000000000000 R09: ffff953fbfe57800
R10: ffff9549af16e8d8 R11: ffff9549af860800 R12: 0000000000000000
R13: 0000000000000020 R14: ffff9539ccb61500 R15: ffffffffb28a7259
FS: 0000000000000000(0000) GS:ffff95430dc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f821354cc72 CR3: 0000001816a6a000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc0309622>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffffb28a8500>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffffb28a0010>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb28a957c>] scsi_request_fn+0x4cc/0x680
[<ffffffffb271d4e9>] __blk_run_queue+0x39/0x50
[<ffffffffb271d5aa>] queue_unplugged+0x2a/0xa0
[<ffffffffb2720d68>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffb27211d4>] blk_finish_plug+0x14/0x40
[<ffffffffc0558e69>] xfs_buf_delwri_submit_buffers+0x1f9/0x240 [xfs]
[<ffffffffc0559b20>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0587e70>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0559b20>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc05880cf>] xfsaild+0x25f/0x6f0 [xfs]
[<ffffffffc0587e70>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffb24bdf21>] kthread+0xd1/0xe0
[<ffffffffb24bde50>] ? insert_kthread_work+0x40/0x40
[<ffffffffb2b255dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb24bde50>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #220891
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
scsi_request_fn
|
vmlinux |
0xffffffff9d6da6c5
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x135
|
|
2
|
__blk_run_queue
|
vmlinux |
0xffffffff9d5436b9
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x39
|
|
3
|
queue_unplugged
|
vmlinux |
0xffffffff9d54377a
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x2a
|
|
4
|
blk_flush_plug_list
|
vmlinux |
0xffffffff9d546fa8
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x1d8
|
|
5
|
blk_finish_plug
|
vmlinux |
0xffffffff9d547414
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x14
|
|
6
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc049a369
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x1e9
|
|
7
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc049b0c0
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x10
|
|
8
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc04c9c60
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x90
|
|
9
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc049b0c0
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x10
|
|
10
|
xfsaild
|
xfs |
0xffffffffc04c9ebf
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x25f
|
|
11
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc04c9c60
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x90
|
|
12
|
kthread
|
vmlinux |
0xffffffff9d2c1c71
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xd1
|
|
13
|
|
vmlinux |
0xffffffff9d2c1ba0
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x40
|
|
14
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff9d975c37
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x21
|
|
15
|
|
vmlinux |
0xffffffff9d2c1ba0
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [xfsaild/dm-0:2818]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev nfit pcspkr joydev sg libnvdimm parport_pc parport vmw_vmci vmw_balloon i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common serio_raw crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm floppy drm e1000
ata_piix mptspi scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 2818 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G L ------------ 3.10.0-957.10.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff993834461040 ti: ffff993835a30000 task.ti: ffff993835a30000
RIP: 0010:[<ffffffff9d6da6c5>] [<ffffffff9d6da6c5>] scsi_request_fn+0x135/0x680
RSP: 0018:ffff993835a33c30 EFLAGS: 00000296
RAX: 0000000000000001 RBX: 0000000000000086 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffff9938b5d1d110
RBP: ffff993835a33c88 R08: ffffffff9e2e4d38 R09: 0000000000000000
R10: ffff9938b6bcfc00 R11: ffff99383509ce00 R12: ffffffff9d2aad8c
R13: ffff993835a33be8 R14: 0000000000000086 R15: ffff9938b5d1d208
FS: 0000000000000000(0000) GS:ffff9938bcc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe35d12279c CR3: 00000000b43fa000 CR4: 00000000000407f0
Call Trace:
[<ffffffff9d5436b9>] __blk_run_queue+0x39/0x50
[<ffffffff9d54377a>] queue_unplugged+0x2a/0xa0
[<ffffffff9d546fa8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff9d547414>] blk_finish_plug+0x14/0x40
[<ffffffffc049a369>] xfs_buf_delwri_submit_buffers+0x1e9/0x230 [xfs]
[<ffffffffc049b0c0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc04c9c60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc049b0c0>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc04c9ebf>] xfsaild+0x25f/0x6f0 [xfs]
[<ffffffffc04c9c60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff9d2c1c71>] kthread+0xd1/0xe0
[<ffffffff9d2c1ba0>] ? insert_kthread_work+0x40/0x40
[<ffffffff9d975c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9d2c1ba0>] ? insert_kthread_work+0x40/0x40
Code: 8b 85 40 04 00 00 a8 02 0f 85 18 01 00 00 4c 89 ff e8 c0 c1 e6 ff 49 8b bd 58 04 00 00 c6 07 00 0f 1f 40 00 fb 66 0f 1f 44 00 00 <4d> 8b b7 00 01 00 00 4d 85 f6 0f 84 5d 04 00 00 49 8b 85 40 04
Complete report #195173
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc03a471e
Build id: 3.10.0-862.el7.x86_64
|
0x24e
|
|
2
|
mptspi_qcmd
|
mptspi |
0xffffffffc03172e0
Build id: 3.10.0-862.el7.x86_64
|
0x50
|
|
3
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffb769b050
Build id: 3.10.0-862.el7.x86_64
|
0xb0
|
|
4
|
scsi_request_fn
|
vmlinux |
0xffffffffb76a45bc
Build id: 3.10.0-862.el7.x86_64
|
0x4cc
|
|
5
|
__blk_run_queue
|
vmlinux |
0xffffffffb7518ad9
Build id: 3.10.0-862.el7.x86_64
|
0x39
|
|
6
|
queue_unplugged
|
vmlinux |
0xffffffffb7518b9a
Build id: 3.10.0-862.el7.x86_64
|
0x2a
|
|
7
|
blk_flush_plug_list
|
vmlinux |
0xffffffffb751c358
Build id: 3.10.0-862.el7.x86_64
|
0x1d8
|
|
8
|
blk_finish_plug
|
vmlinux |
0xffffffffb751c7c4
Build id: 3.10.0-862.el7.x86_64
|
0x14
|
|
9
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc058ce69
Build id: 3.10.0-862.el7.x86_64
|
0x1f9
|
|
10
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc058db20
Build id: 3.10.0-862.el7.x86_64
|
0x10
|
|
11
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc05bbe80
Build id: 3.10.0-862.el7.x86_64
|
0x90
|
|
12
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc058db20
Build id: 3.10.0-862.el7.x86_64
|
0x10
|
|
13
|
xfsaild
|
xfs |
0xffffffffc05bc0df
Build id: 3.10.0-862.el7.x86_64
|
0x25f
|
|
14
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc05bbe80
Build id: 3.10.0-862.el7.x86_64
|
0x90
|
|
15
|
kthread
|
vmlinux |
0xffffffffb72bae31
Build id: 3.10.0-862.el7.x86_64
|
0xd1
|
|
16
|
|
vmlinux |
0xffffffffb72bad60
Build id: 3.10.0-862.el7.x86_64
|
0x40
|
|
17
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb791f637
Build id: 3.10.0-862.el7.x86_64
|
0x21
|
|
18
|
|
vmlinux |
0xffffffffb72bad60
Build id: 3.10.0-862.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [kworker/0:1:15433]
Call Trace:
[<ffffffffc03a471e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc03172e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffb769b050>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb76a45bc>] scsi_request_fn+0x4cc/0x680
[<ffffffffb7518ad9>] __blk_run_queue+0x39/0x50
[<ffffffffb7518b9a>] queue_unplugged+0x2a/0xa0
[<ffffffffb751c358>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffb751c7c4>] blk_finish_plug+0x14/0x40
[<ffffffffc058ce69>] xfs_buf_delwri_submit_buffers+0x1f9/0x240 [xfs]
[<ffffffffc058db20>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc05bbe80>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc058db20>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc05bc0df>] xfsaild+0x25f/0x6f0 [xfs]
[<ffffffffc05bbe80>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffb72bae31>] kthread+0xd1/0xe0
[<ffffffffb72bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb791f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb72bad60>] ? insert_kthread_work+0x40/0x40
Complete report #185252
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_unlock_irqrestore
|
vmlinux |
0xffffffffb43883c5
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x15
|
|
2
|
pvscsi_queue
|
vmw_pvscsi |
0xffffffffc0188632
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x4a2
|
|
3
|
|
vmlinux |
0xffffffffb40eb030
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x170
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffb40e2920
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffffb40ec04c
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffffb3f52769
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffffb3f5282a
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffffb3f56068
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffffb3f564d4
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc03fe3f9
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x1d9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc03fec40
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc042d9a0
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc03fec40
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc042dc1a
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x27a
|
|
15
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc042d9a0
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x90
|
|
16
|
kthread
|
vmlinux |
0xffffffffb3cc6691
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0xd1
|
|
17
|
|
vmlinux |
0xffffffffb3cc65c0
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb4392d37
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x21
|
|
19
|
|
vmlinux |
0xffffffffb3cc65c0
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [xfsaild/dm-0:483]
Modules linked in: xt_CHECKSUM ebtable_filter ebtables ip6table_filter ip6_tables xt_dscp nfnetlink_queue nfnetlink_log bluetooth rfkill xt_multiport iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_policy xt_nat ipt_REJECT nf_reject_ipv4 xt_geoip(OE) xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables tcp_lp fuse devlink tun bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg vmw_balloon parport_pc i2c_piix4 parport vmw_vmci xfs libcrc32c ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt sd_mod fb_sys_fops crc_t10dif crct10dif_generic ttm ahci drm nfit crct10dif_pclmul
crct10dif_common crc32c_intel libnvdimm ata_piix libahci serio_raw libata vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 0 PID: 483 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1127.10.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff95f136f9e2a0 ti: ffff95f136db8000 task.ti: ffff95f136db8000
RIP: 0010:[<ffffffffb43883c5>] [<ffffffffb43883c5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff95f136dbbb70 EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffffb40e9d5e RCX: 0000000000001000
RDX: ffffba5dc0420000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff95f136dbbb70 R08: ffffffffffffffff R09: ffff95f101908800
R10: ffff95f0f5a4b000 R11: ffff95f0f59dc000 R12: 0000000000000000
R13: 0000000000000020 R14: ffff95f0e9554600 R15: ffffffffb40e9ce9
FS: 0000000000000000(0000) GS:ffff95f13fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa27cb6334c CR3: 000000001f5b8000 CR4: 00000000000607f0
Call Trace:
[<ffffffffc0188632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffffb40eb030>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffffb40e2920>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb40ec04c>] scsi_request_fn+0x4ac/0x680
[<ffffffffb3f52769>] __blk_run_queue+0x39/0x50
[<ffffffffb3f5282a>] queue_unplugged+0x2a/0xa0
[<ffffffffb3f56068>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffb3f564d4>] blk_finish_plug+0x14/0x40
[<ffffffffc03fe3f9>] xfs_buf_delwri_submit_buffers+0x1d9/0x240 [xfs]
[<ffffffffc03fec40>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc042d9a0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc03fec40>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc042dc1a>] xfsaild+0x27a/0x760 [xfs]
[<ffffffffc042d9a0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffb3cc6691>] kthread+0xd1/0xe0
[<ffffffffb3cc65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb4392d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb3cc65c0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48
Complete report #131201
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
__blk_run_queue
|
vmlinux |
0xffffffffb5152769
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0x39
|
|
2
|
queue_unplugged
|
vmlinux |
0xffffffffb515282a
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0x2a
|
|
3
|
blk_flush_plug_list
|
vmlinux |
0xffffffffb5156068
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0x1d8
|
|
4
|
blk_finish_plug
|
vmlinux |
0xffffffffb51564d4
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0x14
|
|
5
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc02fe3f9
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0x1d9
|
|
6
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc02fec40
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0x10
|
|
7
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc032d9a0
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0x90
|
|
8
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc02fec40
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0x10
|
|
9
|
xfsaild
|
xfs |
0xffffffffc032dc1a
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0x27a
|
|
10
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc032d9a0
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0x90
|
|
11
|
kthread
|
vmlinux |
0xffffffffb4ec6691
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0xd1
|
|
12
|
|
vmlinux |
0xffffffffb4ec65c0
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0x40
|
|
13
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb5592d37
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0x21
|
|
14
|
|
vmlinux |
0xffffffffb4ec65c0
Build id: 3.10.0-1127.18.2.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 29s! [kworker/1:1:54142]
[<ffffffffb5152769>] __blk_run_queue+0x39/0x50
[<ffffffffb515282a>] queue_unplugged+0x2a/0xa0
[<ffffffffb5156068>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffb51564d4>] blk_finish_plug+0x14/0x40
[<ffffffffc02fe3f9>] xfs_buf_delwri_submit_buffers+0x1d9/0x240 [xfs]
[<ffffffffc02fec40>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc032d9a0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc02fec40>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc032dc1a>] xfsaild+0x27a/0x760 [xfs]
[<ffffffffc032d9a0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffb4ec6691>] kthread+0xd1/0xe0
[<ffffffffb4ec65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb5592d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb4ec65c0>] ? insert_kthread_work+0x40/0x40
Complete report #79410
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc03bd3be
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc046b71e
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc04852e0
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff942e39d0
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff942ed0fc
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff94153519
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff941535da
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff94156e18
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff94157284
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc0639389
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x1d9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0639bd0
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0668a20
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0639bd0
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc0668caa
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x28a
|
|
15
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0668a20
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x90
|
|
16
|
kthread
|
vmlinux |
0xffffffff93ec5c21
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xd1
|
|
17
|
|
vmlinux |
0xffffffff93ec5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff94594df7
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x21
|
|
19
|
|
vmlinux |
0xffffffff93ec5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [xfsaild/dm-0:431]
Modules linked in: talpa_vfshook(OE) talpa_pedconnector(OE) talpa_pedevice(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) event_collector_2_1_364486(OE) tcp_diag inet_diag binfmt_misc xt_physdev br_netfilter ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg
joydev pcspkr vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix nfit mptspi serio_raw scsi_transport_spi mptscsih libata libnvdimm mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: talpa_syscallhook]
CPU: 1 PID: 431 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff90b835921080 ti: ffff90b8358e4000 task.ti: ffff90b8358e4000
RIP: 0010:[<ffffffffc03bd3be>] [<ffffffffc03bd3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff90b8358e7b08 EFLAGS: 00000246
RAX: ffffacf3c0d00000 RBX: ffff90b88cf85db0 RCX: 0000000000000002
RDX: ffff90ba33baa800 RSI: ffff90ba374ea000 RDI: 000000000000000e
RBP: ffff90b8358e7b18 R08: 0000000000000003 R09: 00000000b766dfff
R10: ffff90b8367c6400 R11: ffff90b915dbc000 R12: ffff90b9005b7780
R13: ffff90b8358e7a7c R14: ffffffff9415be3b R15: ffff90b8358e7ac0
FS: 0000000000000000(0000) GS:ffff90ba35640000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000c1e098 CR3: 0000000128216000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc046b71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc04852e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff942e39d0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff942ed0fc>] scsi_request_fn+0x4ac/0x680
[<ffffffff94153519>] __blk_run_queue+0x39/0x50
[<ffffffff941535da>] queue_unplugged+0x2a/0xa0
[<ffffffff94156e18>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff94157284>] blk_finish_plug+0x14/0x40
[<ffffffffc0639389>] xfs_buf_delwri_submit_buffers+0x1d9/0x240 [xfs]
[<ffffffffc0639bd0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0668a20>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0639bd0>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0668caa>] xfsaild+0x28a/0x780 [xfs]
[<ffffffffc0668a20>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff93ec5c21>] kthread+0xd1/0xe0
[<ffffffff93ec5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff94594df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff93ec5b50>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 b3 3c
Complete report #9239
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc03bd3be
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc03dd71e
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc04562e0
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff95adeab0
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff95ae811c
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x4cc
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff9594f079
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff9594f13a
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff95952988
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff95952df4
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc0641d39
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x1c9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0642930
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc06713b0
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0642930
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc067162a
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x27a
|
|
15
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc06713b0
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x90
|
|
16
|
kthread
|
vmlinux |
0xffffffff956c61f1
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xd1
|
|
17
|
|
vmlinux |
0xffffffff956c6120
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff95d8dd37
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x21
|
|
19
|
|
vmlinux |
0xffffffff956c6120
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 25s! [xfsaild/dm-0:984]
Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi kvm_intel ppdev kvm vmw_balloon irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel
lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm ata_piix libnvdimm crct10dif_pclmul crct10dif_common mptspi crc32c_intel libata scsi_transport_spi mptscsih serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 984 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G W L ------------ T 3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffffa009d859a0e0 ti: ffffa009d6eb0000 task.ti: ffffa009d6eb0000
RIP: 0010:[<ffffffffc03bd3be>] [<ffffffffc03bd3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffffa009d6eb3b08 EFLAGS: 00000246
RAX: ffffc1fd43640000 RBX: ffffa009dd4857a0 RCX: 000000000000000d
RDX: ffffa009dafb2600 RSI: ffffa009db33e000 RDI: 000000000000000e
RBP: ffffa009d6eb3b18 R08: 0000000000000002 R09: 000000081ae0bbff
R10: ffffa009ff62d0d8 R11: ffffa009d7c78fc0 R12: ffffa0089cab1e00
R13: ffffa009d6eb3a7c R14: ffffffff959578db R15: ffffa009d6eb3ac0
FS: 0000000000000000(0000) GS:ffffa009ffc40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000035b4aa11b000 CR3: 00000007ede88000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc03dd71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc04562e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff95adeab0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff95ae811c>] scsi_request_fn+0x4cc/0x680
[<ffffffff9594f079>] __blk_run_queue+0x39/0x50
[<ffffffff9594f13a>] queue_unplugged+0x2a/0xa0
[<ffffffff95952988>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff95952df4>] blk_finish_plug+0x14/0x40
[<ffffffffc0641d39>] xfs_buf_delwri_submit_buffers+0x1c9/0x240 [xfs]
[<ffffffffc0642930>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc06713b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0642930>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc067162a>] xfsaild+0x27a/0x760 [xfs]
[<ffffffffc06713b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff956c61f1>] kthread+0xd1/0xe0
[<ffffffff956c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffff95d8dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff956c6120>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 b3 3c
Complete report #7441
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_unlock_irqrestore
|
vmlinux |
0xffffffffac9883c5
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x15
|
|
2
|
pvscsi_queue
|
vmw_pvscsi |
0xffffffffc01e0632
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x4a2
|
|
3
|
|
vmlinux |
0xffffffffac6eaff0
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x170
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffac6e28e0
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffffac6ec00c
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffffac552749
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffffac55280a
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffffac556048
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffffac5564b4
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc04354b9
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x1d9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0435d00
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0464a60
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0435d00
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc0464cda
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x27a
|
|
15
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0464a60
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x90
|
|
16
|
kthread
|
vmlinux |
0xffffffffac2c6691
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0xd1
|
|
17
|
|
vmlinux |
0xffffffffac2c65c0
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffac992d37
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x21
|
|
19
|
|
vmlinux |
0xffffffffac2c65c0
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 37s! [xfsaild/dm-0:430]
Modules linked in: fuse binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel joydev lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm sd_mod nfit crc_t10dif crct10dif_generic drm ata_piix libnvdimm libata crct10dif_pclmul crct10dif_common vmxnet3 crc32c_intel serio_raw vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 430 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G L ------------ 3.10.0-1127.8.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff923c70ca41c0 ti: ffff923c719f4000 task.ti: ffff923c719f4000
RIP: 0010:[<ffffffffac9883c5>] [<ffffffffac9883c5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff923c719f7b70 EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffffac5b1433 RCX: fffffffface34e40
RDX: ffffa87800ea0000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff923c719f7b70 R08: 0000000000000001 R09: 0000000000000000
R10: ffff923c725ab000 R11: ffff923c72719800 R12: 0000000000000000
R13: 0000000000000020 R14: ffff923a414a8480 R15: ffffffffac6e9ca9
FS: 0000000000000000(0000) GS:ffff923c7fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd5352d8780 CR3: 000000023449a000 CR4: 00000000000607f0
Call Trace:
[<ffffffffc01e0632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffffac6eaff0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffffac6e28e0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffac6ec00c>] scsi_request_fn+0x4ac/0x680
[<ffffffffac552749>] __blk_run_queue+0x39/0x50
[<ffffffffac55280a>] queue_unplugged+0x2a/0xa0
[<ffffffffac556048>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffac5564b4>] blk_finish_plug+0x14/0x40
[<ffffffffc04354b9>] xfs_buf_delwri_submit_buffers+0x1d9/0x240 [xfs]
[<ffffffffc0435d00>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0464a60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0435d00>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0464cda>] xfsaild+0x27a/0x760 [xfs]
[<ffffffffc0464a60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffac2c6691>] kthread+0xd1/0xe0
[<ffffffffac2c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffac992d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffac2c65c0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48