Complete report #540688
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc02e33be
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc02cf71e
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc035b2e0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffb28d1700
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffffb28dac9c
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x4cc
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffffb2743849
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffffb274390a
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffffb2747148
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffffb27475b4
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x14
|
|
10
|
_xfs_buf_ioapply
|
xfs |
0xffffffffc054b4b4
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x334
|
|
11
|
|
xfs |
0xffffffffc056d94b
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x2b
|
|
12
|
xfs_buf_submit
|
xfs |
0xffffffffc054cd4c
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x6c
|
|
13
|
xlog_bdstrat
|
xfs |
0xffffffffc056d94b
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x2b
|
|
14
|
xlog_sync
|
xfs |
0xffffffffc056f7d6
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x2e6
|
|
15
|
xlog_state_release_iclog
|
xfs |
0xffffffffc056f95b
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x7b
|
|
16
|
_xfs_log_force
|
xfs |
0xffffffffc0570ab3
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x233
|
|
17
|
|
vmlinux |
0xffffffffb24a9a50
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x130
|
|
18
|
|
xfs |
0xffffffffc057cc4c
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x16c
|
|
19
|
xfs_log_force
|
xfs |
0xffffffffc0570b6c
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x2c
|
|
20
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc057cae0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x90
|
|
21
|
xfsaild
|
xfs |
0xffffffffc057cc4c
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x16c
|
|
22
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc057cae0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x90
|
|
23
|
kthread
|
vmlinux |
0xffffffffb24c1da1
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xd1
|
|
24
|
|
vmlinux |
0xffffffffb24c1cd0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x40
|
|
25
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb2b75c37
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x21
|
|
26
|
|
vmlinux |
0xffffffffb24c1cd0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 38s! [setroubleshootd:19700]
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 38s! [xfsaild/dm-0:3371]
Modules linked in: rpcsec_gss_krb5 nfsv4 nfs fscache xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter cmac arc4 md4 nls_utf8 cifs ccm dns_resolver vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp crc32_pclmul ghash_clmulni_intel ext4 aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mbcache jbd2 joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 nfsd nfs_acl lockd grace auth_rpcgss binfmt_misc sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi
floppy vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix vmxnet3 mptspi libata scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks involflt(POE) dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3371 Comm: xfsaild/dm-0 Kdump: loaded Tainted: P OEL ------------ 3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff9f1cd461e180 ti: ffff9f1c77b00000 task.ti: ffff9f1c77b00000
RIP: 0010:[<ffffffffc02e33be>] [<ffffffffc02e33be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff9f1c77b039b0 EFLAGS: 00000246
RAX: ffffbd9643640000 RBX: ffff9f1977bc5660 RCX: 0000000000000068
RDX: ffff9f1cd5d6d400 RSI: ffff9f1cd6505000 RDI: 000000000000000e
RBP: ffff9f1c77b039c0 R08: 0000000000000003 R09: 000000081338cfff
R10: ffff9f1cd626c0d8 R11: ffff9f15cd7ebdc0 R12: 0000000100000004
R13: ffff9f15d56e9e00 R14: ffff9f1c77b03924 R15: ffffffffb274c09b
FS: 0000000000000000(0000) GS:ffff9f1cffc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f73312f3e94 CR3: 00000000b180c000 CR4: 00000000000007f0
Call Trace:
[<ffffffffc02cf71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc035b2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffb28d1700>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb28dac9c>] scsi_request_fn+0x4cc/0x680
[<ffffffffb2743849>] __blk_run_queue+0x39/0x50
[<ffffffffb274390a>] queue_unplugged+0x2a/0xa0
[<ffffffffb2747148>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffb27475b4>] blk_finish_plug+0x14/0x40
[<ffffffffc054b4b4>] _xfs_buf_ioapply+0x334/0x460 [xfs]
[<ffffffffc056d94b>] ? xlog_bdstrat+0x2b/0x60 [xfs]
[<ffffffffc054cd4c>] xfs_buf_submit+0x6c/0x220 [xfs]
[<ffffffffc056d94b>] xlog_bdstrat+0x2b/0x60 [xfs]
[<ffffffffc056f7d6>] xlog_sync+0x2e6/0x3f0 [xfs]
[<ffffffffc056f95b>] xlog_state_release_iclog+0x7b/0xd0 [xfs]
[<ffffffffc0570ab3>] _xfs_log_force+0x233/0x2c0 [xfs]
[<ffffffffb24a9a50>] ? __internal_add_timer+0x130/0x130
[<ffffffffc057cc4c>] ? xfsaild+0x16c/0x6f0 [xfs]
[<ffffffffc0570b6c>] xfs_log_force+0x2c/0x70 [xfs]
[<ffffffffc057cae0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc057cc4c>] xfsaild+0x16c/0x6f0 [xfs]
[<ffffffffc057cae0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffb24c1da1>] kthread+0xd1/0xe0
[<ffffffffb24c1cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb2b75c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb24c1cd0>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 13 2f
Complete report #66371
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_unlock_irqrestore
|
vmlinux |
0xffffffff816ab915
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x15
|
|
2
|
pvscsi_queue
|
vmw_pvscsi |
0xffffffffc0052632
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x442
|
|
3
|
|
vmlinux |
0xffffffff8146d0e0
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x170
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff8146535a
Build id: 3.10.0-693.11.1.el7.x86_64
|
0xaa
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff8146e11f
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x4df
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff812f7113
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x33
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff812f71ca
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff812fa658
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff812faa64
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x14
|
|
10
|
_xfs_buf_ioapply
|
xfs |
0xffffffffc025f354
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x334
|
|
11
|
|
xfs |
0xffffffffc0280d1b
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x2b
|
|
12
|
xfs_buf_submit
|
xfs |
0xffffffffc0260b5c
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x6c
|
|
13
|
xlog_bdstrat
|
xfs |
0xffffffffc0280d1b
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x2b
|
|
14
|
xlog_sync
|
xfs |
0xffffffffc0282c3e
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x2fe
|
|
15
|
xlog_state_release_iclog
|
xfs |
0xffffffffc0282ddb
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x7b
|
|
16
|
_xfs_log_force
|
xfs |
0xffffffffc0283d23
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x233
|
|
17
|
|
vmlinux |
0xffffffff81098b30
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x70
|
|
18
|
|
xfs |
0xffffffffc028fb3c
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x16c
|
|
19
|
xfs_log_force
|
xfs |
0xffffffffc0283ddc
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x2c
|
|
20
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc028f9d0
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x90
|
|
21
|
xfsaild
|
xfs |
0xffffffffc028fb3c
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x16c
|
|
22
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc028f9d0
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x90
|
|
23
|
kthread
|
vmlinux |
0xffffffff810b099f
Build id: 3.10.0-693.11.1.el7.x86_64
|
0xcf
|
|
24
|
|
vmlinux |
0xffffffff810b08d0
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x40
|
|
25
|
ret_from_fork
|
vmlinux |
0xffffffff816b4fd8
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x58
|
|
26
|
|
vmlinux |
0xffffffff810b08d0
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [xfsaild/dm-0:541]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon nfit pcspkr joydev sg libnvdimm parport_pc vmw_vmci shpchp parport i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx ahci drm_kms_helper libahci syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common ata_piix crc32c_intel
drm serio_raw libata vmxnet3 vmw_pvscsi i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 541 Comm: xfsaild/dm-0 Tainted: G W L ------------ 3.10.0-693.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff8809fef3af70 ti: ffff8809ff36c000 task.ti: ffff8809ff36c000
RIP: 0010:[<ffffffff816ab915>] [<ffffffff816ab915>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8809ff36fa18 EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffff8809ff36f9d8 RCX: 0000000000000600
RDX: ffffc900069a0000 RSI: 0000000000000286 RDI: 0000000000000286
RBP: ffff8809ff36fa18 R08: 0000000000000000 R09: ffff88070c368800
R10: ffff88094faf4700 R11: ffff880a03393000 R12: 0000000000000020
R13: ffff8809de757b00 R14: ffffffff8146bd09 R15: ffff8809ff36f990
FS: 0000000000000000(0000) GS:ffff880a2fd80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055f383ce7244 CR3: 00000000355a7000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8809ff36fa90 ffffffffc0052632 ffff88070c368800 ffff880035cdd000
ffff880035d122c0 ffff880a032277e8 ffffffff8146d0e0 0000000000000006
ffff880a03227754 0000000000000286 ffff88094faf4700 0000000000000000
Call Trace:
[<ffffffffc0052632>] pvscsi_queue+0x442/0x7e1 [vmw_pvscsi]
[<ffffffff8146d0e0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffff8146535a>] scsi_dispatch_cmd+0xaa/0x230
[<ffffffff8146e11f>] scsi_request_fn+0x4df/0x680
[<ffffffff812f7113>] __blk_run_queue+0x33/0x40
[<ffffffff812f71ca>] queue_unplugged+0x2a/0xa0
[<ffffffff812fa658>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff812faa64>] blk_finish_plug+0x14/0x40
[<ffffffffc025f354>] _xfs_buf_ioapply+0x334/0x460 [xfs]
[<ffffffffc0280d1b>] ? xlog_bdstrat+0x2b/0x60 [xfs]
[<ffffffffc0260b5c>] xfs_buf_submit+0x6c/0x220 [xfs]
[<ffffffffc0280d1b>] xlog_bdstrat+0x2b/0x60 [xfs]
[<ffffffffc0282c3e>] xlog_sync+0x2fe/0x420 [xfs]
[<ffffffffc0282ddb>] xlog_state_release_iclog+0x7b/0xd0 [xfs]
[<ffffffffc0283d23>] _xfs_log_force+0x233/0x2c0 [xfs]
[<ffffffff81098b30>] ? internal_add_timer+0x70/0x70
[<ffffffffc028fb3c>] ? xfsaild+0x16c/0x6f0 [xfs]
[<ffffffffc0283ddc>] xfs_log_force+0x2c/0x70 [xfs]
[<ffffffffc028f9d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc028fb3c>] xfsaild+0x16c/0x6f0 [xfs]
[<ffffffffc028f9d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48