Complete report #1111032
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc01163be
Build id: 3.10.0-1127.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc00f071e
Build id: 3.10.0-1127.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc010b2e0
Build id: 3.10.0-1127.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff8f0e2850
Build id: 3.10.0-1127.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff8f0ebefc
Build id: 3.10.0-1127.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff8ef526f9
Build id: 3.10.0-1127.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff8ef527ba
Build id: 3.10.0-1127.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff8ef55ff8
Build id: 3.10.0-1127.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff8ef56464
Build id: 3.10.0-1127.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc03224b9
Build id: 3.10.0-1127.el7.x86_64
|
0x1d9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0322d00
Build id: 3.10.0-1127.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0351a60
Build id: 3.10.0-1127.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0322d00
Build id: 3.10.0-1127.el7.x86_64
|
0x10
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 39s! [xfsaild/dm-2:701]
Modules linked in: binfmt_misc vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd sb_edac soundcore iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd sg pcspkr vmw_balloon joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper ata_piix syscopyarea sysfillrect sysimgblt fb_sys_fops ttm libata nfit drm mptspi scsi_transport_spi mptscsih mptbase crct10dif_pclmul crct10dif_common libnvdimm crc32c_intel e1000 serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 701 Comm: xfsaild/dm-2 Kdump: loaded Tainted: G W L ------------ 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff96edb2e71070 ti: ffff96ed3a19c000 task.ti: ffff96ed3a19c000
RIP: 0010:[<ffffffffc01163be>] [<ffffffffc01163be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff96ed3a19fb08 EFLAGS: 00000246
RAX: ffffbcd040a40000 RBX: ffff96eceaca3ea0 RCX: 0000000000000003
RDX: ffff96ecb66f1c00 RSI: ffff96ecb6727000 RDI: 000000000000000e
RBP: ffff96ed3a19fb18 R08: 0000000000000002 R09: 00000000ba1ae1ff
R10: ffff96edb2a22400 R11: ffff96ec8003f9c0 R12: ffff96ed15dfcf00
R13: ffff96ed3a19fa7c R14: ffffffff8ef5b01b R15: ffff96ed3a19fac0
FS: 0000000000000000(0000) GS:ffff96edb9680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00015f000 CR3: 00000000b118c000 CR4: 00000000003607e0
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 39s! [java:10373]
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 39s! [node_exporter:1697]
Call Trace:
[<ffffffffc00f071e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc010b2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff8f0e2850>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff8f0ebefc>] scsi_request_fn+0x4ac/0x660
[<ffffffff8ef526f9>] __blk_run_queue+0x39/0x50
[<ffffffff8ef527ba>] queue_unplugged+0x2a/0xa0
[<ffffffff8ef55ff8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff8ef56464>] blk_finish_plug+0x14/0x40
[<ffffffffc03224b9>] xfs_buf_delwri_submit_buffers+0x1d9/0x240 [xfs]
[<ffffffffc0322d00>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0351a60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0322d00>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
Complete report #887943
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc02f244e
Build id: 3.10.0-1160.80.1.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc031474e
Build id: 3.10.0-1160.80.1.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc032c2f0
Build id: 3.10.0-1160.80.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffa9d02a90
Build id: 3.10.0-1160.80.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffffa9d0c52c
Build id: 3.10.0-1160.80.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffffa9b68119
Build id: 3.10.0-1160.80.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffffa9b681ea
Build id: 3.10.0-1160.80.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffffa9b6bc38
Build id: 3.10.0-1160.80.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffffa9b6c0a4
Build id: 3.10.0-1160.80.1.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc052f979
Build id: 3.10.0-1160.80.1.el7.x86_64
|
0x1d9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0530200
Build id: 3.10.0-1160.80.1.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc055fe70
Build id: 3.10.0-1160.80.1.el7.x86_64
|
0xa0
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0530200
Build id: 3.10.0-1160.80.1.el7.x86_64
|
0x10
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [xfsaild/dm-1:413]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace
sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix serio_raw libata mptspi scsi_transport_spi mptscsih mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 413 Comm: xfsaild/dm-1 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.80.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9d8073243180 ti: ffff9d8073674000 task.ti: ffff9d8073674000
RIP: 0010:[<ffffffffc02f244e>] [<ffffffffc02f244e>] mpt_put_msg_frame+0x5e/0x90 [mptbase]
RSP: 0018:ffff9d8073677b08 EFLAGS: 00000246
RAX: ffffa94f40800000 RBX: ffff9d7fca1b82b0 RCX: 0000000000000016
RDX: ffff9d7f75f42e00 RSI: ffff9d80785df000 RDI: 000000000000000e
RBP: ffff9d8073677b18 R08: 0000000000000002 R09: 000000013776ffff
R10: ffff9d7f75ee8400 R11: ffff9d7fcc353880 R12: ffff9d7f5ce43c80
R13: ffff9d8073677a7c R14: ffffffffa9b70ecb R15: ffff9d8073677ac0
FS: 0000000000000000(0000) GS:ffff9d807fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fde738fe9d0 CR3: 000000009ecbc000 CR4: 00000000000407f0
Call Trace:
[<ffffffffc031474e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc032c2f0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffa9d02a90>] scsi_dispatch_cmd+0xb0/0x250
[<ffffffffa9d0c52c>] scsi_request_fn+0x4ac/0x680
[<ffffffffa9b68119>] __blk_run_queue+0x39/0x50
[<ffffffffa9b681ea>] queue_unplugged+0x2a/0xa0
[<ffffffffa9b6bc38>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffa9b6c0a4>] blk_finish_plug+0x14/0x50
[<ffffffffc052f979>] xfs_buf_delwri_submit_buffers+0x1d9/0x250 [xfs]
[<ffffffffc0530200>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc055fe70>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [xfs]
[<ffffffffc0530200>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
Complete report #792891
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc03f071e
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x24e
|
|
2
|
mptspi_qcmd
|
mptspi |
0xffffffffc03e42e0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x50
|
|
3
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffba0e3b90
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xb0
|
|
4
|
scsi_request_fn
|
vmlinux |
0xffffffffba0ed2bc
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x4ac
|
|
5
|
__blk_run_queue
|
vmlinux |
0xffffffffb9f536d9
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x39
|
|
6
|
queue_unplugged
|
vmlinux |
0xffffffffb9f5379a
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x2a
|
|
7
|
blk_flush_plug_list
|
vmlinux |
0xffffffffb9f56fd8
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x1d8
|
|
8
|
blk_finish_plug
|
vmlinux |
0xffffffffb9f57444
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x14
|
|
9
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc063b389
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x1d9
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [kworker/4:2:30152]
Call Trace:
[<ffffffffc03f071e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc03e42e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffba0e3b90>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffba0ed2bc>] scsi_request_fn+0x4ac/0x680
[<ffffffffb9f536d9>] __blk_run_queue+0x39/0x50
[<ffffffffb9f5379a>] queue_unplugged+0x2a/0xa0
[<ffffffffb9f56fd8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffb9f57444>] blk_finish_plug+0x14/0x40
[<ffffffffc063b389>] xfs_buf_delwri_submit_buffers+0x1d9/0x240 [xfs]
Complete report #572935
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc03de3be
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc03ca71e
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc04562e0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff9bad1700
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff9badac9c
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x4cc
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff9b943849
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff9b94390a
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff9b947148
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff9b9475b4
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc062b0f9
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x1f9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc062be50
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc065aae0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x90
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 38s! [kworker/1:1:26201]
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 38s! [xfsaild/dm-6:10437]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 nfs fscache cmac arc4 md4 nls_utf8 cifs ccm dns_resolver vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp crc32_pclmul ghash_clmulni_intel ext4 aesni_intel mbcache lrw gf128mul jbd2 glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci nfsd nfs_acl lockd auth_rpcgss grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 ata_generic
pata_acpi floppy vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi libata scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks involflt(POE) dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 10437 Comm: xfsaild/dm-6 Kdump: loaded Tainted: P OEL ------------ 3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8af72ef00000 ti: ffff8af733fe4000 task.ti: ffff8af733fe4000
RIP: 0010:[<ffffffffc03de3be>] [<ffffffffc03de3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff8af733fe7b18 EFLAGS: 00000246
RAX: ffffa465436c0000 RBX: ffff8af730272fd0 RCX: 0000000000000028
RDX: ffff8af79671d000 RSI: ffff8af79ac8d000 RDI: 000000000000000e
RBP: ffff8af733fe7b28 R08: 0000000000000003 R09: 00000007b42c3fff
R10: ffff8af79ac8e0d8 R11: ffff8af0a80e6380 R12: ffff8af5a7889500
R13: ffff8af733fe7a8c R14: ffffffff9b94c09b R15: ffff8af733fe7ad0
FS: 0000000000000000(0000) GS:ffff8af7bfc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe731bd8c19 CR3: 00000000b14c0000 CR4: 00000000000007f0
Call Trace:
[<ffffffffc03ca71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc04562e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff9bad1700>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff9badac9c>] scsi_request_fn+0x4cc/0x680
[<ffffffff9b943849>] __blk_run_queue+0x39/0x50
[<ffffffff9b94390a>] queue_unplugged+0x2a/0xa0
[<ffffffff9b947148>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff9b9475b4>] blk_finish_plug+0x14/0x40
[<ffffffffc062b0f9>] xfs_buf_delwri_submit_buffers+0x1f9/0x240 [xfs]
[<ffffffffc062be50>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc065aae0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Complete report #233460
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc00ba3be
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc00ec71e
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc00b32e0
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff984e3e70
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff984ed59c
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff98353869
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff9835392a
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff98357268
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff983576d4
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc0316389
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x1d9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0316bd0
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x10
|
|
12
|
?
xfs_trans_ail_cursor_first
|
xfs |
0xffffffffc0345a40
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x90
|
|
13
|
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0316bd0
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x10
|
|
14
|
xfsaild
|
xfs |
0xffffffffc0345cca
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x28a
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [xfsaild/dm-0:503]
Modules linked in: binfmt_misc ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 xt_limit xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat vmw_vsock_vmci_transport vsock nf_conntrack sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi ahci vmwgfx crct10dif_pclmul crct10dif_common ata_piix libahci crc32c_intel drm_kms_helper syscopyarea sysfillrect serio_raw sysimgblt fb_sys_fops ttm libata drm mptspi scsi_transport_spi mptscsih vmxnet3 mptbase nfit drm_panel_orientation_quirks
libnvdimm dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 503 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/19/2018
task: ffff9530fb5f3180 ti: ffff9530767f4000 task.ti: ffff9530767f4000
RIP: 0010:[<ffffffffc00ba3be>] [<ffffffffc00ba3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff9530767f7b08 EFLAGS: 00000246
RAX: ffffb12940700000 RBX: ffff9530b2f5c8b0 RCX: 0000000000000039
RDX: ffff9530767a8a00 RSI: ffff953076770000 RDI: 000000000000000e
RBP: ffff9530767f7b18 R08: 0000000000000003 R09: 000000009cea3fff
R10: ffff95316d1a2800 R11: ffff95316f255500 R12: ffff9530cb0d0180
R13: ffff9530767f7a7c R14: ffffffff9835c28b R15: ffff9530767f7ac0
FS: 0000000000000000(0000) GS:ffff953179640000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6232061c30 CR3: 00000000b365c000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc00ec71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc00b32e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff984e3e70>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff984ed59c>] scsi_request_fn+0x4ac/0x680
[<ffffffff98353869>] __blk_run_queue+0x39/0x50
[<ffffffff9835392a>] queue_unplugged+0x2a/0xa0
[<ffffffff98357268>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff983576d4>] blk_finish_plug+0x14/0x40
[<ffffffffc0316389>] xfs_buf_delwri_submit_buffers+0x1d9/0x240 [xfs]
[<ffffffffc0316bd0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0345a40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0316bd0>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0345cca>] xfsaild+0x28a/0x780 [xfs]
Complete report #200987
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc03113be
Build id: 3.10.0-957.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc033371e
Build id: 3.10.0-957.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc039f2e0
Build id: 3.10.0-957.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff8b2d0960
Build id: 3.10.0-957.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff8b2d9efc
Build id: 3.10.0-957.el7.x86_64
|
0x4cc
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff8b142ff9
Build id: 3.10.0-957.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff8b1430ba
Build id: 3.10.0-957.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff8b1468e8
Build id: 3.10.0-957.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff8b146d54
Build id: 3.10.0-957.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc056f369
Build id: 3.10.0-957.el7.x86_64
|
0x1e9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc05700c0
Build id: 3.10.0-957.el7.x86_64
|
0x10
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 71s! [xfsaild/dm-0:2388]
Modules linked in: binfmt_misc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_nat nf_conntrack_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip6table_filter ip6_tables iptable_filter nf_defrag_ipv4 iptable_mangle ip_tables ip_set nfnetlink fuse tun bridge stp llc devlink dm_snapshot dm_bufio vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd vmw_balloon sg pcspkr joydev parport_pc parport vmw_vmci i2c_piix4 xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi e1000 vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi
scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ebtables]
CPU: 1 PID: 2388 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G L ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9d2ef2c65140 ti: ffff9d2cf51f4000 task.ti: ffff9d2cf51f4000
RIP: 0010:[<ffffffffc03113be>] [<ffffffffc03113be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff9d2cf51f7b18 EFLAGS: 00000246
RAX: ffffc0d280f00000 RBX: ffff9d2cd76e0ad0 RCX: 0000000000000002
RDX: ffff9d2ef2c6b200 RSI: ffff9d2ef2fbd000 RDI: 000000000000000e
RBP: ffff9d2cf51f7b28 R08: 0000000000000003 R09: 0000000126c10fff
R10: ffff9d2ef2fbe0d8 R11: ffff9d2d77d8c8c0 R12: ffff9d2cd75cd800
R13: ffff9d2cf51f7a8c R14: ffffffff8b14b83b R15: ffff9d2cf51f7ad0
FS: 0000000000000000(0000) GS:ffff9d2effd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb8e5c0f000 CR3: 00000000350ca000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc033371e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc039f2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff8b2d0960>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff8b2d9efc>] scsi_request_fn+0x4cc/0x680
[<ffffffff8b142ff9>] __blk_run_queue+0x39/0x50
[<ffffffff8b1430ba>] queue_unplugged+0x2a/0xa0
[<ffffffff8b1468e8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff8b146d54>] blk_finish_plug+0x14/0x40
[<ffffffffc056f369>] xfs_buf_delwri_submit_buffers+0x1e9/0x230 [xfs]
[<ffffffffc05700c0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
Complete report #97227
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc00923be
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc00c971e
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc00e22e0
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff9dae3e70
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff9daed59c
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff9d953869
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff9d95392a
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff9d957268
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff9d9576d4
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc0318389
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x1d9
|
|
11
|
?
xfs_buf_delwri_submit_nowait
|
xfs |
0xffffffffc0318bd0
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x10
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [xfsaild/dm-0:432]
Modules linked in: tcp_diag inet_diag talpa_vfshook(OE) talpa_pedconnector(OE) talpa_pedevice(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) event_collector_2_1_364486(OE) binfmt_misc xt_physdev br_netfilter ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd pcspkr
sg vmw_balloon joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common libata nfit crc32c_intel libnvdimm mptspi serio_raw scsi_transport_spi mptscsih vmxnet3 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 432 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff88a532f43180 ti: ffff88a533f10000 task.ti: ffff88a533f10000
RIP: 0010:[<ffffffffc00923be>] [<ffffffffc00923be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff88a533f13b08 EFLAGS: 00000246
RAX: ffffa67b40cc0000 RBX: ffff88a3297c59b0 RCX: 0000000000000026
RDX: ffff88a533c93400 RSI: ffff88a534226000 RDI: 000000000000000e
RBP: ffff88a533f13b18 R08: 0000000000000003 R09: 00000001a546bfff
R10: ffff88a533cc8000 R11: ffff88a532e84000 R12: ffff88a42c488c00
R13: ffff88a533f13a7c R14: ffffffff9d95c28b R15: ffff88a533f13ac0
FS: 0000000000000000(0000) GS:ffff88a535680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3667239080 CR3: 000000023250e000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc00c971e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc00e22e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff9dae3e70>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff9daed59c>] scsi_request_fn+0x4ac/0x680
[<ffffffff9d953869>] __blk_run_queue+0x39/0x50
[<ffffffff9d95392a>] queue_unplugged+0x2a/0xa0
[<ffffffff9d957268>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff9d9576d4>] blk_finish_plug+0x14/0x40
[<ffffffffc0318389>] xfs_buf_delwri_submit_buffers+0x1d9/0x240 [xfs]
[<ffffffffc0318bd0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
Complete report #2761
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc02fe3be
Build id: 3.10.0-957.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc032071e
Build id: 3.10.0-957.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc03382e0
Build id: 3.10.0-957.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff9c6d0960
Build id: 3.10.0-957.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff9c6d9efc
Build id: 3.10.0-957.el7.x86_64
|
0x4cc
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff9c542ff9
Build id: 3.10.0-957.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff9c5430ba
Build id: 3.10.0-957.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff9c5468e8
Build id: 3.10.0-957.el7.x86_64
|
0x1d8
|
|
9
|
blk_finish_plug
|
vmlinux |
0xffffffff9c546d54
Build id: 3.10.0-957.el7.x86_64
|
0x14
|
|
10
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc0597369
Build id: 3.10.0-957.el7.x86_64
|
0x1e9
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 37s! [xfsaild/dm-0:3129]
{ 3}
Modules linked in:
xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep sunrpc snd_seq_midi snd_seq_midi_event coretemp snd_ens1371 snd_rawmidi snd_ac97_codec iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd vmw_balloon ac97_bus snd_seq snd_seq_device snd_pcm pcspkr btusb btrtl btbcm
snd_timer btintel snd bluetooth joydev soundcore sg rfkill parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_pclmul crct10dif_common crc32c_intel serio_raw nfit libnvdimm ata_generic pata_acpi vmwgfx e1000 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ata_piix ttm libata drm mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3129 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G L ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffffa057f5a29040 ti: ffffa05834ae0000 task.ti: ffffa05834ae0000
RIP: 0010:[<ffffffffc02fe3be>] [<ffffffffc02fe3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffffa05834ae3b18 EFLAGS: 00000246
RAX: ffffab87c0380000 RBX: ffffa057f582ffd0 RCX: 000000000000001d
RDX: ffffa057f4c04e00 RSI: ffffa057f4eb1000 RDI: 000000000000000e
RBP: ffffa05834ae3b28 R08: 0000000000000003 R09: 0000000076cf5fff
R10: ffffa057f4a9a0d8 R11: ffffa05828ec2680 R12: ffffa05834524f00
R13: ffffa05834ae3a8c R14: ffffffff9c54b83b R15: ffffa05834ae3ad0
FS: 0000000000000000(0000) GS:ffffa0583b600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f33c4dde019 CR3: 0000000055a06000 CR4: 00000000003607f0
Call Trace:
[<ffffffffc032071e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc03382e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff9c6d0960>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff9c6d9efc>] scsi_request_fn+0x4cc/0x680
[<ffffffff9c542ff9>] __blk_run_queue+0x39/0x50
[<ffffffff9c5430ba>] queue_unplugged+0x2a/0xa0
[<ffffffff9c5468e8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff9c546d54>] blk_finish_plug+0x14/0x40
[<ffffffffc0597369>] xfs_buf_delwri_submit_buffers+0x1e9/0x230 [xfs]