Complete report #871323
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc023544e
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc030674e
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc020b2f0
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff87702fe0
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff8770cb8c
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff87568649
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff8756871a
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff8756c168
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_queue_bio
|
vmlinux |
0xffffffff8756c342
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x182
|
|
10
|
generic_make_request
|
vmlinux |
0xffffffff8756a3a7
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x157
|
|
11
|
submit_bio
|
vmlinux |
0xffffffff8756a650
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x70
|
|
12
|
?
__mem_cgroup_commit_charge
|
vmlinux |
0xffffffff8744bea2
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xe2
|
|
13
|
do_mpage_readpage
|
vmlinux |
0xffffffff874a3e87
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x317
|
|
14
|
|
ext4 |
0xffffffffc06dba00
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x220
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [oracle:2108]
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:37:crtc-0] flip_done timed out
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 26s! [oracle:7851]
Modules linked in: vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 nfit libnvdimm coretemp ppdev iosf_mbi crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix scsi_transport_spi drm mptscsih crct10dif_pclmul crct10dif_common libata mptbase crc32c_intel serio_raw vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 12 PID: 7851 Comm: oracle Kdump: loaded Tainted: G L ------------ 3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9c3f81884200 ti: ffff9c3f14b84000 task.ti: ffff9c3f14b84000
RIP: 0010:[<ffffffffc023544e>] [<ffffffffc023544e>] mpt_put_msg_frame+0x5e/0x90 [mptbase]
RSP: 0018:ffff9c3f14b87690 EFLAGS: 00000246
RAX: ffffbdccc1c00000 RBX: ffff9c42b91f3fe0 RCX: 0000000000000042
RDX: ffff9c42b98ac200 RSI: ffff9c413da45000 RDI: 000000000000000e
RBP: ffff9c3f14b876a0 R08: 0000000000000003 R09: 000000037c620fff
R10: ffff9c3fb7ab3800 R11: ffff9c41c1de9dc0 R12: 0000000100000080
R13: ffff9c42b8921200 R14: ffff9c3f14b87604 R15: ffffffff875713fb
FS: 00007fa1eb543740(0000) GS:ffff9c42bfc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000cfe1b000 CR3: 00000001e69e2000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc030674e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc020b2f0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff87702fe0>] scsi_dispatch_cmd+0xb0/0x250
[<ffffffff8770cb8c>] scsi_request_fn+0x4ac/0x680
[<ffffffff87568649>] __blk_run_queue+0x39/0x50
[<ffffffff8756871a>] queue_unplugged+0x2a/0xa0
[<ffffffff8756c168>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff8756c342>] blk_queue_bio+0x182/0x400
[<ffffffff8756a3a7>] generic_make_request+0x157/0x390
[<ffffffff8756a650>] submit_bio+0x70/0x150
[<ffffffff8744bea2>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
[<ffffffff874a3e87>] do_mpage_readpage+0x317/0x760
[<ffffffffc06dba00>] ? _ext4_get_block+0x220/0x220 [ext4]
Complete report #203142
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc020b3be
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc022d71e
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc02722e0
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffab4e2990
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffffab4ec0bc
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffffab3527a9
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffffab35286a
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffffab3560a8
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_queue_bio
|
vmlinux |
0xffffffffab356282
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x182
|
|
10
|
generic_make_request
|
vmlinux |
0xffffffffab354437
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x147
|
|
11
|
|
xfs |
0xffffffffc04add60
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x6d0
|
|
12
|
submit_bio
|
vmlinux |
0xffffffffab3546e0
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x70
|
|
13
|
mpage_bio_submit
|
vmlinux |
0xffffffffab292a4a
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x2a
|
|
14
|
mpage_readpages
|
vmlinux |
0xffffffffab293bef
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x13f
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 10414s! [yumBackend.py:51713]
Modules linked in: binfmt_misc ext4 mbcache jbd2 vfat fat uas usb_storage tcp_lp nls_utf8 isofs fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul vmw_balloon glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi
snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm joydev pcspkr snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif crct10dif_generic cdrom crct10dif_pclmul crct10dif_common vmwgfx crc32c_intel ata_generic pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt serio_raw fb_sys_fops ttm nfit drm ata_piix libnvdimm libata mptspi e1000 scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 51713 Comm: yumBackend.py Kdump: loaded Tainted: G L ------------ 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff933873a08000 ti: ffff933874dec000 task.ti: ffff933874dec000
RIP: 0010:[<ffffffffc020b3be>] [<ffffffffc020b3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0000:ffff933874def710 EFLAGS: 00000246
RAX: ffffbe4b00980000 RBX: ffff93387396e200 RCX: 0000000000000018
RDX: ffff9338719a2800 RSI: ffff933871994000 RDI: 000000000000000e
RBP: ffff933874def720 R08: 0000000000000003 R09: 0000000072f73fff
R10: ffff933775f93c00 R11: ffff9337ddd8aa00 R12: 0000000100000011
R13: ffff9337f2816280 R14: ffff933874def684 R15: ffffffffab35b0cb
FS: 00007f6fe5b7a740(0000) GS:ffff933879600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000189dec839928 CR3: 0000000135daa000 CR4: 00000000001007f0
Call Trace:
[<ffffffffc022d71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc02722e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffab4e2990>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffab4ec0bc>] scsi_request_fn+0x4ac/0x680
[<ffffffffab3527a9>] __blk_run_queue+0x39/0x50
[<ffffffffab35286a>] queue_unplugged+0x2a/0xa0
[<ffffffffab3560a8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffab356282>] blk_queue_bio+0x182/0x400
[<ffffffffab354437>] generic_make_request+0x147/0x380
[<ffffffffc04add60>] ? __xfs_get_blocks+0x6d0/0x6d0 [xfs]
[<ffffffffab3546e0>] submit_bio+0x70/0x150
[<ffffffffab292a4a>] mpage_bio_submit+0x2a/0x40
[<ffffffffab293bef>] mpage_readpages+0x13f/0x170
Complete report #1247117
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc01e771e
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x24e
|
|
2
|
mptspi_qcmd
|
mptspi |
0xffffffffc01792e0
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x50
|
|
3
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffb34deab0
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xb0
|
|
4
|
scsi_request_fn
|
vmlinux |
0xffffffffb34e811c
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x4cc
|
|
5
|
__blk_run_queue
|
vmlinux |
0xffffffffb334f079
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x39
|
|
6
|
queue_unplugged
|
vmlinux |
0xffffffffb334f13a
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x2a
|
|
7
|
blk_flush_plug_list
|
vmlinux |
0xffffffffb3352988
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x1d8
|
|
8
|
blk_queue_bio
|
vmlinux |
0xffffffffb3352b62
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x182
|
|
9
|
generic_make_request
|
vmlinux |
0xffffffffb3350d27
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x147
|
|
10
|
|
vmlinux |
0xffffffffb328767b
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x20b
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 35s! [ksoftirqd/0:6]
Call Trace:
[<ffffffffc01e771e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc01792e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffb34deab0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb34e811c>] scsi_request_fn+0x4cc/0x680
[<ffffffffb334f079>] __blk_run_queue+0x39/0x50
[<ffffffffb334f13a>] queue_unplugged+0x2a/0xa0
[<ffffffffb3352988>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffb3352b62>] blk_queue_bio+0x182/0x400
[<ffffffffb3350d27>] generic_make_request+0x147/0x380
[<ffffffffb328767b>] ? __bio_add_page+0x20b/0x2b0
Complete report #823140
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc03c044e
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc03e274e
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc03fa2f0
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff97d02ae0
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff97d0c68c
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff97b68159
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff97b6822a
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff97b6bc78
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_queue_bio
|
vmlinux |
0xffffffff97b6be52
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x182
|
|
10
|
generic_make_request
|
vmlinux |
0xffffffff97b69eb7
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x157
|
|
11
|
|
vmlinux |
0xffffffff97a9a76b
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x21b
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [xfsaild/dm-0:456]
Modules linked in: binfmt_misc xt_multiport ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec iosf_mbi crc32_pclmul ghash_clmulni_intel ac97_bus snd_seq aesni_intel snd_seq_device ppdev lrw gf128mul snd_pcm glue_helper ablk_helper cryptd vmw_balloon snd_timer snd joydev soundcore pcspkr sg i2c_piix4 vmw_vmci parport_pc
parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul crct10dif_common nfit drm crc32c_intel libnvdimm libata serio_raw e1000 mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 456 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G W L ------------ 3.10.0-1160.83.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9f78ef950000 ti: ffff9f78ef958000 task.ti: ffff9f78ef958000
RIP: 0010:[<ffffffffc03c044e>] [<ffffffffc03c044e>] mpt_put_msg_frame+0x5e/0x90 [mptbase]
RSP: 0018:ffff9f78ef95b940 EFLAGS: 00000246
RAX: ffffc36c006c0000 RBX: ffff9f77f6579a00 RCX: 0000000000000016
RDX: ffff9f77f566fe00 RSI: ffff9f78f6351000 RDI: 000000000000000e
RBP: ffff9f78ef95b950 R08: 0000000000000002 R09: 0000000100be41ff
R10: ffff9f77f5605400 R11: ffff9f77f4dffd40 R12: 0000000100000001
R13: ffff9f785182ba80 R14: ffff9f78ef95b8b4 R15: ffffffff97b70f0b
FS: 0000000000000000(0000) GS:ffff9f78f9600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055d11fce1b40 CR3: 00000000adaea000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc03e274e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc03fa2f0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff97d02ae0>] scsi_dispatch_cmd+0xb0/0x250
[<ffffffff97d0c68c>] scsi_request_fn+0x4ac/0x680
[<ffffffff97b68159>] __blk_run_queue+0x39/0x50
[<ffffffff97b6822a>] queue_unplugged+0x2a/0xa0
[<ffffffff97b6bc78>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff97b6be52>] blk_queue_bio+0x182/0x400
[<ffffffff97b69eb7>] generic_make_request+0x157/0x390
[<ffffffff97a9a76b>] ? __bio_add_page+0x21b/0x2c0
Complete report #608191
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc032c3be
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc036971e
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc02f12e0
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffb84e39d0
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffffb84ed0fc
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffffb8353519
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffffb83535da
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffffb8356e18
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_queue_bio
|
vmlinux |
0xffffffffb8356ff2
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x182
|
|
10
|
generic_make_request
|
vmlinux |
0xffffffffb83551a7
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x147
|
|
11
|
|
vmlinux |
0xffffffffb828b57b
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x20b
|
|
12
|
submit_bio
|
vmlinux |
0xffffffffb8355450
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x70
|
|
13
|
_xfs_buf_ioapply
|
xfs |
0xffffffffc056f283
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x2f3
|
|
14
|
?
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc05712e0
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x130
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 21s! [xfsaild/dm-0:475]
Modules linked in: iptable_filter xt_multiport sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper sg ppdev ablk_helper cryptd vmw_balloon pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm ata_piix crc32c_intel mptspi scsi_transport_spi serio_raw e1000 libata mptscsih vmxnet3 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 475 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G W L ------------ 3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff8d42043c3180 ti: ffff8d42b5740000 task.ti: ffff8d42b5740000
RIP: 0010:[<ffffffffc032c3be>] [<ffffffffc032c3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff8d42b5743940 EFLAGS: 00000246
RAX: ffff9a2f40f40000 RBX: ffff8d41f2fc7320 RCX: 0000000000000004
RDX: ffff8d42b5493600 RSI: ffff8d42b58ec000 RDI: 000000000000000e
RBP: ffff8d42b5743950 R08: 0000000000000003 R09: 00000001c5411fff
R10: ffff8d40b5d5ac00 R11: ffff8d40b5d10c40 R12: 0000000100000002
R13: ffff8d42b6f7fb00 R14: ffff8d42b57438b4 R15: ffffffffb835be3b
FS: 0000000000000000(0000) GS:ffff8d42bfd80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055f2c75d8000 CR3: 00000000ba86c000 CR4: 00000000000607e0
Call Trace:
[<ffffffffc036971e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc02f12e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffb84e39d0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb84ed0fc>] scsi_request_fn+0x4ac/0x680
[<ffffffffb8353519>] __blk_run_queue+0x39/0x50
[<ffffffffb83535da>] queue_unplugged+0x2a/0xa0
[<ffffffffb8356e18>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffb8356ff2>] blk_queue_bio+0x182/0x400
[<ffffffffb83551a7>] generic_make_request+0x147/0x380
[<ffffffffb828b57b>] ? __bio_add_page+0x20b/0x2b0
[<ffffffffb8355450>] submit_bio+0x70/0x150
[<ffffffffc056f283>] _xfs_buf_ioapply+0x2f3/0x460 [xfs]
[<ffffffffc05712e0>] ? xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
Complete report #233476
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc015f3be
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc00e671e
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc019b2e0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffa18e3fa0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffffa18ed6cc
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffffa1753939
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffffa17539fa
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffffa1757338
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_queue_bio
|
vmlinux |
0xffffffffa1757512
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x182
|
|
10
|
generic_make_request
|
vmlinux |
0xffffffffa17555c7
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x147
|
|
11
|
|
vmlinux |
0xffffffffa168b66b
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x20b
|
|
12
|
submit_bio
|
vmlinux |
0xffffffffa1755870
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x70
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [postmaster:1801]
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [xfsaild/dm-0:523]
Modules linked in: binfmt_misc ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ipt_REJECT nf_reject_ipv4 xt_limit xt_conntrack nf_conntrack vmw_vsock_vmci_transport vsock iptable_filter sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg pcspkr joydev vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea sysfillrect sysimgblt crc32c_intel fb_sys_fops ttm drm ahci serio_raw mptspi scsi_transport_spi libahci ata_piix mptscsih nfit mptbase libata vmxnet3 libnvdimm drm_panel_orientation_quirks
dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 523 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/19/2018
task: ffff8be67a838000 ti: ffff8be5f59b4000 task.ti: ffff8be5f59b4000
RIP: 0010:[<ffffffffc015f3be>] [<ffffffffc015f3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff8be5f59b7940 EFLAGS: 00000246
RAX: ffff9969806c0000 RBX: ffff8be5ce56ae40 RCX: 0000000000000039
RDX: ffff8be5f671aa00 RSI: ffff8be5f65ce000 RDI: 000000000000000e
RBP: ffff8be5f59b7950 R08: 0000000000000003 R09: 0000000011689fff
R10: ffff8be6ed31dc00 R11: ffff8be6e213dc00 R12: 0000000100000003
R13: ffff8be614e93000 R14: ffff8be5f59b78b4 R15: ffffffffa175c35b
FS: 0000000000000000(0000) GS:ffff8be6f9600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f31c5fd15b0 CR3: 0000000030610000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc00e671e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc019b2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffa18e3fa0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffa18ed6cc>] scsi_request_fn+0x4ac/0x680
[<ffffffffa1753939>] __blk_run_queue+0x39/0x50
[<ffffffffa17539fa>] queue_unplugged+0x2a/0xa0
[<ffffffffa1757338>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffa1757512>] blk_queue_bio+0x182/0x400
[<ffffffffa17555c7>] generic_make_request+0x147/0x380
[<ffffffffa168b66b>] ? __bio_add_page+0x20b/0x2b0
[<ffffffffa1755870>] submit_bio+0x70/0x150
Complete report #198299
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc02e83be
Build id: 3.10.0-1160.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc03b971e
Build id: 3.10.0-1160.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc03b22e0
Build id: 3.10.0-1160.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff828e3980
Build id: 3.10.0-1160.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff828ed0ac
Build id: 3.10.0-1160.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff827534d9
Build id: 3.10.0-1160.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff8275359a
Build id: 3.10.0-1160.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff82756dd8
Build id: 3.10.0-1160.el7.x86_64
|
0x1d8
|
|
9
|
blk_queue_bio
|
vmlinux |
0xffffffff82756fb2
Build id: 3.10.0-1160.el7.x86_64
|
0x182
|
|
10
|
generic_make_request
|
vmlinux |
0xffffffff82755167
Build id: 3.10.0-1160.el7.x86_64
|
0x147
|
|
11
|
|
vmlinux |
0xffffffff8268b53b
Build id: 3.10.0-1160.el7.x86_64
|
0x20b
|
|
12
|
submit_bio
|
vmlinux |
0xffffffff82755410
Build id: 3.10.0-1160.el7.x86_64
|
0x70
|
|
13
|
_xfs_buf_ioapply
|
xfs |
0xffffffffc057f283
Build id: 3.10.0-1160.el7.x86_64
|
0x2f3
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 25s! [xfsaild/dm-0:436]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 crc32_pclmul ghash_clmulni_intel snd_rawmidi snd_ac97_codec ppdev ac97_bus snd_seq snd_seq_device snd_pcm aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd snd_timer joydev pcspkr
snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32c_intel mptspi scsi_transport_spi ata_piix drm mptscsih nfit e1000 serio_raw libnvdimm libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 436 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9281f4be0000 ti: ffff9281f4bdc000 task.ti: ffff9281f4bdc000
RIP: 0010:[<ffffffffc02e83be>] [<ffffffffc02e83be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff9281f4bdf940 EFLAGS: 00000246
RAX: ffffb4fb00d40000 RBX: ffff9281c13c5f20 RCX: 0000000000000007
RDX: ffff928310032e00 RSI: ffff928310662000 RDI: 000000000000000e
RBP: ffff9281f4bdf950 R08: 0000000000000003 R09: 00000000aea85fff
R10: ffff9283102e8c00 R11: ffff92826ba87d40 R12: 0000000100000002
R13: ffff928208395380 R14: ffff9281f4bdf8b4 R15: ffffffff8275bdfb
FS: 0000000000000000(0000) GS:ffff92831f0c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4a4f770f00 CR3: 00000000b0832000 CR4: 0000000000340fe0
Call Trace:
[<ffffffffc03b971e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc03b22e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff828e3980>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff828ed0ac>] scsi_request_fn+0x4ac/0x680
[<ffffffff827534d9>] __blk_run_queue+0x39/0x50
[<ffffffff8275359a>] queue_unplugged+0x2a/0xa0
[<ffffffff82756dd8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff82756fb2>] blk_queue_bio+0x182/0x400
[<ffffffff82755167>] generic_make_request+0x147/0x380
[<ffffffff8268b53b>] ? __bio_add_page+0x20b/0x2b0
[<ffffffff82755410>] submit_bio+0x70/0x150
[<ffffffffc057f283>] _xfs_buf_ioapply+0x2f3/0x460 [xfs]
Complete report #125070
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc031a3be
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc030e71e
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc02e82e0
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff89ae3e70
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff89aed59c
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff89953809
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff899538ca
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff89957208
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_queue_bio
|
vmlinux |
0xffffffff899573e2
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x182
|
|
10
|
generic_make_request
|
vmlinux |
0xffffffff89955497
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x147
|
|
11
|
|
vmlinux |
0xffffffff8988b53b
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x20b
|
|
12
|
submit_bio
|
vmlinux |
0xffffffff89955740
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x70
|
|
13
|
_xfs_buf_ioapply
|
xfs |
0xffffffffc0502283
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x2f3
|
|
14
|
?
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc05042e0
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x130
|
|
15
|
__xfs_buf_submit
|
xfs |
0xffffffffc0503c72
Build id: 3.10.0-1160.21.1.el7.x86_64
|
0x72
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [xfsaild/dm-0:432]
Modules linked in: tcp_diag inet_diag iptable_filter event_collector_2_1_364486(OE) binfmt_misc talpa_vfshook(OE) talpa_pedconnector(OE) talpa_pedevice(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) vmw_vsock_vmci_transport vsock crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd pcspkr joydev sg vmw_balloon parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm ata_piix mptspi scsi_transport_spi mptscsih mptbase vmxnet3 nfit libata libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
fuse
CPU: 0 PID: 432 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff91e0f1c13180 ti: ffff91e0f1ca0000 task.ti: ffff91e0f1ca0000
RIP: 0010:[<ffffffffc031a3be>] [<ffffffffc031a3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff91e0f1ca3940 EFLAGS: 00000246
RAX: ffff9e16c1180000 RBX: ffff91e036bc5800 RCX: 000000000000000f
RDX: ffff91def5c8b800 RSI: ffff91e0f4079000 RDI: 000000000000000e
RBP: ffff91e0f1ca3950 R08: 0000000000000002 R09: 00000001747b7fff
R10: ffff91def5cedc00 R11: ffff91e0031bcc40 R12: 0000000100000001
R13: ffff91e0c8a85f80 R14: ffff91e0f1ca38b4 R15: ffffffff8995c22b
FS: 0000000000000000(0000) GS:ffff91e0f5600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f37f3004000 CR3: 0000000104e72000 CR4: 00000000000007f0
Call Trace:
[<ffffffffc030e71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc02e82e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff89ae3e70>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff89aed59c>] scsi_request_fn+0x4ac/0x680
[<ffffffff89953809>] __blk_run_queue+0x39/0x50
[<ffffffff899538ca>] queue_unplugged+0x2a/0xa0
[<ffffffff89957208>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff899573e2>] blk_queue_bio+0x182/0x400
[<ffffffff89955497>] generic_make_request+0x147/0x380
[<ffffffff8988b53b>] ? __bio_add_page+0x20b/0x2b0
[<ffffffff89955740>] submit_bio+0x70/0x150
[<ffffffffc0502283>] _xfs_buf_ioapply+0x2f3/0x460 [xfs]
[<ffffffffc05042e0>] ? xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc0503c72>] __xfs_buf_submit+0x72/0x250 [xfs]
Complete report #57815
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc02303be
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc02a971e
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc024b2e0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff862e3b90
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff862ed2bc
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff861536d9
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff8615379a
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff86156fd8
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_queue_bio
|
vmlinux |
0xffffffff861571b2
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x182
|
|
10
|
generic_make_request
|
vmlinux |
0xffffffff86155367
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x147
|
|
11
|
|
vmlinux |
0xffffffff8608b71b
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x20b
|
|
12
|
submit_bio
|
vmlinux |
0xffffffff86155610
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x70
|
|
13
|
_xfs_buf_ioapply
|
xfs |
0xffffffffc04c3283
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x2f3
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 25s! [xfsaild/dm-0:432]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_CHECKSUM xt_addrtype ipt_MASQUERADE nf_nat_masquerade_ipv4 br_netfilter tun overlay(T) devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd pcspkr joydev vmw_vmci sg i2c_piix4 parport_pc parport
ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit ata_piix drm libata libnvdimm mptspi scsi_transport_spi mptscsih crct10dif_pclmul crct10dif_common crc32c_intel mptbase e1000 serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 432 Comm: xfsaild/dm-0 Tainted: G W L ------------ T 3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff893bb5cc9080 ti: ffff893d7051c000 task.ti: ffff893d7051c000
RIP: 0010:[<ffffffffc02303be>] [<ffffffffc02303be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff893d7051f940 EFLAGS: 00000246
RAX: ffffb3f200d40000 RBX: ffff893d32812c40 RCX: 0000000000000001
RDX: ffff893d71236c00 RSI: ffff893d71951000 RDI: 000000000000000e
RBP: ffff893d7051f950 R08: 0000000000000003 R09: 00000000b40d8fff
R10: ffff893d71a83400 R11: ffff893d719e1880 R12: 0000000100000003
R13: ffff893c2c571e00 R14: ffff893d7051f8b4 R15: ffffffff8615bffb
FS: 0000000000000000(0000) GS:ffff893d766c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb407cea238 CR3: 00000001f4a10000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc02a971e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc024b2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff862e3b90>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff862ed2bc>] scsi_request_fn+0x4ac/0x680
[<ffffffff861536d9>] __blk_run_queue+0x39/0x50
[<ffffffff8615379a>] queue_unplugged+0x2a/0xa0
[<ffffffff86156fd8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff861571b2>] blk_queue_bio+0x182/0x400
[<ffffffff86155367>] generic_make_request+0x147/0x380
[<ffffffff8608b71b>] ? __bio_add_page+0x20b/0x2b0
[<ffffffff86155610>] submit_bio+0x70/0x150
[<ffffffffc04c3283>] _xfs_buf_ioapply+0x2f3/0x460 [xfs]
Complete report #20224
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc04253be
Build id: 3.10.0-1160.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc044771e
Build id: 3.10.0-1160.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc04ba2e0
Build id: 3.10.0-1160.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffb82e3980
Build id: 3.10.0-1160.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffffb82ed0ac
Build id: 3.10.0-1160.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffffb81534d9
Build id: 3.10.0-1160.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffffb815359a
Build id: 3.10.0-1160.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffffb8156dd8
Build id: 3.10.0-1160.el7.x86_64
|
0x1d8
|
|
9
|
blk_queue_bio
|
vmlinux |
0xffffffffb8156fb2
Build id: 3.10.0-1160.el7.x86_64
|
0x182
|
|
10
|
generic_make_request
|
vmlinux |
0xffffffffb8155167
Build id: 3.10.0-1160.el7.x86_64
|
0x147
|
|
11
|
|
vmlinux |
0xffffffffb808b53b
Build id: 3.10.0-1160.el7.x86_64
|
0x20b
|
|
12
|
submit_bio
|
vmlinux |
0xffffffffb8155410
Build id: 3.10.0-1160.el7.x86_64
|
0x70
|
|
13
|
_xfs_buf_ioapply
|
xfs |
0xffffffffc0725283
Build id: 3.10.0-1160.el7.x86_64
|
0x2f3
|
|
14
|
?
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc07272e0
Build id: 3.10.0-1160.el7.x86_64
|
0x130
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 24s! [xfsaild/dm-0:471]
Modules linked in: tcp_lp nls_utf8 isofs xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore coretemp sg iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper
ablk_helper cryptd joydev pcspkr vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit ata_piix libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel libata mptspi scsi_transport_spi serio_raw e1000 mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 6 PID: 471 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8ab434ebc200 ti: ffff8ab630088000 task.ti: ffff8ab630088000
RIP: 0010:[<ffffffffc04253be>] [<ffffffffc04253be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff8ab63008b940 EFLAGS: 00000246
RAX: ffffafec00f80000 RBX: ffff8ab5e166ed00 RCX: 000000000000000d
RDX: ffff8ab43543a400 RSI: ffff8ab63181b000 RDI: 000000000000000e
RBP: ffff8ab63008b950 R08: 0000000000000002 R09: 0000000035c12fff
R10: ffff8ab6325fe800 R11: ffff8ab43535ddc0 R12: 0000000100000001
R13: ffff8ab6319f0a80 R14: ffff8ab63008b8b4 R15: ffffffffb815bdfb
FS: 0000000000000000(0000) GS:ffff8ab635780000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f76d36065e6 CR3: 00000000b282c000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc044771e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc04ba2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffb82e3980>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb82ed0ac>] scsi_request_fn+0x4ac/0x680
[<ffffffffb81534d9>] __blk_run_queue+0x39/0x50
[<ffffffffb815359a>] queue_unplugged+0x2a/0xa0
[<ffffffffb8156dd8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffb8156fb2>] blk_queue_bio+0x182/0x400
[<ffffffffb8155167>] generic_make_request+0x147/0x380
[<ffffffffb808b53b>] ? __bio_add_page+0x20b/0x2b0
[<ffffffffb8155410>] submit_bio+0x70/0x150
[<ffffffffc0725283>] _xfs_buf_ioapply+0x2f3/0x460 [xfs]
[<ffffffffc07272e0>] ? xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
Complete report #14542
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc01e43be
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc027f71e
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc03362e0
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffff8aadc890
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffff8aae5efc
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x4cc
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffff8a94cf59
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffff8a94d01a
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffff8a950858
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x1d8
|
|
9
|
blk_queue_bio
|
vmlinux |
0xffffffff8a950a32
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x182
|
|
10
|
generic_make_request
|
vmlinux |
0xffffffff8a94ebf7
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x147
|
|
11
|
|
vmlinux |
0xffffffff8a88553b
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x20b
|
|
12
|
submit_bio
|
vmlinux |
0xffffffff8a94eea0
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x70
|
|
13
|
_xfs_buf_ioapply
|
xfs |
0xffffffffc047a003
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x2f3
|
|
14
|
|
xfs |
0xffffffffc047bb4a
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x1a
|
|
15
|
__xfs_buf_submit
|
xfs |
0xffffffffc047b8c1
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x61
|
|
16
|
xfs_buf_submit
|
xfs |
0xffffffffc047bb4a
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x1a
|
|
17
|
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc047bd9c
Build id: 3.10.0-1062.1.2.el7.x86_64
|
0x22c
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 28s! [xfsaild/dm-0:563]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink binfmt_misc vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus coretemp snd_seq iosf_mbi crc32_pclmul snd_seq_device snd_pcm ghash_clmulni_intel aesni_intel ppdev snd_timer snd lrw gf128mul vmw_balloon glue_helper soundcore ablk_helper cryptd sg joydev pcspkr parport_pc parport vmw_vmci i2c_piix4 sch_fq_codel ip_tables xfs libcrc32c sr_mod cdrom sd_mod ata_generic
crc_t10dif pata_acpi crct10dif_generic vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nfit ttm mptspi libnvdimm serio_raw scsi_transport_spi drm mptscsih e1000 ata_piix libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]
CPU: 3 PID: 563 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G W L ------------ T 3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9da9f5cf9070 ti: ffff9dabeffe4000 task.ti: ffff9dabeffe4000
RIP: 0010:[<ffffffffc01e43be>] [<ffffffffc01e43be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff9dabeffe7930 EFLAGS: 00000246
RAX: ffffb5b080d00000 RBX: ffff9da9e314e100 RCX: 0000000000000001
RDX: ffff9dab6b3cd000 RSI: ffff9dab3c441000 RDI: 000000000000000e
RBP: ffff9dabeffe7940 R08: 0000000000000002 R09: 0000000215ef8fff
R10: ffff9dabf00b90d8 R11: ffff9daba3b66f40 R12: 0000000100000001
R13: ffff9dabf3b3f600 R14: ffff9dabeffe78a4 R15: ffffffff8a9557ab
FS: 0000000000000000(0000) GS:ffff9dabf56c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f14b4df9cb8 CR3: 0000000180354000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc027f71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc03362e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff8aadc890>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff8aae5efc>] scsi_request_fn+0x4cc/0x680
[<ffffffff8a94cf59>] __blk_run_queue+0x39/0x50
[<ffffffff8a94d01a>] queue_unplugged+0x2a/0xa0
[<ffffffff8a950858>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff8a950a32>] blk_queue_bio+0x182/0x400
[<ffffffff8a94ebf7>] generic_make_request+0x147/0x380
[<ffffffff8a88553b>] ? __bio_add_page+0x20b/0x2b0
[<ffffffff8a94eea0>] submit_bio+0x70/0x150
[<ffffffffc047a003>] _xfs_buf_ioapply+0x2f3/0x460 [xfs]
[<ffffffffc047bb4a>] ? xfs_buf_submit+0x1a/0x40 [xfs]
[<ffffffffc047b8c1>] __xfs_buf_submit+0x61/0x210 [xfs]
[<ffffffffc047bb4a>] xfs_buf_submit+0x1a/0x40 [xfs]
[<ffffffffc047bd9c>] xfs_buf_delwri_submit_buffers+0x22c/0x240 [xfs]
Complete report #4325
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
mpt_put_msg_frame
|
mptbase |
0xffffffffc04473be
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x5e
|
|
2
|
mptscsih_qcmd
|
mptscsih |
0xffffffffc046971e
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x24e
|
|
3
|
mptspi_qcmd
|
mptspi |
0xffffffffc04622e0
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x50
|
|
4
|
scsi_dispatch_cmd
|
vmlinux |
0xffffffffaf6e2990
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xb0
|
|
5
|
scsi_request_fn
|
vmlinux |
0xffffffffaf6ec0bc
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x4ac
|
|
6
|
__blk_run_queue
|
vmlinux |
0xffffffffaf5527a9
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x39
|
|
7
|
queue_unplugged
|
vmlinux |
0xffffffffaf55286a
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x2a
|
|
8
|
blk_flush_plug_list
|
vmlinux |
0xffffffffaf5560a8
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x1d8
|
|
9
|
blk_queue_bio
|
vmlinux |
0xffffffffaf556282
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x182
|
|
10
|
generic_make_request
|
vmlinux |
0xffffffffaf554437
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x147
|
|
11
|
|
vmlinux |
0xffffffffaf48aa5b
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x20b
|
|
12
|
submit_bio
|
vmlinux |
0xffffffffaf5546e0
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x70
|
|
13
|
_xfs_buf_ioapply
|
xfs |
0xffffffffc069f2f3
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x2f3
|
|
14
|
?
xfs_buf_delwri_submit_buffers
|
xfs |
0xffffffffc06a1350
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x130
|
|
15
|
__xfs_buf_submit
|
xfs |
0xffffffffc06a0ce2
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x72
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [xfsaild/dm-0:444]
Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock overlay(T) coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg joydev vmw_balloon pcspkr parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel mptspi serio_raw libata scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 444 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G L ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/30/2013
task: ffff95a275b61070 ti: ffff95a273eb4000 task.ti: ffff95a273eb4000
RIP: 0010:[<ffffffffc04473be>] [<ffffffffc04473be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff95a273eb7940 EFLAGS: 00000246
RAX: ffffad4e80f80000 RBX: ffff95a0ea780060 RCX: 0000000000000007
RDX: ffff95a275f27400 RSI: ffff95a275f73000 RDI: 000000000000000e
RBP: ffff95a273eb7950 R08: 0000000000000003 R09: 000000002a637fff
R10: ffff95a275eb3c00 R11: ffff95a275a1a1c0 R12: 0000000100000004
R13: ffff95a159312780 R14: ffff95a273eb78b4 R15: ffffffffaf55b0cb
FS: 0000000000000000(0000) GS:ffff95a27fcc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5fcfc12200 CR3: 00000001b7364000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc046971e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc04622e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffaf6e2990>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffaf6ec0bc>] scsi_request_fn+0x4ac/0x680
[<ffffffffaf5527a9>] __blk_run_queue+0x39/0x50
[<ffffffffaf55286a>] queue_unplugged+0x2a/0xa0
[<ffffffffaf5560a8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffaf556282>] blk_queue_bio+0x182/0x400
[<ffffffffaf554437>] generic_make_request+0x147/0x380
[<ffffffffaf48aa5b>] ? __bio_add_page+0x20b/0x2b0
[<ffffffffaf5546e0>] submit_bio+0x70/0x150
[<ffffffffc069f2f3>] _xfs_buf_ioapply+0x2f3/0x460 [xfs]
[<ffffffffc06a1350>] ? xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc06a0ce2>] __xfs_buf_submit+0x72/0x250 [xfs]