Info

Permalink
Function
xfs_inode_buf_verify
First occurrence
2020-09-26
Last occurrence
2025-12-23
State
NEW
Components
kernel
Quality
-32

Statistics

Operating system Count
CentOS 7 46
Architecture Count
x86_64 46
Related packages Count
kernel 46
 0:3.10.0-1160.114.2.el7 7
 0:3.10.0-1160.el7 5
 0:3.10.0-957.el7 4
 0:3.10.0-1127.19.1.el7 4
 0:3.10.0-1160.119.1.el7 4
 0:3.10.0-1160.108.1.el7 3
 0:3.10.0-1062.el7 2
 0:3.10.0-1127.18.2.el7 2
 0:3.10.0-1160.11.1.el7 2
 0:3.10.0-1160.80.1.el7 2
 0:3.10.0-1160.92.1.el7 2
 0:3.10.0-1160.21.1.el7 2
 0:3.10.0-862.el7 1
 0:3.10.0-1062.12.1.el7 1
 0:3.10.0-1062.9.1.el7 1
 0:3.10.0-1160.25.1.el7 1
 0:3.10.0-1160.49.1.el7 1
 0:3.10.0-1160.83.1.el7 1
 0:3.10.0-1160.90.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #991487
Frame # Function Binary Source or offset Line
1
xfs_inode_buf_verify
xfs 0xffffffffc0686b58
Build id: 3.10.0-1062.el7.x86_64 		0x98
2
? xfs_buf_submit
xfs 0xffffffffc0692b4a
Build id: 3.10.0-1062.el7.x86_64 		0x1a
3
xfs_inode_buf_write_verify
xfs 0xffffffffc0686bd0
Build id: 3.10.0-1062.el7.x86_64 		0x10
4
_xfs_buf_ioapply
xfs 0xffffffffc0690da7
Build id: 3.10.0-1062.el7.x86_64 		0x97
5
? wake_up_state
vmlinux 0xffffffffa96da0b0
Build id: 3.10.0-1062.el7.x86_64 		0x20
6
? xfs_buf_submit
xfs 0xffffffffc0692b4a
Build id: 3.10.0-1062.el7.x86_64 		0x1a
7
__xfs_buf_submit
xfs 0xffffffffc06928c1
Build id: 3.10.0-1062.el7.x86_64 		0x61
8
xfs_buf_submit
xfs 0xffffffffc0692b4a
Build id: 3.10.0-1062.el7.x86_64 		0x1a
9
xfs_buf_delwri_submit_buffers
xfs 0xffffffffc0692d9c
Build id: 3.10.0-1062.el7.x86_64 		0x22c
10
? xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0693930
Build id: 3.10.0-1062.el7.x86_64 		0x10
11
? xfs_trans_ail_cursor_first
xfs 0xffffffffc06c23d0
Build id: 3.10.0-1062.el7.x86_64 		0x90
12
xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0693930
Build id: 3.10.0-1062.el7.x86_64 		0x10
13
xfsaild
xfs 0xffffffffc06c264a
Build id: 3.10.0-1062.el7.x86_64 		0x27a
14
? xfs_trans_ail_cursor_first
xfs 0xffffffffc06c23d0
Build id: 3.10.0-1062.el7.x86_64 		0x90
15
kthread
vmlinux 0xffffffffa96c50d1
Build id: 3.10.0-1062.el7.x86_64 		0xd1
16
? insert_kthread_work
vmlinux 0xffffffffa96c5000
Build id: 3.10.0-1062.el7.x86_64 		0x40
17
ret_from_fork_nospec_begin
vmlinux 0xffffffffa9d8bd37
Build id: 3.10.0-1062.el7.x86_64 		0x21
18
? insert_kthread_work
vmlinux 0xffffffffa96c5000
Build id: 3.10.0-1062.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 24s! [xfsaild/dm-0:819]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun bridge stp llc devlink nf_nat_ftp nf_conntrack_ftp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ktap_113097(OE) ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc squashfs loop ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg joydev vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c
sr_mod cdrom sd_mod vmwgfx crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ata_generic pata_acpi ttm ahci drm libahci nfit ata_piix libnvdimm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmw_pvscsi vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 819 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff97f968e0a0e0 ti: ffff97f966f78000 task.ti: ffff97f966f78000
RIP: 0010:[<ffffffffc0686b58>]  [<ffffffffc0686b58>] xfs_inode_buf_verify+0x98/0x100 [xfs]
RSP: 0018:ffff97f966f7bc18  EFLAGS: 00010286
RAX: ffff97f965278000 RBX: 0000000000000001 RCX: 0000000000000009
RDX: ffff97f965278000 RSI: 0000000000001000 RDI: ffff97f845984900
RBP: ffff97f966f7bc40 R08: ffff97f845984980 R09: 0000000000000000
R10: ffff97f951ff6918 R11: 0000000413f49000 R12: ffff97f9697a8000
R13: 00000000017aa7c2 R14: ffff97f9576b6180 R15: 0000000566f7bba0
FS:  0000000000000000(0000) GS:ffff97f97fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f588f22d871 CR3: 0000000034ad8000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffc0692b4a>] ? xfs_buf_submit+0x1a/0x40 [xfs]
[<ffffffffc0686bd0>] xfs_inode_buf_write_verify+0x10/0x20 [xfs]
[<ffffffffc0690da7>] _xfs_buf_ioapply+0x97/0x460 [xfs]
[<ffffffffa96da0b0>] ? wake_up_state+0x20/0x20
[<ffffffffc0692b4a>] ? xfs_buf_submit+0x1a/0x40 [xfs]
[<ffffffffc06928c1>] __xfs_buf_submit+0x61/0x210 [xfs]
[<ffffffffc0692b4a>] xfs_buf_submit+0x1a/0x40 [xfs]
[<ffffffffc0692d9c>] xfs_buf_delwri_submit_buffers+0x22c/0x240 [xfs]
[<ffffffffc0693930>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc06c23d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0693930>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc06c264a>] xfsaild+0x27a/0x760 [xfs]
[<ffffffffc06c23d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffa96c50d1>] kthread+0xd1/0xe0
[<ffffffffa96c5000>] ? insert_kthread_work+0x40/0x40
[<ffffffffa9d8bd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa96c5000>] ? insert_kthread_work+0x40/0x40
Code: 3f bf 00 00 4c 89 f7 e8 b7 e7 00 00 44 39 eb 74 49 66 90 41 0f b6 8c 24 92 00 00 00 89 de 4c 89 f7 d3 e6 48 63 f6 e8 f8 c5 00 00 <66> 81 38 49 4e 75 b9 0f b6 50 04 41 0f b7 44 24 7c 83 e0 0f 66
Complete report #719358
Frame # Function Binary Source or offset Line
1
xfs_inode_buf_verify
xfs 0xffffffffc0298ca2
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x82
2
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc02a58d0
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x130
3
xfs_inode_buf_write_verify
xfs 0xffffffffc0298da0
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x10
4
_xfs_buf_ioapply
xfs 0xffffffffc02a3577
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x97
5
? wake_up_state
vmlinux 0xffffffffb9ce1170
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x20
6
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc02a58d0
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x130
7
__xfs_buf_submit
xfs 0xffffffffc02a5262
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x72
8
xfs_buf_delwri_submit_buffers
xfs 0xffffffffc02a58d0
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x130
9
? xfs_buf_delwri_submit_nowait
xfs 0xffffffffc02a6200
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x10
10
? xfs_trans_ail_cursor_first
xfs 0xffffffffc02d5e70
Build id: 3.10.0-1160.90.1.el7.x86_64 		0xa0
11
xfs_buf_delwri_submit_nowait
xfs 0xffffffffc02a6200
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x10
12
xfsaild
xfs 0xffffffffc02d60fa
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x28a
13
? xfs_trans_ail_cursor_first
xfs 0xffffffffc02d5e70
Build id: 3.10.0-1160.90.1.el7.x86_64 		0xa0
14
kthread
vmlinux 0xffffffffb9ccb621
Build id: 3.10.0-1160.90.1.el7.x86_64 		0xd1
15
? insert_kthread_work
vmlinux 0xffffffffb9ccb550
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffffba3c51f7
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x21
17
? insert_kthread_work
vmlinux 0xffffffffb9ccb550
Build id: 3.10.0-1160.90.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 25s! [xfsaild/dm-1:408]
Modules linked in: veth falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_15003(E) xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_addrtype iptable_filter br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay joydev snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore i2c_piix4 virtio_rng virtio_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ip_tables xfs libcrc32c ata_generic pata_acpi virtio_blk virtio_console qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
ttm drm ata_piix crct10dif_pclmul e1000 crct10dif_common crc32c_intel serio_raw virtio_pci libata virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 408 Comm: xfsaild/dm-1 Tainted: P            EL ------------   3.10.0-1160.90.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.15.0-2.module+el8.6.0+20659+3dcf7c70 04/01/2014
task: ffff8aeeb5871080 ti: ffff8aefb2ef8000 task.ti: ffff8aefb2ef8000
RIP: 0010:[<ffffffffc0298ca2>]  [<ffffffffc0298ca2>] xfs_inode_buf_verify+0x82/0x170 [xfs]
RSP: 0018:ffff8aefb2efbc18  EFLAGS: 00010286
RAX: ffff8aefb4736000 RBX: ffff8aeec8f7e918 RCX: 0000000000000008
RDX: ffff8aefb4736000 RSI: 0000000000000000 RDI: ffff8aef38c5b800
RBP: ffff8aefb2efbc50 R08: ffff8aef38c5b880 R09: 0000000000000000
R10: 0000000000000005 R11: ffff8aeeb67f4408 R12: 00000000c8cfd638
R13: ffff8aeeb5c78400 R14: ffff8aefbfca4980 R15: 0000000000000004
FS:  0000000000000000(0000) GS:ffff8aefbfc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0f91d28000 CR3: 000000013ce10000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc02a58d0>] ? xfs_buf_delwri_submit_buffers+0x130/0x250 [xfs]
[<ffffffffc0298da0>] xfs_inode_buf_write_verify+0x10/0x20 [xfs]
[<ffffffffc02a3577>] _xfs_buf_ioapply+0x97/0x460 [xfs]
[<ffffffffb9ce1170>] ? wake_up_state+0x20/0x20
[<ffffffffc02a58d0>] ? xfs_buf_delwri_submit_buffers+0x130/0x250 [xfs]
[<ffffffffc02a5262>] __xfs_buf_submit+0x72/0x250 [xfs]
[<ffffffffc02a58d0>] xfs_buf_delwri_submit_buffers+0x130/0x250 [xfs]
[<ffffffffc02a6200>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc02d5e70>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [xfs]
[<ffffffffc02a6200>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc02d60fa>] xfsaild+0x28a/0x780 [xfs]
[<ffffffffc02d5e70>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [xfs]
[<ffffffffb9ccb621>] kthread+0xd1/0xe0
[<ffffffffb9ccb550>] ? insert_kthread_work+0x40/0x40
[<ffffffffba3c51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb9ccb550>] ? insert_kthread_work+0x40/0x40
Code: ff 0f b7 83 82 00 00 00 41 d3 ff 44 0f af f8 45 85 ff 7e 48 0f b6 8b 92 00 00 00 44 89 e6 4c 89 ef d3 e6 48 63 f6 e8 4e cd 00 00 <66> 81 38 49 4e 8b 50 60 48 89 c1 0f ca 74 3f 45 84 f6 0f 84 a6
Complete report #633126
Frame # Function Binary Source or offset Line
1
xfs_inode_buf_verify
xfs 0xffffffffc0509ca2
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x82
2
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc05168d0
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x130
3
xfs_inode_buf_write_verify
xfs 0xffffffffc0509da0
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x10
4
_xfs_buf_ioapply
xfs 0xffffffffc0514577
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x97
5
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc05168d0
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x130
6
__xfs_buf_submit
xfs 0xffffffffc0516262
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x72
7
xfs_buf_delwri_submit_buffers
xfs 0xffffffffc05168d0
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x130
8
? xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0517200
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x10
9
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0546e70
Build id: 3.10.0-1160.83.1.el7.x86_64 		0xa0
10
xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0517200
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x10
11
xfsaild
xfs 0xffffffffc05470fa
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x28a
12
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0546e70
Build id: 3.10.0-1160.83.1.el7.x86_64 		0xa0
13
kthread
vmlinux 0xffffffffb9acb511
Build id: 3.10.0-1160.83.1.el7.x86_64 		0xd1
14
? insert_kthread_work
vmlinux 0xffffffffb9acb440
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x40
15
ret_from_fork_nospec_begin
vmlinux 0xffffffffba1c51dd
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x7
16
? insert_kthread_work
vmlinux 0xffffffffb9acb440
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [xfsaild/dm-0:403]
Modules linked in: binfmt_misc vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c vmwgfx drm_kms_helper sr_mod cdrom syscopyarea ata_generic pata_acpi sysfillrect sysimgblt fb_sys_fops ttm drm sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common ata_piix crc32c_intel nfit serio_raw libata vmxnet3 libnvdimm vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 403 Comm: xfsaild/dm-0 Tainted: G             L ------------   3.10.0-1160.83.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffff9a4d36e7d280 ti: ffff9a4d35b7c000 task.ti: ffff9a4d35b7c000
RIP: 0010:[<ffffffffc0509ca2>]  [<ffffffffc0509ca2>] xfs_inode_buf_verify+0x82/0x170 [xfs]
RSP: 0000:ffff9a4d35b7fc18  EFLAGS: 00010286
RAX: ffff9a4d385bd000 RBX: ffff9a4d361cce00 RCX: 0000000000000009
RDX: ffff9a4d385bd000 RSI: 0000000000000000 RDI: ffff9a4d05cf5380
RBP: ffff9a4d35b7fc50 R08: ffff9a4d05cf5400 R09: 0000000000000000
R10: ffff9a4d369a3ef8 R11: 0000000035200000 R12: ffff9a4d3606e880
R13: 0000000b352ab180 R14: ffff9a4d38f36818 R15: ffffffffb9d6be7a
FS:  0000000000000000(0000) GS:ffff9a4d3c600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdfc8f25ff0 CR3: 000000003821e000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc05168d0>] ? xfs_buf_delwri_submit_buffers+0x130/0x250 [xfs]
[<ffffffffc0509da0>] xfs_inode_buf_write_verify+0x10/0x20 [xfs]
[<ffffffffc0514577>] _xfs_buf_ioapply+0x97/0x460 [xfs]
[<ffffffffc05168d0>] ? xfs_buf_delwri_submit_buffers+0x130/0x250 [xfs]
[<ffffffffc0516262>] __xfs_buf_submit+0x72/0x250 [xfs]
[<ffffffffc05168d0>] xfs_buf_delwri_submit_buffers+0x130/0x250 [xfs]
[<ffffffffc0517200>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0546e70>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [xfs]
[<ffffffffc0517200>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc05470fa>] xfsaild+0x28a/0x780 [xfs]
[<ffffffffc0546e70>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [xfs]
[<ffffffffb9acb511>] kthread+0xd1/0xe0
[<ffffffffb9acb440>] ? insert_kthread_work+0x40/0x40
[<ffffffffba1c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb9acb440>] ? insert_kthread_work+0x40/0x40
Code: ff 0f b7 83 82 00 00 00 41 d3 ff 44 0f af f8 45 85 ff 7e 48 0f b6 8b 92 00 00 00 44 89 e6 4c 89 ef d3 e6 48 63 f6 e8 4e cd 00 00 <66> 81 38 49 4e 8b 50 60 48 89 c1 0f ca 74 3f 45 84 f6 0f 84 a6
Complete report #329632
Frame # Function Binary Source or offset Line
1
xfs_inode_buf_verify
xfs 0xffffffffc0443a72
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x82
2
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc04502e0
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x130
3
xfs_inode_buf_write_verify
xfs 0xffffffffc0443b60
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x10
4
_xfs_buf_ioapply
xfs 0xffffffffc044e027
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x97
5
? wake_up_state
vmlinux 0xffffffffa96db0c0
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x20
6
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc04502e0
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x130
7
__xfs_buf_submit
xfs 0xffffffffc044fc72
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x72
8
xfs_buf_delwri_submit_buffers
xfs 0xffffffffc04502e0
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x130
9
? xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0450bd0
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x10
10
? xfs_trans_ail_cursor_first
xfs 0xffffffffc047fa40
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x90
11
xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0450bd0
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x10
12
xfsaild
xfs 0xffffffffc047fcca
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x28a
13
? xfs_trans_ail_cursor_first
xfs 0xffffffffc047fa40
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x90
14
kthread
vmlinux 0xffffffffa96c5da1
Build id: 3.10.0-1160.21.1.el7.x86_64 		0xd1
15
? insert_kthread_work
vmlinux 0xffffffffa96c5cd0
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffffa9d95df7
Build id: 3.10.0-1160.21.1.el7.x86_64 		0x21 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [mysqld:7709]
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [xfsaild/dm-1:434]
Modules linked in: talpa_vfshook(OE) talpa_pedconnector(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) talpa_pedevice(OE) tcp_diag inet_diag xt_connlimit ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 ip6table_filter nf_log_ipv4 nf_log_common nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd sg vmw_balloon pcspkr joydev parport_pc vmw_vmci parport i2c_piix4 tcp_htcp
ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw e1000 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: talpa_syscallhook]
CPU: 2 PID: 434 Comm: xfsaild/dm-1 Kdump: loaded Tainted: G        W  OEL ------------   3.10.0-1160.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8a4f37fab180 ti: ffff8a4f3bbf0000 task.ti: ffff8a4f3bbf0000
RIP: 0010:[<ffffffffc0443a72>]  [<ffffffffc0443a72>] xfs_inode_buf_verify+0x82/0x160 [xfs]
RSP: 0018:ffff8a4f3bbf3c18  EFLAGS: 00010286
RAX: ffff8a50dee99000 RBX: ffff8a4f3c3e8000 RCX: 0000000000000008
RDX: ffff8a50dee99000 RSI: 0000000000001000 RDI: ffff8a513baf8900
RBP: ffff8a4f3bbf3c50 R08: ffff8a513baf8980 R09: 0000000000000000
R10: ffff8a504515a718 R11: 00000001515a3000 R12: ffff8a51c2d53080
R13: 0000000e59fe1080 R14: ffff8a504515a718 R15: ffffffffa995740a
FS:  0000000000000000(0000) GS:ffff8a51ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd9e6c4a000 CR3: 00000001f6242000 CR4: 00000000000407e0
Call Trace:
[<ffffffffc04502e0>] ? xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc0443b60>] xfs_inode_buf_write_verify+0x10/0x20 [xfs]
[<ffffffffc044e027>] _xfs_buf_ioapply+0x97/0x460 [xfs]
[<ffffffffa96db0c0>] ? wake_up_state+0x20/0x20
[<ffffffffc04502e0>] ? xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc044fc72>] __xfs_buf_submit+0x72/0x250 [xfs]
[<ffffffffc04502e0>] xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc0450bd0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc047fa40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0450bd0>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc047fcca>] xfsaild+0x28a/0x780 [xfs]
[<ffffffffc047fa40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffa96c5da1>] kthread+0xd1/0xe0
[<ffffffffa96c5cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa9d95df7>] ret_from_fork_nospec_begin+0x21/0x21
Complete report #304188
Frame # Function Binary Source or offset Line
1
xfs_inode_buf_verify
xfs 0xffffffffc06f3a72
Build id: 3.10.0-1160.el7.x86_64 		0x82
2
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc07002e0
Build id: 3.10.0-1160.el7.x86_64 		0x130
3
xfs_inode_buf_write_verify
xfs 0xffffffffc06f3b60
Build id: 3.10.0-1160.el7.x86_64 		0x10
4
_xfs_buf_ioapply
xfs 0xffffffffc06fe027
Build id: 3.10.0-1160.el7.x86_64 		0x97
5
? wake_up_state
vmlinux 0xffffffff8f8daaf0
Build id: 3.10.0-1160.el7.x86_64 		0x20
6
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc07002e0
Build id: 3.10.0-1160.el7.x86_64 		0x130
7
__xfs_buf_submit
xfs 0xffffffffc06ffc72
Build id: 3.10.0-1160.el7.x86_64 		0x72
8
xfs_buf_delwri_submit_buffers
xfs 0xffffffffc07002e0
Build id: 3.10.0-1160.el7.x86_64 		0x130
9
? xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0700bd0
Build id: 3.10.0-1160.el7.x86_64 		0x10
10
? xfs_trans_ail_cursor_first
xfs 0xffffffffc072fa20
Build id: 3.10.0-1160.el7.x86_64 		0x90
11
xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0700bd0
Build id: 3.10.0-1160.el7.x86_64 		0x10
12
xfsaild
xfs 0xffffffffc072fcaa
Build id: 3.10.0-1160.el7.x86_64 		0x28a
13
? xfs_trans_ail_cursor_first
xfs 0xffffffffc072fa20
Build id: 3.10.0-1160.el7.x86_64 		0x90
14
kthread
vmlinux 0xffffffff8f8c5c21
Build id: 3.10.0-1160.el7.x86_64 		0xd1
15
? insert_kthread_work
vmlinux 0xffffffff8f8c5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffff8ff93df7
Build id: 3.10.0-1160.el7.x86_64 		0x21
17
? insert_kthread_work
vmlinux 0xffffffff8f8c5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [xfsaild/dm-0:407]
Modules linked in: tcp_lp nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel ac97_bus aesni_intel snd_seq snd_seq_device snd_pcm lrw gf128mul glue_helper ablk_helper cryptd pcspkr snd_timer vmw_balloon snd soundcore joydev sg parport_pc parport vmw_vmci i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm nfit libnvdimm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libata mptspi serio_raw scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 407 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff8bf7b1040000 ti: ffff8bf7b10c4000 task.ti: ffff8bf7b10c4000
RIP: 0010:[<ffffffffc06f3a72>]  [<ffffffffc06f3a72>] xfs_inode_buf_verify+0x82/0x160 [xfs]
RSP: 0018:ffff8bf7b10c7c18  EFLAGS: 00010286
RAX: ffff8bf70827c000 RBX: ffff8bf5b5ed8000 RCX: 0000000000000009
RDX: ffff8bf70827c000 RSI: 0000000000000000 RDI: ffff8bf72ec10480
RBP: ffff8bf7b10c7c50 R08: ffff8bf72ec10500 R09: 0000000000000000
R10: ffff8bf73c7b6018 R11: 000000010036a000 R12: ffff8bf792fbaa00
R13: 0000000db10c7bb0 R14: ffff8bf73c7b6018 R15: ffffffff8fb56fda
FS:  0000000000000000(0000) GS:ffff8bf7b5600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9870f80000 CR3: 00000000ada8e000 CR4: 00000000007607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffffc07002e0>] ? xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc06f3b60>] xfs_inode_buf_write_verify+0x10/0x20 [xfs]
[<ffffffffc06fe027>] _xfs_buf_ioapply+0x97/0x460 [xfs]
[<ffffffff8f8daaf0>] ? wake_up_state+0x20/0x20
[<ffffffffc07002e0>] ? xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc06ffc72>] __xfs_buf_submit+0x72/0x250 [xfs]
[<ffffffffc07002e0>] xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc0700bd0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc072fa20>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0700bd0>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc072fcaa>] xfsaild+0x28a/0x780 [xfs]
[<ffffffffc072fa20>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff8f8c5c21>] kthread+0xd1/0xe0
[<ffffffff8f8c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff8ff93df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8f8c5b50>] ? insert_kthread_work+0x40/0x40
Code: ff 0f b7 83 82 00 00 00 41 d3 ff 44 0f af f8 45 85 ff 7e 48 0f b6 8b 92 00 00 00 44 89 e6 4c 89 ef d3 e6 48 63 f6 e8 7e c9 00 00 <66> 81 38 49 4e 8b 50 60 48 89 c1 0f ca 74 37 45 84 f6 0f 84 96
Complete report #137980
Frame # Function Binary Source or offset Line
1
xfs_inode_buf_verify
xfs 0xffffffffc053aaf2
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x82
2
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc0547350
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x130
3
xfs_inode_buf_write_verify
xfs 0xffffffffc053abe0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x10
4
_xfs_buf_ioapply
xfs 0xffffffffc0545097
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x97
5
? wake_up_state
vmlinux 0xffffffffa16db990
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x20
6
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc0547350
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x130
7
__xfs_buf_submit
xfs 0xffffffffc0546ce2
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x72
8
xfs_buf_delwri_submit_buffers
xfs 0xffffffffc0547350
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x130
9
? xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0547c40
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x10
10
? xfs_trans_ail_cursor_first
xfs 0xffffffffc05769a0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x90
11
xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0547c40
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x10
12
xfsaild
xfs 0xffffffffc0576c1a
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x27a
13
? xfs_trans_ail_cursor_first
xfs 0xffffffffc05769a0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x90 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 28s! [ora_mman_xe:1826]
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [xfsaild/dm-0:415]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter ipmi_devintf ipmi_msghandler vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev joydev pcspkr vmw_balloon vmxnet3 sg parport_pc parport vmw_vmci i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif drm_kms_helper crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit libnvdimm drm crct10dif_pclmul ata_piix crct10dif_common
crc32c_intel serio_raw libata vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 415 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G             L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff8d22289c8000 ti: ffff8d22289d0000 task.ti: ffff8d22289d0000
RIP: 0010:[<ffffffffc053aaf2>]  [<ffffffffc053aaf2>] xfs_inode_buf_verify+0x82/0x160 [xfs]
RSP: 0018:ffff8d22289d3c18  EFLAGS: 00010286
RAX: ffff8d215ef83000 RBX: ffff8d2229548000 RCX: 0000000000000009
RDX: ffff8d215ef83000 RSI: 0000000000001000 RDI: ffff8d222a722a80
RBP: ffff8d22289d3c50 R08: ffff8d222a722b00 R09: 0000000000000000
R10: ffff8d218821b818 R11: 000000042e312000 R12: 0000000030f8d419
R13: 00000001289d3bb0 R14: ffff8d218821b818 R15: ffffffffa19562de
FS:  0000000000000000(0000) GS:ffff8d223fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1cd6e91218 CR3: 00000000baf28000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc0547350>] ? xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc053abe0>] xfs_inode_buf_write_verify+0x10/0x20 [xfs]
[<ffffffffc0545097>] _xfs_buf_ioapply+0x97/0x460 [xfs]
[<ffffffffa16db990>] ? wake_up_state+0x20/0x20
[<ffffffffc0547350>] ? xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc0546ce2>] __xfs_buf_submit+0x72/0x250 [xfs]
[<ffffffffc0547350>] xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc0547c40>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc05769a0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0547c40>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0576c1a>] xfsaild+0x27a/0x760 [xfs]
[<ffffffffc05769a0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
Complete report #24369
Frame # Function Binary Source or offset Line
1
xfs_inode_buf_verify
xfs 0xffffffffc0314a72
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x82
2
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc03212e0
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x130
3
xfs_inode_buf_write_verify
xfs 0xffffffffc0314b60
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x10
4
_xfs_buf_ioapply
xfs 0xffffffffc031f027
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x97
5
? wake_up_state
vmlinux 0xffffffffb4edb190
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x20
6
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc03212e0
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x130
7
__xfs_buf_submit
xfs 0xffffffffc0320c72
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x72
8
xfs_buf_delwri_submit_buffers
xfs 0xffffffffc03212e0
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x130
9
? xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0321bd0
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x10
10
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0350a40
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x90
11
xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0321bd0
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x10
12
xfsaild
xfs 0xffffffffc0350cca
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x28a
13
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0350a40
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x90
14
kthread
vmlinux 0xffffffffb4ec5e71
Build id: 3.10.0-1160.11.1.el7.x86_64 		0xd1
15
? insert_kthread_work
vmlinux 0xffffffffb4ec5da0
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffffb5593df7
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x21
17
? insert_kthread_work
vmlinux 0xffffffffb4ec5da0
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [xfsaild/dm-0:423]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon sg pcspkr coretemp joydev i2c_piix4 parport_pc vmw_vmci parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic crct10dif_common drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ata_piix ttm drm serio_raw libata mptspi scsi_transport_spi mptscsih e1000 mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 423 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 06/22/2012
task: ffff95a3f58bc200 ti: ffff95a3f5ff4000 task.ti: ffff95a3f5ff4000
RIP: 0010:[<ffffffffc0314a72>]  [<ffffffffc0314a72>] xfs_inode_buf_verify+0x82/0x160 [xfs]
RSP: 0018:ffff95a3f5ff7c18  EFLAGS: 00010286
RAX: ffff95a5373af000 RBX: ffff95a575e0ce00 RCX: 0000000000000009
RDX: ffff95a5373af000 RSI: 0000000000000000 RDI: ffff95a574ddb000
RBP: ffff95a3f5ff7c50 R08: ffff95a574ddb080 R09: 0000000000000000
R10: ffff95a53fe73418 R11: 0000000000000000 R12: ffff95a46abdb300
R13: 00000001f5ff7bb0 R14: ffff95a53fe73418 R15: ffffffffb51571da
FS:  0000000000000000(0000) GS:ffff95a57fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f62a67b8000 CR3: 0000000178b84000 CR4: 00000000000407f0
Call Trace:
[<ffffffffc03212e0>] ? xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc0314b60>] xfs_inode_buf_write_verify+0x10/0x20 [xfs]
[<ffffffffc031f027>] _xfs_buf_ioapply+0x97/0x460 [xfs]
[<ffffffffb4edb190>] ? wake_up_state+0x20/0x20
[<ffffffffc03212e0>] ? xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc0320c72>] __xfs_buf_submit+0x72/0x250 [xfs]
[<ffffffffc03212e0>] xfs_buf_delwri_submit_buffers+0x130/0x240 [xfs]
[<ffffffffc0321bd0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0350a40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0321bd0>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc0350cca>] xfsaild+0x28a/0x780 [xfs]
[<ffffffffc0350a40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffb4ec5e71>] kthread+0xd1/0xe0
[<ffffffffb4ec5da0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb5593df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb4ec5da0>] ? insert_kthread_work+0x40/0x40
Code: ff 0f b7 83 82 00 00 00 41 d3 ff 44 0f af f8 45 85 ff 7e 48 0f b6 8b 92 00 00 00 44 89 e6 4c 89 ef d3 e6 48 63 f6 e8 7e c9 00 00 <66> 81 38 49 4e 8b 50 60 48 89 c1 0f ca 74 37 45 84 f6 0f 84 96
Complete report #19539
Frame # Function Binary Source or offset Line
1
xfs_inode_buf_verify
xfs 0xffffffffc0325b58
Build id: 3.10.0-1062.el7.x86_64 		0x98
2
? xfs_buf_submit
xfs 0xffffffffc0331b4a
Build id: 3.10.0-1062.el7.x86_64 		0x1a
3
xfs_inode_buf_write_verify
xfs 0xffffffffc0325bd0
Build id: 3.10.0-1062.el7.x86_64 		0x10
4
_xfs_buf_ioapply
xfs 0xffffffffc032fda7
Build id: 3.10.0-1062.el7.x86_64 		0x97
5
? wake_up_state
vmlinux 0xffffffff878da0b0
Build id: 3.10.0-1062.el7.x86_64 		0x20
6
? xfs_buf_submit
xfs 0xffffffffc0331b4a
Build id: 3.10.0-1062.el7.x86_64 		0x1a
7
__xfs_buf_submit
xfs 0xffffffffc03318c1
Build id: 3.10.0-1062.el7.x86_64 		0x61
8
xfs_buf_submit
xfs 0xffffffffc0331b4a
Build id: 3.10.0-1062.el7.x86_64 		0x1a
9
xfs_buf_delwri_submit_buffers
xfs 0xffffffffc0331d9c
Build id: 3.10.0-1062.el7.x86_64 		0x22c
10
? xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0332930
Build id: 3.10.0-1062.el7.x86_64 		0x10
11
? xfs_trans_ail_cursor_first
xfs 0xffffffffc03613d0
Build id: 3.10.0-1062.el7.x86_64 		0x90
12
xfs_buf_delwri_submit_nowait
xfs 0xffffffffc0332930
Build id: 3.10.0-1062.el7.x86_64 		0x10
13
xfsaild
xfs 0xffffffffc036164a
Build id: 3.10.0-1062.el7.x86_64 		0x27a
14
? xfs_trans_ail_cursor_first
xfs 0xffffffffc03613d0
Build id: 3.10.0-1062.el7.x86_64 		0x90
15
kthread
vmlinux 0xffffffff878c50d1
Build id: 3.10.0-1062.el7.x86_64 		0xd1
16
? insert_kthread_work
vmlinux 0xffffffff878c5000
Build id: 3.10.0-1062.el7.x86_64 		0x40
17
ret_from_fork_nospec_begin
vmlinux 0xffffffff87f8bd37
Build id: 3.10.0-1062.el7.x86_64 		0x21
18
? insert_kthread_work
vmlinux 0xffffffff878c5000
Build id: 3.10.0-1062.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [xfsaild/dm-0:696]
Modules linked in: tcp_lp fuse uinput xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun bridge stp llc devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ppdev crc32_pclmul snd_hda_codec_generic ghash_clmulni_intel snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device aesni_intel snd_pcm lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr joydev virtio_rng virtio_balloon snd_timer parport_pc snd parport
soundcore i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom virtio_console virtio_net virtio_blk qxl drm_kms_helper ata_generic pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata floppy virtio_pci serio_raw virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 696 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff9da0f4bbc1c0 ti: ffff9da2e91b0000 task.ti: ffff9da2e91b0000
RIP: 0010:[<ffffffffc0325b58>]  [<ffffffffc0325b58>] xfs_inode_buf_verify+0x98/0x100 [xfs]
RSP: 0018:ffff9da2e91b3c18  EFLAGS: 00010286
RAX: ffff9d9ef50e4000 RBX: 0000000000000008 RCX: 0000000000000009
RDX: ffff9d9ef50e4000 RSI: 0000000000001000 RDI: ffff9da1fe359680
RBP: ffff9da2e91b3c40 R08: ffff9da1fe359700 R09: 0000000000000000
R10: fffff1d86f89e500 R11: ffff9da0f4ffdc08 R12: ffff9d9ef559e000
R13: 0000000000000020 R14: ffff9da1fe359680 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff9da2ffc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000099001000 CR3: 0000000235010000 CR4: 00000000000006f0
Call Trace:
[<ffffffffc0331b4a>] ? xfs_buf_submit+0x1a/0x40 [xfs]
[<ffffffffc0325bd0>] xfs_inode_buf_write_verify+0x10/0x20 [xfs]
[<ffffffffc032fda7>] _xfs_buf_ioapply+0x97/0x460 [xfs]
[<ffffffff878da0b0>] ? wake_up_state+0x20/0x20
[<ffffffffc0331b4a>] ? xfs_buf_submit+0x1a/0x40 [xfs]
[<ffffffffc03318c1>] __xfs_buf_submit+0x61/0x210 [xfs]
[<ffffffffc0331b4a>] xfs_buf_submit+0x1a/0x40 [xfs]
[<ffffffffc0331d9c>] xfs_buf_delwri_submit_buffers+0x22c/0x240 [xfs]
[<ffffffffc0332930>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc03613d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0332930>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc036164a>] xfsaild+0x27a/0x760 [xfs]
[<ffffffffc03613d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff878c50d1>] kthread+0xd1/0xe0
[<ffffffff878c5000>] ? insert_kthread_work+0x40/0x40
[<ffffffff87f8bd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff878c5000>] ? insert_kthread_work+0x40/0x40
Code: 3f bf 00 00 4c 89 f7 e8 b7 e7 00 00 44 39 eb 74 49 66 90 41 0f b6 8c 24 92 00 00 00 89 de 4c 89 f7 d3 e6 48 63 f6 e8 f8 c5 00 00 <66> 81 38 49 4e 75 b9 0f b6 50 04 41 0f b7 44 24 7c 83 e0 0f 66
Complete report #2620
Frame # Function Binary Source or offset Line
1
xfs_inode_buf_verify
xfs 0xffffffffc0672e68
Build id: 3.10.0-957.el7.x86_64 		0x98
2
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc067f2a8
Build id: 3.10.0-957.el7.x86_64 		0x128
3
xfs_inode_buf_write_verify
xfs 0xffffffffc0672ee0
Build id: 3.10.0-957.el7.x86_64 		0x10
4
_xfs_buf_ioapply
xfs 0xffffffffc067d1b7
Build id: 3.10.0-957.el7.x86_64 		0x97
5
? wake_up_state
vmlinux 0xffffffff868d67b0
Build id: 3.10.0-957.el7.x86_64 		0x20
6
? xfs_buf_delwri_submit_buffers
xfs 0xffffffffc067f2a8
Build id: 3.10.0-957.el7.x86_64 		0x128
7
xfs_buf_submit
xfs 0xffffffffc067efcc
Build id: 3.10.0-957.el7.x86_64 		0x6c
8
xfs_buf_delwri_submit_buffers
xfs 0xffffffffc067f2a8
Build id: 3.10.0-957.el7.x86_64 		0x128
9
? xfs_buf_delwri_submit_nowait
xfs 0xffffffffc06800c0
Build id: 3.10.0-957.el7.x86_64 		0x10
10
? xfs_trans_ail_cursor_first
xfs 0xffffffffc06aec60
Build id: 3.10.0-957.el7.x86_64 		0x90
11
xfs_buf_delwri_submit_nowait
xfs 0xffffffffc06800c0
Build id: 3.10.0-957.el7.x86_64 		0x10
12
xfsaild
xfs 0xffffffffc06aeebf
Build id: 3.10.0-957.el7.x86_64 		0x25f
13
? xfs_trans_ail_cursor_first
xfs 0xffffffffc06aec60
Build id: 3.10.0-957.el7.x86_64 		0x90
14
kthread
vmlinux 0xffffffff868c1c31
Build id: 3.10.0-957.el7.x86_64 		0xd1
15
? insert_kthread_work
vmlinux 0xffffffff868c1b60
Build id: 3.10.0-957.el7.x86_64 		0x40
16
ret_from_fork_nospec_begin
vmlinux 0xffffffff86f74c1d
Build id: 3.10.0-957.el7.x86_64 		0x7
17
? insert_kthread_work
vmlinux 0xffffffff868c1b60
Build id: 3.10.0-957.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [xfsaild/dm-0:4385]
Modules linked in: tcp_lp nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vmw_vmci vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi snd_ens1371 crc32_pclmul ghash_clmulni_intel snd_rawmidi aesni_intel snd_ac97_codec lrw gf128mul glue_helper ablk_helper ppdev ac97_bus cryptd vmw_balloon snd_seq snd_seq_device snd_pcm snd_timer snd pcspkr joydev soundcore parport_pc parport sg i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_pclmul crct10dif_common crc32c_intel ata_generic
pata_acpi serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm e1000 drm ata_piix mptspi scsi_transport_spi mptscsih mptbase libata drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 4385 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8a85f5e74100 ti: ffff8a85f4b54000 task.ti: ffff8a85f4b54000
RIP: 0010:[<ffffffffc0672e68>]  [<ffffffffc0672e68>] xfs_inode_buf_verify+0x98/0x100 [xfs]
RSP: 0018:ffff8a85f4b57c40  EFLAGS: 00010286
RAX: ffff8a87336dc000 RBX: 0000000000000001 RCX: 0000000000000009
RDX: ffff8a87336dc000 RSI: 0000000000003000 RDI: ffff8a87322e8780
RBP: ffff8a85f4b57c68 R08: ffff8a87322e87f0 R09: 0000000000000000
R10: ffff8a861a295518 R11: 0000080000000200 R12: ffff8a85f5128000
R13: 00000000b55aad4c R14: ffff8a86dd94e580 R15: 00000003f4b57bd0
FS:  0000000000000000(0000) GS:ffff8a8738640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000008fec30 CR3: 0000000124dc2000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc067f2a8>] ? xfs_buf_delwri_submit_buffers+0x128/0x230 [xfs]
[<ffffffffc0672ee0>] xfs_inode_buf_write_verify+0x10/0x20 [xfs]
[<ffffffffc067d1b7>] _xfs_buf_ioapply+0x97/0x460 [xfs]
[<ffffffff868d67b0>] ? wake_up_state+0x20/0x20
[<ffffffffc067f2a8>] ? xfs_buf_delwri_submit_buffers+0x128/0x230 [xfs]
[<ffffffffc067efcc>] xfs_buf_submit+0x6c/0x220 [xfs]
[<ffffffffc067f2a8>] xfs_buf_delwri_submit_buffers+0x128/0x230 [xfs]
[<ffffffffc06800c0>] ? xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc06aec60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc06800c0>] xfs_buf_delwri_submit_nowait+0x10/0x20 [xfs]
[<ffffffffc06aeebf>] xfsaild+0x25f/0x6f0 [xfs]
[<ffffffffc06aec60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff868c1c31>] kthread+0xd1/0xe0
[<ffffffff868c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff86f74c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff868c1b60>] ? insert_kthread_work+0x40/0x40
Code: 5f c0 00 00 4c 89 f7 e8 57 ec 00 00 44 39 eb 74 49 66 90 41 0f b6 8c 24 92 00 00 00 89 de 4c 89 f7 d3 e6 48 63 f6 e8 78 ca 00 00 <66> 81 38 49 4e 75 b9 0f b6 50 04 41 0f b7 44 24 7c 83 e0 0f 66