Info

Permalink
Function
_raw_spin_lock
First occurrence
2020-09-22
Last occurrence
2025-05-26
State
NEW
Components
kernel
Quality
-24

Statistics

Operating system Count
CentOS 7 620
Architecture Count
x86_64 620
Related packages Count
kernel 620
 0:3.10.0-1127.19.1.el7 128
 0:3.10.0-1160.118.1.el7 108
 0:3.10.0-1160.108.1.el7 58
 0:3.10.0-1160.el7 35
 0:3.10.0-1062.9.1.el7 32
 0:3.10.0-1160.105.1.el7 26
 0:3.10.0-1062.18.1.el7 25
 0:3.10.0-1160.36.2.el7 16
 0:3.10.0-1160.76.1.el7 15
 0:3.10.0-693.11.1.el7 15
 0:3.10.0-1160.119.1.el7 13
 0:3.10.0-693.el7 12
 0:3.10.0-957.el7 11
 0:3.10.0-1160.90.1.el7 9
 0:3.10.0-1160.11.1.el7 9
 0:3.10.0-693.5.2.el7 9
 0:3.10.0-957.5.1.el7 7
 0:3.10.0-1062.el7 6
 0:3.10.0-1127.18.2.el7 5
 0:3.10.0-1160.42.2.el7 5
 0:3.10.0-1160.49.1.el7 5
 0:3.10.0-1160.6.1.el7 5
 0:3.10.0-1160.81.1.el7 4
 0:3.10.0-862.el7 4
 0:3.10.0-1160.31.1.el7 4
 0:3.10.0-1160.66.1.el7 4
 0:3.10.0-1160.95.1.el7 4
 0:3.10.0-957.27.2.el7 4
 0:3.10.0-1127.el7 4
 0:3.10.0-1160.45.1.el7 3
 0:3.10.0-1160.92.1.el7 3
 0:3.10.0-957.21.3.el7 3
 0:3.10.0-957.10.1.el7 2
 0:3.10.0-1062.4.1.el7 2
 0:3.10.0-1160.2.2.el7 2
 0:3.10.0-1160.53.1.el7 2
 0:3.10.0-1160.59.1.el7 2
 0:3.10.0-862.3.2.el7 2
 0:3.10.0-1160.25.1.el7 2
 0:3.10.0-1062.12.1.el7 1
 0:3.10.0-1127.10.1.el7 1
 0:3.10.0-1127.13.1.el7 1
 0:3.10.0-1160.102.1.el7 1
 0:3.10.0-1160.114.2.el7 1
 0:3.10.0-1160.24.1.el7 1
 0:3.10.0-1160.62.1.el7 1
 0:3.10.0-1160.71.1.el7 1
 0:3.10.0-1160.83.1.el7 1
 0:3.10.0-862.11.6.el7 1
 0:3.10.0-862.9.1.el7 1
 0:3.10.0-957.12.2.el7 1
 0:3.10.0-1062.4.3.el7 1
 0:3.10.0-1062.1.1.el7 1
 0:3.10.0-862.3.3.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1419337
Frame # Function Binary Source or offset Line
1
xfs_inode_to_disk
xfs 0xffffffffc042614e
Build id: 3.10.0-1160.119.1.el7.x86_64
0xe
2
? xfs_iflush_int
xfs 0xffffffffc0447fbf
Build id: 3.10.0-1160.119.1.el7.x86_64
0xcf
3
xfs_iflush_cluster
xfs 0xffffffffc044844b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x2ab
4
xfs_iflush
xfs 0xffffffffc044875d
Build id: 3.10.0-1160.119.1.el7.x86_64
0x1dd
5
xfs_inode_item_push
xfs 0xffffffffc045af26
Build id: 3.10.0-1160.119.1.el7.x86_64
0x156
6
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0462e70
Build id: 3.10.0-1160.119.1.el7.x86_64
0xa0
7
xfsaild
xfs 0xffffffffc04631ff
Build id: 3.10.0-1160.119.1.el7.x86_64
0x38f
8
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0462e70
Build id: 3.10.0-1160.119.1.el7.x86_64
0xa0
9
kthread
vmlinux 0xffffffff91ccb621
Build id: 3.10.0-1160.119.1.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffff91ccb550
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffff923c51dd
Build id: 3.10.0-1160.119.1.el7.x86_64
0x7
12
? insert_kthread_work
vmlinux 0xffffffff91ccb550
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 3044s! [xfsaild/dm-0:424]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache vboxsf(OE) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc snd_intel8x0 snd_ac97_codec ppdev ac97_bus snd_seq snd_seq_device snd_pcm vboxguest(OE) sg snd_timer joydev pcspkr snd soundcore i2c_piix4 parport_pc parport video ip_tables xfs libcrc32c sd_mod crc_t10dif
crct10dif_generic crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea ahci ata_piix sysfillrect libahci sysimgblt fb_sys_fops ttm crc32c_intel serio_raw drm libata e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 424 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff9aec93a0a100 ti: ffff9aec936cc000 task.ti: ffff9aec936cc000
RIP: 0010:[<ffffffffc042614e>]  [<ffffffffc042614e>] xfs_inode_to_disk+0xe/0x210 [xfs]
RSP: 0018:ffff9aec936cfca0  EFLAGS: 00010246
RAX: 0000000000000001 RBX: ffffffffc0431deb RCX: 0000000000004e49
RDX: 000003a900008391 RSI: ffff9aec97d6fc00 RDI: ffff9aebb5ba61c0
RBP: ffff9aec936cfce8 R08: 0000000000000000 R09: 0000000000000000
R10: ffff9aec9b001700 R11: ffff9aec936ac000 R12: ffff9aec936cfd10
R13: ffff9aec5f469780 R14: 0000000000030001 R15: ffff9aebb545fb00
FS:  0000000000000000(0000) GS:ffff9aec9fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f643aba3000 CR3: 00000000427b2000 CR4: 00000000000206e0
Call Trace:
[<ffffffffc0447fbf>] ? xfs_iflush_int+0xcf/0x2b0 [xfs]
[<ffffffffc044844b>] xfs_iflush_cluster+0x2ab/0x3e0 [xfs]
[<ffffffffc044875d>] xfs_iflush+0x1dd/0x250 [xfs]
[<ffffffffc045af26>] xfs_inode_item_push+0x156/0x1a0 [xfs]
[<ffffffffc0462e70>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [xfs]
[<ffffffffc04631ff>] xfsaild+0x38f/0x780 [xfs]
[<ffffffffc0462e70>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [xfs]
[<ffffffff91ccb621>] kthread+0xd1/0xe0
[<ffffffff91ccb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff923c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff91ccb550>] ? insert_kthread_work+0x40/0x40
Code: 89 83 0c 01 00 00 ba 02 00 00 00 e9 3b fe ff ff 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 45 31 c0 b9 49 4e 00 00 55 <66> 89 0e 66 44 89 46 06 45 31 c9 0f b6 87 00 01 00 00 48 89 e5 
Complete report #724152
Frame # Function Binary Source or offset Line
1
_raw_spin_lock
vmlinux 0xffffffffb65886bc
Build id: 3.10.0-1127.18.2.el7.x86_64
0xc
2
? xfs_iflush_cluster
xfs 0xffffffffc042d75b
Build id: 3.10.0-1127.18.2.el7.x86_64
0x18b
3
xfs_iflush
xfs 0xffffffffc042db85
Build id: 3.10.0-1127.18.2.el7.x86_64
0x1d5
4
xfs_inode_item_push
xfs 0xffffffffc043fc5e
Build id: 3.10.0-1127.18.2.el7.x86_64
0x14e
5
? xfs_trans_ail_cursor_first
xfs 0xffffffffc04479a0
Build id: 3.10.0-1127.18.2.el7.x86_64
0x90
6
xfsaild
xfs 0xffffffffc0447d0f
Build id: 3.10.0-1127.18.2.el7.x86_64
0x36f
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc04479a0
Build id: 3.10.0-1127.18.2.el7.x86_64
0x90
8
kthread
vmlinux 0xffffffffb5ec6691
Build id: 3.10.0-1127.18.2.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffffb5ec65c0
Build id: 3.10.0-1127.18.2.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffffb6592d37
Build id: 3.10.0-1127.18.2.el7.x86_64
0x21
11
? insert_kthread_work
vmlinux 0xffffffffb5ec65c0
Build id: 3.10.0-1127.18.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 52s! [xfsaild/dm-9:889]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 openafs(POE) binfmt_misc
ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit ata_piix mptspi scsi_transport_spi crct10dif_pclmul crct10dif_common mptscsih libnvdimm crc32c_intel libata serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 889 Comm: xfsaild/dm-9 Kdump: loaded Tainted: P           OEL ------------   3.10.0-1127.18.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff912ad79d41c0 ti: ffff912ad79ec000 task.ti: ffff912ad79ec000
RIP: 0010:[<ffffffffb65886bc>]  [<ffffffffb65886bc>] _raw_spin_lock+0xc/0x30
RSP: 0018:ffff912ad79efcf0  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000014 RCX: 00000000000fffff
RDX: 0000000000000001 RSI: 0000000000026ffe RDI: ffff9121e75efc94
RBP: ffff912ad79efd50 R08: 0000000000000002 R09: 0000000000000001
R10: ffff91233fc03700 R11: ffff912aae136e00 R12: 00000000b5ba633a
R13: ffff9123feaa6540 R14: ffff912ad8abb670 R15: ffff912ad79efca0
FS:  0000000000000000(0000) GS:ffff912ada600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000560444833000 CR3: 0000000f2a166000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc042d75b>] ? xfs_iflush_cluster+0x18b/0x3e0 [xfs]
[<ffffffffc042db85>] xfs_iflush+0x1d5/0x250 [xfs]
[<ffffffffc043fc5e>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc04479a0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0447d0f>] xfsaild+0x36f/0x760 [xfs]
[<ffffffffc04479a0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffb5ec6691>] kthread+0xd1/0xe0
[<ffffffffb5ec65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb6592d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb5ec65c0>] ? insert_kthread_work+0x40/0x40
Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 49 19 ff ff 5d 
Complete report #714695
Frame # Function Binary Source or offset Line
1
_raw_spin_lock
vmlinux 0xffffffffabf6a3ec
Build id: 3.10.0-957.el7.x86_64
0xc
2
? xfs_iflush_cluster
xfs 0xffffffffc0542a93
Build id: 3.10.0-957.el7.x86_64
0x193
3
xfs_iflush
xfs 0xffffffffc0545f60
Build id: 3.10.0-957.el7.x86_64
0x1e0
4
xfs_inode_item_push
xfs 0xffffffffc055816e
Build id: 3.10.0-957.el7.x86_64
0x14e
5
? xfs_trans_ail_cursor_first
xfs 0xffffffffc055fc60
Build id: 3.10.0-957.el7.x86_64
0x90
6
xfsaild
xfs 0xffffffffc055ff87
Build id: 3.10.0-957.el7.x86_64
0x327
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc055fc60
Build id: 3.10.0-957.el7.x86_64
0x90
8
kthread
vmlinux 0xffffffffab8c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffffab8c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffffabf74c37
Build id: 3.10.0-957.el7.x86_64
0x21
11
? insert_kthread_work
vmlinux 0xffffffffab8c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [xfsaild/dm-0:4889]
Modules linked in: veth xt_nat xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) overlay(T) sunrpc snd_seq_midi snd_seq_midi_event ppdev vmw_balloon sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus joydev snd_seq snd_seq_device pcspkr snd_pcm snd_timer snd soundcore sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_pclmul
crct10dif_common crc32c_intel vmwgfx ata_generic serio_raw pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm e1000 drm mptspi scsi_transport_spi mptscsih ata_piix mptbase libata drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 4889 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9e21d8462080 ti: ffff9e21d63cc000 task.ti: ffff9e21d63cc000
RIP: 0010:[<ffffffffabf6a3ec>]  [<ffffffffabf6a3ec>] _raw_spin_lock+0xc/0x30
RSP: 0018:ffff9e21d63cfd00  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000014 RCX: 0000000001ffffff
RDX: 0000000000000001 RSI: 0000000000000008 RDI: ffff9e1f66e55374
RBP: ffff9e21d63cfd60 R08: 7038000000000000 R09: 1f63df5eb81c0000
R10: ffff9e1b3fc07700 R11: ffff9e1f2737c9f8 R12: 0000000067370c6e
R13: ffff9e1f66e55280 R14: ffff9e21d9218270 R15: ffff9e21d63cfcb0
FS:  0000000000000000(0000) GS:ffff9e21dd680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa87b722600 CR3: 00000005e7c10000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc0542a93>] ? xfs_iflush_cluster+0x193/0x440 [xfs]
[<ffffffffc0545f60>] xfs_iflush+0x1e0/0x250 [xfs]
[<ffffffffc055816e>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc055fc60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc055ff87>] xfsaild+0x327/0x6f0 [xfs]
[<ffffffffc055fc60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffab8c1c31>] kthread+0xd1/0xe0
[<ffffffffab8c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffabf74c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffab8c1b60>] ? insert_kthread_work+0x40/0x40
Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 40 1b ff ff 5d 
Complete report #530214
Frame # Function Binary Source or offset Line
1
xfs_iflush_cluster
xfs 0xffffffffc04d47c9
Build id: 3.10.0-1160.76.1.el7.x86_64
0x139
2
xfs_iflush
xfs 0xffffffffc04d4c45
Build id: 3.10.0-1160.76.1.el7.x86_64
0x1d5
3
xfs_inode_item_push
xfs 0xffffffffc04e6d5e
Build id: 3.10.0-1160.76.1.el7.x86_64
0x14e
4
? xfs_trans_ail_cursor_first
xfs 0xffffffffc04eea80
Build id: 3.10.0-1160.76.1.el7.x86_64
0x90
5
xfsaild
xfs 0xffffffffc04eee0f
Build id: 3.10.0-1160.76.1.el7.x86_64
0x38f
6
? xfs_trans_ail_cursor_first
xfs 0xffffffffc04eea80
Build id: 3.10.0-1160.76.1.el7.x86_64
0x90
7
kthread
vmlinux 0xffffffffb3cc5f91
Build id: 3.10.0-1160.76.1.el7.x86_64
0xd1
8
? insert_kthread_work
vmlinux 0xffffffffb3cc5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
9
ret_from_fork_nospec_begin
vmlinux 0xffffffffb4399de4
Build id: 3.10.0-1160.76.1.el7.x86_64
0xe
10
? insert_kthread_work
vmlinux 0xffffffffb3cc5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [xfsaild/dm-0:510]
Modules linked in: xt_time xt_conntrack tcp_diag inet_diag xt_owner ip6t_REJECT nf_reject_ipv6 ip6table_filter xt_nat xt_NFLOG xt_set xt_multiport ip_set_bitmap_port ip_set_list_set ip_set_hash_net kcare(OE) iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter ext4 mbcache jbd2 nfnetlink_log loop ip_set vmw_vsock_vmci_transport vsock crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper joydev cryptd pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic syscopyarea nfit sysfillrect
sysimgblt fb_sys_fops libnvdimm ttm ata_generic pata_acpi drm ata_piix libata mptspi scsi_transport_spi mptscsih mptbase crct10dif_pclmul crct10dif_common vmxnet3 serio_raw crc32c_intel floppy drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 510 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9ef517fe5280 ti: ffff9ef517f3c000 task.ti: ffff9ef517f3c000
RIP: 0010:[<ffffffffc04d47c9>]  [<ffffffffc04d47c9>] xfs_iflush_cluster+0x139/0x3e0 [xfs]
RSP: 0018:ffff9ef517f3fcf8  EFLAGS: 00000282
RAX: ffff9eef5df014d0 RBX: ffff9ef23b3bc790 RCX: 0000000000ffffff
RDX: 0000000001000000 RSI: 0000000000000008 RDI: ffff9ef287d98c1c
RBP: ffff9ef517f3fd50 R08: ffff9ef84aa85600 R09: ffff9ef84aa85600
R10: ffff9ef84aa855d0 R11: ffff9ef517e39000 R12: 0000000000000000
R13: ffff9eefc8b28480 R14: ffffffffc029206b R15: ffff9ef517f3fc88
FS:  00007fac30bf7700(0000) GS:ffff9eefca400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000c00060f940 CR3: 000000057aa1a000 CR4: 00000000000407f0
Call Trace:
[<ffffffffc04d4c45>] xfs_iflush+0x1d5/0x250 [xfs]
[<ffffffffc04e6d5e>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc04eea80>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc04eee0f>] xfsaild+0x38f/0x780 [xfs]
[<ffffffffc04eea80>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffb3cc5f91>] kthread+0xd1/0xe0
[<ffffffffb3cc5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb4399de4>] ret_from_fork_nospec_begin+0xe/0x21
[<ffffffffb3cc5ec0>] ? insert_kthread_work+0x40/0x40
Code: 00 ba 01 00 00 00 48 d3 e2 48 89 d1 83 e9 01 21 c8 48 3b 45 b8 0f 85 2d 02 00 00 c6 07 00 66 66 66 90 49 8b 47 70 48 85 c0 74 0c <f7> 80 a8 00 00 00 df 47 00 00 75 0b 41 8b 87 d8 00 00 00 85 c0 
Complete report #318728
Frame # Function Binary Source or offset Line
1
xfs_iflush_cluster
xfs 0xffffffffc029a68c
Build id: 3.10.0-693.el7.x86_64
0x13c
2
xfs_iflush
xfs 0xffffffffc029e390
Build id: 3.10.0-693.el7.x86_64
0x1e0
3
xfs_inode_item_push
xfs 0xffffffffc02afea2
Build id: 3.10.0-693.el7.x86_64
0x102
4
? xfs_trans_ail_cursor_first
xfs 0xffffffffc02b7a10
Build id: 3.10.0-693.el7.x86_64
0x90
5
xfsaild
xfs 0xffffffffc02b7d31
Build id: 3.10.0-693.el7.x86_64
0x321
6
? xfs_trans_ail_cursor_first
xfs 0xffffffffc02b7a10
Build id: 3.10.0-693.el7.x86_64
0x90
7
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
8
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
9
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 25s! [xfsaild/dm-4:845]
Modules linked in: arc4 md4 xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nls_utf8 cifs dns_resolver vmw_vsock_vmci_transport vsock sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg parport_pc i2c_piix4 parport shpchp vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptsas scsi_transport_sas
drm ata_piix mptscsih crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 845 Comm: xfsaild/dm-4 Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8800357d0000 ti: ffff8800357cc000 task.ti: ffff8800357cc000
RIP: 0010:[<ffffffffc029a68c>]  [<ffffffffc029a68c>] xfs_iflush_cluster+0x13c/0x440 [xfs]
RSP: 0018:ffff8800357cfd10  EFLAGS: 00010286
RAX: ffff880c0268f8c0 RBX: 0000000000000000 RCX: 0000000007ffffff
RDX: 0000000008000000 RSI: 0000000000000001 RDI: ffff880c0bcf04b4
RBP: ffff8800357cfd68 R08: 0000000000019c00 R09: 000000000000007f
R10: 0000000000000000 R11: 0000000000000220 R12: ffffffffc02a80a7
R13: ffff8800357cfd68 R14: ffff8800357cfcb0 R15: ffffffff816a976a
FS:  0000000000000000(0000) GS:ffff880c0f300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7cab9fc408 CR3: 0000000b87e24000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff880c07e4fd80 ffff880c02bea000 0000000000000060 ffff880c0a898000
ffffffe000000000 0000000000000100 ffff8809b4df4b00 00000000ffffff8b
ffff8800357cfdd0 ffff8809b4df4bf4 0000000000000000 ffff8800357cfdc0
Call Trace:
[<ffffffffc029e390>] xfs_iflush+0x1e0/0x250 [xfs]
[<ffffffffc02afea2>] xfs_inode_item_push+0x102/0x150 [xfs]
[<ffffffffc02b7a10>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc02b7d31>] xfsaild+0x321/0x6f0 [xfs]
[<ffffffffc02b7a10>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 00 48 d3 e2 48 89 d1 83 e9 01 21 c8 48 3b 45 b8 0f 85 2a 02 00 00 c6 07 00 0f 1f 40 00 49 8b 87 88 00 00 00 48 85 c0 74 0c <f7> 80 98 00 00 00 ff 47 00 00 75 0b 41 8b 87 f0 00 00 00 85 c0 
Complete report #169180
Frame # Function Binary Source or offset Line
1
_raw_spin_lock
vmlinux 0xffffffff9558b85c
Build id: 3.10.0-1160.25.1.el7.x86_64
0xc
2
? xfs_iflush_cluster
xfs 0xffffffffc02f980b
Build id: 3.10.0-1160.25.1.el7.x86_64
0x18b
3
xfs_iflush
xfs 0xffffffffc02f9c35
Build id: 3.10.0-1160.25.1.el7.x86_64
0x1d5
4
xfs_inode_item_push
xfs 0xffffffffc030bd1e
Build id: 3.10.0-1160.25.1.el7.x86_64
0x14e
5
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0313a40
Build id: 3.10.0-1160.25.1.el7.x86_64
0x90
6
xfsaild
xfs 0xffffffffc0313dcf
Build id: 3.10.0-1160.25.1.el7.x86_64
0x38f
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0313a40
Build id: 3.10.0-1160.25.1.el7.x86_64
0x90
8
kthread
vmlinux 0xffffffff94ec5da1
Build id: 3.10.0-1160.25.1.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffff94ec5cd0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffff95595ddd
Build id: 3.10.0-1160.25.1.el7.x86_64
0x7
11
? insert_kthread_work
vmlinux 0xffffffff94ec5cd0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 28s! [xfsaild/sda3:326]
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:37:crtc-0] flip_done timed out
Modules linked in: veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache overlay nf_conntrack_netbios_ns nf_conntrack_broadcast xt_nat xt_mark ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash
dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul snd_ens1371 glue_helper snd_rawmidi ablk_helper snd_ac97_codec ppdev cryptd ac97_bus snd_seq joydev snd_seq_device snd_pcm vmw_balloon pcspkr snd_timer snd soundcore sg vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sch_fq_codel sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi nfit ata_piix crct10dif_pclmul crct10dif_common scsi_transport_spi libata mptscsih crc32c_intel libnvdimm serio_raw drm_panel_orientation_quirks e1000 mptbase fuse
CPU: 0 PID: 326 Comm: xfsaild/sda3 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8c17752a4200 ti: ffff8c17751d8000 task.ti: ffff8c17751d8000
RIP: 0010:[<ffffffff9558b85c>]  [<ffffffff9558b85c>] _raw_spin_lock+0xc/0x30
RSP: 0018:ffff8c17751dbcf0  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000007fffffff
RDX: 0000000000000001 RSI: 000000000008b81f RDI: ffff8c191698b19c
RBP: ffff8c17751dbd50 R08: 0000000000000021 R09: 0000000000000001
R10: ffff8c18bfc07700 R11: ffff8c1972edc000 R12: 00000000597e425e
R13: ffff8c1743593840 R14: ffff8c197194b270 R15: ffff8c17751dbca0
FS:  0000000000000000(0000) GS:ffff8c1975600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2415750880 CR3: 00000000a646c000 CR4: 00000000003607f0
Call Trace:
[<ffffffffc02f980b>] ? xfs_iflush_cluster+0x18b/0x3e0 [xfs]
[<ffffffffc02f9c35>] xfs_iflush+0x1d5/0x250 [xfs]
[<ffffffffc030bd1e>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc0313a40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0313dcf>] xfsaild+0x38f/0x780 [xfs]
[<ffffffffc0313a40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff94ec5da1>] kthread+0xd1/0xe0
[<ffffffff94ec5cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffff95595ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff94ec5cd0>] ? insert_kthread_work+0x40/0x40
Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 da 18 ff ff 5d 
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:33:plane-0] flip_done timed out
Complete report #145439
Frame # Function Binary Source or offset Line
1
_raw_spin_lock
vmlinux 0xffffffff816abbfc
Build id: 3.10.0-693.5.2.el7.x86_64
0xc
2
? xfs_iflush_cluster
xfs 0xffffffffc02bd703
Build id: 3.10.0-693.5.2.el7.x86_64
0x193
3
xfs_iflush
xfs 0xffffffffc02c13b0
Build id: 3.10.0-693.5.2.el7.x86_64
0x1e0
4
xfs_inode_item_push
xfs 0xffffffffc02d2ec2
Build id: 3.10.0-693.5.2.el7.x86_64
0x102
5
? xfs_trans_ail_cursor_first
xfs 0xffffffffc02daa30
Build id: 3.10.0-693.5.2.el7.x86_64
0x90
6
xfsaild
xfs 0xffffffffc02dad51
Build id: 3.10.0-693.5.2.el7.x86_64
0x321
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc02daa30
Build id: 3.10.0-693.5.2.el7.x86_64
0x90
8
kthread
vmlinux 0xffffffff810b099f
Build id: 3.10.0-693.5.2.el7.x86_64
0xcf
9
? insert_kthread_work
vmlinux 0xffffffff810b08d0
Build id: 3.10.0-693.5.2.el7.x86_64
0x40
10
ret_from_fork
vmlinux 0xffffffff816b4fd8
Build id: 3.10.0-693.5.2.el7.x86_64
0x58
11
? insert_kthread_work
vmlinux 0xffffffff810b08d0
Build id: 3.10.0-693.5.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [xfsaild/dm-0:406]
Modules linked in: tcp_lp nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter mvfs(OE) vmw_vsock_vmci_transport vsock coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper nfit ablk_helper cryptd vmw_balloon joydev sg pcspkr libnvdimm parport_pc parport shpchp vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
crct10dif_pclmul crct10dif_common ttm crc32c_intel mptspi drm scsi_transport_spi serio_raw mptscsih e1000 mptbase ata_piix libata i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 406 Comm: xfsaild/dm-0 Tainted: G           OEL ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff8800359ddee0 ti: ffff880035b34000 task.ti: ffff880035b34000
RIP: 0010:[<ffffffff816abbfc>]  [<ffffffff816abbfc>] _raw_spin_lock+0xc/0x30
RSP: 0018:ffff880035b37d08  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffffffffc02cb0c7 RCX: 000000000019baff
RDX: 0000000000000001 RSI: 0000000000000031 RDI: ffff88004c42c4b4
RBP: ffff880035b37d68 R08: 0000000000019c00 R09: 000000000019baff
R10: 0000000000000000 R11: 0000000000000220 R12: ffff880035b37cb0
R13: ffffffff816a982a R14: ffff880035b37c70 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff880229c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f8ac54ec000 CR3: 000000020fe07000 CR4: 00000000001407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffffc02bd703 ffff8800b5a0fe00 ffff88021d06dc00 000000000019bac0
ffff88021b2ad000 fffffff000000000 0000000000000080 ffff880189e54780
00000000ffffff8b ffff880035b37dd0 ffff880189e54874 0000000000000000
Call Trace:
[<ffffffffc02bd703>] ? xfs_iflush_cluster+0x193/0x440 [xfs]
[<ffffffffc02c13b0>] xfs_iflush+0x1e0/0x250 [xfs]
[<ffffffffc02d2ec2>] xfs_inode_item_push+0x102/0x150 [xfs]
[<ffffffffc02daa30>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc02dad51>] xfsaild+0x321/0x6f0 [xfs]
[<ffffffffc02daa30>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 a4 2a ff ff 5d 
Complete report #21599
Frame # Function Binary Source or offset Line
1
xfs_iflush_cluster
xfs 0xffffffffc06b9799
Build id: 3.10.0-1160.6.1.el7.x86_64
0x139
2
xfs_iflush
xfs 0xffffffffc06b9c15
Build id: 3.10.0-1160.6.1.el7.x86_64
0x1d5
3
xfs_inode_item_push
xfs 0xffffffffc06cbcfe
Build id: 3.10.0-1160.6.1.el7.x86_64
0x14e
4
? xfs_trans_ail_cursor_first
xfs 0xffffffffc06d3a20
Build id: 3.10.0-1160.6.1.el7.x86_64
0x90
5
xfsaild
xfs 0xffffffffc06d3daf
Build id: 3.10.0-1160.6.1.el7.x86_64
0x38f
6
? xfs_trans_ail_cursor_first
xfs 0xffffffffc06d3a20
Build id: 3.10.0-1160.6.1.el7.x86_64
0x90
7
kthread
vmlinux 0xffffffffa86c5c21
Build id: 3.10.0-1160.6.1.el7.x86_64
0xd1
8
? insert_kthread_work
vmlinux 0xffffffffa86c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
9
ret_from_fork_nospec_begin
vmlinux 0xffffffffa8d94df7
Build id: 3.10.0-1160.6.1.el7.x86_64
0x21
10
? insert_kthread_work
vmlinux 0xffffffffa86c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [xfsaild/dm-0:535]
Modules linked in: xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables br_netfilter overlay(T) tcp_lp nls_utf8 isofs binfmt_misc tun devlink bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport vmw_vmci i2c_piix4 xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul mptspi crct10dif_common ahci scsi_transport_spi crc32c_intel ata_piix libahci drm mptscsih
serio_raw libata mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ip_tables]
CPU: 3 PID: 535 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9ae3e23c0000 ti: ffff9ae3edbb8000 task.ti: ffff9ae3edbb8000
RIP: 0010:[<ffffffffc06b9799>]  [<ffffffffc06b9799>] xfs_iflush_cluster+0x139/0x3e0 [xfs]
RSP: 0018:ffff9ae3edbbbcf8  EFLAGS: 00010282
RAX: ffff9aed17ed1970 RBX: ffff9aed28993000 RCX: 0000000001ffffff
RDX: 0000000002000000 RSI: 0000000000000008 RDI: ffff9aed1a85939c
RBP: ffff9ae3edbbbd50 R08: 00000000ef9e0c10 R09: 0000000049f6176c
R10: 00000000b0e6d7ff R11: ffff9aed1b87cdf8 R12: ffff9aed00000000
R13: ffff9ae3e8bbba00 R14: ffffffffc063906b R15: ffff9ae3edbbbc88
FS:  0000000000000000(0000) GS:ffff9b01bd6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6d0dc06330 CR3: 00000001d53a2000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc06b9c15>] xfs_iflush+0x1d5/0x250 [xfs]
[<ffffffffc06cbcfe>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc06d3a20>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc06d3daf>] xfsaild+0x38f/0x780 [xfs]
[<ffffffffc06d3a20>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffa86c5c21>] kthread+0xd1/0xe0
[<ffffffffa86c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffa8d94df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa86c5b50>] ? insert_kthread_work+0x40/0x40
Code: 00 ba 01 00 00 00 48 d3 e2 48 89 d1 83 e9 01 21 c8 48 3b 45 b8 0f 85 2d 02 00 00 c6 07 00 0f 1f 40 00 49 8b 47 70 48 85 c0 74 0c <f7> 80 a8 00 00 00 df 47 00 00 75 0b 41 8b 87 d8 00 00 00 85 c0 
Complete report #21284
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff8651751d
Build id: 3.10.0-1160.6.1.el7.x86_64
0x1d
2
queued_spin_lock_slowpath
vmlinux 0xffffffff86b7b43b
Build id: 3.10.0-1160.6.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff86b89b50
Build id: 3.10.0-1160.6.1.el7.x86_64
0x20
4
xfs_iflush_cluster
xfs 0xffffffffc06d77eb
Build id: 3.10.0-1160.6.1.el7.x86_64
0x18b
5
xfs_iflush
xfs 0xffffffffc06d7c15
Build id: 3.10.0-1160.6.1.el7.x86_64
0x1d5
6
xfs_inode_item_push
xfs 0xffffffffc06e9cfe
Build id: 3.10.0-1160.6.1.el7.x86_64
0x14e
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc06f1a20
Build id: 3.10.0-1160.6.1.el7.x86_64
0x90
8
xfsaild
xfs 0xffffffffc06f1daf
Build id: 3.10.0-1160.6.1.el7.x86_64
0x38f
9
? xfs_trans_ail_cursor_first
xfs 0xffffffffc06f1a20
Build id: 3.10.0-1160.6.1.el7.x86_64
0x90
10
kthread
vmlinux 0xffffffff864c5c21
Build id: 3.10.0-1160.6.1.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff864c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffff86b94df7
Build id: 3.10.0-1160.6.1.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffff864c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [xfsaild/dm-2:790]
Modules linked in: tcp_diag inet_diag binfmt_misc ipmi_devintf ipmi_msghandler xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat_ipv4 nf_nat overlay(T) vmw_vsock_vmci_transport vsock iptable_filter coretemp crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon sg joydev pcspkr vmw_vmci parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common libata mptspi scsi_transport_spi mptscsih serio_raw mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror
dm_region_hash dm_log dm_mod fuse nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack libcrc32c br_netfilter bridge stp llc
CPU: 2 PID: 790 Comm: xfsaild/dm-2 Tainted: G             L ------------ T 3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff9b665df8e300 ti: ffff9b664c78c000 task.ti: ffff9b664c78c000
RIP: 0010:[<ffffffff8651751d>]  [<ffffffff8651751d>] native_queued_spin_lock_slowpath+0x1d/0x200
RSP: 0018:ffff9b664c78fcc8  EFLAGS: 00000293
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9b6a6243c09c
RBP: ffff9b664c78fcc8 R08: c038000000000000 R09: 6875561ee01c0000
R10: ffff9b667fc07700 R11: ffff9b696afe09fc R12: ffff9b664c78fc90
R13: ffff9b6ad65fd810 R14: ffffffffc06c180b R15: ffff9b664c78fc60
FS:  0000000000000000(0000) GS:ffff9b6d1d680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbca0f53000 CR3: 00000001407e8000 CR4: 00000000000007e0
Call Trace:
[<ffffffff86b7b43b>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff86b89b50>] _raw_spin_lock+0x20/0x30
[<ffffffffc06d77eb>] xfs_iflush_cluster+0x18b/0x3e0 [xfs]
[<ffffffffc06d7c15>] xfs_iflush+0x1d5/0x250 [xfs]
[<ffffffffc06e9cfe>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc06f1a20>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc06f1daf>] xfsaild+0x38f/0x780 [xfs]
[<ffffffffc06f1a20>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff864c5c21>] kthread+0xd1/0xe0
[<ffffffff864c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff86b94df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff864c5b50>] ? insert_kthread_work+0x40/0x40
Code: 47 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 <8b> 17 85 d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 0f 
Complete report #12917
Frame # Function Binary Source or offset Line
1
_raw_spin_lock
vmlinux 0xffffffff816abc0c
Build id: 3.10.0-693.11.1.el7.x86_64
0xc
2
? xfs_iflush_cluster
xfs 0xffffffffc024b6a3
Build id: 3.10.0-693.11.1.el7.x86_64
0x193
3
xfs_iflush
xfs 0xffffffffc024f350
Build id: 3.10.0-693.11.1.el7.x86_64
0x1e0
4
xfs_inode_item_push
xfs 0xffffffffc0260e62
Build id: 3.10.0-693.11.1.el7.x86_64
0x102
5
? xfs_trans_ail_cursor_first
xfs 0xffffffffc02689d0
Build id: 3.10.0-693.11.1.el7.x86_64
0x90
6
xfsaild
xfs 0xffffffffc0268cf1
Build id: 3.10.0-693.11.1.el7.x86_64
0x321
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc02689d0
Build id: 3.10.0-693.11.1.el7.x86_64
0x90
8
kthread
vmlinux 0xffffffff810b099f
Build id: 3.10.0-693.11.1.el7.x86_64
0xcf
9
? insert_kthread_work
vmlinux 0xffffffff810b08d0
Build id: 3.10.0-693.11.1.el7.x86_64
0x40
10
ret_from_fork
vmlinux 0xffffffff816b4fd8
Build id: 3.10.0-693.11.1.el7.x86_64
0x58
11
? insert_kthread_work
vmlinux 0xffffffff810b08d0
Build id: 3.10.0-693.11.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 43s! [xfsaild/dm-0:419]
Modules linked in: binfmt_misc tun snd_hda_codec_generic iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_hda_intel lrw snd_hda_codec gf128mul glue_helper snd_hda_core ablk_helper snd_hwdep snd_seq cryptd snd_seq_device snd_pcm snd_timer snd ppdev soundcore i2c_piix4 pcspkr joydev sg parport_pc virtio_balloon parport ip_tables xfs libcrc32c sr_mod cdrom qxl ata_generic pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt virtio_blk fb_sys_fops virtio_net virtio_console ttm drm ata_piix crct10dif_pclmul crct10dif_common floppy libata crc32c_intel i2c_core serio_raw virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 419 Comm: xfsaild/dm-0 Tainted: G             L ------------   3.10.0-693.11.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Ubuntu-1.8.2-1ubuntu1 04/01/2014
task: ffff8800b9a78fd0 ti: ffff8800b9e4c000 task.ti: ffff8800b9e4c000
RIP: 0010:[<ffffffff816abc0c>]  [<ffffffff816abc0c>] _raw_spin_lock+0xc/0x30
RSP: 0018:ffff8800b9e4fd08  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff880069b280f4 RCX: 0000000003ffffff
RDX: 0000000000000001 RSI: 0000000000000008 RDI: ffff880069b280f4
RBP: ffff8800b9e4fd68 R08: 0000000000000064 R09: ffff8800bc342200
R10: ffff880088c1e964 R11: ffff8800b9f24000 R12: ffff8800bc341a40
R13: ffff88005f378400 R14: 00000000000000a0 R15: ffff880069b28000
FS:  0000000000000000(0000) GS:ffff8800c1280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000016c90001 CR3: 00000000019f2000 CR4: 00000000003406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffffc024b6a3 ffff880007b1aa80 ffff8800bad489c0 00000000002fb6a0
ffff8800b9f24000 ffffffe000000002 0000000000000100 ffff8800bc341a40
00000000ffffff8b ffff8800b9e4fdd0 ffff8800bc341b34 0000000000000000
Call Trace:
[<ffffffffc024b6a3>] ? xfs_iflush_cluster+0x193/0x440 [xfs]
[<ffffffffc024f350>] xfs_iflush+0x1e0/0x250 [xfs]
[<ffffffffc0260e62>] xfs_inode_item_push+0x102/0x150 [xfs]
[<ffffffffc02689d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0268cf1>] xfsaild+0x321/0x6f0 [xfs]
[<ffffffffc02689d0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 a4 2a ff ff 5d 
Complete report #4088
Frame # Function Binary Source or offset Line
1
_raw_spin_lock
vmlinux 0xffffffffb2d8331c
Build id: 3.10.0-1062.9.1.el7.x86_64
0xc
2
? xfs_iflush_cluster
xfs 0xffffffffc042428b
Build id: 3.10.0-1062.9.1.el7.x86_64
0x18b
3
xfs_iflush
xfs 0xffffffffc0427650
Build id: 3.10.0-1062.9.1.el7.x86_64
0x1e0
4
xfs_inode_item_push
xfs 0xffffffffc04396de
Build id: 3.10.0-1062.9.1.el7.x86_64
0x14e
5
? xfs_trans_ail_cursor_first
xfs 0xffffffffc04413b0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x90
6
xfsaild
xfs 0xffffffffc044171f
Build id: 3.10.0-1062.9.1.el7.x86_64
0x36f
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc04413b0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x90
8
kthread
vmlinux 0xffffffffb26c61f1
Build id: 3.10.0-1062.9.1.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffffb26c6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffffb2d8dd1d
Build id: 3.10.0-1062.9.1.el7.x86_64
0x7
11
? insert_kthread_work
vmlinux 0xffffffffb26c6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [xfsaild/dm-0:546]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter binfmt_misc sunrpc nfit libnvdimm ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper sg ablk_helper cryptd joydev pcspkr virtio_balloon parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom cirrus ata_generic drm_kms_helper pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm virtio_console virtio_blk ata_piix crct10dif_pclmul drm 8139too libata crct10dif_common crc32c_intel virtio_pci virtio_ring serio_raw 8139cp floppy drm_panel_orientation_quirks
mii virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 546 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff8b3231dd3150 ti: ffff8b30f5174000 task.ti: ffff8b30f5174000
RIP: 0010:[<ffffffffb2d8331c>]  [<ffffffffb2d8331c>] _raw_spin_lock+0xc/0x30
RSP: 0000:ffff8b30f5177cf0  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8b31d7a18c14 RCX: 0000000001ffffff
RDX: 0000000000000001 RSI: 0000000000000040 RDI: ffff8b31d7a18c14
RBP: ffff8b30f5177d50 R08: 0000000000000000 R09: 0000000000000001
R10: ffff8b3239c03700 R11: ffff8b32324ea5f8 R12: ffff8b3233a75640
R13: ffff8b3233000e00 R14: 0000000000000040 R15: ffff8b31d7a18b40
FS:  0000000000000000(0000) GS:ffff8b323a180000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000455b0003 CR3: 0000000148410000 CR4: 00000000007606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffc042428b>] ? xfs_iflush_cluster+0x18b/0x440 [xfs]
[<ffffffffc0427650>] xfs_iflush+0x1e0/0x260 [xfs]
[<ffffffffc04396de>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc04413b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc044171f>] xfsaild+0x36f/0x760 [xfs]
[<ffffffffc04413b0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffb26c61f1>] kthread+0xd1/0xe0
[<ffffffffb26c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffffb2d8dd1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb26c6120>] ? insert_kthread_work+0x40/0x40
Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 0f 21 ff ff 5d 
Complete report #3143
Frame # Function Binary Source or offset Line
1
native_queued_spin_lock_slowpath
vmlinux 0xffffffff9bd18093
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1d3
2
queued_spin_lock_slowpath
vmlinux 0xffffffff9c37a024
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb
3
_raw_spin_lock
vmlinux 0xffffffff9c3886d0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20
4
xfs_iflush_cluster
xfs 0xffffffffc049b75b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x18b
5
xfs_iflush
xfs 0xffffffffc049bb85
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1d5
6
xfs_inode_item_push
xfs 0xffffffffc04adc5e
Build id: 3.10.0-1127.19.1.el7.x86_64
0x14e
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc04b59a0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x90
8
xfsaild
xfs 0xffffffffc04b5d0f
Build id: 3.10.0-1127.19.1.el7.x86_64
0x36f
9
? xfs_trans_ail_cursor_first
xfs 0xffffffffc04b59a0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x90
10
kthread
vmlinux 0xffffffff9bcc6691
Build id: 3.10.0-1127.19.1.el7.x86_64
0xd1
11
? insert_kthread_work
vmlinux 0xffffffff9bcc65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffff9c392d37
Build id: 3.10.0-1127.19.1.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffff9bcc65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [xfsaild/dm-0:457]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nf_nat_ftp nf_conntrack_ftp nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw vboxnetadp(OE) nf_conntrack vboxnetflt(OE) ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vboxdrv(OE) sunrpc ext4 mbcache jbd2 ppdev mei_wdt iTCO_wdt iTCO_vendor_support joydev intel_powerclamp coretemp kvm_intel kvm snd_hda_codec_hdmi snd_hda_codec_realtek
snd_hda_codec_generic snd_hda_intel snd_hda_codec raid1 snd_hda_core snd_hwdep snd_seq irqbypass snd_seq_device tpm_infineon snd_pcm sg snd_timer snd soundcore parport_pc pcspkr mei_me i2c_i801 parport asus_atk0110 mei lpc_ich acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi i915 i2c_algo_bit video iosf_mbi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci drm libahci pata_via crc32c_intel libata serio_raw r8169 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]
CPU: 1 PID: 457 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G           OE  ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: System manufacturer System Product Name/P7H55-M, BIOS 1101    08/18/2010
task: ffff9ed5759a8000 ti: ffff9ed575878000 task.ti: ffff9ed575878000
RIP: 0010:[<ffffffff9bd18093>]  [<ffffffff9bd18093>] native_queued_spin_lock_slowpath+0x1d3/0x200
RSP: 0018:ffff9ed57587bcc8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000001
RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff9ed541a98414
RBP: ffff9ed57587bcc8 R08: 0000000000000101 R09: 0000000000000001
R10: ffff9ed6bfc03700 R11: ffff9ed575956000 R12: ffff9ed57587bc90
R13: ffff9ed61aead6b0 R14: ffffffffc048587b R15: ffff9ed57587bc60
FS:  0000000000000000(0000) GS:ffff9ed757240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006e6bd4 CR3: 00000000d9610000 CR4: 00000000000207e0
Call Trace:
[<ffffffff9c37a024>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff9c3886d0>] _raw_spin_lock+0x20/0x30
[<ffffffffc049b75b>] xfs_iflush_cluster+0x18b/0x3e0 [xfs]
[<ffffffffc049bb85>] xfs_iflush+0x1d5/0x250 [xfs]
[<ffffffffc04adc5e>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc04b59a0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc04b5d0f>] xfsaild+0x36f/0x760 [xfs]
[<ffffffffc04b59a0>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff9bcc6691>] kthread+0xd1/0xe0
[<ffffffff9bcc65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff9c392d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9bcc65c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 74 f4 e9 93 fe ff ff 0f 1f 80 00 00 00 00 83 fa 01 75 11 0f 1f 00 e9 68 fe ff ff 0f 1f 00 85 c0 74 0c f3 90 8b 07 0f b6 c0 <83> f8 03 75 f0 b8 01 00 00 00 66 89 07 5d c3 66 0f 1f 44 00 00 
Complete report #13624
Frame # Function Binary Source or offset Line
1
xfs_inode_to_disk
xfs 0xffffffffc028702e
Build id: 3.10.0-693.el7.x86_64
0xe
2
? xfs_iflush_int
xfs 0xffffffffc02a3ebd
Build id: 3.10.0-693.el7.x86_64
0xbd
3
xfs_iflush
xfs 0xffffffffc02a836c
Build id: 3.10.0-693.el7.x86_64
0x1bc
4
xfs_inode_item_push
xfs 0xffffffffc02b9ea2
Build id: 3.10.0-693.el7.x86_64
0x102
5
? xfs_trans_ail_cursor_first
xfs 0xffffffffc02c1a10
Build id: 3.10.0-693.el7.x86_64
0x90
6
xfsaild
xfs 0xffffffffc02c1d31
Build id: 3.10.0-693.el7.x86_64
0x321
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc02c1a10
Build id: 3.10.0-693.el7.x86_64
0x90
8
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
9
? do_exit
vmlinux 0xffffffff8108ddeb
Build id: 3.10.0-693.el7.x86_64
0x6bb
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
11
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
12
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 26s! [xfsaild/dm-2:958]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq sb_edac snd_seq_device edac_core snd_pcm iosf_mbi crc32_pclmul ghash_clmulni_intel snd_timer snd aesni_intel soundcore nfit lrw vmw_balloon gf128mul glue_helper ppdev ablk_helper joydev pcspkr vmw_vmci cryptd libnvdimm sg i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic
pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mptspi ttm scsi_transport_spi ata_piix mptscsih crct10dif_pclmul drm libata e1000 crct10dif_common mptbase crc32c_intel serio_raw i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 958 Comm: xfsaild/dm-2 Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff88043f8ddee0 ti: ffff880420760000 task.ti: ffff880420760000
RIP: 0010:[<ffffffffc028702e>]  [<ffffffffc028702e>] xfs_inode_to_disk+0xe/0x210 [xfs]
RSP: 0018:ffff880420763d20  EFLAGS: 00010246
RAX: 0000000000000023 RBX: ffff88042639f000 RCX: 0000000000004e49
RDX: 000000010003449b RSI: ffff8803e83ca000 RDI: ffff8803c49f2d00
RBP: ffff880420763d68 R08: 0000000000000000 R09: 0000000000000023
R10: 0000000000000000 R11: 0000000000000001 R12: ffff880428deb2c0
R13: ffffffffc02d6e00 R14: ffffffffc0293180 R15: ffff880420763cc0
FS:  0000000000000000(0000) GS:ffff88042d780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9e48ffc000 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff880420763d68 ffffffffc02a3ebd ffffffe000000020 00000000ed8e43dc
ffff8803c49f2d00 ffff88042639f000 ffff880420763dd0 ffff8803c49f2df4
0000000000000000 ffff880420763dc0 ffffffffc02a836c ffff880400000000
Call Trace:
[<ffffffffc02a3ebd>] ? xfs_iflush_int+0xbd/0x2b0 [xfs]
[<ffffffffc02a836c>] xfs_iflush+0x1bc/0x250 [xfs]
[<ffffffffc02b9ea2>] xfs_inode_item_push+0x102/0x150 [xfs]
[<ffffffffc02c1a10>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc02c1d31>] xfsaild+0x321/0x6f0 [xfs]
[<ffffffffc02c1a10>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 89 83 14 01 00 00 ba 02 00 00 00 e9 3b fe ff ff 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 45 31 c0 b9 49 4e 00 00 55 <66> 89 0e 66 44 89 46 06 45 31 c9 0f b6 87 08 01 00 00 48 89 e5 
Complete report #804133
Frame # Function Binary Source or offset Line
1
xfs_inode_to_disk
xfs 0xffffffffc054514e
Build id: 3.10.0-1160.95.1.el7.x86_64
0xe
2
? xfs_iflush_int
xfs 0xffffffffc0566fbf
Build id: 3.10.0-1160.95.1.el7.x86_64
0xcf
3
xfs_iflush
xfs 0xffffffffc056773c
Build id: 3.10.0-1160.95.1.el7.x86_64
0x1bc
4
xfs_inode_item_push
xfs 0xffffffffc0579f26
Build id: 3.10.0-1160.95.1.el7.x86_64
0x156
5
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0581e70
Build id: 3.10.0-1160.95.1.el7.x86_64
0xa0
6
xfsaild
xfs 0xffffffffc05821ff
Build id: 3.10.0-1160.95.1.el7.x86_64
0x38f
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0581e70
Build id: 3.10.0-1160.95.1.el7.x86_64
0xa0
8
kthread
vmlinux 0xffffffffad8cb621
Build id: 3.10.0-1160.95.1.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffffad8cb550
Build id: 3.10.0-1160.95.1.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffffadfc51f7
Build id: 3.10.0-1160.95.1.el7.x86_64
0x21
11
? insert_kthread_work
vmlinux 0xffffffffad8cb550
Build id: 3.10.0-1160.95.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 3072s! [xfsaild/dm-0:494]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT devlink nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc vfat fat snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device iTCO_wdt snd_pcm iTCO_vendor_support joydev snd_timer pcspkr sg i2c_i801 virtio_rng lpc_ich snd soundcore ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common
virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm virtio_console drm_panel_orientation_quirks uas usb_storage ahci libahci libata virtio_pci virtio_ring e1000 virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 5 PID: 494 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.95.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 2/2/2022
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 3072s! [xfsaild/dm-0:494]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT devlink nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc vfat fat snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device iTCO_wdt snd_pcm iTCO_vendor_support joydev snd_timer pcspkr sg i2c_i801 virtio_rng lpc_ich snd soundcore ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common
virtio_gpu drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm virtio_console drm_panel_orientation_quirks uas usb_storage ahci libahci libata virtio_pci virtio_ring e1000 virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 5 PID: 494 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.95.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS unknown 2/2/2022
RIP: 0010:[<ffffffffc054514e>]  [<ffffffffc054514e>] xfs_inode_to_disk+0xe/0x210 [xfs]
RSP: 0018:ffff8f5a8239bd08  EFLAGS: 00010246
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000004e49
RDX: 0000001e000045ac RSI: ffff8f5a85c7da00 RDI: ffff8f59c2b58400
RBP: ffff8f5a8239bd50 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000014e1f40 R11: 0000000000000001 R12: ffffffffc0592040
R13: ffffffffc05514a0 R14: ffff8f5a8239bca8 R15: ffff8f5a842f7540
FS:  0000000000000000(0000) GS:ffff8f5a89940000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe809704354 CR3: 00000006486d0000 CR4: 00000000000026e0
Call Trace:
[<ffffffffc0566fbf>] ? xfs_iflush_int+0xcf/0x2b0 [xfs]
[<ffffffffc056773c>] xfs_iflush+0x1bc/0x250 [xfs]
[<ffffffffc0579f26>] xfs_inode_item_push+0x156/0x1a0 [xfs]
[<ffffffffc0581e70>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [xfs]
[<ffffffffc05821ff>] xfsaild+0x38f/0x780 [xfs]
[<ffffffffc0581e70>] ? xfs_trans_ail_cursor_first+0xa0/0xa0 [xfs]
[<ffffffffad8cb621>] kthread+0xd1/0xe0
[<ffffffffad8cb550>] ? insert_kthread_work+0x40/0x40
[<ffffffffadfc51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffad8cb550>] ? insert_kthread_work+0x40/0x40
Code: 89 83 0c 01 00 00 ba 02 00 00 00 e9 3b fe ff ff 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 45 31 c0 b9 49 4e 00 00 55 <66> 89 0e 66 44 89 46 06 45 31 c9 0f b6 87 00 01 00 00 48 89 e5 
Complete report #332382
Frame # Function Binary Source or offset Line
1
xfs_inode_to_disk
xfs 0xffffffffc04f1f0e
Build id: 3.10.0-1160.36.2.el7.x86_64
0xe
2
? xfs_iflush_int
xfs 0xffffffffc051349f
Build id: 3.10.0-1160.36.2.el7.x86_64
0xcf
3
xfs_iflush
xfs 0xffffffffc0513c14
Build id: 3.10.0-1160.36.2.el7.x86_64
0x1b4
4
xfs_inode_item_push
xfs 0xffffffffc0525d1e
Build id: 3.10.0-1160.36.2.el7.x86_64
0x14e
5
? xfs_trans_ail_cursor_first
xfs 0xffffffffc052da40
Build id: 3.10.0-1160.36.2.el7.x86_64
0x90
6
xfsaild
xfs 0xffffffffc052ddcf
Build id: 3.10.0-1160.36.2.el7.x86_64
0x38f
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc052da40
Build id: 3.10.0-1160.36.2.el7.x86_64
0x90
8
kthread
vmlinux 0xffffffffaa6c5e41
Build id: 3.10.0-1160.36.2.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffffaa6c5d70
Build id: 3.10.0-1160.36.2.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffffaad95df7
Build id: 3.10.0-1160.36.2.el7.x86_64
0x21
11
? insert_kthread_work
vmlinux 0xffffffffaa6c5d70
Build id: 3.10.0-1160.36.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 52s! [xfsaild/dm-2:3485]
Modules linked in: xt_multiport binfmt_misc btrfs raid6_pq xor xt_CHECKSUM tun devlink xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 af_key ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac ppdev coretemp sg pcspkr vmw_balloon iosf_mbi joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c dm_crypt drbg sr_mod sd_mod cdrom
crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel drm_kms_helper ata_piix ghash_clmulni_intel syscopyarea sysfillrect sysimgblt fb_sys_fops libata ttm aesni_intel mptspi scsi_transport_spi drm glue_helper mptscsih lrw gf128mul ablk_helper cryptd mptbase serio_raw vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 3485 Comm: xfsaild/dm-2 Kdump: loaded Tainted: G    B        L ------------   3.10.0-1160.36.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff8ce7f154e300 ti: ffff8ce7f02b8000 task.ti: ffff8ce7f02b8000
RIP: 0010:[<ffffffffc04f1f0e>]  [<ffffffffc04f1f0e>] xfs_inode_to_disk+0xe/0x210 [xfs]
RSP: 0018:ffff8ce7f02bbd08  EFLAGS: 00010246
RAX: 0000000000000001 RBX: ffff8ce71523612b RCX: 0000000000004e49
RDX: 0000084d00007db4 RSI: ffff8ce79e104900 RDI: ffff8ce7915c6940
RBP: ffff8ce7f02bbd50 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000004000000 R11: 0000000000000001 R12: ffffffffc04df127
R13: ffff8ce7f02bbc88 R14: 000000000407958b R15: ffff8ce7ef859000
FS:  0000000000000000(0000) GS:ffff8ce7ffc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000e26158 CR3: 0000000132236000 CR4: 00000000000607f0
Call Trace:
[<ffffffffc051349f>] ? xfs_iflush_int+0xcf/0x2b0 [xfs]
[<ffffffffc0513c14>] xfs_iflush+0x1b4/0x250 [xfs]
[<ffffffffc0525d1e>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc052da40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc052ddcf>] xfsaild+0x38f/0x780 [xfs]
[<ffffffffc052da40>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffaa6c5e41>] kthread+0xd1/0xe0
[<ffffffffaa6c5d70>] ? insert_kthread_work+0x40/0x40
[<ffffffffaad95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffaa6c5d70>] ? insert_kthread_work+0x40/0x40
Code: 89 83 0c 01 00 00 ba 02 00 00 00 e9 3b fe ff ff 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 45 31 c0 b9 49 4e 00 00 55 <66> 89 0e 66 44 89 46 06 45 31 c9 0f b6 87 00 01 00 00 48 89 e5 
Complete report #328444
Frame # Function Binary Source or offset Line
1
xfs_iflush_int
xfs 0xffffffffc07103df
Build id: 3.10.0-1160.el7.x86_64
0x2f
2
xfs_iflush
xfs 0xffffffffc0710bf4
Build id: 3.10.0-1160.el7.x86_64
0x1b4
3
xfs_inode_item_push
xfs 0xffffffffc0722cfe
Build id: 3.10.0-1160.el7.x86_64
0x14e
4
? xfs_trans_ail_cursor_first
xfs 0xffffffffc072aa20
Build id: 3.10.0-1160.el7.x86_64
0x90
5
xfsaild
xfs 0xffffffffc072adaf
Build id: 3.10.0-1160.el7.x86_64
0x38f
6
? xfs_trans_ail_cursor_first
xfs 0xffffffffc072aa20
Build id: 3.10.0-1160.el7.x86_64
0x90
7
kthread
vmlinux 0xffffffffa4ac5c21
Build id: 3.10.0-1160.el7.x86_64
0xd1
8
? insert_kthread_work
vmlinux 0xffffffffa4ac5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
9
ret_from_fork_nospec_begin
vmlinux 0xffffffffa5193df7
Build id: 3.10.0-1160.el7.x86_64
0x21
10
? insert_kthread_work
vmlinux 0xffffffffa4ac5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 26s! [xfsaild/dm-0:445]
Modules linked in: rfcomm ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache iosf_mbi jbd2 crc32_pclmul ghash_clmulni_intel ppdev aesni_intel snd_ens1371 vmw_balloon btusb snd_rawmidi btrtl lrw btbcm gf128mul btintel glue_helper snd_ac97_codec ablk_helper bluetooth cryptd ac97_bus snd_seq snd_seq_device snd_pcm joydev pcspkr
rfkill snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper sr_mod cdrom syscopyarea sysfillrect sysimgblt fb_sys_fops ata_generic ttm pata_acpi drm ata_piix crct10dif_pclmul crct10dif_common nfit libnvdimm libata mptspi scsi_transport_spi mptscsih crc32c_intel e1000 mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 445 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffffa020ac3dc200 ti: ffffa020a9da4000 task.ti: ffffa020a9da4000
RIP: 0010:[<ffffffffc07103df>]  [<ffffffffc07103df>] xfs_iflush_int+0x2f/0x2b0 [xfs]
RSP: 0018:ffffa020a9da7d18  EFLAGS: 00010286
RAX: ffffa01d915e3400 RBX: ffffa020a4869e28 RCX: ffffa02057099b00
RDX: ffffa01d915e3000 RSI: 0000000000001400 RDI: ffffa01fe971de00
RBP: ffffa020a9da7d50 R08: 0000000000000000 R09: 0000000000030000
R10: ffffa020ad65f120 R11: 0000000000000001 R12: 0000000000000000
R13: ffffa020ac1c9980 R14: ffffffffc073b040 R15: ffffffffc06faeb0
FS:  0000000000000000(0000) GS:ffffa020ad640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3c87fe7000 CR3: 000000010059e000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc0710bf4>] xfs_iflush+0x1b4/0x250 [xfs]
[<ffffffffc0722cfe>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc072aa20>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc072adaf>] xfsaild+0x38f/0x780 [xfs]
[<ffffffffc072aa20>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffa4ac5c21>] kthread+0xd1/0xe0
[<ffffffffa4ac5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffa5193df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa4ac5b50>] ? insert_kthread_work+0x40/0x40
Code: 00 55 48 89 e5 41 57 49 89 f7 41 56 41 55 41 54 53 48 89 fb 48 83 ec 10 0f b7 77 32 4c 8b 67 70 4c 8b 37 4c 89 ff e8 11 b0 fe ff <44> 0f b7 08 49 89 c5 66 41 81 f9 49 4e 0f 85 de 01 00 00 0f b7 
Complete report #309220
Frame # Function Binary Source or offset Line
1
xfs_iflush_fork
xfs 0xffffffffc05ee510
Build id: 3.10.0-957.el7.x86_64
-
2
? xfs_iflush_int
xfs 0xffffffffc060e179
Build id: 3.10.0-957.el7.x86_64
0xe9
3
xfs_iflush
xfs 0xffffffffc0611f3c
Build id: 3.10.0-957.el7.x86_64
0x1bc
4
xfs_inode_item_push
xfs 0xffffffffc062416e
Build id: 3.10.0-957.el7.x86_64
0x14e
5
? xfs_trans_ail_cursor_first
xfs 0xffffffffc062bc60
Build id: 3.10.0-957.el7.x86_64
0x90
6
xfsaild
xfs 0xffffffffc062bf87
Build id: 3.10.0-957.el7.x86_64
0x327
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc062bc60
Build id: 3.10.0-957.el7.x86_64
0x90
8
kthread
vmlinux 0xffffffff94ec1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffff94ec1b60
Build id: 3.10.0-957.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffff95574c37
Build id: 3.10.0-957.el7.x86_64
0x21
11
? insert_kthread_work
vmlinux 0xffffffff94ec1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [xfsaild/dm-0:3267]
Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ppdev coretemp iosf_mbi crc32_pclmul vmw_balloon ghash_clmulni_intel cryptd joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi vmxnet3 vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci ata_piix mptspi libata drm_panel_orientation_quirks scsi_transport_spi mptscsih mptbase dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3267 Comm: xfsaild/dm-0 Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff97fff2da4100 ti: ffff97fff2f78000 task.ti: ffff97fff2f78000
RIP: 0010:[<ffffffffc05ee510>]  [<ffffffffc05ee510>] xfs_iflush_fork+0x0/0x220 [xfs]
RSP: 0018:ffff97fff2f7bd20  EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000001aca000 RCX: 0000000000000000
RDX: ffff97ff73fdbe30 RSI: ffff97fff5ae5a00 RDI: ffff97ff6c2a7080
RBP: ffff97fff2f7bd60 R08: 0000000000000000 R09: 0000000000000000
R10: 000000000000b78f R11: 0000000000000001 R12: 000000000000b78f
R13: 0000000000000001 R14: ffffffffc063bf00 R15: ffff97fff2f7bd60
FS:  0000000000000000(0000) GS:ffff97fff9600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4d6d519000 CR3: 00000000b8d3e000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc060e179>] ? xfs_iflush_int+0xe9/0x2e0 [xfs]
[<ffffffffc0611f3c>] xfs_iflush+0x1bc/0x250 [xfs]
[<ffffffffc062416e>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc062bc60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc062bf87>] xfsaild+0x327/0x6f0 [xfs]
[<ffffffffc062bc60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff94ec1c31>] kthread+0xd1/0xe0
[<ffffffff94ec1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff95574c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff94ec1b60>] ? insert_kthread_work+0x40/0x40
Code: c4 10 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 44 00 00 48 8b 47 38 48 89 45 d0 e9 64 ff ff ff 31 c0 eb da 0f 1f 80 00 00 00 00 <0f> 1f 44 00 00 55 48 85 d2 48 89 e5 0f 84 c6 00 00 00 85 c9 0f 
Complete report #193080
Frame # Function Binary Source or offset Line
1
xfs_iflush_int
xfs 0xffffffffc0297e32
Build id: 3.10.0-693.el7.x86_64
0x32
2
xfs_iflush
xfs 0xffffffffc029c36c
Build id: 3.10.0-693.el7.x86_64
0x1bc
3
xfs_inode_item_push
xfs 0xffffffffc02adea2
Build id: 3.10.0-693.el7.x86_64
0x102
4
? xfs_trans_ail_cursor_first
xfs 0xffffffffc02b5a10
Build id: 3.10.0-693.el7.x86_64
0x90
5
xfsaild
xfs 0xffffffffc02b5d31
Build id: 3.10.0-693.el7.x86_64
0x321
6
? xfs_trans_ail_cursor_first
xfs 0xffffffffc02b5a10
Build id: 3.10.0-693.el7.x86_64
0x90
7
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
8
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
9
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 50s! [xfsaild/dm-0:400]
Modules linked in: xt_nat veth ip6table_mangle ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) vmhgfs(OE) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device coretemp iosf_mbi crc32_pclmul snd_pcm ghash_clmulni_intel snd_timer ppdev snd soundcore aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon
joydev pcspkr sg nfit libnvdimm parport_pc parport vmw_vmci i2c_piix4 shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix serio_raw mptspi scsi_transport_spi libata mptscsih mptbase e1000 i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 400 Comm: xfsaild/dm-0 Tainted: G           OEL ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff880034468000 ti: ffff8800347e4000 task.ti: ffff8800347e4000
RIP: 0010:[<ffffffffc0297e32>]  [<ffffffffc0297e32>] xfs_iflush_int+0x32/0x2b0 [xfs]
RSP: 0000:ffff8800347e7d30  EFLAGS: 00010286
RAX: ffff88003b292000 RBX: ffff8800347e7d28 RCX: ffff88002f1bee80
RDX: ffff88003b292000 RSI: 0000000000002000 RDI: ffff880039f7b000
RBP: ffff8800347e7d68 R08: 0000000000000000 R09: 0000000000030000
R10: 0000000000000000 R11: 0000000000000001 R12: ffff880035ad6000
R13: 0000000000000000 R14: ffff880034555140 R15: ffffffffc02cae00
FS:  0000000000000000(0000) GS:ffff88003c600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000019cf7ac CR3: 000000003a16a000 CR4: 00000000003407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffe000000020 0000000019d1b27c ffff88001f9ee540 ffff880035ad6000
ffff8800347e7dd0 ffff88001f9ee634 0000000000000000 ffff8800347e7dc0
ffffffffc029c36c ffff880000000000 ffff880039f7b000 ffff88003b292000
Call Trace:
[<ffffffffc029c36c>] xfs_iflush+0x1bc/0x250 [xfs]
[<ffffffffc02adea2>] xfs_inode_item_push+0x102/0x150 [xfs]
[<ffffffffc02b5a10>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc02b5d31>] xfsaild+0x321/0x6f0 [xfs]
[<ffffffffc02b5a10>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 89 e5 41 57 49 89 f7 41 56 41 55 41 54 53 48 89 fb 48 83 ec 10 0f b7 77 32 4c 8b a7 88 00 00 00 4c 8b 37 4c 89 ff e8 3e f6 fe ff <44> 0f b7 08 49 89 c5 66 41 81 f9 49 4e 0f 85 cb 01 00 00 0f b7 
Complete report #83349
Frame # Function Binary Source or offset Line
1
xfs_iflush_int
xfs 0xffffffffc06750c2
Build id: 3.10.0-957.el7.x86_64
0x32
2
xfs_iflush_cluster
xfs 0xffffffffc0675bae
Build id: 3.10.0-957.el7.x86_64
0x2ae
3
xfs_iflush
xfs 0xffffffffc0678f60
Build id: 3.10.0-957.el7.x86_64
0x1e0
4
xfs_inode_item_push
xfs 0xffffffffc068b16e
Build id: 3.10.0-957.el7.x86_64
0x14e
5
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0692c60
Build id: 3.10.0-957.el7.x86_64
0x90
6
xfsaild
xfs 0xffffffffc0692f87
Build id: 3.10.0-957.el7.x86_64
0x327
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0692c60
Build id: 3.10.0-957.el7.x86_64
0x90
8
kthread
vmlinux 0xffffffff88ec1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffff88ec1b60
Build id: 3.10.0-957.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffff89574c37
Build id: 3.10.0-957.el7.x86_64
0x21
11
? insert_kthread_work
vmlinux 0xffffffff88ec1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [xfsaild/dm-0:3129]
Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev vmw_balloon cryptd pcspkr joydev snd_seq_midi parport_pc parport snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd i2c_piix4 soundcore
sg vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod crct10dif_pclmul crct10dif_common cdrom crc32c_intel serio_raw ata_generic pata_acpi e1000 nfit libnvdimm vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi drm ata_piix scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 3129 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff987772a830c0 ti: ffff987774bc0000 task.ti: ffff987774bc0000
RIP: 0010:[<ffffffffc06750c2>]  [<ffffffffc06750c2>] xfs_iflush_int+0x32/0x2e0 [xfs]
RSP: 0018:ffff987774bc3cc0  EFLAGS: 00010286
RAX: ffff987776c23600 RBX: ffff987774bc3ca0 RCX: 0000000000000000
RDX: ffff987776c23000 RSI: 0000000000000600 RDI: ffff9877782a2880
RBP: ffff987774bc3cf8 R08: 000000000000002a R09: 0000000000000001
R10: ffff98777f807700 R11: ffff987776c243f8 R12: ffffffff88e36a4f
R13: ffff987774bc3c38 R14: 0000000000000200 R15: ffff9877782a2880
FS:  0000000000000000(0000) GS:ffff98777b680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd0cbbc1000 CR3: 0000000078b52000 CR4: 00000000000607e0
Call Trace:
[<ffffffffc0675bae>] xfs_iflush_cluster+0x2ae/0x440 [xfs]
[<ffffffffc0678f60>] xfs_iflush+0x1e0/0x250 [xfs]
[<ffffffffc068b16e>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc0692c60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0692f87>] xfsaild+0x327/0x6f0 [xfs]
[<ffffffffc0692c60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff88ec1c31>] kthread+0xd1/0xe0
[<ffffffff88ec1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff89574c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff88ec1b60>] ? insert_kthread_work+0x40/0x40
Code: 89 e5 41 57 49 89 f7 41 56 41 55 41 54 53 48 89 fb 48 83 ec 10 0f b7 77 32 4c 8b a7 88 00 00 00 4c 8b 37 4c 89 ff e8 1e e8 fe ff <44> 0f b7 08 49 89 c5 66 41 81 f9 49 4e 0f 85 fb 01 00 00 0f b7 
Complete report #18700
Frame # Function Binary Source or offset Line
1
xfs_iflush_int
xfs 0xffffffffc06364e2
Build id: 3.10.0-862.3.2.el7.x86_64
0x32
2
xfs_iflush
xfs 0xffffffffc063a26c
Build id: 3.10.0-862.3.2.el7.x86_64
0x1bc
3
xfs_inode_item_push
xfs 0xffffffffc064c3ee
Build id: 3.10.0-862.3.2.el7.x86_64
0x14e
4
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0653e80
Build id: 3.10.0-862.3.2.el7.x86_64
0x90
5
xfsaild
xfs 0xffffffffc06541a7
Build id: 3.10.0-862.3.2.el7.x86_64
0x327
6
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0653e80
Build id: 3.10.0-862.3.2.el7.x86_64
0x90
7
kthread
vmlinux 0xffffffff950bb161
Build id: 3.10.0-862.3.2.el7.x86_64
0xd1
8
? insert_kthread_work
vmlinux 0xffffffff950bb090
Build id: 3.10.0-862.3.2.el7.x86_64
0x40
9
ret_from_fork_nospec_begin
vmlinux 0xffffffff95720677
Build id: 3.10.0-862.3.2.el7.x86_64
0x21
10
? insert_kthread_work
vmlinux 0xffffffff950bb090
Build id: 3.10.0-862.3.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [xfsaild/dm-0:394]
Modules linked in: vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev sg pcspkr nfit libnvdimm parport_pc shpchp parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata mptspi serio_raw scsi_transport_spi mptscsih e1000 mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 394 Comm: xfsaild/dm-0 Kdump: loaded Tainted: G             L ------------   3.10.0-862.3.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff97f4eb7a6eb0 ti: ffff97f0f67d4000 task.ti: ffff97f0f67d4000
RIP: 0010:[<ffffffffc06364e2>]  [<ffffffffc06364e2>] xfs_iflush_int+0x32/0x2e0 [xfs]
RSP: 0000:ffff97f0f67d7d28  EFLAGS: 00010286
RAX: ffff97f4e9a2c400 RBX: ffff97f4e9ad9000 RCX: ffff97f48d3be900
RDX: ffff97f4e9a2c000 RSI: 0000000000002400 RDI: ffff97f4ec2c5300
RBP: ffff97f0f67d7d60 R08: 0000000000000000 R09: 0000000000030000
R10: ffff97f4ffc1bac0 R11: 0000000000000001 R12: 0000000000000000
R13: ffff97f4e8e2da80 R14: ffffffffc0669230 R15: ffffffffc06250f0
FS:  0000000000000000(0000) GS:ffff97f4ffc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f53196b6000 CR3: 000000043f8a4000 CR4: 00000000000407f0
Call Trace:
[<ffffffffc063a26c>] xfs_iflush+0x1bc/0x250 [xfs]
[<ffffffffc064c3ee>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc0653e80>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc06541a7>] xfsaild+0x327/0x6f0 [xfs]
[<ffffffffc0653e80>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff950bb161>] kthread+0xd1/0xe0
[<ffffffff950bb090>] ? insert_kthread_work+0x40/0x40
[<ffffffff95720677>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff950bb090>] ? insert_kthread_work+0x40/0x40
Code: 89 e5 41 57 49 89 f7 41 56 41 55 41 54 53 48 89 fb 48 83 ec 10 0f b7 77 32 4c 8b a7 88 00 00 00 4c 8b 37 4c 89 ff e8 fe ee fe ff <44> 0f b7 08 49 89 c5 66 41 81 f9 49 4e 0f 85 fb 01 00 00 0f b7 
Complete report #15870
Frame # Function Binary Source or offset Line
1
xfs_iflush_int
xfs 0xffffffffc020de52
Build id: 3.10.0-693.5.2.el7.x86_64
0x32
2
xfs_iflush
xfs 0xffffffffc021238c
Build id: 3.10.0-693.5.2.el7.x86_64
0x1bc
3
xfs_inode_item_push
xfs 0xffffffffc0223ec2
Build id: 3.10.0-693.5.2.el7.x86_64
0x102
4
? xfs_trans_ail_cursor_first
xfs 0xffffffffc022ba30
Build id: 3.10.0-693.5.2.el7.x86_64
0x90
5
xfsaild
xfs 0xffffffffc022bd51
Build id: 3.10.0-693.5.2.el7.x86_64
0x321
6
? xfs_trans_ail_cursor_first
xfs 0xffffffffc022ba30
Build id: 3.10.0-693.5.2.el7.x86_64
0x90
7
kthread
vmlinux 0xffffffff810b099f
Build id: 3.10.0-693.5.2.el7.x86_64
0xcf
8
? insert_kthread_work
vmlinux 0xffffffff810b08d0
Build id: 3.10.0-693.5.2.el7.x86_64
0x40
9
ret_from_fork
vmlinux 0xffffffff816b4fd8
Build id: 3.10.0-693.5.2.el7.x86_64
0x58
10
? insert_kthread_work
vmlinux 0xffffffff810b08d0
Build id: 3.10.0-693.5.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 34s! [xfsaild/vda1:306]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep ppdev snd_seq snd_seq_device snd_pcm snd_timer snd soundcore joydev virtio_balloon pcspkr i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c ata_generic pata_acpi qxl drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm virtio_net virtio_blk virtio_console ahci libahci crct10dif_pclmul crct10dif_common drm crc32c_intel ata_piix serio_raw libata floppy virtio_pci virtio_ring virtio i2c_core
CPU: 2 PID: 306 Comm: xfsaild/vda1 Tainted: G             L ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff8800348ceeb0 ti: ffff88003592c000 task.ti: ffff88003592c000
RIP: 0010:[<ffffffffc020de52>]  [<ffffffffc020de52>] xfs_iflush_int+0x32/0x2b0 [xfs]
RSP: 0000:ffff88003592fd30  EFLAGS: 00010286
RAX: ffff880138bf7c00 RBX: ffff88013641bc00 RCX: ffff8801368bec00
RDX: ffff880138bf7000 RSI: 0000000000003c00 RDI: ffff8801368bed80
RBP: ffff88003592fd68 R08: 0000000000000000 R09: 0000000000030000
R10: 0000000000000000 R11: 0000000000000001 R12: ffff8800354ef5a0
R13: ffff88003592fdd0 R14: ffff8800357cd000 R15: ffff8801368bed80
FS:  0000000000000000(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000a6920002 CR3: 00000000040c6000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffe000000020 00000000f381ae60 ffff88013641bc00 ffff8800357cd000
ffff88003592fdd0 ffff88013641bcf4 0000000000000000 ffff88003592fdc0
ffffffffc021238c ffff880000000000 ffff8801368bed80 ffff880138bf7c00
Call Trace:
[<ffffffffc021238c>] xfs_iflush+0x1bc/0x250 [xfs]
[<ffffffffc0223ec2>] xfs_inode_item_push+0x102/0x150 [xfs]
[<ffffffffc022ba30>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc022bd51>] xfsaild+0x321/0x6f0 [xfs]
[<ffffffffc022ba30>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: 89 e5 41 57 49 89 f7 41 56 41 55 41 54 53 48 89 fb 48 83 ec 10 0f b7 77 32 4c 8b a7 88 00 00 00 4c 8b 37 4c 89 ff e8 3e f6 fe ff <44> 0f b7 08 49 89 c5 66 41 81 f9 49 4e 0f 85 cb 01 00 00 0f b7 
Complete report #316
Frame # Function Binary Source or offset Line
1
xfs_inode_to_disk
xfs 0xffffffffc035e29e
Build id: 3.10.0-957.el7.x86_64
0xe
2
? xfs_iflush_int
xfs 0xffffffffc037c15b
Build id: 3.10.0-957.el7.x86_64
0xcb
3
xfs_iflush
xfs 0xffffffffc037ff3c
Build id: 3.10.0-957.el7.x86_64
0x1bc
4
xfs_inode_item_push
xfs 0xffffffffc039216e
Build id: 3.10.0-957.el7.x86_64
0x14e
5
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0399c60
Build id: 3.10.0-957.el7.x86_64
0x90
6
xfsaild
xfs 0xffffffffc0399f87
Build id: 3.10.0-957.el7.x86_64
0x327
7
? xfs_trans_ail_cursor_first
xfs 0xffffffffc0399c60
Build id: 3.10.0-957.el7.x86_64
0x90
8
kthread
vmlinux 0xffffffffa20c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffffa20c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffffa2774c1d
Build id: 3.10.0-957.el7.x86_64
0x7
11
? insert_kthread_work
vmlinux 0xffffffffa20c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [xfsaild/dm-0:4385]
Modules linked in: tcp_lp nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vmw_vmci vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd vmw_balloon snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus joydev snd_seq pcspkr snd_seq_device snd_pcm snd_timer snd soundcore sg parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic
pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix e1000 mptspi libata scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 4385 Comm: xfsaild/dm-0 Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9717f5e65140 ti: ffff9717f4b7c000 task.ti: ffff9717f4b7c000
RIP: 0010:[<ffffffffc035e29e>]  [<ffffffffc035e29e>] xfs_inode_to_disk+0xe/0x210 [xfs]
RSP: 0018:ffff9717f4b7fd18  EFLAGS: 00010246
RAX: 0000000000008000 RBX: 0000000000000000 RCX: 0000000000004e49
RDX: 0000074400004a76 RSI: ffff971810490a00 RDI: ffff9718d9321680
RBP: ffff9717f4b7fd60 R08: 0000000000000000 R09: 0000000000000000
R10: ffff97193861f120 R11: 0000000000000001 R12: ffffffffc03a9f00
R13: ffffffffc036a5f0 R14: ffff9717f4b7fcb8 R15: ffff9717f4a3b200
FS:  0000000000000000(0000) GS:ffff971938600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcc3d988024 CR3: 000000012fd4a000 CR4: 00000000003607f0
Call Trace:
[<ffffffffc037c15b>] ? xfs_iflush_int+0xcb/0x2e0 [xfs]
[<ffffffffc037ff3c>] xfs_iflush+0x1bc/0x250 [xfs]
[<ffffffffc039216e>] xfs_inode_item_push+0x14e/0x1a0 [xfs]
[<ffffffffc0399c60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffc0399f87>] xfsaild+0x327/0x6f0 [xfs]
[<ffffffffc0399c60>] ? xfs_trans_ail_cursor_first+0x90/0x90 [xfs]
[<ffffffffa20c1c31>] kthread+0xd1/0xe0
[<ffffffffa20c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa2774c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa20c1b60>] ? insert_kthread_work+0x40/0x40
Code: 89 83 24 01 00 00 ba 02 00 00 00 e9 3b fe ff ff 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 45 31 c0 b9 49 4e 00 00 55 <66> 89 0e 66 44 89 46 06 45 31 c9 0f b6 87 18 01 00 00 48 89 e5