NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:2:5544]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_16108(E) falcon_lsm_pinned_16004(E) nls_utf8 cifs ccm dns_resolver ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sb_edac joydev iosf_mbi hv_utils sg ptp pps_core hv_balloon pcspkr i2c_piix4 ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic crct10dif_common ata_generic pata_acpi ata_piix hv_storvsc
hv_netvsc scsi_transport_fc hyperv_keyboard hid_hyperv scsi_tgt libata hyperv_fb hv_vmbus floppy serio_raw dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal]
CPU: 2 PID: 5544 Comm: kworker/2:2 Kdump: loaded Tainted: P        W   EL ------------   3.10.0-1160.90.1.el7.x86_64 #1
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  05/18/2018
Workqueue: xfs-data/dm-0 xfs_end_io [xfs]
task: ffff952ec1599080 ti: ffff952ec1948000 task.ti: ffff952ec1948000
RIP: 0010:[<ffffffffc0355dc7>]  [<ffffffffc0355dc7>] xfs_destroy_ioend+0x147/0x260 [xfs]
RSP: 0018:ffff952ec194bd88  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffd73d50a9b940 RCX: 0000000000000034
RDX: ffff952ffab05e38 RSI: ffff952ffab05e38 RDI: 0000000000000202
RBP: ffff952ec194bdf8 R08: 7010000000000000 R09: 2ffab05e38080000
R10: a30a94966ba9b940 R11: 0000000000000000 R12: 000000008c655a8b
R13: ffffd73d50a9b940 R14: ffffffffbd99cf59 R15: ffff952ec194bd18
FS:  0000000000000000(0000) GS:ffff952fc1680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000689000 CR3: 0000000001430000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc0355f35>] xfs_end_io+0x55/0x90 [xfs]
[<ffffffffbd6c32ef>] process_one_work+0x17f/0x440
[<ffffffffbd6c4436>] worker_thread+0x126/0x3c0
[<ffffffffbd6c4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffffbd6cb621>] kthread+0xd1/0xe0
[<ffffffffbd6cb550>] ? insert_kthread_work+0x40/0x40
[<ffffffffbddc51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffbd6cb550>] ? insert_kthread_work+0x40/0x40
Code: 45 0c 44 39 e0 76 bd 48 8b 03 f6 c4 01 45 0f 45 f7 44 03 63 20 48 8b 5b 08 48 39 da 75 e0 45 89 f4 f0 80 22 ef 48 8b 7d c8 57 9d <0f> 1f 44 00 00 45 84 e4 75 09 49 8b 7d 00 e8 36 2d 47 fd 83 45 

NMI watchdog: BUG: soft lockup - CPU#28 stuck for 22s! [kworker/28:2:79906]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nslink(OE) sb_edac iTCO_wdt intel_powerclamp iTCO_vendor_support coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr osst st ch lpc_ich ioatdma hpilo hpwdt sg wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs sd_mod crc_t10dif crct10dif_generic qla2xxx mgag200 drm_kms_helper syscopyarea bnx2x sysfillrect sysimgblt fb_sys_fops
ttm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm nvme_fc nvme_fabrics igb nvme_core scsi_transport_fc mdio hpsa scsi_tgt libcrc32c dca ptp drm_panel_orientation_quirks pps_core scsi_transport_sas i2c_algo_bit dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 28 PID: 79906 Comm: kworker/28:2 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.108.1.el7.x86_64 #1
Hardware name: HP ProLiant DL580 Gen8, BIOS P79 11/26/2014
Workqueue: xfs-data/dm-2 xfs_end_io [xfs]
task: ffff8da210980000 ti: ffff8da081690000 task.ti: ffff8da081690000
RIP: 0010:[<ffffffffc08a5df0>]  [<ffffffffc08a5df0>] xfs_destroy_ioend+0x170/0x260 [xfs]
RSP: 0018:ffff8da081693d88  EFLAGS: 00000286
RAX: ffff8db1084baa00 RBX: ffffd9806f4eb3c0 RCX: 0000000000000030
RDX: 000000000000000d RSI: ffffd9806f4eb3c0 RDI: ffff8db43ffc51e8
RBP: ffff8da081693df8 R08: 9e00000000000000 R09: 6601bd3acf000000
R10: 99fe1c45a04eb3c0 R11: 0000000000000000 R12: 0000000076cf37e1
R13: ffffd9806f4eb3c0 R14: ffffffff88d9d0a9 R15: ffff8da081693d18
FS:  0000000000000000(0000) GS:ffff8db3fec00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5131b07be8 CR3: 00000001ff610000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc08a5f35>] xfs_end_io+0x55/0x90 [xfs]
[<ffffffff88ac32ef>] process_one_work+0x17f/0x440
[<ffffffff88ac4436>] worker_thread+0x126/0x3c0
[<ffffffff88ac4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff88acb621>] kthread+0xd1/0xe0
[<ffffffff88acb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff891c51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff88acb550>] ? insert_kthread_work+0x40/0x40
Code: 57 9d 0f 1f 44 00 00 45 84 e4 75 09 49 8b 7d 00 e8 a6 2d 32 c8 83 45 c0 01 48 8b 45 b8 8b 4d c0 4c 63 e9 49 c1 e5 04 4c 03 68 70 <0f> b7 40 28 39 c8 7e 39 49 8b 45 00 48 8b 10 80 e6 08 0f 85 18 

NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [kworker/30:2:51743]
Modules linked in: cdc_ether joydev usbnet mii binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bonding sunrpc vfat fat iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure pcspkr sg i2c_i801 ipmi_ssif mei_me mei wmi ipmi_si ipmi_devintf ipmi_msghandler pinctrl_lewisburg pinctrl_intel acpi_power_meter acpi_pad ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm crc32c_intel i40e igb ahci smartpqi libahci drm_panel_orientation_quirks scsi_transport_sas dca ptp i2c_algo_bit pps_core libata dm_mirror dm_region_hash dm_log dm_mod
CPU: 30 PID: 51743 Comm: kworker/30:2 Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: Lenovo ThinkServer SR660 V2/SR660 V2, BIOS XWE103I-1.70 03/22/2022
Workqueue: xfs-data/dm-5 xfs_end_io [xfs]
task: ffff8e417fc31040 ti: ffff8e015c494000 task.ti: ffff8e015c494000
RIP: 0010:[<ffffffffc0738660>]  [<ffffffffc0738660>] xfs_destroy_ioend+0xe0/0x170 [xfs]
RSP: 0018:ffff8e015c497da8  EFLAGS: 00000206
RAX: ffff8e0202cea000 RBX: ffff8e015c497d50 RCX: dead000000000200
RDX: ffff8e318b6fc208 RSI: 0000000000000293 RDI: 0000000000000293
RBP: ffff8e015c497df8 R08: ffff8e242602bd60 R09: ffff8e21d3332080
R10: 0000000000000000 R11: 0000000000000400 R12: fffffc7c9d9f4d40
R13: 0000000000000202 R14: ffff8e318b6fc208 R15: fffffc7c9d9f4d40
FS:  0000000000000000(0000) GS:ffff8e40ff180000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000001433074ca8 CR3: 0000000f7b010000 CR4: 0000000000760fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffffc0738745>] xfs_end_io+0x55/0x90 [xfs]
[<ffffffff826b9d4f>] process_one_work+0x17f/0x440
[<ffffffff826bade6>] worker_thread+0x126/0x3c0
[<ffffffff826bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff826c1c31>] kthread+0xd1/0xe0
[<ffffffff826c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff82d74c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff826c1b60>] ? insert_kthread_work+0x40/0x40
Code: d0 4c 8b 5f 38 85 c0 40 0f 94 c6 45 01 e5 e8 68 e2 24 c2 48 8b 55 d0 48 39 d3 75 c7 0f 1f 80 00 00 00 00 83 45 c8 01 48 8b 45 c0 <8b> 4d c8 4c 63 f9 49 c1 e7 04 4c 03 78 70 0f b7 40 28 39 c8 7e 

NMI watchdog: BUG: soft lockup - CPU#59 stuck for 22s! [kworker/59:30:149892]
Modules linked in: tcp_diag udp_diag inet_diag ppdev sg parport_pc joydev virtio_balloon parport i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod cdrom virtio_net virtio_scsi virtio_console virtio_blk ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci i2c_core virtio_ring serio_raw virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 59 PID: 149892 Comm: kworker/59:30 Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
Workqueue: xfs-data/dm-2 xfs_end_io [xfs]
task: ffff984c223f0000 ti: ffff984cec6dc000 task.ti: ffff984cec6dc000
RIP: 0010:[<ffffffffc06267e5>]  [<ffffffffc06267e5>] xfs_destroy_ioend+0x105/0x160 [xfs]
RSP: 0018:ffff984cec6dfdb0  EFLAGS: 00010206
RAX: ffffe2c12c2bfd00 RBX: ffffe2c17a1b46c0 RCX: 0000000000000000
RDX: 0000000000000027 RSI: ffffe2c17a1b46c0 RDI: ffff984d7fec9e68
RBP: ffff984cec6dfe00 R08: 3600000000000000 R09: 8b05e86d1b000000
R10: 74f9fa545f1b46c0 R11: 0000000000000000 R12: ffffffff00000002
R13: ffffffff96fa0edd R14: ffff984cec6dfd58 R15: ffffe2c17a1b46c0
FS:  0000000000000000(0000) GS:ffff984cfdcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000001c2103b CR3: 00000001c140e000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc0626897>] xfs_end_io+0x57/0x80 [xfs]
[<ffffffff96eb2dff>] process_one_work+0x17f/0x440
[<ffffffff96eb3ac6>] worker_thread+0x126/0x3c0
[<ffffffff96eb39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff96ebae31>] kthread+0xd1/0xe0
[<ffffffff96ebad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff9751f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff96ebad60>] ? insert_kthread_work+0x40/0x40
Code: 8b 55 cc 4c 63 fa 49 c1 e7 04 4c 03 78 70 0f b7 40 28 39 d0 7e 28 41 8b 4f 0c 45 8b 6f 08 41 01 cd 41 8d 45 ff 89 45 d4 49 8b 07 <48> 8b 10 80 e6 08 0f 85 77 ff ff ff 0f 0b 0f 1f 44 00 00 48 8b 

NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [kworker/11:5:25076]
Modules linked in: tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev pcspkr cryptd vmw_balloon joydev sg parport_pc vmw_vmci i2c_piix4 parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix nfit crct10dif_pclmul crct10dif_common libata mptspi crc32c_intel libnvdimm scsi_transport_spi
serio_raw mptscsih mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 11 PID: 25076 Comm: kworker/11:5 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018
Workqueue: xfs-data/sda2 xfs_end_io [xfs]
task: ffff8fa605804200 ti: ffff8fa399c34000 task.ti: ffff8fa399c34000
RIP: 0010:[<ffffffffc06398d1>]  [<ffffffffc06398d1>] xfs_destroy_ioend+0x81/0x260 [xfs]
RSP: 0018:ffff8fa399c37d88  EFLAGS: 00010202
RAX: ffff8fa776c67b00 RBX: ffffffff8aa27022 RCX: 000000018020001d
RDX: 000000018020001e RSI: ffff8fa5efc3e700 RDI: 0000000040000000
RBP: ffff8fa399c37df8 R08: ffff8fa69d3a1200 R09: 000000018020001d
R10: 000000009d3a0001 R11: ffff8fa69d3a1200 R12: 0000000000000001
R13: 0000000000000000 R14: ffff8fa774d75800 R15: ffff8fa4bfc03700
FS:  0000000000000000(0000) GS:ffff8fa777100000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4599fa51a8 CR3: 0000000052a10000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc0639b05>] xfs_end_io+0x55/0x90 [xfs]
[<ffffffff8a8bde8f>] process_one_work+0x17f/0x440
[<ffffffff8a8befa6>] worker_thread+0x126/0x3c0
[<ffffffff8a8bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8a8c5e61>] kthread+0xd1/0xe0
[<ffffffff8a8c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffff8af95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8a8c5d90>] ? insert_kthread_work+0x40/0x40
Code: 8b 75 a8 48 8b 45 b8 48 39 f0 0f 84 74 01 00 00 48 8b 40 48 48 89 45 b0 48 8b 45 b8 66 83 78 28 00 4c 8b 60 70 0f 84 30 01 00 00 <49> 8b 04 24 48 8b 10 80 e6 08 0f 84 f7 00 00 00 c7 45 c0 00 00 

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:20:10570]
Modules linked in: tcp_lp fuse xt_multiport xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun bridge stp llc devlink nf_nat_ftp nf_conntrack_ftp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp ppdev iosf_mbi crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c
sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi scsi_transport_spi libata mptscsih crct10dif_pclmul crct10dif_common serio_raw crc32c_intel mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 10570 Comm: kworker/3:20 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/30/2013
Workqueue: xfs-data/sdb1 xfs_end_io [xfs]
task: ffff8bb574aab150 ti: ffff8bb704d10000 task.ti: ffff8bb704d10000
RIP: 0010:[<ffffffffc0452737>]  [<ffffffffc0452737>] xfs_destroy_ioend+0x147/0x260 [xfs]
RSP: 0018:ffff8bb704d13d88  EFLAGS: 00000202
RAX: 0000000000000000 RBX: fffffa4010a1ce80 RCX: 0000000000000034
RDX: ffff8bb75a508000 RSI: ffff8bb75a508000 RDI: 0000000000000202
RBP: ffff8bb704d13df8 R08: 0010000000000000 R09: b75a508000080000
R10: 16ffb7b8d6a1ce80 R11: 0000000000000000 R12: 000000002544918d
R13: fffffa4010a1ce80 R14: ffffffff9917fed9 R15: ffff8bb704d13d18
FS:  0000000000000000(0000) GS:ffff8bb6f7cc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002464918 CR3: 000000042eee8000 CR4: 00000000000407e0
Call Trace:
[<ffffffffc04528a5>] xfs_end_io+0x55/0x90 [xfs]
[<ffffffff98ebd0ff>] process_one_work+0x17f/0x440
[<ffffffff98ebe216>] worker_thread+0x126/0x3c0
[<ffffffff98ebe0f0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff98ec50d1>] kthread+0xd1/0xe0
[<ffffffff98ec5000>] ? insert_kthread_work+0x40/0x40
[<ffffffff9958cd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff98ec5000>] ? insert_kthread_work+0x40/0x40
Code: 45 0c 44 39 e0 76 bd 48 8b 03 f6 c4 01 45 0f 45 f7 44 03 63 20 48 8b 5b 08 48 39 da 75 e0 45 89 f4 f0 80 22 ef 48 8b 7d c8 57 9d <0f> 1f 44 00 00 45 84 e4 75 09 49 8b 7d 00 e8 e6 93 b6 d8 83 45