NMI watchdog: BUG: soft lockup - CPU#35 stuck for 22s! [10.62.105.205-m:142649]
Modules linked in: vfat fat isofs xfs libcrc32c iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache sr_mod cdrom dm_mirror dm_region_hash dm_log skx_edac edac_core coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_service_time iTCO_wdt iTCO_vendor_support uas usb_storage joydev sg pcspkr mei_me mei shpchp lpc_ich i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_cpufreq dm_multipath dm_mod nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel mpt3sas ahci libahci crc_t10dif crct10dif_generic crct10dif_pclmul libata scsi_transport_fc scsi_tgt
i40e(OE) raid_class ptp crct10dif_common i2c_core scsi_transport_sas pps_core
CPU: 35 PID: 142649 Comm: 10.62.105.205-m Tainted: G           OE  ------------   3.10.0-693.el7.x86_64 #1
Hardware name: Sugon I620-G30/60P24-US, BIOS 0JGST219 05/30/2019
task: ffff880480505ee0 ti: ffff88048dd34000 task.ti: ffff88048dd34000
RIP: 0010:[<ffffffff8132ba5c>]  [<ffffffff8132ba5c>] memcmp+0x2c/0x50
RSP: 0018:ffff88048dd37d78  EFLAGS: 00000297
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 000000000000000a
RDX: 000000000000000b RSI: ffff88048dd37e18 RDI: ffff8808563e9198
RBP: ffff88048dd37d78 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 7fffffffffffffff R12: 0000000000000000
R13: 7fffffffffffffff R14: 000000000000000b R15: ffff88048dd37d50
FS:  0000000000000000(0000) GS:ffff88046ebc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc483303010 CR3: 000000045ebd6000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff88048dd37de8 ffffffffc07dfbde ffff88048dd37e14 ffff88048b7e1228
ffff88048b7e12b0 0000000000000010 ffff88012ba7b000 00ffffff8121d05e
ffff88048b7e1190 ffff88048b7e1228 ffff88012ba7b0c8 ffff88012ba7b000
Call Trace:
[<ffffffffc07dfbde>] nfs_inode_find_state_and_recover+0xae/0x190 [nfsv4]
[<ffffffffc07e29bb>] nfs_reap_expired_delegations+0x1bb/0x220 [nfsv4]
[<ffffffffc07dec22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc07deef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc07def0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 48 85 d2 48 89 e5 74 3c 0f b6 07 0f b6 0e 29 c8 75 27 48 83 ea 01 31 c9 eb 1a 0f 1f 44 00 00 44 0f b6 4c 0f 01 44 0f b6 44 0e 01 <48> 83 c1 01 45 29 c1 75 0b 48 39 d1 75 e6 5d c3 0f 1f 40 00 44 

NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [172.16.8.6-mana:349192]
Modules linked in: nfsv3 tcp_lp rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt ipmi_ssif iTCO_vendor_support dell_smbios sparse_keymap dcdbas pcspkr sg ipmi_si wmi ipmi_devintf ipmi_msghandler mei_me shpchp i2c_i801 mei lpc_ich acpi_power_meter nfit libnvdimm acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs
libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ahci libahci megaraid_sas tg3 libata ptp i2c_core pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 18 PID: 349192 Comm: 172.16.8.6-mana Not tainted 3.10.0-693.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge T440/021KCD, BIOS 2.7.7 05/06/2020
task: ffff8817f4a05ee0 ti: ffff8801eb2c4000 task.ti: ffff8801eb2c4000
RIP: 0010:[<ffffffff8132ba56>]  [<ffffffff8132ba56>] memcmp+0x26/0x50
RSP: 0018:ffff8801eb2c7d78  EFLAGS: 00000293
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000003
RDX: 000000000000000b RSI: ffff8801eb2c7e18 RDI: ffff881819844e98
RBP: ffff8801eb2c7d78 R08: 0000000000000061 R09: 000000000000005e
R10: ffff88181cc59c40 R11: ffffea00048da200 R12: ffff88181cc59c40
R13: ffffea00048da200 R14: 000000000000000b R15: ffff8801eb2c7d50
FS:  0000000000000000(0000) GS:ffff88181cc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f91643b2000 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8801eb2c7de8 ffffffffc077bbde ffff8801eb2c7e14 ffff8810be7705e0
ffff8810be770668 ffffffffffffff10 ffff8817f9e57000 00ffffff8121d05e
ffff8810be770548 ffff8810be7705e0 ffff8817f9e570c8 ffff8817f9e57000
Call Trace:
[<ffffffffc077bbde>] nfs_inode_find_state_and_recover+0xae/0x190 [nfsv4]
[<ffffffffc077e9bb>] nfs_reap_expired_delegations+0x1bb/0x220 [nfsv4]
[<ffffffffc077ac22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc077aef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc077af0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 55 48 85 d2 48 89 e5 74 3c 0f b6 07 0f b6 0e 29 c8 75 27 48 83 ea 01 31 c9 eb 1a 0f 1f 44 00 00 44 0f b6 4c 0f 01 <44> 0f b6 44 0e 01 48 83 c1 01 45 29 c1 75 0b 48 39 d1 75 e6 5d 

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [10.1.15.5-manag:103781]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ext4 mbcache jbd2 ppdev joydev sg i2c_piix4 parport_pc pcspkr virtio_balloon parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom virtio_blk virtio_console virtio_net ata_generic virtio_scsi pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci floppy virtio_ring i2c_core serio_raw virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 103781 Comm: 10.1.15.5-manag Not tainted 3.10.0-693.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff8803c6b18000 ti: ffff8800ba370000 task.ti: ffff8800ba370000
RIP: 0010:[<ffffffffc059e41e>]  [<ffffffffc059e41e>] nfs_mark_return_delegation.isra.4+0xe/0x20 [nfsv4]
RSP: 0018:ffff8800ba373dc0  EFLAGS: 00000206
RAX: ffff880035f77000 RBX: 00000000cc7298fc RCX: 000000000000000f
RDX: 000000000000000f RSI: ffff880556920000 RDI: ffff880035f70800
RBP: ffff8800ba373dc0 R08: 0000000000000000 R09: 0000000000000000
R10: ffff88042ced9c40 R11: ffffea0010900f00 R12: 0000000000000010
R13: ffffffffc059e4ce R14: ffffffffffffff10 R15: ffff880035f70800
FS:  0000000000000000(0000) GS:ffff88042cec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f82be2c20a0 CR3: 00000000019f2000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8800ba373de8 ffffffffc059e551 ffff88048716aed0 ffff88003561a8c8
ffff88003561a800 ffff8800ba373e58 ffffffffc059f9ae ffffffffc05b83c0
ffff88042e6672c0 ffff880556920000 00000000fffffff5 002a122a613ba9ae
Call Trace:
[<ffffffffc059e551>] nfs_revoke_delegation+0x71/0x90 [nfsv4]
[<ffffffffc059f9ae>] nfs_reap_expired_delegations+0x1ae/0x220 [nfsv4]
[<ffffffffc059bc22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc059bef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc059bf0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 55 48 89 e5 f0 80 4e 48 04 48 8b 07 f0 80 88 28 01 00 00 20 5d c3 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 f0 80 4e 48 02 <48> 8b 07 f0 80 88 28 01 00 00 20 5d c3 0f 1f 44 00 00 0f 1f 44 

NMI watchdog: BUG: soft lockup - CPU#50 stuck for 22s! [10.210.38.12-ma:194020]
Modules linked in: panfs(POE) 8021q garp mrp stp llc nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache iTCO_wdt iTCO_vendor_support ib_isert iscsi_target_mod ib_srpt target_core_mod rpcrdma ib_iser libiscsi scsi_transport_iscsi ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm skx_edac edac_core intel_powerclamp coretemp ib_ucm intel_rapl iosf_mbi kvm_intel kvm ib_uverbs ib_umad irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper rdma_cm cryptd ib_cm iw_cm joydev pcspkr sg mei_me mei shpchp lpc_ich i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler mlx5_ib ib_core nfit libnvdimm acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul ast crct10dif_common
crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mlx5_core ttm drm igb nvme ahci devlink libahci nvme_core dca ptp libata i2c_algo_bit i2c_core pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 50 PID: 194020 Comm: 10.210.38.12-ma Tainted: P           OE  ------------   3.10.0-693.el7.x86_64 #1
Hardware name: Penguin Computing Relion XE1112/MR91-FS0-ZB, BIOS R03 06/03/2019
task: ffff884616300fd0 ti: ffff89234cb58000 task.ti: ffff89234cb58000
RIP: 0010:[<ffffffffc0b89429>]  [<ffffffffc0b89429>] nfs_mark_return_delegation.isra.4+0x19/0x20 [nfsv4]
RSP: 0018:ffff89234cb5bdc0  EFLAGS: 00000202
RAX: ffff897d3975ac00 RBX: 000000000ada7bfc RCX: 000000000000000f
RDX: 000000000000000f RSI: ffff88bccd217d00 RDI: ffff885363f1d000
RBP: ffff89234cb5bdc0 R08: 0000000000000000 R09: 0000000000000000
R10: ffff897d3c299c40 R11: ffffea05f4cb8100 R12: ffff88bce724f200
R13: 000000000000000b R14: 000000000000000b R15: ffff88b34590b600
FS:  0000000000000000(0000) GS:ffff897d3c280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f50ab594dc9 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff89234cb5bde8 ffffffffc0b89551 ffff88bccbe68e10 ffff897d3975acc8
ffff897d3975ac00 ffff89234cb5be58 ffffffffc0b8a9ae ffffffffc0ba33c0
ffff897cee3a8f00 ffff88bccd217d00 0000000000000000 1f5cd8026693e0ac
Call Trace:
[<ffffffffc0b89551>] nfs_revoke_delegation+0x71/0x90 [nfsv4]
[<ffffffffc0b8a9ae>] nfs_reap_expired_delegations+0x1ae/0x220 [nfsv4]
[<ffffffffc0b86c22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc0b86ef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc0b86f0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 48 8b 07 f0 80 88 28 01 00 00 20 5d c3 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 f0 80 4e 48 02 48 8b 07 f0 80 88 28 01 00 00 20 <5d> c3 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 41 55 4c 8d af