NMI watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [10.10.0.2-manag:30885]
Modules linked in: 8021q garp mrp stp llc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper iTCO_wdt ablk_helper iTCO_vendor_support mei_me mei sg mxm_wmi lpc_ich ipmi_ssif wmi ipmi_si ipmi_devintf cryptd ipmi_msghandler pcspkr dcdbas acpi_power_meter shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe drm ahci libahci crct10dif_pclmul crct10dif_common crc32c_intel libata mdio ptp pps_core dca i2c_core
CPU: 20 PID: 30885 Comm: 10.10.0.2-manag Not tainted 3.10.0-693.17.1.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge C6320/082F9M, BIOS 2.6.0 10/27/2017
task: ffff881046b96eb0 ti: ffff88055aa7c000 task.ti: ffff88055aa7c000
RIP: 0010:[<ffffffffc047cf63>]  [<ffffffffc047cf63>] nfs_sb_deactive+0x13/0x30 [nfs]
RSP: 0018:ffff88055aa7fd00  EFLAGS: 00000206
RAX: ffff88105aedd800 RBX: 0000000000000004 RCX: ffff88105aedd800
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff88105aeda800
RBP: ffff88055aa7fd40 R08: ffff88105aeda88c R09: ffffffffc06284ba
R10: ffff88085e89b9c0 R11: ffffea000142ca00 R12: ffff88105aedd800
R13: 0000000000000000 R14: ffff88105aeda88c R15: ffffffffc06284ba
FS:  0000000000000000(0000) GS:ffff88085e880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd69af82180 CR3: 00000000019fa000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc0645765>] ? nfs_delegation_reap_unclaimed+0x115/0x140 [nfsv4]
[<ffffffffc063fe68>] nfs4_state_clear_reclaim_reboot+0x138/0x160 [nfsv4]
[<ffffffffc0640460>] nfs4_recovery_handle_error+0x40/0x180 [nfsv4]
[<ffffffffc0641483>] nfs4_do_reclaim+0x503/0x6c0 [nfsv4]
[<ffffffffc0641bec>] nfs4_state_manager+0x5ac/0x8c0 [nfsv4]
[<ffffffffc0641f00>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc0641f1f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b270f>] kthread+0xcf/0xe0
[<ffffffff810c0f53>] ? finish_task_switch+0x93/0x160
[<ffffffff810b2640>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b8798>] ret_from_fork+0x58/0x90
[<ffffffff810b2640>] ? insert_kthread_work+0x40/0x40
Code: 0f 83 c0 01 39 c8 75 f0 31 c0 5d c3 0f 1f 40 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 48 8b 87 50 03 00 00 f0 ff 88 88 05 00 00 <0f> 94 c2 84 d2 75 06 c3 0f 1f 44 00 00 55 48 89 e5 e8 07 96 d8 

NMI watchdog: BUG: soft lockup - CPU#8 stuck for 23s! [10.1.1.1-manage:439156]
Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache 8021q garp mrp stp llc ip6_tables iptable_filter rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass iTCO_wdt iTCO_vendor_support dell_smbios sparse_keymap dcdbas crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ipmi_si ipmi_devintf ipmi_msghandler pcspkr lpc_ich i2c_i801 nfit mei_me mei libnvdimm shpchp acpi_power_meter acpi_pad wmi nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic
mgag200 i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea qede sysfillrect sysimgblt fb_sys_fops ttm qed drm ahci libahci libata megaraid_sas i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 8 PID: 439156 Comm: 10.1.1.1-manage Tainted: G             L ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge MX740c/00Y7XY, BIOS 2.3.8 07/30/2019
task: ffff8820820beeb0 ti: ffff881f2fbe4000 task.ti: ffff881f2fbe4000
RIP: 0010:[<ffffffff816abc00>]  [<ffffffff816abc00>] _raw_spin_lock+0x10/0x30
RSP: 0018:ffff881f2fbe7cc0  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff881f2fbe7c60 RCX: ffff8820d19b2800
RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff8826cec043d0
RBP: ffff881f2fbe7cd8 R08: ffff8820d19b788c R09: ffffffffc08754ba
R10: ffff881ffcd19c80 R11: ffffea00ffe4da00 R12: ffff88060830b900
R13: ffffffffc0269a41 R14: ffff881f2fbe7c40 R15: 00000000ffffd8cf
FS:  0000000000000000(0000) GS:ffff881ffcd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002af4173ef000 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffff8121d16e ffff881fe57609d0 ffff881fe5760980 ffff881f2fbe7cf8
ffffffffc089140a ffff881fe5760980 ffff883ff7d7acc8 ffff881f2fbe7d40
ffffffffc08926fc ffff881fe5760980 00000000d1b7634e 0000000000000000
Call Trace:
[<ffffffff8121d16e>] ? igrab+0x1e/0x60
[<ffffffffc089140a>] nfs_delegation_grab_inode+0x2a/0x50 [nfsv4]
[<ffffffffc08926fc>] nfs_delegation_reap_unclaimed+0xac/0x140 [nfsv4]
[<ffffffffc088ce68>] nfs4_state_clear_reclaim_reboot+0x138/0x160 [nfsv4]
[<ffffffffc088d460>] nfs4_recovery_handle_error+0x40/0x180 [nfsv4]
[<ffffffffc088e483>] nfs4_do_reclaim+0x503/0x6c0 [nfsv4]
[<ffffffffc088ebec>] nfs4_state_manager+0x5ac/0x8c0 [nfsv4]
[<ffffffffc088ef00>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc088ef1f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 a4 2a ff ff 5d c3 0f 1f 40 

NMI watchdog: BUG: soft lockup - CPU#66 stuck for 22s! [10.200.0.70-man:17953]
Modules linked in: joydev uinput xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nfsv3 nfs_acl cmac arc4 md4 nls_utf8 cifs rpcsec_gss_krb5 ccm nfsv4 dns_resolver nfs lockd grace ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter cachefiles fscache dm_mirror dm_region_hash dm_log dm_mod vfat iTCO_wdt iTCO_vendor_support fat sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm
irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr ipmi_si ipmi_devintf ipmi_msghandler i2c_i801 pcc_cpufreq hpwdt hpilo lpc_ich wmi acpi_power_meter ioatdma dca auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common ttm crc32c_intel ahci libahci drm libata tg3 ptp pps_core drm_panel_orientation_quirks
CPU: 66 PID: 17953 Comm: 10.200.0.70-man Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-957.el7.centos.plus.x86_64 #1
Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 04/29/2021
task: ffff9634a86930c0 ti: ffff965353624000 task.ti: ffff965353624000
RIP: 0010:[<ffffffffc07741c3>]  [<ffffffffc07741c3>] nfs_sb_deactive+0x13/0x30 [nfs]
RSP: 0018:ffff965353627d00  EFLAGS: 00000202
RAX: ffff9634a93a3000 RBX: ffff96537f29f160 RCX: ffff9634a93a3000
RDX: 0000000000000000 RSI: 0000000000000009 RDI: ffff96437dec2800
RBP: ffff965353627d40 R08: ffff96437dec288c R09: ffffffffc07af58a
R10: ffff96537f29f160 R11: ffffe10e0092ce00 R12: ffff96537f29f160
R13: ffff965353627cf8 R14: ffff96437dec288c R15: ffff965353627cf8
FS:  0000000000000000(0000) GS:ffff96537f280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000111 CR3: 0000000353a10000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc07cdba5>] ? nfs_delegation_reap_unclaimed+0x115/0x140 [nfsv4]
[<ffffffffc07c7e48>] nfs4_state_clear_reclaim_reboot+0x138/0x160 [nfsv4]
[<ffffffffc07c87e0>] nfs4_recovery_handle_error+0x40/0x170 [nfsv4]
[<ffffffffc07c98e4>] nfs4_do_reclaim+0x5f4/0x6c0 [nfsv4]
[<ffffffffc07c9f64>] nfs4_state_manager+0x5b4/0x8d0 [nfsv4]
[<ffffffffc07ca280>] ? nfs4_state_manager+0x8d0/0x8d0 [nfsv4]
[<ffffffffc07ca29f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffffa72c2c31>] kthread+0xd1/0xe0
[<ffffffffa72c2b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa7986c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa72c2b60>] ? insert_kthread_work+0x40/0x40
Code: 0f 83 c0 01 39 c8 75 f0 31 c0 5d c3 0f 1f 40 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 48 8b 87 50 03 00 00 f0 ff 88 88 05 00 00 <0f> 94 c2 84 d2 75 06 c3 0f 1f 44 00 00 55 48 89 e5 e8 67 18 cd 

NMI watchdog: BUG: soft lockup - CPU#22 stuck for 23s! [192.168.10.43-m:50234]
Modules linked in: joydev uinput xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink cmac nfsv3 nfs_acl arc4 md4 nls_utf8 cifs ccm rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter cachefiles fscache dm_mirror dm_region_hash dm_log dm_mod vfat fat sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul iTCO_wdt
iTCO_vendor_support ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg lpc_ich i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler hpilo hpwdt ioatdma pcc_cpufreq dca wmi acpi_power_meter auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt ahci fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel libahci drm libata tg3 ptp pps_core drm_panel_orientation_quirks
CPU: 22 PID: 50234 Comm: 192.168.10.43-m Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.centos.plus.x86_64 #1
Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 04/29/2021
task: ffff884216a4b0c0 ti: ffff88410abac000 task.ti: ffff88410abac000
RIP: 0010:[<ffffffffc07e81c3>]  [<ffffffffc07e81c3>] nfs_sb_deactive+0x13/0x30 [nfs]
RSP: 0018:ffff88410abafd00  EFLAGS: 00000206
RAX: ffff883b33e1a000 RBX: ffff884a1ec9f160 RCX: ffff883b33e1a000
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff8841e83fa800
RBP: ffff88410abafd40 R08: ffff8841e83fa88c R09: ffffffffc099458a
R10: ffff884a1ec9f160 R11: ffffe1e4deebf000 R12: ffff884a1ec9f160
R13: ffff88410abafcf8 R14: ffff8841e83fa88c R15: ffff88410abafcf8
FS:  0000000000000000(0000) GS:ffff884a1ec80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f57644ffc20 CR3: 0000000e28410000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc09b2ba5>] ? nfs_delegation_reap_unclaimed+0x115/0x140 [nfsv4]
[<ffffffffc09ace48>] nfs4_state_clear_reclaim_reboot+0x138/0x160 [nfsv4]
[<ffffffffc09ad7e0>] nfs4_recovery_handle_error+0x40/0x170 [nfsv4]
[<ffffffffc09ae8e4>] nfs4_do_reclaim+0x5f4/0x6c0 [nfsv4]
[<ffffffffc09aef64>] nfs4_state_manager+0x5b4/0x8d0 [nfsv4]
[<ffffffffc09af280>] ? nfs4_state_manager+0x8d0/0x8d0 [nfsv4]
[<ffffffffc09af29f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff94cc2c31>] kthread+0xd1/0xe0
[<ffffffff94cc2b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff95386c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff94cc2b60>] ? insert_kthread_work+0x40/0x40
Code: 0f 83 c0 01 39 c8 75 f0 31 c0 5d c3 0f 1f 40 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 48 8b 87 50 03 00 00 f0 ff 88 88 05 00 00 <0f> 94 c2 84 d2 75 06 c3 0f 1f 44 00 00 55 48 89 e5 e8 67 d8 65 

NMI watchdog: BUG: soft lockup - CPU#53 stuck for 22s! [192.168.10.43-m:38592]
Modules linked in: joydev uinput xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nfsv3 nfs_acl cmac arc4 md4 rpcsec_gss_krb5 nls_utf8 nfsv4 cifs ccm nfs dns_resolver lockd grace ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter cachefiles fscache dm_mirror dm_region_hash dm_log dm_mod sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel vfat fat kvm irqbypass crc32_pclmul iTCO_wdt
iTCO_vendor_support ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg i2c_i801 lpc_ich hpwdt hpilo ipmi_si ipmi_devintf ipmi_msghandler ioatdma pcc_cpufreq dca wmi acpi_power_meter auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common ahci crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm libahci libata drm tg3 ptp pps_core drm_panel_orientation_quirks
CPU: 53 PID: 38592 Comm: 192.168.10.43-m Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-957.el7.centos.plus.x86_64 #1
Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 04/29/2021
task: ffff9e6cb3d02080 ti: ffff9e6c42da4000 task.ti: ffff9e6c42da4000
RIP: 0010:[<ffffffff8e17c3f0>]  [<ffffffff8e17c3f0>] _raw_spin_lock+0x10/0x30
RSP: 0000:ffff9e6c42da7ce0  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff9e6cb3d9508c RCX: ffff9e6cb3d97000
RDX: 0000000000000001 RSI: 0000000000000008 RDI: ffff9e73907fd150
RBP: ffff9e6c42da7cf8 R08: ffff9e6cb3d9508c R09: ffffffffc0b1c58a
R10: ffff9e739fc5f160 R11: ffffe3f5413c9a00 R12: ffff9e739fc5f160
R13: ffffe3f5413c9a00 R14: ffff9e6c42da7c60 R15: ffff9e6c42da7cd8
FS:  0000000000000000(0000) GS:ffff9e739fc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000310cb8 CR3: 00000002cfc10000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc0b397ab>] ? nfs_delegation_grab_inode+0x1b/0x50 [nfsv4]
[<ffffffffc0b3ab3c>] nfs_delegation_reap_unclaimed+0xac/0x140 [nfsv4]
[<ffffffffc0b34e48>] nfs4_state_clear_reclaim_reboot+0x138/0x160 [nfsv4]
[<ffffffffc0b357e0>] nfs4_recovery_handle_error+0x40/0x170 [nfsv4]
[<ffffffffc0b368e4>] nfs4_do_reclaim+0x5f4/0x6c0 [nfsv4]
[<ffffffffc0b36f64>] nfs4_state_manager+0x5b4/0x8d0 [nfsv4]
[<ffffffffc0b37280>] ? nfs4_state_manager+0x8d0/0x8d0 [nfsv4]
[<ffffffffc0b3729f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff8dac2c31>] kthread+0xd1/0xe0
[<ffffffff8dac2b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff8e186c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8dac2b60>] ? insert_kthread_work+0x40/0x40
Code: 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 40 1b ff ff 5d c3 0f 1f 40 

NMI watchdog: BUG: soft lockup - CPU#43 stuck for 22s! [20.20.20.24-man:87019]
Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 8021q garp mrp stp llc dns_resolver nfs fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm i40iw ip6_tables ib_core iptable_filter skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support dell_smbios sg sparse_keymap dcdbas ipmi_si pcspkr ipmi_devintf i2c_i801 ipmi_msghandler lpc_ich shpchp mei_me wmi mei nfit acpi_power_meter libnvdimm acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif
crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect crct10dif_pclmul sysimgblt crct10dif_common fb_sys_fops crc32c_intel ttm drm i40e igb ahci libahci libata megaraid_sas dca i2c_algo_bit ptp i2c_core pps_core
CPU: 43 PID: 87019 Comm: 20.20.20.24-man Tainted: G             L ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R740/01YM03, BIOS 2.2.11 06/13/2019
task: ffff881ac1a04f10 ti: ffff881092638000 task.ti: ffff881092638000
RIP: 0010:[<ffffffffc097e6d8>]  [<ffffffffc097e6d8>] nfs_delegation_reap_unclaimed+0x88/0x140 [nfsv4]
RSP: 0000:ffff88109263bd08  EFLAGS: 00000283
RAX: 0000000000000005 RBX: ffff88203c359c80 RCX: ffff882022eec000
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff882022eef000
RBP: ffff88109263bd40 R08: ffff882022eef08c R09: ffffffffc09614ba
R10: ffff88203c359c80 R11: ffffea0080d4fc00 R12: ffff882022eec000
R13: ffff88109263bcf8 R14: ffff882022eef08c R15: ffffffffc09614ba
FS:  0000000000000000(0000) GS:ffff88203c340000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6b93c7f903 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8810834f1980 0000000099ba55bd 0000000000000000 ffff881141068640
ffff881033e2f000 ffff881141068650 ffff881141068650 ffff88109263bda8
ffffffffc0978e68 ffff881033e2f000 ffff881033e2f0c8 ffff881033e2f130
Call Trace:
[<ffffffffc0978e68>] nfs4_state_clear_reclaim_reboot+0x138/0x160 [nfsv4]
[<ffffffffc0979460>] nfs4_recovery_handle_error+0x40/0x180 [nfsv4]
[<ffffffffc097a483>] nfs4_do_reclaim+0x503/0x6c0 [nfsv4]
[<ffffffffc097abec>] nfs4_state_manager+0x5ac/0x8c0 [nfsv4]
[<ffffffffc097af00>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc097af1f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: c8 48 8b 5d c8 4c 39 f3 75 1d e9 94 00 00 00 0f 1f 40 00 48 8b 03 48 89 45 c8 48 8b 5d c8 4c 39 f3 0f 84 7c 00 00 00 48 8b 43 48 <a8> 10 75 e4 48 8b 43 48 a8 01 74 dc 49 8b bf 70 03 00 00 e8 60 

NMI watchdog: BUG: soft lockup - CPU#21 stuck for 22s! [10.1.1.1-manage:373631]
Modules linked in: nfsv3 rpcsec_gss_krb5 8021q garp nfsv4 mrp stp llc dns_resolver nfs fscache ip6_tables iptable_filter rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support sg dell_smbios sparse_keymap dcdbas pcspkr lpc_ich shpchp i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm mei_me mei wmi tpm_crb acpi_pad acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif
crct10dif_generic sr_mod cdrom mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ahci libahci tg3 bnxt_en libata megaraid_sas ptp i2c_core pps_core
CPU: 21 PID: 373631 Comm: 10.1.1.1-manage Not tainted 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.8.2 08/31/2020
task: ffff880064130fd0 ti: ffff880269eb8000 task.ti: ffff880269eb8000
RIP: 0010:[<ffffffffc0813f63>]  [<ffffffffc0813f63>] nfs_sb_deactive+0x13/0x30 [nfs]
RSP: 0018:ffff880269ebbd00  EFLAGS: 00000202
RAX: ffff8802fb41d000 RBX: ffff880474099c80 RCX: ffff8802fb41d000
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffff8802fb41d800
RBP: ffff880269ebbd40 R08: ffff8802fb41d88c R09: ffffffffc08504ba
R10: ffff880474099c80 R11: ffffea0011d19e00 R12: ffff880474099c80
R13: ffff880269ebbcf8 R14: ffff8802fb41d88c R15: ffff880269ebbcf8
FS:  0000000000000000(0000) GS:ffff880474080000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002ba56851bc2d CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffffc086d765 ffff880236265f80 000000006e5601c6 0000000000000000
ffff8802761ac240 ffff880474678800 ffff8802761ac250 ffff8802761ac250
ffff880269ebbda8 ffffffffc0867e68 ffff880474678800 ffff8804746788c8
Call Trace:
[<ffffffffc086d765>] ? nfs_delegation_reap_unclaimed+0x115/0x140 [nfsv4]
[<ffffffffc0867e68>] nfs4_state_clear_reclaim_reboot+0x138/0x160 [nfsv4]
[<ffffffffc0868460>] nfs4_recovery_handle_error+0x40/0x180 [nfsv4]
[<ffffffffc0869483>] nfs4_do_reclaim+0x503/0x6c0 [nfsv4]
[<ffffffffc0869bec>] nfs4_state_manager+0x5ac/0x8c0 [nfsv4]
[<ffffffffc0869f00>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc0869f1f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: 0f 83 c0 01 39 c8 75 f0 31 c0 5d c3 0f 1f 40 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 48 8b 87 50 03 00 00 f0 ff 88 88 05 00 00 <0f> 94 c2 84 d2 75 06 c3 0f 1f 44 00 00 55 48 89 e5 e8 c7 05 9f 

NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [20.0.0.11-manag:9395]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache 8021q garp mrp stp llc ip6_tables iptable_filter xfs libcrc32c sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul iTCO_wdt sg iTCO_vendor_support glue_helper ablk_helper cryptd dcdbas mxm_wmi ipmi_si mei_me pcspkr mei ipmi_devintf wmi lpc_ich acpi_power_meter ipmi_msghandler shpchp tpm_crb nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci megaraid_sas tg3 libata i2c_core ptp pps_core
CPU: 4 PID: 9395 Comm: 20.0.0.11-manag Not tainted 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R530/0CN7X8, BIOS 2.11.0 12/23/2019
task: ffff880a67bbbf40 ti: ffff880890c44000 task.ti: ffff880890c44000
RIP: 0010:[<ffffffff816abc00>]  [<ffffffff816abc00>] _raw_spin_lock+0x10/0x30
RSP: 0018:ffff880890c47ce0  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff880666699c80 RCX: ffff8802f9f5c000
RDX: 0000000000000001 RSI: 0000000000000004 RDI: ffff88076140f9d0
RBP: ffff880890c47cf8 R08: ffff8802f9f5b88c R09: ffffffffc07514ba
R10: ffff880666699c80 R11: ffffea0000f28000 R12: ffff880666699c80
R13: ffff880890c47cd8 R14: ffff880890c47c60 R15: ffff880890c47cd8
FS:  0000000000000000(0000) GS:ffff880666680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f84deefae40 CR3: 00000000019f2000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffffc076d3fb ffff88076140f980 ffff88066400bcc8 ffff880890c47d40
ffffffffc076e6fc ffff88076140f980 0000000081d2e770 0000000000000000
ffff880816dd9840 ffff88066400bc00 ffff880816dd9850 ffff880816dd9850
Call Trace:
[<ffffffffc076d3fb>] ? nfs_delegation_grab_inode+0x1b/0x50 [nfsv4]
[<ffffffffc076e6fc>] nfs_delegation_reap_unclaimed+0xac/0x140 [nfsv4]
[<ffffffffc0768e68>] nfs4_state_clear_reclaim_reboot+0x138/0x160 [nfsv4]
[<ffffffffc0769460>] nfs4_recovery_handle_error+0x40/0x180 [nfsv4]
[<ffffffffc076a483>] nfs4_do_reclaim+0x503/0x6c0 [nfsv4]
[<ffffffffc076abec>] nfs4_state_manager+0x5ac/0x8c0 [nfsv4]
[<ffffffffc076af00>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc076af1f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 <85> c0 75 01 c3 55 89 c6 48 89 e5 e8 a4 2a ff ff 5d c3 0f 1f 40 

NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [10.30.62.64-man:16801]
Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache nvidia_uvm(OE) xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm ib_core mlx5_ib(OE) mlx5_core(OE) mlx_compat(OE) mlxfw(OE) devlink sunrpc dm_mirror dm_region_hash dm_log dm_mod nvidia_drm(POE) nvidia_modeset(POE) iTCO_wdt iTCO_vendor_support dcdbas nvidia(POE) sb_edac coretemp intel_rapl iosf_mbi kvm irqbypass crc32_pclmul
ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg lpc_ich mei_me mei ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mxm_wmi mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm bnx2x crct10dif_pclmul drm crct10dif_common crc32c_intel ahci libahci igb drm_panel_orientation_quirks mdio libcrc32c libata dca ptp pps_core megaraid_sas(OE) i2c_algo_bit wmi
CPU: 15 PID: 16801 Comm: 10.30.62.64-man Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R730/072T6D, BIOS 2.5.5 08/16/2017
task: ffff96f4a92b2080 ti: ffff96e842740000 task.ti: ffff96e842740000
RIP: 0010:[<ffffffffc0c211c3>]  [<ffffffffc0c211c3>] nfs_sb_deactive+0x13/0x30 [nfs]
RSP: 0018:ffff96e842743d00  EFLAGS: 00000206
RAX: ffff96f6f66ba000 RBX: ffff9706fdfdf160 RCX: ffff96f6f66ba000
RDX: 0000000000000000 RSI: 0000000000000006 RDI: ffff96f6f66bb800
RBP: ffff96e842743d40 R08: ffff96f6f66bb88c R09: ffffffffc0c5c58a
R10: ffff9706fdfdf160 R11: ffffea1e40e91000 R12: ffff96f6f66ba000
R13: ffff96e842743cf8 R14: ffff96f6f66bb88c R15: ffffffffc0c5c58a
FS:  0000000000000000(0000) GS:ffff9706fdfc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3074365ca0 CR3: 0000001f7fc10000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc0c7aba5>] ? nfs_delegation_reap_unclaimed+0x115/0x140 [nfsv4]
[<ffffffffc0c74e48>] nfs4_state_clear_reclaim_reboot+0x138/0x160 [nfsv4]
[<ffffffffc0c757e0>] nfs4_recovery_handle_error+0x40/0x170 [nfsv4]
[<ffffffffc0c768e4>] nfs4_do_reclaim+0x5f4/0x6c0 [nfsv4]
[<ffffffffc0c76f64>] nfs4_state_manager+0x5b4/0x8d0 [nfsv4]
[<ffffffffc0c77280>] ? nfs4_state_manager+0x8d0/0x8d0 [nfsv4]
[<ffffffffc0c7729f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffffa68c1c31>] kthread+0xd1/0xe0
[<ffffffffa68c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa6f74c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa68c1b60>] ? insert_kthread_work+0x40/0x40
Code: 0f 83 c0 01 39 c8 75 f0 31 c0 5d c3 0f 1f 40 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 48 8b 87 50 03 00 00 f0 ff 88 88 05 00 00 <0f> 94 c2 84 d2 75 06 c3 0f 1f 44 00 00 55 48 89 e5 e8 67 38 e2