NMI watchdog: BUG: soft lockup - CPU#33 stuck for 23s! [SystemLogFlush:79817]
Modules linked in: nfnetlink_queue nfnetlink_log bluetooth rfkill ip_set binfmt_misc nfnetlink br_netfilter bridge stp llc overlay(T) vfat fat ext4 mbcache jbd2 skx_edac coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt iTCO_vendor_support mei_me mei sg i2c_i801 joydev megaraid_sas pcspkr i2c_core shpchp lpc_ich nfit acpi_power_meter libnvdimm acpi_cpufreq xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mlx5_core crct10dif_pclmul crct10dif_common crc32c_intel ahci i40e libahci libata mlxfw devlink ptp pps_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 33 PID: 79817 Comm: SystemLogFlush Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: XFUSION 2288H V5/BC11SPSCB10, BIOS 8.27 03/08/2022
task: ffff8ccccb310fd0 ti: ffff8cdbed398000 task.ti: ffff8cdbed398000
RIP: 0010:[<ffffffff82908802>]  [<ffffffff82908802>] native_queued_spin_lock_slowpath+0x122/0x200
RSP: 0018:ffff8cdbed39bb88  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8ccde7b0bc78 RCX: 0000000001090000
RDX: ffff8cf38ef19700 RSI: 0000000001610001 RDI: ffff8cf38ba00c80
RBP: ffff8cdbed39bb88 R08: ffff8cf38ec59700 R09: 0000000000000000
R10: ffff8cf38ec5ba60 R11: ffffdddcfc45d800 R12: ffff8cdbed39bb28
R13: ffff8ccde7b0bd48 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00007f317d5ff700(0000) GS:ffff8cf38ec40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2abdf66010 CR3: 0000004a12b6a000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff82f0842a>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff82f15680>] _raw_spin_lock+0x20/0x30
[<ffffffffc05fc437>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc05e8365>] ext4_ext_map_blocks+0x7b5/0xf60 [ext4]
[<ffffffffc05eabbb>] ? __ext4_handle_dirty_metadata+0x8b/0x220 [ext4]
[<ffffffffc05b8998>] ext4_map_blocks+0x98/0x6e0 [ext4]
[<ffffffffc05b592b>] ? __ext4_new_inode+0xf1b/0x12d0 [ext4]
[<ffffffffc05b9ac5>] ext4_getblk+0x65/0x200 [ext4]
[<ffffffffc05b9c87>] ext4_bread+0x27/0xc0 [ext4]
[<ffffffffc05c1f2a>] ext4_append+0x6a/0x110 [ext4]
[<ffffffffc05c727f>] ext4_mkdir+0x24f/0x410 [ext4]
[<ffffffff82a276ed>] vfs_mkdir+0xbd/0x170
[<ffffffff82a2dcfa>] SyS_mkdirat+0xca/0x100
[<ffffffff82a2dd49>] SyS_mkdir+0x19/0x20
[<ffffffff82f1f7d5>] system_call_fastpath+0x1c/0x21
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 00 97 01 00 48 03 14 c5 a0 53 53 83 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 

NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [python:10518]
Modules linked in: binfmt_misc udp_diag nf_conntrack_ftp tcp_diag inet_diag xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr vfat fat joydev sg ipmi_ssif ipmi_si ipmi_devintf
ipmi_msghandler pinctrl_amd i2c_designware_platform i2c_designware_core igb acpi_cpufreq i2c_piix4 sch_fq_codel ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci libata drm_panel_orientation_quirks megaraid_sas ixgbe mdio ptp pps_core nfit dca libnvdimm
CPU: 18 PID: 10518 Comm: python Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: Suma R6240A0/62DB32, BIOS CXWH052028 05/09/2022
task: ffff9be96dc78000 ti: ffff9c089819c000 task.ti: ffff9c089819c000
RIP: 0010:[<ffffffffafb12182>]  [<ffffffffafb12182>] native_queued_spin_lock_slowpath+0x122/0x200
RSP: 0018:ffff9c089819fb80  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff9bd8d341afb8 RCX: 0000000000910000
RDX: ffff9be84f71b780 RSI: 0000000000a10101 RDI: ffff9c28f9a79c80
RBP: ffff9c089819fb80 R08: ffff9be84f69b780 R09: 0000000000000000
R10: ffff9be84f69f0c0 R11: ffffef1801826680 R12: 0000000000000000
R13: ffff9c089819fb20 R14: ffff9bd8d341ad28 R15: 0000000000000000
FS:  00007f16e8d48740(0000) GS:ffff9be84f680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000563f193d51c0 CR3: 00000050670ae000 CR4: 0000000000340fe0
Call Trace:
[<ffffffffb015bf73>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffffb016a420>] _raw_spin_lock+0x20/0x30
[<ffffffffc06b2bb7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc069e8f5>] ext4_ext_map_blocks+0x7b5/0xf60 [ext4]
[<ffffffffafc78f9d>] ? __getblk+0x2d/0x300
[<ffffffffc06a11bb>] ? __ext4_handle_dirty_metadata+0x8b/0x220 [ext4]
[<ffffffffc066db38>] ext4_map_blocks+0x98/0x6e0 [ext4]
[<ffffffffc066a9fb>] ? __ext4_new_inode+0xf1b/0x12e0 [ext4]
[<ffffffffc066ecc5>] ext4_getblk+0x65/0x200 [ext4]
[<ffffffffc066ee87>] ext4_bread+0x27/0xc0 [ext4]
[<ffffffffc06772aa>] ext4_append+0x6a/0x110 [ext4]
[<ffffffffc067c5cf>] ext4_mkdir+0x24f/0x410 [ext4]
[<ffffffffafc4de7d>] vfs_mkdir+0xbd/0x170
[<ffffffffafc5431a>] SyS_mkdirat+0xca/0x100
[<ffffffffafc54369>] SyS_mkdir+0x19/0x20
[<ffffffffb0174ddb>] system_call_fastpath+0x22/0x27
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 b7 01 00 48 03 14 c5 a0 b9 74 b0 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 

NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [perl:51420]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc ext4 mbcache jbd2 iTCO_wdt iTCO_vendor_support mxm_wmi dcdbas sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci megaraid_sas tg3 crct10dif_pclmul libahci crct10dif_common crc32c_intel libata drm_panel_orientation_quirks ptp pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 14 PID: 51420 Comm: perl Kdump: loaded Tainted: G             L ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.1.7 06/16/2016
task: ffff8901ec5b20e0 ti: ffff88cb047c0000 task.ti: ffff88cb047c0000
RIP: 0010:[<ffffffff91f17fd0>]  [<ffffffff91f17fd0>] native_queued_spin_lock_slowpath+0x120/0x200
RSP: 0018:ffff88cb047c3b80  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88a4ae7dd400 RCX: 0000000000710000
RDX: ffff88a1ff9db8c0 RSI: 0000000001f10101 RDI: ffff8901fd516480
RBP: ffff88cb047c3b80 R08: ffff88a1ff3db8c0 R09: 0000000000000000
R10: ffff88a1ff3df0c0 R11: ffffe825287a31c0 R12: 0000000000000000
R13: ffff88cb047c3b20 R14: ffff88a4ae7dd170 R15: 0000000000000000
FS:  00007fb048adb740(0000) GS:ffff88a1ff3c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb04865b270 CR3: 0000008104e0a000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9257a004>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff925886b0>] _raw_spin_lock+0x20/0x30
[<ffffffffc107d8a7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc1069305>] ext4_ext_map_blocks+0x7d5/0xfa0 [ext4]
[<ffffffff920861ed>] ? __getblk+0x2d/0x300
[<ffffffffc106bcbb>] ? __ext4_handle_dirty_metadata+0x8b/0x220 [ext4]
[<ffffffffc1038068>] ext4_map_blocks+0x98/0x6e0 [ext4]
[<ffffffffc1034e5b>] ? __ext4_new_inode+0xf1b/0x12e0 [ext4]
[<ffffffffc1039245>] ext4_getblk+0x65/0x200 [ext4]
[<ffffffffc1039407>] ext4_bread+0x27/0xc0 [ext4]
[<ffffffffc104194a>] ext4_append+0x6a/0x110 [ext4]
[<ffffffffc1046c67>] ext4_mkdir+0x237/0x410 [ext4]
[<ffffffff9205a5af>] vfs_mkdir+0x10f/0x1d0
[<ffffffff920608aa>] SyS_mkdirat+0xca/0x100
[<ffffffff920608f9>] SyS_mkdir+0x19/0x20
[<ffffffff92592ed2>] system_call_fastpath+0x25/0x2a
Code: c1 e8 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 0f b5 92 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 <f3> 90 41 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [libdzmq:package:83964]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache binfmt_misc ext4 mbcache jbd2 iTCO_wdt iTCO_vendor_support mxm_wmi dcdbas sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr lpc_ich mei_me mei sg ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci libata crct10dif_pclmul crct10dif_common crc32c_intel megaraid_sas tg3 drm_panel_orientation_quirks ptp pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 83964 Comm: libdzmq:package Kdump: loaded Not tainted 3.10.0-1127.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.1.7 06/16/2016
task: ffff8959f8e7d230 ti: ffff89680ddb4000 task.ti: ffff89680ddb4000
RIP: 0010:[<ffffffffb0d17fd6>]  [<ffffffffb0d17fd6>] native_queued_spin_lock_slowpath+0x126/0x200
RSP: 0018:ffff89680ddb7b90  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff895cbd6237c8 RCX: 0000000000090000
RDX: ffff8959ff49b8c0 RSI: 0000000000a10101 RDI: ffff88fc2dabac80
RBP: ffff89680ddb7b90 R08: ffff89b9fea1b8c0 R09: 0000000000000000
R10: ffff89b9fea1f0c0 R11: ffffe2a38051d2c0 R12: 0000000000000000
R13: ffff89680ddb7b30 R14: ffff895cbd623538 R15: 0000000000000000
FS:  00007f390e89db80(0000) GS:ffff89b9fea00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001673000 CR3: 000000701af3c000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb137a004>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffffb13886b0>] _raw_spin_lock+0x20/0x30
[<ffffffffc0e498a7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc0e35305>] ext4_ext_map_blocks+0x7d5/0xfa0 [ext4]
[<ffffffffb0e861ed>] ? __getblk+0x2d/0x300
[<ffffffffc0e37cbb>] ? __ext4_handle_dirty_metadata+0x8b/0x220 [ext4]
[<ffffffffc0e04068>] ext4_map_blocks+0x98/0x6e0 [ext4]
[<ffffffffc0e00e5b>] ? __ext4_new_inode+0xf1b/0x12e0 [ext4]
[<ffffffffc0e05245>] ext4_getblk+0x65/0x200 [ext4]
[<ffffffffc0e05407>] ext4_bread+0x27/0xc0 [ext4]
[<ffffffffc0e0d94a>] ext4_append+0x6a/0x110 [ext4]
[<ffffffffc0e12c67>] ext4_mkdir+0x237/0x410 [ext4]
[<ffffffffb0e5a5af>] vfs_mkdir+0x10f/0x1d0
[<ffffffffb0e608aa>] SyS_mkdirat+0xca/0x100
[<ffffffffb1392ed2>] system_call_fastpath+0x25/0x2a
Code: 0d 48 98 83 e2 30 48 81 c2 c0 b8 01 00 48 03 14 c5 e0 0f 95 b1 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2