Backtrace

NMI watchdog: BUG: soft lockup - CPU#35 stuck for 22s! [kiblnd_sd_04_01:4063]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter mgc(OE) lustre(OE) lmv(OE) mdc(OE) fid(OE) osc(OE) lov(OE) fld(OE) ko2iblnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) crc_t10dif crct10dif_generic rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) dell_rbu sunrpc nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) nvidia(POE) vfat fat kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper dell_smbios ablk_helper cryptd dcdbas dell_wmi_descriptor pcspkr i2c_piix4 ipmi_si ipmi_devintf ipmi_msghandler
acpi_power_meter knem(OE) binfmt_misc ip_tables ext4 mbcache jbd2 mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci crct10dif_pclmul libahci crct10dif_common crc32c_intel drm mlx5_core(OE) libata tg3 bnxt_en nvme nvme_core mlxfw(OE) psample auxiliary(OE) mlx_compat(OE) devlink ptp drm_panel_orientation_quirks pps_core wmi dm_mirror dm_region_hash dm_log dm_mod xpmem(OE) fuse
CPU: 35 PID: 4063 Comm: kiblnd_sd_04_01 Kdump: loaded Tainted: P           OEL ------------   3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R7525/0590KW, BIOS 2.10.2 10/25/2022
task: ffff9cc8b5a34200 ti: ffff9cc8b4204000 task.ti: ffff9cc8b4204000
RIP: 0010:[<ffffffffa93c844e>]  [<ffffffffa93c844e>] get_page_from_freelist+0x18e/0xac0
RSP: 0018:ffff9cc8b4207710  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff9cc8b4207700 RCX: 0000000000000002
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9d493feda000
RBP: ffff9cc8b4207818 R08: 0000000000000041 R09: 0000000000000000
R10: 0000000000002265 R11: ffffffffffffff83 R12: ffffffffa95a681d
R13: ffff9cc8b4207690 R14: ffff9d893feda4a0 R15: 000000000000000a
FS:  0000000000000000(0000) GS:ffff9d48bfcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f937e9a8000 CR3: 0000010fccc10000 CR4: 0000000000760ee0
PKRU: 00000000
Call Trace:
[<ffffffffa93c8f04>] __alloc_pages_nodemask+0x184/0xbe0
[<ffffffffa9586a45>] ? cpumask_next_and+0x35/0x50
[<ffffffffa97fb493>] ? alloc_iova+0x33/0x1c0
[<ffffffffa94193b8>] alloc_pages_current+0x98/0x110
[<ffffffffa93c393e>] __get_free_pages+0xe/0x40
[<ffffffffa93c3986>] get_zeroed_page+0x16/0x20
[<ffffffffa97fea32>] iommu_map_page+0x182/0x4c0
[<ffffffffa9800f7d>] map_sg+0x17d/0x2f0
[<ffffffffc074b227>] kiblnd_map_tx+0x2a7/0x430 [ko2iblnd]
[<ffffffffc074bd62>] kiblnd_setup_rd_kiov+0x252/0x3c0 [ko2iblnd]
[<ffffffffc07511be>] kiblnd_recv+0x3be/0x7f0 [ko2iblnd]
[<ffffffffc09a01e8>] lnet_ni_recv+0xc8/0x330 [lnet]
[<ffffffffc09a80e9>] lnet_parse_local+0x4c9/0xd40 [lnet]
[<ffffffffa92d757f>] ? ttwu_do_activate+0x6f/0x80
[<ffffffffc09a92fa>] lnet_parse+0x99a/0x11f0 [lnet]
[<ffffffffa92d30c2>] ? __wake_up_common+0x82/0x120
[<ffffffffc057b7fb>] ? mlx5_ib_poll_cq+0x40b/0xdd0 [mlx5_ib]
[<ffffffffc07519a3>] kiblnd_handle_rx+0x213/0x6b0 [ko2iblnd]
[<ffffffffc0758b62>] kiblnd_scheduler+0xfa2/0x11c0 [ko2iblnd]
[<ffffffffa92dadf0>] ? wake_up_state+0x20/0x20
[<ffffffffc0757bc0>] ? kiblnd_cq_event+0x90/0x90 [ko2iblnd]
[<ffffffffa92c5e61>] kthread+0xd1/0xe0
[<ffffffffa92c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffa9995de4>] ret_from_fork_nospec_begin+0xe/0x21
[<ffffffffa92c5d90>] ? insert_kthread_work+0x40/0x40
Code: 60 ff ff ff 04 75 46 44 8b 85 60 ff ff ff 8b 8d 18 ff ff ff 45 31 c9 8b 75 c4 4c 89 ff 45 89 c4 41 83 e4 03 49 63 c4 49 8b 1c c7 <49> 8b 87 d8 05 00 00 48 85 c0 48 89 da 4c 0f 49 c8 e8 6c af ff