ABRT Analytics - Problem #518568 - kernel in xprt_lookup

Info

Permalink
Function: xprt_lookup_rqst
First occurrence: 2020-12-02
Last occurrence: 2024-11-13
State: NEW
Components: kernel Tainted
Quality: -29

Statistics

Operating system	Count
CentOS 7	747

Architecture	Count
x86_64	747


Related packages	Count
kernel	747
0:3.10.0-693.2.2.el7	730
0:3.10.0-862.3.3.el7	13
0:3.10.0-1160.71.1.el7	2
0:3.10.0-1160.53.1.el7	1
0:3.10.0-693.5.2.el7	1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace

Complete report #1421388

Frame #	Function	Binary	Source or offset	Line
1	xprt_lookup_rqst	sunrpc	0xffffffffc02d83d6 Build id: 3.10.0-693.5.2.el7.x86_64	0x46
2	xs_tcp_data_recv	sunrpc	0xffffffffc02ddeb5 Build id: 3.10.0-693.5.2.el7.x86_64	0x4c5
3	? __put_single_page	vmlinux	0xffffffff81190353 Build id: 3.10.0-693.5.2.el7.x86_64	0x23
4	? skb_release_data	vmlinux	0xffffffff81572d82 Build id: 3.10.0-693.5.2.el7.x86_64	0x132
5	? xs_tcp_state_change	sunrpc	0xffffffffc02dd9f0 Build id: 3.10.0-693.5.2.el7.x86_64	0x2b0
6	tcp_read_sock	vmlinux	0xffffffff815d7b69 Build id: 3.10.0-693.5.2.el7.x86_64	0xa9
7	xs_tcp_data_receive_workfn	sunrpc	0xffffffffc02da863 Build id: 3.10.0-693.5.2.el7.x86_64	0xb3
8	process_one_work	vmlinux	0xffffffff810a882a Build id: 3.10.0-693.5.2.el7.x86_64	0x17a
9	worker_thread	vmlinux	0xffffffff810a94f6 Build id: 3.10.0-693.5.2.el7.x86_64	0x126
10	? manage_workers.isra.24	vmlinux	0xffffffff810a93d0 Build id: 3.10.0-693.5.2.el7.x86_64	0x2a0
11	kthread	vmlinux	0xffffffff810b099f Build id: 3.10.0-693.5.2.el7.x86_64	0xcf
12	? insert_kthread_work	vmlinux	0xffffffff810b08d0 Build id: 3.10.0-693.5.2.el7.x86_64	0x40
13	ret_from_fork	vmlinux	0xffffffff816b4fd8 Build id: 3.10.0-693.5.2.el7.x86_64	0x58
14	? insert_kthread_work	vmlinux	0xffffffff810b08d0 Build id: 3.10.0-693.5.2.el7.x86_64	0x40

NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:1H:298178]
Modules linked in: fuse rpcsec_gss_krb5 nfsv4 dns_resolver nfsv3 8021q nfs garp mrp stp llc fscache rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib ip6_tables rdma_ucm ib_ucm ib_uverbs iptable_filter ib_umad rdma_cm ib_cm iw_cm ib_core skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel iTCO_wdt dell_smbios iTCO_vendor_support sparse_keymap kvm dcdbas irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace binfmt_misc sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif
crct10dif_generic mgag200 i2c_algo_bit crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci tg3 libahci bnxt_en libata ptp i2c_core pps_core megaraid_sas(OE)
CPU: 0 PID: 298178 Comm: kworker/0:1H Tainted: G           OE  ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.13.3 12/17/2021
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff8801062ddee0 ti: ffff880104684000 task.ti: ffff880104684000
RIP: 0010:[<ffffffffc02d83d6>]  [<ffffffffc02d83d6>] xprt_lookup_rqst+0x46/0x120 [sunrpc]
RSP: 0018:ffff880104687cb0  EFLAGS: 00000202
RAX: ffff88015900fd18 RBX: ffffffffffffff7c RCX: ffff880026d434b8
RDX: ffff88000bcf44b8 RSI: 00000000f1ce43fd RDI: ffff88015900f800
RBP: ffff880104687cd0 R08: ffff881819478900 R09: 0000000000000004
R10: 0000000000000004 R11: ffff88015900fe00 R12: ffffea000459c000
R13: 0000000000007120 R14: 00000000000074a4 R15: ffff8818aaed2b90
FS:  0000000000000000(0000) GS:ffff881818a00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b204a11e990 CR3: 00000000019f2000 CR4: 00000000003407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000004 ffff88015900f800 ffff88015900fcd0 ffff88015900fdfc
ffff880104687d58 ffffffffc02ddeb5 ffffffff81190353 ffff881819478900
0000000000001f5c 0000000000009b4c 0000000000000000 ffffffff81572d82
Call Trace:
[<ffffffffc02ddeb5>] xs_tcp_data_recv+0x4c5/0xab0 [sunrpc]
[<ffffffff81190353>] ? __put_single_page+0x23/0x30
[<ffffffff81572d82>] ? skb_release_data+0x132/0x140
[<ffffffffc02dd9f0>] ? xs_tcp_state_change+0x2b0/0x2b0 [sunrpc]
[<ffffffff815d7b69>] tcp_read_sock+0xa9/0x1e0
[<ffffffffc02da863>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffff810a882a>] process_one_work+0x17a/0x440
[<ffffffff810a94f6>] worker_thread+0x126/0x3c0
[<ffffffff810a93d0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: f4 53 48 8b 97 18 05 00 00 48 39 d0 48 8d 9a 48 ff ff ff 74 29 39 72 e0 75 11 eb 79 66 0f 1f 84 00 00 00 00 00 44 39 61 e0 74 6a <48> 8b 8b b8 00 00 00 48 39 c8 48 8d 99 48 ff ff ff 75 e7 f6 05

Complete report #1191970

Frame #	Function	Binary	Source or offset	Line
1	xprt_lookup_rqst	sunrpc	0xffffffffc01483d6 Build id: 3.10.0-693.2.2.el7.x86_64	0x46
2	xs_tcp_data_recv	sunrpc	0xffffffffc014dea5 Build id: 3.10.0-693.2.2.el7.x86_64	0x4c5
3	? kmem_cache_free	vmlinux	0xffffffff811de9e2 Build id: 3.10.0-693.2.2.el7.x86_64	0x1e2
4	? xs_tcp_state_change	sunrpc	0xffffffffc014d9e0 Build id: 3.10.0-693.2.2.el7.x86_64	0x2b0
5	tcp_read_sock	vmlinux	0xffffffff815d7b19 Build id: 3.10.0-693.2.2.el7.x86_64	0xa9
6	xs_tcp_data_receive_workfn	sunrpc	0xffffffffc014a863 Build id: 3.10.0-693.2.2.el7.x86_64	0xb3
7	process_one_work	vmlinux	0xffffffff810a881a Build id: 3.10.0-693.2.2.el7.x86_64	0x17a
8	worker_thread	vmlinux	0xffffffff810a94e6 Build id: 3.10.0-693.2.2.el7.x86_64	0x126
9	? manage_workers.isra.24	vmlinux	0xffffffff810a93c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x2a0
10	kthread	vmlinux	0xffffffff810b098f Build id: 3.10.0-693.2.2.el7.x86_64	0xcf
11	? insert_kthread_work	vmlinux	0xffffffff810b08c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x40
12	ret_from_fork	vmlinux	0xffffffff816b4f58 Build id: 3.10.0-693.2.2.el7.x86_64	0x58
13	? insert_kthread_work	vmlinux	0xffffffff810b08c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x40

NMI watchdog: BUG: soft lockup - CPU#28 stuck for 24s! [kworker/28:1H:376188]
Modules linked in: fuse squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) ebtable_filter devlink ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt irqbypass iTCO_vendor_support dell_smbios dcdbas sparse_keymap crc32_pclmul crc32c_intel mgag200 ttm ghash_clmulni_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data
dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci libata i40e(OE) megaraid_sas tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 28 PID: 376188 Comm: kworker/28:1H Tainted: G        W  OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.21.1 03/07/2024
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff882f47220000 ti: ffff88022e12c000 task.ti: ffff88022e12c000
RIP: 0010:[<ffffffffc01483d6>]  [<ffffffffc01483d6>] xprt_lookup_rqst+0x46/0x120 [sunrpc]
RSP: 0018:ffff88022e12fcb0  EFLAGS: 00000202
RAX: ffff885f35612d18 RBX: ffffffff815ddf5a RCX: ffff883bfd856ab8
RDX: ffff8831111176b8 RSI: 000000007490382a RDI: ffff885f35612800
RBP: ffff88022e12fcd0 R08: ffff884b2936e300 R09: 0000000000000004
R10: 0000000000000004 R11: ffff885f35612e00 R12: 0000000000000020
R13: ffff88336f478a00 R14: ffff88045dd48f80 R15: ffff88336f478a00
FS:  0000000000000000(0000) GS:ffff882f7dd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaaaf427000 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000004 ffff885f35612800 ffff885f35612cd0 ffff885f35612dfc
ffff88022e12fd58 ffffffffc014dea5 ffff884b2936ca00 ffff884b2936e300
00000000000030c0 0000000000000f70 0000000000000000 ffff88022e12fd30
Call Trace:
[<ffffffffc014dea5>] xs_tcp_data_recv+0x4c5/0xab0 [sunrpc]
[<ffffffff811de9e2>] ? kmem_cache_free+0x1e2/0x200
[<ffffffffc014d9e0>] ? xs_tcp_state_change+0x2b0/0x2b0 [sunrpc]
[<ffffffff815d7b19>] tcp_read_sock+0xa9/0x1e0
[<ffffffffc014a863>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f58>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: f4 53 48 8b 97 18 05 00 00 48 39 d0 48 8d 9a 48 ff ff ff 74 29 39 72 e0 75 11 eb 79 66 0f 1f 84 00 00 00 00 00 44 39 61 e0 74 6a <48> 8b 8b b8 00 00 00 48 39 c8 48 8d 99 48 ff ff ff 75 e7 f6 05

Complete report #1189498

Frame #	Function	Binary	Source or offset	Line
1	xprt_lookup_rqst	sunrpc	0xffffffffc01173d6 Build id: 3.10.0-693.2.2.el7.x86_64	0x46
2	xs_tcp_data_recv	sunrpc	0xffffffffc011cea5 Build id: 3.10.0-693.2.2.el7.x86_64	0x4c5
3	? kmem_cache_free	vmlinux	0xffffffff811de9e2 Build id: 3.10.0-693.2.2.el7.x86_64	0x1e2
4	? xs_tcp_state_change	sunrpc	0xffffffffc011c9e0 Build id: 3.10.0-693.2.2.el7.x86_64	0x2b0
5	tcp_read_sock	vmlinux	0xffffffff815d7b19 Build id: 3.10.0-693.2.2.el7.x86_64	0xa9
6	xs_tcp_data_receive_workfn	sunrpc	0xffffffffc0119863 Build id: 3.10.0-693.2.2.el7.x86_64	0xb3
7	process_one_work	vmlinux	0xffffffff810a881a Build id: 3.10.0-693.2.2.el7.x86_64	0x17a
8	worker_thread	vmlinux	0xffffffff810a94e6 Build id: 3.10.0-693.2.2.el7.x86_64	0x126
9	? manage_workers.isra.24	vmlinux	0xffffffff810a93c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x2a0
10	kthread	vmlinux	0xffffffff810b098f Build id: 3.10.0-693.2.2.el7.x86_64	0xcf
11	? finish_task_switch	vmlinux	0xffffffff810bf107 Build id: 3.10.0-693.2.2.el7.x86_64	0x57
12	? insert_kthread_work	vmlinux	0xffffffff810b08c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x40
13	ret_from_fork	vmlinux	0xffffffff816b4f58 Build id: 3.10.0-693.2.2.el7.x86_64	0x58
14	? insert_kthread_work	vmlinux	0xffffffff810b08c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x40

NMI watchdog: BUG: soft lockup - CPU#59 stuck for 21s! [kworker/59:2H:368087]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass dell_smbios dcdbas sparse_keymap crc32_pclmul crc32c_intel mgag200 ghash_clmulni_intel ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data
dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa sg e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix sd_mod crc_t10dif mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun crct10dif_pclmul crct10dif_common ahci libahci libata megaraid_sas i40e(OE) tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 59 PID: 368087 Comm: kworker/59:2H Tainted: G           OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.21.1 03/07/2024
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff88023f030000 ti: ffff880193df0000 task.ti: ffff880193df0000
RIP: 0010:[<ffffffffc01173d6>]  [<ffffffffc01173d6>] xprt_lookup_rqst+0x46/0x120 [sunrpc]
RSP: 0018:ffff880193df3cb0  EFLAGS: 00000206
RAX: ffff882f7a62d518 RBX: 000000000000017a RCX: ffff88490aae9cb8
RDX: ffff88316006c2b8 RSI: 0000000015c0ddf2 RDI: ffff882f7a62d000
RBP: ffff880193df3cd0 R08: 0000000000000101 R09: 0000000000000004
R10: 0000000000000004 R11: ffff882f7a62d600 R12: ffff88603ffd7000
R13: ffff88603ffd7068 R14: ffffffff8118b1fe R15: ffff880193df3ca0
FS:  0000000000000000(0000) GS:ffff885f7c540000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffff40f4fd0 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000004 ffff882f7a62d000 ffff882f7a62d4d0 ffff882f7a62d5fc
ffff880193df3d58 ffffffffc011cea5 ffff88317869de00 ffff88317869f100
0000000000001f04 0000000000001258 0000000000000000 ffff880193df3d30
Call Trace:
[<ffffffffc011cea5>] xs_tcp_data_recv+0x4c5/0xab0 [sunrpc]
[<ffffffff811de9e2>] ? kmem_cache_free+0x1e2/0x200
[<ffffffffc011c9e0>] ? xs_tcp_state_change+0x2b0/0x2b0 [sunrpc]
[<ffffffff815d7b19>] tcp_read_sock+0xa9/0x1e0
[<ffffffffc0119863>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810bf107>] ? finish_task_switch+0x57/0x160
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f58>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: f4 53 48 8b 97 18 05 00 00 48 39 d0 48 8d 9a 48 ff ff ff 74 29 39 72 e0 75 11 eb 79 66 0f 1f 84 00 00 00 00 00 44 39 61 e0 74 6a <48> 8b 8b b8 00 00 00 48 39 c8 48 8d 99 48 ff ff ff 75 e7 f6 05

Complete report #1153839

Frame #	Function	Binary	Source or offset	Line
1	xprt_lookup_rqst	sunrpc	0xffffffffc01503d6 Build id: 3.10.0-693.2.2.el7.x86_64	0x46
2	xs_tcp_data_recv	sunrpc	0xffffffffc0155ea5 Build id: 3.10.0-693.2.2.el7.x86_64	0x4c5
3	? xs_tcp_state_change	sunrpc	0xffffffffc01559e0 Build id: 3.10.0-693.2.2.el7.x86_64	0x2b0
4	tcp_read_sock	vmlinux	0xffffffff815d7b19 Build id: 3.10.0-693.2.2.el7.x86_64	0xa9
5	xs_tcp_data_receive_workfn	sunrpc	0xffffffffc0152863 Build id: 3.10.0-693.2.2.el7.x86_64	0xb3
6	process_one_work	vmlinux	0xffffffff810a881a Build id: 3.10.0-693.2.2.el7.x86_64	0x17a
7	worker_thread	vmlinux	0xffffffff810a94e6 Build id: 3.10.0-693.2.2.el7.x86_64	0x126
8	? manage_workers.isra.24	vmlinux	0xffffffff810a93c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x2a0
9	kthread	vmlinux	0xffffffff810b098f Build id: 3.10.0-693.2.2.el7.x86_64	0xcf
10	? do_exit	vmlinux	0xffffffff8108ddeb Build id: 3.10.0-693.2.2.el7.x86_64	0x6bb
11	? insert_kthread_work	vmlinux	0xffffffff810b08c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x40
12	ret_from_fork	vmlinux	0xffffffff816b4f58 Build id: 3.10.0-693.2.2.el7.x86_64	0x58
13	? insert_kthread_work	vmlinux	0xffffffff810b08c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x40

NMI watchdog: BUG: soft lockup - CPU#41 stuck for 24s! [kworker/41:1H:293474]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver dell_rbu dm_mirror dm_region_hash dm_log joydev skx_edac dell_smbios edac_core sparse_keymap dcdbas intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul crc32c_intel ghash_clmulni_intel mgag200 aesni_intel ttm lrw gf128mul glue_helper ablk_helper cryptd drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm pcspkr mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data
dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci libata i40e(OE) megaraid_sas tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 41 PID: 293474 Comm: kworker/41:1H Tainted: G           OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/0WKGTH, BIOS 2.21.1 03/07/2024
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff88304c2c8fd0 ti: ffff8831ff35c000 task.ti: ffff8831ff35c000
RIP: 0010:[<ffffffffc01503d6>]  [<ffffffffc01503d6>] xprt_lookup_rqst+0x46/0x120 [sunrpc]
RSP: 0018:ffff8831ff35fcb0  EFLAGS: 00000206
RAX: ffff885f743c8518 RBX: 0000000000000090 RCX: ffff882f18a352b8
RDX: ffff8851fa586ab8 RSI: 0000000063c84622 RDI: ffff885f743c8000
RBP: ffff8831ff35fcd0 R08: ffff880373a2a800 R09: 0000000000000004
R10: 0000000000000004 R11: ffff885f743c8600 R12: 0000000000000090
R13: ffff8831ff35c000 R14: 0000000000000090 R15: 000010f8743c8258
FS:  0000000000000000(0000) GS:ffff885f7c300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffff01b0a38 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000004 ffff885f743c8000 ffff885f743c84d0 ffff885f743c85fc
ffff8831ff35fd58 ffffffffc0155ea5 0000000000000090 ffff880373a2a800
0000000000001370 0000000000001dec 0000000000000000 ffff880373a2a800
Call Trace:
[<ffffffffc0155ea5>] xs_tcp_data_recv+0x4c5/0xab0 [sunrpc]
[<ffffffffc01559e0>] ? xs_tcp_state_change+0x2b0/0x2b0 [sunrpc]
[<ffffffff815d7b19>] tcp_read_sock+0xa9/0x1e0
[<ffffffffc0152863>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f58>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: f4 53 48 8b 97 18 05 00 00 48 39 d0 48 8d 9a 48 ff ff ff 74 29 39 72 e0 75 11 eb 79 66 0f 1f 84 00 00 00 00 00 44 39 61 e0 74 6a <48> 8b 8b b8 00 00 00 48 39 c8 48 8d 99 48 ff ff ff 75 e7 f6 05

Complete report #1137649

Frame #	Function	Binary	Source or offset	Line
1	xprt_lookup_rqst	sunrpc	0xffffffffc014e3d6 Build id: 3.10.0-693.2.2.el7.x86_64	0x46
2	xs_tcp_data_recv	sunrpc	0xffffffffc0153ea5 Build id: 3.10.0-693.2.2.el7.x86_64	0x4c5
3	? kmem_cache_free	vmlinux	0xffffffff811de9e2 Build id: 3.10.0-693.2.2.el7.x86_64	0x1e2
4	? xs_tcp_state_change	sunrpc	0xffffffffc01539e0 Build id: 3.10.0-693.2.2.el7.x86_64	0x2b0
5	tcp_read_sock	vmlinux	0xffffffff815d7b19 Build id: 3.10.0-693.2.2.el7.x86_64	0xa9
6	xs_tcp_data_receive_workfn	sunrpc	0xffffffffc0150863 Build id: 3.10.0-693.2.2.el7.x86_64	0xb3
7	process_one_work	vmlinux	0xffffffff810a881a Build id: 3.10.0-693.2.2.el7.x86_64	0x17a
8	worker_thread	vmlinux	0xffffffff810a94e6 Build id: 3.10.0-693.2.2.el7.x86_64	0x126
9	? manage_workers.isra.24	vmlinux	0xffffffff810a93c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x2a0
10	kthread	vmlinux	0xffffffff810b098f Build id: 3.10.0-693.2.2.el7.x86_64	0xcf
11	? insert_kthread_work	vmlinux	0xffffffff810b08c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x40
12	ret_from_fork	vmlinux	0xffffffff816b4f58 Build id: 3.10.0-693.2.2.el7.x86_64	0x58
13	? insert_kthread_work	vmlinux	0xffffffff810b08c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x40

NMI watchdog: BUG: soft lockup - CPU#17 stuck for 21s! [kworker/17:1H:211233]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) squashfs loop overlay(T) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi joydev kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass mgag200 crc32_pclmul crc32c_intel ttm drm_kms_helper syscopyarea dell_smbios sysfillrect dcdbas sysimgblt fb_sys_fops sparse_keymap drm ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data
dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci libata i40e(OE) megaraid_sas tg3(OE) ptp pps_core [last unloaded: tracedev]
CPU: 17 PID: 211233 Comm: kworker/17:1H Tainted: G    B      OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.21.1 03/07/2024
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff88309c820fd0 ti: ffff884b97fa8000 task.ti: ffff884b97fa8000
RIP: 0010:[<ffffffffc014e3d6>]  [<ffffffffc014e3d6>] xprt_lookup_rqst+0x46/0x120 [sunrpc]
RSP: 0018:ffff884b97fabcb0  EFLAGS: 00000212
RAX: ffff8830b5d61d18 RBX: ffff885031b4a400 RCX: ffff8803f7fab4b8
RDX: ffff880405f36cb8 RSI: 00000000e8f1b9d4 RDI: ffff8830b5d61800
RBP: ffff884b97fabcd0 R08: ffff88425560e100 R09: 0000000000000004
R10: 0000000000000004 R11: ffff8830b5d61e00 R12: 0000000031b48001
R13: ffffea0140c6d200 R14: ffff8803bf51838e R15: ffff884b97fabca8
FS:  0000000000000000(0000) GS:ffff885fb8000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffff076b012 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000004 ffff8830b5d61800 ffff8830b5d61cd0 ffff8830b5d61dfc
ffff884b97fabd58 ffffffffc0153ea5 ffff88425560cf00 ffff88425560e100
0000000000000340 0000000000000838 0000000000000000 ffff884b97fabd30
Call Trace:
[<ffffffffc0153ea5>] xs_tcp_data_recv+0x4c5/0xab0 [sunrpc]
[<ffffffff811de9e2>] ? kmem_cache_free+0x1e2/0x200
[<ffffffffc01539e0>] ? xs_tcp_state_change+0x2b0/0x2b0 [sunrpc]
[<ffffffff815d7b19>] tcp_read_sock+0xa9/0x1e0
[<ffffffffc0150863>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f58>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: f4 53 48 8b 97 18 05 00 00 48 39 d0 48 8d 9a 48 ff ff ff 74 29 39 72 e0 75 11 eb 79 66 0f 1f 84 00 00 00 00 00 44 39 61 e0 74 6a <48> 8b 8b b8 00 00 00 48 39 c8 48 8d 99 48 ff ff ff 75 e7 f6 05

Complete report #1089811

Frame #	Function	Binary	Source or offset	Line
1	xprt_lookup_rqst	sunrpc	0xffffffffc018e3d6 Build id: 3.10.0-693.2.2.el7.x86_64	0x46
2	xs_tcp_data_recv	sunrpc	0xffffffffc0193ea5 Build id: 3.10.0-693.2.2.el7.x86_64	0x4c5
3	? kmem_cache_free	vmlinux	0xffffffff811de9e2 Build id: 3.10.0-693.2.2.el7.x86_64	0x1e2
4	? xs_tcp_state_change	sunrpc	0xffffffffc01939e0 Build id: 3.10.0-693.2.2.el7.x86_64	0x2b0
5	tcp_read_sock	vmlinux	0xffffffff815d7b19 Build id: 3.10.0-693.2.2.el7.x86_64	0xa9
6	xs_tcp_data_receive_workfn	sunrpc	0xffffffffc0190863 Build id: 3.10.0-693.2.2.el7.x86_64	0xb3
7	process_one_work	vmlinux	0xffffffff810a881a Build id: 3.10.0-693.2.2.el7.x86_64	0x17a
8	worker_thread	vmlinux	0xffffffff810a94e6 Build id: 3.10.0-693.2.2.el7.x86_64	0x126
9	? manage_workers.isra.24	vmlinux	0xffffffff810a93c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x2a0
10	kthread	vmlinux	0xffffffff810b098f Build id: 3.10.0-693.2.2.el7.x86_64	0xcf
11	? do_exit	vmlinux	0xffffffff8108ddeb Build id: 3.10.0-693.2.2.el7.x86_64	0x6bb
12	? insert_kthread_work	vmlinux	0xffffffff810b08c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x40
13	ret_from_fork	vmlinux	0xffffffff816b4f58 Build id: 3.10.0-693.2.2.el7.x86_64	0x58
14	? insert_kthread_work	vmlinux	0xffffffff810b08c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x40

NMI watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [kworker/9:1H:76870]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) squashfs loop overlay(T) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel iTCO_wdt kvm iTCO_vendor_support irqbypass dell_smbios crc32_pclmul dcdbas sparse_keymap crc32c_intel ghash_clmulni_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt aesni_intel fb_sys_fops lrw gf128mul glue_helper drm ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data
dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci libata i40e(OE) megaraid_sas tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 9 PID: 76870 Comm: kworker/9:1H Tainted: G           OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/08CYF7, BIOS 2.21.1 03/07/2024
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff8831130a0000 ti: ffff8831f91e8000 task.ti: ffff8831f91e8000
RIP: 0010:[<ffffffffc018e3d6>]  [<ffffffffc018e3d6>] xprt_lookup_rqst+0x46/0x120 [sunrpc]
RSP: 0018:ffff8831f91ebcb0  EFLAGS: 00000206
RAX: ffff885f74121518 RBX: 0000000000000202 RCX: ffff882708dcc4b8
RDX: ffff8829d1c24eb8 RSI: 00000000815d74c0 RDI: ffff885f74121000
RBP: ffff8831f91ebcd0 R08: ffff88305c532f00 R09: 0000000000000004
R10: 0000000000000004 R11: ffff885f74121600 R12: ffffffff8118aea4
R13: ffff8831f91ebc78 R14: ffff88603ffd7000 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffff885f7bf00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aaaab17ac70 CR3: 0000005f38bb3000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000004 ffff885f74121000 ffff885f741214d0 ffff885f741215fc
ffff8831f91ebd58 ffffffffc0193ea5 ffff8830f64bac00 ffff88305c532f00
0000000000000144 00000000000009b4 0000000000000000 ffff8831f91ebd30
Call Trace:
[<ffffffffc0193ea5>] xs_tcp_data_recv+0x4c5/0xab0 [sunrpc]
[<ffffffff811de9e2>] ? kmem_cache_free+0x1e2/0x200
[<ffffffffc01939e0>] ? xs_tcp_state_change+0x2b0/0x2b0 [sunrpc]
[<ffffffff815d7b19>] tcp_read_sock+0xa9/0x1e0
[<ffffffffc0190863>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f58>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: f4 53 48 8b 97 18 05 00 00 48 39 d0 48 8d 9a 48 ff ff ff 74 29 39 72 e0 75 11 eb 79 66 0f 1f 84 00 00 00 00 00 44 39 61 e0 74 6a <48> 8b 8b b8 00 00 00 48 39 c8 48 8d 99 48 ff ff ff 75 e7 f6 05

Complete report #1019758

Frame #	Function	Binary	Source or offset	Line
1	xprt_lookup_rqst	sunrpc	0xffffffffc014e3dd Build id: 3.10.0-693.2.2.el7.x86_64	0x4d
2	xs_tcp_data_recv	sunrpc	0xffffffffc0153ea5 Build id: 3.10.0-693.2.2.el7.x86_64	0x4c5
3	? kmem_cache_free	vmlinux	0xffffffff811de9e2 Build id: 3.10.0-693.2.2.el7.x86_64	0x1e2
4	? xs_tcp_state_change	sunrpc	0xffffffffc01539e0 Build id: 3.10.0-693.2.2.el7.x86_64	0x2b0
5	tcp_read_sock	vmlinux	0xffffffff815d7b19 Build id: 3.10.0-693.2.2.el7.x86_64	0xa9
6	xs_tcp_data_receive_workfn	sunrpc	0xffffffffc0150863 Build id: 3.10.0-693.2.2.el7.x86_64	0xb3
7	process_one_work	vmlinux	0xffffffff810a881a Build id: 3.10.0-693.2.2.el7.x86_64	0x17a
8	worker_thread	vmlinux	0xffffffff810a94e6 Build id: 3.10.0-693.2.2.el7.x86_64	0x126
9	? manage_workers.isra.24	vmlinux	0xffffffff810a93c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x2a0
10	kthread	vmlinux	0xffffffff810b098f Build id: 3.10.0-693.2.2.el7.x86_64	0xcf
11	? insert_kthread_work	vmlinux	0xffffffff810b08c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x40
12	ret_from_fork	vmlinux	0xffffffff816b4f58 Build id: 3.10.0-693.2.2.el7.x86_64	0x58
13	? insert_kthread_work	vmlinux	0xffffffff810b08c0 Build id: 3.10.0-693.2.2.el7.x86_64	0x40

NMI watchdog: BUG: soft lockup - CPU#11 stuck for 21s! [kworker/11:2H:112355]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt dell_smbios iTCO_vendor_support sparse_keymap irqbypass dcdbas crc32_pclmul crc32c_intel mgag200 ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ghash_clmulni_intel drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei shpchp i2c_i801 lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_pad acpi_power_meter binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data
dm_bufio dm_mod udf crc_itu_t vfat fat br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci libata i40e(OE) megaraid_sas tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 11 PID: 112355 Comm: kworker/11:2H Tainted: G           OEL ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.21.1 03/07/2024
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff88607bd52f70 ti: ffff884df61c8000 task.ti: ffff884df61c8000
RIP: 0010:[<ffffffffc014e3dd>]  [<ffffffffc014e3dd>] xprt_lookup_rqst+0x4d/0x120 [sunrpc]
RSP: 0018:ffff884df61cbcb0  EFLAGS: 00000212
RAX: ffff885f75c59518 RBX: 0000000000000010 RCX: ffff8857df261ab8
RDX: ffff880180783eb8 RSI: 000000008043900d RDI: ffff885f75c59000
RBP: ffff884df61cbcd0 R08: ffff883800900d00 R09: 0000000000000004
R10: 0000000000000004 R11: ffff885f75c59600 R12: ffffffffffffff10
R13: ffff885f75c595fc R14: ffff883226c29c5a R15: 0000000000000004
FS:  0000000000000000(0000) GS:ffff885fb7f40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aac7d200000 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000000000004 ffff885f75c59000 ffff885f75c594d0 ffff885f75c595fc
ffff884df61cbd58 ffffffffc0153ea5 ffff8833963c7b00 ffff883800900d00
0000000000000154 00000000000009cc 0000000000000000 ffff884df61cbd30
Call Trace:
[<ffffffffc0153ea5>] xs_tcp_data_recv+0x4c5/0xab0 [sunrpc]
[<ffffffff811de9e2>] ? kmem_cache_free+0x1e2/0x200
[<ffffffffc01539e0>] ? xs_tcp_state_change+0x2b0/0x2b0 [sunrpc]
[<ffffffff815d7b19>] tcp_read_sock+0xa9/0x1e0
[<ffffffffc0150863>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f58>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 48 39 d0 48 8d 9a 48 ff ff ff 74 29 39 72 e0 75 11 eb 79 66 0f 1f 84 00 00 00 00 00 44 39 61 e0 74 6a 48 8b 8b b8 00 00 00 <48> 39 c8 48 8d 99 48 ff ff ff 75 e7 f6 05 5c 89 03 00 01 0f 85

Complete report #1002672

Frame #	Function	Binary	Source or offset	Line
1	xprt_lookup_rqst	sunrpc	0xffffffffc03ed7b0 Build id: 3.10.0-1160.53.1.el7.x86_64	0x40
2	xs_tcp_data_recv	sunrpc	0xffffffffc03f3105 Build id: 3.10.0-1160.53.1.el7.x86_64	0x4c5
3	? xs_tcp_setup_socket	sunrpc	0xffffffffc03f2c40 Build id: 3.10.0-1160.53.1.el7.x86_64	0x490
4	tcp_read_sock	vmlinux	0xffffffffb3eaff0b Build id: 3.10.0-1160.53.1.el7.x86_64	0xab
5	xs_tcp_data_receive_workfn	sunrpc	0xffffffffc03efc03 Build id: 3.10.0-1160.53.1.el7.x86_64	0xb3
6	process_one_work	vmlinux	0xffffffffb38bde8f Build id: 3.10.0-1160.53.1.el7.x86_64	0x17f
7	worker_thread	vmlinux	0xffffffffb38befa6 Build id: 3.10.0-1160.53.1.el7.x86_64	0x126
8	? manage_workers.isra.26	vmlinux	0xffffffffb38bee80 Build id: 3.10.0-1160.53.1.el7.x86_64	0x2a0
9	kthread	vmlinux	0xffffffffb38c5e61 Build id: 3.10.0-1160.53.1.el7.x86_64	0xd1
10	? insert_kthread_work	vmlinux	0xffffffffb38c5d90 Build id: 3.10.0-1160.53.1.el7.x86_64	0x40
11	ret_from_fork_nospec_begin	vmlinux	0xffffffffb3f95ddd Build id: 3.10.0-1160.53.1.el7.x86_64	0x7
12	? insert_kthread_work	vmlinux	0xffffffffb38c5d90 Build id: 3.10.0-1160.53.1.el7.x86_64	0x40

NMI watchdog: BUG: soft lockup - CPU#4 stuck for 37s! [sssd_pam:20975]
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 37s! [kworker/5:0H:32726]
Modules linked in: binfmt_misc nfsv3 nfs_acl nfs lockd grace fscache xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter tun bridge stp llc vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg parport_pc vmw_vmci i2c_piix4 parport auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nfit ttm libnvdimm sd_mod ahci crc_t10dif crct10dif_generic
drm libahci ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 5 PID: 32726 Comm: kworker/5:0H Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff8ec0b0bec200 ti: ffff8ebe0e814000 task.ti: ffff8ebe0e814000
RIP: 0010:[<ffffffffc03ed7b0>]  [<ffffffffc03ed7b0>] xprt_lookup_rqst+0x40/0x130 [sunrpc]
RSP: 0018:ffff8ebe0e817cb0  EFLAGS: 00010206
RAX: ffff8ec095d02518 RBX: ffff8ebe35dde000 RCX: ffff8ebf081fe8b8
RDX: ffff8ebf0ac326b8 RSI: 00000000f8f23989 RDI: ffff8ec095d02000
RBP: ffff8ebe0e817cd0 R08: 0000000001000000 R09: c4fd377ffb3026a8
R10: ffff8ebf0a589b00 R11: 000000000000000c R12: ffff8ebeb7e8ae48
R13: 0000000000000000 R14: 0000000000000202 R15: 000000050dc18780
FS:  0000000000000000(0000) GS:ffff8ec0bfd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f95ec96a000 CR3: 00000001099cc000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffc03f3105>] xs_tcp_data_recv+0x4c5/0xad0 [sunrpc]
[<ffffffffc03f2c40>] ? xs_tcp_setup_socket+0x490/0x490 [sunrpc]
[<ffffffffb3eaff0b>] tcp_read_sock+0xab/0x1f0
[<ffffffffc03efc03>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffffb38bde8f>] process_one_work+0x17f/0x440
[<ffffffffb38befa6>] worker_thread+0x126/0x3c0
[<ffffffffb38bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb38c5e61>] kthread+0xd1/0xe0
[<ffffffffb38c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffb3f95ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb38c5d90>] ? insert_kthread_work+0x40/0x40
Code: 89 fd 41 54 41 89 f4 53 48 8b 97 18 05 00 00 48 39 d0 48 8d 9a 48 ff ff ff 74 29 39 72 e0 75 11 e9 7e 00 00 00 66 0f 1f 44 00 00 <44> 39 61 e0 74 72 48 8b 8b b8 00 00 00 48 39 c8 48 8d 99 48 ff

Complete report #512752

Frame #	Function	Binary	Source or offset	Line
1	xprt_lookup_rqst	sunrpc	0xffffffffc0739806 Build id: 3.10.0-1160.71.1.el7.x86_64	0x46
2	xs_tcp_data_recv	sunrpc	0xffffffffc073f5c5 Build id: 3.10.0-1160.71.1.el7.x86_64	0x4c5
3	? xs_tcp_state_change	sunrpc	0xffffffffc073f100 Build id: 3.10.0-1160.71.1.el7.x86_64	0x2f0
4	tcp_read_sock	vmlinux	0xffffffff9beb2f0b Build id: 3.10.0-1160.71.1.el7.x86_64	0xab
5	xs_tcp_data_receive_workfn	sunrpc	0xffffffffc073be93 Build id: 3.10.0-1160.71.1.el7.x86_64	0xb3
6	process_one_work	vmlinux	0xffffffff9b8bdfbf Build id: 3.10.0-1160.71.1.el7.x86_64	0x17f
7	worker_thread	vmlinux	0xffffffff9b8bf0d6 Build id: 3.10.0-1160.71.1.el7.x86_64	0x126
8	? manage_workers.isra.26	vmlinux	0xffffffff9b8befb0 Build id: 3.10.0-1160.71.1.el7.x86_64	0x2a0
9	kthread	vmlinux	0xffffffff9b8c5f91 Build id: 3.10.0-1160.71.1.el7.x86_64	0xd1
10	? insert_kthread_work	vmlinux	0xffffffff9b8c5ec0 Build id: 3.10.0-1160.71.1.el7.x86_64	0x40
11	ret_from_fork_nospec_begin	vmlinux	0xffffffff9bf99df7 Build id: 3.10.0-1160.71.1.el7.x86_64	0x21
12	? insert_kthread_work	vmlinux	0xffffffff9b8c5ec0 Build id: 3.10.0-1160.71.1.el7.x86_64	0x40

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [kworker/3:1H:14145]
Modules linked in: tcp_diag udp_diag inet_diag tcp_lp netatop(OE) nfsv3 rpcsec_gss_krb5 nfsv4 iscsi_tcp libiscsi_tcp libiscsi dns_resolver nfs scsi_transport_iscsi fscache ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables vmw_vsock_vmci_transport vsock iosf_mbi kvm_intel kvm irqbypass ppdev crc32_pclmul ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c ata_generic sd_mod sr_mod crc_t10dif crct10dif_generic cdrom pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm ahci crc32c_intel vmw_pvscsi ata_piix
libahci serio_raw libata vmxnet3 nfit drm_panel_orientation_quirks libnvdimm dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: netatop]
CPU: 3 PID: 14145 Comm: kworker/3:1H Tainted: G           OE  ------------   3.10.0-1160.71.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff9120ede28000 ti: ffff9120ed250000 task.ti: ffff9120ed250000
RIP: 0010:[<ffffffffc0739806>]  [<ffffffffc0739806>] xprt_lookup_rqst+0x46/0x130 [sunrpc]
RSP: 0018:ffff9120ed253cb0  EFLAGS: 00000297
RAX: ffff9120f6405520 RBX: ffff9120f6405258 RCX: ffff91206a5cb4b8
RDX: ffff9120dd07c6b8 RSI: 00000000785b29b9 RDI: ffff9120f6405000
RBP: ffff9120ed253cd0 R08: 0000000000000004 R09: 0000000000000004
R10: ffff9120e7144800 R11: 00000000000061a6 R12: ffffffffc0742c04
R13: ffff9120ed253c58 R14: ffff9120f9278a00 R15: ffff9120f6405258
FS:  0000000000000000(0000) GS:ffff9122756c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000c6d088 CR3: 0000000234296000 CR4: 00000000000607e0
Call Trace:
[<ffffffffc073f5c5>] xs_tcp_data_recv+0x4c5/0xad0 [sunrpc]
[<ffffffffc073f100>] ? xs_tcp_state_change+0x2f0/0x2f0 [sunrpc]
[<ffffffff9beb2f0b>] tcp_read_sock+0xab/0x1f0
[<ffffffffc073be93>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffff9b8bdfbf>] process_one_work+0x17f/0x440
[<ffffffff9b8bf0d6>] worker_thread+0x126/0x3c0
[<ffffffff9b8befb0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff9b8c5f91>] kthread+0xd1/0xe0
[<ffffffff9b8c5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffff9bf99df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9b8c5ec0>] ? insert_kthread_work+0x40/0x40
Code: f4 53 48 8b 97 20 05 00 00 48 39 d0 48 8d 9a 48 ff ff ff 74 29 39 72 e0 75 11 e9 7e 00 00 00 66 0f 1f 44 00 00 44 39 61 e0 74 72 <48> 8b 8b b8 00 00 00 48 39 c8 48 8d 99 48 ff ff ff 75 e7 f6 05

Complete report #19397

Frame #	Function	Binary	Source or offset	Line
1	xprt_lookup_rqst	sunrpc	0xffffffffc0b2d4d6 Build id: 3.10.0-862.3.3.el7.x86_64	0x46
2	xs_tcp_data_recv	sunrpc	0xffffffffc0b32f25 Build id: 3.10.0-862.3.3.el7.x86_64	0x4c5
3	? xs_tcp_setup_socket	sunrpc	0xffffffffc0b32a60 Build id: 3.10.0-862.3.3.el7.x86_64	0x490
4	tcp_read_sock	vmlinux	0xffffffffa324044b Build id: 3.10.0-862.3.3.el7.x86_64	0xab
5	xs_tcp_data_receive_workfn	sunrpc	0xffffffffc0b2fa83 Build id: 3.10.0-862.3.3.el7.x86_64	0xb3
6	process_one_work	vmlinux	0xffffffffa2cb312f Build id: 3.10.0-862.3.3.el7.x86_64	0x17f
7	worker_thread	vmlinux	0xffffffffa2cb3df6 Build id: 3.10.0-862.3.3.el7.x86_64	0x126
8	? manage_workers.isra.24	vmlinux	0xffffffffa2cb3cd0 Build id: 3.10.0-862.3.3.el7.x86_64	0x2a0
9	kthread	vmlinux	0xffffffffa2cbb161 Build id: 3.10.0-862.3.3.el7.x86_64	0xd1
10	? insert_kthread_work	vmlinux	0xffffffffa2cbb090 Build id: 3.10.0-862.3.3.el7.x86_64	0x40
11	ret_from_fork_nospec_begin	vmlinux	0xffffffffa332065d Build id: 3.10.0-862.3.3.el7.x86_64	0x7
12	? insert_kthread_work	vmlinux	0xffffffffa2cbb090 Build id: 3.10.0-862.3.3.el7.x86_64	0x40

NMI watchdog: BUG: soft lockup - CPU#44 stuck for 22s! [kworker/44:1H:286704]
Modules linked in: squashfs loop overlay(T) 8021q garp mrp stp llc beegfs(OE) nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ucm rpcrdma rdma_ucm ib_uverbs ib_iser rdma_cm iw_cm opa_vnic libiscsi ib_umad ib_ipoib scsi_transport_iscsi ib_cm iTCO_wdt iTCO_vendor_support sunrpc skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm hfi1 ast irqbypass crc32_pclmul ttm ipmi_ssif ghash_clmulni_intel drm_kms_helper aesni_intel lrw syscopyarea gf128mul sysfillrect glue_helper sysimgblt ablk_helper fb_sys_fops cryptd rdmavt ixgbe drm ib_core pcspkr ptp joydev pps_core mei_me mdio i2c_algo_bit i2c_i801 sg dca mei i2c_core shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler
nfit libnvdimm acpi_pad acpi_power_meter binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci libata
CPU: 44 PID: 286704 Comm: kworker/44:1H Tainted: G        W  OE  ------------ T 3.10.0-862.3.3.el7.x86_64 #1
Hardware name: Intel Corporation S2600BPB/S2600BPB, BIOS SE5C620.86B.00.01.0013.030920180427 03/09/2018
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff939e20b31fa0 ti: ffff938caf750000 task.ti: ffff938caf750000
RIP: 0010:[<ffffffffc0b2d4d6>]  [<ffffffffc0b2d4d6>] xprt_lookup_rqst+0x46/0x130 [sunrpc]
RSP: 0018:ffff938caf753cb0  EFLAGS: 00000297
RAX: ffff93bc1dbae518 RBX: ffff938caf753ca0 RCX: ffff93abb49e38b8
RDX: ffff93abb49e38b8 RSI: 00000000f524d585 RDI: ffff93bc1dbae000
RBP: ffff938caf753cd0 R08: ffff93bc0ee9bf00 R09: ffffffffc08bd0ba
R10: ffff93a41e01c310 R11: ffffd793f03fe700 R12: 000000000000002c
R13: ffff939cf0fa6ce4 R14: ffff93a8f06ffd00 R15: ffff939ed5f4be00
FS:  0000000000000000(0000) GS:ffff93a41e000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff6ada3d420 CR3: 00000011f7ba4000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffffc0b32f25>] xs_tcp_data_recv+0x4c5/0xad0 [sunrpc]
[<ffffffffc0b32a60>] ? xs_tcp_setup_socket+0x490/0x490 [sunrpc]
[<ffffffffa324044b>] tcp_read_sock+0xab/0x1f0
[<ffffffffc0b2fa83>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffffa2cb312f>] process_one_work+0x17f/0x440
[<ffffffffa2cb3df6>] worker_thread+0x126/0x3c0
[<ffffffffa2cb3cd0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffa2cbb161>] kthread+0xd1/0xe0
[<ffffffffa2cbb090>] ? insert_kthread_work+0x40/0x40
[<ffffffffa332065d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa2cbb090>] ? insert_kthread_work+0x40/0x40
Code: f4 53 48 8b 97 18 05 00 00 48 39 d0 48 8d 9a 48 ff ff ff 74 29 39 72 e0 75 11 e9 7e 00 00 00 66 0f 1f 44 00 00 44 39 61 e0 74 72 <48> 8b 8b b8 00 00 00 48 39 c8 48 8d 99 48 ff ff ff 75 e7 f6 05