Complete report #1425213
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_atomic_dec_and_lock
|
vmlinux |
0xffffffff9e74a54c
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
|
2
|
__put_nfs_open_context
|
nfs |
0xffffffffc09eb226
Build id: 3.10.0-862.el7.x86_64
|
0x46
|
|
3
|
put_nfs_open_context
|
nfs |
0xffffffffc09eb2f0
Build id: 3.10.0-862.el7.x86_64
|
0x10
|
|
4
|
nfs_free_request
|
nfs |
0xffffffffc09f5d9c
Build id: 3.10.0-862.el7.x86_64
|
0xac
|
|
5
|
nfs_page_group_destroy
|
nfs |
0xffffffffc09f5f44
Build id: 3.10.0-862.el7.x86_64
|
0x54
|
|
6
|
nfs_unlock_and_release_request
|
nfs |
0xffffffffc09f5f87
Build id: 3.10.0-862.el7.x86_64
|
0x27
|
|
7
|
nfs_commit_release_pages
|
nfs |
0xffffffffc09fa224
Build id: 3.10.0-862.el7.x86_64
|
0x74
|
|
8
|
nfs_commit_release
|
nfs |
0xffffffffc09f8c6c
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
|
9
|
rpc_free_task
|
sunrpc |
0xffffffffc05a8d73
Build id: 3.10.0-862.el7.x86_64
|
0x33
|
|
10
|
rpc_async_release
|
sunrpc |
0xffffffffc05a8dc2
Build id: 3.10.0-862.el7.x86_64
|
0x12
|
|
11
|
process_one_work
|
vmlinux |
0xffffffff9e4b2dff
Build id: 3.10.0-862.el7.x86_64
|
0x17f
|
|
12
|
worker_thread
|
vmlinux |
0xffffffff9e4b3ac6
Build id: 3.10.0-862.el7.x86_64
|
0x126
|
|
13
|
|
vmlinux |
0xffffffff9e4b39a0
Build id: 3.10.0-862.el7.x86_64
|
0x2a0
|
|
14
|
kthread
|
vmlinux |
0xffffffff9e4bae31
Build id: 3.10.0-862.el7.x86_64
|
0xd1
|
|
15
|
|
vmlinux |
0xffffffff9e4bad60
Build id: 3.10.0-862.el7.x86_64
|
0x40
|
|
16
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff9eb1f637
Build id: 3.10.0-862.el7.x86_64
|
0x21
|
|
17
|
|
vmlinux |
0xffffffff9e4bad60
Build id: 3.10.0-862.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#49 stuck for 22s! [kworker/49:0:25181]
yrfs: info (func:nic_detector_peer_detect line:269 comm:yrfs_nic_detect pid:3530): all peer nic recovered
Modules linked in:
veth xt_addrtype br_netfilter yrfs(OE) cmac arc4 md4 nls_utf8 cifs ccm xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) mlx4_ib(OE) ib_core(OE) mlx4_core(OE) mlx_compat(OE) devlink overlay(T) sunrpc vfat fat sb_edac coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
ablk_helper cryptd iTCO_wdt iTCO_vendor_support mxm_wmi ses mei_me enclosure i2c_i801 pcspkr sg mei joydev ipmi_si ipmi_devintf shpchp lpc_ich ipmi_msghandler wmi acpi_power_meter knem(OE) ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci mpt3sas ixgbe igb libata crct10dif_pclmul crct10dif_common crc32c_intel megaraid_sas mdio i2c_algo_bit ptp raid_class pps_core i2c_core scsi_transport_sas dca dm_mirror dm_region_hash dm_log dm_mod
CPU: 49 PID: 25181 Comm: kworker/49:0 Kdump: loaded Tainted: G OEL ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: 4U Rack Server/60G24-US, BIOS 227 05/07/2018
Workqueue: nfsiod rpc_async_release [sunrpc]
task: ffff96df9a6e9fa0 ti: ffff96deb6768000 task.ti: ffff96deb6768000
RIP: 0010:[<ffffffff9e74a54c>] [<ffffffff9e74a54c>] _atomic_dec_and_lock+0x1c/0x70
RSP: 0018:ffff96deb676bca8 EFLAGS: 00000246
RAX: 000000000016f32b RBX: ffff96deb676bce0 RCX: 000000000016f32a
RDX: 000000000016f32b RSI: ffff96f6c527cc00 RDI: ffff970ddaefbd80
RBP: ffff96deb676bcb8 R08: 8028000000000000 R09: f6e0aaf040140000
R10: 18473f26c3b1dac0 R11: 0000000000000000 R12: 0000000000000010
R13: ffffffff9e5f62d3 R14: ffffffffffffff10 R15: ffff96dfbfc03900
FS: 0000000000000000(0000) GS:ffff96f65fbc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000549d1a82000 CR3: 000000181aac0000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc09eb226>] __put_nfs_open_context+0x46/0x100 [nfs]
[<ffffffffc09eb2f0>] put_nfs_open_context+0x10/0x20 [nfs]
[<ffffffffc09f5d9c>] nfs_free_request+0xac/0x200 [nfs]
[<ffffffffc09f5f44>] nfs_page_group_destroy+0x54/0x70 [nfs]
[<ffffffffc09f5f87>] nfs_unlock_and_release_request+0x27/0x30 [nfs]
[<ffffffffc09fa224>] nfs_commit_release_pages+0x74/0x230 [nfs]
[<ffffffffc09f8c6c>] nfs_commit_release+0x1c/0x30 [nfs]
[<ffffffffc05a8d73>] rpc_free_task+0x33/0x70 [sunrpc]
[<ffffffffc05a8dc2>] rpc_async_release+0x12/0x20 [sunrpc]
[<ffffffff9e4b2dff>] process_one_work+0x17f/0x440
[<ffffffff9e4b3ac6>] worker_thread+0x126/0x3c0
[<ffffffff9e4b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff9e4bae31>] kthread+0xd1/0xe0
[<ffffffff9e4bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff9eb1f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9e4bad60>] ? insert_kthread_work+0x40/0x40
Code: 77 dd 5b 41 5c 5d c3 90 90 90 90 90 90 90 90 55 48 89 e5 53 48 89 fb 48 83 ec 08 8b 17 83 fa 01 74 2e 8d 4a ff 89 d0 f0 0f b1 0f <39> c2 89 c1 75 09 31 c0 48 83 c4 08 5b 5d c3 83 f9 01 74 11 8d
Complete report #686810
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_atomic_dec_and_lock
|
vmlinux |
0xffffffff81322a1c
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x1c
|
|
2
|
__put_nfs_open_context
|
nfs |
0xffffffffc0800076
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x46
|
|
3
|
put_nfs_open_context
|
nfs |
0xffffffffc0800140
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x10
|
|
4
|
nfs_free_request
|
nfs |
0xffffffffc080ab5c
Build id: 3.10.0-693.5.2.el7.x86_64
|
0xac
|
|
5
|
nfs_page_group_destroy
|
nfs |
0xffffffffc080ad04
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x54
|
|
6
|
nfs_unlock_and_release_request
|
nfs |
0xffffffffc080ad47
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x27
|
|
7
|
nfs_commit_release_pages
|
nfs |
0xffffffffc080ee64
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x74
|
|
8
|
|
nfs |
0xffffffffc080f2a9
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x39
|
|
9
|
nfs_commit_release
|
nfs |
0xffffffffc080d916
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x16
|
|
10
|
rpc_free_task
|
sunrpc |
0xffffffffc0370a20
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x30
|
|
11
|
rpc_async_release
|
sunrpc |
0xffffffffc0370a72
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x12
|
|
12
|
process_one_work
|
vmlinux |
0xffffffff810a882a
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x17a
|
|
13
|
worker_thread
|
vmlinux |
0xffffffff810a94f6
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x126
|
|
14
|
|
vmlinux |
0xffffffff810a93d0
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x2a0
|
|
15
|
kthread
|
vmlinux |
0xffffffff810b099f
Build id: 3.10.0-693.5.2.el7.x86_64
|
0xcf
|
|
16
|
|
vmlinux |
0xffffffff8108ddfb
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x6bb
|
|
17
|
|
vmlinux |
0xffffffff810b08d0
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x40
|
|
18
|
ret_from_fork
|
vmlinux |
0xffffffff816b4fd8
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x58
|
|
19
|
|
vmlinux |
0xffffffff810b08d0
Build id: 3.10.0-693.5.2.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/1:0:26471]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfsv3 nfs fscache 8021q garp mrp stp llc ip6_tables iptable_filter rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm mlx5_ib ib_core xfs libcrc32c skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass vfat fat iTCO_wdt crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_vendor_support joydev sg pcspkr ipmi_si ipmi_devintf ipmi_msghandler mei_me nfit mei i2c_i801 lpc_ich libnvdimm wmi shpchp acpi_cpufreq acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif
crct10dif_generic ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect crct10dif_pclmul sysimgblt crct10dif_common fb_sys_fops crc32c_intel mlx5_core ttm ixgbe mpt3sas drm ahci libahci libata raid_class scsi_transport_sas devlink mdio ptp pps_core i2c_core dca
CPU: 1 PID: 26471 Comm: kworker/1:0 Tainted: G L ------------ 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Ace Computers Ace /X11DPT-PS, BIOS 3.4 10/31/2020
Workqueue: nfsiod rpc_async_release [sunrpc]
task: ffff88166562dee0 ti: ffff880063054000 task.ti: ffff880063054000
RIP: 0010:[<ffffffff81322a1c>] [<ffffffff81322a1c>] _atomic_dec_and_lock+0x1c/0x70
RSP: 0018:ffff880063057ca8 EFLAGS: 00000246
RBP: ffff880063057cb8 R08: 8028000000000000 R09: 0ef79ced40140000
R10: f0ea66d0a7275005 R11: 0000000000000000 R12: 000000000040003f
R13: 0000000000000000 R14: 0000000000000202 R15: ffffea003bde7380
FS: 0000000000000000(0000) GS:ffff8817dda40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f39611fc4f0 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
ffff880ef79ced40 ffff8805d1033600 ffff880063057cf8 ffffffffc0800076
0000000063057ce8 ffff880ef79ced00 ffff8805d1033600 ffff8805d1033600
Call Trace:
[<ffffffffc0800076>] __put_nfs_open_context+0x46/0x100 [nfs]
[<ffffffffc0800140>] put_nfs_open_context+0x10/0x20 [nfs]
[<ffffffffc080ab5c>] nfs_free_request+0xac/0x200 [nfs]
[<ffffffffc080ad04>] nfs_page_group_destroy+0x54/0x70 [nfs]
[<ffffffffc080ad47>] nfs_unlock_and_release_request+0x27/0x30 [nfs]
[<ffffffffc080ee64>] nfs_commit_release_pages+0x74/0x230 [nfs]
[<ffffffffc080f2a9>] ? nfs_write_completion+0x39/0x160 [nfs]
[<ffffffffc080d916>] nfs_commit_release+0x16/0x30 [nfs]
[<ffffffffc0370a20>] rpc_free_task+0x30/0x70 [sunrpc]
[<ffffffffc0370a72>] rpc_async_release+0x12/0x20 [sunrpc]
[<ffffffff810a882a>] process_one_work+0x17a/0x440
[<ffffffff810a94f6>] worker_thread+0x126/0x3c0
[<ffffffff810a93d0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff8108ddfb>] ? do_exit+0x6bb/0xa40
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: 77 dd 5b 41 5c 5d c3 90 90 90 90 90 90 90 90 55 48 89 e5 53 48 89 fb 48 83 ec 08 8b 17 83 fa 01 74 2e 8d 4a ff 89 d0 f0 0f b1 0f <39> c2 89 c1 75 09 31 c0 48 83 c4 08 5b 5d c3 83 f9 01 74 11 8d