Complete report #1424246
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
__smp_mb__before_atomic
|
vmlinux |
0xffffffff9e4c46aa
Build id: 3.10.0-862.el7.x86_64
|
0xa
|
|
2
|
|
nfs |
0xffffffffc09f5cbc
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
|
3
|
nfs_unlock_and_release_request
|
nfs |
0xffffffffc09f5f72
Build id: 3.10.0-862.el7.x86_64
|
0x12
|
|
4
|
nfs_commit_release_pages
|
nfs |
0xffffffffc09fa224
Build id: 3.10.0-862.el7.x86_64
|
0x74
|
|
5
|
|
vmlinux |
0xffffffff9eb1551e
Build id: 3.10.0-862.el7.x86_64
|
0x1e
|
|
6
|
nfs_commit_release
|
nfs |
0xffffffffc09f8c6c
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
|
7
|
rpc_free_task
|
sunrpc |
0xffffffffc05a8d73
Build id: 3.10.0-862.el7.x86_64
|
0x33
|
|
8
|
rpc_async_release
|
sunrpc |
0xffffffffc05a8dc2
Build id: 3.10.0-862.el7.x86_64
|
0x12
|
|
9
|
process_one_work
|
vmlinux |
0xffffffff9e4b2dff
Build id: 3.10.0-862.el7.x86_64
|
0x17f
|
|
10
|
worker_thread
|
vmlinux |
0xffffffff9e4b3ac6
Build id: 3.10.0-862.el7.x86_64
|
0x126
|
|
11
|
|
vmlinux |
0xffffffff9e4b39a0
Build id: 3.10.0-862.el7.x86_64
|
0x2a0
|
|
12
|
kthread
|
vmlinux |
0xffffffff9e4bae31
Build id: 3.10.0-862.el7.x86_64
|
0xd1
|
|
13
|
|
vmlinux |
0xffffffff9e4bad60
Build id: 3.10.0-862.el7.x86_64
|
0x40
|
|
14
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff9eb1f637
Build id: 3.10.0-862.el7.x86_64
|
0x21
|
|
15
|
|
vmlinux |
0xffffffff9e4bad60
Build id: 3.10.0-862.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#36 stuck for 22s! [kworker/36:1:32506]
yrfs: info (func:nic_detector_peer_detect line:269 comm:yrfs_nic_detect pid:3530): all peer nic recovered
Modules linked in:
veth xt_addrtype br_netfilter yrfs(OE) cmac arc4 md4 nls_utf8 cifs ccm xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) mlx4_ib(OE) ib_core(OE) mlx4_core(OE) mlx_compat(OE) devlink overlay(T) sunrpc vfat fat sb_edac coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
ablk_helper cryptd iTCO_wdt iTCO_vendor_support mxm_wmi ses mei_me enclosure i2c_i801 pcspkr sg mei joydev ipmi_si ipmi_devintf shpchp lpc_ich ipmi_msghandler wmi acpi_power_meter knem(OE) ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci mpt3sas ixgbe igb libata crct10dif_pclmul crct10dif_common crc32c_intel megaraid_sas mdio i2c_algo_bit ptp raid_class pps_core i2c_core scsi_transport_sas dca dm_mirror dm_region_hash dm_log dm_mod
CPU: 36 PID: 32506 Comm: kworker/36:1 Kdump: loaded Tainted: G OE ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: 4U Rack Server/60G24-US, BIOS 227 05/07/2018
Workqueue: nfsiod rpc_async_release [sunrpc]
task: ffff96e025c48fd0 ti: ffff96deb67a0000 task.ti: ffff96deb67a0000
RIP: 0010:[<ffffffff9e4c46aa>] [<ffffffff9e4c46aa>] __smp_mb__before_atomic+0xa/0x10
RSP: 0018:ffff96deb67a3d38 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 0000000000000005 RCX: 0000000000000034
RDX: ffff970640be39bc RSI: fffff9ee0d39a540 RDI: ffff970640be3980
RBP: ffff96deb67a3d50 R08: 2a00000000000000 R09: e7b834e695000000
R10: 1847c5077839a540 R11: 0000000000000000 R12: 0000000000000034
R13: ffff970640be2f80 R14: 8000000000000000 R15: 0640be2fc0000000
FS: 0000000000000000(0000) GS:ffff96f65f880000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0010a1000 CR3: 00000001e5a0e000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc09f5cbc>] ? nfs_unlock_request+0x1c/0x50 [nfs]
[<ffffffffc09f5f72>] nfs_unlock_and_release_request+0x12/0x30 [nfs]
[<ffffffffc09fa224>] nfs_commit_release_pages+0x74/0x230 [nfs]
[<ffffffff9eb1551e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffc09f8c6c>] nfs_commit_release+0x1c/0x30 [nfs]
[<ffffffffc05a8d73>] rpc_free_task+0x33/0x70 [sunrpc]
[<ffffffffc05a8dc2>] rpc_async_release+0x12/0x20 [sunrpc]
[<ffffffff9e4b2dff>] process_one_work+0x17f/0x440
[<ffffffff9e4b3ac6>] worker_thread+0x126/0x3c0
[<ffffffff9e4b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff9e4bae31>] kthread+0xd1/0xe0
[<ffffffff9e4bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff9eb1f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9e4bad60>] ? insert_kthread_work+0x40/0x40
Code: 00 55 b8 00 02 00 00 48 89 e5 5d c3 0f 1f 44 00 00 55 b8 00 40 00 00 48 89 e5 5d c3 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 5d <c3> 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 5d c3 0f 1f 44 00
Complete report #988952
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
nfs_unlock_request
|
nfs |
0xffffffffc07d1a85
Build id: 3.10.0-693.el7.x86_64
|
0x25
|
|
2
|
nfs_unlock_and_release_request
|
nfs |
0xffffffffc07d1d32
Build id: 3.10.0-693.el7.x86_64
|
0x12
|
|
3
|
nfs_commit_release_pages
|
nfs |
0xffffffffc07d5e64
Build id: 3.10.0-693.el7.x86_64
|
0x74
|
|
4
|
|
sunrpc |
0xffffffffc00c0f56
Build id: 3.10.0-693.el7.x86_64
|
0x1f6
|
|
5
|
|
vmlinux |
0xffffffff816ab9ee
Build id: 3.10.0-693.el7.x86_64
|
0x1e
|
|
6
|
nfs_commit_release
|
nfs |
0xffffffffc07d4916
Build id: 3.10.0-693.el7.x86_64
|
0x16
|
|
7
|
rpc_free_task
|
sunrpc |
0xffffffffc00cda10
Build id: 3.10.0-693.el7.x86_64
|
0x30
|
|
8
|
rpc_async_release
|
sunrpc |
0xffffffffc00cda62
Build id: 3.10.0-693.el7.x86_64
|
0x12
|
|
9
|
process_one_work
|
vmlinux |
0xffffffff810a881a
Build id: 3.10.0-693.el7.x86_64
|
0x17a
|
|
10
|
worker_thread
|
vmlinux |
0xffffffff810a94e6
Build id: 3.10.0-693.el7.x86_64
|
0x126
|
|
11
|
|
vmlinux |
0xffffffff810a93c0
Build id: 3.10.0-693.el7.x86_64
|
0x2a0
|
|
12
|
kthread
|
vmlinux |
0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
|
0xcf
|
|
13
|
|
vmlinux |
0xffffffff8108ddeb
Build id: 3.10.0-693.el7.x86_64
|
0x6bb
|
|
14
|
|
vmlinux |
0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
|
0x40
|
|
15
|
ret_from_fork
|
vmlinux |
0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
|
0x58
|
|
16
|
|
vmlinux |
0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#41 stuck for 22s! [kworker/41:0:387190]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink tun xt_addrtype br_netfilter overlay(T) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support dell_smbios dcdbas sparse_keymap skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel
lrw ipmi_ssif gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci tg3 megaraid_sas ixgbe(OE) libata dca ptp i2c_core pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 41 PID: 387190 Comm: kworker/41:0 Tainted: G OE ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R840/0TF0V7, BIOS 2.4.7 10/27/2019
Workqueue: nfsiod rpc_async_release [sunrpc]
task: ffff88612714cf10 ti: ffff8921c0a00000 task.ti: ffff8921c0a00000
RIP: 0010:[<ffffffffc07d1a85>] [<ffffffffc07d1a85>] nfs_unlock_request+0x25/0x50 [nfs]
RSP: 0000:ffff8921c0a03d40 EFLAGS: 00000246
RAX: 0000000000000001 RBX: 00000000fe1f8980 RCX: 0000000000000034
RDX: ffff8861234580bc RSI: ffffea00450ca600 RDI: ffff886123458080
RBP: ffff8921c0a03d50 R08: 3000000000000000 R09: a801143298000000
R10: 57fed5cdad0ca600 R11: 0000000000000000 R12: 0000000000000005
R13: ffff8861234580c0 R14: 0000000000000034 R15: ffff886123509300
FS: 0000000000000000(0000) GS:ffff88bebe480000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b112a9118a0 CR3: 0000001efe7fa000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff886123458080 ffff88c24aaccfc0 ffff8921c0a03d68 ffffffffc07d1d32
ffff886123458080 ffff8921c0a03dd0 ffffffffc07d5e64 ffffffffc00c0f56
0000000000000246 ffff886060732e18 ffff8921c0a03da0 ffffffff816ab9ee
Call Trace:
[<ffffffffc07d1d32>] nfs_unlock_and_release_request+0x12/0x30 [nfs]
[<ffffffffc07d5e64>] nfs_commit_release_pages+0x74/0x230 [nfs]
[<ffffffffc00c0f56>] ? call_transmit+0x1f6/0x2c0 [sunrpc]
[<ffffffff816ab9ee>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffc07d4916>] nfs_commit_release+0x16/0x30 [nfs]
[<ffffffffc00cda10>] rpc_free_task+0x30/0x70 [sunrpc]
[<ffffffffc00cda62>] rpc_async_release+0x12/0x20 [sunrpc]
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: c3 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 41 54 53 48 8b 47 40 48 89 fb a8 01 74 22 e8 f4 86 8e c0 4c 8d 63 40 f0 80 63 40 fe <e8> f6 86 8e c0 4c 89 e7 31 f6 e8 1c fe 8d c0 5b 41 5c 5d c3 48