Info

Permalink
Function
nfs_sb_active
First occurrence
2021-04-16
Last occurrence
2021-05-18
State
NEW
Components
kernel
Quality
-21

Statistics

Operating system Count
CentOS 7 8,179
Architecture Count
x86_64 8,179
Related packages Count
kernel 8,179
 0:3.10.0-693.el7 7,026
 0:3.10.0-957.el7 1,153

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1090021
Frame # Function Binary Source or offset Line
1
nfs_sb_deactive
nfs 0xffffffffc05c2f53
Build id: 3.10.0-693.el7.x86_64
0x13
2
? nfs_reap_expired_delegations
nfsv4 0xffffffffc061c97d
Build id: 3.10.0-693.el7.x86_64
0x17d
3
nfs4_state_manager
nfsv4 0xffffffffc0618c22
Build id: 3.10.0-693.el7.x86_64
0x5f2
4
? nfs4_state_manager
nfsv4 0xffffffffc0618ef0
Build id: 3.10.0-693.el7.x86_64
0x8c0
5
nfs4_run_state_manager
nfsv4 0xffffffffc0618f0f
Build id: 3.10.0-693.el7.x86_64
0x1f
6
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
7
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
8
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
9
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#50 stuck for 22s! [10.1.1.151-mana:60076]
Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt dell_smbios iTCO_vendor_support sparse_keymap dcdbas vfat fat intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif pcspkr sg mei_me mei i2c_i801 shpchp wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper
syscopyarea crct10dif_pclmul sysfillrect crct10dif_common sysimgblt fb_sys_fops crc32c_intel ttm drm ahci libahci tg3 bnxt_en libata megaraid_sas i2c_core ptp pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 50 PID: 60076 Comm: 10.1.1.151-mana Tainted: G        W      ------------   3.10.0-693.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R750/06V45N, BIOS 1.3.8 08/31/2021
task: ffff883f5fb48000 ti: ffff8841a8328000 task.ti: ffff8841a8328000
RIP: 0010:[<ffffffffc05c2f53>]  [<ffffffffc05c2f53>] nfs_sb_deactive+0x13/0x30 [nfs]
RSP: 0018:ffff8841a832bdf0  EFLAGS: 00000202
RAX: ffff883f5b688800 RBX: ffff883f7fa59c40 RCX: ffff883f5b688800
RDX: 0000000000000000 RSI: 0000000000000008 RDI: ffff883f5b68b000
RBP: ffff8841a832be58 R08: ffff883f5b68b08c R09: ffffffff81185477
R10: ffff883f7fa59c40 R11: ffffea0106334b00 R12: ffff883f7fa59c40
R13: ffff8841a832bde8 R14: ffff883f5b68b08c R15: ffff8841a832bde8
FS:  0000000000000000(0000) GS:ffff883f7fa40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b6bc6229a00 CR3: 0000007f64626000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffffc061c97d ffffffffc0635340 ffff887f7823d680 ffff88010f350800
01000000a832be38 24fb4cfe67301375 000000043946735b 000000005be2f302
ffff883f7d580400 ffff883f7d580400 ffff883f7d5804c8 ffff883f7d580530
Call Trace:
[<ffffffffc061c97d>] ? nfs_reap_expired_delegations+0x17d/0x220 [nfsv4]
[<ffffffffc0618c22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc0618ef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc0618f0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 0f 83 c0 01 39 c8 75 f0 31 c0 5d c3 0f 1f 40 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 48 8b 87 50 03 00 00 f0 ff 88 88 05 00 00 <0f> 94 c2 84 d2 75 06 c3 0f 1f 44 00 00 55 48 89 e5 e8 57 16 c4 
Complete report #1006424
Frame # Function Binary Source or offset Line
1
nfs4_state_mark_reclaim_nograce
nfsv4 0xffffffffc05e9336
Build id: 3.10.0-693.el7.x86_64
0x6
2
? nfs_inode_find_state_and_recover
nfsv4 0xffffffffc05ecbee
Build id: 3.10.0-693.el7.x86_64
0xbe
3
nfs_reap_expired_delegations
nfsv4 0xffffffffc05ef9bb
Build id: 3.10.0-693.el7.x86_64
0x1bb
4
nfs4_state_manager
nfsv4 0xffffffffc05ebc22
Build id: 3.10.0-693.el7.x86_64
0x5f2
5
? nfs4_state_manager
nfsv4 0xffffffffc05ebef0
Build id: 3.10.0-693.el7.x86_64
0x8c0
6
nfs4_run_state_manager
nfsv4 0xffffffffc05ebf0f
Build id: 3.10.0-693.el7.x86_64
0x1f
7
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
8
? do_exit
vmlinux 0xffffffff8108ddeb
Build id: 3.10.0-693.el7.x86_64
0x6bb
9
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
10
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
11
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [135.252.218.207:95396]
Modules linked in: tcp_diag inet_diag binfmt_misc tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 tun devlink ebtable_filter ebtables ip6table_filter ip6_tables xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc mvfs(OE) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache overlay(T) vmw_vsock_vmci_transport vsock ppdev vmw_balloon crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx
crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix mptspi drm nfit scsi_transport_spi libata mptscsih serio_raw libnvdimm mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 13 PID: 95396 Comm: 135.252.218.207 Tainted: G           OE  ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff881bef9a0000 ti: ffff8800bb444000 task.ti: ffff8800bb444000
RIP: 0010:[<ffffffffc05e9336>]  [<ffffffffc05e9336>] nfs4_state_mark_reclaim_nograce+0x6/0x30 [nfsv4]
RSP: 0018:ffff8800bb447d78  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8800bb447d00 RCX: 000000000000000b
RDX: 000000000000000b RSI: ffff884b32a1e3c0 RDI: ffff88316127d800
RBP: ffff8800bb447de8 R08: 0000000000000000 R09: 0000000000000000
R10: ffff883175959c40 R11: ffffea0060ebe200 R12: 0000000000000010
R13: ffffffffc05ec631 R14: ffffffffffffff10 R15: ffff88316127d800
FS:  0000000000000000(0000) GS:ffff883175940000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2af78bf918 CR3: 000000316e695000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff8800bb447de8 ffffffffc05ecbee ffff8800bb447e14 ffff8826e104f050
ffff8826e104f0d8 0000000000000010 ffff88316127d800 00ffffff8121d05e
ffff8826e104efb8 ffff8826e104f050 ffff88316127d8c8 ffff88316127d800
Call Trace:
[<ffffffffc05ecbee>] ? nfs_inode_find_state_and_recover+0xbe/0x190 [nfsv4]
[<ffffffffc05ef9bb>] nfs_reap_expired_delegations+0x1bb/0x220 [nfsv4]
[<ffffffffc05ebc22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc05ebef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc05ebf0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 0f 1f 44 00 00 66 66 66 66 90 55 48 8b 86 a0 00 00 00 48 89 e5 48 89 87 a0 00 00 00 f0 ff 80 24 01 00 00 5d c3 66 66 66 66 90 55 <48> 89 e5 f0 80 4e 40 80 f0 80 66 40 bf 48 8b 46 30 f0 80 48 48 
Complete report #971322
Frame # Function Binary Source or offset Line
1
nfs4_state_mark_reclaim_nograce
nfsv4 0xffffffffc185e343
Build id: 3.10.0-693.el7.x86_64
0x13
2
nfs_inode_find_state_and_recover
nfsv4 0xffffffffc1861bee
Build id: 3.10.0-693.el7.x86_64
0xbe
3
nfs_reap_expired_delegations
nfsv4 0xffffffffc18649bb
Build id: 3.10.0-693.el7.x86_64
0x1bb
4
nfs4_state_manager
nfsv4 0xffffffffc1860c22
Build id: 3.10.0-693.el7.x86_64
0x5f2
5
? nfs4_state_manager
nfsv4 0xffffffffc1860ef0
Build id: 3.10.0-693.el7.x86_64
0x8c0
6
nfs4_run_state_manager
nfsv4 0xffffffffc1860f0f
Build id: 3.10.0-693.el7.x86_64
0x1f
7
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
8
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
9
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! [10.210.38.12-ma:5472]
Modules linked in: nvidia_uvm(OE) nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) vfio_iommu_type1 vfio 8021q garp mrp stp llc panfs(POE) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ucm dm_mirror dm_region_hash dm_log dm_mod rpcrdma rdma_ucm ib_uverbs ib_umad ib_iser ib_ipoib rdma_cm iw_cm libiscsi scsi_transport_iscsi ib_cm skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd mlx5_ib ib_core pcspkr joydev sg mei_me mei shpchp i2c_i801 lpc_ich ipmi_si nfit libnvdimm acpi_cpufreq acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace
sunrpc ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic ast i2c_algo_bit mlx5_core drm_kms_helper ttm syscopyarea sysfillrect sysimgblt fb_sys_fops drm crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci ptp pps_core libata devlink i2c_core wmi ipmi_devintf ipmi_msghandler [last unloaded: mdev]
CPU: 25 PID: 5472 Comm: 10.210.38.12-ma Tainted: P           OE  ------------   3.10.0-693.el7.x86_64 #1
Hardware name: Supermicro SYS-1029GP-TR/X11DPG-SN, BIOS 2.0b 02/26/2018
task: ffff8816ae3a6eb0 ti: ffff8809d3bac000 task.ti: ffff8809d3bac000
RIP: 0010:[<ffffffffc185e343>]  [<ffffffffc185e343>] nfs4_state_mark_reclaim_nograce+0x13/0x30 [nfsv4]
RSP: 0018:ffff8809d3bafd78  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffff8809d3bafd00 RCX: 000000000000000b
RDX: 000000000000000b RSI: ffff8809dcf4d140 RDI: ffff88180c27e000
RBP: ffff8809d3bafd78 R08: 0000000000000000 R09: 0000000000000000
R10: ffff88180dc59c40 R11: ffffea0043bfa100 R12: 0000000000000010
R13: ffffffffc1861631 R14: ffffffffffffff10 R15: ffff88180c27e000
FS:  0000000000000000(0000) GS:ffff88180dc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9fa0f9f1a0 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8809d3bafde8 ffffffffc1861bee ffff8809d3bafe14 ffff88029061aed0
ffff88029061af58 0000000000000010 ffff88180c27e000 00ffffff8121d05e
ffff88029061ae38 ffff88029061aed0 ffff88180c27e0c8 ffff88180c27e000
Call Trace:
[<ffffffffc1861bee>] nfs_inode_find_state_and_recover+0xbe/0x190 [nfsv4]
[<ffffffffc18649bb>] nfs_reap_expired_delegations+0x1bb/0x220 [nfsv4]
[<ffffffffc1860c22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc1860ef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc1860f0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 86 a0 00 00 00 48 89 e5 48 89 87 a0 00 00 00 f0 ff 80 24 01 00 00 5d c3 0f 1f 44 00 00 55 48 89 e5 f0 80 4e 40 80 f0 80 66 40 bf <48> 8b 46 30 f0 80 48 48 02 f0 80 8f 28 01 00 00 10 b8 01 00 00 
Complete report #937586
Frame # Function Binary Source or offset Line
1
nfs_delegation_grab_inode
nfsv4 0xffffffffc07813a5
Build id: 3.10.0-693.el7.x86_64
0x5
2
? nfs_reap_expired_delegations
nfsv4 0xffffffffc07828be
Build id: 3.10.0-693.el7.x86_64
0xbe
3
nfs4_state_manager
nfsv4 0xffffffffc077ec22
Build id: 3.10.0-693.el7.x86_64
0x5f2
4
? nfs4_state_manager
nfsv4 0xffffffffc077eef0
Build id: 3.10.0-693.el7.x86_64
0x8c0
5
nfs4_run_state_manager
nfsv4 0xffffffffc077ef0f
Build id: 3.10.0-693.el7.x86_64
0x1f
6
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
7
? do_exit
vmlinux 0xffffffff8108ddeb
Build id: 3.10.0-693.el7.x86_64
0x6bb
8
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
9
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [10.1.1.151-mana:19113]
Modules linked in: fuse rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt dell_smbios iTCO_vendor_support sparse_keymap dcdbas intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif pcspkr sg mei_me mei shpchp i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_pad acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul ttm crct10dif_common crc32c_intel drm ahci libahci tg3 libata megaraid_sas bnxt_en ptp i2c_core pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 10 PID: 19113 Comm: 10.1.1.151-mana Tainted: G        W      ------------   3.10.0-693.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R750/06V45N, BIOS 1.3.8 08/31/2021
task: ffff880026d29fa0 ti: ffff880113290000 task.ti: ffff880113290000
RIP: 0010:[<ffffffffc07813a5>]  [<ffffffffc07813a5>] nfs_delegation_grab_inode+0x5/0x50 [nfsv4]
RSP: 0018:ffff880113293df0  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff882fafb59c40 RCX: ffff882f9c790800
RDX: 0000000000000008 RSI: 0000000000000008 RDI: ffff882facf31c00
RBP: ffff880113293e58 R08: ffff882f9c79408c R09: ffffffff81185477
R10: ffff882fafb59c40 R11: ffffea0001642a80 R12: ffff882fafb59c40
R13: ffff880113293de8 R14: ffff882f9c79408c R15: ffff880113293de8
FS:  0000000000000000(0000) GS:ffff882fafb40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055f3ba9a3ff0 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffffc07828be ffffffffc079b340 ffff886246cdcb40 ffff882facf31c00
0100000013293e38 cf12717965e13624 000000044b34a795 00000000a902b340
ffff882fabb94400 ffff882fabb94400 ffff882fabb944c8 ffff882fabb94530
Call Trace:
[<ffffffffc07828be>] ? nfs_reap_expired_delegations+0xbe/0x220 [nfsv4]
[<ffffffffc077ec22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc077eef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc077ef0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 89 d1 48 8d 56 20 48 8b 76 10 e8 17 c6 fe ff 41 89 c4 48 89 df e8 8c ff ff ff 5b 44 89 e0 41 5c 5d c3 0f 1f 40 00 0f 1f 44 00 00 <55> 48 89 e5 41 54 49 89 fc 53 48 8d 5f 50 48 89 df e8 75 a7 f2 
Complete report #261725
Frame # Function Binary Source or offset Line
1
nfs_delegation_grab_inode
nfsv4 0xffffffffc09183ae
Build id: 3.10.0-693.el7.x86_64
0xe
2
nfs_reap_expired_delegations
nfsv4 0xffffffffc09198be
Build id: 3.10.0-693.el7.x86_64
0xbe
3
nfs4_state_manager
nfsv4 0xffffffffc0915c22
Build id: 3.10.0-693.el7.x86_64
0x5f2
4
? nfs4_state_manager
nfsv4 0xffffffffc0915ef0
Build id: 3.10.0-693.el7.x86_64
0x8c0
5
nfs4_run_state_manager
nfsv4 0xffffffffc0915f0f
Build id: 3.10.0-693.el7.x86_64
0x1f
6
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
7
? do_exit
vmlinux 0xffffffff8108ddeb
Build id: 3.10.0-693.el7.x86_64
0x6bb
8
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
9
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [192.168.0.200-m:6560]
Modules linked in: arc4 md4 nls_utf8 cifs fuse rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) vfat fat snd_hda_codec_hdmi dm_mirror dm_region_hash dm_log dm_mod snd_hda_intel snd_hda_codec snd_hda_core intel_powerclamp coretemp snd_hwdep kvm_intel kvm snd_seq joydev snd_seq_device snd_pcm irqbypass crc32_pclmul ghash_clmulni_intel snd_timer aesni_intel snd lrw gf128mul glue_helper ablk_helper cryptd soundcore ppdev sg pcspkr parport_pc shpchp parport tpm_crb acpi_pad int3400_thermal
nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic nouveau mxm_wmi i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci libata crct10dif_pclmul crct10dif_common crc32c_intel r8169 serio_raw mii i2c_hid i2c_core video wmi
CPU: 4 PID: 6560 Comm: 192.168.0.200-m Tainted: G           OEL ------------   3.10.0-693.el7.x86_64 #1
Hardware name: LENOVO 90KQCTO1WW/317A, BIOS M31KT22A 10/15/2020
task: ffff8807bf371fa0 ti: ffff8807e5a90000 task.ti: ffff8807e5a90000
RIP: 0010:[<ffffffffc09183ae>]  [<ffffffffc09183ae>] nfs_delegation_grab_inode+0xe/0x50 [nfsv4]
RSP: 0018:ffff8807e5a93de0  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff88036ae932e8 RCX: ffff880493698800
RDX: 0000000000000007 RSI: 0000000000000007 RDI: ffff8804be362580
RBP: ffff8807e5a93de8 R08: ffff88049369f88c R09: 0000000000000000
R10: 000000002ef2be01 R11: ffffea0020bbca00 R12: ffff8807e5a93d78
R13: ffff88036ae932e8 R14: 000000006d714489 R15: ffffea0020bbca00
FS:  0000000000000000(0000) GS:ffff88083eb00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffc08f4fd5 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff8804bcfc48c8 ffff8807e5a93e58 ffffffffc09198be ffffffffc09323c0
ffff880838857440 ffff8804be362580 00000000c04433e0 000c3d7c61514939
0000000400000000 000000006d714489 ffff8804bcfc4800 00000000ffffd8e5
Call Trace:
[<ffffffffc09198be>] nfs_reap_expired_delegations+0xbe/0x220 [nfsv4]
[<ffffffffc0915c22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc0915ef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc0915f0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 10 e8 17 c6 fe ff 41 89 c4 48 89 df e8 8c ff ff ff 5b 44 89 e0 41 5c 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 41 54 49 89 fc <53> 48 8d 5f 50 48 89 df e8 75 37 d9 c0 49 8b 7c 24 18 48 85 ff 
Complete report #131319
Frame # Function Binary Source or offset Line
1
nfs4_state_mark_reclaim_nograce
nfsv4 0xffffffffc07d334c
Build id: 3.10.0-693.el7.x86_64
0x1c
2
nfs_inode_find_state_and_recover
nfsv4 0xffffffffc07d6bee
Build id: 3.10.0-693.el7.x86_64
0xbe
3
nfs_reap_expired_delegations
nfsv4 0xffffffffc07d99bb
Build id: 3.10.0-693.el7.x86_64
0x1bb
4
nfs4_state_manager
nfsv4 0xffffffffc07d5c22
Build id: 3.10.0-693.el7.x86_64
0x5f2
5
? nfs4_state_manager
nfsv4 0xffffffffc07d5ef0
Build id: 3.10.0-693.el7.x86_64
0x8c0
6
nfs4_run_state_manager
nfsv4 0xffffffffc07d5f0f
Build id: 3.10.0-693.el7.x86_64
0x1f
7
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
8
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
9
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [10.20.240.196-m:29248]
Modules linked in: nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ftdi_sio vfat fat intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel iTCO_wdt iTCO_vendor_support kvm snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_hda_intel sg snd_hda_codec snd_hda_core snd_hwdep snd_seq irqbypass snd_seq_device acpi_pad snd_pcm snd_timer snd soundcore crc32_pclmul i2c_i801 ghash_clmulni_intel mei_me mei shpchp pcspkr cryptd nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod
sr_mod cdrom crc_t10dif crct10dif_generic i915 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm ahci igb libahci libata crct10dif_pclmul crct10dif_common crc32c_intel ptp pps_core serio_raw dca i2c_algo_bit i2c_hid video i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 29248 Comm: 10.20.240.196-m Not tainted 3.10.0-693.el7.x86_64 #1
Hardware name: PFU LIMITED AR8300M300L/310L/MBE-561A, BIOS SPL:1.11 10/21/2020
task: ffff880430284f10 ti: ffff8803c13ec000 task.ti: ffff8803c13ec000
RIP: 0010:[<ffffffffc07d334c>]  [<ffffffffc07d334c>] nfs4_state_mark_reclaim_nograce+0x1c/0x30 [nfsv4]
RSP: 0018:ffff8803c13efd78  EFLAGS: 00000202
RAX: ffff88044de64400 RBX: ffff8803c13efd00 RCX: 000000000000000b
RDX: 000000000000000b RSI: ffff88043d2800c0 RDI: ffff880453e89000
RBP: ffff8803c13efd78 R08: 0000000000000000 R09: 0000000000000000
R10: ffff88046ed19c40 R11: ffffea0011725380 R12: 0000000000000010
R13: ffffffffc07d6640 R14: ffffffffffffff10 R15: ffff880453e89000
FS:  0000000000000000(0000) GS:ffff88046ed00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7b5209e000 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8803c13efde8 ffffffffc07d6bee ffff8803c13efe14 ffff88045ba51228
ffff88045ba512b0 0000000000000003 ffff880453e89000 00ffffff8121d05e
ffff88045ba51190 ffff88045ba51228 ffff880453e890c8 ffff880453e89000
Call Trace:
[<ffffffffc07d6bee>] nfs_inode_find_state_and_recover+0xbe/0x190 [nfsv4]
[<ffffffffc07d99bb>] nfs_reap_expired_delegations+0x1bb/0x220 [nfsv4]
[<ffffffffc07d5c22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc07d5ef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc07d5f0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 89 87 a0 00 00 00 f0 ff 80 24 01 00 00 5d c3 0f 1f 44 00 00 55 48 89 e5 f0 80 4e 40 80 f0 80 66 40 bf 48 8b 46 30 f0 80 48 48 02 <f0> 80 8f 28 01 00 00 10 b8 01 00 00 00 5d c3 0f 1f 44 00 00 0f 
Complete report #1032079
Frame # Function Binary Source or offset Line
1
nfs_sb_active
nfs 0xffffffffc088de6e
Build id: 3.10.0-693.el7.x86_64
0x2e
2
nfs_reap_expired_delegations
nfsv4 0xffffffffc09b88b2
Build id: 3.10.0-693.el7.x86_64
0xb2
3
nfs4_state_manager
nfsv4 0xffffffffc09b4c22
Build id: 3.10.0-693.el7.x86_64
0x5f2
4
? nfs4_state_manager
nfsv4 0xffffffffc09b4ef0
Build id: 3.10.0-693.el7.x86_64
0x8c0
5
nfs4_run_state_manager
nfsv4 0xffffffffc09b4f0f
Build id: 3.10.0-693.el7.x86_64
0x1f
6
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
7
? do_exit
vmlinux 0xffffffff8108ddeb
Build id: 3.10.0-693.el7.x86_64
0x6bb
8
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
9
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [10.210.38.12-ma:74271]
Modules linked in: panfs(POE) 8021q garp mrp stp llc nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ib_isert iscsi_target_mod iTCO_wdt iTCO_vendor_support ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi ib_ucm rpcrdma kvm_intel rdma_ucm kvm ib_uverbs ib_iser irqbypass ib_ipoib libiscsi crc32_pclmul scsi_transport_iscsi ghash_clmulni_intel aesni_intel lrw ib_umad gf128mul glue_helper ablk_helper cryptd rdma_cm ib_cm joydev pcspkr iw_cm sg i2c_i801 mei_me mei shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad mlx5_ib ib_core nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper crct10dif_pclmul
crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops mlx5_core ttm drm igb ahci libahci libata nvme nvme_core devlink dca i2c_algo_bit ptp i2c_core pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 74271 Comm: 10.210.38.12-ma Tainted: P           OE  ------------   3.10.0-693.el7.x86_64 #1
Hardware name: Penguin Computing Relion XE1112/MR91-FS0-ZB, BIOS R03 06/03/2019
task: ffff8887d9bb2f70 ti: ffff88c409a9c000 task.ti: ffff88c409a9c000
RIP: 0010:[<ffffffffc088de6e>]  [<ffffffffc088de6e>] nfs_sb_active+0x2e/0x80 [nfs]
RSP: 0018:ffff88c409a9fde8  EFLAGS: 00000246
RAX: 0000000000000004 RBX: ffff880497f87200 RCX: ffff88015b876000
RDX: 0000000000000004 RSI: 0000000000000005 RDI: ffff88015b875800
RBP: ffff88c409a9fde8 R08: ffff88015b87588c R09: 0000000000000000
R10: ffff88bd3da59c40 R11: ffffea02f4ea2800 R12: ffff88c409a9fd78
R13: ffff885b7810f050 R14: 000000007a27b1ac R15: ffff88bd320ad000
FS:  0000000000000000(0000) GS:ffff88bd3da40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2385d61000 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff88c409a9fe58 ffffffffc09b88b2 ffffffffc09d13c0 ffff88bd3b279380
ffff886e79b03f80 0000000000000000 3c8ac4026693e0ac 0000000400000000
000000007a27b1ac ffff88bd320ad000 00000000ffffd8e5 ffff88bd320ad0c8
Call Trace:
[<ffffffffc09b88b2>] nfs_reap_expired_delegations+0xb2/0x220 [nfsv4]
[<ffffffffc09b4c22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc09b4ef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc09b4f0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 55 8b 97 8c 00 00 00 48 8b 8f 50 03 00 00 48 89 e5 85 d2 74 5b 8d 72 01 4c 8d 87 8c 00 00 00 89 d0 f0 0f b1 b7 8c 00 00 00 <39> d0 89 c6 75 2b b8 01 00 00 00 f0 0f c1 81 88 05 00 00 85 c0 
Complete report #261596
Frame # Function Binary Source or offset Line
1
nfs_sb_deactive
nfs 0xffffffffc08bff53
Build id: 3.10.0-693.el7.x86_64
0x13
2
? nfs_reap_expired_delegations
nfsv4 0xffffffffc091995e
Build id: 3.10.0-693.el7.x86_64
0x15e
3
nfs4_state_manager
nfsv4 0xffffffffc0915c22
Build id: 3.10.0-693.el7.x86_64
0x5f2
4
? nfs4_state_manager
nfsv4 0xffffffffc0915ef0
Build id: 3.10.0-693.el7.x86_64
0x8c0
5
nfs4_run_state_manager
nfsv4 0xffffffffc0915f0f
Build id: 3.10.0-693.el7.x86_64
0x1f
6
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
7
? do_exit
vmlinux 0xffffffff8108ddeb
Build id: 3.10.0-693.el7.x86_64
0x6bb
8
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
9
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [192.168.0.200-m:6560]
Modules linked in: arc4 md4 nls_utf8 cifs fuse rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) vfat fat snd_hda_codec_hdmi dm_mirror dm_region_hash dm_log dm_mod snd_hda_intel snd_hda_codec snd_hda_core intel_powerclamp coretemp snd_hwdep kvm_intel kvm snd_seq joydev snd_seq_device snd_pcm irqbypass crc32_pclmul ghash_clmulni_intel snd_timer aesni_intel snd lrw gf128mul glue_helper ablk_helper cryptd soundcore ppdev sg pcspkr parport_pc shpchp parport tpm_crb acpi_pad int3400_thermal
nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic nouveau mxm_wmi i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci libata crct10dif_pclmul crct10dif_common crc32c_intel r8169 serio_raw mii i2c_hid i2c_core video wmi
CPU: 4 PID: 6560 Comm: 192.168.0.200-m Tainted: G           OEL ------------   3.10.0-693.el7.x86_64 #1
Hardware name: LENOVO 90KQCTO1WW/317A, BIOS M31KT22A 10/15/2020
task: ffff8807bf371fa0 ti: ffff8807e5a90000 task.ti: ffff8807e5a90000
RIP: 0010:[<ffffffffc08bff53>]  [<ffffffffc08bff53>] nfs_sb_deactive+0x13/0x30 [nfs]
RSP: 0018:ffff8807e5a93df0  EFLAGS: 00000246
RAX: ffff880493698800 RBX: ffff8804bcfc4800 RCX: 00000000000003b5
RDX: 0000000000000001 RSI: ffff88036ae93370 RDI: ffff88049369f800
RBP: ffff8807e5a93e58 R08: ffff880838857440 R09: 0000000000000000
R10: 000000002ef2be01 R11: ffffea0020bbca00 R12: ffff88036ae932e8
R13: ffffffffc0919a6d R14: ffff8807e5a93d78 R15: ffff88036ae932e8
FS:  0000000000000000(0000) GS:ffff88083eb00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffc08f4fe4 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffffc091995e ffffffffc09323c0 ffff880838857440 ffff8804be362580
00000000c04433e0 000c3d7c61514939 0000000400000000 000000006d714489
ffff8804bcfc4800 00000000ffffd8e5 ffff8804bcfc48c8 ffff8804bcfc4930
Call Trace:
[<ffffffffc091995e>] ? nfs_reap_expired_delegations+0x15e/0x220 [nfsv4]
[<ffffffffc0915c22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc0915ef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc0915f0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 0f 83 c0 01 39 c8 75 f0 31 c0 5d c3 0f 1f 40 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 48 8b 87 50 03 00 00 f0 ff 88 88 05 00 00 <0f> 94 c2 84 d2 75 06 c3 0f 1f 44 00 00 55 48 89 e5 e8 57 46 94 
Complete report #807120
Frame # Function Binary Source or offset Line
1
nfs_sb_active
nfs 0xffffffffc07e5e85
Build id: 3.10.0-693.el7.x86_64
0x45
2
nfs_reap_expired_delegations
nfsv4 0xffffffffc08968b2
Build id: 3.10.0-693.el7.x86_64
0xb2
3
nfs4_state_manager
nfsv4 0xffffffffc0892c22
Build id: 3.10.0-693.el7.x86_64
0x5f2
4
? nfs4_state_manager
nfsv4 0xffffffffc0892ef0
Build id: 3.10.0-693.el7.x86_64
0x8c0
5
nfs4_run_state_manager
nfsv4 0xffffffffc0892f0f
Build id: 3.10.0-693.el7.x86_64
0x1f
6
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
7
? do_exit
vmlinux 0xffffffff8108ddeb
Build id: 3.10.0-693.el7.x86_64
0x6bb
8
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
9
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#25 stuck for 23s! [10.62.105.205-m:134410]
Modules linked in: tcp_lp fuse rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache sr_mod cdrom dm_mirror dm_region_hash dm_log skx_edac edac_core coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_service_time iTCO_wdt iTCO_vendor_support uas usb_storage joydev sg pcspkr mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_cpufreq dm_multipath nfsd dm_mod auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod lpfc i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel mpt3sas ahci libahci crc_t10dif crct10dif_generic libata crct10dif_pclmul scsi_transport_fc scsi_tgt i40e(OE) raid_class
ptp i2c_core crct10dif_common scsi_transport_sas pps_core
CPU: 25 PID: 134410 Comm: 10.62.105.205-m Tainted: G           OEL ------------   3.10.0-693.el7.x86_64 #1
Hardware name: Sugon I620-G30/60P24-US, BIOS 0JGST219 05/30/2019
task: ffff8804701e9fa0 ti: ffff880835bb0000 task.ti: ffff880835bb0000
RIP: 0010:[<ffffffffc07e5e85>]  [<ffffffffc07e5e85>] nfs_sb_active+0x45/0x80 [nfs]
RSP: 0018:ffff880835bb3de8  EFLAGS: 00000206
RAX: 0000000000000071 RBX: ffff88086635fd40 RCX: ffff880179131000
RDX: 0000000000000008 RSI: 0000000000000008 RDI: ffff880179133800
RBP: ffff880835bb3de8 R08: ffff88017913388c R09: 0000000000000000
R10: 0000000000000000 R11: 7fffffffffffffff R12: ffff880835bb3d78
R13: ffff880501ace408 R14: 00000000db21002e R15: ffff880835bb3dc0
FS:  0000000000000000(0000) GS:ffff88046e940000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff4663319e5 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff880835bb3e58 ffffffffc08968b2 ffffffffc08af3c0 ffff88081da39500
ffff88085e271c00 0100000000000000 b0036ce861b040a1 000000040089a28f
00000000db21002e ffff88046c019000 ffff88046c019000 ffff88046c0190c8
Call Trace:
[<ffffffffc08968b2>] nfs_reap_expired_delegations+0xb2/0x220 [nfsv4]
[<ffffffffc0892c22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc0892ef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc0892f0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 8d 72 01 4c 8d 87 8c 00 00 00 89 d0 f0 0f b1 b7 8c 00 00 00 39 d0 89 c6 75 2b b8 01 00 00 00 f0 0f c1 81 88 05 00 00 85 c0 74 13 <f0> ff 8f 8c 00 00 00 b8 01 00 00 00 5d c3 0f 1f 44 00 00 b8 01 
Complete report #631061
Frame # Function Binary Source or offset Line
1
nfs_sb_active
nfs 0xffffffffc06b1e6e
Build id: 3.10.0-693.el7.x86_64
0x2e
2
nfs_reap_expired_delegations
nfsv4 0xffffffffc07098b2
Build id: 3.10.0-693.el7.x86_64
0xb2
3
nfs4_state_manager
nfsv4 0xffffffffc0705c22
Build id: 3.10.0-693.el7.x86_64
0x5f2
4
? nfs4_state_manager
nfsv4 0xffffffffc0705ef0
Build id: 3.10.0-693.el7.x86_64
0x8c0
5
nfs4_run_state_manager
nfsv4 0xffffffffc0705f0f
Build id: 3.10.0-693.el7.x86_64
0x1f
6
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
7
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
8
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
9
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [10.200.20.11-ma:64654]
Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_raw iptable_security sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif iTCO_wdt iTCO_vendor_support ipmi_si sg acpi_power_meter ipmi_devintf ipmi_msghandler hpwdt hpilo pcspkr wmi ioatdma pcc_cpufreq dca shpchp lpc_ich nfsd auth_rpcgss nfs_acl lockd grace sunrpc
ip_tables xfs libcrc32c ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common tg3 crc32c_intel serio_raw hpsa i2c_core ptp scsi_transport_sas pps_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 64654 Comm: 10.200.20.11-ma Not tainted 3.10.0-693.el7.x86_64 #1
Hardware name: H3C FlexServer R390, BIOS P70 02/10/2014
task: ffff8808083a4f10 ti: ffff8804e16ac000 task.ti: ffff8804e16ac000
RIP: 0010:[<ffffffffc06b1e6e>]  [<ffffffffc06b1e6e>] nfs_sb_active+0x2e/0x80 [nfs]
RSP: 0018:ffff8804e16afde8  EFLAGS: 00000246
RAX: 0000000000000004 RBX: ffff8800665781c8 RCX: ffff8804b9f99000
RDX: 0000000000000004 RSI: 0000000000000005 RDI: ffff8804b9f9c000
RBP: ffff8804e16afde8 R08: ffff8804b9f9c08c R09: 0000000000000000
R10: 000000002d4cac01 R11: ffffea0020b53200 R12: ffff8804e16afd78
R13: ffff8800665781c8 R14: 000000005de5cfe5 R15: 0000000000000202
FS:  0000000000000000(0000) GS:ffff88042f640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb9f4a49100 CR3: 00000000019f2000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff8804e16afe58 ffffffffc07098b2 ffffffffc07223c0 ffff8806c1ad60c0
ffff880428b1ab00 00000000c03963e0 00133e596140abd2 0000000400000000
000000005de5cfe5 ffff88041d3ae000 00000000ffffd8e5 ffff88041d3ae0c8
Call Trace:
[<ffffffffc07098b2>] nfs_reap_expired_delegations+0xb2/0x220 [nfsv4]
[<ffffffffc0705c22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc0705ef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc0705f0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 55 8b 97 8c 00 00 00 48 8b 8f 50 03 00 00 48 89 e5 85 d2 74 5b 8d 72 01 4c 8d 87 8c 00 00 00 89 d0 f0 0f b1 b7 8c 00 00 00 <39> d0 89 c6 75 2b b8 01 00 00 00 f0 0f c1 81 88 05 00 00 85 c0 
Complete report #478683
Frame # Function Binary Source or offset Line
1
nfs_sb_active
nfs 0xffffffffc04d1122
Build id: 3.10.0-957.el7.x86_64
0x32
2
nfs_reap_expired_delegations
nfsv4 0xffffffffc0528d52
Build id: 3.10.0-957.el7.x86_64
0xb2
3
nfs4_state_manager
nfsv4 0xffffffffc0524fad
Build id: 3.10.0-957.el7.x86_64
0x5fd
4
? nfs4_state_manager
nfsv4 0xffffffffc0525280
Build id: 3.10.0-957.el7.x86_64
0x8d0
5
nfs4_run_state_manager
nfsv4 0xffffffffc052529f
Build id: 3.10.0-957.el7.x86_64
0x1f
6
kthread
vmlinux 0xffffffffb38c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
7
? insert_kthread_work
vmlinux 0xffffffffb38c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
8
ret_from_fork_nospec_begin
vmlinux 0xffffffffb3f74c1d
Build id: 3.10.0-957.el7.x86_64
0x7
9
? insert_kthread_work
vmlinux 0xffffffffb38c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [192.21.13.24-ma:22489]
Modules linked in: vmsecmod(OE) binfmt_misc nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack libcrc32c rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc nfit libnvdimm iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw sd_mod crc_t10dif gf128mul glue_helper crct10dif_generic ablk_helper cryptd sg ppdev uas usb_storage parport_pc parport pcspkr i2c_piix4 joydev cdp(OE) ip_tables ext4 mbcache jbd2 virtio_balloon virtio_console virtio_blk virtio_net ata_generic pata_acpi crct10dif_pclmul crct10dif_common cirrus crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm virtio_pci virtio_ring virtio ata_piix drm_panel_orientation_quirks floppy libata dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vmsec_nfq]
CPU: 3 PID: 22489 Comm: 192.21.13.24-ma Kdump: loaded Tainted: G           OE  ------------   3.10.0-957.el7.x86_64 #1
Hardware name: OpenStack Foundation OpenStack Nova, BIOS 1.10.2-1ubuntu1 04/01/2014
task: ffffa06a790c4100 ti: ffffa069e8afc000 task.ti: ffffa069e8afc000
RIP: 0010:[<ffffffffc04d1122>]  [<ffffffffc04d1122>] nfs_sb_active+0x32/0x80 [nfs]
RSP: 0018:ffffa069e8affde8  EFLAGS: 00000246
RAX: 0000000000000003 RBX: ffffa069f604d000 RCX: ffffa069f604d000
RDX: 0000000000000003 RSI: 0000000000000003 RDI: ffffa06af9644000
RBP: ffffa069e8affde8 R08: ffffa06af964408c R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: ffffa06af964408c
R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffa06affd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f486478eac0 CR3: 0000000083e10000 CR4: 00000000007606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffc0528d52>] nfs_reap_expired_delegations+0xb2/0x230 [nfsv4]
[<ffffffffc0524fad>] nfs4_state_manager+0x5fd/0x8d0 [nfsv4]
[<ffffffffc0525280>] ? nfs4_state_manager+0x8d0/0x8d0 [nfsv4]
[<ffffffffc052529f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffffb38c1c31>] kthread+0xd1/0xe0
[<ffffffffb38c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb3f74c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb38c1b60>] ? insert_kthread_work+0x40/0x40
Code: 97 8c 00 00 00 48 8b 8f 50 03 00 00 48 89 e5 85 d2 74 5b 8d 72 01 4c 8d 87 8c 00 00 00 89 d0 f0 0f b1 b7 8c 00 00 00 39 d0 89 c6 <75> 2b b8 01 00 00 00 f0 0f c1 81 88 05 00 00 85 c0 74 13 f0 ff 
Complete report #312618
Frame # Function Binary Source or offset Line
1
nfs_sb_active
nfs 0xffffffffc0506e81
Build id: 3.10.0-693.el7.x86_64
0x41
2
nfs_reap_expired_delegations
nfsv4 0xffffffffc055e8b2
Build id: 3.10.0-693.el7.x86_64
0xb2
3
nfs4_state_manager
nfsv4 0xffffffffc055ac22
Build id: 3.10.0-693.el7.x86_64
0x5f2
4
? nfs4_state_manager
nfsv4 0xffffffffc055aef0
Build id: 3.10.0-693.el7.x86_64
0x8c0
5
nfs4_run_state_manager
nfsv4 0xffffffffc055af0f
Build id: 3.10.0-693.el7.x86_64
0x1f
6
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
7
? do_exit
vmlinux 0xffffffff8108ddeb
Build id: 3.10.0-693.el7.x86_64
0x6bb
8
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
9
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
10
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [10.1.15.5-manag:135171]
Modules linked in: binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ext4 mbcache jbd2 ppdev virtio_balloon joydev sg i2c_piix4 pcspkr parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic virtio_console virtio_blk virtio_scsi pata_acpi virtio_net cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci virtio_ring serio_raw i2c_core floppy virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 135171 Comm: 10.1.15.5-manag Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff88024620af70 ti: ffff880191798000 task.ti: ffff880191798000
RIP: 0010:[<ffffffffc0506e81>]  [<ffffffffc0506e81>] nfs_sb_active+0x41/0x80 [nfs]
RSP: 0018:ffff88019179bde8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff8804222dfc98 RCX: ffff880035683000
RDX: 0000000000000005 RSI: 0000000000000005 RDI: ffff880035682000
RBP: ffff88019179bde8 R08: ffff88003568208c R09: 0000000000000000
R10: ffff88042cf59c40 R11: ffffea00066af700 R12: ffff88019179bd78
R13: ffff8804222dfc98 R14: 0000000070d412af R15: ffffea00066af700
FS:  0000000000000000(0000) GS:ffff88042cf40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fefb46399e5 CR3: 00000000019f2000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff88019179be58 ffffffffc055e8b2 ffffffffc05773c0 ffff880423b42540
ffff8801a22c5980 00000000fffffff5 003e3a3d613ba9ae 0000000400000000
0000000070d412af ffff88042db39400 ffff88042db39400 ffff88042db394c8
Call Trace:
[<ffffffffc055e8b2>] nfs_reap_expired_delegations+0xb2/0x220 [nfsv4]
[<ffffffffc055ac22>] nfs4_state_manager+0x5f2/0x8c0 [nfsv4]
[<ffffffffc055aef0>] ? nfs4_state_manager+0x8c0/0x8c0 [nfsv4]
[<ffffffffc055af0f>] nfs4_run_state_manager+0x1f/0x40 [nfsv4]
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff8108ddeb>] ? do_exit+0x6bb/0xa40
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 85 d2 74 5b 8d 72 01 4c 8d 87 8c 00 00 00 89 d0 f0 0f b1 b7 8c 00 00 00 39 d0 89 c6 75 2b b8 01 00 00 00 f0 0f c1 81 88 05 00 00 <85> c0 74 13 f0 ff 8f 8c 00 00 00 b8 01 00 00 00 5d c3 0f 1f 44