Info

Permalink
Function
_raw_spin_unlock_irqrestore
First occurrence
2020-09-28
Last occurrence
2020-09-28
State
NEW
Components
kernel
Quality
-46

Statistics

Operating system Count
CentOS 7 2,758
Architecture Count
x86_64 2,758
Related packages Count
kernel 2,578
 0:3.10.0-693.21.1.el7 1,218
 0:3.10.0-1062.12.1.el7 788
 0:3.10.0-957.5.1.el7 439
 0:3.10.0-957.el7 59
 0:3.10.0-862.11.6.el7 28
 0:3.10.0-862.14.4.el7 18
 0:3.10.0-957.1.3.el7 13
 0:3.10.0-957.21.3.el7 7
 0:3.10.0-957.27.2.el7 5
 0:3.10.0-862.el7 2
 0:3.10.0-957.21.2.el7 1
kernel-plus 180
 0:3.10.0-957.el7.centos.plus 180

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #37700
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff8cb6a0f5
Build id: 3.10.0-957.el7.x86_64
0x15
2
prepare_to_wait
vmlinux 0xffffffff8c4c27f6
Build id: 3.10.0-957.el7.x86_64
0x56
3
nfs_wait_client_init_complete
nfs 0xffffffffc0938358
Build id: 3.10.0-957.el7.x86_64
0x88
4
? wake_up_atomic_t
vmlinux 0xffffffff8c4c2d00
Build id: 3.10.0-957.el7.x86_64
0x30
5
nfs_get_client
nfs 0xffffffffc0939eab
Build id: 3.10.0-957.el7.x86_64
0x22b
6
nfs4_set_client
nfsv4 0xffffffffc09a339c
Build id: 3.10.0-957.el7.x86_64
0x9c
7
nfs4_create_server
nfsv4 0xffffffffc09a4b4e
Build id: 3.10.0-957.el7.x86_64
0x13e
8
nfs4_remote_mount
nfsv4 0xffffffffc099bbae
Build id: 3.10.0-957.el7.x86_64
0x2e
9
mount_fs
vmlinux 0xffffffff8c645bde
Build id: 3.10.0-957.el7.x86_64
0x3e
10
? __alloc_percpu
vmlinux 0xffffffff8c5daff5
Build id: 3.10.0-957.el7.x86_64
0x15
11
vfs_kern_mount
vmlinux 0xffffffff8c663777
Build id: 3.10.0-957.el7.x86_64
0x67
12
nfs_do_root_mount
nfsv4 0xffffffffc099bad6
Build id: 3.10.0-957.el7.x86_64
0x86
13
nfs4_try_mount
nfsv4 0xffffffffc099bed4
Build id: 3.10.0-957.el7.x86_64
0x44
14
? get_nfs_version
nfs 0xffffffffc093a6e7
Build id: 3.10.0-957.el7.x86_64
0x27
15
nfs_fs_mount
nfs 0xffffffffc0946d3b
Build id: 3.10.0-957.el7.x86_64
0x4cb
16
? nfs_clone_super
nfs 0xffffffffc0947ca0
Build id: 3.10.0-957.el7.x86_64
0x140
17
? param_set_portnr
nfs 0xffffffffc0945b40
Build id: 3.10.0-957.el7.x86_64
0x70
18
mount_fs
vmlinux 0xffffffff8c645bde
Build id: 3.10.0-957.el7.x86_64
0x3e
19
? __alloc_percpu
vmlinux 0xffffffff8c5daff5
Build id: 3.10.0-957.el7.x86_64
0x15
20
vfs_kern_mount
vmlinux 0xffffffff8c663777
Build id: 3.10.0-957.el7.x86_64
0x67
21
do_mount
vmlinux 0xffffffff8c665d9f
Build id: 3.10.0-957.el7.x86_64
0x1ef
22
? __check_object_size
vmlinux 0xffffffff8c63e03a
Build id: 3.10.0-957.el7.x86_64
0x1ca
23
? kmem_cache_alloc_trace
vmlinux 0xffffffff8c61c4dc
Build id: 3.10.0-957.el7.x86_64
0x3c
24
SyS_mount
vmlinux 0xffffffff8c666bd3
Build id: 3.10.0-957.el7.x86_64
0x83
25
system_call_fastpath
vmlinux 0xffffffff8cb74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
26
? system_call_after_swapgs
vmlinux 0xffffffff8cb74d21
Build id: 3.10.0-957.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [mount.nfs:4067]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) dm_mirror dm_region_hash dm_log dm_mod snd_hda_codec_realtek snd_hda_codec_generic intel_powerclamp coretemp intel_rapl kvm_intel kvm irqbypass crc32_pclmul ppdev gpio_ich iTCO_wdt ghash_clmulni_intel iTCO_vendor_support snd_hda_intel snd_hda_codec
cryptd snd_hda_core snd_hwdep snd_seq snd_seq_device pcspkr snd_pcm snd_timer sg wmi mei_me mei snd parport_pc parport i2c_i801 lpc_ich soundcore auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i915 crct10dif_pclmul crct10dif_common crc32c_intel serio_raw i2c_algo_bit video iosf_mbi drm_kms_helper ahci syscopyarea sysfillrect sysimgblt fb_sys_fops libahci drm libata r8169 mii drm_panel_orientation_quirks
CPU: 3 PID: 4067 Comm: mount.nfs Tainted: G           OEL ------------   3.10.0-957.el7.x86_64 #1
Hardware name: LENOVO 3157AD8/To be filled by O.E.M., BIOS 9QKT34AUS 12/16/2011
task: ffff92a6f2f74100 ti: ffff92a56d590000 task.ti: ffff92a56d590000
RIP: 0010:[<ffffffff8cb6a0f5>]  [<ffffffff8cb6a0f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff92a56d593a70  EFLAGS: 00000246
RAX: 0000000000000246 RBX: ffff92a56d5a3728 RCX: ffff92a6f2f74100
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff92a56d593a70 R08: ffff92a6e8653ad0 R09: ffff92a6ffa9b780
R10: ffff92a56d5a3728 R11: 0000000000000001 R12: ffff92a6ffb9ab80
R13: ffff92a56d593a78 R14: ffff92a6f2f74798 R15: 000000016d61e1e8
FS:  00007f20da26a880(0000) GS:ffff92a6ffb80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe110e79000 CR3: 00000000ad5c2000 CR4: 00000000000607e0
Call Trace:
[<ffffffff8c4c27f6>] prepare_to_wait+0x56/0x90
[<ffffffffc0938358>] nfs_wait_client_init_complete+0x88/0xe0 [nfs]
[<ffffffff8c4c2d00>] ? wake_up_atomic_t+0x30/0x30
[<ffffffffc0939eab>] nfs_get_client+0x22b/0x470 [nfs]
[<ffffffffc09a339c>] nfs4_set_client+0x9c/0x150 [nfsv4]
[<ffffffffc09a4b4e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc099bbae>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffff8c645bde>] mount_fs+0x3e/0x1b0
[<ffffffff8c5daff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff8c663777>] vfs_kern_mount+0x67/0x110
[<ffffffffc099bad6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc099bed4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc093a6e7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc0946d3b>] nfs_fs_mount+0x4cb/0xdc0 [nfs]
[<ffffffffc0947ca0>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc0945b40>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffff8c645bde>] mount_fs+0x3e/0x1b0
[<ffffffff8c5daff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff8c663777>] vfs_kern_mount+0x67/0x110
[<ffffffff8c665d9f>] do_mount+0x1ef/0xce0
[<ffffffff8c63e03a>] ? __check_object_size+0x1ca/0x250
[<ffffffff8c61c4dc>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffff8c666bd3>] SyS_mount+0x83/0xd0
[<ffffffff8cb74ddb>] system_call_fastpath+0x22/0x27
[<ffffffff8cb74d21>] ? system_call_after_swapgs+0xae/0x146
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48 
Complete report #999785
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffa137c0f5
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x15
2
finish_wait
vmlinux 0xffffffffa0cc3956
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x56
3
nfs_wait_client_init_complete
nfs 0xffffffffc4597371
Build id: 3.10.0-957.el7.centos.plus.x86_64
0xa1
4
? wake_up_atomic_t
vmlinux 0xffffffffa0cc3d00
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x30
5
nfs_get_client
nfs 0xffffffffc4598eab
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x22b
6
nfs4_set_client
nfsv4 0xffffffffc0cc939c
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x9c
7
nfs4_create_server
nfsv4 0xffffffffc0ccab4e
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x13e
8
? mutex_unlock
vmlinux 0xffffffffa1377dbb
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x1b
9
nfs4_remote_mount
nfsv4 0xffffffffc0cc1bae
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x2e
10
mount_fs
vmlinux 0xffffffffa0e46bde
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x3e
11
? __alloc_percpu
vmlinux 0xffffffffa0ddbff5
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x15
12
vfs_kern_mount
vmlinux 0xffffffffa0e64777
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x67
13
nfs_do_root_mount
nfsv4 0xffffffffc0cc1ad6
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x86
14
nfs4_try_mount
nfsv4 0xffffffffc0cc1ed4
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x44
15
? get_nfs_version
nfs 0xffffffffc45996e7
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x27
16
nfs_fs_mount
nfs 0xffffffffc45a5d3b
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x4cb
17
? nfs_clone_super
nfs 0xffffffffc45a6ca0
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x140
18
? param_set_portnr
nfs 0xffffffffc45a4b40
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x70
19
mount_fs
vmlinux 0xffffffffa0e46bde
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x3e
20
? __alloc_percpu
vmlinux 0xffffffffa0ddbff5
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x15
21
vfs_kern_mount
vmlinux 0xffffffffa0e64777
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x67
22
do_mount
vmlinux 0xffffffffa0e66d9f
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x1ef
23
? __check_object_size
vmlinux 0xffffffffa0e3f03a
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x1ca
24
? kmem_cache_alloc_trace
vmlinux 0xffffffffa0e1d4dc
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x3c
25
SyS_mount
vmlinux 0xffffffffa0e67bd3
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x83
26
tracesys
vmlinux 0xffffffffa138705b
Build id: 3.10.0-957.el7.centos.plus.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [mount.nfs:7912]
Modules linked in: overlay cmac arc4 md4 rpcsec_gss_krb5 nfsv4 nls_utf8 cifs nfs lockd ccm dns_resolver grace ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter cachefiles fscache dm_mirror dm_region_hash dm_log dm_mod nvidia_drm(POE) nvidia_modeset(POE) nvidia_uvm(OE) nvidia(POE) snd_hda_codec_hdmi snd_hda_codec_generic vfat ppdev fat hp_wmi sparse_keymap rfkill intel_powerclamp coretemp kvm_intel kvm
irqbypass snd_hda_intel crc32_pclmul snd_usb_audio ghash_clmulni_intel snd_hda_codec uvcvideo hid_wacom aesni_intel drm_kms_helper videobuf2_vmalloc videobuf2_memops videobuf2_core lrw videodev gf128mul glue_helper ablk_helper e1000e(OE) cryptd snd_hda_core drm joydev snd_usbmidi_lib snd_hwdep snd_rawmidi snd_seq snd_seq_device snd_pcm serio_raw ptp pps_core snd_timer drm_panel_orientation_quirks pcspkr syscopyarea snd sysfillrect sysimgblt fb_sys_fops soundcore wmi parport_pc parport video int3400_thermal tpm_crb acpi_pad auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 ahci libahci libata crct10dif_pclmul crct10dif_common crc32c_intel nvme nvme_core
CPU: 14 PID: 7912 Comm: mount.nfs Tainted: P           OEL ------------ T 3.10.0-957.el7.centos.plus.x86_64 #1
Hardware name: HP HP Z1 Entry Tower G6/870C, BIOS S01 Ver. 02.18.00 12/14/2023
task: ffff8857d9f4c100 ti: ffff8857d861c000 task.ti: ffff8857d861c000
RIP: 0010:[<ffffffffa137c0f5>]  [<ffffffffa137c0f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8857d861fa78  EFLAGS: 00000287
RAX: ffff8857e13cfad0 RBX: ffff8857e3bced28 RCX: dead000000000200
RDX: ffff8857e6b2bad0 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff8857d861fa78 R08: ffff8857d861fad0 R09: ffff88584ec5b7c0
R10: ffff8857e3bced28 R11: 0000000000000001 R12: ffff88584ed9abc0
R13: ffff8857d861fa70 R14: ffff8857d9f4c798 R15: 00000001d861f9d8
FS:  00007f00625de880(0000) GS:ffff88584ed80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2763fffd90 CR3: 0000000f98624000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffa0cc3956>] finish_wait+0x56/0x70
[<ffffffffc4597371>] nfs_wait_client_init_complete+0xa1/0xe0 [nfs]
[<ffffffffa0cc3d00>] ? wake_up_atomic_t+0x30/0x30
[<ffffffffc4598eab>] nfs_get_client+0x22b/0x470 [nfs]
[<ffffffffc0cc939c>] nfs4_set_client+0x9c/0x150 [nfsv4]
[<ffffffffc0ccab4e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffa1377dbb>] ? mutex_unlock+0x1b/0x20
[<ffffffffc0cc1bae>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffffa0e46bde>] mount_fs+0x3e/0x1b0
[<ffffffffa0ddbff5>] ? __alloc_percpu+0x15/0x20
[<ffffffffa0e64777>] vfs_kern_mount+0x67/0x110
[<ffffffffc0cc1ad6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc0cc1ed4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc45996e7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc45a5d3b>] nfs_fs_mount+0x4cb/0xdc0 [nfs]
[<ffffffffc45a6ca0>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc45a4b40>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffffa0e46bde>] mount_fs+0x3e/0x1b0
[<ffffffffa0ddbff5>] ? __alloc_percpu+0x15/0x20
[<ffffffffa0e64777>] vfs_kern_mount+0x67/0x110
[<ffffffffa0e66d9f>] do_mount+0x1ef/0xce0
[<ffffffffa0e3f03a>] ? __check_object_size+0x1ca/0x250
[<ffffffffa0e1d4dc>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffffa0e67bd3>] SyS_mount+0x83/0xd0
[<ffffffffa138705b>] tracesys+0xa3/0xc9
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #935303
Frame # Function Binary Source or offset Line
1
_raw_spin_lock_irqsave
vmlinux 0xffffffff89d6a450
Build id: 3.10.0-957.el7.x86_64
-
2
? finish_wait
vmlinux 0xffffffff896c2938
Build id: 3.10.0-957.el7.x86_64
0x38
3
nfs_wait_client_init_complete
nfs 0xffffffffc0b7c371
Build id: 3.10.0-957.el7.x86_64
0xa1
4
? wake_up_atomic_t
vmlinux 0xffffffff896c2d00
Build id: 3.10.0-957.el7.x86_64
0x30
5
nfs_get_client
nfs 0xffffffffc0b7deab
Build id: 3.10.0-957.el7.x86_64
0x22b
6
nfs4_set_client
nfsv4 0xffffffffc0be739c
Build id: 3.10.0-957.el7.x86_64
0x9c
7
nfs4_create_server
nfsv4 0xffffffffc0be8b4e
Build id: 3.10.0-957.el7.x86_64
0x13e
8
nfs4_remote_mount
nfsv4 0xffffffffc0bdfbae
Build id: 3.10.0-957.el7.x86_64
0x2e
9
mount_fs
vmlinux 0xffffffff89845bde
Build id: 3.10.0-957.el7.x86_64
0x3e
10
? __alloc_percpu
vmlinux 0xffffffff897daff5
Build id: 3.10.0-957.el7.x86_64
0x15
11
vfs_kern_mount
vmlinux 0xffffffff89863777
Build id: 3.10.0-957.el7.x86_64
0x67
12
nfs_do_root_mount
nfsv4 0xffffffffc0bdfad6
Build id: 3.10.0-957.el7.x86_64
0x86
13
nfs4_try_mount
nfsv4 0xffffffffc0bdfed4
Build id: 3.10.0-957.el7.x86_64
0x44
14
? get_nfs_version
nfs 0xffffffffc0b7e6e7
Build id: 3.10.0-957.el7.x86_64
0x27
15
nfs_fs_mount
nfs 0xffffffffc0b8ad3b
Build id: 3.10.0-957.el7.x86_64
0x4cb
16
? nfs_clone_super
nfs 0xffffffffc0b8bca0
Build id: 3.10.0-957.el7.x86_64
0x140
17
? param_set_portnr
nfs 0xffffffffc0b89b40
Build id: 3.10.0-957.el7.x86_64
0x70
18
mount_fs
vmlinux 0xffffffff89845bde
Build id: 3.10.0-957.el7.x86_64
0x3e
19
? __alloc_percpu
vmlinux 0xffffffff897daff5
Build id: 3.10.0-957.el7.x86_64
0x15
20
vfs_kern_mount
vmlinux 0xffffffff89863777
Build id: 3.10.0-957.el7.x86_64
0x67
21
do_mount
vmlinux 0xffffffff89865d9f
Build id: 3.10.0-957.el7.x86_64
0x1ef
22
? __check_object_size
vmlinux 0xffffffff8983e03a
Build id: 3.10.0-957.el7.x86_64
0x1ca
23
? kmem_cache_alloc_trace
vmlinux 0xffffffff8981c4dc
Build id: 3.10.0-957.el7.x86_64
0x3c
24
SyS_mount
vmlinux 0xffffffff89866bd3
Build id: 3.10.0-957.el7.x86_64
0x83
25
system_call_fastpath
vmlinux 0xffffffff89d74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [mount.nfs:15993]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache cuse fuse rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) sunrpc iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif joydev pcspkr mei_me lpc_ich mei i2c_i801 sg ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter wmi acpi_pad binfmt_misc knem(OE) ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) ast crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx5_core(OE) igb drm ahci libahci mlxfw(OE) psample dca i2c_algo_bit
auxiliary(OE) devlink libata mlx_compat(OE) ptp drm_panel_orientation_quirks pps_core nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod xpmem(OE)
CPU: 22 PID: 15993 Comm: mount.nfs Tainted: G           OE  ------------   3.10.0-957.el7.x86_64 #1
Hardware name: THTF Chaoqiang R628/T1DM-E2, BIOS T1DM56.03 09/28/2019
task: ffff8d9bdb34e180 ti: ffff8d9bce744000 task.ti: ffff8d9bce744000
RIP: 0010:[<ffffffff89d6a450>]  [<ffffffff89d6a450>] _raw_spin_lock_irqsave+0x0/0x40
RSP: 0018:ffff8d9bce747a80  EFLAGS: 00000293
RAX: ffff8d9bdb34e180 RBX: ffff8d9bcea9bf28 RCX: ffff8d9bdb34e180
RDX: 0000000000000004 RSI: ffff8d9bce747ab8 RDI: ffffffffc0ba5070
RBP: ffff8d9bce747aa8 R08: ffffffffc0ba5078 R09: ffff8d9bd7b9bf00
R10: ffff8d9bcea9bf28 R11: 0000000000000005 R12: ffff8d9bcea9bf28
R13: ffff8d9bce747a70 R14: ffff8d9bddb1ab80 R15: ffff8d9bce747a70
FS:  00007f5289e28880(0000) GS:ffff8d9bddb00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f923f5c8b90 CR3: 0000002fd8722000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff896c2938>] ? finish_wait+0x38/0x70
[<ffffffffc0b7c371>] nfs_wait_client_init_complete+0xa1/0xe0 [nfs]
[<ffffffff896c2d00>] ? wake_up_atomic_t+0x30/0x30
[<ffffffffc0b7deab>] nfs_get_client+0x22b/0x470 [nfs]
[<ffffffffc0be739c>] nfs4_set_client+0x9c/0x150 [nfsv4]
[<ffffffffc0be8b4e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc0bdfbae>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffff89845bde>] mount_fs+0x3e/0x1b0
[<ffffffff897daff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff89863777>] vfs_kern_mount+0x67/0x110
[<ffffffffc0bdfad6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc0bdfed4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc0b7e6e7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc0b8ad3b>] nfs_fs_mount+0x4cb/0xdc0 [nfs]
[<ffffffffc0b8bca0>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc0b89b40>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffff89845bde>] mount_fs+0x3e/0x1b0
[<ffffffff897daff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff89863777>] vfs_kern_mount+0x67/0x110
[<ffffffff89865d9f>] do_mount+0x1ef/0xce0
[<ffffffff8983e03a>] ? __check_object_size+0x1ca/0x250
[<ffffffff8981c4dc>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffff89866bd3>] SyS_mount+0x83/0xd0
[<ffffffff89d74ddb>] system_call_fastpath+0x22/0x27
Code: 02 00 00 31 c0 ba 01 00 00 00 f0 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 fd 1a ff ff 5d c3 90 66 2e 0f 1f 84 00 00 00 00 00 <0f> 1f 44 00 00 55 48 89 e5 53 9c 58 0f 1f 44 00 00 48 89 c3 fa 
Complete report #935302
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff9096a0f5
Build id: 3.10.0-957.el7.x86_64
0x15
2
finish_wait
vmlinux 0xffffffff902c2956
Build id: 3.10.0-957.el7.x86_64
0x56
3
nfs_wait_client_init_complete
nfs 0xffffffffc094f371
Build id: 3.10.0-957.el7.x86_64
0xa1
4
? wake_up_atomic_t
vmlinux 0xffffffff902c2d00
Build id: 3.10.0-957.el7.x86_64
0x30
5
nfs_get_client
nfs 0xffffffffc0950eab
Build id: 3.10.0-957.el7.x86_64
0x22b
6
nfs4_set_client
nfsv4 0xffffffffc09ba39c
Build id: 3.10.0-957.el7.x86_64
0x9c
7
nfs4_create_server
nfsv4 0xffffffffc09bbb4e
Build id: 3.10.0-957.el7.x86_64
0x13e
8
nfs4_remote_mount
nfsv4 0xffffffffc09b2bae
Build id: 3.10.0-957.el7.x86_64
0x2e
9
mount_fs
vmlinux 0xffffffff90445bde
Build id: 3.10.0-957.el7.x86_64
0x3e
10
? __alloc_percpu
vmlinux 0xffffffff903daff5
Build id: 3.10.0-957.el7.x86_64
0x15
11
vfs_kern_mount
vmlinux 0xffffffff90463777
Build id: 3.10.0-957.el7.x86_64
0x67
12
nfs_do_root_mount
nfsv4 0xffffffffc09b2ad6
Build id: 3.10.0-957.el7.x86_64
0x86
13
nfs4_try_mount
nfsv4 0xffffffffc09b2ed4
Build id: 3.10.0-957.el7.x86_64
0x44
14
? get_nfs_version
nfs 0xffffffffc09516e7
Build id: 3.10.0-957.el7.x86_64
0x27
15
nfs_fs_mount
nfs 0xffffffffc095dd3b
Build id: 3.10.0-957.el7.x86_64
0x4cb
16
? nfs_clone_super
nfs 0xffffffffc095eca0
Build id: 3.10.0-957.el7.x86_64
0x140
17
? param_set_portnr
nfs 0xffffffffc095cb40
Build id: 3.10.0-957.el7.x86_64
0x70
18
mount_fs
vmlinux 0xffffffff90445bde
Build id: 3.10.0-957.el7.x86_64
0x3e
19
? __alloc_percpu
vmlinux 0xffffffff903daff5
Build id: 3.10.0-957.el7.x86_64
0x15
20
vfs_kern_mount
vmlinux 0xffffffff90463777
Build id: 3.10.0-957.el7.x86_64
0x67
21
do_mount
vmlinux 0xffffffff90465d9f
Build id: 3.10.0-957.el7.x86_64
0x1ef
22
? __check_object_size
vmlinux 0xffffffff9043e03a
Build id: 3.10.0-957.el7.x86_64
0x1ca
23
? kmem_cache_alloc_trace
vmlinux 0xffffffff9041c4dc
Build id: 3.10.0-957.el7.x86_64
0x3c
24
SyS_mount
vmlinux 0xffffffff90466bd3
Build id: 3.10.0-957.el7.x86_64
0x83
25
system_call_fastpath
vmlinux 0xffffffff90974ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [mount.nfs:16043]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache cuse fuse rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) sunrpc iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif pcspkr joydev i2c_i801 sg lpc_ich ipmi_si ipmi_devintf ipmi_msghandler mei_me mei wmi acpi_pad acpi_power_meter binfmt_misc knem(OE) ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) ast drm_kms_helper syscopyarea sysfillrect crct10dif_pclmul crct10dif_common sysimgblt crc32c_intel fb_sys_fops ttm drm drm_panel_orientation_quirks mlx5_core(OE) igb ahci libahci
mlxfw(OE) psample dca auxiliary(OE) i2c_algo_bit devlink libata mlx_compat(OE) ptp pps_core nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod xpmem(OE)
CPU: 16 PID: 16043 Comm: mount.nfs Tainted: G           OE  ------------   3.10.0-957.el7.x86_64 #1
Hardware name: THTF Chaoqiang R628/T1DM-E2, BIOS T1DM56.03 09/28/2019
task: ffff8e04d588d140 ti: ffff8e1cc82ec000 task.ti: ffff8e1cc82ec000
RIP: 0010:[<ffffffff9096a0f5>]  [<ffffffff9096a0f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8e1cc82efa78  EFLAGS: 00000293
RAX: ffffffffc0978078 RBX: ffff8e1cd3e26b28 RCX: dead000000000200
RDX: ffffffffc0978078 RSI: 0000000000000293 RDI: 0000000000000293
RBP: ffff8e1cc82efa78 R08: ffff8e1cc82efad0 R09: ffff8e1cd2afa300
R10: ffff8e1cd3e26b28 R11: 0000000000000005 R12: ffff8e04dfa1ab80
R13: ffff8e1cc82efa70 R14: ffff8e04d588d7d8 R15: 00000001c82ef9d8
FS:  00007fd947dcb880(0000) GS:ffff8e04dfa00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005564600ee362 CR3: 0000002fd038e000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff902c2956>] finish_wait+0x56/0x70
[<ffffffffc094f371>] nfs_wait_client_init_complete+0xa1/0xe0 [nfs]
[<ffffffff902c2d00>] ? wake_up_atomic_t+0x30/0x30
[<ffffffffc0950eab>] nfs_get_client+0x22b/0x470 [nfs]
[<ffffffffc09ba39c>] nfs4_set_client+0x9c/0x150 [nfsv4]
[<ffffffffc09bbb4e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc09b2bae>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffff90445bde>] mount_fs+0x3e/0x1b0
[<ffffffff903daff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff90463777>] vfs_kern_mount+0x67/0x110
[<ffffffffc09b2ad6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc09b2ed4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc09516e7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc095dd3b>] nfs_fs_mount+0x4cb/0xdc0 [nfs]
[<ffffffffc095eca0>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc095cb40>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffff90445bde>] mount_fs+0x3e/0x1b0
[<ffffffff903daff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff90463777>] vfs_kern_mount+0x67/0x110
[<ffffffff90465d9f>] do_mount+0x1ef/0xce0
[<ffffffff9043e03a>] ? __check_object_size+0x1ca/0x250
[<ffffffff9041c4dc>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffff90466bd3>] SyS_mount+0x83/0xd0
[<ffffffff90974ddb>] system_call_fastpath+0x22/0x27
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #935300
Frame # Function Binary Source or offset Line
1
prepare_to_wait
vmlinux 0xffffffff818c27fb
Build id: 3.10.0-957.el7.x86_64
0x5b
2
nfs_wait_client_init_complete
nfs 0xffffffffc0c2f358
Build id: 3.10.0-957.el7.x86_64
0x88
3
? wake_up_atomic_t
vmlinux 0xffffffff818c2d00
Build id: 3.10.0-957.el7.x86_64
0x30
4
nfs_get_client
nfs 0xffffffffc0c30eab
Build id: 3.10.0-957.el7.x86_64
0x22b
5
nfs4_set_client
nfsv4 0xffffffffc0cb239c
Build id: 3.10.0-957.el7.x86_64
0x9c
6
nfs4_create_server
nfsv4 0xffffffffc0cb3b4e
Build id: 3.10.0-957.el7.x86_64
0x13e
7
nfs4_remote_mount
nfsv4 0xffffffffc0caabae
Build id: 3.10.0-957.el7.x86_64
0x2e
8
mount_fs
vmlinux 0xffffffff81a45bde
Build id: 3.10.0-957.el7.x86_64
0x3e
9
? __alloc_percpu
vmlinux 0xffffffff819daff5
Build id: 3.10.0-957.el7.x86_64
0x15
10
vfs_kern_mount
vmlinux 0xffffffff81a63777
Build id: 3.10.0-957.el7.x86_64
0x67
11
nfs_do_root_mount
nfsv4 0xffffffffc0caaad6
Build id: 3.10.0-957.el7.x86_64
0x86
12
nfs4_try_mount
nfsv4 0xffffffffc0caaed4
Build id: 3.10.0-957.el7.x86_64
0x44
13
? get_nfs_version
nfs 0xffffffffc0c316e7
Build id: 3.10.0-957.el7.x86_64
0x27
14
nfs_fs_mount
nfs 0xffffffffc0c3dd3b
Build id: 3.10.0-957.el7.x86_64
0x4cb
15
? nfs_clone_super
nfs 0xffffffffc0c3eca0
Build id: 3.10.0-957.el7.x86_64
0x140
16
? param_set_portnr
nfs 0xffffffffc0c3cb40
Build id: 3.10.0-957.el7.x86_64
0x70
17
mount_fs
vmlinux 0xffffffff81a45bde
Build id: 3.10.0-957.el7.x86_64
0x3e
18
? __alloc_percpu
vmlinux 0xffffffff819daff5
Build id: 3.10.0-957.el7.x86_64
0x15
19
vfs_kern_mount
vmlinux 0xffffffff81a63777
Build id: 3.10.0-957.el7.x86_64
0x67
20
do_mount
vmlinux 0xffffffff81a65d9f
Build id: 3.10.0-957.el7.x86_64
0x1ef
21
? __check_object_size
vmlinux 0xffffffff81a3e03a
Build id: 3.10.0-957.el7.x86_64
0x1ca
22
? kmem_cache_alloc_trace
vmlinux 0xffffffff81a1c4dc
Build id: 3.10.0-957.el7.x86_64
0x3c
23
SyS_mount
vmlinux 0xffffffff81a66bd3
Build id: 3.10.0-957.el7.x86_64
0x83
24
system_call_fastpath
vmlinux 0xffffffff81f74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [mount.nfs:16052]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 cuse fuse dns_resolver nfs lockd grace fscache rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) sunrpc iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif pcspkr joydev sg mei_me lpc_ich i2c_i801 mei wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad binfmt_misc knem(OE) ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) ast crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mlx5_core(OE) igb ahci mlxfw(OE) psample auxiliary(OE) devlink
dca i2c_algo_bit libahci mlx_compat(OE) drm_panel_orientation_quirks ptp pps_core libata nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod xpmem(OE)
CPU: 26 PID: 16052 Comm: mount.nfs Tainted: G           OE  ------------   3.10.0-957.el7.x86_64 #1
Hardware name: THTF T1DM/T1DM-E2, BIOS T1DM56.03 09/28/2019
task: ffff977ac92ab0c0 ti: ffff977ac62d4000 task.ti: ffff977ac62d4000
RIP: 0010:[<ffffffff818c27fb>]  [<ffffffff818c27fb>] prepare_to_wait+0x5b/0x90
RSP: 0018:ffff977ac62d7a98  EFLAGS: 00000286
RAX: 0000000000000246 RBX: ffffffffc0c58078 RCX: ffff977ac92ab0c0
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff977ac62d7aa8 R08: ffffffffc0c58078 R09: ffff977ace72a400
R10: ffff9792ca77af28 R11: 0000000000000005 R12: ffff9792ca77af28
R13: 0000000000000005 R14: ffff9792ca77af28 R15: ffff977ac62d7a70
FS:  00007f012dfa6880(0000) GS:ffff9792ddc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055f8bd693002 CR3: 00000017bf626000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffc0c2f358>] nfs_wait_client_init_complete+0x88/0xe0 [nfs]
[<ffffffff818c2d00>] ? wake_up_atomic_t+0x30/0x30
[<ffffffffc0c30eab>] nfs_get_client+0x22b/0x470 [nfs]
[<ffffffffc0cb239c>] nfs4_set_client+0x9c/0x150 [nfsv4]
[<ffffffffc0cb3b4e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc0caabae>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffff81a45bde>] mount_fs+0x3e/0x1b0
[<ffffffff819daff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff81a63777>] vfs_kern_mount+0x67/0x110
[<ffffffffc0caaad6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc0caaed4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc0c316e7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc0c3dd3b>] nfs_fs_mount+0x4cb/0xdc0 [nfs]
[<ffffffffc0c3eca0>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc0c3cb40>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffff81a45bde>] mount_fs+0x3e/0x1b0
[<ffffffff819daff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff81a63777>] vfs_kern_mount+0x67/0x110
[<ffffffff81a65d9f>] do_mount+0x1ef/0xce0
[<ffffffff81a3e03a>] ? __check_object_size+0x1ca/0x250
[<ffffffff81a1c4dc>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffff81a66bd3>] SyS_mount+0x83/0xd0
[<ffffffff81f74ddb>] system_call_fastpath+0x22/0x27
Code: 89 6d e0 48 8b 55 e0 65 48 8b 0c 25 80 0e 01 00 48 87 11 4c 89 e7 48 89 c6 48 89 55 e0 48 8b 55 e0 e8 ea 78 6a 00 48 83 c4 10 5b <41> 5c 41 5d 5d c3 0f 1f 80 00 00 00 00 49 8b 54 24 08 49 8d 74 
Complete report #907861
Frame # Function Binary Source or offset Line
1
nfs_get_client
nfs 0xffffffffc0cabe99
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x219
2
nfs4_set_client
nfsv4 0xffffffffc0d8a39c
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x9c
3
nfs4_create_server
nfsv4 0xffffffffc0d8bb4e
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x13e
4
nfs4_remote_mount
nfsv4 0xffffffffc0d82bae
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x2e
5
mount_fs
vmlinux 0xffffffff87e46bde
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x3e
6
? __alloc_percpu
vmlinux 0xffffffff87ddbff5
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x15
7
vfs_kern_mount
vmlinux 0xffffffff87e64777
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x67
8
nfs_do_root_mount
nfsv4 0xffffffffc0d82ad6
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x86
9
nfs4_try_mount
nfsv4 0xffffffffc0d82ed4
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x44
10
? get_nfs_version
nfs 0xffffffffc0cac6e7
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x27
11
nfs_fs_mount
nfs 0xffffffffc0cb8d3b
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x4cb
12
? nfs_clone_super
nfs 0xffffffffc0cb9ca0
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x140
13
? param_set_portnr
nfs 0xffffffffc0cb7b40
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x70
14
mount_fs
vmlinux 0xffffffff87e46bde
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x3e
15
? __alloc_percpu
vmlinux 0xffffffff87ddbff5
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x15
16
vfs_kern_mount
vmlinux 0xffffffff87e64777
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x67
17
do_mount
vmlinux 0xffffffff87e66d9f
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x1ef
18
? __check_object_size
vmlinux 0xffffffff87e3f03a
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x1ca
19
? kmem_cache_alloc_trace
vmlinux 0xffffffff87e1d4dc
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x3c
20
SyS_mount
vmlinux 0xffffffff87e67bd3
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x83
21
tracesys
vmlinux 0xffffffff8838705b
Build id: 3.10.0-957.el7.centos.plus.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mount.nfs:3586]
Modules linked in: nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) tcp_lp binfmt_misc fuse uinput xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink cmac nfsv3 nfs_acl md4 nls_utf8 cifs ccm rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep cachefiles fscache dm_mirror dm_region_hash dm_log dm_mod hp_wmi sparse_keymap ppdev vfat
fat arc4 mei_wdt iTCO_wdt iTCO_vendor_support intel_pmc_core intel_powerclamp coretemp intel_rapl iwlmvm kvm mac80211 snd_hda_codec_hdmi irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul snd_hda_codec_realtek glue_helper ablk_helper cryptd snd_hda_codec_generic serio_raw iwlwifi cfg80211 uvcvideo pcspkr videobuf2_vmalloc videobuf2_memops videobuf2_core videodev joydev snd_usb_audio btusb btrtl btbcm btintel snd_usbmidi_lib bluetooth snd_rawmidi sg rfkill wmi i915 snd_hda_intel parport_pc snd_hda_codec parport e1000e(OE) snd_hda_core iosf_mbi video i2c_algo_bit snd_hwdep snd_seq drm_kms_helper tpm_infineon snd_seq_device acpi_pad drm snd_pcm mei_me snd_timer snd drm_panel_orientation_quirks syscopyarea sysfillrect mei sysimgblt fb_sys_fops soundcore ptp i2c_i801 pps_core
auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ahci libahci libata
CPU: 1 PID: 3586 Comm: mount.nfs Kdump: loaded Tainted: G        W  OE  ------------ T 3.10.0-957.el7.centos.plus.x86_64 #1
Hardware name: HP HP EliteDesk 800 G2 DM 35W/8055, BIOS N21 Ver. 02.53 04/16/2021
task: ffff8f7558f630c0 ti: ffff8f74d2610000 task.ti: ffff8f74d2610000
RIP: 0010:[<ffffffffc0cabe99>]  [<ffffffffc0cabe99>] nfs_get_client+0x219/0x470 [nfs]
RSP: 0000:ffff8f74d2613b08  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffffff87cc3956 RCX: 0000000000000002
RDX: ffff8f73ee822b60 RSI: 0000000000000000 RDI: ffff8f7535eb4840
RBP: ffff8f74d2613b60 R08: ffff8f74d2613ad0 R09: ffff8f73ee822b00
R10: ffff8f73a508b128 R11: 0000000000000001 R12: 0000000000000293
R13: 0000000000000293 R14: ffffffffc0cd3070 R15: ffff8f74d2613ad0
FS:  00007f1ab95e8880(0000) GS:ffff8f756f440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002486db0 CR3: 0000000192b3e000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc0d8a39c>] nfs4_set_client+0x9c/0x150 [nfsv4]
[<ffffffffc0d8bb4e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc0d82bae>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffff87e46bde>] mount_fs+0x3e/0x1b0
[<ffffffff87ddbff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff87e64777>] vfs_kern_mount+0x67/0x110
[<ffffffffc0d82ad6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc0d82ed4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc0cac6e7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc0cb8d3b>] nfs_fs_mount+0x4cb/0xdc0 [nfs]
[<ffffffffc0cb9ca0>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc0cb7b40>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffff87e46bde>] mount_fs+0x3e/0x1b0
[<ffffffff87ddbff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff87e64777>] vfs_kern_mount+0x67/0x110
[<ffffffff87e66d9f>] do_mount+0x1ef/0xce0
[<ffffffff87e3f03a>] ? __check_object_size+0x1ca/0x250
[<ffffffff87e1d4dc>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffff87e67bd3>] SyS_mount+0x83/0xd0
[<ffffffff8838705b>] tracesys+0xa3/0xc9
Code: 48 c7 c7 60 ee cc c0 31 c0 e8 8b 17 6c c7 48 89 d8 e9 4f 01 00 00 0f 1f 40 00 4c 89 4d b8 48 89 55 c0 4c 89 55 c8 f0 41 ff 04 24 <4c> 89 ef c6 07 00 0f 1f 40 00 4c 89 e7 e8 25 e4 ff ff 4c 89 e7 
Complete report #852583
Frame # Function Binary Source or offset Line
1
finish_wait
vmlinux 0xffffffff89cc3916
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x16
2
nfs_wait_client_init_complete
nfs 0xffffffffc0667371
Build id: 3.10.0-957.el7.centos.plus.x86_64
0xa1
3
? wake_up_atomic_t
vmlinux 0xffffffff89cc3d00
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x30
4
nfs_get_client
nfs 0xffffffffc0668eab
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x22b
5
nfs4_set_client
nfsv4 0xffffffffc06ea39c
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x9c
6
nfs4_create_server
nfsv4 0xffffffffc06ebb4e
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x13e
7
nfs4_remote_mount
nfsv4 0xffffffffc06e2bae
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x2e
8
mount_fs
vmlinux 0xffffffff89e46bde
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x3e
9
? __alloc_percpu
vmlinux 0xffffffff89ddbff5
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x15
10
vfs_kern_mount
vmlinux 0xffffffff89e64777
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x67
11
nfs_do_root_mount
nfsv4 0xffffffffc06e2ad6
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x86
12
nfs4_try_mount
nfsv4 0xffffffffc06e2ed4
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x44
13
? get_nfs_version
nfs 0xffffffffc06696e7
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x27
14
nfs_fs_mount
nfs 0xffffffffc0675d3b
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x4cb
15
? nfs_clone_super
nfs 0xffffffffc0676ca0
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x140
16
? param_set_portnr
nfs 0xffffffffc0674b40
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x70
17
mount_fs
vmlinux 0xffffffff89e46bde
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x3e
18
? __alloc_percpu
vmlinux 0xffffffff89ddbff5
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x15
19
vfs_kern_mount
vmlinux 0xffffffff89e64777
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x67
20
do_mount
vmlinux 0xffffffff89e66d9f
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x1ef
21
? __check_object_size
vmlinux 0xffffffff89e3f03a
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x1ca
22
? kmem_cache_alloc_trace
vmlinux 0xffffffff89e1d4dc
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x3c
23
SyS_mount
vmlinux 0xffffffff89e67bd3
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x83
24
system_call_fastpath
vmlinux 0xffffffff8a386ddb
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [mount.nfs:6424]
Modules linked in: joydev uinput xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nfsv3 cmac nfs_acl arc4 md4 nls_utf8 cifs rpcsec_gss_krb5 nfsv4 ccm dns_resolver nfs lockd grace ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter cachefiles fscache dm_mirror dm_region_hash dm_log dm_mod vfat fat iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm
irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr i2c_i801 hpwdt hpilo ipmi_si ipmi_devintf ipmi_msghandler pcc_cpufreq lpc_ich ioatdma acpi_power_meter wmi dca auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper ahci crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt libahci fb_sys_fops ttm libata drm tg3 ptp pps_core drm_panel_orientation_quirks
CPU: 10 PID: 6424 Comm: mount.nfs Kdump: loaded Tainted: G        W      ------------   3.10.0-957.el7.centos.plus.x86_64 #1
Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 04/29/2021
task: ffff8a7f7e7ea080 ti: ffff8a7c5ba60000 task.ti: ffff8a7c5ba60000
RIP: 0010:[<ffffffff89cc3916>]  [<ffffffff89cc3916>] finish_wait+0x16/0x70
RSP: 0000:ffff8a7c5ba63a98  EFLAGS: 00000202
RAX: ffff8a7f7e7ea080 RBX: ffff8a75f9677ad0 RCX: ffff8a7f7e7ea080
RDX: 0000000000000004 RSI: ffff8a7c5ba63ab8 RDI: ffffffffc0690070
RBP: ffff8a7c5ba63aa8 R08: ffff8a75f9677ad0 R09: ffff8a8447373e00
R10: ffff8a7f58f33528 R11: 0000000000000001 R12: ffff8a7f58f33528
R13: 0000000000000001 R14: ffff8a7f58f33528 R15: ffff8a7c5ba63a70
FS:  00007f7fe37de880(0000) GS:ffff8a7c5f680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5f1d3b2a7c CR3: 00000002a0584000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc0667371>] nfs_wait_client_init_complete+0xa1/0xe0 [nfs]
[<ffffffff89cc3d00>] ? wake_up_atomic_t+0x30/0x30
[<ffffffffc0668eab>] nfs_get_client+0x22b/0x470 [nfs]
[<ffffffffc06ea39c>] nfs4_set_client+0x9c/0x150 [nfsv4]
[<ffffffffc06ebb4e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc06e2bae>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffff89e46bde>] mount_fs+0x3e/0x1b0
[<ffffffff89ddbff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff89e64777>] vfs_kern_mount+0x67/0x110
[<ffffffffc06e2ad6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc06e2ed4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc06696e7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc0675d3b>] nfs_fs_mount+0x4cb/0xdc0 [nfs]
[<ffffffffc0676ca0>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc0674b40>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffff89e46bde>] mount_fs+0x3e/0x1b0
[<ffffffff89ddbff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff89e64777>] vfs_kern_mount+0x67/0x110
[<ffffffff89e66d9f>] do_mount+0x1ef/0xce0
[<ffffffff89e3f03a>] ? __check_object_size+0x1ca/0x250
[<ffffffff89e1d4dc>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffff89e67bd3>] SyS_mount+0x83/0xd0
[<ffffffff8a386ddb>] system_call_fastpath+0x22/0x27
Code: df e8 ef 87 6b 00 5b 41 5c 41 5d 5d c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 65 48 8b 04 25 80 0e 01 00 48 89 e5 41 56 41 55 <49> 89 fd 41 54 4c 8d 66 18 53 48 c7 00 00 00 00 00 48 89 f3 4c 
Complete report #786327
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff9b97c0f5
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x15
2
finish_wait
vmlinux 0xffffffff9b2c3956
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x56
3
nfs_wait_client_init_complete
nfs 0xffffffffc0a0e371
Build id: 3.10.0-957.el7.centos.plus.x86_64
0xa1
4
? wake_up_atomic_t
vmlinux 0xffffffff9b2c3d00
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x30
5
nfs_get_client
nfs 0xffffffffc0a0feab
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x22b
6
nfs4_set_client
nfsv4 0xffffffffc0ab139c
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x9c
7
nfs4_create_server
nfsv4 0xffffffffc0ab2b4e
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x13e
8
nfs4_remote_mount
nfsv4 0xffffffffc0aa9bae
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x2e
9
mount_fs
vmlinux 0xffffffff9b446bde
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x3e
10
? __alloc_percpu
vmlinux 0xffffffff9b3dbff5
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x15
11
vfs_kern_mount
vmlinux 0xffffffff9b464777
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x67
12
nfs_do_root_mount
nfsv4 0xffffffffc0aa9ad6
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x86
13
nfs4_try_mount
nfsv4 0xffffffffc0aa9ed4
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x44
14
? get_nfs_version
nfs 0xffffffffc0a106e7
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x27
15
nfs_fs_mount
nfs 0xffffffffc0a1cd3b
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x4cb
16
? nfs_clone_super
nfs 0xffffffffc0a1dca0
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x140
17
? param_set_portnr
nfs 0xffffffffc0a1bb40
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x70
18
mount_fs
vmlinux 0xffffffff9b446bde
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x3e
19
? __alloc_percpu
vmlinux 0xffffffff9b3dbff5
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x15
20
vfs_kern_mount
vmlinux 0xffffffff9b464777
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x67
21
do_mount
vmlinux 0xffffffff9b466d9f
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x1ef
22
? __check_object_size
vmlinux 0xffffffff9b43f03a
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x1ca
23
? kmem_cache_alloc_trace
vmlinux 0xffffffff9b41d4dc
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x3c
24
SyS_mount
vmlinux 0xffffffff9b467bd3
Build id: 3.10.0-957.el7.centos.plus.x86_64
0x83
25
tracesys
vmlinux 0xffffffff9b98705b
Build id: 3.10.0-957.el7.centos.plus.x86_64
0xa3
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [mount.nfs:61751]
Modules linked in: joydev uinput xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nfsv3 nfs_acl cmac arc4 md4 nls_utf8 cifs ccm rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod vfat fat sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul iTCO_wdt iTCO_vendor_support
ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg hpilo hpwdt lpc_ich i2c_i801 ioatdma dca ipmi_si ipmi_devintf ipmi_msghandler pcc_cpufreq acpi_power_meter wmi auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 ahci i2c_algo_bit drm_kms_helper crct10dif_pclmul libahci crct10dif_common crc32c_intel syscopyarea libata sysfillrect sysimgblt fb_sys_fops tg3 ttm drm ptp pps_core drm_panel_orientation_quirks
CPU: 11 PID: 61751 Comm: mount.nfs Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-957.el7.centos.plus.x86_64 #1
Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 04/29/2021
task: ffff97b3a66a0000 ti: ffff97b741bd0000 task.ti: ffff97b741bd0000
RIP: 0010:[<ffffffff9b97c0f5>]  [<ffffffff9b97c0f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff97b741bd3a78  EFLAGS: 00000287
RAX: ffff97be9b79fad0 RBX: ffff97b715b8e928 RCX: dead000000000200
RDX: ffff97b78d2b3ad0 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff97b741bd3a78 R08: ffff97b741bd3ad0 R09: ffff97bf9f01b7c0
R10: ffff97b715b8e928 R11: 0000000000000001 R12: ffff97b79f6dabc0
R13: ffff97b741bd3a78 R14: ffff97b3a66a0698 R15: 0000000141bd39d8
FS:  00007f0ae0380880(0000) GS:ffff97b79f6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056455259128c CR3: 0000000841abe000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9b2c3956>] finish_wait+0x56/0x70
[<ffffffffc0a0e371>] nfs_wait_client_init_complete+0xa1/0xe0 [nfs]
[<ffffffff9b2c3d00>] ? wake_up_atomic_t+0x30/0x30
[<ffffffffc0a0feab>] nfs_get_client+0x22b/0x470 [nfs]
[<ffffffffc0ab139c>] nfs4_set_client+0x9c/0x150 [nfsv4]
[<ffffffffc0ab2b4e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc0aa9bae>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffff9b446bde>] mount_fs+0x3e/0x1b0
[<ffffffff9b3dbff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff9b464777>] vfs_kern_mount+0x67/0x110
[<ffffffffc0aa9ad6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc0aa9ed4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc0a106e7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc0a1cd3b>] nfs_fs_mount+0x4cb/0xdc0 [nfs]
[<ffffffffc0a1dca0>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc0a1bb40>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffff9b446bde>] mount_fs+0x3e/0x1b0
[<ffffffff9b3dbff5>] ? __alloc_percpu+0x15/0x20
[<ffffffff9b464777>] vfs_kern_mount+0x67/0x110
[<ffffffff9b466d9f>] do_mount+0x1ef/0xce0
[<ffffffff9b43f03a>] ? __check_object_size+0x1ca/0x250
[<ffffffff9b41d4dc>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffff9b467bd3>] SyS_mount+0x83/0xd0
[<ffffffff9b98705b>] tracesys+0xa3/0xc9
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #334379
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffb536a175
Build id: 3.10.0-957.1.3.el7.x86_64
0x15
2
prepare_to_wait
vmlinux 0xffffffffb4cc27f6
Build id: 3.10.0-957.1.3.el7.x86_64
0x56
3
nfs_wait_client_init_complete
nfs 0xffffffffc1a46358
Build id: 3.10.0-957.1.3.el7.x86_64
0x88
4
? wake_up_atomic_t
vmlinux 0xffffffffb4cc2d00
Build id: 3.10.0-957.1.3.el7.x86_64
0x30
5
nfs_get_client
nfs 0xffffffffc1a47eab
Build id: 3.10.0-957.1.3.el7.x86_64
0x22b
6
nfs4_set_client
nfsv4 0xffffffffc1ab139c
Build id: 3.10.0-957.1.3.el7.x86_64
0x9c
7
nfs4_create_server
nfsv4 0xffffffffc1ab2b4e
Build id: 3.10.0-957.1.3.el7.x86_64
0x13e
8
nfs4_remote_mount
nfsv4 0xffffffffc1aa9bae
Build id: 3.10.0-957.1.3.el7.x86_64
0x2e
9
mount_fs
vmlinux 0xffffffffb4e45e4e
Build id: 3.10.0-957.1.3.el7.x86_64
0x3e
10
? __alloc_percpu
vmlinux 0xffffffffb4ddb265
Build id: 3.10.0-957.1.3.el7.x86_64
0x15
11
vfs_kern_mount
vmlinux 0xffffffffb4e639e7
Build id: 3.10.0-957.1.3.el7.x86_64
0x67
12
nfs_do_root_mount
nfsv4 0xffffffffc1aa9ad6
Build id: 3.10.0-957.1.3.el7.x86_64
0x86
13
nfs4_try_mount
nfsv4 0xffffffffc1aa9ed4
Build id: 3.10.0-957.1.3.el7.x86_64
0x44
14
? get_nfs_version
nfs 0xffffffffc1a486e7
Build id: 3.10.0-957.1.3.el7.x86_64
0x27
15
nfs_fs_mount
nfs 0xffffffffc1a54d3b
Build id: 3.10.0-957.1.3.el7.x86_64
0x4cb
16
? nfs_clone_super
nfs 0xffffffffc1a55ca0
Build id: 3.10.0-957.1.3.el7.x86_64
0x140
17
? param_set_portnr
nfs 0xffffffffc1a53b40
Build id: 3.10.0-957.1.3.el7.x86_64
0x70
18
mount_fs
vmlinux 0xffffffffb4e45e4e
Build id: 3.10.0-957.1.3.el7.x86_64
0x3e
19
? __alloc_percpu
vmlinux 0xffffffffb4ddb265
Build id: 3.10.0-957.1.3.el7.x86_64
0x15
20
vfs_kern_mount
vmlinux 0xffffffffb4e639e7
Build id: 3.10.0-957.1.3.el7.x86_64
0x67
21
do_mount
vmlinux 0xffffffffb4e6600f
Build id: 3.10.0-957.1.3.el7.x86_64
0x1ef
22
? __check_object_size
vmlinux 0xffffffffb4e3e2aa
Build id: 3.10.0-957.1.3.el7.x86_64
0x1ca
23
? kmem_cache_alloc_trace
vmlinux 0xffffffffb4e1c74c
Build id: 3.10.0-957.1.3.el7.x86_64
0x3c
24
SyS_mount
vmlinux 0xffffffffb4e66e43
Build id: 3.10.0-957.1.3.el7.x86_64
0x83
25
system_call_fastpath
vmlinux 0xffffffffb5374ddb
Build id: 3.10.0-957.1.3.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [mount.nfs:14049]
Modules linked in: nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM overlay(T) ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace ipt_REJECT nf_reject_ipv4 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6_tables iptable_filter vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) cachefiles fscache vboxdrv(OE) iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm snd_hda_codec_hdmi irqbypass crc32_pclmul ghash_clmulni_intel cryptd snd_hda_codec_realtek snd_usb_audio snd_hda_codec_generic uvcvideo videobuf2_vmalloc videobuf2_memops
snd_hda_intel videobuf2_core joydev videodev snd_usbmidi_lib snd_hda_codec snd_rawmidi snd_hda_core intel_wmi_thunderbolt snd_hwdep snd_seq pcspkr snd_seq_device snd_pcm sg snd_timer i2c_i801 snd lpc_ich ioatdma soundcore acpi_power_meter binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ahci drm crc32c_intel igb libahci libata ptp pps_core dca drm_panel_orientation_quirks i2c_algo_bit ipmi_devintf ipmi_msghandler wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 14049 Comm: mount.nfs Tainted: P          IOEL ------------ T 3.10.0-957.1.3.el7.x86_64 #1
Hardware name: Supermicro SYS-7038A-I/X10DAI, BIOS 1.0c 01/29/2015
task: ffff972e90265140 ti: ffff972694358000 task.ti: ffff972694358000
RIP: 0010:[<ffffffffb536a175>]  [<ffffffffb536a175>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff97269435ba70  EFLAGS: 00000246
RAX: 0000000000000246 RBX: ffff972e990d4728 RCX: ffff972e90265140
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff97269435ba70 R08: ffffffffc1a6f078 R09: ffff97269b774b00
R10: ffff972e990d4728 R11: 0000000000000005 R12: ffff97269f89ab80
R13: ffff97269435ba70 R14: ffff972e902657d8 R15: 00000001b4ce53df
FS:  00007fc0c452e880(0000) GS:ffff97269f880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fee114a87c0 CR3: 0000000822328000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb4cc27f6>] prepare_to_wait+0x56/0x90
[<ffffffffc1a46358>] nfs_wait_client_init_complete+0x88/0xe0 [nfs]
[<ffffffffb4cc2d00>] ? wake_up_atomic_t+0x30/0x30
[<ffffffffc1a47eab>] nfs_get_client+0x22b/0x470 [nfs]
[<ffffffffc1ab139c>] nfs4_set_client+0x9c/0x150 [nfsv4]
[<ffffffffc1ab2b4e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc1aa9bae>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffffb4e45e4e>] mount_fs+0x3e/0x1b0
[<ffffffffb4ddb265>] ? __alloc_percpu+0x15/0x20
[<ffffffffb4e639e7>] vfs_kern_mount+0x67/0x110
[<ffffffffc1aa9ad6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc1aa9ed4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc1a486e7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc1a54d3b>] nfs_fs_mount+0x4cb/0xdc0 [nfs]
[<ffffffffc1a55ca0>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc1a53b40>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffffb4e45e4e>] mount_fs+0x3e/0x1b0
[<ffffffffb4ddb265>] ? __alloc_percpu+0x15/0x20
[<ffffffffb4e639e7>] vfs_kern_mount+0x67/0x110
[<ffffffffb4e6600f>] do_mount+0x1ef/0xce0
[<ffffffffb4e3e2aa>] ? __check_object_size+0x1ca/0x250
[<ffffffffb4e1c74c>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffffb4e66e43>] SyS_mount+0x83/0xd0
[<ffffffffb5374ddb>] system_call_fastpath+0x22/0x27
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #263362
Frame # Function Binary Source or offset Line
1
nfs_get_client
nfs 0xffffffffc1acce80
Build id: 3.10.0-862.11.6.el7.x86_64
0x210
2
nfs4_set_client
nfsv4 0xffffffffc1b53518
Build id: 3.10.0-862.11.6.el7.x86_64
0x98
3
nfs4_create_server
nfsv4 0xffffffffc1b54cbe
Build id: 3.10.0-862.11.6.el7.x86_64
0x13e
4
nfs4_remote_mount
nfsv4 0xffffffffc1b4bd9e
Build id: 3.10.0-862.11.6.el7.x86_64
0x2e
5
mount_fs
vmlinux 0xffffffff9c223c2e
Build id: 3.10.0-862.11.6.el7.x86_64
0x3e
6
? __alloc_percpu
vmlinux 0xffffffff9c1bb1e5
Build id: 3.10.0-862.11.6.el7.x86_64
0x15
7
vfs_kern_mount
vmlinux 0xffffffff9c241157
Build id: 3.10.0-862.11.6.el7.x86_64
0x67
8
nfs_do_root_mount
nfsv4 0xffffffffc1b4bcc6
Build id: 3.10.0-862.11.6.el7.x86_64
0x86
9
nfs4_try_mount
nfsv4 0xffffffffc1b4c0c4
Build id: 3.10.0-862.11.6.el7.x86_64
0x44
10
? get_nfs_version
nfs 0xffffffffc1acd6d7
Build id: 3.10.0-862.11.6.el7.x86_64
0x27
11
nfs_fs_mount
nfs 0xffffffffc1ad9cdb
Build id: 3.10.0-862.11.6.el7.x86_64
0x4cb
12
? nfs_clone_super
nfs 0xffffffffc1adac20
Build id: 3.10.0-862.11.6.el7.x86_64
0x140
13
? param_set_portnr
nfs 0xffffffffc1ad8ae0
Build id: 3.10.0-862.11.6.el7.x86_64
0x70
14
mount_fs
vmlinux 0xffffffff9c223c2e
Build id: 3.10.0-862.11.6.el7.x86_64
0x3e
15
? __alloc_percpu
vmlinux 0xffffffff9c1bb1e5
Build id: 3.10.0-862.11.6.el7.x86_64
0x15
16
vfs_kern_mount
vmlinux 0xffffffff9c241157
Build id: 3.10.0-862.11.6.el7.x86_64
0x67
17
do_mount
vmlinux 0xffffffff9c24377f
Build id: 3.10.0-862.11.6.el7.x86_64
0x1ef
18
? kmem_cache_alloc_trace
vmlinux 0xffffffff9c1faf2c
Build id: 3.10.0-862.11.6.el7.x86_64
0x3c
19
SyS_mount
vmlinux 0xffffffff9c2445b3
Build id: 3.10.0-862.11.6.el7.x86_64
0x83
20
system_call_fastpath
vmlinux 0xffffffff9c72579b
Build id: 3.10.0-862.11.6.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [mount.nfs:2335]
Modules linked in: devlink ppdev rpcsec_gss_krb5 vmnet(OE) auth_rpcgss parport_pc parport nfsv4 vmw_vsock_vmci_transport vsock vmw_vmci dns_resolver nfs vmmon(OE) lockd grace fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) snd_hda_codec_hdmi sb_edac intel_powerclamp coretemp mei_wdt intel_rapl iosf_mbi kvm_intel
iTCO_wdt iTCO_vendor_support kvm dcdbas irqbypass snd_hda_codec_realtek crc32_pclmul snd_hda_codec_generic ghash_clmulni_intel snd_hda_intel drm_kms_helper aesni_intel lrw joydev snd_hda_codec gf128mul glue_helper ablk_helper syscopyarea cryptd sysfillrect sysimgblt fb_sys_fops snd_hda_core drm snd_hwdep snd_seq snd_seq_device snd_pcm pcspkr ipmi_devintf i2c_i801 ipmi_msghandler mei_me snd_timer lpc_ich snd sg mei soundcore shpchp binfmt_misc ip_tables xfs sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ahci bnx2x libahci libata crct10dif_pclmul crct10dif_common e1000e igb mpt2sas crc32c_intel serio_raw dca i2c_algo_bit i2c_core mdio ptp raid_class pps_core scsi_transport_sas libcrc32c wmi
CPU: 2 PID: 2335 Comm: mount.nfs Tainted: P           OEL ------------   3.10.0-862.11.6.el7.x86_64 #1
Hardware name: Dell Inc. Precision T7610/0NK70N, BIOS A08 07/24/2014
task: ffff966f8051dee0 ti: ffff966f818ac000 task.ti: ffff966f818ac000
RIP: 0010:[<ffffffffc1acce80>]  [<ffffffffc1acce80>] nfs_get_client+0x210/0x470 [nfs]
RSP: 0018:ffff966f818afb08  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffffff9c0beb96 RCX: 0000000000000003
RDX: ffff966f9cad6360 RSI: ffff966f9cad63b8 RDI: ffff966f9cad63b8
RBP: ffff966f818afb60 R08: ffff966f818afad0 R09: ffff966f9cad6300
R10: ffff966fadac8f28 R11: 0000000000000005 R12: 0000000000000293
R13: 0000000000000293 R14: ffffffffc1af6070 R15: ffff966f818afad0
FS:  00007f367f470880(0000) GS:ffff966fefc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fddb1be15f0 CR3: 000000101aa9e000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc1b53518>] nfs4_set_client+0x98/0x130 [nfsv4]
[<ffffffffc1b54cbe>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc1b4bd9e>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffff9c223c2e>] mount_fs+0x3e/0x1b0
[<ffffffff9c1bb1e5>] ? __alloc_percpu+0x15/0x20
[<ffffffff9c241157>] vfs_kern_mount+0x67/0x110
[<ffffffffc1b4bcc6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc1b4c0c4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc1acd6d7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc1ad9cdb>] nfs_fs_mount+0x4cb/0xda0 [nfs]
[<ffffffffc1adac20>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc1ad8ae0>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffff9c223c2e>] mount_fs+0x3e/0x1b0
[<ffffffff9c1bb1e5>] ? __alloc_percpu+0x15/0x20
[<ffffffff9c241157>] vfs_kern_mount+0x67/0x110
[<ffffffff9c24377f>] do_mount+0x1ef/0xce0
[<ffffffff9c1faf2c>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffff9c2445b3>] SyS_mount+0x83/0xd0
[<ffffffff9c72579b>] system_call_fastpath+0x22/0x27
Code: 00 00 49 8b 77 08 48 89 c2 48 c7 c7 c8 15 af c1 31 c0 e8 ea 03 c4 da 48 89 d8 e9 4f 01 00 00 0f 1f 40 00 4c 89 4d b8 48 89 55 c0 <4c> 89 55 c8 f0 41 ff 04 24 4c 89 ef c6 07 00 0f 1f 40 00 4c 89 
Complete report #198449
Frame # Function Binary Source or offset Line
1
nfs_get_client
nfs 0xffffffffc0c70e9b
Build id: 3.10.0-862.14.4.el7.x86_64
0x22b
2
nfs4_set_client
nfsv4 0xffffffffc0cda518
Build id: 3.10.0-862.14.4.el7.x86_64
0x98
3
nfs4_create_server
nfsv4 0xffffffffc0cdbcbe
Build id: 3.10.0-862.14.4.el7.x86_64
0x13e
4
nfs4_remote_mount
nfsv4 0xffffffffc0cd2d9e
Build id: 3.10.0-862.14.4.el7.x86_64
0x2e
5
mount_fs
vmlinux 0xffffffff9e023cee
Build id: 3.10.0-862.14.4.el7.x86_64
0x3e
6
? __alloc_percpu
vmlinux 0xffffffff9dfbb2a5
Build id: 3.10.0-862.14.4.el7.x86_64
0x15
7
vfs_kern_mount
vmlinux 0xffffffff9e041217
Build id: 3.10.0-862.14.4.el7.x86_64
0x67
8
nfs_do_root_mount
nfsv4 0xffffffffc0cd2cc6
Build id: 3.10.0-862.14.4.el7.x86_64
0x86
9
nfs4_try_mount
nfsv4 0xffffffffc0cd30c4
Build id: 3.10.0-862.14.4.el7.x86_64
0x44
10
? get_nfs_version
nfs 0xffffffffc0c716d7
Build id: 3.10.0-862.14.4.el7.x86_64
0x27
11
nfs_fs_mount
nfs 0xffffffffc0c7dcdb
Build id: 3.10.0-862.14.4.el7.x86_64
0x4cb
12
? nfs_clone_super
nfs 0xffffffffc0c7ec20
Build id: 3.10.0-862.14.4.el7.x86_64
0x140
13
? param_set_portnr
nfs 0xffffffffc0c7cae0
Build id: 3.10.0-862.14.4.el7.x86_64
0x70
14
mount_fs
vmlinux 0xffffffff9e023cee
Build id: 3.10.0-862.14.4.el7.x86_64
0x3e
15
? __alloc_percpu
vmlinux 0xffffffff9dfbb2a5
Build id: 3.10.0-862.14.4.el7.x86_64
0x15
16
vfs_kern_mount
vmlinux 0xffffffff9e041217
Build id: 3.10.0-862.14.4.el7.x86_64
0x67
17
do_mount
vmlinux 0xffffffff9e04383f
Build id: 3.10.0-862.14.4.el7.x86_64
0x1ef
18
? kmem_cache_alloc_trace
vmlinux 0xffffffff9dffafec
Build id: 3.10.0-862.14.4.el7.x86_64
0x3c
19
SyS_mount
vmlinux 0xffffffff9e044673
Build id: 3.10.0-862.14.4.el7.x86_64
0x83
20
system_call_fastpath
vmlinux 0xffffffff9e52579b
Build id: 3.10.0-862.14.4.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mount.nfs:1371]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc arc4 ath9k ath9k_common ath9k_hw ath mac80211 cfg80211 intel_powerclamp coretemp intel_rapl kvm_intel ath3k kvm btusb btrtl btbcm btintel bluetooth irqbypass crc32_pclmul ghash_clmulni_intel rfkill aesni_intel dell_wmi lrw dell_smbios gf128mul dcdbas iTCO_wdt glue_helper ablk_helper iTCO_vendor_support snd_hda_codec_realtek snd_hda_codec_generic snd_hda_codec_hdmi pinctrl_sunrisepoint sparse_keymap pinctrl_intel cryptd i2c_i801 mei_me mei sg snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd pcspkr soundcore acpi_pad shpchp ip_tables xfs libcrc32c rtsx_usb_sdmmc mmc_core rtsx_usb sr_mod sd_mod cdrom crc_t10dif crct10dif_generic i915 nouveau mxm_wmi
ttm iosf_mbi i2c_algo_bit ahci drm_kms_helper libahci syscopyarea sysfillrect sysimgblt fb_sys_fops libata drm crct10dif_pclmul crct10dif_common crc32c_intel r8169 serio_raw mii wmi i2c_hid video i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1371 Comm: mount.nfs Kdump: loaded Tainted: G             L ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: Dell Inc. Vostro 3668/0N9Y46, BIOS 1.7.0 01/30/2018
task: ffffa098ab484f10 ti: ffffa0988b6a8000 task.ti: ffffa0988b6a8000
RIP: 0010:[<ffffffffc0c70e9b>]  [<ffffffffc0c70e9b>] nfs_get_client+0x22b/0x470 [nfs]
RSP: 0018:ffffa0988b6abb08  EFLAGS: 00000286
RAX: 00000000fffffe00 RBX: ffffffff9debeb96 RCX: 0000000000000000
RDX: ffffffffc0c9a078 RSI: 0000000000000206 RDI: 0000000000000206
RBP: ffffa0988b6abb60 R08: ffffa0988b6abad0 R09: ffffa098bec19740
R10: ffffa0989d637328 R11: 0000000000000000 R12: 0000000000000206
R13: ffffffff9e51b7d7 R14: ffffa0988b6aba78 R15: ffffffff9e50dc5f
FS:  00007ff02b5ba880(0000) GS:ffffa098bec40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f061322a000 CR3: 00000004646fc000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc0cda518>] nfs4_set_client+0x98/0x130 [nfsv4]
[<ffffffffc0cdbcbe>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc0cd2d9e>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffff9e023cee>] mount_fs+0x3e/0x1b0
[<ffffffff9dfbb2a5>] ? __alloc_percpu+0x15/0x20
[<ffffffff9e041217>] vfs_kern_mount+0x67/0x110
[<ffffffffc0cd2cc6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc0cd30c4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc0c716d7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc0c7dcdb>] nfs_fs_mount+0x4cb/0xda0 [nfs]
[<ffffffffc0c7ec20>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc0c7cae0>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffff9e023cee>] mount_fs+0x3e/0x1b0
[<ffffffff9dfbb2a5>] ? __alloc_percpu+0x15/0x20
[<ffffffff9e041217>] vfs_kern_mount+0x67/0x110
[<ffffffff9e04383f>] do_mount+0x1ef/0xce0
[<ffffffff9dffafec>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffff9e044673>] SyS_mount+0x83/0xd0
[<ffffffff9e52579b>] system_call_fastpath+0x22/0x27
Code: 4f 01 00 00 0f 1f 40 00 4c 89 4d b8 48 89 55 c0 4c 89 55 c8 f0 41 ff 04 24 4c 89 ef c6 07 00 0f 1f 40 00 4c 89 e7 e8 25 e4 ff ff <4c> 89 e7 e8 9d f2 ff ff 4c 89 ef e8 85 a8 8a dd 4c 8b 55 c8 48 
Complete report #160976
Frame # Function Binary Source or offset Line
1
finish_wait
vmlinux 0xffffffffa7cc3ba9
Build id: 3.10.0-957.27.2.el7.x86_64
0x59
2
nfs_wait_client_init_complete
nfs 0xffffffffc0b25371
Build id: 3.10.0-957.27.2.el7.x86_64
0xa1
3
? wake_up_atomic_t
vmlinux 0xffffffffa7cc3f50
Build id: 3.10.0-957.27.2.el7.x86_64
0x30
4
nfs_get_client
nfs 0xffffffffc0b26eab
Build id: 3.10.0-957.27.2.el7.x86_64
0x22b
5
nfs4_set_client
nfsv4 0xffffffffc0c7a39c
Build id: 3.10.0-957.27.2.el7.x86_64
0x9c
6
nfs4_create_server
nfsv4 0xffffffffc0c7bb4e
Build id: 3.10.0-957.27.2.el7.x86_64
0x13e
7
nfs4_remote_mount
nfsv4 0xffffffffc0c72bae
Build id: 3.10.0-957.27.2.el7.x86_64
0x2e
8
mount_fs
vmlinux 0xffffffffa7e4723e
Build id: 3.10.0-957.27.2.el7.x86_64
0x3e
9
? __alloc_percpu
vmlinux 0xffffffffa7ddc6d5
Build id: 3.10.0-957.27.2.el7.x86_64
0x15
10
vfs_kern_mount
vmlinux 0xffffffffa7e64e67
Build id: 3.10.0-957.27.2.el7.x86_64
0x67
11
nfs_do_root_mount
nfsv4 0xffffffffc0c72ad6
Build id: 3.10.0-957.27.2.el7.x86_64
0x86
12
nfs4_try_mount
nfsv4 0xffffffffc0c72ed4
Build id: 3.10.0-957.27.2.el7.x86_64
0x44
13
? get_nfs_version
nfs 0xffffffffc0b276e7
Build id: 3.10.0-957.27.2.el7.x86_64
0x27
14
nfs_fs_mount
nfs 0xffffffffc0b33d3b
Build id: 3.10.0-957.27.2.el7.x86_64
0x4cb
15
? nfs_clone_super
nfs 0xffffffffc0b34ca0
Build id: 3.10.0-957.27.2.el7.x86_64
0x140
16
? param_set_portnr
nfs 0xffffffffc0b32b40
Build id: 3.10.0-957.27.2.el7.x86_64
0x70
17
mount_fs
vmlinux 0xffffffffa7e4723e
Build id: 3.10.0-957.27.2.el7.x86_64
0x3e
18
? __alloc_percpu
vmlinux 0xffffffffa7ddc6d5
Build id: 3.10.0-957.27.2.el7.x86_64
0x15
19
vfs_kern_mount
vmlinux 0xffffffffa7e64e67
Build id: 3.10.0-957.27.2.el7.x86_64
0x67
20
do_mount
vmlinux 0xffffffffa7e6748f
Build id: 3.10.0-957.27.2.el7.x86_64
0x1ef
21
? __check_object_size
vmlinux 0xffffffffa7e3f69a
Build id: 3.10.0-957.27.2.el7.x86_64
0x1ca
22
? kmem_cache_alloc_trace
vmlinux 0xffffffffa7e1ea2c
Build id: 3.10.0-957.27.2.el7.x86_64
0x3c
23
SyS_mount
vmlinux 0xffffffffa7e682c3
Build id: 3.10.0-957.27.2.el7.x86_64
0x83
24
sys_talpa_mount
talpa_syscallhook 0xffffffffc08378b6
Build id: 3.10.0-957.27.2.el7.x86_64
0x66
25
system_call_fastpath
vmlinux 0xffffffffa8376ddb
Build id: 3.10.0-957.27.2.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [mount.nfs:422]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter nfsv3 nfs_acl rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache talpa_vfshook(OE) talpa_pedconnector(OE) talpa_pedevice(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) ext4 mbcache jbd2 intel_pmc_core intel_powerclamp coretemp intel_rapl iTCO_wdt iTCO_vendor_support kvm ppdev asus_wmi sparse_keymap rfkill mxm_wmi irqbypass crc32_pclmul snd_hda_codec_hdmi ghash_clmulni_intel snd_hda_codec_realtek aesni_intel snd_hda_codec_generic lrw gf128mul glue_helper
ablk_helper cryptd snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device pcspkr snd_pcm snd_timer snd soundcore i2c_i801 sg parport_pc parport pinctrl_sunrisepoint pinctrl_intel acpi_pad wmi mei_me mei sch_fq_codel auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i915 crct10dif_pclmul crct10dif_common crc32c_intel serio_raw r8169 mii i2c_algo_bit iosf_mbi drm_kms_helper ahci video libahci syscopyarea sysfillrect sysimgblt fb_sys_fops i2c_hid drm libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 422 Comm: mount.nfs Kdump: loaded Tainted: G           OE  ------------   3.10.0-957.27.2.el7.x86_64 #1
Hardware name: System manufacturer System Product Name/H110M-R, BIOS 3401 03/20/2017
task: ffff9fe3b410e180 ti: ffff9fe3a5568000 task.ti: ffff9fe3a5568000
RIP: 0010:[<ffffffffa7cc3ba9>]  [<ffffffffa7cc3ba9>] finish_wait+0x59/0x70
RSP: 0018:ffff9fe3a556ba50  EFLAGS: 00000283
RAX: ffffffffc0b4e078 RBX: ffff9fe3a556ba88 RCX: dead000000000200
RDX: ffffffffc0b4e078 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff9fe3a556ba60 R08: ffff9fe3a556ba88 R09: ffff9fe3fc496b00
R10: ffff9fe4ce368528 R11: 0000000000000000 R12: ffff9fe4ce368528
R13: 0000000000000000 R14: ffff9fe4ce368528 R15: ffff9fe3a556ba30
FS:  00007fd2328f9880(0000) GS:ffff9fe576d80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc82f595010 CR3: 00000000740a0000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc0b25371>] nfs_wait_client_init_complete+0xa1/0xe0 [nfs]
[<ffffffffa7cc3f50>] ? wake_up_atomic_t+0x30/0x30
[<ffffffffc0b26eab>] nfs_get_client+0x22b/0x470 [nfs]
[<ffffffffc0c7a39c>] nfs4_set_client+0x9c/0x150 [nfsv4]
[<ffffffffc0c7bb4e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc0c72bae>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffffa7e4723e>] mount_fs+0x3e/0x1b0
[<ffffffffa7ddc6d5>] ? __alloc_percpu+0x15/0x20
[<ffffffffa7e64e67>] vfs_kern_mount+0x67/0x110
[<ffffffffc0c72ad6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc0c72ed4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc0b276e7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc0b33d3b>] nfs_fs_mount+0x4cb/0xdc0 [nfs]
[<ffffffffc0b34ca0>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc0b32b40>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffffa7e4723e>] mount_fs+0x3e/0x1b0
[<ffffffffa7ddc6d5>] ? __alloc_percpu+0x15/0x20
[<ffffffffa7e64e67>] vfs_kern_mount+0x67/0x110
[<ffffffffa7e6748f>] do_mount+0x1ef/0xce0
[<ffffffffa7e3f69a>] ? __check_object_size+0x1ca/0x250
[<ffffffffa7e1ea2c>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffffa7e682c3>] SyS_mount+0x83/0xd0
[<ffffffffc08378b6>] sys_talpa_mount+0x66/0x100 [talpa_syscallhook]
[<ffffffffa8376ddb>] system_call_fastpath+0x22/0x27
Code: 74 30 4c 89 ef e8 68 8c 6a 00 4c 89 e7 49 89 c6 e8 5d 2d 2d 00 4c 89 63 18 4c 89 63 20 4c 89 f6 4c 89 ef e8 da 88 6a 00 5b 41 5c <41> 5d 41 5e 5d c3 90 4c 3b 66 20 75 ca eb ee 0f 1f 84 00 00 00 
Complete report #123804
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff84f1b445
Build id: 3.10.0-862.14.4.el7.x86_64
0x15
2
prepare_to_wait
vmlinux 0xffffffff848bea36
Build id: 3.10.0-862.14.4.el7.x86_64
0x56
3
nfs_wait_client_init_complete
nfs 0xffffffffc0e71348
Build id: 3.10.0-862.14.4.el7.x86_64
0x88
4
? wake_up_atomic_t
vmlinux 0xffffffff848bef10
Build id: 3.10.0-862.14.4.el7.x86_64
0x30
5
nfs_get_client
nfs 0xffffffffc0e72e9b
Build id: 3.10.0-862.14.4.el7.x86_64
0x22b
6
nfs4_set_client
nfsv4 0xffffffffc0f90518
Build id: 3.10.0-862.14.4.el7.x86_64
0x98
7
nfs4_create_server
nfsv4 0xffffffffc0f91cbe
Build id: 3.10.0-862.14.4.el7.x86_64
0x13e
8
nfs4_remote_mount
nfsv4 0xffffffffc0f88d9e
Build id: 3.10.0-862.14.4.el7.x86_64
0x2e
9
mount_fs
vmlinux 0xffffffff84a23cee
Build id: 3.10.0-862.14.4.el7.x86_64
0x3e
10
? __alloc_percpu
vmlinux 0xffffffff849bb2a5
Build id: 3.10.0-862.14.4.el7.x86_64
0x15
11
vfs_kern_mount
vmlinux 0xffffffff84a41217
Build id: 3.10.0-862.14.4.el7.x86_64
0x67
12
nfs_do_root_mount
nfsv4 0xffffffffc0f88cc6
Build id: 3.10.0-862.14.4.el7.x86_64
0x86
13
nfs4_try_mount
nfsv4 0xffffffffc0f890c4
Build id: 3.10.0-862.14.4.el7.x86_64
0x44
14
? get_nfs_version
nfs 0xffffffffc0e736d7
Build id: 3.10.0-862.14.4.el7.x86_64
0x27
15
nfs_fs_mount
nfs 0xffffffffc0e7fcdb
Build id: 3.10.0-862.14.4.el7.x86_64
0x4cb
16
? nfs_clone_super
nfs 0xffffffffc0e80c20
Build id: 3.10.0-862.14.4.el7.x86_64
0x140
17
? param_set_portnr
nfs 0xffffffffc0e7eae0
Build id: 3.10.0-862.14.4.el7.x86_64
0x70
18
mount_fs
vmlinux 0xffffffff84a23cee
Build id: 3.10.0-862.14.4.el7.x86_64
0x3e
19
? __alloc_percpu
vmlinux 0xffffffff849bb2a5
Build id: 3.10.0-862.14.4.el7.x86_64
0x15
20
vfs_kern_mount
vmlinux 0xffffffff84a41217
Build id: 3.10.0-862.14.4.el7.x86_64
0x67
21
do_mount
vmlinux 0xffffffff84a4383f
Build id: 3.10.0-862.14.4.el7.x86_64
0x1ef
22
? kmem_cache_alloc_trace
vmlinux 0xffffffff849fafec
Build id: 3.10.0-862.14.4.el7.x86_64
0x3c
23
SyS_mount
vmlinux 0xffffffff84a44673
Build id: 3.10.0-862.14.4.el7.x86_64
0x83
24
system_call_fastpath
vmlinux 0xffffffff84f2579b
Build id: 3.10.0-862.14.4.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [mount.nfs:1443]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc snd_hda_codec_realtek snd_hda_codec_generic arc4 ath9k snd_hda_codec_hdmi intel_powerclamp coretemp ath9k_common intel_rapl ath9k_hw kvm_intel ath snd_hda_intel kvm snd_hda_codec mac80211 iTCO_wdt iTCO_vendor_support snd_hda_core dell_wmi snd_hwdep dell_smbios sparse_keymap snd_seq irqbypass crc32_pclmul snd_seq_device ghash_clmulni_intel ath3k dcdbas btusb snd_pcm btrtl btbcm btintel bluetooth cfg80211 aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr mei_me snd_timer mei pinctrl_sunrisepoint i2c_i801 pinctrl_intel rfkill snd soundcore shpchp acpi_pad ip_tables xfs libcrc32c rtsx_usb_sdmmc mmc_core rtsx_usb sr_mod sd_mod cdrom crc_t10dif crct10dif_generic i915 nouveau mxm_wmi
ttm iosf_mbi i2c_algo_bit drm_kms_helper ahci syscopyarea sysfillrect sysimgblt fb_sys_fops libahci drm libata crct10dif_pclmul crct10dif_common crc32c_intel r8169 serio_raw mii wmi i2c_hid video i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 1443 Comm: mount.nfs Kdump: loaded Tainted: G             L ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: Dell Inc. Vostro 3668/0N9Y46, BIOS 1.7.0 01/30/2018
task: ffff9be198b9dee0 ti: ffff9bddb4e34000 task.ti: ffff9bddb4e34000
RIP: 0010:[<ffffffff84f1b445>]  [<ffffffff84f1b445>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9bddb4e37a70  EFLAGS: 00000246
RAX: 0000000000000246 RBX: ffff9be1a9235528 RCX: ffff9be198b9dee0
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff9bddb4e37a70 R08: ffff9bddb4e27ad0 R09: ffff9be1bec99740
R10: ffff9be1a9235528 R11: 0000000000000000 R12: ffff9be1bedd8b40
R13: ffff9bddb4e37a70 R14: ffff9be198b9e578 R15: 00000001848e066d
FS:  00007fb50f9c9880(0000) GS:ffff9be1bedc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056125f836000 CR3: 00000004683da000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff848bea36>] prepare_to_wait+0x56/0x90
[<ffffffffc0e71348>] nfs_wait_client_init_complete+0x88/0xe0 [nfs]
[<ffffffff848bef10>] ? wake_up_atomic_t+0x30/0x30
[<ffffffffc0e72e9b>] nfs_get_client+0x22b/0x470 [nfs]
[<ffffffffc0f90518>] nfs4_set_client+0x98/0x130 [nfsv4]
[<ffffffffc0f91cbe>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc0f88d9e>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffff84a23cee>] mount_fs+0x3e/0x1b0
[<ffffffff849bb2a5>] ? __alloc_percpu+0x15/0x20
[<ffffffff84a41217>] vfs_kern_mount+0x67/0x110
[<ffffffffc0f88cc6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc0f890c4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc0e736d7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc0e7fcdb>] nfs_fs_mount+0x4cb/0xda0 [nfs]
[<ffffffffc0e80c20>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc0e7eae0>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffff84a23cee>] mount_fs+0x3e/0x1b0
[<ffffffff849bb2a5>] ? __alloc_percpu+0x15/0x20
[<ffffffff84a41217>] vfs_kern_mount+0x67/0x110
[<ffffffff84a4383f>] do_mount+0x1ef/0xce0
[<ffffffff849fafec>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffff84a44673>] SyS_mount+0x83/0xd0
[<ffffffff84f2579b>] system_call_fastpath+0x22/0x27
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #52216
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff816b6545
Build id: 3.10.0-693.21.1.el7.x86_64
0x15
2
finish_wait
vmlinux 0xffffffff810b4d86
Build id: 3.10.0-693.21.1.el7.x86_64
0x56
3
nfs_wait_client_init_complete
nfs 0xffffffffc054a361
Build id: 3.10.0-693.21.1.el7.x86_64
0xa1
4
? wake_up_atomic_t
vmlinux 0xffffffff810b4fc0
Build id: 3.10.0-693.21.1.el7.x86_64
0x30
5
nfs_get_client
nfs 0xffffffffc054be9b
Build id: 3.10.0-693.21.1.el7.x86_64
0x22b
6
nfs4_set_client
nfsv4 0xffffffffc05b4fd8
Build id: 3.10.0-693.21.1.el7.x86_64
0x98
7
nfs4_create_server
nfsv4 0xffffffffc05b677e
Build id: 3.10.0-693.21.1.el7.x86_64
0x13e
8
nfs4_remote_mount
nfsv4 0xffffffffc05ad91e
Build id: 3.10.0-693.21.1.el7.x86_64
0x2e
9
mount_fs
vmlinux 0xffffffff81209f1e
Build id: 3.10.0-693.21.1.el7.x86_64
0x3e
10
? __alloc_percpu
vmlinux 0xffffffff811aa685
Build id: 3.10.0-693.21.1.el7.x86_64
0x15
11
vfs_kern_mount
vmlinux 0xffffffff81226d57
Build id: 3.10.0-693.21.1.el7.x86_64
0x67
12
nfs_do_root_mount
nfsv4 0xffffffffc05ad846
Build id: 3.10.0-693.21.1.el7.x86_64
0x86
13
nfs4_try_mount
nfsv4 0xffffffffc05adc44
Build id: 3.10.0-693.21.1.el7.x86_64
0x44
14
? get_nfs_version
nfs 0xffffffffc054c6d7
Build id: 3.10.0-693.21.1.el7.x86_64
0x27
15
nfs_fs_mount
nfs 0xffffffffc0558c9b
Build id: 3.10.0-693.21.1.el7.x86_64
0x4cb
16
? nfs_clone_super
nfs 0xffffffffc0559be0
Build id: 3.10.0-693.21.1.el7.x86_64
0x140
17
? param_set_portnr
nfs 0xffffffffc0557aa0
Build id: 3.10.0-693.21.1.el7.x86_64
0x70
18
mount_fs
vmlinux 0xffffffff81209f1e
Build id: 3.10.0-693.21.1.el7.x86_64
0x3e
19
? __alloc_percpu
vmlinux 0xffffffff811aa685
Build id: 3.10.0-693.21.1.el7.x86_64
0x15
20
vfs_kern_mount
vmlinux 0xffffffff81226d57
Build id: 3.10.0-693.21.1.el7.x86_64
0x67
21
do_mount
vmlinux 0xffffffff81229263
Build id: 3.10.0-693.21.1.el7.x86_64
0x233
22
SyS_mount
vmlinux 0xffffffff81229ea6
Build id: 3.10.0-693.21.1.el7.x86_64
0x96
23
system_call_fastpath
vmlinux 0xffffffff816c0715
Build id: 3.10.0-693.21.1.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mount.nfs:977]
Modules linked in: uinput ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache snd_hda_codec_generic snd_hda_intel snd_hda_codec iosf_mbi snd_hda_core crc32_pclmul snd_hwdep snd_seq snd_seq_device ghash_clmulni_intel snd_pcm aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr virtio_balloon snd_timer sg snd soundcore i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_blk virtio_net qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel floppy virtio_pci serio_raw virtio_ring virtio i2c_core dm_mirror dm_region_hash dm_log
dm_mod
CPU: 1 PID: 977 Comm: mount.nfs Not tainted 3.10.0-693.21.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.10.2-1 04/01/2014
task: ffff880134ba0000 ti: ffff880137cdc000 task.ti: ffff880137cdc000
RIP: 0010:[<ffffffff816b6545>]  [<ffffffff816b6545>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff880137cdfa60  EFLAGS: 00000283
RAX: ffffffffc0575078 RBX: ffff8800b509ad28 RCX: dead000000000200
RDX: ffffffffc0575078 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff880137cdfa60 R08: ffff880137cdfab8 R09: ffff8800b723fa00
R10: ffff8800b509ad28 R11: ffffea0002d45300 R12: ffff8800b509ad28
R13: ffff880137cdfa60 R14: 0000000000000040 R15: ffff880137cdfa58
FS:  00007f0643806880(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f01a3529000 CR3: 00000000b73b4000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff810b4d86>] finish_wait+0x56/0x70
[<ffffffffc054a361>] nfs_wait_client_init_complete+0xa1/0xe0 [nfs]
[<ffffffff810b4fc0>] ? wake_up_atomic_t+0x30/0x30
[<ffffffffc054be9b>] nfs_get_client+0x22b/0x470 [nfs]
[<ffffffffc05b4fd8>] nfs4_set_client+0x98/0x130 [nfsv4]
[<ffffffffc05b677e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc05ad91e>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffff81209f1e>] mount_fs+0x3e/0x1b0
[<ffffffff811aa685>] ? __alloc_percpu+0x15/0x20
[<ffffffff81226d57>] vfs_kern_mount+0x67/0x110
[<ffffffffc05ad846>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc05adc44>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc054c6d7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc0558c9b>] nfs_fs_mount+0x4cb/0xda0 [nfs]
[<ffffffffc0559be0>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc0557aa0>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffff81209f1e>] mount_fs+0x3e/0x1b0
[<ffffffff811aa685>] ? __alloc_percpu+0x15/0x20
[<ffffffff81226d57>] vfs_kern_mount+0x67/0x110
[<ffffffff81229263>] do_mount+0x233/0xaf0
[<ffffffff81229ea6>] SyS_mount+0x96/0xf0
[<ffffffff816c0715>] system_call_fastpath+0x1c/0x21
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #14207
Frame # Function Binary Source or offset Line
1
prepare_to_wait
vmlinux 0xffffffff9fec27fb
Build id: 3.10.0-957.1.3.el7.x86_64
0x5b
2
nfs_wait_client_init_complete
nfs 0xffffffffc0939358
Build id: 3.10.0-957.1.3.el7.x86_64
0x88
3
? wake_up_atomic_t
vmlinux 0xffffffff9fec2d00
Build id: 3.10.0-957.1.3.el7.x86_64
0x30
4
nfs_get_client
nfs 0xffffffffc093aeab
Build id: 3.10.0-957.1.3.el7.x86_64
0x22b
5
nfs4_set_client
nfsv4 0xffffffffc09f939c
Build id: 3.10.0-957.1.3.el7.x86_64
0x9c
6
nfs4_create_server
nfsv4 0xffffffffc09fab4e
Build id: 3.10.0-957.1.3.el7.x86_64
0x13e
7
nfs4_remote_mount
nfsv4 0xffffffffc09f1bae
Build id: 3.10.0-957.1.3.el7.x86_64
0x2e
8
mount_fs
vmlinux 0xffffffffa0045e4e
Build id: 3.10.0-957.1.3.el7.x86_64
0x3e
9
? __alloc_percpu
vmlinux 0xffffffff9ffdb265
Build id: 3.10.0-957.1.3.el7.x86_64
0x15
10
vfs_kern_mount
vmlinux 0xffffffffa00639e7
Build id: 3.10.0-957.1.3.el7.x86_64
0x67
11
nfs_do_root_mount
nfsv4 0xffffffffc09f1ad6
Build id: 3.10.0-957.1.3.el7.x86_64
0x86
12
nfs4_try_mount
nfsv4 0xffffffffc09f1ed4
Build id: 3.10.0-957.1.3.el7.x86_64
0x44
13
? get_nfs_version
nfs 0xffffffffc093b6e7
Build id: 3.10.0-957.1.3.el7.x86_64
0x27
14
nfs_fs_mount
nfs 0xffffffffc0947d3b
Build id: 3.10.0-957.1.3.el7.x86_64
0x4cb
15
? nfs_clone_super
nfs 0xffffffffc0948ca0
Build id: 3.10.0-957.1.3.el7.x86_64
0x140
16
? param_set_portnr
nfs 0xffffffffc0946b40
Build id: 3.10.0-957.1.3.el7.x86_64
0x70
17
mount_fs
vmlinux 0xffffffffa0045e4e
Build id: 3.10.0-957.1.3.el7.x86_64
0x3e
18
? __alloc_percpu
vmlinux 0xffffffff9ffdb265
Build id: 3.10.0-957.1.3.el7.x86_64
0x15
19
vfs_kern_mount
vmlinux 0xffffffffa00639e7
Build id: 3.10.0-957.1.3.el7.x86_64
0x67
20
do_mount
vmlinux 0xffffffffa006600f
Build id: 3.10.0-957.1.3.el7.x86_64
0x1ef
21
? __check_object_size
vmlinux 0xffffffffa003e2aa
Build id: 3.10.0-957.1.3.el7.x86_64
0x1ca
22
? kmem_cache_alloc_trace
vmlinux 0xffffffffa001c74c
Build id: 3.10.0-957.1.3.el7.x86_64
0x3c
23
SyS_mount
vmlinux 0xffffffffa0066e43
Build id: 3.10.0-957.1.3.el7.x86_64
0x83
24
system_call_fastpath
vmlinux 0xffffffffa0574ddb
Build id: 3.10.0-957.1.3.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [mount.nfs:221230]
Modules linked in: fuse rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache devlink bnx2fc cnic uio fcoe libfcoe libfc scsi_transport_fc scsi_tgt ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iTCO_wdt iTCO_vendor_support dcdbas sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass dm_service_time pcspkr sg ipmi_si ipmi_devintf ipmi_msghandler lpc_ich mei_me acpi_power_meter mei acpi_pad
bridge dm_multipath bonding ip_tables ext4 mbcache jbd2 dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel mgag200 8021q garp drm_kms_helper ghash_clmulni_intel mrp stp aesni_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm lrw llc gf128mul glue_helper ablk_helper cryptd drm drm_panel_orientation_quirks igb i2c_algo_bit ahci ixgbe libahci scsi_transport_iscsi libata megaraid_sas dca ptp pps_core mdio wmi sunrpc dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 221230 Comm: mount.nfs Kdump: loaded Not tainted 3.10.0-957.1.3.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R630/0CNCJW, BIOS 1.0.4 08/28/2014
task: ffff9616dc7a4100 ti: ffff96173f94c000 task.ti: ffff96173f94c000
RIP: 0010:[<ffffffff9fec27fb>]  [<ffffffff9fec27fb>] prepare_to_wait+0x5b/0x90
RSP: 0018:ffff96173f94fa98  EFLAGS: 00000286
RAX: 0000000000000246 RBX: ffff96173f94fad0 RCX: ffff9616dc7a4100
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff96173f94faa8 R08: ffffffffc0962078 R09: ffff96173f945900
R10: ffff9616edf9c128 R11: 0000000000000005 R12: ffff9616edf9c128
R13: 0000000000000005 R14: ffff9616edf9c128 R15: ffff96173f94fa78
FS:  00007f3a46e0d880(0000) GS:ffff9616fae80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f750903a910 CR3: 0000000ff8ad2000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc0939358>] nfs_wait_client_init_complete+0x88/0xe0 [nfs]
[<ffffffff9fec2d00>] ? wake_up_atomic_t+0x30/0x30
[<ffffffffc093aeab>] nfs_get_client+0x22b/0x470 [nfs]
[<ffffffffc09f939c>] nfs4_set_client+0x9c/0x150 [nfsv4]
[<ffffffffc09fab4e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc09f1bae>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffffa0045e4e>] mount_fs+0x3e/0x1b0
[<ffffffff9ffdb265>] ? __alloc_percpu+0x15/0x20
[<ffffffffa00639e7>] vfs_kern_mount+0x67/0x110
[<ffffffffc09f1ad6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc09f1ed4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc093b6e7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc0947d3b>] nfs_fs_mount+0x4cb/0xdc0 [nfs]
[<ffffffffc0948ca0>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc0946b40>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffffa0045e4e>] mount_fs+0x3e/0x1b0
[<ffffffff9ffdb265>] ? __alloc_percpu+0x15/0x20
[<ffffffffa00639e7>] vfs_kern_mount+0x67/0x110
[<ffffffffa006600f>] do_mount+0x1ef/0xce0
[<ffffffffa003e2aa>] ? __check_object_size+0x1ca/0x250
[<ffffffffa001c74c>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffffa0066e43>] SyS_mount+0x83/0xd0
[<ffffffffa0574ddb>] system_call_fastpath+0x22/0x27
Code: 89 6d e0 48 8b 55 e0 65 48 8b 0c 25 80 0e 01 00 48 87 11 4c 89 e7 48 89 c6 48 89 55 e0 48 8b 55 e0 e8 6a 79 6a 00 48 83 c4 10 5b <41> 5c 41 5d 5d c3 0f 1f 80 00 00 00 00 49 8b 54 24 08 49 8d 74 
Complete report #3233
Frame # Function Binary Source or offset Line
1
finish_wait
vmlinux 0xffffffffb26c2906
Build id: 3.10.0-957.el7.x86_64
0x6
2
? nfs_wait_client_init_complete
nfs 0xffffffffc068d371
Build id: 3.10.0-957.el7.x86_64
0xa1
3
? wake_up_atomic_t
vmlinux 0xffffffffb26c2d00
Build id: 3.10.0-957.el7.x86_64
0x30
4
nfs_get_client
nfs 0xffffffffc068eeab
Build id: 3.10.0-957.el7.x86_64
0x22b
5
nfs4_set_client
nfsv4 0xffffffffc06f839c
Build id: 3.10.0-957.el7.x86_64
0x9c
6
nfs4_create_server
nfsv4 0xffffffffc06f9b4e
Build id: 3.10.0-957.el7.x86_64
0x13e
7
nfs4_remote_mount
nfsv4 0xffffffffc06f0bae
Build id: 3.10.0-957.el7.x86_64
0x2e
8
mount_fs
vmlinux 0xffffffffb2845bde
Build id: 3.10.0-957.el7.x86_64
0x3e
9
? __alloc_percpu
vmlinux 0xffffffffb27daff5
Build id: 3.10.0-957.el7.x86_64
0x15
10
vfs_kern_mount
vmlinux 0xffffffffb2863777
Build id: 3.10.0-957.el7.x86_64
0x67
11
nfs_do_root_mount
nfsv4 0xffffffffc06f0ad6
Build id: 3.10.0-957.el7.x86_64
0x86
12
nfs4_try_mount
nfsv4 0xffffffffc06f0ed4
Build id: 3.10.0-957.el7.x86_64
0x44
13
? get_nfs_version
nfs 0xffffffffc068f6e7
Build id: 3.10.0-957.el7.x86_64
0x27
14
nfs_fs_mount
nfs 0xffffffffc069bd3b
Build id: 3.10.0-957.el7.x86_64
0x4cb
15
? nfs_clone_super
nfs 0xffffffffc069cca0
Build id: 3.10.0-957.el7.x86_64
0x140
16
? param_set_portnr
nfs 0xffffffffc069ab40
Build id: 3.10.0-957.el7.x86_64
0x70
17
mount_fs
vmlinux 0xffffffffb2845bde
Build id: 3.10.0-957.el7.x86_64
0x3e
18
? __alloc_percpu
vmlinux 0xffffffffb27daff5
Build id: 3.10.0-957.el7.x86_64
0x15
19
vfs_kern_mount
vmlinux 0xffffffffb2863777
Build id: 3.10.0-957.el7.x86_64
0x67
20
do_mount
vmlinux 0xffffffffb2865d9f
Build id: 3.10.0-957.el7.x86_64
0x1ef
21
? __check_object_size
vmlinux 0xffffffffb283e03a
Build id: 3.10.0-957.el7.x86_64
0x1ca
22
? kmem_cache_alloc_trace
vmlinux 0xffffffffb281c4dc
Build id: 3.10.0-957.el7.x86_64
0x3c
23
SyS_mount
vmlinux 0xffffffffb2866bd3
Build id: 3.10.0-957.el7.x86_64
0x83
24
system_call_fastpath
vmlinux 0xffffffffb2d74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [mount.nfs:2103]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg parport_pc parport vmw_vmci
i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ata_piix libahci drm crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw nfit vmxnet3 libnvdimm vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 18 PID: 2103 Comm: mount.nfs Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/09/2019
task: ffff8b7335621040 ti: ffff8b72b3798000 task.ti: ffff8b72b3798000
RIP: 0010:[<ffffffffb26c2906>]  [<ffffffffb26c2906>] finish_wait+0x6/0x70
RSP: 0018:ffff8b72b379baa8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff8b7335621040 RCX: ffff8b7335621040
RDX: 0000000000000004 RSI: ffff8b72b379bab8 RDI: ffffffffc06b6070
RBP: ffff8b72b379baf8 R08: ffffffffc06b6078 R09: ffff8b729947b400
R10: ffff8b73e9b09b28 R11: 0000000000000000 R12: ffffffffc06b6078
R13: ffff8b729947b400 R14: ffff8b73e9b09b28 R15: 0000000000000000
FS:  00007f9112dd9880(0000) GS:ffff8b743f680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555eac84f800 CR3: 00000000b3470000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc068d371>] ? nfs_wait_client_init_complete+0xa1/0xe0 [nfs]
[<ffffffffb26c2d00>] ? wake_up_atomic_t+0x30/0x30
[<ffffffffc068eeab>] nfs_get_client+0x22b/0x470 [nfs]
[<ffffffffc06f839c>] nfs4_set_client+0x9c/0x150 [nfsv4]
[<ffffffffc06f9b4e>] nfs4_create_server+0x13e/0x3b0 [nfsv4]
[<ffffffffc06f0bae>] nfs4_remote_mount+0x2e/0x60 [nfsv4]
[<ffffffffb2845bde>] mount_fs+0x3e/0x1b0
[<ffffffffb27daff5>] ? __alloc_percpu+0x15/0x20
[<ffffffffb2863777>] vfs_kern_mount+0x67/0x110
[<ffffffffc06f0ad6>] nfs_do_root_mount+0x86/0xc0 [nfsv4]
[<ffffffffc06f0ed4>] nfs4_try_mount+0x44/0xc0 [nfsv4]
[<ffffffffc068f6e7>] ? get_nfs_version+0x27/0x90 [nfs]
[<ffffffffc069bd3b>] nfs_fs_mount+0x4cb/0xdc0 [nfs]
[<ffffffffc069cca0>] ? nfs_clone_super+0x140/0x140 [nfs]
[<ffffffffc069ab40>] ? param_set_portnr+0x70/0x70 [nfs]
[<ffffffffb2845bde>] mount_fs+0x3e/0x1b0
[<ffffffffb27daff5>] ? __alloc_percpu+0x15/0x20
[<ffffffffb2863777>] vfs_kern_mount+0x67/0x110
[<ffffffffb2865d9f>] do_mount+0x1ef/0xce0
[<ffffffffb283e03a>] ? __check_object_size+0x1ca/0x250
[<ffffffffb281c4dc>] ? kmem_cache_alloc_trace+0x3c/0x200
[<ffffffffb2866bd3>] SyS_mount+0x83/0xd0
[<ffffffffb2d74ddb>] system_call_fastpath+0x22/0x27
Code: 7c 24 18 49 89 c5 e8 aa 21 2d 00 4c 89 ee 48 89 df e8 ef 77 6a 00 5b 41 5c 41 5d 5d c3 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 <65> 48 8b 04 25 80 0e 01 00 48 89 e5 41 56 41 55 49 89 fd 41 54