| Operating system | Count |
|---|---|
| CentOS 7 | 4 |
| Architecture | Count |
|---|---|
| x86_64 | 4 |
| Related packages | Count |
|---|---|
| kernel | 4 |
Packages names with count less than the total count of reports are most probably not the cause of the problem.
No reports in the last 20 days.
NMI watchdog: BUG: soft lockup - CPU#30 stuck for 22s! [node_exporter:200343] Modules linked in: tcp_diag inet_diag snd_seq snd_seq_device snd_timer snd soundcore veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack libcrc32c br_netfilter bridge overlay(T) nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache 8021q garp mrp stp llc bonding rpcrdma ib_isert iscsi_target_mod ib_iser libiscsi scsi_transport_iscsi ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_ucm ib_uverbs ib_umad rdma_cm ib_cm iw_cm i40iw ib_core ip6_tables skx_edac edac_core coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt joydev iTCO_vendor_support sg nfit pcspkr i2c_i801 libnvdimm shpchp lpc_ich mei_me mei ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod sr_mod crc_t10dif cdrom crct10dif_generic uas usb_storage ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe drm igb crct10dif_pclmul i40e mpt3sas crct10dif_common crc32c_intel ahci libahci libata mdio dca raid_class i2c_algo_bit scsi_transport_sas ptp i2c_core pps_core CPU: 30 PID: 200343 Comm: node_exporter Tainted: G W L ------------ T 3.10.0-693.5.2.el7.x86_64 #1 Hardware name: Sugon CX50-G35/60P16B-TC4600T, BIOS 0KLSN218 05/22/2019 task: ffff882f7cac2f70 ti: ffff8821a5b0c000 task.ti: ffff8821a5b0c000 RIP: 0010:[<ffffffff810fa366>] [<ffffffff810fa366>] native_queued_spin_lock_slowpath+0x146/0x1e0 RSP: 0018:ffff8821a5b0f7f8 EFLAGS: 00000202 RAX: 0000000000000101 RBX: ffff8821a5b0f788 RCX: 0000000000f10000 RDX: 0000000000f10101 RSI: 0000000000000101 RDI: ffff882fdcb797b0 RBP: ffff8821a5b0f7f8 R08: ffff882fddc97880 R09: 0000000000000000 R10: ffff8821a5b0fad0 R11: ffffea0080fd7000 R12: 0000000000000000 R13: 00000000000022f4 R14: 000000000000015b R15: ffff882fdd242e80 FS: 000000c0001bd090(0000) GS:ffff882fddc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000c000cd4000 CR3: 000000184e136000 CR4: 00000000003407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Stack: ffff8821a5b0f808 ffffffff8169e6bf ffff8821a5b0f818 ffffffff816abc53 ffff8821a5b0f838 ffffffff8156cfd0 ffff8821a5b0fad0 000000000000007c ffff8821a5b0f900 ffffffff815db0eb 0000000000000032 0000000000016cc0 Call Trace: [<ffffffff8169e6bf>] queued_spin_lock_slowpath+0xb/0xf [<ffffffff816abc53>] _raw_spin_lock_bh+0x33/0x40 [<ffffffff8156cfd0>] lock_sock_nested+0x20/0x50 [<ffffffff815db0eb>] tcp_sendmsg+0x3b/0xc30 [<ffffffff81606943>] inet_sendmsg+0x63/0xb0 [<ffffffff8156a580>] sock_sendmsg+0xb0/0xf0 [<ffffffff813227c5>] ? cpumask_next_and+0x35/0x50 [<ffffffff810d1c52>] ? load_balance+0x1e2/0x9a0 [<ffffffff8156a5f7>] kernel_sendmsg+0x37/0x50 [<ffffffffc02cdc69>] xs_send_kvec+0xa9/0xb0 [sunrpc] [<ffffffffc02cdf07>] xs_sendpages+0x67/0x1f0 [sunrpc] [<ffffffffc093cd10>] ? nfs3_xdr_enc_access3args+0x30/0x30 [nfsv3] [<ffffffffc02cf821>] xs_tcp_send_request+0x91/0x220 [sunrpc] [<ffffffffc02cc065>] xprt_transmit+0x65/0x310 [sunrpc] [<ffffffffc02c7f30>] call_transmit+0x1d0/0x2c0 [sunrpc] [<ffffffffc02c7d60>] ? call_decode+0x880/0x880 [sunrpc] [<ffffffffc02c7d60>] ? call_decode+0x880/0x880 [sunrpc] [<ffffffffc02d5017>] __rpc_execute+0x97/0x410 [sunrpc] [<ffffffff810b18e5>] ? wake_up_bit+0x25/0x30 [<ffffffffc02d67b8>] rpc_execute+0x68/0xb0 [sunrpc] [<ffffffffc02c6776>] rpc_run_task+0xf6/0x150 [sunrpc] [<ffffffffc02c68e0>] rpc_call_sync+0x50/0xc0 [sunrpc] [<ffffffffc093b5eb>] nfs3_rpc_wrapper.constprop.11+0x6b/0xb0 [nfsv3] [<ffffffffc093c376>] nfs3_proc_statfs+0x56/0xb0 [nfsv3] [<ffffffffc0877fe8>] nfs_statfs+0x68/0x190 [nfs] [<ffffffff81234891>] statfs_by_dentry+0xa1/0x140 [<ffffffff8123494b>] vfs_statfs+0x1b/0xb0 [<ffffffff81234a35>] user_statfs+0x55/0xa0 [<ffffffff81234aa7>] SYSC_statfs+0x27/0x60 [<ffffffff81234cae>] SyS_statfs+0xe/0x10 [<ffffffff816b5089>] system_call_fastpath+0x16/0x1b Code: 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 85 c0 74 21 83 f8 03 75 10 eb 1a 66 2e 0f 1f 84 00 00 00 00 00 85 c0 74 0c f3 90 <8b> 17 0f b7 c2 83 f8 03 75 f0 be 01 00 00 00 eb 15 66 0f 1f 84
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [k3s-server:52508] Modules linked in: xt_statistic tcp_lp nls_utf8 isofs ip6t_MASQUERADE nf_nat_masquerade_ipv6 vxlan ip6_udp_tunnel udp_tunnel ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle xt_physdev xt_NFLOG nfnetlink_log xt_limit xt_mark xt_multiport xt_comment ip_set veth xt_nat nf_conntrack_netlink xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache overlay(T) br_netfilter bridge stp llc vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 iosf_mbi snd_rawmidi crc32_pclmul snd_ac97_codec ppdev ghash_clmulni_intel vmw_balloon aesni_intel ac97_bus lrw gf128mul snd_seq glue_helper ablk_helper cryptd snd_seq_device snd_pcm joydev pcspkr snd_timer snd sg vmw_vmci soundcore i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi nfit crct10dif_pclmul crct10dif_common crc32c_intel scsi_transport_spi libnvdimm mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse CPU: 13 PID: 52508 Comm: k3s-server Kdump: loaded Tainted: G W L ------------ T 3.10.0-1160.118.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020 task: ffff9b763d2a3180 ti: ffff9b728e064000 task.ti: ffff9b728e064000 RIP: 0010:[<ffffffff9211eb5d>] [<ffffffff9211eb5d>] native_queued_spin_lock_slowpath+0x1d/0x200 RSP: 0018:ffff9b728e067730 EFLAGS: 00000293 RAX: 0000000000000001 RBX: ffff9b76aeb41f00 RCX: 0000000000000001 RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff9b76fbe36cf0 RBP: ffff9b728e067730 R08: 0000000000000000 R09: 0000000000000000 R10: ffff9b728e067a10 R11: 27b407bedb21b59c R12: ffff9b728e0676b8 R13: 0000000000000000 R14: 0000000000000001 R15: 000000000000ffcb FS: 00007fc0e2946f38(0000) GS:ffff9b775d940000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fdcb1efdd70 CR3: 000000038a3e6000 CR4: 00000000007607e0 PKRU: 55555554 Call Trace: [<ffffffff927ac21a>] queued_spin_lock_slowpath+0xb/0x13 [<ffffffff927ba777>] _raw_spin_lock_bh+0x37/0x40 [<ffffffff926629b0>] lock_sock_nested+0x20/0x50 [<ffffffff926de1db>] tcp_sendmsg+0x3b/0xc60 [<ffffffff9270b4e9>] inet_sendmsg+0x69/0xb0 [<ffffffff923b7fab>] ? _kstrtoull+0x3b/0xb0 [<ffffffff9265e4f6>] sock_sendmsg+0xb6/0x100 [<ffffffff923b8034>] ? kstrtoull+0x14/0x20 [<ffffffff9213b8c2>] ? make_kgid+0x12/0x20 [<ffffffffc04afc4f>] ? xdr_inline_decode+0x2f/0xd0 [sunrpc] [<ffffffff9265e577>] kernel_sendmsg+0x37/0x50 [<ffffffffc049a651>] xs_send_kvec+0xb1/0xc0 [sunrpc] [<ffffffffc049a957>] xs_sendpages+0x67/0x200 [sunrpc] [<ffffffffc049c58f>] xs_tcp_send_request+0xaf/0x250 [sunrpc] [<ffffffffc049866c>] xprt_transmit+0x6c/0x360 [sunrpc] [<ffffffffc0494080>] call_transmit+0x1d0/0x2c0 [sunrpc] [<ffffffffc0493eb0>] ? call_decode+0x880/0x880 [sunrpc] [<ffffffffc0493eb0>] ? call_decode+0x880/0x880 [sunrpc] [<ffffffffc04a1969>] __rpc_execute+0x99/0x420 [sunrpc] [<ffffffff920cc745>] ? wake_up_bit+0x25/0x30 [<ffffffffc04a4268>] rpc_execute+0x68/0xc0 [sunrpc] [<ffffffffc0492856>] rpc_run_task+0xf6/0x150 [sunrpc] [<ffffffffc080a943>] nfs4_call_sync_sequence+0x63/0xa0 [nfsv4] [<ffffffffc080b878>] _nfs4_proc_statfs+0xc8/0xf0 [nfsv4] [<ffffffffc08176a6>] nfs4_proc_statfs+0x66/0xa0 [nfsv4] [<ffffffffc079c62e>] nfs_statfs+0x6e/0x1a0 [nfs] [<ffffffff922588ba>] ? __check_object_size+0x1ca/0x250 [<ffffffff92294337>] statfs_by_dentry+0xa7/0x140 [<ffffffff922943eb>] vfs_statfs+0x1b/0xc0 [<ffffffff922946c5>] user_statfs+0x55/0xb0 [<ffffffff92294747>] SYSC_statfs+0x27/0x60 [<ffffffff9229492e>] SyS_statfs+0xe/0x20 [<ffffffff927c539a>] system_call_fastpath+0x25/0x2a Code: 47 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 <8b> 17 85 d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 cc
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kubelet:2325] Modules linked in: ipt_rpfilter xt_set iptable_raw ip_set_hash_ip ip_set_hash_net ip_set ipip tunnel4 ip_tunnel xt_multiport veth xt_statistic xt_recent xt_nat rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev sg coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper pcspkr joydev cryptd vmw_balloon parport_pc parport i2c_piix4 vmw_vmci nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper serio_raw syscopyarea sysfillrect ahci sysimgblt fb_sys_fops mptspi ttm libahci scsi_transport_spi nfit drm e1000 ata_piix libata mptscsih libnvdimm mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse CPU: 3 PID: 2325 Comm: kubelet Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015 task: ffff8c53d534b180 ti: ffff8c53f3edc000 task.ti: ffff8c53f3edc000 RIP: 0010:[<ffffffff873174cd>] [<ffffffff873174cd>] native_queued_spin_lock_slowpath+0x1d/0x200 RSP: 0018:ffff8c53f3edf730 EFLAGS: 00000293 RAX: 0000000000000001 RBX: ffff8c53f3edf7a0 RCX: 0000000000000001 RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffff8c546f6b0ff0 RBP: ffff8c53f3edf730 R08: 0000000000000000 R09: 0000000000000000 R10: ffff8c53f3edfa10 R11: 061c144823b8e5fc R12: ffff8c53f3edf6b0 R13: ffff8c545156e300 R14: ffff8c546f8e4000 R15: ffffffff875b4a00 FS: 00007f7d02dea700(0000) GS:ffff8c54796c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000c420155010 CR3: 00000000b107e000 CR4: 00000000003607e0 Call Trace: [<ffffffff8797b37b>] queued_spin_lock_slowpath+0xb/0xf [<ffffffff87989ad3>] _raw_spin_lock_bh+0x33/0x40 [<ffffffff878385e0>] lock_sock_nested+0x20/0x50 [<ffffffff878b0dab>] tcp_sendmsg+0x3b/0xc60 [<ffffffff872e54c1>] ? check_preempt_wakeup+0x101/0x220 [<ffffffff872d7202>] ? check_preempt_curr+0x92/0xa0 [<ffffffff878dd589>] inet_sendmsg+0x69/0xb0 [<ffffffff878343a6>] sock_sendmsg+0xb6/0xf0 [<ffffffff872c6d3b>] ? autoremove_wake_function+0x2b/0x40 [<ffffffff872e4306>] ? update_curr+0x86/0x1e0 [<ffffffff872d2fd2>] ? __wake_up_common+0x82/0x120 [<ffffffff87834417>] kernel_sendmsg+0x37/0x50 [<ffffffffc067a019>] xs_send_kvec+0xa9/0xb0 [sunrpc] [<ffffffffc067a307>] xs_sendpages+0x67/0x200 [sunrpc] [<ffffffffc067bcf1>] xs_tcp_send_request+0xa1/0x230 [sunrpc] [<ffffffffc067838c>] xprt_transmit+0x6c/0x360 [sunrpc] [<ffffffffc0673fb0>] call_transmit+0x1d0/0x2c0 [sunrpc] [<ffffffffc0673de0>] ? call_decode+0x880/0x880 [sunrpc] [<ffffffffc0673de0>] ? call_decode+0x880/0x880 [sunrpc] [<ffffffffc06811c9>] __rpc_execute+0x99/0x420 [sunrpc] [<ffffffff872c6cd5>] ? wake_up_bit+0x25/0x30 [<ffffffffc0683a48>] rpc_execute+0x68/0xc0 [sunrpc] [<ffffffffc06727e6>] rpc_run_task+0xf6/0x150 [sunrpc] [<ffffffffc082ba23>] nfs4_call_sync_sequence+0x63/0xa0 [nfsv4] [<ffffffffc082c9a8>] _nfs4_proc_statfs+0xc8/0xf0 [nfsv4] [<ffffffffc0838196>] nfs4_proc_statfs+0x66/0xa0 [nfsv4] [<ffffffffc07f62de>] nfs_statfs+0x6e/0x190 [nfs] [<ffffffff87484b77>] statfs_by_dentry+0xa7/0x140 [<ffffffff87484c2b>] vfs_statfs+0x1b/0xc0 [<ffffffff87484f05>] user_statfs+0x55/0xa0 [<ffffffff87484f77>] SYSC_statfs+0x27/0x60 [<ffffffff8748517e>] SyS_statfs+0xe/0x10 [<ffffffff87993f92>] system_call_fastpath+0x25/0x2a Code: 47 fe ff ff 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 66 90 b9 01 00 00 00 8b 17 85 d2 74 0d 83 fa 03 74 08 f3 90 <8b> 17 85 d2 75 f3 89 d0 f0 0f b1 0f 39 c2 75 e3 5d 66 90 c3 0f [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:37:crtc-0] flip_done timed out [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:33:plane-0] flip_done timed out