| Operating system | Count |
|---|---|
| CentOS 7 | 44 |
| Architecture | Count |
|---|---|
| x86_64 | 44 |
| Related packages | Count |
|---|---|
| kernel | 44 |
Packages names with count less than the total count of reports are most probably not the cause of the problem.
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 25s! [k3s-server:61818] NMI watchdog: BUG: soft lockup - CPU#15 stuck for 21s! [kworker/15:1:107317] Modules linked in: xt_statistic tcp_lp nls_utf8 isofs ip6t_MASQUERADE nf_nat_masquerade_ipv6 vxlan ip6_udp_tunnel udp_tunnel ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle xt_physdev xt_NFLOG nfnetlink_log xt_limit xt_mark xt_multiport xt_comment ip_set veth xt_nat nf_conntrack_netlink xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache overlay(T) br_netfilter bridge stp llc vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 iosf_mbi snd_rawmidi crc32_pclmul snd_ac97_codec ppdev ghash_clmulni_intel vmw_balloon aesni_intel ac97_bus lrw gf128mul snd_seq glue_helper ablk_helper cryptd snd_seq_device snd_pcm joydev pcspkr snd_timer snd sg vmw_vmci soundcore i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi nfit crct10dif_pclmul crct10dif_common crc32c_intel scsi_transport_spi libnvdimm mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse CPU: 15 PID: 107317 Comm: kworker/15:1 Kdump: loaded Tainted: G W L ------------ T 3.10.0-1160.118.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020 Workqueue: events_freezable vmballoon_work [vmw_balloon] task: ffff9b7291a8e300 ti: ffff9b76e17ac000 task.ti: ffff9b76e17ac000 NMI watchdog: BUG: soft lockup - CPU#4 stuck for 21s! [kworker/4:0:100916] RIP: 0010:[<ffffffffc05b1c15>] [<ffffffffc05b1c15>] vmballoon_work+0xc5/0x70f [vmw_balloon] NMI watchdog: BUG: soft lockup - CPU#7 stuck for 21s! [java:19228] RSP: 0018:ffff9b76e17afdd0 EFLAGS: 00000246 RAX: 0000000000000000 RBX: ffffffff920e8699 RCX: 0000000000000001 RDX: 0000000000005670 RSI: 0000000000000000 RDI: ffffffff92c08d00 RBP: ffff9b76e17afe18 R08: ffffffffc05b4560 R09: 0000fe92aafb4558 R10: 0000fe92aafb4558 R11: ffff9b729646ff00 R12: ffffffff920ec651 R13: ffff9b76e17afd80 R14: ffffffff920eb01c R15: ffff9b76e17afd78 FS: 0000000000000000(0000) GS:ffff9b775d9c0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000c000349000 CR3: 000000034e392000 CR4: 00000000007607e0 PKRU: 00000000 Call Trace: [<ffffffff920c32ef>] process_one_work+0x17f/0x440 [<ffffffff920c4436>] worker_thread+0x126/0x3c0
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 45s! [kworker/0:0:20127] Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc vmw_vsock_vmci_transport vsock sunrpc sb_edac ext4 coretemp mbcache jbd2 iosf_mbi crc32_pclmul squashfs loop ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon joydev pcspkr i2c_piix4 sg vmw_vmci parport_pc parport ip_tables xfs libcrc32c vmwgfx sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi libata mptscsih mptbase serio_raw crct10dif_pclmul crct10dif_common crc32c_intel vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse CPU: 0 PID: 20127 Comm: kworker/0:0 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.76.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 Workqueue: events_freezable vmballoon_work [vmw_balloon] task: ffff93fd5bd36300 ti: ffff93f81cd68000 task.ti: ffff93f81cd68000 RIP: 0010:[<ffffffffc046fbed>] [<ffffffffc046fbed>] vmballoon_work+0xbd/0x6ff [vmw_balloon] RSP: 0018:ffff93f81cd6bdd0 EFLAGS: 00000246 RAX: 0000000000000000 RBX: 0000000000000086 RCX: 0000000000000001 RDX: 0000000000005670 RSI: 0000000000000000 RDI: ffffffff81c08d00 RBP: ffff93f81cd6be18 R08: ffffffffc0472560 R09: 0000fee32ae72558 R10: 0000fee32ae72558 R11: 0000000000000005 R12: ffffffff810aeefc R13: ffff93f81cd6bd88 R14: 0000000000000086 R15: ffffffff81c9a420 FS: 0000000000000000(0000) GS:ffff93fd7fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f1070263003 CR3: 000000081a44e000 CR4: 00000000000607f0 Call Trace: [<ffffffff810bdfbf>] process_one_work+0x17f/0x440
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
vmballoon_work
|
vmw_balloon |
0xffffffffc0489bed
Build id: 3.10.0-957.21.3.el7.x86_64 |
0xbd |
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [kworker/0:2:24077] Modules linked in: tcp_diag inet_diag vmw_vsock_vmci_transport vsock coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg pcspkr joydev vmw_balloon parport_pc parport vmw_vmci i2c_piix4 nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables xfs libcrc32c vmwgfx drm_kms_helper sr_mod cdrom syscopyarea ata_generic sysfillrect pata_acpi sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm ata_piix crct10dif_pclmul libata crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod CPU: 0 PID: 24077 Comm: kworker/0:2 Kdump: loaded Tainted: G L ------------ 3.10.0-957.21.3.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 10/22/2013 Workqueue: events_freezable vmballoon_work [vmw_balloon] task: ffff8f526ffcc100 ti: ffff8f54293e4000 task.ti: ffff8f54293e4000 RIP: 0010:[<ffffffffc0489bed>] [<ffffffffc0489bed>] vmballoon_work+0xbd/0x6ff [vmw_balloon]
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
vmballoon_work
|
vmw_balloon |
0xffffffffc0582bed
Build id: 3.10.0-1160.36.2.el7.x86_64 |
0xbd |
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 59s! [sh:23893]
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 59s! [kworker/3:1:24855]
Modules linked in: nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c xt_multiport iptable_filter nfsv3 nfs_acl nfs lockd grace fscache vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul pcspkr joydev ppdev glue_helper vmw_balloon ablk_helper cryptd sg parport_pc parport vmw_vmci i2c_piix4 ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ata_piix libahci drm mptspi scsi_transport_spi libata mptscsih crct10dif_pclmul crct10dif_common crc32c_intel serio_raw mptbase vmxnet3 drm_panel_orientation_quirks fuse
CPU: 3 PID: 24855 Comm: kworker/3:1 Tainted: G W L ------------ 3.10.0-1160.36.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
Workqueue: events_freezable vmballoon_work [vmw_balloon]
task: ffff94940159a100 ti: ffff94950ff0c000 task.ti: ffff94950ff0c000
INFO: rcu_sched detected stalls on CPUs/tasks: { 0
RIP: 0010:[<ffffffffc0582bed>]
[<ffffffffc0582bed>] vmballoon_work+0xbd/0x6ff [vmw_balloon]
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
vmballoon_work
|
vmw_balloon |
0xffffffffc0585bed
Build id: 3.10.0-957.el7.x86_64 |
0xbd |
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 27s! [kworker/2:0:115387] Modules linked in: veth xt_nat xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) overlay(T) sunrpc snd_seq_midi snd_seq_midi_event coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi ppdev snd_ac97_codec ac97_bus snd_seq snd_seq_device vmw_balloon snd_pcm snd_timer snd soundcore pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm e1000 mptspi scsi_transport_spi mptscsih mptbase ata_piix nfit libata libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod CPU: 2 PID: 115387 Comm: kworker/2:0 Kdump: loaded Tainted: G OEL ------------ T 3.10.0-957.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015 Workqueue: events_freezable vmballoon_work [vmw_balloon] task: ffff8e5fbf7b4100 ti: ffff8e5f61b48000 task.ti: ffff8e5f61b48000 RIP: 0010:[<ffffffffc0585bed>] [<ffffffffc0585bed>] vmballoon_work+0xbd/0x6ff [vmw_balloon] RSP: 0000:ffff8e5f61b4bdd0 EFLAGS: 00000246 RAX: 0000000000000000 RBX: ffff8e5f61b4bda0 RCX: 0000000000000001 NMI watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [nexus_QuartzSch:24096] RDX: 0000000000005670 RSI: 0000000000000000 RDI: ffffffffbda08cc0 RBP: ffff8e5f61b4be18 R08: ffffffffc0588560 R09: 0000fe9daaf88558 R10: 0000fe9daaf88558 R11: 0000000000000000 R12: 0000000000000001 R13: ffff8e5ff589ab80 R14: ffff8e5ff589ac00 R15: ffff8e5fbf7b4168 FS: 0000000000000000(0000) GS:ffff8e5ff5880000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000003b46d78 CR3: 0000000114810000 CR4: 00000000003607e0 Modules linked in: veth xt_nat xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) overlay(T) sunrpc snd_seq_midi snd_seq_midi_event coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi ppdev snd_ac97_codec ac97_bus snd_seq snd_seq_device vmw_balloon snd_pcm snd_timer snd soundcore pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_pclmul Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
vmballoon_work
|
vmw_balloon |
0xffffffffc0568bed
Build id: 3.10.0-957.10.1.el7.x86_64 |
0xbd |
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 473s! [kworker/2:0:20]
Modules linked in: binfmt_misc vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix vmxnet3 libata vmw_pvscsi drm_panel_orientation_quirks nfit libnvdimm floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: cvblk]
CPU: 2 PID: 20 Comm: kworker/2:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-957.10.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: events_freezable vmballoon_work [vmw_balloon]
task: ffff8dd2b9896180 ti: ffff8dd2b98c0000 task.ti: ffff8dd2b98c0000
RIP: 0010:[<ffffffffc0568bed>] [<ffffffffc0568bed>] vmballoon_work+0xbd/0x6ff [vmw_balloon]
RSP: 0018:ffff8dd2b98c3dd0 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8dd2b98c3da0 RCX: 0000000000000001
RDX: 0000000000005670 RSI: 0000000000000000 RDI: ffffffffafa08cc0
RBP: ffff8dd2b98c3e18 R08: ffffffffc056b560 R09: 0000fea4eaf6b558
R10: 0000fea4eaf6b558 R11: 0000000000000000 R12: 0000000000000001
R13: ffff8dd56d69ab80 R14: ffff8dd56d69ac00 R15: ffff8dd2b98961e8
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 239s! [vmtoolsd:9967]
FS: 0000000000000000(0000) GS:ffff8dd56d680000(0000) knlGS:0000000000000000
INFO: rcu_sched detected stalls on CPUs/tasks: {
Modules linked in:
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb8e306c000 CR3: 000000040f326000 CR4: 00000000007607e0
}
(detected by 0, t=511343 jiffies, g=195381, c=195380, q=4147)
All QSes seen, last rcu_sched kthread activity 451341 (4306840479-4306389138), jiffies_till_next_fqs=3
binfmt_misc vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel
swapper/0 R running task
0 0 0 0x00000008
Call Trace:
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 25s! [systemd-journal:597] ata_generic pata_acpi [<ffffffff816c0715>] system_call_fastpath+0x1c/0x21 [<ffffffffc04a0fe8>] vmballoon_work+0x3c8/0x63d [vmw_balloon]