| Operating system | Count |
|---|---|
| CentOS 7 | 53 |
| Architecture | Count |
|---|---|
| x86_64 | 53 |
| Related packages | Count |
|---|---|
| kernel | 53 |
Packages names with count less than the total count of reports are most probably not the cause of the problem.
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
multi_cpu_stop
|
vmlinux |
0xffffffff8111681a
Build id: 3.10.0-693.5.2.el7.x86_64 |
0x4a |
NMI watchdog: BUG: soft lockup - CPU#25 stuck for 22s! [rm:24694] NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [migration/18:101] Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_15402(E) 8021q garp mrp stp llc rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache panfs(POE) falcon_lsm_pinned_15309(E) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) mlx5_fpga_tools(OE) ip6_tables dell_rbu iptable_filter dcdbas iTCO_wdt iTCO_vendor_support sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_si ipmi_devintf ipmi_msghandler cdc_ether raid0 joydev mei_me mei acpi_pad pcspkr acpi_power_meter usbnet sg mii shpchp lpc_ich wmi nfsd auth_rpcgss knem(OE) nfs_acl lockd binfmt_misc grace sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic uas usb_storage mgag200 i2c_algo_bit drm_kms_helper crct10dif_pclmul syscopyarea ixgbe crct10dif_common sysfillrect sysimgblt crc32c_intel fb_sys_fops ttm bnx2x drm dca ahci libahci nvme libata nvme_core megaraid_sas i2c_core mdio libcrc32c ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_ib(OE) mlx4_ib(OE) mlx4_en(OE) ib_uverbs(OE) ib_core(OE) mlx5_core(OE) mlxfw(OE) vfio_mdev(OE) vfio_iommu_type1 vfio mdev(OE) ptp pps_core mlx4_core(OE) devlink mlx_compat(OE) [last unloaded: falcon_kal] CPU: 18 PID: 101 Comm: migration/18 Tainted: P OEL ------------ 3.10.0-693.5.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R920/0Y4CNC, BIOS 1.8.0 05/23/2018 task: ffff89209d285ee0 ti: ffff89209d2a8000 task.ti: ffff89209d2a8000 RIP: 0010:[<ffffffff8111681a>] [<ffffffff8111681a>] multi_cpu_stop+0x4a/0x100 RSP: 0000:ffff89209d2abd98 EFLAGS: 00000246 RAX: 0000000000000001 RBX: 0000000000016cc0 RCX: dead000000000200 RDX: ffff89dd3f70ffb0 RSI: 0000000000000286 RDI: ffff896cb530fb70 RBP: ffff89209d2abdc0 R08: ffff896cb530fb10 R09: 0000000000000001 R10: 000000000000b865 R11: 0000000000000001 R12: 0000000000000000 R13: ffff89209d2abdec R14: ffff8921bfc08800 R15: ffff89209d3fcc08 FS: 0000000000000000(0000) GS:ffff89dd3f700000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000003c32840 CR3: 00000000019f2000 CR4: 00000000001407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Stack: ffff89dd3f70ffa8 ffff896cb530fb70 ffff89dd3f70ffa0 ffff896cb530fb98 ffffffff811167d0 ffff89209d2abe90 ffffffff81116ac7 ffff89209d2abfd8 Modules linked in: ffff89dd3f70ffb0 0000000000000000 0000000000000000 ffff89dd3f716cc0 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
multi_cpu_stop
|
vmlinux |
0xffffffff9a935b7a
Build id: 3.10.0-1160.59.1.el7.x86_64 |
0x4a |
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u32:2:25778] dm_mod Modules linked in: fuse veth CPU: 7 PID: 45 Comm: migration/7 Kdump: loaded Tainted: G W L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1 xt_nat Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 CPU: 1 PID: 13 Comm: migration/1 Kdump: loaded Tainted: G W L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1 task: ffff91813b120000 ti: ffff91813b11c000 task.ti: ffff91813b11c000 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 RIP: 0010:[<ffffffff9a935b7a>] [<ffffffff9a935b7a>] multi_cpu_stop+0x4a/0x110
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
multi_cpu_stop
|
vmlinux |
0xffffffff839265ea
Build id: 3.10.0-862.el7.x86_64 |
0x4a |
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [migration/13:75] Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache veth vxlan ip6_udp_tunnel udp_tunnel xt_set ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_bitmap_port ip_set_hash_ipport ip_set dummy ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc symap_custom_3_10_0_862_el7_x86_64(POE) ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink symev_custom_3_10_0_862_el7_x86_64(OE) nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack overlay(T) vmw_vsock_vmci_transport vsock sunrpc xfs libcrc32c ppdev vmw_balloon joydev pcspkr sg nfit shpchp parport_pc i2c_piix4 libnvdimm vmw_vmci parport ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci ata_piix libahci serio_raw libata vmxnet3 i2c_core vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod CPU: 13 PID: 75 Comm: migration/13 Kdump: loaded Tainted: P W OEL ------------ T 3.10.0-862.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018 task: ffff8cf829fb2f70 ti: ffff8cf829fd0000 task.ti: ffff8cf829fd0000 RIP: 0010:[<ffffffff839265ea>] [<ffffffff839265ea>] multi_cpu_stop+0x4a/0x110 RSP: 0000:ffff8cf829fd3d98 EFLAGS: 00000246 RAX: 0000000000000001 RBX: ffff8cf829fd3d20 RCX: dead000000000200 RDX: ffff8d077f254030 RSI: 0000000000000282 RDI: ffff8cfe61717ac0 RBP: ffff8cf829fd3dc0 R08: ffff8cfe61717a90 R09: 0000000000000000 R10: 000000000000b85e R11: 0000000000000001 R12: 000000000000000d R13: 000000000000000e R14: ffff8d077f298b00 R15: 000000000000000e FS: 0000000000000000(0000) GS:ffff8d077f240000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000002b7e270 CR3: 0000001c90c0e000 CR4: 00000000000007e0 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
multi_cpu_stop
|
vmlinux |
0xffffffff89d2fd4a
Build id: 3.10.0-957.el7.x86_64 |
0x4a |
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 36s! [kworker/u40:0:652955] crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm mptspi scsi_transport_spi mptscsih ahci vmxnet3 libahci mptbase drm_panel_orientation_quirks ata_piix libata dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dattobd] Modules linked in: CPU: 2 PID: 19 Comm: migration/2 Kdump: loaded Tainted: G W OEL ------------ 3.10.0-957.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 task: ffff98680ee3d140 ti: ffff98680ee8c000 task.ti: ffff98680ee8c000 dattobd(OE) sfs(OE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 RIP: 0010:[<ffffffff89d2fd4a>] tun [<ffffffff89d2fd4a>] multi_cpu_stop+0x4a/0x110
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
multi_cpu_stop
|
vmlinux |
0xffffffff99935b7a
Build id: 3.10.0-1160.42.2.el7.x86_64 |
0x4a |
NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [migration/19:105]
Modules linked in: qax_tq_base(OE) binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul vmw_balloon glue_helper ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif syscopyarea crct10dif_generic sysfillrect sysimgblt fb_sys_fops ttm ahci drm ata_piix libahci nfit libata libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw
vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: qax_tq_base]
CPU: 19 PID: 105 Comm: migration/19 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.42.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9a31f0f63180 ti: ffff9a31f0f74000 task.ti: ffff9a31f0f74000
RIP: 0010:[<ffffffff99935b7a>] [<ffffffff99935b7a>] multi_cpu_stop+0x4a/0x110
NMI watchdog: BUG: soft lockup - CPU#18 stuck for 29s! [java:12574]
RSP: 0000:ffff9a31f0f77d98 EFLAGS: 00000246
RAX: 0000000000000001 RBX: ffff9a31f0fe1000 RCX: dead000000000200
RDX: ffff9a37674960b0 RSI: 0000000000000286 RDI: ffff9a315de6bb20
RBP: ffff9a31f0f77dc0 R08: ffff9a315de6baf0 R09: 0000000000000001
R10: ffff9a3767480000 R11: 0000000000000000 R12: 0000000000000000
R13: ffff9a31f0f77d70 R14: ffffffff998de305 R15: ffff9a31f0fc0800
FS: 0000000000000000(0000) GS:ffff9a3767480000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f01927c5000 CR3: 000000049d1e4000 CR4: 00000000007607e0
Modules linked in: qax_tq_base(OE) binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE
INFO: rcu_sched detected stalls on CPUs/tasks:
{
nf_nat_masquerade_ipv4
iptable_nat nf_nat_ipv4
14
nf_nat
nf_conntrack_ipv4
}
(detected by 17, t=62219 jiffies, g=198920671, c=198920670, q=4904)
Task dump for CPU 14:
nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge
vmtoolsd R
running task
stp
0 1087 1 0x0000008a
Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
multi_cpu_stop
|
vmlinux |
0xffffffff810f2952
Build id: 3.10.0-229.el7.x86_64 |
0x82 |
BUG: soft lockup - CPU#8 stuck for 23s! [migration/8:83] BUG: soft lockup - CPU#14 stuck for 26s! [migration/14:116] Modules linked in: binfmt_misc coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr vmw_vmci sg i2c_piix4 shpchp parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper ata_piix mptspi ttm scsi_transport_spi crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw mptscsih libata i2c_core mptbase vmxnet3 floppy dm_mirror dm_region_hash dm_log dm_mod CPU: 8 PID: 83 Comm: migration/8 Not tainted 3.10.0-229.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014 task: ffff880cb1858b60 ti: ffff880cb1864000 task.ti: ffff880cb1864000 RIP: 0010:[<ffffffff810f2952>] [<ffffffff810f2952>] multi_cpu_stop+0x82/0xf0