| Operating system | Count |
|---|---|
| CentOS 7 | 18 |
| Architecture | Count |
|---|---|
| x86_64 | 18 |
| Related packages | Count |
|---|---|
| kernel | 18 |
Packages names with count less than the total count of reports are most probably not the cause of the problem.
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
avtab_search_node
|
vmlinux |
0xffffffffb5b2f3a3
Build id: 3.10.0-1160.119.1.el7.x86_64 |
0x53 |
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 3160s! [gmain:30745] Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink vboxsf(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc snd_intel8x0 snd_ac97_codec ac97_bus snd_seq ppdev snd_seq_device snd_pcm sg joydev snd_timer vboxguest(OE) pcspkr i2c_piix4 snd parport_pc parport video soundcore ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ata_piix drm libahci crc32c_intel e1000 serio_raw libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse CPU: 1 PID: 30745 Comm: gmain Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.119.1.el7.x86_64 #1 Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 task: ffff9369dc40e300 ti: ffff936a103f0000 task.ti: ffff936a103f0000 RIP: 0010:[<ffffffffb5b2f3a3>] [<ffffffffb5b2f3a3>] avtab_search_node+0x53/0xa0 02:12:43.409501 timesync vgsvcTimeSyncWorker: Radical host time change: 3 394 742 000 000ns (HostNow=1 782 526 366 445 000 000 ns HostLast=1 782 522 971 703 000 000 ns) NMI watchdog: BUG: soft lockup - CPU#3 stuck for 3160s! [kglobalaccel:2943] Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink vboxsf(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc snd_intel8x0 snd_ac97_codec ac97_bus snd_seq ppdev snd_seq_device snd_pcm sg joydev snd_timer vboxguest(OE) pcspkr i2c_piix4 snd parport_pc parport video soundcore ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci ata_piix drm libahci crc32c_intel e1000 serio_raw libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse CPU: 3 PID: 2943 Comm: kglobalaccel Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.119.1.el7.x86_64 #1 Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006 task: ffff9369f54f5280 ti: ffff936a551d8000 task.ti: ffff936a551d8000 RIP: 0010:[<ffffffffb58a953a>] [<ffffffffb58a953a>] __do_softirq+0x9a/0x290 RSP: 0000:ffff936adfd83f00 EFLAGS: 00010206 RSP: 0018:ffff936a103f39e8 EFLAGS: 00010206 RAX: ffff936a551dbfd8 RBX: ffff936adfd83ec0 RCX: 0000000000000003 RDX: 0000000129ed236c RSI: 0000000000000000 RDI: ffff9369f54f5280 RBP: ffff936adfd83f60 R08: 0000000000000000 R09: ffffffffb5a72550 R10: ffffffffb6a8a4b0 R11: 0000000000007ffe R12: ffff936adfd83e78 R13: ffffffffb5fc63f2 R14: ffff936adfd83f60 R15: ffff936a551dbfd8 FS: 00002b59470328c0(0000) GS:ffff936adfd80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f7c1015a000 CR3: 00000000951d6000 CR4: 00000000000206e0 RAX: ffff936a83b180c0 RBX: ffff9369dc40e998 RCX: 0000000000000120 RDX: 000000000000090b RSI: 0000000000000007 RDI: ffffffffb6a7d708 RBP: ffff936a103f39e8 R08: 0000000000000707 R09: ffff936a8b020000 R10: 0000000000000487 R11: 0000000000000003 R12: ffff936adfc80000 R13: ffffffffb582c671 R14: ffff936a103f3998 R15: ffff936a103f39a8 FS: 00007fe678d9c700(0000) GS:ffff936adfc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000412515 CR3: 0000000020452000 CR4: 00000000000206e0 02:12:43.414261 timesync vgsvcTimeSyncWorker: Radical guest time change: 3 393 061 515 000ns (GuestNow=1 782 526 363 409 485 000 ns GuestLast=1 782 522 970 347 970 000 ns fSetTimeLastLoop=false) Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
avtab_search_node
|
vmlinux |
0xffffffffb351bd25
Build id: 3.10.0-1160.62.1.el7.x86_64 |
0x55 |
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 44s! [systemd-journal:588] Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13109(E) tcp_lp falcon_lsm_pinned_12905(E) nfsv3 nfs_acl nfs lockd grace fscache xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd pcspkr joydev sg parport_pc vmw_vmci parport i2c_piix4 binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper ahci crct10dif_pclmul crct10dif_common syscopyarea sysfillrect ata_piix libahci sysimgblt fb_sys_fops crc32c_intel ttm serio_raw libata drm vmxnet3 nfit libnvdimm vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal] CPU: 1 PID: 588 Comm: systemd-journal Kdump: loaded Tainted: P EL ------------ 3.10.0-1160.62.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020 task: ffff918c328cb180 ti: ffff918bbba2c000 task.ti: ffff918bbba2c000 RIP: 0010:[<ffffffffb351bd25>] [<ffffffffb351bd25>] avtab_search_node+0x55/0x90 RSP: 0018:ffff918bbba2f9d0 EFLAGS: 00000206 RAX: ffff918bb65bf828 RBX: 000000000000090b RCX: 0000000000000cdc RDX: 000000000000090a RSI: 0000000000000007 RDI: ffffffffb444c6c8 RBP: ffff918bbba2f9d0 R08: 0000000000000707 R09: ffff918b7b594000 R10: 0000000000003377 R11: 0000000000000003 R12: 0000000000000707 R13: ffff918b99664000 R14: 0000000000002433 R15: 0000000000000003 FS: 00007f6834939880(0000) GS:ffff918c3fd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 NMI watchdog: BUG: soft lockup - CPU#0 stuck for 44s! [kworker/0:1H:737] CR2: 00007fabe50b1000 CR3: 00000000bb4cc000 CR4: 00000000000607e0 Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13109(E) tcp_lp falcon_lsm_pinned_12905(E) nfsv3 nfs_acl nfs lockd grace fscache xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd pcspkr joydev sg parport_pc vmw_vmci parport i2c_piix4 binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
avtab_search_node
|
vmlinux |
0xffffffffa1f1a7b0
Build id: 3.10.0-1127.8.2.el7.x86_64 |
0x50 |
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [master:1309] Modules linked in: fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock coretemp crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul crct10dif_common crc32c_intel mptspi serio_raw scsi_transport_spi mptscsih mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler] CPU: 3 PID: 1309 Comm: master Kdump: loaded Tainted: G L ------------ 3.10.0-1127.8.2.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014 task: ffffa0beaec8d230 ti: ffffa0beac3a0000 task.ti: ffffa0beac3a0000 RIP: 0010:[<ffffffffa1f1a7b0>] [<ffffffffa1f1a7b0>] avtab_search_node+0x50/0x90 RSP: 0000:ffffa0beac3a39c8 EFLAGS: 00010286 RAX: ffffa0be8a8a15e8 RBX: 0000000000000141 RCX: 0000000000000148 RDX: 0000000000000cdc RSI: 0000000000000018 RDI: ffffffffa2e4b688 RBP: ffffa0beac3a39c8 R08: 0000000000000707 R09: ffffa0be81a14000 R10: 0000000000000538 R11: 0000000000000000 R12: 0000000000000707 R13: ffffa0bbd98cc000 R14: 000000000000051c R15: 0000000000000000 FS: 00007fed820ce880(0000) GS:ffffa0bebfd80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000002677370 CR3: 000000032c3de000 CR4: 00000000000007e0 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
avtab_search_node
|
vmlinux |
0xffffffffb9f1a7a7
Build id: 3.10.0-1127.13.1.el7.x86_64 |
0x47 |
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 31s! [qmgr:1496] Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic vmwgfx sr_mod cdrom ata_generic drm_kms_helper pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common mptspi scsi_transport_spi mptscsih e1000 mptbase crc32c_intel ata_piix libata serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod CPU: 1 PID: 1496 Comm: qmgr Kdump: loaded Tainted: G W L ------------ 3.10.0-1127.13.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015 task: ffff8b5f7a42a0e0 ti: ffff8b5f7a6dc000 task.ti: ffff8b5f7a6dc000 RIP: 0010:[<ffffffffb9f1a7a7>] [<ffffffffb9f1a7a7>] avtab_search_node+0x47/0x90 RSP: 0018:ffff8b5f7a6dfbd0 EFLAGS: 00010206 RAX: 000000000000057d RBX: 0000000000000000 RCX: 0000000000000cdc RDX: 0000000000000cd9 RSI: 000000000000000d RDI: ffffffffbae4b6c8 RBP: ffff8b5f7a6dfbd0 R08: 0000000000000707 R09: ffff8b62ecd34000 R10: 000000000000337d R11: 0000000000000246 R12: ffffffffb9e9c508 R13: ffff8b5f7a6dfb80 R14: 0000000000000086 R15: ffff8b5f7a6dfb38 FS: 00007f85f8ef2880(0000) GS:ffff8b62ee640000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fcaaaffcc78 CR3: 000000040cef0000 CR4: 00000000001607e0 Call Trace: