| Operating system | Count |
|---|---|
| CentOS 7 | 71 |
| Architecture | Count |
|---|---|
| x86_64 | 71 |
| Related packages | Count |
|---|---|
| kernel | 71 |
Packages names with count less than the total count of reports are most probably not the cause of the problem.
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
dev_gro_receive
|
vmlinux |
0xffffffffb844e898
Build id: 3.10.0-1062.4.1.el7.x86_64 |
0x1f8 |
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u48:4:31518] Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13207(E) falcon_lsm_pinned_13109(E) falcon_lsm_pinned_13005(E) nls_utf8 isofs fuse rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache devlink binfmt_misc falcon_lsm_pinned_12905(E) ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_hda_codec_generic lrw gf128mul glue_helper ablk_helper cryptd snd_hda_intel joydev snd_hda_codec virtio_balloon snd_hda_core sg snd_hwdep snd_seq parport_pc snd_seq_device parport snd_pcm snd_timer snd pcspkr i2c_piix4 soundcore auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw crct10dif_pclmul crct10dif_common virtio_pci crc32c_intel floppy virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_kal] CPU: 0 PID: 31518 Comm: kworker/u48:4 Kdump: loaded Tainted: P W EL ------------ 3.10.0-1062.4.1.el7.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Workqueue: writeback bdi_writeback_workfn task: ffffa0d509388000 ti: ffffa0cc2b224000 task.ti: ffffa0cc2b224000 RIP: 0010:[<ffffffffb844e898>] [<ffffffffb844e898>] dev_gro_receive+0x1f8/0x470 RSP: 0018:ffffa0d71f003db0 EFLAGS: 00010206 RAX: ffffffffb8b67f60 RBX: ffffffffb8b67f80 RCX: 0000000000000040 RDX: 0000000000000074 RSI: ffffa0a88fd0df00 RDI: 000000000000000e RBP: ffffa0d71f003df8 R08: ffffa0cbb3beba00 R09: 000000000000000c R10: ffffa0d71f01f120 R11: fffffbef833f4300 R12: ffffa0d71f003d28 R13: ffffffffb858cefa R14: ffffa0d71f003df8 R15: ffffffffb8b67f80 FS: 0000000000000000(0000) GS:ffffa0d71f000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000027f3d000 CR3: 0000002f19f60000 CR4: 00000000001606f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
dev_gro_receive
|
vmlinux |
0xffffffff81587bf7
Build id: 3.10.0-693.el7.x86_64 |
0x47 |
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 32s! [vsftpd:14630] Modules linked in: vxlan ip6_udp_tunnel udp_tunnel iptable_mangle xt_mark veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vmw_vsock_vmci_transport vsock vboxdrv(OE) sb_edac edac_core coretemp iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd nfit ppdev pcspkr vmw_balloon joydev libnvdimm sg parport_pc parport shpchp vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod vmwgfx crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul libata crct10dif_common crc32c_intel serio_raw vmxnet3 i2c_core vmw_pvscsi floppy dm_mirror dm_region_hash dm_log dm_mod CPU: 3 PID: 14630 Comm: vsftpd Tainted: G OEL ------------ T 3.10.0-693.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016 task: ffff88081e07af70 ti: ffff8805b075c000 task.ti: ffff8805b075c000 RIP: 0010:[<ffffffff81587bf7>] [<ffffffff81587bf7>] dev_gro_receive+0x47/0x3d0 NMI watchdog: BUG: soft lockup - CPU#2 stuck for 35s! [kworker/2:1:12115] RSP: 0018:ffff88081fcc3d30 EFLAGS: 00010286 RAX: ffff8805746042c0 RBX: 0000000000000008 RCX: ffff88101d76c000 RDX: 0000000000000082 RSI: ffff8805b47ca500 RDI: ffff88101d76d460 RBP: ffff88081fcc3d78 R08: ffff880574603c80 R09: 0000000000000000 R10: ffff88081ca97af0 R11: ffff880817321068 R12: ffff88081fcc3ca8 R13: ffffffff816b5c1d R14: ffff88081fcc3d78 R15: ffff8805b47ca500 FS: 00007fc6b21ea880(0000) GS:ffff88081fcc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000ce205210 CR3: 00000005ae2f8000 CR4: 00000000000407e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Stack: Modules linked in: vxlan ip6_udp_tunnel udp_tunnel iptable_mangle xt_mark veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink 0000000000000780 0000000000000202 ffff88057474c400 00000000951a77c9 ffff88101d76d538 ffff8805b47ca500 ffff88101d76d460 ffff8805b47ca500 ffff88081afce430 ffff88081fcc3da0 ffffffff81588278 ffff88101d76d538 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
dev_gro_receive
|
vmlinux |
0xffffffffb3c5089c
Build id: 3.10.0-1062.9.1.el7.x86_64 |
0x1fc |
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [zabbix_agentd:14522] Modules linked in: veth binfmt_misc xt_nat xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink vmw_vsock_vmci_transport vsock ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) sunrpc ppdev vmw_balloon sg edac_mce_amd pcspkr joydev parport_pc i2c_piix4 parport pcc_cpufreq vmw_vmci sch_fq_codel ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix e1000 serio_raw libata mptspi scsi_transport_spi mptscsih mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler] CPU: 0 PID: 14522 Comm: zabbix_agentd Kdump: loaded Tainted: G L ------------ T 3.10.0-1062.9.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014 task: ffff9e4032510000 ti: ffff9e4019328000 task.ti: ffff9e4019328000 RIP: 0010:[<ffffffffb3c5089c>] [<ffffffffb3c5089c>] dev_gro_receive+0x1fc/0x470 RSP: 0018:ffff9e403fc03cc8 EFLAGS: 00000296 RAX: ffffffffb43690c0 RBX: ffffffffb4366ea0 RCX: 0000000000000040 RDX: 000000000000007c RSI: ffff9e3ff5ff5b00 RDI: 000000000000000e RBP: ffff9e403fc03d10 R08: ffff9e3e2ecb30c0 R09: 000000022ce52640 R10: 0000000000000004 R11: 0000000000000005 R12: ffff9e403fc03c38 R13: ffffffffb3d8eefa R14: ffff9e403fc03d10 R15: ffffffffb43690e0 FS: 00007f8cf0e58840(0000) GS:ffff9e403fc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00000000004ec3ea CR3: 0000000210334000 CR4: 00000000000007f0 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
dev_gro_receive
|
vmlinux |
0xffffffff815910e7
Build id: 3.10.0-693.21.1.el7.x86_64 |
0x217 |
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 24s! [swapper/4:0] Modules linked in: iptable_filter fuse binfmt_misc vmw_vsock_vmci_transport vsock sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev ppdev vmw_balloon sg parport_pc vmw_vmci parport shpchp i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm crct10dif_pclmul libata mptspi crct10dif_common crc32c_intel scsi_transport_spi serio_raw mptscsih e1000 mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod CPU: 4 PID: 0 Comm: swapper/4 Tainted: G W L ------------ 3.10.0-693.21.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014 task: ffff880175bfcf10 ti: ffff880175414000 task.ti: ffff880175414000 RIP: 0010:[<ffffffff815910e7>] [<ffffffff815910e7>] dev_gro_receive+0x217/0x3e0 RSP: 0018:ffff88073fd03c68 EFLAGS: 00010202 RAX: ffffffff81b39520 RBX: ffff88073fd13968 RCX: ffff88071f942000 RDX: 000000000000009c RSI: ffff88071caa1200 RDI: ffff88071f942b70 RBP: ffff88073fd03cb0 R08: ffff880160ac1100 R09: 11800000d60b4e00 R10: 004500088801b356 R11: 5000ffffffffffff R12: ffff88073fd03bd8 R13: ffffffff816c1732 R14: ffff88073fd03cb0 R15: ffff88071caa1200 FS: 0000000000000000(0000) GS:ffff88073fd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f48a86dfa70 CR3: 00000006d1f4a000 CR4: 00000000003607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
dev_gro_receive
|
vmlinux |
0xffffffff8363a29e
Build id: 3.10.0-957.5.1.el7.x86_64 |
0x25e |
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 34s! [xfsaild/dm-0:2416] Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables gsch(OE) redirfs(OE) dsa_filter(POE) tun devlink rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi floppy vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm vmxnet3 ata_piix libata drm mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables] CPU: 3 PID: 2416 Comm: xfsaild/dm-0 Kdump: loaded Tainted: P OEL ------------ 3.10.0-957.5.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 task: ffff8c3875fc30c0 ti: ffff8c3875288000 task.ti: ffff8c3875288000 RIP: 0010:[<ffffffff8363a29e>] [<ffffffff8363a29e>] dev_gro_receive+0x25e/0x470 RSP: 0018:ffff8c397fd83d68 EFLAGS: 00010246 RAX: ffffffff83d63f80 RBX: ffff8c3875148048 RCX: ffff8c3945cae0c0 RDX: 0000000000000000 RSI: 0000000000000001 RDI: 000000000000000e RBP: ffff8c397fd83db0 R08: ffff8c3945cada80 R09: 0000000000000000 R10: ffff8c38f0441a58 R11: ffff8c3973b988c0 R12: ffff8c397fd83cd8 R13: ffffffff83775df2 R14: ffff8c397fd83db0 R15: ffffffff83d63fa0 FS: 0000000000000000(0000) GS:ffff8c397fd80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f8d0a5089e5 CR3: 0000000005a6c000 CR4: 00000000003607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: