| Operating system | Count |
|---|---|
| CentOS 7 | 388 |
| Architecture | Count |
|---|---|
| x86_64 | 388 |
| Related packages | Count |
|---|---|
| kernel | 381 |
| kernel-lt | 7 |
Packages names with count less than the total count of reports are most probably not the cause of the problem.
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
smp_call_function_many
|
vmlinux |
0xffffffff810f99b6
Build id: 3.10.0-693.5.2.el7.x86_64 |
0x206 |
NMI watchdog: BUG: soft lockup - CPU#49 stuck for 22s! [falcon-sensor:6678] Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(E) falcon_kal(E) falcon_lsm_pinned_17219(E) falcon_lsm_pinned_17129(E) falcon_lsm_pinned_17106(E) falcon_lsm_pinned_17005(E) falcon_lsm_pinned_16903(E) falcon_lsm_pinned_16803(E) falcon_lsm_pinned_16703(E) falcon_lsm_pinned_16604(E) falcon_lsm_pinned_16407(E) falcon_lsm_pinned_16405(E) falcon_lsm_pinned_16404(E) falcon_lsm_pinned_16303(E) falcon_lsm_pinned_16206(E) falcon_lsm_pinned_16108(E) squashfs loop overlay(T) falcon_lsm_pinned_16004(E) falcon_lsm_pinned_15907(E) falcon_lsm_pinned_15805(E) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache panfs(POE) 8021q garp mrp stp llc falcon_lsm_pinned_15705(E) rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) mlx5_fpga_tools(OE) ip6_tables iptable_filter vfat fat kvm irqbypass crc32_pclmul dell_smbios ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sparse_keymap i2c_piix4 dcdbas pcspkr ipmi_si ipmi_devintf sg ipmi_msghandler tpm_crb shpchp acpi_power_meter wmi mlx5_ib(OE) mlx4_ib(OE) ib_uverbs(OE) mlx4_en(OE) mlx5_core(OE) mlxfw(OE) vfio_mdev(OE) vfio_iommu_type1 vfio mdev(OE) nfsd auth_rpcgss nfs_acl lockd grace sunrpc mlx4_core(OE) knem(OE) binfmt_misc devlink ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci crct10dif_pclmul libahci crct10dif_common tg3 crc32c_intel libata ptp megaraid_sas i2c_core pps_core ib_ipoib(OE) ib_cm(OE) ib_umad(OE) ib_core(OE) mlx_compat(OE) [last unloaded: falcon_kal] CPU: 49 PID: 6678 Comm: falcon-sensor Tainted: P W OEL ------------ T 3.10.0-693.5.2.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R6515/0J91V2, BIOS 2.11.4 03/22/2023 task: ffff883f78506eb0 ti: ffff881bf75d0000 task.ti: ffff881bf75d0000 RIP: 0010:[<ffffffff810f99b6>] [<ffffffff810f99b6>] smp_call_function_many+0x206/0x260 RSP: 0000:ffff881bf75d3c50 EFLAGS: 00000202 RAX: 0000000000000000 RBX: ffffffff8106e4a0 RCX: ffff883f7d81b510 RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000000 RBP: ffff881bf75d3c88 R08: ffff880126750c00 R09: ffffffff813228a9 R10: ffff883f7e459c80 R11: ffffea006e45d200 R12: 0000000000000282 R13: ffff881bf75d3c00 R14: 0000000000000066 R15: 000200da00000000 FS: 00007f10551fd700(0000) GS:ffff883f7e440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f0ffc0075c0 CR3: 0000001c72b11000 CR4: 00000000003406e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Stack: 00000001f75d3ca8 ffff883e270f2f08 ffff883e270f2bc0 00007f0ffc0075c0 0000000000000000 ffff883e270f2f08 ffff881375e9d038 ffff881bf75d3cd8 ffffffff8106e668 ffff883e270f2bc0 00007f0ffc0075c0 0000000000000000 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
smp_call_function_many
|
vmlinux |
0xffffffffb291519a
Build id: 3.10.0-1062.4.1.el7.x86_64 |
0x20a |
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [BESClient:2819] Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(E) falcon_kal(E) falcon_lsm_pinned_17005(E) veth nf_conntrack_netlink xt_addrtype br_netfilter dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache falcon_lsm_pinned_16903(E) uinput devlink binfmt_misc ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_nat nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter snd_hda_codec_generic iTCO_wdt iTCO_vendor_support qxl snd_hda_intel ttm snd_hda_codec drm_kms_helper snd_hda_core iosf_mbi snd_hwdep kvm_intel syscopyarea snd_seq sysfillrect sysimgblt snd_seq_device kvm fb_sys_fops snd_pcm irqbypass drm crc32_pclmul ghash_clmulni_intel snd_timer snd aesni_intel soundcore lrw drm_panel_orientation_quirks gf128mul lpc_ich i2c_i801 glue_helper ablk_helper joydev virtio_balloon cryptd virtio_rng pcspkr auth_rpcgss sunrpc ip_tables xfs libcrc32c ahci libahci virtio_net virtio_console virtio_blk virtio_scsi libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_kal] NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [BESClient:2811] CPU: 2 PID: 2819 Comm: BESClient Kdump: loaded Tainted: P EL ------------ 3.10.0-1062.4.1.el7.x86_64 #1 Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 task: ffff97477423c1c0 ti: ffff97475ce60000 task.ti: ffff97475ce60000 Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(E) falcon_kal(E) falcon_lsm_pinned_17005(E) veth nf_conntrack_netlink RIP: 0010:[<ffffffffb291519a>] xt_addrtype [<ffffffffb291519a>] smp_call_function_many+0x20a/0x270
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
smp_call_function_many
|
vmlinux |
0xffffffff9e91173a
Build id: 3.10.0-957.el7.x86_64 |
0x20a |
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [kswapd0:106] Modules linked in: binfmt_misc vmw_vsock_vmci_transport vsock sunrpc vfat fat ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper sg cryptd vmw_vmci pcspkr joydev parport_pc parport i2c_piix4 tpm_crb ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic e1000e crct10dif_pclmul crct10dif_common vmwgfx crc32c_intel serio_raw mptsas ptp pps_core scsi_transport_sas drm_kms_helper mptscsih mptbase syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm ata_piix nfit libata libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod CPU: 1 PID: 106 Comm: kswapd0 Kdump: loaded Tainted: G W L ------------ 3.10.0-957.el7.x86_64 #1 Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.17369862.B64.2012240522 12/24/2020 task: ffff9254bf8d5140 ti: ffff92549c7d8000 task.ti: ffff92549c7d8000 RIP: 0010:[<ffffffff9e91173a>] [<ffffffff9e91173a>] smp_call_function_many+0x20a/0x270 RSP: 0018:ffff92549c7db950 EFLAGS: 00000202 RAX: 000000000000000e RBX: 000800009e87a3d0 RCX: ffff92549f5a0490 RDX: 000000000000000e RSI: 0000000000000010 RDI: 0000000000000000 RBP: ffff92549c7db988 R08: ffff924df4bfb400 R09: ffffffff9eb755c9 R10: ffff92549f25f160 R11: fffff143441e8200 R12: 0000000000000286 R13: 0000000000000286 R14: ffff92549c7db900 R15: ffffffff9e856b82 FS: 0000000000000000(0000) GS:ffff92549f240000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 000000000043172c CR3: 00000005bb210000 CR4: 0000000000760fe0 PKRU: 00000000 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
smp_call_function_many
|
vmlinux |
0xffffffff81d1761a
Build id: 3.10.0-1127.19.1.el7.x86_64 |
0x20a |
NMI watchdog: BUG: soft lockup - CPU#60 stuck for 30s! [swapper/60:0] nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci nfit ata_piix libahci drm crct10dif_pclmul libnvdimm crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod Modules linked in: tcp_lp fuse veeamsnap(OE) veth binfmt_misc nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter nls_utf8 cifs ccm dns_resolver overlay(T) vmw_vsock_vmci_transport vsock ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect CPU: 41 PID: 54165 Comm: RemoteInvocatio Kdump: loaded Tainted: G OEL ------------ T 3.10.0-1127.19.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017 task: ffff9bfe2a05d230 ti: ffff9bf09c568000 task.ti: ffff9bf09c568000 sysimgblt fb_sys_fops ttm ahci nfit ata_piix libahci drm crct10dif_pclmul libnvdimm crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod RIP: 0010:[<ffffffff81d1761a>] [<ffffffff81d1761a>] smp_call_function_many+0x20a/0x270
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
smp_call_function_many
|
vmlinux |
0xffffffff810f98e2
Build id: 3.10.0-514.el7.x86_64 |
0x202 |
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:13561] Modules linked in: fuse uinput xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_hda_codec_generic iosf_mbi crc32_pclmul snd_hda_intel ghash_clmulni_intel aesni_intel snd_hda_codec lrw gf128mul glue_helper ablk_helper snd_hda_core snd_hwdep snd_seq ppdev snd_seq_device cryptd snd_pcm snd_timer snd pcspkr parport_pc virtio_balloon parport sg i2c_piix4 soundcore nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_blk virtio_net qxl drm_kms_helper crct10dif_pclmul syscopyarea crct10dif_common sysfillrect crc32c_intel sysimgblt ata_piix fb_sys_fops serio_raw ttm drm virtio_pci virtio_ring virtio libata i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod CPU: 2 PID: 13561 Comm: kworker/u8:0 Tainted: G L ------------ 3.10.0-514.el7.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Workqueue: writeback bdi_writeback_workfn (flush-253:0) task: ffff8801c1bd5e20 ti: ffff88012fb98000 task.ti: ffff88012fb98000 RIP: 0010:[<ffffffff810f98e2>] [<ffffffff810f98e2>] smp_call_function_many+0x202/0x260
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
smp_call_function_many
|
vmlinux |
0xffffffff82116e4e
Build id: 3.10.0-1160.15.2.el7.x86_64 |
0x20e |
NMI watchdog: BUG: soft lockup - CPU#32 stuck for 23s! [samtools:2488] Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_14203(E) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache uinput devlink binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter falcon_lsm_pinned_14108(E) snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm qxl snd_timer ttm drm_kms_helper iTCO_wdt syscopyarea crc32_pclmul ghash_clmulni_intel aesni_intel lrw sysfillrect gf128mul sysimgblt glue_helper ppdev fb_sys_fops ablk_helper snd iTCO_vendor_support joydev cryptd parport_pc drm virtio_rng pcspkr i2c_i801 lpc_ich soundcore virtio_balloon parport drm_panel_orientation_quirks nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c virtio_net net_failover virtio_console virtio_blk failover crct10dif_pclmul ahci virtio_pci libahci crct10dif_common serio_raw crc32c_intel libata virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal] CPU: 32 PID: 2488 Comm: samtools Kdump: loaded Tainted: P W EL ------------ 3.10.0-1160.15.2.el7.x86_64 #1 Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.13.0-2.module+el8.4.0+534+4680a14e 04/01/2014 task: ffff8b144cb85280 ti: ffff8b1fb5030000 task.ti: ffff8b1fb5030000 RIP: 0010:[<ffffffff82116e4e>] [<ffffffff82116e4e>] smp_call_function_many+0x20e/0x270 NMI watchdog: BUG: soft lockup - CPU#182 stuck for 26s! [python:108243] NMI watchdog: BUG: soft lockup - CPU#201 stuck for 26s! [python:108089] RSP: 0018:ffff8b1fb5033a48 EFLAGS: 00000202 RAX: 00000000000000c9 RBX: 0000000000000010 RCX: ffff8cfbae860910 RDX: 00000000000000c9 RSI: 00000000000000ff RDI: 00000000000000c0 NMI watchdog: BUG: soft lockup - CPU#63 stuck for 23s! [sort:128334] RBP: ffff8b1fb5033a80 R08: ffff8b10a1f40818 R09: ffffffff82386479 Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) R10: ffff8cfbabe1f160 R11: fffff7080751e600 R12: ffffffffffffff03 Modules linked in: falcon_lsm_pinned_14203(E) R13: ffff8b10d479e800 R14: 00000000000000ff R15: 0000017ce3a00000 falcon_lsm_serviceable(PE) rpcsec_gss_krb5 FS: 00007fb6607d3740(0000) GS:ffff8cfbabe00000(0000) knlGS:0000000000000000 falcon_nf_netcontain(PE) nfsv4 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 falcon_kal(E) dns_resolver CR2: 00007f2141d04024 CR3: 000000085f790000 CR4: 0000000000760ee0 falcon_lsm_pinned_14203(E) nfs rpcsec_gss_krb5 PKRU: 55555554 fscache nfsv4 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
smp_call_function_many
|
vmlinux |
0xffffffff810d6d62
Build id: 3.10.0-229.14.1.el7.x86_64 |
0x202 |
BUG: soft lockup - CPU#5 stuck for 23s! [asterisk:95669] Modules linked in: binfmt_misc ebtable_filter ebtables ip6table_filter ip6_tables tun bridge stp llc dahdi(OF) crc_ccitt netatop(OF) xt_CHECKSUM iptable_mangle xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_nat_ipv4 nf_nat nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_recent xt_multiport xt_mark xt_conntrack nf_conntrack iptable_filter ext4 mbcache jbd2 coretemp snd_ens1371 snd_rawmidi snd_ac97_codec crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq snd_seq_device aesni_intel snd_pcm lrw ppdev gf128mul glue_helper snd_timer ablk_helper vmw_balloon cryptd snd ahci pcspkr soundcore sg libahci vmw_vmci i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common ttm crc32c_intel mptspi scsi_transport_spi drm ata_piix serio_raw mptscsih libata e1000 i2c_core mptbase dm_mirror dm_region_hash dm_log dm_mod CPU: 5 PID: 95669 Comm: asterisk Tainted: GF O-------------- 3.10.0-229.14.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015 BUG: soft lockup - CPU#12 stuck for 23s! [gdrive:95717] task: ffff8800b9aaa220 ti: ffff88010f5f4000 task.ti: ffff88010f5f4000 Modules linked in: binfmt_misc ebtable_filter ebtables ip6table_filter ip6_tables tun bridge stp llc dahdi(OF) crc_ccitt netatop(OF) xt_CHECKSUM iptable_mangle xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_nat_ipv4 nf_nat nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_recent xt_multiport xt_mark xt_conntrack nf_conntrack iptable_filter ext4 mbcache jbd2 coretemp snd_ens1371 snd_rawmidi snd_ac97_codec crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq snd_seq_device aesni_intel snd_pcm lrw ppdev gf128mul glue_helper snd_timer ablk_helper vmw_balloon cryptd snd ahci pcspkr soundcore sg libahci vmw_vmci i2c_piix4 RIP: 0010:[<ffffffff810d6d62>] [<ffffffff810d6d62>] smp_call_function_many+0x202/0x260
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
smp_call_function_many
|
vmlinux |
0xffffffffb0f16c0a
Build id: 3.10.0-1160.el7.x86_64 |
0x20a |
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [popen.rb:42:57892] NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [popen.rb:41:57891] Modules linked in: veth br_netfilter tun devlink overlay(T) bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec iosf_mbi crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq snd_seq_device aesni_intel snd_pcm lrw gf128mul ppdev glue_helper ablk_helper cryptd snd_timer vmw_balloon snd soundcore joydev sg pcspkr i2c_piix4 vmw_vmci parport_pc parport xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm nfit libnvdimm drm serio_raw mptspi e1000 scsi_transport_spi ata_piix mptscsih libata mptbase drm_panel_orientation_quirks fuse [last unloaded: ip_tables] CPU: 1 PID: 57891 Comm: popen.rb:41 Kdump: loaded Tainted: G B L ------------ T 3.10.0-1160.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020 task: ffff9c858197d280 ti: ffff9c8484070000 task.ti: ffff9c8484070000 Modules linked in: veth br_netfilter tun RIP: 0010:[<ffffffffb0f16c0a>] devlink [<ffffffffb0f16c0a>] smp_call_function_many+0x20a/0x270
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
smp_call_function_many
|
vmlinux |
0xffffffffb63170da
Build id: 3.10.0-1160.45.1.el7.x86_64 |
0x20a |
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 25s! [dsa.Scheduler_0:28045] Modules linked in: gsch(OE) redirfs(OE) bmhook(OE) tmhook(OE) dsa_filter(POE) dsa_filter_hook(OE) squashfs loop vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg vmw_vmci parport_pc parport i2c_piix4 nfsd nfs_acl lockd binfmt_misc grace auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_piix libnvdimm libata vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse CPU: 1 PID: 28045 Comm: dsa.Scheduler_0 Kdump: loaded Tainted: P OEL ------------ 3.10.0-1160.45.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018 task: ffff8b6cb35dc200 ti: ffff8b6ca7d70000 task.ti: ffff8b6ca7d70000 RIP: 0010:[<ffffffffb63170da>] [<ffffffffb63170da>] smp_call_function_many+0x20a/0x270 NMI watchdog: BUG: soft lockup - CPU#2 stuck for 25s! [wrs.worker_0001:5493] RSP: 0018:ffff8b6ca7d73b58 EFLAGS: 00000202 RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff8b6cffd204d0 RDX: 0000000000000002 RSI: 0000000000000004 RDI: 0000000000000000 RBP: ffff8b6ca7d73b90 R08: ffff8b6cf6b1d800 R09: ffffffffb6586bf9 R10: ffff8b6cffc9f160 R11: ffffec64404ce400 R12: 0000000000000292 R13: ffff8b6ca7d73b08 R14: 0000000000000001 R15: 0000000000000001 FS: 00007f3b3707d700(0000) GS:ffff8b6cffc80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3b314fb008 CR3: 00000000b2d36000 CR4: 00000000000007e0 Modules linked in: gsch(OE) redirfs(OE) bmhook(OE) tmhook(OE) dsa_filter(POE) dsa_filter_hook(OE) squashfs loop vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg vmw_vmci parport_pc parport i2c_piix4 nfsd nfs_acl lockd binfmt_misc grace auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_piix libnvdimm libata vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse CPU: 2 PID: 5493 Comm: wrs.worker_0001 Kdump: loaded Tainted: P OEL ------------ 3.10.0-1160.45.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018 task: ffff8b6be0cd3180 ti: ffff8b6cb3484000 task.ti: ffff8b6cb3484000 RIP: 0010:[<ffffffffb6844d8b>] [<ffffffffb6844d8b>] __build_skb+0xdb/0x1b0 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
smp_call_function_many
|
vmlinux |
0xffffffff97116dda
Build id: 3.10.0-1160.21.1.el7.x86_64 |
0x20a |
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 21s! [titanagent:18095] Modules linked in: binfmt_misc tcp_diag inet_diag nf_conntrack_ftp tq_base(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc nfit libnvdimm ppdev coretemp vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm ata_piix libata crct10dif_pclmul vmxnet3 crct10dif_common crc32c_intel serio_raw vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse CPU: 13 PID: 18095 Comm: titanagent Kdump: loaded Tainted: G W OEL ------------ 3.10.0-1160.21.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 task: ffff8ec36bff3180 ti: ffff8ec36e79c000 task.ti: ffff8ec36e79c000 RIP: 0010:[<ffffffff97116dda>] [<ffffffff97116dda>] smp_call_function_many+0x20a/0x270 RSP: 0000:ffff8ec36e79faa0 EFLAGS: 00000202 RAX: 0000000000000004 RBX: 000800009707e6a0 RCX: ffff8ec2e51208b8 RDX: 0000000000000004 RSI: 0000000000000012 RDI: 0000000000000000 RBP: ffff8ec36e79fad8 R08: ffff8ec613770c00 R09: ffffffff97386419 R10: ffff8ec83fc5f160 R11: ffffd2bb0b71d000 R12: 0000000000000282 R13: 0000000000000282 R14: ffff8ec36e79fa50 R15: ffff8ec36e79fdd0 FS: 00007fe2127f4700(0000) GS:ffff8ec83fc40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fe2127f4a08 CR3: 0000000605f8e000 CR4: 00000000003607e0 NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [migration/14:81] Modules linked in: binfmt_misc tcp_diag inet_diag nf_conntrack_ftp tq_base(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc nfit libnvdimm ppdev coretemp vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic drm ata_piix libata crct10dif_pclmul vmxnet3 crct10dif_common crc32c_intel serio_raw vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse CPU: 14 PID: 81 Comm: migration/14 Kdump: loaded Tainted: G W OEL ------------ 3.10.0-1160.21.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 task: ffff8ec613b54200 ti: ffff8ec613b64000 task.ti: ffff8ec613b64000 RIP: 0010:[<ffffffff9713587a>] [<ffffffff9713587a>] multi_cpu_stop+0x4a/0x110 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: