WARNING: CPU: 41 PID: 215 at net/sched/sch_generic.c:356 dev_watchdog+0x248/0x260
NETDEV WATCHDOG: eno1 (i40e): transmit queue 53 timed out
Modules linked in: nfsv3 tcp_diag inet_diag veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass
crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_hdmi pcspkr snd_hda_intel snd_hda_codec joydev snd_hda_core snd_seq snd_hwdep snd_seq_device snd_pcm snd_timer snd soundcore sg lpc_ich i2c_i801 ipmi_si acpi_cpufreq mei_me mei acpi_power_meter acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul ast crct10dif_common crc32c_intel i2c_algo_bit drm_kms_helper i40e ttm syscopyarea sysfillrect sysimgblt fb_sys_fops drm ahci ptp pps_core megaraid_sas libahci libata nvme drm_panel_orientation_quirks nvme_core wmi ipmi_devintf ipmi_msghandler nfit libnvdimm
CPU: 41 PID: 215 Comm: migration/41 Kdump: loaded Tainted: P           OE  ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: Supermicro SYS-4029GP-TRT/X11DPG-OT-CPU, BIOS 3.2a 12/10/2019
Call Trace:
<IRQ>  [<ffffffffa2d61dc1>] dump_stack+0x19/0x1b
[<ffffffffa2697648>] __warn+0xd8/0x100
[<ffffffffa26976cf>] warn_slowpath_fmt+0x5f/0x80
[<ffffffffa2c65bb8>] dev_watchdog+0x248/0x260
[<ffffffffa2c65970>] ? dev_deactivate_queue.constprop.26+0x60/0x60
[<ffffffffa26a7f58>] call_timer_fn+0x38/0x110
[<ffffffffa2c65970>] ? dev_deactivate_queue.constprop.26+0x60/0x60
[<ffffffffa26aa3bd>] run_timer_softirq+0x24d/0x300
[<ffffffffa26a0f05>] __do_softirq+0xf5/0x280
[<ffffffffa2d7832c>] call_softirq+0x1c/0x30
[<ffffffffa262e675>] do_softirq+0x65/0xa0
[<ffffffffa26a1285>] irq_exit+0x105/0x110
[<ffffffffa2d796c8>] smp_apic_timer_interrupt+0x48/0x60
[<ffffffffa2d75df2>] apic_timer_interrupt+0x162/0x170
<EOI>  [<ffffffffa272fd4a>] ? multi_cpu_stop+0x4a/0x110
[<ffffffffa272fd00>] ? cpu_stop_should_run+0x50/0x50
[<ffffffffa2730009>] cpu_stopper_thread+0x99/0x150
[<ffffffffa2d6770f>] ? __schedule+0x3ff/0x890
[<ffffffffa26ca5c4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa26ca480>] ? lg_double_unlock+0x40/0x40
[<ffffffffa26c1c31>] kthread+0xd1/0xe0
[<ffffffffa26c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa2d74c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa26c1b60>] ? insert_kthread_work+0x40/0x40

WARNING: CPU: 32 PID: 169 at net/sched/sch_generic.c:356 dev_watchdog+0x248/0x260
NETDEV WATCHDOG: eth3 (mlx5_core): transmit queue 21 timed out
Modules linked in: nvidia_uvm(OE) petrelfs(OE) cuse fuse bonding rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) sunrpc nvidia_drm(POE) nvidia_modeset(POE) iTCO_wdt iTCO_vendor_support vfat fat coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd nvidia(POE) pcspkr joydev cdc_ether usbnet mii sg mei_me mei i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler pinctrl_lewisburg pinctrl_intel acpi_cpufreq acpi_power_meter knem(OE) ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic nouveau crct10dif_pclmul crct10dif_common crc32c_intel nvme ast video mxm_wmi nvme_core i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt
mlxfw(OE) vfio_mdev(OE) fb_sys_fops vfio_iommu_type1 ttm vfio mdev(OE) mlx_compat(OE) devlink drm ixgbe ahci mpt3sas libahci mdio libata ptp drm_panel_orientation_quirks raid_class pps_core scsi_transport_sas dca wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 32 PID: 169 Comm: migration/32 Kdump: loaded Tainted: P        W  OE  ------------   3.10.0-957.el7.x86_64 #1
Hardware name: Nettrix X660 G45/60WB32, BIOS NKMH041042 10/28/2022
Call Trace:
<IRQ>  [<ffffffff8cf61dc1>] dump_stack+0x19/0x1b
[<ffffffff8c897648>] __warn+0xd8/0x100
[<ffffffff8c8976cf>] warn_slowpath_fmt+0x5f/0x80
[<ffffffff8ce65bb8>] dev_watchdog+0x248/0x260
[<ffffffff8ce65970>] ? dev_deactivate_queue.constprop.26+0x60/0x60
[<ffffffff8c8a7f58>] call_timer_fn+0x38/0x110
[<ffffffff8ce65970>] ? dev_deactivate_queue.constprop.26+0x60/0x60
[<ffffffff8c8aa3bd>] run_timer_softirq+0x24d/0x300
[<ffffffff8c8a0f05>] __do_softirq+0xf5/0x280
[<ffffffff8cf7832c>] call_softirq+0x1c/0x30
[<ffffffff8c82e675>] do_softirq+0x65/0xa0
[<ffffffff8c8a1285>] irq_exit+0x105/0x110
[<ffffffff8cf796c8>] smp_apic_timer_interrupt+0x48/0x60
[<ffffffff8cf75df2>] apic_timer_interrupt+0x162/0x170
<EOI>  [<ffffffff8c92fd4f>] ? multi_cpu_stop+0x4f/0x110
[<ffffffff8c92fd00>] ? cpu_stop_should_run+0x50/0x50
[<ffffffff8c930009>] cpu_stopper_thread+0x99/0x150
[<ffffffff8cf6770f>] ? __schedule+0x3ff/0x890
[<ffffffff8c8ca5c4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff8c8ca480>] ? lg_double_unlock+0x40/0x40
[<ffffffff8c8c1c31>] kthread+0xd1/0xe0
[<ffffffff8c8c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff8cf74c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff8c8c1b60>] ? insert_kthread_work+0x40/0x40

WARNING: CPU: 13 PID: 75 at net/sched/sch_generic.c:356 dev_watchdog+0x248/0x260
NETDEV WATCHDOG: ens16 (ixgbevf): transmit queue 1 timed out
sched: RT throttling activated
Modules linked in: binfmt_misc xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) iptable_filter sunrpc ext4 mbcache jbd2 ppdev sb_edac iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel parport_pc lrw gf128mul parport sg glue_helper pcspkr i2c_piix4 joydev ablk_helper virtio_balloon cryptd ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_console virtio_scsi bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm ata_piix libahci libata crct10dif_pclmul crct10dif_common virtio_pci crc32c_intel ixgbevf virtio_ring serio_raw
virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 13 PID: 75 Comm: migration/13 Kdump: loaded Tainted: G               ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
Call Trace:
<IRQ>  [<ffffffffadb7ffa5>] dump_stack+0x19/0x1b
[<ffffffffad49bd18>] __warn+0xd8/0x100
[<ffffffffad49bd9f>] warn_slowpath_fmt+0x5f/0x80
[<ffffffffc059b02d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffffadac0780>] ? tcp_init_xmit_timers+0x30/0x30
[<ffffffffada811e8>] dev_watchdog+0x248/0x260
[<ffffffffada80fa0>] ? dev_deactivate_queue.constprop.27+0x60/0x60
[<ffffffffad4ac7c8>] call_timer_fn+0x38/0x110
[<ffffffffada80fa0>] ? dev_deactivate_queue.constprop.27+0x60/0x60
[<ffffffffad4aec5d>] run_timer_softirq+0x24d/0x300
[<ffffffffad4a5695>] __do_softirq+0xf5/0x280
[<ffffffffadb9642c>] call_softirq+0x1c/0x30
[<ffffffffad42f715>] do_softirq+0x65/0xa0
[<ffffffffad4a5a15>] irq_exit+0x105/0x110
[<ffffffffadb979c8>] smp_apic_timer_interrupt+0x48/0x60
[<ffffffffadb93efa>] apic_timer_interrupt+0x16a/0x170
<EOI>  [<ffffffffad53603a>] ? multi_cpu_stop+0x4a/0x110
[<ffffffffad535ff0>] ? cpu_stop_should_run+0x50/0x50
[<ffffffffad5362f9>] cpu_stopper_thread+0x99/0x150
[<ffffffffadb85942>] ? __schedule+0x402/0x840
[<ffffffffad4cf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffad4cefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffffad4c6691>] kthread+0xd1/0xe0
[<ffffffffad4c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffadb92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffad4c65c0>] ? insert_kthread_work+0x40/0x40

WARNING: CPU: 3 PID: 24 at net/sched/sch_generic.c:356 dev_watchdog+0x248/0x260
NETDEV WATCHDOG: ens192 (vmxnet3): transmit queue 6 timed out
Modules linked in: binfmt_misc vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd nfit ppdev sg pcspkr parport_pc vmw_balloon joydev parport libnvdimm i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm ata_piix vmxnet3 libata vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 24 Comm: migration/3 Kdump: loaded Not tainted 3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Call Trace:
<IRQ>  [<ffffffff90f63107>] dump_stack+0x19/0x1b
[<ffffffff90897768>] __warn+0xd8/0x100
[<ffffffff908977ef>] warn_slowpath_fmt+0x5f/0x80
[<ffffffff90e66c38>] dev_watchdog+0x248/0x260
[<ffffffff90e669f0>] ? dev_deactivate_queue.constprop.26+0x60/0x60
[<ffffffff908a80c8>] call_timer_fn+0x38/0x110
[<ffffffff90e669f0>] ? dev_deactivate_queue.constprop.26+0x60/0x60
[<ffffffff908aa52d>] run_timer_softirq+0x24d/0x300
[<ffffffff908a1075>] __do_softirq+0xf5/0x280
[<ffffffff90f7932c>] call_softirq+0x1c/0x30
[<ffffffff9082e675>] do_softirq+0x65/0xa0
[<ffffffff908a13f5>] irq_exit+0x105/0x110
[<ffffffff90f7a6e8>] smp_apic_timer_interrupt+0x48/0x60
[<ffffffff90f76df2>] apic_timer_interrupt+0x162/0x170
<EOI>  [<ffffffff9093026a>] ? multi_cpu_stop+0x4a/0x110
[<ffffffff90930220>] ? cpu_stop_should_run+0x50/0x50
[<ffffffff90930529>] cpu_stopper_thread+0x99/0x150
[<ffffffff90f68a7a>] ? __schedule+0x42a/0x860
[<ffffffff908ca734>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff908ca5f0>] ? lg_double_unlock+0x40/0x40
[<ffffffff908c1da1>] kthread+0xd1/0xe0
[<ffffffff908c1cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffff90f75c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff908c1cd0>] ? insert_kthread_work+0x40/0x40