Info

Permalink
Function
e1000_xmit_frame
First occurrence
2020-10-10
Last occurrence
2026-08-12
State
NEW
Components
kernel
Quality
-50

Statistics

Operating system Count
CentOS 7 31
Architecture Count
x86_64 31
Related packages Count
kernel 31
 0:3.10.0-1062.9.1.el7 9
 0:3.10.0-1160.el7 8
 0:3.10.0-1127.18.2.el7 2
 0:3.10.0-1127.19.1.el7 2
 0:3.10.0-1127.el7 2
 0:3.10.0-1160.6.1.el7 2
 0:3.10.0-1160.42.2.el7 1
 0:3.10.0-957.10.1.el7 1
 0:3.10.0-1160.62.1.el7 1
 0:3.10.0-862.9.1.el7 1
 0:3.10.0-1160.76.1.el7 1
 0:3.10.0-1160.119.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #1420139
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc03d7877
Build id: 3.10.0-1160.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffa9254386
Build id: 3.10.0-1160.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffa928242a
Build id: 3.10.0-1160.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffa9257301
Build id: 3.10.0-1160.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffffa92574d0
Build id: 3.10.0-1160.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffffa92a5373
Build id: 3.10.0-1160.el7.x86_64
0x593
7
ip_output
vmlinux 0xffffffffa92a59db
Build id: 3.10.0-1160.el7.x86_64
0x7b
8
? __kmalloc_node_track_caller
vmlinux 0xffffffffa8e2a765
Build id: 3.10.0-1160.el7.x86_64
0x35
9
? __alloc_skb
vmlinux 0xffffffffa923e13d
Build id: 3.10.0-1160.el7.x86_64
0x8d
10
ip_local_out_sk
vmlinux 0xffffffffa92a3407
Build id: 3.10.0-1160.el7.x86_64
0x37
11
ip_queue_xmit
vmlinux 0xffffffffa92a37a4
Build id: 3.10.0-1160.el7.x86_64
0x144
12
tcp_transmit_skb
vmlinux 0xffffffffa92bde94
Build id: 3.10.0-1160.el7.x86_64
0x4e4
13
tcp_write_xmit
vmlinux 0xffffffffa92be51a
Build id: 3.10.0-1160.el7.x86_64
0x18a
14
tcp_tsq_handler.part.33
vmlinux 0xffffffffa92bf105
Build id: 3.10.0-1160.el7.x86_64
0x35
15
tcp_tasklet_func
vmlinux 0xffffffffa92bf2f8
Build id: 3.10.0-1160.el7.x86_64
0x108
16
tasklet_action
vmlinux 0xffffffffa8ca49e3
Build id: 3.10.0-1160.el7.x86_64
0x83
17
__do_softirq
vmlinux 0xffffffffa8ca4b95
Build id: 3.10.0-1160.el7.x86_64
0xf5
18
run_ksoftirqd
vmlinux 0xffffffffa8ca4d58
Build id: 3.10.0-1160.el7.x86_64
0x38
19
smpboot_thread_fn
vmlinux 0xffffffffa8cce674
Build id: 3.10.0-1160.el7.x86_64
0x144
20
? lg_double_unlock
vmlinux 0xffffffffa8cce530
Build id: 3.10.0-1160.el7.x86_64
0x40
21
kthread
vmlinux 0xffffffffa8cc5c21
Build id: 3.10.0-1160.el7.x86_64
0xd1
22
? insert_kthread_work
vmlinux 0xffffffffa8cc5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
23
ret_from_fork_nospec_begin
vmlinux 0xffffffffa9393df7
Build id: 3.10.0-1160.el7.x86_64
0x21
24
? insert_kthread_work
vmlinux 0xffffffffa8cc5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 32s! [ksoftirqd/8:49]
Modules linked in: vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq sb_edac snd_seq_device iosf_mbi snd_pcm crc32_pclmul ghash_clmulni_intel snd_timer aesni_intel snd lrw gf128mul glue_helper soundcore ablk_helper ppdev sg cryptd joydev pcspkr vmw_balloon parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm nfit mptspi libahci scsi_transport_spi ata_piix libnvdimm mptscsih crct10dif_pclmul libata crct10dif_common mptbase crc32c_intel serio_raw e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 8 PID: 49 Comm: ksoftirqd/8 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffffa03677b68000 ti: ffffa03677b70000 task.ti: ffffa03677b70000
RIP: 0010:[<ffffffffc03d7877>]  [<ffffffffc03d7877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffffa03677b739b0  EFLAGS: 00000286
RAX: ffffb19183203818 RBX: 0000000000000001 RCX: 00000000000000b4
RDX: 00000000000000b4 RSI: ffffa03535cce0c0 RDI: ffffa036463c6000
RBP: ffffa03677b73a30 R08: 0000000000000100 R09: 0000000026100000
R10: 646ce42899546d04 R11: 0000000000000000 R12: 0000000000000202
R13: 00000007af1dc818 R14: 0000000135cce0c0 R15: ffffffffa8fb1650
FS:  0000000000000000(0000) GS:ffffa03d1d800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f86b1378080 CR3: 000000017dbba000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa9254386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa928242a>] sch_direct_xmit+0x11a/0x250
[<ffffffffa9257301>] __dev_queue_xmit+0x491/0x650
[<ffffffffa92574d0>] dev_queue_xmit+0x10/0x20
[<ffffffffa92a5373>] ip_finish_output+0x593/0x8d0
[<ffffffffa92a59db>] ip_output+0x7b/0xf0
[<ffffffffa8e2a765>] ? __kmalloc_node_track_caller+0x35/0x290
[<ffffffffa923e13d>] ? __alloc_skb+0x8d/0x2d0
[<ffffffffa92a3407>] ip_local_out_sk+0x37/0x40
[<ffffffffa92a37a4>] ip_queue_xmit+0x144/0x3c0
[<ffffffffa92bde94>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffa92be51a>] tcp_write_xmit+0x18a/0xd40
[<ffffffffa92bf105>] tcp_tsq_handler.part.33+0x35/0x40
[<ffffffffa92bf2f8>] tcp_tasklet_func+0x108/0x140
[<ffffffffa8ca49e3>] tasklet_action+0x83/0x140
[<ffffffffa8ca4b95>] __do_softirq+0xf5/0x280
[<ffffffffa8ca4d58>] run_ksoftirqd+0x38/0x50
[<ffffffffa8cce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa8cce530>] ? lg_double_unlock+0x40/0x40
[<ffffffffa8cc5c21>] kthread+0xd1/0xe0
[<ffffffffa8cc5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffa9393df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa8cc5b50>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #1286886
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc03f844c
Build id: 3.10.0-1160.119.1.el7.x86_64
0xaec
2
dev_hard_start_xmit
vmlinux 0xffffffff9607edc6
Build id: 3.10.0-1160.119.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff960add72
Build id: 3.10.0-1160.119.1.el7.x86_64
0x122
4
__dev_queue_xmit
vmlinux 0xffffffff96081dd1
Build id: 3.10.0-1160.119.1.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffff96081fa0
Build id: 3.10.0-1160.119.1.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffff960d2523
Build id: 3.10.0-1160.119.1.el7.x86_64
0x593
7
ip_output
vmlinux 0xffffffff960d2b9b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffff960d1f90
Build id: 3.10.0-1160.119.1.el7.x86_64
0xa60
9
ip_local_out_sk
vmlinux 0xffffffff960d0587
Build id: 3.10.0-1160.119.1.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffff960d0924
Build id: 3.10.0-1160.119.1.el7.x86_64
0x144
11
tcp_transmit_skb
vmlinux 0xffffffff960eb5f4
Build id: 3.10.0-1160.119.1.el7.x86_64
0x4e4
12
tcp_write_xmit
vmlinux 0xffffffff960ebc7a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x18a
13
tcp_tsq_handler.part.33
vmlinux 0xffffffff960ec865
Build id: 3.10.0-1160.119.1.el7.x86_64
0x35
14
tcp_tasklet_func
vmlinux 0xffffffff960eca68
Build id: 3.10.0-1160.119.1.el7.x86_64
0x108
15
tasklet_action
vmlinux 0xffffffff95aa93e3
Build id: 3.10.0-1160.119.1.el7.x86_64
0x83
16
__do_softirq
vmlinux 0xffffffff95aa9595
Build id: 3.10.0-1160.119.1.el7.x86_64
0xf5
17
run_ksoftirqd
vmlinux 0xffffffff95aa9768
Build id: 3.10.0-1160.119.1.el7.x86_64
0x38
18
smpboot_thread_fn
vmlinux 0xffffffff95ad43fc
Build id: 3.10.0-1160.119.1.el7.x86_64
0x14c
19
? lg_double_unlock
vmlinux 0xffffffff95ad42b0
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
20
kthread
vmlinux 0xffffffff95acb621
Build id: 3.10.0-1160.119.1.el7.x86_64
0xd1
21
? insert_kthread_work
vmlinux 0xffffffff95acb550
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
22
ret_from_fork_nospec_begin
vmlinux 0xffffffff961c51f7
Build id: 3.10.0-1160.119.1.el7.x86_64
0x21
23
? insert_kthread_work
vmlinux 0xffffffff95acb550
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 27s! [ksoftirqd/2:19]
Modules linked in: xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_addrtype iptable_filter br_netfilter bridge stp llc nfsv3 nfs_acl rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache overlay(T) bochs_drm ttm edac_mce_amd drm_kms_helper syscopyarea sysfillrect sysimgblt aesni_intel fb_sys_fops lrw gf128mul drm glue_helper ablk_helper cryptd joydev virtio_balloon pcspkr floppy drm_panel_orientation_quirks i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common sg ata_generic pata_acpi ata_piix virtio_console libata crc32c_intel vmxnet3 virtio_pci virtio_ring vmw_pvscsi serio_raw virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
task: ffff8a203af78000 ti: ffff8a203af74000 task.ti: ffff8a203af74000
RIP: 0010:[<ffffffffc03f844c>]  [<ffffffffc03f844c>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0018:ffff8a203af779a0  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffff8a20ffd13940 RCX: 0000000000000128
RDX: ffff9d7280e60610 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff8a203af77a30 R08: 00000001a411f000 R09: 0000000016a04042
R10: 0000000000000001 R11: ffff8a1f7a94f270 R12: ffff8a20e0f94090
R13: ffff8a20ffd13940 R14: ffff8a20e0f9410c R15: ffff8a20e0f940f8
FS:  0000000000000000(0000) GS:ffff8a20ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fbce6633988 CR3: 0000000235aaa000 CR4: 00000000000007e0
Call Trace:
[<ffffffff9607edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff960add72>] sch_direct_xmit+0x122/0x250
[<ffffffff96081dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffff96081fa0>] dev_queue_xmit+0x10/0x20
[<ffffffff960d2523>] ip_finish_output+0x593/0x8d0
[<ffffffff960d2b9b>] ip_output+0x7b/0x100
[<ffffffff960d1f90>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff960d0587>] ip_local_out_sk+0x37/0x40
[<ffffffff960d0924>] ip_queue_xmit+0x144/0x3d0
[<ffffffff960eb5f4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff960ebc7a>] tcp_write_xmit+0x18a/0xd40
[<ffffffff960ec865>] tcp_tsq_handler.part.33+0x35/0x40
[<ffffffff960eca68>] tcp_tasklet_func+0x108/0x140
[<ffffffff95aa93e3>] tasklet_action+0x83/0x140
[<ffffffff95aa9595>] __do_softirq+0xf5/0x290
[<ffffffff95aa9768>] run_ksoftirqd+0x38/0x60
[<ffffffff95ad43fc>] smpboot_thread_fn+0x14c/0x1b0
[<ffffffff95ad42b0>] ? lg_double_unlock+0x40/0x40
[<ffffffff95acb621>] kthread+0xd1/0xe0
[<ffffffff95acb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff961c51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff95acb550>] ? insert_kthread_work+0x40/0x40
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41 
Complete report #745872
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0292877
Build id: 3.10.0-1160.76.1.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffaa6416e7
Build id: 3.10.0-1160.76.1.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffaa658f86
Build id: 3.10.0-1160.76.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffffaa68709a
Build id: 3.10.0-1160.76.1.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffaa65bf01
Build id: 3.10.0-1160.76.1.el7.x86_64
0x491
6
dev_queue_xmit
vmlinux 0xffffffffaa65c0d0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffffaa6aac5b
Build id: 3.10.0-1160.76.1.el7.x86_64
0x67b
8
ip_output
vmlinux 0xffffffffaa6ab1db
Build id: 3.10.0-1160.76.1.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffffaa6aa5e0
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffffaa6a8c07
Build id: 3.10.0-1160.76.1.el7.x86_64
0x37
11
ip_queue_xmit
vmlinux 0xffffffffaa6a8fa4
Build id: 3.10.0-1160.76.1.el7.x86_64
0x144
12
tcp_transmit_skb
vmlinux 0xffffffffaa6c3734
Build id: 3.10.0-1160.76.1.el7.x86_64
0x4e4
13
tcp_write_xmit
vmlinux 0xffffffffaa6c3dba
Build id: 3.10.0-1160.76.1.el7.x86_64
0x18a
14
tcp_tsq_handler.part.33
vmlinux 0xffffffffaa6c49a5
Build id: 3.10.0-1160.76.1.el7.x86_64
0x35
15
tcp_tasklet_func
vmlinux 0xffffffffaa6c4b98
Build id: 3.10.0-1160.76.1.el7.x86_64
0x108
16
tasklet_action
vmlinux 0xffffffffaa0a4bd3
Build id: 3.10.0-1160.76.1.el7.x86_64
0x83
17
__do_softirq
vmlinux 0xffffffffaa0a4d85
Build id: 3.10.0-1160.76.1.el7.x86_64
0xf5
18
run_ksoftirqd
vmlinux 0xffffffffaa0a4f48
Build id: 3.10.0-1160.76.1.el7.x86_64
0x38
19
smpboot_thread_fn
vmlinux 0xffffffffaa0ce9e4
Build id: 3.10.0-1160.76.1.el7.x86_64
0x144
20
? lg_double_unlock
vmlinux 0xffffffffaa0ce8a0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
21
kthread
vmlinux 0xffffffffaa0c5f91
Build id: 3.10.0-1160.76.1.el7.x86_64
0xd1
22
? insert_kthread_work
vmlinux 0xffffffffaa0c5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
23
ret_from_fork_nospec_begin
vmlinux 0xffffffffaa799df7
Build id: 3.10.0-1160.76.1.el7.x86_64
0x21
24
? insert_kthread_work
vmlinux 0xffffffffaa0c5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [ksoftirqd/0:6]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus sb_edac snd_seq snd_seq_device iosf_mbi crc32_pclmul snd_pcm ghash_clmulni_intel
aesni_intel snd_timer btusb lrw snd btrtl btbcm btintel gf128mul bluetooth soundcore glue_helper rfkill ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix nfit mptspi libnvdimm libata scsi_transport_spi serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks synosnap(OE) fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffffa0ea6c4b5280 ti: ffffa0ea6c574000 task.ti: ffffa0ea6c574000
RIP: 0010:[<ffffffffc0292877>]  [<ffffffffc0292877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffffa0ea6c5779b0  EFLAGS: 00000286
RAX: ffffc47c431c3818 RBX: 0000000000000001 RCX: 0000000000000047
RDX: 0000000000000047 RSI: ffffa0f118fa7580 RDI: ffffa0f11c25e000
RBP: ffffa0ea6c577a30 R08: 0000000000000100 R09: 0000000022100000
R10: 0045000824353e56 R11: 0000000000000000 R12: 0000000000000202
R13: 00000004c44b78ce R14: 000000011c8f1100 R15: ffffffffaa3b20b0
FS:  0000000000000000(0000) GS:ffffa0f11d600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000a94c000 CR3: 00000007ac6e0000 CR4: 00000000003607f0
Call Trace:
[<ffffffffaa6416e7>] ? kfree_skbmem+0x37/0x90
[<ffffffffaa658f86>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffaa68709a>] sch_direct_xmit+0x11a/0x250
[<ffffffffaa65bf01>] __dev_queue_xmit+0x491/0x650
[<ffffffffaa65c0d0>] dev_queue_xmit+0x10/0x20
[<ffffffffaa6aac5b>] ip_finish_output+0x67b/0x8d0
[<ffffffffaa6ab1db>] ip_output+0x7b/0xf0
[<ffffffffaa6aa5e0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffaa6a8c07>] ip_local_out_sk+0x37/0x40
[<ffffffffaa6a8fa4>] ip_queue_xmit+0x144/0x3c0
[<ffffffffaa6c3734>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffaa6c3dba>] tcp_write_xmit+0x18a/0xd40
[<ffffffffaa6c49a5>] tcp_tsq_handler.part.33+0x35/0x40
[<ffffffffaa6c4b98>] tcp_tasklet_func+0x108/0x140
[<ffffffffaa0a4bd3>] tasklet_action+0x83/0x140
[<ffffffffaa0a4d85>] __do_softirq+0xf5/0x280
[<ffffffffaa0a4f48>] run_ksoftirqd+0x38/0x50
[<ffffffffaa0ce9e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffaa0ce8a0>] ? lg_double_unlock+0x40/0x40
[<ffffffffaa0c5f91>] kthread+0xd1/0xe0
[<ffffffffaa0c5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffffaa799df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffaa0c5ec0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #661517
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0335867
Build id: 3.10.0-862.9.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffa81ec226
Build id: 3.10.0-862.9.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffa8217aba
Build id: 3.10.0-862.9.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffa81ef111
Build id: 3.10.0-862.9.1.el7.x86_64
0x4a1
5
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc094b2ce
Build id: 3.10.0-862.9.1.el7.x86_64
0x4e
6
dev_queue_xmit
vmlinux 0xffffffffa81ef2e0
Build id: 3.10.0-862.9.1.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffffa82384a6
Build id: 3.10.0-862.9.1.el7.x86_64
0x546
8
ip_output
vmlinux 0xffffffffa8238a03
Build id: 3.10.0-862.9.1.el7.x86_64
0x73
9
? __ip_append_data.isra.50
vmlinux 0xffffffffa8237f60
Build id: 3.10.0-862.9.1.el7.x86_64
0xa50
10
ip_local_out_sk
vmlinux 0xffffffffa82365f7
Build id: 3.10.0-862.9.1.el7.x86_64
0x37
11
ip_queue_xmit
vmlinux 0xffffffffa8236963
Build id: 3.10.0-862.9.1.el7.x86_64
0x143
12
tcp_transmit_skb
vmlinux 0xffffffffa8250844
Build id: 3.10.0-862.9.1.el7.x86_64
0x4e4
13
tcp_write_xmit
vmlinux 0xffffffffa8250eb2
Build id: 3.10.0-862.9.1.el7.x86_64
0x172
14
tcp_tsq_handler.part.31
vmlinux 0xffffffffa8251a75
Build id: 3.10.0-862.9.1.el7.x86_64
0x35
15
tcp_tasklet_func
vmlinux 0xffffffffa8251c58
Build id: 3.10.0-862.9.1.el7.x86_64
0x108
16
tasklet_action
vmlinux 0xffffffffa7c9aed3
Build id: 3.10.0-862.9.1.el7.x86_64
0x83
17
__do_softirq
vmlinux 0xffffffffa7c9b085
Build id: 3.10.0-862.9.1.el7.x86_64
0xf5
18
run_ksoftirqd
vmlinux 0xffffffffa7c9b248
Build id: 3.10.0-862.9.1.el7.x86_64
0x38
19
smpboot_thread_fn
vmlinux 0xffffffffa7cc3dd4
Build id: 3.10.0-862.9.1.el7.x86_64
0x144
20
? lg_double_unlock
vmlinux 0xffffffffa7cc3c90
Build id: 3.10.0-862.9.1.el7.x86_64
0x40
21
kthread
vmlinux 0xffffffffa7cbb621
Build id: 3.10.0-862.9.1.el7.x86_64
0xd1
22
? insert_kthread_work
vmlinux 0xffffffffa7cbb550
Build id: 3.10.0-862.9.1.el7.x86_64
0x40
23
ret_from_fork_nospec_begin
vmlinux 0xffffffffa83205f7
Build id: 3.10.0-862.9.1.el7.x86_64
0x21
24
? insert_kthread_work
vmlinux 0xffffffffa7cbb550
Build id: 3.10.0-862.9.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [ksoftirqd/2:19]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 ppdev snd_rawmidi snd_ac97_codec aesni_intel ac97_bus lrw gf128mul snd_seq vmw_balloon glue_helper ablk_helper cryptd snd_seq_device snd_pcm pcspkr joydev snd_timer snd soundcore vmw_vmci sg nfit i2c_piix4 shpchp libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod
crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul drm_kms_helper crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt mptspi fb_sys_fops ata_piix ttm scsi_transport_spi libata mptscsih serio_raw e1000 drm mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Tainted: G             L ------------   3.10.0-862.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff921c74e68fd0 ti: ffff921c74e74000 task.ti: ffff921c74e74000
RIP: 0010:[<ffffffffc0335867>]  [<ffffffffc0335867>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff921c74e779c0  EFLAGS: 00000286
RAX: ffffb7a8831c3818 RBX: 0000000000000001 RCX: 0000000000000050
RDX: 0000000000000050 RSI: ffff922292ede9c0 RDI: ffff922272afc000
RBP: ffff921c74e77a40 R08: 0000000000000100 R09: 0000000026100000
R10: a9ac3e45da62bc3d R11: 0000000000000000 R12: 0000000000000202
R13: 00000004950f12e8 R14: 0000000100000020 R15: ffffffffa7f748b0
FS:  0000000000000000(0000) GS:ffff922273080000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffd16f975b8 CR3: 00000004d640e000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa81ec226>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa8217aba>] sch_direct_xmit+0x11a/0x250
[<ffffffffa81ef111>] __dev_queue_xmit+0x4a1/0x660
[<ffffffffc094b2ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffffa81ef2e0>] dev_queue_xmit+0x10/0x20
[<ffffffffa82384a6>] ip_finish_output+0x546/0x7a0
[<ffffffffa8238a03>] ip_output+0x73/0xe0
[<ffffffffa8237f60>] ? __ip_append_data.isra.50+0xa50/0xa50
[<ffffffffa82365f7>] ip_local_out_sk+0x37/0x40
[<ffffffffa8236963>] ip_queue_xmit+0x143/0x3a0
[<ffffffffa8250844>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffa8250eb2>] tcp_write_xmit+0x172/0xd00
[<ffffffffa8251a75>] tcp_tsq_handler.part.31+0x35/0x40
[<ffffffffa8251c58>] tcp_tasklet_func+0x108/0x140
[<ffffffffa7c9aed3>] tasklet_action+0x83/0x140
[<ffffffffa7c9b085>] __do_softirq+0xf5/0x280
[<ffffffffa7c9b248>] run_ksoftirqd+0x38/0x50
[<ffffffffa7cc3dd4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa7cc3c90>] ? lg_double_unlock+0x40/0x40
[<ffffffffa7cbb621>] kthread+0xd1/0xe0
[<ffffffffa7cbb550>] ? insert_kthread_work+0x40/0x40
[<ffffffffa83205f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa7cbb550>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #555170
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc050f877
Build id: 3.10.0-1160.62.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff88a58f86
Build id: 3.10.0-1160.62.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff88a8709a
Build id: 3.10.0-1160.62.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff88a5bf01
Build id: 3.10.0-1160.62.1.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffff88a5c0d0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x10
6
neigh_resolve_output
vmlinux 0xffffffff88a68dfd
Build id: 3.10.0-1160.62.1.el7.x86_64
0x11d
7
ip_finish_output
vmlinux 0xffffffff88aaa85c
Build id: 3.10.0-1160.62.1.el7.x86_64
0x27c
8
ip_output
vmlinux 0xffffffff88aab1db
Build id: 3.10.0-1160.62.1.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffff88aaa5e0
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffff88aa8c07
Build id: 3.10.0-1160.62.1.el7.x86_64
0x37
11
ip_queue_xmit
vmlinux 0xffffffff88aa8fa4
Build id: 3.10.0-1160.62.1.el7.x86_64
0x144
12
tcp_transmit_skb
vmlinux 0xffffffff88ac3724
Build id: 3.10.0-1160.62.1.el7.x86_64
0x4e4
13
tcp_write_xmit
vmlinux 0xffffffff88ac3daa
Build id: 3.10.0-1160.62.1.el7.x86_64
0x18a
14
tcp_tsq_handler.part.33
vmlinux 0xffffffff88ac4995
Build id: 3.10.0-1160.62.1.el7.x86_64
0x35
15
tcp_tasklet_func
vmlinux 0xffffffff88ac4b88
Build id: 3.10.0-1160.62.1.el7.x86_64
0x108
16
tasklet_action
vmlinux 0xffffffff884a4bd3
Build id: 3.10.0-1160.62.1.el7.x86_64
0x83
17
__do_softirq
vmlinux 0xffffffff884a4d85
Build id: 3.10.0-1160.62.1.el7.x86_64
0xf5
18
run_ksoftirqd
vmlinux 0xffffffff884a4f48
Build id: 3.10.0-1160.62.1.el7.x86_64
0x38
19
smpboot_thread_fn
vmlinux 0xffffffff884cea04
Build id: 3.10.0-1160.62.1.el7.x86_64
0x144
20
? lg_double_unlock
vmlinux 0xffffffff884ce8c0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x40
21
kthread
vmlinux 0xffffffff884c5fb1
Build id: 3.10.0-1160.62.1.el7.x86_64
0xd1
22
? insert_kthread_work
vmlinux 0xffffffff884c5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x40
23
ret_from_fork_nospec_begin
vmlinux 0xffffffff88b99ddd
Build id: 3.10.0-1160.62.1.el7.x86_64
0x7
24
? insert_kthread_work
vmlinux 0xffffffff884c5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 24s! [kworker/u256:1:93638]
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [ksoftirqd/0:6]
Modules linked in: snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 aesni_intel snd_rawmidi lrw xfs snd_ac97_codec gf128mul glue_helper ppdev ablk_helper libcrc32c vmw_balloon ac97_bus cryptd snd_seq snd_seq_device snd_pcm joydev pcspkr snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect crct10dif_common sysimgblt crc32c_intel fb_sys_fops ttm ata_piix drm e1000 libata mptspi nfit serio_raw scsi_transport_spi mptscsih libnvdimm mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6 Comm: ksoftirqd/0 Tainted: G             L ------------   3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff90edfafe5280 ti: ffff90edfa8a4000 task.ti: ffff90edfa8a4000
RIP: 0010:[<ffffffffc050f877>]  [<ffffffffc050f877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff90edfa8a7970  EFLAGS: 00000286
RAX: ffffa559410c3818 RBX: 0000000000000001 RCX: 0000000000000051
RDX: 0000000000000051 RSI: ffff90eeb1477140 RDI: ffff90edfae64000
RBP: ffff90edfa8a79f0 R08: 0000000000000100 R09: 0000000026100000
R10: d3243af5d5be7886 R11: 0000000000000000 R12: 0000000000000202
R13: 00000001beb3558c R14: 0000000100000000 R15: ffffffff887b2090
FS:  0000000000000000(0000) GS:ffff90eeb5600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f671f24b000 CR3: 000000023318a000 CR4: 00000000003607f0
Call Trace:
[<ffffffff88a58f86>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff88a8709a>] sch_direct_xmit+0x11a/0x250
[<ffffffff88a5bf01>] __dev_queue_xmit+0x491/0x650
[<ffffffff88a5c0d0>] dev_queue_xmit+0x10/0x20
[<ffffffff88a68dfd>] neigh_resolve_output+0x11d/0x220
[<ffffffff88aaa85c>] ip_finish_output+0x27c/0x8d0
[<ffffffff88aab1db>] ip_output+0x7b/0xf0
[<ffffffff88aaa5e0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff88aa8c07>] ip_local_out_sk+0x37/0x40
[<ffffffff88aa8fa4>] ip_queue_xmit+0x144/0x3c0
[<ffffffff88ac3724>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff88ac3daa>] tcp_write_xmit+0x18a/0xd40
[<ffffffff88ac4995>] tcp_tsq_handler.part.33+0x35/0x40
[<ffffffff88ac4b88>] tcp_tasklet_func+0x108/0x140
[<ffffffff884a4bd3>] tasklet_action+0x83/0x140
[<ffffffff884a4d85>] __do_softirq+0xf5/0x280
[<ffffffff884a4f48>] run_ksoftirqd+0x38/0x50
[<ffffffff884cea04>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff884ce8c0>] ? lg_double_unlock+0x40/0x40
[<ffffffff884c5fb1>] kthread+0xd1/0xe0
[<ffffffff884c5ee0>] ? insert_kthread_work+0x40/0x40
[<ffffffff88b99ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff884c5ee0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #217846
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc02d4857
Build id: 3.10.0-957.10.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff99639ce6
Build id: 3.10.0-957.10.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff99666d6a
Build id: 3.10.0-957.10.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff9963cc61
Build id: 3.10.0-957.10.1.el7.x86_64
0x491
5
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc05e02ce
Build id: 3.10.0-957.10.1.el7.x86_64
0x4e
6
dev_queue_xmit
vmlinux 0xffffffff9963ce30
Build id: 3.10.0-957.10.1.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffff996897b6
Build id: 3.10.0-957.10.1.el7.x86_64
0x556
8
ip_output
vmlinux 0xffffffff99689d13
Build id: 3.10.0-957.10.1.el7.x86_64
0x73
9
? __ip_append_data.isra.51
vmlinux 0xffffffff99689260
Build id: 3.10.0-957.10.1.el7.x86_64
0xa50
10
ip_local_out_sk
vmlinux 0xffffffff99687897
Build id: 3.10.0-957.10.1.el7.x86_64
0x37
11
ip_queue_xmit
vmlinux 0xffffffff99687c34
Build id: 3.10.0-957.10.1.el7.x86_64
0x144
12
tcp_transmit_skb
vmlinux 0xffffffff996a1fe4
Build id: 3.10.0-957.10.1.el7.x86_64
0x4e4
13
tcp_write_xmit
vmlinux 0xffffffff996a2652
Build id: 3.10.0-957.10.1.el7.x86_64
0x172
14
tcp_tsq_handler.part.31
vmlinux 0xffffffff996a3215
Build id: 3.10.0-957.10.1.el7.x86_64
0x35
15
tcp_tasklet_func
vmlinux 0xffffffff996a33f8
Build id: 3.10.0-957.10.1.el7.x86_64
0x108
16
tasklet_action
vmlinux 0xffffffff990a0d93
Build id: 3.10.0-957.10.1.el7.x86_64
0x83
17
__do_softirq
vmlinux 0xffffffff990a0f45
Build id: 3.10.0-957.10.1.el7.x86_64
0xf5
18
run_ksoftirqd
vmlinux 0xffffffff990a1108
Build id: 3.10.0-957.10.1.el7.x86_64
0x38
19
smpboot_thread_fn
vmlinux 0xffffffff990ca604
Build id: 3.10.0-957.10.1.el7.x86_64
0x144
20
? lg_double_unlock
vmlinux 0xffffffff990ca4c0
Build id: 3.10.0-957.10.1.el7.x86_64
0x40
21
kthread
vmlinux 0xffffffff990c1c71
Build id: 3.10.0-957.10.1.el7.x86_64
0xd1
22
? insert_kthread_work
vmlinux 0xffffffff990c1ba0
Build id: 3.10.0-957.10.1.el7.x86_64
0x40
23
ret_from_fork_nospec_begin
vmlinux 0xffffffff99775c37
Build id: 3.10.0-957.10.1.el7.x86_64
0x21
24
? insert_kthread_work
vmlinux 0xffffffff990c1ba0
Build id: 3.10.0-957.10.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [ksoftirqd/0:3]
Modules linked in: binfmt_misc tun nls_utf8 cifs ccm dns_resolver ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter sunrpc ext4 mbcache jbd2 ppdev joydev sg virtio_balloon parport_pc parport pcspkr i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_blk cirrus floppy drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix serio_raw libata e1000 drm_panel_orientation_quirks virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Kdump: loaded Not tainted 3.10.0-957.10.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5.1-0-g8936dbb-20141113_115728-nilsson.home.kraxel.org 04/01/2014
task: ffff9fb23d20a080 ti: ffff9fb23d294000 task.ti: ffff9fb23d294000
RIP: 0010:[<ffffffffc02d4857>]  [<ffffffffc02d4857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9fb23d2979c0  EFLAGS: 00010286
RAX: ffffbdc240e83818 RBX: 0000000000000001 RCX: 00000000000000bf
RDX: 00000000000000bf RSI: ffff9fb2bfe99ac0 RDI: ffff9fb2b7616000
RBP: ffff9fb23d297a40 R08: 0000000000000100 R09: 0000000026100000
R10: 23b78740a3c947d5 R11: 0000000000000000 R12: 0000000000000202
R13: 000000010a2abbc4 R14: 00000001bfe99ac0 R15: ffffffff993a0b80
FS:  0000000000000000(0000) GS:ffff9fb2bfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff69f3a002c CR3: 000000003598a000 CR4: 00000000000006f0
Call Trace:
[<ffffffff99639ce6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff99666d6a>] sch_direct_xmit+0x11a/0x250
[<ffffffff9963cc61>] __dev_queue_xmit+0x491/0x650
[<ffffffffc05e02ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffff9963ce30>] dev_queue_xmit+0x10/0x20
[<ffffffff996897b6>] ip_finish_output+0x556/0x7b0
[<ffffffff99689d13>] ip_output+0x73/0xe0
[<ffffffff99689260>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffff99687897>] ip_local_out_sk+0x37/0x40
[<ffffffff99687c34>] ip_queue_xmit+0x144/0x3c0
[<ffffffff996a1fe4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff996a2652>] tcp_write_xmit+0x172/0xd00
[<ffffffff996a3215>] tcp_tsq_handler.part.31+0x35/0x40
[<ffffffff996a33f8>] tcp_tasklet_func+0x108/0x140
[<ffffffff990a0d93>] tasklet_action+0x83/0x140
[<ffffffff990a0f45>] __do_softirq+0xf5/0x280
[<ffffffff990a1108>] run_ksoftirqd+0x38/0x50
[<ffffffff990ca604>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff990ca4c0>] ? lg_double_unlock+0x40/0x40
[<ffffffff990c1c71>] kthread+0xd1/0xe0
[<ffffffff990c1ba0>] ? insert_kthread_work+0x40/0x40
[<ffffffff99775c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff990c1ba0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #130468
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc03cc877
Build id: 3.10.0-1160.6.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffa18543e6
Build id: 3.10.0-1160.6.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffa188248a
Build id: 3.10.0-1160.6.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffa1857361
Build id: 3.10.0-1160.6.1.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffffa1857530
Build id: 3.10.0-1160.6.1.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffffa18a53e3
Build id: 3.10.0-1160.6.1.el7.x86_64
0x593
7
ip_output
vmlinux 0xffffffffa18a5a4b
Build id: 3.10.0-1160.6.1.el7.x86_64
0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffffa18a4e50
Build id: 3.10.0-1160.6.1.el7.x86_64
0xa60
9
ip_local_out_sk
vmlinux 0xffffffffa18a3477
Build id: 3.10.0-1160.6.1.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffffa18a3814
Build id: 3.10.0-1160.6.1.el7.x86_64
0x144
11
tcp_transmit_skb
vmlinux 0xffffffffa18bdf24
Build id: 3.10.0-1160.6.1.el7.x86_64
0x4e4
12
tcp_write_xmit
vmlinux 0xffffffffa18be5aa
Build id: 3.10.0-1160.6.1.el7.x86_64
0x18a
13
tcp_tsq_handler.part.33
vmlinux 0xffffffffa18bf195
Build id: 3.10.0-1160.6.1.el7.x86_64
0x35
14
tcp_tasklet_func
vmlinux 0xffffffffa18bf388
Build id: 3.10.0-1160.6.1.el7.x86_64
0x108
15
tasklet_action
vmlinux 0xffffffffa12a49e3
Build id: 3.10.0-1160.6.1.el7.x86_64
0x83
16
__do_softirq
vmlinux 0xffffffffa12a4b95
Build id: 3.10.0-1160.6.1.el7.x86_64
0xf5
17
run_ksoftirqd
vmlinux 0xffffffffa12a4d58
Build id: 3.10.0-1160.6.1.el7.x86_64
0x38
18
smpboot_thread_fn
vmlinux 0xffffffffa12ce674
Build id: 3.10.0-1160.6.1.el7.x86_64
0x144
19
? lg_double_unlock
vmlinux 0xffffffffa12ce530
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
20
kthread
vmlinux 0xffffffffa12c5c21
Build id: 3.10.0-1160.6.1.el7.x86_64
0xd1
21
? insert_kthread_work
vmlinux 0xffffffffa12c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
22
ret_from_fork_nospec_begin
vmlinux 0xffffffffa1994df7
Build id: 3.10.0-1160.6.1.el7.x86_64
0x21
23
? insert_kthread_work
vmlinux 0xffffffffa12c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [ksoftirqd/0:6]
Modules linked in: binfmt_misc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event coretemp crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi snd_ac97_codec aesni_intel ac97_bus snd_seq lrw gf128mul glue_helper snd_seq_device ablk_helper snd_pcm snd_timer snd cryptd ppdev soundcore
vmw_balloon pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci crct10dif_pclmul crct10dif_common drm crc32c_intel ata_piix libahci libata mptspi serio_raw scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
task: ffff89f0bc74b180 ti: ffff89f0bc758000 task.ti: ffff89f0bc758000
RIP: 0010:[<ffffffffc03cc877>]  [<ffffffffc03cc877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0000:ffff89f0bc75b9b0  EFLAGS: 00000286
RAX: ffff97a1c0e03818 RBX: ffff89f044cb8590 RCX: 00000000000000d6
RDX: 00000000000000d6 RSI: ffff89f075a2ba40 RDI: ffff89f074bd8000
RBP: ffff89f0bc75ba30 R08: 0000000000000100 R09: 0000000026100000
R10: 0000000080000000 R11: 0000000000000000 R12: ffffffffc08a9fad
R13: ffff89f0bc75b9f0 R14: ffff89f0b7a04200 R15: 000000005b953fda
FS:  0000000000000000(0000) GS:ffff89f0bd600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000559c18419090 CR3: 000000006a160000 CR4: 00000000000007f0
Call Trace:
[<ffffffffa18543e6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa188248a>] sch_direct_xmit+0x11a/0x250
[<ffffffffa1857361>] __dev_queue_xmit+0x491/0x650
[<ffffffffa1857530>] dev_queue_xmit+0x10/0x20
[<ffffffffa18a53e3>] ip_finish_output+0x593/0x8d0
[<ffffffffa18a5a4b>] ip_output+0x7b/0xf0
[<ffffffffa18a4e50>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffa18a3477>] ip_local_out_sk+0x37/0x40
[<ffffffffa18a3814>] ip_queue_xmit+0x144/0x3c0
[<ffffffffa18bdf24>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffa18be5aa>] tcp_write_xmit+0x18a/0xd40
[<ffffffffa18bf195>] tcp_tsq_handler.part.33+0x35/0x40
[<ffffffffa18bf388>] tcp_tasklet_func+0x108/0x140
[<ffffffffa12a49e3>] tasklet_action+0x83/0x140
[<ffffffffa12a4b95>] __do_softirq+0xf5/0x280
[<ffffffffa12a4d58>] run_ksoftirqd+0x38/0x50
[<ffffffffa12ce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa12ce530>] ? lg_double_unlock+0x40/0x40
[<ffffffffa12c5c21>] kthread+0xd1/0xe0
[<ffffffffa12c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffa1994df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa12c5b50>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #7191
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc03d9877
Build id: 3.10.0-1062.9.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff9b84f346
Build id: 3.10.0-1062.9.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff9b87d0ba
Build id: 3.10.0-1062.9.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff9b8522b1
Build id: 3.10.0-1062.9.1.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffff9b852480
Build id: 3.10.0-1062.9.1.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffff9b8a0893
Build id: 3.10.0-1062.9.1.el7.x86_64
0x593
7
ip_output
vmlinux 0xffffffff9b8a0efb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffff9b8a0300
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa60
9
ip_local_out_sk
vmlinux 0xffffffff9b89e927
Build id: 3.10.0-1062.9.1.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffff9b89ecc4
Build id: 3.10.0-1062.9.1.el7.x86_64
0x144
11
tcp_transmit_skb
vmlinux 0xffffffff9b8b9374
Build id: 3.10.0-1062.9.1.el7.x86_64
0x4e4
12
tcp_write_xmit
vmlinux 0xffffffff9b8b99fa
Build id: 3.10.0-1062.9.1.el7.x86_64
0x18a
13
tcp_tsq_handler.part.33
vmlinux 0xffffffff9b8ba5e5
Build id: 3.10.0-1062.9.1.el7.x86_64
0x35
14
tcp_tasklet_func
vmlinux 0xffffffff9b8ba7c8
Build id: 3.10.0-1062.9.1.el7.x86_64
0x108
15
tasklet_action
vmlinux 0xffffffff9b2a5153
Build id: 3.10.0-1062.9.1.el7.x86_64
0x83
16
__do_softirq
vmlinux 0xffffffff9b2a5305
Build id: 3.10.0-1062.9.1.el7.x86_64
0xf5
17
run_ksoftirqd
vmlinux 0xffffffff9b2a54c8
Build id: 3.10.0-1062.9.1.el7.x86_64
0x38
18
smpboot_thread_fn
vmlinux 0xffffffff9b2cec24
Build id: 3.10.0-1062.9.1.el7.x86_64
0x144
19
? lg_double_unlock
vmlinux 0xffffffff9b2ceae0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
20
kthread
vmlinux 0xffffffff9b2c61f1
Build id: 3.10.0-1062.9.1.el7.x86_64
0xd1
21
? insert_kthread_work
vmlinux 0xffffffff9b2c6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
22
ret_from_fork_nospec_begin
vmlinux 0xffffffff9b98dd37
Build id: 3.10.0-1062.9.1.el7.x86_64
0x21
23
? insert_kthread_work
vmlinux 0xffffffff9b2c6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 45s! [ksoftirqd/4:29]
ata2: lost interrupt (Status 0x50)
ata2.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
ata2.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in#012         Get event status notification 4a 01 00 00 10 00 00 00 08 00res 58/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x6 (timeout)
ata2.00: status: { DRDY DRQ }
ata2: soft resetting link
Modules linked in: xt_time nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ip6table_filter ip6_tables tcp_diag inet_diag ipt_REJECT nf_reject_ipv4 xt_owner xt_multiport iptable_filter ext4 mbcache jbd2 loop dm_mirror dm_region_hash dm_log dm_mod ppdev sg parport_pc parport pcspkr i2c_piix4 joydev ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm virtio_blk drm ata_piix e1000 libata virtio_pci serio_raw virtio_ring virtio drm_panel_orientation_quirks floppy
CPU: 4 PID: 29 Comm: ksoftirqd/4 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
task: ffff9f467a265230 ti: ffff9f467a2a4000 task.ti: ffff9f467a2a4000
RIP: 0010:[<ffffffffc03d9877>]  [<ffffffffc03d9877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9f467a2a79b0  EFLAGS: 00000286
RAX: ffffb6d3c1c03818 RBX: 0000000000000001 RCX: 0000000000000094
RDX: 0000000000000094 RSI: ffff9f45369abdc0 RDI: ffff9f49151fc000
RBP: ffff9f467a2a7a30 R08: 0000000000000100 R09: 0000000026100000
R10: 0000004500089a06 R11: 0000000000000000 R12: 0000000000000202
R13: 0000000386b048ce R14: 000000017a2a7920 R15: ffffffff9b5acd80
FS:  0000000000000000(0000) GS:ffff9f4927d00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1802f16cfb CR3: 00000000b9e10000 CR4: 00000000000006e0
Call Trace:
[<ffffffff9b84f346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff9b87d0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffff9b8522b1>] __dev_queue_xmit+0x491/0x650
[<ffffffff9b852480>] dev_queue_xmit+0x10/0x20
[<ffffffff9b8a0893>] ip_finish_output+0x593/0x8d0
[<ffffffff9b8a0efb>] ip_output+0x7b/0xf0
[<ffffffff9b8a0300>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff9b89e927>] ip_local_out_sk+0x37/0x40
[<ffffffff9b89ecc4>] ip_queue_xmit+0x144/0x3c0
[<ffffffff9b8b9374>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff9b8b99fa>] tcp_write_xmit+0x18a/0xd40
[<ffffffff9b8ba5e5>] tcp_tsq_handler.part.33+0x35/0x40
[<ffffffff9b8ba7c8>] tcp_tasklet_func+0x108/0x140
[<ffffffff9b2a5153>] tasklet_action+0x83/0x140
[<ffffffff9b2a5305>] __do_softirq+0xf5/0x280
[<ffffffff9b2a54c8>] run_ksoftirqd+0x38/0x50
[<ffffffff9b2cec24>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff9b2ceae0>] ? lg_double_unlock+0x40/0x40
[<ffffffff9b2c61f1>] kthread+0xd1/0xe0
[<ffffffff9b2c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffff9b98dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9b2c6120>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00