Info

Permalink
Function
e1000_xmit_frame
First occurrence
2021-01-28
Last occurrence
2021-01-28
State
NEW
Components
kernel
Quality
-92

Statistics

Operating system Count
CentOS 7 112
Architecture Count
x86_64 112
Related packages Count
kernel 112
 0:3.10.0-1160.6.1.el7 50
 0:3.10.0-1160.el7 15
 0:3.10.0-1127.10.1.el7 12
 0:3.10.0-1062.el7 6
 0:3.10.0-862.el7 6
 0:3.10.0-1160.76.1.el7 4
 0:3.10.0-1160.119.1.el7 3
 0:3.10.0-1160.62.1.el7 3
 0:3.10.0-1127.el7 2
 0:3.10.0-1062.9.1.el7 2
 0:3.10.0-1160.53.1.el7 2
 0:3.10.0-1160.71.1.el7 2
 0:3.10.0-1160.83.1.el7 2
 0:3.10.0-1160.99.1.el7 1
 0:3.10.0-957.el7 1
 0:3.10.0-1160.81.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1179609
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0352877
Build id: 3.10.0-1062.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffa2e4d346
Build id: 3.10.0-1062.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffa2e7b0ba
Build id: 3.10.0-1062.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffa2e502b1
Build id: 3.10.0-1062.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffffa2e50480
Build id: 3.10.0-1062.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffffa2e9e97b
Build id: 3.10.0-1062.el7.x86_64
0x67b
7
ip_output
vmlinux 0xffffffffa2e9eefb
Build id: 3.10.0-1062.el7.x86_64
0x7b
8
? internal_add_timer
vmlinux 0xffffffffa28acd8d
Build id: 3.10.0-1062.el7.x86_64
0x7d
9
ip_local_out_sk
vmlinux 0xffffffffa2e9c927
Build id: 3.10.0-1062.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffffa2e9ccc4
Build id: 3.10.0-1062.el7.x86_64
0x144
11
tcp_transmit_skb
vmlinux 0xffffffffa2eb7304
Build id: 3.10.0-1062.el7.x86_64
0x4e4
12
tcp_send_ack
vmlinux 0xffffffffa2eb94ff
Build id: 3.10.0-1062.el7.x86_64
0x11f
13
__tcp_ack_snd_check
vmlinux 0xffffffffa2eae15b
Build id: 3.10.0-1062.el7.x86_64
0x6b
14
tcp_rcv_established
vmlinux 0xffffffffa2eb3c74
Build id: 3.10.0-1062.el7.x86_64
0x264
15
tcp_v4_do_rcv
vmlinux 0xffffffffa2ebecda
Build id: 3.10.0-1062.el7.x86_64
0x10a
16
tcp_v4_rcv
vmlinux 0xffffffffa2ec03ec
Build id: 3.10.0-1062.el7.x86_64
0x7dc
17
? __udp4_lib_rcv
vmlinux 0xffffffffa2ecbe78
Build id: 3.10.0-1062.el7.x86_64
0x578
18
ip_local_deliver_finish
vmlinux 0xffffffffa2e9886d
Build id: 3.10.0-1062.el7.x86_64
0xbd
19
ip_local_deliver
vmlinux 0xffffffffa2e98b60
Build id: 3.10.0-1062.el7.x86_64
0x60
20
? tcp_v4_early_demux
vmlinux 0xffffffffa2ebfbc2
Build id: 3.10.0-1062.el7.x86_64
0x112
21
ip_rcv_finish
vmlinux 0xffffffffa2e984d0
Build id: 3.10.0-1062.el7.x86_64
0x90
22
ip_rcv
vmlinux 0xffffffffa2e98ea0
Build id: 3.10.0-1062.el7.x86_64
0x2c0
23
? udp4_lib_lookup_skb
vmlinux 0xffffffffa2ecb111
Build id: 3.10.0-1062.el7.x86_64
0x61
24
? udp_gro_receive
vmlinux 0xffffffffa2ecd0f7
Build id: 3.10.0-1062.el7.x86_64
0x57
25
__netif_receive_skb_core
vmlinux 0xffffffffa2e4dbd9
Build id: 3.10.0-1062.el7.x86_64
0x729
26
? tcp4_gro_receive
vmlinux 0xffffffffa2ec5500
Build id: 3.10.0-1062.el7.x86_64
0x90
27
? __getnstimeofday64
vmlinux 0xffffffffa2905d5f
Build id: 3.10.0-1062.el7.x86_64
0x3f
28
__netif_receive_skb
vmlinux 0xffffffffa2e4ded8
Build id: 3.10.0-1062.el7.x86_64
0x18
29
netif_receive_skb_internal
vmlinux 0xffffffffa2e4df60
Build id: 3.10.0-1062.el7.x86_64
0x40
30
napi_gro_receive
vmlinux 0xffffffffa2e4ebe8
Build id: 3.10.0-1062.el7.x86_64
0xd8
31
e1000_clean_rx_irq
e1000 0xffffffffc035345b
Build id: 3.10.0-1062.el7.x86_64
0x2bb
32
e1000_clean
e1000 0xffffffffc0354100
Build id: 3.10.0-1062.el7.x86_64
0x280
33
? update_curr
vmlinux 0xffffffffa28e375c
Build id: 3.10.0-1062.el7.x86_64
0x14c
34
net_rx_action
vmlinux 0xffffffffa2e4e57f
Build id: 3.10.0-1062.el7.x86_64
0x26f
35
__do_softirq
vmlinux 0xffffffffa28a41e5
Build id: 3.10.0-1062.el7.x86_64
0xf5
36
run_ksoftirqd
vmlinux 0xffffffffa28a43a8
Build id: 3.10.0-1062.el7.x86_64
0x38
37
smpboot_thread_fn
vmlinux 0xffffffffa28cdb04
Build id: 3.10.0-1062.el7.x86_64
0x144
38
? lg_double_unlock
vmlinux 0xffffffffa28cd9c0
Build id: 3.10.0-1062.el7.x86_64
0x40
39
kthread
vmlinux 0xffffffffa28c50d1
Build id: 3.10.0-1062.el7.x86_64
0xd1
40
? insert_kthread_work
vmlinux 0xffffffffa28c5000
Build id: 3.10.0-1062.el7.x86_64
0x40
41
ret_from_fork_nospec_begin
vmlinux 0xffffffffa2f8bd37
Build id: 3.10.0-1062.el7.x86_64
0x21
42
? insert_kthread_work
vmlinux 0xffffffffa28c5000
Build id: 3.10.0-1062.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [ksoftirqd/2:19]
Modules linked in: vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon btusb snd_ens1371 btrtl btbcm btintel aesni_intel snd_rawmidi bluetooth lrw gf128mul glue_helper snd_ac97_codec ablk_helper cryptd ac97_bus snd_seq snd_seq_device joydev pcspkr snd_pcm rfkill sg snd_timer snd soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi libata scsi_transport_spi mptscsih e1000 crct10dif_pclmul mptbase crct10dif_common crc32c_intel nfit serio_raw libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8892f7235230 ti: ffff8892f7258000 task.ti: ffff8892f7258000
RIP: 0010:[<ffffffffc0352877>]  [<ffffffffc0352877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8892f725b550  EFLAGS: 00000286
RAX: ffff9546806c3818 RBX: 0000000000000010 RCX: 0000000000000034
RDX: 0000000000000034 RSI: ffff8892f283cec0 RDI: ffff8892f511e000
RBP: ffff8892f725b5d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: ffffffffffffff10
R13: ffff8892f511e000 R14: ffff8892f283cec0 R15: 0000000000000032
FS:  0000000000000000(0000) GS:ffff8892f9680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5fe4b16c20 CR3: 00000000b223a000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffa2e4d346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa2e7b0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffffa2e502b1>] __dev_queue_xmit+0x491/0x650
[<ffffffffa2e50480>] dev_queue_xmit+0x10/0x20
[<ffffffffa2e9e97b>] ip_finish_output+0x67b/0x8d0
[<ffffffffa2e9eefb>] ip_output+0x7b/0xf0
[<ffffffffa28acd8d>] ? internal_add_timer+0x7d/0xc0
[<ffffffffa2e9c927>] ip_local_out_sk+0x37/0x40
[<ffffffffa2e9ccc4>] ip_queue_xmit+0x144/0x3c0
[<ffffffffa2eb7304>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffa2eb94ff>] tcp_send_ack+0x11f/0x170
[<ffffffffa2eae15b>] __tcp_ack_snd_check+0x6b/0x90
[<ffffffffa2eb3c74>] tcp_rcv_established+0x264/0x770
[<ffffffffa2ebecda>] tcp_v4_do_rcv+0x10a/0x350
[<ffffffffa2ec03ec>] tcp_v4_rcv+0x7dc/0x9e0
[<ffffffffa2ecbe78>] ? __udp4_lib_rcv+0x578/0xbe0
[<ffffffffa2e9886d>] ip_local_deliver_finish+0xbd/0x200
[<ffffffffa2e98b60>] ip_local_deliver+0x60/0xe0
[<ffffffffa2ebfbc2>] ? tcp_v4_early_demux+0x112/0x160
[<ffffffffa2e984d0>] ip_rcv_finish+0x90/0x370
[<ffffffffa2e98ea0>] ip_rcv+0x2c0/0x420
[<ffffffffa2ecb111>] ? udp4_lib_lookup_skb+0x61/0x70
[<ffffffffa2ecd0f7>] ? udp_gro_receive+0x57/0x170
[<ffffffffa2e4dbd9>] __netif_receive_skb_core+0x729/0xa10
[<ffffffffa2ec5500>] ? tcp4_gro_receive+0x90/0x1a0
[<ffffffffa2905d5f>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffffa2e4ded8>] __netif_receive_skb+0x18/0x60
[<ffffffffa2e4df60>] netif_receive_skb_internal+0x40/0xc0
[<ffffffffa2e4ebe8>] napi_gro_receive+0xd8/0x100
[<ffffffffc035345b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc0354100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffa28e375c>] ? update_curr+0x14c/0x1e0
[<ffffffffa2e4e57f>] net_rx_action+0x26f/0x390
[<ffffffffa28a41e5>] __do_softirq+0xf5/0x280
[<ffffffffa28a43a8>] run_ksoftirqd+0x38/0x50
[<ffffffffa28cdb04>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa28cd9c0>] ? lg_double_unlock+0x40/0x40
[<ffffffffa28c50d1>] kthread+0xd1/0xe0
[<ffffffffa28c5000>] ? insert_kthread_work+0x40/0x40
[<ffffffffa2f8bd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa28c5000>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:37:crtc-0] flip_done timed out
Complete report #991389
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc03ca8e7
Build id: 3.10.0-1160.81.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff9387edc6
Build id: 3.10.0-1160.81.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff938add72
Build id: 3.10.0-1160.81.1.el7.x86_64
0x122
4
__dev_queue_xmit
vmlinux 0xffffffff93881dd1
Build id: 3.10.0-1160.81.1.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffff93881fa0
Build id: 3.10.0-1160.81.1.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffff938d260b
Build id: 3.10.0-1160.81.1.el7.x86_64
0x67b
7
ip_output
vmlinux 0xffffffff938d2b9b
Build id: 3.10.0-1160.81.1.el7.x86_64
0x7b
8
? __wake_up_common
vmlinux 0xffffffff932d8cb2
Build id: 3.10.0-1160.81.1.el7.x86_64
0x82
9
ip_local_out_sk
vmlinux 0xffffffff938d0587
Build id: 3.10.0-1160.81.1.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffff938d0924
Build id: 3.10.0-1160.81.1.el7.x86_64
0x144
11
tcp_transmit_skb
vmlinux 0xffffffff938eb5f4
Build id: 3.10.0-1160.81.1.el7.x86_64
0x4e4
12
tcp_send_ack
vmlinux 0xffffffff938ed83f
Build id: 3.10.0-1160.81.1.el7.x86_64
0x11f
13
__tcp_ack_snd_check
vmlinux 0xffffffff938e1c6b
Build id: 3.10.0-1160.81.1.el7.x86_64
0x6b
14
tcp_rcv_established
vmlinux 0xffffffff938e7f85
Build id: 3.10.0-1160.81.1.el7.x86_64
0x425
15
tcp_v4_do_rcv
vmlinux 0xffffffff938f30f2
Build id: 3.10.0-1160.81.1.el7.x86_64
0x112
16
tcp_v4_rcv
vmlinux 0xffffffff938f498c
Build id: 3.10.0-1160.81.1.el7.x86_64
0x7dc
17
ip_local_deliver_finish
vmlinux 0xffffffff938cc34d
Build id: 3.10.0-1160.81.1.el7.x86_64
0xbd
18
ip_local_deliver
vmlinux 0xffffffff938cc660
Build id: 3.10.0-1160.81.1.el7.x86_64
0x60
19
? tcp_v4_early_demux
vmlinux 0xffffffff938f4162
Build id: 3.10.0-1160.81.1.el7.x86_64
0x122
20
ip_rcv_finish
vmlinux 0xffffffff938cbfa0
Build id: 3.10.0-1160.81.1.el7.x86_64
0x90
21
ip_rcv
vmlinux 0xffffffff938cc9a0
Build id: 3.10.0-1160.81.1.el7.x86_64
0x2c0
22
? udp4_lib_lookup_skb
vmlinux 0xffffffff938ff941
Build id: 3.10.0-1160.81.1.el7.x86_64
0x61
23
? udp_gro_receive
vmlinux 0xffffffff939019c7
Build id: 3.10.0-1160.81.1.el7.x86_64
0x57
24
__netif_receive_skb_core
vmlinux 0xffffffff9387f669
Build id: 3.10.0-1160.81.1.el7.x86_64
0x739
25
? tcp4_gro_receive
vmlinux 0xffffffff938f9b00
Build id: 3.10.0-1160.81.1.el7.x86_64
0x30
26
? inet_gro_receive
vmlinux 0xffffffff9390bcf9
Build id: 3.10.0-1160.81.1.el7.x86_64
0x249
27
__netif_receive_skb
vmlinux 0xffffffff9387f968
Build id: 3.10.0-1160.81.1.el7.x86_64
0x18
28
netif_receive_skb_internal
vmlinux 0xffffffff9387fa00
Build id: 3.10.0-1160.81.1.el7.x86_64
0x40
29
napi_gro_receive
vmlinux 0xffffffff938806b8
Build id: 3.10.0-1160.81.1.el7.x86_64
0xd8
30
e1000_clean_rx_irq
e1000 0xffffffffc03cb4cb
Build id: 3.10.0-1160.81.1.el7.x86_64
0x2bb
31
e1000_clean
e1000 0xffffffffc03cc180
Build id: 3.10.0-1160.81.1.el7.x86_64
0x280
32
? mod_timer
vmlinux 0xffffffff932b3a7c
Build id: 3.10.0-1160.81.1.el7.x86_64
0x10c
33
net_rx_action
vmlinux 0xffffffff93880037
Build id: 3.10.0-1160.81.1.el7.x86_64
0x277
34
__do_softirq
vmlinux 0xffffffff932a9545
Build id: 3.10.0-1160.81.1.el7.x86_64
0xf5
35
run_ksoftirqd
vmlinux 0xffffffff932a9718
Build id: 3.10.0-1160.81.1.el7.x86_64
0x38
36
smpboot_thread_fn
vmlinux 0xffffffff932d42ac
Build id: 3.10.0-1160.81.1.el7.x86_64
0x14c
37
? lg_double_unlock
vmlinux 0xffffffff932d4160
Build id: 3.10.0-1160.81.1.el7.x86_64
0x40
38
kthread
vmlinux 0xffffffff932cb4d1
Build id: 3.10.0-1160.81.1.el7.x86_64
0xd1
39
? insert_kthread_work
vmlinux 0xffffffff932cb400
Build id: 3.10.0-1160.81.1.el7.x86_64
0x40
40
ret_from_fork_nospec_begin
vmlinux 0xffffffff939c51dd
Build id: 3.10.0-1160.81.1.el7.x86_64
0x7
41
? insert_kthread_work
vmlinux 0xffffffff932cb400
Build id: 3.10.0-1160.81.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [ksoftirqd/7:44]
Modules linked in: vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event vfat fat snd_ens1371 iosf_mbi crc32_pclmul ghash_clmulni_intel snd_rawmidi snd_ac97_codec aesni_intel lrw gf128mul ac97_bus snd_seq snd_seq_device snd_pcm glue_helper ablk_helper snd_timer snd cryptd sg soundcore vmw_balloon ppdev pcspkr tpm_crb joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci ata_piix libahci libata mptsas nfit scsi_transport_sas crct10dif_pclmul crct10dif_common libnvdimm crc32c_intel mptscsih e1000 mptbase serio_raw drm_panel_orientation_quirks fuse
CPU: 7 PID: 44 Comm: ksoftirqd/7 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.14410784.B64.1908150010 08/15/2019
task: ffff9bb2359d2100 ti: ffff9bb2359e8000 task.ti: ffff9bb2359e8000
RIP: 0010:[<ffffffffc03ca8e7>]  [<ffffffffc03ca8e7>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9bb2359eb550  EFLAGS: 00000286
RAX: ffffb362c1bc3818 RBX: 0000000000000001 RCX: 000000000000003a
RDX: 000000000000003a RSI: ffff9bb4ea2a4e80 RDI: ffff9bb4ecb60000
RBP: ffff9bb2359eb5d0 R08: 0000000000000000 R09: 0000000022100000
R10: 3400004500083338 R11: 0000000000000000 R12: 0000000000000202
R13: 00000004288c58ce R14: 00000001ea2a4e80 R15: ffffffff935c8a90
FS:  0000000000000000(0000) GS:ffff9bb4ed7c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6320de2080 CR3: 000000042b258000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffff9387edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff938add72>] sch_direct_xmit+0x122/0x250
[<ffffffff93881dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffff93881fa0>] dev_queue_xmit+0x10/0x20
[<ffffffff938d260b>] ip_finish_output+0x67b/0x8d0
[<ffffffff938d2b9b>] ip_output+0x7b/0x100
[<ffffffff932d8cb2>] ? __wake_up_common+0x82/0x120
[<ffffffff938d0587>] ip_local_out_sk+0x37/0x40
[<ffffffff938d0924>] ip_queue_xmit+0x144/0x3d0
[<ffffffff938eb5f4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff938ed83f>] tcp_send_ack+0x11f/0x180
[<ffffffff938e1c6b>] __tcp_ack_snd_check+0x6b/0xa0
[<ffffffff938e7f85>] tcp_rcv_established+0x425/0x7a0
[<ffffffff938f30f2>] tcp_v4_do_rcv+0x112/0x360
[<ffffffff938f498c>] tcp_v4_rcv+0x7dc/0x9e0
[<ffffffff938cc34d>] ip_local_deliver_finish+0xbd/0x210
[<ffffffff938cc660>] ip_local_deliver+0x60/0xe0
[<ffffffff938f4162>] ? tcp_v4_early_demux+0x122/0x170
[<ffffffff938cbfa0>] ip_rcv_finish+0x90/0x380
[<ffffffff938cc9a0>] ip_rcv+0x2c0/0x420
[<ffffffff938ff941>] ? udp4_lib_lookup_skb+0x61/0x70
[<ffffffff939019c7>] ? udp_gro_receive+0x57/0x180
[<ffffffff9387f669>] __netif_receive_skb_core+0x739/0xa20
[<ffffffff938f9b00>] ? tcp4_gro_receive+0x30/0x1a0
[<ffffffff9390bcf9>] ? inet_gro_receive+0x249/0x290
[<ffffffff9387f968>] __netif_receive_skb+0x18/0x70
[<ffffffff9387fa00>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff938806b8>] napi_gro_receive+0xd8/0x100
[<ffffffffc03cb4cb>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc03cc180>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff932b3a7c>] ? mod_timer+0x10c/0x230
[<ffffffff93880037>] net_rx_action+0x277/0x3a0
[<ffffffff932a9545>] __do_softirq+0xf5/0x290
[<ffffffff932a9718>] run_ksoftirqd+0x38/0x60
[<ffffffff932d42ac>] smpboot_thread_fn+0x14c/0x1b0
[<ffffffff932d4160>] ? lg_double_unlock+0x40/0x40
[<ffffffff932cb4d1>] kthread+0xd1/0xe0
[<ffffffff932cb400>] ? insert_kthread_work+0x40/0x40
[<ffffffff939c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff932cb400>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc 
Complete report #987370
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc03a4857
Build id: 3.10.0-957.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffa7a38c66
Build id: 3.10.0-957.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffa7a65cea
Build id: 3.10.0-957.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffa7a3bbe1
Build id: 3.10.0-957.el7.x86_64
0x491
5
? local_bh_enable
vmlinux 0xffffffffa74a0387
Build id: 3.10.0-957.el7.x86_64
0x17
6
dev_queue_xmit
vmlinux 0xffffffffa7a3bdb0
Build id: 3.10.0-957.el7.x86_64
0x10
7
neigh_resolve_output
vmlinux 0xffffffffa7a4861d
Build id: 3.10.0-957.el7.x86_64
0x11d
8
ip_finish_output
vmlinux 0xffffffffa7a8845c
Build id: 3.10.0-957.el7.x86_64
0x27c
9
ip_output
vmlinux 0xffffffffa7a88c93
Build id: 3.10.0-957.el7.x86_64
0x73
10
? __ip_append_data.isra.51
vmlinux 0xffffffffa7a881e0
Build id: 3.10.0-957.el7.x86_64
0xa50
11
ip_local_out_sk
vmlinux 0xffffffffa7a86817
Build id: 3.10.0-957.el7.x86_64
0x37
12
ip_queue_xmit
vmlinux 0xffffffffa7a86bb4
Build id: 3.10.0-957.el7.x86_64
0x144
13
tcp_transmit_skb
vmlinux 0xffffffffa7aa0f64
Build id: 3.10.0-957.el7.x86_64
0x4e4
14
tcp_send_ack
vmlinux 0xffffffffa7aa2fef
Build id: 3.10.0-957.el7.x86_64
0x11f
15
__tcp_ack_snd_check
vmlinux 0xffffffffa7a9796b
Build id: 3.10.0-957.el7.x86_64
0x6b
16
tcp_rcv_established
vmlinux 0xffffffffa7a9d944
Build id: 3.10.0-957.el7.x86_64
0x264
17
tcp_v4_do_rcv
vmlinux 0xffffffffa7aa86da
Build id: 3.10.0-957.el7.x86_64
0x10a
18
tcp_v4_rcv
vmlinux 0xffffffffa7aa9dcc
Build id: 3.10.0-957.el7.x86_64
0x7bc
19
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc07642ce
Build id: 3.10.0-957.el7.x86_64
0x4e
20
ip_local_deliver_finish
vmlinux 0xffffffffa7a8285d
Build id: 3.10.0-957.el7.x86_64
0xbd
21
ip_local_deliver
vmlinux 0xffffffffa7a82b49
Build id: 3.10.0-957.el7.x86_64
0x59
22
? ip_rcv_finish
vmlinux 0xffffffffa7a827a0
Build id: 3.10.0-957.el7.x86_64
0x370
23
ip_rcv_finish
vmlinux 0xffffffffa7a824c0
Build id: 3.10.0-957.el7.x86_64
0x90
24
ip_rcv
vmlinux 0xffffffffa7a82e79
Build id: 3.10.0-957.el7.x86_64
0x2b9
25
? inet_del_offload
vmlinux 0xffffffffa7a82430
Build id: 3.10.0-957.el7.x86_64
0x40
26
__netif_receive_skb_core
vmlinux 0xffffffffa7a394f9
Build id: 3.10.0-957.el7.x86_64
0x729
27
? tcp4_gro_receive
vmlinux 0xffffffffa7aaee00
Build id: 3.10.0-957.el7.x86_64
0x80
28
? inet_gro_receive
vmlinux 0xffffffffa7ac06c9
Build id: 3.10.0-957.el7.x86_64
0x249
29
__netif_receive_skb
vmlinux 0xffffffffa7a397f8
Build id: 3.10.0-957.el7.x86_64
0x18
30
netif_receive_skb_internal
vmlinux 0xffffffffa7a39880
Build id: 3.10.0-957.el7.x86_64
0x40
31
napi_gro_receive
vmlinux 0xffffffffa7a3a508
Build id: 3.10.0-957.el7.x86_64
0xd8
32
e1000_clean_rx_irq
e1000 0xffffffffc03a543b
Build id: 3.10.0-957.el7.x86_64
0x2bb
33
e1000_clean
e1000 0xffffffffc03a60dd
Build id: 3.10.0-957.el7.x86_64
0x27d
34
? tcp_v4_md5_lookup
vmlinux 0xffffffffa7aa5ac3
Build id: 3.10.0-957.el7.x86_64
0x13
35
? update_curr
vmlinux 0xffffffffa74e00bc
Build id: 3.10.0-957.el7.x86_64
0x14c
36
net_rx_action
vmlinux 0xffffffffa7a39e9f
Build id: 3.10.0-957.el7.x86_64
0x26f
37
__do_softirq
vmlinux 0xffffffffa74a0f05
Build id: 3.10.0-957.el7.x86_64
0xf5
38
run_ksoftirqd
vmlinux 0xffffffffa74a10c8
Build id: 3.10.0-957.el7.x86_64
0x38
39
smpboot_thread_fn
vmlinux 0xffffffffa74ca5c4
Build id: 3.10.0-957.el7.x86_64
0x144
40
? lg_double_unlock
vmlinux 0xffffffffa74ca480
Build id: 3.10.0-957.el7.x86_64
0x40
41
kthread
vmlinux 0xffffffffa74c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
42
? insert_kthread_work
vmlinux 0xffffffffa74c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
43
ret_from_fork_nospec_begin
vmlinux 0xffffffffa7b74c37
Build id: 3.10.0-957.el7.x86_64
0x21
44
? insert_kthread_work
vmlinux 0xffffffffa74c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#23 stuck for 23s! [ksoftirqd/23:124]
Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink bluetooth rfkill xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter binfmt_misc sunrpc ppdev parport_pc parport sg i2c_piix4 virtio_balloon pcspkr joydev ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_blk virtio_console cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt floppy fb_sys_fops ttm ata_piix drm libata crc32c_intel e1000 serio_raw virtio_pci drm_panel_orientation_quirks virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 23 PID: 124 Comm: ksoftirqd/23 Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: Bochs Bochs, BIOS rel-1.7.5.1-20180921_224418 04/01/2014
task: ffff8cd8291a0000 ti: ffff8cd82919c000 task.ti: ffff8cd82919c000
RIP: 0010:[<ffffffffc03a4857>]  [<ffffffffc03a4857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8cd82919f540  EFLAGS: 00000286
RAX: ffffa0a946a43818 RBX: 0000000000000001 RCX: 000000000000003a
RDX: 000000000000003a RSI: ffff8ce6ffe915c0 RDI: ffff8ce6be39e000
RBP: ffff8cd82919f5c0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: 0000000000000202
R13: 0000000e973574da R14: 00000001be39e000 R15: ffffffffa77a04b0
FS:  0000000000000000(0000) GS:ffff8ce6bf3c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000f2a2c8 CR3: 0000000f5de10000 CR4: 00000000000206e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa7a38c66>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa7a65cea>] sch_direct_xmit+0x11a/0x250
[<ffffffffa7a3bbe1>] __dev_queue_xmit+0x491/0x650
[<ffffffffa74a0387>] ? local_bh_enable+0x17/0x20
[<ffffffffa7a3bdb0>] dev_queue_xmit+0x10/0x20
[<ffffffffa7a4861d>] neigh_resolve_output+0x11d/0x220
[<ffffffffa7a8845c>] ip_finish_output+0x27c/0x7b0
[<ffffffffa7a88c93>] ip_output+0x73/0xe0
[<ffffffffa7a881e0>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffa7a86817>] ip_local_out_sk+0x37/0x40
[<ffffffffa7a86bb4>] ip_queue_xmit+0x144/0x3c0
[<ffffffffa7aa0f64>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffa7aa2fef>] tcp_send_ack+0x11f/0x170
[<ffffffffa7a9796b>] __tcp_ack_snd_check+0x6b/0x90
[<ffffffffa7a9d944>] tcp_rcv_established+0x264/0x770
[<ffffffffa7aa86da>] tcp_v4_do_rcv+0x10a/0x350
[<ffffffffa7aa9dcc>] tcp_v4_rcv+0x7bc/0x9c0
[<ffffffffc07642ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffffa7a8285d>] ip_local_deliver_finish+0xbd/0x200
[<ffffffffa7a82b49>] ip_local_deliver+0x59/0xd0
[<ffffffffa7a827a0>] ? ip_rcv_finish+0x370/0x370
[<ffffffffa7a824c0>] ip_rcv_finish+0x90/0x370
[<ffffffffa7a82e79>] ip_rcv+0x2b9/0x410
[<ffffffffa7a82430>] ? inet_del_offload+0x40/0x40
[<ffffffffa7a394f9>] __netif_receive_skb_core+0x729/0xa10
[<ffffffffa7aaee00>] ? tcp4_gro_receive+0x80/0x1a0
[<ffffffffa7ac06c9>] ? inet_gro_receive+0x249/0x290
[<ffffffffa7a397f8>] __netif_receive_skb+0x18/0x60
[<ffffffffa7a39880>] netif_receive_skb_internal+0x40/0xc0
[<ffffffffa7a3a508>] napi_gro_receive+0xd8/0x100
[<ffffffffc03a543b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc03a60dd>] e1000_clean+0x27d/0x8e0 [e1000]
[<ffffffffa7aa5ac3>] ? tcp_v4_md5_lookup+0x13/0x20
[<ffffffffa74e00bc>] ? update_curr+0x14c/0x1e0
[<ffffffffa7a39e9f>] net_rx_action+0x26f/0x390
[<ffffffffa74a0f05>] __do_softirq+0xf5/0x280
[<ffffffffa74a10c8>] run_ksoftirqd+0x38/0x50
[<ffffffffa74ca5c4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa74ca480>] ? lg_double_unlock+0x40/0x40
[<ffffffffa74c1c31>] kthread+0xd1/0xe0
[<ffffffffa74c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa7b74c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa74c1b60>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #765297
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc02e7877
Build id: 3.10.0-1062.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff9084d346
Build id: 3.10.0-1062.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff9087b0ba
Build id: 3.10.0-1062.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff908502b1
Build id: 3.10.0-1062.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffff90850480
Build id: 3.10.0-1062.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffff9089e97b
Build id: 3.10.0-1062.el7.x86_64
0x67b
7
ip_output
vmlinux 0xffffffff9089eefb
Build id: 3.10.0-1062.el7.x86_64
0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffff9089e300
Build id: 3.10.0-1062.el7.x86_64
0xa60
9
ip_local_out_sk
vmlinux 0xffffffff9089c927
Build id: 3.10.0-1062.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffff9089ccc4
Build id: 3.10.0-1062.el7.x86_64
0x144
11
tcp_transmit_skb
vmlinux 0xffffffff908b7304
Build id: 3.10.0-1062.el7.x86_64
0x4e4
12
tcp_send_ack
vmlinux 0xffffffff908b94ff
Build id: 3.10.0-1062.el7.x86_64
0x11f
13
__tcp_ack_snd_check
vmlinux 0xffffffff908ae15b
Build id: 3.10.0-1062.el7.x86_64
0x6b
14
tcp_rcv_state_process
vmlinux 0xffffffff908b4e36
Build id: 3.10.0-1062.el7.x86_64
0xa36
15
tcp_v4_do_rcv
vmlinux 0xffffffff908bed4b
Build id: 3.10.0-1062.el7.x86_64
0x17b
16
tcp_v4_rcv
vmlinux 0xffffffff908c03ec
Build id: 3.10.0-1062.el7.x86_64
0x7dc
17
ip_local_deliver_finish
vmlinux 0xffffffff9089886d
Build id: 3.10.0-1062.el7.x86_64
0xbd
18
ip_local_deliver
vmlinux 0xffffffff90898b60
Build id: 3.10.0-1062.el7.x86_64
0x60
19
? tcp_v4_early_demux
vmlinux 0xffffffff908bfbc2
Build id: 3.10.0-1062.el7.x86_64
0x112
20
ip_rcv_finish
vmlinux 0xffffffff908984d0
Build id: 3.10.0-1062.el7.x86_64
0x90
21
ip_rcv
vmlinux 0xffffffff90898ea0
Build id: 3.10.0-1062.el7.x86_64
0x2c0
22
? udp4_lib_lookup_skb
vmlinux 0xffffffff908cb111
Build id: 3.10.0-1062.el7.x86_64
0x61
23
? udp_gro_receive
vmlinux 0xffffffff908cd0f7
Build id: 3.10.0-1062.el7.x86_64
0x57
24
__netif_receive_skb_core
vmlinux 0xffffffff9084dbd9
Build id: 3.10.0-1062.el7.x86_64
0x729
25
? tcp4_gro_receive
vmlinux 0xffffffff908c5500
Build id: 3.10.0-1062.el7.x86_64
0x90
26
? __getnstimeofday64
vmlinux 0xffffffff90305d5f
Build id: 3.10.0-1062.el7.x86_64
0x3f
27
__netif_receive_skb
vmlinux 0xffffffff9084ded8
Build id: 3.10.0-1062.el7.x86_64
0x18
28
netif_receive_skb_internal
vmlinux 0xffffffff9084df60
Build id: 3.10.0-1062.el7.x86_64
0x40
29
napi_gro_receive
vmlinux 0xffffffff9084ebe8
Build id: 3.10.0-1062.el7.x86_64
0xd8
30
e1000_clean_rx_irq
e1000 0xffffffffc02e845b
Build id: 3.10.0-1062.el7.x86_64
0x2bb
31
e1000_clean
e1000 0xffffffffc02e9100
Build id: 3.10.0-1062.el7.x86_64
0x280
32
? update_curr
vmlinux 0xffffffff902e375c
Build id: 3.10.0-1062.el7.x86_64
0x14c
33
net_rx_action
vmlinux 0xffffffff9084e57f
Build id: 3.10.0-1062.el7.x86_64
0x26f
34
__do_softirq
vmlinux 0xffffffff902a41e5
Build id: 3.10.0-1062.el7.x86_64
0xf5
35
run_ksoftirqd
vmlinux 0xffffffff902a43a8
Build id: 3.10.0-1062.el7.x86_64
0x38
36
smpboot_thread_fn
vmlinux 0xffffffff902cdb04
Build id: 3.10.0-1062.el7.x86_64
0x144
37
? lg_double_unlock
vmlinux 0xffffffff902cd9c0
Build id: 3.10.0-1062.el7.x86_64
0x40
38
kthread
vmlinux 0xffffffff902c50d1
Build id: 3.10.0-1062.el7.x86_64
0xd1
39
? insert_kthread_work
vmlinux 0xffffffff902c5000
Build id: 3.10.0-1062.el7.x86_64
0x40
40
ret_from_fork_nospec_begin
vmlinux 0xffffffff9098bd37
Build id: 3.10.0-1062.el7.x86_64
0x21
41
? insert_kthread_work
vmlinux 0xffffffff902c5000
Build id: 3.10.0-1062.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ksoftirqd/0:6]
Modules linked in: bnep vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 btusb btrtl snd_rawmidi snd_ac97_codec ac97_bus snd_seq btbcm btintel snd_seq_device bluetooth snd_pcm snd_timer snd soundcore iosf_mbi crc32_pclmul ghash_clmulni_intel rfkill aesni_intel lrw gf128mul ppdev vmw_balloon glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common crc32c_intel ttm nfit mptspi ata_piix libnvdimm drm scsi_transport_spi serio_raw mptscsih libata e1000 mptbase drm_panel_orientation_quirks
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9fcfb64b5230 ti: ffff9fcfb6550000 task.ti: ffff9fcfb6550000
RIP: 0010:[<ffffffffc02e7877>]  [<ffffffffc02e7877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9fcfb6553520  EFLAGS: 00000286
RAX: ffffb26c431c3818 RBX: 0000000000000001 RCX: 0000000000000009
RDX: 0000000000000009 RSI: ffff9fd4612398c0 RDI: ffff9fd464dbe000
RBP: ffff9fcfb65535a0 R08: 0000000000000000 R09: 0000000022100000
R10: 66b6a8c0a3ee0640 R11: 0000000000000000 R12: 0000000000000202
R13: 000000061e3284c2 R14: 00000001000861e6 R15: ffffffff905aabc0
FS:  0000000000000000(0000) GS:ffff9fd465600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f51fc5f51a8 CR3: 00000005bb246000 CR4: 00000000003607f0
Call Trace:
[<ffffffff9084d346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff9087b0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffff908502b1>] __dev_queue_xmit+0x491/0x650
[<ffffffff90850480>] dev_queue_xmit+0x10/0x20
[<ffffffff9089e97b>] ip_finish_output+0x67b/0x8d0
[<ffffffff9089eefb>] ip_output+0x7b/0xf0
[<ffffffff9089e300>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff9089c927>] ip_local_out_sk+0x37/0x40
[<ffffffff9089ccc4>] ip_queue_xmit+0x144/0x3c0
[<ffffffff908b7304>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff908b94ff>] tcp_send_ack+0x11f/0x170
[<ffffffff908ae15b>] __tcp_ack_snd_check+0x6b/0x90
[<ffffffff908b4e36>] tcp_rcv_state_process+0xa36/0xf50
[<ffffffff908bed4b>] tcp_v4_do_rcv+0x17b/0x350
[<ffffffff908c03ec>] tcp_v4_rcv+0x7dc/0x9e0
[<ffffffff9089886d>] ip_local_deliver_finish+0xbd/0x200
[<ffffffff90898b60>] ip_local_deliver+0x60/0xe0
[<ffffffff908bfbc2>] ? tcp_v4_early_demux+0x112/0x160
[<ffffffff908984d0>] ip_rcv_finish+0x90/0x370
[<ffffffff90898ea0>] ip_rcv+0x2c0/0x420
[<ffffffff908cb111>] ? udp4_lib_lookup_skb+0x61/0x70
[<ffffffff908cd0f7>] ? udp_gro_receive+0x57/0x170
[<ffffffff9084dbd9>] __netif_receive_skb_core+0x729/0xa10
[<ffffffff908c5500>] ? tcp4_gro_receive+0x90/0x1a0
[<ffffffff90305d5f>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffff9084ded8>] __netif_receive_skb+0x18/0x60
[<ffffffff9084df60>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff9084ebe8>] napi_gro_receive+0xd8/0x100
[<ffffffffc02e845b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc02e9100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff902e375c>] ? update_curr+0x14c/0x1e0
[<ffffffff9084e57f>] net_rx_action+0x26f/0x390
[<ffffffff902a41e5>] __do_softirq+0xf5/0x280
[<ffffffff902a43a8>] run_ksoftirqd+0x38/0x50
[<ffffffff902cdb04>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff902cd9c0>] ? lg_double_unlock+0x40/0x40
[<ffffffff902c50d1>] kthread+0xd1/0xe0
[<ffffffff902c5000>] ? insert_kthread_work+0x40/0x40
[<ffffffff9098bd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff902c5000>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #556725
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc050f877
Build id: 3.10.0-1160.62.1.el7.x86_64
0x7f7
2
? e1000_xmit_frame
e1000 0xffffffffc050f877
Build id: 3.10.0-1160.62.1.el7.x86_64
0x7f7
3
dev_hard_start_xmit
vmlinux 0xffffffff88a58f86
Build id: 3.10.0-1160.62.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffff88a8709a
Build id: 3.10.0-1160.62.1.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffff88a5bf01
Build id: 3.10.0-1160.62.1.el7.x86_64
0x491
6
dev_queue_xmit
vmlinux 0xffffffff88a5c0d0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffff88aaac5b
Build id: 3.10.0-1160.62.1.el7.x86_64
0x67b
8
ip_output
vmlinux 0xffffffff88aab1db
Build id: 3.10.0-1160.62.1.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffff88aaa5e0
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffff88aa8c07
Build id: 3.10.0-1160.62.1.el7.x86_64
0x37
11
ip_queue_xmit
vmlinux 0xffffffff88aa8fa4
Build id: 3.10.0-1160.62.1.el7.x86_64
0x144
12
tcp_transmit_skb
vmlinux 0xffffffff88ac3724
Build id: 3.10.0-1160.62.1.el7.x86_64
0x4e4
13
tcp_send_ack
vmlinux 0xffffffff88ac593f
Build id: 3.10.0-1160.62.1.el7.x86_64
0x11f
14
__tcp_ack_snd_check
vmlinux 0xffffffff88ab9f2b
Build id: 3.10.0-1160.62.1.el7.x86_64
0x6b
15
tcp_rcv_state_process
vmlinux 0xffffffff88ac11d6
Build id: 3.10.0-1160.62.1.el7.x86_64
0xa36
16
tcp_v4_do_rcv
vmlinux 0xffffffff88acb0ab
Build id: 3.10.0-1160.62.1.el7.x86_64
0x17b
17
tcp_v4_rcv
vmlinux 0xffffffff88acc88c
Build id: 3.10.0-1160.62.1.el7.x86_64
0x7dc
18
? call_rcu_sched
vmlinux 0xffffffff88559bed
Build id: 3.10.0-1160.62.1.el7.x86_64
0x1d
19
? page_frag_free
vmlinux 0xffffffff885c7438
Build id: 3.10.0-1160.62.1.el7.x86_64
0x68
20
? skb_release_data
vmlinux 0xffffffff88a42722
Build id: 3.10.0-1160.62.1.el7.x86_64
0x142
21
? kfree_skbmem
vmlinux 0xffffffff88a416e7
Build id: 3.10.0-1160.62.1.el7.x86_64
0x37
22
ip_local_deliver_finish
vmlinux 0xffffffff88aa4a8d
Build id: 3.10.0-1160.62.1.el7.x86_64
0xbd
23
ip_local_deliver
vmlinux 0xffffffff88aa4d80
Build id: 3.10.0-1160.62.1.el7.x86_64
0x60
24
? tcp_v4_early_demux
vmlinux 0xffffffff88acc062
Build id: 3.10.0-1160.62.1.el7.x86_64
0x112
25
ip_rcv_finish
vmlinux 0xffffffff88aa46f0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x90
26
ip_rcv
vmlinux 0xffffffff88aa50c0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x2c0
27
? udp_gro_receive
vmlinux 0xffffffff88ad95c7
Build id: 3.10.0-1160.62.1.el7.x86_64
0x57
28
__netif_receive_skb_core
vmlinux 0xffffffff88a59819
Build id: 3.10.0-1160.62.1.el7.x86_64
0x729
29
? tcp4_gro_receive
vmlinux 0xffffffff88ad1a00
Build id: 3.10.0-1160.62.1.el7.x86_64
0xf0
30
? __getnstimeofday64
vmlinux 0xffffffff8850805f
Build id: 3.10.0-1160.62.1.el7.x86_64
0x3f
31
__netif_receive_skb
vmlinux 0xffffffff88a59b18
Build id: 3.10.0-1160.62.1.el7.x86_64
0x18
32
netif_receive_skb_internal
vmlinux 0xffffffff88a59ba0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x40
33
napi_gro_receive
vmlinux 0xffffffff88a5a828
Build id: 3.10.0-1160.62.1.el7.x86_64
0xd8
34
e1000_clean_rx_irq
e1000 0xffffffffc051045b
Build id: 3.10.0-1160.62.1.el7.x86_64
0x2bb
35
e1000_clean
e1000 0xffffffffc0511100
Build id: 3.10.0-1160.62.1.el7.x86_64
0x280
36
net_rx_action
vmlinux 0xffffffff88a5a1bf
Build id: 3.10.0-1160.62.1.el7.x86_64
0x26f
37
__do_softirq
vmlinux 0xffffffff884a4d85
Build id: 3.10.0-1160.62.1.el7.x86_64
0xf5
38
run_ksoftirqd
vmlinux 0xffffffff884a4f48
Build id: 3.10.0-1160.62.1.el7.x86_64
0x38
39
smpboot_thread_fn
vmlinux 0xffffffff884cea04
Build id: 3.10.0-1160.62.1.el7.x86_64
0x144
40
? lg_double_unlock
vmlinux 0xffffffff884ce8c0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x40
41
kthread
vmlinux 0xffffffff884c5fb1
Build id: 3.10.0-1160.62.1.el7.x86_64
0xd1
42
? insert_kthread_work
vmlinux 0xffffffff884c5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x40
43
ret_from_fork_nospec_begin
vmlinux 0xffffffff88b99ddd
Build id: 3.10.0-1160.62.1.el7.x86_64
0x7
44
? insert_kthread_work
vmlinux 0xffffffff884c5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 24s! [Timer-Zimbra:10292]
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ksoftirqd/0:6]
mptscsih: ioc0: attempting task abort! (sc=ffff90ee3c780380)
sd 0:0:0:0: [sda] CDB: Read(10) 28 00 10 0f b0 00 00 00 08 00
Modules linked in: snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 aesni_intel snd_rawmidi lrw xfs snd_ac97_codec gf128mul glue_helper ppdev ablk_helper libcrc32c vmw_balloon ac97_bus cryptd snd_seq snd_seq_device snd_pcm joydev pcspkr snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea crct10dif_pclmul sysfillrect crct10dif_common sysimgblt crc32c_intel fb_sys_fops ttm ata_piix drm e1000 libata mptspi nfit serio_raw scsi_transport_spi mptscsih libnvdimm mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6 Comm: ksoftirqd/0 Tainted: G             L ------------   3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff90edfafe5280 ti: ffff90edfa8a4000 task.ti: ffff90edfa8a4000
RIP: 0010:[<ffffffffc050f877>]  [<ffffffffc050f877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff90edfa8a7520  EFLAGS: 00000286
RAX: ffffa559410c3818 RBX: 0000000000002508 RCX: 00000000000000f5
RDX: 00000000000000f5 RSI: ffff90eeb1477140 RDI: ffff90edfae64000
RBP: ffff90edfa8a75a0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: ffffffff88a832d5
R13: ffffffff887b2a9c R14: ffff90edfa8a7490 R15: ffff90eeb1477140
FS:  0000000000000000(0000) GS:ffff90eeb5600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001cc41a0 CR3: 0000000071a10000 CR4: 00000000003607f0
Call Trace:
[<ffffffffc050f877>] ? e1000_xmit_frame+0x7f7/0x1120 [e1000]
[<ffffffff88a58f86>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff88a8709a>] sch_direct_xmit+0x11a/0x250
[<ffffffff88a5bf01>] __dev_queue_xmit+0x491/0x650
[<ffffffff88a5c0d0>] dev_queue_xmit+0x10/0x20
[<ffffffff88aaac5b>] ip_finish_output+0x67b/0x8d0
[<ffffffff88aab1db>] ip_output+0x7b/0xf0
[<ffffffff88aaa5e0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff88aa8c07>] ip_local_out_sk+0x37/0x40
[<ffffffff88aa8fa4>] ip_queue_xmit+0x144/0x3c0
[<ffffffff88ac3724>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff88ac593f>] tcp_send_ack+0x11f/0x170
[<ffffffff88ab9f2b>] __tcp_ack_snd_check+0x6b/0x90
[<ffffffff88ac11d6>] tcp_rcv_state_process+0xa36/0xf50
[<ffffffff88acb0ab>] tcp_v4_do_rcv+0x17b/0x350
[<ffffffff88acc88c>] tcp_v4_rcv+0x7dc/0x9e0
[<ffffffff88559bed>] ? call_rcu_sched+0x1d/0x20
[<ffffffff885c7438>] ? page_frag_free+0x68/0x80
[<ffffffff88a42722>] ? skb_release_data+0x142/0x150
[<ffffffff88a416e7>] ? kfree_skbmem+0x37/0x90
[<ffffffff88aa4a8d>] ip_local_deliver_finish+0xbd/0x200
[<ffffffff88aa4d80>] ip_local_deliver+0x60/0xe0
[<ffffffff88acc062>] ? tcp_v4_early_demux+0x112/0x160
[<ffffffff88aa46f0>] ip_rcv_finish+0x90/0x370
[<ffffffff88aa50c0>] ip_rcv+0x2c0/0x420
[<ffffffff88ad95c7>] ? udp_gro_receive+0x57/0x170
[<ffffffff88a59819>] __netif_receive_skb_core+0x729/0xa10
[<ffffffff88ad1a00>] ? tcp4_gro_receive+0xf0/0x1a0
[<ffffffff8850805f>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffff88a59b18>] __netif_receive_skb+0x18/0x60
[<ffffffff88a59ba0>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff88a5a828>] napi_gro_receive+0xd8/0x100
[<ffffffffc051045b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc0511100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff88a5a1bf>] net_rx_action+0x26f/0x390
[<ffffffff884a4d85>] __do_softirq+0xf5/0x280
[<ffffffff884a4f48>] run_ksoftirqd+0x38/0x50
[<ffffffff884cea04>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff884ce8c0>] ? lg_double_unlock+0x40/0x40
[<ffffffff884c5fb1>] kthread+0xd1/0xe0
[<ffffffff884c5ee0>] ? insert_kthread_work+0x40/0x40
[<ffffffff88b99ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff884c5ee0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #332930
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0126867
Build id: 3.10.0-862.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff9efeb166
Build id: 3.10.0-862.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff9f016a2a
Build id: 3.10.0-862.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff9efee051
Build id: 3.10.0-862.el7.x86_64
0x4a1
5
dev_queue_xmit
vmlinux 0xffffffff9efee220
Build id: 3.10.0-862.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffff9f037416
Build id: 3.10.0-862.el7.x86_64
0x546
7
ip_output
vmlinux 0xffffffff9f037973
Build id: 3.10.0-862.el7.x86_64
0x73
8
? __ip_local_out_sk
vmlinux 0xffffffff9f035526
Build id: 3.10.0-862.el7.x86_64
0xf6
9
ip_local_out_sk
vmlinux 0xffffffff9f035567
Build id: 3.10.0-862.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffff9f0358d3
Build id: 3.10.0-862.el7.x86_64
0x143
11
tcp_transmit_skb
vmlinux 0xffffffff9f04f7b4
Build id: 3.10.0-862.el7.x86_64
0x4e4
12
tcp_send_ack
vmlinux 0xffffffff9f05182f
Build id: 3.10.0-862.el7.x86_64
0x11f
13
__tcp_ack_snd_check
vmlinux 0xffffffff9f04600b
Build id: 3.10.0-862.el7.x86_64
0x6b
14
tcp_rcv_established
vmlinux 0xffffffff9f04c357
Build id: 3.10.0-862.el7.x86_64
0x417
15
? sock_def_readable
vmlinux 0xffffffff9efd005a
Build id: 3.10.0-862.el7.x86_64
0x3a
16
tcp_v4_do_rcv
vmlinux 0xffffffff9f056dda
Build id: 3.10.0-862.el7.x86_64
0x10a
17
tcp_v4_rcv
vmlinux 0xffffffff9f058568
Build id: 3.10.0-862.el7.x86_64
0x788
18
? iptable_filter_hook
iptable_filter 0xffffffffc04eb036
Build id: 3.10.0-862.el7.x86_64
0x36
19
ip_local_deliver_finish
vmlinux 0xffffffff9f031699
Build id: 3.10.0-862.el7.x86_64
0xb9
20
ip_local_deliver
vmlinux 0xffffffff9f031989
Build id: 3.10.0-862.el7.x86_64
0x59
21
? ip_rcv_finish
vmlinux 0xffffffff9f0315e0
Build id: 3.10.0-862.el7.x86_64
0x370
22
ip_rcv_finish
vmlinux 0xffffffff9f031300
Build id: 3.10.0-862.el7.x86_64
0x90
23
ip_rcv
vmlinux 0xffffffff9f031cb9
Build id: 3.10.0-862.el7.x86_64
0x2b9
24
? udp4_lib_lookup_skb
vmlinux 0xffffffff9f0630b1
Build id: 3.10.0-862.el7.x86_64
0x61
25
? udp_gro_receive
vmlinux 0xffffffff9f064fa7
Build id: 3.10.0-862.el7.x86_64
0x57
26
__netif_receive_skb_core
vmlinux 0xffffffff9efeb9f9
Build id: 3.10.0-862.el7.x86_64
0x729
27
? tcp4_gro_receive
vmlinux 0xffffffff9f05d600
Build id: 3.10.0-862.el7.x86_64
0xe0
28
? __getnstimeofday64
vmlinux 0xffffffff9eaf8c5f
Build id: 3.10.0-862.el7.x86_64
0x3f
29
__netif_receive_skb
vmlinux 0xffffffff9efebd08
Build id: 3.10.0-862.el7.x86_64
0x18
30
netif_receive_skb_internal
vmlinux 0xffffffff9efebd90
Build id: 3.10.0-862.el7.x86_64
0x40
31
napi_gro_receive
vmlinux 0xffffffff9efec9b8
Build id: 3.10.0-862.el7.x86_64
0xd8
32
e1000_clean_rx_irq
e1000 0xffffffffc012744b
Build id: 3.10.0-862.el7.x86_64
0x2bb
33
e1000_clean
e1000 0xffffffffc01280eb
Build id: 3.10.0-862.el7.x86_64
0x27b
34
net_rx_action
vmlinux 0xffffffff9efec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
35
__do_softirq
vmlinux 0xffffffff9ea9a945
Build id: 3.10.0-862.el7.x86_64
0xf5
36
run_ksoftirqd
vmlinux 0xffffffff9ea9ab08
Build id: 3.10.0-862.el7.x86_64
0x38
37
smpboot_thread_fn
vmlinux 0xffffffff9eac35e4
Build id: 3.10.0-862.el7.x86_64
0x144
38
? lg_double_unlock
vmlinux 0xffffffff9eac34a0
Build id: 3.10.0-862.el7.x86_64
0x40
39
kthread
vmlinux 0xffffffff9eabae31
Build id: 3.10.0-862.el7.x86_64
0xd1
40
? insert_kthread_work
vmlinux 0xffffffff9eabad60
Build id: 3.10.0-862.el7.x86_64
0x40
41
ret_from_fork_nospec_begin
vmlinux 0xffffffff9f11f637
Build id: 3.10.0-862.el7.x86_64
0x21
42
? insert_kthread_work
vmlinux 0xffffffff9eabad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 24s! [ksoftirqd/4:29]
Modules linked in: iptable_filter snd_seq_midi snd_seq_midi_event sb_edac coretemp iosf_mbi snd_ens1371 crc32_pclmul snd_rawmidi ghash_clmulni_intel snd_ac97_codec aesni_intel ac97_bus lrw snd_seq gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm ppdev vmw_balloon snd_timer snd soundcore joydev pcspkr i2c_piix4 sg vmw_vmci shpchp parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common mptspi libata scsi_transport_spi crc32c_intel mptscsih e1000 mptbase serio_raw i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 29 Comm: ksoftirqd/4 Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
task: ffff8f8d748aaf70 ti: ffff8f8d748d8000 task.ti: ffff8f8d748d8000
RIP: 0010:[<ffffffffc0126867>]  [<ffffffffc0126867>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8f8d748db580  EFLAGS: 00000286
RAX: ffffa64b43203818 RBX: ffff8f9393349140 RCX: 0000000000000013
RDX: 0000000000000013 RSI: ffff8f9393349140 RDI: ffff8f9398cf4000
RBP: ffff8f8d748db600 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: 000000000000000f
R13: ffffa64b43203818 R14: 0000000000000100 R15: 0000000022100000
FS:  0000000000000000(0000) GS:ffff8f9397900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd54c1db000 CR3: 0000000476b96000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9efeb166>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff9f016a2a>] sch_direct_xmit+0x11a/0x250
[<ffffffff9efee051>] __dev_queue_xmit+0x4a1/0x660
[<ffffffff9efee220>] dev_queue_xmit+0x10/0x20
[<ffffffff9f037416>] ip_finish_output+0x546/0x7a0
[<ffffffff9f037973>] ip_output+0x73/0xe0
[<ffffffff9f035526>] ? __ip_local_out_sk+0xf6/0x100
[<ffffffff9f035567>] ip_local_out_sk+0x37/0x40
[<ffffffff9f0358d3>] ip_queue_xmit+0x143/0x3a0
[<ffffffff9f04f7b4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff9f05182f>] tcp_send_ack+0x11f/0x170
[<ffffffff9f04600b>] __tcp_ack_snd_check+0x6b/0xa0
[<ffffffff9f04c357>] tcp_rcv_established+0x417/0x770
[<ffffffff9efd005a>] ? sock_def_readable+0x3a/0x70
[<ffffffff9f056dda>] tcp_v4_do_rcv+0x10a/0x350
[<ffffffff9f058568>] tcp_v4_rcv+0x788/0x990
[<ffffffffc04eb036>] ? iptable_filter_hook+0x36/0x80 [iptable_filter]
[<ffffffff9f031699>] ip_local_deliver_finish+0xb9/0x200
[<ffffffff9f031989>] ip_local_deliver+0x59/0xd0
[<ffffffff9f0315e0>] ? ip_rcv_finish+0x370/0x370
[<ffffffff9f031300>] ip_rcv_finish+0x90/0x370
[<ffffffff9f031cb9>] ip_rcv+0x2b9/0x410
[<ffffffff9f0630b1>] ? udp4_lib_lookup_skb+0x61/0x70
[<ffffffff9f064fa7>] ? udp_gro_receive+0x57/0x170
[<ffffffff9efeb9f9>] __netif_receive_skb_core+0x729/0xa20
[<ffffffff9f05d600>] ? tcp4_gro_receive+0xe0/0x1a0
[<ffffffff9eaf8c5f>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffff9efebd08>] __netif_receive_skb+0x18/0x60
[<ffffffff9efebd90>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff9efec9b8>] napi_gro_receive+0xd8/0x100
[<ffffffffc012744b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc01280eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffff9efec3af>] net_rx_action+0x26f/0x390
[<ffffffff9ea9a945>] __do_softirq+0xf5/0x280
[<ffffffff9ea9ab08>] run_ksoftirqd+0x38/0x50
[<ffffffff9eac35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff9eac34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffff9eabae31>] kthread+0xd1/0xe0
[<ffffffff9eabad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff9f11f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9eabad60>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #172011
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc013b867
Build id: 3.10.0-862.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffb4feb166
Build id: 3.10.0-862.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffb5016a2a
Build id: 3.10.0-862.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffb4fee051
Build id: 3.10.0-862.el7.x86_64
0x4a1
5
dev_queue_xmit
vmlinux 0xffffffffb4fee220
Build id: 3.10.0-862.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffffb5037416
Build id: 3.10.0-862.el7.x86_64
0x546
7
ip_output
vmlinux 0xffffffffb5037973
Build id: 3.10.0-862.el7.x86_64
0x73
8
ip_local_out_sk
vmlinux 0xffffffffb5035567
Build id: 3.10.0-862.el7.x86_64
0x37
9
ip_queue_xmit
vmlinux 0xffffffffb50358d3
Build id: 3.10.0-862.el7.x86_64
0x143
10
tcp_transmit_skb
vmlinux 0xffffffffb504f7b4
Build id: 3.10.0-862.el7.x86_64
0x4e4
11
tcp_send_ack
vmlinux 0xffffffffb505182f
Build id: 3.10.0-862.el7.x86_64
0x11f
12
__tcp_ack_snd_check
vmlinux 0xffffffffb504600b
Build id: 3.10.0-862.el7.x86_64
0x6b
13
tcp_rcv_established
vmlinux 0xffffffffb504c1a4
Build id: 3.10.0-862.el7.x86_64
0x264
14
tcp_v4_do_rcv
vmlinux 0xffffffffb5056dda
Build id: 3.10.0-862.el7.x86_64
0x10a
15
tcp_v4_rcv
vmlinux 0xffffffffb5058568
Build id: 3.10.0-862.el7.x86_64
0x788
16
? neigh_update
vmlinux 0xffffffffb4ffc0a9
Build id: 3.10.0-862.el7.x86_64
0xc9
17
? skb_release_data
vmlinux 0xffffffffb4fd6102
Build id: 3.10.0-862.el7.x86_64
0x132
18
ip_local_deliver_finish
vmlinux 0xffffffffb5031699
Build id: 3.10.0-862.el7.x86_64
0xb9
19
ip_local_deliver
vmlinux 0xffffffffb5031989
Build id: 3.10.0-862.el7.x86_64
0x59
20
? tcp_v4_early_demux
vmlinux 0xffffffffb5057d92
Build id: 3.10.0-862.el7.x86_64
0x112
21
ip_rcv_finish
vmlinux 0xffffffffb5031300
Build id: 3.10.0-862.el7.x86_64
0x90
22
ip_rcv
vmlinux 0xffffffffb5031cb9
Build id: 3.10.0-862.el7.x86_64
0x2b9
23
? udp4_lib_lookup_skb
vmlinux 0xffffffffb50630b1
Build id: 3.10.0-862.el7.x86_64
0x61
24
? udp_gro_receive
vmlinux 0xffffffffb5064fa7
Build id: 3.10.0-862.el7.x86_64
0x57
25
__netif_receive_skb_core
vmlinux 0xffffffffb4feb9f9
Build id: 3.10.0-862.el7.x86_64
0x729
26
? tcp4_gro_receive
vmlinux 0xffffffffb505d600
Build id: 3.10.0-862.el7.x86_64
0xe0
27
? __getnstimeofday64
vmlinux 0xffffffffb4af8c5f
Build id: 3.10.0-862.el7.x86_64
0x3f
28
__netif_receive_skb
vmlinux 0xffffffffb4febd08
Build id: 3.10.0-862.el7.x86_64
0x18
29
netif_receive_skb_internal
vmlinux 0xffffffffb4febd90
Build id: 3.10.0-862.el7.x86_64
0x40
30
napi_gro_receive
vmlinux 0xffffffffb4fec9b8
Build id: 3.10.0-862.el7.x86_64
0xd8
31
e1000_clean_rx_irq
e1000 0xffffffffc013c44b
Build id: 3.10.0-862.el7.x86_64
0x2bb
32
e1000_clean
e1000 0xffffffffc013d0eb
Build id: 3.10.0-862.el7.x86_64
0x27b
33
? update_curr
vmlinux 0xffffffffb4ad816c
Build id: 3.10.0-862.el7.x86_64
0x14c
34
? account_entity_dequeue
vmlinux 0xffffffffb4ad4ade
Build id: 3.10.0-862.el7.x86_64
0xae
35
? __schedule
vmlinux 0xffffffffb5112b14
Build id: 3.10.0-862.el7.x86_64
0x614
36
net_rx_action
vmlinux 0xffffffffb4fec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
37
__do_softirq
vmlinux 0xffffffffb4a9a945
Build id: 3.10.0-862.el7.x86_64
0xf5
38
run_ksoftirqd
vmlinux 0xffffffffb4a9ab08
Build id: 3.10.0-862.el7.x86_64
0x38
39
smpboot_thread_fn
vmlinux 0xffffffffb4ac35e4
Build id: 3.10.0-862.el7.x86_64
0x144
40
? lg_double_unlock
vmlinux 0xffffffffb4ac34a0
Build id: 3.10.0-862.el7.x86_64
0x40
41
kthread
vmlinux 0xffffffffb4abae31
Build id: 3.10.0-862.el7.x86_64
0xd1
42
? insert_kthread_work
vmlinux 0xffffffffb4abad60
Build id: 3.10.0-862.el7.x86_64
0x40
43
ret_from_fork_nospec_begin
vmlinux 0xffffffffb511f61d
Build id: 3.10.0-862.el7.x86_64
0x7
44
? insert_kthread_work
vmlinux 0xffffffffb4abad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ksoftirqd/2:19]
Modules linked in: binfmt_misc tun devlink ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device iosf_mbi snd_pcm snd_timer crc32_pclmul snd soundcore ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev nfit vmw_balloon sg pcspkr joydev libnvdimm parport_pc parport i2c_piix4 shpchp vmw_vmci xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm ata_piix libata crc32c_intel drm serio_raw mptspi scsi_transport_spi mptscsih e1000 mptbase i2c_core [last unloaded: ip_tables]
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9197fa850fd0 ti: ffff9197fa85c000 task.ti: ffff9197fa85c000
RIP: 0010:[<ffffffffc013b867>]  [<ffffffffc013b867>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9197fa85f580  EFLAGS: 00000286
RAX: ffffa32ec1983818 RBX: 0000000000000001 RCX: 000000000000003b
RDX: 000000000000003b RSI: ffff9199ad700e80 RDI: ffff9199bf926000
RBP: ffff9197fa85f600 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: 0000000000000202
R13: 0000000233b62d0c R14: 0000000100000100 R15: ffffffffb4d73e00
FS:  0000000000000000(0000) GS:ffff9199b1680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1f1b6dd100 CR3: 00000002b2bc4000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb4feb166>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb5016a2a>] sch_direct_xmit+0x11a/0x250
[<ffffffffb4fee051>] __dev_queue_xmit+0x4a1/0x660
[<ffffffffb4fee220>] dev_queue_xmit+0x10/0x20
[<ffffffffb5037416>] ip_finish_output+0x546/0x7a0
[<ffffffffb5037973>] ip_output+0x73/0xe0
[<ffffffffb5035567>] ip_local_out_sk+0x37/0x40
[<ffffffffb50358d3>] ip_queue_xmit+0x143/0x3a0
[<ffffffffb504f7b4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffb505182f>] tcp_send_ack+0x11f/0x170
[<ffffffffb504600b>] __tcp_ack_snd_check+0x6b/0xa0
[<ffffffffb504c1a4>] tcp_rcv_established+0x264/0x770
[<ffffffffb5056dda>] tcp_v4_do_rcv+0x10a/0x350
[<ffffffffb5058568>] tcp_v4_rcv+0x788/0x990
[<ffffffffb4ffc0a9>] ? neigh_update+0xc9/0x650
[<ffffffffb4fd6102>] ? skb_release_data+0x132/0x140
[<ffffffffb5031699>] ip_local_deliver_finish+0xb9/0x200
[<ffffffffb5031989>] ip_local_deliver+0x59/0xd0
[<ffffffffb5057d92>] ? tcp_v4_early_demux+0x112/0x160
[<ffffffffb5031300>] ip_rcv_finish+0x90/0x370
[<ffffffffb5031cb9>] ip_rcv+0x2b9/0x410
[<ffffffffb50630b1>] ? udp4_lib_lookup_skb+0x61/0x70
[<ffffffffb5064fa7>] ? udp_gro_receive+0x57/0x170
[<ffffffffb4feb9f9>] __netif_receive_skb_core+0x729/0xa20
[<ffffffffb505d600>] ? tcp4_gro_receive+0xe0/0x1a0
[<ffffffffb4af8c5f>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffffb4febd08>] __netif_receive_skb+0x18/0x60
[<ffffffffb4febd90>] netif_receive_skb_internal+0x40/0xc0
[<ffffffffb4fec9b8>] napi_gro_receive+0xd8/0x100
[<ffffffffc013c44b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc013d0eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffffb4ad816c>] ? update_curr+0x14c/0x1e0
[<ffffffffb4ad4ade>] ? account_entity_dequeue+0xae/0xd0
[<ffffffffb5112b14>] ? __schedule+0x614/0xa20
[<ffffffffb4fec3af>] net_rx_action+0x26f/0x390
[<ffffffffb4a9a945>] __do_softirq+0xf5/0x280
[<ffffffffb4a9ab08>] run_ksoftirqd+0x38/0x50
[<ffffffffb4ac35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb4ac34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffffb4abae31>] kthread+0xd1/0xe0
[<ffffffffb4abad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb511f61d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb4abad60>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #126079
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00de877
Build id: 3.10.0-1160.6.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff8c6543e6
Build id: 3.10.0-1160.6.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff8c68248a
Build id: 3.10.0-1160.6.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff8c657361
Build id: 3.10.0-1160.6.1.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffff8c657530
Build id: 3.10.0-1160.6.1.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffff8c6a54cb
Build id: 3.10.0-1160.6.1.el7.x86_64
0x67b
7
ip_output
vmlinux 0xffffffff8c6a5a4b
Build id: 3.10.0-1160.6.1.el7.x86_64
0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffff8c6a4e50
Build id: 3.10.0-1160.6.1.el7.x86_64
0xa60
9
ip_local_out_sk
vmlinux 0xffffffff8c6a3477
Build id: 3.10.0-1160.6.1.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffff8c6a3814
Build id: 3.10.0-1160.6.1.el7.x86_64
0x144
11
tcp_transmit_skb
vmlinux 0xffffffff8c6bdf24
Build id: 3.10.0-1160.6.1.el7.x86_64
0x4e4
12
tcp_send_ack
vmlinux 0xffffffff8c6c013f
Build id: 3.10.0-1160.6.1.el7.x86_64
0x11f
13
__tcp_ack_snd_check
vmlinux 0xffffffff8c6b4ceb
Build id: 3.10.0-1160.6.1.el7.x86_64
0x6b
14
tcp_rcv_established
vmlinux 0xffffffff8c6ba814
Build id: 3.10.0-1160.6.1.el7.x86_64
0x264
15
tcp_v4_do_rcv
vmlinux 0xffffffff8c6c583a
Build id: 3.10.0-1160.6.1.el7.x86_64
0x10a
16
tcp_v4_rcv
vmlinux 0xffffffff8c6c708c
Build id: 3.10.0-1160.6.1.el7.x86_64
0x7dc
17
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc066c2ce
Build id: 3.10.0-1160.6.1.el7.x86_64
0x4e
18
ip_local_deliver_finish
vmlinux 0xffffffff8c69f2fd
Build id: 3.10.0-1160.6.1.el7.x86_64
0xbd
19
ip_local_deliver
vmlinux 0xffffffff8c69f5f0
Build id: 3.10.0-1160.6.1.el7.x86_64
0x60
20
? ip_rcv_finish
vmlinux 0xffffffff8c69f240
Build id: 3.10.0-1160.6.1.el7.x86_64
0x370
21
ip_rcv_finish
vmlinux 0xffffffff8c69ef60
Build id: 3.10.0-1160.6.1.el7.x86_64
0x90
22
ip_rcv
vmlinux 0xffffffff8c69f930
Build id: 3.10.0-1160.6.1.el7.x86_64
0x2c0
23
? packet_rcv
vmlinux 0xffffffff8c75fc02
Build id: 3.10.0-1160.6.1.el7.x86_64
0x272
24
? inet_del_offload
vmlinux 0xffffffff8c69eed0
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
25
__netif_receive_skb_core
vmlinux 0xffffffff8c654c79
Build id: 3.10.0-1160.6.1.el7.x86_64
0x729
26
? tcp4_gro_receive
vmlinux 0xffffffff8c6cc200
Build id: 3.10.0-1160.6.1.el7.x86_64
0xf0
27
? __getnstimeofday64
vmlinux 0xffffffff8c10792f
Build id: 3.10.0-1160.6.1.el7.x86_64
0x3f
28
__netif_receive_skb
vmlinux 0xffffffff8c654f78
Build id: 3.10.0-1160.6.1.el7.x86_64
0x18
29
netif_receive_skb_internal
vmlinux 0xffffffff8c655000
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
30
napi_gro_receive
vmlinux 0xffffffff8c655c88
Build id: 3.10.0-1160.6.1.el7.x86_64
0xd8
31
e1000_clean_rx_irq
e1000 0xffffffffc00df45b
Build id: 3.10.0-1160.6.1.el7.x86_64
0x2bb
32
e1000_clean
e1000 0xffffffffc00e0100
Build id: 3.10.0-1160.6.1.el7.x86_64
0x280
33
? tcp_v4_md5_lookup
vmlinux 0xffffffff8c6c2bd3
Build id: 3.10.0-1160.6.1.el7.x86_64
0x13
34
? update_curr
vmlinux 0xffffffff8c0e441c
Build id: 3.10.0-1160.6.1.el7.x86_64
0x14c
35
net_rx_action
vmlinux 0xffffffff8c65561f
Build id: 3.10.0-1160.6.1.el7.x86_64
0x26f
36
__do_softirq
vmlinux 0xffffffff8c0a4b95
Build id: 3.10.0-1160.6.1.el7.x86_64
0xf5
37
run_ksoftirqd
vmlinux 0xffffffff8c0a4d58
Build id: 3.10.0-1160.6.1.el7.x86_64
0x38
38
smpboot_thread_fn
vmlinux 0xffffffff8c0ce674
Build id: 3.10.0-1160.6.1.el7.x86_64
0x144
39
? lg_double_unlock
vmlinux 0xffffffff8c0ce530
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
40
kthread
vmlinux 0xffffffff8c0c5c21
Build id: 3.10.0-1160.6.1.el7.x86_64
0xd1
41
? insert_kthread_work
vmlinux 0xffffffff8c0c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
42
ret_from_fork_nospec_begin
vmlinux 0xffffffff8c794ddd
Build id: 3.10.0-1160.6.1.el7.x86_64
0x7
43
? insert_kthread_work
vmlinux 0xffffffff8c0c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [ksoftirqd/3:24]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event crc32_pclmul ghash_clmulni_intel aesni_intel lrw snd_ens1371 gf128mul glue_helper snd_rawmidi ablk_helper snd_ac97_codec cryptd ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore ppdev sg pcspkr joydev vmw_balloon parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata nfit libnvdimm mptspi scsi_transport_spi mptscsih crct10dif_pclmul crct10dif_common crc32c_intel e1000 mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 24 Comm: ksoftirqd/3 Kdump: loaded Tainted: G               ------------ T 3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8990a4fc3180 ti: ffff8990a4fd0000 task.ti: ffff8990a4fd0000
RIP: 0010:[<ffffffffc00de877>]  [<ffffffffc00de877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8990a4fd3550  EFLAGS: 00000286
RAX: ffffa61f41983818 RBX: 0000000000000001 RCX: 00000000000000b0
RDX: 00000000000000b0 RSI: ffff8992e8bb1500 RDI: ffff8990ad38a000
RBP: ffff8990a4fd35d0 R08: 0000000000000100 R09: 0000000022100000
R10: c09c280000450008 R11: 0000000000000000 R12: 0000000000000202
R13: 00000001a84accda R14: 00000001bf18f2f8 R15: ffffffff8c3b1690
FS:  0000000000000000(0000) GS:ffff8992eaec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f280c0d9000 CR3: 000000022cc10000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8c6543e6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff8c68248a>] sch_direct_xmit+0x11a/0x250
[<ffffffff8c657361>] __dev_queue_xmit+0x491/0x650
[<ffffffff8c657530>] dev_queue_xmit+0x10/0x20
[<ffffffff8c6a54cb>] ip_finish_output+0x67b/0x8d0
[<ffffffff8c6a5a4b>] ip_output+0x7b/0xf0
[<ffffffff8c6a4e50>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff8c6a3477>] ip_local_out_sk+0x37/0x40
[<ffffffff8c6a3814>] ip_queue_xmit+0x144/0x3c0
[<ffffffff8c6bdf24>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff8c6c013f>] tcp_send_ack+0x11f/0x170
[<ffffffff8c6b4ceb>] __tcp_ack_snd_check+0x6b/0x90
[<ffffffff8c6ba814>] tcp_rcv_established+0x264/0x770
[<ffffffff8c6c583a>] tcp_v4_do_rcv+0x10a/0x350
[<ffffffff8c6c708c>] tcp_v4_rcv+0x7dc/0x9e0
[<ffffffffc066c2ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffff8c69f2fd>] ip_local_deliver_finish+0xbd/0x200
[<ffffffff8c69f5f0>] ip_local_deliver+0x60/0xe0
[<ffffffff8c69f240>] ? ip_rcv_finish+0x370/0x370
[<ffffffff8c69ef60>] ip_rcv_finish+0x90/0x370
[<ffffffff8c69f930>] ip_rcv+0x2c0/0x420
[<ffffffff8c75fc02>] ? packet_rcv+0x272/0x430
[<ffffffff8c69eed0>] ? inet_del_offload+0x40/0x40
[<ffffffff8c654c79>] __netif_receive_skb_core+0x729/0xa10
[<ffffffff8c6cc200>] ? tcp4_gro_receive+0xf0/0x1a0
[<ffffffff8c10792f>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffff8c654f78>] __netif_receive_skb+0x18/0x60
[<ffffffff8c655000>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff8c655c88>] napi_gro_receive+0xd8/0x100
[<ffffffffc00df45b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc00e0100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff8c6c2bd3>] ? tcp_v4_md5_lookup+0x13/0x20
[<ffffffff8c0e441c>] ? update_curr+0x14c/0x1e0
[<ffffffff8c65561f>] net_rx_action+0x26f/0x390
[<ffffffff8c0a4b95>] __do_softirq+0xf5/0x280
[<ffffffff8c0a4d58>] run_ksoftirqd+0x38/0x50
[<ffffffff8c0ce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff8c0ce530>] ? lg_double_unlock+0x40/0x40
[<ffffffff8c0c5c21>] kthread+0xd1/0xe0
[<ffffffff8c0c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff8c794ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff8c0c5b50>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #52621
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0111877
Build id: 3.10.0-1127.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffb3c53386
Build id: 3.10.0-1127.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffb3c8131a
Build id: 3.10.0-1127.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffb3c56301
Build id: 3.10.0-1127.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffffb3c564d0
Build id: 3.10.0-1127.el7.x86_64
0x10
6
neigh_resolve_output
vmlinux 0xffffffffb3c6312d
Build id: 3.10.0-1127.el7.x86_64
0x11d
7
ip_finish_output
vmlinux 0xffffffffb3ca3e9c
Build id: 3.10.0-1127.el7.x86_64
0x27c
8
ip_output
vmlinux 0xffffffffb3ca481b
Build id: 3.10.0-1127.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffffb3ca3c20
Build id: 3.10.0-1127.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffffb3ca2247
Build id: 3.10.0-1127.el7.x86_64
0x37
11
ip_queue_xmit
vmlinux 0xffffffffb3ca25e4
Build id: 3.10.0-1127.el7.x86_64
0x144
12
tcp_transmit_skb
vmlinux 0xffffffffb3cbccc4
Build id: 3.10.0-1127.el7.x86_64
0x4e4
13
tcp_send_ack
vmlinux 0xffffffffb3cbeecf
Build id: 3.10.0-1127.el7.x86_64
0x11f
14
__tcp_ack_snd_check
vmlinux 0xffffffffb3cb3abb
Build id: 3.10.0-1127.el7.x86_64
0x6b
15
tcp_rcv_state_process
vmlinux 0xffffffffb3cba796
Build id: 3.10.0-1127.el7.x86_64
0xa36
16
? local_bh_enable
vmlinux 0xffffffffb36a4b17
Build id: 3.10.0-1127.el7.x86_64
0x17
17
? nfnetlink_has_listeners
nfnetlink 0xffffffffc06bb165
Build id: 3.10.0-1127.el7.x86_64
0x15
18
? ctnetlink_conntrack_event
nf_conntrack_netlink 0xffffffffc06cac94
Build id: 3.10.0-1127.el7.x86_64
0x454
19
tcp_v4_do_rcv
vmlinux 0xffffffffb3cc472b
Build id: 3.10.0-1127.el7.x86_64
0x17b
20
tcp_v4_rcv
vmlinux 0xffffffffb3cc5dcc
Build id: 3.10.0-1127.el7.x86_64
0x7dc
21
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc03ed2ce
Build id: 3.10.0-1127.el7.x86_64
0x4e
22
ip_local_deliver_finish
vmlinux 0xffffffffb3c9e0fd
Build id: 3.10.0-1127.el7.x86_64
0xbd
23
ip_local_deliver
vmlinux 0xffffffffb3c9e3f0
Build id: 3.10.0-1127.el7.x86_64
0x60
24
? ip_rcv_finish
vmlinux 0xffffffffb3c9e040
Build id: 3.10.0-1127.el7.x86_64
0x370
25
ip_rcv_finish
vmlinux 0xffffffffb3c9dd60
Build id: 3.10.0-1127.el7.x86_64
0x90
26
ip_rcv
vmlinux 0xffffffffb3c9e730
Build id: 3.10.0-1127.el7.x86_64
0x2c0
27
? inet_del_offload
vmlinux 0xffffffffb3c9dcd0
Build id: 3.10.0-1127.el7.x86_64
0x40
28
__netif_receive_skb_core
vmlinux 0xffffffffb3c53c19
Build id: 3.10.0-1127.el7.x86_64
0x729
29
? tcp4_gro_receive
vmlinux 0xffffffffb3ccaf00
Build id: 3.10.0-1127.el7.x86_64
0xb0
30
? __getnstimeofday64
vmlinux 0xffffffffb370825f
Build id: 3.10.0-1127.el7.x86_64
0x3f
31
__netif_receive_skb
vmlinux 0xffffffffb3c53f18
Build id: 3.10.0-1127.el7.x86_64
0x18
32
netif_receive_skb_internal
vmlinux 0xffffffffb3c53fa0
Build id: 3.10.0-1127.el7.x86_64
0x40
33
napi_gro_receive
vmlinux 0xffffffffb3c54c28
Build id: 3.10.0-1127.el7.x86_64
0xd8
34
e1000_clean_rx_irq
e1000 0xffffffffc011245b
Build id: 3.10.0-1127.el7.x86_64
0x2bb
35
e1000_clean
e1000 0xffffffffc0113100
Build id: 3.10.0-1127.el7.x86_64
0x280
36
? update_curr
vmlinux 0xffffffffb36e4eac
Build id: 3.10.0-1127.el7.x86_64
0x14c
37
net_rx_action
vmlinux 0xffffffffb3c545bf
Build id: 3.10.0-1127.el7.x86_64
0x26f
38
__do_softirq
vmlinux 0xffffffffb36a5695
Build id: 3.10.0-1127.el7.x86_64
0xf5
39
run_ksoftirqd
vmlinux 0xffffffffb36a5858
Build id: 3.10.0-1127.el7.x86_64
0x38
40
smpboot_thread_fn
vmlinux 0xffffffffb36cf0e4
Build id: 3.10.0-1127.el7.x86_64
0x144
41
? lg_double_unlock
vmlinux 0xffffffffb36cefa0
Build id: 3.10.0-1127.el7.x86_64
0x40
42
kthread
vmlinux 0xffffffffb36c6691
Build id: 3.10.0-1127.el7.x86_64
0xd1
43
? insert_kthread_work
vmlinux 0xffffffffb36c65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
44
ret_from_fork_nospec_begin
vmlinux 0xffffffffb3d92d37
Build id: 3.10.0-1127.el7.x86_64
0x21
45
? insert_kthread_work
vmlinux 0xffffffffb36c65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ksoftirqd/0:6]
Modules linked in: tcp_lp fuse xt_statistic veth nf_tables vxlan ip6_udp_tunnel udp_tunnel xt_nat xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter devlink ebtables ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq joydev pcspkr snd_seq_device snd_pcm snd_timer snd i2c_piix4 sg soundcore vmw_vmci parport_pc parport nf_conntrack_ipv4 nf_defrag_ipv4
ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix mptspi e1000 scsi_transport_spi libata serio_raw mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9142bcfe5230 ti: ffff9142bc8a4000 task.ti: ffff9142bc8a4000
RIP: 0010:[<ffffffffc0111877>]  [<ffffffffc0111877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9142bc8a74e0  EFLAGS: 00000286
RAX: ffffbbfc00cc3818 RBX: 0000000000000001 RCX: 00000000000000d9
RDX: 00000000000000d9 RSI: ffff914370a1eb80 RDI: ffff9143745e4000
RBP: ffff9142bc8a7560 R08: 0000000000000100 R09: 0000000022100000
R10: c83ea8c0b3630640 R11: 0000000000000000 R12: 0000000000000202
R13: 00000001dfc36cc2 R14: 00000001bc8a74a8 R15: ffffffffb39b0770
FS:  0000000000000000(0000) GS:ffff914375600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f59bcbc5000 CR3: 000000022efca000 CR4: 00000000001607f0
Call Trace:
[<ffffffffb3c53386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb3c8131a>] sch_direct_xmit+0x11a/0x250
[<ffffffffb3c56301>] __dev_queue_xmit+0x491/0x650
[<ffffffffb3c564d0>] dev_queue_xmit+0x10/0x20
[<ffffffffb3c6312d>] neigh_resolve_output+0x11d/0x220
[<ffffffffb3ca3e9c>] ip_finish_output+0x27c/0x8d0
[<ffffffffb3ca481b>] ip_output+0x7b/0xf0
[<ffffffffb3ca3c20>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffb3ca2247>] ip_local_out_sk+0x37/0x40
[<ffffffffb3ca25e4>] ip_queue_xmit+0x144/0x3c0
[<ffffffffb3cbccc4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffb3cbeecf>] tcp_send_ack+0x11f/0x170
[<ffffffffb3cb3abb>] __tcp_ack_snd_check+0x6b/0x90
[<ffffffffb3cba796>] tcp_rcv_state_process+0xa36/0xf50
[<ffffffffb36a4b17>] ? local_bh_enable+0x17/0x20
[<ffffffffc06bb165>] ? nfnetlink_has_listeners+0x15/0x20 [nfnetlink]
[<ffffffffc06cac94>] ? ctnetlink_conntrack_event+0x454/0x6c0 [nf_conntrack_netlink]
[<ffffffffb3cc472b>] tcp_v4_do_rcv+0x17b/0x350
[<ffffffffb3cc5dcc>] tcp_v4_rcv+0x7dc/0x9e0
[<ffffffffc03ed2ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffffb3c9e0fd>] ip_local_deliver_finish+0xbd/0x200
[<ffffffffb3c9e3f0>] ip_local_deliver+0x60/0xe0
[<ffffffffb3c9e040>] ? ip_rcv_finish+0x370/0x370
[<ffffffffb3c9dd60>] ip_rcv_finish+0x90/0x370
[<ffffffffb3c9e730>] ip_rcv+0x2c0/0x420
[<ffffffffb3c9dcd0>] ? inet_del_offload+0x40/0x40
[<ffffffffb3c53c19>] __netif_receive_skb_core+0x729/0xa10
[<ffffffffb3ccaf00>] ? tcp4_gro_receive+0xb0/0x1a0
[<ffffffffb370825f>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffffb3c53f18>] __netif_receive_skb+0x18/0x60
[<ffffffffb3c53fa0>] netif_receive_skb_internal+0x40/0xc0
[<ffffffffb3c54c28>] napi_gro_receive+0xd8/0x100
[<ffffffffc011245b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc0113100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffb36e4eac>] ? update_curr+0x14c/0x1e0
[<ffffffffb3c545bf>] net_rx_action+0x26f/0x390
[<ffffffffb36a5695>] __do_softirq+0xf5/0x280
[<ffffffffb36a5858>] run_ksoftirqd+0x38/0x50
[<ffffffffb36cf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb36cefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffffb36c6691>] kthread+0xd1/0xe0
[<ffffffffb36c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb3d92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb36c65c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00