Info

Permalink
Function
e1000_xmit_frame
First occurrence
2021-02-01
Last occurrence
2022-04-19
State
NEW
Components
kernel
Quality
-60

Statistics

Operating system Count
CentOS 7 37
Architecture Count
x86_64 37
Related packages Count
kernel 37
 0:3.10.0-693.el7 6
 0:3.10.0-862.el7 5
 0:3.10.0-1062.18.1.el7 4
 0:3.10.0-1160.el7 3
 0:3.10.0-693.21.1.el7 3
 0:3.10.0-1160.119.1.el7 3
 0:3.10.0-514.el7 2
 0:3.10.0-514.16.1.el7 2
 0:3.10.0-1160.114.2.el7 2
 0:3.10.0-1127.el7 1
 0:3.10.0-1160.36.2.el7 1
 0:3.10.0-1160.59.1.el7 1
 0:3.10.0-693.2.2.el7 1
 0:3.10.0-1160.80.1.el7 1
 0:3.10.0-1160.102.1.el7 1
 0:3.10.0-1062.4.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #503974
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc027d877
Build id: 3.10.0-1160.59.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffad658f96
Build id: 3.10.0-1160.59.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffad6870aa
Build id: 3.10.0-1160.59.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffad65bf11
Build id: 3.10.0-1160.59.1.el7.x86_64
0x491
5
dev_queue_xmit_sk
vmlinux 0xffffffffad65c103
Build id: 3.10.0-1160.59.1.el7.x86_64
0x13
6
arp_xmit
vmlinux 0xffffffffad6da15a
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3a
7
? arp_create
vmlinux 0xffffffffad6d9d93
Build id: 3.10.0-1160.59.1.el7.x86_64
0x1c3
8
arp_send_dst.part.16
vmlinux 0xffffffffad6da215
Build id: 3.10.0-1160.59.1.el7.x86_64
0x45
9
arp_process
vmlinux 0xffffffffad6db1f7
Build id: 3.10.0-1160.59.1.el7.x86_64
0x877
10
? ip_rcv_finish
vmlinux 0xffffffffad6a49e0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x370
11
arp_rcv
vmlinux 0xffffffffad6db3ad
Build id: 3.10.0-1160.59.1.el7.x86_64
0x12d
12
? inet_del_offload
vmlinux 0xffffffffad6a4670
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
13
__netif_receive_skb_core
vmlinux 0xffffffffad659829
Build id: 3.10.0-1160.59.1.el7.x86_64
0x729
14
? tcp4_gro_receive
vmlinux 0xffffffffad6d1a00
Build id: 3.10.0-1160.59.1.el7.x86_64
0xe0
15
? __getnstimeofday64
vmlinux 0xffffffffad107f0f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3f
16
__netif_receive_skb
vmlinux 0xffffffffad659b28
Build id: 3.10.0-1160.59.1.el7.x86_64
0x18
17
netif_receive_skb_internal
vmlinux 0xffffffffad659bb0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
18
napi_gro_receive
vmlinux 0xffffffffad65a838
Build id: 3.10.0-1160.59.1.el7.x86_64
0xd8
19
e1000_clean_rx_irq
e1000 0xffffffffc027e45b
Build id: 3.10.0-1160.59.1.el7.x86_64
0x2bb
20
e1000_clean
e1000 0xffffffffc027f100
Build id: 3.10.0-1160.59.1.el7.x86_64
0x280
21
? nf_ct_delete
nf_conntrack 0xffffffffc076202d
Build id: 3.10.0-1160.59.1.el7.x86_64
0x18d
22
? tcp_init_xmit_timers
vmlinux 0xffffffffad6c71f0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x30
23
net_rx_action
vmlinux 0xffffffffad65a1cf
Build id: 3.10.0-1160.59.1.el7.x86_64
0x26f
24
__do_softirq
vmlinux 0xffffffffad0a4bf5
Build id: 3.10.0-1160.59.1.el7.x86_64
0xf5
25
run_ksoftirqd
vmlinux 0xffffffffad0a4db8
Build id: 3.10.0-1160.59.1.el7.x86_64
0x38
26
smpboot_thread_fn
vmlinux 0xffffffffad0ce8b4
Build id: 3.10.0-1160.59.1.el7.x86_64
0x144
27
? lg_double_unlock
vmlinux 0xffffffffad0ce770
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
28
kthread
vmlinux 0xffffffffad0c5e61
Build id: 3.10.0-1160.59.1.el7.x86_64
0xd1
29
? insert_kthread_work
vmlinux 0xffffffffad0c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffffad799df7
Build id: 3.10.0-1160.59.1.el7.x86_64
0x21
31
? insert_kthread_work
vmlinux 0xffffffffad0c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [ksoftirqd/2:19]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter binfmt_misc overlay(T) vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd sg vmw_balloon pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common ata_piix drm crc32c_intel
mptspi nfit serio_raw scsi_transport_spi libnvdimm libata drm_panel_orientation_quirks mptscsih e1000 mptbase dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffffa0cf454fd280 ti: ffffa0cf45540000 task.ti: ffffa0cf45540000
RIP: 0010:[<ffffffffc027d877>]  [<ffffffffc027d877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0000:ffffa0cf455437c0  EFLAGS: 00000286
RAX: ffffb10000683818 RBX: 0000000000000730 RCX: 000000000000003b
RDX: 000000000000003b RSI: ffffa0cef4de7140 RDI: ffffa0cf6ee60000
RBP: ffffa0cf45543840 R08: 0000000000000100 R09: 0000000002000000
R10: 0000000080000000 R11: 0000000000000000 R12: ffffffffad6832e5
R13: ffffffffad2267ed R14: ffffa0cf455437c8 R15: ffffa0cf2ad03000
FS:  0000000000000000(0000) GS:ffffa0cf6f880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe1561683b4 CR3: 000000007a5ea000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffad658f96>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffad6870aa>] sch_direct_xmit+0x11a/0x250
[<ffffffffad65bf11>] __dev_queue_xmit+0x491/0x650
[<ffffffffad65c103>] dev_queue_xmit_sk+0x13/0x20
[<ffffffffad6da15a>] arp_xmit+0x3a/0xb0
[<ffffffffad6d9d93>] ? arp_create+0x1c3/0x230
[<ffffffffad6da215>] arp_send_dst.part.16+0x45/0x50
[<ffffffffad6db1f7>] arp_process+0x877/0x8e0
[<ffffffffad6a49e0>] ? ip_rcv_finish+0x370/0x370
[<ffffffffad6db3ad>] arp_rcv+0x12d/0x1b0
[<ffffffffad6a4670>] ? inet_del_offload+0x40/0x40
[<ffffffffad659829>] __netif_receive_skb_core+0x729/0xa10
[<ffffffffad6d1a00>] ? tcp4_gro_receive+0xe0/0x1a0
[<ffffffffad107f0f>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffffad659b28>] __netif_receive_skb+0x18/0x60
[<ffffffffad659bb0>] netif_receive_skb_internal+0x40/0xc0
[<ffffffffad65a838>] napi_gro_receive+0xd8/0x100
[<ffffffffc027e45b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc027f100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffc076202d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffffad6c71f0>] ? tcp_init_xmit_timers+0x30/0x30
[<ffffffffad65a1cf>] net_rx_action+0x26f/0x390
[<ffffffffad0a4bf5>] __do_softirq+0xf5/0x280
[<ffffffffad0a4db8>] run_ksoftirqd+0x38/0x50
[<ffffffffad0ce8b4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffad0ce770>] ? lg_double_unlock+0x40/0x40
[<ffffffffad0c5e61>] kthread+0xd1/0xe0
[<ffffffffad0c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffad799df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffad0c5d90>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #246854
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0460877
Build id: 3.10.0-1062.18.1.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffff818376e7
Build id: 3.10.0-1062.18.1.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffff8184f346
Build id: 3.10.0-1062.18.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffff8187d0ca
Build id: 3.10.0-1062.18.1.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffff818522c1
Build id: 3.10.0-1062.18.1.el7.x86_64
0x491
6
dev_queue_xmit_sk
vmlinux 0xffffffff818524b3
Build id: 3.10.0-1062.18.1.el7.x86_64
0x13
7
arp_xmit
vmlinux 0xffffffff818cfd0a
Build id: 3.10.0-1062.18.1.el7.x86_64
0x3a
8
? arp_create
vmlinux 0xffffffff818cf943
Build id: 3.10.0-1062.18.1.el7.x86_64
0x1c3
9
arp_send_dst.part.16
vmlinux 0xffffffff818cfdc5
Build id: 3.10.0-1062.18.1.el7.x86_64
0x45
10
arp_process
vmlinux 0xffffffff818d0da7
Build id: 3.10.0-1062.18.1.el7.x86_64
0x877
11
? ip_rcv_finish
vmlinux 0xffffffff8189a7d0
Build id: 3.10.0-1062.18.1.el7.x86_64
0x370
12
arp_rcv
vmlinux 0xffffffff818d0f5d
Build id: 3.10.0-1062.18.1.el7.x86_64
0x12d
13
? packet_rcv
vmlinux 0xffffffff8195aaec
Build id: 3.10.0-1062.18.1.el7.x86_64
0x4c
14
__netif_receive_skb_core
vmlinux 0xffffffff8184fbd9
Build id: 3.10.0-1062.18.1.el7.x86_64
0x729
15
? __getnstimeofday64
vmlinux 0xffffffff8130790f
Build id: 3.10.0-1062.18.1.el7.x86_64
0x3f
16
__netif_receive_skb
vmlinux 0xffffffff8184fed8
Build id: 3.10.0-1062.18.1.el7.x86_64
0x18
17
netif_receive_skb_internal
vmlinux 0xffffffff8184ff60
Build id: 3.10.0-1062.18.1.el7.x86_64
0x40
18
napi_gro_receive
vmlinux 0xffffffff81850be8
Build id: 3.10.0-1062.18.1.el7.x86_64
0xd8
19
e1000_clean_rx_irq
e1000 0xffffffffc046145b
Build id: 3.10.0-1062.18.1.el7.x86_64
0x2bb
20
e1000_clean
e1000 0xffffffffc0462100
Build id: 3.10.0-1062.18.1.el7.x86_64
0x280
21
? nf_ct_delete
nf_conntrack 0xffffffffc08cd02d
Build id: 3.10.0-1062.18.1.el7.x86_64
0x18d
22
? tcp_init_xmit_timers
vmlinux 0xffffffff818bce30
Build id: 3.10.0-1062.18.1.el7.x86_64
0x30
23
net_rx_action
vmlinux 0xffffffff8185057f
Build id: 3.10.0-1062.18.1.el7.x86_64
0x26f
24
__do_softirq
vmlinux 0xffffffff812a5435
Build id: 3.10.0-1062.18.1.el7.x86_64
0xf5
25
run_ksoftirqd
vmlinux 0xffffffff812a55f8
Build id: 3.10.0-1062.18.1.el7.x86_64
0x38
26
smpboot_thread_fn
vmlinux 0xffffffff812ced54
Build id: 3.10.0-1062.18.1.el7.x86_64
0x144
27
? lg_double_unlock
vmlinux 0xffffffff812cec10
Build id: 3.10.0-1062.18.1.el7.x86_64
0x40
28
kthread
vmlinux 0xffffffff812c6321
Build id: 3.10.0-1062.18.1.el7.x86_64
0xd1
29
? insert_kthread_work
vmlinux 0xffffffff812c6250
Build id: 3.10.0-1062.18.1.el7.x86_64
0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffff8198dd24
Build id: 3.10.0-1062.18.1.el7.x86_64
0xe
31
? insert_kthread_work
vmlinux 0xffffffff812c6250
Build id: 3.10.0-1062.18.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 36s! [ksoftirqd/2:19]
Modules linked in: tcp_lp nls_utf8 isofs xt_nat xt_CHECKSUM iptable_mangle veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 nf_conntrack_netlink nfnetlink tun xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) fuse vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec crc32_pclmul ac97_bus snd_seq snd_seq_device ghash_clmulni_intel ppdev snd_pcm vmw_balloon aesni_intel snd_timer lrw snd gf128mul soundcore glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper ata_piix syscopyarea mptspi sysfillrect nfit scsi_transport_spi sysimgblt e1000 fb_sys_fops libata mptscsih ttm mptbase drm libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff921db4e6d230 ti: ffff921db4eb0000 task.ti: ffff921db4eb0000
RIP: 0010:[<ffffffffc0460877>]  [<ffffffffc0460877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff921db4eb37c0  EFLAGS: 00000286
RAX: ffffa6aac31c3818 RBX: 0000000000000086 RCX: 000000000000006f
RDX: 000000000000006f RSI: ffff9223aa6e7880 RDI: ffff921db50dc000
RBP: ffff921db4eb3840 R08: 0000000000000100 R09: 0000000002000000
R10: 0000000080000000 R11: 0000000000000000 R12: ffffffff812d7a0f
R13: ffffffff81423c7d R14: ffff921db4eb37c8 R15: ffff921ca0ca2000
FS:  00007f966d3fd700(0000) GS:ffff9223b3080000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f966b766000 CR3: 000000034e410000 CR4: 00000000003407e0
Call Trace:
[<ffffffff818376e7>] ? kfree_skbmem+0x37/0x90
[<ffffffff8184f346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff8187d0ca>] sch_direct_xmit+0x11a/0x250
[<ffffffff818522c1>] __dev_queue_xmit+0x491/0x650
[<ffffffff818524b3>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff818cfd0a>] arp_xmit+0x3a/0xb0
[<ffffffff818cf943>] ? arp_create+0x1c3/0x230
[<ffffffff818cfdc5>] arp_send_dst.part.16+0x45/0x50
[<ffffffff818d0da7>] arp_process+0x877/0x8e0
[<ffffffff8189a7d0>] ? ip_rcv_finish+0x370/0x370
[<ffffffff818d0f5d>] arp_rcv+0x12d/0x1b0
[<ffffffff8195aaec>] ? packet_rcv+0x4c/0x430
[<ffffffff8184fbd9>] __netif_receive_skb_core+0x729/0xa10
[<ffffffff8130790f>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffff8184fed8>] __netif_receive_skb+0x18/0x60
[<ffffffff8184ff60>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff81850be8>] napi_gro_receive+0xd8/0x100
[<ffffffffc046145b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc0462100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffc08cd02d>] ? nf_ct_delete+0x18d/0x270 [nf_conntrack]
[<ffffffff818bce30>] ? tcp_init_xmit_timers+0x30/0x30
[<ffffffff8185057f>] net_rx_action+0x26f/0x390
[<ffffffff812a5435>] __do_softirq+0xf5/0x280
[<ffffffff812a55f8>] run_ksoftirqd+0x38/0x50
[<ffffffff812ced54>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff812cec10>] ? lg_double_unlock+0x40/0x40
[<ffffffff812c6321>] kthread+0xd1/0xe0
[<ffffffff812c6250>] ? insert_kthread_work+0x40/0x40
[<ffffffff8198dd24>] ret_from_fork_nospec_begin+0xe/0x21
[<ffffffff812c6250>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #1418479
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc03cf8e7
Build id: 3.10.0-1160.119.1.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffff93066c7f
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3f
3
dev_hard_start_xmit
vmlinux 0xffffffff9307edc6
Build id: 3.10.0-1160.119.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffff930add72
Build id: 3.10.0-1160.119.1.el7.x86_64
0x122
5
__dev_queue_xmit
vmlinux 0xffffffff93081dd1
Build id: 3.10.0-1160.119.1.el7.x86_64
0x491
6
dev_queue_xmit_sk
vmlinux 0xffffffff93081fc3
Build id: 3.10.0-1160.119.1.el7.x86_64
0x13
7
arp_xmit
vmlinux 0xffffffff9310258a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3a
8
? arp_create
vmlinux 0xffffffff931021b3
Build id: 3.10.0-1160.119.1.el7.x86_64
0x1c3
9
arp_send_dst.part.16
vmlinux 0xffffffff93102645
Build id: 3.10.0-1160.119.1.el7.x86_64
0x45
10
arp_process
vmlinux 0xffffffff93103657
Build id: 3.10.0-1160.119.1.el7.x86_64
0x877
11
? ip_rcv_finish
vmlinux 0xffffffff930cc290
Build id: 3.10.0-1160.119.1.el7.x86_64
0x380
12
arp_rcv
vmlinux 0xffffffff9310380d
Build id: 3.10.0-1160.119.1.el7.x86_64
0x12d
13
? packet_rcv
vmlinux 0xffffffff9318b1fc
Build id: 3.10.0-1160.119.1.el7.x86_64
0x4c
14
__netif_receive_skb_core
vmlinux 0xffffffff9307f669
Build id: 3.10.0-1160.119.1.el7.x86_64
0x739
15
? __getnstimeofday64
vmlinux 0xffffffff92b0eabf
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3f
16
__netif_receive_skb
vmlinux 0xffffffff9307f968
Build id: 3.10.0-1160.119.1.el7.x86_64
0x18
17
netif_receive_skb_internal
vmlinux 0xffffffff9307fa00
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
18
napi_gro_receive
vmlinux 0xffffffff930806b8
Build id: 3.10.0-1160.119.1.el7.x86_64
0xd8
19
e1000_clean_rx_irq
e1000 0xffffffffc03d04cb
Build id: 3.10.0-1160.119.1.el7.x86_64
0x2bb
20
e1000_clean
e1000 0xffffffffc03d1180
Build id: 3.10.0-1160.119.1.el7.x86_64
0x280
21
? try_to_wake_up
vmlinux 0xffffffff92ae0e92
Build id: 3.10.0-1160.119.1.el7.x86_64
0x192
22
net_rx_action
vmlinux 0xffffffff93080037
Build id: 3.10.0-1160.119.1.el7.x86_64
0x277
23
__do_softirq
vmlinux 0xffffffff92aa9595
Build id: 3.10.0-1160.119.1.el7.x86_64
0xf5
24
run_ksoftirqd
vmlinux 0xffffffff92aa9768
Build id: 3.10.0-1160.119.1.el7.x86_64
0x38
25
smpboot_thread_fn
vmlinux 0xffffffff92ad43fc
Build id: 3.10.0-1160.119.1.el7.x86_64
0x14c
26
? lg_double_unlock
vmlinux 0xffffffff92ad42b0
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
27
kthread
vmlinux 0xffffffff92acb621
Build id: 3.10.0-1160.119.1.el7.x86_64
0xd1
28
? insert_kthread_work
vmlinux 0xffffffff92acb550
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffff931c51dd
Build id: 3.10.0-1160.119.1.el7.x86_64
0x7
30
? insert_kthread_work
vmlinux 0xffffffff92acb550
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [ksoftirqd/1:14]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr sg joydev vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel mptspi ata_piix serio_raw nfit e1000 scsi_transport_spi mptscsih libnvdimm mptbase libata
drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 14 Comm: ksoftirqd/1 Tainted: G        W    L ------------   3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9ed571928000 ti: ffff9ed571924000 task.ti: ffff9ed571924000
RIP: 0010:[<ffffffffc03cf8e7>]  [<ffffffffc03cf8e7>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9ed5719277c0  EFLAGS: 00010286
RAX: ffffb4c3c31c3818 RBX: 0000000000000001 RCX: 00000000000000fb
RDX: 00000000000000fb RSI: ffff9ed960c68140 RDI: ffff9ed964f60000
RBP: ffff9ed571927840 R08: 0000000000000100 R09: 0000000002000000
R10: 0000000080000000 R11: 0000000000000000 R12: 0000000000000202
R13: 000000039ef27a02 R14: 0000000111fa2300 R15: ffffffff92dc9150
FS:  0000000000000000(0000) GS:ffff9ed966240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f464627be98 CR3: 00000005a25fc000 CR4: 0000000000360fe0
Call Trace:
[<ffffffff93066c7f>] ? kfree_skbmem+0x3f/0xa0
[<ffffffff9307edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff930add72>] sch_direct_xmit+0x122/0x250
[<ffffffff93081dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffff93081fc3>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff9310258a>] arp_xmit+0x3a/0xb0
[<ffffffff931021b3>] ? arp_create+0x1c3/0x240
[<ffffffff93102645>] arp_send_dst.part.16+0x45/0x50
[<ffffffff93103657>] arp_process+0x877/0x8e0
[<ffffffff930cc290>] ? ip_rcv_finish+0x380/0x380
[<ffffffff9310380d>] arp_rcv+0x12d/0x1b0
[<ffffffff9318b1fc>] ? packet_rcv+0x4c/0x440
[<ffffffff9307f669>] __netif_receive_skb_core+0x739/0xa20
[<ffffffff92b0eabf>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffff9307f968>] __netif_receive_skb+0x18/0x70
[<ffffffff9307fa00>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff930806b8>] napi_gro_receive+0xd8/0x100
[<ffffffffc03d04cb>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc03d1180>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff92ae0e92>] ? try_to_wake_up+0x192/0x3a0
[<ffffffff93080037>] net_rx_action+0x277/0x3a0
[<ffffffff92aa9595>] __do_softirq+0xf5/0x290
[<ffffffff92aa9768>] run_ksoftirqd+0x38/0x60
[<ffffffff92ad43fc>] smpboot_thread_fn+0x14c/0x1b0
[<ffffffff92ad42b0>] ? lg_double_unlock+0x40/0x40
[<ffffffff92acb621>] kthread+0xd1/0xe0
[<ffffffff92acb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff931c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff92acb550>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc 
Complete report #1302901
Frame # Function Binary Source or offset Line
1
dev_hard_start_xmit
vmlinux 0xffffffffa167edc6
Build id: 3.10.0-1160.119.1.el7.x86_64
0x246
2
sch_direct_xmit
vmlinux 0xffffffffa16add72
Build id: 3.10.0-1160.119.1.el7.x86_64
0x122
3
__dev_queue_xmit
vmlinux 0xffffffffa1681dd1
Build id: 3.10.0-1160.119.1.el7.x86_64
0x491
4
dev_queue_xmit_sk
vmlinux 0xffffffffa1681fc3
Build id: 3.10.0-1160.119.1.el7.x86_64
0x13
5
arp_xmit
vmlinux 0xffffffffa170258a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3a
6
? arp_create
vmlinux 0xffffffffa17021b3
Build id: 3.10.0-1160.119.1.el7.x86_64
0x1c3
7
arp_send_dst.part.16
vmlinux 0xffffffffa1702645
Build id: 3.10.0-1160.119.1.el7.x86_64
0x45
8
arp_process
vmlinux 0xffffffffa1703657
Build id: 3.10.0-1160.119.1.el7.x86_64
0x877
9
? ip_rcv_finish
vmlinux 0xffffffffa16cc290
Build id: 3.10.0-1160.119.1.el7.x86_64
0x380
10
arp_rcv
vmlinux 0xffffffffa170380d
Build id: 3.10.0-1160.119.1.el7.x86_64
0x12d
11
? packet_rcv
vmlinux 0xffffffffa178b1fc
Build id: 3.10.0-1160.119.1.el7.x86_64
0x4c
12
__netif_receive_skb_core
vmlinux 0xffffffffa167f669
Build id: 3.10.0-1160.119.1.el7.x86_64
0x739
13
? __getnstimeofday64
vmlinux 0xffffffffa110eabf
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3f
14
__netif_receive_skb
vmlinux 0xffffffffa167f968
Build id: 3.10.0-1160.119.1.el7.x86_64
0x18
15
netif_receive_skb_internal
vmlinux 0xffffffffa167fa00
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
16
napi_gro_receive
vmlinux 0xffffffffa16806b8
Build id: 3.10.0-1160.119.1.el7.x86_64
0xd8
17
e1000_clean_rx_irq
e1000 0xffffffffc01984cb
Build id: 3.10.0-1160.119.1.el7.x86_64
0x2bb
18
e1000_clean
e1000 0xffffffffc0199180
Build id: 3.10.0-1160.119.1.el7.x86_64
0x280
19
net_rx_action
vmlinux 0xffffffffa1680037
Build id: 3.10.0-1160.119.1.el7.x86_64
0x277
20
__do_softirq
vmlinux 0xffffffffa10a9595
Build id: 3.10.0-1160.119.1.el7.x86_64
0xf5
21
run_ksoftirqd
vmlinux 0xffffffffa10a9768
Build id: 3.10.0-1160.119.1.el7.x86_64
0x38
22
smpboot_thread_fn
vmlinux 0xffffffffa10d43fc
Build id: 3.10.0-1160.119.1.el7.x86_64
0x14c
23
? lg_double_unlock
vmlinux 0xffffffffa10d42b0
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
24
kthread
vmlinux 0xffffffffa10cb621
Build id: 3.10.0-1160.119.1.el7.x86_64
0xd1
25
? insert_kthread_work
vmlinux 0xffffffffa10cb550
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
26
ret_from_fork_nospec_begin
vmlinux 0xffffffffa17c51f7
Build id: 3.10.0-1160.119.1.el7.x86_64
0x21
27
? insert_kthread_work
vmlinux 0xffffffffa10cb550
Build id: 3.10.0-1160.119.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [node:30426]
INFO: rcu_sched detected stalls on CPUs/tasks: { 0} 
Modules linked in:
xt_CHECKSUM
iptable_mangle
ipt_MASQUERADE
[<ffffffffa167edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa16add72>] sch_direct_xmit+0x122/0x250
[<ffffffffa1681dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffffa1681fc3>] dev_queue_xmit_sk+0x13/0x20
[<ffffffffa170258a>] arp_xmit+0x3a/0xb0
[<ffffffffa17021b3>] ? arp_create+0x1c3/0x240
[<ffffffffa1702645>] arp_send_dst.part.16+0x45/0x50
[<ffffffffa1703657>] arp_process+0x877/0x8e0
[<ffffffffa16cc290>] ? ip_rcv_finish+0x380/0x380
[<ffffffffa170380d>] arp_rcv+0x12d/0x1b0
[<ffffffffa178b1fc>] ? packet_rcv+0x4c/0x440
[<ffffffffa167f669>] __netif_receive_skb_core+0x739/0xa20
[<ffffffffa110eabf>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffffa167f968>] __netif_receive_skb+0x18/0x70
[<ffffffffa167fa00>] netif_receive_skb_internal+0x40/0xc0
[<ffffffffa16806b8>] napi_gro_receive+0xd8/0x100
[<ffffffffc01984cb>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc0199180>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffa1680037>] net_rx_action+0x277/0x3a0
[<ffffffffa10a9595>] __do_softirq+0xf5/0x290
[<ffffffffa10a9768>] run_ksoftirqd+0x38/0x60
[<ffffffffa10d43fc>] smpboot_thread_fn+0x14c/0x1b0
[<ffffffffa10d42b0>] ? lg_double_unlock+0x40/0x40
[<ffffffffa10cb621>] kthread+0xd1/0xe0
[<ffffffffa10cb550>] ? insert_kthread_work+0x40/0x40
[<ffffffffa17c51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa10cb550>] ? insert_kthread_work+0x40/0x40
Complete report #1203124
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00e9ee7
Build id: 3.10.0-693.el7.x86_64
0x947
2
dev_hard_start_xmit
vmlinux 0xffffffff81586771
Build id: 3.10.0-693.el7.x86_64
0x171
3
sch_direct_xmit
vmlinux 0xffffffff815af50a
Build id: 3.10.0-693.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff81589566
Build id: 3.10.0-693.el7.x86_64
0x226
5
dev_queue_xmit_sk
vmlinux 0xffffffff815898c3
Build id: 3.10.0-693.el7.x86_64
0x13
6
arp_xmit
vmlinux 0xffffffff815fe063
Build id: 3.10.0-693.el7.x86_64
0x33
7
? arp_create
vmlinux 0xffffffff815fdcc0
Build id: 3.10.0-693.el7.x86_64
0x1c0
8
arp_send_dst.part.15
vmlinux 0xffffffff815fe115
Build id: 3.10.0-693.el7.x86_64
0x45
9
arp_process
vmlinux 0xffffffff815fef5e
Build id: 3.10.0-693.el7.x86_64
0x70e
10
? ip_local_deliver
vmlinux 0xffffffff815ca339
Build id: 3.10.0-693.el7.x86_64
0x59
11
? ip_rcv_finish
vmlinux 0xffffffff815c9fa0
Build id: 3.10.0-693.el7.x86_64
0x350
12
arp_rcv
vmlinux 0xffffffff815ff115
Build id: 3.10.0-693.el7.x86_64
0x125
13
? inet_del_offload
vmlinux 0xffffffff815c9c50
Build id: 3.10.0-693.el7.x86_64
0x40
14
__netif_receive_skb_core
vmlinux 0xffffffff81586f22
Build id: 3.10.0-693.el7.x86_64
0x572
15
? inet_gro_receive
vmlinux 0xffffffff81607087
Build id: 3.10.0-693.el7.x86_64
0x247
16
__netif_receive_skb
vmlinux 0xffffffff81587188
Build id: 3.10.0-693.el7.x86_64
0x18
17
netif_receive_skb_internal
vmlinux 0xffffffff81587210
Build id: 3.10.0-693.el7.x86_64
0x40
18
napi_gro_receive
vmlinux 0xffffffff81588318
Build id: 3.10.0-693.el7.x86_64
0xd8
19
e1000_clean_rx_irq
e1000 0xffffffffc00eb47c
Build id: 3.10.0-693.el7.x86_64
0x2ac
20
e1000_clean
e1000 0xffffffffc00ebed8
Build id: 3.10.0-693.el7.x86_64
0x278
21
net_rx_action
vmlinux 0xffffffff8158799d
Build id: 3.10.0-693.el7.x86_64
0x16d
22
__do_softirq
vmlinux 0xffffffff81090b3f
Build id: 3.10.0-693.el7.x86_64
0xef
23
run_ksoftirqd
vmlinux 0xffffffff81090d08
Build id: 3.10.0-693.el7.x86_64
0x38
24
smpboot_thread_fn
vmlinux 0xffffffff810b909f
Build id: 3.10.0-693.el7.x86_64
0x12f
25
? lg_double_unlock
vmlinux 0xffffffff810b8f70
Build id: 3.10.0-693.el7.x86_64
0x40
26
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
27
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
28
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
29
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [ksoftirqd/0:3]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 aesni_intel lrw gf128mul glue_helper ablk_helper snd_rawmidi cryptd snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm vmw_balloon joydev snd_timer pcspkr snd sg vmw_vmci nfit i2c_piix4 soundcore shpchp libnvdimm parport_pc parport nfsd auth_rpcgss
nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi e1000 scsi_transport_spi crct10dif_pclmul mptscsih crct10dif_common crc32c_intel mptbase serio_raw i2c_core
CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G             L ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff880174b49fa0 ti: ffff880174b6c000 task.ti: ffff880174b6c000
RIP: 0010:[<ffffffffc00e9ee7>]  [<ffffffffc00e9ee7>] e1000_xmit_frame+0x947/0x1040 [e1000]
RSP: 0000:ffff880174b6f800  EFLAGS: 00000286
RAX: ffffc900034c3818 RBX: 0000000000000202 RCX: 00000000000000f2
RDX: 00000000000000f2 RSI: ffff8800361244c0 RDI: ffff88081d530000
RBP: ffff880174b6f880 R08: 0000000000000100 R09: 0000000002000000
R10: 0000000000000000 R11: 0000000000000000 R12: 00000001361244c0
R13: ffffffff813492f0 R14: ffff880174b6f7c8 R15: 0000000000000202
FS:  0000000000000000(0000) GS:ffff88081d600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005623ec7ad808 CR3: 00000000019f2000 CR4: 00000000001407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff880174b6f8b0 ffff88081d66c098 000000f11d530000 0000000000000000
0000000017feb700 ffff880000000000 ffff880800000000 0000000100001000
ffff8806b87e2800 ffff88081d530000 ffff88081d66c000 ffff88081d530000
Call Trace:
[<ffffffff81586771>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff815af50a>] sch_direct_xmit+0x11a/0x240
[<ffffffff81589566>] __dev_queue_xmit+0x226/0x550
[<ffffffff815898c3>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff815fe063>] arp_xmit+0x33/0xa0
[<ffffffff815fdcc0>] ? arp_create+0x1c0/0x230
[<ffffffff815fe115>] arp_send_dst.part.15+0x45/0x50
[<ffffffff815fef5e>] arp_process+0x70e/0x780
[<ffffffff815ca339>] ? ip_local_deliver+0x59/0xd0
[<ffffffff815c9fa0>] ? ip_rcv_finish+0x350/0x350
[<ffffffff815ff115>] arp_rcv+0x125/0x1b0
[<ffffffff815c9c50>] ? inet_del_offload+0x40/0x40
[<ffffffff81586f22>] __netif_receive_skb_core+0x572/0x7c0
[<ffffffff81607087>] ? inet_gro_receive+0x247/0x290
[<ffffffff81587188>] __netif_receive_skb+0x18/0x60
[<ffffffff81587210>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff81588318>] napi_gro_receive+0xd8/0x130
[<ffffffffc00eb47c>] e1000_clean_rx_irq+0x2ac/0x4f0 [e1000]
[<ffffffffc00ebed8>] e1000_clean+0x278/0x8d0 [e1000]
[<ffffffff8158799d>] net_rx_action+0x16d/0x380
[<ffffffff81090b3f>] __do_softirq+0xef/0x280
[<ffffffff81090d08>] run_ksoftirqd+0x38/0x50
[<ffffffff810b909f>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f70>] ? lg_double_unlock+0x40/0x40
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #1151077
Frame # Function Binary Source or offset Line
1
__getnstimeofday64
vmlinux 0xffffffffb5505e86
Build id: 3.10.0-1062.4.1.el7.x86_64
0x96
2
getnstimeofday64
vmlinux 0xffffffffb5505ece
Build id: 3.10.0-1062.4.1.el7.x86_64
0xe
3
ktime_get_real
vmlinux 0xffffffffb5505f85
Build id: 3.10.0-1062.4.1.el7.x86_64
0x25
4
dev_hard_start_xmit
vmlinux 0xffffffffb5a4d2ed
Build id: 3.10.0-1062.4.1.el7.x86_64
0x1ed
5
sch_direct_xmit
vmlinux 0xffffffffb5a7b0ba
Build id: 3.10.0-1062.4.1.el7.x86_64
0x11a
6
__dev_queue_xmit
vmlinux 0xffffffffb5a502b1
Build id: 3.10.0-1062.4.1.el7.x86_64
0x491
7
dev_queue_xmit_sk
vmlinux 0xffffffffb5a504a3
Build id: 3.10.0-1062.4.1.el7.x86_64
0x13
8
arp_xmit
vmlinux 0xffffffffb5acdcda
Build id: 3.10.0-1062.4.1.el7.x86_64
0x3a
9
? arp_create
vmlinux 0xffffffffb5acd913
Build id: 3.10.0-1062.4.1.el7.x86_64
0x1c3
10
arp_send_dst.part.16
vmlinux 0xffffffffb5acdd95
Build id: 3.10.0-1062.4.1.el7.x86_64
0x45
11
arp_process
vmlinux 0xffffffffb5aced77
Build id: 3.10.0-1062.4.1.el7.x86_64
0x877
12
? ttwu_do_wakeup
vmlinux 0xffffffffb54d6619
Build id: 3.10.0-1062.4.1.el7.x86_64
0x19
13
arp_rcv
vmlinux 0xffffffffb5acef2d
Build id: 3.10.0-1062.4.1.el7.x86_64
0x12d
14
? packet_rcv
vmlinux 0xffffffffb5b58a3c
Build id: 3.10.0-1062.4.1.el7.x86_64
0x4c
15
__netif_receive_skb_core
vmlinux 0xffffffffb5a4dbd9
Build id: 3.10.0-1062.4.1.el7.x86_64
0x729
16
? __getnstimeofday64
vmlinux 0xffffffffb5505e00
Build id: 3.10.0-1062.4.1.el7.x86_64
0x10
17
? __getnstimeofday64
vmlinux 0xffffffffb5505e2f
Build id: 3.10.0-1062.4.1.el7.x86_64
0x3f
18
__netif_receive_skb
vmlinux 0xffffffffb5a4ded8
Build id: 3.10.0-1062.4.1.el7.x86_64
0x18
19
netif_receive_skb_internal
vmlinux 0xffffffffb5a4df60
Build id: 3.10.0-1062.4.1.el7.x86_64
0x40
20
napi_gro_receive
vmlinux 0xffffffffb5a4ebe8
Build id: 3.10.0-1062.4.1.el7.x86_64
0xd8
21
e1000_clean_rx_irq
e1000 0xffffffffc01e845b
Build id: 3.10.0-1062.4.1.el7.x86_64
0x2bb
22
e1000_clean
e1000 0xffffffffc01e9100
Build id: 3.10.0-1062.4.1.el7.x86_64
0x280
23
? update_curr
vmlinux 0xffffffffb54e382c
Build id: 3.10.0-1062.4.1.el7.x86_64
0x14c
24
net_rx_action
vmlinux 0xffffffffb5a4e57f
Build id: 3.10.0-1062.4.1.el7.x86_64
0x26f
25
__do_softirq
vmlinux 0xffffffffb54a42c5
Build id: 3.10.0-1062.4.1.el7.x86_64
0xf5
26
run_ksoftirqd
vmlinux 0xffffffffb54a4488
Build id: 3.10.0-1062.4.1.el7.x86_64
0x38
27
smpboot_thread_fn
vmlinux 0xffffffffb54cdbe4
Build id: 3.10.0-1062.4.1.el7.x86_64
0x144
28
? lg_double_unlock
vmlinux 0xffffffffb54cdaa0
Build id: 3.10.0-1062.4.1.el7.x86_64
0x40
29
kthread
vmlinux 0xffffffffb54c51b1
Build id: 3.10.0-1062.4.1.el7.x86_64
0xd1
30
? insert_kthread_work
vmlinux 0xffffffffb54c50e0
Build id: 3.10.0-1062.4.1.el7.x86_64
0x40
31
ret_from_fork_nospec_begin
vmlinux 0xffffffffb5b8bd1d
Build id: 3.10.0-1062.4.1.el7.x86_64
0x7
32
? insert_kthread_work
vmlinux 0xffffffffb54c50e0
Build id: 3.10.0-1062.4.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 4097s! [ksoftirqd/1:14]
Modules linked in: tcp_lp binfmt_misc fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc vboxvideo(OE) ttm crc32_pclmul ghash_clmulni_intel ppdev drm_kms_helper syscopyarea sysfillrect aesni_intel sysimgblt fb_sys_fops lrw vboxguest(OE) gf128mul glue_helper drm ablk_helper joydev cryptd pcspkr sg i2c_piix4 parport_pc parport video drm_panel_orientation_quirks
ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ahci libahci libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw e1000 dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 14 Comm: ksoftirqd/1 Kdump: loaded Tainted: G           OE  ------------   3.10.0-1062.4.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffffa0a47c1a8000 ti: ffffa0a47c1b0000 task.ti: ffffa0a47c1b0000
RIP: 0010:[<ffffffffb5505e86>]  [<ffffffffb5505e86>] __getnstimeofday64+0x96/0xd0
RSP: 0018:ffffa0a47c1b37f0  EFLAGS: 00010202
RAX: 0911b6089a218519 RBX: ffffa0a479343d28 RCX: 0000000067bf153c
RDX: 00000000248f4daa RSI: 000003fca0cd93be RDI: 0016027a0e197e92
RBP: ffffa0a47c1b3808 R08: 000000000080004f R09: 0000000000000000
R10: 0000000000000000 R11: ffffa0a45c5d641e R12: 0000000000000000
R13: ffffa0a4793439c0 R14: ffffa0a47b99ad00 R15: 00000000d2362676
FS:  0000000000000000(0000) GS:ffffa0a47fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000909423b81f0 CR3: 00000000797ee000 CR4: 00000000000206e0
Call Trace:
[<ffffffffb5505ece>] getnstimeofday64+0xe/0x30
[<ffffffffb5505f85>] ktime_get_real+0x25/0x70
[<ffffffffb5a4d2ed>] dev_hard_start_xmit+0x1ed/0x3b0
[<ffffffffb5a7b0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffffb5a502b1>] __dev_queue_xmit+0x491/0x650
[<ffffffffb5a504a3>] dev_queue_xmit_sk+0x13/0x20
[<ffffffffb5acdcda>] arp_xmit+0x3a/0xb0
[<ffffffffb5acd913>] ? arp_create+0x1c3/0x230
[<ffffffffb5acdd95>] arp_send_dst.part.16+0x45/0x50
[<ffffffffb5aced77>] arp_process+0x877/0x8e0
[<ffffffffb54d6619>] ? ttwu_do_wakeup+0x19/0xe0
[<ffffffffb5acef2d>] arp_rcv+0x12d/0x1b0
[<ffffffffb5b58a3c>] ? packet_rcv+0x4c/0x430
[<ffffffffb5a4dbd9>] __netif_receive_skb_core+0x729/0xa10
[<ffffffffb5505e00>] ? __getnstimeofday64+0x10/0xd0
[<ffffffffb5505e2f>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffffb5a4ded8>] __netif_receive_skb+0x18/0x60
[<ffffffffb5a4df60>] netif_receive_skb_internal+0x40/0xc0
[<ffffffffb5a4ebe8>] napi_gro_receive+0xd8/0x100
[<ffffffffc01e845b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc01e9100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffb54e382c>] ? update_curr+0x14c/0x1e0
[<ffffffffb5a4e57f>] net_rx_action+0x26f/0x390
[<ffffffffb54a42c5>] __do_softirq+0xf5/0x280
[<ffffffffb54a4488>] run_ksoftirqd+0x38/0x50
[<ffffffffb54cdbe4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb54cdaa0>] ? lg_double_unlock+0x40/0x40
[<ffffffffb54c51b1>] kthread+0xd1/0xe0
[<ffffffffb54c50e0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb5b8bd1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb54c50e0>] ? insert_kthread_work+0x40/0x40
Code: 48 21 c6 44 89 c0 48 0f 49 d6 48 0f af d0 48 8d 04 3a 48 d3 f8 48 8b 0b 48 3d ff c9 9a 3b 76 35 31 d2 0f 1f 00 48 2d 00 ca 9a 3b <83> c2 01 48 3d ff c9 9a 3b 77 ef 48 01 ca 48 89 43 08 48 89 13 
Complete report #978108
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc01ab877
Build id: 3.10.0-1160.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffb263c6f7
Build id: 3.10.0-1160.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffb2654386
Build id: 3.10.0-1160.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffffb268242a
Build id: 3.10.0-1160.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffb2657301
Build id: 3.10.0-1160.el7.x86_64
0x491
6
dev_queue_xmit_sk
vmlinux 0xffffffffb26574f3
Build id: 3.10.0-1160.el7.x86_64
0x13
7
arp_xmit
vmlinux 0xffffffffb26d48ba
Build id: 3.10.0-1160.el7.x86_64
0x3a
8
? arp_create
vmlinux 0xffffffffb26d44f3
Build id: 3.10.0-1160.el7.x86_64
0x1c3
9
arp_send_dst.part.16
vmlinux 0xffffffffb26d4975
Build id: 3.10.0-1160.el7.x86_64
0x45
10
arp_process
vmlinux 0xffffffffb26d5957
Build id: 3.10.0-1160.el7.x86_64
0x877
11
arp_rcv
vmlinux 0xffffffffb26d5b0d
Build id: 3.10.0-1160.el7.x86_64
0x12d
12
? packet_rcv
vmlinux 0xffffffffb275f91c
Build id: 3.10.0-1160.el7.x86_64
0x4c
13
__netif_receive_skb_core
vmlinux 0xffffffffb2654c19
Build id: 3.10.0-1160.el7.x86_64
0x729
14
? __getnstimeofday64
vmlinux 0xffffffffb21078df
Build id: 3.10.0-1160.el7.x86_64
0x3f
15
__netif_receive_skb
vmlinux 0xffffffffb2654f18
Build id: 3.10.0-1160.el7.x86_64
0x18
16
netif_receive_skb_internal
vmlinux 0xffffffffb2654fa0
Build id: 3.10.0-1160.el7.x86_64
0x40
17
napi_gro_receive
vmlinux 0xffffffffb2655c28
Build id: 3.10.0-1160.el7.x86_64
0xd8
18
e1000_clean_rx_irq
e1000 0xffffffffc01ac45b
Build id: 3.10.0-1160.el7.x86_64
0x2bb
19
e1000_clean
e1000 0xffffffffc01ad100
Build id: 3.10.0-1160.el7.x86_64
0x280
20
? __queue_work
vmlinux 0xffffffffb20bbb9c
Build id: 3.10.0-1160.el7.x86_64
0x13c
21
net_rx_action
vmlinux 0xffffffffb26555bf
Build id: 3.10.0-1160.el7.x86_64
0x26f
22
__do_softirq
vmlinux 0xffffffffb20a4b95
Build id: 3.10.0-1160.el7.x86_64
0xf5
23
run_ksoftirqd
vmlinux 0xffffffffb20a4d58
Build id: 3.10.0-1160.el7.x86_64
0x38
24
smpboot_thread_fn
vmlinux 0xffffffffb20ce674
Build id: 3.10.0-1160.el7.x86_64
0x144
25
? lg_double_unlock
vmlinux 0xffffffffb20ce530
Build id: 3.10.0-1160.el7.x86_64
0x40
26
kthread
vmlinux 0xffffffffb20c5c21
Build id: 3.10.0-1160.el7.x86_64
0xd1
27
? insert_kthread_work
vmlinux 0xffffffffb20c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
28
ret_from_fork_nospec_begin
vmlinux 0xffffffffb2793df7
Build id: 3.10.0-1160.el7.x86_64
0x21
29
? insert_kthread_work
vmlinux 0xffffffffb20c5b50
Build id: 3.10.0-1160.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [ksoftirqd/0:6]
Modules linked in: tcp_lp bnep nfnetlink_queue nfnetlink_log bluetooth rfkill xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd sg joydev vmw_balloon pcspkr parport_pc parport i2c_piix4 vmw_vmci ip_tables
xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix drm nfit mptspi libata serio_raw scsi_transport_spi libnvdimm mptscsih e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8d8370bad280 ti: ffff8d83713e0000 task.ti: ffff8d83713e0000
RIP: 0010:[<ffffffffc01ab877>]  [<ffffffffc01ab877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8d83713e37c0  EFLAGS: 00000286
RAX: ffffb2bc81c43818 RBX: 0000000000000010 RCX: 000000000000005c
RDX: 000000000000005c RSI: ffff8d8373fa3a80 RDI: ffff8d837b3bc000
RBP: ffff8d83713e3840 R08: 0000000000000100 R09: 0000000002000000
R10: 0000000080000000 R11: 0000000000000000 R12: ffffffffffffff10
R13: ffff8d837b3bc000 R14: ffff8d8373fa3a80 R15: 000000000000005b
FS:  0000000000000000(0000) GS:ffff8d8480600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8c8736d000 CR3: 000000027a10c000 CR4: 00000000003607f0
Call Trace:
[<ffffffffb263c6f7>] ? kfree_skbmem+0x37/0x90
[<ffffffffb2654386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb268242a>] sch_direct_xmit+0x11a/0x250
[<ffffffffb2657301>] __dev_queue_xmit+0x491/0x650
[<ffffffffb26574f3>] dev_queue_xmit_sk+0x13/0x20
[<ffffffffb26d48ba>] arp_xmit+0x3a/0xb0
[<ffffffffb26d44f3>] ? arp_create+0x1c3/0x230
[<ffffffffb26d4975>] arp_send_dst.part.16+0x45/0x50
[<ffffffffb26d5957>] arp_process+0x877/0x8e0
[<ffffffffb26d5b0d>] arp_rcv+0x12d/0x1b0
[<ffffffffb275f91c>] ? packet_rcv+0x4c/0x430
[<ffffffffb2654c19>] __netif_receive_skb_core+0x729/0xa10
[<ffffffffb21078df>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffffb2654f18>] __netif_receive_skb+0x18/0x60
[<ffffffffb2654fa0>] netif_receive_skb_internal+0x40/0xc0
[<ffffffffb2655c28>] napi_gro_receive+0xd8/0x100
[<ffffffffc01ac45b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc01ad100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffb20bbb9c>] ? __queue_work+0x13c/0x3f0
[<ffffffffb26555bf>] net_rx_action+0x26f/0x390
[<ffffffffb20a4b95>] __do_softirq+0xf5/0x280
[<ffffffffb20a4d58>] run_ksoftirqd+0x38/0x50
[<ffffffffb20ce674>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffb20ce530>] ? lg_double_unlock+0x40/0x40
[<ffffffffb20c5c21>] kthread+0xd1/0xe0
[<ffffffffb20c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffb2793df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb20c5b50>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #903321
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc014b8e7
Build id: 3.10.0-1160.102.1.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffff8f466c7f
Build id: 3.10.0-1160.102.1.el7.x86_64
0x3f
3
dev_hard_start_xmit
vmlinux 0xffffffff8f47edc6
Build id: 3.10.0-1160.102.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffff8f4add72
Build id: 3.10.0-1160.102.1.el7.x86_64
0x122
5
__qdisc_run
vmlinux 0xffffffff8f4adf2e
Build id: 3.10.0-1160.102.1.el7.x86_64
0x8e
6
__dev_queue_xmit
vmlinux 0xffffffff8f481de9
Build id: 3.10.0-1160.102.1.el7.x86_64
0x4a9
7
dev_queue_xmit_sk
vmlinux 0xffffffff8f481fc3
Build id: 3.10.0-1160.102.1.el7.x86_64
0x13
8
arp_xmit
vmlinux 0xffffffff8f50258a
Build id: 3.10.0-1160.102.1.el7.x86_64
0x3a
9
? arp_create
vmlinux 0xffffffff8f5021b3
Build id: 3.10.0-1160.102.1.el7.x86_64
0x1c3
10
arp_send_dst.part.16
vmlinux 0xffffffff8f502645
Build id: 3.10.0-1160.102.1.el7.x86_64
0x45
11
arp_process
vmlinux 0xffffffff8f503657
Build id: 3.10.0-1160.102.1.el7.x86_64
0x877
12
arp_rcv
vmlinux 0xffffffff8f50380d
Build id: 3.10.0-1160.102.1.el7.x86_64
0x12d
13
? packet_rcv
vmlinux 0xffffffff8f58b1fc
Build id: 3.10.0-1160.102.1.el7.x86_64
0x4c
14
__netif_receive_skb_core
vmlinux 0xffffffff8f47f669
Build id: 3.10.0-1160.102.1.el7.x86_64
0x739
15
? __getnstimeofday64
vmlinux 0xffffffff8ef0ea9f
Build id: 3.10.0-1160.102.1.el7.x86_64
0x3f
16
__netif_receive_skb
vmlinux 0xffffffff8f47f968
Build id: 3.10.0-1160.102.1.el7.x86_64
0x18
17
netif_receive_skb_internal
vmlinux 0xffffffff8f47fa00
Build id: 3.10.0-1160.102.1.el7.x86_64
0x40
18
napi_gro_receive
vmlinux 0xffffffff8f4806b8
Build id: 3.10.0-1160.102.1.el7.x86_64
0xd8
19
e1000_clean_rx_irq
e1000 0xffffffffc014c4cb
Build id: 3.10.0-1160.102.1.el7.x86_64
0x2bb
20
e1000_clean
e1000 0xffffffffc014d180
Build id: 3.10.0-1160.102.1.el7.x86_64
0x280
21
? sched_clock_cpu
vmlinux 0xffffffff8eee4345
Build id: 3.10.0-1160.102.1.el7.x86_64
0x85
22
net_rx_action
vmlinux 0xffffffff8f480037
Build id: 3.10.0-1160.102.1.el7.x86_64
0x277
23
__do_softirq
vmlinux 0xffffffff8eea9595
Build id: 3.10.0-1160.102.1.el7.x86_64
0xf5
24
run_ksoftirqd
vmlinux 0xffffffff8eea9768
Build id: 3.10.0-1160.102.1.el7.x86_64
0x38
25
smpboot_thread_fn
vmlinux 0xffffffff8eed43fc
Build id: 3.10.0-1160.102.1.el7.x86_64
0x14c
26
? lg_double_unlock
vmlinux 0xffffffff8eed42b0
Build id: 3.10.0-1160.102.1.el7.x86_64
0x40
27
kthread
vmlinux 0xffffffff8eecb621
Build id: 3.10.0-1160.102.1.el7.x86_64
0xd1
28
? insert_kthread_work
vmlinux 0xffffffff8eecb550
Build id: 3.10.0-1160.102.1.el7.x86_64
0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffff8f5c51dd
Build id: 3.10.0-1160.102.1.el7.x86_64
0x7
30
? insert_kthread_work
vmlinux 0xffffffff8eecb550
Build id: 3.10.0-1160.102.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ksoftirqd/0:6]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter devlink ebtables iptable_filter vmw_vsock_vmci_transport vsock fuse ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables snd_seq_midi snd_seq_midi_event ext4 iosf_mbi crc32_pclmul ghash_clmulni_intel mbcache jbd2 ppdev snd_ens1371 aesni_intel snd_rawmidi vmw_balloon snd_ac97_codec lrw gf128mul glue_helper ac97_bus ablk_helper cryptd snd_seq snd_seq_device snd_pcm joydev pcspkr snd_timer sg snd soundcore vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod sr_mod
cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix nfit serio_raw libnvdimm libata mptspi e1000 scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.102.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff9ce6792ad280 ti: ffff9ce6793e0000 task.ti: ffff9ce6793e0000
RIP: 0010:[<ffffffffc014b8e7>]  [<ffffffffc014b8e7>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9ce6793e3768  EFLAGS: 00000286
RAX: ffffab88c1c43818 RBX: 000000000000004b RCX: 0000000000000095
RDX: 0000000000000095 RSI: ffff9ce7a6eaaec0 RDI: ffff9ce7aaa76000
RBP: ffff9ce6793e37e8 R08: 0000000000000100 R09: 0000000022100000
R10: 6504000000000000 R11: 0000000000000000 R12: 0000000259197402
R13: 000000018f481fa0 R14: ffffffff8f1c9110 R15: ffff9ce6793e3720
FS:  0000000000000000(0000) GS:ffff9ce7ab800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056282a188598 CR3: 00000002a4fac000 CR4: 00000000003607f0
Call Trace:
[<ffffffff8f466c7f>] ? kfree_skbmem+0x3f/0xa0
[<ffffffff8f47edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff8f4add72>] sch_direct_xmit+0x122/0x250
[<ffffffff8f4adf2e>] __qdisc_run+0x8e/0x360
[<ffffffff8f481de9>] __dev_queue_xmit+0x4a9/0x650
[<ffffffff8f481fc3>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff8f50258a>] arp_xmit+0x3a/0xb0
[<ffffffff8f5021b3>] ? arp_create+0x1c3/0x240
[<ffffffff8f502645>] arp_send_dst.part.16+0x45/0x50
[<ffffffff8f503657>] arp_process+0x877/0x8e0
[<ffffffff8f50380d>] arp_rcv+0x12d/0x1b0
[<ffffffff8f58b1fc>] ? packet_rcv+0x4c/0x440
[<ffffffff8f47f669>] __netif_receive_skb_core+0x739/0xa20
[<ffffffff8ef0ea9f>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffff8f47f968>] __netif_receive_skb+0x18/0x70
[<ffffffff8f47fa00>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff8f4806b8>] napi_gro_receive+0xd8/0x100
[<ffffffffc014c4cb>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc014d180>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff8eee4345>] ? sched_clock_cpu+0x85/0xc0
[<ffffffff8f480037>] net_rx_action+0x277/0x3a0
[<ffffffff8eea9595>] __do_softirq+0xf5/0x290
[<ffffffff8eea9768>] run_ksoftirqd+0x38/0x60
[<ffffffff8eed43fc>] smpboot_thread_fn+0x14c/0x1b0
[<ffffffff8eed42b0>] ? lg_double_unlock+0x40/0x40
[<ffffffff8eecb621>] kthread+0xd1/0xe0
[<ffffffff8eecb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff8f5c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff8eecb550>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc 
Complete report #895492
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00e4ee7
Build id: 3.10.0-693.21.1.el7.x86_64
0x947
2
dev_hard_start_xmit
vmlinux 0xffffffff8158fa64
Build id: 3.10.0-693.21.1.el7.x86_64
0x194
3
sch_direct_xmit
vmlinux 0xffffffff815b8d2a
Build id: 3.10.0-693.21.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff815928b6
Build id: 3.10.0-693.21.1.el7.x86_64
0x226
5
dev_queue_xmit_sk
vmlinux 0xffffffff81592c13
Build id: 3.10.0-693.21.1.el7.x86_64
0x13
6
arp_xmit
vmlinux 0xffffffff81608103
Build id: 3.10.0-693.21.1.el7.x86_64
0x33
7
? arp_create
vmlinux 0xffffffff81607d63
Build id: 3.10.0-693.21.1.el7.x86_64
0x1c3
8
arp_send_dst.part.15
vmlinux 0xffffffff816081b5
Build id: 3.10.0-693.21.1.el7.x86_64
0x45
9
arp_process
vmlinux 0xffffffff81608ffe
Build id: 3.10.0-693.21.1.el7.x86_64
0x70e
10
? ip_local_deliver
vmlinux 0xffffffff815d3e19
Build id: 3.10.0-693.21.1.el7.x86_64
0x59
11
? ip_rcv_finish
vmlinux 0xffffffff815d3a70
Build id: 3.10.0-693.21.1.el7.x86_64
0x370
12
arp_rcv
vmlinux 0xffffffff816091b5
Build id: 3.10.0-693.21.1.el7.x86_64
0x125
13
? inet_del_offload
vmlinux 0xffffffff815d3700
Build id: 3.10.0-693.21.1.el7.x86_64
0x40
14
__netif_receive_skb_core
vmlinux 0xffffffff81590234
Build id: 3.10.0-693.21.1.el7.x86_64
0x594
15
? __build_skb
vmlinux 0xffffffff815810aa
Build id: 3.10.0-693.21.1.el7.x86_64
0x2a
16
? __getnstimeofday64
vmlinux 0xffffffff810ee42f
Build id: 3.10.0-693.21.1.el7.x86_64
0x3f
17
__netif_receive_skb
vmlinux 0xffffffff81590498
Build id: 3.10.0-693.21.1.el7.x86_64
0x18
18
netif_receive_skb_internal
vmlinux 0xffffffff81590520
Build id: 3.10.0-693.21.1.el7.x86_64
0x40
19
napi_gro_receive
vmlinux 0xffffffff81591648
Build id: 3.10.0-693.21.1.el7.x86_64
0xd8
20
e1000_clean_rx_irq
e1000 0xffffffffc00e647f
Build id: 3.10.0-693.21.1.el7.x86_64
0x2af
21
e1000_clean
e1000 0xffffffffc00e6eeb
Build id: 3.10.0-693.21.1.el7.x86_64
0x27b
22
? update_curr
vmlinux 0xffffffff810d11c4
Build id: 3.10.0-693.21.1.el7.x86_64
0x104
23
? account_entity_dequeue
vmlinux 0xffffffff810cdbce
Build id: 3.10.0-693.21.1.el7.x86_64
0xae
24
net_rx_action
vmlinux 0xffffffff81590cc3
Build id: 3.10.0-693.21.1.el7.x86_64
0x173
25
__do_softirq
vmlinux 0xffffffff81094035
Build id: 3.10.0-693.21.1.el7.x86_64
0xf5
26
run_ksoftirqd
vmlinux 0xffffffff810941f8
Build id: 3.10.0-693.21.1.el7.x86_64
0x38
27
smpboot_thread_fn
vmlinux 0xffffffff810bc774
Build id: 3.10.0-693.21.1.el7.x86_64
0x144
28
? lg_double_unlock
vmlinux 0xffffffff810bc630
Build id: 3.10.0-693.21.1.el7.x86_64
0x40
29
kthread
vmlinux 0xffffffff810b4031
Build id: 3.10.0-693.21.1.el7.x86_64
0xd1
30
? insert_kthread_work
vmlinux 0xffffffff810b3f60
Build id: 3.10.0-693.21.1.el7.x86_64
0x40
31
ret_from_fork
vmlinux 0xffffffff816c0577
Build id: 3.10.0-693.21.1.el7.x86_64
0x77
32
? insert_kthread_work
vmlinux 0xffffffff810b3f60
Build id: 3.10.0-693.21.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [ksoftirqd/1:13]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_ftp ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac edac_core coretemp iosf_mbi crc32_pclmul ext4 ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec mbcache ac97_bus jbd2 snd_seq snd_seq_device snd_pcm
aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr snd_timer snd sg nfit soundcore i2c_piix4 vmw_vmci shpchp libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic vmwgfx pata_acpi drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm serio_raw mptspi e1000 scsi_transport_spi mptscsih mptbase ata_piix libata i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 13 Comm: ksoftirqd/1 Tainted: G             L ------------   3.10.0-693.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff88017c9aaf70 ti: ffff88017c9b8000 task.ti: ffff88017c9b8000
RIP: 0010:[<ffffffffc00e4ee7>]  [<ffffffffc00e4ee7>] e1000_xmit_frame+0x947/0x1040 [e1000]
RSP: 0018:ffff88017c9bb800  EFLAGS: 00000286
RAX: ffffc90000e03818 RBX: 0000000000000000 RCX: 0000000000000015
RDX: 0000000000000015 RSI: ffff88023147a200 RDI: ffff880235b86000
RBP: ffff88017c9bb880 R08: 0000000000000100 R09: 0000000002000000
R10: 0000000080000000 R11: 0000000000000000 R12: ffff8800b8b0a100
R13: ffff88017fc00e00 R14: ffffffff811e1f29 R15: ffff88017c9bb828
FS:  0000000000000000(0000) GS:ffff880236640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f1ebf9f6000 CR3: 0000000225e5e000 CR4: 00000000001607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8158fa64>] dev_hard_start_xmit+0x194/0x3d0
[<ffffffff815b8d2a>] sch_direct_xmit+0x11a/0x240
[<ffffffff815928b6>] __dev_queue_xmit+0x226/0x550
[<ffffffff81592c13>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff81608103>] arp_xmit+0x33/0xa0
[<ffffffff81607d63>] ? arp_create+0x1c3/0x230
[<ffffffff816081b5>] arp_send_dst.part.15+0x45/0x50
[<ffffffff81608ffe>] arp_process+0x70e/0x780
[<ffffffff815d3e19>] ? ip_local_deliver+0x59/0xd0
[<ffffffff815d3a70>] ? ip_rcv_finish+0x370/0x370
[<ffffffff816091b5>] arp_rcv+0x125/0x1b0
[<ffffffff815d3700>] ? inet_del_offload+0x40/0x40
[<ffffffff81590234>] __netif_receive_skb_core+0x594/0x7e0
[<ffffffff815810aa>] ? __build_skb+0x2a/0x1b0
[<ffffffff810ee42f>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffff81590498>] __netif_receive_skb+0x18/0x60
[<ffffffff81590520>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff81591648>] napi_gro_receive+0xd8/0x130
[<ffffffffc00e647f>] e1000_clean_rx_irq+0x2af/0x500 [e1000]
[<ffffffffc00e6eeb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffff810d11c4>] ? update_curr+0x104/0x190
[<ffffffff810cdbce>] ? account_entity_dequeue+0xae/0xd0
[<ffffffff81590cc3>] net_rx_action+0x173/0x380
[<ffffffff81094035>] __do_softirq+0xf5/0x280
[<ffffffff810941f8>] run_ksoftirqd+0x38/0x50
[<ffffffff810bc774>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff810bc630>] ? lg_double_unlock+0x40/0x40
[<ffffffff810b4031>] kthread+0xd1/0xe0
[<ffffffff810b3f60>] ? insert_kthread_work+0x40/0x40
[<ffffffff816c0577>] ret_from_fork+0x77/0xb0
[<ffffffff810b3f60>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #767074
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00d98e7
Build id: 3.10.0-1160.80.1.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffb4a66c7f
Build id: 3.10.0-1160.80.1.el7.x86_64
0x3f
3
dev_hard_start_xmit
vmlinux 0xffffffffb4a7edc6
Build id: 3.10.0-1160.80.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffffb4aadd72
Build id: 3.10.0-1160.80.1.el7.x86_64
0x122
5
__qdisc_run
vmlinux 0xffffffffb4aadf2e
Build id: 3.10.0-1160.80.1.el7.x86_64
0x8e
6
__dev_queue_xmit
vmlinux 0xffffffffb4a81de9
Build id: 3.10.0-1160.80.1.el7.x86_64
0x4a9
7
dev_queue_xmit_sk
vmlinux 0xffffffffb4a81fc3
Build id: 3.10.0-1160.80.1.el7.x86_64
0x13
8
arp_xmit
vmlinux 0xffffffffb4b0258a
Build id: 3.10.0-1160.80.1.el7.x86_64
0x3a
9
? arp_create
vmlinux 0xffffffffb4b021b3
Build id: 3.10.0-1160.80.1.el7.x86_64
0x1c3
10
arp_send_dst.part.16
vmlinux 0xffffffffb4b02645
Build id: 3.10.0-1160.80.1.el7.x86_64
0x45
11
arp_process
vmlinux 0xffffffffb4b03657
Build id: 3.10.0-1160.80.1.el7.x86_64
0x877
12
? ip_rcv_finish
vmlinux 0xffffffffb4acc290
Build id: 3.10.0-1160.80.1.el7.x86_64
0x380
13
arp_rcv
vmlinux 0xffffffffb4b0380d
Build id: 3.10.0-1160.80.1.el7.x86_64
0x12d
14
? packet_rcv
vmlinux 0xffffffffb4b8b1fc
Build id: 3.10.0-1160.80.1.el7.x86_64
0x4c
15
__netif_receive_skb_core
vmlinux 0xffffffffb4a7f669
Build id: 3.10.0-1160.80.1.el7.x86_64
0x739
16
? __getnstimeofday64
vmlinux 0xffffffffb450e8df
Build id: 3.10.0-1160.80.1.el7.x86_64
0x3f
17
__netif_receive_skb
vmlinux 0xffffffffb4a7f968
Build id: 3.10.0-1160.80.1.el7.x86_64
0x18
18
netif_receive_skb_internal
vmlinux 0xffffffffb4a7fa00
Build id: 3.10.0-1160.80.1.el7.x86_64
0x40
19
napi_gro_receive
vmlinux 0xffffffffb4a806b8
Build id: 3.10.0-1160.80.1.el7.x86_64
0xd8
20
e1000_clean_rx_irq
e1000 0xffffffffc00da4cb
Build id: 3.10.0-1160.80.1.el7.x86_64
0x2bb
21
e1000_clean
e1000 0xffffffffc00db180
Build id: 3.10.0-1160.80.1.el7.x86_64
0x280
22
net_rx_action
vmlinux 0xffffffffb4a80037
Build id: 3.10.0-1160.80.1.el7.x86_64
0x277
23
__do_softirq
vmlinux 0xffffffffb44a9545
Build id: 3.10.0-1160.80.1.el7.x86_64
0xf5
24
run_ksoftirqd
vmlinux 0xffffffffb44a9718
Build id: 3.10.0-1160.80.1.el7.x86_64
0x38
25
smpboot_thread_fn
vmlinux 0xffffffffb44d42ac
Build id: 3.10.0-1160.80.1.el7.x86_64
0x14c
26
? lg_double_unlock
vmlinux 0xffffffffb44d4160
Build id: 3.10.0-1160.80.1.el7.x86_64
0x40
27
kthread
vmlinux 0xffffffffb44cb4d1
Build id: 3.10.0-1160.80.1.el7.x86_64
0xd1
28
? insert_kthread_work
vmlinux 0xffffffffb44cb400
Build id: 3.10.0-1160.80.1.el7.x86_64
0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffffb4bc51dd
Build id: 3.10.0-1160.80.1.el7.x86_64
0x7
30
? insert_kthread_work
vmlinux 0xffffffffb44cb400
Build id: 3.10.0-1160.80.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 21s! [ksoftirqd/6:39]
Modules linked in: binfmt_misc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm kvm_intel snd_timer snd kvm irqbypass soundcore crc32_pclmul ghash_clmulni_intel aesni_intel
lrw gf128mul glue_helper ablk_helper pcspkr sg cryptd ppdev joydev vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel nfit mptspi libnvdimm e1000 scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 6 PID: 39 Comm: ksoftirqd/6 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.80.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9634f8cd4200 ti: ffff9634f8cf4000 task.ti: ffff9634f8cf4000
RIP: 0010:[<ffffffffc00d98e7>]  [<ffffffffc00d98e7>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9634f8cf7768  EFLAGS: 00000286
RAX: ffffb85901983818 RBX: 0000000000000042 RCX: 000000000000000e
RDX: 000000000000000e RSI: ffff9633b5510140 RDI: ffff9634f94be000
RBP: ffff9634f8cf77e8 R08: 0000000000000100 R09: 0000000022100000
R10: 3400004500081896 R11: 0000000000000000 R12: 00000001463a6cce
R13: 00000001f211c280 R14: ffffffffb47c8a90 R15: ffff9634f8cf7720
FS:  0000000000000000(0000) GS:ffff963600780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f16ec96e000 CR3: 000000027d338000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffb4a66c7f>] ? kfree_skbmem+0x3f/0xa0
[<ffffffffb4a7edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb4aadd72>] sch_direct_xmit+0x122/0x250
[<ffffffffb4aadf2e>] __qdisc_run+0x8e/0x360
[<ffffffffb4a81de9>] __dev_queue_xmit+0x4a9/0x650
[<ffffffffb4a81fc3>] dev_queue_xmit_sk+0x13/0x20
[<ffffffffb4b0258a>] arp_xmit+0x3a/0xb0
[<ffffffffb4b021b3>] ? arp_create+0x1c3/0x240
[<ffffffffb4b02645>] arp_send_dst.part.16+0x45/0x50
[<ffffffffb4b03657>] arp_process+0x877/0x8e0
[<ffffffffb4acc290>] ? ip_rcv_finish+0x380/0x380
[<ffffffffb4b0380d>] arp_rcv+0x12d/0x1b0
[<ffffffffb4b8b1fc>] ? packet_rcv+0x4c/0x440
[<ffffffffb4a7f669>] __netif_receive_skb_core+0x739/0xa20
[<ffffffffb450e8df>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffffb4a7f968>] __netif_receive_skb+0x18/0x70
[<ffffffffb4a7fa00>] netif_receive_skb_internal+0x40/0xc0
[<ffffffffb4a806b8>] napi_gro_receive+0xd8/0x100
[<ffffffffc00da4cb>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc00db180>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffffb4a80037>] net_rx_action+0x277/0x3a0
[<ffffffffb44a9545>] __do_softirq+0xf5/0x290
[<ffffffffb44a9718>] run_ksoftirqd+0x38/0x60
[<ffffffffb44d42ac>] smpboot_thread_fn+0x14c/0x1b0
[<ffffffffb44d4160>] ? lg_double_unlock+0x40/0x40
[<ffffffffb44cb4d1>] kthread+0xd1/0xe0
[<ffffffffb44cb400>] ? insert_kthread_work+0x40/0x40
[<ffffffffb4bc51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb44cb400>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc 
Complete report #701046
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc007cee7
Build id: 3.10.0-693.2.2.el7.x86_64
0x947
2
dev_hard_start_xmit
vmlinux 0xffffffff81586771
Build id: 3.10.0-693.2.2.el7.x86_64
0x171
3
sch_direct_xmit
vmlinux 0xffffffff815af52a
Build id: 3.10.0-693.2.2.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff81589566
Build id: 3.10.0-693.2.2.el7.x86_64
0x226
5
dev_queue_xmit_sk
vmlinux 0xffffffff815898c3
Build id: 3.10.0-693.2.2.el7.x86_64
0x13
6
arp_xmit
vmlinux 0xffffffff815fe083
Build id: 3.10.0-693.2.2.el7.x86_64
0x33
7
? arp_create
vmlinux 0xffffffff815fdce0
Build id: 3.10.0-693.2.2.el7.x86_64
0x1c0
8
arp_send_dst.part.15
vmlinux 0xffffffff815fe135
Build id: 3.10.0-693.2.2.el7.x86_64
0x45
9
arp_process
vmlinux 0xffffffff815fef7e
Build id: 3.10.0-693.2.2.el7.x86_64
0x70e
10
? ip_local_deliver
vmlinux 0xffffffff815ca359
Build id: 3.10.0-693.2.2.el7.x86_64
0x59
11
? ___slab_alloc
vmlinux 0xffffffff811dd579
Build id: 3.10.0-693.2.2.el7.x86_64
0x209
12
? ip_rcv_finish
vmlinux 0xffffffff815c9fc0
Build id: 3.10.0-693.2.2.el7.x86_64
0x350
13
arp_rcv
vmlinux 0xffffffff815ff135
Build id: 3.10.0-693.2.2.el7.x86_64
0x125
14
? inet_del_offload
vmlinux 0xffffffff815c9c70
Build id: 3.10.0-693.2.2.el7.x86_64
0x40
15
__netif_receive_skb_core
vmlinux 0xffffffff81586f22
Build id: 3.10.0-693.2.2.el7.x86_64
0x572
16
? __build_skb
vmlinux 0xffffffff81577e8a
Build id: 3.10.0-693.2.2.el7.x86_64
0x2a
17
__netif_receive_skb
vmlinux 0xffffffff81587188
Build id: 3.10.0-693.2.2.el7.x86_64
0x18
18
netif_receive_skb_internal
vmlinux 0xffffffff81587210
Build id: 3.10.0-693.2.2.el7.x86_64
0x40
19
napi_gro_receive
vmlinux 0xffffffff81588318
Build id: 3.10.0-693.2.2.el7.x86_64
0xd8
20
e1000_clean_rx_irq
e1000 0xffffffffc007e47c
Build id: 3.10.0-693.2.2.el7.x86_64
0x2ac
21
e1000_clean
e1000 0xffffffffc007eed8
Build id: 3.10.0-693.2.2.el7.x86_64
0x278
22
net_rx_action
vmlinux 0xffffffff8158799d
Build id: 3.10.0-693.2.2.el7.x86_64
0x16d
23
__do_softirq
vmlinux 0xffffffff81090b3f
Build id: 3.10.0-693.2.2.el7.x86_64
0xef
24
run_ksoftirqd
vmlinux 0xffffffff81090d08
Build id: 3.10.0-693.2.2.el7.x86_64
0x38
25
smpboot_thread_fn
vmlinux 0xffffffff810b909f
Build id: 3.10.0-693.2.2.el7.x86_64
0x12f
26
? lg_double_unlock
vmlinux 0xffffffff810b8f70
Build id: 3.10.0-693.2.2.el7.x86_64
0x40
27
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.2.2.el7.x86_64
0xcf
28
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.2.2.el7.x86_64
0x40
29
ret_from_fork
vmlinux 0xffffffff816b4f58
Build id: 3.10.0-693.2.2.el7.x86_64
0x58
30
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.2.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ksoftirqd/0:3]
Modules linked in: xt_multiport ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipportip ip_set_hash_ipport dummy vxlan ip6_udp_tunnel udp_tunnel ipt_rpfilter xt_set iptable_raw ip_set_hash_ip ip_set_hash_net ip_set ipip tunnel4 ip_tunnel xt_statistic ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_mangle ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_filter iptable_mangle ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables xt_comment xt_mark binfmt_misc veth xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul
ppdev snd_ens1371 ghash_clmulni_intel snd_rawmidi aesni_intel lrw gf128mul snd_ac97_codec glue_helper ablk_helper cryptd ac97_bus snd_seq vmw_balloon snd_seq_device snd_pcm pcspkr joydev snd_timer snd soundcore nfit sg libnvdimm parport_pc parport vmw_vmci i2c_piix4 shpchp ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper ata_piix mptspi scsi_transport_spi syscopyarea sysfillrect mptscsih sysimgblt fb_sys_fops ttm libata mptbase drm e1000 crct10dif_pclmul crct10dif_common crc32c_intel serio_raw i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G             L ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff88007b451fa0 ti: ffff88007b4cc000 task.ti: ffff88007b4cc000
RIP: 0010:[<ffffffffc007cee7>]  [<ffffffffc007cee7>] e1000_xmit_frame+0x947/0x1040 [e1000]
RSP: 0018:ffff88007b4cf800  EFLAGS: 00000286
RAX: ffffc90000403818 RBX: ffff880008135d00 RCX: 0000000000000064
RDX: 0000000000000064 RSI: ffff8800362eeb00 RDI: ffff880079f92000
RBP: ffff88007b4cf880 R08: 0000000000000100 R09: 0000000002000000
R10: 0000000080000000 R11: 0000000000000000 R12: 00000000c0653f47
R13: 128ea8c07b4cf7a0 R14: ffff880000000000 R15: ffffffff815cfacf
FS:  0000000000000000(0000) GS:ffff88007c600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000004036f9 CR3: 0000000079370000 CR4: 00000000001407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff88007b4cf8b0 ffff88007b345098 000000637a122e00 0000000000000000
000000007971c000 ffff880000000000 ffff880000000000 0000000100001000
ffff8800087aee00 ffff880079f92000 ffff88007b345000 ffff880079f92000
Call Trace:
[<ffffffff81586771>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff815af52a>] sch_direct_xmit+0x11a/0x240
[<ffffffff81589566>] __dev_queue_xmit+0x226/0x550
[<ffffffff815898c3>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff815fe083>] arp_xmit+0x33/0xa0
[<ffffffff815fdce0>] ? arp_create+0x1c0/0x230
[<ffffffff815fe135>] arp_send_dst.part.15+0x45/0x50
[<ffffffff815fef7e>] arp_process+0x70e/0x780
[<ffffffff815ca359>] ? ip_local_deliver+0x59/0xd0
[<ffffffff811dd579>] ? ___slab_alloc+0x209/0x4f0
[<ffffffff815c9fc0>] ? ip_rcv_finish+0x350/0x350
[<ffffffff815ff135>] arp_rcv+0x125/0x1b0
[<ffffffff815c9c70>] ? inet_del_offload+0x40/0x40
[<ffffffff81586f22>] __netif_receive_skb_core+0x572/0x7c0
[<ffffffff81577e8a>] ? __build_skb+0x2a/0x1b0
[<ffffffff81587188>] __netif_receive_skb+0x18/0x60
[<ffffffff81587210>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff81588318>] napi_gro_receive+0xd8/0x130
[<ffffffffc007e47c>] e1000_clean_rx_irq+0x2ac/0x4f0 [e1000]
[<ffffffffc007eed8>] e1000_clean+0x278/0x8d0 [e1000]
[<ffffffff8158799d>] net_rx_action+0x16d/0x380
[<ffffffff81090b3f>] __do_softirq+0xef/0x280
[<ffffffff81090d08>] run_ksoftirqd+0x38/0x50
[<ffffffff810b909f>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f70>] ? lg_double_unlock+0x40/0x40
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f58>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #584620
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0104867
Build id: 3.10.0-862.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffff939d4397
Build id: 3.10.0-862.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffff939eb166
Build id: 3.10.0-862.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffff93a16a2a
Build id: 3.10.0-862.el7.x86_64
0x11a
5
__qdisc_run
vmlinux 0xffffffff93a16bee
Build id: 3.10.0-862.el7.x86_64
0x8e
6
__dev_queue_xmit
vmlinux 0xffffffff939ee069
Build id: 3.10.0-862.el7.x86_64
0x4b9
7
dev_queue_xmit_sk
vmlinux 0xffffffff939ee243
Build id: 3.10.0-862.el7.x86_64
0x13
8
arp_xmit
vmlinux 0xffffffff93a65ae3
Build id: 3.10.0-862.el7.x86_64
0x33
9
? arp_create
vmlinux 0xffffffff93a65743
Build id: 3.10.0-862.el7.x86_64
0x1c3
10
arp_send_dst.part.16
vmlinux 0xffffffff93a65b95
Build id: 3.10.0-862.el7.x86_64
0x45
11
arp_process
vmlinux 0xffffffff93a66b77
Build id: 3.10.0-862.el7.x86_64
0x877
12
? sk_run_filter
vmlinux 0xffffffff93a061f0
Build id: 3.10.0-862.el7.x86_64
0x550
13
? ttwu_do_activate.constprop.93
vmlinux 0xffffffff934cba9d
Build id: 3.10.0-862.el7.x86_64
0x5d
14
arp_rcv
vmlinux 0xffffffff93a66d25
Build id: 3.10.0-862.el7.x86_64
0x125
15
? packet_rcv
vmlinux 0xffffffff93aed08c
Build id: 3.10.0-862.el7.x86_64
0x4c
16
? default_wake_function
vmlinux 0xffffffff934cee92
Build id: 3.10.0-862.el7.x86_64
0x12
17
__netif_receive_skb_core
vmlinux 0xffffffff939eb9f9
Build id: 3.10.0-862.el7.x86_64
0x729
18
? __getnstimeofday64
vmlinux 0xffffffff934f8c5f
Build id: 3.10.0-862.el7.x86_64
0x3f
19
__netif_receive_skb
vmlinux 0xffffffff939ebd08
Build id: 3.10.0-862.el7.x86_64
0x18
20
netif_receive_skb_internal
vmlinux 0xffffffff939ebd90
Build id: 3.10.0-862.el7.x86_64
0x40
21
napi_gro_receive
vmlinux 0xffffffff939ec9b8
Build id: 3.10.0-862.el7.x86_64
0xd8
22
e1000_clean_rx_irq
e1000 0xffffffffc010544b
Build id: 3.10.0-862.el7.x86_64
0x2bb
23
e1000_clean
e1000 0xffffffffc01060eb
Build id: 3.10.0-862.el7.x86_64
0x27b
24
net_rx_action
vmlinux 0xffffffff939ec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
25
__do_softirq
vmlinux 0xffffffff9349a945
Build id: 3.10.0-862.el7.x86_64
0xf5
26
run_ksoftirqd
vmlinux 0xffffffff9349ab08
Build id: 3.10.0-862.el7.x86_64
0x38
27
smpboot_thread_fn
vmlinux 0xffffffff934c35e4
Build id: 3.10.0-862.el7.x86_64
0x144
28
? lg_double_unlock
vmlinux 0xffffffff934c34a0
Build id: 3.10.0-862.el7.x86_64
0x40
29
kthread
vmlinux 0xffffffff934bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
30
? insert_kthread_work
vmlinux 0xffffffff934bad60
Build id: 3.10.0-862.el7.x86_64
0x40
31
ret_from_fork_nospec_begin
vmlinux 0xffffffff93b1f637
Build id: 3.10.0-862.el7.x86_64
0x21
32
? insert_kthread_work
vmlinux 0xffffffff934bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [ksoftirqd/0:3]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 devlink xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi snd_ens1371 kvm_intel snd_rawmidi kvm snd_ac97_codec ac97_bus snd_seq irqbypass snd_seq_device crc32_pclmul snd_pcm ghash_clmulni_intel aesni_intel snd_timer lrw snd gf128mul glue_helper ablk_helper
ppdev cryptd soundcore vmw_balloon sg pcspkr vmw_vmci i2c_piix4 joydev shpchp nfit libnvdimm parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ata_piix ttm drm crct10dif_pclmul crct10dif_common mptspi scsi_transport_spi crc32c_intel e1000 mptscsih libata mptbase serio_raw i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9bef7d719fa0 ti: ffff9bef7d794000 task.ti: ffff9bef7d794000
RIP: 0010:[<ffffffffc0104867>]  [<ffffffffc0104867>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9bef7d797778  EFLAGS: 00000286
RAX: ffffa8e000f83818 RBX: 00000000000000a6 RCX: 0000000000000098
RDX: 0000000000000098 RSI: ffff9befbfebf700 RDI: ffff9bef7d622000
RBP: ffff9bef7d7977f8 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: 00000001aa6a80ce
R13: 0000000195fb3900 R14: ffffffff93773e00 R15: ffff9bef7d797730
FS:  0000000000000000(0000) GS:ffff9befb7600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00002abf7af67f70 CR3: 00000001a8a6e000 CR4: 00000000001607f0
Call Trace:
[<ffffffff939d4397>] ? kfree_skbmem+0x37/0x90
[<ffffffff939eb166>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff93a16a2a>] sch_direct_xmit+0x11a/0x250
[<ffffffff93a16bee>] __qdisc_run+0x8e/0x360
[<ffffffff939ee069>] __dev_queue_xmit+0x4b9/0x660
[<ffffffff939ee243>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff93a65ae3>] arp_xmit+0x33/0xa0
[<ffffffff93a65743>] ? arp_create+0x1c3/0x230
[<ffffffff93a65b95>] arp_send_dst.part.16+0x45/0x50
[<ffffffff93a66b77>] arp_process+0x877/0x8e0
[<ffffffff93a061f0>] ? sk_run_filter+0x550/0x760
[<ffffffff934cba9d>] ? ttwu_do_activate.constprop.93+0x5d/0x70
[<ffffffff93a66d25>] arp_rcv+0x125/0x1b0
[<ffffffff93aed08c>] ? packet_rcv+0x4c/0x430
[<ffffffff934cee92>] ? default_wake_function+0x12/0x20
[<ffffffff939eb9f9>] __netif_receive_skb_core+0x729/0xa20
[<ffffffff934f8c5f>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffff939ebd08>] __netif_receive_skb+0x18/0x60
[<ffffffff939ebd90>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff939ec9b8>] napi_gro_receive+0xd8/0x100
[<ffffffffc010544b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc01060eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffff939ec3af>] net_rx_action+0x26f/0x390
[<ffffffff9349a945>] __do_softirq+0xf5/0x280
[<ffffffff9349ab08>] run_ksoftirqd+0x38/0x50
[<ffffffff934c35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff934c34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffff934bae31>] kthread+0xd1/0xe0
[<ffffffff934bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff93b1f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff934bad60>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #581436
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00cdee7
Build id: 3.10.0-693.el7.x86_64
0x947
2
? kfree_skbmem
vmlinux 0xffffffff815710d7
Build id: 3.10.0-693.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffff81586771
Build id: 3.10.0-693.el7.x86_64
0x171
4
sch_direct_xmit
vmlinux 0xffffffff815af50a
Build id: 3.10.0-693.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffff81589566
Build id: 3.10.0-693.el7.x86_64
0x226
6
dev_queue_xmit_sk
vmlinux 0xffffffff815898c3
Build id: 3.10.0-693.el7.x86_64
0x13
7
arp_xmit
vmlinux 0xffffffff815fe063
Build id: 3.10.0-693.el7.x86_64
0x33
8
? arp_create
vmlinux 0xffffffff815fdcc0
Build id: 3.10.0-693.el7.x86_64
0x1c0
9
arp_send_dst.part.15
vmlinux 0xffffffff815fe115
Build id: 3.10.0-693.el7.x86_64
0x45
10
arp_process
vmlinux 0xffffffff815fef5e
Build id: 3.10.0-693.el7.x86_64
0x70e
11
? ip_local_deliver
vmlinux 0xffffffff815ca392
Build id: 3.10.0-693.el7.x86_64
0xb2
12
? ip_rcv_finish
vmlinux 0xffffffff815c9fa0
Build id: 3.10.0-693.el7.x86_64
0x350
13
arp_rcv
vmlinux 0xffffffff815ff115
Build id: 3.10.0-693.el7.x86_64
0x125
14
? packet_rcv
vmlinux 0xffffffff8168438c
Build id: 3.10.0-693.el7.x86_64
0x4c
15
__netif_receive_skb_core
vmlinux 0xffffffff81586f22
Build id: 3.10.0-693.el7.x86_64
0x572
16
? __getnstimeofday64
vmlinux 0xffffffff810ea8ca
Build id: 3.10.0-693.el7.x86_64
0x3a
17
__netif_receive_skb
vmlinux 0xffffffff81587188
Build id: 3.10.0-693.el7.x86_64
0x18
18
netif_receive_skb_internal
vmlinux 0xffffffff81587210
Build id: 3.10.0-693.el7.x86_64
0x40
19
napi_gro_receive
vmlinux 0xffffffff81588318
Build id: 3.10.0-693.el7.x86_64
0xd8
20
e1000_clean_rx_irq
e1000 0xffffffffc00cf47c
Build id: 3.10.0-693.el7.x86_64
0x2ac
21
e1000_clean
e1000 0xffffffffc00cfed8
Build id: 3.10.0-693.el7.x86_64
0x278
22
net_rx_action
vmlinux 0xffffffff8158799d
Build id: 3.10.0-693.el7.x86_64
0x16d
23
__do_softirq
vmlinux 0xffffffff81090b3f
Build id: 3.10.0-693.el7.x86_64
0xef
24
run_ksoftirqd
vmlinux 0xffffffff81090d08
Build id: 3.10.0-693.el7.x86_64
0x38
25
smpboot_thread_fn
vmlinux 0xffffffff810b909f
Build id: 3.10.0-693.el7.x86_64
0x12f
26
? lg_double_unlock
vmlinux 0xffffffff810b8f70
Build id: 3.10.0-693.el7.x86_64
0x40
27
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
28
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
29
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
30
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [ksoftirqd/0:3]
Modules linked in: xt_CHECKSUM tun ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus joydev pcspkr snd_seq snd_seq_device
snd_pcm snd_timer snd sg soundcore nfit libnvdimm parport_pc parport vmw_vmci i2c_piix4 shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx serio_raw drm_kms_helper mptspi scsi_transport_spi mptscsih syscopyarea sysfillrect sysimgblt fb_sys_fops e1000 ttm mptbase drm ata_piix libata i2c_core
CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff88007a4c9fa0 ti: ffff88007a4ec000 task.ti: ffff88007a4ec000
RIP: 0010:[<ffffffffc00cdee7>]  [<ffffffffc00cdee7>] e1000_xmit_frame+0x947/0x1040 [e1000]
RSP: 0018:ffff88007a4ef800  EFLAGS: 00000286
RAX: ffffc90000403818 RBX: 0000000000000000 RCX: 000000000000002e
RDX: 000000000000002e RSI: ffff880035173140 RDI: ffff880078832000
RBP: ffff88007a4ef880 R08: 0000000000000100 R09: 0000000002000000
R10: 0000000080000000 R11: 0000000000000000 R12: ffffffff810c4810
R13: ffff88007682dee0 R14: 0000000000000000 R15: ffff880059f11100
FS:  0000000000000000(0000) GS:ffff88007b600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbe12bc3000 CR3: 0000000077d79000 CR4: 00000000003407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffff815710d7 ffff880079dbe098 0000002d81573004 0000000000000000
0000000075a04800 ffff880000000000 ffffffff00000000 0000000100001000
ffff88006cef9800 ffff880078832000 ffff880079dbe000 ffff880078832000
Call Trace:
[<ffffffff815710d7>] ? kfree_skbmem+0x37/0x90
[<ffffffff81586771>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff815af50a>] sch_direct_xmit+0x11a/0x240
[<ffffffff81589566>] __dev_queue_xmit+0x226/0x550
[<ffffffff815898c3>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff815fe063>] arp_xmit+0x33/0xa0
[<ffffffff815fdcc0>] ? arp_create+0x1c0/0x230
[<ffffffff815fe115>] arp_send_dst.part.15+0x45/0x50
[<ffffffff815fef5e>] arp_process+0x70e/0x780
[<ffffffff815ca392>] ? ip_local_deliver+0xb2/0xd0
[<ffffffff815c9fa0>] ? ip_rcv_finish+0x350/0x350
[<ffffffff815ff115>] arp_rcv+0x125/0x1b0
[<ffffffff8168438c>] ? packet_rcv+0x4c/0x420
[<ffffffff81586f22>] __netif_receive_skb_core+0x572/0x7c0
[<ffffffff810ea8ca>] ? __getnstimeofday64+0x3a/0xd0
[<ffffffff81587188>] __netif_receive_skb+0x18/0x60
[<ffffffff81587210>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff81588318>] napi_gro_receive+0xd8/0x130
[<ffffffffc00cf47c>] e1000_clean_rx_irq+0x2ac/0x4f0 [e1000]
[<ffffffffc00cfed8>] e1000_clean+0x278/0x8d0 [e1000]
[<ffffffff8158799d>] net_rx_action+0x16d/0x380
[<ffffffff81090b3f>] __do_softirq+0xef/0x280
[<ffffffff81090d08>] run_ksoftirqd+0x38/0x50
[<ffffffff810b909f>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f70>] ? lg_double_unlock+0x40/0x40
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #175985
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffa012bd57
Build id: 3.10.0-514.el7.x86_64
0x957
2
dev_hard_start_xmit
vmlinux 0xffffffff8156edc1
Build id: 3.10.0-514.el7.x86_64
0x171
3
sch_direct_xmit
vmlinux 0xffffffff81596424
Build id: 3.10.0-514.el7.x86_64
0x104
4
__dev_queue_xmit
vmlinux 0xffffffff815719bc
Build id: 3.10.0-514.el7.x86_64
0x23c
5
dev_queue_xmit_sk
vmlinux 0xffffffff81571d23
Build id: 3.10.0-514.el7.x86_64
0x13
6
arp_xmit
vmlinux 0xffffffff815e4063
Build id: 3.10.0-514.el7.x86_64
0x33
7
? arp_create
vmlinux 0xffffffff815e3cc0
Build id: 3.10.0-514.el7.x86_64
0x1c0
8
arp_send_dst.part.15
vmlinux 0xffffffff815e4115
Build id: 3.10.0-514.el7.x86_64
0x45
9
arp_process
vmlinux 0xffffffff815e4f5e
Build id: 3.10.0-514.el7.x86_64
0x70e
10
? tcp_rcv_established
vmlinux 0xffffffff815ca8bd
Build id: 3.10.0-514.el7.x86_64
0x1bd
11
arp_rcv
vmlinux 0xffffffff815e5115
Build id: 3.10.0-514.el7.x86_64
0x125
12
? tcp_v4_rcv
vmlinux 0xffffffff815d6b5c
Build id: 3.10.0-514.el7.x86_64
0x7ac
13
__netif_receive_skb_core
vmlinux 0xffffffff8156f582
Build id: 3.10.0-514.el7.x86_64
0x582
14
__netif_receive_skb
vmlinux 0xffffffff8156f818
Build id: 3.10.0-514.el7.x86_64
0x18
15
netif_receive_skb_internal
vmlinux 0xffffffff8156f8a0
Build id: 3.10.0-514.el7.x86_64
0x40
16
napi_gro_receive
vmlinux 0xffffffff81570a28
Build id: 3.10.0-514.el7.x86_64
0xd8
17
e1000_clean_rx_irq
e1000 0xffffffffa012d2fc
Build id: 3.10.0-514.el7.x86_64
0x2ac
18
e1000_clean
e1000 0xffffffffa012dd31
Build id: 3.10.0-514.el7.x86_64
0x281
19
net_rx_action
vmlinux 0xffffffff815700b0
Build id: 3.10.0-514.el7.x86_64
0x170
20
__do_softirq
vmlinux 0xffffffff8108f0ff
Build id: 3.10.0-514.el7.x86_64
0xef
21
run_ksoftirqd
vmlinux 0xffffffff8108f2c8
Build id: 3.10.0-514.el7.x86_64
0x38
22
smpboot_thread_fn
vmlinux 0xffffffff810b90af
Build id: 3.10.0-514.el7.x86_64
0x12f
23
? lg_double_unlock
vmlinux 0xffffffff810b8f80
Build id: 3.10.0-514.el7.x86_64
0x90
24
kthread
vmlinux 0xffffffff810b052f
Build id: 3.10.0-514.el7.x86_64
0xcf
25
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
26
ret_from_fork
vmlinux 0xffffffff81696518
Build id: 3.10.0-514.el7.x86_64
0x58
27
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [ksoftirqd/0:3]
Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_nat_ipv4 nf_nat br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip6table_filter ip6_tables snd_seq_midi snd_seq_midi_event intel_powerclamp coretemp iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi vmw_balloon snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm pcspkr snd_timer snd soundcore sg i2c_piix4 vmw_vmci shpchp nfit libnvdimm parport_pc parport nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul
crct10dif_common crc32c_intel vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw ttm drm ata_piix mptspi scsi_transport_spi mptscsih e1000 mptbase i2c_core libata fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G             L ------------ T 3.10.0-514.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff880149731f60 ti: ffff880149038000 task.ti: ffff880149038000
RIP: 0010:[<ffffffffa012bd57>]  [<ffffffffa012bd57>] e1000_xmit_frame+0x957/0x1040 [e1000]
RSP: 0000:ffff88014903b808  EFLAGS: 00000286
RAX: ffffc90000d03818 RBX: ffff88014903b7a0 RCX: 00000000000000a3
RDX: 00000000000000a3 RSI: ffff88014475da00 RDI: ffff88014703e000
RBP: ffff88014903b888 R08: 0000000000000100 R09: 0000000002000000
R10: ffff88014703e000 R11: 0000000000000000 R12: ffff880009e7ae00
R13: 0000000063a78946 R14: ffffff000322dad8 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff88014b800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000c00012f010 CR3: 000000011fea3000 CR4: 00000000001407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff88014903b8b8 ffff880148a45098 000000a275dbb000 0000000000000000
0000000042b57900 ffff880000000000 ffff880100000000 0000000100001000
ffff880009e7b500 ffff88014703e000 ffff880148a45000 ffff88014703e000
Call Trace:
[<ffffffff8156edc1>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff81596424>] sch_direct_xmit+0x104/0x200
[<ffffffff815719bc>] __dev_queue_xmit+0x23c/0x570
[<ffffffff81571d23>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff815e4063>] arp_xmit+0x33/0xa0
[<ffffffff815e3cc0>] ? arp_create+0x1c0/0x230
[<ffffffff815e4115>] arp_send_dst.part.15+0x45/0x50
[<ffffffff815e4f5e>] arp_process+0x70e/0x780
[<ffffffff815ca8bd>] ? tcp_rcv_established+0x1bd/0x760
[<ffffffff815e5115>] arp_rcv+0x125/0x1b0
[<ffffffff815d6b5c>] ? tcp_v4_rcv+0x7ac/0x9a0
[<ffffffff8156f582>] __netif_receive_skb_core+0x582/0x800
[<ffffffff8156f818>] __netif_receive_skb+0x18/0x60
[<ffffffff8156f8a0>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff81570a28>] napi_gro_receive+0xd8/0x130
[<ffffffffa012d2fc>] e1000_clean_rx_irq+0x2ac/0x4f0 [e1000]
[<ffffffffa012dd31>] e1000_clean+0x281/0x8f0 [e1000]
[<ffffffff815700b0>] net_rx_action+0x170/0x380
[<ffffffff8108f0ff>] __do_softirq+0xef/0x280
[<ffffffff8108f2c8>] run_ksoftirqd+0x38/0x50
[<ffffffff810b90af>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f80>] ? lg_double_unlock+0x90/0x90
[<ffffffff810b052f>] kthread+0xcf/0xe0
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81696518>] ret_from_fork+0x58/0x90
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #170010
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc003fee7
Build id: 3.10.0-693.el7.x86_64
0x947
2
dev_hard_start_xmit
vmlinux 0xffffffff81586771
Build id: 3.10.0-693.el7.x86_64
0x171
3
sch_direct_xmit
vmlinux 0xffffffff815af50a
Build id: 3.10.0-693.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff81589566
Build id: 3.10.0-693.el7.x86_64
0x226
5
dev_queue_xmit_sk
vmlinux 0xffffffff815898c3
Build id: 3.10.0-693.el7.x86_64
0x13
6
arp_xmit
vmlinux 0xffffffff815fe063
Build id: 3.10.0-693.el7.x86_64
0x33
7
? arp_create
vmlinux 0xffffffff815fdcc0
Build id: 3.10.0-693.el7.x86_64
0x1c0
8
arp_send_dst.part.15
vmlinux 0xffffffff815fe115
Build id: 3.10.0-693.el7.x86_64
0x45
9
arp_process
vmlinux 0xffffffff815fef5e
Build id: 3.10.0-693.el7.x86_64
0x70e
10
? ip_local_deliver
vmlinux 0xffffffff815ca339
Build id: 3.10.0-693.el7.x86_64
0x59
11
? ip_rcv_finish
vmlinux 0xffffffff815c9fa0
Build id: 3.10.0-693.el7.x86_64
0x350
12
arp_rcv
vmlinux 0xffffffff815ff115
Build id: 3.10.0-693.el7.x86_64
0x125
13
? inet_del_offload
vmlinux 0xffffffff815c9c50
Build id: 3.10.0-693.el7.x86_64
0x40
14
__netif_receive_skb_core
vmlinux 0xffffffff81586f22
Build id: 3.10.0-693.el7.x86_64
0x572
15
? tcp4_gro_receive
vmlinux 0xffffffff815f5944
Build id: 3.10.0-693.el7.x86_64
0x134
16
? inet_gro_receive
vmlinux 0xffffffff81607087
Build id: 3.10.0-693.el7.x86_64
0x247
17
__netif_receive_skb
vmlinux 0xffffffff81587188
Build id: 3.10.0-693.el7.x86_64
0x18
18
netif_receive_skb_internal
vmlinux 0xffffffff81587210
Build id: 3.10.0-693.el7.x86_64
0x40
19
napi_gro_receive
vmlinux 0xffffffff81588318
Build id: 3.10.0-693.el7.x86_64
0xd8
20
e1000_clean_rx_irq
e1000 0xffffffffc004147c
Build id: 3.10.0-693.el7.x86_64
0x2ac
21
e1000_clean
e1000 0xffffffffc0041ed8
Build id: 3.10.0-693.el7.x86_64
0x278
22
net_rx_action
vmlinux 0xffffffff8158799d
Build id: 3.10.0-693.el7.x86_64
0x16d
23
__do_softirq
vmlinux 0xffffffff81090b3f
Build id: 3.10.0-693.el7.x86_64
0xef
24
run_ksoftirqd
vmlinux 0xffffffff81090d08
Build id: 3.10.0-693.el7.x86_64
0x38
25
smpboot_thread_fn
vmlinux 0xffffffff810b909f
Build id: 3.10.0-693.el7.x86_64
0x12f
26
? lg_double_unlock
vmlinux 0xffffffff810b8f70
Build id: 3.10.0-693.el7.x86_64
0x40
27
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
28
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
29
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
30
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [ksoftirqd/1:13]
Modules linked in: xt_comment xt_mark veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel cryptd ac97_bus snd_seq vmw_balloon snd_seq_device snd_pcm joydev pcspkr snd_timer snd soundcore nfit sg vmw_vmci libnvdimm i2c_piix4 parport_pc parport shpchp ip_tables xfs libcrc32c vmwgfx sd_mod sr_mod cdrom crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul ata_generic crct10dif_common crc32c_intel pata_acpi
drm serio_raw mptspi scsi_transport_spi ata_piix mptscsih libata mptbase i2c_core e1000 dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 13 Comm: ksoftirqd/1 Tainted: G             L ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff88007a13af70 ti: ffff88007a154000 task.ti: ffff88007a154000
RIP: 0010:[<ffffffffc003fee7>]  [<ffffffffc003fee7>] e1000_xmit_frame+0x947/0x1040 [e1000]
RSP: 0000:ffff88007a157800  EFLAGS: 00000286
RAX: ffffc90000383818 RBX: ffffffff815cdf31 RCX: 00000000000000b2
RDX: 00000000000000b2 RSI: ffff880035ddc140 RDI: ffff880078b40000
RBP: ffff88007a157880 R08: 0000000000000100 R09: 0000000002000000
R10: 0000000080000000 R11: 0000000000000000 R12: ffff880079ee5800
R13: ffff880056f64b00 R14: 00000000399a4f16 R15: ffffffff815cf810
FS:  0000000000000000(0000) GS:ffff88007b640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9198f65090 CR3: 00000000230a2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff88007a1578b0 ffff880079de3098 000000b1ffff0000 0000000000000000
0000000056f66700 0000000100000000 0000000000000000 0000000100001000
ffff880056f66700 ffff880078b40000 ffff880079de3000 ffff880078b40000
Call Trace:
[<ffffffff81586771>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff815af50a>] sch_direct_xmit+0x11a/0x240
[<ffffffff81589566>] __dev_queue_xmit+0x226/0x550
[<ffffffff815898c3>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff815fe063>] arp_xmit+0x33/0xa0
[<ffffffff815fdcc0>] ? arp_create+0x1c0/0x230
[<ffffffff815fe115>] arp_send_dst.part.15+0x45/0x50
[<ffffffff815fef5e>] arp_process+0x70e/0x780
[<ffffffff815ca339>] ? ip_local_deliver+0x59/0xd0
[<ffffffff815c9fa0>] ? ip_rcv_finish+0x350/0x350
[<ffffffff815ff115>] arp_rcv+0x125/0x1b0
[<ffffffff815c9c50>] ? inet_del_offload+0x40/0x40
[<ffffffff81586f22>] __netif_receive_skb_core+0x572/0x7c0
[<ffffffff815f5944>] ? tcp4_gro_receive+0x134/0x1b0
[<ffffffff81607087>] ? inet_gro_receive+0x247/0x290
[<ffffffff81587188>] __netif_receive_skb+0x18/0x60
[<ffffffff81587210>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff81588318>] napi_gro_receive+0xd8/0x130
[<ffffffffc004147c>] e1000_clean_rx_irq+0x2ac/0x4f0 [e1000]
[<ffffffffc0041ed8>] e1000_clean+0x278/0x8d0 [e1000]
[<ffffffff8158799d>] net_rx_action+0x16d/0x380
[<ffffffff81090b3f>] __do_softirq+0xef/0x280
[<ffffffff81090d08>] run_ksoftirqd+0x38/0x50
[<ffffffff810b909f>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f70>] ? lg_double_unlock+0x40/0x40
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #165948
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffa00b0d57
Build id: 3.10.0-514.16.1.el7.x86_64
0x957
2
dev_hard_start_xmit
vmlinux 0xffffffff8156f911
Build id: 3.10.0-514.16.1.el7.x86_64
0x171
3
sch_direct_xmit
vmlinux 0xffffffff81596f84
Build id: 3.10.0-514.16.1.el7.x86_64
0x104
4
__dev_queue_xmit
vmlinux 0xffffffff8157250c
Build id: 3.10.0-514.16.1.el7.x86_64
0x23c
5
dev_queue_xmit_sk
vmlinux 0xffffffff81572873
Build id: 3.10.0-514.16.1.el7.x86_64
0x13
6
arp_xmit
vmlinux 0xffffffff815e4bf3
Build id: 3.10.0-514.16.1.el7.x86_64
0x33
7
? arp_create
vmlinux 0xffffffff815e4850
Build id: 3.10.0-514.16.1.el7.x86_64
0x1c0
8
arp_send_dst.part.15
vmlinux 0xffffffff815e4ca5
Build id: 3.10.0-514.16.1.el7.x86_64
0x45
9
arp_process
vmlinux 0xffffffff815e5aee
Build id: 3.10.0-514.16.1.el7.x86_64
0x70e
10
? ip_local_deliver
vmlinux 0xffffffff815b1379
Build id: 3.10.0-514.16.1.el7.x86_64
0x59
11
? ip_rcv_finish
vmlinux 0xffffffff815b0fe0
Build id: 3.10.0-514.16.1.el7.x86_64
0x350
12
arp_rcv
vmlinux 0xffffffff815e5ca5
Build id: 3.10.0-514.16.1.el7.x86_64
0x125
13
? inet_del_offload
vmlinux 0xffffffff815b0c90
Build id: 3.10.0-514.16.1.el7.x86_64
0x40
14
__netif_receive_skb_core
vmlinux 0xffffffff815700d2
Build id: 3.10.0-514.16.1.el7.x86_64
0x582
15
? tcp4_gro_receive
vmlinux 0xffffffff815dc694
Build id: 3.10.0-514.16.1.el7.x86_64
0x134
16
__netif_receive_skb
vmlinux 0xffffffff81570368
Build id: 3.10.0-514.16.1.el7.x86_64
0x18
17
netif_receive_skb_internal
vmlinux 0xffffffff815703f0
Build id: 3.10.0-514.16.1.el7.x86_64
0x40
18
napi_gro_receive
vmlinux 0xffffffff81571578
Build id: 3.10.0-514.16.1.el7.x86_64
0xd8
19
e1000_clean_rx_irq
e1000 0xffffffffa00b22fc
Build id: 3.10.0-514.16.1.el7.x86_64
0x2ac
20
e1000_clean
e1000 0xffffffffa00b2d31
Build id: 3.10.0-514.16.1.el7.x86_64
0x281
21
net_rx_action
vmlinux 0xffffffff81570c00
Build id: 3.10.0-514.16.1.el7.x86_64
0x170
22
__do_softirq
vmlinux 0xffffffff8108f63f
Build id: 3.10.0-514.16.1.el7.x86_64
0xef
23
run_ksoftirqd
vmlinux 0xffffffff8108f808
Build id: 3.10.0-514.16.1.el7.x86_64
0x38
24
smpboot_thread_fn
vmlinux 0xffffffff810b95bf
Build id: 3.10.0-514.16.1.el7.x86_64
0x12f
25
? lg_double_unlock
vmlinux 0xffffffff810b9490
Build id: 3.10.0-514.16.1.el7.x86_64
0x90
26
kthread
vmlinux 0xffffffff810b0a4f
Build id: 3.10.0-514.16.1.el7.x86_64
0xcf
27
? kthread_create_on_node
vmlinux 0xffffffff810b0980
Build id: 3.10.0-514.16.1.el7.x86_64
0x140
28
ret_from_fork
vmlinux 0xffffffff816970d8
Build id: 3.10.0-514.16.1.el7.x86_64
0x58
29
? kthread_create_on_node
vmlinux 0xffffffff810b0980
Build id: 3.10.0-514.16.1.el7.x86_64
0x140
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [ksoftirqd/1:13]
Modules linked in: tcp_diag inet_diag fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event intel_powerclamp coretemp snd_ens1371 snd_rawmidi iosf_mbi snd_ac97_codec ac97_bus crc32_pclmul ghash_clmulni_intel snd_seq aesni_intel snd_seq_device snd_pcm lrw snd_timer
gf128mul glue_helper snd ablk_helper ppdev nfit cryptd soundcore i2c_piix4 vmw_balloon pcspkr vmw_vmci libnvdimm sg parport_pc parport shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect ata_piix mptspi sysimgblt fb_sys_fops scsi_transport_spi ttm serio_raw libata mptscsih e1000 drm mptbase i2c_core fjes
CPU: 1 PID: 13 Comm: ksoftirqd/1 Tainted: G             L ------------   3.10.0-514.16.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff88017ca9af10 ti: ffff88017cab0000 task.ti: ffff88017cab0000
RIP: 0010:[<ffffffffa00b0d57>]  [<ffffffffa00b0d57>] e1000_xmit_frame+0x957/0x1040 [e1000]
RSP: 0018:ffff88017cab3808  EFLAGS: 00000286
RAX: ffffc90000cc3818 RBX: 00000001dbe73e02 RCX: 0000000000000073
RDX: 0000000000000073 RSI: ffff880035cd74c0 RDI: ffff88023f918000
RBP: ffff88017cab3888 R08: 0000000000000100 R09: 0000000002000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff8133b2c3
R13: ffff88017cab37d0 R14: 0000000000000001 R15: 0000000000000202
FS:  0000000000000000(0000) GS:ffff880235640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000448c90 CR3: 0000000231a5c000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff88017cab38b8 ffff8802360ed098 0000007200000046 0000000000000000
0000000031329200 ffff880200000000 ffff880200000000 0000000100001000
ffff88022875e300 ffff88023f918000 ffff8802360ed000 ffff88023f918000
Call Trace:
[<ffffffff8156f911>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff81596f84>] sch_direct_xmit+0x104/0x200
[<ffffffff8157250c>] __dev_queue_xmit+0x23c/0x570
[<ffffffff81572873>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff815e4bf3>] arp_xmit+0x33/0xa0
[<ffffffff815e4850>] ? arp_create+0x1c0/0x230
[<ffffffff815e4ca5>] arp_send_dst.part.15+0x45/0x50
[<ffffffff815e5aee>] arp_process+0x70e/0x780
[<ffffffff815b1379>] ? ip_local_deliver+0x59/0xd0
[<ffffffff815b0fe0>] ? ip_rcv_finish+0x350/0x350
[<ffffffff815e5ca5>] arp_rcv+0x125/0x1b0
[<ffffffff815b0c90>] ? inet_del_offload+0x40/0x40
[<ffffffff815700d2>] __netif_receive_skb_core+0x582/0x800
[<ffffffff815dc694>] ? tcp4_gro_receive+0x134/0x1b0
[<ffffffff81570368>] __netif_receive_skb+0x18/0x60
[<ffffffff815703f0>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff81571578>] napi_gro_receive+0xd8/0x130
[<ffffffffa00b22fc>] e1000_clean_rx_irq+0x2ac/0x4f0 [e1000]
[<ffffffffa00b2d31>] e1000_clean+0x281/0x8f0 [e1000]
[<ffffffff81570c00>] net_rx_action+0x170/0x380
[<ffffffff8108f63f>] __do_softirq+0xef/0x280
[<ffffffff8108f808>] run_ksoftirqd+0x38/0x50
[<ffffffff810b95bf>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b9490>] ? lg_double_unlock+0x90/0x90
[<ffffffff810b0a4f>] kthread+0xcf/0xe0
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[<ffffffff816970d8>] ret_from_fork+0x58/0x90
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #155634
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc03fe877
Build id: 3.10.0-1127.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffff9ac3b6f7
Build id: 3.10.0-1127.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffff9ac53386
Build id: 3.10.0-1127.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffff9ac8131a
Build id: 3.10.0-1127.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffff9ac56301
Build id: 3.10.0-1127.el7.x86_64
0x491
6
dev_queue_xmit_sk
vmlinux 0xffffffff9ac564f3
Build id: 3.10.0-1127.el7.x86_64
0x13
7
arp_xmit
vmlinux 0xffffffff9acd368a
Build id: 3.10.0-1127.el7.x86_64
0x3a
8
? arp_create
vmlinux 0xffffffff9acd32c3
Build id: 3.10.0-1127.el7.x86_64
0x1c3
9
arp_send_dst.part.16
vmlinux 0xffffffff9acd3745
Build id: 3.10.0-1127.el7.x86_64
0x45
10
arp_process
vmlinux 0xffffffff9acd4727
Build id: 3.10.0-1127.el7.x86_64
0x877
11
? ip_rcv_finish
vmlinux 0xffffffff9ac9e040
Build id: 3.10.0-1127.el7.x86_64
0x370
12
arp_rcv
vmlinux 0xffffffff9acd48dd
Build id: 3.10.0-1127.el7.x86_64
0x12d
13
? packet_rcv
vmlinux 0xffffffff9ad5e5dc
Build id: 3.10.0-1127.el7.x86_64
0x4c
14
__netif_receive_skb_core
vmlinux 0xffffffff9ac53c19
Build id: 3.10.0-1127.el7.x86_64
0x729
15
? inet_gro_receive
vmlinux 0xffffffff9acdcb89
Build id: 3.10.0-1127.el7.x86_64
0x249
16
__netif_receive_skb
vmlinux 0xffffffff9ac53f18
Build id: 3.10.0-1127.el7.x86_64
0x18
17
netif_receive_skb_internal
vmlinux 0xffffffff9ac53fa0
Build id: 3.10.0-1127.el7.x86_64
0x40
18
napi_gro_receive
vmlinux 0xffffffff9ac54c28
Build id: 3.10.0-1127.el7.x86_64
0xd8
19
e1000_clean_rx_irq
e1000 0xffffffffc03ff45b
Build id: 3.10.0-1127.el7.x86_64
0x2bb
20
e1000_clean
e1000 0xffffffffc0400100
Build id: 3.10.0-1127.el7.x86_64
0x280
21
? update_curr
vmlinux 0xffffffff9a6e4eac
Build id: 3.10.0-1127.el7.x86_64
0x14c
22
net_rx_action
vmlinux 0xffffffff9ac545bf
Build id: 3.10.0-1127.el7.x86_64
0x26f
23
__do_softirq
vmlinux 0xffffffff9a6a5695
Build id: 3.10.0-1127.el7.x86_64
0xf5
24
run_ksoftirqd
vmlinux 0xffffffff9a6a5858
Build id: 3.10.0-1127.el7.x86_64
0x38
25
smpboot_thread_fn
vmlinux 0xffffffff9a6cf0e4
Build id: 3.10.0-1127.el7.x86_64
0x144
26
? lg_double_unlock
vmlinux 0xffffffff9a6cefa0
Build id: 3.10.0-1127.el7.x86_64
0x40
27
kthread
vmlinux 0xffffffff9a6c6691
Build id: 3.10.0-1127.el7.x86_64
0xd1
28
? insert_kthread_work
vmlinux 0xffffffff9a6c65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffff9ad92d37
Build id: 3.10.0-1127.el7.x86_64
0x21
30
? insert_kthread_work
vmlinux 0xffffffff9a6c65c0
Build id: 3.10.0-1127.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [ksoftirqd/2:19]
Modules linked in: binfmt_misc tcp_lp nls_utf8 isofs fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr joydev sg parport_pc parport i2c_piix4
vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper nfit syscopyarea sysfillrect sysimgblt ata_piix fb_sys_fops ttm libnvdimm serio_raw drm libata e1000 mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks
CPU: 2 PID: 19 Comm: ksoftirqd/2 Kdump: loaded Not tainted 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8cd278ead230 ti: ffff8cd278ec8000 task.ti: ffff8cd278ec8000
RIP: 0010:[<ffffffffc03fe877>]  [<ffffffffc03fe877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8cd278ecb7c0  EFLAGS: 00000286
RAX: ffff9d61c0803818 RBX: ffff8cd2773e6a00 RCX: 00000000000000f4
RDX: 00000000000000f4 RSI: ffff8cd2357af140 RDI: ffff8cd277736000
RBP: ffff8cd278ecb840 R08: 0000000000000100 R09: 0000000002000000
R10: 0000000080000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000096 R14: ffff8cd278ecb770 R15: 0000000000000086
FS:  0000000000000000(0000) GS:ffff8cd279480000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd7d879a4a0 CR3: 000000000e59a000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9ac3b6f7>] ? kfree_skbmem+0x37/0x90
[<ffffffff9ac53386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff9ac8131a>] sch_direct_xmit+0x11a/0x250
[<ffffffff9ac56301>] __dev_queue_xmit+0x491/0x650
[<ffffffff9ac564f3>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff9acd368a>] arp_xmit+0x3a/0xb0
[<ffffffff9acd32c3>] ? arp_create+0x1c3/0x230
[<ffffffff9acd3745>] arp_send_dst.part.16+0x45/0x50
[<ffffffff9acd4727>] arp_process+0x877/0x8e0
[<ffffffff9ac9e040>] ? ip_rcv_finish+0x370/0x370
[<ffffffff9acd48dd>] arp_rcv+0x12d/0x1b0
[<ffffffff9ad5e5dc>] ? packet_rcv+0x4c/0x430
[<ffffffff9ac53c19>] __netif_receive_skb_core+0x729/0xa10
[<ffffffff9acdcb89>] ? inet_gro_receive+0x249/0x290
[<ffffffff9ac53f18>] __netif_receive_skb+0x18/0x60
[<ffffffff9ac53fa0>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff9ac54c28>] napi_gro_receive+0xd8/0x100
[<ffffffffc03ff45b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc0400100>] e1000_clean+0x280/0x8e0 [e1000]
[<ffffffff9a6e4eac>] ? update_curr+0x14c/0x1e0
[<ffffffff9ac545bf>] net_rx_action+0x26f/0x390
[<ffffffff9a6a5695>] __do_softirq+0xf5/0x280
[<ffffffff9a6a5858>] run_ksoftirqd+0x38/0x50
[<ffffffff9a6cf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff9a6cefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffff9a6c6691>] kthread+0xd1/0xe0
[<ffffffff9a6c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff9ad92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9a6c65c0>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #86283
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffa00e2d57
Build id: 3.10.0-514.el7.x86_64
0x957
2
dev_hard_start_xmit
vmlinux 0xffffffff8156edc1
Build id: 3.10.0-514.el7.x86_64
0x171
3
sch_direct_xmit
vmlinux 0xffffffff81596424
Build id: 3.10.0-514.el7.x86_64
0x104
4
__dev_queue_xmit
vmlinux 0xffffffff815719bc
Build id: 3.10.0-514.el7.x86_64
0x23c
5
dev_queue_xmit_sk
vmlinux 0xffffffff81571d23
Build id: 3.10.0-514.el7.x86_64
0x13
6
arp_xmit
vmlinux 0xffffffff815e4063
Build id: 3.10.0-514.el7.x86_64
0x33
7
? arp_create
vmlinux 0xffffffff815e3cc0
Build id: 3.10.0-514.el7.x86_64
0x1c0
8
arp_send_dst.part.15
vmlinux 0xffffffff815e4115
Build id: 3.10.0-514.el7.x86_64
0x45
9
arp_process
vmlinux 0xffffffff815e4f5e
Build id: 3.10.0-514.el7.x86_64
0x70e
10
? ip_local_deliver
vmlinux 0xffffffff815b0862
Build id: 3.10.0-514.el7.x86_64
0xb2
11
? ip_rcv_finish
vmlinux 0xffffffff815b0470
Build id: 3.10.0-514.el7.x86_64
0x350
12
arp_rcv
vmlinux 0xffffffff815e5115
Build id: 3.10.0-514.el7.x86_64
0x125
13
? inet_del_offload
vmlinux 0xffffffff815b0120
Build id: 3.10.0-514.el7.x86_64
0x40
14
__netif_receive_skb_core
vmlinux 0xffffffff8156f582
Build id: 3.10.0-514.el7.x86_64
0x582
15
__netif_receive_skb
vmlinux 0xffffffff8156f818
Build id: 3.10.0-514.el7.x86_64
0x18
16
netif_receive_skb_internal
vmlinux 0xffffffff8156f8a0
Build id: 3.10.0-514.el7.x86_64
0x40
17
napi_gro_receive
vmlinux 0xffffffff81570a28
Build id: 3.10.0-514.el7.x86_64
0xd8
18
e1000_clean_rx_irq
e1000 0xffffffffa00e42fc
Build id: 3.10.0-514.el7.x86_64
0x2ac
19
e1000_clean
e1000 0xffffffffa00e4d31
Build id: 3.10.0-514.el7.x86_64
0x281
20
net_rx_action
vmlinux 0xffffffff815700b0
Build id: 3.10.0-514.el7.x86_64
0x170
21
__do_softirq
vmlinux 0xffffffff8108f0ff
Build id: 3.10.0-514.el7.x86_64
0xef
22
run_ksoftirqd
vmlinux 0xffffffff8108f2c8
Build id: 3.10.0-514.el7.x86_64
0x38
23
smpboot_thread_fn
vmlinux 0xffffffff810b90af
Build id: 3.10.0-514.el7.x86_64
0x12f
24
? lg_double_unlock
vmlinux 0xffffffff810b8f80
Build id: 3.10.0-514.el7.x86_64
0x90
25
kthread
vmlinux 0xffffffff810b052f
Build id: 3.10.0-514.el7.x86_64
0xcf
26
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
27
ret_from_fork
vmlinux 0xffffffff81696518
Build id: 3.10.0-514.el7.x86_64
0x58
28
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 28s! [ksoftirqd/4:28]
Modules linked in: binfmt_misc ext4 mbcache jbd2 xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event intel_powerclamp coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus ppdev snd_seq snd_seq_device
pcspkr snd_pcm sg vmw_balloon snd_timer snd soundcore parport_pc parport shpchp i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm mptspi ata_piix scsi_transport_spi e1000 mptscsih libata mptbase i2c_core floppy fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 28 Comm: ksoftirqd/4 Tainted: G             L ------------   3.10.0-514.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
task: ffff880179761f60 ti: ffff880179770000 task.ti: ffff880179770000
RIP: 0010:[<ffffffffa00e2d57>]  [<ffffffffa00e2d57>] e1000_xmit_frame+0x957/0x1040 [e1000]
RSP: 0018:ffff880179773808  EFLAGS: 00000286
RAX: ffffc900060c3818 RBX: 0000000fcba1d802 RCX: 0000000000000023
RDX: 0000000000000023 RSI: ffff880ff8baa600 RDI: ffff880ffcf4e000
RBP: ffff880179773888 R08: 0000000000000100 R09: 0000000002000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff8133aba3
R13: ffff8801797737d0 R14: 0000000000000001 R15: 0000000000000202
FS:  0000000000000000(0000) GS:ffff880ffd700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f31d8778000 CR3: 0000000fd178c000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff8801797738b8 ffff880ffda2c098 0000002279773850 0000000000000000
00000000f4ea4700 ffff880f00000000 ffff880f00000000 0000000100001000
ffff880c6ba6be00 ffff880ffcf4e000 ffff880ffda2c000 ffff880ffcf4e000
Call Trace:
[<ffffffff8156edc1>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff81596424>] sch_direct_xmit+0x104/0x200
[<ffffffff815719bc>] __dev_queue_xmit+0x23c/0x570
[<ffffffff81571d23>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff815e4063>] arp_xmit+0x33/0xa0
[<ffffffff815e3cc0>] ? arp_create+0x1c0/0x230
[<ffffffff815e4115>] arp_send_dst.part.15+0x45/0x50
[<ffffffff815e4f5e>] arp_process+0x70e/0x780
[<ffffffff815b0862>] ? ip_local_deliver+0xb2/0xd0
[<ffffffff815b0470>] ? ip_rcv_finish+0x350/0x350
[<ffffffff815e5115>] arp_rcv+0x125/0x1b0
[<ffffffff815b0120>] ? inet_del_offload+0x40/0x40
[<ffffffff8156f582>] __netif_receive_skb_core+0x582/0x800
[<ffffffff8156f818>] __netif_receive_skb+0x18/0x60
[<ffffffff8156f8a0>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff81570a28>] napi_gro_receive+0xd8/0x130
[<ffffffffa00e42fc>] e1000_clean_rx_irq+0x2ac/0x4f0 [e1000]
[<ffffffffa00e4d31>] e1000_clean+0x281/0x8f0 [e1000]
[<ffffffff815700b0>] net_rx_action+0x170/0x380
[<ffffffff8108f0ff>] __do_softirq+0xef/0x280
[<ffffffff8108f2c8>] run_ksoftirqd+0x38/0x50
[<ffffffff810b90af>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f80>] ? lg_double_unlock+0x90/0x90
[<ffffffff810b052f>] kthread+0xcf/0xe0
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81696518>] ret_from_fork+0x58/0x90
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #56266
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc038f867
Build id: 3.10.0-862.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff9b7eb166
Build id: 3.10.0-862.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff9b816a2a
Build id: 3.10.0-862.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff9b7ee051
Build id: 3.10.0-862.el7.x86_64
0x4a1
5
dev_queue_xmit_sk
vmlinux 0xffffffff9b7ee243
Build id: 3.10.0-862.el7.x86_64
0x13
6
arp_xmit
vmlinux 0xffffffff9b865ae3
Build id: 3.10.0-862.el7.x86_64
0x33
7
? arp_create
vmlinux 0xffffffff9b865743
Build id: 3.10.0-862.el7.x86_64
0x1c3
8
arp_send_dst.part.16
vmlinux 0xffffffff9b865b95
Build id: 3.10.0-862.el7.x86_64
0x45
9
arp_process
vmlinux 0xffffffff9b866b77
Build id: 3.10.0-862.el7.x86_64
0x877
10
? ip_local_deliver_finish
vmlinux 0xffffffff9b831699
Build id: 3.10.0-862.el7.x86_64
0xb9
11
? ip_local_deliver
vmlinux 0xffffffff9b831989
Build id: 3.10.0-862.el7.x86_64
0x59
12
? ip_rcv_finish
vmlinux 0xffffffff9b8315e0
Build id: 3.10.0-862.el7.x86_64
0x370
13
arp_rcv
vmlinux 0xffffffff9b866d25
Build id: 3.10.0-862.el7.x86_64
0x125
14
? inet_del_offload
vmlinux 0xffffffff9b831270
Build id: 3.10.0-862.el7.x86_64
0x40
15
__netif_receive_skb_core
vmlinux 0xffffffff9b7eb9f9
Build id: 3.10.0-862.el7.x86_64
0x729
16
? tcp4_gro_receive
vmlinux 0xffffffff9b85d600
Build id: 3.10.0-862.el7.x86_64
0xe0
17
? inet_gro_receive
vmlinux 0xffffffff9b86edb9
Build id: 3.10.0-862.el7.x86_64
0x249
18
__netif_receive_skb
vmlinux 0xffffffff9b7ebd08
Build id: 3.10.0-862.el7.x86_64
0x18
19
netif_receive_skb_internal
vmlinux 0xffffffff9b7ebd90
Build id: 3.10.0-862.el7.x86_64
0x40
20
napi_gro_receive
vmlinux 0xffffffff9b7ec9b8
Build id: 3.10.0-862.el7.x86_64
0xd8
21
e1000_clean_rx_irq
e1000 0xffffffffc039044b
Build id: 3.10.0-862.el7.x86_64
0x2bb
22
e1000_clean
e1000 0xffffffffc03910eb
Build id: 3.10.0-862.el7.x86_64
0x27b
23
? update_curr
vmlinux 0xffffffff9b2d816c
Build id: 3.10.0-862.el7.x86_64
0x14c
24
? account_entity_dequeue
vmlinux 0xffffffff9b2d4ade
Build id: 3.10.0-862.el7.x86_64
0xae
25
net_rx_action
vmlinux 0xffffffff9b7ec3af
Build id: 3.10.0-862.el7.x86_64
0x26f
26
__do_softirq
vmlinux 0xffffffff9b29a945
Build id: 3.10.0-862.el7.x86_64
0xf5
27
run_ksoftirqd
vmlinux 0xffffffff9b29ab08
Build id: 3.10.0-862.el7.x86_64
0x38
28
smpboot_thread_fn
vmlinux 0xffffffff9b2c35e4
Build id: 3.10.0-862.el7.x86_64
0x144
29
? lg_double_unlock
vmlinux 0xffffffff9b2c34a0
Build id: 3.10.0-862.el7.x86_64
0x40
30
kthread
vmlinux 0xffffffff9b2bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
31
? insert_kthread_work
vmlinux 0xffffffff9b2bad60
Build id: 3.10.0-862.el7.x86_64
0x40
32
ret_from_fork_nospec_begin
vmlinux 0xffffffff9b91f637
Build id: 3.10.0-862.el7.x86_64
0x21
33
? insert_kthread_work
vmlinux 0xffffffff9b2bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ksoftirqd/0:3]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd ppdev joydev pcspkr soundcore sg nfit vmw_balloon libnvdimm parport_pc parport i2c_piix4 shpchp vmw_vmci ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm crc32c_intel ata_piix libata mptspi scsi_transport_spi e1000 serio_raw mptscsih mptbase i2c_core
CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9878b5f19fa0 ti: ffff9878b5f9c000 task.ti: ffff9878b5f9c000
RIP: 0010:[<ffffffffc038f867>]  [<ffffffffc038f867>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9878b5f9f7d0  EFLAGS: 00000286
RAX: ffffa83000f83818 RBX: 0000000000000001 RCX: 00000000000000dd
RDX: 00000000000000dd RSI: ffff987775eadc40 RDI: ffff9878b136c000
RBP: ffff9878b5f9f850 R08: 0000000000000100 R09: 0000000002000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000202
R13: 0000000172545602 R14: 0000000100000246 R15: ffffffff9b573e00
FS:  0000000000000000(0000) GS:ffff9878b8600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff454cdf000 CR3: 00000000a7b06000 CR4: 00000000003607f0
Call Trace:
[<ffffffff9b7eb166>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff9b816a2a>] sch_direct_xmit+0x11a/0x250
[<ffffffff9b7ee051>] __dev_queue_xmit+0x4a1/0x660
[<ffffffff9b7ee243>] dev_queue_xmit_sk+0x13/0x20
[<ffffffff9b865ae3>] arp_xmit+0x33/0xa0
[<ffffffff9b865743>] ? arp_create+0x1c3/0x230
[<ffffffff9b865b95>] arp_send_dst.part.16+0x45/0x50
[<ffffffff9b866b77>] arp_process+0x877/0x8e0
[<ffffffff9b831699>] ? ip_local_deliver_finish+0xb9/0x200
[<ffffffff9b831989>] ? ip_local_deliver+0x59/0xd0
[<ffffffff9b8315e0>] ? ip_rcv_finish+0x370/0x370
[<ffffffff9b866d25>] arp_rcv+0x125/0x1b0
[<ffffffff9b831270>] ? inet_del_offload+0x40/0x40
[<ffffffff9b7eb9f9>] __netif_receive_skb_core+0x729/0xa20
[<ffffffff9b85d600>] ? tcp4_gro_receive+0xe0/0x1a0
[<ffffffff9b86edb9>] ? inet_gro_receive+0x249/0x290
[<ffffffff9b7ebd08>] __netif_receive_skb+0x18/0x60
[<ffffffff9b7ebd90>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff9b7ec9b8>] napi_gro_receive+0xd8/0x100
[<ffffffffc039044b>] e1000_clean_rx_irq+0x2bb/0x510 [e1000]
[<ffffffffc03910eb>] e1000_clean+0x27b/0x8e0 [e1000]
[<ffffffff9b2d816c>] ? update_curr+0x14c/0x1e0
[<ffffffff9b2d4ade>] ? account_entity_dequeue+0xae/0xd0
[<ffffffff9b7ec3af>] net_rx_action+0x26f/0x390
[<ffffffff9b29a945>] __do_softirq+0xf5/0x280
[<ffffffff9b29ab08>] run_ksoftirqd+0x38/0x50
[<ffffffff9b2c35e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff9b2c34a0>] ? lg_double_unlock+0x40/0x40
[<ffffffff9b2bae31>] kthread+0xd1/0xe0
[<ffffffff9b2bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff9b91f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9b2bad60>] ? insert_kthread_work+0x40/0x40
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00