Info

Permalink
Function
vmxnet3_xmit_frame
First occurrence
2020-12-14
Last occurrence
2020-12-14
State
NEW
Components
kernel
Quality
-87

Statistics

Operating system Count
CentOS 7 21
Architecture Count
x86_64 21
Related packages Count
kernel 21
 0:3.10.0-1062.9.1.el7 8
 0:3.10.0-1160.el7 8
 0:3.10.0-1127.13.1.el7 2
 0:3.10.0-693.2.2.el7 1
 0:3.10.0-1160.119.1.el7 1
 0:3.10.0-1127.18.2.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #346978
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc01773dc
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xaec
2
dev_hard_start_xmit
vmlinux 0xffffffff8dc53386
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffff8dc8131a
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff8dc56301
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x491
5
dev_queue_xmit
vmlinux 0xffffffff8dc564d0
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x10
6
ip_finish_output
vmlinux 0xffffffff8dca429b
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x67b
7
ip_output
vmlinux 0xffffffff8dca481b
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x7b
8
? __wake_up_common
vmlinux 0xffffffff8d6d3c52
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x82
9
? __slab_free
vmlinux 0xffffffff8d825fe0
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x1b0
10
ip_local_out_sk
vmlinux 0xffffffff8dca2247
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x37
11
ip_queue_xmit
vmlinux 0xffffffff8dca25e4
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x144
12
tcp_transmit_skb
vmlinux 0xffffffff8dcbccd4
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x4e4
13
tcp_send_ack
vmlinux 0xffffffff8dcbeedf
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x11f
14
__tcp_ack_snd_check
vmlinux 0xffffffff8dcb3abb
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x6b
15
tcp_rcv_established
vmlinux 0xffffffff8dcb95e4
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x264
16
tcp_v4_do_rcv
vmlinux 0xffffffff8dcc46ca
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x10a
17
tcp_v4_rcv
vmlinux 0xffffffff8dcc5ddc
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x7dc
18
? __rmqueue
vmlinux 0xffffffff8d7c5efa
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x8a
19
? zone_statistics
vmlinux 0xffffffff8d7e05c8
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x88
20
ip_local_deliver_finish
vmlinux 0xffffffff8dc9e0fd
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xbd
21
ip_local_deliver
vmlinux 0xffffffff8dc9e3f0
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x60
22
? tcp_v4_early_demux
vmlinux 0xffffffff8dcc55b2
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x112
23
ip_rcv_finish
vmlinux 0xffffffff8dc9dd60
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x90
24
ip_rcv
vmlinux 0xffffffff8dc9e730
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x2c0
25
__netif_receive_skb_core
vmlinux 0xffffffff8dc53c19
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x729
26
__netif_receive_skb
vmlinux 0xffffffff8dc53f18
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x18
27
netif_receive_skb_internal
vmlinux 0xffffffff8dc53fa0
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x40
28
netif_receive_skb
vmlinux 0xffffffff8dc5403c
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x1c
29
vmxnet3_rq_rx_complete
vmxnet3 0xffffffffc0178639
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x8a9
30
vmxnet3_poll_rx_only
vmxnet3 0xffffffffc01790e6
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x36
31
net_rx_action
vmlinux 0xffffffff8dc545bf
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x26f
32
__do_softirq
vmlinux 0xffffffff8d6a5695
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xf5
33
run_ksoftirqd
vmlinux 0xffffffff8d6a5858
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x38
34
smpboot_thread_fn
vmlinux 0xffffffff8d6cf0e4
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x144
35
? lg_double_unlock
vmlinux 0xffffffff8d6cefa0
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x40
36
kthread
vmlinux 0xffffffff8d6c6691
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xd1
37
? insert_kthread_work
vmlinux 0xffffffff8d6c65c0
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x40
38
ret_from_fork_nospec_begin
vmlinux 0xffffffff8dd92d37
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x21
39
? insert_kthread_work
vmlinux 0xffffffff8d6c65c0
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 41s! [ksoftirqd/7:44]
Modules linked in: udp_diag tcp_diag inet_diag binfmt_misc vmw_vsock_vmci_transport vsock sunrpc vfat fat sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd sg vmw_balloon pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul mptspi crct10dif_common scsi_transport_spi crc32c_intel mptscsih nfit serio_raw mptbase libnvdimm vmxnet3 drm_panel_orientation_quirks floppy
CPU: 7 PID: 44 Comm: ksoftirqd/7 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1127.18.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/19/2018
task: ffff8baff9d01070 ti: ffff8baff9d0c000 task.ti: ffff8baff9d0c000
RIP: 0010:[<ffffffffc01773dc>]  [<ffffffffc01773dc>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0000:ffff8baff9d0f5e0  EFLAGS: 00000282
RAX: 0000000000000000 RBX: 0101000006ff9202 RCX: 000000000000011e
RDX: ffffaaa4418b0638 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff8baff9d0f670 R08: 65770a0501013a65 R09: 0000000000c8404e
R10: 0101000012ff9205 R11: ffff8bb2addf31d0 R12: ffff8baff9d0f5f0
R13: ffff8baff9d0f5d0 R14: 0000000000000283 R15: ffff8bb291facc00
FS:  0000000000000000(0000) GS:ffff8bb2bfdc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f59b39f9018 CR3: 00000000ba926000 CR4: 00000000000407e0
Call Trace:
[<ffffffff8dc53386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff8dc8131a>] sch_direct_xmit+0x11a/0x250
[<ffffffff8dc56301>] __dev_queue_xmit+0x491/0x650
[<ffffffff8dc564d0>] dev_queue_xmit+0x10/0x20
[<ffffffff8dca429b>] ip_finish_output+0x67b/0x8d0
[<ffffffff8dca481b>] ip_output+0x7b/0xf0
[<ffffffff8d6d3c52>] ? __wake_up_common+0x82/0x120
[<ffffffff8d825fe0>] ? __slab_free+0x1b0/0x290
[<ffffffff8dca2247>] ip_local_out_sk+0x37/0x40
[<ffffffff8dca25e4>] ip_queue_xmit+0x144/0x3c0
[<ffffffff8dcbccd4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff8dcbeedf>] tcp_send_ack+0x11f/0x170
[<ffffffff8dcb3abb>] __tcp_ack_snd_check+0x6b/0x90
[<ffffffff8dcb95e4>] tcp_rcv_established+0x264/0x770
[<ffffffff8dcc46ca>] tcp_v4_do_rcv+0x10a/0x350
[<ffffffff8dcc5ddc>] tcp_v4_rcv+0x7dc/0x9e0
[<ffffffff8d7c5efa>] ? __rmqueue+0x8a/0x460
[<ffffffff8d7e05c8>] ? zone_statistics+0x88/0xa0
[<ffffffff8dc9e0fd>] ip_local_deliver_finish+0xbd/0x200
[<ffffffff8dc9e3f0>] ip_local_deliver+0x60/0xe0
[<ffffffff8dcc55b2>] ? tcp_v4_early_demux+0x112/0x160
[<ffffffff8dc9dd60>] ip_rcv_finish+0x90/0x370
[<ffffffff8dc9e730>] ip_rcv+0x2c0/0x420
[<ffffffff8dc53c19>] __netif_receive_skb_core+0x729/0xa10
[<ffffffff8dc53f18>] __netif_receive_skb+0x18/0x60
[<ffffffff8dc53fa0>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff8dc5403c>] netif_receive_skb+0x1c/0x70
[<ffffffffc0178639>] vmxnet3_rq_rx_complete+0x8a9/0xef0 [vmxnet3]
[<ffffffffc01790e6>] vmxnet3_poll_rx_only+0x36/0xa0 [vmxnet3]
[<ffffffff8dc545bf>] net_rx_action+0x26f/0x390
[<ffffffff8d6a5695>] __do_softirq+0xf5/0x280
[<ffffffff8d6a5858>] run_ksoftirqd+0x38/0x50
[<ffffffff8d6cf0e4>] smpboot_thread_fn+0x144/0x1a0
[<ffffffff8d6cefa0>] ? lg_double_unlock+0x40/0x40
[<ffffffff8d6c6691>] kthread+0xd1/0xe0
[<ffffffff8d6c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff8dd92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8d6c65c0>] ? insert_kthread_work+0x40/0x40
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41
Complete report #22798
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc02233dc
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xaec
2
? ipt_do_table
ip_tables 0xffffffffc0320719
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x359
3
dev_hard_start_xmit
vmlinux 0xffffffffa9e4f346
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffffa9e7d0ba
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffa9e522b1
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x491
6
dev_queue_xmit
vmlinux 0xffffffffa9e52480
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x10
7
ip_finish_output
vmlinux 0xffffffffa9ea097b
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x67b
8
ip_output
vmlinux 0xffffffffa9ea0efb
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffffa9ea0300
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xa60
10
ip_local_out_sk
vmlinux 0xffffffffa9e9e927
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x37
11
ip_queue_xmit
vmlinux 0xffffffffa9e9ecc4
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x144
12
tcp_transmit_skb
vmlinux 0xffffffffa9eb9374
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x4e4
13
tcp_send_ack
vmlinux 0xffffffffa9ebb56f
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x11f
14
__tcp_ack_snd_check
vmlinux 0xffffffffa9eb016b
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x6b
15
tcp_rcv_established
vmlinux 0xffffffffa9eb5e37
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x417
16
tcp_v4_do_rcv
vmlinux 0xffffffffa9ec0d5a
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x10a
17
tcp_v4_rcv
vmlinux 0xffffffffa9ec246c
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x7dc
18
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc06b72ce
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x4e
19
ip_local_deliver_finish
vmlinux 0xffffffffa9e9a86d
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xbd
20
ip_local_deliver
vmlinux 0xffffffffa9e9ab60
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x60
21
? ip_rcv_finish
vmlinux 0xffffffffa9e9a7b0
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x370
22
ip_rcv_finish
vmlinux 0xffffffffa9e9a4d0
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x90
23
ip_rcv
vmlinux 0xffffffffa9e9aea0
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x2c0
24
? inet_del_offload
vmlinux 0xffffffffa9e9a440
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x40
25
__netif_receive_skb_core
vmlinux 0xffffffffa9e4fbd9
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x729
26
? tcp4_gro_receive
vmlinux 0xffffffffa9ec7600
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x110
27
? __getnstimeofday64
vmlinux 0xffffffffa99076bf
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x3f
28
__netif_receive_skb
vmlinux 0xffffffffa9e4fed8
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x18
29
netif_receive_skb_internal
vmlinux 0xffffffffa9e4ff60
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x40
30
napi_gro_receive
vmlinux 0xffffffffa9e50be8
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xd8
31
vmxnet3_rq_rx_complete
vmxnet3 0xffffffffc02246b4
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x924
32
vmxnet3_poll_rx_only
vmxnet3 0xffffffffc02250e6
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x36
33
net_rx_action
vmlinux 0xffffffffa9e5057f
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x26f
34
__do_softirq
vmlinux 0xffffffffa98a5305
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xf5
35
run_ksoftirqd
vmlinux 0xffffffffa98a54c8
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x38
36
smpboot_thread_fn
vmlinux 0xffffffffa98cec24
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x144
37
? lg_double_unlock
vmlinux 0xffffffffa98ceae0
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x40
38
kthread
vmlinux 0xffffffffa98c61f1
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xd1
39
? insert_kthread_work
vmlinux 0xffffffffa98c6120
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x40
40
ret_from_fork_nospec_begin
vmlinux 0xffffffffa9f8dd37
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x21
41
? insert_kthread_work
vmlinux 0xffffffffa98c6120
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [ksoftirqd/0:6]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter tcp_diag inet_diag vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul mptspi crct10dif_common scsi_transport_spi crc32c_intel mptscsih
libata serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6 Comm: ksoftirqd/0 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/30/2013
task: ffff8da6274eb150 ti: ffff8da627500000 task.ti: ffff8da627500000
RIP: 0010:[<ffffffffc02233dc>]  [<ffffffffc02233dc>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0018:ffff8da6275035e0  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffff8da627503638 RCX: 0000000000000197
RDX: ffffb7bb43180608 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff8da627503670 R08: 0000000000000050 R09: 0000000000c84042
R10: 3400004500082443 R11: ffff8da61b037960 R12: ffff8da627503628
R13: ffff8da67a146d90 R14: ffffffffa9a24a99 R15: ffffffffc069cf8d
FS:  0000000000000000(0000) GS:ffff8da628800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f41d10e03b8 CR3: 0000000305610000 CR4: 00000000000407f0
Call Trace:
[<ffffffffc0320719>] ? ipt_do_table+0x359/0x740 [ip_tables]
[<ffffffffa9e4f346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa9e7d0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffffa9e522b1>] __dev_queue_xmit+0x491/0x650
[<ffffffffa9e52480>] dev_queue_xmit+0x10/0x20
[<ffffffffa9ea097b>] ip_finish_output+0x67b/0x8d0
[<ffffffffa9ea0efb>] ip_output+0x7b/0xf0
[<ffffffffa9ea0300>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffa9e9e927>] ip_local_out_sk+0x37/0x40
[<ffffffffa9e9ecc4>] ip_queue_xmit+0x144/0x3c0
[<ffffffffa9eb9374>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffa9ebb56f>] tcp_send_ack+0x11f/0x170
[<ffffffffa9eb016b>] __tcp_ack_snd_check+0x6b/0x90
[<ffffffffa9eb5e37>] tcp_rcv_established+0x417/0x770
[<ffffffffa9ec0d5a>] tcp_v4_do_rcv+0x10a/0x350
[<ffffffffa9ec246c>] tcp_v4_rcv+0x7dc/0x9e0
[<ffffffffc06b72ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffffa9e9a86d>] ip_local_deliver_finish+0xbd/0x200
[<ffffffffa9e9ab60>] ip_local_deliver+0x60/0xe0
[<ffffffffa9e9a7b0>] ? ip_rcv_finish+0x370/0x370
[<ffffffffa9e9a4d0>] ip_rcv_finish+0x90/0x370
[<ffffffffa9e9aea0>] ip_rcv+0x2c0/0x420
[<ffffffffa9e9a440>] ? inet_del_offload+0x40/0x40
[<ffffffffa9e4fbd9>] __netif_receive_skb_core+0x729/0xa10
[<ffffffffa9ec7600>] ? tcp4_gro_receive+0x110/0x1a0
[<ffffffffa99076bf>] ? __getnstimeofday64+0x3f/0xd0
[<ffffffffa9e4fed8>] __netif_receive_skb+0x18/0x60
[<ffffffffa9e4ff60>] netif_receive_skb_internal+0x40/0xc0
[<ffffffffa9e50be8>] napi_gro_receive+0xd8/0x100
[<ffffffffc02246b4>] vmxnet3_rq_rx_complete+0x924/0xef0 [vmxnet3]
[<ffffffffc02250e6>] vmxnet3_poll_rx_only+0x36/0xa0 [vmxnet3]
[<ffffffffa9e5057f>] net_rx_action+0x26f/0x390
[<ffffffffa98a5305>] __do_softirq+0xf5/0x280
[<ffffffffa98a54c8>] run_ksoftirqd+0x38/0x50
[<ffffffffa98cec24>] smpboot_thread_fn+0x144/0x1a0
[<ffffffffa98ceae0>] ? lg_double_unlock+0x40/0x40
[<ffffffffa98c61f1>] kthread+0xd1/0xe0
[<ffffffffa98c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffffa9f8dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa98c6120>] ? insert_kthread_work+0x40/0x40
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41
Complete report #21509
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc00554dd
Build id: 3.10.0-693.2.2.el7.x86_64 		0xb5d
2
? kfree_skbmem
vmlinux 0xffffffff815710d7
Build id: 3.10.0-693.2.2.el7.x86_64 		0x37
3
dev_hard_start_xmit
vmlinux 0xffffffff81586771
Build id: 3.10.0-693.2.2.el7.x86_64 		0x171
4
sch_direct_xmit
vmlinux 0xffffffff815af52a
Build id: 3.10.0-693.2.2.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffff81589566
Build id: 3.10.0-693.2.2.el7.x86_64 		0x226
6
dev_queue_xmit
vmlinux 0xffffffff815898a0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x10
7
ip_finish_output
vmlinux 0xffffffff815cfd5e
Build id: 3.10.0-693.2.2.el7.x86_64 		0x52e
8
ip_output
vmlinux 0xffffffff815d02b3
Build id: 3.10.0-693.2.2.el7.x86_64 		0x73
9
? __ip_append_data.isra.48
vmlinux 0xffffffff815cf830
Build id: 3.10.0-693.2.2.el7.x86_64 		0xa00
10
ip_local_out_sk
vmlinux 0xffffffff815cdf51
Build id: 3.10.0-693.2.2.el7.x86_64 		0x31
11
ip_queue_xmit
vmlinux 0xffffffff815ce2c3
Build id: 3.10.0-693.2.2.el7.x86_64 		0x143
12
tcp_transmit_skb
vmlinux 0xffffffff815e7c2f
Build id: 3.10.0-693.2.2.el7.x86_64 		0x4cf
13
tcp_send_ack
vmlinux 0xffffffff815e9c4c
Build id: 3.10.0-693.2.2.el7.x86_64 		0x11c
14
__tcp_ack_snd_check
vmlinux 0xffffffff815defcb
Build id: 3.10.0-693.2.2.el7.x86_64 		0x6b
15
tcp_rcv_established
vmlinux 0xffffffff815e46b4
Build id: 3.10.0-693.2.2.el7.x86_64 		0x264
16
tcp_v4_do_rcv
vmlinux 0xffffffff815ef17a
Build id: 3.10.0-693.2.2.el7.x86_64 		0x10a
17
tcp_v4_rcv
vmlinux 0xffffffff815f08c8
Build id: 3.10.0-693.2.2.el7.x86_64 		0x798
18
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc06b4306
Build id: 3.10.0-693.2.2.el7.x86_64 		0x86
19
ip_local_deliver_finish
vmlinux 0xffffffff815ca074
Build id: 3.10.0-693.2.2.el7.x86_64 		0xb4
20
ip_local_deliver
vmlinux 0xffffffff815ca359
Build id: 3.10.0-693.2.2.el7.x86_64 		0x59
21
? ip_rcv_finish
vmlinux 0xffffffff815c9fc0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x350
22
ip_rcv_finish
vmlinux 0xffffffff815c9cfa
Build id: 3.10.0-693.2.2.el7.x86_64 		0x8a
23
ip_rcv
vmlinux 0xffffffff815ca686
Build id: 3.10.0-693.2.2.el7.x86_64 		0x2b6
24
? inet_del_offload
vmlinux 0xffffffff815c9c70
Build id: 3.10.0-693.2.2.el7.x86_64 		0x40
25
__netif_receive_skb_core
vmlinux 0xffffffff81586f22
Build id: 3.10.0-693.2.2.el7.x86_64 		0x572
26
? __getnstimeofday64
vmlinux 0xffffffff810ea8ca
Build id: 3.10.0-693.2.2.el7.x86_64 		0x3a
27
__netif_receive_skb
vmlinux 0xffffffff81587188
Build id: 3.10.0-693.2.2.el7.x86_64 		0x18
28
netif_receive_skb_internal
vmlinux 0xffffffff81587210
Build id: 3.10.0-693.2.2.el7.x86_64 		0x40
29
napi_gro_receive
vmlinux 0xffffffff81588318
Build id: 3.10.0-693.2.2.el7.x86_64 		0xd8
30
vmxnet3_rq_rx_complete
vmxnet3 0xffffffffc0053fac
Build id: 3.10.0-693.2.2.el7.x86_64 		0x82c
31
vmxnet3_poll_rx_only
vmxnet3 0xffffffffc005450a
Build id: 3.10.0-693.2.2.el7.x86_64 		0x3a
32
net_rx_action
vmlinux 0xffffffff8158799d
Build id: 3.10.0-693.2.2.el7.x86_64 		0x16d
33
__do_softirq
vmlinux 0xffffffff81090b3f
Build id: 3.10.0-693.2.2.el7.x86_64 		0xef
34
run_ksoftirqd
vmlinux 0xffffffff81090d08
Build id: 3.10.0-693.2.2.el7.x86_64 		0x38
35
smpboot_thread_fn
vmlinux 0xffffffff810b909f
Build id: 3.10.0-693.2.2.el7.x86_64 		0x12f
36
? lg_double_unlock
vmlinux 0xffffffff810b8f70
Build id: 3.10.0-693.2.2.el7.x86_64 		0x40
37
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.2.2.el7.x86_64 		0xcf
38
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x40
39
ret_from_fork
vmlinux 0xffffffff816b4f58
Build id: 3.10.0-693.2.2.el7.x86_64 		0x58
40
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [ksoftirqd/0:3]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun lp ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus ppdev snd_seq snd_seq_device pcspkr snd_pcm snd_timer
vmw_balloon snd soundcore joydev nfit sg libnvdimm parport_pc parport i2c_piix4 shpchp vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea sysfillrect sysimgblt crc32c_intel fb_sys_fops ttm drm serio_raw ata_piix mptspi scsi_transport_spi mptscsih libata mptbase vmxnet3 i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3 Comm: ksoftirqd/0 Tainted: G           OE  ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff88017a251fa0 ti: ffff88017a360000 task.ti: ffff88017a360000
RIP: 0010:[<ffffffffc00554dd>]  [<ffffffffc00554dd>] vmxnet3_xmit_frame+0xb5d/0xed0 [vmxnet3]
RSP: 0018:ffff88017a363640  EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffffffff81e87180 RCX: 0000000000000095
RDX: ffffc900018b4600 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff88017a3636d0 R08: 0c00bc92fc565000 R09: 0000000000c84036
R10: ffff88041a76c940 R11: 0040298528001045 R12: ffffffff81e87180
R13: 0000000000000000 R14: ffffffffc068374d R15: ffff88017a363670
FS:  0000000000000000(0000) GS:ffff88042d600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fac7d76b2e6 CR3: 0000000420efb000 CR4: 00000000003407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff880428b20000 ffff88017a363658 ffffffff815710d7 ffff88017a363678
ffff88041a76c940 0000000000000283 ffff880428b208e0 000088017a3636d0
0000002200000014 ffff8800a74bc800 0000000000000036 ffff880400000002
Call Trace:
[<ffffffff815710d7>] ? kfree_skbmem+0x37/0x90
[<ffffffff81586771>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff815af52a>] sch_direct_xmit+0x11a/0x240
[<ffffffff81589566>] __dev_queue_xmit+0x226/0x550
[<ffffffff815898a0>] dev_queue_xmit+0x10/0x20
[<ffffffff815cfd5e>] ip_finish_output+0x52e/0x780
[<ffffffff815d02b3>] ip_output+0x73/0xe0
[<ffffffff815cf830>] ? __ip_append_data.isra.48+0xa00/0xa00
[<ffffffff815cdf51>] ip_local_out_sk+0x31/0x40
[<ffffffff815ce2c3>] ip_queue_xmit+0x143/0x3a0
[<ffffffff815e7c2f>] tcp_transmit_skb+0x4cf/0x9b0
[<ffffffff815e9c4c>] tcp_send_ack+0x11c/0x170
[<ffffffff815defcb>] __tcp_ack_snd_check+0x6b/0xa0
[<ffffffff815e46b4>] tcp_rcv_established+0x264/0x760
[<ffffffff815ef17a>] tcp_v4_do_rcv+0x10a/0x340
[<ffffffff815f08c8>] tcp_v4_rcv+0x798/0x9a0
[<ffffffffc06b4306>] ? ipv4_confirm+0x86/0x100 [nf_conntrack_ipv4]
[<ffffffff815ca074>] ip_local_deliver_finish+0xb4/0x1f0
[<ffffffff815ca359>] ip_local_deliver+0x59/0xd0
[<ffffffff815c9fc0>] ? ip_rcv_finish+0x350/0x350
[<ffffffff815c9cfa>] ip_rcv_finish+0x8a/0x350
[<ffffffff815ca686>] ip_rcv+0x2b6/0x410
[<ffffffff815c9c70>] ? inet_del_offload+0x40/0x40
[<ffffffff81586f22>] __netif_receive_skb_core+0x572/0x7c0
[<ffffffff810ea8ca>] ? __getnstimeofday64+0x3a/0xd0
[<ffffffff81587188>] __netif_receive_skb+0x18/0x60
[<ffffffff81587210>] netif_receive_skb_internal+0x40/0xc0
[<ffffffff81588318>] napi_gro_receive+0xd8/0x130
[<ffffffffc0053fac>] vmxnet3_rq_rx_complete+0x82c/0xc20 [vmxnet3]
[<ffffffffc005450a>] vmxnet3_poll_rx_only+0x3a/0xb0 [vmxnet3]
[<ffffffff8158799d>] net_rx_action+0x16d/0x380
[<ffffffff81090b3f>] __do_softirq+0xef/0x280
[<ffffffff81090d08>] run_ksoftirqd+0x38/0x50
[<ffffffff810b909f>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f70>] ? lg_double_unlock+0x40/0x40
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f58>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: f7 ff ff c7 02 00 00 00 00 41 8b 95 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 21 00 00 41 8b 4d 34 89 0a <e9> 6b f7 ff ff 48 8b 5d 80 e9 9a fc ff ff 0f b7 b3 c0 00 00 00