Info

Permalink
Function
e1000_xmit_frame
First occurrence
2020-09-30
Last occurrence
2026-05-22
State
NEW
Components
kernel
Quality
-47

Statistics

Operating system Count
CentOS 7 68
Architecture Count
x86_64 68
Related packages Count
kernel 68
 0:3.10.0-693.el7 33
 0:3.10.0-1160.el7 5
 0:3.10.0-1160.119.1.el7 4
 0:3.10.0-1127.13.1.el7 4
 0:3.10.0-862.el7 2
 0:3.10.0-1062.el7 2
 0:3.10.0-862.14.4.el7 2
 0:3.10.0-957.5.1.el7 2
 0:3.10.0-957.el7 2
 0:3.10.0-1160.99.1.el7 2
 0:3.10.0-1062.9.1.el7 1
 0:3.10.0-1127.18.2.el7 1
 0:3.10.0-1160.118.1.el7 1
 0:3.10.0-1160.76.1.el7 1
 0:3.10.0-1160.81.1.el7 1
 0:3.10.0-1160.6.1.el7 1
 0:3.10.0-1160.45.1.el7 1
 0:3.10.0-1160.71.1.el7 1
 0:3.10.0-1160.83.1.el7 1
 0:3.10.0-1160.59.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1335154
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc016e877
Build id: 3.10.0-1062.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffab44d346
Build id: 3.10.0-1062.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffab47b0ba
Build id: 3.10.0-1062.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffab4502b1
Build id: 3.10.0-1062.el7.x86_64
0x491
5
? local_bh_enable
vmlinux 0xffffffffaaea3667
Build id: 3.10.0-1062.el7.x86_64
0x17
6
dev_queue_xmit
vmlinux 0xffffffffab450480
Build id: 3.10.0-1062.el7.x86_64
0x10
7
neigh_resolve_output
vmlinux 0xffffffffab45cfed
Build id: 3.10.0-1062.el7.x86_64
0x11d
8
ip_finish_output
vmlinux 0xffffffffab49e57c
Build id: 3.10.0-1062.el7.x86_64
0x27c
9
ip_output
vmlinux 0xffffffffab49eefb
Build id: 3.10.0-1062.el7.x86_64
0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffffab49e300
Build id: 3.10.0-1062.el7.x86_64
0xa60
11
ip_local_out_sk
vmlinux 0xffffffffab49c927
Build id: 3.10.0-1062.el7.x86_64
0x37
12
ip_queue_xmit
vmlinux 0xffffffffab49ccc4
Build id: 3.10.0-1062.el7.x86_64
0x144
13
tcp_transmit_skb
vmlinux 0xffffffffab4b7304
Build id: 3.10.0-1062.el7.x86_64
0x4e4
14
tcp_write_xmit
vmlinux 0xffffffffab4b798a
Build id: 3.10.0-1062.el7.x86_64
0x18a
15
__tcp_push_pending_frames
vmlinux 0xffffffffab4b87be
Build id: 3.10.0-1062.el7.x86_64
0x2e
16
tcp_send_fin
vmlinux 0xffffffffab4ba442
Build id: 3.10.0-1062.el7.x86_64
0x62
17
tcp_close
vmlinux 0xffffffffab4ab3a0
Build id: 3.10.0-1062.el7.x86_64
0x3a0
18
inet_release
vmlinux 0xffffffffab4d52ed
Build id: 3.10.0-1062.el7.x86_64
0x7d
19
sock_release
vmlinux 0xffffffffab42c765
Build id: 3.10.0-1062.el7.x86_64
0x25
20
sock_close
vmlinux 0xffffffffab42c7e2
Build id: 3.10.0-1062.el7.x86_64
0x12
21
__fput
vmlinux 0xffffffffab04a9cc
Build id: 3.10.0-1062.el7.x86_64
0xec
22
____fput
vmlinux 0xffffffffab04ac2e
Build id: 3.10.0-1062.el7.x86_64
0xe
23
task_work_run
vmlinux 0xffffffffaaec1c0b
Build id: 3.10.0-1062.el7.x86_64
0xbb
24
do_notify_resume
vmlinux 0xffffffffaae2cc65
Build id: 3.10.0-1062.el7.x86_64
0xa5
25
int_signal
vmlinux 0xffffffffab58c23b
Build id: 3.10.0-1062.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 37s! [redis-server:1377]
Modules linked in: tun bridge stp llc devlink ip_set nfnetlink bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi ppdev snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus btusb snd_seq joydev btrtl pcspkr btbcm snd_seq_device btintel bluetooth snd_pcm rfkill snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom vmwgfx ata_generic drm_kms_helper pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi drm scsi_transport_spi crct10dif_pclmul crct10dif_common ata_piix mptscsih crc32c_intel nfit serio_raw libata libnvdimm e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash
dm_log dm_mod [last unloaded: ip_tables]
CPU: 1 PID: 1377 Comm: redis-server Kdump: loaded Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff981833240000 ti: ffff98182d214000 task.ti: ffff98182d214000
RIP: 0010:[<ffffffffc016e877>]  [<ffffffffc016e877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff98182d2179c0  EFLAGS: 00000286
RAX: ffffb926c0983818 RBX: ffff98182d2179b8 RCX: 00000000000000f6
RDX: 00000000000000f6 RSI: ffff9818b1ba4100 RDI: ffff9818b57d0000
RBP: ffff98182d217a40 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: 0000000000000042
R13: ffff9818b536ac00 R14: ffff9818b1abcc00 R15: ffff9818b536ac00
FS:  00007f8b835e7f80(0000) GS:ffff9818b9640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8b835ff000 CR3: 00000000ad212000 CR4: 00000000003607e0
Call Trace:
[<ffffffffab44d346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffab47b0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffffab4502b1>] __dev_queue_xmit+0x491/0x650
[<ffffffffaaea3667>] ? local_bh_enable+0x17/0x20
[<ffffffffab450480>] dev_queue_xmit+0x10/0x20
[<ffffffffab45cfed>] neigh_resolve_output+0x11d/0x220
[<ffffffffab49e57c>] ip_finish_output+0x27c/0x8d0
[<ffffffffab49eefb>] ip_output+0x7b/0xf0
[<ffffffffab49e300>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffab49c927>] ip_local_out_sk+0x37/0x40
[<ffffffffab49ccc4>] ip_queue_xmit+0x144/0x3c0
[<ffffffffab4b7304>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffab4b798a>] tcp_write_xmit+0x18a/0xd40
[<ffffffffab4b87be>] __tcp_push_pending_frames+0x2e/0xc0
[<ffffffffab4ba442>] tcp_send_fin+0x62/0x190
[<ffffffffab4ab3a0>] tcp_close+0x3a0/0x400
[<ffffffffab4d52ed>] inet_release+0x7d/0x90
[<ffffffffab42c765>] sock_release+0x25/0x90
[<ffffffffab42c7e2>] sock_close+0x12/0x20
[<ffffffffab04a9cc>] __fput+0xec/0x260
[<ffffffffab04ac2e>] ____fput+0xe/0x10
[<ffffffffaaec1c0b>] task_work_run+0xbb/0xe0
[<ffffffffaae2cc65>] do_notify_resume+0xa5/0xc0
[<ffffffffab58c23b>] int_signal+0x12/0x17
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #819090
Frame # Function Binary Source or offset Line
1
? kfree_skbmem
vmlinux 0xffffffffa4466c7f
Build id: 3.10.0-1160.99.1.el7.x86_64
0x3f
2
dev_hard_start_xmit
vmlinux 0xffffffffa447edc6
Build id: 3.10.0-1160.99.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffa44add72
Build id: 3.10.0-1160.99.1.el7.x86_64
0x122
4
__dev_queue_xmit
vmlinux 0xffffffffa4481dd1
Build id: 3.10.0-1160.99.1.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffffa4481fa0
Build id: 3.10.0-1160.99.1.el7.x86_64
0x10
6
neigh_resolve_output
vmlinux 0xffffffffa448f03d
Build id: 3.10.0-1160.99.1.el7.x86_64
0x11d
7
ip_finish_output
vmlinux 0xffffffffa44d220c
Build id: 3.10.0-1160.99.1.el7.x86_64
0x27c
8
ip_output
vmlinux 0xffffffffa44d2b9b
Build id: 3.10.0-1160.99.1.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffffa44d1f90
Build id: 3.10.0-1160.99.1.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffffa44d0587
Build id: 3.10.0-1160.99.1.el7.x86_64
0x37
11
ip_queue_xmit
vmlinux 0xffffffffa44d0924
Build id: 3.10.0-1160.99.1.el7.x86_64
0x144
12
tcp_transmit_skb
vmlinux 0xffffffffa44eb5f4
Build id: 3.10.0-1160.99.1.el7.x86_64
0x4e4
13
tcp_write_xmit
vmlinux 0xffffffffa44ebc7a
Build id: 3.10.0-1160.99.1.el7.x86_64
0x18a
14
__tcp_push_pending_frames
vmlinux 0xffffffffa44ecace
Build id: 3.10.0-1160.99.1.el7.x86_64
0x2e
15
tcp_send_fin
vmlinux 0xffffffffa44ee7b2
Build id: 3.10.0-1160.99.1.el7.x86_64
0x62
16
tcp_close
vmlinux 0xffffffffa44df300
Build id: 3.10.0-1160.99.1.el7.x86_64
0x3a0
17
inet_release
vmlinux 0xffffffffa4509dfd
Build id: 3.10.0-1160.99.1.el7.x86_64
0x7d
18
sock_release
vmlinux 0xffffffffa445d865
Build id: 3.10.0-1160.99.1.el7.x86_64
0x25
19
sock_close
vmlinux 0xffffffffa445d8e2
Build id: 3.10.0-1160.99.1.el7.x86_64
0x12
20
__fput
vmlinux 0xffffffffa405df7c
Build id: 3.10.0-1160.99.1.el7.x86_64
0xec
21
____fput
vmlinux 0xffffffffa405e1ae
Build id: 3.10.0-1160.99.1.el7.x86_64
0xe
22
task_work_run
vmlinux 0xffffffffa3ec7f6b
Build id: 3.10.0-1160.99.1.el7.x86_64
0xbb
23
do_notify_resume
vmlinux 0xffffffffa3e2dcfd
Build id: 3.10.0-1160.99.1.el7.x86_64
0xad
24
int_signal
vmlinux 0xffffffffa45c56f7
Build id: 3.10.0-1160.99.1.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [kswapd0:35]
RSP: 0018:ffff8f8e338039c0  EFLAGS: 00010286
RAX: ffffa14940d03818 RBX: 0000000000000001 RCX: 00000000000000ff
RDX: 00000000000000ff RSI: ffff8f8e304ac740 RDI: ffff8f8e344c2000
RBP: ffff8f8e33803a40 R08: 0000000000000100 R09: 0000000022100000
R10: 101f280000450008 R11: 0000000000000000 R12: 0000000000000202
R13: 00000001cfa91cda R14: 0000000100000000 R15: ffffffffa41c9110
FS:  00007f95e589ca80(0000) GS:ffff8f8e35640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3539e1d000 CR3: 0000000205354000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa4466c7f>] ? kfree_skbmem+0x3f/0xa0
[<ffffffffa447edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa44add72>] sch_direct_xmit+0x122/0x250
[<ffffffffa4481dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffffa4481fa0>] dev_queue_xmit+0x10/0x20
[<ffffffffa448f03d>] neigh_resolve_output+0x11d/0x220
[<ffffffffa44d220c>] ip_finish_output+0x27c/0x8d0
[<ffffffffa44d2b9b>] ip_output+0x7b/0x100
[<ffffffffa44d1f90>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffa44d0587>] ip_local_out_sk+0x37/0x40
[<ffffffffa44d0924>] ip_queue_xmit+0x144/0x3d0
[<ffffffffa44eb5f4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffa44ebc7a>] tcp_write_xmit+0x18a/0xd40
[<ffffffffa44ecace>] __tcp_push_pending_frames+0x2e/0xd0
[<ffffffffa44ee7b2>] tcp_send_fin+0x62/0x190
[<ffffffffa44df300>] tcp_close+0x3a0/0x400
[<ffffffffa4509dfd>] inet_release+0x7d/0x90
[<ffffffffa445d865>] sock_release+0x25/0x90
[<ffffffffa445d8e2>] sock_close+0x12/0x20
[<ffffffffa405df7c>] __fput+0xec/0x230
[<ffffffffa405e1ae>] ____fput+0xe/0x20
[<ffffffffa3ec7f6b>] task_work_run+0xbb/0xe0
[<ffffffffa3e2dcfd>] do_notify_resume+0xad/0xd0
[<ffffffffa45c56f7>] int_signal+0x12/0x17
Complete report #635850
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0399877
Build id: 3.10.0-1160.59.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff86058f96
Build id: 3.10.0-1160.59.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff860870aa
Build id: 3.10.0-1160.59.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff8605bf11
Build id: 3.10.0-1160.59.1.el7.x86_64
0x491
5
? local_bh_enable
vmlinux 0xffffffff85aa4077
Build id: 3.10.0-1160.59.1.el7.x86_64
0x17
6
dev_queue_xmit
vmlinux 0xffffffff8605c0e0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x10
7
neigh_resolve_output
vmlinux 0xffffffff86068e0d
Build id: 3.10.0-1160.59.1.el7.x86_64
0x11d
8
ip_finish_output
vmlinux 0xffffffff860aa86c
Build id: 3.10.0-1160.59.1.el7.x86_64
0x27c
9
ip_output
vmlinux 0xffffffff860ab1eb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffff860aa5f0
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa60
11
ip_local_out_sk
vmlinux 0xffffffff860a8c17
Build id: 3.10.0-1160.59.1.el7.x86_64
0x37
12
ip_queue_xmit
vmlinux 0xffffffff860a8fb4
Build id: 3.10.0-1160.59.1.el7.x86_64
0x144
13
tcp_transmit_skb
vmlinux 0xffffffff860c3734
Build id: 3.10.0-1160.59.1.el7.x86_64
0x4e4
14
tcp_write_xmit
vmlinux 0xffffffff860c3dba
Build id: 3.10.0-1160.59.1.el7.x86_64
0x18a
15
__tcp_push_pending_frames
vmlinux 0xffffffff860c4bfe
Build id: 3.10.0-1160.59.1.el7.x86_64
0x2e
16
tcp_send_fin
vmlinux 0xffffffff860c6892
Build id: 3.10.0-1160.59.1.el7.x86_64
0x62
17
tcp_close
vmlinux 0xffffffff860b76d0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x3a0
18
inet_release
vmlinux 0xffffffff860e17fd
Build id: 3.10.0-1160.59.1.el7.x86_64
0x7d
19
sock_release
vmlinux 0xffffffff86038765
Build id: 3.10.0-1160.59.1.el7.x86_64
0x25
20
sock_close
vmlinux 0xffffffff860387e2
Build id: 3.10.0-1160.59.1.el7.x86_64
0x12
21
__fput
vmlinux 0xffffffff85c5088c
Build id: 3.10.0-1160.59.1.el7.x86_64
0xec
22
____fput
vmlinux 0xffffffff85c50abe
Build id: 3.10.0-1160.59.1.el7.x86_64
0xe
23
task_work_run
vmlinux 0xffffffff85ac299b
Build id: 3.10.0-1160.59.1.el7.x86_64
0xbb
24
do_notify_resume
vmlinux 0xffffffff85a2cc65
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa5
25
int_signal
vmlinux 0xffffffff8619a2ef
Build id: 3.10.0-1160.59.1.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 54s! [yumBackend.py:28107]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc edac_mce_amd crc32_pclmul ghash_clmulni_intel aesni_intel joydev ppdev lrw gf128mul glue_helper ablk_helper snd_intel8x0 cryptd snd_ac97_codec ac97_bus snd_seq pcspkr snd_seq_device snd_pcm sg snd_timer parport_pc vboxguest(OE) i2c_piix4 snd parport video soundcore ip_tables xfs
libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm pata_acpi crct10dif_pclmul crct10dif_common drm ahci serio_raw crc32c_intel drm_panel_orientation_quirks libahci ata_piix e1000 libata dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 28107 Comm: yumBackend.py Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff900bd35e2100 ti: ffff900b5ef1c000 task.ti: ffff900b5ef1c000
RIP: 0010:[<ffffffffc0399877>]  [<ffffffffc0399877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff900b5ef1f9c0  EFLAGS: 00010286
RAX: ffffa4ea80843818 RBX: ffff900af5d9c600 RCX: 000000000000000b
RDX: 000000000000000b RSI: ffff900af5c59cc0 RDI: ffff900bd8bd6000
RBP: ffff900b5ef1fa40 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: ffff900af5d9c608
R13: ffff900ac3513a20 R14: 0000000000000f14 R15: ffff900ac3513ad0
FS:  00007fe56d685740(0000) GS:ffff900bdfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000008c5bb58 CR3: 000000000d822000 CR4: 00000000000006e0
Call Trace:
[<ffffffff86058f96>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff860870aa>] sch_direct_xmit+0x11a/0x250
[<ffffffff8605bf11>] __dev_queue_xmit+0x491/0x650
[<ffffffff85aa4077>] ? local_bh_enable+0x17/0x20
[<ffffffff8605c0e0>] dev_queue_xmit+0x10/0x20
[<ffffffff86068e0d>] neigh_resolve_output+0x11d/0x220
[<ffffffff860aa86c>] ip_finish_output+0x27c/0x8d0
[<ffffffff860ab1eb>] ip_output+0x7b/0xf0
[<ffffffff860aa5f0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff860a8c17>] ip_local_out_sk+0x37/0x40
[<ffffffff860a8fb4>] ip_queue_xmit+0x144/0x3c0
[<ffffffff860c3734>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff860c3dba>] tcp_write_xmit+0x18a/0xd40
[<ffffffff860c4bfe>] __tcp_push_pending_frames+0x2e/0xc0
[<ffffffff860c6892>] tcp_send_fin+0x62/0x190
[<ffffffff860b76d0>] tcp_close+0x3a0/0x400
[<ffffffff860e17fd>] inet_release+0x7d/0x90
[<ffffffff86038765>] sock_release+0x25/0x90
[<ffffffff860387e2>] sock_close+0x12/0x20
[<ffffffff85c5088c>] __fput+0xec/0x230
[<ffffffff85c50abe>] ____fput+0xe/0x10
[<ffffffff85ac299b>] task_work_run+0xbb/0xe0
[<ffffffff85a2cc65>] do_notify_resume+0xa5/0xc0
[<ffffffff8619a2ef>] int_signal+0x12/0x17
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #633125
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc02e744c
Build id: 3.10.0-1160.83.1.el7.x86_64
0xaec
2
? dev_hard_start_xmit
vmlinux 0xffffffffba07edc6
Build id: 3.10.0-1160.83.1.el7.x86_64
0x246
3
dev_hard_start_xmit
vmlinux 0xffffffffba07edc6
Build id: 3.10.0-1160.83.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffffba0add72
Build id: 3.10.0-1160.83.1.el7.x86_64
0x122
5
__dev_queue_xmit
vmlinux 0xffffffffba081dd1
Build id: 3.10.0-1160.83.1.el7.x86_64
0x491
6
dev_queue_xmit
vmlinux 0xffffffffba081fa0
Build id: 3.10.0-1160.83.1.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffffba0d260b
Build id: 3.10.0-1160.83.1.el7.x86_64
0x67b
8
? __enqueue_entity
vmlinux 0xffffffffb9ae5bf8
Build id: 3.10.0-1160.83.1.el7.x86_64
0x78
9
ip_output
vmlinux 0xffffffffba0d2b9b
Build id: 3.10.0-1160.83.1.el7.x86_64
0x7b
10
? update_curr
vmlinux 0xffffffffb9ae8bb4
Build id: 3.10.0-1160.83.1.el7.x86_64
0x164
11
ip_local_out_sk
vmlinux 0xffffffffba0d0587
Build id: 3.10.0-1160.83.1.el7.x86_64
0x37
12
ip_queue_xmit
vmlinux 0xffffffffba0d0924
Build id: 3.10.0-1160.83.1.el7.x86_64
0x144
13
tcp_transmit_skb
vmlinux 0xffffffffba0eb5f4
Build id: 3.10.0-1160.83.1.el7.x86_64
0x4e4
14
tcp_write_xmit
vmlinux 0xffffffffba0ebc7a
Build id: 3.10.0-1160.83.1.el7.x86_64
0x18a
15
__tcp_push_pending_frames
vmlinux 0xffffffffba0ecace
Build id: 3.10.0-1160.83.1.el7.x86_64
0x2e
16
tcp_send_fin
vmlinux 0xffffffffba0ee7b2
Build id: 3.10.0-1160.83.1.el7.x86_64
0x62
17
tcp_close
vmlinux 0xffffffffba0df300
Build id: 3.10.0-1160.83.1.el7.x86_64
0x3a0
18
inet_release
vmlinux 0xffffffffba109dfd
Build id: 3.10.0-1160.83.1.el7.x86_64
0x7d
19
sock_release
vmlinux 0xffffffffba05d865
Build id: 3.10.0-1160.83.1.el7.x86_64
0x25
20
sock_close
vmlinux 0xffffffffba05d8e2
Build id: 3.10.0-1160.83.1.el7.x86_64
0x12
21
__fput
vmlinux 0xffffffffb9c5db8c
Build id: 3.10.0-1160.83.1.el7.x86_64
0xec
22
____fput
vmlinux 0xffffffffb9c5ddbe
Build id: 3.10.0-1160.83.1.el7.x86_64
0xe
23
task_work_run
vmlinux 0xffffffffb9ac7e5b
Build id: 3.10.0-1160.83.1.el7.x86_64
0xbb
24
do_notify_resume
vmlinux 0xffffffffb9a2dcfd
Build id: 3.10.0-1160.83.1.el7.x86_64
0xad
25
int_signal
vmlinux 0xffffffffba1c56f7
Build id: 3.10.0-1160.83.1.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [isc-socket:97223]
Modules linked in: binfmt_misc vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c vmwgfx drm_kms_helper sr_mod cdrom syscopyarea ata_generic pata_acpi sysfillrect sysimgblt fb_sys_fops ttm drm sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common ata_piix crc32c_intel nfit serio_raw libata vmxnet3 libnvdimm vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 97223 Comm: isc-socket Tainted: G             L ------------   3.10.0-1160.83.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffff9a4d360ad280 ti: ffff9a4d382e4000 task.ti: ffff9a4d382e4000
RIP: 0010:[<ffffffffc02e744c>]  [<ffffffffc02e744c>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0018:ffff9a4d382e79f0  EFLAGS: 00000286
RAX: 0000000000000000 RBX: 0000000000000042 RCX: 000000000000004e
RDX: ffffb321801d8600 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff9a4d382e7a80 R08: 359587a935001860 R09: 0000000000c80036
R10: 57f6794004ad3595 R11: ffff9a4d39eaa4d0 R12: ffff9a4d382e7a00
R13: 0000000000000283 R14: 0000000000000283 R15: ffffffffc02e744c
FS:  00007f16e53c1700(0000) GS:ffff9a4d3c600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001fc5ed0 CR3: 0000000035294000 CR4: 00000000001607f0
Call Trace:
[<ffffffffba07edc6>] ? dev_hard_start_xmit+0x246/0x3b0
[<ffffffffba07edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffba0add72>] sch_direct_xmit+0x122/0x250
[<ffffffffba081dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffffba081fa0>] dev_queue_xmit+0x10/0x20
[<ffffffffba0d260b>] ip_finish_output+0x67b/0x8d0
[<ffffffffb9ae5bf8>] ? __enqueue_entity+0x78/0x80
[<ffffffffba0d2b9b>] ip_output+0x7b/0x100
[<ffffffffb9ae8bb4>] ? update_curr+0x164/0x1f0
[<ffffffffba0d0587>] ip_local_out_sk+0x37/0x40
[<ffffffffba0d0924>] ip_queue_xmit+0x144/0x3d0
[<ffffffffba0eb5f4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffba0ebc7a>] tcp_write_xmit+0x18a/0xd40
[<ffffffffba0ecace>] __tcp_push_pending_frames+0x2e/0xd0
[<ffffffffba0ee7b2>] tcp_send_fin+0x62/0x190
[<ffffffffba0df300>] tcp_close+0x3a0/0x400
[<ffffffffba109dfd>] inet_release+0x7d/0x90
[<ffffffffba05d865>] sock_release+0x25/0x90
[<ffffffffba05d8e2>] sock_close+0x12/0x20
[<ffffffffb9c5db8c>] __fput+0xec/0x230
[<ffffffffb9c5ddbe>] ____fput+0xe/0x20
[<ffffffffb9ac7e5b>] task_work_run+0xbb/0xe0
[<ffffffffb9a2dcfd>] do_notify_resume+0xad/0xd0
[<ffffffffba1c56f7>] int_signal+0x12/0x17
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41 
Complete report #399238
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00a5ee7
Build id: 3.10.0-693.el7.x86_64
0x947
2
? ipt_do_table
ip_tables 0xffffffffc00c7d06
Build id: 3.10.0-693.el7.x86_64
0x336
3
dev_hard_start_xmit
vmlinux 0xffffffff81586771
Build id: 3.10.0-693.el7.x86_64
0x171
4
sch_direct_xmit
vmlinux 0xffffffff815af50a
Build id: 3.10.0-693.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffff81589566
Build id: 3.10.0-693.el7.x86_64
0x226
6
dev_queue_xmit
vmlinux 0xffffffff815898a0
Build id: 3.10.0-693.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffff815cfd3e
Build id: 3.10.0-693.el7.x86_64
0x52e
8
ip_output
vmlinux 0xffffffff815d0293
Build id: 3.10.0-693.el7.x86_64
0x73
9
? __ip_append_data.isra.48
vmlinux 0xffffffff815cf810
Build id: 3.10.0-693.el7.x86_64
0xa00
10
ip_local_out_sk
vmlinux 0xffffffff815cdf31
Build id: 3.10.0-693.el7.x86_64
0x31
11
ip_queue_xmit
vmlinux 0xffffffff815ce2a3
Build id: 3.10.0-693.el7.x86_64
0x143
12
tcp_transmit_skb
vmlinux 0xffffffff815e7c0f
Build id: 3.10.0-693.el7.x86_64
0x4cf
13
tcp_write_xmit
vmlinux 0xffffffff815e824a
Build id: 3.10.0-693.el7.x86_64
0x15a
14
__tcp_push_pending_frames
vmlinux 0xffffffff815e904e
Build id: 3.10.0-693.el7.x86_64
0x2e
15
tcp_send_fin
vmlinux 0xffffffff815eaae2
Build id: 3.10.0-693.el7.x86_64
0x62
16
tcp_close
vmlinux 0xffffffff815dc150
Build id: 3.10.0-693.el7.x86_64
0x3a0
17
inet_release
vmlinux 0xffffffff816052b7
Build id: 3.10.0-693.el7.x86_64
0x77
18
sock_release
vmlinux 0xffffffff8156999f
Build id: 3.10.0-693.el7.x86_64
0x1f
19
sock_close
vmlinux 0xffffffff81569a12
Build id: 3.10.0-693.el7.x86_64
0x12
20
__fput
vmlinux 0xffffffff81202fb9
Build id: 3.10.0-693.el7.x86_64
0xe9
21
____fput
vmlinux 0xffffffff8120321e
Build id: 3.10.0-693.el7.x86_64
0xe
22
task_work_run
vmlinux 0xffffffff810ad247
Build id: 3.10.0-693.el7.x86_64
0xa7
23
do_notify_resume
vmlinux 0xffffffff8102ab62
Build id: 3.10.0-693.el7.x86_64
0x92
24
int_signal
vmlinux 0xffffffff816b527d
Build id: 3.10.0-693.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 69s! [mysqld:4445]
Modules linked in: br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr joydev nfit sg libnvdimm parport_pc parport vmw_vmci shpchp i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea crct10dif_pclmul crct10dif_common sysfillrect sysimgblt fb_sys_fops crc32c_intel
ttm drm serio_raw mptspi scsi_transport_spi mptscsih mptbase ata_piix e1000 libata i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 4445 Comm: mysqld Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff880063b58000 ti: ffff880076c1c000 task.ti: ffff880076c1c000
RIP: 0010:[<ffffffffc00a5ee7>]  [<ffffffffc00a5ee7>] e1000_xmit_frame+0x947/0x1040 [e1000]
RSP: 0018:ffff880076c1fa40  EFLAGS: 00000286
RAX: ffffc90000443818 RBX: ffffffff81098a93 RCX: 000000000000001f
RDX: 000000000000001f RSI: ffff880035153140 RDI: ffff880078a90000
RBP: ffff880076c1fac0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: ffff88007a168000
R13: ffff8800597a9710 R14: ffff88007a168000 R15: 00000000e562afe7
FS:  00007f5cdd5bb700(0000) GS:ffff88007b640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f45d8666960 CR3: 00000000769fc000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffffc00c7d06 ffff880079dda098 0000001dc00c7d06 0000000900000000
00000000793d2900 ffff880000000000 ffffffff00000000 0000000100001000
ffff8800768028f8 ffff880078a90000 ffff880079dda000 ffff880078a90000
Call Trace:
[<ffffffffc00c7d06>] ? ipt_do_table+0x336/0x710 [ip_tables]
[<ffffffff81586771>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff815af50a>] sch_direct_xmit+0x11a/0x240
[<ffffffff81589566>] __dev_queue_xmit+0x226/0x550
[<ffffffff815898a0>] dev_queue_xmit+0x10/0x20
[<ffffffff815cfd3e>] ip_finish_output+0x52e/0x780
[<ffffffff815d0293>] ip_output+0x73/0xe0
[<ffffffff815cf810>] ? __ip_append_data.isra.48+0xa00/0xa00
[<ffffffff815cdf31>] ip_local_out_sk+0x31/0x40
[<ffffffff815ce2a3>] ip_queue_xmit+0x143/0x3a0
[<ffffffff815e7c0f>] tcp_transmit_skb+0x4cf/0x9b0
[<ffffffff815e824a>] tcp_write_xmit+0x15a/0xce0
[<ffffffff815e904e>] __tcp_push_pending_frames+0x2e/0xc0
[<ffffffff815eaae2>] tcp_send_fin+0x62/0x190
[<ffffffff815dc150>] tcp_close+0x3a0/0x3f0
[<ffffffff816052b7>] inet_release+0x77/0x80
[<ffffffff8156999f>] sock_release+0x1f/0x80
[<ffffffff81569a12>] sock_close+0x12/0x20
[<ffffffff81202fb9>] __fput+0xe9/0x260
[<ffffffff8120321e>] ____fput+0xe/0x10
[<ffffffff810ad247>] task_work_run+0xa7/0xf0
[<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
[<ffffffff816b527d>] int_signal+0x12/0x17
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #311327
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0350877
Build id: 3.10.0-1160.45.1.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffa203e6f7
Build id: 3.10.0-1160.45.1.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffa2055f96
Build id: 3.10.0-1160.45.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffffa20840aa
Build id: 3.10.0-1160.45.1.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffa2058f11
Build id: 3.10.0-1160.45.1.el7.x86_64
0x491
6
dev_queue_xmit
vmlinux 0xffffffffa20590e0
Build id: 3.10.0-1160.45.1.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffffa20a7c6b
Build id: 3.10.0-1160.45.1.el7.x86_64
0x67b
8
ip_output
vmlinux 0xffffffffa20a81eb
Build id: 3.10.0-1160.45.1.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffffa20a75f0
Build id: 3.10.0-1160.45.1.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffffa20a5c17
Build id: 3.10.0-1160.45.1.el7.x86_64
0x37
11
ip_queue_xmit
vmlinux 0xffffffffa20a5fb4
Build id: 3.10.0-1160.45.1.el7.x86_64
0x144
12
tcp_transmit_skb
vmlinux 0xffffffffa20c06f4
Build id: 3.10.0-1160.45.1.el7.x86_64
0x4e4
13
tcp_write_xmit
vmlinux 0xffffffffa20c0d7a
Build id: 3.10.0-1160.45.1.el7.x86_64
0x18a
14
__tcp_push_pending_frames
vmlinux 0xffffffffa20c1bbe
Build id: 3.10.0-1160.45.1.el7.x86_64
0x2e
15
tcp_send_fin
vmlinux 0xffffffffa20c3852
Build id: 3.10.0-1160.45.1.el7.x86_64
0x62
16
tcp_close
vmlinux 0xffffffffa20b46d0
Build id: 3.10.0-1160.45.1.el7.x86_64
0x3a0
17
inet_release
vmlinux 0xffffffffa20de7bd
Build id: 3.10.0-1160.45.1.el7.x86_64
0x7d
18
sock_release
vmlinux 0xffffffffa2035765
Build id: 3.10.0-1160.45.1.el7.x86_64
0x25
19
sock_close
vmlinux 0xffffffffa20357e2
Build id: 3.10.0-1160.45.1.el7.x86_64
0x12
20
__fput
vmlinux 0xffffffffa1c5088c
Build id: 3.10.0-1160.45.1.el7.x86_64
0xec
21
____fput
vmlinux 0xffffffffa1c50abe
Build id: 3.10.0-1160.45.1.el7.x86_64
0xe
22
task_work_run
vmlinux 0xffffffffa1ac299b
Build id: 3.10.0-1160.45.1.el7.x86_64
0xbb
23
do_notify_resume
vmlinux 0xffffffffa1a2cc65
Build id: 3.10.0-1160.45.1.el7.x86_64
0xa5
24
int_signal
vmlinux 0xffffffffa21962ef
Build id: 3.10.0-1160.45.1.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 642s! [zabbix_server:5170]
Modules linked in: tcp_diag inet_diag veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter binfmt_misc xt_multiport xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter ipmi_devintf ipmi_msghandler overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 aesni_intel snd_rawmidi snd_ac97_codec lrw gf128mul ac97_bus snd_seq ppdev snd_seq_device snd_pcm glue_helper ablk_helper snd_timer snd cryptd soundcore vmw_balloon pcspkr sg joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c
sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic vmwgfx pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix nfit libnvdimm serio_raw libata scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 5170 Comm: zabbix_server Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff90c842062100 ti: ffff90c710060000 task.ti: ffff90c710060000
RIP: 0010:[<ffffffffc0350877>]  [<ffffffffc0350877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff90c710063a00  EFLAGS: 00000286
RAX: ffffa6df80d43818 RBX: 0000000000000001 RCX: 0000000000000017
RDX: 0000000000000017 RSI: ffff90c8f0f53400 RDI: ffff90c8f4666000
RBP: ffff90c710063a80 R08: 0000000000000100 R09: 0000000022100000
R10: 340000450008d94e R11: 0000000000000000 R12: 0000000000000202
R13: 00000001d756c4ce R14: 00000001f0f53400 R15: ffffffffa1db22b0
FS:  00007f4eb4630880(0000) GS:ffff90c8f5600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4eb465e000 CR3: 000000005005e000 CR4: 00000000003607f0
Call Trace:
[<ffffffffa203e6f7>] ? kfree_skbmem+0x37/0x90
[<ffffffffa2055f96>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa20840aa>] sch_direct_xmit+0x11a/0x250
[<ffffffffa2058f11>] __dev_queue_xmit+0x491/0x650
[<ffffffffa20590e0>] dev_queue_xmit+0x10/0x20
[<ffffffffa20a7c6b>] ip_finish_output+0x67b/0x8d0
[<ffffffffa20a81eb>] ip_output+0x7b/0xf0
[<ffffffffa20a75f0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffa20a5c17>] ip_local_out_sk+0x37/0x40
[<ffffffffa20a5fb4>] ip_queue_xmit+0x144/0x3c0
[<ffffffffa20c06f4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffa20c0d7a>] tcp_write_xmit+0x18a/0xd40
[<ffffffffa20c1bbe>] __tcp_push_pending_frames+0x2e/0xc0
[<ffffffffa20c3852>] tcp_send_fin+0x62/0x190
[<ffffffffa20b46d0>] tcp_close+0x3a0/0x400
[<ffffffffa20de7bd>] inet_release+0x7d/0x90
[<ffffffffa2035765>] sock_release+0x25/0x90
[<ffffffffa20357e2>] sock_close+0x12/0x20
[<ffffffffa1c5088c>] __fput+0xec/0x230
[<ffffffffa1c50abe>] ____fput+0xe/0x10
[<ffffffffa1ac299b>] task_work_run+0xbb/0xe0
[<ffffffffa1a2cc65>] do_notify_resume+0xa5/0xc0
[<ffffffffa21962ef>] int_signal+0x12/0x17
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #19357
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc04e3857
Build id: 3.10.0-957.5.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffba838ce6
Build id: 3.10.0-957.5.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffba865d6a
Build id: 3.10.0-957.5.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffba83bc61
Build id: 3.10.0-957.5.1.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffffba83be30
Build id: 3.10.0-957.5.1.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffffba8887b6
Build id: 3.10.0-957.5.1.el7.x86_64
0x556
7
ip_output
vmlinux 0xffffffffba888d13
Build id: 3.10.0-957.5.1.el7.x86_64
0x73
8
? __ip_append_data.isra.51
vmlinux 0xffffffffba888260
Build id: 3.10.0-957.5.1.el7.x86_64
0xa50
9
ip_local_out_sk
vmlinux 0xffffffffba886897
Build id: 3.10.0-957.5.1.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffffba886c34
Build id: 3.10.0-957.5.1.el7.x86_64
0x144
11
tcp_transmit_skb
vmlinux 0xffffffffba8a0fe4
Build id: 3.10.0-957.5.1.el7.x86_64
0x4e4
12
tcp_write_xmit
vmlinux 0xffffffffba8a1652
Build id: 3.10.0-957.5.1.el7.x86_64
0x172
13
__tcp_push_pending_frames
vmlinux 0xffffffffba8a245e
Build id: 3.10.0-957.5.1.el7.x86_64
0x2e
14
tcp_send_fin
vmlinux 0xffffffffba8a3f52
Build id: 3.10.0-957.5.1.el7.x86_64
0x62
15
tcp_close
vmlinux 0xffffffffba895190
Build id: 3.10.0-957.5.1.el7.x86_64
0x3a0
16
inet_release
vmlinux 0xffffffffba8be8ed
Build id: 3.10.0-957.5.1.el7.x86_64
0x7d
17
sock_release
vmlinux 0xffffffffba818765
Build id: 3.10.0-957.5.1.el7.x86_64
0x25
18
sock_close
vmlinux 0xffffffffba8187e2
Build id: 3.10.0-957.5.1.el7.x86_64
0x12
19
__fput
vmlinux 0xffffffffba44398c
Build id: 3.10.0-957.5.1.el7.x86_64
0xec
20
____fput
vmlinux 0xffffffffba443bee
Build id: 3.10.0-957.5.1.el7.x86_64
0xe
21
task_work_run
vmlinux 0xffffffffba2be7db
Build id: 3.10.0-957.5.1.el7.x86_64
0xbb
22
do_notify_resume
vmlinux 0xffffffffba22bc65
Build id: 3.10.0-957.5.1.el7.x86_64
0xa5
23
int_signal
vmlinux 0xffffffffba975124
Build id: 3.10.0-957.5.1.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [nginx:6401]
Modules linked in: fuse xt_CHECKSUM xt_set xt_multiport ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip_set_hash_ip devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc coretemp crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd snd_seq_midi vmw_balloon snd_seq_midi_event pcspkr snd_ens1371 snd_rawmidi snd_ac97_codec joydev ac97_bus
snd_seq snd_seq_device snd_pcm snd_timer snd soundcore sg floppy parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw pata_acpi nfit vmwgfx libnvdimm e1000 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi mptscsih libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6401 Comm: nginx Kdump: loaded Tainted: G             L ------------   3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8b9c63701040 ti: ffff8b9c68390000 task.ti: ffff8b9c68390000
RIP: 0010:[<ffffffffc04e3857>]  [<ffffffffc04e3857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8b9c68393a20  EFLAGS: 00000286
RAX: ffffb7f640943818 RBX: ffffffffba83ba15 RCX: 00000000000000d0
RDX: 00000000000000d0 RSI: ffff8b9bf55ee400 RDI: ffff8b9c7a806000
RBP: ffff8b9c68393aa0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: ffffffffc08b578d
R13: ffff8b9c68393a60 R14: ffff8b9c68fcae00 R15: 0000000000000246
FS:  00007f079d6ad840(0000) GS:ffff8b9c7b400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f32721129a2 CR3: 00000000a3b56000 CR4: 00000000000007f0
Call Trace:
[<ffffffffba838ce6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffba865d6a>] sch_direct_xmit+0x11a/0x250
[<ffffffffba83bc61>] __dev_queue_xmit+0x491/0x650
[<ffffffffba83be30>] dev_queue_xmit+0x10/0x20
[<ffffffffba8887b6>] ip_finish_output+0x556/0x7b0
[<ffffffffba888d13>] ip_output+0x73/0xe0
[<ffffffffba888260>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffba886897>] ip_local_out_sk+0x37/0x40
[<ffffffffba886c34>] ip_queue_xmit+0x144/0x3c0
[<ffffffffba8a0fe4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffba8a1652>] tcp_write_xmit+0x172/0xd00
[<ffffffffba8a245e>] __tcp_push_pending_frames+0x2e/0xc0
[<ffffffffba8a3f52>] tcp_send_fin+0x62/0x190
[<ffffffffba895190>] tcp_close+0x3a0/0x400
[<ffffffffba8be8ed>] inet_release+0x7d/0x90
[<ffffffffba818765>] sock_release+0x25/0x90
[<ffffffffba8187e2>] sock_close+0x12/0x20
[<ffffffffba44398c>] __fput+0xec/0x260
[<ffffffffba443bee>] ____fput+0xe/0x10
[<ffffffffba2be7db>] task_work_run+0xbb/0xe0
[<ffffffffba22bc65>] do_notify_resume+0xa5/0xc0
[<ffffffffba975124>] int_signal+0x12/0x17
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #10625
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0131867
Build id: 3.10.0-862.14.4.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffb4df02e6
Build id: 3.10.0-862.14.4.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffb4e1bc0a
Build id: 3.10.0-862.14.4.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffb4df31d1
Build id: 3.10.0-862.14.4.el7.x86_64
0x4a1
5
dev_queue_xmit
vmlinux 0xffffffffb4df33a0
Build id: 3.10.0-862.14.4.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffffb4e3c5d6
Build id: 3.10.0-862.14.4.el7.x86_64
0x546
7
ip_output
vmlinux 0xffffffffb4e3cb33
Build id: 3.10.0-862.14.4.el7.x86_64
0x73
8
? __ip_append_data.isra.50
vmlinux 0xffffffffb4e3c090
Build id: 3.10.0-862.14.4.el7.x86_64
0xa50
9
ip_local_out_sk
vmlinux 0xffffffffb4e3a727
Build id: 3.10.0-862.14.4.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffffb4e3aa93
Build id: 3.10.0-862.14.4.el7.x86_64
0x143
11
tcp_transmit_skb
vmlinux 0xffffffffb4e54bd4
Build id: 3.10.0-862.14.4.el7.x86_64
0x4e4
12
tcp_write_xmit
vmlinux 0xffffffffb4e55242
Build id: 3.10.0-862.14.4.el7.x86_64
0x172
13
__tcp_push_pending_frames
vmlinux 0xffffffffb4e5604e
Build id: 3.10.0-862.14.4.el7.x86_64
0x2e
14
tcp_send_fin
vmlinux 0xffffffffb4e57b02
Build id: 3.10.0-862.14.4.el7.x86_64
0x62
15
tcp_close
vmlinux 0xffffffffb4e48d80
Build id: 3.10.0-862.14.4.el7.x86_64
0x3a0
16
inet_release
vmlinux 0xffffffffb4e7236d
Build id: 3.10.0-862.14.4.el7.x86_64
0x7d
17
sock_release
vmlinux 0xffffffffb4dd0a35
Build id: 3.10.0-862.14.4.el7.x86_64
0x25
18
sock_close
vmlinux 0xffffffffb4dd0ab2
Build id: 3.10.0-862.14.4.el7.x86_64
0x12
19
__fput
vmlinux 0xffffffffb4a214fc
Build id: 3.10.0-862.14.4.el7.x86_64
0xec
20
____fput
vmlinux 0xffffffffb4a2175e
Build id: 3.10.0-862.14.4.el7.x86_64
0xe
21
task_work_run
vmlinux 0xffffffffb48bab8b
Build id: 3.10.0-862.14.4.el7.x86_64
0xbb
22
do_notify_resume
vmlinux 0xffffffffb482bc55
Build id: 3.10.0-862.14.4.el7.x86_64
0xa5
23
int_signal
vmlinux 0xffffffffb4f25ae4
Build id: 3.10.0-862.14.4.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 33s! [java:63603]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_seq_midi snd_seq_midi_event lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer ppdev snd vmw_balloon soundcore pcspkr nfit sg joydev libnvdimm parport_pc parport vmw_vmci shpchp i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm mptspi serio_raw scsi_transport_spi e1000 mptscsih ata_piix libata i2c_core mptbase dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 63603 Comm: java Tainted: G        W    L ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff9eb3ef5deeb0 ti: ffff9eb65efa8000 task.ti: ffff9eb65efa8000
RIP: 0010:[<ffffffffc0131867>]  [<ffffffffc0131867>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9eb65efaba20  EFLAGS: 00010286
RAX: ffffc1d8c1903818 RBX: ffff9eb4b9888000 RCX: 0000000000000014
RDX: 0000000000000014 RSI: ffff9eb375dd6500 RDI: ffff9eb76d47c000
RBP: ffff9eb65efabaa0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: ffffffffc060a7bd
R13: ffff9eb65efaba60 R14: 0000000000000000 R15: 0000000000000000
FS:  00007f4f7da1f700(0000) GS:ffff9eb76d640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4faf8a2710 CR3: 00000000af560000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb4df02e6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb4e1bc0a>] sch_direct_xmit+0x11a/0x250
[<ffffffffb4df31d1>] __dev_queue_xmit+0x4a1/0x660
[<ffffffffb4df33a0>] dev_queue_xmit+0x10/0x20
[<ffffffffb4e3c5d6>] ip_finish_output+0x546/0x7a0
[<ffffffffb4e3cb33>] ip_output+0x73/0xe0
[<ffffffffb4e3c090>] ? __ip_append_data.isra.50+0xa50/0xa50
[<ffffffffb4e3a727>] ip_local_out_sk+0x37/0x40
[<ffffffffb4e3aa93>] ip_queue_xmit+0x143/0x3a0
[<ffffffffb4e54bd4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffb4e55242>] tcp_write_xmit+0x172/0xd00
[<ffffffffb4e5604e>] __tcp_push_pending_frames+0x2e/0xc0
[<ffffffffb4e57b02>] tcp_send_fin+0x62/0x190
[<ffffffffb4e48d80>] tcp_close+0x3a0/0x400
[<ffffffffb4e7236d>] inet_release+0x7d/0x90
[<ffffffffb4dd0a35>] sock_release+0x25/0x90
[<ffffffffb4dd0ab2>] sock_close+0x12/0x20
[<ffffffffb4a214fc>] __fput+0xec/0x260
[<ffffffffb4a2175e>] ____fput+0xe/0x10
[<ffffffffb48bab8b>] task_work_run+0xbb/0xe0
[<ffffffffb482bc55>] do_notify_resume+0xa5/0xc0
[<ffffffffb4f25ae4>] int_signal+0x12/0x17
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #7986
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0319877
Build id: 3.10.0-1127.13.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff96c53386
Build id: 3.10.0-1127.13.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff96c8131a
Build id: 3.10.0-1127.13.1.el7.x86_64
0x11a
4
__qdisc_run
vmlinux 0xffffffff96c814de
Build id: 3.10.0-1127.13.1.el7.x86_64
0x8e
5
__dev_queue_xmit
vmlinux 0xffffffff96c56319
Build id: 3.10.0-1127.13.1.el7.x86_64
0x4a9
6
dev_queue_xmit
vmlinux 0xffffffff96c564d0
Build id: 3.10.0-1127.13.1.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffff96ca429b
Build id: 3.10.0-1127.13.1.el7.x86_64
0x67b
8
ip_output
vmlinux 0xffffffff96ca481b
Build id: 3.10.0-1127.13.1.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffff96ca3c20
Build id: 3.10.0-1127.13.1.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffff96ca2247
Build id: 3.10.0-1127.13.1.el7.x86_64
0x37
11
ip_queue_xmit
vmlinux 0xffffffff96ca25e4
Build id: 3.10.0-1127.13.1.el7.x86_64
0x144
12
tcp_transmit_skb
vmlinux 0xffffffff96cbccd4
Build id: 3.10.0-1127.13.1.el7.x86_64
0x4e4
13
tcp_write_xmit
vmlinux 0xffffffff96cbd35a
Build id: 3.10.0-1127.13.1.el7.x86_64
0x18a
14
__tcp_push_pending_frames
vmlinux 0xffffffff96cbe19e
Build id: 3.10.0-1127.13.1.el7.x86_64
0x2e
15
tcp_send_fin
vmlinux 0xffffffff96cbfe22
Build id: 3.10.0-1127.13.1.el7.x86_64
0x62
16
tcp_close
vmlinux 0xffffffff96cb0d00
Build id: 3.10.0-1127.13.1.el7.x86_64
0x3a0
17
inet_release
vmlinux 0xffffffff96cdad3d
Build id: 3.10.0-1127.13.1.el7.x86_64
0x7d
18
sock_release
vmlinux 0xffffffff96c32765
Build id: 3.10.0-1127.13.1.el7.x86_64
0x25
19
sock_close
vmlinux 0xffffffff96c327e2
Build id: 3.10.0-1127.13.1.el7.x86_64
0x12
20
__fput
vmlinux 0xffffffff9684f49c
Build id: 3.10.0-1127.13.1.el7.x86_64
0xec
21
____fput
vmlinux 0xffffffff9684f6ce
Build id: 3.10.0-1127.13.1.el7.x86_64
0xe
22
task_work_run
vmlinux 0xffffffff966c31cb
Build id: 3.10.0-1127.13.1.el7.x86_64
0xbb
23
do_exit
vmlinux 0xffffffff966a2404
Build id: 3.10.0-1127.13.1.el7.x86_64
0x2d4
24
? __fput
vmlinux 0xffffffff9684f545
Build id: 3.10.0-1127.13.1.el7.x86_64
0x195
25
? system_call_after_swapgs
vmlinux 0xffffffff96d92e15
Build id: 3.10.0-1127.13.1.el7.x86_64
0xa2
26
do_group_exit
vmlinux 0xffffffff966a2bcf
Build id: 3.10.0-1127.13.1.el7.x86_64
0x3f
27
SyS_exit_group
vmlinux 0xffffffff966a2c44
Build id: 3.10.0-1127.13.1.el7.x86_64
0x14
28
system_call_fastpath
vmlinux 0xffffffff96d92ed2
Build id: 3.10.0-1127.13.1.el7.x86_64
0x25
29
? system_call_after_swapgs
vmlinux 0xffffffff96d92e15
Build id: 3.10.0-1127.13.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 54s! [sshd:12077]
Modules linked in: ip6table_filter ip6_tables xt_time nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack tcp_diag inet_diag sysdig_probe(OE) ipt_REJECT nf_reject_ipv4 xt_owner xt_multiport iptable_filter ext4 mbcache jbd2 loop dm_mirror dm_region_hash dm_log dm_mod sg ppdev pcspkr parport_pc parport i2c_piix4 joydev ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper virtio_blk syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata e1000 serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy
CPU: 3 PID: 12077 Comm: sshd Kdump: loaded Tainted: G           OEL ------------   3.10.0-1127.13.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
task: ffff882f8c32e2a0 ti: ffff883083b94000 task.ti: ffff883083b94000
RIP: 0010:[<ffffffffc0319877>]  [<ffffffffc0319877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff883083b978f8  EFLAGS: 00000286
RAX: ffffb0ce41bc3818 RBX: 000000000000004a RCX: 0000000000000024
RDX: 0000000000000024 RSI: ffff883394698f80 RDI: ffff8833949b4000
RBP: ffff883083b97978 R08: 0000000000000100 R09: 0000000022100000
R10: 2af4064000400000 R11: 0000000000000000 R12: 0000000273c908c6
R13: 0000000196825e9d R14: ffffffff969b0830 R15: ffff883083b978b0
FS:  0000000000000000(0000) GS:ffff8833a7cc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055ffb692cae0 CR3: 0000000141410000 CR4: 00000000000006e0
Call Trace:
[<ffffffff96c53386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff96c8131a>] sch_direct_xmit+0x11a/0x250
[<ffffffff96c814de>] __qdisc_run+0x8e/0x360
[<ffffffff96c56319>] __dev_queue_xmit+0x4a9/0x650
[<ffffffff96c564d0>] dev_queue_xmit+0x10/0x20
[<ffffffff96ca429b>] ip_finish_output+0x67b/0x8d0
[<ffffffff96ca481b>] ip_output+0x7b/0xf0
[<ffffffff96ca3c20>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff96ca2247>] ip_local_out_sk+0x37/0x40
[<ffffffff96ca25e4>] ip_queue_xmit+0x144/0x3c0
[<ffffffff96cbccd4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff96cbd35a>] tcp_write_xmit+0x18a/0xd40
[<ffffffff96cbe19e>] __tcp_push_pending_frames+0x2e/0xc0
[<ffffffff96cbfe22>] tcp_send_fin+0x62/0x190
[<ffffffff96cb0d00>] tcp_close+0x3a0/0x400
[<ffffffff96cdad3d>] inet_release+0x7d/0x90
[<ffffffff96c32765>] sock_release+0x25/0x90
[<ffffffff96c327e2>] sock_close+0x12/0x20
[<ffffffff9684f49c>] __fput+0xec/0x230
[<ffffffff9684f6ce>] ____fput+0xe/0x10
[<ffffffff966c31cb>] task_work_run+0xbb/0xe0
[<ffffffff966a2404>] do_exit+0x2d4/0xa20
[<ffffffff9684f545>] ? __fput+0x195/0x230
[<ffffffff96d92e15>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff966a2bcf>] do_group_exit+0x3f/0xa0
[<ffffffff966a2c44>] SyS_exit_group+0x14/0x20
[<ffffffff96d92ed2>] system_call_fastpath+0x25/0x2a
[<ffffffff96d92e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #4331
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00a8877
Build id: 3.10.0-1127.18.2.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff8ee53386
Build id: 3.10.0-1127.18.2.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff8ee8131a
Build id: 3.10.0-1127.18.2.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff8ee56301
Build id: 3.10.0-1127.18.2.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffff8ee564d0
Build id: 3.10.0-1127.18.2.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffff8eea429b
Build id: 3.10.0-1127.18.2.el7.x86_64
0x67b
7
ip_output
vmlinux 0xffffffff8eea481b
Build id: 3.10.0-1127.18.2.el7.x86_64
0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffff8eea3c20
Build id: 3.10.0-1127.18.2.el7.x86_64
0xa60
9
ip_local_out_sk
vmlinux 0xffffffff8eea2247
Build id: 3.10.0-1127.18.2.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffff8eea25e4
Build id: 3.10.0-1127.18.2.el7.x86_64
0x144
11
tcp_transmit_skb
vmlinux 0xffffffff8eebccd4
Build id: 3.10.0-1127.18.2.el7.x86_64
0x4e4
12
tcp_write_xmit
vmlinux 0xffffffff8eebd35a
Build id: 3.10.0-1127.18.2.el7.x86_64
0x18a
13
__tcp_push_pending_frames
vmlinux 0xffffffff8eebe19e
Build id: 3.10.0-1127.18.2.el7.x86_64
0x2e
14
tcp_send_fin
vmlinux 0xffffffff8eebfe22
Build id: 3.10.0-1127.18.2.el7.x86_64
0x62
15
tcp_close
vmlinux 0xffffffff8eeb0d00
Build id: 3.10.0-1127.18.2.el7.x86_64
0x3a0
16
inet_release
vmlinux 0xffffffff8eedad3d
Build id: 3.10.0-1127.18.2.el7.x86_64
0x7d
17
sock_release
vmlinux 0xffffffff8ee32765
Build id: 3.10.0-1127.18.2.el7.x86_64
0x25
18
sock_close
vmlinux 0xffffffff8ee327e2
Build id: 3.10.0-1127.18.2.el7.x86_64
0x12
19
__fput
vmlinux 0xffffffff8ea4f4bc
Build id: 3.10.0-1127.18.2.el7.x86_64
0xec
20
____fput
vmlinux 0xffffffff8ea4f6ee
Build id: 3.10.0-1127.18.2.el7.x86_64
0xe
21
task_work_run
vmlinux 0xffffffff8e8c31cb
Build id: 3.10.0-1127.18.2.el7.x86_64
0xbb
22
do_notify_resume
vmlinux 0xffffffff8e82cc65
Build id: 3.10.0-1127.18.2.el7.x86_64
0xa5
23
int_signal
vmlinux 0xffffffff8ef9322f
Build id: 3.10.0-1127.18.2.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 30s! [srs:9047]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus ghash_clmulni_intel snd_seq ppdev snd_seq_device aesni_intel snd_pcm lrw gf128mul vmw_balloon glue_helper ablk_helper cryptd
joydev pcspkr snd_timer sg snd vmw_vmci soundcore i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nfit ttm crct10dif_pclmul crct10dif_common crc32c_intel libnvdimm drm ata_piix mptspi libata scsi_transport_spi e1000 serio_raw mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 9047 Comm: srs Tainted: G             L ------------   3.10.0-1127.18.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff9546f70a9070 ti: ffff95468974c000 task.ti: ffff95468974c000
RIP: 0010:[<ffffffffc00a8877>]  [<ffffffffc00a8877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff95468974fa00  EFLAGS: 00000286
RAX: ffffad73c0743818 RBX: ffff95468974f9c8 RCX: 000000000000005d
RDX: 000000000000005d RSI: ffff9546b5151700 RDI: ffff9546f72cc000
RBP: ffff95468974fa80 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: 0000000000000000
R13: ffff9546bd17e000 R14: 0000000000000000 R15: ffff95468a8160f8
FS:  00007fd31efa2740(0000) GS:ffff9546fc640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd70ed07830 CR3: 000000004703e000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8ee53386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff8ee8131a>] sch_direct_xmit+0x11a/0x250
[<ffffffff8ee56301>] __dev_queue_xmit+0x491/0x650
[<ffffffff8ee564d0>] dev_queue_xmit+0x10/0x20
[<ffffffff8eea429b>] ip_finish_output+0x67b/0x8d0
[<ffffffff8eea481b>] ip_output+0x7b/0xf0
[<ffffffff8eea3c20>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff8eea2247>] ip_local_out_sk+0x37/0x40
[<ffffffff8eea25e4>] ip_queue_xmit+0x144/0x3c0
[<ffffffff8eebccd4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff8eebd35a>] tcp_write_xmit+0x18a/0xd40
[<ffffffff8eebe19e>] __tcp_push_pending_frames+0x2e/0xc0
[<ffffffff8eebfe22>] tcp_send_fin+0x62/0x190
[<ffffffff8eeb0d00>] tcp_close+0x3a0/0x400
[<ffffffff8eedad3d>] inet_release+0x7d/0x90
[<ffffffff8ee32765>] sock_release+0x25/0x90
[<ffffffff8ee327e2>] sock_close+0x12/0x20
[<ffffffff8ea4f4bc>] __fput+0xec/0x230
[<ffffffff8ea4f6ee>] ____fput+0xe/0x10
[<ffffffff8e8c31cb>] task_work_run+0xbb/0xe0
[<ffffffff8e82cc65>] do_notify_resume+0xa5/0xc0
[<ffffffff8ef9322f>] int_signal+0x12/0x17
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00