Info

Permalink
Function
e1000_xmit_frame
First occurrence
2020-11-08
Last occurrence
2026-07-06
State
NEW
Components
kernel
Quality
-53

Statistics

Operating system Count
CentOS 7 25
Architecture Count
x86_64 25
Related packages Count
kernel 25
 0:3.10.0-1160.11.1.el7 7
 0:3.10.0-327.el7 4
 0:3.10.0-1160.71.1.el7 3
 0:3.10.0-1160.102.1.el7 2
 0:3.10.0-514.26.1.el7 1
 0:3.10.0-514.el7 1
 0:3.10.0-1160.el7 1
 0:3.10.0-862.9.1.el7 1
 0:3.10.0-957.el7 1
 0:3.10.0-1160.59.1.el7 1
 0:3.10.0-1160.119.1.el7.tuxcare.els12 1
 0:3.10.0-1160.92.1.el7 1
 0:3.10.0-1160.119.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #1417450
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff90bba385
Build id: 3.10.0-1160.119.1.el7.x86_64
0x15
2
vmxnet3_xmit_frame
vmxnet3 0xffffffffc009740f
Build id: 3.10.0-1160.119.1.el7.x86_64
0xaaf
3
? vmxnet3_xmit_frame
vmxnet3 0xffffffffc009740f
Build id: 3.10.0-1160.119.1.el7.x86_64
0xaaf
4
dev_hard_start_xmit
vmlinux 0xffffffff90a7edc6
Build id: 3.10.0-1160.119.1.el7.x86_64
0x246
5
sch_direct_xmit
vmlinux 0xffffffff90aadd72
Build id: 3.10.0-1160.119.1.el7.x86_64
0x122
6
__dev_queue_xmit
vmlinux 0xffffffff90a81dd1
Build id: 3.10.0-1160.119.1.el7.x86_64
0x491
7
dev_queue_xmit
vmlinux 0xffffffff90a81fa0
Build id: 3.10.0-1160.119.1.el7.x86_64
0x10
8
ip_finish_output
vmlinux 0xffffffff90ad260b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x67b
9
ip_output
vmlinux 0xffffffff90ad2b9b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x7b
10
? __alloc_skb
vmlinux 0xffffffff90a687fd
Build id: 3.10.0-1160.119.1.el7.x86_64
0x5d
11
ip_local_out_sk
vmlinux 0xffffffff90ad0587
Build id: 3.10.0-1160.119.1.el7.x86_64
0x37
12
ip_queue_xmit
vmlinux 0xffffffff90ad0924
Build id: 3.10.0-1160.119.1.el7.x86_64
0x144
13
tcp_transmit_skb
vmlinux 0xffffffff90aeb5f4
Build id: 3.10.0-1160.119.1.el7.x86_64
0x4e4
14
tcp_send_ack
vmlinux 0xffffffff90aed83f
Build id: 3.10.0-1160.119.1.el7.x86_64
0x11f
15
__tcp_ack_snd_check
vmlinux 0xffffffff90ae1c6b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x6b
16
tcp_rcv_established
vmlinux 0xffffffff90ae7dc7
Build id: 3.10.0-1160.119.1.el7.x86_64
0x267
17
tcp_v4_do_rcv
vmlinux 0xffffffff90af30f2
Build id: 3.10.0-1160.119.1.el7.x86_64
0x112
18
? sk_wait_data
vmlinux 0xffffffff90a63fda
Build id: 3.10.0-1160.119.1.el7.x86_64
0xda
19
tcp_prequeue_process
vmlinux 0xffffffff90adaf37
Build id: 3.10.0-1160.119.1.el7.x86_64
0x77
20
tcp_recvmsg
vmlinux 0xffffffff90adbeac
Build id: 3.10.0-1160.119.1.el7.x86_64
0x88c
21
inet_recvmsg
vmlinux 0xffffffff90b0b690
Build id: 3.10.0-1160.119.1.el7.x86_64
0x80
22
sock_aio_read.part.9
vmlinux 0xffffffff90a5d7dc
Build id: 3.10.0-1160.119.1.el7.x86_64
0x14c
23
sock_aio_read
vmlinux 0xffffffff90a5d829
Build id: 3.10.0-1160.119.1.el7.x86_64
0x29
24
do_sync_read
vmlinux 0xffffffff9065b1d3
Build id: 3.10.0-1160.119.1.el7.x86_64
0x93
25
vfs_read
vmlinux 0xffffffff9065bc9d
Build id: 3.10.0-1160.119.1.el7.x86_64
0x14d
26
SyS_read
vmlinux 0xffffffff9065ca65
Build id: 3.10.0-1160.119.1.el7.x86_64
0x55
27
system_call_fastpath
vmlinux 0xffffffff90bc539a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 333s! [oracle:25870]
Modules linked in: vmw_vsock_vmci_transport vsock sunrpc ext4 nfit mbcache jbd2 libnvdimm coretemp iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg vmw_vmci parport_pc i2c_piix4 parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi crct10dif_pclmul libata crct10dif_common mptscsih serio_raw crc32c_intel mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 15 PID: 25870 Comm: oracle Kdump: loaded Tainted: G             L ------------   3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8d9af7990000 ti: ffff8d9d00624000 task.ti: ffff8d9d00624000
RIP: 0010:[<ffffffff90bba385>]  [<ffffffff90bba385>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8d9d006277f0  EFLAGS: 00000283
RAX: ffff8d9cfbba1a00 RBX: ffff8d9d00627958 RCX: 0000000000000000
RDX: ffff8d9a34412500 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff8d9d006277f0 R08: 50005123a9565000 R09: 0000000000c8004e
R10: 6503a8c02f6d0640 R11: ffff8d9cfbba1a00 R12: ffffffff9079c615
R13: ffff8d9d00627778 R14: 0000000000000000 R15: ffffffff904ee68c
FS:  00007fc12a1af740(0000) GS:ffff8d9dffd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5d1919a000 CR3: 000000014149e000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc009740f>] vmxnet3_xmit_frame+0xaaf/0xfd0 [vmxnet3]
[<ffffffffc009740f>] ? vmxnet3_xmit_frame+0xaaf/0xfd0 [vmxnet3]
[<ffffffff90a7edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff90aadd72>] sch_direct_xmit+0x122/0x250
[<ffffffff90a81dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffff90a81fa0>] dev_queue_xmit+0x10/0x20
[<ffffffff90ad260b>] ip_finish_output+0x67b/0x8d0
[<ffffffff90ad2b9b>] ip_output+0x7b/0x100
[<ffffffff90a687fd>] ? __alloc_skb+0x5d/0x2d0
[<ffffffff90ad0587>] ip_local_out_sk+0x37/0x40
[<ffffffff90ad0924>] ip_queue_xmit+0x144/0x3d0
[<ffffffff90aeb5f4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff90aed83f>] tcp_send_ack+0x11f/0x180
[<ffffffff90ae1c6b>] __tcp_ack_snd_check+0x6b/0xa0
[<ffffffff90ae7dc7>] tcp_rcv_established+0x267/0x7a0
[<ffffffff90af30f2>] tcp_v4_do_rcv+0x112/0x360
[<ffffffff90a63fda>] ? sk_wait_data+0xda/0x120
[<ffffffff90adaf37>] tcp_prequeue_process+0x77/0xc0
[<ffffffff90adbeac>] tcp_recvmsg+0x88c/0xb30
[<ffffffff90b0b690>] inet_recvmsg+0x80/0xb0
[<ffffffff90a5d7dc>] sock_aio_read.part.9+0x14c/0x170
[<ffffffff90a5d829>] sock_aio_read+0x29/0x40
[<ffffffff9065b1d3>] do_sync_read+0x93/0xe0
[<ffffffff9065bc9d>] vfs_read+0x14d/0x170
[<ffffffff9065ca65>] SyS_read+0x55/0xd0
[<ffffffff90bc539a>] system_call_fastpath+0x25/0x2a
Code: 07 00 0f 1f 40 00 5d c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 cc cc cc cc 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #1297391
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc03828e7
Build id: 3.10.0-1160.92.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff8327edc6
Build id: 3.10.0-1160.92.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff832add72
Build id: 3.10.0-1160.92.1.el7.x86_64
0x122
4
__dev_queue_xmit
vmlinux 0xffffffff83281dd1
Build id: 3.10.0-1160.92.1.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffff83281fa0
Build id: 3.10.0-1160.92.1.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffff832d260b
Build id: 3.10.0-1160.92.1.el7.x86_64
0x67b
7
ip_output
vmlinux 0xffffffff832d2b9b
Build id: 3.10.0-1160.92.1.el7.x86_64
0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffff832d1f90
Build id: 3.10.0-1160.92.1.el7.x86_64
0xa60
9
ip_local_out_sk
vmlinux 0xffffffff832d0587
Build id: 3.10.0-1160.92.1.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffff832d0924
Build id: 3.10.0-1160.92.1.el7.x86_64
0x144
11
tcp_transmit_skb
vmlinux 0xffffffff832eb5f4
Build id: 3.10.0-1160.92.1.el7.x86_64
0x4e4
12
tcp_send_ack
vmlinux 0xffffffff832ed83f
Build id: 3.10.0-1160.92.1.el7.x86_64
0x11f
13
__tcp_ack_snd_check
vmlinux 0xffffffff832e1c6b
Build id: 3.10.0-1160.92.1.el7.x86_64
0x6b
14
tcp_rcv_state_process
vmlinux 0xffffffff832e8fd6
Build id: 3.10.0-1160.92.1.el7.x86_64
0xa36
15
? __tcp_ack_snd_check
vmlinux 0xffffffff832e1c8f
Build id: 3.10.0-1160.92.1.el7.x86_64
0x8f
16
? tcp_rcv_established
vmlinux 0xffffffff832e7dc7
Build id: 3.10.0-1160.92.1.el7.x86_64
0x267
17
tcp_v4_do_rcv
vmlinux 0xffffffff832f316b
Build id: 3.10.0-1160.92.1.el7.x86_64
0x18b
18
? sk_wait_data
vmlinux 0xffffffff83263fda
Build id: 3.10.0-1160.92.1.el7.x86_64
0xda
19
tcp_prequeue_process
vmlinux 0xffffffff832daf37
Build id: 3.10.0-1160.92.1.el7.x86_64
0x77
20
tcp_recvmsg
vmlinux 0xffffffff832dbeac
Build id: 3.10.0-1160.92.1.el7.x86_64
0x88c
21
inet_recvmsg
vmlinux 0xffffffff8330b690
Build id: 3.10.0-1160.92.1.el7.x86_64
0x80
22
sock_aio_read.part.9
vmlinux 0xffffffff8325d7dc
Build id: 3.10.0-1160.92.1.el7.x86_64
0x14c
23
? secure_tcp_sequence_number
vmlinux 0xffffffff832747a9
Build id: 3.10.0-1160.92.1.el7.x86_64
0x69
24
sock_aio_read
vmlinux 0xffffffff8325d829
Build id: 3.10.0-1160.92.1.el7.x86_64
0x29
25
do_sync_read
vmlinux 0xffffffff82e5b113
Build id: 3.10.0-1160.92.1.el7.x86_64
0x93
26
vfs_read
vmlinux 0xffffffff82e5bbdd
Build id: 3.10.0-1160.92.1.el7.x86_64
0x14d
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 32s! [zabbix_agentd:1098]
Modules linked in: tcp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi snd_ens1371 crc32_pclmul snd_rawmidi snd_ac97_codec ghash_clmulni_intel ac97_bus ppdev snd_seq snd_seq_device vmw_balloon aesni_intel snd_pcm lrw gf128mul glue_helper ablk_helper cryptd snd_timer snd joydev pcspkr sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm libnvdimm mptspi scsi_transport_spi mptscsih ata_piix mptbase crct10dif_pclmul crct10dif_common libata e1000 crc32c_intel serio_raw drm_panel_orientation_quirks fuse
CPU: 3 PID: 1098 Comm: zabbix_agentd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.92.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff982959f68000 ti: ffff982959f70000 task.ti: ffff982959f70000
RIP: 0010:[<ffffffffc03828e7>]  [<ffffffffc03828e7>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff982959f737e0  EFLAGS: 00000286
RAX: ffffa60a834c3818 RBX: 0000000000000001 RCX: 000000000000005f
RDX: 000000000000005f RSI: ffff98277abef040 RDI: ffff98295c02c000
RBP: ffff982959f73860 R08: 0000000000000100 R09: 0000000022100000
R10: 9101a8c0de1b0640 R11: 0000000000000000 R12: 0000000000000202
R13: 00000007f7e294c2 R14: 0000000100000202 R15: ffffffff82fc8fd0
FS:  00007f918f03c840(0000) GS:ffff98295d6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001dedb20 CR3: 0000000819f66000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff8327edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff832add72>] sch_direct_xmit+0x122/0x250
[<ffffffff83281dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffff83281fa0>] dev_queue_xmit+0x10/0x20
[<ffffffff832d260b>] ip_finish_output+0x67b/0x8d0
[<ffffffff832d2b9b>] ip_output+0x7b/0x100
[<ffffffff832d1f90>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff832d0587>] ip_local_out_sk+0x37/0x40
[<ffffffff832d0924>] ip_queue_xmit+0x144/0x3d0
[<ffffffff832eb5f4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff832ed83f>] tcp_send_ack+0x11f/0x180
[<ffffffff832e1c6b>] __tcp_ack_snd_check+0x6b/0xa0
[<ffffffff832e8fd6>] tcp_rcv_state_process+0xa36/0xf50
[<ffffffff832e1c8f>] ? __tcp_ack_snd_check+0x8f/0xa0
[<ffffffff832e7dc7>] ? tcp_rcv_established+0x267/0x7a0
[<ffffffff832f316b>] tcp_v4_do_rcv+0x18b/0x360
[<ffffffff83263fda>] ? sk_wait_data+0xda/0x120
[<ffffffff832daf37>] tcp_prequeue_process+0x77/0xc0
[<ffffffff832dbeac>] tcp_recvmsg+0x88c/0xb30
[<ffffffff8330b690>] inet_recvmsg+0x80/0xb0
[<ffffffff8325d7dc>] sock_aio_read.part.9+0x14c/0x170
[<ffffffff832747a9>] ? secure_tcp_sequence_number+0x69/0xa0
[<ffffffff8325d829>] sock_aio_read+0x29/0x40
[<ffffffff82e5b113>] do_sync_read+0x93/0xe0
[<ffffffff82e5bbdd>] vfs_read+0x14d/0x170
Complete report #1093736
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc015e44c
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0xaec
2
dev_hard_start_xmit
vmlinux 0xffffffffb4a7edd6
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffb4aadd82
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x122
4
__dev_queue_xmit
vmlinux 0xffffffffb4a81de1
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffffb4a81fb0
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffffb4ad264b
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x67b
7
ip_output
vmlinux 0xffffffffb4ad2bdb
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffffb4ad1fd0
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0xa60
9
ip_local_out_sk
vmlinux 0xffffffffb4ad05c7
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffffb4ad0964
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x144
11
tcp_transmit_skb
vmlinux 0xffffffffb4aeb634
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x4e4
12
tcp_send_ack
vmlinux 0xffffffffb4aed87f
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x11f
13
__tcp_ack_snd_check
vmlinux 0xffffffffb4ae1cab
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x6b
14
tcp_rcv_established
vmlinux 0xffffffffb4ae7e07
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x267
15
tcp_v4_do_rcv
vmlinux 0xffffffffb4af30f2
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x112
16
? sk_wait_data
vmlinux 0xffffffffb4a63fda
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0xda
17
tcp_prequeue_process
vmlinux 0xffffffffb4adaf77
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x77
18
tcp_recvmsg
vmlinux 0xffffffffb4adbeec
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x88c
19
? xfs_iunlock
xfs 0xffffffffc03dcf44
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x124
20
inet_recvmsg
vmlinux 0xffffffffb4b0b690
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x80
21
sock_aio_read.part.9
vmlinux 0xffffffffb4a5d7dc
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x14c
22
sock_aio_read
vmlinux 0xffffffffb4a5d829
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x29
23
do_sync_read
vmlinux 0xffffffffb465b233
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x93
24
vfs_read
vmlinux 0xffffffffb465bcfd
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x14d
25
SyS_read
vmlinux 0xffffffffb465cac5
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x55
26
system_call_fastpath
vmlinux 0xffffffffb4bc539a
Build id: 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 29s! [exim:19737]
Modules linked in: binfmt_misc ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 ip6table_filter xt_time nf_log_ipv4 nf_log_common nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit vfat fat isofs iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 xt_owner xt_multiport iptable_filter tcp_diag inet_diag ext4 mbcache jbd2 loop sunrpc coretemp crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix scsi_transport_spi drm mptscsih mptbase crct10dif_pclmul libata crct10dif_common crc32c_intel serio_raw vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 19737 Comm: exim Kdump: loaded Not tainted 3.10.0-1160.119.1.el7.tuxcare.els12.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffff8c012a3c6300 ti: ffff8bff467f4000 task.ti: ffff8bff467f4000
RIP: 0010:[<ffffffffc015e44c>]  [<ffffffffc015e44c>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0018:ffff8bff467f7800  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffff8c013fc93940 RCX: 0000000000000161
RDX: ffffa1f18189e608 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff8bff467f7890 R08: 0c005e65540a1100 R09: 0000000000c8404e
R10: 953a49d013f10640 R11: ffff8c012ce05600 R12: ffff8c009861a950
R13: ffff8c013fc93940 R14: 0000000000000001 R15: 0000000000001000
FS:  00007fd4e100b880(0000) GS:ffff8c013fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007efcb644a000 CR3: 0000000117690000 CR4: 00000000000007e0
Call Trace:
[<ffffffffb4a7edd6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb4aadd82>] sch_direct_xmit+0x122/0x250
[<ffffffffb4a81de1>] __dev_queue_xmit+0x491/0x650
[<ffffffffb4a81fb0>] dev_queue_xmit+0x10/0x20
[<ffffffffb4ad264b>] ip_finish_output+0x67b/0x8d0
[<ffffffffb4ad2bdb>] ip_output+0x7b/0x100
[<ffffffffb4ad1fd0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffb4ad05c7>] ip_local_out_sk+0x37/0x40
[<ffffffffb4ad0964>] ip_queue_xmit+0x144/0x3d0
[<ffffffffb4aeb634>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffb4aed87f>] tcp_send_ack+0x11f/0x180
[<ffffffffb4ae1cab>] __tcp_ack_snd_check+0x6b/0xa0
[<ffffffffb4ae7e07>] tcp_rcv_established+0x267/0x7a0
[<ffffffffb4af30f2>] tcp_v4_do_rcv+0x112/0x360
[<ffffffffb4a63fda>] ? sk_wait_data+0xda/0x120
[<ffffffffb4adaf77>] tcp_prequeue_process+0x77/0xc0
[<ffffffffb4adbeec>] tcp_recvmsg+0x88c/0xb30
[<ffffffffc03dcf44>] ? xfs_iunlock+0x124/0x130 [xfs]
[<ffffffffb4b0b690>] inet_recvmsg+0x80/0xb0
[<ffffffffb4a5d7dc>] sock_aio_read.part.9+0x14c/0x170
[<ffffffffb4a5d829>] sock_aio_read+0x29/0x40
[<ffffffffb465b233>] do_sync_read+0x93/0xe0
[<ffffffffb465bcfd>] vfs_read+0x14d/0x170
[<ffffffffb465cac5>] SyS_read+0x55/0xd0
[<ffffffffb4bc539a>] system_call_fastpath+0x25/0x2a
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41 
Complete report #905813
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0487877
Build id: 3.10.0-1160.59.1.el7.x86_64
0x7f7
2
? tcp_in_window
nf_conntrack 0xffffffffc0a5dfad
Build id: 3.10.0-1160.59.1.el7.x86_64
0xfd
3
dev_hard_start_xmit
vmlinux 0xffffffff9c458f96
Build id: 3.10.0-1160.59.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffff9c4870aa
Build id: 3.10.0-1160.59.1.el7.x86_64
0x11a
5
__qdisc_run
vmlinux 0xffffffff9c48726e
Build id: 3.10.0-1160.59.1.el7.x86_64
0x8e
6
__dev_queue_xmit
vmlinux 0xffffffff9c45bc98
Build id: 3.10.0-1160.59.1.el7.x86_64
0x218
7
? iptable_nat_ipv4_fn
iptable_nat 0xffffffffc0a94020
Build id: 3.10.0-1160.59.1.el7.x86_64
0x20
8
dev_queue_xmit
vmlinux 0xffffffff9c45c0e0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x10
9
neigh_resolve_output
vmlinux 0xffffffff9c468e0d
Build id: 3.10.0-1160.59.1.el7.x86_64
0x11d
10
ip_finish_output
vmlinux 0xffffffff9c4aa86c
Build id: 3.10.0-1160.59.1.el7.x86_64
0x27c
11
ip_output
vmlinux 0xffffffff9c4ab1eb
Build id: 3.10.0-1160.59.1.el7.x86_64
0x7b
12
? __ip_append_data.isra.50
vmlinux 0xffffffff9c4aa5f0
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa60
13
ip_local_out_sk
vmlinux 0xffffffff9c4a8c17
Build id: 3.10.0-1160.59.1.el7.x86_64
0x37
14
ip_queue_xmit
vmlinux 0xffffffff9c4a8fb4
Build id: 3.10.0-1160.59.1.el7.x86_64
0x144
15
tcp_transmit_skb
vmlinux 0xffffffff9c4c3734
Build id: 3.10.0-1160.59.1.el7.x86_64
0x4e4
16
tcp_send_ack
vmlinux 0xffffffff9c4c594f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x11f
17
__tcp_ack_snd_check
vmlinux 0xffffffff9c4b9f3b
Build id: 3.10.0-1160.59.1.el7.x86_64
0x6b
18
tcp_rcv_established
vmlinux 0xffffffff9c4c01b5
Build id: 3.10.0-1160.59.1.el7.x86_64
0x425
19
tcp_v4_do_rcv
vmlinux 0xffffffff9c4cb04a
Build id: 3.10.0-1160.59.1.el7.x86_64
0x10a
20
? sk_wait_data
vmlinux 0xffffffff9c43eb4a
Build id: 3.10.0-1160.59.1.el7.x86_64
0xda
21
tcp_prequeue_process
vmlinux 0xffffffff9c4b3357
Build id: 3.10.0-1160.59.1.el7.x86_64
0x77
22
tcp_recvmsg
vmlinux 0xffffffff9c4b42b4
Build id: 3.10.0-1160.59.1.el7.x86_64
0x884
23
inet_recvmsg
vmlinux 0xffffffff9c4e2ff0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x80
24
sock_aio_read.part.9
vmlinux 0xffffffff9c4386ec
Build id: 3.10.0-1160.59.1.el7.x86_64
0x14c
25
sock_aio_read
vmlinux 0xffffffff9c438731
Build id: 3.10.0-1160.59.1.el7.x86_64
0x21
26
do_sync_read
vmlinux 0xffffffff9c04d9c3
Build id: 3.10.0-1160.59.1.el7.x86_64
0x93
27
vfs_read
vmlinux 0xffffffff9c04e4a5
Build id: 3.10.0-1160.59.1.el7.x86_64
0x145
28
SyS_read
vmlinux 0xffffffff9c04f27f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x7f
29
system_call_fastpath
vmlinux 0xffffffff9c599f92
Build id: 3.10.0-1160.59.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 32s! [kworker/0:1:31613]
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 30s! [dbserver_d:23194]
Modules linked in: binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 snd_ens1371 snd_rawmidi snd_ac97_codec crc32_pclmul ac97_bus snd_seq ghash_clmulni_intel snd_seq_device ppdev nfit snd_pcm aesni_intel libnvdimm lrw gf128mul vmw_balloon snd_timer glue_helper ablk_helper cryptd snd soundcore joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm ata_piix mptspi crct10dif_pclmul drm crct10dif_common crc32c_intel scsi_transport_spi libata serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 23194 Comm: dbserver_d Kdump: loaded Tainted: G             L ------------   3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff95d6c69ae300 ti: ffff95d8eeaf4000 task.ti: ffff95d8eeaf4000
RIP: 0010:[<ffffffffc0487877>]  [<ffffffffc0487877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff95d8eeaf7778  EFLAGS: 00010286
RAX: ffffbb8f40e03818 RBX: 000000000000003c RCX: 00000000000000a2
RDX: 00000000000000a2 RSI: ffff95d8f7527e00 RDI: ffff95d8f4f80000
RBP: ffff95d8eeaf77f8 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: 00000002338de402
R13: 0000000100000400 R14: ffffffff9c1b22c0 R15: ffff95d8eeaf7730
FS:  00007f4151fca700(0000) GS:ffff95d8f66c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffd67851718 CR3: 0000000002c7a000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc0a5dfad>] ? tcp_in_window+0xfd/0xa60 [nf_conntrack]
[<ffffffff9c458f96>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff9c4870aa>] sch_direct_xmit+0x11a/0x250
[<ffffffff9c48726e>] __qdisc_run+0x8e/0x360
[<ffffffff9c45bc98>] __dev_queue_xmit+0x218/0x650
[<ffffffffc0a94020>] ? iptable_nat_ipv4_fn+0x20/0x20 [iptable_nat]
[<ffffffff9c45c0e0>] dev_queue_xmit+0x10/0x20
[<ffffffff9c468e0d>] neigh_resolve_output+0x11d/0x220
[<ffffffff9c4aa86c>] ip_finish_output+0x27c/0x8d0
[<ffffffff9c4ab1eb>] ip_output+0x7b/0xf0
[<ffffffff9c4aa5f0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff9c4a8c17>] ip_local_out_sk+0x37/0x40
[<ffffffff9c4a8fb4>] ip_queue_xmit+0x144/0x3c0
[<ffffffff9c4c3734>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff9c4c594f>] tcp_send_ack+0x11f/0x170
[<ffffffff9c4b9f3b>] __tcp_ack_snd_check+0x6b/0x90
[<ffffffff9c4c01b5>] tcp_rcv_established+0x425/0x7a0
[<ffffffff9c4cb04a>] tcp_v4_do_rcv+0x10a/0x350
[<ffffffff9c43eb4a>] ? sk_wait_data+0xda/0x120
[<ffffffff9c4b3357>] tcp_prequeue_process+0x77/0xc0
[<ffffffff9c4b42b4>] tcp_recvmsg+0x884/0xb30
[<ffffffff9c4e2ff0>] inet_recvmsg+0x80/0xb0
[<ffffffff9c4386ec>] sock_aio_read.part.9+0x14c/0x170
[<ffffffff9c438731>] sock_aio_read+0x21/0x30
[<ffffffff9c04d9c3>] do_sync_read+0x93/0xe0
[<ffffffff9c04e4a5>] vfs_read+0x145/0x170
[<ffffffff9c04f27f>] SyS_read+0x7f/0xf0
[<ffffffff9c599f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #764959
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc01dc3dc
Build id: 3.10.0-1160.11.1.el7.x86_64
0xaec
2
? kfree_skbmem
vmlinux 0xffffffffb483c6f7
Build id: 3.10.0-1160.11.1.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffb4853f96
Build id: 3.10.0-1160.11.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffffb488203a
Build id: 3.10.0-1160.11.1.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffb4856f11
Build id: 3.10.0-1160.11.1.el7.x86_64
0x491
6
dev_queue_xmit
vmlinux 0xffffffffb48570e0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffffb48a507b
Build id: 3.10.0-1160.11.1.el7.x86_64
0x67b
8
ip_output
vmlinux 0xffffffffb48a55fb
Build id: 3.10.0-1160.11.1.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffffb48a4a00
Build id: 3.10.0-1160.11.1.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffffb48a3027
Build id: 3.10.0-1160.11.1.el7.x86_64
0x37
11
ip_queue_xmit
vmlinux 0xffffffffb48a33c4
Build id: 3.10.0-1160.11.1.el7.x86_64
0x144
12
tcp_transmit_skb
vmlinux 0xffffffffb48bdad4
Build id: 3.10.0-1160.11.1.el7.x86_64
0x4e4
13
tcp_send_ack
vmlinux 0xffffffffb48bfcef
Build id: 3.10.0-1160.11.1.el7.x86_64
0x11f
14
__tcp_ack_snd_check
vmlinux 0xffffffffb48b489b
Build id: 3.10.0-1160.11.1.el7.x86_64
0x6b
15
tcp_rcv_established
vmlinux 0xffffffffb48ba3c4
Build id: 3.10.0-1160.11.1.el7.x86_64
0x264
16
tcp_v4_do_rcv
vmlinux 0xffffffffb48c53ea
Build id: 3.10.0-1160.11.1.el7.x86_64
0x10a
17
? sk_wait_data
vmlinux 0xffffffffb4839b4a
Build id: 3.10.0-1160.11.1.el7.x86_64
0xda
18
tcp_prequeue_process
vmlinux 0xffffffffb48ad767
Build id: 3.10.0-1160.11.1.el7.x86_64
0x77
19
tcp_recvmsg
vmlinux 0xffffffffb48ae6c4
Build id: 3.10.0-1160.11.1.el7.x86_64
0x884
20
inet_recvmsg
vmlinux 0xffffffffb48dd370
Build id: 3.10.0-1160.11.1.el7.x86_64
0x80
21
sock_aio_read.part.9
vmlinux 0xffffffffb48336ec
Build id: 3.10.0-1160.11.1.el7.x86_64
0x14c
22
sock_aio_read
vmlinux 0xffffffffb4833731
Build id: 3.10.0-1160.11.1.el7.x86_64
0x21
23
do_sync_read
vmlinux 0xffffffffb444d163
Build id: 3.10.0-1160.11.1.el7.x86_64
0x93
24
vfs_read
vmlinux 0xffffffffb444dc45
Build id: 3.10.0-1160.11.1.el7.x86_64
0x145
25
SyS_read
vmlinux 0xffffffffb444ea1f
Build id: 3.10.0-1160.11.1.el7.x86_64
0x7f
26
system_call_fastpath
vmlinux 0xffffffffb4993f92
Build id: 3.10.0-1160.11.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 31s! [iget:76221]
Modules linked in: nls_utf8 udf crc_itu_t vfat fat uas usb_storage uvcvideo videobuf2_vmalloc videobuf2_memops videobuf2_core videodev xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack vmw_vsock_vmci_transport vsock fuse ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables vmhgfs(OE) iptable_filter sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 crc32_pclmul ghash_clmulni_intel ppdev snd_ens1371 snd_rawmidi
snd_ac97_codec ac97_bus aesni_intel lrw vmw_balloon snd_seq gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm snd_timer pcspkr joydev snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata nfit crct10dif_pclmul crct10dif_common libnvdimm crc32c_intel mptspi scsi_transport_spi mptscsih serio_raw mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 76221 Comm: iget Tainted: G           OEL ------------   3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8ffa0cff2100 ti: ffff8ffc2f170000 task.ti: ffff8ffc2f170000
RIP: 0010:[<ffffffffc01dc3dc>]  [<ffffffffc01dc3dc>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0018:ffff8ffc2f173800  EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffffffffc080dfad RCX: 0000000000000010
RDX: ffffab1240ce6630 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff8ffc2f173890 R08: 7ae11c51b68daed1 R09: 0000000000c84036
R10: e68b65760bef7ae1 R11: ffff8ffab92480f0 R12: 7ae11c51b68daed1
R13: 0000000000c84036 R14: e68b65760bef7ae1 R15: ffff8ffab92480b0
FS:  00007f8492fd7700(0000) GS:ffff8ffc35700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f15fc341000 CR3: 000000022fb38000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb483c6f7>] ? kfree_skbmem+0x37/0x90
[<ffffffffb4853f96>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb488203a>] sch_direct_xmit+0x11a/0x250
[<ffffffffb4856f11>] __dev_queue_xmit+0x491/0x650
[<ffffffffb48570e0>] dev_queue_xmit+0x10/0x20
[<ffffffffb48a507b>] ip_finish_output+0x67b/0x8d0
[<ffffffffb48a55fb>] ip_output+0x7b/0xf0
[<ffffffffb48a4a00>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffb48a3027>] ip_local_out_sk+0x37/0x40
[<ffffffffb48a33c4>] ip_queue_xmit+0x144/0x3c0
[<ffffffffb48bdad4>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffb48bfcef>] tcp_send_ack+0x11f/0x170
[<ffffffffb48b489b>] __tcp_ack_snd_check+0x6b/0x90
[<ffffffffb48ba3c4>] tcp_rcv_established+0x264/0x770
[<ffffffffb48c53ea>] tcp_v4_do_rcv+0x10a/0x350
[<ffffffffb4839b4a>] ? sk_wait_data+0xda/0x120
[<ffffffffb48ad767>] tcp_prequeue_process+0x77/0xc0
[<ffffffffb48ae6c4>] tcp_recvmsg+0x884/0xb30
[<ffffffffb48dd370>] inet_recvmsg+0x80/0xb0
[<ffffffffb48336ec>] sock_aio_read.part.9+0x14c/0x170
[<ffffffffb4833731>] sock_aio_read+0x21/0x30
[<ffffffffb444d163>] do_sync_read+0x93/0xe0
[<ffffffffb444dc45>] vfs_read+0x145/0x170
[<ffffffffb444ea1f>] SyS_read+0x7f/0xf0
[<ffffffffb4993f92>] system_call_fastpath+0x25/0x2a
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41 
Complete report #734051
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0330877
Build id: 3.10.0-1160.71.1.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffff9f2416e7
Build id: 3.10.0-1160.71.1.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffff9f258f86
Build id: 3.10.0-1160.71.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffff9f28709a
Build id: 3.10.0-1160.71.1.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffff9f25bf01
Build id: 3.10.0-1160.71.1.el7.x86_64
0x491
6
? iptable_nat_ipv4_fn
iptable_nat 0xffffffffc090c020
Build id: 3.10.0-1160.71.1.el7.x86_64
0x20
7
dev_queue_xmit
vmlinux 0xffffffff9f25c0d0
Build id: 3.10.0-1160.71.1.el7.x86_64
0x10
8
neigh_resolve_output
vmlinux 0xffffffff9f268dfd
Build id: 3.10.0-1160.71.1.el7.x86_64
0x11d
9
ip_finish_output
vmlinux 0xffffffff9f2aa85c
Build id: 3.10.0-1160.71.1.el7.x86_64
0x27c
10
ip_output
vmlinux 0xffffffff9f2ab1db
Build id: 3.10.0-1160.71.1.el7.x86_64
0x7b
11
? __ip_append_data.isra.50
vmlinux 0xffffffff9f2aa5e0
Build id: 3.10.0-1160.71.1.el7.x86_64
0xa60
12
ip_local_out_sk
vmlinux 0xffffffff9f2a8c07
Build id: 3.10.0-1160.71.1.el7.x86_64
0x37
13
ip_queue_xmit
vmlinux 0xffffffff9f2a8fa4
Build id: 3.10.0-1160.71.1.el7.x86_64
0x144
14
tcp_transmit_skb
vmlinux 0xffffffff9f2c3734
Build id: 3.10.0-1160.71.1.el7.x86_64
0x4e4
15
tcp_send_ack
vmlinux 0xffffffff9f2c594f
Build id: 3.10.0-1160.71.1.el7.x86_64
0x11f
16
__tcp_ack_snd_check
vmlinux 0xffffffff9f2b9f3b
Build id: 3.10.0-1160.71.1.el7.x86_64
0x6b
17
tcp_rcv_established
vmlinux 0xffffffff9f2bfff7
Build id: 3.10.0-1160.71.1.el7.x86_64
0x267
18
tcp_v4_do_rcv
vmlinux 0xffffffff9f2cb04a
Build id: 3.10.0-1160.71.1.el7.x86_64
0x10a
19
? sk_wait_data
vmlinux 0xffffffff9f23eb3a
Build id: 3.10.0-1160.71.1.el7.x86_64
0xda
20
tcp_prequeue_process
vmlinux 0xffffffff9f2b3357
Build id: 3.10.0-1160.71.1.el7.x86_64
0x77
21
tcp_recvmsg
vmlinux 0xffffffff9f2b42b4
Build id: 3.10.0-1160.71.1.el7.x86_64
0x884
22
inet_recvmsg
vmlinux 0xffffffff9f2e2ff0
Build id: 3.10.0-1160.71.1.el7.x86_64
0x80
23
sock_aio_read.part.9
vmlinux 0xffffffff9f2386dc
Build id: 3.10.0-1160.71.1.el7.x86_64
0x14c
24
? __mem_cgroup_commit_charge
vmlinux 0xffffffff9ee3ed52
Build id: 3.10.0-1160.71.1.el7.x86_64
0xe2
25
sock_aio_read
vmlinux 0xffffffff9f238721
Build id: 3.10.0-1160.71.1.el7.x86_64
0x21
26
do_sync_read
vmlinux 0xffffffff9ee4d963
Build id: 3.10.0-1160.71.1.el7.x86_64
0x93
27
vfs_read
vmlinux 0xffffffff9ee4e3e5
Build id: 3.10.0-1160.71.1.el7.x86_64
0x145
28
SyS_read
vmlinux 0xffffffff9ee4f185
Build id: 3.10.0-1160.71.1.el7.x86_64
0x55
29
system_call_fastpath
vmlinux 0xffffffff9f399f92
Build id: 3.10.0-1160.71.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [python3:116925]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 bridge stp llc ebtable_filter ebtables devlink tun ipmi_devintf ipmi_msghandler ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack ip6table_filter ip6_tables iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle vmw_vsock_vmci_transport vsock bnep sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus aesni_intel btusb snd_seq btrtl btbcm btintel lrw gf128mul glue_helper ablk_helper snd_seq_device bluetooth snd_pcm ppdev rfkill cryptd snd_timer snd soundcore joydev pcspkr vmw_balloon sg parport_pc parport i2c_piix4 vmw_vmci sch_fq_codel ip_tables
xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm crc32c_intel ata_piix nfit drm libnvdimm libata serio_raw mptspi scsi_transport_spi mptscsih e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 116925 Comm: python3 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.71.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9a17b4939080 ti: ffff9a16cb544000 task.ti: ffff9a16cb544000
RIP: 0010:[<ffffffffc0330877>]  [<ffffffffc0330877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0000:ffff9a16cb5477d0  EFLAGS: 00010286
RAX: ffffa537c0d03818 RBX: ffff9a16cb547828 RCX: 00000000000000db
RDX: 00000000000000db RSI: ffff9a1831cc0140 RDI: ffff9a183673c000
RBP: ffff9a16cb547850 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: 0000000000000001
R13: 0000000000000000 R14: ffffe63500fbf380 R15: ffff9a17ff1df9c0
FS:  00007fd4727ae740(0000) GS:ffff9a1837640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055f0783a7ebd CR3: 000000006a086000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9f2416e7>] ? kfree_skbmem+0x37/0x90
[<ffffffff9f258f86>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff9f28709a>] sch_direct_xmit+0x11a/0x250
[<ffffffff9f25bf01>] __dev_queue_xmit+0x491/0x650
[<ffffffffc090c020>] ? iptable_nat_ipv4_fn+0x20/0x20 [iptable_nat]
[<ffffffff9f25c0d0>] dev_queue_xmit+0x10/0x20
[<ffffffff9f268dfd>] neigh_resolve_output+0x11d/0x220
[<ffffffff9f2aa85c>] ip_finish_output+0x27c/0x8d0
[<ffffffff9f2ab1db>] ip_output+0x7b/0xf0
[<ffffffff9f2aa5e0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff9f2a8c07>] ip_local_out_sk+0x37/0x40
[<ffffffff9f2a8fa4>] ip_queue_xmit+0x144/0x3c0
[<ffffffff9f2c3734>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffff9f2c594f>] tcp_send_ack+0x11f/0x170
[<ffffffff9f2b9f3b>] __tcp_ack_snd_check+0x6b/0x90
[<ffffffff9f2bfff7>] tcp_rcv_established+0x267/0x7a0
[<ffffffff9f2cb04a>] tcp_v4_do_rcv+0x10a/0x350
[<ffffffff9f23eb3a>] ? sk_wait_data+0xda/0x120
[<ffffffff9f2b3357>] tcp_prequeue_process+0x77/0xc0
[<ffffffff9f2b42b4>] tcp_recvmsg+0x884/0xb30
[<ffffffff9f2e2ff0>] inet_recvmsg+0x80/0xb0
[<ffffffff9f2386dc>] sock_aio_read.part.9+0x14c/0x170
[<ffffffff9ee3ed52>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
[<ffffffff9f238721>] sock_aio_read+0x21/0x30
[<ffffffff9ee4d963>] do_sync_read+0x93/0xe0
[<ffffffff9ee4e3e5>] vfs_read+0x145/0x170
[<ffffffff9ee4f185>] SyS_read+0x55/0xd0
[<ffffffff9f399f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #649500
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0138857
Build id: 3.10.0-957.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffb4c38c66
Build id: 3.10.0-957.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffb4c65cea
Build id: 3.10.0-957.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffb4c3bbe1
Build id: 3.10.0-957.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffffb4c3bdb0
Build id: 3.10.0-957.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffffb4c88736
Build id: 3.10.0-957.el7.x86_64
0x556
7
ip_output
vmlinux 0xffffffffb4c88c93
Build id: 3.10.0-957.el7.x86_64
0x73
8
? __ip_append_data.isra.51
vmlinux 0xffffffffb4c881e0
Build id: 3.10.0-957.el7.x86_64
0xa50
9
ip_local_out_sk
vmlinux 0xffffffffb4c86817
Build id: 3.10.0-957.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffffb4c86bb4
Build id: 3.10.0-957.el7.x86_64
0x144
11
tcp_transmit_skb
vmlinux 0xffffffffb4ca0f64
Build id: 3.10.0-957.el7.x86_64
0x4e4
12
tcp_send_ack
vmlinux 0xffffffffb4ca2fef
Build id: 3.10.0-957.el7.x86_64
0x11f
13
__tcp_ack_snd_check
vmlinux 0xffffffffb4c9796b
Build id: 3.10.0-957.el7.x86_64
0x6b
14
tcp_rcv_established
vmlinux 0xffffffffb4c9d944
Build id: 3.10.0-957.el7.x86_64
0x264
15
tcp_v4_do_rcv
vmlinux 0xffffffffb4ca86da
Build id: 3.10.0-957.el7.x86_64
0x10a
16
? sk_wait_data
vmlinux 0xffffffffb4c1eaaa
Build id: 3.10.0-957.el7.x86_64
0xda
17
tcp_prequeue_process
vmlinux 0xffffffffb4c90da7
Build id: 3.10.0-957.el7.x86_64
0x77
18
tcp_recvmsg
vmlinux 0xffffffffb4c91d04
Build id: 3.10.0-957.el7.x86_64
0x884
19
inet_recvmsg
vmlinux 0xffffffffb4cc0060
Build id: 3.10.0-957.el7.x86_64
0x80
20
sock_aio_read.part.9
vmlinux 0xffffffffb4c186ec
Build id: 3.10.0-957.el7.x86_64
0x14c
21
sock_aio_read
vmlinux 0xffffffffb4c18731
Build id: 3.10.0-957.el7.x86_64
0x21
22
do_sync_read
vmlinux 0xffffffffb48404d3
Build id: 3.10.0-957.el7.x86_64
0x93
23
vfs_read
vmlinux 0xffffffffb4840fb5
Build id: 3.10.0-957.el7.x86_64
0x145
24
SyS_read
vmlinux 0xffffffffb4841dcf
Build id: 3.10.0-957.el7.x86_64
0x7f
25
? system_call_after_swapgs
vmlinux 0xffffffffb4d74d21
Build id: 3.10.0-957.el7.x86_64
0xae
26
system_call_fastpath
vmlinux 0xffffffffb4d74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
27
? system_call_after_swapgs
vmlinux 0xffffffffb4d74d21
Build id: 3.10.0-957.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#17 stuck for 24s! [oracle:49088]
Modules linked in: tcp_lp nls_utf8 isofs fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ppdev sg parport_pc joydev virtio_balloon parport pcspkr i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm floppy ata_piix drm libata virtio_pci serio_raw
virtio_ring e1000 virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 17 PID: 49088 Comm: oracle Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014
task: ffff9ce474626180 ti: ffff9ce29ab64000 task.ti: ffff9ce29ab64000
RIP: 0010:[<ffffffffc0138857>]  [<ffffffffc0138857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9ce29ab67820  EFLAGS: 00000286
RAX: ffffc0c6c6b03818 RBX: 0000000000000001 RCX: 0000000000000007
RDX: 0000000000000007 RSI: ffff9ce47a631100 RDI: ffff9ce48119c000
RBP: ffff9ce29ab678a0 R08: 0000000000000000 R09: 0000000022100000
R10: 2e63a80ae3f80640 R11: 0000000000000000 R12: 0000000000000202
R13: 0000000cfab1f0c2 R14: 000000017a631100 R15: ffffffffb49a04b0
FS:  00007f12661b9740(0000) GS:ffff9ce482a40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1265035b30 CR3: 0000000ee763e000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb4c38c66>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb4c65cea>] sch_direct_xmit+0x11a/0x250
[<ffffffffb4c3bbe1>] __dev_queue_xmit+0x491/0x650
[<ffffffffb4c3bdb0>] dev_queue_xmit+0x10/0x20
[<ffffffffb4c88736>] ip_finish_output+0x556/0x7b0
[<ffffffffb4c88c93>] ip_output+0x73/0xe0
[<ffffffffb4c881e0>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffb4c86817>] ip_local_out_sk+0x37/0x40
[<ffffffffb4c86bb4>] ip_queue_xmit+0x144/0x3c0
[<ffffffffb4ca0f64>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffb4ca2fef>] tcp_send_ack+0x11f/0x170
[<ffffffffb4c9796b>] __tcp_ack_snd_check+0x6b/0x90
[<ffffffffb4c9d944>] tcp_rcv_established+0x264/0x770
[<ffffffffb4ca86da>] tcp_v4_do_rcv+0x10a/0x350
[<ffffffffb4c1eaaa>] ? sk_wait_data+0xda/0x120
[<ffffffffb4c90da7>] tcp_prequeue_process+0x77/0xc0
[<ffffffffb4c91d04>] tcp_recvmsg+0x884/0xb30
[<ffffffffb4cc0060>] inet_recvmsg+0x80/0xb0
[<ffffffffb4c186ec>] sock_aio_read.part.9+0x14c/0x170
[<ffffffffb4c18731>] sock_aio_read+0x21/0x30
[<ffffffffb48404d3>] do_sync_read+0x93/0xe0
[<ffffffffb4840fb5>] vfs_read+0x145/0x170
[<ffffffffb4841dcf>] SyS_read+0x7f/0xf0
[<ffffffffb4d74d21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffb4d74ddb>] system_call_fastpath+0x22/0x27
[<ffffffffb4d74d21>] ? system_call_after_swapgs+0xae/0x146
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #305461
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc042b867
Build id: 3.10.0-862.9.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffaf7ec226
Build id: 3.10.0-862.9.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffaf817aba
Build id: 3.10.0-862.9.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffaf7ef111
Build id: 3.10.0-862.9.1.el7.x86_64
0x4a1
5
dev_queue_xmit
vmlinux 0xffffffffaf7ef2e0
Build id: 3.10.0-862.9.1.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffffaf8384a6
Build id: 3.10.0-862.9.1.el7.x86_64
0x546
7
ip_output
vmlinux 0xffffffffaf838a03
Build id: 3.10.0-862.9.1.el7.x86_64
0x73
8
? __ip_append_data.isra.50
vmlinux 0xffffffffaf837f60
Build id: 3.10.0-862.9.1.el7.x86_64
0xa50
9
ip_local_out_sk
vmlinux 0xffffffffaf8365f7
Build id: 3.10.0-862.9.1.el7.x86_64
0x37
10
ip_queue_xmit
vmlinux 0xffffffffaf836963
Build id: 3.10.0-862.9.1.el7.x86_64
0x143
11
tcp_transmit_skb
vmlinux 0xffffffffaf850844
Build id: 3.10.0-862.9.1.el7.x86_64
0x4e4
12
tcp_send_ack
vmlinux 0xffffffffaf8528bf
Build id: 3.10.0-862.9.1.el7.x86_64
0x11f
13
__tcp_ack_snd_check
vmlinux 0xffffffffaf84709b
Build id: 3.10.0-862.9.1.el7.x86_64
0x6b
14
tcp_rcv_established
vmlinux 0xffffffffaf84d3e7
Build id: 3.10.0-862.9.1.el7.x86_64
0x417
15
tcp_v4_do_rcv
vmlinux 0xffffffffaf857e6a
Build id: 3.10.0-862.9.1.el7.x86_64
0x10a
16
? sk_wait_data
vmlinux 0xffffffffaf7d284a
Build id: 3.10.0-862.9.1.el7.x86_64
0xda
17
tcp_prequeue_process
vmlinux 0xffffffffaf840927
Build id: 3.10.0-862.9.1.el7.x86_64
0x77
18
tcp_recvmsg
vmlinux 0xffffffffaf841884
Build id: 3.10.0-862.9.1.el7.x86_64
0x884
19
inet_recvmsg
vmlinux 0xffffffffaf86f7f0
Build id: 3.10.0-862.9.1.el7.x86_64
0x80
20
sock_aio_read.part.10
vmlinux 0xffffffffaf7cc9bc
Build id: 3.10.0-862.9.1.el7.x86_64
0x14c
21
sock_aio_read
vmlinux 0xffffffffaf7cca01
Build id: 3.10.0-862.9.1.el7.x86_64
0x21
22
do_sync_read
vmlinux 0xffffffffaf41a8d3
Build id: 3.10.0-862.9.1.el7.x86_64
0x93
23
vfs_read
vmlinux 0xffffffffaf41b3a5
Build id: 3.10.0-862.9.1.el7.x86_64
0x145
24
SyS_read
vmlinux 0xffffffffaf41c1cf
Build id: 3.10.0-862.9.1.el7.x86_64
0x7f
25
system_call_fastpath
vmlinux 0xffffffffaf920795
Build id: 3.10.0-862.9.1.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [oracle_36383_pd:36383]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 aesni_intel snd_rawmidi snd_ac97_codec lrw gf128mul glue_helper ablk_helper ac97_bus snd_seq ppdev cryptd vmw_balloon snd_seq_device snd_pcm snd_timer snd pcspkr joydev soundcore shpchp sg vmw_vmci nfit libnvdimm i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom
crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm serio_raw e1000 ata_piix mptspi scsi_transport_spi mptscsih mptbase libata i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 36383 Comm: oracle_36383_pd Kdump: loaded Tainted: G        W    L ------------   3.10.0-862.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff900cd2ad0000 ti: ffff900cd2ac4000 task.ti: ffff900cd2ac4000
RIP: 0010:[<ffffffffc042b867>]  [<ffffffffc042b867>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff900cd2ac7820  EFLAGS: 00000286
RAX: ffff9d6d03503818 RBX: 0000000000000001 RCX: 00000000000000c9
RDX: 00000000000000c9 RSI: ffff900cdfda31c0 RDI: ffff901137438000
RBP: ffff900cd2ac78a0 R08: 0000000000000100 R09: 0000000022100000
R10: 12bd280000450008 R11: 0000000000000000 R12: 0000000000000202
R13: 00000006f44800da R14: 0000000139cb1098 R15: ffffffffaf5748b0
FS:  00007fcbc2c20280(0000) GS:ffff901138040000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc6bf73a590 CR3: 0000000292b8c000 CR4: 00000000001607e0
Call Trace:
[<ffffffffaf7ec226>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffaf817aba>] sch_direct_xmit+0x11a/0x250
[<ffffffffaf7ef111>] __dev_queue_xmit+0x4a1/0x660
[<ffffffffaf7ef2e0>] dev_queue_xmit+0x10/0x20
[<ffffffffaf8384a6>] ip_finish_output+0x546/0x7a0
[<ffffffffaf838a03>] ip_output+0x73/0xe0
[<ffffffffaf837f60>] ? __ip_append_data.isra.50+0xa50/0xa50
[<ffffffffaf8365f7>] ip_local_out_sk+0x37/0x40
[<ffffffffaf836963>] ip_queue_xmit+0x143/0x3a0
[<ffffffffaf850844>] tcp_transmit_skb+0x4e4/0x9e0
[<ffffffffaf8528bf>] tcp_send_ack+0x11f/0x170
[<ffffffffaf84709b>] __tcp_ack_snd_check+0x6b/0xa0
[<ffffffffaf84d3e7>] tcp_rcv_established+0x417/0x770
[<ffffffffaf857e6a>] tcp_v4_do_rcv+0x10a/0x350
[<ffffffffaf7d284a>] ? sk_wait_data+0xda/0x120
[<ffffffffaf840927>] tcp_prequeue_process+0x77/0xc0
[<ffffffffaf841884>] tcp_recvmsg+0x884/0xb30
[<ffffffffaf86f7f0>] inet_recvmsg+0x80/0xb0
[<ffffffffaf7cc9bc>] sock_aio_read.part.10+0x14c/0x170
[<ffffffffaf7cca01>] sock_aio_read+0x21/0x30
[<ffffffffaf41a8d3>] do_sync_read+0x93/0xe0
[<ffffffffaf41b3a5>] vfs_read+0x145/0x170
[<ffffffffaf41c1cf>] SyS_read+0x7f/0xf0
[<ffffffffaf920795>] system_call_fastpath+0x1c/0x21
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #183608
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffa00bf63a
Build id: 3.10.0-327.el7.x86_64
0xaca
2
? ipt_do_table
ip_tables 0xffffffffa031e137
Build id: 3.10.0-327.el7.x86_64
0x337
3
dev_hard_start_xmit
vmlinux 0xffffffff8152c671
Build id: 3.10.0-327.el7.x86_64
0x171
4
sch_direct_xmit
vmlinux 0xffffffff8154cd74
Build id: 3.10.0-327.el7.x86_64
0x104
5
dev_queue_xmit
vmlinux 0xffffffff8152cae6
Build id: 3.10.0-327.el7.x86_64
0x236
6
ip_finish_output
vmlinux 0xffffffff8156ae1d
Build id: 3.10.0-327.el7.x86_64
0x53d
7
ip_output
vmlinux 0xffffffff8156bdcf
Build id: 3.10.0-327.el7.x86_64
0x6f
8
? ip_fragment
vmlinux 0xffffffff8156a8e0
Build id: 3.10.0-327.el7.x86_64
0x8b0
9
ip_local_out_sk
vmlinux 0xffffffff81569a41
Build id: 3.10.0-327.el7.x86_64
0x31
10
ip_queue_xmit
vmlinux 0xffffffff81569da1
Build id: 3.10.0-327.el7.x86_64
0x141
11
tcp_transmit_skb
vmlinux 0xffffffff815833a4
Build id: 3.10.0-327.el7.x86_64
0x494
12
tcp_send_ack
vmlinux 0xffffffff815853dc
Build id: 3.10.0-327.el7.x86_64
0x11c
13
__tcp_ack_snd_check
vmlinux 0xffffffff8157a7eb
Build id: 3.10.0-327.el7.x86_64
0x6b
14
tcp_rcv_established
vmlinux 0xffffffff8157fe64
Build id: 3.10.0-327.el7.x86_64
0x264
15
tcp_v4_do_rcv
vmlinux 0xffffffff8158a6ea
Build id: 3.10.0-327.el7.x86_64
0x10a
16
? sk_wait_data
vmlinux 0xffffffff815155da
Build id: 3.10.0-327.el7.x86_64
0xda
17
tcp_prequeue_process
vmlinux 0xffffffff81573931
Build id: 3.10.0-327.el7.x86_64
0x71
18
tcp_recvmsg
vmlinux 0xffffffff81574ae9
Build id: 3.10.0-327.el7.x86_64
0x919
19
inet_recvmsg
vmlinux 0xffffffff815a10eb
Build id: 3.10.0-327.el7.x86_64
0x7b
20
sock_aio_read.part.7
vmlinux 0xffffffff8150ffb6
Build id: 3.10.0-327.el7.x86_64
0x146
21
sock_aio_read
vmlinux 0xffffffff8150fff1
Build id: 3.10.0-327.el7.x86_64
0x21
22
do_sync_read
vmlinux 0xffffffff811ddcdd
Build id: 3.10.0-327.el7.x86_64
0x8d
23
vfs_read
vmlinux 0xffffffff811de4e5
Build id: 3.10.0-327.el7.x86_64
0x145
24
SyS_read
vmlinux 0xffffffff811def8f
Build id: 3.10.0-327.el7.x86_64
0x7f
25
system_call_fastpath
vmlinux 0xffffffff81645909
Build id: 3.10.0-327.el7.x86_64
0x16
BUG: soft lockup - CPU#5 stuck for 35s! [oracle_130483_p:130483]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event ppdev vmw_balloon snd_ens1371 crc32_pclmul snd_rawmidi snd_ac97_codec ghash_clmulni_intel ac97_bus snd_seq aesni_intel lrw snd_seq_device gf128mul glue_helper snd_pcm ablk_helper snd_timer snd cryptd soundcore sg pcspkr parport_pc parport vmw_vmci i2c_piix4 shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi
vmwgfx drm_kms_helper ttm crct10dif_pclmul drm crct10dif_common e1000 ata_piix crc32c_intel serio_raw mptspi scsi_transport_spi mptscsih libata mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 130483 Comm: oracle_130483_p Tainted: G             L ------------   3.10.0-327.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff880108a62e00 ti: ffff88013a460000 task.ti: ffff88013a460000
RIP: 0010:[<ffffffffa00bf63a>]  [<ffffffffa00bf63a>] e1000_xmit_frame+0xaca/0x10b0 [e1000]
RSP: 0018:ffff88013a463878  EFLAGS: 00000286
RAX: ffffc9000b443818 RBX: 00000001f41e94c2 RCX: ffff88061ccfc000
RDX: 0000000000000011 RSI: 00000000ffffffff RDI: ffff88061ccfc000
RBP: ffff88013a4638f8 R08: 0000000000000200 R09: 0000000022100000
R10: 9401a8c061100640 R11: 0000000000000000 R12: ffffffff81313f63
R13: ffff88013a463840 R14: 0000000000000001 R15: 0000000000000202
FS:  00007ff935bf5f40(0000) GS:ffff8806266a0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc66820b000 CR3: 0000000178770000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffffffffa031e137 ffff88061b922098 0000000fa031e137 0000000900000000
0000000000000000 ffffff0000000000 0000000100001000 ffff88017743c800
ffff880035d9e180 ffff88061ccfc000 ffff88061b922000 ffff88061ccfc000
Call Trace:
[<ffffffffa031e137>] ? ipt_do_table+0x337/0x701 [ip_tables]
[<ffffffff8152c671>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff8154cd74>] sch_direct_xmit+0x104/0x200
[<ffffffff8152cae6>] dev_queue_xmit+0x236/0x570
[<ffffffff8156ae1d>] ip_finish_output+0x53d/0x7d0
[<ffffffff8156bdcf>] ip_output+0x6f/0xe0
[<ffffffff8156a8e0>] ? ip_fragment+0x8b0/0x8b0
[<ffffffff81569a41>] ip_local_out_sk+0x31/0x40
[<ffffffff81569da1>] ip_queue_xmit+0x141/0x3d0
[<ffffffff815833a4>] tcp_transmit_skb+0x494/0x980
[<ffffffff815853dc>] tcp_send_ack+0x11c/0x170
[<ffffffff8157a7eb>] __tcp_ack_snd_check+0x6b/0xa0
[<ffffffff8157fe64>] tcp_rcv_established+0x264/0x750
[<ffffffff8158a6ea>] tcp_v4_do_rcv+0x10a/0x340
[<ffffffff815155da>] ? sk_wait_data+0xda/0x120
[<ffffffff81573931>] tcp_prequeue_process+0x71/0xb0
[<ffffffff81574ae9>] tcp_recvmsg+0x919/0xbc0
[<ffffffff815a10eb>] inet_recvmsg+0x7b/0xa0
[<ffffffff8150ffb6>] sock_aio_read.part.7+0x146/0x160
[<ffffffff8150fff1>] sock_aio_read+0x21/0x30
[<ffffffff811ddcdd>] do_sync_read+0x8d/0xd0
[<ffffffff811de4e5>] vfs_read+0x145/0x170
[<ffffffff811def8f>] SyS_read+0x7f/0xe0
[<ffffffff81645909>] system_call_fastpath+0x16/0x1b
Code: 14 48 8b 45 c8 48 8b 80 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 4d c8 41 0f b7 47 2a 41 8b 57 18 48 03 81 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 44 00 
Complete report #14066
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffa00add57
Build id: 3.10.0-514.el7.x86_64
0x957
2
? ipt_do_table
ip_tables 0xffffffffa036cd07
Build id: 3.10.0-514.el7.x86_64
0x337
3
dev_hard_start_xmit
vmlinux 0xffffffff8156edc1
Build id: 3.10.0-514.el7.x86_64
0x171
4
sch_direct_xmit
vmlinux 0xffffffff81596424
Build id: 3.10.0-514.el7.x86_64
0x104
5
__dev_queue_xmit
vmlinux 0xffffffff815719bc
Build id: 3.10.0-514.el7.x86_64
0x23c
6
dev_queue_xmit
vmlinux 0xffffffff81571d00
Build id: 3.10.0-514.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffff815b5716
Build id: 3.10.0-514.el7.x86_64
0x466
8
ip_output
vmlinux 0xffffffff815b6713
Build id: 3.10.0-514.el7.x86_64
0x73
9
? ip_fragment.constprop.54
vmlinux 0xffffffff815b52b0
Build id: 3.10.0-514.el7.x86_64
0x90
10
ip_local_out_sk
vmlinux 0xffffffff815b43d1
Build id: 3.10.0-514.el7.x86_64
0x31
11
ip_queue_xmit
vmlinux 0xffffffff815b4743
Build id: 3.10.0-514.el7.x86_64
0x143
12
tcp_transmit_skb
vmlinux 0xffffffff815cdeaf
Build id: 3.10.0-514.el7.x86_64
0x4af
13
tcp_send_ack
vmlinux 0xffffffff815cfecc
Build id: 3.10.0-514.el7.x86_64
0x11c
14
__tcp_ack_snd_check
vmlinux 0xffffffff815c52db
Build id: 3.10.0-514.el7.x86_64
0x6b
15
tcp_rcv_established
vmlinux 0xffffffff815cab17
Build id: 3.10.0-514.el7.x86_64
0x417
16
tcp_v4_do_rcv
vmlinux 0xffffffff815d53fa
Build id: 3.10.0-514.el7.x86_64
0x10a
17
? sk_wait_data
vmlinux 0xffffffff8155985a
Build id: 3.10.0-514.el7.x86_64
0xda
18
tcp_prequeue_process
vmlinux 0xffffffff815be301
Build id: 3.10.0-514.el7.x86_64
0x71
19
tcp_recvmsg
vmlinux 0xffffffff815bf47c
Build id: 3.10.0-514.el7.x86_64
0x8ac
20
inet_recvmsg
vmlinux 0xffffffff815ec86b
Build id: 3.10.0-514.el7.x86_64
0x7b
21
sock_aio_read.part.7
vmlinux 0xffffffff81554576
Build id: 3.10.0-514.el7.x86_64
0x146
22
sock_aio_read
vmlinux 0xffffffff815545b1
Build id: 3.10.0-514.el7.x86_64
0x21
23
do_sync_read
vmlinux 0xffffffff811fd7ed
Build id: 3.10.0-514.el7.x86_64
0x8d
24
vfs_read
vmlinux 0xffffffff811fe04d
Build id: 3.10.0-514.el7.x86_64
0x14d
25
SyS_read
vmlinux 0xffffffff811feb6f
Build id: 3.10.0-514.el7.x86_64
0x7f
26
system_call_fastpath
vmlinux 0xffffffff816965c9
Build id: 3.10.0-514.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [HikariPool-1 co:12043]
Modules linked in: xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle overlay ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc vmw_vsock_vmci_transport vsock sunrpc nfit libnvdimm snd_seq_midi snd_seq_midi_event ppdev vmw_balloon intel_powerclamp snd_ens1371 snd_rawmidi snd_ac97_codec coretemp ac97_bus iosf_mbi snd_seq crc32_pclmul ghash_clmulni_intel aesni_intel lrw sg snd_seq_device gf128mul snd_pcm snd_timer snd soundcore glue_helper ablk_helper cryptd pcspkr shpchp i2c_piix4 parport_pc parport vmw_vmci ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic
pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel mptspi vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops scsi_transport_spi mptscsih ttm ata_piix mptbase drm e1000 libata i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 12043 Comm: HikariPool-1 co Tainted: G        W    L ------------   3.10.0-514.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8801e4b01f60 ti: ffff88021b7bc000 task.ti: ffff88021b7bc000
RIP: 0010:[<ffffffffa00add57>]  [<ffffffffa00add57>] e1000_xmit_frame+0x957/0x1040 [e1000]
RSP: 0018:ffff88021b7bf860  EFLAGS: 00000286
RAX: ffffc90000cc3818 RBX: ffffffff81096702 RCX: 0000000000000054
RDX: 0000000000000054 RSI: ffff880230e2c680 RDI: ffff880235450000
RBP: ffff88021b7bf8e0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff880184aac000
R13: ffff8801e469b010 R14: ffff880184aac000 R15: ffff88021b7bf810
FS:  00007fc90f9da700(0000) GS:ffff880235640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc9b7c3bdb8 CR3: 000000021ae58000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffffa036cd07 ffff880235f6c098 00000052a036cd07 0000000900000000
000000001b7bf900 ffffffff00000000 0000000000000000 0000000100001000
ffff8801c3a95100 ffff880235450000 ffff880235f6c000 ffff880235450000
Call Trace:
[<ffffffffa036cd07>] ? ipt_do_table+0x337/0x710 [ip_tables]
[<ffffffff8156edc1>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff81596424>] sch_direct_xmit+0x104/0x200
[<ffffffff815719bc>] __dev_queue_xmit+0x23c/0x570
[<ffffffff81571d00>] dev_queue_xmit+0x10/0x20
[<ffffffff815b5716>] ip_finish_output+0x466/0x750
[<ffffffff815b6713>] ip_output+0x73/0xe0
[<ffffffff815b52b0>] ? ip_fragment.constprop.54+0x90/0x90
[<ffffffff815b43d1>] ip_local_out_sk+0x31/0x40
[<ffffffff815b4743>] ip_queue_xmit+0x143/0x3a0
[<ffffffff815cdeaf>] tcp_transmit_skb+0x4af/0x990
[<ffffffff815cfecc>] tcp_send_ack+0x11c/0x170
[<ffffffff815c52db>] __tcp_ack_snd_check+0x6b/0xa0
[<ffffffff815cab17>] tcp_rcv_established+0x417/0x760
[<ffffffff815d53fa>] tcp_v4_do_rcv+0x10a/0x340
[<ffffffff8155985a>] ? sk_wait_data+0xda/0x120
[<ffffffff815be301>] tcp_prequeue_process+0x71/0xb0
[<ffffffff815bf47c>] tcp_recvmsg+0x8ac/0xb50
[<ffffffff815ec86b>] inet_recvmsg+0x7b/0xa0
[<ffffffff81554576>] sock_aio_read.part.7+0x146/0x160
[<ffffffff815545b1>] sock_aio_read+0x21/0x30
[<ffffffff811fd7ed>] do_sync_read+0x8d/0xd0
[<ffffffff811fe04d>] vfs_read+0x14d/0x170
[<ffffffff811feb6f>] SyS_read+0x7f/0xe0
[<ffffffff816965c9>] system_call_fastpath+0x16/0x1b
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00