Info

Permalink
Function
e1000_xmit_frame
First occurrence
2020-09-22
Last occurrence
2026-09-15
State
NEW
Components
kernel
Quality
-75

Statistics

Operating system Count
CentOS 7 247
Architecture Count
x86_64 247
Related packages Count
kernel 247
 0:3.10.0-1160.el7 66
 0:3.10.0-1062.el7 21
 0:3.10.0-1160.41.1.el7 15
 0:3.10.0-957.el7 12
 0:3.10.0-1160.119.1.el7 10
 0:3.10.0-1160.71.1.el7 10
 0:3.10.0-1160.80.1.el7 10
 0:3.10.0-1160.59.1.el7 9
 0:3.10.0-1127.el7 5
 0:3.10.0-862.el7 5
 0:3.10.0-1160.102.1.el7 5
 0:3.10.0-1127.19.1.el7 4
 0:3.10.0-1160.25.1.el7 4
 0:3.10.0-1160.45.1.el7 4
 0:3.10.0-1160.92.1.el7 4
 0:3.10.0-957.12.1.el7 4
 0:3.10.0-1062.9.1.el7 4
 0:3.10.0-1160.90.1.el7 4
 0:3.10.0-1160.114.2.el7 3
 0:3.10.0-1160.62.1.el7 3
 0:3.10.0-1160.76.1.el7 3
 0:3.10.0-1160.88.1.el7 3
 0:3.10.0-1127.18.2.el7 3
 0:3.10.0-1160.118.1.el7 3
 0:3.10.0-1160.66.1.el7 3
 0:3.10.0-957.1.3.el7 2
 0:3.10.0-1160.31.1.el7 2
 0:3.10.0-1160.49.1.el7 2
 0:3.10.0-1160.6.1.el7 2
 0:3.10.0-1160.105.1.el7 2
 0:3.10.0-1160.108.1.el7 2
 0:3.10.0-1160.11.1.el7 2
 0:3.10.0-693.el7 2
 0:3.10.0-1127.10.1.el7 2
 0:3.10.0-1160.24.1.el7 2
 0:3.10.0-957.5.1.el7 2
 0:3.10.0-1062.1.2.el7 1
 0:3.10.0-1160.95.1.el7 1
 0:3.10.0-957.10.1.el7 1
 0:3.10.0-957.27.2.el7 1
 0:3.10.0-1160.15.2.el7 1
 0:3.10.0-1160.81.1.el7 1
 0:3.10.0-1062.18.1.el7 1
 0:3.10.0-957.21.3.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1265440
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc03d544c
Build id: 3.10.0-1160.119.1.el7.x86_64 		0xaec
2
? kfree_skbmem
vmlinux 0xffffffff89a66c7f
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x3f
3
dev_hard_start_xmit
vmlinux 0xffffffff89a7edc6
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffff89aadd72
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x122
5
__dev_queue_xmit
vmlinux 0xffffffff89a81dd1
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x491
6
? local_bh_enable
vmlinux 0xffffffff894a89b7
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x17
7
dev_queue_xmit
vmlinux 0xffffffff89a81fa0
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x10
8
neigh_resolve_output
vmlinux 0xffffffff89a8f03d
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x11d
9
ip_finish_output
vmlinux 0xffffffff89ad220c
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x27c
10
ip_output
vmlinux 0xffffffff89ad2b9b
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x7b
11
? __ip_append_data.isra.50
vmlinux 0xffffffff89ad1f90
Build id: 3.10.0-1160.119.1.el7.x86_64 		0xa60
12
ip_local_out_sk
vmlinux 0xffffffff89ad0587
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x37
13
ip_send_skb
vmlinux 0xffffffff89ad3636
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x16
14
udp_send_skb
vmlinux 0xffffffff89afc3ac
Build id: 3.10.0-1160.119.1.el7.x86_64 		0xac
15
? ip_copy_metadata
vmlinux 0xffffffff89ad02b0
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x170
16
udp_sendmsg
vmlinux 0xffffffff89afd9a1
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x5e1
17
? __pollwait
vmlinux 0xffffffff89672550
Build id: 3.10.0-1160.119.1.el7.x86_64 		0xf0
18
? sock_has_perm
vmlinux 0xffffffff8971f7d5
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x75
19
inet_sendmsg
vmlinux 0xffffffff89b0b4e9
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x69
20
? selinux_socket_sendmsg
vmlinux 0xffffffff8971f913
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x23
21
sock_sendmsg
vmlinux 0xffffffff89a5e4f6
Build id: 3.10.0-1160.119.1.el7.x86_64 		0xb6
22
? sidtab_find_context
vmlinux 0xffffffff8972defe
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x19e
23
___sys_sendmsg
vmlinux 0xffffffff89a5f3f9
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x3e9
24
? __ip_route_output_key_hash
vmlinux 0xffffffff89ac8c63
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x223
25
? ip4_datagram_release_cb
vmlinux 0xffffffff89af9cdf
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x6f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff89bba56e
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x1e
27
? udp_lib_rehash
vmlinux 0xffffffff89afcecf
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x11f
28
? _raw_spin_unlock_bh
vmlinux 0xffffffff89bba56e
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x1e
29
? release_sock
vmlinux 0xffffffff89a63ea8
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x128
30
? ip4_datagram_connect
vmlinux 0xffffffff89afa147
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x37
31
? inet_dgram_connect
vmlinux 0xffffffff89b0a0e4
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x34
32
__sys_sendmsg
vmlinux 0xffffffff89a60b41
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x51
33
SyS_sendmsg
vmlinux 0xffffffff89a60b92
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x12
34
system_call_fastpath
vmlinux 0xffffffff89bc539a
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 43s! [chronyd:702]
Modules linked in: xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_addrtype iptable_filter br_netfilter bridge stp llc nfsv3 nfs_acl rpcsec_gss_krb5 nfsv4 dns_resolver nfs lockd grace fscache overlay(T) bochs_drm ttm drm_kms_helper syscopyarea sysfillrect sysimgblt edac_mce_amd fb_sys_fops drm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr floppy drm_panel_orientation_quirks virtio_balloon i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common sg ata_generic pata_acpi virtio_console ata_piix libata crc32c_intel serio_raw vmxnet3 virtio_pci vmw_pvscsi virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 702 Comm: chronyd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
task: ffff9aac764b2100 ti: ffff9aae7514c000 task.ti: ffff9aae7514c000
RIP: 0010:[<ffffffffc03d544c>]  [<ffffffffc03d544c>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0018:ffff9aae7514f700  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffff9aac75428c28 RCX: 000000000000008b
RDX: ffffbb8540e88608 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff9aae7514f790 R08: 000000000000008a R09: 0000000000a0005a
R10: ffff9aae7fc1f120 R11: ffff9aacfb3068a0 R12: ffff9aae76229700
R13: ffffdb85fd010016 R14: ffffffff89b120cb R15: ffff9aae7514f6e8
FS:  00007fd4da69a740(0000) GS:ffff9aae7fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f83ba74da98 CR3: 0000000235150000 CR4: 00000000000007f0
Call Trace:
[<ffffffff89a66c7f>] ? kfree_skbmem+0x3f/0xa0
[<ffffffff89a7edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff89aadd72>] sch_direct_xmit+0x122/0x250
[<ffffffff89a81dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffff894a89b7>] ? local_bh_enable+0x17/0x20
[<ffffffff89a81fa0>] dev_queue_xmit+0x10/0x20
[<ffffffff89a8f03d>] neigh_resolve_output+0x11d/0x220
[<ffffffff89ad220c>] ip_finish_output+0x27c/0x8d0
[<ffffffff89ad2b9b>] ip_output+0x7b/0x100
[<ffffffff89ad1f90>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff89ad0587>] ip_local_out_sk+0x37/0x40
[<ffffffff89ad3636>] ip_send_skb+0x16/0x50
[<ffffffff89afc3ac>] udp_send_skb+0xac/0x2b0
[<ffffffff89ad02b0>] ? ip_copy_metadata+0x170/0x170
[<ffffffff89afd9a1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff89672550>] ? __pollwait+0xf0/0xf0
[<ffffffff8971f7d5>] ? sock_has_perm+0x75/0xa0
[<ffffffff89b0b4e9>] inet_sendmsg+0x69/0xb0
[<ffffffff8971f913>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff89a5e4f6>] sock_sendmsg+0xb6/0x100
[<ffffffff8972defe>] ? sidtab_find_context+0x19e/0x1b0
[<ffffffff89a5f3f9>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff89ac8c63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffff89af9cdf>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff89bba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff89afcecf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff89bba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff89a63ea8>] ? release_sock+0x128/0x180
[<ffffffff89afa147>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff89b0a0e4>] ? inet_dgram_connect+0x34/0x90
[<ffffffff89a60b41>] __sys_sendmsg+0x51/0x90
[<ffffffff89a60b92>] SyS_sendmsg+0x12/0x30
[<ffffffff89bc539a>] system_call_fastpath+0x25/0x2a
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41
Complete report #924906
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc01038e7
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff8667edc6
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffff866add72
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x122
4
__qdisc_run
vmlinux 0xffffffff866adf2e
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x8e
5
__dev_queue_xmit
vmlinux 0xffffffff86681b58
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x218
6
dev_queue_xmit
vmlinux 0xffffffff86681fa0
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x10
7
ip_finish_output
vmlinux 0xffffffff866d260b
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x67b
8
ip_output
vmlinux 0xffffffff866d2b9b
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffff866d1f90
Build id: 3.10.0-1160.88.1.el7.x86_64 		0xa60
10
ip_local_out_sk
vmlinux 0xffffffff866d0587
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x37
11
ip_send_skb
vmlinux 0xffffffff866d3636
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffff866fc3ac
Build id: 3.10.0-1160.88.1.el7.x86_64 		0xac
13
? ip_copy_metadata
vmlinux 0xffffffff866d02b0
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffff866fd9a1
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x5e1
15
? __pollwait
vmlinux 0xffffffff86272490
Build id: 3.10.0-1160.88.1.el7.x86_64 		0xf0
16
? sock_has_perm
vmlinux 0xffffffff8631f635
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x75
17
inet_sendmsg
vmlinux 0xffffffff8670b4e9
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x69
18
? selinux_socket_sendmsg
vmlinux 0xffffffff8631f773
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x23
19
sock_sendmsg
vmlinux 0xffffffff8665e4f6
Build id: 3.10.0-1160.88.1.el7.x86_64 		0xb6
20
? __pollwait
vmlinux 0xffffffff86272490
Build id: 3.10.0-1160.88.1.el7.x86_64 		0xf0
21
? sidtab_find_context
vmlinux 0xffffffff8632dd5e
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x19e
22
___sys_sendmsg
vmlinux 0xffffffff8665f3f9
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x3e9
23
? __ip_route_output_key_hash
vmlinux 0xffffffff866c8c63
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x223
24
? ip4_datagram_release_cb
vmlinux 0xffffffff866f9cdf
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffff867ba56e
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x1e
26
? udp_lib_rehash
vmlinux 0xffffffff866fcecf
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffff867ba56e
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x1e
28
? release_sock
vmlinux 0xffffffff86663ea8
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x128
29
? ip4_datagram_connect
vmlinux 0xffffffff866fa147
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x37
30
? inet_dgram_connect
vmlinux 0xffffffff8670a0e4
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x34
31
__sys_sendmsg
vmlinux 0xffffffff86660b41
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x51
32
SyS_sendmsg
vmlinux 0xffffffff86660b92
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x12
33
system_call_fastpath
vmlinux 0xffffffff867c539a
Build id: 3.10.0-1160.88.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 27s! [chronyd:747]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter binfmt_misc vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq iosf_mbi snd_seq_device crc32_pclmul ghash_clmulni_intel snd_pcm aesni_intel lrw snd_timer gf128mul snd glue_helper soundcore ablk_helper cryptd sg ppdev pcspkr joydev vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod sr_mod
cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper ata_piix syscopyarea libata sysfillrect sysimgblt fb_sys_fops ttm nfit drm libnvdimm e1000 mptspi crct10dif_pclmul crct10dif_common scsi_transport_spi mptscsih crc32c_intel mptbase serio_raw drm_panel_orientation_quirks fuse
CPU: 7 PID: 747 Comm: chronyd Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9b67ec909080 ti: ffff9b67ebfdc000 task.ti: ffff9b67ebfdc000
RIP: 0010:[<ffffffffc01038e7>]  [<ffffffffc01038e7>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9b67ebfdf6f8  EFLAGS: 00000286
RAX: ffffb9da41c03818 RBX: ffff9b67ebfdf698 RCX: 0000000000000017
RDX: 0000000000000063 RSI: 0000000000000063 RDI: ffff9b67e7f89600
RBP: ffff9b67ebfdf778 R08: 0000000000000000 R09: 0000000002000000
R10: 0000000080000000 R11: 0000000000000000 R12: 0000000000000246
R13: 00000001ebfdf6e8 R14: ffffffff863c8fd0 R15: ffff9b67ebfdf6b0
FS:  00007fbf74c44740(0000) GS:ffff9b67ed7c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f16bc01e4f0 CR3: 000000042bfd6000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff8667edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff866add72>] sch_direct_xmit+0x122/0x250
[<ffffffff866adf2e>] __qdisc_run+0x8e/0x360
[<ffffffff86681b58>] __dev_queue_xmit+0x218/0x650
[<ffffffff86681fa0>] dev_queue_xmit+0x10/0x20
[<ffffffff866d260b>] ip_finish_output+0x67b/0x8d0
[<ffffffff866d2b9b>] ip_output+0x7b/0x100
[<ffffffff866d1f90>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff866d0587>] ip_local_out_sk+0x37/0x40
[<ffffffff866d3636>] ip_send_skb+0x16/0x50
[<ffffffff866fc3ac>] udp_send_skb+0xac/0x2b0
[<ffffffff866d02b0>] ? ip_copy_metadata+0x170/0x170
[<ffffffff866fd9a1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff86272490>] ? __pollwait+0xf0/0xf0
[<ffffffff8631f635>] ? sock_has_perm+0x75/0xa0
[<ffffffff8670b4e9>] inet_sendmsg+0x69/0xb0
[<ffffffff8631f773>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff8665e4f6>] sock_sendmsg+0xb6/0x100
[<ffffffff86272490>] ? __pollwait+0xf0/0xf0
[<ffffffff8632dd5e>] ? sidtab_find_context+0x19e/0x1b0
[<ffffffff8665f3f9>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff866c8c63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffff866f9cdf>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff867ba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff866fcecf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff867ba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff86663ea8>] ? release_sock+0x128/0x180
[<ffffffff866fa147>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff8670a0e4>] ? inet_dgram_connect+0x34/0x90
[<ffffffff86660b41>] __sys_sendmsg+0x51/0x90
[<ffffffff86660b92>] SyS_sendmsg+0x12/0x30
[<ffffffff867c539a>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc
Complete report #784774
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc04243dc
Build id: 3.10.0-1160.el7.x86_64 		0xaec
2
dev_hard_start_xmit
vmlinux 0xffffffff92e54386
Build id: 3.10.0-1160.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffff92e8242a
Build id: 3.10.0-1160.el7.x86_64 		0x11a
4
__qdisc_run
vmlinux 0xffffffff92e825ee
Build id: 3.10.0-1160.el7.x86_64 		0x8e
5
__dev_queue_xmit
vmlinux 0xffffffff92e57319
Build id: 3.10.0-1160.el7.x86_64 		0x4a9
6
dev_queue_xmit
vmlinux 0xffffffff92e574d0
Build id: 3.10.0-1160.el7.x86_64 		0x10
7
ip_finish_output
vmlinux 0xffffffff92ea545b
Build id: 3.10.0-1160.el7.x86_64 		0x67b
8
ip_output
vmlinux 0xffffffff92ea59db
Build id: 3.10.0-1160.el7.x86_64 		0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffff92ea4de0
Build id: 3.10.0-1160.el7.x86_64 		0xa60
10
ip_local_out_sk
vmlinux 0xffffffff92ea3407
Build id: 3.10.0-1160.el7.x86_64 		0x37
11
ip_send_skb
vmlinux 0xffffffff92ea6446
Build id: 3.10.0-1160.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffff92ece85c
Build id: 3.10.0-1160.el7.x86_64 		0xac
13
? ip_copy_metadata
vmlinux 0xffffffff92ea3140
Build id: 3.10.0-1160.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffff92ecfdc1
Build id: 3.10.0-1160.el7.x86_64 		0x5e1
15
? __pollwait
vmlinux 0xffffffff92a63e50
Build id: 3.10.0-1160.el7.x86_64 		0xf0
16
? sock_has_perm
vmlinux 0xffffffff92b0be65
Build id: 3.10.0-1160.el7.x86_64 		0x75
17
inet_sendmsg
vmlinux 0xffffffff92edd589
Build id: 3.10.0-1160.el7.x86_64 		0x69
18
? selinux_socket_sendmsg
vmlinux 0xffffffff92b0bf93
Build id: 3.10.0-1160.el7.x86_64 		0x23
19
sock_sendmsg
vmlinux 0xffffffff92e343a6
Build id: 3.10.0-1160.el7.x86_64 		0xb6
20
? sidtab_find_context
vmlinux 0xffffffff92b1a0ca
Build id: 3.10.0-1160.el7.x86_64 		0x18a
21
___sys_sendmsg
vmlinux 0xffffffff92e35269
Build id: 3.10.0-1160.el7.x86_64 		0x3e9
22
? __ip_route_output_key_hash
vmlinux 0xffffffff92e9bc63
Build id: 3.10.0-1160.el7.x86_64 		0x223
23
? ip4_datagram_release_cb
vmlinux 0xffffffff92ecc28f
Build id: 3.10.0-1160.el7.x86_64 		0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff92f8992e
Build id: 3.10.0-1160.el7.x86_64 		0x1e
25
? udp_lib_rehash
vmlinux 0xffffffff92ecf33f
Build id: 3.10.0-1160.el7.x86_64 		0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff92f8992e
Build id: 3.10.0-1160.el7.x86_64 		0x1e
27
? release_sock
vmlinux 0xffffffff92e39a18
Build id: 3.10.0-1160.el7.x86_64 		0x128
28
? ip4_datagram_connect
vmlinux 0xffffffff92ecc6f7
Build id: 3.10.0-1160.el7.x86_64 		0x37
29
? inet_dgram_connect
vmlinux 0xffffffff92edc214
Build id: 3.10.0-1160.el7.x86_64 		0x34
30
? SYSC_connect
vmlinux 0xffffffff92e3496d
Build id: 3.10.0-1160.el7.x86_64 		0xed
31
__sys_sendmsg
vmlinux 0xffffffff92e36921
Build id: 3.10.0-1160.el7.x86_64 		0x51
32
SyS_sendmsg
vmlinux 0xffffffff92e36972
Build id: 3.10.0-1160.el7.x86_64 		0x12
33
system_call_fastpath
vmlinux 0xffffffff92f93f92
Build id: 3.10.0-1160.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 28s! [chronyd:851]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr joydev vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm nfit serio_raw drm ahci libahci vmxnet3 libnvdimm ata_piix vmw_pvscsi libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 851 Comm: chronyd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8dc26478c200 ti: ffff8dc27a3bc000 task.ti: ffff8dc27a3bc000
RIP: 0010:[<ffffffffc04243dc>]  [<ffffffffc04243dc>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0018:ffff8dc27a3bf6e8  EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffffffff92ee3f63 RCX: 000000000000010d
RDX: ffff9b8a01900600 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff8dc27a3bf778 R08: 0c008cc4e3565000 R09: 0000000000c80036
R10: 000803965a290c00 R11: ffff8dc5208610c0 R12: ffffffff93515e40
R13: 0000000000000000 R14: 0000000000000004 R15: 0000000000000000
FS:  00007fed1cdd9740(0000) GS:ffff8dc52d6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0000e9010 CR3: 00000001793da000 CR4: 00000000003607e0
Call Trace:
[<ffffffff92e54386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff92e8242a>] sch_direct_xmit+0x11a/0x250
[<ffffffff92e825ee>] __qdisc_run+0x8e/0x360
[<ffffffff92e57319>] __dev_queue_xmit+0x4a9/0x650
[<ffffffff92e574d0>] dev_queue_xmit+0x10/0x20
[<ffffffff92ea545b>] ip_finish_output+0x67b/0x8d0
[<ffffffff92ea59db>] ip_output+0x7b/0xf0
[<ffffffff92ea4de0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff92ea3407>] ip_local_out_sk+0x37/0x40
[<ffffffff92ea6446>] ip_send_skb+0x16/0x50
[<ffffffff92ece85c>] udp_send_skb+0xac/0x2b0
[<ffffffff92ea3140>] ? ip_copy_metadata+0x170/0x170
[<ffffffff92ecfdc1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff92a63e50>] ? __pollwait+0xf0/0xf0
[<ffffffff92b0be65>] ? sock_has_perm+0x75/0x90
[<ffffffff92edd589>] inet_sendmsg+0x69/0xb0
[<ffffffff92b0bf93>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff92e343a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff92b1a0ca>] ? sidtab_find_context+0x18a/0x1a0
[<ffffffff92e35269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff92e9bc63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffff92ecc28f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff92f8992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff92ecf33f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff92f8992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff92e39a18>] ? release_sock+0x128/0x180
[<ffffffff92ecc6f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff92edc214>] ? inet_dgram_connect+0x34/0x90
[<ffffffff92e3496d>] ? SYSC_connect+0xed/0x130
[<ffffffff92e36921>] __sys_sendmsg+0x51/0x90
[<ffffffff92e36972>] SyS_sendmsg+0x12/0x20
[<ffffffff92f93f92>] system_call_fastpath+0x25/0x2a
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41
Complete report #608378
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00958e7
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff8847edc6
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffff884add72
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x122
4
__dev_queue_xmit
vmlinux 0xffffffff88481dd1
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x491
5
? local_bh_enable
vmlinux 0xffffffff87ea8967
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x17
6
dev_queue_xmit
vmlinux 0xffffffff88481fa0
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x10
7
neigh_resolve_output
vmlinux 0xffffffff8848f03d
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x11d
8
ip_finish_output
vmlinux 0xffffffff884d220c
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x27c
9
ip_output
vmlinux 0xffffffff884d2b9b
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffff884d1f90
Build id: 3.10.0-1160.81.1.el7.x86_64 		0xa60
11
ip_local_out_sk
vmlinux 0xffffffff884d0587
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffff884d3636
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffff884fc3ac
Build id: 3.10.0-1160.81.1.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffff884d02b0
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffff884fd9a1
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x5e1
16
? __wake_up_common
vmlinux 0xffffffff87ed8cb2
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x82
17
? ep_poll_callback
vmlinux 0xffffffff880acbf8
Build id: 3.10.0-1160.81.1.el7.x86_64 		0xf8
18
? sock_has_perm
vmlinux 0xffffffff8811f105
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x75
19
inet_sendmsg
vmlinux 0xffffffff8850b4e9
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x69
20
? selinux_socket_sendmsg
vmlinux 0xffffffff8811f243
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x23
21
sock_sendmsg
vmlinux 0xffffffff8845e4f6
Build id: 3.10.0-1160.81.1.el7.x86_64 		0xb6
22
? unix_dgram_sendmsg
vmlinux 0xffffffff8854481b
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x3cb
23
? sidtab_find_context
vmlinux 0xffffffff8812d82e
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x19e
24
___sys_sendmsg
vmlinux 0xffffffff8845f3f9
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x3e9
25
? __ip_route_output_key_hash
vmlinux 0xffffffff884c8c63
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x223
26
? ip4_datagram_release_cb
vmlinux 0xffffffff884f9cdf
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x6f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffff885ba56e
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x1e
28
? udp_lib_rehash
vmlinux 0xffffffff884fcecf
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x11f
29
? _raw_spin_unlock_bh
vmlinux 0xffffffff885ba56e
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x1e
30
? release_sock
vmlinux 0xffffffff88463ea8
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x128
31
? ip4_datagram_connect
vmlinux 0xffffffff884fa147
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x37
32
? inet_dgram_connect
vmlinux 0xffffffff8850a0e4
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x34
33
__sys_sendmsg
vmlinux 0xffffffff88460b41
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x51
34
SyS_sendmsg
vmlinux 0xffffffff88460b92
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x12
35
system_call_fastpath
vmlinux 0xffffffff885c539a
Build id: 3.10.0-1160.81.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 3138s! [chronyd:729]
Modules linked in: tcp_lp binfmt_misc xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) sunrpc intel_powerclamp crc32_pclmul ghash_clmulni_intel squashfs loop snd_intel8x0 snd_ac97_codec ppdev aesni_intel ac97_bus lrw gf128mul snd_seq glue_helper snd_seq_device ablk_helper cryptd snd_pcm joydev pcspkr snd_timer snd sg parport_pc parport video soundcore i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod cdrom crct10dif_generic ata_generic pata_acpi ata_piix libata vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
ttm drm e1000 crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 729 Comm: chronyd Tainted: G               ------------ T 3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff90b119c2a100 ti: ffff90b119cdc000 task.ti: ffff90b119cdc000
RIP: 0010:[<ffffffffc00958e7>]  [<ffffffffc00958e7>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff90b119cdf710  EFLAGS: 00000286
RAX: ffffa083c0f43818 RBX: ffffffff88467453 RCX: 0000000000000060
RDX: 0000000000000060 RSI: ffff90b075de7200 RDI: ffff90b255cc6000
RBP: ffff90b119cdf790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffffffff87e706ce R15: ffff90b119cdf678
FS:  00007f9e7e301740(0000) GS:ffff90b25fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0610d6c000 CR3: 00000000dba00000 CR4: 00000000000606f0
Call Trace:
[<ffffffff8847edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff884add72>] sch_direct_xmit+0x122/0x250
[<ffffffff88481dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffff87ea8967>] ? local_bh_enable+0x17/0x20
[<ffffffff88481fa0>] dev_queue_xmit+0x10/0x20
[<ffffffff8848f03d>] neigh_resolve_output+0x11d/0x220
[<ffffffff884d220c>] ip_finish_output+0x27c/0x8d0
[<ffffffff884d2b9b>] ip_output+0x7b/0x100
[<ffffffff884d1f90>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff884d0587>] ip_local_out_sk+0x37/0x40
[<ffffffff884d3636>] ip_send_skb+0x16/0x50
[<ffffffff884fc3ac>] udp_send_skb+0xac/0x2b0
[<ffffffff884d02b0>] ? ip_copy_metadata+0x170/0x170
[<ffffffff884fd9a1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff87ed8cb2>] ? __wake_up_common+0x82/0x120
[<ffffffff880acbf8>] ? ep_poll_callback+0xf8/0x220
[<ffffffff8811f105>] ? sock_has_perm+0x75/0xa0
[<ffffffff8850b4e9>] inet_sendmsg+0x69/0xb0
[<ffffffff8811f243>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff8845e4f6>] sock_sendmsg+0xb6/0x100
[<ffffffff8854481b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffff8812d82e>] ? sidtab_find_context+0x19e/0x1b0
[<ffffffff8845f3f9>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff884c8c63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffff884f9cdf>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff885ba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff884fcecf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff885ba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff88463ea8>] ? release_sock+0x128/0x180
[<ffffffff884fa147>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff8850a0e4>] ? inet_dgram_connect+0x34/0x90
[<ffffffff88460b41>] __sys_sendmsg+0x51/0x90
[<ffffffff88460b92>] SyS_sendmsg+0x12/0x30
[<ffffffff885c539a>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc
Complete report #605718
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc045b877
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffb3058f96
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffffb30870aa
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffb305bf11
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x491
5
dev_queue_xmit
vmlinux 0xffffffffb305c0e0
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x10
6
ip_finish_output
vmlinux 0xffffffffb30aac6b
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x67b
7
ip_output
vmlinux 0xffffffffb30ab1eb
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffffb30aa5f0
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xa60
9
ip_local_out_sk
vmlinux 0xffffffffb30a8c17
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x37
10
ip_send_skb
vmlinux 0xffffffffb30abc56
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x16
11
udp_send_skb
vmlinux 0xffffffffb30d40fc
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xac
12
? ip_copy_metadata
vmlinux 0xffffffffb30a8950
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x170
13
udp_sendmsg
vmlinux 0xffffffffb30d5661
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x5e1
14
? __alloc_skb
vmlinux 0xffffffffb30431ad
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x8d
15
? ep_poll_callback
vmlinux 0xffffffffb2c9da38
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xf8
16
? sock_has_perm
vmlinux 0xffffffffb2d0c895
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x75
17
inet_sendmsg
vmlinux 0xffffffffb30e2e49
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x69
18
? selinux_socket_sendmsg
vmlinux 0xffffffffb2d0c9c3
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x23
19
sock_sendmsg
vmlinux 0xffffffffb30393a6
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xb6
20
? unix_dgram_sendmsg
vmlinux 0xffffffffb311b04b
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x3cb
21
? sidtab_find_context
vmlinux 0xffffffffb2d1ab1a
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x18a
22
___sys_sendmsg
vmlinux 0xffffffffb303a269
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x3e9
23
? __ip_route_output_key_hash
vmlinux 0xffffffffb30a1483
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x223
24
? ip4_datagram_release_cb
vmlinux 0xffffffffb30d1b2f
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffb318ebae
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x1e
26
? udp_lib_rehash
vmlinux 0xffffffffb30d4bdf
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffb318ebae
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x1e
28
? release_sock
vmlinux 0xffffffffb303ea18
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x128
29
? ip4_datagram_connect
vmlinux 0xffffffffb30d1f97
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x37
30
? inet_dgram_connect
vmlinux 0xffffffffb30e1ad4
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x34
31
? SYSC_connect
vmlinux 0xffffffffb303996d
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xed
32
__sys_sendmsg
vmlinux 0xffffffffb303b921
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x51
33
SyS_sendmsg
vmlinux 0xffffffffb303b972
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x12
34
system_call_fastpath
vmlinux 0xffffffffb3199f92
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 24s! [chronyd:1313]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink binfmt_misc overlay(T) vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt ata_piix fb_sys_fops ttm drm nfit libata libnvdimm crct10dif_pclmul crct10dif_common
e1000 mptspi crc32c_intel scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 1313 Comm: chronyd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8ee721cdd280 ti: ffff8ee7217f8000 task.ti: ffff8ee7217f8000
RIP: 0010:[<ffffffffc045b877>]  [<ffffffffc045b877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8ee7217fb750  EFLAGS: 00000286
RAX: ffff9f9380ac3818 RBX: ffffffffb3041e73 RCX: 000000000000000b
RDX: 000000000000000b RSI: ffff8ee6b58fd5c0 RDI: ffff8ee72dda6000
RBP: ffff8ee7217fb7d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffffffffc09018dd R15: ffff8ee7217fb6e8
FS:  00007ff8cd852740(0000) GS:ffff8ee72f880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6d1655a000 CR3: 00000000aaf0a000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffb3058f96>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb30870aa>] sch_direct_xmit+0x11a/0x250
[<ffffffffb305bf11>] __dev_queue_xmit+0x491/0x650
[<ffffffffb305c0e0>] dev_queue_xmit+0x10/0x20
[<ffffffffb30aac6b>] ip_finish_output+0x67b/0x8d0
[<ffffffffb30ab1eb>] ip_output+0x7b/0xf0
[<ffffffffb30aa5f0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffb30a8c17>] ip_local_out_sk+0x37/0x40
[<ffffffffb30abc56>] ip_send_skb+0x16/0x50
[<ffffffffb30d40fc>] udp_send_skb+0xac/0x2b0
[<ffffffffb30a8950>] ? ip_copy_metadata+0x170/0x170
[<ffffffffb30d5661>] udp_sendmsg+0x5e1/0xa70
[<ffffffffb30431ad>] ? __alloc_skb+0x8d/0x2d0
[<ffffffffb2c9da38>] ? ep_poll_callback+0xf8/0x220
[<ffffffffb2d0c895>] ? sock_has_perm+0x75/0x90
[<ffffffffb30e2e49>] inet_sendmsg+0x69/0xb0
[<ffffffffb2d0c9c3>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffffb30393a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffb311b04b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffb2d1ab1a>] ? sidtab_find_context+0x18a/0x1a0
[<ffffffffb303a269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffb30a1483>] ? __ip_route_output_key_hash+0x223/0x7e0
[<ffffffffb30d1b2f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffb318ebae>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb30d4bdf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffb318ebae>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb303ea18>] ? release_sock+0x128/0x180
[<ffffffffb30d1f97>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffb30e1ad4>] ? inet_dgram_connect+0x34/0x90
[<ffffffffb303996d>] ? SYSC_connect+0xed/0x130
[<ffffffffb303b921>] __sys_sendmsg+0x51/0x90
[<ffffffffb303b972>] SyS_sendmsg+0x12/0x20
[<ffffffffb3199f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:37:crtc-0] flip_done timed out
Complete report #526574
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc03d8877
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x7f7
2
? __alloc_skb
vmlinux 0xffffffffaf8401ad
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x8d
3
dev_hard_start_xmit
vmlinux 0xffffffffaf855f96
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffffaf8840aa
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffaf858f11
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x491
6
dev_queue_xmit
vmlinux 0xffffffffaf8590e0
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x10
7
ip_finish_output
vmlinux 0xffffffffaf8a70eb
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x67b
8
ip_output
vmlinux 0xffffffffaf8a766b
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffffaf8a6a70
Build id: 3.10.0-1160.24.1.el7.x86_64 		0xa60
10
ip_local_out_sk
vmlinux 0xffffffffaf8a5097
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x37
11
ip_send_skb
vmlinux 0xffffffffaf8a80d6
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffffaf8d053c
Build id: 3.10.0-1160.24.1.el7.x86_64 		0xac
13
? ip_copy_metadata
vmlinux 0xffffffffaf8a4dd0
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffffaf8d1aa1
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x5e1
15
? __pollwait
vmlinux 0xffffffffaf464100
Build id: 3.10.0-1160.24.1.el7.x86_64 		0xf0
16
? sock_has_perm
vmlinux 0xffffffffaf50c2a5
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x75
17
inet_sendmsg
vmlinux 0xffffffffaf8df289
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x69
18
? selinux_socket_sendmsg
vmlinux 0xffffffffaf50c3d3
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x23
19
sock_sendmsg
vmlinux 0xffffffffaf8363a6
Build id: 3.10.0-1160.24.1.el7.x86_64 		0xb6
20
? sidtab_find_context
vmlinux 0xffffffffaf51a52a
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x18a
21
___sys_sendmsg
vmlinux 0xffffffffaf837269
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x3e9
22
? __ip_route_output_key_hash
vmlinux 0xffffffffaf89d8f3
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x223
23
? ip4_datagram_release_cb
vmlinux 0xffffffffaf8cdf6f
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffaf98b67e
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x1e
25
? udp_lib_rehash
vmlinux 0xffffffffaf8d101f
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffffaf98b67e
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x1e
27
? release_sock
vmlinux 0xffffffffaf83ba18
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x128
28
? ip4_datagram_connect
vmlinux 0xffffffffaf8ce3d7
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x37
29
? inet_dgram_connect
vmlinux 0xffffffffaf8ddf14
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x34
30
? SYSC_connect
vmlinux 0xffffffffaf83696d
Build id: 3.10.0-1160.24.1.el7.x86_64 		0xed
31
__sys_sendmsg
vmlinux 0xffffffffaf838921
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x51
32
SyS_sendmsg
vmlinux 0xffffffffaf838972
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x12
33
system_call_fastpath
vmlinux 0xffffffffaf995f92
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 21s! [chronyd:956]
Modules linked in: tcp_diag inet_diag cfg80211 rfkill sunrpc nfit libnvdimm iosf_mbi sg crc32_pclmul ppdev ghash_clmulni_intel parport_pc aesni_intel parport i2c_piix4 virtio_balloon lrw gf128mul pcspkr glue_helper ablk_helper joydev cryptd ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi ata_piix libata e1000 serio_raw virtio_pci crct10dif_pclmul crct10dif_common crc32c_intel virtio_ring floppy virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 7 PID: 956 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.24.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffffa03fd83e9080 ti: ffffa03fd97f4000 task.ti: ffffa03fd97f4000
RIP: 0010:[<ffffffffc03d8877>]  [<ffffffffc03d8877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffffa03fd97f7750  EFLAGS: 00000286
RAX: ffffb3eb83583818 RBX: ffffffffaf83ee73 RCX: 0000000000000009
RDX: 0000000000000009 RSI: ffffa03fe17f6980 RDI: ffffa03fc43c4000
RBP: ffffa03fd97f77d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: ffffa03fd97f76c8 R14: ffffa03fce3f8188 R15: ffffa03fd97f76b8
FS:  00007f74f2ff8740(0000) GS:ffffa046bfdc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd42297c9e5 CR3: 0000000160f92000 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaf8401ad>] ? __alloc_skb+0x8d/0x2d0
[<ffffffffaf855f96>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffaf8840aa>] sch_direct_xmit+0x11a/0x250
[<ffffffffaf858f11>] __dev_queue_xmit+0x491/0x650
[<ffffffffaf8590e0>] dev_queue_xmit+0x10/0x20
[<ffffffffaf8a70eb>] ip_finish_output+0x67b/0x8d0
[<ffffffffaf8a766b>] ip_output+0x7b/0xf0
[<ffffffffaf8a6a70>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffaf8a5097>] ip_local_out_sk+0x37/0x40
[<ffffffffaf8a80d6>] ip_send_skb+0x16/0x50
[<ffffffffaf8d053c>] udp_send_skb+0xac/0x2b0
[<ffffffffaf8a4dd0>] ? ip_copy_metadata+0x170/0x170
[<ffffffffaf8d1aa1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffaf464100>] ? __pollwait+0xf0/0xf0
[<ffffffffaf50c2a5>] ? sock_has_perm+0x75/0x90
[<ffffffffaf8df289>] inet_sendmsg+0x69/0xb0
[<ffffffffaf50c3d3>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffffaf8363a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffaf51a52a>] ? sidtab_find_context+0x18a/0x1a0
[<ffffffffaf837269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffaf89d8f3>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffffaf8cdf6f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffaf98b67e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffaf8d101f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffaf98b67e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffaf83ba18>] ? release_sock+0x128/0x180
[<ffffffffaf8ce3d7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffaf8ddf14>] ? inet_dgram_connect+0x34/0x90
[<ffffffffaf83696d>] ? SYSC_connect+0xed/0x130
[<ffffffffaf838921>] __sys_sendmsg+0x51/0x90
[<ffffffffaf838972>] SyS_sendmsg+0x12/0x20
[<ffffffffaf995f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #511557
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc012f877
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff98458f96
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffff984870aa
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff9845bf11
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x491
5
? local_bh_enable
vmlinux 0xffffffff97ea4077
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x17
6
dev_queue_xmit
vmlinux 0xffffffff9845c0e0
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x10
7
neigh_resolve_output
vmlinux 0xffffffff98468e0d
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x11d
8
ip_finish_output
vmlinux 0xffffffff984aa86c
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x27c
9
ip_output
vmlinux 0xffffffff984ab1eb
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffff984aa5f0
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xa60
11
ip_local_out_sk
vmlinux 0xffffffff984a8c17
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffff984abc56
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffff984d40fc
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffff984a8950
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffff984d5661
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x5e1
16
? ep_poll_callback
vmlinux 0xffffffff9809da38
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xf8
17
? avc_has_perm_flags
vmlinux 0xffffffff9810c06c
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xdc
18
? sock_has_perm
vmlinux 0xffffffff9810c895
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x75
19
inet_sendmsg
vmlinux 0xffffffff984e2e49
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x69
20
? selinux_socket_sendmsg
vmlinux 0xffffffff9810c9c3
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x23
21
sock_sendmsg
vmlinux 0xffffffff984393a6
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xb6
22
? avc_compute_av
vmlinux 0xffffffff985856c6
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x1d2
23
? xfrm_replay_advance_bmp
vmlinux 0xffffffff98510005
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x85
24
? sidtab_find_context
vmlinux 0xffffffff9811ab1a
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x18a
25
___sys_sendmsg
vmlinux 0xffffffff9843a269
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x3e9
26
? __ip_route_output_key_hash
vmlinux 0xffffffff984a1483
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x223
27
? ip4_datagram_release_cb
vmlinux 0xffffffff984d1b2f
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x6f
28
? _raw_spin_unlock_bh
vmlinux 0xffffffff9858ebae
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x1e
29
? udp_lib_rehash
vmlinux 0xffffffff984d4bdf
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x11f
30
? _raw_spin_unlock_bh
vmlinux 0xffffffff9858ebae
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x1e
31
? release_sock
vmlinux 0xffffffff9843ea18
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x128
32
? ip4_datagram_connect
vmlinux 0xffffffff984d1f97
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x37
33
? inet_dgram_connect
vmlinux 0xffffffff984e1ad4
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x34
34
? SYSC_connect
vmlinux 0xffffffff9843996d
Build id: 3.10.0-1160.59.1.el7.x86_64 		0xed
35
__sys_sendmsg
vmlinux 0xffffffff9843b921
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x51
36
SyS_sendmsg
vmlinux 0xffffffff9843b972
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x12
37
system_call_fastpath
vmlinux 0xffffffff98599f92
Build id: 3.10.0-1160.59.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 25s! [chronyd:1178]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter binfmt_misc overlay(T) vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg vmw_balloon joydev pcspkr parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel
nfit mptspi libnvdimm scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks e1000 serio_raw dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 1178 Comm: chronyd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9dc2b70ed280 ti: ffff9dc2b4ab4000 task.ti: ffff9dc2b4ab4000
RIP: 0010:[<ffffffffc012f877>]  [<ffffffffc012f877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9dc2b4ab7710  EFLAGS: 00000286
RAX: ffffb67e806c3818 RBX: ffffffff98441e73 RCX: 00000000000000ac
RDX: 00000000000000ac RSI: ffff9dc3300d5180 RDI: ffff9dc3343f8000
RBP: ffff9dc2b4ab7790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffff9dc2b4934300 R15: 0000000000010000
FS:  00007f36e6182740(0000) GS:ffff9dc3396c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f248b405000 CR3: 00000000b4f50000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff98458f96>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff984870aa>] sch_direct_xmit+0x11a/0x250
[<ffffffff9845bf11>] __dev_queue_xmit+0x491/0x650
[<ffffffff97ea4077>] ? local_bh_enable+0x17/0x20
[<ffffffff9845c0e0>] dev_queue_xmit+0x10/0x20
[<ffffffff98468e0d>] neigh_resolve_output+0x11d/0x220
[<ffffffff984aa86c>] ip_finish_output+0x27c/0x8d0
[<ffffffff984ab1eb>] ip_output+0x7b/0xf0
[<ffffffff984aa5f0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff984a8c17>] ip_local_out_sk+0x37/0x40
[<ffffffff984abc56>] ip_send_skb+0x16/0x50
[<ffffffff984d40fc>] udp_send_skb+0xac/0x2b0
[<ffffffff984a8950>] ? ip_copy_metadata+0x170/0x170
[<ffffffff984d5661>] udp_sendmsg+0x5e1/0xa70
[<ffffffff9809da38>] ? ep_poll_callback+0xf8/0x220
[<ffffffff9810c06c>] ? avc_has_perm_flags+0xdc/0x1c0
[<ffffffff9810c895>] ? sock_has_perm+0x75/0x90
[<ffffffff984e2e49>] inet_sendmsg+0x69/0xb0
[<ffffffff9810c9c3>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff984393a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff985856c6>] ? avc_compute_av+0x1d2/0x1e4
[<ffffffff98510005>] ? xfrm_replay_advance_bmp+0x85/0x150
[<ffffffff9811ab1a>] ? sidtab_find_context+0x18a/0x1a0
[<ffffffff9843a269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff984a1483>] ? __ip_route_output_key_hash+0x223/0x7e0
[<ffffffff984d1b2f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff9858ebae>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff984d4bdf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff9858ebae>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff9843ea18>] ? release_sock+0x128/0x180
[<ffffffff984d1f97>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff984e1ad4>] ? inet_dgram_connect+0x34/0x90
[<ffffffff9843996d>] ? SYSC_connect+0xed/0x130
[<ffffffff9843b921>] __sys_sendmsg+0x51/0x90
[<ffffffff9843b972>] SyS_sendmsg+0x12/0x20
[<ffffffff98599f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #449306
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc031c877
Build id: 3.10.0-1160.el7.x86_64 		0x7f7
2
? kfree_skbmem
vmlinux 0xffffffff98e3c6f7
Build id: 3.10.0-1160.el7.x86_64 		0x37
3
dev_hard_start_xmit
vmlinux 0xffffffff98e54386
Build id: 3.10.0-1160.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffff98e8242a
Build id: 3.10.0-1160.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffff98e57301
Build id: 3.10.0-1160.el7.x86_64 		0x491
6
dev_queue_xmit
vmlinux 0xffffffff98e574d0
Build id: 3.10.0-1160.el7.x86_64 		0x10
7
ip_finish_output
vmlinux 0xffffffff98ea545b
Build id: 3.10.0-1160.el7.x86_64 		0x67b
8
ip_output
vmlinux 0xffffffff98ea59db
Build id: 3.10.0-1160.el7.x86_64 		0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffff98ea4de0
Build id: 3.10.0-1160.el7.x86_64 		0xa60
10
ip_local_out_sk
vmlinux 0xffffffff98ea3407
Build id: 3.10.0-1160.el7.x86_64 		0x37
11
ip_send_skb
vmlinux 0xffffffff98ea6446
Build id: 3.10.0-1160.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffff98ece85c
Build id: 3.10.0-1160.el7.x86_64 		0xac
13
? ip_copy_metadata
vmlinux 0xffffffff98ea3140
Build id: 3.10.0-1160.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffff98ecfdc1
Build id: 3.10.0-1160.el7.x86_64 		0x5e1
15
? __pollwait
vmlinux 0xffffffff98a63e50
Build id: 3.10.0-1160.el7.x86_64 		0xf0
16
? sock_has_perm
vmlinux 0xffffffff98b0be65
Build id: 3.10.0-1160.el7.x86_64 		0x75
17
inet_sendmsg
vmlinux 0xffffffff98edd589
Build id: 3.10.0-1160.el7.x86_64 		0x69
18
? selinux_socket_sendmsg
vmlinux 0xffffffff98b0bf93
Build id: 3.10.0-1160.el7.x86_64 		0x23
19
sock_sendmsg
vmlinux 0xffffffff98e343a6
Build id: 3.10.0-1160.el7.x86_64 		0xb6
20
? sidtab_find_context
vmlinux 0xffffffff98b1a0ca
Build id: 3.10.0-1160.el7.x86_64 		0x18a
21
___sys_sendmsg
vmlinux 0xffffffff98e35269
Build id: 3.10.0-1160.el7.x86_64 		0x3e9
22
? __ip_route_output_key_hash
vmlinux 0xffffffff98e9bc63
Build id: 3.10.0-1160.el7.x86_64 		0x223
23
? ip4_datagram_release_cb
vmlinux 0xffffffff98ecc28f
Build id: 3.10.0-1160.el7.x86_64 		0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff98f8992e
Build id: 3.10.0-1160.el7.x86_64 		0x1e
25
? udp_lib_rehash
vmlinux 0xffffffff98ecf33f
Build id: 3.10.0-1160.el7.x86_64 		0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff98f8992e
Build id: 3.10.0-1160.el7.x86_64 		0x1e
27
? release_sock
vmlinux 0xffffffff98e39a18
Build id: 3.10.0-1160.el7.x86_64 		0x128
28
? ip4_datagram_connect
vmlinux 0xffffffff98ecc6f7
Build id: 3.10.0-1160.el7.x86_64 		0x37
29
? inet_dgram_connect
vmlinux 0xffffffff98edc214
Build id: 3.10.0-1160.el7.x86_64 		0x34
30
? SYSC_connect
vmlinux 0xffffffff98e3496d
Build id: 3.10.0-1160.el7.x86_64 		0xed
31
__sys_sendmsg
vmlinux 0xffffffff98e36921
Build id: 3.10.0-1160.el7.x86_64 		0x51
32
SyS_sendmsg
vmlinux 0xffffffff98e36972
Build id: 3.10.0-1160.el7.x86_64 		0x12
33
system_call_fastpath
vmlinux 0xffffffff98f93f92
Build id: 3.10.0-1160.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [chronyd:658]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 iosf_mbi crc32_pclmul snd_rawmidi snd_ac97_codec ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ac97_bus snd_seq snd_seq_device snd_pcm
cryptd snd_timer snd ppdev vmw_balloon soundcore joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi mptspi vmwgfx scsi_transport_spi drm_kms_helper syscopyarea sysfillrect sysimgblt ata_piix crct10dif_pclmul fb_sys_fops mptscsih crct10dif_common ttm nfit libata drm mptbase crc32c_intel e1000 drm_panel_orientation_quirks serio_raw libnvdimm fuse
CPU: 0 PID: 658 Comm: chronyd Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8a62312e1080 ti: ffff8a62341cc000 task.ti: ffff8a62341cc000
RIP: 0010:[<ffffffffc031c877>]  [<ffffffffc031c877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8a62341cf750  EFLAGS: 00000286
RAX: ffffb16b80443818 RBX: ffffffff98e3ce73 RCX: 00000000000000ee
RDX: 00000000000000ee RSI: ffff8a61f5e79e00 RDI: ffff8a6234bea000
RBP: ffff8a62341cf7d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffffffffc085a8dd R15: ffff8a62341cf6e8
FS:  00007f4655a65740(0000) GS:ffff8a623b600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f91f0f5d000 CR3: 0000000075b30000 CR4: 00000000003607f0
Call Trace:
[<ffffffff98e3c6f7>] ? kfree_skbmem+0x37/0x90
[<ffffffff98e54386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff98e8242a>] sch_direct_xmit+0x11a/0x250
[<ffffffff98e57301>] __dev_queue_xmit+0x491/0x650
[<ffffffff98e574d0>] dev_queue_xmit+0x10/0x20
[<ffffffff98ea545b>] ip_finish_output+0x67b/0x8d0
[<ffffffff98ea59db>] ip_output+0x7b/0xf0
[<ffffffff98ea4de0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff98ea3407>] ip_local_out_sk+0x37/0x40
[<ffffffff98ea6446>] ip_send_skb+0x16/0x50
[<ffffffff98ece85c>] udp_send_skb+0xac/0x2b0
[<ffffffff98ea3140>] ? ip_copy_metadata+0x170/0x170
[<ffffffff98ecfdc1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff98a63e50>] ? __pollwait+0xf0/0xf0
[<ffffffff98b0be65>] ? sock_has_perm+0x75/0x90
[<ffffffff98edd589>] inet_sendmsg+0x69/0xb0
[<ffffffff98b0bf93>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff98e343a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff98b1a0ca>] ? sidtab_find_context+0x18a/0x1a0
[<ffffffff98e35269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff98e9bc63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffff98ecc28f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff98f8992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff98ecf33f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff98f8992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff98e39a18>] ? release_sock+0x128/0x180
[<ffffffff98ecc6f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff98edc214>] ? inet_dgram_connect+0x34/0x90
[<ffffffff98e3496d>] ? SYSC_connect+0xed/0x130
[<ffffffff98e36921>] __sys_sendmsg+0x51/0x90
[<ffffffff98e36972>] SyS_sendmsg+0x12/0x20
[<ffffffff98f93f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #334502
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc036d877
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff9f653386
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffff9f68131a
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff9f656301
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x491
5
dev_queue_xmit
vmlinux 0xffffffff9f6564d0
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x10
6
ip_finish_output
vmlinux 0xffffffff9f6a429b
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x67b
7
ip_output
vmlinux 0xffffffff9f6a481b
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffff9f6a3c20
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xa60
9
ip_local_out_sk
vmlinux 0xffffffff9f6a2247
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x37
10
ip_send_skb
vmlinux 0xffffffff9f6a5286
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x16
11
udp_send_skb
vmlinux 0xffffffff9f6cd65c
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xac
12
? ip_copy_metadata
vmlinux 0xffffffff9f6a1f80
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x170
13
udp_sendmsg
vmlinux 0xffffffff9f6cebc1
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x5e1
14
? __wake_up_common
vmlinux 0xffffffff9f0d3c52
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x82
15
? ep_poll_callback
vmlinux 0xffffffff9f29c4a8
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xf8
16
? sock_has_perm
vmlinux 0xffffffff9f30b1a5
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x75
17
inet_sendmsg
vmlinux 0xffffffff9f6dc389
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x69
18
? selinux_socket_sendmsg
vmlinux 0xffffffff9f30b2d3
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x23
19
sock_sendmsg
vmlinux 0xffffffff9f6333a6
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xb6
20
? avc_compute_av
vmlinux 0xffffffff9f77f426
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x1ec
21
? sidtab_find_context
vmlinux 0xffffffff9f31936a
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x18a
22
___sys_sendmsg
vmlinux 0xffffffff9f634269
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x3e9
23
? __ip_route_output_key_hash
vmlinux 0xffffffff9f69aacf
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x20f
24
? ip4_datagram_release_cb
vmlinux 0xffffffff9f6cb06f
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffff9f78856e
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x1e
26
? udp_lib_rehash
vmlinux 0xffffffff9f6ce13f
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffff9f78856e
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x1e
28
? release_sock
vmlinux 0xffffffff9f638a18
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x128
29
? ip4_datagram_connect
vmlinux 0xffffffff9f6cb4d7
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x37
30
? inet_dgram_connect
vmlinux 0xffffffff9f6db014
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x34
31
? SYSC_connect
vmlinux 0xffffffff9f63396d
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xed
32
__sys_sendmsg
vmlinux 0xffffffff9f635921
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x51
33
SyS_sendmsg
vmlinux 0xffffffff9f635972
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x12
34
system_call_fastpath
vmlinux 0xffffffff9f792ed2
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 51s! [chronyd:747]
Modules linked in: fuse xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip_vs nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock overlay(T) sunrpc sb_edac iosf_mbi kvm_intel kvm irqbypass crc32_pclmul nfit ghash_clmulni_intel aesni_intel libnvdimm lrw gf128mul glue_helper ablk_helper ppdev cryptd
vmw_balloon pcspkr sg joydev vmxnet3 parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel e1000 drm ata_piix serio_raw libata mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 747 Comm: chronyd Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1127.18.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff886c76f2d230 ti: ffff886c7145c000 task.ti: ffff886c7145c000
RIP: 0010:[<ffffffffc036d877>]  [<ffffffffc036d877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff886c7145f750  EFLAGS: 00000286
RAX: ffff98a6c0d03818 RBX: ffffffff9f63be73 RCX: 00000000000000d8
RDX: 00000000000000d8 RSI: ffff886c2002c040 RDI: ffff886bbce64000
RBP: ffff886c7145f7d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffff886bbfc07700 R15: ffff886bbfc07700
FS:  00007fc7d977e740(0000) GS:ffff886c756c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f03b75c9000 CR3: 000000022fff2000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9f653386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff9f68131a>] sch_direct_xmit+0x11a/0x250
[<ffffffff9f656301>] __dev_queue_xmit+0x491/0x650
[<ffffffff9f6564d0>] dev_queue_xmit+0x10/0x20
[<ffffffff9f6a429b>] ip_finish_output+0x67b/0x8d0
[<ffffffff9f6a481b>] ip_output+0x7b/0xf0
[<ffffffff9f6a3c20>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff9f6a2247>] ip_local_out_sk+0x37/0x40
[<ffffffff9f6a5286>] ip_send_skb+0x16/0x50
[<ffffffff9f6cd65c>] udp_send_skb+0xac/0x2b0
[<ffffffff9f6a1f80>] ? ip_copy_metadata+0x170/0x170
[<ffffffff9f6cebc1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff9f0d3c52>] ? __wake_up_common+0x82/0x120
[<ffffffff9f29c4a8>] ? ep_poll_callback+0xf8/0x220
[<ffffffff9f30b1a5>] ? sock_has_perm+0x75/0x90
[<ffffffff9f6dc389>] inet_sendmsg+0x69/0xb0
[<ffffffff9f30b2d3>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff9f6333a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff9f77f426>] ? avc_compute_av+0x1ec/0x1fe
[<ffffffff9f31936a>] ? sidtab_find_context+0x18a/0x1a0
[<ffffffff9f634269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff9f69aacf>] ? __ip_route_output_key_hash+0x20f/0x7d0
[<ffffffff9f6cb06f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff9f78856e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff9f6ce13f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff9f78856e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff9f638a18>] ? release_sock+0x128/0x180
[<ffffffff9f6cb4d7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff9f6db014>] ? inet_dgram_connect+0x34/0x90
[<ffffffff9f63396d>] ? SYSC_connect+0xed/0x130
[<ffffffff9f635921>] __sys_sendmsg+0x51/0x90
[<ffffffff9f635972>] SyS_sendmsg+0x12/0x20
[<ffffffff9f792ed2>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #254207
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0236877
Build id: 3.10.0-1062.el7.x86_64 		0x7f7
2
? local_bh_enable
vmlinux 0xffffffff8c6a3667
Build id: 3.10.0-1062.el7.x86_64 		0x17
3
dev_hard_start_xmit
vmlinux 0xffffffff8cc4d346
Build id: 3.10.0-1062.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffff8cc7b0ba
Build id: 3.10.0-1062.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffff8cc502b1
Build id: 3.10.0-1062.el7.x86_64 		0x491
6
? local_bh_enable
vmlinux 0xffffffff8c6a3667
Build id: 3.10.0-1062.el7.x86_64 		0x17
7
dev_queue_xmit
vmlinux 0xffffffff8cc50480
Build id: 3.10.0-1062.el7.x86_64 		0x10
8
neigh_resolve_output
vmlinux 0xffffffff8cc5cfed
Build id: 3.10.0-1062.el7.x86_64 		0x11d
9
ip_finish_output
vmlinux 0xffffffff8cc9e57c
Build id: 3.10.0-1062.el7.x86_64 		0x27c
10
ip_output
vmlinux 0xffffffff8cc9eefb
Build id: 3.10.0-1062.el7.x86_64 		0x7b
11
? __ip_append_data.isra.50
vmlinux 0xffffffff8cc9e300
Build id: 3.10.0-1062.el7.x86_64 		0xa60
12
ip_local_out_sk
vmlinux 0xffffffff8cc9c927
Build id: 3.10.0-1062.el7.x86_64 		0x37
13
ip_send_skb
vmlinux 0xffffffff8cc9f966
Build id: 3.10.0-1062.el7.x86_64 		0x16
14
udp_send_skb
vmlinux 0xffffffff8ccc7c6c
Build id: 3.10.0-1062.el7.x86_64 		0xac
15
? ip_copy_metadata
vmlinux 0xffffffff8cc9c660
Build id: 3.10.0-1062.el7.x86_64 		0x170
16
udp_sendmsg
vmlinux 0xffffffff8ccc91d1
Build id: 3.10.0-1062.el7.x86_64 		0x5e1
17
? __pollwait
vmlinux 0xffffffff8c85e540
Build id: 3.10.0-1062.el7.x86_64 		0xf0
18
? sock_has_perm
vmlinux 0xffffffff8c905955
Build id: 3.10.0-1062.el7.x86_64 		0x75
19
inet_sendmsg
vmlinux 0xffffffff8ccd6939
Build id: 3.10.0-1062.el7.x86_64 		0x69
20
? selinux_socket_sendmsg
vmlinux 0xffffffff8c905a83
Build id: 3.10.0-1062.el7.x86_64 		0x23
21
sock_sendmsg
vmlinux 0xffffffff8cc2d3a6
Build id: 3.10.0-1062.el7.x86_64 		0xb6
22
? sidtab_find_context
vmlinux 0xffffffff8c913b9a
Build id: 3.10.0-1062.el7.x86_64 		0x18a
23
___sys_sendmsg
vmlinux 0xffffffff8cc2e269
Build id: 3.10.0-1062.el7.x86_64 		0x3e9
24
? __ip_route_output_key_hash
vmlinux 0xffffffff8cc95257
Build id: 3.10.0-1062.el7.x86_64 		0x207
25
? ip4_datagram_release_cb
vmlinux 0xffffffff8ccc567f
Build id: 3.10.0-1062.el7.x86_64 		0x6f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff8cd8196e
Build id: 3.10.0-1062.el7.x86_64 		0x1e
27
? udp_lib_rehash
vmlinux 0xffffffff8ccc874f
Build id: 3.10.0-1062.el7.x86_64 		0x11f
28
? _raw_spin_unlock_bh
vmlinux 0xffffffff8cd8196e
Build id: 3.10.0-1062.el7.x86_64 		0x1e
29
? release_sock
vmlinux 0xffffffff8cc32a10
Build id: 3.10.0-1062.el7.x86_64 		0x120
30
? ip4_datagram_connect
vmlinux 0xffffffff8ccc5ae7
Build id: 3.10.0-1062.el7.x86_64 		0x37
31
? inet_dgram_connect
vmlinux 0xffffffff8ccd55c4
Build id: 3.10.0-1062.el7.x86_64 		0x34
32
? SYSC_connect
vmlinux 0xffffffff8cc2d96d
Build id: 3.10.0-1062.el7.x86_64 		0xed
33
__sys_sendmsg
vmlinux 0xffffffff8cc2f921
Build id: 3.10.0-1062.el7.x86_64 		0x51
34
SyS_sendmsg
vmlinux 0xffffffff8cc2f972
Build id: 3.10.0-1062.el7.x86_64 		0x12
35
system_call_fastpath
vmlinux 0xffffffff8cd8bede
Build id: 3.10.0-1062.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 43s! [chronyd:900]
Modules linked in: tcp_lp nls_utf8 isofs fuse xt_CHECKSUM tun bridge devlink stp llc target_core_user uio target_core_pscsi target_core_file target_core_iblock iscsi_target_mod target_core_mod ip6t_MASQUERADE nf_nat_masquerade_ipv6 xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_mark ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event raid456 async_raid6_recov async_memcpy
async_pq raid6_pq ppdev coretemp async_xor xor iosf_mbi async_tx crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul snd_ens1371 glue_helper snd_rawmidi ablk_helper snd_ac97_codec cryptd ac97_bus pcspkr snd_seq snd_seq_device snd_pcm snd_timer snd sg joydev soundcore parport_pc parport pcc_cpufreq vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod cdrom crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ata_piix ttm crc32c_intel drm serio_raw mptspi nfit scsi_transport_spi mptscsih e1000 libata mptbase libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 900 Comm: chronyd Kdump: loaded Not tainted 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff9e7680915230 ti: ffff9e767fa6c000 task.ti: ffff9e767fa6c000
RIP: 0010:[<ffffffffc0236877>]  [<ffffffffc0236877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9e767fa6f710  EFLAGS: 00000286
RAX: ffffb8f040703818 RBX: ffffffff8cc35e63 RCX: 000000000000008a
RDX: 000000000000008a RSI: ffff9e7635950940 RDI: ffff9e768164a000
RBP: ffff9e767fa6f790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffff9e768164a000 R15: ffff9e767fa6f6f0
FS:  00007fc3e52cc740(0000) GS:ffff9e7684640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3291d37000 CR3: 00000000813ac000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8c6a3667>] ? local_bh_enable+0x17/0x20
[<ffffffff8cc4d346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff8cc7b0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffff8cc502b1>] __dev_queue_xmit+0x491/0x650
[<ffffffff8c6a3667>] ? local_bh_enable+0x17/0x20
[<ffffffff8cc50480>] dev_queue_xmit+0x10/0x20
[<ffffffff8cc5cfed>] neigh_resolve_output+0x11d/0x220
[<ffffffff8cc9e57c>] ip_finish_output+0x27c/0x8d0
[<ffffffff8cc9eefb>] ip_output+0x7b/0xf0
[<ffffffff8cc9e300>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff8cc9c927>] ip_local_out_sk+0x37/0x40
[<ffffffff8cc9f966>] ip_send_skb+0x16/0x50
[<ffffffff8ccc7c6c>] udp_send_skb+0xac/0x2b0
[<ffffffff8cc9c660>] ? ip_copy_metadata+0x170/0x170
[<ffffffff8ccc91d1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff8c85e540>] ? __pollwait+0xf0/0xf0
[<ffffffff8c905955>] ? sock_has_perm+0x75/0x90
[<ffffffff8ccd6939>] inet_sendmsg+0x69/0xb0
[<ffffffff8c905a83>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff8cc2d3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff8c913b9a>] ? sidtab_find_context+0x18a/0x1a0
[<ffffffff8cc2e269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff8cc95257>] ? __ip_route_output_key_hash+0x207/0x7b0
[<ffffffff8ccc567f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff8cd8196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff8ccc874f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff8cd8196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff8cc32a10>] ? release_sock+0x120/0x170
[<ffffffff8ccc5ae7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff8ccd55c4>] ? inet_dgram_connect+0x34/0x90
[<ffffffff8cc2d96d>] ? SYSC_connect+0xed/0x130
[<ffffffff8cc2f921>] __sys_sendmsg+0x51/0x90
[<ffffffff8cc2f972>] SyS_sendmsg+0x12/0x20
[<ffffffff8cd8bede>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #249912
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc016d877
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffbd053386
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffffbd08131a
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffbd056301
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x491
5
? local_bh_enable
vmlinux 0xffffffffbcaa4b17
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x17
6
dev_queue_xmit
vmlinux 0xffffffffbd0564d0
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x10
7
neigh_resolve_output
vmlinux 0xffffffffbd06312d
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x11d
8
ip_finish_output
vmlinux 0xffffffffbd0a3e9c
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x27c
9
ip_output
vmlinux 0xffffffffbd0a481b
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffffbd0a3c20
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xa60
11
ip_local_out_sk
vmlinux 0xffffffffbd0a2247
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffffbd0a5286
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffffbd0cd65c
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffffbd0a1f80
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffffbd0cebc1
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x5e1
16
? __wake_up_common
vmlinux 0xffffffffbcad3c52
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x82
17
? ep_poll_callback
vmlinux 0xffffffffbcc9c4a8
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xf8
18
? sock_has_perm
vmlinux 0xffffffffbcd0b1a5
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x75
19
inet_sendmsg
vmlinux 0xffffffffbd0dc389
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x69
20
? selinux_socket_sendmsg
vmlinux 0xffffffffbcd0b2d3
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x23
21
sock_sendmsg
vmlinux 0xffffffffbd0333a6
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xb6
22
? unix_dgram_sendmsg
vmlinux 0xffffffffbd11421b
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x3cb
23
? sidtab_find_context
vmlinux 0xffffffffbcd1936a
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x18a
24
___sys_sendmsg
vmlinux 0xffffffffbd034269
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x3e9
25
? __ip_route_output_key_hash
vmlinux 0xffffffffbd09aacf
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x20f
26
? ip4_datagram_release_cb
vmlinux 0xffffffffbd0cb06f
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x6f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffbd18856e
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x1e
28
? udp_lib_rehash
vmlinux 0xffffffffbd0ce13f
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x11f
29
? _raw_spin_unlock_bh
vmlinux 0xffffffffbd18856e
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x1e
30
? release_sock
vmlinux 0xffffffffbd038a18
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x128
31
? ip4_datagram_connect
vmlinux 0xffffffffbd0cb4d7
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x37
32
? inet_dgram_connect
vmlinux 0xffffffffbd0db014
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x34
33
? SYSC_connect
vmlinux 0xffffffffbd03396d
Build id: 3.10.0-1127.18.2.el7.x86_64 		0xed
34
__sys_sendmsg
vmlinux 0xffffffffbd035921
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x51
35
SyS_sendmsg
vmlinux 0xffffffffbd035972
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x12
36
system_call_fastpath
vmlinux 0xffffffffbd192ed2
Build id: 3.10.0-1127.18.2.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 29s! [chronyd:739]
Modules linked in: fuse xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip_vs ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper nfit ablk_helper cryptd libnvdimm ppdev vmw_balloon sg joydev pcspkr parport_pc parport
vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix mptspi libata scsi_transport_spi e1000 mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 739 Comm: chronyd Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1127.18.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9a9c75f8b150 ti: ffff9a9cf5270000 task.ti: ffff9a9cf5270000
RIP: 0010:[<ffffffffc016d877>]  [<ffffffffc016d877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9a9cf5273710  EFLAGS: 00000286
RAX: ffffba2d006c3818 RBX: ffffffffbd03be73 RCX: 00000000000000a2
RDX: 00000000000000a2 RSI: ffff9a9cf1bc44c0 RDI: ffff9a9cf5712000
RBP: ffff9a9cf5273790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffff9a9cf5421a00 R15: 0000000100010000
FS:  00007ff3e7c91740(0000) GS:ffff9a9cf9640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f12ffb707d0 CR3: 0000000135f46000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffbd053386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffbd08131a>] sch_direct_xmit+0x11a/0x250
[<ffffffffbd056301>] __dev_queue_xmit+0x491/0x650
[<ffffffffbcaa4b17>] ? local_bh_enable+0x17/0x20
[<ffffffffbd0564d0>] dev_queue_xmit+0x10/0x20
[<ffffffffbd06312d>] neigh_resolve_output+0x11d/0x220
[<ffffffffbd0a3e9c>] ip_finish_output+0x27c/0x8d0
[<ffffffffbd0a481b>] ip_output+0x7b/0xf0
[<ffffffffbd0a3c20>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffbd0a2247>] ip_local_out_sk+0x37/0x40
[<ffffffffbd0a5286>] ip_send_skb+0x16/0x50
[<ffffffffbd0cd65c>] udp_send_skb+0xac/0x2b0
[<ffffffffbd0a1f80>] ? ip_copy_metadata+0x170/0x170
[<ffffffffbd0cebc1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffbcad3c52>] ? __wake_up_common+0x82/0x120
[<ffffffffbcc9c4a8>] ? ep_poll_callback+0xf8/0x220
[<ffffffffbcd0b1a5>] ? sock_has_perm+0x75/0x90
[<ffffffffbd0dc389>] inet_sendmsg+0x69/0xb0
[<ffffffffbcd0b2d3>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffffbd0333a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffbd11421b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffbcd1936a>] ? sidtab_find_context+0x18a/0x1a0
[<ffffffffbd034269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffbd09aacf>] ? __ip_route_output_key_hash+0x20f/0x7d0
[<ffffffffbd0cb06f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffbd18856e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffbd0ce13f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffbd18856e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffbd038a18>] ? release_sock+0x128/0x180
[<ffffffffbd0cb4d7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffbd0db014>] ? inet_dgram_connect+0x34/0x90
[<ffffffffbd03396d>] ? SYSC_connect+0xed/0x130
[<ffffffffbd035921>] __sys_sendmsg+0x51/0x90
[<ffffffffbd035972>] SyS_sendmsg+0x12/0x20
[<ffffffffbd192ed2>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #10367
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0263877
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffb823b6f7
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffb8253386
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffffb828131a
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffb8256301
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x491
6
? local_bh_enable
vmlinux 0xffffffffb7ca4b17
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x17
7
dev_queue_xmit
vmlinux 0xffffffffb82564d0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x10
8
neigh_resolve_output
vmlinux 0xffffffffb826312d
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x11d
9
ip_finish_output
vmlinux 0xffffffffb82a3e9c
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x27c
10
ip_output
vmlinux 0xffffffffb82a481b
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x7b
11
? __ip_append_data.isra.50
vmlinux 0xffffffffb82a3c20
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xa60
12
ip_local_out_sk
vmlinux 0xffffffffb82a2247
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x37
13
ip_send_skb
vmlinux 0xffffffffb82a5286
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x16
14
udp_send_skb
vmlinux 0xffffffffb82cd65c
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xac
15
? ip_copy_metadata
vmlinux 0xffffffffb82a1f80
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x170
16
udp_sendmsg
vmlinux 0xffffffffb82cebc1
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x5e1
17
? __pollwait
vmlinux 0xffffffffb7e63430
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xf0
18
? sock_has_perm
vmlinux 0xffffffffb7f0b1d5
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x75
19
inet_sendmsg
vmlinux 0xffffffffb82dc389
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x69
20
? selinux_socket_sendmsg
vmlinux 0xffffffffb7f0b303
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x23
21
sock_sendmsg
vmlinux 0xffffffffb82333a6
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xb6
22
? sidtab_find_context
vmlinux 0xffffffffb7f1939a
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x18a
23
___sys_sendmsg
vmlinux 0xffffffffb8234269
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x3e9
24
? __ip_route_output_key_hash
vmlinux 0xffffffffb829aacf
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x20f
25
? ip4_datagram_release_cb
vmlinux 0xffffffffb82cb06f
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x6f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffffb838856e
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x1e
27
? udp_lib_rehash
vmlinux 0xffffffffb82ce13f
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x11f
28
? _raw_spin_unlock_bh
vmlinux 0xffffffffb838856e
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x1e
29
? release_sock
vmlinux 0xffffffffb8238a18
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x128
30
? ip4_datagram_connect
vmlinux 0xffffffffb82cb4d7
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x37
31
? inet_dgram_connect
vmlinux 0xffffffffb82db014
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x34
32
? SYSC_connect
vmlinux 0xffffffffb823396d
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xed
33
__sys_sendmsg
vmlinux 0xffffffffb8235921
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x51
34
SyS_sendmsg
vmlinux 0xffffffffb8235972
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x12
35
system_call_fastpath
vmlinux 0xffffffffb8392ed2
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 263s! [chronyd:707]
Modules linked in: tcp_lp nls_utf8 isofs rfcomm fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper snd_ens1371 cryptd snd_rawmidi snd_ac97_codec ac97_bus snd_seq btusb
btrtl btbcm btintel vmw_balloon bluetooth snd_seq_device snd_pcm snd_timer snd pcspkr soundcore rfkill joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm crct10dif_pclmul crct10dif_common libata crc32c_intel mptspi serio_raw scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 707 Comm: chronyd Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
task: ffff91be3b89b150 ti: ffff91be3b9fc000 task.ti: ffff91be3b9fc000
RIP: 0010:[<ffffffffc0263877>]  [<ffffffffc0263877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff91be3b9ff710  EFLAGS: 00000286
RAX: ffffb1a180283818 RBX: ffffffffb823be73 RCX: 00000000000000d0
RDX: 00000000000000d0 RSI: ffff91be362f1100 RDI: ffff91be3b3c4000
RBP: ffff91be3b9ff790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
CPU: 0 PID: 707 Comm: chronyd Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
task: ffff91be3b89b150 ti: ffff91be3b9fc000 task.ti: ffff91be3b9fc000
RIP: 0010:[<ffffffffc0263877>]  [<ffffffffc0263877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff91be3b9ff710  EFLAGS: 00000286
RAX: ffffb1a180283818 RBX: ffffffffb823be73 RCX: 00000000000000d0
RDX: 00000000000000d0 RSI: ffff91be362f1100 RDI: ffff91be3b3c4000
RBP: ffff91be3b9ff790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
FS:  00007ffae85af740(0000) GS:ffff91be3d600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2283655000 CR3: 000000003b608000 CR4: 00000000001607f0
Call Trace:
[<ffffffffb823b6f7>] ? kfree_skbmem+0x37/0x90
[<ffffffffb8253386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb828131a>] sch_direct_xmit+0x11a/0x250
[<ffffffffb8256301>] __dev_queue_xmit+0x491/0x650
[<ffffffffb7ca4b17>] ? local_bh_enable+0x17/0x20
[<ffffffffb82564d0>] dev_queue_xmit+0x10/0x20
[<ffffffffb826312d>] neigh_resolve_output+0x11d/0x220
[<ffffffffb82a3e9c>] ip_finish_output+0x27c/0x8d0
[<ffffffffb82a481b>] ip_output+0x7b/0xf0
[<ffffffffb82a3c20>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffb82a2247>] ip_local_out_sk+0x37/0x40
[<ffffffffb82a5286>] ip_send_skb+0x16/0x50
[<ffffffffb82cd65c>] udp_send_skb+0xac/0x2b0
[<ffffffffb82a1f80>] ? ip_copy_metadata+0x170/0x170
[<ffffffffb82cebc1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffb7e63430>] ? __pollwait+0xf0/0xf0
[<ffffffffb7f0b1d5>] ? sock_has_perm+0x75/0x90
[<ffffffffb82dc389>] inet_sendmsg+0x69/0xb0
[<ffffffffb7f0b303>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffffb82333a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffb7f1939a>] ? sidtab_find_context+0x18a/0x1a0
[<ffffffffb8234269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffb829aacf>] ? __ip_route_output_key_hash+0x20f/0x7d0
[<ffffffffb82cb06f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffb838856e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb82ce13f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffb838856e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb8238a18>] ? release_sock+0x128/0x180
[<ffffffffb82cb4d7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffb82db014>] ? inet_dgram_connect+0x34/0x90
[<ffffffffb823396d>] ? SYSC_connect+0xed/0x130
[<ffffffffb8235921>] __sys_sendmsg+0x51/0x90
[<ffffffffb8235972>] SyS_sendmsg+0x12/0x20
[<ffffffffb8392ed2>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #1320103
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0113857
Build id: 3.10.0-957.5.1.el7.x86_64 		0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffb5621717
Build id: 3.10.0-957.5.1.el7.x86_64 		0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffb5638ce6
Build id: 3.10.0-957.5.1.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffffb5665d6a
Build id: 3.10.0-957.5.1.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffb563bc61
Build id: 3.10.0-957.5.1.el7.x86_64 		0x491
6
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc06a52ce
Build id: 3.10.0-957.5.1.el7.x86_64 		0x4e
7
dev_queue_xmit
vmlinux 0xffffffffb563be30
Build id: 3.10.0-957.5.1.el7.x86_64 		0x10
8
ip_finish_output
vmlinux 0xffffffffb56887b6
Build id: 3.10.0-957.5.1.el7.x86_64 		0x556
9
ip_output
vmlinux 0xffffffffb5688d13
Build id: 3.10.0-957.5.1.el7.x86_64 		0x73
10
? __ip_append_data.isra.51
vmlinux 0xffffffffb5688260
Build id: 3.10.0-957.5.1.el7.x86_64 		0xa50
11
ip_local_out_sk
vmlinux 0xffffffffb5686897
Build id: 3.10.0-957.5.1.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffffb5689776
Build id: 3.10.0-957.5.1.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffffb56b158c
Build id: 3.10.0-957.5.1.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffffb56865e0
Build id: 3.10.0-957.5.1.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffffb56b2875
Build id: 3.10.0-957.5.1.el7.x86_64 		0x335
16
? __pollwait
vmlinux 0xffffffffb5257320
Build id: 3.10.0-957.5.1.el7.x86_64 		0xf0
17
? sock_has_perm
vmlinux 0xffffffffb52fc985
Build id: 3.10.0-957.5.1.el7.x86_64 		0x75
18
inet_sendmsg
vmlinux 0xffffffffb56bff39
Build id: 3.10.0-957.5.1.el7.x86_64 		0x69
19
? selinux_socket_sendmsg
vmlinux 0xffffffffb52fcab3
Build id: 3.10.0-957.5.1.el7.x86_64 		0x23
20
sock_sendmsg
vmlinux 0xffffffffb56193a6
Build id: 3.10.0-957.5.1.el7.x86_64 		0xb6
21
___sys_sendmsg
vmlinux 0xffffffffb561a269
Build id: 3.10.0-957.5.1.el7.x86_64 		0x3e9
22
? ip4_datagram_release_cb
vmlinux 0xffffffffb56af00f
Build id: 3.10.0-957.5.1.el7.x86_64 		0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffffb576a31e
Build id: 3.10.0-957.5.1.el7.x86_64 		0x1e
24
? udp_lib_rehash
vmlinux 0xffffffffb56b1e8f
Build id: 3.10.0-957.5.1.el7.x86_64 		0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffb576a31e
Build id: 3.10.0-957.5.1.el7.x86_64 		0x1e
26
? release_sock
vmlinux 0xffffffffb561ea00
Build id: 3.10.0-957.5.1.el7.x86_64 		0x120
27
? ip4_datagram_connect
vmlinux 0xffffffffb56af477
Build id: 3.10.0-957.5.1.el7.x86_64 		0x37
28
? inet_dgram_connect
vmlinux 0xffffffffb56bebc4
Build id: 3.10.0-957.5.1.el7.x86_64 		0x34
29
? SYSC_connect
vmlinux 0xffffffffb561996d
Build id: 3.10.0-957.5.1.el7.x86_64 		0xed
30
__sys_sendmsg
vmlinux 0xffffffffb561b921
Build id: 3.10.0-957.5.1.el7.x86_64 		0x51
31
SyS_sendmsg
vmlinux 0xffffffffb561b972
Build id: 3.10.0-957.5.1.el7.x86_64 		0x12
32
system_call_fastpath
vmlinux 0xffffffffb5774ddb
Build id: 3.10.0-957.5.1.el7.x86_64 		0x22 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 39s! [chronyd:9741]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat vmhgfs(OE) ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr snd_seq_midi snd_seq_midi_event snd_ens1371 btusb snd_rawmidi btrtl btbcm
snd_ac97_codec btintel bluetooth ac97_bus snd_seq snd_seq_device snd_pcm rfkill snd_timer snd sg soundcore parport_pc parport vmw_vmci i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmwgfx ata_generic pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm e1000 mptspi ata_piix scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm
CPU: 1 PID: 9741 Comm: chronyd Kdump: loaded Tainted: G           OE  ------------   3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff91a9ed25a080 ti: ffff91a879be4000 task.ti: ffff91a879be4000
RIP: 0010:[<ffffffffc0113857>]  [<ffffffffc0113857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff91a879be7760  EFLAGS: 00010286
RAX: ffff9d05c1043818 RBX: ffffffffb5622c73 RCX: 0000000000000043
RDX: 0000000000000043 RSI: ffff91a9f1a7ff40 RDI: ffff91a9ff89e000
RBP: ffff91a879be77e0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 00000001c066e915 R15: ffffffffb53a0a90
FS:  00007f4ece4b5740(0000) GS:ffff91a9f5640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f78a6aeb008 CR3: 00000000b9ba8000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb5621717>] ? kfree_skbmem+0x37/0x90
[<ffffffffb5638ce6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb5665d6a>] sch_direct_xmit+0x11a/0x250
[<ffffffffb563bc61>] __dev_queue_xmit+0x491/0x650
[<ffffffffc06a52ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffffb563be30>] dev_queue_xmit+0x10/0x20
[<ffffffffb56887b6>] ip_finish_output+0x556/0x7b0
[<ffffffffb5688d13>] ip_output+0x73/0xe0
[<ffffffffb5688260>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffb5686897>] ip_local_out_sk+0x37/0x40
[<ffffffffb5689776>] ip_send_skb+0x16/0x50
[<ffffffffb56b158c>] udp_send_skb+0xac/0x2b0
[<ffffffffb56865e0>] ? ip_copy_metadata+0x170/0x170
[<ffffffffb56b2875>] udp_sendmsg+0x335/0xa10
[<ffffffffb5257320>] ? __pollwait+0xf0/0xf0
[<ffffffffb52fc985>] ? sock_has_perm+0x75/0x90
[<ffffffffb56bff39>] inet_sendmsg+0x69/0xb0
[<ffffffffb52fcab3>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffffb56193a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffb561a269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffb56af00f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffb576a31e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb56b1e8f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffb576a31e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb561ea00>] ? release_sock+0x120/0x170
[<ffffffffb56af477>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffb56bebc4>] ? inet_dgram_connect+0x34/0x90
[<ffffffffb561996d>] ? SYSC_connect+0xed/0x130
[<ffffffffb561b921>] __sys_sendmsg+0x51/0x90
[<ffffffffb561b972>] SyS_sendmsg+0x12/0x20
[<ffffffffb5774ddb>] system_call_fastpath+0x22/0x27
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #1238564
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0266877
Build id: 3.10.0-1062.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff9a04d346
Build id: 3.10.0-1062.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffff9a07b0ba
Build id: 3.10.0-1062.el7.x86_64 		0x11a
4
__qdisc_run
vmlinux 0xffffffff9a07b27e
Build id: 3.10.0-1062.el7.x86_64 		0x8e
5
__dev_queue_xmit
vmlinux 0xffffffff9a0502c9
Build id: 3.10.0-1062.el7.x86_64 		0x4a9
6
dev_queue_xmit
vmlinux 0xffffffff9a050480
Build id: 3.10.0-1062.el7.x86_64 		0x10
7
ip_finish_output
vmlinux 0xffffffff9a09e97b
Build id: 3.10.0-1062.el7.x86_64 		0x67b
8
ip_output
vmlinux 0xffffffff9a09eefb
Build id: 3.10.0-1062.el7.x86_64 		0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffff9a09e300
Build id: 3.10.0-1062.el7.x86_64 		0xa60
10
ip_local_out_sk
vmlinux 0xffffffff9a09c927
Build id: 3.10.0-1062.el7.x86_64 		0x37
11
ip_send_skb
vmlinux 0xffffffff9a09f966
Build id: 3.10.0-1062.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffff9a0c7c6c
Build id: 3.10.0-1062.el7.x86_64 		0xac
13
? ip_copy_metadata
vmlinux 0xffffffff9a09c660
Build id: 3.10.0-1062.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffff9a0c91d1
Build id: 3.10.0-1062.el7.x86_64 		0x5e1
15
? __pollwait
vmlinux 0xffffffff99c5e540
Build id: 3.10.0-1062.el7.x86_64 		0xf0
16
? __pollwait
vmlinux 0xffffffff99c5e540
Build id: 3.10.0-1062.el7.x86_64 		0xf0
17
inet_sendmsg
vmlinux 0xffffffff9a0d6939
Build id: 3.10.0-1062.el7.x86_64 		0x69
18
? __pollwait
vmlinux 0xffffffff99c5e540
Build id: 3.10.0-1062.el7.x86_64 		0xf0
19
sock_sendmsg
vmlinux 0xffffffff9a02d3a6
Build id: 3.10.0-1062.el7.x86_64 		0xb6
20
? __pollwait
vmlinux 0xffffffff99c5e540
Build id: 3.10.0-1062.el7.x86_64 		0xf0
21
___sys_sendmsg
vmlinux 0xffffffff9a02e269
Build id: 3.10.0-1062.el7.x86_64 		0x3e9
22
? __ip_route_output_key_hash
vmlinux 0xffffffff9a095257
Build id: 3.10.0-1062.el7.x86_64 		0x207
23
? ip4_datagram_release_cb
vmlinux 0xffffffff9a0c567f
Build id: 3.10.0-1062.el7.x86_64 		0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff9a18196e
Build id: 3.10.0-1062.el7.x86_64 		0x1e
25
? udp_lib_rehash
vmlinux 0xffffffff9a0c874f
Build id: 3.10.0-1062.el7.x86_64 		0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff9a18196e
Build id: 3.10.0-1062.el7.x86_64 		0x1e
27
? release_sock
vmlinux 0xffffffff9a032a10
Build id: 3.10.0-1062.el7.x86_64 		0x120
28
? ip4_datagram_connect
vmlinux 0xffffffff9a0c5ae7
Build id: 3.10.0-1062.el7.x86_64 		0x37
29
? inet_dgram_connect
vmlinux 0xffffffff9a0d55c4
Build id: 3.10.0-1062.el7.x86_64 		0x34
30
? SYSC_connect
vmlinux 0xffffffff9a02d96d
Build id: 3.10.0-1062.el7.x86_64 		0xed
31
__sys_sendmsg
vmlinux 0xffffffff9a02f921
Build id: 3.10.0-1062.el7.x86_64 		0x51
32
SyS_sendmsg
vmlinux 0xffffffff9a02f972
Build id: 3.10.0-1062.el7.x86_64 		0x12
33
system_call_fastpath
vmlinux 0xffffffff9a18bede
Build id: 3.10.0-1062.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [chronyd:1502]
Modules linked in: xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype nf_conntrack_netlink iptable_filter tcp_lp veth fuse xt_nat br_netfilter overlay(T) tun bridge stp llc devlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper pcspkr joydev cryptd sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul drm crct10dif_common nfit crc32c_intel libata libnvdimm mptspi e1000 scsi_transport_spi serio_raw mptscsih
mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: iptable_filter]
CPU: 2 PID: 1502 Comm: chronyd Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9b19a80ed230 ti: ffff9b19aeed4000 task.ti: ffff9b19aeed4000
RIP: 0010:[<ffffffffc0266877>]  [<ffffffffc0266877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9b19aeed76f8  EFLAGS: 00000286
RAX: ffffa65783203818 RBX: 000000000000004a RCX: 000000000000001b
RDX: 000000000000001b RSI: ffff9b19ab59f600 RDI: ffff9b19aee76000
RBP: ffff9b19aeed7778 R08: 0000000000000000 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: 000000045e3998c6
R13: 00000001aeed76e8 R14: ffffffff99daabc0 R15: ffff9b19aeed76b0
FS:  00007f568e524740(0000) GS:ffff9b19af880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fffa6ac4030 CR3: 00000005656a4000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9a04d346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff9a07b0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffff9a07b27e>] __qdisc_run+0x8e/0x360
[<ffffffff9a0502c9>] __dev_queue_xmit+0x4a9/0x650
[<ffffffff9a050480>] dev_queue_xmit+0x10/0x20
[<ffffffff9a09e97b>] ip_finish_output+0x67b/0x8d0
[<ffffffff9a09eefb>] ip_output+0x7b/0xf0
[<ffffffff9a09e300>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff9a09c927>] ip_local_out_sk+0x37/0x40
[<ffffffff9a09f966>] ip_send_skb+0x16/0x50
[<ffffffff9a0c7c6c>] udp_send_skb+0xac/0x2b0
[<ffffffff9a09c660>] ? ip_copy_metadata+0x170/0x170
[<ffffffff9a0c91d1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff99c5e540>] ? __pollwait+0xf0/0xf0
[<ffffffff99c5e540>] ? __pollwait+0xf0/0xf0
[<ffffffff9a0d6939>] inet_sendmsg+0x69/0xb0
[<ffffffff99c5e540>] ? __pollwait+0xf0/0xf0
[<ffffffff9a02d3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff99c5e540>] ? __pollwait+0xf0/0xf0
[<ffffffff9a02e269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff9a095257>] ? __ip_route_output_key_hash+0x207/0x7b0
[<ffffffff9a0c567f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff9a18196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff9a0c874f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff9a18196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff9a032a10>] ? release_sock+0x120/0x170
[<ffffffff9a0c5ae7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff9a0d55c4>] ? inet_dgram_connect+0x34/0x90
[<ffffffff9a02d96d>] ? SYSC_connect+0xed/0x130
[<ffffffff9a02f921>] __sys_sendmsg+0x51/0x90
[<ffffffff9a02f972>] SyS_sendmsg+0x12/0x20
[<ffffffff9a18bede>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #1112492
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc011b877
Build id: 3.10.0-1062.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffb604d346
Build id: 3.10.0-1062.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffffb607b0ba
Build id: 3.10.0-1062.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffb60502b1
Build id: 3.10.0-1062.el7.x86_64 		0x491
5
dev_queue_xmit
vmlinux 0xffffffffb6050480
Build id: 3.10.0-1062.el7.x86_64 		0x10
6
neigh_resolve_output
vmlinux 0xffffffffb605cfed
Build id: 3.10.0-1062.el7.x86_64 		0x11d
7
ip_finish_output
vmlinux 0xffffffffb609e57c
Build id: 3.10.0-1062.el7.x86_64 		0x27c
8
ip_output
vmlinux 0xffffffffb609eefb
Build id: 3.10.0-1062.el7.x86_64 		0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffffb609e300
Build id: 3.10.0-1062.el7.x86_64 		0xa60
10
ip_local_out_sk
vmlinux 0xffffffffb609c927
Build id: 3.10.0-1062.el7.x86_64 		0x37
11
ip_send_skb
vmlinux 0xffffffffb609f966
Build id: 3.10.0-1062.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffffb60c7c6c
Build id: 3.10.0-1062.el7.x86_64 		0xac
13
? ip_copy_metadata
vmlinux 0xffffffffb609c660
Build id: 3.10.0-1062.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffffb60c91d1
Build id: 3.10.0-1062.el7.x86_64 		0x5e1
15
? __alloc_skb
vmlinux 0xffffffffb603716d
Build id: 3.10.0-1062.el7.x86_64 		0x8d
16
? ep_poll_callback
vmlinux 0xffffffffb5c96f4e
Build id: 3.10.0-1062.el7.x86_64 		0xee
17
? __alloc_skb
vmlinux 0xffffffffb6037181
Build id: 3.10.0-1062.el7.x86_64 		0xa1
18
? __wake_up_common
vmlinux 0xffffffffb5ad24d2
Build id: 3.10.0-1062.el7.x86_64 		0x82
19
inet_sendmsg
vmlinux 0xffffffffb60d6939
Build id: 3.10.0-1062.el7.x86_64 		0x69
20
sock_sendmsg
vmlinux 0xffffffffb602d3a6
Build id: 3.10.0-1062.el7.x86_64 		0xb6
21
? unix_dgram_sendmsg
vmlinux 0xffffffffb610e6db
Build id: 3.10.0-1062.el7.x86_64 		0x3cb
22
___sys_sendmsg
vmlinux 0xffffffffb602e269
Build id: 3.10.0-1062.el7.x86_64 		0x3e9
23
? __ip_route_output_key_hash
vmlinux 0xffffffffb6095257
Build id: 3.10.0-1062.el7.x86_64 		0x207
24
? ip4_datagram_release_cb
vmlinux 0xffffffffb60c567f
Build id: 3.10.0-1062.el7.x86_64 		0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffb618196e
Build id: 3.10.0-1062.el7.x86_64 		0x1e
26
? udp_lib_rehash
vmlinux 0xffffffffb60c874f
Build id: 3.10.0-1062.el7.x86_64 		0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffb618196e
Build id: 3.10.0-1062.el7.x86_64 		0x1e
28
? release_sock
vmlinux 0xffffffffb6032a10
Build id: 3.10.0-1062.el7.x86_64 		0x120
29
? ip4_datagram_connect
vmlinux 0xffffffffb60c5ae7
Build id: 3.10.0-1062.el7.x86_64 		0x37
30
? inet_dgram_connect
vmlinux 0xffffffffb60d55c4
Build id: 3.10.0-1062.el7.x86_64 		0x34
31
? SYSC_connect
vmlinux 0xffffffffb602d96d
Build id: 3.10.0-1062.el7.x86_64 		0xed
32
__sys_sendmsg
vmlinux 0xffffffffb602f921
Build id: 3.10.0-1062.el7.x86_64 		0x51
33
SyS_sendmsg
vmlinux 0xffffffffb602f972
Build id: 3.10.0-1062.el7.x86_64 		0x12
34
system_call_fastpath
vmlinux 0xffffffffb618bede
Build id: 3.10.0-1062.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 26s! [chronyd:1365]
Modules linked in: nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev pcspkr parport_pc parport snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper nfit syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libnvdimm mptspi crct10dif_pclmul crct10dif_common libata scsi_transport_spi mptscsih crc32c_intel e1000 mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 1365 Comm: chronyd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff93a768739070 ti: ffff93a7692fc000 task.ti: ffff93a7692fc000
RIP: 0010:[<ffffffffc011b877>]  [<ffffffffc011b877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff93a7692ff710  EFLAGS: 00000286
RAX: ffffab7881cc3818 RBX: ffffffffb6035e63 RCX: 0000000000000094
RDX: 0000000000000094 RSI: ffff93a769029bc0 RDI: ffff93a666bfe000
RBP: ffff93a7692ff790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 00000001669d8400 R15: ffffffffb5daabc0
FS:  00007f1aee32f740(0000) GS:ffff93a76d780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f33230b1f13 CR3: 00000003254fa000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffb604d346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb607b0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffffb60502b1>] __dev_queue_xmit+0x491/0x650
[<ffffffffb6050480>] dev_queue_xmit+0x10/0x20
[<ffffffffb605cfed>] neigh_resolve_output+0x11d/0x220
[<ffffffffb609e57c>] ip_finish_output+0x27c/0x8d0
[<ffffffffb609eefb>] ip_output+0x7b/0xf0
[<ffffffffb609e300>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffb609c927>] ip_local_out_sk+0x37/0x40
[<ffffffffb609f966>] ip_send_skb+0x16/0x50
[<ffffffffb60c7c6c>] udp_send_skb+0xac/0x2b0
[<ffffffffb609c660>] ? ip_copy_metadata+0x170/0x170
[<ffffffffb60c91d1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffb603716d>] ? __alloc_skb+0x8d/0x2d0
[<ffffffffb5c96f4e>] ? ep_poll_callback+0xee/0x210
[<ffffffffb6037181>] ? __alloc_skb+0xa1/0x2d0
[<ffffffffb5ad24d2>] ? __wake_up_common+0x82/0x120
[<ffffffffb60d6939>] inet_sendmsg+0x69/0xb0
[<ffffffffb602d3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffb610e6db>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffb602e269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffb6095257>] ? __ip_route_output_key_hash+0x207/0x7b0
[<ffffffffb60c567f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffb618196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb60c874f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffb618196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb6032a10>] ? release_sock+0x120/0x170
[<ffffffffb60c5ae7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffb60d55c4>] ? inet_dgram_connect+0x34/0x90
[<ffffffffb602d96d>] ? SYSC_connect+0xed/0x130
[<ffffffffb602f921>] __sys_sendmsg+0x51/0x90
[<ffffffffb602f972>] SyS_sendmsg+0x12/0x20
[<ffffffffb618bede>] system_call_fastpath+0x25/0x2a
Complete report #1093212
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00e98e7
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff92a7edc6
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffff92aadd72
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x122
4
__dev_queue_xmit
vmlinux 0xffffffff92a81dd1
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x491
5
dev_queue_xmit
vmlinux 0xffffffff92a81fa0
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x10
6
neigh_resolve_output
vmlinux 0xffffffff92a8f03d
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x11d
7
ip_finish_output
vmlinux 0xffffffff92ad220c
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x27c
8
ip_output
vmlinux 0xffffffff92ad2b9b
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffff92ad1f90
Build id: 3.10.0-1160.102.1.el7.x86_64 		0xa60
10
ip_local_out_sk
vmlinux 0xffffffff92ad0587
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x37
11
ip_send_skb
vmlinux 0xffffffff92ad3636
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffff92afc3ac
Build id: 3.10.0-1160.102.1.el7.x86_64 		0xac
13
? ip_copy_metadata
vmlinux 0xffffffff92ad02b0
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffff92afd9a1
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x5e1
15
? __pollwait
vmlinux 0xffffffff92672500
Build id: 3.10.0-1160.102.1.el7.x86_64 		0xf0
16
? __pollwait
vmlinux 0xffffffff92672500
Build id: 3.10.0-1160.102.1.el7.x86_64 		0xf0
17
inet_sendmsg
vmlinux 0xffffffff92b0b4e9
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x69
18
? __pollwait
vmlinux 0xffffffff92672500
Build id: 3.10.0-1160.102.1.el7.x86_64 		0xf0
19
sock_sendmsg
vmlinux 0xffffffff92a5e4f6
Build id: 3.10.0-1160.102.1.el7.x86_64 		0xb6
20
___sys_sendmsg
vmlinux 0xffffffff92a5f3f9
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x3e9
21
? __ip_route_output_key_hash
vmlinux 0xffffffff92ac8c63
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x223
22
? ip4_datagram_release_cb
vmlinux 0xffffffff92af9cdf
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffff92bba56e
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x1e
24
? udp_lib_rehash
vmlinux 0xffffffff92afcecf
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffff92bba56e
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x1e
26
? release_sock
vmlinux 0xffffffff92a63ea8
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x128
27
? ip4_datagram_connect
vmlinux 0xffffffff92afa147
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x37
28
? inet_dgram_connect
vmlinux 0xffffffff92b0a0e4
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x34
29
__sys_sendmsg
vmlinux 0xffffffff92a60b41
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x51
30
SyS_sendmsg
vmlinux 0xffffffff92a60b92
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x12
31
system_call_fastpath
vmlinux 0xffffffff92bc539a
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [chronyd:625]
Modules linked in: nfnetlink_queue nfnetlink_log bluetooth rfkill binfmt_misc isofs loop veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul snd_ens1371 vmw_balloon glue_helper snd_rawmidi snd_ac97_codec ablk_helper cryptd ppdev ac97_bus snd_seq snd_seq_device joydev pcspkr snd_pcm snd_timer snd soundcore sg parport_pc parport
i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit crct10dif_pclmul crct10dif_common crc32c_intel libnvdimm ata_piix mptspi libata e1000 scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 1 PID: 625 Comm: chronyd Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.102.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9469267f4200 ti: ffff94692be24000 task.ti: ffff94692be24000
RIP: 0010:[<ffffffffc00e98e7>]  [<ffffffffc00e98e7>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff94692be27710  EFLAGS: 00000286
RAX: ffffaa7181c03818 RBX: ffffffff92a67453 RCX: 00000000000000e1
RDX: 00000000000000e1 RSI: ffff946928d6b140 RDI: ffff94692c438000
RBP: ffff94692be27790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffff9469202ff500 R15: 0000000100010000
FS:  00007fd2c7246740(0000) GS:ffff94692d640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7139a132e0 CR3: 000000042be68000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff92a7edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff92aadd72>] sch_direct_xmit+0x122/0x250
[<ffffffff92a81dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffff92a81fa0>] dev_queue_xmit+0x10/0x20
[<ffffffff92a8f03d>] neigh_resolve_output+0x11d/0x220
[<ffffffff92ad220c>] ip_finish_output+0x27c/0x8d0
[<ffffffff92ad2b9b>] ip_output+0x7b/0x100
[<ffffffff92ad1f90>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff92ad0587>] ip_local_out_sk+0x37/0x40
[<ffffffff92ad3636>] ip_send_skb+0x16/0x50
[<ffffffff92afc3ac>] udp_send_skb+0xac/0x2b0
[<ffffffff92ad02b0>] ? ip_copy_metadata+0x170/0x170
[<ffffffff92afd9a1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff92672500>] ? __pollwait+0xf0/0xf0
[<ffffffff92672500>] ? __pollwait+0xf0/0xf0
[<ffffffff92b0b4e9>] inet_sendmsg+0x69/0xb0
[<ffffffff92672500>] ? __pollwait+0xf0/0xf0
[<ffffffff92a5e4f6>] sock_sendmsg+0xb6/0x100
[<ffffffff92a5f3f9>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff92ac8c63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffff92af9cdf>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff92bba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff92afcecf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff92bba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff92a63ea8>] ? release_sock+0x128/0x180
[<ffffffff92afa147>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff92b0a0e4>] ? inet_dgram_connect+0x34/0x90
[<ffffffff92a60b41>] __sys_sendmsg+0x51/0x90
[<ffffffff92a60b92>] SyS_sendmsg+0x12/0x30
[<ffffffff92bc539a>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc
Complete report #1085513
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc0287bcc
Build id: 3.10.0-957.21.3.el7.x86_64 		0xaec
2
? nf_nat_setup_info
nf_nat 0xffffffffc09b03b7
Build id: 3.10.0-957.21.3.el7.x86_64 		0x2f7
3
dev_hard_start_xmit
vmlinux 0xffffffffb9e39ce6
Build id: 3.10.0-957.21.3.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffffb9e66d6a
Build id: 3.10.0-957.21.3.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffb9e3cc61
Build id: 3.10.0-957.21.3.el7.x86_64 		0x491
6
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc09a12ce
Build id: 3.10.0-957.21.3.el7.x86_64 		0x4e
7
dev_queue_xmit
vmlinux 0xffffffffb9e3ce30
Build id: 3.10.0-957.21.3.el7.x86_64 		0x10
8
ip_finish_output
vmlinux 0xffffffffb9e897b6
Build id: 3.10.0-957.21.3.el7.x86_64 		0x556
9
ip_output
vmlinux 0xffffffffb9e89d13
Build id: 3.10.0-957.21.3.el7.x86_64 		0x73
10
? __ip_append_data.isra.51
vmlinux 0xffffffffb9e89260
Build id: 3.10.0-957.21.3.el7.x86_64 		0xa50
11
ip_local_out_sk
vmlinux 0xffffffffb9e87897
Build id: 3.10.0-957.21.3.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffffb9e8a776
Build id: 3.10.0-957.21.3.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffffb9eb26ac
Build id: 3.10.0-957.21.3.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffffb9e875e0
Build id: 3.10.0-957.21.3.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffffb9eb3995
Build id: 3.10.0-957.21.3.el7.x86_64 		0x335
16
? default_wake_function
vmlinux 0xffffffffb98d6b72
Build id: 3.10.0-957.21.3.el7.x86_64 		0x12
17
? __wake_up_common
vmlinux 0xffffffffb98cbc0b
Build id: 3.10.0-957.21.3.el7.x86_64 		0x5b
18
? ep_poll_callback
vmlinux 0xffffffffb9a8f53e
Build id: 3.10.0-957.21.3.el7.x86_64 		0xee
19
inet_sendmsg
vmlinux 0xffffffffb9ec1199
Build id: 3.10.0-957.21.3.el7.x86_64 		0x69
20
? selinux_socket_sendmsg
vmlinux 0xffffffffb9afcd83
Build id: 3.10.0-957.21.3.el7.x86_64 		0x23
21
sock_sendmsg
vmlinux 0xffffffffb9e1a3a6
Build id: 3.10.0-957.21.3.el7.x86_64 		0xb6
22
? unix_dgram_sendmsg
vmlinux 0xffffffffb9ef8a1b
Build id: 3.10.0-957.21.3.el7.x86_64 		0x3cb
23
___sys_sendmsg
vmlinux 0xffffffffb9e1b269
Build id: 3.10.0-957.21.3.el7.x86_64 		0x3e9
24
? ip4_datagram_release_cb
vmlinux 0xffffffffb9eb012f
Build id: 3.10.0-957.21.3.el7.x86_64 		0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffb9f6b5fe
Build id: 3.10.0-957.21.3.el7.x86_64 		0x1e
26
? udp_lib_rehash
vmlinux 0xffffffffb9eb2faf
Build id: 3.10.0-957.21.3.el7.x86_64 		0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffb9f6b5fe
Build id: 3.10.0-957.21.3.el7.x86_64 		0x1e
28
? release_sock
vmlinux 0xffffffffb9e1fa00
Build id: 3.10.0-957.21.3.el7.x86_64 		0x120
29
? ip4_datagram_connect
vmlinux 0xffffffffb9eb0597
Build id: 3.10.0-957.21.3.el7.x86_64 		0x37
30
? inet_dgram_connect
vmlinux 0xffffffffb9ebfe24
Build id: 3.10.0-957.21.3.el7.x86_64 		0x34
31
? SYSC_connect
vmlinux 0xffffffffb9e1a96d
Build id: 3.10.0-957.21.3.el7.x86_64 		0xed
32
__sys_sendmsg
vmlinux 0xffffffffb9e1c921
Build id: 3.10.0-957.21.3.el7.x86_64 		0x51
33
SyS_sendmsg
vmlinux 0xffffffffb9e1c972
Build id: 3.10.0-957.21.3.el7.x86_64 		0x12
34
system_call_fastpath
vmlinux 0xffffffffb9f75ddb
Build id: 3.10.0-957.21.3.el7.x86_64 		0x22
35
? system_call_after_swapgs
vmlinux 0xffffffffb9f75d21
Build id: 3.10.0-957.21.3.el7.x86_64 		0xae 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 38s! [iidbms:19431]
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 38s! [chronyd:6790]
Modules linked in: arc4 md4 xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 nfs fscache nls_utf8 cifs ccm dns_resolver vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp crc32_pclmul ext4 ghash_clmulni_intel mbcache jbd2 aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr parport_pc parport sg vmw_vmci i2c_piix4 nfsd nfs_acl lockd grace auth_rpcgss binfmt_misc sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm vmxnet3 mptspi scsi_transport_spi mptscsih ata_piix mptbase libata drm_panel_orientation_quirks involflt(OE) dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 6790 Comm: chronyd Kdump: loaded Tainted: G           OEL ------------   3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8b32b2ea0000 ti: ffff8b39b4270000 task.ti: ffff8b39b4270000
RIP: 0010:[<ffffffffc0287bcc>]  [<ffffffffc0287bcc>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0018:ffff8b39b4273750  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffff8b39b42737c8 RCX: 00000000000000b5
RDX: ffff9c414313a610 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff8b39b42737e0 R08: 0000000000000000 R09: 0000000000a0405a
R10: 0000000000000000 R11: ffff8b39ad366b40 R12: ffffffffc097a8bd
R13: ffff8b39b42736f8 R14: ffffffffb98a04f7 R15: ffff8b39b42736b8
FS:  00007fa5f249c740(0000) GS:ffff8b3a3fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3b53d78000 CR3: 00000007b426c000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc09b03b7>] ? nf_nat_setup_info+0x2f7/0x320 [nf_nat]
[<ffffffffb9e39ce6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb9e66d6a>] sch_direct_xmit+0x11a/0x250
[<ffffffffb9e3cc61>] __dev_queue_xmit+0x491/0x650
[<ffffffffc09a12ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffffb9e3ce30>] dev_queue_xmit+0x10/0x20
[<ffffffffb9e897b6>] ip_finish_output+0x556/0x7b0
[<ffffffffb9e89d13>] ip_output+0x73/0xe0
[<ffffffffb9e89260>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffb9e87897>] ip_local_out_sk+0x37/0x40
[<ffffffffb9e8a776>] ip_send_skb+0x16/0x50
[<ffffffffb9eb26ac>] udp_send_skb+0xac/0x2b0
[<ffffffffb9e875e0>] ? ip_copy_metadata+0x170/0x170
[<ffffffffb9eb3995>] udp_sendmsg+0x335/0xa10
[<ffffffffb98d6b72>] ? default_wake_function+0x12/0x20
[<ffffffffb98cbc0b>] ? __wake_up_common+0x5b/0x90
[<ffffffffb9a8f53e>] ? ep_poll_callback+0xee/0x210
[<ffffffffb9ec1199>] inet_sendmsg+0x69/0xb0
[<ffffffffb9afcd83>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffffb9e1a3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffb9ef8a1b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffb9e1b269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffb9eb012f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffb9f6b5fe>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb9eb2faf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffb9f6b5fe>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb9e1fa00>] ? release_sock+0x120/0x170
[<ffffffffb9eb0597>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffb9ebfe24>] ? inet_dgram_connect+0x34/0x90
[<ffffffffb9e1a96d>] ? SYSC_connect+0xed/0x130
[<ffffffffb9e1c921>] __sys_sendmsg+0x51/0x90
[<ffffffffb9e1c972>] SyS_sendmsg+0x12/0x20
[<ffffffffb9f75ddb>] system_call_fastpath+0x22/0x27
[<ffffffffb9f75d21>] ? system_call_after_swapgs+0xae/0x146
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41
Complete report #1004891
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc037e8e7
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffaee7edc6
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffffaeeadd72
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x122
4
__dev_queue_xmit
vmlinux 0xffffffffaee81dd1
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x491
5
? local_bh_enable
vmlinux 0xffffffffae8a89b7
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x17
6
dev_queue_xmit
vmlinux 0xffffffffaee81fa0
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x10
7
neigh_resolve_output
vmlinux 0xffffffffaee8f03d
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x11d
8
ip_finish_output
vmlinux 0xffffffffaeed220c
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x27c
9
ip_output
vmlinux 0xffffffffaeed2b9b
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffffaeed1f90
Build id: 3.10.0-1160.119.1.el7.x86_64 		0xa60
11
ip_local_out_sk
vmlinux 0xffffffffaeed0587
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffffaeed3636
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffffaeefc3ac
Build id: 3.10.0-1160.119.1.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffffaeed02b0
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffffaeefd9a1
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x5e1
16
inet_sendmsg
vmlinux 0xffffffffaef0b4e9
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x69
17
? radix_tree_lookup_slot
vmlinux 0xffffffffaeba3c02
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x22
18
sock_sendmsg
vmlinux 0xffffffffaee5e4f6
Build id: 3.10.0-1160.119.1.el7.x86_64 		0xb6
19
? xfs_iunlock
xfs 0xffffffffc0551ed4
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x124
20
___sys_sendmsg
vmlinux 0xffffffffaee5f3f9
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x3e9
21
? __ip_route_output_key_hash
vmlinux 0xffffffffaeec8c63
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x223
22
? ip4_datagram_release_cb
vmlinux 0xffffffffaeef9cdf
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffffaefba56e
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x1e
24
? udp_lib_rehash
vmlinux 0xffffffffaeefcecf
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffaefba56e
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x1e
26
? release_sock
vmlinux 0xffffffffaee63ea8
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x128
27
? ip4_datagram_connect
vmlinux 0xffffffffaeefa147
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x37
28
? inet_dgram_connect
vmlinux 0xffffffffaef0a0e4
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x34
29
__sys_sendmsg
vmlinux 0xffffffffaee60b41
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x51
30
SyS_sendmsg
vmlinux 0xffffffffaee60b92
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x12
31
system_call_fastpath
vmlinux 0xffffffffaefc539a
Build id: 3.10.0-1160.119.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 36s! [chronyd:694]
Modules linked in: xt_conntrack iptable_filter iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack sunrpc kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper pcspkr cryptd sg joydev virtio_balloon i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_blk bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata e1000 crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 694 Comm: chronyd Tainted: G             L ------------   3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
task: ffff8c34cc81d280 ti: ffff8c34cba14000 task.ti: ffff8c34cba14000
RIP: 0010:[<ffffffffc037e8e7>]  [<ffffffffc037e8e7>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8c34cba17710  EFLAGS: 00000286
RAX: ffffb06540143818 RBX: ffffffffaee67453 RCX: 000000000000005c
RDX: 000000000000005c RSI: ffff8c34cf1bd4c0 RDI: ffff8c34cbf04000
RBP: ffff8c34cba17790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffff8c34cd801700 R15: ffff8c34cd801700
FS:  00007f7dd3cdc740(0000) GS:ffff8c34cfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7dd2c97f00 CR3: 000000000c8cc000 CR4: 00000000003407f0
Call Trace:
[<ffffffffaee7edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffaeeadd72>] sch_direct_xmit+0x122/0x250
[<ffffffffaee81dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffffae8a89b7>] ? local_bh_enable+0x17/0x20
[<ffffffffaee81fa0>] dev_queue_xmit+0x10/0x20
[<ffffffffaee8f03d>] neigh_resolve_output+0x11d/0x220
[<ffffffffaeed220c>] ip_finish_output+0x27c/0x8d0
[<ffffffffaeed2b9b>] ip_output+0x7b/0x100
[<ffffffffaeed1f90>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffaeed0587>] ip_local_out_sk+0x37/0x40
[<ffffffffaeed3636>] ip_send_skb+0x16/0x50
[<ffffffffaeefc3ac>] udp_send_skb+0xac/0x2b0
[<ffffffffaeed02b0>] ? ip_copy_metadata+0x170/0x170
[<ffffffffaeefd9a1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffaef0b4e9>] inet_sendmsg+0x69/0xb0
[<ffffffffaeba3c02>] ? radix_tree_lookup_slot+0x22/0x50
[<ffffffffaee5e4f6>] sock_sendmsg+0xb6/0x100
[<ffffffffc0551ed4>] ? xfs_iunlock+0x124/0x130 [xfs]
[<ffffffffaee5f3f9>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffaeec8c63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffffaeef9cdf>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffaefba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffffaeefcecf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffaefba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffffaee63ea8>] ? release_sock+0x128/0x180
[<ffffffffaeefa147>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffaef0a0e4>] ? inet_dgram_connect+0x34/0x90
[<ffffffffaee60b41>] __sys_sendmsg+0x51/0x90
[<ffffffffaee60b92>] SyS_sendmsg+0x12/0x30
[<ffffffffaefc539a>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc
Complete report #992687
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000e 0xffffffffc0439e23
Build id: 3.10.0-1127.el7.x86_64 		0xc13
2
dev_hard_start_xmit
vmlinux 0xffffffffb0853386
Build id: 3.10.0-1127.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffffb088131a
Build id: 3.10.0-1127.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffb0856301
Build id: 3.10.0-1127.el7.x86_64 		0x491
5
? local_bh_enable
vmlinux 0xffffffffb02a4b17
Build id: 3.10.0-1127.el7.x86_64 		0x17
6
dev_queue_xmit
vmlinux 0xffffffffb08564d0
Build id: 3.10.0-1127.el7.x86_64 		0x10
7
neigh_resolve_output
vmlinux 0xffffffffb086312d
Build id: 3.10.0-1127.el7.x86_64 		0x11d
8
ip_finish_output
vmlinux 0xffffffffb08a3e9c
Build id: 3.10.0-1127.el7.x86_64 		0x27c
9
ip_output
vmlinux 0xffffffffb08a481b
Build id: 3.10.0-1127.el7.x86_64 		0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffffb08a3c20
Build id: 3.10.0-1127.el7.x86_64 		0xa60
11
ip_local_out_sk
vmlinux 0xffffffffb08a2247
Build id: 3.10.0-1127.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffffb08a5286
Build id: 3.10.0-1127.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffffb08cd64c
Build id: 3.10.0-1127.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffffb08a1f80
Build id: 3.10.0-1127.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffffb08cebb1
Build id: 3.10.0-1127.el7.x86_64 		0x5e1
16
? __wake_up_common
vmlinux 0xffffffffb02d3c52
Build id: 3.10.0-1127.el7.x86_64 		0x82
17
? ep_poll_callback
vmlinux 0xffffffffb049c4a8
Build id: 3.10.0-1127.el7.x86_64 		0xf8
18
? __wake_up_common
vmlinux 0xffffffffb02d3c52
Build id: 3.10.0-1127.el7.x86_64 		0x82
19
inet_sendmsg
vmlinux 0xffffffffb08dc379
Build id: 3.10.0-1127.el7.x86_64 		0x69
20
sock_sendmsg
vmlinux 0xffffffffb08333a6
Build id: 3.10.0-1127.el7.x86_64 		0xb6
21
? unix_dgram_sendmsg
vmlinux 0xffffffffb09141fb
Build id: 3.10.0-1127.el7.x86_64 		0x3cb
22
___sys_sendmsg
vmlinux 0xffffffffb0834269
Build id: 3.10.0-1127.el7.x86_64 		0x3e9
23
? __ip_route_output_key_hash
vmlinux 0xffffffffb089aacf
Build id: 3.10.0-1127.el7.x86_64 		0x20f
24
? ip4_datagram_release_cb
vmlinux 0xffffffffb08cb05f
Build id: 3.10.0-1127.el7.x86_64 		0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffb098854e
Build id: 3.10.0-1127.el7.x86_64 		0x1e
26
? udp_lib_rehash
vmlinux 0xffffffffb08ce12f
Build id: 3.10.0-1127.el7.x86_64 		0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffb098854e
Build id: 3.10.0-1127.el7.x86_64 		0x1e
28
? release_sock
vmlinux 0xffffffffb0838a18
Build id: 3.10.0-1127.el7.x86_64 		0x128
29
? ip4_datagram_connect
vmlinux 0xffffffffb08cb4c7
Build id: 3.10.0-1127.el7.x86_64 		0x37
30
? inet_dgram_connect
vmlinux 0xffffffffb08db004
Build id: 3.10.0-1127.el7.x86_64 		0x34
31
? SYSC_connect
vmlinux 0xffffffffb083396d
Build id: 3.10.0-1127.el7.x86_64 		0xed
32
__sys_sendmsg
vmlinux 0xffffffffb0835921
Build id: 3.10.0-1127.el7.x86_64 		0x51
33
SyS_sendmsg
vmlinux 0xffffffffb0835972
Build id: 3.10.0-1127.el7.x86_64 		0x12
34
system_call_fastpath
vmlinux 0xffffffffb0992ed2
Build id: 3.10.0-1127.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 38s! [chronyd:1024]
Modules linked in: tcp_lp nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc vfat fat snd_hda_codec_generic ppdev snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep iosf_mbi crc32_pclmul snd_seq snd_seq_device pcspkr vmw_balloon snd_pcm ghash_clmulni_intel joydev cryptd sg snd_timer snd vmw_vmci soundcore i2c_piix4 tpm_crb parport_pc parport sch_fq_codel ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt nfit crct10dif_pclmul
fb_sys_fops crct10dif_common ttm ahci libnvdimm crc32c_intel e1000e mptsas drm libahci ata_piix scsi_transport_sas serio_raw libata mptscsih mptbase ptp pps_core drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1024 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.12343141.B64.1902160724 02/16/2019
task: ffff9a4bf78c0000 ti: ffff9a4bf7200000 task.ti: ffff9a4bf7200000
RIP: 0010:[<ffffffffc0439e23>]  [<ffffffffc0439e23>] e1000_xmit_frame+0xc13/0xfa0 [e1000e]
RSP: 0018:ffff9a4bf7203718  EFLAGS: 00000246
RAX: 00000000000000ad RBX: ffff9a4bf72036b8 RCX: 0000000000000100
RDX: ffffc074c0743818 RSI: 0000000000000013 RDI: ffff9a4b8f40e100
RBP: ffff9a4bf7203790 R08: 0000000000000000 R09: ffff9a4b8d3208c0
R10: 0000000000000000 R11: 0000000027567202 R12: 0000000000000246
R13: ffff9a4bf9001700 R14: ffff9a4bf9001700 R15: 0000000000000020
FS:  00007f92a50df740(0000) GS:ffff9a4bffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7432f60000 CR3: 0000000077206000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb0853386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb088131a>] sch_direct_xmit+0x11a/0x250
[<ffffffffb0856301>] __dev_queue_xmit+0x491/0x650
[<ffffffffb02a4b17>] ? local_bh_enable+0x17/0x20
[<ffffffffb08564d0>] dev_queue_xmit+0x10/0x20
[<ffffffffb086312d>] neigh_resolve_output+0x11d/0x220
[<ffffffffb08a3e9c>] ip_finish_output+0x27c/0x8d0
[<ffffffffb08a481b>] ip_output+0x7b/0xf0
[<ffffffffb08a3c20>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffb08a2247>] ip_local_out_sk+0x37/0x40
[<ffffffffb08a5286>] ip_send_skb+0x16/0x50
[<ffffffffb08cd64c>] udp_send_skb+0xac/0x2b0
[<ffffffffb08a1f80>] ? ip_copy_metadata+0x170/0x170
[<ffffffffb08cebb1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffb02d3c52>] ? __wake_up_common+0x82/0x120
[<ffffffffb049c4a8>] ? ep_poll_callback+0xf8/0x220
[<ffffffffb02d3c52>] ? __wake_up_common+0x82/0x120
[<ffffffffb08dc379>] inet_sendmsg+0x69/0xb0
[<ffffffffb08333a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffb09141fb>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffb0834269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffb089aacf>] ? __ip_route_output_key_hash+0x20f/0x7d0
[<ffffffffb08cb05f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffb098854e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb08ce12f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffb098854e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb0838a18>] ? release_sock+0x128/0x180
[<ffffffffb08cb4c7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffb08db004>] ? inet_dgram_connect+0x34/0x90
[<ffffffffb083396d>] ? SYSC_connect+0xed/0x130
[<ffffffffb0835921>] __sys_sendmsg+0x51/0x90
[<ffffffffb0835972>] SyS_sendmsg+0x12/0x20
[<ffffffffb0992ed2>] system_call_fastpath+0x25/0x2a
Code: 8b 45 c8 48 8b 80 00 03 00 00 f6 80 98 00 00 00 03 74 18 48 8b 45 c8 f6 80 11 37 00 00 08 75 4b 49 8b 57 30 41 0f b7 47 20 89 02 <31> c0 e9 5d f8 ff ff 4c 8b 7d b8 48 8b 5d 88 e9 15 fd ff ff 0f
Complete report #986965
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc03e43dc
Build id: 3.10.0-1160.el7.x86_64 		0xaec
2
dev_hard_start_xmit
vmlinux 0xffffffff84854386
Build id: 3.10.0-1160.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffff8488242a
Build id: 3.10.0-1160.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff84857301
Build id: 3.10.0-1160.el7.x86_64 		0x491
5
? local_bh_enable
vmlinux 0xffffffff842a4017
Build id: 3.10.0-1160.el7.x86_64 		0x17
6
dev_queue_xmit
vmlinux 0xffffffff848574d0
Build id: 3.10.0-1160.el7.x86_64 		0x10
7
neigh_resolve_output
vmlinux 0xffffffff848641fd
Build id: 3.10.0-1160.el7.x86_64 		0x11d
8
ip_finish_output
vmlinux 0xffffffff848a505c
Build id: 3.10.0-1160.el7.x86_64 		0x27c
9
ip_output
vmlinux 0xffffffff848a59db
Build id: 3.10.0-1160.el7.x86_64 		0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffff848a4de0
Build id: 3.10.0-1160.el7.x86_64 		0xa60
11
ip_local_out_sk
vmlinux 0xffffffff848a3407
Build id: 3.10.0-1160.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffff848a6446
Build id: 3.10.0-1160.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffff848ce85c
Build id: 3.10.0-1160.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffff848a3140
Build id: 3.10.0-1160.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffff848cfdc1
Build id: 3.10.0-1160.el7.x86_64 		0x5e1
16
? __pollwait
vmlinux 0xffffffff84463e50
Build id: 3.10.0-1160.el7.x86_64 		0xf0
17
? sock_has_perm
vmlinux 0xffffffff8450be65
Build id: 3.10.0-1160.el7.x86_64 		0x75
18
inet_sendmsg
vmlinux 0xffffffff848dd589
Build id: 3.10.0-1160.el7.x86_64 		0x69
19
? selinux_socket_sendmsg
vmlinux 0xffffffff8450bf93
Build id: 3.10.0-1160.el7.x86_64 		0x23
20
sock_sendmsg
vmlinux 0xffffffff848343a6
Build id: 3.10.0-1160.el7.x86_64 		0xb6
21
___sys_sendmsg
vmlinux 0xffffffff84835269
Build id: 3.10.0-1160.el7.x86_64 		0x3e9
22
? __ip_route_output_key_hash
vmlinux 0xffffffff8489bc63
Build id: 3.10.0-1160.el7.x86_64 		0x223
23
? ip4_datagram_release_cb
vmlinux 0xffffffff848cc28f
Build id: 3.10.0-1160.el7.x86_64 		0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff8498992e
Build id: 3.10.0-1160.el7.x86_64 		0x1e
25
? udp_lib_rehash
vmlinux 0xffffffff848cf33f
Build id: 3.10.0-1160.el7.x86_64 		0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff8498992e
Build id: 3.10.0-1160.el7.x86_64 		0x1e
27
? release_sock
vmlinux 0xffffffff84839a18
Build id: 3.10.0-1160.el7.x86_64 		0x128
28
? ip4_datagram_connect
vmlinux 0xffffffff848cc6f7
Build id: 3.10.0-1160.el7.x86_64 		0x37
29
? inet_dgram_connect
vmlinux 0xffffffff848dc214
Build id: 3.10.0-1160.el7.x86_64 		0x34
30
? SYSC_connect
vmlinux 0xffffffff8483496d
Build id: 3.10.0-1160.el7.x86_64 		0xed
31
__sys_sendmsg
vmlinux 0xffffffff84836921
Build id: 3.10.0-1160.el7.x86_64 		0x51
32
SyS_sendmsg
vmlinux 0xffffffff84836972
Build id: 3.10.0-1160.el7.x86_64 		0x12
33
system_call_fastpath
vmlinux 0xffffffff84993f92
Build id: 3.10.0-1160.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 186s! [chronyd:794]
Modules linked in: tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc vfat fat nls_utf8 isofs loop intel_powerclamp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg tpm_crb parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper nfit libnvdimm syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul crct10dif_common
crc32c_intel libata drm serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 794 Comm: chronyd Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware20,1/440BX Desktop Reference Platform, BIOS VMW201.00V.20648489.B64.2210180829 10/18/2022
task: ffff8ae0ce2a1080 ti: ffff8ae0ce308000 task.ti: ffff8ae0ce308000
RIP: 0010:[<ffffffffc03e43dc>]  [<ffffffffc03e43dc>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0018:ffff8ae0ce30b700  EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffffffffc09869d7 RCX: 000000000000003a
RDX: ffffa8f00031c600 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff8ae0ce30b790 R08: 0000000000000039 R09: 0000000000a0405a
R10: ffff8ae0cdd836b0 R11: ffff8ae1375d8390 R12: ffff8ae0d59a4200
R13: 000106d03fd1acc0 R14: ffffffff8483e13d R15: 000000003fd1acc0
FS:  00007f6da7143740(0000) GS:ffff8ae13fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00001baeb72e95d0 CR3: 000000000e306000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffff84854386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff8488242a>] sch_direct_xmit+0x11a/0x250
[<ffffffff84857301>] __dev_queue_xmit+0x491/0x650
[<ffffffff842a4017>] ? local_bh_enable+0x17/0x20
[<ffffffff848574d0>] dev_queue_xmit+0x10/0x20
[<ffffffff848641fd>] neigh_resolve_output+0x11d/0x220
[<ffffffff848a505c>] ip_finish_output+0x27c/0x8d0
[<ffffffff848a59db>] ip_output+0x7b/0xf0
[<ffffffff848a4de0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff848a3407>] ip_local_out_sk+0x37/0x40
[<ffffffff848a6446>] ip_send_skb+0x16/0x50
[<ffffffff848ce85c>] udp_send_skb+0xac/0x2b0
[<ffffffff848a3140>] ? ip_copy_metadata+0x170/0x170
[<ffffffff848cfdc1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff84463e50>] ? __pollwait+0xf0/0xf0
[<ffffffff8450be65>] ? sock_has_perm+0x75/0x90
[<ffffffff848dd589>] inet_sendmsg+0x69/0xb0
[<ffffffff8450bf93>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff848343a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff84835269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff8489bc63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffff848cc28f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff8498992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff848cf33f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff8498992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff84839a18>] ? release_sock+0x128/0x180
[<ffffffff848cc6f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff848dc214>] ? inet_dgram_connect+0x34/0x90
[<ffffffff8483496d>] ? SYSC_connect+0xed/0x130
[<ffffffff84836921>] __sys_sendmsg+0x51/0x90
[<ffffffff84836972>] SyS_sendmsg+0x12/0x20
[<ffffffff84993f92>] system_call_fastpath+0x25/0x2a
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41
Complete report #942971
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc03a7877
Build id: 3.10.0-1127.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffa3853386
Build id: 3.10.0-1127.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffffa388131a
Build id: 3.10.0-1127.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffa3856301
Build id: 3.10.0-1127.el7.x86_64 		0x491
5
? local_bh_enable
vmlinux 0xffffffffa32a4b17
Build id: 3.10.0-1127.el7.x86_64 		0x17
6
dev_queue_xmit
vmlinux 0xffffffffa38564d0
Build id: 3.10.0-1127.el7.x86_64 		0x10
7
neigh_resolve_output
vmlinux 0xffffffffa386312d
Build id: 3.10.0-1127.el7.x86_64 		0x11d
8
ip_finish_output
vmlinux 0xffffffffa38a3e9c
Build id: 3.10.0-1127.el7.x86_64 		0x27c
9
ip_output
vmlinux 0xffffffffa38a481b
Build id: 3.10.0-1127.el7.x86_64 		0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffffa38a3c20
Build id: 3.10.0-1127.el7.x86_64 		0xa60
11
ip_local_out_sk
vmlinux 0xffffffffa38a2247
Build id: 3.10.0-1127.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffffa38a5286
Build id: 3.10.0-1127.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffffa38cd64c
Build id: 3.10.0-1127.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffffa38a1f80
Build id: 3.10.0-1127.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffffa38cebb1
Build id: 3.10.0-1127.el7.x86_64 		0x5e1
16
? __pollwait
vmlinux 0xffffffffa3463380
Build id: 3.10.0-1127.el7.x86_64 		0xf0
17
? __pollwait
vmlinux 0xffffffffa3463380
Build id: 3.10.0-1127.el7.x86_64 		0xf0
18
inet_sendmsg
vmlinux 0xffffffffa38dc379
Build id: 3.10.0-1127.el7.x86_64 		0x69
19
? __pollwait
vmlinux 0xffffffffa3463380
Build id: 3.10.0-1127.el7.x86_64 		0xf0
20
sock_sendmsg
vmlinux 0xffffffffa38333a6
Build id: 3.10.0-1127.el7.x86_64 		0xb6
21
___sys_sendmsg
vmlinux 0xffffffffa3834269
Build id: 3.10.0-1127.el7.x86_64 		0x3e9
22
? __ip_route_output_key_hash
vmlinux 0xffffffffa389aacf
Build id: 3.10.0-1127.el7.x86_64 		0x20f
23
? ip4_datagram_release_cb
vmlinux 0xffffffffa38cb05f
Build id: 3.10.0-1127.el7.x86_64 		0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffa398854e
Build id: 3.10.0-1127.el7.x86_64 		0x1e
25
? udp_lib_rehash
vmlinux 0xffffffffa38ce12f
Build id: 3.10.0-1127.el7.x86_64 		0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffffa398854e
Build id: 3.10.0-1127.el7.x86_64 		0x1e
27
? release_sock
vmlinux 0xffffffffa3838a18
Build id: 3.10.0-1127.el7.x86_64 		0x128
28
? ip4_datagram_connect
vmlinux 0xffffffffa38cb4c7
Build id: 3.10.0-1127.el7.x86_64 		0x37
29
? inet_dgram_connect
vmlinux 0xffffffffa38db004
Build id: 3.10.0-1127.el7.x86_64 		0x34
30
? SYSC_connect
vmlinux 0xffffffffa383396d
Build id: 3.10.0-1127.el7.x86_64 		0xed
31
__sys_sendmsg
vmlinux 0xffffffffa3835921
Build id: 3.10.0-1127.el7.x86_64 		0x51
32
SyS_sendmsg
vmlinux 0xffffffffa3835972
Build id: 3.10.0-1127.el7.x86_64 		0x12
33
system_call_fastpath
vmlinux 0xffffffffa3992ed2
Build id: 3.10.0-1127.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [chronyd:834]
Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm dns_resolver xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables tun xt_conntrack iptable_filter vmw_vsock_vmci_transport vsock xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev snd_timer vmw_balloon snd sg crc32_pclmul soundcore ghash_clmulni_intel aesni_intel pcspkr lrw gf128mul glue_helper ablk_helper cryptd joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx ata_piix drm_kms_helper syscopyarea
sysfillrect sysimgblt libata fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm nfit e1000 serio_raw libnvdimm mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 834 Comm: chronyd Kdump: loaded Tainted: G        W    L ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff911b354f0000 ti: ffff912027c18000 task.ti: ffff912027c18000
RIP: 0010:[<ffffffffc03a7877>]  [<ffffffffc03a7877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff912027c1b710  EFLAGS: 00000286
RAX: ffffae0a83503818 RBX: ffffffffa383be73 RCX: 000000000000007d
RDX: 000000000000007d RSI: ffff912024194bc0 RDI: ffff911c77b54000
RBP: ffff912027c1b790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 000000017fc07700 R15: ffffffffa35b0770
FS:  00007f7a83e7f740(0000) GS:ffff912029640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc08710c2c9 CR3: 00000005227a0000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa3853386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa388131a>] sch_direct_xmit+0x11a/0x250
[<ffffffffa3856301>] __dev_queue_xmit+0x491/0x650
[<ffffffffa32a4b17>] ? local_bh_enable+0x17/0x20
[<ffffffffa38564d0>] dev_queue_xmit+0x10/0x20
[<ffffffffa386312d>] neigh_resolve_output+0x11d/0x220
[<ffffffffa38a3e9c>] ip_finish_output+0x27c/0x8d0
[<ffffffffa38a481b>] ip_output+0x7b/0xf0
[<ffffffffa38a3c20>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffa38a2247>] ip_local_out_sk+0x37/0x40
[<ffffffffa38a5286>] ip_send_skb+0x16/0x50
[<ffffffffa38cd64c>] udp_send_skb+0xac/0x2b0
[<ffffffffa38a1f80>] ? ip_copy_metadata+0x170/0x170
[<ffffffffa38cebb1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffa3463380>] ? __pollwait+0xf0/0xf0
[<ffffffffa3463380>] ? __pollwait+0xf0/0xf0
[<ffffffffa38dc379>] inet_sendmsg+0x69/0xb0
[<ffffffffa3463380>] ? __pollwait+0xf0/0xf0
[<ffffffffa38333a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffa3834269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffa389aacf>] ? __ip_route_output_key_hash+0x20f/0x7d0
[<ffffffffa38cb05f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffa398854e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa38ce12f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffa398854e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa3838a18>] ? release_sock+0x128/0x180
[<ffffffffa38cb4c7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffa38db004>] ? inet_dgram_connect+0x34/0x90
[<ffffffffa383396d>] ? SYSC_connect+0xed/0x130
[<ffffffffa3835921>] __sys_sendmsg+0x51/0x90
[<ffffffffa3835972>] SyS_sendmsg+0x12/0x20
[<ffffffffa3992ed2>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #846920
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc033e857
Build id: 3.10.0-957.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffab438c66
Build id: 3.10.0-957.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffffab465cea
Build id: 3.10.0-957.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffab43bbe1
Build id: 3.10.0-957.el7.x86_64 		0x491
5
dev_queue_xmit
vmlinux 0xffffffffab43bdb0
Build id: 3.10.0-957.el7.x86_64 		0x10
6
neigh_resolve_output
vmlinux 0xffffffffab44861d
Build id: 3.10.0-957.el7.x86_64 		0x11d
7
ip_finish_output
vmlinux 0xffffffffab48845c
Build id: 3.10.0-957.el7.x86_64 		0x27c
8
ip_output
vmlinux 0xffffffffab488c93
Build id: 3.10.0-957.el7.x86_64 		0x73
9
? __ip_append_data.isra.51
vmlinux 0xffffffffab4881e0
Build id: 3.10.0-957.el7.x86_64 		0xa50
10
ip_local_out_sk
vmlinux 0xffffffffab486817
Build id: 3.10.0-957.el7.x86_64 		0x37
11
ip_send_skb
vmlinux 0xffffffffab4896f6
Build id: 3.10.0-957.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffffab4b150c
Build id: 3.10.0-957.el7.x86_64 		0xac
13
? ip_copy_metadata
vmlinux 0xffffffffab486560
Build id: 3.10.0-957.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffffab4b27f5
Build id: 3.10.0-957.el7.x86_64 		0x335
15
? default_wake_function
vmlinux 0xffffffffaaed67c2
Build id: 3.10.0-957.el7.x86_64 		0x12
16
? __wake_up_common
vmlinux 0xffffffffaaecba9b
Build id: 3.10.0-957.el7.x86_64 		0x5b
17
? ep_poll_callback
vmlinux 0xffffffffab08ecde
Build id: 3.10.0-957.el7.x86_64 		0xee
18
inet_sendmsg
vmlinux 0xffffffffab4bfeb9
Build id: 3.10.0-957.el7.x86_64 		0x69
19
sock_sendmsg
vmlinux 0xffffffffab4193a6
Build id: 3.10.0-957.el7.x86_64 		0xb6
20
? unix_dgram_sendmsg
vmlinux 0xffffffffab4f769b
Build id: 3.10.0-957.el7.x86_64 		0x3cb
21
___sys_sendmsg
vmlinux 0xffffffffab41a269
Build id: 3.10.0-957.el7.x86_64 		0x3e9
22
? ip4_datagram_release_cb
vmlinux 0xffffffffab4aef8f
Build id: 3.10.0-957.el7.x86_64 		0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffffab56a29e
Build id: 3.10.0-957.el7.x86_64 		0x1e
24
? udp_lib_rehash
vmlinux 0xffffffffab4b1e0f
Build id: 3.10.0-957.el7.x86_64 		0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffab56a29e
Build id: 3.10.0-957.el7.x86_64 		0x1e
26
? release_sock
vmlinux 0xffffffffab41e980
Build id: 3.10.0-957.el7.x86_64 		0x120
27
? ip4_datagram_connect
vmlinux 0xffffffffab4af3f7
Build id: 3.10.0-957.el7.x86_64 		0x37
28
? inet_dgram_connect
vmlinux 0xffffffffab4beb44
Build id: 3.10.0-957.el7.x86_64 		0x34
29
? SYSC_connect
vmlinux 0xffffffffab41996d
Build id: 3.10.0-957.el7.x86_64 		0xed
30
__sys_sendmsg
vmlinux 0xffffffffab41b921
Build id: 3.10.0-957.el7.x86_64 		0x51
31
SyS_sendmsg
vmlinux 0xffffffffab41b972
Build id: 3.10.0-957.el7.x86_64 		0x12
32
system_call_fastpath
vmlinux 0xffffffffab574ddb
Build id: 3.10.0-957.el7.x86_64 		0x22 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [chronyd:6225]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables tcp_lp nfnetlink_queue nfnetlink_log bluetooth rfkill fuse tun devlink bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr snd_ens1371 snd_rawmidi
snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore sg parport_pc parport i2c_piix4 vmw_vmci xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw nfit libnvdimm e1000 ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi drm_panel_orientation_quirks libata scsi_transport_spi mptscsih mptbase dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 1 PID: 6225 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9c6bb9fca080 ti: ffff9c6b75928000 task.ti: ffff9c6b75928000
RIP: 0010:[<ffffffffc033e857>]  [<ffffffffc033e857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9c6b7592b720  EFLAGS: 00000286
RAX: ffffc01dc0403818 RBX: ffffffffab422bf3 RCX: 00000000000000c4
RDX: 00000000000000c4 RSI: ffff9c6b74c362c0 RDI: ffff9c6bb989e000
RBP: ffff9c6b7592b7a0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 0000000000000000 R15: 0000000000000000
FS:  00007f4043541740(0000) GS:ffff9c6bbb640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001b15030 CR3: 000000003592c000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffab438c66>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffab465cea>] sch_direct_xmit+0x11a/0x250
[<ffffffffab43bbe1>] __dev_queue_xmit+0x491/0x650
[<ffffffffab43bdb0>] dev_queue_xmit+0x10/0x20
[<ffffffffab44861d>] neigh_resolve_output+0x11d/0x220
[<ffffffffab48845c>] ip_finish_output+0x27c/0x7b0
[<ffffffffab488c93>] ip_output+0x73/0xe0
[<ffffffffab4881e0>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffab486817>] ip_local_out_sk+0x37/0x40
[<ffffffffab4896f6>] ip_send_skb+0x16/0x50
[<ffffffffab4b150c>] udp_send_skb+0xac/0x2b0
[<ffffffffab486560>] ? ip_copy_metadata+0x170/0x170
[<ffffffffab4b27f5>] udp_sendmsg+0x335/0xa10
[<ffffffffaaed67c2>] ? default_wake_function+0x12/0x20
[<ffffffffaaecba9b>] ? __wake_up_common+0x5b/0x90
[<ffffffffab08ecde>] ? ep_poll_callback+0xee/0x210
[<ffffffffab4bfeb9>] inet_sendmsg+0x69/0xb0
[<ffffffffab4193a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffab4f769b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffab41a269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffab4aef8f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffab56a29e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffab4b1e0f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffab56a29e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffab41e980>] ? release_sock+0x120/0x170
[<ffffffffab4af3f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffab4beb44>] ? inet_dgram_connect+0x34/0x90
[<ffffffffab41996d>] ? SYSC_connect+0xed/0x130
[<ffffffffab41b921>] __sys_sendmsg+0x51/0x90
[<ffffffffab41b972>] SyS_sendmsg+0x12/0x20
[<ffffffffab574ddb>] system_call_fastpath+0x22/0x27
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #813175
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc023144c
Build id: 3.10.0-1160.92.1.el7.x86_64 		0xaec
2
? nf_nat_setup_info
nf_nat 0xffffffffc066d457
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x2f7
3
dev_hard_start_xmit
vmlinux 0xffffffff96c7edc6
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffff96cadd72
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x122
5
__dev_queue_xmit
vmlinux 0xffffffff96c81dd1
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x491
6
dev_queue_xmit
vmlinux 0xffffffff96c81fa0
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x10
7
ip_finish_output
vmlinux 0xffffffff96cd260b
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x67b
8
ip_output
vmlinux 0xffffffff96cd2b9b
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffff96cd1f90
Build id: 3.10.0-1160.92.1.el7.x86_64 		0xa60
10
ip_local_out_sk
vmlinux 0xffffffff96cd0587
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x37
11
ip_send_skb
vmlinux 0xffffffff96cd3636
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffff96cfc3ac
Build id: 3.10.0-1160.92.1.el7.x86_64 		0xac
13
? ip_copy_metadata
vmlinux 0xffffffff96cd02b0
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffff96cfd9a1
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x5e1
15
? __pollwait
vmlinux 0xffffffff96872490
Build id: 3.10.0-1160.92.1.el7.x86_64 		0xf0
16
? __pollwait
vmlinux 0xffffffff96872490
Build id: 3.10.0-1160.92.1.el7.x86_64 		0xf0
17
inet_sendmsg
vmlinux 0xffffffff96d0b4e9
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x69
18
? __pollwait
vmlinux 0xffffffff96872490
Build id: 3.10.0-1160.92.1.el7.x86_64 		0xf0
19
sock_sendmsg
vmlinux 0xffffffff96c5e4f6
Build id: 3.10.0-1160.92.1.el7.x86_64 		0xb6
20
___sys_sendmsg
vmlinux 0xffffffff96c5f3f9
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x3e9
21
? __ip_route_output_key_hash
vmlinux 0xffffffff96cc8c63
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x223
22
? ip4_datagram_release_cb
vmlinux 0xffffffff96cf9cdf
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffff96dba56e
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x1e
24
? udp_lib_rehash
vmlinux 0xffffffff96cfcecf
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffff96dba56e
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x1e
26
? release_sock
vmlinux 0xffffffff96c63ea8
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x128
27
? ip4_datagram_connect
vmlinux 0xffffffff96cfa147
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x37
28
? inet_dgram_connect
vmlinux 0xffffffff96d0a0e4
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x34
29
__sys_sendmsg
vmlinux 0xffffffff96c60b41
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x51
30
SyS_sendmsg
vmlinux 0xffffffff96c60b92
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x12
31
system_call_fastpath
vmlinux 0xffffffff96dc539a
Build id: 3.10.0-1160.92.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [chronyd:713]
Modules linked in: binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon sg joydev pcspkr parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea crc_t10dif sysfillrect sysimgblt
fb_sys_fops crct10dif_generic ttm crct10dif_pclmul drm crct10dif_common crc32c_intel ata_piix nfit libata serio_raw libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 713 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.92.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8f2734c3b180 ti: ffff8f27362b8000 task.ti: ffff8f27362b8000
RIP: 0010:[<ffffffffc023144c>]  [<ffffffffc023144c>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0018:ffff8f27362bb740  EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffff8f27362bb7b8 RCX: 00000000000000d6
RDX: ffffb169c035c600 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff8f27362bb7d0 R08: 0000000000000000 R09: 0000000000a0005a
R10: 0000000000000000 R11: ffff8f27366f0d50 R12: ffffffffc063f99d
R13: ffff8f27362bb6e8 R14: ffffffff966a89b7 R15: ffff8f27362bb6a8
FS:  00007fb97843f740(0000) GS:ffff8f273b600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006e9360 CR3: 00000000762c0000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc066d457>] ? nf_nat_setup_info+0x2f7/0x360 [nf_nat]
[<ffffffff96c7edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff96cadd72>] sch_direct_xmit+0x122/0x250
[<ffffffff96c81dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffff96c81fa0>] dev_queue_xmit+0x10/0x20
[<ffffffff96cd260b>] ip_finish_output+0x67b/0x8d0
[<ffffffff96cd2b9b>] ip_output+0x7b/0x100
[<ffffffff96cd1f90>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff96cd0587>] ip_local_out_sk+0x37/0x40
[<ffffffff96cd3636>] ip_send_skb+0x16/0x50
[<ffffffff96cfc3ac>] udp_send_skb+0xac/0x2b0
[<ffffffff96cd02b0>] ? ip_copy_metadata+0x170/0x170
[<ffffffff96cfd9a1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff96872490>] ? __pollwait+0xf0/0xf0
[<ffffffff96872490>] ? __pollwait+0xf0/0xf0
[<ffffffff96d0b4e9>] inet_sendmsg+0x69/0xb0
[<ffffffff96872490>] ? __pollwait+0xf0/0xf0
[<ffffffff96c5e4f6>] sock_sendmsg+0xb6/0x100
[<ffffffff96c5f3f9>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff96cc8c63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffff96cf9cdf>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff96dba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff96cfcecf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff96dba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff96c63ea8>] ? release_sock+0x128/0x180
[<ffffffff96cfa147>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff96d0a0e4>] ? inet_dgram_connect+0x34/0x90
[<ffffffff96c60b41>] __sys_sendmsg+0x51/0x90
[<ffffffff96c60b92>] SyS_sendmsg+0x12/0x30
[<ffffffff96dc539a>] system_call_fastpath+0x25/0x2a
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41
Complete report #706286
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc018d877
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffafa4f346
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffffafa7d0ca
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffafa522c1
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x491
5
dev_queue_xmit
vmlinux 0xffffffffafa52490
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x10
6
ip_finish_output
vmlinux 0xffffffffafaa099b
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x67b
7
ip_output
vmlinux 0xffffffffafaa0f1b
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffffafaa0320
Build id: 3.10.0-1062.18.1.el7.x86_64 		0xa60
9
ip_local_out_sk
vmlinux 0xffffffffafa9e947
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x37
10
ip_send_skb
vmlinux 0xffffffffafaa1986
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x16
11
udp_send_skb
vmlinux 0xffffffffafac9d0c
Build id: 3.10.0-1062.18.1.el7.x86_64 		0xac
12
? ip_copy_metadata
vmlinux 0xffffffffafa9e680
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x170
13
udp_sendmsg
vmlinux 0xffffffffafacb271
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x5e1
14
? ep_poll_callback
vmlinux 0xffffffffaf69934e
Build id: 3.10.0-1062.18.1.el7.x86_64 		0xee
15
? _raw_spin_unlock_irqrestore
vmlinux 0xffffffffafb83815
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x15
16
inet_sendmsg
vmlinux 0xffffffffafad89c9
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x69
17
sock_sendmsg
vmlinux 0xffffffffafa2f3a6
Build id: 3.10.0-1062.18.1.el7.x86_64 		0xb6
18
? unix_dgram_sendmsg
vmlinux 0xffffffffafb1077b
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x3cb
19
___sys_sendmsg
vmlinux 0xffffffffafa30269
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x3e9
20
? __ip_route_output_key_hash
vmlinux 0xffffffffafa97277
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x207
21
? ip4_datagram_release_cb
vmlinux 0xffffffffafac771f
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x6f
22
? _raw_spin_unlock_bh
vmlinux 0xffffffffafb839be
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x1e
23
? udp_lib_rehash
vmlinux 0xffffffffafaca7ef
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x11f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffafb839be
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x1e
25
? release_sock
vmlinux 0xffffffffafa34a10
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x120
26
? ip4_datagram_connect
vmlinux 0xffffffffafac7b87
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x37
27
? inet_dgram_connect
vmlinux 0xffffffffafad7654
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x34
28
? SYSC_connect
vmlinux 0xffffffffafa2f96d
Build id: 3.10.0-1062.18.1.el7.x86_64 		0xed
29
__sys_sendmsg
vmlinux 0xffffffffafa31921
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x51
30
SyS_sendmsg
vmlinux 0xffffffffafa31972
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x12
31
system_call_fastpath
vmlinux 0xffffffffafb8dede
Build id: 3.10.0-1062.18.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 29s! [chronyd:893]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter binfmt_misc vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata nfit mptspi libnvdimm crct10dif_pclmul crct10dif_common
scsi_transport_spi mptscsih crc32c_intel e1000 mptbase serio_raw drm_panel_orientation_quirks
CPU: 1 PID: 893 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff8e8375cd9070 ti: ffff8e83fb340000 task.ti: ffff8e83fb340000
RIP: 0010:[<ffffffffc018d877>]  [<ffffffffc018d877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8e83fb343750  EFLAGS: 00000286
RAX: ffffb2f800d03818 RBX: ffffffffafa37e63 RCX: 00000000000000e5
RDX: 00000000000000e5 RSI: ffff8e8570db4400 RDI: ffff8e84bce64000
RBP: ffff8e83fb3437d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 00000001c06898dd R15: ffffffffaf7acf90
FS:  00007f41d8506740(0000) GS:ffff8e8575640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8f10b7c000 CR3: 00000000bb326000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffafa4f346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffafa7d0ca>] sch_direct_xmit+0x11a/0x250
[<ffffffffafa522c1>] __dev_queue_xmit+0x491/0x650
[<ffffffffafa52490>] dev_queue_xmit+0x10/0x20
[<ffffffffafaa099b>] ip_finish_output+0x67b/0x8d0
[<ffffffffafaa0f1b>] ip_output+0x7b/0xf0
[<ffffffffafaa0320>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffafa9e947>] ip_local_out_sk+0x37/0x40
[<ffffffffafaa1986>] ip_send_skb+0x16/0x50
[<ffffffffafac9d0c>] udp_send_skb+0xac/0x2b0
[<ffffffffafa9e680>] ? ip_copy_metadata+0x170/0x170
[<ffffffffafacb271>] udp_sendmsg+0x5e1/0xa70
[<ffffffffaf69934e>] ? ep_poll_callback+0xee/0x210
[<ffffffffafb83815>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[<ffffffffafad89c9>] inet_sendmsg+0x69/0xb0
[<ffffffffafa2f3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffafb1077b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffafa30269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffafa97277>] ? __ip_route_output_key_hash+0x207/0x7b0
[<ffffffffafac771f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffafb839be>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffafaca7ef>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffafb839be>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffafa34a10>] ? release_sock+0x120/0x170
[<ffffffffafac7b87>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffafad7654>] ? inet_dgram_connect+0x34/0x90
[<ffffffffafa2f96d>] ? SYSC_connect+0xed/0x130
[<ffffffffafa31921>] __sys_sendmsg+0x51/0x90
[<ffffffffafa31972>] SyS_sendmsg+0x12/0x20
[<ffffffffafb8dede>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #580144
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0447857
Build id: 3.10.0-957.el7.x86_64 		0x7f7
2
? kfree_skbmem
vmlinux 0xffffffff89621697
Build id: 3.10.0-957.el7.x86_64 		0x37
3
dev_hard_start_xmit
vmlinux 0xffffffff89638c66
Build id: 3.10.0-957.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffff89665cea
Build id: 3.10.0-957.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffff8963bbe1
Build id: 3.10.0-957.el7.x86_64 		0x491
6
? local_bh_enable
vmlinux 0xffffffff890a0387
Build id: 3.10.0-957.el7.x86_64 		0x17
7
dev_queue_xmit
vmlinux 0xffffffff8963bdb0
Build id: 3.10.0-957.el7.x86_64 		0x10
8
neigh_resolve_output
vmlinux 0xffffffff8964861d
Build id: 3.10.0-957.el7.x86_64 		0x11d
9
ip_finish_output
vmlinux 0xffffffff8968845c
Build id: 3.10.0-957.el7.x86_64 		0x27c
10
ip_output
vmlinux 0xffffffff89688c93
Build id: 3.10.0-957.el7.x86_64 		0x73
11
? __ip_append_data.isra.51
vmlinux 0xffffffff896881e0
Build id: 3.10.0-957.el7.x86_64 		0xa50
12
ip_local_out_sk
vmlinux 0xffffffff89686817
Build id: 3.10.0-957.el7.x86_64 		0x37
13
ip_send_skb
vmlinux 0xffffffff896896f6
Build id: 3.10.0-957.el7.x86_64 		0x16
14
udp_send_skb
vmlinux 0xffffffff896b150c
Build id: 3.10.0-957.el7.x86_64 		0xac
15
? ip_copy_metadata
vmlinux 0xffffffff89686560
Build id: 3.10.0-957.el7.x86_64 		0x170
16
udp_sendmsg
vmlinux 0xffffffff896b27f5
Build id: 3.10.0-957.el7.x86_64 		0x335
17
? default_wake_function
vmlinux 0xffffffff890d67c2
Build id: 3.10.0-957.el7.x86_64 		0x12
18
? __wake_up_common
vmlinux 0xffffffff890cba9b
Build id: 3.10.0-957.el7.x86_64 		0x5b
19
? sock_has_perm
vmlinux 0xffffffff892fc3a5
Build id: 3.10.0-957.el7.x86_64 		0x75
20
inet_sendmsg
vmlinux 0xffffffff896bfeb9
Build id: 3.10.0-957.el7.x86_64 		0x69
21
? selinux_socket_sendmsg
vmlinux 0xffffffff892fc4d3
Build id: 3.10.0-957.el7.x86_64 		0x23
22
sock_sendmsg
vmlinux 0xffffffff896193a6
Build id: 3.10.0-957.el7.x86_64 		0xb6
23
? unix_dgram_sendmsg
vmlinux 0xffffffff896f769b
Build id: 3.10.0-957.el7.x86_64 		0x3cb
24
___sys_sendmsg
vmlinux 0xffffffff8961a269
Build id: 3.10.0-957.el7.x86_64 		0x3e9
25
? ip4_datagram_release_cb
vmlinux 0xffffffff896aef8f
Build id: 3.10.0-957.el7.x86_64 		0x6f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff8976a29e
Build id: 3.10.0-957.el7.x86_64 		0x1e
27
? udp_lib_rehash
vmlinux 0xffffffff896b1e0f
Build id: 3.10.0-957.el7.x86_64 		0x11f
28
? _raw_spin_unlock_bh
vmlinux 0xffffffff8976a29e
Build id: 3.10.0-957.el7.x86_64 		0x1e
29
? release_sock
vmlinux 0xffffffff8961e980
Build id: 3.10.0-957.el7.x86_64 		0x120
30
? ip4_datagram_connect
vmlinux 0xffffffff896af3f7
Build id: 3.10.0-957.el7.x86_64 		0x37
31
? inet_dgram_connect
vmlinux 0xffffffff896beb44
Build id: 3.10.0-957.el7.x86_64 		0x34
32
? SYSC_connect
vmlinux 0xffffffff8961996d
Build id: 3.10.0-957.el7.x86_64 		0xed
33
__sys_sendmsg
vmlinux 0xffffffff8961b921
Build id: 3.10.0-957.el7.x86_64 		0x51
34
SyS_sendmsg
vmlinux 0xffffffff8961b972
Build id: 3.10.0-957.el7.x86_64 		0x12
35
system_call_fastpath
vmlinux 0xffffffff89774ddb
Build id: 3.10.0-957.el7.x86_64 		0x22 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [chronyd:8619]
Modules linked in: tcp_lp rfcomm fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd snd_ens1371 snd_rawmidi snd_ac97_codec
ac97_bus snd_seq btusb snd_seq_device btrtl btbcm btintel snd_pcm bluetooth pcspkr joydev snd_timer snd soundcore rfkill sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi drm e1000 mptscsih ata_piix mptbase libata drm_panel_orientation_quirks nfit libnvdimm
CPU: 2 PID: 8619 Comm: chronyd Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9e6df5ea30c0 ti: ffff9e6e76db8000 task.ti: ffff9e6e76db8000
RIP: 0010:[<ffffffffc0447857>]  [<ffffffffc0447857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9e6e76dbb720  EFLAGS: 00000286
RAX: ffffb21d009c3818 RBX: ffffffff89622bf3 RCX: 000000000000004e
RDX: 000000000000004e RSI: ffff9e6df5c82fc0 RDI: ffff9e6ef557a000
RBP: ffff9e6e76dbb7a0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 0000000000000000 R15: 0000000000000000
FS:  00007f4300f90740(0000) GS:ffff9e6ef9680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f10c3716000 CR3: 00000000b6dd6000 CR4: 00000000003607e0
Call Trace:
[<ffffffff89621697>] ? kfree_skbmem+0x37/0x90
[<ffffffff89638c66>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff89665cea>] sch_direct_xmit+0x11a/0x250
[<ffffffff8963bbe1>] __dev_queue_xmit+0x491/0x650
[<ffffffff890a0387>] ? local_bh_enable+0x17/0x20
[<ffffffff8963bdb0>] dev_queue_xmit+0x10/0x20
[<ffffffff8964861d>] neigh_resolve_output+0x11d/0x220
[<ffffffff8968845c>] ip_finish_output+0x27c/0x7b0
[<ffffffff89688c93>] ip_output+0x73/0xe0
[<ffffffff896881e0>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffff89686817>] ip_local_out_sk+0x37/0x40
[<ffffffff896896f6>] ip_send_skb+0x16/0x50
[<ffffffff896b150c>] udp_send_skb+0xac/0x2b0
[<ffffffff89686560>] ? ip_copy_metadata+0x170/0x170
[<ffffffff896b27f5>] udp_sendmsg+0x335/0xa10
[<ffffffff890d67c2>] ? default_wake_function+0x12/0x20
[<ffffffff890cba9b>] ? __wake_up_common+0x5b/0x90
[<ffffffff892fc3a5>] ? sock_has_perm+0x75/0x90
[<ffffffff896bfeb9>] inet_sendmsg+0x69/0xb0
[<ffffffff892fc4d3>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff896193a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff896f769b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffff8961a269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff896aef8f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff8976a29e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff896b1e0f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff8976a29e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff8961e980>] ? release_sock+0x120/0x170
[<ffffffff896af3f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff896beb44>] ? inet_dgram_connect+0x34/0x90
[<ffffffff8961996d>] ? SYSC_connect+0xed/0x130
[<ffffffff8961b921>] __sys_sendmsg+0x51/0x90
[<ffffffff8961b972>] SyS_sendmsg+0x12/0x20
[<ffffffff89774ddb>] system_call_fastpath+0x22/0x27
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #543230
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0227877
Build id: 3.10.0-1160.el7.x86_64 		0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffbda3c6f7
Build id: 3.10.0-1160.el7.x86_64 		0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffbda54386
Build id: 3.10.0-1160.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffffbda8242a
Build id: 3.10.0-1160.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffbda57301
Build id: 3.10.0-1160.el7.x86_64 		0x491
6
dev_queue_xmit
vmlinux 0xffffffffbda574d0
Build id: 3.10.0-1160.el7.x86_64 		0x10
7
neigh_resolve_output
vmlinux 0xffffffffbda641fd
Build id: 3.10.0-1160.el7.x86_64 		0x11d
8
ip_finish_output
vmlinux 0xffffffffbdaa505c
Build id: 3.10.0-1160.el7.x86_64 		0x27c
9
ip_output
vmlinux 0xffffffffbdaa59db
Build id: 3.10.0-1160.el7.x86_64 		0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffffbdaa4de0
Build id: 3.10.0-1160.el7.x86_64 		0xa60
11
ip_local_out_sk
vmlinux 0xffffffffbdaa3407
Build id: 3.10.0-1160.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffffbdaa6446
Build id: 3.10.0-1160.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffffbdace85c
Build id: 3.10.0-1160.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffffbdaa3140
Build id: 3.10.0-1160.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffffbdacfdc1
Build id: 3.10.0-1160.el7.x86_64 		0x5e1
16
? __pollwait
vmlinux 0xffffffffbd663e50
Build id: 3.10.0-1160.el7.x86_64 		0xf0
17
? sock_has_perm
vmlinux 0xffffffffbd70be65
Build id: 3.10.0-1160.el7.x86_64 		0x75
18
inet_sendmsg
vmlinux 0xffffffffbdadd589
Build id: 3.10.0-1160.el7.x86_64 		0x69
19
? selinux_socket_sendmsg
vmlinux 0xffffffffbd70bf93
Build id: 3.10.0-1160.el7.x86_64 		0x23
20
sock_sendmsg
vmlinux 0xffffffffbda343a6
Build id: 3.10.0-1160.el7.x86_64 		0xb6
21
? unix_dgram_sendmsg
vmlinux 0xffffffffbdb154bb
Build id: 3.10.0-1160.el7.x86_64 		0x3cb
22
___sys_sendmsg
vmlinux 0xffffffffbda35269
Build id: 3.10.0-1160.el7.x86_64 		0x3e9
23
? __ip_route_output_key_hash
vmlinux 0xffffffffbda9bc63
Build id: 3.10.0-1160.el7.x86_64 		0x223
24
? ip4_datagram_release_cb
vmlinux 0xffffffffbdacc28f
Build id: 3.10.0-1160.el7.x86_64 		0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffbdb8992e
Build id: 3.10.0-1160.el7.x86_64 		0x1e
26
? udp_lib_rehash
vmlinux 0xffffffffbdacf33f
Build id: 3.10.0-1160.el7.x86_64 		0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffbdb8992e
Build id: 3.10.0-1160.el7.x86_64 		0x1e
28
? release_sock
vmlinux 0xffffffffbda39a18
Build id: 3.10.0-1160.el7.x86_64 		0x128
29
? ip4_datagram_connect
vmlinux 0xffffffffbdacc6f7
Build id: 3.10.0-1160.el7.x86_64 		0x37
30
? inet_dgram_connect
vmlinux 0xffffffffbdadc214
Build id: 3.10.0-1160.el7.x86_64 		0x34
31
? SYSC_connect
vmlinux 0xffffffffbda3496d
Build id: 3.10.0-1160.el7.x86_64 		0xed
32
__sys_sendmsg
vmlinux 0xffffffffbda36921
Build id: 3.10.0-1160.el7.x86_64 		0x51
33
SyS_sendmsg
vmlinux 0xffffffffbda36972
Build id: 3.10.0-1160.el7.x86_64 		0x12
34
system_call_fastpath
vmlinux 0xffffffffbdb93f92
Build id: 3.10.0-1160.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 71s! [chronyd:763]
Modules linked in: tun devlink bridge stp llc ip_set nfnetlink bnep vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel btusb lrw gf128mul ppdev btrtl glue_helper btbcm ablk_helper btintel cryptd bluetooth vmw_balloon rfkill sg pcspkr joydev parport_pc parport snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore i2c_piix4 vmw_vmci xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crc32c_intel nfit libnvdimm drm ata_piix serio_raw e1000 mptspi scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash
dm_log dm_mod fuse [last unloaded: ip_tables]
CPU: 3 PID: 763 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffffa0bcb518b180 ti: ffffa0bcf4bcc000 task.ti: ffffa0bcf4bcc000
RIP: 0010:[<ffffffffc0227877>]  [<ffffffffc0227877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffffa0bcf4bcf710  EFLAGS: 00000286
RAX: ffffbe4180383818 RBX: ffffffffbda3ce73 RCX: 0000000000000038
RDX: 0000000000000038 RSI: ffffa0bcb4d6b140 RDI: ffffa0bcf59da000
RBP: ffffa0bcf4bcf790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 0000000000000800 R15: 0000000000000400
FS:  00007effe2acc740(0000) GS:ffffa0bcf8cc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f64ab0ba000 CR3: 000000007292e000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffbda3c6f7>] ? kfree_skbmem+0x37/0x90
[<ffffffffbda54386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffbda8242a>] sch_direct_xmit+0x11a/0x250
[<ffffffffbda57301>] __dev_queue_xmit+0x491/0x650
[<ffffffffbda574d0>] dev_queue_xmit+0x10/0x20
[<ffffffffbda641fd>] neigh_resolve_output+0x11d/0x220
[<ffffffffbdaa505c>] ip_finish_output+0x27c/0x8d0
[<ffffffffbdaa59db>] ip_output+0x7b/0xf0
[<ffffffffbdaa4de0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffbdaa3407>] ip_local_out_sk+0x37/0x40
[<ffffffffbdaa6446>] ip_send_skb+0x16/0x50
[<ffffffffbdace85c>] udp_send_skb+0xac/0x2b0
[<ffffffffbdaa3140>] ? ip_copy_metadata+0x170/0x170
[<ffffffffbdacfdc1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffbd663e50>] ? __pollwait+0xf0/0xf0
[<ffffffffbd70be65>] ? sock_has_perm+0x75/0x90
[<ffffffffbdadd589>] inet_sendmsg+0x69/0xb0
[<ffffffffbd70bf93>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffffbda343a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffbdb154bb>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffbda35269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffbda9bc63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffffbdacc28f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffbdb8992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffbdacf33f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffbdb8992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffbda39a18>] ? release_sock+0x128/0x180
[<ffffffffbdacc6f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffbdadc214>] ? inet_dgram_connect+0x34/0x90
[<ffffffffbda3496d>] ? SYSC_connect+0xed/0x130
[<ffffffffbda36921>] __sys_sendmsg+0x51/0x90
[<ffffffffbda36972>] SyS_sendmsg+0x12/0x20
[<ffffffffbdb93f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #529022
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc02ab857
Build id: 3.10.0-957.5.1.el7.x86_64 		0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffbec21717
Build id: 3.10.0-957.5.1.el7.x86_64 		0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffbec38ce6
Build id: 3.10.0-957.5.1.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffffbec65d6a
Build id: 3.10.0-957.5.1.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffbec3bc61
Build id: 3.10.0-957.5.1.el7.x86_64 		0x491
6
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc080f2ce
Build id: 3.10.0-957.5.1.el7.x86_64 		0x4e
7
dev_queue_xmit
vmlinux 0xffffffffbec3be30
Build id: 3.10.0-957.5.1.el7.x86_64 		0x10
8
ip_finish_output
vmlinux 0xffffffffbec887b6
Build id: 3.10.0-957.5.1.el7.x86_64 		0x556
9
ip_output
vmlinux 0xffffffffbec88d13
Build id: 3.10.0-957.5.1.el7.x86_64 		0x73
10
? __ip_append_data.isra.51
vmlinux 0xffffffffbec88260
Build id: 3.10.0-957.5.1.el7.x86_64 		0xa50
11
ip_local_out_sk
vmlinux 0xffffffffbec86897
Build id: 3.10.0-957.5.1.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffffbec89776
Build id: 3.10.0-957.5.1.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffffbecb158c
Build id: 3.10.0-957.5.1.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffffbec865e0
Build id: 3.10.0-957.5.1.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffffbecb2875
Build id: 3.10.0-957.5.1.el7.x86_64 		0x335
16
? default_wake_function
vmlinux 0xffffffffbe6d6802
Build id: 3.10.0-957.5.1.el7.x86_64 		0x12
17
? __wake_up_common
vmlinux 0xffffffffbe6cbadb
Build id: 3.10.0-957.5.1.el7.x86_64 		0x5b
18
? sock_has_perm
vmlinux 0xffffffffbe8fc985
Build id: 3.10.0-957.5.1.el7.x86_64 		0x75
19
inet_sendmsg
vmlinux 0xffffffffbecbff39
Build id: 3.10.0-957.5.1.el7.x86_64 		0x69
20
? selinux_socket_sendmsg
vmlinux 0xffffffffbe8fcab3
Build id: 3.10.0-957.5.1.el7.x86_64 		0x23
21
sock_sendmsg
vmlinux 0xffffffffbec193a6
Build id: 3.10.0-957.5.1.el7.x86_64 		0xb6
22
? unix_dgram_sendmsg
vmlinux 0xffffffffbecf771b
Build id: 3.10.0-957.5.1.el7.x86_64 		0x3cb
23
___sys_sendmsg
vmlinux 0xffffffffbec1a269
Build id: 3.10.0-957.5.1.el7.x86_64 		0x3e9
24
? ip4_datagram_release_cb
vmlinux 0xffffffffbecaf00f
Build id: 3.10.0-957.5.1.el7.x86_64 		0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffbed6a31e
Build id: 3.10.0-957.5.1.el7.x86_64 		0x1e
26
? udp_lib_rehash
vmlinux 0xffffffffbecb1e8f
Build id: 3.10.0-957.5.1.el7.x86_64 		0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffbed6a31e
Build id: 3.10.0-957.5.1.el7.x86_64 		0x1e
28
? release_sock
vmlinux 0xffffffffbec1ea00
Build id: 3.10.0-957.5.1.el7.x86_64 		0x120
29
? ip4_datagram_connect
vmlinux 0xffffffffbecaf477
Build id: 3.10.0-957.5.1.el7.x86_64 		0x37
30
? inet_dgram_connect
vmlinux 0xffffffffbecbebc4
Build id: 3.10.0-957.5.1.el7.x86_64 		0x34
31
? SYSC_connect
vmlinux 0xffffffffbec1996d
Build id: 3.10.0-957.5.1.el7.x86_64 		0xed
32
__sys_sendmsg
vmlinux 0xffffffffbec1b921
Build id: 3.10.0-957.5.1.el7.x86_64 		0x51
33
SyS_sendmsg
vmlinux 0xffffffffbec1b972
Build id: 3.10.0-957.5.1.el7.x86_64 		0x12
34
system_call_fastpath
vmlinux 0xffffffffbed74ddb
Build id: 3.10.0-957.5.1.el7.x86_64 		0x22 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 31s! [chronyd:9190]
Modules linked in: tcp_lp rfcomm xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack fuse ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep sunrpc snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi joydev btusb snd_ac97_codec btrtl btbcm ac97_bus
btintel pcspkr bluetooth snd_seq snd_seq_device snd_pcm snd_timer rfkill snd soundcore sg vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common ata_generic pata_acpi crc32c_intel vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi e1000 libata scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 9190 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8d712a3b8000 ti: ffff8d6f39d50000 task.ti: ffff8d6f39d50000
RIP: 0010:[<ffffffffc02ab857>]  [<ffffffffc02ab857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8d6f39d53760  EFLAGS: 00000286
RAX: ffffb10181c03818 RBX: ffffffffbec22c73 RCX: 0000000000000006
RDX: 0000000000000006 RSI: ffff8d712f344140 RDI: ffff8d70e622c000
RBP: ffff8d6f39d537e0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 00000001c07d8915 R15: ffffffffbe9a0a90
FS:  00007f10f5b1b740(0000) GS:ffff8d7133680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1620e15000 CR3: 00000000ba80e000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffbec21717>] ? kfree_skbmem+0x37/0x90
[<ffffffffbec38ce6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffbec65d6a>] sch_direct_xmit+0x11a/0x250
[<ffffffffbec3bc61>] __dev_queue_xmit+0x491/0x650
[<ffffffffc080f2ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffffbec3be30>] dev_queue_xmit+0x10/0x20
[<ffffffffbec887b6>] ip_finish_output+0x556/0x7b0
[<ffffffffbec88d13>] ip_output+0x73/0xe0
[<ffffffffbec88260>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffbec86897>] ip_local_out_sk+0x37/0x40
[<ffffffffbec89776>] ip_send_skb+0x16/0x50
[<ffffffffbecb158c>] udp_send_skb+0xac/0x2b0
[<ffffffffbec865e0>] ? ip_copy_metadata+0x170/0x170
[<ffffffffbecb2875>] udp_sendmsg+0x335/0xa10
[<ffffffffbe6d6802>] ? default_wake_function+0x12/0x20
[<ffffffffbe6cbadb>] ? __wake_up_common+0x5b/0x90
[<ffffffffbe8fc985>] ? sock_has_perm+0x75/0x90
[<ffffffffbecbff39>] inet_sendmsg+0x69/0xb0
[<ffffffffbe8fcab3>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffffbec193a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffbecf771b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffbec1a269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffbecaf00f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffbed6a31e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffbecb1e8f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffbed6a31e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffbec1ea00>] ? release_sock+0x120/0x170
[<ffffffffbecaf477>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffbecbebc4>] ? inet_dgram_connect+0x34/0x90
[<ffffffffbec1996d>] ? SYSC_connect+0xed/0x130
[<ffffffffbec1b921>] __sys_sendmsg+0x51/0x90
[<ffffffffbec1b972>] SyS_sendmsg+0x12/0x20
[<ffffffffbed74ddb>] system_call_fastpath+0x22/0x27
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #467484
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc02ec877
Build id: 3.10.0-1062.el7.x86_64 		0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffb2a356e7
Build id: 3.10.0-1062.el7.x86_64 		0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffb2a4d346
Build id: 3.10.0-1062.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffffb2a7b0ba
Build id: 3.10.0-1062.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffb2a502b1
Build id: 3.10.0-1062.el7.x86_64 		0x491
6
dev_queue_xmit
vmlinux 0xffffffffb2a50480
Build id: 3.10.0-1062.el7.x86_64 		0x10
7
neigh_resolve_output
vmlinux 0xffffffffb2a5cfed
Build id: 3.10.0-1062.el7.x86_64 		0x11d
8
ip_finish_output
vmlinux 0xffffffffb2a9e57c
Build id: 3.10.0-1062.el7.x86_64 		0x27c
9
ip_output
vmlinux 0xffffffffb2a9eefb
Build id: 3.10.0-1062.el7.x86_64 		0x7b
10
? __ip_local_out_sk
vmlinux 0xffffffffb2a9c8e2
Build id: 3.10.0-1062.el7.x86_64 		0x102
11
ip_local_out_sk
vmlinux 0xffffffffb2a9c927
Build id: 3.10.0-1062.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffffb2a9f966
Build id: 3.10.0-1062.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffffb2ac7c6c
Build id: 3.10.0-1062.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffffb2a9c660
Build id: 3.10.0-1062.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffffb2ac91d1
Build id: 3.10.0-1062.el7.x86_64 		0x5e1
16
? ep_poll_callback
vmlinux 0xffffffffb2696f4e
Build id: 3.10.0-1062.el7.x86_64 		0xee
17
? __wake_up_common
vmlinux 0xffffffffb24d24d2
Build id: 3.10.0-1062.el7.x86_64 		0x82
18
inet_sendmsg
vmlinux 0xffffffffb2ad6939
Build id: 3.10.0-1062.el7.x86_64 		0x69
19
sock_sendmsg
vmlinux 0xffffffffb2a2d3a6
Build id: 3.10.0-1062.el7.x86_64 		0xb6
20
? unix_dgram_sendmsg
vmlinux 0xffffffffb2b0e6db
Build id: 3.10.0-1062.el7.x86_64 		0x3cb
21
___sys_sendmsg
vmlinux 0xffffffffb2a2e269
Build id: 3.10.0-1062.el7.x86_64 		0x3e9
22
? __ip_route_output_key_hash
vmlinux 0xffffffffb2a95257
Build id: 3.10.0-1062.el7.x86_64 		0x207
23
? ip4_datagram_release_cb
vmlinux 0xffffffffb2ac567f
Build id: 3.10.0-1062.el7.x86_64 		0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffb2b8196e
Build id: 3.10.0-1062.el7.x86_64 		0x1e
25
? udp_lib_rehash
vmlinux 0xffffffffb2ac874f
Build id: 3.10.0-1062.el7.x86_64 		0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffffb2b8196e
Build id: 3.10.0-1062.el7.x86_64 		0x1e
27
? release_sock
vmlinux 0xffffffffb2a32a10
Build id: 3.10.0-1062.el7.x86_64 		0x120
28
? ip4_datagram_connect
vmlinux 0xffffffffb2ac5ae7
Build id: 3.10.0-1062.el7.x86_64 		0x37
29
? inet_dgram_connect
vmlinux 0xffffffffb2ad55c4
Build id: 3.10.0-1062.el7.x86_64 		0x34
30
? SYSC_connect
vmlinux 0xffffffffb2a2d96d
Build id: 3.10.0-1062.el7.x86_64 		0xed
31
__sys_sendmsg
vmlinux 0xffffffffb2a2f921
Build id: 3.10.0-1062.el7.x86_64 		0x51
32
SyS_sendmsg
vmlinux 0xffffffffb2a2f972
Build id: 3.10.0-1062.el7.x86_64 		0x12
33
system_call_fastpath
vmlinux 0xffffffffb2b8bede
Build id: 3.10.0-1062.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 25s! [chronyd:987]
Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink bnep vmw_vsock_vmci_transport vsock sunrpc btusb btrtl btbcm btintel bluetooth rfkill ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw vmw_balloon gf128mul pcspkr glue_helper ablk_helper joydev cryptd sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt ata_piix fb_sys_fops ttm nfit libata libnvdimm mptspi drm crct10dif_pclmul crct10dif_common scsi_transport_spi crc32c_intel mptscsih mptbase serio_raw e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 987 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8f02b160e2a0 ti: ffff8f02b5180000 task.ti: ffff8f02b5180000
RIP: 0010:[<ffffffffc02ec877>]  [<ffffffffc02ec877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8f02b5183710  EFLAGS: 00000286
RAX: ffff9f8e40983818 RBX: ffffffffb2a35e63 RCX: 00000000000000c6
RDX: 00000000000000c6 RSI: ffff8f01b5c32580 RDI: ffff8f02b5662000
RBP: ffff8f02b5183790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 0000000000000001 R15: ffff8f02b96dac40
FS:  00007fd19492d740(0000) GS:ffff8f02b96c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007faa17730000 CR3: 00000001352a4000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb2a356e7>] ? kfree_skbmem+0x37/0x90
[<ffffffffb2a4d346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb2a7b0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffffb2a502b1>] __dev_queue_xmit+0x491/0x650
[<ffffffffb2a50480>] dev_queue_xmit+0x10/0x20
[<ffffffffb2a5cfed>] neigh_resolve_output+0x11d/0x220
[<ffffffffb2a9e57c>] ip_finish_output+0x27c/0x8d0
[<ffffffffb2a9eefb>] ip_output+0x7b/0xf0
[<ffffffffb2a9c8e2>] ? __ip_local_out_sk+0x102/0x110
[<ffffffffb2a9c927>] ip_local_out_sk+0x37/0x40
[<ffffffffb2a9f966>] ip_send_skb+0x16/0x50
[<ffffffffb2ac7c6c>] udp_send_skb+0xac/0x2b0
[<ffffffffb2a9c660>] ? ip_copy_metadata+0x170/0x170
[<ffffffffb2ac91d1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffb2696f4e>] ? ep_poll_callback+0xee/0x210
[<ffffffffb24d24d2>] ? __wake_up_common+0x82/0x120
[<ffffffffb2ad6939>] inet_sendmsg+0x69/0xb0
[<ffffffffb2a2d3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffb2b0e6db>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffb2a2e269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffb2a95257>] ? __ip_route_output_key_hash+0x207/0x7b0
[<ffffffffb2ac567f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffb2b8196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb2ac874f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffb2b8196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb2a32a10>] ? release_sock+0x120/0x170
[<ffffffffb2ac5ae7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffb2ad55c4>] ? inet_dgram_connect+0x34/0x90
[<ffffffffb2a2d96d>] ? SYSC_connect+0xed/0x130
[<ffffffffb2a2f921>] __sys_sendmsg+0x51/0x90
[<ffffffffb2a2f972>] SyS_sendmsg+0x12/0x20
[<ffffffffb2b8bede>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #371331
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00f4877
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffaea55f96
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffffaea840aa
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffaea58f11
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x491
5
dev_queue_xmit
vmlinux 0xffffffffaea590e0
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x10
6
ip_finish_output
vmlinux 0xffffffffaeaa7c6b
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x67b
7
ip_output
vmlinux 0xffffffffaeaa81eb
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffffaeaa75f0
Build id: 3.10.0-1160.41.1.el7.x86_64 		0xa60
9
ip_local_out_sk
vmlinux 0xffffffffaeaa5c17
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x37
10
ip_send_skb
vmlinux 0xffffffffaeaa8c56
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x16
11
udp_send_skb
vmlinux 0xffffffffaead10bc
Build id: 3.10.0-1160.41.1.el7.x86_64 		0xac
12
? ip_copy_metadata
vmlinux 0xffffffffaeaa5950
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x170
13
udp_sendmsg
vmlinux 0xffffffffaead2621
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x5e1
14
? __pollwait
vmlinux 0xffffffffae664620
Build id: 3.10.0-1160.41.1.el7.x86_64 		0xf0
15
? __pollwait
vmlinux 0xffffffffae664620
Build id: 3.10.0-1160.41.1.el7.x86_64 		0xf0
16
inet_sendmsg
vmlinux 0xffffffffaeadfe09
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x69
17
? __pollwait
vmlinux 0xffffffffae664620
Build id: 3.10.0-1160.41.1.el7.x86_64 		0xf0
18
sock_sendmsg
vmlinux 0xffffffffaea363a6
Build id: 3.10.0-1160.41.1.el7.x86_64 		0xb6
19
___sys_sendmsg
vmlinux 0xffffffffaea37269
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x3e9
20
? __ip_route_output_key_hash
vmlinux 0xffffffffaea9e483
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x223
21
? ip4_datagram_release_cb
vmlinux 0xffffffffaeaceaef
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x6f
22
? _raw_spin_unlock_bh
vmlinux 0xffffffffaeb8b93e
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x1e
23
? udp_lib_rehash
vmlinux 0xffffffffaead1b9f
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x11f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffaeb8b93e
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x1e
25
? release_sock
vmlinux 0xffffffffaea3ba18
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x128
26
? ip4_datagram_connect
vmlinux 0xffffffffaeacef57
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x37
27
? inet_dgram_connect
vmlinux 0xffffffffaeadea94
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x34
28
? SYSC_connect
vmlinux 0xffffffffaea3696d
Build id: 3.10.0-1160.41.1.el7.x86_64 		0xed
29
__sys_sendmsg
vmlinux 0xffffffffaea38921
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x51
30
SyS_sendmsg
vmlinux 0xffffffffaea38972
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x12
31
system_call_fastpath
vmlinux 0xffffffffaeb95f92
Build id: 3.10.0-1160.41.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [chronyd:612]
Modules linked in: lp tcp_lp rpcsec_gss_krb5 xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus vmw_balloon snd_seq aesni_intel
lrw gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm joydev pcspkr sg snd_timer snd soundcore vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper nfit syscopyarea sysfillrect sysimgblt crct10dif_pclmul crct10dif_common libnvdimm fb_sys_fops ttm crc32c_intel ata_piix mptspi libata scsi_transport_spi mptscsih drm e1000 serio_raw mptbase drm_panel_orientation_quirks fuse
CPU: 1 PID: 612 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.41.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9338b4524200 ti: ffff93383679c000 task.ti: ffff93383679c000
RIP: 0010:[<ffffffffc00f4877>]  [<ffffffffc00f4877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff93383679f750  EFLAGS: 00000286
RAX: ffffb79ec0703818 RBX: ffffffffaea3ee73 RCX: 0000000000000014
RDX: 0000000000000014 RSI: ffff9337b5e37100 RDI: ffff9338b4648000
RBP: ffff93383679f7d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffffffffc06cb8dd R15: ffff93383679f6e8
FS:  00007f18d43df740(0000) GS:ffff9338b9640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0476e083bc CR3: 00000000b66a6000 CR4: 00000000003607e0
Call Trace:
[<ffffffffaea55f96>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffaea840aa>] sch_direct_xmit+0x11a/0x250
[<ffffffffaea58f11>] __dev_queue_xmit+0x491/0x650
[<ffffffffaea590e0>] dev_queue_xmit+0x10/0x20
[<ffffffffaeaa7c6b>] ip_finish_output+0x67b/0x8d0
[<ffffffffaeaa81eb>] ip_output+0x7b/0xf0
[<ffffffffaeaa75f0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffaeaa5c17>] ip_local_out_sk+0x37/0x40
[<ffffffffaeaa8c56>] ip_send_skb+0x16/0x50
[<ffffffffaead10bc>] udp_send_skb+0xac/0x2b0
[<ffffffffaeaa5950>] ? ip_copy_metadata+0x170/0x170
[<ffffffffaead2621>] udp_sendmsg+0x5e1/0xa70
[<ffffffffae664620>] ? __pollwait+0xf0/0xf0
[<ffffffffae664620>] ? __pollwait+0xf0/0xf0
[<ffffffffaeadfe09>] inet_sendmsg+0x69/0xb0
[<ffffffffae664620>] ? __pollwait+0xf0/0xf0
[<ffffffffaea363a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffaea37269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffaea9e483>] ? __ip_route_output_key_hash+0x223/0x7e0
[<ffffffffaeaceaef>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffaeb8b93e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffaead1b9f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffaeb8b93e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffaea3ba18>] ? release_sock+0x128/0x180
[<ffffffffaeacef57>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffaeadea94>] ? inet_dgram_connect+0x34/0x90
[<ffffffffaea3696d>] ? SYSC_connect+0xed/0x130
[<ffffffffaea38921>] __sys_sendmsg+0x51/0x90
[<ffffffffaea38972>] SyS_sendmsg+0x12/0x20
[<ffffffffaeb95f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #351193
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc03c5867
Build id: 3.10.0-862.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff979eb166
Build id: 3.10.0-862.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffff97a16a2a
Build id: 3.10.0-862.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff979ee051
Build id: 3.10.0-862.el7.x86_64 		0x4a1
5
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc094a2ce
Build id: 3.10.0-862.el7.x86_64 		0x4e
6
dev_queue_xmit
vmlinux 0xffffffff979ee220
Build id: 3.10.0-862.el7.x86_64 		0x10
7
ip_finish_output
vmlinux 0xffffffff97a37416
Build id: 3.10.0-862.el7.x86_64 		0x546
8
ip_output
vmlinux 0xffffffff97a37973
Build id: 3.10.0-862.el7.x86_64 		0x73
9
? __ip_append_data.isra.50
vmlinux 0xffffffff97a36ed0
Build id: 3.10.0-862.el7.x86_64 		0xa50
10
ip_local_out_sk
vmlinux 0xffffffff97a35567
Build id: 3.10.0-862.el7.x86_64 		0x37
11
ip_send_skb
vmlinux 0xffffffff97a383c6
Build id: 3.10.0-862.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffff97a5fc4c
Build id: 3.10.0-862.el7.x86_64 		0xac
13
? ip_copy_metadata
vmlinux 0xffffffff97a352b0
Build id: 3.10.0-862.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffff97a610fc
Build id: 3.10.0-862.el7.x86_64 		0x32c
15
? __alloc_skb
vmlinux 0xffffffff979d6d0d
Build id: 3.10.0-862.el7.x86_64 		0x8d
16
? __alloc_skb
vmlinux 0xffffffff979d6d21
Build id: 3.10.0-862.el7.x86_64 		0xa1
17
? ep_poll_callback
vmlinux 0xffffffff97667bee
Build id: 3.10.0-862.el7.x86_64 		0xee
18
inet_sendmsg
vmlinux 0xffffffff97a6e5a9
Build id: 3.10.0-862.el7.x86_64 		0x69
19
sock_sendmsg
vmlinux 0xffffffff979cc396
Build id: 3.10.0-862.el7.x86_64 		0xb6
20
? unix_dgram_sendmsg
vmlinux 0xffffffff97aa4dab
Build id: 3.10.0-862.el7.x86_64 		0x3cb
21
___sys_sendmsg
vmlinux 0xffffffff979cd199
Build id: 3.10.0-862.el7.x86_64 		0x3a9
22
? ip4_datagram_release_cb
vmlinux 0xffffffff97a5d72f
Build id: 3.10.0-862.el7.x86_64 		0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffff97b1551e
Build id: 3.10.0-862.el7.x86_64 		0x1e
24
? udp_lib_rehash
vmlinux 0xffffffff97a6054f
Build id: 3.10.0-862.el7.x86_64 		0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffff97b1551e
Build id: 3.10.0-862.el7.x86_64 		0x1e
26
? release_sock
vmlinux 0xffffffff979d1660
Build id: 3.10.0-862.el7.x86_64 		0x120
27
? ip4_datagram_connect
vmlinux 0xffffffff97a5db97
Build id: 3.10.0-862.el7.x86_64 		0x37
28
? inet_dgram_connect
vmlinux 0xffffffff97a6d234
Build id: 3.10.0-862.el7.x86_64 		0x34
29
? SYSC_connect
vmlinux 0xffffffff979cc93d
Build id: 3.10.0-862.el7.x86_64 		0xed
30
__sys_sendmsg
vmlinux 0xffffffff979ce761
Build id: 3.10.0-862.el7.x86_64 		0x51
31
SyS_sendmsg
vmlinux 0xffffffff979ce7b2
Build id: 3.10.0-862.el7.x86_64 		0x12
32
system_call_fastpath
vmlinux 0xffffffff97b1f7d5
Build id: 3.10.0-862.el7.x86_64 		0x1c 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [chronyd:669]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi kvm_intel kvm snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus irqbypass snd_seq crc32_pclmul ppdev snd_seq_device snd_pcm ghash_clmulni_intel cryptd vmw_balloon joydev pcspkr snd_timer snd sg soundcore i2c_piix4 vmw_vmci shpchp nfit libnvdimm parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul syscopyarea crct10dif_common
sysfillrect sysimgblt fb_sys_fops crc32c_intel ttm drm e1000 mptspi serio_raw scsi_transport_spi mptscsih ata_piix mptbase i2c_core libata floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 669 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff9f1ef117dee0 ti: ffff9f1e39de8000 task.ti: ffff9f1e39de8000
RIP: 0010:[<ffffffffc03c5867>]  [<ffffffffc03c5867>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9f1e39deb760  EFLAGS: 00000286
RAX: ffffaa5cc0f03818 RBX: ffffffff979d58b3 RCX: 0000000000000066
RDX: 0000000000000066 RSI: ffff9f1db5ea7e40 RDI: ffff9f1ef4a88000
RBP: ffff9f1e39deb7e0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 00000001c09248f5 R15: ffffffff97773e00
FS:  00007ff4196ac740(0000) GS:ffff9f1ef9600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd84b192000 CR3: 00000000ba56a000 CR4: 00000000001607f0
Call Trace:
[<ffffffff979eb166>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff97a16a2a>] sch_direct_xmit+0x11a/0x250
[<ffffffff979ee051>] __dev_queue_xmit+0x4a1/0x660
[<ffffffffc094a2ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffff979ee220>] dev_queue_xmit+0x10/0x20
[<ffffffff97a37416>] ip_finish_output+0x546/0x7a0
[<ffffffff97a37973>] ip_output+0x73/0xe0
[<ffffffff97a36ed0>] ? __ip_append_data.isra.50+0xa50/0xa50
[<ffffffff97a35567>] ip_local_out_sk+0x37/0x40
[<ffffffff97a383c6>] ip_send_skb+0x16/0x50
[<ffffffff97a5fc4c>] udp_send_skb+0xac/0x2b0
[<ffffffff97a352b0>] ? ip_copy_metadata+0x170/0x170
[<ffffffff97a610fc>] udp_sendmsg+0x32c/0xa00
[<ffffffff979d6d0d>] ? __alloc_skb+0x8d/0x2d0
[<ffffffff979d6d21>] ? __alloc_skb+0xa1/0x2d0
[<ffffffff97667bee>] ? ep_poll_callback+0xee/0x210
[<ffffffff97a6e5a9>] inet_sendmsg+0x69/0xb0
[<ffffffff979cc396>] sock_sendmsg+0xb6/0xf0
[<ffffffff97aa4dab>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffff979cd199>] ___sys_sendmsg+0x3a9/0x3c0
[<ffffffff97a5d72f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff97b1551e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff97a6054f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff97b1551e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff979d1660>] ? release_sock+0x120/0x170
[<ffffffff97a5db97>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff97a6d234>] ? inet_dgram_connect+0x34/0x90
[<ffffffff979cc93d>] ? SYSC_connect+0xed/0x130
[<ffffffff979ce761>] __sys_sendmsg+0x51/0x90
[<ffffffff979ce7b2>] SyS_sendmsg+0x12/0x20
[<ffffffff97b1f7d5>] system_call_fastpath+0x1c/0x21
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #286329
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc02b3877
Build id: 3.10.0-1160.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff8d254386
Build id: 3.10.0-1160.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffff8d28242a
Build id: 3.10.0-1160.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff8d257301
Build id: 3.10.0-1160.el7.x86_64 		0x491
5
dev_queue_xmit
vmlinux 0xffffffff8d2574d0
Build id: 3.10.0-1160.el7.x86_64 		0x10
6
ip_finish_output
vmlinux 0xffffffff8d2a545b
Build id: 3.10.0-1160.el7.x86_64 		0x67b
7
ip_output
vmlinux 0xffffffff8d2a59db
Build id: 3.10.0-1160.el7.x86_64 		0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffff8d2a4de0
Build id: 3.10.0-1160.el7.x86_64 		0xa60
9
ip_local_out_sk
vmlinux 0xffffffff8d2a3407
Build id: 3.10.0-1160.el7.x86_64 		0x37
10
ip_send_skb
vmlinux 0xffffffff8d2a6446
Build id: 3.10.0-1160.el7.x86_64 		0x16
11
udp_send_skb
vmlinux 0xffffffff8d2ce85c
Build id: 3.10.0-1160.el7.x86_64 		0xac
12
? ip_copy_metadata
vmlinux 0xffffffff8d2a3140
Build id: 3.10.0-1160.el7.x86_64 		0x170
13
udp_sendmsg
vmlinux 0xffffffff8d2cfdc1
Build id: 3.10.0-1160.el7.x86_64 		0x5e1
14
? __enqueue_entity
vmlinux 0xffffffff8ccdfb48
Build id: 3.10.0-1160.el7.x86_64 		0x78
15
? check_preempt_curr
vmlinux 0xffffffff8ccd7202
Build id: 3.10.0-1160.el7.x86_64 		0x92
16
? ttwu_do_wakeup
vmlinux 0xffffffff8ccd7229
Build id: 3.10.0-1160.el7.x86_64 		0x19
17
? ttwu_do_activate
vmlinux 0xffffffff8ccd735f
Build id: 3.10.0-1160.el7.x86_64 		0x6f
18
inet_sendmsg
vmlinux 0xffffffff8d2dd589
Build id: 3.10.0-1160.el7.x86_64 		0x69
19
sock_sendmsg
vmlinux 0xffffffff8d2343a6
Build id: 3.10.0-1160.el7.x86_64 		0xb6
20
? __wake_up_common
vmlinux 0xffffffff8ccd2fd2
Build id: 3.10.0-1160.el7.x86_64 		0x82
21
? sock_recvmsg
vmlinux 0xffffffff8d2344f5
Build id: 3.10.0-1160.el7.x86_64 		0xc5
22
? ep_poll_callback
vmlinux 0xffffffff8ce9d008
Build id: 3.10.0-1160.el7.x86_64 		0xf8
23
___sys_sendmsg
vmlinux 0xffffffff8d235269
Build id: 3.10.0-1160.el7.x86_64 		0x3e9
24
? __ip_route_output_key_hash
vmlinux 0xffffffff8d29bc63
Build id: 3.10.0-1160.el7.x86_64 		0x223
25
? ip4_datagram_release_cb
vmlinux 0xffffffff8d2cc28f
Build id: 3.10.0-1160.el7.x86_64 		0x6f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff8d38992e
Build id: 3.10.0-1160.el7.x86_64 		0x1e
27
? udp_lib_rehash
vmlinux 0xffffffff8d2cf33f
Build id: 3.10.0-1160.el7.x86_64 		0x11f
28
? _raw_spin_unlock_bh
vmlinux 0xffffffff8d38992e
Build id: 3.10.0-1160.el7.x86_64 		0x1e
29
? release_sock
vmlinux 0xffffffff8d239a18
Build id: 3.10.0-1160.el7.x86_64 		0x128
30
? ip4_datagram_connect
vmlinux 0xffffffff8d2cc6f7
Build id: 3.10.0-1160.el7.x86_64 		0x37
31
? inet_dgram_connect
vmlinux 0xffffffff8d2dc214
Build id: 3.10.0-1160.el7.x86_64 		0x34
32
? SYSC_connect
vmlinux 0xffffffff8d234910
Build id: 3.10.0-1160.el7.x86_64 		0x90
33
__sys_sendmsg
vmlinux 0xffffffff8d236921
Build id: 3.10.0-1160.el7.x86_64 		0x51
34
SyS_sendmsg
vmlinux 0xffffffff8d236972
Build id: 3.10.0-1160.el7.x86_64 		0x12
35
system_call_fastpath
vmlinux 0xffffffff8d393f92
Build id: 3.10.0-1160.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 29s! [isc-worker0000:51810]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype tun br_netfilter devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus iosf_mbi snd_seq crc32_pclmul snd_seq_device ghash_clmulni_intel
snd_pcm ppdev aesni_intel lrw snd_timer gf128mul glue_helper vmw_balloon ablk_helper cryptd snd soundcore joydev pcspkr vmw_vmci sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix mptspi nfit scsi_transport_spi libata libnvdimm serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks fuse
CPU: 0 PID: 51810 Comm: isc-worker0000 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff95c561080000 ti: ffff95c567c68000 task.ti: ffff95c567c68000
RIP: 0010:[<ffffffffc02b3877>]  [<ffffffffc02b3877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff95c567c6b750  EFLAGS: 00000286
RAX: ffffbe4940403818 RBX: 0000000000000000 RCX: 00000000000000f7
RDX: 00000000000000f7 RSI: ffff95c5359c4140 RDI: ffff95c5769be000
RBP: ffff95c567c6b7d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: ffff95c567c6b7b8
R13: 0000400067c6b728 R14: ffffffffc08118dd R15: ffff95c567c6b6e8
FS:  00007fcf702aa700(0000) GS:ffff95c57b600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc8dc8c7fb8 CR3: 000000006103c000 CR4: 00000000003607f0
Call Trace:
[<ffffffff8d254386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff8d28242a>] sch_direct_xmit+0x11a/0x250
[<ffffffff8d257301>] __dev_queue_xmit+0x491/0x650
[<ffffffff8d2574d0>] dev_queue_xmit+0x10/0x20
[<ffffffff8d2a545b>] ip_finish_output+0x67b/0x8d0
[<ffffffff8d2a59db>] ip_output+0x7b/0xf0
[<ffffffff8d2a4de0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff8d2a3407>] ip_local_out_sk+0x37/0x40
[<ffffffff8d2a6446>] ip_send_skb+0x16/0x50
[<ffffffff8d2ce85c>] udp_send_skb+0xac/0x2b0
[<ffffffff8d2a3140>] ? ip_copy_metadata+0x170/0x170
[<ffffffff8d2cfdc1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff8ccdfb48>] ? __enqueue_entity+0x78/0x80
[<ffffffff8ccd7202>] ? check_preempt_curr+0x92/0xa0
[<ffffffff8ccd7229>] ? ttwu_do_wakeup+0x19/0xe0
[<ffffffff8ccd735f>] ? ttwu_do_activate+0x6f/0x80
[<ffffffff8d2dd589>] inet_sendmsg+0x69/0xb0
[<ffffffff8d2343a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff8ccd2fd2>] ? __wake_up_common+0x82/0x120
[<ffffffff8d2344f5>] ? sock_recvmsg+0xc5/0x100
[<ffffffff8ce9d008>] ? ep_poll_callback+0xf8/0x220
[<ffffffff8d235269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff8d29bc63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffff8d2cc28f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff8d38992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff8d2cf33f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff8d38992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff8d239a18>] ? release_sock+0x128/0x180
[<ffffffff8d2cc6f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff8d2dc214>] ? inet_dgram_connect+0x34/0x90
[<ffffffff8d234910>] ? SYSC_connect+0x90/0x130
[<ffffffff8d236921>] __sys_sendmsg+0x51/0x90
[<ffffffff8d236972>] SyS_sendmsg+0x12/0x20
[<ffffffff8d393f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #207460
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc01a5867
Build id: 3.10.0-862.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff979eb166
Build id: 3.10.0-862.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffff97a16a2a
Build id: 3.10.0-862.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff979ee051
Build id: 3.10.0-862.el7.x86_64 		0x4a1
5
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc05f72ce
Build id: 3.10.0-862.el7.x86_64 		0x4e
6
dev_queue_xmit
vmlinux 0xffffffff979ee220
Build id: 3.10.0-862.el7.x86_64 		0x10
7
ip_finish_output
vmlinux 0xffffffff97a37416
Build id: 3.10.0-862.el7.x86_64 		0x546
8
ip_output
vmlinux 0xffffffff97a37973
Build id: 3.10.0-862.el7.x86_64 		0x73
9
? __ip_append_data.isra.50
vmlinux 0xffffffff97a36ed0
Build id: 3.10.0-862.el7.x86_64 		0xa50
10
ip_local_out_sk
vmlinux 0xffffffff97a35567
Build id: 3.10.0-862.el7.x86_64 		0x37
11
ip_send_skb
vmlinux 0xffffffff97a383c6
Build id: 3.10.0-862.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffff97a5fc4c
Build id: 3.10.0-862.el7.x86_64 		0xac
13
? ip_copy_metadata
vmlinux 0xffffffff97a352b0
Build id: 3.10.0-862.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffff97a610fc
Build id: 3.10.0-862.el7.x86_64 		0x32c
15
? poll_select_copy_remaining
vmlinux 0xffffffff97630830
Build id: 3.10.0-862.el7.x86_64 		0x150
16
? poll_select_copy_remaining
vmlinux 0xffffffff97630830
Build id: 3.10.0-862.el7.x86_64 		0x150
17
inet_sendmsg
vmlinux 0xffffffff97a6e5a9
Build id: 3.10.0-862.el7.x86_64 		0x69
18
sock_sendmsg
vmlinux 0xffffffff979cc396
Build id: 3.10.0-862.el7.x86_64 		0xb6
19
? unix_dgram_sendmsg
vmlinux 0xffffffff97aa4dab
Build id: 3.10.0-862.el7.x86_64 		0x3cb
20
___sys_sendmsg
vmlinux 0xffffffff979cd199
Build id: 3.10.0-862.el7.x86_64 		0x3a9
21
? ip4_datagram_release_cb
vmlinux 0xffffffff97a5d72f
Build id: 3.10.0-862.el7.x86_64 		0x6f
22
? _raw_spin_unlock_bh
vmlinux 0xffffffff97b1551e
Build id: 3.10.0-862.el7.x86_64 		0x1e
23
? udp_lib_rehash
vmlinux 0xffffffff97a6054f
Build id: 3.10.0-862.el7.x86_64 		0x11f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff97b1551e
Build id: 3.10.0-862.el7.x86_64 		0x1e
25
? release_sock
vmlinux 0xffffffff979d1660
Build id: 3.10.0-862.el7.x86_64 		0x120
26
? ip4_datagram_connect
vmlinux 0xffffffff97a5db97
Build id: 3.10.0-862.el7.x86_64 		0x37
27
? inet_dgram_connect
vmlinux 0xffffffff97a6d234
Build id: 3.10.0-862.el7.x86_64 		0x34
28
? SYSC_connect
vmlinux 0xffffffff979cc93d
Build id: 3.10.0-862.el7.x86_64 		0xed
29
__sys_sendmsg
vmlinux 0xffffffff979ce761
Build id: 3.10.0-862.el7.x86_64 		0x51
30
SyS_sendmsg
vmlinux 0xffffffff979ce7b2
Build id: 3.10.0-862.el7.x86_64 		0x12
31
system_call_fastpath
vmlinux 0xffffffff97b1f7d5
Build id: 3.10.0-862.el7.x86_64 		0x1c 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 30s! [chronyd:798]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod ppdev coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd pcspkr joydev vmw_vmci sg i2c_piix4 shpchp nfit libnvdimm parport_pc parport sch_fq_codel
ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm e1000 mptspi ata_piix scsi_transport_spi crct10dif_pclmul crct10dif_common mptscsih crc32c_intel libata mptbase serio_raw i2c_core
CPU: 1 PID: 798 Comm: chronyd Not tainted 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffffa0d8f7755ee0 ti: ffffa0d8f764c000 task.ti: ffffa0d8f764c000
RIP: 0010:[<ffffffffc01a5867>]  [<ffffffffc01a5867>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffffa0d8f764f760  EFLAGS: 00000286
RAX: ffffabecc0703818 RBX: ffffffff979d58b3 RCX: 0000000000000028
RDX: 0000000000000028 RSI: ffffa0d875d0f3c0 RDI: ffffa0d975550000
RBP: ffffa0d8f764f7e0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffffffffc05bf8f5 R15: ffffa0d8f764f6f8
FS:  00007f83d19d7740(0000) GS:ffffa0d979640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fefe310c000 CR3: 00000000b75ec000 CR4: 00000000003607e0
Call Trace:
[<ffffffff979eb166>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff97a16a2a>] sch_direct_xmit+0x11a/0x250
[<ffffffff979ee051>] __dev_queue_xmit+0x4a1/0x660
[<ffffffffc05f72ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffff979ee220>] dev_queue_xmit+0x10/0x20
[<ffffffff97a37416>] ip_finish_output+0x546/0x7a0
[<ffffffff97a37973>] ip_output+0x73/0xe0
[<ffffffff97a36ed0>] ? __ip_append_data.isra.50+0xa50/0xa50
[<ffffffff97a35567>] ip_local_out_sk+0x37/0x40
[<ffffffff97a383c6>] ip_send_skb+0x16/0x50
[<ffffffff97a5fc4c>] udp_send_skb+0xac/0x2b0
[<ffffffff97a352b0>] ? ip_copy_metadata+0x170/0x170
[<ffffffff97a610fc>] udp_sendmsg+0x32c/0xa00
[<ffffffff97630830>] ? poll_select_copy_remaining+0x150/0x150
[<ffffffff97630830>] ? poll_select_copy_remaining+0x150/0x150
[<ffffffff97a6e5a9>] inet_sendmsg+0x69/0xb0
[<ffffffff979cc396>] sock_sendmsg+0xb6/0xf0
[<ffffffff97aa4dab>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffff979cd199>] ___sys_sendmsg+0x3a9/0x3c0
[<ffffffff97a5d72f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff97b1551e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff97a6054f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff97b1551e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff979d1660>] ? release_sock+0x120/0x170
[<ffffffff97a5db97>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff97a6d234>] ? inet_dgram_connect+0x34/0x90
[<ffffffff979cc93d>] ? SYSC_connect+0xed/0x130
[<ffffffff979ce761>] __sys_sendmsg+0x51/0x90
[<ffffffff979ce7b2>] SyS_sendmsg+0x12/0x20
[<ffffffff97b1f7d5>] system_call_fastpath+0x1c/0x21
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #117689
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0073ee7
Build id: 3.10.0-693.el7.x86_64 		0x947
2
dev_hard_start_xmit
vmlinux 0xffffffff81586771
Build id: 3.10.0-693.el7.x86_64 		0x171
3
sch_direct_xmit
vmlinux 0xffffffff815af50a
Build id: 3.10.0-693.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff81589566
Build id: 3.10.0-693.el7.x86_64 		0x226
5
dev_queue_xmit
vmlinux 0xffffffff815898a0
Build id: 3.10.0-693.el7.x86_64 		0x10
6
neigh_resolve_output
vmlinux 0xffffffff8159527b
Build id: 3.10.0-693.el7.x86_64 		0x11b
7
ip_finish_output
vmlinux 0xffffffff815cfaa7
Build id: 3.10.0-693.el7.x86_64 		0x297
8
ip_output
vmlinux 0xffffffff815d0293
Build id: 3.10.0-693.el7.x86_64 		0x73
9
? __ip_append_data.isra.48
vmlinux 0xffffffff815cf810
Build id: 3.10.0-693.el7.x86_64 		0xa00
10
ip_local_out_sk
vmlinux 0xffffffff815cdf31
Build id: 3.10.0-693.el7.x86_64 		0x31
11
ip_send_skb
vmlinux 0xffffffff815d0cd6
Build id: 3.10.0-693.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffff815f81b0
Build id: 3.10.0-693.el7.x86_64 		0x160
13
? ip_copy_metadata
vmlinux 0xffffffff815cdc80
Build id: 3.10.0-693.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffff815f92cc
Build id: 3.10.0-693.el7.x86_64 		0x32c
15
? poll_select_copy_remaining
vmlinux 0xffffffff81215dd0
Build id: 3.10.0-693.el7.x86_64 		0x150
16
? poll_select_copy_remaining
vmlinux 0xffffffff81215dd0
Build id: 3.10.0-693.el7.x86_64 		0x150
17
inet_sendmsg
vmlinux 0xffffffff81606893
Build id: 3.10.0-693.el7.x86_64 		0x63
18
sock_sendmsg
vmlinux 0xffffffff8156a580
Build id: 3.10.0-693.el7.x86_64 		0xb0
19
? unix_dgram_sendmsg
vmlinux 0xffffffff8163c506
Build id: 3.10.0-693.el7.x86_64 		0x3c6
20
___sys_sendmsg
vmlinux 0xffffffff8156ae29
Build id: 3.10.0-693.el7.x86_64 		0x3a9
21
? ip4_datagram_release_cb
vmlinux 0xffffffff815f5a29
Build id: 3.10.0-693.el7.x86_64 		0x69
22
? _raw_spin_unlock_bh
vmlinux 0xffffffff816ab9ee
Build id: 3.10.0-693.el7.x86_64 		0x1e
23
? udp_lib_rehash
vmlinux 0xffffffff815f7c3d
Build id: 3.10.0-693.el7.x86_64 		0xfd
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff816ab9ee
Build id: 3.10.0-693.el7.x86_64 		0x1e
25
? release_sock
vmlinux 0xffffffff8156e998
Build id: 3.10.0-693.el7.x86_64 		0x118
26
? ip4_datagram_connect
vmlinux 0xffffffff815f5e87
Build id: 3.10.0-693.el7.x86_64 		0x37
27
? inet_dgram_connect
vmlinux 0xffffffff8160556e
Build id: 3.10.0-693.el7.x86_64 		0x2e
28
? SYSC_connect
vmlinux 0xffffffff8156a497
Build id: 3.10.0-693.el7.x86_64 		0xe7
29
__sys_sendmsg
vmlinux 0xffffffff8156b5f1
Build id: 3.10.0-693.el7.x86_64 		0x51
30
SyS_sendmsg
vmlinux 0xffffffff8156b642
Build id: 3.10.0-693.el7.x86_64 		0x12
31
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64 		0x16 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 56s! [chronyd:1091]
Modules linked in: tcp_lp nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper snd_ens1371 cryptd vmw_balloon snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm joydev pcspkr snd_timer snd nfit soundcore sg libnvdimm parport_pc parport vmw_vmci i2c_piix4 shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif
crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul crct10dif_common fb_sys_fops ttm crc32c_intel drm ata_piix mptspi scsi_transport_spi libata mptscsih serio_raw e1000 mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 1091 Comm: chronyd Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff88013143dee0 ti: ffff880135a28000 task.ti: ffff880135a28000
RIP: 0010:[<ffffffffc0073ee7>]  [<ffffffffc0073ee7>] e1000_xmit_frame+0x947/0x1040 [e1000]
RSP: 0018:ffff880135a2b778  EFLAGS: 00000286
RAX: ffffc90000703818 RBX: ffff880135a2b718 RCX: 0000000000000008
RDX: 0000000000000008 RSI: ffff8800361290c0 RDI: ffff880135574000
RBP: ffff880135a2b7f8 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: ffff880135a2b740 R14: 0000000000000202 R15: 0000000000000202
FS:  00007fd0db3cf740(0000) GS:ffff880139600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006dde1c CR3: 0000000132c21000 CR4: 00000000000407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 ffff880136f56098 000000065198a7bd 0000000900000000
0000000000000200 ffff880100000000 ffff880100000000 0000000100001000
ffff8800b3e0c700 ffff880135574000 ffff880136f56000 ffff880135574000
Call Trace:
[<ffffffff81586771>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff815af50a>] sch_direct_xmit+0x11a/0x240
[<ffffffff81589566>] __dev_queue_xmit+0x226/0x550
[<ffffffff815898a0>] dev_queue_xmit+0x10/0x20
[<ffffffff8159527b>] neigh_resolve_output+0x11b/0x220
[<ffffffff815cfaa7>] ip_finish_output+0x297/0x780
[<ffffffff815d0293>] ip_output+0x73/0xe0
[<ffffffff815cf810>] ? __ip_append_data.isra.48+0xa00/0xa00
[<ffffffff815cdf31>] ip_local_out_sk+0x31/0x40
[<ffffffff815d0cd6>] ip_send_skb+0x16/0x50
[<ffffffff815f81b0>] udp_send_skb+0x160/0x290
[<ffffffff815cdc80>] ? ip_copy_metadata+0x170/0x170
[<ffffffff815f92cc>] udp_sendmsg+0x32c/0xa00
[<ffffffff81215dd0>] ? poll_select_copy_remaining+0x150/0x150
[<ffffffff81215dd0>] ? poll_select_copy_remaining+0x150/0x150
[<ffffffff81606893>] inet_sendmsg+0x63/0xb0
[<ffffffff8156a580>] sock_sendmsg+0xb0/0xf0
[<ffffffff8163c506>] ? unix_dgram_sendmsg+0x3c6/0x770
[<ffffffff8156ae29>] ___sys_sendmsg+0x3a9/0x3c0
[<ffffffff815f5a29>] ? ip4_datagram_release_cb+0x69/0x1b0
[<ffffffff816ab9ee>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff815f7c3d>] ? udp_lib_rehash+0xfd/0x130
[<ffffffff816ab9ee>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff8156e998>] ? release_sock+0x118/0x170
[<ffffffff815f5e87>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff8160556e>] ? inet_dgram_connect+0x2e/0x80
[<ffffffff8156a497>] ? SYSC_connect+0xe7/0x120
[<ffffffff8156b5f1>] __sys_sendmsg+0x51/0x90
[<ffffffff8156b642>] SyS_sendmsg+0x12/0x20
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #107043
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc04dbee7
Build id: 3.10.0-693.el7.x86_64 		0x947
2
dev_hard_start_xmit
vmlinux 0xffffffff81586771
Build id: 3.10.0-693.el7.x86_64 		0x171
3
sch_direct_xmit
vmlinux 0xffffffff815af50a
Build id: 3.10.0-693.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff81589566
Build id: 3.10.0-693.el7.x86_64 		0x226
5
dev_queue_xmit
vmlinux 0xffffffff815898a0
Build id: 3.10.0-693.el7.x86_64 		0x10
6
neigh_resolve_output
vmlinux 0xffffffff8159527b
Build id: 3.10.0-693.el7.x86_64 		0x11b
7
ip_finish_output
vmlinux 0xffffffff815cfaa7
Build id: 3.10.0-693.el7.x86_64 		0x297
8
ip_output
vmlinux 0xffffffff815d0293
Build id: 3.10.0-693.el7.x86_64 		0x73
9
? __ip_append_data.isra.48
vmlinux 0xffffffff815cf810
Build id: 3.10.0-693.el7.x86_64 		0xa00
10
ip_local_out_sk
vmlinux 0xffffffff815cdf31
Build id: 3.10.0-693.el7.x86_64 		0x31
11
ip_send_skb
vmlinux 0xffffffff815d0cd6
Build id: 3.10.0-693.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffff815f81b0
Build id: 3.10.0-693.el7.x86_64 		0x160
13
? ip_copy_metadata
vmlinux 0xffffffff815cdc80
Build id: 3.10.0-693.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffff815f92cc
Build id: 3.10.0-693.el7.x86_64 		0x32c
15
? default_wake_function
vmlinux 0xffffffff810c4822
Build id: 3.10.0-693.el7.x86_64 		0x12
16
? sock_has_perm
vmlinux 0xffffffff812b4d65
Build id: 3.10.0-693.el7.x86_64 		0x75
17
inet_sendmsg
vmlinux 0xffffffff81606893
Build id: 3.10.0-693.el7.x86_64 		0x63
18
? selinux_socket_sendmsg
vmlinux 0xffffffff812b4e93
Build id: 3.10.0-693.el7.x86_64 		0x23
19
sock_sendmsg
vmlinux 0xffffffff8156a580
Build id: 3.10.0-693.el7.x86_64 		0xb0
20
? unix_dgram_sendmsg
vmlinux 0xffffffff8163c506
Build id: 3.10.0-693.el7.x86_64 		0x3c6
21
___sys_sendmsg
vmlinux 0xffffffff8156ae29
Build id: 3.10.0-693.el7.x86_64 		0x3a9
22
? ip4_datagram_release_cb
vmlinux 0xffffffff815f5a29
Build id: 3.10.0-693.el7.x86_64 		0x69
23
? _raw_spin_unlock_bh
vmlinux 0xffffffff816ab9ee
Build id: 3.10.0-693.el7.x86_64 		0x1e
24
? udp_lib_rehash
vmlinux 0xffffffff815f7c3d
Build id: 3.10.0-693.el7.x86_64 		0xfd
25
? _raw_spin_unlock_bh
vmlinux 0xffffffff816ab9ee
Build id: 3.10.0-693.el7.x86_64 		0x1e
26
? release_sock
vmlinux 0xffffffff8156e998
Build id: 3.10.0-693.el7.x86_64 		0x118
27
? ip4_datagram_connect
vmlinux 0xffffffff815f5e87
Build id: 3.10.0-693.el7.x86_64 		0x37
28
? inet_dgram_connect
vmlinux 0xffffffff8160556e
Build id: 3.10.0-693.el7.x86_64 		0x2e
29
? SYSC_connect
vmlinux 0xffffffff8156a497
Build id: 3.10.0-693.el7.x86_64 		0xe7
30
__sys_sendmsg
vmlinux 0xffffffff8156b5f1
Build id: 3.10.0-693.el7.x86_64 		0x51
31
SyS_sendmsg
vmlinux 0xffffffff8156b642
Build id: 3.10.0-693.el7.x86_64 		0x12
32
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64 		0x16 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 30s! [chronyd:655]
Modules linked in: usblp ext4 mbcache jbd2 nf_conntrack_ftp nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event ppdev coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel cryptd vmw_balloon snd_ens1371 snd_rawmidi snd_ac97_codec pcspkr joydev ac97_bus snd_seq snd_seq_device e1000 snd_pcm sg nfit snd_timer snd
soundcore libnvdimm parport_pc parport vmwgfx ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm mptspi mptscsih mptbase scsi_transport_spi i2c_piix4 shpchp i2c_core vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libata serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 655 Comm: chronyd Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff880035e0bf40 ti: ffff8800bb3a4000 task.ti: ffff8800bb3a4000
RIP: 0010:[<ffffffffc04dbee7>]  [<ffffffffc04dbee7>] e1000_xmit_frame+0x947/0x1040 [e1000]
RSP: 0018:ffff8800bb3a7778  EFLAGS: 00000286
RAX: ffffc90001203818 RBX: ffff8800bb3a7718 RCX: 000000000000009a
RDX: 000000000000009a RSI: ffff8800b8ca1140 RDI: ffff8801354da000
RBP: ffff8800bb3a77f8 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: ffff8800bb3a7740 R14: 0000000000000202 R15: 0000000000000202
FS:  00007f5bd4a36740(0000) GS:ffff880139600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9be9fb0000 CR3: 00000000baa3a000 CR4: 00000000003407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 ffff880136f98098 0000009827985a54 0000000900000000
0000000000000200 ffff880100000000 ffff880000000000 0000000100001000
ffff880070a00b00 ffff8801354da000 ffff880136f98000 ffff8801354da000
Call Trace:
[<ffffffff81586771>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff815af50a>] sch_direct_xmit+0x11a/0x240
[<ffffffff81589566>] __dev_queue_xmit+0x226/0x550
[<ffffffff815898a0>] dev_queue_xmit+0x10/0x20
[<ffffffff8159527b>] neigh_resolve_output+0x11b/0x220
[<ffffffff815cfaa7>] ip_finish_output+0x297/0x780
[<ffffffff815d0293>] ip_output+0x73/0xe0
[<ffffffff815cf810>] ? __ip_append_data.isra.48+0xa00/0xa00
[<ffffffff815cdf31>] ip_local_out_sk+0x31/0x40
[<ffffffff815d0cd6>] ip_send_skb+0x16/0x50
[<ffffffff815f81b0>] udp_send_skb+0x160/0x290
[<ffffffff815cdc80>] ? ip_copy_metadata+0x170/0x170
[<ffffffff815f92cc>] udp_sendmsg+0x32c/0xa00
[<ffffffff810c4822>] ? default_wake_function+0x12/0x20
[<ffffffff812b4d65>] ? sock_has_perm+0x75/0x90
[<ffffffff81606893>] inet_sendmsg+0x63/0xb0
[<ffffffff812b4e93>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff8156a580>] sock_sendmsg+0xb0/0xf0
[<ffffffff8163c506>] ? unix_dgram_sendmsg+0x3c6/0x770
[<ffffffff8156ae29>] ___sys_sendmsg+0x3a9/0x3c0
[<ffffffff815f5a29>] ? ip4_datagram_release_cb+0x69/0x1b0
[<ffffffff816ab9ee>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff815f7c3d>] ? udp_lib_rehash+0xfd/0x130
[<ffffffff816ab9ee>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff8156e998>] ? release_sock+0x118/0x170
[<ffffffff815f5e87>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff8160556e>] ? inet_dgram_connect+0x2e/0x80
[<ffffffff8156a497>] ? SYSC_connect+0xe7/0x120
[<ffffffff8156b5f1>] __sys_sendmsg+0x51/0x90
[<ffffffff8156b642>] SyS_sendmsg+0x12/0x20
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #96221
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc01e3877
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffa423c6f7
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffa4253f96
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffffa428203a
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffa4256f11
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x491
6
dev_queue_xmit
vmlinux 0xffffffffa42570e0
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x10
7
ip_finish_output
vmlinux 0xffffffffa42a507b
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x67b
8
ip_output
vmlinux 0xffffffffa42a55fb
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffffa42a4a00
Build id: 3.10.0-1160.11.1.el7.x86_64 		0xa60
10
ip_local_out_sk
vmlinux 0xffffffffa42a3027
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x37
11
ip_send_skb
vmlinux 0xffffffffa42a6066
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffffa42ce49c
Build id: 3.10.0-1160.11.1.el7.x86_64 		0xac
13
? ip_copy_metadata
vmlinux 0xffffffffa42a2d60
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffffa42cfa01
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x5e1
15
? __pollwait
vmlinux 0xffffffffa3e64030
Build id: 3.10.0-1160.11.1.el7.x86_64 		0xf0
16
? __pollwait
vmlinux 0xffffffffa3e64030
Build id: 3.10.0-1160.11.1.el7.x86_64 		0xf0
17
inet_sendmsg
vmlinux 0xffffffffa42dd1c9
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x69
18
? enqueue_entity
vmlinux 0xffffffffa3ce684f
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x2ef
19
sock_sendmsg
vmlinux 0xffffffffa42343a6
Build id: 3.10.0-1160.11.1.el7.x86_64 		0xb6
20
? check_preempt_curr
vmlinux 0xffffffffa3cd7452
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x92
21
? ttwu_do_wakeup
vmlinux 0xffffffffa3cd7479
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x19
22
___sys_sendmsg
vmlinux 0xffffffffa4235269
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x3e9
23
? __ip_route_output_key_hash
vmlinux 0xffffffffa429b883
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x223
24
? ip4_datagram_release_cb
vmlinux 0xffffffffa42cbecf
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffa438958e
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x1e
26
? udp_lib_rehash
vmlinux 0xffffffffa42cef7f
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffa438958e
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x1e
28
? release_sock
vmlinux 0xffffffffa4239a18
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x128
29
? ip4_datagram_connect
vmlinux 0xffffffffa42cc337
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x37
30
? inet_dgram_connect
vmlinux 0xffffffffa42dbe54
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x34
31
? SYSC_connect
vmlinux 0xffffffffa423496d
Build id: 3.10.0-1160.11.1.el7.x86_64 		0xed
32
__sys_sendmsg
vmlinux 0xffffffffa4236921
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x51
33
SyS_sendmsg
vmlinux 0xffffffffa4236972
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x12
34
system_call_fastpath
vmlinux 0xffffffffa4393f92
Build id: 3.10.0-1160.11.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [chronyd:636]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371
snd_rawmidi snd_ac97_codec ac97_bus snd_seq pcspkr joydev ppdev i2c_piix4 snd_seq_device snd_pcm sg snd_timer snd vmw_balloon soundcore vmw_vmci parport_pc parport binfmt_misc ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper nfit syscopyarea sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm libnvdimm ata_piix drm mptspi scsi_transport_spi libata mptscsih e1000 mptbase serio_raw drm_panel_orientation_quirks fuse
CPU: 0 PID: 636 Comm: chronyd Tainted: G             L ------------ T 3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8cdeaf0ae300 ti: ffff8cdb3b9bc000 task.ti: ffff8cdb3b9bc000
RIP: 0010:[<ffffffffc01e3877>]  [<ffffffffc01e3877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8cdb3b9bf750  EFLAGS: 00000286
RAX: ffffa9c281983818 RBX: ffffffffa423ce73 RCX: 000000000000001b
RDX: 000000000000001b RSI: ffff8cdeaca7a140 RDI: ffff8cdbe53fc000
RBP: ffff8cdb3b9bf7d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 00000001c073d8dd R15: ffffffffa3fb1850
FS:  00007f00743a7740(0000) GS:ffff8cdead600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f961d570000 CR3: 00000000bb9a2000 CR4: 00000000003607f0
Call Trace:
[<ffffffffa423c6f7>] ? kfree_skbmem+0x37/0x90
[<ffffffffa4253f96>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa428203a>] sch_direct_xmit+0x11a/0x250
[<ffffffffa4256f11>] __dev_queue_xmit+0x491/0x650
[<ffffffffa42570e0>] dev_queue_xmit+0x10/0x20
[<ffffffffa42a507b>] ip_finish_output+0x67b/0x8d0
[<ffffffffa42a55fb>] ip_output+0x7b/0xf0
[<ffffffffa42a4a00>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffa42a3027>] ip_local_out_sk+0x37/0x40
[<ffffffffa42a6066>] ip_send_skb+0x16/0x50
[<ffffffffa42ce49c>] udp_send_skb+0xac/0x2b0
[<ffffffffa42a2d60>] ? ip_copy_metadata+0x170/0x170
[<ffffffffa42cfa01>] udp_sendmsg+0x5e1/0xa70
[<ffffffffa3e64030>] ? __pollwait+0xf0/0xf0
[<ffffffffa3e64030>] ? __pollwait+0xf0/0xf0
[<ffffffffa42dd1c9>] inet_sendmsg+0x69/0xb0
[<ffffffffa3ce684f>] ? enqueue_entity+0x2ef/0xbe0
[<ffffffffa42343a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffa3cd7452>] ? check_preempt_curr+0x92/0xa0
[<ffffffffa3cd7479>] ? ttwu_do_wakeup+0x19/0xe0
[<ffffffffa4235269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffa429b883>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffffa42cbecf>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffa438958e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa42cef7f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffa438958e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa4239a18>] ? release_sock+0x128/0x180
[<ffffffffa42cc337>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffa42dbe54>] ? inet_dgram_connect+0x34/0x90
[<ffffffffa423496d>] ? SYSC_connect+0xed/0x130
[<ffffffffa4236921>] __sys_sendmsg+0x51/0x90
[<ffffffffa4236972>] SyS_sendmsg+0x12/0x20
[<ffffffffa4393f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #46058
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc01a4877
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffa7e4f346
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffffa7e7d0ba
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffa7e522b1
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x491
5
? local_bh_enable
vmlinux 0xffffffffa78a4787
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x17
6
dev_queue_xmit
vmlinux 0xffffffffa7e52480
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x10
7
neigh_resolve_output
vmlinux 0xffffffffa7e5efed
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x11d
8
ip_finish_output
vmlinux 0xffffffffa7ea057c
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x27c
9
ip_output
vmlinux 0xffffffffa7ea0efb
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffffa7ea0300
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xa60
11
ip_local_out_sk
vmlinux 0xffffffffa7e9e927
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffffa7ea1966
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffffa7ec9cec
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffffa7e9e660
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffffa7ecb251
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x5e1
16
? __pollwait
vmlinux 0xffffffffa7a606a0
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xf0
17
? __pollwait
vmlinux 0xffffffffa7a606a0
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xf0
18
inet_sendmsg
vmlinux 0xffffffffa7ed89b9
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x69
19
sock_sendmsg
vmlinux 0xffffffffa7e2f3a6
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xb6
20
? unix_dgram_sendmsg
vmlinux 0xffffffffa7f1075b
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x3cb
21
___sys_sendmsg
vmlinux 0xffffffffa7e30269
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x3e9
22
? __ip_route_output_key_hash
vmlinux 0xffffffffa7e97257
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x207
23
? ip4_datagram_release_cb
vmlinux 0xffffffffa7ec76ff
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffa7f831ce
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x1e
25
? udp_lib_rehash
vmlinux 0xffffffffa7eca7cf
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffffa7f831ce
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x1e
27
? release_sock
vmlinux 0xffffffffa7e34a10
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x120
28
? ip4_datagram_connect
vmlinux 0xffffffffa7ec7b67
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x37
29
? inet_dgram_connect
vmlinux 0xffffffffa7ed7644
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x34
30
? SYSC_connect
vmlinux 0xffffffffa7e2f96d
Build id: 3.10.0-1062.9.1.el7.x86_64 		0xed
31
__sys_sendmsg
vmlinux 0xffffffffa7e31921
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x51
32
SyS_sendmsg
vmlinux 0xffffffffa7e31972
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x12
33
system_call_fastpath
vmlinux 0xffffffffa7f8dede
Build id: 3.10.0-1062.9.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 70s! [chronyd:892]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 br_netfilter tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock overlay(T) sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 sb_edac snd_rawmidi snd_ac97_codec coretemp ppdev iosf_mbi ac97_bus crc32_pclmul snd_seq snd_seq_device vmw_balloon snd_pcm ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_timer snd joydev soundcore pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c vmwgfx sr_mod cdrom ata_generic pata_acpi drm_kms_helper sd_mod crc_t10dif crct10dif_generic
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul libata crct10dif_common crc32c_intel mptspi scsi_transport_spi serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 892 Comm: chronyd Tainted: G             L ------------ T 3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
task: ffff962eb5145230 ti: ffff962eb8ad8000 task.ti: ffff962eb8ad8000
RIP: 0010:[<ffffffffc01a4877>]  [<ffffffffc01a4877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff962eb8adb710  EFLAGS: 00000286
RAX: ffffb06880283818 RBX: ffffffffa7e37e63 RCX: 00000000000000e0
RDX: 00000000000000e0 RSI: ffff962e80089100 RDI: ffff962ebb35c000
RBP: ffff962eb8adb790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffff962ebbdbdb00 R15: 0000000000010000
FS:  00007f94db776740(0000) GS:ffff962ebd600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f728a2a1740 CR3: 0000000038ad6000 CR4: 00000000001607f0
Call Trace:
[<ffffffffa7e4f346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa7e7d0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffffa7e522b1>] __dev_queue_xmit+0x491/0x650
[<ffffffffa78a4787>] ? local_bh_enable+0x17/0x20
[<ffffffffa7e52480>] dev_queue_xmit+0x10/0x20
[<ffffffffa7e5efed>] neigh_resolve_output+0x11d/0x220
[<ffffffffa7ea057c>] ip_finish_output+0x27c/0x8d0
[<ffffffffa7ea0efb>] ip_output+0x7b/0xf0
[<ffffffffa7ea0300>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffa7e9e927>] ip_local_out_sk+0x37/0x40
[<ffffffffa7ea1966>] ip_send_skb+0x16/0x50
[<ffffffffa7ec9cec>] udp_send_skb+0xac/0x2b0
[<ffffffffa7e9e660>] ? ip_copy_metadata+0x170/0x170
[<ffffffffa7ecb251>] udp_sendmsg+0x5e1/0xa70
[<ffffffffa7a606a0>] ? __pollwait+0xf0/0xf0
[<ffffffffa7a606a0>] ? __pollwait+0xf0/0xf0
[<ffffffffa7ed89b9>] inet_sendmsg+0x69/0xb0
[<ffffffffa7e2f3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffa7f1075b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffa7e30269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffa7e97257>] ? __ip_route_output_key_hash+0x207/0x7b0
[<ffffffffa7ec76ff>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffa7f831ce>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa7eca7cf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffa7f831ce>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa7e34a10>] ? release_sock+0x120/0x170
[<ffffffffa7ec7b67>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffa7ed7644>] ? inet_dgram_connect+0x34/0x90
[<ffffffffa7e2f96d>] ? SYSC_connect+0xed/0x130
[<ffffffffa7e31921>] __sys_sendmsg+0x51/0x90
[<ffffffffa7e31972>] SyS_sendmsg+0x12/0x20
[<ffffffffa7f8dede>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #24776
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00be877
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff9fa53386
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffff9fa8131a
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff9fa56301
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x491
5
dev_queue_xmit
vmlinux 0xffffffff9fa564d0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x10
6
ip_finish_output
vmlinux 0xffffffff9faa429b
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x67b
7
ip_output
vmlinux 0xffffffff9faa481b
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffff9faa3c20
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xa60
9
ip_local_out_sk
vmlinux 0xffffffff9faa2247
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x37
10
ip_send_skb
vmlinux 0xffffffff9faa5286
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x16
11
udp_send_skb
vmlinux 0xffffffff9facd65c
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xac
12
? ip_copy_metadata
vmlinux 0xffffffff9faa1f80
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x170
13
udp_sendmsg
vmlinux 0xffffffff9facebc1
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x5e1
14
? __pollwait
vmlinux 0xffffffff9f663430
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xf0
15
? __pollwait
vmlinux 0xffffffff9f663430
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xf0
16
inet_sendmsg
vmlinux 0xffffffff9fadc389
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x69
17
? __pollwait
vmlinux 0xffffffff9f663430
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xf0
18
sock_sendmsg
vmlinux 0xffffffff9fa333a6
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xb6
19
___sys_sendmsg
vmlinux 0xffffffff9fa34269
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x3e9
20
? __ip_route_output_key_hash
vmlinux 0xffffffff9fa9aacf
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x20f
21
? ip4_datagram_release_cb
vmlinux 0xffffffff9facb06f
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x6f
22
? _raw_spin_unlock_bh
vmlinux 0xffffffff9fb8856e
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x1e
23
? udp_lib_rehash
vmlinux 0xffffffff9face13f
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x11f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff9fb8856e
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x1e
25
? release_sock
vmlinux 0xffffffff9fa38a18
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x128
26
? ip4_datagram_connect
vmlinux 0xffffffff9facb4d7
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x37
27
? inet_dgram_connect
vmlinux 0xffffffff9fadb014
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x34
28
? SYSC_connect
vmlinux 0xffffffff9fa3396d
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xed
29
__sys_sendmsg
vmlinux 0xffffffff9fa35921
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x51
30
SyS_sendmsg
vmlinux 0xffffffff9fa35972
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x12
31
system_call_fastpath
vmlinux 0xffffffff9fb92ed2
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x25
32
? system_call_after_swapgs
vmlinux 0xffffffff9fb92e15
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xa2 
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [chronyd:551]
Modules linked in: xt_time nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ip6table_filter ip6_tables tcp_diag inet_diag ipt_REJECT nf_reject_ipv4 xt_owner xt_multiport iptable_filter ext4 mbcache jbd2 loop dm_mirror dm_region_hash dm_log dm_mod sg pcspkr i2c_piix4 joydev ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper virtio_blk syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata e1000 virtio_pci serio_raw virtio_ring virtio drm_panel_orientation_quirks floppy
CPU: 4 PID: 551 Comm: chronyd Kdump: loaded Tainted: G        W    L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-48-gd9c812dda519-prebuilt.qemu.org 04/01/2014
task: ffff967f542020e0 ti: ffff967f54f3c000 task.ti: ffff967f54f3c000
RIP: 0010:[<ffffffffc00be877>]  [<ffffffffc00be877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff967f54f3f750  EFLAGS: 00000286
RAX: ffffafa181c43818 RBX: ffffffff9fa3be73 RCX: 00000000000000fa
RDX: 00000000000000fa RSI: ffff967e01debf40 RDI: ffff967f54f20000
RBP: ffff967f54f3f7d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: ffff967f54f3f6c8 R14: ffff967f52412e88 R15: ffff967f54f3f6b8
FS:  00007f1ec4614740(0000) GS:ffff967f67d00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b2c9e67dc70 CR3: 0000000414168000 CR4: 00000000000006e0
Call Trace:
[<ffffffff9fa53386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff9fa8131a>] sch_direct_xmit+0x11a/0x250
[<ffffffff9fa56301>] __dev_queue_xmit+0x491/0x650
[<ffffffff9fa564d0>] dev_queue_xmit+0x10/0x20
[<ffffffff9faa429b>] ip_finish_output+0x67b/0x8d0
[<ffffffff9faa481b>] ip_output+0x7b/0xf0
[<ffffffff9faa3c20>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff9faa2247>] ip_local_out_sk+0x37/0x40
[<ffffffff9faa5286>] ip_send_skb+0x16/0x50
[<ffffffff9facd65c>] udp_send_skb+0xac/0x2b0
[<ffffffff9faa1f80>] ? ip_copy_metadata+0x170/0x170
[<ffffffff9facebc1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff9f663430>] ? __pollwait+0xf0/0xf0
[<ffffffff9f663430>] ? __pollwait+0xf0/0xf0
[<ffffffff9fadc389>] inet_sendmsg+0x69/0xb0
[<ffffffff9f663430>] ? __pollwait+0xf0/0xf0
[<ffffffff9fa333a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff9fa34269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff9fa9aacf>] ? __ip_route_output_key_hash+0x20f/0x7d0
[<ffffffff9facb06f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff9fb8856e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff9face13f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff9fb8856e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff9fa38a18>] ? release_sock+0x128/0x180
[<ffffffff9facb4d7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff9fadb014>] ? inet_dgram_connect+0x34/0x90
[<ffffffff9fa3396d>] ? SYSC_connect+0xed/0x130
[<ffffffff9fa35921>] __sys_sendmsg+0x51/0x90
[<ffffffff9fa35972>] SyS_sendmsg+0x12/0x20
[<ffffffff9fb92ed2>] system_call_fastpath+0x25/0x2a
[<ffffffff9fb92e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #19313
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc04de857
Build id: 3.10.0-957.27.2.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffa683ace6
Build id: 3.10.0-957.27.2.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffffa6867d6a
Build id: 3.10.0-957.27.2.el7.x86_64 		0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffa683dc61
Build id: 3.10.0-957.27.2.el7.x86_64 		0x491
5
? local_bh_enable
vmlinux 0xffffffffa62a15d7
Build id: 3.10.0-957.27.2.el7.x86_64 		0x17
6
dev_queue_xmit
vmlinux 0xffffffffa683de30
Build id: 3.10.0-957.27.2.el7.x86_64 		0x10
7
neigh_resolve_output
vmlinux 0xffffffffa684a69d
Build id: 3.10.0-957.27.2.el7.x86_64 		0x11d
8
ip_finish_output
vmlinux 0xffffffffa688a4dc
Build id: 3.10.0-957.27.2.el7.x86_64 		0x27c
9
ip_output
vmlinux 0xffffffffa688ad13
Build id: 3.10.0-957.27.2.el7.x86_64 		0x73
10
? __ip_append_data.isra.51
vmlinux 0xffffffffa688a260
Build id: 3.10.0-957.27.2.el7.x86_64 		0xa50
11
ip_local_out_sk
vmlinux 0xffffffffa6888897
Build id: 3.10.0-957.27.2.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffffa688b776
Build id: 3.10.0-957.27.2.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffffa68b36ac
Build id: 3.10.0-957.27.2.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffffa68885e0
Build id: 3.10.0-957.27.2.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffffa68b4995
Build id: 3.10.0-957.27.2.el7.x86_64 		0x335
16
? __pollwait
vmlinux 0xffffffffa64583a0
Build id: 3.10.0-957.27.2.el7.x86_64 		0xf0
17
? __pollwait
vmlinux 0xffffffffa64583a0
Build id: 3.10.0-957.27.2.el7.x86_64 		0xf0
18
inet_sendmsg
vmlinux 0xffffffffa68c2199
Build id: 3.10.0-957.27.2.el7.x86_64 		0x69
19
sock_sendmsg
vmlinux 0xffffffffa681b3a6
Build id: 3.10.0-957.27.2.el7.x86_64 		0xb6
20
___sys_sendmsg
vmlinux 0xffffffffa681c269
Build id: 3.10.0-957.27.2.el7.x86_64 		0x3e9
21
? ip4_datagram_release_cb
vmlinux 0xffffffffa68b112f
Build id: 3.10.0-957.27.2.el7.x86_64 		0x6f
22
? _raw_spin_unlock_bh
vmlinux 0xffffffffa696c63e
Build id: 3.10.0-957.27.2.el7.x86_64 		0x1e
23
? udp_lib_rehash
vmlinux 0xffffffffa68b3faf
Build id: 3.10.0-957.27.2.el7.x86_64 		0x11f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffa696c63e
Build id: 3.10.0-957.27.2.el7.x86_64 		0x1e
25
? release_sock
vmlinux 0xffffffffa6820a00
Build id: 3.10.0-957.27.2.el7.x86_64 		0x120
26
? ip4_datagram_connect
vmlinux 0xffffffffa68b1597
Build id: 3.10.0-957.27.2.el7.x86_64 		0x37
27
? inet_dgram_connect
vmlinux 0xffffffffa68c0e24
Build id: 3.10.0-957.27.2.el7.x86_64 		0x34
28
? SYSC_connect
vmlinux 0xffffffffa681b96d
Build id: 3.10.0-957.27.2.el7.x86_64 		0xed
29
__sys_sendmsg
vmlinux 0xffffffffa681d921
Build id: 3.10.0-957.27.2.el7.x86_64 		0x51
30
SyS_sendmsg
vmlinux 0xffffffffa681d972
Build id: 3.10.0-957.27.2.el7.x86_64 		0x12
31
system_call_fastpath
vmlinux 0xffffffffa6976ddb
Build id: 3.10.0-957.27.2.el7.x86_64 		0x22 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 47s! [vmtoolsd:1012]
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 47s! [chronyd:1033]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq ppdev snd_seq_device iosf_mbi snd_pcm crc32_pclmul ghash_clmulni_intel aesni_intel vmw_balloon lrw snd_timer gf128mul glue_helper ablk_helper cryptd snd soundcore pcspkr sg joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
ttm ata_piix drm crct10dif_pclmul crct10dif_common libata crc32c_intel e1000 nfit libnvdimm serio_raw mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1033 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-957.27.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff92a5f22730c0 ti: ffff92a577268000 task.ti: ffff92a577268000
RIP: 0010:[<ffffffffc04de857>]  [<ffffffffc04de857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff92a57726b720  EFLAGS: 00000286
RAX: ffffaf7640903818 RBX: ffffffffa6824c73 RCX: 000000000000001d
RDX: 000000000000001d RSI: ffff92a4f567e140 RDI: ffff92a5f8bf4000
RBP: ffff92a57726b7a0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 0000000100000000 R15: ffffffffa65a23e0
FS:  00007f3bf37ee740(0000) GS:ffff92a5fa640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f56cba03000 CR3: 00000000b7120000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa683ace6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa6867d6a>] sch_direct_xmit+0x11a/0x250
[<ffffffffa683dc61>] __dev_queue_xmit+0x491/0x650
[<ffffffffa62a15d7>] ? local_bh_enable+0x17/0x20
[<ffffffffa683de30>] dev_queue_xmit+0x10/0x20
[<ffffffffa684a69d>] neigh_resolve_output+0x11d/0x220
[<ffffffffa688a4dc>] ip_finish_output+0x27c/0x7b0
[<ffffffffa688ad13>] ip_output+0x73/0xe0
[<ffffffffa688a260>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffa6888897>] ip_local_out_sk+0x37/0x40
[<ffffffffa688b776>] ip_send_skb+0x16/0x50
[<ffffffffa68b36ac>] udp_send_skb+0xac/0x2b0
[<ffffffffa68885e0>] ? ip_copy_metadata+0x170/0x170
[<ffffffffa68b4995>] udp_sendmsg+0x335/0xa10
[<ffffffffa64583a0>] ? __pollwait+0xf0/0xf0
[<ffffffffa64583a0>] ? __pollwait+0xf0/0xf0
[<ffffffffa68c2199>] inet_sendmsg+0x69/0xb0
[<ffffffffa681b3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffa681c269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffa68b112f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffa696c63e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa68b3faf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffa696c63e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa6820a00>] ? release_sock+0x120/0x170
[<ffffffffa68b1597>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffa68c0e24>] ? inet_dgram_connect+0x34/0x90
[<ffffffffa681b96d>] ? SYSC_connect+0xed/0x130
[<ffffffffa681d921>] __sys_sendmsg+0x51/0x90
[<ffffffffa681d972>] SyS_sendmsg+0x12/0x20
[<ffffffffa6976ddb>] system_call_fastpath+0x22/0x27
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #6725
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00d8877
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffa304d346
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x246
3
sch_direct_xmit
vmlinux 0xffffffffa307b0ba
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x11a
4
__qdisc_run
vmlinux 0xffffffffa307b27e
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x8e
5
__dev_queue_xmit
vmlinux 0xffffffffa30502c9
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x4a9
6
dev_queue_xmit
vmlinux 0xffffffffa3050480
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x10
7
ip_finish_output
vmlinux 0xffffffffa309e97b
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x67b
8
ip_output
vmlinux 0xffffffffa309eefb
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffffa309e300
Build id: 3.10.0-1062.1.2.el7.x86_64 		0xa60
10
ip_local_out_sk
vmlinux 0xffffffffa309c927
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x37
11
ip_send_skb
vmlinux 0xffffffffa309f966
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x16
12
udp_send_skb
vmlinux 0xffffffffa30c7ccc
Build id: 3.10.0-1062.1.2.el7.x86_64 		0xac
13
? ip_copy_metadata
vmlinux 0xffffffffa309c660
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x170
14
udp_sendmsg
vmlinux 0xffffffffa30c9231
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x5e1
15
? ep_poll_callback
vmlinux 0xffffffffa2c96f4e
Build id: 3.10.0-1062.1.2.el7.x86_64 		0xee
16
? __wake_up_common
vmlinux 0xffffffffa2ad24d2
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x82
17
inet_sendmsg
vmlinux 0xffffffffa30d6999
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x69
18
sock_sendmsg
vmlinux 0xffffffffa302d3a6
Build id: 3.10.0-1062.1.2.el7.x86_64 		0xb6
19
? unix_dgram_sendmsg
vmlinux 0xffffffffa310e73b
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x3cb
20
___sys_sendmsg
vmlinux 0xffffffffa302e269
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x3e9
21
? __ip_route_output_key_hash
vmlinux 0xffffffffa3095257
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x207
22
? ip4_datagram_release_cb
vmlinux 0xffffffffa30c56df
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffffa31819ce
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x1e
24
? udp_lib_rehash
vmlinux 0xffffffffa30c87af
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffa31819ce
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x1e
26
? release_sock
vmlinux 0xffffffffa3032a10
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x120
27
? ip4_datagram_connect
vmlinux 0xffffffffa30c5b47
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x37
28
? inet_dgram_connect
vmlinux 0xffffffffa30d5624
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x34
29
? SYSC_connect
vmlinux 0xffffffffa302d96d
Build id: 3.10.0-1062.1.2.el7.x86_64 		0xed
30
__sys_sendmsg
vmlinux 0xffffffffa302f921
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x51
31
SyS_sendmsg
vmlinux 0xffffffffa302f972
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x12
32
system_call_fastpath
vmlinux 0xffffffffa318cede
Build id: 3.10.0-1062.1.2.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 30s! [chronyd:1100]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) tcp_lp nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink binfmt_misc vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq coretemp snd_seq_device snd_pcm ppdev iosf_mbi snd_timer crc32_pclmul ghash_clmulni_intel aesni_intel snd lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd soundcore joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci sch_fq_codel ip_tables xfs libcrc32c
sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm ata_piix mptspi nfit crc32c_intel drm scsi_transport_spi libnvdimm mptscsih mptbase libata e1000 serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]
CPU: 0 PID: 1100 Comm: chronyd Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8ca5f4bed230 ti: ffff8ca5eabb8000 task.ti: ffff8ca5eabb8000
RIP: 0010:[<ffffffffc00d8877>]  [<ffffffffc00d8877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8ca5eabbb6f8  EFLAGS: 00000286
RAX: ffff985240d43818 RBX: ffff8ca5eda5e300 RCX: 0000000000000076
RDX: 0000000000000076 RSI: ffff8ca3f5e27b40 RDI: ffff8ca5f4c16000
RBP: ffff8ca5eabbb778 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: ffffffffa30dd313
R13: ffff8ca5eabbb6e8 R14: ffffffffa3712a40 R15: 0000000000000000
FS:  00007f1a23cb1740(0000) GS:ffff8ca5f5600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4a76819680 CR3: 0000000229a70000 CR4: 00000000001607f0
Call Trace:
[<ffffffffa304d346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa307b0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffffa307b27e>] __qdisc_run+0x8e/0x360
[<ffffffffa30502c9>] __dev_queue_xmit+0x4a9/0x650
[<ffffffffa3050480>] dev_queue_xmit+0x10/0x20
[<ffffffffa309e97b>] ip_finish_output+0x67b/0x8d0
[<ffffffffa309eefb>] ip_output+0x7b/0xf0
[<ffffffffa309e300>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffa309c927>] ip_local_out_sk+0x37/0x40
[<ffffffffa309f966>] ip_send_skb+0x16/0x50
[<ffffffffa30c7ccc>] udp_send_skb+0xac/0x2b0
[<ffffffffa309c660>] ? ip_copy_metadata+0x170/0x170
[<ffffffffa30c9231>] udp_sendmsg+0x5e1/0xa70
[<ffffffffa2c96f4e>] ? ep_poll_callback+0xee/0x210
[<ffffffffa2ad24d2>] ? __wake_up_common+0x82/0x120
[<ffffffffa30d6999>] inet_sendmsg+0x69/0xb0
[<ffffffffa302d3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffa310e73b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffa302e269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffa3095257>] ? __ip_route_output_key_hash+0x207/0x7b0
[<ffffffffa30c56df>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffa31819ce>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa30c87af>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffa31819ce>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa3032a10>] ? release_sock+0x120/0x170
[<ffffffffa30c5b47>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffa30d5624>] ? inet_dgram_connect+0x34/0x90
[<ffffffffa302d96d>] ? SYSC_connect+0xed/0x130
[<ffffffffa302f921>] __sys_sendmsg+0x51/0x90
[<ffffffffa302f972>] SyS_sendmsg+0x12/0x20
[<ffffffffa318cede>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00
Complete report #326
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0418857
Build id: 3.10.0-957.el7.x86_64 		0x7f7
2
? kfree_skbmem
vmlinux 0xffffffff85c21697
Build id: 3.10.0-957.el7.x86_64 		0x37
3
dev_hard_start_xmit
vmlinux 0xffffffff85c38c66
Build id: 3.10.0-957.el7.x86_64 		0x246
4
sch_direct_xmit
vmlinux 0xffffffff85c65cea
Build id: 3.10.0-957.el7.x86_64 		0x11a
5
__dev_queue_xmit
vmlinux 0xffffffff85c3bbe1
Build id: 3.10.0-957.el7.x86_64 		0x491
6
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc08f72ce
Build id: 3.10.0-957.el7.x86_64 		0x4e
7
dev_queue_xmit
vmlinux 0xffffffff85c3bdb0
Build id: 3.10.0-957.el7.x86_64 		0x10
8
ip_finish_output
vmlinux 0xffffffff85c88736
Build id: 3.10.0-957.el7.x86_64 		0x556
9
ip_output
vmlinux 0xffffffff85c88c93
Build id: 3.10.0-957.el7.x86_64 		0x73
10
? __ip_append_data.isra.51
vmlinux 0xffffffff85c881e0
Build id: 3.10.0-957.el7.x86_64 		0xa50
11
ip_local_out_sk
vmlinux 0xffffffff85c86817
Build id: 3.10.0-957.el7.x86_64 		0x37
12
ip_send_skb
vmlinux 0xffffffff85c896f6
Build id: 3.10.0-957.el7.x86_64 		0x16
13
udp_send_skb
vmlinux 0xffffffff85cb150c
Build id: 3.10.0-957.el7.x86_64 		0xac
14
? ip_copy_metadata
vmlinux 0xffffffff85c86560
Build id: 3.10.0-957.el7.x86_64 		0x170
15
udp_sendmsg
vmlinux 0xffffffff85cb27f5
Build id: 3.10.0-957.el7.x86_64 		0x335
16
? __pollwait
vmlinux 0xffffffff85856d70
Build id: 3.10.0-957.el7.x86_64 		0xf0
17
? __pollwait
vmlinux 0xffffffff85856d70
Build id: 3.10.0-957.el7.x86_64 		0xf0
18
inet_sendmsg
vmlinux 0xffffffff85cbfeb9
Build id: 3.10.0-957.el7.x86_64 		0x69
19
? __pollwait
vmlinux 0xffffffff85856d70
Build id: 3.10.0-957.el7.x86_64 		0xf0
20
sock_sendmsg
vmlinux 0xffffffff85c193a6
Build id: 3.10.0-957.el7.x86_64 		0xb6
21
? unix_dgram_sendmsg
vmlinux 0xffffffff85cf769b
Build id: 3.10.0-957.el7.x86_64 		0x3cb
22
___sys_sendmsg
vmlinux 0xffffffff85c1a269
Build id: 3.10.0-957.el7.x86_64 		0x3e9
23
? ip4_datagram_release_cb
vmlinux 0xffffffff85caef8f
Build id: 3.10.0-957.el7.x86_64 		0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff85d6a29e
Build id: 3.10.0-957.el7.x86_64 		0x1e
25
? udp_lib_rehash
vmlinux 0xffffffff85cb1e0f
Build id: 3.10.0-957.el7.x86_64 		0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff85d6a29e
Build id: 3.10.0-957.el7.x86_64 		0x1e
27
? release_sock
vmlinux 0xffffffff85c1e980
Build id: 3.10.0-957.el7.x86_64 		0x120
28
? ip4_datagram_connect
vmlinux 0xffffffff85caf3f7
Build id: 3.10.0-957.el7.x86_64 		0x37
29
? inet_dgram_connect
vmlinux 0xffffffff85cbeb44
Build id: 3.10.0-957.el7.x86_64 		0x34
30
? SYSC_connect
vmlinux 0xffffffff85c1996d
Build id: 3.10.0-957.el7.x86_64 		0xed
31
__sys_sendmsg
vmlinux 0xffffffff85c1b921
Build id: 3.10.0-957.el7.x86_64 		0x51
32
SyS_sendmsg
vmlinux 0xffffffff85c1b972
Build id: 3.10.0-957.el7.x86_64 		0x12
33
system_call_fastpath
vmlinux 0xffffffff85d74ddb
Build id: 3.10.0-957.el7.x86_64 		0x22 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [chronyd:8863]
Modules linked in: tcp_lp nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vmw_vmci vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore joydev pcspkr sg parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_pclmul crct10dif_common crc32c_intel ata_generic
pata_acpi serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi scsi_transport_spi e1000 libata mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 8863 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8e34701fb0c0 ti: ffff8e3473030000 task.ti: ffff8e3473030000
RIP: 0010:[<ffffffffc0418857>]  [<ffffffffc0418857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8e3473033760  EFLAGS: 00000286
RAX: ffffb9dec0f83818 RBX: ffffffff85c22bf3 RCX: 0000000000000090
RDX: 0000000000000090 RSI: ffff8e346f6ffcc0 RDI: ffff8e3475e24000
RBP: ffff8e34730337e0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffffffffc08d1915 R15: ffff8e34730336f8
FS:  00007f6a313ec740(0000) GS:ffff8e3478640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f218d2a1000 CR3: 0000000170c0a000 CR4: 00000000003607e0
Call Trace:
[<ffffffff85c21697>] ? kfree_skbmem+0x37/0x90
[<ffffffff85c38c66>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff85c65cea>] sch_direct_xmit+0x11a/0x250
[<ffffffff85c3bbe1>] __dev_queue_xmit+0x491/0x650
[<ffffffffc08f72ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffff85c3bdb0>] dev_queue_xmit+0x10/0x20
[<ffffffff85c88736>] ip_finish_output+0x556/0x7b0
[<ffffffff85c88c93>] ip_output+0x73/0xe0
[<ffffffff85c881e0>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffff85c86817>] ip_local_out_sk+0x37/0x40
[<ffffffff85c896f6>] ip_send_skb+0x16/0x50
[<ffffffff85cb150c>] udp_send_skb+0xac/0x2b0
[<ffffffff85c86560>] ? ip_copy_metadata+0x170/0x170
[<ffffffff85cb27f5>] udp_sendmsg+0x335/0xa10
[<ffffffff85856d70>] ? __pollwait+0xf0/0xf0
[<ffffffff85856d70>] ? __pollwait+0xf0/0xf0
[<ffffffff85cbfeb9>] inet_sendmsg+0x69/0xb0
[<ffffffff85856d70>] ? __pollwait+0xf0/0xf0
[<ffffffff85c193a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff85cf769b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffff85c1a269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff85caef8f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff85d6a29e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff85cb1e0f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff85d6a29e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff85c1e980>] ? release_sock+0x120/0x170
[<ffffffff85caf3f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff85cbeb44>] ? inet_dgram_connect+0x34/0x90
[<ffffffff85c1996d>] ? SYSC_connect+0xed/0x130
[<ffffffff85c1b921>] __sys_sendmsg+0x51/0x90
[<ffffffff85c1b972>] SyS_sendmsg+0x12/0x20
[<ffffffff85d74ddb>] system_call_fastpath+0x22/0x27
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00