Info

Permalink
Function
e1000_xmit_frame
First occurrence
2020-09-22
Last occurrence
2024-05-11
State
NEW
Components
kernel
Quality
-73

Statistics

Operating system Count
CentOS 7 166
Architecture Count
x86_64 166
Related packages Count
kernel 166
 0:3.10.0-1160.el7 55
 0:3.10.0-1062.el7 18
 0:3.10.0-957.el7 17
 0:3.10.0-1160.41.1.el7 13
 0:3.10.0-1160.102.1.el7 5
 0:3.10.0-957.12.1.el7 4
 0:3.10.0-1062.9.1.el7 4
 0:3.10.0-957.5.1.el7 3
 0:3.10.0-1127.el7 3
 0:3.10.0-1160.90.1.el7 3
 0:3.10.0-1160.118.1.el7 3
 0:3.10.0-1160.66.1.el7 3
 0:3.10.0-1160.92.1.el7 3
 0:3.10.0-957.1.3.el7 2
 0:3.10.0-1127.19.1.el7 2
 0:3.10.0-1160.105.1.el7 2
 0:3.10.0-1160.108.1.el7 2
 0:3.10.0-1160.62.1.el7 2
 0:3.10.0-1160.11.1.el7 2
 0:3.10.0-1160.25.1.el7 2
 0:3.10.0-693.el7 2
 0:3.10.0-862.el7 2
 0:3.10.0-1127.10.1.el7 2
 0:3.10.0-1062.1.2.el7 1
 0:3.10.0-957.10.1.el7 1
 0:3.10.0-957.27.2.el7 1
 0:3.10.0-1160.15.2.el7 1
 0:3.10.0-1160.59.1.el7 1
 0:3.10.0-1160.88.1.el7 1
 0:3.10.0-1160.45.1.el7 1
 0:3.10.0-1160.71.1.el7 1
 0:3.10.0-1160.80.1.el7 1
 0:3.10.0-1062.18.1.el7 1
 0:3.10.0-1160.119.1.el7 1
 0:3.10.0-957.21.3.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #19383
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc04e3857
Build id: 3.10.0-957.5.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffba838ce6
Build id: 3.10.0-957.5.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffba865d6a
Build id: 3.10.0-957.5.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffba83bc61
Build id: 3.10.0-957.5.1.el7.x86_64
0x491
5
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc08e52ce
Build id: 3.10.0-957.5.1.el7.x86_64
0x4e
6
dev_queue_xmit
vmlinux 0xffffffffba83be30
Build id: 3.10.0-957.5.1.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffffba8887b6
Build id: 3.10.0-957.5.1.el7.x86_64
0x556
8
ip_output
vmlinux 0xffffffffba888d13
Build id: 3.10.0-957.5.1.el7.x86_64
0x73
9
? __ip_append_data.isra.51
vmlinux 0xffffffffba888260
Build id: 3.10.0-957.5.1.el7.x86_64
0xa50
10
ip_local_out_sk
vmlinux 0xffffffffba886897
Build id: 3.10.0-957.5.1.el7.x86_64
0x37
11
ip_send_skb
vmlinux 0xffffffffba889776
Build id: 3.10.0-957.5.1.el7.x86_64
0x16
12
udp_send_skb
vmlinux 0xffffffffba8b158c
Build id: 3.10.0-957.5.1.el7.x86_64
0xac
13
? ip_copy_metadata
vmlinux 0xffffffffba8865e0
Build id: 3.10.0-957.5.1.el7.x86_64
0x170
14
udp_sendmsg
vmlinux 0xffffffffba8b2875
Build id: 3.10.0-957.5.1.el7.x86_64
0x335
15
? __pollwait
vmlinux 0xffffffffba457320
Build id: 3.10.0-957.5.1.el7.x86_64
0xf0
16
? __pollwait
vmlinux 0xffffffffba457320
Build id: 3.10.0-957.5.1.el7.x86_64
0xf0
17
inet_sendmsg
vmlinux 0xffffffffba8bff39
Build id: 3.10.0-957.5.1.el7.x86_64
0x69
18
? __pollwait
vmlinux 0xffffffffba457320
Build id: 3.10.0-957.5.1.el7.x86_64
0xf0
19
sock_sendmsg
vmlinux 0xffffffffba8193a6
Build id: 3.10.0-957.5.1.el7.x86_64
0xb6
20
___sys_sendmsg
vmlinux 0xffffffffba81a269
Build id: 3.10.0-957.5.1.el7.x86_64
0x3e9
21
? ip4_datagram_release_cb
vmlinux 0xffffffffba8af00f
Build id: 3.10.0-957.5.1.el7.x86_64
0x6f
22
? _raw_spin_unlock_bh
vmlinux 0xffffffffba96a31e
Build id: 3.10.0-957.5.1.el7.x86_64
0x1e
23
? udp_lib_rehash
vmlinux 0xffffffffba8b1e8f
Build id: 3.10.0-957.5.1.el7.x86_64
0x11f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffba96a31e
Build id: 3.10.0-957.5.1.el7.x86_64
0x1e
25
? release_sock
vmlinux 0xffffffffba81ea00
Build id: 3.10.0-957.5.1.el7.x86_64
0x120
26
? ip4_datagram_connect
vmlinux 0xffffffffba8af477
Build id: 3.10.0-957.5.1.el7.x86_64
0x37
27
__sys_sendmsg
vmlinux 0xffffffffba81b921
Build id: 3.10.0-957.5.1.el7.x86_64
0x51
28
SyS_sendmsg
vmlinux 0xffffffffba81b972
Build id: 3.10.0-957.5.1.el7.x86_64
0x12
29
system_call_fastpath
vmlinux 0xffffffffba974ddb
Build id: 3.10.0-957.5.1.el7.x86_64
0x22
30
? system_call_after_swapgs
vmlinux 0xffffffffba974d21
Build id: 3.10.0-957.5.1.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 59s! [chronyd:5667]
Modules linked in: fuse xt_CHECKSUM xt_set xt_multiport ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip_set_hash_ip devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc coretemp crc32_pclmul ghash_clmulni_intel aesni_intel ppdev lrw gf128mul glue_helper ablk_helper cryptd snd_seq_midi vmw_balloon snd_seq_midi_event pcspkr snd_ens1371 snd_rawmidi snd_ac97_codec joydev ac97_bus
snd_seq snd_seq_device snd_pcm snd_timer snd soundcore sg floppy parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw pata_acpi nfit vmwgfx libnvdimm e1000 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi mptscsih libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 5667 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8b9bf599d140 ti: ffff8b9c7a92c000 task.ti: ffff8b9c7a92c000
RIP: 0010:[<ffffffffc04e3857>]  [<ffffffffc04e3857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8b9c7a92f760  EFLAGS: 00000286
RAX: ffffb7f640943818 RBX: ffffffffba822c73 RCX: 00000000000000c2
RDX: 00000000000000c2 RSI: ffff8b9bf55ee400 RDI: ffff8b9c7a806000
RBP: ffff8b9c7a92f7e0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffffffffc08ae915 R15: ffff8b9c7a92f6f8
FS:  00007f293b741740(0000) GS:ffff8b9c7b440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006dee1c CR3: 00000000ba92a000 CR4: 00000000000007e0
Call Trace:
[<ffffffffba838ce6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffba865d6a>] sch_direct_xmit+0x11a/0x250
[<ffffffffba83bc61>] __dev_queue_xmit+0x491/0x650
[<ffffffffc08e52ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffffba83be30>] dev_queue_xmit+0x10/0x20
[<ffffffffba8887b6>] ip_finish_output+0x556/0x7b0
[<ffffffffba888d13>] ip_output+0x73/0xe0
[<ffffffffba888260>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffba886897>] ip_local_out_sk+0x37/0x40
[<ffffffffba889776>] ip_send_skb+0x16/0x50
[<ffffffffba8b158c>] udp_send_skb+0xac/0x2b0
[<ffffffffba8865e0>] ? ip_copy_metadata+0x170/0x170
[<ffffffffba8b2875>] udp_sendmsg+0x335/0xa10
[<ffffffffba457320>] ? __pollwait+0xf0/0xf0
[<ffffffffba457320>] ? __pollwait+0xf0/0xf0
[<ffffffffba8bff39>] inet_sendmsg+0x69/0xb0
[<ffffffffba457320>] ? __pollwait+0xf0/0xf0
[<ffffffffba8193a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffba81a269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffba8af00f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffba96a31e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffba8b1e8f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffba96a31e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffba81ea00>] ? release_sock+0x120/0x170
[<ffffffffba8af477>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffba81b921>] __sys_sendmsg+0x51/0x90
[<ffffffffba81b972>] SyS_sendmsg+0x12/0x20
[<ffffffffba974ddb>] system_call_fastpath+0x22/0x27
[<ffffffffba974d21>] ? system_call_after_swapgs+0xae/0x146
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #1320103
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0113857
Build id: 3.10.0-957.5.1.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffb5621717
Build id: 3.10.0-957.5.1.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffb5638ce6
Build id: 3.10.0-957.5.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffffb5665d6a
Build id: 3.10.0-957.5.1.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffb563bc61
Build id: 3.10.0-957.5.1.el7.x86_64
0x491
6
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc06a52ce
Build id: 3.10.0-957.5.1.el7.x86_64
0x4e
7
dev_queue_xmit
vmlinux 0xffffffffb563be30
Build id: 3.10.0-957.5.1.el7.x86_64
0x10
8
ip_finish_output
vmlinux 0xffffffffb56887b6
Build id: 3.10.0-957.5.1.el7.x86_64
0x556
9
ip_output
vmlinux 0xffffffffb5688d13
Build id: 3.10.0-957.5.1.el7.x86_64
0x73
10
? __ip_append_data.isra.51
vmlinux 0xffffffffb5688260
Build id: 3.10.0-957.5.1.el7.x86_64
0xa50
11
ip_local_out_sk
vmlinux 0xffffffffb5686897
Build id: 3.10.0-957.5.1.el7.x86_64
0x37
12
ip_send_skb
vmlinux 0xffffffffb5689776
Build id: 3.10.0-957.5.1.el7.x86_64
0x16
13
udp_send_skb
vmlinux 0xffffffffb56b158c
Build id: 3.10.0-957.5.1.el7.x86_64
0xac
14
? ip_copy_metadata
vmlinux 0xffffffffb56865e0
Build id: 3.10.0-957.5.1.el7.x86_64
0x170
15
udp_sendmsg
vmlinux 0xffffffffb56b2875
Build id: 3.10.0-957.5.1.el7.x86_64
0x335
16
? __pollwait
vmlinux 0xffffffffb5257320
Build id: 3.10.0-957.5.1.el7.x86_64
0xf0
17
? sock_has_perm
vmlinux 0xffffffffb52fc985
Build id: 3.10.0-957.5.1.el7.x86_64
0x75
18
inet_sendmsg
vmlinux 0xffffffffb56bff39
Build id: 3.10.0-957.5.1.el7.x86_64
0x69
19
? selinux_socket_sendmsg
vmlinux 0xffffffffb52fcab3
Build id: 3.10.0-957.5.1.el7.x86_64
0x23
20
sock_sendmsg
vmlinux 0xffffffffb56193a6
Build id: 3.10.0-957.5.1.el7.x86_64
0xb6
21
___sys_sendmsg
vmlinux 0xffffffffb561a269
Build id: 3.10.0-957.5.1.el7.x86_64
0x3e9
22
? ip4_datagram_release_cb
vmlinux 0xffffffffb56af00f
Build id: 3.10.0-957.5.1.el7.x86_64
0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffffb576a31e
Build id: 3.10.0-957.5.1.el7.x86_64
0x1e
24
? udp_lib_rehash
vmlinux 0xffffffffb56b1e8f
Build id: 3.10.0-957.5.1.el7.x86_64
0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffb576a31e
Build id: 3.10.0-957.5.1.el7.x86_64
0x1e
26
? release_sock
vmlinux 0xffffffffb561ea00
Build id: 3.10.0-957.5.1.el7.x86_64
0x120
27
? ip4_datagram_connect
vmlinux 0xffffffffb56af477
Build id: 3.10.0-957.5.1.el7.x86_64
0x37
28
? inet_dgram_connect
vmlinux 0xffffffffb56bebc4
Build id: 3.10.0-957.5.1.el7.x86_64
0x34
29
? SYSC_connect
vmlinux 0xffffffffb561996d
Build id: 3.10.0-957.5.1.el7.x86_64
0xed
30
__sys_sendmsg
vmlinux 0xffffffffb561b921
Build id: 3.10.0-957.5.1.el7.x86_64
0x51
31
SyS_sendmsg
vmlinux 0xffffffffb561b972
Build id: 3.10.0-957.5.1.el7.x86_64
0x12
32
system_call_fastpath
vmlinux 0xffffffffb5774ddb
Build id: 3.10.0-957.5.1.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 39s! [chronyd:9741]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat vmhgfs(OE) ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod ppdev crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr snd_seq_midi snd_seq_midi_event snd_ens1371 btusb snd_rawmidi btrtl btbcm
snd_ac97_codec btintel bluetooth ac97_bus snd_seq snd_seq_device snd_pcm rfkill snd_timer snd sg soundcore parport_pc parport vmw_vmci i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmwgfx ata_generic pata_acpi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm e1000 mptspi ata_piix scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm
CPU: 1 PID: 9741 Comm: chronyd Kdump: loaded Tainted: G           OE  ------------   3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff91a9ed25a080 ti: ffff91a879be4000 task.ti: ffff91a879be4000
RIP: 0010:[<ffffffffc0113857>]  [<ffffffffc0113857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff91a879be7760  EFLAGS: 00010286
RAX: ffff9d05c1043818 RBX: ffffffffb5622c73 RCX: 0000000000000043
RDX: 0000000000000043 RSI: ffff91a9f1a7ff40 RDI: ffff91a9ff89e000
RBP: ffff91a879be77e0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 00000001c066e915 R15: ffffffffb53a0a90
FS:  00007f4ece4b5740(0000) GS:ffff91a9f5640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f78a6aeb008 CR3: 00000000b9ba8000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb5621717>] ? kfree_skbmem+0x37/0x90
[<ffffffffb5638ce6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb5665d6a>] sch_direct_xmit+0x11a/0x250
[<ffffffffb563bc61>] __dev_queue_xmit+0x491/0x650
[<ffffffffc06a52ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffffb563be30>] dev_queue_xmit+0x10/0x20
[<ffffffffb56887b6>] ip_finish_output+0x556/0x7b0
[<ffffffffb5688d13>] ip_output+0x73/0xe0
[<ffffffffb5688260>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffb5686897>] ip_local_out_sk+0x37/0x40
[<ffffffffb5689776>] ip_send_skb+0x16/0x50
[<ffffffffb56b158c>] udp_send_skb+0xac/0x2b0
[<ffffffffb56865e0>] ? ip_copy_metadata+0x170/0x170
[<ffffffffb56b2875>] udp_sendmsg+0x335/0xa10
[<ffffffffb5257320>] ? __pollwait+0xf0/0xf0
[<ffffffffb52fc985>] ? sock_has_perm+0x75/0x90
[<ffffffffb56bff39>] inet_sendmsg+0x69/0xb0
[<ffffffffb52fcab3>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffffb56193a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffb561a269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffb56af00f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffb576a31e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb56b1e8f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffb576a31e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb561ea00>] ? release_sock+0x120/0x170
[<ffffffffb56af477>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffb56bebc4>] ? inet_dgram_connect+0x34/0x90
[<ffffffffb561996d>] ? SYSC_connect+0xed/0x130
[<ffffffffb561b921>] __sys_sendmsg+0x51/0x90
[<ffffffffb561b972>] SyS_sendmsg+0x12/0x20
[<ffffffffb5774ddb>] system_call_fastpath+0x22/0x27
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #1085513
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc0287bcc
Build id: 3.10.0-957.21.3.el7.x86_64
0xaec
2
? nf_nat_setup_info
nf_nat 0xffffffffc09b03b7
Build id: 3.10.0-957.21.3.el7.x86_64
0x2f7
3
dev_hard_start_xmit
vmlinux 0xffffffffb9e39ce6
Build id: 3.10.0-957.21.3.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffffb9e66d6a
Build id: 3.10.0-957.21.3.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffb9e3cc61
Build id: 3.10.0-957.21.3.el7.x86_64
0x491
6
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc09a12ce
Build id: 3.10.0-957.21.3.el7.x86_64
0x4e
7
dev_queue_xmit
vmlinux 0xffffffffb9e3ce30
Build id: 3.10.0-957.21.3.el7.x86_64
0x10
8
ip_finish_output
vmlinux 0xffffffffb9e897b6
Build id: 3.10.0-957.21.3.el7.x86_64
0x556
9
ip_output
vmlinux 0xffffffffb9e89d13
Build id: 3.10.0-957.21.3.el7.x86_64
0x73
10
? __ip_append_data.isra.51
vmlinux 0xffffffffb9e89260
Build id: 3.10.0-957.21.3.el7.x86_64
0xa50
11
ip_local_out_sk
vmlinux 0xffffffffb9e87897
Build id: 3.10.0-957.21.3.el7.x86_64
0x37
12
ip_send_skb
vmlinux 0xffffffffb9e8a776
Build id: 3.10.0-957.21.3.el7.x86_64
0x16
13
udp_send_skb
vmlinux 0xffffffffb9eb26ac
Build id: 3.10.0-957.21.3.el7.x86_64
0xac
14
? ip_copy_metadata
vmlinux 0xffffffffb9e875e0
Build id: 3.10.0-957.21.3.el7.x86_64
0x170
15
udp_sendmsg
vmlinux 0xffffffffb9eb3995
Build id: 3.10.0-957.21.3.el7.x86_64
0x335
16
? default_wake_function
vmlinux 0xffffffffb98d6b72
Build id: 3.10.0-957.21.3.el7.x86_64
0x12
17
? __wake_up_common
vmlinux 0xffffffffb98cbc0b
Build id: 3.10.0-957.21.3.el7.x86_64
0x5b
18
? ep_poll_callback
vmlinux 0xffffffffb9a8f53e
Build id: 3.10.0-957.21.3.el7.x86_64
0xee
19
inet_sendmsg
vmlinux 0xffffffffb9ec1199
Build id: 3.10.0-957.21.3.el7.x86_64
0x69
20
? selinux_socket_sendmsg
vmlinux 0xffffffffb9afcd83
Build id: 3.10.0-957.21.3.el7.x86_64
0x23
21
sock_sendmsg
vmlinux 0xffffffffb9e1a3a6
Build id: 3.10.0-957.21.3.el7.x86_64
0xb6
22
? unix_dgram_sendmsg
vmlinux 0xffffffffb9ef8a1b
Build id: 3.10.0-957.21.3.el7.x86_64
0x3cb
23
___sys_sendmsg
vmlinux 0xffffffffb9e1b269
Build id: 3.10.0-957.21.3.el7.x86_64
0x3e9
24
? ip4_datagram_release_cb
vmlinux 0xffffffffb9eb012f
Build id: 3.10.0-957.21.3.el7.x86_64
0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffb9f6b5fe
Build id: 3.10.0-957.21.3.el7.x86_64
0x1e
26
? udp_lib_rehash
vmlinux 0xffffffffb9eb2faf
Build id: 3.10.0-957.21.3.el7.x86_64
0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffb9f6b5fe
Build id: 3.10.0-957.21.3.el7.x86_64
0x1e
28
? release_sock
vmlinux 0xffffffffb9e1fa00
Build id: 3.10.0-957.21.3.el7.x86_64
0x120
29
? ip4_datagram_connect
vmlinux 0xffffffffb9eb0597
Build id: 3.10.0-957.21.3.el7.x86_64
0x37
30
? inet_dgram_connect
vmlinux 0xffffffffb9ebfe24
Build id: 3.10.0-957.21.3.el7.x86_64
0x34
31
? SYSC_connect
vmlinux 0xffffffffb9e1a96d
Build id: 3.10.0-957.21.3.el7.x86_64
0xed
32
__sys_sendmsg
vmlinux 0xffffffffb9e1c921
Build id: 3.10.0-957.21.3.el7.x86_64
0x51
33
SyS_sendmsg
vmlinux 0xffffffffb9e1c972
Build id: 3.10.0-957.21.3.el7.x86_64
0x12
34
system_call_fastpath
vmlinux 0xffffffffb9f75ddb
Build id: 3.10.0-957.21.3.el7.x86_64
0x22
35
? system_call_after_swapgs
vmlinux 0xffffffffb9f75d21
Build id: 3.10.0-957.21.3.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 38s! [iidbms:19431]
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 38s! [chronyd:6790]
Modules linked in: arc4 md4 xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter rpcsec_gss_krb5 nfsv4 nfs fscache nls_utf8 cifs ccm dns_resolver vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp crc32_pclmul ext4 ghash_clmulni_intel mbcache jbd2 aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr parport_pc parport sg vmw_vmci i2c_piix4 nfsd nfs_acl lockd grace auth_rpcgss binfmt_misc sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm vmxnet3 mptspi scsi_transport_spi mptscsih ata_piix mptbase libata drm_panel_orientation_quirks involflt(OE) dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 6790 Comm: chronyd Kdump: loaded Tainted: G           OEL ------------   3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8b32b2ea0000 ti: ffff8b39b4270000 task.ti: ffff8b39b4270000
RIP: 0010:[<ffffffffc0287bcc>]  [<ffffffffc0287bcc>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0018:ffff8b39b4273750  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffff8b39b42737c8 RCX: 00000000000000b5
RDX: ffff9c414313a610 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff8b39b42737e0 R08: 0000000000000000 R09: 0000000000a0405a
R10: 0000000000000000 R11: ffff8b39ad366b40 R12: ffffffffc097a8bd
R13: ffff8b39b42736f8 R14: ffffffffb98a04f7 R15: ffff8b39b42736b8
FS:  00007fa5f249c740(0000) GS:ffff8b3a3fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3b53d78000 CR3: 00000007b426c000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc09b03b7>] ? nf_nat_setup_info+0x2f7/0x320 [nf_nat]
[<ffffffffb9e39ce6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb9e66d6a>] sch_direct_xmit+0x11a/0x250
[<ffffffffb9e3cc61>] __dev_queue_xmit+0x491/0x650
[<ffffffffc09a12ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffffb9e3ce30>] dev_queue_xmit+0x10/0x20
[<ffffffffb9e897b6>] ip_finish_output+0x556/0x7b0
[<ffffffffb9e89d13>] ip_output+0x73/0xe0
[<ffffffffb9e89260>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffb9e87897>] ip_local_out_sk+0x37/0x40
[<ffffffffb9e8a776>] ip_send_skb+0x16/0x50
[<ffffffffb9eb26ac>] udp_send_skb+0xac/0x2b0
[<ffffffffb9e875e0>] ? ip_copy_metadata+0x170/0x170
[<ffffffffb9eb3995>] udp_sendmsg+0x335/0xa10
[<ffffffffb98d6b72>] ? default_wake_function+0x12/0x20
[<ffffffffb98cbc0b>] ? __wake_up_common+0x5b/0x90
[<ffffffffb9a8f53e>] ? ep_poll_callback+0xee/0x210
[<ffffffffb9ec1199>] inet_sendmsg+0x69/0xb0
[<ffffffffb9afcd83>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffffb9e1a3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffb9ef8a1b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffb9e1b269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffb9eb012f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffb9f6b5fe>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb9eb2faf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffb9f6b5fe>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb9e1fa00>] ? release_sock+0x120/0x170
[<ffffffffb9eb0597>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffb9ebfe24>] ? inet_dgram_connect+0x34/0x90
[<ffffffffb9e1a96d>] ? SYSC_connect+0xed/0x130
[<ffffffffb9e1c921>] __sys_sendmsg+0x51/0x90
[<ffffffffb9e1c972>] SyS_sendmsg+0x12/0x20
[<ffffffffb9f75ddb>] system_call_fastpath+0x22/0x27
[<ffffffffb9f75d21>] ? system_call_after_swapgs+0xae/0x146
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41 
Complete report #986965
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc03e43dc
Build id: 3.10.0-1160.el7.x86_64
0xaec
2
dev_hard_start_xmit
vmlinux 0xffffffff84854386
Build id: 3.10.0-1160.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff8488242a
Build id: 3.10.0-1160.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff84857301
Build id: 3.10.0-1160.el7.x86_64
0x491
5
? local_bh_enable
vmlinux 0xffffffff842a4017
Build id: 3.10.0-1160.el7.x86_64
0x17
6
dev_queue_xmit
vmlinux 0xffffffff848574d0
Build id: 3.10.0-1160.el7.x86_64
0x10
7
neigh_resolve_output
vmlinux 0xffffffff848641fd
Build id: 3.10.0-1160.el7.x86_64
0x11d
8
ip_finish_output
vmlinux 0xffffffff848a505c
Build id: 3.10.0-1160.el7.x86_64
0x27c
9
ip_output
vmlinux 0xffffffff848a59db
Build id: 3.10.0-1160.el7.x86_64
0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffff848a4de0
Build id: 3.10.0-1160.el7.x86_64
0xa60
11
ip_local_out_sk
vmlinux 0xffffffff848a3407
Build id: 3.10.0-1160.el7.x86_64
0x37
12
ip_send_skb
vmlinux 0xffffffff848a6446
Build id: 3.10.0-1160.el7.x86_64
0x16
13
udp_send_skb
vmlinux 0xffffffff848ce85c
Build id: 3.10.0-1160.el7.x86_64
0xac
14
? ip_copy_metadata
vmlinux 0xffffffff848a3140
Build id: 3.10.0-1160.el7.x86_64
0x170
15
udp_sendmsg
vmlinux 0xffffffff848cfdc1
Build id: 3.10.0-1160.el7.x86_64
0x5e1
16
? __pollwait
vmlinux 0xffffffff84463e50
Build id: 3.10.0-1160.el7.x86_64
0xf0
17
? sock_has_perm
vmlinux 0xffffffff8450be65
Build id: 3.10.0-1160.el7.x86_64
0x75
18
inet_sendmsg
vmlinux 0xffffffff848dd589
Build id: 3.10.0-1160.el7.x86_64
0x69
19
? selinux_socket_sendmsg
vmlinux 0xffffffff8450bf93
Build id: 3.10.0-1160.el7.x86_64
0x23
20
sock_sendmsg
vmlinux 0xffffffff848343a6
Build id: 3.10.0-1160.el7.x86_64
0xb6
21
___sys_sendmsg
vmlinux 0xffffffff84835269
Build id: 3.10.0-1160.el7.x86_64
0x3e9
22
? __ip_route_output_key_hash
vmlinux 0xffffffff8489bc63
Build id: 3.10.0-1160.el7.x86_64
0x223
23
? ip4_datagram_release_cb
vmlinux 0xffffffff848cc28f
Build id: 3.10.0-1160.el7.x86_64
0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff8498992e
Build id: 3.10.0-1160.el7.x86_64
0x1e
25
? udp_lib_rehash
vmlinux 0xffffffff848cf33f
Build id: 3.10.0-1160.el7.x86_64
0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff8498992e
Build id: 3.10.0-1160.el7.x86_64
0x1e
27
? release_sock
vmlinux 0xffffffff84839a18
Build id: 3.10.0-1160.el7.x86_64
0x128
28
? ip4_datagram_connect
vmlinux 0xffffffff848cc6f7
Build id: 3.10.0-1160.el7.x86_64
0x37
29
? inet_dgram_connect
vmlinux 0xffffffff848dc214
Build id: 3.10.0-1160.el7.x86_64
0x34
30
? SYSC_connect
vmlinux 0xffffffff8483496d
Build id: 3.10.0-1160.el7.x86_64
0xed
31
__sys_sendmsg
vmlinux 0xffffffff84836921
Build id: 3.10.0-1160.el7.x86_64
0x51
32
SyS_sendmsg
vmlinux 0xffffffff84836972
Build id: 3.10.0-1160.el7.x86_64
0x12
33
system_call_fastpath
vmlinux 0xffffffff84993f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 186s! [chronyd:794]
Modules linked in: tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc vfat fat nls_utf8 isofs loop intel_powerclamp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg tpm_crb parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper nfit libnvdimm syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul crct10dif_common
crc32c_intel libata drm serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 794 Comm: chronyd Kdump: loaded Not tainted 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware20,1/440BX Desktop Reference Platform, BIOS VMW201.00V.20648489.B64.2210180829 10/18/2022
task: ffff8ae0ce2a1080 ti: ffff8ae0ce308000 task.ti: ffff8ae0ce308000
RIP: 0010:[<ffffffffc03e43dc>]  [<ffffffffc03e43dc>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0018:ffff8ae0ce30b700  EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffffffffc09869d7 RCX: 000000000000003a
RDX: ffffa8f00031c600 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff8ae0ce30b790 R08: 0000000000000039 R09: 0000000000a0405a
R10: ffff8ae0cdd836b0 R11: ffff8ae1375d8390 R12: ffff8ae0d59a4200
R13: 000106d03fd1acc0 R14: ffffffff8483e13d R15: 000000003fd1acc0
FS:  00007f6da7143740(0000) GS:ffff8ae13fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00001baeb72e95d0 CR3: 000000000e306000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffff84854386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff8488242a>] sch_direct_xmit+0x11a/0x250
[<ffffffff84857301>] __dev_queue_xmit+0x491/0x650
[<ffffffff842a4017>] ? local_bh_enable+0x17/0x20
[<ffffffff848574d0>] dev_queue_xmit+0x10/0x20
[<ffffffff848641fd>] neigh_resolve_output+0x11d/0x220
[<ffffffff848a505c>] ip_finish_output+0x27c/0x8d0
[<ffffffff848a59db>] ip_output+0x7b/0xf0
[<ffffffff848a4de0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff848a3407>] ip_local_out_sk+0x37/0x40
[<ffffffff848a6446>] ip_send_skb+0x16/0x50
[<ffffffff848ce85c>] udp_send_skb+0xac/0x2b0
[<ffffffff848a3140>] ? ip_copy_metadata+0x170/0x170
[<ffffffff848cfdc1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff84463e50>] ? __pollwait+0xf0/0xf0
[<ffffffff8450be65>] ? sock_has_perm+0x75/0x90
[<ffffffff848dd589>] inet_sendmsg+0x69/0xb0
[<ffffffff8450bf93>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff848343a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff84835269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff8489bc63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffff848cc28f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff8498992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff848cf33f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff8498992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff84839a18>] ? release_sock+0x128/0x180
[<ffffffff848cc6f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff848dc214>] ? inet_dgram_connect+0x34/0x90
[<ffffffff8483496d>] ? SYSC_connect+0xed/0x130
[<ffffffff84836921>] __sys_sendmsg+0x51/0x90
[<ffffffff84836972>] SyS_sendmsg+0x12/0x20
[<ffffffff84993f92>] system_call_fastpath+0x25/0x2a
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41 
Complete report #813175
Frame # Function Binary Source or offset Line
1
vmxnet3_xmit_frame
vmxnet3 0xffffffffc023144c
Build id: 3.10.0-1160.92.1.el7.x86_64
0xaec
2
? nf_nat_setup_info
nf_nat 0xffffffffc066d457
Build id: 3.10.0-1160.92.1.el7.x86_64
0x2f7
3
dev_hard_start_xmit
vmlinux 0xffffffff96c7edc6
Build id: 3.10.0-1160.92.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffff96cadd72
Build id: 3.10.0-1160.92.1.el7.x86_64
0x122
5
__dev_queue_xmit
vmlinux 0xffffffff96c81dd1
Build id: 3.10.0-1160.92.1.el7.x86_64
0x491
6
dev_queue_xmit
vmlinux 0xffffffff96c81fa0
Build id: 3.10.0-1160.92.1.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffff96cd260b
Build id: 3.10.0-1160.92.1.el7.x86_64
0x67b
8
ip_output
vmlinux 0xffffffff96cd2b9b
Build id: 3.10.0-1160.92.1.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffff96cd1f90
Build id: 3.10.0-1160.92.1.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffff96cd0587
Build id: 3.10.0-1160.92.1.el7.x86_64
0x37
11
ip_send_skb
vmlinux 0xffffffff96cd3636
Build id: 3.10.0-1160.92.1.el7.x86_64
0x16
12
udp_send_skb
vmlinux 0xffffffff96cfc3ac
Build id: 3.10.0-1160.92.1.el7.x86_64
0xac
13
? ip_copy_metadata
vmlinux 0xffffffff96cd02b0
Build id: 3.10.0-1160.92.1.el7.x86_64
0x170
14
udp_sendmsg
vmlinux 0xffffffff96cfd9a1
Build id: 3.10.0-1160.92.1.el7.x86_64
0x5e1
15
? __pollwait
vmlinux 0xffffffff96872490
Build id: 3.10.0-1160.92.1.el7.x86_64
0xf0
16
? __pollwait
vmlinux 0xffffffff96872490
Build id: 3.10.0-1160.92.1.el7.x86_64
0xf0
17
inet_sendmsg
vmlinux 0xffffffff96d0b4e9
Build id: 3.10.0-1160.92.1.el7.x86_64
0x69
18
? __pollwait
vmlinux 0xffffffff96872490
Build id: 3.10.0-1160.92.1.el7.x86_64
0xf0
19
sock_sendmsg
vmlinux 0xffffffff96c5e4f6
Build id: 3.10.0-1160.92.1.el7.x86_64
0xb6
20
___sys_sendmsg
vmlinux 0xffffffff96c5f3f9
Build id: 3.10.0-1160.92.1.el7.x86_64
0x3e9
21
? __ip_route_output_key_hash
vmlinux 0xffffffff96cc8c63
Build id: 3.10.0-1160.92.1.el7.x86_64
0x223
22
? ip4_datagram_release_cb
vmlinux 0xffffffff96cf9cdf
Build id: 3.10.0-1160.92.1.el7.x86_64
0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffff96dba56e
Build id: 3.10.0-1160.92.1.el7.x86_64
0x1e
24
? udp_lib_rehash
vmlinux 0xffffffff96cfcecf
Build id: 3.10.0-1160.92.1.el7.x86_64
0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffff96dba56e
Build id: 3.10.0-1160.92.1.el7.x86_64
0x1e
26
? release_sock
vmlinux 0xffffffff96c63ea8
Build id: 3.10.0-1160.92.1.el7.x86_64
0x128
27
? ip4_datagram_connect
vmlinux 0xffffffff96cfa147
Build id: 3.10.0-1160.92.1.el7.x86_64
0x37
28
? inet_dgram_connect
vmlinux 0xffffffff96d0a0e4
Build id: 3.10.0-1160.92.1.el7.x86_64
0x34
29
__sys_sendmsg
vmlinux 0xffffffff96c60b41
Build id: 3.10.0-1160.92.1.el7.x86_64
0x51
30
SyS_sendmsg
vmlinux 0xffffffff96c60b92
Build id: 3.10.0-1160.92.1.el7.x86_64
0x12
31
system_call_fastpath
vmlinux 0xffffffff96dc539a
Build id: 3.10.0-1160.92.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 27s! [chronyd:713]
Modules linked in: binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon sg joydev pcspkr parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea crc_t10dif sysfillrect sysimgblt
fb_sys_fops crct10dif_generic ttm crct10dif_pclmul drm crct10dif_common crc32c_intel ata_piix nfit libata serio_raw libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 713 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.92.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8f2734c3b180 ti: ffff8f27362b8000 task.ti: ffff8f27362b8000
RIP: 0010:[<ffffffffc023144c>]  [<ffffffffc023144c>] vmxnet3_xmit_frame+0xaec/0xfd0 [vmxnet3]
RSP: 0018:ffff8f27362bb740  EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffff8f27362bb7b8 RCX: 00000000000000d6
RDX: ffffb169c035c600 RSI: 0000000000000287 RDI: 0000000000000287
RBP: ffff8f27362bb7d0 R08: 0000000000000000 R09: 0000000000a0005a
R10: 0000000000000000 R11: ffff8f27366f0d50 R12: ffffffffc063f99d
R13: ffff8f27362bb6e8 R14: ffffffff966a89b7 R15: ffff8f27362bb6a8
FS:  00007fb97843f740(0000) GS:ffff8f273b600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006e9360 CR3: 00000000762c0000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc066d457>] ? nf_nat_setup_info+0x2f7/0x360 [nf_nat]
[<ffffffff96c7edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff96cadd72>] sch_direct_xmit+0x122/0x250
[<ffffffff96c81dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffff96c81fa0>] dev_queue_xmit+0x10/0x20
[<ffffffff96cd260b>] ip_finish_output+0x67b/0x8d0
[<ffffffff96cd2b9b>] ip_output+0x7b/0x100
[<ffffffff96cd1f90>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff96cd0587>] ip_local_out_sk+0x37/0x40
[<ffffffff96cd3636>] ip_send_skb+0x16/0x50
[<ffffffff96cfc3ac>] udp_send_skb+0xac/0x2b0
[<ffffffff96cd02b0>] ? ip_copy_metadata+0x170/0x170
[<ffffffff96cfd9a1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff96872490>] ? __pollwait+0xf0/0xf0
[<ffffffff96872490>] ? __pollwait+0xf0/0xf0
[<ffffffff96d0b4e9>] inet_sendmsg+0x69/0xb0
[<ffffffff96872490>] ? __pollwait+0xf0/0xf0
[<ffffffff96c5e4f6>] sock_sendmsg+0xb6/0x100
[<ffffffff96c5f3f9>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff96cc8c63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffff96cf9cdf>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff96dba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff96cfcecf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff96dba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff96c63ea8>] ? release_sock+0x128/0x180
[<ffffffff96cfa147>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff96d0a0e4>] ? inet_dgram_connect+0x34/0x90
[<ffffffff96c60b41>] __sys_sendmsg+0x51/0x90
[<ffffffff96c60b92>] SyS_sendmsg+0x12/0x30
[<ffffffff96dc539a>] system_call_fastpath+0x25/0x2a
Code: ff c7 02 00 00 00 00 41 8b 94 24 e8 00 00 00 48 8b 4d d0 8d 14 d5 00 06 00 00 48 63 d2 48 03 91 b8 23 00 00 41 8b 4c 24 34 89 0a <e9> d0 f7 ff ff 0f 1f 80 00 00 00 00 48 c7 45 c8 00 40 00 00 41 
Complete report #1238564
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0266877
Build id: 3.10.0-1062.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff9a04d346
Build id: 3.10.0-1062.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff9a07b0ba
Build id: 3.10.0-1062.el7.x86_64
0x11a
4
__qdisc_run
vmlinux 0xffffffff9a07b27e
Build id: 3.10.0-1062.el7.x86_64
0x8e
5
__dev_queue_xmit
vmlinux 0xffffffff9a0502c9
Build id: 3.10.0-1062.el7.x86_64
0x4a9
6
dev_queue_xmit
vmlinux 0xffffffff9a050480
Build id: 3.10.0-1062.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffff9a09e97b
Build id: 3.10.0-1062.el7.x86_64
0x67b
8
ip_output
vmlinux 0xffffffff9a09eefb
Build id: 3.10.0-1062.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffff9a09e300
Build id: 3.10.0-1062.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffff9a09c927
Build id: 3.10.0-1062.el7.x86_64
0x37
11
ip_send_skb
vmlinux 0xffffffff9a09f966
Build id: 3.10.0-1062.el7.x86_64
0x16
12
udp_send_skb
vmlinux 0xffffffff9a0c7c6c
Build id: 3.10.0-1062.el7.x86_64
0xac
13
? ip_copy_metadata
vmlinux 0xffffffff9a09c660
Build id: 3.10.0-1062.el7.x86_64
0x170
14
udp_sendmsg
vmlinux 0xffffffff9a0c91d1
Build id: 3.10.0-1062.el7.x86_64
0x5e1
15
? __pollwait
vmlinux 0xffffffff99c5e540
Build id: 3.10.0-1062.el7.x86_64
0xf0
16
? __pollwait
vmlinux 0xffffffff99c5e540
Build id: 3.10.0-1062.el7.x86_64
0xf0
17
inet_sendmsg
vmlinux 0xffffffff9a0d6939
Build id: 3.10.0-1062.el7.x86_64
0x69
18
? __pollwait
vmlinux 0xffffffff99c5e540
Build id: 3.10.0-1062.el7.x86_64
0xf0
19
sock_sendmsg
vmlinux 0xffffffff9a02d3a6
Build id: 3.10.0-1062.el7.x86_64
0xb6
20
? __pollwait
vmlinux 0xffffffff99c5e540
Build id: 3.10.0-1062.el7.x86_64
0xf0
21
___sys_sendmsg
vmlinux 0xffffffff9a02e269
Build id: 3.10.0-1062.el7.x86_64
0x3e9
22
? __ip_route_output_key_hash
vmlinux 0xffffffff9a095257
Build id: 3.10.0-1062.el7.x86_64
0x207
23
? ip4_datagram_release_cb
vmlinux 0xffffffff9a0c567f
Build id: 3.10.0-1062.el7.x86_64
0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff9a18196e
Build id: 3.10.0-1062.el7.x86_64
0x1e
25
? udp_lib_rehash
vmlinux 0xffffffff9a0c874f
Build id: 3.10.0-1062.el7.x86_64
0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff9a18196e
Build id: 3.10.0-1062.el7.x86_64
0x1e
27
? release_sock
vmlinux 0xffffffff9a032a10
Build id: 3.10.0-1062.el7.x86_64
0x120
28
? ip4_datagram_connect
vmlinux 0xffffffff9a0c5ae7
Build id: 3.10.0-1062.el7.x86_64
0x37
29
? inet_dgram_connect
vmlinux 0xffffffff9a0d55c4
Build id: 3.10.0-1062.el7.x86_64
0x34
30
? SYSC_connect
vmlinux 0xffffffff9a02d96d
Build id: 3.10.0-1062.el7.x86_64
0xed
31
__sys_sendmsg
vmlinux 0xffffffff9a02f921
Build id: 3.10.0-1062.el7.x86_64
0x51
32
SyS_sendmsg
vmlinux 0xffffffff9a02f972
Build id: 3.10.0-1062.el7.x86_64
0x12
33
system_call_fastpath
vmlinux 0xffffffff9a18bede
Build id: 3.10.0-1062.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [chronyd:1502]
Modules linked in: xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype nf_conntrack_netlink iptable_filter tcp_lp veth fuse xt_nat br_netfilter overlay(T) tun bridge stp llc devlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper pcspkr joydev cryptd sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul drm crct10dif_common nfit crc32c_intel libata libnvdimm mptspi e1000 scsi_transport_spi serio_raw mptscsih
mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: iptable_filter]
CPU: 2 PID: 1502 Comm: chronyd Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9b19a80ed230 ti: ffff9b19aeed4000 task.ti: ffff9b19aeed4000
RIP: 0010:[<ffffffffc0266877>]  [<ffffffffc0266877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9b19aeed76f8  EFLAGS: 00000286
RAX: ffffa65783203818 RBX: 000000000000004a RCX: 000000000000001b
RDX: 000000000000001b RSI: ffff9b19ab59f600 RDI: ffff9b19aee76000
RBP: ffff9b19aeed7778 R08: 0000000000000000 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: 000000045e3998c6
R13: 00000001aeed76e8 R14: ffffffff99daabc0 R15: ffff9b19aeed76b0
FS:  00007f568e524740(0000) GS:ffff9b19af880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fffa6ac4030 CR3: 00000005656a4000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9a04d346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff9a07b0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffff9a07b27e>] __qdisc_run+0x8e/0x360
[<ffffffff9a0502c9>] __dev_queue_xmit+0x4a9/0x650
[<ffffffff9a050480>] dev_queue_xmit+0x10/0x20
[<ffffffff9a09e97b>] ip_finish_output+0x67b/0x8d0
[<ffffffff9a09eefb>] ip_output+0x7b/0xf0
[<ffffffff9a09e300>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff9a09c927>] ip_local_out_sk+0x37/0x40
[<ffffffff9a09f966>] ip_send_skb+0x16/0x50
[<ffffffff9a0c7c6c>] udp_send_skb+0xac/0x2b0
[<ffffffff9a09c660>] ? ip_copy_metadata+0x170/0x170
[<ffffffff9a0c91d1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff99c5e540>] ? __pollwait+0xf0/0xf0
[<ffffffff99c5e540>] ? __pollwait+0xf0/0xf0
[<ffffffff9a0d6939>] inet_sendmsg+0x69/0xb0
[<ffffffff99c5e540>] ? __pollwait+0xf0/0xf0
[<ffffffff9a02d3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff99c5e540>] ? __pollwait+0xf0/0xf0
[<ffffffff9a02e269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff9a095257>] ? __ip_route_output_key_hash+0x207/0x7b0
[<ffffffff9a0c567f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff9a18196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff9a0c874f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff9a18196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff9a032a10>] ? release_sock+0x120/0x170
[<ffffffff9a0c5ae7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff9a0d55c4>] ? inet_dgram_connect+0x34/0x90
[<ffffffff9a02d96d>] ? SYSC_connect+0xed/0x130
[<ffffffff9a02f921>] __sys_sendmsg+0x51/0x90
[<ffffffff9a02f972>] SyS_sendmsg+0x12/0x20
[<ffffffff9a18bede>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #1112492
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc011b877
Build id: 3.10.0-1062.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffb604d346
Build id: 3.10.0-1062.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffb607b0ba
Build id: 3.10.0-1062.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffb60502b1
Build id: 3.10.0-1062.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffffb6050480
Build id: 3.10.0-1062.el7.x86_64
0x10
6
neigh_resolve_output
vmlinux 0xffffffffb605cfed
Build id: 3.10.0-1062.el7.x86_64
0x11d
7
ip_finish_output
vmlinux 0xffffffffb609e57c
Build id: 3.10.0-1062.el7.x86_64
0x27c
8
ip_output
vmlinux 0xffffffffb609eefb
Build id: 3.10.0-1062.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffffb609e300
Build id: 3.10.0-1062.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffffb609c927
Build id: 3.10.0-1062.el7.x86_64
0x37
11
ip_send_skb
vmlinux 0xffffffffb609f966
Build id: 3.10.0-1062.el7.x86_64
0x16
12
udp_send_skb
vmlinux 0xffffffffb60c7c6c
Build id: 3.10.0-1062.el7.x86_64
0xac
13
? ip_copy_metadata
vmlinux 0xffffffffb609c660
Build id: 3.10.0-1062.el7.x86_64
0x170
14
udp_sendmsg
vmlinux 0xffffffffb60c91d1
Build id: 3.10.0-1062.el7.x86_64
0x5e1
15
? __alloc_skb
vmlinux 0xffffffffb603716d
Build id: 3.10.0-1062.el7.x86_64
0x8d
16
? ep_poll_callback
vmlinux 0xffffffffb5c96f4e
Build id: 3.10.0-1062.el7.x86_64
0xee
17
? __alloc_skb
vmlinux 0xffffffffb6037181
Build id: 3.10.0-1062.el7.x86_64
0xa1
18
? __wake_up_common
vmlinux 0xffffffffb5ad24d2
Build id: 3.10.0-1062.el7.x86_64
0x82
19
inet_sendmsg
vmlinux 0xffffffffb60d6939
Build id: 3.10.0-1062.el7.x86_64
0x69
20
sock_sendmsg
vmlinux 0xffffffffb602d3a6
Build id: 3.10.0-1062.el7.x86_64
0xb6
21
? unix_dgram_sendmsg
vmlinux 0xffffffffb610e6db
Build id: 3.10.0-1062.el7.x86_64
0x3cb
22
___sys_sendmsg
vmlinux 0xffffffffb602e269
Build id: 3.10.0-1062.el7.x86_64
0x3e9
23
? __ip_route_output_key_hash
vmlinux 0xffffffffb6095257
Build id: 3.10.0-1062.el7.x86_64
0x207
24
? ip4_datagram_release_cb
vmlinux 0xffffffffb60c567f
Build id: 3.10.0-1062.el7.x86_64
0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffb618196e
Build id: 3.10.0-1062.el7.x86_64
0x1e
26
? udp_lib_rehash
vmlinux 0xffffffffb60c874f
Build id: 3.10.0-1062.el7.x86_64
0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffb618196e
Build id: 3.10.0-1062.el7.x86_64
0x1e
28
? release_sock
vmlinux 0xffffffffb6032a10
Build id: 3.10.0-1062.el7.x86_64
0x120
29
? ip4_datagram_connect
vmlinux 0xffffffffb60c5ae7
Build id: 3.10.0-1062.el7.x86_64
0x37
30
? inet_dgram_connect
vmlinux 0xffffffffb60d55c4
Build id: 3.10.0-1062.el7.x86_64
0x34
31
? SYSC_connect
vmlinux 0xffffffffb602d96d
Build id: 3.10.0-1062.el7.x86_64
0xed
32
__sys_sendmsg
vmlinux 0xffffffffb602f921
Build id: 3.10.0-1062.el7.x86_64
0x51
33
SyS_sendmsg
vmlinux 0xffffffffb602f972
Build id: 3.10.0-1062.el7.x86_64
0x12
34
system_call_fastpath
vmlinux 0xffffffffb618bede
Build id: 3.10.0-1062.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 26s! [chronyd:1365]
Modules linked in: nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev pcspkr parport_pc parport snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper nfit syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libnvdimm mptspi crct10dif_pclmul crct10dif_common libata scsi_transport_spi mptscsih crc32c_intel e1000 mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 1365 Comm: chronyd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff93a768739070 ti: ffff93a7692fc000 task.ti: ffff93a7692fc000
RIP: 0010:[<ffffffffc011b877>]  [<ffffffffc011b877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff93a7692ff710  EFLAGS: 00000286
RAX: ffffab7881cc3818 RBX: ffffffffb6035e63 RCX: 0000000000000094
RDX: 0000000000000094 RSI: ffff93a769029bc0 RDI: ffff93a666bfe000
RBP: ffff93a7692ff790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 00000001669d8400 R15: ffffffffb5daabc0
FS:  00007f1aee32f740(0000) GS:ffff93a76d780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f33230b1f13 CR3: 00000003254fa000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffb604d346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb607b0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffffb60502b1>] __dev_queue_xmit+0x491/0x650
[<ffffffffb6050480>] dev_queue_xmit+0x10/0x20
[<ffffffffb605cfed>] neigh_resolve_output+0x11d/0x220
[<ffffffffb609e57c>] ip_finish_output+0x27c/0x8d0
[<ffffffffb609eefb>] ip_output+0x7b/0xf0
[<ffffffffb609e300>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffb609c927>] ip_local_out_sk+0x37/0x40
[<ffffffffb609f966>] ip_send_skb+0x16/0x50
[<ffffffffb60c7c6c>] udp_send_skb+0xac/0x2b0
[<ffffffffb609c660>] ? ip_copy_metadata+0x170/0x170
[<ffffffffb60c91d1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffb603716d>] ? __alloc_skb+0x8d/0x2d0
[<ffffffffb5c96f4e>] ? ep_poll_callback+0xee/0x210
[<ffffffffb6037181>] ? __alloc_skb+0xa1/0x2d0
[<ffffffffb5ad24d2>] ? __wake_up_common+0x82/0x120
[<ffffffffb60d6939>] inet_sendmsg+0x69/0xb0
[<ffffffffb602d3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffb610e6db>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffb602e269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffb6095257>] ? __ip_route_output_key_hash+0x207/0x7b0
[<ffffffffb60c567f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffb618196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb60c874f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffb618196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb6032a10>] ? release_sock+0x120/0x170
[<ffffffffb60c5ae7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffb60d55c4>] ? inet_dgram_connect+0x34/0x90
[<ffffffffb602d96d>] ? SYSC_connect+0xed/0x130
[<ffffffffb602f921>] __sys_sendmsg+0x51/0x90
[<ffffffffb602f972>] SyS_sendmsg+0x12/0x20
[<ffffffffb618bede>] system_call_fastpath+0x25/0x2a
Complete report #1093212
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00e98e7
Build id: 3.10.0-1160.102.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff92a7edc6
Build id: 3.10.0-1160.102.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff92aadd72
Build id: 3.10.0-1160.102.1.el7.x86_64
0x122
4
__dev_queue_xmit
vmlinux 0xffffffff92a81dd1
Build id: 3.10.0-1160.102.1.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffff92a81fa0
Build id: 3.10.0-1160.102.1.el7.x86_64
0x10
6
neigh_resolve_output
vmlinux 0xffffffff92a8f03d
Build id: 3.10.0-1160.102.1.el7.x86_64
0x11d
7
ip_finish_output
vmlinux 0xffffffff92ad220c
Build id: 3.10.0-1160.102.1.el7.x86_64
0x27c
8
ip_output
vmlinux 0xffffffff92ad2b9b
Build id: 3.10.0-1160.102.1.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffff92ad1f90
Build id: 3.10.0-1160.102.1.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffff92ad0587
Build id: 3.10.0-1160.102.1.el7.x86_64
0x37
11
ip_send_skb
vmlinux 0xffffffff92ad3636
Build id: 3.10.0-1160.102.1.el7.x86_64
0x16
12
udp_send_skb
vmlinux 0xffffffff92afc3ac
Build id: 3.10.0-1160.102.1.el7.x86_64
0xac
13
? ip_copy_metadata
vmlinux 0xffffffff92ad02b0
Build id: 3.10.0-1160.102.1.el7.x86_64
0x170
14
udp_sendmsg
vmlinux 0xffffffff92afd9a1
Build id: 3.10.0-1160.102.1.el7.x86_64
0x5e1
15
? __pollwait
vmlinux 0xffffffff92672500
Build id: 3.10.0-1160.102.1.el7.x86_64
0xf0
16
? __pollwait
vmlinux 0xffffffff92672500
Build id: 3.10.0-1160.102.1.el7.x86_64
0xf0
17
inet_sendmsg
vmlinux 0xffffffff92b0b4e9
Build id: 3.10.0-1160.102.1.el7.x86_64
0x69
18
? __pollwait
vmlinux 0xffffffff92672500
Build id: 3.10.0-1160.102.1.el7.x86_64
0xf0
19
sock_sendmsg
vmlinux 0xffffffff92a5e4f6
Build id: 3.10.0-1160.102.1.el7.x86_64
0xb6
20
___sys_sendmsg
vmlinux 0xffffffff92a5f3f9
Build id: 3.10.0-1160.102.1.el7.x86_64
0x3e9
21
? __ip_route_output_key_hash
vmlinux 0xffffffff92ac8c63
Build id: 3.10.0-1160.102.1.el7.x86_64
0x223
22
? ip4_datagram_release_cb
vmlinux 0xffffffff92af9cdf
Build id: 3.10.0-1160.102.1.el7.x86_64
0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffff92bba56e
Build id: 3.10.0-1160.102.1.el7.x86_64
0x1e
24
? udp_lib_rehash
vmlinux 0xffffffff92afcecf
Build id: 3.10.0-1160.102.1.el7.x86_64
0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffff92bba56e
Build id: 3.10.0-1160.102.1.el7.x86_64
0x1e
26
? release_sock
vmlinux 0xffffffff92a63ea8
Build id: 3.10.0-1160.102.1.el7.x86_64
0x128
27
? ip4_datagram_connect
vmlinux 0xffffffff92afa147
Build id: 3.10.0-1160.102.1.el7.x86_64
0x37
28
? inet_dgram_connect
vmlinux 0xffffffff92b0a0e4
Build id: 3.10.0-1160.102.1.el7.x86_64
0x34
29
__sys_sendmsg
vmlinux 0xffffffff92a60b41
Build id: 3.10.0-1160.102.1.el7.x86_64
0x51
30
SyS_sendmsg
vmlinux 0xffffffff92a60b92
Build id: 3.10.0-1160.102.1.el7.x86_64
0x12
31
system_call_fastpath
vmlinux 0xffffffff92bc539a
Build id: 3.10.0-1160.102.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [chronyd:625]
Modules linked in: nfnetlink_queue nfnetlink_log bluetooth rfkill binfmt_misc isofs loop veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul snd_ens1371 vmw_balloon glue_helper snd_rawmidi snd_ac97_codec ablk_helper cryptd ppdev ac97_bus snd_seq snd_seq_device joydev pcspkr snd_pcm snd_timer snd soundcore sg parport_pc parport
i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit crct10dif_pclmul crct10dif_common crc32c_intel libnvdimm ata_piix mptspi libata e1000 scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 1 PID: 625 Comm: chronyd Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.102.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9469267f4200 ti: ffff94692be24000 task.ti: ffff94692be24000
RIP: 0010:[<ffffffffc00e98e7>]  [<ffffffffc00e98e7>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff94692be27710  EFLAGS: 00000286
RAX: ffffaa7181c03818 RBX: ffffffff92a67453 RCX: 00000000000000e1
RDX: 00000000000000e1 RSI: ffff946928d6b140 RDI: ffff94692c438000
RBP: ffff94692be27790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffff9469202ff500 R15: 0000000100010000
FS:  00007fd2c7246740(0000) GS:ffff94692d640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7139a132e0 CR3: 000000042be68000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff92a7edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff92aadd72>] sch_direct_xmit+0x122/0x250
[<ffffffff92a81dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffff92a81fa0>] dev_queue_xmit+0x10/0x20
[<ffffffff92a8f03d>] neigh_resolve_output+0x11d/0x220
[<ffffffff92ad220c>] ip_finish_output+0x27c/0x8d0
[<ffffffff92ad2b9b>] ip_output+0x7b/0x100
[<ffffffff92ad1f90>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff92ad0587>] ip_local_out_sk+0x37/0x40
[<ffffffff92ad3636>] ip_send_skb+0x16/0x50
[<ffffffff92afc3ac>] udp_send_skb+0xac/0x2b0
[<ffffffff92ad02b0>] ? ip_copy_metadata+0x170/0x170
[<ffffffff92afd9a1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff92672500>] ? __pollwait+0xf0/0xf0
[<ffffffff92672500>] ? __pollwait+0xf0/0xf0
[<ffffffff92b0b4e9>] inet_sendmsg+0x69/0xb0
[<ffffffff92672500>] ? __pollwait+0xf0/0xf0
[<ffffffff92a5e4f6>] sock_sendmsg+0xb6/0x100
[<ffffffff92a5f3f9>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff92ac8c63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffff92af9cdf>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff92bba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff92afcecf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff92bba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffff92a63ea8>] ? release_sock+0x128/0x180
[<ffffffff92afa147>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff92b0a0e4>] ? inet_dgram_connect+0x34/0x90
[<ffffffff92a60b41>] __sys_sendmsg+0x51/0x90
[<ffffffff92a60b92>] SyS_sendmsg+0x12/0x30
[<ffffffff92bc539a>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc 
Complete report #1004891
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc037e8e7
Build id: 3.10.0-1160.119.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffaee7edc6
Build id: 3.10.0-1160.119.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffaeeadd72
Build id: 3.10.0-1160.119.1.el7.x86_64
0x122
4
__dev_queue_xmit
vmlinux 0xffffffffaee81dd1
Build id: 3.10.0-1160.119.1.el7.x86_64
0x491
5
? local_bh_enable
vmlinux 0xffffffffae8a89b7
Build id: 3.10.0-1160.119.1.el7.x86_64
0x17
6
dev_queue_xmit
vmlinux 0xffffffffaee81fa0
Build id: 3.10.0-1160.119.1.el7.x86_64
0x10
7
neigh_resolve_output
vmlinux 0xffffffffaee8f03d
Build id: 3.10.0-1160.119.1.el7.x86_64
0x11d
8
ip_finish_output
vmlinux 0xffffffffaeed220c
Build id: 3.10.0-1160.119.1.el7.x86_64
0x27c
9
ip_output
vmlinux 0xffffffffaeed2b9b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffffaeed1f90
Build id: 3.10.0-1160.119.1.el7.x86_64
0xa60
11
ip_local_out_sk
vmlinux 0xffffffffaeed0587
Build id: 3.10.0-1160.119.1.el7.x86_64
0x37
12
ip_send_skb
vmlinux 0xffffffffaeed3636
Build id: 3.10.0-1160.119.1.el7.x86_64
0x16
13
udp_send_skb
vmlinux 0xffffffffaeefc3ac
Build id: 3.10.0-1160.119.1.el7.x86_64
0xac
14
? ip_copy_metadata
vmlinux 0xffffffffaeed02b0
Build id: 3.10.0-1160.119.1.el7.x86_64
0x170
15
udp_sendmsg
vmlinux 0xffffffffaeefd9a1
Build id: 3.10.0-1160.119.1.el7.x86_64
0x5e1
16
inet_sendmsg
vmlinux 0xffffffffaef0b4e9
Build id: 3.10.0-1160.119.1.el7.x86_64
0x69
17
? radix_tree_lookup_slot
vmlinux 0xffffffffaeba3c02
Build id: 3.10.0-1160.119.1.el7.x86_64
0x22
18
sock_sendmsg
vmlinux 0xffffffffaee5e4f6
Build id: 3.10.0-1160.119.1.el7.x86_64
0xb6
19
? xfs_iunlock
xfs 0xffffffffc0551ed4
Build id: 3.10.0-1160.119.1.el7.x86_64
0x124
20
___sys_sendmsg
vmlinux 0xffffffffaee5f3f9
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3e9
21
? __ip_route_output_key_hash
vmlinux 0xffffffffaeec8c63
Build id: 3.10.0-1160.119.1.el7.x86_64
0x223
22
? ip4_datagram_release_cb
vmlinux 0xffffffffaeef9cdf
Build id: 3.10.0-1160.119.1.el7.x86_64
0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffffaefba56e
Build id: 3.10.0-1160.119.1.el7.x86_64
0x1e
24
? udp_lib_rehash
vmlinux 0xffffffffaeefcecf
Build id: 3.10.0-1160.119.1.el7.x86_64
0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffaefba56e
Build id: 3.10.0-1160.119.1.el7.x86_64
0x1e
26
? release_sock
vmlinux 0xffffffffaee63ea8
Build id: 3.10.0-1160.119.1.el7.x86_64
0x128
27
? ip4_datagram_connect
vmlinux 0xffffffffaeefa147
Build id: 3.10.0-1160.119.1.el7.x86_64
0x37
28
? inet_dgram_connect
vmlinux 0xffffffffaef0a0e4
Build id: 3.10.0-1160.119.1.el7.x86_64
0x34
29
__sys_sendmsg
vmlinux 0xffffffffaee60b41
Build id: 3.10.0-1160.119.1.el7.x86_64
0x51
30
SyS_sendmsg
vmlinux 0xffffffffaee60b92
Build id: 3.10.0-1160.119.1.el7.x86_64
0x12
31
system_call_fastpath
vmlinux 0xffffffffaefc539a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 36s! [chronyd:694]
Modules linked in: xt_conntrack iptable_filter iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack sunrpc kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper pcspkr cryptd sg joydev virtio_balloon i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_blk bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata e1000 crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 694 Comm: chronyd Tainted: G             L ------------   3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.3-0-ga6ed6b701f0a-prebuilt.qemu.org 04/01/2014
task: ffff8c34cc81d280 ti: ffff8c34cba14000 task.ti: ffff8c34cba14000
RIP: 0010:[<ffffffffc037e8e7>]  [<ffffffffc037e8e7>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8c34cba17710  EFLAGS: 00000286
RAX: ffffb06540143818 RBX: ffffffffaee67453 RCX: 000000000000005c
RDX: 000000000000005c RSI: ffff8c34cf1bd4c0 RDI: ffff8c34cbf04000
RBP: ffff8c34cba17790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffff8c34cd801700 R15: ffff8c34cd801700
FS:  00007f7dd3cdc740(0000) GS:ffff8c34cfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7dd2c97f00 CR3: 000000000c8cc000 CR4: 00000000003407f0
Call Trace:
[<ffffffffaee7edc6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffaeeadd72>] sch_direct_xmit+0x122/0x250
[<ffffffffaee81dd1>] __dev_queue_xmit+0x491/0x650
[<ffffffffae8a89b7>] ? local_bh_enable+0x17/0x20
[<ffffffffaee81fa0>] dev_queue_xmit+0x10/0x20
[<ffffffffaee8f03d>] neigh_resolve_output+0x11d/0x220
[<ffffffffaeed220c>] ip_finish_output+0x27c/0x8d0
[<ffffffffaeed2b9b>] ip_output+0x7b/0x100
[<ffffffffaeed1f90>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffaeed0587>] ip_local_out_sk+0x37/0x40
[<ffffffffaeed3636>] ip_send_skb+0x16/0x50
[<ffffffffaeefc3ac>] udp_send_skb+0xac/0x2b0
[<ffffffffaeed02b0>] ? ip_copy_metadata+0x170/0x170
[<ffffffffaeefd9a1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffaef0b4e9>] inet_sendmsg+0x69/0xb0
[<ffffffffaeba3c02>] ? radix_tree_lookup_slot+0x22/0x50
[<ffffffffaee5e4f6>] sock_sendmsg+0xb6/0x100
[<ffffffffc0551ed4>] ? xfs_iunlock+0x124/0x130 [xfs]
[<ffffffffaee5f3f9>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffaeec8c63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffffaeef9cdf>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffaefba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffffaeefcecf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffaefba56e>] ? _raw_spin_unlock_bh+0x1e/0x30
[<ffffffffaee63ea8>] ? release_sock+0x128/0x180
[<ffffffffaeefa147>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffaef0a0e4>] ? inet_dgram_connect+0x34/0x90
[<ffffffffaee60b41>] __sys_sendmsg+0x51/0x90
[<ffffffffaee60b92>] SyS_sendmsg+0x12/0x30
[<ffffffffaefc539a>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc 
Complete report #992687
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000e 0xffffffffc0439e23
Build id: 3.10.0-1127.el7.x86_64
0xc13
2
dev_hard_start_xmit
vmlinux 0xffffffffb0853386
Build id: 3.10.0-1127.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffb088131a
Build id: 3.10.0-1127.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffb0856301
Build id: 3.10.0-1127.el7.x86_64
0x491
5
? local_bh_enable
vmlinux 0xffffffffb02a4b17
Build id: 3.10.0-1127.el7.x86_64
0x17
6
dev_queue_xmit
vmlinux 0xffffffffb08564d0
Build id: 3.10.0-1127.el7.x86_64
0x10
7
neigh_resolve_output
vmlinux 0xffffffffb086312d
Build id: 3.10.0-1127.el7.x86_64
0x11d
8
ip_finish_output
vmlinux 0xffffffffb08a3e9c
Build id: 3.10.0-1127.el7.x86_64
0x27c
9
ip_output
vmlinux 0xffffffffb08a481b
Build id: 3.10.0-1127.el7.x86_64
0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffffb08a3c20
Build id: 3.10.0-1127.el7.x86_64
0xa60
11
ip_local_out_sk
vmlinux 0xffffffffb08a2247
Build id: 3.10.0-1127.el7.x86_64
0x37
12
ip_send_skb
vmlinux 0xffffffffb08a5286
Build id: 3.10.0-1127.el7.x86_64
0x16
13
udp_send_skb
vmlinux 0xffffffffb08cd64c
Build id: 3.10.0-1127.el7.x86_64
0xac
14
? ip_copy_metadata
vmlinux 0xffffffffb08a1f80
Build id: 3.10.0-1127.el7.x86_64
0x170
15
udp_sendmsg
vmlinux 0xffffffffb08cebb1
Build id: 3.10.0-1127.el7.x86_64
0x5e1
16
? __wake_up_common
vmlinux 0xffffffffb02d3c52
Build id: 3.10.0-1127.el7.x86_64
0x82
17
? ep_poll_callback
vmlinux 0xffffffffb049c4a8
Build id: 3.10.0-1127.el7.x86_64
0xf8
18
? __wake_up_common
vmlinux 0xffffffffb02d3c52
Build id: 3.10.0-1127.el7.x86_64
0x82
19
inet_sendmsg
vmlinux 0xffffffffb08dc379
Build id: 3.10.0-1127.el7.x86_64
0x69
20
sock_sendmsg
vmlinux 0xffffffffb08333a6
Build id: 3.10.0-1127.el7.x86_64
0xb6
21
? unix_dgram_sendmsg
vmlinux 0xffffffffb09141fb
Build id: 3.10.0-1127.el7.x86_64
0x3cb
22
___sys_sendmsg
vmlinux 0xffffffffb0834269
Build id: 3.10.0-1127.el7.x86_64
0x3e9
23
? __ip_route_output_key_hash
vmlinux 0xffffffffb089aacf
Build id: 3.10.0-1127.el7.x86_64
0x20f
24
? ip4_datagram_release_cb
vmlinux 0xffffffffb08cb05f
Build id: 3.10.0-1127.el7.x86_64
0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffb098854e
Build id: 3.10.0-1127.el7.x86_64
0x1e
26
? udp_lib_rehash
vmlinux 0xffffffffb08ce12f
Build id: 3.10.0-1127.el7.x86_64
0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffb098854e
Build id: 3.10.0-1127.el7.x86_64
0x1e
28
? release_sock
vmlinux 0xffffffffb0838a18
Build id: 3.10.0-1127.el7.x86_64
0x128
29
? ip4_datagram_connect
vmlinux 0xffffffffb08cb4c7
Build id: 3.10.0-1127.el7.x86_64
0x37
30
? inet_dgram_connect
vmlinux 0xffffffffb08db004
Build id: 3.10.0-1127.el7.x86_64
0x34
31
? SYSC_connect
vmlinux 0xffffffffb083396d
Build id: 3.10.0-1127.el7.x86_64
0xed
32
__sys_sendmsg
vmlinux 0xffffffffb0835921
Build id: 3.10.0-1127.el7.x86_64
0x51
33
SyS_sendmsg
vmlinux 0xffffffffb0835972
Build id: 3.10.0-1127.el7.x86_64
0x12
34
system_call_fastpath
vmlinux 0xffffffffb0992ed2
Build id: 3.10.0-1127.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 38s! [chronyd:1024]
Modules linked in: tcp_lp nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc vfat fat snd_hda_codec_generic ppdev snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep iosf_mbi crc32_pclmul snd_seq snd_seq_device pcspkr vmw_balloon snd_pcm ghash_clmulni_intel joydev cryptd sg snd_timer snd vmw_vmci soundcore i2c_piix4 tpm_crb parport_pc parport sch_fq_codel ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt nfit crct10dif_pclmul
fb_sys_fops crct10dif_common ttm ahci libnvdimm crc32c_intel e1000e mptsas drm libahci ata_piix scsi_transport_sas serio_raw libata mptscsih mptbase ptp pps_core drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1024 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.12343141.B64.1902160724 02/16/2019
task: ffff9a4bf78c0000 ti: ffff9a4bf7200000 task.ti: ffff9a4bf7200000
RIP: 0010:[<ffffffffc0439e23>]  [<ffffffffc0439e23>] e1000_xmit_frame+0xc13/0xfa0 [e1000e]
RSP: 0018:ffff9a4bf7203718  EFLAGS: 00000246
RAX: 00000000000000ad RBX: ffff9a4bf72036b8 RCX: 0000000000000100
RDX: ffffc074c0743818 RSI: 0000000000000013 RDI: ffff9a4b8f40e100
RBP: ffff9a4bf7203790 R08: 0000000000000000 R09: ffff9a4b8d3208c0
R10: 0000000000000000 R11: 0000000027567202 R12: 0000000000000246
R13: ffff9a4bf9001700 R14: ffff9a4bf9001700 R15: 0000000000000020
FS:  00007f92a50df740(0000) GS:ffff9a4bffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7432f60000 CR3: 0000000077206000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb0853386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb088131a>] sch_direct_xmit+0x11a/0x250
[<ffffffffb0856301>] __dev_queue_xmit+0x491/0x650
[<ffffffffb02a4b17>] ? local_bh_enable+0x17/0x20
[<ffffffffb08564d0>] dev_queue_xmit+0x10/0x20
[<ffffffffb086312d>] neigh_resolve_output+0x11d/0x220
[<ffffffffb08a3e9c>] ip_finish_output+0x27c/0x8d0
[<ffffffffb08a481b>] ip_output+0x7b/0xf0
[<ffffffffb08a3c20>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffb08a2247>] ip_local_out_sk+0x37/0x40
[<ffffffffb08a5286>] ip_send_skb+0x16/0x50
[<ffffffffb08cd64c>] udp_send_skb+0xac/0x2b0
[<ffffffffb08a1f80>] ? ip_copy_metadata+0x170/0x170
[<ffffffffb08cebb1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffb02d3c52>] ? __wake_up_common+0x82/0x120
[<ffffffffb049c4a8>] ? ep_poll_callback+0xf8/0x220
[<ffffffffb02d3c52>] ? __wake_up_common+0x82/0x120
[<ffffffffb08dc379>] inet_sendmsg+0x69/0xb0
[<ffffffffb08333a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffb09141fb>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffb0834269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffb089aacf>] ? __ip_route_output_key_hash+0x20f/0x7d0
[<ffffffffb08cb05f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffb098854e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb08ce12f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffb098854e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb0838a18>] ? release_sock+0x128/0x180
[<ffffffffb08cb4c7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffb08db004>] ? inet_dgram_connect+0x34/0x90
[<ffffffffb083396d>] ? SYSC_connect+0xed/0x130
[<ffffffffb0835921>] __sys_sendmsg+0x51/0x90
[<ffffffffb0835972>] SyS_sendmsg+0x12/0x20
[<ffffffffb0992ed2>] system_call_fastpath+0x25/0x2a
Code: 8b 45 c8 48 8b 80 00 03 00 00 f6 80 98 00 00 00 03 74 18 48 8b 45 c8 f6 80 11 37 00 00 08 75 4b 49 8b 57 30 41 0f b7 47 20 89 02 <31> c0 e9 5d f8 ff ff 4c 8b 7d b8 48 8b 5d 88 e9 15 fd ff ff 0f 
Complete report #942971
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc03a7877
Build id: 3.10.0-1127.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffa3853386
Build id: 3.10.0-1127.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffa388131a
Build id: 3.10.0-1127.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffa3856301
Build id: 3.10.0-1127.el7.x86_64
0x491
5
? local_bh_enable
vmlinux 0xffffffffa32a4b17
Build id: 3.10.0-1127.el7.x86_64
0x17
6
dev_queue_xmit
vmlinux 0xffffffffa38564d0
Build id: 3.10.0-1127.el7.x86_64
0x10
7
neigh_resolve_output
vmlinux 0xffffffffa386312d
Build id: 3.10.0-1127.el7.x86_64
0x11d
8
ip_finish_output
vmlinux 0xffffffffa38a3e9c
Build id: 3.10.0-1127.el7.x86_64
0x27c
9
ip_output
vmlinux 0xffffffffa38a481b
Build id: 3.10.0-1127.el7.x86_64
0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffffa38a3c20
Build id: 3.10.0-1127.el7.x86_64
0xa60
11
ip_local_out_sk
vmlinux 0xffffffffa38a2247
Build id: 3.10.0-1127.el7.x86_64
0x37
12
ip_send_skb
vmlinux 0xffffffffa38a5286
Build id: 3.10.0-1127.el7.x86_64
0x16
13
udp_send_skb
vmlinux 0xffffffffa38cd64c
Build id: 3.10.0-1127.el7.x86_64
0xac
14
? ip_copy_metadata
vmlinux 0xffffffffa38a1f80
Build id: 3.10.0-1127.el7.x86_64
0x170
15
udp_sendmsg
vmlinux 0xffffffffa38cebb1
Build id: 3.10.0-1127.el7.x86_64
0x5e1
16
? __pollwait
vmlinux 0xffffffffa3463380
Build id: 3.10.0-1127.el7.x86_64
0xf0
17
? __pollwait
vmlinux 0xffffffffa3463380
Build id: 3.10.0-1127.el7.x86_64
0xf0
18
inet_sendmsg
vmlinux 0xffffffffa38dc379
Build id: 3.10.0-1127.el7.x86_64
0x69
19
? __pollwait
vmlinux 0xffffffffa3463380
Build id: 3.10.0-1127.el7.x86_64
0xf0
20
sock_sendmsg
vmlinux 0xffffffffa38333a6
Build id: 3.10.0-1127.el7.x86_64
0xb6
21
___sys_sendmsg
vmlinux 0xffffffffa3834269
Build id: 3.10.0-1127.el7.x86_64
0x3e9
22
? __ip_route_output_key_hash
vmlinux 0xffffffffa389aacf
Build id: 3.10.0-1127.el7.x86_64
0x20f
23
? ip4_datagram_release_cb
vmlinux 0xffffffffa38cb05f
Build id: 3.10.0-1127.el7.x86_64
0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffa398854e
Build id: 3.10.0-1127.el7.x86_64
0x1e
25
? udp_lib_rehash
vmlinux 0xffffffffa38ce12f
Build id: 3.10.0-1127.el7.x86_64
0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffffa398854e
Build id: 3.10.0-1127.el7.x86_64
0x1e
27
? release_sock
vmlinux 0xffffffffa3838a18
Build id: 3.10.0-1127.el7.x86_64
0x128
28
? ip4_datagram_connect
vmlinux 0xffffffffa38cb4c7
Build id: 3.10.0-1127.el7.x86_64
0x37
29
? inet_dgram_connect
vmlinux 0xffffffffa38db004
Build id: 3.10.0-1127.el7.x86_64
0x34
30
? SYSC_connect
vmlinux 0xffffffffa383396d
Build id: 3.10.0-1127.el7.x86_64
0xed
31
__sys_sendmsg
vmlinux 0xffffffffa3835921
Build id: 3.10.0-1127.el7.x86_64
0x51
32
SyS_sendmsg
vmlinux 0xffffffffa3835972
Build id: 3.10.0-1127.el7.x86_64
0x12
33
system_call_fastpath
vmlinux 0xffffffffa3992ed2
Build id: 3.10.0-1127.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [chronyd:834]
Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm dns_resolver xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables tun xt_conntrack iptable_filter vmw_vsock_vmci_transport vsock xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev snd_timer vmw_balloon snd sg crc32_pclmul soundcore ghash_clmulni_intel aesni_intel pcspkr lrw gf128mul glue_helper ablk_helper cryptd joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx ata_piix drm_kms_helper syscopyarea
sysfillrect sysimgblt libata fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm nfit e1000 serio_raw libnvdimm mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 834 Comm: chronyd Kdump: loaded Tainted: G        W    L ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff911b354f0000 ti: ffff912027c18000 task.ti: ffff912027c18000
RIP: 0010:[<ffffffffc03a7877>]  [<ffffffffc03a7877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff912027c1b710  EFLAGS: 00000286
RAX: ffffae0a83503818 RBX: ffffffffa383be73 RCX: 000000000000007d
RDX: 000000000000007d RSI: ffff912024194bc0 RDI: ffff911c77b54000
RBP: ffff912027c1b790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 000000017fc07700 R15: ffffffffa35b0770
FS:  00007f7a83e7f740(0000) GS:ffff912029640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc08710c2c9 CR3: 00000005227a0000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa3853386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa388131a>] sch_direct_xmit+0x11a/0x250
[<ffffffffa3856301>] __dev_queue_xmit+0x491/0x650
[<ffffffffa32a4b17>] ? local_bh_enable+0x17/0x20
[<ffffffffa38564d0>] dev_queue_xmit+0x10/0x20
[<ffffffffa386312d>] neigh_resolve_output+0x11d/0x220
[<ffffffffa38a3e9c>] ip_finish_output+0x27c/0x8d0
[<ffffffffa38a481b>] ip_output+0x7b/0xf0
[<ffffffffa38a3c20>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffa38a2247>] ip_local_out_sk+0x37/0x40
[<ffffffffa38a5286>] ip_send_skb+0x16/0x50
[<ffffffffa38cd64c>] udp_send_skb+0xac/0x2b0
[<ffffffffa38a1f80>] ? ip_copy_metadata+0x170/0x170
[<ffffffffa38cebb1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffa3463380>] ? __pollwait+0xf0/0xf0
[<ffffffffa3463380>] ? __pollwait+0xf0/0xf0
[<ffffffffa38dc379>] inet_sendmsg+0x69/0xb0
[<ffffffffa3463380>] ? __pollwait+0xf0/0xf0
[<ffffffffa38333a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffa3834269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffa389aacf>] ? __ip_route_output_key_hash+0x20f/0x7d0
[<ffffffffa38cb05f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffa398854e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa38ce12f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffa398854e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa3838a18>] ? release_sock+0x128/0x180
[<ffffffffa38cb4c7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffa38db004>] ? inet_dgram_connect+0x34/0x90
[<ffffffffa383396d>] ? SYSC_connect+0xed/0x130
[<ffffffffa3835921>] __sys_sendmsg+0x51/0x90
[<ffffffffa3835972>] SyS_sendmsg+0x12/0x20
[<ffffffffa3992ed2>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #846920
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc033e857
Build id: 3.10.0-957.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffab438c66
Build id: 3.10.0-957.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffab465cea
Build id: 3.10.0-957.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffab43bbe1
Build id: 3.10.0-957.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffffab43bdb0
Build id: 3.10.0-957.el7.x86_64
0x10
6
neigh_resolve_output
vmlinux 0xffffffffab44861d
Build id: 3.10.0-957.el7.x86_64
0x11d
7
ip_finish_output
vmlinux 0xffffffffab48845c
Build id: 3.10.0-957.el7.x86_64
0x27c
8
ip_output
vmlinux 0xffffffffab488c93
Build id: 3.10.0-957.el7.x86_64
0x73
9
? __ip_append_data.isra.51
vmlinux 0xffffffffab4881e0
Build id: 3.10.0-957.el7.x86_64
0xa50
10
ip_local_out_sk
vmlinux 0xffffffffab486817
Build id: 3.10.0-957.el7.x86_64
0x37
11
ip_send_skb
vmlinux 0xffffffffab4896f6
Build id: 3.10.0-957.el7.x86_64
0x16
12
udp_send_skb
vmlinux 0xffffffffab4b150c
Build id: 3.10.0-957.el7.x86_64
0xac
13
? ip_copy_metadata
vmlinux 0xffffffffab486560
Build id: 3.10.0-957.el7.x86_64
0x170
14
udp_sendmsg
vmlinux 0xffffffffab4b27f5
Build id: 3.10.0-957.el7.x86_64
0x335
15
? default_wake_function
vmlinux 0xffffffffaaed67c2
Build id: 3.10.0-957.el7.x86_64
0x12
16
? __wake_up_common
vmlinux 0xffffffffaaecba9b
Build id: 3.10.0-957.el7.x86_64
0x5b
17
? ep_poll_callback
vmlinux 0xffffffffab08ecde
Build id: 3.10.0-957.el7.x86_64
0xee
18
inet_sendmsg
vmlinux 0xffffffffab4bfeb9
Build id: 3.10.0-957.el7.x86_64
0x69
19
sock_sendmsg
vmlinux 0xffffffffab4193a6
Build id: 3.10.0-957.el7.x86_64
0xb6
20
? unix_dgram_sendmsg
vmlinux 0xffffffffab4f769b
Build id: 3.10.0-957.el7.x86_64
0x3cb
21
___sys_sendmsg
vmlinux 0xffffffffab41a269
Build id: 3.10.0-957.el7.x86_64
0x3e9
22
? ip4_datagram_release_cb
vmlinux 0xffffffffab4aef8f
Build id: 3.10.0-957.el7.x86_64
0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffffab56a29e
Build id: 3.10.0-957.el7.x86_64
0x1e
24
? udp_lib_rehash
vmlinux 0xffffffffab4b1e0f
Build id: 3.10.0-957.el7.x86_64
0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffab56a29e
Build id: 3.10.0-957.el7.x86_64
0x1e
26
? release_sock
vmlinux 0xffffffffab41e980
Build id: 3.10.0-957.el7.x86_64
0x120
27
? ip4_datagram_connect
vmlinux 0xffffffffab4af3f7
Build id: 3.10.0-957.el7.x86_64
0x37
28
? inet_dgram_connect
vmlinux 0xffffffffab4beb44
Build id: 3.10.0-957.el7.x86_64
0x34
29
? SYSC_connect
vmlinux 0xffffffffab41996d
Build id: 3.10.0-957.el7.x86_64
0xed
30
__sys_sendmsg
vmlinux 0xffffffffab41b921
Build id: 3.10.0-957.el7.x86_64
0x51
31
SyS_sendmsg
vmlinux 0xffffffffab41b972
Build id: 3.10.0-957.el7.x86_64
0x12
32
system_call_fastpath
vmlinux 0xffffffffab574ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [chronyd:6225]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables tcp_lp nfnetlink_queue nfnetlink_log bluetooth rfkill fuse tun devlink bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr snd_ens1371 snd_rawmidi
snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore sg parport_pc parport i2c_piix4 vmw_vmci xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw nfit libnvdimm e1000 ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi drm_panel_orientation_quirks libata scsi_transport_spi mptscsih mptbase dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 1 PID: 6225 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9c6bb9fca080 ti: ffff9c6b75928000 task.ti: ffff9c6b75928000
RIP: 0010:[<ffffffffc033e857>]  [<ffffffffc033e857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9c6b7592b720  EFLAGS: 00000286
RAX: ffffc01dc0403818 RBX: ffffffffab422bf3 RCX: 00000000000000c4
RDX: 00000000000000c4 RSI: ffff9c6b74c362c0 RDI: ffff9c6bb989e000
RBP: ffff9c6b7592b7a0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 0000000000000000 R15: 0000000000000000
FS:  00007f4043541740(0000) GS:ffff9c6bbb640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001b15030 CR3: 000000003592c000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffab438c66>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffab465cea>] sch_direct_xmit+0x11a/0x250
[<ffffffffab43bbe1>] __dev_queue_xmit+0x491/0x650
[<ffffffffab43bdb0>] dev_queue_xmit+0x10/0x20
[<ffffffffab44861d>] neigh_resolve_output+0x11d/0x220
[<ffffffffab48845c>] ip_finish_output+0x27c/0x7b0
[<ffffffffab488c93>] ip_output+0x73/0xe0
[<ffffffffab4881e0>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffab486817>] ip_local_out_sk+0x37/0x40
[<ffffffffab4896f6>] ip_send_skb+0x16/0x50
[<ffffffffab4b150c>] udp_send_skb+0xac/0x2b0
[<ffffffffab486560>] ? ip_copy_metadata+0x170/0x170
[<ffffffffab4b27f5>] udp_sendmsg+0x335/0xa10
[<ffffffffaaed67c2>] ? default_wake_function+0x12/0x20
[<ffffffffaaecba9b>] ? __wake_up_common+0x5b/0x90
[<ffffffffab08ecde>] ? ep_poll_callback+0xee/0x210
[<ffffffffab4bfeb9>] inet_sendmsg+0x69/0xb0
[<ffffffffab4193a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffab4f769b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffab41a269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffab4aef8f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffab56a29e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffab4b1e0f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffab56a29e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffab41e980>] ? release_sock+0x120/0x170
[<ffffffffab4af3f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffab4beb44>] ? inet_dgram_connect+0x34/0x90
[<ffffffffab41996d>] ? SYSC_connect+0xed/0x130
[<ffffffffab41b921>] __sys_sendmsg+0x51/0x90
[<ffffffffab41b972>] SyS_sendmsg+0x12/0x20
[<ffffffffab574ddb>] system_call_fastpath+0x22/0x27
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #706286
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc018d877
Build id: 3.10.0-1062.18.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffafa4f346
Build id: 3.10.0-1062.18.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffafa7d0ca
Build id: 3.10.0-1062.18.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffafa522c1
Build id: 3.10.0-1062.18.1.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffffafa52490
Build id: 3.10.0-1062.18.1.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffffafaa099b
Build id: 3.10.0-1062.18.1.el7.x86_64
0x67b
7
ip_output
vmlinux 0xffffffffafaa0f1b
Build id: 3.10.0-1062.18.1.el7.x86_64
0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffffafaa0320
Build id: 3.10.0-1062.18.1.el7.x86_64
0xa60
9
ip_local_out_sk
vmlinux 0xffffffffafa9e947
Build id: 3.10.0-1062.18.1.el7.x86_64
0x37
10
ip_send_skb
vmlinux 0xffffffffafaa1986
Build id: 3.10.0-1062.18.1.el7.x86_64
0x16
11
udp_send_skb
vmlinux 0xffffffffafac9d0c
Build id: 3.10.0-1062.18.1.el7.x86_64
0xac
12
? ip_copy_metadata
vmlinux 0xffffffffafa9e680
Build id: 3.10.0-1062.18.1.el7.x86_64
0x170
13
udp_sendmsg
vmlinux 0xffffffffafacb271
Build id: 3.10.0-1062.18.1.el7.x86_64
0x5e1
14
? ep_poll_callback
vmlinux 0xffffffffaf69934e
Build id: 3.10.0-1062.18.1.el7.x86_64
0xee
15
? _raw_spin_unlock_irqrestore
vmlinux 0xffffffffafb83815
Build id: 3.10.0-1062.18.1.el7.x86_64
0x15
16
inet_sendmsg
vmlinux 0xffffffffafad89c9
Build id: 3.10.0-1062.18.1.el7.x86_64
0x69
17
sock_sendmsg
vmlinux 0xffffffffafa2f3a6
Build id: 3.10.0-1062.18.1.el7.x86_64
0xb6
18
? unix_dgram_sendmsg
vmlinux 0xffffffffafb1077b
Build id: 3.10.0-1062.18.1.el7.x86_64
0x3cb
19
___sys_sendmsg
vmlinux 0xffffffffafa30269
Build id: 3.10.0-1062.18.1.el7.x86_64
0x3e9
20
? __ip_route_output_key_hash
vmlinux 0xffffffffafa97277
Build id: 3.10.0-1062.18.1.el7.x86_64
0x207
21
? ip4_datagram_release_cb
vmlinux 0xffffffffafac771f
Build id: 3.10.0-1062.18.1.el7.x86_64
0x6f
22
? _raw_spin_unlock_bh
vmlinux 0xffffffffafb839be
Build id: 3.10.0-1062.18.1.el7.x86_64
0x1e
23
? udp_lib_rehash
vmlinux 0xffffffffafaca7ef
Build id: 3.10.0-1062.18.1.el7.x86_64
0x11f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffafb839be
Build id: 3.10.0-1062.18.1.el7.x86_64
0x1e
25
? release_sock
vmlinux 0xffffffffafa34a10
Build id: 3.10.0-1062.18.1.el7.x86_64
0x120
26
? ip4_datagram_connect
vmlinux 0xffffffffafac7b87
Build id: 3.10.0-1062.18.1.el7.x86_64
0x37
27
? inet_dgram_connect
vmlinux 0xffffffffafad7654
Build id: 3.10.0-1062.18.1.el7.x86_64
0x34
28
? SYSC_connect
vmlinux 0xffffffffafa2f96d
Build id: 3.10.0-1062.18.1.el7.x86_64
0xed
29
__sys_sendmsg
vmlinux 0xffffffffafa31921
Build id: 3.10.0-1062.18.1.el7.x86_64
0x51
30
SyS_sendmsg
vmlinux 0xffffffffafa31972
Build id: 3.10.0-1062.18.1.el7.x86_64
0x12
31
system_call_fastpath
vmlinux 0xffffffffafb8dede
Build id: 3.10.0-1062.18.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 29s! [chronyd:893]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter binfmt_misc vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata nfit mptspi libnvdimm crct10dif_pclmul crct10dif_common
scsi_transport_spi mptscsih crc32c_intel e1000 mptbase serio_raw drm_panel_orientation_quirks
CPU: 1 PID: 893 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff8e8375cd9070 ti: ffff8e83fb340000 task.ti: ffff8e83fb340000
RIP: 0010:[<ffffffffc018d877>]  [<ffffffffc018d877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8e83fb343750  EFLAGS: 00000286
RAX: ffffb2f800d03818 RBX: ffffffffafa37e63 RCX: 00000000000000e5
RDX: 00000000000000e5 RSI: ffff8e8570db4400 RDI: ffff8e84bce64000
RBP: ffff8e83fb3437d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 00000001c06898dd R15: ffffffffaf7acf90
FS:  00007f41d8506740(0000) GS:ffff8e8575640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8f10b7c000 CR3: 00000000bb326000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffafa4f346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffafa7d0ca>] sch_direct_xmit+0x11a/0x250
[<ffffffffafa522c1>] __dev_queue_xmit+0x491/0x650
[<ffffffffafa52490>] dev_queue_xmit+0x10/0x20
[<ffffffffafaa099b>] ip_finish_output+0x67b/0x8d0
[<ffffffffafaa0f1b>] ip_output+0x7b/0xf0
[<ffffffffafaa0320>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffafa9e947>] ip_local_out_sk+0x37/0x40
[<ffffffffafaa1986>] ip_send_skb+0x16/0x50
[<ffffffffafac9d0c>] udp_send_skb+0xac/0x2b0
[<ffffffffafa9e680>] ? ip_copy_metadata+0x170/0x170
[<ffffffffafacb271>] udp_sendmsg+0x5e1/0xa70
[<ffffffffaf69934e>] ? ep_poll_callback+0xee/0x210
[<ffffffffafb83815>] ? _raw_spin_unlock_irqrestore+0x15/0x20
[<ffffffffafad89c9>] inet_sendmsg+0x69/0xb0
[<ffffffffafa2f3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffafb1077b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffafa30269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffafa97277>] ? __ip_route_output_key_hash+0x207/0x7b0
[<ffffffffafac771f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffafb839be>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffafaca7ef>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffafb839be>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffafa34a10>] ? release_sock+0x120/0x170
[<ffffffffafac7b87>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffafad7654>] ? inet_dgram_connect+0x34/0x90
[<ffffffffafa2f96d>] ? SYSC_connect+0xed/0x130
[<ffffffffafa31921>] __sys_sendmsg+0x51/0x90
[<ffffffffafa31972>] SyS_sendmsg+0x12/0x20
[<ffffffffafb8dede>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #580144
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0447857
Build id: 3.10.0-957.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffff89621697
Build id: 3.10.0-957.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffff89638c66
Build id: 3.10.0-957.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffff89665cea
Build id: 3.10.0-957.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffff8963bbe1
Build id: 3.10.0-957.el7.x86_64
0x491
6
? local_bh_enable
vmlinux 0xffffffff890a0387
Build id: 3.10.0-957.el7.x86_64
0x17
7
dev_queue_xmit
vmlinux 0xffffffff8963bdb0
Build id: 3.10.0-957.el7.x86_64
0x10
8
neigh_resolve_output
vmlinux 0xffffffff8964861d
Build id: 3.10.0-957.el7.x86_64
0x11d
9
ip_finish_output
vmlinux 0xffffffff8968845c
Build id: 3.10.0-957.el7.x86_64
0x27c
10
ip_output
vmlinux 0xffffffff89688c93
Build id: 3.10.0-957.el7.x86_64
0x73
11
? __ip_append_data.isra.51
vmlinux 0xffffffff896881e0
Build id: 3.10.0-957.el7.x86_64
0xa50
12
ip_local_out_sk
vmlinux 0xffffffff89686817
Build id: 3.10.0-957.el7.x86_64
0x37
13
ip_send_skb
vmlinux 0xffffffff896896f6
Build id: 3.10.0-957.el7.x86_64
0x16
14
udp_send_skb
vmlinux 0xffffffff896b150c
Build id: 3.10.0-957.el7.x86_64
0xac
15
? ip_copy_metadata
vmlinux 0xffffffff89686560
Build id: 3.10.0-957.el7.x86_64
0x170
16
udp_sendmsg
vmlinux 0xffffffff896b27f5
Build id: 3.10.0-957.el7.x86_64
0x335
17
? default_wake_function
vmlinux 0xffffffff890d67c2
Build id: 3.10.0-957.el7.x86_64
0x12
18
? __wake_up_common
vmlinux 0xffffffff890cba9b
Build id: 3.10.0-957.el7.x86_64
0x5b
19
? sock_has_perm
vmlinux 0xffffffff892fc3a5
Build id: 3.10.0-957.el7.x86_64
0x75
20
inet_sendmsg
vmlinux 0xffffffff896bfeb9
Build id: 3.10.0-957.el7.x86_64
0x69
21
? selinux_socket_sendmsg
vmlinux 0xffffffff892fc4d3
Build id: 3.10.0-957.el7.x86_64
0x23
22
sock_sendmsg
vmlinux 0xffffffff896193a6
Build id: 3.10.0-957.el7.x86_64
0xb6
23
? unix_dgram_sendmsg
vmlinux 0xffffffff896f769b
Build id: 3.10.0-957.el7.x86_64
0x3cb
24
___sys_sendmsg
vmlinux 0xffffffff8961a269
Build id: 3.10.0-957.el7.x86_64
0x3e9
25
? ip4_datagram_release_cb
vmlinux 0xffffffff896aef8f
Build id: 3.10.0-957.el7.x86_64
0x6f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff8976a29e
Build id: 3.10.0-957.el7.x86_64
0x1e
27
? udp_lib_rehash
vmlinux 0xffffffff896b1e0f
Build id: 3.10.0-957.el7.x86_64
0x11f
28
? _raw_spin_unlock_bh
vmlinux 0xffffffff8976a29e
Build id: 3.10.0-957.el7.x86_64
0x1e
29
? release_sock
vmlinux 0xffffffff8961e980
Build id: 3.10.0-957.el7.x86_64
0x120
30
? ip4_datagram_connect
vmlinux 0xffffffff896af3f7
Build id: 3.10.0-957.el7.x86_64
0x37
31
? inet_dgram_connect
vmlinux 0xffffffff896beb44
Build id: 3.10.0-957.el7.x86_64
0x34
32
? SYSC_connect
vmlinux 0xffffffff8961996d
Build id: 3.10.0-957.el7.x86_64
0xed
33
__sys_sendmsg
vmlinux 0xffffffff8961b921
Build id: 3.10.0-957.el7.x86_64
0x51
34
SyS_sendmsg
vmlinux 0xffffffff8961b972
Build id: 3.10.0-957.el7.x86_64
0x12
35
system_call_fastpath
vmlinux 0xffffffff89774ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [chronyd:8619]
Modules linked in: tcp_lp rfcomm fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmw_balloon cryptd snd_ens1371 snd_rawmidi snd_ac97_codec
ac97_bus snd_seq btusb snd_seq_device btrtl btbcm btintel snd_pcm bluetooth pcspkr joydev snd_timer snd soundcore rfkill sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi drm e1000 mptscsih ata_piix mptbase libata drm_panel_orientation_quirks nfit libnvdimm
CPU: 2 PID: 8619 Comm: chronyd Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff9e6df5ea30c0 ti: ffff9e6e76db8000 task.ti: ffff9e6e76db8000
RIP: 0010:[<ffffffffc0447857>]  [<ffffffffc0447857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9e6e76dbb720  EFLAGS: 00000286
RAX: ffffb21d009c3818 RBX: ffffffff89622bf3 RCX: 000000000000004e
RDX: 000000000000004e RSI: ffff9e6df5c82fc0 RDI: ffff9e6ef557a000
RBP: ffff9e6e76dbb7a0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 0000000000000000 R15: 0000000000000000
FS:  00007f4300f90740(0000) GS:ffff9e6ef9680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f10c3716000 CR3: 00000000b6dd6000 CR4: 00000000003607e0
Call Trace:
[<ffffffff89621697>] ? kfree_skbmem+0x37/0x90
[<ffffffff89638c66>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff89665cea>] sch_direct_xmit+0x11a/0x250
[<ffffffff8963bbe1>] __dev_queue_xmit+0x491/0x650
[<ffffffff890a0387>] ? local_bh_enable+0x17/0x20
[<ffffffff8963bdb0>] dev_queue_xmit+0x10/0x20
[<ffffffff8964861d>] neigh_resolve_output+0x11d/0x220
[<ffffffff8968845c>] ip_finish_output+0x27c/0x7b0
[<ffffffff89688c93>] ip_output+0x73/0xe0
[<ffffffff896881e0>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffff89686817>] ip_local_out_sk+0x37/0x40
[<ffffffff896896f6>] ip_send_skb+0x16/0x50
[<ffffffff896b150c>] udp_send_skb+0xac/0x2b0
[<ffffffff89686560>] ? ip_copy_metadata+0x170/0x170
[<ffffffff896b27f5>] udp_sendmsg+0x335/0xa10
[<ffffffff890d67c2>] ? default_wake_function+0x12/0x20
[<ffffffff890cba9b>] ? __wake_up_common+0x5b/0x90
[<ffffffff892fc3a5>] ? sock_has_perm+0x75/0x90
[<ffffffff896bfeb9>] inet_sendmsg+0x69/0xb0
[<ffffffff892fc4d3>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff896193a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff896f769b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffff8961a269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff896aef8f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff8976a29e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff896b1e0f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff8976a29e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff8961e980>] ? release_sock+0x120/0x170
[<ffffffff896af3f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff896beb44>] ? inet_dgram_connect+0x34/0x90
[<ffffffff8961996d>] ? SYSC_connect+0xed/0x130
[<ffffffff8961b921>] __sys_sendmsg+0x51/0x90
[<ffffffff8961b972>] SyS_sendmsg+0x12/0x20
[<ffffffff89774ddb>] system_call_fastpath+0x22/0x27
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #543230
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0227877
Build id: 3.10.0-1160.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffbda3c6f7
Build id: 3.10.0-1160.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffbda54386
Build id: 3.10.0-1160.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffffbda8242a
Build id: 3.10.0-1160.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffbda57301
Build id: 3.10.0-1160.el7.x86_64
0x491
6
dev_queue_xmit
vmlinux 0xffffffffbda574d0
Build id: 3.10.0-1160.el7.x86_64
0x10
7
neigh_resolve_output
vmlinux 0xffffffffbda641fd
Build id: 3.10.0-1160.el7.x86_64
0x11d
8
ip_finish_output
vmlinux 0xffffffffbdaa505c
Build id: 3.10.0-1160.el7.x86_64
0x27c
9
ip_output
vmlinux 0xffffffffbdaa59db
Build id: 3.10.0-1160.el7.x86_64
0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffffbdaa4de0
Build id: 3.10.0-1160.el7.x86_64
0xa60
11
ip_local_out_sk
vmlinux 0xffffffffbdaa3407
Build id: 3.10.0-1160.el7.x86_64
0x37
12
ip_send_skb
vmlinux 0xffffffffbdaa6446
Build id: 3.10.0-1160.el7.x86_64
0x16
13
udp_send_skb
vmlinux 0xffffffffbdace85c
Build id: 3.10.0-1160.el7.x86_64
0xac
14
? ip_copy_metadata
vmlinux 0xffffffffbdaa3140
Build id: 3.10.0-1160.el7.x86_64
0x170
15
udp_sendmsg
vmlinux 0xffffffffbdacfdc1
Build id: 3.10.0-1160.el7.x86_64
0x5e1
16
? __pollwait
vmlinux 0xffffffffbd663e50
Build id: 3.10.0-1160.el7.x86_64
0xf0
17
? sock_has_perm
vmlinux 0xffffffffbd70be65
Build id: 3.10.0-1160.el7.x86_64
0x75
18
inet_sendmsg
vmlinux 0xffffffffbdadd589
Build id: 3.10.0-1160.el7.x86_64
0x69
19
? selinux_socket_sendmsg
vmlinux 0xffffffffbd70bf93
Build id: 3.10.0-1160.el7.x86_64
0x23
20
sock_sendmsg
vmlinux 0xffffffffbda343a6
Build id: 3.10.0-1160.el7.x86_64
0xb6
21
? unix_dgram_sendmsg
vmlinux 0xffffffffbdb154bb
Build id: 3.10.0-1160.el7.x86_64
0x3cb
22
___sys_sendmsg
vmlinux 0xffffffffbda35269
Build id: 3.10.0-1160.el7.x86_64
0x3e9
23
? __ip_route_output_key_hash
vmlinux 0xffffffffbda9bc63
Build id: 3.10.0-1160.el7.x86_64
0x223
24
? ip4_datagram_release_cb
vmlinux 0xffffffffbdacc28f
Build id: 3.10.0-1160.el7.x86_64
0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffbdb8992e
Build id: 3.10.0-1160.el7.x86_64
0x1e
26
? udp_lib_rehash
vmlinux 0xffffffffbdacf33f
Build id: 3.10.0-1160.el7.x86_64
0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffbdb8992e
Build id: 3.10.0-1160.el7.x86_64
0x1e
28
? release_sock
vmlinux 0xffffffffbda39a18
Build id: 3.10.0-1160.el7.x86_64
0x128
29
? ip4_datagram_connect
vmlinux 0xffffffffbdacc6f7
Build id: 3.10.0-1160.el7.x86_64
0x37
30
? inet_dgram_connect
vmlinux 0xffffffffbdadc214
Build id: 3.10.0-1160.el7.x86_64
0x34
31
? SYSC_connect
vmlinux 0xffffffffbda3496d
Build id: 3.10.0-1160.el7.x86_64
0xed
32
__sys_sendmsg
vmlinux 0xffffffffbda36921
Build id: 3.10.0-1160.el7.x86_64
0x51
33
SyS_sendmsg
vmlinux 0xffffffffbda36972
Build id: 3.10.0-1160.el7.x86_64
0x12
34
system_call_fastpath
vmlinux 0xffffffffbdb93f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 71s! [chronyd:763]
Modules linked in: tun devlink bridge stp llc ip_set nfnetlink bnep vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel btusb lrw gf128mul ppdev btrtl glue_helper btbcm ablk_helper btintel cryptd bluetooth vmw_balloon rfkill sg pcspkr joydev parport_pc parport snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore i2c_piix4 vmw_vmci xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crc32c_intel nfit libnvdimm drm ata_piix serio_raw e1000 mptspi scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash
dm_log dm_mod fuse [last unloaded: ip_tables]
CPU: 3 PID: 763 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffffa0bcb518b180 ti: ffffa0bcf4bcc000 task.ti: ffffa0bcf4bcc000
RIP: 0010:[<ffffffffc0227877>]  [<ffffffffc0227877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffffa0bcf4bcf710  EFLAGS: 00000286
RAX: ffffbe4180383818 RBX: ffffffffbda3ce73 RCX: 0000000000000038
RDX: 0000000000000038 RSI: ffffa0bcb4d6b140 RDI: ffffa0bcf59da000
RBP: ffffa0bcf4bcf790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 0000000000000800 R15: 0000000000000400
FS:  00007effe2acc740(0000) GS:ffffa0bcf8cc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f64ab0ba000 CR3: 000000007292e000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffbda3c6f7>] ? kfree_skbmem+0x37/0x90
[<ffffffffbda54386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffbda8242a>] sch_direct_xmit+0x11a/0x250
[<ffffffffbda57301>] __dev_queue_xmit+0x491/0x650
[<ffffffffbda574d0>] dev_queue_xmit+0x10/0x20
[<ffffffffbda641fd>] neigh_resolve_output+0x11d/0x220
[<ffffffffbdaa505c>] ip_finish_output+0x27c/0x8d0
[<ffffffffbdaa59db>] ip_output+0x7b/0xf0
[<ffffffffbdaa4de0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffbdaa3407>] ip_local_out_sk+0x37/0x40
[<ffffffffbdaa6446>] ip_send_skb+0x16/0x50
[<ffffffffbdace85c>] udp_send_skb+0xac/0x2b0
[<ffffffffbdaa3140>] ? ip_copy_metadata+0x170/0x170
[<ffffffffbdacfdc1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffbd663e50>] ? __pollwait+0xf0/0xf0
[<ffffffffbd70be65>] ? sock_has_perm+0x75/0x90
[<ffffffffbdadd589>] inet_sendmsg+0x69/0xb0
[<ffffffffbd70bf93>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffffbda343a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffbdb154bb>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffbda35269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffbda9bc63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffffbdacc28f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffbdb8992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffbdacf33f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffbdb8992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffbda39a18>] ? release_sock+0x128/0x180
[<ffffffffbdacc6f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffbdadc214>] ? inet_dgram_connect+0x34/0x90
[<ffffffffbda3496d>] ? SYSC_connect+0xed/0x130
[<ffffffffbda36921>] __sys_sendmsg+0x51/0x90
[<ffffffffbda36972>] SyS_sendmsg+0x12/0x20
[<ffffffffbdb93f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #529022
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc02ab857
Build id: 3.10.0-957.5.1.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffbec21717
Build id: 3.10.0-957.5.1.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffbec38ce6
Build id: 3.10.0-957.5.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffffbec65d6a
Build id: 3.10.0-957.5.1.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffbec3bc61
Build id: 3.10.0-957.5.1.el7.x86_64
0x491
6
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc080f2ce
Build id: 3.10.0-957.5.1.el7.x86_64
0x4e
7
dev_queue_xmit
vmlinux 0xffffffffbec3be30
Build id: 3.10.0-957.5.1.el7.x86_64
0x10
8
ip_finish_output
vmlinux 0xffffffffbec887b6
Build id: 3.10.0-957.5.1.el7.x86_64
0x556
9
ip_output
vmlinux 0xffffffffbec88d13
Build id: 3.10.0-957.5.1.el7.x86_64
0x73
10
? __ip_append_data.isra.51
vmlinux 0xffffffffbec88260
Build id: 3.10.0-957.5.1.el7.x86_64
0xa50
11
ip_local_out_sk
vmlinux 0xffffffffbec86897
Build id: 3.10.0-957.5.1.el7.x86_64
0x37
12
ip_send_skb
vmlinux 0xffffffffbec89776
Build id: 3.10.0-957.5.1.el7.x86_64
0x16
13
udp_send_skb
vmlinux 0xffffffffbecb158c
Build id: 3.10.0-957.5.1.el7.x86_64
0xac
14
? ip_copy_metadata
vmlinux 0xffffffffbec865e0
Build id: 3.10.0-957.5.1.el7.x86_64
0x170
15
udp_sendmsg
vmlinux 0xffffffffbecb2875
Build id: 3.10.0-957.5.1.el7.x86_64
0x335
16
? default_wake_function
vmlinux 0xffffffffbe6d6802
Build id: 3.10.0-957.5.1.el7.x86_64
0x12
17
? __wake_up_common
vmlinux 0xffffffffbe6cbadb
Build id: 3.10.0-957.5.1.el7.x86_64
0x5b
18
? sock_has_perm
vmlinux 0xffffffffbe8fc985
Build id: 3.10.0-957.5.1.el7.x86_64
0x75
19
inet_sendmsg
vmlinux 0xffffffffbecbff39
Build id: 3.10.0-957.5.1.el7.x86_64
0x69
20
? selinux_socket_sendmsg
vmlinux 0xffffffffbe8fcab3
Build id: 3.10.0-957.5.1.el7.x86_64
0x23
21
sock_sendmsg
vmlinux 0xffffffffbec193a6
Build id: 3.10.0-957.5.1.el7.x86_64
0xb6
22
? unix_dgram_sendmsg
vmlinux 0xffffffffbecf771b
Build id: 3.10.0-957.5.1.el7.x86_64
0x3cb
23
___sys_sendmsg
vmlinux 0xffffffffbec1a269
Build id: 3.10.0-957.5.1.el7.x86_64
0x3e9
24
? ip4_datagram_release_cb
vmlinux 0xffffffffbecaf00f
Build id: 3.10.0-957.5.1.el7.x86_64
0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffbed6a31e
Build id: 3.10.0-957.5.1.el7.x86_64
0x1e
26
? udp_lib_rehash
vmlinux 0xffffffffbecb1e8f
Build id: 3.10.0-957.5.1.el7.x86_64
0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffbed6a31e
Build id: 3.10.0-957.5.1.el7.x86_64
0x1e
28
? release_sock
vmlinux 0xffffffffbec1ea00
Build id: 3.10.0-957.5.1.el7.x86_64
0x120
29
? ip4_datagram_connect
vmlinux 0xffffffffbecaf477
Build id: 3.10.0-957.5.1.el7.x86_64
0x37
30
? inet_dgram_connect
vmlinux 0xffffffffbecbebc4
Build id: 3.10.0-957.5.1.el7.x86_64
0x34
31
? SYSC_connect
vmlinux 0xffffffffbec1996d
Build id: 3.10.0-957.5.1.el7.x86_64
0xed
32
__sys_sendmsg
vmlinux 0xffffffffbec1b921
Build id: 3.10.0-957.5.1.el7.x86_64
0x51
33
SyS_sendmsg
vmlinux 0xffffffffbec1b972
Build id: 3.10.0-957.5.1.el7.x86_64
0x12
34
system_call_fastpath
vmlinux 0xffffffffbed74ddb
Build id: 3.10.0-957.5.1.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 31s! [chronyd:9190]
Modules linked in: tcp_lp rfcomm xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack fuse ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep sunrpc snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi joydev btusb snd_ac97_codec btrtl btbcm ac97_bus
btintel pcspkr bluetooth snd_seq snd_seq_device snd_pcm snd_timer rfkill snd soundcore sg vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common ata_generic pata_acpi crc32c_intel vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi e1000 libata scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 9190 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8d712a3b8000 ti: ffff8d6f39d50000 task.ti: ffff8d6f39d50000
RIP: 0010:[<ffffffffc02ab857>]  [<ffffffffc02ab857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8d6f39d53760  EFLAGS: 00000286
RAX: ffffb10181c03818 RBX: ffffffffbec22c73 RCX: 0000000000000006
RDX: 0000000000000006 RSI: ffff8d712f344140 RDI: ffff8d70e622c000
RBP: ffff8d6f39d537e0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 00000001c07d8915 R15: ffffffffbe9a0a90
FS:  00007f10f5b1b740(0000) GS:ffff8d7133680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1620e15000 CR3: 00000000ba80e000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffbec21717>] ? kfree_skbmem+0x37/0x90
[<ffffffffbec38ce6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffbec65d6a>] sch_direct_xmit+0x11a/0x250
[<ffffffffbec3bc61>] __dev_queue_xmit+0x491/0x650
[<ffffffffc080f2ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffffbec3be30>] dev_queue_xmit+0x10/0x20
[<ffffffffbec887b6>] ip_finish_output+0x556/0x7b0
[<ffffffffbec88d13>] ip_output+0x73/0xe0
[<ffffffffbec88260>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffbec86897>] ip_local_out_sk+0x37/0x40
[<ffffffffbec89776>] ip_send_skb+0x16/0x50
[<ffffffffbecb158c>] udp_send_skb+0xac/0x2b0
[<ffffffffbec865e0>] ? ip_copy_metadata+0x170/0x170
[<ffffffffbecb2875>] udp_sendmsg+0x335/0xa10
[<ffffffffbe6d6802>] ? default_wake_function+0x12/0x20
[<ffffffffbe6cbadb>] ? __wake_up_common+0x5b/0x90
[<ffffffffbe8fc985>] ? sock_has_perm+0x75/0x90
[<ffffffffbecbff39>] inet_sendmsg+0x69/0xb0
[<ffffffffbe8fcab3>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffffbec193a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffbecf771b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffbec1a269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffbecaf00f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffbed6a31e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffbecb1e8f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffbed6a31e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffbec1ea00>] ? release_sock+0x120/0x170
[<ffffffffbecaf477>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffbecbebc4>] ? inet_dgram_connect+0x34/0x90
[<ffffffffbec1996d>] ? SYSC_connect+0xed/0x130
[<ffffffffbec1b921>] __sys_sendmsg+0x51/0x90
[<ffffffffbec1b972>] SyS_sendmsg+0x12/0x20
[<ffffffffbed74ddb>] system_call_fastpath+0x22/0x27
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #467484
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc02ec877
Build id: 3.10.0-1062.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffb2a356e7
Build id: 3.10.0-1062.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffb2a4d346
Build id: 3.10.0-1062.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffffb2a7b0ba
Build id: 3.10.0-1062.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffb2a502b1
Build id: 3.10.0-1062.el7.x86_64
0x491
6
dev_queue_xmit
vmlinux 0xffffffffb2a50480
Build id: 3.10.0-1062.el7.x86_64
0x10
7
neigh_resolve_output
vmlinux 0xffffffffb2a5cfed
Build id: 3.10.0-1062.el7.x86_64
0x11d
8
ip_finish_output
vmlinux 0xffffffffb2a9e57c
Build id: 3.10.0-1062.el7.x86_64
0x27c
9
ip_output
vmlinux 0xffffffffb2a9eefb
Build id: 3.10.0-1062.el7.x86_64
0x7b
10
? __ip_local_out_sk
vmlinux 0xffffffffb2a9c8e2
Build id: 3.10.0-1062.el7.x86_64
0x102
11
ip_local_out_sk
vmlinux 0xffffffffb2a9c927
Build id: 3.10.0-1062.el7.x86_64
0x37
12
ip_send_skb
vmlinux 0xffffffffb2a9f966
Build id: 3.10.0-1062.el7.x86_64
0x16
13
udp_send_skb
vmlinux 0xffffffffb2ac7c6c
Build id: 3.10.0-1062.el7.x86_64
0xac
14
? ip_copy_metadata
vmlinux 0xffffffffb2a9c660
Build id: 3.10.0-1062.el7.x86_64
0x170
15
udp_sendmsg
vmlinux 0xffffffffb2ac91d1
Build id: 3.10.0-1062.el7.x86_64
0x5e1
16
? ep_poll_callback
vmlinux 0xffffffffb2696f4e
Build id: 3.10.0-1062.el7.x86_64
0xee
17
? __wake_up_common
vmlinux 0xffffffffb24d24d2
Build id: 3.10.0-1062.el7.x86_64
0x82
18
inet_sendmsg
vmlinux 0xffffffffb2ad6939
Build id: 3.10.0-1062.el7.x86_64
0x69
19
sock_sendmsg
vmlinux 0xffffffffb2a2d3a6
Build id: 3.10.0-1062.el7.x86_64
0xb6
20
? unix_dgram_sendmsg
vmlinux 0xffffffffb2b0e6db
Build id: 3.10.0-1062.el7.x86_64
0x3cb
21
___sys_sendmsg
vmlinux 0xffffffffb2a2e269
Build id: 3.10.0-1062.el7.x86_64
0x3e9
22
? __ip_route_output_key_hash
vmlinux 0xffffffffb2a95257
Build id: 3.10.0-1062.el7.x86_64
0x207
23
? ip4_datagram_release_cb
vmlinux 0xffffffffb2ac567f
Build id: 3.10.0-1062.el7.x86_64
0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffb2b8196e
Build id: 3.10.0-1062.el7.x86_64
0x1e
25
? udp_lib_rehash
vmlinux 0xffffffffb2ac874f
Build id: 3.10.0-1062.el7.x86_64
0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffffb2b8196e
Build id: 3.10.0-1062.el7.x86_64
0x1e
27
? release_sock
vmlinux 0xffffffffb2a32a10
Build id: 3.10.0-1062.el7.x86_64
0x120
28
? ip4_datagram_connect
vmlinux 0xffffffffb2ac5ae7
Build id: 3.10.0-1062.el7.x86_64
0x37
29
? inet_dgram_connect
vmlinux 0xffffffffb2ad55c4
Build id: 3.10.0-1062.el7.x86_64
0x34
30
? SYSC_connect
vmlinux 0xffffffffb2a2d96d
Build id: 3.10.0-1062.el7.x86_64
0xed
31
__sys_sendmsg
vmlinux 0xffffffffb2a2f921
Build id: 3.10.0-1062.el7.x86_64
0x51
32
SyS_sendmsg
vmlinux 0xffffffffb2a2f972
Build id: 3.10.0-1062.el7.x86_64
0x12
33
system_call_fastpath
vmlinux 0xffffffffb2b8bede
Build id: 3.10.0-1062.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 25s! [chronyd:987]
Modules linked in: ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink bnep vmw_vsock_vmci_transport vsock sunrpc btusb btrtl btbcm btintel bluetooth rfkill ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw vmw_balloon gf128mul pcspkr glue_helper ablk_helper joydev cryptd sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt ata_piix fb_sys_fops ttm nfit libata libnvdimm mptspi drm crct10dif_pclmul crct10dif_common scsi_transport_spi crc32c_intel mptscsih mptbase serio_raw e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 987 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8f02b160e2a0 ti: ffff8f02b5180000 task.ti: ffff8f02b5180000
RIP: 0010:[<ffffffffc02ec877>]  [<ffffffffc02ec877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8f02b5183710  EFLAGS: 00000286
RAX: ffff9f8e40983818 RBX: ffffffffb2a35e63 RCX: 00000000000000c6
RDX: 00000000000000c6 RSI: ffff8f01b5c32580 RDI: ffff8f02b5662000
RBP: ffff8f02b5183790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 0000000000000001 R15: ffff8f02b96dac40
FS:  00007fd19492d740(0000) GS:ffff8f02b96c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007faa17730000 CR3: 00000001352a4000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb2a356e7>] ? kfree_skbmem+0x37/0x90
[<ffffffffb2a4d346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffb2a7b0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffffb2a502b1>] __dev_queue_xmit+0x491/0x650
[<ffffffffb2a50480>] dev_queue_xmit+0x10/0x20
[<ffffffffb2a5cfed>] neigh_resolve_output+0x11d/0x220
[<ffffffffb2a9e57c>] ip_finish_output+0x27c/0x8d0
[<ffffffffb2a9eefb>] ip_output+0x7b/0xf0
[<ffffffffb2a9c8e2>] ? __ip_local_out_sk+0x102/0x110
[<ffffffffb2a9c927>] ip_local_out_sk+0x37/0x40
[<ffffffffb2a9f966>] ip_send_skb+0x16/0x50
[<ffffffffb2ac7c6c>] udp_send_skb+0xac/0x2b0
[<ffffffffb2a9c660>] ? ip_copy_metadata+0x170/0x170
[<ffffffffb2ac91d1>] udp_sendmsg+0x5e1/0xa70
[<ffffffffb2696f4e>] ? ep_poll_callback+0xee/0x210
[<ffffffffb24d24d2>] ? __wake_up_common+0x82/0x120
[<ffffffffb2ad6939>] inet_sendmsg+0x69/0xb0
[<ffffffffb2a2d3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffb2b0e6db>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffb2a2e269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffb2a95257>] ? __ip_route_output_key_hash+0x207/0x7b0
[<ffffffffb2ac567f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffb2b8196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb2ac874f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffb2b8196e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffb2a32a10>] ? release_sock+0x120/0x170
[<ffffffffb2ac5ae7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffb2ad55c4>] ? inet_dgram_connect+0x34/0x90
[<ffffffffb2a2d96d>] ? SYSC_connect+0xed/0x130
[<ffffffffb2a2f921>] __sys_sendmsg+0x51/0x90
[<ffffffffb2a2f972>] SyS_sendmsg+0x12/0x20
[<ffffffffb2b8bede>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #371331
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00f4877
Build id: 3.10.0-1160.41.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffaea55f96
Build id: 3.10.0-1160.41.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffaea840aa
Build id: 3.10.0-1160.41.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffaea58f11
Build id: 3.10.0-1160.41.1.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffffaea590e0
Build id: 3.10.0-1160.41.1.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffffaeaa7c6b
Build id: 3.10.0-1160.41.1.el7.x86_64
0x67b
7
ip_output
vmlinux 0xffffffffaeaa81eb
Build id: 3.10.0-1160.41.1.el7.x86_64
0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffffaeaa75f0
Build id: 3.10.0-1160.41.1.el7.x86_64
0xa60
9
ip_local_out_sk
vmlinux 0xffffffffaeaa5c17
Build id: 3.10.0-1160.41.1.el7.x86_64
0x37
10
ip_send_skb
vmlinux 0xffffffffaeaa8c56
Build id: 3.10.0-1160.41.1.el7.x86_64
0x16
11
udp_send_skb
vmlinux 0xffffffffaead10bc
Build id: 3.10.0-1160.41.1.el7.x86_64
0xac
12
? ip_copy_metadata
vmlinux 0xffffffffaeaa5950
Build id: 3.10.0-1160.41.1.el7.x86_64
0x170
13
udp_sendmsg
vmlinux 0xffffffffaead2621
Build id: 3.10.0-1160.41.1.el7.x86_64
0x5e1
14
? __pollwait
vmlinux 0xffffffffae664620
Build id: 3.10.0-1160.41.1.el7.x86_64
0xf0
15
? __pollwait
vmlinux 0xffffffffae664620
Build id: 3.10.0-1160.41.1.el7.x86_64
0xf0
16
inet_sendmsg
vmlinux 0xffffffffaeadfe09
Build id: 3.10.0-1160.41.1.el7.x86_64
0x69
17
? __pollwait
vmlinux 0xffffffffae664620
Build id: 3.10.0-1160.41.1.el7.x86_64
0xf0
18
sock_sendmsg
vmlinux 0xffffffffaea363a6
Build id: 3.10.0-1160.41.1.el7.x86_64
0xb6
19
___sys_sendmsg
vmlinux 0xffffffffaea37269
Build id: 3.10.0-1160.41.1.el7.x86_64
0x3e9
20
? __ip_route_output_key_hash
vmlinux 0xffffffffaea9e483
Build id: 3.10.0-1160.41.1.el7.x86_64
0x223
21
? ip4_datagram_release_cb
vmlinux 0xffffffffaeaceaef
Build id: 3.10.0-1160.41.1.el7.x86_64
0x6f
22
? _raw_spin_unlock_bh
vmlinux 0xffffffffaeb8b93e
Build id: 3.10.0-1160.41.1.el7.x86_64
0x1e
23
? udp_lib_rehash
vmlinux 0xffffffffaead1b9f
Build id: 3.10.0-1160.41.1.el7.x86_64
0x11f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffaeb8b93e
Build id: 3.10.0-1160.41.1.el7.x86_64
0x1e
25
? release_sock
vmlinux 0xffffffffaea3ba18
Build id: 3.10.0-1160.41.1.el7.x86_64
0x128
26
? ip4_datagram_connect
vmlinux 0xffffffffaeacef57
Build id: 3.10.0-1160.41.1.el7.x86_64
0x37
27
? inet_dgram_connect
vmlinux 0xffffffffaeadea94
Build id: 3.10.0-1160.41.1.el7.x86_64
0x34
28
? SYSC_connect
vmlinux 0xffffffffaea3696d
Build id: 3.10.0-1160.41.1.el7.x86_64
0xed
29
__sys_sendmsg
vmlinux 0xffffffffaea38921
Build id: 3.10.0-1160.41.1.el7.x86_64
0x51
30
SyS_sendmsg
vmlinux 0xffffffffaea38972
Build id: 3.10.0-1160.41.1.el7.x86_64
0x12
31
system_call_fastpath
vmlinux 0xffffffffaeb95f92
Build id: 3.10.0-1160.41.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [chronyd:612]
Modules linked in: lp tcp_lp rpcsec_gss_krb5 xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus vmw_balloon snd_seq aesni_intel
lrw gf128mul glue_helper ablk_helper cryptd snd_seq_device snd_pcm joydev pcspkr sg snd_timer snd soundcore vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper nfit syscopyarea sysfillrect sysimgblt crct10dif_pclmul crct10dif_common libnvdimm fb_sys_fops ttm crc32c_intel ata_piix mptspi libata scsi_transport_spi mptscsih drm e1000 serio_raw mptbase drm_panel_orientation_quirks fuse
CPU: 1 PID: 612 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.41.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9338b4524200 ti: ffff93383679c000 task.ti: ffff93383679c000
RIP: 0010:[<ffffffffc00f4877>]  [<ffffffffc00f4877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff93383679f750  EFLAGS: 00000286
RAX: ffffb79ec0703818 RBX: ffffffffaea3ee73 RCX: 0000000000000014
RDX: 0000000000000014 RSI: ffff9337b5e37100 RDI: ffff9338b4648000
RBP: ffff93383679f7d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffffffffc06cb8dd R15: ffff93383679f6e8
FS:  00007f18d43df740(0000) GS:ffff9338b9640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0476e083bc CR3: 00000000b66a6000 CR4: 00000000003607e0
Call Trace:
[<ffffffffaea55f96>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffaea840aa>] sch_direct_xmit+0x11a/0x250
[<ffffffffaea58f11>] __dev_queue_xmit+0x491/0x650
[<ffffffffaea590e0>] dev_queue_xmit+0x10/0x20
[<ffffffffaeaa7c6b>] ip_finish_output+0x67b/0x8d0
[<ffffffffaeaa81eb>] ip_output+0x7b/0xf0
[<ffffffffaeaa75f0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffaeaa5c17>] ip_local_out_sk+0x37/0x40
[<ffffffffaeaa8c56>] ip_send_skb+0x16/0x50
[<ffffffffaead10bc>] udp_send_skb+0xac/0x2b0
[<ffffffffaeaa5950>] ? ip_copy_metadata+0x170/0x170
[<ffffffffaead2621>] udp_sendmsg+0x5e1/0xa70
[<ffffffffae664620>] ? __pollwait+0xf0/0xf0
[<ffffffffae664620>] ? __pollwait+0xf0/0xf0
[<ffffffffaeadfe09>] inet_sendmsg+0x69/0xb0
[<ffffffffae664620>] ? __pollwait+0xf0/0xf0
[<ffffffffaea363a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffaea37269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffaea9e483>] ? __ip_route_output_key_hash+0x223/0x7e0
[<ffffffffaeaceaef>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffaeb8b93e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffaead1b9f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffaeb8b93e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffaea3ba18>] ? release_sock+0x128/0x180
[<ffffffffaeacef57>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffaeadea94>] ? inet_dgram_connect+0x34/0x90
[<ffffffffaea3696d>] ? SYSC_connect+0xed/0x130
[<ffffffffaea38921>] __sys_sendmsg+0x51/0x90
[<ffffffffaea38972>] SyS_sendmsg+0x12/0x20
[<ffffffffaeb95f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #351193
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc03c5867
Build id: 3.10.0-862.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff979eb166
Build id: 3.10.0-862.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff97a16a2a
Build id: 3.10.0-862.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff979ee051
Build id: 3.10.0-862.el7.x86_64
0x4a1
5
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc094a2ce
Build id: 3.10.0-862.el7.x86_64
0x4e
6
dev_queue_xmit
vmlinux 0xffffffff979ee220
Build id: 3.10.0-862.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffff97a37416
Build id: 3.10.0-862.el7.x86_64
0x546
8
ip_output
vmlinux 0xffffffff97a37973
Build id: 3.10.0-862.el7.x86_64
0x73
9
? __ip_append_data.isra.50
vmlinux 0xffffffff97a36ed0
Build id: 3.10.0-862.el7.x86_64
0xa50
10
ip_local_out_sk
vmlinux 0xffffffff97a35567
Build id: 3.10.0-862.el7.x86_64
0x37
11
ip_send_skb
vmlinux 0xffffffff97a383c6
Build id: 3.10.0-862.el7.x86_64
0x16
12
udp_send_skb
vmlinux 0xffffffff97a5fc4c
Build id: 3.10.0-862.el7.x86_64
0xac
13
? ip_copy_metadata
vmlinux 0xffffffff97a352b0
Build id: 3.10.0-862.el7.x86_64
0x170
14
udp_sendmsg
vmlinux 0xffffffff97a610fc
Build id: 3.10.0-862.el7.x86_64
0x32c
15
? __alloc_skb
vmlinux 0xffffffff979d6d0d
Build id: 3.10.0-862.el7.x86_64
0x8d
16
? __alloc_skb
vmlinux 0xffffffff979d6d21
Build id: 3.10.0-862.el7.x86_64
0xa1
17
? ep_poll_callback
vmlinux 0xffffffff97667bee
Build id: 3.10.0-862.el7.x86_64
0xee
18
inet_sendmsg
vmlinux 0xffffffff97a6e5a9
Build id: 3.10.0-862.el7.x86_64
0x69
19
sock_sendmsg
vmlinux 0xffffffff979cc396
Build id: 3.10.0-862.el7.x86_64
0xb6
20
? unix_dgram_sendmsg
vmlinux 0xffffffff97aa4dab
Build id: 3.10.0-862.el7.x86_64
0x3cb
21
___sys_sendmsg
vmlinux 0xffffffff979cd199
Build id: 3.10.0-862.el7.x86_64
0x3a9
22
? ip4_datagram_release_cb
vmlinux 0xffffffff97a5d72f
Build id: 3.10.0-862.el7.x86_64
0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffff97b1551e
Build id: 3.10.0-862.el7.x86_64
0x1e
24
? udp_lib_rehash
vmlinux 0xffffffff97a6054f
Build id: 3.10.0-862.el7.x86_64
0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffff97b1551e
Build id: 3.10.0-862.el7.x86_64
0x1e
26
? release_sock
vmlinux 0xffffffff979d1660
Build id: 3.10.0-862.el7.x86_64
0x120
27
? ip4_datagram_connect
vmlinux 0xffffffff97a5db97
Build id: 3.10.0-862.el7.x86_64
0x37
28
? inet_dgram_connect
vmlinux 0xffffffff97a6d234
Build id: 3.10.0-862.el7.x86_64
0x34
29
? SYSC_connect
vmlinux 0xffffffff979cc93d
Build id: 3.10.0-862.el7.x86_64
0xed
30
__sys_sendmsg
vmlinux 0xffffffff979ce761
Build id: 3.10.0-862.el7.x86_64
0x51
31
SyS_sendmsg
vmlinux 0xffffffff979ce7b2
Build id: 3.10.0-862.el7.x86_64
0x12
32
system_call_fastpath
vmlinux 0xffffffff97b1f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [chronyd:669]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi kvm_intel kvm snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus irqbypass snd_seq crc32_pclmul ppdev snd_seq_device snd_pcm ghash_clmulni_intel cryptd vmw_balloon joydev pcspkr snd_timer snd sg soundcore i2c_piix4 vmw_vmci shpchp nfit libnvdimm parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul syscopyarea crct10dif_common
sysfillrect sysimgblt fb_sys_fops crc32c_intel ttm drm e1000 mptspi serio_raw scsi_transport_spi mptscsih ata_piix mptbase i2c_core libata floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 669 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff9f1ef117dee0 ti: ffff9f1e39de8000 task.ti: ffff9f1e39de8000
RIP: 0010:[<ffffffffc03c5867>]  [<ffffffffc03c5867>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff9f1e39deb760  EFLAGS: 00000286
RAX: ffffaa5cc0f03818 RBX: ffffffff979d58b3 RCX: 0000000000000066
RDX: 0000000000000066 RSI: ffff9f1db5ea7e40 RDI: ffff9f1ef4a88000
RBP: ffff9f1e39deb7e0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 00000001c09248f5 R15: ffffffff97773e00
FS:  00007ff4196ac740(0000) GS:ffff9f1ef9600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd84b192000 CR3: 00000000ba56a000 CR4: 00000000001607f0
Call Trace:
[<ffffffff979eb166>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff97a16a2a>] sch_direct_xmit+0x11a/0x250
[<ffffffff979ee051>] __dev_queue_xmit+0x4a1/0x660
[<ffffffffc094a2ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffff979ee220>] dev_queue_xmit+0x10/0x20
[<ffffffff97a37416>] ip_finish_output+0x546/0x7a0
[<ffffffff97a37973>] ip_output+0x73/0xe0
[<ffffffff97a36ed0>] ? __ip_append_data.isra.50+0xa50/0xa50
[<ffffffff97a35567>] ip_local_out_sk+0x37/0x40
[<ffffffff97a383c6>] ip_send_skb+0x16/0x50
[<ffffffff97a5fc4c>] udp_send_skb+0xac/0x2b0
[<ffffffff97a352b0>] ? ip_copy_metadata+0x170/0x170
[<ffffffff97a610fc>] udp_sendmsg+0x32c/0xa00
[<ffffffff979d6d0d>] ? __alloc_skb+0x8d/0x2d0
[<ffffffff979d6d21>] ? __alloc_skb+0xa1/0x2d0
[<ffffffff97667bee>] ? ep_poll_callback+0xee/0x210
[<ffffffff97a6e5a9>] inet_sendmsg+0x69/0xb0
[<ffffffff979cc396>] sock_sendmsg+0xb6/0xf0
[<ffffffff97aa4dab>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffff979cd199>] ___sys_sendmsg+0x3a9/0x3c0
[<ffffffff97a5d72f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff97b1551e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff97a6054f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff97b1551e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff979d1660>] ? release_sock+0x120/0x170
[<ffffffff97a5db97>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff97a6d234>] ? inet_dgram_connect+0x34/0x90
[<ffffffff979cc93d>] ? SYSC_connect+0xed/0x130
[<ffffffff979ce761>] __sys_sendmsg+0x51/0x90
[<ffffffff979ce7b2>] SyS_sendmsg+0x12/0x20
[<ffffffff97b1f7d5>] system_call_fastpath+0x1c/0x21
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #286329
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc02b3877
Build id: 3.10.0-1160.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff8d254386
Build id: 3.10.0-1160.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff8d28242a
Build id: 3.10.0-1160.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff8d257301
Build id: 3.10.0-1160.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffff8d2574d0
Build id: 3.10.0-1160.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffff8d2a545b
Build id: 3.10.0-1160.el7.x86_64
0x67b
7
ip_output
vmlinux 0xffffffff8d2a59db
Build id: 3.10.0-1160.el7.x86_64
0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffff8d2a4de0
Build id: 3.10.0-1160.el7.x86_64
0xa60
9
ip_local_out_sk
vmlinux 0xffffffff8d2a3407
Build id: 3.10.0-1160.el7.x86_64
0x37
10
ip_send_skb
vmlinux 0xffffffff8d2a6446
Build id: 3.10.0-1160.el7.x86_64
0x16
11
udp_send_skb
vmlinux 0xffffffff8d2ce85c
Build id: 3.10.0-1160.el7.x86_64
0xac
12
? ip_copy_metadata
vmlinux 0xffffffff8d2a3140
Build id: 3.10.0-1160.el7.x86_64
0x170
13
udp_sendmsg
vmlinux 0xffffffff8d2cfdc1
Build id: 3.10.0-1160.el7.x86_64
0x5e1
14
? __enqueue_entity
vmlinux 0xffffffff8ccdfb48
Build id: 3.10.0-1160.el7.x86_64
0x78
15
? check_preempt_curr
vmlinux 0xffffffff8ccd7202
Build id: 3.10.0-1160.el7.x86_64
0x92
16
? ttwu_do_wakeup
vmlinux 0xffffffff8ccd7229
Build id: 3.10.0-1160.el7.x86_64
0x19
17
? ttwu_do_activate
vmlinux 0xffffffff8ccd735f
Build id: 3.10.0-1160.el7.x86_64
0x6f
18
inet_sendmsg
vmlinux 0xffffffff8d2dd589
Build id: 3.10.0-1160.el7.x86_64
0x69
19
sock_sendmsg
vmlinux 0xffffffff8d2343a6
Build id: 3.10.0-1160.el7.x86_64
0xb6
20
? __wake_up_common
vmlinux 0xffffffff8ccd2fd2
Build id: 3.10.0-1160.el7.x86_64
0x82
21
? sock_recvmsg
vmlinux 0xffffffff8d2344f5
Build id: 3.10.0-1160.el7.x86_64
0xc5
22
? ep_poll_callback
vmlinux 0xffffffff8ce9d008
Build id: 3.10.0-1160.el7.x86_64
0xf8
23
___sys_sendmsg
vmlinux 0xffffffff8d235269
Build id: 3.10.0-1160.el7.x86_64
0x3e9
24
? __ip_route_output_key_hash
vmlinux 0xffffffff8d29bc63
Build id: 3.10.0-1160.el7.x86_64
0x223
25
? ip4_datagram_release_cb
vmlinux 0xffffffff8d2cc28f
Build id: 3.10.0-1160.el7.x86_64
0x6f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff8d38992e
Build id: 3.10.0-1160.el7.x86_64
0x1e
27
? udp_lib_rehash
vmlinux 0xffffffff8d2cf33f
Build id: 3.10.0-1160.el7.x86_64
0x11f
28
? _raw_spin_unlock_bh
vmlinux 0xffffffff8d38992e
Build id: 3.10.0-1160.el7.x86_64
0x1e
29
? release_sock
vmlinux 0xffffffff8d239a18
Build id: 3.10.0-1160.el7.x86_64
0x128
30
? ip4_datagram_connect
vmlinux 0xffffffff8d2cc6f7
Build id: 3.10.0-1160.el7.x86_64
0x37
31
? inet_dgram_connect
vmlinux 0xffffffff8d2dc214
Build id: 3.10.0-1160.el7.x86_64
0x34
32
? SYSC_connect
vmlinux 0xffffffff8d234910
Build id: 3.10.0-1160.el7.x86_64
0x90
33
__sys_sendmsg
vmlinux 0xffffffff8d236921
Build id: 3.10.0-1160.el7.x86_64
0x51
34
SyS_sendmsg
vmlinux 0xffffffff8d236972
Build id: 3.10.0-1160.el7.x86_64
0x12
35
system_call_fastpath
vmlinux 0xffffffff8d393f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 29s! [isc-worker0000:51810]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype tun br_netfilter devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus iosf_mbi snd_seq crc32_pclmul snd_seq_device ghash_clmulni_intel
snd_pcm ppdev aesni_intel lrw snd_timer gf128mul glue_helper vmw_balloon ablk_helper cryptd snd soundcore joydev pcspkr vmw_vmci sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix mptspi nfit scsi_transport_spi libata libnvdimm serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks fuse
CPU: 0 PID: 51810 Comm: isc-worker0000 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff95c561080000 ti: ffff95c567c68000 task.ti: ffff95c567c68000
RIP: 0010:[<ffffffffc02b3877>]  [<ffffffffc02b3877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff95c567c6b750  EFLAGS: 00000286
RAX: ffffbe4940403818 RBX: 0000000000000000 RCX: 00000000000000f7
RDX: 00000000000000f7 RSI: ffff95c5359c4140 RDI: ffff95c5769be000
RBP: ffff95c567c6b7d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: ffff95c567c6b7b8
R13: 0000400067c6b728 R14: ffffffffc08118dd R15: ffff95c567c6b6e8
FS:  00007fcf702aa700(0000) GS:ffff95c57b600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc8dc8c7fb8 CR3: 000000006103c000 CR4: 00000000003607f0
Call Trace:
[<ffffffff8d254386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff8d28242a>] sch_direct_xmit+0x11a/0x250
[<ffffffff8d257301>] __dev_queue_xmit+0x491/0x650
[<ffffffff8d2574d0>] dev_queue_xmit+0x10/0x20
[<ffffffff8d2a545b>] ip_finish_output+0x67b/0x8d0
[<ffffffff8d2a59db>] ip_output+0x7b/0xf0
[<ffffffff8d2a4de0>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff8d2a3407>] ip_local_out_sk+0x37/0x40
[<ffffffff8d2a6446>] ip_send_skb+0x16/0x50
[<ffffffff8d2ce85c>] udp_send_skb+0xac/0x2b0
[<ffffffff8d2a3140>] ? ip_copy_metadata+0x170/0x170
[<ffffffff8d2cfdc1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff8ccdfb48>] ? __enqueue_entity+0x78/0x80
[<ffffffff8ccd7202>] ? check_preempt_curr+0x92/0xa0
[<ffffffff8ccd7229>] ? ttwu_do_wakeup+0x19/0xe0
[<ffffffff8ccd735f>] ? ttwu_do_activate+0x6f/0x80
[<ffffffff8d2dd589>] inet_sendmsg+0x69/0xb0
[<ffffffff8d2343a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff8ccd2fd2>] ? __wake_up_common+0x82/0x120
[<ffffffff8d2344f5>] ? sock_recvmsg+0xc5/0x100
[<ffffffff8ce9d008>] ? ep_poll_callback+0xf8/0x220
[<ffffffff8d235269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff8d29bc63>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffff8d2cc28f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff8d38992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff8d2cf33f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff8d38992e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff8d239a18>] ? release_sock+0x128/0x180
[<ffffffff8d2cc6f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff8d2dc214>] ? inet_dgram_connect+0x34/0x90
[<ffffffff8d234910>] ? SYSC_connect+0x90/0x130
[<ffffffff8d236921>] __sys_sendmsg+0x51/0x90
[<ffffffff8d236972>] SyS_sendmsg+0x12/0x20
[<ffffffff8d393f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #207460
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc01a5867
Build id: 3.10.0-862.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff979eb166
Build id: 3.10.0-862.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff97a16a2a
Build id: 3.10.0-862.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff979ee051
Build id: 3.10.0-862.el7.x86_64
0x4a1
5
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc05f72ce
Build id: 3.10.0-862.el7.x86_64
0x4e
6
dev_queue_xmit
vmlinux 0xffffffff979ee220
Build id: 3.10.0-862.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffff97a37416
Build id: 3.10.0-862.el7.x86_64
0x546
8
ip_output
vmlinux 0xffffffff97a37973
Build id: 3.10.0-862.el7.x86_64
0x73
9
? __ip_append_data.isra.50
vmlinux 0xffffffff97a36ed0
Build id: 3.10.0-862.el7.x86_64
0xa50
10
ip_local_out_sk
vmlinux 0xffffffff97a35567
Build id: 3.10.0-862.el7.x86_64
0x37
11
ip_send_skb
vmlinux 0xffffffff97a383c6
Build id: 3.10.0-862.el7.x86_64
0x16
12
udp_send_skb
vmlinux 0xffffffff97a5fc4c
Build id: 3.10.0-862.el7.x86_64
0xac
13
? ip_copy_metadata
vmlinux 0xffffffff97a352b0
Build id: 3.10.0-862.el7.x86_64
0x170
14
udp_sendmsg
vmlinux 0xffffffff97a610fc
Build id: 3.10.0-862.el7.x86_64
0x32c
15
? poll_select_copy_remaining
vmlinux 0xffffffff97630830
Build id: 3.10.0-862.el7.x86_64
0x150
16
? poll_select_copy_remaining
vmlinux 0xffffffff97630830
Build id: 3.10.0-862.el7.x86_64
0x150
17
inet_sendmsg
vmlinux 0xffffffff97a6e5a9
Build id: 3.10.0-862.el7.x86_64
0x69
18
sock_sendmsg
vmlinux 0xffffffff979cc396
Build id: 3.10.0-862.el7.x86_64
0xb6
19
? unix_dgram_sendmsg
vmlinux 0xffffffff97aa4dab
Build id: 3.10.0-862.el7.x86_64
0x3cb
20
___sys_sendmsg
vmlinux 0xffffffff979cd199
Build id: 3.10.0-862.el7.x86_64
0x3a9
21
? ip4_datagram_release_cb
vmlinux 0xffffffff97a5d72f
Build id: 3.10.0-862.el7.x86_64
0x6f
22
? _raw_spin_unlock_bh
vmlinux 0xffffffff97b1551e
Build id: 3.10.0-862.el7.x86_64
0x1e
23
? udp_lib_rehash
vmlinux 0xffffffff97a6054f
Build id: 3.10.0-862.el7.x86_64
0x11f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff97b1551e
Build id: 3.10.0-862.el7.x86_64
0x1e
25
? release_sock
vmlinux 0xffffffff979d1660
Build id: 3.10.0-862.el7.x86_64
0x120
26
? ip4_datagram_connect
vmlinux 0xffffffff97a5db97
Build id: 3.10.0-862.el7.x86_64
0x37
27
? inet_dgram_connect
vmlinux 0xffffffff97a6d234
Build id: 3.10.0-862.el7.x86_64
0x34
28
? SYSC_connect
vmlinux 0xffffffff979cc93d
Build id: 3.10.0-862.el7.x86_64
0xed
29
__sys_sendmsg
vmlinux 0xffffffff979ce761
Build id: 3.10.0-862.el7.x86_64
0x51
30
SyS_sendmsg
vmlinux 0xffffffff979ce7b2
Build id: 3.10.0-862.el7.x86_64
0x12
31
system_call_fastpath
vmlinux 0xffffffff97b1f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 30s! [chronyd:798]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod ppdev coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd pcspkr joydev vmw_vmci sg i2c_piix4 shpchp nfit libnvdimm parport_pc parport sch_fq_codel
ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm e1000 mptspi ata_piix scsi_transport_spi crct10dif_pclmul crct10dif_common mptscsih crc32c_intel libata mptbase serio_raw i2c_core
CPU: 1 PID: 798 Comm: chronyd Not tainted 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffffa0d8f7755ee0 ti: ffffa0d8f764c000 task.ti: ffffa0d8f764c000
RIP: 0010:[<ffffffffc01a5867>]  [<ffffffffc01a5867>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffffa0d8f764f760  EFLAGS: 00000286
RAX: ffffabecc0703818 RBX: ffffffff979d58b3 RCX: 0000000000000028
RDX: 0000000000000028 RSI: ffffa0d875d0f3c0 RDI: ffffa0d975550000
RBP: ffffa0d8f764f7e0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffffffffc05bf8f5 R15: ffffa0d8f764f6f8
FS:  00007f83d19d7740(0000) GS:ffffa0d979640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fefe310c000 CR3: 00000000b75ec000 CR4: 00000000003607e0
Call Trace:
[<ffffffff979eb166>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff97a16a2a>] sch_direct_xmit+0x11a/0x250
[<ffffffff979ee051>] __dev_queue_xmit+0x4a1/0x660
[<ffffffffc05f72ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffff979ee220>] dev_queue_xmit+0x10/0x20
[<ffffffff97a37416>] ip_finish_output+0x546/0x7a0
[<ffffffff97a37973>] ip_output+0x73/0xe0
[<ffffffff97a36ed0>] ? __ip_append_data.isra.50+0xa50/0xa50
[<ffffffff97a35567>] ip_local_out_sk+0x37/0x40
[<ffffffff97a383c6>] ip_send_skb+0x16/0x50
[<ffffffff97a5fc4c>] udp_send_skb+0xac/0x2b0
[<ffffffff97a352b0>] ? ip_copy_metadata+0x170/0x170
[<ffffffff97a610fc>] udp_sendmsg+0x32c/0xa00
[<ffffffff97630830>] ? poll_select_copy_remaining+0x150/0x150
[<ffffffff97630830>] ? poll_select_copy_remaining+0x150/0x150
[<ffffffff97a6e5a9>] inet_sendmsg+0x69/0xb0
[<ffffffff979cc396>] sock_sendmsg+0xb6/0xf0
[<ffffffff97aa4dab>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffff979cd199>] ___sys_sendmsg+0x3a9/0x3c0
[<ffffffff97a5d72f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff97b1551e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff97a6054f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff97b1551e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff979d1660>] ? release_sock+0x120/0x170
[<ffffffff97a5db97>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff97a6d234>] ? inet_dgram_connect+0x34/0x90
[<ffffffff979cc93d>] ? SYSC_connect+0xed/0x130
[<ffffffff979ce761>] __sys_sendmsg+0x51/0x90
[<ffffffff979ce7b2>] SyS_sendmsg+0x12/0x20
[<ffffffff97b1f7d5>] system_call_fastpath+0x1c/0x21
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #117689
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0073ee7
Build id: 3.10.0-693.el7.x86_64
0x947
2
dev_hard_start_xmit
vmlinux 0xffffffff81586771
Build id: 3.10.0-693.el7.x86_64
0x171
3
sch_direct_xmit
vmlinux 0xffffffff815af50a
Build id: 3.10.0-693.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff81589566
Build id: 3.10.0-693.el7.x86_64
0x226
5
dev_queue_xmit
vmlinux 0xffffffff815898a0
Build id: 3.10.0-693.el7.x86_64
0x10
6
neigh_resolve_output
vmlinux 0xffffffff8159527b
Build id: 3.10.0-693.el7.x86_64
0x11b
7
ip_finish_output
vmlinux 0xffffffff815cfaa7
Build id: 3.10.0-693.el7.x86_64
0x297
8
ip_output
vmlinux 0xffffffff815d0293
Build id: 3.10.0-693.el7.x86_64
0x73
9
? __ip_append_data.isra.48
vmlinux 0xffffffff815cf810
Build id: 3.10.0-693.el7.x86_64
0xa00
10
ip_local_out_sk
vmlinux 0xffffffff815cdf31
Build id: 3.10.0-693.el7.x86_64
0x31
11
ip_send_skb
vmlinux 0xffffffff815d0cd6
Build id: 3.10.0-693.el7.x86_64
0x16
12
udp_send_skb
vmlinux 0xffffffff815f81b0
Build id: 3.10.0-693.el7.x86_64
0x160
13
? ip_copy_metadata
vmlinux 0xffffffff815cdc80
Build id: 3.10.0-693.el7.x86_64
0x170
14
udp_sendmsg
vmlinux 0xffffffff815f92cc
Build id: 3.10.0-693.el7.x86_64
0x32c
15
? poll_select_copy_remaining
vmlinux 0xffffffff81215dd0
Build id: 3.10.0-693.el7.x86_64
0x150
16
? poll_select_copy_remaining
vmlinux 0xffffffff81215dd0
Build id: 3.10.0-693.el7.x86_64
0x150
17
inet_sendmsg
vmlinux 0xffffffff81606893
Build id: 3.10.0-693.el7.x86_64
0x63
18
sock_sendmsg
vmlinux 0xffffffff8156a580
Build id: 3.10.0-693.el7.x86_64
0xb0
19
? unix_dgram_sendmsg
vmlinux 0xffffffff8163c506
Build id: 3.10.0-693.el7.x86_64
0x3c6
20
___sys_sendmsg
vmlinux 0xffffffff8156ae29
Build id: 3.10.0-693.el7.x86_64
0x3a9
21
? ip4_datagram_release_cb
vmlinux 0xffffffff815f5a29
Build id: 3.10.0-693.el7.x86_64
0x69
22
? _raw_spin_unlock_bh
vmlinux 0xffffffff816ab9ee
Build id: 3.10.0-693.el7.x86_64
0x1e
23
? udp_lib_rehash
vmlinux 0xffffffff815f7c3d
Build id: 3.10.0-693.el7.x86_64
0xfd
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff816ab9ee
Build id: 3.10.0-693.el7.x86_64
0x1e
25
? release_sock
vmlinux 0xffffffff8156e998
Build id: 3.10.0-693.el7.x86_64
0x118
26
? ip4_datagram_connect
vmlinux 0xffffffff815f5e87
Build id: 3.10.0-693.el7.x86_64
0x37
27
? inet_dgram_connect
vmlinux 0xffffffff8160556e
Build id: 3.10.0-693.el7.x86_64
0x2e
28
? SYSC_connect
vmlinux 0xffffffff8156a497
Build id: 3.10.0-693.el7.x86_64
0xe7
29
__sys_sendmsg
vmlinux 0xffffffff8156b5f1
Build id: 3.10.0-693.el7.x86_64
0x51
30
SyS_sendmsg
vmlinux 0xffffffff8156b642
Build id: 3.10.0-693.el7.x86_64
0x12
31
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 56s! [chronyd:1091]
Modules linked in: tcp_lp nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper snd_ens1371 cryptd vmw_balloon snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm joydev pcspkr snd_timer snd nfit soundcore sg libnvdimm parport_pc parport vmw_vmci i2c_piix4 shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif
crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul crct10dif_common fb_sys_fops ttm crc32c_intel drm ata_piix mptspi scsi_transport_spi libata mptscsih serio_raw e1000 mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 1091 Comm: chronyd Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff88013143dee0 ti: ffff880135a28000 task.ti: ffff880135a28000
RIP: 0010:[<ffffffffc0073ee7>]  [<ffffffffc0073ee7>] e1000_xmit_frame+0x947/0x1040 [e1000]
RSP: 0018:ffff880135a2b778  EFLAGS: 00000286
RAX: ffffc90000703818 RBX: ffff880135a2b718 RCX: 0000000000000008
RDX: 0000000000000008 RSI: ffff8800361290c0 RDI: ffff880135574000
RBP: ffff880135a2b7f8 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: ffff880135a2b740 R14: 0000000000000202 R15: 0000000000000202
FS:  00007fd0db3cf740(0000) GS:ffff880139600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006dde1c CR3: 0000000132c21000 CR4: 00000000000407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 ffff880136f56098 000000065198a7bd 0000000900000000
0000000000000200 ffff880100000000 ffff880100000000 0000000100001000
ffff8800b3e0c700 ffff880135574000 ffff880136f56000 ffff880135574000
Call Trace:
[<ffffffff81586771>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff815af50a>] sch_direct_xmit+0x11a/0x240
[<ffffffff81589566>] __dev_queue_xmit+0x226/0x550
[<ffffffff815898a0>] dev_queue_xmit+0x10/0x20
[<ffffffff8159527b>] neigh_resolve_output+0x11b/0x220
[<ffffffff815cfaa7>] ip_finish_output+0x297/0x780
[<ffffffff815d0293>] ip_output+0x73/0xe0
[<ffffffff815cf810>] ? __ip_append_data.isra.48+0xa00/0xa00
[<ffffffff815cdf31>] ip_local_out_sk+0x31/0x40
[<ffffffff815d0cd6>] ip_send_skb+0x16/0x50
[<ffffffff815f81b0>] udp_send_skb+0x160/0x290
[<ffffffff815cdc80>] ? ip_copy_metadata+0x170/0x170
[<ffffffff815f92cc>] udp_sendmsg+0x32c/0xa00
[<ffffffff81215dd0>] ? poll_select_copy_remaining+0x150/0x150
[<ffffffff81215dd0>] ? poll_select_copy_remaining+0x150/0x150
[<ffffffff81606893>] inet_sendmsg+0x63/0xb0
[<ffffffff8156a580>] sock_sendmsg+0xb0/0xf0
[<ffffffff8163c506>] ? unix_dgram_sendmsg+0x3c6/0x770
[<ffffffff8156ae29>] ___sys_sendmsg+0x3a9/0x3c0
[<ffffffff815f5a29>] ? ip4_datagram_release_cb+0x69/0x1b0
[<ffffffff816ab9ee>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff815f7c3d>] ? udp_lib_rehash+0xfd/0x130
[<ffffffff816ab9ee>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff8156e998>] ? release_sock+0x118/0x170
[<ffffffff815f5e87>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff8160556e>] ? inet_dgram_connect+0x2e/0x80
[<ffffffff8156a497>] ? SYSC_connect+0xe7/0x120
[<ffffffff8156b5f1>] __sys_sendmsg+0x51/0x90
[<ffffffff8156b642>] SyS_sendmsg+0x12/0x20
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #107043
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc04dbee7
Build id: 3.10.0-693.el7.x86_64
0x947
2
dev_hard_start_xmit
vmlinux 0xffffffff81586771
Build id: 3.10.0-693.el7.x86_64
0x171
3
sch_direct_xmit
vmlinux 0xffffffff815af50a
Build id: 3.10.0-693.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff81589566
Build id: 3.10.0-693.el7.x86_64
0x226
5
dev_queue_xmit
vmlinux 0xffffffff815898a0
Build id: 3.10.0-693.el7.x86_64
0x10
6
neigh_resolve_output
vmlinux 0xffffffff8159527b
Build id: 3.10.0-693.el7.x86_64
0x11b
7
ip_finish_output
vmlinux 0xffffffff815cfaa7
Build id: 3.10.0-693.el7.x86_64
0x297
8
ip_output
vmlinux 0xffffffff815d0293
Build id: 3.10.0-693.el7.x86_64
0x73
9
? __ip_append_data.isra.48
vmlinux 0xffffffff815cf810
Build id: 3.10.0-693.el7.x86_64
0xa00
10
ip_local_out_sk
vmlinux 0xffffffff815cdf31
Build id: 3.10.0-693.el7.x86_64
0x31
11
ip_send_skb
vmlinux 0xffffffff815d0cd6
Build id: 3.10.0-693.el7.x86_64
0x16
12
udp_send_skb
vmlinux 0xffffffff815f81b0
Build id: 3.10.0-693.el7.x86_64
0x160
13
? ip_copy_metadata
vmlinux 0xffffffff815cdc80
Build id: 3.10.0-693.el7.x86_64
0x170
14
udp_sendmsg
vmlinux 0xffffffff815f92cc
Build id: 3.10.0-693.el7.x86_64
0x32c
15
? default_wake_function
vmlinux 0xffffffff810c4822
Build id: 3.10.0-693.el7.x86_64
0x12
16
? sock_has_perm
vmlinux 0xffffffff812b4d65
Build id: 3.10.0-693.el7.x86_64
0x75
17
inet_sendmsg
vmlinux 0xffffffff81606893
Build id: 3.10.0-693.el7.x86_64
0x63
18
? selinux_socket_sendmsg
vmlinux 0xffffffff812b4e93
Build id: 3.10.0-693.el7.x86_64
0x23
19
sock_sendmsg
vmlinux 0xffffffff8156a580
Build id: 3.10.0-693.el7.x86_64
0xb0
20
? unix_dgram_sendmsg
vmlinux 0xffffffff8163c506
Build id: 3.10.0-693.el7.x86_64
0x3c6
21
___sys_sendmsg
vmlinux 0xffffffff8156ae29
Build id: 3.10.0-693.el7.x86_64
0x3a9
22
? ip4_datagram_release_cb
vmlinux 0xffffffff815f5a29
Build id: 3.10.0-693.el7.x86_64
0x69
23
? _raw_spin_unlock_bh
vmlinux 0xffffffff816ab9ee
Build id: 3.10.0-693.el7.x86_64
0x1e
24
? udp_lib_rehash
vmlinux 0xffffffff815f7c3d
Build id: 3.10.0-693.el7.x86_64
0xfd
25
? _raw_spin_unlock_bh
vmlinux 0xffffffff816ab9ee
Build id: 3.10.0-693.el7.x86_64
0x1e
26
? release_sock
vmlinux 0xffffffff8156e998
Build id: 3.10.0-693.el7.x86_64
0x118
27
? ip4_datagram_connect
vmlinux 0xffffffff815f5e87
Build id: 3.10.0-693.el7.x86_64
0x37
28
? inet_dgram_connect
vmlinux 0xffffffff8160556e
Build id: 3.10.0-693.el7.x86_64
0x2e
29
? SYSC_connect
vmlinux 0xffffffff8156a497
Build id: 3.10.0-693.el7.x86_64
0xe7
30
__sys_sendmsg
vmlinux 0xffffffff8156b5f1
Build id: 3.10.0-693.el7.x86_64
0x51
31
SyS_sendmsg
vmlinux 0xffffffff8156b642
Build id: 3.10.0-693.el7.x86_64
0x12
32
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 30s! [chronyd:655]
Modules linked in: usblp ext4 mbcache jbd2 nf_conntrack_ftp nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event ppdev coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel cryptd vmw_balloon snd_ens1371 snd_rawmidi snd_ac97_codec pcspkr joydev ac97_bus snd_seq snd_seq_device e1000 snd_pcm sg nfit snd_timer snd
soundcore libnvdimm parport_pc parport vmwgfx ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm mptspi mptscsih mptbase scsi_transport_spi i2c_piix4 shpchp i2c_core vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libata serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 655 Comm: chronyd Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff880035e0bf40 ti: ffff8800bb3a4000 task.ti: ffff8800bb3a4000
RIP: 0010:[<ffffffffc04dbee7>]  [<ffffffffc04dbee7>] e1000_xmit_frame+0x947/0x1040 [e1000]
RSP: 0018:ffff8800bb3a7778  EFLAGS: 00000286
RAX: ffffc90001203818 RBX: ffff8800bb3a7718 RCX: 000000000000009a
RDX: 000000000000009a RSI: ffff8800b8ca1140 RDI: ffff8801354da000
RBP: ffff8800bb3a77f8 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: ffff8800bb3a7740 R14: 0000000000000202 R15: 0000000000000202
FS:  00007f5bd4a36740(0000) GS:ffff880139600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9be9fb0000 CR3: 00000000baa3a000 CR4: 00000000003407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 ffff880136f98098 0000009827985a54 0000000900000000
0000000000000200 ffff880100000000 ffff880000000000 0000000100001000
ffff880070a00b00 ffff8801354da000 ffff880136f98000 ffff8801354da000
Call Trace:
[<ffffffff81586771>] dev_hard_start_xmit+0x171/0x3b0
[<ffffffff815af50a>] sch_direct_xmit+0x11a/0x240
[<ffffffff81589566>] __dev_queue_xmit+0x226/0x550
[<ffffffff815898a0>] dev_queue_xmit+0x10/0x20
[<ffffffff8159527b>] neigh_resolve_output+0x11b/0x220
[<ffffffff815cfaa7>] ip_finish_output+0x297/0x780
[<ffffffff815d0293>] ip_output+0x73/0xe0
[<ffffffff815cf810>] ? __ip_append_data.isra.48+0xa00/0xa00
[<ffffffff815cdf31>] ip_local_out_sk+0x31/0x40
[<ffffffff815d0cd6>] ip_send_skb+0x16/0x50
[<ffffffff815f81b0>] udp_send_skb+0x160/0x290
[<ffffffff815cdc80>] ? ip_copy_metadata+0x170/0x170
[<ffffffff815f92cc>] udp_sendmsg+0x32c/0xa00
[<ffffffff810c4822>] ? default_wake_function+0x12/0x20
[<ffffffff812b4d65>] ? sock_has_perm+0x75/0x90
[<ffffffff81606893>] inet_sendmsg+0x63/0xb0
[<ffffffff812b4e93>] ? selinux_socket_sendmsg+0x23/0x30
[<ffffffff8156a580>] sock_sendmsg+0xb0/0xf0
[<ffffffff8163c506>] ? unix_dgram_sendmsg+0x3c6/0x770
[<ffffffff8156ae29>] ___sys_sendmsg+0x3a9/0x3c0
[<ffffffff815f5a29>] ? ip4_datagram_release_cb+0x69/0x1b0
[<ffffffff816ab9ee>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff815f7c3d>] ? udp_lib_rehash+0xfd/0x130
[<ffffffff816ab9ee>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff8156e998>] ? release_sock+0x118/0x170
[<ffffffff815f5e87>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff8160556e>] ? inet_dgram_connect+0x2e/0x80
[<ffffffff8156a497>] ? SYSC_connect+0xe7/0x120
[<ffffffff8156b5f1>] __sys_sendmsg+0x51/0x90
[<ffffffff8156b642>] SyS_sendmsg+0x12/0x20
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #96221
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc01e3877
Build id: 3.10.0-1160.11.1.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffffa423c6f7
Build id: 3.10.0-1160.11.1.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffffa4253f96
Build id: 3.10.0-1160.11.1.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffffa428203a
Build id: 3.10.0-1160.11.1.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffffa4256f11
Build id: 3.10.0-1160.11.1.el7.x86_64
0x491
6
dev_queue_xmit
vmlinux 0xffffffffa42570e0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffffa42a507b
Build id: 3.10.0-1160.11.1.el7.x86_64
0x67b
8
ip_output
vmlinux 0xffffffffa42a55fb
Build id: 3.10.0-1160.11.1.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffffa42a4a00
Build id: 3.10.0-1160.11.1.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffffa42a3027
Build id: 3.10.0-1160.11.1.el7.x86_64
0x37
11
ip_send_skb
vmlinux 0xffffffffa42a6066
Build id: 3.10.0-1160.11.1.el7.x86_64
0x16
12
udp_send_skb
vmlinux 0xffffffffa42ce49c
Build id: 3.10.0-1160.11.1.el7.x86_64
0xac
13
? ip_copy_metadata
vmlinux 0xffffffffa42a2d60
Build id: 3.10.0-1160.11.1.el7.x86_64
0x170
14
udp_sendmsg
vmlinux 0xffffffffa42cfa01
Build id: 3.10.0-1160.11.1.el7.x86_64
0x5e1
15
? __pollwait
vmlinux 0xffffffffa3e64030
Build id: 3.10.0-1160.11.1.el7.x86_64
0xf0
16
? __pollwait
vmlinux 0xffffffffa3e64030
Build id: 3.10.0-1160.11.1.el7.x86_64
0xf0
17
inet_sendmsg
vmlinux 0xffffffffa42dd1c9
Build id: 3.10.0-1160.11.1.el7.x86_64
0x69
18
? enqueue_entity
vmlinux 0xffffffffa3ce684f
Build id: 3.10.0-1160.11.1.el7.x86_64
0x2ef
19
sock_sendmsg
vmlinux 0xffffffffa42343a6
Build id: 3.10.0-1160.11.1.el7.x86_64
0xb6
20
? check_preempt_curr
vmlinux 0xffffffffa3cd7452
Build id: 3.10.0-1160.11.1.el7.x86_64
0x92
21
? ttwu_do_wakeup
vmlinux 0xffffffffa3cd7479
Build id: 3.10.0-1160.11.1.el7.x86_64
0x19
22
___sys_sendmsg
vmlinux 0xffffffffa4235269
Build id: 3.10.0-1160.11.1.el7.x86_64
0x3e9
23
? __ip_route_output_key_hash
vmlinux 0xffffffffa429b883
Build id: 3.10.0-1160.11.1.el7.x86_64
0x223
24
? ip4_datagram_release_cb
vmlinux 0xffffffffa42cbecf
Build id: 3.10.0-1160.11.1.el7.x86_64
0x6f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffa438958e
Build id: 3.10.0-1160.11.1.el7.x86_64
0x1e
26
? udp_lib_rehash
vmlinux 0xffffffffa42cef7f
Build id: 3.10.0-1160.11.1.el7.x86_64
0x11f
27
? _raw_spin_unlock_bh
vmlinux 0xffffffffa438958e
Build id: 3.10.0-1160.11.1.el7.x86_64
0x1e
28
? release_sock
vmlinux 0xffffffffa4239a18
Build id: 3.10.0-1160.11.1.el7.x86_64
0x128
29
? ip4_datagram_connect
vmlinux 0xffffffffa42cc337
Build id: 3.10.0-1160.11.1.el7.x86_64
0x37
30
? inet_dgram_connect
vmlinux 0xffffffffa42dbe54
Build id: 3.10.0-1160.11.1.el7.x86_64
0x34
31
? SYSC_connect
vmlinux 0xffffffffa423496d
Build id: 3.10.0-1160.11.1.el7.x86_64
0xed
32
__sys_sendmsg
vmlinux 0xffffffffa4236921
Build id: 3.10.0-1160.11.1.el7.x86_64
0x51
33
SyS_sendmsg
vmlinux 0xffffffffa4236972
Build id: 3.10.0-1160.11.1.el7.x86_64
0x12
34
system_call_fastpath
vmlinux 0xffffffffa4393f92
Build id: 3.10.0-1160.11.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [chronyd:636]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371
snd_rawmidi snd_ac97_codec ac97_bus snd_seq pcspkr joydev ppdev i2c_piix4 snd_seq_device snd_pcm sg snd_timer snd vmw_balloon soundcore vmw_vmci parport_pc parport binfmt_misc ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper nfit syscopyarea sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm libnvdimm ata_piix drm mptspi scsi_transport_spi libata mptscsih e1000 mptbase serio_raw drm_panel_orientation_quirks fuse
CPU: 0 PID: 636 Comm: chronyd Tainted: G             L ------------ T 3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8cdeaf0ae300 ti: ffff8cdb3b9bc000 task.ti: ffff8cdb3b9bc000
RIP: 0010:[<ffffffffc01e3877>]  [<ffffffffc01e3877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8cdb3b9bf750  EFLAGS: 00000286
RAX: ffffa9c281983818 RBX: ffffffffa423ce73 RCX: 000000000000001b
RDX: 000000000000001b RSI: ffff8cdeaca7a140 RDI: ffff8cdbe53fc000
RBP: ffff8cdb3b9bf7d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 00000001c073d8dd R15: ffffffffa3fb1850
FS:  00007f00743a7740(0000) GS:ffff8cdead600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f961d570000 CR3: 00000000bb9a2000 CR4: 00000000003607f0
Call Trace:
[<ffffffffa423c6f7>] ? kfree_skbmem+0x37/0x90
[<ffffffffa4253f96>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa428203a>] sch_direct_xmit+0x11a/0x250
[<ffffffffa4256f11>] __dev_queue_xmit+0x491/0x650
[<ffffffffa42570e0>] dev_queue_xmit+0x10/0x20
[<ffffffffa42a507b>] ip_finish_output+0x67b/0x8d0
[<ffffffffa42a55fb>] ip_output+0x7b/0xf0
[<ffffffffa42a4a00>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffa42a3027>] ip_local_out_sk+0x37/0x40
[<ffffffffa42a6066>] ip_send_skb+0x16/0x50
[<ffffffffa42ce49c>] udp_send_skb+0xac/0x2b0
[<ffffffffa42a2d60>] ? ip_copy_metadata+0x170/0x170
[<ffffffffa42cfa01>] udp_sendmsg+0x5e1/0xa70
[<ffffffffa3e64030>] ? __pollwait+0xf0/0xf0
[<ffffffffa3e64030>] ? __pollwait+0xf0/0xf0
[<ffffffffa42dd1c9>] inet_sendmsg+0x69/0xb0
[<ffffffffa3ce684f>] ? enqueue_entity+0x2ef/0xbe0
[<ffffffffa42343a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffa3cd7452>] ? check_preempt_curr+0x92/0xa0
[<ffffffffa3cd7479>] ? ttwu_do_wakeup+0x19/0xe0
[<ffffffffa4235269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffa429b883>] ? __ip_route_output_key_hash+0x223/0x7f0
[<ffffffffa42cbecf>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffa438958e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa42cef7f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffa438958e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa4239a18>] ? release_sock+0x128/0x180
[<ffffffffa42cc337>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffa42dbe54>] ? inet_dgram_connect+0x34/0x90
[<ffffffffa423496d>] ? SYSC_connect+0xed/0x130
[<ffffffffa4236921>] __sys_sendmsg+0x51/0x90
[<ffffffffa4236972>] SyS_sendmsg+0x12/0x20
[<ffffffffa4393f92>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #46058
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc01a4877
Build id: 3.10.0-1062.9.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffa7e4f346
Build id: 3.10.0-1062.9.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffa7e7d0ba
Build id: 3.10.0-1062.9.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffa7e522b1
Build id: 3.10.0-1062.9.1.el7.x86_64
0x491
5
? local_bh_enable
vmlinux 0xffffffffa78a4787
Build id: 3.10.0-1062.9.1.el7.x86_64
0x17
6
dev_queue_xmit
vmlinux 0xffffffffa7e52480
Build id: 3.10.0-1062.9.1.el7.x86_64
0x10
7
neigh_resolve_output
vmlinux 0xffffffffa7e5efed
Build id: 3.10.0-1062.9.1.el7.x86_64
0x11d
8
ip_finish_output
vmlinux 0xffffffffa7ea057c
Build id: 3.10.0-1062.9.1.el7.x86_64
0x27c
9
ip_output
vmlinux 0xffffffffa7ea0efb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x7b
10
? __ip_append_data.isra.50
vmlinux 0xffffffffa7ea0300
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa60
11
ip_local_out_sk
vmlinux 0xffffffffa7e9e927
Build id: 3.10.0-1062.9.1.el7.x86_64
0x37
12
ip_send_skb
vmlinux 0xffffffffa7ea1966
Build id: 3.10.0-1062.9.1.el7.x86_64
0x16
13
udp_send_skb
vmlinux 0xffffffffa7ec9cec
Build id: 3.10.0-1062.9.1.el7.x86_64
0xac
14
? ip_copy_metadata
vmlinux 0xffffffffa7e9e660
Build id: 3.10.0-1062.9.1.el7.x86_64
0x170
15
udp_sendmsg
vmlinux 0xffffffffa7ecb251
Build id: 3.10.0-1062.9.1.el7.x86_64
0x5e1
16
? __pollwait
vmlinux 0xffffffffa7a606a0
Build id: 3.10.0-1062.9.1.el7.x86_64
0xf0
17
? __pollwait
vmlinux 0xffffffffa7a606a0
Build id: 3.10.0-1062.9.1.el7.x86_64
0xf0
18
inet_sendmsg
vmlinux 0xffffffffa7ed89b9
Build id: 3.10.0-1062.9.1.el7.x86_64
0x69
19
sock_sendmsg
vmlinux 0xffffffffa7e2f3a6
Build id: 3.10.0-1062.9.1.el7.x86_64
0xb6
20
? unix_dgram_sendmsg
vmlinux 0xffffffffa7f1075b
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3cb
21
___sys_sendmsg
vmlinux 0xffffffffa7e30269
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3e9
22
? __ip_route_output_key_hash
vmlinux 0xffffffffa7e97257
Build id: 3.10.0-1062.9.1.el7.x86_64
0x207
23
? ip4_datagram_release_cb
vmlinux 0xffffffffa7ec76ff
Build id: 3.10.0-1062.9.1.el7.x86_64
0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffa7f831ce
Build id: 3.10.0-1062.9.1.el7.x86_64
0x1e
25
? udp_lib_rehash
vmlinux 0xffffffffa7eca7cf
Build id: 3.10.0-1062.9.1.el7.x86_64
0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffffa7f831ce
Build id: 3.10.0-1062.9.1.el7.x86_64
0x1e
27
? release_sock
vmlinux 0xffffffffa7e34a10
Build id: 3.10.0-1062.9.1.el7.x86_64
0x120
28
? ip4_datagram_connect
vmlinux 0xffffffffa7ec7b67
Build id: 3.10.0-1062.9.1.el7.x86_64
0x37
29
? inet_dgram_connect
vmlinux 0xffffffffa7ed7644
Build id: 3.10.0-1062.9.1.el7.x86_64
0x34
30
? SYSC_connect
vmlinux 0xffffffffa7e2f96d
Build id: 3.10.0-1062.9.1.el7.x86_64
0xed
31
__sys_sendmsg
vmlinux 0xffffffffa7e31921
Build id: 3.10.0-1062.9.1.el7.x86_64
0x51
32
SyS_sendmsg
vmlinux 0xffffffffa7e31972
Build id: 3.10.0-1062.9.1.el7.x86_64
0x12
33
system_call_fastpath
vmlinux 0xffffffffa7f8dede
Build id: 3.10.0-1062.9.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 70s! [chronyd:892]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_nat nf_conntrack ipt_REJECT nf_reject_ipv4 br_netfilter tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock overlay(T) sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 sb_edac snd_rawmidi snd_ac97_codec coretemp ppdev iosf_mbi ac97_bus crc32_pclmul snd_seq snd_seq_device vmw_balloon snd_pcm ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_timer snd joydev soundcore pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c vmwgfx sr_mod cdrom ata_generic pata_acpi drm_kms_helper sd_mod crc_t10dif crct10dif_generic
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul libata crct10dif_common crc32c_intel mptspi scsi_transport_spi serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 892 Comm: chronyd Tainted: G             L ------------ T 3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/31/2013
task: ffff962eb5145230 ti: ffff962eb8ad8000 task.ti: ffff962eb8ad8000
RIP: 0010:[<ffffffffc01a4877>]  [<ffffffffc01a4877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff962eb8adb710  EFLAGS: 00000286
RAX: ffffb06880283818 RBX: ffffffffa7e37e63 RCX: 00000000000000e0
RDX: 00000000000000e0 RSI: ffff962e80089100 RDI: ffff962ebb35c000
RBP: ffff962eb8adb790 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffff962ebbdbdb00 R15: 0000000000010000
FS:  00007f94db776740(0000) GS:ffff962ebd600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f728a2a1740 CR3: 0000000038ad6000 CR4: 00000000001607f0
Call Trace:
[<ffffffffa7e4f346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa7e7d0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffffa7e522b1>] __dev_queue_xmit+0x491/0x650
[<ffffffffa78a4787>] ? local_bh_enable+0x17/0x20
[<ffffffffa7e52480>] dev_queue_xmit+0x10/0x20
[<ffffffffa7e5efed>] neigh_resolve_output+0x11d/0x220
[<ffffffffa7ea057c>] ip_finish_output+0x27c/0x8d0
[<ffffffffa7ea0efb>] ip_output+0x7b/0xf0
[<ffffffffa7ea0300>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffa7e9e927>] ip_local_out_sk+0x37/0x40
[<ffffffffa7ea1966>] ip_send_skb+0x16/0x50
[<ffffffffa7ec9cec>] udp_send_skb+0xac/0x2b0
[<ffffffffa7e9e660>] ? ip_copy_metadata+0x170/0x170
[<ffffffffa7ecb251>] udp_sendmsg+0x5e1/0xa70
[<ffffffffa7a606a0>] ? __pollwait+0xf0/0xf0
[<ffffffffa7a606a0>] ? __pollwait+0xf0/0xf0
[<ffffffffa7ed89b9>] inet_sendmsg+0x69/0xb0
[<ffffffffa7e2f3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffa7f1075b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffa7e30269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffa7e97257>] ? __ip_route_output_key_hash+0x207/0x7b0
[<ffffffffa7ec76ff>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffa7f831ce>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa7eca7cf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffa7f831ce>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa7e34a10>] ? release_sock+0x120/0x170
[<ffffffffa7ec7b67>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffa7ed7644>] ? inet_dgram_connect+0x34/0x90
[<ffffffffa7e2f96d>] ? SYSC_connect+0xed/0x130
[<ffffffffa7e31921>] __sys_sendmsg+0x51/0x90
[<ffffffffa7e31972>] SyS_sendmsg+0x12/0x20
[<ffffffffa7f8dede>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #24776
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00be877
Build id: 3.10.0-1127.19.1.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffff9fa53386
Build id: 3.10.0-1127.19.1.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffff9fa8131a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffff9fa56301
Build id: 3.10.0-1127.19.1.el7.x86_64
0x491
5
dev_queue_xmit
vmlinux 0xffffffff9fa564d0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x10
6
ip_finish_output
vmlinux 0xffffffff9faa429b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x67b
7
ip_output
vmlinux 0xffffffff9faa481b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x7b
8
? __ip_append_data.isra.50
vmlinux 0xffffffff9faa3c20
Build id: 3.10.0-1127.19.1.el7.x86_64
0xa60
9
ip_local_out_sk
vmlinux 0xffffffff9faa2247
Build id: 3.10.0-1127.19.1.el7.x86_64
0x37
10
ip_send_skb
vmlinux 0xffffffff9faa5286
Build id: 3.10.0-1127.19.1.el7.x86_64
0x16
11
udp_send_skb
vmlinux 0xffffffff9facd65c
Build id: 3.10.0-1127.19.1.el7.x86_64
0xac
12
? ip_copy_metadata
vmlinux 0xffffffff9faa1f80
Build id: 3.10.0-1127.19.1.el7.x86_64
0x170
13
udp_sendmsg
vmlinux 0xffffffff9facebc1
Build id: 3.10.0-1127.19.1.el7.x86_64
0x5e1
14
? __pollwait
vmlinux 0xffffffff9f663430
Build id: 3.10.0-1127.19.1.el7.x86_64
0xf0
15
? __pollwait
vmlinux 0xffffffff9f663430
Build id: 3.10.0-1127.19.1.el7.x86_64
0xf0
16
inet_sendmsg
vmlinux 0xffffffff9fadc389
Build id: 3.10.0-1127.19.1.el7.x86_64
0x69
17
? __pollwait
vmlinux 0xffffffff9f663430
Build id: 3.10.0-1127.19.1.el7.x86_64
0xf0
18
sock_sendmsg
vmlinux 0xffffffff9fa333a6
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb6
19
___sys_sendmsg
vmlinux 0xffffffff9fa34269
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3e9
20
? __ip_route_output_key_hash
vmlinux 0xffffffff9fa9aacf
Build id: 3.10.0-1127.19.1.el7.x86_64
0x20f
21
? ip4_datagram_release_cb
vmlinux 0xffffffff9facb06f
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6f
22
? _raw_spin_unlock_bh
vmlinux 0xffffffff9fb8856e
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1e
23
? udp_lib_rehash
vmlinux 0xffffffff9face13f
Build id: 3.10.0-1127.19.1.el7.x86_64
0x11f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff9fb8856e
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1e
25
? release_sock
vmlinux 0xffffffff9fa38a18
Build id: 3.10.0-1127.19.1.el7.x86_64
0x128
26
? ip4_datagram_connect
vmlinux 0xffffffff9facb4d7
Build id: 3.10.0-1127.19.1.el7.x86_64
0x37
27
? inet_dgram_connect
vmlinux 0xffffffff9fadb014
Build id: 3.10.0-1127.19.1.el7.x86_64
0x34
28
? SYSC_connect
vmlinux 0xffffffff9fa3396d
Build id: 3.10.0-1127.19.1.el7.x86_64
0xed
29
__sys_sendmsg
vmlinux 0xffffffff9fa35921
Build id: 3.10.0-1127.19.1.el7.x86_64
0x51
30
SyS_sendmsg
vmlinux 0xffffffff9fa35972
Build id: 3.10.0-1127.19.1.el7.x86_64
0x12
31
system_call_fastpath
vmlinux 0xffffffff9fb92ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
32
? system_call_after_swapgs
vmlinux 0xffffffff9fb92e15
Build id: 3.10.0-1127.19.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [chronyd:551]
Modules linked in: xt_time nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ip6table_filter ip6_tables tcp_diag inet_diag ipt_REJECT nf_reject_ipv4 xt_owner xt_multiport iptable_filter ext4 mbcache jbd2 loop dm_mirror dm_region_hash dm_log dm_mod sg pcspkr i2c_piix4 joydev ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper virtio_blk syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata e1000 virtio_pci serio_raw virtio_ring virtio drm_panel_orientation_quirks floppy
CPU: 4 PID: 551 Comm: chronyd Kdump: loaded Tainted: G        W    L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-48-gd9c812dda519-prebuilt.qemu.org 04/01/2014
task: ffff967f542020e0 ti: ffff967f54f3c000 task.ti: ffff967f54f3c000
RIP: 0010:[<ffffffffc00be877>]  [<ffffffffc00be877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff967f54f3f750  EFLAGS: 00000286
RAX: ffffafa181c43818 RBX: ffffffff9fa3be73 RCX: 00000000000000fa
RDX: 00000000000000fa RSI: ffff967e01debf40 RDI: ffff967f54f20000
RBP: ffff967f54f3f7d0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: ffff967f54f3f6c8 R14: ffff967f52412e88 R15: ffff967f54f3f6b8
FS:  00007f1ec4614740(0000) GS:ffff967f67d00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b2c9e67dc70 CR3: 0000000414168000 CR4: 00000000000006e0
Call Trace:
[<ffffffff9fa53386>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff9fa8131a>] sch_direct_xmit+0x11a/0x250
[<ffffffff9fa56301>] __dev_queue_xmit+0x491/0x650
[<ffffffff9fa564d0>] dev_queue_xmit+0x10/0x20
[<ffffffff9faa429b>] ip_finish_output+0x67b/0x8d0
[<ffffffff9faa481b>] ip_output+0x7b/0xf0
[<ffffffff9faa3c20>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffff9faa2247>] ip_local_out_sk+0x37/0x40
[<ffffffff9faa5286>] ip_send_skb+0x16/0x50
[<ffffffff9facd65c>] udp_send_skb+0xac/0x2b0
[<ffffffff9faa1f80>] ? ip_copy_metadata+0x170/0x170
[<ffffffff9facebc1>] udp_sendmsg+0x5e1/0xa70
[<ffffffff9f663430>] ? __pollwait+0xf0/0xf0
[<ffffffff9f663430>] ? __pollwait+0xf0/0xf0
[<ffffffff9fadc389>] inet_sendmsg+0x69/0xb0
[<ffffffff9f663430>] ? __pollwait+0xf0/0xf0
[<ffffffff9fa333a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff9fa34269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff9fa9aacf>] ? __ip_route_output_key_hash+0x20f/0x7d0
[<ffffffff9facb06f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff9fb8856e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff9face13f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff9fb8856e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff9fa38a18>] ? release_sock+0x128/0x180
[<ffffffff9facb4d7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff9fadb014>] ? inet_dgram_connect+0x34/0x90
[<ffffffff9fa3396d>] ? SYSC_connect+0xed/0x130
[<ffffffff9fa35921>] __sys_sendmsg+0x51/0x90
[<ffffffff9fa35972>] SyS_sendmsg+0x12/0x20
[<ffffffff9fb92ed2>] system_call_fastpath+0x25/0x2a
[<ffffffff9fb92e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #19313
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc04de857
Build id: 3.10.0-957.27.2.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffa683ace6
Build id: 3.10.0-957.27.2.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffa6867d6a
Build id: 3.10.0-957.27.2.el7.x86_64
0x11a
4
__dev_queue_xmit
vmlinux 0xffffffffa683dc61
Build id: 3.10.0-957.27.2.el7.x86_64
0x491
5
? local_bh_enable
vmlinux 0xffffffffa62a15d7
Build id: 3.10.0-957.27.2.el7.x86_64
0x17
6
dev_queue_xmit
vmlinux 0xffffffffa683de30
Build id: 3.10.0-957.27.2.el7.x86_64
0x10
7
neigh_resolve_output
vmlinux 0xffffffffa684a69d
Build id: 3.10.0-957.27.2.el7.x86_64
0x11d
8
ip_finish_output
vmlinux 0xffffffffa688a4dc
Build id: 3.10.0-957.27.2.el7.x86_64
0x27c
9
ip_output
vmlinux 0xffffffffa688ad13
Build id: 3.10.0-957.27.2.el7.x86_64
0x73
10
? __ip_append_data.isra.51
vmlinux 0xffffffffa688a260
Build id: 3.10.0-957.27.2.el7.x86_64
0xa50
11
ip_local_out_sk
vmlinux 0xffffffffa6888897
Build id: 3.10.0-957.27.2.el7.x86_64
0x37
12
ip_send_skb
vmlinux 0xffffffffa688b776
Build id: 3.10.0-957.27.2.el7.x86_64
0x16
13
udp_send_skb
vmlinux 0xffffffffa68b36ac
Build id: 3.10.0-957.27.2.el7.x86_64
0xac
14
? ip_copy_metadata
vmlinux 0xffffffffa68885e0
Build id: 3.10.0-957.27.2.el7.x86_64
0x170
15
udp_sendmsg
vmlinux 0xffffffffa68b4995
Build id: 3.10.0-957.27.2.el7.x86_64
0x335
16
? __pollwait
vmlinux 0xffffffffa64583a0
Build id: 3.10.0-957.27.2.el7.x86_64
0xf0
17
? __pollwait
vmlinux 0xffffffffa64583a0
Build id: 3.10.0-957.27.2.el7.x86_64
0xf0
18
inet_sendmsg
vmlinux 0xffffffffa68c2199
Build id: 3.10.0-957.27.2.el7.x86_64
0x69
19
sock_sendmsg
vmlinux 0xffffffffa681b3a6
Build id: 3.10.0-957.27.2.el7.x86_64
0xb6
20
___sys_sendmsg
vmlinux 0xffffffffa681c269
Build id: 3.10.0-957.27.2.el7.x86_64
0x3e9
21
? ip4_datagram_release_cb
vmlinux 0xffffffffa68b112f
Build id: 3.10.0-957.27.2.el7.x86_64
0x6f
22
? _raw_spin_unlock_bh
vmlinux 0xffffffffa696c63e
Build id: 3.10.0-957.27.2.el7.x86_64
0x1e
23
? udp_lib_rehash
vmlinux 0xffffffffa68b3faf
Build id: 3.10.0-957.27.2.el7.x86_64
0x11f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffffa696c63e
Build id: 3.10.0-957.27.2.el7.x86_64
0x1e
25
? release_sock
vmlinux 0xffffffffa6820a00
Build id: 3.10.0-957.27.2.el7.x86_64
0x120
26
? ip4_datagram_connect
vmlinux 0xffffffffa68b1597
Build id: 3.10.0-957.27.2.el7.x86_64
0x37
27
? inet_dgram_connect
vmlinux 0xffffffffa68c0e24
Build id: 3.10.0-957.27.2.el7.x86_64
0x34
28
? SYSC_connect
vmlinux 0xffffffffa681b96d
Build id: 3.10.0-957.27.2.el7.x86_64
0xed
29
__sys_sendmsg
vmlinux 0xffffffffa681d921
Build id: 3.10.0-957.27.2.el7.x86_64
0x51
30
SyS_sendmsg
vmlinux 0xffffffffa681d972
Build id: 3.10.0-957.27.2.el7.x86_64
0x12
31
system_call_fastpath
vmlinux 0xffffffffa6976ddb
Build id: 3.10.0-957.27.2.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 47s! [vmtoolsd:1012]
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 47s! [chronyd:1033]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq ppdev snd_seq_device iosf_mbi snd_pcm crc32_pclmul ghash_clmulni_intel aesni_intel vmw_balloon lrw snd_timer gf128mul glue_helper ablk_helper cryptd snd soundcore pcspkr sg joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
ttm ata_piix drm crct10dif_pclmul crct10dif_common libata crc32c_intel e1000 nfit libnvdimm serio_raw mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1033 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-957.27.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff92a5f22730c0 ti: ffff92a577268000 task.ti: ffff92a577268000
RIP: 0010:[<ffffffffc04de857>]  [<ffffffffc04de857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff92a57726b720  EFLAGS: 00000286
RAX: ffffaf7640903818 RBX: ffffffffa6824c73 RCX: 000000000000001d
RDX: 000000000000001d RSI: ffff92a4f567e140 RDI: ffff92a5f8bf4000
RBP: ffff92a57726b7a0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: 0000000100000000 R15: ffffffffa65a23e0
FS:  00007f3bf37ee740(0000) GS:ffff92a5fa640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f56cba03000 CR3: 00000000b7120000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa683ace6>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa6867d6a>] sch_direct_xmit+0x11a/0x250
[<ffffffffa683dc61>] __dev_queue_xmit+0x491/0x650
[<ffffffffa62a15d7>] ? local_bh_enable+0x17/0x20
[<ffffffffa683de30>] dev_queue_xmit+0x10/0x20
[<ffffffffa684a69d>] neigh_resolve_output+0x11d/0x220
[<ffffffffa688a4dc>] ip_finish_output+0x27c/0x7b0
[<ffffffffa688ad13>] ip_output+0x73/0xe0
[<ffffffffa688a260>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffffa6888897>] ip_local_out_sk+0x37/0x40
[<ffffffffa688b776>] ip_send_skb+0x16/0x50
[<ffffffffa68b36ac>] udp_send_skb+0xac/0x2b0
[<ffffffffa68885e0>] ? ip_copy_metadata+0x170/0x170
[<ffffffffa68b4995>] udp_sendmsg+0x335/0xa10
[<ffffffffa64583a0>] ? __pollwait+0xf0/0xf0
[<ffffffffa64583a0>] ? __pollwait+0xf0/0xf0
[<ffffffffa68c2199>] inet_sendmsg+0x69/0xb0
[<ffffffffa681b3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffa681c269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffa68b112f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffa696c63e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa68b3faf>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffa696c63e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa6820a00>] ? release_sock+0x120/0x170
[<ffffffffa68b1597>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffa68c0e24>] ? inet_dgram_connect+0x34/0x90
[<ffffffffa681b96d>] ? SYSC_connect+0xed/0x130
[<ffffffffa681d921>] __sys_sendmsg+0x51/0x90
[<ffffffffa681d972>] SyS_sendmsg+0x12/0x20
[<ffffffffa6976ddb>] system_call_fastpath+0x22/0x27
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #6725
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc00d8877
Build id: 3.10.0-1062.1.2.el7.x86_64
0x7f7
2
dev_hard_start_xmit
vmlinux 0xffffffffa304d346
Build id: 3.10.0-1062.1.2.el7.x86_64
0x246
3
sch_direct_xmit
vmlinux 0xffffffffa307b0ba
Build id: 3.10.0-1062.1.2.el7.x86_64
0x11a
4
__qdisc_run
vmlinux 0xffffffffa307b27e
Build id: 3.10.0-1062.1.2.el7.x86_64
0x8e
5
__dev_queue_xmit
vmlinux 0xffffffffa30502c9
Build id: 3.10.0-1062.1.2.el7.x86_64
0x4a9
6
dev_queue_xmit
vmlinux 0xffffffffa3050480
Build id: 3.10.0-1062.1.2.el7.x86_64
0x10
7
ip_finish_output
vmlinux 0xffffffffa309e97b
Build id: 3.10.0-1062.1.2.el7.x86_64
0x67b
8
ip_output
vmlinux 0xffffffffa309eefb
Build id: 3.10.0-1062.1.2.el7.x86_64
0x7b
9
? __ip_append_data.isra.50
vmlinux 0xffffffffa309e300
Build id: 3.10.0-1062.1.2.el7.x86_64
0xa60
10
ip_local_out_sk
vmlinux 0xffffffffa309c927
Build id: 3.10.0-1062.1.2.el7.x86_64
0x37
11
ip_send_skb
vmlinux 0xffffffffa309f966
Build id: 3.10.0-1062.1.2.el7.x86_64
0x16
12
udp_send_skb
vmlinux 0xffffffffa30c7ccc
Build id: 3.10.0-1062.1.2.el7.x86_64
0xac
13
? ip_copy_metadata
vmlinux 0xffffffffa309c660
Build id: 3.10.0-1062.1.2.el7.x86_64
0x170
14
udp_sendmsg
vmlinux 0xffffffffa30c9231
Build id: 3.10.0-1062.1.2.el7.x86_64
0x5e1
15
? ep_poll_callback
vmlinux 0xffffffffa2c96f4e
Build id: 3.10.0-1062.1.2.el7.x86_64
0xee
16
? __wake_up_common
vmlinux 0xffffffffa2ad24d2
Build id: 3.10.0-1062.1.2.el7.x86_64
0x82
17
inet_sendmsg
vmlinux 0xffffffffa30d6999
Build id: 3.10.0-1062.1.2.el7.x86_64
0x69
18
sock_sendmsg
vmlinux 0xffffffffa302d3a6
Build id: 3.10.0-1062.1.2.el7.x86_64
0xb6
19
? unix_dgram_sendmsg
vmlinux 0xffffffffa310e73b
Build id: 3.10.0-1062.1.2.el7.x86_64
0x3cb
20
___sys_sendmsg
vmlinux 0xffffffffa302e269
Build id: 3.10.0-1062.1.2.el7.x86_64
0x3e9
21
? __ip_route_output_key_hash
vmlinux 0xffffffffa3095257
Build id: 3.10.0-1062.1.2.el7.x86_64
0x207
22
? ip4_datagram_release_cb
vmlinux 0xffffffffa30c56df
Build id: 3.10.0-1062.1.2.el7.x86_64
0x6f
23
? _raw_spin_unlock_bh
vmlinux 0xffffffffa31819ce
Build id: 3.10.0-1062.1.2.el7.x86_64
0x1e
24
? udp_lib_rehash
vmlinux 0xffffffffa30c87af
Build id: 3.10.0-1062.1.2.el7.x86_64
0x11f
25
? _raw_spin_unlock_bh
vmlinux 0xffffffffa31819ce
Build id: 3.10.0-1062.1.2.el7.x86_64
0x1e
26
? release_sock
vmlinux 0xffffffffa3032a10
Build id: 3.10.0-1062.1.2.el7.x86_64
0x120
27
? ip4_datagram_connect
vmlinux 0xffffffffa30c5b47
Build id: 3.10.0-1062.1.2.el7.x86_64
0x37
28
? inet_dgram_connect
vmlinux 0xffffffffa30d5624
Build id: 3.10.0-1062.1.2.el7.x86_64
0x34
29
? SYSC_connect
vmlinux 0xffffffffa302d96d
Build id: 3.10.0-1062.1.2.el7.x86_64
0xed
30
__sys_sendmsg
vmlinux 0xffffffffa302f921
Build id: 3.10.0-1062.1.2.el7.x86_64
0x51
31
SyS_sendmsg
vmlinux 0xffffffffa302f972
Build id: 3.10.0-1062.1.2.el7.x86_64
0x12
32
system_call_fastpath
vmlinux 0xffffffffa318cede
Build id: 3.10.0-1062.1.2.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 30s! [chronyd:1100]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) tcp_lp nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink binfmt_misc vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq coretemp snd_seq_device snd_pcm ppdev iosf_mbi snd_timer crc32_pclmul ghash_clmulni_intel aesni_intel snd lrw vmw_balloon gf128mul glue_helper ablk_helper cryptd soundcore joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci sch_fq_codel ip_tables xfs libcrc32c
sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops crct10dif_pclmul crct10dif_common ttm ata_piix mptspi nfit crc32c_intel drm scsi_transport_spi libnvdimm mptscsih mptbase libata e1000 serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler]
CPU: 0 PID: 1100 Comm: chronyd Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
task: ffff8ca5f4bed230 ti: ffff8ca5eabb8000 task.ti: ffff8ca5eabb8000
RIP: 0010:[<ffffffffc00d8877>]  [<ffffffffc00d8877>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8ca5eabbb6f8  EFLAGS: 00000286
RAX: ffff985240d43818 RBX: ffff8ca5eda5e300 RCX: 0000000000000076
RDX: 0000000000000076 RSI: ffff8ca3f5e27b40 RDI: ffff8ca5f4c16000
RBP: ffff8ca5eabbb778 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000080000000 R11: 0000000000000000 R12: ffffffffa30dd313
R13: ffff8ca5eabbb6e8 R14: ffffffffa3712a40 R15: 0000000000000000
FS:  00007f1a23cb1740(0000) GS:ffff8ca5f5600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4a76819680 CR3: 0000000229a70000 CR4: 00000000001607f0
Call Trace:
[<ffffffffa304d346>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffffa307b0ba>] sch_direct_xmit+0x11a/0x250
[<ffffffffa307b27e>] __qdisc_run+0x8e/0x360
[<ffffffffa30502c9>] __dev_queue_xmit+0x4a9/0x650
[<ffffffffa3050480>] dev_queue_xmit+0x10/0x20
[<ffffffffa309e97b>] ip_finish_output+0x67b/0x8d0
[<ffffffffa309eefb>] ip_output+0x7b/0xf0
[<ffffffffa309e300>] ? __ip_append_data.isra.50+0xa60/0xa60
[<ffffffffa309c927>] ip_local_out_sk+0x37/0x40
[<ffffffffa309f966>] ip_send_skb+0x16/0x50
[<ffffffffa30c7ccc>] udp_send_skb+0xac/0x2b0
[<ffffffffa309c660>] ? ip_copy_metadata+0x170/0x170
[<ffffffffa30c9231>] udp_sendmsg+0x5e1/0xa70
[<ffffffffa2c96f4e>] ? ep_poll_callback+0xee/0x210
[<ffffffffa2ad24d2>] ? __wake_up_common+0x82/0x120
[<ffffffffa30d6999>] inet_sendmsg+0x69/0xb0
[<ffffffffa302d3a6>] sock_sendmsg+0xb6/0xf0
[<ffffffffa310e73b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffffa302e269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffffa3095257>] ? __ip_route_output_key_hash+0x207/0x7b0
[<ffffffffa30c56df>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffffa31819ce>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa30c87af>] ? udp_lib_rehash+0x11f/0x150
[<ffffffffa31819ce>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffffa3032a10>] ? release_sock+0x120/0x170
[<ffffffffa30c5b47>] ? ip4_datagram_connect+0x37/0x50
[<ffffffffa30d5624>] ? inet_dgram_connect+0x34/0x90
[<ffffffffa302d96d>] ? SYSC_connect+0xed/0x130
[<ffffffffa302f921>] __sys_sendmsg+0x51/0x90
[<ffffffffa302f972>] SyS_sendmsg+0x12/0x20
[<ffffffffa318cede>] system_call_fastpath+0x25/0x2a
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00 
Complete report #326
Frame # Function Binary Source or offset Line
1
e1000_xmit_frame
e1000 0xffffffffc0418857
Build id: 3.10.0-957.el7.x86_64
0x7f7
2
? kfree_skbmem
vmlinux 0xffffffff85c21697
Build id: 3.10.0-957.el7.x86_64
0x37
3
dev_hard_start_xmit
vmlinux 0xffffffff85c38c66
Build id: 3.10.0-957.el7.x86_64
0x246
4
sch_direct_xmit
vmlinux 0xffffffff85c65cea
Build id: 3.10.0-957.el7.x86_64
0x11a
5
__dev_queue_xmit
vmlinux 0xffffffff85c3bbe1
Build id: 3.10.0-957.el7.x86_64
0x491
6
? ipv4_confirm
nf_conntrack_ipv4 0xffffffffc08f72ce
Build id: 3.10.0-957.el7.x86_64
0x4e
7
dev_queue_xmit
vmlinux 0xffffffff85c3bdb0
Build id: 3.10.0-957.el7.x86_64
0x10
8
ip_finish_output
vmlinux 0xffffffff85c88736
Build id: 3.10.0-957.el7.x86_64
0x556
9
ip_output
vmlinux 0xffffffff85c88c93
Build id: 3.10.0-957.el7.x86_64
0x73
10
? __ip_append_data.isra.51
vmlinux 0xffffffff85c881e0
Build id: 3.10.0-957.el7.x86_64
0xa50
11
ip_local_out_sk
vmlinux 0xffffffff85c86817
Build id: 3.10.0-957.el7.x86_64
0x37
12
ip_send_skb
vmlinux 0xffffffff85c896f6
Build id: 3.10.0-957.el7.x86_64
0x16
13
udp_send_skb
vmlinux 0xffffffff85cb150c
Build id: 3.10.0-957.el7.x86_64
0xac
14
? ip_copy_metadata
vmlinux 0xffffffff85c86560
Build id: 3.10.0-957.el7.x86_64
0x170
15
udp_sendmsg
vmlinux 0xffffffff85cb27f5
Build id: 3.10.0-957.el7.x86_64
0x335
16
? __pollwait
vmlinux 0xffffffff85856d70
Build id: 3.10.0-957.el7.x86_64
0xf0
17
? __pollwait
vmlinux 0xffffffff85856d70
Build id: 3.10.0-957.el7.x86_64
0xf0
18
inet_sendmsg
vmlinux 0xffffffff85cbfeb9
Build id: 3.10.0-957.el7.x86_64
0x69
19
? __pollwait
vmlinux 0xffffffff85856d70
Build id: 3.10.0-957.el7.x86_64
0xf0
20
sock_sendmsg
vmlinux 0xffffffff85c193a6
Build id: 3.10.0-957.el7.x86_64
0xb6
21
? unix_dgram_sendmsg
vmlinux 0xffffffff85cf769b
Build id: 3.10.0-957.el7.x86_64
0x3cb
22
___sys_sendmsg
vmlinux 0xffffffff85c1a269
Build id: 3.10.0-957.el7.x86_64
0x3e9
23
? ip4_datagram_release_cb
vmlinux 0xffffffff85caef8f
Build id: 3.10.0-957.el7.x86_64
0x6f
24
? _raw_spin_unlock_bh
vmlinux 0xffffffff85d6a29e
Build id: 3.10.0-957.el7.x86_64
0x1e
25
? udp_lib_rehash
vmlinux 0xffffffff85cb1e0f
Build id: 3.10.0-957.el7.x86_64
0x11f
26
? _raw_spin_unlock_bh
vmlinux 0xffffffff85d6a29e
Build id: 3.10.0-957.el7.x86_64
0x1e
27
? release_sock
vmlinux 0xffffffff85c1e980
Build id: 3.10.0-957.el7.x86_64
0x120
28
? ip4_datagram_connect
vmlinux 0xffffffff85caf3f7
Build id: 3.10.0-957.el7.x86_64
0x37
29
? inet_dgram_connect
vmlinux 0xffffffff85cbeb44
Build id: 3.10.0-957.el7.x86_64
0x34
30
? SYSC_connect
vmlinux 0xffffffff85c1996d
Build id: 3.10.0-957.el7.x86_64
0xed
31
__sys_sendmsg
vmlinux 0xffffffff85c1b921
Build id: 3.10.0-957.el7.x86_64
0x51
32
SyS_sendmsg
vmlinux 0xffffffff85c1b972
Build id: 3.10.0-957.el7.x86_64
0x12
33
system_call_fastpath
vmlinux 0xffffffff85d74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [chronyd:8863]
Modules linked in: tcp_lp nls_utf8 isofs fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vmw_vmci vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore joydev pcspkr sg parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_pclmul crct10dif_common crc32c_intel ata_generic
pata_acpi serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi scsi_transport_spi e1000 libata mptscsih mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 8863 Comm: chronyd Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff8e34701fb0c0 ti: ffff8e3473030000 task.ti: ffff8e3473030000
RIP: 0010:[<ffffffffc0418857>]  [<ffffffffc0418857>] e1000_xmit_frame+0x7f7/0x1120 [e1000]
RSP: 0018:ffff8e3473033760  EFLAGS: 00000286
RAX: ffffb9dec0f83818 RBX: ffffffff85c22bf3 RCX: 0000000000000090
RDX: 0000000000000090 RSI: ffff8e346f6ffcc0 RDI: ffff8e3475e24000
RBP: ffff8e34730337e0 R08: 0000000000000100 R09: 0000000022100000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000246
R13: 0000000000000246 R14: ffffffffc08d1915 R15: ffff8e34730336f8
FS:  00007f6a313ec740(0000) GS:ffff8e3478640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f218d2a1000 CR3: 0000000170c0a000 CR4: 00000000003607e0
Call Trace:
[<ffffffff85c21697>] ? kfree_skbmem+0x37/0x90
[<ffffffff85c38c66>] dev_hard_start_xmit+0x246/0x3b0
[<ffffffff85c65cea>] sch_direct_xmit+0x11a/0x250
[<ffffffff85c3bbe1>] __dev_queue_xmit+0x491/0x650
[<ffffffffc08f72ce>] ? ipv4_confirm+0x4e/0x100 [nf_conntrack_ipv4]
[<ffffffff85c3bdb0>] dev_queue_xmit+0x10/0x20
[<ffffffff85c88736>] ip_finish_output+0x556/0x7b0
[<ffffffff85c88c93>] ip_output+0x73/0xe0
[<ffffffff85c881e0>] ? __ip_append_data.isra.51+0xa50/0xa50
[<ffffffff85c86817>] ip_local_out_sk+0x37/0x40
[<ffffffff85c896f6>] ip_send_skb+0x16/0x50
[<ffffffff85cb150c>] udp_send_skb+0xac/0x2b0
[<ffffffff85c86560>] ? ip_copy_metadata+0x170/0x170
[<ffffffff85cb27f5>] udp_sendmsg+0x335/0xa10
[<ffffffff85856d70>] ? __pollwait+0xf0/0xf0
[<ffffffff85856d70>] ? __pollwait+0xf0/0xf0
[<ffffffff85cbfeb9>] inet_sendmsg+0x69/0xb0
[<ffffffff85856d70>] ? __pollwait+0xf0/0xf0
[<ffffffff85c193a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff85cf769b>] ? unix_dgram_sendmsg+0x3cb/0x770
[<ffffffff85c1a269>] ___sys_sendmsg+0x3e9/0x400
[<ffffffff85caef8f>] ? ip4_datagram_release_cb+0x6f/0x1c0
[<ffffffff85d6a29e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff85cb1e0f>] ? udp_lib_rehash+0x11f/0x150
[<ffffffff85d6a29e>] ? _raw_spin_unlock_bh+0x1e/0x20
[<ffffffff85c1e980>] ? release_sock+0x120/0x170
[<ffffffff85caf3f7>] ? ip4_datagram_connect+0x37/0x50
[<ffffffff85cbeb44>] ? inet_dgram_connect+0x34/0x90
[<ffffffff85c1996d>] ? SYSC_connect+0xed/0x130
[<ffffffff85c1b921>] __sys_sendmsg+0x51/0x90
[<ffffffff85c1b972>] SyS_sendmsg+0x12/0x20
[<ffffffff85d74ddb>] system_call_fastpath+0x22/0x27
Code: 00 00 20 74 10 48 8b 83 00 03 00 00 f6 80 98 00 00 00 03 74 16 48 8b 5d c8 41 0f b7 47 2a 41 8b 57 18 48 03 83 90 0c 00 00 89 10 <48> 83 c4 58 31 c0 5b 41 5c 41 5d 41 5e 41 5f 5d c3 0f 1f 84 00