Info

Permalink
Function
smp_call_function_many
First occurrence
2020-09-22
Last occurrence
2025-06-18
State
NEW
Components
kernel
Quality
-48

Statistics

Operating system Count
CentOS 7 625
Architecture Count
x86_64 625
Related packages Count
kernel 625
 0:3.10.0-693.5.2.el7 278
 0:3.10.0-693.el7 226
 0:3.10.0-514.el7 50
 0:3.10.0-514.21.1.el7 23
 0:3.10.0-693.21.1.el7 16
 0:3.10.0-693.11.1.el7 15
 0:3.10.0-514.26.2.el7 4
 0:3.10.0-514.10.2.el7 3
 0:3.10.0-514.21.2.el7 2
 0:3.10.0-514.16.1.el7 1
 0:3.10.0-693.2.2.el7 1
 0:3.10.0-1062.el7 1
 0:3.10.0-1127.19.1.el7 1
 0:3.10.0-1160.83.1.el7 1
 0:3.10.0-957.21.3.el7 1
 0:3.10.0-957.el7 1
 0:3.10.0-693.17.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #1170709
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
? __netif_receive_skb_core
vmlinux 0xffffffff815869df
Build id: 3.10.0-693.el7.x86_64
0x2f
3
? setup_data_read
vmlinux 0xffffffff81031c80
Build id: 3.10.0-693.el7.x86_64
0x90
4
? __netif_receive_skb_core
vmlinux 0xffffffff815869e0
Build id: 3.10.0-693.el7.x86_64
0x30
5
on_each_cpu
vmlinux 0xffffffff810f9a5d
Build id: 3.10.0-693.el7.x86_64
0x2d
6
? __netif_receive_skb_core
vmlinux 0xffffffff815869df
Build id: 3.10.0-693.el7.x86_64
0x2f
7
text_poke_bp
vmlinux 0xffffffff810324d4
Build id: 3.10.0-693.el7.x86_64
0x94
8
arch_jump_label_transform
vmlinux 0xffffffff8102f558
Build id: 3.10.0-693.el7.x86_64
0x68
9
__jump_label_update
vmlinux 0xffffffff8117fb8f
Build id: 3.10.0-693.el7.x86_64
0x5f
10
jump_label_update
vmlinux 0xffffffff8117fc4d
Build id: 3.10.0-693.el7.x86_64
0x9d
11
__static_key_slow_dec
vmlinux 0xffffffff8117fefd
Build id: 3.10.0-693.el7.x86_64
0x4d
12
static_key_slow_dec
vmlinux 0xffffffff8117ff82
Build id: 3.10.0-693.el7.x86_64
0x22
13
net_disable_timestamp
vmlinux 0xffffffff81583c48
Build id: 3.10.0-693.el7.x86_64
0x38
14
sock_disable_timestamp
vmlinux 0xffffffff8156c675
Build id: 3.10.0-693.el7.x86_64
0x45
15
__sk_free
vmlinux 0xffffffff8156d5d2
Build id: 3.10.0-693.el7.x86_64
0x62
16
sk_free
vmlinux 0xffffffff8156d6f1
Build id: 3.10.0-693.el7.x86_64
0x21
17
sk_common_release
vmlinux 0xffffffff8156d7ae
Build id: 3.10.0-693.el7.x86_64
0xae
18
udp_lib_close
vmlinux 0xffffffff815f8b59
Build id: 3.10.0-693.el7.x86_64
0x9
19
inet_release
vmlinux 0xffffffff816052b7
Build id: 3.10.0-693.el7.x86_64
0x77
20
sock_release
vmlinux 0xffffffff8156999f
Build id: 3.10.0-693.el7.x86_64
0x1f
21
sock_close
vmlinux 0xffffffff81569a12
Build id: 3.10.0-693.el7.x86_64
0x12
22
__fput
vmlinux 0xffffffff81202fb9
Build id: 3.10.0-693.el7.x86_64
0xe9
23
____fput
vmlinux 0xffffffff8120321e
Build id: 3.10.0-693.el7.x86_64
0xe
24
task_work_run
vmlinux 0xffffffff810ad247
Build id: 3.10.0-693.el7.x86_64
0xa7
25
do_notify_resume
vmlinux 0xffffffff8102ab62
Build id: 3.10.0-693.el7.x86_64
0x92
26
int_signal
vmlinux 0xffffffff816b527d
Build id: 3.10.0-693.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 28s! [chronyd:769]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) vmhgfs(OE) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 crc32_pclmul ghash_clmulni_intel ppdev snd_ens1371 snd_rawmidi aesni_intel lrw gf128mul glue_helper snd_ac97_codec ablk_helper cryptd ac97_bus vmw_balloon snd_seq snd_seq_device snd_pcm pcspkr joydev snd_timer snd sg nfit soundcore vmw_vmci libnvdimm i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix serio_raw e1000 libata mptspi scsi_transport_spi mptscsih mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 769 Comm: chronyd Tainted: G           OEL ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8800bb748000 ti: ffff8802341d0000 task.ti: ffff8802341d0000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8802341d3c40  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000c000081031c80 RCX: ffff88023561af70
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8802341d3c78 R08: ffff88017cf53c00 R09: ffffffff813227d9
R10: ffff8802356d9c80 R11: ffffea0000d33400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8802341d3bf0 R15: 0000000000000282
FS:  00007f91d87d2740(0000) GS:ffff8802356c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8eb48a5000 CR3: 000000022bb64000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001341d3c50 ffffffff81b1c820 ffffffff815869df ffffffff81031c80
0000000000000000 ffffffff815869e0 0000000000000000 ffff8802341d3ca0
ffffffff810f9a5d ffffffff815869df ffff8802341d3cf3 0000000000000005
Call Trace:
[<ffffffff815869df>] ? __netif_receive_skb_core+0x2f/0x7c0
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff815869e0>] ? __netif_receive_skb_core+0x30/0x7c0
[<ffffffff810f9a5d>] on_each_cpu+0x2d/0x60
[<ffffffff815869df>] ? __netif_receive_skb_core+0x2f/0x7c0
[<ffffffff810324d4>] text_poke_bp+0x94/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb8f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc4d>] jump_label_update+0x9d/0xb0
[<ffffffff8117fefd>] __static_key_slow_dec+0x4d/0xb0
[<ffffffff8117ff82>] static_key_slow_dec+0x22/0x50
[<ffffffff81583c48>] net_disable_timestamp+0x38/0x40
[<ffffffff8156c675>] sock_disable_timestamp+0x45/0x50
[<ffffffff8156d5d2>] __sk_free+0x62/0x160
[<ffffffff8156d6f1>] sk_free+0x21/0x30
[<ffffffff8156d7ae>] sk_common_release+0xae/0xf0
[<ffffffff815f8b59>] udp_lib_close+0x9/0x10
[<ffffffff816052b7>] inet_release+0x77/0x80
[<ffffffff8156999f>] sock_release+0x1f/0x80
[<ffffffff81569a12>] sock_close+0x12/0x20
[<ffffffff81202fb9>] __fput+0xe9/0x260
[<ffffffff8120321e>] ____fput+0xe/0x10
[<ffffffff810ad247>] task_work_run+0xa7/0xf0
[<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
[<ffffffff816b527d>] int_signal+0x12/0x17
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #975387
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f933a
Build id: 3.10.0-693.5.2.el7.x86_64
0xfa
2
? setup_data_read
vmlinux 0xffffffff81031c80
Build id: 3.10.0-693.5.2.el7.x86_64
0x90
3
? setup_data_read
vmlinux 0xffffffff81031c80
Build id: 3.10.0-693.5.2.el7.x86_64
0x90
4
? setup_data_read
vmlinux 0xffffffff81031c80
Build id: 3.10.0-693.5.2.el7.x86_64
0x90
5
smp_call_function_single
vmlinux 0xffffffff810f943f
Build id: 3.10.0-693.5.2.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813227c5
Build id: 3.10.0-693.5.2.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f99d3
Build id: 3.10.0-693.5.2.el7.x86_64
0x223
8
? dev_hard_start_xmit
vmlinux 0xffffffff81586833
Build id: 3.10.0-693.5.2.el7.x86_64
0x233
9
? setup_data_read
vmlinux 0xffffffff81031c80
Build id: 3.10.0-693.5.2.el7.x86_64
0x90
10
? dev_hard_start_xmit
vmlinux 0xffffffff81586834
Build id: 3.10.0-693.5.2.el7.x86_64
0x234
11
on_each_cpu
vmlinux 0xffffffff810f9a6d
Build id: 3.10.0-693.5.2.el7.x86_64
0x2d
12
? dev_hard_start_xmit
vmlinux 0xffffffff81586833
Build id: 3.10.0-693.5.2.el7.x86_64
0x233
13
text_poke_bp
vmlinux 0xffffffff810324aa
Build id: 3.10.0-693.5.2.el7.x86_64
0x6a
14
arch_jump_label_transform
vmlinux 0xffffffff8102f558
Build id: 3.10.0-693.5.2.el7.x86_64
0x68
15
__jump_label_update
vmlinux 0xffffffff8117fb9f
Build id: 3.10.0-693.5.2.el7.x86_64
0x5f
16
jump_label_update
vmlinux 0xffffffff8117fc5d
Build id: 3.10.0-693.5.2.el7.x86_64
0x9d
17
__static_key_slow_dec
vmlinux 0xffffffff8117ff0d
Build id: 3.10.0-693.5.2.el7.x86_64
0x4d
18
static_key_slow_dec
vmlinux 0xffffffff8117ff92
Build id: 3.10.0-693.5.2.el7.x86_64
0x22
19
net_disable_timestamp
vmlinux 0xffffffff81583c48
Build id: 3.10.0-693.5.2.el7.x86_64
0x38
20
sock_disable_timestamp
vmlinux 0xffffffff8156c675
Build id: 3.10.0-693.5.2.el7.x86_64
0x45
21
__sk_free
vmlinux 0xffffffff8156d5d2
Build id: 3.10.0-693.5.2.el7.x86_64
0x62
22
sk_free
vmlinux 0xffffffff8156d6f1
Build id: 3.10.0-693.5.2.el7.x86_64
0x21
23
sk_common_release
vmlinux 0xffffffff8156d7ae
Build id: 3.10.0-693.5.2.el7.x86_64
0xae
24
udp_lib_close
vmlinux 0xffffffff815f8c09
Build id: 3.10.0-693.5.2.el7.x86_64
0x9
25
inet_release
vmlinux 0xffffffff81605367
Build id: 3.10.0-693.5.2.el7.x86_64
0x77
26
sock_release
vmlinux 0xffffffff8156999f
Build id: 3.10.0-693.5.2.el7.x86_64
0x1f
27
sock_close
vmlinux 0xffffffff81569a12
Build id: 3.10.0-693.5.2.el7.x86_64
0x12
28
__fput
vmlinux 0xffffffff81202f39
Build id: 3.10.0-693.5.2.el7.x86_64
0xe9
29
____fput
vmlinux 0xffffffff8120319e
Build id: 3.10.0-693.5.2.el7.x86_64
0xe
30
task_work_run
vmlinux 0xffffffff810ad257
Build id: 3.10.0-693.5.2.el7.x86_64
0xa7
31
do_notify_resume
vmlinux 0xffffffff8102ab62
Build id: 3.10.0-693.5.2.el7.x86_64
0x92
32
int_signal
vmlinux 0xffffffff816b533d
Build id: 3.10.0-693.5.2.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [chronyd:714]
Modules linked in: binfmt_misc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun des_generic arc4 md4 nls_utf8 cifs dns_resolver ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev joydev pcspkr vmw_balloon sg vmw_vmci parport_pc parport shpchp i2c_piix4
nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi crct10dif_pclmul crct10dif_common scsi_transport_spi crc32c_intel serio_raw mptscsih i2c_core ata_piix mptbase vmxnet3 libata floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 714 Comm: chronyd Tainted: G             L ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8800b9c36eb0 ti: ffff8800b8290000 task.ti: ffff8800b8290000
RIP: 0010:[<ffffffff810f933a>]  [<ffffffff810f933a>] generic_exec_single+0xfa/0x1a0
RSP: 0000:ffff8800b8293bb0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffff8800b8293b80 RCX: 0000000000000830
RDX: 0000000000000002 RSI: 0000000000000002 RDI: 0000000000000282
RBP: ffff8800b8293c00 R08: ffffffff816ce6e0 R09: 0000000000000000
R10: 000000000000bf1d R11: 0000000000000001 R12: ffffffff81215dc0
R13: ffff8800b8293ad0 R14: ffff880000000000 R15: 00000000000000db
FS:  00007f0141bbc740(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc6aa0e8000 CR3: 00000000baf52000 CR4: 00000000001407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81031c80 0000000000000000
0000000000000002 00000000aa710456 ffffffffff527000 0000000000000001
ffffffff81031c80 ffffffff81031c80 ffff8800b8293c30 ffffffff810f943f
Call Trace:
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff810f943f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813227c5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f99d3>] smp_call_function_many+0x223/0x260
[<ffffffff81586833>] ? dev_hard_start_xmit+0x233/0x3b0
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff81586834>] ? dev_hard_start_xmit+0x234/0x3b0
[<ffffffff810f9a6d>] on_each_cpu+0x2d/0x60
[<ffffffff81586833>] ? dev_hard_start_xmit+0x233/0x3b0
[<ffffffff810324aa>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb9f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc5d>] jump_label_update+0x9d/0xb0
[<ffffffff8117ff0d>] __static_key_slow_dec+0x4d/0xb0
[<ffffffff8117ff92>] static_key_slow_dec+0x22/0x50
[<ffffffff81583c48>] net_disable_timestamp+0x38/0x40
[<ffffffff8156c675>] sock_disable_timestamp+0x45/0x50
[<ffffffff8156d5d2>] __sk_free+0x62/0x160
[<ffffffff8156d6f1>] sk_free+0x21/0x30
[<ffffffff8156d7ae>] sk_common_release+0xae/0xf0
[<ffffffff815f8c09>] udp_lib_close+0x9/0x10
[<ffffffff81605367>] inet_release+0x77/0x80
[<ffffffff8156999f>] sock_release+0x1f/0x80
[<ffffffff81569a12>] sock_close+0x12/0x20
[<ffffffff81202f39>] __fput+0xe9/0x260
[<ffffffff8120319e>] ____fput+0xe/0x10
[<ffffffff810ad257>] task_work_run+0xa7/0xf0
[<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
[<ffffffff816b533d>] int_signal+0x12/0x17
Code: 00 78 01 00 48 89 de 48 03 14 c5 e0 fd b0 81 48 89 df e8 3a db 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #671336
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99b6
Build id: 3.10.0-693.11.1.el7.x86_64
0x206
2
? netif_receive_skb_internal
vmlinux 0xffffffff815871f0
Build id: 3.10.0-693.11.1.el7.x86_64
0x20
3
? setup_data_read
vmlinux 0xffffffff81031c80
Build id: 3.10.0-693.11.1.el7.x86_64
0x90
4
? netif_receive_skb_internal
vmlinux 0xffffffff815871f1
Build id: 3.10.0-693.11.1.el7.x86_64
0x21
5
on_each_cpu
vmlinux 0xffffffff810f9a6d
Build id: 3.10.0-693.11.1.el7.x86_64
0x2d
6
? netif_receive_skb_internal
vmlinux 0xffffffff815871f0
Build id: 3.10.0-693.11.1.el7.x86_64
0x20
7
text_poke_bp
vmlinux 0xffffffff810324aa
Build id: 3.10.0-693.11.1.el7.x86_64
0x6a
8
arch_jump_label_transform
vmlinux 0xffffffff8102f558
Build id: 3.10.0-693.11.1.el7.x86_64
0x68
9
__jump_label_update
vmlinux 0xffffffff8117fb9f
Build id: 3.10.0-693.11.1.el7.x86_64
0x5f
10
jump_label_update
vmlinux 0xffffffff8117fc5d
Build id: 3.10.0-693.11.1.el7.x86_64
0x9d
11
__static_key_slow_dec
vmlinux 0xffffffff8117ff0d
Build id: 3.10.0-693.11.1.el7.x86_64
0x4d
12
static_key_slow_dec
vmlinux 0xffffffff8117ff92
Build id: 3.10.0-693.11.1.el7.x86_64
0x22
13
net_disable_timestamp
vmlinux 0xffffffff81583c48
Build id: 3.10.0-693.11.1.el7.x86_64
0x38
14
sock_disable_timestamp
vmlinux 0xffffffff8156c675
Build id: 3.10.0-693.11.1.el7.x86_64
0x45
15
__sk_free
vmlinux 0xffffffff8156d5d2
Build id: 3.10.0-693.11.1.el7.x86_64
0x62
16
sk_free
vmlinux 0xffffffff8156d6f1
Build id: 3.10.0-693.11.1.el7.x86_64
0x21
17
sk_common_release
vmlinux 0xffffffff8156d7ae
Build id: 3.10.0-693.11.1.el7.x86_64
0xae
18
udp_lib_close
vmlinux 0xffffffff815f8c09
Build id: 3.10.0-693.11.1.el7.x86_64
0x9
19
inet_release
vmlinux 0xffffffff81605367
Build id: 3.10.0-693.11.1.el7.x86_64
0x77
20
sock_release
vmlinux 0xffffffff8156999f
Build id: 3.10.0-693.11.1.el7.x86_64
0x1f
21
sock_close
vmlinux 0xffffffff81569a12
Build id: 3.10.0-693.11.1.el7.x86_64
0x12
22
__fput
vmlinux 0xffffffff81202db9
Build id: 3.10.0-693.11.1.el7.x86_64
0xe9
23
____fput
vmlinux 0xffffffff8120301e
Build id: 3.10.0-693.11.1.el7.x86_64
0xe
24
task_work_run
vmlinux 0xffffffff810ad257
Build id: 3.10.0-693.11.1.el7.x86_64
0xa7
25
do_notify_resume
vmlinux 0xffffffff8102ab62
Build id: 3.10.0-693.11.1.el7.x86_64
0x92
26
int_signal
vmlinux 0xffffffff816b533d
Build id: 3.10.0-693.11.1.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [chronyd:1109]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc intel_powerclamp coretemp kvm_intel sr_mod kvm cdrom irqbypass ipmi_ssif crc32_pclmul ghash_clmulni_intel ipmi_si aesni_intel lrw sg iTCO_wdt iTCO_vendor_support acpi_power_meter ipmi_devintf ipmi_msghandler gf128mul pcspkr glue_helper tpm_infineon i2c_i801 ablk_helper lpc_ich ahci cryptd libahci libata shpchp i7core_edac edac_core acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm drm igb mptsas scsi_transport_sas crct10dif_pclmul ptp crct10dif_common mptscsih pps_core crc32c_intel dca mptbase i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 10 PID: 1109 Comm: chronyd Tainted: G          I  L ------------   3.10.0-693.11.1.el7.x86_64 #1
Hardware name: FUJITSU                          PRIMERGY RX300 S6             /D2619, BIOS 6.00 Rev. 1.04.2619.N1           05/18/2010
task: ffff8803378d1fa0 ti: ffff88033a860000 task.ti: ffff88033a860000
RIP: 0010:[<ffffffff810f99b6>]  [<ffffffff810f99b6>] smp_call_function_many+0x206/0x260
RSP: 0018:ffff88033a863c40  EFLAGS: 00000202
RAX: 000000000000000d RBX: 0000000000000286 RCX: ffff880237bdb0d0
RDX: 000000000000000d RSI: 0000000000000018 RDI: 0000000000000000
RBP: ffff88033a863c78 R08: ffff8802be0e6400 R09: ffffffff81322729
R10: ffff88033fb19c80 R11: ffffea0008c0ee00 R12: 0000000000000286
R13: 0000000000000018 R14: ffff88033a863bb8 R15: 0000000000000286
FS:  00007f8eef294740(0000) GS:ffff88033fb00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056514f5ce1c0 CR3: 000000033271a000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000000013a863c50 ffffffff81b1c820 ffffffff815871f0 ffffffff81031c80
0000000000000000 ffffffff815871f1 0000000000000000 ffff88033a863ca0
ffffffff810f9a6d ffffffff815871f0 ffff88033a863cf3 0000000000000005
Call Trace:
[<ffffffff815871f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff815871f1>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff810f9a6d>] on_each_cpu+0x2d/0x60
[<ffffffff815871f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff810324aa>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb9f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc5d>] jump_label_update+0x9d/0xb0
[<ffffffff8117ff0d>] __static_key_slow_dec+0x4d/0xb0
[<ffffffff8117ff92>] static_key_slow_dec+0x22/0x50
[<ffffffff81583c48>] net_disable_timestamp+0x38/0x40
[<ffffffff8156c675>] sock_disable_timestamp+0x45/0x50
[<ffffffff8156d5d2>] __sk_free+0x62/0x160
[<ffffffff8156d6f1>] sk_free+0x21/0x30
[<ffffffff8156d7ae>] sk_common_release+0xae/0xf0
[<ffffffff815f8c09>] udp_lib_close+0x9/0x10
[<ffffffff81605367>] inet_release+0x77/0x80
[<ffffffff8156999f>] sock_release+0x1f/0x80
[<ffffffff81569a12>] sock_close+0x12/0x20
[<ffffffff81202db9>] __fput+0xe9/0x260
[<ffffffff8120301e>] ____fput+0xe/0x10
[<ffffffff810ad257>] task_work_run+0xa7/0xf0
[<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
[<ffffffff816b533d>] int_signal+0x12/0x17
Code: 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 e5 44 a2 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #351028
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810fb21a
Build id: 3.10.0-693.17.1.el7.x86_64
0xfa
2
? setup_data_read
vmlinux 0xffffffff81031b80
Build id: 3.10.0-693.17.1.el7.x86_64
0x90
3
? setup_data_read
vmlinux 0xffffffff81031b80
Build id: 3.10.0-693.17.1.el7.x86_64
0x90
4
? setup_data_read
vmlinux 0xffffffff81031b80
Build id: 3.10.0-693.17.1.el7.x86_64
0x90
5
smp_call_function_single
vmlinux 0xffffffff810fb31f
Build id: 3.10.0-693.17.1.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff81324bd5
Build id: 3.10.0-693.17.1.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810fb8b3
Build id: 3.10.0-693.17.1.el7.x86_64
0x223
8
? dev_hard_start_xmit
vmlinux 0xffffffff81588833
Build id: 3.10.0-693.17.1.el7.x86_64
0x233
9
? setup_data_read
vmlinux 0xffffffff81031b80
Build id: 3.10.0-693.17.1.el7.x86_64
0x90
10
? dev_hard_start_xmit
vmlinux 0xffffffff81588834
Build id: 3.10.0-693.17.1.el7.x86_64
0x234
11
on_each_cpu
vmlinux 0xffffffff810fb94d
Build id: 3.10.0-693.17.1.el7.x86_64
0x2d
12
? dev_hard_start_xmit
vmlinux 0xffffffff81588833
Build id: 3.10.0-693.17.1.el7.x86_64
0x233
13
text_poke_bp
vmlinux 0xffffffff810323f7
Build id: 3.10.0-693.17.1.el7.x86_64
0xb7
14
arch_jump_label_transform
vmlinux 0xffffffff8102f458
Build id: 3.10.0-693.17.1.el7.x86_64
0x68
15
__jump_label_update
vmlinux 0xffffffff81181a6f
Build id: 3.10.0-693.17.1.el7.x86_64
0x5f
16
jump_label_update
vmlinux 0xffffffff81181b2d
Build id: 3.10.0-693.17.1.el7.x86_64
0x9d
17
__static_key_slow_dec
vmlinux 0xffffffff81181ddd
Build id: 3.10.0-693.17.1.el7.x86_64
0x4d
18
static_key_slow_dec
vmlinux 0xffffffff81181e62
Build id: 3.10.0-693.17.1.el7.x86_64
0x22
19
net_disable_timestamp
vmlinux 0xffffffff81585c48
Build id: 3.10.0-693.17.1.el7.x86_64
0x38
20
sock_disable_timestamp
vmlinux 0xffffffff8156e675
Build id: 3.10.0-693.17.1.el7.x86_64
0x45
21
__sk_free
vmlinux 0xffffffff8156f5d2
Build id: 3.10.0-693.17.1.el7.x86_64
0x62
22
sk_free
vmlinux 0xffffffff8156f6f1
Build id: 3.10.0-693.17.1.el7.x86_64
0x21
23
sk_common_release
vmlinux 0xffffffff8156f7ae
Build id: 3.10.0-693.17.1.el7.x86_64
0xae
24
udp_lib_close
vmlinux 0xffffffff815facf9
Build id: 3.10.0-693.17.1.el7.x86_64
0x9
25
inet_release
vmlinux 0xffffffff816074e7
Build id: 3.10.0-693.17.1.el7.x86_64
0x77
26
sock_release
vmlinux 0xffffffff8156b99f
Build id: 3.10.0-693.17.1.el7.x86_64
0x1f
27
sock_close
vmlinux 0xffffffff8156ba12
Build id: 3.10.0-693.17.1.el7.x86_64
0x12
28
__fput
vmlinux 0xffffffff81204f79
Build id: 3.10.0-693.17.1.el7.x86_64
0xe9
29
____fput
vmlinux 0xffffffff812051de
Build id: 3.10.0-693.17.1.el7.x86_64
0xe
30
task_work_run
vmlinux 0xffffffff810aefc7
Build id: 3.10.0-693.17.1.el7.x86_64
0xa7
31
do_notify_resume
vmlinux 0xffffffff8102ab52
Build id: 3.10.0-693.17.1.el7.x86_64
0x92
32
int_signal
vmlinux 0xffffffff816b8d37
Build id: 3.10.0-693.17.1.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [chronyd:764]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ext4 mbcache jbd2 edac_mce_amd edac_core ppdev crc32_pclmul ghash_clmulni_intel snd_seq_midi snd_seq_midi_event aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon snd_ens1371 snd_rawmidi snd_ac97_codec joydev pcspkr ac97_bus snd_seq
snd_seq_device snd_pcm snd_timer snd sg soundcore vmw_vmci nfit shpchp i2c_piix4 libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw ata_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm pata_acpi drm mptspi vmxnet3 scsi_transport_spi mptscsih ata_piix mptbase libata i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 764 Comm: chronyd Tainted: G           OEL ------------   3.10.0-693.17.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff880093e00fd0 ti: ffff88009d040000 task.ti: ffff88009d040000
RIP: 0010:[<ffffffff810fb21a>]  [<ffffffff810fb21a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88009d043bb0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff88009d043b80 RCX: 0000000000000002
RDX: ffffffff816d69f0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff88009d043c00 R08: ffffffff816d69e8 R09: 0000000000000016
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000010
R13: ffffffff8105a436 R14: ffffffffffffff10 R15: 0000000000000282
FS:  00007fdddff2e740(0000) GS:ffff88009d400000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006dde20 CR3: 000000009d044000 CR4: 00000000001407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff81031b80>] ? setup_data_read+0x90/0x90
[<ffffffff81031b80>] ? setup_data_read+0x90/0x90
[<ffffffff81031b80>] ? setup_data_read+0x90/0x90
[<ffffffff810fb31f>] smp_call_function_single+0x5f/0xa0
[<ffffffff81324bd5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810fb8b3>] smp_call_function_many+0x223/0x260
[<ffffffff81588833>] ? dev_hard_start_xmit+0x233/0x3b0
[<ffffffff81031b80>] ? setup_data_read+0x90/0x90
[<ffffffff81588834>] ? dev_hard_start_xmit+0x234/0x3b0
[<ffffffff810fb94d>] on_each_cpu+0x2d/0x60
[<ffffffff81588833>] ? dev_hard_start_xmit+0x233/0x3b0
[<ffffffff810323f7>] text_poke_bp+0xb7/0xf0
[<ffffffff8102f458>] arch_jump_label_transform+0x68/0xb0
[<ffffffff81181a6f>] __jump_label_update+0x5f/0x80
[<ffffffff81181b2d>] jump_label_update+0x9d/0xb0
[<ffffffff81181ddd>] __static_key_slow_dec+0x4d/0xb0
[<ffffffff81181e62>] static_key_slow_dec+0x22/0x50
[<ffffffff81585c48>] net_disable_timestamp+0x38/0x40
[<ffffffff8156e675>] sock_disable_timestamp+0x45/0x50
[<ffffffff8156f5d2>] __sk_free+0x62/0x160
[<ffffffff8156f6f1>] sk_free+0x21/0x30
[<ffffffff8156f7ae>] sk_common_release+0xae/0xf0
[<ffffffff815facf9>] udp_lib_close+0x9/0x10
[<ffffffff816074e7>] inet_release+0x77/0x80
[<ffffffff8156b99f>] sock_release+0x1f/0x80
[<ffffffff8156ba12>] sock_close+0x12/0x20
[<ffffffff81204f79>] __fput+0xe9/0x260
[<ffffffff812051de>] ____fput+0xe/0x10
[<ffffffff810aefc7>] task_work_run+0xa7/0xf0
[<ffffffff8102ab52>] do_notify_resume+0x92/0xb0
[<ffffffff816b8d37>] int_signal+0x12/0x17
Code: 40 95 01 00 48 89 de 48 03 14 c5 20 8e b1 81 48 89 df e8 5a e0 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #197333
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810fd5fa
Build id: 3.10.0-693.21.1.el7.x86_64
0x20a
2
? netif_receive_skb_internal
vmlinux 0xffffffff81590500
Build id: 3.10.0-693.21.1.el7.x86_64
0x20
3
? setup_data_read
vmlinux 0xffffffff81031c20
Build id: 3.10.0-693.21.1.el7.x86_64
0x90
4
? netif_receive_skb_internal
vmlinux 0xffffffff81590501
Build id: 3.10.0-693.21.1.el7.x86_64
0x21
5
on_each_cpu
vmlinux 0xffffffff810fd6bd
Build id: 3.10.0-693.21.1.el7.x86_64
0x2d
6
? netif_receive_skb_internal
vmlinux 0xffffffff81590500
Build id: 3.10.0-693.21.1.el7.x86_64
0x20
7
text_poke_bp
vmlinux 0xffffffff81032497
Build id: 3.10.0-693.21.1.el7.x86_64
0xb7
8
arch_jump_label_transform
vmlinux 0xffffffff8102f4c8
Build id: 3.10.0-693.21.1.el7.x86_64
0x68
9
__jump_label_update
vmlinux 0xffffffff81183faf
Build id: 3.10.0-693.21.1.el7.x86_64
0x5f
10
jump_label_update
vmlinux 0xffffffff8118408d
Build id: 3.10.0-693.21.1.el7.x86_64
0x9d
11
__static_key_slow_dec
vmlinux 0xffffffff8118433d
Build id: 3.10.0-693.21.1.el7.x86_64
0x4d
12
static_key_slow_dec
vmlinux 0xffffffff811843c2
Build id: 3.10.0-693.21.1.el7.x86_64
0x22
13
net_disable_timestamp
vmlinux 0xffffffff8158cef8
Build id: 3.10.0-693.21.1.el7.x86_64
0x38
14
sock_disable_timestamp
vmlinux 0xffffffff81575785
Build id: 3.10.0-693.21.1.el7.x86_64
0x45
15
__sk_free
vmlinux 0xffffffff815766f5
Build id: 3.10.0-693.21.1.el7.x86_64
0x65
16
sk_free
vmlinux 0xffffffff81576811
Build id: 3.10.0-693.21.1.el7.x86_64
0x21
17
sk_common_release
vmlinux 0xffffffff815768d7
Build id: 3.10.0-693.21.1.el7.x86_64
0xb7
18
udp_lib_close
vmlinux 0xffffffff81602b19
Build id: 3.10.0-693.21.1.el7.x86_64
0x9
19
inet_release
vmlinux 0xffffffff8160f36d
Build id: 3.10.0-693.21.1.el7.x86_64
0x7d
20
sock_release
vmlinux 0xffffffff81572795
Build id: 3.10.0-693.21.1.el7.x86_64
0x25
21
sock_close
vmlinux 0xffffffff81572812
Build id: 3.10.0-693.21.1.el7.x86_64
0x12
22
__fput
vmlinux 0xffffffff8120791c
Build id: 3.10.0-693.21.1.el7.x86_64
0xec
23
____fput
vmlinux 0xffffffff81207b7e
Build id: 3.10.0-693.21.1.el7.x86_64
0xe
24
task_work_run
vmlinux 0xffffffff810b087b
Build id: 3.10.0-693.21.1.el7.x86_64
0xbb
25
do_notify_resume
vmlinux 0xffffffff8102ab52
Build id: 3.10.0-693.21.1.el7.x86_64
0x92
26
int_signal
vmlinux 0xffffffff816c0a5d
Build id: 3.10.0-693.21.1.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [chronyd:1024]
Modules linked in: sctp_diag sctp dccp_diag dccp tcp_diag udp_diag inet_diag unix_diag veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype xt_conntrack br_netfilter bridge stp llc gsch(OE) redirfs(OE) dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter vmw_vsock_vmci_transport vsock ppdev vmw_balloon sg i2c_piix4 pcspkr vmw_vmci joydev shpchp nfit libnvdimm parport_pc parport tcp_htcp ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi libata mptscsih serio_raw mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 1024 Comm: chronyd Tainted: G           OEL ------------   3.10.0-693.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff88083fe8dee0 ti: ffff8800358b4000 task.ti: ffff8800358b4000
RIP: 0010:[<ffffffff810fd5fa>]  [<ffffffff810fd5fa>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8800358b7c40  EFLAGS: 00000202
RAX: 0000000000000017 RBX: 0000000000000286 RCX: ffff88081dbdcce0
RDX: 0000000000000017 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8800358b7c78 R08: ffff880174534800 R09: ffffffff81328989
R10: ffff88081d71ba00 R11: ffffea002050b800 R12: 0000000000000286
R13: ffff8800358b7bf0 R14: 0000047400000040 R15: 00000000000000db
FS:  00007fcd044af740(0000) GS:ffff88081d700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1132676000 CR3: 0000000035a00000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff81590500>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81031c20>] ? setup_data_read+0x90/0x90
[<ffffffff81590501>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff810fd6bd>] on_each_cpu+0x2d/0x60
[<ffffffff81590500>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81032497>] text_poke_bp+0xb7/0xf0
[<ffffffff8102f4c8>] arch_jump_label_transform+0x68/0xb0
[<ffffffff81183faf>] __jump_label_update+0x5f/0xa0
[<ffffffff8118408d>] jump_label_update+0x9d/0xb0
[<ffffffff8118433d>] __static_key_slow_dec+0x4d/0xb0
[<ffffffff811843c2>] static_key_slow_dec+0x22/0x50
[<ffffffff8158cef8>] net_disable_timestamp+0x38/0x40
[<ffffffff81575785>] sock_disable_timestamp+0x45/0x50
[<ffffffff815766f5>] __sk_free+0x65/0x160
[<ffffffff81576811>] sk_free+0x21/0x30
[<ffffffff815768d7>] sk_common_release+0xb7/0xf0
[<ffffffff81602b19>] udp_lib_close+0x9/0x10
[<ffffffff8160f36d>] inet_release+0x7d/0x90
[<ffffffff81572795>] sock_release+0x25/0x90
[<ffffffff81572812>] sock_close+0x12/0x20
[<ffffffff8120791c>] __fput+0xec/0x260
[<ffffffff81207b7e>] ____fput+0xe/0x10
[<ffffffff810b087b>] task_work_run+0xbb/0xe0
[<ffffffff8102ab52>] do_notify_resume+0x92/0xb0
[<ffffffff816c0a5d>] int_signal+0x12/0x17
Code: 48 63 35 8e 1b a3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 b2 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d 1b a3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #105569
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
? netif_rx_internal
vmlinux 0xffffffff815861a2
Build id: 3.10.0-693.el7.x86_64
0x22
3
? setup_data_read
vmlinux 0xffffffff81031c80
Build id: 3.10.0-693.el7.x86_64
0x90
4
? netif_rx_internal
vmlinux 0xffffffff815861a3
Build id: 3.10.0-693.el7.x86_64
0x23
5
on_each_cpu
vmlinux 0xffffffff810f9a5d
Build id: 3.10.0-693.el7.x86_64
0x2d
6
? netif_rx_internal
vmlinux 0xffffffff815861a2
Build id: 3.10.0-693.el7.x86_64
0x22
7
text_poke_bp
vmlinux 0xffffffff810324f7
Build id: 3.10.0-693.el7.x86_64
0xb7
8
arch_jump_label_transform
vmlinux 0xffffffff8102f558
Build id: 3.10.0-693.el7.x86_64
0x68
9
__jump_label_update
vmlinux 0xffffffff8117fb8f
Build id: 3.10.0-693.el7.x86_64
0x5f
10
jump_label_update
vmlinux 0xffffffff8117fc4d
Build id: 3.10.0-693.el7.x86_64
0x9d
11
__static_key_slow_dec
vmlinux 0xffffffff8117fefd
Build id: 3.10.0-693.el7.x86_64
0x4d
12
static_key_slow_dec
vmlinux 0xffffffff8117ff82
Build id: 3.10.0-693.el7.x86_64
0x22
13
net_disable_timestamp
vmlinux 0xffffffff81583c48
Build id: 3.10.0-693.el7.x86_64
0x38
14
sock_disable_timestamp
vmlinux 0xffffffff8156c675
Build id: 3.10.0-693.el7.x86_64
0x45
15
__sk_free
vmlinux 0xffffffff8156d5d2
Build id: 3.10.0-693.el7.x86_64
0x62
16
sk_free
vmlinux 0xffffffff8156d6f1
Build id: 3.10.0-693.el7.x86_64
0x21
17
sk_common_release
vmlinux 0xffffffff8156d7ae
Build id: 3.10.0-693.el7.x86_64
0xae
18
udp_lib_close
vmlinux 0xffffffff815f8b59
Build id: 3.10.0-693.el7.x86_64
0x9
19
inet_release
vmlinux 0xffffffff816052b7
Build id: 3.10.0-693.el7.x86_64
0x77
20
sock_release
vmlinux 0xffffffff8156999f
Build id: 3.10.0-693.el7.x86_64
0x1f
21
sock_close
vmlinux 0xffffffff81569a12
Build id: 3.10.0-693.el7.x86_64
0x12
22
__fput
vmlinux 0xffffffff81202fb9
Build id: 3.10.0-693.el7.x86_64
0xe9
23
____fput
vmlinux 0xffffffff8120321e
Build id: 3.10.0-693.el7.x86_64
0xe
24
task_work_run
vmlinux 0xffffffff810ad247
Build id: 3.10.0-693.el7.x86_64
0xa7
25
do_notify_resume
vmlinux 0xffffffff8102ab62
Build id: 3.10.0-693.el7.x86_64
0x92
26
int_signal
vmlinux 0xffffffff816b527d
Build id: 3.10.0-693.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [chronyd:811]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event coretemp crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi snd_ac97_codec ac97_bus ppdev aesni_intel snd_seq lrw gf128mul glue_helper ablk_helper cryptd snd_seq_device vmw_balloon snd_pcm snd_timer pcspkr joydev snd soundcore sg vmw_vmci i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm crc32c_intel ata_piix mptspi libata e1000 scsi_transport_spi serio_raw mptscsih mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 811 Comm: chronyd Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2012
task: ffff8802356c1fa0 ti: ffff88022a6fc000 task.ti: ffff88022a6fc000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff88022a6ffc40  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0000000000000286 RCX: ffff88023661afc0
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff88022a6ffc78 R08: ffff88017ceac800 R09: ffffffff813227d9
R10: ffff880236799c80 R11: ffffea000809f400 R12: 0000000000000286
R13: ffff88022a6ffbf0 R14: ffff88022a6ffbb8 R15: 0000000000000286
FS:  00007fa56b853740(0000) GS:ffff880236780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4699ac8000 CR3: 000000022c21a000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000000012a6ffc50 ffffffff81b1c820 ffffffff815861a2 ffffffff81031c80
0000000000000000 ffffffff815861a3 0000000000000000 ffff88022a6ffca0
ffffffff810f9a5d ffffffff815861a2 ffff88022a6ffcf3 0000000000000005
Call Trace:
[<ffffffff815861a2>] ? netif_rx_internal+0x22/0x110
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff815861a3>] ? netif_rx_internal+0x23/0x110
[<ffffffff810f9a5d>] on_each_cpu+0x2d/0x60
[<ffffffff815861a2>] ? netif_rx_internal+0x22/0x110
[<ffffffff810324f7>] text_poke_bp+0xb7/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb8f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc4d>] jump_label_update+0x9d/0xb0
[<ffffffff8117fefd>] __static_key_slow_dec+0x4d/0xb0
[<ffffffff8117ff82>] static_key_slow_dec+0x22/0x50
[<ffffffff81583c48>] net_disable_timestamp+0x38/0x40
[<ffffffff8156c675>] sock_disable_timestamp+0x45/0x50
[<ffffffff8156d5d2>] __sk_free+0x62/0x160
[<ffffffff8156d6f1>] sk_free+0x21/0x30
[<ffffffff8156d7ae>] sk_common_release+0xae/0xf0
[<ffffffff815f8b59>] udp_lib_close+0x9/0x10
[<ffffffff816052b7>] inet_release+0x77/0x80
[<ffffffff8156999f>] sock_release+0x1f/0x80
[<ffffffff81569a12>] sock_close+0x12/0x20
[<ffffffff81202fb9>] __fput+0xe9/0x260
[<ffffffff8120321e>] ____fput+0xe/0x10
[<ffffffff810ad247>] task_work_run+0xa7/0xf0
[<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
[<ffffffff816b527d>] int_signal+0x12/0x17
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #96772
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99b2
Build id: 3.10.0-693.5.2.el7.x86_64
0x202
2
? __netif_receive_skb_core
vmlinux 0xffffffff815869df
Build id: 3.10.0-693.5.2.el7.x86_64
0x2f
3
? setup_data_read
vmlinux 0xffffffff81031c80
Build id: 3.10.0-693.5.2.el7.x86_64
0x90
4
? __netif_receive_skb_core
vmlinux 0xffffffff815869e0
Build id: 3.10.0-693.5.2.el7.x86_64
0x30
5
on_each_cpu
vmlinux 0xffffffff810f9a6d
Build id: 3.10.0-693.5.2.el7.x86_64
0x2d
6
? __netif_receive_skb_core
vmlinux 0xffffffff815869df
Build id: 3.10.0-693.5.2.el7.x86_64
0x2f
7
text_poke_bp
vmlinux 0xffffffff810324aa
Build id: 3.10.0-693.5.2.el7.x86_64
0x6a
8
arch_jump_label_transform
vmlinux 0xffffffff8102f558
Build id: 3.10.0-693.5.2.el7.x86_64
0x68
9
__jump_label_update
vmlinux 0xffffffff8117fb9f
Build id: 3.10.0-693.5.2.el7.x86_64
0x5f
10
jump_label_update
vmlinux 0xffffffff8117fc5d
Build id: 3.10.0-693.5.2.el7.x86_64
0x9d
11
__static_key_slow_dec
vmlinux 0xffffffff8117ff0d
Build id: 3.10.0-693.5.2.el7.x86_64
0x4d
12
static_key_slow_dec
vmlinux 0xffffffff8117ff92
Build id: 3.10.0-693.5.2.el7.x86_64
0x22
13
net_disable_timestamp
vmlinux 0xffffffff81583c48
Build id: 3.10.0-693.5.2.el7.x86_64
0x38
14
sock_disable_timestamp
vmlinux 0xffffffff8156c675
Build id: 3.10.0-693.5.2.el7.x86_64
0x45
15
__sk_free
vmlinux 0xffffffff8156d5d2
Build id: 3.10.0-693.5.2.el7.x86_64
0x62
16
sk_free
vmlinux 0xffffffff8156d6f1
Build id: 3.10.0-693.5.2.el7.x86_64
0x21
17
sk_common_release
vmlinux 0xffffffff8156d7ae
Build id: 3.10.0-693.5.2.el7.x86_64
0xae
18
udp_lib_close
vmlinux 0xffffffff815f8c09
Build id: 3.10.0-693.5.2.el7.x86_64
0x9
19
inet_release
vmlinux 0xffffffff81605367
Build id: 3.10.0-693.5.2.el7.x86_64
0x77
20
sock_release
vmlinux 0xffffffff8156999f
Build id: 3.10.0-693.5.2.el7.x86_64
0x1f
21
sock_close
vmlinux 0xffffffff81569a12
Build id: 3.10.0-693.5.2.el7.x86_64
0x12
22
__fput
vmlinux 0xffffffff81202f39
Build id: 3.10.0-693.5.2.el7.x86_64
0xe9
23
____fput
vmlinux 0xffffffff8120319e
Build id: 3.10.0-693.5.2.el7.x86_64
0xe
24
task_work_run
vmlinux 0xffffffff810ad257
Build id: 3.10.0-693.5.2.el7.x86_64
0xa7
25
do_notify_resume
vmlinux 0xffffffff8102ab62
Build id: 3.10.0-693.5.2.el7.x86_64
0x92
26
int_signal
vmlinux 0xffffffff816b533d
Build id: 3.10.0-693.5.2.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [chronyd:602]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul snd_hda_codec_generic glue_helper ablk_helper snd_hda_intel snd_hda_codec ppdev cryptd parport_pc snd_hda_core snd_hwdep snd_seq parport i2c_piix4 pcspkr joydev snd_seq_device virtio_balloon snd_pcm snd_timer snd soundcore nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c ata_generic pata_acpi qxl ahci drm_kms_helper libahci syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm virtio_console virtio_net virtio_blk crct10dif_pclmul crct10dif_common ata_piix libata crc32c_intel serio_raw virtio_pci virtio_ring i2c_core virtio floppy
CPU: 3 PID: 602 Comm: chronyd Tainted: G             L ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff880135268fd0 ti: ffff8800ba10c000 task.ti: ffff8800ba10c000
RIP: 0010:[<ffffffff810f99b2>]  [<ffffffff810f99b2>] smp_call_function_many+0x202/0x260
RSP: 0000:ffff8800ba10fc40  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 0000000000000004 RCX: ffff88013fc9ade0
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8800ba10fc78 R08: ffff880139bb1800 R09: ffffffff813228a9
R10: ffff88013fd99c80 R11: ffffea000072d800 R12: 0000000000017800
R13: ffffffff81031c80 R14: 0000000000000000 R15: ffff88013fd97840
FS:  00007ff1db184740(0000) GS:ffff88013fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000005d469003 CR3: 00000000ba0a6000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001ba10fc50 ffffffff81b1c820 ffffffff815869df ffffffff81031c80
0000000000000000 ffffffff815869e0 0000000000000000 ffff8800ba10fca0
ffffffff810f9a6d ffffffff815869df ffff8800ba10fcf3 0000000000000005
Call Trace:
[<ffffffff815869df>] ? __netif_receive_skb_core+0x2f/0x7c0
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff815869e0>] ? __netif_receive_skb_core+0x30/0x7c0
[<ffffffff810f9a6d>] on_each_cpu+0x2d/0x60
[<ffffffff815869df>] ? __netif_receive_skb_core+0x2f/0x7c0
[<ffffffff810324aa>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb9f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc5d>] jump_label_update+0x9d/0xb0
[<ffffffff8117ff0d>] __static_key_slow_dec+0x4d/0xb0
[<ffffffff8117ff92>] static_key_slow_dec+0x22/0x50
[<ffffffff81583c48>] net_disable_timestamp+0x38/0x40
[<ffffffff8156c675>] sock_disable_timestamp+0x45/0x50
[<ffffffff8156d5d2>] __sk_free+0x62/0x160
[<ffffffff8156d6f1>] sk_free+0x21/0x30
[<ffffffff8156d7ae>] sk_common_release+0xae/0xf0
[<ffffffff815f8c09>] udp_lib_close+0x9/0x10
[<ffffffff81605367>] inet_release+0x77/0x80
[<ffffffff8156999f>] sock_release+0x1f/0x80
[<ffffffff81569a12>] sock_close+0x12/0x20
[<ffffffff81202f39>] __fput+0xe9/0x260
[<ffffffff8120319e>] ____fput+0xe/0x10
[<ffffffff810ad257>] task_work_run+0xa7/0xf0
[<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
[<ffffffff816b533d>] int_signal+0x12/0x17
Code: 48 63 35 16 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 e5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #63834
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffff81685fac
Build id: 3.10.0-514.el7.x86_64
0x19
2
warn_slowpath_common
vmlinux 0xffffffff81085820
Build id: 3.10.0-514.el7.x86_64
0x70
3
? netif_rx_internal
vmlinux 0xffffffff8156e7e2
Build id: 3.10.0-514.el7.x86_64
0x22
4
warn_slowpath_null
vmlinux 0xffffffff8108596a
Build id: 3.10.0-514.el7.x86_64
0x1a
5
text_poke
vmlinux 0xffffffff8168f695
Build id: 3.10.0-514.el7.x86_64
0x225
6
? netif_rx_internal
vmlinux 0xffffffff8156e7e3
Build id: 3.10.0-514.el7.x86_64
0x23
7
? netif_rx_internal
vmlinux 0xffffffff8156e7e2
Build id: 3.10.0-514.el7.x86_64
0x22
8
? netif_rx_internal
vmlinux 0xffffffff8156e7e3
Build id: 3.10.0-514.el7.x86_64
0x23
9
text_poke_bp
vmlinux 0xffffffff81032584
Build id: 3.10.0-514.el7.x86_64
0xa4
10
arch_jump_label_transform
vmlinux 0xffffffff8102f508
Build id: 3.10.0-514.el7.x86_64
0x68
11
__jump_label_update
vmlinux 0xffffffff8117d90f
Build id: 3.10.0-514.el7.x86_64
0x5f
12
jump_label_update
vmlinux 0xffffffff8117d9cd
Build id: 3.10.0-514.el7.x86_64
0x9d
13
__static_key_slow_dec
vmlinux 0xffffffff8117dbdd
Build id: 3.10.0-514.el7.x86_64
0x4d
14
static_key_slow_dec
vmlinux 0xffffffff8117dc52
Build id: 3.10.0-514.el7.x86_64
0x12
15
net_disable_timestamp
vmlinux 0xffffffff8156c5a8
Build id: 3.10.0-514.el7.x86_64
0x38
16
sock_disable_timestamp
vmlinux 0xffffffff81557255
Build id: 3.10.0-514.el7.x86_64
0x45
17
__sk_free
vmlinux 0xffffffff81557f82
Build id: 3.10.0-514.el7.x86_64
0x62
18
sk_free
vmlinux 0xffffffff815580a1
Build id: 3.10.0-514.el7.x86_64
0x21
19
sk_common_release
vmlinux 0xffffffff8155815e
Build id: 3.10.0-514.el7.x86_64
0xae
20
udp_lib_close
vmlinux 0xffffffff815de939
Build id: 3.10.0-514.el7.x86_64
0x9
21
inet_release
vmlinux 0xffffffff815eb157
Build id: 3.10.0-514.el7.x86_64
0x77
22
sock_release
vmlinux 0xffffffff815545df
Build id: 3.10.0-514.el7.x86_64
0x1f
23
sock_close
vmlinux 0xffffffff81554652
Build id: 3.10.0-514.el7.x86_64
0x12
24
__fput
vmlinux 0xffffffff811fff09
Build id: 3.10.0-514.el7.x86_64
0xe9
25
____fput
vmlinux 0xffffffff812001be
Build id: 3.10.0-514.el7.x86_64
0xe
26
task_work_run
vmlinux 0xffffffff810accc7
Build id: 3.10.0-514.el7.x86_64
0xa7
27
do_notify_resume
vmlinux 0xffffffff8102ab22
Build id: 3.10.0-514.el7.x86_64
0x92
28
int_signal
vmlinux 0xffffffff8169687d
Build id: 3.10.0-514.el7.x86_64
0x12
WARNING: at arch/x86/kernel/alternative.c:577 text_poke+0x225/0x230()
Modules linked in: linux_user_bde(POE) linux_kernel_bde(POE) dm_mirror dm_region_hash dm_log dm_mod intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel iTCO_wdt iTCO_vendor_support mxm_wmi aesni_intel lrw gf128mul mei_me glue_helper sb_edac ablk_helper cryptd mei edac_core i2c_i801 pcspkr lpc_ich shpchp wmi acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 ixgbe igb mdio crct10dif_pclmul crct10dif_common ptp i2c_algo_bit crc32c_intel i2c_core pps_core nvme dca fjes [last unloaded: pci_mfd]
CPU: 1 PID: 916 Comm: chronyd Tainted: P        W  OE  ------------   3.10.0-514.el7.x86_64 #1
Hardware name: Default string Default string/Default string, BIOS 5.11 03/07/2017
0000000000000000 00000000da71cc87 ffff88046aaa7c28 ffffffff81685fac
ffff88046aaa7c60 ffffffff81085820 ffffea0000055b80 ffffffff8156e7e2
0000000000000001 ffff88046aaa7d0b 0000000000000000 ffff88046aaa7c70
Call Trace:
[<ffffffff81685fac>] dump_stack+0x19/0x1b
[<ffffffff81085820>] warn_slowpath_common+0x70/0xb0
[<ffffffff8156e7e2>] ? netif_rx_internal+0x22/0x120
[<ffffffff8108596a>] warn_slowpath_null+0x1a/0x20
[<ffffffff8168f695>] text_poke+0x225/0x230
[<ffffffff8156e7e3>] ? netif_rx_internal+0x23/0x120
[<ffffffff8156e7e2>] ? netif_rx_internal+0x22/0x120
[<ffffffff8156e7e3>] ? netif_rx_internal+0x23/0x120
[<ffffffff81032584>] text_poke_bp+0xa4/0xf0
[<ffffffff8102f508>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117d90f>] __jump_label_update+0x5f/0x80
[<ffffffff8117d9cd>] jump_label_update+0x9d/0xb0
[<ffffffff8117dbdd>] __static_key_slow_dec+0x4d/0xb0
[<ffffffff8117dc52>] static_key_slow_dec+0x12/0x20
[<ffffffff8156c5a8>] net_disable_timestamp+0x38/0x40
[<ffffffff81557255>] sock_disable_timestamp+0x45/0x50
[<ffffffff81557f82>] __sk_free+0x62/0x160
[<ffffffff815580a1>] sk_free+0x21/0x30
[<ffffffff8155815e>] sk_common_release+0xae/0xf0
[<ffffffff815de939>] udp_lib_close+0x9/0x10
[<ffffffff815eb157>] inet_release+0x77/0x80
[<ffffffff815545df>] sock_release+0x1f/0x80
[<ffffffff81554652>] sock_close+0x12/0x20
[<ffffffff811fff09>] __fput+0xe9/0x260
[<ffffffff812001be>] ____fput+0xe/0x10
[<ffffffff810accc7>] task_work_run+0xa7/0xe0
[<ffffffff8102ab22>] do_notify_resume+0x92/0xb0
[<ffffffff8169687d>] int_signal+0x12/0x17
Complete report #3886
Frame # Function Binary Source or offset Line
1
text_poke
vmlinux 0xffffffff816adb98
Build id: 3.10.0-693.5.2.el7.x86_64
0x188
2
? __netif_receive_skb_core
vmlinux 0xffffffff815869df
Build id: 3.10.0-693.5.2.el7.x86_64
0x2f
3
? __netif_receive_skb_core
vmlinux 0xffffffff815869e0
Build id: 3.10.0-693.5.2.el7.x86_64
0x30
4
text_poke_bp
vmlinux 0xffffffff810324c1
Build id: 3.10.0-693.5.2.el7.x86_64
0x81
5
arch_jump_label_transform
vmlinux 0xffffffff8102f558
Build id: 3.10.0-693.5.2.el7.x86_64
0x68
6
__jump_label_update
vmlinux 0xffffffff8117fb9f
Build id: 3.10.0-693.5.2.el7.x86_64
0x5f
7
jump_label_update
vmlinux 0xffffffff8117fc5d
Build id: 3.10.0-693.5.2.el7.x86_64
0x9d
8
__static_key_slow_dec
vmlinux 0xffffffff8117ff0d
Build id: 3.10.0-693.5.2.el7.x86_64
0x4d
9
static_key_slow_dec
vmlinux 0xffffffff8117ff92
Build id: 3.10.0-693.5.2.el7.x86_64
0x22
10
net_disable_timestamp
vmlinux 0xffffffff81583c48
Build id: 3.10.0-693.5.2.el7.x86_64
0x38
11
sock_disable_timestamp
vmlinux 0xffffffff8156c675
Build id: 3.10.0-693.5.2.el7.x86_64
0x45
12
__sk_free
vmlinux 0xffffffff8156d5d2
Build id: 3.10.0-693.5.2.el7.x86_64
0x62
13
sk_free
vmlinux 0xffffffff8156d6f1
Build id: 3.10.0-693.5.2.el7.x86_64
0x21
14
sk_common_release
vmlinux 0xffffffff8156d7ae
Build id: 3.10.0-693.5.2.el7.x86_64
0xae
15
udp_lib_close
vmlinux 0xffffffff815f8c09
Build id: 3.10.0-693.5.2.el7.x86_64
0x9
16
inet_release
vmlinux 0xffffffff81605367
Build id: 3.10.0-693.5.2.el7.x86_64
0x77
17
sock_release
vmlinux 0xffffffff8156999f
Build id: 3.10.0-693.5.2.el7.x86_64
0x1f
18
sock_close
vmlinux 0xffffffff81569a12
Build id: 3.10.0-693.5.2.el7.x86_64
0x12
19
__fput
vmlinux 0xffffffff81202f39
Build id: 3.10.0-693.5.2.el7.x86_64
0xe9
20
____fput
vmlinux 0xffffffff8120319e
Build id: 3.10.0-693.5.2.el7.x86_64
0xe
21
task_work_run
vmlinux 0xffffffff810ad257
Build id: 3.10.0-693.5.2.el7.x86_64
0xa7
22
do_notify_resume
vmlinux 0xffffffff8102ab62
Build id: 3.10.0-693.5.2.el7.x86_64
0x92
23
int_signal
vmlinux 0xffffffff816b533d
Build id: 3.10.0-693.5.2.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [chronyd:667]
Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd hv_utils joydev ptp pcspkr sg pps_core i2c_piix4 i2c_core ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi hv_storvsc scsi_transport_fc hv_netvsc ata_piix hyperv_keyboard hid_hyperv scsi_tgt libata crct10dif_pclmul
hyperv_fb crct10dif_common crc32c_intel floppy hv_vmbus serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 667 Comm: chronyd Tainted: G             L ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090006  05/23/2012
task: ffff8800eeda4f10 ti: ffff8800f2bc4000 task.ti: ffff8800f2bc4000
RIP: 0010:[<ffffffff816adb98>]  [<ffffffff816adb98>] text_poke+0x188/0x250
RSP: 0018:ffff8800f2bc7c70  EFLAGS: 00000282
RAX: 0000000000000004 RBX: ffffffffff527000 RCX: 00000000fe982203
RDX: 0000000000000004 RSI: 0000000000000000 RDI: 0000000000000282
RBP: ffff8800f2bc7ca0 R08: ffffffff81b1c828 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffff527000
R13: ffffffff810295da R14: ffff8800f2bc7c20 R15: ffffffff810cb63c
FS:  00007f6939530740(0000) GS:ffff880102600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f622faf6000 CR3: 00000000f2ef0000 CR4: 00000000001406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000282 ffffffff815869df ffff8800f2bc7cf3 0000000000000005
ffffffff815869e0 0000000000000000 ffff8800f2bc7ce0 ffffffff810324c1
ccffffff816a982a 000000009fc72ea4 ffffffff81afae10 0000000000000000
Call Trace:
[<ffffffff815869df>] ? __netif_receive_skb_core+0x2f/0x7c0
[<ffffffff815869e0>] ? __netif_receive_skb_core+0x30/0x7c0
[<ffffffff810324c1>] text_poke_bp+0x81/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb9f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc5d>] jump_label_update+0x9d/0xb0
[<ffffffff8117ff0d>] __static_key_slow_dec+0x4d/0xb0
[<ffffffff8117ff92>] static_key_slow_dec+0x22/0x50
[<ffffffff81583c48>] net_disable_timestamp+0x38/0x40
[<ffffffff8156c675>] sock_disable_timestamp+0x45/0x50
[<ffffffff8156d5d2>] __sk_free+0x62/0x160
[<ffffffff8156d6f1>] sk_free+0x21/0x30
[<ffffffff8156d7ae>] sk_common_release+0xae/0xf0
[<ffffffff815f8c09>] udp_lib_close+0x9/0x10
[<ffffffff81605367>] inet_release+0x77/0x80
[<ffffffff8156999f>] sock_release+0x1f/0x80
[<ffffffff81569a12>] sock_close+0x12/0x20
[<ffffffff81202f39>] __fput+0xe9/0x260
[<ffffffff8120319e>] ____fput+0xe/0x10
[<ffffffff810ad257>] task_work_run+0xa7/0xf0
[<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
[<ffffffff816b533d>] int_signal+0x12/0x17
Code: 00 31 d2 eb 15 66 0f 1f 44 00 00 41 0f b6 1c 06 41 38 1c 04 0f 85 a9 00 00 00 83 c2 01 48 63 c2 49 39 c5 77 e6 48 8b 7d d0 57 9d <0f> 1f 44 00 00 48 83 c4 08 4c 89 e0 5b 41 5c 41 5d 41 5e 41 5f 
Complete report #1427
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
? dev_hard_start_xmit
vmlinux 0xffffffff81586833
Build id: 3.10.0-693.el7.x86_64
0x233
3
? setup_data_read
vmlinux 0xffffffff81031c80
Build id: 3.10.0-693.el7.x86_64
0x90
4
? dev_hard_start_xmit
vmlinux 0xffffffff81586834
Build id: 3.10.0-693.el7.x86_64
0x234
5
on_each_cpu
vmlinux 0xffffffff810f9a5d
Build id: 3.10.0-693.el7.x86_64
0x2d
6
? dev_hard_start_xmit
vmlinux 0xffffffff81586833
Build id: 3.10.0-693.el7.x86_64
0x233
7
text_poke_bp
vmlinux 0xffffffff810324f7
Build id: 3.10.0-693.el7.x86_64
0xb7
8
arch_jump_label_transform
vmlinux 0xffffffff8102f558
Build id: 3.10.0-693.el7.x86_64
0x68
9
__jump_label_update
vmlinux 0xffffffff8117fb8f
Build id: 3.10.0-693.el7.x86_64
0x5f
10
jump_label_update
vmlinux 0xffffffff8117fc4d
Build id: 3.10.0-693.el7.x86_64
0x9d
11
__static_key_slow_dec
vmlinux 0xffffffff8117fefd
Build id: 3.10.0-693.el7.x86_64
0x4d
12
static_key_slow_dec
vmlinux 0xffffffff8117ff82
Build id: 3.10.0-693.el7.x86_64
0x22
13
net_disable_timestamp
vmlinux 0xffffffff81583c48
Build id: 3.10.0-693.el7.x86_64
0x38
14
sock_disable_timestamp
vmlinux 0xffffffff8156c675
Build id: 3.10.0-693.el7.x86_64
0x45
15
__sk_free
vmlinux 0xffffffff8156d5d2
Build id: 3.10.0-693.el7.x86_64
0x62
16
sk_free
vmlinux 0xffffffff8156d6f1
Build id: 3.10.0-693.el7.x86_64
0x21
17
sk_common_release
vmlinux 0xffffffff8156d7ae
Build id: 3.10.0-693.el7.x86_64
0xae
18
udp_lib_close
vmlinux 0xffffffff815f8b59
Build id: 3.10.0-693.el7.x86_64
0x9
19
inet_release
vmlinux 0xffffffff816052b7
Build id: 3.10.0-693.el7.x86_64
0x77
20
sock_release
vmlinux 0xffffffff8156999f
Build id: 3.10.0-693.el7.x86_64
0x1f
21
sock_close
vmlinux 0xffffffff81569a12
Build id: 3.10.0-693.el7.x86_64
0x12
22
__fput
vmlinux 0xffffffff81202fb9
Build id: 3.10.0-693.el7.x86_64
0xe9
23
____fput
vmlinux 0xffffffff8120321e
Build id: 3.10.0-693.el7.x86_64
0xe
24
task_work_run
vmlinux 0xffffffff810ad247
Build id: 3.10.0-693.el7.x86_64
0xa7
25
do_notify_resume
vmlinux 0xffffffff8102ab62
Build id: 3.10.0-693.el7.x86_64
0x92
26
int_signal
vmlinux 0xffffffff816b527d
Build id: 3.10.0-693.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 30s! [chronyd:1049]
Modules linked in: tcp_lp fuse tun ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac edac_core snd_ens1371 iosf_mbi crc32_pclmul snd_rawmidi snd_ac97_codec ghash_clmulni_intel ac97_bus snd_seq aesni_intel lrw snd_seq_device gf128mul glue_helper snd_pcm ablk_helper vmw_balloon joydev pcspkr ppdev cryptd snd_timer i2c_piix4 vmw_vmci sg snd soundcore shpchp nfit libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi drm ata_piix crct10dif_pclmul e1000 crct10dif_common mptscsih libata i2c_core serio_raw crc32c_intel mptbase dm_mirror dm_region_hash
dm_log dm_mod [last unloaded: ip_tables]
CPU: 10 PID: 1049 Comm: chronyd Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff880423a7af70 ti: ffff8804206fc000 task.ti: ffff8804206fc000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8804206ffc40  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 000c000081031c80 RCX: ffff88042d65b088
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8804206ffc78 R08: ffff88017a395800 R09: ffffffff813227d9
R10: ffff88042d899c80 R11: ffffea000d06c400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8804206ffbf0 R15: 0000000000000282
FS:  00007ff402f54740(0000) GS:ffff88042d880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2db4002028 CR3: 0000000426e5b000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
00000001206ffc50 ffffffff81b1c820 ffffffff81586833 ffffffff81031c80
0000000000000000 ffffffff81586834 0000000000000000 ffff8804206ffca0
ffffffff810f9a5d ffffffff81586833 ffff8804206ffcf3 0000000000000005
Call Trace:
[<ffffffff81586833>] ? dev_hard_start_xmit+0x233/0x3b0
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff81586834>] ? dev_hard_start_xmit+0x234/0x3b0
[<ffffffff810f9a5d>] on_each_cpu+0x2d/0x60
[<ffffffff81586833>] ? dev_hard_start_xmit+0x233/0x3b0
[<ffffffff810324f7>] text_poke_bp+0xb7/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb8f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc4d>] jump_label_update+0x9d/0xb0
[<ffffffff8117fefd>] __static_key_slow_dec+0x4d/0xb0
[<ffffffff8117ff82>] static_key_slow_dec+0x22/0x50
[<ffffffff81583c48>] net_disable_timestamp+0x38/0x40
[<ffffffff8156c675>] sock_disable_timestamp+0x45/0x50
[<ffffffff8156d5d2>] __sk_free+0x62/0x160
[<ffffffff8156d6f1>] sk_free+0x21/0x30
[<ffffffff8156d7ae>] sk_common_release+0xae/0xf0
[<ffffffff815f8b59>] udp_lib_close+0x9/0x10
[<ffffffff816052b7>] inet_release+0x77/0x80
[<ffffffff8156999f>] sock_release+0x1f/0x80
[<ffffffff81569a12>] sock_close+0x12/0x20
[<ffffffff81202fb9>] __fput+0xe9/0x260
[<ffffffff8120321e>] ____fput+0xe/0x10
[<ffffffff810ad247>] task_work_run+0xa7/0xf0
[<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
[<ffffffff816b527d>] int_signal+0x12/0x17
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #606
Frame # Function Binary Source or offset Line
1
generic_exec_single
vmlinux 0xffffffff810f932a
Build id: 3.10.0-693.el7.x86_64
0xfa
2
? setup_data_read
vmlinux 0xffffffff81031c80
Build id: 3.10.0-693.el7.x86_64
0x90
3
? setup_data_read
vmlinux 0xffffffff81031c80
Build id: 3.10.0-693.el7.x86_64
0x90
4
? setup_data_read
vmlinux 0xffffffff81031c80
Build id: 3.10.0-693.el7.x86_64
0x90
5
smp_call_function_single
vmlinux 0xffffffff810f942f
Build id: 3.10.0-693.el7.x86_64
0x5f
6
? cpumask_next_and
vmlinux 0xffffffff813226f5
Build id: 3.10.0-693.el7.x86_64
0x35
7
smp_call_function_many
vmlinux 0xffffffff810f99c3
Build id: 3.10.0-693.el7.x86_64
0x223
8
? netif_receive_skb_internal
vmlinux 0xffffffff815871f0
Build id: 3.10.0-693.el7.x86_64
0x20
9
? setup_data_read
vmlinux 0xffffffff81031c80
Build id: 3.10.0-693.el7.x86_64
0x90
10
? netif_receive_skb_internal
vmlinux 0xffffffff815871f1
Build id: 3.10.0-693.el7.x86_64
0x21
11
on_each_cpu
vmlinux 0xffffffff810f9a5d
Build id: 3.10.0-693.el7.x86_64
0x2d
12
? netif_receive_skb_internal
vmlinux 0xffffffff815871f0
Build id: 3.10.0-693.el7.x86_64
0x20
13
text_poke_bp
vmlinux 0xffffffff810324aa
Build id: 3.10.0-693.el7.x86_64
0x6a
14
arch_jump_label_transform
vmlinux 0xffffffff8102f558
Build id: 3.10.0-693.el7.x86_64
0x68
15
__jump_label_update
vmlinux 0xffffffff8117fb8f
Build id: 3.10.0-693.el7.x86_64
0x5f
16
jump_label_update
vmlinux 0xffffffff8117fc4d
Build id: 3.10.0-693.el7.x86_64
0x9d
17
__static_key_slow_dec
vmlinux 0xffffffff8117fefd
Build id: 3.10.0-693.el7.x86_64
0x4d
18
static_key_slow_dec
vmlinux 0xffffffff8117ff82
Build id: 3.10.0-693.el7.x86_64
0x22
19
net_disable_timestamp
vmlinux 0xffffffff81583c48
Build id: 3.10.0-693.el7.x86_64
0x38
20
sock_disable_timestamp
vmlinux 0xffffffff8156c675
Build id: 3.10.0-693.el7.x86_64
0x45
21
__sk_free
vmlinux 0xffffffff8156d5d2
Build id: 3.10.0-693.el7.x86_64
0x62
22
sk_free
vmlinux 0xffffffff8156d6f1
Build id: 3.10.0-693.el7.x86_64
0x21
23
sk_common_release
vmlinux 0xffffffff8156d7ae
Build id: 3.10.0-693.el7.x86_64
0xae
24
udp_lib_close
vmlinux 0xffffffff815f8b59
Build id: 3.10.0-693.el7.x86_64
0x9
25
inet_release
vmlinux 0xffffffff816052b7
Build id: 3.10.0-693.el7.x86_64
0x77
26
sock_release
vmlinux 0xffffffff8156999f
Build id: 3.10.0-693.el7.x86_64
0x1f
27
sock_close
vmlinux 0xffffffff81569a12
Build id: 3.10.0-693.el7.x86_64
0x12
28
__fput
vmlinux 0xffffffff81202fb9
Build id: 3.10.0-693.el7.x86_64
0xe9
29
____fput
vmlinux 0xffffffff8120321e
Build id: 3.10.0-693.el7.x86_64
0xe
30
task_work_run
vmlinux 0xffffffff810ad247
Build id: 3.10.0-693.el7.x86_64
0xa7
31
do_notify_resume
vmlinux 0xffffffff8102ab62
Build id: 3.10.0-693.el7.x86_64
0x92
32
int_signal
vmlinux 0xffffffff816b527d
Build id: 3.10.0-693.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 43s! [chronyd:860]
Modules linked in: vmw_vsock_vmci_transport vsock coretemp ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper joydev pcspkr ablk_helper cryptd vmw_balloon sg parport_pc parport i2c_piix4 vmw_vmci shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm crc32c_intel serio_raw mptspi scsi_transport_spi ata_piix mptscsih vmxnet3 mptbase libata i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 860 Comm: chronyd Not tainted 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff880036604f10 ti: ffff88007a1e4000 task.ti: ffff88007a1e4000
RIP: 0010:[<ffffffff810f932a>]  [<ffffffff810f932a>] generic_exec_single+0xfa/0x1a0
RSP: 0018:ffff88007a1e7bb0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffff88007a1e7b40 RCX: 0000000000000830
RDX: 0000000000000002 RSI: 0000000000000002 RDI: 0000000000000282
RBP: ffff88007a1e7c00 R08: ffffffff816ce6a0 R09: 0000000000000000
R10: ffff8800630e76b0 R11: ffff88007b1b5b10 R12: 0000000000000010
R13: ffffffff8105a2c6 R14: ffffffffffffff10 R15: 0000000000000282
FS:  00007f680af2c740(0000) GS:ffff88007fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f73bb457000 CR3: 000000007a913000 CR4: 00000000000007f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000000000000 0000000000000000 ffffffff81031c80 0000000000000000
0000000000000002 0000000010ffccf6 ffffffffff527000 0000000000000001
ffffffff81031c80 ffffffff81031c80 ffff88007a1e7c30 ffffffff810f942f
Call Trace:
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff810f942f>] smp_call_function_single+0x5f/0xa0
[<ffffffff813226f5>] ? cpumask_next_and+0x35/0x50
[<ffffffff810f99c3>] smp_call_function_many+0x223/0x260
[<ffffffff815871f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff815871f1>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff810f9a5d>] on_each_cpu+0x2d/0x60
[<ffffffff815871f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff810324aa>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb8f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc4d>] jump_label_update+0x9d/0xb0
[<ffffffff8117fefd>] __static_key_slow_dec+0x4d/0xb0
[<ffffffff8117ff82>] static_key_slow_dec+0x22/0x50
[<ffffffff81583c48>] net_disable_timestamp+0x38/0x40
[<ffffffff8156c675>] sock_disable_timestamp+0x45/0x50
[<ffffffff8156d5d2>] __sk_free+0x62/0x160
[<ffffffff8156d6f1>] sk_free+0x21/0x30
[<ffffffff8156d7ae>] sk_common_release+0xae/0xf0
[<ffffffff815f8b59>] udp_lib_close+0x9/0x10
[<ffffffff816052b7>] inet_release+0x77/0x80
[<ffffffff8156999f>] sock_release+0x1f/0x80
[<ffffffff81569a12>] sock_close+0x12/0x20
[<ffffffff81202fb9>] __fput+0xe9/0x260
[<ffffffff8120321e>] ____fput+0xe/0x10
[<ffffffff810ad247>] task_work_run+0xa7/0xf0
[<ffffffff8102ab62>] do_notify_resume+0x92/0xb0
[<ffffffff816b527d>] int_signal+0x12/0x17
Code: 00 78 01 00 48 89 de 48 03 14 c5 e0 fd b0 81 48 89 df e8 7a da 23 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 
Complete report #255
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f98e2
Build id: 3.10.0-514.el7.x86_64
0x202
2
? netif_receive_skb_internal
vmlinux 0xffffffff8156f880
Build id: 3.10.0-514.el7.x86_64
0x20
3
? setup_data_read
vmlinux 0xffffffff81031d20
Build id: 3.10.0-514.el7.x86_64
0x90
4
? netif_receive_skb_internal
vmlinux 0xffffffff8156f881
Build id: 3.10.0-514.el7.x86_64
0x21
5
on_each_cpu
vmlinux 0xffffffff810f999d
Build id: 3.10.0-514.el7.x86_64
0x2d
6
? netif_receive_skb_internal
vmlinux 0xffffffff8156f880
Build id: 3.10.0-514.el7.x86_64
0x20
7
text_poke_bp
vmlinux 0xffffffff81032574
Build id: 3.10.0-514.el7.x86_64
0x94
8
arch_jump_label_transform
vmlinux 0xffffffff8102f508
Build id: 3.10.0-514.el7.x86_64
0x68
9
__jump_label_update
vmlinux 0xffffffff8117d90f
Build id: 3.10.0-514.el7.x86_64
0x5f
10
jump_label_update
vmlinux 0xffffffff8117d9cd
Build id: 3.10.0-514.el7.x86_64
0x9d
11
__static_key_slow_dec
vmlinux 0xffffffff8117dbdd
Build id: 3.10.0-514.el7.x86_64
0x4d
12
static_key_slow_dec
vmlinux 0xffffffff8117dc52
Build id: 3.10.0-514.el7.x86_64
0x12
13
net_disable_timestamp
vmlinux 0xffffffff8156c5a8
Build id: 3.10.0-514.el7.x86_64
0x38
14
sock_disable_timestamp
vmlinux 0xffffffff81557255
Build id: 3.10.0-514.el7.x86_64
0x45
15
__sk_free
vmlinux 0xffffffff81557f82
Build id: 3.10.0-514.el7.x86_64
0x62
16
sk_free
vmlinux 0xffffffff815580a1
Build id: 3.10.0-514.el7.x86_64
0x21
17
sk_common_release
vmlinux 0xffffffff8155815e
Build id: 3.10.0-514.el7.x86_64
0xae
18
udp_lib_close
vmlinux 0xffffffff815de939
Build id: 3.10.0-514.el7.x86_64
0x9
19
inet_release
vmlinux 0xffffffff815eb157
Build id: 3.10.0-514.el7.x86_64
0x77
20
sock_release
vmlinux 0xffffffff815545df
Build id: 3.10.0-514.el7.x86_64
0x1f
21
sock_close
vmlinux 0xffffffff81554652
Build id: 3.10.0-514.el7.x86_64
0x12
22
__fput
vmlinux 0xffffffff811fff09
Build id: 3.10.0-514.el7.x86_64
0xe9
23
____fput
vmlinux 0xffffffff812001be
Build id: 3.10.0-514.el7.x86_64
0xe
24
task_work_run
vmlinux 0xffffffff810accc7
Build id: 3.10.0-514.el7.x86_64
0xa7
25
do_notify_resume
vmlinux 0xffffffff8102ab22
Build id: 3.10.0-514.el7.x86_64
0x92
26
int_signal
vmlinux 0xffffffff8169687d
Build id: 3.10.0-514.el7.x86_64
0x12
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [chronyd:830]
Modules linked in: fuse uinput xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_hda_codec_generic iosf_mbi crc32_pclmul snd_hda_intel ghash_clmulni_intel aesni_intel snd_hda_codec lrw gf128mul glue_helper ablk_helper snd_hda_core snd_hwdep snd_seq ppdev snd_seq_device cryptd snd_pcm snd_timer snd pcspkr parport_pc virtio_balloon parport sg i2c_piix4 soundcore nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_blk virtio_net qxl drm_kms_helper crct10dif_pclmul syscopyarea crct10dif_common sysfillrect
crc32c_intel sysimgblt ata_piix fb_sys_fops serio_raw ttm drm virtio_pci virtio_ring virtio libata i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 830 Comm: chronyd Tainted: G             L ------------   3.10.0-514.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff880227a4af10 ti: ffff8802287e0000 task.ti: ffff8802287e0000
RIP: 0010:[<ffffffff810f98e2>]  [<ffffffff810f98e2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8802287e3c58  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00000000000000fc RCX: ffff880239d9ac48
RDX: 0000000000000003 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8802287e3c90 R08: ffff880184d9d400 R09: ffffffff81317a89
R10: ffff880239d19b40 R11: ffffea00070d8600 R12: 0000000000000292
R13: ffff8802287e3c08 R14: 0000000000000000 R15: ffff880239d177c0
FS:  00007f3b84467740(0000) GS:ffff880239d00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000ad8e8002 CR3: 0000000228f8c000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000000018106ab88 ffffffff81ae20c0 ffffffff8156f880 ffffffff81031d20
0000000000000000 ffffffff8156f881 0000000000000000 ffff8802287e3cb8
ffffffff810f999d ffffffff8156f880 ffff8802287e3d0b 0000000000000005
Call Trace:
[<ffffffff8156f880>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81031d20>] ? setup_data_read+0x90/0x90
[<ffffffff8156f881>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff810f999d>] on_each_cpu+0x2d/0x60
[<ffffffff8156f880>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81032574>] text_poke_bp+0x94/0xf0
[<ffffffff8102f508>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117d90f>] __jump_label_update+0x5f/0x80
[<ffffffff8117d9cd>] jump_label_update+0x9d/0xb0
[<ffffffff8117dbdd>] __static_key_slow_dec+0x4d/0xb0
[<ffffffff8117dc52>] static_key_slow_dec+0x12/0x20
[<ffffffff8156c5a8>] net_disable_timestamp+0x38/0x40
[<ffffffff81557255>] sock_disable_timestamp+0x45/0x50
[<ffffffff81557f82>] __sk_free+0x62/0x160
[<ffffffff815580a1>] sk_free+0x21/0x30
[<ffffffff8155815e>] sk_common_release+0xae/0xf0
[<ffffffff815de939>] udp_lib_close+0x9/0x10
[<ffffffff815eb157>] inet_release+0x77/0x80
[<ffffffff815545df>] sock_release+0x1f/0x80
[<ffffffff81554652>] sock_close+0x12/0x20
[<ffffffff811fff09>] __fput+0xe9/0x260
[<ffffffff812001be>] ____fput+0xe/0x10
[<ffffffff810accc7>] task_work_run+0xa7/0xe0
[<ffffffff8102ab22>] do_notify_resume+0x92/0xb0
[<ffffffff8169687d>] int_signal+0x12/0x17
Code: 48 63 35 76 9e 9e 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 57 ad 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 45 9e 9e 00 eb b7 0f b6 4d cc 4c 89