Complete report #1272328
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9611173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
|
vmlinux |
0xffffffff96637be2
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
|
3
|
|
vmlinux |
0xffffffff96032f70
Build id: 3.10.0-957.el7.x86_64
|
0xd0
|
|
4
|
|
vmlinux |
0xffffffff96637be3
Build id: 3.10.0-957.el7.x86_64
|
0x23
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff961117fd
Build id: 3.10.0-957.el7.x86_64
|
0x2d
|
|
6
|
|
vmlinux |
0xffffffff96637be2
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff960337e7
Build id: 3.10.0-957.el7.x86_64
|
0xb7
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff96030588
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff961b33ef
Build id: 3.10.0-957.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff961b34cd
Build id: 3.10.0-957.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff961b372d
Build id: 3.10.0-957.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff96633894
Build id: 3.10.0-957.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff96620675
Build id: 3.10.0-957.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff96620a22
Build id: 3.10.0-957.el7.x86_64
|
0x3a2
|
|
15
|
|
vmlinux |
0xffffffff9676f608
Build id: 3.10.0-957.el7.x86_64
|
0x228
|
|
16
|
SyS_setsockopt
|
vmlinux |
0xffffffff9661b726
Build id: 3.10.0-957.el7.x86_64
|
0xe6
|
|
17
|
system_call_fastpath
|
vmlinux |
0xffffffff96774ddb
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [fping:114483]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper snd_seq_midi snd_seq_midi_event cryptd ppdev vmw_balloon joydev pcspkr sg snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq parport_pc
parport snd_seq_device snd_pcm snd_timer snd soundcore vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common vmwgfx crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix e1000 serio_raw mptspi libata scsi_transport_spi mptscsih drm_panel_orientation_quirks mptbase nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 114483 Comm: fping Tainted: G W L ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff97c00e1d30c0 ti: ffff97c0237f0000 task.ti: ffff97c0237f0000
RIP: 0010:[<ffffffff9611173a>] [<ffffffff9611173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff97c0237f3d18 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff97c1b5620678
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff97c0237f3d50 R08: ffff97c0fc85c800 R09: ffffffff963755c9
R10: ffff97c1b579f160 R11: fffff46502d3a600 R12: 0000000000000296
R13: ffff97c0237f3cc8 R14: ffff97c0237f3c88 R15: 0000000000000296
FS: 00007f0ea9deb740(0000) GS:ffff97c1b5780000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc73f932000 CR3: 0000000035c84000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffff96637be2>] ? netif_rx_internal+0x22/0x110
[<ffffffff96032f70>] ? setup_data_read+0xd0/0xd0
[<ffffffff96637be3>] ? netif_rx_internal+0x23/0x110
[<ffffffff961117fd>] on_each_cpu+0x2d/0x60
[<ffffffff96637be2>] ? netif_rx_internal+0x22/0x110
[<ffffffff960337e7>] text_poke_bp+0xb7/0xf0
[<ffffffff96030588>] arch_jump_label_transform+0x68/0xb0
[<ffffffff961b33ef>] __jump_label_update+0x5f/0xa0
[<ffffffff961b34cd>] jump_label_update+0x9d/0xb0
[<ffffffff961b372d>] static_key_slow_inc+0xad/0xb0
[<ffffffff96633894>] net_enable_timestamp+0x44/0x50
[<ffffffff96620675>] sock_enable_timestamp+0x45/0x50
[<ffffffff96620a22>] sock_setsockopt+0x3a2/0x8d0
[<ffffffff9676f608>] ? __do_page_fault+0x228/0x500
[<ffffffff9661b726>] SyS_setsockopt+0xe6/0x100
[<ffffffff96774ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 d4 96 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1175256
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
|
0x202
|
|
2
|
|
vmlinux |
0xffffffff815861a2
Build id: 3.10.0-693.el7.x86_64
|
0x22
|
|
3
|
|
vmlinux |
0xffffffff81031c80
Build id: 3.10.0-693.el7.x86_64
|
0x90
|
|
4
|
|
vmlinux |
0xffffffff815861a3
Build id: 3.10.0-693.el7.x86_64
|
0x23
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810f9a5d
Build id: 3.10.0-693.el7.x86_64
|
0x2d
|
|
6
|
|
vmlinux |
0xffffffff815861a2
Build id: 3.10.0-693.el7.x86_64
|
0x22
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff810324d4
Build id: 3.10.0-693.el7.x86_64
|
0x94
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f558
Build id: 3.10.0-693.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8117fb8f
Build id: 3.10.0-693.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8117fc4d
Build id: 3.10.0-693.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff8117fead
Build id: 3.10.0-693.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff815822a4
Build id: 3.10.0-693.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff815700a5
Build id: 3.10.0-693.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8157042a
Build id: 3.10.0-693.el7.x86_64
|
0x37a
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [chronyd:734]
Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter overlay(T) vmhgfs(OE) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device crc32_pclmul snd_pcm ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon snd_timer sg pcspkr nfit joydev snd soundcore libnvdimm i2c_piix4 vmw_vmci shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables
xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata e1000 crct10dif_pclmul crct10dif_common mptspi crc32c_intel scsi_transport_spi serio_raw mptscsih mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 734 Comm: chronyd Tainted: G OEL ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff88022fb49fa0 ti: ffff8800ba764000 task.ti: ffff8800ba764000
RIP: 0010:[<ffffffff810f99a2>] [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8800ba767d48 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff88023561af20
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8800ba767d80 R08: ffff88017cf53400 R09: ffffffff813227d9
R10: ffff880235659c80 R11: ffffea000662d000 R12: 0000000000000296
R13: ffff8800ba767cf8 R14: ffff8800ba767cb8 R15: 0000000000000296
FS: 00007f88aa34f740(0000) GS:ffff880235640000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc360dc81f0 CR3: 0000000034712000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001ba767d58 ffffffff81b1c820 ffffffff815861a2 ffffffff81031c80
0000000000000000 ffffffff815861a3 0000000000000000 ffff8800ba767da8
ffffffff810f9a5d ffffffff815861a2 ffff8800ba767dfb 0000000000000005
Call Trace:
[<ffffffff815861a2>] ? netif_rx_internal+0x22/0x110
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff815861a3>] ? netif_rx_internal+0x23/0x110
[<ffffffff810f9a5d>] on_each_cpu+0x2d/0x60
[<ffffffff815861a2>] ? netif_rx_internal+0x22/0x110
[<ffffffff810324d4>] text_poke_bp+0x94/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb8f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc4d>] jump_label_update+0x9d/0xb0
[<ffffffff8117fead>] static_key_slow_inc+0xad/0xb0
[<ffffffff815822a4>] net_enable_timestamp+0x44/0x50
[<ffffffff815700a5>] sock_enable_timestamp+0x45/0x50
[<ffffffff8157042a>] sock_setsockopt+0x37a/0x880
Complete report #1084342
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810fd5fa
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x20a
|
|
2
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff81590500
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x20
|
|
3
|
|
vmlinux |
0xffffffff81031c20
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x90
|
|
4
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff81590501
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x21
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810fd6bd
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x2d
|
|
6
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff81590500
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x20
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff8103244a
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f4c8
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff81183faf
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8118408d
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff811842ed
Build id: 3.10.0-693.21.1.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff8158b544
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff81579225
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff815795c2
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x392
|
|
15
|
SyS_setsockopt
|
vmlinux |
0xffffffff815744d6
Build id: 3.10.0-693.21.1.el7.x86_64
|
0xe6
|
|
16
|
tracesys
|
vmlinux |
0xffffffff816c0991
Build id: 3.10.0-693.21.1.el7.x86_64
|
0x9d
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 25s! [chronyd:711]
Modules linked in: ip6table_mangle isofs xfs arc4 md4 xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nls_utf8 cifs dns_resolver vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod edac_core iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr vmw_vmci sg nfit shpchp libnvdimm parport_pc i2c_piix4 parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul ata_piix crct10dif_common mptspi crc32c_intel libata scsi_transport_spi serio_raw mptscsih mptbase vmxnet3 i2c_core floppy
CPU: 6 PID: 711 Comm: chronyd Tainted: G L ------------ 3.10.0-693.21.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff88081b57bf40 ti: ffff880812f6c000 task.ti: ffff880812f6c000
RIP: 0010:[<ffffffff810fd5fa>] [<ffffffff810fd5fa>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff880812f6fd18 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 0000000000019580 RCX: ffff88083fc5cd58
RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff880812f6fd50 R08: ffff880174510800 R09: ffffffff81328989
R10: ffff88083fd9ba00 R11: ffffea0007cd0600 R12: ffffffff813288df
R13: ffff880812f6fca0 R14: 0000000000000286 R15: ffffffff8158f473
FS: 00007f05a48f8740(0000) GS:ffff88083fd80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe6390e8e68 CR3: 00000008127ee000 CR4: 00000000001607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff81590500>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81031c20>] ? setup_data_read+0x90/0x90
[<ffffffff81590501>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff810fd6bd>] on_each_cpu+0x2d/0x60
[<ffffffff81590500>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff8103244a>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f4c8>] arch_jump_label_transform+0x68/0xb0
[<ffffffff81183faf>] __jump_label_update+0x5f/0xa0
[<ffffffff8118408d>] jump_label_update+0x9d/0xb0
[<ffffffff811842ed>] static_key_slow_inc+0xad/0xb0
[<ffffffff8158b544>] net_enable_timestamp+0x44/0x50
[<ffffffff81579225>] sock_enable_timestamp+0x45/0x50
[<ffffffff815795c2>] sock_setsockopt+0x392/0x8a0
[<ffffffff815744d6>] SyS_setsockopt+0xe6/0x100
[<ffffffff816c0991>] tracesys+0x9d/0xc3
Code: 48 63 35 8e 1b a3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 b2 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d 1b a3 00 eb b7 0f b6 4d cc 4c 89
Complete report #912674
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810fb892
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x202
|
|
2
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815891f0
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x20
|
|
3
|
|
vmlinux |
0xffffffff81031b80
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x90
|
|
4
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815891f1
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x21
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810fb94d
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x2d
|
|
6
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815891f0
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x20
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff810323aa
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f458
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff81181a6f
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff81181b2d
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff81181d8d
Build id: 3.10.0-693.17.1.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff815842a4
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff815720a5
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8157242a
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x37a
|
|
15
|
|
vmlinux |
0xffffffff81569340
Build id: 3.10.0-693.17.1.el7.x86_64
|
0xa0
|
|
16
|
SyS_setsockopt
|
vmlinux |
0xffffffff8156d3f6
Build id: 3.10.0-693.17.1.el7.x86_64
|
0xd6
|
|
17
|
system_call_fastpath
|
vmlinux |
0xffffffff816b89fd
Build id: 3.10.0-693.17.1.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [chronyd:774]
Modules linked in: file_protector(OE) snapapi26(OE) nfnetlink_queue nfnetlink_log bluetooth rfkill tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_ftp ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod ext4 sb_edac edac_core mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper
ablk_helper cryptd ppdev vmw_balloon i2c_piix4 joydev pcspkr sg vmw_vmci shpchp nfit libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper ata_piix syscopyarea sysfillrect sysimgblt libata fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel mptspi drm scsi_transport_spi mptscsih serio_raw vmxnet3 mptbase i2c_core [last unloaded: edd]
CPU: 6 PID: 774 Comm: chronyd Tainted: G W OEL ------------ 3.10.0-693.17.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff88032b8b1fa0 ti: ffff880034fb0000 task.ti: ffff880034fb0000
RIP: 0010:[<ffffffff810fb892>] [<ffffffff810fb892>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff880034fb3d18 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff88033161cd30
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff880034fb3d50 R08: ffff88017aca4800 R09: ffffffff81324cb9
R10: ffff88033179b9c0 R11: ffffea000afad400 R12: 0000000000000296
R13: ffff880034fb3cc8 R14: ffff880034fb3cd3 R15: ffffffff815881a2
FS: 00007f8f06387740(0000) GS:ffff880331780000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f97ddaca000 CR3: 0000000034fb4000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff815891f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81031b80>] ? setup_data_read+0x90/0x90
[<ffffffff815891f1>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff810fb94d>] on_each_cpu+0x2d/0x60
[<ffffffff815891f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff810323aa>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f458>] arch_jump_label_transform+0x68/0xb0
[<ffffffff81181a6f>] __jump_label_update+0x5f/0x80
[<ffffffff81181b2d>] jump_label_update+0x9d/0xb0
[<ffffffff81181d8d>] static_key_slow_inc+0xad/0xb0
[<ffffffff815842a4>] net_enable_timestamp+0x44/0x50
[<ffffffff815720a5>] sock_enable_timestamp+0x45/0x50
[<ffffffff8157242a>] sock_setsockopt+0x37a/0x880
[<ffffffff81569340>] ? sock_alloc_file+0xa0/0x140
[<ffffffff8156d3f6>] SyS_setsockopt+0xd6/0xf0
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 76 b6 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 8e b1 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 45 b6 a2 00 eb b7 0f b6 4d cc 4c 89
Complete report #864594
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
|
0x202
|
|
2
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815871f0
Build id: 3.10.0-693.el7.x86_64
|
0x20
|
|
3
|
|
vmlinux |
0xffffffff81031c80
Build id: 3.10.0-693.el7.x86_64
|
0x90
|
|
4
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815871f1
Build id: 3.10.0-693.el7.x86_64
|
0x21
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810f9a5d
Build id: 3.10.0-693.el7.x86_64
|
0x2d
|
|
6
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815871f0
Build id: 3.10.0-693.el7.x86_64
|
0x20
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff810324aa
Build id: 3.10.0-693.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f558
Build id: 3.10.0-693.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8117fb8f
Build id: 3.10.0-693.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8117fc4d
Build id: 3.10.0-693.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff8117fead
Build id: 3.10.0-693.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff815822a4
Build id: 3.10.0-693.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff815700a5
Build id: 3.10.0-693.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8157042a
Build id: 3.10.0-693.el7.x86_64
|
0x37a
|
|
15
|
|
vmlinux |
0xffffffff81567340
Build id: 3.10.0-693.el7.x86_64
|
0xa0
|
|
16
|
SyS_setsockopt
|
vmlinux |
0xffffffff8156b3f6
Build id: 3.10.0-693.el7.x86_64
|
0xd6
|
|
17
|
system_call_fastpath
|
vmlinux |
0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [chronyd:26224]
Modules linked in: tcp_diag inet_diag binfmt_misc sysmon_edr(OE) tun netfilter_edr(OE) ip_set nfnetlink bridge stp llc dm_mirror dm_region_hash dm_log dm_mod ext4 mbcache jbd2 ppdev virtio_balloon parport_pc parport joydev i2c_piix4 pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c ata_generic pata_acpi virtio_net virtio_console virtio_scsi virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci i2c_core virtio_ring serio_raw floppy virtio [last unloaded: ip_tables]
CPU: 15 PID: 26224 Comm: chronyd Tainted: G OEL ------------ 3.10.0-693.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff88044fa04f10 ti: ffff880102848000 task.ti: ffff880102848000
RIP: 0010:[<ffffffff810f99a2>] [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff88010284bd48 EFLAGS: 00000202
RAX: 000000000000000b RBX: 00000000000000fc RCX: ffff88081d0daf98
RDX: 000000000000000b RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff88010284bd80 R08: ffff880174b8f000 R09: ffffffff813227d9
R10: ffff88081d1d9c80 R11: ffffea00040ab400 R12: 0000000000000296
R13: ffff88010284bcf8 R14: ffff88010284bce0 R15: 0000000000000005
FS: 00007fbb68441740(0000) GS:ffff88081d1c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb02345f0c8 CR3: 00000007edfc2000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000000010284bd58 ffffffff81b1c820 ffffffff815871f0 ffffffff81031c80
0000000000000000 ffffffff815871f1 0000000000000000 ffff88010284bda8
ffffffff810f9a5d ffffffff815871f0 ffff88010284bdfb 0000000000000005
Call Trace:
[<ffffffff815871f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff815871f1>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff810f9a5d>] on_each_cpu+0x2d/0x60
[<ffffffff815871f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff810324aa>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb8f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc4d>] jump_label_update+0x9d/0xb0
[<ffffffff8117fead>] static_key_slow_inc+0xad/0xb0
[<ffffffff815822a4>] net_enable_timestamp+0x44/0x50
[<ffffffff815700a5>] sock_enable_timestamp+0x45/0x50
[<ffffffff8157042a>] sock_setsockopt+0x37a/0x880
[<ffffffff81567340>] ? sock_alloc_file+0xa0/0x140
[<ffffffff8156b3f6>] SyS_setsockopt+0xd6/0xf0
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89
Complete report #671432
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f99b2
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x202
|
|
2
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815871f0
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x20
|
|
3
|
|
vmlinux |
0xffffffff81031c80
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x90
|
|
4
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815871f1
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x21
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810f9a6d
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x2d
|
|
6
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815871f0
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x20
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff810324aa
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f558
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8117fb9f
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8117fc5d
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff8117febd
Build id: 3.10.0-693.11.1.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff815822a4
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff815700a5
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8157042a
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x37a
|
|
15
|
SyS_setsockopt
|
vmlinux |
0xffffffff8156b3f6
Build id: 3.10.0-693.11.1.el7.x86_64
|
0xd6
|
|
16
|
system_call_fastpath
|
vmlinux |
0xffffffff816b5089
Build id: 3.10.0-693.11.1.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [chronyd:1109]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc intel_powerclamp coretemp kvm_intel sr_mod kvm cdrom irqbypass ipmi_ssif crc32_pclmul ghash_clmulni_intel ipmi_si aesni_intel lrw sg iTCO_wdt iTCO_vendor_support acpi_power_meter ipmi_devintf ipmi_msghandler gf128mul pcspkr glue_helper tpm_infineon i2c_i801 ablk_helper lpc_ich ahci cryptd libahci libata shpchp i7core_edac edac_core acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm drm igb mptsas scsi_transport_sas crct10dif_pclmul ptp crct10dif_common mptscsih pps_core crc32c_intel dca mptbase i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 15 PID: 1109 Comm: chronyd Tainted: G I L ------------ 3.10.0-693.11.1.el7.x86_64 #1
Hardware name: FUJITSU PRIMERGY RX300 S6 /D2619, BIOS 6.00 Rev. 1.04.2619.N1 05/18/2010
task: ffff8803378d1fa0 ti: ffff88033a860000 task.ti: ffff88033a860000
RIP: 0010:[<ffffffff810f99b2>] [<ffffffff810f99b2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff88033a863d48 EFLAGS: 00000202
RAX: 0000000000000015 RBX: 0000000000017800 RCX: ffff88033fc5b198
RDX: 0000000000000015 RSI: 0000000000000018 RDI: 0000000000000000
RBP: ffff88033a863d80 R08: ffff88017cddf000 R09: ffffffff81322729
R10: ffff880237c59c80 R11: ffffea00028d9600 R12: ffff88033a863cc0
R13: 0000000000000286 R14: 0000000000000010 R15: ffffffff8105558e
FS: 00007f8eef294740(0000) GS:ffff880237c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f58adb2b4f0 CR3: 000000033271a000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000000013a863d58 ffffffff81b1c820 ffffffff815871f0 ffffffff81031c80
0000000000000000 ffffffff815871f1 0000000000000000 ffff88033a863da8
ffffffff810f9a6d ffffffff815871f0 ffff88033a863dfb 0000000000000005
Call Trace:
[<ffffffff815871f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff815871f1>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff810f9a6d>] on_each_cpu+0x2d/0x60
[<ffffffff815871f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff810324aa>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb9f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc5d>] jump_label_update+0x9d/0xb0
[<ffffffff8117febd>] static_key_slow_inc+0xad/0xb0
[<ffffffff815822a4>] net_enable_timestamp+0x44/0x50
[<ffffffff815700a5>] sock_enable_timestamp+0x45/0x50
[<ffffffff8157042a>] sock_setsockopt+0x37a/0x880
[<ffffffff8156b3f6>] SyS_setsockopt+0xd6/0xf0
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 16 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 e5 44 a2 00 eb b7 0f b6 4d cc 4c 89
Complete report #663161
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffaa71173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffffaac39860
Build id: 3.10.0-957.el7.x86_64
|
0x20
|
|
3
|
|
vmlinux |
0xffffffffaa632f70
Build id: 3.10.0-957.el7.x86_64
|
0xd0
|
|
4
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffffaac39861
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffffaa7117fd
Build id: 3.10.0-957.el7.x86_64
|
0x2d
|
|
6
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffffaac39860
Build id: 3.10.0-957.el7.x86_64
|
0x20
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffffaa63379a
Build id: 3.10.0-957.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffffaa630588
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffffaa7b33ef
Build id: 3.10.0-957.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffffaa7b34cd
Build id: 3.10.0-957.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffffaa7b372d
Build id: 3.10.0-957.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffffaac33894
Build id: 3.10.0-957.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffffaac20675
Build id: 3.10.0-957.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffffaac20a22
Build id: 3.10.0-957.el7.x86_64
|
0x3a2
|
|
15
|
|
vmlinux |
0xffffffffaad6f608
Build id: 3.10.0-957.el7.x86_64
|
0x228
|
|
16
|
SyS_setsockopt
|
vmlinux |
0xffffffffaac1b726
Build id: 3.10.0-957.el7.x86_64
|
0xe6
|
|
17
|
system_call_fastpath
|
vmlinux |
0xffffffffaad74ddb
Build id: 3.10.0-957.el7.x86_64
|
0x22
|
NMI watchdog: BUG: soft lockup - CPU#98 stuck for 23s! [ping:243488]
Modules linked in: nvidia_uvm(OE) petrelfs(OE) cuse fuse bonding rdma_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) sunrpc iTCO_wdt iTCO_vendor_support nvidia_drm(POE) nvidia_modeset(POE) coretemp kvm_intel kvm irqbypass crc32_pclmul vfat fat ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd nvidia(POE) pcspkr joydev cdc_ether usbnet mii sg mei_me mei i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler acpi_cpufreq pinctrl_lewisburg pinctrl_intel acpi_power_meter knem(OE) ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic nouveau crct10dif_pclmul crct10dif_common crc32c_intel nvme ast video mxm_wmi nvme_core i2c_algo_bit drm_kms_helper mlx5_core(OE) syscopyarea sysfillrect sysimgblt
fb_sys_fops mlxfw(OE) vfio_mdev(OE) vfio_iommu_type1 ttm vfio mdev(OE) mlx_compat(OE) devlink ixgbe drm ahci mpt3sas libahci libata mdio raid_class ptp drm_panel_orientation_quirks scsi_transport_sas pps_core dca wmi dm_mirror dm_region_hash dm_log dm_mod
CPU: 98 PID: 243488 Comm: ping Kdump: loaded Tainted: P W OEL ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: Nettrix X660 G45/60WB32, BIOS NKMH041042 10/28/2022
task: ffff8e6e7baf6180 ti: ffff8e01a8b48000 task.ti: ffff8e01a8b48000
RIP: 0010:[<ffffffffaa71173a>] [<ffffffffaa71173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8e01a8b4bd18 EFLAGS: 00000202
RAX: 0000000000000024 RBX: 0000008000000062 RCX: ffff8e8afe721560
RDX: 0000000000000024 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8e01a8b4bd50 R08: ffff8e0d4da46400 R09: ffffffffaa9755c9
R10: ffff8e8afee9f160 R11: ffffd759de268e00 R12: 000000fcaa97551f
R13: 0000000000000296 R14: 0000000000000296 R15: ffff8e01a8b4bcc8
FS: 00007f0c13f4b740(0000) GS:ffff8e8afee80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0c13f6b000 CR3: 000000eba9fd4000 CR4: 0000000000760fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffaac39860>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffffaa632f70>] ? setup_data_read+0xd0/0xd0
[<ffffffffaac39861>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffffaa7117fd>] on_each_cpu+0x2d/0x60
[<ffffffffaac39860>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffffaa63379a>] text_poke_bp+0x6a/0xf0
[<ffffffffaa630588>] arch_jump_label_transform+0x68/0xb0
[<ffffffffaa7b33ef>] __jump_label_update+0x5f/0xa0
[<ffffffffaa7b34cd>] jump_label_update+0x9d/0xb0
[<ffffffffaa7b372d>] static_key_slow_inc+0xad/0xb0
[<ffffffffaac33894>] net_enable_timestamp+0x44/0x50
[<ffffffffaac20675>] sock_enable_timestamp+0x45/0x50
[<ffffffffaac20a22>] sock_setsockopt+0x3a2/0x8d0
[<ffffffffaad6f608>] ? __do_page_fault+0x228/0x500
[<ffffffffaac1b726>] SyS_setsockopt+0xe6/0x100
[<ffffffffaad74ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 34 ab f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #636268
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb851761a
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x20a
|
|
2
|
|
vmlinux |
0xffffffffb8a522f2
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x22
|
|
3
|
|
vmlinux |
0xffffffffb8434260
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xd0
|
|
4
|
|
vmlinux |
0xffffffffb8a522f3
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x23
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffffb85176dd
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x2d
|
|
6
|
|
vmlinux |
0xffffffffb8a522f2
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x22
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffffb8434ab4
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x94
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffffb84316a8
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffffb85bb6cf
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffffb85bb7ad
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffffb85bba0d
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffffb8a4db74
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffffb8a3a715
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffffb8a3aaca
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x3aa
|
|
15
|
|
vmlinux |
0xffffffffb8b8d678
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x238
|
|
16
|
SyS_setsockopt
|
vmlinux |
0xffffffffb8a35726
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xe6
|
|
17
|
system_call_fastpath
|
vmlinux |
0xffffffffb8b92ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x25
|
|
18
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffb8b92e15
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xa2
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [ping:22635]
Modules linked in: xt_nat veth tcp_lp nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter fuse dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter iptable_filter devlink ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables binfmt_misc overlay vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon pcspkr parport_pc joydev parport sg i2c_piix4 vmw_vmci ip_tables xfs libcrc32c
sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul mptspi crct10dif_common crc32c_intel scsi_transport_spi serio_raw mptscsih vmxnet3 mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 22635 Comm: ping Kdump: loaded Tainted: G L ------------ 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffff9712445d9070 ti: ffff9712b7c28000 task.ti: ffff9712b7c28000
RIP: 0010:[<ffffffffb851761a>] [<ffffffffb851761a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9712b7c2bd18 EFLAGS: 00000202
RAX: 0000000000000007 RBX: 000000000001b840 RCX: ffff97147fde06c8
RDX: 0000000000000007 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff9712b7c2bd50 R08: ffff9713bcf83800 R09: ffffffffb87851b9
R10: ffff97147fd9f160 R11: ffffecb44179a400 R12: ffffffffb878510f
R13: ffff971275412180 R14: ffffffffb84e69fb R15: ffff9712b7c2bcb0
FS: 00007f361c3ba740(0000) GS:ffff97147fd80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fac3bb61ba0 CR3: 00000001fa464000 CR4: 00000000000407e0
Call Trace:
[<ffffffffb8a522f2>] ? netif_rx_internal+0x22/0x110
[<ffffffffb8434260>] ? setup_data_read+0xd0/0xd0
[<ffffffffb8a522f3>] ? netif_rx_internal+0x23/0x110
[<ffffffffb85176dd>] on_each_cpu+0x2d/0x60
[<ffffffffb8a522f2>] ? netif_rx_internal+0x22/0x110
[<ffffffffb8434ab4>] text_poke_bp+0x94/0xf0
[<ffffffffb84316a8>] arch_jump_label_transform+0x68/0xb0
[<ffffffffb85bb6cf>] __jump_label_update+0x5f/0xa0
[<ffffffffb85bb7ad>] jump_label_update+0x9d/0xb0
[<ffffffffb85bba0d>] static_key_slow_inc+0xad/0xb0
[<ffffffffb8a4db74>] net_enable_timestamp+0x44/0x50
[<ffffffffb8a3a715>] sock_enable_timestamp+0x45/0x50
[<ffffffffb8a3aaca>] sock_setsockopt+0x3aa/0x8f0
[<ffffffffb8b8d678>] ? __do_page_fault+0x238/0x500
[<ffffffffb8a35726>] SyS_setsockopt+0xe6/0x100
[<ffffffffb8b92ed2>] system_call_fastpath+0x25/0x2a
[<ffffffffb8b92e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 15 b9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #553733
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8d71718e
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x20e
|
|
2
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff8dc59b80
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x20
|
|
3
|
|
vmlinux |
0xffffffff8d634260
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xd0
|
|
4
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff8dc59b81
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x21
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff8d71724d
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x2d
|
|
6
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff8dc59b80
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x20
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff8d634a8a
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8d6316a8
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8d7bb39f
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8d7bb47d
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff8d7bb6dd
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff8dc53774
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff8dc40705
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8dc40c87
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x577
|
|
15
|
SyS_setsockopt
|
vmlinux |
0xffffffff8dc3b716
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xe6
|
|
16
|
system_call_fastpath
|
vmlinux |
0xffffffff8dd99f92
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#20 stuck for 23s! [chronyd:1571]
Modules linked in: socwatch2_14(OE) vtsspp(OE) sep5(OE) socperf3(OE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 pax(OE) tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support intel_wmi_thunderbolt mxm_wmi nvidia_drm(POE) nvidia_modeset(POE) vfat nvidia(POE) fat sb_edac intel_powerclamp coretemp snd_hda_codec_hdmi intel_rapl iosf_mbi
kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel snd_hda_codec_realtek aesni_intel snd_hda_codec_generic lrw gf128mul glue_helper ablk_helper cryptd pcspkr snd_hda_intel snd_hda_codec snd_hda_core sg snd_hwdep snd_seq i2c_i801 lpc_ich snd_seq_device snd_pcm joydev mei_me mei snd_timer snd ipmi_ssif ioatdma soundcore ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter acpi_pad ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm igb ahci crct10dif_pclmul crct10dif_common crc32c_intel libahci ptp pps_core dca libata i2c_algo_bit drm_panel_orientation_quirks nfit libnvdimm fuse
CPU: 20 PID: 1571 Comm: chronyd Kdump: loaded Tainted: P OEL ------------ 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: Supermicro Super Server/X10DRG-Q, BIOS 3.2 11/22/2019
task: ffff90d8a0776300 ti: ffff90f8bc574000 task.ti: ffff90f8bc574000
RIP: 0010:[<ffffffff8d71718e>] [<ffffffff8d71718e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff90f8bc577d18 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000000000001b840 RCX: ffff90d8bf620958
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff90f8bc577d50 R08: ffff90d9732d0000 R09: ffffffff8d9869f9
R10: ffff90f8bef1f160 R11: fffffe7f7ff3b200 R12: ffff90f8bc577cc8
R13: 0000000000000003 R14: ffffffff8d827843 R15: ffff90f8bc577ca8
FS: 00007ff5aff02740(0000) GS:ffff90f8bef00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005561e0bf6ba8 CR3: 0000003fe85fc000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8dc59b80>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff8d634260>] ? setup_data_read+0xd0/0xd0
[<ffffffff8dc59b81>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff8d71724d>] on_each_cpu+0x2d/0x60
[<ffffffff8dc59b80>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff8d634a8a>] text_poke_bp+0x6a/0xf0
[<ffffffff8d6316a8>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8d7bb39f>] __jump_label_update+0x5f/0xa0
[<ffffffff8d7bb47d>] jump_label_update+0x9d/0xb0
[<ffffffff8d7bb6dd>] static_key_slow_inc+0xad/0xb0
[<ffffffff8dc53774>] net_enable_timestamp+0x44/0x50
[<ffffffff8dc40705>] sock_enable_timestamp+0x45/0x50
[<ffffffff8dc40c87>] sock_setsockopt+0x577/0x8f0
[<ffffffff8dc3b716>] SyS_setsockopt+0xe6/0x100
[<ffffffff8dd99f92>] system_call_fastpath+0x25/0x2a
Code: 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 35 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #517615
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f670a
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x23a
|
|
2
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff81587de1
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x21
|
|
3
|
smp_call_function
|
vmlinux |
0xffffffff810f67aa
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x3a
|
|
4
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff81587de0
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x20
|
|
5
|
|
vmlinux |
0xffffffff810237e0
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x90
|
|
6
|
on_each_cpu
|
vmlinux |
0xffffffff810f680b
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x2b
|
|
7
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff81587de0
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x20
|
|
8
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff81587de1
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x21
|
|
9
|
text_poke_bp
|
vmlinux |
0xffffffff81024087
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0xb7
|
|
10
|
arch_jump_label_transform
|
vmlinux |
0xffffffff81020e48
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x68
|
|
11
|
__jump_label_update
|
vmlinux |
0xffffffff8118242f
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x5f
|
|
12
|
jump_label_update
|
vmlinux |
0xffffffff811824ed
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x9d
|
|
13
|
static_key_slow_inc
|
vmlinux |
0xffffffff8118274d
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0xad
|
|
14
|
net_enable_timestamp
|
vmlinux |
0xffffffff81582b04
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x44
|
|
15
|
sock_enable_timestamp
|
vmlinux |
0xffffffff815703c5
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x45
|
|
16
|
sock_setsockopt
|
vmlinux |
0xffffffff8157090f
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x53f
|
|
17
|
SyS_setsockopt
|
vmlinux |
0xffffffff8156b416
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0xd6
|
|
18
|
system_call_fastpath
|
vmlinux |
0xffffffff816ba109
Build id: 3.10.0-693.2.2.rt56.623.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [chronyd:1020]
Modules linked in: tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink ext4 mbcache jbd2 sunrpc dm_mirror dm_region_hash dm_log dm_mod ppdev joydev sg edac_core virtio_balloon virtio_rng iosf_mbi crc32_pclmul parport_pc ghash_clmulni_intel parport aesni_intel lrw gf128mul glue_helper ablk_helper i2c_piix4 cryptd i2c_core pcspkr ip_tables xfs libcrc32c sr_mod cdrom virtio_blk virtio_console virtio_net ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_piix virtio_pci libata floppy fuse
CPU: 0 PID: 1020 Comm: chronyd Not tainted 3.10.0-693.2.2.rt56.623.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff8803e9c7e1e0 ti: ffff8803e2b60000 task.ti: ffff8803e2b60000
RIP: 0010:[<ffffffff810f670a>] [<ffffffff810f670a>] smp_call_function_many+0x23a/0x2a0
RSP: 0018:ffff8803e2b63d20 EFLAGS: 00000202
RAX: 0000000000000003 RBX: ffff8803e2b61fd8 RCX: ffff8803efedc838
RDX: 0000000000000003 RSI: 0000000000000019 RDI: 0000000000000000
RBP: ffff8803e2b63d58 R08: ffff8803efe19c08 R09: 0000000000000000
R10: ffff8803efe19c08 R11: 0000000000000000 R12: ffffffff8101a8e2
R13: ffff8803e2b63cd8 R14: ffffffff810c6bfc R15: ffff8803e2b63ca0
FS: 00007efcdf257740(0000) GS:ffff8803efe00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006ec0f8 CR3: 00000003e8810000 CR4: 00000000003406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000100000000 0000000000019bc0 ffff8803e2b60000 ffff8803e2b63fd8
0000000000000000 ffffffff81587de1 0000000000000000 ffff8803e2b63d78
ffffffff810f67aa ffffffff81587de0 ffffffff810237e0 ffff8803e2b63da8
Call Trace:
[<ffffffff81587de1>] ? netif_receive_skb_internal+0x21/0xe0
[<ffffffff810f67aa>] smp_call_function+0x3a/0x70
[<ffffffff81587de0>] ? netif_receive_skb_internal+0x20/0xe0
[<ffffffff810237e0>] ? setup_data_read+0x90/0x90
[<ffffffff810f680b>] on_each_cpu+0x2b/0xc0
[<ffffffff81587de0>] ? netif_receive_skb_internal+0x20/0xe0
[<ffffffff81587de1>] ? netif_receive_skb_internal+0x21/0xe0
[<ffffffff81024087>] text_poke_bp+0xb7/0xf0
[<ffffffff81020e48>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8118242f>] __jump_label_update+0x5f/0x80
[<ffffffff811824ed>] jump_label_update+0x9d/0xb0
[<ffffffff8118274d>] static_key_slow_inc+0xad/0xb0
[<ffffffff81582b04>] net_enable_timestamp+0x44/0x50
[<ffffffff815703c5>] sock_enable_timestamp+0x45/0x50
[<ffffffff8157090f>] sock_setsockopt+0x53f/0x890
[<ffffffff8156b416>] SyS_setsockopt+0xd6/0xf0
[<ffffffff816ba109>] system_call_fastpath+0x16/0x1b
Code: 63 35 ff 19 a6 00 89 c2 39 f0 0f 8d 49 fe ff ff 48 98 49 8b 4d 00 48 03 0c c5 20 a0 b4 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 19 a6 00 eb b7 0f b6 4d cc 4c 89
Complete report #430516
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810fb6b2
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x202
|
|
2
|
?
__netif_receive_skb_core
|
vmlinux |
0xffffffff815889df
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x2f
|
|
3
|
|
vmlinux |
0xffffffff81031b80
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x90
|
|
4
|
?
__netif_receive_skb_core
|
vmlinux |
0xffffffff815889e0
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x30
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810fb76d
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x2d
|
|
6
|
?
__netif_receive_skb_core
|
vmlinux |
0xffffffff815889df
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x2f
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff810323aa
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f458
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8118188f
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8118194d
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff81181bad
Build id: 3.10.0-693.11.6.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff815842a4
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff815720a5
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8157242a
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x37a
|
|
15
|
SyS_setsockopt
|
vmlinux |
0xffffffff8156d3f6
Build id: 3.10.0-693.11.6.el7.x86_64
|
0xd6
|
|
16
|
system_call_fastpath
|
vmlinux |
0xffffffff816b89fd
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x16
|
|
17
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff816b889d
Build id: 3.10.0-693.11.6.el7.x86_64
|
0xca
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [chronyd:656]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_intel8x0 snd_ac97_codec ppdev e1000 pcspkr sg ac97_bus joydev snd_seq snd_seq_device i2c_piix4 i2c_core parport_pc snd_pcm iosf_mbi snd_timer snd soundcore parport ahci libahci video nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi ata_piix serio_raw libata crc32c_intel dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 656 Comm: chronyd Tainted: G L ------------ 3.10.0-693.11.6.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff8800356eeeb0 ti: ffff8800d9a58000 task.ti: ffff8800d9a58000
RIP: 0010:[<ffffffff810fb6b2>] [<ffffffff810fb6b2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8800d9a5bd18 EFLAGS: 00010202
RAX: 0000000000000001 RBX: ffff8800d9a5bca8 RCX: ffff88011fc9be20
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8800d9a5bd50 R08: ffff88011b142400 R09: ffffffff81324989
R10: ffff88011fc1b9c0 R11: ffffea0001149900 R12: ffffffff816b9bff
R13: ffffffff816b9c06 R14: ffffffff816b9c0d R15: ffffffff816b9c14
FS: 00007f968b05d740(0000) GS:ffff88011fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcf5947a000 CR3: 00000000d9a56000 CR4: 00000000000206f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff815889df>] ? __netif_receive_skb_core+0x2f/0x7c0
[<ffffffff81031b80>] ? setup_data_read+0x90/0x90
[<ffffffff815889e0>] ? __netif_receive_skb_core+0x30/0x7c0
[<ffffffff810fb76d>] on_each_cpu+0x2d/0x60
[<ffffffff815889df>] ? __netif_receive_skb_core+0x2f/0x7c0
[<ffffffff810323aa>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f458>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8118188f>] __jump_label_update+0x5f/0x80
[<ffffffff8118194d>] jump_label_update+0x9d/0xb0
[<ffffffff81181bad>] static_key_slow_inc+0xad/0xb0
[<ffffffff815842a4>] net_enable_timestamp+0x44/0x50
[<ffffffff815720a5>] sock_enable_timestamp+0x45/0x50
[<ffffffff8157242a>] sock_setsockopt+0x37a/0x880
[<ffffffff8156d3f6>] SyS_setsockopt+0xd6/0xf0
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
[<ffffffff816b889d>] ? system_call_after_swapgs+0xca/0x214
Code: 48 63 35 56 b8 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 8e b1 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 25 b8 a2 00 eb b7 0f b6 4d cc 4c 89
Complete report #364862
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f99a6
Build id: 3.10.0-693.el7.x86_64
|
0x206
|
|
2
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815871f0
Build id: 3.10.0-693.el7.x86_64
|
0x20
|
|
3
|
|
vmlinux |
0xffffffff81031c80
Build id: 3.10.0-693.el7.x86_64
|
0x90
|
|
4
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815871f1
Build id: 3.10.0-693.el7.x86_64
|
0x21
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810f9a5d
Build id: 3.10.0-693.el7.x86_64
|
0x2d
|
|
6
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815871f0
Build id: 3.10.0-693.el7.x86_64
|
0x20
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff810324aa
Build id: 3.10.0-693.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f558
Build id: 3.10.0-693.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8117fb8f
Build id: 3.10.0-693.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8117fc4d
Build id: 3.10.0-693.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff8117fead
Build id: 3.10.0-693.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff815822a4
Build id: 3.10.0-693.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff815700a5
Build id: 3.10.0-693.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8157042a
Build id: 3.10.0-693.el7.x86_64
|
0x37a
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [chronyd:640]
Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype xt_conntrack iptable_filter nf_conntrack_netlink binfmt_misc fuse xt_nat veth br_netfilter tun overlay(T) ip_set nfnetlink bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event edac_core iosf_mbi crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi snd_ac97_codec aesni_intel lrw gf128mul ac97_bus glue_helper snd_seq ablk_helper cryptd snd_seq_device snd_pcm ppdev snd_timer vmw_balloon snd soundcore pcspkr joydev sg nfit parport_pc libnvdimm parport shpchp vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif
crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi drm scsi_transport_spi mptscsih ata_piix mptbase libata e1000 crct10dif_pclmul crct10dif_common i2c_core crc32c_intel serio_raw [last unloaded: iptable_filter]
CPU: 2 PID: 640 Comm: chronyd Tainted: G L ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff88037f9b4f10 ti: ffff88036dd6c000 task.ti: ffff88036dd6c000
RIP: 0010:[<ffffffff810f99a6>] [<ffffffff810f99a6>] smp_call_function_many+0x206/0x260
RSP: 0018:ffff88036dd6fd48 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff88037021af48
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff88036dd6fd80 R08: ffff88017afb3800 R09: ffffffff813227d9
R10: ffff880370299c80 R11: ffffea0002cf0400 R12: 0000000000000296
R13: ffff88036dd6fcf8 R14: 0000000000000000 R15: ffff880000000000
FS: 00007f9f1ca4f740(0000) GS:ffff880370280000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd5650e5fd8 CR3: 000000036a241000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
000000016dd6fd58 ffffffff81b1c820 ffffffff815871f0 ffffffff81031c80
0000000000000000 ffffffff815871f1 0000000000000000 ffff88036dd6fda8
ffffffff810f9a5d ffffffff815871f0 ffff88036dd6fdfb 0000000000000005
Call Trace:
[<ffffffff815871f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff815871f1>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff810f9a5d>] on_each_cpu+0x2d/0x60
[<ffffffff815871f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff810324aa>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb8f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc4d>] jump_label_update+0x9d/0xb0
[<ffffffff8117fead>] static_key_slow_inc+0xad/0xb0
[<ffffffff815822a4>] net_enable_timestamp+0x44/0x50
[<ffffffff815700a5>] sock_enable_timestamp+0x45/0x50
[<ffffffff8157042a>] sock_setsockopt+0x37a/0x880
Complete report #209919
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f99a6
Build id: 3.10.0-693.el7.x86_64
|
0x206
|
|
2
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815871f0
Build id: 3.10.0-693.el7.x86_64
|
0x20
|
|
3
|
|
vmlinux |
0xffffffff81031c80
Build id: 3.10.0-693.el7.x86_64
|
0x90
|
|
4
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815871f1
Build id: 3.10.0-693.el7.x86_64
|
0x21
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810f9a5d
Build id: 3.10.0-693.el7.x86_64
|
0x2d
|
|
6
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815871f0
Build id: 3.10.0-693.el7.x86_64
|
0x20
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff810324d4
Build id: 3.10.0-693.el7.x86_64
|
0x94
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f558
Build id: 3.10.0-693.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8117fb8f
Build id: 3.10.0-693.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8117fc4d
Build id: 3.10.0-693.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff8117fead
Build id: 3.10.0-693.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff815822a4
Build id: 3.10.0-693.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff815700a5
Build id: 3.10.0-693.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8157042a
Build id: 3.10.0-693.el7.x86_64
|
0x37a
|
|
15
|
|
vmlinux |
0xffffffff81567340
Build id: 3.10.0-693.el7.x86_64
|
0xa0
|
|
16
|
SyS_setsockopt
|
vmlinux |
0xffffffff8156b3f6
Build id: 3.10.0-693.el7.x86_64
|
0xd6
|
|
17
|
system_call_fastpath
|
vmlinux |
0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [chronyd:786]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg pcspkr vmw_balloon joydev parport_pc parport shpchp vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi scsi_transport_spi ata_piix mptscsih mptbase libata
crct10dif_pclmul crct10dif_common crc32c_intel serio_raw e1000 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 786 Comm: chronyd Tainted: G W L ------------ 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff88081abcbf40 ti: ffff88081a238000 task.ti: ffff88081a238000
RIP: 0010:[<ffffffff810f99a6>] [<ffffffff810f99a6>] smp_call_function_many+0x206/0x260
RSP: 0018:ffff88081a23bd48 EFLAGS: 00000202
RAX: 0000000000000005 RBX: 0000000000017800 RCX: ffff88081ef5af48
RDX: 0000000000000005 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff88081a23bd80 R08: ffff880174b1c800 R09: ffffffff813227d9
R10: ffff88081ee99c80 R11: ffffea002052cc00 R12: ffff88081a23bcf8
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
FS: 00007f7e2c5e9740(0000) GS:ffff88081ee80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f403846e000 CR3: 0000000812f94000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000000011a23bd58 ffffffff81b1c820 ffffffff815871f0 ffffffff81031c80
0000000000000000 ffffffff815871f1 0000000000000000 ffff88081a23bda8
ffffffff810f9a5d ffffffff815871f0 ffff88081a23bdfb 0000000000000005
Call Trace:
[<ffffffff815871f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff815871f1>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff810f9a5d>] on_each_cpu+0x2d/0x60
[<ffffffff815871f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff810324d4>] text_poke_bp+0x94/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb8f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc4d>] jump_label_update+0x9d/0xb0
[<ffffffff8117fead>] static_key_slow_inc+0xad/0xb0
[<ffffffff815822a4>] net_enable_timestamp+0x44/0x50
[<ffffffff815700a5>] sock_enable_timestamp+0x45/0x50
[<ffffffff8157042a>] sock_setsockopt+0x37a/0x880
[<ffffffff81567340>] ? sock_alloc_file+0xa0/0x140
[<ffffffff8156b3f6>] SyS_setsockopt+0xd6/0xf0
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #195227
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
|
0x202
|
|
2
|
|
vmlinux |
0xffffffff815861a2
Build id: 3.10.0-693.el7.x86_64
|
0x22
|
|
3
|
|
vmlinux |
0xffffffff81031c80
Build id: 3.10.0-693.el7.x86_64
|
0x90
|
|
4
|
|
vmlinux |
0xffffffff815861a3
Build id: 3.10.0-693.el7.x86_64
|
0x23
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810f9a5d
Build id: 3.10.0-693.el7.x86_64
|
0x2d
|
|
6
|
|
vmlinux |
0xffffffff815861a2
Build id: 3.10.0-693.el7.x86_64
|
0x22
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff810324aa
Build id: 3.10.0-693.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f558
Build id: 3.10.0-693.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8117fb8f
Build id: 3.10.0-693.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8117fc4d
Build id: 3.10.0-693.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff8117fead
Build id: 3.10.0-693.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff815822a4
Build id: 3.10.0-693.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff815700a5
Build id: 3.10.0-693.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8157042a
Build id: 3.10.0-693.el7.x86_64
|
0x37a
|
|
15
|
SyS_setsockopt
|
vmlinux |
0xffffffff8156b3f6
Build id: 3.10.0-693.el7.x86_64
|
0xd6
|
|
16
|
system_call_fastpath
|
vmlinux |
0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [chronyd:660]
Modules linked in: binfmt_misc tcp_lp fuse veth xt_nat br_netfilter tun overlay(T) ip_set nfnetlink bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm ppdev snd_timer snd vmw_balloon soundcore pcspkr sg joydev nfit parport_pc libnvdimm parport vmw_vmci shpchp i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common ata_piix mptspi libata scsi_transport_spi mptscsih mptbase crc32c_intel e1000 i2c_core serio_raw [last unloaded: iptable_filter]
CPU: 0 PID: 660 Comm: chronyd Tainted: G ------------ T 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff8800b64b1fa0 ti: ffff8800b8150000 task.ti: ffff8800b8150000
RIP: 0010:[<ffffffff810f99a2>] [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8800b8153d48 EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff8800ba69a0e0
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8800b8153d80 R08: ffff880087c75400 R09: ffffffff813227d9
R10: ffff8800ba619c80 R11: ffffea0002c48600 R12: 0000000000000296
R13: ffff8800b8153cf8 R14: ffff8800b8153cb8 R15: 0000000000000296
FS: 00007f9a01370740(0000) GS:ffff8800ba600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f47cb31c000 CR3: 00000000ada2a000 CR4: 00000000003407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
00000001b8153d58 ffffffff81b1c820 ffffffff815861a2 ffffffff81031c80
0000000000000000 ffffffff815861a3 0000000000000000 ffff8800b8153da8
ffffffff810f9a5d ffffffff815861a2 ffff8800b8153dfb 0000000000000005
Call Trace:
[<ffffffff815861a2>] ? netif_rx_internal+0x22/0x110
[<ffffffff81031c80>] ? setup_data_read+0x90/0x90
[<ffffffff815861a3>] ? netif_rx_internal+0x23/0x110
[<ffffffff810f9a5d>] on_each_cpu+0x2d/0x60
[<ffffffff815861a2>] ? netif_rx_internal+0x22/0x110
[<ffffffff810324aa>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f558>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117fb8f>] __jump_label_update+0x5f/0x80
[<ffffffff8117fc4d>] jump_label_update+0x9d/0xb0
[<ffffffff8117fead>] static_key_slow_inc+0xad/0xb0
[<ffffffff815822a4>] net_enable_timestamp+0x44/0x50
[<ffffffff815700a5>] sock_enable_timestamp+0x45/0x50
[<ffffffff8157042a>] sock_setsockopt+0x37a/0x880
[<ffffffff8156b3f6>] SyS_setsockopt+0xd6/0xf0
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89
Complete report #92729
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff86516a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x20a
|
|
2
|
|
vmlinux |
0xffffffff86a4e2b2
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x22
|
|
3
|
|
vmlinux |
0xffffffff86434260
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xd0
|
|
4
|
|
vmlinux |
0xffffffff86a4e2b3
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x23
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff86516b3d
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x2d
|
|
6
|
|
vmlinux |
0xffffffff86a4e2b2
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x22
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff86434ad7
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xb7
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff864316a8
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff865ba8df
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff865ba9bd
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff865bac1d
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff86a49b44
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff86a36705
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff86a36aba
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x3aa
|
|
15
|
|
vmlinux |
0xffffffff86b88678
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x238
|
|
16
|
|
vmlinux |
0xffffffff86624f75
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x35
|
|
17
|
|
vmlinux |
0xffffffff864ccf96
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x26
|
|
18
|
SyS_setsockopt
|
vmlinux |
0xffffffff86a31726
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xe6
|
|
19
|
system_call_fastpath
|
vmlinux |
0xffffffff86b8dede
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x25
|
|
20
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff86b8de21
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xae
|
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [check_icmp:49651]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink iptable_filter ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 vmw_vsock_vmci_transport vsock xt_conntrack nf_conntrack ip6table_filter ip6_tables nfit ppdev vmw_balloon edac_mce_amd libnvdimm crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg joydev parport_pc parport vmw_vmci i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common
crc32c_intel mptspi libata scsi_transport_spi e1000 mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 10 PID: 49651 Comm: check_icmp Kdump: loaded Not tainted 3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9daeb174b150 ti: ffff9dae7deb0000 task.ti: ffff9dae7deb0000
RIP: 0010:[<ffffffff86516a7a>] [<ffffffff86516a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9dae7deb3d18 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00000000000000fc RCX: ffff9daf25660768
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9dae7deb3d50 R08: ffff9daa76536000 R09: ffffffff86781839
R10: ffff9daf2589f160 R11: fffff30400d57e00 R12: 0000000000000296
R13: ffff9dae7deb3cc8 R14: ffffffff864e652b R15: ffffffff8662630a
FS: 00007efd87968740(0000) GS:ffff9daf25880000(0000) knlGS:00000000f7712700
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f8423060624 CR3: 00000005dc732000 CR4: 00000000000407e0
Call Trace:
[<ffffffff86a4e2b2>] ? netif_rx_internal+0x22/0x110
[<ffffffff86434260>] ? setup_data_read+0xd0/0xd0
[<ffffffff86a4e2b3>] ? netif_rx_internal+0x23/0x110
[<ffffffff86516b3d>] on_each_cpu+0x2d/0x60
[<ffffffff86a4e2b2>] ? netif_rx_internal+0x22/0x110
[<ffffffff86434ad7>] text_poke_bp+0xb7/0xf0
[<ffffffff864316a8>] arch_jump_label_transform+0x68/0xb0
[<ffffffff865ba8df>] __jump_label_update+0x5f/0xa0
[<ffffffff865ba9bd>] jump_label_update+0x9d/0xb0
[<ffffffff865bac1d>] static_key_slow_inc+0xad/0xb0
[<ffffffff86a49b44>] net_enable_timestamp+0x44/0x50
[<ffffffff86a36705>] sock_enable_timestamp+0x45/0x50
[<ffffffff86a36aba>] sock_setsockopt+0x3aa/0x8f0
[<ffffffff86b88678>] ? __do_page_fault+0x238/0x500
[<ffffffff86624f75>] ? kmem_cache_alloc+0x35/0x1f0
[<ffffffff864ccf96>] ? prepare_creds+0x26/0x1c0
[<ffffffff86a31726>] SyS_setsockopt+0xe6/0x100
[<ffffffff86b8dede>] system_call_fastpath+0x25/0x2a
[<ffffffff86b8de21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 14 87 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #21611
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f9f72
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x202
|
|
2
|
|
vmlinux |
0xffffffff8156f1b2
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x22
|
|
3
|
|
vmlinux |
0xffffffff81031ca0
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x90
|
|
4
|
|
vmlinux |
0xffffffff8156f1b3
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x23
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810fa02d
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x2d
|
|
6
|
|
vmlinux |
0xffffffff8156f1b2
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x22
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff81032517
Build id: 3.10.0-514.21.1.el7.x86_64
|
0xb7
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f508
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8117dfcf
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8117e08d
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff8117e22d
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x6d
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff8156b494
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff8155bad5
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8155be5a
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x37a
|
|
15
|
|
vmlinux |
0xffffffff81552b60
Build id: 3.10.0-514.21.1.el7.x86_64
|
0xa0
|
|
16
|
SyS_setsockopt
|
vmlinux |
0xffffffff81556c56
Build id: 3.10.0-514.21.1.el7.x86_64
|
0xd6
|
|
17
|
system_call_fastpath
|
vmlinux |
0xffffffff816975c9
Build id: 3.10.0-514.21.1.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [chronyd:607]
Modules linked in: ext4 mbcache jbd2 dm_mirror dm_region_hash dm_log dm_mod bochs_drm iosf_mbi ttm crc32_pclmul ghash_clmulni_intel drm_kms_helper aesni_intel syscopyarea lrw gf128mul sysfillrect sysimgblt glue_helper fb_sys_fops drm ablk_helper i2c_piix4 ppdev cryptd sg virtio_balloon shpchp i2c_core parport_pc parport pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_blk ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw virtio_pci virtio_ring virtio floppy
CPU: 0 PID: 607 Comm: chronyd Tainted: G L ------------ 3.10.0-514.21.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
task: ffff88043f86bec0 ti: ffff8804276f8000 task.ti: ffff8804276f8000
RIP: 0010:[<ffffffff810f9f72>] [<ffffffff810f9f72>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8804276fbd48 EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00000000000000fc RCX: ffff88043fdd9fa0
RDX: 0000000000000007 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8804276fbd80 R08: ffff88017fc3f400 R09: ffffffff813182b9
R10: ffff88043fc19b40 R11: ffffea0010af7000 R12: 0000000000000296
R13: ffff8804276fbcf8 R14: ffffffff810c1f65 R15: ffff8804276fbcb0
FS: 00007fc8e80d9740(0000) GS:ffff88043fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fab4db0a238 CR3: 0000000427688000 CR4: 00000000001407f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000000018106acb8 ffffffff81ae6340 ffffffff8156f1b2 ffffffff81031ca0
0000000000000000 ffffffff8156f1b3 0000000000000000 ffff8804276fbda8
ffffffff810fa02d ffffffff8156f1b2 ffff8804276fbdfb 0000000000000005
Call Trace:
[<ffffffff8156f1b2>] ? netif_rx_internal+0x22/0x120
[<ffffffff81031ca0>] ? setup_data_read+0x90/0x90
[<ffffffff8156f1b3>] ? netif_rx_internal+0x23/0x120
[<ffffffff810fa02d>] on_each_cpu+0x2d/0x60
[<ffffffff8156f1b2>] ? netif_rx_internal+0x22/0x120
[<ffffffff81032517>] text_poke_bp+0xb7/0xf0
[<ffffffff8102f508>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117dfcf>] __jump_label_update+0x5f/0x80
[<ffffffff8117e08d>] jump_label_update+0x9d/0xb0
[<ffffffff8117e22d>] static_key_slow_inc+0x6d/0x90
[<ffffffff8156b494>] net_enable_timestamp+0x44/0x50
[<ffffffff8155bad5>] sock_enable_timestamp+0x45/0x50
[<ffffffff8155be5a>] sock_setsockopt+0x37a/0x940
[<ffffffff81552b60>] ? sock_alloc_file+0xa0/0x140
[<ffffffff81556c56>] SyS_setsockopt+0xd6/0xf0
[<ffffffff816975c9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 66 da 9e 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 99 ad 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 35 da 9e 00 eb b7 0f b6 4d cc 4c 89
Complete report #5962
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f9f76
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x206
|
|
2
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff81570460
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x20
|
|
3
|
|
vmlinux |
0xffffffff81031ce0
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x90
|
|
4
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff81570461
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x21
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810fa02d
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x2d
|
|
6
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff81570460
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x20
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff8103250a
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f508
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8117dfdf
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8117e09d
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff8117e23d
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x6d
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff8156b6a4
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff8155bce5
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8155c06a
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x37a
|
|
15
|
SyS_setsockopt
|
vmlinux |
0xffffffff81556e66
Build id: 3.10.0-514.26.2.el7.x86_64
|
0xd6
|
|
16
|
system_call_fastpath
|
vmlinux |
0xffffffff81697809
Build id: 3.10.0-514.26.2.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [dhclient:20394]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock intel_powerclamp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon pcspkr vmwgfx ttm drm_kms_helper syscopyarea nfit sysfillrect libnvdimm shpchp sg sysimgblt fb_sys_fops drm parport_pc
i2c_piix4 i2c_core vmw_vmci parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ahci ata_piix libahci libata e1000 fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 20394 Comm: dhclient Tainted: G OEL ------------ 3.10.0-514.26.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffff880092348000 ti: ffff88013719c000 task.ti: ffff88013719c000
RIP: 0010:[<ffffffff810f9f76>] [<ffffffff810f9f76>] smp_call_function_many+0x206/0x260
RSP: 0018:ffff88013719fd48 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00000000000000fc RCX: ffff88013fd9a6a0
RDX: 0000000000000003 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff88013719fd80 R08: ffff880139be5c00 R09: ffffffff81318499
R10: ffff88013fd19b40 R11: ffffea0004da3900 R12: 0000000000000296
R13: ffff88013719fcf8 R14: 000280da00000000 R15: ffffffff811dd69b
FS: 00007f24abc67880(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f24aa8d9093 CR3: 0000000079fd8000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000000018106acb8 ffffffff81ae6300 ffffffff81570460 ffffffff81031ce0
0000000000000000 ffffffff81570461 0000000000000000 ffff88013719fda8
ffffffff810fa02d ffffffff81570460 ffff88013719fdfb 0000000000000005
Call Trace:
[<ffffffff81570460>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81031ce0>] ? setup_data_read+0x90/0x90
[<ffffffff81570461>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff810fa02d>] on_each_cpu+0x2d/0x60
[<ffffffff81570460>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff8103250a>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f508>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117dfdf>] __jump_label_update+0x5f/0x80
[<ffffffff8117e09d>] jump_label_update+0x9d/0xb0
[<ffffffff8117e23d>] static_key_slow_inc+0x6d/0x90
[<ffffffff8156b6a4>] net_enable_timestamp+0x44/0x50
[<ffffffff8155bce5>] sock_enable_timestamp+0x45/0x50
[<ffffffff8155c06a>] sock_setsockopt+0x37a/0x940
[<ffffffff81556e66>] SyS_setsockopt+0xd6/0xf0
[<ffffffff81697809>] system_call_fastpath+0x16/0x1b
Code: da 9e 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 80 99 ad 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 f5 d9 9e 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #2836
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810fb6b2
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x202
|
|
2
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815891f0
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x20
|
|
3
|
|
vmlinux |
0xffffffff81031b80
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x90
|
|
4
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815891f1
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x21
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810fb76d
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x2d
|
|
6
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff815891f0
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x20
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff810323aa
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f458
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8118188f
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8118194d
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff81181bad
Build id: 3.10.0-693.11.6.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff815842a4
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff815720a5
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8157242a
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x37a
|
|
15
|
|
vmlinux |
0xffffffff81569340
Build id: 3.10.0-693.11.6.el7.x86_64
|
0xa0
|
|
16
|
SyS_setsockopt
|
vmlinux |
0xffffffff8156d3f6
Build id: 3.10.0-693.11.6.el7.x86_64
|
0xd6
|
|
17
|
system_call_fastpath
|
vmlinux |
0xffffffff816b89fd
Build id: 3.10.0-693.11.6.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [chronyd:745]
Modules linked in: tcp_lp arc4 md4 nls_utf8 cifs dns_resolver fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables xt_multiport vmw_vsock_vmci_transport vsock xt_conntrack iptable_filter iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr i2c_piix4 sg vmw_vmci shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix
crct10dif_pclmul crct10dif_common libata mptspi scsi_transport_spi crc32c_intel mptscsih serio_raw mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 745 Comm: chronyd Tainted: G L ------------ 3.10.0-693.11.6.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff880425e0dee0 ti: ffff88043f9f8000 task.ti: ffff88043f9f8000
RIP: 0010:[<ffffffff810fb6b2>] [<ffffffff810fb6b2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff88043f9fbd18 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff88043fc1c5b8
RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff88043f9fbd50 R08: ffff88017a378400 R09: ffffffff81324989
R10: ffff88043fc5b9c0 R11: ffffea0007d9fd00 R12: 0000000000000296
R13: ffff88043f9fbcc8 R14: 0000000000000000 R15: ffffffff811e220a
FS: 00007f6ef0924740(0000) GS:ffff88043fc40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f048622b180 CR3: 000000042e580000 CR4: 00000000001607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff815891f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81031b80>] ? setup_data_read+0x90/0x90
[<ffffffff815891f1>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff810fb76d>] on_each_cpu+0x2d/0x60
[<ffffffff815891f0>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff810323aa>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f458>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8118188f>] __jump_label_update+0x5f/0x80
[<ffffffff8118194d>] jump_label_update+0x9d/0xb0
[<ffffffff81181bad>] static_key_slow_inc+0xad/0xb0
[<ffffffff815842a4>] net_enable_timestamp+0x44/0x50
[<ffffffff815720a5>] sock_enable_timestamp+0x45/0x50
[<ffffffff8157242a>] sock_setsockopt+0x37a/0x880
[<ffffffff81569340>] ? sock_alloc_file+0xa0/0x140
[<ffffffff8156d3f6>] SyS_setsockopt+0xd6/0xf0
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 56 b8 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 8e b1 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 25 b8 a2 00 eb b7 0f b6 4d cc 4c 89
Complete report #2429
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f98e2
Build id: 3.10.0-514.el7.x86_64
|
0x202
|
|
2
|
?
__netif_receive_skb_core
|
vmlinux |
0xffffffff8156f02f
Build id: 3.10.0-514.el7.x86_64
|
0x2f
|
|
3
|
|
vmlinux |
0xffffffff81031d20
Build id: 3.10.0-514.el7.x86_64
|
0x90
|
|
4
|
?
__netif_receive_skb_core
|
vmlinux |
0xffffffff8156f030
Build id: 3.10.0-514.el7.x86_64
|
0x30
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810f999d
Build id: 3.10.0-514.el7.x86_64
|
0x2d
|
|
6
|
?
__netif_receive_skb_core
|
vmlinux |
0xffffffff8156f02f
Build id: 3.10.0-514.el7.x86_64
|
0x2f
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff8103254a
Build id: 3.10.0-514.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f508
Build id: 3.10.0-514.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8117d90f
Build id: 3.10.0-514.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8117d9cd
Build id: 3.10.0-514.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff8117db6d
Build id: 3.10.0-514.el7.x86_64
|
0x6d
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff8156a7d4
Build id: 3.10.0-514.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff8155ae85
Build id: 3.10.0-514.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8155b20a
Build id: 3.10.0-514.el7.x86_64
|
0x37a
|
|
15
|
|
vmlinux |
0xffffffff81551f50
Build id: 3.10.0-514.el7.x86_64
|
0xa0
|
|
16
|
SyS_setsockopt
|
vmlinux |
0xffffffff81556046
Build id: 3.10.0-514.el7.x86_64
|
0xd6
|
|
17
|
system_call_fastpath
|
vmlinux |
0xffffffff816965c9
Build id: 3.10.0-514.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [chronyd:803]
Modules linked in: ext4 mbcache jbd2 loop arc4 md4 nls_utf8 cifs fuse rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache vmw_vsock_vmci_transport vsock intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg pcspkr vmw_balloon i2c_piix4 shpchp parport_pc parport vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ata_piix ttm vmxnet3 drm crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw mptspi scsi_transport_spi mptscsih mptbase i2c_core floppy fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 803 Comm: chronyd Not tainted 3.10.0-514.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/30/2013
task: ffff880233d86dd0 ti: ffff88023321c000 task.ti: ffff88023321c000
RIP: 0010:[<ffffffff810f98e2>] [<ffffffff810f98e2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff88023321fd48 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0000000000017780 RCX: ffff88023fc1ae48
RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff88023321fd80 R08: ffff880184e1e400 R09: ffffffff81317a89
R10: ffff88023fd59b40 R11: ffffea000735b700 R12: ffffffff813179df
R13: ffffffff81058b23 R14: ffffffffffffff10 R15: 0000000000000297
FS: 00007f914f2f8740(0000) GS:ffff88023fd40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1dc1ce8000 CR3: 0000000230e13000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000000018106ab88 ffffffff81ae20c0 ffffffff8156f02f ffffffff81031d20
0000000000000000 ffffffff8156f030 0000000000000000 ffff88023321fda8
ffffffff810f999d ffffffff8156f02f ffff88023321fdfb 0000000000000005
Call Trace:
[<ffffffff8156f02f>] ? __netif_receive_skb_core+0x2f/0x800
[<ffffffff81031d20>] ? setup_data_read+0x90/0x90
[<ffffffff8156f030>] ? __netif_receive_skb_core+0x30/0x800
[<ffffffff810f999d>] on_each_cpu+0x2d/0x60
[<ffffffff8156f02f>] ? __netif_receive_skb_core+0x2f/0x800
[<ffffffff8103254a>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f508>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117d90f>] __jump_label_update+0x5f/0x80
[<ffffffff8117d9cd>] jump_label_update+0x9d/0xb0
[<ffffffff8117db6d>] static_key_slow_inc+0x6d/0x90
[<ffffffff8156a7d4>] net_enable_timestamp+0x44/0x50
[<ffffffff8155ae85>] sock_enable_timestamp+0x45/0x50
[<ffffffff8155b20a>] sock_setsockopt+0x37a/0x940
[<ffffffff81551f50>] ? sock_alloc_file+0xa0/0x140
[<ffffffff81556046>] SyS_setsockopt+0xd6/0xf0
[<ffffffff816965c9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 76 9e 9e 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 57 ad 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 45 9e 9e 00 eb b7 0f b6 4d cc 4c 89
Complete report #143
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f98e2
Build id: 3.10.0-514.el7.x86_64
|
0x202
|
|
2
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff8156f880
Build id: 3.10.0-514.el7.x86_64
|
0x20
|
|
3
|
|
vmlinux |
0xffffffff81031d20
Build id: 3.10.0-514.el7.x86_64
|
0x90
|
|
4
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff8156f881
Build id: 3.10.0-514.el7.x86_64
|
0x21
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff810f999d
Build id: 3.10.0-514.el7.x86_64
|
0x2d
|
|
6
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff8156f880
Build id: 3.10.0-514.el7.x86_64
|
0x20
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff8103254a
Build id: 3.10.0-514.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8102f508
Build id: 3.10.0-514.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8117d90f
Build id: 3.10.0-514.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8117d9cd
Build id: 3.10.0-514.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff8117db6d
Build id: 3.10.0-514.el7.x86_64
|
0x6d
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff8156a7d4
Build id: 3.10.0-514.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff8155ae85
Build id: 3.10.0-514.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8155b20a
Build id: 3.10.0-514.el7.x86_64
|
0x37a
|
|
15
|
SyS_setsockopt
|
vmlinux |
0xffffffff81556046
Build id: 3.10.0-514.el7.x86_64
|
0xd6
|
|
16
|
system_call_fastpath
|
vmlinux |
0xffffffff816965c9
Build id: 3.10.0-514.el7.x86_64
|
0x16
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [chronyd:830]
Modules linked in: fuse uinput xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_hda_codec_generic iosf_mbi crc32_pclmul snd_hda_intel ghash_clmulni_intel aesni_intel snd_hda_codec lrw gf128mul glue_helper ablk_helper snd_hda_core snd_hwdep snd_seq ppdev snd_seq_device cryptd snd_pcm snd_timer snd pcspkr parport_pc virtio_balloon parport sg i2c_piix4 soundcore nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_blk virtio_net qxl drm_kms_helper crct10dif_pclmul syscopyarea crct10dif_common sysfillrect
crc32c_intel sysimgblt ata_piix fb_sys_fops serio_raw ttm drm virtio_pci virtio_ring virtio libata i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 830 Comm: chronyd Tainted: G L ------------ 3.10.0-514.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff880227a4af10 ti: ffff8802287e0000 task.ti: ffff8802287e0000
RIP: 0010:[<ffffffff810f98e2>] [<ffffffff810f98e2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8802287e3d48 EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff880239d19fa0
RDX: 0000000000000002 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8802287e3d80 R08: ffff880187c3a400 R09: ffffffff81317a89
R10: ffff880239c19b40 R11: ffffea0002b55a00 R12: 0000000000000296
R13: ffff8802287e3cf8 R14: ffffffff8156e7e2 R15: 0000000000000282
FS: 00007f3b84467740(0000) GS:ffff880239c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000c0f01000 CR3: 0000000228f8c000 CR4: 00000000000406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
000000018106ab88 ffffffff81ae20c0 ffffffff8156f880 ffffffff81031d20
0000000000000000 ffffffff8156f881 0000000000000000 ffff8802287e3da8
ffffffff810f999d ffffffff8156f880 ffff8802287e3dfb 0000000000000005
Call Trace:
[<ffffffff8156f880>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff81031d20>] ? setup_data_read+0x90/0x90
[<ffffffff8156f881>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff810f999d>] on_each_cpu+0x2d/0x60
[<ffffffff8156f880>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff8103254a>] text_poke_bp+0x6a/0xf0
[<ffffffff8102f508>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8117d90f>] __jump_label_update+0x5f/0x80
[<ffffffff8117d9cd>] jump_label_update+0x9d/0xb0
[<ffffffff8117db6d>] static_key_slow_inc+0x6d/0x90
[<ffffffff8156a7d4>] net_enable_timestamp+0x44/0x50
[<ffffffff8155ae85>] sock_enable_timestamp+0x45/0x50
[<ffffffff8155b20a>] sock_setsockopt+0x37a/0x940
[<ffffffff81556046>] SyS_setsockopt+0xd6/0xf0
[<ffffffff816965c9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 76 9e 9e 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 57 ad 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 45 9e 9e 00 eb b7 0f b6 4d cc 4c 89
Complete report #532781
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff89916c0a
Build id: 3.10.0-1160.el7.x86_64
|
0x20a
|
|
2
|
|
vmlinux |
0xffffffff89e532f2
Build id: 3.10.0-1160.el7.x86_64
|
0x22
|
|
3
|
|
vmlinux |
0xffffffff89834260
Build id: 3.10.0-1160.el7.x86_64
|
0xd0
|
|
4
|
|
vmlinux |
0xffffffff89e532f3
Build id: 3.10.0-1160.el7.x86_64
|
0x23
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff89916ccd
Build id: 3.10.0-1160.el7.x86_64
|
0x2d
|
|
6
|
|
vmlinux |
0xffffffff89e532f2
Build id: 3.10.0-1160.el7.x86_64
|
0x22
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff89834ad7
Build id: 3.10.0-1160.el7.x86_64
|
0xb7
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff898316a8
Build id: 3.10.0-1160.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff899bae8f
Build id: 3.10.0-1160.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff899baf6d
Build id: 3.10.0-1160.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff899bb1cd
Build id: 3.10.0-1160.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff89e4eb74
Build id: 3.10.0-1160.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff89e3b715
Build id: 3.10.0-1160.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff89e3bc97
Build id: 3.10.0-1160.el7.x86_64
|
0x577
|
|
15
|
|
vmlinux |
0xffffffff89a6f463
Build id: 3.10.0-1160.el7.x86_64
|
0x33
|
|
16
|
SyS_setsockopt
|
vmlinux |
0xffffffff89e36726
Build id: 3.10.0-1160.el7.x86_64
|
0xe6
|
|
17
|
system_call_fastpath
|
vmlinux |
0xffffffff89f93f92
Build id: 3.10.0-1160.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [chronyd:623]
Modules linked in: overlay ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec ppdev ac97_bus aesni_intel snd_seq lrw snd_seq_device gf128mul glue_helper ablk_helper cryptd vmw_balloon snd_pcm joydev snd_timer pcspkr snd vmw_vmci
soundcore sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi scsi_transport_spi mptscsih crct10dif_pclmul e1000 crct10dif_common crc32c_intel mptbase nfit libnvdimm serio_raw drm_panel_orientation_quirks fuse
CPU: 1 PID: 623 Comm: chronyd Tainted: G L ------------ 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff89b175e6e300 ti: ffff89b2b9fa4000 task.ti: ffff89b2b9fa4000
RIP: 0010:[<ffffffff89916c0a>] [<ffffffff89916c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff89b2b9fa7d18 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff89b36f220600
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff89b2b9fa7d50 R08: ffff89b2be34b000 R09: ffffffff89b85f99
R10: ffff89b36f25f160 R11: ffffdb9802e9ea00 R12: 0000000000000296
R13: ffff89b2b9fa7cc8 R14: ffff89b175d6c480 R15: ffff89b2b0bb90e8
FS: 00007f4842d66740(0000) GS:ffff89b36f240000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055cb6c4126d8 CR3: 0000000179fa2000 CR4: 00000000003607e0
Call Trace:
[<ffffffff89e532f2>] ? netif_rx_internal+0x22/0x110
[<ffffffff89834260>] ? setup_data_read+0xd0/0xd0
[<ffffffff89e532f3>] ? netif_rx_internal+0x23/0x110
[<ffffffff89916ccd>] on_each_cpu+0x2d/0x60
[<ffffffff89e532f2>] ? netif_rx_internal+0x22/0x110
[<ffffffff89834ad7>] text_poke_bp+0xb7/0xf0
[<ffffffff898316a8>] arch_jump_label_transform+0x68/0xb0
[<ffffffff899bae8f>] __jump_label_update+0x5f/0xa0
[<ffffffff899baf6d>] jump_label_update+0x9d/0xb0
[<ffffffff899bb1cd>] static_key_slow_inc+0xad/0xb0
[<ffffffff89e4eb74>] net_enable_timestamp+0x44/0x50
[<ffffffff89e3b715>] sock_enable_timestamp+0x45/0x50
[<ffffffff89e3bc97>] sock_setsockopt+0x577/0x8f0
[<ffffffff89a6f463>] ? set_close_on_exec+0x33/0x70
[<ffffffff89e36726>] SyS_setsockopt+0xe6/0x100
[<ffffffff89f93f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 55 8a f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #204237
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8b116eea
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x20a
|
|
2
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff8b656b90
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x20
|
|
3
|
|
vmlinux |
0xffffffff8b034260
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xd0
|
|
4
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff8b656b91
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x21
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff8b116fad
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x2d
|
|
6
|
?
netif_receive_skb_internal
|
vmlinux |
0xffffffff8b656b90
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x20
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff8b034ad7
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xb7
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8b0316a8
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8b1bb12f
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8b1bb20d
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff8b1bb46d
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff8b650784
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff8b63d715
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8b63dc97
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x577
|
|
15
|
|
vmlinux |
0xffffffff8b26f713
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x33
|
|
16
|
SyS_setsockopt
|
vmlinux |
0xffffffff8b638726
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xe6
|
|
17
|
system_call_fastpath
|
vmlinux |
0xffffffff8b795f92
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [chronyd:869]
Modules linked in: vsock_diag tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter ipmi_devintf ipmi_msghandler vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ppdev joydev pcspkr vmw_balloon i2c_piix4 parport_pc parport vmw_vmci binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common e1000e syscopyarea sysfillrect
crc32c_intel serio_raw sysimgblt ahci fb_sys_fops ttm libahci ptp pps_core drm nfit ata_piix vmw_pvscsi libnvdimm libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 5 PID: 869 Comm: chronyd Kdump: loaded Tainted: G L ------------ 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
task: ffff94d30dbb3180 ti: ffff94d305fe4000 task.ti: ffff94d305fe4000
RIP: 0010:[<ffffffff8b116eea>] [<ffffffff8b116eea>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff94d305fe7d18 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00000000000000fc RCX: ffff94d9ffc606a0
RDX: 0000000000000001 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff94d305fe7d50 R08: ffff94d331f72800 R09: ffffffff8b386549
R10: ffff94d9ffd5f160 R11: fffffc7f0a33ab00 R12: 0000000000000296
R13: ffff94d305fe7cc8 R14: 0000000000000000 R15: ffffffff8b22825a
FS: 00007f31720ef740(0000) GS:ffff94d9ffd40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3a3054a8c0 CR3: 0000000143baa000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8b656b90>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff8b034260>] ? setup_data_read+0xd0/0xd0
[<ffffffff8b656b91>] ? netif_receive_skb_internal+0x21/0xc0
[<ffffffff8b116fad>] on_each_cpu+0x2d/0x60
[<ffffffff8b656b90>] ? netif_receive_skb_internal+0x20/0xc0
[<ffffffff8b034ad7>] text_poke_bp+0xb7/0xf0
[<ffffffff8b0316a8>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8b1bb12f>] __jump_label_update+0x5f/0xa0
[<ffffffff8b1bb20d>] jump_label_update+0x9d/0xb0
[<ffffffff8b1bb46d>] static_key_slow_inc+0xad/0xb0
[<ffffffff8b650784>] net_enable_timestamp+0x44/0x50
[<ffffffff8b63d715>] sock_enable_timestamp+0x45/0x50
[<ffffffff8b63dc97>] sock_setsockopt+0x577/0x8f0
[<ffffffff8b26f713>] ? set_close_on_exec+0x33/0x70
[<ffffffff8b638726>] SyS_setsockopt+0xe6/0x100
[<ffffffff8b795f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 fe 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 d5 8b f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 87 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #19251
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8f916c0a
Build id: 3.10.0-1160.el7.x86_64
|
0x20a
|
|
2
|
?
__netif_receive_skb_core
|
vmlinux |
0xffffffff8fe54527
Build id: 3.10.0-1160.el7.x86_64
|
0x37
|
|
3
|
|
vmlinux |
0xffffffff8f834260
Build id: 3.10.0-1160.el7.x86_64
|
0xd0
|
|
4
|
?
__netif_receive_skb_core
|
vmlinux |
0xffffffff8fe54528
Build id: 3.10.0-1160.el7.x86_64
|
0x38
|
|
5
|
on_each_cpu
|
vmlinux |
0xffffffff8f916ccd
Build id: 3.10.0-1160.el7.x86_64
|
0x2d
|
|
6
|
?
__netif_receive_skb_core
|
vmlinux |
0xffffffff8fe54527
Build id: 3.10.0-1160.el7.x86_64
|
0x37
|
|
7
|
text_poke_bp
|
vmlinux |
0xffffffff8f834a8a
Build id: 3.10.0-1160.el7.x86_64
|
0x6a
|
|
8
|
arch_jump_label_transform
|
vmlinux |
0xffffffff8f8316a8
Build id: 3.10.0-1160.el7.x86_64
|
0x68
|
|
9
|
__jump_label_update
|
vmlinux |
0xffffffff8f9bae8f
Build id: 3.10.0-1160.el7.x86_64
|
0x5f
|
|
10
|
jump_label_update
|
vmlinux |
0xffffffff8f9baf6d
Build id: 3.10.0-1160.el7.x86_64
|
0x9d
|
|
11
|
static_key_slow_inc
|
vmlinux |
0xffffffff8f9bb1cd
Build id: 3.10.0-1160.el7.x86_64
|
0xad
|
|
12
|
net_enable_timestamp
|
vmlinux |
0xffffffff8fe4eb74
Build id: 3.10.0-1160.el7.x86_64
|
0x44
|
|
13
|
sock_enable_timestamp
|
vmlinux |
0xffffffff8fe3b715
Build id: 3.10.0-1160.el7.x86_64
|
0x45
|
|
14
|
sock_setsockopt
|
vmlinux |
0xffffffff8fe3bc97
Build id: 3.10.0-1160.el7.x86_64
|
0x577
|
|
15
|
|
vmlinux |
0xffffffff8fa6f463
Build id: 3.10.0-1160.el7.x86_64
|
0x33
|
|
16
|
SyS_setsockopt
|
vmlinux |
0xffffffff8fe36726
Build id: 3.10.0-1160.el7.x86_64
|
0xe6
|
|
17
|
system_call_fastpath
|
vmlinux |
0xffffffff8ff93f92
Build id: 3.10.0-1160.el7.x86_64
|
0x25
|
|
18
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff8ff93ed5
Build id: 3.10.0-1160.el7.x86_64
|
0xa2
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 28s! [chronyd:725]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac snd_hda_codec_generic coretemp snd_hda_intel snd_hda_codec snd_hda_core iosf_mbi crc32_pclmul ghash_clmulni_intel snd_hwdep snd_seq aesni_intel snd_seq_device snd_pcm lrw gf128mul glue_helper snd_timer
ablk_helper snd cryptd soundcore ppdev sg pcspkr joydev vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel e1000 mptspi serio_raw scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks floppy fuse
CPU: 0 PID: 725 Comm: chronyd Kdump: loaded Tainted: G W L ------------ 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
task: ffff8d4a79813180 ti: ffff8d4a79b90000 task.ti: ffff8d4a79b90000
RIP: 0010:[<ffffffff8f916c0a>] [<ffffffff8f916c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8d4a79b93d18 EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000000000001b840 RCX: ffff8d4a7b49fc10
RDX: 0000000000000002 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff8d4a79b93d50 R08: ffff8d4a47d8f400 R09: ffffffff8fb85f99
R10: ffff8d4a7b41f160 R11: ffffe86b81e93800 R12: ffff8d4a79b93cc8
R13: 0000000000000000 R14: ffffffff8f8e5f1b R15: ffff8d4a79b93cb0
FS: 00007fabc1698740(0000) GS:ffff8d4a7b400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002a5e1e8 CR3: 00000000b9b8e000 CR4: 00000000000607f0
Call Trace:
[<ffffffff8fe54527>] ? __netif_receive_skb_core+0x37/0xa10
[<ffffffff8f834260>] ? setup_data_read+0xd0/0xd0
[<ffffffff8fe54528>] ? __netif_receive_skb_core+0x38/0xa10
[<ffffffff8f916ccd>] on_each_cpu+0x2d/0x60
[<ffffffff8fe54527>] ? __netif_receive_skb_core+0x37/0xa10
[<ffffffff8f834a8a>] text_poke_bp+0x6a/0xf0
[<ffffffff8f8316a8>] arch_jump_label_transform+0x68/0xb0
[<ffffffff8f9bae8f>] __jump_label_update+0x5f/0xa0
[<ffffffff8f9baf6d>] jump_label_update+0x9d/0xb0
[<ffffffff8f9bb1cd>] static_key_slow_inc+0xad/0xb0
[<ffffffff8fe4eb74>] net_enable_timestamp+0x44/0x50
[<ffffffff8fe3b715>] sock_enable_timestamp+0x45/0x50
[<ffffffff8fe3bc97>] sock_setsockopt+0x577/0x8f0
[<ffffffff8fa6f463>] ? set_close_on_exec+0x33/0x70
[<ffffffff8fe36726>] SyS_setsockopt+0xe6/0x100
[<ffffffff8ff93f92>] system_call_fastpath+0x25/0x2a
[<ffffffff8ff93ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 55 90 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89