Complete report #1403717
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8ed1173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8ec7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff8ec7a608
Build id: 3.10.0-957.el7.x86_64
|
0x68
|
|
4
|
pmdp_splitting_flush
|
vmlinux |
0xffffffff8ec782c0
Build id: 3.10.0-957.el7.x86_64
|
0x30
|
|
5
|
__split_huge_page
|
vmlinux |
0xffffffff8ee2c1bb
Build id: 3.10.0-957.el7.x86_64
|
0xeb
|
|
6
|
split_huge_page_to_list
|
vmlinux |
0xffffffff8ee2c9c6
Build id: 3.10.0-957.el7.x86_64
|
0x76
|
|
7
|
|
vmlinux |
0xffffffff8ec2a621
Build id: 3.10.0-957.el7.x86_64
|
0x151
|
|
8
|
__split_huge_page_pmd
|
vmlinux |
0xffffffff8ee2d820
Build id: 3.10.0-957.el7.x86_64
|
0x1d0
|
|
9
|
unmap_page_range
|
vmlinux |
0xffffffff8ede728d
Build id: 3.10.0-957.el7.x86_64
|
0xbdd
|
|
10
|
unmap_single_vma
|
vmlinux |
0xffffffff8ede7361
Build id: 3.10.0-957.el7.x86_64
|
0x81
|
|
11
|
?
anon_vma_interval_tree_insert
|
vmlinux |
0xffffffff8eddf347
Build id: 3.10.0-957.el7.x86_64
|
0x97
|
|
12
|
zap_page_range
|
vmlinux |
0xffffffff8ede879d
Build id: 3.10.0-957.el7.x86_64
|
0x11d
|
|
13
|
SyS_madvise
|
vmlinux |
0xffffffff8ede368d
Build id: 3.10.0-957.el7.x86_64
|
0x49d
|
|
14
|
?
poll_select_copy_remaining
|
vmlinux |
0xffffffff8ee57044
Build id: 3.10.0-957.el7.x86_64
|
0x144
|
|
15
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff8f374d21
Build id: 3.10.0-957.el7.x86_64
|
0xae
|
|
16
|
tracesys
|
vmlinux |
0xffffffff8f37505b
Build id: 3.10.0-957.el7.x86_64
|
0xa3
|
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [etcd:17549]
Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_multiport ip_set_hash_ip ip_set_hash_net ipip tunnel4 ip_tunnel veth xt_addrtype xt_set ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipport ip_set_hash_ipportip ip_set dummy iptable_raw xt_CT ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter sunrpc ppdev sg virtio_balloon parport_pc parport i2c_piix4 joydev pcspkr br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack
ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_scsi virtio_net virtio_console virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm floppy drm ata_piix libata serio_raw drm_panel_orientation_quirks virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 9 PID: 17549 Comm: etcd Kdump: loaded Tainted: G ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9f688576b0c0 ti: ffff9f6840bf8000 task.ti: ffff9f6840bf8000
RIP: 0010:[<ffffffff8ed1173a>] [<ffffffff8ed1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9f6840bfba58 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9f68bea20500
RDX: 0000000000000000 RSI: 0000000000000030 RDI: 0000000000000000
RBP: ffff9f6840bfba90 R08: ffff9f5a29525400 R09: ffffffff8ef755c9
R10: ffff9f68bec5f160 R11: ffffea4676d3a400 R12: 0000000000000292
R13: ffff9f6840bfba08 R14: ffff9f688576b0c0 R15: ffffffff8ee1d189
FS: 000000c420072090(0000) GS:ffff9f68bec40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe6092015e0 CR3: 0000000f71bac000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8ec7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8ec7a608>] flush_tlb_mm_range+0x68/0x140
[<ffffffff8ec782c0>] pmdp_splitting_flush+0x30/0x40
[<ffffffff8ee2c1bb>] __split_huge_page+0xeb/0x880
[<ffffffff8ee2c9c6>] split_huge_page_to_list+0x76/0xf0
[<ffffffff8ec2a621>] ? __switch_to+0x151/0x580
[<ffffffff8ee2d820>] __split_huge_page_pmd+0x1d0/0x5c0
[<ffffffff8ede728d>] unmap_page_range+0xbdd/0xc30
[<ffffffff8ede7361>] unmap_single_vma+0x81/0xf0
[<ffffffff8eddf347>] ? anon_vma_interval_tree_insert+0x97/0xa0
[<ffffffff8ede879d>] zap_page_range+0x11d/0x190
[<ffffffff8ede368d>] SyS_madvise+0x49d/0xac0
[<ffffffff8ee57044>] ? poll_select_copy_remaining+0x144/0x180
[<ffffffff8f374d21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff8f37505b>] tracesys+0xa3/0xc9
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 94 8f f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #296008
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffacf1185e
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xfffffffface7a5c8
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xfffffffface7a638
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x68
|
|
4
|
pmdp_splitting_flush
|
vmlinux |
0xfffffffface782f0
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x30
|
|
5
|
__split_huge_page
|
vmlinux |
0xffffffffad02c6cb
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xeb
|
|
6
|
|
vmlinux |
0xffffffffacee04e9
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xa9
|
|
7
|
split_huge_page_to_list
|
vmlinux |
0xffffffffad02ced6
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xfffffffface2a621
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x151
|
|
9
|
__split_huge_page_pmd
|
vmlinux |
0xffffffffad02dd30
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x1d0
|
|
10
|
unmap_page_range
|
vmlinux |
0xffffffffacfe771d
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xbdd
|
|
11
|
unmap_single_vma
|
vmlinux |
0xffffffffacfe77f1
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x81
|
|
12
|
zap_page_range
|
vmlinux |
0xffffffffacfe8c2d
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x11d
|
|
13
|
|
vmlinux |
0xffffffffacf0cdc0
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x90
|
|
14
|
SyS_madvise
|
vmlinux |
0xffffffffacfe3b1d
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x49d
|
|
15
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffad575d21
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xae
|
|
16
|
tracesys
|
vmlinux |
0xffffffffad57605b
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xa3
|
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [.NET ThreadPool:981]
Modules linked in: xt_REDIRECT nf_nat_redirect ip_vs_rr xt_ipvs ip_vs xt_nat veth vxlan ip6_udp_tunnel udp_tunnel xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) sunrpc dm_mirror dm_region_hash dm_log dm_mod ppdev sg parport_pc virtio_balloon i2c_piix4 parport pcspkr joydev ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_scsi virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata virtio_pci serio_raw
virtio_ring floppy virtio drm_panel_orientation_quirks
CPU: 7 PID: 981 Comm: .NET ThreadPool Kdump: loaded Tainted: G L ------------ T 3.10.0-957.10.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9eca75a0a080 ti: ffff9ec3001d0000 task.ti: ffff9ec3001d0000
RIP: 0010:[<ffffffffacf1185e>] [<ffffffffacf1185e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff9ec3001d3a58 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9eca9ce20398
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9ec3001d3a90 R08: ffff9ec3f44ad000 R09: ffffffffad175c99
R10: ffff9eca9cfdf160 R11: fffffeb8df0e8200 R12: 0000000000000292
R13: ffff9ec3001d3a08 R14: ffff9ec3f3e19c18 R15: 0000000000000000
FS: 00007f81a7fff700(0000) GS:ffff9eca9cfc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c4201bd010 CR3: 000000083fb7a000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<fffffffface7a5c8>] native_flush_tlb_others+0xb8/0xc0
[<fffffffface7a638>] flush_tlb_mm_range+0x68/0x140
[<fffffffface782f0>] pmdp_splitting_flush+0x30/0x40
[<ffffffffad02c6cb>] __split_huge_page+0xeb/0x880
[<ffffffffacee04e9>] ? update_cfs_shares+0xa9/0xf0
[<ffffffffad02ced6>] split_huge_page_to_list+0x76/0xf0
[<fffffffface2a621>] ? __switch_to+0x151/0x580
[<ffffffffad02dd30>] __split_huge_page_pmd+0x1d0/0x5c0
[<ffffffffacfe771d>] unmap_page_range+0xbdd/0xc30
[<ffffffffacfe77f1>] unmap_single_vma+0x81/0xf0
[<ffffffffacfe8c2d>] zap_page_range+0x11d/0x190
[<ffffffffacf0cdc0>] ? futex_wake+0x90/0x180
[<ffffffffacfe3b1d>] SyS_madvise+0x49d/0xac0
[<ffffffffad575d21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffad57605b>] tracesys+0xa3/0xc9
Code: 82 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 b9 b4 ad f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 3d 82 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee