Complete report #204241
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb391185a
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb387a5c8
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffb387a764
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffb39fda17
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x67
|
|
5
|
wp_page_copy.isra.71
|
vmlinux |
0xffffffffb39e5a4b
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x30b
|
|
6
|
do_wp_page
|
vmlinux |
0xffffffffb39e7cbb
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xfb
|
|
7
|
|
vmlinux |
0xffffffffb39b688b
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x2b
|
|
8
|
|
vmlinux |
0xffffffffb39e4d09
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x139
|
|
9
|
handle_pte_fault
|
vmlinux |
0xffffffffb39e964d
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x37d
|
|
10
|
|
vmlinux |
0xffffffffb38a3c22
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x92
|
|
11
|
handle_mm_fault
|
vmlinux |
0xffffffffb39ec0fd
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x39d
|
|
12
|
__do_page_fault
|
vmlinux |
0xffffffffb3f705e3
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x203
|
|
13
|
trace_do_page_fault
|
vmlinux |
0xffffffffb3f709c6
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x56
|
|
14
|
do_async_page_fault
|
vmlinux |
0xffffffffb3f6ff42
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x22
|
|
15
|
async_page_fault
|
vmlinux |
0xffffffffb3f6c788
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x28
|
NMI watchdog: BUG: soft lockup - CPU#28 stuck for 30s! [runc:59111]
Modules linked in: ip_vs_rr xt_ipvs ip_vs xt_REDIRECT nf_nat_redirect xt_nat veth vxlan ip6_udp_tunnel udp_tunnel xt_mark nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) sunrpc dm_mirror dm_region_hash dm_log dm_mod sg ppdev virtio_balloon joydev parport_pc parport i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm virtio_pci virtio_ring serio_raw
libata floppy drm_panel_orientation_quirks virtio
CPU: 28 PID: 59111 Comm: runc Kdump: loaded Tainted: G L ------------ T 3.10.0-957.10.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9f88f4fb1040 ti: ffff9f9084a48000 task.ti: ffff9f9084a48000
RIP: 0010:[<ffffffffb391185a>] [<ffffffffb391185a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff9f9084a4bb40 EFLAGS: 00000202
RAX: 0000000000000013 RBX: 00080000b387a400 RCX: ffff9f90dd2e07f8
RDX: 0000000000000013 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9f9084a4bb78 R08: ffff9f8a343f2400 R09: ffffffffb3b75c99
R10: ffff9f90dd51f160 R11: fffff1669fa8ec00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff9f9084a4baf0 R15: ffffffffffffffff
FS: 00007f681b561740(0000) GS:ffff9f90dd500000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c42013c150 CR3: 0000000808786000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb387a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb387a764>] flush_tlb_page+0x54/0xa0
[<ffffffffb39fda17>] ptep_clear_flush+0x67/0xa0
[<ffffffffb39e5a4b>] wp_page_copy.isra.71+0x30b/0x560
[<ffffffffb39e7cbb>] do_wp_page+0xfb/0x720
[<ffffffffb39b688b>] ? unlock_page+0x2b/0x30
[<ffffffffb39e4d09>] ? do_read_fault.isra.61+0x139/0x1b0
[<ffffffffb39e964d>] handle_pte_fault+0x37d/0xd10
[<ffffffffb38a3c22>] ? proc_put_long+0x92/0xd0
[<ffffffffb39ec0fd>] handle_mm_fault+0x39d/0x9b0
[<ffffffffb3f705e3>] __do_page_fault+0x203/0x500
[<ffffffffb3f709c6>] trace_do_page_fault+0x56/0x150
[<ffffffffb3f6ff42>] do_async_page_fault+0x22/0xf0
[<ffffffffb3f6c788>] async_page_fault+0x28/0x30
Code: 48 63 35 6e 82 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 b9 54 b4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 3d 82 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1421518
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff810f98e2
Build id: 3.10.0-514.el7.x86_64
|
0x202
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8106c4c8
Build id: 3.10.0-514.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8106c6c5
Build id: 3.10.0-514.el7.x86_64
|
0x55
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff811c0d74
Build id: 3.10.0-514.el7.x86_64
|
0x54
|
|
5
|
wp_page_copy.isra.56
|
vmlinux |
0xffffffff811acffb
Build id: 3.10.0-514.el7.x86_64
|
0x24b
|
|
6
|
do_wp_page
|
vmlinux |
0xffffffff811aeba5
Build id: 3.10.0-514.el7.x86_64
|
0xe5
|
|
7
|
|
vmlinux |
0xffffffff811ac6d4
Build id: 3.10.0-514.el7.x86_64
|
0xe4
|
|
8
|
handle_mm_fault
|
vmlinux |
0xffffffff811b0e3c
Build id: 3.10.0-514.el7.x86_64
|
0x72c
|
|
9
|
|
vmlinux |
0xffffffff810c4cf8
Build id: 3.10.0-514.el7.x86_64
|
0x1c8
|
|
10
|
|
vmlinux |
0xffffffff8161d516
Build id: 3.10.0-514.el7.x86_64
|
0x36
|
|
11
|
__do_page_fault
|
vmlinux |
0xffffffff81691a94
Build id: 3.10.0-514.el7.x86_64
|
0x154
|
|
12
|
trace_do_page_fault
|
vmlinux |
0xffffffff81691e76
Build id: 3.10.0-514.el7.x86_64
|
0x56
|
|
13
|
do_async_page_fault
|
vmlinux |
0xffffffff8169151b
Build id: 3.10.0-514.el7.x86_64
|
0x1b
|
|
14
|
async_page_fault
|
vmlinux |
0xffffffff8168e0b8
Build id: 3.10.0-514.el7.x86_64
|
0x28
|
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kubelet:5103]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter sunrpc snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep bochs_drm snd_seq ttm snd_seq_device drm_kms_helper snd_pcm iTCO_wdt syscopyarea virtio_balloon virtio_net iTCO_vendor_support sysfillrect snd_timer sysimgblt fb_sys_fops virtio_pci virtio_ring drm sg snd pcspkr virtio i2c_i801 lpc_ich shpchp soundcore i2c_core ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_common ahci libahci libata serio_raw vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod
CPU: 10 PID: 5103 Comm: kubelet Tainted: G L ------------ 3.10.0-514.el7.x86_64 #1
Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
task: ffff8800347e9f60 ti: ffff8807a5e8c000 task.ti: ffff8807a5e8c000
RIP: 0010:[<ffffffff810f98e2>] [<ffffffff810f98e2>] smp_call_function_many+0x202/0x260
RSP: 0000:ffff8807a5e8fc30 EFLAGS: 00000202
RAX: 000000000000001b RBX: 000800008106c300 RCX: ffff88085f4daef8
RDX: 000000000000001b RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff8807a5e8fc68 R08: ffff88017c9ad800 R09: ffffffff81317a89
R10: ffff88085f099b40 R11: ffffea001f83ca00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8807a5e8fbe0 R15: ffffffffffffffff
FS: 00007fc64b4be740(0000) GS:ffff88085f080000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c4209ff008 CR3: 0000000830fb7000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001a5e8fc88 ffff88085a2ba288 ffff88085a2b9f40 000000c4209ff008
0000000000000000 ffff88085a2ba288 ffff88083335eff8 ffff8807a5e8fcb8
ffffffff8106c4c8 ffff88085a2b9f40 000000c4209ff008 0000000000000000
Call Trace:
[<ffffffff8106c4c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106c6c5>] flush_tlb_page+0x55/0xa0
[<ffffffff811c0d74>] ptep_clear_flush+0x54/0x80
[<ffffffff811acffb>] wp_page_copy.isra.56+0x24b/0x470
[<ffffffff811aeba5>] do_wp_page+0xe5/0x530
[<ffffffff811ac6d4>] ? do_read_fault.isra.42+0xe4/0x130
[<ffffffff811b0e3c>] handle_mm_fault+0x72c/0xfe0
[<ffffffff810c4cf8>] ? try_to_wake_up+0x1c8/0x330
[<ffffffff8161d516>] ? unix_peer_get+0x36/0x40
[<ffffffff81691a94>] __do_page_fault+0x154/0x450
[<ffffffff81691e76>] trace_do_page_fault+0x56/0x150
[<ffffffff8169151b>] do_async_page_fault+0x1b/0xd0
[<ffffffff8168e0b8>] async_page_fault+0x28/0x30
Code: 48 63 35 76 9e 9e 00 89 c2 39 f0 0f 8d 86 fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 57 ad 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 45 9e 9e 00 eb b7 0f b6 4d cc 4c 89