Complete report #768063
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8b9150ce
Build id: 3.10.0-1062.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8b87d398
Build id: 3.10.0-1062.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8b87d534
Build id: 3.10.0-1062.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff8ba03b18
Build id: 3.10.0-1062.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff8b9fd0a6
Build id: 3.10.0-1062.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff8b9feed3
Build id: 3.10.0-1062.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff8b9ff2e6
Build id: 3.10.0-1062.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff8b9fd000
Build id: 3.10.0-1062.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff8b9fc6e0
Build id: 3.10.0-1062.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff8b9c7a50
Build id: 3.10.0-1062.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff8b9c7cc2
Build id: 3.10.0-1062.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc06b9030
Build id: 3.10.0-1062.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff8bb4eea0
Build id: 3.10.0-1062.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffff8b82b59e
Build id: 3.10.0-1062.el7.x86_64
|
0xce
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc06b8ffb
Build id: 3.10.0-1062.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffff8b9c8d31
Build id: 3.10.0-1062.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffff8ba776c0
Build id: 3.10.0-1062.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8ba78254
Build id: 3.10.0-1062.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8ba7855f
Build id: 3.10.0-1062.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffff8ba78a43
Build id: 3.10.0-1062.el7.x86_64
|
0x263
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8ba7963b
Build id: 3.10.0-1062.el7.x86_64
|
0x2cb
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 24s! [kworker/u256:3:100948]
Modules linked in: binfmt_misc nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq ppdev snd_seq_device snd_pcm vmw_balloon snd_timer snd iosf_mbi crc32_pclmul ghash_clmulni_intel sg soundcore aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit libnvdimm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel mptspi libata scsi_transport_spi e1000 mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 100948 Comm: kworker/u256:3 Kdump: loaded Tainted: G L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff93db6966c1c0 ti: ffff93d9be658000 task.ti: ffff93d9be658000
RIP: 0010:[<ffffffff8b9150ce>] [<ffffffff8b9150ce>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff93d9be65b830 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000800008b87d1d0 RCX: ffff93dc3d6e0628
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff93d9be65b868 R08: ffff93cda9664000 R09: ffffffff8bb7f629
R10: ffff93dc3d71f160 R11: fffffd1076d3c400 R12: 0000000000000286
R13: 0000000000000286 R14: ffff93d9be65b7e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff93dc3d700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000f723463c000 CR3: 0000000fe03f8000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffff8b87d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8b87d534>] flush_tlb_page+0x54/0xa0
[<ffffffff8ba03b18>] ptep_clear_flush+0x68/0xa0
[<ffffffff8b9fd0a6>] page_mkclean_one+0xa6/0xe0
[<ffffffff8b9feed3>] rmap_walk+0xf3/0x360
[<ffffffff8b9ff2e6>] page_mkclean+0x76/0xb0
[<ffffffff8b9fd000>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff8b9fc6e0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff8b9c7a50>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8b9c7cc2>] write_cache_pages+0x1f2/0x470
[<ffffffffc06b9030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff8bb4eea0>] ? submit_bio+0x70/0x150
[<ffffffff8b82b59e>] ? __switch_to+0xce/0x580
[<ffffffffc06b8ffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff8b9c8d31>] do_writepages+0x21/0x50
[<ffffffff8ba776c0>] __writeback_single_inode+0x40/0x260
[<ffffffff8ba78254>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8ba7855f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8ba78a43>] wb_writeback+0x263/0x2f0
[<ffffffff8ba7963b>] bdi_writeback_workfn+0x2cb/0x460
Complete report #21235
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb7f1761e
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb7e7e8f8
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffb7e7ea94
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffb8007888
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffb8000e36
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffb8002c43
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffb8003056
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffb8000d90
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffb8000470
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffb7fc9f70
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffb7fca1e2
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc0440310
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffb7ee8b34
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x144
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc04402db
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffffb7fcb261
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffffb807cc40
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x40
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffffb807d7c4
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x1c4
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffb807dacf
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x9f
|
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [kworker/u40:2:31661]
Modules linked in: xt_nat veth dcdbas nf_conntrack_netlink nfnetlink br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay vmw_vsock_vmci_transport vsock xt_comment xt_conntrack iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ext4 mbcache jbd2 sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi crct10dif_pclmul
nfit crct10dif_common scsi_transport_spi crc32c_intel e1000 mptscsih serio_raw libnvdimm mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 12 PID: 31661 Comm: kworker/u40:2 Kdump: loaded Tainted: G L ------------ 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
task: ffff9709a4da8000 ti: ffff97045c390000 task.ti: ffff97045c390000
RIP: 0010:[<ffffffffb7f1761e>] [<ffffffffb7f1761e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff97045c393830 EFLAGS: 00000202
RAX: 0000000000000012 RBX: ffffffffb7e7e730 RCX: ffff970b7f4a07b8
RDX: 0000000000000012 RSI: 0000000000000014 RDI: 0000000000000000
RBP: ffff97045c393868 R08: ffff96fce97b8800 R09: ffffffffb81851b9
R10: ffff970b7f31f160 R11: ffffeded4568d200 R12: 0000000000000282
R13: ffff97045c3937e0 R14: ffff97045c3937d8 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff970b7f300000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0dac86bfb8 CR3: 0000000c3b410000 CR4: 00000000000407e0
Call Trace:
[<ffffffffb7e7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb7e7ea94>] flush_tlb_page+0x54/0xa0
[<ffffffffb8007888>] ptep_clear_flush+0x68/0xa0
[<ffffffffb8000e36>] page_mkclean_one+0xa6/0xe0
[<ffffffffb8002c43>] rmap_walk+0xf3/0x360
[<ffffffffb8003056>] page_mkclean+0x76/0xb0
[<ffffffffb8000d90>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffb8000470>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb7fc9f70>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb7fca1e2>] write_cache_pages+0x1f2/0x470
[<ffffffffc0440310>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffb7ee8b34>] ? find_busiest_group+0x144/0x990
[<ffffffffc04402db>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb7fcb261>] do_writepages+0x21/0x50
[<ffffffffb807cc40>] __writeback_single_inode+0x40/0x260
[<ffffffffb807d7c4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffb807dacf>] __writeback_inodes_wb+0x9f/0xd0