Info

Permalink
Function
smp_call_function_many
First occurrence
2020-09-28
Last occurrence
2026-02-03
State
NEW
Components
kernel
Quality
-46

Statistics

Operating system Count
CentOS 7 347
Architecture Count
x86_64 347
Related packages Count
kernel 347
 0:3.10.0-693.el7 213
 0:3.10.0-862.el7 93
 0:3.10.0-693.21.1.el7 18
 0:3.10.0-862.9.1.el7 9
 0:3.10.0-862.6.3.el7 6
 0:3.10.0-862.14.4.el7 3
 0:3.10.0-862.3.3.el7 2
 0:3.10.0-693.11.1.el7 1
 0:3.10.0-862.11.6.el7 1
 0:3.10.0-862.3.2.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #826003
Frame # Function Binary Source or offset Line
1
ptep_clear_flush
vmlinux 0xffffffff825d87d0
Build id: 3.10.0-862.el7.x86_64
-
2
? page_mkclean
vmlinux 0xffffffff825d1ba4
Build id: 3.10.0-862.el7.x86_64
0x144
3
clear_page_dirty_for_io
vmlinux 0xffffffff8259f7d0
Build id: 3.10.0-862.el7.x86_64
0x50
4
write_cache_pages
vmlinux 0xffffffff8259fa7a
Build id: 3.10.0-862.el7.x86_64
0x22a
5
? xfs_aops_discard_page
xfs 0xffffffffc03ad850
Build id: 3.10.0-862.el7.x86_64
0x150
6
? submit_bio
vmlinux 0xffffffff8271a940
Build id: 3.10.0-862.el7.x86_64
0x70
7
xfs_vm_writepages
xfs 0xffffffffc03ad65d
Build id: 3.10.0-862.el7.x86_64
0xbd
8
do_writepages
vmlinux 0xffffffff825a0e21
Build id: 3.10.0-862.el7.x86_64
0x21
9
__writeback_single_inode
vmlinux 0xffffffff82648af0
Build id: 3.10.0-862.el7.x86_64
0x40
10
writeback_sb_inodes
vmlinux 0xffffffff82649584
Build id: 3.10.0-862.el7.x86_64
0x1c4
11
__writeback_inodes_wb
vmlinux 0xffffffff826498ef
Build id: 3.10.0-862.el7.x86_64
0x9f
12
wb_writeback
vmlinux 0xffffffff8264a123
Build id: 3.10.0-862.el7.x86_64
0x263
13
bdi_writeback_workfn
vmlinux 0xffffffff8264aaab
Build id: 3.10.0-862.el7.x86_64
0x2cb
14
process_one_work
vmlinux 0xffffffff824b2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
15
worker_thread
vmlinux 0xffffffff824b3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
16
? manage_workers.isra.24
vmlinux 0xffffffff824b39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
17
kthread
vmlinux 0xffffffff824bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
18
? insert_kthread_work
vmlinux 0xffffffff824bad60
Build id: 3.10.0-862.el7.x86_64
0x40
19
ret_from_fork_nospec_begin
vmlinux 0xffffffff82b1f61d
Build id: 3.10.0-862.el7.x86_64
0x7
20
? insert_kthread_work
vmlinux 0xffffffff824bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [kworker/u8:2:31434]
Modules linked in: veth binfmt_misc xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr ppdev nfit joydev vmw_balloon libnvdimm vmw_vmci parport_pc shpchp parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea ahci sysfillrect sysimgblt fb_sys_fops sd_mod ttm crc_t10dif crct10dif_generic libahci ata_piix drm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw i2c_core vmxnet3 vmw_pvscsi dm_mirror dm_region_hash
dm_log dm_mod
CPU: 1 PID: 31434 Comm: kworker/u8:2 Kdump: loaded Tainted: G               ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
task: ffff8a57c8fc1fa0 ti: ffff8a584c6fc000 task.ti: ffff8a584c6fc000
RIP: 0010:[<ffffffff825d87d0>]  [<ffffffff825d87d0>] ptep_clear_flush+0x0/0xa0
RSP: 0000:ffff8a584c6ff9a0  EFLAGS: 00010206
RAX: 8000000000000867 RBX: 00003ffffffff000 RCX: 8000000000000867
RDX: ffff8a54cc654720 RSI: 00007f79ba8e4000 RDI: ffff8a57c2f5d438
RBP: ffff8a584c6ff9f8 R08: 00000000000413d9 R09: 000000000c654000
R10: 000000000000bcc2 R11: 0000000000000001 R12: 00003ffffffff000
R13: 0000000000000000 R14: 0000000000000000 R15: 000000000c654000
FS:  0000000000000000(0000) GS:ffff8a58ffc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6cba19f000 CR3: 00000003eb762000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffff825d1ba4>] ? page_mkclean+0x144/0x210
[<ffffffff8259f7d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8259fa7a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc03ad850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff8271a940>] ? submit_bio+0x70/0x150
[<ffffffffc03ad65d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffff825a0e21>] do_writepages+0x21/0x50
[<ffffffff82648af0>] __writeback_single_inode+0x40/0x260
[<ffffffff82649584>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff826498ef>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8264a123>] wb_writeback+0x263/0x2f0
[<ffffffff8264aaab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff824b2dff>] process_one_work+0x17f/0x440
[<ffffffff824b3ac6>] worker_thread+0x126/0x3c0
[<ffffffff824b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff824bae31>] kthread+0xd1/0xe0
[<ffffffff824bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff82b1f61d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff824bad60>] ? insert_kthread_work+0x40/0x40
Code: ff ff e8 84 8c eb ff 48 8b 7d b0 4c 89 95 78 ff ff ff e8 f4 96 fe ff 4c 8b 95 78 ff ff ff e9 16 fc ff ff 0f 1f 84 00 00 00 00 00 <0f> 1f 44 00 00 55 48 89 e5 41 56 41 55 49 89 f5 41 54 49 89 fc 
Complete report #1383336
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8106e864
Build id: 3.10.0-693.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff811c2524
Build id: 3.10.0-693.el7.x86_64
0x54
5
page_mkclean
vmlinux 0xffffffff811bbf2e
Build id: 3.10.0-693.el7.x86_64
0x12e
6
clear_page_dirty_for_io
vmlinux 0xffffffff8118d9c0
Build id: 3.10.0-693.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffff8118dc6a
Build id: 3.10.0-693.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc02b3980
Build id: 3.10.0-693.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffff812f8dc0
Build id: 3.10.0-693.el7.x86_64
0x70
10
xfs_vm_writepages
xfs 0xffffffffc02b378d
Build id: 3.10.0-693.el7.x86_64
0xbd
11
do_writepages
vmlinux 0xffffffff8118effe
Build id: 3.10.0-693.el7.x86_64
0x1e
12
__writeback_single_inode
vmlinux 0xffffffff8122d7d0
Build id: 3.10.0-693.el7.x86_64
0x40
13
writeback_sb_inodes
vmlinux 0xffffffff8122e414
Build id: 3.10.0-693.el7.x86_64
0x1c4
14
__writeback_inodes_wb
vmlinux 0xffffffff8122e77f
Build id: 3.10.0-693.el7.x86_64
0x9f
15
wb_writeback
vmlinux 0xffffffff8122efb3
Build id: 3.10.0-693.el7.x86_64
0x263
16
bdi_writeback_workfn
vmlinux 0xffffffff8122f45b
Build id: 3.10.0-693.el7.x86_64
0x2cb
17
process_one_work
vmlinux 0xffffffff810a881a
Build id: 3.10.0-693.el7.x86_64
0x17a
18
worker_thread
vmlinux 0xffffffff810a94e6
Build id: 3.10.0-693.el7.x86_64
0x126
19
? manage_workers.isra.24
vmlinux 0xffffffff810a93c0
Build id: 3.10.0-693.el7.x86_64
0x2a0
20
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
21
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
22
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
23
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kworker/u34:1:15321]
Modules linked in: tcp_diag inet_diag xt_iprange nf_conntrack_netlink xt_NFLOG nfnetlink_log fuse binfmt_misc netfilter_edr(OE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun xt_nat xt_mark ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 coretemp ppdev vmw_balloon joydev pcspkr sg vmw_vmci i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd
grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic crct10dif_common pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi drm scsi_transport_spi ata_piix serio_raw mptscsih libata e1000 mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 8 PID: 15321 Comm: kworker/u34:1 Tainted: G           OEL ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8808bd7fbf40 ti: ffff880da42d4000 task.ti: ffff880da42d4000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff880da42d78c0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffff8106e4a0 RCX: ffff880666c1b138
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff880da42d78f8 R08: ffff8806f696fc00 R09: ffffffff813227d9
R10: ffff880ca6c19c80 R11: ffffea003286b000 R12: 0000000000000282
R13: ffff880da42d7870 R14: ffffffff81349a50 R15: ffff880da42d7860
FS:  0000000000000000(0000) GS:ffff880ca6c00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000070831e040 CR3: 00000000019f2000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000100000015 ffff8806646528c8 ffff880664652580 00007f4ba5a2d000
0000000000000000 ffff8806646528c8 ffff88005645b168 ffff880da42d7948
ffffffff8106e668 ffff880664652580 00007f4ba5a2d000 0000000000000000
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e864>] flush_tlb_page+0x54/0xa0
[<ffffffff811c2524>] ptep_clear_flush+0x54/0x80
[<ffffffff811bbf2e>] page_mkclean+0x12e/0x1f0
[<ffffffff8118d9c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8118dc6a>] write_cache_pages+0x22a/0x4d0
[<ffffffffc02b3980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff812f8dc0>] ? submit_bio+0x70/0x150
[<ffffffffc02b378d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffff8118effe>] do_writepages+0x1e/0x40
[<ffffffff8122d7d0>] __writeback_single_inode+0x40/0x220
[<ffffffff8122e414>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff8122e77f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8122efb3>] wb_writeback+0x263/0x2f0
[<ffffffff8122f45b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1137316
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa3d0862a
Build id: 3.10.0-862.6.3.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa3c77788
Build id: 3.10.0-862.6.3.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffa3c77924
Build id: 3.10.0-862.6.3.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffa3dd8ff7
Build id: 3.10.0-862.6.3.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffffa3dd2364
Build id: 3.10.0-862.6.3.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffffa3d9ff90
Build id: 3.10.0-862.6.3.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffffa3da023a
Build id: 3.10.0-862.6.3.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc0405850
Build id: 3.10.0-862.6.3.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffffa3f1b380
Build id: 3.10.0-862.6.3.el7.x86_64
0x70
10
? cpumask_next_and
vmlinux 0xffffffffa3f4acb5
Build id: 3.10.0-862.6.3.el7.x86_64
0x35
11
xfs_vm_writepages
xfs 0xffffffffc040565d
Build id: 3.10.0-862.6.3.el7.x86_64
0xbd
12
do_writepages
vmlinux 0xffffffffa3da15e1
Build id: 3.10.0-862.6.3.el7.x86_64
0x21
13
__writeback_single_inode
vmlinux 0xffffffffa3e49260
Build id: 3.10.0-862.6.3.el7.x86_64
0x40
14
writeback_sb_inodes
vmlinux 0xffffffffa3e49cf4
Build id: 3.10.0-862.6.3.el7.x86_64
0x1c4
15
__writeback_inodes_wb
vmlinux 0xffffffffa3e4a05f
Build id: 3.10.0-862.6.3.el7.x86_64
0x9f
16
wb_writeback
vmlinux 0xffffffffa3e4a893
Build id: 3.10.0-862.6.3.el7.x86_64
0x263
17
? get_nr_inodes
vmlinux 0xffffffffa3e36dcc
Build id: 3.10.0-862.6.3.el7.x86_64
0x4c
18
bdi_writeback_workfn
vmlinux 0xffffffffa3e4b21b
Build id: 3.10.0-862.6.3.el7.x86_64
0x2cb
19
process_one_work
vmlinux 0xffffffffa3cb35ef
Build id: 3.10.0-862.6.3.el7.x86_64
0x17f
20
worker_thread
vmlinux 0xffffffffa3cb4686
Build id: 3.10.0-862.6.3.el7.x86_64
0x126
21
? manage_workers.isra.24
vmlinux 0xffffffffa3cb4560
Build id: 3.10.0-862.6.3.el7.x86_64
0x2a0
22
kthread
vmlinux 0xffffffffa3cbb621
Build id: 3.10.0-862.6.3.el7.x86_64
0xd1
23
? insert_kthread_work
vmlinux 0xffffffffa3cbb550
Build id: 3.10.0-862.6.3.el7.x86_64
0x40
24
ret_from_fork_nospec_begin
vmlinux 0xffffffffa43205f7
Build id: 3.10.0-862.6.3.el7.x86_64
0x21
25
? insert_kthread_work
vmlinux 0xffffffffa3cbb550
Build id: 3.10.0-862.6.3.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u16:2:14150]
Modules linked in: tcp_diag udp_diag inet_diag sg ppdev i2c_piix4 parport_pc pcspkr parport joydev ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata virtio_blk e1000 i2c_core virtio_pci floppy virtio_ring serio_raw virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 14150 Comm: kworker/u16:2 Kdump: loaded Tainted: G             L ------------   3.10.0-862.6.3.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff94686ee8eeb0 ti: ffff94688165c000 task.ti: ffff94688165c000
RIP: 0010:[<ffffffffa3d0862a>]  [<ffffffffa3d0862a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff94688165f8b8  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 00000000000196c0 RCX: ffff94689fd9bfa0
RDX: 0000000000000006 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff94688165f8f0 R08: ffff94677fd47000 R09: ffffffffa3f4ae19
R10: ffff94689fc1bb40 R11: ffffcedc43614c00 R12: ffffffffa3f4ad6f
R13: ffff946636448000 R14: ffff94688939b418 R15: ffff94688939b400
FS:  0000000000000000(0000) GS:ffff94689fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fda8dae3000 CR3: 00000000d876a000 CR4: 00000000000006f0
Call Trace:
[<ffffffffa3c77788>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa3c77924>] flush_tlb_page+0x54/0xa0
[<ffffffffa3dd8ff7>] ptep_clear_flush+0x67/0xa0
[<ffffffffa3dd2364>] page_mkclean+0x144/0x210
[<ffffffffa3d9ff90>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa3da023a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc0405850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffa3f1b380>] ? submit_bio+0x70/0x150
[<ffffffffa3f4acb5>] ? cpumask_next_and+0x35/0x50
[<ffffffffc040565d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffffa3da15e1>] do_writepages+0x21/0x50
[<ffffffffa3e49260>] __writeback_single_inode+0x40/0x260
[<ffffffffa3e49cf4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffa3e4a05f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa3e4a893>] wb_writeback+0x263/0x2f0
[<ffffffffa3e36dcc>] ? get_nr_inodes+0x4c/0x70
[<ffffffffa3e4b21b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa3cb35ef>] process_one_work+0x17f/0x440
[<ffffffffa3cb4686>] worker_thread+0x126/0x3c0
[<ffffffffa3cb4560>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffa3cbb621>] kthread+0xd1/0xe0
[<ffffffffa3cbb550>] ? insert_kthread_work+0x40/0x40
[<ffffffffa43205f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa3cbb550>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 9e ae c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 93 a4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d ae c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #852706
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb4307e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb4277018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffb42771b4
Build id: 3.10.0-862.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffb43d8837
Build id: 3.10.0-862.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffffb43d1ba4
Build id: 3.10.0-862.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffffb439f7d0
Build id: 3.10.0-862.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffffb439fa7a
Build id: 3.10.0-862.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc05b5850
Build id: 3.10.0-862.el7.x86_64
0x150
9
? update_curr
vmlinux 0xffffffffb42d816c
Build id: 3.10.0-862.el7.x86_64
0x14c
10
? __switch_to
vmlinux 0xffffffffb422959e
Build id: 3.10.0-862.el7.x86_64
0xce
11
xfs_vm_writepages
xfs 0xffffffffc05b565d
Build id: 3.10.0-862.el7.x86_64
0xbd
12
do_writepages
vmlinux 0xffffffffb43a0e21
Build id: 3.10.0-862.el7.x86_64
0x21
13
__writeback_single_inode
vmlinux 0xffffffffb4448af0
Build id: 3.10.0-862.el7.x86_64
0x40
14
writeback_sb_inodes
vmlinux 0xffffffffb4449584
Build id: 3.10.0-862.el7.x86_64
0x1c4
15
__writeback_inodes_wb
vmlinux 0xffffffffb44498ef
Build id: 3.10.0-862.el7.x86_64
0x9f
16
bdi_writeback_workfn
vmlinux 0xffffffffb444abb6
Build id: 3.10.0-862.el7.x86_64
0x3d6
17
process_one_work
vmlinux 0xffffffffb42b2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
18
rescuer_thread
vmlinux 0xffffffffb42b3354
Build id: 3.10.0-862.el7.x86_64
0x294
19
? process_one_work
vmlinux 0xffffffffb42b30c0
Build id: 3.10.0-862.el7.x86_64
0x440
20
kthread
vmlinux 0xffffffffb42bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
21
? insert_kthread_work
vmlinux 0xffffffffb42bad60
Build id: 3.10.0-862.el7.x86_64
0x40
22
ret_from_fork_nospec_begin
vmlinux 0xffffffffb491f637
Build id: 3.10.0-862.el7.x86_64
0x21
23
? insert_kthread_work
vmlinux 0xffffffffb42bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [writeback:31]
Modules linked in: xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec crc32_pclmul ac97_bus snd_seq ghash_clmulni_intel ppdev snd_seq_device snd_pcm vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev vmw_vmci pcspkr snd_timer i2c_piix4 snd soundcore shpchp nfit libnvdimm parport_pc parport ip_tables xfs libcrc32c sr_mod
cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi drm ata_piix mptscsih crct10dif_pclmul crct10dif_common libata e1000 crc32c_intel mptbase serio_raw i2c_core
CPU: 3 PID: 31 Comm: writeback Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff89acf9982f70 ti: ffff89afaede4000 task.ti: ffff89afaede4000
RIP: 0010:[<ffffffffb4307e3a>]  [<ffffffffb4307e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff89afaede78e8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff89afad61cf00
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff89afaede7920 R08: ffff89acf9cabc00 R09: ffffffffb454a3d9
R10: ffff89afad6dbb00 R11: ffffefcb41b2ce00 R12: 0000000000000292
R13: ffff89afaede7898 R14: ffff89afae1b70d8 R15: ffff89afa6cf5a40
FS:  0000000000000000(0000) GS:ffff89afad6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6871fb7d00 CR3: 000000041b3b0000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffb4277018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb42771b4>] flush_tlb_page+0x54/0xa0
[<ffffffffb43d8837>] ptep_clear_flush+0x67/0xa0
[<ffffffffb43d1ba4>] page_mkclean+0x144/0x210
[<ffffffffb439f7d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb439fa7a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc05b5850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffb42d816c>] ? update_curr+0x14c/0x1e0
[<ffffffffb422959e>] ? __switch_to+0xce/0x580
[<ffffffffc05b565d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffffb43a0e21>] do_writepages+0x21/0x50
[<ffffffffb4448af0>] __writeback_single_inode+0x40/0x260
[<ffffffffb4449584>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffb44498ef>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb444abb6>] bdi_writeback_workfn+0x3d6/0x460
[<ffffffffb42b2dff>] process_one_work+0x17f/0x440
[<ffffffffb42b3354>] rescuer_thread+0x294/0x3c0
[<ffffffffb42b30c0>] ? process_one_work+0x440/0x440
[<ffffffffb42bae31>] kthread+0xd1/0xe0
[<ffffffffb42bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb491f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb42bad60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 f3 b4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #704868
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa9b07e3e
Build id: 3.10.0-862.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffa9a77018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffa9a771b4
Build id: 3.10.0-862.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffa9bd8837
Build id: 3.10.0-862.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffffa9bd1ba4
Build id: 3.10.0-862.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffffa9b9f7d0
Build id: 3.10.0-862.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffffa9b9fa7a
Build id: 3.10.0-862.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc037e850
Build id: 3.10.0-862.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffffa9d1a940
Build id: 3.10.0-862.el7.x86_64
0x70
10
xfs_vm_writepages
xfs 0xffffffffc037e65d
Build id: 3.10.0-862.el7.x86_64
0xbd
11
do_writepages
vmlinux 0xffffffffa9ba0e21
Build id: 3.10.0-862.el7.x86_64
0x21
12
__writeback_single_inode
vmlinux 0xffffffffa9c48af0
Build id: 3.10.0-862.el7.x86_64
0x40
13
writeback_sb_inodes
vmlinux 0xffffffffa9c49584
Build id: 3.10.0-862.el7.x86_64
0x1c4
14
__writeback_inodes_wb
vmlinux 0xffffffffa9c498ef
Build id: 3.10.0-862.el7.x86_64
0x9f
15
bdi_writeback_workfn
vmlinux 0xffffffffa9c4abb6
Build id: 3.10.0-862.el7.x86_64
0x3d6
16
process_one_work
vmlinux 0xffffffffa9ab2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
17
rescuer_thread
vmlinux 0xffffffffa9ab3354
Build id: 3.10.0-862.el7.x86_64
0x294
18
? process_one_work
vmlinux 0xffffffffa9ab30c0
Build id: 3.10.0-862.el7.x86_64
0x440
19
kthread
vmlinux 0xffffffffa9abae31
Build id: 3.10.0-862.el7.x86_64
0xd1
20
? insert_kthread_work
vmlinux 0xffffffffa9abad60
Build id: 3.10.0-862.el7.x86_64
0x40
21
ret_from_fork_nospec_begin
vmlinux 0xffffffffaa11f637
Build id: 3.10.0-862.el7.x86_64
0x21
22
? insert_kthread_work
vmlinux 0xffffffffa9abad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [writeback:51]
Modules linked in: tcp_lp fuse binfmt_misc af_packet_diag netlink_diag vsock_diag vsock sctp_diag sctp tcp_diag udp_diag inet_diag unix_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink sunrpc ext4 mbcache jbd2 ppdev parport_pc virtio_balloon parport sg i2c_piix4 joydev pcspkr sch_fq_codel ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_scsi virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci floppy i2c_core virtio_ring virtio dm_mirror dm_region_hash
dm_log dm_mod
CPU: 7 PID: 51 Comm: writeback Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff88327408af70 ti: ffff88391ebe4000 task.ti: ffff88391ebe4000
RIP: 0010:[<ffffffffa9b07e3e>]  [<ffffffffa9b07e3e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff88391ebe78e8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00000000000000fc RCX: ffff88391de5ccc0
RDX: 0000000000000001 RSI: 0000000000000060 RDI: 0000000000000000
RBP: ffff88391ebe7920 R08: ffff88327443d800 R09: ffffffffa9d4a3d9
R10: ffff88391dfdbb00 R11: ffffd46d52bca800 R12: 0000000000000292
R13: ffff88391ebe7898 R14: ffff88357c6d9e18 R15: ffff88357c6d9e00
FS:  0000000000000000(0000) GS:ffff88391dfc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f522a5c3b80 CR3: 000000057580e000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa9a77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa9a771b4>] flush_tlb_page+0x54/0xa0
[<ffffffffa9bd8837>] ptep_clear_flush+0x67/0xa0
[<ffffffffa9bd1ba4>] page_mkclean+0x144/0x210
[<ffffffffa9b9f7d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa9b9fa7a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc037e850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffa9d1a940>] ? submit_bio+0x70/0x150
[<ffffffffc037e65d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffffa9ba0e21>] do_writepages+0x21/0x50
[<ffffffffa9c48af0>] __writeback_single_inode+0x40/0x260
[<ffffffffa9c49584>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffa9c498ef>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa9c4abb6>] bdi_writeback_workfn+0x3d6/0x460
[<ffffffffa9ab2dff>] process_one_work+0x17f/0x440
[<ffffffffa9ab3354>] rescuer_thread+0x294/0x3c0
[<ffffffffa9ab30c0>] ? process_one_work+0x440/0x440
[<ffffffffa9abae31>] kthread+0xd1/0xe0
[<ffffffffa9abad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffaa11f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa9abad60>] ? insert_kthread_work+0x40/0x40
Code: b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 73 aa f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #553082
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8106e864
Build id: 3.10.0-693.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff811c2524
Build id: 3.10.0-693.el7.x86_64
0x54
5
page_mkclean
vmlinux 0xffffffff811bbf2e
Build id: 3.10.0-693.el7.x86_64
0x12e
6
clear_page_dirty_for_io
vmlinux 0xffffffff8118d9c0
Build id: 3.10.0-693.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffff8118dc6a
Build id: 3.10.0-693.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc027b980
Build id: 3.10.0-693.el7.x86_64
0x150
9
xfs_vm_writepages
xfs 0xffffffffc027b78d
Build id: 3.10.0-693.el7.x86_64
0xbd
10
do_writepages
vmlinux 0xffffffff8118effe
Build id: 3.10.0-693.el7.x86_64
0x1e
11
__writeback_single_inode
vmlinux 0xffffffff8122d7d0
Build id: 3.10.0-693.el7.x86_64
0x40
12
writeback_sb_inodes
vmlinux 0xffffffff8122e414
Build id: 3.10.0-693.el7.x86_64
0x1c4
13
__writeback_inodes_wb
vmlinux 0xffffffff8122e77f
Build id: 3.10.0-693.el7.x86_64
0x9f
14
wb_writeback
vmlinux 0xffffffff8122efb3
Build id: 3.10.0-693.el7.x86_64
0x263
15
? get_nr_inodes
vmlinux 0xffffffff8121bc3c
Build id: 3.10.0-693.el7.x86_64
0x4c
16
bdi_writeback_workfn
vmlinux 0xffffffff8122f45b
Build id: 3.10.0-693.el7.x86_64
0x2cb
17
process_one_work
vmlinux 0xffffffff810a881a
Build id: 3.10.0-693.el7.x86_64
0x17a
18
worker_thread
vmlinux 0xffffffff810a94e6
Build id: 3.10.0-693.el7.x86_64
0x126
19
? manage_workers.isra.24
vmlinux 0xffffffff810a93c0
Build id: 3.10.0-693.el7.x86_64
0x2a0
20
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
21
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
22
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
23
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u256:0:1651]
Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac edac_core iosf_mbi crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi snd_ac97_codec aesni_intel lrw gf128mul ac97_bus glue_helper snd_seq ablk_helper snd_seq_device cryptd ppdev snd_pcm vmw_balloon snd_timer pcspkr joydev snd soundcore sg vmw_vmci i2c_piix4 shpchp nfit
libnvdimm parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi scsi_transport_spi mptscsih crct10dif_pclmul crct10dif_common mptbase crc32c_intel e1000 serio_raw i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1651 Comm: kworker/u256:0 Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8800b44ebf40 ti: ffff8800b5c70000 task.ti: ffff8800b5c70000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8800b5c738c0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000c00008106e4a0 RCX: ffff88023561af20
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8800b5c738f8 R08: ffff88017cf3bc00 R09: ffffffff813227d9
R10: ffff880235659c80 R11: ffffea000893ac00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8800b5c73870 R15: ffff8802328bb190
FS:  0000000000000000(0000) GS:ffff880235640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f68b0abdff8 CR3: 00000000bbbcb000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
00000001b5c73918 ffff8802329fc808 ffff8802329fc4c0 00007f6964b82000
0000000000000000 ffff8802329fc808 ffff880221ff1c10 ffff8800b5c73948
ffffffff8106e668 ffff8802329fc4c0 00007f6964b82000 0000000000000000
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e864>] flush_tlb_page+0x54/0xa0
[<ffffffff811c2524>] ptep_clear_flush+0x54/0x80
[<ffffffff811bbf2e>] page_mkclean+0x12e/0x1f0
[<ffffffff8118d9c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8118dc6a>] write_cache_pages+0x22a/0x4d0
[<ffffffffc027b980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffc027b78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffff8118effe>] do_writepages+0x1e/0x40
[<ffffffff8122d7d0>] __writeback_single_inode+0x40/0x220
[<ffffffff8122e414>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff8122e77f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8122efb3>] wb_writeback+0x263/0x2f0
[<ffffffff8121bc3c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8122f45b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #534521
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8106e864
Build id: 3.10.0-693.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff811c2524
Build id: 3.10.0-693.el7.x86_64
0x54
5
page_mkclean
vmlinux 0xffffffff811bbf2e
Build id: 3.10.0-693.el7.x86_64
0x12e
6
clear_page_dirty_for_io
vmlinux 0xffffffff8118d9c0
Build id: 3.10.0-693.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffff8118dc6a
Build id: 3.10.0-693.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc021b980
Build id: 3.10.0-693.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffff812f8dc0
Build id: 3.10.0-693.el7.x86_64
0x70
10
xfs_vm_writepages
xfs 0xffffffffc021b78d
Build id: 3.10.0-693.el7.x86_64
0xbd
11
do_writepages
vmlinux 0xffffffff8118effe
Build id: 3.10.0-693.el7.x86_64
0x1e
12
__writeback_single_inode
vmlinux 0xffffffff8122d7d0
Build id: 3.10.0-693.el7.x86_64
0x40
13
writeback_sb_inodes
vmlinux 0xffffffff8122e414
Build id: 3.10.0-693.el7.x86_64
0x1c4
14
__writeback_inodes_wb
vmlinux 0xffffffff8122e77f
Build id: 3.10.0-693.el7.x86_64
0x9f
15
wb_writeback
vmlinux 0xffffffff8122efb3
Build id: 3.10.0-693.el7.x86_64
0x263
16
? get_nr_inodes
vmlinux 0xffffffff8121bc3c
Build id: 3.10.0-693.el7.x86_64
0x4c
17
bdi_writeback_workfn
vmlinux 0xffffffff8122f45b
Build id: 3.10.0-693.el7.x86_64
0x2cb
18
process_one_work
vmlinux 0xffffffff810a881a
Build id: 3.10.0-693.el7.x86_64
0x17a
19
worker_thread
vmlinux 0xffffffff810a94e6
Build id: 3.10.0-693.el7.x86_64
0x126
20
? manage_workers.isra.24
vmlinux 0xffffffff810a93c0
Build id: 3.10.0-693.el7.x86_64
0x2a0
21
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
22
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
23
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
24
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kworker/u320:2:144101]
Modules linked in: uio_ivshmem(OE) gsch(OE) redirfs(OE) binfmt_misc nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter uio ppdev parport_pc sg i2c_piix4 pcspkr parport virtio_balloon joydev nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci floppy virtio_ring i2c_core serio_raw virtio dm_mirror dm_region_hash dm_log dm_mod [last unloaded: redirfs]
CPU: 15 PID: 144101 Comm: kworker/u320:2 Tainted: G           OEL ------------   3.10.0-693.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8807f62e9fa0 ti: ffff8807e5f5c000 task.ti: ffff8807e5f5c000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8807e5f5f8c0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008106e4a0 RCX: ffff88081ce1af98
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff8807e5f5f8f8 R08: ffff880174b8f000 R09: ffffffff813227d9
R10: ffff88081d1d9c80 R11: ffffea001fb18c00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8807e5f5f870 R15: ffff8807fd386eb0
FS:  0000000000000000(0000) GS:ffff88081d1c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f7024002028 CR3: 00000000019f2000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
00000001810c4765 ffff880819321c48 ffff880819321900 00007f0e79099000
0000000000000000 ffff880819321c48 ffff8807f920f4c8 ffff8807e5f5f948
ffffffff8106e668 ffff880819321900 00007f0e79099000 0000000000000000
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e864>] flush_tlb_page+0x54/0xa0
[<ffffffff811c2524>] ptep_clear_flush+0x54/0x80
[<ffffffff811bbf2e>] page_mkclean+0x12e/0x1f0
[<ffffffff8118d9c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8118dc6a>] write_cache_pages+0x22a/0x4d0
[<ffffffffc021b980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff812f8dc0>] ? submit_bio+0x70/0x150
[<ffffffffc021b78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffff8118effe>] do_writepages+0x1e/0x40
[<ffffffff8122d7d0>] __writeback_single_inode+0x40/0x220
[<ffffffff8122e414>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff8122e77f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8122efb3>] wb_writeback+0x263/0x2f0
[<ffffffff8121bc3c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8122f45b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #409307
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb2d07e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb2c77018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffb2c771b4
Build id: 3.10.0-862.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffb2dd8837
Build id: 3.10.0-862.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffffb2dd1ba4
Build id: 3.10.0-862.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffffb2d9f7d0
Build id: 3.10.0-862.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffffb2d9fa7a
Build id: 3.10.0-862.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc0398850
Build id: 3.10.0-862.el7.x86_64
0x150
9
? update_curr
vmlinux 0xffffffffb2cd816c
Build id: 3.10.0-862.el7.x86_64
0x14c
10
? account_entity_dequeue
vmlinux 0xffffffffb2cd4ade
Build id: 3.10.0-862.el7.x86_64
0xae
11
? dequeue_entity
vmlinux 0xffffffffb2cd865c
Build id: 3.10.0-862.el7.x86_64
0x11c
12
xfs_vm_writepages
xfs 0xffffffffc039865d
Build id: 3.10.0-862.el7.x86_64
0xbd
13
do_writepages
vmlinux 0xffffffffb2da0e21
Build id: 3.10.0-862.el7.x86_64
0x21
14
__writeback_single_inode
vmlinux 0xffffffffb2e48af0
Build id: 3.10.0-862.el7.x86_64
0x40
15
? __schedule
vmlinux 0xffffffffb331291c
Build id: 3.10.0-862.el7.x86_64
0x41c
16
writeback_sb_inodes
vmlinux 0xffffffffb2e49584
Build id: 3.10.0-862.el7.x86_64
0x1c4
17
__writeback_inodes_wb
vmlinux 0xffffffffb2e498ef
Build id: 3.10.0-862.el7.x86_64
0x9f
18
wb_writeback
vmlinux 0xffffffffb2e4a123
Build id: 3.10.0-862.el7.x86_64
0x263
19
? get_nr_inodes
vmlinux 0xffffffffb2e3665c
Build id: 3.10.0-862.el7.x86_64
0x4c
20
bdi_writeback_workfn
vmlinux 0xffffffffb2e4aaab
Build id: 3.10.0-862.el7.x86_64
0x2cb
21
process_one_work
vmlinux 0xffffffffb2cb2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
22
worker_thread
vmlinux 0xffffffffb2cb3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
23
? manage_workers.isra.24
vmlinux 0xffffffffb2cb39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
24
kthread
vmlinux 0xffffffffb2cbae31
Build id: 3.10.0-862.el7.x86_64
0xd1
25
? insert_kthread_work
vmlinux 0xffffffffb2cbad60
Build id: 3.10.0-862.el7.x86_64
0x40
26
ret_from_fork_nospec_begin
vmlinux 0xffffffffb331f637
Build id: 3.10.0-862.el7.x86_64
0x21
27
? insert_kthread_work
vmlinux 0xffffffffb2cbad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [kworker/u510:0:9772]
ata2: lost interrupt (Status 0x50)
ata2.01: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
ata2.01: cmd a0/00:00:00:08:00/00:00:00:00:00/b0 tag 0 pio 16392 in#012         Get event status notification 4a 01 00 00 10 00 00 00 08 00res 58/00:02:00:08:00/00:00:00:00:00/b0 Emask 0x6 (timeout)
ata2.01: status: { DRDY DRQ }
ata2: soft resetting link
Modules linked in: xt_iprange ip6table_filter ip6_tables iptable_filter binfmt_misc joydev bochs_drm ttm sb_edac drm_kms_helper iosf_mbi ppdev virtio_rng sg syscopyarea kvm_intel kvm sysfillrect sysimgblt irqbypass fb_sys_fops crc32_pclmul ghash_clmulni_intel i2c_piix4 drm aesni_intel lrw parport_pc i2c_core gf128mul parport glue_helper ablk_helper cryptd pcspkr ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_net virtio_console virtio_scsi crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libata serio_raw virtio_pci virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 9772 Comm: kworker/u510:0 Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff997334bedee0 ti: ffff99724aa80000 task.ti: ffff99724aa80000
RIP: 0010:[<ffffffffb2d07e3a>]  [<ffffffffb2d07e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff99724aa838b8  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 0000000000019680 RCX: ffff99737f69cfa0
RDX: 0000000000000002 RSI: 00000000000000ff RDI: 0000000000000000
RBP: ffff99724aa838f0 R08: ffff9965ea6a3800 R09: ffffffffb2f4a3d9
R10: ffff99737f7dbb00 R11: fffffc19bbf6b200 R12: ffff99724aa83868
R13: ffff9965ffc07500 R14: ffff99724aa838a0 R15: ffffffffb2df7ffa
FS:  0000000000000000(0000) GS:ffff99737f7c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffffffffff600400 CR3: 0000000dcb216000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb2c77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb2c771b4>] flush_tlb_page+0x54/0xa0
[<ffffffffb2dd8837>] ptep_clear_flush+0x67/0xa0
[<ffffffffb2dd1ba4>] page_mkclean+0x144/0x210
[<ffffffffb2d9f7d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb2d9fa7a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc0398850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffb2cd816c>] ? update_curr+0x14c/0x1e0
[<ffffffffb2cd4ade>] ? account_entity_dequeue+0xae/0xd0
[<ffffffffb2cd865c>] ? dequeue_entity+0x11c/0x5e0
[<ffffffffc039865d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffffb2da0e21>] do_writepages+0x21/0x50
[<ffffffffb2e48af0>] __writeback_single_inode+0x40/0x260
[<ffffffffb331291c>] ? __schedule+0x41c/0xa20
[<ffffffffb2e49584>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffb2e498ef>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb2e4a123>] wb_writeback+0x263/0x2f0
[<ffffffffb2e3665c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffb2e4aaab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb2cb2dff>] process_one_work+0x17f/0x440
[<ffffffffb2cb3ac6>] worker_thread+0x126/0x3c0
[<ffffffffb2cb39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffb2cbae31>] kthread+0xd1/0xe0
[<ffffffffb2cbad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb331f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb2cbad60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 93 b3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #279046
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa3b07e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa3a77018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffa3a771b4
Build id: 3.10.0-862.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffa3bd8837
Build id: 3.10.0-862.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffffa3bd1ba4
Build id: 3.10.0-862.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffffa3b9f7d0
Build id: 3.10.0-862.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffffa3b9fa7a
Build id: 3.10.0-862.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc02e6850
Build id: 3.10.0-862.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffffa3d1a940
Build id: 3.10.0-862.el7.x86_64
0x70
10
? __switch_to
vmlinux 0xffffffffa3a2959e
Build id: 3.10.0-862.el7.x86_64
0xce
11
? cpumask_next_and
vmlinux 0xffffffffa3d4a275
Build id: 3.10.0-862.el7.x86_64
0x35
12
xfs_vm_writepages
xfs 0xffffffffc02e665d
Build id: 3.10.0-862.el7.x86_64
0xbd
13
do_writepages
vmlinux 0xffffffffa3ba0e21
Build id: 3.10.0-862.el7.x86_64
0x21
14
__writeback_single_inode
vmlinux 0xffffffffa3c48af0
Build id: 3.10.0-862.el7.x86_64
0x40
15
writeback_sb_inodes
vmlinux 0xffffffffa3c49584
Build id: 3.10.0-862.el7.x86_64
0x1c4
16
__writeback_inodes_wb
vmlinux 0xffffffffa3c498ef
Build id: 3.10.0-862.el7.x86_64
0x9f
17
wb_writeback
vmlinux 0xffffffffa3c4a123
Build id: 3.10.0-862.el7.x86_64
0x263
18
? get_nr_inodes
vmlinux 0xffffffffa3c3665c
Build id: 3.10.0-862.el7.x86_64
0x4c
19
bdi_writeback_workfn
vmlinux 0xffffffffa3c4aaab
Build id: 3.10.0-862.el7.x86_64
0x2cb
20
process_one_work
vmlinux 0xffffffffa3ab2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
21
worker_thread
vmlinux 0xffffffffa3ab3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
22
? manage_workers.isra.24
vmlinux 0xffffffffa3ab39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
23
kthread
vmlinux 0xffffffffa3abae31
Build id: 3.10.0-862.el7.x86_64
0xd1
24
? insert_kthread_work
vmlinux 0xffffffffa3abad60
Build id: 3.10.0-862.el7.x86_64
0x40
25
ret_from_fork_nospec_begin
vmlinux 0xffffffffa411f637
Build id: 3.10.0-862.el7.x86_64
0x21
26
? insert_kthread_work
vmlinux 0xffffffffa3abad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#20 stuck for 67s! [kworker/u320:1:107665]
Modules linked in: udp_diag tcp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink sunrpc ppdev parport_pc joydev pcspkr virtio_balloon sg parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom virtio_console virtio_scsi virtio_blk virtio_net ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci serio_raw virtio_ring i2c_core floppy virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 20 PID: 107665 Comm: kworker/u320:1 Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff90999a07eeb0 ti: ffff908c81a78000 task.ti: ffff908c81a78000
RIP: 0010:[<ffffffffa3b07e3a>]  [<ffffffffa3b07e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff908c81a7b8b8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff90906ce1cfe0
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff908c81a7b8f0 R08: ffff908db9c27c00 R09: ffffffffa3d4a3d9
R10: ffff90906d31bb00 R11: ffffd0f2853a6400 R12: 0000000000000296
R13: ffff908c81a7b868 R14: 0000000000000046 R15: ffff90903a930c00
FS:  0000000000000000(0000) GS:ffff90906d300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000026714014 CR3: 000000008620e000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa3a77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa3a771b4>] flush_tlb_page+0x54/0xa0
[<ffffffffa3bd8837>] ptep_clear_flush+0x67/0xa0
[<ffffffffa3bd1ba4>] page_mkclean+0x144/0x210
[<ffffffffa3b9f7d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa3b9fa7a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc02e6850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffa3d1a940>] ? submit_bio+0x70/0x150
[<ffffffffa3a2959e>] ? __switch_to+0xce/0x580
[<ffffffffa3d4a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffc02e665d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffffa3ba0e21>] do_writepages+0x21/0x50
[<ffffffffa3c48af0>] __writeback_single_inode+0x40/0x260
[<ffffffffa3c49584>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffa3c498ef>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa3c4a123>] wb_writeback+0x263/0x2f0
[<ffffffffa3c3665c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffa3c4aaab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa3ab2dff>] process_one_work+0x17f/0x440
[<ffffffffa3ab3ac6>] worker_thread+0x126/0x3c0
[<ffffffffa3ab39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffa3abae31>] kthread+0xd1/0xe0
[<ffffffffa3abad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa411f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa3abad60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 73 a4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #370742
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb790b0ca
Build id: 3.10.0-862.14.4.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb7879e58
Build id: 3.10.0-862.14.4.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffb7879ff4
Build id: 3.10.0-862.14.4.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffb79dc0f7
Build id: 3.10.0-862.14.4.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffffb79d53c4
Build id: 3.10.0-862.14.4.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffffb79a2530
Build id: 3.10.0-862.14.4.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffffb79a27da
Build id: 3.10.0-862.14.4.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc05a4850
Build id: 3.10.0-862.14.4.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffffb7b1f450
Build id: 3.10.0-862.14.4.el7.x86_64
0x70
10
xfs_vm_writepages
xfs 0xffffffffc05a465d
Build id: 3.10.0-862.14.4.el7.x86_64
0xbd
11
do_writepages
vmlinux 0xffffffffb79a3b81
Build id: 3.10.0-862.14.4.el7.x86_64
0x21
12
__writeback_single_inode
vmlinux 0xffffffffb7a4cfc0
Build id: 3.10.0-862.14.4.el7.x86_64
0x40
13
writeback_sb_inodes
vmlinux 0xffffffffb7a4da54
Build id: 3.10.0-862.14.4.el7.x86_64
0x1c4
14
__writeback_inodes_wb
vmlinux 0xffffffffb7a4ddbf
Build id: 3.10.0-862.14.4.el7.x86_64
0x9f
15
wb_writeback
vmlinux 0xffffffffb7a4e5f3
Build id: 3.10.0-862.14.4.el7.x86_64
0x263
16
bdi_writeback_workfn
vmlinux 0xffffffffb7a4ef7b
Build id: 3.10.0-862.14.4.el7.x86_64
0x2cb
17
process_one_work
vmlinux 0xffffffffb78b613f
Build id: 3.10.0-862.14.4.el7.x86_64
0x17f
18
worker_thread
vmlinux 0xffffffffb78b71d6
Build id: 3.10.0-862.14.4.el7.x86_64
0x126
19
? manage_workers.isra.24
vmlinux 0xffffffffb78b70b0
Build id: 3.10.0-862.14.4.el7.x86_64
0x2a0
20
kthread
vmlinux 0xffffffffb78bdf21
Build id: 3.10.0-862.14.4.el7.x86_64
0xd1
21
? insert_kthread_work
vmlinux 0xffffffffb78bde50
Build id: 3.10.0-862.14.4.el7.x86_64
0x40
22
ret_from_fork_nospec_begin
vmlinux 0xffffffffb7f255dd
Build id: 3.10.0-862.14.4.el7.x86_64
0x7
23
? insert_kthread_work
vmlinux 0xffffffffb78bde50
Build id: 3.10.0-862.14.4.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#20 stuck for 23s! [kworker/u256:3:24558]
Modules linked in: arc4 md4 nls_utf8 cifs ccm dns_resolver ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 vfat fat ppdev iosf_mbi vmw_balloon joydev pcspkr sg nfit vmw_vmci shpchp i2c_piix4 libnvdimm tpm_crb parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx crct10dif_pclmul crct10dif_common
crc32_pclmul crc32c_intel drm_kms_helper ghash_clmulni_intel syscopyarea aesni_intel sysfillrect sysimgblt fb_sys_fops lrw ttm gf128mul glue_helper ahci ablk_helper 8021q cryptd garp drm mrp ata_piix libahci stp llc libata vmxnet3 serio_raw vmw_pvscsi i2c_core scsi_transport_iscsi sunrpc dm_mirror dm_region_hash dm_log dm_mod
CPU: 20 PID: 24558 Comm: kworker/u256:3 Kdump: loaded Tainted: G             L ------------   3.10.0-862.14.4.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.16707776.B64.2008070230 08/07/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9f4a33a1eeb0 ti: ffff9f42c856c000 task.ti: ffff9f42c856c000
RIP: 0010:[<ffffffffb790b0ca>]  [<ffffffffb790b0ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9f42c856f8b8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00000000000000fc RCX: ffff9f733965d1e0
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9f42c856f8f0 R08: ffff9f43f54c0400 R09: ffffffffb7b4ef19
R10: ffff9f7339b1bb40 R11: fffffc08c55f8e00 R12: 0000000000000296
R13: ffff9f42c856f868 R14: 00000000a7762337 R15: ffff9f732ed81d00
FS:  0000000000000000(0000) GS:ffff9f7339b00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000081dd7000 CR3: 0000000190e0e000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffb7879e58>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb7879ff4>] flush_tlb_page+0x54/0xa0
[<ffffffffb79dc0f7>] ptep_clear_flush+0x67/0xa0
[<ffffffffb79d53c4>] page_mkclean+0x144/0x210
[<ffffffffb79a2530>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb79a27da>] write_cache_pages+0x22a/0x4e0
[<ffffffffc05a4850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffb7b1f450>] ? submit_bio+0x70/0x150
[<ffffffffc05a465d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffffb79a3b81>] do_writepages+0x21/0x50
[<ffffffffb7a4cfc0>] __writeback_single_inode+0x40/0x260
[<ffffffffb7a4da54>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffb7a4ddbf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb7a4e5f3>] wb_writeback+0x263/0x2f0
[<ffffffffb7a4ef7b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb78b613f>] process_one_work+0x17f/0x440
[<ffffffffb78b71d6>] worker_thread+0x126/0x3c0
[<ffffffffb78b70b0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffb78bdf21>] kthread+0xd1/0xe0
[<ffffffffb78bde50>] ? insert_kthread_work+0x40/0x40
[<ffffffffb7f255dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb78bde50>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 fe 84 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 54 53 b8 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 84 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #331432
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa0907e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa0877018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffa08771b4
Build id: 3.10.0-862.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffa09d8837
Build id: 3.10.0-862.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffffa09d1ba4
Build id: 3.10.0-862.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffffa099f7d0
Build id: 3.10.0-862.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffffa099fa7a
Build id: 3.10.0-862.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc0471850
Build id: 3.10.0-862.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffffa0b1a940
Build id: 3.10.0-862.el7.x86_64
0x70
10
? cpumask_next_and
vmlinux 0xffffffffa0b4a275
Build id: 3.10.0-862.el7.x86_64
0x35
11
xfs_vm_writepages
xfs 0xffffffffc047165d
Build id: 3.10.0-862.el7.x86_64
0xbd
12
do_writepages
vmlinux 0xffffffffa09a0e21
Build id: 3.10.0-862.el7.x86_64
0x21
13
__writeback_single_inode
vmlinux 0xffffffffa0a48af0
Build id: 3.10.0-862.el7.x86_64
0x40
14
writeback_sb_inodes
vmlinux 0xffffffffa0a49584
Build id: 3.10.0-862.el7.x86_64
0x1c4
15
__writeback_inodes_wb
vmlinux 0xffffffffa0a498ef
Build id: 3.10.0-862.el7.x86_64
0x9f
16
wb_writeback
vmlinux 0xffffffffa0a4a123
Build id: 3.10.0-862.el7.x86_64
0x263
17
? get_nr_inodes
vmlinux 0xffffffffa0a3665c
Build id: 3.10.0-862.el7.x86_64
0x4c
18
bdi_writeback_workfn
vmlinux 0xffffffffa0a4aaab
Build id: 3.10.0-862.el7.x86_64
0x2cb
19
process_one_work
vmlinux 0xffffffffa08b2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
20
worker_thread
vmlinux 0xffffffffa08b3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
21
? manage_workers.isra.24
vmlinux 0xffffffffa08b39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
22
kthread
vmlinux 0xffffffffa08bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
23
? insert_kthread_work
vmlinux 0xffffffffa08bad60
Build id: 3.10.0-862.el7.x86_64
0x40
24
ret_from_fork_nospec_begin
vmlinux 0xffffffffa0f1f637
Build id: 3.10.0-862.el7.x86_64
0x21
25
? insert_kthread_work
vmlinux 0xffffffffa08bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u80:1:18366]
Modules linked in: iptable_filter gsch(OE) redirfs(OE) dsa_filter(POE) binfmt_misc sunrpc ext4 mbcache jbd2 ppdev joydev sg i2c_piix4 pcspkr parport_pc parport virtio_balloon ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_scsi virtio_console virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw virtio_pci i2c_core virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 18366 Comm: kworker/u80:1 Kdump: loaded Tainted: P           OEL ------------   3.10.0-862.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff998b82b80000 ti: ffff998c76a0c000 task.ti: ffff998c76a0c000
RIP: 0010:[<ffffffffa0907e3a>]  [<ffffffffa0907e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff998c76a0f8b8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff998eaec1cc20
RDX: 0000000000000000 RSI: 0000000000000028 RDI: 0000000000000000
RBP: ffff998c76a0f8f0 R08: ffff998bfa347c00 R09: ffffffffa0b4a3d9
R10: ffff998eaecdbb00 R11: ffffef598e3ab000 R12: 0000000000000296
R13: ffff998c76a0f868 R14: 0000000000000046 R15: ffff998e956eb400
FS:  0000000000000000(0000) GS:ffff998eaecc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f8ebdb24000 CR3: 000000014360e000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa0877018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa08771b4>] flush_tlb_page+0x54/0xa0
[<ffffffffa09d8837>] ptep_clear_flush+0x67/0xa0
[<ffffffffa09d1ba4>] page_mkclean+0x144/0x210
[<ffffffffa099f7d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa099fa7a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc0471850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffa0b1a940>] ? submit_bio+0x70/0x150
[<ffffffffa0b4a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffc047165d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffffa09a0e21>] do_writepages+0x21/0x50
[<ffffffffa0a48af0>] __writeback_single_inode+0x40/0x260
[<ffffffffa0a49584>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffa0a498ef>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa0a4a123>] wb_writeback+0x263/0x2f0
[<ffffffffa0a3665c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffa0a4aaab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa08b2dff>] process_one_work+0x17f/0x440
[<ffffffffa08b3ac6>] worker_thread+0x126/0x3c0
[<ffffffffa08b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffa08bae31>] kthread+0xd1/0xe0
[<ffffffffa08bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa0f1f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa08bad60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 53 a1 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #190124
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb9707e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb9677018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffb96771b4
Build id: 3.10.0-862.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffb97d8837
Build id: 3.10.0-862.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffffb97d1ba4
Build id: 3.10.0-862.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffffb979f7d0
Build id: 3.10.0-862.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffffb979fa7a
Build id: 3.10.0-862.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc06b4850
Build id: 3.10.0-862.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffffb991a940
Build id: 3.10.0-862.el7.x86_64
0x70
10
? __switch_to
vmlinux 0xffffffffb962959e
Build id: 3.10.0-862.el7.x86_64
0xce
11
xfs_vm_writepages
xfs 0xffffffffc06b465d
Build id: 3.10.0-862.el7.x86_64
0xbd
12
do_writepages
vmlinux 0xffffffffb97a0e21
Build id: 3.10.0-862.el7.x86_64
0x21
13
__writeback_single_inode
vmlinux 0xffffffffb9848af0
Build id: 3.10.0-862.el7.x86_64
0x40
14
writeback_sb_inodes
vmlinux 0xffffffffb9849584
Build id: 3.10.0-862.el7.x86_64
0x1c4
15
__writeback_inodes_wb
vmlinux 0xffffffffb98498ef
Build id: 3.10.0-862.el7.x86_64
0x9f
16
wb_writeback
vmlinux 0xffffffffb984a123
Build id: 3.10.0-862.el7.x86_64
0x263
17
bdi_writeback_workfn
vmlinux 0xffffffffb984aaab
Build id: 3.10.0-862.el7.x86_64
0x2cb
18
process_one_work
vmlinux 0xffffffffb96b2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
19
worker_thread
vmlinux 0xffffffffb96b3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
20
? manage_workers.isra.24
vmlinux 0xffffffffb96b39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
21
kthread
vmlinux 0xffffffffb96bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
22
? insert_kthread_work
vmlinux 0xffffffffb96bad60
Build id: 3.10.0-862.el7.x86_64
0x40
23
ret_from_fork_nospec_begin
vmlinux 0xffffffffb9d1f637
Build id: 3.10.0-862.el7.x86_64
0x21
24
? insert_kthread_work
vmlinux 0xffffffffb96bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [kworker/u256:0:2734]
Modules linked in: tcp_lp fuse tcp_diag inet_diag xt_set xt_multiport iptable_raw ip_set_hash_ip ip_set_hash_net ip_set veth nf_conntrack_netlink nfnetlink xt_nat xt_statistic ip_vs_sh ip_vs_wrr ip_vs_rr cfg80211 rfkill ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ipt_REJECT nf_reject_ipv4 ip_vs nf_conntrack tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel snd_ens1371 ppdev snd_rawmidi snd_ac97_codec ac97_bus
snd_seq vmw_balloon aesni_intel snd_seq_device lrw gf128mul glue_helper ablk_helper cryptd snd_pcm snd_timer pcspkr joydev snd nfit vmw_vmci i2c_piix4 sg soundcore libnvdimm shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common ata_piix crc32c_intel mptspi libata scsi_transport_spi e1000 serio_raw mptscsih mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 2734 Comm: kworker/u256:0 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9c9588b10fd0 ti: ffff9c95ca224000 task.ti: ffff9c95ca224000
RIP: 0010:[<ffffffffb9707e3a>]  [<ffffffffb9707e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9c95ca2278b8  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff9c992569cd90
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9c95ca2278f0 R08: ffff9c9476543c00 R09: ffffffffb994a3d9
R10: ffff9c992565bb00 R11: ffffe2d04aeff800 R12: 0000000000000296
R13: ffff9c95ca227868 R14: ffff9c9921b9d0d8 R15: ffff9c96a93e1180
FS:  0000000000000000(0000) GS:ffff9c9925640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000519000 CR3: 00000005d0326000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb9677018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb96771b4>] flush_tlb_page+0x54/0xa0
[<ffffffffb97d8837>] ptep_clear_flush+0x67/0xa0
[<ffffffffb97d1ba4>] page_mkclean+0x144/0x210
[<ffffffffb979f7d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb979fa7a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc06b4850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffb991a940>] ? submit_bio+0x70/0x150
[<ffffffffb962959e>] ? __switch_to+0xce/0x580
[<ffffffffc06b465d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffffb97a0e21>] do_writepages+0x21/0x50
[<ffffffffb9848af0>] __writeback_single_inode+0x40/0x260
[<ffffffffb9849584>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffb98498ef>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb984a123>] wb_writeback+0x263/0x2f0
[<ffffffffb984aaab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb96b2dff>] process_one_work+0x17f/0x440
[<ffffffffb96b3ac6>] worker_thread+0x126/0x3c0
[<ffffffffb96b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffb96bae31>] kthread+0xd1/0xe0
[<ffffffffb96bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb9d1f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb96bad60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 33 ba f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #96068
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb0d07e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb0c77018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffb0c771b4
Build id: 3.10.0-862.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffb0dd8837
Build id: 3.10.0-862.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffffb0dd1ba4
Build id: 3.10.0-862.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffffb0d9f7d0
Build id: 3.10.0-862.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffffb0d9fa7a
Build id: 3.10.0-862.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc0235850
Build id: 3.10.0-862.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffffb0f1a940
Build id: 3.10.0-862.el7.x86_64
0x70
10
? __switch_to
vmlinux 0xffffffffb0c2959e
Build id: 3.10.0-862.el7.x86_64
0xce
11
xfs_vm_writepages
xfs 0xffffffffc023565d
Build id: 3.10.0-862.el7.x86_64
0xbd
12
do_writepages
vmlinux 0xffffffffb0da0e21
Build id: 3.10.0-862.el7.x86_64
0x21
13
__writeback_single_inode
vmlinux 0xffffffffb0e48af0
Build id: 3.10.0-862.el7.x86_64
0x40
14
writeback_sb_inodes
vmlinux 0xffffffffb0e49584
Build id: 3.10.0-862.el7.x86_64
0x1c4
15
__writeback_inodes_wb
vmlinux 0xffffffffb0e498ef
Build id: 3.10.0-862.el7.x86_64
0x9f
16
wb_writeback
vmlinux 0xffffffffb0e4a123
Build id: 3.10.0-862.el7.x86_64
0x263
17
? get_nr_inodes
vmlinux 0xffffffffb0e3665c
Build id: 3.10.0-862.el7.x86_64
0x4c
18
bdi_writeback_workfn
vmlinux 0xffffffffb0e4aaab
Build id: 3.10.0-862.el7.x86_64
0x2cb
19
process_one_work
vmlinux 0xffffffffb0cb2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
20
worker_thread
vmlinux 0xffffffffb0cb3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
21
? manage_workers.isra.24
vmlinux 0xffffffffb0cb39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
22
kthread
vmlinux 0xffffffffb0cbae31
Build id: 3.10.0-862.el7.x86_64
0xd1
23
? insert_kthread_work
vmlinux 0xffffffffb0cbad60
Build id: 3.10.0-862.el7.x86_64
0x40
24
ret_from_fork_nospec_begin
vmlinux 0xffffffffb131f637
Build id: 3.10.0-862.el7.x86_64
0x21
25
? insert_kthread_work
vmlinux 0xffffffffb0cbad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [kworker/u320:3:84258]
Modules linked in: tcp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink binfmt_misc sunrpc ppdev parport_pc sg virtio_balloon parport joydev i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod cdrom virtio_blk virtio_console virtio_net virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_generic pata_acpi ata_piix libata virtio_pci virtio_ring i2c_core serio_raw virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 22 PID: 84258 Comm: kworker/u320:3 Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff977b40eb3f40 ti: ffff977b88738000 task.ti: ffff977b88738000
RIP: 0010:[<ffffffffb0d07e3a>]  [<ffffffffb0d07e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff977b8873b8b8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff97831ce1d030
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff977b8873b8f0 R08: ffff977c73c40c00 R09: ffffffffb0f4a3d9
R10: ffff97831d39bb00 R11: ffffda9c00a00c00 R12: 0000000000000296
R13: ffff977b8873b868 R14: 0000000000000046 R15: ffffffffb0df7ffa
FS:  0000000000000000(0000) GS:ffff97831d380000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000006a413d2b4 CR3: 00000002b93b2000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb0c77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb0c771b4>] flush_tlb_page+0x54/0xa0
[<ffffffffb0dd8837>] ptep_clear_flush+0x67/0xa0
[<ffffffffb0dd1ba4>] page_mkclean+0x144/0x210
[<ffffffffb0d9f7d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb0d9fa7a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc0235850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffb0f1a940>] ? submit_bio+0x70/0x150
[<ffffffffb0c2959e>] ? __switch_to+0xce/0x580
[<ffffffffc023565d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffffb0da0e21>] do_writepages+0x21/0x50
[<ffffffffb0e48af0>] __writeback_single_inode+0x40/0x260
[<ffffffffb0e49584>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffb0e498ef>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb0e4a123>] wb_writeback+0x263/0x2f0
[<ffffffffb0e3665c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffb0e4aaab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb0cb2dff>] process_one_work+0x17f/0x440
[<ffffffffb0cb3ac6>] worker_thread+0x126/0x3c0
[<ffffffffb0cb39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffb0cbae31>] kthread+0xd1/0xe0
[<ffffffffb0cbad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb131f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb0cbad60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 93 b1 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #3134
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8106e864
Build id: 3.10.0-693.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff811c2524
Build id: 3.10.0-693.el7.x86_64
0x54
5
page_mkclean
vmlinux 0xffffffff811bbf2e
Build id: 3.10.0-693.el7.x86_64
0x12e
6
clear_page_dirty_for_io
vmlinux 0xffffffff8118d9c0
Build id: 3.10.0-693.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffff8118dc6a
Build id: 3.10.0-693.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc029f980
Build id: 3.10.0-693.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffff812f8dc0
Build id: 3.10.0-693.el7.x86_64
0x70
10
xfs_vm_writepages
xfs 0xffffffffc029f78d
Build id: 3.10.0-693.el7.x86_64
0xbd
11
do_writepages
vmlinux 0xffffffff8118effe
Build id: 3.10.0-693.el7.x86_64
0x1e
12
__writeback_single_inode
vmlinux 0xffffffff8122d7d0
Build id: 3.10.0-693.el7.x86_64
0x40
13
writeback_sb_inodes
vmlinux 0xffffffff8122e414
Build id: 3.10.0-693.el7.x86_64
0x1c4
14
__writeback_inodes_wb
vmlinux 0xffffffff8122e77f
Build id: 3.10.0-693.el7.x86_64
0x9f
15
wb_writeback
vmlinux 0xffffffff8122efb3
Build id: 3.10.0-693.el7.x86_64
0x263
16
? get_nr_inodes
vmlinux 0xffffffff8121bc3c
Build id: 3.10.0-693.el7.x86_64
0x4c
17
bdi_writeback_workfn
vmlinux 0xffffffff8122f45b
Build id: 3.10.0-693.el7.x86_64
0x2cb
18
process_one_work
vmlinux 0xffffffff810a881a
Build id: 3.10.0-693.el7.x86_64
0x17a
19
worker_thread
vmlinux 0xffffffff810a94e6
Build id: 3.10.0-693.el7.x86_64
0x126
20
? manage_workers.isra.24
vmlinux 0xffffffff810a93c0
Build id: 3.10.0-693.el7.x86_64
0x2a0
21
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64
0xcf
22
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
23
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64
0x58
24
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kworker/u256:2:88347]
Modules linked in: tcp_lp fuse tun ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac edac_core snd_ens1371 iosf_mbi crc32_pclmul snd_rawmidi snd_ac97_codec ghash_clmulni_intel ac97_bus snd_seq aesni_intel lrw snd_seq_device gf128mul glue_helper snd_pcm ablk_helper vmw_balloon joydev pcspkr ppdev cryptd snd_timer i2c_piix4 vmw_vmci sg snd soundcore shpchp nfit libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi drm ata_piix crct10dif_pclmul e1000 crct10dif_common mptscsih libata i2c_core serio_raw crc32c_intel mptbase dm_mirror dm_region_hash
dm_log dm_mod [last unloaded: ip_tables]
CPU: 7 PID: 88347 Comm: kworker/u256:2 Not tainted 3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8803b6f3cf10 ti: ffff8803adf5c000 task.ti: ffff8803adf5c000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff8803adf5f8c0  EFLAGS: 00000202
RAX: 0000000000000009 RBX: 000800008106e4a0 RCX: ffff88042d85b010
RDX: 0000000000000009 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8803adf5f8f8 R08: ffff88017a394c00 R09: ffffffff813227d9
R10: ffff88042d7d9c80 R11: ffffea0002799000 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8803adf5f870 R15: ffff8803adf5f860
FS:  0000000000000000(0000) GS:ffff88042d7c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd6ebe7c800 CR3: 00000000019f2000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000100000018 ffff880428c70fc8 ffff880428c70c80 00007fd6ebe79000
0000000000000000 ffff880428c70fc8 ffff880426f1a3c8 ffff8803adf5f948
ffffffff8106e668 ffff880428c70c80 00007fd6ebe79000 0000000000000000
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e864>] flush_tlb_page+0x54/0xa0
[<ffffffff811c2524>] ptep_clear_flush+0x54/0x80
[<ffffffff811bbf2e>] page_mkclean+0x12e/0x1f0
[<ffffffff8118d9c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8118dc6a>] write_cache_pages+0x22a/0x4d0
[<ffffffffc029f980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff812f8dc0>] ? submit_bio+0x70/0x150
[<ffffffffc029f78d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffff8118effe>] do_writepages+0x1e/0x40
[<ffffffff8122d7d0>] __writeback_single_inode+0x40/0x220
[<ffffffff8122e414>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff8122e77f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8122efb3>] wb_writeback+0x263/0x2f0
[<ffffffff8121bc3c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8122f45b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89