Info

Permalink
Function
smp_call_function_many
First occurrence
2020-09-24
Last occurrence
2021-05-10
State
NEW
Components
kernel
Quality
-57

Statistics

Operating system Count
CentOS 7 457
Architecture Count
x86_64 457
Related packages Count
kernel 457
 0:3.10.0-1062.9.1.el7 255
 0:3.10.0-957.el7 99
 0:3.10.0-1160.76.1.el7 28
 0:3.10.0-862.el7 15
 0:3.10.0-1062.4.3.el7 14
 0:3.10.0-1160.102.1.el7 13
 0:3.10.0-1160.105.1.el7 8
 0:3.10.0-1160.45.1.el7 7
 0:3.10.0-1160.59.1.el7 3
 0:3.10.0-1062.el7 3
 0:3.10.0-957.21.3.el7 2
 0:3.10.0-1160.83.1.el7 2
 0:3.10.0-1160.108.1.el7 2
 0:3.10.0-1062.18.1.el7 2
 0:3.10.0-1160.53.1.el7 1
 0:3.10.0-1160.92.1.el7 1
 0:3.10.0-1160.95.1.el7 1
 0:3.10.0-1160.119.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1418287
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff995150ca
Build id: 3.10.0-1062.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9947d398
Build id: 3.10.0-1062.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff9947d534
Build id: 3.10.0-1062.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff99603b18
Build id: 3.10.0-1062.el7.x86_64
0x68
5
page_mkclean_one
vmlinux 0xffffffff995fd0a6
Build id: 3.10.0-1062.el7.x86_64
0xa6
6
rmap_walk
vmlinux 0xffffffff995feed3
Build id: 3.10.0-1062.el7.x86_64
0xf3
7
page_mkclean
vmlinux 0xffffffff995ff2e6
Build id: 3.10.0-1062.el7.x86_64
0x76
8
? page_referenced_one
vmlinux 0xffffffff995fd000
Build id: 3.10.0-1062.el7.x86_64
0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffff995fc6e0
Build id: 3.10.0-1062.el7.x86_64
0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffff995c7a50
Build id: 3.10.0-1062.el7.x86_64
0x50
11
write_cache_pages
vmlinux 0xffffffff995c7cc2
Build id: 3.10.0-1062.el7.x86_64
0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc0601030
Build id: 3.10.0-1062.el7.x86_64
0xa0
13
? update_curr
vmlinux 0xffffffff994e375c
Build id: 3.10.0-1062.el7.x86_64
0x14c
14
? update_group_power
vmlinux 0xffffffff994e71f4
Build id: 3.10.0-1062.el7.x86_64
0x174
15
? cpumask_next_and
vmlinux 0xffffffff9977f475
Build id: 3.10.0-1062.el7.x86_64
0x35
16
xfs_vm_writepages
xfs 0xffffffffc0600ffb
Build id: 3.10.0-1062.el7.x86_64
0x6b
17
do_writepages
vmlinux 0xffffffff995c8d31
Build id: 3.10.0-1062.el7.x86_64
0x21
18
__writeback_single_inode
vmlinux 0xffffffff996776c0
Build id: 3.10.0-1062.el7.x86_64
0x40
19
writeback_sb_inodes
vmlinux 0xffffffff99678254
Build id: 3.10.0-1062.el7.x86_64
0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffff9967855f
Build id: 3.10.0-1062.el7.x86_64
0x9f
21
wb_writeback
vmlinux 0xffffffff99678a43
Build id: 3.10.0-1062.el7.x86_64
0x263
22
? get_nr_inodes
vmlinux 0xffffffff9966490c
Build id: 3.10.0-1062.el7.x86_64
0x4c
23
bdi_writeback_workfn
vmlinux 0xffffffff9967963b
Build id: 3.10.0-1062.el7.x86_64
0x2cb
24
process_one_work
vmlinux 0xffffffff994bd0ff
Build id: 3.10.0-1062.el7.x86_64
0x17f
25
worker_thread
vmlinux 0xffffffff994be216
Build id: 3.10.0-1062.el7.x86_64
0x126
26
? manage_workers.isra.26
vmlinux 0xffffffff994be0f0
Build id: 3.10.0-1062.el7.x86_64
0x2a0
27
kthread
vmlinux 0xffffffff994c50d1
Build id: 3.10.0-1062.el7.x86_64
0xd1
28
? insert_kthread_work
vmlinux 0xffffffff994c5000
Build id: 3.10.0-1062.el7.x86_64
0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffff99b8bd37
Build id: 3.10.0-1062.el7.x86_64
0x21
30
? insert_kthread_work
vmlinux 0xffffffff994c5000
Build id: 3.10.0-1062.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u256:1:120459]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi ppdev sb_edac snd_seq_midi_event vmw_balloon iosf_mbi kvm_intel kvm irqbypass snd_ens1371 crc32_pclmul snd_rawmidi ghash_clmulni_intel snd_ac97_codec aesni_intel ac97_bus snd_seq lrw gf128mul snd_seq_device glue_helper snd_pcm ablk_helper cryptd snd_timer snd vmw_vmci joydev pcspkr sg i2c_piix4 soundcore parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops mptspi ttm scsi_transport_spi ata_piix nfit crct10dif_pclmul drm crct10dif_common libnvdimm mptscsih libata e1000 serio_raw mptbase crc32c_intel drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 120459 Comm: kworker/u256:1 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9747f178e2a0 ti: ffff974477d7c000 task.ti: ffff974477d7c000
RIP: 0010:[<ffffffff995150ca>]  [<ffffffff995150ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff974477d7f830  EFLAGS: 00000202
RAX: 000000000000000d RBX: 0000000000000001 RCX: ffff9747fd960498
RDX: 000000000000000d RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff974477d7f868 R08: ffff974539ca5000 R09: ffffffff9977f629
R10: ffff9747fd65f160 R11: ffffdcf3904e8200 R12: 0000000000013338
R13: 000000fc9977f57f R14: 0000000000000286 R15: 0000000000000286
FS:  0000000000000000(0000) GS:ffff9747fd640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa4b829dd00 CR3: 0000000431610000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9947d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9947d534>] flush_tlb_page+0x54/0xa0
[<ffffffff99603b18>] ptep_clear_flush+0x68/0xa0
[<ffffffff995fd0a6>] page_mkclean_one+0xa6/0xe0
[<ffffffff995feed3>] rmap_walk+0xf3/0x360
[<ffffffff995ff2e6>] page_mkclean+0x76/0xb0
[<ffffffff995fd000>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff995fc6e0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff995c7a50>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff995c7cc2>] write_cache_pages+0x1f2/0x470
[<ffffffffc0601030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff994e375c>] ? update_curr+0x14c/0x1e0
[<ffffffff994e71f4>] ? update_group_power+0x174/0x280
[<ffffffff9977f475>] ? cpumask_next_and+0x35/0x50
[<ffffffffc0600ffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff995c8d31>] do_writepages+0x21/0x50
[<ffffffff996776c0>] __writeback_single_inode+0x40/0x260
[<ffffffff99678254>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff9967855f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff99678a43>] wb_writeback+0x263/0x2f0
[<ffffffff9966490c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9967963b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff994bd0ff>] process_one_work+0x17f/0x440
[<ffffffff994be216>] worker_thread+0x126/0x3c0
[<ffffffff994be0f0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff994c50d1>] kthread+0xd1/0xe0
[<ffffffff994c5000>] ? insert_kthread_work+0x40/0x40
[<ffffffff99b8bd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff994c5000>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 14 9a f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1356519
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbc707e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbc677018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffbc6771b4
Build id: 3.10.0-862.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffbc7d8837
Build id: 3.10.0-862.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffffbc7d1ba4
Build id: 3.10.0-862.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffffbc79f7d0
Build id: 3.10.0-862.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffffbc79fa7a
Build id: 3.10.0-862.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc0478850
Build id: 3.10.0-862.el7.x86_64
0x150
9
? update_group_power
vmlinux 0xffffffffbc6db898
Build id: 3.10.0-862.el7.x86_64
0x28
10
? cpumask_next_and
vmlinux 0xffffffffbc94a275
Build id: 3.10.0-862.el7.x86_64
0x35
11
xfs_vm_writepages
xfs 0xffffffffc047865d
Build id: 3.10.0-862.el7.x86_64
0xbd
12
do_writepages
vmlinux 0xffffffffbc7a0e21
Build id: 3.10.0-862.el7.x86_64
0x21
13
__writeback_single_inode
vmlinux 0xffffffffbc848af0
Build id: 3.10.0-862.el7.x86_64
0x40
14
writeback_sb_inodes
vmlinux 0xffffffffbc849584
Build id: 3.10.0-862.el7.x86_64
0x1c4
15
__writeback_inodes_wb
vmlinux 0xffffffffbc8498ef
Build id: 3.10.0-862.el7.x86_64
0x9f
16
wb_writeback
vmlinux 0xffffffffbc84a123
Build id: 3.10.0-862.el7.x86_64
0x263
17
? get_nr_inodes
vmlinux 0xffffffffbc83665c
Build id: 3.10.0-862.el7.x86_64
0x4c
18
bdi_writeback_workfn
vmlinux 0xffffffffbc84aaab
Build id: 3.10.0-862.el7.x86_64
0x2cb
19
process_one_work
vmlinux 0xffffffffbc6b2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
20
worker_thread
vmlinux 0xffffffffbc6b3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
21
? manage_workers.isra.24
vmlinux 0xffffffffbc6b39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
22
kthread
vmlinux 0xffffffffbc6bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
23
? insert_kthread_work
vmlinux 0xffffffffbc6bad60
Build id: 3.10.0-862.el7.x86_64
0x40
24
ret_from_fork_nospec_begin
vmlinux 0xffffffffbcd1f637
Build id: 3.10.0-862.el7.x86_64
0x21
25
? insert_kthread_work
vmlinux 0xffffffffbc6bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 27s! [kworker/u256:2:6368]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 vmw_balloon snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm pcspkr joydev sg snd_timer nfit snd soundcore vmw_vmci libnvdimm shpchp i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix crct10dif_pclmul crct10dif_common crc32c_intel scsi_transport_spi libata e1000 mptscsih i2c_core serio_raw mptbase
CPU: 3 PID: 6368 Comm: kworker/u256:2 Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff8d170e021fa0 ti: ffff8d173a658000 task.ti: ffff8d173a658000
RIP: 0010:[<ffffffffbc707e3a>]  [<ffffffffbc707e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8d173a65b8b8  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff8d17b969cf00
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8d173a65b8f0 R08: ffff8d17b762bc00 R09: ffffffffbc94a3d9
R10: ffff8d17b96dbb00 R11: ffffecf8c4d0b600 R12: 0000000000000296
R13: ffff8d173a65b868 R14: ffff8d173a65b858 R15: ffff8d17b4cecb00
FS:  0000000000000000(0000) GS:ffff8d17b96c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb404b30000 CR3: 000000012ec0e000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffbc677018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbc6771b4>] flush_tlb_page+0x54/0xa0
[<ffffffffbc7d8837>] ptep_clear_flush+0x67/0xa0
[<ffffffffbc7d1ba4>] page_mkclean+0x144/0x210
[<ffffffffbc79f7d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffbc79fa7a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc0478850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffbc6db898>] ? update_group_power+0x28/0x280
[<ffffffffbc94a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffc047865d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffffbc7a0e21>] do_writepages+0x21/0x50
[<ffffffffbc848af0>] __writeback_single_inode+0x40/0x260
[<ffffffffbc849584>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffbc8498ef>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffbc84a123>] wb_writeback+0x263/0x2f0
[<ffffffffbc83665c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffbc84aaab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffbc6b2dff>] process_one_work+0x17f/0x440
[<ffffffffbc6b3ac6>] worker_thread+0x126/0x3c0
[<ffffffffbc6b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffbc6bae31>] kthread+0xd1/0xe0
[<ffffffffbc6bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffbcd1f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffbc6bad60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 33 bd f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1257505
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff98316cba
Build id: 3.10.0-1062.18.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9827e608
Build id: 3.10.0-1062.18.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff9827e7a4
Build id: 3.10.0-1062.18.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff98405638
Build id: 3.10.0-1062.18.1.el7.x86_64
0x68
5
page_mkclean_one
vmlinux 0xffffffff983febe6
Build id: 3.10.0-1062.18.1.el7.x86_64
0xa6
6
rmap_walk
vmlinux 0xffffffff984009f3
Build id: 3.10.0-1062.18.1.el7.x86_64
0xf3
7
page_mkclean
vmlinux 0xffffffff98400e06
Build id: 3.10.0-1062.18.1.el7.x86_64
0x76
8
? page_referenced_one
vmlinux 0xffffffff983feb40
Build id: 3.10.0-1062.18.1.el7.x86_64
0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffff983fe220
Build id: 3.10.0-1062.18.1.el7.x86_64
0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffff983c93b0
Build id: 3.10.0-1062.18.1.el7.x86_64
0x50
11
write_cache_pages
vmlinux 0xffffffff983c9622
Build id: 3.10.0-1062.18.1.el7.x86_64
0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc06e3030
Build id: 3.10.0-1062.18.1.el7.x86_64
0xa0
13
? submit_bio
vmlinux 0xffffffff985511e0
Build id: 3.10.0-1062.18.1.el7.x86_64
0x70
14
? update_group_power
vmlinux 0xffffffff982e84f4
Build id: 3.10.0-1062.18.1.el7.x86_64
0x174
15
? cpumask_next_and
vmlinux 0xffffffff98581845
Build id: 3.10.0-1062.18.1.el7.x86_64
0x35
16
xfs_vm_writepages
xfs 0xffffffffc06e2ffb
Build id: 3.10.0-1062.18.1.el7.x86_64
0x6b
17
do_writepages
vmlinux 0xffffffff983ca6a1
Build id: 3.10.0-1062.18.1.el7.x86_64
0x21
18
__writeback_single_inode
vmlinux 0xffffffff98479b00
Build id: 3.10.0-1062.18.1.el7.x86_64
0x40
19
? wake_up_bit
vmlinux 0xffffffff982c73d5
Build id: 3.10.0-1062.18.1.el7.x86_64
0x25
20
writeback_sb_inodes
vmlinux 0xffffffff9847a694
Build id: 3.10.0-1062.18.1.el7.x86_64
0x1c4
21
__writeback_inodes_wb
vmlinux 0xffffffff9847a99f
Build id: 3.10.0-1062.18.1.el7.x86_64
0x9f
22
wb_writeback
vmlinux 0xffffffff9847ae83
Build id: 3.10.0-1062.18.1.el7.x86_64
0x263
23
? get_nr_inodes
vmlinux 0xffffffff98466d2c
Build id: 3.10.0-1062.18.1.el7.x86_64
0x4c
24
bdi_writeback_workfn
vmlinux 0xffffffff9847ba7b
Build id: 3.10.0-1062.18.1.el7.x86_64
0x2cb
25
process_one_work
vmlinux 0xffffffff982be34f
Build id: 3.10.0-1062.18.1.el7.x86_64
0x17f
26
worker_thread
vmlinux 0xffffffff982bf466
Build id: 3.10.0-1062.18.1.el7.x86_64
0x126
27
? manage_workers.isra.26
vmlinux 0xffffffff982bf340
Build id: 3.10.0-1062.18.1.el7.x86_64
0x2a0
28
kthread
vmlinux 0xffffffff982c6321
Build id: 3.10.0-1062.18.1.el7.x86_64
0xd1
29
? insert_kthread_work
vmlinux 0xffffffff982c6250
Build id: 3.10.0-1062.18.1.el7.x86_64
0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffff9898dd37
Build id: 3.10.0-1062.18.1.el7.x86_64
0x21
31
? insert_kthread_work
vmlinux 0xffffffff982c6250
Build id: 3.10.0-1062.18.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [kworker/u50:1:163]
Modules linked in: cmac arc4 md4 xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink nls_utf8 cifs ccm dns_resolver vmw_vsock_vmci_transport vsock sunrpc ppdev sb_edac vmw_balloon coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm ata_piix libahci nfit crct10dif_pclmul libnvdimm
libata crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 12 PID: 163 Comm: kworker/u50:1 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8d0cece10000 ti: ffff8d0cece18000 task.ti: ffff8d0cece18000
RIP: 0010:[<ffffffff98316cba>]  [<ffffffff98316cba>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8d0cece1b830  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009827e440 RCX: ffff8d08efa20818
RDX: 0000000000000000 RSI: 0000000000000018 RDI: 0000000000000000
RBP: ffff8d0cece1b868 R08: ffff8d097a56fc00 R09: ffffffff985819f9
R10: ffff8d0cff61f160 R11: fffff3ef000f6400 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8d0cece1b7e0 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8d0cff600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000015f16b0 CR3: 0000000665610000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9827e608>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9827e7a4>] flush_tlb_page+0x54/0xa0
[<ffffffff98405638>] ptep_clear_flush+0x68/0xa0
[<ffffffff983febe6>] page_mkclean_one+0xa6/0xe0
[<ffffffff984009f3>] rmap_walk+0xf3/0x360
[<ffffffff98400e06>] page_mkclean+0x76/0xb0
[<ffffffff983feb40>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff983fe220>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff983c93b0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff983c9622>] write_cache_pages+0x1f2/0x470
[<ffffffffc06e3030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff985511e0>] ? submit_bio+0x70/0x150
[<ffffffff982e84f4>] ? update_group_power+0x174/0x280
[<ffffffff98581845>] ? cpumask_next_and+0x35/0x50
[<ffffffffc06e2ffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff983ca6a1>] do_writepages+0x21/0x50
[<ffffffff98479b00>] __writeback_single_inode+0x40/0x260
[<ffffffff982c73d5>] ? wake_up_bit+0x25/0x30
[<ffffffff9847a694>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff9847a99f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff9847ae83>] wb_writeback+0x263/0x2f0
[<ffffffff98466d2c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9847ba7b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff982be34f>] process_one_work+0x17f/0x440
[<ffffffff982bf466>] worker_thread+0x126/0x3c0
[<ffffffff982bf340>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff982c6321>] kthread+0xd1/0xe0
[<ffffffff982c6250>] ? insert_kthread_work+0x40/0x40
[<ffffffff9898dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff982c6250>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ee 5c c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 e8 f4 98 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5c c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1146480
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9f51e23a
Build id: 3.10.0-1160.105.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff9f481fa8
Build id: 3.10.0-1160.105.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff9f482144
Build id: 3.10.0-1160.105.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff9f614228
Build id: 3.10.0-1160.105.1.el7.x86_64
0x68
5
page_mkclean_one
vmlinux 0xffffffff9f60d5fe
Build id: 3.10.0-1160.105.1.el7.x86_64
0xae
6
rmap_walk
vmlinux 0xffffffff9f60f473
Build id: 3.10.0-1160.105.1.el7.x86_64
0xf3
7
page_mkclean
vmlinux 0xffffffff9f60f896
Build id: 3.10.0-1160.105.1.el7.x86_64
0x76
8
? page_referenced_one
vmlinux 0xffffffff9f60d550
Build id: 3.10.0-1160.105.1.el7.x86_64
0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffff9f60cc20
Build id: 3.10.0-1160.105.1.el7.x86_64
0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffff9f5d5770
Build id: 3.10.0-1160.105.1.el7.x86_64
0x50
11
write_cache_pages
vmlinux 0xffffffff9f5d59e2
Build id: 3.10.0-1160.105.1.el7.x86_64
0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc052d6c0
Build id: 3.10.0-1160.105.1.el7.x86_64
0xb0
13
? update_group_power
vmlinux 0xffffffff9f4ee68c
Build id: 3.10.0-1160.105.1.el7.x86_64
0x17c
14
? cpumask_next_and
vmlinux 0xffffffff9f79c5d5
Build id: 3.10.0-1160.105.1.el7.x86_64
0x35
15
xfs_vm_writepages
xfs 0xffffffffc052d67b
Build id: 3.10.0-1160.105.1.el7.x86_64
0x6b
16
do_writepages
vmlinux 0xffffffff9f5d6ab1
Build id: 3.10.0-1160.105.1.el7.x86_64
0x21
17
__writeback_single_inode
vmlinux 0xffffffff9f68ccf0
Build id: 3.10.0-1160.105.1.el7.x86_64
0x40
18
writeback_sb_inodes
vmlinux 0xffffffff9f68d8bc
Build id: 3.10.0-1160.105.1.el7.x86_64
0x1cc
19
__writeback_inodes_wb
vmlinux 0xffffffff9f68dbc7
Build id: 3.10.0-1160.105.1.el7.x86_64
0xa7
20
wb_writeback
vmlinux 0xffffffff9f68e0a3
Build id: 3.10.0-1160.105.1.el7.x86_64
0x263
21
? get_nr_inodes
vmlinux 0xffffffff9f67906c
Build id: 3.10.0-1160.105.1.el7.x86_64
0x4c
22
bdi_writeback_workfn
vmlinux 0xffffffff9f68ecaf
Build id: 3.10.0-1160.105.1.el7.x86_64
0x2cf
23
process_one_work
vmlinux 0xffffffff9f4c32ef
Build id: 3.10.0-1160.105.1.el7.x86_64
0x17f
24
worker_thread
vmlinux 0xffffffff9f4c4436
Build id: 3.10.0-1160.105.1.el7.x86_64
0x126
25
? manage_workers.isra.26
vmlinux 0xffffffff9f4c4310
Build id: 3.10.0-1160.105.1.el7.x86_64
0x2b0
26
kthread
vmlinux 0xffffffff9f4cb621
Build id: 3.10.0-1160.105.1.el7.x86_64
0xd1
27
? insert_kthread_work
vmlinux 0xffffffff9f4cb550
Build id: 3.10.0-1160.105.1.el7.x86_64
0x40
28
ret_from_fork_nospec_begin
vmlinux 0xffffffff9fbc51dd
Build id: 3.10.0-1160.105.1.el7.x86_64
0x7
29
? insert_kthread_work
vmlinux 0xffffffff9f4cb550
Build id: 3.10.0-1160.105.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#31 stuck for 67s! [kworker/u256:0:93948]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev vmw_balloon snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device iosf_mbi crc32_pclmul snd_pcm ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_timer snd joydev
pcspkr sg vmw_vmci soundcore i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit ata_piix libnvdimm libata mptspi scsi_transport_spi e1000 crct10dif_pclmul mptscsih crct10dif_common mptbase crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 31 PID: 93948 Comm: kworker/u256:0 Not tainted 3.10.0-1160.105.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff966520929080 ti: ffff9664e2d70000 task.ti: ffff9664e2d70000
RIP: 0010:[<ffffffff9f51e23a>]  [<ffffffff9f51e23a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9664e2d73830  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800009f481b10 RCX: ffff966cfd6a0ab0
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9664e2d73868 R08: ffff965e58df2800 R09: ffffffff9f79c7b9
R10: ffff966cfdddf160 R11: fffffd8caf34dc00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9664e2d737e0 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff966cfddc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5a447029e5 CR3: 0000000ffc552000 CR4: 0000000000760fe0
PKRU: 00000000
Call Trace:
[<ffffffff9f481fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9f482144>] flush_tlb_page+0x54/0xa0
[<ffffffff9f614228>] ptep_clear_flush+0x68/0xa0
[<ffffffff9f60d5fe>] page_mkclean_one+0xae/0xe0
[<ffffffff9f60f473>] rmap_walk+0xf3/0x370
[<ffffffff9f60f896>] page_mkclean+0x76/0xb0
[<ffffffff9f60d550>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff9f60cc20>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff9f5d5770>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff9f5d59e2>] write_cache_pages+0x1f2/0x480
[<ffffffffc052d6c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffff9f4ee68c>] ? update_group_power+0x17c/0x280
[<ffffffff9f79c5d5>] ? cpumask_next_and+0x35/0x50
[<ffffffffc052d67b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffff9f5d6ab1>] do_writepages+0x21/0x50
[<ffffffff9f68ccf0>] __writeback_single_inode+0x40/0x260
[<ffffffff9f68d8bc>] writeback_sb_inodes+0x1cc/0x430
[<ffffffff9f68dbc7>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffff9f68e0a3>] wb_writeback+0x263/0x2f0
[<ffffffff9f67906c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9f68ecaf>] bdi_writeback_workfn+0x2cf/0x470
[<ffffffff9f4c32ef>] process_one_work+0x17f/0x440
[<ffffffff9f4c4436>] worker_thread+0x126/0x3c0
[<ffffffff9f4c4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff9f4cb621>] kthread+0xd1/0xe0
[<ffffffff9f4cb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff9fbc51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff9f4cb550>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 80 19 15 a0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #850337
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb971173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb967a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffb967a734
Build id: 3.10.0-957.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffb97fd587
Build id: 3.10.0-957.el7.x86_64
0x67
5
page_mkclean_one
vmlinux 0xffffffffb97f6b66
Build id: 3.10.0-957.el7.x86_64
0xa6
6
rmap_walk
vmlinux 0xffffffffb97f8983
Build id: 3.10.0-957.el7.x86_64
0xf3
7
page_mkclean
vmlinux 0xffffffffb97f8d96
Build id: 3.10.0-957.el7.x86_64
0x76
8
? page_referenced_one
vmlinux 0xffffffffb97f6ac0
Build id: 3.10.0-957.el7.x86_64
0x180
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffb97f6200
Build id: 3.10.0-957.el7.x86_64
0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffb97c1d60
Build id: 3.10.0-957.el7.x86_64
0x50
11
write_cache_pages
vmlinux 0xffffffffb97c200a
Build id: 3.10.0-957.el7.x86_64
0x22a
12
? xfs_aops_discard_page
xfs 0xffffffffc05325b0
Build id: 3.10.0-957.el7.x86_64
0x150
13
? submit_bio
vmlinux 0xffffffffb9944f30
Build id: 3.10.0-957.el7.x86_64
0x70
14
? update_group_power
vmlinux 0xffffffffb96e3868
Build id: 3.10.0-957.el7.x86_64
0x28
15
xfs_vm_writepages
xfs 0xffffffffc05323ab
Build id: 3.10.0-957.el7.x86_64
0x6b
16
do_writepages
vmlinux 0xffffffffb97c3181
Build id: 3.10.0-957.el7.x86_64
0x21
17
__writeback_single_inode
vmlinux 0xffffffffb986f930
Build id: 3.10.0-957.el7.x86_64
0x40
18
writeback_sb_inodes
vmlinux 0xffffffffb98703c4
Build id: 3.10.0-957.el7.x86_64
0x1c4
19
__writeback_inodes_wb
vmlinux 0xffffffffb987072f
Build id: 3.10.0-957.el7.x86_64
0x9f
20
wb_writeback
vmlinux 0xffffffffb9870f63
Build id: 3.10.0-957.el7.x86_64
0x263
21
? get_nr_inodes
vmlinux 0xffffffffb985cd7c
Build id: 3.10.0-957.el7.x86_64
0x4c
22
bdi_writeback_workfn
vmlinux 0xffffffffb98718eb
Build id: 3.10.0-957.el7.x86_64
0x2cb
23
process_one_work
vmlinux 0xffffffffb96b9d4f
Build id: 3.10.0-957.el7.x86_64
0x17f
24
worker_thread
vmlinux 0xffffffffb96bade6
Build id: 3.10.0-957.el7.x86_64
0x126
25
? manage_workers.isra.25
vmlinux 0xffffffffb96bacc0
Build id: 3.10.0-957.el7.x86_64
0x2a0
26
kthread
vmlinux 0xffffffffb96c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
27
? insert_kthread_work
vmlinux 0xffffffffb96c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
28
ret_from_fork_nospec_begin
vmlinux 0xffffffffb9d74c37
Build id: 3.10.0-957.el7.x86_64
0x21
29
? insert_kthread_work
vmlinux 0xffffffffb96c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 26s! [kworker/u256:1:14135]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac coretemp iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device
snd_pcm joydev pcspkr snd_timer snd soundcore sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ata_generic pata_acpi serio_raw vmwgfx drm_kms_helper e1000 syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi scsi_transport_spi ata_piix mptscsih libata mptbase nfit drm_panel_orientation_quirks libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 14135 Comm: kworker/u256:1 Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8ca926b0a080 ti: ffff8ca927244000 task.ti: ffff8ca927244000
RIP: 0010:[<ffffffffb971173a>]  [<ffffffffb971173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8ca927247820  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000b967a3d0 RCX: ffff8ca9b5660678
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8ca927247858 R08: ffff8ca8fc85c800 R09: ffffffffb99755c9
R10: ffff8ca9b579f160 R11: ffffe377485fae00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8ca9272477d0 R15: 1400000000000001
FS:  0000000000000000(0000) GS:ffff8ca9b5780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc8f8bc3000 CR3: 000000012e210000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb967a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb967a734>] flush_tlb_page+0x54/0xa0
[<ffffffffb97fd587>] ptep_clear_flush+0x67/0xa0
[<ffffffffb97f6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffffb97f8983>] rmap_walk+0xf3/0x360
[<ffffffffb97f8d96>] page_mkclean+0x76/0xb0
[<ffffffffb97f6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffffb97f6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb97c1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb97c200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc05325b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffb9944f30>] ? submit_bio+0x70/0x150
[<ffffffffb96e3868>] ? update_group_power+0x28/0x280
[<ffffffffc05323ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb97c3181>] do_writepages+0x21/0x50
[<ffffffffb986f930>] __writeback_single_inode+0x40/0x260
[<ffffffffb98703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffb987072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb9870f63>] wb_writeback+0x263/0x2f0
[<ffffffffb985cd7c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffb98718eb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb96b9d4f>] process_one_work+0x17f/0x440
[<ffffffffb96bade6>] worker_thread+0x126/0x3c0
[<ffffffffb96bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffb96c1c31>] kthread+0xd1/0xe0
[<ffffffffb96c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb9d74c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb96c1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 34 ba f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #777029
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9771e0ba
Build id: 3.10.0-1160.83.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff97681f88
Build id: 3.10.0-1160.83.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff97682124
Build id: 3.10.0-1160.83.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff97813ef8
Build id: 3.10.0-1160.83.1.el7.x86_64
0x68
5
page_mkclean_one
vmlinux 0xffffffff9780d2de
Build id: 3.10.0-1160.83.1.el7.x86_64
0xae
6
rmap_walk
vmlinux 0xffffffff9780f143
Build id: 3.10.0-1160.83.1.el7.x86_64
0xf3
7
page_mkclean
vmlinux 0xffffffff9780f566
Build id: 3.10.0-1160.83.1.el7.x86_64
0x76
8
? page_referenced_one
vmlinux 0xffffffff9780d230
Build id: 3.10.0-1160.83.1.el7.x86_64
0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffff9780c8c0
Build id: 3.10.0-1160.83.1.el7.x86_64
0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffff977d55e0
Build id: 3.10.0-1160.83.1.el7.x86_64
0x50
11
write_cache_pages
vmlinux 0xffffffff977d5852
Build id: 3.10.0-1160.83.1.el7.x86_64
0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc06be6c0
Build id: 3.10.0-1160.83.1.el7.x86_64
0xb0
13
? update_group_power
vmlinux 0xffffffff976ee3b8
Build id: 3.10.0-1160.83.1.el7.x86_64
0x28
14
xfs_vm_writepages
xfs 0xffffffffc06be67b
Build id: 3.10.0-1160.83.1.el7.x86_64
0x6b
15
do_writepages
vmlinux 0xffffffff977d6921
Build id: 3.10.0-1160.83.1.el7.x86_64
0x21
16
__writeback_single_inode
vmlinux 0xffffffff9788c790
Build id: 3.10.0-1160.83.1.el7.x86_64
0x40
17
? wake_up_bit
vmlinux 0xffffffff976cc635
Build id: 3.10.0-1160.83.1.el7.x86_64
0x25
18
writeback_sb_inodes
vmlinux 0xffffffff9788d35c
Build id: 3.10.0-1160.83.1.el7.x86_64
0x1cc
19
__writeback_inodes_wb
vmlinux 0xffffffff9788d667
Build id: 3.10.0-1160.83.1.el7.x86_64
0xa7
20
wb_writeback
vmlinux 0xffffffff9788db43
Build id: 3.10.0-1160.83.1.el7.x86_64
0x263
21
? get_nr_inodes
vmlinux 0xffffffff97878abc
Build id: 3.10.0-1160.83.1.el7.x86_64
0x4c
22
bdi_writeback_workfn
vmlinux 0xffffffff9788e74f
Build id: 3.10.0-1160.83.1.el7.x86_64
0x2cf
23
process_one_work
vmlinux 0xffffffff976c31df
Build id: 3.10.0-1160.83.1.el7.x86_64
0x17f
24
worker_thread
vmlinux 0xffffffff976c4326
Build id: 3.10.0-1160.83.1.el7.x86_64
0x126
25
? manage_workers.isra.26
vmlinux 0xffffffff976c4200
Build id: 3.10.0-1160.83.1.el7.x86_64
0x2b0
26
kthread
vmlinux 0xffffffff976cb511
Build id: 3.10.0-1160.83.1.el7.x86_64
0xd1
27
? insert_kthread_work
vmlinux 0xffffffff976cb440
Build id: 3.10.0-1160.83.1.el7.x86_64
0x40
28
ret_from_fork_nospec_begin
vmlinux 0xffffffff97dc51dd
Build id: 3.10.0-1160.83.1.el7.x86_64
0x7
29
? insert_kthread_work
vmlinux 0xffffffff976cb440
Build id: 3.10.0-1160.83.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kworker/u256:0:8657]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter binfmt_misc vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev iosf_mbi vmw_balloon crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi snd_ac97_codec aesni_intel lrw ac97_bus gf128mul glue_helper ablk_helper snd_seq cryptd snd_seq_device snd_pcm joydev pcspkr snd_timer snd sg vmw_vmci soundcore i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix drm libata nfit mptspi scsi_transport_spi libnvdimm serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks fuse
CPU: 7 PID: 8657 Comm: kworker/u256:0 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.83.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff8e1d0fb4e300 ti: ffff8e1d07e44000 task.ti: ffff8e1d07e44000
RIP: 0010:[<ffffffff9771e0ba>]  [<ffffffff9771e0ba>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8e1d07e47830  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0008000097681af0 RCX: ffff8e1f5d6206f0
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8e1d07e47868 R08: ffff8e18ac55d000 R09: ffffffff9799c179
R10: ffff8e1f5d7df160 R11: ffffcac8e0631800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8e1d07e477e0 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8e1f5d7c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1228002748 CR3: 0000000364010000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffff97681f88>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff97682124>] flush_tlb_page+0x54/0xa0
[<ffffffff97813ef8>] ptep_clear_flush+0x68/0xa0
[<ffffffff9780d2de>] page_mkclean_one+0xae/0xe0
[<ffffffff9780f143>] rmap_walk+0xf3/0x370
[<ffffffff9780f566>] page_mkclean+0x76/0xb0
[<ffffffff9780d230>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff9780c8c0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff977d55e0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff977d5852>] write_cache_pages+0x1f2/0x480
[<ffffffffc06be6c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffff976ee3b8>] ? update_group_power+0x28/0x280
[<ffffffffc06be67b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffff977d6921>] do_writepages+0x21/0x50
[<ffffffff9788c790>] __writeback_single_inode+0x40/0x260
[<ffffffff976cc635>] ? wake_up_bit+0x25/0x30
[<ffffffff9788d35c>] writeback_sb_inodes+0x1cc/0x430
[<ffffffff9788d667>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffff9788db43>] wb_writeback+0x263/0x2f0
[<ffffffff97878abc>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9788e74f>] bdi_writeback_workfn+0x2cf/0x470
[<ffffffff976c31df>] process_one_work+0x17f/0x440
[<ffffffff976c4326>] worker_thread+0x126/0x3c0
[<ffffffff976c4200>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff976cb511>] kthread+0xd1/0xe0
[<ffffffff976cb440>] ? insert_kthread_work+0x40/0x40
[<ffffffff97dc51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff976cb440>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 35 98 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d 19 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #754062
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff86907e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff86877018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff868771b4
Build id: 3.10.0-862.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff869d8837
Build id: 3.10.0-862.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffff869d1ba4
Build id: 3.10.0-862.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffff8699f7d0
Build id: 3.10.0-862.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffff8699fa7a
Build id: 3.10.0-862.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc0374850
Build id: 3.10.0-862.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffff86b1a940
Build id: 3.10.0-862.el7.x86_64
0x70
10
? update_group_power
vmlinux 0xffffffff868db9e4
Build id: 3.10.0-862.el7.x86_64
0x174
11
? cpumask_next_and
vmlinux 0xffffffff86b4a275
Build id: 3.10.0-862.el7.x86_64
0x35
12
xfs_vm_writepages
xfs 0xffffffffc037465d
Build id: 3.10.0-862.el7.x86_64
0xbd
13
do_writepages
vmlinux 0xffffffff869a0e21
Build id: 3.10.0-862.el7.x86_64
0x21
14
__writeback_single_inode
vmlinux 0xffffffff86a48af0
Build id: 3.10.0-862.el7.x86_64
0x40
15
writeback_sb_inodes
vmlinux 0xffffffff86a49584
Build id: 3.10.0-862.el7.x86_64
0x1c4
16
__writeback_inodes_wb
vmlinux 0xffffffff86a498ef
Build id: 3.10.0-862.el7.x86_64
0x9f
17
wb_writeback
vmlinux 0xffffffff86a4a123
Build id: 3.10.0-862.el7.x86_64
0x263
18
? get_nr_inodes
vmlinux 0xffffffff86a3665c
Build id: 3.10.0-862.el7.x86_64
0x4c
19
bdi_writeback_workfn
vmlinux 0xffffffff86a4aaab
Build id: 3.10.0-862.el7.x86_64
0x2cb
20
process_one_work
vmlinux 0xffffffff868b2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
21
worker_thread
vmlinux 0xffffffff868b3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
22
? manage_workers.isra.24
vmlinux 0xffffffff868b39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
23
kthread
vmlinux 0xffffffff868bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
24
? insert_kthread_work
vmlinux 0xffffffff868bad60
Build id: 3.10.0-862.el7.x86_64
0x40
25
ret_from_fork_nospec_begin
vmlinux 0xffffffff86f1f637
Build id: 3.10.0-862.el7.x86_64
0x21
26
? insert_kthread_work
vmlinux 0xffffffff868bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u256:3:6368]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmhgfs(OE) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 crc32_pclmul snd_rawmidi ghash_clmulni_intel snd_ac97_codec aesni_intel ac97_bus lrw gf128mul glue_helper snd_seq snd_seq_device snd_pcm ablk_helper cryptd snd_timer snd soundcore
joydev pcspkr sg ppdev vmw_balloon nfit parport_pc libnvdimm parport shpchp i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel mptspi ata_piix scsi_transport_spi mptscsih libata e1000 mptbase serio_raw i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 6368 Comm: kworker/u256:3 Kdump: loaded Tainted: G           OE  ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff99ab35f3eeb0 ti: ffff99ab7e0e0000 task.ti: ffff99ab7e0e0000
RIP: 0010:[<ffffffff86907e3a>]  [<ffffffff86907e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff99ab7e0e38b8  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 00000000000000fc RCX: ffff99ad3579cd90
RDX: 0000000000000006 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff99ab7e0e38f0 R08: ffff99ac7c853400 R09: ffffffff86b4a3d9
R10: ffff99ad3565bb00 R11: ffffea44c854ca00 R12: 0000000000000296
R13: ffff99ab7e0e3868 R14: ffff99ad318940d8 R15: ffff99ab34cb2680
FS:  0000000000000000(0000) GS:ffff99ad35640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8604bd4000 CR3: 00000000ae43e000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffff86877018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff868771b4>] flush_tlb_page+0x54/0xa0
[<ffffffff869d8837>] ptep_clear_flush+0x67/0xa0
[<ffffffff869d1ba4>] page_mkclean+0x144/0x210
[<ffffffff8699f7d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8699fa7a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc0374850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff86b1a940>] ? submit_bio+0x70/0x150
[<ffffffff868db9e4>] ? update_group_power+0x174/0x280
[<ffffffff86b4a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffc037465d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffff869a0e21>] do_writepages+0x21/0x50
[<ffffffff86a48af0>] __writeback_single_inode+0x40/0x260
[<ffffffff86a49584>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff86a498ef>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff86a4a123>] wb_writeback+0x263/0x2f0
[<ffffffff86a3665c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff86a4aaab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff868b2dff>] process_one_work+0x17f/0x440
[<ffffffff868b3ac6>] worker_thread+0x126/0x3c0
[<ffffffff868b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff868bae31>] kthread+0xd1/0xe0
[<ffffffff868bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff86f1f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff868bad60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 53 87 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #372627
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff87b07e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff87a77018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff87a771b4
Build id: 3.10.0-862.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff87bd8837
Build id: 3.10.0-862.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffff87bd1ba4
Build id: 3.10.0-862.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffff87b9f7d0
Build id: 3.10.0-862.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffff87b9fa7a
Build id: 3.10.0-862.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc0456850
Build id: 3.10.0-862.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffff87d1a940
Build id: 3.10.0-862.el7.x86_64
0x70
10
? update_group_power
vmlinux 0xffffffff87adb9e4
Build id: 3.10.0-862.el7.x86_64
0x174
11
xfs_vm_writepages
xfs 0xffffffffc045665d
Build id: 3.10.0-862.el7.x86_64
0xbd
12
do_writepages
vmlinux 0xffffffff87ba0e21
Build id: 3.10.0-862.el7.x86_64
0x21
13
__writeback_single_inode
vmlinux 0xffffffff87c48af0
Build id: 3.10.0-862.el7.x86_64
0x40
14
writeback_sb_inodes
vmlinux 0xffffffff87c49584
Build id: 3.10.0-862.el7.x86_64
0x1c4
15
__writeback_inodes_wb
vmlinux 0xffffffff87c498ef
Build id: 3.10.0-862.el7.x86_64
0x9f
16
wb_writeback
vmlinux 0xffffffff87c4a123
Build id: 3.10.0-862.el7.x86_64
0x263
17
? get_nr_inodes
vmlinux 0xffffffff87c3665c
Build id: 3.10.0-862.el7.x86_64
0x4c
18
bdi_writeback_workfn
vmlinux 0xffffffff87c4aaab
Build id: 3.10.0-862.el7.x86_64
0x2cb
19
process_one_work
vmlinux 0xffffffff87ab2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
20
worker_thread
vmlinux 0xffffffff87ab3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
21
? manage_workers.isra.24
vmlinux 0xffffffff87ab39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
22
kthread
vmlinux 0xffffffff87abae31
Build id: 3.10.0-862.el7.x86_64
0xd1
23
? insert_kthread_work
vmlinux 0xffffffff87abad60
Build id: 3.10.0-862.el7.x86_64
0x40
24
ret_from_fork_nospec_begin
vmlinux 0xffffffff8811f637
Build id: 3.10.0-862.el7.x86_64
0x21
25
? insert_kthread_work
vmlinux 0xffffffff87abad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [kworker/u256:1:5354]
Modules linked in: tcp_lp fuse vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper snd_seq ablk_helper cryptd snd_seq_device snd_pcm snd_timer snd vmw_balloon soundcore nfit ppdev joydev pcspkr sg libnvdimm parport_pc parport shpchp i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm ata_piix mptspi crc32c_intel libata scsi_transport_spi e1000 mptscsih serio_raw mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 5354 Comm: kworker/u256:1 Kdump: loaded Tainted: G        W    L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff99178f802f70 ti: ffff9917a7ef0000 task.ti: ffff9917a7ef0000
RIP: 0010:[<ffffffff87b07e3a>]  [<ffffffff87b07e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9917a7ef38b8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9917fb61cd98
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9917a7ef38f0 R08: ffff9917fa6c4000 R09: ffffffff87d4a3d9
R10: ffff9917fb71bb00 R11: ffffd5f441e22200 R12: 0000000000000296
R13: ffff9917a7ef3868 R14: ffff9917cd0de918 R15: ffff9917cd0de900
FS:  0000000000000000(0000) GS:ffff9917fb700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000f059f950 CR3: 000000002260e000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffff87a77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff87a771b4>] flush_tlb_page+0x54/0xa0
[<ffffffff87bd8837>] ptep_clear_flush+0x67/0xa0
[<ffffffff87bd1ba4>] page_mkclean+0x144/0x210
[<ffffffff87b9f7d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff87b9fa7a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc0456850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff87d1a940>] ? submit_bio+0x70/0x150
[<ffffffff87adb9e4>] ? update_group_power+0x174/0x280
[<ffffffffc045665d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffff87ba0e21>] do_writepages+0x21/0x50
[<ffffffff87c48af0>] __writeback_single_inode+0x40/0x260
[<ffffffff87c49584>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff87c498ef>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff87c4a123>] wb_writeback+0x263/0x2f0
[<ffffffff87c3665c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff87c4aaab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff87ab2dff>] process_one_work+0x17f/0x440
[<ffffffff87ab3ac6>] worker_thread+0x126/0x3c0
[<ffffffff87ab39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff87abae31>] kthread+0xd1/0xe0
[<ffffffff87abad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff8811f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff87abad60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 73 88 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #367360
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb1d170de
Build id: 3.10.0-1160.59.1.el7.x86_64
0x20e
2
native_flush_tlb_others
vmlinux 0xffffffffb1c7e258
Build id: 3.10.0-1160.59.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffb1c7e3f4
Build id: 3.10.0-1160.59.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffb1e08108
Build id: 3.10.0-1160.59.1.el7.x86_64
0x68
5
page_mkclean_one
vmlinux 0xffffffffb1e016a6
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa6
6
rmap_walk
vmlinux 0xffffffffb1e034b3
Build id: 3.10.0-1160.59.1.el7.x86_64
0xf3
7
page_mkclean
vmlinux 0xffffffffb1e038c6
Build id: 3.10.0-1160.59.1.el7.x86_64
0x76
8
? page_referenced_one
vmlinux 0xffffffffb1e01600
Build id: 3.10.0-1160.59.1.el7.x86_64
0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffb1e00ce0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffb1dca6c0
Build id: 3.10.0-1160.59.1.el7.x86_64
0x50
11
write_cache_pages
vmlinux 0xffffffffb1dca932
Build id: 3.10.0-1160.59.1.el7.x86_64
0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc046c290
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa0
13
? submit_bio
vmlinux 0xffffffffb1f55e80
Build id: 3.10.0-1160.59.1.el7.x86_64
0x70
14
? update_group_power
vmlinux 0xffffffffb1ce8118
Build id: 3.10.0-1160.59.1.el7.x86_64
0x28
15
xfs_vm_writepages
xfs 0xffffffffc046c25b
Build id: 3.10.0-1160.59.1.el7.x86_64
0x6b
16
do_writepages
vmlinux 0xffffffffb1dcb9b1
Build id: 3.10.0-1160.59.1.el7.x86_64
0x21
17
__writeback_single_inode
vmlinux 0xffffffffb1e7df10
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
18
? wake_up_bit
vmlinux 0xffffffffb1cc6f15
Build id: 3.10.0-1160.59.1.el7.x86_64
0x25
19
writeback_sb_inodes
vmlinux 0xffffffffb1e7ea94
Build id: 3.10.0-1160.59.1.el7.x86_64
0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffffb1e7ed9f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x9f
21
wb_writeback
vmlinux 0xffffffffb1e7f273
Build id: 3.10.0-1160.59.1.el7.x86_64
0x263
22
? get_nr_inodes
vmlinux 0xffffffffb1e6ae6c
Build id: 3.10.0-1160.59.1.el7.x86_64
0x4c
23
bdi_writeback_workfn
vmlinux 0xffffffffb1e7fe6b
Build id: 3.10.0-1160.59.1.el7.x86_64
0x2cb
24
process_one_work
vmlinux 0xffffffffb1cbde8f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x17f
25
worker_thread
vmlinux 0xffffffffb1cbefa6
Build id: 3.10.0-1160.59.1.el7.x86_64
0x126
26
? manage_workers.isra.26
vmlinux 0xffffffffb1cbee80
Build id: 3.10.0-1160.59.1.el7.x86_64
0x2a0
27
kthread
vmlinux 0xffffffffb1cc5e61
Build id: 3.10.0-1160.59.1.el7.x86_64
0xd1
28
? insert_kthread_work
vmlinux 0xffffffffb1cc5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffffb2399df7
Build id: 3.10.0-1160.59.1.el7.x86_64
0x21
30
? insert_kthread_work
vmlinux 0xffffffffb1cc5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u256:2:110743]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul vmw_balloon glue_helper ablk_helper snd_ens1371 cryptd snd_rawmidi snd_ac97_codec ac97_bus snd_seq joydev snd_seq_device pcspkr snd_pcm
snd_timer sg snd soundcore i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper nfit syscopyarea sysfillrect libnvdimm sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix libata e1000 mptspi scsi_transport_spi mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 110743 Comm: kworker/u256:2 Kdump: loaded Not tainted 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8d8d4bf7e300 ti: ffff8d8d4bfdc000 task.ti: ffff8d8d4bfdc000
RIP: 0010:[<ffffffffb1d170de>]  [<ffffffffb1d170de>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8d8d4bfdf830  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000b1c7ddd0 RCX: ffff8d8dba420650
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8d8d4bfdf868 R08: ffff8d8db403c800 R09: ffffffffb1f86c09
R10: ffff8d8dba4df160 R11: ffffd2d681dd3800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8d8d4bfdf7e0 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8d8dba4c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa2e260d000 CR3: 0000000049e10000 CR4: 00000000000607e0
Call Trace:
[<ffffffffb1c7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb1c7e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffb1e08108>] ptep_clear_flush+0x68/0xa0
[<ffffffffb1e016a6>] page_mkclean_one+0xa6/0xe0
[<ffffffffb1e034b3>] rmap_walk+0xf3/0x360
[<ffffffffb1e038c6>] page_mkclean+0x76/0xb0
[<ffffffffb1e01600>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffb1e00ce0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb1dca6c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb1dca932>] write_cache_pages+0x1f2/0x470
[<ffffffffc046c290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffb1f55e80>] ? submit_bio+0x70/0x150
[<ffffffffb1ce8118>] ? update_group_power+0x28/0x280
[<ffffffffc046c25b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb1dcb9b1>] do_writepages+0x21/0x50
[<ffffffffb1e7df10>] __writeback_single_inode+0x40/0x260
[<ffffffffb1cc6f15>] ? wake_up_bit+0x25/0x30
[<ffffffffb1e7ea94>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffb1e7ed9f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb1e7f273>] wb_writeback+0x263/0x2f0
[<ffffffffb1e6ae6c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffb1e7fe6b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb1cbde8f>] process_one_work+0x17f/0x440
[<ffffffffb1cbefa6>] worker_thread+0x126/0x3c0
[<ffffffffb1cbee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb1cc5e61>] kthread+0xd1/0xe0
[<ffffffffb1cc5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffb2399df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb1cc5d90>] ? insert_kthread_work+0x40/0x40
Code: 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 95 b2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #324071
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffbe707e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffbe677018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffbe6771b4
Build id: 3.10.0-862.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffbe7d8837
Build id: 3.10.0-862.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffffbe7d1ba4
Build id: 3.10.0-862.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffffbe79f7d0
Build id: 3.10.0-862.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffffbe79fa7a
Build id: 3.10.0-862.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc0388850
Build id: 3.10.0-862.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffffbe91a940
Build id: 3.10.0-862.el7.x86_64
0x70
10
? update_group_power
vmlinux 0xffffffffbe6db898
Build id: 3.10.0-862.el7.x86_64
0x28
11
xfs_vm_writepages
xfs 0xffffffffc038865d
Build id: 3.10.0-862.el7.x86_64
0xbd
12
do_writepages
vmlinux 0xffffffffbe7a0e21
Build id: 3.10.0-862.el7.x86_64
0x21
13
__writeback_single_inode
vmlinux 0xffffffffbe848af0
Build id: 3.10.0-862.el7.x86_64
0x40
14
writeback_sb_inodes
vmlinux 0xffffffffbe849584
Build id: 3.10.0-862.el7.x86_64
0x1c4
15
__writeback_inodes_wb
vmlinux 0xffffffffbe8498ef
Build id: 3.10.0-862.el7.x86_64
0x9f
16
wb_writeback
vmlinux 0xffffffffbe84a123
Build id: 3.10.0-862.el7.x86_64
0x263
17
? get_nr_inodes
vmlinux 0xffffffffbe83665c
Build id: 3.10.0-862.el7.x86_64
0x4c
18
bdi_writeback_workfn
vmlinux 0xffffffffbe84aaab
Build id: 3.10.0-862.el7.x86_64
0x2cb
19
process_one_work
vmlinux 0xffffffffbe6b2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
20
worker_thread
vmlinux 0xffffffffbe6b3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
21
? manage_workers.isra.24
vmlinux 0xffffffffbe6b39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
22
kthread
vmlinux 0xffffffffbe6bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
23
? insert_kthread_work
vmlinux 0xffffffffbe6bad60
Build id: 3.10.0-862.el7.x86_64
0x40
24
ret_from_fork_nospec_begin
vmlinux 0xffffffffbed1f61d
Build id: 3.10.0-862.el7.x86_64
0x7
25
? insert_kthread_work
vmlinux 0xffffffffbe6bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u256:2:23039]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink bnep vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul snd_ens1371 snd_rawmidi btusb snd_ac97_codec btrtl btbcm btintel ac97_bus bluetooth glue_helper snd_seq snd_seq_device snd_pcm snd_timer ablk_helper snd rfkill cryptd soundcore sg pcspkr joydev ppdev nfit libnvdimm parport_pc parport shpchp vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic
pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi crct10dif_pclmul crct10dif_common crc32c_intel scsi_transport_spi libata mptscsih e1000 serio_raw mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 23039 Comm: kworker/u256:2 Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff946037899fa0 ti: ffff945ff1340000 task.ti: ffff945ff1340000
RIP: 0010:[<ffffffffbe707e3a>]  [<ffffffffbe707e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff945ff13438b8  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00000000000000fc RCX: ffff94603b71cd48
RDX: 0000000000000004 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff945ff13438f0 R08: ffff94603a6c3800 R09: ffffffffbe94a3d9
R10: ffff94603b69bb00 R11: ffffdf1f00269200 R12: 0000000000000296
R13: ffff945ff1343868 R14: ffff94601f7db718 R15: ffff94601f7db700
FS:  0000000000000000(0000) GS:ffff94603b680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002122038 CR3: 000000007700e000 CR4: 00000000003607e0
Call Trace:
[<ffffffffbe677018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbe6771b4>] flush_tlb_page+0x54/0xa0
[<ffffffffbe7d8837>] ptep_clear_flush+0x67/0xa0
[<ffffffffbe7d1ba4>] page_mkclean+0x144/0x210
[<ffffffffbe79f7d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffbe79fa7a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc0388850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffbe91a940>] ? submit_bio+0x70/0x150
[<ffffffffbe6db898>] ? update_group_power+0x28/0x280
[<ffffffffc038865d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffffbe7a0e21>] do_writepages+0x21/0x50
[<ffffffffbe848af0>] __writeback_single_inode+0x40/0x260
[<ffffffffbe849584>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffbe8498ef>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffbe84a123>] wb_writeback+0x263/0x2f0
[<ffffffffbe83665c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffbe84aaab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffbe6b2dff>] process_one_work+0x17f/0x440
[<ffffffffbe6b3ac6>] worker_thread+0x126/0x3c0
[<ffffffffbe6b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffbe6bae31>] kthread+0xd1/0xe0
[<ffffffffbe6bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffbed1f61d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffbe6bad60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 33 bf f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #98413
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8dd16a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff8dc7e478
Build id: 3.10.0-1062.9.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff8dc7e614
Build id: 3.10.0-1062.9.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff8de059d8
Build id: 3.10.0-1062.9.1.el7.x86_64
0x68
5
page_mkclean_one
vmlinux 0xffffffff8ddfef66
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
6
rmap_walk
vmlinux 0xffffffff8de00d93
Build id: 3.10.0-1062.9.1.el7.x86_64
0xf3
7
page_mkclean
vmlinux 0xffffffff8de011a6
Build id: 3.10.0-1062.9.1.el7.x86_64
0x76
8
? page_referenced_one
vmlinux 0xffffffff8ddfeec0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffff8ddfe5a0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffff8ddc98e0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x50
11
write_cache_pages
vmlinux 0xffffffff8ddc9b52
Build id: 3.10.0-1062.9.1.el7.x86_64
0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc03c9030
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa0
13
? submit_bio
vmlinux 0xffffffff8df50fd0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x70
14
? update_group_power
vmlinux 0xffffffff8dce83b4
Build id: 3.10.0-1062.9.1.el7.x86_64
0x174
15
? cpumask_next_and
vmlinux 0xffffffff8df81635
Build id: 3.10.0-1062.9.1.el7.x86_64
0x35
16
xfs_vm_writepages
xfs 0xffffffffc03c8ffb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x6b
17
do_writepages
vmlinux 0xffffffff8ddcabc1
Build id: 3.10.0-1062.9.1.el7.x86_64
0x21
18
__writeback_single_inode
vmlinux 0xffffffff8de79840
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
19
? wake_up_bit
vmlinux 0xffffffff8dcc72a5
Build id: 3.10.0-1062.9.1.el7.x86_64
0x25
20
writeback_sb_inodes
vmlinux 0xffffffff8de7a3d4
Build id: 3.10.0-1062.9.1.el7.x86_64
0x1c4
21
__writeback_inodes_wb
vmlinux 0xffffffff8de7a6df
Build id: 3.10.0-1062.9.1.el7.x86_64
0x9f
22
wb_writeback
vmlinux 0xffffffff8de7abc3
Build id: 3.10.0-1062.9.1.el7.x86_64
0x263
23
? get_nr_inodes
vmlinux 0xffffffff8de66a6c
Build id: 3.10.0-1062.9.1.el7.x86_64
0x4c
24
bdi_writeback_workfn
vmlinux 0xffffffff8de7b7bb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x2cb
25
process_one_work
vmlinux 0xffffffff8dcbe21f
Build id: 3.10.0-1062.9.1.el7.x86_64
0x17f
26
worker_thread
vmlinux 0xffffffff8dcbf336
Build id: 3.10.0-1062.9.1.el7.x86_64
0x126
27
? manage_workers.isra.26
vmlinux 0xffffffff8dcbf210
Build id: 3.10.0-1062.9.1.el7.x86_64
0x2a0
28
kthread
vmlinux 0xffffffff8dcc61f1
Build id: 3.10.0-1062.9.1.el7.x86_64
0xd1
29
? insert_kthread_work
vmlinux 0xffffffff8dcc6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffff8e38dd37
Build id: 3.10.0-1062.9.1.el7.x86_64
0x21
31
? insert_kthread_work
vmlinux 0xffffffff8dcc6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u256:2:9969]
Modules linked in: fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock vboxpci(OE) vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) sunrpc snd_seq_midi snd_seq_midi_event nls_utf8 isofs snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus squashfs snd_seq coretemp iosf_mbi snd_seq_device snd_pcm kvm_intel kvm loop snd_timer snd ppdev soundcore irqbypass crc32_pclmul pcspkr ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev parport_pc parport pcc_cpufreq vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod
cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm mptspi scsi_transport_spi e1000 mptscsih ata_piix nfit libnvdimm libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 9969 Comm: kworker/u256:2 Kdump: loaded Tainted: G           OE  ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8e931175d230 ti: ffff8e91cec14000 task.ti: ffff8e91cec14000
RIP: 0010:[<ffffffff8dd16a6a>]  [<ffffffff8dd16a6a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8e91cec17830  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008dc7e2b0 RCX: ffff8e933b420600
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8e91cec17868 R08: ffff8e92fd0cb400 R09: ffffffff8df817e9
R10: ffff8e933b45f160 R11: fffff21346be9a00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8e91cec177e0 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8e933b440000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8a58007000 CR3: 000000011fc10000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8dc7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8dc7e614>] flush_tlb_page+0x54/0xa0
[<ffffffff8de059d8>] ptep_clear_flush+0x68/0xa0
[<ffffffff8ddfef66>] page_mkclean_one+0xa6/0xe0
[<ffffffff8de00d93>] rmap_walk+0xf3/0x360
[<ffffffff8de011a6>] page_mkclean+0x76/0xb0
[<ffffffff8ddfeec0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff8ddfe5a0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff8ddc98e0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8ddc9b52>] write_cache_pages+0x1f2/0x470
[<ffffffffc03c9030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff8df50fd0>] ? submit_bio+0x70/0x150
[<ffffffff8dce83b4>] ? update_group_power+0x174/0x280
[<ffffffff8df81635>] ? cpumask_next_and+0x35/0x50
[<ffffffffc03c8ffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff8ddcabc1>] do_writepages+0x21/0x50
[<ffffffff8de79840>] __writeback_single_inode+0x40/0x260
[<ffffffff8dcc72a5>] ? wake_up_bit+0x25/0x30
[<ffffffff8de7a3d4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8de7a6df>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8de7abc3>] wb_writeback+0x263/0x2f0
[<ffffffff8de66a6c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8de7b7bb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff8dcbe21f>] process_one_work+0x17f/0x440
[<ffffffff8dcbf336>] worker_thread+0x126/0x3c0
[<ffffffff8dcbf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8dcc61f1>] kthread+0xd1/0xe0
[<ffffffff8dcc6120>] ? insert_kthread_work+0x40/0x40
[<ffffffff8e38dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8dcc6120>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 fe 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 94 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #94899
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb571173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb567a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffb567a734
Build id: 3.10.0-957.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffb57fd587
Build id: 3.10.0-957.el7.x86_64
0x67
5
page_mkclean_one
vmlinux 0xffffffffb57f6b66
Build id: 3.10.0-957.el7.x86_64
0xa6
6
rmap_walk
vmlinux 0xffffffffb57f8983
Build id: 3.10.0-957.el7.x86_64
0xf3
7
page_mkclean
vmlinux 0xffffffffb57f8d96
Build id: 3.10.0-957.el7.x86_64
0x76
8
? page_referenced_one
vmlinux 0xffffffffb57f6ac0
Build id: 3.10.0-957.el7.x86_64
0x180
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffb57f6200
Build id: 3.10.0-957.el7.x86_64
0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffb57c1d60
Build id: 3.10.0-957.el7.x86_64
0x50
11
write_cache_pages
vmlinux 0xffffffffb57c200a
Build id: 3.10.0-957.el7.x86_64
0x22a
12
? xfs_aops_discard_page
xfs 0xffffffffc04385b0
Build id: 3.10.0-957.el7.x86_64
0x150
13
? submit_bio
vmlinux 0xffffffffb5944f30
Build id: 3.10.0-957.el7.x86_64
0x70
14
? update_group_power
vmlinux 0xffffffffb56e3868
Build id: 3.10.0-957.el7.x86_64
0x28
15
? cpumask_next_and
vmlinux 0xffffffffb5975465
Build id: 3.10.0-957.el7.x86_64
0x35
16
xfs_vm_writepages
xfs 0xffffffffc04383ab
Build id: 3.10.0-957.el7.x86_64
0x6b
17
do_writepages
vmlinux 0xffffffffb57c3181
Build id: 3.10.0-957.el7.x86_64
0x21
18
__writeback_single_inode
vmlinux 0xffffffffb586f930
Build id: 3.10.0-957.el7.x86_64
0x40
19
writeback_sb_inodes
vmlinux 0xffffffffb58703c4
Build id: 3.10.0-957.el7.x86_64
0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffffb587072f
Build id: 3.10.0-957.el7.x86_64
0x9f
21
wb_writeback
vmlinux 0xffffffffb5870f63
Build id: 3.10.0-957.el7.x86_64
0x263
22
? get_nr_inodes
vmlinux 0xffffffffb585cd7c
Build id: 3.10.0-957.el7.x86_64
0x4c
23
bdi_writeback_workfn
vmlinux 0xffffffffb58718eb
Build id: 3.10.0-957.el7.x86_64
0x2cb
24
process_one_work
vmlinux 0xffffffffb56b9d4f
Build id: 3.10.0-957.el7.x86_64
0x17f
25
worker_thread
vmlinux 0xffffffffb56bade6
Build id: 3.10.0-957.el7.x86_64
0x126
26
? manage_workers.isra.25
vmlinux 0xffffffffb56bacc0
Build id: 3.10.0-957.el7.x86_64
0x2a0
27
kthread
vmlinux 0xffffffffb56c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
28
? insert_kthread_work
vmlinux 0xffffffffb56c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffffb5d74c37
Build id: 3.10.0-957.el7.x86_64
0x21
30
? insert_kthread_work
vmlinux 0xffffffffb56c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u8:2:10419]
Modules linked in: nfsv3 nfs_acl nfs lockd grace fscache ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter qxl ttm drm_kms_helper syscopyarea sysfillrect sysimgblt ppdev fb_sys_fops drm sg drm_panel_orientation_quirks pcspkr joydev virtio_balloon i2c_piix4 parport_pc parport auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi virtio_console
serio_raw floppy ata_piix e1000 libata virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 10419 Comm: kworker/u8:2 Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9291f420b0c0 ti: ffff92932c2a4000 task.ti: ffff92932c2a4000
RIP: 0010:[<ffffffffb571173a>]  [<ffffffffb571173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff92932c2a7820  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000b567a3d0 RCX: ffff92933fc1fb80
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff92932c2a7858 R08: ffff9292fd6d1000 R09: ffffffffb59755c9
R10: ffff92933fc9f160 R11: fffff8a701689700 R12: 0000000000000286
R13: 0000000000000286 R14: ffff92932c2a77d0 R15: ffff92932c2a7830
FS:  0000000000000000(0000) GS:ffff92933fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe7840ed5d0 CR3: 0000000159c10000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb567a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb567a734>] flush_tlb_page+0x54/0xa0
[<ffffffffb57fd587>] ptep_clear_flush+0x67/0xa0
[<ffffffffb57f6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffffb57f8983>] rmap_walk+0xf3/0x360
[<ffffffffb57f8d96>] page_mkclean+0x76/0xb0
[<ffffffffb57f6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffffb57f6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb57c1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb57c200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc04385b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffb5944f30>] ? submit_bio+0x70/0x150
[<ffffffffb56e3868>] ? update_group_power+0x28/0x280
[<ffffffffb5975465>] ? cpumask_next_and+0x35/0x50
[<ffffffffc04383ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb57c3181>] do_writepages+0x21/0x50
[<ffffffffb586f930>] __writeback_single_inode+0x40/0x260
[<ffffffffb58703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffb587072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb5870f63>] wb_writeback+0x263/0x2f0
[<ffffffffb585cd7c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffb58718eb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb56b9d4f>] process_one_work+0x17f/0x440
[<ffffffffb56bade6>] worker_thread+0x126/0x3c0
[<ffffffffb56bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffb56c1c31>] kthread+0xd1/0xe0
[<ffffffffb56c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb5d74c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb56c1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 34 b6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #70431
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff85d161da
Build id: 3.10.0-1062.4.3.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff85c7e478
Build id: 3.10.0-1062.4.3.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffff85c7e614
Build id: 3.10.0-1062.4.3.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffff85e05148
Build id: 3.10.0-1062.4.3.el7.x86_64
0x68
5
page_mkclean_one
vmlinux 0xffffffff85dfe6d6
Build id: 3.10.0-1062.4.3.el7.x86_64
0xa6
6
rmap_walk
vmlinux 0xffffffff85e00503
Build id: 3.10.0-1062.4.3.el7.x86_64
0xf3
7
page_mkclean
vmlinux 0xffffffff85e00916
Build id: 3.10.0-1062.4.3.el7.x86_64
0x76
8
? page_referenced_one
vmlinux 0xffffffff85dfe630
Build id: 3.10.0-1062.4.3.el7.x86_64
0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffff85dfdd10
Build id: 3.10.0-1062.4.3.el7.x86_64
0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffff85dc9050
Build id: 3.10.0-1062.4.3.el7.x86_64
0x50
11
write_cache_pages
vmlinux 0xffffffff85dc92c2
Build id: 3.10.0-1062.4.3.el7.x86_64
0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc03c5030
Build id: 3.10.0-1062.4.3.el7.x86_64
0xa0
13
? submit_bio
vmlinux 0xffffffff85f50750
Build id: 3.10.0-1062.4.3.el7.x86_64
0x70
14
? update_group_power
vmlinux 0xffffffff85ce8304
Build id: 3.10.0-1062.4.3.el7.x86_64
0x174
15
? cpumask_next_and
vmlinux 0xffffffff85f80d25
Build id: 3.10.0-1062.4.3.el7.x86_64
0x35
16
xfs_vm_writepages
xfs 0xffffffffc03c4ffb
Build id: 3.10.0-1062.4.3.el7.x86_64
0x6b
17
do_writepages
vmlinux 0xffffffff85dca331
Build id: 3.10.0-1062.4.3.el7.x86_64
0x21
18
__writeback_single_inode
vmlinux 0xffffffff85e78fb0
Build id: 3.10.0-1062.4.3.el7.x86_64
0x40
19
writeback_sb_inodes
vmlinux 0xffffffff85e79b44
Build id: 3.10.0-1062.4.3.el7.x86_64
0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffff85e79e4f
Build id: 3.10.0-1062.4.3.el7.x86_64
0x9f
21
wb_writeback
vmlinux 0xffffffff85e7a333
Build id: 3.10.0-1062.4.3.el7.x86_64
0x263
22
? get_nr_inodes
vmlinux 0xffffffff85e661dc
Build id: 3.10.0-1062.4.3.el7.x86_64
0x4c
23
bdi_writeback_workfn
vmlinux 0xffffffff85e7af2b
Build id: 3.10.0-1062.4.3.el7.x86_64
0x2cb
24
process_one_work
vmlinux 0xffffffff85cbe21f
Build id: 3.10.0-1062.4.3.el7.x86_64
0x17f
25
worker_thread
vmlinux 0xffffffff85cbf336
Build id: 3.10.0-1062.4.3.el7.x86_64
0x126
26
? manage_workers.isra.26
vmlinux 0xffffffff85cbf210
Build id: 3.10.0-1062.4.3.el7.x86_64
0x2a0
27
kthread
vmlinux 0xffffffff85cc61f1
Build id: 3.10.0-1062.4.3.el7.x86_64
0xd1
28
? insert_kthread_work
vmlinux 0xffffffff85cc6120
Build id: 3.10.0-1062.4.3.el7.x86_64
0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffff8638cd37
Build id: 3.10.0-1062.4.3.el7.x86_64
0x21
30
? insert_kthread_work
vmlinux 0xffffffff85cc6120
Build id: 3.10.0-1062.4.3.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 24s! [kworker/u256:2:57834]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel ppdev vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport pcc_cpufreq vmw_vmci i2c_piix4 sch_fq_codel ip_tables xfs libcrc32c sr_mod
cdrom sd_mod ata_generic crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit ata_piix libata e1000 crct10dif_pclmul libnvdimm crct10dif_common crc32c_intel mptspi scsi_transport_spi mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 12 PID: 57834 Comm: kworker/u256:2 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.4.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff98320f629070 ti: ffff9832303b4000 task.ti: ffff9832303b4000
RIP: 0010:[<ffffffff85d161da>]  [<ffffffff85d161da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9832303b7830  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 0008000085c7e2b0 RCX: ffff98323a6a0740
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9832303b7868 R08: ffff983206309400 R09: ffffffff85f80ed9
R10: ffff98323a91f160 R11: fffffaef42e99e00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9832303b77e0 R15: ffff9832303b7840
FS:  0000000000000000(0000) GS:ffff98323a900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f51f6f90000 CR3: 0000000019010000 CR4: 00000000001607e0
Call Trace:
[<ffffffff85c7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff85c7e614>] flush_tlb_page+0x54/0xa0
[<ffffffff85e05148>] ptep_clear_flush+0x68/0xa0
[<ffffffff85dfe6d6>] page_mkclean_one+0xa6/0xe0
[<ffffffff85e00503>] rmap_walk+0xf3/0x360
[<ffffffff85e00916>] page_mkclean+0x76/0xb0
[<ffffffff85dfe630>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff85dfdd10>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff85dc9050>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff85dc92c2>] write_cache_pages+0x1f2/0x470
[<ffffffffc03c5030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff85f50750>] ? submit_bio+0x70/0x150
[<ffffffff85ce8304>] ? update_group_power+0x174/0x280
[<ffffffff85f80d25>] ? cpumask_next_and+0x35/0x50
[<ffffffffc03c4ffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff85dca331>] do_writepages+0x21/0x50
[<ffffffff85e78fb0>] __writeback_single_inode+0x40/0x260
[<ffffffff85e79b44>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff85e79e4f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff85e7a333>] wb_writeback+0x263/0x2f0
[<ffffffff85e661dc>] ? get_nr_inodes+0x4c/0x70
[<ffffffff85e7af2b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff85cbe21f>] process_one_work+0x17f/0x440
[<ffffffff85cbf336>] worker_thread+0x126/0x3c0
[<ffffffff85cbf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff85cc61f1>] kthread+0xd1/0xe0
[<ffffffff85cc6120>] ? insert_kthread_work+0x40/0x40
[<ffffffff8638cd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff85cc6120>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 0e 57 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 d7 94 86 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 dd 56 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #17844
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa711173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa707a598
Build id: 3.10.0-957.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffa707a734
Build id: 3.10.0-957.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffa71fd587
Build id: 3.10.0-957.el7.x86_64
0x67
5
page_mkclean_one
vmlinux 0xffffffffa71f6b66
Build id: 3.10.0-957.el7.x86_64
0xa6
6
rmap_walk
vmlinux 0xffffffffa71f8983
Build id: 3.10.0-957.el7.x86_64
0xf3
7
page_mkclean
vmlinux 0xffffffffa71f8d96
Build id: 3.10.0-957.el7.x86_64
0x76
8
? page_referenced_one
vmlinux 0xffffffffa71f6ac0
Build id: 3.10.0-957.el7.x86_64
0x180
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffa71f6200
Build id: 3.10.0-957.el7.x86_64
0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffa71c1d60
Build id: 3.10.0-957.el7.x86_64
0x50
11
write_cache_pages
vmlinux 0xffffffffa71c200a
Build id: 3.10.0-957.el7.x86_64
0x22a
12
? xfs_aops_discard_page
xfs 0xffffffffc03a25b0
Build id: 3.10.0-957.el7.x86_64
0x150
13
? submit_bio
vmlinux 0xffffffffa7344f30
Build id: 3.10.0-957.el7.x86_64
0x70
14
? update_group_power
vmlinux 0xffffffffa70e3868
Build id: 3.10.0-957.el7.x86_64
0x28
15
? cpumask_next_and
vmlinux 0xffffffffa7375465
Build id: 3.10.0-957.el7.x86_64
0x35
16
xfs_vm_writepages
xfs 0xffffffffc03a23ab
Build id: 3.10.0-957.el7.x86_64
0x6b
17
do_writepages
vmlinux 0xffffffffa71c3181
Build id: 3.10.0-957.el7.x86_64
0x21
18
__writeback_single_inode
vmlinux 0xffffffffa726f930
Build id: 3.10.0-957.el7.x86_64
0x40
19
writeback_sb_inodes
vmlinux 0xffffffffa72703c4
Build id: 3.10.0-957.el7.x86_64
0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffffa727072f
Build id: 3.10.0-957.el7.x86_64
0x9f
21
wb_writeback
vmlinux 0xffffffffa7270f63
Build id: 3.10.0-957.el7.x86_64
0x263
22
? get_nr_inodes
vmlinux 0xffffffffa725cd7c
Build id: 3.10.0-957.el7.x86_64
0x4c
23
bdi_writeback_workfn
vmlinux 0xffffffffa72718eb
Build id: 3.10.0-957.el7.x86_64
0x2cb
24
process_one_work
vmlinux 0xffffffffa70b9d4f
Build id: 3.10.0-957.el7.x86_64
0x17f
25
worker_thread
vmlinux 0xffffffffa70bade6
Build id: 3.10.0-957.el7.x86_64
0x126
26
? manage_workers.isra.25
vmlinux 0xffffffffa70bacc0
Build id: 3.10.0-957.el7.x86_64
0x2a0
27
kthread
vmlinux 0xffffffffa70c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
28
? insert_kthread_work
vmlinux 0xffffffffa70c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffffa7774c37
Build id: 3.10.0-957.el7.x86_64
0x21
30
? insert_kthread_work
vmlinux 0xffffffffa70c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u256:1:6751]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmhgfs(OE) vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp snd_seq_midi snd_seq_midi_event iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 ppdev snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device vmw_balloon snd_pcm snd_timer snd soundcore joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi
vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm ata_piix e1000 libata nfit libnvdimm mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 6751 Comm: kworker/u256:1 Kdump: loaded Tainted: G           OEL ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8fc513488000 ti: ffff8fc5dfebc000 task.ti: ffff8fc5dfebc000
RIP: 0010:[<ffffffffa711173a>]  [<ffffffffa711173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8fc5dfebf820  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a707a3d0 RCX: ffff8fc5f96204b8
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8fc5dfebf858 R08: ffff8fc5f762c400 R09: ffffffffa73755c9
R10: ffff8fc5f969f160 R11: fffff16444c79600 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8fc5dfebf7d0 R15: 0000000180080008
FS:  0000000000000000(0000) GS:ffff8fc5f9680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f473d673420 CR3: 00000000b32fe000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa707a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa707a734>] flush_tlb_page+0x54/0xa0
[<ffffffffa71fd587>] ptep_clear_flush+0x67/0xa0
[<ffffffffa71f6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffffa71f8983>] rmap_walk+0xf3/0x360
[<ffffffffa71f8d96>] page_mkclean+0x76/0xb0
[<ffffffffa71f6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffffa71f6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffa71c1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa71c200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc03a25b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffa7344f30>] ? submit_bio+0x70/0x150
[<ffffffffa70e3868>] ? update_group_power+0x28/0x280
[<ffffffffa7375465>] ? cpumask_next_and+0x35/0x50
[<ffffffffc03a23ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa71c3181>] do_writepages+0x21/0x50
[<ffffffffa726f930>] __writeback_single_inode+0x40/0x260
[<ffffffffa72703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffa727072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa7270f63>] wb_writeback+0x263/0x2f0
[<ffffffffa725cd7c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffa72718eb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa70b9d4f>] process_one_work+0x17f/0x440
[<ffffffffa70bade6>] worker_thread+0x126/0x3c0
[<ffffffffa70bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffa70c1c31>] kthread+0xd1/0xe0
[<ffffffffa70c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa7774c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa70c1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 d4 a7 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #12574
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa4307e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa4277018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffa42771b4
Build id: 3.10.0-862.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffa43d8837
Build id: 3.10.0-862.el7.x86_64
0x67
5
page_mkclean
vmlinux 0xffffffffa43d1ba4
Build id: 3.10.0-862.el7.x86_64
0x144
6
clear_page_dirty_for_io
vmlinux 0xffffffffa439f7d0
Build id: 3.10.0-862.el7.x86_64
0x50
7
write_cache_pages
vmlinux 0xffffffffa439fa7a
Build id: 3.10.0-862.el7.x86_64
0x22a
8
? xfs_aops_discard_page
xfs 0xffffffffc0436850
Build id: 3.10.0-862.el7.x86_64
0x150
9
? submit_bio
vmlinux 0xffffffffa451a940
Build id: 3.10.0-862.el7.x86_64
0x70
10
? update_group_power
vmlinux 0xffffffffa42db898
Build id: 3.10.0-862.el7.x86_64
0x28
11
? cpumask_next_and
vmlinux 0xffffffffa454a275
Build id: 3.10.0-862.el7.x86_64
0x35
12
xfs_vm_writepages
xfs 0xffffffffc043665d
Build id: 3.10.0-862.el7.x86_64
0xbd
13
do_writepages
vmlinux 0xffffffffa43a0e21
Build id: 3.10.0-862.el7.x86_64
0x21
14
__writeback_single_inode
vmlinux 0xffffffffa4448af0
Build id: 3.10.0-862.el7.x86_64
0x40
15
writeback_sb_inodes
vmlinux 0xffffffffa4449584
Build id: 3.10.0-862.el7.x86_64
0x1c4
16
__writeback_inodes_wb
vmlinux 0xffffffffa44498ef
Build id: 3.10.0-862.el7.x86_64
0x9f
17
wb_writeback
vmlinux 0xffffffffa444a123
Build id: 3.10.0-862.el7.x86_64
0x263
18
? get_nr_inodes
vmlinux 0xffffffffa443665c
Build id: 3.10.0-862.el7.x86_64
0x4c
19
bdi_writeback_workfn
vmlinux 0xffffffffa444aaab
Build id: 3.10.0-862.el7.x86_64
0x2cb
20
process_one_work
vmlinux 0xffffffffa42b2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
21
worker_thread
vmlinux 0xffffffffa42b3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
22
? manage_workers.isra.24
vmlinux 0xffffffffa42b39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
23
kthread
vmlinux 0xffffffffa42bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
24
? insert_kthread_work
vmlinux 0xffffffffa42bad60
Build id: 3.10.0-862.el7.x86_64
0x40
25
ret_from_fork_nospec_begin
vmlinux 0xffffffffa491f637
Build id: 3.10.0-862.el7.x86_64
0x21
26
? insert_kthread_work
vmlinux 0xffffffffa42bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 54s! [kworker/u64:3:2417]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 xt_addrtype nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat xt_conntrack nf_conntrack br_netfilter ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd vmw_balloon joydev pcspkr sg parport_pc floppy parport vmw_vmci shpchp i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom vmwgfx ata_generic drm_kms_helper syscopyarea sysfillrect sysimgblt pata_acpi fb_sys_fops ttm mptspi scsi_transport_spi mptscsih drm ata_piix crct10dif_pclmul
crct10dif_common libata e1000 crc32c_intel mptbase serio_raw i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 2417 Comm: kworker/u64:3 Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff986b067d8fd0 ti: ffff9869c36a8000 task.ti: ffff9869c36a8000
RIP: 0010:[<ffffffffa4307e3a>]  [<ffffffffa4307e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9869c36ab8b8  EFLAGS: 00000202
RAX: 0000000000000005 RBX: 0000000000019680 RCX: ffff986b6ef5cdb8
RDX: 0000000000000005 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff9869c36ab8f0 R08: ffff9868ba383800 R09: ffffffffa454a3d9
R10: ffff986b6eedbb00 R11: fffff5134a0fc200 R12: ffff9869c36ab868
R13: ffffffffa4574a23 R14: ffff9869c36ab858 R15: ffff986a126e8000
FS:  0000000000000000(0000) GS:ffff986b6eec0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fe5b3e31000 CR3: 00000002dbbb0000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa4277018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa42771b4>] flush_tlb_page+0x54/0xa0
[<ffffffffa43d8837>] ptep_clear_flush+0x67/0xa0
[<ffffffffa43d1ba4>] page_mkclean+0x144/0x210
[<ffffffffa439f7d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa439fa7a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc0436850>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffa451a940>] ? submit_bio+0x70/0x150
[<ffffffffa42db898>] ? update_group_power+0x28/0x280
[<ffffffffa454a275>] ? cpumask_next_and+0x35/0x50
[<ffffffffc043665d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffffa43a0e21>] do_writepages+0x21/0x50
[<ffffffffa4448af0>] __writeback_single_inode+0x40/0x260
[<ffffffffa4449584>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffa44498ef>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa444a123>] wb_writeback+0x263/0x2f0
[<ffffffffa443665c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffa444aaab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa42b2dff>] process_one_work+0x17f/0x440
[<ffffffffa42b3ac6>] worker_thread+0x126/0x3c0
[<ffffffffa42b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffa42bae31>] kthread+0xd1/0xe0
[<ffffffffa42bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa491f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa42bad60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 f3 a4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #9406
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffaef11a8a
Build id: 3.10.0-957.21.3.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaee7a6a8
Build id: 3.10.0-957.21.3.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffaee7a844
Build id: 3.10.0-957.21.3.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffaeffdb97
Build id: 3.10.0-957.21.3.el7.x86_64
0x67
5
page_mkclean_one
vmlinux 0xffffffffaeff7176
Build id: 3.10.0-957.21.3.el7.x86_64
0xa6
6
rmap_walk
vmlinux 0xffffffffaeff8f93
Build id: 3.10.0-957.21.3.el7.x86_64
0xf3
7
page_mkclean
vmlinux 0xffffffffaeff93a6
Build id: 3.10.0-957.21.3.el7.x86_64
0x76
8
? page_referenced_one
vmlinux 0xffffffffaeff70d0
Build id: 3.10.0-957.21.3.el7.x86_64
0x180
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffaeff6810
Build id: 3.10.0-957.21.3.el7.x86_64
0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffaefc2320
Build id: 3.10.0-957.21.3.el7.x86_64
0x50
11
write_cache_pages
vmlinux 0xffffffffaefc2592
Build id: 3.10.0-957.21.3.el7.x86_64
0x1f2
12
? xfs_aops_discard_page
xfs 0xffffffffc03737d0
Build id: 3.10.0-957.21.3.el7.x86_64
0x150
13
? submit_bio
vmlinux 0xffffffffaf145790
Build id: 3.10.0-957.21.3.el7.x86_64
0x70
14
? update_group_power
vmlinux 0xffffffffaeee3cd4
Build id: 3.10.0-957.21.3.el7.x86_64
0x174
15
xfs_vm_writepages
xfs 0xffffffffc03735cb
Build id: 3.10.0-957.21.3.el7.x86_64
0x6b
16
do_writepages
vmlinux 0xffffffffaefc36d1
Build id: 3.10.0-957.21.3.el7.x86_64
0x21
17
__writeback_single_inode
vmlinux 0xffffffffaf0701b0
Build id: 3.10.0-957.21.3.el7.x86_64
0x40
18
writeback_sb_inodes
vmlinux 0xffffffffaf070c44
Build id: 3.10.0-957.21.3.el7.x86_64
0x1c4
19
__writeback_inodes_wb
vmlinux 0xffffffffaf070faf
Build id: 3.10.0-957.21.3.el7.x86_64
0x9f
20
wb_writeback
vmlinux 0xffffffffaf0717e3
Build id: 3.10.0-957.21.3.el7.x86_64
0x263
21
? get_nr_inodes
vmlinux 0xffffffffaf05d5fc
Build id: 3.10.0-957.21.3.el7.x86_64
0x4c
22
bdi_writeback_workfn
vmlinux 0xffffffffaf07216b
Build id: 3.10.0-957.21.3.el7.x86_64
0x2cb
23
process_one_work
vmlinux 0xffffffffaeeb9ebf
Build id: 3.10.0-957.21.3.el7.x86_64
0x17f
24
worker_thread
vmlinux 0xffffffffaeebaf56
Build id: 3.10.0-957.21.3.el7.x86_64
0x126
25
? manage_workers.isra.25
vmlinux 0xffffffffaeebae30
Build id: 3.10.0-957.21.3.el7.x86_64
0x2a0
26
kthread
vmlinux 0xffffffffaeec1da1
Build id: 3.10.0-957.21.3.el7.x86_64
0xd1
27
? insert_kthread_work
vmlinux 0xffffffffaeec1cd0
Build id: 3.10.0-957.21.3.el7.x86_64
0x40
28
ret_from_fork_nospec_begin
vmlinux 0xffffffffaf575c37
Build id: 3.10.0-957.21.3.el7.x86_64
0x21
29
? insert_kthread_work
vmlinux 0xffffffffaeec1cd0
Build id: 3.10.0-957.21.3.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 22s! [kworker/u510:1:237332]
<EOI> 
Modules linked in:
veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev sg cryptd virtio_balloon i2c_piix4 joydev pcspkr parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul drm crct10dif_common libata virtio_pci virtio_ring crc32c_intel drm_panel_orientation_quirks
serio_raw virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 15 PID: 237332 Comm: kworker/u510:1 Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.21.3.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-20180921_191058-build10b184b164b178 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
task: ffff9bd628368000 ti: ffff9bcf64c8c000 task.ti: ffff9bcf64c8c000
RIP: 0010:[<ffffffffaef11a8a>]  [<ffffffffaef11a8a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9bcf64c8f830  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000aee7a4e0 RCX: ffff9bd71b6207e0
RDX: 0000000000000000 RSI: 00000000000000ff RDI: 0000000000000000
RBP: ffff9bcf64c8f868 R08: ffff9bd0745bd800 R09: ffffffffaf175e39
R10: ffff9bd71b9df160 R11: ffffdeebd4a8bc00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9bcf64c8f7e0 R15: 0000000000000001
FS:  0000000000000000(0000) GS:ffff9bd71b9c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f100a9f5000 CR3: 00000007f96f2000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaee7a6a8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaee7a844>] flush_tlb_page+0x54/0xa0
[<ffffffffaeffdb97>] ptep_clear_flush+0x67/0xa0
[<ffffffffaeff7176>] page_mkclean_one+0xa6/0xe0
[<ffffffffaeff8f93>] rmap_walk+0xf3/0x360
[<ffffffffaeff93a6>] page_mkclean+0x76/0xb0
[<ffffffffaeff70d0>] ? page_referenced_one+0x180/0x180
[<ffffffffaeff6810>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffaefc2320>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffaefc2592>] write_cache_pages+0x1f2/0x470
[<ffffffffc03737d0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffaf145790>] ? submit_bio+0x70/0x150
[<ffffffffaeee3cd4>] ? update_group_power+0x174/0x280
[<ffffffffc03735cb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffaefc36d1>] do_writepages+0x21/0x50
[<ffffffffaf0701b0>] __writeback_single_inode+0x40/0x260
[<ffffffffaf070c44>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffaf070faf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffaf0717e3>] wb_writeback+0x263/0x2f0
[<ffffffffaf05d5fc>] ? get_nr_inodes+0x4c/0x70
[<ffffffffaf07216b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffaeeb9ebf>] process_one_work+0x17f/0x440
[<ffffffffaeebaf56>] worker_thread+0x126/0x3c0
[<ffffffffaeebae30>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffaeec1da1>] kthread+0xd1/0xe0
[<ffffffffaeec1cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffffaf575c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffaeec1cd0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 3e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 bc b4 af f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1437
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb2716a6a
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb267e478
Build id: 3.10.0-1062.9.1.el7.x86_64
0xb8
3
flush_tlb_page
vmlinux 0xffffffffb267e614
Build id: 3.10.0-1062.9.1.el7.x86_64
0x54
4
ptep_clear_flush
vmlinux 0xffffffffb28059d8
Build id: 3.10.0-1062.9.1.el7.x86_64
0x68
5
page_mkclean_one
vmlinux 0xffffffffb27fef66
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa6
6
rmap_walk
vmlinux 0xffffffffb2800d93
Build id: 3.10.0-1062.9.1.el7.x86_64
0xf3
7
page_mkclean
vmlinux 0xffffffffb28011a6
Build id: 3.10.0-1062.9.1.el7.x86_64
0x76
8
? page_referenced_one
vmlinux 0xffffffffb27feec0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffb27fe5a0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffb27c98e0
Build id: 3.10.0-1062.9.1.el7.x86_64
0x50
11
write_cache_pages
vmlinux 0xffffffffb27c9b52
Build id: 3.10.0-1062.9.1.el7.x86_64
0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc040b030
Build id: 3.10.0-1062.9.1.el7.x86_64
0xa0
13
? set_next_entity
vmlinux 0xffffffffb26e29ec
Build id: 3.10.0-1062.9.1.el7.x86_64
0x3c
14
? update_group_power
vmlinux 0xffffffffb26e83b4
Build id: 3.10.0-1062.9.1.el7.x86_64
0x174
15
? cpumask_next_and
vmlinux 0xffffffffb2981635
Build id: 3.10.0-1062.9.1.el7.x86_64
0x35
16
? find_busiest_group
vmlinux 0xffffffffb26e8604
Build id: 3.10.0-1062.9.1.el7.x86_64
0x144
17
xfs_vm_writepages
xfs 0xffffffffc040affb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x6b
18
do_writepages
vmlinux 0xffffffffb27cabc1
Build id: 3.10.0-1062.9.1.el7.x86_64
0x21
19
__writeback_single_inode
vmlinux 0xffffffffb2879840
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
20
? check_preempt_wakeup
vmlinux 0xffffffffb26e5ac1
Build id: 3.10.0-1062.9.1.el7.x86_64
0x101
21
writeback_sb_inodes
vmlinux 0xffffffffb287a3d4
Build id: 3.10.0-1062.9.1.el7.x86_64
0x1c4
22
__writeback_inodes_wb
vmlinux 0xffffffffb287a6df
Build id: 3.10.0-1062.9.1.el7.x86_64
0x9f
23
wb_writeback
vmlinux 0xffffffffb287abc3
Build id: 3.10.0-1062.9.1.el7.x86_64
0x263
24
? get_nr_inodes
vmlinux 0xffffffffb2866a6c
Build id: 3.10.0-1062.9.1.el7.x86_64
0x4c
25
bdi_writeback_workfn
vmlinux 0xffffffffb287b7bb
Build id: 3.10.0-1062.9.1.el7.x86_64
0x2cb
26
process_one_work
vmlinux 0xffffffffb26be21f
Build id: 3.10.0-1062.9.1.el7.x86_64
0x17f
27
worker_thread
vmlinux 0xffffffffb26bf336
Build id: 3.10.0-1062.9.1.el7.x86_64
0x126
28
? manage_workers.isra.26
vmlinux 0xffffffffb26bf210
Build id: 3.10.0-1062.9.1.el7.x86_64
0x2a0
29
kthread
vmlinux 0xffffffffb26c61f1
Build id: 3.10.0-1062.9.1.el7.x86_64
0xd1
30
? insert_kthread_work
vmlinux 0xffffffffb26c6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
31
ret_from_fork_nospec_begin
vmlinux 0xffffffffb2d8dd1d
Build id: 3.10.0-1062.9.1.el7.x86_64
0x7
32
? insert_kthread_work
vmlinux 0xffffffffb26c6120
Build id: 3.10.0-1062.9.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u20:3:4176]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter binfmt_misc sunrpc nfit libnvdimm ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper sg ablk_helper cryptd joydev pcspkr virtio_balloon parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom cirrus ata_generic drm_kms_helper pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm virtio_console virtio_blk ata_piix crct10dif_pclmul drm 8139too libata crct10dif_common crc32c_intel virtio_pci virtio_ring serio_raw 8139cp floppy drm_panel_orientation_quirks
mii virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 4176 Comm: kworker/u20:3 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8b31178e8000 ti: ffff8b31cb01c000 task.ti: ffff8b31cb01c000
RIP: 0010:[<ffffffffb2716a6a>]  [<ffffffffb2716a6a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8b31cb01f830  EFLAGS: 00000202
RAX: 0000000000000008 RBX: 000000000000000a RCX: ffff8b323a420398
RDX: 0000000000000008 RSI: 000000000000000a RDI: 0000000000000000
RBP: ffff8b31cb01f868 R08: ffff8b3237aba800 R09: ffffffffb29817e9
R10: ffff8b323a21f160 R11: ffffd6dcc189a000 R12: 000000000001b800
R13: ffffffffb267e2b0 R14: ffff8b31cb01f878 R15: ffff8b323a21b840
FS:  0000000000000000(0000) GS:ffff8b323a200000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000e055a004 CR3: 0000000148410000 CR4: 00000000007606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffffb267e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb267e614>] flush_tlb_page+0x54/0xa0
[<ffffffffb28059d8>] ptep_clear_flush+0x68/0xa0
[<ffffffffb27fef66>] page_mkclean_one+0xa6/0xe0
[<ffffffffb2800d93>] rmap_walk+0xf3/0x360
[<ffffffffb28011a6>] page_mkclean+0x76/0xb0
[<ffffffffb27feec0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffb27fe5a0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb27c98e0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb27c9b52>] write_cache_pages+0x1f2/0x470
[<ffffffffc040b030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffb26e29ec>] ? set_next_entity+0x3c/0xe0
[<ffffffffb26e83b4>] ? update_group_power+0x174/0x280
[<ffffffffb2981635>] ? cpumask_next_and+0x35/0x50
[<ffffffffb26e8604>] ? find_busiest_group+0x144/0x990
[<ffffffffc040affb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb27cabc1>] do_writepages+0x21/0x50
[<ffffffffb2879840>] __writeback_single_inode+0x40/0x260
[<ffffffffb26e5ac1>] ? check_preempt_wakeup+0x101/0x220
[<ffffffffb287a3d4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffb287a6df>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb287abc3>] wb_writeback+0x263/0x2f0
[<ffffffffb2866a6c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffb287b7bb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb26be21f>] process_one_work+0x17f/0x440
[<ffffffffb26bf336>] worker_thread+0x126/0x3c0
[<ffffffffb26bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb26c61f1>] kthread+0xd1/0xe0
[<ffffffffb26c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffffb2d8dd1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb26c6120>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 fe 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 34 b3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89