Info

Permalink
Function
smp_call_function_many
First occurrence
2020-10-05
Last occurrence
2026-01-18
State
NEW
Components
kernel
Quality
-64

Statistics

Operating system Count
CentOS 7 1,145
Architecture Count
x86_64 1,145
Related packages Count
kernel 1,145
 0:3.10.0-1160.76.1.el7 530
 0:3.10.0-1062.el7 210
 0:3.10.0-1160.66.1.el7 67
 0:3.10.0-1160.36.2.el7 57
 0:3.10.0-1160.45.1.el7 41
 0:3.10.0-1160.el7 32
 0:3.10.0-1062.12.1.el7 23
 0:3.10.0-1160.108.1.el7 19
 0:3.10.0-1160.2.2.el7 18
 0:3.10.0-1127.19.1.el7 16
 0:3.10.0-1062.9.1.el7 12
 0:3.10.0-1160.88.1.el7 11
 0:3.10.0-1160.102.1.el7 10
 0:3.10.0-1160.119.1.el7 10
 0:3.10.0-1160.15.2.el7 8
 0:3.10.0-1160.25.1.el7 8
 0:3.10.0-1160.6.1.el7 7
 0:3.10.0-1160.81.1.el7 7
 0:3.10.0-1062.4.3.el7 6
 0:3.10.0-1127.el7 6
 0:3.10.0-1160.71.1.el7 6
 0:3.10.0-1160.59.1.el7 5
 0:3.10.0-1160.62.1.el7 5
 0:3.10.0-1127.18.2.el7 3
 0:3.10.0-1160.11.1.el7 3
 0:3.10.0-1160.24.1.el7 3
 0:3.10.0-1160.53.1.el7 3
 0:3.10.0-1062.1.1.el7 2
 0:3.10.0-1062.18.1.el7 2
 0:3.10.0-1160.118.1.el7 2
 0:3.10.0-1160.90.1.el7 2
 0:3.10.0-1160.92.1.el7 2
 0:3.10.0-1160.99.1.el7 2
 0:3.10.0-1127.13.1.el7 2
 0:3.10.0-957.el7 2
 0:3.10.0-1160.42.2.el7 1
 0:3.10.0-1160.49.1.el7 1
 0:3.10.0-1160.83.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1244408
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa2f1173a
Build id: 3.10.0-957.el7.x86_64 		0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa2e7a598
Build id: 3.10.0-957.el7.x86_64 		0xb8
3
flush_tlb_page
vmlinux 0xffffffffa2e7a734
Build id: 3.10.0-957.el7.x86_64 		0x54
4
ptep_clear_flush
vmlinux 0xffffffffa2ffd587
Build id: 3.10.0-957.el7.x86_64 		0x67
5
page_mkclean_one
vmlinux 0xffffffffa2ff6b66
Build id: 3.10.0-957.el7.x86_64 		0xa6
6
rmap_walk
vmlinux 0xffffffffa2ff8983
Build id: 3.10.0-957.el7.x86_64 		0xf3
7
page_mkclean
vmlinux 0xffffffffa2ff8d96
Build id: 3.10.0-957.el7.x86_64 		0x76
8
? page_referenced_one
vmlinux 0xffffffffa2ff6ac0
Build id: 3.10.0-957.el7.x86_64 		0x180
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffa2ff6200
Build id: 3.10.0-957.el7.x86_64 		0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffa2fc1d60
Build id: 3.10.0-957.el7.x86_64 		0x50
11
write_cache_pages
vmlinux 0xffffffffa2fc200a
Build id: 3.10.0-957.el7.x86_64 		0x22a
12
? xfs_aops_discard_page
xfs 0xffffffffc053a5b0
Build id: 3.10.0-957.el7.x86_64 		0x150
13
? submit_bio
vmlinux 0xffffffffa3144f30
Build id: 3.10.0-957.el7.x86_64 		0x70
14
? update_curr
vmlinux 0xffffffffa2ee00bc
Build id: 3.10.0-957.el7.x86_64 		0x14c
15
? update_group_power
vmlinux 0xffffffffa2ee39b4
Build id: 3.10.0-957.el7.x86_64 		0x174
16
xfs_vm_writepages
xfs 0xffffffffc053a3ab
Build id: 3.10.0-957.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffffa2fc3181
Build id: 3.10.0-957.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffffa306f930
Build id: 3.10.0-957.el7.x86_64 		0x40
19
writeback_sb_inodes
vmlinux 0xffffffffa30703c4
Build id: 3.10.0-957.el7.x86_64 		0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffffa307072f
Build id: 3.10.0-957.el7.x86_64 		0x9f
21
wb_writeback
vmlinux 0xffffffffa3070f63
Build id: 3.10.0-957.el7.x86_64 		0x263
22
? bdi_dirty_limit
vmlinux 0xffffffffa2fc2830
Build id: 3.10.0-957.el7.x86_64 		0x40
23
bdi_writeback_workfn
vmlinux 0xffffffffa30717ec
Build id: 3.10.0-957.el7.x86_64 		0x1cc
24
process_one_work
vmlinux 0xffffffffa2eb9d4f
Build id: 3.10.0-957.el7.x86_64 		0x17f
25
worker_thread
vmlinux 0xffffffffa2ebade6
Build id: 3.10.0-957.el7.x86_64 		0x126
26
? manage_workers.isra.25
vmlinux 0xffffffffa2ebacc0
Build id: 3.10.0-957.el7.x86_64 		0x2a0
27
kthread
vmlinux 0xffffffffa2ec1c31
Build id: 3.10.0-957.el7.x86_64 		0xd1
28
? insert_kthread_work
vmlinux 0xffffffffa2ec1b60
Build id: 3.10.0-957.el7.x86_64 		0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffffa3574c37
Build id: 3.10.0-957.el7.x86_64 		0x21
30
? insert_kthread_work
vmlinux 0xffffffffa2ec1b60
Build id: 3.10.0-957.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u96:0:5751]
Modules linked in: ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipportip ip_set_hash_ipport xt_set xt_addrtype xt_multiport ip_set_hash_ip ip_set_hash_net ip_set ipip tunnel4 ip_tunnel veth rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache dummy iptable_raw xt_CT ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter sunrpc ppdev sg virtio_balloon joydev parport_pc parport i2c_piix4 pcspkr br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack
ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_blk virtio_console virtio_scsi virtio_net cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm floppy ata_piix libata serio_raw virtio_pci drm_panel_orientation_quirks virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 5751 Comm: kworker/u96:0 Kdump: loaded Tainted: G             L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
task: ffff9668b8bd8000 ti: ffff967214a5c000 task.ti: ffff967214a5c000
RIP: 0010:[<ffffffffa2f1173a>]  [<ffffffffa2f1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff967214a5f820  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a2e7a3d0 RCX: ffff96723ea1fb80
RDX: 0000000000000000 RSI: 0000000000000030 RDI: 0000000000000000
RBP: ffff967214a5f858 R08: ffff9663a9523400 R09: ffffffffa31755c9
R10: ffff96723ea5f160 R11: ffffd949fe2adc00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff967214a5f7d0 R15: ffff9663bfc03300
FS:  0000000000000000(0000) GS:ffff96723ea40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb49d7aa020 CR3: 000000015a810000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa2e7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa2e7a734>] flush_tlb_page+0x54/0xa0
[<ffffffffa2ffd587>] ptep_clear_flush+0x67/0xa0
[<ffffffffa2ff6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffffa2ff8983>] rmap_walk+0xf3/0x360
[<ffffffffa2ff8d96>] page_mkclean+0x76/0xb0
[<ffffffffa2ff6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffffa2ff6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffa2fc1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa2fc200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc053a5b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffa3144f30>] ? submit_bio+0x70/0x150
[<ffffffffa2ee00bc>] ? update_curr+0x14c/0x1e0
[<ffffffffa2ee39b4>] ? update_group_power+0x174/0x280
[<ffffffffc053a3ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa2fc3181>] do_writepages+0x21/0x50
[<ffffffffa306f930>] __writeback_single_inode+0x40/0x260
[<ffffffffa30703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffa307072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa3070f63>] wb_writeback+0x263/0x2f0
[<ffffffffa2fc2830>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffffa30717ec>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffffa2eb9d4f>] process_one_work+0x17f/0x440
[<ffffffffa2ebade6>] worker_thread+0x126/0x3c0
[<ffffffffa2ebacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffa2ec1c31>] kthread+0xd1/0xe0
[<ffffffffa2ec1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa3574c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa2ec1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 b4 a3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #532708
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffabf17228
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x208
2
native_flush_tlb_others
vmlinux 0xffffffffabe7e258
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xb8
3
flush_tlb_page
vmlinux 0xffffffffabe7e3f4
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x54
4
ptep_clear_flush
vmlinux 0xffffffffac0080f8
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x68
5
page_mkclean_one
vmlinux 0xffffffffac001696
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa6
6
rmap_walk
vmlinux 0xffffffffac0034a3
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xf3
7
page_mkclean
vmlinux 0xffffffffac0038b6
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x76
8
? page_referenced_one
vmlinux 0xffffffffac0015f0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffac000cd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffabfca6d0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x50
11
write_cache_pages
vmlinux 0xffffffffabfca942
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc0321290
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa0
13
? __enqueue_entity
vmlinux 0xffffffffabedfd28
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x78
14
? put_prev_entity
vmlinux 0xffffffffabee64cb
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x8b
15
? pick_next_entity
vmlinux 0xffffffffabee2429
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xa9
16
? set_next_entity
vmlinux 0xffffffffabee283c
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x3c
17
xfs_vm_writepages
xfs 0xffffffffc032125b
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x6b
18
do_writepages
vmlinux 0xffffffffabfcb9c1
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x21
19
__writeback_single_inode
vmlinux 0xffffffffac07df30
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x40
20
writeback_sb_inodes
vmlinux 0xffffffffac07eab4
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x1c4
21
__writeback_inodes_wb
vmlinux 0xffffffffac07edbf
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x9f
22
wb_writeback
vmlinux 0xffffffffac07f293
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x263
23
? bdi_dirty_limit
vmlinux 0xffffffffabfcb130
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x40
24
bdi_writeback_workfn
vmlinux 0xffffffffac07fd8c
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x1cc
25
process_one_work
vmlinux 0xffffffffabebdfdf
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x17f
26
worker_thread
vmlinux 0xffffffffabebf0f6
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x126
27
? manage_workers.isra.26
vmlinux 0xffffffffabebefd0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x2a0
28
kthread
vmlinux 0xffffffffabec5fb1
Build id: 3.10.0-1160.62.1.el7.x86_64 		0xd1
29
? insert_kthread_work
vmlinux 0xffffffffabec5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffffac599df7
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x21
31
? insert_kthread_work
vmlinux 0xffffffffabec5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [kworker/u16:2:29583]
Modules linked in: tcp_diag udp_diag inet_diag tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nfsv3 nf_conntrack nfs_acl ip6table_filter ip6_tables rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_set ip_set iptable_filter sunrpc snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer ppdev snd parport_pc soundcore sg parport pcspkr joydev i2c_piix4 virtio_balloon ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm drm ata_piix libata virtio_net net_failover virtio_blk failover drm_panel_orientation_quirks floppy virtio_pci serio_raw virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 5 PID: 29583 Comm: kworker/u16:2 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9a9e261f4200 ti: ffff9a95d0284000 task.ti: ffff9a95d0284000
RIP: 0010:[<ffffffffabf17228>]  [<ffffffffabf17228>] smp_call_function_many+0x208/0x270
RSP: 0018:ffff9a95d0287830  EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffffffffabe7ddd0 RCX: ffff9a9fcb8a04a0
RDX: 0000000000000002 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff9a95d0287868 R08: ffff9a8f4fd49800 R09: ffffffffac1869d9
R10: ffff9a9fcb95f160 R11: ffffea3462e25a00 R12: ffff9a95d0287800
R13: ffffffffac18692f R14: 0000000000000000 R15: ffff9a9f7e8d6b04
FS:  0000000000000000(0000) GS:ffff9a9fcb940000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1064627500 CR3: 0000000959e10000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffabe7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffabe7e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffac0080f8>] ptep_clear_flush+0x68/0xa0
[<ffffffffac001696>] page_mkclean_one+0xa6/0xe0
[<ffffffffac0034a3>] rmap_walk+0xf3/0x360
[<ffffffffac0038b6>] page_mkclean+0x76/0xb0
[<ffffffffac0015f0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffac000cd0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffabfca6d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffabfca942>] write_cache_pages+0x1f2/0x470
[<ffffffffc0321290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffabedfd28>] ? __enqueue_entity+0x78/0x80
[<ffffffffabee64cb>] ? put_prev_entity+0x8b/0x400
[<ffffffffabee2429>] ? pick_next_entity+0xa9/0x190
[<ffffffffabee283c>] ? set_next_entity+0x3c/0xe0
[<ffffffffc032125b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffabfcb9c1>] do_writepages+0x21/0x50
[<ffffffffac07df30>] __writeback_single_inode+0x40/0x260
[<ffffffffac07eab4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffac07edbf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffac07f293>] wb_writeback+0x263/0x2f0
[<ffffffffabfcb130>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffffac07fd8c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffffabebdfdf>] process_one_work+0x17f/0x440
[<ffffffffabebf0f6>] worker_thread+0x126/0x3c0
[<ffffffffabebefd0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffabec5fb1>] kthread+0xd1/0xe0
[<ffffffffabec5ee0>] ? insert_kthread_work+0x40/0x40
[<ffffffffac599df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffabec5ee0>] ? insert_kthread_work+0x40/0x40
Code: 28 00 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 b5 ac f6 41 20 01 74 cd 0f 1f 44 00 00 <f3> 90 f6 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc
Complete report #1172777
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffaef16c0a
Build id: 3.10.0-1160.el7.x86_64 		0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffaee7e868
Build id: 3.10.0-1160.el7.x86_64 		0xb8
3
flush_tlb_page
vmlinux 0xffffffffaee7ea04
Build id: 3.10.0-1160.el7.x86_64 		0x54
4
ptep_clear_flush
vmlinux 0xffffffffaf007728
Build id: 3.10.0-1160.el7.x86_64 		0x68
5
page_mkclean_one
vmlinux 0xffffffffaf000cd6
Build id: 3.10.0-1160.el7.x86_64 		0xa6
6
rmap_walk
vmlinux 0xffffffffaf002ae3
Build id: 3.10.0-1160.el7.x86_64 		0xf3
7
page_mkclean
vmlinux 0xffffffffaf002ef6
Build id: 3.10.0-1160.el7.x86_64 		0x76
8
? page_referenced_one
vmlinux 0xffffffffaf000c30
Build id: 3.10.0-1160.el7.x86_64 		0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffaf000310
Build id: 3.10.0-1160.el7.x86_64 		0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffaefc9af0
Build id: 3.10.0-1160.el7.x86_64 		0x50
11
write_cache_pages
vmlinux 0xffffffffaefc9d62
Build id: 3.10.0-1160.el7.x86_64 		0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc03f7290
Build id: 3.10.0-1160.el7.x86_64 		0xa0
13
? __switch_to
vmlinux 0xffffffffaee2b59e
Build id: 3.10.0-1160.el7.x86_64 		0xce
14
xfs_vm_writepages
xfs 0xffffffffc03f725b
Build id: 3.10.0-1160.el7.x86_64 		0x6b
15
do_writepages
vmlinux 0xffffffffaefcade1
Build id: 3.10.0-1160.el7.x86_64 		0x21
16
__writeback_single_inode
vmlinux 0xffffffffaf07d6f0
Build id: 3.10.0-1160.el7.x86_64 		0x40
17
? wake_up_bit
vmlinux 0xffffffffaeec6cd5
Build id: 3.10.0-1160.el7.x86_64 		0x25
18
writeback_sb_inodes
vmlinux 0xffffffffaf07e274
Build id: 3.10.0-1160.el7.x86_64 		0x1c4
19
__writeback_inodes_wb
vmlinux 0xffffffffaf07e57f
Build id: 3.10.0-1160.el7.x86_64 		0x9f
20
wb_writeback
vmlinux 0xffffffffaf07ea53
Build id: 3.10.0-1160.el7.x86_64 		0x263
21
? bdi_dirty_limit
vmlinux 0xffffffffaefca550
Build id: 3.10.0-1160.el7.x86_64 		0x40
22
bdi_writeback_workfn
vmlinux 0xffffffffaf07f54c
Build id: 3.10.0-1160.el7.x86_64 		0x1cc
23
process_one_work
vmlinux 0xffffffffaeebdc4f
Build id: 3.10.0-1160.el7.x86_64 		0x17f
24
worker_thread
vmlinux 0xffffffffaeebed66
Build id: 3.10.0-1160.el7.x86_64 		0x126
25
? manage_workers.isra.26
vmlinux 0xffffffffaeebec40
Build id: 3.10.0-1160.el7.x86_64 		0x2a0
26
kthread
vmlinux 0xffffffffaeec5c21
Build id: 3.10.0-1160.el7.x86_64 		0xd1
27
? insert_kthread_work
vmlinux 0xffffffffaeec5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40
28
ret_from_fork_nospec_begin
vmlinux 0xffffffffaf593df7
Build id: 3.10.0-1160.el7.x86_64 		0x21
29
? insert_kthread_work
vmlinux 0xffffffffaeec5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u320:0:5]
Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter uio_ivshmem(OE) uio gsch(OE) redirfs(OE) sunrpc ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr virtio_balloon sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover virtio_console virtio_scsi virtio_blk failover cirrus drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 5 Comm: kworker/u320:0 Kdump: loaded Tainted: G           OE  ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff99b7b1754200 ti: ffff99b7b2368000 task.ti: ffff99b7b2368000
RIP: 0010:[<ffffffffaef16c0a>]  [<ffffffffaef16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff99b7b236b830  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00080000aee7e6a0 RCX: ffff99be5cf20400
RDX: 0000000000000004 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff99b7b236b868 R08: ffff99b7b2353000 R09: ffffffffaf185f99
R10: ffff99be5ce5f160 R11: ffffd3c49d193200 R12: 0000000000000286
R13: 0000000000000286 R14: ffff99b7b236b7e0 R15: ffff99b7a13eda84
FS:  0000000000000000(0000) GS:ffff99be5ce40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000448f30 CR3: 00000001497bc000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaee7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaee7ea04>] flush_tlb_page+0x54/0xa0
[<ffffffffaf007728>] ptep_clear_flush+0x68/0xa0
[<ffffffffaf000cd6>] page_mkclean_one+0xa6/0xe0
[<ffffffffaf002ae3>] rmap_walk+0xf3/0x360
[<ffffffffaf002ef6>] page_mkclean+0x76/0xb0
[<ffffffffaf000c30>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffaf000310>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffaefc9af0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffaefc9d62>] write_cache_pages+0x1f2/0x470
[<ffffffffc03f7290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffaee2b59e>] ? __switch_to+0xce/0x580
[<ffffffffc03f725b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffaefcade1>] do_writepages+0x21/0x50
[<ffffffffaf07d6f0>] __writeback_single_inode+0x40/0x260
[<ffffffffaeec6cd5>] ? wake_up_bit+0x25/0x30
[<ffffffffaf07e274>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffaf07e57f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffaf07ea53>] wb_writeback+0x263/0x2f0
[<ffffffffaefca550>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffffaf07f54c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffffaeebdc4f>] process_one_work+0x17f/0x440
[<ffffffffaeebed66>] worker_thread+0x126/0x3c0
[<ffffffffaeebec40>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffaeec5c21>] kthread+0xd1/0xe0
[<ffffffffaeec5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffaf593df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffaeec5b50>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 b5 af f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1013674
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9db1e238
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x208
2
native_flush_tlb_others
vmlinux 0xffffffff9da81fa8
Build id: 3.10.0-1160.102.1.el7.x86_64 		0xb8
3
flush_tlb_page
vmlinux 0xffffffff9da82144
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x54
4
ptep_clear_flush
vmlinux 0xffffffff9dc14228
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x68
5
page_mkclean_one
vmlinux 0xffffffff9dc0d5fe
Build id: 3.10.0-1160.102.1.el7.x86_64 		0xae
6
rmap_walk
vmlinux 0xffffffff9dc0f473
Build id: 3.10.0-1160.102.1.el7.x86_64 		0xf3
7
page_mkclean
vmlinux 0xffffffff9dc0f896
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x76
8
? page_referenced_one
vmlinux 0xffffffff9dc0d550
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffff9dc0cc20
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffff9dbd5770
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x50
11
write_cache_pages
vmlinux 0xffffffff9dbd59e2
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x1f2
12
? __enqueue_entity
vmlinux 0xffffffff9dae5d58
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x78
13
? xfs_vm_writepages
xfs 0xffffffffc06a86c0
Build id: 3.10.0-1160.102.1.el7.x86_64 		0xb0
14
xfs_vm_writepages
xfs 0xffffffffc06a867b
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x6b
15
do_writepages
vmlinux 0xffffffff9dbd6ab1
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x21
16
__writeback_single_inode
vmlinux 0xffffffff9dc8ccf0
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x40
17
? wake_up_bit
vmlinux 0xffffffff9dacc745
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x25
18
writeback_sb_inodes
vmlinux 0xffffffff9dc8d8bc
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x1cc
19
__writeback_inodes_wb
vmlinux 0xffffffff9dc8dbc7
Build id: 3.10.0-1160.102.1.el7.x86_64 		0xa7
20
wb_writeback
vmlinux 0xffffffff9dc8e0a3
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x263
21
? bdi_dirty_limit
vmlinux 0xffffffff9dbd61f0
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x40
22
bdi_writeback_workfn
vmlinux 0xffffffff9dc8ebac
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x1cc
23
process_one_work
vmlinux 0xffffffff9dac32ef
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x17f
24
worker_thread
vmlinux 0xffffffff9dac4436
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x126
25
? manage_workers.isra.26
vmlinux 0xffffffff9dac4310
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x2b0
26
kthread
vmlinux 0xffffffff9dacb621
Build id: 3.10.0-1160.102.1.el7.x86_64 		0xd1
27
? insert_kthread_work
vmlinux 0xffffffff9dacb550
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x40
28
ret_from_fork_nospec_begin
vmlinux 0xffffffff9e1c51dd
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x7
29
? insert_kthread_work
vmlinux 0xffffffff9dacb550
Build id: 3.10.0-1160.102.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u200:5:53738]
Modules linked in: tcp_lp tcp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 nfs_acl nfs lockd grace fscache kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev pcspkr virtio_balloon i2c_piix4 auth_rpcgss sunrpc binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom virtio_scsi virtio_net net_failover failover virtio_console ata_generic pata_acpi bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul
crct10dif_common crc32c_intel virtio_pci virtio_ring serio_raw virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 53738 Comm: kworker/u200:5 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.102.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9f08525fa100 ti: ffff9e3e6c484000 task.ti: ffff9e3e6c484000
RIP: 0010:[<ffffffff9db1e238>]  [<ffffffff9db1e238>] smp_call_function_many+0x208/0x270
RSP: 0018:ffff9e3e6c487830  EFLAGS: 00000202
RAX: 0000000000000009 RBX: 000800009da81b10 RCX: ffff9f0a24c60650
RDX: 0000000000000009 RSI: 0000000000000064 RDI: 0000000000000000
RBP: ffff9e3e6c487868 R08: ffff9d2ed20aa800 R09: ffffffff9dd9c7b9
R10: ffff9f0a24adf160 R11: ffffdd8fc3039200 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9e3e6c4877e0 R15: ffff9f0a13064008
FS:  0000000000000000(0000) GS:ffff9f0a24ac0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000ea6e1c CR3: 000001e06e420000 CR4: 0000000000760fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffff9da81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9da82144>] flush_tlb_page+0x54/0xa0
[<ffffffff9dc14228>] ptep_clear_flush+0x68/0xa0
[<ffffffff9dc0d5fe>] page_mkclean_one+0xae/0xe0
[<ffffffff9dc0f473>] rmap_walk+0xf3/0x370
[<ffffffff9dc0f896>] page_mkclean+0x76/0xb0
[<ffffffff9dc0d550>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff9dc0cc20>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff9dbd5770>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff9dbd59e2>] write_cache_pages+0x1f2/0x480
[<ffffffff9dae5d58>] ? __enqueue_entity+0x78/0x80
[<ffffffffc06a86c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffffc06a867b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffff9dbd6ab1>] do_writepages+0x21/0x50
[<ffffffff9dc8ccf0>] __writeback_single_inode+0x40/0x260
[<ffffffff9dacc745>] ? wake_up_bit+0x25/0x30
[<ffffffff9dc8d8bc>] writeback_sb_inodes+0x1cc/0x430
[<ffffffff9dc8dbc7>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffff9dc8e0a3>] wb_writeback+0x263/0x2f0
[<ffffffff9dbd61f0>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffff9dc8ebac>] bdi_writeback_workfn+0x1cc/0x470
[<ffffffff9dac32ef>] process_one_work+0x17f/0x440
[<ffffffff9dac4436>] worker_thread+0x126/0x3c0
[<ffffffff9dac4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff9dacb621>] kthread+0xd1/0xe0
[<ffffffff9dacb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff9e1c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff9dacb550>] ? insert_kthread_work+0x40/0x40
Code: 29 00 48 63 35 4e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 19 75 9e f6 41 20 01 74 cd 0f 1f 44 00 00 <f3> 90 f6 41 20 01 75 f8 48 63 35 1d 18 c4 00 eb b7 0f b6 4d cc
Complete report #936560
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff89b1718a
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff89a7e258
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xb8
3
flush_tlb_page
vmlinux 0xffffffff89a7e3f4
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x54
4
ptep_clear_flush
vmlinux 0xffffffff89c08108
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x68
5
page_mkclean_one
vmlinux 0xffffffff89c01686
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xa6
6
rmap_walk
vmlinux 0xffffffff89c034b3
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xf3
7
page_mkclean
vmlinux 0xffffffff89c038c6
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x76
8
? page_referenced_one
vmlinux 0xffffffff89c015e0
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffff89c00cc0
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffff89bca700
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x50
11
write_cache_pages
vmlinux 0xffffffff89bca972
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc06aa290
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xa0
13
? submit_bio
vmlinux 0xffffffff89d55c70
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x70
14
? update_group_power
vmlinux 0xffffffff89ae82d4
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x174
15
xfs_vm_writepages
xfs 0xffffffffc06aa25b
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x6b
16
do_writepages
vmlinux 0xffffffff89bcb9f1
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x21
17
__writeback_single_inode
vmlinux 0xffffffff89c7df50
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x40
18
? wake_up_bit
vmlinux 0xffffffff89ac7045
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x25
19
writeback_sb_inodes
vmlinux 0xffffffff89c7ead4
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffff89c7eddf
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x9f
21
wb_writeback
vmlinux 0xffffffff89c7f2b3
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x263
22
? bdi_dirty_limit
vmlinux 0xffffffff89bcb160
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x40
23
bdi_writeback_workfn
vmlinux 0xffffffff89c7fdac
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x1cc
24
process_one_work
vmlinux 0xffffffff89abdfbf
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x17f
25
worker_thread
vmlinux 0xffffffff89abf0d6
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x126
26
? manage_workers.isra.26
vmlinux 0xffffffff89abefb0
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x2a0
27
kthread
vmlinux 0xffffffff89ac5f91
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xd1
28
? insert_kthread_work
vmlinux 0xffffffff89ac5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffff8a199df7
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x21
30
? insert_kthread_work
vmlinux 0xffffffff89ac5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [kworker/u256:1:129553]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus sb_edac snd_seq snd_seq_device iosf_mbi crc32_pclmul snd_pcm ghash_clmulni_intel
btusb snd_timer btrtl aesni_intel btbcm lrw gf128mul btintel snd glue_helper bluetooth ablk_helper soundcore cryptd ppdev rfkill joydev vmw_balloon pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix drm nfit crct10dif_pclmul crct10dif_common crc32c_intel scsi_transport_spi libnvdimm libata mptscsih serio_raw e1000 mptbase drm_panel_orientation_quirks synosnap(OE) fuse
CPU: 9 PID: 129553 Comm: kworker/u256:1 Kdump: loaded Tainted: G        W  OEL ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffffa122bd5b3180 ti: ffffa125c88fc000 task.ti: ffffa125c88fc000
RIP: 0010:[<ffffffff89b1718a>]  [<ffffffff89b1718a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa125c88ff830  EFLAGS: 00000202
RAX: 000000000000000b RBX: 0008000089a7ddd0 RCX: ffffa1261d8e0740
RDX: 000000000000000b RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffffa125c88ff868 R08: ffffa11f6c55d800 R09: ffffffff89d869f9
R10: ffffa1261d85f160 R11: fffff1a312a29400 R12: 0000000000000286
R13: 0000000000000286 R14: ffffa125c88ff7e0 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffa1261d840000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f980b224b90 CR3: 00000006fa010000 CR4: 00000000003607e0
Call Trace:
[<ffffffff89a7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff89a7e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffff89c08108>] ptep_clear_flush+0x68/0xa0
[<ffffffff89c01686>] page_mkclean_one+0xa6/0xe0
[<ffffffff89c034b3>] rmap_walk+0xf3/0x360
[<ffffffff89c038c6>] page_mkclean+0x76/0xb0
[<ffffffff89c015e0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff89c00cc0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff89bca700>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff89bca972>] write_cache_pages+0x1f2/0x470
[<ffffffffc06aa290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff89d55c70>] ? submit_bio+0x70/0x150
[<ffffffff89ae82d4>] ? update_group_power+0x174/0x280
[<ffffffffc06aa25b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff89bcb9f1>] do_writepages+0x21/0x50
[<ffffffff89c7df50>] __writeback_single_inode+0x40/0x260
[<ffffffff89ac7045>] ? wake_up_bit+0x25/0x30
[<ffffffff89c7ead4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff89c7eddf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff89c7f2b3>] wb_writeback+0x263/0x2f0
[<ffffffff89bcb160>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffff89c7fdac>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffff89abdfbf>] process_one_work+0x17f/0x440
[<ffffffff89abf0d6>] worker_thread+0x126/0x3c0
[<ffffffff89abefb0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff89ac5f91>] kthread+0xd1/0xe0
[<ffffffff89ac5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffff8a199df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff89ac5ec0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 be 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 75 8a f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #749902
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb691718a
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb687e258
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xb8
3
flush_tlb_page
vmlinux 0xffffffffb687e3f4
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x54
4
ptep_clear_flush
vmlinux 0xffffffffb6a08108
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x68
5
page_mkclean_one
vmlinux 0xffffffffb6a01686
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xa6
6
rmap_walk
vmlinux 0xffffffffb6a034b3
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xf3
7
page_mkclean
vmlinux 0xffffffffb6a038c6
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x76
8
? page_referenced_one
vmlinux 0xffffffffb6a015e0
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffb6a00cc0
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffb69ca700
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x50
11
write_cache_pages
vmlinux 0xffffffffb69ca972
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc04e1290
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xa0
13
? set_next_entity
vmlinux 0xffffffffb68e283c
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x3c
14
xfs_vm_writepages
xfs 0xffffffffc04e125b
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x6b
15
do_writepages
vmlinux 0xffffffffb69cb9f1
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x21
16
__writeback_single_inode
vmlinux 0xffffffffb6a7df50
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x40
17
writeback_sb_inodes
vmlinux 0xffffffffb6a7ead4
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x1c4
18
__writeback_inodes_wb
vmlinux 0xffffffffb6a7eddf
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x9f
19
wb_writeback
vmlinux 0xffffffffb6a7f2b3
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x263
20
? bdi_dirty_limit
vmlinux 0xffffffffb69cb160
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x40
21
bdi_writeback_workfn
vmlinux 0xffffffffb6a7fdac
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x1cc
22
process_one_work
vmlinux 0xffffffffb68bdfbf
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x17f
23
worker_thread
vmlinux 0xffffffffb68bf0d6
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x126
24
? manage_workers.isra.26
vmlinux 0xffffffffb68befb0
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x2a0
25
kthread
vmlinux 0xffffffffb68c5f91
Build id: 3.10.0-1160.76.1.el7.x86_64 		0xd1
26
? insert_kthread_work
vmlinux 0xffffffffb68c5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x40
27
ret_from_fork_nospec_begin
vmlinux 0xffffffffb6f99df7
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x21
28
? insert_kthread_work
vmlinux 0xffffffffb68c5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 24s! [kworker/u192:2:73004]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) sunrpc ppdev sg joydev virtio_balloon parport_pc parport i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod cdrom virtio_net ata_generic pata_acpi net_failover virtio_console virtio_scsi failover virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ata_piix ttm libata drm serio_raw virtio_pci virtio_ring floppy virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 4 PID: 73004 Comm: kworker/u192:2 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
task: ffff956b950c5280 ti: ffff956b44acc000 task.ti: ffff956b44acc000
RIP: 0010:[<ffffffffb691718a>]  [<ffffffffb691718a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff956b44acf830  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00080000b687ddd0 RCX: ffff956d2dee0478
RDX: 0000000000000003 RSI: 0000000000000060 RDI: 0000000000000000
RBP: ffff956b44acf868 R08: ffff956a763bbc00 R09: ffffffffb6b869f9
R10: ffff956d2df1f160 R11: ffffdc6ccaddce00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff956b44acf7e0 R15: ffff9569355f1884
FS:  0000000000000000(0000) GS:ffff956d2df00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000559d830dfaf6 CR3: 000000009e3c0000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb687e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb687e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffb6a08108>] ptep_clear_flush+0x68/0xa0
[<ffffffffb6a01686>] page_mkclean_one+0xa6/0xe0
[<ffffffffb6a034b3>] rmap_walk+0xf3/0x360
[<ffffffffb6a038c6>] page_mkclean+0x76/0xb0
[<ffffffffb6a015e0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffb6a00cc0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb69ca700>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb69ca972>] write_cache_pages+0x1f2/0x470
[<ffffffffc04e1290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffb68e283c>] ? set_next_entity+0x3c/0xe0
[<ffffffffc04e125b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb69cb9f1>] do_writepages+0x21/0x50
[<ffffffffb6a7df50>] __writeback_single_inode+0x40/0x260
[<ffffffffb6a7ead4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffb6a7eddf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb6a7f2b3>] wb_writeback+0x263/0x2f0
[<ffffffffb69cb160>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffffb6a7fdac>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffffb68bdfbf>] process_one_work+0x17f/0x440
[<ffffffffb68bf0d6>] worker_thread+0x126/0x3c0
[<ffffffffb68befb0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb68c5f91>] kthread+0xd1/0xe0
[<ffffffffb68c5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb6f99df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb68c5ec0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 be 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 55 b7 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #738980
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb3116eea
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb307e868
Build id: 3.10.0-1160.25.1.el7.x86_64 		0xb8
3
flush_tlb_page
vmlinux 0xffffffffb307ea04
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x54
4
ptep_clear_flush
vmlinux 0xffffffffb32079d8
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x68
5
page_mkclean_one
vmlinux 0xffffffffb3200f86
Build id: 3.10.0-1160.25.1.el7.x86_64 		0xa6
6
rmap_walk
vmlinux 0xffffffffb3202d93
Build id: 3.10.0-1160.25.1.el7.x86_64 		0xf3
7
page_mkclean
vmlinux 0xffffffffb32031a6
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x76
8
? page_referenced_one
vmlinux 0xffffffffb3200ee0
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffb32005c0
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffb31c9d10
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x50
11
write_cache_pages
vmlinux 0xffffffffb31c9f82
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc05f7290
Build id: 3.10.0-1160.25.1.el7.x86_64 		0xa0
13
? update_group_power
vmlinux 0xffffffffb30e8114
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x174
14
? cpumask_next_and
vmlinux 0xffffffffb3386395
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x35
15
? find_busiest_group
vmlinux 0xffffffffb30e8364
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x144
16
xfs_vm_writepages
xfs 0xffffffffc05f725b
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffffb31cb001
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffffb327d9a0
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x40
19
writeback_sb_inodes
vmlinux 0xffffffffb327e524
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffffb327e82f
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x9f
21
wb_writeback
vmlinux 0xffffffffb327ed03
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x263
22
? bdi_dirty_limit
vmlinux 0xffffffffb31ca770
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x40
23
bdi_writeback_workfn
vmlinux 0xffffffffb327f7fc
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x1cc
24
process_one_work
vmlinux 0xffffffffb30bddcf
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x17f
25
worker_thread
vmlinux 0xffffffffb30beee6
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x126
26
? manage_workers.isra.26
vmlinux 0xffffffffb30bedc0
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x2a0
27
kthread
vmlinux 0xffffffffb30c5da1
Build id: 3.10.0-1160.25.1.el7.x86_64 		0xd1
28
? insert_kthread_work
vmlinux 0xffffffffb30c5cd0
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffffb3795df7
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x21
30
? insert_kthread_work
vmlinux 0xffffffffb30c5cd0
Build id: 3.10.0-1160.25.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#22 stuck for 62s! [kworker/u66:0:9589]
Modules linked in: binfmt_misc xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul vmw_balloon glue_helper ablk_helper pcspkr cryptd sg joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci ata_piix nfit crct10dif_pclmul crct10dif_common libata crc32c_intel libnvdimm serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks
dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 22 PID: 9589 Comm: kworker/u66:0 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff918c95d30000 ti: ffff91909eefc000 task.ti: ffff91909eefc000
RIP: 0010:[<ffffffffb3116eea>]  [<ffffffffb3116eea>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff91909eeff830  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000b307e6a0 RCX: ffff9189490209a8
RDX: 0000000000000000 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff91909eeff868 R08: ffff9189dc745c00 R09: ffffffffb3386549
R10: ffff91911279f160 R11: ffffc21b20f2f200 R12: 0000000000000286
R13: 0000000000000286 R14: ffff91909eeff7e0 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff919112780000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000003ff5c62a0 CR3: 0000000f36410000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb307e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb307ea04>] flush_tlb_page+0x54/0xa0
[<ffffffffb32079d8>] ptep_clear_flush+0x68/0xa0
[<ffffffffb3200f86>] page_mkclean_one+0xa6/0xe0
[<ffffffffb3202d93>] rmap_walk+0xf3/0x360
[<ffffffffb32031a6>] page_mkclean+0x76/0xb0
[<ffffffffb3200ee0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffb32005c0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb31c9d10>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb31c9f82>] write_cache_pages+0x1f2/0x470
[<ffffffffc05f7290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffb30e8114>] ? update_group_power+0x174/0x280
[<ffffffffb3386395>] ? cpumask_next_and+0x35/0x50
[<ffffffffb30e8364>] ? find_busiest_group+0x144/0x990
[<ffffffffc05f725b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb31cb001>] do_writepages+0x21/0x50
[<ffffffffb327d9a0>] __writeback_single_inode+0x40/0x260
[<ffffffffb327e524>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffb327e82f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb327ed03>] wb_writeback+0x263/0x2f0
[<ffffffffb31ca770>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffffb327f7fc>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffffb30bddcf>] process_one_work+0x17f/0x440
[<ffffffffb30beee6>] worker_thread+0x126/0x3c0
[<ffffffffb30bedc0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb30c5da1>] kthread+0xd1/0xe0
[<ffffffffb30c5cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb3795df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb30c5cd0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 fe 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 d5 b3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 87 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #644075
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffac71e0ba
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffac681f88
Build id: 3.10.0-1160.83.1.el7.x86_64 		0xb8
3
flush_tlb_page
vmlinux 0xffffffffac682124
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x54
4
ptep_clear_flush
vmlinux 0xffffffffac813ef8
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x68
5
page_mkclean_one
vmlinux 0xffffffffac80d2de
Build id: 3.10.0-1160.83.1.el7.x86_64 		0xae
6
rmap_walk
vmlinux 0xffffffffac80f143
Build id: 3.10.0-1160.83.1.el7.x86_64 		0xf3
7
page_mkclean
vmlinux 0xffffffffac80f566
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x76
8
? page_referenced_one
vmlinux 0xffffffffac80d230
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffac80c8c0
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffac7d55e0
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x50
11
write_cache_pages
vmlinux 0xffffffffac7d5852
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc029a6c0
Build id: 3.10.0-1160.83.1.el7.x86_64 		0xb0
13
? default_wake_function
vmlinux 0xffffffffac6e1072
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x12
14
? update_group_power
vmlinux 0xffffffffac6ee50c
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x17c
15
? cpumask_next_and
vmlinux 0xffffffffac99bf95
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x35
16
xfs_vm_writepages
xfs 0xffffffffc029a67b
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffffac7d6921
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffffac88c790
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x40
19
writeback_sb_inodes
vmlinux 0xffffffffac88d35c
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x1cc
20
__writeback_inodes_wb
vmlinux 0xffffffffac88d667
Build id: 3.10.0-1160.83.1.el7.x86_64 		0xa7
21
wb_writeback
vmlinux 0xffffffffac88db43
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x263
22
? bdi_dirty_limit
vmlinux 0xffffffffac7d6060
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x40
23
bdi_writeback_workfn
vmlinux 0xffffffffac88e64c
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x1cc
24
process_one_work
vmlinux 0xffffffffac6c31df
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x17f
25
worker_thread
vmlinux 0xffffffffac6c4326
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x126
26
? manage_workers.isra.26
vmlinux 0xffffffffac6c4200
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x2b0
27
kthread
vmlinux 0xffffffffac6cb511
Build id: 3.10.0-1160.83.1.el7.x86_64 		0xd1
28
? insert_kthread_work
vmlinux 0xffffffffac6cb440
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffffacdc51f7
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x21
30
? insert_kthread_work
vmlinux 0xffffffffac6cb440
Build id: 3.10.0-1160.83.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u12:1:7273]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_addrtype iptable_filter br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_14713(E) overlay snd_hda_codec_generic iosf_mbi crc32_pclmul ghash_clmulni_intel snd_hda_intel aesni_intel snd_hda_codec lrw gf128mul glue_helper snd_hda_core snd_hwdep pcspkr snd_seq ablk_helper cryptd joydev virtio_rng snd_seq_device i2c_piix4 virtio_balloon snd_pcm snd_timer snd soundcore ip_tables xfs libcrc32c ata_generic pata_acpi virtio_blk virtio_console qxl drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul crct10dif_common floppy crc32c_intel virtio_pci e1000 serio_raw virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 7273 Comm: kworker/u12:1 Tainted: P            E  ------------   3.10.0-1160.83.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.15.0-2.module+el8.6.0+20659+3dcf7c70 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:1)
task: ffff913360afc200 ti: ffff91335ad2c000 task.ti: ffff91335ad2c000
RIP: 0010:[<ffffffffac71e0ba>]  [<ffffffffac71e0ba>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff91335ad2f830  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00080000ac681af0 RCX: ffff91347fce0348
RDX: 0000000000000003 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff91335ad2f868 R08: ffff91343bed9400 R09: ffffffffac99c179
R10: ffff91347fc9f160 R11: fffff1c7c6ddb100 R12: 0000000000000286
R13: 0000000000000286 R14: ffff91335ad2f7e0 R15: ffff913476481030
FS:  0000000000000000(0000) GS:ffff91347fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00187a00d CR3: 0000000019546000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffac681f88>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffac682124>] flush_tlb_page+0x54/0xa0
[<ffffffffac813ef8>] ptep_clear_flush+0x68/0xa0
[<ffffffffac80d2de>] page_mkclean_one+0xae/0xe0
[<ffffffffac80f143>] rmap_walk+0xf3/0x370
[<ffffffffac80f566>] page_mkclean+0x76/0xb0
[<ffffffffac80d230>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffac80c8c0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffac7d55e0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffac7d5852>] write_cache_pages+0x1f2/0x480
[<ffffffffc029a6c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffffac6e1072>] ? default_wake_function+0x12/0x20
[<ffffffffac6ee50c>] ? update_group_power+0x17c/0x280
[<ffffffffac99bf95>] ? cpumask_next_and+0x35/0x50
[<ffffffffc029a67b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffffac7d6921>] do_writepages+0x21/0x50
[<ffffffffac88c790>] __writeback_single_inode+0x40/0x260
[<ffffffffac88d35c>] writeback_sb_inodes+0x1cc/0x430
[<ffffffffac88d667>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffffac88db43>] wb_writeback+0x263/0x2f0
[<ffffffffac7d6060>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffffac88e64c>] bdi_writeback_workfn+0x1cc/0x470
[<ffffffffac6c31df>] process_one_work+0x17f/0x440
[<ffffffffac6c4326>] worker_thread+0x126/0x3c0
[<ffffffffac6c4200>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffffac6cb511>] kthread+0xd1/0xe0
[<ffffffffac6cb440>] ? insert_kthread_work+0x40/0x40
[<ffffffffacdc51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffac6cb440>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 8e 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 35 ad f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d 19 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #493505
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff86f16c5a
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff86e7e868
Build id: 3.10.0-1160.6.1.el7.x86_64 		0xb8
3
flush_tlb_page
vmlinux 0xffffffff86e7ea04
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x54
4
ptep_clear_flush
vmlinux 0xffffffff87007768
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x68
5
page_mkclean_one
vmlinux 0xffffffff87000d16
Build id: 3.10.0-1160.6.1.el7.x86_64 		0xa6
6
rmap_walk
vmlinux 0xffffffff87002b23
Build id: 3.10.0-1160.6.1.el7.x86_64 		0xf3
7
page_mkclean
vmlinux 0xffffffff87002f36
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x76
8
? page_referenced_one
vmlinux 0xffffffff87000c70
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffff87000350
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffff86fc9b30
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x50
11
write_cache_pages
vmlinux 0xffffffff86fc9da2
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc0685290
Build id: 3.10.0-1160.6.1.el7.x86_64 		0xa0
13
? __rmqueue
vmlinux 0xffffffff86fc5b2a
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x8a
14
? update_curr
vmlinux 0xffffffff86ee441c
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x14c
15
? account_entity_dequeue
vmlinux 0xffffffff86ee116e
Build id: 3.10.0-1160.6.1.el7.x86_64 		0xae
16
xfs_vm_writepages
xfs 0xffffffffc068525b
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffff86fcae21
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffff8707d730
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x40
19
? __schedule
vmlinux 0xffffffff87586d4f
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x3af
20
writeback_sb_inodes
vmlinux 0xffffffff8707e2b4
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x1c4
21
__writeback_inodes_wb
vmlinux 0xffffffff8707e5bf
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x9f
22
wb_writeback
vmlinux 0xffffffff8707ea93
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x263
23
? bdi_dirty_limit
vmlinux 0xffffffff86fca590
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x40
24
bdi_writeback_workfn
vmlinux 0xffffffff8707f58c
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x1cc
25
process_one_work
vmlinux 0xffffffff86ebdc4f
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x17f
26
worker_thread
vmlinux 0xffffffff86ebed66
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x126
27
? manage_workers.isra.26
vmlinux 0xffffffff86ebec40
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x2a0
28
kthread
vmlinux 0xffffffff86ec5c21
Build id: 3.10.0-1160.6.1.el7.x86_64 		0xd1
29
? insert_kthread_work
vmlinux 0xffffffff86ec5b50
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffff87594ddd
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x7
31
? insert_kthread_work
vmlinux 0xffffffff86ec5b50
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [kworker/u8:0:5]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink sunrpc snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm joydev snd_timer snd iosf_mbi vboxguest(OE) soundcore sg parport_pc video pcspkr parport i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops pata_acpi ahci ttm crc32c_intel libahci serio_raw drm e1000 ata_piix drm_panel_orientation_quirks libata dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 5 Comm: kworker/u8:0 Kdump: loaded Tainted: G        W  OEL ------------   3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffffa0251bb94200 ti: ffffa0251bbc8000 task.ti: ffffa0251bbc8000
RIP: 0010:[<ffffffff86f16c5a>]  [<ffffffff86f16c5a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa0251bbcb830  EFLAGS: 00010202
RAX: 0000000000000001 RBX: ffffffff86e7e6a0 RCX: ffffa028d22a0428
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffffa0251bbcb868 R08: ffffa0251bbae400 R09: ffffffff87185fd9
R10: ffffa028d231f160 R11: ffffe2a80d5d0300 R12: ffffa0251bbcb800
R13: ffffffff87185f2f R14: ffffa0251bbcb7d8 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffffa028d2300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7d773f75b8 CR3: 00000004fd9e4000 CR4: 00000000000206e0
Call Trace:
[<ffffffff86e7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff86e7ea04>] flush_tlb_page+0x54/0xa0
[<ffffffff87007768>] ptep_clear_flush+0x68/0xa0
[<ffffffff87000d16>] page_mkclean_one+0xa6/0xe0
[<ffffffff87002b23>] rmap_walk+0xf3/0x360
[<ffffffff87002f36>] page_mkclean+0x76/0xb0
[<ffffffff87000c70>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff87000350>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff86fc9b30>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff86fc9da2>] write_cache_pages+0x1f2/0x470
[<ffffffffc0685290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff86fc5b2a>] ? __rmqueue+0x8a/0x460
[<ffffffff86ee441c>] ? update_curr+0x14c/0x1e0
[<ffffffff86ee116e>] ? account_entity_dequeue+0xae/0xd0
[<ffffffffc068525b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff86fcae21>] do_writepages+0x21/0x50
[<ffffffff8707d730>] __writeback_single_inode+0x40/0x260
[<ffffffff87586d4f>] ? __schedule+0x3af/0x860
[<ffffffff8707e2b4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8707e5bf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8707ea93>] wb_writeback+0x263/0x2f0
[<ffffffff86fca590>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffff8707f58c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffff86ebdc4f>] process_one_work+0x17f/0x440
[<ffffffff86ebed66>] worker_thread+0x126/0x3c0
[<ffffffff86ebec40>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff86ec5c21>] kthread+0xd1/0xe0
[<ffffffff86ec5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff87594ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff86ec5b50>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 b5 87 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:37:crtc-0] flip_done timed out
Complete report #439204
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff85f1720a
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x20a
2
native_flush_tlb_others
vmlinux 0xffffffff85e7e258
Build id: 3.10.0-1160.66.1.el7.x86_64 		0xb8
3
flush_tlb_page
vmlinux 0xffffffff85e7e3f4
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x54
4
ptep_clear_flush
vmlinux 0xffffffff86008108
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x68
5
page_mkclean_one
vmlinux 0xffffffff860016a6
Build id: 3.10.0-1160.66.1.el7.x86_64 		0xa6
6
rmap_walk
vmlinux 0xffffffff860034b3
Build id: 3.10.0-1160.66.1.el7.x86_64 		0xf3
7
page_mkclean
vmlinux 0xffffffff860038c6
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x76
8
? page_referenced_one
vmlinux 0xffffffff86001600
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffff86000ce0
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffff85fca6e0
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x50
11
write_cache_pages
vmlinux 0xffffffff85fca952
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc0478290
Build id: 3.10.0-1160.66.1.el7.x86_64 		0xa0
13
? __enqueue_entity
vmlinux 0xffffffff85edfd08
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x78
14
? put_prev_entity
vmlinux 0xffffffff85ee64ab
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x8b
15
? pick_next_entity
vmlinux 0xffffffff85ee2409
Build id: 3.10.0-1160.66.1.el7.x86_64 		0xa9
16
? set_next_entity
vmlinux 0xffffffff85ee281c
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x3c
17
xfs_vm_writepages
xfs 0xffffffffc047825b
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x6b
18
do_writepages
vmlinux 0xffffffff85fcb9d1
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x21
19
__writeback_single_inode
vmlinux 0xffffffff8607df40
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x40
20
writeback_sb_inodes
vmlinux 0xffffffff8607eac4
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x1c4
21
__writeback_inodes_wb
vmlinux 0xffffffff8607edcf
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x9f
22
wb_writeback
vmlinux 0xffffffff8607f2a3
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x263
23
? bdi_dirty_limit
vmlinux 0xffffffff85fcb140
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x40
24
bdi_writeback_workfn
vmlinux 0xffffffff8607fd9c
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x1cc
25
process_one_work
vmlinux 0xffffffff85ebdfbf
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x17f
26
worker_thread
vmlinux 0xffffffff85ebf0d6
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x126
27
? manage_workers.isra.26
vmlinux 0xffffffff85ebefb0
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x2a0
28
kthread
vmlinux 0xffffffff85ec5f91
Build id: 3.10.0-1160.66.1.el7.x86_64 		0xd1
29
? insert_kthread_work
vmlinux 0xffffffff85ec5ec0
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffff86599df7
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x21
31
? insert_kthread_work
vmlinux 0xffffffff85ec5ec0
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u12:2:7305]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13501(E) snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq iosf_mbi snd_seq_device crc32_pclmul snd_pcm ghash_clmulni_intel snd_timer snd aesni_intel soundcore lrw gf128mul glue_helper ablk_helper cryptd joydev virtio_rng ppdev virtio_balloon i2c_piix4 parport_pc parport pcspkr ip_tables xfs libcrc32c ata_generic pata_acpi virtio_console virtio_blk qxl drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common libata e1000 crc32c_intel serio_raw virtio_pci virtio_ring virtio floppy drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 7305 Comm: kworker/u12:2 Tainted: P            E  ------------   3.10.0-1160.66.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module+el8.3.0+7860+a7792d29 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:1)
task: ffff8ae1fdc2b180 ti: ffff8ae0b46a8000 task.ti: ffff8ae0b46a8000
RIP: 0010:[<ffffffff85f1720a>]  [<ffffffff85f1720a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8ae0b46ab830  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 0008000085e7ddd0 RCX: ffff8ae1ffce0348
RDX: 0000000000000003 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff8ae0b46ab868 R08: ffff8ae1bbed9400 R09: ffffffff861869e9
R10: ffff8ae1ffc9f160 R11: ffffe43a04084700 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8ae0b46ab7e0 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff8ae1ffc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c001c15000 CR3: 000000009a460000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff85e7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff85e7e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffff86008108>] ptep_clear_flush+0x68/0xa0
[<ffffffff860016a6>] page_mkclean_one+0xa6/0xe0
[<ffffffff860034b3>] rmap_walk+0xf3/0x360
[<ffffffff860038c6>] page_mkclean+0x76/0xb0
[<ffffffff86001600>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff86000ce0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff85fca6e0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff85fca952>] write_cache_pages+0x1f2/0x470
[<ffffffffc0478290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff85edfd08>] ? __enqueue_entity+0x78/0x80
[<ffffffff85ee64ab>] ? put_prev_entity+0x8b/0x400
[<ffffffff85ee2409>] ? pick_next_entity+0xa9/0x190
[<ffffffff85ee281c>] ? set_next_entity+0x3c/0xe0
[<ffffffffc047825b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff85fcb9d1>] do_writepages+0x21/0x50
[<ffffffff8607df40>] __writeback_single_inode+0x40/0x260
[<ffffffff8607eac4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8607edcf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8607f2a3>] wb_writeback+0x263/0x2f0
[<ffffffff85fcb140>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffff8607fd9c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffff85ebdfbf>] process_one_work+0x17f/0x440
[<ffffffff85ebf0d6>] worker_thread+0x126/0x3c0
[<ffffffff85ebefb0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff85ec5f91>] kthread+0xd1/0xe0
[<ffffffff85ec5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffff86599df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff85ec5ec0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 9e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 b5 86 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d 87 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #367914
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa8716c5a
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffa867e868
Build id: 3.10.0-1160.6.1.el7.x86_64 		0xb8
3
flush_tlb_page
vmlinux 0xffffffffa867ea04
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x54
4
ptep_clear_flush
vmlinux 0xffffffffa8807768
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x68
5
page_mkclean_one
vmlinux 0xffffffffa8800d16
Build id: 3.10.0-1160.6.1.el7.x86_64 		0xa6
6
rmap_walk
vmlinux 0xffffffffa8802b23
Build id: 3.10.0-1160.6.1.el7.x86_64 		0xf3
7
page_mkclean
vmlinux 0xffffffffa8802f36
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x76
8
? page_referenced_one
vmlinux 0xffffffffa8800c70
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffa8800350
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffa87c9b30
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x50
11
write_cache_pages
vmlinux 0xffffffffa87c9da2
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc0395290
Build id: 3.10.0-1160.6.1.el7.x86_64 		0xa0
13
? submit_bio
vmlinux 0xffffffffa8955450
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x70
14
? cpumask_next_and
vmlinux 0xffffffffa8985e25
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x35
15
? find_busiest_group
vmlinux 0xffffffffa86e80a4
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x144
16
xfs_vm_writepages
xfs 0xffffffffc039525b
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffffa87cae21
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffffa887d730
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x40
19
? wake_up_bit
vmlinux 0xffffffffa86c6cd5
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x25
20
writeback_sb_inodes
vmlinux 0xffffffffa887e2b4
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x1c4
21
__writeback_inodes_wb
vmlinux 0xffffffffa887e5bf
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x9f
22
wb_writeback
vmlinux 0xffffffffa887ea93
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x263
23
? bdi_dirty_limit
vmlinux 0xffffffffa87ca590
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x40
24
bdi_writeback_workfn
vmlinux 0xffffffffa887f58c
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x1cc
25
process_one_work
vmlinux 0xffffffffa86bdc4f
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x17f
26
worker_thread
vmlinux 0xffffffffa86bed66
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x126
27
? manage_workers.isra.26
vmlinux 0xffffffffa86bec40
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x2a0
28
kthread
vmlinux 0xffffffffa86c5c21
Build id: 3.10.0-1160.6.1.el7.x86_64 		0xd1
29
? insert_kthread_work
vmlinux 0xffffffffa86c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffffa8d94ddd
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x7
31
? insert_kthread_work
vmlinux 0xffffffffa86c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u10:0:6936]
Modules linked in: tcp_lp nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter sunrpc snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer joydev ppdev snd iosf_mbi vboxguest(OE) parport_pc sg soundcore video i2c_piix4 parport pcspkr binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt ahci fb_sys_fops ata_piix libahci ttm drm libata serio_raw e1000 crc32c_intel drm_panel_orientation_quirks dm_mirror dm_region_hash
dm_log dm_mod fuse
CPU: 3 PID: 6936 Comm: kworker/u10:0 Kdump: loaded Tainted: G           OE  ------------   3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff933353506300 ti: ffff933222e14000 task.ti: ffff933222e14000
RIP: 0010:[<ffffffffa8716c5a>]  [<ffffffffa8716c5a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff933222e17830  EFLAGS: 00010202
RAX: 0000000000000002 RBX: ffff933222e177f0 RCX: ffff93344f220450
RDX: 0000000000000002 RSI: 0000000000000005 RDI: 0000000000000000
RBP: ffff933222e17868 R08: ffff9331b8f63c00 R09: ffffffffa8985fd9
R10: ffff93344f31f160 R11: ffffd2cfc5e9f000 R12: ffff933222e17800
R13: ffffffffa8985f2f R14: ffff9331bfc01500 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff93344f300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f44c38e4af8 CR3: 00000002aa610000 CR4: 00000000000206e0
Call Trace:
[<ffffffffa867e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa867ea04>] flush_tlb_page+0x54/0xa0
[<ffffffffa8807768>] ptep_clear_flush+0x68/0xa0
[<ffffffffa8800d16>] page_mkclean_one+0xa6/0xe0
[<ffffffffa8802b23>] rmap_walk+0xf3/0x360
[<ffffffffa8802f36>] page_mkclean+0x76/0xb0
[<ffffffffa8800c70>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffa8800350>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffa87c9b30>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa87c9da2>] write_cache_pages+0x1f2/0x470
[<ffffffffc0395290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffa8955450>] ? submit_bio+0x70/0x150
[<ffffffffa8985e25>] ? cpumask_next_and+0x35/0x50
[<ffffffffa86e80a4>] ? find_busiest_group+0x144/0x990
[<ffffffffc039525b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa87cae21>] do_writepages+0x21/0x50
[<ffffffffa887d730>] __writeback_single_inode+0x40/0x260
[<ffffffffa86c6cd5>] ? wake_up_bit+0x25/0x30
[<ffffffffa887e2b4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffa887e5bf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa887ea93>] wb_writeback+0x263/0x2f0
[<ffffffffa87ca590>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffffa887f58c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffffa86bdc4f>] process_one_work+0x17f/0x440
[<ffffffffa86bed66>] worker_thread+0x126/0x3c0
[<ffffffffa86bec40>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffa86c5c21>] kthread+0xd1/0xe0
[<ffffffffa86c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffa8d94ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa86c5b50>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 35 a9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #330238
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffae1170da
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffae07e258
Build id: 3.10.0-1160.49.1.el7.x86_64 		0xb8
3
flush_tlb_page
vmlinux 0xffffffffae07e3f4
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x54
4
ptep_clear_flush
vmlinux 0xffffffffae208108
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x68
5
page_mkclean_one
vmlinux 0xffffffffae2016a6
Build id: 3.10.0-1160.49.1.el7.x86_64 		0xa6
6
rmap_walk
vmlinux 0xffffffffae2034b3
Build id: 3.10.0-1160.49.1.el7.x86_64 		0xf3
7
page_mkclean
vmlinux 0xffffffffae2038c6
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x76
8
? page_referenced_one
vmlinux 0xffffffffae201600
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffffae200ce0
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffffae1ca6c0
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x50
11
write_cache_pages
vmlinux 0xffffffffae1ca932
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc0541290
Build id: 3.10.0-1160.49.1.el7.x86_64 		0xa0
13
? cpumask_next_and
vmlinux 0xffffffffae386a45
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x35
14
? load_balance
vmlinux 0xffffffffae0e8eee
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x1ee
15
? cpumask_next_and
vmlinux 0xffffffffae386a45
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x35
16
xfs_vm_writepages
xfs 0xffffffffc054125b
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffffae1cb9b1
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffffae27df10
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x40
19
writeback_sb_inodes
vmlinux 0xffffffffae27ea94
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffffae27ed9f
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x9f
21
wb_writeback
vmlinux 0xffffffffae27f273
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x263
22
? bdi_dirty_limit
vmlinux 0xffffffffae1cb120
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x40
23
bdi_writeback_workfn
vmlinux 0xffffffffae27fd6c
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x1cc
24
process_one_work
vmlinux 0xffffffffae0bde8f
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x17f
25
worker_thread
vmlinux 0xffffffffae0befa6
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x126
26
? manage_workers.isra.26
vmlinux 0xffffffffae0bee80
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x2a0
27
kthread
vmlinux 0xffffffffae0c5e61
Build id: 3.10.0-1160.49.1.el7.x86_64 		0xd1
28
? insert_kthread_work
vmlinux 0xffffffffae0c5d90
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffffae795df7
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x21
30
? insert_kthread_work
vmlinux 0xffffffffae0c5d90
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u256:1:86445]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel btusb snd_ens1371 lrw gf128mul snd_rawmidi snd_ac97_codec
btrtl glue_helper btbcm btintel bluetooth ablk_helper ac97_bus snd_seq rfkill snd_seq_device cryptd snd_pcm snd_timer snd soundcore pcspkr ppdev joydev vmw_balloon sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul drm crct10dif_common crc32c_intel nfit libata libnvdimm e1000 mptspi scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 1 PID: 86445 Comm: kworker/u256:1 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff8f6eaa5de300 ti: ffff8f6eb8ccc000 task.ti: ffff8f6eb8ccc000
RIP: 0010:[<ffffffffae1170da>]  [<ffffffffae1170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8f6eb8ccf830  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00080000ae07ddd0 RCX: ffff8f6efd0a0600
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8f6eb8ccf868 R08: ffff8f6a3005c000 R09: ffffffffae386bf9
R10: ffff8f6efd05f160 R11: ffffd092d8d92200 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8f6eb8ccf7e0 R15: 0000000300000001
FS:  0000000000000000(0000) GS:ffff8f6efd040000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f24a9c61000 CR3: 00000005e9e7e000 CR4: 00000000001607e0
Call Trace:
[<ffffffffae07e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffae07e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffae208108>] ptep_clear_flush+0x68/0xa0
[<ffffffffae2016a6>] page_mkclean_one+0xa6/0xe0
[<ffffffffae2034b3>] rmap_walk+0xf3/0x360
[<ffffffffae2038c6>] page_mkclean+0x76/0xb0
[<ffffffffae201600>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffae200ce0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffae1ca6c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffae1ca932>] write_cache_pages+0x1f2/0x470
[<ffffffffc0541290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffae386a45>] ? cpumask_next_and+0x35/0x50
[<ffffffffae0e8eee>] ? load_balance+0x1ee/0xa10
[<ffffffffae386a45>] ? cpumask_next_and+0x35/0x50
[<ffffffffc054125b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffae1cb9b1>] do_writepages+0x21/0x50
[<ffffffffae27df10>] __writeback_single_inode+0x40/0x260
[<ffffffffae27ea94>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffae27ed9f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffae27f273>] wb_writeback+0x263/0x2f0
[<ffffffffae1cb120>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffffae27fd6c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffffae0bde8f>] process_one_work+0x17f/0x440
[<ffffffffae0befa6>] worker_thread+0x126/0x3c0
[<ffffffffae0bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffae0c5e61>] kthread+0xd1/0xe0
[<ffffffffae0c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffae795df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffae0c5d90>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 d5 ae f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #5648
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff879150ce
Build id: 3.10.0-1062.el7.x86_64 		0x20e
2
native_flush_tlb_others
vmlinux 0xffffffff8787d398
Build id: 3.10.0-1062.el7.x86_64 		0xb8
3
flush_tlb_page
vmlinux 0xffffffff8787d534
Build id: 3.10.0-1062.el7.x86_64 		0x54
4
ptep_clear_flush
vmlinux 0xffffffff87a03b18
Build id: 3.10.0-1062.el7.x86_64 		0x68
5
page_mkclean_one
vmlinux 0xffffffff879fd0a6
Build id: 3.10.0-1062.el7.x86_64 		0xa6
6
rmap_walk
vmlinux 0xffffffff879feed3
Build id: 3.10.0-1062.el7.x86_64 		0xf3
7
page_mkclean
vmlinux 0xffffffff879ff2e6
Build id: 3.10.0-1062.el7.x86_64 		0x76
8
? page_referenced_one
vmlinux 0xffffffff879fd000
Build id: 3.10.0-1062.el7.x86_64 		0x1b0
9
? percpu_flush_tlb_batch_pages
vmlinux 0xffffffff879fc6e0
Build id: 3.10.0-1062.el7.x86_64 		0x20
10
clear_page_dirty_for_io
vmlinux 0xffffffff879c7a50
Build id: 3.10.0-1062.el7.x86_64 		0x50
11
write_cache_pages
vmlinux 0xffffffff879c7cc2
Build id: 3.10.0-1062.el7.x86_64 		0x1f2
12
? xfs_vm_writepages
xfs 0xffffffffc032b030
Build id: 3.10.0-1062.el7.x86_64 		0xa0
13
? submit_bio
vmlinux 0xffffffff87b4eea0
Build id: 3.10.0-1062.el7.x86_64 		0x70
14
xfs_vm_writepages
xfs 0xffffffffc032affb
Build id: 3.10.0-1062.el7.x86_64 		0x6b
15
do_writepages
vmlinux 0xffffffff879c8d31
Build id: 3.10.0-1062.el7.x86_64 		0x21
16
__writeback_single_inode
vmlinux 0xffffffff87a776c0
Build id: 3.10.0-1062.el7.x86_64 		0x40
17
? wake_up_bit
vmlinux 0xffffffff878c6185
Build id: 3.10.0-1062.el7.x86_64 		0x25
18
writeback_sb_inodes
vmlinux 0xffffffff87a78254
Build id: 3.10.0-1062.el7.x86_64 		0x1c4
19
__writeback_inodes_wb
vmlinux 0xffffffff87a7855f
Build id: 3.10.0-1062.el7.x86_64 		0x9f
20
wb_writeback
vmlinux 0xffffffff87a78a43
Build id: 3.10.0-1062.el7.x86_64 		0x263
21
? bdi_dirty_limit
vmlinux 0xffffffff879c84b0
Build id: 3.10.0-1062.el7.x86_64 		0x40
22
bdi_writeback_workfn
vmlinux 0xffffffff87a7953c
Build id: 3.10.0-1062.el7.x86_64 		0x1cc
23
process_one_work
vmlinux 0xffffffff878bd0ff
Build id: 3.10.0-1062.el7.x86_64 		0x17f
24
worker_thread
vmlinux 0xffffffff878be216
Build id: 3.10.0-1062.el7.x86_64 		0x126
25
? manage_workers.isra.26
vmlinux 0xffffffff878be0f0
Build id: 3.10.0-1062.el7.x86_64 		0x2a0
26
kthread
vmlinux 0xffffffff878c50d1
Build id: 3.10.0-1062.el7.x86_64 		0xd1
27
? insert_kthread_work
vmlinux 0xffffffff878c5000
Build id: 3.10.0-1062.el7.x86_64 		0x40
28
ret_from_fork_nospec_begin
vmlinux 0xffffffff87f8bd37
Build id: 3.10.0-1062.el7.x86_64 		0x21
29
? insert_kthread_work
vmlinux 0xffffffff878c5000
Build id: 3.10.0-1062.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u8:0:11843]
Modules linked in: tcp_lp fuse uinput xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun bridge stp llc devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ppdev crc32_pclmul snd_hda_codec_generic ghash_clmulni_intel snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device aesni_intel snd_pcm lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr joydev virtio_rng virtio_balloon snd_timer parport_pc snd parport
soundcore i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom virtio_console virtio_net virtio_blk qxl drm_kms_helper ata_generic pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata floppy virtio_pci serio_raw virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 11843 Comm: kworker/u8:0 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9d9f54885230 ti: ffff9da2df7d8000 task.ti: ffff9da2df7d8000
RIP: 0010:[<ffffffff879150ce>]  [<ffffffff879150ce>] smp_call_function_many+0x20e/0x270
RSP: 0000:ffff9da2df7db830  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 0000000000000004 RCX: ffff9da2ffd9fc10
RDX: 0000000000000003 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff9da2df7db868 R08: ffff9da03fd52000 R09: ffffffff87b7f629
R10: ffff9da2ffc1f160 R11: ffffe212cfb9d400 R12: 000000000001b7c0
R13: ffffffff8787d1d0 R14: ffff9da2df7db878 R15: ffff9da2ffc1b800
FS:  0000000000000000(0000) GS:ffff9da2ffc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000a8a0d000 CR3: 0000000235010000 CR4: 00000000000006f0
Call Trace:
[<ffffffff8787d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8787d534>] flush_tlb_page+0x54/0xa0
[<ffffffff87a03b18>] ptep_clear_flush+0x68/0xa0
[<ffffffff879fd0a6>] page_mkclean_one+0xa6/0xe0
[<ffffffff879feed3>] rmap_walk+0xf3/0x360
[<ffffffff879ff2e6>] page_mkclean+0x76/0xb0
[<ffffffff879fd000>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff879fc6e0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff879c7a50>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff879c7cc2>] write_cache_pages+0x1f2/0x470
[<ffffffffc032b030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff87b4eea0>] ? submit_bio+0x70/0x150
[<ffffffffc032affb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff879c8d31>] do_writepages+0x21/0x50
[<ffffffff87a776c0>] __writeback_single_inode+0x40/0x260
[<ffffffff878c6185>] ? wake_up_bit+0x25/0x30
[<ffffffff87a78254>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff87a7855f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff87a78a43>] wb_writeback+0x263/0x2f0
[<ffffffff879c84b0>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffff87a7953c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffff878bd0ff>] process_one_work+0x17f/0x440
[<ffffffff878be216>] worker_thread+0x126/0x3c0
[<ffffffff878be0f0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff878c50d1>] kthread+0xd1/0xe0
[<ffffffff878c5000>] ? insert_kthread_work+0x40/0x40
[<ffffffff87f8bd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff878c5000>] ? insert_kthread_work+0x40/0x40
Code: 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 54 88 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee