Complete report #1326667
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff89116c0a
Build id: 3.10.0-1160.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8907e868
Build id: 3.10.0-1160.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8907ea04
Build id: 3.10.0-1160.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff89207728
Build id: 3.10.0-1160.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff89200cd6
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff89202ae3
Build id: 3.10.0-1160.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff89202ef6
Build id: 3.10.0-1160.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff89200c30
Build id: 3.10.0-1160.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff89200310
Build id: 3.10.0-1160.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff891c9af0
Build id: 3.10.0-1160.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff891c9d62
Build id: 3.10.0-1160.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc0605290
Build id: 3.10.0-1160.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff890e23a9
Build id: 3.10.0-1160.el7.x86_64
|
0xa9
|
|
14
|
|
vmlinux |
0xffffffff890e98ce
Build id: 3.10.0-1160.el7.x86_64
|
0x61e
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc060525b
Build id: 3.10.0-1160.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffff891cade1
Build id: 3.10.0-1160.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffff8927d6f0
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8927e274
Build id: 3.10.0-1160.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8927e57f
Build id: 3.10.0-1160.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffff8927ea53
Build id: 3.10.0-1160.el7.x86_64
|
0x263
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8927f54c
Build id: 3.10.0-1160.el7.x86_64
|
0x1cc
|
|
22
|
process_one_work
|
vmlinux |
0xffffffff890bdc4f
Build id: 3.10.0-1160.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffff890bed66
Build id: 3.10.0-1160.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffff890bec40
Build id: 3.10.0-1160.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffff890c5c21
Build id: 3.10.0-1160.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffff890c5b50
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff89793df7
Build id: 3.10.0-1160.el7.x86_64
|
0x21
|
|
28
|
|
vmlinux |
0xffffffff890c5b50
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 21s! [kworker/u256:1:53616]
Modules linked in: xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype nf_conntrack_netlink iptable_filter tcp_diag inet_diag xt_nat veth br_netfilter tun devlink overlay(T) bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev snd_ens1371 vmw_balloon snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device sb_edac snd_pcm iosf_mbi crc32_pclmul ghash_clmulni_intel snd_timer snd aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
pata_acpi ttm drm mptspi nfit crct10dif_pclmul scsi_transport_spi crct10dif_common ata_piix libnvdimm mptscsih crc32c_intel serio_raw libata e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: iptable_filter]
CPU: 4 PID: 53616 Comm: kworker/u256:1 Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
task: ffff8bb9c3d5b180 ti: ffff8bb9c1efc000 task.ti: ffff8bb9c1efc000
RIP: 0010:[<ffffffff89116c0a>] [<ffffffff89116c0a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8bb9c1eff830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0008000000000010 RCX: ffff8bc0fd620678
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8bb9c1eff868 R08: ffff8ba28677bc00 R09: ffffffff89385f99
R10: ffff8bc0fd71f160 R11: ffffe250620a9e00 R12: 0000000000000286
R13: 0000000000000000 R14: ffff8bb9c1eff7a0 R15: 0000000000000286
FS: 0000000000000000(0000) GS:ffff8bc0fd700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7a1411e650 CR3: 000000012fbfc000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8907e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8907ea04>] flush_tlb_page+0x54/0xa0
[<ffffffff89207728>] ptep_clear_flush+0x68/0xa0
[<ffffffff89200cd6>] page_mkclean_one+0xa6/0xe0
[<ffffffff89202ae3>] rmap_walk+0xf3/0x360
[<ffffffff89202ef6>] page_mkclean+0x76/0xb0
[<ffffffff89200c30>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff89200310>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff891c9af0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff891c9d62>] write_cache_pages+0x1f2/0x470
[<ffffffffc0605290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff890e23a9>] ? pick_next_entity+0xa9/0x190
[<ffffffff890e98ce>] ? pick_next_task_fair+0x61e/0x870
[<ffffffffc060525b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff891cade1>] do_writepages+0x21/0x50
[<ffffffff8927d6f0>] __writeback_single_inode+0x40/0x260
[<ffffffff8927e274>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8927e57f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8927ea53>] wb_writeback+0x263/0x2f0
[<ffffffff8927f54c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffff890bdc4f>] process_one_work+0x17f/0x440
[<ffffffff890bed66>] worker_thread+0x126/0x3c0
[<ffffffff890bec40>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff890c5c21>] kthread+0xd1/0xe0
[<ffffffff890c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff89793df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff890c5b50>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 d5 89 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1294329
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa4916c0a
Build id: 3.10.0-1160.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa487e868
Build id: 3.10.0-1160.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffa487ea04
Build id: 3.10.0-1160.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffa4a07728
Build id: 3.10.0-1160.el7.x86_64
|
0x68
|
|
5
|
|
vmlinux |
0xffffffffa4a076c0
Build id: 3.10.0-1160.el7.x86_64
|
0x70
|
|
6
|
page_mkclean_one
|
vmlinux |
0xffffffffa4a00cd6
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
|
7
|
rmap_walk
|
vmlinux |
0xffffffffa4a02ae3
Build id: 3.10.0-1160.el7.x86_64
|
0xf3
|
|
8
|
page_mkclean
|
vmlinux |
0xffffffffa4a02ef6
Build id: 3.10.0-1160.el7.x86_64
|
0x76
|
|
9
|
|
vmlinux |
0xffffffffa4a00c30
Build id: 3.10.0-1160.el7.x86_64
|
0x1b0
|
|
10
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffa4a00310
Build id: 3.10.0-1160.el7.x86_64
|
0x20
|
|
11
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffa49c9af0
Build id: 3.10.0-1160.el7.x86_64
|
0x50
|
|
12
|
write_cache_pages
|
vmlinux |
0xffffffffa49c9d62
Build id: 3.10.0-1160.el7.x86_64
|
0x1f2
|
|
13
|
|
xfs |
0xffffffffc0714290
Build id: 3.10.0-1160.el7.x86_64
|
0xa0
|
|
14
|
|
vmlinux |
0xffffffffa4b55410
Build id: 3.10.0-1160.el7.x86_64
|
0x70
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc071425b
Build id: 3.10.0-1160.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffffa49cade1
Build id: 3.10.0-1160.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffffa4a7d6f0
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
|
18
|
|
vmlinux |
0xffffffffa48c6cd5
Build id: 3.10.0-1160.el7.x86_64
|
0x25
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffffa4a7e274
Build id: 3.10.0-1160.el7.x86_64
|
0x1c4
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffa4a7e57f
Build id: 3.10.0-1160.el7.x86_64
|
0x9f
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffffa4a7ea53
Build id: 3.10.0-1160.el7.x86_64
|
0x263
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffa4a7f64b
Build id: 3.10.0-1160.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffa48bdc4f
Build id: 3.10.0-1160.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffa48bed66
Build id: 3.10.0-1160.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffa48bec40
Build id: 3.10.0-1160.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffffa48c5c21
Build id: 3.10.0-1160.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffa48c5b50
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffa4f93df7
Build id: 3.10.0-1160.el7.x86_64
|
0x21
|
|
29
|
|
vmlinux |
0xffffffffa48c5b50
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [kworker/u256:0:60483]
Modules linked in: tcp_diag inet_diag xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev vmw_balloon snd_ens1371 snd_rawmidi snd_ac97_codec sb_edac ac97_bus snd_seq iosf_mbi snd_seq_device crc32_pclmul snd_pcm
ghash_clmulni_intel aesni_intel snd_timer lrw gf128mul glue_helper ablk_helper cryptd sg snd pcspkr soundcore joydev vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ata_piix nfit ttm mptspi scsi_transport_spi drm mptscsih libata libnvdimm crct10dif_pclmul mptbase crct10dif_common e1000 crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 22 PID: 60483 Comm: kworker/u256:0 Kdump: loaded Tainted: G W L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
task: ffff8a0abc62a100 ti: ffff8a0b5a1a0000 task.ti: ffff8a0b5a1a0000
RIP: 0010:[<ffffffffa4916c0a>] [<ffffffffa4916c0a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8a0b5a1a3830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a487e6a0 RCX: ffff8a1efd620948
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8a0b5a1a3868 R08: ffff8a008d748400 R09: ffffffffa4b85f99
R10: ffff8a1efdb9f160 R11: ffffe5f844c4f000 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8a0b5a1a37e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff8a1efdb80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007efefe6dd000 CR3: 0000000142782000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa487e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa487ea04>] flush_tlb_page+0x54/0xa0
[<ffffffffa4a07728>] ptep_clear_flush+0x68/0xa0
[<ffffffffa4a076c0>] ? walk_page_vma+0x70/0x70
[<ffffffffa4a00cd6>] page_mkclean_one+0xa6/0xe0
[<ffffffffa4a02ae3>] rmap_walk+0xf3/0x360
[<ffffffffa4a02ef6>] page_mkclean+0x76/0xb0
[<ffffffffa4a00c30>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffa4a00310>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffa49c9af0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa49c9d62>] write_cache_pages+0x1f2/0x470
[<ffffffffc0714290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffa4b55410>] ? submit_bio+0x70/0x150
[<ffffffffc071425b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa49cade1>] do_writepages+0x21/0x50
[<ffffffffa4a7d6f0>] __writeback_single_inode+0x40/0x260
[<ffffffffa48c6cd5>] ? wake_up_bit+0x25/0x30
[<ffffffffa4a7e274>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffa4a7e57f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa4a7ea53>] wb_writeback+0x263/0x2f0
[<ffffffffa4a7f64b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa48bdc4f>] process_one_work+0x17f/0x440
[<ffffffffa48bed66>] worker_thread+0x126/0x3c0
[<ffffffffa48bec40>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffa48c5c21>] kthread+0xd1/0xe0
[<ffffffffa48c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffa4f93df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa48c5b50>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 55 a5 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #768063
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8b9150ce
Build id: 3.10.0-1062.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8b87d398
Build id: 3.10.0-1062.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8b87d534
Build id: 3.10.0-1062.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff8ba03b18
Build id: 3.10.0-1062.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff8b9fd0a6
Build id: 3.10.0-1062.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff8b9feed3
Build id: 3.10.0-1062.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff8b9ff2e6
Build id: 3.10.0-1062.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff8b9fd000
Build id: 3.10.0-1062.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff8b9fc6e0
Build id: 3.10.0-1062.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff8b9c7a50
Build id: 3.10.0-1062.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff8b9c7cc2
Build id: 3.10.0-1062.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc06b9030
Build id: 3.10.0-1062.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff8bb4eea0
Build id: 3.10.0-1062.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffff8b82b59e
Build id: 3.10.0-1062.el7.x86_64
|
0xce
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc06b8ffb
Build id: 3.10.0-1062.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffff8b9c8d31
Build id: 3.10.0-1062.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffff8ba776c0
Build id: 3.10.0-1062.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8ba78254
Build id: 3.10.0-1062.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8ba7855f
Build id: 3.10.0-1062.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffff8ba78a43
Build id: 3.10.0-1062.el7.x86_64
|
0x263
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8ba7963b
Build id: 3.10.0-1062.el7.x86_64
|
0x2cb
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 24s! [kworker/u256:3:100948]
Modules linked in: binfmt_misc nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq ppdev snd_seq_device snd_pcm vmw_balloon snd_timer snd iosf_mbi crc32_pclmul ghash_clmulni_intel sg soundcore aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit libnvdimm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel mptspi libata scsi_transport_spi e1000 mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 100948 Comm: kworker/u256:3 Kdump: loaded Tainted: G L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff93db6966c1c0 ti: ffff93d9be658000 task.ti: ffff93d9be658000
RIP: 0010:[<ffffffff8b9150ce>] [<ffffffff8b9150ce>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff93d9be65b830 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000800008b87d1d0 RCX: ffff93dc3d6e0628
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff93d9be65b868 R08: ffff93cda9664000 R09: ffffffff8bb7f629
R10: ffff93dc3d71f160 R11: fffffd1076d3c400 R12: 0000000000000286
R13: 0000000000000286 R14: ffff93d9be65b7e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff93dc3d700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000f723463c000 CR3: 0000000fe03f8000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffff8b87d398>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8b87d534>] flush_tlb_page+0x54/0xa0
[<ffffffff8ba03b18>] ptep_clear_flush+0x68/0xa0
[<ffffffff8b9fd0a6>] page_mkclean_one+0xa6/0xe0
[<ffffffff8b9feed3>] rmap_walk+0xf3/0x360
[<ffffffff8b9ff2e6>] page_mkclean+0x76/0xb0
[<ffffffff8b9fd000>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff8b9fc6e0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff8b9c7a50>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8b9c7cc2>] write_cache_pages+0x1f2/0x470
[<ffffffffc06b9030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff8bb4eea0>] ? submit_bio+0x70/0x150
[<ffffffff8b82b59e>] ? __switch_to+0xce/0x580
[<ffffffffc06b8ffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff8b9c8d31>] do_writepages+0x21/0x50
[<ffffffff8ba776c0>] __writeback_single_inode+0x40/0x260
[<ffffffff8ba78254>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8ba7855f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8ba78a43>] wb_writeback+0x263/0x2f0
[<ffffffff8ba7963b>] bdi_writeback_workfn+0x2cb/0x460
Complete report #1244606
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9311e25a
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff93081fa8
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff93082144
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff93214278
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff9320d64e
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xae
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff9320f4c3
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff9320f8e6
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff9320d5a0
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff9320cc70
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff931d57c0
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff931d5a32
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x1f2
|
|
12
|
|
vmlinux |
0xffffffff933687d9
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x39
|
|
13
|
|
xfs |
0xffffffffc05026c0
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xb0
|
|
14
|
|
vmlinux |
0xffffffff9336a537
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x157
|
|
15
|
|
vmlinux |
0xffffffff9339c615
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x35
|
|
16
|
xfs_vm_writepages
|
xfs |
0xffffffffc050267b
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x6b
|
|
17
|
do_writepages
|
vmlinux |
0xffffffff931d6b01
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x21
|
|
18
|
__writeback_single_inode
|
vmlinux |
0xffffffff9328cd40
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x40
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffff9328d90c
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x1cc
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff9328dc17
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xa7
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffff9328e0f3
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x263
|
|
22
|
|
vmlinux |
0xffffffff932790bc
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x4c
|
|
23
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff9328ecff
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x2cf
|
|
24
|
process_one_work
|
vmlinux |
0xffffffff930c32ef
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x17f
|
|
25
|
worker_thread
|
vmlinux |
0xffffffff930c4436
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x126
|
|
26
|
|
vmlinux |
0xffffffff930c4310
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x2b0
|
|
27
|
kthread
|
vmlinux |
0xffffffff930cb621
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xd1
|
|
28
|
|
vmlinux |
0xffffffff930cb550
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x40
|
|
29
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff937c51dd
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x7
|
|
30
|
|
vmlinux |
0xffffffff930cb550
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kworker/u256:0:27656]
Modules linked in: tcp_lp veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq sg pcspkr snd_seq_device joydev snd_pcm snd_timer ppdev snd soundcore crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix libnvdimm libata mptspi serio_raw scsi_transport_spi mptscsih e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 7 PID: 27656 Comm: kworker/u256:0 Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9c52c3326300 ti: ffff9c579bab8000 task.ti: ffff9c579bab8000
RIP: 0010:[<ffffffff9311e25a>] [<ffffffff9311e25a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9c579babb830 EFLAGS: 00000202
RAX: 0000000000000002 RBX: 0008000093081b10 RCX: ffff9c579d6a06f0
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9c579babb868 R08: ffff9c50ec55d000 R09: ffffffff9339c7f9
R10: ffff9c579d7df160 R11: ffffe5c61f9f6e00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9c579babb7e0 R15: ffffffff931cc4ae
FS: 0000000000000000(0000) GS:ffff9c579d7c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f637819d1c0 CR3: 00000003d3810000 CR4: 0000000000760fe0
PKRU: 00000000
Call Trace:
[<ffffffff93081fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff93082144>] flush_tlb_page+0x54/0xa0
[<ffffffff93214278>] ptep_clear_flush+0x68/0xa0
[<ffffffff9320d64e>] page_mkclean_one+0xae/0xe0
[<ffffffff9320f4c3>] rmap_walk+0xf3/0x370
[<ffffffff9320f8e6>] page_mkclean+0x76/0xb0
[<ffffffff9320d5a0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff9320cc70>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff931d57c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff931d5a32>] write_cache_pages+0x1f2/0x480
[<ffffffff933687d9>] ? __blk_run_queue+0x39/0x50
[<ffffffffc05026c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffff9336a537>] ? generic_make_request+0x157/0x390
[<ffffffff9339c615>] ? cpumask_next_and+0x35/0x50
[<ffffffffc050267b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffff931d6b01>] do_writepages+0x21/0x50
[<ffffffff9328cd40>] __writeback_single_inode+0x40/0x260
[<ffffffff9328d90c>] writeback_sb_inodes+0x1cc/0x430
[<ffffffff9328dc17>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffff9328e0f3>] wb_writeback+0x263/0x2f0
[<ffffffff932790bc>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9328ecff>] bdi_writeback_workfn+0x2cf/0x470
[<ffffffff930c32ef>] process_one_work+0x17f/0x440
[<ffffffff930c4436>] worker_thread+0x126/0x3c0
[<ffffffff930c4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff930cb621>] kthread+0xd1/0xe0
[<ffffffff930cb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff937c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff930cb550>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ae 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 19 d5 93 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 7d 18 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1159584
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
native_queued_spin_lock_slowpath
|
vmlinux |
0xffffffff94f1776d
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x1d
|
|
2
|
queued_spin_lock_slowpath
|
vmlinux |
0xffffffff9557afe5
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xb
|
|
3
|
_raw_spin_lock
|
vmlinux |
0xffffffff955896f0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x20
|
|
4
|
__page_check_address
|
vmlinux |
0xffffffff95000b12
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x102
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff95000e54
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x44
|
|
6
|
|
vmlinux |
0xffffffff952ec2a0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x170
|
|
7
|
rmap_walk
|
vmlinux |
0xffffffff95002cc3
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xf3
|
|
8
|
page_mkclean
|
vmlinux |
0xffffffff950030d6
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x76
|
|
9
|
|
vmlinux |
0xffffffff95000e10
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x1b0
|
|
10
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff950004f0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x20
|
|
11
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff94fc9cc0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x50
|
|
12
|
write_cache_pages
|
vmlinux |
0xffffffff94fc9f32
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x1f2
|
|
13
|
|
xfs |
0xffffffffc03bd290
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xa0
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc03bd25b
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffff94fcafb1
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffff9507d8d0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x40
|
|
17
|
|
vmlinux |
0xffffffff94ec6f25
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x25
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffff9507e454
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff9507e75f
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffff9507ec33
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x263
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff9507f82b
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x2cb
|
|
22
|
process_one_work
|
vmlinux |
0xffffffff94ebde9f
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffff94ebefb6
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffff94ebee90
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffff94ec5e71
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffff94ec5da0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff95593ddd
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x7
|
|
28
|
|
vmlinux |
0xffffffff94ec5da0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/u33:1:16643]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(E) falcon_kal(E) falcon_lsm_pinned_17507(E) binfmt_misc falcon_lsm_pinned_17405(E) falcon_lsm_pinned_17306(E) falcon_lsm_pinned_17219(E) falcon_lsm_pinned_17129(E) falcon_lsm_pinned_17011(E) falcon_lsm_pinned_17005(E) falcon_lsm_pinned_16903(E) falcon_lsm_pinned_16803(E) falcon_lsm_pinned_16703(E) tcp_diag udp_diag inet_diag falcon_lsm_pinned_16604(E) vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon pcspkr nfit sg joydev libnvdimm vmw_vmci parport_pc i2c_piix4 parport auth_rpcgss sunrpc ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect crc_t10dif sysimgblt
crct10dif_generic fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal]
CPU: 3 PID: 16643 Comm: kworker/u33:1 Kdump: loaded Tainted: P EL ------------ 3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
task: ffff8dd194e40000 ti: ffff8dd0833dc000 task.ti: ffff8dd0833dc000
RIP: 0010:[<ffffffff94f1776d>] [<ffffffff94f1776d>] native_queued_spin_lock_slowpath+0x1d/0x200
RSP: 0000:ffff8dd0833df8b8 EFLAGS: 00000293
RAX: 0000000000000001 RBX: ffff8dd1856bb100 RCX: 0000000000000001
RDX: 0000000000000001 RSI: 0000000000000001 RDI: fffff834090bebf0
RBP: ffff8dd0833df8b8 R08: 0000000000000000 R09: 0000000242faf000
R10: 0000000000000400 R11: 0000000000000000 R12: ffff8dd08ab33300
R13: ffff8dd1856bb1d8 R14: 00000000d7790f6d R15: ffff8dcf8373dfe0
FS: 0000000000000000(0000) GS:ffff8dd1b7cc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fac0e76cf00 CR3: 000000030d810000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9557afe5>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff955896f0>] _raw_spin_lock+0x20/0x30
[<ffffffff95000b12>] __page_check_address+0x102/0x250
[<ffffffff95000e54>] page_mkclean_one+0x44/0xe0
[<ffffffff952ec2a0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffff95002cc3>] rmap_walk+0xf3/0x360
[<ffffffff950030d6>] page_mkclean+0x76/0xb0
[<ffffffff95000e10>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff950004f0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff94fc9cc0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff94fc9f32>] write_cache_pages+0x1f2/0x470
[<ffffffffc03bd290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffc03bd25b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff94fcafb1>] do_writepages+0x21/0x50
[<ffffffff9507d8d0>] __writeback_single_inode+0x40/0x260
[<ffffffff94ec6f25>] ? wake_up_bit+0x25/0x30
[<ffffffff9507e454>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff9507e75f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff9507ec33>] wb_writeback+0x263/0x2f0
[<ffffffff9507f82b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff94ebde9f>] process_one_work+0x17f/0x440
[<ffffffff94ebefb6>] worker_thread+0x126/0x3c0
[<ffffffff94ebee90>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff94ec5e71>] kthread+0xd1/0xe0
[<ffffffff94ec5da0>] ? insert_kthread_work+0x40/0x40
[<ffffffff95593ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff94ec5da0>] ? insert_kthread_work+0x40/0x40
Complete report #1147950
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8ff1173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8fe7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8fe7a734
Build id: 3.10.0-957.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff8fffd587
Build id: 3.10.0-957.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff8fff6b66
Build id: 3.10.0-957.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff8fff8983
Build id: 3.10.0-957.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff8fff8d96
Build id: 3.10.0-957.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff8fff6ac0
Build id: 3.10.0-957.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff8fff6200
Build id: 3.10.0-957.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff8ffc1d60
Build id: 3.10.0-957.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff8ffc200a
Build id: 3.10.0-957.el7.x86_64
|
0x22a
|
|
12
|
|
xfs |
0xffffffffc04d65b0
Build id: 3.10.0-957.el7.x86_64
|
0x150
|
|
13
|
xfs_vm_writepages
|
xfs |
0xffffffffc04d63ab
Build id: 3.10.0-957.el7.x86_64
|
0x6b
|
|
14
|
do_writepages
|
vmlinux |
0xffffffff8ffc3181
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
15
|
__writeback_single_inode
|
vmlinux |
0xffffffff9006f930
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
16
|
writeback_sb_inodes
|
vmlinux |
0xffffffff900703c4
Build id: 3.10.0-957.el7.x86_64
|
0x1c4
|
|
17
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff9007072f
Build id: 3.10.0-957.el7.x86_64
|
0x9f
|
|
18
|
wb_writeback
|
vmlinux |
0xffffffff90070f63
Build id: 3.10.0-957.el7.x86_64
|
0x263
|
|
19
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff900718eb
Build id: 3.10.0-957.el7.x86_64
|
0x2cb
|
|
20
|
process_one_work
|
vmlinux |
0xffffffff8feb9d4f
Build id: 3.10.0-957.el7.x86_64
|
0x17f
|
|
21
|
worker_thread
|
vmlinux |
0xffffffff8febade6
Build id: 3.10.0-957.el7.x86_64
|
0x126
|
|
22
|
|
vmlinux |
0xffffffff8febacc0
Build id: 3.10.0-957.el7.x86_64
|
0x2a0
|
|
23
|
kthread
|
vmlinux |
0xffffffff8fec1c31
Build id: 3.10.0-957.el7.x86_64
|
0xd1
|
|
24
|
|
vmlinux |
0xffffffff8fec1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
25
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff90574c1d
Build id: 3.10.0-957.el7.x86_64
|
0x7
|
|
26
|
|
vmlinux |
0xffffffff8fec1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#15 stuck for 23s! [kworker/u48:1:5199]
Modules linked in: binfmt_misc tcp_lp fuse uinput xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter sunrpc nfit libnvdimm snd_hda_codec_generic ppdev iosf_mbi crc32_pclmul snd_hda_intel ghash_clmulni_intel snd_hda_codec aesni_intel snd_hda_core snd_hwdep lrw gf128mul snd_seq snd_seq_device glue_helper ablk_helper snd_pcm cryptd snd_timer parport_pc parport snd soundcore joydev pcspkr sg i2c_piix4 virtio_balloon ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_blk virtio_console virtio_net qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
ttm floppy drm ata_piix crct10dif_pclmul libata crct10dif_common crc32c_intel serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 15 PID: 5199 Comm: kworker/u48:1 Kdump: loaded Tainted: G L ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: writeback bdi_writeback_workfn (flush-252:16)
task: ffff988f19e56180 ti: ffff989085d2c000 task.ti: ffff989085d2c000
RIP: 0010:[<ffffffff8ff1173a>] [<ffffffff8ff1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff989085d2f820 EFLAGS: 00000202
RAX: 0000000000000016 RBX: 000800008fe7a3d0 RCX: ffff98ae3f5a0618
RDX: 0000000000000016 RSI: 0000000000000018 RDI: 0000000000000000
RBP: ffff989085d2f858 R08: ffff988fd30fac00 R09: ffffffff901755c9
R10: ffff98ae3f3df160 R11: fffff4e94c9ee200 R12: 0000000000000286
R13: 0000000000000286 R14: ffff989085d2f7d0 R15: ffff9891a7b89400
FS: 0000000000000000(0000) GS:ffff98ae3f3c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5d249f8010 CR3: 0000000553010000 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8fe7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8fe7a734>] flush_tlb_page+0x54/0xa0
[<ffffffff8fffd587>] ptep_clear_flush+0x67/0xa0
[<ffffffff8fff6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffff8fff8983>] rmap_walk+0xf3/0x360
[<ffffffff8fff8d96>] page_mkclean+0x76/0xb0
[<ffffffff8fff6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffff8fff6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff8ffc1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8ffc200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc04d65b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffc04d63ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff8ffc3181>] do_writepages+0x21/0x50
[<ffffffff9006f930>] __writeback_single_inode+0x40/0x260
[<ffffffff900703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff9007072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff90070f63>] wb_writeback+0x263/0x2f0
[<ffffffff900718eb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff8feb9d4f>] process_one_work+0x17f/0x440
[<ffffffff8febade6>] worker_thread+0x126/0x3c0
[<ffffffff8febacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff8fec1c31>] kthread+0xd1/0xe0
[<ffffffff8fec1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff90574c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff8fec1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 b4 90 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1147646
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffaf5170da
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffaf47e258
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffaf47e3f4
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffaf608108
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffaf6016a6
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffaf6034b3
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffaf6038c6
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffaf601600
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffaf600ce0
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffaf5ca6c0
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffaf5ca932
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc067c290
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffaf755e70
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffffaf786a45
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x35
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc067c25b
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffffaf5cb9b1
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffffaf67df10
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffffaf67ea94
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffaf67ed9f
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffffaf67f273
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffffaf66ae6c
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffaf67fe6b
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffaf4bde8f
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffaf4befa6
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffaf4bee80
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffffaf4c5e61
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffaf4c5d90
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffafb95de4
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0xe
|
|
29
|
|
vmlinux |
0xffffffffaf4c5d90
Build id: 3.10.0-1160.53.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u66:1:13161]
Modules linked in: bmsensor(OE) tmhook(OE) ip6table_filter ip6_tables iptable_filter tcp_diag inet_diag binfmt_misc vmw_vsock_vmci_transport vsock crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg vmw_vmci parport_pc parport i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm nfit crct10dif_pclmul libata mptspi crct10dif_common libnvdimm crc32c_intel scsi_transport_spi serio_raw mptscsih mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: bmsensor]
CPU: 5 PID: 13161 Comm: kworker/u66:1 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9cf3a3f39080 ti: ffff9cf380c3c000 task.ti: ffff9cf380c3c000
RIP: 0010:[<ffffffffaf5170da>] [<ffffffffaf5170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9cf380c3f830 EFLAGS: 00000202
RAX: 0000000000000013 RBX: 00080000af47ddd0 RCX: ffff9cf77bda0978
RDX: 0000000000000013 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff9cf380c3f868 R08: ffff9cf3ffc09c00 R09: ffffffffaf786bf9
R10: ffff9cf47bc9f160 R11: fffff2ea4a040c00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9cf380c3f7e0 R15: 0000000000000000
FS: 00007f6a064fe700(0000) GS:ffff9cf47bc80000(0000) knlGS:000000000a029840
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2f5c03f2c8 CR3: 000000002bc10000 CR4: 00000000003407e0
Call Trace:
[<ffffffffaf47e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaf47e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffaf608108>] ptep_clear_flush+0x68/0xa0
[<ffffffffaf6016a6>] page_mkclean_one+0xa6/0xe0
[<ffffffffaf6034b3>] rmap_walk+0xf3/0x360
[<ffffffffaf6038c6>] page_mkclean+0x76/0xb0
[<ffffffffaf601600>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffaf600ce0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffaf5ca6c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffaf5ca932>] write_cache_pages+0x1f2/0x470
[<ffffffffc067c290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffaf755e70>] ? submit_bio+0x70/0x150
[<ffffffffaf786a45>] ? cpumask_next_and+0x35/0x50
[<ffffffffc067c25b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffaf5cb9b1>] do_writepages+0x21/0x50
[<ffffffffaf67df10>] __writeback_single_inode+0x40/0x260
[<ffffffffaf67ea94>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffaf67ed9f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffaf67f273>] wb_writeback+0x263/0x2f0
[<ffffffffaf66ae6c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffaf67fe6b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffaf4bde8f>] process_one_work+0x17f/0x440
[<ffffffffaf4befa6>] worker_thread+0x126/0x3c0
[<ffffffffaf4bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffaf4c5e61>] kthread+0xd1/0xe0
[<ffffffffaf4c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffafb95de4>] ret_from_fork_nospec_begin+0xe/0x21
[<ffffffffaf4c5d90>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 15 b0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1088346
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8b316c0a
Build id: 3.10.0-1160.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8b27e868
Build id: 3.10.0-1160.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8b27ea04
Build id: 3.10.0-1160.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff8b407728
Build id: 3.10.0-1160.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff8b400cd6
Build id: 3.10.0-1160.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff8b402ae3
Build id: 3.10.0-1160.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff8b402ef6
Build id: 3.10.0-1160.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff8b400c30
Build id: 3.10.0-1160.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff8b400310
Build id: 3.10.0-1160.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff8b3c9af0
Build id: 3.10.0-1160.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff8b3c9d62
Build id: 3.10.0-1160.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc05a5290
Build id: 3.10.0-1160.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff8b22b59e
Build id: 3.10.0-1160.el7.x86_64
|
0xce
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc05a525b
Build id: 3.10.0-1160.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffff8b3cade1
Build id: 3.10.0-1160.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffff8b47d6f0
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8b47e274
Build id: 3.10.0-1160.el7.x86_64
|
0x1c4
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8b47e57f
Build id: 3.10.0-1160.el7.x86_64
|
0x9f
|
|
19
|
wb_writeback
|
vmlinux |
0xffffffff8b47ea53
Build id: 3.10.0-1160.el7.x86_64
|
0x263
|
|
20
|
|
vmlinux |
0xffffffff8b46a67c
Build id: 3.10.0-1160.el7.x86_64
|
0x4c
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8b47f64b
Build id: 3.10.0-1160.el7.x86_64
|
0x2cb
|
|
22
|
process_one_work
|
vmlinux |
0xffffffff8b2bdc4f
Build id: 3.10.0-1160.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffff8b2bed66
Build id: 3.10.0-1160.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffff8b2bec40
Build id: 3.10.0-1160.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffff8b2c5c21
Build id: 3.10.0-1160.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffff8b2c5b50
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff8b993df7
Build id: 3.10.0-1160.el7.x86_64
|
0x21
|
|
28
|
|
vmlinux |
0xffffffff8b2c5b50
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [kworker/u312:0:69901]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack ipt_REJECT nf_reject_ipv4 rpcsec_gss_krb5 auth_rpcgss tun nfsv4 dns_resolver nfs bridge stp llc lockd grace fscache ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter ip_vs nf_conntrack sunrpc ppdev sg parport_pc parport i2c_piix4 pcspkr virtio_balloon joydev binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover virtio_console virtio_blk virtio_scsi failover cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata virtio_pci virtio_ring serio_raw floppy virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 12 PID: 69901 Comm: kworker/u312:0 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff998ac0389080 ti: ffff997d00dd4000 task.ti: ffff997d00dd4000
RIP: 0010:[<ffffffff8b316c0a>] [<ffffffff8b316c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff997d00dd7830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008b27e6a0 RCX: ffff998dfce205b8
RDX: 0000000000000000 RSI: 000000000000009c RDI: 0000000000000000
RBP: ffff997d00dd7868 R08: ffff996f85b6e400 R09: ffffffff8b585f99
R10: ffff998dfd11f160 R11: fffff5d8c8457a00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff997d00dd77e0 R15: ffff9978e0761884
FS: 0000000000000000(0000) GS:ffff998dfd100000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005569c3135000 CR3: 00000018bec10000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8b27e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8b27ea04>] flush_tlb_page+0x54/0xa0
[<ffffffff8b407728>] ptep_clear_flush+0x68/0xa0
[<ffffffff8b400cd6>] page_mkclean_one+0xa6/0xe0
[<ffffffff8b402ae3>] rmap_walk+0xf3/0x360
[<ffffffff8b402ef6>] page_mkclean+0x76/0xb0
[<ffffffff8b400c30>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff8b400310>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff8b3c9af0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8b3c9d62>] write_cache_pages+0x1f2/0x470
[<ffffffffc05a5290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff8b22b59e>] ? __switch_to+0xce/0x580
[<ffffffffc05a525b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff8b3cade1>] do_writepages+0x21/0x50
[<ffffffff8b47d6f0>] __writeback_single_inode+0x40/0x260
[<ffffffff8b47e274>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8b47e57f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8b47ea53>] wb_writeback+0x263/0x2f0
[<ffffffff8b46a67c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8b47f64b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff8b2bdc4f>] process_one_work+0x17f/0x440
[<ffffffff8b2bed66>] worker_thread+0x126/0x3c0
[<ffffffff8b2bec40>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8b2c5c21>] kthread+0xd1/0xe0
[<ffffffff8b2c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff8b993df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8b2c5b50>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 f5 8b f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1072613
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb19170da
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb187e258
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffb187e3f4
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffb1a08108
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffb1a016a6
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffb1a034b3
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffb1a038c6
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffb1a01600
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffb1a00ce0
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffb19ca6c0
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffb19ca932
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc01a8290
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffb18de305
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x85
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc01a825b
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffffb19cb9b1
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffffb1a7df10
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x40
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffffb1a7ea94
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x1c4
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffb1a7ed9f
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x9f
|
|
19
|
wb_writeback
|
vmlinux |
0xffffffffb1a7f273
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x263
|
|
20
|
|
vmlinux |
0xffffffffb1a6ae6c
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x4c
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffb1a7fe6b
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x2cb
|
|
22
|
process_one_work
|
vmlinux |
0xffffffffb18bde8f
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffffb18befa6
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffffb18bee80
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffffb18c5e61
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffffb18c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb1f99df7
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x21
|
|
28
|
|
vmlinux |
0xffffffffb18c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [kworker/u33:2:4494]
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [kworker/u34:2:25580]
Modules linked in: nf_conntrack_netlink xt_REDIRECT nf_nat_redirect xt_comment iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack vmw_vsock_vmci_transport vsock vmwgfx iosf_mbi ttm crc32_pclmul ghash_clmulni_intel e1000e drm_kms_helper aesni_intel syscopyarea lrw gf128mul sysfillrect glue_helper sysimgblt ablk_helper fb_sys_fops nfit cryptd drm ptp pcspkr joydev sg pps_core ppdev parport_pc drm_panel_orientation_quirks vmw_balloon parport libnvdimm i2c_piix4 vmw_vmci fuse auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 12 PID: 25580 Comm: kworker/u34:2 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff980535319080 ti: ffff980342220000 task.ti: ffff980342220000
RIP: 0010:[<ffffffffb19170da>] [<ffffffffb19170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff980342223830 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000b187ddd0 RCX: ffff980337c60818
RDX: 0000000000000001 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff980342223868 R08: ffff9803bb1a9800 R09: ffffffffb1b86c09
R10: ffff98053fd1f160 R11: ffffedd80ff0c800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9803422237e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff98053fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1a8a3192e0 CR3: 0000000176c10000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffb187e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb187e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffb1a08108>] ptep_clear_flush+0x68/0xa0
[<ffffffffb1a016a6>] page_mkclean_one+0xa6/0xe0
[<ffffffffb1a034b3>] rmap_walk+0xf3/0x360
[<ffffffffb1a038c6>] page_mkclean+0x76/0xb0
[<ffffffffb1a01600>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffb1a00ce0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb19ca6c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb19ca932>] write_cache_pages+0x1f2/0x470
[<ffffffffc01a8290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffb18de305>] ? sched_clock_cpu+0x85/0xc0
[<ffffffffc01a825b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb19cb9b1>] do_writepages+0x21/0x50
[<ffffffffb1a7df10>] __writeback_single_inode+0x40/0x260
[<ffffffffb1a7ea94>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffb1a7ed9f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb1a7f273>] wb_writeback+0x263/0x2f0
[<ffffffffb1a6ae6c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffb1a7fe6b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb18bde8f>] process_one_work+0x17f/0x440
[<ffffffffb18befa6>] worker_thread+0x126/0x3c0
[<ffffffffb18bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb18c5e61>] kthread+0xd1/0xe0
[<ffffffffb18c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffb1f99df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb18c5d90>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 55 b2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1012340
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9da81fa8
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0xb8
|
|
2
|
flush_tlb_page
|
vmlinux |
0xffffffff9da82144
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x54
|
|
3
|
ptep_clear_flush
|
vmlinux |
0xffffffff9dc14228
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x68
|
|
4
|
page_mkclean_one
|
vmlinux |
0xffffffff9dc0d5fe
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0xae
|
|
5
|
rmap_walk
|
vmlinux |
0xffffffff9dc0f473
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0xf3
|
|
6
|
page_mkclean
|
vmlinux |
0xffffffff9dc0f896
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x76
|
|
7
|
|
vmlinux |
0xffffffff9dc0d550
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x1b0
|
|
8
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff9dc0cc20
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x20
|
|
9
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff9dbd5770
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x50
|
|
10
|
write_cache_pages
|
vmlinux |
0xffffffff9dbd59e2
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x1f2
|
|
11
|
|
xfs |
0xffffffffc06a86c0
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0xb0
|
|
12
|
xfs_vm_writepages
|
xfs |
0xffffffffc06a867b
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x6b
|
|
13
|
do_writepages
|
vmlinux |
0xffffffff9dbd6ab1
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x21
|
|
14
|
__writeback_single_inode
|
vmlinux |
0xffffffff9dc8ccf0
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x40
|
|
15
|
writeback_sb_inodes
|
vmlinux |
0xffffffff9dc8d8bc
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x1cc
|
|
16
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff9dc8dbc7
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0xa7
|
|
17
|
wb_writeback
|
vmlinux |
0xffffffff9dc8e0a3
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x263
|
|
18
|
|
vmlinux |
0xffffffff9dc7906c
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x4c
|
|
19
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff9dc8ecaf
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x2cf
|
|
20
|
process_one_work
|
vmlinux |
0xffffffff9dac32ef
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x17f
|
|
21
|
worker_thread
|
vmlinux |
0xffffffff9dac4436
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x126
|
|
22
|
|
vmlinux |
0xffffffff9dac4310
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x2b0
|
|
23
|
kthread
|
vmlinux |
0xffffffff9dacb621
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0xd1
|
|
24
|
|
vmlinux |
0xffffffff9dacb550
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x40
|
|
25
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff9e1c51dd
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x7
|
|
26
|
|
vmlinux |
0xffffffff9dacb550
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#75 stuck for 23s! [spectre:4615]
RAX: 0000000000000040 RBX: 000800009da81b10 RCX: ffff9f0a25a20bc8
RDX: 0000000000000040 RSI: 0000000000000064 RDI: 0000000000000040
RBP: ffff9d2eff22f868 R08: ffff9d2ed1fc3408 R09: ffffffff9dd9c7b9
R10: ffff9f0a2539f160 R11: ffffdd8fc310f200 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9d2eff22f7e0 R15: ffff9f0a00000000
FS: 0000000000000000(0000) GS:ffff9f0a25380000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6d6f5cd860 CR3: 000001de4b3b6000 CR4: 0000000000760fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe07f0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffff9da81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9da82144>] flush_tlb_page+0x54/0xa0
[<ffffffff9dc14228>] ptep_clear_flush+0x68/0xa0
[<ffffffff9dc0d5fe>] page_mkclean_one+0xae/0xe0
[<ffffffff9dc0f473>] rmap_walk+0xf3/0x370
[<ffffffff9dc0f896>] page_mkclean+0x76/0xb0
[<ffffffff9dc0d550>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff9dc0cc20>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff9dbd5770>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff9dbd59e2>] write_cache_pages+0x1f2/0x480
[<ffffffffc06a86c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffffc06a867b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffff9dbd6ab1>] do_writepages+0x21/0x50
[<ffffffff9dc8ccf0>] __writeback_single_inode+0x40/0x260
[<ffffffff9dc8d8bc>] writeback_sb_inodes+0x1cc/0x430
[<ffffffff9dc8dbc7>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffff9dc8e0a3>] wb_writeback+0x263/0x2f0
[<ffffffff9dc7906c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9dc8ecaf>] bdi_writeback_workfn+0x2cf/0x470
[<ffffffff9dac32ef>] process_one_work+0x17f/0x440
[<ffffffff9dac4436>] worker_thread+0x126/0x3c0
[<ffffffff9dac4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff9dacb621>] kthread+0xd1/0xe0
[<ffffffff9dacb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff9e1c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff9dacb550>] ? insert_kthread_work+0x40/0x40
Complete report #1008983
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb2b1760a
Build id: 3.10.0-1127.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb2a7e8f8
Build id: 3.10.0-1127.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffb2a7ea94
Build id: 3.10.0-1127.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffb2c077f8
Build id: 3.10.0-1127.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffb2c00da6
Build id: 3.10.0-1127.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffb2c02bb3
Build id: 3.10.0-1127.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffb2c02fc6
Build id: 3.10.0-1127.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffb2c00d00
Build id: 3.10.0-1127.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffb2c003e0
Build id: 3.10.0-1127.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffb2bc9ec0
Build id: 3.10.0-1127.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffb2bca132
Build id: 3.10.0-1127.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc068f3d0
Build id: 3.10.0-1127.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffb2d54630
Build id: 3.10.0-1127.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffffb2ae4eac
Build id: 3.10.0-1127.el7.x86_64
|
0x14c
|
|
15
|
|
vmlinux |
0xffffffffb3185830
Build id: 3.10.0-1127.el7.x86_64
|
0x310
|
|
16
|
|
vmlinux |
0xffffffffb3185824
Build id: 3.10.0-1127.el7.x86_64
|
0x304
|
|
17
|
|
vmlinux |
0xffffffffb3185830
Build id: 3.10.0-1127.el7.x86_64
|
0x310
|
|
18
|
xfs_vm_writepages
|
xfs |
0xffffffffc068f39b
Build id: 3.10.0-1127.el7.x86_64
|
0x6b
|
|
19
|
do_writepages
|
vmlinux |
0xffffffffb2bcb1b1
Build id: 3.10.0-1127.el7.x86_64
|
0x21
|
|
20
|
__writeback_single_inode
|
vmlinux |
0xffffffffb2c7cbd0
Build id: 3.10.0-1127.el7.x86_64
|
0x40
|
|
21
|
|
vmlinux |
0xffffffffb2ac7745
Build id: 3.10.0-1127.el7.x86_64
|
0x25
|
|
22
|
writeback_sb_inodes
|
vmlinux |
0xffffffffb2c7d764
Build id: 3.10.0-1127.el7.x86_64
|
0x1c4
|
|
23
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffb2c7da6f
Build id: 3.10.0-1127.el7.x86_64
|
0x9f
|
|
24
|
wb_writeback
|
vmlinux |
0xffffffffb2c7df53
Build id: 3.10.0-1127.el7.x86_64
|
0x263
|
|
25
|
|
vmlinux |
0xffffffffb2c69bac
Build id: 3.10.0-1127.el7.x86_64
|
0x4c
|
|
26
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffb2c7eb4b
Build id: 3.10.0-1127.el7.x86_64
|
0x2cb
|
|
27
|
process_one_work
|
vmlinux |
0xffffffffb2abe6bf
Build id: 3.10.0-1127.el7.x86_64
|
0x17f
|
|
28
|
worker_thread
|
vmlinux |
0xffffffffb2abf7d6
Build id: 3.10.0-1127.el7.x86_64
|
0x126
|
|
29
|
|
vmlinux |
0xffffffffb2abf6b0
Build id: 3.10.0-1127.el7.x86_64
|
0x2a0
|
|
30
|
kthread
|
vmlinux |
0xffffffffb2ac6691
Build id: 3.10.0-1127.el7.x86_64
|
0xd1
|
|
31
|
|
vmlinux |
0xffffffffb2ac65c0
Build id: 3.10.0-1127.el7.x86_64
|
0x40
|
|
32
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb3192d37
Build id: 3.10.0-1127.el7.x86_64
|
0x21
|
|
33
|
|
vmlinux |
0xffffffffb2ac65c0
Build id: 3.10.0-1127.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u256:2:40601]
Modules linked in: binfmt_misc vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec sb_edac ac97_bus iosf_mbi crc32_pclmul snd_seq ghash_clmulni_intel aesni_intel snd_seq_device snd_pcm lrw gf128mul ppdev glue_helper snd_timer snd vmw_balloon ablk_helper cryptd soundcore pcspkr joydev sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata e1000 crct10dif_pclmul crct10dif_common nfit crc32c_intel mptspi libnvdimm scsi_transport_spi mptscsih mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 40601 Comm: kworker/u256:2 Kdump: loaded Tainted: G W L ------------ 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9913b54ae2a0 ti: ffff9912ded6c000 task.ti: ffff9912ded6c000
RIP: 0010:[<ffffffffb2b1760a>] [<ffffffffb2b1760a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9912ded6f830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000b2a7e730 RCX: ffff9913b9620628
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9912ded6f868 R08: ffff9913b7663400 R09: ffffffffb2d85099
R10: ffff9913b969f160 R11: ffffe72c41514600 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9912ded6f7e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff9913b9680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000455000 CR3: 0000000038e10000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb2a7e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb2a7ea94>] flush_tlb_page+0x54/0xa0
[<ffffffffb2c077f8>] ptep_clear_flush+0x68/0xa0
[<ffffffffb2c00da6>] page_mkclean_one+0xa6/0xe0
[<ffffffffb2c02bb3>] rmap_walk+0xf3/0x360
[<ffffffffb2c02fc6>] page_mkclean+0x76/0xb0
[<ffffffffb2c00d00>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffb2c003e0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb2bc9ec0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb2bca132>] write_cache_pages+0x1f2/0x470
[<ffffffffc068f3d0>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffb2d54630>] ? submit_bio+0x70/0x150
[<ffffffffb2ae4eac>] ? update_curr+0x14c/0x1e0
[<ffffffffb3185830>] ? __schedule+0x310/0x840
[<ffffffffb3185824>] ? __schedule+0x304/0x840
[<ffffffffb3185830>] ? __schedule+0x310/0x840
[<ffffffffc068f39b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb2bcb1b1>] do_writepages+0x21/0x50
[<ffffffffb2c7cbd0>] __writeback_single_inode+0x40/0x260
[<ffffffffb2ac7745>] ? wake_up_bit+0x25/0x30
[<ffffffffb2c7d764>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffb2c7da6f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb2c7df53>] wb_writeback+0x263/0x2f0
[<ffffffffb2c69bac>] ? get_nr_inodes+0x4c/0x70
[<ffffffffb2c7eb4b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb2abe6bf>] process_one_work+0x17f/0x440
[<ffffffffb2abf7d6>] worker_thread+0x126/0x3c0
[<ffffffffb2abf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb2ac6691>] kthread+0xd1/0xe0
[<ffffffffb2ac65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb3192d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb2ac65c0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 1e 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 0f 75 b3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 7a c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #990488
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
__page_check_address
|
vmlinux |
0xffffffff8340d26c
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x10c
|
|
2
|
page_mkclean_one
|
vmlinux |
0xffffffff8340d5b4
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x44
|
|
3
|
rmap_walk
|
vmlinux |
0xffffffff8340f493
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0xf3
|
|
4
|
page_mkclean
|
vmlinux |
0xffffffff8340f8b6
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x76
|
|
5
|
|
vmlinux |
0xffffffff8340d570
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x1b0
|
|
6
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff8340cc40
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x20
|
|
7
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff833d5790
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x50
|
|
8
|
write_cache_pages
|
vmlinux |
0xffffffff833d5a02
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x1f2
|
|
9
|
|
xfs |
0xffffffffc02406c0
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0xb0
|
|
10
|
xfs_vm_writepages
|
xfs |
0xffffffffc024067b
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x6b
|
|
11
|
do_writepages
|
vmlinux |
0xffffffff833d6ad1
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x21
|
|
12
|
__writeback_single_inode
|
vmlinux |
0xffffffff8348cd10
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x40
|
|
13
|
|
vmlinux |
0xffffffff832cc745
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x25
|
|
14
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8348d8dc
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x1cc
|
|
15
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8348dbe7
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0xa7
|
|
16
|
wb_writeback
|
vmlinux |
0xffffffff8348e0c3
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x263
|
|
17
|
|
vmlinux |
0xffffffff8347908c
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x4c
|
|
18
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8348eccf
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x2cf
|
|
19
|
process_one_work
|
vmlinux |
0xffffffff832c32ef
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x17f
|
|
20
|
worker_thread
|
vmlinux |
0xffffffff832c4436
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x126
|
|
21
|
|
vmlinux |
0xffffffff832c4310
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x2b0
|
|
22
|
kthread
|
vmlinux |
0xffffffff832cb621
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0xd1
|
|
23
|
|
vmlinux |
0xffffffff832cb550
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x40
|
|
24
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff839c51f7
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x21
|
|
25
|
|
vmlinux |
0xffffffff832cb550
Build id: 3.10.0-1160.114.2.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u4:0:15101]
Modules linked in: binfmt_misc xt_limit ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_multiport xt_conntrack nf_conntrack iptable_filter sg pcspkr joydev i2c_piix4 virtio_balloon ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover virtio_blk failover bochs_drm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 15101 Comm: kworker/u4:0 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.114.2.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff885138c8d280 ti: ffff885195188000 task.ti: ffff885195188000
RIP: 0010:[<ffffffff8340d26c>] [<ffffffff8340d26c>] __page_check_address+0x10c/0x260
RSP: 0000:ffff88519518b8e8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff88519518b92c RCX: 00003ffffffff000
RDX: 00003ffffffff000 RSI: 00003ffffffff000 RDI: ffffd7a2c8d46c30
RBP: ffff88519518b910 R08: 0000000000000000 R09: 00000002351b0000
R10: a174b63af259f580 R11: 0000000000000000 R12: 00007feb06d4e000
R13: ffffffff8340d262 R14: ffff88519518b878 R15: 00003ffffffff000
FS: 0000000000000000(0000) GS:ffff8852bfc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7d18e8d000 CR3: 000000019c610000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8340d5b4>] page_mkclean_one+0x44/0xe0
[<ffffffff8340f493>] rmap_walk+0xf3/0x370
[<ffffffff8340f8b6>] page_mkclean+0x76/0xb0
[<ffffffff8340d570>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff8340cc40>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff833d5790>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff833d5a02>] write_cache_pages+0x1f2/0x480
[<ffffffffc02406c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffffc024067b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffff833d6ad1>] do_writepages+0x21/0x50
[<ffffffff8348cd10>] __writeback_single_inode+0x40/0x260
[<ffffffff832cc745>] ? wake_up_bit+0x25/0x30
[<ffffffff8348d8dc>] writeback_sb_inodes+0x1cc/0x430
[<ffffffff8348dbe7>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffff8348e0c3>] wb_writeback+0x263/0x2f0
[<ffffffff8347908c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8348eccf>] bdi_writeback_workfn+0x2cf/0x470
[<ffffffff832c32ef>] process_one_work+0x17f/0x440
[<ffffffff832c4436>] worker_thread+0x126/0x3c0
[<ffffffff832c4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff832cb621>] kthread+0xd1/0xe0
[<ffffffff832cb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff839c51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff832cb550>] ? insert_kthread_work+0x40/0x40
Code: ff 3f 00 00 48 21 c1 48 21 ca 48 c1 ea 06 48 03 15 9a 8a a3 00 4c 8d 7a 30 4c 89 ff e8 ae d4 5a 00 48 ba 00 f0 ff ff ff 3f 00 00 <49> 8b 3e 48 89 d1 48 f7 d1 48 0b 0d 84 8b a3 00 48 21 f9 f7 c1
Complete report #986430
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffbc31e1ce
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffbc281fa8
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffbc282144
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffbc4141b8
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffbc40d58e
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xae
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffbc40f403
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffbc40f826
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffbc40d4e0
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffbc40cbb0
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffbc3d5700
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffbc3d5972
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc05206c0
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xb0
|
|
13
|
|
vmlinux |
0xffffffffbc2e8cc4
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x164
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc052067b
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffffbc3d6a41
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffffbc48cc80
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x40
|
|
17
|
|
vmlinux |
0xffffffffbc2da89e
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x4e
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffffbc48d84c
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x1cc
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffbc48db57
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xa7
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffffbc48e033
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffffbc478ffc
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffbc48ec3f
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x2cf
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffbc2c32ef
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffbc2c4436
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffbc2c4310
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x2b0
|
|
26
|
kthread
|
vmlinux |
0xffffffffbc2cb621
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffbc2cb550
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffbc9c51dd
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x7
|
|
29
|
|
vmlinux |
0xffffffffbc2cb550
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#34 stuck for 22s! [kworker/u256:2:106909]
Modules linked in: tcp_lp nls_utf8 isofs xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 devlink tun ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ext4 mbcache jbd2 nfit libnvdimm iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg joydev virtio_balloon i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif ata_generic crct10dif_generic pata_acpi
bochs_drm virtio_net net_failover drm_kms_helper virtio_scsi failover syscopyarea sysfillrect ahci sysimgblt fb_sys_fops ata_piix ttm libahci crct10dif_pclmul crct10dif_common virtio_pci drm virtio_ring libata serio_raw crc32c_intel drm_panel_orientation_quirks virtio floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 34 PID: 106909 Comm: kworker/u256:2 Tainted: G L ------------ 3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffffa1e47c166300 ti: ffffa1c9b1910000 task.ti: ffffa1c9b1910000
RIP: 0010:[<ffffffffbc31e1ce>] [<ffffffffbc31e1ce>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffffa1c9b1913830 EFLAGS: 00000202
RAX: 000000000000006d RBX: 00080000bc281b10 RCX: ffffa1ef7b160b28
RDX: 000000000000006d RSI: 0000000000000080 RDI: 0000000000000040
RBP: ffffa1c9b1913868 R08: ffffa114e2903408 R09: ffffffffbc59c669
R10: ffffa1ef79e9f160 R11: fffff6a3de38b000 R12: 0000000000000286
R13: 0000000000000286 R14: ffffa1c9b19137e0 R15: ffffa1c9b19137f0
FS: 0000000000000000(0000) GS:ffffa1ef79e80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000dfb5888c CR3: 000000dd8e616000 CR4: 0000000000760fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffffbc281fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbc282144>] flush_tlb_page+0x54/0xa0
[<ffffffffbc4141b8>] ptep_clear_flush+0x68/0xa0
[<ffffffffbc40d58e>] page_mkclean_one+0xae/0xe0
[<ffffffffbc40f403>] rmap_walk+0xf3/0x370
[<ffffffffbc40f826>] page_mkclean+0x76/0xb0
[<ffffffffbc40d4e0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffbc40cbb0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffbc3d5700>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffbc3d5972>] write_cache_pages+0x1f2/0x480
[<ffffffffc05206c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffffbc2e8cc4>] ? update_curr+0x164/0x1f0
[<ffffffffc052067b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffffbc3d6a41>] do_writepages+0x21/0x50
[<ffffffffbc48cc80>] __writeback_single_inode+0x40/0x260
[<ffffffffbc2da89e>] ? finish_task_switch+0x4e/0x1c0
[<ffffffffbc48d84c>] writeback_sb_inodes+0x1cc/0x430
[<ffffffffbc48db57>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffffbc48e033>] wb_writeback+0x263/0x2f0
[<ffffffffbc478ffc>] ? get_nr_inodes+0x4c/0x70
[<ffffffffbc48ec3f>] bdi_writeback_workfn+0x2cf/0x470
[<ffffffffbc2c32ef>] process_one_work+0x17f/0x440
[<ffffffffbc2c4436>] worker_thread+0x126/0x3c0
[<ffffffffbc2c4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffffbc2cb621>] kthread+0xd1/0xe0
[<ffffffffbc2cb550>] ? insert_kthread_work+0x40/0x40
[<ffffffffbc9c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffbc2cb550>] ? insert_kthread_work+0x40/0x40
Code: 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 f5 bc f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #924771
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8e11722a
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8e07e258
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8e07e3f4
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff8e2080f8
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff8e201696
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff8e2034a3
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff8e2038b6
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff8e2015f0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff8e200cd0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff8e1ca6d0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff8e1ca942
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc06a8290
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff8e0e64cb
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x8b
|
|
14
|
|
vmlinux |
0xffffffff8e0e2429
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xa9
|
|
15
|
|
vmlinux |
0xffffffff8e0e283c
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x3c
|
|
16
|
xfs_vm_writepages
|
xfs |
0xffffffffc06a825b
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x6b
|
|
17
|
do_writepages
|
vmlinux |
0xffffffff8e1cb9c1
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x21
|
|
18
|
__writeback_single_inode
|
vmlinux |
0xffffffff8e27df30
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x40
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8e27eab4
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x1c4
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8e27edbf
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x9f
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffff8e27f293
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x263
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8e27fd8c
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x1cc
|
|
23
|
process_one_work
|
vmlinux |
0xffffffff8e0bdfdf
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffff8e0bf0f6
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffff8e0befd0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffff8e0c5fb1
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffff8e0c5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff8e799ddd
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x7
|
|
29
|
|
vmlinux |
0xffffffff8e0c5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 69s! [java:4196]
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 69s! [kworker/u256:3:117059]
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi kvm_intel kvm snd_ens1371 irqbypass snd_rawmidi crc32_pclmul snd_ac97_codec ac97_bus ghash_clmulni_intel snd_seq aesni_intel lrw snd_seq_device gf128mul snd_pcm snd_timer ppdev glue_helper ablk_helper cryptd snd vmw_balloon soundcore joydev pcspkr sg parport_pc
parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mptspi ttm crct10dif_pclmul crct10dif_common scsi_transport_spi ata_piix crc32c_intel mptscsih drm nfit libata mptbase serio_raw libnvdimm e1000 drm_panel_orientation_quirks fuse
CPU: 3 PID: 117059 Comm: kworker/u256:3 Kdump: loaded Tainted: G W L ------------ 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff8a80c267e300 ti: ffff8a7f1e6b4000 task.ti: ffff8a7f1e6b4000
RIP: 0010:[<ffffffff8e11722a>] [<ffffffff8e11722a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8a7f1e6b7830 EFLAGS: 00000202
RAX: 0000000000000006 RBX: 000800008e07ddd0 RCX: ffff8a822d7a0650
RDX: 0000000000000006 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8a7f1e6b7868 R08: ffff8a7f75ccb800 R09: ffffffff8e3869d9
R10: ffff8a822d6df160 R11: ffffd8a58fcb4e00 R12: 0000000000000286
R13: 0000000000000000 R14: ffff8a7f1e6b77a0 R15: 0000000000000286
FS: 0000000000000000(0000) GS:ffff8a822d6c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f88002d8880 CR3: 0000000402116000 CR4: 00000000003607e0
Modules linked in: xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter overlay vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi kvm_intel kvm snd_ens1371 irqbypass snd_rawmidi crc32_pclmul snd_ac97_codec ac97_bus ghash_clmulni_intel snd_seq aesni_intel lrw snd_seq_device gf128mul snd_pcm snd_timer ppdev glue_helper ablk_helper cryptd snd vmw_balloon soundcore joydev pcspkr sg parport_pc
parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mptspi ttm crct10dif_pclmul crct10dif_common scsi_transport_spi ata_piix crc32c_intel mptscsih drm nfit libata mptbase serio_raw libnvdimm e1000 drm_panel_orientation_quirks fuse
CPU: 4 PID: 4196 Comm: java Kdump: loaded Tainted: G W L ------------ 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8a8227cee300 ti: ffff8a82157e4000 task.ti: ffff8a82157e4000
RIP: 0010:[<ffffffff8e11722a>] [<ffffffff8e11722a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8a82157e7cb0 EFLAGS: 00000202
RAX: 0000000000000006 RBX: 000800008e07ddd0 RCX: ffff8a822d7a0678
RDX: 0000000000000006 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8a82157e7ce8 R08: ffff8a7f75ccbc00 R09: ffffffff8e3869d9
R10: ffff8a822d71f160 R11: ffffd8a588ca9200 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8a82157e7c60 R15: 00007f17bd240000
FS: 00007f17c146eb10(0000) GS:ffff8a822d700000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f17bd1fffc0 CR3: 000000040609e000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8e07e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8e07e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffff8e2080f8>] ptep_clear_flush+0x68/0xa0
[<ffffffff8e201696>] page_mkclean_one+0xa6/0xe0
[<ffffffff8e2034a3>] rmap_walk+0xf3/0x360
[<ffffffff8e2038b6>] page_mkclean+0x76/0xb0
[<ffffffff8e2015f0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff8e200cd0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff8e1ca6d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8e1ca942>] write_cache_pages+0x1f2/0x470
[<ffffffffc06a8290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff8e0e64cb>] ? put_prev_entity+0x8b/0x400
[<ffffffff8e0e2429>] ? pick_next_entity+0xa9/0x190
[<ffffffff8e0e283c>] ? set_next_entity+0x3c/0xe0
[<ffffffffc06a825b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff8e1cb9c1>] do_writepages+0x21/0x50
[<ffffffff8e27df30>] __writeback_single_inode+0x40/0x260
[<ffffffff8e27eab4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8e27edbf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8e27f293>] wb_writeback+0x263/0x2f0
[<ffffffff8e27fd8c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffff8e0bdfdf>] process_one_work+0x17f/0x440
[<ffffffff8e0bf0f6>] worker_thread+0x126/0x3c0
[<ffffffff8e0befd0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8e0c5fb1>] kthread+0xd1/0xe0
[<ffffffff8e0c5ee0>] ? insert_kthread_work+0x40/0x40
[<ffffffff8e799ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff8e0c5ee0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 d5 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #921749
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9751e23a
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff97481fa8
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff97482144
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff97614228
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff9760d5fe
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0xae
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff9760f473
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff9760f896
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff9760d550
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff9760cc20
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff975d5770
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff975d59e2
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc067c6c0
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0xb0
|
|
13
|
|
vmlinux |
0xffffffff9776a7a0
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffff974e1182
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x12
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc067c67b
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffff975d6ab1
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffff9768ccf0
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x40
|
|
18
|
|
vmlinux |
0xffffffff974cc745
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x25
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffff9768d8bc
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x1cc
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff9768dbc7
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0xa7
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffff9768e0a3
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x263
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff9768ecaf
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x2cf
|
|
23
|
process_one_work
|
vmlinux |
0xffffffff974c32ef
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffff974c4436
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffff974c4310
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x2b0
|
|
26
|
kthread
|
vmlinux |
0xffffffff974cb621
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffff974cb550
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff97bc51f7
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x21
|
|
29
|
|
vmlinux |
0xffffffff974cb550
Build id: 3.10.0-1160.102.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [kworker/u256:4:98666]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_15805(E) nf_conntrack_netlink xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock falcon_lsm_pinned_15604(E) dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul
ppdev ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul nfit crct10dif_common mptspi crc32c_intel scsi_transport_spi libata libnvdimm mptscsih mptbase serio_raw vmxnet3 drm_panel_orientation_quirks fuse [last unloaded: falcon_kal]
CPU: 9 PID: 98666 Comm: kworker/u256:4 Kdump: loaded Tainted: P E ------------ T 3.10.0-1160.102.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff8913d00ee300 ti: ffff890a07688000 task.ti: ffff890a07688000
RIP: 0010:[<ffffffff9751e23a>] [<ffffffff9751e23a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff890a0768b830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0008000097481b10 RCX: ffff89197d620740
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff890a0768b868 R08: ffff890ad96a7800 R09: ffffffff9779c7b9
R10: ffff89197d85f160 R11: ffffe02ca57b4c00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff890a0768b7e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff89197d840000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5c5ab6f008 CR3: 000000079c410000 CR4: 00000000001607e0
Call Trace:
[<ffffffff97481fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff97482144>] flush_tlb_page+0x54/0xa0
[<ffffffff97614228>] ptep_clear_flush+0x68/0xa0
[<ffffffff9760d5fe>] page_mkclean_one+0xae/0xe0
[<ffffffff9760f473>] rmap_walk+0xf3/0x370
[<ffffffff9760f896>] page_mkclean+0x76/0xb0
[<ffffffff9760d550>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff9760cc20>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff975d5770>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff975d59e2>] write_cache_pages+0x1f2/0x480
[<ffffffffc067c6c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffff9776a7a0>] ? submit_bio+0x70/0x150
[<ffffffff974e1182>] ? default_wake_function+0x12/0x20
[<ffffffffc067c67b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffff975d6ab1>] do_writepages+0x21/0x50
[<ffffffff9768ccf0>] __writeback_single_inode+0x40/0x260
[<ffffffff974cc745>] ? wake_up_bit+0x25/0x30
[<ffffffff9768d8bc>] writeback_sb_inodes+0x1cc/0x430
[<ffffffff9768dbc7>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffff9768e0a3>] wb_writeback+0x263/0x2f0
[<ffffffff9768ecaf>] bdi_writeback_workfn+0x2cf/0x470
[<ffffffff974c32ef>] process_one_work+0x17f/0x440
[<ffffffff974c4436>] worker_thread+0x126/0x3c0
[<ffffffff974c4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff974cb621>] kthread+0xd1/0xe0
[<ffffffff974cb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff97bc51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff974cb550>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 19 15 98 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 18 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #880601
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffbc11e07a
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffbc081f48
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffbc0820e4
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffbc213eb8
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffbc20d29e
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xae
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffbc20f103
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffbc20f526
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffbc20d1f0
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffbc20c880
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffbc1d55a0
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffbc1d5812
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc06236c0
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xb0
|
|
13
|
|
vmlinux |
0xffffffffbc0ea05b
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x8b
|
|
14
|
|
vmlinux |
0xffffffffbc0e84c9
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xa9
|
|
15
|
|
vmlinux |
0xffffffffbc0e88bc
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x3c
|
|
16
|
xfs_vm_writepages
|
xfs |
0xffffffffc062367b
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x6b
|
|
17
|
do_writepages
|
vmlinux |
0xffffffffbc1d68e1
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x21
|
|
18
|
__writeback_single_inode
|
vmlinux |
0xffffffffbc28c750
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x40
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffffbc28d31c
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x1cc
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffbc28d627
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xa7
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffffbc28db03
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x263
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffbc28e60c
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x1cc
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffbc0c319f
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffbc0c42e6
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffbc0c41c0
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x2b0
|
|
26
|
kthread
|
vmlinux |
0xffffffffbc0cb4d1
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffbc0cb400
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffbc7c51dd
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x7
|
|
29
|
|
vmlinux |
0xffffffffbc0cb400
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [kworker/u16:0:12495]
Modules linked in: af_packet_diag netlink_diag vsock_diag tcp_diag udp_diag inet_diag unix_diag xt_comment xt_conntrack iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_filter vmw_vsock_vmci_transport vsock sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci parport_pc parport i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata mptspi scsi_transport_spi nfit mptscsih crct10dif_pclmul crct10dif_common crc32c_intel serio_raw libnvdimm mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash
dm_log dm_mod fuse
CPU: 4 PID: 12495 Comm: kworker/u16:0 Tainted: G W OEL ------------ 3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9906a0722100 ti: ffff990229770000 task.ti: ffff990229770000
RIP: 0010:[<ffffffffbc11e07a>] [<ffffffffbc11e07a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff990229773830 EFLAGS: 00000202
RAX: 0000000000000006 RBX: 00080000bc081ab0 RCX: ffff9909ffda0678
RDX: 0000000000000006 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff990229773868 R08: ffff99032c413000 R09: ffffffffbc39c139
R10: ffff9909ffd1f160 R11: fffff9b840209a00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9902297737e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff9909ffd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7d30dc1000 CR3: 000000004aa10000 CR4: 00000000000607e0
Call Trace:
[<ffffffffbc081f48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbc0820e4>] flush_tlb_page+0x54/0xa0
[<ffffffffbc213eb8>] ptep_clear_flush+0x68/0xa0
[<ffffffffbc20d29e>] page_mkclean_one+0xae/0xe0
[<ffffffffbc20f103>] rmap_walk+0xf3/0x370
[<ffffffffbc20f526>] page_mkclean+0x76/0xb0
[<ffffffffbc20d1f0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffbc20c880>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffbc1d55a0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffbc1d5812>] write_cache_pages+0x1f2/0x480
[<ffffffffc06236c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffffbc0ea05b>] ? put_prev_entity+0x8b/0x400
[<ffffffffbc0e84c9>] ? pick_next_entity+0xa9/0x190
[<ffffffffbc0e88bc>] ? set_next_entity+0x3c/0xe0
[<ffffffffc062367b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffffbc1d68e1>] do_writepages+0x21/0x50
[<ffffffffbc28c750>] __writeback_single_inode+0x40/0x260
[<ffffffffbc28d31c>] writeback_sb_inodes+0x1cc/0x430
[<ffffffffbc28d627>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffffbc28db03>] wb_writeback+0x263/0x2f0
[<ffffffffbc28e60c>] bdi_writeback_workfn+0x1cc/0x470
[<ffffffffbc0c319f>] process_one_work+0x17f/0x440
[<ffffffffbc0c42e6>] worker_thread+0x126/0x3c0
[<ffffffffbc0c41c0>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffffbc0cb4d1>] kthread+0xd1/0xe0
[<ffffffffbc0cb400>] ? insert_kthread_work+0x40/0x40
[<ffffffffbc7c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffbc0cb400>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ce 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 d5 bc f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 19 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #868741
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8ab1e23a
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8aa81fa8
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8aa82144
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff8ac14228
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff8ac0d5fe
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0xae
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff8ac0f473
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff8ac0f896
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff8ac0d550
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff8ac0cc20
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff8abd5770
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff8abd59e2
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc05356c0
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0xb0
|
|
13
|
|
vmlinux |
0xffffffff8ad9c5d5
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x35
|
|
14
|
|
vmlinux |
0xffffffff8ad9c5d5
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x35
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc053567b
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffff8abd6ab1
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffff8ac8ccf0
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x40
|
|
18
|
|
vmlinux |
0xffffffff8aacc745
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x25
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8ac8d8bc
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x1cc
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8ac8dbc7
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0xa7
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffff8ac8e0a3
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x263
|
|
22
|
|
vmlinux |
0xffffffff8ac7906c
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x4c
|
|
23
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8ac8ecaf
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x2cf
|
|
24
|
process_one_work
|
vmlinux |
0xffffffff8aac32ef
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x17f
|
|
25
|
worker_thread
|
vmlinux |
0xffffffff8aac4436
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x126
|
|
26
|
|
vmlinux |
0xffffffff8aac4310
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x2b0
|
|
27
|
kthread
|
vmlinux |
0xffffffff8aacb621
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0xd1
|
|
28
|
|
vmlinux |
0xffffffff8aacb550
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x40
|
|
29
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff8b1c51dd
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x7
|
|
30
|
|
vmlinux |
0xffffffff8aacb550
Build id: 3.10.0-1160.99.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u256:2:65937]
Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus iosf_mbi snd_seq crc32_pclmul ghash_clmulni_intel snd_seq_device snd_pcm aesni_intel lrw snd_timer gf128mul snd glue_helper ablk_helper soundcore
cryptd sg ppdev pcspkr vmw_vmci joydev vmw_balloon i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix nfit mptspi libata scsi_transport_spi libnvdimm e1000 mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 5 PID: 65937 Comm: kworker/u256:2 Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.99.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff976c53b31080 ti: ffff976d8e284000 task.ti: ffff976d8e284000
RIP: 0010:[<ffffffff8ab1e23a>] [<ffffffff8ab1e23a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff976d8e287830 EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800008aa81b10 RCX: ffff976e756a06a0
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff976d8e287868 R08: ffff976dba88c000 R09: ffffffff8ad9c7b9
R10: ffff976e7575f160 R11: fffffad948a3d000 R12: 0000000000000286
R13: 0000000000000286 R14: ffff976d8e2877e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff976e75740000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f065a01c068 CR3: 00000001cec10000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8aa81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8aa82144>] flush_tlb_page+0x54/0xa0
[<ffffffff8ac14228>] ptep_clear_flush+0x68/0xa0
[<ffffffff8ac0d5fe>] page_mkclean_one+0xae/0xe0
[<ffffffff8ac0f473>] rmap_walk+0xf3/0x370
[<ffffffff8ac0f896>] page_mkclean+0x76/0xb0
[<ffffffff8ac0d550>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff8ac0cc20>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff8abd5770>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8abd59e2>] write_cache_pages+0x1f2/0x480
[<ffffffffc05356c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffff8ad9c5d5>] ? cpumask_next_and+0x35/0x50
[<ffffffff8ad9c5d5>] ? cpumask_next_and+0x35/0x50
[<ffffffffc053567b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffff8abd6ab1>] do_writepages+0x21/0x50
[<ffffffff8ac8ccf0>] __writeback_single_inode+0x40/0x260
[<ffffffff8aacc745>] ? wake_up_bit+0x25/0x30
[<ffffffff8ac8d8bc>] writeback_sb_inodes+0x1cc/0x430
[<ffffffff8ac8dbc7>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffff8ac8e0a3>] wb_writeback+0x263/0x2f0
[<ffffffff8ac7906c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8ac8ecaf>] bdi_writeback_workfn+0x2cf/0x470
[<ffffffff8aac32ef>] process_one_work+0x17f/0x440
[<ffffffff8aac4436>] worker_thread+0x126/0x3c0
[<ffffffff8aac4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff8aacb621>] kthread+0xd1/0xe0
[<ffffffff8aacb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff8b1c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff8aacb550>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 19 75 8b f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 18 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #862070
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8e11760a
Build id: 3.10.0-1127.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8e07e8f8
Build id: 3.10.0-1127.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8e07ea94
Build id: 3.10.0-1127.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff8e2077f8
Build id: 3.10.0-1127.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff8e200da6
Build id: 3.10.0-1127.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff8e202bb3
Build id: 3.10.0-1127.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff8e202fc6
Build id: 3.10.0-1127.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff8e200d00
Build id: 3.10.0-1127.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff8e2003e0
Build id: 3.10.0-1127.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff8e1c9ec0
Build id: 3.10.0-1127.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff8e1ca132
Build id: 3.10.0-1127.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc05de3d0
Build id: 3.10.0-1127.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff8e354630
Build id: 3.10.0-1127.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffff8e02b59e
Build id: 3.10.0-1127.el7.x86_64
|
0xce
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc05de39b
Build id: 3.10.0-1127.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffff8e1cb1b1
Build id: 3.10.0-1127.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffff8e27cbd0
Build id: 3.10.0-1127.el7.x86_64
|
0x40
|
|
18
|
|
vmlinux |
0xffffffff8e0c7745
Build id: 3.10.0-1127.el7.x86_64
|
0x25
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8e27d764
Build id: 3.10.0-1127.el7.x86_64
|
0x1c4
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8e27da6f
Build id: 3.10.0-1127.el7.x86_64
|
0x9f
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffff8e27df53
Build id: 3.10.0-1127.el7.x86_64
|
0x263
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8e27eb4b
Build id: 3.10.0-1127.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffff8e0be6bf
Build id: 3.10.0-1127.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffff8e0bf7d6
Build id: 3.10.0-1127.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffff8e0bf6b0
Build id: 3.10.0-1127.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffff8e0c6691
Build id: 3.10.0-1127.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffff8e0c65c0
Build id: 3.10.0-1127.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff8e792d37
Build id: 3.10.0-1127.el7.x86_64
|
0x21
|
|
29
|
|
vmlinux |
0xffffffff8e0c65c0
Build id: 3.10.0-1127.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [kworker/u32:2:24560]
Modules linked in: ip6table_filter ip6_tables xt_owner tcp_diag inet_diag binfmt_misc iptable_filter vmw_vsock_vmci_transport vsock sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper pcspkr cryptd joydev sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod ahci crc_t10dif ata_piix libahci crct10dif_generic drm nfit libata crct10dif_pclmul crct10dif_common libnvdimm crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: talpa_syscallhook]
CPU: 9 PID: 24560 Comm: kworker/u32:2 Kdump: loaded Tainted: G W OEL ------------ T 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff98fef8abe2a0 ti: ffff98fcdc838000 task.ti: ffff98fcdc838000
RIP: 0010:[<ffffffff8e11760a>] [<ffffffff8e11760a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff98fcdc83b830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008e07e730 RCX: ffff990bbf220740
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff98fcdc83b868 R08: ffff98fd29710800 R09: ffffffff8e385099
R10: ffff990bbf45f160 R11: ffffe965f082c000 R12: 0000000000000286
R13: 0000000000000286 R14: ffff98fcdc83b7e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff990bbf440000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4736537008 CR3: 0000000107e4a000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8e07e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8e07ea94>] flush_tlb_page+0x54/0xa0
[<ffffffff8e2077f8>] ptep_clear_flush+0x68/0xa0
[<ffffffff8e200da6>] page_mkclean_one+0xa6/0xe0
[<ffffffff8e202bb3>] rmap_walk+0xf3/0x360
[<ffffffff8e202fc6>] page_mkclean+0x76/0xb0
[<ffffffff8e200d00>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff8e2003e0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff8e1c9ec0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8e1ca132>] write_cache_pages+0x1f2/0x470
[<ffffffffc05de3d0>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff8e354630>] ? submit_bio+0x70/0x150
[<ffffffff8e02b59e>] ? __switch_to+0xce/0x580
[<ffffffffc05de39b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff8e1cb1b1>] do_writepages+0x21/0x50
[<ffffffff8e27cbd0>] __writeback_single_inode+0x40/0x260
[<ffffffff8e0c7745>] ? wake_up_bit+0x25/0x30
[<ffffffff8e27d764>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8e27da6f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8e27df53>] wb_writeback+0x263/0x2f0
[<ffffffff8e27eb4b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff8e0be6bf>] process_one_work+0x17f/0x440
[<ffffffff8e0bf7d6>] worker_thread+0x126/0x3c0
[<ffffffff8e0bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8e0c6691>] kthread+0xd1/0xe0
[<ffffffff8e0c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff8e792d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8e0c65c0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 1e 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 0f d5 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 7a c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #851133
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff957170de
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9567e258
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff9567e3f4
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff95808108
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff958016a6
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff958034b3
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff958038c6
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff95801600
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff95800ce0
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff957ca6c0
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff957ca932
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc0591290
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff95955e80
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffff95986a55
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x35
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc059125b
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffff957cb9b1
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffff9587df10
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffff9587ea94
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff9587ed9f
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffff9587f273
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffff9586ae6c
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff9587fe6b
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffff956bde8f
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffff956befa6
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffff956bee80
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffff956c5e61
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffff956c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff95d99ddd
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x7
|
|
29
|
|
vmlinux |
0xffffffff956c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kworker/u65:3:15449]
Modules linked in: tcp_diag udp_diag inet_diag xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper vmw_balloon cryptd sg pcspkr joydev vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci nfit ata_piix libahci drm
libnvdimm crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 10 PID: 15449 Comm: kworker/u65:3 Tainted: G W L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8accc580b180 ti: ffff8acc05290000 task.ti: ffff8acc05290000
RIP: 0010:[<ffffffff957170de>] [<ffffffff957170de>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8acc05293830 EFLAGS: 00000202
RAX: 0000000000000005 RBX: 000800009567ddd0 RCX: ffff8adaffb607c8
RDX: 0000000000000005 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff8acc05293868 R08: ffff8acc5949e400 R09: ffffffff95986c09
R10: ffff8adaffc9f160 R11: ffffebbb3ffb2400 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8acc052937e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff8adaffc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c00a694b80 CR3: 00000001d0ea8000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffff9567e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9567e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffff95808108>] ptep_clear_flush+0x68/0xa0
[<ffffffff958016a6>] page_mkclean_one+0xa6/0xe0
[<ffffffff958034b3>] rmap_walk+0xf3/0x360
[<ffffffff958038c6>] page_mkclean+0x76/0xb0
[<ffffffff95801600>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff95800ce0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff957ca6c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff957ca932>] write_cache_pages+0x1f2/0x470
[<ffffffffc0591290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff95955e80>] ? submit_bio+0x70/0x150
[<ffffffff95986a55>] ? cpumask_next_and+0x35/0x50
[<ffffffffc059125b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff957cb9b1>] do_writepages+0x21/0x50
[<ffffffff9587df10>] __writeback_single_inode+0x40/0x260
[<ffffffff9587ea94>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff9587ed9f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff9587f273>] wb_writeback+0x263/0x2f0
[<ffffffff9586ae6c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9587fe6b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff956bde8f>] process_one_work+0x17f/0x440
[<ffffffff956befa6>] worker_thread+0x126/0x3c0
[<ffffffff956bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff956c5e61>] kthread+0xd1/0xe0
[<ffffffff956c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffff95d99ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff956c5d90>] ? insert_kthread_work+0x40/0x40
Code: 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 35 96 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #820934
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffaed1173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffaec7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffaec7a734
Build id: 3.10.0-957.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffaedfd587
Build id: 3.10.0-957.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffaedf6b66
Build id: 3.10.0-957.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffaedf8983
Build id: 3.10.0-957.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffaedf8d96
Build id: 3.10.0-957.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffaedf6ac0
Build id: 3.10.0-957.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffaedf6200
Build id: 3.10.0-957.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffaedc1d60
Build id: 3.10.0-957.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffaedc200a
Build id: 3.10.0-957.el7.x86_64
|
0x22a
|
|
12
|
|
xfs |
0xffffffffc05635b0
Build id: 3.10.0-957.el7.x86_64
|
0x150
|
|
13
|
|
vmlinux |
0xffffffffaef44f30
Build id: 3.10.0-957.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffffaec2a59e
Build id: 3.10.0-957.el7.x86_64
|
0xce
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc05633ab
Build id: 3.10.0-957.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffffaedc3181
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffffaee6f930
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffffaee703c4
Build id: 3.10.0-957.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffaee7072f
Build id: 3.10.0-957.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffffaee70f63
Build id: 3.10.0-957.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffffaee5cd7c
Build id: 3.10.0-957.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffaee718eb
Build id: 3.10.0-957.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffaecb9d4f
Build id: 3.10.0-957.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffaecbade6
Build id: 3.10.0-957.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffaecbacc0
Build id: 3.10.0-957.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffffaecc1c31
Build id: 3.10.0-957.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffaecc1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffaf374c37
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
29
|
|
vmlinux |
0xffffffffaecc1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u80:1:37162]
Modules linked in: binfmt_misc sysmon_edr(OE) netfilter_edr(OE) ppdev sg i2c_piix4 joydev virtio_balloon parport_pc parport pcspkr ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_blk virtio_console virtio_scsi virtio_net serio_raw cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm floppy drm ata_piix libata virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 37162 Comm: kworker/u80:1 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9d008c622080 ti: ffff9d001f2f0000 task.ti: ffff9d001f2f0000
RIP: 0010:[<ffffffffaed1173a>] [<ffffffffaed1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9d001f2f3820 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000aec7a3d0 RCX: ffff9d00dec5f610
RDX: 0000000000000001 RSI: 0000000000000028 RDI: 0000000000000000
RBP: ffff9d001f2f3858 R08: ffff9cfa3fd5f400 R09: ffffffffaef755c9
R10: ffff9d00dec1f160 R11: ffffed438d3c8400 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9d001f2f37d0 R15: ffff9d001f2f37a0
FS: 0000000000000000(0000) GS:ffff9d00dec00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fea2bdb3500 CR3: 0000000244410000 CR4: 00000000000006f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffaec7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffaec7a734>] flush_tlb_page+0x54/0xa0
[<ffffffffaedfd587>] ptep_clear_flush+0x67/0xa0
[<ffffffffaedf6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffffaedf8983>] rmap_walk+0xf3/0x360
[<ffffffffaedf8d96>] page_mkclean+0x76/0xb0
[<ffffffffaedf6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffffaedf6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffaedc1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffaedc200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc05635b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffaef44f30>] ? submit_bio+0x70/0x150
[<ffffffffaec2a59e>] ? __switch_to+0xce/0x580
[<ffffffffc05633ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffaedc3181>] do_writepages+0x21/0x50
[<ffffffffaee6f930>] __writeback_single_inode+0x40/0x260
[<ffffffffaee703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffaee7072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffaee70f63>] wb_writeback+0x263/0x2f0
[<ffffffffaee5cd7c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffaee718eb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffaecb9d4f>] process_one_work+0x17f/0x440
[<ffffffffaecbade6>] worker_thread+0x126/0x3c0
[<ffffffffaecbacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffaecc1c31>] kthread+0xd1/0xe0
[<ffffffffaecc1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffaf374c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffaecc1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 94 af f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #811645
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb851173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb847a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffb847a734
Build id: 3.10.0-957.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffb85fd587
Build id: 3.10.0-957.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffb85f6b66
Build id: 3.10.0-957.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffb85f8983
Build id: 3.10.0-957.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffb85f8d96
Build id: 3.10.0-957.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffb85f6ac0
Build id: 3.10.0-957.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffb85f6200
Build id: 3.10.0-957.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffb85c1d60
Build id: 3.10.0-957.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffb85c200a
Build id: 3.10.0-957.el7.x86_64
|
0x22a
|
|
12
|
|
xfs |
0xffffffffc05d35b0
Build id: 3.10.0-957.el7.x86_64
|
0x150
|
|
13
|
|
vmlinux |
0xffffffffb8744f30
Build id: 3.10.0-957.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffffb8744f48
Build id: 3.10.0-957.el7.x86_64
|
0x88
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc05d33ab
Build id: 3.10.0-957.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffffb85c3181
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffffb866f930
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffffb86703c4
Build id: 3.10.0-957.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffb867072f
Build id: 3.10.0-957.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffffb8670f63
Build id: 3.10.0-957.el7.x86_64
|
0x263
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffb86718eb
Build id: 3.10.0-957.el7.x86_64
|
0x2cb
|
|
22
|
process_one_work
|
vmlinux |
0xffffffffb84b9d4f
Build id: 3.10.0-957.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffffb84bade6
Build id: 3.10.0-957.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffffb84bacc0
Build id: 3.10.0-957.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffffb84c1c31
Build id: 3.10.0-957.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffffb84c1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb8b74c1d
Build id: 3.10.0-957.el7.x86_64
|
0x7
|
|
28
|
|
vmlinux |
0xffffffffb84c1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [kworker/u32:0:4466]
Modules linked in: tcp_lp nls_utf8 isofs fuse veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter binfmt_misc vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper vmw_balloon cryptd sg pcspkr joydev parport_pc i2c_piix4 parport vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
ttm ahci drm libahci crct10dif_pclmul crct10dif_common crc32c_intel vmxnet3 serio_raw ata_piix libata vmw_pvscsi drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 10 PID: 4466 Comm: kworker/u32:0 Tainted: G L ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9193f62d2080 ti: ffff919383b5c000 task.ti: ffff919383b5c000
RIP: 0010:[<ffffffffb851173a>] [<ffffffffb851173a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff919383b5f820 EFLAGS: 00000202
RAX: 0000000000000009 RBX: 00080000b847a3d0 RCX: ffff9194df460718
RDX: 0000000000000009 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff919383b5f858 R08: ffff918e34405800 R09: ffffffffb87755c9
R10: ffff9194df49f160 R11: fffff71e41501000 R12: 0000000000000286
R13: 0000000000000286 R14: ffff919383b5f7d0 R15: ffff918e3fc03300
FS: 0000000000000000(0000) GS:ffff9194df480000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000a3f000 CR3: 00000001e8810000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffffb847a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb847a734>] flush_tlb_page+0x54/0xa0
[<ffffffffb85fd587>] ptep_clear_flush+0x67/0xa0
[<ffffffffb85f6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffffb85f8983>] rmap_walk+0xf3/0x360
[<ffffffffb85f8d96>] page_mkclean+0x76/0xb0
[<ffffffffb85f6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffffb85f6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb85c1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb85c200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc05d35b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffb8744f30>] ? submit_bio+0x70/0x150
[<ffffffffb8744f48>] ? submit_bio+0x88/0x150
[<ffffffffc05d33ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb85c3181>] do_writepages+0x21/0x50
[<ffffffffb866f930>] __writeback_single_inode+0x40/0x260
[<ffffffffb86703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffb867072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb8670f63>] wb_writeback+0x263/0x2f0
[<ffffffffb86718eb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb84b9d4f>] process_one_work+0x17f/0x440
[<ffffffffb84bade6>] worker_thread+0x126/0x3c0
[<ffffffffb84bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffb84c1c31>] kthread+0xd1/0xe0
[<ffffffffb84c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb8b74c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb84c1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 14 b9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #792384
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb931e07a
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb9281f48
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffb92820e4
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffb9413eb8
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffb940d29e
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xae
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffb940f103
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffb940f526
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffb940d1f0
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffb940c880
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffb93d55a0
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffb93d5812
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc05b36c0
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xb0
|
|
13
|
|
vmlinux |
0xffffffffb92e908c
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x11c
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc05b367b
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffffb93d68e1
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffffb948c750
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x40
|
|
17
|
|
vmlinux |
0xffffffffb92cc5f5
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x25
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffffb948d31c
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x1cc
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffb948d627
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xa7
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffffb948db03
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffffb9478a7c
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffb948e70f
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x2cf
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffb92c319f
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffb92c42e6
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffb92c41c0
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x2b0
|
|
26
|
kthread
|
vmlinux |
0xffffffffb92cb4d1
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffb92cb400
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb99c51dd
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x7
|
|
29
|
|
vmlinux |
0xffffffffb92cb400
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u256:0:115897]
Modules linked in: vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod vfat fat snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore pcspkr ppdev joydev tpm_crb sg iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd parport_pc parport vmw_balloon i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_generic pata_acpi ahci drm ata_piix mptsas libahci crct10dif_pclmul crct10dif_common nfit crc32c_intel scsi_transport_sas libata mptscsih e1000 serio_raw libnvdimm mptbase drm_panel_orientation_quirks fuse
CPU: 2 PID: 115897 Comm: kworker/u256:0 Kdump: loaded Tainted: G W L ------------ 3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.14410784.B64.1908150010 08/15/2019
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff930733743180 ti: ffff93040e814000 task.ti: ffff93040e814000
RIP: 0010:[<ffffffffb931e07a>] [<ffffffffb931e07a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff93040e817830 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00080000b9281ab0 RCX: ffff93082d6e0628
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff93040e817868 R08: ffff930575cdb400 R09: ffffffffb959c139
R10: ffff93082d69f160 R11: ffffeaf250aff600 R12: 0000000000000286
R13: 0000000000000286 R14: ffff93040e8177e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff93082d680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f406ec15003 CR3: 000000005ce10000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffb9281f48>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb92820e4>] flush_tlb_page+0x54/0xa0
[<ffffffffb9413eb8>] ptep_clear_flush+0x68/0xa0
[<ffffffffb940d29e>] page_mkclean_one+0xae/0xe0
[<ffffffffb940f103>] rmap_walk+0xf3/0x370
[<ffffffffb940f526>] page_mkclean+0x76/0xb0
[<ffffffffb940d1f0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffb940c880>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb93d55a0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb93d5812>] write_cache_pages+0x1f2/0x480
[<ffffffffc05b36c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffffb92e908c>] ? dequeue_entity+0x11c/0x5d0
[<ffffffffc05b367b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffffb93d68e1>] do_writepages+0x21/0x50
[<ffffffffb948c750>] __writeback_single_inode+0x40/0x260
[<ffffffffb92cc5f5>] ? wake_up_bit+0x25/0x30
[<ffffffffb948d31c>] writeback_sb_inodes+0x1cc/0x430
[<ffffffffb948d627>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffffb948db03>] wb_writeback+0x263/0x2f0
[<ffffffffb9478a7c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffb948e70f>] bdi_writeback_workfn+0x2cf/0x470
[<ffffffffb92c319f>] process_one_work+0x17f/0x440
[<ffffffffb92c42e6>] worker_thread+0x126/0x3c0
[<ffffffffb92c41c0>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffffb92cb4d1>] kthread+0xd1/0xe0
[<ffffffffb92cb400>] ? insert_kthread_work+0x40/0x40
[<ffffffffb99c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb92cb400>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ce 19 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 f5 b9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 19 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #771563
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8e11e1ca
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8e081fa8
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8e082144
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff8e2141b8
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff8e20d58e
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xae
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff8e20f403
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff8e20f826
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff8e20d4e0
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff8e20cbb0
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff8e1d5700
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff8e1d5972
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc05db6c0
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xb0
|
|
13
|
|
vmlinux |
0xffffffff8e39c485
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x35
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc05db67b
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffff8e1d6a41
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffff8e28cc80
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x40
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8e28d84c
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x1cc
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8e28db57
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xa7
|
|
19
|
wb_writeback
|
vmlinux |
0xffffffff8e28e033
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x263
|
|
20
|
|
vmlinux |
0xffffffff8e278ffc
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x4c
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8e28ec3f
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x2cf
|
|
22
|
process_one_work
|
vmlinux |
0xffffffff8e0c32ef
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffff8e0c4436
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffff8e0c4310
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x2b0
|
|
25
|
kthread
|
vmlinux |
0xffffffff8e0cb621
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffff8e0cb550
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff8e7c51dd
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x7
|
|
28
|
|
vmlinux |
0xffffffff8e0cb550
Build id: 3.10.0-1160.88.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 25s! [kworker/u256:3:14820]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter devlink ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq ppdev snd_seq_device snd_pcm aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd snd_timer joydev pcspkr snd vmw_vmci sg soundcore i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm nfit ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata libnvdimm mptspi e1000 scsi_transport_spi serio_raw mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 4 PID: 14820 Comm: kworker/u256:3 Tainted: G L ------------ 3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff8fc06a930000 ti: ffff8fbf160cc000 task.ti: ffff8fbf160cc000
RIP: 0010:[<ffffffff8e11e1ca>] [<ffffffff8e11e1ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8fbf160cf830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008e081b10 RCX: ffff8fc12a020678
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8fbf160cf868 R08: ffff8fbff93cbc00 R09: ffffffff8e39c669
R10: ffff8fc12a11f160 R11: ffffe648094dac00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8fbf160cf7e0 R15: 0000000000000286
FS: 0000000000000000(0000) GS:ffff8fc12a100000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2920268000 CR3: 00000000bbc10000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8e081fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8e082144>] flush_tlb_page+0x54/0xa0
[<ffffffff8e2141b8>] ptep_clear_flush+0x68/0xa0
[<ffffffff8e20d58e>] page_mkclean_one+0xae/0xe0
[<ffffffff8e20f403>] rmap_walk+0xf3/0x370
[<ffffffff8e20f826>] page_mkclean+0x76/0xb0
[<ffffffff8e20d4e0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff8e20cbb0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff8e1d5700>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8e1d5972>] write_cache_pages+0x1f2/0x480
[<ffffffffc05db6c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffff8e39c485>] ? cpumask_next_and+0x35/0x50
[<ffffffffc05db67b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffff8e1d6a41>] do_writepages+0x21/0x50
[<ffffffff8e28cc80>] __writeback_single_inode+0x40/0x260
[<ffffffff8e28d84c>] writeback_sb_inodes+0x1cc/0x430
[<ffffffff8e28db57>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffff8e28e033>] wb_writeback+0x263/0x2f0
[<ffffffff8e278ffc>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8e28ec3f>] bdi_writeback_workfn+0x2cf/0x470
[<ffffffff8e0c32ef>] process_one_work+0x17f/0x440
[<ffffffff8e0c4436>] worker_thread+0x126/0x3c0
[<ffffffff8e0c4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff8e0cb621>] kthread+0xd1/0xe0
[<ffffffff8e0cb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff8e7c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff8e0cb550>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 7e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 d5 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #762032
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
page_mkclean_one
|
vmlinux |
0xffffffff89600d90
Build id: 3.10.0-1127.19.1.el7.x86_64
|
-
|
|
2
|
|
vmlinux |
0xffffffff89602c43
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xf3
|
|
3
|
page_mkclean
|
vmlinux |
0xffffffff89603056
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x76
|
|
4
|
|
vmlinux |
0xffffffff89600d90
Build id: 3.10.0-1127.19.1.el7.x86_64
|
-
|
|
5
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff89600470
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x20
|
|
6
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff895c9f70
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x50
|
|
7
|
write_cache_pages
|
vmlinux |
0xffffffff895ca1e2
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x1f2
|
|
8
|
|
xfs |
0xffffffffc0648310
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xa0
|
|
9
|
xfs_vm_writepages
|
xfs |
0xffffffffc06482db
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x6b
|
|
10
|
do_writepages
|
vmlinux |
0xffffffff895cb261
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x21
|
|
11
|
__writeback_single_inode
|
vmlinux |
0xffffffff8967cc40
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x40
|
|
12
|
|
vmlinux |
0xffffffff894c7745
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x25
|
|
13
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8967d7c4
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x1c4
|
|
14
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8967dacf
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x9f
|
|
15
|
wb_writeback
|
vmlinux |
0xffffffff8967dfa3
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x263
|
|
16
|
|
vmlinux |
0xffffffff89669c5c
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x4c
|
|
17
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8967eb9b
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x2cb
|
|
18
|
process_one_work
|
vmlinux |
0xffffffff894be6bf
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x17f
|
|
19
|
worker_thread
|
vmlinux |
0xffffffff894bf7d6
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x126
|
|
20
|
|
vmlinux |
0xffffffff894bf6b0
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x2a0
|
|
21
|
kthread
|
vmlinux |
0xffffffff894c6691
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xd1
|
|
22
|
|
vmlinux |
0xffffffff894c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x40
|
|
23
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff89b92d37
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x21
|
|
24
|
|
vmlinux |
0xffffffff894c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u2:0:24091]
Modules linked in: sunrpc intel_powerclamp crc32_pclmul ghash_clmulni_intel snd_intel8x0 joydev ppdev snd_ac97_codec ac97_bus snd_seq aesni_intel snd_seq_device snd_pcm lrw gf128mul glue_helper ablk_helper cryptd pcspkr snd_timer sg snd parport_pc parport video vboxguest(OE) i2c_piix4 soundcore binfmt_misc sch_fq_codel ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci ata_piix libahci libata e1000 crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 24091 Comm: kworker/u2:0 Tainted: G OEL ------------ 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff982d8edee2a0 ti: ffff982d84bbc000 task.ti: ffff982d84bbc000
RIP: 0010:[<ffffffff89600d90>] [<ffffffff89600d90>] page_mkclean_one+0x0/0xe0
RSP: 0018:ffff982d84bbf960 EFLAGS: 00000206
RAX: ffffffff89600d90 RBX: ffffffff896009c9 RCX: ffff982d84bbf9c4
RDX: 00007f2e53def000 RSI: ffff982de9a8b510 RDI: ffffe39502021840
RBP: ffff982d84bbf9b0 R08: 0000000000000000 R09: 0000000000000000
R10: 71abdb8ca2021800 R11: 0000000000000000 R12: ffff982e1f132ca8
R13: 8000000080861867 R14: ffffffff8947ea78 R15: ffff982d84bbf8d8
FS: 0000000000000000(0000) GS:ffff982e99c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007facaefd8000 CR3: 00000000d7d6c000 CR4: 00000000000606f0
Call Trace:
[<ffffffff89602c43>] ? rmap_walk+0xf3/0x360
[<ffffffff89603056>] page_mkclean+0x76/0xb0
[<ffffffff89600d90>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff89600470>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff895c9f70>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff895ca1e2>] write_cache_pages+0x1f2/0x470
[<ffffffffc0648310>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffc06482db>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff895cb261>] do_writepages+0x21/0x50
[<ffffffff8967cc40>] __writeback_single_inode+0x40/0x260
[<ffffffff894c7745>] ? wake_up_bit+0x25/0x30
[<ffffffff8967d7c4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8967dacf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8967dfa3>] wb_writeback+0x263/0x2f0
[<ffffffff89669c5c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8967eb9b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff894be6bf>] process_one_work+0x17f/0x440
[<ffffffff894bf7d6>] worker_thread+0x126/0x3c0
[<ffffffff894bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff894c6691>] kthread+0xd1/0xe0
[<ffffffff894c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff89b92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff894c65c0>] ? insert_kthread_work+0x40/0x40
Code: 00 e9 5d ff ff ff 4c 89 f6 e8 7d da 01 00 e9 2e ff ff ff 4c 89 f6 e8 70 da 01 00 eb d1 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 <0f> 1f 44 00 00 55 41 b8 01 00 00 00 48 89 e5 41 57 49 89 f7 41
Complete report #716739
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff82b1e1ce
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff82a81fa8
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff82a82144
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff82c141b8
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff82c0d58e
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0xae
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff82c0f403
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff82c0f826
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff82c0d4e0
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff82c0cbb0
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff82bd5700
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff82bd5972
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc04816c0
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0xb0
|
|
13
|
|
vmlinux |
0xffffffff82aea1ab
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x8b
|
|
14
|
|
vmlinux |
0xffffffff82ae8619
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0xa9
|
|
15
|
|
vmlinux |
0xffffffff82ae8a0c
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x3c
|
|
16
|
xfs_vm_writepages
|
xfs |
0xffffffffc048167b
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x6b
|
|
17
|
do_writepages
|
vmlinux |
0xffffffff82bd6a41
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x21
|
|
18
|
__writeback_single_inode
|
vmlinux |
0xffffffff82c8cc80
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x40
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffff82c8d84c
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x1cc
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff82c8db57
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0xa7
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffff82c8e033
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x263
|
|
22
|
|
vmlinux |
0xffffffff82c78ffc
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x4c
|
|
23
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff82c8ec3f
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x2cf
|
|
24
|
process_one_work
|
vmlinux |
0xffffffff82ac32ef
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x17f
|
|
25
|
worker_thread
|
vmlinux |
0xffffffff82ac4436
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x126
|
|
26
|
|
vmlinux |
0xffffffff82ac4310
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x2b0
|
|
27
|
kthread
|
vmlinux |
0xffffffff82acb621
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0xd1
|
|
28
|
|
vmlinux |
0xffffffff82acb550
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x40
|
|
29
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff831c51f7
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x21
|
|
30
|
|
vmlinux |
0xffffffff82acb550
Build id: 3.10.0-1160.90.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u12:0:18076]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay iptable_filter snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm virtio_balloon iosf_mbi snd_timer crc32_pclmul snd ghash_clmulni_intel soundcore aesni_intel joydev lrw pcspkr gf128mul glue_helper ablk_helper i2c_piix4 cryptd ip_tables xfs libcrc32c ata_generic pata_acpi virtio_console virtio_blk qxl drm_kms_helper syscopyarea ata_piix sysfillrect sysimgblt fb_sys_fops ttm libata drm crct10dif_pclmul crct10dif_common crc32c_intel e1000 virtio_pci serio_raw
virtio_ring virtio floppy drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 18076 Comm: kworker/u12:0 Kdump: loaded Not tainted 3.10.0-1160.90.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: writeback bdi_writeback_workfn (flush-253:1)
task: ffff8d9f34a1e300 ti: ffff8d9f35e04000 task.ti: ffff8d9f35e04000
RIP: 0010:[<ffffffff82b1e1ce>] [<ffffffff82b1e1ce>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8d9f35e07830 EFLAGS: 00000202
RAX: 0000000000000005 RBX: 0008000082a81b10 RCX: ffff8d9f3fd60348
RDX: 0000000000000005 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff8d9f35e07868 R08: ffff8d9e7aed9400 R09: ffffffff82d9c669
R10: ffff8d9f3fc9f160 R11: ffffe65ec8d52d00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8d9f35e077e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff8d9f3fc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbc8dfc2000 CR3: 0000000021410000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff82a81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff82a82144>] flush_tlb_page+0x54/0xa0
[<ffffffff82c141b8>] ptep_clear_flush+0x68/0xa0
[<ffffffff82c0d58e>] page_mkclean_one+0xae/0xe0
[<ffffffff82c0f403>] rmap_walk+0xf3/0x370
[<ffffffff82c0f826>] page_mkclean+0x76/0xb0
[<ffffffff82c0d4e0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff82c0cbb0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff82bd5700>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff82bd5972>] write_cache_pages+0x1f2/0x480
[<ffffffffc04816c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffff82aea1ab>] ? put_prev_entity+0x8b/0x400
[<ffffffff82ae8619>] ? pick_next_entity+0xa9/0x190
[<ffffffff82ae8a0c>] ? set_next_entity+0x3c/0xe0
[<ffffffffc048167b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffff82bd6a41>] do_writepages+0x21/0x50
[<ffffffff82c8cc80>] __writeback_single_inode+0x40/0x260
[<ffffffff82c8d84c>] writeback_sb_inodes+0x1cc/0x430
[<ffffffff82c8db57>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffff82c8e033>] wb_writeback+0x263/0x2f0
[<ffffffff82c78ffc>] ? get_nr_inodes+0x4c/0x70
[<ffffffff82c8ec3f>] bdi_writeback_workfn+0x2cf/0x470
[<ffffffff82ac32ef>] process_one_work+0x17f/0x440
[<ffffffff82ac4436>] worker_thread+0x126/0x3c0
[<ffffffff82ac4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff82acb621>] kthread+0xd1/0xe0
[<ffffffff82acb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff831c51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff82acb550>] ? insert_kthread_work+0x40/0x40
Code: 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 75 83 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #704423
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9f51173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9f47a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff9f47a734
Build id: 3.10.0-957.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff9f5fd587
Build id: 3.10.0-957.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff9f5f6b66
Build id: 3.10.0-957.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff9f5f8983
Build id: 3.10.0-957.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff9f5f8d96
Build id: 3.10.0-957.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff9f5f6ac0
Build id: 3.10.0-957.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff9f5f6200
Build id: 3.10.0-957.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff9f5c1d60
Build id: 3.10.0-957.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff9f5c200a
Build id: 3.10.0-957.el7.x86_64
|
0x22a
|
|
12
|
|
xfs |
0xffffffffc04605b0
Build id: 3.10.0-957.el7.x86_64
|
0x150
|
|
13
|
|
vmlinux |
0xffffffff9f744f30
Build id: 3.10.0-957.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffff9f42a59e
Build id: 3.10.0-957.el7.x86_64
|
0xce
|
|
15
|
|
vmlinux |
0xffffffff9f775465
Build id: 3.10.0-957.el7.x86_64
|
0x35
|
|
16
|
xfs_vm_writepages
|
xfs |
0xffffffffc04603ab
Build id: 3.10.0-957.el7.x86_64
|
0x6b
|
|
17
|
do_writepages
|
vmlinux |
0xffffffff9f5c3181
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
18
|
__writeback_single_inode
|
vmlinux |
0xffffffff9f66f930
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffff9f6703c4
Build id: 3.10.0-957.el7.x86_64
|
0x1c4
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff9f67072f
Build id: 3.10.0-957.el7.x86_64
|
0x9f
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffff9f670f63
Build id: 3.10.0-957.el7.x86_64
|
0x263
|
|
22
|
|
vmlinux |
0xffffffff9f65cd7c
Build id: 3.10.0-957.el7.x86_64
|
0x4c
|
|
23
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff9f6718eb
Build id: 3.10.0-957.el7.x86_64
|
0x2cb
|
|
24
|
process_one_work
|
vmlinux |
0xffffffff9f4b9d4f
Build id: 3.10.0-957.el7.x86_64
|
0x17f
|
|
25
|
worker_thread
|
vmlinux |
0xffffffff9f4bade6
Build id: 3.10.0-957.el7.x86_64
|
0x126
|
|
26
|
|
vmlinux |
0xffffffff9f4bacc0
Build id: 3.10.0-957.el7.x86_64
|
0x2a0
|
|
27
|
kthread
|
vmlinux |
0xffffffff9f4c1c31
Build id: 3.10.0-957.el7.x86_64
|
0xd1
|
|
28
|
|
vmlinux |
0xffffffff9f4c1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
29
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff9fb74c37
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
30
|
|
vmlinux |
0xffffffff9f4c1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kworker/u256:0:128009]
Modules linked in: veth xt_nat xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) overlay(T) sunrpc snd_seq_midi snd_seq_midi_event sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm joydev snd_timer snd pcspkr sg soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic
pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm serio_raw ata_piix mptspi libata scsi_transport_spi e1000 nfit mptscsih libnvdimm mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 128009 Comm: kworker/u256:0 Kdump: loaded Tainted: G W OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8b76ebc9d140 ti: ffff8b7625a30000 task.ti: ffff8b7625a30000
RIP: 0010:[<ffffffff9f51173a>] [<ffffffff9f51173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b7625a33820 EFLAGS: 00000202
RAX: 0000000000000006 RBX: 000800009f47a3d0 RCX: ffff8b76ed7a06a0
RDX: 0000000000000006 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8b7625a33858 R08: ffff8b7439cb4c00 R09: ffffffff9f7755c9
R10: ffff8b76ed7df160 R11: ffffefb40bee9c00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8b7625a337d0 R15: 1400000000000001
FS: 0000000000000000(0000) GS:ffff8b76ed7c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c4200ac0d0 CR3: 00000003eb316000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9f47a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9f47a734>] flush_tlb_page+0x54/0xa0
[<ffffffff9f5fd587>] ptep_clear_flush+0x67/0xa0
[<ffffffff9f5f6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffff9f5f8983>] rmap_walk+0xf3/0x360
[<ffffffff9f5f8d96>] page_mkclean+0x76/0xb0
[<ffffffff9f5f6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffff9f5f6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff9f5c1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff9f5c200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc04605b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff9f744f30>] ? submit_bio+0x70/0x150
[<ffffffff9f42a59e>] ? __switch_to+0xce/0x580
[<ffffffff9f775465>] ? cpumask_next_and+0x35/0x50
[<ffffffffc04603ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff9f5c3181>] do_writepages+0x21/0x50
[<ffffffff9f66f930>] __writeback_single_inode+0x40/0x260
[<ffffffff9f6703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff9f67072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff9f670f63>] wb_writeback+0x263/0x2f0
[<ffffffff9f65cd7c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9f6718eb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff9f4b9d4f>] process_one_work+0x17f/0x440
[<ffffffff9f4bade6>] worker_thread+0x126/0x3c0
[<ffffffff9f4bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff9f4c1c31>] kthread+0xd1/0xe0
[<ffffffff9f4c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff9fb74c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9f4c1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 14 a0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #645126
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8af1722a
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8ae7e258
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8ae7e3f4
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff8b0080f8
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff8b001696
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff8b0034a3
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff8b0038b6
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff8b0015f0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff8b000cd0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff8afca6d0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff8afca942
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc047f290
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff8aedfd28
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x78
|
|
14
|
|
vmlinux |
0xffffffff8aee64cb
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x8b
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc047f25b
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffff8afcb9c1
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffff8b07df30
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8b07eab4
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8b07edbf
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffff8b07f293
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x263
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8b07fe8b
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x2cb
|
|
22
|
process_one_work
|
vmlinux |
0xffffffff8aebdfdf
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffff8aebf0f6
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffff8aebefd0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffff8aec5fb1
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffff8aec5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff8b599df7
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x21
|
|
28
|
|
vmlinux |
0xffffffff8aec5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [kworker/u256:4:33547]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) drbg ansi_cprng ccm ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc arc4 snd_seq_midi snd_seq_midi_event rtl8192cu rtl_usb rtl8192c_common rtlwifi mac80211 snd_ens1371 sb_edac snd_rawmidi snd_ac97_codec ac97_bus cfg80211 iosf_mbi snd_seq
crc32_pclmul snd_seq_device ghash_clmulni_intel snd_pcm aesni_intel lrw snd_timer gf128mul rfkill glue_helper snd ablk_helper soundcore sg cryptd pcspkr ppdev joydev vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm e1000 nfit ata_piix libnvdimm mptspi scsi_transport_spi mptscsih libata crct10dif_pclmul crct10dif_common crc32c_intel mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 33547 Comm: kworker/u256:4 Kdump: loaded Tainted: G L ------------ T 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9d32bb85a100 ti: ffff9d33b89d8000 task.ti: ffff9d33b89d8000
RIP: 0010:[<ffffffff8af1722a>] [<ffffffff8af1722a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9d33b89db830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008ae7ddd0 RCX: ffff9d34f1620628
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9d33b89db868 R08: ffff9d33b8ccb400 R09: ffffffff8b1869d9
R10: ffff9d34f169f160 R11: ffffe3c0858c2800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9d33b89db7e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff9d34f1680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe3efd64010 CR3: 00000000a3ade000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8ae7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8ae7e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffff8b0080f8>] ptep_clear_flush+0x68/0xa0
[<ffffffff8b001696>] page_mkclean_one+0xa6/0xe0
[<ffffffff8b0034a3>] rmap_walk+0xf3/0x360
[<ffffffff8b0038b6>] page_mkclean+0x76/0xb0
[<ffffffff8b0015f0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff8b000cd0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff8afca6d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8afca942>] write_cache_pages+0x1f2/0x470
[<ffffffffc047f290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff8aedfd28>] ? __enqueue_entity+0x78/0x80
[<ffffffff8aee64cb>] ? put_prev_entity+0x8b/0x400
[<ffffffffc047f25b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff8afcb9c1>] do_writepages+0x21/0x50
[<ffffffff8b07df30>] __writeback_single_inode+0x40/0x260
[<ffffffff8b07eab4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8b07edbf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8b07f293>] wb_writeback+0x263/0x2f0
[<ffffffff8b07fe8b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff8aebdfdf>] process_one_work+0x17f/0x440
[<ffffffff8aebf0f6>] worker_thread+0x126/0x3c0
[<ffffffff8aebefd0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8aec5fb1>] kthread+0xd1/0xe0
[<ffffffff8aec5ee0>] ? insert_kthread_work+0x40/0x40
[<ffffffff8b599df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8aec5ee0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 b5 8b f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #622390
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
page_mkclean_one
|
vmlinux |
0xffffffffa200d247
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x57
|
|
2
|
rmap_walk
|
vmlinux |
0xffffffffa200f103
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xf3
|
|
3
|
page_mkclean
|
vmlinux |
0xffffffffa200f526
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x76
|
|
4
|
|
vmlinux |
0xffffffffa200d1f0
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x1b0
|
|
5
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffa200c880
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x20
|
|
6
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffa1fd55a0
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x50
|
|
7
|
write_cache_pages
|
vmlinux |
0xffffffffa1fd5812
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x1f2
|
|
8
|
|
xfs |
0xffffffffc03f16c0
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xb0
|
|
9
|
|
vmlinux |
0xffffffffa1ee84c9
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xa9
|
|
10
|
xfs_vm_writepages
|
xfs |
0xffffffffc03f167b
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x6b
|
|
11
|
do_writepages
|
vmlinux |
0xffffffffa1fd68e1
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x21
|
|
12
|
__writeback_single_inode
|
vmlinux |
0xffffffffa208c750
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x40
|
|
13
|
writeback_sb_inodes
|
vmlinux |
0xffffffffa208d31c
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x1cc
|
|
14
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffa208d627
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xa7
|
|
15
|
wb_writeback
|
vmlinux |
0xffffffffa208db03
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x263
|
|
16
|
|
vmlinux |
0xffffffffa2078a7c
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x4c
|
|
17
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffa208e70f
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x2cf
|
|
18
|
process_one_work
|
vmlinux |
0xffffffffa1ec319f
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x17f
|
|
19
|
worker_thread
|
vmlinux |
0xffffffffa1ec42e6
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x126
|
|
20
|
|
vmlinux |
0xffffffffa1ec41c0
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x2b0
|
|
21
|
kthread
|
vmlinux |
0xffffffffa1ecb4d1
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0xd1
|
|
22
|
|
vmlinux |
0xffffffffa1ecb400
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x40
|
|
23
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffa25c51dd
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x7
|
|
24
|
|
vmlinux |
0xffffffffa1ecb400
Build id: 3.10.0-1160.81.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 26s! [kworker/u129:1:55948]
Modules linked in: tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon edac_mce_amd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic vmwgfx crct10dif_common drm_kms_helper syscopyarea ata_generic sysfillrect pata_acpi sysimgblt fb_sys_fops ttm ahci mptsas drm ata_piix libahci scsi_transport_sas mptscsih libata e1000 mptbase serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 5 PID: 55948 Comm: kworker/u129:1 Kdump: loaded Tainted: G W L ------------ 3.10.0-1160.81.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8adb0c394200 ti: ffff8ae0848fc000 task.ti: ffff8ae0848fc000
RIP: 0010:[<ffffffffa200d247>] [<ffffffffa200d247>] page_mkclean_one+0x57/0xe0
RSP: 0018:ffff8ae0848ff920 EFLAGS: 00000282
RAX: 8000000271530867 RBX: 00007f184ea5c000 RCX: 8000000271530825
RDX: 0000000000000001 RSI: 00007f184ea5c000 RDI: ffffe3b60b2497f0
RBP: ffff8ae0848ff958 R08: ffff8ad9b9171808 R09: ffffffffa219c139
R10: ffff8adb24b5f160 R11: ffffe3b60a0f5400 R12: 00000000535986c1
R13: 0000000000000000 R14: 00007f184ea5c000 R15: ffff8adb23448000
FS: 00007f7618e3c780(0000) GS:ffff8adb24b40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000002f4d7af0c CR3: 00000002e0410000 CR4: 00000000000007e0
Call Trace:
[<ffffffffa200f103>] rmap_walk+0xf3/0x370
[<ffffffffa200f526>] page_mkclean+0x76/0xb0
[<ffffffffa200d1f0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffa200c880>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffa1fd55a0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa1fd5812>] write_cache_pages+0x1f2/0x480
[<ffffffffc03f16c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffffa1ee84c9>] ? pick_next_entity+0xa9/0x190
[<ffffffffc03f167b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffffa1fd68e1>] do_writepages+0x21/0x50
[<ffffffffa208c750>] __writeback_single_inode+0x40/0x260
[<ffffffffa208d31c>] writeback_sb_inodes+0x1cc/0x430
[<ffffffffa208d627>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffffa208db03>] wb_writeback+0x263/0x2f0
[<ffffffffa2078a7c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffa208e70f>] bdi_writeback_workfn+0x2cf/0x470
[<ffffffffa1ec319f>] process_one_work+0x17f/0x440
[<ffffffffa1ec42e6>] worker_thread+0x126/0x3c0
[<ffffffffa1ec41c0>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffffa1ecb4d1>] kthread+0xd1/0xe0
[<ffffffffa1ecb400>] ? insert_kthread_work+0x40/0x40
[<ffffffffa25c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa1ecb400>] ? insert_kthread_work+0x40/0x40
Code: 40 65 48 8b 04 25 28 00 00 00 48 89 45 d0 31 c0 4c 89 e6 e8 ac fb ff ff 48 85 c0 49 89 c6 74 26 31 d2 f6 00 42 75 4d 48 8b 7d c8 <c6> 07 00 66 66 66 90 85 d2 74 10 49 83 bc 24 40 03 00 00 00 75
Complete report #595892
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff84717188
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x208
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8467e258
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8467e3f4
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff84808108
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff84801686
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff848034b3
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff848038c6
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff848015e0
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff84800cc0
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff847ca700
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff847ca972
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc07a5290
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xa0
|
|
13
|
xfs_vm_writepages
|
xfs |
0xffffffffc07a525b
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x6b
|
|
14
|
do_writepages
|
vmlinux |
0xffffffff847cb9f1
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x21
|
|
15
|
__writeback_single_inode
|
vmlinux |
0xffffffff8487df50
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x40
|
|
16
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8487ead4
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x1c4
|
|
17
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8487eddf
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x9f
|
|
18
|
wb_writeback
|
vmlinux |
0xffffffff8487f2b3
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x263
|
|
19
|
|
vmlinux |
0xffffffff8486ac2c
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x4c
|
|
20
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8487feab
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x2cb
|
|
21
|
process_one_work
|
vmlinux |
0xffffffff846bdfbf
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x17f
|
|
22
|
worker_thread
|
vmlinux |
0xffffffff846bf0d6
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x126
|
|
23
|
|
vmlinux |
0xffffffff846befb0
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x2a0
|
|
24
|
kthread
|
vmlinux |
0xffffffff846c5f91
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xd1
|
|
25
|
|
vmlinux |
0xffffffff846c5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x40
|
|
26
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff84d99de4
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xe
|
|
27
|
|
vmlinux |
0xffffffff846c5ec0
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u32:9:424]
Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm dns_resolver vfat fat snd_hda_codec_hdmi ppdev edac_mce_amd snd_hda_intel snd_hda_codec kvm snd_hda_core snd_hwdep snd_seq snd_seq_device irqbypass snd_pcm crc32_pclmul ghash_clmulni_intel aesni_intel snd_timer lrw snd joydev gf128mul glue_helper ablk_helper cryptd soundcore nvme pcspkr k10temp sg ccp i2c_piix4 nvme_core parport_pc parport i2c_designware_platform gpio_amdpt i2c_designware_core pinctrl_amd acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic nouveau video mxm_wmi i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm libahci libata crct10dif_pclmul crct10dif_common crc32c_intel sdhci_acpi serio_raw r8169 iosf_mbi sdhci drm_panel_orientation_quirks mmc_core wmi dm_mirror
dm_region_hash dm_log dm_mod
CPU: 5 PID: 424 Comm: kworker/u32:9 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: Gigabyte Technology Co., Ltd. B450 AORUS M/B450 AORUS M, BIOS F5 01/25/2019
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
task: ffff8fa82c19b180 ti: ffff8faeb7d40000 task.ti: ffff8faeb7d40000
RIP: 0010:[<ffffffff84717188>] [<ffffffff84717188>] smp_call_function_many+0x208/0x270
RSP: 0018:ffff8faeb7d43830 EFLAGS: 00000202
RAX: 000000000000000e RBX: ffffffff8467ddd0 RCX: ffff8faedefa06a0
RDX: 000000000000000e RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff8faeb7d43868 R08: ffff8fa82c650c00 R09: ffffffff849869f9
R10: ffff8faeded5f160 R11: ffffe1e89f11c200 R12: 0000000000000282
R13: ffff8faeb7d437e0 R14: 0000000000000000 R15: 0000000000000000
FS: 00007fe3098db700(0000) GS:ffff8faeded40000(0000) knlGS:00000000f524f800
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002892000 CR3: 00000005f7210000 CR4: 00000000003407e0
Call Trace:
[<ffffffff8467e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8467e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffff84808108>] ptep_clear_flush+0x68/0xa0
[<ffffffff84801686>] page_mkclean_one+0xa6/0xe0
[<ffffffff848034b3>] rmap_walk+0xf3/0x360
[<ffffffff848038c6>] page_mkclean+0x76/0xb0
[<ffffffff848015e0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff84800cc0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff847ca700>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff847ca972>] write_cache_pages+0x1f2/0x470
[<ffffffffc07a5290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffc07a525b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff847cb9f1>] do_writepages+0x21/0x50
[<ffffffff8487df50>] __writeback_single_inode+0x40/0x260
[<ffffffff8487ead4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8487eddf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8487f2b3>] wb_writeback+0x263/0x2f0
[<ffffffff8486ac2c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8487feab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff846bdfbf>] process_one_work+0x17f/0x440
[<ffffffff846bf0d6>] worker_thread+0x126/0x3c0
[<ffffffff846befb0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff846c5f91>] kthread+0xd1/0xe0
[<ffffffff846c5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffff84d99de4>] ret_from_fork_nospec_begin+0xe/0x21
[<ffffffff846c5ec0>] ? insert_kthread_work+0x40/0x40
Code: 28 00 48 63 35 be 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 35 85 f6 41 20 01 74 cd 0f 1f 44 00 00 <f3> 90 f6 41 20 01 75 f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc
Complete report #595157
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffbbb170da
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffbba7e258
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffbba7e3f4
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffbbc08108
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffbbc016a6
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffbbc034b3
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffbbc038c6
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffbbc01600
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffbbc00ce0
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffbbbca6c0
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffbbbca932
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc050b290
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0xa0
|
|
13
|
xfs_vm_writepages
|
xfs |
0xffffffffc050b25b
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x6b
|
|
14
|
do_writepages
|
vmlinux |
0xffffffffbbbcb9b1
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x21
|
|
15
|
__writeback_single_inode
|
vmlinux |
0xffffffffbbc7def0
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x40
|
|
16
|
|
vmlinux |
0xffffffffbbac6f15
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x25
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffffbbc7ea74
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x1c4
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffbbc7ed7f
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x9f
|
|
19
|
wb_writeback
|
vmlinux |
0xffffffffbbc7f253
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x263
|
|
20
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffbbc7fe4b
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x2cb
|
|
21
|
process_one_work
|
vmlinux |
0xffffffffbbabde8f
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x17f
|
|
22
|
worker_thread
|
vmlinux |
0xffffffffbbabefa6
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x126
|
|
23
|
|
vmlinux |
0xffffffffbbabee80
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x2a0
|
|
24
|
kthread
|
vmlinux |
0xffffffffbbac5e61
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0xd1
|
|
25
|
|
vmlinux |
0xffffffffbbac5d90
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x40
|
|
26
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffbc195df7
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x21
|
|
27
|
|
vmlinux |
0xffffffffbbac5d90
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u256:0:87948]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi kvm_intel kvm snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq irqbypass snd_seq_device snd_pcm crc32_pclmul ghash_clmulni_intel snd_timer aesni_intel snd lrw soundcore gf128mul glue_helper ablk_helper cryptd ppdev pcspkr vmw_balloon vmw_vmci sg joydev i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic
ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul crct10dif_common crc32c_intel nfit e1000 mptspi scsi_transport_spi libnvdimm mptscsih serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 87948 Comm: kworker/u256:0 Kdump: loaded Tainted: G W L ------------ T 3.10.0-1160.42.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8a76b6beb180 ti: ffff8a74eb070000 task.ti: ffff8a74eb070000
RIP: 0010:[<ffffffffbbb170da>] [<ffffffffbbb170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8a74eb073830 EFLAGS: 00000202
RAX: 0000000000000005 RBX: 00080000bba7ddd0 RCX: ffff8a77c9760600
RDX: 0000000000000005 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8a74eb073868 R08: ffff8a75b70ccc00 R09: ffffffffbbd86bd9
R10: ffff8a77c965f160 R11: ffffc79f06516000 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8a74eb0737e0 R15: 0000000000000286
FS: 0000000000000000(0000) GS:ffff8a77c9640000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fef6400c000 CR3: 0000000165610000 CR4: 00000000001607e0
Call Trace:
[<ffffffffbba7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffbba7e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffbbc08108>] ptep_clear_flush+0x68/0xa0
[<ffffffffbbc016a6>] page_mkclean_one+0xa6/0xe0
[<ffffffffbbc034b3>] rmap_walk+0xf3/0x360
[<ffffffffbbc038c6>] page_mkclean+0x76/0xb0
[<ffffffffbbc01600>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffbbc00ce0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffbbbca6c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffbbbca932>] write_cache_pages+0x1f2/0x470
[<ffffffffc050b290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffc050b25b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffbbbcb9b1>] do_writepages+0x21/0x50
[<ffffffffbbc7def0>] __writeback_single_inode+0x40/0x260
[<ffffffffbbac6f15>] ? wake_up_bit+0x25/0x30
[<ffffffffbbc7ea74>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffbbc7ed7f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffbbc7f253>] wb_writeback+0x263/0x2f0
[<ffffffffbbc7fe4b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffbbabde8f>] process_one_work+0x17f/0x440
[<ffffffffbbabefa6>] worker_thread+0x126/0x3c0
[<ffffffffbbabee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffbbac5e61>] kthread+0xd1/0xe0
[<ffffffffbbac5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffbc195df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffbbac5d90>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 75 bc f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #591219
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffab5170da
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffab47e258
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffab47e3f4
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffab608108
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffab6016a6
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffab6034b3
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffab6038c6
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffab601600
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffab600ce0
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffab5ca6c0
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffab5ca932
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc066c290
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffabae2e49
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x69
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc066c25b
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffffab5cb9b1
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffffab67df10
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x40
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffffab67ea94
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x1c4
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffab67ed9f
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x9f
|
|
19
|
wb_writeback
|
vmlinux |
0xffffffffab67f273
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x263
|
|
20
|
|
vmlinux |
0xffffffffab66ae6c
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x4c
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffab67fe6b
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x2cb
|
|
22
|
process_one_work
|
vmlinux |
0xffffffffab4bde8f
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffffab4befa6
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffffab4bee80
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffffab4c5e61
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffffab4c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffabb99df7
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x21
|
|
28
|
|
vmlinux |
0xffffffffab4c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u24:10:17259]
Modules linked in: tcp_diag inet_diag uinput nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_service_time sd_mod crc_t10dif crct10dif_generic iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq crc32_pclmul snd_seq_device ghash_clmulni_intel snd_pcm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_timer sg joydev snd pcspkr i2c_piix4 virtio_balloon soundcore nfsd dm_multipath auth_rpcgss nfs_acl lockd grace binfmt_misc sunrpc ip_tables xfs libcrc32c sr_mod cdrom virtio_net virtio_console net_failover virtio_scsi virtio_blk failover ata_generic pata_acpi qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel
serio_raw virtio_pci floppy virtio_ring drm_panel_orientation_quirks virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 17259 Comm: kworker/u24:10 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9e59a2e14200 ti: ffff9e5447e0c000 task.ti: ffff9e5447e0c000
RIP: 0010:[<ffffffffab5170da>] [<ffffffffab5170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9e5447e0f830 EFLAGS: 00000202
RAX: 0000000000000005 RBX: 00080000ab47ddd0 RCX: ffff9e5b1f360400
RDX: 0000000000000005 RSI: 000000000000000c RDI: 0000000000000000
RBP: ffff9e5447e0f868 R08: ffff9e546cbe0000 R09: ffffffffab786c09
R10: ffff9e5b1f25f160 R11: ffffd82d93351600 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9e5447e0f7e0 R15: ffff9e5b19d31200
FS: 0000000000000000(0000) GS:ffff9e5b1f240000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6c3773a13e CR3: 00000008091b0000 CR4: 0000000000020ee0
Call Trace:
[<ffffffffab47e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffab47e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffab608108>] ptep_clear_flush+0x68/0xa0
[<ffffffffab6016a6>] page_mkclean_one+0xa6/0xe0
[<ffffffffab6034b3>] rmap_walk+0xf3/0x360
[<ffffffffab6038c6>] page_mkclean+0x76/0xb0
[<ffffffffab601600>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffab600ce0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffab5ca6c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffab5ca932>] write_cache_pages+0x1f2/0x470
[<ffffffffc066c290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffabae2e49>] ? inet_sendmsg+0x69/0xb0
[<ffffffffc066c25b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffab5cb9b1>] do_writepages+0x21/0x50
[<ffffffffab67df10>] __writeback_single_inode+0x40/0x260
[<ffffffffab67ea94>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffab67ed9f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffab67f273>] wb_writeback+0x263/0x2f0
[<ffffffffab66ae6c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffab67fe6b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffab4bde8f>] process_one_work+0x17f/0x440
[<ffffffffab4befa6>] worker_thread+0x126/0x3c0
[<ffffffffab4bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffab4c5e61>] kthread+0xd1/0xe0
[<ffffffffab4c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffabb99df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffab4c5d90>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 15 ac f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #584019
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9e91173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9e87a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff9e87a734
Build id: 3.10.0-957.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff9e9fd587
Build id: 3.10.0-957.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff9e9f6b66
Build id: 3.10.0-957.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff9e9f8983
Build id: 3.10.0-957.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff9e9f8d96
Build id: 3.10.0-957.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff9e9f6ac0
Build id: 3.10.0-957.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff9e9f6200
Build id: 3.10.0-957.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff9e9c1d60
Build id: 3.10.0-957.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff9e9c200a
Build id: 3.10.0-957.el7.x86_64
|
0x22a
|
|
12
|
|
xfs |
0xffffffffc06ca5b0
Build id: 3.10.0-957.el7.x86_64
|
0x150
|
|
13
|
|
vmlinux |
0xffffffff9eb44f30
Build id: 3.10.0-957.el7.x86_64
|
0x70
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc06ca3ab
Build id: 3.10.0-957.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffff9e9c3181
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffff9ea6f930
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffff9ea703c4
Build id: 3.10.0-957.el7.x86_64
|
0x1c4
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff9ea7072f
Build id: 3.10.0-957.el7.x86_64
|
0x9f
|
|
19
|
wb_writeback
|
vmlinux |
0xffffffff9ea70f63
Build id: 3.10.0-957.el7.x86_64
|
0x263
|
|
20
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff9ea718eb
Build id: 3.10.0-957.el7.x86_64
|
0x2cb
|
|
21
|
process_one_work
|
vmlinux |
0xffffffff9e8b9d4f
Build id: 3.10.0-957.el7.x86_64
|
0x17f
|
|
22
|
worker_thread
|
vmlinux |
0xffffffff9e8bade6
Build id: 3.10.0-957.el7.x86_64
|
0x126
|
|
23
|
|
vmlinux |
0xffffffff9e8bacc0
Build id: 3.10.0-957.el7.x86_64
|
0x2a0
|
|
24
|
kthread
|
vmlinux |
0xffffffff9e8c1c31
Build id: 3.10.0-957.el7.x86_64
|
0xd1
|
|
25
|
|
vmlinux |
0xffffffff9e8c1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
26
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff9ef74c1d
Build id: 3.10.0-957.el7.x86_64
|
0x7
|
|
27
|
|
vmlinux |
0xffffffff9e8c1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 21s! [kworker/u32:2:16545]
Modules linked in: binfmt_misc vmw_vsock_vmci_transport vsock sunrpc vfat fat ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper sg cryptd vmw_vmci pcspkr joydev parport_pc parport i2c_piix4 tpm_crb ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic e1000e crct10dif_pclmul crct10dif_common vmwgfx crc32c_intel serio_raw mptsas ptp pps_core scsi_transport_sas drm_kms_helper mptscsih mptbase syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm ata_piix nfit libata libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 12 PID: 16545 Comm: kworker/u32:2 Kdump: loaded Tainted: G W L ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.17369862.B64.2012240522 12/24/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff925475355140 ti: ffff92533074c000 task.ti: ffff92533074c000
RIP: 0010:[<ffffffff9e91173a>] [<ffffffff9e91173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff92533074f820 EFLAGS: 00000202
RAX: 000000000000000e RBX: 000800009e87a3d0 RCX: ffff92549f5a0768
RDX: 000000000000000e RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff92533074f858 R08: ffff924df4bfe000 R09: ffffffff9eb755c9
R10: ffff92549f51f160 R11: fffff1435b5f8800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff92533074f7d0 R15: ffff924dffc03300
FS: 0000000000000000(0000) GS:ffff92549f500000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f085a38e700 CR3: 00000005bb210000 CR4: 0000000000760fe0
PKRU: 00000000
Call Trace:
[<ffffffff9e87a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9e87a734>] flush_tlb_page+0x54/0xa0
[<ffffffff9e9fd587>] ptep_clear_flush+0x67/0xa0
[<ffffffff9e9f6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffff9e9f8983>] rmap_walk+0xf3/0x360
[<ffffffff9e9f8d96>] page_mkclean+0x76/0xb0
[<ffffffff9e9f6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffff9e9f6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff9e9c1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff9e9c200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc06ca5b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff9eb44f30>] ? submit_bio+0x70/0x150
[<ffffffffc06ca3ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff9e9c3181>] do_writepages+0x21/0x50
[<ffffffff9ea6f930>] __writeback_single_inode+0x40/0x260
[<ffffffff9ea703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff9ea7072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff9ea70f63>] wb_writeback+0x263/0x2f0
[<ffffffff9ea718eb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff9e8b9d4f>] process_one_work+0x17f/0x440
[<ffffffff9e8bade6>] worker_thread+0x126/0x3c0
[<ffffffff9e8bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff9e8c1c31>] kthread+0xd1/0xe0
[<ffffffff9e8c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff9ef74c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff9e8c1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 54 9f f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #528499
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff88f1185a
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff88e7a5c8
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff88e7a764
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff88ffd937
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff88ff6f16
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff88ff8d33
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff88ff9146
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff88ff6e70
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff88ff65b0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff88fc2010
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff88fc22ba
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x22a
|
|
12
|
|
xfs |
0xffffffffc03945b0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x150
|
|
13
|
|
vmlinux |
0xffffffff88ee064c
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x11c
|
|
14
|
|
vmlinux |
0xffffffff89175a45
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x35
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc03943ab
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffff88fc3431
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffff8906fee0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffff89070974
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff89070cdf
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffff89071513
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffff8905d32c
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff89071e9b
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffff88eb9d8f
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffff88ebae26
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffff88ebad00
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffff88ec1c71
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffff88ec1ba0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff89574c1d
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x7
|
|
29
|
|
vmlinux |
0xffffffff88ec1ba0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [kworker/u128:2:60626]
Modules linked in: vfat fat uas usb_storage cmac arc4 md4 nls_utf8 cifs ccm dns_resolver fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun nfsv3 nfs_acl bridge nfs stp llc lockd grace fscache ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter mvfs(OE) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec joydev ac97_bus snd_seq pcspkr snd_seq_device snd_pcm snd_timer snd sg soundcore vmw_vmci i2c_piix4 nfit libnvdimm parport_pc
parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw ata_piix syscopyarea sysfillrect sysimgblt libata fb_sys_fops ttm mptspi scsi_transport_spi mptscsih drm e1000 mptbase drm_panel_orientation_quirks floppy
CPU: 5 PID: 60626 Comm: kworker/u128:2 Kdump: loaded Tainted: G W OEL ------------ 3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff8abe59691040 ti: ffff8abe53698000 task.ti: ffff8abe53698000
RIP: 0010:[<ffffffff88f1185a>] [<ffffffff88f1185a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8abe5369b820 EFLAGS: 00000202
RAX: 0000000000000003 RBX: ffffffff88e7a400 RCX: ffff8abe764e0650
RDX: 0000000000000003 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff8abe5369b858 R08: ffff8ab9b6644800 R09: ffffffff89175ba9
R10: ffff8abe7655f160 R11: ffffd236c1009000 R12: 0000000000000282
R13: ffff8abe5369b7d0 R14: ffffffffffffffff R15: ffffffffffffffff
FS: 0000000000000000(0000) GS:ffff8abe76540000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc48a24b000 CR3: 00000004d4c10000 CR4: 00000000001607e0
Call Trace:
[<ffffffff88e7a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff88e7a764>] flush_tlb_page+0x54/0xa0
[<ffffffff88ffd937>] ptep_clear_flush+0x67/0xa0
[<ffffffff88ff6f16>] page_mkclean_one+0xa6/0xe0
[<ffffffff88ff8d33>] rmap_walk+0xf3/0x360
[<ffffffff88ff9146>] page_mkclean+0x76/0xb0
[<ffffffff88ff6e70>] ? page_referenced_one+0x180/0x180
[<ffffffff88ff65b0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff88fc2010>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff88fc22ba>] write_cache_pages+0x22a/0x4e0
[<ffffffffc03945b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff88ee064c>] ? dequeue_entity+0x11c/0x5e0
[<ffffffff89175a45>] ? cpumask_next_and+0x35/0x50
[<ffffffffc03943ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff88fc3431>] do_writepages+0x21/0x50
[<ffffffff8906fee0>] __writeback_single_inode+0x40/0x260
[<ffffffff89070974>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff89070cdf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff89071513>] wb_writeback+0x263/0x2f0
[<ffffffff8905d32c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff89071e9b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff88eb9d8f>] process_one_work+0x17f/0x440
[<ffffffff88ebae26>] worker_thread+0x126/0x3c0
[<ffffffff88ebad00>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff88ec1c71>] kthread+0xd1/0xe0
[<ffffffff88ec1ba0>] ? insert_kthread_work+0x40/0x40
[<ffffffff89574c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff88ec1ba0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 2e 82 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 b4 89 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 fd 81 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #514292
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa431722a
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa427e258
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffa427e3f4
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffa44080f8
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffa4401696
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffa44034a3
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffa44038b6
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffa44015f0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffa4400cd0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffa43ca6d0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffa43ca942
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc03aa290
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffa42e64cb
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x8b
|
|
14
|
|
vmlinux |
0xffffffffa42e283c
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x3c
|
|
15
|
|
vmlinux |
0xffffffffa4586825
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x35
|
|
16
|
xfs_vm_writepages
|
xfs |
0xffffffffc03aa25b
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x6b
|
|
17
|
do_writepages
|
vmlinux |
0xffffffffa43cb9c1
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x21
|
|
18
|
__writeback_single_inode
|
vmlinux |
0xffffffffa447df30
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x40
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffffa447eab4
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x1c4
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffa447edbf
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x9f
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffffa447f293
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x263
|
|
22
|
|
vmlinux |
0xffffffffa446ac0c
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x4c
|
|
23
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffa447fe8b
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x2cb
|
|
24
|
process_one_work
|
vmlinux |
0xffffffffa42bdfdf
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x17f
|
|
25
|
worker_thread
|
vmlinux |
0xffffffffa42bf0f6
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x126
|
|
26
|
|
vmlinux |
0xffffffffa42befd0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x2a0
|
|
27
|
kthread
|
vmlinux |
0xffffffffa42c5fb1
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xd1
|
|
28
|
|
vmlinux |
0xffffffffa42c5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x40
|
|
29
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffa4999df7
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x21
|
|
30
|
|
vmlinux |
0xffffffffa42c5ee0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u16:1:28783]
Modules linked in: tcp_diag udp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat ipt_REJECT nf_reject_ipv4 tun nf_conntrack_ipv4 nf_defrag_ipv4 bridge xt_conntrack stp nf_conntrack llc ebtable_filter ebtables devlink ip6table_filter ip6_tables nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_set ip_set iptable_filter sunrpc snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore ppdev sg virtio_balloon parport_pc parport i2c_piix4 pcspkr joydev ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi cirrus drm_kms_helper virtio_net net_failover syscopyarea virtio_blk
failover sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata virtio_pci serio_raw virtio_ring floppy virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 28783 Comm: kworker/u16:1 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff907b81988000 ti: ffff90780b0c4000 task.ti: ffff90780b0c4000
RIP: 0010:[<ffffffffa431722a>] [<ffffffffa431722a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff90780b0c7830 EFLAGS: 00000202
RAX: 0000000000000003 RBX: ffffffffa427ddd0 RCX: ffff907bcb8e0400
RDX: 0000000000000003 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff90780b0c7868 R08: ffff906b4fd48800 R09: ffffffffa45869d9
R10: ffff907bcb85f160 R11: ffffefd23a1e6a00 R12: ffff90780b0c7800
R13: ffffffffa458692f R14: 1f140f0a0b0c77c8 R15: ffff907a00000000
FS: 0000000000000000(0000) GS:ffff907bcb840000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0d3415e000 CR3: 0000000939410000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa427e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa427e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffa44080f8>] ptep_clear_flush+0x68/0xa0
[<ffffffffa4401696>] page_mkclean_one+0xa6/0xe0
[<ffffffffa44034a3>] rmap_walk+0xf3/0x360
[<ffffffffa44038b6>] page_mkclean+0x76/0xb0
[<ffffffffa44015f0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffa4400cd0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffa43ca6d0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa43ca942>] write_cache_pages+0x1f2/0x470
[<ffffffffc03aa290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffa42e64cb>] ? put_prev_entity+0x8b/0x400
[<ffffffffa42e283c>] ? set_next_entity+0x3c/0xe0
[<ffffffffa4586825>] ? cpumask_next_and+0x35/0x50
[<ffffffffc03aa25b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa43cb9c1>] do_writepages+0x21/0x50
[<ffffffffa447df30>] __writeback_single_inode+0x40/0x260
[<ffffffffa447eab4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffa447edbf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa447f293>] wb_writeback+0x263/0x2f0
[<ffffffffa446ac0c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffa447fe8b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa42bdfdf>] process_one_work+0x17f/0x440
[<ffffffffa42bf0f6>] worker_thread+0x126/0x3c0
[<ffffffffa42befd0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffa42c5fb1>] kthread+0xd1/0xe0
[<ffffffffa42c5ee0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa4999df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa42c5ee0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 f5 a4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #489282
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9af1720a
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9ae7e258
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff9ae7e3f4
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff9b008128
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff9b0016a6
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff9b0034d3
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff9b0038e6
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff9b001600
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff9b000ce0
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff9afca720
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff9afca992
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc03e8290
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0xa0
|
|
13
|
xfs_vm_writepages
|
xfs |
0xffffffffc03e825b
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x6b
|
|
14
|
do_writepages
|
vmlinux |
0xffffffff9afcba11
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x21
|
|
15
|
__writeback_single_inode
|
vmlinux |
0xffffffff9b07df70
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x40
|
|
16
|
|
vmlinux |
0xffffffff9aec7045
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x25
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffff9b07eaf4
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x1c4
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff9b07edff
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x9f
|
|
19
|
wb_writeback
|
vmlinux |
0xffffffff9b07f2d3
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x263
|
|
20
|
|
vmlinux |
0xffffffff9b06ac4c
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x4c
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff9b07fecb
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x2cb
|
|
22
|
process_one_work
|
vmlinux |
0xffffffff9aebdfbf
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffff9aebf0d6
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffff9aebefb0
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffff9aec5f91
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffff9aec5ec0
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff9b599df7
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x21
|
|
28
|
|
vmlinux |
0xffffffff9aec5ec0
Build id: 3.10.0-1160.71.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u8:1:32308]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink sunrpc snd_intel8x0 snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm snd_timer sg iosf_mbi snd parport_pc soundcore parport i2c_piix4 pcspkr video ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw ahci libahci ata_piix drm crc32c_intel e1000 libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 32308 Comm: kworker/u8:1 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.71.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9a6cce9b1080 ti: ffff9a6c240e8000 task.ti: ffff9a6c240e8000
RIP: 0010:[<ffffffff9af1720a>] [<ffffffff9af1720a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9a6c240eb830 EFLAGS: 00010202
RAX: 0000000000000001 RBX: ffff9a6c240eb7f0 RCX: ffff9a6d1fc9fc10
RDX: 0000000000000001 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff9a6c240eb868 R08: ffff9a6d1b152000 R09: ffffffff9b186a19
R10: ffff9a6d1fc1f160 R11: fffff812412ee500 R12: ffff9a6c240eb800
R13: ffffffff9b18696f R14: 00000000000000fc R15: 00000000000008fc
FS: 0000000000000000(0000) GS:ffff9a6d1fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f57af1f4d00 CR3: 0000000023810000 CR4: 00000000000206f0
Call Trace:
[<ffffffff9ae7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9ae7e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffff9b008128>] ptep_clear_flush+0x68/0xa0
[<ffffffff9b0016a6>] page_mkclean_one+0xa6/0xe0
[<ffffffff9b0034d3>] rmap_walk+0xf3/0x360
[<ffffffff9b0038e6>] page_mkclean+0x76/0xb0
[<ffffffff9b001600>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff9b000ce0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff9afca720>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff9afca992>] write_cache_pages+0x1f2/0x470
[<ffffffffc03e8290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffc03e825b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff9afcba11>] do_writepages+0x21/0x50
[<ffffffff9b07df70>] __writeback_single_inode+0x40/0x260
[<ffffffff9aec7045>] ? wake_up_bit+0x25/0x30
[<ffffffff9b07eaf4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff9b07edff>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff9b07f2d3>] wb_writeback+0x263/0x2f0
[<ffffffff9b06ac4c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9b07fecb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff9aebdfbf>] process_one_work+0x17f/0x440
[<ffffffff9aebf0d6>] worker_thread+0x126/0x3c0
[<ffffffff9aebefb0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff9aec5f91>] kthread+0xd1/0xe0
[<ffffffff9aec5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffff9b599df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9aec5ec0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 9e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 b5 9b f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 6d 87 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #483081
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff87b1173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff87a7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff87a7a734
Build id: 3.10.0-957.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff87bfd587
Build id: 3.10.0-957.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff87bf6b66
Build id: 3.10.0-957.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff87bf8983
Build id: 3.10.0-957.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff87bf8d96
Build id: 3.10.0-957.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff87bf6ac0
Build id: 3.10.0-957.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff87bf6200
Build id: 3.10.0-957.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff87bc1d60
Build id: 3.10.0-957.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff87bc200a
Build id: 3.10.0-957.el7.x86_64
|
0x22a
|
|
12
|
|
xfs |
0xffffffffc05d35b0
Build id: 3.10.0-957.el7.x86_64
|
0x150
|
|
13
|
|
vmlinux |
0xffffffff87ad0887
Build id: 3.10.0-957.el7.x86_64
|
0x57
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc05d33ab
Build id: 3.10.0-957.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffff87bc3181
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffff87c6f930
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffff87c703c4
Build id: 3.10.0-957.el7.x86_64
|
0x1c4
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff87c7072f
Build id: 3.10.0-957.el7.x86_64
|
0x9f
|
|
19
|
wb_writeback
|
vmlinux |
0xffffffff87c70f63
Build id: 3.10.0-957.el7.x86_64
|
0x263
|
|
20
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff87c718eb
Build id: 3.10.0-957.el7.x86_64
|
0x2cb
|
|
21
|
process_one_work
|
vmlinux |
0xffffffff87ab9d4f
Build id: 3.10.0-957.el7.x86_64
|
0x17f
|
|
22
|
worker_thread
|
vmlinux |
0xffffffff87abade6
Build id: 3.10.0-957.el7.x86_64
|
0x126
|
|
23
|
|
vmlinux |
0xffffffff87abacc0
Build id: 3.10.0-957.el7.x86_64
|
0x2a0
|
|
24
|
kthread
|
vmlinux |
0xffffffff87ac1c31
Build id: 3.10.0-957.el7.x86_64
|
0xd1
|
|
25
|
|
vmlinux |
0xffffffff87ac1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
26
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff88174c37
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
27
|
|
vmlinux |
0xffffffff87ac1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [kworker/u256:1:20619]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw snd_ens1371 gf128mul glue_helper ablk_helper cryptd vmw_balloon snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer joydev snd soundcore pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel
vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata e1000 mptspi scsi_transport_spi mptscsih drm_panel_orientation_quirks mptbase nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 20619 Comm: kworker/u256:1 Kdump: loaded Tainted: G L ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9b01bc416180 ti: ffff9b02516b8000 task.ti: ffff9b02516b8000
RIP: 0010:[<ffffffff87b1173a>] [<ffffffff87b1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9b02516bb820 EFLAGS: 00000202
RAX: 0000000000000004 RBX: 0008000087a7a3d0 RCX: ffff9b0275720490
RDX: 0000000000000004 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9b02516bb858 R08: ffff9b01bc85bc00 R09: ffffffff87d755c9
R10: ffff9b027565f160 R11: fffffc0d0899ba00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9b02516bb7d0 R15: 1400000000000001
FS: 0000000000000000(0000) GS:ffff9b0275640000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0dfcd23300 CR3: 00000001fcc10000 CR4: 00000000001607e0
Call Trace:
[<ffffffff87a7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff87a7a734>] flush_tlb_page+0x54/0xa0
[<ffffffff87bfd587>] ptep_clear_flush+0x67/0xa0
[<ffffffff87bf6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffff87bf8983>] rmap_walk+0xf3/0x360
[<ffffffff87bf8d96>] page_mkclean+0x76/0xb0
[<ffffffff87bf6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffff87bf6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff87bc1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff87bc200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc05d35b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff87ad0887>] ? finish_task_switch+0x57/0x1c0
[<ffffffffc05d33ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff87bc3181>] do_writepages+0x21/0x50
[<ffffffff87c6f930>] __writeback_single_inode+0x40/0x260
[<ffffffff87c703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff87c7072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff87c70f63>] wb_writeback+0x263/0x2f0
[<ffffffff87c718eb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff87ab9d4f>] process_one_work+0x17f/0x440
[<ffffffff87abade6>] worker_thread+0x126/0x3c0
[<ffffffff87abacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff87ac1c31>] kthread+0xd1/0xe0
[<ffffffff87ac1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff88174c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff87ac1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 74 88 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #471296
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9e116e4e
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9e07e868
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff9e07ea04
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff9e207908
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff9e200eb6
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff9e202cc3
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff9e2030d6
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff9e200e10
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff9e2004f0
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff9e1c9c60
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff9e1c9ed2
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc0448290
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xa0
|
|
13
|
xfs_vm_writepages
|
xfs |
0xffffffffc044825b
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x6b
|
|
14
|
do_writepages
|
vmlinux |
0xffffffff9e1caf51
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x21
|
|
15
|
__writeback_single_inode
|
vmlinux |
0xffffffff9e27d8d0
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x40
|
|
16
|
|
vmlinux |
0xffffffff9e0c6ec5
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x25
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffff9e27e454
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x1c4
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff9e27e75f
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x9f
|
|
19
|
wb_writeback
|
vmlinux |
0xffffffff9e27ec33
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x263
|
|
20
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff9e27f72c
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x1cc
|
|
21
|
process_one_work
|
vmlinux |
0xffffffff9e0bde3f
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x17f
|
|
22
|
worker_thread
|
vmlinux |
0xffffffff9e0bef56
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x126
|
|
23
|
|
vmlinux |
0xffffffff9e0bee30
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x2a0
|
|
24
|
kthread
|
vmlinux |
0xffffffff9e0c5e11
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xd1
|
|
25
|
|
vmlinux |
0xffffffff9e0c5d40
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x40
|
|
26
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff9e794df7
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x21
|
|
27
|
|
vmlinux |
0xffffffff9e0c5d40
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#20 stuck for 22s! [kworker/u256:2:474]
Modules linked in: bmhook(OE) tmhook(OE) dsa_filter(POE) dsa_filter_hook(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 ppdev vmw_balloon sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport
i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel nfit libata mptspi libnvdimm scsi_transport_spi serio_raw mptscsih mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 20 PID: 474 Comm: kworker/u256:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff983f65b93180 ti: ffff983f9a3b8000 task.ti: ffff983f9a3b8000
RIP: 0010:[<ffffffff9e116e4e>] [<ffffffff9e116e4e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff983f9a3bb830 EFLAGS: 00000202
RAX: 000000000000000a RBX: 000800009e07e6a0 RCX: ffff984a4d8a08f8
RDX: 000000000000000a RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff983f9a3bb868 R08: ffff983f61bafc00 R09: ffffffff9e386479
R10: ffff984a4db1f160 R11: ffffe8b2abb36c00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff983f9a3bb7e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff984a4db00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f59fa6e6ed8 CR3: 00000002f0210000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9e07e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9e07ea04>] flush_tlb_page+0x54/0xa0
[<ffffffff9e207908>] ptep_clear_flush+0x68/0xa0
[<ffffffff9e200eb6>] page_mkclean_one+0xa6/0xe0
[<ffffffff9e202cc3>] rmap_walk+0xf3/0x360
[<ffffffff9e2030d6>] page_mkclean+0x76/0xb0
[<ffffffff9e200e10>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff9e2004f0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff9e1c9c60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff9e1c9ed2>] write_cache_pages+0x1f2/0x470
[<ffffffffc0448290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffc044825b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff9e1caf51>] do_writepages+0x21/0x50
[<ffffffff9e27d8d0>] __writeback_single_inode+0x40/0x260
[<ffffffff9e0c6ec5>] ? wake_up_bit+0x25/0x30
[<ffffffff9e27e454>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff9e27e75f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff9e27ec33>] wb_writeback+0x263/0x2f0
[<ffffffff9e27f72c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffff9e0bde3f>] process_one_work+0x17f/0x440
[<ffffffff9e0bef56>] worker_thread+0x126/0x3c0
[<ffffffff9e0bee30>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff9e0c5e11>] kthread+0xd1/0xe0
[<ffffffff9e0c5d40>] ? insert_kthread_work+0x40/0x40
[<ffffffff9e794df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9e0c5d40>] ? insert_kthread_work+0x40/0x40
Code: 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 d5 9e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 ad 86 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #468671
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9fd16a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9fc7e478
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff9fc7e614
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff9fe05a48
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff9fdfefd6
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff9fe00e03
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff9fe01216
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff9fdfef30
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff9fdfe610
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff9fdc9900
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff9fdc9b72
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc04a9030
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff9fce29fc
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x3c
|
|
14
|
|
vmlinux |
0xffffffff9fc2b59e
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xce
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc04a8ffb
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffff9fdcabe1
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffff9fe798c0
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffff9fe7a454
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff9fe7a75f
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffff9fe7ac43
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffff9fe66aec
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff9fe7b83b
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffff9fcbe21f
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffff9fcbf336
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffff9fcbf210
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffff9fcc61f1
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffff9fcc6120
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffa038dd37
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x21
|
|
29
|
|
vmlinux |
0xffffffff9fcc6120
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kworker/u320:3:9659]
Modules linked in: kbase(OE) xt_NFQUEUE nfnetlink_log nfnetlink nfring(OE) tcp_lp fuse vmsecmod(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss nf_conntrack_ipv6 nf_defrag_ipv6 nfsv4 nf_conntrack_ipv4 nf_defrag_ipv4 dns_resolver nfs nf_conntrack lockd grace fscache sunrpc ppdev parport_pc sg parport virtio_balloon i2c_piix4 pcspkr joydev ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_blk virtio_console virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci virtio_ring floppy serio_raw virtio
drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vmsec_nfq]
CPU: 7 PID: 9659 Comm: kworker/u320:3 Kdump: loaded Tainted: GF OEL ------------ 3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8c921c271070 ti: ffff8c9b6cbf8000 task.ti: ffff8c9b6cbf8000
RIP: 0010:[<ffffffff9fd16a7a>] [<ffffffff9fd16a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8c9b6cbfb830 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 000800009fc7e2b0 RCX: ffff8ca17ce604f0
RDX: 0000000000000001 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff8c9b6cbfb868 R08: ffff8c92e95dc400 R09: ffffffff9ff81839
R10: ffff8ca17cfdf160 R11: fffff26b3e4cf800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8c9b6cbfb7e0 R15: ffff8ca17ce31874
FS: 0000000000000000(0000) GS:ffff8ca17cfc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6d73560871 CR3: 00000009fd810000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9fc7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9fc7e614>] flush_tlb_page+0x54/0xa0
[<ffffffff9fe05a48>] ptep_clear_flush+0x68/0xa0
[<ffffffff9fdfefd6>] page_mkclean_one+0xa6/0xe0
[<ffffffff9fe00e03>] rmap_walk+0xf3/0x360
[<ffffffff9fe01216>] page_mkclean+0x76/0xb0
[<ffffffff9fdfef30>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff9fdfe610>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff9fdc9900>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff9fdc9b72>] write_cache_pages+0x1f2/0x470
[<ffffffffc04a9030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff9fce29fc>] ? set_next_entity+0x3c/0xe0
[<ffffffff9fc2b59e>] ? __switch_to+0xce/0x580
[<ffffffffc04a8ffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff9fdcabe1>] do_writepages+0x21/0x50
[<ffffffff9fe798c0>] __writeback_single_inode+0x40/0x260
[<ffffffff9fe7a454>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff9fe7a75f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff9fe7ac43>] wb_writeback+0x263/0x2f0
[<ffffffff9fe66aec>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9fe7b83b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff9fcbe21f>] process_one_work+0x17f/0x440
[<ffffffff9fcbf336>] worker_thread+0x126/0x3c0
[<ffffffff9fcbf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff9fcc61f1>] kthread+0xd1/0xe0
[<ffffffff9fcc6120>] ? insert_kthread_work+0x40/0x40
[<ffffffffa038dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9fcc6120>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 94 a0 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #395834
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa9916e4a
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa987e868
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffa987ea04
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffa9a07908
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffa9a00eb6
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffa9a02cc3
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffa9a030d6
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffa9a00e10
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffa9a004f0
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffa99c9c60
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffa99c9ed2
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc03d4290
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffa982b59e
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xce
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc03d425b
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffffa99caf51
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffffa9a7d8d0
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x40
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffffa9a7e454
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x1c4
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffa9a7e75f
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x9f
|
|
19
|
wb_writeback
|
vmlinux |
0xffffffffa9a7ec33
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x263
|
|
20
|
|
vmlinux |
0xffffffffa9a6a85c
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x4c
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffa9a7f82b
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x2cb
|
|
22
|
process_one_work
|
vmlinux |
0xffffffffa98bde3f
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffffa98bef56
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffffa98bee30
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffffa98c5e11
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffffa98c5d40
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffa9f94de4
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xe
|
|
28
|
|
vmlinux |
0xffffffffa98c5d40
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#173 stuck for 22s! [kworker/u510:6:201565]
Modules linked in: rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache tcp_lp uinput devlink binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13402(E) snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep qxl snd_seq snd_seq_device ttm snd_pcm drm_kms_helper snd_timer crc32_pclmul syscopyarea sysfillrect
ghash_clmulni_intel aesni_intel iTCO_wdt sysimgblt iTCO_vendor_support ppdev fb_sys_fops snd lrw gf128mul glue_helper ablk_helper parport_pc joydev cryptd drm i2c_i801 soundcore virtio_rng virtio_balloon lpc_ich pcspkr parport drm_panel_orientation_quirks nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c virtio_net net_failover virtio_console virtio_blk failover ahci virtio_pci crct10dif_pclmul libahci crct10dif_common libata crc32c_intel virtio_ring serio_raw virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 173 PID: 201565 Comm: kworker/u510:6 Kdump: loaded Tainted: P EL ------------ 3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.13.0-2.module+el8.4.0+534+4680a14e 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8b18f977c200 ti: ffff8b20167a0000 task.ti: ffff8b20167a0000
RIP: 0010:[<ffffffffa9916e4a>] [<ffffffffa9916e4a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b20167a3830 EFLAGS: 00000202
RAX: 0000000000000036 RBX: 00080000a987e6a0 RCX: ffff8d012c3a1f18
RDX: 0000000000000036 RSI: 00000000000000ff RDI: 0000000000000000
RBP: ffff8b20167a3868 R08: ffff8b1636763c00 R09: ffffffffa9b86479
R10: ffff8d012e15f160 R11: ffffdb2fc8c4d800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8b20167a37e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff8d012e140000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000003829008 CR3: 00000103e9410000 CR4: 0000000000760ee0
PKRU: 00000000
Call Trace:
[<ffffffffa987e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa987ea04>] flush_tlb_page+0x54/0xa0
[<ffffffffa9a07908>] ptep_clear_flush+0x68/0xa0
[<ffffffffa9a00eb6>] page_mkclean_one+0xa6/0xe0
[<ffffffffa9a02cc3>] rmap_walk+0xf3/0x360
[<ffffffffa9a030d6>] page_mkclean+0x76/0xb0
[<ffffffffa9a00e10>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffa9a004f0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffa99c9c60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa99c9ed2>] write_cache_pages+0x1f2/0x470
[<ffffffffc03d4290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffa982b59e>] ? __switch_to+0xce/0x580
[<ffffffffc03d425b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa99caf51>] do_writepages+0x21/0x50
[<ffffffffa9a7d8d0>] __writeback_single_inode+0x40/0x260
[<ffffffffa9a7e454>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffa9a7e75f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa9a7ec33>] wb_writeback+0x263/0x2f0
[<ffffffffa9a6a85c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffa9a7f82b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa98bde3f>] process_one_work+0x17f/0x440
[<ffffffffa98bef56>] worker_thread+0x126/0x3c0
[<ffffffffa98bee30>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffa98c5e11>] kthread+0xd1/0xe0
[<ffffffffa98c5d40>] ? insert_kthread_work+0x40/0x40
[<ffffffffa9f94de4>] ret_from_fork_nospec_begin+0xe/0x21
[<ffffffffa98c5d40>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 de 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 55 aa f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ad 86 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #362472
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa5d170da
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa5c7e258
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffa5c7e3f4
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffa5e08108
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffa5e016a6
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffa5e034b3
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffa5e038c6
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffa5e01600
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffa5e00ce0
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffa5dca6c0
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffa5dca932
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc0584290
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffa5e64987
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x47
|
|
14
|
|
vmlinux |
0xffffffffa5f86a25
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x35
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc058425b
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffffa5dcb9b1
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffffa5e7def0
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffffa5e7ea74
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffa5e7ed7f
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffffa5e7f253
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffffa5e6ae4c
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffa5e7fe4b
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffa5cbde8f
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffa5cbefa6
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffa5cbee80
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffffa5cc5e61
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffa5cc5d90
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffa6395ddd
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x7
|
|
29
|
|
vmlinux |
0xffffffffa5cc5d90
Build id: 3.10.0-1160.42.2.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u256:1:18818]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log snd_seq_midi snd_seq_midi_event dm_mod iosf_mbi crc32_pclmul snd_ens1371 snd_rawmidi ghash_clmulni_intel snd_ac97_codec ac97_bus snd_seq snd_seq_device ppdev snd_pcm aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper
cryptd snd_timer snd joydev pcspkr soundcore sg vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libnvdimm mptspi scsi_transport_spi libata mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks fuse
CPU: 3 PID: 18818 Comm: kworker/u256:1 Kdump: loaded Not tainted 3.10.0-1160.42.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff8c16d7122100 ti: ffff8c1553504000 task.ti: ffff8c1553504000
RIP: 0010:[<ffffffffa5d170da>] [<ffffffffa5d170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8c1553507830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a5c7ddd0 RCX: ffff8c16f2620650
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8c1553507868 R08: ffff8c15784ccc00 R09: ffffffffa5f86bd9
R10: ffff8c16f26df160 R11: fffffba9492fee00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8c15535077e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff8c16f26c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000010f10d7 CR3: 0000000258a98000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa5c7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa5c7e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffa5e08108>] ptep_clear_flush+0x68/0xa0
[<ffffffffa5e016a6>] page_mkclean_one+0xa6/0xe0
[<ffffffffa5e034b3>] rmap_walk+0xf3/0x360
[<ffffffffa5e038c6>] page_mkclean+0x76/0xb0
[<ffffffffa5e01600>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffa5e00ce0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffa5dca6c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa5dca932>] write_cache_pages+0x1f2/0x470
[<ffffffffc0584290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffa5e64987>] ? poll_freewait+0x47/0xa0
[<ffffffffa5f86a25>] ? cpumask_next_and+0x35/0x50
[<ffffffffc058425b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa5dcb9b1>] do_writepages+0x21/0x50
[<ffffffffa5e7def0>] __writeback_single_inode+0x40/0x260
[<ffffffffa5e7ea74>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffa5e7ed7f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa5e7f253>] wb_writeback+0x263/0x2f0
[<ffffffffa5e6ae4c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffa5e7fe4b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa5cbde8f>] process_one_work+0x17f/0x440
[<ffffffffa5cbefa6>] worker_thread+0x126/0x3c0
[<ffffffffa5cbee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffa5cc5e61>] kthread+0xd1/0xe0
[<ffffffffa5cc5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffa6395ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa5cc5d90>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 95 a6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #361781
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8d3170da
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8d27e258
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8d27e3f4
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff8d408108
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff8d4016a6
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff8d4034b3
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff8d4038c6
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff8d401600
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff8d400ce0
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff8d3ca6c0
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff8d3ca932
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc02c3290
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff8d555e70
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffff8d586a45
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x35
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc02c325b
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffff8d3cb9b1
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffff8d47df10
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8d47ea94
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8d47ed9f
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffff8d47f273
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffff8d46ae6c
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8d47fe6b
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffff8d2bde8f
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffff8d2befa6
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffff8d2bee80
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffff8d2c5e61
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffff8d2c5d90
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff8d995df7
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x21
|
|
29
|
|
vmlinux |
0xffffffff8d2c5d90
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u12:0:24369]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13005(E) snd_hda_codec_generic snd_hda_intel iosf_mbi snd_hda_codec snd_hda_core crc32_pclmul ghash_clmulni_intel snd_hwdep snd_seq aesni_intel snd_seq_device lrw snd_pcm gf128mul glue_helper ablk_helper cryptd virtio_rng snd_timer ppdev joydev parport_pc snd soundcore pcspkr virtio_balloon i2c_piix4 parport ip_tables xfs libcrc32c ata_generic pata_acpi virtio_blk virtio_console qxl drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata e1000 crct10dif_pclmul crct10dif_common virtio_pci crc32c_intel virtio_ring serio_raw floppy virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 24369 Comm: kworker/u12:0 Tainted: P E ------------ 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.13.0-2.module+el8.3.0+7860+a7792d29 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:6)
task: ffff8b2be7776300 ti: ffff8b2a01444000 task.ti: ffff8b2a01444000
RIP: 0010:[<ffffffff8d3170da>] [<ffffffff8d3170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b2a01447830 EFLAGS: 00000202
RAX: 0000000000000004 RBX: 000800008d27ddd0 RCX: ffff8b2d3fd20348
RDX: 0000000000000004 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff8b2a01447868 R08: ffff8b2a76311400 R09: ffffffff8d586bf9
R10: ffff8b2d3fc9f160 R11: ffffe5688f985c00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8b2a014477e0 R15: ffff8b2ca04b0ea0
FS: 0000000000000000(0000) GS:ffff8b2d3fc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9a0af0e050 CR3: 0000000332210000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8d27e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8d27e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffff8d408108>] ptep_clear_flush+0x68/0xa0
[<ffffffff8d4016a6>] page_mkclean_one+0xa6/0xe0
[<ffffffff8d4034b3>] rmap_walk+0xf3/0x360
[<ffffffff8d4038c6>] page_mkclean+0x76/0xb0
[<ffffffff8d401600>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff8d400ce0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff8d3ca6c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8d3ca932>] write_cache_pages+0x1f2/0x470
[<ffffffffc02c3290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff8d555e70>] ? submit_bio+0x70/0x150
[<ffffffff8d586a45>] ? cpumask_next_and+0x35/0x50
[<ffffffffc02c325b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff8d3cb9b1>] do_writepages+0x21/0x50
[<ffffffff8d47df10>] __writeback_single_inode+0x40/0x260
[<ffffffff8d47ea94>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8d47ed9f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8d47f273>] wb_writeback+0x263/0x2f0
[<ffffffff8d46ae6c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8d47fe6b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff8d2bde8f>] process_one_work+0x17f/0x440
[<ffffffff8d2befa6>] worker_thread+0x126/0x3c0
[<ffffffff8d2bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8d2c5e61>] kthread+0xd1/0xe0
[<ffffffff8d2c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffff8d995df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8d2c5d90>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 f5 8d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #349890
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffac716eaa
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffac67e868
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffac67ea04
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffac807908
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffac800eb6
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffac802cc3
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffac8030d6
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffac800e10
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffac8004f0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffac7c9cc0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffac7c9f32
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc0442290
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffac955610
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffffac62b59e
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xce
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc044225b
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffffac7cafb1
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffffac87d8d0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x40
|
|
18
|
|
vmlinux |
0xffffffffac6c6f25
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x25
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffffac87e454
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x1c4
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffac87e75f
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x9f
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffffac87ec33
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x263
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffac87f82b
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffac6bde9f
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffac6befb6
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffac6bee90
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffffac6c5e71
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffac6c5da0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffacd93ddd
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x7
|
|
29
|
|
vmlinux |
0xffffffffac6c5da0
Build id: 3.10.0-1160.11.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/u256:0:97646]
Modules linked in: tcp_lp veth xt_nat xt_CHECKSUM iptable_mangle nf_conntrack_netlink ipt_MASQUERADE nfnetlink nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ipt_REJECT nf_reject_ipv4 xt_addrtype xt_conntrack nf_nat tun nf_conntrack br_netfilter bridge stp llc overlay(T) ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd vmw_vmci pcspkr joydev sg vmw_balloon i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect nfit sysimgblt fb_sys_fops ttm libnvdimm drm ata_piix e1000
libata crct10dif_pclmul crct10dif_common mptspi scsi_transport_spi crc32c_intel mptscsih mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 97646 Comm: kworker/u256:0 Tainted: G W L ------------ T 3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
task: ffff9528d9a3e300 ti: ffff9528e2b48000 task.ti: ffff9528e2b48000
RIP: 0010:[<ffffffffac716eaa>] [<ffffffffac716eaa>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff9528e2b4b830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000ac67e6a0 RCX: ffff952af1620650
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9528e2b4b868 R08: ffff952928fccc00 R09: ffffffffac986199
R10: ffff952af16df160 R11: ffffee204717ba00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9528e2b4b7e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff952af16c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9d00c18000 CR3: 0000000318b04000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffffac67e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffac67ea04>] flush_tlb_page+0x54/0xa0
[<ffffffffac807908>] ptep_clear_flush+0x68/0xa0
[<ffffffffac800eb6>] page_mkclean_one+0xa6/0xe0
[<ffffffffac802cc3>] rmap_walk+0xf3/0x360
[<ffffffffac8030d6>] page_mkclean+0x76/0xb0
[<ffffffffac800e10>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffac8004f0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffac7c9cc0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffac7c9f32>] write_cache_pages+0x1f2/0x470
[<ffffffffc0442290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffac955610>] ? submit_bio+0x70/0x150
[<ffffffffac62b59e>] ? __switch_to+0xce/0x580
[<ffffffffc044225b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffac7cafb1>] do_writepages+0x21/0x50
[<ffffffffac87d8d0>] __writeback_single_inode+0x40/0x260
[<ffffffffac6c6f25>] ? wake_up_bit+0x25/0x30
[<ffffffffac87e454>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffac87e75f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffac87ec33>] wb_writeback+0x263/0x2f0
[<ffffffffac87f82b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffac6bde9f>] process_one_work+0x17f/0x440
[<ffffffffac6befb6>] worker_thread+0x126/0x3c0
[<ffffffffac6bee90>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffac6c5e71>] kthread+0xd1/0xe0
[<ffffffffac6c5da0>] ? insert_kthread_work+0x40/0x40
[<ffffffffacd93ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffac6c5da0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 7e 86 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 35 ad f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 86 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #342967
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffadd16a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffadc7e478
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffadc7e614
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffade05a48
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffaddfefd6
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffade00e03
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffade01216
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffaddfef30
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffaddfe610
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffaddc9900
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffaddc9b72
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc05f5030
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffadf51020
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x70
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc05f4ffb
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffffaddcabe1
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffffade798c0
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
|
17
|
|
vmlinux |
0xffffffffadcc72a5
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x25
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffffade7a454
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffade7a75f
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffffade7ac43
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffffade66aec
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffade7b83b
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffadcbe21f
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffadcbf336
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffadcbf210
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffffadcc61f1
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffadcc6120
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffae38dd37
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x21
|
|
29
|
|
vmlinux |
0xffffffffadcc6120
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [kworker/u320:0:15052]
Modules linked in: tcp_lp fuse kbase(OE) xt_NFQUEUE nfnetlink_log nfnetlink nfring(OE) vmsecmod(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc ppdev joydev virtio_balloon sg i2c_piix4 pcspkr parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_net virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci virtio_ring serio_raw floppy virtio
drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vmsec_nfq]
CPU: 11 PID: 15052 Comm: kworker/u320:0 Kdump: loaded Tainted: GF OEL ------------ 3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9c2366788000 ti: ffff9c2be8e0c000 task.ti: ffff9c2be8e0c000
RIP: 0010:[<ffffffffadd16a7a>] [<ffffffffadd16a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9c2be8e0f830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000adc7e2b0 RCX: ffff9c2cfce20590
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9c2be8e0f868 R08: ffff9c1e695dd400 R09: ffffffffadf81839
R10: ffff9c2cfd0df160 R11: fffff054b58ba400 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9c2be8e0f7e0 R15: 0000000180080008
FS: 0000000000000000(0000) GS:ffff9c2cfd0c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fee937fea08 CR3: 0000000226a10000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffadc7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffadc7e614>] flush_tlb_page+0x54/0xa0
[<ffffffffade05a48>] ptep_clear_flush+0x68/0xa0
[<ffffffffaddfefd6>] page_mkclean_one+0xa6/0xe0
[<ffffffffade00e03>] rmap_walk+0xf3/0x360
[<ffffffffade01216>] page_mkclean+0x76/0xb0
[<ffffffffaddfef30>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffaddfe610>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffaddc9900>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffaddc9b72>] write_cache_pages+0x1f2/0x470
[<ffffffffc05f5030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffadf51020>] ? submit_bio+0x70/0x150
[<ffffffffc05f4ffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffaddcabe1>] do_writepages+0x21/0x50
[<ffffffffade798c0>] __writeback_single_inode+0x40/0x260
[<ffffffffadcc72a5>] ? wake_up_bit+0x25/0x30
[<ffffffffade7a454>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffade7a75f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffade7ac43>] wb_writeback+0x263/0x2f0
[<ffffffffade66aec>] ? get_nr_inodes+0x4c/0x70
[<ffffffffade7b83b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffadcbe21f>] process_one_work+0x17f/0x440
[<ffffffffadcbf336>] worker_thread+0x126/0x3c0
[<ffffffffadcbf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffadcc61f1>] kthread+0xd1/0xe0
[<ffffffffadcc6120>] ? insert_kthread_work+0x40/0x40
[<ffffffffae38dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffadcc6120>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 94 ae f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #332672
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff98b1185a
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff98a7a5c8
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff98a7a764
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff98bfd937
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff98bf6f16
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff98bf8d33
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff98bf9146
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff98bf6e70
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff98bf65b0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff98bc2010
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff98bc22ba
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x22a
|
|
12
|
|
xfs |
0xffffffffc057f5b0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x150
|
|
13
|
|
vmlinux |
0xffffffff98d45510
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x70
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc057f3ab
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffff98bc3431
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffff98c6fee0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x40
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffff98c70974
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x1c4
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff98c70cdf
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x9f
|
|
19
|
wb_writeback
|
vmlinux |
0xffffffff98c71513
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x263
|
|
20
|
|
vmlinux |
0xffffffff98c5d32c
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x4c
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff98c71e9b
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x2cb
|
|
22
|
process_one_work
|
vmlinux |
0xffffffff98ab9d8f
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffff98abae26
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffff98abad00
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffff98ac1c71
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffff98ac1ba0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff99174c1d
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x7
|
|
28
|
|
vmlinux |
0xffffffff98ac1ba0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u128:1:754]
Modules linked in: vfat fat uas usb_storage cmac arc4 md4 nls_utf8 cifs ccm dns_resolver fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc nfsv3 nfs_acl nfs ebtable_filter lockd ebtables grace fscache ip6table_filter ip6_tables devlink iptable_filter mvfs(OE) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi snd_ac97_codec pcspkr joydev ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd sg soundcore vmw_vmci i2c_piix4 nfit libnvdimm parport_pc
parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw ata_piix syscopyarea sysfillrect sysimgblt fb_sys_fops ttm libata drm e1000 mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks floppy
CPU: 2 PID: 754 Comm: kworker/u128:1 Kdump: loaded Tainted: G OE ------------ 3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff97174fb22080 ti: ffff971356984000 task.ti: ffff971356984000
RIP: 0010:[<ffffffff98b1185a>] [<ffffffff98b1185a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff971356987820 EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffff98a7a400 RCX: ffff9718f64204b8
RDX: 0000000000000000 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff971356987858 R08: ffff971436643c00 R09: ffffffff98d75ba9
R10: ffff9718f649f160 R11: ffffdddccdb6b600 R12: 0000000000000282
R13: ffff9713569877d0 R14: 000000008bfdf1ca R15: 1400000000000001
FS: 0000000000000000(0000) GS:ffff9718f6480000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdee4c4b0a0 CR3: 0000000366010000 CR4: 00000000001607e0
Call Trace:
[<ffffffff98a7a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff98a7a764>] flush_tlb_page+0x54/0xa0
[<ffffffff98bfd937>] ptep_clear_flush+0x67/0xa0
[<ffffffff98bf6f16>] page_mkclean_one+0xa6/0xe0
[<ffffffff98bf8d33>] rmap_walk+0xf3/0x360
[<ffffffff98bf9146>] page_mkclean+0x76/0xb0
[<ffffffff98bf6e70>] ? page_referenced_one+0x180/0x180
[<ffffffff98bf65b0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff98bc2010>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff98bc22ba>] write_cache_pages+0x22a/0x4e0
[<ffffffffc057f5b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff98d45510>] ? submit_bio+0x70/0x150
[<ffffffffc057f3ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff98bc3431>] do_writepages+0x21/0x50
[<ffffffff98c6fee0>] __writeback_single_inode+0x40/0x260
[<ffffffff98c70974>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff98c70cdf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff98c71513>] wb_writeback+0x263/0x2f0
[<ffffffff98c5d32c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff98c71e9b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff98ab9d8f>] process_one_work+0x17f/0x440
[<ffffffff98abae26>] worker_thread+0x126/0x3c0
[<ffffffff98abad00>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff98ac1c71>] kthread+0xd1/0xe0
[<ffffffff98ac1ba0>] ? insert_kthread_work+0x40/0x40
[<ffffffff99174c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff98ac1ba0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 2e 82 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 74 99 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 fd 81 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #310693
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff83116cba
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8307e608
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8307e7a4
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff83205638
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff831febe6
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff832009f3
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff83200e06
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff831feb40
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff831fe220
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff831c93b0
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff831c9622
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc0518030
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff833511e0
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x70
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc0517ffb
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffff831ca6a1
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffff83279b00
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x40
|
|
17
|
|
vmlinux |
0xffffffff830c73d5
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x25
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8327a694
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8327a99f
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffff8327ae83
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x263
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8327ba7b
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x2cb
|
|
22
|
process_one_work
|
vmlinux |
0xffffffff830be34f
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffff830bf466
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffff830bf340
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffff830c6321
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffff830c6250
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff8378dd37
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x21
|
|
28
|
|
vmlinux |
0xffffffff830c6250
Build id: 3.10.0-1062.18.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [kworker/u65:2:23376]
Modules linked in: binfmt_misc cmac arc4 md4 nls_utf8 cifs ccm dns_resolver ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_comment xt_mark iptable_filter iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_netlink nfnetlink iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi overlay(T) vmw_vsock_vmci_transport falcon_lsm_serviceable(PE) vsock falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_12803(E) ext4 sb_edac mbcache iosf_mbi jbd2 crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper pcc_cpufreq cryptd ppdev vmw_balloon pcspkr joydev parport_pc parport vmw_vmci sg i2c_piix4 br_netfilter bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack
auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi nfit crct10dif_pclmul libata crct10dif_common crc32c_intel mptscsih libnvdimm serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 11 PID: 23376 Comm: kworker/u65:2 Kdump: loaded Tainted: P EL ------------ T 3.10.0-1062.18.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff89033d00b150 ti: ffff88fca5824000 task.ti: ffff88fca5824000
RIP: 0010:[<ffffffff83116cba>] [<ffffffff83116cba>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff88fca5827830 EFLAGS: 00000202
RAX: 000000000000001f RBX: 000800008307e440 RCX: ffff89133f5e07f0
RDX: 000000000000001f RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff88fca5827868 R08: ffff88f4a8259c00 R09: ffffffff833819f9
R10: ffff89033fcdf160 R11: ffffc4febacbb400 R12: 0000000000000286
R13: 0000000000000286 R14: ffff88fca58277e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff89033fcc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6d2f1903e8 CR3: 0000000169810000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8307e608>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8307e7a4>] flush_tlb_page+0x54/0xa0
[<ffffffff83205638>] ptep_clear_flush+0x68/0xa0
[<ffffffff831febe6>] page_mkclean_one+0xa6/0xe0
[<ffffffff832009f3>] rmap_walk+0xf3/0x360
[<ffffffff83200e06>] page_mkclean+0x76/0xb0
[<ffffffff831feb40>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff831fe220>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff831c93b0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff831c9622>] write_cache_pages+0x1f2/0x470
[<ffffffffc0518030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff833511e0>] ? submit_bio+0x70/0x150
[<ffffffffc0517ffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff831ca6a1>] do_writepages+0x21/0x50
[<ffffffff83279b00>] __writeback_single_inode+0x40/0x260
[<ffffffff830c73d5>] ? wake_up_bit+0x25/0x30
[<ffffffff8327a694>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8327a99f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8327ae83>] wb_writeback+0x263/0x2f0
[<ffffffff8327ba7b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff830be34f>] process_one_work+0x17f/0x440
[<ffffffff830bf466>] worker_thread+0x126/0x3c0
[<ffffffff830bf340>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff830c6321>] kthread+0xd1/0xe0
[<ffffffff830c6250>] ? insert_kthread_work+0x40/0x40
[<ffffffff8378dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff830c6250>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ee 5c c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 e8 d4 83 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5c c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #289745
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb751173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb747a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffb747a734
Build id: 3.10.0-957.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffb75fd587
Build id: 3.10.0-957.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffb75f6b66
Build id: 3.10.0-957.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffb75f8983
Build id: 3.10.0-957.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffb75f8d96
Build id: 3.10.0-957.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffb75f6ac0
Build id: 3.10.0-957.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffb75f6200
Build id: 3.10.0-957.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffb75c1d60
Build id: 3.10.0-957.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffb75c200a
Build id: 3.10.0-957.el7.x86_64
|
0x22a
|
|
12
|
|
xfs |
0xffffffffc04695b0
Build id: 3.10.0-957.el7.x86_64
|
0x150
|
|
13
|
|
vmlinux |
0xffffffffb7744f30
Build id: 3.10.0-957.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffffb7775465
Build id: 3.10.0-957.el7.x86_64
|
0x35
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc04693ab
Build id: 3.10.0-957.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffffb75c3181
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffffb766f930
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffffb76703c4
Build id: 3.10.0-957.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffb767072f
Build id: 3.10.0-957.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffffb7670f63
Build id: 3.10.0-957.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffffb765cd7c
Build id: 3.10.0-957.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffb76718eb
Build id: 3.10.0-957.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffb74b9d4f
Build id: 3.10.0-957.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffb74bade6
Build id: 3.10.0-957.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffb74bacc0
Build id: 3.10.0-957.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffffb74c1c31
Build id: 3.10.0-957.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffb74c1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb7b74c37
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
29
|
|
vmlinux |
0xffffffffb74c1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u256:1:87737]
Modules linked in: veth xt_nat xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock fuse vmhgfs(OE) overlay(T) sunrpc snd_seq_midi snd_seq_midi_event coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel snd_ens1371 lrw snd_rawmidi gf128mul glue_helper snd_ac97_codec ablk_helper cryptd ac97_bus snd_seq snd_seq_device ppdev snd_pcm snd_timer vmw_balloon snd soundcore pcspkr sg joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul
crct10dif_common crc32c_intel ata_generic pata_acpi serio_raw vmwgfx e1000 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi mptscsih libata mptbase drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 87737 Comm: kworker/u256:1 Kdump: loaded Tainted: G OEL ------------ T 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffffa0d2ee8630c0 ti: ffffa0d3534f8000 task.ti: ffffa0d3534f8000
RIP: 0010:[<ffffffffb751173a>] [<ffffffffb751173a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffffa0d3534fb820 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00080000b747a3d0 RCX: ffffa0d4126e04b8
RDX: 0000000000000003 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffffa0d3534fb858 R08: ffffa0d3bd25c400 R09: ffffffffb77755c9
R10: ffffa0d41269f160 R11: ffffdfb306fbc600 R12: 0000000000000286
R13: 0000000000000286 R14: ffffa0d3534fb7d0 R15: 1400000000000001
FS: 0000000000000000(0000) GS:ffffa0d412680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055fb73410270 CR3: 00000000b4498000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb747a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb747a734>] flush_tlb_page+0x54/0xa0
[<ffffffffb75fd587>] ptep_clear_flush+0x67/0xa0
[<ffffffffb75f6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffffb75f8983>] rmap_walk+0xf3/0x360
[<ffffffffb75f8d96>] page_mkclean+0x76/0xb0
[<ffffffffb75f6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffffb75f6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb75c1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb75c200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc04695b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffb7744f30>] ? submit_bio+0x70/0x150
[<ffffffffb7775465>] ? cpumask_next_and+0x35/0x50
[<ffffffffc04693ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb75c3181>] do_writepages+0x21/0x50
[<ffffffffb766f930>] __writeback_single_inode+0x40/0x260
[<ffffffffb76703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffb767072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb7670f63>] wb_writeback+0x263/0x2f0
[<ffffffffb765cd7c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffb76718eb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb74b9d4f>] process_one_work+0x17f/0x440
[<ffffffffb74bade6>] worker_thread+0x126/0x3c0
[<ffffffffb74bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffb74c1c31>] kthread+0xd1/0xe0
[<ffffffffb74c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffb7b74c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb74c1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 14 b8 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #246018
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa1316bda
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa127e258
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffa127e3f4
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffa1407bf8
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffa14011a6
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffa1402fb3
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffa14033c6
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffa1401100
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffa14007e0
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffa13ca1c0
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffa13ca432
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc0318290
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffa12e4a1c
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x11c
|
|
14
|
|
vmlinux |
0xffffffffa12e6021
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x31
|
|
15
|
|
vmlinux |
0xffffffffa12e22e9
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xa9
|
|
16
|
|
vmlinux |
0xffffffffa12e258c
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x3c
|
|
17
|
xfs_vm_writepages
|
xfs |
0xffffffffc031825b
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x6b
|
|
18
|
do_writepages
|
vmlinux |
0xffffffffa13cb4b1
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x21
|
|
19
|
__writeback_single_inode
|
vmlinux |
0xffffffffa147d9d0
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x40
|
|
20
|
writeback_sb_inodes
|
vmlinux |
0xffffffffa147e554
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x1c4
|
|
21
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffa147e85f
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x9f
|
|
22
|
wb_writeback
|
vmlinux |
0xffffffffa147ed33
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x263
|
|
23
|
|
vmlinux |
0xffffffffa146a95c
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x4c
|
|
24
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffa147f92b
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x2cb
|
|
25
|
process_one_work
|
vmlinux |
0xffffffffa12bde5f
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x17f
|
|
26
|
worker_thread
|
vmlinux |
0xffffffffa12bef76
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x126
|
|
27
|
|
vmlinux |
0xffffffffa12bee50
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x2a0
|
|
28
|
kthread
|
vmlinux |
0xffffffffa12c5e31
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0xd1
|
|
29
|
|
vmlinux |
0xffffffffa12c5d60
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x40
|
|
30
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffa1995df7
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x21
|
|
31
|
|
vmlinux |
0xffffffffa12c5d60
Build id: 3.10.0-1160.31.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [kworker/u8:3:24159]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables vsock_diag tcp_diag udp_diag inet_diag unix_diag tun devlink bridge stp llc ip_set vmw_vsock_vmci_transport vsock iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport vmw_vmci i2c_piix4 nfsd nfs_acl lockd grace auth_rpcgss
sunrpc xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw ttm vmxnet3 mptspi drm ata_piix scsi_transport_spi mptscsih nfit mptbase libata libnvdimm drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ip_tables]
CPU: 1 PID: 24159 Comm: kworker/u8:3 Kdump: loaded Tainted: G W L ------------ 3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9df328830000 ti: ffff9df34dd10000 task.ti: ffff9df34dd10000
RIP: 0010:[<ffffffffa1316bda>] [<ffffffffa1316bda>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9df34dd13830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffa127ddd0 RCX: ffff9df4ffc20600
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff9df34dd13868 R08: ffff9df236359800 R09: ffffffffa15866a9
R10: ffff9df4ffc9f160 R11: ffffe90e507ce200 R12: ffff9df34dd13800
R13: ffffffffa15865ff R14: 0000000000000000 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff9df4ffc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7751ae0000 CR3: 000000042db7e000 CR4: 00000000000407e0
Call Trace:
[<ffffffffa127e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa127e3f4>] flush_tlb_page+0x54/0xa0
[<ffffffffa1407bf8>] ptep_clear_flush+0x68/0xa0
[<ffffffffa14011a6>] page_mkclean_one+0xa6/0xe0
[<ffffffffa1402fb3>] rmap_walk+0xf3/0x360
[<ffffffffa14033c6>] page_mkclean+0x76/0xb0
[<ffffffffa1401100>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffa14007e0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffa13ca1c0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa13ca432>] write_cache_pages+0x1f2/0x470
[<ffffffffc0318290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffa12e4a1c>] ? dequeue_entity+0x11c/0x5c0
[<ffffffffa12e6021>] ? put_prev_entity+0x31/0x400
[<ffffffffa12e22e9>] ? pick_next_entity+0xa9/0x190
[<ffffffffa12e258c>] ? set_next_entity+0x3c/0xe0
[<ffffffffc031825b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa13cb4b1>] do_writepages+0x21/0x50
[<ffffffffa147d9d0>] __writeback_single_inode+0x40/0x260
[<ffffffffa147e554>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffa147e85f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa147ed33>] wb_writeback+0x263/0x2f0
[<ffffffffa146a95c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffa147f92b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa12bde5f>] process_one_work+0x17f/0x440
[<ffffffffa12bef76>] worker_thread+0x126/0x3c0
[<ffffffffa12bee50>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffa12c5e31>] kthread+0xd1/0xe0
[<ffffffffa12c5d60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa1995df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa12c5d60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ce 8a c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 f5 a1 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 8a c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #169507
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff94f16eea
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff94e7e868
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff94e7ea04
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff950079d8
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff95000f86
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff95002d93
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff950031a6
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff95000ee0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff950005c0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff94fc9d10
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff94fc9f82
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc02dd290
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff94ee2499
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xa9
|
|
14
|
|
vmlinux |
0xffffffff94ee273c
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x3c
|
|
15
|
|
vmlinux |
0xffffffff94e2b59e
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xce
|
|
16
|
xfs_vm_writepages
|
xfs |
0xffffffffc02dd25b
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x6b
|
|
17
|
do_writepages
|
vmlinux |
0xffffffff94fcb001
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x21
|
|
18
|
__writeback_single_inode
|
vmlinux |
0xffffffff9507d9a0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x40
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffff9507e524
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x1c4
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff9507e82f
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x9f
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffff9507ed03
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x263
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff9507f8fb
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffff94ebddcf
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffff94ebeee6
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffff94ebedc0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffff94ec5da1
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffff94ec5cd0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff95595ddd
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x7
|
|
29
|
|
vmlinux |
0xffffffff94ec5cd0
Build id: 3.10.0-1160.25.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u256:3:58535]
Modules linked in: veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache overlay nf_conntrack_netbios_ns nf_conntrack_broadcast xt_nat xt_mark ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash
dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul snd_ens1371 glue_helper snd_rawmidi ablk_helper snd_ac97_codec ppdev cryptd ac97_bus snd_seq joydev snd_seq_device snd_pcm vmw_balloon pcspkr snd_timer snd soundcore sg vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sch_fq_codel sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi nfit ata_piix crct10dif_pclmul crct10dif_common scsi_transport_spi libata mptscsih crc32c_intel libnvdimm serio_raw drm_panel_orientation_quirks e1000 mptbase fuse
CPU: 2 PID: 58535 Comm: kworker/u256:3 Kdump: loaded Tainted: G W L ------------ 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:5)
task: ffff8c1746d19080 ti: ffff8c19494e4000 task.ti: ffff8c19494e4000
RIP: 0010:[<ffffffff94f16eea>] [<ffffffff94f16eea>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8c19494e7830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0008000094e7e6a0 RCX: ffff8c1975620628
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8c19494e7868 R08: ffff8c18be34b400 R09: ffffffff95186549
R10: ffff8c197569f160 R11: ffffd2ecc69a3400 R12: 0000000000000286
R13: 0000000000000018 R14: ffff8c19494e77a0 R15: 0000000000000286
FS: 0000000000000000(0000) GS:ffff8c1975680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2415647228 CR3: 00000001cf648000 CR4: 00000000003607e0
Call Trace:
[<ffffffff94e7e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff94e7ea04>] flush_tlb_page+0x54/0xa0
[<ffffffff950079d8>] ptep_clear_flush+0x68/0xa0
[<ffffffff95000f86>] page_mkclean_one+0xa6/0xe0
[<ffffffff95002d93>] rmap_walk+0xf3/0x360
[<ffffffff950031a6>] page_mkclean+0x76/0xb0
[<ffffffff95000ee0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff950005c0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff94fc9d10>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff94fc9f82>] write_cache_pages+0x1f2/0x470
[<ffffffffc02dd290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff94ee2499>] ? pick_next_entity+0xa9/0x190
[<ffffffff94ee273c>] ? set_next_entity+0x3c/0xe0
[<ffffffff94e2b59e>] ? __switch_to+0xce/0x580
[<ffffffffc02dd25b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff94fcb001>] do_writepages+0x21/0x50
[<ffffffff9507d9a0>] __writeback_single_inode+0x40/0x260
[<ffffffff9507e524>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff9507e82f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff9507ed03>] wb_writeback+0x263/0x2f0
[<ffffffff9507f8fb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff94ebddcf>] process_one_work+0x17f/0x440
[<ffffffff94ebeee6>] worker_thread+0x126/0x3c0
[<ffffffff94ebedc0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff94ec5da1>] kthread+0xd1/0xe0
[<ffffffff94ec5cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffff95595ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff94ec5cd0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 fe 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 b5 95 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 87 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #143128
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa1b1185a
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa1a7a5c8
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffa1a7a764
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffa1bfd937
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffa1bf6f16
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffa1bf8d33
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffa1bf9146
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffa1bf6e70
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffa1bf65b0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffa1bc2010
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffa1bc22ba
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x22a
|
|
12
|
|
xfs |
0xffffffffc06635b0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x150
|
|
13
|
|
vmlinux |
0xffffffffa1d45510
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffffa1d75a45
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x35
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc06633ab
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffffa1bc3431
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffffa1c6fee0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffffa1c70974
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffa1c70cdf
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffffa1c71513
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffffa1c5d32c
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffa1c71e9b
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffa1ab9d8f
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffa1abae26
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffa1abad00
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffffa1ac1c71
Build id: 3.10.0-957.5.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffa1ac1ba0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffa2174c1d
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x7
|
|
29
|
|
vmlinux |
0xffffffffa1ac1ba0
Build id: 3.10.0-957.5.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [kworker/u128:0:40519]
Modules linked in: vfat fat uas usb_storage cmac arc4 md4 nls_utf8 cifs ccm dns_resolver fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc nfsv3 nfs_acl ebtable_filter nfs ebtables lockd grace fscache ip6table_filter ip6_tables devlink iptable_filter mvfs(OE) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev snd_ens1371 snd_rawmidi pcspkr snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd sg soundcore nfit i2c_piix4 vmw_vmci libnvdimm parport_pc
parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm e1000 drm mptspi scsi_transport_spi ata_piix mptscsih mptbase libata drm_panel_orientation_quirks floppy
CPU: 6 PID: 40519 Comm: kworker/u128:0 Kdump: loaded Tainted: G W OE ------------ 3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff8e5004750000 ti: ffff8e516cfb0000 task.ti: ffff8e516cfb0000
RIP: 0010:[<ffffffffa1b1185a>] [<ffffffffa1b1185a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8e516cfb3820 EFLAGS: 00000202
RAX: 0000000000000007 RBX: ffffffffa1a7a400 RCX: ffff8e52765e0678
RDX: 0000000000000007 RSI: 0000000000000040 RDI: 0000000000000000
RBP: ffff8e516cfb3858 R08: ffff8e4db6644c00 R09: ffffffffa1d75ba9
R10: ffff8e527659f160 R11: fffff97cd6599200 R12: 0000000000000282
R13: ffff8e516cfb37d0 R14: 00000000c2dee95a R15: 1400000000000001
FS: 0000000000000000(0000) GS:ffff8e5276580000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6cd385da92 CR3: 000000021aa10000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa1a7a5c8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa1a7a764>] flush_tlb_page+0x54/0xa0
[<ffffffffa1bfd937>] ptep_clear_flush+0x67/0xa0
[<ffffffffa1bf6f16>] page_mkclean_one+0xa6/0xe0
[<ffffffffa1bf8d33>] rmap_walk+0xf3/0x360
[<ffffffffa1bf9146>] page_mkclean+0x76/0xb0
[<ffffffffa1bf6e70>] ? page_referenced_one+0x180/0x180
[<ffffffffa1bf65b0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffa1bc2010>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa1bc22ba>] write_cache_pages+0x22a/0x4e0
[<ffffffffc06635b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffa1d45510>] ? submit_bio+0x70/0x150
[<ffffffffa1d75a45>] ? cpumask_next_and+0x35/0x50
[<ffffffffc06633ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa1bc3431>] do_writepages+0x21/0x50
[<ffffffffa1c6fee0>] __writeback_single_inode+0x40/0x260
[<ffffffffa1c70974>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffa1c70cdf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa1c71513>] wb_writeback+0x263/0x2f0
[<ffffffffa1c5d32c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffa1c71e9b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa1ab9d8f>] process_one_work+0x17f/0x440
[<ffffffffa1abae26>] worker_thread+0x126/0x3c0
[<ffffffffa1abad00>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffa1ac1c71>] kthread+0xd1/0xe0
[<ffffffffa1ac1ba0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa2174c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa1ac1ba0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 2e 82 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 74 a2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 fd 81 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #87552
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff85f1173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff85e7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff85e7a734
Build id: 3.10.0-957.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff85ffd587
Build id: 3.10.0-957.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff85ff6b66
Build id: 3.10.0-957.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff85ff8983
Build id: 3.10.0-957.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff85ff8d96
Build id: 3.10.0-957.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff85ff6ac0
Build id: 3.10.0-957.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff85ff6200
Build id: 3.10.0-957.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff85fc1d60
Build id: 3.10.0-957.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff85fc200a
Build id: 3.10.0-957.el7.x86_64
|
0x22a
|
|
12
|
|
xfs |
0xffffffffc04845b0
Build id: 3.10.0-957.el7.x86_64
|
0x150
|
|
13
|
|
vmlinux |
0xffffffff861467de
Build id: 3.10.0-957.el7.x86_64
|
0xce
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc04843ab
Build id: 3.10.0-957.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffff85fc3181
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffff8606f930
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffff860703c4
Build id: 3.10.0-957.el7.x86_64
|
0x1c4
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8607072f
Build id: 3.10.0-957.el7.x86_64
|
0x9f
|
|
19
|
wb_writeback
|
vmlinux |
0xffffffff86070f63
Build id: 3.10.0-957.el7.x86_64
|
0x263
|
|
20
|
|
vmlinux |
0xffffffff8605cd7c
Build id: 3.10.0-957.el7.x86_64
|
0x4c
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff860718eb
Build id: 3.10.0-957.el7.x86_64
|
0x2cb
|
|
22
|
process_one_work
|
vmlinux |
0xffffffff85eb9d4f
Build id: 3.10.0-957.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffff85ebade6
Build id: 3.10.0-957.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffff85ebacc0
Build id: 3.10.0-957.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffff85ec1c31
Build id: 3.10.0-957.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffff85ec1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff86574c37
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
28
|
|
vmlinux |
0xffffffff85ec1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [kworker/u320:0:65563]
Modules linked in: binfmt_misc ip6t_MASQUERADE nf_nat_masquerade_ipv6 xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_mark ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables nfnetlink_queue nfnetlink_log bluetooth rfkill ip_set nfnetlink bridge stp llc ext4 mbcache jbd2 ppdev sg parport_pc parport i2c_piix4 virtio_balloon joydev pcspkr xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_scsi virtio_blk cirrus
serio_raw drm_kms_helper floppy syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 12 PID: 65563 Comm: kworker/u320:0 Kdump: loaded Tainted: G L ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff89f0407e8000 ti: ffff89eefa470000 task.ti: ffff89eefa470000
RIP: 0010:[<ffffffff85f1173a>] [<ffffffff85f1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff89eefa473820 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0008000085e7a3d0 RCX: ffff89f69ce20578
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff89eefa473858 R08: ffff89eff44ae400 R09: ffffffff861755c9
R10: ffff89f69d11f160 R11: ffffe64f202b6000 R12: 0000000000000286
R13: 0000000000000286 R14: ffff89eefa4737d0 R15: 0000000000000080
FS: 0000000000000000(0000) GS:ffff89f69d100000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000a7c2c0 CR3: 000000080ae10000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff85e7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff85e7a734>] flush_tlb_page+0x54/0xa0
[<ffffffff85ffd587>] ptep_clear_flush+0x67/0xa0
[<ffffffff85ff6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffff85ff8983>] rmap_walk+0xf3/0x360
[<ffffffff85ff8d96>] page_mkclean+0x76/0xb0
[<ffffffff85ff6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffff85ff6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff85fc1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff85fc200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc04845b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff861467de>] ? blk_flush_plug_list+0xce/0x230
[<ffffffffc04843ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff85fc3181>] do_writepages+0x21/0x50
[<ffffffff8606f930>] __writeback_single_inode+0x40/0x260
[<ffffffff860703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff8607072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff86070f63>] wb_writeback+0x263/0x2f0
[<ffffffff8605cd7c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff860718eb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff85eb9d4f>] process_one_work+0x17f/0x440
[<ffffffff85ebade6>] worker_thread+0x126/0x3c0
[<ffffffff85ebacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff85ec1c31>] kthread+0xd1/0xe0
[<ffffffff85ec1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff86574c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff85ec1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 b4 86 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #64553
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8c716a7a
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8c67e478
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8c67e614
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff8c805a48
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff8c7fefd6
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff8c800e03
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff8c801216
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff8c7fef30
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff8c7fe610
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff8c7c9900
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff8c7c9b72
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc0349030
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff8c951020
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffff8c981685
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x35
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc0348ffb
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffff8c7cabe1
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffff8c8798c0
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
|
18
|
|
vmlinux |
0xffffffff8c6c72a5
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x25
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffff8c87a454
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1c4
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff8c87a75f
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x9f
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffff8c87ac43
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x263
|
|
22
|
|
vmlinux |
0xffffffff8c866aec
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x4c
|
|
23
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8c87b83b
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x2cb
|
|
24
|
process_one_work
|
vmlinux |
0xffffffff8c6be21f
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x17f
|
|
25
|
worker_thread
|
vmlinux |
0xffffffff8c6bf336
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x126
|
|
26
|
|
vmlinux |
0xffffffff8c6bf210
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x2a0
|
|
27
|
kthread
|
vmlinux |
0xffffffff8c6c61f1
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xd1
|
|
28
|
|
vmlinux |
0xffffffff8c6c6120
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
|
29
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff8cd8dd37
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x21
|
|
30
|
|
vmlinux |
0xffffffff8c6c6120
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 21s! [kworker/u16:2:25778]
Modules linked in: nf_log_ipv4 nf_log_common xt_LOG xt_recent ip_set_hash_net tcp_diag udp_diag inet_diag tcp_lp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack fuse nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_set ip_set nfnetlink iptable_filter sunrpc snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore ppdev sg i2c_piix4 virtio_balloon pcspkr joydev parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common virtio_net
virtio_blk ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata virtio_pci floppy virtio_ring serio_raw virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 6 PID: 25778 Comm: kworker/u16:2 Kdump: loaded Tainted: G L ------------ 3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9a12217120e0 ti: ffff9a13d63d8000 task.ti: ffff9a13d63d8000
RIP: 0010:[<ffffffff8c716a7a>] [<ffffffff8c716a7a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9a13d63db830 EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffffffff8c67e2b0 RCX: ffff9a144b9204c8
RDX: 0000000000000004 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff9a13d63db868 R08: ffff9a03e1ba0000 R09: ffffffff8c981839
R10: ffff9a144b99f160 R11: ffffdac41bc4fc00 R12: ffff9a13d63db800
R13: ffffffff8c98178f R14: 0000000000000082 R15: ffff9a13ffd11000
FS: 0000000000000000(0000) GS:ffff9a144b980000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000039d3d600 CR3: 00000007bf810000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8c67e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8c67e614>] flush_tlb_page+0x54/0xa0
[<ffffffff8c805a48>] ptep_clear_flush+0x68/0xa0
[<ffffffff8c7fefd6>] page_mkclean_one+0xa6/0xe0
[<ffffffff8c800e03>] rmap_walk+0xf3/0x360
[<ffffffff8c801216>] page_mkclean+0x76/0xb0
[<ffffffff8c7fef30>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff8c7fe610>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff8c7c9900>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff8c7c9b72>] write_cache_pages+0x1f2/0x470
[<ffffffffc0349030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff8c951020>] ? submit_bio+0x70/0x150
[<ffffffff8c981685>] ? cpumask_next_and+0x35/0x50
[<ffffffffc0348ffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff8c7cabe1>] do_writepages+0x21/0x50
[<ffffffff8c8798c0>] __writeback_single_inode+0x40/0x260
[<ffffffff8c6c72a5>] ? wake_up_bit+0x25/0x30
[<ffffffff8c87a454>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8c87a75f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8c87ac43>] wb_writeback+0x263/0x2f0
[<ffffffff8c866aec>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8c87b83b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff8c6be21f>] process_one_work+0x17f/0x440
[<ffffffff8c6bf336>] worker_thread+0x126/0x3c0
[<ffffffff8c6bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8c6c61f1>] kthread+0xd1/0xe0
[<ffffffff8c6c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffff8cd8dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8c6c6120>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ee 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 34 8d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #22388
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa5b1173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa5a7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffa5a7a734
Build id: 3.10.0-957.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffa5bfd587
Build id: 3.10.0-957.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffa5bf6b66
Build id: 3.10.0-957.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffa5bf8983
Build id: 3.10.0-957.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffa5bf8d96
Build id: 3.10.0-957.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffa5bf6ac0
Build id: 3.10.0-957.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffa5bf6200
Build id: 3.10.0-957.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffa5bc1d60
Build id: 3.10.0-957.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffa5bc200a
Build id: 3.10.0-957.el7.x86_64
|
0x22a
|
|
12
|
|
xfs |
0xffffffffc04c95b0
Build id: 3.10.0-957.el7.x86_64
|
0x150
|
|
13
|
|
vmlinux |
0xffffffffa5d44f30
Build id: 3.10.0-957.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffffa5a2a59e
Build id: 3.10.0-957.el7.x86_64
|
0xce
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc04c93ab
Build id: 3.10.0-957.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffffa5bc3181
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffffa5c6f930
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffffa5c703c4
Build id: 3.10.0-957.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffa5c7072f
Build id: 3.10.0-957.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffffa5c70f63
Build id: 3.10.0-957.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffffa5c5cd7c
Build id: 3.10.0-957.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffa5c718eb
Build id: 3.10.0-957.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffa5ab9d4f
Build id: 3.10.0-957.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffa5abade6
Build id: 3.10.0-957.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffa5abacc0
Build id: 3.10.0-957.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffffa5ac1c31
Build id: 3.10.0-957.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffa5ac1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffa6174c37
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
29
|
|
vmlinux |
0xffffffffa5ac1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [kworker/u320:0:11792]
Modules linked in: binfmt_misc ext4 mbcache jbd2 ppdev sg joydev virtio_balloon parport_pc parport i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw ttm floppy drm ata_piix libata virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 13 PID: 11792 Comm: kworker/u320:0 Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9da9c6e1a080 ti: ffff9da61e3fc000 task.ti: ffff9da61e3fc000
RIP: 0010:[<ffffffffa5b1173a>] [<ffffffffa5b1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9da61e3ff820 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a5a7a3d0 RCX: ffff9da9dce205a0
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9da61e3ff858 R08: ffff9da3344ae800 R09: ffffffffa5d755c9
R10: ffff9da9dd15f160 R11: ffffe9990063c600 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9da61e3ff7d0 R15: ffff9da61e3ff7a0
FS: 0000000000000000(0000) GS:ffff9da9dd140000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7db7e58b80 CR3: 000000059d010000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa5a7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa5a7a734>] flush_tlb_page+0x54/0xa0
[<ffffffffa5bfd587>] ptep_clear_flush+0x67/0xa0
[<ffffffffa5bf6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffffa5bf8983>] rmap_walk+0xf3/0x360
[<ffffffffa5bf8d96>] page_mkclean+0x76/0xb0
[<ffffffffa5bf6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffffa5bf6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffa5bc1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa5bc200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc04c95b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffa5d44f30>] ? submit_bio+0x70/0x150
[<ffffffffa5a2a59e>] ? __switch_to+0xce/0x580
[<ffffffffc04c93ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa5bc3181>] do_writepages+0x21/0x50
[<ffffffffa5c6f930>] __writeback_single_inode+0x40/0x260
[<ffffffffa5c703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffa5c7072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa5c70f63>] wb_writeback+0x263/0x2f0
[<ffffffffa5c5cd7c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffa5c718eb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa5ab9d4f>] process_one_work+0x17f/0x440
[<ffffffffa5abade6>] worker_thread+0x126/0x3c0
[<ffffffffa5abacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffa5ac1c31>] kthread+0xd1/0xe0
[<ffffffffa5ac1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa6174c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa5ac1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 74 a6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #20903
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb1316c5a
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb127e868
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffb127ea04
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffb1407768
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffb1400d16
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffb1402b23
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffb1402f36
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffb1400c70
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffb1400350
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffb13c9b30
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffb13c9da2
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc0525290
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffb1555450
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x70
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc052525b
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffffb13cae21
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffffb147d730
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x40
|
|
17
|
|
vmlinux |
0xffffffffb12c6cd5
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x25
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffffb147e2b4
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffb147e5bf
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffffb147ea93
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffffb146a6bc
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffb147f68b
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffb12bdc4f
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffb12bed66
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffb12bec40
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffffb12c5c21
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffb12c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb1994df7
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x21
|
|
29
|
|
vmlinux |
0xffffffffb12c5b50
Build id: 3.10.0-1160.6.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [kworker/u50:3:6152]
Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm d2dctl(POE) vhost_net vhost macvtap macvlan xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack bridge stp llc ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter jc42 vfat fat ext4 mbcache jbd2 ipmi_ssif intel_powerclamp coretemp kvm_intel kvm irqbypass iTCO_wdt
crc32_pclmul ghash_clmulni_intel aesni_intel cdc_ether iTCO_vendor_support lrw gf128mul glue_helper usbnet ipmi_si ipmi_devintf ablk_helper cryptd ipmi_msghandler ioatdma pcspkr mii sg i2c_i801 lpc_ich i7core_edac dca acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_generic pata_acpi drm ata_piix libata megaraid_sas crct10dif_pclmul crct10dif_common crc32c_intel serio_raw bnx2 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 6 PID: 6152 Comm: kworker/u50:3 Kdump: loaded Tainted: P W OEL ------------ 3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: IBM System x3650 M3 -[7945K3G]-/69Y4438, BIOS -[D6E164AUS-1.22]- 06/04/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:1)
task: ffff9289e1ff8000 ti: ffff92941b258000 task.ti: ffff92941b258000
RIP: 0010:[<ffffffffb1316c5a>] [<ffffffffb1316c5a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff92941b25b830 EFLAGS: 00000202
RAX: 0000000000000003 RBX: ffffffffb127e6a0 RCX: ffff92898fae0728
RDX: 0000000000000003 RSI: 0000000000000018 RDI: 0000000000000000
RBP: ffff92941b25b868 R08: ffff9289e1fd0000 R09: ffffffffb1585fd9
R10: ffff92958f21f160 R11: ffffe6f1db7abc00 R12: 0000000000000282
R13: ffff92941b25b7e0 R14: ffff92941b25b7d8 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff92958f200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fae179288af CR3: 00000016cc610000 CR4: 00000000000227e0
Call Trace:
[<ffffffffb127e868>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb127ea04>] flush_tlb_page+0x54/0xa0
[<ffffffffb1407768>] ptep_clear_flush+0x68/0xa0
[<ffffffffb1400d16>] page_mkclean_one+0xa6/0xe0
[<ffffffffb1402b23>] rmap_walk+0xf3/0x360
[<ffffffffb1402f36>] page_mkclean+0x76/0xb0
[<ffffffffb1400c70>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffb1400350>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb13c9b30>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb13c9da2>] write_cache_pages+0x1f2/0x470
[<ffffffffc0525290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffb1555450>] ? submit_bio+0x70/0x150
[<ffffffffc052525b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb13cae21>] do_writepages+0x21/0x50
[<ffffffffb147d730>] __writeback_single_inode+0x40/0x260
[<ffffffffb12c6cd5>] ? wake_up_bit+0x25/0x30
[<ffffffffb147e2b4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffb147e5bf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb147ea93>] wb_writeback+0x263/0x2f0
[<ffffffffb146a6bc>] ? get_nr_inodes+0x4c/0x70
[<ffffffffb147f68b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb12bdc4f>] process_one_work+0x17f/0x440
[<ffffffffb12bed66>] worker_thread+0x126/0x3c0
[<ffffffffb12bec40>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb12c5c21>] kthread+0xd1/0xe0
[<ffffffffb12c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffb1994df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb12c5b50>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 f5 b1 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #18318
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff8991760a
Build id: 3.10.0-1127.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff8987e8f8
Build id: 3.10.0-1127.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff8987ea94
Build id: 3.10.0-1127.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff89a077f8
Build id: 3.10.0-1127.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff89a00da6
Build id: 3.10.0-1127.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff89a02bb3
Build id: 3.10.0-1127.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff89a02fc6
Build id: 3.10.0-1127.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff89a00d00
Build id: 3.10.0-1127.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff89a003e0
Build id: 3.10.0-1127.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff899c9ec0
Build id: 3.10.0-1127.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff899ca132
Build id: 3.10.0-1127.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc03963d0
Build id: 3.10.0-1127.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffff898e2e89
Build id: 3.10.0-1127.el7.x86_64
|
0xa9
|
|
14
|
|
vmlinux |
0xffffffff898e103c
Build id: 3.10.0-1127.el7.x86_64
|
0x3c
|
|
15
|
|
vmlinux |
0xffffffff8982b59e
Build id: 3.10.0-1127.el7.x86_64
|
0xce
|
|
16
|
xfs_vm_writepages
|
xfs |
0xffffffffc039639b
Build id: 3.10.0-1127.el7.x86_64
|
0x6b
|
|
17
|
do_writepages
|
vmlinux |
0xffffffff899cb1b1
Build id: 3.10.0-1127.el7.x86_64
|
0x21
|
|
18
|
__writeback_single_inode
|
vmlinux |
0xffffffff89a7cbd0
Build id: 3.10.0-1127.el7.x86_64
|
0x40
|
|
19
|
writeback_sb_inodes
|
vmlinux |
0xffffffff89a7d764
Build id: 3.10.0-1127.el7.x86_64
|
0x1c4
|
|
20
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff89a7da6f
Build id: 3.10.0-1127.el7.x86_64
|
0x9f
|
|
21
|
wb_writeback
|
vmlinux |
0xffffffff89a7df53
Build id: 3.10.0-1127.el7.x86_64
|
0x263
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff89a7eb4b
Build id: 3.10.0-1127.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffff898be6bf
Build id: 3.10.0-1127.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffff898bf7d6
Build id: 3.10.0-1127.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffff898bf6b0
Build id: 3.10.0-1127.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffff898c6691
Build id: 3.10.0-1127.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffff898c65c0
Build id: 3.10.0-1127.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff89f92d37
Build id: 3.10.0-1127.el7.x86_64
|
0x21
|
|
29
|
|
vmlinux |
0xffffffff898c65c0
Build id: 3.10.0-1127.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [kworker/u256:2:60387]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables br_netfilter overlay(T) tcp_lp nls_utf8 isofs fuse tun devlink bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi kvm_intel snd_ens1371 snd_rawmidi kvm snd_ac97_codec irqbypass ac97_bus crc32_pclmul snd_seq ghash_clmulni_intel snd_seq_device snd_pcm snd_timer ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd snd i2c_piix4 sg pcspkr joydev vmw_vmci soundcore parport_pc parport xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper
syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit mptspi scsi_transport_spi ata_piix mptscsih crct10dif_pclmul crct10dif_common drm libnvdimm libata mptbase e1000 serio_raw crc32c_intel drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 5 PID: 60387 Comm: kworker/u256:2 Kdump: loaded Tainted: G ------------ T 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9376f44e9070 ti: ffff9378b1450000 task.ti: ffff9378b1450000
RIP: 0010:[<ffffffff8991760a>] [<ffffffff8991760a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9378b1453830 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0000000000000005 RCX: ffff9378b56206a0
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9378b1453868 R08: ffff9377fc629400 R09: ffffffff89b85099
R10: ffff9378b575f160 R11: ffffebc3c73d7000 R12: 0000000000013378
R13: 000000fc89b84fef R14: 0000000000000286 R15: 0000000000000286
FS: 0000000000000000(0000) GS:ffff9378b5740000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5b41bf79e5 CR3: 0000000096a9a000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8987e8f8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8987ea94>] flush_tlb_page+0x54/0xa0
[<ffffffff89a077f8>] ptep_clear_flush+0x68/0xa0
[<ffffffff89a00da6>] page_mkclean_one+0xa6/0xe0
[<ffffffff89a02bb3>] rmap_walk+0xf3/0x360
[<ffffffff89a02fc6>] page_mkclean+0x76/0xb0
[<ffffffff89a00d00>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffff89a003e0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff899c9ec0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff899ca132>] write_cache_pages+0x1f2/0x470
[<ffffffffc03963d0>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff898e2e89>] ? pick_next_entity+0xa9/0x190
[<ffffffff898e103c>] ? set_next_entity+0x3c/0xe0
[<ffffffff8982b59e>] ? __switch_to+0xce/0x580
[<ffffffffc039639b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff899cb1b1>] do_writepages+0x21/0x50
[<ffffffff89a7cbd0>] __writeback_single_inode+0x40/0x260
[<ffffffff89a7d764>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff89a7da6f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff89a7df53>] wb_writeback+0x263/0x2f0
[<ffffffff89a7eb4b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff898be6bf>] process_one_work+0x17f/0x440
[<ffffffff898bf7d6>] worker_thread+0x126/0x3c0
[<ffffffff898bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff898c6691>] kthread+0xd1/0xe0
[<ffffffff898c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff89f92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff898c65c0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 1e 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 0f 55 8a f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 7a c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #15032
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
__page_check_address
|
vmlinux |
0xffffffff83ff6e0c
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x10c
|
|
2
|
page_mkclean_one
|
vmlinux |
0xffffffff83ff7114
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x44
|
|
3
|
rmap_walk
|
vmlinux |
0xffffffff83ff8f93
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xf3
|
|
4
|
page_mkclean
|
vmlinux |
0xffffffff83ff93a6
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x76
|
|
5
|
|
vmlinux |
0xffffffff83ff70d0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x180
|
|
6
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff83ff6810
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x20
|
|
7
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff83fc2320
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x50
|
|
8
|
write_cache_pages
|
vmlinux |
0xffffffff83fc2592
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x1f2
|
|
9
|
|
xfs |
0xffffffffc04b47d0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x150
|
|
10
|
xfs_vm_writepages
|
xfs |
0xffffffffc04b45cb
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x6b
|
|
11
|
do_writepages
|
vmlinux |
0xffffffff83fc36d1
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x21
|
|
12
|
__writeback_single_inode
|
vmlinux |
0xffffffff840701b0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x40
|
|
13
|
writeback_sb_inodes
|
vmlinux |
0xffffffff84070c44
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x1c4
|
|
14
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff84070faf
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x9f
|
|
15
|
wb_writeback
|
vmlinux |
0xffffffff840717e3
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x263
|
|
16
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff8407216b
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x2cb
|
|
17
|
process_one_work
|
vmlinux |
0xffffffff83eb9ebf
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x17f
|
|
18
|
worker_thread
|
vmlinux |
0xffffffff83ebaf56
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x126
|
|
19
|
|
vmlinux |
0xffffffff83ebae30
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x2a0
|
|
20
|
kthread
|
vmlinux |
0xffffffff83ec1da1
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xd1
|
|
21
|
|
vmlinux |
0xffffffff83ec1cd0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x40
|
|
22
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff84575c37
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x21
|
|
23
|
|
vmlinux |
0xffffffff83ec1cd0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u8:0:11112]
FS: 00007f5b76250740(0000) GS:ffff8cdfbfc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f73d12af000 CR3: 0000000104044000 CR4: 00000000003607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg vmw_balloon pcspkr joydev i2c_piix4 parport_pc parport vmw_vmci ip_tables xfs libcrc32c
ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi scsi_transport_spi ata_piix mptscsih libata e1000 crct10dif_pclmul crct10dif_common mptbase crc32c_intel serio_raw nfit libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 11112 Comm: kworker/u8:0 Kdump: loaded Tainted: G W L ------------ 3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8cde835a2080 ti: ffff8cdfb5b04000 task.ti: ffff8cdfb5b04000
RIP: 0010:[<ffffffff83ff6e0c>] [<ffffffff83ff6e0c>] __page_check_address+0x10c/0x250
RSP: 0018:ffff8cdfb5b078e8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8cde19157980 RCX: 00003ffffffff000
RDX: 00003ffffffff000 RSI: 00003ffffffff000 RDI: ffffd81441ac8df0
RBP: ffff8cdfb5b07910 R08: 0000000000000000 R09: 000000006b237000
R10: ffff8cdf987cb118 R11: d5d5d55555d5d5d5 R12: ffff8cdfb47e0000
R13: ffff8cde35eb2830 R14: ffffffff84167e6e R15: ffff8cde35eb2848
FS: 0000000000000000(0000) GS:ffff8cdfbfd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa9aa4d9ef0 CR3: 000000010206a000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff83ff7114>] page_mkclean_one+0x44/0xe0
[<ffffffff83ff8f93>] rmap_walk+0xf3/0x360
[<ffffffff83ff93a6>] page_mkclean+0x76/0xb0
[<ffffffff83ff70d0>] ? page_referenced_one+0x180/0x180
[<ffffffff83ff6810>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff83fc2320>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff83fc2592>] write_cache_pages+0x1f2/0x470
[<ffffffffc04b47d0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffc04b45cb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff83fc36d1>] do_writepages+0x21/0x50
[<ffffffff840701b0>] __writeback_single_inode+0x40/0x260
[<ffffffff84070c44>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff84070faf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff840717e3>] wb_writeback+0x263/0x2f0
[<ffffffff8407216b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff83eb9ebf>] process_one_work+0x17f/0x440
[<ffffffff83ebaf56>] worker_thread+0x126/0x3c0
[<ffffffff83ebae30>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff83ec1da1>] kthread+0xd1/0xe0
[<ffffffff83ec1cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffff84575c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff83ec1cd0>] ? insert_kthread_work+0x40/0x40
Code: ff 3f 00 00 48 21 c1 48 21 ca 48 c1 ea 06 48 03 15 ba e3 a4 00 4c 8d 7a 30 4c 89 ff e8 3e 49 57 00 48 ba 00 f0 ff ff ff 3f 00 00 <49> 8b 3e 48 89 d1 48 f7 d1 48 0b 0d 9c e4 a4 00 48 21 f9 f7 c1
Complete report #13277
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb2311a8a
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb227a6a8
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffb227a844
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffb23fdb97
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffb23f7176
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffb23f8f93
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffb23f93a6
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffb23f70d0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffb23f6810
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffb23c2320
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffb23c2592
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc03a87d0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x150
|
|
13
|
|
vmlinux |
0xffffffffb2545790
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x70
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc03a85cb
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffffb23c36d1
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffffb24701b0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x40
|
|
17
|
writeback_sb_inodes
|
vmlinux |
0xffffffffb2470c44
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x1c4
|
|
18
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffb2470faf
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x9f
|
|
19
|
wb_writeback
|
vmlinux |
0xffffffffb24717e3
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x263
|
|
20
|
|
vmlinux |
0xffffffffb245d5fc
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x4c
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffb247216b
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x2cb
|
|
22
|
process_one_work
|
vmlinux |
0xffffffffb22b9ebf
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffffb22baf56
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffffb22bae30
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffffb22c1da1
Build id: 3.10.0-957.21.3.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffffb22c1cd0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb2975c37
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x21
|
|
28
|
|
vmlinux |
0xffffffffb22c1cd0
Build id: 3.10.0-957.21.3.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u12:1:18546]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock ppdev sb_edac iosf_mbi vmw_balloon crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev pcspkr parport_pc vmw_vmci parport i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt crct10dif_pclmul fb_sys_fops crct10dif_common crc32c_intel ttm serio_raw drm ata_piix libata e1000
mptspi nfit scsi_transport_spi mptscsih libnvdimm mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 18546 Comm: kworker/u12:1 Kdump: loaded Tainted: G L ------------ 3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
Workqueue: writeback bdi_writeback_workfn (flush-253:1)
task: ffff9c0f1c73d140 ti: ffff9c0e02f38000 task.ti: ffff9c0e02f38000
RIP: 0010:[<ffffffffb2311a8a>] [<ffffffffb2311a8a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9c0e02f3b830 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000b227a4e0 RCX: ffff9c0f3fc60600
RDX: 0000000000000001 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff9c0e02f3b868 R08: ffff9c0c7a375400 R09: ffffffffb2575e39
R10: ffff9c0f3fcdf160 R11: ffffd9e0d086e600 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9c0e02f3b7e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff9c0f3fcc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007faa37eaf008 CR3: 00000002f3a10000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb227a6a8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb227a844>] flush_tlb_page+0x54/0xa0
[<ffffffffb23fdb97>] ptep_clear_flush+0x67/0xa0
[<ffffffffb23f7176>] page_mkclean_one+0xa6/0xe0
[<ffffffffb23f8f93>] rmap_walk+0xf3/0x360
[<ffffffffb23f93a6>] page_mkclean+0x76/0xb0
[<ffffffffb23f70d0>] ? page_referenced_one+0x180/0x180
[<ffffffffb23f6810>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb23c2320>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb23c2592>] write_cache_pages+0x1f2/0x470
[<ffffffffc03a87d0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffb2545790>] ? submit_bio+0x70/0x150
[<ffffffffc03a85cb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb23c36d1>] do_writepages+0x21/0x50
[<ffffffffb24701b0>] __writeback_single_inode+0x40/0x260
[<ffffffffb2470c44>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffb2470faf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb24717e3>] wb_writeback+0x263/0x2f0
[<ffffffffb245d5fc>] ? get_nr_inodes+0x4c/0x70
[<ffffffffb247216b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb22b9ebf>] process_one_work+0x17f/0x440
[<ffffffffb22baf56>] worker_thread+0x126/0x3c0
[<ffffffffb22bae30>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffb22c1da1>] kthread+0xd1/0xe0
[<ffffffffb22c1cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb2975c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb22c1cd0>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 3e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 bc f4 b2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #13267
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff82b1173a
Build id: 3.10.0-957.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff82a7a598
Build id: 3.10.0-957.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffff82a7a734
Build id: 3.10.0-957.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffff82bfd587
Build id: 3.10.0-957.el7.x86_64
|
0x67
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffff82bf6b66
Build id: 3.10.0-957.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffff82bf8983
Build id: 3.10.0-957.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffff82bf8d96
Build id: 3.10.0-957.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffff82bf6ac0
Build id: 3.10.0-957.el7.x86_64
|
0x180
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffff82bf6200
Build id: 3.10.0-957.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffff82bc1d60
Build id: 3.10.0-957.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffff82bc200a
Build id: 3.10.0-957.el7.x86_64
|
0x22a
|
|
12
|
|
xfs |
0xffffffffc02b25b0
Build id: 3.10.0-957.el7.x86_64
|
0x150
|
|
13
|
|
vmlinux |
0xffffffff82d44f30
Build id: 3.10.0-957.el7.x86_64
|
0x70
|
|
14
|
|
vmlinux |
0xffffffff82d75465
Build id: 3.10.0-957.el7.x86_64
|
0x35
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc02b23ab
Build id: 3.10.0-957.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffff82bc3181
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffff82c6f930
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffff82c703c4
Build id: 3.10.0-957.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffff82c7072f
Build id: 3.10.0-957.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffff82c70f63
Build id: 3.10.0-957.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffff82c5cd7c
Build id: 3.10.0-957.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffff82c718eb
Build id: 3.10.0-957.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffff82ab9d4f
Build id: 3.10.0-957.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffff82abade6
Build id: 3.10.0-957.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffff82abacc0
Build id: 3.10.0-957.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffff82ac1c31
Build id: 3.10.0-957.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffff82ac1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff83174c37
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
29
|
|
vmlinux |
0xffffffff82ac1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u320:5:75090]
Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop overlay ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ppdev parport_pc sg joydev virtio_balloon parport i2c_piix4 pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom virtio_scsi virtio_blk virtio_net
virtio_console ata_generic pata_acpi floppy cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm virtio_pci libata drm_panel_orientation_quirks serio_raw virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 75090 Comm: kworker/u320:5 Kdump: loaded Tainted: G L ------------ 3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:6)
task: ffff978d16f3e180 ti: ffff978a1e544000 task.ti: ffff978a1e544000
RIP: 0010:[<ffffffff82b1173a>] [<ffffffff82b1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff978a1e547820 EFLAGS: 00000202
RAX: 0000000000000005 RBX: 0008000082a7a3d0 RCX: ffff978dacf60320
RDX: 0000000000000005 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff978a1e547858 R08: ffff978af9c23c00 R09: ffffffff82d755c9
R10: ffff978dacf1f160 R11: fffff666d09f7600 R12: 0000000000000286
R13: 0000000000000286 R14: ffff978a1e5477d0 R15: ffff978d7dfe8660
FS: 0000000000000000(0000) GS:ffff978dacf00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c4200d1010 CR3: 000000024c210000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff82a7a598>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff82a7a734>] flush_tlb_page+0x54/0xa0
[<ffffffff82bfd587>] ptep_clear_flush+0x67/0xa0
[<ffffffff82bf6b66>] page_mkclean_one+0xa6/0xe0
[<ffffffff82bf8983>] rmap_walk+0xf3/0x360
[<ffffffff82bf8d96>] page_mkclean+0x76/0xb0
[<ffffffff82bf6ac0>] ? page_referenced_one+0x180/0x180
[<ffffffff82bf6200>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffff82bc1d60>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffff82bc200a>] write_cache_pages+0x22a/0x4e0
[<ffffffffc02b25b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff82d44f30>] ? submit_bio+0x70/0x150
[<ffffffff82d75465>] ? cpumask_next_and+0x35/0x50
[<ffffffffc02b23ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff82bc3181>] do_writepages+0x21/0x50
[<ffffffff82c6f930>] __writeback_single_inode+0x40/0x260
[<ffffffff82c703c4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff82c7072f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff82c70f63>] wb_writeback+0x263/0x2f0
[<ffffffff82c5cd7c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff82c718eb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff82ab9d4f>] process_one_work+0x17f/0x440
[<ffffffff82abade6>] worker_thread+0x126/0x3c0
[<ffffffff82abacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff82ac1c31>] kthread+0xd1/0xe0
[<ffffffff82ac1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff83174c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff82ac1b60>] ? insert_kthread_work+0x40/0x40
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 74 83 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #8409
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb7b16a7e
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb7a7e478
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffb7a7e614
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffb7c05a48
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffb7bfefd6
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffb7c00e03
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffb7c01216
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffb7bfef30
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffb7bfe610
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffb7bc9900
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffb7bc9b72
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc0248030
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffb7d51020
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x70
|
|
14
|
xfs_vm_writepages
|
xfs |
0xffffffffc0247ffb
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x6b
|
|
15
|
do_writepages
|
vmlinux |
0xffffffffb7bcabe1
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x21
|
|
16
|
__writeback_single_inode
|
vmlinux |
0xffffffffb7c798c0
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
|
17
|
|
vmlinux |
0xffffffffb7ac72a5
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x25
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffffb7c7a454
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffb7c7a75f
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffffb7c7ac43
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x263
|
|
21
|
|
vmlinux |
0xffffffffb7c66aec
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x4c
|
|
22
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffb7c7b83b
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x2cb
|
|
23
|
process_one_work
|
vmlinux |
0xffffffffb7abe21f
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x17f
|
|
24
|
worker_thread
|
vmlinux |
0xffffffffb7abf336
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x126
|
|
25
|
|
vmlinux |
0xffffffffb7abf210
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x2a0
|
|
26
|
kthread
|
vmlinux |
0xffffffffb7ac61f1
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0xd1
|
|
27
|
|
vmlinux |
0xffffffffb7ac6120
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
|
28
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb818dd37
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x21
|
|
29
|
|
vmlinux |
0xffffffffb7ac6120
Build id: 3.10.0-1062.12.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u320:0:35070]
Modules linked in: xt_NFQUEUE nfnetlink_log nfnetlink nfring(OE) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache sunrpc ppdev sg virtio_balloon joydev i2c_piix4 pcspkr parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata virtio_pci serio_raw virtio_ring virtio drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 35070 Comm: kworker/u320:0 Kdump: loaded Tainted: G OEL ------------ 3.10.0-1062.12.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff98565f7fb150 ti: ffff9856e7e7c000 task.ti: ffff9856e7e7c000
RIP: 0010:[<ffffffffb7b16a7e>] [<ffffffffb7b16a7e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff9856e7e7f830 EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00080000b7a7e2b0 RCX: ffff985e5cf20450
RDX: 0000000000000004 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9856e7e7f868 R08: ffff9857b44abc00 R09: ffffffffb7d81839
R10: ffff985e5cedf160 R11: ffffd77a40907a00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff9856e7e7f7e0 R15: ffff985e58fe3c00
FS: 0000000000000000(0000) GS:ffff985e5cec0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9ed0af3a08 CR3: 0000000083610000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb7a7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb7a7e614>] flush_tlb_page+0x54/0xa0
[<ffffffffb7c05a48>] ptep_clear_flush+0x68/0xa0
[<ffffffffb7bfefd6>] page_mkclean_one+0xa6/0xe0
[<ffffffffb7c00e03>] rmap_walk+0xf3/0x360
[<ffffffffb7c01216>] page_mkclean+0x76/0xb0
[<ffffffffb7bfef30>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffb7bfe610>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffb7bc9900>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffb7bc9b72>] write_cache_pages+0x1f2/0x470
[<ffffffffc0248030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffb7d51020>] ? submit_bio+0x70/0x150
[<ffffffffc0247ffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb7bcabe1>] do_writepages+0x21/0x50
[<ffffffffb7c798c0>] __writeback_single_inode+0x40/0x260
[<ffffffffb7ac72a5>] ? wake_up_bit+0x25/0x30
[<ffffffffb7c7a454>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffb7c7a75f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb7c7ac43>] wb_writeback+0x263/0x2f0
[<ffffffffb7c66aec>] ? get_nr_inodes+0x4c/0x70
[<ffffffffb7c7b83b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb7abe21f>] process_one_work+0x17f/0x440
[<ffffffffb7abf336>] worker_thread+0x126/0x3c0
[<ffffffffb7abf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb7ac61f1>] kthread+0xd1/0xe0
[<ffffffffb7ac6120>] ? insert_kthread_work+0x40/0x40
[<ffffffffb818dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb7ac6120>] ? insert_kthread_work+0x40/0x40
Code: 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 74 b8 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 bd 5e c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #3641
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa6d16a6e
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x20e
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffa6c7e478
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_page
|
vmlinux |
0xffffffffa6c7e614
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x54
|
|
4
|
ptep_clear_flush
|
vmlinux |
0xffffffffa6e059d8
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x68
|
|
5
|
page_mkclean_one
|
vmlinux |
0xffffffffa6dfef66
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xa6
|
|
6
|
rmap_walk
|
vmlinux |
0xffffffffa6e00d93
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xf3
|
|
7
|
page_mkclean
|
vmlinux |
0xffffffffa6e011a6
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x76
|
|
8
|
|
vmlinux |
0xffffffffa6dfeec0
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x1b0
|
|
9
|
?
percpu_flush_tlb_batch_pages
|
vmlinux |
0xffffffffa6dfe5a0
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x20
|
|
10
|
clear_page_dirty_for_io
|
vmlinux |
0xffffffffa6dc98e0
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x50
|
|
11
|
write_cache_pages
|
vmlinux |
0xffffffffa6dc9b52
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x1f2
|
|
12
|
|
xfs |
0xffffffffc0462030
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xa0
|
|
13
|
|
vmlinux |
0xffffffffa6ce29ec
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x3c
|
|
14
|
|
vmlinux |
0xffffffffa6c2b59e
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xce
|
|
15
|
xfs_vm_writepages
|
xfs |
0xffffffffc0461ffb
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x6b
|
|
16
|
do_writepages
|
vmlinux |
0xffffffffa6dcabc1
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x21
|
|
17
|
__writeback_single_inode
|
vmlinux |
0xffffffffa6e79840
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x40
|
|
18
|
writeback_sb_inodes
|
vmlinux |
0xffffffffa6e7a3d4
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x1c4
|
|
19
|
__writeback_inodes_wb
|
vmlinux |
0xffffffffa6e7a6df
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x9f
|
|
20
|
wb_writeback
|
vmlinux |
0xffffffffa6e7abc3
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x263
|
|
21
|
bdi_writeback_workfn
|
vmlinux |
0xffffffffa6e7b7bb
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x2cb
|
|
22
|
process_one_work
|
vmlinux |
0xffffffffa6cbe21f
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x17f
|
|
23
|
worker_thread
|
vmlinux |
0xffffffffa6cbf336
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x126
|
|
24
|
|
vmlinux |
0xffffffffa6cbf210
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x2a0
|
|
25
|
kthread
|
vmlinux |
0xffffffffa6cc61f1
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0xd1
|
|
26
|
|
vmlinux |
0xffffffffa6cc6120
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x40
|
|
27
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffa738dd37
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x21
|
|
28
|
|
vmlinux |
0xffffffffa6cc6120
Build id: 3.10.0-1062.9.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u256:1:81074]
Modules linked in: xt_nat xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables veth tcp_lp fuse br_netfilter tun devlink overlay(T) bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vmw_vmci vsock sunrpc snd_seq_midi snd_seq_midi_event coretemp ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 vmw_balloon snd_rawmidi snd_ac97_codec ac97_bus aesni_intel snd_seq lrw gf128mul glue_helper snd_seq_device snd_pcm ablk_helper cryptd snd_timer pcspkr joydev snd sg soundcore parport_pc parport pcc_cpufreq i2c_piix4 xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm crct10dif_pclmul crct10dif_common mptspi crc32c_intel scsi_transport_spi serio_raw drm mptscsih ata_piix e1000 nfit libnvdimm libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 2 PID: 81074 Comm: kworker/u256:1 Kdump: loaded Tainted: G L ------------ T 3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff96b30f6b62a0 ti: ffff96b2e69d0000 task.ti: ffff96b2e69d0000
RIP: 0010:[<ffffffffa6d16a6e>] [<ffffffffa6d16a6e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff96b2e69d3830 EFLAGS: 00000202
RAX: 0000000000000001 RBX: 00080000a6c7e2b0 RCX: ffff96b33a6604b0
RDX: 0000000000000001 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff96b2e69d3868 R08: ffff96b3065c4400 R09: ffffffffa6f817e9
R10: ffff96b33a69f160 R11: ffffd2cc427db000 R12: 0000000000000286
R13: 0000000000000286 R14: ffff96b2e69d37e0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff96b33a680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6043e12000 CR3: 000000008bfda000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa6c7e478>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa6c7e614>] flush_tlb_page+0x54/0xa0
[<ffffffffa6e059d8>] ptep_clear_flush+0x68/0xa0
[<ffffffffa6dfef66>] page_mkclean_one+0xa6/0xe0
[<ffffffffa6e00d93>] rmap_walk+0xf3/0x360
[<ffffffffa6e011a6>] page_mkclean+0x76/0xb0
[<ffffffffa6dfeec0>] ? page_referenced_one+0x1b0/0x1b0
[<ffffffffa6dfe5a0>] ? percpu_flush_tlb_batch_pages+0x20/0x20
[<ffffffffa6dc98e0>] clear_page_dirty_for_io+0x50/0xd0
[<ffffffffa6dc9b52>] write_cache_pages+0x1f2/0x470
[<ffffffffc0462030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffa6ce29ec>] ? set_next_entity+0x3c/0xe0
[<ffffffffa6c2b59e>] ? __switch_to+0xce/0x580
[<ffffffffc0461ffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa6dcabc1>] do_writepages+0x21/0x50
[<ffffffffa6e79840>] __writeback_single_inode+0x40/0x260
[<ffffffffa6e7a3d4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffa6e7a6df>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa6e7abc3>] wb_writeback+0x263/0x2f0
[<ffffffffa6e7b7bb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa6cbe21f>] process_one_work+0x17f/0x440
[<ffffffffa6cbf336>] worker_thread+0x126/0x3c0
[<ffffffffa6cbf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffa6cc61f1>] kthread+0xd1/0xe0
[<ffffffffa6cc6120>] ? insert_kthread_work+0x40/0x40
[<ffffffffa738dd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa6cc6120>] ? insert_kthread_work+0x40/0x40
Code: 5e c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 20 e8 94 a7 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 cd 5e c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee