Info

Permalink
Function
__list_del_entry
First occurrence
2020-09-30
Last occurrence
2020-09-30
State
NEW
Components
kernel
Quality
-24

Statistics

Operating system Count
CentOS 7 277
Architecture Count
x86_64 277
Related packages Count
kernel 277
 0:3.10.0-1160.45.1.el7 142
 0:3.10.0-693.2.2.el7 76
 0:3.10.0-957.12.2.el7 22
 0:3.10.0-1160.49.1.el7 13
 0:3.10.0-1160.31.1.el7 4
 0:3.10.0-1160.119.1.el7 3
 0:3.10.0-693.5.2.el7 3
 0:3.10.0-1160.88.1.el7 2
 0:3.10.0-1127.19.1.el7 2
 0:3.10.0-1160.66.1.el7 2
 0:3.10.0-862.2.3.el7 1
 0:3.10.0-1160.11.1.el7 1
 0:3.10.0-862.el7 1
 0:3.10.0-1160.53.1.el7 1
 0:3.10.0-1160.71.1.el7 1
 0:3.10.0-957.27.2.el7 1
 0:3.10.0-1160.83.1.el7 1
 0:3.10.0-1160.92.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #1032006
Frame # Function Binary Source or offset Line
1
__list_del_entry
vmlinux 0xffffffff8133da0e
Build id: 3.10.0-693.5.2.el7.x86_64
0x3e
2
queue_io
vmlinux 0xffffffff8122ecf6
Build id: 3.10.0-693.5.2.el7.x86_64
0xe6
3
wb_writeback
vmlinux 0xffffffff8122f0a4
Build id: 3.10.0-693.5.2.el7.x86_64
0x244
4
? get_nr_inodes
vmlinux 0xffffffff8121bd4c
Build id: 3.10.0-693.5.2.el7.x86_64
0x4c
5
bdi_writeback_workfn
vmlinux 0xffffffff8122f56b
Build id: 3.10.0-693.5.2.el7.x86_64
0x2cb
6
process_one_work
vmlinux 0xffffffff810a882a
Build id: 3.10.0-693.5.2.el7.x86_64
0x17a
7
worker_thread
vmlinux 0xffffffff810a94f6
Build id: 3.10.0-693.5.2.el7.x86_64
0x126
8
? manage_workers.isra.24
vmlinux 0xffffffff810a93d0
Build id: 3.10.0-693.5.2.el7.x86_64
0x2a0
9
kthread
vmlinux 0xffffffff810b099f
Build id: 3.10.0-693.5.2.el7.x86_64
0xcf
10
? insert_kthread_work
vmlinux 0xffffffff810b08d0
Build id: 3.10.0-693.5.2.el7.x86_64
0x40
11
ret_from_fork
vmlinux 0xffffffff816b4fd8
Build id: 3.10.0-693.5.2.el7.x86_64
0x58
12
? insert_kthread_work
vmlinux 0xffffffff810b08d0
Build id: 3.10.0-693.5.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [kworker/u256:1:76227]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev hyperv_keyboard hv_balloon pcspkr hyperv_fb nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c ata_generic pata_acpi sd_mod
crc_t10dif crct10dif_generic hv_netvsc hv_storvsc scsi_transport_fc hv_utils ptp pps_core hid_hyperv scsi_tgt crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libata serio_raw floppy hv_vmbus dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 76227 Comm: kworker/u256:1 Tainted: G             L ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS 090007  05/18/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff880027520fd0 ti: ffff880005908000 task.ti: ffff880005908000
RIP: 0010:[<ffffffff8133da0e>]  [<ffffffff8133da0e>] __list_del_entry+0x3e/0xd0
RSP: 0000:ffff88000590bc80  EFLAGS: 00010246
RAX: ffff8800172f5ff8 RBX: ffff88000590bca8 RCX: dead000000000200
RDX: ffff88003505d2d0 RSI: ffff88000590bda8 RDI: ffff88006edbf2b8
RBP: ffff88000590bc80 R08: ffff88006edbf2b8 R09: 000000000000ffff
R10: dffeb3e8ebc5d250 R11: 0000000000000000 R12: ffff88003505d2e0
R13: ffffffff810b18e5 R14: ffff88000590bc00 R15: 000000005d25c5a5
FS:  0000000000000000(0000) GS:ffff88007b600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe8ee1e5e27 CR3: 000000007334c000 CR4: 00000000000406f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff88000590bcf0 ffffffff8122ecf6 ffff88003505d2e0 000000000590bce0
ffff88003505d2d0 ffff88000590bda8 ffff88000590bcb0 ffff88000590bcb0
000000005d25c5a5 ffff88000590bda8 ffff88003505d0e0 ffff88003505d238
Call Trace:
[<ffffffff8122ecf6>] queue_io+0xe6/0x250
[<ffffffff8122f0a4>] wb_writeback+0x244/0x2f0
[<ffffffff8121bd4c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8122f56b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff810a882a>] process_one_work+0x17a/0x440
[<ffffffff810a94f6>] worker_thread+0x126/0x3c0
[<ffffffff810a93d0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: 89 e5 48 39 ca 74 29 48 b9 00 02 00 00 00 00 ad de 48 39 c8 74 7a 4c 8b 00 4c 39 c7 75 53 4c 8b 42 08 4c 39 c7 75 2b 48 89 42 08 <48> 89 10 5d c3 49 89 d0 48 89 f9 48 c7 c2 f0 2d 93 81 be 35 00 
Complete report #981009
Frame # Function Binary Source or offset Line
1
__list_del_entry
vmlinux 0xffffffff8133db22
Build id: 3.10.0-693.2.2.el7.x86_64
0x12
2
? queue_io
vmlinux 0xffffffff8122ed86
Build id: 3.10.0-693.2.2.el7.x86_64
0xe6
3
wb_writeback
vmlinux 0xffffffff8122f134
Build id: 3.10.0-693.2.2.el7.x86_64
0x244
4
bdi_writeback_workfn
vmlinux 0xffffffff8122f445
Build id: 3.10.0-693.2.2.el7.x86_64
0x115
5
process_one_work
vmlinux 0xffffffff810a881a
Build id: 3.10.0-693.2.2.el7.x86_64
0x17a
6
worker_thread
vmlinux 0xffffffff810a94e6
Build id: 3.10.0-693.2.2.el7.x86_64
0x126
7
? manage_workers.isra.24
vmlinux 0xffffffff810a93c0
Build id: 3.10.0-693.2.2.el7.x86_64
0x2a0
8
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.2.2.el7.x86_64
0xcf
9
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.2.2.el7.x86_64
0x40
10
ret_from_fork
vmlinux 0xffffffff816b4f58
Build id: 3.10.0-693.2.2.el7.x86_64
0x58
11
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.2.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/u897:1:207593]
Modules linked in: squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dell_rbu dm_mirror dm_region_hash dm_log joydev iTCO_wdt skx_edac iTCO_vendor_support edac_core intel_powerclamp coretemp intel_rapl iosf_mbi mgag200 ttm kvm_intel drm_kms_helper kvm dell_smbios syscopyarea dcdbas sysfillrect sparse_keymap sysimgblt irqbypass fb_sys_fops crc32_pclmul drm crc32c_intel ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr mei_me mei i2c_i801 shpchp lpc_ich wmi ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad binfmt_misc ip_tables bnxt_en nvme nvme_core qed hpilo dm_thin_pool dm_bio_prison dm_persistent_data dm_bufio dm_mod udf crc_itu_t vfat fat
br_netfilter bridge stp llc xfs isci libsas qla3xxx hpsa e1000e ixgbe(OE) igb(OE) i2c_algo_bit i2c_core dca aacraid aic79xx ata_piix mpt2sas raid_class mptspi scsi_transport_spi mptsas mptscsih mptbase scsi_transport_sas arcmsr sata_nv sata_svw bnx2x(OE) mdio libcrc32c bnx2(OE) ext4 mbcache jbd2 sata_sil e1000 nfsv3 nfs_acl nfs lockd grace sunrpc fscache tun sd_mod crc_t10dif crct10dif_generic sg crct10dif_pclmul crct10dif_common ahci libahci i40e(OE) megaraid_sas libata tg3(OE) ptp pps_core [last unloaded: ipmi_msghandler]
CPU: 2 PID: 207593 Comm: kworker/u897:1 Tainted: G           OE  ------------ T 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R440/04JN2K, BIOS 2.21.1 03/07/2024
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff881b302ebf40 ti: ffff88079ced8000 task.ti: ffff88079ced8000
RIP: 0010:[<ffffffff8133db22>]  [<ffffffff8133db22>] __list_del_entry+0x12/0xd0
RSP: 0018:ffff88079cedbc80  EFLAGS: 00000202
RAX: ffff883d493b8238 RBX: ffff880e736a0238 RCX: dead000000000100
RDX: ffff882fb11fca00 RSI: ffff88079cedbcb0 RDI: ffff8832aa6b8238
RBP: ffff88079cedbcf0 R08: ffff8844297b0238 R09: 0000000180400021
R10: 000000001b7f8701 R11: ffffea00046dfe00 R12: 000000008503ba01
R13: ffffea00ce140ec0 R14: 0000000000000000 R15: ffff88079cedbc80
FS:  0000000000000000(0000) GS:ffff882fb8a40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000801514208 CR3: 000000524b1ad000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff88079cedbcf0 ffffffff8122ed86 ffff882fb11fca10 00000000ac2b4000
ffff882fb11fca00 ffff88338503bc40 ffff88262a5e8238 ffff880aa13c23f8
000000005ba805d6 ffff88338503bc40 ffff882fb11fc810 ffff882fb11fc968
Call Trace:
[<ffffffff8122ed86>] ? queue_io+0xe6/0x250
[<ffffffff8122f134>] wb_writeback+0x244/0x2f0
[<ffffffff8122f445>] bdi_writeback_workfn+0x115/0x460
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f58>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: e8 f4 9e d4 ff e9 73 ff ff ff 0f 1f 44 00 00 66 2e 0f 1f 84 00 00 00 00 00 55 48 8b 17 48 b9 00 01 00 00 00 00 ad de 48 8b 47 08 <48> 89 e5 48 39 ca 74 29 48 b9 00 02 00 00 00 00 ad de 48 39 c8 
Complete report #962971
Frame # Function Binary Source or offset Line
1
queue_io
vmlinux 0xffffffff9fc7eeee
Build id: 3.10.0-1160.49.1.el7.x86_64
0x11e
2
wb_writeback
vmlinux 0xffffffff9fc7f257
Build id: 3.10.0-1160.49.1.el7.x86_64
0x247
3
bdi_writeback_workfn
vmlinux 0xffffffff9fc7fe6b
Build id: 3.10.0-1160.49.1.el7.x86_64
0x2cb
4
process_one_work
vmlinux 0xffffffff9fabde8f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x17f
5
worker_thread
vmlinux 0xffffffff9fabefa6
Build id: 3.10.0-1160.49.1.el7.x86_64
0x126
6
? manage_workers.isra.26
vmlinux 0xffffffff9fabee80
Build id: 3.10.0-1160.49.1.el7.x86_64
0x2a0
7
kthread
vmlinux 0xffffffff9fac5e61
Build id: 3.10.0-1160.49.1.el7.x86_64
0xd1
8
? insert_kthread_work
vmlinux 0xffffffff9fac5d90
Build id: 3.10.0-1160.49.1.el7.x86_64
0x40
9
ret_from_fork_nospec_begin
vmlinux 0xffffffffa0195df7
Build id: 3.10.0-1160.49.1.el7.x86_64
0x21
10
? insert_kthread_work
vmlinux 0xffffffff9fac5d90
Build id: 3.10.0-1160.49.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [kworker/u256:2:52329]
Modules linked in: veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) binfmt_misc sunrpc joydev ppdev sg parport_pc parport intel_rapl iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd i2c_piix4 pcspkr ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix xen_netfront crct10dif_pclmul crct10dif_common xen_blkfront libata crc32c_intel floppy drm_panel_orientation_quirks
serio_raw dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 5 PID: 52329 Comm: kworker/u256:2 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: Xen HVM domU, BIOS 4.7.6-6.1 08/24/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
task: ffff9a71c125e300 ti: ffff9a7566368000 task.ti: ffff9a7566368000
RIP: 0010:[<ffffffff9fc7eeee>]  [<ffffffff9fc7eeee>] queue_io+0x11e/0x240
RSP: 0018:ffff9a756636bc90  EFLAGS: 00000216
RAX: 0000000101c88878 RBX: 0000000000000e07 RCX: 0000000000000000
RDX: ffff9a756636bd30 RSI: ffff9a756636bcb0 RDI: ffff9a73fef2e770
RBP: ffff9a756636bcf0 R08: ffff9a73fef2e3b0 R09: 0000000000000000
R10: 0000000000000e07 R11: ffff9a756636b8c6 R12: ffff9a73fef2e770
R13: ffff9a756636bcf0 R14: 0000000000000e07 R15: ffff9a756636b8c6
FS:  0000000000000000(0000) GS:ffff9a757d340000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000268e9ac8c000 CR3: 0000000184c10000 CR4: 00000000001606e0
Call Trace:
[<ffffffff9fc7f257>] wb_writeback+0x247/0x2f0
[<ffffffff9fc7fe6b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff9fabde8f>] process_one_work+0x17f/0x440
[<ffffffff9fabefa6>] worker_thread+0x126/0x3c0
[<ffffffff9fabee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff9fac5e61>] kthread+0xd1/0xe0
[<ffffffff9fac5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffa0195df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9fac5d90>] ? insert_kthread_work+0x40/0x40
Code: 49 89 c7 48 8b 45 a8 48 3b 83 98 00 00 00 74 1d 48 8b 45 b0 4c 8b ab a0 00 00 00 48 8b 50 10 48 85 d2 74 99 49 8b 45 e8 48 39 02 <79> 90 8b 45 b8 85 c0 75 59 48 8b 45 c0 4c 39 f0 74 21 48 8b 93 
Complete report #747321
Frame # Function Binary Source or offset Line
1
__list_add
vmlinux 0xffffffffad1bd183
Build id: 3.10.0-1160.92.1.el7.x86_64
0x33
2
queue_io
vmlinux 0xffffffffad08dc5b
Build id: 3.10.0-1160.92.1.el7.x86_64
0xcb
3
wb_writeback
vmlinux 0xffffffffad08e017
Build id: 3.10.0-1160.92.1.el7.x86_64
0x247
4
? bdi_dirty_limit
vmlinux 0xffffffffacfd6180
Build id: 3.10.0-1160.92.1.el7.x86_64
0x40
5
bdi_writeback_workfn
vmlinux 0xffffffffad08eb3c
Build id: 3.10.0-1160.92.1.el7.x86_64
0x1cc
6
process_one_work
vmlinux 0xffffffffacec32ef
Build id: 3.10.0-1160.92.1.el7.x86_64
0x17f
7
worker_thread
vmlinux 0xffffffffacec4436
Build id: 3.10.0-1160.92.1.el7.x86_64
0x126
8
? manage_workers.isra.26
vmlinux 0xffffffffacec4310
Build id: 3.10.0-1160.92.1.el7.x86_64
0x2b0
9
kthread
vmlinux 0xffffffffacecb621
Build id: 3.10.0-1160.92.1.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffffacecb550
Build id: 3.10.0-1160.92.1.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffffad5c51f7
Build id: 3.10.0-1160.92.1.el7.x86_64
0x21
12
? insert_kthread_work
vmlinux 0xffffffffacecb550
Build id: 3.10.0-1160.92.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u65:2:19148]
Modules linked in: nfsv3 nfs fscache rpcsec_gss_krb5 xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter w83795 w83627ehf hwmon_vid dm_mirror dm_region_hash dm_log dm_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw sp5100_tco gf128mul glue_helper ablk_helper cryptd pcspkr k10temp fam15h_power joydev i2c_piix4 ipmi_ssif osst st sg ipmi_si ipmi_devintf ipmi_msghandler tpm_infineon acpi_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic
qla2xxx ast i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci e1000e libahci pata_atiixp mpt2sas nvme_fc libata crct10dif_pclmul crct10dif_common nvme_fabrics crc32c_intel firewire_ohci nvme_core firewire_core serio_raw scsi_transport_fc drm_panel_orientation_quirks crc_itu_t ptp raid_class scsi_transport_sas scsi_tgt pps_core fuse
CPU: 0 PID: 19148 Comm: kworker/u65:2 Kdump: loaded Tainted: G        W      ------------   3.10.0-1160.92.1.el7.x86_64 #1
Hardware name: ASUSTeK Computer INC. KGP(M)E-D16/KGP(M)E-D16, BIOS 3201    02/19/2013
Workqueue: writeback bdi_writeback_workfn (flush-8:240)
task: ffff9da9689c8000 ti: ffff9da81e9b0000 task.ti: ffff9da81e9b0000
RIP: 0010:[<ffffffffad1bd183>]  [<ffffffffad1bd183>] __list_add+0x33/0xc0
RSP: 0018:ffff9da81e9b3c68  EFLAGS: 00000202
RAX: ffff9da81e9b3cb0 RBX: ffff9da98dfcc728 RCX: dead000000000200
RDX: ffff9daa7e81ddb8 RSI: ffff9da81e9b3cb0 RDI: ffff9daa7e8a15b8
RBP: ffff9da81e9b3c80 R08: ffff9daa7e81ddb8 R09: 0000000000000002
R10: 00000000000007e9 R11: ffff9da81e9b38c6 R12: fffffffface9f83a
R13: ffff9da81e9b3c08 R14: ffff9da81e9b3c18 R15: ffff9da81e9b3bc8
FS:  00007f446b528840(0000) GS:ffff9da98fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000001818068 CR3: 0000000c0f1ac000 CR4: 00000000000407f0
Call Trace:
[<ffffffffad08dc5b>] queue_io+0xcb/0x240
[<ffffffffad08e017>] wb_writeback+0x247/0x2f0
[<ffffffffacfd6180>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffffad08eb3c>] bdi_writeback_workfn+0x1cc/0x470
[<ffffffffacec32ef>] process_one_work+0x17f/0x440
[<ffffffffacec4436>] worker_thread+0x126/0x3c0
[<ffffffffacec4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffffacecb621>] kthread+0xd1/0xe0
[<ffffffffacecb550>] ? insert_kthread_work+0x40/0x40
[<ffffffffad5c51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffacecb550>] ? insert_kthread_work+0x40/0x40
Code: f5 41 54 49 89 d4 53 4c 8b 42 08 48 89 fb 49 39 f0 75 2e 4d 8b 45 00 4d 39 c4 75 6c 4c 39 e3 74 42 4c 39 eb 74 3d 49 89 5c 24 08 <4c> 89 6b 08 4c 89 23 49 89 5d 00 5b 41 5c 41 5d 5d c3 cc cc cc 
Complete report #750121
Frame # Function Binary Source or offset Line
1
__list_del_entry
vmlinux 0xffffffff9f7a676c
Build id: 3.10.0-1160.45.1.el7.x86_64
0x2c
2
queue_io
vmlinux 0xffffffff9f67ee8c
Build id: 3.10.0-1160.45.1.el7.x86_64
0xbc
3
wb_writeback
vmlinux 0xffffffff9f67f257
Build id: 3.10.0-1160.45.1.el7.x86_64
0x247
4
bdi_writeback_workfn
vmlinux 0xffffffff9f67fcb5
Build id: 3.10.0-1160.45.1.el7.x86_64
0x115
5
process_one_work
vmlinux 0xffffffff9f4bde8f
Build id: 3.10.0-1160.45.1.el7.x86_64
0x17f
6
worker_thread
vmlinux 0xffffffff9f4befa6
Build id: 3.10.0-1160.45.1.el7.x86_64
0x126
7
? manage_workers.isra.26
vmlinux 0xffffffff9f4bee80
Build id: 3.10.0-1160.45.1.el7.x86_64
0x2a0
8
kthread
vmlinux 0xffffffff9f4c5e61
Build id: 3.10.0-1160.45.1.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffff9f4c5d90
Build id: 3.10.0-1160.45.1.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffff9fb95ddd
Build id: 3.10.0-1160.45.1.el7.x86_64
0x7
11
? insert_kthread_work
vmlinux 0xffffffff9f4c5d90
Build id: 3.10.0-1160.45.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#22 stuck for 23s! [kworker/u258:3:88853]
Modules linked in: tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag squashfs overlay(T) 8021q garp mrp stp llc mmfs26(OE) mmfslinux(OE) tracedev(OE) loop nfsv3 rpcsec_gss_krb5 nfs_acl auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6table_raw ip6table_filter ip6_tables rpcrdma iptable_raw iptable_security iptable_mangle iptable_nat nf_nat_ipv4 ib_isert nf_nat iscsi_target_mod nf_conntrack_ipv4 nf_defrag_ipv4 xt_comment xt_multiport xt_conntrack ib_iser libiscsi nf_conntrack scsi_transport_iscsi iptable_filter ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ipoib rdma_ucm ib_uverbs ib_umad sunrpc rdma_cm ib_cm iw_cm dm_mirror dm_region_hash dm_log dm_mod dell_smbios iTCO_wdt iTCO_vendor_support dell_wmi_descriptor dcdbas ext4 mbcache jbd2 i10nm_edac
nfit libnvdimm intel_powerclamp coretemp intel_rapl bnxt_re iosf_mbi kvm_intel ib_core kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg mei_me mei i2c_i801 wmi ipmi_si ipmi_devintf ipmi_msghandler pinctrl_lewisburg pinctrl_intel acpi_power_meter acpi_pad binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul tg3 ahci crct10dif_common crc32c_intel bnxt_en libahci megaraid_sas nvme libata ptp nvme_core devlink drm_panel_orientation_quirks pps_core
CPU: 22 PID: 88853 Comm: kworker/u258:3 Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R650/0PYXKY, BIOS 1.5.5 02/10/2022
Workqueue: writeback bdi_writeback_workfn (flush-0:44)
task: ffff9a2b43d79080 ti: ffff9a174fd94000 task.ti: ffff9a174fd94000
RIP: 0010:[<ffffffff9f7a676c>]  [<ffffffff9f7a676c>] __list_del_entry+0x2c/0xd0
RSP: 0018:ffff9a174fd97c80  EFLAGS: 00000202
RAX: ffff9aa45f727ca8 RBX: 00000000b4f67c01 RCX: dead000000000200
RDX: ffff9a2d9a8749f0 RSI: ffff9a174fd97cb0 RDI: ffff9a9c578d22a8
RBP: ffff9a174fd97c80 R08: ffff9a9c578d22a8 R09: 000000018040002c
R10: 000000009e741401 R11: ffff9aad9e7414c0 R12: 00000000f8017d01
R13: ffff9a174fd97c88 R14: ffffffff9f62551a R15: ffff9a174fd97c80
FS:  0000000000000000(0000) GS:ffff9a2dbf6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000068018440000 CR3: 000000922f210000 CR4: 0000000000760fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffff9f67ee8c>] queue_io+0xbc/0x240
[<ffffffff9f67f257>] wb_writeback+0x247/0x2f0
[<ffffffff9f67fcb5>] bdi_writeback_workfn+0x115/0x460
[<ffffffff9f4bde8f>] process_one_work+0x17f/0x440
[<ffffffff9f4befa6>] worker_thread+0x126/0x3c0
[<ffffffff9f4bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff9f4c5e61>] kthread+0xd1/0xe0
[<ffffffff9f4c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffff9fb95ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff9f4c5d90>] ? insert_kthread_work+0x40/0x40
Code: 48 8b 17 48 b9 00 01 00 00 00 00 ad de 48 8b 47 08 48 89 e5 48 39 ca 74 29 48 b9 00 02 00 00 00 00 ad de 48 39 c8 74 7a 4c 8b 00 <4c> 39 c7 75 53 4c 8b 42 08 4c 39 c7 75 2b 48 89 42 08 48 89 10 
Complete report #684120
Frame # Function Binary Source or offset Line
1
queue_io
vmlinux 0xffffffff91c8d772
Build id: 3.10.0-1160.83.1.el7.x86_64
0xd2
2
wb_writeback
vmlinux 0xffffffff91c8db27
Build id: 3.10.0-1160.83.1.el7.x86_64
0x247
3
bdi_writeback_workfn
vmlinux 0xffffffff91c8e595
Build id: 3.10.0-1160.83.1.el7.x86_64
0x115
4
process_one_work
vmlinux 0xffffffff91ac31df
Build id: 3.10.0-1160.83.1.el7.x86_64
0x17f
5
worker_thread
vmlinux 0xffffffff91ac4326
Build id: 3.10.0-1160.83.1.el7.x86_64
0x126
6
? manage_workers.isra.26
vmlinux 0xffffffff91ac4200
Build id: 3.10.0-1160.83.1.el7.x86_64
0x2b0
7
kthread
vmlinux 0xffffffff91acb511
Build id: 3.10.0-1160.83.1.el7.x86_64
0xd1
8
? insert_kthread_work
vmlinux 0xffffffff91acb440
Build id: 3.10.0-1160.83.1.el7.x86_64
0x40
9
ret_from_fork_nospec_begin
vmlinux 0xffffffff921c51f7
Build id: 3.10.0-1160.83.1.el7.x86_64
0x21
10
? insert_kthread_work
vmlinux 0xffffffff91acb440
Build id: 3.10.0-1160.83.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u32:2:28525]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_14812(E) falcon_lsm_pinned_14713(E) falcon_lsm_pinned_14604(E) tcp_diag inet_diag xt_nat veth xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink binfmt_misc overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ppdev nfit libnvdimm parport_pc i2c_piix4 parport pcspkr iosf_mbi auth_rpcgss
ip_tables xfs libcrc32c crct10dif_pclmul crct10dif_common 8021q crc32_pclmul crc32c_intel garp mrp stp ghash_clmulni_intel llc scsi_transport_iscsi aesni_intel lrw gf128mul glue_helper ablk_helper nvme cryptd serio_raw ena nvme_core sunrpc dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal]
CPU: 1 PID: 28525 Comm: kworker/u32:2 Kdump: loaded Tainted: P            EL ------------ T 3.10.0-1160.83.1.el7.x86_64 #1
Hardware name: Amazon EC2 c5.4xlarge/, BIOS 1.0 10/16/2017
Workqueue: writeback bdi_writeback_workfn (flush-259:0)
task: ffff9146586e0000 ti: ffff914d9aa68000 task.ti: ffff914d9aa68000
RIP: 0010:[<ffffffff91c8d772>]  [<ffffffff91c8d772>] queue_io+0xd2/0x240
RSP: 0018:ffff914d9aa6bc90  EFLAGS: 00000206
RAX: ffff914672957000 RBX: 0000000066cdae01 RCX: dead000000000200
RDX: ffff914e151afa30 RSI: ffff914d9aa6bcb0 RDI: ffff914e151af2b0
RBP: ffff914d9aa6bcf0 R08: ffff914e151afa30 R09: 000000018040001d
R10: 0000000066cda401 R11: ffff914766cda480 R12: ffff9146f6e5cd30
R13: ffff914d9aa6bc88 R14: 000000001333a501 R15: ffff914e1333ac40
FS:  0000000000000000(0000) GS:ffff914e1f240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7c2fc7b59f CR3: 000000024c410000 CR4: 00000000007606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffff91c8db27>] wb_writeback+0x247/0x2f0
[<ffffffff91c8e595>] bdi_writeback_workfn+0x115/0x470
[<ffffffff91ac31df>] process_one_work+0x17f/0x440
[<ffffffff91ac4326>] worker_thread+0x126/0x3c0
[<ffffffff91ac4200>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff91acb511>] kthread+0xd1/0xe0
[<ffffffff91acb440>] ? insert_kthread_work+0x40/0x40
[<ffffffff921c51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff91acb440>] ? insert_kthread_work+0x40/0x40
Code: 75 c0 4c 89 75 c8 eb 47 90 4c 89 ef 41 83 c4 01 e8 b4 f5 12 00 48 8b 55 c0 4c 89 f6 4c 89 ef e8 e5 f4 12 00 49 8b 85 40 ff ff ff <48> 3b 05 ef 83 ad 00 0f 84 11 01 00 00 4d 85 ff 74 0f 4c 39 f8 
Complete report #80001
Frame # Function Binary Source or offset Line
1
queue_io
vmlinux 0xffffffffa08712ec
Build id: 3.10.0-957.12.2.el7.x86_64
0xfc
2
wb_writeback
vmlinux 0xffffffffa0871687
Build id: 3.10.0-957.12.2.el7.x86_64
0x247
3
bdi_writeback_workfn
vmlinux 0xffffffffa0871e75
Build id: 3.10.0-957.12.2.el7.x86_64
0x115
4
? try_to_wake_up
vmlinux 0xffffffffa06d6820
Build id: 3.10.0-957.12.2.el7.x86_64
0x190
5
process_one_work
vmlinux 0xffffffffa06b9e3f
Build id: 3.10.0-957.12.2.el7.x86_64
0x17f
6
worker_thread
vmlinux 0xffffffffa06baed6
Build id: 3.10.0-957.12.2.el7.x86_64
0x126
7
? manage_workers.isra.25
vmlinux 0xffffffffa06badb0
Build id: 3.10.0-957.12.2.el7.x86_64
0x2a0
8
kthread
vmlinux 0xffffffffa06c1d21
Build id: 3.10.0-957.12.2.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffffa06c1c50
Build id: 3.10.0-957.12.2.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffffa0d75c37
Build id: 3.10.0-957.12.2.el7.x86_64
0x21
11
? insert_kthread_work
vmlinux 0xffffffffa06c1c50
Build id: 3.10.0-957.12.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#22 stuck for 22s! [kworker/u65:0:19849]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) ext4 mbcache jbd2 overlay(T) squashfs loop nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache 8021q garp mrp stp llc ip6table_filter ip6table_raw ip6_tables xt_comment xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev i2c_i801 lpc_ich hpilo hpwdt ioatdma ipmi_si sg ipmi_devintf ipmi_msghandler pcc_cpufreq wmi acpi_power_meter binfmt_misc
ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common serio_raw crc32c_intel ixgbe ahci drm_panel_orientation_quirks libahci igb libata hpsa mdio ptp pps_core dca scsi_transport_sas i2c_algo_bit [last unloaded: tracedev]
CPU: 22 PID: 19849 Comm: kworker/u65:0 Kdump: loaded Tainted: G        W  OE  ------------ T 3.10.0-957.12.2.el7.x86_64 #1
Hardware name: HP ProLiant DL160 Gen9/ProLiant DL160 Gen9, BIOS U20 07/21/2019
Workqueue: writeback bdi_writeback_workfn (flush-8:16)
task: ffff8df46dcc1040 ti: ffff8decfcea0000 task.ti: ffff8decfcea0000
RIP: 0010:[<ffffffffa08712ec>]  [<ffffffffa08712ec>] queue_io+0xfc/0x250
RSP: 0018:ffff8decfcea3c90  EFLAGS: 00000206
RAX: ffff8dfe004f0248 RBX: 000000000f0a0701 RCX: dead000000000200
RDX: ffff8e038f490248 RSI: ffff8decfcea3cb0 RDI: ffff8e147cfe9000
RBP: ffff8decfcea3cf0 R08: ffff8e038f490248 R09: 0000000180400011
R10: 00000000bea22d01 R11: fffffe1c80fa8880 R12: ffffffffa081a2c2
R13: ffff8decfcea3cf0 R14: 0000000000000000 R15: ffff8dd5ffc000c0
FS:  0000000000000000(0000) GS:ffff8df47fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aecacacf010 CR3: 0000001bf078a000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa0871687>] wb_writeback+0x247/0x2f0
[<ffffffffa0871e75>] bdi_writeback_workfn+0x115/0x460
[<ffffffffa06d6820>] ? try_to_wake_up+0x190/0x390
[<ffffffffa06b9e3f>] process_one_work+0x17f/0x440
[<ffffffffa06baed6>] worker_thread+0x126/0x3c0
[<ffffffffa06badb0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffa06c1d21>] kthread+0xd1/0xe0
[<ffffffffa06c1c50>] ? insert_kthread_work+0x40/0x40
[<ffffffffa0d75c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa06c1c50>] ? insert_kthread_work+0x40/0x40
Code: 49 8b 4f e8 48 39 08 78 5e 4c 89 ff 41 83 c4 01 e8 9a 3e 12 00 48 8b 55 c0 4c 89 ee 4c 89 ff e8 cb 3d 12 00 49 8b bf 40 ff ff ff <e8> 0f 14 01 00 85 c0 75 ab 4d 85 f6 0f 84 f2 00 00 00 49 8b 87 
Complete report #37793
Frame # Function Binary Source or offset Line
1
__list_del_entry
vmlinux 0xffffffffb899519c
Build id: 3.10.0-957.12.2.el7.x86_64
0x2c
2
queue_io
vmlinux 0xffffffffb88712d6
Build id: 3.10.0-957.12.2.el7.x86_64
0xe6
3
wb_writeback
vmlinux 0xffffffffb8871687
Build id: 3.10.0-957.12.2.el7.x86_64
0x247
4
bdi_writeback_workfn
vmlinux 0xffffffffb8871e75
Build id: 3.10.0-957.12.2.el7.x86_64
0x115
5
? __schedule
vmlinux 0xffffffffb8d689ba
Build id: 3.10.0-957.12.2.el7.x86_64
0x42a
6
process_one_work
vmlinux 0xffffffffb86b9e3f
Build id: 3.10.0-957.12.2.el7.x86_64
0x17f
7
worker_thread
vmlinux 0xffffffffb86baed6
Build id: 3.10.0-957.12.2.el7.x86_64
0x126
8
? manage_workers.isra.25
vmlinux 0xffffffffb86badb0
Build id: 3.10.0-957.12.2.el7.x86_64
0x2a0
9
kthread
vmlinux 0xffffffffb86c1d21
Build id: 3.10.0-957.12.2.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffffb86c1c50
Build id: 3.10.0-957.12.2.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffffb8d75c37
Build id: 3.10.0-957.12.2.el7.x86_64
0x21
12
? insert_kthread_work
vmlinux 0xffffffffb86c1c50
Build id: 3.10.0-957.12.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#62 stuck for 23s! [kworker/u146:0:35404]
Modules linked in: 8021q garp mrp stp llc squashfs loop overlay(T) mmfs26(OE) mmfslinux(OE) tracedev(OE) rpcsec_gss_krb5 auth_rpcgss nfsv3 nfs_acl nfsv4 dns_resolver nfs lockd grace fscache ip6table_filter ip6table_raw ip6_tables ib_isert iscsi_target_mod xt_comment xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ib_srpt target_core_mod nf_conntrack iptable_mangle iptable_security iptable_raw ib_srp scsi_transport_srp scsi_tgt ib_ucm rpcrdma dm_mirror dm_region_hash dm_log sunrpc dm_mod rdma_ucm ib_uverbs ib_iser rdma_cm iw_cm libiscsi ib_umad ib_ipoib scsi_transport_iscsi ib_cm iTCO_wdt iTCO_vendor_support mxm_wmi ext4 sb_edac mbcache intel_powerclamp coretemp jbd2 intel_rapl iosf_mbi kvm irqbypass crc32_pclmul ghash_clmulni_intel
cryptd mlx5_ib ib_core joydev pcspkr mei_me sg i2c_i801 mei lpc_ich ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_power_meter acpi_pad binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops mlx5_core ttm igb ahci mlxfw devlink drm crct10dif_pclmul libahci ptp crct10dif_common crc32c_intel nvme libata pps_core drm_panel_orientation_quirks nvme_core dca i2c_algo_bit
CPU: 62 PID: 35404 Comm: kworker/u146:0 Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-957.12.2.el7.x86_64 #1
Hardware name: Supermicro SYS-6018R-TDW/X10DDW-i, BIOS 3.0a 03/01/2018
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff9345a0835140 ti: ffff93394cea4000 task.ti: ffff93394cea4000
RIP: 0010:[<ffffffffb899519c>]  [<ffffffffb899519c>] __list_del_entry+0x2c/0xd0
RSP: 0018:ffff93394cea7c80  EFLAGS: 00000206
RAX: ffff92e3dc1a0248 RBX: 0000000061bae101 RCX: dead000000000200
RDX: ffff930676358380 RSI: ffff93394cea7cb0 RDI: ffff931edd8b7688
RBP: ffff93394cea7c80 R08: ffff931edd8b7688 R09: 0000000180400020
R10: 0000000061bae001 R11: ffffeafd7d86eb80 R12: 0000000061bae901
R13: ffff93394cea7c88 R14: ffff92c87fc03b00 R15: ffff93394cea7c80
FS:  0000000000000000(0000) GS:ffff93467f280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2abfd93000 CR3: 0000007f0178c000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb88712d6>] queue_io+0xe6/0x250
[<ffffffffb8871687>] wb_writeback+0x247/0x2f0
[<ffffffffb8871e75>] bdi_writeback_workfn+0x115/0x460
[<ffffffffb8d689ba>] ? __schedule+0x42a/0x860
[<ffffffffb86b9e3f>] process_one_work+0x17f/0x440
[<ffffffffb86baed6>] worker_thread+0x126/0x3c0
[<ffffffffb86badb0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffb86c1d21>] kthread+0xd1/0xe0
[<ffffffffb86c1c50>] ? insert_kthread_work+0x40/0x40
[<ffffffffb8d75c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb86c1c50>] ? insert_kthread_work+0x40/0x40
Code: 48 8b 17 48 b9 00 01 00 00 00 00 ad de 48 8b 47 08 48 89 e5 48 39 ca 74 29 48 b9 00 02 00 00 00 00 ad de 48 39 c8 74 7a 4c 8b 00 <4c> 39 c7 75 53 4c 8b 42 08 4c 39 c7 75 2b 48 89 42 08 48 89 10 
Complete report #533663
Frame # Function Binary Source or offset Line
1
__list_del_entry
vmlinux 0xffffffff9dfa4d2c
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2c
2
queue_io
vmlinux 0xffffffff9de7dbbc
Build id: 3.10.0-1127.19.1.el7.x86_64
0xbc
3
wb_writeback
vmlinux 0xffffffff9de7df87
Build id: 3.10.0-1127.19.1.el7.x86_64
0x247
4
? get_nr_inodes
vmlinux 0xffffffff9de69c5c
Build id: 3.10.0-1127.19.1.el7.x86_64
0x4c
5
bdi_writeback_workfn
vmlinux 0xffffffff9de7eb9b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2cb
6
process_one_work
vmlinux 0xffffffff9dcbe6bf
Build id: 3.10.0-1127.19.1.el7.x86_64
0x17f
7
worker_thread
vmlinux 0xffffffff9dcbf7d6
Build id: 3.10.0-1127.19.1.el7.x86_64
0x126
8
? manage_workers.isra.26
vmlinux 0xffffffff9dcbf6b0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2a0
9
kthread
vmlinux 0xffffffff9dcc6691
Build id: 3.10.0-1127.19.1.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffff9dcc65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffff9e392d37
Build id: 3.10.0-1127.19.1.el7.x86_64
0x21
12
? insert_kthread_work
vmlinux 0xffffffff9dcc65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 28s! [kworker/u2:1:6270]
Modules linked in: tcp_lp nls_utf8 isofs fuse vboxsf(OE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc ext4 intel_pmc_core vboxvideo(OE) intel_powerclamp ttm iosf_mbi crc32_pclmul joydev drm_kms_helper ghash_clmulni_intel snd_intel8x0 ppdev mbcache snd_ac97_codec jbd2 syscopyarea ac97_bus sysfillrect sysimgblt snd_seq fb_sys_fops aesni_intel
drm snd_seq_device snd_pcm lrw gf128mul glue_helper ablk_helper cryptd snd_timer pcspkr sg snd i2c_piix4 vboxguest(OE) soundcore drm_panel_orientation_quirks parport_pc parport video sch_fq_codel ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod cdrom crct10dif_generic ata_generic pata_acpi ahci libahci ata_piix libata e1000 crct10dif_pclmul crct10dif_common crc32c_intel serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 6270 Comm: kworker/u2:1 Kdump: loaded Tainted: G           OE  ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Workqueue: writeback bdi_writeback_workfn (flush-vboxsf-2)
task: ffffa149d48920e0 ti: ffffa149d5778000 task.ti: ffffa149d5778000
RIP: 0010:[<ffffffff9dfa4d2c>]  [<ffffffff9dfa4d2c>] __list_del_entry+0x2c/0xd0
RSP: 0018:ffffa149d577bc80  EFLAGS: 00000206
RAX: ffffa149d8f907d8 RBX: ffffa149d577bca8 RCX: dead000000000200
RDX: ffffa148f5f56e50 RSI: ffffa149d577bcb0 RDI: ffffa149d8f8fca8
RBP: ffffa149d577bc80 R08: ffffa149d8f8fca8 R09: 000000000000ffff
R10: 7adbc9733eb56dd0 R11: 0000000000000000 R12: ffffffff9de7ce94
R13: ffffa149d577bc00 R14: ffffa14992f39000 R15: ffffa1499138e000
FS:  0000000000000000(0000) GS:ffffa149dfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fcf8b4d3000 CR3: 00000000bd234000 CR4: 00000000000606f0
Call Trace:
[<ffffffff9de7dbbc>] queue_io+0xbc/0x240
[<ffffffff9de7df87>] wb_writeback+0x247/0x2f0
[<ffffffff9de69c5c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9de7eb9b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff9dcbe6bf>] process_one_work+0x17f/0x440
[<ffffffff9dcbf7d6>] worker_thread+0x126/0x3c0
[<ffffffff9dcbf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff9dcc6691>] kthread+0xd1/0xe0
[<ffffffff9dcc65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff9e392d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9dcc65c0>] ? insert_kthread_work+0x40/0x40
Code: 48 8b 17 48 b9 00 01 00 00 00 00 ad de 48 8b 47 08 48 89 e5 48 39 ca 74 29 48 b9 00 02 00 00 00 00 ad de 48 39 c8 74 7a 4c 8b 00 <4c> 39 c7 75 53 4c 8b 42 08 4c 39 c7 75 2b 48 89 42 08 48 89 10 
Complete report #470184
Frame # Function Binary Source or offset Line
1
dump_stack
vmlinux 0xffffffff94d865a9
Build id: 3.10.0-1160.66.1.el7.x86_64
0x19
2
__warn
vmlinux 0xffffffff9469b408
Build id: 3.10.0-1160.66.1.el7.x86_64
0xd8
3
warn_slowpath_fmt
vmlinux 0xffffffff9469b48f
Build id: 3.10.0-1160.66.1.el7.x86_64
0x5f
4
__list_del_entry
vmlinux 0xffffffff949a65d1
Build id: 3.10.0-1160.66.1.el7.x86_64
0xa1
5
queue_io
vmlinux 0xffffffff9487eebc
Build id: 3.10.0-1160.66.1.el7.x86_64
0xbc
6
wb_writeback
vmlinux 0xffffffff9487f287
Build id: 3.10.0-1160.66.1.el7.x86_64
0x247
7
? get_nr_inodes
vmlinux 0xffffffff9486ac1c
Build id: 3.10.0-1160.66.1.el7.x86_64
0x4c
8
bdi_writeback_workfn
vmlinux 0xffffffff9487fe9b
Build id: 3.10.0-1160.66.1.el7.x86_64
0x2cb
9
process_one_work
vmlinux 0xffffffff946bdfbf
Build id: 3.10.0-1160.66.1.el7.x86_64
0x17f
10
worker_thread
vmlinux 0xffffffff946bf0d6
Build id: 3.10.0-1160.66.1.el7.x86_64
0x126
11
? manage_workers.isra.26
vmlinux 0xffffffff946befb0
Build id: 3.10.0-1160.66.1.el7.x86_64
0x2a0
12
kthread
vmlinux 0xffffffff946c5f91
Build id: 3.10.0-1160.66.1.el7.x86_64
0xd1
13
? insert_kthread_work
vmlinux 0xffffffff946c5ec0
Build id: 3.10.0-1160.66.1.el7.x86_64
0x40
14
ret_from_fork_nospec_begin
vmlinux 0xffffffff94d99df7
Build id: 3.10.0-1160.66.1.el7.x86_64
0x21
15
? insert_kthread_work
vmlinux 0xffffffff946c5ec0
Build id: 3.10.0-1160.66.1.el7.x86_64
0x40
WARNING: CPU: 1 PID: 22276 at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0
list_del corruption. prev->next should be ffff9299c78471b8, but was feff9299c78471b8
Modules linked in: tcp_lp xt_nat veth nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) dm_crypt drbg ansi_cprng ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter hwmon_vid sunrpc ext4 mbcache jbd2 intel_powerclamp coretemp intel_rapl kvm_intel kvm ppdev mei_wdt iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul ghash_clmulni_intel snd_hda_codec_realtek
aesni_intel snd_hda_codec_generic snd_hda_codec_hdmi lrw gf128mul glue_helper sg ablk_helper cryptd pcspkr snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm parport_pc parport mei_me ite_cir rc_core snd_timer snd mei i2c_i801 soundcore lpc_ich ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i915 i2c_algo_bit iosf_mbi drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci drm libahci libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw r8169 drm_panel_orientation_quirks video dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ipmi_msghandler]
CPU: 1 PID: 22276 Comm: kworker/u8:2 Kdump: loaded Tainted: G        W      ------------ T 3.10.0-1160.66.1.el7.x86_64 #1
Hardware name: BIOSTAR Group Hi-Fi H87S3+/Hi-Fi H87S3+, BIOS 4.6.5 05/09/2013
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
Call Trace:
[<ffffffff94d865a9>] dump_stack+0x19/0x1b
[<ffffffff9469b408>] __warn+0xd8/0x100
[<ffffffff9469b48f>] warn_slowpath_fmt+0x5f/0x80
[<ffffffff949a65d1>] __list_del_entry+0xa1/0xd0
[<ffffffff9487eebc>] queue_io+0xbc/0x240
[<ffffffff9487f287>] wb_writeback+0x247/0x2f0
[<ffffffff9486ac1c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9487fe9b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff946bdfbf>] process_one_work+0x17f/0x440
[<ffffffff946bf0d6>] worker_thread+0x126/0x3c0
[<ffffffff946befb0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff946c5f91>] kthread+0xd1/0xe0
[<ffffffff946c5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffff94d99df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff946c5ec0>] ? insert_kthread_work+0x40/0x40
Complete report #352087
Frame # Function Binary Source or offset Line
1
__list_del_entry
vmlinux 0xffffffffa31a622e
Build id: 3.10.0-1160.31.1.el7.x86_64
0x3e
2
queue_io
vmlinux 0xffffffffa307e94c
Build id: 3.10.0-1160.31.1.el7.x86_64
0xbc
3
wb_writeback
vmlinux 0xffffffffa307ed17
Build id: 3.10.0-1160.31.1.el7.x86_64
0x247
4
? get_nr_inodes
vmlinux 0xffffffffa306a95c
Build id: 3.10.0-1160.31.1.el7.x86_64
0x4c
5
bdi_writeback_workfn
vmlinux 0xffffffffa307f92b
Build id: 3.10.0-1160.31.1.el7.x86_64
0x2cb
6
process_one_work
vmlinux 0xffffffffa2ebde5f
Build id: 3.10.0-1160.31.1.el7.x86_64
0x17f
7
worker_thread
vmlinux 0xffffffffa2ebef76
Build id: 3.10.0-1160.31.1.el7.x86_64
0x126
8
? manage_workers.isra.26
vmlinux 0xffffffffa2ebee50
Build id: 3.10.0-1160.31.1.el7.x86_64
0x2a0
9
kthread
vmlinux 0xffffffffa2ec5e31
Build id: 3.10.0-1160.31.1.el7.x86_64
0xd1
10
? SyS_futex
vmlinux 0xffffffffa2f151f0
Build id: 3.10.0-1160.31.1.el7.x86_64
0x80
11
? insert_kthread_work
vmlinux 0xffffffffa2ec5d60
Build id: 3.10.0-1160.31.1.el7.x86_64
0x40
12
ret_from_fork_nospec_begin
vmlinux 0xffffffffa3595df7
Build id: 3.10.0-1160.31.1.el7.x86_64
0x21
13
? insert_kthread_work
vmlinux 0xffffffffa2ec5d60
Build id: 3.10.0-1160.31.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 41s! [kworker/u256:2:57547]
Modules linked in: ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables ipt_rpfilter xt_set iptable_raw ip_set_hash_ip ip_set_hash_net ip_set ipip tunnel4 ip_tunnel xt_nat xt_statistic ipt_REJECT nf_reject_ipv4 iptable_mangle xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_nat_ipv4 nf_nat rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache overlay(T) vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev joydev sg pcspkr vmw_balloon parport_pc parport vmw_vmci i2c_piix4 nfsd auth_rpcgss nf_conntrack_ipv4 nfs_acl nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs lockd
grace sunrpc nf_conntrack libcrc32c ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ahci ata_piix libahci libata serio_raw nfit libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse br_netfilter bridge stp llc
CPU: 0 PID: 57547 Comm: kworker/u256:2 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffffa080bde76300 ti: ffffa08414114000 task.ti: ffffa08414114000
RIP: 0010:[<ffffffffa31a622e>]  [<ffffffffa31a622e>] __list_del_entry+0x3e/0xd0
RSP: 0018:ffffa08414117c80  EFLAGS: 00010246
RAX: ffffa07c3b6e91b8 RBX: ffffa08414117ca8 RCX: dead000000000200
RDX: ffffa08417cb8380 RSI: ffffa08414117cb0 RDI: ffffa07e3034edb8
RBP: ffffa08414117c80 R08: ffffa07e3034edb8 R09: 000000000000ffff
R10: 7def41560bcb8300 R11: 0000000000000005 R12: ffffffffa307dc24
R13: ffffa08414117c00 R14: ffffa07d7fd51000 R15: ffffa07c365180f0
FS:  0000000000000000(0000) GS:ffffa0841d600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000166030 CR3: 0000000815192000 CR4: 00000000001607f0
Call Trace:
[<ffffffffa307e94c>] queue_io+0xbc/0x240
[<ffffffffa307ed17>] wb_writeback+0x247/0x2f0
[<ffffffffa306a95c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffa307f92b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa2ebde5f>] process_one_work+0x17f/0x440
[<ffffffffa2ebef76>] worker_thread+0x126/0x3c0
[<ffffffffa2ebee50>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffa2ec5e31>] kthread+0xd1/0xe0
[<ffffffffa2f151f0>] ? SyS_futex+0x80/0x190
[<ffffffffa2ec5d60>] ? insert_kthread_work+0x40/0x40
[<ffffffffa3595df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa2ec5d60>] ? insert_kthread_work+0x40/0x40
Code: 89 e5 48 39 ca 74 29 48 b9 00 02 00 00 00 00 ad de 48 39 c8 74 7a 4c 8b 00 4c 39 c7 75 53 4c 8b 42 08 4c 39 c7 75 2b 48 89 42 08 <48> 89 10 5d c3 49 89 d0 48 89 f9 48 c7 c2 f8 1f 8b a3 be 35 00 
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:37:crtc-0] flip_done timed out
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:37:crtc-0] flip_done timed out
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:37:crtc-0] flip_done timed out
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:33:plane-0] flip_done timed out
Complete report #322600
Frame # Function Binary Source or offset Line
1
__list_del_entry
vmlinux 0xffffffff9239692e
Build id: 3.10.0-957.27.2.el7.x86_64
0x3e
2
queue_io
vmlinux 0xffffffff92272286
Build id: 3.10.0-957.27.2.el7.x86_64
0xe6
3
wb_writeback
vmlinux 0xffffffff92272637
Build id: 3.10.0-957.27.2.el7.x86_64
0x247
4
? get_nr_inodes
vmlinux 0xffffffff9225e46c
Build id: 3.10.0-957.27.2.el7.x86_64
0x4c
5
bdi_writeback_workfn
vmlinux 0xffffffff92272fdb
Build id: 3.10.0-957.27.2.el7.x86_64
0x2cb
6
process_one_work
vmlinux 0xffffffff920baf9f
Build id: 3.10.0-957.27.2.el7.x86_64
0x17f
7
worker_thread
vmlinux 0xffffffff920bc036
Build id: 3.10.0-957.27.2.el7.x86_64
0x126
8
? manage_workers.isra.25
vmlinux 0xffffffff920bbf10
Build id: 3.10.0-957.27.2.el7.x86_64
0x2a0
9
kthread
vmlinux 0xffffffff920c2e81
Build id: 3.10.0-957.27.2.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffff920c2db0
Build id: 3.10.0-957.27.2.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffff92776c37
Build id: 3.10.0-957.27.2.el7.x86_64
0x21
12
? insert_kthread_work
vmlinux 0xffffffff920c2db0
Build id: 3.10.0-957.27.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u256:1:115116]
Modules linked in: tcp_diag udp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev sb_edac vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci crct10dif_pclmul crct10dif_common crc32c_intel libahci ata_piix drm libata nfit libnvdimm
serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 115116 Comm: kworker/u256:1 Kdump: loaded Tainted: G             L ------------   3.10.0-957.27.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff92ffc0b8c100 ti: ffff92fcda93c000 task.ti: ffff92fcda93c000
RIP: 0010:[<ffffffff9239692e>]  [<ffffffff9239692e>] __list_del_entry+0x3e/0xd0
RSP: 0000:ffff92fcda93fc80  EFLAGS: 00010246
RAX: ffff9302639ebe48 RBX: ffff92fcda93fcb0 RCX: dead000000000200
RDX: ffff93025cf30380 RSI: ffff92fcda93fcb0 RDI: ffff92ff83089508
RBP: ffff92fcda93fc80 R08: ffff92ff83089508 R09: ffff92ffc0b8c100
R10: 00000000003dbba9 R11: 0000000000000000 R12: 00000000003dbba9
R13: 0000000000000000 R14: ffff92fcda93fc00 R15: ffff92fcda93fc80
FS:  0000000000000000(0000) GS:ffff930265640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f904eb61000 CR3: 00000005fe2ec000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff92272286>] queue_io+0xe6/0x250
[<ffffffff92272637>] wb_writeback+0x247/0x2f0
[<ffffffff9225e46c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff92272fdb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff920baf9f>] process_one_work+0x17f/0x440
[<ffffffff920bc036>] worker_thread+0x126/0x3c0
[<ffffffff920bbf10>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff920c2e81>] kthread+0xd1/0xe0
[<ffffffff920c2db0>] ? insert_kthread_work+0x40/0x40
[<ffffffff92776c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff920c2db0>] ? insert_kthread_work+0x40/0x40
Code: 89 e5 48 39 ca 74 29 48 b9 00 02 00 00 00 00 ad de 48 39 c8 74 7a 4c 8b 00 4c 39 c7 75 53 4c 8b 42 08 4c 39 c7 75 2b 48 89 42 08 <48> 89 10 5d c3 49 89 d0 48 89 f9 48 c7 c2 a0 5b aa 92 be 35 00 
Complete report #258682
Frame # Function Binary Source or offset Line
1
__list_del_entry
vmlinux 0xffffffffbd5a4d3e
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3e
2
queue_io
vmlinux 0xffffffffbd47dbbc
Build id: 3.10.0-1127.19.1.el7.x86_64
0xbc
3
wb_writeback
vmlinux 0xffffffffbd47df87
Build id: 3.10.0-1127.19.1.el7.x86_64
0x247
4
? get_nr_inodes
vmlinux 0xffffffffbd469c5c
Build id: 3.10.0-1127.19.1.el7.x86_64
0x4c
5
bdi_writeback_workfn
vmlinux 0xffffffffbd47eb9b
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2cb
6
process_one_work
vmlinux 0xffffffffbd2be6bf
Build id: 3.10.0-1127.19.1.el7.x86_64
0x17f
7
worker_thread
vmlinux 0xffffffffbd2bf7d6
Build id: 3.10.0-1127.19.1.el7.x86_64
0x126
8
? manage_workers.isra.26
vmlinux 0xffffffffbd2bf6b0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2a0
9
kthread
vmlinux 0xffffffffbd2c6691
Build id: 3.10.0-1127.19.1.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffffbd2c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffffbd992d37
Build id: 3.10.0-1127.19.1.el7.x86_64
0x21
12
? insert_kthread_work
vmlinux 0xffffffffbd2c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/u4:2:23347]
Modules linked in: vsock_diag tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter ipmi_devintf ipmi_msghandler vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr joydev vmw_balloon parport_pc i2c_piix4 parport vmw_vmci binfmt_misc ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci nfit drm
libnvdimm ata_piix libahci e1000e crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw ptp pps_core vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 23347 Comm: kworker/u4:2 Kdump: loaded Tainted: G             L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8c8e4f6d8000 ti: ffff8c8f2c6b4000 task.ti: ffff8c8f2c6b4000
RIP: 0010:[<ffffffffbd5a4d3e>]  [<ffffffffbd5a4d3e>] __list_del_entry+0x3e/0xd0
RSP: 0018:ffff8c8f2c6b7c80  EFLAGS: 00010246
RAX: ffff8c8ea0420228 RBX: fffffffffffffffe RCX: dead000000000200
RDX: ffff8c8f28468380 RSI: ffff8c8f2c6b7cb0 RDI: ffff8c8bba9a5c68
RBP: ffff8c8f2c6b7c80 R08: ffff8c8bba9a5c68 R09: 000000000000ffff
R10: cdc2eb751c468300 R11: 0000000000000100 R12: 000000000000ffff
R13: 000000000000ffff R14: cdc2eb751c468300 R15: 0000000000000100
FS:  0000000000000000(0000) GS:ffff8c8f3fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6c29aeb6c8 CR3: 000000042d776000 CR4: 00000000003607f0
Call Trace:
[<ffffffffbd47dbbc>] queue_io+0xbc/0x240
[<ffffffffbd47df87>] wb_writeback+0x247/0x2f0
[<ffffffffbd469c5c>] ? get_nr_inodes+0x4c/0x70
[<ffffffffbd47eb9b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffbd2be6bf>] process_one_work+0x17f/0x440
[<ffffffffbd2bf7d6>] worker_thread+0x126/0x3c0
[<ffffffffbd2bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffbd2c6691>] kthread+0xd1/0xe0
[<ffffffffbd2c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffbd992d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffbd2c65c0>] ? insert_kthread_work+0x40/0x40
Code: 89 e5 48 39 ca 74 29 48 b9 00 02 00 00 00 00 ad de 48 39 c8 74 7a 4c 8b 00 4c 39 c7 75 53 4c 8b 42 08 4c 39 c7 75 2b 48 89 42 08 <48> 89 10 5d c3 49 89 d0 48 89 f9 48 c7 c2 18 09 cb bd be 35 00 
Complete report #224463
Frame # Function Binary Source or offset Line
1
__list_add
vmlinux 0xffffffff8c5a6163
Build id: 3.10.0-1160.31.1.el7.x86_64
0x33
2
queue_io
vmlinux 0xffffffff8c47e95b
Build id: 3.10.0-1160.31.1.el7.x86_64
0xcb
3
wb_writeback
vmlinux 0xffffffff8c47ed17
Build id: 3.10.0-1160.31.1.el7.x86_64
0x247
4
? get_nr_inodes
vmlinux 0xffffffff8c46a95c
Build id: 3.10.0-1160.31.1.el7.x86_64
0x4c
5
bdi_writeback_workfn
vmlinux 0xffffffff8c47f92b
Build id: 3.10.0-1160.31.1.el7.x86_64
0x2cb
6
process_one_work
vmlinux 0xffffffff8c2bde5f
Build id: 3.10.0-1160.31.1.el7.x86_64
0x17f
7
worker_thread
vmlinux 0xffffffff8c2bef76
Build id: 3.10.0-1160.31.1.el7.x86_64
0x126
8
? manage_workers.isra.26
vmlinux 0xffffffff8c2bee50
Build id: 3.10.0-1160.31.1.el7.x86_64
0x2a0
9
kthread
vmlinux 0xffffffff8c2c5e31
Build id: 3.10.0-1160.31.1.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffff8c2c5d60
Build id: 3.10.0-1160.31.1.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffff8c995df7
Build id: 3.10.0-1160.31.1.el7.x86_64
0x21
12
? insert_kthread_work
vmlinux 0xffffffff8c2c5d60
Build id: 3.10.0-1160.31.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/u16:1:31519]
Modules linked in: tcp_lp cfg80211 rfkill veth xt_nat xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink tun xt_addrtype br_netfilter devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ppdev joydev sg vmw_balloon parport_pc parport vmw_vmci
i2c_piix4 sch_fq_codel ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm sd_mod crc_t10dif ahci crct10dif_generic libahci ata_piix nfit crct10dif_pclmul crct10dif_common crc32c_intel libnvdimm libata serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 31519 Comm: kworker/u16:1 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9f54a4bee300 ti: ffff9f56703dc000 task.ti: ffff9f56703dc000
RIP: 0010:[<ffffffff8c5a6163>]  [<ffffffff8c5a6163>] __list_add+0x33/0xc0
RSP: 0000:ffff9f56703dfc68  EFLAGS: 00010202
RAX: ffff9f55399e8d70 RBX: ffff9f56b275c000 RCX: dead000000000200
RDX: ffff9f56703dfcb0 RSI: ffff9f56703dfcb0 RDI: ffff9f5691857df0
RBP: ffff9f56703dfc80 R08: ffff9f56703dfcb0 R09: 000000000000ffff
R10: 82a4e002a3d50300 R11: 0000000000000000 R12: ffffffff8c2c6ee5
R13: ffff9f56703dfbe8 R14: 0000000006a25540 R15: 0000000000000000
FS:  0000000000000000(0000) GS:ffff9f56bfcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f66c5beb710 CR3: 00000001d3084000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8c47e95b>] queue_io+0xcb/0x240
[<ffffffff8c47ed17>] wb_writeback+0x247/0x2f0
[<ffffffff8c46a95c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8c47f92b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff8c2bde5f>] process_one_work+0x17f/0x440
[<ffffffff8c2bef76>] worker_thread+0x126/0x3c0
[<ffffffff8c2bee50>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8c2c5e31>] kthread+0xd1/0xe0
[<ffffffff8c2c5d60>] ? insert_kthread_work+0x40/0x40
[<ffffffff8c995df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8c2c5d60>] ? insert_kthread_work+0x40/0x40
Code: f5 41 54 49 89 d4 53 4c 8b 42 08 48 89 fb 49 39 f0 75 2a 4d 8b 45 00 4d 39 c4 75 68 4c 39 e3 74 3e 4c 39 eb 74 39 49 89 5c 24 08 <4c> 89 6b 08 4c 89 23 49 89 5d 00 5b 41 5c 41 5d 5d c3 49 89 d1 
Complete report #210652
Frame # Function Binary Source or offset Line
1
__list_del_entry
vmlinux 0xffffffff8556834e
Build id: 3.10.0-862.el7.x86_64
0x3e
2
queue_io
vmlinux 0xffffffff85449d56
Build id: 3.10.0-862.el7.x86_64
0xe6
3
wb_writeback
vmlinux 0xffffffff8544a107
Build id: 3.10.0-862.el7.x86_64
0x247
4
? get_nr_inodes
vmlinux 0xffffffff8543665c
Build id: 3.10.0-862.el7.x86_64
0x4c
5
bdi_writeback_workfn
vmlinux 0xffffffff8544aaab
Build id: 3.10.0-862.el7.x86_64
0x2cb
6
process_one_work
vmlinux 0xffffffff852b2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
7
worker_thread
vmlinux 0xffffffff852b3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
8
? manage_workers.isra.24
vmlinux 0xffffffff852b39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
9
kthread
vmlinux 0xffffffff852bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffff852bad60
Build id: 3.10.0-862.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffff8591f637
Build id: 3.10.0-862.el7.x86_64
0x21
12
? insert_kthread_work
vmlinux 0xffffffff852bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [kworker/u49:0:7865]
Modules linked in: tcp_lp fuse binfmt_misc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nf_nat_ftp nf_conntrack_ftp ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr joydev sg nfit i2c_piix4 shpchp vmw_vmci parport_pc libnvdimm
parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm ata_piix libahci crct10dif_pclmul libata crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 7865 Comm: kworker/u49:0 Kdump: loaded Not tainted 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Workqueue: writeback bdi_writeback_workfn (flush-253:2)
task: ffff8e5a6ced9fa0 ti: ffff8e657eb90000 task.ti: ffff8e657eb90000
RIP: 0010:[<ffffffff8556834e>]  [<ffffffff8556834e>] __list_del_entry+0x3e/0xd0
RSP: 0000:ffff8e657eb93c80  EFLAGS: 00010246
RAX: ffff8e71a1748248 RBX: ffff8e657eb93cb0 RCX: dead000000000200
RDX: ffff8e658d778380 RSI: ffff8e657eb93cb0 RDI: ffff8e64f31154c8
RBP: ffff8e657eb93c80 R08: ffff8e64f31154c8 R09: 000000000000ffff
R10: c669588781778300 R11: 0000000000000001 R12: c669588781778300
R13: 0000000000000001 R14: ffff8e657eb93c00 R15: ffff8e657eb93c80
FS:  0000000000000000(0000) GS:ffff8e658db00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fff60e6c000 CR3: 000000160880e000 CR4: 00000000000607e0
Call Trace:
[<ffffffff85449d56>] queue_io+0xe6/0x250
[<ffffffff8544a107>] wb_writeback+0x247/0x2f0
[<ffffffff8543665c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8544aaab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff852b2dff>] process_one_work+0x17f/0x440
[<ffffffff852b3ac6>] worker_thread+0x126/0x3c0
[<ffffffff852b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff852bae31>] kthread+0xd1/0xe0
[<ffffffff852bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff8591f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff852bad60>] ? insert_kthread_work+0x40/0x40
Code: 89 e5 48 39 ca 74 29 48 b9 00 02 00 00 00 00 ad de 48 39 c8 74 7a 4c 8b 00 4c 39 c7 75 53 4c 8b 42 08 4c 39 c7 75 2b 48 89 42 08 <48> 89 10 5d c3 49 89 d0 48 89 f9 48 c7 c2 58 7a c9 85 be 35 00 
Complete report #61307
Frame # Function Binary Source or offset Line
1
__list_del_entry
vmlinux 0xffffffffb63a5d1e
Build id: 3.10.0-1160.11.1.el7.x86_64
0x3e
2
queue_io
vmlinux 0xffffffffb627e84c
Build id: 3.10.0-1160.11.1.el7.x86_64
0xbc
3
wb_writeback
vmlinux 0xffffffffb627ec17
Build id: 3.10.0-1160.11.1.el7.x86_64
0x247
4
? bdi_dirty_limit
vmlinux 0xffffffffb61ca720
Build id: 3.10.0-1160.11.1.el7.x86_64
0x40
5
bdi_writeback_workfn
vmlinux 0xffffffffb627f72c
Build id: 3.10.0-1160.11.1.el7.x86_64
0x1cc
6
process_one_work
vmlinux 0xffffffffb60bde9f
Build id: 3.10.0-1160.11.1.el7.x86_64
0x17f
7
worker_thread
vmlinux 0xffffffffb60befb6
Build id: 3.10.0-1160.11.1.el7.x86_64
0x126
8
? manage_workers.isra.26
vmlinux 0xffffffffb60bee90
Build id: 3.10.0-1160.11.1.el7.x86_64
0x2a0
9
kthread
vmlinux 0xffffffffb60c5e71
Build id: 3.10.0-1160.11.1.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffffb60c5da0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffffb6793df7
Build id: 3.10.0-1160.11.1.el7.x86_64
0x21
12
? insert_kthread_work
vmlinux 0xffffffffb60c5da0
Build id: 3.10.0-1160.11.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 24s! [kworker/u40:1:27435]
Modules linked in: xt_multiport rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache nf_nat_ftp nf_conntrack_ftp ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp ppdev iosf_mbi vmw_balloon crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper joydev pcspkr cryptd sg parport_pc vmw_vmci parport i2c_piix4 ip_tables
xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix scsi_transport_spi crct10dif_pclmul crct10dif_common libata drm mptscsih serio_raw crc32c_intel mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 12 PID: 27435 Comm: kworker/u40:1 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/30/2013
Workqueue: writeback bdi_writeback_workfn (flush-0:40)
task: ffff9d8627a6e300 ti: ffff9d8637dbc000 task.ti: ffff9d8637dbc000
RIP: 0010:[<ffffffffb63a5d1e>]  [<ffffffffb63a5d1e>] __list_del_entry+0x3e/0xd0
RSP: 0018:ffff9d8637dbfc80  EFLAGS: 00010246
RAX: ffff9d7b52b82fb8 RBX: ffff9d8637dbfca8 RCX: dead000000000200
RDX: ffff9d7c2bbfb238 RSI: ffff9d8637dbfcb0 RDI: ffff9d8313a002b0
RBP: ffff9d8637dbfc80 R08: ffff9d8313a002b0 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffffc06a3480 R12: ffffffffb627db24
R13: ffff9d8637dbfc00 R14: ffff9d7c42bd8800 R15: ffff9d8313a7cb78
FS:  0000000000000000(0000) GS:ffff9d875ab00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f87433d8000 CR3: 000000014e3a8000 CR4: 00000000000407e0
Call Trace:
[<ffffffffb627e84c>] queue_io+0xbc/0x240
[<ffffffffb627ec17>] wb_writeback+0x247/0x2f0
[<ffffffffb61ca720>] ? bdi_dirty_limit+0x40/0xe0
[<ffffffffb627f72c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffffb60bde9f>] process_one_work+0x17f/0x440
[<ffffffffb60befb6>] worker_thread+0x126/0x3c0
[<ffffffffb60bee90>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb60c5e71>] kthread+0xd1/0xe0
[<ffffffffb60c5da0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb6793df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb60c5da0>] ? insert_kthread_work+0x40/0x40
Code: 89 e5 48 39 ca 74 29 48 b9 00 02 00 00 00 00 ad de 48 39 c8 74 7a 4c 8b 00 4c 39 c7 75 53 4c 8b 42 08 4c 39 c7 75 2b 48 89 42 08 <48> 89 10 5d c3 49 89 d0 48 89 f9 48 c7 c2 58 19 ab b6 be 35 00 
Complete report #10356
Frame # Function Binary Source or offset Line
1
__list_del_entry
vmlinux 0xffffffff883682ee
Build id: 3.10.0-862.2.3.el7.x86_64
0x3e
2
queue_io
vmlinux 0xffffffff88249cf6
Build id: 3.10.0-862.2.3.el7.x86_64
0xe6
3
wb_writeback
vmlinux 0xffffffff8824a0a7
Build id: 3.10.0-862.2.3.el7.x86_64
0x247
4
? get_nr_inodes
vmlinux 0xffffffff882365fc
Build id: 3.10.0-862.2.3.el7.x86_64
0x4c
5
bdi_writeback_workfn
vmlinux 0xffffffff8824aa4b
Build id: 3.10.0-862.2.3.el7.x86_64
0x2cb
6
process_one_work
vmlinux 0xffffffff880b2dff
Build id: 3.10.0-862.2.3.el7.x86_64
0x17f
7
worker_thread
vmlinux 0xffffffff880b3ac6
Build id: 3.10.0-862.2.3.el7.x86_64
0x126
8
? manage_workers.isra.24
vmlinux 0xffffffff880b39a0
Build id: 3.10.0-862.2.3.el7.x86_64
0x2a0
9
kthread
vmlinux 0xffffffff880bae31
Build id: 3.10.0-862.2.3.el7.x86_64
0xd1
10
? insert_kthread_work
vmlinux 0xffffffff880bad60
Build id: 3.10.0-862.2.3.el7.x86_64
0x40
11
ret_from_fork_nospec_begin
vmlinux 0xffffffff8871f5f7
Build id: 3.10.0-862.2.3.el7.x86_64
0x21
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [kworker/u8:1:22218]
Modules linked in: binfmt_misc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev joydev pcspkr nfit vmw_balloon sg libnvdimm i2c_piix4 parport_pc shpchp parport vmw_vmci ip_tables xfs libcrc32c
sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix mptspi drm libata scsi_transport_spi crct10dif_pclmul crct10dif_common e1000 mptscsih crc32c_intel mptbase serio_raw i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 22218 Comm: kworker/u8:1 Kdump: loaded Tainted: G             L ------------   3.10.0-862.2.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff93f589d81fa0 ti: ffff93f555f3c000 task.ti: ffff93f555f3c000
RIP: 0010:[<ffffffff883682ee>]  [<ffffffff883682ee>] __list_del_entry+0x3e/0xd0
RSP: 0000:ffff93f555f3fc80  EFLAGS: 00010246
RAX: ffff93f5f69a09c8 RBX: ffff93f555f3fcb0 RCX: dead000000000200
RDX: ffff93f6f3088380 RSI: ffff93f555f3fcb0 RDI: ffff93f59a50cd88
RBP: ffff93f555f3fc80 R08: ffff93f59a50cd88 R09: 000000000000ffff
R10: b023c7d4e7088300 R11: 0000000000000001 R12: b023c7d4e7088300
R13: 0000000000000001 R14: ffff93f589d82008 R15: ffff93f555f3fc80
FS:  0000000000000000(0000) GS:ffff93f6ffc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f063ced9050 CR3: 00000000bb6ac000 CR4: 00000000000607e0
Call Trace:
[<ffffffff88249cf6>] queue_io+0xe6/0x250
[<ffffffff8824a0a7>] wb_writeback+0x247/0x2f0
[<ffffffff882365fc>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8824aa4b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff880b2dff>] process_one_work+0x17f/0x440
[<ffffffff880b3ac6>] worker_thread+0x126/0x3c0
[<ffffffff880b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff880bae31>] kthread+0xd1/0xe0
[<ffffffff880bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff8871f5f7>] ret_from_fork_nospec_begin+0x21/0x21
Complete report #4110
Frame # Function Binary Source or offset Line
1
__list_del_entry
vmlinux 0xffffffff9f59519c
Build id: 3.10.0-957.12.2.el7.x86_64
0x2c
2
queue_io
vmlinux 0xffffffff9f4712d6
Build id: 3.10.0-957.12.2.el7.x86_64
0xe6
3
wb_writeback
vmlinux 0xffffffff9f471687
Build id: 3.10.0-957.12.2.el7.x86_64
0x247
4
bdi_writeback_workfn
vmlinux 0xffffffff9f471e75
Build id: 3.10.0-957.12.2.el7.x86_64
0x115
5
process_one_work
vmlinux 0xffffffff9f2b9e3f
Build id: 3.10.0-957.12.2.el7.x86_64
0x17f
6
worker_thread
vmlinux 0xffffffff9f2baed6
Build id: 3.10.0-957.12.2.el7.x86_64
0x126
7
? manage_workers.isra.25
vmlinux 0xffffffff9f2badb0
Build id: 3.10.0-957.12.2.el7.x86_64
0x2a0
8
kthread
vmlinux 0xffffffff9f2c1d21
Build id: 3.10.0-957.12.2.el7.x86_64
0xd1
9
? insert_kthread_work
vmlinux 0xffffffff9f2c1c50
Build id: 3.10.0-957.12.2.el7.x86_64
0x40
10
ret_from_fork_nospec_begin
vmlinux 0xffffffff9f975c37
Build id: 3.10.0-957.12.2.el7.x86_64
0x21
11
? insert_kthread_work
vmlinux 0xffffffff9f2c1c50
Build id: 3.10.0-957.12.2.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [kworker/u66:2:13366]
Modules linked in: mmfs26(OE) mmfslinux(OE) tracedev(OE) ext4 mbcache jbd2 squashfs loop overlay(T) binfmt_misc nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache 8021q garp mrp stp llc ip6table_filter ip6table_raw ip6_tables xt_comment xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt irqbypass crc32_pclmul iTCO_vendor_support ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ipmi_si joydev pcspkr cryptd hpilo hpwdt i2c_i801 sg lpc_ich ipmi_devintf ipmi_msghandler pcc_cpufreq ioatdma wmi acpi_power_meter
ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ahci drm_kms_helper libahci syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm crc32c_intel ixgbe libata serio_raw hpsa drm_panel_orientation_quirks igb scsi_transport_sas mdio ptp pps_core dca i2c_algo_bit [last unloaded: tracedev]
CPU: 14 PID: 13366 Comm: kworker/u66:2 Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-957.12.2.el7.x86_64 #1
Hardware name: HP ProLiant DL160 Gen9/ProLiant DL160 Gen9, BIOS U20 07/21/2019
Workqueue: writeback bdi_writeback_workfn (flush-8:16)
task: ffff95941ab05140 ti: ffff959ed796c000 task.ti: ffff959ed796c000
RIP: 0010:[<ffffffff9f59519c>]  [<ffffffff9f59519c>] __list_del_entry+0x2c/0xd0
RSP: 0018:ffff959ed796fc80  EFLAGS: 00000206
RAX: ffff958ef0e7af48 RBX: 00000000e141e101 RCX: dead000000000200
RDX: ffff958938fc8380 RSI: ffff959ed796fcb0 RDI: ffff95a24dde5c48
RBP: ffff959ed796fc80 R08: ffff95a24dde5c48 R09: 000000010040002d
R10: 000000003c2cbf01 R11: ffffddc37ff0b2c0 R12: 00000000239ced01
R13: ffff959ed796fc88 R14: ffff956abfc03b00 R15: ffff959ed796fc80
FS:  0000000000000000(0000) GS:ffff95a93f380000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c0003b1000 CR3: 0000001145010000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9f4712d6>] queue_io+0xe6/0x250
[<ffffffff9f471687>] wb_writeback+0x247/0x2f0
[<ffffffff9f471e75>] bdi_writeback_workfn+0x115/0x460
[<ffffffff9f2b9e3f>] process_one_work+0x17f/0x440
[<ffffffff9f2baed6>] worker_thread+0x126/0x3c0
[<ffffffff9f2badb0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff9f2c1d21>] kthread+0xd1/0xe0
[<ffffffff9f2c1c50>] ? insert_kthread_work+0x40/0x40
[<ffffffff9f975c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9f2c1c50>] ? insert_kthread_work+0x40/0x40
Code: 48 8b 17 48 b9 00 01 00 00 00 00 ad de 48 8b 47 08 48 89 e5 48 39 ca 74 29 48 b9 00 02 00 00 00 00 ad de 48 39 c8 74 7a 4c 8b 00 <4c> 39 c7 75 53 4c 8b 42 08 4c 39 c7 75 2b 48 89 42 08 48 89 10