Info

Permalink
Function
_raw_spin_unlock_irqrestore
First occurrence
2023-06-08
Last occurrence
2026-03-12
State
NEW
Components
kernel
Quality
-31

Statistics

Operating system Count
CentOS 7 1,288
Architecture Count
x86_64 1,288
Related packages Count
kernel 1,288
 0:3.10.0-957.el7 1,134
 0:3.10.0-1160.108.1.el7 133
 0:3.10.0-1160.59.1.el7 16
 0:3.10.0-693.21.1.el7 3
 0:3.10.0-862.el7 2

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1361294
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff8a915375
Build id: 3.10.0-862.el7.x86_64
0x15
2
__wake_up
vmlinux 0xffffffff8a2c7a14
Build id: 3.10.0-862.el7.x86_64
0x44
3
__wake_up_bit
vmlinux 0xffffffff8a2bbc60
Build id: 3.10.0-862.el7.x86_64
0x50
4
end_page_writeback
vmlinux 0xffffffff8a393a71
Build id: 3.10.0-862.el7.x86_64
0x41
5
end_buffer_async_write
vmlinux 0xffffffff8a4528d1
Build id: 3.10.0-862.el7.x86_64
0xb1
6
xfs_destroy_ioend
xfs 0xffffffffc05887aa
Build id: 3.10.0-862.el7.x86_64
0xca
7
xfs_end_io
xfs 0xffffffffc0588897
Build id: 3.10.0-862.el7.x86_64
0x57
8
process_one_work
vmlinux 0xffffffff8a2b2dff
Build id: 3.10.0-862.el7.x86_64
0x17f
9
worker_thread
vmlinux 0xffffffff8a2b3ac6
Build id: 3.10.0-862.el7.x86_64
0x126
10
? manage_workers.isra.24
vmlinux 0xffffffff8a2b39a0
Build id: 3.10.0-862.el7.x86_64
0x2a0
11
kthread
vmlinux 0xffffffff8a2bae31
Build id: 3.10.0-862.el7.x86_64
0xd1
12
? insert_kthread_work
vmlinux 0xffffffff8a2bad60
Build id: 3.10.0-862.el7.x86_64
0x40
13
ret_from_fork_nospec_begin
vmlinux 0xffffffff8a91f637
Build id: 3.10.0-862.el7.x86_64
0x21
14
? insert_kthread_work
vmlinux 0xffffffff8a2bad60
Build id: 3.10.0-862.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#64 stuck for 22s! [kworker/64:0:56924]
Modules linked in: veth yrfs(OE) cmac arc4 md4 nls_utf8 cifs xt_addrtype ccm br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter fuse rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache rdma_ucm(OE) ib_ucm(OE) rdma_cm(OE) iw_cm(OE) ib_ipoib(OE) ib_cm(OE) ib_uverbs(OE) ib_umad(OE) mlx5_fpga_tools(OE) mlx5_ib(OE) mlx5_core(OE) mlxfw(OE) mlx4_en(OE) mlx4_ib(OE) ib_core(OE) mlx4_core(OE) mlx_compat(OE) devlink overlay(T) sunrpc vfat fat sb_edac coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul
iTCO_wdt iTCO_vendor_support glue_helper ablk_helper mxm_wmi cryptd ses enclosure joydev sg pcspkr i2c_i801 lpc_ich mei_me mei shpchp ipmi_si ipmi_devintf wmi ipmi_msghandler acpi_power_meter knem(OE) ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ixgbe drm igb ahci mpt3sas crct10dif_pclmul crct10dif_common libahci crc32c_intel libata megaraid_sas i2c_algo_bit mdio ptp i2c_core raid_class pps_core scsi_transport_sas dca dm_mirror dm_region_hash dm_log dm_mod
CPU: 64 PID: 56924 Comm: kworker/64:0 Kdump: loaded Tainted: G           OE  ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name:   4U Rack Server/60G24-US, BIOS 227 05/07/2018
Workqueue: xfs-data/sdc xfs_end_io [xfs]
task: ffff989663460000 ti: ffff987cf66d8000 task.ti: ffff987cf66d8000
RIP: 0010:[<ffffffff8a915375>]  [<ffffffff8a915375>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff987cf66dbcf0  EFLAGS: 00000216
RAX: ffff9894fff92d48 RBX: ffff987cf66dbc98 RCX: dead000000000200
RDX: ffff9894fff92d30 RSI: 0000000000000216 RDI: 0000000000000216
RBP: ffff987cf66dbcf0 R08: ffff9895204b7d60 R09: ffff98955b0c1fa0
R10: 0000000000000000 R11: 0000000000000400 R12: ffff987cf66dbc70
R13: 0000000000000003 R14: 0000000000000000 R15: ffff9894fff92d30
FS:  0000000000000000(0000) GS:ffff98ac9f300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b07b64d0000 CR3: 00000018b9352000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8a2c7a14>] __wake_up+0x44/0x50
[<ffffffff8a2bbc60>] __wake_up_bit+0x50/0x70
[<ffffffff8a393a71>] end_page_writeback+0x41/0x60
[<ffffffff8a4528d1>] end_buffer_async_write+0xb1/0x120
[<ffffffffc05887aa>] xfs_destroy_ioend+0xca/0x160 [xfs]
[<ffffffffc0588897>] xfs_end_io+0x57/0x80 [xfs]
[<ffffffff8a2b2dff>] process_one_work+0x17f/0x440
[<ffffffff8a2b3ac6>] worker_thread+0x126/0x3c0
[<ffffffff8a2b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff8a2bae31>] kthread+0xd1/0xe0
[<ffffffff8a2bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffff8a91f637>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8a2bad60>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #827667
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff82d6a0f5
Build id: 3.10.0-957.el7.x86_64
0x15
2
__wake_up
vmlinux 0xffffffff826cec64
Build id: 3.10.0-957.el7.x86_64
0x44
3
__wake_up_bit
vmlinux 0xffffffff826c2b10
Build id: 3.10.0-957.el7.x86_64
0x50
4
end_page_writeback
vmlinux 0xffffffff827b5fd1
Build id: 3.10.0-957.el7.x86_64
0x41
5
end_buffer_async_write
vmlinux 0xffffffff82879831
Build id: 3.10.0-957.el7.x86_64
0xb1
6
xfs_destroy_ioend
xfs 0xffffffffc0738648
Build id: 3.10.0-957.el7.x86_64
0xc8
7
xfs_end_io
xfs 0xffffffffc0738745
Build id: 3.10.0-957.el7.x86_64
0x55
8
process_one_work
vmlinux 0xffffffff826b9d4f
Build id: 3.10.0-957.el7.x86_64
0x17f
9
worker_thread
vmlinux 0xffffffff826bade6
Build id: 3.10.0-957.el7.x86_64
0x126
10
? manage_workers.isra.25
vmlinux 0xffffffff826bacc0
Build id: 3.10.0-957.el7.x86_64
0x2a0
11
kthread
vmlinux 0xffffffff826c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
12
? insert_kthread_work
vmlinux 0xffffffff826c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
13
ret_from_fork_nospec_begin
vmlinux 0xffffffff82d74c1d
Build id: 3.10.0-957.el7.x86_64
0x7
14
? insert_kthread_work
vmlinux 0xffffffff826c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#26 stuck for 22s! [kworker/26:2:158195]
Modules linked in: cdc_ether joydev usbnet mii binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bonding sunrpc vfat fat iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure pcspkr sg i2c_i801 ipmi_ssif mei_me mei wmi ipmi_si ipmi_devintf ipmi_msghandler pinctrl_lewisburg pinctrl_intel acpi_power_meter acpi_pad ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm crc32c_intel i40e igb ahci smartpqi libahci drm_panel_orientation_quirks scsi_transport_sas dca ptp i2c_algo_bit pps_core libata dm_mirror dm_region_hash dm_log dm_mod
CPU: 26 PID: 158195 Comm: kworker/26:2 Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: Lenovo ThinkServer SR660 V2/SR660 V2, BIOS XWE103I-1.70 03/22/2022
Workqueue: xfs-data/dm-2 xfs_end_io [xfs]
task: ffff8e219793c100 ti: ffff8e015c454000 task.ti: ffff8e015c454000
RIP: 0010:[<ffffffff82d6a0f5>]  [<ffffffff82d6a0f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8e015c457ce8  EFLAGS: 00000293
RAX: ffff8e417ffbe008 RBX: ffffffff826d67c2 RCX: dead000000000200
RDX: ffff8e417ffbdff0 RSI: 0000000000000293 RDI: 0000000000000293
RBP: ffff8e015c457ce8 R08: ffff8e29dee53d60 R09: ffff8e21d33ca080
R10: 0000000000000000 R11: 0000000000000400 R12: 0000000000000003
R13: 0000000000000000 R14: ffff8e417ffbdff0 R15: ffff8e417ffbe008
FS:  0000000000000000(0000) GS:ffff8e40ff080000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7968f78000 CR3: 0000000f7b010000 CR4: 0000000000760fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffff826cec64>] __wake_up+0x44/0x50
[<ffffffff826c2b10>] __wake_up_bit+0x50/0x70
[<ffffffff827b5fd1>] end_page_writeback+0x41/0x60
[<ffffffff82879831>] end_buffer_async_write+0xb1/0x120
[<ffffffffc0738648>] xfs_destroy_ioend+0xc8/0x170 [xfs]
[<ffffffffc0738745>] xfs_end_io+0x55/0x90 [xfs]
[<ffffffff826b9d4f>] process_one_work+0x17f/0x440
[<ffffffff826bade6>] worker_thread+0x126/0x3c0
[<ffffffff826bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff826c1c31>] kthread+0xd1/0xe0
[<ffffffff826c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff82d74c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff826c1b60>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #792148
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffbad6a0f5
Build id: 3.10.0-957.el7.x86_64
0x15
2
__wake_up
vmlinux 0xffffffffba6cec64
Build id: 3.10.0-957.el7.x86_64
0x44
3
__wake_up_bit
vmlinux 0xffffffffba6c2b10
Build id: 3.10.0-957.el7.x86_64
0x50
4
end_page_writeback
vmlinux 0xffffffffba7b5fd1
Build id: 3.10.0-957.el7.x86_64
0x41
5
end_buffer_async_write
vmlinux 0xffffffffba879831
Build id: 3.10.0-957.el7.x86_64
0xb1
6
xfs_destroy_ioend
xfs 0xffffffffc0909648
Build id: 3.10.0-957.el7.x86_64
0xc8
7
xfs_end_io
xfs 0xffffffffc0909745
Build id: 3.10.0-957.el7.x86_64
0x55
8
process_one_work
vmlinux 0xffffffffba6b9d4f
Build id: 3.10.0-957.el7.x86_64
0x17f
9
worker_thread
vmlinux 0xffffffffba6bade6
Build id: 3.10.0-957.el7.x86_64
0x126
10
? manage_workers.isra.25
vmlinux 0xffffffffba6bacc0
Build id: 3.10.0-957.el7.x86_64
0x2a0
11
kthread
vmlinux 0xffffffffba6c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
12
? insert_kthread_work
vmlinux 0xffffffffba6c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
13
ret_from_fork_nospec_begin
vmlinux 0xffffffffbad74c1d
Build id: 3.10.0-957.el7.x86_64
0x7
14
? insert_kthread_work
vmlinux 0xffffffffba6c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#34 stuck for 21s! [kworker/34:1:5814]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter mpt3sas mpt2sas raid_class mptctl mptbase rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache nls_utf8 isofs loop sunrpc dm_mirror dm_region_hash dm_log dm_mod ses sd_mod enclosure scsi_transport_sas crc_t10dif crct10dif_generic sg iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm vfat irqbypass fat crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_hdmi snd_hda_codec_realtek
snd_hda_codec_generic pcspkr megaraid_sas snd_hda_intel joydev snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm ipmi_ssif snd_timer mei_me snd mei lpc_ich soundcore i2c_i801 ipmi_si ipmi_devintf ipmi_msghandler acpi_cpufreq acpi_power_meter acpi_pad sch_fq_codel ip_tables xfs libcrc32c nouveau ast video crct10dif_pclmul mxm_wmi crct10dif_common i2c_algo_bit crc32c_intel drm_kms_helper i40e syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ptp drm pps_core nvme ahci nvme_core libahci libata drm_panel_orientation_quirks wmi nfit libnvdimm
CPU: 34 PID: 5814 Comm: kworker/34:1 Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: Supermicro X11DAi-N/X11DAi-N, BIOS 3.1 05/03/2019
Workqueue: xfs-data/sda1 xfs_end_io [xfs]
task: ffff96571ec76180 ti: ffff962923a54000 task.ti: ffff962923a54000
RIP: 0010:[<ffffffffbad6a0f5>]  [<ffffffffbad6a0f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff962923a57ce8  EFLAGS: 00000293
RAX: ffff9668fffc05b8 RBX: ffffffffba6d67c2 RCX: dead000000000200
RDX: ffff9668fffc05a0 RSI: 0000000000000293 RDI: 0000000000000293
RBP: ffff962923a57ce8 R08: ffff96686fb6fd60 R09: ffff96495288d140
R10: 0000000000000000 R11: 0000000000000400 R12: 0000000000000003
R13: 0000000000000000 R14: ffff9668fffc05a0 R15: ffff9668fffc05b8
FS:  0000000000000000(0000) GS:ffff96687db80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c000a1b000 CR3: 000000017be10000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffffba6cec64>] __wake_up+0x44/0x50
[<ffffffffba6c2b10>] __wake_up_bit+0x50/0x70
[<ffffffffba7b5fd1>] end_page_writeback+0x41/0x60
[<ffffffffba879831>] end_buffer_async_write+0xb1/0x120
[<ffffffffc0909648>] xfs_destroy_ioend+0xc8/0x170 [xfs]
[<ffffffffc0909745>] xfs_end_io+0x55/0x90 [xfs]
[<ffffffffba6b9d4f>] process_one_work+0x17f/0x440
[<ffffffffba6bade6>] worker_thread+0x126/0x3c0
[<ffffffffba6bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffba6c1c31>] kthread+0xd1/0xe0
[<ffffffffba6c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffbad74c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffba6c1b60>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #733118
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff82d6a0f5
Build id: 3.10.0-957.el7.x86_64
0x15
2
__wake_up
vmlinux 0xffffffff826cec64
Build id: 3.10.0-957.el7.x86_64
0x44
3
__wake_up_bit
vmlinux 0xffffffff826c2b10
Build id: 3.10.0-957.el7.x86_64
0x50
4
end_page_writeback
vmlinux 0xffffffff827b5fd1
Build id: 3.10.0-957.el7.x86_64
0x41
5
end_buffer_async_write
vmlinux 0xffffffff82879831
Build id: 3.10.0-957.el7.x86_64
0xb1
6
xfs_destroy_ioend
xfs 0xffffffffc0738648
Build id: 3.10.0-957.el7.x86_64
0xc8
7
xfs_end_io
xfs 0xffffffffc0738745
Build id: 3.10.0-957.el7.x86_64
0x55
8
process_one_work
vmlinux 0xffffffff826b9d4f
Build id: 3.10.0-957.el7.x86_64
0x17f
9
worker_thread
vmlinux 0xffffffff826bade6
Build id: 3.10.0-957.el7.x86_64
0x126
10
? manage_workers.isra.25
vmlinux 0xffffffff826bacc0
Build id: 3.10.0-957.el7.x86_64
0x2a0
11
kthread
vmlinux 0xffffffff826c1c31
Build id: 3.10.0-957.el7.x86_64
0xd1
12
? finish_task_switch
vmlinux 0xffffffff826d0887
Build id: 3.10.0-957.el7.x86_64
0x57
13
? insert_kthread_work
vmlinux 0xffffffff826c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
14
ret_from_fork_nospec_begin
vmlinux 0xffffffff82d74c1d
Build id: 3.10.0-957.el7.x86_64
0x7
15
? insert_kthread_work
vmlinux 0xffffffff826c1b60
Build id: 3.10.0-957.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#18 stuck for 22s! [kworker/18:0:70263]
Modules linked in: cdc_ether joydev usbnet mii binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bonding sunrpc vfat fat iTCO_wdt iTCO_vendor_support intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ses enclosure pcspkr sg i2c_i801 ipmi_ssif mei_me mei wmi ipmi_si ipmi_devintf ipmi_msghandler pinctrl_lewisburg pinctrl_intel acpi_power_meter acpi_pad ip_tables xfs libcrc32c raid1 sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm crc32c_intel i40e igb ahci smartpqi libahci drm_panel_orientation_quirks scsi_transport_sas dca ptp i2c_algo_bit pps_core libata dm_mirror dm_region_hash dm_log dm_mod
CPU: 18 PID: 70263 Comm: kworker/18:0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: Lenovo ThinkServer SR660 V2/SR660 V2, BIOS XWE103I-1.70 03/22/2022
Workqueue: xfs-data/dm-5 xfs_end_io [xfs]
task: ffff8e40f07e6180 ti: ffff8e015c41c000 task.ti: ffff8e015c41c000
RIP: 0010:[<ffffffff82d6a0f5>]  [<ffffffff82d6a0f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8e015c41fce8  EFLAGS: 00000293
RAX: ffff8e217ff83008 RBX: ffffffff826d67c2 RCX: dead000000000200
RDX: ffff8e217ff82ff0 RSI: 0000000000000293 RDI: 0000000000000293
RBP: ffff8e015c41fce8 R08: ffff8e065a0cfd60 R09: ffffffff836e9c00
R10: 0000000000000000 R11: 0000000000000400 R12: 0000000000000003
R13: 0000000000000000 R14: ffff8e217ff82ff0 R15: ffff8e217ff83008
FS:  0000000000000000(0000) GS:ffff8e40fee80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000021f8fc7000 CR3: 0000000f7b010000 CR4: 0000000000760fe0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffff826cec64>] __wake_up+0x44/0x50
[<ffffffff826c2b10>] __wake_up_bit+0x50/0x70
[<ffffffff827b5fd1>] end_page_writeback+0x41/0x60
[<ffffffff82879831>] end_buffer_async_write+0xb1/0x120
[<ffffffffc0738648>] xfs_destroy_ioend+0xc8/0x170 [xfs]
[<ffffffffc0738745>] xfs_end_io+0x55/0x90 [xfs]
[<ffffffff826b9d4f>] process_one_work+0x17f/0x440
[<ffffffff826bade6>] worker_thread+0x126/0x3c0
[<ffffffff826bacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff826c1c31>] kthread+0xd1/0xe0
[<ffffffff826d0887>] ? finish_task_switch+0x57/0x1c0
[<ffffffff826c1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff82d74c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff826c1b60>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #730031
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff816b6545
Build id: 3.10.0-693.21.1.el7.x86_64
0x15
2
__wake_up
vmlinux 0xffffffff810c0bd4
Build id: 3.10.0-693.21.1.el7.x86_64
0x44
3
__wake_up_bit
vmlinux 0xffffffff810b4f40
Build id: 3.10.0-693.21.1.el7.x86_64
0x50
4
end_page_writeback
vmlinux 0xffffffff81186b21
Build id: 3.10.0-693.21.1.el7.x86_64
0x41
5
end_buffer_async_write
vmlinux 0xffffffff8123bf81
Build id: 3.10.0-693.21.1.el7.x86_64
0xb1
6
xfs_destroy_ioend
xfs 0xffffffffc0292bfa
Build id: 3.10.0-693.21.1.el7.x86_64
0xca
7
xfs_end_io
xfs 0xffffffffc0292ce7
Build id: 3.10.0-693.21.1.el7.x86_64
0x57
8
process_one_work
vmlinux 0xffffffff810abe2f
Build id: 3.10.0-693.21.1.el7.x86_64
0x17f
9
worker_thread
vmlinux 0xffffffff810acaf6
Build id: 3.10.0-693.21.1.el7.x86_64
0x126
10
? manage_workers.isra.24
vmlinux 0xffffffff810ac9d0
Build id: 3.10.0-693.21.1.el7.x86_64
0x2a0
11
kthread
vmlinux 0xffffffff810b4031
Build id: 3.10.0-693.21.1.el7.x86_64
0xd1
12
? insert_kthread_work
vmlinux 0xffffffff810b3f60
Build id: 3.10.0-693.21.1.el7.x86_64
0x40
13
ret_from_fork
vmlinux 0xffffffff816c0577
Build id: 3.10.0-693.21.1.el7.x86_64
0x77
14
? insert_kthread_work
vmlinux 0xffffffff810b3f60
Build id: 3.10.0-693.21.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#16 stuck for 22s! [kworker/16:0:37430]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables devlink binfmt_misc tun ip_set nfnetlink bridge stp llc intel_powerclamp coretemp kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd gpio_ich iTCO_wdt iTCO_vendor_support dcdbas joydev ipmi_si ipmi_devintf pcspkr ipmi_msghandler sg wmi i7core_edac acpi_power_meter
shpchp lpc_ich edac_core nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul ata_piix crct10dif_common libata crc32c_intel serio_raw megaraid_sas i2c_core bnx2 dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 16 PID: 37430 Comm: kworker/16:0 Not tainted 3.10.0-693.21.1.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R810/0TT6JF, BIOS 2.9.0 07/29/2013
Workqueue: xfs-data/dm-0 xfs_end_io [xfs]
task: ffff8800596d1fa0 ti: ffff8810865e8000 task.ti: ffff8810865e8000
RIP: 0010:[<ffffffff816b6545>]  [<ffffffff816b6545>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8810865ebcf0  EFLAGS: 00000283
RAX: ffff88307ffb83f0 RBX: ffff8810865ebc98 RCX: dead000000000200
RDX: ffff88307ffb83d8 RSI: 0000000000000283 RDI: 0000000000000283
RBP: ffff8810865ebcf0 R08: ffff883087e57b10 R09: ffff880169466eb0
R10: 0000000000000000 R11: 0000000000000400 R12: ffff8810865ebc70
R13: 0000000000000003 R14: 0000000000000000 R15: ffff88307ffb83d8
FS:  0000000000000000(0000) GS:ffff88103f900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fdaed85b22b CR3: 0000000001a02000 CR4: 00000000000207e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff810c0bd4>] __wake_up+0x44/0x50
[<ffffffff810b4f40>] __wake_up_bit+0x50/0x70
[<ffffffff81186b21>] end_page_writeback+0x41/0x60
[<ffffffff8123bf81>] end_buffer_async_write+0xb1/0x120
[<ffffffffc0292bfa>] xfs_destroy_ioend+0xca/0x160 [xfs]
[<ffffffffc0292ce7>] xfs_end_io+0x57/0x80 [xfs]
[<ffffffff810abe2f>] process_one_work+0x17f/0x440
[<ffffffff810acaf6>] worker_thread+0x126/0x3c0
[<ffffffff810ac9d0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b4031>] kthread+0xd1/0xe0
[<ffffffff810b3f60>] ? insert_kthread_work+0x40/0x40
[<ffffffff816c0577>] ret_from_fork+0x77/0xb0
[<ffffffff810b3f60>] ? insert_kthread_work+0x40/0x40
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48 
Complete report #908106
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff891ba385
Build id: 3.10.0-1160.108.1.el7.x86_64
0x15
2
__wake_up_common_lock
vmlinux 0xffffffff88ad8f31
Build id: 3.10.0-1160.108.1.el7.x86_64
0x91
3
? task_rq_unlock
vmlinux 0xffffffff88ad8990
Build id: 3.10.0-1160.108.1.el7.x86_64
0x20
4
__wake_up
vmlinux 0xffffffff88ad8f73
Build id: 3.10.0-1160.108.1.el7.x86_64
0x13
5
__wake_up_bit
vmlinux 0xffffffff88acc590
Build id: 3.10.0-1160.108.1.el7.x86_64
0x50
6
end_page_writeback
vmlinux 0xffffffff88bc8bc1
Build id: 3.10.0-1160.108.1.el7.x86_64
0x41
7
xfs_destroy_ioend
xfs 0xffffffffc08a5dda
Build id: 3.10.0-1160.108.1.el7.x86_64
0x15a
8
xfs_end_io
xfs 0xffffffffc08a5f35
Build id: 3.10.0-1160.108.1.el7.x86_64
0x55
9
process_one_work
vmlinux 0xffffffff88ac32ef
Build id: 3.10.0-1160.108.1.el7.x86_64
0x17f
10
worker_thread
vmlinux 0xffffffff88ac4436
Build id: 3.10.0-1160.108.1.el7.x86_64
0x126
11
? manage_workers.isra.26
vmlinux 0xffffffff88ac4310
Build id: 3.10.0-1160.108.1.el7.x86_64
0x2b0
12
kthread
vmlinux 0xffffffff88acb621
Build id: 3.10.0-1160.108.1.el7.x86_64
0xd1
13
? insert_kthread_work
vmlinux 0xffffffff88acb550
Build id: 3.10.0-1160.108.1.el7.x86_64
0x40
14
ret_from_fork_nospec_begin
vmlinux 0xffffffff891c51f7
Build id: 3.10.0-1160.108.1.el7.x86_64
0x21
15
? insert_kthread_work
vmlinux 0xffffffff88acb550
Build id: 3.10.0-1160.108.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#32 stuck for 22s! [kworker/32:0:62894]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nslink(OE) sb_edac iTCO_wdt intel_powerclamp iTCO_vendor_support coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr osst st ch lpc_ich ioatdma hpilo hpwdt sg wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter pcc_cpufreq nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs sd_mod crc_t10dif crct10dif_generic qla2xxx mgag200 drm_kms_helper syscopyarea bnx2x sysfillrect sysimgblt fb_sys_fops
ttm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw drm nvme_fc nvme_fabrics igb nvme_core scsi_transport_fc mdio hpsa scsi_tgt libcrc32c dca ptp drm_panel_orientation_quirks pps_core scsi_transport_sas i2c_algo_bit dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 32 PID: 62894 Comm: kworker/32:0 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.108.1.el7.x86_64 #1
Hardware name: HP ProLiant DL580 Gen8, BIOS P79 11/26/2014
Workqueue: xfs-data/dm-2 xfs_end_io [xfs]
task: ffff8d93cfba8000 ti: ffff8da2cdce4000 task.ti: ffff8da2cdce4000
RIP: 0010:[<ffffffff891ba385>]  [<ffffffff891ba385>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8da2cdce7cb0  EFLAGS: 00000292
RAX: 0000000000000001 RBX: 00000001cdce7c40 RCX: dead000000000200
RDX: ffff8da43ff994d8 RSI: 0000000000000292 RDI: 0000000000000292
RBP: ffff8da2cdce7cb0 R08: ffff8db3f126bc28 R09: 0000000000000002
R10: 0000000000000000 R11: 0000000000000400 R12: ffff8da2cdce7c48
R13: ffffffff88ae1182 R14: ffff8da2cdce7c20 R15: ffff8da43ff994d8
FS:  0000000000000000(0000) GS:ffff8db3fe480000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8c46044a00 CR3: 00000016fd87c000 CR4: 00000000001607e0
Call Trace:
[<ffffffff88ad8f31>] __wake_up_common_lock+0x91/0xc0
[<ffffffff88ad8990>] ? task_rq_unlock+0x20/0x20
[<ffffffff88ad8f73>] __wake_up+0x13/0x20
[<ffffffff88acc590>] __wake_up_bit+0x50/0x70
[<ffffffff88bc8bc1>] end_page_writeback+0x41/0x60
[<ffffffffc08a5dda>] xfs_destroy_ioend+0x15a/0x260 [xfs]
[<ffffffffc08a5f35>] xfs_end_io+0x55/0x90 [xfs]
[<ffffffff88ac32ef>] process_one_work+0x17f/0x440
[<ffffffff88ac4436>] worker_thread+0x126/0x3c0
[<ffffffff88ac4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff88acb621>] kthread+0xd1/0xe0
[<ffffffff88acb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff891c51f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff88acb550>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 cc cc cc cc 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #900010
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff99f8ea05
Build id: 3.10.0-1160.59.1.el7.x86_64
0x15
2
__wake_up_common_lock
vmlinux 0xffffffff998d31f1
Build id: 3.10.0-1160.59.1.el7.x86_64
0x91
3
? task_rq_unlock
vmlinux 0xffffffff998d2c60
Build id: 3.10.0-1160.59.1.el7.x86_64
0x20
4
__wake_up
vmlinux 0xffffffff998d3233
Build id: 3.10.0-1160.59.1.el7.x86_64
0x13
5
__wake_up_bit
vmlinux 0xffffffff998c6d60
Build id: 3.10.0-1160.59.1.el7.x86_64
0x50
6
end_page_writeback
vmlinux 0xffffffff999bdf01
Build id: 3.10.0-1160.59.1.el7.x86_64
0x41
7
xfs_destroy_ioend
xfs 0xffffffffc078e9aa
Build id: 3.10.0-1160.59.1.el7.x86_64
0x15a
8
xfs_end_io
xfs 0xffffffffc078eb05
Build id: 3.10.0-1160.59.1.el7.x86_64
0x55
9
process_one_work
vmlinux 0xffffffff998bde8f
Build id: 3.10.0-1160.59.1.el7.x86_64
0x17f
10
worker_thread
vmlinux 0xffffffff998befa6
Build id: 3.10.0-1160.59.1.el7.x86_64
0x126
11
? manage_workers.isra.26
vmlinux 0xffffffff998bee80
Build id: 3.10.0-1160.59.1.el7.x86_64
0x2a0
12
kthread
vmlinux 0xffffffff998c5e61
Build id: 3.10.0-1160.59.1.el7.x86_64
0xd1
13
? insert_kthread_work
vmlinux 0xffffffff998c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
14
ret_from_fork_nospec_begin
vmlinux 0xffffffff99f99de4
Build id: 3.10.0-1160.59.1.el7.x86_64
0xe
15
? insert_kthread_work
vmlinux 0xffffffff998c5d90
Build id: 3.10.0-1160.59.1.el7.x86_64
0x40
NMI watchdog: BUG: soft lockup - CPU#8 stuck for 22s! [kworker/8:4:73940]
Modules linked in: ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dell_smbios dell_wmi_descriptor dcdbas vfat fat amd64_edac_mod edac_mce_amd kvm_amd kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd dm_service_time pcspkr joydev sg k10temp i2c_piix4 wmi ipmi_si ipmi_devintf ipmi_msghandler tpm_crb acpi_power_meter dm_multipath dm_mod binfmt_misc nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic mgag200
i2c_algo_bit drm_kms_helper ahci syscopyarea sysfillrect sysimgblt fb_sys_fops ttm libahci mpt3sas crct10dif_pclmul crct10dif_common libata i40e tg3 raid_class drm crc32c_intel ptp scsi_transport_sas drm_panel_orientation_quirks pps_core megaraid_sas fuse
CPU: 8 PID: 73940 Comm: kworker/8:4 Not tainted 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R7525/0590KW, BIOS 2.3.6 07/06/2021
Workqueue: xfs-data/dm-1 xfs_end_io [xfs]
task: ffff8bc12f5aa100 ti: ffff8bc1c2a5c000 task.ti: ffff8bc1c2a5c000
RIP: 0010:[<ffffffff99f8ea05>]  [<ffffffff99f8ea05>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8bc1c2a5fcb0  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 000000017f186240 RCX: dead000000000200
RDX: ffff8bc0ec05fbb8 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffff8bc1c2a5fcb0 R08: ffff8bc0ec05fbb8 R09: 0000000000000002
R10: 0000000000000000 R11: 0000000000000400 R12: ffff8bc1c2a5fc48
R13: ffffffff998dae02 R14: ffff8bc1c2a5fc20 R15: ffff8bc0ec05fba0
FS:  0000000000000000(0000) GS:ffff8bc13ea00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000010de6491a050 CR3: 0000000ad4986000 CR4: 0000000000340fe0
Call Trace:
[<ffffffff998d31f1>] __wake_up_common_lock+0x91/0xc0
[<ffffffff998d2c60>] ? task_rq_unlock+0x20/0x20
[<ffffffff998d3233>] __wake_up+0x13/0x20
[<ffffffff998c6d60>] __wake_up_bit+0x50/0x70
[<ffffffff999bdf01>] end_page_writeback+0x41/0x60
[<ffffffffc078e9aa>] xfs_destroy_ioend+0x15a/0x260 [xfs]
[<ffffffffc078eb05>] xfs_end_io+0x55/0x90 [xfs]
[<ffffffff998bde8f>] process_one_work+0x17f/0x440
[<ffffffff998befa6>] worker_thread+0x126/0x3c0
[<ffffffff998bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff998c5e61>] kthread+0xd1/0xe0
[<ffffffff998c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffff99f99de4>] ret_from_fork_nospec_begin+0xe/0x21
[<ffffffff998c5d90>] ? insert_kthread_work+0x40/0x40
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48