Complete report #271311
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
lg_global_lock
|
vmlinux |
0xffffffffabec3b7f
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x3f
|
|
2
|
queue_stop_cpus_work
|
vmlinux |
0xffffffffabf26edc
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x3c
|
|
3
|
?
dyntick_save_progress_counter
|
vmlinux |
0xffffffffabf48a70
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x30
|
|
4
|
__stop_cpus
|
vmlinux |
0xffffffffabf271a2
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x52
|
|
5
|
?
dyntick_save_progress_counter
|
vmlinux |
0xffffffffabf48a70
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x30
|
|
6
|
try_stop_cpus
|
vmlinux |
0xffffffffabf2757a
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x3a
|
|
7
|
synchronize_sched_expedited
|
vmlinux |
0xffffffffabf4a8a4
Build id: 3.10.0-862.9.1.el7.x86_64
|
0xe4
|
|
8
|
synchronize_rcu_expedited
|
vmlinux |
0xffffffffabf4aa4e
Build id: 3.10.0-862.9.1.el7.x86_64
|
0xe
|
|
9
|
synchronize_net
|
vmlinux |
0xffffffffac3e79ad
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x2d
|
|
10
|
dev_deactivate_many
|
vmlinux |
0xffffffffac418637
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x247
|
|
11
|
dev_deactivate
|
vmlinux |
0xffffffffac418683
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x43
|
|
12
|
linkwatch_do_dev
|
vmlinux |
0xffffffffac4060b8
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x38
|
|
13
|
__linkwatch_run_queue
|
vmlinux |
0xffffffffac4063d9
Build id: 3.10.0-862.9.1.el7.x86_64
|
0xf9
|
|
14
|
linkwatch_event
|
vmlinux |
0xffffffffac406495
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x25
|
|
15
|
process_one_work
|
vmlinux |
0xffffffffabeb35ef
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x17f
|
|
16
|
worker_thread
|
vmlinux |
0xffffffffabeb4686
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x126
|
|
17
|
|
vmlinux |
0xffffffffabeb4560
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x2a0
|
|
18
|
kthread
|
vmlinux |
0xffffffffabebb621
Build id: 3.10.0-862.9.1.el7.x86_64
|
0xd1
|
|
19
|
|
vmlinux |
0xffffffffabebb550
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x40
|
|
20
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffac5205f7
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x21
|
|
21
|
|
vmlinux |
0xffffffffabebb550
Build id: 3.10.0-862.9.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [kworker/2:0:130557]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_ens1371 snd_rawmidi ppdev snd_ac97_codec ac97_bus vmw_balloon snd_seq snd_seq_device snd_pcm snd_timer pcspkr joydev snd soundcore sg vmw_vmci i2c_piix4 shpchp nfit libnvdimm parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod sr_mod cdrom
crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw e1000 mptspi scsi_transport_spi mptscsih mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 130557 Comm: kworker/2:0 Kdump: loaded Tainted: G W L ------------ 3.10.0-862.9.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
Workqueue: events linkwatch_event
task: ffff8b628a7abf40 ti: ffff8b6008f70000 task.ti: ffff8b6008f70000
RIP: 0010:[<ffffffffabec3b7f>] [<ffffffffabec3b7f>] lg_global_lock+0x3f/0x80
RSP: 0018:ffff8b6008f73be8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff8b6175688000 RCX: 000000000000003d
RDX: 000000000000007d RSI: 0000000000000040 RDI: ffff8b66f9f54040
RBP: ffff8b6008f73c08 R08: ffffffffacb420a8 R09: 0000000000000000
R10: ffffffffacb41e20 R11: 0000000000000000 R12: 0000000000000000
R13: ffff8b66f8098b40 R14: ffff8b62f4bf76c0 R15: 0000000000000000
FS: 0000000000000000(0000) GS:ffff8b66f8080000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f266ff52620 CR3: 00000002f500e000 CR4: 00000000001607e0
Call Trace:
[<ffffffffabf26edc>] queue_stop_cpus_work+0x3c/0xa0
[<ffffffffabf48a70>] ? dyntick_save_progress_counter+0x30/0x30
[<ffffffffabf271a2>] __stop_cpus+0x52/0x90
[<ffffffffabf48a70>] ? dyntick_save_progress_counter+0x30/0x30
[<ffffffffabf2757a>] try_stop_cpus+0x3a/0x60
[<ffffffffabf4a8a4>] synchronize_sched_expedited+0xe4/0x1e0
[<ffffffffabf4aa4e>] synchronize_rcu_expedited+0xe/0x10
[<ffffffffac3e79ad>] synchronize_net+0x2d/0x40
[<ffffffffac418637>] dev_deactivate_many+0x247/0x250
[<ffffffffac418683>] dev_deactivate+0x43/0x70
[<ffffffffac4060b8>] linkwatch_do_dev+0x38/0x60
[<ffffffffac4063d9>] __linkwatch_run_queue+0xf9/0x190
[<ffffffffac406495>] linkwatch_event+0x25/0x30
[<ffffffffabeb35ef>] process_one_work+0x17f/0x440
[<ffffffffabeb4686>] worker_thread+0x126/0x3c0
[<ffffffffabeb4560>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffabebb621>] kthread+0xd1/0xe0
[<ffffffffabebb550>] ? insert_kthread_work+0x40/0x40
[<ffffffffac5205f7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffabebb550>] ? insert_kthread_work+0x40/0x40
Code: ff ff ff ff 4c 8b 2d b1 f2 74 00 41 bc 01 00 00 00 eb 21 66 0f 1f 84 00 00 00 00 00 48 98 49 8b 3e 48 03 3c c5 a0 53 b3 ac 31 c0 <f0> 44 0f b1 27 85 c0 75 28 48 63 35 15 f9 c7 00 83 c3 01 4c 89
Complete report #246791
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
lg_global_lock
|
vmlinux |
0xffffffff996c36bf
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x3f
|
|
2
|
queue_stop_cpus_work
|
vmlinux |
0xffffffff99726a1c
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x3c
|
|
3
|
?
dyntick_save_progress_counter
|
vmlinux |
0xffffffff997485f0
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x30
|
|
4
|
__stop_cpus
|
vmlinux |
0xffffffff99726ce2
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x52
|
|
5
|
?
dyntick_save_progress_counter
|
vmlinux |
0xffffffff997485f0
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x30
|
|
6
|
try_stop_cpus
|
vmlinux |
0xffffffff997270ba
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x3a
|
|
7
|
synchronize_sched_expedited
|
vmlinux |
0xffffffff9974a424
Build id: 3.10.0-862.3.2.el7.x86_64
|
0xe4
|
|
8
|
synchronize_rcu_expedited
|
vmlinux |
0xffffffff9974a5ce
Build id: 3.10.0-862.3.2.el7.x86_64
|
0xe
|
|
9
|
synchronize_net
|
vmlinux |
0xffffffff99be78ed
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x2d
|
|
10
|
dev_deactivate_many
|
vmlinux |
0xffffffff99c185a7
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x247
|
|
11
|
dev_deactivate
|
vmlinux |
0xffffffff99c185f3
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x43
|
|
12
|
linkwatch_do_dev
|
vmlinux |
0xffffffff99c05ff8
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x38
|
|
13
|
__linkwatch_run_queue
|
vmlinux |
0xffffffff99c06319
Build id: 3.10.0-862.3.2.el7.x86_64
|
0xf9
|
|
14
|
linkwatch_event
|
vmlinux |
0xffffffff99c063d5
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x25
|
|
15
|
process_one_work
|
vmlinux |
0xffffffff996b312f
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x17f
|
|
16
|
worker_thread
|
vmlinux |
0xffffffff996b3df6
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x126
|
|
17
|
|
vmlinux |
0xffffffff996b3cd0
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x2a0
|
|
18
|
kthread
|
vmlinux |
0xffffffff996bb161
Build id: 3.10.0-862.3.2.el7.x86_64
|
0xd1
|
|
19
|
|
vmlinux |
0xffffffff996bb090
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x40
|
|
20
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff99d2065d
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x7
|
|
21
|
|
vmlinux |
0xffffffff996bb090
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [kworker/0:2:54445]
Modules linked in: xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmhgfs(OE) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev snd_ens1371 aesni_intel snd_rawmidi snd_ac97_codec lrw gf128mul glue_helper ablk_helper cryptd ac97_bus snd_seq vmw_balloon snd_seq_device snd_pcm joydev pcspkr snd_timer snd nfit sg soundcore libnvdimm vmw_vmci i2c_piix4 shpchp parport_pc parport ip_tables
xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix e1000 mptspi libata serio_raw scsi_transport_spi mptscsih mptbase i2c_core
CPU: 0 PID: 54445 Comm: kworker/0:2 Kdump: loaded Tainted: G OEL ------------ T 3.10.0-862.3.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
Workqueue: events linkwatch_event
task: ffff9302825c5ee0 ti: ffff9302ef4a8000 task.ti: ffff9302ef4a8000
RIP: 0010:[<ffffffff996c36bf>] [<ffffffff996c36bf>] lg_global_lock+0x3f/0x80
RSP: 0000:ffff9302ef4abbe8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffffffff9a6bfc00 RCX: 000000000000002f
RDX: 000000000000002f RSI: 0000000000000080 RDI: ffff9302fc1d4000
RBP: ffff9302ef4abc08 R08: ffffffff9a341fe0 R09: 0000000000000000
R10: ffffffff9a341d60 R11: 0000000000000000 R12: ffff930286158780
R13: ffff930286158780 R14: 00000000ffffffff R15: ffffffffc0497150
FS: 0000000000000000(0000) GS:ffff9302fb600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f30d64d8d10 CR3: 0000000045e5c000 CR4: 00000000003607f0
Call Trace:
[<ffffffff99726a1c>] queue_stop_cpus_work+0x3c/0xa0
[<ffffffff997485f0>] ? dyntick_save_progress_counter+0x30/0x30
[<ffffffff99726ce2>] __stop_cpus+0x52/0x90
[<ffffffff997485f0>] ? dyntick_save_progress_counter+0x30/0x30
[<ffffffff997270ba>] try_stop_cpus+0x3a/0x60
[<ffffffff9974a424>] synchronize_sched_expedited+0xe4/0x1e0
[<ffffffff9974a5ce>] synchronize_rcu_expedited+0xe/0x10
[<ffffffff99be78ed>] synchronize_net+0x2d/0x40
[<ffffffff99c185a7>] dev_deactivate_many+0x247/0x250
[<ffffffff99c185f3>] dev_deactivate+0x43/0x70
[<ffffffff99c05ff8>] linkwatch_do_dev+0x38/0x60
[<ffffffff99c06319>] __linkwatch_run_queue+0xf9/0x190
[<ffffffff99c063d5>] linkwatch_event+0x25/0x30
[<ffffffff996b312f>] process_one_work+0x17f/0x440
[<ffffffff996b3df6>] worker_thread+0x126/0x3c0
[<ffffffff996b3cd0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff996bb161>] kthread+0xd1/0xe0
[<ffffffff996bb090>] ? insert_kthread_work+0x40/0x40
[<ffffffff99d2065d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff996bb090>] ? insert_kthread_work+0x40/0x40
Code: ff ff ff ff 4c 8b 2d 71 f6 74 00 41 bc 01 00 00 00 eb 21 66 0f 1f 84 00 00 00 00 00 48 98 49 8b 3e 48 03 3c c5 e0 52 33 9a 31 c0 <3e> 44 0f b1 27 85 c0 75 28 48 63 35 15 fd c7 00 83 c3 01 4c 89
Complete report #20826
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_unlock_irqrestore
|
vmlinux |
0xffffffff9516a0f5
Build id: 3.10.0-957.el7.x86_64
|
0x15
|
|
2
|
cpu_stop_queue_work
|
vmlinux |
0xffffffff94b2fc8f
Build id: 3.10.0-957.el7.x86_64
|
0x4f
|
|
3
|
queue_stop_cpus_work
|
vmlinux |
0xffffffff94b2fe70
Build id: 3.10.0-957.el7.x86_64
|
0x60
|
|
4
|
?
dyntick_save_progress_counter
|
vmlinux |
0xffffffff94b52100
Build id: 3.10.0-957.el7.x86_64
|
0x30
|
|
5
|
__stop_cpus
|
vmlinux |
0xffffffff94b30112
Build id: 3.10.0-957.el7.x86_64
|
0x52
|
|
6
|
?
dyntick_save_progress_counter
|
vmlinux |
0xffffffff94b52100
Build id: 3.10.0-957.el7.x86_64
|
0x30
|
|
7
|
try_stop_cpus
|
vmlinux |
0xffffffff94b304ea
Build id: 3.10.0-957.el7.x86_64
|
0x3a
|
|
8
|
synchronize_sched_expedited
|
vmlinux |
0xffffffff94b53f34
Build id: 3.10.0-957.el7.x86_64
|
0xe4
|
|
9
|
synchronize_rcu_expedited
|
vmlinux |
0xffffffff94b540de
Build id: 3.10.0-957.el7.x86_64
|
0xe
|
|
10
|
synchronize_net
|
vmlinux |
0xffffffff950340fd
Build id: 3.10.0-957.el7.x86_64
|
0x2d
|
|
11
|
dev_deactivate_many
|
vmlinux |
0xffffffff95066a37
Build id: 3.10.0-957.el7.x86_64
|
0x247
|
|
12
|
dev_deactivate
|
vmlinux |
0xffffffff95066a83
Build id: 3.10.0-957.el7.x86_64
|
0x43
|
|
13
|
linkwatch_do_dev
|
vmlinux |
0xffffffff95053c68
Build id: 3.10.0-957.el7.x86_64
|
0x38
|
|
14
|
__linkwatch_run_queue
|
vmlinux |
0xffffffff95053f89
Build id: 3.10.0-957.el7.x86_64
|
0xf9
|
|
15
|
linkwatch_event
|
vmlinux |
0xffffffff95054045
Build id: 3.10.0-957.el7.x86_64
|
0x25
|
|
16
|
process_one_work
|
vmlinux |
0xffffffff94ab9d4f
Build id: 3.10.0-957.el7.x86_64
|
0x17f
|
|
17
|
worker_thread
|
vmlinux |
0xffffffff94abade6
Build id: 3.10.0-957.el7.x86_64
|
0x126
|
|
18
|
|
vmlinux |
0xffffffff94abacc0
Build id: 3.10.0-957.el7.x86_64
|
0x2a0
|
|
19
|
kthread
|
vmlinux |
0xffffffff94ac1c31
Build id: 3.10.0-957.el7.x86_64
|
0xd1
|
|
20
|
|
vmlinux |
0xffffffff94ac1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
|
21
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff95174c37
Build id: 3.10.0-957.el7.x86_64
|
0x21
|
|
22
|
|
vmlinux |
0xffffffff94ac1b60
Build id: 3.10.0-957.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [kworker/0:1:10337]
Modules linked in: tun devlink ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock sunrpc ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel cryptd vmw_balloon snd_seq_midi snd_seq_midi_event pcspkr joydev snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore sg parport_pc parport i2c_piix4 vmw_vmci xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi libata scsi_transport_spi nfit e1000 mptscsih mptbase libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 0 PID: 10337 Comm: kworker/0:1 Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
Workqueue: events linkwatch_event
task: ffff9935439dc100 ti: ffff99354055c000 task.ti: ffff99354055c000
RIP: 0010:[<ffffffff9516a0f5>] [<ffffffff9516a0f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff99354055fbd8 EFLAGS: 00000282
RAX: 0000000000000001 RBX: 0000000000000000 RCX: ffff99354055c000
RDX: 0000000000000000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff99354055fbd8 R08: ffffffff9521dd80 R09: ffff9935439dc100
R10: ffffffff957583e0 R11: 0000000000000005 R12: ffffffff94ad300f
R13: ffff99354055fb60 R14: 0000000000000000 R15: ffff99357c61ab80
FS: 0000000000000000(0000) GS:ffff99357c600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f13c9e3c019 CR3: 0000000033a94000 CR4: 00000000000607f0
Call Trace:
[<ffffffff94b2fc8f>] cpu_stop_queue_work+0x4f/0x70
[<ffffffff94b2fe70>] queue_stop_cpus_work+0x60/0xa0
[<ffffffff94b52100>] ? dyntick_save_progress_counter+0x30/0x30
[<ffffffff94b30112>] __stop_cpus+0x52/0x90
[<ffffffff94b52100>] ? dyntick_save_progress_counter+0x30/0x30
[<ffffffff94b304ea>] try_stop_cpus+0x3a/0x60
[<ffffffff94b53f34>] synchronize_sched_expedited+0xe4/0x1e0
[<ffffffff94b540de>] synchronize_rcu_expedited+0xe/0x10
[<ffffffff950340fd>] synchronize_net+0x2d/0x40
[<ffffffff95066a37>] dev_deactivate_many+0x247/0x250
[<ffffffff95066a83>] dev_deactivate+0x43/0x70
[<ffffffff95053c68>] linkwatch_do_dev+0x38/0x60
[<ffffffff95053f89>] __linkwatch_run_queue+0xf9/0x190
[<ffffffff95054045>] linkwatch_event+0x25/0x30
[<ffffffff94ab9d4f>] process_one_work+0x17f/0x440
[<ffffffff94abade6>] worker_thread+0x126/0x3c0
[<ffffffff94abacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffff94ac1c31>] kthread+0xd1/0xe0
[<ffffffff94ac1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffff95174c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff94ac1b60>] ? insert_kthread_work+0x40/0x40
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48
Complete report #2048
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
lg_global_lock
|
vmlinux |
0xffffffffac8c338f
Build id: 3.10.0-862.el7.x86_64
|
0x3f
|
|
2
|
queue_stop_cpus_work
|
vmlinux |
0xffffffffac9266ec
Build id: 3.10.0-862.el7.x86_64
|
0x3c
|
|
3
|
?
dyntick_save_progress_counter
|
vmlinux |
0xffffffffac948250
Build id: 3.10.0-862.el7.x86_64
|
0x30
|
|
4
|
__stop_cpus
|
vmlinux |
0xffffffffac9269b2
Build id: 3.10.0-862.el7.x86_64
|
0x52
|
|
5
|
?
dyntick_save_progress_counter
|
vmlinux |
0xffffffffac948250
Build id: 3.10.0-862.el7.x86_64
|
0x30
|
|
6
|
try_stop_cpus
|
vmlinux |
0xffffffffac926d8a
Build id: 3.10.0-862.el7.x86_64
|
0x3a
|
|
7
|
synchronize_sched_expedited
|
vmlinux |
0xffffffffac94a084
Build id: 3.10.0-862.el7.x86_64
|
0xe4
|
|
8
|
synchronize_rcu_expedited
|
vmlinux |
0xffffffffac94a22e
Build id: 3.10.0-862.el7.x86_64
|
0xe
|
|
9
|
synchronize_net
|
vmlinux |
0xffffffffacde68ed
Build id: 3.10.0-862.el7.x86_64
|
0x2d
|
|
10
|
dev_deactivate_many
|
vmlinux |
0xfffffffface175a7
Build id: 3.10.0-862.el7.x86_64
|
0x247
|
|
11
|
dev_deactivate
|
vmlinux |
0xfffffffface175f3
Build id: 3.10.0-862.el7.x86_64
|
0x43
|
|
12
|
linkwatch_do_dev
|
vmlinux |
0xfffffffface04ff8
Build id: 3.10.0-862.el7.x86_64
|
0x38
|
|
13
|
__linkwatch_run_queue
|
vmlinux |
0xfffffffface05319
Build id: 3.10.0-862.el7.x86_64
|
0xf9
|
|
14
|
linkwatch_event
|
vmlinux |
0xfffffffface053d5
Build id: 3.10.0-862.el7.x86_64
|
0x25
|
|
15
|
process_one_work
|
vmlinux |
0xffffffffac8b2dff
Build id: 3.10.0-862.el7.x86_64
|
0x17f
|
|
16
|
worker_thread
|
vmlinux |
0xffffffffac8b3ac6
Build id: 3.10.0-862.el7.x86_64
|
0x126
|
|
17
|
|
vmlinux |
0xffffffffac8b39a0
Build id: 3.10.0-862.el7.x86_64
|
0x2a0
|
|
18
|
kthread
|
vmlinux |
0xffffffffac8bae31
Build id: 3.10.0-862.el7.x86_64
|
0xd1
|
|
19
|
|
vmlinux |
0xffffffffac8bad60
Build id: 3.10.0-862.el7.x86_64
|
0x40
|
|
20
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffacf1f61d
Build id: 3.10.0-862.el7.x86_64
|
0x7
|
|
21
|
|
vmlinux |
0xffffffffac8bad60
Build id: 3.10.0-862.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 28s! [kworker/0:1:12784]
mptscsih e1000 mptbase ata_piix libata i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 12784 Comm: kworker/0:1 Kdump: loaded Not tainted 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
Workqueue: events linkwatch_event
task: ffff98a0b85a9fa0 ti: ffff98a134084000 task.ti: ffff98a134084000
RIP: 0010:[<ffffffffac8c338f>] [<ffffffffac8c338f>] lg_global_lock+0x3f/0x80
RSP: 0018:ffff98a134087be8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffffffffac8cffff RCX: 000000000000000b
RDX: 000000000000004b RSI: 0000000000000040 RDI: ffff98a13a8d4000
RBP: ffff98a134087c08 R08: ffffffffad5420a8 R09: 0000000000000000
R10: ffffffffad541e20 R11: 0000000000000000 R12: ffff98a134087bc0
R13: ffffffffad7bebed R14: ffffffffad267a62 R15: ffff98a134087bd0
FS: 0000000000000000(0000) GS:ffff98a139600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa0220d5000 CR3: 000000006900e000 CR4: 00000000003607f0
Call Trace:
[<ffffffffac9266ec>] queue_stop_cpus_work+0x3c/0xa0
[<ffffffffac948250>] ? dyntick_save_progress_counter+0x30/0x30
[<ffffffffac9269b2>] __stop_cpus+0x52/0x90
[<ffffffffac948250>] ? dyntick_save_progress_counter+0x30/0x30
[<ffffffffac926d8a>] try_stop_cpus+0x3a/0x60
[<ffffffffac94a084>] synchronize_sched_expedited+0xe4/0x1e0
[<ffffffffac94a22e>] synchronize_rcu_expedited+0xe/0x10
[<ffffffffacde68ed>] synchronize_net+0x2d/0x40
[<fffffffface175a7>] dev_deactivate_many+0x247/0x250
[<fffffffface175f3>] dev_deactivate+0x43/0x70
[<fffffffface04ff8>] linkwatch_do_dev+0x38/0x60
[<fffffffface05319>] __linkwatch_run_queue+0xf9/0x190
[<fffffffface053d5>] linkwatch_event+0x25/0x30
[<ffffffffac8b2dff>] process_one_work+0x17f/0x440
[<ffffffffac8b3ac6>] worker_thread+0x126/0x3c0
[<ffffffffac8b39a0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffffac8bae31>] kthread+0xd1/0xe0
[<ffffffffac8bad60>] ? insert_kthread_work+0x40/0x40
[<ffffffffacf1f61d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffac8bad60>] ? insert_kthread_work+0x40/0x40
Code: ff ff ff ff 4c 8b 2d e1 f8 74 00 41 bc 01 00 00 00 eb 21 66 0f 1f 84 00 00 00 00 00 48 98 49 8b 3e 48 03 3c c5 a0 53 53 ad 31 c0 <f0> 44 0f b1 27 85 c0 75 28 48 63 35 05 01 c8 00 83 c3 01 4c 89
Complete report #806807
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_unlock_irqrestore
|
vmlinux |
0xffffffffa5dba385
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x15
|
|
2
|
cpu_stop_queue_work
|
vmlinux |
0xffffffffa573d57d
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x7d
|
|
3
|
?
get_state_synchronize_rcu
|
vmlinux |
0xffffffffa57606b0
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x20
|
|
4
|
queue_stop_cpus_work
|
vmlinux |
0xffffffffa573d7a0
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x60
|
|
5
|
?
get_state_synchronize_rcu
|
vmlinux |
0xffffffffa57606b0
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x20
|
|
6
|
__stop_cpus
|
vmlinux |
0xffffffffa573da62
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x52
|
|
7
|
?
get_state_synchronize_rcu
|
vmlinux |
0xffffffffa57606b0
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x20
|
|
8
|
try_stop_cpus
|
vmlinux |
0xffffffffa573de8a
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x3a
|
|
9
|
synchronize_sched_expedited
|
vmlinux |
0xffffffffa5762617
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0xe7
|
|
10
|
synchronize_rcu_expedited
|
vmlinux |
0xffffffffa576280e
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0xe
|
|
11
|
synchronize_net
|
vmlinux |
0xffffffffa5c798dd
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x2d
|
|
12
|
dev_deactivate_many
|
vmlinux |
0xffffffffa5caead7
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x247
|
|
13
|
dev_deactivate
|
vmlinux |
0xffffffffa5caeb23
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x43
|
|
14
|
linkwatch_do_dev
|
vmlinux |
0xffffffffa5c9a9a0
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x40
|
|
15
|
__linkwatch_run_queue
|
vmlinux |
0xffffffffa5c9acc9
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0xf9
|
|
16
|
linkwatch_event
|
vmlinux |
0xffffffffa5c9ad85
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x25
|
|
17
|
process_one_work
|
vmlinux |
0xffffffffa56c31df
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x17f
|
|
18
|
worker_thread
|
vmlinux |
0xffffffffa56c4326
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x126
|
|
19
|
|
vmlinux |
0xffffffffa56c4200
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x2b0
|
|
20
|
kthread
|
vmlinux |
0xffffffffa56cb511
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0xd1
|
|
21
|
|
vmlinux |
0xffffffffa56cb440
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x40
|
|
22
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffa5dc51dd
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x7
|
|
23
|
|
vmlinux |
0xffffffffa56cb440
Build id: 3.10.0-1160.83.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [kworker/0:3:116493]
Modules linked in: tcp_lp rfcomm xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter bnep vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper snd_ens1371 ablk_helper ppdev cryptd snd_rawmidi snd_ac97_codec btusb ac97_bus btrtl btbcm
snd_seq vmw_balloon btintel bluetooth snd_seq_device snd_pcm pcspkr snd_timer joydev rfkill snd soundcore sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper ata_piix syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel libata nfit drm libnvdimm serio_raw mptspi scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 0 PID: 116493 Comm: kworker/0:3 Tainted: G L ------------ 3.10.0-1160.83.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
Workqueue: events linkwatch_event
task: ffff9e160e8e6300 ti: ffff9e160dcc0000 task.ti: ffff9e160dcc0000
RIP: 0010:[<ffffffffa5dba385>] [<ffffffffa5dba385>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9e160dcc3bb0 EFLAGS: 00000246
RAX: 0000000000000246 RBX: ffff9e163c61acc0 RCX: ffff9e163b44f318
RDX: 0000000000000001 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff9e160dcc3bb0 R08: ffff9e163c6160b0 R09: 0000000000000000
R10: ffffffffa635e3a0 R11: fffffb9ec08c5300 R12: 000000000dcc3b70
R13: ffffffffa56e9f0a R14: ffff9e160dcc3b58 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff9e163c600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6e954fc000 CR3: 0000000033b8c000 CR4: 00000000001607f0
Call Trace:
[<ffffffffa573d57d>] cpu_stop_queue_work+0x7d/0xe0
[<ffffffffa57606b0>] ? get_state_synchronize_rcu+0x20/0x20
[<ffffffffa573d7a0>] queue_stop_cpus_work+0x60/0xb0
[<ffffffffa57606b0>] ? get_state_synchronize_rcu+0x20/0x20
[<ffffffffa573da62>] __stop_cpus+0x52/0x90
[<ffffffffa57606b0>] ? get_state_synchronize_rcu+0x20/0x20
[<ffffffffa573de8a>] try_stop_cpus+0x3a/0x60
[<ffffffffa5762617>] synchronize_sched_expedited+0xe7/0x1f0
[<ffffffffa576280e>] synchronize_rcu_expedited+0xe/0x20
[<ffffffffa5c798dd>] synchronize_net+0x2d/0x40
[<ffffffffa5caead7>] dev_deactivate_many+0x247/0x250
[<ffffffffa5caeb23>] dev_deactivate+0x43/0x70
[<ffffffffa5c9a9a0>] linkwatch_do_dev+0x40/0x70
[<ffffffffa5c9acc9>] __linkwatch_run_queue+0xf9/0x190
[<ffffffffa5c9ad85>] linkwatch_event+0x25/0x30
[<ffffffffa56c31df>] process_one_work+0x17f/0x440
[<ffffffffa56c4326>] worker_thread+0x126/0x3c0
[<ffffffffa56c4200>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffffa56cb511>] kthread+0xd1/0xe0
[<ffffffffa56cb440>] ? insert_kthread_work+0x40/0x40
[<ffffffffa5dc51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffa56cb440>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 cc cc cc cc 0f 1f 44 00 00 55 48 89 e5 48
Complete report #718541
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
lg_global_lock
|
vmlinux |
0xffffffffb4ace41f
Build id: 3.10.0-1160.el7.x86_64
|
0x3f
|
|
2
|
queue_stop_cpus_work
|
vmlinux |
0xffffffffb4b357ac
Build id: 3.10.0-1160.el7.x86_64
|
0x3c
|
|
3
|
?
get_state_synchronize_rcu
|
vmlinux |
0xffffffffb4b57cc0
Build id: 3.10.0-1160.el7.x86_64
|
0x20
|
|
4
|
__stop_cpus
|
vmlinux |
0xffffffffb4b35a72
Build id: 3.10.0-1160.el7.x86_64
|
0x52
|
|
5
|
?
get_state_synchronize_rcu
|
vmlinux |
0xffffffffb4b57cc0
Build id: 3.10.0-1160.el7.x86_64
|
0x20
|
|
6
|
try_stop_cpus
|
vmlinux |
0xffffffffb4b35e8a
Build id: 3.10.0-1160.el7.x86_64
|
0x3a
|
|
7
|
synchronize_sched_expedited
|
vmlinux |
0xffffffffb4b59b07
Build id: 3.10.0-1160.el7.x86_64
|
0xe7
|
|
8
|
synchronize_rcu_expedited
|
vmlinux |
0xffffffffb4b59cce
Build id: 3.10.0-1160.el7.x86_64
|
0xe
|
|
9
|
synchronize_net
|
vmlinux |
0xffffffffb504f08d
Build id: 3.10.0-1160.el7.x86_64
|
0x2d
|
|
10
|
dev_deactivate_many
|
vmlinux |
0xffffffffb5083187
Build id: 3.10.0-1160.el7.x86_64
|
0x247
|
|
11
|
dev_deactivate
|
vmlinux |
0xffffffffb50831d3
Build id: 3.10.0-1160.el7.x86_64
|
0x43
|
|
12
|
linkwatch_do_dev
|
vmlinux |
0xffffffffb506f910
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
|
13
|
__linkwatch_run_queue
|
vmlinux |
0xffffffffb506fc29
Build id: 3.10.0-1160.el7.x86_64
|
0xf9
|
|
14
|
linkwatch_event
|
vmlinux |
0xffffffffb506fce5
Build id: 3.10.0-1160.el7.x86_64
|
0x25
|
|
15
|
process_one_work
|
vmlinux |
0xffffffffb4abdc4f
Build id: 3.10.0-1160.el7.x86_64
|
0x17f
|
|
16
|
worker_thread
|
vmlinux |
0xffffffffb4abed66
Build id: 3.10.0-1160.el7.x86_64
|
0x126
|
|
17
|
|
vmlinux |
0xffffffffb4abec40
Build id: 3.10.0-1160.el7.x86_64
|
0x2a0
|
|
18
|
kthread
|
vmlinux |
0xffffffffb4ac5c21
Build id: 3.10.0-1160.el7.x86_64
|
0xd1
|
|
19
|
|
vmlinux |
0xffffffffb4ac5b50
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
|
20
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffffb5193df7
Build id: 3.10.0-1160.el7.x86_64
|
0x21
|
|
21
|
|
vmlinux |
0xffffffffb4ac5b50
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [kworker/2:0:52394]
Modules linked in: xt_statistic veth vxlan ip6_udp_tunnel udp_tunnel xt_nat nf_tables ip_set ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_mangle ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment xt_mark nf_conntrack_netlink nfnetlink xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink overlay(T) vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel snd_timer snd
lrw gf128mul soundcore glue_helper ablk_helper cryptd pcspkr joydev vmw_balloon sg parport_pc parport i2c_piix4 vmw_vmci br_netfilter bridge stp llc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata serio_raw mptspi scsi_transport_spi mptscsih nfit libnvdimm mptbase e1000 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 52394 Comm: kworker/2:0 Tainted: G W L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: events linkwatch_event
task: ffff94dcbe22d280 ti: ffff94dd70790000 task.ti: ffff94dd70790000
RIP: 0010:[<ffffffffb4ace41f>] [<ffffffffb4ace41f>] lg_global_lock+0x3f/0x80
RSP: 0018:ffff94dd70793be8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 00000000a7b8a602 RCX: 0000000000000023
RDX: 0000000000000023 RSI: 0000000000000080 RDI: ffff94dd79ed6080
RBP: ffff94dd70793c08 R08: ffffffffb575e100 R09: 0000000000000000
R10: ffffffffb575de80 R11: 0000000000000001 R12: ffff94dd70793c00
R13: ffff94dcb07dc600 R14: 0000000000000000 R15: 0000000000000001
FS: 0000000000000000(0000) GS:ffff94dd79680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000004de8650 CR3: 000000009e6c8000 CR4: 00000000003607e0
Call Trace:
[<ffffffffb4b357ac>] queue_stop_cpus_work+0x3c/0xa0
[<ffffffffb4b57cc0>] ? get_state_synchronize_rcu+0x20/0x20
[<ffffffffb4b35a72>] __stop_cpus+0x52/0x90
[<ffffffffb4b57cc0>] ? get_state_synchronize_rcu+0x20/0x20
[<ffffffffb4b35e8a>] try_stop_cpus+0x3a/0x60
[<ffffffffb4b59b07>] synchronize_sched_expedited+0xe7/0x1e0
[<ffffffffb4b59cce>] synchronize_rcu_expedited+0xe/0x10
[<ffffffffb504f08d>] synchronize_net+0x2d/0x40
[<ffffffffb5083187>] dev_deactivate_many+0x247/0x250
[<ffffffffb50831d3>] dev_deactivate+0x43/0x70
[<ffffffffb506f910>] linkwatch_do_dev+0x40/0x60
[<ffffffffb506fc29>] __linkwatch_run_queue+0xf9/0x190
[<ffffffffb506fce5>] linkwatch_event+0x25/0x30
[<ffffffffb4abdc4f>] process_one_work+0x17f/0x440
[<ffffffffb4abed66>] worker_thread+0x126/0x3c0
[<ffffffffb4abec40>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb4ac5c21>] kthread+0xd1/0xe0
[<ffffffffb4ac5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffb5193df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb4ac5b50>] ? insert_kthread_work+0x40/0x40
Complete report #345121
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_unlock_irqrestore
|
vmlinux |
0xffffffff93589785
Build id: 3.10.0-1160.el7.x86_64
|
0x15
|
|
2
|
try_to_wake_up
|
vmlinux |
0xffffffff92eda830
Build id: 3.10.0-1160.el7.x86_64
|
0x190
|
|
3
|
wake_up_q
|
vmlinux |
0xffffffff92edaaab
Build id: 3.10.0-1160.el7.x86_64
|
0x5b
|
|
4
|
cpu_stop_queue_work
|
vmlinux |
0xffffffff92f355b5
Build id: 3.10.0-1160.el7.x86_64
|
0x85
|
|
5
|
?
get_state_synchronize_rcu
|
vmlinux |
0xffffffff92f57cc0
Build id: 3.10.0-1160.el7.x86_64
|
0x20
|
|
6
|
queue_stop_cpus_work
|
vmlinux |
0xffffffff92f357d0
Build id: 3.10.0-1160.el7.x86_64
|
0x60
|
|
7
|
?
get_state_synchronize_rcu
|
vmlinux |
0xffffffff92f57cc0
Build id: 3.10.0-1160.el7.x86_64
|
0x20
|
|
8
|
__stop_cpus
|
vmlinux |
0xffffffff92f35a72
Build id: 3.10.0-1160.el7.x86_64
|
0x52
|
|
9
|
?
get_state_synchronize_rcu
|
vmlinux |
0xffffffff92f57cc0
Build id: 3.10.0-1160.el7.x86_64
|
0x20
|
|
10
|
try_stop_cpus
|
vmlinux |
0xffffffff92f35e8a
Build id: 3.10.0-1160.el7.x86_64
|
0x3a
|
|
11
|
synchronize_sched_expedited
|
vmlinux |
0xffffffff92f59b07
Build id: 3.10.0-1160.el7.x86_64
|
0xe7
|
|
12
|
synchronize_rcu_expedited
|
vmlinux |
0xffffffff92f59cce
Build id: 3.10.0-1160.el7.x86_64
|
0xe
|
|
13
|
synchronize_net
|
vmlinux |
0xffffffff9344f08d
Build id: 3.10.0-1160.el7.x86_64
|
0x2d
|
|
14
|
dev_deactivate_many
|
vmlinux |
0xffffffff93483187
Build id: 3.10.0-1160.el7.x86_64
|
0x247
|
|
15
|
dev_deactivate
|
vmlinux |
0xffffffff934831d3
Build id: 3.10.0-1160.el7.x86_64
|
0x43
|
|
16
|
linkwatch_do_dev
|
vmlinux |
0xffffffff9346f910
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
|
17
|
__linkwatch_run_queue
|
vmlinux |
0xffffffff9346fc29
Build id: 3.10.0-1160.el7.x86_64
|
0xf9
|
|
18
|
linkwatch_event
|
vmlinux |
0xffffffff9346fce5
Build id: 3.10.0-1160.el7.x86_64
|
0x25
|
|
19
|
process_one_work
|
vmlinux |
0xffffffff92ebdc4f
Build id: 3.10.0-1160.el7.x86_64
|
0x17f
|
|
20
|
worker_thread
|
vmlinux |
0xffffffff92ebed66
Build id: 3.10.0-1160.el7.x86_64
|
0x126
|
|
21
|
|
vmlinux |
0xffffffff92ebec40
Build id: 3.10.0-1160.el7.x86_64
|
0x2a0
|
|
22
|
kthread
|
vmlinux |
0xffffffff92ec5c21
Build id: 3.10.0-1160.el7.x86_64
|
0xd1
|
|
23
|
|
vmlinux |
0xffffffff92ec5b50
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
|
24
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff93593df7
Build id: 3.10.0-1160.el7.x86_64
|
0x21
|
|
25
|
|
vmlinux |
0xffffffff92ec5b50
Build id: 3.10.0-1160.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [kworker/2:2:123047]
Modules linked in: rfcomm ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache iosf_mbi jbd2 crc32_pclmul ppdev ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec btusb ac97_bus btrtl vmw_balloon snd_seq btbcm aesni_intel btintel bluetooth lrw gf128mul snd_seq_device glue_helper ablk_helper cryptd snd_pcm joydev pcspkr
snd_timer rfkill snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit serio_raw libnvdimm mptspi ata_piix scsi_transport_spi e1000 mptscsih libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 123047 Comm: kworker/2:2 Kdump: loaded Tainted: G L ------------ 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
Workqueue: events linkwatch_event
task: ffff8a9b2efea100 ti: ffff8a9b0af30000 task.ti: ffff8a9b0af30000
RIP: 0010:[<ffffffff93589785>] [<ffffffff93589785>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8a9b0af33b38 EFLAGS: 00000296
RAX: 0000000000000000 RBX: ffff8a9b0af33ad0 RCX: 0000000000000004
RDX: 0000000000000000 RSI: 0000000000000296 RDI: 0000000000000296
RBP: ffff8a9b0af33b38 R08: ffffffff93613f08 R09: ffff8a9b22fc8000
R10: ffffffff93b5de80 R11: 6e776f4420736920 R12: ffff8a9b0af33ab0
R13: ffffffff92e63cd3 R14: ffff8a9b0af33aa0 R15: 0000000000000003
FS: 0000000000000000(0000) GS:ffff8a9ded680000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2b58e23000 CR3: 00000003b9c9e000 CR4: 00000000003607e0
Call Trace:
[<ffffffff92eda830>] try_to_wake_up+0x190/0x390
[<ffffffff92edaaab>] wake_up_q+0x5b/0x80
[<ffffffff92f355b5>] cpu_stop_queue_work+0x85/0xe0
[<ffffffff92f57cc0>] ? get_state_synchronize_rcu+0x20/0x20
[<ffffffff92f357d0>] queue_stop_cpus_work+0x60/0xa0
[<ffffffff92f57cc0>] ? get_state_synchronize_rcu+0x20/0x20
[<ffffffff92f35a72>] __stop_cpus+0x52/0x90
[<ffffffff92f57cc0>] ? get_state_synchronize_rcu+0x20/0x20
[<ffffffff92f35e8a>] try_stop_cpus+0x3a/0x60
[<ffffffff92f59b07>] synchronize_sched_expedited+0xe7/0x1e0
[<ffffffff92f59cce>] synchronize_rcu_expedited+0xe/0x10
[<ffffffff9344f08d>] synchronize_net+0x2d/0x40
[<ffffffff93483187>] dev_deactivate_many+0x247/0x250
[<ffffffff934831d3>] dev_deactivate+0x43/0x70
[<ffffffff9346f910>] linkwatch_do_dev+0x40/0x60
[<ffffffff9346fc29>] __linkwatch_run_queue+0xf9/0x190
[<ffffffff9346fce5>] linkwatch_event+0x25/0x30
[<ffffffff92ebdc4f>] process_one_work+0x17f/0x440
[<ffffffff92ebed66>] worker_thread+0x126/0x3c0
[<ffffffff92ebec40>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff92ec5c21>] kthread+0xd1/0xe0
[<ffffffff92ec5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff93593df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff92ec5b50>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #8937
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
lg_global_lock
|
vmlinux |
0xffffffff998cee8f
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x3f
|
|
2
|
queue_stop_cpus_work
|
vmlinux |
0xffffffff9993613c
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x3c
|
|
3
|
?
get_state_synchronize_rcu
|
vmlinux |
0xffffffff999585d0
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x20
|
|
4
|
__stop_cpus
|
vmlinux |
0xffffffff99936402
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x52
|
|
5
|
?
get_state_synchronize_rcu
|
vmlinux |
0xffffffff999585d0
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x20
|
|
6
|
try_stop_cpus
|
vmlinux |
0xffffffff9993681a
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x3a
|
|
7
|
synchronize_sched_expedited
|
vmlinux |
0xffffffff9995a417
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xe7
|
|
8
|
synchronize_rcu_expedited
|
vmlinux |
0xffffffff9995a5de
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xe
|
|
9
|
synchronize_net
|
vmlinux |
0xffffffff99e4e08d
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x2d
|
|
10
|
dev_deactivate_many
|
vmlinux |
0xffffffff99e82067
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x247
|
|
11
|
dev_deactivate
|
vmlinux |
0xffffffff99e820b3
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x43
|
|
12
|
linkwatch_do_dev
|
vmlinux |
0xffffffff99e6e7f8
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x38
|
|
13
|
__linkwatch_run_queue
|
vmlinux |
0xffffffff99e6eb19
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xf9
|
|
14
|
linkwatch_event
|
vmlinux |
0xffffffff99e6ebd5
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x25
|
|
15
|
process_one_work
|
vmlinux |
0xffffffff998be6bf
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x17f
|
|
16
|
worker_thread
|
vmlinux |
0xffffffff998bf7d6
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x126
|
|
17
|
|
vmlinux |
0xffffffff998bf6b0
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x2a0
|
|
18
|
kthread
|
vmlinux |
0xffffffff998c6691
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0xd1
|
|
19
|
|
vmlinux |
0xffffffff998c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x40
|
|
20
|
ret_from_fork_nospec_begin
|
vmlinux |
0xffffffff99f92d37
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x21
|
|
21
|
|
vmlinux |
0xffffffff998c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64
|
0x40
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [kworker/0:0:42276]
Modules linked in: binfmt_misc tcp_lp rfcomm xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter vmw_vsock_vmci_transport ip6_tables vsock iptable_filter fuse bnep sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec btusb ac97_bus snd_seq btrtl iosf_mbi snd_seq_device btbcm crc32_pclmul snd_pcm ghash_clmulni_intel btintel snd_timer aesni_intel bluetooth
snd lrw gf128mul glue_helper ablk_helper ppdev soundcore rfkill cryptd sg pcspkr joydev vmw_balloon parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit mptspi scsi_transport_spi drm ata_piix crct10dif_pclmul mptscsih crct10dif_common mptbase crc32c_intel libnvdimm e1000 libata serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 42276 Comm: kworker/0:0 Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Workqueue: events linkwatch_event
task: ffff9df3b5541070 ti: ffff9df3d4514000 task.ti: ffff9df3d4514000
RIP: 0010:[<ffffffff998cee8f>] [<ffffffff998cee8f>] lg_global_lock+0x3f/0x80
RSP: 0000:ffff9df3d4517be8 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff9df558ebffff RCX: 0000000000000011
RDX: 0000000000000051 RSI: 0000000000000040 RDI: ffff9df5b6a56080
RBP: ffff9df3d4517c08 R08: ffffffff9a55ddc8 R09: 0000000000000000
R10: ffffffff9a55db40 R11: 0000000000000005 R12: ffff9df3d4517bc0
R13: ffffffff9a802bae R14: ffffffff9a27ee9a R15: ffff9df3d4517bd0
FS: 0000000000000000(0000) GS:ffff9df5b5600000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b845ad2f000 CR3: 000000033ca10000 CR4: 00000000003607f0
Call Trace:
[<ffffffff9993613c>] queue_stop_cpus_work+0x3c/0xa0
[<ffffffff999585d0>] ? get_state_synchronize_rcu+0x20/0x20
[<ffffffff99936402>] __stop_cpus+0x52/0x90
[<ffffffff999585d0>] ? get_state_synchronize_rcu+0x20/0x20
[<ffffffff9993681a>] try_stop_cpus+0x3a/0x60
[<ffffffff9995a417>] synchronize_sched_expedited+0xe7/0x1e0
[<ffffffff9995a5de>] synchronize_rcu_expedited+0xe/0x10
[<ffffffff99e4e08d>] synchronize_net+0x2d/0x40
[<ffffffff99e82067>] dev_deactivate_many+0x247/0x250
[<ffffffff99e820b3>] dev_deactivate+0x43/0x70
[<ffffffff99e6e7f8>] linkwatch_do_dev+0x38/0x60
[<ffffffff99e6eb19>] __linkwatch_run_queue+0xf9/0x190
[<ffffffff99e6ebd5>] linkwatch_event+0x25/0x30
[<ffffffff998be6bf>] process_one_work+0x17f/0x440
[<ffffffff998bf7d6>] worker_thread+0x126/0x3c0
[<ffffffff998bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff998c6691>] kthread+0xd1/0xe0
[<ffffffff998c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff99f92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff998c65c0>] ? insert_kthread_work+0x40/0x40
Code: ff ff ff ff 4c 8b 2d 41 43 74 00 41 bc 01 00 00 00 eb 21 66 0f 1f 84 00 00 00 00 00 48 98 49 8b 3e 48 03 3c c5 a0 10 55 9a 31 c0 <f0> 44 0f b1 27 85 c0 75 28 48 63 35 25 03 c9 00 83 c3 01 4c 89