Info

Permalink
Function
mpt_put_msg_frame
First occurrence
2020-09-26
Last occurrence
2021-11-17
State
NEW
Components
kernel
Quality
-64

Statistics

Operating system Count
CentOS 7 428
Architecture Count
x86_64 428
Related packages Count
kernel 428
 0:3.10.0-1127.19.1.el7 106
 0:3.10.0-1160.108.1.el7 85
 0:3.10.0-1127.10.1.el7 41
 0:3.10.0-1062.9.1.el7 31
 0:3.10.0-1160.42.2.el7 30
 0:3.10.0-1160.el7 18
 0:3.10.0-1160.53.1.el7 16
 0:3.10.0-1160.25.1.el7 12
 0:3.10.0-1127.el7 7
 0:3.10.0-957.el7 7
 0:3.10.0-1160.119.1.el7 6
 0:3.10.0-1160.49.1.el7 6
 0:3.10.0-1160.36.2.el7 6
 0:3.10.0-1160.31.1.el7 5
 0:3.10.0-1062.12.1.el7 5
 0:3.10.0-1160.90.1.el7 5
 0:3.10.0-1160.66.1.el7 4
 0:3.10.0-1160.71.1.el7 3
 0:3.10.0-1160.76.1.el7 3
 0:3.10.0-1062.4.1.el7 3
 0:3.10.0-693.2.2.el7 3
 0:3.10.0-693.el7 3
 0:3.10.0-957.21.3.el7 3
 0:3.10.0-1160.11.1.el7 2
 0:3.10.0-1062.4.3.el7 2
 0:3.10.0-1062.el7 2
 0:3.10.0-1160.102.1.el7 2
 0:3.10.0-957.5.1.el7 2
 0:3.10.0-1160.45.1.el7 1
 0:3.10.0-1160.6.1.el7 1
 0:3.10.0-1160.95.1.el7 1
 0:3.10.0-1160.62.1.el7 1
 0:3.10.0-1160.88.1.el7 1
 0:3.10.0-957.1.3.el7 1
 0:3.10.0-1160.59.1.el7 1
 0:3.10.0-1160.24.1.el7 1
 0:3.10.0-957.10.1.el7 1
 0:3.10.0-957.27.2.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1255254
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff987ba385
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x15
2
pvscsi_queue
vmw_pvscsi 0xffffffffc00796a2
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x4a2
3
? scsi_test_unit_ready
vmlinux 0xffffffff9850bcb0
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x170
4
scsi_dispatch_cmd
vmlinux 0xffffffff98503160
Build id: 3.10.0-1160.108.1.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffff9850cd0c
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffff9838e1de
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffff98368799
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffff9836c6c3
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffff9836a4f7
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x157
10
submit_bio
vmlinux 0xffffffff9836a7a0
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffff981d4f64
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc031d341
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc031db54
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffff981d5a0c
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc031d6c0
Build id: 3.10.0-1160.108.1.el7.x86_64 		0xb0
16
? cpumask_next_and
vmlinux 0xffffffff9839c5d5
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x35
17
xfs_vm_writepages
xfs 0xffffffffc031d67b
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x6b
18
do_writepages
vmlinux 0xffffffff981d6ab1
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x21
19
__writeback_single_inode
vmlinux 0xffffffff9828ccf0
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x40
20
writeback_sb_inodes
vmlinux 0xffffffff9828d8bc
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x1cc
21
__writeback_inodes_wb
vmlinux 0xffffffff9828dbc7
Build id: 3.10.0-1160.108.1.el7.x86_64 		0xa7
22
wb_writeback
vmlinux 0xffffffff9828e0a3
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x263
23
? get_nr_inodes
vmlinux 0xffffffff9827906c
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x4c
24
bdi_writeback_workfn
vmlinux 0xffffffff9828ecaf
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x2cf
25
process_one_work
vmlinux 0xffffffff980c32ef
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x17f
26
worker_thread
vmlinux 0xffffffff980c4436
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x126
27
? manage_workers.isra.26
vmlinux 0xffffffff980c4310
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x2b0
28
kthread
vmlinux 0xffffffff980cb621
Build id: 3.10.0-1160.108.1.el7.x86_64 		0xd1
29
? insert_kthread_work
vmlinux 0xffffffff980cb550
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffff987c51dd
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x7
31
? insert_kthread_work
vmlinux 0xffffffff980cb550
Build id: 3.10.0-1160.108.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#26 stuck for 38s! [kworker/u62:2:2426]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter nfsv3 nfs_acl nfs lockd grace fscache vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod ahci crc_t10dif crct10dif_generic drm ata_piix libahci
nfit libata libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 26 PID: 2426 Comm: kworker/u62:2 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.108.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff8e448ecc3180 ti: ffff8e49c066c000 task.ti: ffff8e49c066c000
RIP: 0010:[<ffffffff987ba385>]  [<ffffffff987ba385>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8e49c066f770  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffff983c9e83 RCX: ffffffff98c34f40
RDX: ffffa589cc4d0000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff8e49c066f770 R08: 0000000000000001 R09: 0000000000000000
R10: ffff8e42205c4000 R11: ffff8e4e09bb8000 R12: 0000000000000000
R13: 0000000000000020 R14: ffff8e40514cfd80 R15: ffffffff9850a8e9
FS:  0000000000000000(0000) GS:ffff8e4e0d4c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5a30007b20 CR3: 000000128f320000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffc00796a2>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffff9850bcb0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffff98503160>] scsi_dispatch_cmd+0xb0/0x250
[<ffffffff9850cd0c>] scsi_request_fn+0x4ac/0x680
[<ffffffff9838e1de>] ? deadline_add_request+0x6e/0x90
[<ffffffff98368799>] __blk_run_queue+0x39/0x50
[<ffffffff9836c6c3>] blk_queue_bio+0x3b3/0x400
[<ffffffff9836a4f7>] generic_make_request+0x157/0x390
[<ffffffff9836a7a0>] submit_bio+0x70/0x150
[<ffffffff981d4f64>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc031d341>] xfs_submit_ioend.isra.12+0x61/0xf0 [xfs]
[<ffffffffc031db54>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffff981d5a0c>] write_cache_pages+0x21c/0x480
[<ffffffffc031d6c0>] ? xfs_vm_writepages+0xb0/0xb0 [xfs]
[<ffffffff9839c5d5>] ? cpumask_next_and+0x35/0x50
[<ffffffffc031d67b>] xfs_vm_writepages+0x6b/0xb0 [xfs]
[<ffffffff981d6ab1>] do_writepages+0x21/0x50
[<ffffffff9828ccf0>] __writeback_single_inode+0x40/0x260
[<ffffffff9828d8bc>] writeback_sb_inodes+0x1cc/0x430
[<ffffffff9828dbc7>] __writeback_inodes_wb+0xa7/0xe0
[<ffffffff9828e0a3>] wb_writeback+0x263/0x2f0
[<ffffffff9827906c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9828ecaf>] bdi_writeback_workfn+0x2cf/0x470
[<ffffffff980c32ef>] process_one_work+0x17f/0x440
[<ffffffff980c4436>] worker_thread+0x126/0x3c0
[<ffffffff980c4310>] ? manage_workers.isra.26+0x2b0/0x2b0
[<ffffffff980cb621>] kthread+0xd1/0xe0
[<ffffffff980cb550>] ? insert_kthread_work+0x40/0x40
[<ffffffff987c51dd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff980cb550>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 cc cc cc cc 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 cc cc cc cc 0f 1f 44 00 00 55 48 89 e5 48
Complete report #1141754
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffb2b81125
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x15
2
ata_scsi_queuecmd
libata 0xffffffffc0493d65
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x155
3
? ata_scsi_invalid_field
libata 0xffffffffc0490a40
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffffb28dd100
Build id: 3.10.0-1062.4.1.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffffb28e676c
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x4cc
6
? deadline_add_request
vmlinux 0xffffffffb2771f4e
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffffb274d7b9
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffffb27514d3
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffffb274f467
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffffb274f710
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffffb25c7834
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc053bcd1
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc053c4c4
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffffb25c82ac
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc053c030
Build id: 3.10.0-1062.4.1.el7.x86_64 		0xa0
16
xfs_vm_writepages
xfs 0xffffffffc053bffb
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffffb25c92f1
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffffb2677f70
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x40
19
? wake_up_bit
vmlinux 0xffffffffb24c6265
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x25
20
writeback_sb_inodes
vmlinux 0xffffffffb2678b04
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x1c4
21
__writeback_inodes_wb
vmlinux 0xffffffffb2678e0f
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x9f
22
wb_writeback
vmlinux 0xffffffffb26792f3
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x263
23
bdi_writeback_workfn
vmlinux 0xffffffffb2679d35
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x115
24
? __schedule
vmlinux 0xffffffffb2b7e78a
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x66a
25
process_one_work
vmlinux 0xffffffffb24bd1df
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x17f
26
worker_thread
vmlinux 0xffffffffb24be2f6
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x126
27
? manage_workers.isra.26
vmlinux 0xffffffffb24be1d0
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x2a0
28
kthread
vmlinux 0xffffffffb24c51b1
Build id: 3.10.0-1062.4.1.el7.x86_64 		0xd1
29
? insert_kthread_work
vmlinux 0xffffffffb24c50e0
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffffb2b8bd1d
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x7
31
? insert_kthread_work
vmlinux 0xffffffffb24c50e0
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u4:3:17756]
Modules linked in: tcp_lp binfmt_misc fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc crc32_pclmul vboxvideo(OE) ttm ghash_clmulni_intel drm_kms_helper ppdev syscopyarea vboxguest(OE) sysfillrect sysimgblt aesni_intel fb_sys_fops lrw gf128mul glue_helper drm ablk_helper cryptd joydev sg pcspkr parport_pc parport video drm_panel_orientation_quirks
i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ahci libahci libata crct10dif_pclmul crct10dif_common e1000 crc32c_intel serio_raw dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 17756 Comm: kworker/u4:3 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1062.4.1.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff951c414f8000 ti: ffff951c45804000 task.ti: ffff951c45804000
RIP: 0010:[<ffffffffb2b81125>]  [<ffffffffb2b81125>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff951c458077a0  EFLAGS: 00010282
RAX: 0000000000000001 RBX: ffff951c75e98000 RCX: 0000000000000000
RDX: ffff951c75ae3c18 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff951c458077a0 R08: ffffffffffffffff R09: 0000000014439fff
R10: ffff951c759aa0d8 R11: ffff951c4160a380 R12: ffff951c45807748
R13: ffff951c41975e00 R14: ffff951c759ab000 R15: ffff951c759aa000
FS:  0000000000000000(0000) GS:ffff951cbfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fce20f13b10 CR3: 000000007b13c000 CR4: 00000000000206e0
Call Trace:
[<ffffffffc0493d65>] ata_scsi_queuecmd+0x155/0x450 [libata]
[<ffffffffc0490a40>] ? ata_scsi_invalid_field+0x50/0x50 [libata]
[<ffffffffb28dd100>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb28e676c>] scsi_request_fn+0x4cc/0x680
[<ffffffffb2771f4e>] ? deadline_add_request+0x6e/0x90
[<ffffffffb274d7b9>] __blk_run_queue+0x39/0x50
[<ffffffffb27514d3>] blk_queue_bio+0x3b3/0x400
[<ffffffffb274f467>] generic_make_request+0x147/0x380
[<ffffffffb274f710>] submit_bio+0x70/0x150
[<ffffffffb25c7834>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc053bcd1>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc053c4c4>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffffb25c82ac>] write_cache_pages+0x21c/0x470
[<ffffffffc053c030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffc053bffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb25c92f1>] do_writepages+0x21/0x50
[<ffffffffb2677f70>] __writeback_single_inode+0x40/0x260
[<ffffffffb24c6265>] ? wake_up_bit+0x25/0x30
[<ffffffffb2678b04>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffb2678e0f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb26792f3>] wb_writeback+0x263/0x2f0
[<ffffffffb2679d35>] bdi_writeback_workfn+0x115/0x460
[<ffffffffb2b7e78a>] ? __schedule+0x66a/0x9c0
[<ffffffffb24bd1df>] process_one_work+0x17f/0x440
[<ffffffffb24be2f6>] worker_thread+0x126/0x3c0
[<ffffffffb24be1d0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb24c51b1>] kthread+0xd1/0xe0
[<ffffffffb24c50e0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb2b8bd1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb24c50e0>] ? insert_kthread_work+0x40/0x40
Code: 36 44 99 ff 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 e8 16 44 99 ff 66 90 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #1026380
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc02b53be
Build id: 3.10.0-1160.el7.x86_64 		0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc02dc71e
Build id: 3.10.0-1160.el7.x86_64 		0x24e
3
mptspi_qcmd
mptspi 0xffffffffc02ab2e0
Build id: 3.10.0-1160.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffffb68e3980
Build id: 3.10.0-1160.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffffb68ed0ac
Build id: 3.10.0-1160.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffffb6777f8e
Build id: 3.10.0-1160.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffffb67534d9
Build id: 3.10.0-1160.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffffb67571e3
Build id: 3.10.0-1160.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffffb6755167
Build id: 3.10.0-1160.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffffb6755410
Build id: 3.10.0-1160.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffffb65c9314
Build id: 3.10.0-1160.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc0528f31
Build id: 3.10.0-1160.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc0529724
Build id: 3.10.0-1160.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffffb65c9d8c
Build id: 3.10.0-1160.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc0529290
Build id: 3.10.0-1160.el7.x86_64 		0xa0
16
? pick_next_entity
vmlinux 0xffffffffb64e23a9
Build id: 3.10.0-1160.el7.x86_64 		0xa9
17
? set_next_entity
vmlinux 0xffffffffb64e055c
Build id: 3.10.0-1160.el7.x86_64 		0x3c
18
? __switch_to
vmlinux 0xffffffffb642b59e
Build id: 3.10.0-1160.el7.x86_64 		0xce
19
xfs_vm_writepages
xfs 0xffffffffc052925b
Build id: 3.10.0-1160.el7.x86_64 		0x6b
20
do_writepages
vmlinux 0xffffffffb65cade1
Build id: 3.10.0-1160.el7.x86_64 		0x21
21
__writeback_single_inode
vmlinux 0xffffffffb667d6f0
Build id: 3.10.0-1160.el7.x86_64 		0x40
22
writeback_sb_inodes
vmlinux 0xffffffffb667e274
Build id: 3.10.0-1160.el7.x86_64 		0x1c4
23
__writeback_inodes_wb
vmlinux 0xffffffffb667e57f
Build id: 3.10.0-1160.el7.x86_64 		0x9f
24
wb_writeback
vmlinux 0xffffffffb667ea53
Build id: 3.10.0-1160.el7.x86_64 		0x263
25
bdi_writeback_workfn
vmlinux 0xffffffffb667f54c
Build id: 3.10.0-1160.el7.x86_64 		0x1cc
26
process_one_work
vmlinux 0xffffffffb64bdc4f
Build id: 3.10.0-1160.el7.x86_64 		0x17f
27
worker_thread
vmlinux 0xffffffffb64bed66
Build id: 3.10.0-1160.el7.x86_64 		0x126
28
? manage_workers.isra.26
vmlinux 0xffffffffb64bec40
Build id: 3.10.0-1160.el7.x86_64 		0x2a0
29
kthread
vmlinux 0xffffffffb64c5c21
Build id: 3.10.0-1160.el7.x86_64 		0xd1
30
? insert_kthread_work
vmlinux 0xffffffffb64c5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40
31
ret_from_fork_nospec_begin
vmlinux 0xffffffffb6b93df7
Build id: 3.10.0-1160.el7.x86_64 		0x21
32
? insert_kthread_work
vmlinux 0xffffffffb64c5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 39s! [kworker/u256:2:123460]
Modules linked in: vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi iosf_mbi snd_ac97_codec crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq snd_seq_device snd_pcm aesni_intel snd_timer lrw gf128mul snd ppdev glue_helper ablk_helper soundcore sg joydev cryptd pcspkr vmw_balloon parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul crct10dif_common crc32c_intel mptspi nfit e1000 scsi_transport_spi serio_raw mptscsih drm_panel_orientation_quirks mptbase libnvdimm dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 123460 Comm: kworker/u256:2 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9d15f3190000 ti: ffff9d15f00fc000 task.ti: ffff9d15f00fc000
RIP: 0010:[<ffffffffc02b53be>]  [<ffffffffc02b53be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff9d15f00ff708  EFLAGS: 00000246
RAX: ffffc22c806c0000 RBX: ffff9d164eca0000 RCX: 0000000000000010
RDX: ffff9d1575d77a00 RSI: ffff9d166d476000 RDI: 000000000000000e
RBP: ffff9d15f00ff718 R08: 0000000000000002 R09: 000000012aa27fff
R10: ffff9d166d241800 R11: ffff9d15c55ead80 R12: ffff9d15b182d800
R13: ffff9d15f00ff67c R14: ffffffffb675bdfb R15: ffff9d15f00ff6c0
FS:  0000000000000000(0000) GS:ffff9d1679640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001a8b150 CR3: 00000001392a0000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc02dc71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc02ab2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffb68e3980>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb68ed0ac>] scsi_request_fn+0x4ac/0x680
[<ffffffffb6777f8e>] ? deadline_add_request+0x6e/0x90
[<ffffffffb67534d9>] __blk_run_queue+0x39/0x50
[<ffffffffb67571e3>] blk_queue_bio+0x3b3/0x400
[<ffffffffb6755167>] generic_make_request+0x147/0x380
[<ffffffffb6755410>] submit_bio+0x70/0x150
[<ffffffffb65c9314>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc0528f31>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc0529724>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffffb65c9d8c>] write_cache_pages+0x21c/0x470
[<ffffffffc0529290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffb64e23a9>] ? pick_next_entity+0xa9/0x190
[<ffffffffb64e055c>] ? set_next_entity+0x3c/0xe0
[<ffffffffb642b59e>] ? __switch_to+0xce/0x580
[<ffffffffc052925b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb65cade1>] do_writepages+0x21/0x50
[<ffffffffb667d6f0>] __writeback_single_inode+0x40/0x260
[<ffffffffb667e274>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffb667e57f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb667ea53>] wb_writeback+0x263/0x2f0
[<ffffffffb667f54c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffffb64bdc4f>] process_one_work+0x17f/0x440
[<ffffffffb64bed66>] worker_thread+0x126/0x3c0
[<ffffffffb64bec40>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb64c5c21>] kthread+0xd1/0xe0
[<ffffffffb64c5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffb6b93df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb64c5b50>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 33 2c
Complete report #660120
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffba38e9f5
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x15
2
pvscsi_queue
vmw_pvscsi 0xffffffffc00a5632
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x4a2
3
? scsi_test_unit_ready
vmlinux 0xffffffffba0ecd60
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x170
4
scsi_dispatch_cmd
vmlinux 0xffffffffba0e4650
Build id: 3.10.0-1160.66.1.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffffba0edd7c
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffffb9f789de
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffffb9f53d09
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffffb9f57ad3
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffffb9f559b7
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffffb9f55c60
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffffb9dc9f04
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc161bf31
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc161c724
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffffb9dca97c
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc161c290
Build id: 3.10.0-1160.66.1.el7.x86_64 		0xa0
16
xfs_vm_writepages
xfs 0xffffffffc161c25b
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffffb9dcb9d1
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffffb9e7df40
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x40
19
writeback_sb_inodes
vmlinux 0xffffffffb9e7eac4
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffffb9e7edcf
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x9f
21
wb_writeback
vmlinux 0xffffffffb9e7f2a3
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x263
22
bdi_writeback_workfn
vmlinux 0xffffffffb9e7fe9b
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x2cb
23
process_one_work
vmlinux 0xffffffffb9cbdfbf
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x17f
24
worker_thread
vmlinux 0xffffffffb9cbf0d6
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x126
25
? manage_workers.isra.26
vmlinux 0xffffffffb9cbefb0
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x2a0
26
kthread
vmlinux 0xffffffffb9cc5f91
Build id: 3.10.0-1160.66.1.el7.x86_64 		0xd1
27
? insert_kthread_work
vmlinux 0xffffffffb9cc5ec0
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x40
28
ret_from_fork_nospec_begin
vmlinux 0xffffffffba399df7
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x21
29
? insert_kthread_work
vmlinux 0xffffffffb9cc5ec0
Build id: 3.10.0-1160.66.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [kworker/u16:0:23795]
Modules linked in: nvidia_uvm(OE) xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 tun ebtable_filter ebtables ip6table_filter devlink ip6_tables rpcsec_gss_krb5 nfsv4 dns_resolver xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc nfsv3 nfs_acl nfs lockd grace fscache overlay(T) vmw_vsock_vmci_transport vsock sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd sg pcspkr joydev vmw_balloon vmw_vmci i2c_piix4 parport_pc parport binfmt_misc auth_rpcgss sunrpc ip_tables xfs libcrc32c nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper
sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix libahci serio_raw nfit libata libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 5 PID: 23795 Comm: kworker/u16:0 Kdump: loaded Tainted: P           OEL ------------ T 3.10.0-1160.66.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff88b1f1b7d280 ti: ffff889bf0510000 task.ti: ffff889bf0510000
RIP: 0010:[<ffffffffba38e9f5>]  [<ffffffffba38e9f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff889bf0513770  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffffb9fb2d73 RCX: ffffffffba834f40
RDX: ffffb55b59260000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff889bf0513770 R08: 0000000000000001 R09: 0000000000000000
R10: ffff88b2356ae800 R11: ffff88b2352c3800 R12: 0000000000000000
R13: 0000000000000020 R14: ffff88b1f2042a80 R15: ffffffffba0eba19
FS:  0000000000000000(0000) GS:ffff88b2bfd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002b2353b91001 CR3: 00000020a8b66000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc00a5632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffffba0ecd60>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffffba0e4650>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffba0edd7c>] scsi_request_fn+0x4ac/0x680
[<ffffffffb9f789de>] ? deadline_add_request+0x6e/0x90
[<ffffffffb9f53d09>] __blk_run_queue+0x39/0x50
[<ffffffffb9f57ad3>] blk_queue_bio+0x3b3/0x400
[<ffffffffb9f559b7>] generic_make_request+0x147/0x380
[<ffffffffb9f55c60>] submit_bio+0x70/0x150
[<ffffffffb9dc9f04>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc161bf31>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc161c724>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffffb9dca97c>] write_cache_pages+0x21c/0x470
[<ffffffffc161c290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffc161c25b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb9dcb9d1>] do_writepages+0x21/0x50
[<ffffffffb9e7df40>] __writeback_single_inode+0x40/0x260
[<ffffffffb9e7eac4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffb9e7edcf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb9e7f2a3>] wb_writeback+0x263/0x2f0
[<ffffffffb9e7fe9b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb9cbdfbf>] process_one_work+0x17f/0x440
[<ffffffffb9cbf0d6>] worker_thread+0x126/0x3c0
[<ffffffffb9cbefb0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb9cc5f91>] kthread+0xd1/0xe0
[<ffffffffb9cc5ec0>] ? insert_kthread_work+0x40/0x40
[<ffffffffba399df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb9cc5ec0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #573841
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc00123be
Build id: 3.10.0-693.el7.x86_64 		0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc0034729
Build id: 3.10.0-693.el7.x86_64 		0x249
3
mptspi_qcmd
mptspi 0xffffffffc000b2b0
Build id: 3.10.0-693.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffff8146540a
Build id: 3.10.0-693.el7.x86_64 		0xaa
5
scsi_request_fn
vmlinux 0xffffffff8146e1cf
Build id: 3.10.0-693.el7.x86_64 		0x4df
6
__blk_run_queue
vmlinux 0xffffffff812f71d3
Build id: 3.10.0-693.el7.x86_64 		0x33
7
blk_queue_bio
vmlinux 0xffffffff812faad3
Build id: 3.10.0-693.el7.x86_64 		0x363
8
generic_make_request
vmlinux 0xffffffff812f8b45
Build id: 3.10.0-693.el7.x86_64 		0x105
9
submit_bio
vmlinux 0xffffffff812f8dc0
Build id: 3.10.0-693.el7.x86_64 		0x70
10
? __test_set_page_writeback
vmlinux 0xffffffff8118d4ca
Build id: 3.10.0-693.el7.x86_64 		0xca
11
xfs_submit_ioend.isra.16
xfs 0xffffffffc02475e1
Build id: 3.10.0-693.el7.x86_64 		0x61
12
xfs_do_writepage
xfs 0xffffffffc0247e44
Build id: 3.10.0-693.el7.x86_64 		0x4c4
13
write_cache_pages
vmlinux 0xffffffff8118dc91
Build id: 3.10.0-693.el7.x86_64 		0x251
14
? xfs_aops_discard_page
xfs 0xffffffffc0247980
Build id: 3.10.0-693.el7.x86_64 		0x150
15
xfs_vm_writepages
xfs 0xffffffffc024778d
Build id: 3.10.0-693.el7.x86_64 		0xbd
16
do_writepages
vmlinux 0xffffffff8118effe
Build id: 3.10.0-693.el7.x86_64 		0x1e
17
__writeback_single_inode
vmlinux 0xffffffff8122d7d0
Build id: 3.10.0-693.el7.x86_64 		0x40
18
writeback_sb_inodes
vmlinux 0xffffffff8122e414
Build id: 3.10.0-693.el7.x86_64 		0x1c4
19
__writeback_inodes_wb
vmlinux 0xffffffff8122e77f
Build id: 3.10.0-693.el7.x86_64 		0x9f
20
wb_writeback
vmlinux 0xffffffff8122efb3
Build id: 3.10.0-693.el7.x86_64 		0x263
21
? get_nr_inodes
vmlinux 0xffffffff8121bc3c
Build id: 3.10.0-693.el7.x86_64 		0x4c
22
bdi_writeback_workfn
vmlinux 0xffffffff8122f45b
Build id: 3.10.0-693.el7.x86_64 		0x2cb
23
process_one_work
vmlinux 0xffffffff810a881a
Build id: 3.10.0-693.el7.x86_64 		0x17a
24
worker_thread
vmlinux 0xffffffff810a94e6
Build id: 3.10.0-693.el7.x86_64 		0x126
25
? manage_workers.isra.24
vmlinux 0xffffffff810a93c0
Build id: 3.10.0-693.el7.x86_64 		0x2a0
26
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.el7.x86_64 		0xcf
27
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64 		0x40
28
ret_from_fork
vmlinux 0xffffffff816b4f18
Build id: 3.10.0-693.el7.x86_64 		0x58
29
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u256:1:19215]
<EOI> 
Modules linked in:
nls_utf8 isofs rfcomm fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 btusb btrtl btbcm btintel iosf_mbi crc32_pclmul ghash_clmulni_intel bluetooth aesni_intel lrw snd_rawmidi gf128mul snd_ac97_codec glue_helper ac97_bus snd_seq ppdev
ablk_helper snd_seq_device rfkill snd_pcm snd_timer snd cryptd soundcore pcspkr vmw_balloon nfit sg joydev libnvdimm parport_pc parport shpchp i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix e1000 libata serio_raw mptspi scsi_transport_spi mptscsih mptbase i2c_core
CPU: 1 PID: 19215 Comm: kworker/u256:1 Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff88018b621fa0 ti: ffff88022ffac000 task.ti: ffff88022ffac000
RIP: 0010:[<ffffffffc00123be>]  [<ffffffffc00123be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff88022ffaf6f0  EFLAGS: 00000246
RAX: ffffc90000f80000 RBX: ffff88018c7e2500 RCX: 000000000000001d
RDX: ffff880036327600 RSI: ffff88003623b000 RDI: 000000000000000e
RBP: ffff88022ffaf700 R08: 0000000000000002 R09: ffff8800362380d8
R10: ffff8801f6e816c0 R11: ffff8801f6e816c0 R12: 0000000100000001
R13: ffff8801c6b27a80 R14: ffff88022ffaf664 R15: ffffffff812ff13b
FS:  0000000000000000(0000) GS:ffff880235640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007eff47e6b400 CR3: 000000009ecad000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff88003623bd30 000000000000003c ffff88022ffaf7a8 ffffffffc0034729
ffff88003623b008 040000002ffaf768 000000000000001d ffff8801f6e816c0
0000000000000074 ffff88003623b188 000000600000001d ffff880230443310
Call Trace:
[<ffffffffc0034729>] mptscsih_qcmd+0x249/0x820 [mptscsih]
[<ffffffffc000b2b0>] mptspi_qcmd+0x50/0xe0 [mptspi]
[<ffffffff8146540a>] scsi_dispatch_cmd+0xaa/0x230
[<ffffffff8146e1cf>] scsi_request_fn+0x4df/0x680
[<ffffffff812f71d3>] __blk_run_queue+0x33/0x40
[<ffffffff812faad3>] blk_queue_bio+0x363/0x3a0
[<ffffffff812f8b45>] generic_make_request+0x105/0x310
[<ffffffff812f8dc0>] submit_bio+0x70/0x150
[<ffffffff8118d4ca>] ? __test_set_page_writeback+0xca/0x160
[<ffffffffc02475e1>] xfs_submit_ioend.isra.16+0x61/0xe0 [xfs]
[<ffffffffc0247e44>] xfs_do_writepage+0x4c4/0x530 [xfs]
[<ffffffff8118dc91>] write_cache_pages+0x251/0x4d0
[<ffffffffc0247980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffc024778d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffff8118effe>] do_writepages+0x1e/0x40
[<ffffffff8122d7d0>] __writeback_single_inode+0x40/0x220
[<ffffffff8122e414>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff8122e77f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8122efb3>] wb_writeback+0x263/0x2f0
[<ffffffff8121bc3c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8122f45b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f18>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 03 02
Complete report #412304
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff9d9883c5
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x15
2
pvscsi_queue
vmw_pvscsi 0xffffffffc0148632
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x4a2
3
? scsi_test_unit_ready
vmlinux 0xffffffff9d6eb030
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x170
4
scsi_dispatch_cmd
vmlinux 0xffffffff9d6e2920
Build id: 3.10.0-1127.10.1.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffff9d6ec04c
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffff9d57716e
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffff9d552769
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffff9d556473
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffff9d5543f7
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffff9d5546a0
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffff9d3c9744
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc038cfb1
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc038d7a4
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffff9d3ca1bc
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc038d310
Build id: 3.10.0-1127.10.1.el7.x86_64 		0xa0
16
xfs_vm_writepages
xfs 0xffffffffc038d2db
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffff9d3cb211
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffff9d47cc30
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x40
19
? wake_up_bit
vmlinux 0xffffffff9d2c7745
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x25
20
writeback_sb_inodes
vmlinux 0xffffffff9d47d7c4
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x1c4
21
__writeback_inodes_wb
vmlinux 0xffffffff9d47dacf
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x9f
22
wb_writeback
vmlinux 0xffffffff9d47dfb3
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x263
23
? get_nr_inodes
vmlinux 0xffffffff9d469c0c
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x4c
24
bdi_writeback_workfn
vmlinux 0xffffffff9d47ebab
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x2cb
25
process_one_work
vmlinux 0xffffffff9d2be6bf
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x17f
26
worker_thread
vmlinux 0xffffffff9d2bf7d6
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x126
27
? manage_workers.isra.26
vmlinux 0xffffffff9d2bf6b0
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x2a0
28
kthread
vmlinux 0xffffffff9d2c6691
Build id: 3.10.0-1127.10.1.el7.x86_64 		0xd1
29
? insert_kthread_work
vmlinux 0xffffffff9d2c65c0
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffff9d992d37
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x21
31
? insert_kthread_work
vmlinux 0xffffffff9d2c65c0
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 38s! [kworker/u2:1:18533]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13402(E) falcon_lsm_pinned_13207(E) falcon_lsm_pinned_13109(E) falcon_lsm_pinned_13005(E) tcp_diag udp_diag inet_diag falcon_lsm_pinned_12905(E) falcon_lsm_pinned_12803(E) falcon_lsm_pinned_12104(E) nfsv3 nfs_acl nfs lockd grace fscache binfmt_misc vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd pcspkr vmw_balloon joydev sg parport_pc i2c_piix4 parport vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt sd_mod crc_t10dif fb_sys_fops crct10dif_generic ttm drm ata_piix crct10dif_pclmul crct10dif_common
libata nfit crc32c_intel serio_raw libnvdimm vmw_pvscsi vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_kal]
CPU: 0 PID: 18533 Comm: kworker/u2:1 Kdump: loaded Tainted: P            EL ------------   3.10.0-1127.10.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Workqueue: writeback bdi_writeback_workfn (flush-253:1)
task: ffff986a0b583150 ti: ffff986adaf74000 task.ti: ffff986adaf74000
RIP: 0010:[<ffffffff9d9883c5>]  [<ffffffff9d9883c5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff986adaf77770  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffff9d5b1473 RCX: ffffffff9de34e40
RDX: ffffa86c81b90000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff986adaf77770 R08: 0000000000000001 R09: 0000000000000000
R10: ffff986cab158000 R11: ffff986c3afbf000 R12: 0000000000000000
R13: 0000000000000020 R14: ffff9868b67fea80 R15: ffffffff9d6e9ce9
FS:  0000000000000000(0000) GS:ffff986cbfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6a81ad1aa8 CR3: 0000000429c2c000 CR4: 00000000001607f0
Call Trace:
[<ffffffffc0148632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffff9d6eb030>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffff9d6e2920>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff9d6ec04c>] scsi_request_fn+0x4ac/0x680
[<ffffffff9d57716e>] ? deadline_add_request+0x6e/0x90
[<ffffffff9d552769>] __blk_run_queue+0x39/0x50
[<ffffffff9d556473>] blk_queue_bio+0x3b3/0x400
[<ffffffff9d5543f7>] generic_make_request+0x147/0x380
[<ffffffff9d5546a0>] submit_bio+0x70/0x150
[<ffffffff9d3c9744>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc038cfb1>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc038d7a4>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffff9d3ca1bc>] write_cache_pages+0x21c/0x470
[<ffffffffc038d310>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffc038d2db>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff9d3cb211>] do_writepages+0x21/0x50
[<ffffffff9d47cc30>] __writeback_single_inode+0x40/0x260
[<ffffffff9d2c7745>] ? wake_up_bit+0x25/0x30
[<ffffffff9d47d7c4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff9d47dacf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff9d47dfb3>] wb_writeback+0x263/0x2f0
[<ffffffff9d469c0c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff9d47ebab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff9d2be6bf>] process_one_work+0x17f/0x440
[<ffffffff9d2bf7d6>] worker_thread+0x126/0x3c0
[<ffffffff9d2bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff9d2c6691>] kthread+0xd1/0xe0
[<ffffffff9d2c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff9d992d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff9d2c65c0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #363632
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc02643be
Build id: 3.10.0-957.21.3.el7.x86_64 		0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc02da71e
Build id: 3.10.0-957.21.3.el7.x86_64 		0x24e
3
mptspi_qcmd
mptspi 0xffffffffc02c32e0
Build id: 3.10.0-957.21.3.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffffa14d1700
Build id: 3.10.0-957.21.3.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffffa14dac9c
Build id: 3.10.0-957.21.3.el7.x86_64 		0x4cc
6
? deadline_add_request
vmlinux 0xffffffffa1367e6e
Build id: 3.10.0-957.21.3.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffffa1343849
Build id: 3.10.0-957.21.3.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffffa1347553
Build id: 3.10.0-957.21.3.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffffa13454e7
Build id: 3.10.0-957.21.3.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffffa1345790
Build id: 3.10.0-957.21.3.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffffa11c1b44
Build id: 3.10.0-957.21.3.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc04b4471
Build id: 3.10.0-957.21.3.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc04b4c74
Build id: 3.10.0-957.21.3.el7.x86_64 		0x4a4
14
write_cache_pages
vmlinux 0xffffffffa11c25bc
Build id: 3.10.0-957.21.3.el7.x86_64 		0x21c
15
? xfs_aops_discard_page
xfs 0xffffffffc04b47d0
Build id: 3.10.0-957.21.3.el7.x86_64 		0x150
16
xfs_vm_writepages
xfs 0xffffffffc04b45cb
Build id: 3.10.0-957.21.3.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffffa11c36d1
Build id: 3.10.0-957.21.3.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffffa12701b0
Build id: 3.10.0-957.21.3.el7.x86_64 		0x40
19
writeback_sb_inodes
vmlinux 0xffffffffa1270c44
Build id: 3.10.0-957.21.3.el7.x86_64 		0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffffa1270faf
Build id: 3.10.0-957.21.3.el7.x86_64 		0x9f
21
wb_writeback
vmlinux 0xffffffffa12717e3
Build id: 3.10.0-957.21.3.el7.x86_64 		0x263
22
bdi_writeback_workfn
vmlinux 0xffffffffa127216b
Build id: 3.10.0-957.21.3.el7.x86_64 		0x2cb
23
process_one_work
vmlinux 0xffffffffa10b9ebf
Build id: 3.10.0-957.21.3.el7.x86_64 		0x17f
24
worker_thread
vmlinux 0xffffffffa10baf56
Build id: 3.10.0-957.21.3.el7.x86_64 		0x126
25
? manage_workers.isra.25
vmlinux 0xffffffffa10bae30
Build id: 3.10.0-957.21.3.el7.x86_64 		0x2a0
26
kthread
vmlinux 0xffffffffa10c1da1
Build id: 3.10.0-957.21.3.el7.x86_64 		0xd1
27
? insert_kthread_work
vmlinux 0xffffffffa10c1cd0
Build id: 3.10.0-957.21.3.el7.x86_64 		0x40
28
ret_from_fork_nospec_begin
vmlinux 0xffffffffa1775c37
Build id: 3.10.0-957.21.3.el7.x86_64 		0x21
29
? insert_kthread_work
vmlinux 0xffffffffa10c1cd0
Build id: 3.10.0-957.21.3.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 38s! [awk:20258]
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 38s! [kworker/u8:0:19772]
Modules linked in: nfsv3 rpcsec_gss_krb5 nfsv4 xt_CHECKSUM iptable_mangle nfs fscache ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter cmac arc4 md4 nls_utf8 cifs ccm dns_resolver vmw_vsock_vmci_transport vsock ppdev vmw_balloon coretemp crc32_pclmul ghash_clmulni_intel ext4 aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev mbcache jbd2 pcspkr sg parport_pc parport vmw_vmci i2c_piix4 nfsd nfs_acl lockd grace auth_rpcgss binfmt_misc sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif sr_mod crct10dif_generic cdrom crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 ata_generic
pata_acpi vmwgfx floppy drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix scsi_transport_spi mptscsih libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 19772 Comm: kworker/u8:0 Kdump: loaded Tainted: G             L ------------   3.10.0-957.21.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/14/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffffa118b2bc5140 ti: ffffa11875828000 task.ti: ffffa11875828000
RIP: 0010:[<ffffffffc02643be>]  [<ffffffffc02643be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffffa1187582b708  EFLAGS: 00000246
RAX: ffffb9c3835c0000 RBX: ffffa110e5127020 RCX: 0000000000000035
RDX: ffffa118db0a7a00 RSI: ffffa118ff91d000 RDI: 000000000000000e
RBP: ffffa1187582b718 R08: 0000000000000003 R09: 00000005b7a83fff
R10: ffffa115dd78c0d8 R11: ffffa118d93776c0 R12: ffffa11231034300
R13: ffffa1187582b67c R14: ffffffffa134c09b R15: ffffa1187582b6c0
FS:  0000000000000000(0000) GS:ffffa118ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fabc3e89353 CR3: 0000000804634000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc02da71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc02c32e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffa14d1700>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffa14dac9c>] scsi_request_fn+0x4cc/0x680
[<ffffffffa1367e6e>] ? deadline_add_request+0x6e/0x90
[<ffffffffa1343849>] __blk_run_queue+0x39/0x50
[<ffffffffa1347553>] blk_queue_bio+0x3b3/0x400
[<ffffffffa13454e7>] generic_make_request+0x147/0x380
[<ffffffffa1345790>] submit_bio+0x70/0x150
[<ffffffffa11c1b44>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc04b4471>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc04b4c74>] xfs_do_writepage+0x4a4/0x510 [xfs]
[<ffffffffa11c25bc>] write_cache_pages+0x21c/0x470
[<ffffffffc04b47d0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffc04b45cb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa11c36d1>] do_writepages+0x21/0x50
[<ffffffffa12701b0>] __writeback_single_inode+0x40/0x260
[<ffffffffa1270c44>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffa1270faf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa12717e3>] wb_writeback+0x263/0x2f0
[<ffffffffa127216b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffa10b9ebf>] process_one_work+0x17f/0x440
[<ffffffffa10baf56>] worker_thread+0x126/0x3c0
[<ffffffffa10bae30>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffa10c1da1>] kthread+0xd1/0xe0
[<ffffffffa10c1cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa1775c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa10c1cd0>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 23 27
Complete report #358751
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffa738b795
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x15
2
pvscsi_queue
vmw_pvscsi 0xffffffffc03ff632
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x4a2
3
? scsi_test_unit_ready
vmlinux 0xffffffffa70ecf70
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x170
4
scsi_dispatch_cmd
vmlinux 0xffffffffa70e4860
Build id: 3.10.0-1160.49.1.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffffa70edf8c
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffffa6f78bee
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffffa6f53f39
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffffa6f57ce3
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffffa6f55bc7
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffffa6f55e70
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffffa6dc9ee4
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc063ff31
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc0640724
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffffa6dca95c
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc0640290
Build id: 3.10.0-1160.49.1.el7.x86_64 		0xa0
16
? __enqueue_entity
vmlinux 0xffffffffa6cdfbd8
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x78
17
? put_prev_entity
vmlinux 0xffffffffa6ce637b
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x8b
18
xfs_vm_writepages
xfs 0xffffffffc064025b
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x6b
19
do_writepages
vmlinux 0xffffffffa6dcb9b1
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x21
20
__writeback_single_inode
vmlinux 0xffffffffa6e7df10
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x40
21
writeback_sb_inodes
vmlinux 0xffffffffa6e7ea94
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x1c4
22
__writeback_inodes_wb
vmlinux 0xffffffffa6e7ed9f
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x9f
23
bdi_writeback_workfn
vmlinux 0xffffffffa6e7ff76
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x3d6
24
process_one_work
vmlinux 0xffffffffa6cbde8f
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x17f
25
rescuer_thread
vmlinux 0xffffffffa6cbe3e4
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x294
26
? process_one_work
vmlinux 0xffffffffa6cbe150
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x440
27
kthread
vmlinux 0xffffffffa6cc5e61
Build id: 3.10.0-1160.49.1.el7.x86_64 		0xd1
28
? insert_kthread_work
vmlinux 0xffffffffa6cc5d90
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffffa7395df7
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x21
30
? insert_kthread_work
vmlinux 0xffffffffa6cc5d90
Build id: 3.10.0-1160.49.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [openvpn:1148]
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [writeback:31]
Modules linked in: xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype br_netfilter rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs tun lockd grace fscache binfmt_misc overlay(T) nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd pcspkr
vmw_balloon joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic ata_piix syscopyarea sysfillrect sysimgblt fb_sys_fops ttm libata e1000 crct10dif_pclmul crct10dif_common drm crc32c_intel serio_raw vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 31 Comm: writeback Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 10/22/2013
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8f1bb5c73180 ti: ffff8f1bb5cc4000 task.ti: ffff8f1bb5cc4000
RIP: 0010:[<ffffffffa738b795>]  [<ffffffffa738b795>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8f1bb5cc77a0  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffffa6fb2f83 RCX: ffffffffa7834f40
RDX: ffffa3fb01b90000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff8f1bb5cc77a0 R08: 0000000000000001 R09: 0000000000000000
R10: ffff8f1e6b08baf8 R11: ffff8f1e6bea7800 R12: ffff8f1bb5cc7788
R13: 0000000000000001 R14: ffff8f1bb5cc7750 R15: ffffffffa6f55acf
FS:  0000000000000000(0000) GS:ffff8f1e7fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc3cb8ec000 CR3: 000000022d616000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc03ff632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffffa70ecf70>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffffa70e4860>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffa70edf8c>] scsi_request_fn+0x4ac/0x680
[<ffffffffa6f78bee>] ? deadline_add_request+0x6e/0x90
[<ffffffffa6f53f39>] __blk_run_queue+0x39/0x50
[<ffffffffa6f57ce3>] blk_queue_bio+0x3b3/0x400
[<ffffffffa6f55bc7>] generic_make_request+0x147/0x380
[<ffffffffa6f55e70>] submit_bio+0x70/0x150
[<ffffffffa6dc9ee4>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc063ff31>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc0640724>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffffa6dca95c>] write_cache_pages+0x21c/0x470
[<ffffffffc0640290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffa6cdfbd8>] ? __enqueue_entity+0x78/0x80
[<ffffffffa6ce637b>] ? put_prev_entity+0x8b/0x400
[<ffffffffc064025b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa6dcb9b1>] do_writepages+0x21/0x50
[<ffffffffa6e7df10>] __writeback_single_inode+0x40/0x260
[<ffffffffa6e7ea94>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffa6e7ed9f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa6e7ff76>] bdi_writeback_workfn+0x3d6/0x460
[<ffffffffa6cbde8f>] process_one_work+0x17f/0x440
[<ffffffffa6cbe3e4>] rescuer_thread+0x294/0x3c0
[<ffffffffa6cbe150>] ? process_one_work+0x440/0x440
[<ffffffffa6cc5e61>] kthread+0xd1/0xe0
[<ffffffffa6cc5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffa7395df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa6cc5d90>] ? insert_kthread_work+0x40/0x40
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48
Complete report #331054
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc03e23be
Build id: 3.10.0-957.5.1.el7.x86_64 		0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc03bd71e
Build id: 3.10.0-957.5.1.el7.x86_64 		0x24e
3
mptspi_qcmd
mptspi 0xffffffffc03db2e0
Build id: 3.10.0-957.5.1.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffffb96d0fe0
Build id: 3.10.0-957.5.1.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffffb96da57c
Build id: 3.10.0-957.5.1.el7.x86_64 		0x4cc
6
? deadline_add_request
vmlinux 0xffffffffb9567bde
Build id: 3.10.0-957.5.1.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffffb95435d9
Build id: 3.10.0-957.5.1.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffffb95472d3
Build id: 3.10.0-957.5.1.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffffb9545267
Build id: 3.10.0-957.5.1.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffffb9545510
Build id: 3.10.0-957.5.1.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffffb93c1834
Build id: 3.10.0-957.5.1.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc0616251
Build id: 3.10.0-957.5.1.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc0616a54
Build id: 3.10.0-957.5.1.el7.x86_64 		0x4a4
14
write_cache_pages
vmlinux 0xffffffffb93c22e4
Build id: 3.10.0-957.5.1.el7.x86_64 		0x254
15
? xfs_aops_discard_page
xfs 0xffffffffc06165b0
Build id: 3.10.0-957.5.1.el7.x86_64 		0x150
16
? submit_bio
vmlinux 0xffffffffb9545510
Build id: 3.10.0-957.5.1.el7.x86_64 		0x70
17
? __switch_to
vmlinux 0xffffffffb922a59e
Build id: 3.10.0-957.5.1.el7.x86_64 		0xce
18
xfs_vm_writepages
xfs 0xffffffffc06163ab
Build id: 3.10.0-957.5.1.el7.x86_64 		0x6b
19
do_writepages
vmlinux 0xffffffffb93c3431
Build id: 3.10.0-957.5.1.el7.x86_64 		0x21
20
__writeback_single_inode
vmlinux 0xffffffffb946fee0
Build id: 3.10.0-957.5.1.el7.x86_64 		0x40
21
writeback_sb_inodes
vmlinux 0xffffffffb9470974
Build id: 3.10.0-957.5.1.el7.x86_64 		0x1c4
22
__writeback_inodes_wb
vmlinux 0xffffffffb9470cdf
Build id: 3.10.0-957.5.1.el7.x86_64 		0x9f
23
wb_writeback
vmlinux 0xffffffffb9471513
Build id: 3.10.0-957.5.1.el7.x86_64 		0x263
24
bdi_writeback_workfn
vmlinux 0xffffffffb9471d9c
Build id: 3.10.0-957.5.1.el7.x86_64 		0x1cc
25
process_one_work
vmlinux 0xffffffffb92b9d8f
Build id: 3.10.0-957.5.1.el7.x86_64 		0x17f
26
worker_thread
vmlinux 0xffffffffb92bae26
Build id: 3.10.0-957.5.1.el7.x86_64 		0x126
27
? manage_workers.isra.25
vmlinux 0xffffffffb92bad00
Build id: 3.10.0-957.5.1.el7.x86_64 		0x2a0
28
kthread
vmlinux 0xffffffffb92c1c71
Build id: 3.10.0-957.5.1.el7.x86_64 		0xd1
29
? insert_kthread_work
vmlinux 0xffffffffb92c1ba0
Build id: 3.10.0-957.5.1.el7.x86_64 		0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffffb9974c37
Build id: 3.10.0-957.5.1.el7.x86_64 		0x21
31
? insert_kthread_work
vmlinux 0xffffffffb92c1ba0
Build id: 3.10.0-957.5.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 25s! [kworker/u256:1:100830]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ppdev vmw_balloon joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi e1000 libata drm_panel_orientation_quirks mptscsih mptbase nfit libnvdimm floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 100830 Comm: kworker/u256:1 Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.5.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Workqueue: writeback bdi_writeback_workfn (flush-253:1)
task: ffff90d584fe1040 ti: ffff90d5b7770000 task.ti: ffff90d5b7770000
RIP: 0010:[<ffffffffc03e23be>]  [<ffffffffc03e23be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff90d5b77736f8  EFLAGS: 00000246
RAX: ffffac42c0d00000 RBX: ffff90d58e1cdab0 RCX: 0000000000000007
RDX: ffff90d5b6548800 RSI: ffff90d7b129f000 RDI: 000000000000000e
RBP: ffff90d5b7773708 R08: 0000000000000003 R09: 00000000ba544fff
R10: ffff90d7b28ae0d8 R11: ffff90d60b8c8700 R12: ffff90d7aff75080
R13: ffff90d5b777366c R14: ffffffffb954be1b R15: ffff90d5b77736b0
FS:  0000000000000000(0000) GS:ffff90d7b57c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2a614fc9ab CR3: 000000004a2d0000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc03bd71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc03db2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffb96d0fe0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb96da57c>] scsi_request_fn+0x4cc/0x680
[<ffffffffb9567bde>] ? deadline_add_request+0x6e/0x90
[<ffffffffb95435d9>] __blk_run_queue+0x39/0x50
[<ffffffffb95472d3>] blk_queue_bio+0x3b3/0x400
[<ffffffffb9545267>] generic_make_request+0x147/0x380
[<ffffffffb9545510>] submit_bio+0x70/0x150
[<ffffffffb93c1834>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc0616251>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc0616a54>] xfs_do_writepage+0x4a4/0x510 [xfs]
[<ffffffffb93c22e4>] write_cache_pages+0x254/0x4e0
[<ffffffffc06165b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffb9545510>] ? submit_bio+0x70/0x150
[<ffffffffb922a59e>] ? __switch_to+0xce/0x580
[<ffffffffc06163ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb93c3431>] do_writepages+0x21/0x50
[<ffffffffb946fee0>] __writeback_single_inode+0x40/0x260
[<ffffffffb9470974>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffb9470cdf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb9471513>] wb_writeback+0x263/0x2f0
[<ffffffffb9471d9c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffffb92b9d8f>] process_one_work+0x17f/0x440
[<ffffffffb92bae26>] worker_thread+0x126/0x3c0
[<ffffffffb92bad00>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffb92c1c71>] kthread+0xd1/0xe0
[<ffffffffb92c1ba0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb9974c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb92c1ba0>] ? insert_kthread_work+0x40/0x40
Complete report #327232
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffa6b6c495
Build id: 3.10.0-957.27.2.el7.x86_64 		0x15
2
pvscsi_queue
vmw_pvscsi 0xffffffffc0437622
Build id: 3.10.0-957.27.2.el7.x86_64 		0x4a2
3
? scsi_test_unit_ready
vmlinux 0xffffffffa68db410
Build id: 3.10.0-957.27.2.el7.x86_64 		0x170
4
scsi_dispatch_cmd
vmlinux 0xffffffffa68d2ee0
Build id: 3.10.0-957.27.2.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffffa68dc47c
Build id: 3.10.0-957.27.2.el7.x86_64 		0x4cc
6
? deadline_add_request
vmlinux 0xffffffffa67694de
Build id: 3.10.0-957.27.2.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffffa6744e59
Build id: 3.10.0-957.27.2.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffffa6748b63
Build id: 3.10.0-957.27.2.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffffa6746af7
Build id: 3.10.0-957.27.2.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffffa6746da0
Build id: 3.10.0-957.27.2.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffffa65c2c14
Build id: 3.10.0-957.27.2.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc06a4471
Build id: 3.10.0-957.27.2.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc06a4c74
Build id: 3.10.0-957.27.2.el7.x86_64 		0x4a4
14
write_cache_pages
vmlinux 0xffffffffa65c368c
Build id: 3.10.0-957.27.2.el7.x86_64 		0x21c
15
? xfs_aops_discard_page
xfs 0xffffffffc06a47d0
Build id: 3.10.0-957.27.2.el7.x86_64 		0x150
16
xfs_vm_writepages
xfs 0xffffffffc06a45cb
Build id: 3.10.0-957.27.2.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffffa65c47a1
Build id: 3.10.0-957.27.2.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffffa6671020
Build id: 3.10.0-957.27.2.el7.x86_64 		0x40
19
writeback_sb_inodes
vmlinux 0xffffffffa6671ab4
Build id: 3.10.0-957.27.2.el7.x86_64 		0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffffa6671e1f
Build id: 3.10.0-957.27.2.el7.x86_64 		0x9f
21
wb_writeback
vmlinux 0xffffffffa6672653
Build id: 3.10.0-957.27.2.el7.x86_64 		0x263
22
bdi_writeback_workfn
vmlinux 0xffffffffa6672edc
Build id: 3.10.0-957.27.2.el7.x86_64 		0x1cc
23
process_one_work
vmlinux 0xffffffffa64baf9f
Build id: 3.10.0-957.27.2.el7.x86_64 		0x17f
24
worker_thread
vmlinux 0xffffffffa64bc036
Build id: 3.10.0-957.27.2.el7.x86_64 		0x126
25
? manage_workers.isra.25
vmlinux 0xffffffffa64bbf10
Build id: 3.10.0-957.27.2.el7.x86_64 		0x2a0
26
kthread
vmlinux 0xffffffffa64c2e81
Build id: 3.10.0-957.27.2.el7.x86_64 		0xd1
27
? insert_kthread_work
vmlinux 0xffffffffa64c2db0
Build id: 3.10.0-957.27.2.el7.x86_64 		0x40
28
ret_from_fork_nospec_begin
vmlinux 0xffffffffa6b76c37
Build id: 3.10.0-957.27.2.el7.x86_64 		0x21
29
? insert_kthread_work
vmlinux 0xffffffffa64c2db0
Build id: 3.10.0-957.27.2.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u2:2:12986]
Modules linked in: vmw_vsock_vmci_transport vsock crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul vmw_balloon glue_helper ablk_helper cryptd sg pcspkr joydev parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect ata_generic pata_acpi sysimgblt fb_sys_fops ttm serio_raw drm ata_piix vmxnet3 ahci libahci libata nfit libnvdimm vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 12986 Comm: kworker/u2:2 Kdump: loaded Tainted: G             L ------------   3.10.0-957.27.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff88d8aaa130c0 ti: ffff88d88cb84000 task.ti: ffff88d88cb84000
RIP: 0010:[<ffffffffa6b6c495>]  [<ffffffffa6b6c495>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff88d88cb87770  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffffa67a3023 RCX: ffff88d9390c1060
RDX: ffffaac4c0770000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff88d88cb87770 R08: 0000000000000000 R09: ffff88d8a0668a00
R10: ffff88d83677e0d8 R11: ffff88d83677e800 R12: 0000000000000000
R13: 0000000000000020 R14: ffff88d88cfa7900 R15: ffffffffa68da169
FS:  0000000000000000(0000) GS:ffff88d93fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9b9df12850 CR3: 00000000a1dec000 CR4: 00000000000007f0
Call Trace:
[<ffffffffc0437622>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffffa68db410>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffffa68d2ee0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffa68dc47c>] scsi_request_fn+0x4cc/0x680
[<ffffffffa67694de>] ? deadline_add_request+0x6e/0x90
[<ffffffffa6744e59>] __blk_run_queue+0x39/0x50
[<ffffffffa6748b63>] blk_queue_bio+0x3b3/0x400
[<ffffffffa6746af7>] generic_make_request+0x147/0x380
[<ffffffffa6746da0>] submit_bio+0x70/0x150
[<ffffffffa65c2c14>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc06a4471>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc06a4c74>] xfs_do_writepage+0x4a4/0x510 [xfs]
[<ffffffffa65c368c>] write_cache_pages+0x21c/0x470
[<ffffffffc06a47d0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffc06a45cb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa65c47a1>] do_writepages+0x21/0x50
[<ffffffffa6671020>] __writeback_single_inode+0x40/0x260
[<ffffffffa6671ab4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffa6671e1f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa6672653>] wb_writeback+0x263/0x2f0
[<ffffffffa6672edc>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffffa64baf9f>] process_one_work+0x17f/0x440
[<ffffffffa64bc036>] worker_thread+0x126/0x3c0
[<ffffffffa64bbf10>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffa64c2e81>] kthread+0xd1/0xe0
[<ffffffffa64c2db0>] ? insert_kthread_work+0x40/0x40
[<ffffffffa6b76c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa64c2db0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48
Complete report #308132
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc03c43be
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc044371e
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x24e
3
mptspi_qcmd
mptspi 0xffffffffc042c2e0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffff8cae2990
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffff8caec0bc
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffff8c9771ae
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffff8c9527a9
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffff8c9564b3
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffff8c954437
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffff8c9546e0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffff8c7c9794
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc05fafb1
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc05fb7a4
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffff8c7ca20c
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc05fb310
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xa0
16
? set_next_entity
vmlinux 0xffffffff8c6e103c
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x3c
17
? __switch_to
vmlinux 0xffffffff8c62b59e
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xce
18
xfs_vm_writepages
xfs 0xffffffffc05fb2db
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x6b
19
do_writepages
vmlinux 0xffffffff8c7cb261
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x21
20
__writeback_single_inode
vmlinux 0xffffffff8c87cc40
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x40
21
writeback_sb_inodes
vmlinux 0xffffffff8c87d7c4
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x1c4
22
__writeback_inodes_wb
vmlinux 0xffffffff8c87dacf
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x9f
23
wb_writeback
vmlinux 0xffffffff8c87dfa3
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x263
24
bdi_writeback_workfn
vmlinux 0xffffffff8c87eb9b
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x2cb
25
process_one_work
vmlinux 0xffffffff8c6be6bf
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x17f
26
worker_thread
vmlinux 0xffffffff8c6bf7d6
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x126
27
? manage_workers.isra.26
vmlinux 0xffffffff8c6bf6b0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x2a0
28
kthread
vmlinux 0xffffffff8c6c6691
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xd1
29
? insert_kthread_work
vmlinux 0xffffffff8c6c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffff8cd92d37
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x21
31
? insert_kthread_work
vmlinux 0xffffffff8c6c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 27s! [kworker/u8:1:24365]
Modules linked in: fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock coretemp crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg vmw_vmci parport_pc parport i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom
sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix mptspi scsi_transport_spi serio_raw mptscsih libata vmxnet3 mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 24365 Comm: kworker/u8:1 Kdump: loaded Not tainted 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/30/2013
Workqueue: writeback bdi_writeback_workfn (flush-253:1)
task: ffff9daf599741c0 ti: ffff9db01d200000 task.ti: ffff9db01d200000
RIP: 0010:[<ffffffffc03c43be>]  [<ffffffffc03c43be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff9db01d203708  EFLAGS: 00000246
RAX: ffffb45000ec0000 RBX: ffff9db036f4e000 RCX: 000000000000001f
RDX: ffff9daef5bf2400 RSI: ffff9daef5cba000 RDI: 000000000000000e
RBP: ffff9db01d203718 R08: 0000000000000002 R09: 000000015c7eafff
R10: ffff9daef5bc7800 R11: ffff9db027c98e00 R12: ffff9db032a5ef00
R13: ffff9db01d20367c R14: ffffffff8c95b0cb R15: ffff9db01d2036c0
FS:  0000000000000000(0000) GS:ffff9db03fd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff1963ea000 CR3: 00000001726b6000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc044371e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc042c2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff8cae2990>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff8caec0bc>] scsi_request_fn+0x4ac/0x680
[<ffffffff8c9771ae>] ? deadline_add_request+0x6e/0x90
[<ffffffff8c9527a9>] __blk_run_queue+0x39/0x50
[<ffffffff8c9564b3>] blk_queue_bio+0x3b3/0x400
[<ffffffff8c954437>] generic_make_request+0x147/0x380
[<ffffffff8c9546e0>] submit_bio+0x70/0x150
[<ffffffff8c7c9794>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc05fafb1>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc05fb7a4>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffff8c7ca20c>] write_cache_pages+0x21c/0x470
[<ffffffffc05fb310>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff8c6e103c>] ? set_next_entity+0x3c/0xe0
[<ffffffff8c62b59e>] ? __switch_to+0xce/0x580
[<ffffffffc05fb2db>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff8c7cb261>] do_writepages+0x21/0x50
[<ffffffff8c87cc40>] __writeback_single_inode+0x40/0x260
[<ffffffff8c87d7c4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8c87dacf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8c87dfa3>] wb_writeback+0x263/0x2f0
[<ffffffff8c87eb9b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff8c6be6bf>] process_one_work+0x17f/0x440
[<ffffffff8c6bf7d6>] worker_thread+0x126/0x3c0
[<ffffffff8c6bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8c6c6691>] kthread+0xd1/0xe0
[<ffffffff8c6c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff8cd92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8c6c65c0>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 23 3d
Complete report #287481
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc028f3be
Build id: 3.10.0-1160.el7.x86_64 		0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc02de71e
Build id: 3.10.0-1160.el7.x86_64 		0x24e
3
mptspi_qcmd
mptspi 0xffffffffc02aa2e0
Build id: 3.10.0-1160.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffff8d0e3980
Build id: 3.10.0-1160.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffff8d0ed0ac
Build id: 3.10.0-1160.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffff8cf77f8e
Build id: 3.10.0-1160.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffff8cf534d9
Build id: 3.10.0-1160.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffff8cf571e3
Build id: 3.10.0-1160.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffff8cf55167
Build id: 3.10.0-1160.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffff8cf55410
Build id: 3.10.0-1160.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffff8cdc9314
Build id: 3.10.0-1160.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc052cf31
Build id: 3.10.0-1160.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc052d724
Build id: 3.10.0-1160.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffff8cdc9d8c
Build id: 3.10.0-1160.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc052d290
Build id: 3.10.0-1160.el7.x86_64 		0xa0
16
? submit_bio
vmlinux 0xffffffff8cf55410
Build id: 3.10.0-1160.el7.x86_64 		0x70
17
xfs_vm_writepages
xfs 0xffffffffc052d25b
Build id: 3.10.0-1160.el7.x86_64 		0x6b
18
do_writepages
vmlinux 0xffffffff8cdcade1
Build id: 3.10.0-1160.el7.x86_64 		0x21
19
__writeback_single_inode
vmlinux 0xffffffff8ce7d6f0
Build id: 3.10.0-1160.el7.x86_64 		0x40
20
? wake_up_bit
vmlinux 0xffffffff8ccc6cd5
Build id: 3.10.0-1160.el7.x86_64 		0x25
21
writeback_sb_inodes
vmlinux 0xffffffff8ce7e274
Build id: 3.10.0-1160.el7.x86_64 		0x1c4
22
__writeback_inodes_wb
vmlinux 0xffffffff8ce7e57f
Build id: 3.10.0-1160.el7.x86_64 		0x9f
23
wb_writeback
vmlinux 0xffffffff8ce7ea53
Build id: 3.10.0-1160.el7.x86_64 		0x263
24
? get_nr_inodes
vmlinux 0xffffffff8ce6a67c
Build id: 3.10.0-1160.el7.x86_64 		0x4c
25
bdi_writeback_workfn
vmlinux 0xffffffff8ce7f64b
Build id: 3.10.0-1160.el7.x86_64 		0x2cb
26
process_one_work
vmlinux 0xffffffff8ccbdc4f
Build id: 3.10.0-1160.el7.x86_64 		0x17f
27
worker_thread
vmlinux 0xffffffff8ccbed66
Build id: 3.10.0-1160.el7.x86_64 		0x126
28
? manage_workers.isra.26
vmlinux 0xffffffff8ccbec40
Build id: 3.10.0-1160.el7.x86_64 		0x2a0
29
kthread
vmlinux 0xffffffff8ccc5c21
Build id: 3.10.0-1160.el7.x86_64 		0xd1
30
? insert_kthread_work
vmlinux 0xffffffff8ccc5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40
31
ret_from_fork_nospec_begin
vmlinux 0xffffffff8d393df7
Build id: 3.10.0-1160.el7.x86_64 		0x21
32
? insert_kthread_work
vmlinux 0xffffffff8ccc5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 52s! [kworker/u256:0:125171]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype tun br_netfilter devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus iosf_mbi snd_seq crc32_pclmul snd_seq_device ghash_clmulni_intel
snd_pcm ppdev aesni_intel lrw snd_timer gf128mul glue_helper vmw_balloon ablk_helper cryptd snd soundcore joydev pcspkr vmw_vmci sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm ata_piix mptspi nfit scsi_transport_spi libata libnvdimm serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks fuse
CPU: 1 PID: 125171 Comm: kworker/u256:0 Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
Workqueue: writeback bdi_writeback_workfn (flush-8:0)
task: ffff95c50cfa3180 ti: ffff95c5023f4000 task.ti: ffff95c5023f4000
RIP: 0010:[<ffffffffc028f3be>]  [<ffffffffc028f3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0000:ffff95c5023f7708  EFLAGS: 00000246
RAX: ffffbe49403c0000 RBX: ffff95c574f33000 RCX: 000000000000001a
RDX: ffff95c535ac4e00 RSI: ffff95c5358ae000 RDI: 000000000000000e
RBP: ffff95c5023f7718 R08: 0000000000000002 R09: 000000003f7e4fff
R10: ffff95c535b26800 R11: ffff95c5053f6a00 R12: ffff95c5128bc900
R13: ffff95c5023f767c R14: ffffffff8cf5bdfb R15: ffff95c5023f76c0
FS:  0000000000000000(0000) GS:ffff95c57b640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff96369a7d0 CR3: 0000000051810000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc02de71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc02aa2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff8d0e3980>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff8d0ed0ac>] scsi_request_fn+0x4ac/0x680
[<ffffffff8cf77f8e>] ? deadline_add_request+0x6e/0x90
[<ffffffff8cf534d9>] __blk_run_queue+0x39/0x50
[<ffffffff8cf571e3>] blk_queue_bio+0x3b3/0x400
[<ffffffff8cf55167>] generic_make_request+0x147/0x380
[<ffffffff8cf55410>] submit_bio+0x70/0x150
[<ffffffff8cdc9314>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc052cf31>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc052d724>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffff8cdc9d8c>] write_cache_pages+0x21c/0x470
[<ffffffffc052d290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff8cf55410>] ? submit_bio+0x70/0x150
[<ffffffffc052d25b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff8cdcade1>] do_writepages+0x21/0x50
[<ffffffff8ce7d6f0>] __writeback_single_inode+0x40/0x260
[<ffffffff8ccc6cd5>] ? wake_up_bit+0x25/0x30
[<ffffffff8ce7e274>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8ce7e57f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8ce7ea53>] wb_writeback+0x263/0x2f0
[<ffffffff8ce6a67c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8ce7f64b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff8ccbdc4f>] process_one_work+0x17f/0x440
[<ffffffff8ccbed66>] worker_thread+0x126/0x3c0
[<ffffffff8ccbec40>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8ccc5c21>] kthread+0xd1/0xe0
[<ffffffff8ccc5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffff8d393df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8ccc5b50>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 d3 29
Complete report #247262
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc03193be
Build id: 3.10.0-1160.el7.x86_64 		0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc033b71e
Build id: 3.10.0-1160.el7.x86_64 		0x24e
3
mptspi_qcmd
mptspi 0xffffffffc03342e0
Build id: 3.10.0-1160.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffffacee3980
Build id: 3.10.0-1160.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffffaceed0ac
Build id: 3.10.0-1160.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffffacd77f8e
Build id: 3.10.0-1160.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffffacd534d9
Build id: 3.10.0-1160.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffffacd571e3
Build id: 3.10.0-1160.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffffacd55167
Build id: 3.10.0-1160.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffffacd55410
Build id: 3.10.0-1160.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffffacbc9314
Build id: 3.10.0-1160.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc05bbf31
Build id: 3.10.0-1160.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc05bc724
Build id: 3.10.0-1160.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffffacbc9d8c
Build id: 3.10.0-1160.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc05bc290
Build id: 3.10.0-1160.el7.x86_64 		0xa0
16
? __switch_to
vmlinux 0xffffffffaca2b59e
Build id: 3.10.0-1160.el7.x86_64 		0xce
17
xfs_vm_writepages
xfs 0xffffffffc05bc25b
Build id: 3.10.0-1160.el7.x86_64 		0x6b
18
do_writepages
vmlinux 0xffffffffacbcade1
Build id: 3.10.0-1160.el7.x86_64 		0x21
19
__writeback_single_inode
vmlinux 0xffffffffacc7d6f0
Build id: 3.10.0-1160.el7.x86_64 		0x40
20
writeback_sb_inodes
vmlinux 0xffffffffacc7e274
Build id: 3.10.0-1160.el7.x86_64 		0x1c4
21
__writeback_inodes_wb
vmlinux 0xffffffffacc7e57f
Build id: 3.10.0-1160.el7.x86_64 		0x9f
22
wb_writeback
vmlinux 0xffffffffacc7ea53
Build id: 3.10.0-1160.el7.x86_64 		0x263
23
bdi_writeback_workfn
vmlinux 0xffffffffacc7f64b
Build id: 3.10.0-1160.el7.x86_64 		0x2cb
24
process_one_work
vmlinux 0xffffffffacabdc4f
Build id: 3.10.0-1160.el7.x86_64 		0x17f
25
worker_thread
vmlinux 0xffffffffacabed66
Build id: 3.10.0-1160.el7.x86_64 		0x126
26
? manage_workers.isra.26
vmlinux 0xffffffffacabec40
Build id: 3.10.0-1160.el7.x86_64 		0x2a0
27
kthread
vmlinux 0xffffffffacac5c21
Build id: 3.10.0-1160.el7.x86_64 		0xd1
28
? insert_kthread_work
vmlinux 0xffffffffacac5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffffad193df7
Build id: 3.10.0-1160.el7.x86_64 		0x21
30
? insert_kthread_work
vmlinux 0xffffffffacac5b50
Build id: 3.10.0-1160.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 24s! [kworker/u256:0:102932]
Modules linked in: tcp_lp nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg parport_pc parport snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm nfit drm crct10dif_pclmul libnvdimm crct10dif_common crc32c_intel ata_piix e1000 libata mptspi serio_raw scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 102932 Comm: kworker/u256:0 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff91e288a2a100 ti: ffff91e2b3ad0000 task.ti: ffff91e2b3ad0000
RIP: 0010:[<ffffffffc03193be>]  [<ffffffffc03193be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff91e2b3ad3708  EFLAGS: 00000246
RAX: ffff9dbc80440000 RBX: ffff91e2996931a0 RCX: 000000000000000b
RDX: ffff91e27545ee00 RSI: ffff91e275dc6000 RDI: 000000000000000e
RBP: ffff91e2b3ad3718 R08: 0000000000000002 R09: 000000001f2e6fff
R10: ffff91e2757f5c00 R11: ffff91e2997d79c0 R12: ffff91e242ddd980
R13: ffff91e2b3ad367c R14: ffffffffacd5bdfb R15: ffff91e2b3ad36c0
FS:  0000000000000000(0000) GS:ffff91e2bb640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7364104000 CR3: 000000004f1fa000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc033b71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc03342e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffacee3980>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffaceed0ac>] scsi_request_fn+0x4ac/0x680
[<ffffffffacd77f8e>] ? deadline_add_request+0x6e/0x90
[<ffffffffacd534d9>] __blk_run_queue+0x39/0x50
[<ffffffffacd571e3>] blk_queue_bio+0x3b3/0x400
[<ffffffffacd55167>] generic_make_request+0x147/0x380
[<ffffffffacd55410>] submit_bio+0x70/0x150
[<ffffffffacbc9314>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc05bbf31>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc05bc724>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffffacbc9d8c>] write_cache_pages+0x21c/0x470
[<ffffffffc05bc290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffaca2b59e>] ? __switch_to+0xce/0x580
[<ffffffffc05bc25b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffacbcade1>] do_writepages+0x21/0x50
[<ffffffffacc7d6f0>] __writeback_single_inode+0x40/0x260
[<ffffffffacc7e274>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffacc7e57f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffacc7ea53>] wb_writeback+0x263/0x2f0
[<ffffffffacc7f64b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffacabdc4f>] process_one_work+0x17f/0x440
[<ffffffffacabed66>] worker_thread+0x126/0x3c0
[<ffffffffacabec40>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffacac5c21>] kthread+0xd1/0xe0
[<ffffffffacac5b50>] ? insert_kthread_work+0x40/0x40
[<ffffffffad193df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffacac5b50>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 73 32
Complete report #244512
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc004c3be
Build id: 3.10.0-693.2.2.el7.x86_64 		0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc008c729
Build id: 3.10.0-693.2.2.el7.x86_64 		0x249
3
mptspi_qcmd
mptspi 0xffffffffc00452b0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffff8146561a
Build id: 3.10.0-693.2.2.el7.x86_64 		0xaa
5
scsi_request_fn
vmlinux 0xffffffff8146e3df
Build id: 3.10.0-693.2.2.el7.x86_64 		0x4df
6
__blk_run_queue
vmlinux 0xffffffff812f73e3
Build id: 3.10.0-693.2.2.el7.x86_64 		0x33
7
blk_queue_bio
vmlinux 0xffffffff812face3
Build id: 3.10.0-693.2.2.el7.x86_64 		0x363
8
generic_make_request
vmlinux 0xffffffff812f8d55
Build id: 3.10.0-693.2.2.el7.x86_64 		0x105
9
submit_bio
vmlinux 0xffffffff812f8fd0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x70
10
? __test_set_page_writeback
vmlinux 0xffffffff8118d4ca
Build id: 3.10.0-693.2.2.el7.x86_64 		0xca
11
xfs_submit_ioend.isra.16
xfs 0xffffffffc02835e1
Build id: 3.10.0-693.2.2.el7.x86_64 		0x61
12
xfs_do_writepage
xfs 0xffffffffc0283e44
Build id: 3.10.0-693.2.2.el7.x86_64 		0x4c4
13
write_cache_pages
vmlinux 0xffffffff8118dc91
Build id: 3.10.0-693.2.2.el7.x86_64 		0x251
14
? xfs_aops_discard_page
xfs 0xffffffffc0283980
Build id: 3.10.0-693.2.2.el7.x86_64 		0x150
15
? submit_bio
vmlinux 0xffffffff812f8fd0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x70
16
xfs_vm_writepages
xfs 0xffffffffc028378d
Build id: 3.10.0-693.2.2.el7.x86_64 		0xbd
17
do_writepages
vmlinux 0xffffffff8118effe
Build id: 3.10.0-693.2.2.el7.x86_64 		0x1e
18
__writeback_single_inode
vmlinux 0xffffffff8122d970
Build id: 3.10.0-693.2.2.el7.x86_64 		0x40
19
writeback_sb_inodes
vmlinux 0xffffffff8122e5b4
Build id: 3.10.0-693.2.2.el7.x86_64 		0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffff8122e91f
Build id: 3.10.0-693.2.2.el7.x86_64 		0x9f
21
wb_writeback
vmlinux 0xffffffff8122f153
Build id: 3.10.0-693.2.2.el7.x86_64 		0x263
22
bdi_writeback_workfn
vmlinux 0xffffffff8122f5fb
Build id: 3.10.0-693.2.2.el7.x86_64 		0x2cb
23
process_one_work
vmlinux 0xffffffff810a881a
Build id: 3.10.0-693.2.2.el7.x86_64 		0x17a
24
worker_thread
vmlinux 0xffffffff810a94e6
Build id: 3.10.0-693.2.2.el7.x86_64 		0x126
25
? manage_workers.isra.24
vmlinux 0xffffffff810a93c0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x2a0
26
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.2.2.el7.x86_64 		0xcf
27
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x40
28
ret_from_fork
vmlinux 0xffffffff816b4f58
Build id: 3.10.0-693.2.2.el7.x86_64 		0x58
29
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 31s! [kworker/3:2:73086]
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 31s! [kworker/u256:0:71832]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac edac_core coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul snd_ens1371 glue_helper snd_rawmidi ablk_helper snd_ac97_codec cryptd ac97_bus snd_seq snd_seq_device snd_pcm ppdev snd_timer snd nfit soundcore vmw_balloon pcspkr libnvdimm sg joydev parport_pc parport shpchp vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic
pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm ata_piix e1000 libata mptspi scsi_transport_spi mptscsih mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 71832 Comm: kworker/u256:0 Not tainted 3.10.0-693.2.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8800b6f70fd0 ti: ffff880234b1c000 task.ti: ffff880234b1c000
RIP: 0010:[<ffffffffc004c3be>]  [<ffffffffc004c3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff880234b1f6f0  EFLAGS: 00000246
RAX: ffffc90000d40000 RBX: ffff88023f8cba00 RCX: 000000000000000a
RDX: ffff880231a66a00 RSI: ffff880035dc5000 RDI: 000000000000000e
RBP: ffff880234b1f700 R08: 0000000000000002 R09: ffff880035dc60d8
R10: ffff8800355e5500 R11: ffff8800355e5500 R12: 0000000100000001
R13: ffff8800b8ebd980 R14: ffff880234b1f664 R15: ffffffff812ff34b
FS:  0000000000000000(0000) GS:ffff880235680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbedb06f000 CR3: 000000020bb77000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff880035dc5d30 000000000000003c ffff880234b1f7a8 ffffffffc008c729
ffff880035dc5008 0400000034b1f768 000000000000000a ffff8800355e5500
0000000000000028 ffff880035dc5188 000000600000000a ffff880035982bf0
Call Trace:
[<ffffffffc008c729>] mptscsih_qcmd+0x249/0x820 [mptscsih]
[<ffffffffc00452b0>] mptspi_qcmd+0x50/0xe0 [mptspi]
[<ffffffff8146561a>] scsi_dispatch_cmd+0xaa/0x230
[<ffffffff8146e3df>] scsi_request_fn+0x4df/0x680
[<ffffffff812f73e3>] __blk_run_queue+0x33/0x40
[<ffffffff812face3>] blk_queue_bio+0x363/0x3a0
[<ffffffff812f8d55>] generic_make_request+0x105/0x310
[<ffffffff812f8fd0>] submit_bio+0x70/0x150
[<ffffffff8118d4ca>] ? __test_set_page_writeback+0xca/0x160
[<ffffffffc02835e1>] xfs_submit_ioend.isra.16+0x61/0xe0 [xfs]
[<ffffffffc0283e44>] xfs_do_writepage+0x4c4/0x530 [xfs]
[<ffffffff8118dc91>] write_cache_pages+0x251/0x4d0
[<ffffffffc0283980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff812f8fd0>] ? submit_bio+0x70/0x150
[<ffffffffc028378d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffff8118effe>] do_writepages+0x1e/0x40
[<ffffffff8122d970>] __writeback_single_inode+0x40/0x220
[<ffffffff8122e5b4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff8122e91f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8122f153>] wb_writeback+0x263/0x2f0
[<ffffffff8122f5fb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f58>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 a3 05
Complete report #236282
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc01c23be
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc01e971e
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x24e
3
mptspi_qcmd
mptspi 0xffffffffc02b72e0
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffffa62e4810
Build id: 3.10.0-1160.36.2.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffffa62edf3c
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffffa6178b9e
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffffa6153ee9
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffffa6157c93
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffffa6155b77
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffffa6155e20
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffffa5fc9ec4
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc043df31
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc043e724
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffffa5fca93c
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc043e290
Build id: 3.10.0-1160.36.2.el7.x86_64 		0xa0
16
? __enqueue_entity
vmlinux 0xffffffffa5edfbb8
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x78
17
? put_prev_entity
vmlinux 0xffffffffa5ee635b
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x8b
18
xfs_vm_writepages
xfs 0xffffffffc043e25b
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x6b
19
do_writepages
vmlinux 0xffffffffa5fcb991
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x21
20
__writeback_single_inode
vmlinux 0xffffffffa607dec0
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x40
21
writeback_sb_inodes
vmlinux 0xffffffffa607ea44
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x1c4
22
__writeback_inodes_wb
vmlinux 0xffffffffa607ed4f
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x9f
23
wb_writeback
vmlinux 0xffffffffa607f223
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x263
24
bdi_writeback_workfn
vmlinux 0xffffffffa607fd1c
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x1cc
25
process_one_work
vmlinux 0xffffffffa5ebde6f
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x17f
26
worker_thread
vmlinux 0xffffffffa5ebef86
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x126
27
? manage_workers.isra.26
vmlinux 0xffffffffa5ebee60
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x2a0
28
kthread
vmlinux 0xffffffffa5ec5e41
Build id: 3.10.0-1160.36.2.el7.x86_64 		0xd1
29
? insert_kthread_work
vmlinux 0xffffffffa5ec5d70
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffffa6595df7
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x21
31
? insert_kthread_work
vmlinux 0xffffffffa5ec5d70
Build id: 3.10.0-1160.36.2.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 50s! [kworker/u8:1:7238]
Modules linked in: ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat vmw_vsock_vmci_transport vsock nf_conntrack sunrpc coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd pcspkr vmw_balloon joydev sg parport_pc i2c_piix4 parport vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul ata_piix crct10dif_common drm mptspi e1000 crc32c_intel scsi_transport_spi mptscsih serio_raw libata mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 7238 Comm: kworker/u8:1 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.36.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/30/2013
Workqueue: writeback bdi_writeback_workfn (flush-253:1)
task: ffff88112438a100 ti: ffff881177cac000 task.ti: ffff881177cac000
RIP: 0010:[<ffffffffc01c23be>]  [<ffffffffc01c23be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff881177caf708  EFLAGS: 00000246
RAX: ffff9d6bc0480000 RBX: ffff881176f83000 RCX: 000000000000001f
RDX: ffff881135f54000 RSI: ffff881135e77000 RDI: 000000000000000e
RBP: ffff881177caf718 R08: 0000000000000002 R09: 000000006b7b0fff
R10: ffff881135425000 R11: ffff8811264b1180 R12: ffff881137dc7800
R13: ffff881177caf67c R14: ffffffffa615c8ab R15: ffff881177caf6c0
FS:  0000000000000000(0000) GS:ffff88117fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f04c7d17000 CR3: 0000000076c50000 CR4: 00000000000407e0
Call Trace:
[<ffffffffc01e971e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc02b72e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffa62e4810>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffa62edf3c>] scsi_request_fn+0x4ac/0x680
[<ffffffffa6178b9e>] ? deadline_add_request+0x6e/0x90
[<ffffffffa6153ee9>] __blk_run_queue+0x39/0x50
[<ffffffffa6157c93>] blk_queue_bio+0x3b3/0x400
[<ffffffffa6155b77>] generic_make_request+0x147/0x380
[<ffffffffa6155e20>] submit_bio+0x70/0x150
[<ffffffffa5fc9ec4>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc043df31>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc043e724>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffffa5fca93c>] write_cache_pages+0x21c/0x470
[<ffffffffc043e290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffa5edfbb8>] ? __enqueue_entity+0x78/0x80
[<ffffffffa5ee635b>] ? put_prev_entity+0x8b/0x400
[<ffffffffc043e25b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffa5fcb991>] do_writepages+0x21/0x50
[<ffffffffa607dec0>] __writeback_single_inode+0x40/0x260
[<ffffffffa607ea44>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffa607ed4f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffa607f223>] wb_writeback+0x263/0x2f0
[<ffffffffa607fd1c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffffa5ebde6f>] process_one_work+0x17f/0x440
[<ffffffffa5ebef86>] worker_thread+0x126/0x3c0
[<ffffffffa5ebee60>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffa5ec5e41>] kthread+0xd1/0xe0
[<ffffffffa5ec5d70>] ? insert_kthread_work+0x40/0x40
[<ffffffffa6595df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffa5ec5d70>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 03 1d
Complete report #233698
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffb8f8b4d5
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x15
2
pvscsi_queue
vmw_pvscsi 0xffffffffc0248632
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x4a2
3
? scsi_test_unit_ready
vmlinux 0xffffffffb8cec6b0
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x170
4
scsi_dispatch_cmd
vmlinux 0xffffffffb8ce3fa0
Build id: 3.10.0-1160.24.1.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffffb8ced6cc
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffffb8b7853e
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffffb8b53939
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffffb8b57743
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffffb8b555c7
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffffb8b55870
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffffb89c9534
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc0483f31
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc0484724
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffffb89c9fac
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc0484290
Build id: 3.10.0-1160.24.1.el7.x86_64 		0xa0
16
? __enqueue_entity
vmlinux 0xffffffffb88dfd08
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x78
17
? put_prev_entity
vmlinux 0xffffffffb88e622b
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x8b
18
? __switch_to
vmlinux 0xffffffffb882b59e
Build id: 3.10.0-1160.24.1.el7.x86_64 		0xce
19
xfs_vm_writepages
xfs 0xffffffffc048425b
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x6b
20
do_writepages
vmlinux 0xffffffffb89cb001
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x21
21
__writeback_single_inode
vmlinux 0xffffffffb8a7d9a0
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x40
22
? wake_up_bit
vmlinux 0xffffffffb88c6e55
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x25
23
writeback_sb_inodes
vmlinux 0xffffffffb8a7e524
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x1c4
24
__writeback_inodes_wb
vmlinux 0xffffffffb8a7e82f
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x9f
25
wb_writeback
vmlinux 0xffffffffb8a7ed03
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x263
26
bdi_writeback_workfn
vmlinux 0xffffffffb8a7f8fb
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x2cb
27
process_one_work
vmlinux 0xffffffffb88bddcf
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x17f
28
worker_thread
vmlinux 0xffffffffb88beee6
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x126
29
? manage_workers.isra.26
vmlinux 0xffffffffb88bedc0
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x2a0
30
kthread
vmlinux 0xffffffffb88c5da1
Build id: 3.10.0-1160.24.1.el7.x86_64 		0xd1
31
? insert_kthread_work
vmlinux 0xffffffffb88c5cd0
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x40
32
ret_from_fork_nospec_begin
vmlinux 0xffffffffb8f95df7
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x21
33
? insert_kthread_work
vmlinux 0xffffffffb88c5cd0
Build id: 3.10.0-1160.24.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 21s! [kworker/u4:31:396]
Modules linked in: tcp_diag inet_diag talpa_vfshook(OE) talpa_pedconnector(OE) talpa_pedevice(OE) talpa_vcdevice(OE) talpa_core(OE) talpa_linux(OE) talpa_syscallhook(OE) vmw_vsock_vmci_transport vsock nf_conntrack_ipv4 nf_defrag_ipv4 xt_owner xt_conntrack nf_conntrack xt_multiport iptable_filter sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd joydev pcspkr sg vmw_vmci parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt crc_t10dif fb_sys_fops crct10dif_generic ttm drm crct10dif_pclmul crct10dif_common ahci crc32c_intel libahci serio_raw vmxnet3 ata_piix vmw_pvscsi libata nfit libnvdimm drm_panel_orientation_quirks
dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 396 Comm: kworker/u4:31 Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.24.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff90bfb5f35280 ti: ffff90bfb5f4c000 task.ti: ffff90bfb5f4c000
RIP: 0010:[<ffffffffb8f8b4d5>]  [<ffffffffb8f8b4d5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff90bfb5f4f770  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffffb8bb28d3 RCX: ffff90c0efbb1060
RDX: ffffa9d080ea0000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff90bfb5f4f770 R08: 0000000000000000 R09: ffff90c16ef00100
R10: ffff90c1b4e69800 R11: ffff90c1b5d96000 R12: 0000000000000000
R13: 0000000000000020 R14: ffff90c0b5a37980 R15: ffffffffb8ceb369
FS:  0000000000000000(0000) GS:ffff90c1bfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000560f1ef9e3f0 CR3: 000000017939e000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc0248632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffffb8cec6b0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffffb8ce3fa0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb8ced6cc>] scsi_request_fn+0x4ac/0x680
[<ffffffffb8b7853e>] ? deadline_add_request+0x6e/0x90
[<ffffffffb8b53939>] __blk_run_queue+0x39/0x50
[<ffffffffb8b57743>] blk_queue_bio+0x3b3/0x400
[<ffffffffb8b555c7>] generic_make_request+0x147/0x380
[<ffffffffb8b55870>] submit_bio+0x70/0x150
[<ffffffffb89c9534>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc0483f31>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc0484724>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffffb89c9fac>] write_cache_pages+0x21c/0x470
[<ffffffffc0484290>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffb88dfd08>] ? __enqueue_entity+0x78/0x80
[<ffffffffb88e622b>] ? put_prev_entity+0x8b/0x400
[<ffffffffb882b59e>] ? __switch_to+0xce/0x580
[<ffffffffc048425b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffb89cb001>] do_writepages+0x21/0x50
[<ffffffffb8a7d9a0>] __writeback_single_inode+0x40/0x260
[<ffffffffb88c6e55>] ? wake_up_bit+0x25/0x30
[<ffffffffb8a7e524>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffb8a7e82f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffb8a7ed03>] wb_writeback+0x263/0x2f0
[<ffffffffb8a7f8fb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffb88bddcf>] process_one_work+0x17f/0x440
[<ffffffffb88beee6>] worker_thread+0x126/0x3c0
[<ffffffffb88bedc0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb88c5da1>] kthread+0xd1/0xe0
[<ffffffffb88c5cd0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb8f95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb88c5cd0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #225290
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc00743be
Build id: 3.10.0-693.2.2.el7.x86_64 		0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc00d5729
Build id: 3.10.0-693.2.2.el7.x86_64 		0x249
3
mptspi_qcmd
mptspi 0xffffffffc015f2b0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffff8146561a
Build id: 3.10.0-693.2.2.el7.x86_64 		0xaa
5
scsi_request_fn
vmlinux 0xffffffff8146e3df
Build id: 3.10.0-693.2.2.el7.x86_64 		0x4df
6
__blk_run_queue
vmlinux 0xffffffff812f73e3
Build id: 3.10.0-693.2.2.el7.x86_64 		0x33
7
blk_queue_bio
vmlinux 0xffffffff812face3
Build id: 3.10.0-693.2.2.el7.x86_64 		0x363
8
generic_make_request
vmlinux 0xffffffff812f8d55
Build id: 3.10.0-693.2.2.el7.x86_64 		0x105
9
submit_bio
vmlinux 0xffffffff812f8fd0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x70
10
? __test_set_page_writeback
vmlinux 0xffffffff8118d4ca
Build id: 3.10.0-693.2.2.el7.x86_64 		0xca
11
xfs_submit_ioend.isra.16
xfs 0xffffffffc02845e1
Build id: 3.10.0-693.2.2.el7.x86_64 		0x61
12
xfs_do_writepage
xfs 0xffffffffc0284e44
Build id: 3.10.0-693.2.2.el7.x86_64 		0x4c4
13
write_cache_pages
vmlinux 0xffffffff8118dc91
Build id: 3.10.0-693.2.2.el7.x86_64 		0x251
14
? xfs_aops_discard_page
xfs 0xffffffffc0284980
Build id: 3.10.0-693.2.2.el7.x86_64 		0x150
15
? dequeue_entity
vmlinux 0xffffffff810cdc7c
Build id: 3.10.0-693.2.2.el7.x86_64 		0x11c
16
xfs_vm_writepages
xfs 0xffffffffc028478d
Build id: 3.10.0-693.2.2.el7.x86_64 		0xbd
17
do_writepages
vmlinux 0xffffffff8118effe
Build id: 3.10.0-693.2.2.el7.x86_64 		0x1e
18
__writeback_single_inode
vmlinux 0xffffffff8122d970
Build id: 3.10.0-693.2.2.el7.x86_64 		0x40
19
writeback_sb_inodes
vmlinux 0xffffffff8122e5b4
Build id: 3.10.0-693.2.2.el7.x86_64 		0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffff8122e91f
Build id: 3.10.0-693.2.2.el7.x86_64 		0x9f
21
wb_writeback
vmlinux 0xffffffff8122f153
Build id: 3.10.0-693.2.2.el7.x86_64 		0x263
22
? get_nr_inodes
vmlinux 0xffffffff8121bddc
Build id: 3.10.0-693.2.2.el7.x86_64 		0x4c
23
bdi_writeback_workfn
vmlinux 0xffffffff8122f5fb
Build id: 3.10.0-693.2.2.el7.x86_64 		0x2cb
24
process_one_work
vmlinux 0xffffffff810a881a
Build id: 3.10.0-693.2.2.el7.x86_64 		0x17a
25
worker_thread
vmlinux 0xffffffff810a94e6
Build id: 3.10.0-693.2.2.el7.x86_64 		0x126
26
? manage_workers.isra.24
vmlinux 0xffffffff810a93c0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x2a0
27
kthread
vmlinux 0xffffffff810b098f
Build id: 3.10.0-693.2.2.el7.x86_64 		0xcf
28
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x40
29
ret_from_fork
vmlinux 0xffffffff816b4f58
Build id: 3.10.0-693.2.2.el7.x86_64 		0x58
30
? insert_kthread_work
vmlinux 0xffffffff810b08c0
Build id: 3.10.0-693.2.2.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 103s! [kworker/u256:2:10771]
Modules linked in: fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ext4 mbcache jbd2 ppdev pcspkr vmw_balloon sg joydev parport_pc parport shpchp vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_common ata_generic
pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect crc32c_intel sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi drm mptscsih serio_raw ata_piix libata mptbase vmxnet3 i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 10771 Comm: kworker/u256:2 Tainted: G             L ------------   3.10.0-693.2.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff88018fedcf10 ti: ffff8801c3afc000 task.ti: ffff8801c3afc000
RIP: 0010:[<ffffffffc00743be>]  [<ffffffffc00743be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff8801c3aff6f0  EFLAGS: 00000246
RAX: ffffc90000d40000 RBX: ffff880231191500 RCX: 0000000000000002
RDX: ffff88023110c400 RSI: ffff880231f4d000 RDI: 000000000000000e
RBP: ffff8801c3aff700 R08: 0000000000000002 R09: ffff880231f4f0d8
R10: ffff880002efd880 R11: ffff880002efd880 R12: 0000000100000001
R13: ffff880035dedf80 R14: ffff8801c3aff664 R15: ffffffff812ff34b
FS:  0000000000000000(0000) GS:ffff8802356c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f9ef89f3000 CR3: 00000000b9a1a000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff880231f4dd30 000000000000003c ffff8801c3aff7a8 ffffffffc00d5729
ffff880231f4d008 04000000c3aff768 0000000000000002 ffff880002efd880
0000000000000008 ffff880231f4d188 0000006000000002 ffff8802313828f0
Call Trace:
[<ffffffffc00d5729>] mptscsih_qcmd+0x249/0x820 [mptscsih]
[<ffffffffc015f2b0>] mptspi_qcmd+0x50/0xe0 [mptspi]
[<ffffffff8146561a>] scsi_dispatch_cmd+0xaa/0x230
[<ffffffff8146e3df>] scsi_request_fn+0x4df/0x680
[<ffffffff812f73e3>] __blk_run_queue+0x33/0x40
[<ffffffff812face3>] blk_queue_bio+0x363/0x3a0
[<ffffffff812f8d55>] generic_make_request+0x105/0x310
[<ffffffff812f8fd0>] submit_bio+0x70/0x150
[<ffffffff8118d4ca>] ? __test_set_page_writeback+0xca/0x160
[<ffffffffc02845e1>] xfs_submit_ioend.isra.16+0x61/0xe0 [xfs]
[<ffffffffc0284e44>] xfs_do_writepage+0x4c4/0x530 [xfs]
[<ffffffff8118dc91>] write_cache_pages+0x251/0x4d0
[<ffffffffc0284980>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffff810cdc7c>] ? dequeue_entity+0x11c/0x5d0
[<ffffffffc028478d>] xfs_vm_writepages+0xbd/0xe0 [xfs]
[<ffffffff8118effe>] do_writepages+0x1e/0x40
[<ffffffff8122d970>] __writeback_single_inode+0x40/0x220
[<ffffffff8122e5b4>] writeback_sb_inodes+0x1c4/0x490
[<ffffffff8122e91f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8122f153>] wb_writeback+0x263/0x2f0
[<ffffffff8121bddc>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8122f5fb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff810a881a>] process_one_work+0x17a/0x440
[<ffffffff810a94e6>] worker_thread+0x126/0x3c0
[<ffffffff810a93c0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b098f>] kthread+0xcf/0xe0
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4f58>] ret_from_fork+0x58/0x90
[<ffffffff810b08c0>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 23 08
Complete report #133719
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffbd7883a5
Build id: 3.10.0-1127.el7.x86_64 		0x15
2
pvscsi_queue
vmw_pvscsi 0xffffffffc0302632
Build id: 3.10.0-1127.el7.x86_64 		0x4a2
3
? scsi_test_unit_ready
vmlinux 0xffffffffbd4eaee0
Build id: 3.10.0-1127.el7.x86_64 		0x170
4
scsi_dispatch_cmd
vmlinux 0xffffffffbd4e2850
Build id: 3.10.0-1127.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffffbd4ebefc
Build id: 3.10.0-1127.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffffbd37709e
Build id: 3.10.0-1127.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffffbd3526f9
Build id: 3.10.0-1127.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffffbd356403
Build id: 3.10.0-1127.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffffbd354387
Build id: 3.10.0-1127.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffffbd354630
Build id: 3.10.0-1127.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffffbd1c96e4
Build id: 3.10.0-1127.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc056a071
Build id: 3.10.0-1127.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc056a864
Build id: 3.10.0-1127.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffffbd1ca15c
Build id: 3.10.0-1127.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc056a3d0
Build id: 3.10.0-1127.el7.x86_64 		0xa0
16
xfs_vm_writepages
xfs 0xffffffffc056a39b
Build id: 3.10.0-1127.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffffbd1cb1b1
Build id: 3.10.0-1127.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffffbd27cbd0
Build id: 3.10.0-1127.el7.x86_64 		0x40
19
? wake_up_bit
vmlinux 0xffffffffbd0c7745
Build id: 3.10.0-1127.el7.x86_64 		0x25
20
writeback_sb_inodes
vmlinux 0xffffffffbd27d764
Build id: 3.10.0-1127.el7.x86_64 		0x1c4
21
__writeback_inodes_wb
vmlinux 0xffffffffbd27da6f
Build id: 3.10.0-1127.el7.x86_64 		0x9f
22
wb_writeback
vmlinux 0xffffffffbd27df53
Build id: 3.10.0-1127.el7.x86_64 		0x263
23
bdi_writeback_workfn
vmlinux 0xffffffffbd27eb4b
Build id: 3.10.0-1127.el7.x86_64 		0x2cb
24
process_one_work
vmlinux 0xffffffffbd0be6bf
Build id: 3.10.0-1127.el7.x86_64 		0x17f
25
worker_thread
vmlinux 0xffffffffbd0bf7d6
Build id: 3.10.0-1127.el7.x86_64 		0x126
26
? manage_workers.isra.26
vmlinux 0xffffffffbd0bf6b0
Build id: 3.10.0-1127.el7.x86_64 		0x2a0
27
kthread
vmlinux 0xffffffffbd0c6691
Build id: 3.10.0-1127.el7.x86_64 		0xd1
28
? insert_kthread_work
vmlinux 0xffffffffbd0c65c0
Build id: 3.10.0-1127.el7.x86_64 		0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffffbd792d37
Build id: 3.10.0-1127.el7.x86_64 		0x21
30
? insert_kthread_work
vmlinux 0xffffffffbd0c65c0
Build id: 3.10.0-1127.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 24s! [kworker/u2:0:8656]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables binfmt_misc nls_utf8 isofs fuse tun devlink bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd pcspkr vmw_balloon joydev sg parport_pc parport i2c_piix4 vmw_vmci xfs libcrc32c sr_mod cdrom
vmwgfx drm_kms_helper sd_mod ahci ata_generic pata_acpi libahci crc_t10dif syscopyarea sysfillrect crct10dif_generic sysimgblt fb_sys_fops ttm ata_piix drm crct10dif_pclmul crct10dif_common crc32c_intel vmxnet3 nfit libnvdimm libata serio_raw vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 0 PID: 8656 Comm: kworker/u2:0 Kdump: loaded Tainted: G             L ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9ec9820120e0 ti: ffff9ec982434000 task.ti: ffff9ec982434000
RIP: 0010:[<ffffffffbd7883a5>]  [<ffffffffbd7883a5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9ec982437770  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffffbd3b13b3 RCX: ffffffffbdc34e40
RDX: ffffaa47c0790000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff9ec982437770 R08: 0000000000000001 R09: 0000000000000000
R10: ffff9ec9b1674400 R11: ffff9ec8b5fc0000 R12: 0000000000000000
R13: 0000000000000020 R14: ffff9ec8be71cc00 R15: ffffffffbd4e9b99
FS:  0000000000000000(0000) GS:ffff9ec9bcc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f35a1970000 CR3: 00000000b0a22000 CR4: 00000000000607f0
Call Trace:
[<ffffffffc0302632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffffbd4eaee0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffffbd4e2850>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffbd4ebefc>] scsi_request_fn+0x4ac/0x660
[<ffffffffbd37709e>] ? deadline_add_request+0x6e/0x90
[<ffffffffbd3526f9>] __blk_run_queue+0x39/0x50
[<ffffffffbd356403>] blk_queue_bio+0x3b3/0x400
[<ffffffffbd354387>] generic_make_request+0x147/0x380
[<ffffffffbd354630>] submit_bio+0x70/0x150
[<ffffffffbd1c96e4>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc056a071>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc056a864>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffffbd1ca15c>] write_cache_pages+0x21c/0x470
[<ffffffffc056a3d0>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffc056a39b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffbd1cb1b1>] do_writepages+0x21/0x50
[<ffffffffbd27cbd0>] __writeback_single_inode+0x40/0x260
[<ffffffffbd0c7745>] ? wake_up_bit+0x25/0x30
[<ffffffffbd27d764>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffbd27da6f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffbd27df53>] wb_writeback+0x263/0x2f0
[<ffffffffbd27eb4b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffbd0be6bf>] process_one_work+0x17f/0x440
[<ffffffffbd0bf7d6>] worker_thread+0x126/0x3c0
[<ffffffffbd0bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffbd0c6691>] kthread+0xd1/0xe0
[<ffffffffbd0c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffbd792d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffbd0c65c0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48
Complete report #115987
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff929883c5
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x15
2
pvscsi_queue
vmw_pvscsi 0xffffffffc024a632
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x4a2
3
? scsi_test_unit_ready
vmlinux 0xffffffff926eb0a0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x170
4
scsi_dispatch_cmd
vmlinux 0xffffffff926e2990
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffff926ec0bc
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffff925771ae
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffff925527a9
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffff925564b3
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffff92554437
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffff925546e0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffff923c9794
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc04dffb1
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc04e07a4
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffff923ca20c
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc04e0310
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xa0
16
? __schedule
vmlinux 0xffffffff92985850
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x310
17
? __schedule
vmlinux 0xffffffff92985844
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x304
18
? __schedule
vmlinux 0xffffffff92985850
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x310
19
? __schedule
vmlinux 0xffffffff92985844
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x304
20
? __schedule
vmlinux 0xffffffff92985850
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x310
21
? __switch_to
vmlinux 0xffffffff9222b59e
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xce
22
xfs_vm_writepages
xfs 0xffffffffc04e02db
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x6b
23
do_writepages
vmlinux 0xffffffff923cb261
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x21
24
__writeback_single_inode
vmlinux 0xffffffff9247cc40
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x40
25
writeback_sb_inodes
vmlinux 0xffffffff9247d7c4
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x1c4
26
__writeback_inodes_wb
vmlinux 0xffffffff9247dacf
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x9f
27
wb_writeback
vmlinux 0xffffffff9247dfa3
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x263
28
bdi_writeback_workfn
vmlinux 0xffffffff9247eb9b
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x2cb
29
process_one_work
vmlinux 0xffffffff922be6bf
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x17f
30
worker_thread
vmlinux 0xffffffff922bf7d6
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x126
31
? manage_workers.isra.26
vmlinux 0xffffffff922bf6b0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x2a0
32
kthread
vmlinux 0xffffffff922c6691
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xd1
33
? insert_kthread_work
vmlinux 0xffffffff922c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x40
34
ret_from_fork_nospec_begin
vmlinux 0xffffffff92992d37
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x21
35
? insert_kthread_work
vmlinux 0xffffffff922c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [kworker/u4:0:14945]
Modules linked in: vsock_diag tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter ipmi_devintf ipmi_msghandler vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper sg ppdev ablk_helper cryptd pcspkr vmw_balloon joydev parport_pc i2c_piix4 parport vmw_vmci binfmt_misc ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci nfit libnvdimm
drm e1000e ata_piix libahci crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw ptp pps_core vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 14945 Comm: kworker/u4:0 Kdump: loaded Tainted: G             L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8d238cede2a0 ti: ffff8d2175700000 task.ti: ffff8d2175700000
RIP: 0010:[<ffffffff929883c5>]  [<ffffffff929883c5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8d2175703770  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffff925b14b3 RCX: ffffffff92e34e40
RDX: ffffb47e01b90000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff8d2175703770 R08: 0000000000000001 R09: 0000000000000000
R10: ffff8d256932f000 R11: ffff8d22c4bf5800 R12: 0000000000000000
R13: 0000000000000020 R14: ffff8d256dfa2000 R15: ffffffff926e9d59
FS:  0000000000000000(0000) GS:ffff8d257fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fdb68fbe000 CR3: 000000000b674000 CR4: 00000000003607f0
Call Trace:
[<ffffffffc024a632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffff926eb0a0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffff926e2990>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff926ec0bc>] scsi_request_fn+0x4ac/0x680
[<ffffffff925771ae>] ? deadline_add_request+0x6e/0x90
[<ffffffff925527a9>] __blk_run_queue+0x39/0x50
[<ffffffff925564b3>] blk_queue_bio+0x3b3/0x400
[<ffffffff92554437>] generic_make_request+0x147/0x380
[<ffffffff925546e0>] submit_bio+0x70/0x150
[<ffffffff923c9794>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc04dffb1>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc04e07a4>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffff923ca20c>] write_cache_pages+0x21c/0x470
[<ffffffffc04e0310>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff92985850>] ? __schedule+0x310/0x840
[<ffffffff92985844>] ? __schedule+0x304/0x840
[<ffffffff92985850>] ? __schedule+0x310/0x840
[<ffffffff92985844>] ? __schedule+0x304/0x840
[<ffffffff92985850>] ? __schedule+0x310/0x840
[<ffffffff9222b59e>] ? __switch_to+0xce/0x580
[<ffffffffc04e02db>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff923cb261>] do_writepages+0x21/0x50
[<ffffffff9247cc40>] __writeback_single_inode+0x40/0x260
[<ffffffff9247d7c4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff9247dacf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff9247dfa3>] wb_writeback+0x263/0x2f0
[<ffffffff9247eb9b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff922be6bf>] process_one_work+0x17f/0x440
[<ffffffff922bf7d6>] worker_thread+0x126/0x3c0
[<ffffffff922bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff922c6691>] kthread+0xd1/0xe0
[<ffffffff922c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff92992d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff922c65c0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #24347
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff87f883c5
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x15
2
pvscsi_queue
vmw_pvscsi 0xffffffffc0062632
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x4a2
3
? scsi_test_unit_ready
vmlinux 0xffffffff87ceb0a0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x170
4
scsi_dispatch_cmd
vmlinux 0xffffffff87ce2990
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffff87cec0bc
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffff87b771ae
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffff87b527a9
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffff87b564b3
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffff87b54437
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffff87b546e0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffff879c9794
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc028cfb1
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc028d7a4
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffff879ca20c
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc028d310
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xa0
16
xfs_vm_writepages
xfs 0xffffffffc028d2db
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffff879cb261
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffff87a7cc40
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x40
19
writeback_sb_inodes
vmlinux 0xffffffff87a7d7c4
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x1c4
20
__writeback_inodes_wb
vmlinux 0xffffffff87a7dacf
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x9f
21
wb_writeback
vmlinux 0xffffffff87a7dfa3
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x263
22
? get_nr_inodes
vmlinux 0xffffffff87a69c5c
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x4c
23
bdi_writeback_workfn
vmlinux 0xffffffff87a7eb9b
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x2cb
24
process_one_work
vmlinux 0xffffffff878be6bf
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x17f
25
worker_thread
vmlinux 0xffffffff878bf7d6
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x126
26
? manage_workers.isra.26
vmlinux 0xffffffff878bf6b0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x2a0
27
kthread
vmlinux 0xffffffff878c6691
Build id: 3.10.0-1127.19.1.el7.x86_64 		0xd1
28
? insert_kthread_work
vmlinux 0xffffffff878c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffff87f92d37
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x21
30
? insert_kthread_work
vmlinux 0xffffffff878c65c0
Build id: 3.10.0-1127.19.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [kworker/u4:0:1939]
Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ppdev vmw_balloon pcspkr joydev sg vmw_vmci parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect crc_t10dif sysimgblt fb_sys_fops crct10dif_generic ttm crct10dif_common drm ahci ata_piix libahci libata crc32c_intel serio_raw vmxnet3
vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 1939 Comm: kworker/u4:0 Kdump: loaded Tainted: G        W    L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9c10b682c1c0 ti: ffff9c1016334000 task.ti: ffff9c1016334000
RIP: 0010:[<ffffffff87f883c5>]  [<ffffffff87f883c5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9c1016337770  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffff87bb14b3 RCX: ffff9c10b9097060
RDX: ffffb1a300770000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff9c1016337770 R08: 0000000000000000 R09: ffff9c10b7e6d300
R10: ffff9c0fb6712c00 R11: ffff9c0fb661f000 R12: 0000000000000000
R13: 0000000000000020 R14: ffff9c102c70af00 R15: ffffffff87ce9d59
FS:  0000000000000000(0000) GS:ffff9c10bfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fea6b1ca000 CR3: 00000000b28b8000 CR4: 00000000000007f0
Call Trace:
[<ffffffffc0062632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffff87ceb0a0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffff87ce2990>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff87cec0bc>] scsi_request_fn+0x4ac/0x680
[<ffffffff87b771ae>] ? deadline_add_request+0x6e/0x90
[<ffffffff87b527a9>] __blk_run_queue+0x39/0x50
[<ffffffff87b564b3>] blk_queue_bio+0x3b3/0x400
[<ffffffff87b54437>] generic_make_request+0x147/0x380
[<ffffffff87b546e0>] submit_bio+0x70/0x150
[<ffffffff879c9794>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc028cfb1>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc028d7a4>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffff879ca20c>] write_cache_pages+0x21c/0x470
[<ffffffffc028d310>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffc028d2db>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff879cb261>] do_writepages+0x21/0x50
[<ffffffff87a7cc40>] __writeback_single_inode+0x40/0x260
[<ffffffff87a7d7c4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff87a7dacf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff87a7dfa3>] wb_writeback+0x263/0x2f0
[<ffffffff87a69c5c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff87a7eb9b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff878be6bf>] process_one_work+0x17f/0x440
[<ffffffff878bf7d6>] worker_thread+0x126/0x3c0
[<ffffffff878bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff878c6691>] kthread+0xd1/0xe0
[<ffffffff878c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff87f92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff878c65c0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48
Complete report #16500
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffaf36a175
Build id: 3.10.0-957.1.3.el7.x86_64 		0x15
2
pvscsi_queue
vmw_pvscsi 0xffffffffc04cc622
Build id: 3.10.0-957.1.3.el7.x86_64 		0x4a2
3
? scsi_test_unit_ready
vmlinux 0xffffffffaf0d9100
Build id: 3.10.0-957.1.3.el7.x86_64 		0x170
4
scsi_dispatch_cmd
vmlinux 0xffffffffaf0d0bd0
Build id: 3.10.0-957.1.3.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffffaf0da16c
Build id: 3.10.0-957.1.3.el7.x86_64 		0x4cc
6
? deadline_add_request
vmlinux 0xffffffffaef6786e
Build id: 3.10.0-957.1.3.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffffaef43269
Build id: 3.10.0-957.1.3.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffffaef46f63
Build id: 3.10.0-957.1.3.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffffaef44ef7
Build id: 3.10.0-957.1.3.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffffaef451a0
Build id: 3.10.0-957.1.3.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffffaedc17f4
Build id: 3.10.0-957.1.3.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc0688251
Build id: 3.10.0-957.1.3.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc0688a54
Build id: 3.10.0-957.1.3.el7.x86_64 		0x4a4
14
write_cache_pages
vmlinux 0xffffffffaedc22a4
Build id: 3.10.0-957.1.3.el7.x86_64 		0x254
15
? xfs_aops_discard_page
xfs 0xffffffffc06885b0
Build id: 3.10.0-957.1.3.el7.x86_64 		0x150
16
? submit_bio
vmlinux 0xffffffffaef451a0
Build id: 3.10.0-957.1.3.el7.x86_64 		0x70
17
xfs_vm_writepages
xfs 0xffffffffc06883ab
Build id: 3.10.0-957.1.3.el7.x86_64 		0x6b
18
do_writepages
vmlinux 0xffffffffaedc33f1
Build id: 3.10.0-957.1.3.el7.x86_64 		0x21
19
__writeback_single_inode
vmlinux 0xffffffffaee6fba0
Build id: 3.10.0-957.1.3.el7.x86_64 		0x40
20
writeback_sb_inodes
vmlinux 0xffffffffaee70634
Build id: 3.10.0-957.1.3.el7.x86_64 		0x1c4
21
__writeback_inodes_wb
vmlinux 0xffffffffaee7099f
Build id: 3.10.0-957.1.3.el7.x86_64 		0x9f
22
wb_writeback
vmlinux 0xffffffffaee711d3
Build id: 3.10.0-957.1.3.el7.x86_64 		0x263
23
bdi_writeback_workfn
vmlinux 0xffffffffaee71a5c
Build id: 3.10.0-957.1.3.el7.x86_64 		0x1cc
24
process_one_work
vmlinux 0xffffffffaecb9d4f
Build id: 3.10.0-957.1.3.el7.x86_64 		0x17f
25
worker_thread
vmlinux 0xffffffffaecbade6
Build id: 3.10.0-957.1.3.el7.x86_64 		0x126
26
? manage_workers.isra.25
vmlinux 0xffffffffaecbacc0
Build id: 3.10.0-957.1.3.el7.x86_64 		0x2a0
27
kthread
vmlinux 0xffffffffaecc1c31
Build id: 3.10.0-957.1.3.el7.x86_64 		0xd1
28
? insert_kthread_work
vmlinux 0xffffffffaecc1b60
Build id: 3.10.0-957.1.3.el7.x86_64 		0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffffaf374c1d
Build id: 3.10.0-957.1.3.el7.x86_64 		0x7
30
? insert_kthread_work
vmlinux 0xffffffffaecc1b60
Build id: 3.10.0-957.1.3.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 28s! [kworker/u4:2:25513]
Modules linked in: arc4 md4 nls_utf8 cifs ccm dns_resolver binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm vmxnet3 drm ahci vmw_pvscsi libahci drm_panel_orientation_quirks
ata_generic pata_acpi ata_piix nfit libata libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 25513 Comm: kworker/u4:2 Kdump: loaded Tainted: G             L ------------   3.10.0-957.1.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
Workqueue: writeback bdi_writeback_workfn (flush-253:1)
task: ffff91a8ab7f5140 ti: ffff91a521908000 task.ti: ffff91a521908000
RIP: 0010:[<ffffffffaf36a175>]  [<ffffffffaf36a175>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff91a52190b760  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffffaefa1363 RCX: ffff91a8bfca3060
RDX: ffffb12181b80000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff91a52190b760 R08: 0000000000000000 R09: ffff91a60e9b7e00
R10: ffff91a8aa1fb0d8 R11: ffff91a8aa1fb800 R12: 0000000000000000
R13: 0000000000000020 R14: ffff91a75f372000 R15: ffffffffaf0d7e59
FS:  0000000000000000(0000) GS:ffff91a8bfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007feceb744080 CR3: 000000041d704000 CR4: 00000000007607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffffc04cc622>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffffaf0d9100>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffffaf0d0bd0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffaf0da16c>] scsi_request_fn+0x4cc/0x680
[<ffffffffaef6786e>] ? deadline_add_request+0x6e/0x90
[<ffffffffaef43269>] __blk_run_queue+0x39/0x50
[<ffffffffaef46f63>] blk_queue_bio+0x3b3/0x400
[<ffffffffaef44ef7>] generic_make_request+0x147/0x380
[<ffffffffaef451a0>] submit_bio+0x70/0x150
[<ffffffffaedc17f4>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc0688251>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc0688a54>] xfs_do_writepage+0x4a4/0x510 [xfs]
[<ffffffffaedc22a4>] write_cache_pages+0x254/0x4e0
[<ffffffffc06885b0>] ? xfs_aops_discard_page+0x150/0x150 [xfs]
[<ffffffffaef451a0>] ? submit_bio+0x70/0x150
[<ffffffffc06883ab>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffaedc33f1>] do_writepages+0x21/0x50
[<ffffffffaee6fba0>] __writeback_single_inode+0x40/0x260
[<ffffffffaee70634>] writeback_sb_inodes+0x1c4/0x490
[<ffffffffaee7099f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffaee711d3>] wb_writeback+0x263/0x2f0
[<ffffffffaee71a5c>] bdi_writeback_workfn+0x1cc/0x460
[<ffffffffaecb9d4f>] process_one_work+0x17f/0x440
[<ffffffffaecbade6>] worker_thread+0x126/0x3c0
[<ffffffffaecbacc0>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffaecc1c31>] kthread+0xd1/0xe0
[<ffffffffaecc1b60>] ? insert_kthread_work+0x40/0x40
[<ffffffffaf374c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffaecc1b60>] ? insert_kthread_work+0x40/0x40
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #16461
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc03843be
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc03a671e
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x24e
3
mptspi_qcmd
mptspi 0xffffffffc03772e0
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffff8ccde1a0
Build id: 3.10.0-1062.4.3.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffff8cce780c
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x4cc
6
? deadline_add_request
vmlinux 0xffffffff8cb72f8e
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffff8cb4e7f9
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffff8cb52513
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffff8cb504a7
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffff8cb50750
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffff8c9c8874
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc05e4cd1
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc05e54c4
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffff8c9c92ec
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc05e5030
Build id: 3.10.0-1062.4.3.el7.x86_64 		0xa0
16
xfs_vm_writepages
xfs 0xffffffffc05e4ffb
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x6b
17
do_writepages
vmlinux 0xffffffff8c9ca331
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x21
18
__writeback_single_inode
vmlinux 0xffffffff8ca78fb0
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x40
19
? wake_up_bit
vmlinux 0xffffffff8c8c72a5
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x25
20
writeback_sb_inodes
vmlinux 0xffffffff8ca79b44
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x1c4
21
__writeback_inodes_wb
vmlinux 0xffffffff8ca79e4f
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x9f
22
wb_writeback
vmlinux 0xffffffff8ca7a333
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x263
23
bdi_writeback_workfn
vmlinux 0xffffffff8ca7af2b
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x2cb
24
process_one_work
vmlinux 0xffffffff8c8be21f
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x17f
25
worker_thread
vmlinux 0xffffffff8c8bf336
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x126
26
? manage_workers.isra.26
vmlinux 0xffffffff8c8bf210
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x2a0
27
kthread
vmlinux 0xffffffff8c8c61f1
Build id: 3.10.0-1062.4.3.el7.x86_64 		0xd1
28
? insert_kthread_work
vmlinux 0xffffffff8c8c6120
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x40
29
ret_from_fork_nospec_begin
vmlinux 0xffffffff8cf8cd37
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x21
30
? insert_kthread_work
vmlinux 0xffffffff8c8c6120
Build id: 3.10.0-1062.4.3.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kworker/u21:1:24775]
Modules linked in: fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sb_edac coretemp iosf_mbi ppdev crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd i2c_piix4 joydev pcspkr sg vmw_vmci parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif
crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm crct10dif_pclmul crct10dif_common crc32c_intel libata mptspi serio_raw scsi_transport_spi e1000 mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 7 PID: 24775 Comm: kworker/u21:1 Kdump: loaded Tainted: G             L ------------   3.10.0-1062.4.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/30/2013
Workqueue: writeback bdi_writeback_workfn (flush-253:1)
task: ffff976cee3e20e0 ti: ffff976dad658000 task.ti: ffff976dad658000
RIP: 0010:[<ffffffffc03843be>]  [<ffffffffc03843be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff976dad65b708  EFLAGS: 00000246
RAX: ffffb600c1bc0000 RBX: ffff976d95660060 RCX: 0000000000000012
RDX: ffff976db92c5e00 RSI: ffff976b111fc000 RDI: 000000000000000e
RBP: ffff976dad65b718 R08: 0000000000000003 R09: 00000002d7d61fff
R10: ffff9769b5ffc0d8 R11: ffff976d58704c40 R12: ffff976d7fa50780
R13: ffff976dad65b67c R14: ffffffff8cb5705b R15: ffff976dad65b6c0
FS:  0000000000000000(0000) GS:ffff976dbfc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9b72277688 CR3: 0000000197efc000 CR4: 00000000000407e0
Call Trace:
[<ffffffffc03a671e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc03772e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff8ccde1a0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff8cce780c>] scsi_request_fn+0x4cc/0x680
[<ffffffff8cb72f8e>] ? deadline_add_request+0x6e/0x90
[<ffffffff8cb4e7f9>] __blk_run_queue+0x39/0x50
[<ffffffff8cb52513>] blk_queue_bio+0x3b3/0x400
[<ffffffff8cb504a7>] generic_make_request+0x147/0x380
[<ffffffff8cb50750>] submit_bio+0x70/0x150
[<ffffffff8c9c8874>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc05e4cd1>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc05e54c4>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffff8c9c92ec>] write_cache_pages+0x21c/0x470
[<ffffffffc05e5030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffc05e4ffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff8c9ca331>] do_writepages+0x21/0x50
[<ffffffff8ca78fb0>] __writeback_single_inode+0x40/0x260
[<ffffffff8c8c72a5>] ? wake_up_bit+0x25/0x30
[<ffffffff8ca79b44>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8ca79e4f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8ca7a333>] wb_writeback+0x263/0x2f0
[<ffffffff8ca7af2b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff8c8be21f>] process_one_work+0x17f/0x440
[<ffffffff8c8bf336>] worker_thread+0x126/0x3c0
[<ffffffff8c8bf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8c8c61f1>] kthread+0xd1/0xe0
[<ffffffff8c8c6120>] ? insert_kthread_work+0x40/0x40
[<ffffffff8cf8cd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8c8c6120>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 23 39
Complete report #15279
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff88181125
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x15
2
pvscsi_queue
vmw_pvscsi 0xffffffffc03b2632
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x4a2
3
? scsi_test_unit_ready
vmlinux 0xffffffff87ee5700
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x170
4
scsi_dispatch_cmd
vmlinux 0xffffffff87edd100
Build id: 3.10.0-1062.4.1.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffff87ee676c
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x4cc
6
? deadline_add_request
vmlinux 0xffffffff87d71f4e
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffff87d4d7b9
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffff87d514d3
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffff87d4f467
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffff87d4f710
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffff87bc7834
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc05fecd1
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc05ff4c4
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffff87bc82ac
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc05ff030
Build id: 3.10.0-1062.4.1.el7.x86_64 		0xa0
16
? submit_bio
vmlinux 0xffffffff87d4f710
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x70
17
? __switch_to
vmlinux 0xffffffff87a2b59e
Build id: 3.10.0-1062.4.1.el7.x86_64 		0xce
18
xfs_vm_writepages
xfs 0xffffffffc05feffb
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x6b
19
do_writepages
vmlinux 0xffffffff87bc92f1
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x21
20
__writeback_single_inode
vmlinux 0xffffffff87c77f70
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x40
21
writeback_sb_inodes
vmlinux 0xffffffff87c78b04
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x1c4
22
__writeback_inodes_wb
vmlinux 0xffffffff87c78e0f
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x9f
23
wb_writeback
vmlinux 0xffffffff87c792f3
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x263
24
bdi_writeback_workfn
vmlinux 0xffffffff87c79eeb
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x2cb
25
process_one_work
vmlinux 0xffffffff87abd1df
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x17f
26
worker_thread
vmlinux 0xffffffff87abe2f6
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x126
27
? manage_workers.isra.26
vmlinux 0xffffffff87abe1d0
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x2a0
28
kthread
vmlinux 0xffffffff87ac51b1
Build id: 3.10.0-1062.4.1.el7.x86_64 		0xd1
29
? insert_kthread_work
vmlinux 0xffffffff87ac50e0
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x40
30
ret_from_fork_nospec_begin
vmlinux 0xffffffff8818bd37
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x21
31
? insert_kthread_work
vmlinux 0xffffffff87ac50e0
Build id: 3.10.0-1062.4.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 40s! [kworker/u8:0:22922]
Modules linked in: nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper vmw_balloon ablk_helper cryptd pcspkr sg joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables
xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit crct10dif_pclmul crct10dif_common crc32c_intel libnvdimm serio_raw drm ata_piix libata vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 22922 Comm: kworker/u8:0 Kdump: loaded Tainted: G             L ------------ T 3.10.0-1062.4.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/03/2018
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff99b8a50641c0 ti: ffff99b8038b8000 task.ti: ffff99b8038b8000
RIP: 0010:[<ffffffff88181125>]  [<ffffffff88181125>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0000:ffff99b8038bb770  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffffffff87dac073 RCX: ffffffff88634800
RDX: ffffbee180e90000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff99b8038bb770 R08: 0000000000000001 R09: 0000000000000000
R10: ffff99ba332b68d8 R11: ffff99ba332b7000 R12: 0000000000000000
R13: 0000000000000020 R14: ffff99b8a5dc7980 R15: ffffffff87ee4459
FS:  0000000000000000(0000) GS:ffff99ba3fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001365e00 CR3: 0000000082610000 CR4: 00000000000607f0
Call Trace:
[<ffffffffc03b2632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffff87ee5700>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffff87edd100>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff87ee676c>] scsi_request_fn+0x4cc/0x680
[<ffffffff87d71f4e>] ? deadline_add_request+0x6e/0x90
[<ffffffff87d4d7b9>] __blk_run_queue+0x39/0x50
[<ffffffff87d514d3>] blk_queue_bio+0x3b3/0x400
[<ffffffff87d4f467>] generic_make_request+0x147/0x380
[<ffffffff87d4f710>] submit_bio+0x70/0x150
[<ffffffff87bc7834>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc05fecd1>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc05ff4c4>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffff87bc82ac>] write_cache_pages+0x21c/0x470
[<ffffffffc05ff030>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff87d4f710>] ? submit_bio+0x70/0x150
[<ffffffff87a2b59e>] ? __switch_to+0xce/0x580
[<ffffffffc05feffb>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff87bc92f1>] do_writepages+0x21/0x50
[<ffffffff87c77f70>] __writeback_single_inode+0x40/0x260
[<ffffffff87c78b04>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff87c78e0f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff87c792f3>] wb_writeback+0x263/0x2f0
[<ffffffff87c79eeb>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff87abd1df>] process_one_work+0x17f/0x440
[<ffffffff87abe2f6>] worker_thread+0x126/0x3c0
[<ffffffff87abe1d0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff87ac51b1>] kthread+0xd1/0xe0
[<ffffffff87ac50e0>] ? insert_kthread_work+0x40/0x40
[<ffffffff8818bd37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff87ac50e0>] ? insert_kthread_work+0x40/0x40
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48
Complete report #4938
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc023f3be
Build id: 3.10.0-1127.el7.x86_64 		0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc025a71e
Build id: 3.10.0-1127.el7.x86_64 		0x24e
3
mptspi_qcmd
mptspi 0xffffffffc01e42e0
Build id: 3.10.0-1127.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffffadee2850
Build id: 3.10.0-1127.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffffadeebefc
Build id: 3.10.0-1127.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffffadd7709e
Build id: 3.10.0-1127.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffffadd526f9
Build id: 3.10.0-1127.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffffadd56403
Build id: 3.10.0-1127.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffffadd54387
Build id: 3.10.0-1127.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffffadd54630
Build id: 3.10.0-1127.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffffadbc96e4
Build id: 3.10.0-1127.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc0438071
Build id: 3.10.0-1127.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc0438864
Build id: 3.10.0-1127.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffffadbca15c
Build id: 3.10.0-1127.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc04383d0
Build id: 3.10.0-1127.el7.x86_64 		0xa0
16
? __enqueue_entity
vmlinux 0xffffffffadae0628
Build id: 3.10.0-1127.el7.x86_64 		0x78
17
? cpumask_next_and
vmlinux 0xffffffffadd84ee5
Build id: 3.10.0-1127.el7.x86_64 		0x35
18
? find_busiest_group
vmlinux 0xffffffffadae8ae4
Build id: 3.10.0-1127.el7.x86_64 		0x144
19
xfs_vm_writepages
xfs 0xffffffffc043839b
Build id: 3.10.0-1127.el7.x86_64 		0x6b
20
do_writepages
vmlinux 0xffffffffadbcb1b1
Build id: 3.10.0-1127.el7.x86_64 		0x21
21
__writeback_single_inode
vmlinux 0xffffffffadc7cbd0
Build id: 3.10.0-1127.el7.x86_64 		0x40
22
? wake_up_bit
vmlinux 0xffffffffadac7745
Build id: 3.10.0-1127.el7.x86_64 		0x25
23
writeback_sb_inodes
vmlinux 0xffffffffadc7d764
Build id: 3.10.0-1127.el7.x86_64 		0x1c4
24
__writeback_inodes_wb
vmlinux 0xffffffffadc7da6f
Build id: 3.10.0-1127.el7.x86_64 		0x9f
25
wb_writeback
vmlinux 0xffffffffadc7df53
Build id: 3.10.0-1127.el7.x86_64 		0x263
26
? get_nr_inodes
vmlinux 0xffffffffadc69bac
Build id: 3.10.0-1127.el7.x86_64 		0x4c
27
bdi_writeback_workfn
vmlinux 0xffffffffadc7eb4b
Build id: 3.10.0-1127.el7.x86_64 		0x2cb
28
process_one_work
vmlinux 0xffffffffadabe6bf
Build id: 3.10.0-1127.el7.x86_64 		0x17f
29
worker_thread
vmlinux 0xffffffffadabf7d6
Build id: 3.10.0-1127.el7.x86_64 		0x126
30
? manage_workers.isra.26
vmlinux 0xffffffffadabf6b0
Build id: 3.10.0-1127.el7.x86_64 		0x2a0
31
kthread
vmlinux 0xffffffffadac6691
Build id: 3.10.0-1127.el7.x86_64 		0xd1
32
? insert_kthread_work
vmlinux 0xffffffffadac65c0
Build id: 3.10.0-1127.el7.x86_64 		0x40
33
ret_from_fork_nospec_begin
vmlinux 0xffffffffae192d37
Build id: 3.10.0-1127.el7.x86_64 		0x21
34
? insert_kthread_work
vmlinux 0xffffffffadac65c0
Build id: 3.10.0-1127.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 27s! [kworker/u256:2:19298]
Modules linked in: binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus aesni_intel lrw snd_seq gf128mul glue_helper ablk_helper cryptd vmw_balloon snd_seq_device joydev pcspkr snd_pcm snd_timer snd soundcore sg parport_pc parport i2c_piix4 vmw_vmci
ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx e1000 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops serio_raw ttm mptspi drm ata_piix scsi_transport_spi mptscsih mptbase libata nfit drm_panel_orientation_quirks libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 19298 Comm: kworker/u256:2 Kdump: loaded Tainted: G             L ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9bf52da81070 ti: ffff9bf519e34000 task.ti: ffff9bf519e34000
RIP: 0010:[<ffffffffc023f3be>]  [<ffffffffc023f3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff9bf519e37708  EFLAGS: 00000246
RAX: ffffbf79803c0000 RBX: ffff9bf519fb1000 RCX: 0000000000000001
RDX: ffff9bf4f59a0c00 RSI: ffff9bf4f5aec000 RDI: 000000000000000e
RBP: ffff9bf519e37718 R08: 0000000000000002 R09: 000000006a0c3fff
R10: ffff9bf4f66ef800 R11: ffff9bf536fe0a80 R12: ffff9bf536c0eb80
R13: ffff9bf519e3767c R14: ffffffffadd5b01b R15: ffff9bf519e376c0
FS:  0000000000000000(0000) GS:ffff9bf53b640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9131dd3000 CR3: 000000007830c000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc025a71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc01e42e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffadee2850>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffadeebefc>] scsi_request_fn+0x4ac/0x660
[<ffffffffadd7709e>] ? deadline_add_request+0x6e/0x90
[<ffffffffadd526f9>] __blk_run_queue+0x39/0x50
[<ffffffffadd56403>] blk_queue_bio+0x3b3/0x400
[<ffffffffadd54387>] generic_make_request+0x147/0x380
[<ffffffffadd54630>] submit_bio+0x70/0x150
[<ffffffffadbc96e4>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc0438071>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc0438864>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffffadbca15c>] write_cache_pages+0x21c/0x470
[<ffffffffc04383d0>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffffadae0628>] ? __enqueue_entity+0x78/0x80
[<ffffffffadd84ee5>] ? cpumask_next_and+0x35/0x50
[<ffffffffadae8ae4>] ? find_busiest_group+0x144/0x990
[<ffffffffc043839b>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffffadbcb1b1>] do_writepages+0x21/0x50
[<ffffffffadc7cbd0>] __writeback_single_inode+0x40/0x260
[<ffffffffadac7745>] ? wake_up_bit+0x25/0x30
[<ffffffffadc7d764>] writeback_sb_inodes+0x1c4/0x430
[<ffffffffadc7da6f>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffffadc7df53>] wb_writeback+0x263/0x2f0
[<ffffffffadc69bac>] ? get_nr_inodes+0x4c/0x70
[<ffffffffadc7eb4b>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffffadabe6bf>] process_one_work+0x17f/0x440
[<ffffffffadabf7d6>] worker_thread+0x126/0x3c0
[<ffffffffadabf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffadac6691>] kthread+0xd1/0xe0
[<ffffffffadac65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffffae192d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffadac65c0>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 d3 24
Complete report #2485
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc02023be
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc021d71e
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x24e
3
mptspi_qcmd
mptspi 0xffffffffc02952e0
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffff86ae2920
Build id: 3.10.0-1127.10.1.el7.x86_64 		0xb0
5
scsi_request_fn
vmlinux 0xffffffff86aec04c
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x4ac
6
? deadline_add_request
vmlinux 0xffffffff8697716e
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x6e
7
__blk_run_queue
vmlinux 0xffffffff86952769
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x39
8
blk_queue_bio
vmlinux 0xffffffff86956473
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x3b3
9
generic_make_request
vmlinux 0xffffffff869543f7
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x147
10
submit_bio
vmlinux 0xffffffff869546a0
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x70
11
? __test_set_page_writeback
vmlinux 0xffffffff867c9744
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x54
12
xfs_submit_ioend.isra.12
xfs 0xffffffffc0444fb1
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x61
13
xfs_do_writepage
xfs 0xffffffffc04457a4
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x494
14
write_cache_pages
vmlinux 0xffffffff867ca1bc
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x21c
15
? xfs_vm_writepages
xfs 0xffffffffc0445310
Build id: 3.10.0-1127.10.1.el7.x86_64 		0xa0
16
? cpumask_next_and
vmlinux 0xffffffff86984fc5
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x35
17
? find_busiest_group
vmlinux 0xffffffff866e8ae4
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x144
18
xfs_vm_writepages
xfs 0xffffffffc04452db
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x6b
19
do_writepages
vmlinux 0xffffffff867cb211
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x21
20
__writeback_single_inode
vmlinux 0xffffffff8687cc30
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x40
21
writeback_sb_inodes
vmlinux 0xffffffff8687d7c4
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x1c4
22
__writeback_inodes_wb
vmlinux 0xffffffff8687dacf
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x9f
23
wb_writeback
vmlinux 0xffffffff8687dfb3
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x263
24
? get_nr_inodes
vmlinux 0xffffffff86869c0c
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x4c
25
bdi_writeback_workfn
vmlinux 0xffffffff8687ebab
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x2cb
26
process_one_work
vmlinux 0xffffffff866be6bf
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x17f
27
worker_thread
vmlinux 0xffffffff866bf7d6
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x126
28
? manage_workers.isra.26
vmlinux 0xffffffff866bf6b0
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x2a0
29
kthread
vmlinux 0xffffffff866c6691
Build id: 3.10.0-1127.10.1.el7.x86_64 		0xd1
30
? insert_kthread_work
vmlinux 0xffffffff866c65c0
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x40
31
ret_from_fork_nospec_begin
vmlinux 0xffffffff86d92d37
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x21
32
? insert_kthread_work
vmlinux 0xffffffff866c65c0
Build id: 3.10.0-1127.10.1.el7.x86_64 		0x40 
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [kworker/u256:1:33654]
Modules linked in: act_police sch_ingress cls_u32 sch_sfq sch_cbq sch_tbf xt_nat veth binfmt_misc xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat_ipv4 nf_nat overlay(T) vmw_vsock_vmci_transport vsock iptable_filter sunrpc coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ppdev joydev pcspkr vmw_balloon parport_pc parport vmw_vmci ip_tables xfs sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul ata_piix crct10dif_common crc32c_intel mptspi scsi_transport_spi libata mptscsih serio_raw mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack libcrc32c br_netfilter bridge stp llc
CPU: 10 PID: 33654 Comm: kworker/u256:1 Tainted: G             L ------------ T 3.10.0-1127.10.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff8849d57741c0 ti: ffff8849ba308000 task.ti: ffff8849ba308000
RIP: 0010:[<ffffffffc02023be>]  [<ffffffffc02023be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff8849ba30b708  EFLAGS: 00000246
RAX: ffffad3f03500000 RBX: ffff8841f53e53d0 RCX: 0000000000000009
RDX: ffff8849d741ce00 RSI: ffff8849d8808000 RDI: 000000000000000e
RBP: ffff8849ba30b718 R08: 0000000000000003 R09: 0000000035f7afff
R10: ffff8849d88af400 R11: ffff8849d658c380 R12: ffff8849b9eb6880
R13: ffff8849ba30b67c R14: ffffffff8695b08b R15: ffff8849ba30b6c0
FS:  0000000000000000(0000) GS:ffff8849dd880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2f2eb277e0 CR3: 000000081c8b0000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc021d71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc02952e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff86ae2920>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff86aec04c>] scsi_request_fn+0x4ac/0x680
[<ffffffff8697716e>] ? deadline_add_request+0x6e/0x90
[<ffffffff86952769>] __blk_run_queue+0x39/0x50
[<ffffffff86956473>] blk_queue_bio+0x3b3/0x400
[<ffffffff869543f7>] generic_make_request+0x147/0x380
[<ffffffff869546a0>] submit_bio+0x70/0x150
[<ffffffff867c9744>] ? __test_set_page_writeback+0x54/0x1a0
[<ffffffffc0444fb1>] xfs_submit_ioend.isra.12+0x61/0xe0 [xfs]
[<ffffffffc04457a4>] xfs_do_writepage+0x494/0x550 [xfs]
[<ffffffff867ca1bc>] write_cache_pages+0x21c/0x470
[<ffffffffc0445310>] ? xfs_vm_writepages+0xa0/0xa0 [xfs]
[<ffffffff86984fc5>] ? cpumask_next_and+0x35/0x50
[<ffffffff866e8ae4>] ? find_busiest_group+0x144/0x990
[<ffffffffc04452db>] xfs_vm_writepages+0x6b/0xa0 [xfs]
[<ffffffff867cb211>] do_writepages+0x21/0x50
[<ffffffff8687cc30>] __writeback_single_inode+0x40/0x260
[<ffffffff8687d7c4>] writeback_sb_inodes+0x1c4/0x430
[<ffffffff8687dacf>] __writeback_inodes_wb+0x9f/0xd0
[<ffffffff8687dfb3>] wb_writeback+0x263/0x2f0
[<ffffffff86869c0c>] ? get_nr_inodes+0x4c/0x70
[<ffffffff8687ebab>] bdi_writeback_workfn+0x2cb/0x460
[<ffffffff866be6bf>] process_one_work+0x17f/0x440
[<ffffffff866bf7d6>] worker_thread+0x126/0x3c0
[<ffffffff866bf6b0>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff866c6691>] kthread+0xd1/0xe0
[<ffffffff866c65c0>] ? insert_kthread_work+0x40/0x40
[<ffffffff86d92d37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff866c65c0>] ? insert_kthread_work+0x40/0x40
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 03 21