ABRT Analytics - Problem #823337 - kernel in mptspi

Info

Permalink
Function: mptspi_qcmd
First occurrence: 2020-10-15
Last occurrence: 2026-10-10
State: NEW
Components: kernel
Quality: -24

Statistics

Operating system	Count
CentOS 7	41

Architecture	Count
x86_64	41


Related packages	Count
kernel	41
0:3.10.0-1160.119.1.el7	6
0:3.10.0-862.el7	5
0:3.10.0-1160.53.1.el7	4
0:3.10.0-1127.19.1.el7	4
0:3.10.0-957.el7	3
0:3.10.0-1160.36.2.el7	3
0:3.10.0-1062.el7	2
0:3.10.0-1160.41.1.el7	2
0:3.10.0-1160.81.1.el7	2
0:3.10.0-957.21.3.el7	1
0:3.10.0-862.2.3.el7	1
0:3.10.0-1062.9.1.el7	1
0:3.10.0-1160.6.1.el7	1
0:3.10.0-1160.42.2.el7	1
0:3.10.0-1160.25.1.el7	1
0:3.10.0-1160.59.1.el7	1
0:3.10.0-957.12.2.el7	1
0:3.10.0-1160.88.1.el7	1
0:3.10.0-1160.11.1.el7	1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace

Complete report #221043

Frame #	Function	Binary	Source or offset	Line
1	__blk_run_queue	vmlinux	0xffffffffad118ad9 Build id: 3.10.0-862.el7.x86_64	0x39
2	blk_run_queue	vmlinux	0xffffffffad118b56 Build id: 3.10.0-862.el7.x86_64	0x26
3	scsi_run_queue	vmlinux	0xffffffffad2a2b58 Build id: 3.10.0-862.el7.x86_64	0x258
4	scsi_next_command	vmlinux	0xffffffffad2a47b0 Build id: 3.10.0-862.el7.x86_64	0x20
5	scsi_end_request	vmlinux	0xffffffffad2a4905 Build id: 3.10.0-862.el7.x86_64	0x135
6	scsi_io_completion	vmlinux	0xffffffffad2a4b78 Build id: 3.10.0-862.el7.x86_64	0x168
7	scsi_finish_command	vmlinux	0xffffffffad29a05c Build id: 3.10.0-862.el7.x86_64	0xdc
8	scsi_softirq_done	vmlinux	0xffffffffad2a40c2 Build id: 3.10.0-862.el7.x86_64	0x132
9	blk_done_softirq	vmlinux	0xffffffffad122796 Build id: 3.10.0-862.el7.x86_64	0x96
10	__do_softirq	vmlinux	0xfffffffface9a945 Build id: 3.10.0-862.el7.x86_64	0xf5
11	call_softirq	vmlinux	0xffffffffad522d2c Build id: 3.10.0-862.el7.x86_64	0x1c
12	do_softirq	vmlinux	0xfffffffface2d625 Build id: 3.10.0-862.el7.x86_64	0x65
13	irq_exit	vmlinux	0xfffffffface9acc5 Build id: 3.10.0-862.el7.x86_64	0x105
14	smp_call_function_single_interrupt	vmlinux	0xfffffffface54799 Build id: 3.10.0-862.el7.x86_64	0x39
15	call_function_single_interrupt	vmlinux	0xffffffffad521aa2 Build id: 3.10.0-862.el7.x86_64	0x162

NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [swapper/0:0]
[<ffffffffad118ad9>] __blk_run_queue+0x39/0x50
[<ffffffffad118b56>] blk_run_queue+0x26/0x40
[<ffffffffad2a2b58>] scsi_run_queue+0x258/0x2f0
[<ffffffffad2a47b0>] scsi_next_command+0x20/0x40
[<ffffffffad2a4905>] scsi_end_request+0x135/0x1e0
[<ffffffffad2a4b78>] scsi_io_completion+0x168/0x6a0
[<ffffffffad29a05c>] scsi_finish_command+0xdc/0x140
[<ffffffffad2a40c2>] scsi_softirq_done+0x132/0x160
[<ffffffffad122796>] blk_done_softirq+0x96/0xc0
[<fffffffface9a945>] __do_softirq+0xf5/0x280
[<ffffffffad522d2c>] call_softirq+0x1c/0x30
[<fffffffface2d625>] do_softirq+0x65/0xa0
[<fffffffface9acc5>] irq_exit+0x105/0x110
[<fffffffface54799>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffffad521aa2>] call_function_single_interrupt+0x162/0x170

Complete report #174551

Frame #	Function	Binary	Source or offset	Line
1	scsi_next_command	vmlinux	0xffffffffb0ca47b0 Build id: 3.10.0-862.el7.x86_64	0x20
2	scsi_end_request	vmlinux	0xffffffffb0ca4905 Build id: 3.10.0-862.el7.x86_64	0x135
3	scsi_io_completion	vmlinux	0xffffffffb0ca4b78 Build id: 3.10.0-862.el7.x86_64	0x168
4	scsi_finish_command	vmlinux	0xffffffffb0c9a05c Build id: 3.10.0-862.el7.x86_64	0xdc
5	scsi_softirq_done	vmlinux	0xffffffffb0ca40c2 Build id: 3.10.0-862.el7.x86_64	0x132
6	blk_done_softirq	vmlinux	0xffffffffb0b22796 Build id: 3.10.0-862.el7.x86_64	0x96
7	__do_softirq	vmlinux	0xffffffffb089a945 Build id: 3.10.0-862.el7.x86_64	0xf5
8	call_softirq	vmlinux	0xffffffffb0f22d2c Build id: 3.10.0-862.el7.x86_64	0x1c
9	do_softirq	vmlinux	0xffffffffb082d625 Build id: 3.10.0-862.el7.x86_64	0x65
10	irq_exit	vmlinux	0xffffffffb089acc5 Build id: 3.10.0-862.el7.x86_64	0x105
11	smp_call_function_single_interrupt	vmlinux	0xffffffffb0854799 Build id: 3.10.0-862.el7.x86_64	0x39
12	call_function_single_interrupt	vmlinux	0xffffffffb0f21aa2 Build id: 3.10.0-862.el7.x86_64	0x162

NMI watchdog: BUG: soft lockup - CPU#6 stuck for 32s! [java:26606]
[<ffffffffb0ca47b0>] scsi_next_command+0x20/0x40
[<ffffffffb0ca4905>] scsi_end_request+0x135/0x1e0
[<ffffffffb0ca4b78>] scsi_io_completion+0x168/0x6a0
[<ffffffffb0c9a05c>] scsi_finish_command+0xdc/0x140
[<ffffffffb0ca40c2>] scsi_softirq_done+0x132/0x160
[<ffffffffb0b22796>] blk_done_softirq+0x96/0xc0
[<ffffffffb089a945>] __do_softirq+0xf5/0x280
[<ffffffffb0f22d2c>] call_softirq+0x1c/0x30
[<ffffffffb082d625>] do_softirq+0x65/0xa0
[<ffffffffb089acc5>] irq_exit+0x105/0x110
[<ffffffffb0854799>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffffb0f21aa2>] call_function_single_interrupt+0x162/0x170

Complete report #11543

Frame #	Function	Binary	Source or offset	Line
1	scsi_run_queue	vmlinux	0xffffffff960d9248 Build id: 3.10.0-957.21.3.el7.x86_64	0x258
2	scsi_next_command	vmlinux	0xffffffff960dae90 Build id: 3.10.0-957.21.3.el7.x86_64	0x20
3	scsi_end_request	vmlinux	0xffffffff960dafe5 Build id: 3.10.0-957.21.3.el7.x86_64	0x135
4	scsi_io_completion	vmlinux	0xffffffff960db258 Build id: 3.10.0-957.21.3.el7.x86_64	0x168
5	scsi_finish_command	vmlinux	0xffffffff960d070c Build id: 3.10.0-957.21.3.el7.x86_64	0xdc
6	scsi_softirq_done	vmlinux	0xffffffff960da7a2 Build id: 3.10.0-957.21.3.el7.x86_64	0x132
7	blk_done_softirq	vmlinux	0xffffffff95f4d546 Build id: 3.10.0-957.21.3.el7.x86_64	0x96
8	__do_softirq	vmlinux	0xffffffff95ca1075 Build id: 3.10.0-957.21.3.el7.x86_64	0xf5
9	call_softirq	vmlinux	0xffffffff9637932c Build id: 3.10.0-957.21.3.el7.x86_64	0x1c
10	do_softirq	vmlinux	0xffffffff95c2e675 Build id: 3.10.0-957.21.3.el7.x86_64	0x65
11	irq_exit	vmlinux	0xffffffff95ca13f5 Build id: 3.10.0-957.21.3.el7.x86_64	0x105
12	smp_call_function_single_interrupt	vmlinux	0xffffffff95c573f9 Build id: 3.10.0-957.21.3.el7.x86_64	0x39
13	call_function_single_interrupt	vmlinux	0xffffffff963780a2 Build id: 3.10.0-957.21.3.el7.x86_64	0x162

NMI watchdog: BUG: soft lockup - CPU#5 stuck for 27s! [smbd:10868]
[<ffffffff960d9248>] scsi_run_queue+0x258/0x2f0
[<ffffffff960dae90>] scsi_next_command+0x20/0x40
[<ffffffff960dafe5>] scsi_end_request+0x135/0x1e0
[<ffffffff960db258>] scsi_io_completion+0x168/0x6a0
[<ffffffff960d070c>] scsi_finish_command+0xdc/0x140
[<ffffffff960da7a2>] scsi_softirq_done+0x132/0x160
[<ffffffff95f4d546>] blk_done_softirq+0x96/0xc0
[<ffffffff95ca1075>] __do_softirq+0xf5/0x280
[<ffffffff9637932c>] call_softirq+0x1c/0x30
[<ffffffff95c2e675>] do_softirq+0x65/0xa0
[<ffffffff95ca13f5>] irq_exit+0x105/0x110
[<ffffffff95c573f9>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffff963780a2>] call_function_single_interrupt+0x162/0x170

Complete report #8775

Frame #	Function	Binary	Source or offset	Line
1	blk_run_queue	vmlinux	0xffffffff9e94cfd6 Build id: 3.10.0-1062.el7.x86_64	0x26
2	scsi_run_queue	vmlinux	0xffffffff9eae44a8 Build id: 3.10.0-1062.el7.x86_64	0x258
3	scsi_next_command	vmlinux	0xffffffff9eae60f0 Build id: 3.10.0-1062.el7.x86_64	0x20
4	scsi_end_request	vmlinux	0xffffffff9eae6245 Build id: 3.10.0-1062.el7.x86_64	0x135
5	scsi_io_completion	vmlinux	0xffffffff9eae64b8 Build id: 3.10.0-1062.el7.x86_64	0x168
6	scsi_finish_command	vmlinux	0xffffffff9eadb89c Build id: 3.10.0-1062.el7.x86_64	0xdc
7	scsi_softirq_done	vmlinux	0xffffffff9eae5a02 Build id: 3.10.0-1062.el7.x86_64	0x132
8	blk_done_softirq	vmlinux	0xffffffff9e956c56 Build id: 3.10.0-1062.el7.x86_64	0x96
9	__do_softirq	vmlinux	0xffffffff9e6a41e5 Build id: 3.10.0-1062.el7.x86_64	0xf5
10	call_softirq	vmlinux	0xffffffff9ed8f42c Build id: 3.10.0-1062.el7.x86_64	0x1c
11	do_softirq	vmlinux	0xffffffff9e62f675 Build id: 3.10.0-1062.el7.x86_64	0x65
12	irq_exit	vmlinux	0xffffffff9e6a4565 Build id: 3.10.0-1062.el7.x86_64	0x105
13	smp_call_function_single_interrupt	vmlinux	0xffffffff9e659099 Build id: 3.10.0-1062.el7.x86_64	0x39
14	call_function_single_interrupt	vmlinux	0xffffffff9ed8e1a2 Build id: 3.10.0-1062.el7.x86_64	0x162

NMI watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [ksoftirqd/0:6]
[<ffffffff9e94cfd6>] blk_run_queue+0x26/0x40
[<ffffffff9eae44a8>] scsi_run_queue+0x258/0x2f0
[<ffffffff9eae60f0>] scsi_next_command+0x20/0x40
[<ffffffff9eae6245>] scsi_end_request+0x135/0x1e0
[<ffffffff9eae64b8>] scsi_io_completion+0x168/0x6a0
[<ffffffff9eadb89c>] scsi_finish_command+0xdc/0x140
[<ffffffff9eae5a02>] scsi_softirq_done+0x132/0x160
[<ffffffff9e956c56>] blk_done_softirq+0x96/0xc0
[<ffffffff9e6a41e5>] __do_softirq+0xf5/0x280
[<ffffffff9ed8f42c>] call_softirq+0x1c/0x30
[<ffffffff9e62f675>] do_softirq+0x65/0xa0
[<ffffffff9e6a4565>] irq_exit+0x105/0x110
[<ffffffff9e659099>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffff9ed8e1a2>] call_function_single_interrupt+0x162/0x170

Complete report #797096

Frame #	Function	Binary	Source or offset	Line
1	mptspi_qcmd	mptspi	0xffffffffc033c2f0 Build id: 3.10.0-1160.81.1.el7.x86_64	0x50
2	scsi_dispatch_cmd	vmlinux	0xffffffffb0302a90 Build id: 3.10.0-1160.81.1.el7.x86_64	0xb0
3	scsi_request_fn	vmlinux	0xffffffffb030c52c Build id: 3.10.0-1160.81.1.el7.x86_64	0x4ac
4	? kmem_cache_free	vmlinux	0xffffffffb003375a Build id: 3.10.0-1160.81.1.el7.x86_64	0x1fa
5	__blk_run_queue	vmlinux	0xffffffffb0168119 Build id: 3.10.0-1160.81.1.el7.x86_64	0x39
6	blk_run_queue	vmlinux	0xffffffffb01681a6 Build id: 3.10.0-1160.81.1.el7.x86_64	0x26
7	scsi_run_queue	vmlinux	0xffffffffb030aa18 Build id: 3.10.0-1160.81.1.el7.x86_64	0x258
8	scsi_next_command	vmlinux	0xffffffffb030c740 Build id: 3.10.0-1160.81.1.el7.x86_64	0x20
9	scsi_end_request	vmlinux	0xffffffffb030c8ad Build id: 3.10.0-1160.81.1.el7.x86_64	0x14d
10	scsi_io_completion	vmlinux	0xffffffffb030cb30 Build id: 3.10.0-1160.81.1.el7.x86_64	0x170
11	scsi_finish_command	vmlinux	0xffffffffb0301a00 Build id: 3.10.0-1160.81.1.el7.x86_64	0xe0
12	scsi_softirq_done	vmlinux	0xffffffffb030c048 Build id: 3.10.0-1160.81.1.el7.x86_64	0x138
13	blk_done_softirq	vmlinux	0xffffffffb0172346 Build id: 3.10.0-1160.81.1.el7.x86_64	0x96
14	__do_softirq	vmlinux	0xffffffffafea9545 Build id: 3.10.0-1160.81.1.el7.x86_64	0xf5
15	call_softirq	vmlinux	0xffffffffb05c8aac Build id: 3.10.0-1160.81.1.el7.x86_64	0x1c
16	do_softirq	vmlinux	0xffffffffafe30825 Build id: 3.10.0-1160.81.1.el7.x86_64	0x65
17	irq_exit	vmlinux	0xffffffffafea98f5 Build id: 3.10.0-1160.81.1.el7.x86_64	0x115
18	smp_call_function_single_interrupt	vmlinux	0xffffffffafe5c4d9 Build id: 3.10.0-1160.81.1.el7.x86_64	0x39
19	call_function_single_interrupt	vmlinux	0xffffffffb05c7772 Build id: 3.10.0-1160.81.1.el7.x86_64	0x172

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [node:24517]
[<ffffffffc033c2f0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffb0302a90>] scsi_dispatch_cmd+0xb0/0x250
[<ffffffffb030c52c>] scsi_request_fn+0x4ac/0x680
[<ffffffffb003375a>] ? kmem_cache_free+0x1fa/0x220
[<ffffffffb0168119>] __blk_run_queue+0x39/0x50
[<ffffffffb01681a6>] blk_run_queue+0x26/0x40
[<ffffffffb030aa18>] scsi_run_queue+0x258/0x2f0
[<ffffffffb030c740>] scsi_next_command+0x20/0x40
[<ffffffffb030c8ad>] scsi_end_request+0x14d/0x200
[<ffffffffb030cb30>] scsi_io_completion+0x170/0x720
[<ffffffffb0301a00>] scsi_finish_command+0xe0/0x140
[<ffffffffb030c048>] scsi_softirq_done+0x138/0x170
[<ffffffffb0172346>] blk_done_softirq+0x96/0xd0
[<ffffffffafea9545>] __do_softirq+0xf5/0x290
[<ffffffffb05c8aac>] call_softirq+0x1c/0x30
[<ffffffffafe30825>] do_softirq+0x65/0xa0
[<ffffffffafea98f5>] irq_exit+0x115/0x120
[<ffffffffafe5c4d9>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffffb05c7772>] call_function_single_interrupt+0x172/0x180

Complete report #442318

Frame #	Function	Binary	Source or offset	Line
1	scsi_request_fn	vmlinux	0xffffffff9e0edf3c Build id: 3.10.0-1160.36.2.el7.x86_64	0x4ac
2	__blk_run_queue	vmlinux	0xffffffff9df53ee9 Build id: 3.10.0-1160.36.2.el7.x86_64	0x39
3	blk_run_queue	vmlinux	0xffffffff9df53f66 Build id: 3.10.0-1160.36.2.el7.x86_64	0x26
4	scsi_run_queue	vmlinux	0xffffffff9e0ec498 Build id: 3.10.0-1160.36.2.el7.x86_64	0x258
5	scsi_next_command	vmlinux	0xffffffff9e0ee150 Build id: 3.10.0-1160.36.2.el7.x86_64	0x20
6	scsi_end_request	vmlinux	0xffffffff9e0ee2a5 Build id: 3.10.0-1160.36.2.el7.x86_64	0x135
7	scsi_io_completion	vmlinux	0xffffffff9e0ee518 Build id: 3.10.0-1160.36.2.el7.x86_64	0x168
8	? sched_clock	vmlinux	0xffffffff9dc35c19 Build id: 3.10.0-1160.36.2.el7.x86_64	0x9
9	scsi_finish_command	vmlinux	0xffffffff9e0e380c Build id: 3.10.0-1160.36.2.el7.x86_64	0xdc
10	scsi_softirq_done	vmlinux	0xffffffff9e0eda60 Build id: 3.10.0-1160.36.2.el7.x86_64	0x130
11	blk_done_softirq	vmlinux	0xffffffff9df5dce6 Build id: 3.10.0-1160.36.2.el7.x86_64	0x96
12	__do_softirq	vmlinux	0xffffffff9dca4bd5 Build id: 3.10.0-1160.36.2.el7.x86_64	0xf5
13	call_softirq	vmlinux	0xffffffff9e3994ec Build id: 3.10.0-1160.36.2.el7.x86_64	0x1c
14	do_softirq	vmlinux	0xffffffff9dc2f715 Build id: 3.10.0-1160.36.2.el7.x86_64	0x65
15	irq_exit	vmlinux	0xffffffff9dca4f55 Build id: 3.10.0-1160.36.2.el7.x86_64	0x105
16	smp_call_function_single_interrupt	vmlinux	0xffffffff9dc599c9 Build id: 3.10.0-1160.36.2.el7.x86_64	0x39
17	call_function_single_interrupt	vmlinux	0xffffffff9e39826a Build id: 3.10.0-1160.36.2.el7.x86_64	0x16a

NMI watchdog: BUG: soft lockup - CPU#0 stuck for 41s! [xfsaild/dm-0:432]
[<ffffffff9e0edf3c>] scsi_request_fn+0x4ac/0x680
[<ffffffff9df53ee9>] __blk_run_queue+0x39/0x50
[<ffffffff9df53f66>] blk_run_queue+0x26/0x40
[<ffffffff9e0ec498>] scsi_run_queue+0x258/0x2f0
[<ffffffff9e0ee150>] scsi_next_command+0x20/0x40
[<ffffffff9e0ee2a5>] scsi_end_request+0x135/0x1e0
[<ffffffff9e0ee518>] scsi_io_completion+0x168/0x720
[<ffffffff9dc35c19>] ? sched_clock+0x9/0x10
[<ffffffff9e0e380c>] scsi_finish_command+0xdc/0x140
[<ffffffff9e0eda60>] scsi_softirq_done+0x130/0x160
[<ffffffff9df5dce6>] blk_done_softirq+0x96/0xc0
[<ffffffff9dca4bd5>] __do_softirq+0xf5/0x280
[<ffffffff9e3994ec>] call_softirq+0x1c/0x30
[<ffffffff9dc2f715>] do_softirq+0x65/0xa0
[<ffffffff9dca4f55>] irq_exit+0x105/0x110
[<ffffffff9dc599c9>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffff9e39826a>] call_function_single_interrupt+0x16a/0x170

Complete report #369491

Frame #	Function	Binary	Source or offset	Line
1	? scsi_test_unit_ready	vmlinux	0xffffffffb46ec2a0 Build id: 3.10.0-1160.11.1.el7.x86_64	0x170
2	scsi_dispatch_cmd	vmlinux	0xffffffffb46e3b90 Build id: 3.10.0-1160.11.1.el7.x86_64	0xb0
3	scsi_request_fn	vmlinux	0xffffffffb46ed2bc Build id: 3.10.0-1160.11.1.el7.x86_64	0x4ac
4	__blk_run_queue	vmlinux	0xffffffffb45536d9 Build id: 3.10.0-1160.11.1.el7.x86_64	0x39
5	blk_run_queue	vmlinux	0xffffffffb4553756 Build id: 3.10.0-1160.11.1.el7.x86_64	0x26
6	scsi_run_queue	vmlinux	0xffffffffb46eb818 Build id: 3.10.0-1160.11.1.el7.x86_64	0x258
7	scsi_next_command	vmlinux	0xffffffffb46ed4d0 Build id: 3.10.0-1160.11.1.el7.x86_64	0x20
8	scsi_end_request	vmlinux	0xffffffffb46ed625 Build id: 3.10.0-1160.11.1.el7.x86_64	0x135
9	scsi_io_completion	vmlinux	0xffffffffb46ed898 Build id: 3.10.0-1160.11.1.el7.x86_64	0x168
10	? sched_clock	vmlinux	0xffffffffb4235c19 Build id: 3.10.0-1160.11.1.el7.x86_64	0x9
11	scsi_finish_command	vmlinux	0xffffffffb46e2b8c Build id: 3.10.0-1160.11.1.el7.x86_64	0xdc
12	scsi_softirq_done	vmlinux	0xffffffffb46ecde0 Build id: 3.10.0-1160.11.1.el7.x86_64	0x130
13	blk_done_softirq	vmlinux	0xffffffffb455d426 Build id: 3.10.0-1160.11.1.el7.x86_64	0x96
14	__do_softirq	vmlinux	0xffffffffb42a4b95 Build id: 3.10.0-1160.11.1.el7.x86_64	0xf5
15	call_softirq	vmlinux	0xffffffffb49974ec Build id: 3.10.0-1160.11.1.el7.x86_64	0x1c
16	do_softirq	vmlinux	0xffffffffb422f715 Build id: 3.10.0-1160.11.1.el7.x86_64	0x65
17	irq_exit	vmlinux	0xffffffffb42a4f15 Build id: 3.10.0-1160.11.1.el7.x86_64	0x105
18	smp_call_function_single_interrupt	vmlinux	0xffffffffb42599c9 Build id: 3.10.0-1160.11.1.el7.x86_64	0x39
19	call_function_single_interrupt	vmlinux	0xffffffffb499626a Build id: 3.10.0-1160.11.1.el7.x86_64	0x16a

NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [swapper/6:0]
[<ffffffffb46ec2a0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffffb46e3b90>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb46ed2bc>] scsi_request_fn+0x4ac/0x680
[<ffffffffb45536d9>] __blk_run_queue+0x39/0x50
[<ffffffffb4553756>] blk_run_queue+0x26/0x40
[<ffffffffb46eb818>] scsi_run_queue+0x258/0x2f0
[<ffffffffb46ed4d0>] scsi_next_command+0x20/0x40
[<ffffffffb46ed625>] scsi_end_request+0x135/0x1e0
[<ffffffffb46ed898>] scsi_io_completion+0x168/0x720
[<ffffffffb4235c19>] ? sched_clock+0x9/0x10
[<ffffffffb46e2b8c>] scsi_finish_command+0xdc/0x140
[<ffffffffb46ecde0>] scsi_softirq_done+0x130/0x160
[<ffffffffb455d426>] blk_done_softirq+0x96/0xc0
[<ffffffffb42a4b95>] __do_softirq+0xf5/0x280
[<ffffffffb49974ec>] call_softirq+0x1c/0x30
[<ffffffffb422f715>] do_softirq+0x65/0xa0
[<ffffffffb42a4f15>] irq_exit+0x105/0x110
[<ffffffffb42599c9>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffffb499626a>] call_function_single_interrupt+0x16a/0x170

Complete report #174694

Frame #	Function	Binary	Source or offset	Line
1	scsi_dispatch_cmd	vmlinux	0xffffffffb9e9b050 Build id: 3.10.0-862.el7.x86_64	0xb0
2	scsi_request_fn	vmlinux	0xffffffffb9ea45bc Build id: 3.10.0-862.el7.x86_64	0x4cc
3	__blk_run_queue	vmlinux	0xffffffffb9d18ad9 Build id: 3.10.0-862.el7.x86_64	0x39
4	blk_run_queue	vmlinux	0xffffffffb9d18b56 Build id: 3.10.0-862.el7.x86_64	0x26
5	scsi_run_queue	vmlinux	0xffffffffb9ea2b58 Build id: 3.10.0-862.el7.x86_64	0x258
6	scsi_next_command	vmlinux	0xffffffffb9ea47b0 Build id: 3.10.0-862.el7.x86_64	0x20
7	scsi_end_request	vmlinux	0xffffffffb9ea4905 Build id: 3.10.0-862.el7.x86_64	0x135
8	scsi_io_completion	vmlinux	0xffffffffb9ea4b78 Build id: 3.10.0-862.el7.x86_64	0x168
9	scsi_finish_command	vmlinux	0xffffffffb9e9a05c Build id: 3.10.0-862.el7.x86_64	0xdc
10	scsi_softirq_done	vmlinux	0xffffffffb9ea40c2 Build id: 3.10.0-862.el7.x86_64	0x132
11	blk_done_softirq	vmlinux	0xffffffffb9d22796 Build id: 3.10.0-862.el7.x86_64	0x96
12	__do_softirq	vmlinux	0xffffffffb9a9a945 Build id: 3.10.0-862.el7.x86_64	0xf5
13	call_softirq	vmlinux	0xffffffffba122d2c Build id: 3.10.0-862.el7.x86_64	0x1c
14	do_softirq	vmlinux	0xffffffffb9a2d625 Build id: 3.10.0-862.el7.x86_64	0x65
15	irq_exit	vmlinux	0xffffffffb9a9acc5 Build id: 3.10.0-862.el7.x86_64	0x105
16	smp_call_function_single_interrupt	vmlinux	0xffffffffb9a54799 Build id: 3.10.0-862.el7.x86_64	0x39
17	call_function_single_interrupt	vmlinux	0xffffffffba121aa2 Build id: 3.10.0-862.el7.x86_64	0x162

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 42s! [swapper/3:0]
[<ffffffffb9e9b050>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb9ea45bc>] scsi_request_fn+0x4cc/0x680
[<ffffffffb9d18ad9>] __blk_run_queue+0x39/0x50
[<ffffffffb9d18b56>] blk_run_queue+0x26/0x40
[<ffffffffb9ea2b58>] scsi_run_queue+0x258/0x2f0
[<ffffffffb9ea47b0>] scsi_next_command+0x20/0x40
[<ffffffffb9ea4905>] scsi_end_request+0x135/0x1e0
[<ffffffffb9ea4b78>] scsi_io_completion+0x168/0x6a0
[<ffffffffb9e9a05c>] scsi_finish_command+0xdc/0x140
[<ffffffffb9ea40c2>] scsi_softirq_done+0x132/0x160
[<ffffffffb9d22796>] blk_done_softirq+0x96/0xc0
[<ffffffffb9a9a945>] __do_softirq+0xf5/0x280
[<ffffffffba122d2c>] call_softirq+0x1c/0x30
[<ffffffffb9a2d625>] do_softirq+0x65/0xa0
[<ffffffffb9a9acc5>] irq_exit+0x105/0x110
[<ffffffffb9a54799>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffffba121aa2>] call_function_single_interrupt+0x162/0x170

Complete report #146674

Frame #	Function	Binary	Source or offset	Line
1	mptspi_qcmd	mptspi	0xffffffffc03172e0 Build id: 3.10.0-957.12.2.el7.x86_64	0x50
2	scsi_dispatch_cmd	vmlinux	0xffffffff8d2d1630 Build id: 3.10.0-957.12.2.el7.x86_64	0xb0
3	scsi_request_fn	vmlinux	0xffffffff8d2dabcc Build id: 3.10.0-957.12.2.el7.x86_64	0x4cc
4	__blk_run_queue	vmlinux	0xffffffff8d143789 Build id: 3.10.0-957.12.2.el7.x86_64	0x39
5	blk_run_queue	vmlinux	0xffffffff8d143806 Build id: 3.10.0-957.12.2.el7.x86_64	0x26
6	scsi_run_queue	vmlinux	0xffffffff8d2d9178 Build id: 3.10.0-957.12.2.el7.x86_64	0x258
7	scsi_next_command	vmlinux	0xffffffff8d2dadc0 Build id: 3.10.0-957.12.2.el7.x86_64	0x20
8	scsi_end_request	vmlinux	0xffffffff8d2daf15 Build id: 3.10.0-957.12.2.el7.x86_64	0x135
9	scsi_io_completion	vmlinux	0xffffffff8d2db188 Build id: 3.10.0-957.12.2.el7.x86_64	0x168
10	scsi_finish_command	vmlinux	0xffffffff8d2d063c Build id: 3.10.0-957.12.2.el7.x86_64	0xdc
11	scsi_softirq_done	vmlinux	0xffffffff8d2da6d2 Build id: 3.10.0-957.12.2.el7.x86_64	0x132
12	blk_done_softirq	vmlinux	0xffffffff8d14d476 Build id: 3.10.0-957.12.2.el7.x86_64	0x96
13	__do_softirq	vmlinux	0xffffffff8cea0ff5 Build id: 3.10.0-957.12.2.el7.x86_64	0xf5
14	call_softirq	vmlinux	0xffffffff8d57932c Build id: 3.10.0-957.12.2.el7.x86_64	0x1c
15	do_softirq	vmlinux	0xffffffff8ce2e675 Build id: 3.10.0-957.12.2.el7.x86_64	0x65
16	irq_exit	vmlinux	0xffffffff8cea1375 Build id: 3.10.0-957.12.2.el7.x86_64	0x105
17	smp_call_function_single_interrupt	vmlinux	0xffffffff8ce573a9 Build id: 3.10.0-957.12.2.el7.x86_64	0x39
18	call_function_single_interrupt	vmlinux	0xffffffff8d5780a2 Build id: 3.10.0-957.12.2.el7.x86_64	0x162

NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [mysqld:15371]
[<ffffffffc03172e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff8d2d1630>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff8d2dabcc>] scsi_request_fn+0x4cc/0x680
[<ffffffff8d143789>] __blk_run_queue+0x39/0x50
[<ffffffff8d143806>] blk_run_queue+0x26/0x40
[<ffffffff8d2d9178>] scsi_run_queue+0x258/0x2f0
[<ffffffff8d2dadc0>] scsi_next_command+0x20/0x40
[<ffffffff8d2daf15>] scsi_end_request+0x135/0x1e0
[<ffffffff8d2db188>] scsi_io_completion+0x168/0x6a0
[<ffffffff8d2d063c>] scsi_finish_command+0xdc/0x140
[<ffffffff8d2da6d2>] scsi_softirq_done+0x132/0x160
[<ffffffff8d14d476>] blk_done_softirq+0x96/0xc0
[<ffffffff8cea0ff5>] __do_softirq+0xf5/0x280
[<ffffffff8d57932c>] call_softirq+0x1c/0x30
[<ffffffff8ce2e675>] do_softirq+0x65/0xa0
[<ffffffff8cea1375>] irq_exit+0x105/0x110
[<ffffffff8ce573a9>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffff8d5780a2>] call_function_single_interrupt+0x162/0x170

Complete report #79179

Frame #	Function	Binary	Source or offset	Line
1	scsi_dispatch_cmd	vmlinux	0xffffffffb6ee2990 Build id: 3.10.0-1127.19.1.el7.x86_64	0xb0
2	scsi_request_fn	vmlinux	0xffffffffb6eec0bc Build id: 3.10.0-1127.19.1.el7.x86_64	0x4ac
3	__blk_run_queue	vmlinux	0xffffffffb6d527a9 Build id: 3.10.0-1127.19.1.el7.x86_64	0x39
4	blk_run_queue	vmlinux	0xffffffffb6d52826 Build id: 3.10.0-1127.19.1.el7.x86_64	0x26
5	scsi_run_queue	vmlinux	0xffffffffb6eea618 Build id: 3.10.0-1127.19.1.el7.x86_64	0x258
6	scsi_next_command	vmlinux	0xffffffffb6eec2d0 Build id: 3.10.0-1127.19.1.el7.x86_64	0x20
7	scsi_end_request	vmlinux	0xffffffffb6eec425 Build id: 3.10.0-1127.19.1.el7.x86_64	0x135
8	scsi_io_completion	vmlinux	0xffffffffb6eec698 Build id: 3.10.0-1127.19.1.el7.x86_64	0x168
9	? sched_clock	vmlinux	0xffffffffb6a35c19 Build id: 3.10.0-1127.19.1.el7.x86_64	0x9
10	scsi_finish_command	vmlinux	0xffffffffb6ee198c Build id: 3.10.0-1127.19.1.el7.x86_64	0xdc
11	scsi_softirq_done	vmlinux	0xffffffffb6eebbe0 Build id: 3.10.0-1127.19.1.el7.x86_64	0x130
12	blk_done_softirq	vmlinux	0xffffffffb6d5c4f6 Build id: 3.10.0-1127.19.1.el7.x86_64	0x96
13	__do_softirq	vmlinux	0xffffffffb6aa5695 Build id: 3.10.0-1127.19.1.el7.x86_64	0xf5
14	call_softirq	vmlinux	0xffffffffb719642c Build id: 3.10.0-1127.19.1.el7.x86_64	0x1c
15	do_softirq	vmlinux	0xffffffffb6a2f715 Build id: 3.10.0-1127.19.1.el7.x86_64	0x65
16	irq_exit	vmlinux	0xffffffffb6aa5a15 Build id: 3.10.0-1127.19.1.el7.x86_64	0x105
17	smp_call_function_single_interrupt	vmlinux	0xffffffffb6a598c9 Build id: 3.10.0-1127.19.1.el7.x86_64	0x39
18	call_function_single_interrupt	vmlinux	0xffffffffb71951aa Build id: 3.10.0-1127.19.1.el7.x86_64	0x16a

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 31s! [ora_dbw0_xe:5961]
[<ffffffffb6ee2990>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb6eec0bc>] scsi_request_fn+0x4ac/0x680
[<ffffffffb6d527a9>] __blk_run_queue+0x39/0x50
[<ffffffffb6d52826>] blk_run_queue+0x26/0x40
[<ffffffffb6eea618>] scsi_run_queue+0x258/0x2f0
[<ffffffffb6eec2d0>] scsi_next_command+0x20/0x40
[<ffffffffb6eec425>] scsi_end_request+0x135/0x1e0
[<ffffffffb6eec698>] scsi_io_completion+0x168/0x720
[<ffffffffb6a35c19>] ? sched_clock+0x9/0x10
[<ffffffffb6ee198c>] scsi_finish_command+0xdc/0x140
[<ffffffffb6eebbe0>] scsi_softirq_done+0x130/0x160
[<ffffffffb6d5c4f6>] blk_done_softirq+0x96/0xc0
[<ffffffffb6aa5695>] __do_softirq+0xf5/0x280
[<ffffffffb719642c>] call_softirq+0x1c/0x30
[<ffffffffb6a2f715>] do_softirq+0x65/0xa0
[<ffffffffb6aa5a15>] irq_exit+0x105/0x110
[<ffffffffb6a598c9>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffffb71951aa>] call_function_single_interrupt+0x16a/0x170

Complete report #24104

Frame #	Function	Binary	Source or offset	Line
1	scsi_request_fn	vmlinux	0xffffffff844e811c Build id: 3.10.0-1062.9.1.el7.x86_64	0x4cc
2	__blk_run_queue	vmlinux	0xffffffff8434f079 Build id: 3.10.0-1062.9.1.el7.x86_64	0x39
3	blk_run_queue	vmlinux	0xffffffff8434f0f6 Build id: 3.10.0-1062.9.1.el7.x86_64	0x26
4	scsi_run_queue	vmlinux	0xffffffff844e66c8 Build id: 3.10.0-1062.9.1.el7.x86_64	0x258
5	scsi_next_command	vmlinux	0xffffffff844e8310 Build id: 3.10.0-1062.9.1.el7.x86_64	0x20
6	scsi_end_request	vmlinux	0xffffffff844e8465 Build id: 3.10.0-1062.9.1.el7.x86_64	0x135
7	scsi_io_completion	vmlinux	0xffffffff844e86d8 Build id: 3.10.0-1062.9.1.el7.x86_64	0x168
8	scsi_finish_command	vmlinux	0xffffffff844ddabc Build id: 3.10.0-1062.9.1.el7.x86_64	0xdc
9	scsi_softirq_done	vmlinux	0xffffffff844e7c22 Build id: 3.10.0-1062.9.1.el7.x86_64	0x132
10	blk_done_softirq	vmlinux	0xffffffff84358d86 Build id: 3.10.0-1062.9.1.el7.x86_64	0x96
11	__do_softirq	vmlinux	0xffffffff840a5305 Build id: 3.10.0-1062.9.1.el7.x86_64	0xf5
12	call_softirq	vmlinux	0xffffffff8479142c Build id: 3.10.0-1062.9.1.el7.x86_64	0x1c
13	do_softirq	vmlinux	0xffffffff8402f715 Build id: 3.10.0-1062.9.1.el7.x86_64	0x65
14	irq_exit	vmlinux	0xffffffff840a5685 Build id: 3.10.0-1062.9.1.el7.x86_64	0x105
15	smp_call_function_single_interrupt	vmlinux	0xffffffff84059469 Build id: 3.10.0-1062.9.1.el7.x86_64	0x39
16	call_function_single_interrupt	vmlinux	0xffffffff847901aa Build id: 3.10.0-1062.9.1.el7.x86_64	0x16a

NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [kswapd0:158]
[<ffffffff844e811c>] scsi_request_fn+0x4cc/0x680
[<ffffffff8434f079>] __blk_run_queue+0x39/0x50
[<ffffffff8434f0f6>] blk_run_queue+0x26/0x40
[<ffffffff844e66c8>] scsi_run_queue+0x258/0x2f0
[<ffffffff844e8310>] scsi_next_command+0x20/0x40
[<ffffffff844e8465>] scsi_end_request+0x135/0x1e0
[<ffffffff844e86d8>] scsi_io_completion+0x168/0x6a0
[<ffffffff844ddabc>] scsi_finish_command+0xdc/0x140
[<ffffffff844e7c22>] scsi_softirq_done+0x132/0x160
[<ffffffff84358d86>] blk_done_softirq+0x96/0xc0
[<ffffffff840a5305>] __do_softirq+0xf5/0x280
[<ffffffff8479142c>] call_softirq+0x1c/0x30
[<ffffffff8402f715>] do_softirq+0x65/0xa0
[<ffffffff840a5685>] irq_exit+0x105/0x110
[<ffffffff84059469>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffff847901aa>] call_function_single_interrupt+0x16a/0x170

Complete report #13343

Frame #	Function	Binary	Source or offset	Line
1	mptspi_qcmd	mptspi	0xffffffffc04062e0 Build id: 3.10.0-862.2.3.el7.x86_64	0x50
2	scsi_dispatch_cmd	vmlinux	0xffffffff9049aff0 Build id: 3.10.0-862.2.3.el7.x86_64	0xb0
3	scsi_request_fn	vmlinux	0xffffffff904a455c Build id: 3.10.0-862.2.3.el7.x86_64	0x4cc
4	__blk_run_queue	vmlinux	0xffffffff90318a79 Build id: 3.10.0-862.2.3.el7.x86_64	0x39
5	blk_run_queue	vmlinux	0xffffffff90318af6 Build id: 3.10.0-862.2.3.el7.x86_64	0x26
6	scsi_run_queue	vmlinux	0xffffffff904a2af8 Build id: 3.10.0-862.2.3.el7.x86_64	0x258
7	scsi_next_command	vmlinux	0xffffffff904a4750 Build id: 3.10.0-862.2.3.el7.x86_64	0x20
8	scsi_end_request	vmlinux	0xffffffff904a48a5 Build id: 3.10.0-862.2.3.el7.x86_64	0x135
9	scsi_io_completion	vmlinux	0xffffffff904a4b18 Build id: 3.10.0-862.2.3.el7.x86_64	0x168
10	? __netif_receive_skb	vmlinux	0xffffffff905ebd08 Build id: 3.10.0-862.2.3.el7.x86_64	0x18
11	scsi_finish_command	vmlinux	0xffffffff90499ffc Build id: 3.10.0-862.2.3.el7.x86_64	0xdc
12	scsi_softirq_done	vmlinux	0xffffffff904a4062 Build id: 3.10.0-862.2.3.el7.x86_64	0x132
13	blk_done_softirq	vmlinux	0xffffffff90322736 Build id: 3.10.0-862.2.3.el7.x86_64	0x96
14	__do_softirq	vmlinux	0xffffffff9009a945 Build id: 3.10.0-862.2.3.el7.x86_64	0xf5
15	call_softirq	vmlinux	0xffffffff90722cec Build id: 3.10.0-862.2.3.el7.x86_64	0x1c
16	do_softirq	vmlinux	0xffffffff9002d625 Build id: 3.10.0-862.2.3.el7.x86_64	0x65
17	irq_exit	vmlinux	0xffffffff9009acc5 Build id: 3.10.0-862.2.3.el7.x86_64	0x105
18	smp_call_function_single_interrupt	vmlinux	0xffffffff90054779 Build id: 3.10.0-862.2.3.el7.x86_64	0x39
19	call_function_single_interrupt	vmlinux	0xffffffff90721a62 Build id: 3.10.0-862.2.3.el7.x86_64	0x162

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 32s! [zmlogger:4633]
[<ffffffffc04062e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff9049aff0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff904a455c>] scsi_request_fn+0x4cc/0x680
[<ffffffff90318a79>] __blk_run_queue+0x39/0x50
[<ffffffff90318af6>] blk_run_queue+0x26/0x40
[<ffffffff904a2af8>] scsi_run_queue+0x258/0x2f0
[<ffffffff904a4750>] scsi_next_command+0x20/0x40
[<ffffffff904a48a5>] scsi_end_request+0x135/0x1e0
[<ffffffff904a4b18>] scsi_io_completion+0x168/0x6a0
[<ffffffff905ebd08>] ? __netif_receive_skb+0x18/0x60
[<ffffffff90499ffc>] scsi_finish_command+0xdc/0x140
[<ffffffff904a4062>] scsi_softirq_done+0x132/0x160
[<ffffffff90322736>] blk_done_softirq+0x96/0xc0
[<ffffffff9009a945>] __do_softirq+0xf5/0x280
[<ffffffff90722cec>] call_softirq+0x1c/0x30
[<ffffffff9002d625>] do_softirq+0x65/0xa0
[<ffffffff9009acc5>] irq_exit+0x105/0x110
[<ffffffff90054779>] smp_call_function_single_interrupt+0x39/0x40
[<ffffffff90721a62>] call_function_single_interrupt+0x162/0x170