Info

Permalink
Function
mpt_put_msg_frame
First occurrence
2020-10-09
Last occurrence
2026-07-22
State
NEW
Components
kernel
Quality
-44

Statistics

Operating system Count
CentOS 7 166
Architecture Count
x86_64 166
Related packages Count
kernel 166
 0:3.10.0-1160.53.1.el7 37
 0:3.10.0-1127.19.1.el7 34
 0:3.10.0-1160.25.1.el7 15
 0:3.10.0-1160.el7 13
 0:3.10.0-1160.59.1.el7 8
 0:3.10.0-1160.42.2.el7 6
 0:3.10.0-1160.45.1.el7 5
 0:3.10.0-1160.6.1.el7 5
 0:3.10.0-1160.119.1.el7 4
 0:3.10.0-1160.21.1.el7 4
 0:3.10.0-1160.36.2.el7 4
 0:3.10.0-1160.76.1.el7 4
 0:3.10.0-1160.31.1.el7 4
 0:3.10.0-1127.18.2.el7 3
 0:3.10.0-1160.15.2.el7 3
 0:3.10.0-1160.62.1.el7 3
 0:3.10.0-1160.118.1.el7 3
 0:3.10.0-1160.49.1.el7 3
 0:3.10.0-1127.8.2.el7 2
 0:3.10.0-1160.108.1.el7 2
 0:3.10.0-1160.83.1.el7 2
 0:3.10.0-1160.41.1.el7 1
 0:3.10.0-1127.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #349532
Frame # Function Binary Source or offset Line
1
mpt_fault_reset_work
mptbase 0xffffffffc01cafa9
Build id: 3.10.0-1160.49.1.el7.x86_64
0xc9
2
mptscsih_qcmd
mptscsih 0xffffffffc023971e
Build id: 3.10.0-1160.49.1.el7.x86_64
0x24e
3
mptspi_qcmd
mptspi 0xffffffffc024d2e0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffffa60e4860
Build id: 3.10.0-1160.49.1.el7.x86_64
0xb0
5
scsi_request_fn
vmlinux 0xffffffffa60edf8c
Build id: 3.10.0-1160.49.1.el7.x86_64
0x4ac
6
__blk_run_queue
vmlinux 0xffffffffa5f53f39
Build id: 3.10.0-1160.49.1.el7.x86_64
0x39
7
queue_unplugged
vmlinux 0xffffffffa5f53ffa
Build id: 3.10.0-1160.49.1.el7.x86_64
0x2a
8
blk_flush_plug_list
vmlinux 0xffffffffa5f578d8
Build id: 3.10.0-1160.49.1.el7.x86_64
0x1d8
9
blk_finish_plug
vmlinux 0xffffffffa5f57d44
Build id: 3.10.0-1160.49.1.el7.x86_64
0x14
10
_xfs_buf_ioapply
xfs 0xffffffffc03f52c4
Build id: 3.10.0-1160.49.1.el7.x86_64
0x334
11
? xlog_bdstrat
xfs 0xffffffffc04176c7
Build id: 3.10.0-1160.49.1.el7.x86_64
0x37
12
__xfs_buf_submit
xfs 0xffffffffc03f6c72
Build id: 3.10.0-1160.49.1.el7.x86_64
0x72
13
xlog_bdstrat
xfs 0xffffffffc04176c7
Build id: 3.10.0-1160.49.1.el7.x86_64
0x37
14
xlog_sync
xfs 0xffffffffc0419526
Build id: 3.10.0-1160.49.1.el7.x86_64
0x2e6
15
xlog_state_release_iclog
xfs 0xffffffffc04196ab
Build id: 3.10.0-1160.49.1.el7.x86_64
0x7b
16
xfs_log_force_lsn
xfs 0xffffffffc041ac51
Build id: 3.10.0-1160.49.1.el7.x86_64
0x1f1
17
? filemap_fdatawait_range
vmlinux 0xffffffffa5dbd58f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x1f
18
? down_read
vmlinux 0xffffffffa6388632
Build id: 3.10.0-1160.49.1.el7.x86_64
0x12
19
xfs_file_fsync
xfs 0xffffffffc03fadfd
Build id: 3.10.0-1160.49.1.el7.x86_64
0xfd
20
do_fsync
vmlinux 0xffffffffa5e84227
Build id: 3.10.0-1160.49.1.el7.x86_64
0x67
21
SyS_fsync
vmlinux 0xffffffffa5e84510
Build id: 3.10.0-1160.49.1.el7.x86_64
0x10
22
system_call_fastpath
vmlinux 0xffffffffa6395f92
Build id: 3.10.0-1160.49.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 41s! [kworker/u16:1:20473]
Modules linked in: cmac arc4 md4 nls_utf8 cifs ccm dns_resolver iptable_nat nf_nat_ipv4 nf_nat iptable_mangle nf_log_ipv4 nf_log_common xt_LOG nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack vmw_vsock_vmci_transport vsock iptable_filter sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw vmw_balloon gf128mul glue_helper ablk_helper pcspkr cryptd sg joydev vmw_vmci parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi crct10dif_pclmul ata_piix crct10dif_common drm scsi_transport_spi crc32c_intel mptscsih libata serio_raw mptbase nfit vmxnet3 libnvdimm drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log
dm_mod fuse
CPU: 4 PID: 20473 Comm: kworker/u16:1 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: mpt_poll_0 mpt_fault_reset_work [mptbase]
task: ffff99b016642100 ti: ffff99b01ee34000 task.ti: ffff99b01ee34000
RIP: 0010:[<ffffffffc01cafa9>]  [<ffffffffc01cafa9>] mpt_fault_reset_work+0xc9/0x47a [mptbase]
RSP: 0018:ffff99b01ee37d90  EFLAGS: 00000202
RAX: ffffa8c701bc0000 RBX: 0000000000001191 RCX: 0000000000000000
RDX: 0000000024000000 RSI: 0000000000000096 RDI: ffff99b01de81d60
RBP: ffff99b01ee37e18 R08: ffffffffa641e640 R09: ffff99b016642100
R10: 993f220fa8681d60 R11: ffff99adb723ff00 R12: ffffffffa5ce594a
R13: ffff99b01ee37d40 R14: 0000000000000001 R15: ffff99b02dd1acc0
FS:  0000000000000000(0000) GS:ffff99b02dd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3071d7e000 CR3: 00000002efe10000 CR4: 00000000001607e0
iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw vmw_balloon gf128mul glue_helper ablk_helper pcspkr cryptd sg joydev vmw_vmci parport_pc parport ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi crct10dif_pclmul ata_piix crct10dif_common drm scsi_transport_spi crc32c_intel mptscsih libata serio_raw mptbase nfit vmxnet3 libnvdimm drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 11066 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff99b01d8e0000 ti: ffff99adb735c000 task.ti: ffff99adb735c000
RIP: 0010:[<ffffffffc01be3be>]  [<ffffffffc01be3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff99adb735fa70  EFLAGS: 00000246
RAX: ffffa8c701bc0000 RBX: ffff99af8ad13aa0 RCX: 0000000000000002
RDX: ffff99ad3651c200 RSI: ffff99b01de81000 RDI: 000000000000000e
RBP: ffff99adb735fa80 R08: 0000000000000003 R09: 000000031c3e0bff
R10: ffff99aff0124800 R11: ffff99b01db03800 R12: 0000000100000006
R13: ffff99b016aa9380 R14: ffff99adb735f9e4 R15: ffffffffa5f5c8fb
FS:  00007f9b241d6700(0000) GS:ffff99b02dcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007feefb3b7e50 CR3: 000000030ed8e000 CR4: 00000000001607e0
Call Trace:
[<ffffffffc023971e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc024d2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffa60e4860>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffa60edf8c>] scsi_request_fn+0x4ac/0x680
[<ffffffffa5f53f39>] __blk_run_queue+0x39/0x50
[<ffffffffa5f53ffa>] queue_unplugged+0x2a/0xa0
[<ffffffffa5f578d8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffa5f57d44>] blk_finish_plug+0x14/0x40
[<ffffffffc03f52c4>] _xfs_buf_ioapply+0x334/0x460 [xfs]
[<ffffffffc04176c7>] ? xlog_bdstrat+0x37/0x70 [xfs]
[<ffffffffc03f6c72>] __xfs_buf_submit+0x72/0x250 [xfs]
[<ffffffffc04176c7>] xlog_bdstrat+0x37/0x70 [xfs]
[<ffffffffc0419526>] xlog_sync+0x2e6/0x3f0 [xfs]
[<ffffffffc04196ab>] xlog_state_release_iclog+0x7b/0xd0 [xfs]
[<ffffffffc041ac51>] xfs_log_force_lsn+0x1f1/0x370 [xfs]
[<ffffffffa5dbd58f>] ? filemap_fdatawait_range+0x1f/0x30
[<ffffffffa6388632>] ? down_read+0x12/0x40
[<ffffffffc03fadfd>] xfs_file_fsync+0xfd/0x1c0 [xfs]
[<ffffffffa5e84227>] do_fsync+0x67/0xb0
[<ffffffffa5e84510>] SyS_fsync+0x10/0x20
[<ffffffffa6395f92>] system_call_fastpath+0x25/0x2a
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 c3 1c 
Complete report #330992
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc045d3be
Build id: 3.10.0-1160.49.1.el7.x86_64
0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc04bc71e
Build id: 3.10.0-1160.49.1.el7.x86_64
0x24e
3
mptspi_qcmd
mptspi 0xffffffffc05142e0
Build id: 3.10.0-1160.49.1.el7.x86_64
0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffff8ace4860
Build id: 3.10.0-1160.49.1.el7.x86_64
0xb0
5
scsi_request_fn
vmlinux 0xffffffff8acedf8c
Build id: 3.10.0-1160.49.1.el7.x86_64
0x4ac
6
__blk_run_queue
vmlinux 0xffffffff8ab53f39
Build id: 3.10.0-1160.49.1.el7.x86_64
0x39
7
queue_unplugged
vmlinux 0xffffffff8ab53ffa
Build id: 3.10.0-1160.49.1.el7.x86_64
0x2a
8
blk_flush_plug_list
vmlinux 0xffffffff8ab578d8
Build id: 3.10.0-1160.49.1.el7.x86_64
0x1d8
9
blk_finish_plug
vmlinux 0xffffffff8ab57d44
Build id: 3.10.0-1160.49.1.el7.x86_64
0x14
10
_xfs_buf_ioapply
xfs 0xffffffffc06ae2c4
Build id: 3.10.0-1160.49.1.el7.x86_64
0x334
11
? xlog_bdstrat
xfs 0xffffffffc06d06c7
Build id: 3.10.0-1160.49.1.el7.x86_64
0x37
12
__xfs_buf_submit
xfs 0xffffffffc06afc72
Build id: 3.10.0-1160.49.1.el7.x86_64
0x72
13
xlog_bdstrat
xfs 0xffffffffc06d06c7
Build id: 3.10.0-1160.49.1.el7.x86_64
0x37
14
xlog_sync
xfs 0xffffffffc06d2526
Build id: 3.10.0-1160.49.1.el7.x86_64
0x2e6
15
xlog_state_release_iclog
xfs 0xffffffffc06d26ab
Build id: 3.10.0-1160.49.1.el7.x86_64
0x7b
16
xfs_log_force_lsn
xfs 0xffffffffc06d3c51
Build id: 3.10.0-1160.49.1.el7.x86_64
0x1f1
17
? filemap_fdatawait_range
vmlinux 0xffffffff8a9bd58f
Build id: 3.10.0-1160.49.1.el7.x86_64
0x1f
18
? down_read
vmlinux 0xffffffff8af88632
Build id: 3.10.0-1160.49.1.el7.x86_64
0x12
19
xfs_file_fsync
xfs 0xffffffffc06b3dfd
Build id: 3.10.0-1160.49.1.el7.x86_64
0xfd
20
do_fsync
vmlinux 0xffffffff8aa84227
Build id: 3.10.0-1160.49.1.el7.x86_64
0x67
21
? system_call_after_swapgs
vmlinux 0xffffffff8af95ec9
Build id: 3.10.0-1160.49.1.el7.x86_64
0x96
22
SyS_fsync
vmlinux 0xffffffff8aa84510
Build id: 3.10.0-1160.49.1.el7.x86_64
0x10
23
system_call_fastpath
vmlinux 0xffffffff8af95f92
Build id: 3.10.0-1160.49.1.el7.x86_64
0x25
24
? system_call_after_swapgs
vmlinux 0xffffffff8af95ed5
Build id: 3.10.0-1160.49.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [nagios:70219]
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [postdrop:11879]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock ppdev pcspkr vmw_balloon joydev sg parport_pc parport i2c_piix4 vmw_vmci nfsd nfs_acl lockd grace auth_rpcgss sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi scsi_transport_spi mptscsih drm nfit ata_piix libnvdimm libata mptbase e1000 serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 11879 Comm: postdrop Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8ce3709e3180 ti: ffff8ce189ed4000 task.ti: ffff8ce189ed4000
RIP: 0010:[<ffffffffc045d3be>]  [<ffffffffc045d3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0000:ffff8ce189ed7a70  EFLAGS: 00000246
RAX: ffff9d9040d00000 RBX: ffff8ce1adba8a00 RCX: 0000000000000001
RDX: ffff8ce175d06600 RSI: ffff8ce3759ac000 RDI: 000000000000000e
RBP: ffff8ce189ed7a80 R08: 0000000000000002 R09: 000000003559d7ff
R10: ffff8ce371b46800 R11: ffff8ce363abaf40 R12: 0000000100000001
R13: ffff8ce187cde580 R14: ffff8ce189ed79e4 R15: ffffffff8ab5c8fb
FS:  00007f7c99a38880(0000) GS:ffff8ce375680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0198544db0 CR3: 000000006db02000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc04bc71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc05142e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff8ace4860>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff8acedf8c>] scsi_request_fn+0x4ac/0x680
[<ffffffff8ab53f39>] __blk_run_queue+0x39/0x50
[<ffffffff8ab53ffa>] queue_unplugged+0x2a/0xa0
[<ffffffff8ab578d8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff8ab57d44>] blk_finish_plug+0x14/0x40
[<ffffffffc06ae2c4>] _xfs_buf_ioapply+0x334/0x460 [xfs]
[<ffffffffc06d06c7>] ? xlog_bdstrat+0x37/0x70 [xfs]
[<ffffffffc06afc72>] __xfs_buf_submit+0x72/0x250 [xfs]
[<ffffffffc06d06c7>] xlog_bdstrat+0x37/0x70 [xfs]
[<ffffffffc06d2526>] xlog_sync+0x2e6/0x3f0 [xfs]
[<ffffffffc06d26ab>] xlog_state_release_iclog+0x7b/0xd0 [xfs]
[<ffffffffc06d3c51>] xfs_log_force_lsn+0x1f1/0x370 [xfs]
[<ffffffff8a9bd58f>] ? filemap_fdatawait_range+0x1f/0x30
[<ffffffff8af88632>] ? down_read+0x12/0x40
[<ffffffffc06b3dfd>] xfs_file_fsync+0xfd/0x1c0 [xfs]
[<ffffffff8aa84227>] do_fsync+0x67/0xb0
[<ffffffff8af95ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff8aa84510>] SyS_fsync+0x10/0x20
[<ffffffff8af95f92>] system_call_fastpath+0x25/0x2a
[<ffffffff8af95ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 b3 46 
Complete report #225642
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffff8b78b565
Build id: 3.10.0-1160.25.1.el7.x86_64
0x15
2
pvscsi_queue
vmw_pvscsi 0xffffffffc0239632
Build id: 3.10.0-1160.25.1.el7.x86_64
0x4a2
3
? scsi_test_unit_ready
vmlinux 0xffffffff8b4ec6b0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x170
4
scsi_dispatch_cmd
vmlinux 0xffffffff8b4e3fa0
Build id: 3.10.0-1160.25.1.el7.x86_64
0xb0
5
scsi_request_fn
vmlinux 0xffffffff8b4ed6cc
Build id: 3.10.0-1160.25.1.el7.x86_64
0x4ac
6
__blk_run_queue
vmlinux 0xffffffff8b353939
Build id: 3.10.0-1160.25.1.el7.x86_64
0x39
7
queue_unplugged
vmlinux 0xffffffff8b3539fa
Build id: 3.10.0-1160.25.1.el7.x86_64
0x2a
8
blk_flush_plug_list
vmlinux 0xffffffff8b357338
Build id: 3.10.0-1160.25.1.el7.x86_64
0x1d8
9
blk_finish_plug
vmlinux 0xffffffff8b3577a4
Build id: 3.10.0-1160.25.1.el7.x86_64
0x14
10
_xfs_buf_ioapply
xfs 0xffffffffc04672c4
Build id: 3.10.0-1160.25.1.el7.x86_64
0x334
11
? xlog_bdstrat
xfs 0xffffffffc04896c7
Build id: 3.10.0-1160.25.1.el7.x86_64
0x37
12
__xfs_buf_submit
xfs 0xffffffffc0468c72
Build id: 3.10.0-1160.25.1.el7.x86_64
0x72
13
xlog_bdstrat
xfs 0xffffffffc04896c7
Build id: 3.10.0-1160.25.1.el7.x86_64
0x37
14
xlog_sync
xfs 0xffffffffc048b4d6
Build id: 3.10.0-1160.25.1.el7.x86_64
0x2e6
15
xlog_state_release_iclog
xfs 0xffffffffc048b65b
Build id: 3.10.0-1160.25.1.el7.x86_64
0x7b
16
xfs_log_force_lsn
xfs 0xffffffffc048cc01
Build id: 3.10.0-1160.25.1.el7.x86_64
0x1f1
17
? filemap_fdatawait_range
vmlinux 0xffffffff8b1bd36f
Build id: 3.10.0-1160.25.1.el7.x86_64
0x1f
18
? down_read
vmlinux 0xffffffff8b788212
Build id: 3.10.0-1160.25.1.el7.x86_64
0x12
19
xfs_file_fsync
xfs 0xffffffffc046cdfd
Build id: 3.10.0-1160.25.1.el7.x86_64
0xfd
20
do_fsync
vmlinux 0xffffffff8b283cb7
Build id: 3.10.0-1160.25.1.el7.x86_64
0x67
21
SyS_fsync
vmlinux 0xffffffff8b283fa0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x10
22
system_call_fastpath
vmlinux 0xffffffff8b795f92
Build id: 3.10.0-1160.25.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [nagios:30710]
Modules linked in: vsock_diag tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter ipmi_devintf ipmi_msghandler vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg ppdev joydev pcspkr vmw_balloon i2c_piix4 parport_pc parport vmw_vmci binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper crct10dif_pclmul crct10dif_common e1000e syscopyarea sysfillrect
crc32c_intel serio_raw sysimgblt ahci fb_sys_fops ttm libahci ptp pps_core drm nfit ata_piix vmw_pvscsi libnvdimm libata drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 30710 Comm: nagios Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
task: ffff94d61a579080 ti: ffff94d3307a8000 task.ti: ffff94d3307a8000
RIP: 0010:[<ffffffff8b78b565>]  [<ffffffff8b78b565>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff94d3307abad8  EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffff94d3307aba98 RCX: ffffffff8bc34f40
RDX: ffffbcc8036c0000 RSI: 0000000000000286 RDI: 0000000000000286
RBP: ffff94d3307abad8 R08: 0000000000000001 R09: 0000000000000000
R10: ffff94d329fac800 R11: ffff94d9ffca9800 R12: 0000000000000020
R13: ffff94d38b63fe00 R14: ffffffff8b4eb369 R15: ffff94d3307aba50
FS:  00007fa5c65f0740(0000) GS:ffff94d9ffc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fce33a5a000 CR3: 000000024533c000 CR4: 00000000003607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc0239632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffff8b4ec6b0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffff8b4e3fa0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff8b4ed6cc>] scsi_request_fn+0x4ac/0x680
[<ffffffff8b353939>] __blk_run_queue+0x39/0x50
[<ffffffff8b3539fa>] queue_unplugged+0x2a/0xa0
[<ffffffff8b357338>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff8b3577a4>] blk_finish_plug+0x14/0x40
[<ffffffffc04672c4>] _xfs_buf_ioapply+0x334/0x460 [xfs]
[<ffffffffc04896c7>] ? xlog_bdstrat+0x37/0x70 [xfs]
[<ffffffffc0468c72>] __xfs_buf_submit+0x72/0x250 [xfs]
[<ffffffffc04896c7>] xlog_bdstrat+0x37/0x70 [xfs]
[<ffffffffc048b4d6>] xlog_sync+0x2e6/0x3f0 [xfs]
[<ffffffffc048b65b>] xlog_state_release_iclog+0x7b/0xd0 [xfs]
[<ffffffffc048cc01>] xfs_log_force_lsn+0x1f1/0x370 [xfs]
[<ffffffff8b1bd36f>] ? filemap_fdatawait_range+0x1f/0x30
[<ffffffff8b788212>] ? down_read+0x12/0x40
[<ffffffffc046cdfd>] xfs_file_fsync+0xfd/0x1c0 [xfs]
[<ffffffff8b283cb7>] do_fsync+0x67/0xb0
[<ffffffff8b283fa0>] SyS_fsync+0x10/0x20
[<ffffffff8b795f92>] system_call_fastpath+0x25/0x2a
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #106635
Frame # Function Binary Source or offset Line
1
_raw_spin_unlock_irqrestore
vmlinux 0xffffffffbb1883c5
Build id: 3.10.0-1127.19.1.el7.x86_64
0x15
2
pvscsi_queue
vmw_pvscsi 0xffffffffc005d632
Build id: 3.10.0-1127.19.1.el7.x86_64
0x4a2
3
? scsi_test_unit_ready
vmlinux 0xffffffffbaeeb0a0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x170
4
scsi_dispatch_cmd
vmlinux 0xffffffffbaee2990
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb0
5
scsi_request_fn
vmlinux 0xffffffffbaeec0bc
Build id: 3.10.0-1127.19.1.el7.x86_64
0x4ac
6
__blk_run_queue
vmlinux 0xffffffffbad527a9
Build id: 3.10.0-1127.19.1.el7.x86_64
0x39
7
queue_unplugged
vmlinux 0xffffffffbad5286a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2a
8
blk_flush_plug_list
vmlinux 0xffffffffbad560a8
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1d8
9
blk_finish_plug
vmlinux 0xffffffffbad56514
Build id: 3.10.0-1127.19.1.el7.x86_64
0x14
10
_xfs_buf_ioapply
xfs 0xffffffffc0324334
Build id: 3.10.0-1127.19.1.el7.x86_64
0x334
11
? xlog_bdstrat
xfs 0xffffffffc0346617
Build id: 3.10.0-1127.19.1.el7.x86_64
0x37
12
__xfs_buf_submit
xfs 0xffffffffc0325ce2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x72
13
xlog_bdstrat
xfs 0xffffffffc0346617
Build id: 3.10.0-1127.19.1.el7.x86_64
0x37
14
xlog_sync
xfs 0xffffffffc0348426
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2e6
15
xlog_state_release_iclog
xfs 0xffffffffc03485ab
Build id: 3.10.0-1127.19.1.el7.x86_64
0x7b
16
xfs_log_force_lsn
xfs 0xffffffffc0349b51
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1f1
17
? filemap_fdatawait_range
vmlinux 0xffffffffbabbd90f
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1f
18
? down_read
vmlinux 0xffffffffbb1850a2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x12
19
xfs_file_fsync
xfs 0xffffffffc0329e6d
Build id: 3.10.0-1127.19.1.el7.x86_64
0xfd
20
do_fsync
vmlinux 0xffffffffbac82f57
Build id: 3.10.0-1127.19.1.el7.x86_64
0x67
21
SyS_fsync
vmlinux 0xffffffffbac83240
Build id: 3.10.0-1127.19.1.el7.x86_64
0x10
22
system_call_fastpath
vmlinux 0xffffffffbb192ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [9_dirty_io_sche:12830]
Modules linked in: vsock_diag tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter ipmi_devintf ipmi_msghandler vmw_vsock_vmci_transport vsock sunrpc sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd pcspkr joydev vmw_balloon sg parport_pc parport i2c_piix4 vmw_vmci binfmt_misc ip_tables xfs libcrc32c ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix ahci
drm libahci crct10dif_pclmul crct10dif_common e1000e nfit crc32c_intel libnvdimm libata serio_raw ptp pps_core vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 12830 Comm: 9_dirty_io_sche Kdump: loaded Tainted: G             L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8e7509398000 ti: ffff8e773bd48000 task.ti: ffff8e773bd48000
RIP: 0010:[<ffffffffbb1883c5>]  [<ffffffffbb1883c5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8e773bd4bad8  EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffff8e773bd4ba98 RCX: ffffffffbb634e40
RDX: ffffa64701b90000 RSI: 0000000000000286 RDI: 0000000000000286
RBP: ffff8e773bd4bad8 R08: 0000000000000001 R09: 0000000000000000
R10: ffff8e78e9709c00 R11: ffff8e78e9928000 R12: 0000000000000020
R13: ffff8e78acfdd080 R14: ffffffffbaee9d59 R15: ffff8e773bd4ba50
FS:  00007f5971efb700(0000) GS:ffff8e78ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000006dee20 CR3: 00000001a1de0000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc005d632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffffbaeeb0a0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffffbaee2990>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffbaeec0bc>] scsi_request_fn+0x4ac/0x680
[<ffffffffbad527a9>] __blk_run_queue+0x39/0x50
[<ffffffffbad5286a>] queue_unplugged+0x2a/0xa0
[<ffffffffbad560a8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffbad56514>] blk_finish_plug+0x14/0x40
[<ffffffffc0324334>] _xfs_buf_ioapply+0x334/0x460 [xfs]
[<ffffffffc0346617>] ? xlog_bdstrat+0x37/0x70 [xfs]
[<ffffffffc0325ce2>] __xfs_buf_submit+0x72/0x250 [xfs]
[<ffffffffc0346617>] xlog_bdstrat+0x37/0x70 [xfs]
[<ffffffffc0348426>] xlog_sync+0x2e6/0x3f0 [xfs]
[<ffffffffc03485ab>] xlog_state_release_iclog+0x7b/0xd0 [xfs]
[<ffffffffc0349b51>] xfs_log_force_lsn+0x1f1/0x370 [xfs]
[<ffffffffbabbd90f>] ? filemap_fdatawait_range+0x1f/0x30
[<ffffffffbb1850a2>] ? down_read+0x12/0x40
[<ffffffffc0329e6d>] xfs_file_fsync+0xfd/0x1c0 [xfs]
[<ffffffffbac82f57>] do_fsync+0x67/0xb0
[<ffffffffbac83240>] SyS_fsync+0x10/0x20
[<ffffffffbb192ed2>] system_call_fastpath+0x25/0x2a
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48 
Complete report #7108
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc01aa3be
Build id: 3.10.0-1127.19.1.el7.x86_64
0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc01d171e
Build id: 3.10.0-1127.19.1.el7.x86_64
0x24e
3
mptspi_qcmd
mptspi 0xffffffffc010f2e0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffff844e2990
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb0
5
scsi_request_fn
vmlinux 0xffffffff844ec0bc
Build id: 3.10.0-1127.19.1.el7.x86_64
0x4ac
6
__blk_run_queue
vmlinux 0xffffffff843527a9
Build id: 3.10.0-1127.19.1.el7.x86_64
0x39
7
queue_unplugged
vmlinux 0xffffffff8435286a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2a
8
blk_flush_plug_list
vmlinux 0xffffffff843560a8
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1d8
9
blk_finish_plug
vmlinux 0xffffffff84356514
Build id: 3.10.0-1127.19.1.el7.x86_64
0x14
10
_xfs_buf_ioapply
xfs 0xffffffffc0334334
Build id: 3.10.0-1127.19.1.el7.x86_64
0x334
11
? xlog_bdstrat
xfs 0xffffffffc0356617
Build id: 3.10.0-1127.19.1.el7.x86_64
0x37
12
__xfs_buf_submit
xfs 0xffffffffc0335ce2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x72
13
xlog_bdstrat
xfs 0xffffffffc0356617
Build id: 3.10.0-1127.19.1.el7.x86_64
0x37
14
xlog_sync
xfs 0xffffffffc0358426
Build id: 3.10.0-1127.19.1.el7.x86_64
0x2e6
15
xlog_state_release_iclog
xfs 0xffffffffc03585ab
Build id: 3.10.0-1127.19.1.el7.x86_64
0x7b
16
xfs_log_force_lsn
xfs 0xffffffffc0359b51
Build id: 3.10.0-1127.19.1.el7.x86_64
0x1f1
17
? down_read
vmlinux 0xffffffff847850a2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x12
18
xfs_dir_fsync
xfs 0xffffffffc033988a
Build id: 3.10.0-1127.19.1.el7.x86_64
0x7a
19
do_fsync
vmlinux 0xffffffff84282f57
Build id: 3.10.0-1127.19.1.el7.x86_64
0x67
20
SyS_fsync
vmlinux 0xffffffff84283240
Build id: 3.10.0-1127.19.1.el7.x86_64
0x10
21
system_call_fastpath
vmlinux 0xffffffff84792ed2
Build id: 3.10.0-1127.19.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [exim:20076]
Modules linked in: tcp_diag udp_diag inet_diag unix_diag ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 ip6table_filter nf_log_ipv4 nf_log_common xt_owner nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi libata scsi_transport_spi mptscsih crct10dif_pclmul crct10dif_common drm_panel_orientation_quirks mptbase crc32c_intel serio_raw vmxnet3 floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 20076 Comm: exim Kdump: loaded Tainted: G             L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff98c95144e2a0 ti: ffff98c5e9f94000 task.ti: ffff98c5e9f94000
RIP: 0010:[<ffffffffc01aa3be>]  [<ffffffffc01aa3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff98c5e9f97a90  EFLAGS: 00000246
RAX: ffffb29d01c00000 RBX: ffff98c96e65e800 RCX: 000000000000001a
RDX: ffff98c718ccb400 RSI: ffff98c96beb3000 RDI: 000000000000000e
RBP: ffff98c5e9f97aa0 R08: 0000000000000002 R09: 00000003b57e3fff
R10: ffff98c96b3f8400 R11: ffff98c6f17e7180 R12: 0000000100000001
R13: ffff98c906012d80 R14: ffff98c5e9f97a04 R15: ffffffff8435b0cb
FS:  00007fa802483840(0000) GS:ffff98c97fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000505769 CR3: 0000000035a96000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc01d171e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc010f2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff844e2990>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff844ec0bc>] scsi_request_fn+0x4ac/0x680
[<ffffffff843527a9>] __blk_run_queue+0x39/0x50
[<ffffffff8435286a>] queue_unplugged+0x2a/0xa0
[<ffffffff843560a8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff84356514>] blk_finish_plug+0x14/0x40
[<ffffffffc0334334>] _xfs_buf_ioapply+0x334/0x460 [xfs]
[<ffffffffc0356617>] ? xlog_bdstrat+0x37/0x70 [xfs]
[<ffffffffc0335ce2>] __xfs_buf_submit+0x72/0x250 [xfs]
[<ffffffffc0356617>] xlog_bdstrat+0x37/0x70 [xfs]
[<ffffffffc0358426>] xlog_sync+0x2e6/0x3f0 [xfs]
[<ffffffffc03585ab>] xlog_state_release_iclog+0x7b/0xd0 [xfs]
[<ffffffffc0359b51>] xfs_log_force_lsn+0x1f1/0x370 [xfs]
[<ffffffff847850a2>] ? down_read+0x12/0x40
[<ffffffffc033988a>] xfs_dir_fsync+0x7a/0xe0 [xfs]
[<ffffffff84282f57>] do_fsync+0x67/0xb0
[<ffffffff84283240>] SyS_fsync+0x10/0x20
[<ffffffff84792ed2>] system_call_fastpath+0x25/0x2a
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 83 1b