Info

Permalink
Function
mpt_put_msg_frame
First occurrence
2020-10-09
Last occurrence
2024-05-04
State
NEW
Components
kernel
Quality
-17

Statistics

Operating system Count
CentOS 7 47
Architecture Count
x86_64 47
Related packages Count
kernel 47
 0:3.10.0-957.12.2.el7 12
 0:3.10.0-862.el7 9
 0:3.10.0-1160.45.1.el7 5
 0:3.10.0-1160.el7 4
 0:3.10.0-1127.10.1.el7 4
 0:3.10.0-1160.11.1.el7 2
 0:3.10.0-957.el7 2
 0:3.10.0-1062.el7 2
 0:3.10.0-1160.119.1.el7 2
 0:3.10.0-1127.19.1.el7 1
 0:3.10.0-1160.71.1.el7 1
 0:3.10.0-1160.80.1.el7 1
 0:3.10.0-1160.66.1.el7 1
 0:3.10.0-1160.41.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #324057
Frame # Function Binary Source or offset Line
1
mptscsih_qcmd
mptscsih 0xffffffffc031071e
Build id: 3.10.0-957.el7.x86_64
0x24e
2
mptspi_qcmd
mptspi 0xffffffffc02b52e0
Build id: 3.10.0-957.el7.x86_64
0x50
3
scsi_dispatch_cmd
vmlinux 0xffffffff8ead0960
Build id: 3.10.0-957.el7.x86_64
0xb0
4
scsi_request_fn
vmlinux 0xffffffff8ead9efc
Build id: 3.10.0-957.el7.x86_64
0x4cc
5
? deadline_add_request
vmlinux 0xffffffff8e9675fe
Build id: 3.10.0-957.el7.x86_64
0x6e
6
__blk_run_queue
vmlinux 0xffffffff8e942ff9
Build id: 3.10.0-957.el7.x86_64
0x39
7
blk_queue_bio
vmlinux 0xffffffff8e946cf3
Build id: 3.10.0-957.el7.x86_64
0x3b3
8
generic_make_request
vmlinux 0xffffffff8e944c87
Build id: 3.10.0-957.el7.x86_64
0x147
9
submit_bio
vmlinux 0xffffffff8e944f30
Build id: 3.10.0-957.el7.x86_64
0x70
10
? bio_alloc_bioset
vmlinux 0xffffffff8e87e2e5
Build id: 3.10.0-957.el7.x86_64
0x115
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 40s! [kworker/u256:1:12427]
RSP: 0018:ffff944dbd373748  EFLAGS: 00000246
RAX: ffffbe13c03c0000 RBX: ffff944df896b188 RCX: 0000000000000003
RDX: ffff944db5ebd800 RSI: ffff944db5ea9000 RDI: 000000000000000e
RBP: ffff944dbd373758 R08: 0000000000000002 R09: 000000005a506fff
R10: ffff944db5f340d8 R11: ffff944df44e1c00 R12: ffff944df6c92b80
R13: ffff944dbd3736bc R14: ffffffff8e94b83b R15: ffff944dbd373700
FS:  00007fdae695e740(0000) GS:ffff944dfb600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007efecdcc3100 CR3: 0000000049298000 CR4: 00000000003607f0
Call Trace:
[<ffffffffc031071e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc02b52e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff8ead0960>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff8ead9efc>] scsi_request_fn+0x4cc/0x680
[<ffffffff8e9675fe>] ? deadline_add_request+0x6e/0x90
[<ffffffff8e942ff9>] __blk_run_queue+0x39/0x50
[<ffffffff8e946cf3>] blk_queue_bio+0x3b3/0x400
[<ffffffff8e944c87>] generic_make_request+0x147/0x380
[<ffffffff8e944f30>] submit_bio+0x70/0x150
[<ffffffff8e87e2e5>] ? bio_alloc_bioset+0x115/0x310
Complete report #220881
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc019f3be
Build id: 3.10.0-1062.el7.x86_64
0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc01ba71e
Build id: 3.10.0-1062.el7.x86_64
0x24e
3
mptspi_qcmd
mptspi 0xffffffffc024d2e0
Build id: 3.10.0-1062.el7.x86_64
0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffff844dc890
Build id: 3.10.0-1062.el7.x86_64
0xb0
5
scsi_request_fn
vmlinux 0xffffffff844e5efc
Build id: 3.10.0-1062.el7.x86_64
0x4cc
6
? deadline_add_request
vmlinux 0xffffffff843716de
Build id: 3.10.0-1062.el7.x86_64
0x6e
7
__blk_run_queue
vmlinux 0xffffffff8434cf59
Build id: 3.10.0-1062.el7.x86_64
0x39
8
blk_queue_bio
vmlinux 0xffffffff84350c63
Build id: 3.10.0-1062.el7.x86_64
0x3b3
9
generic_make_request
vmlinux 0xffffffff8434ebf7
Build id: 3.10.0-1062.el7.x86_64
0x147
10
submit_bio
vmlinux 0xffffffff8434eea0
Build id: 3.10.0-1062.el7.x86_64
0x70
11
? bio_alloc_bioset
vmlinux 0xffffffff84286325
Build id: 3.10.0-1062.el7.x86_64
0x115
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 31s! [vmtoolsd:1139]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_pcm joydev pcspkr snd_timer snd sg soundcore vmw_vmci i2c_piix4 parport_pc parport sch_fq_codel ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea nfit
sysfillrect sysimgblt fb_sys_fops libnvdimm ttm ata_piix crct10dif_pclmul crct10dif_common drm mptspi scsi_transport_spi crc32c_intel libata mptscsih mptbase e1000 serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1139 Comm: vmtoolsd Kdump: loaded Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9b84b5658000 ti: ffff9b86ceecc000 task.ti: ffff9b86ceecc000
RIP: 0010:[<ffffffffc019f3be>]  [<ffffffffc019f3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff9b86ceecf768  EFLAGS: 00000246
RAX: ffffbc2281940000 RBX: ffff9b8676ed97a0 RCX: 000000000000000e
RDX: ffff9b86d4631400 RSI: ffff9b86d5923000 RDI: 000000000000000e
RBP: ffff9b86ceecf778 R08: 0000000000000002 R09: 000000021a2c3fff
R10: ffff9b86db4ec0d8 R11: ffff9b86bdf79c00 R12: ffff9b8654ef0300
R13: ffff9b86ceecf6dc R14: ffffffff843557ab R15: ffff9b86ceecf720
FS:  00007fab6ba3e880(0000) GS:ffff9b86d9840000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002234f28 CR3: 000000024ef6e000 CR4: 00000000003607e0
Call Trace:
[<ffffffffc01ba71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc024d2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff844dc890>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff844e5efc>] scsi_request_fn+0x4cc/0x680
[<ffffffff843716de>] ? deadline_add_request+0x6e/0x90
[<ffffffff8434cf59>] __blk_run_queue+0x39/0x50
[<ffffffff84350c63>] blk_queue_bio+0x3b3/0x400
[<ffffffff8434ebf7>] generic_make_request+0x147/0x380
[<ffffffff8434eea0>] submit_bio+0x70/0x150
[<ffffffff84286325>] ? bio_alloc_bioset+0x115/0x310
Complete report #1399318
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc018644e
Build id: 3.10.0-1160.119.1.el7.x86_64
0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc01a674e
Build id: 3.10.0-1160.119.1.el7.x86_64
0x24e
3
mptspi_qcmd
mptspi 0xffffffffc00c32f0
Build id: 3.10.0-1160.119.1.el7.x86_64
0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffffae503030
Build id: 3.10.0-1160.119.1.el7.x86_64
0xb0
5
scsi_request_fn
vmlinux 0xffffffffae50cbdc
Build id: 3.10.0-1160.119.1.el7.x86_64
0x4ac
6
? deadline_add_request
vmlinux 0xffffffffae38e21e
Build id: 3.10.0-1160.119.1.el7.x86_64
0x6e
7
__blk_run_queue
vmlinux 0xffffffffae3687d9
Build id: 3.10.0-1160.119.1.el7.x86_64
0x39
8
blk_queue_bio
vmlinux 0xffffffffae36c703
Build id: 3.10.0-1160.119.1.el7.x86_64
0x3b3
9
generic_make_request
vmlinux 0xffffffffae36a537
Build id: 3.10.0-1160.119.1.el7.x86_64
0x157
10
submit_bio
vmlinux 0xffffffffae36a7e0
Build id: 3.10.0-1160.119.1.el7.x86_64
0x70
11
? sched_clock_cpu
vmlinux 0xffffffffae0e4345
Build id: 3.10.0-1160.119.1.el7.x86_64
0x85
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kworker/u8:0:20701]
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [mysqld:3020]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter binfmt_misc overlay(T) vmw_vsock_vmci_transport vsock sunrpc nfit libnvdimm coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr joydev vmw_balloon sg parport_pc i2c_piix4 parport vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx ata_piix drm_kms_helper syscopyarea sysfillrect sysimgblt mptspi fb_sys_fops libata ttm
scsi_transport_spi crct10dif_pclmul mptscsih crct10dif_common mptbase drm crc32c_intel serio_raw vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 3020 Comm: mysqld Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff91abb7f21080 ti: ffff91acf8f38000 task.ti: ffff91acf8f38000
RIP: 0010:[<ffffffffc018644e>]  [<ffffffffc018644e>] mpt_put_msg_frame+0x5e/0x90 [mptbase]
RSP: 0018:ffff91acf8f3bb08  EFLAGS: 00000246
RAX: ffffa23f40f40000 RBX: ffff91abb7d11000 RCX: 000000000000000a
RDX: ffff91ab35e36e00 RSI: ffff91ad32a5c000 RDI: 000000000000000e
RBP: ffff91acf8f3bb18 R08: 0000000000000002 R09: 00000002083b0fff
R10: ffff91ad32704c00 R11: ffff91ab6e866e00 R12: ffff91ac0293c300
R13: ffff91acf8f3ba7c R14: ffffffffae37158b R15: ffff91acf8f3bac0
FS:  00007f45012aa700(0000) GS:ffff91ad3fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f146c90a530 CR3: 0000000234cf0000 CR4: 00000000003607f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc01a674e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc00c32f0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffae503030>] scsi_dispatch_cmd+0xb0/0x250
[<ffffffffae50cbdc>] scsi_request_fn+0x4ac/0x680
[<ffffffffae38e21e>] ? deadline_add_request+0x6e/0x90
[<ffffffffae3687d9>] __blk_run_queue+0x39/0x50
[<ffffffffae36c703>] blk_queue_bio+0x3b3/0x400
[<ffffffffae36a537>] generic_make_request+0x157/0x390
[<ffffffffae36a7e0>] submit_bio+0x70/0x150
[<ffffffffae0e4345>] ? sched_clock_cpu+0x85/0xc0
Complete report #735332
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc00a43be
Build id: 3.10.0-862.el7.x86_64
0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc011b71e
Build id: 3.10.0-862.el7.x86_64
0x24e
3
mptspi_qcmd
mptspi 0xffffffffc00bf2e0
Build id: 3.10.0-862.el7.x86_64
0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffff8d29b050
Build id: 3.10.0-862.el7.x86_64
0xb0
5
scsi_request_fn
vmlinux 0xffffffff8d2a45bc
Build id: 3.10.0-862.el7.x86_64
0x4cc
6
? deadline_add_request
vmlinux 0xffffffff8d13c84e
Build id: 3.10.0-862.el7.x86_64
0x6e
7
__blk_run_queue
vmlinux 0xffffffff8d118ad9
Build id: 3.10.0-862.el7.x86_64
0x39
8
blk_queue_bio
vmlinux 0xffffffff8d11c763
Build id: 3.10.0-862.el7.x86_64
0x3b3
9
generic_make_request
vmlinux 0xffffffff8d11a6bb
Build id: 3.10.0-862.el7.x86_64
0x10b
10
submit_bio
vmlinux 0xffffffff8d11a940
Build id: 3.10.0-862.el7.x86_64
0x70
11
? __schedule
vmlinux 0xffffffff8d512b08
Build id: 3.10.0-862.el7.x86_64
0x608
12
? __schedule
vmlinux 0xffffffff8d512b14
Build id: 3.10.0-862.el7.x86_64
0x614
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 42s! [kworker/u256:2:12425]
Modules linked in: fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event ext4 mbcache jbd2 crc32_pclmul ghash_clmulni_intel ppdev snd_ens1371 aesni_intel snd_rawmidi lrw gf128mul glue_helper ablk_helper cryptd snd_ac97_codec ac97_bus snd_seq vmw_balloon snd_seq_device joydev pcspkr snd_pcm snd_timer snd nfit sg soundcore libnvdimm vmw_vmci i2c_piix4 shpchp parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel drm serio_raw e1000 mptspi ata_piix scsi_transport_spi mptscsih libata mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 12425 Comm: kworker/u256:2 Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff95f7324f4f10 ti: ffff95f736910000 task.ti: ffff95f736910000
RIP: 0010:[<ffffffffc00a43be>]  [<ffffffffc00a43be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff95f736913888  EFLAGS: 00000246
RAX: ffffba29c06c0000 RBX: ffff95f74b653800 RCX: 000000000000000d
RDX: ffff95f6f5c58200 RSI: ffff95f7f18d6000 RDI: 000000000000000e
RBP: ffff95f736913898 R08: 0000000000000002 R09: 000000012871cfff
R10: ffff95f6f56c30d8 R11: ffff95f7ee462f40 R12: ffff95f74ad8b180
R13: ffff95f7369137fc R14: ffffffff8d1212cb R15: ffff95f736913840
FS:  0000000000000000(0000) GS:ffff95f7f9600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbce9841000 CR3: 0000000133aac000 CR4: 00000000007607f0
PKRU: 00000000
Call Trace:
[<ffffffffc011b71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc00bf2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff8d29b050>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff8d2a45bc>] scsi_request_fn+0x4cc/0x680
[<ffffffff8d13c84e>] ? deadline_add_request+0x6e/0x90
[<ffffffff8d118ad9>] __blk_run_queue+0x39/0x50
[<ffffffff8d11c763>] blk_queue_bio+0x3b3/0x400
[<ffffffff8d11a6bb>] generic_make_request+0x10b/0x320
[<ffffffff8d11a940>] submit_bio+0x70/0x150
[<ffffffff8d512b08>] ? __schedule+0x608/0xa20
[<ffffffff8d512b14>] ? __schedule+0x614/0xa20
Complete report #343393
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc028d3be
Build id: 3.10.0-1160.41.1.el7.x86_64
0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc02b471e
Build id: 3.10.0-1160.41.1.el7.x86_64
0x24e
3
mptspi_qcmd
mptspi 0xffffffffc02cc2e0
Build id: 3.10.0-1160.41.1.el7.x86_64
0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffffb1ce4840
Build id: 3.10.0-1160.41.1.el7.x86_64
0xb0
5
scsi_request_fn
vmlinux 0xffffffffb1cedf6c
Build id: 3.10.0-1160.41.1.el7.x86_64
0x4ac
6
? deadline_add_request
vmlinux 0xffffffffb1b78bce
Build id: 3.10.0-1160.41.1.el7.x86_64
0x6e
7
__blk_run_queue
vmlinux 0xffffffffb1b53f19
Build id: 3.10.0-1160.41.1.el7.x86_64
0x39
8
blk_queue_bio
vmlinux 0xffffffffb1b57cc3
Build id: 3.10.0-1160.41.1.el7.x86_64
0x3b3
9
generic_make_request
vmlinux 0xffffffffb1b55ba7
Build id: 3.10.0-1160.41.1.el7.x86_64
0x147
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [kworker/u128:0:18268]
Modules linked in: br_netfilter xt_nat xt_recent xt_mark xt_comment xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock overlay(T) sunrpc snd_seq_midi snd_seq_midi_event coretemp snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus iosf_mbi snd_seq crc32_pclmul ghash_clmulni_intel aesni_intel snd_seq_device snd_pcm lrw gf128mul glue_helper ablk_helper snd_timer snd joydev pcspkr cryptd ppdev vmw_balloon soundcore sg parport_pc parport i2c_piix4 vmw_vmci sch_fq_codel binfmt_misc ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif
crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci nfit libahci ata_piix drm libnvdimm crct10dif_pclmul libata crct10dif_common mptspi crc32c_intel scsi_transport_spi mptscsih serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 18268 Comm: kworker/u128:0 Tainted: G        W    L ------------ T 3.10.0-1160.41.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff93a1fbae9080 ti: ffff93925d038000 task.ti: ffff93925d038000
RIP: 0010:[<ffffffffc028d3be>]  [<ffffffffc028d3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 24s! [xfsaild/dm-0:589]
RSP: 0018:ffff93925d03b888  EFLAGS: 00000246
RAX: ffffa7d906480000 RBX: ffff939fbcea59d0 RCX: 0000000000000001
RDX: ffff93a1f8825600 RSI: ffff93a1f91d0000 RDI: 000000000000000e
RBP: ffff93925d03b898 R08: 0000000000000003 R09: 0000000ff88e2fff
R10: ffff93935933d400 R11: ffff939eb4c664c0 R12: ffff93a17159c900
R13: ffff93925d03b7fc R14: ffffffffb1b5c8db R15: ffff93925d03b840
FS:  0000000000000000(0000) GS:ffff93a1fe6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f55876f4c60 CR3: 0000000f97846000 CR4: 00000000001607e0
Modules linked in: br_netfilter xt_nat xt_recent xt_mark xt_comment xt_addrtype xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock overlay(T) sunrpc snd_seq_midi snd_seq_midi_event coretemp snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus iosf_mbi snd_seq crc32_pclmul ghash_clmulni_intel aesni_intel snd_seq_device snd_pcm lrw gf128mul glue_helper ablk_helper snd_timer snd joydev pcspkr cryptd ppdev vmw_balloon soundcore sg parport_pc parport i2c_piix4 vmw_vmci sch_fq_codel binfmt_misc ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif
Call Trace:
[<ffffffffc02b471e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc02cc2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffb1ce4840>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb1cedf6c>] scsi_request_fn+0x4ac/0x680
[<ffffffffb1b78bce>] ? deadline_add_request+0x6e/0x90
[<ffffffffb1b53f19>] __blk_run_queue+0x39/0x50
[<ffffffffb1b57cc3>] blk_queue_bio+0x3b3/0x400
[<ffffffffb1b55ba7>] generic_make_request+0x147/0x380
Complete report #211132
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc00903be
Build id: 3.10.0-862.el7.x86_64
0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc012771e
Build id: 3.10.0-862.el7.x86_64
0x24e
3
mptspi_qcmd
mptspi 0xffffffffc011c2e0
Build id: 3.10.0-862.el7.x86_64
0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffffa8a9b050
Build id: 3.10.0-862.el7.x86_64
0xb0
5
scsi_request_fn
vmlinux 0xffffffffa8aa45bc
Build id: 3.10.0-862.el7.x86_64
0x4cc
6
? deadline_add_request
vmlinux 0xffffffffa893c84e
Build id: 3.10.0-862.el7.x86_64
0x6e
7
__blk_run_queue
vmlinux 0xffffffffa8918ad9
Build id: 3.10.0-862.el7.x86_64
0x39
8
blk_queue_bio
vmlinux 0xffffffffa891c763
Build id: 3.10.0-862.el7.x86_64
0x3b3
9
generic_make_request
vmlinux 0xffffffffa891a6bb
Build id: 3.10.0-862.el7.x86_64
0x10b
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 60s! [kube-scheduler:5118]
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 62s! [kube-controller:5102]
Modules linked in: veth vxlan ip6_udp_tunnel udp_tunnel rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace ip6table_nat fscache nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_conntrack_netlink br_netfilter overlay(T) xt_addrtype xt_set xt_mark ip_set_hash_ipportip ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipport ip_set nfnetlink dummy xt_comment xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper
sg cryptd ppdev pcspkr vmw_balloon joydev vmw_vmci i2c_piix4 parport_pc parport shpchp sch_fq_codel ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif crct10dif_generic cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci mptspi scsi_transport_spi ata_piix libahci mptscsih libata mptbase crct10dif_pclmul crct10dif_common crc32c_intel serio_raw i2c_core vmxnet3 dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 5102 Comm: kube-controller Kdump: loaded Tainted: G             L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff99b074bd6eb0 ti: ffff99b0b7fd8000 task.ti: ffff99b0b7fd8000
RIP: 0010:[<ffffffffc00903be>]  [<ffffffffc00903be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff99b0b7fdbae8  EFLAGS: 00000246
RAX: ffffb578c1c80000 RBX: ffff99acdd5f6000 RCX: 0000000000000007
RDX: ffff99ae7fd45c00 RSI: ffff99b0ecb30000 RDI: 000000000000000e
RBP: ffff99b0b7fdbaf8 R08: 0000000000000002 R09: 00000001a5e33fff
R10: ffff99b0ec7a90d8 R11: ffff99ad17d4a840 R12: ffff99b016beac00
R13: ffff99b0b7fdba5c R14: ffffffffa89212cb R15: ffff99b0b7fdbaa0
FS:  000000c0001ab210(0000) GS:ffff99b0ffc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8313c6c000 CR3: 0000000411aac000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc012771e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc011c2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffa8a9b050>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffa8aa45bc>] scsi_request_fn+0x4cc/0x680
[<ffffffffa893c84e>] ? deadline_add_request+0x6e/0x90
[<ffffffffa8918ad9>] __blk_run_queue+0x39/0x50
[<ffffffffa891c763>] blk_queue_bio+0x3b3/0x400
[<ffffffffa891a6bb>] generic_make_request+0x10b/0x320
Complete report #14593
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc032b3be
Build id: 3.10.0-1127.10.1.el7.x86_64
0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc037871e
Build id: 3.10.0-1127.10.1.el7.x86_64
0x24e
3
mptspi_qcmd
mptspi 0xffffffffc038e2e0
Build id: 3.10.0-1127.10.1.el7.x86_64
0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffff9dce2920
Build id: 3.10.0-1127.10.1.el7.x86_64
0xb0
5
scsi_request_fn
vmlinux 0xffffffff9dcec04c
Build id: 3.10.0-1127.10.1.el7.x86_64
0x4ac
6
? deadline_add_request
vmlinux 0xffffffff9db7716e
Build id: 3.10.0-1127.10.1.el7.x86_64
0x6e
7
__blk_run_queue
vmlinux 0xffffffff9db52769
Build id: 3.10.0-1127.10.1.el7.x86_64
0x39
8
blk_queue_bio
vmlinux 0xffffffff9db56473
Build id: 3.10.0-1127.10.1.el7.x86_64
0x3b3
9
generic_make_request
vmlinux 0xffffffff9db543f7
Build id: 3.10.0-1127.10.1.el7.x86_64
0x147
10
submit_bio
vmlinux 0xffffffff9db546a0
Build id: 3.10.0-1127.10.1.el7.x86_64
0x70
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [dockerd:1550]
Modules linked in: tcp_lp xt_nat veth vxlan ip6_udp_tunnel udp_tunnel xt_mark nf_conntrack_netlink nfnetlink xt_CHECKSUM iptable_mangle xt_addrtype br_netfilter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter overlay(T) fuse vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev snd_ens1371 vmw_balloon snd_rawmidi aesni_intel snd_ac97_codec lrw gf128mul ac97_bus glue_helper ablk_helper cryptd snd_seq snd_seq_device snd_pcm joydev snd_timer pcspkr snd soundcore sg i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs libcrc32c sr_mod
cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi drm crct10dif_pclmul crct10dif_common scsi_transport_spi crc32c_intel mptscsih e1000 ata_piix nfit mptbase libata libnvdimm serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 1550 Comm: dockerd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1127.10.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff916633cbd230 ti: ffff916634e8c000 task.ti: ffff916634e8c000
RIP: 0010:[<ffffffffc032b3be>]  [<ffffffffc032b3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0000:ffff916634e8f9f8  EFLAGS: 00000246
RAX: ffffba6d40d00000 RBX: ffff9164c4264100 RCX: 0000000000000007
RDX: ffff9166327c1400 RSI: ffff916631c4d000 RDI: 000000000000000e
RBP: ffff916634e8fa08 R08: 0000000000000002 R09: 0000000009351fff
R10: ffff9166325a7000 R11: ffff9164b5643b80 R12: ffff916491b95080
R13: ffff916634e8f96c R14: ffffffff9db5b08b R15: ffff916634e8f9b0
FS:  00007f785de68700(0000) GS:ffff916637680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000561ab80df737 CR3: 000000017cf12000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc037871e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc038e2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff9dce2920>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff9dcec04c>] scsi_request_fn+0x4ac/0x680
[<ffffffff9db7716e>] ? deadline_add_request+0x6e/0x90
[<ffffffff9db52769>] __blk_run_queue+0x39/0x50
[<ffffffff9db56473>] blk_queue_bio+0x3b3/0x400
[<ffffffff9db543f7>] generic_make_request+0x147/0x380
[<ffffffff9db546a0>] submit_bio+0x70/0x150
Complete report #7071
Frame # Function Binary Source or offset Line
1
mpt_put_msg_frame
mptbase 0xffffffffc01aa3be
Build id: 3.10.0-1127.19.1.el7.x86_64
0x5e
2
mptscsih_qcmd
mptscsih 0xffffffffc01d171e
Build id: 3.10.0-1127.19.1.el7.x86_64
0x24e
3
mptspi_qcmd
mptspi 0xffffffffc010f2e0
Build id: 3.10.0-1127.19.1.el7.x86_64
0x50
4
scsi_dispatch_cmd
vmlinux 0xffffffff844e2990
Build id: 3.10.0-1127.19.1.el7.x86_64
0xb0
5
scsi_request_fn
vmlinux 0xffffffff844ec0bc
Build id: 3.10.0-1127.19.1.el7.x86_64
0x4ac
6
? deadline_add_request
vmlinux 0xffffffff843771ae
Build id: 3.10.0-1127.19.1.el7.x86_64
0x6e
7
__blk_run_queue
vmlinux 0xffffffff843527a9
Build id: 3.10.0-1127.19.1.el7.x86_64
0x39
8
blk_queue_bio
vmlinux 0xffffffff843564b3
Build id: 3.10.0-1127.19.1.el7.x86_64
0x3b3
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [mysqld:6630]
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [mysqld:6662]
Modules linked in: tcp_diag udp_diag inet_diag unix_diag ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 ip6table_filter nf_log_ipv4 nf_log_common xt_owner nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi libata scsi_transport_spi mptscsih crct10dif_pclmul crct10dif_common drm_panel_orientation_quirks mptbase crc32c_intel serio_raw vmxnet3 floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 6662 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff98c951509070 ti: ffff98c96e32c000 task.ti: ffff98c96e32c000
RIP: 0010:[<ffffffffc01aa3be>]  [<ffffffffc01aa3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff98c96e32faf8  EFLAGS: 00000246
RAX: ffffb29d01c00000 RBX: ffff98c910eaa3b0 RCX: 0000000000000015
RDX: ffff98c718ccb400 RSI: ffff98c96beb3000 RDI: 000000000000000e
RBP: ffff98c96e32fb08 R08: 0000000000000003 R09: 00000000a9de0fff
R10: ffff98c96b3f8400 R11: ffff98c8f5a6da40 R12: ffff98c8f5548d80
R13: ffff98c96e32fa6c R14: ffffffff8435b0cb R15: ffff98c96e32fab0
FS:  00007f5993fff700(0000) GS:ffff98c97fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa394d1b310 CR3: 00000000355e4000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffc01d171e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc010f2e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff844e2990>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff844ec0bc>] scsi_request_fn+0x4ac/0x680
[<ffffffff843771ae>] ? deadline_add_request+0x6e/0x90
[<ffffffff843527a9>] __blk_run_queue+0x39/0x50
[<ffffffff843564b3>] blk_queue_bio+0x3b3/0x400