NMI watchdog: BUG: soft lockup - CPU#6 stuck for 40s! [oracle:121011]
[<ffffffff93cdc890>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff93ce5efc>] scsi_request_fn+0x4cc/0x680
[<ffffffff93b4cf59>] __blk_run_queue+0x39/0x50
[<ffffffff93b4d01a>] queue_unplugged+0x2a/0xa0
[<ffffffff93b50858>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff93b50cc4>] blk_finish_plug+0x14/0x40
[<ffffffff93a8cd97>] do_blockdev_direct_IO+0x1a87/0x20a0
[<ffffffffc046cc40>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffff93a8d405>] __blockdev_direct_IO+0x55/0x60
[<ffffffffc046cc40>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffffc046cc60>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[<ffffffffc0478080>] xfs_file_dio_aio_write+0x180/0x390 [xfs]
[<ffffffffc046cc40>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffffc046cc60>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[<ffffffffc0478652>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[<ffffffff93a47ba3>] do_sync_write+0x93/0xe0
[<ffffffff93a48690>] vfs_write+0xc0/0x1f0
[<ffffffff93a49672>] SyS_pwrite64+0x92/0xc0
[<ffffffff93f8bede>] system_call_fastpath+0x25/0x2a

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [mysqld:30035]
Modules linked in: iptable_filter xt_multiport sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper sg ppdev ablk_helper cryptd vmw_balloon pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common drm ata_piix crc32c_intel mptspi scsi_transport_spi serio_raw e1000 libata mptscsih vmxnet3 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 30035 Comm: mysqld Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.6.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff8d42b5bb9080 ti: ffff8d41f5f98000 task.ti: ffff8d41f5f98000
RIP: 0010:[<ffffffffc032c3be>]  [<ffffffffc032c3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0018:ffff8d41f5f9b820  EFLAGS: 00000246
RAX: ffff9a2f40f40000 RBX: ffff8d4252550220 RCX: 0000000000000000
RDX: ffff8d42b5493600 RSI: ffff8d42b58ec000 RDI: 000000000000000e
RBP: ffff8d41f5f9b830 R08: 0000000000000003 R09: 000000005bd51fff
R10: ffff8d40b5d5ac00 R11: ffff8d42457f8540 R12: 0000000100000002
R13: ffff8d429115c900 R14: ffff8d41f5f9b794 R15: ffffffffb835be3b
FS:  00007fe6290b3700(0000) GS:ffff8d42bfc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000401008 CR3: 000000005b0f0000 CR4: 00000000000607e0
Call Trace:
[<ffffffffc036971e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc02f12e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffffb84e39d0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffb84ed0fc>] scsi_request_fn+0x4ac/0x680
[<ffffffffb8353519>] __blk_run_queue+0x39/0x50
[<ffffffffb83535da>] queue_unplugged+0x2a/0xa0
[<ffffffffb8356e18>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffb8357284>] blk_finish_plug+0x14/0x40
[<ffffffffb8292e47>] do_blockdev_direct_IO+0x1a97/0x20b0
[<ffffffffc056aea0>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffffb82934b5>] __blockdev_direct_IO+0x55/0x60
[<ffffffffc056aea0>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffffc056aec0>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[<ffffffffc0576480>] xfs_file_dio_aio_write+0x180/0x390 [xfs]
[<ffffffffc056aea0>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffffc056aec0>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[<ffffffffc0576a52>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[<ffffffffb824d0a3>] do_sync_write+0x93/0xe0
[<ffffffffb824db90>] vfs_write+0xc0/0x1f0
[<ffffffffb824eb32>] SyS_pwrite64+0x92/0xc0
[<ffffffffb8794ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffffb8794f92>] system_call_fastpath+0x25/0x2a
[<ffffffffb8794ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 a3 33 

NMI watchdog: BUG: soft lockup - CPU#0 stuck for 21s! [mysqld:3449]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev pcspkr joydev vmw_balloon sg shpchp i2c_piix4 parport_pc vmw_vmci parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod crc_t10dif crct10dif_generic
ahci ata_piix libahci drm libata crct10dif_common i2c_core crc32c_intel serio_raw vmxnet3 vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 3449 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-862.6.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8c9e8a7d6eb0 ti: ffff8c9caca84000 task.ti: ffff8c9caca84000
RIP: 0010:[<ffffffffa0d164f5>]  [<ffffffffa0d164f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8c9caca87890  EFLAGS: 00000286
RAX: 0000000000000000 RBX: ffffffffa09754d3 RCX: ffff8c9ebfe31060
RDX: ffff98e1c1ba0000 RSI: 0000000000000286 RDI: 0000000000000286
RBP: ffff8c9caca87890 R08: 0000000000000000 R09: ffff8c9eacea3800
R10: ffff8c9eb07360d8 R11: ffff8c9ebf61e000 R12: 0000000000000000
R13: 0000000000000020 R14: ffff8c9e46b4d080 R15: ffffffffa0aa2da9
FS:  00007f7c7cc11700(0000) GS:ffff8c9ebfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6143c5d000 CR3: 00000002ace68000 CR4: 00000000000007f0
Call Trace:
[<ffffffffc02a7622>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffffa0aa4050>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffffa0a9bb60>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffffa0aa50cc>] scsi_request_fn+0x4cc/0x680
[<ffffffffa0919519>] __blk_run_queue+0x39/0x50
[<ffffffffa09195da>] queue_unplugged+0x2a/0xa0
[<ffffffffa091cd98>] blk_flush_plug_list+0x1d8/0x230
[<ffffffffa091d204>] blk_finish_plug+0x14/0x40
[<ffffffffa085e5d7>] do_blockdev_direct_IO+0x1a87/0x20a0
[<ffffffffc04f8030>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffffa085ec45>] __blockdev_direct_IO+0x55/0x60
[<ffffffffc04f8030>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffffc04f8050>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[<ffffffffc0502f69>] xfs_file_dio_aio_write+0x179/0x340 [xfs]
[<ffffffffc04f8030>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffffc04f8050>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[<ffffffffa091c87d>] ? blk_peek_request+0x9d/0x2a0
[<ffffffffc05034f2>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[<ffffffffa081a9b3>] do_sync_write+0x93/0xe0
[<ffffffffa081b490>] vfs_write+0xc0/0x1f0
[<ffffffffa081c482>] SyS_pwrite64+0x92/0xc0
[<ffffffffa0d206d5>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffa0d20795>] system_call_fastpath+0x1c/0x21
[<ffffffffa0d206e1>] ? system_call_after_swapgs+0xae/0x146
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48 

NMI watchdog: BUG: soft lockup - CPU#2 stuck for 35s! [kworker/u8:3:13804]
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 38s! [oracle:15368]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) vmw_vsock_vmci_transport falcon_lsm_pinned_11906(E) vsock sunrpc crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd pcspkr joydev sg parport_pc i2c_piix4 parport vmw_vmci binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix ahci nfit libahci libata libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 15368 Comm: oracle Tainted: P            EL ------------   3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9f393a151080 ti: ffff9f338f808000 task.ti: ffff9f338f808000
RIP: 0010:[<ffffffff88d8b565>]  [<ffffffff88d8b565>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff9f338f80b888  EFLAGS: 00000282
RAX: 0000000000000000 RBX: ffff9f338f80b848 RCX: ffffffff89234f40
RDX: ffffbb05c3610000 RSI: 0000000000000282 RDI: 0000000000000282
RBP: ffff9f338f80b888 R08: 0000000000000001 R09: 0000000000000000
R10: ffff9f34f03c6800 R11: ffff9f34f2798800 R12: 0000000000000020
R13: ffff9f3738c2dc80 R14: ffffffff88aeb369 R15: ffff9f338f80b800
FS:  00007f76d1ac2740(0000) GS:ffff9f39bfc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000002f0b9a4 CR3: 000000013f080000 CR4: 00000000000007f0
Call Trace:
[<ffffffffc0279632>] pvscsi_queue+0x4a2/0x920 [vmw_pvscsi]
[<ffffffff88aec6b0>] ? scsi_test_unit_ready+0x170/0x170
[<ffffffff88ae3fa0>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff88aed6cc>] scsi_request_fn+0x4ac/0x680
[<ffffffff88953939>] __blk_run_queue+0x39/0x50
[<ffffffff889539fa>] queue_unplugged+0x2a/0xa0
[<ffffffff88957338>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff889577a4>] blk_finish_plug+0x14/0x40
[<ffffffff88893247>] do_blockdev_direct_IO+0x1a97/0x20b0
[<ffffffffc053eea0>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffff888938b5>] __blockdev_direct_IO+0x55/0x60
[<ffffffffc053eea0>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffffc053eec0>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[<ffffffffc054a480>] xfs_file_dio_aio_write+0x180/0x390 [xfs]
[<ffffffffc053eea0>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffffc053eec0>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[<ffffffff887ee08a>] ? __do_fault.isra.61+0x8a/0x100
[<ffffffffc054aa52>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[<ffffffff8884d313>] do_sync_write+0x93/0xe0
[<ffffffff8884de00>] vfs_write+0xc0/0x1f0
[<ffffffff8884eda2>] SyS_pwrite64+0x92/0xc0
[<ffffffff88d95ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff88d95f92>] system_call_fastpath+0x25/0x2a
[<ffffffff88d95ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 07 00 66 66 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 c6 07 00 66 66 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48 

NMI watchdog: BUG: soft lockup - CPU#5 stuck for 23s! [METALOG:74778]
Modules linked in: rpcsec_gss_krb5 xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat_ipv4 nf_nat overlay(T) vmw_vsock_vmci_transport vsock iptable_filter coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmw_vmci cryptd ppdev pcspkr sg vmw_balloon joydev parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul drm crct10dif_common ata_piix crc32c_intel mptspi scsi_transport_spi mptscsih libata serio_raw mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod nf_conntrack_ipv4
nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack libcrc32c br_netfilter bridge stp llc
CPU: 5 PID: 74778 Comm: METALOG Tainted: G             L ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff9f7303e641c0 ti: ffff9f73360d8000 task.ti: ffff9f73360d8000
RIP: 0010:[<ffffffffc03ac3be>]  [<ffffffffc03ac3be>] mpt_put_msg_frame+0x5e/0x80 [mptbase]
RSP: 0000:ffff9f73360db820  EFLAGS: 00000246
RAX: ffffb73a83200000 RBX: ffff9f6dd8d86960 RCX: 0000000000000004
RDX: ffff9f759870f400 RSI: ffff9f7598f9b000 RDI: 000000000000000e
RBP: ffff9f73360db830 R08: 0000000000000003 R09: 0000000814ac8fff
R10: ffff9f759888a000 R11: ffff9f759c33e840 R12: 0000000100000004
R13: ffff9f7331572400 R14: ffff9f73360db794 R15: ffffffff8295b0cb
FS:  00007fb41495b700(0000) GS:ffff9f759d740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4f469bb000 CR3: 00000007ddc9c000 CR4: 00000000000007e0
Call Trace:
[<ffffffffc042c71e>] mptscsih_qcmd+0x24e/0x830 [mptscsih]
[<ffffffffc04152e0>] mptspi_qcmd+0x50/0xf0 [mptspi]
[<ffffffff82ae2990>] scsi_dispatch_cmd+0xb0/0x240
[<ffffffff82aec0bc>] scsi_request_fn+0x4ac/0x680
[<ffffffff829527a9>] __blk_run_queue+0x39/0x50
[<ffffffff8295286a>] queue_unplugged+0x2a/0xa0
[<ffffffff829560a8>] blk_flush_plug_list+0x1d8/0x230
[<ffffffff82956514>] blk_finish_plug+0x14/0x40
[<ffffffff82892317>] do_blockdev_direct_IO+0x1a87/0x20a0
[<ffffffffc05fef20>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffff82892985>] __blockdev_direct_IO+0x55/0x60
[<ffffffffc05fef20>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffffc05fef40>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[<ffffffffc060a4b0>] xfs_file_dio_aio_write+0x180/0x390 [xfs]
[<ffffffffc05fef20>] ? xfs_setfilesize+0x80/0x80 [xfs]
[<ffffffffc05fef40>] ? xfs_get_blocks_direct+0x20/0x20 [xfs]
[<ffffffff8284d141>] ? vfs_write+0x1/0x1f0
[<ffffffffc060aa82>] xfs_file_aio_write+0x102/0x1b0 [xfs]
[<ffffffff8284c713>] do_sync_write+0x93/0xe0
[<ffffffff8284d200>] vfs_write+0xc0/0x1f0
[<ffffffff8284e192>] SyS_pwrite64+0x92/0xc0
[<ffffffff82d92e09>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff82d92ed2>] system_call_fastpath+0x25/0x2a
[<ffffffff82d92e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f6 86 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 <5b> 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 a3 3b