NMI watchdog: BUG: soft lockup - CPU#0 stuck for 95s! [kworker/u256:0:100670]
{ 0 1 2 3 6} 
Modules linked in:
tcp_lp nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event crc32_pclmul ppdev ghash_clmulni_intel snd_ens1371 vmw_balloon snd_rawmidi snd_ac97_codec aesni_intel ac97_bus lrw gf128mul snd_seq glue_helper ablk_helper cryptd snd_seq_device snd_pcm joydev pcspkr snd_timer sg snd soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
ata_piix ttm libata e1000 drm nfit mptspi scsi_transport_spi libnvdimm mptscsih crct10dif_pclmul crct10dif_common crc32c_intel mptbase serio_raw drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 100670 Comm: kworker/u256:0 Kdump: loaded Tainted: G             L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
task: ffff9aec2d2bd280 ti: ffff9aed94e54000 task.ti: ffff9aed94e54000
RIP: 0010:[<ffffffffba116562>]  [<ffffffffba116562>] generic_exec_single+0x102/0x1c0
(detected by 5, t=103227 jiffies, g=10409713, c=10409712, q=2840)
(t=103227 jiffies g=10409713 c=10409712 q=2840)
Task dump for CPU 0:
RSP: 0018:ffff9aed94e577a0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9aed94e57770 RCX: 0000000000000004
RDX: ffffffffba813f10 RSI: 0000000000000080 RDI: 0000000000000286
RBP: ffff9aed94e577f0 R08: ffffffffba813f08 R09: 00000000a2b11000
R10: 0000000000006000 R11: 0000000000000005 R12: ffff9aeeaf685480
kworker/u256:0  R  running task        0 100670      2 0x00000088
R13: 0000000000011220 R14: ffff9aed94e57770 R15: ffff9aeeaf6854b0
FS:  0000000000000000(0000) GS:ffff9aeeb1600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2bf1a84170 CR3: 000000032bc10000 CR4: 0000000000760ff0
Workqueue: writeback bdi_writeback_workfn (flush-253:0)
Call Trace:
[<ffffffffba0bdc4f>] ? process_one_work+0x17f/0x440

NMI watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [java:91199]
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [java:53178]
Modules linked in: vmw_vsock_vmci_transport vsock vfat fat sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg tpm_crb parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw e1000e ahci mptsas drm scsi_transport_sas ata_piix libahci mptscsih libata mptbase ptp pps_core nfit libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 53178 Comm: java Kdump: loaded Tainted: G        W    L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware7,1/440BX Desktop Reference Platform, BIOS VMW71.00V.17369862.B64.2012240522 12/24/2020
task: ffff980d18ae0000 ti: ffff980d7c710000 task.ti: ffff980d7c710000
Modules linked in: vmw_vsock_vmci_transport vsock vfat fat sb_edac iosf_mbi
RIP: 0010:[<ffffffffabd1109a>] 
[<ffffffffabd1109a>] generic_exec_single+0xfa/0x1b0

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [dracut:1759]
Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter snd_hda_codec_generic iosf_mbi snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm ppdev snd_timer crc32_pclmul ghash_clmulni_intel aesni_intel snd lrw gf128mul glue_helper ablk_helper soundcore i2c_piix4 pcspkr joydev cryptd parport_pc parport virtio_rng sg virtio_balloon ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi
virtio_blk virtio_net virtio_console qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul crct10dif_common crc32c_intel drm_panel_orientation_quirks floppy serio_raw virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 1759 Comm: dracut Tainted: G             L ------------   3.10.0-1062.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff96b43456d230 ti: ffff96b428ec4000 task.ti: ffff96b428ec4000
RIP: 0010:[<ffffffffacd14a22>]  [<ffffffffacd14a22>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff96b428ec7c50  EFLAGS: 00000202
RAX: 0000000000000004 RBX: ffff96b428ec7c20 RCX: 000000000000003c
RDX: 000000000000000f RSI: 0000000000000004 RDI: 0000000000000282
RBP: ffff96b428ec7ca0 R08: ffffffffad413500 R09: ffffffffacc9776c
R10: ffff96b428aaa510 R11: 0000000000000000 R12: ffff96b428ec7bd8
R13: fffffdaa485bdc00 R14: 00007fffeb264000 R15: 0000000000000000
FS:  00007f8f89fbe740(0000) GS:ffff96b43fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000001071f68 CR3: 000000022ab7c000 CR4: 00000000000406e0
Call Trace:

NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [systemd-journal:828]
Modules linked in: tcp_diag udp_diag inet_diag 8021q garp mrp stp llc nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter bonding nls_utf8 isofs xfs libcrc32c intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt sg i2c_i801 iTCO_vendor_support sb_edac edac_core pcspkr i2c_core mei_me mei acpi_power_meter shpchp lpc_ich ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ixgbe ahci mpt3sas libahci mdio ptp raid_class libata pps_core scsi_transport_sas dca scsi_ep_front(OE) fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 10 PID: 828 Comm: systemd-journal Tainted: G        W  OEL ------------   3.10.0-514.el7.x86_64 #1
Hardware name: Huawei RH2288H V3/BC11HGSA0, BIOS 5.03 07/25/2018
task: ffff88203ce3de20 ti: ffff88203d0ac000 task.ti: ffff88203d0ac000
RIP: 0010:[<ffffffff810f925e>]  [<ffffffff810f925e>] generic_exec_single+0xfe/0x1a0
RSP: 0000:ffff88203d0afbd0  EFLAGS: 00000202
RAX: ffff880b50f5de00 RBX: ffffffff810c1945 RCX: ffff880b50f5de10
RDX: ffff88203f4d7780 RSI: ffff88203d0afbd0 RDI: ffff88203d0afbd0
RBP: ffff88203d0afc18 R08: 0000000000000001 R09: ffffea0050cb23c0
R10: 0000000000003746 R11: 8000000001c9d225 R12: ffff88103b7b8fb0
R13: 0000000000000002 R14: 00000000810c0c18 R15: ffff88207ffd8008
FS:  00007f3fa273e880(0000) GS:ffff88203f000000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3fa3c84f00 CR3: 000000103aab3000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff880b50f5de10 0000000000000000 ffffffff8106c300 ffff88203d0afca0
0000000000000003 0000000027b7faef 0000000000000027 ffffffff8106c300
0000000000000003 0000000027b7faef 0000000000000027 ffffffff8106c300
ffffffff8106c300 ffff88203d0afc48 ffffffff810f935f ffff88203d0afc48
ffffffff8106c300 ffff88203d0afc48 ffffffff810f935f ffff88203d0afc48
Call Trace:

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [vmtoolsd:6213]
Modules linked in: xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables tcp_lp rfcomm fuse tun devlink ip_set nfnetlink bridge stp llc vmw_vsock_vmci_transport vsock bnep sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr btusb btrtl
btbcm btintel bluetooth snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq rfkill snd_seq_device snd_pcm snd_timer snd soundcore sg parport_pc parport vmw_vmci i2c_piix4 xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_pclmul crct10dif_common crc32c_intel serio_raw nfit e1000 libnvdimm ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi scsi_transport_spi mptscsih ata_piix mptbase libata drm_panel_orientation_quirks [last unloaded: ip_tables]
CPU: 1 PID: 6213 Comm: vmtoolsd Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff94b236b1a080 ti: ffff94b236b34000 task.ti: ffff94b236b34000
RIP: 0010:[<ffffffff8f91109a>]  [<ffffffff8f91109a>] generic_exec_single+0xfa/0x1b0
[<ffffffff8f82e675>] do_softirq+0x65/0xa0
[<ffffffff8f8a1285>] irq_exit+0x105/0x110

NMI watchdog: BUG: soft lockup - CPU#18 stuck for 35s! [UndertowEventHa:12565]
dm_region_hash dm_log
Modules linked in:
dm_mod fuse
dm_mod fuse
[last unloaded: tmhook]
CPU: 17 PID: 9899 Comm: UndertowEventHa Kdump: loaded Tainted: P           OEL ------------   3.10.0-1160.15.2.el7.x86_64 #1
gsch(OE)
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8cdc61fb4200 ti: ffff8ce48cb18000 task.ti: ffff8ce48cb18000
[last unloaded: tmhook]
RIP: 0010:[<ffffffffa0b167a2>]  [<ffffffffa0b167a2>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff8ce48cb1bc20  EFLAGS: 00000202
CPU: 23 PID: 11995 Comm: UndertowEventHa Kdump: loaded Tainted: P           OEL ------------   3.10.0-1160.15.2.el7.x86_64 #1
RAX: 0000000000000080 RBX: ffff8ce48cb1bbf0 RCX: 0000000000000015
RDX: ffffffffa12169d0 RSI: 0000000000000080 RDI: 0000000000000282
RBP: ffff8ce48cb1bc70 R08: ffffffffa12169c8 R09: ffffffffa0c02798
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
R10: ffff8ce70da5f0a0 R11: ffffd5b9c50ffa40 R12: ffff8ce70d3bac00
task: ffff8ce22f150000 ti: ffff8ce49f304000 task.ti: ffff8ce49f304000
R13: 00000000b7642292 R14: ffffffffff0a0004 R15: 0000000000000100
FS:  00007fa028c11700(0000) GS:ffff8ce70da40000(0000) knlGS:0000000000000000
redirfs(OE)
RIP: 0010:[<ffffffffa0b16e4a>]  [<ffffffffa0b16e4a>] smp_call_function_many+0x20a/0x270
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
RSP: 0018:ffff8ce49f307cb0  EFLAGS: 00000202
RAX: 0000000000000009 RBX: 00080000a0a7e6a0 RCX: ffff8ce70d860970
RDX: 0000000000000009 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8ce49f307ce8 R08: ffff8cdc24ff8800 R09: ffffffffa0d86479
CR2: 00007fa0340a7000 CR3: 000000099f656000 CR4: 00000000003607e0
R10: ffff8ce70dbdf160 R11: ffffd5b9db73f400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8ce49f307c60 R15: 00007f81e64f1000
FS:  00007f81db7df700(0000) GS:ffff8ce70dbc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f81e64f1000 CR3: 000000099ded0000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:

BUG: soft lockup - CPU#4 stuck for 23s! [kworker/u256:0:105377]
Modules linked in: binfmt_misc tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables dahdi(OF) crc_ccitt fuse netatop(OF) xt_CHECKSUM iptable_mangle xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype iptable_nat nf_nat_ipv4 nf_nat nf_log_ipv4 nf_log_common xt_LOG ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_recent xt_multiport xt_mark xt_conntrack nf_conntrack iptable_filter ext4 mbcache jbd2 coretemp snd_ens1371 snd_rawmidi snd_ac97_codec crc32_pclmul ac97_bus ghash_clmulni_intel snd_seq snd_seq_device snd_pcm aesni_intel ppdev lrw gf128mul glue_helper ablk_helper vmw_balloon snd_timer cryptd snd ahci pcspkr soundcore libahci sg vmw_vmci i2c_piix4 shpchp parport_pc parport nfsd auth_rpcgss nfs_acl lockd sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod
crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper ttm crct10dif_pclmul crct10dif_common crc32c_intel mptspi drm serio_raw scsi_transport_spi ata_piix e1000 mptscsih libata mptbase i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 105377 Comm: kworker/u256:0 Tainted: GF          O--------------   3.10.0-229.14.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015
BUG: soft lockup - CPU#6 stuck for 22s! [ftdc:4344]
Workqueue: writeback bdi_writeback_workfn
INFO: rcu_sched detected stalls on CPUs/tasks:
(flush-253:1)
Modules linked in:
{
binfmt_misc
task: ffff880002615b00 ti: ffff8800a7c68000 task.ti: ffff8800a7c68000
} 
tun bridge stp
RIP: 0010:[<ffffffff810d66ea>] 
[<ffffffff810d66ea>] generic_exec_single+0xfa/0x1a0

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [agent_app:8909]
Modules linked in: kbase(OE) udp_diag tcp_diag inet_diag binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter sunrpc ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel virtio_balloon joydev lrw gf128mul parport_pc glue_helper ablk_helper parport cryptd i2c_piix4 pcspkr ip_tables ext4 mbcache jbd2 ata_generic pata_acpi virtio_net virtio_blk virtio_console cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm floppy ata_piix drm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw virtio_pci virtio_ring virtio
drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: kbase]
CPU: 1 PID: 8909 Comm: agent_app Kdump: loaded Tainted: G           OEL ------------   3.10.0-957.el7.x86_64 #1
Hardware name: Fedora Project OpenStack Nova, BIOS 0.5.1 01/01/2011
task: ffff933dbaf89040 ti: ffff933d8e274000 task.ti: ffff933d8e274000
RIP: 0010:[<ffffffffaf11109e>]  [<ffffffffaf11109e>] generic_exec_single+0xfe/0x1b0
RSP: 0018:ffff933d8e277c30  EFLAGS: 00000202
RAX: 0000000000000010 RBX: ffff933d8e277c00 RCX: 0000000000000030
RDX: 000000000000ffff RSI: 0000000000000010 RDI: 0000000000000286
RBP: ffff933d8e277c78 R08: ffffffffaf814180 R09: ffffffffaf1f838c
R10: ffff933dbf25f0a0 R11: ffffdc363fe4aa40 R12: 0000000000000040
R13: 0000000000000000 R14: ffffffffc05a0dd6 R15: ffff933d8e277bf0
FS:  00007f990beec740(0000) GS:ffff933dbf240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f990bf04000 CR3: 0000000fd2f1a000 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace: