NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [minio:3938]
Modules linked in: nf_conntrack_netlink xt_addrtype xt_CHECKSUM br_netfilter ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci i2c_piix4 parport_pc parport ip_tables xfs libcrc32c
sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm libnvdimm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel mptspi libata scsi_transport_spi mptscsih e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 7 PID: 3938 Comm: minio Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.99.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8c76f9791080 ti: ffff8c76f7cac000 task.ti: ffff8c76f7cac000
RIP: 0010:[<ffffffffa531e23e>]  [<ffffffffa531e23e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff8c76f7cafc70  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00080000a5281b10 RCX: ffff8c78757206f0
RDX: 0000000000000004 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff8c76f7cafca8 R08: ffff8c77ba88c800 R09: ffffffffa559c7b9
R10: ffff8c78757df160 R11: fffff88141e7c800 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8c76f7cafc20 R15: 0000000000000001
FS:  000000c00007a890(0000) GS:ffff8c78757c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f24602ae000 CR3: 00000002336fc000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa5281fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa528201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa53fa07e>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffa53fbbff>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa53fbd33>] tlb_finish_mmu+0x23/0x40
[<ffffffffa53ff27a>] zap_page_range+0xea/0x140
[<ffffffffa52b8862>] ? __set_current_blocked+0x42/0x70
[<ffffffffa53f966d>] SyS_madvise+0x49d/0xac0
[<ffffffffa54ae42c>] ? SyS_epoll_wait+0xac/0x110
[<ffffffffa59c539a>] system_call_fastpath+0x25/0x2a
Code: 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 40 19 f5 a5 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 1d 18 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 

NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [calico-node:69824]
Modules linked in: squashfs loop xt_multiport veth ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables ipt_rpfilter xt_set iptable_raw ip_set_hash_ip ip_set_hash_net ip_set ipip tunnel4 ip_tunnel xt_nat xt_statistic ipt_REJECT nf_reject_ipv4 iptable_mangle xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_nat_ipv4 nf_nat rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache overlay(T) vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev joydev sg pcspkr vmw_balloon parport_pc parport vmw_vmci i2c_piix4 nfsd auth_rpcgss nf_conntrack_ipv4 nfs_acl nf_defrag_ipv4 ip_vs_sh
ip_vs_wrr ip_vs_rr ip_vs lockd grace sunrpc nf_conntrack libcrc32c ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ahci ata_piix libahci libata serio_raw nfit libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse br_netfilter bridge stp llc
CPU: 4 PID: 69824 Comm: calico-node Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.31.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffffa08414306300 ti: ffffa082ac57c000 task.ti: ffffa082ac57c000
RIP: 0010:[<ffffffffa2f16bde>]  [<ffffffffa2f16bde>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffffa082ac57fc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00080000a2e7ddd0 RCX: ffffa0841d620678
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffffa082ac57fca8 R08: ffffa07d6c57d800 R09: ffffffffa31866a9
R10: ffffa0841d71f160 R11: ffffeac1e03f9800 R12: 0000000000000282
R13: 0000000000000282 R14: ffffa082ac57fc20 R15: 0000000000000001
FS:  00007f3376ffd700(0000) GS:ffffa0841d700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c001789000 CR3: 0000000815192000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa2e7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffa2e7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffa2fee176>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffa2fefc3f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffa2fefd63>] tlb_finish_mmu+0x23/0x30
[<ffffffffa2ff322a>] zap_page_range+0xea/0x130
[<ffffffffa2eb3862>] ? __set_current_blocked+0x42/0x70
[<ffffffffa2fed71d>] SyS_madvise+0x49d/0xac0
[<ffffffffa3595f92>] system_call_fastpath+0x25/0x2a
Code: 8a c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 b5 a3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 8a c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
[drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [PLANE:33:plane-0] flip_done timed out

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [qtp468567811-13:25816]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(E) falcon_kal(E) falcon_lsm_pinned_17905(E) xt_nat veth falcon_lsm_pinned_17804(E) xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) ext4 mbcache jbd2 snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core iosf_mbi crc32_pclmul snd_hwdep snd_seq ghash_clmulni_intel snd_seq_device snd_pcm aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev snd_timer virtio_balloon pcspkr virtio_rng snd i2c_piix4 soundcore ip_tables xfs libcrc32c ata_generic pata_acpi virtio_console virtio_blk qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
ttm e1000 floppy drm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel serio_raw libata virtio_pci virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_kal]
CPU: 1 PID: 25816 Comm: qtp468567811-13 Tainted: P            EL ------------ T 3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+90052+d3bf71d8 04/01/2014
task: ffff93c125c48000 ti: ffff93c208628000 task.ti: ffff93c208628000
RIP: 0010:[<ffffffff9891e25a>]  [<ffffffff9891e25a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff93c20862bc70  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000c000098881b10 RCX: ffff93c33fc20320
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff93c20862bca8 R08: ffff93c27aec1800 R09: ffffffff98b9c7f9
R10: ffff93c33fc9f160 R11: ffffeabc04494600 R12: 0000000000000282
R13: 0000000000000282 R14: ffff93c20862bc20 R15: 0000000000000001
FS:  00007fa48181c640(0000) GS:ffff93c33fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa5808c62b0 CR3: 00000000a7d22000 CR4: 00000000001606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff98881fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9888201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff989fa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff989fbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff989fbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffff989ff2ca>] zap_page_range+0xea/0x140
[<ffffffff988e1118>] ? wake_up_q+0x58/0x90
[<ffffffff989f96bd>] SyS_madvise+0x49d/0xac0
[<ffffffff9895f95b>] ? __seccomp_filter+0x5b/0x310
[<ffffffff988b8875>] ? __set_current_blocked+0x55/0x70
[<ffffffff98fc562e>] tracesys+0xa6/0xcc
Code: 48 63 35 ae 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 19 55 99 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 7d 18 c4 00 eb b7 0f b6 4d cc 4c 89 

NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [armeria-common-:18299]
Modules linked in: veth dummy vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_defrag_ipv6 ip_set_hash_ip xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink br_netfilter overlay(T) iptable_nat nf_nat_ipv4 nf_nat xt_comment xt_conntrack xt_addrtype bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 iptable_mangle nf_conntrack ip_set iptable_filter vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul joydev glue_helper ablk_helper cryptd vmw_balloon ppdev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif syscopyarea crct10dif_generic sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix
crct10dif_pclmul crct10dif_common crc32c_intel libata serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: ip6_tables]
CPU: 6 PID: 18299 Comm: armeria-common- Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffff937f910e0000 ti: ffff937f91108000 task.ti: ffff937f91108000
RIP: 0010:[<ffffffffb3b170de>]  [<ffffffffb3b170de>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff937f9110bc70  EFLAGS: 00000202
RAX: 0000000000000005 RBX: ffffffffb3a7ddd0 RCX: ffff937fbfd606c8
RDX: 0000000000000005 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff937f9110bca8 R08: ffff9378ec412400 R09: ffffffffb3d86c09
R10: ffff937fbfd9f160 R11: ffffde30577e6800 R12: ffff937f9110bc40
R13: ffffffffb3d86b5f R14: ffff937f9ddbc000 R15: 0000000000000001
FS:  00007f443e3fb640(0000) GS:ffff937fbfd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f249e282380 CR3: 00000000a0e28000 CR4: 00000000001607e0
Call Trace:
[<ffffffffb3a7e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb3a7e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb3bee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffffb3bf013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb3bf0263>] tlb_finish_mmu+0x23/0x30
[<ffffffffb3bf372a>] zap_page_range+0xea/0x130
[<ffffffffb3bfd36c>] ? do_mmap+0x39c/0x590
[<ffffffffb3bedc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffb3bdedab>] ? vm_mmap_pgoff+0xeb/0x120
[<ffffffffb3ab3892>] ? __set_current_blocked+0x42/0x70
[<ffffffffb4199f92>] system_call_fastpath+0x25/0x2a
Code: 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 75 b4 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee