NMI watchdog: BUG: soft lockup - CPU#5 stuck for 24s! [kworker/5:0H:31723]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter nfsv3 nfs_acl nfs lockd grace fscache vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg joydev vmw_balloon pcspkr vmw_vmci parport_pc i2c_piix4 parport auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif syscopyarea sysfillrect crct10dif_generic sysimgblt fb_sys_fops ttm ahci libahci
drm crct10dif_pclmul ata_piix crct10dif_common nfit crc32c_intel serio_raw libata libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 5 PID: 31723 Comm: kworker/5:0H Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff9f7cef128000 ti: ffff9f7a58aac000 task.ti: ffff9f7a58aac000
RIP: 0010:[<ffffffffb5b97ae9>]  [<ffffffffb5b97ae9>] memcpy+0x29/0x110
RSP: 0018:ffff9f7a58aafb80  EFLAGS: 00010202
RAX: ffff9f7bc574b000 RBX: ffff9f7af1c45600 RCX: ffff9f7cef128000
RDX: 0000000000000220 RSI: ffff9f7b4518166a RDI: ffff9f7bc574b000
RBP: ffff9f7a58aafbe0 R08: 1343403bb2e5f01f R09: 7912653053a35ab3
R10: 375fc7c7b141e0a3 R11: d53a745b70abe831 R12: ffff9f7af1c45600
R13: ffff9f7a58aafb80 R14: ffffffffb5e65e0d R15: ffff9f7a58aafb18
FS:  0000000000000000(0000) GS:ffff9f7cffd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc06ca50009 CR3: 00000002b3cca000 CR4: 00000000007607e0
PKRU: 00000000
Call Trace:
[<ffffffffb5e406f3>] ? skb_copy_bits+0x133/0x2c0
[<ffffffffb5e40796>] skb_copy_bits+0x1d6/0x2c0
[<ffffffffc0422cfd>] xdr_skb_read_bits+0x2d/0x50 [sunrpc]
[<ffffffffc0422dd1>] xdr_partial_copy_from_skb+0xb1/0x240 [sunrpc]
[<ffffffffc0422cd0>] ? xprt_create_transport+0x260/0x260 [sunrpc]
[<ffffffffc042718e>] xs_tcp_data_recv+0x54e/0xad0 [sunrpc]
[<ffffffffc0426c40>] ? xs_tcp_setup_socket+0x490/0x490 [sunrpc]
[<ffffffffb5eaff0b>] tcp_read_sock+0xab/0x1f0
[<ffffffffc0423c03>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffffb58bde8f>] process_one_work+0x17f/0x440
[<ffffffffb58befa6>] worker_thread+0x126/0x3c0
[<ffffffffb58bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb58c5e61>] kthread+0xd1/0xe0
[<ffffffffb58c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffffb5f95ddd>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb58c5d90>] ? insert_kthread_work+0x40/0x40
Code: 90 90 48 89 f8 48 83 fa 20 72 7e 40 38 fe 7c 35 48 83 ea 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 8b 5e 18 48 8d 76 20 <4c> 89 07 4c 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 

NMI watchdog: BUG: soft lockup - CPU#253 stuck for 22s! [kworker/253:0H:255287]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_14203(E) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache uinput devlink binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter falcon_lsm_pinned_14108(E) snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm qxl snd_timer ttm drm_kms_helper iTCO_wdt syscopyarea
crc32_pclmul ghash_clmulni_intel aesni_intel lrw sysfillrect gf128mul sysimgblt glue_helper ppdev fb_sys_fops ablk_helper snd iTCO_vendor_support joydev cryptd parport_pc drm virtio_rng pcspkr i2c_i801 lpc_ich soundcore virtio_balloon parport drm_panel_orientation_quirks nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c virtio_net net_failover virtio_console virtio_blk failover crct10dif_pclmul ahci virtio_pci libahci crct10dif_common serio_raw crc32c_intel libata virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: falcon_kal]
CPU: 253 PID: 255287 Comm: kworker/253:0H Kdump: loaded Tainted: P        W   EL ------------   3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.13.0-2.module+el8.4.0+534+4680a14e 04/01/2014
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff8b1fc3df0000 ti: ffff8b1fbb27c000 task.ti: ffff8b1fbb27c000
RIP: 0010:[<ffffffff82397346>]  [<ffffffff82397346>] memcpy+0x6/0x110
RSP: 0018:ffff8b1fbb27fbe8  EFLAGS: 00000286
RAX: ffff8c8e21479000 RBX: ffff8b20dbda8d30 RCX: 00000000000005a0
RDX: 00000000000005a0 RSI: ffff8b16063d614e RDI: ffff8c8e21479000
RBP: ffff8b1fbb27fc48 R08: 00000000000005a0 R09: 00000000000005a0
R10: ffff8b204c5acd00 R11: 00000000000016a0 R12: ffff8b1fbb27fb88
R13: ffffffff820bb244 R14: ffff8b1fbb27fb50 R15: ffffffff820db085
FS:  0000000000000000(0000) GS:ffff8cfbaf540000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f207c7a5000 CR3: 000000044636a000 CR4: 0000000000760ee0
PKRU: 00000000
Call Trace:
[<ffffffff8263f6f3>] ? skb_copy_bits+0x133/0x2c0
[<ffffffffc0275cfd>] xdr_skb_read_bits+0x2d/0x50 [sunrpc]
[<ffffffffc0275dd1>] xdr_partial_copy_from_skb+0xb1/0x240 [sunrpc]
[<ffffffffc0275cd0>] ? xprt_create_transport+0x260/0x260 [sunrpc]
[<ffffffffc027a261>] xs_tcp_data_recv+0x621/0xad0 [sunrpc]
[<ffffffff82226822>] ? kmem_cache_free+0x1e2/0x200
[<ffffffffc0279c40>] ? xs_tcp_setup_socket+0x490/0x490 [sunrpc]
[<ffffffff826ae31b>] tcp_read_sock+0xab/0x1f0
[<ffffffffc0276c03>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffff820bde3f>] process_one_work+0x17f/0x440
[<ffffffff820bef56>] worker_thread+0x126/0x3c0
[<ffffffff820bee30>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff820c5e11>] kthread+0xd1/0xe0
[<ffffffff820c5d40>] ? insert_kthread_work+0x40/0x40
[<ffffffff82794de4>] ret_from_fork_nospec_begin+0xe/0x21
[<ffffffff820c5d40>] ? insert_kthread_work+0x40/0x40
Code: ab 82 31 c0 e8 fc 62 d0 ff 0f ae e8 0f 31 48 c1 e2 20 89 c0 48 09 c2 48 31 d3 e9 7b ff ff ff 90 90 90 90 90 90 48 89 f8 48 89 d1 <f3> a4 c3 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 

NMI watchdog: BUG: soft lockup - CPU#218 stuck for 21s! [kworker/218:1H:44983]
Modules linked in: veth ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter overlay(T) tcp_lp rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache uinput devlink binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13501(E) snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device
snd_pcm snd_timer qxl ttm snd drm_kms_helper ppdev syscopyarea sysfillrect iTCO_wdt iTCO_vendor_support crc32_pclmul parport_pc ghash_clmulni_intel aesni_intel sysimgblt lrw gf128mul pcspkr fb_sys_fops glue_helper ablk_helper soundcore lpc_ich cryptd joydev parport i2c_i801 virtio_balloon virtio_rng drm drm_panel_orientation_quirks nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c virtio_net virtio_console virtio_blk net_failover failover ahci libahci crct10dif_pclmul crct10dif_common virtio_pci crc32c_intel libata virtio_ring serio_raw virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 218 PID: 44983 Comm: kworker/218:1H Kdump: loaded Tainted: P            EL ------------ T 3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.13.0-2.module+el8.4.0+534+4680a14e 04/01/2014
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff8f8550ff3180 ti: ffff8f86debb8000 task.ti: ffff8f86debb8000
RIP: 0010:[<ffffffffb8d97346>]  [<ffffffffb8d97346>] memcpy+0x6/0x110
RSP: 0000:ffff8f86debbbbe8  EFLAGS: 00000282
RAX: ffff912ff2f07000 RBX: ffff8f980da59000 RCX: 0000000000000162
RDX: 0000000000000162 RSI: ffff8f8bf138ae9e RDI: ffff912ff2f07000
RBP: ffff8f86debbbc48 R08: 0000000000000162 R09: 0000000000000162
R10: ffff8f971e903200 R11: 0000000000003cf2 R12: ffff8f86debbbb88
R13: ffffffffb8abb244 R14: ffff8f86debbbb50 R15: ffffffffb8adb085
FS:  0000000000000000(0000) GS:ffff916beec80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000020f76a0 CR3: 0000000418f5c000 CR4: 0000000000760ee0
PKRU: 00000000
Call Trace:
[<ffffffffb903f6f3>] ? skb_copy_bits+0x133/0x2c0
[<ffffffffc049ecfd>] xdr_skb_read_bits+0x2d/0x50 [sunrpc]
[<ffffffffc049edd1>] xdr_partial_copy_from_skb+0xb1/0x240 [sunrpc]
[<ffffffffc049ecd0>] ? xprt_create_transport+0x260/0x260 [sunrpc]
[<ffffffffc04a3261>] xs_tcp_data_recv+0x621/0xad0 [sunrpc]
[<ffffffffb8c26822>] ? kmem_cache_free+0x1e2/0x200
[<ffffffffc04a2c40>] ? xs_tcp_setup_socket+0x490/0x490 [sunrpc]
[<ffffffffb90ae31b>] tcp_read_sock+0xab/0x1f0
[<ffffffffc049fc03>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffffb8abde3f>] process_one_work+0x17f/0x440
[<ffffffffb8abef56>] worker_thread+0x126/0x3c0
[<ffffffffb8abee30>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffffb8ac5e11>] kthread+0xd1/0xe0
[<ffffffffb8ac5d40>] ? insert_kthread_work+0x40/0x40
[<ffffffffb9194de4>] ret_from_fork_nospec_begin+0xe/0x21
[<ffffffffb8ac5d40>] ? insert_kthread_work+0x40/0x40
Code: 4b b9 31 c0 e8 fc 62 d0 ff 0f ae e8 0f 31 48 c1 e2 20 89 c0 48 09 c2 48 31 d3 e9 7b ff ff ff 90 90 90 90 90 90 48 89 f8 48 89 d1 <f3> a4 c3 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 

NMI watchdog: BUG: soft lockup - CPU#7 stuck for 22s! [kworker/7:2H:15000]
Modules linked in: xt_multiport rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter sunrpc snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer ppdev iosf_mbi snd crc32_pclmul soundcore ghash_clmulni_intel sg aesni_intel virtio_rng lrw gf128mul parport_pc virtio_balloon joydev glue_helper parport ablk_helper i2c_piix4 cryptd pcspkr binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover qxl virtio_console virtio_blk failover drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul libata crct10dif_common floppy virtio_pci crc32c_intel
virtio_ring serio_raw drm_panel_orientation_quirks virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 7 PID: 15000 Comm: kworker/7:2H Kdump: loaded Not tainted 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff88aa81e6a100 ti: ffff88aab5d4c000 task.ti: ffff88aab5d4c000
RIP: 0010:[<ffffffff82b97ac6>]  [<ffffffff82b97ac6>] memcpy+0x6/0x110
RSP: 0018:ffff88aab5d4fbe8  EFLAGS: 00010282
RAX: ffff88e8dad0e000 RBX: 0000000000001000 RCX: 00000000000008b8
RDX: 00000000000008b8 RSI: ffff88e425d5e2e6 RDI: ffff88e8dad0e000
RBP: ffff88aab5d4fc48 R08: 00000000000008b8 R09: 00000000000008b8
R10: ffff88ac34f78500 R11: 0000000000005a80 R12: 00000000000001c8
R13: 0000000000000000 R14: 0000000000000b50 R15: 000000000000003e
FS:  0000000000000000(0000) GS:ffff88e9bf1c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000c9ff007 CR3: 0000003f2a4a6000 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff82e406f3>] ? skb_copy_bits+0x133/0x2c0
[<ffffffffc0796cfd>] xdr_skb_read_bits+0x2d/0x50 [sunrpc]
[<ffffffffc0796dd1>] xdr_partial_copy_from_skb+0xb1/0x240 [sunrpc]
[<ffffffffc0796cd0>] ? xprt_create_transport+0x260/0x260 [sunrpc]
[<ffffffffc079b261>] xs_tcp_data_recv+0x621/0xad0 [sunrpc]
[<ffffffff82a27022>] ? kmem_cache_free+0x1e2/0x200
[<ffffffffc079ac40>] ? xs_tcp_setup_socket+0x490/0x490 [sunrpc]
[<ffffffff82eaff0b>] tcp_read_sock+0xab/0x1f0
[<ffffffffc0797c03>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffff828bde8f>] process_one_work+0x17f/0x440
[<ffffffff828befa6>] worker_thread+0x126/0x3c0
[<ffffffff828bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff828c5e61>] kthread+0xd1/0xe0
[<ffffffff828c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffff82f95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff828c5d90>] ? insert_kthread_work+0x40/0x40
Code: 2b 83 31 c0 e8 3c 5c d0 ff 0f ae e8 0f 31 48 c1 e2 20 89 c0 48 09 c2 48 31 d3 e9 7b ff ff ff 90 90 90 90 90 90 48 89 f8 48 89 d1 <f3> a4 c3 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 

NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [kworker/2:2H:73455]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache bnep vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec iosf_mbi crc32_pclmul ac97_bus snd_seq btusb snd_seq_device btrtl ppdev ghash_clmulni_intel snd_pcm btbcm btintel aesni_intel bluetooth snd_timer snd lrw gf128mul glue_helper ablk_helper cryptd rfkill soundcore vmw_balloon pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod
cdrom sd_mod crc_t10dif ata_generic crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit ata_piix libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel mptspi libata scsi_transport_spi serio_raw mptscsih e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 73455 Comm: kworker/2:2H Kdump: loaded Tainted: G             L ------------   3.10.0-1160.42.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff89ba7a460000 ti: ffff89baf272c000 task.ti: ffff89baf272c000
RIP: 0010:[<ffffffff8db97b08>]  [<ffffffff8db97b08>] memcpy+0x68/0x110
RSP: 0018:ffff89baf272fbe8  EFLAGS: 00010206
RAX: ffff89baa1bee000 RBX: ffff89bb33fbb800 RCX: ffff89ba7a460000
RDX: 000000000000040c RSI: ffff89b9987d8e0a RDI: ffff89baa1bee44c
RBP: ffff89baf272fc48 R08: 62347830202c3232 R09: 3620202c32323220
R10: 202c30204c58500a R11: 633562347830202c R12: ffff89bb33659740
R13: ffff89baf1c42d00 R14: 00000000662404af R15: 1701a8c0f272fb70
FS:  0000000000000000(0000) GS:ffff89bb35680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000564ff4352580 CR3: 0000000234342000 CR4: 00000000003607e0
Call Trace:
[<ffffffff8de406f3>] ? skb_copy_bits+0x133/0x2c0
[<ffffffffc07b7cfd>] xdr_skb_read_bits+0x2d/0x50 [sunrpc]
[<ffffffffc07b7dd1>] xdr_partial_copy_from_skb+0xb1/0x240 [sunrpc]
[<ffffffffc07b7cd0>] ? xprt_create_transport+0x260/0x260 [sunrpc]
[<ffffffffc07bc261>] xs_tcp_data_recv+0x621/0xad0 [sunrpc]
[<ffffffff8de3f732>] ? skb_release_data+0x142/0x150
[<ffffffffc07bbc40>] ? xs_tcp_setup_socket+0x490/0x490 [sunrpc]
[<ffffffff8deaff0b>] tcp_read_sock+0xab/0x1f0
[<ffffffffc07b8c03>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffff8d8bde8f>] process_one_work+0x17f/0x440
[<ffffffff8d8befa6>] worker_thread+0x126/0x3c0
[<ffffffff8d8bee80>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff8d8c5e61>] kthread+0xd1/0xe0
[<ffffffff8d8c5d90>] ? insert_kthread_work+0x40/0x40
[<ffffffff8df95df7>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffff8d8c5d90>] ? insert_kthread_work+0x40/0x40
Code: d4 83 c2 20 eb 44 48 01 d6 48 01 d7 48 83 ea 20 0f 1f 00 48 83 ea 20 4c 8b 46 f8 4c 8b 4e f0 4c 8b 56 e8 4c 8b 5e e0 48 8d 76 e0 <4c> 89 47 f8 4c 89 4f f0 4c 89 57 e8 4c 89 5f e0 48 8d 7f e0 73 

NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [kworker/5:0H:7671]
Modules linked in: vsock_diag tcp_diag udp_diag inet_diag unix_diag tcp_lp fuse nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_CHECKSUM tun devlink nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_MASQUERADE nf_nat_masquerade_ipv6 xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_mark ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc
ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg joydev pcspkr vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common ahci crc32c_intel ata_piix serio_raw drm libahci libata vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 5 PID: 7671 Comm: kworker/5:0H Kdump: loaded Tainted: G             L ------------   3.10.0-957.10.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff9745b5cde180 ti: ffff97547e258000 task.ti: ffff97547e258000
RIP: 0010:[<ffffffffb0f86a08>]  [<ffffffffb0f86a08>] memcpy+0x58/0x110
RSP: 0018:ffff97547e25bbf8  EFLAGS: 00000202
RAX: ffff974c79077e18 RBX: 0000000000000010 RCX: ffff97547e258000
RDX: 00000000000001a8 RSI: ffff972b88f47aea RDI: ffff974c79078000
RBP: ffff97547e25bc50 R08: 09cd1104e50404e7 R09: 00000000000001e8
R10: 00000000000001e8 R11: ffff974c79077e18 R12: ffffffffffffff10
R13: ffff974c790d42d0 R14: ffff972d3474fe8a R15: 0000000000000290
FS:  0000000000000000(0000) GS:ffff9763bf340000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6bd6b69720 CR3: 0000003f30ffa000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 00000000
Call Trace:
[<ffffffffb122300b>] ? skb_copy_bits+0x13b/0x280
[<ffffffffc07a7c5d>] xdr_skb_read_bits+0x2d/0x50 [sunrpc]
[<ffffffffc07a7d31>] xdr_partial_copy_from_skb+0xb1/0x210 [sunrpc]
[<ffffffffc07a7c30>] ? xprt_create_transport+0x260/0x260 [sunrpc]
[<ffffffffc07ac121>] xs_tcp_data_recv+0x621/0xad0 [sunrpc]
[<ffffffffb0e1bc72>] ? kmem_cache_free+0x1e2/0x200
[<ffffffffc07abb00>] ? xs_tcp_setup_socket+0x490/0x490 [sunrpc]
[<ffffffffb12919db>] tcp_read_sock+0xab/0x1f0
[<ffffffffc07a8b03>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffffb0cb9d8f>] process_one_work+0x17f/0x440
[<ffffffffb0cbae26>] worker_thread+0x126/0x3c0
[<ffffffffb0cbad00>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffb0cc1c71>] kthread+0xd1/0xe0
[<ffffffffb0cc1ba0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb1375c1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffffb0cc1ba0>] ? insert_kthread_work+0x40/0x40
Code: 89 4f 08 4c 89 57 10 4c 89 5f 18 48 8d 7f 20 73 d4 83 c2 20 eb 44 48 01 d6 48 01 d7 48 83 ea 20 0f 1f 00 48 83 ea 20 4c 8b 46 f8 <4c> 8b 4e f0 4c 8b 56 e8 4c 8b 5e e0 48 8d 76 e0 4c 89 47 f8 4c 

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [kworker/3:1H:310]
Modules linked in: fuse uinput xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_hda_codec_generic snd_hda_intel snd_hda_codec ppdev snd_hda_core snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore parport_pc joydev pcspkr parport i2c_piix4 virtio_balloon nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c ata_generic pata_acpi qxl drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm ahci virtio_net virtio_console virtio_blk drm libahci ata_piix libata crct10dif_pclmul crct10dif_common serio_raw virtio_pci crc32c_intel virtio_ring virtio i2c_core floppy
CPU: 3 PID: 310 Comm: kworker/3:1H Tainted: G             L ------------   3.10.0-693.5.2.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff880134fb9fa0 ti: ffff880035cc0000 task.ti: ffff880035cc0000
RIP: 0010:[<ffffffff81331726>]  [<ffffffff81331726>] memcpy+0x6/0x110
RSP: 0018:ffff880035cc3bf8  EFLAGS: 00010286
RAX: ffff8800b7260000 RBX: 00000000000003ec RCX: 00000000000003ec
RDX: 00000000000003ec RSI: ffff88006aed120a RDI: ffff8800b7260000
RBP: ffff880035cc3c50 R08: ffff88005c3b9400 R09: 00000000000003ec
R10: 00000000000003ec R11: ffff8800b7260000 R12: 0000000000009a74
R13: 000000000000000e R14: 0000000000009e60 R15: ffff88005c3b9400
FS:  0000000000000000(0000) GS:ffff88013fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000af5b6003 CR3: 000000006acfb000 CR4: 00000000001406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffffffff8157198b ffff880035cc3fd8 00009e6000000082 00000000000003ec
ffff880035cc0000 ffff8800000003ec ffff880035cc3ce8 00000000000003ec
0000000000001000 ffff880035cc3fd8 ffff880035cc3fd8 ffff880035cc3c70
Call Trace:
[<ffffffff8157198b>] ? skb_copy_bits+0x13b/0x280
[<ffffffffc02a09dd>] xdr_skb_read_bits+0x2d/0x50 [sunrpc]
[<ffffffffc02a0aae>] xdr_partial_copy_from_skb+0xae/0x210 [sunrpc]
[<ffffffffc02a09b0>] ? xprt_create_transport+0x260/0x260 [sunrpc]
[<ffffffffc02a4ffe>] xs_tcp_data_recv+0x60e/0xab0 [sunrpc]
[<ffffffffc02a49f0>] ? xs_tcp_state_change+0x2b0/0x2b0 [sunrpc]
[<ffffffff815d7b69>] tcp_read_sock+0xa9/0x1e0
[<ffffffffc02a1863>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffff810a882a>] process_one_work+0x17a/0x440
[<ffffffff810a94f6>] worker_thread+0x126/0x3c0
[<ffffffff810a93d0>] ? manage_workers.isra.24+0x2a0/0x2a0
[<ffffffff810b099f>] kthread+0xcf/0xe0
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
[<ffffffff816b4fd8>] ret_from_fork+0x58/0x90
[<ffffffff810b08d0>] ? insert_kthread_work+0x40/0x40
Code: 93 81 31 c0 e8 ac 86 d5 ff 66 66 90 0f ae e8 0f 31 48 c1 e2 20 89 c0 48 09 c2 48 31 d3 e9 78 ff ff ff 90 90 90 48 89 f8 48 89 d1 <f3> a4 c3 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 

NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kworker/1:0H:25527]
Modules linked in: binfmt_misc udp_diag tcp_diag inet_diag nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev virtio_balloon pcspkr parport_pc parport i2c_piix4 ip_tables ext4 mbcache jbd2 virtio_blk virtio_net crct10dif_pclmul crct10dif_common crc32c_intel ata_generic pata_acpi serio_raw cirrus drm_kms_helper syscopyarea sysfillrect floppy sysimgblt fb_sys_fops ttm ata_piix drm libata virtio_pci virtio_ring
drm_panel_orientation_quirks virtio
CPU: 1 PID: 25527 Comm: kworker/1:0H Tainted: G             L ------------   3.10.0-957.10.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffff9296761a0000 ti: ffff929644c20000 task.ti: ffff929644c20000
RIP: 0010:[<ffffffffb0b869bd>]  [<ffffffffb0b869bd>] memcpy+0xd/0x110
RSP: 0018:ffff929644c23bf8  EFLAGS: 00010246
RAX: ffff9296a7690000 RBX: ffff92965c56bc58 RCX: 0000000000000097
RDX: 0000000000000000 RSI: ffff92964fb8a7de RDI: ffff9296a7690000
RBP: ffff929644c23c50 R08: ffff9296bb067600 R09: 00000000000004b8
R10: 00000000000004b8 R11: ffff9296a7690000 R12: 0000000000000000
R13: ffffffffb08b7ad2 R14: ffff929644c23b88 R15: ffffffffb08b7264
FS:  0000000000000000(0000) GS:ffff9296bfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000b025c001 CR3: 000000000bfbe000 CR4: 00000000000406e0
Call Trace:
[<ffffffffb0e2300b>] ? skb_copy_bits+0x13b/0x280
[<ffffffffc05bdc5d>] xdr_skb_read_bits+0x2d/0x50 [sunrpc]
[<ffffffffc05bdd31>] xdr_partial_copy_from_skb+0xb1/0x210 [sunrpc]
[<ffffffffc05bdc30>] ? xprt_create_transport+0x260/0x260 [sunrpc]
[<ffffffffc05c204e>] xs_tcp_data_recv+0x54e/0xad0 [sunrpc]
[<ffffffffc05c1b00>] ? xs_tcp_setup_socket+0x490/0x490 [sunrpc]
[<ffffffffb0e919db>] tcp_read_sock+0xab/0x1f0
[<ffffffffc05beb03>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffffb08b9d8f>] process_one_work+0x17f/0x440
[<ffffffffb08bae26>] worker_thread+0x126/0x3c0
[<ffffffffb08bad00>] ? manage_workers.isra.25+0x2a0/0x2a0
[<ffffffffb08c1c71>] kthread+0xd1/0xe0
[<ffffffffb08c1ba0>] ? insert_kthread_work+0x40/0x40
[<ffffffffb0f75c37>] ret_from_fork_nospec_begin+0x21/0x21
[<ffffffffb08c1ba0>] ? insert_kthread_work+0x40/0x40
Code: d1 ff 0f ae e8 0f 31 48 c1 e2 20 89 c0 48 09 c2 48 31 d3 e9 7b ff ff ff 90 90 90 90 90 90 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b 4e 08 4c 8b 56 10 4c 

NMI watchdog: BUG: soft lockup - CPU#5 stuck for 24s! [kworker/5:1H:7094]
Modules linked in: uinput binfmt_misc xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ppdev sg iosf_mbi crc32_pclmul virtio_rng ghash_clmulni_intel parport_pc i2c_piix4 parport aesni_intel pcspkr joydev nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables
xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif ata_generic pata_acpi virtio_net virtio_console virtio_scsi qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata crct10dif_pclmul virtio_pci virtio_ring floppy serio_raw virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log authenc cmac zlib wp512 twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common tea sha512_ssse3 sha512_generic serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic seed salsa20_generic salsa20_x86_64 rmd320 rmd256 rmd160 rmd128 michael_mic md4 khazad ghash_generic gcm fcrypt drbg dm_crypt dm_mod des_generic deflate cts crypto_null crct10dif_generic crct10dif_common crc32c_intel ccm cast6_avx_x86_64 cast6_generic cast_common camellia_generic
camellia_aesni_avx2 camellia_aesni_avx_x86_64 ablk_helper cryptd camellia_x86_64 xts lrw gf128mul glue_helper blowfish_generic blowfish_x86_64 blowfish_common arc4 ansi_cprng [last unloaded: tcrypt]
CPU: 5 PID: 7094 Comm: kworker/5:1H Kdump: loaded Tainted: G             L ------------   3.10.0-1062.9.1.el7.x86_64 #1
Hardware name: oVirt oVirt Node, BIOS 1.11.0-2.el7 04/01/2014
Workqueue: xprtiod xs_tcp_data_receive_workfn [sunrpc]
task: ffffa1112fe60000 ti: ffffa110f65f0000 task.ti: ffffa110f65f0000
RIP: 0010:[<ffffffff90d92686>]  [<ffffffff90d92686>] memcpy+0x6/0x110
RSP: 0018:ffffa110f65f3be8  EFLAGS: 00010282
RAX: ffffa13e5823c000 RBX: 0000000000001000 RCX: 00000000000000f8
RDX: 00000000000000f8 RSI: ffffa13a37a7f4fe RDI: ffffa13e5823c000
RBP: ffffa110f65f3c48 R08: 00000000000000f8 R09: 00000000000000f8
R10: ffffa17efdbf4a00 R11: 00000000000005a8 R12: 00000000000004b0
R13: 0000000000000000 R14: 00000000000005a8 R15: 000000000000003e
FS:  0000000000000000(0000) GS:ffffa17fbd140000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000002d6a1005 CR3: 0000006ee1ef6000 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff91039973>] ? skb_copy_bits+0x133/0x2c0
[<ffffffffc078ccfd>] xdr_skb_read_bits+0x2d/0x50 [sunrpc]
[<ffffffffc078cdd1>] xdr_partial_copy_from_skb+0xb1/0x240 [sunrpc]
[<ffffffffc078ccd0>] ? xprt_create_transport+0x260/0x260 [sunrpc]
[<ffffffffc079118e>] xs_tcp_data_recv+0x54e/0xad0 [sunrpc]
[<ffffffffc0790c40>] ? xs_tcp_setup_socket+0x490/0x490 [sunrpc]
[<ffffffff910a8beb>] tcp_read_sock+0xab/0x1f0
[<ffffffffc078dc03>] xs_tcp_data_receive_workfn+0xb3/0x140 [sunrpc]
[<ffffffff90abe21f>] process_one_work+0x17f/0x440
[<ffffffff90abf336>] worker_thread+0x126/0x3c0
[<ffffffff90abf210>] ? manage_workers.isra.26+0x2a0/0x2a0
[<ffffffff90ac61f1>] kthread+0xd1/0xe0
[<ffffffff90ac6120>] ? insert_kthread_work+0x40/0x40
[<ffffffff9118dd1d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff90ac6120>] ? insert_kthread_work+0x40/0x40
Code: 4a 91 31 c0 e8 5c b7 d0 ff 0f ae e8 0f 31 48 c1 e2 20 89 c0 48 09 c2 48 31 d3 e9 7b ff ff ff 90 90 90 90 90 90 48 89 f8 48 89 d1 <f3> a4 c3 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 20 4c 8b 06 4c 8b