NMI watchdog: BUG: soft lockup - CPU#11 stuck for 22s! [telegraf:76449]
Modules linked in: joydev uinput xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nfsv3 nfs_acl cmac arc4 rpcsec_gss_krb5 md4 nls_utf8 nfsv4 cifs nfs ccm lockd dns_resolver grace ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter cachefiles fscache dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support vfat fat sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm
irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr sg i2c_i801 hpilo lpc_ich hpwdt ipmi_si ipmi_devintf ioatdma ipmi_msghandler dca pcc_cpufreq wmi acpi_power_meter auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 ahci i2c_algo_bit crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel libahci syscopyarea sysfillrect sysimgblt libata tg3 fb_sys_fops ttm ptp drm pps_core drm_panel_orientation_quirks
CPU: 11 PID: 76449 Comm: telegraf Kdump: loaded Tainted: G        W      ------------ T 3.10.0-957.el7.centos.plus.x86_64 #1
Hardware name: HP ProLiant DL360 Gen9/ProLiant DL360 Gen9, BIOS P89 04/29/2021
task: ffff9d8518389040 ti: ffff9d85c0bb8000 task.ti: ffff9d85c0bb8000
RIP: 0010:[<ffffffff9cb1209a>]  [<ffffffff9cb1209a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9d85c0bbbcd0  EFLAGS: 00000202
RAX: ffff9d84f0aa7900 RBX: ffff9d939f0d95a0 RCX: ffff9d84f0aa79a0
RDX: ffff9d857f51b740 RSI: ffff9d85c0bbbcd0 RDI: ffff9d85c0bbbcd0
RBP: ffff9d85c0bbbd20 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000001000 R11: ffffffff9cbd6e1d R12: ffffffff9cb81a14
R13: ffff9d85c0bbbc60 R14: 0000000000000000 R15: ffff9d939f0d9410
FS:  00007f4aadffb700(0000) GS:ffff9d857f6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c001e7d000 CR3: 00000020227aa000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9cdbbef0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff9cdbbef0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff9cb121af>] smp_call_function_single+0x5f/0xa0
[<ffffffff9cdbc04d>] rdmsr_on_cpu+0x5d/0x90
[<ffffffffc0af82b3>] show_temp+0xb3/0xe0 [coretemp]
[<ffffffff9ceb4fe3>] dev_attr_show+0x23/0x60
[<ffffffff9d177e12>] ? mutex_lock+0x12/0x2f
[<ffffffff9ccccadf>] sysfs_kf_seq_show+0xcf/0x1f0
[<ffffffff9cccb216>] kernfs_seq_show+0x26/0x30
[<ffffffff9cc698c0>] seq_read+0x130/0x440
[<ffffffff9cccbb65>] kernfs_fop_read+0x105/0x170
[<ffffffff9cc41f0f>] vfs_read+0x9f/0x170
[<ffffffff9cc42dcf>] SyS_read+0x7f/0xf0
[<ffffffff9d18705b>] tracesys+0xa3/0xc9
Code: 40 b7 01 00 48 89 de 48 03 14 c5 20 c2 74 9d 48 89 df e8 1a bf 28 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 

NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [python:4085]
Modules linked in: usdm_drv(OE) igb_uio(OE) sha512_ssse3 sha512_generic icp_qa_al(OE) bpctl_driver(OE) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vfat fat dm_mirror dm_region_hash dm_log dm_mod skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm iTCO_wdt iTCO_vendor_support irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ipmi_ssif
joydev lpc_ich sg mei_me i2c_i801 shpchp mei ipmi_si ipmi_devintf ipmi_msghandler nfit libnvdimm acpi_power_meter acpi_pad acpi_cpufreq ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel ixgbe ahci drm i40e igb libahci libata mdio qat_c62x(OE) ptp intel_qat(OE) pps_core i2c_algo_bit dca uio i2c_core
CPU: 19 PID: 4085 Comm: python Kdump: loaded Tainted: G        W  OE  ------------   3.10.0-862.el7.x86_64 #1
Hardware name: Supermicro Super Server/X11DPL-i, BIOS 3.8a 10/28/2022
task: ffff926be760eeb0 ti: ffff92673874c000 task.ti: ffff92673874c000
RIP: 0010:[<ffffffff9d70779a>]  [<ffffffff9d70779a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff92673874fcd0  EFLAGS: 00000202
RAX: 00000000000001c0 RBX: 0000000000000286 RCX: 0000000000000003
RDX: ffff9265328ff838 RSI: 0000000000000000 RDI: 0000000000000286
RBP: ffff92673874fd20 R08: ffff9265328ff808 R09: 0000000000000007
R10: 0000000000001000 R11: ffffffff9d7b32ed R12: ffff92673874fca0
R13: ffff9265328ff800 R14: 0000000000000010 R15: ffffffff9d94a275
FS:  00007f632fe09700(0000) GS:ffff92741c040000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6338684000 CR3: 0000000dbbe96000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff9d97d910>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff9d97d910>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff9d7078af>] smp_call_function_single+0x5f/0xa0
[<ffffffff9d97daad>] rdmsr_on_cpu+0x5d/0x90
[<ffffffffc06a32b3>] show_temp+0xb3/0xe0 [coretemp]
[<ffffffff9da6e2e3>] dev_attr_show+0x23/0x60
[<ffffffff9dd11002>] ? mutex_lock+0x12/0x2f
[<ffffffff9d8a3e8f>] sysfs_kf_seq_show+0xcf/0x1f0
[<ffffffff9d8a25d6>] kernfs_seq_show+0x26/0x30
[<ffffffff9d841410>] seq_read+0x110/0x3f0
[<ffffffff9d8a2e35>] kernfs_fop_read+0xf5/0x160
[<ffffffff9d81ab3f>] vfs_read+0x9f/0x170
[<ffffffff9d81ba0f>] SyS_read+0x7f/0xf0
[<ffffffff9dd1f7d5>] system_call_fastpath+0x1c/0x21
Code: 80 96 01 00 48 89 de 48 03 14 c5 a0 53 33 9e 48 89 df e8 4a 87 25 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 

NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [sensors:24850]
Modules linked in: 8021q garp mrp xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt ib_ucm dm_mirror dm_region_hash dm_log dm_mod rpcrdma rdma_ucm ib_uverbs ib_iser rdma_cm ib_umad iw_cm libiscsi ib_ipoib scsi_transport_iscsi ib_cm iTCO_wdt iTCO_vendor_support gpio_ich ppdev coretemp i2c_algo_bit ttm kvm_intel drm_kms_helper syscopyarea sysfillrect kvm sysimgblt irqbypass fb_sys_fops lpc_ich ata_generic serio_raw sg drm pcspkr pata_acpi i2c_i801 drm_panel_orientation_quirks e1000e
ib_mthca floppy parport_pc parport ptp ib_core ioatdma pps_core i5000_edac i5k_amb dca nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod crct10dif_common cdrom ahci ata_piix libahci libata [last unloaded: ipmi_msghandler]
CPU: 4 PID: 24850 Comm: sensors Kdump: loaded Tainted: G          I    ------------   3.10.0-957.12.1.el7.x86_64 #1
Hardware name: System manufacturer System Product Name/DSBF-D/1U, BIOS 1007     12/04/2007
task: ffff9a89ea19c100 ti: ffff9a89e57c8000 task.ti: ffff9a89e57c8000
RIP: 0010:[<ffffffff895111ee>]  [<ffffffff895111ee>] generic_exec_single+0xfe/0x1b0
RSP: 0018:ffff9a89e57cbcd0  EFLAGS: 00000202
RAX: 00000000000008fb RBX: ffff9a89ec756500 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff9a89e57cbd20 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000001000 R11: ffffffff895d626d R12: 00000000f8b6ddc3
R13: 0000000000000000 R14: 00000000f8b6ddc3 R15: ffff9a89e3ae7948
FS:  00002aec8d5a2600(0000) GS:ffff9a89ffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002aec8d590000 CR3: 00000004257e4000 CR4: 00000000000007e0
Call Trace:
[<ffffffff897aa890>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff897aa890>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff895112ff>] smp_call_function_single+0x5f/0xa0
[<ffffffff897aa9ed>] rdmsr_on_cpu+0x5d/0x90
[<ffffffffc06342b3>] show_temp+0xb3/0xe0 [coretemp]
[<ffffffff898a3e73>] dev_attr_show+0x23/0x60
[<ffffffff89b67072>] ? mutex_lock+0x12/0x2f
[<ffffffff896cc13f>] sysfs_kf_seq_show+0xcf/0x1f0
[<ffffffff896ca876>] kernfs_seq_show+0x26/0x30
[<ffffffff89668ef0>] seq_read+0x130/0x440
[<ffffffff896cb1c5>] kernfs_fop_read+0x105/0x170
[<ffffffff8964153f>] vfs_read+0x9f/0x170
[<ffffffff896423ff>] SyS_read+0x7f/0xf0
[<ffffffff89b75ce1>] ? system_call_after_swapgs+0xae/0x146
[<ffffffff89b75d9b>] system_call_fastpath+0x22/0x27
[<ffffffff89b75ce1>] ? system_call_after_swapgs+0xae/0x146
Code: 48 89 de 48 03 14 c5 a0 ba 14 8a 48 89 df e8 6a b7 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 f6 43 20 01 <75> f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 00 0f 85 87 

NMI watchdog: BUG: soft lockup - CPU#51 stuck for 22s! [sensors:46487]
Modules linked in: nvidia(OE+) tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver mmfs26(OE) mmfslinux(OE) tracedev(OE) nfsv3 nfs_acl nfs lockd grace fscache ip6table_filter ip6table_raw ip6_tables xt_comment xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi ext4 kvm_intel kvm mbcache jbd2 irqbypass crc32_pclmul ghash_clmulni_intel snd_hda_codec_hdmi aesni_intel snd_hda_codec_realtek lrw gf128mul glue_helper pcspkr ablk_helper snd_hda_codec_generic cryptd intel_wmi_thunderbolt i2c_i801 snd_hda_intel
lpc_ich snd_hda_codec joydev mei_me snd_seq snd_hda_core sg mei snd_seq_device snd_hwdep snd_pcm snd_timer snd ioatdma soundcore ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea mlx5_core sysfillrect sysimgblt fb_sys_fops igb ttm ahci mxm_wmi mlxfw devlink libahci crct10dif_pclmul drm crct10dif_common ptp libata crc32c_intel dca nvme pps_core i2c_algo_bit nvme_core drm_panel_orientation_quirks wmi
CPU: 51 PID: 46487 Comm: sensors Kdump: loaded Tainted: G           OEL ------------   3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: Supermicro SYS-7048GR-TR/X10DRG-Q, BIOS 2.0 12/31/2015
task: ffffa141699dc200 ti: ffffa14fcec18000 task.ti: ffffa14fcec18000
RIP: 0010:[<ffffffffb2f16a36>]  [<ffffffffb2f16a36>] generic_exec_single+0x106/0x1c0
RSP: 0018:ffffa14fcec1bcd0  EFLAGS: 00000202
RAX: 0000000000000048 RBX: 0000000000000000 RCX: 0000000000000038
RDX: 0000000000000048 RSI: 0000000000000008 RDI: 0000000000000282
RBP: ffffa14fcec1bd20 R08: ffffffffb36137c8 R09: 0000000000000000
R10: 0000000000001000 R11: ffffffffb2fde9fd R12: ffffa15dac41ca00
R13: ffffa13593b70000 R14: 00000000420e96f9 R15: 0000000000000000
FS:  00007f33e2f5f740(0000) GS:ffffa15dbfc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f33e2f79000 CR3: 0000003f04938000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb31bc0b0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffffb31bc0b0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffffb2f16b4f>] smp_call_function_single+0x5f/0xa0
[<ffffffffb31bc20d>] rdmsr_on_cpu+0x5d/0x90
[<ffffffffc0ba62b3>] show_temp+0xb3/0xe0 [coretemp]
[<ffffffffb32b70e3>] dev_attr_show+0x23/0x60
[<ffffffffb35875b2>] ? mutex_lock+0x12/0x2f
[<ffffffffb30dbedf>] sysfs_kf_seq_show+0xcf/0x1f0
[<ffffffffb30da596>] kernfs_seq_show+0x26/0x30
[<ffffffffb3076d10>] seq_read+0x130/0x450
[<ffffffffb30daef5>] kernfs_fop_read+0x105/0x170
[<ffffffffb304e3ff>] vfs_read+0x9f/0x170
[<ffffffffb304f27f>] SyS_read+0x7f/0xf0
[<ffffffffb3595f92>] system_call_fastpath+0x25/0x2a
Code: 14 c5 60 15 b5 b3 48 89 df e8 87 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 f6 43 20 01 <75> f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 00 0f 85 8f 

NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [AsyncMetrics:42749]
Modules linked in: binfmt_misc tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event sb_edac coretemp iosf_mbi kvm_intel ext4 kvm ppdev irqbypass mbcache crc32_pclmul ghash_clmulni_intel jbd2 aesni_intel snd_ens1371 lrw gf128mul glue_helper ablk_helper snd_rawmidi snd_ac97_codec cryptd vmw_balloon ac97_bus snd_seq snd_seq_device snd_pcm snd_timer
snd soundcore joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm libata e1000 mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 2 PID: 42749 Comm: AsyncMetrics Kdump: loaded Not tainted 3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/20/2014
task: ffff9cf5c504b0c0 ti: ffff9cf645e90000 task.ti: ffff9cf645e90000
RIP: 0010:[<ffffffff8271109a>]  [<ffffffff8271109a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9cf645e93cd0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9cf645e93ca0 RCX: 0000000000000001
RDX: ffffffff82e12d90 RSI: 0000000000000080 RDI: 0000000000000286
RBP: ffff9cf645e93d18 R08: ffffffff82e12d88 R09: 0000000000000000
R10: 0000000000001000 R11: 0000000000000293 R12: 0000000000000000
R13: ffff9cf645e93c4e R14: 0000000a0000ffff R15: ffffffff82980ee3
FS:  00007fd581809700(0000) GS:ffff9cf777680000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f44372ebff8 CR3: 000000010eb38000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff829aa1f0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff829aa1f0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff827111af>] smp_call_function_single+0x5f/0xa0
[<ffffffff829aa34d>] rdmsr_on_cpu+0x5d/0x90
[<ffffffffc08562b3>] show_temp+0xb3/0xe0 [coretemp]
[<ffffffff82aa32e3>] dev_attr_show+0x23/0x60
[<ffffffff82d65e12>] ? mutex_lock+0x12/0x2f
[<ffffffff828cbadf>] sysfs_kf_seq_show+0xcf/0x1f0
[<ffffffff828ca216>] kernfs_seq_show+0x26/0x30
[<ffffffff828688c0>] seq_read+0x130/0x440
[<ffffffff828cab65>] kernfs_fop_read+0x105/0x170
[<ffffffff82840f0f>] vfs_read+0x9f/0x170
[<ffffffff82841fc2>] SyS_pread64+0x92/0xc0
[<ffffffff82d74ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 34 83 48 89 df e8 1a b2 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 

NMI watchdog: BUG: soft lockup - CPU#43 stuck for 22s! [sensors:25309]
Modules linked in: nvidia_uvm(OE) binfmt_misc mmfs26(OE) mmfslinux(OE) tracedev(OE) nfsv3 nfs_acl nfs lockd grace fscache 8021q garp mrp xt_comment xt_multiport ip6table_raw ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 xt_conntrack iptable_filter iptable_raw iptable_security iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) bridge stp llc ip_set sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support sb_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm nouveau irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ses enclosure scsi_transport_sas
sg i2c_i801 mei_me lpc_ich joydev mei video ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ahci ttm mlx5_core libahci ixgbe drm mxm_wmi mlxfw devlink mdio crct10dif_pclmul crct10dif_common libata megaraid_sas ptp crc32c_intel drm_panel_orientation_quirks pps_core dca wmi [last unloaded: ip_tables]
CPU: 43 PID: 25309 Comm: sensors Kdump: loaded Tainted: P        W  OEL ------------   3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: NVIDIA DGX-1 with V100/DGX-1 with V100, BIOS S2W_3A08 02/14/2019
task: ffff9035f19cd280 ti: ffff90365b80c000 task.ti: ffff90365b80c000
RIP: 0010:[<ffffffff9d916a32>]  [<ffffffff9d916a32>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff90365b80fcd0  EFLAGS: 00000202
RAX: 0000000000000050 RBX: 0000000000000286 RCX: 0000000000000030
RDX: 000000000000ffff RSI: 0000000000000010 RDI: 0000000000000286
RBP: ffff90365b80fd20 R08: ffff8fbf1e69a008 R09: 0000000000000002
R10: 0000000000001000 R11: ffffffff9d9de9fd R12: ffff90365b80fca0
R13: ffff8fbf1e69a000 R14: 0000000000040001 R15: ffffffff9db86a45
FS:  00007f9dbf095740(0000) GS:ffff8ffcff9c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9dbf0b2000 CR3: 0000003c82f04000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9dbbc0b0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff9dbbc0b0>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff9d916b4f>] smp_call_function_single+0x5f/0xa0
[<ffffffff9dbbc20d>] rdmsr_on_cpu+0x5d/0x90
[<ffffffffc0b562b3>] show_temp+0xb3/0xe0 [coretemp]
[<ffffffff9dcb70e3>] dev_attr_show+0x23/0x60
[<ffffffff9df875b2>] ? mutex_lock+0x12/0x2f
[<ffffffff9dadbedf>] sysfs_kf_seq_show+0xcf/0x1f0
[<ffffffff9dada596>] kernfs_seq_show+0x26/0x30
[<ffffffff9da76d10>] seq_read+0x130/0x450
[<ffffffff9dadaef5>] kernfs_fop_read+0x105/0x170
[<ffffffff9da4e3ff>] vfs_read+0x9f/0x170
[<ffffffff9da4f27f>] SyS_read+0x7f/0xf0
[<ffffffff9df95f92>] system_call_fastpath+0x25/0x2a
Code: 89 de 48 03 14 c5 60 15 55 9e 48 89 df e8 87 71 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [sadc:28933]
Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp xfs iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper libcrc32c ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg nfit libnvdimm parport_pc vmw_vmci parport i2c_piix4 sch_fq_codel ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic serio_raw crct10dif_pclmul crct10dif_common crc32c_intel vmxnet3 vmwgfx drm_kms_helper floppy syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm ahci libahci drm ata_piix libata vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 28933 Comm: sadc Kdump: loaded Tainted: G             L ------------   3.10.0-957.1.3.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9c0f06544100 ti: ffff9c123bff0000 task.ti: ffff9c123bff0000
RIP: 0010:[<ffffffff8fb1117a>]  [<ffffffff8fb1117a>] generic_exec_single+0xfa/0x1b0
RSP: 0018:ffff9c123bff3cd0  EFLAGS: 00000202
RAX: 00000000000008fd RBX: ffff9c13293c04a0 RCX: 0000000000000000
RDX: 00000000000008fb RSI: 00000000000000fb RDI: 0000000000000286
RBP: ffff9c123bff3d20 R08: 0000000000000001 R09: 0000000000000000
R10: 0000000000001000 R11: ffffffff8fbd608d R12: 00000000ff5b9cab
R13: 0000000000000000 R14: 00000000ff5b9cab R15: ffff9c130bf57dc8
FS:  00007f12383c8740(0000) GS:ffff9c133fcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f12383f1000 CR3: 000000040f606000 CR4: 00000000001607e0
Call Trace:
[<ffffffff8fdaa460>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff8fdaa460>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff8fb1128f>] smp_call_function_single+0x5f/0xa0
[<ffffffff8fdaa5bd>] rdmsr_on_cpu+0x5d/0x90
[<ffffffffc04d12b3>] show_temp+0xb3/0xe0 [coretemp]
[<ffffffff8fea3553>] dev_attr_show+0x23/0x60
[<ffffffff90165e92>] ? mutex_lock+0x12/0x2f
[<ffffffff8fccbd4f>] sysfs_kf_seq_show+0xcf/0x1f0
[<ffffffff8fcca486>] kernfs_seq_show+0x26/0x30
[<ffffffff8fc68b30>] seq_read+0x130/0x440
[<ffffffff8fccadd5>] kernfs_fop_read+0x105/0x170
[<ffffffff8fc4117f>] vfs_read+0x9f/0x170
[<ffffffff8fc4203f>] SyS_read+0x7f/0xf0
[<ffffffff90174ddb>] system_call_fastpath+0x22/0x27
Code: 00 b7 01 00 48 89 de 48 03 14 c5 60 b9 74 90 48 89 df e8 aa b3 27 00 84 c0 75 46 45 85 ed 74 11 f6 43 20 01 74 0b 0f 1f 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00 

NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [python:16253]
Modules linked in: xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_nat nf_nat_ipv4 nf_nat overlay(T) vmw_vsock_vmci_transport vsock iptable_filter coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmw_vmci cryptd ppdev pcspkr sg vmw_balloon joydev parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul drm crct10dif_common ata_piix crc32c_intel mptspi scsi_transport_spi mptscsih libata serio_raw mptbase vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod nf_conntrack_ipv4 nf_defrag_ipv4
ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs nf_conntrack libcrc32c br_netfilter bridge stp llc
CPU: 11 PID: 16253 Comm: python Tainted: G             L ------------ T 3.10.0-1127.19.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff9f756b5c20e0 ti: ffff9f6e37f7c000 task.ti: ffff9f6e37f7c000
RIP: 0010:[<ffffffff82716f72>]  [<ffffffff82716f72>] generic_exec_single+0x102/0x1c0
RSP: 0018:ffff9f6e37f7fcd0  EFLAGS: 00000202
RAX: 0000000000000080 RBX: ffff9f6e37f7fca0 RCX: 000000000000000a
RDX: ffffffff82e14d50 RSI: 0000000000000080 RDI: 0000000000000286
RBP: ffff9f6e37f7fd20 R08: ffffffff82e14d48 R09: 0000000000000000
R10: 0000000000001000 R11: ffffffff827de28d R12: 0000000000000000
R13: 0000000000000000 R14: 00003e4eeee6bc83 R15: ffffffffc0490a09
FS:  00007febf02f3700(0000) GS:ffff9f759d8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007febfb69d000 CR3: 00000000b7e28000 CR4: 00000000000007e0
Call Trace:
[<ffffffff829ba600>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff829ba600>] ? sbitmap_queue_init_node+0x1b0/0x1b0
[<ffffffff8271708f>] smp_call_function_single+0x5f/0xa0
[<ffffffff829ba75d>] rdmsr_on_cpu+0x5d/0x90
[<ffffffffc08322b3>] show_temp+0xb3/0xe0 [coretemp]
[<ffffffff82ab5213>] dev_attr_show+0x23/0x60
[<ffffffff82d84022>] ? mutex_lock+0x12/0x2f
[<ffffffff828da90f>] sysfs_kf_seq_show+0xcf/0x1f0
[<ffffffff828d8fc6>] kernfs_seq_show+0x26/0x30
[<ffffffff82875a50>] seq_read+0x130/0x440
[<ffffffff828d9925>] kernfs_fop_read+0x105/0x170
[<ffffffff8284d06f>] vfs_read+0x9f/0x170
[<ffffffff8284dedf>] SyS_read+0x7f/0xf0
[<ffffffff82d92e15>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff82d92ed2>] system_call_fastpath+0x25/0x2a
[<ffffffff82d92e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 89 de 48 03 14 c5 a0 10 35 83 48 89 df e8 07 52 28 00 84 c0 75 53 45 85 ed 74 16 f6 43 20 01 74 10 0f 1f 84 00 00 00 00 00 f3 90 <f6> 43 20 01 75 f8 31 c0 48 8b 7c 24 28 65 48 33 3c 25 28 00 00