NMI watchdog: BUG: soft lockup - CPU#14 stuck for 23s! [spades:62868]
Modules linked in: binfmt_misc fuse ext4 mbcache jbd2 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache ipmi_si vfat fat uas usb_storage mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase dell_rbu nf_conntrack_ftp ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support gpio_ich intel_powerclamp dcdbas coretemp joydev kvm_intel kvm irqbypass bnx2 serio_raw pcspkr sg ipmi_ssif ipmi_devintf ipmi_msghandler i7core_edac edac_core ata_generic pata_acpi acpi_power_meter lpc_ich
shpchp nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_common mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crc32c_intel libata megaraid_sas i2c_core [last unloaded: ipmi_si]
CPU: 14 PID: 62868 Comm: spades Tainted: G             L ------------   3.10.0-693.11.1.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R910/0P658H, BIOS 1.2.0 06/22/2010
task: ffff8840a7c0dee0 ti: ffff883ff87e4000 task.ti: ffff883ff87e4000
RIP: 0010:[<ffffffff810fa336>]  [<ffffffff810fa336>] native_queued_spin_lock_slowpath+0x116/0x1e0
RSP: 0018:ffff883ff87e78c8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff88564dafe4c8 RCX: 0000000000710000
RDX: ffff881fff8d7880 RSI: 0000000000610101 RDI: ffff887ffa751480
RBP: ffff883ff87e78c8 R08: ffff885fff8d7880 R09: 0000000000000000
R10: ffff885fff8d9be0 R11: ffffea016010ed00 R12: ffff883ff87e7868
R13: ffff88564dafe598 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  0000000000d422d0(0063) GS:ffff885fff8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000004c2000 CR3: 0000002aae325000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff883ff87e78d8 ffffffff8169e6cf ffff883ff87e78e8 ffffffff816abc20
ffff883ff87e7918 ffffffffc08038c7 ffff88564dafe598 0000000000000000
ffff8858043b4c00 0000000000000000 ffff883ff87e7a00 ffffffffc07ef7f5
Call Trace:
[<ffffffff8169e6cf>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff816abc20>] _raw_spin_lock+0x20/0x30
[<ffffffffc08038c7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc07ef7f5>] ext4_ext_map_blocks+0x795/0xf40 [ext4]
[<ffffffff811df65a>] ? kmem_cache_alloc+0x1ba/0x1e0
[<ffffffffc07bee10>] ext4_da_get_block_prep+0x370/0x4b0 [ext4]
[<ffffffff81236213>] ? alloc_page_buffers+0x63/0xc0
[<ffffffff8123830a>] __block_write_begin+0x17a/0x4b0
[<ffffffffc07beaa0>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc07f1ae9>] ? __ext4_journal_start_sb+0x69/0xe0 [ext4]
[<ffffffffc07c5337>] ext4_da_write_begin+0x167/0x350 [ext4]
[<ffffffff811824de>] generic_file_buffered_write+0x11e/0x2a0
[<ffffffff81184902>] __generic_file_aio_write+0x1e2/0x400
[<ffffffff812d3d43>] ? ima_file_check+0x63/0x1b0
[<ffffffff81184b79>] generic_file_aio_write+0x59/0xa0
[<ffffffffc07b9eeb>] ext4_file_write+0xdb/0x470 [ext4]
[<ffffffff8118296b>] ? unlock_page+0x2b/0x30
[<ffffffff811ad616>] ? do_read_fault.isra.44+0xe6/0x130
[<ffffffff8120006d>] do_sync_write+0x8d/0xd0
[<ffffffff81200b2d>] vfs_write+0xbd/0x1e0
[<ffffffff8120193f>] SyS_write+0x7f/0xe0
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 0d 48 98 83 e2 30 48 81 c2 80 78 01 00 48 03 14 c5 e0 fd b0 81 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 41 8b 40 08 <85> c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 17 0f b7 c2 

NMI watchdog: BUG: soft lockup - CPU#12 stuck for 23s! [kswapd0:152]
Modules linked in: binfmt_misc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr hpilo lpc_ich sg hpwdt ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi shpchp acpi_power_meter pcc_cpufreq ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 crct10dif_pclmul crct10dif_common drm_kms_helper igb serio_raw crc32c_intel syscopyarea ahci sysfillrect sysimgblt fb_sys_fops libahci ttm ptp libata pps_core drm dca hpsa i2c_algo_bit i2c_core scsi_transport_sas
CPU: 12 PID: 152 Comm: kswapd0 Not tainted 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: HP ProLiant DL380e Gen8, BIOS P73 08/02/2014
task: ffff88017be41fa0 ti: ffff880274a84000 task.ti: ffff880274a84000
RIP: 0010:[<ffffffffc03a64c7>]  [<ffffffffc03a64c7>] ext4_inode_touch_time_cmp+0x67/0x90 [ext4]
NMI watchdog: BUG: soft lockup - CPU#21 stuck for 23s! [in:imjournal:1187]
Modules linked in: binfmt_misc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr hpilo lpc_ich sg hpwdt ioatdma ipmi_si ipmi_devintf ipmi_msghandler wmi shpchp acpi_power_meter pcc_cpufreq ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 crct10dif_pclmul crct10dif_common drm_kms_helper igb serio_raw crc32c_intel syscopyarea ahci sysfillrect sysimgblt fb_sys_fops libahci ttm ptp libata pps_core drm dca hpsa i2c_algo_bit i2c_core scsi_transport_sas
CPU: 21 PID: 1187 Comm: in:imjournal Not tainted 3.10.0-693.5.2.el7.x86_64 #1
Hardware name: HP ProLiant DL380e Gen8, BIOS P73 08/02/2014
task: ffff88017bd03f40 ti: ffff880262f68000 task.ti: ffff880262f68000
RIP: 0010:[<ffffffff810fa3de>]  [<ffffffff810fa3de>] native_queued_spin_lock_slowpath+0x1be/0x1e0
RSP: 0018:ffff880262f6b8c8  EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffff880002b48c18 RCX: 0000000000000001
RDX: 0000000000000101 RSI: 0000000000000001 RDI: ffff88027473dc80
RBP: ffff880262f6b8c8 R08: 0000000000000101 R09: ffffffffc03937ca
R10: ffff88027fa59be0 R11: ffffea0001714680 R12: ffff880262f6b868
R13: ffff880002b48ce8 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00007fbda1a12700(0000) GS:ffff88027fa40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f1aa861a000 CR3: 0000000273259000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff880262f6b8d8 ffffffff8169e6bf ffff880262f6b8e8 ffffffff816abc10
ffff880262f6b918 ffffffffc03a78b7 ffff880002b48ce8 0000000000000000
ffff88005c51af00 0000000000000000 ffff880262f6ba00 ffffffffc03937e5
Call Trace:
[<ffffffff8169e6bf>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff816abc10>] _raw_spin_lock+0x20/0x30
[<ffffffffc03a78b7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc03937e5>] ext4_ext_map_blocks+0x795/0xf40 [ext4]
[<ffffffff811df64a>] ? kmem_cache_alloc+0x1ba/0x1e0
[<ffffffffc0362e00>] ext4_da_get_block_prep+0x370/0x4b0 [ext4]
[<ffffffff81236393>] ? alloc_page_buffers+0x63/0xc0
[<ffffffff8123848a>] __block_write_begin+0x17a/0x4b0
[<ffffffffc0362a90>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc0395ad9>] ? __ext4_journal_start_sb+0x69/0xe0 [ext4]
[<ffffffffc0369327>] ext4_da_write_begin+0x167/0x350 [ext4]
[<ffffffff811824de>] generic_file_buffered_write+0x11e/0x2a0
[<ffffffff81184902>] __generic_file_aio_write+0x1e2/0x400
[<ffffffff81184b79>] generic_file_aio_write+0x59/0xa0
[<ffffffffc035deeb>] ext4_file_write+0xdb/0x470 [ext4]
[<ffffffff81211c2d>] ? putname+0x3d/0x60
[<ffffffff81212662>] ? SYSC_renameat2+0x122/0x5a0
[<ffffffff812001ed>] do_sync_write+0x8d/0xd0
[<ffffffff81200cad>] vfs_write+0xbd/0x1e0
[<ffffffff81201abf>] SyS_write+0x7f/0xe0
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 37 81 fe 00 01 00 00 74 f4 e9 95 fe ff ff 0f 1f 80 00 00 00 00 83 fa 01 75 11 0f 1f 00 e9 69 fe ff ff 0f 1f 00 85 c0 74 0c f3 90 <8b> 07 0f b6 c0 83 f8 03 75 f0 b8 01 00 00 00 66 89 07 5d c3 f3 

NMI watchdog: BUG: soft lockup - CPU#22 stuck for 23s! [postmaster:8525]
Modules linked in: xt_multiport ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter arc4 rpcsec_gss_krb5 md4 nfsv4 nls_utf8 cifs nfs dns_resolver fscache mxm_wmi iTCO_wdt iTCO_vendor_support dcdbas ext4 mbcache jbd2 sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr lpc_ich mei_me sg mei ipmi_si ipmi_devintf ipmi_msghandler shpchp wmi acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm igb crct10dif_pclmul drm crct10dif_common crc32c_intel ahci
libahci libata ptp megaraid_sas pps_core dca i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 22 PID: 8525 Comm: postmaster Tainted: G             L ------------   3.10.0-693.11.6.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge T630/0NT78X, BIOS 2.5.4 08/17/2017
task: ffff88181aee9fa0 ti: ffff8817d52f0000 task.ti: ffff8817d52f0000
RIP: 0010:[<ffffffff810fc032>]  [<ffffffff810fc032>] native_queued_spin_lock_slowpath+0x112/0x1e0
RSP: 0018:ffff8817d52f3898  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8803613ad8b0 RCX: 0000000000b10000
RDX: ffff88301e3995c0 RSI: 0000000001690001 RDI: ffff883019982480
RBP: ffff8817d52f3898 R08: ffff88181e8d95c0 R09: 0000000000000000
R10: ffff88181e8db920 R11: ffffea00a56488c0 R12: ffff8817d52f3838
R13: ffff8803613ad980 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00007faa19e598c0(0000) GS:ffff88181e8c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007faa19e98000 CR3: 000000181b21c000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff816a070f>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff816ade20>] _raw_spin_lock+0x20/0x30
[<ffffffffc06028d7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc05ee805>] ext4_ext_map_blocks+0x795/0xf40 [ext4]
[<ffffffff811e161a>] ? kmem_cache_alloc+0x1ba/0x1e0
[<ffffffffc05bde10>] ext4_da_get_block_prep+0x370/0x4b0 [ext4]
[<ffffffff81238333>] ? alloc_page_buffers+0x63/0xc0
[<ffffffff8123a42a>] __block_write_begin+0x17a/0x4b0
[<ffffffffc05bdaa0>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc05f0af9>] ? __ext4_journal_start_sb+0x69/0xe0 [ext4]
[<ffffffffc05c4337>] ext4_da_write_begin+0x167/0x350 [ext4]
[<ffffffff811841ce>] generic_file_buffered_write+0x11e/0x2a0
[<ffffffff811865f2>] __generic_file_aio_write+0x1e2/0x400
[<ffffffff812d5f00>] ? process_measurement+0x210/0x250
[<ffffffff81186869>] generic_file_aio_write+0x59/0xa0
[<ffffffffc05b8eeb>] ext4_file_write+0xdb/0x470 [ext4]
[<ffffffff811f4d72>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
[<ffffffff8120202d>] do_sync_write+0x8d/0xd0
[<ffffffff81202aed>] vfs_write+0xbd/0x1e0
[<ffffffff812038ff>] SyS_write+0x7f/0xe0
[<ffffffff816b89fd>] system_call_fastpath+0x16/0x1b
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 c0 95 01 00 48 03 14 c5 20 8e b1 81 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b 

NMI watchdog: BUG: soft lockup - CPU#13 stuck for 23s! [python:252594]
Modules linked in: fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_service_time dell_smbios sparse_keymap iTCO_wdt iTCO_vendor_support dcdbas ext4 mbcache jbd2 skx_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ipmi_ssif glue_helper ablk_helper joydev cryptd
pcspkr sg ipmi_si ipmi_devintf ipmi_msghandler nfit wmi acpi_power_meter acpi_pad i2c_i801 libnvdimm lpc_ich mei_me mei shpchp dm_multipath nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc32c_intel lpfc mgag200 i2c_algo_bit drm_kms_helper megaraid_sas syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crc_t10dif crct10dif_generic ahci crct10dif_pclmul drm scsi_transport_fc libahci tg3 libata scsi_tgt crct10dif_common ptp pps_core i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 13 PID: 252594 Comm: python Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R940xa/08XR9M, BIOS 1.3.8 11/21/2018
task: ffff8822339f0fd0 ti: ffff8802edf20000 task.ti: ffff8802edf20000
RIP: 0010:[<ffffffff810fa322>]  [<ffffffff810fa322>] native_queued_spin_lock_slowpath+0x112/0x1e0
RSP: 0018:ffff8802edf238c8  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8822a9ea8408 RCX: 0000000000690000
RDX: ffff883ffe297880 RSI: 0000000000490101 RDI: ffff887fe5b38c80
RBP: ffff8802edf238c8 R08: ffff883ffe2d7880 R09: 0000000000000000
R10: ffff883ffe2d9be0 R11: ffffea0184ecf6c0 R12: ffff8802edf23868
R13: ffff8822a9ea84d8 R14: 0000000000000000 R15: 1fffffffffffffff
FS:  00002af9e9879380(0000) GS:ffff883ffe2c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00002afa03446024 CR3: 00000001e6f87000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff8802edf238d8 ffffffff8169e5ff ffff8802edf238e8 ffffffff816abb50
ffff8802edf23918 ffffffffc07c98b7 ffff8822a9ea84d8 0000000000000000
ffff88613b3db240 0000000000000000 ffff8802edf23a00 ffffffffc07b57e5
Call Trace:
[<ffffffff8169e5ff>] queued_spin_lock_slowpath+0xb/0xf
[<ffffffff816abb50>] _raw_spin_lock+0x20/0x30
[<ffffffffc07c98b7>] ext4_es_lru_add+0x57/0x90 [ext4]
[<ffffffffc07b57e5>] ext4_ext_map_blocks+0x795/0xf40 [ext4]
[<ffffffff811df6ca>] ? kmem_cache_alloc+0x1ba/0x1e0
[<ffffffffc0784e00>] ext4_da_get_block_prep+0x370/0x4b0 [ext4]
[<ffffffff81236283>] ? alloc_page_buffers+0x63/0xc0
[<ffffffff8123837a>] __block_write_begin+0x17a/0x4b0
[<ffffffffc0784a90>] ? ext4_da_invalidatepage+0x320/0x320 [ext4]
[<ffffffffc07b7ad9>] ? __ext4_journal_start_sb+0x69/0xe0 [ext4]
[<ffffffffc078b327>] ext4_da_write_begin+0x167/0x350 [ext4]
[<ffffffff811824ce>] generic_file_buffered_write+0x11e/0x2a0
[<ffffffff811848f2>] __generic_file_aio_write+0x1e2/0x400
[<ffffffff81184b69>] generic_file_aio_write+0x59/0xa0
[<ffffffffc077feeb>] ext4_file_write+0xdb/0x470 [ext4]
[<ffffffff811c43dd>] ? free_pages_and_swap_cache+0xad/0xd0
[<ffffffff811ad08c>] ? tlb_flush_mmu.part.63+0x6c/0xc0
[<ffffffff811ae375>] ? tlb_finish_mmu+0x55/0x60
[<ffffffff8120026d>] do_sync_write+0x8d/0xd0
[<ffffffff81200d2d>] vfs_write+0xbd/0x1e0
[<ffffffff81201b3f>] SyS_write+0x7f/0xe0
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 13 48 c1 ea 0d 48 98 83 e2 30 48 81 c2 80 78 01 00 48 03 14 c5 e0 fd b0 81 4c 89 02 41 8b 40 08 85 c0 75 0f 0f 1f 44 00 00 f3 90 <41> 8b 40 08 85 c0 74 f6 4d 8b 08 4d 85 c9 74 04 41 0f 18 09 8b