| Operating system | Count |
|---|---|
| CentOS 7 | 271 |
| Architecture | Count |
|---|---|
| x86_64 | 271 |
| Related packages | Count |
|---|---|
| kernel | 262 |
| kernel-lt | 9 |
Packages names with count less than the total count of reports are most probably not the cause of the problem.
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
cmpxchg_double_slab.isra.50
|
vmlinux |
0xffffffff96832954
Build id: 3.10.0-1160.81.1.el7.x86_64 |
0x24 |
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [kthreadd:2] Modules linked in: kcare(OE) xt_time xt_conntrack xt_owner ipt_REJECT nf_reject_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter xt_nat ip6t_REJECT nf_reject_ipv6 xt_NFLOG xt_set xt_multiport ip_set_bitmap_port ip_set_list_set ip_set_hash_net ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6table_filter ip6_tables ip_set tcp_diag inet_diag ext4 mbcache jbd2 nfnetlink_log loop vmw_vsock_vmci_transport vsock crc32_pclmul ghash_clmulni_intel ppdev aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr vmw_vmci sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ata_piix ttm libata drm nfit mptspi libnvdimm scsi_transport_spi crct10dif_pclmul crct10dif_common mptscsih crc32c_intel mptbase serio_raw vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse CPU: 3 PID: 2 Comm: kthreadd Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.81.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018 task: ffff8b2a32e61080 ti: ffff8b2a32e6c000 task.ti: ffff8b2a32e6c000 RIP: 0010:[<ffffffff96832954>] [<ffffffff96832954>] cmpxchg_double_slab.isra.50+0x24/0xf0 RSP: 0000:ffff8b2e4a5839b0 EFLAGS: 00000206 RAX: ffff8b2e0181b300 RBX: 0000000000000000 RCX: 000000010020001e RDX: 000000010020001f RSI: ffffe623d5060680 RDI: 0000000040000000 RBP: ffff8b2e4a583a00 R08: ffff8b2e0181ad00 R09: 000000010020001e R10: ffff8b2e0181b300 R11: ffff8b2e0181ad00 R12: ffff8b2e4a583928 R13: ffffffff96dc63f2 R14: ffff8b2e4a583a00 R15: ffff8b2e0181ad00 FS: 00007f5f78823700(0000) GS:ffff8b2e4a580000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 00007f59468b7000 CR3: 0000000fe4d98000 CR4: 00000000000407e0 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
cmpxchg_double_slab.isra.50
|
vmlinux |
0xffffffffac625934
Build id: 3.10.0-1127.19.1.el7.x86_64 |
0x24 |
NMI watchdog: BUG: soft lockup - CPU#17 stuck for 24s! [swapper/17:0] Modules linked in: tcp_lp gsch(OE) redirfs(OE) fuse dsa_filter(POE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 ppdev vmw_balloon sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi crct10dif_pclmul crct10dif_common scsi_transport_spi libata crc32c_intel mptscsih serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod CPU: 17 PID: 0 Comm: swapper/17 Kdump: loaded Tainted: P OEL ------------ 3.10.0-1127.19.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 task: ffff8a64ed6de2a0 ti: ffff8a64ed6f8000 task.ti: ffff8a64ed6f8000 RIP: 0010:[<ffffffffac625934>] [<ffffffffac625934>] cmpxchg_double_slab.isra.50+0x24/0xf0 RSP: 0018:ffff8a6f8da43b50 EFLAGS: 00010206 RAX: 0000000000000000 RBX: ffff8a6f8da43ad8 RCX: 0000000180040003 RDX: 0000000180040004 RSI: fffffe0def393c00 RDI: 0000000040000000 RBP: ffff8a6f8da43ba0 R08: ffff8a6f4e4f4000 R09: 0000000180040003 R10: 0000000000000000 R11: ffff8a6f4e4f4000 R12: ffff8a6f8da43ac8 R13: ffffffffacb93efa R14: ffff8a6f8da43ba0 R15: ffff8a6f4e4f4000 FS: 0000000000000000(0000) GS:ffff8a6f8da40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f8318cdec70 CR3: 0000000bedd4e000 CR4: 00000000003607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
cmpxchg_double_slab.isra.51
|
vmlinux |
0xffffffffb861b6a4
Build id: 3.10.0-957.12.2.el7.x86_64 |
0x24 |
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [trivial-rewrite:30350] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 33s! [kworker/0:1:30433] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack binfmt_misc ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac coretemp ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc vmw_vmci i2c_piix4 parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm vmxnet3 mptspi libata scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod CPU: 0 PID: 30433 Comm: kworker/0:1 Kdump: loaded Tainted: G L ------------ 3.10.0-957.12.2.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 NMI watchdog: BUG: soft lockup - CPU#1 stuck for 27s! [java:27291] Workqueue: events neigh_periodic_work task: ffff9ed64c2c6180 ti: ffff9ec32acbc000 task.ti: ffff9ec32acbc000 Modules linked in: ip6t_rpfilter RIP: 0010:[<ffffffffb861b6a4>] [<ffffffffb861b6a4>] cmpxchg_double_slab.isra.51+0x24/0xf0
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
cmpxchg_double_slab.isra.54
|
vmlinux |
0xffffffff811e3e34
Build id: 4.4.60-1.el7.elrepo.x86_64 |
0x24 |
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [puppet:30641] sched: RT throttling activated Modules linked in: zram lz4_compress vmw_vsock_vmci_transport vsock ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables iptable_filter sunrpc coretemp crct10dif_pclmul ppdev crc32_pclmul ghash_clmulni_intel vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr input_leds sg parport_pc parport 8250_fintek vmw_vmci shpchp i2c_piix4 acpi_cpufreq binfmt_misc sch_fq_codel ip_tables ext4 mbcache jbd2 sr_mod cdrom ata_generic pata_acpi sd_mod vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crc32c_intel ata_piix mptspi scsi_transport_spi mptscsih serio_raw mptbase vmxnet3 drm libata floppy fjes dm_mirror dm_region_hash dm_log dm_mod CPU: 0 PID: 30641 Comm: puppet Not tainted 4.4.60-1.el7.elrepo.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 task: ffff88001366e900 ti: ffff88001da98000 task.ti: ffff88001da98000 RIP: 0010:[<ffffffff811e3e34>] [<ffffffff811e3e34>] cmpxchg_double_slab.isra.54+0x24/0xe0 RSP: 0000:ffff8801b8e03ce0 EFLAGS: 00010206 RAX: ffff8801010fe9c0 RBX: ffff8801010fe640 RCX: 0000000100400001 RDX: 0000000100400002 RSI: ffffea0004043f80 RDI: 0000000040000000 RBP: ffff8801b8e03d30 R08: ffff8801010fe640 R09: 0000000100400001 R10: ffff8801010fe9c0 R11: ffffea0004043f80 R12: ffffea0004043f80 R13: 0000000000000000 R14: 0000000000400001 R15: ffff8801b8803b00 FS: 00007f29e4c59740(0000) GS:ffff8801b8e00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00000000079f1008 CR3: 000000001da78000 CR4: 00000000000006f0 Stack: 0000000000000003 ffff8801b8e0f0c0 ffff8801b8e0f168 ffff8801b8e03d40 ffffffff810ed5f2 00a33f26f0708078 ffff8801b8e0f0d8 ffff8801b8e0d600 ffffc90001e4d460 ffff8801010fe9c0 ffff8801b8e03df0 ffffffff811e5a26 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
cmpxchg_double_slab.isra.51
|
vmlinux |
0xffffffffa761b5aa
Build id: 3.10.0-957.10.1.el7.x86_64 |
0x2a |
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 22s! [kworker/u20:2:20436] Modules linked in: tcp_lp nls_utf8 isofs fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ixgbe_zc(OE) ptp pps_core vxlan ip6_udp_tunnel udp_tunnel uio pf_ring(OE) ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd vmw_balloon joydev pcspkr parport_pc parport sg vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmwgfx drm_kms_helper syscopyarea floppy sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: pps_core] CPU: 9 PID: 20436 Comm: kworker/u20:2 Kdump: loaded Tainted: G OEL ------------ 3.10.0-957.10.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 Workqueue: netns cleanup_net task: ffff934ceff3c100 ti: ffff934b99178000 task.ti: ffff934b99178000 RIP: 0010:[<ffffffffa761b5aa>] [<ffffffffa761b5aa>] cmpxchg_double_slab.isra.51+0x2a/0xf0 RSP: 0018:ffff934cf6403b40 EFLAGS: 00000246 RAX: ffff934b997dbb00 RBX: ffffffffc08bf016 RCX: 0000000100200018 RDX: 0000000100200019 RSI: fffff12a8d65f680 RDI: 0000000040000000 RBP: ffff934cf6403b90 R08: ffff934b997dbe00 R09: 0000000100200018 R10: ffff934b997dbb00 R11: fffff12a8d65f680 R12: ffff934cf6403ab8 R13: ffffffffa7b76df2 R14: ffff934cf6403b90 R15: ffff934b997dbe00 FS: 0000000000000000(0000) GS:ffff934cf6400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fbc985b1e98 CR3: 00000003be210000 CR4: 00000000000607e0 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
cmpxchg_double_slab.isra.51
|
vmlinux |
0xffffffff811e2cca
Build id: 3.10.0-693.21.1.el7.x86_64 |
0x2a |
NMI watchdog: BUG: soft lockup - CPU#24 stuck for 22s! [swapper/24:0] Modules linked in: cdc_ether usbnet mii vhost_net vhost macvtap macvlan ebt_arp tun fuse mpt3sas mpt2sas raid_class scsi_transport_sas mptctl mptbase nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache devlink dell_rbu geneve ip6_udp_tunnel udp_tunnel bnx2fc cnic uio fcoe libfcoe libfc scsi_transport_fc scsi_tgt scsi_transport_iscsi ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack openvswitch ip_set nfnetlink ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter sunrpc iTCO_wdt iTCO_vendor_support mxm_wmi dcdbas sb_edac edac_core intel_powerclamp coretemp intel_rapl iosf_mbi dm_service_time kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev sg pcspkr ipmi_si ipmi_devintf ipmi_msghandler shpchp acpi_power_meter mei_me mei wmi lpc_ich dm_multipath ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel ahci igb libahci ixgbe(OE) libata megaraid_sas ptp i2c_algo_bit pps_core i2c_core dca dm_mirror dm_region_hash dm_log dm_mod 8021q garp mrp bridge stp llc bonding CPU: 24 PID: 0 Comm: swapper/24 Tainted: G W OEL ------------ 3.10.0-693.21.1.el7.x86_64 #1 Hardware name: Dell Inc. PowerEdge R730xd/072T6D, BIOS 2.4.3 01/17/2017 task: ffff88012e13af70 ti: ffff88012e154000 task.ti: ffff88012e154000 RIP: 0010:[<ffffffff811e2cca>] [<ffffffff811e2cca>] cmpxchg_double_slab.isra.51+0x2a/0xf0 RSP: 0018:ffff885efe903d20 EFLAGS: 00000246 RAX: ffff882c05b1e3c0 RBX: ffffffff811e12ac RCX: 00000001802a0004 RDX: 00000001802a0005 RSI: ffffea00b016c780 RDI: 0000000040060000 RBP: ffff885efe903d70 R08: ffff882c05b1e300 R09: 00000001802a0004 R10: ffff882c05b1e3c0 R11: ffffea00b016c780 R12: ffff885efe903c98 R13: ffffffff816c1732 R14: ffff885efe903d70 R15: ffff882c05b1e300 FS: 0000000000000000(0000) GS:ffff885efe900000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f1f8c4745f0 CR3: 0000000001a02000 CR4: 00000000003627e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
cmpxchg_double_slab.isra.52
|
vmlinux |
0xffffffffaf222214
Build id: 3.10.0-1062.el7.x86_64 |
0x24 |
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 21s! [fileStore_clien:22020] Modules linked in: nfsv3 nfsv4 dns_resolver nfs fscache ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables rpcsec_gss_krb5 binfmt_misc ip_set nfnetlink vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg vmw_vmci i2c_piix4 parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace sunrpc xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod drm_kms_helper crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci nfit libahci drm ata_piix libnvdimm libata crct10dif_pclmul crct10dif_common crc32c_intel vmw_pvscsi serio_raw vmxnet3 drm_panel_orientation_quirks [last unloaded: ip_tables] CPU: 9 PID: 22020 Comm: fileStore_clien Kdump: loaded Tainted: G L ------------ 3.10.0-1062.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018 task: ffff8fe72c7e5230 ti: ffff8fe729f28000 task.ti: ffff8fe729f28000 RIP: 0010:[<ffffffffaf222214>] [<ffffffffaf222214>] cmpxchg_double_slab.isra.52+0x24/0xf0 RSP: 0000:ffff8feca7443990 EFLAGS: 00010206 RAX: 0000000000000000 RBX: ffff8feca7443940 RCX: 000000018020001f RDX: 0000000100200020 RSI: ffffea575880ea80 RDI: 0000000040000000 RBP: ffff8feca74439e0 R08: ffff8feca03ab900 R09: 000000018020001f R10: 0000000000000000 R11: ffff8feca03ab900 R12: ffff8feca7443908 R13: ffffffffaf78cef2 R14: ffff8feca74439e0 R15: ffff8feca03ab900 FS: 00007ef9b6d24700(0000) GS:ffff8feca7440000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007efbfff39258 CR3: 000000048372a000 CR4: 00000000007607e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 PKRU: 55555554 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
cmpxchg_double_slab.isra.52
|
vmlinux |
0xffffffff8a822214
Build id: 3.10.0-1062.1.2.el7.x86_64 |
0x24 |
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 127s! [dockerd:4775] Modules linked in: veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink binfmt_misc vmw_vsock_vmci_transport vsock sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus coretemp snd_seq iosf_mbi crc32_pclmul snd_seq_device snd_pcm ghash_clmulni_intel aesni_intel ppdev snd_timer snd lrw gf128mul vmw_balloon glue_helper soundcore ablk_helper cryptd sg joydev pcspkr parport_pc parport vmw_vmci i2c_piix4 sch_fq_codel ip_tables xfs libcrc32c sr_mod cdrom sd_mod ata_generic crc_t10dif pata_acpi crct10dif_generic vmwgfx crct10dif_pclmul crct10dif_common crc32c_intel drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops nfit ttm mptspi libnvdimm serio_raw scsi_transport_spi drm mptscsih e1000 ata_piix libata mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler] CPU: 2 PID: 4775 Comm: dockerd Kdump: loaded Tainted: G W L ------------ T 3.10.0-1062.1.2.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015 task: ffff9dabf03c1070 ti: ffff9dab87a14000 task.ti: ffff9dab87a14000 RIP: 0010:[<ffffffff8a822214>] [<ffffffff8a822214>] cmpxchg_double_slab.isra.52+0x24/0xf0 RSP: 0018:ffff9dabf5683c80 EFLAGS: 00010206 RAX: ffff9da9fcdb1b80 RBX: ffff9dabeaa902c0 RCX: 0000000180330026 RDX: 0000000180330027 RSI: fffff1e880f36c00 RDI: 0000000040080000 RBP: ffff9dabf5683cd0 R08: ffff9da9fcdb1400 R09: 0000000180330026 R10: ffff9da9fcdb1b80 R11: ffff9da9fcdb1400 R12: ffff9dabf5683bf8 R13: ffffffff8ad8defa R14: ffff9dabf5683cd0 R15: ffff9da9fcdb1400 FS: 00007fe90569a700(0000) GS:ffff9dabf5680000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000002621268 CR3: 00000001b8780000 CR4: 00000000001607e0 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
cmpxchg_double_slab.isra.51
|
vmlinux |
0xffffffff811dbf94
Build id: 3.10.0-514.el7.x86_64 |
0x24 |
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 25s! [abrtd:869] Modules linked in: xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle overlay ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc vmw_vsock_vmci_transport vsock sunrpc nfit libnvdimm snd_seq_midi snd_seq_midi_event ppdev vmw_balloon intel_powerclamp snd_ens1371 snd_rawmidi snd_ac97_codec coretemp ac97_bus iosf_mbi snd_seq crc32_pclmul ghash_clmulni_intel aesni_intel lrw sg snd_seq_device gf128mul snd_pcm snd_timer snd soundcore glue_helper ablk_helper cryptd pcspkr shpchp i2c_piix4 parport_pc parport vmw_vmci ip_tables xfs libcrc32c sd_mod sr_mod crc_t10dif cdrom crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel mptspi vmwgfx drm_kms_helper serio_raw syscopyarea sysfillrect sysimgblt fb_sys_fops scsi_transport_spi mptscsih ttm ata_piix mptbase drm e1000 libata i2c_core fjes dm_mirror dm_region_hash dm_log dm_mod CPU: 0 PID: 869 Comm: abrtd Tainted: G W L ------------ 3.10.0-514.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/02/2015 task: ffff8800bbb36dd0 ti: ffff880235974000 task.ti: ffff880235974000 RIP: 0010:[<ffffffff811dbf94>] [<ffffffff811dbf94>] cmpxchg_double_slab.isra.51+0x24/0xf0 RSP: 0018:ffff880235603c60 EFLAGS: 00010206 RAX: ffff88006e3bee00 RBX: ffff880066fd9e40 RCX: 0000000180400012 RDX: 0000000180400013 RSI: ffffea0001b8ef00 RDI: 0000000040082000 RBP: ffff880235603cb0 R08: ffff88006e3bd100 R09: 0000000180400012 R10: ffff88006e3bee00 R11: ffffea0001b8ef00 R12: ffff880235603bd8 R13: ffffffff8169721d R14: ffff880235603cb0 R15: ffff88006e3bd100 FS: 00007f56c1b75900(0000) GS:ffff880235600000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000002681bf0 CR3: 00000002358ba000 CR4: 00000000001407f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Stack: ffff880235603c70 ffffffff810f2944 ffff880235603cc0 ffffffff810b4e46 0031682276bccec8 ffffea00019bf640 ffff880235603d48 ffffffff811dc1f1 ffffc90001e40e00 ffffea0001b8ef00 ffff880235603d68 ffffffff811dc1f1 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
cmpxchg_double_slab.isra.50
|
vmlinux |
0xffffffffb34258b4
Build id: 3.10.0-1160.6.1.el7.x86_64 |
0x24 |
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 25s! [kworker/1:1:4568] Modules linked in: vmfsrtx_export(POE) vmfsrtx_bridge(POE) vmfsrtx_bridge_helper(OE) vmfsrtx_dev(POE) vmfsrtx_dev_helper(POE) xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ppdev ablk_helper cryptd joydev pcspkr vmw_balloon sg parport_pc vmw_vmci parport i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm nfit crct10dif_pclmul crct10dif_common crc32c_intel libnvdimm ata_piix mptspi scsi_transport_spi mptscsih libata mptbase serio_raw vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse [last unloaded: vmfsrtx_bridge_helper] CPU: 1 PID: 4568 Comm: kworker/1:1 Kdump: loaded Tainted: P W OEL ------------ 3.10.0-1160.6.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018 Workqueue: xfs-sync/dm-2 xfs_log_worker [xfs] task: ffff9e2f7e798000 ti: ffff9e2fbbbbc000 task.ti: ffff9e2fbbbbc000 RIP: 0010:[<ffffffffb34258b4>] [<ffffffffb34258b4>] cmpxchg_double_slab.isra.50+0x24/0xf0 RSP: 0018:ffff9e4d7ee43a60 EFLAGS: 00010206 RAX: 0000000000000000 RBX: ffff9e4d7ee43a10 RCX: 000000018020001f RDX: 0000000100200020 RSI: ffffd8824756e900 RDI: 0000000040000000 RBP: ffff9e4d7ee43ab0 R08: ffff9e2f95ba5100 R09: 000000018020001f R10: 0000000000000000 R11: ffff9e2f95ba5100 R12: ffff9e4d7ee439d8 R13: ffffffffb3995fba R14: ffff9e4d7ee43ab0 R15: ffff9e2f95ba5100 FS: 0000000000000000(0000) GS:ffff9e4d7ee40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fdd995c5000 CR3: 0000000163b68000 CR4: 00000000000407e0 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
cmpxchg_double_slab.isra.50
|
vmlinux |
0xffffffff93225904
Build id: 3.10.0-1127.13.1.el7.x86_64 |
0x24 |
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 21s! [zabbix_server:20823] Modules linked in: binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev sg pcspkr vmw_vmci parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix scsi_transport_spi mptscsih crct10dif_pclmul libata crct10dif_common crc32c_intel drm e1000 mptbase serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ipmi_msghandler] CPU: 3 PID: 20823 Comm: zabbix_server Kdump: loaded Tainted: G W L ------------ 3.10.0-1127.13.1.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015 task: ffffa0dd46f75230 ti: ffffa0df8ad7c000 task.ti: ffffa0df8ad7c000 RIP: 0010:[<ffffffff93225904>] [<ffffffff93225904>] cmpxchg_double_slab.isra.50+0x24/0xf0 RSP: 0000:ffffa0dfbfcc3d30 EFLAGS: 00010206 RAX: ffffa0dd1191d230 RBX: ffffa0dfbfcc3ce0 RCX: 0000000180070004 RDX: 0000000180070005 RSI: fffff45640464600 RDI: 0000000044040000 RBP: ffffa0dfbfcc3d80 R08: ffffa0dd1191e2a0 R09: 0000000180070004 R10: ffffa0dd1191d230 R11: ffffa0dd1191e2a0 R12: ffffa0dfbfcc3ca8 R13: ffffffff93793efa R14: ffffa0dfbfcc3d80 R15: ffffa0dd1191e2a0 FS: 00007f8509a0bc40(0000) GS:ffffa0dfbfcc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f3888010160 CR3: 00000002b1e5a000 CR4: 00000000000607e0 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
cmpxchg_double_slab.isra.50
|
vmlinux |
0xffffffff8c8258a4
Build id: 3.10.0-1127.el7.x86_64 |
0x24 |
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [watchdog/0:11] Modules linked in: tcp_lp fuse xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd pcspkr joydev vmw_balloon sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common crc32c_intel ata_piix drm serio_raw libata e1000 mptspi scsi_transport_spi mptscsih mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod CPU: 0 PID: 11 Comm: watchdog/0 Kdump: loaded Not tainted 3.10.0-1127.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/30/2013 task: ffff8d89b94fb150 ti: ffff8d89b9508000 task.ti: ffff8d89b9508000 RIP: 0010:[<ffffffff8c8258a4>] [<ffffffff8c8258a4>] cmpxchg_double_slab.isra.50+0x24/0xf0 RSP: 0000:ffff8d89bfc03800 EFLAGS: 00010206 RAX: ffff8d88df207700 RBX: ffff8d89bfc037b0 RCX: 000000018040000a RDX: 000000018040000b RSI: ffffe72a817c81c0 RDI: 0000000040000000 RBP: ffff8d89bfc03850 R08: ffff8d88df2074c0 R09: 000000018040000a R10: ffff8d88df207700 R11: ffff8d88df2074c0 R12: ffff8d89bfc03778 R13: ffffffff8cd93efa R14: ffff8d89bfc03850 R15: ffff8d88df2074c0 FS: 0000000000000000(0000) GS:ffff8d89bfc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f78a9b2a000 CR3: 00000000b6a82000 CR4: 00000000000007f0 Call Trace:
| Frame # | Function | Binary | Source or offset | Line |
|---|---|---|---|---|
| 1 |
cmpxchg_double_slab.isra.51
|
vmlinux |
0xffffffff81e1b304
Build id: 3.10.0-957.1.3.el7.x86_64 |
0x24 |
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 21s! [labmedica_hub.c:10280] crct10dif_generic sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci crct10dif_pclmul crct10dif_common crc32c_intel ata_piix serio_raw mptspi drm libahci scsi_transport_spi mptscsih e1000 libata mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: dsa_filter] CPU: 5 PID: 10280 Comm: labmedica_hub.c Kdump: loaded Tainted: P OEL ------------ 3.10.0-957.1.3.el7.x86_64 #1 Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018 task: ffff9010411630c0 ti: ffff900ea354c000 task.ti: ffff900ea354c000 RIP: 0010:[<ffffffff81e1b304>] [<ffffffff81e1b304>] cmpxchg_double_slab.isra.51+0x24/0xf0 RSP: 0018:ffff901e7f343c50 EFLAGS: 00010206 RAX: ffff901520963540 RBX: ffff901e7f343c10 RCX: 0000000180190004 RDX: 0000000180190005 RSI: ffffd73bda825880 RDI: 0000000040080000 RBP: ffff901e7f343ca0 R08: ffff901520962000 R09: 0000000180190004 R10: ffff901520963540 R11: ffffd73bda825880 R12: ffff901e7f343bc8 R13: ffffffff82375df2 R14: ffff901e7f343ca0 R15: ffff901520962000 FS: 0000000000000000(0000) GS:ffff901e7f340000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fa3365541a0 CR3: 0000000511f5c000 CR4: 00000000000607e0 Call Trace: