Info

Permalink
Function
multi_cpu_stop
First occurrence
2020-09-22
Last occurrence
2026-09-15
State
NEW
Components
kernel
Quality
-22

Statistics

Operating system Count
CentOS 7 1,188
Architecture Count
x86_64 1,188
Related packages Count
kernel 1,188
 0:3.10.0-514.el7 723
 0:3.10.0-514.10.2.el7 420
 0:3.10.0-514.26.2.el7 23
 0:3.10.0-514.16.1.el7 12
 0:3.10.0-327.18.2.el7 8
 0:3.10.0-514.el7_lustre 2

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #672855
Frame # Function Binary Source or offset Line
1
multi_cpu_stop
vmlinux 0xffffffff81116143
Build id: 3.10.0-514.el7.x86_64
0x83
2
cpu_stopper_thread
vmlinux 0xffffffff811163c6
Build id: 3.10.0-514.el7.x86_64
0x96
3
? __schedule
vmlinux 0xffffffff8168b070
Build id: 3.10.0-514.el7.x86_64
0x3b0
4
smpboot_thread_fn
vmlinux 0xffffffff810b90af
Build id: 3.10.0-514.el7.x86_64
0x12f
5
? lg_double_unlock
vmlinux 0xffffffff810b8f80
Build id: 3.10.0-514.el7.x86_64
0x90
6
kthread
vmlinux 0xffffffff810b052f
Build id: 3.10.0-514.el7.x86_64
0xcf
7
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
8
ret_from_fork
vmlinux 0xffffffff81696518
Build id: 3.10.0-514.el7.x86_64
0x58
9
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [migration/19:104]
NMI watchdog: BUG: soft lockup - CPU#19 stuck for 22s! [migration/19:104]
Modules linked in: tcp_diag udp_diag inet_diag 8021q garp mrp stp llc nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack iptable_filter bonding nls_utf8 isofs xfs libcrc32c intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd iTCO_wdt i2c_i801 iTCO_vendor_support sb_edac sg pcspkr edac_core i2c_core shpchp mei_me acpi_power_meter lpc_ich mei ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic crct10dif_pclmul crct10dif_common crc32c_intel ixgbe ahci libahci mpt3sas mdio libata ptp raid_class pps_core scsi_transport_sas dca scsi_ep_front(OE) fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 19 PID: 104 Comm: migration/19 Tainted: G        W  OEL ------------   3.10.0-514.el7.x86_64 #1
Hardware name: Huawei RH2288H V3/BC11HGSA0, BIOS 5.03 07/25/2018
task: ffff8810f962ce70 ti: ffff8810f9654000 task.ti: ffff8810f9654000
RIP: 0010:[<ffffffff81116143>]  [<ffffffff81116143>] multi_cpu_stop+0x83/0xf0
RSP: 0018:ffff8810f9657da0  EFLAGS: 00000293
RAX: ffffffff816b2e20 RBX: ffff8810f9657dc0 RCX: dead000000000200
RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff881b35e239b0
RBP: ffff8810f9657dc0 R08: 0000000000000000 R09: 0000000000000001
R10: 000000000000b9e7 R11: 0000000000000001 R12: ffff8810f90c0400
R13: ffffffff810c970b R14: ffff8810f9657d20 R15: ffff88203a2e6dd0
FS:  0000000000000000(0000) GS:ffff88203f240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa786102000 CR3: 00000000019ba000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff88203f24ff08 ffff88203f24ff00 ffff881b35e239b0 ffff881b35e239d8
ffff8810f9657e90 ffffffff811163c6 ffff8810f9657fd8 ffff88203f24ff10
0000000000000000 0000000000000000 ffff88203f256c40 ffff88203cd47080
Call Trace:
[<ffffffff811163c6>] cpu_stopper_thread+0x96/0x170
[<ffffffff8168b070>] ? __schedule+0x3b0/0x990
[<ffffffff810b90af>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f80>] ? lg_double_unlock+0x90/0x90
[<ffffffff810b052f>] kthread+0xcf/0xe0
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81696518>] ret_from_fork+0x58/0x90
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
Code: f6 75 65 f0 ff 4b 24 0f 94 c1 84 c9 44 89 e2 74 0f 8b 43 20 8b 73 10 8d 48 01 89 73 24 89 4b 20 83 fa 04 74 23 f3 90 44 8b 63 20 <41> 39 d4 74 f0 41 83 fc 02 75 c2 fa 66 0f 1f 44 00 00 eb c4 66 
Complete report #500881
Frame # Function Binary Source or offset Line
1
multi_cpu_stop
vmlinux 0xffffffff811167df
Build id: 3.10.0-514.26.2.el7.x86_64
0x7f
2
cpu_stopper_thread
vmlinux 0xffffffff81116a66
Build id: 3.10.0-514.26.2.el7.x86_64
0x96
3
? __schedule
vmlinux 0xffffffff8168c1f0
Build id: 3.10.0-514.26.2.el7.x86_64
0x3b0
4
smpboot_thread_fn
vmlinux 0xffffffff810b95bf
Build id: 3.10.0-514.26.2.el7.x86_64
0x12f
5
? lg_double_unlock
vmlinux 0xffffffff810b9490
Build id: 3.10.0-514.26.2.el7.x86_64
0x90
6
kthread
vmlinux 0xffffffff810b0a4f
Build id: 3.10.0-514.26.2.el7.x86_64
0xcf
7
? kthread_create_on_node
vmlinux 0xffffffff810b0980
Build id: 3.10.0-514.26.2.el7.x86_64
0x140
8
ret_from_fork
vmlinux 0xffffffff81697758
Build id: 3.10.0-514.26.2.el7.x86_64
0x58
9
? kthread_create_on_node
vmlinux 0xffffffff810b0980
Build id: 3.10.0-514.26.2.el7.x86_64
0x140
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [migration/4:28]
Modules linked in: veth xt_nat ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 xt_addrtype iptable_filter xt_conntrack nf_nat nf_conntrack br_netfilter bridge stp llc overlay(T) nfsv4 dns_resolver nfs fscache rpcsec_gss_krb5 ext4 mbcache jbd2 kvm_amd dcdbas kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_devintf ses enclosure pcspkr sp5100_tco ipmi_si shpchp ipmi_msghandler amd64_edac_mod sg i2c_piix4 edac_mce_amd edac_core fam15h_power k10temp acpi_power_meter nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
ttm crct10dif_pclmul crct10dif_common drm crc32c_intel ahci mpt2sas libahci serio_raw libata raid_class i2c_core megaraid_sas scsi_transport_sas bnx2 fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 28 Comm: migration/4 Tainted: G        W      ------------ T 3.10.0-514.26.2.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R815/0W13NR, BIOS 3.1.1 04/13/2013
task: ffff880169a10000 ti: ffff880169a0c000 task.ti: ffff880169a0c000
RIP: 0010:[<ffffffff811167df>]  [<ffffffff811167df>] multi_cpu_stop+0x7f/0xf0
RSP: 0000:ffff880169a0fda0  EFLAGS: 00000293
RAX: ffffffff816b6e60 RBX: ffff880169a0fd30 RCX: dead000000000200
RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff881fda797b70
RBP: ffff880169a0fdc0 R08: 0000000000000000 R09: 0000000000000001
R10: 000000000000b996 R11: 0000000000000001 R12: ffff880169a0fd20
R13: ffff880261876dd0 R14: 0000000000000004 R15: 0000000000000027
FS:  00007fa30effd700(0000) GS:ffff880fdfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007fa387f5d020 CR3: 000000701fe7d000 CR4: 00000000000407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff880fdfd0ff08 ffff880fdfd0ff00 ffff881fda797b70 ffff881fda797b98
ffff880169a0fe90 ffffffff81116a66 ffff880169a0ffd8 ffff880fdfd0ff10
0000000000000000 0000000000000000 ffff880fdfd16c40 ffff884fdeb25140
Call Trace:
[<ffffffff81116a66>] cpu_stopper_thread+0x96/0x170
[<ffffffff8168c1f0>] ? __schedule+0x3b0/0x990
[<ffffffff810b95bf>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b9490>] ? lg_double_unlock+0x90/0x90
[<ffffffff810b0a4f>] kthread+0xcf/0xe0
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81697758>] ret_from_fork+0x58/0x90
[<ffffffff810b0980>] ? kthread_create_on_node+0x140/0x140
Code: 75 05 45 84 f6 75 65 f0 ff 4b 24 0f 94 c1 84 c9 44 89 e2 74 0f 8b 43 20 8b 73 10 8d 48 01 89 73 24 89 4b 20 83 fa 04 74 23 f3 90 <44> 8b 63 20 41 39 d4 74 f0 41 83 fc 02 75 c2 fa 66 66 90 66 66 
Complete report #124131
Frame # Function Binary Source or offset Line
1
multi_cpu_stop
vmlinux 0xffffffff811033df
Build id: 3.10.0-327.18.2.el7.x86_64
0x7f
2
cpu_stopper_thread
vmlinux 0xffffffff81103666
Build id: 3.10.0-327.18.2.el7.x86_64
0x96
3
? __schedule
vmlinux 0xffffffff8163aae8
Build id: 3.10.0-327.18.2.el7.x86_64
0x2d8
4
smpboot_thread_fn
vmlinux 0xffffffff810ae54f
Build id: 3.10.0-327.18.2.el7.x86_64
0xff
5
? schedule
vmlinux 0xffffffff8163b139
Build id: 3.10.0-327.18.2.el7.x86_64
0x29
6
? lg_double_unlock
vmlinux 0xffffffff810ae450
Build id: 3.10.0-327.18.2.el7.x86_64
0x90
7
kthread
vmlinux 0xffffffff810a5aef
Build id: 3.10.0-327.18.2.el7.x86_64
0xcf
8
? kthread_create_on_node
vmlinux 0xffffffff810a5a20
Build id: 3.10.0-327.18.2.el7.x86_64
0x140
9
ret_from_fork
vmlinux 0xffffffff81646118
Build id: 3.10.0-327.18.2.el7.x86_64
0x58
10
? kthread_create_on_node
vmlinux 0xffffffff810a5a20
Build id: 3.10.0-327.18.2.el7.x86_64
0x140
BUG: soft lockup - CPU#17 stuck for 22s! [migration/17:334]
Modules linked in: xt_nat iptable_nat nf_nat_ipv4 nf_nat ebt_dnat ebtable_nat ipt_REJECT tun xt_CHECKSUM iptable_mangle ebt_ip ebt_arp veth rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache bridge stp llc nf_conntrack_ipv4 nf_defrag_ipv4 xt_physdev xt_conntrack nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter intel_powerclamp coretemp intel_rapl kvm_intel kvm iTCO_wdt iTCO_vendor_support crc32_pclmul ghash_clmulni_intel dcdbas aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ipmi_ssif ixgbe pcspkr mdio sg mei_me lpc_ich i2c_i801 ioatdma mfd_core sb_edac shpchp mei edac_core wmi ipmi_si ipmi_msghandler acpi_pad nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ast syscopyarea sysfillrect sysimgblt drm_kms_helper
crct10dif_pclmul crct10dif_common crc32c_intel ttm ahci igb libahci drm libata ptp pps_core dca i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 17 PID: 334 Comm: migration/17 Tainted: G             L ------------   3.10.0-327.18.2.el7.x86_64 #1
Hardware name: Dell BANFF12/0TTH1R, BIOS 2.6.0 10/09/2014
task: ffff880c3c382e00 ti: ffff880c3bc08000 task.ti: ffff880c3bc08000
RIP: 0010:[<ffffffff811033df>]  [<ffffffff811033df>] multi_cpu_stop+0x7f/0xf0
RSP: 0000:ffff880c3bc0bd88  EFLAGS: 00000293
RAX: ffffffff81660ae0 RBX: ffff880c4f6147c0 RCX: dead000000200200
RDX: 0000000000000001 RSI: 0000000000000282 RDI: ffff880a7314fb50
RBP: ffff880c3bc0bda8 R08: 0000000000000000 R09: 0000000000000001
R10: 000000000000b5c9 R11: 0000000000000002 R12: ffffffff810bd24b
R13: ffff880c3bc0bd08 R14: ffff880bde382280 R15: ffffffff810b5db5
FS:  0000000000000000(0000) GS:ffff880c4f720000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f4c9da59298 CR3: 000000000194a000 CR4: 00000000000427e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff880c4f72dce8 ffff880c4f72dce0 ffff880a7314fb50 ffff880a7314fb78
ffff880c3bc0be78 ffffffff81103666 ffff880c3bc0bfd8 ffff880c4f72dcf0
0000000000000000 0000000000000000 ffff880c4f7347c0 ffff88144d279f40
Call Trace:
[<ffffffff81103666>] cpu_stopper_thread+0x96/0x170
[<ffffffff8163aae8>] ? __schedule+0x2d8/0x900
[<ffffffff810ae54f>] smpboot_thread_fn+0xff/0x1a0
[<ffffffff8163b139>] ? schedule+0x29/0x70
[<ffffffff810ae450>] ? lg_double_unlock+0x90/0x90
[<ffffffff810a5aef>] kthread+0xcf/0xe0
[<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81646118>] ret_from_fork+0x58/0x90
[<ffffffff810a5a20>] ? kthread_create_on_node+0x140/0x140
Code: 75 05 45 84 ed 75 65 f0 ff 4b 24 0f 94 c1 84 c9 44 89 e2 74 0f 8b 43 20 8b 73 10 8d 48 01 89 73 24 89 4b 20 83 fa 04 74 23 f3 90 <44> 8b 63 20 41 39 d4 74 f0 41 83 fc 02 75 c2 fa 66 66 90 66 66 
Complete report #32095
Frame # Function Binary Source or offset Line
1
multi_cpu_stop
vmlinux 0xffffffff8111613f
Build id: 3.10.0-514.el7_lustre.x86_64
0x7f
2
cpu_stopper_thread
vmlinux 0xffffffff811163c6
Build id: 3.10.0-514.el7_lustre.x86_64
0x96
3
? __schedule
vmlinux 0xffffffff8168b1c0
Build id: 3.10.0-514.el7_lustre.x86_64
0x3b0
4
smpboot_thread_fn
vmlinux 0xffffffff810b90af
Build id: 3.10.0-514.el7_lustre.x86_64
0x12f
5
? lg_double_unlock
vmlinux 0xffffffff810b8f80
Build id: 3.10.0-514.el7_lustre.x86_64
0x90
6
kthread
vmlinux 0xffffffff810b052f
Build id: 3.10.0-514.el7_lustre.x86_64
0xcf
7
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7_lustre.x86_64
0x140
8
ret_from_fork
vmlinux 0xffffffff81696658
Build id: 3.10.0-514.el7_lustre.x86_64
0x58
9
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7_lustre.x86_64
0x140
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [migration/5:34]
Modules linked in: fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 vxlan ip6_udp_tunnel udp_tunnel iptable_mangle xt_mark veth iptable_filter xt_conntrack nf_nat bridge stp llc nf_conntrack_netlink nfnetlink nf_conntrack nfsv3 nfs fscache osc(OE) mgc(OE) lustre(OE) lmv(OE) mdc(OE) lov(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) sha512_ssse3 sha512_generic overlay(T) tun crypto_null libcfs(OE) xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp l2tp_ppp esp4 l2tp_netlink l2tp_core pppox ah4 ppp_generic slhc af_key dm_mirror dm_region_hash dm_log dm_mod intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul
iTCO_wdt ipmi_ssif ghash_clmulni_intel ipmi_devintf aesni_intel iTCO_vendor_support lrw gf128mul glue_helper ablk_helper cryptd dcdbas ipmi_si sg pcspkr ipmi_msghandler mei_me mei lpc_ich shpchp acpi_power_meter sb_edac wmi edac_core nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic mgag200 i2c_algo_bit drm_kms_helper crct10dif_pclmul crct10dif_common syscopyarea crc32c_intel sysfillrect sysimgblt fb_sys_fops ttm drm ahci libahci bnx2x i2c_core libata mdio libcrc32c ptp megaraid_sas pps_core fjes
CPU: 5 PID: 34 Comm: migration/5 Tainted: G           OEL ------------ T 3.10.0-514.el7_lustre.x86_64 #1
Hardware name: Dell Inc. PowerEdge R730/0H21J3, BIOS 1.2.10 03/09/2015
task: ffff8808fc035e20 ti: ffff8808fc088000 task.ti: ffff8808fc088000
RIP: 0010:[<ffffffff8111613f>]  [<ffffffff8111613f>] multi_cpu_stop+0x7f/0xf0
RSP: 0000:ffff8808fc08bda0  EFLAGS: 00000293
RAX: ffffffff816aff20 RBX: ffff8808fc08bd40 RCX: dead000000000200
RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff880845bbbb70
RBP: ffff8808fc08bdc0 R08: 0000000000000000 R09: 0000000000000001
R10: 000000000000bb63 R11: 0000000000000001 R12: ffff88105e496c40
R13: ffff8808fc1dc000 R14: 0000000000000000 R15: ffff8808fc08bdec
FS:  0000000000000000(0000) GS:ffff88105e480000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0a3d4ddbf0 CR3: 00000000019ba000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff88105e48ff08 ffff88105e48ff00 ffff880845bbbb70 ffff880845bbbb98
ffff8808fc08be90 ffffffff811163c6 ffff8808fc08bfd8 ffff88105e48ff10
0000000000000000 0000000000000000 ffff88105e496c40 ffff88105dbd0c80
Call Trace:
[<ffffffff811163c6>] cpu_stopper_thread+0x96/0x170
[<ffffffff8168b1c0>] ? __schedule+0x3b0/0x990
[<ffffffff810b90af>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f80>] ? lg_double_unlock+0x90/0x90
[<ffffffff810b052f>] kthread+0xcf/0xe0
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81696658>] ret_from_fork+0x58/0x90
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
Code: 75 05 45 84 f6 75 65 f0 ff 4b 24 0f 94 c1 84 c9 44 89 e2 74 0f 8b 43 20 8b 73 10 8d 48 01 89 73 24 89 4b 20 83 fa 04 74 23 f3 90 <44> 8b 63 20 41 39 d4 74 f0 41 83 fc 02 75 c2 fa 66 0f 1f 44 00 
Complete report #2155
Frame # Function Binary Source or offset Line
1
multi_cpu_stop
vmlinux 0xffffffff8111613f
Build id: 3.10.0-514.el7.x86_64
0x7f
2
cpu_stopper_thread
vmlinux 0xffffffff811163c6
Build id: 3.10.0-514.el7.x86_64
0x96
3
? __schedule
vmlinux 0xffffffff8168b070
Build id: 3.10.0-514.el7.x86_64
0x3b0
4
smpboot_thread_fn
vmlinux 0xffffffff810b90af
Build id: 3.10.0-514.el7.x86_64
0x12f
5
? lg_double_unlock
vmlinux 0xffffffff810b8f80
Build id: 3.10.0-514.el7.x86_64
0x90
6
kthread
vmlinux 0xffffffff810b052f
Build id: 3.10.0-514.el7.x86_64
0xcf
7
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
8
ret_from_fork
vmlinux 0xffffffff81696518
Build id: 3.10.0-514.el7.x86_64
0x58
9
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
NMI watchdog: BUG: soft lockup - CPU#11 stuck for 23s! [migration/11:64]
Modules linked in: xt_REDIRECT nf_nat_redirect ip_vs_rr xt_ipvs ip_vs xt_nat veth vxlan ip6_udp_tunnel udp_tunnel iptable_mangle xt_mark xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_nat_ipv4 nf_nat br_netfilter bridge stp llc overlay(T) ipmi_devintf iTCO_wdt iTCO_vendor_support gpio_ich dcdbas intel_powerclamp coretemp kvm irqbypass ipmi_ssif pcspkr sg ipmi_si ipmi_msghandler acpi_power_meter wmi i7core_edac edac_core shpchp lpc_ich nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common sr_mod cdrom ata_generic pata_acpi mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crc32c_intel serio_raw ata_piix
mptsas libata scsi_transport_sas i2c_core mptscsih mptbase bnx2 fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 11 PID: 64 Comm: migration/11 Tainted: G        W I  L ------------ T 3.10.0-514.el7.x86_64 #1
Hardware name: Dell Inc. PowerEdge R410/0WWR83, BIOS 1.11.0 07/20/2012
task: ffff88018272de20 ti: ffff88018275c000 task.ti: ffff88018275c000
RIP: 0010:[<ffffffff8111613f>]  [<ffffffff8111613f>] multi_cpu_stop+0x7f/0xf0
RSP: 0000:ffff88018275fda0  EFLAGS: 00000293
RAX: ffffffff816af9a0 RBX: ffff880501c58fb0 RCX: dead000000000200
RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff8806414cbb70
RBP: ffff88018275fdc0 R08: 0000000000000000 R09: 0000000000000001
R10: 000000000000b4fc R11: 0000000000000001 R12: 000000008275fd38
R13: ffffffff810bcf71 R14: ffff88018275fd28 R15: ffffffff811160c0
FS:  0000000000000000(0000) GS:ffff88042fb40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f49b0059610 CR3: 00000005820c2000 CR4: 00000000000007e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
ffff88042fb4ff08 ffff88042fb4ff00 ffff8806414cbb70 ffff8806414cbb98
ffff88018275fe90 ffffffff811163c6 ffff88018275ffd8 ffff88042fb4ff10
0000000000000000 0000000000000000 ffff88042fb56c40 ffff880617280000
Call Trace:
[<ffffffff811163c6>] cpu_stopper_thread+0x96/0x170
[<ffffffff8168b070>] ? __schedule+0x3b0/0x990
[<ffffffff810b90af>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f80>] ? lg_double_unlock+0x90/0x90
[<ffffffff810b052f>] kthread+0xcf/0xe0
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81696518>] ret_from_fork+0x58/0x90
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
Code: 75 05 45 84 f6 75 65 f0 ff 4b 24 0f 94 c1 84 c9 44 89 e2 74 0f 8b 43 20 8b 73 10 8d 48 01 89 73 24 89 4b 20 83 fa 04 74 23 f3 90 <44> 8b 63 20 41 39 d4 74 f0 41 83 fc 02 75 c2 fa 66 66 90 66 66 
Complete report #73
Frame # Function Binary Source or offset Line
1
multi_cpu_stop
vmlinux 0xffffffff81116143
Build id: 3.10.0-514.el7.x86_64
0x83
2
cpu_stopper_thread
vmlinux 0xffffffff811163c6
Build id: 3.10.0-514.el7.x86_64
0x96
3
? __schedule
vmlinux 0xffffffff8168b070
Build id: 3.10.0-514.el7.x86_64
0x3b0
4
smpboot_thread_fn
vmlinux 0xffffffff810b90af
Build id: 3.10.0-514.el7.x86_64
0x12f
5
? lg_double_unlock
vmlinux 0xffffffff810b8f80
Build id: 3.10.0-514.el7.x86_64
0x90
6
kthread
vmlinux 0xffffffff810b052f
Build id: 3.10.0-514.el7.x86_64
0xcf
7
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
8
ret_from_fork
vmlinux 0xffffffff81696518
Build id: 3.10.0-514.el7.x86_64
0x58
9
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [migration/10:60]
Modules linked in: nfnetlink_queue nfnetlink_log nfnetlink bluetooth rfkill vmw_vsock_vmci_transport vsock intel_powerclamp coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg vmw_balloon pcspkr parport_pc parport shpchp i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom ata_generic sd_mod crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect mptspi sysimgblt scsi_transport_spi fb_sys_fops ttm crct10dif_pclmul crct10dif_common ata_piix drm crc32c_intel serio_raw mptscsih libata mptbase vmxnet3 i2c_core floppy fjes dm_mirror dm_region_hash dm_log dm_mod
CPU: 10 PID: 60 Comm: migration/10 Tainted: G             L ------------   3.10.0-514.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff8804c5108000 ti: ffff8804c5110000 task.ti: ffff8804c5110000
RIP: 0010:[<ffffffff81116143>]  [<ffffffff81116143>] multi_cpu_stop+0x83/0xf0
RSP: 0000:ffff8804c5113da0  EFLAGS: 00000293
RAX: ffffffff816afea0 RBX: ffff8804c5113dc0 RCX: dead000000000200
RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff88042d2afb70
RBP: ffff8804c5113dc0 R08: 0000000000000000 R09: 0000000000000000
R10: 000000000000bcfd R11: 0000000000000001 R12: ffff8804c7c09000
R13: ffffffff810c970b R14: ffff8804c5113d20 R15: ffff88054123bec0
FS:  0000000000000000(0000) GS:ffff880637d00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6abd2d2a08 CR3: 00000000019ba000 CR4: 00000000003407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff880637d0ff08 ffff880637d0ff00 ffff88042d2afb70 ffff88042d2afb98
ffff8804c5113e90 ffffffff811163c6 ffff8804c5113fd8 ffff880637d0ff10
0000000000000000 0000000000000000 ffff880637d16c40 ffff880234b58c80
Call Trace:
[<ffffffff811163c6>] cpu_stopper_thread+0x96/0x170
[<ffffffff8168b070>] ? __schedule+0x3b0/0x990
[<ffffffff810b90af>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f80>] ? lg_double_unlock+0x90/0x90
[<ffffffff810b052f>] kthread+0xcf/0xe0
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81696518>] ret_from_fork+0x58/0x90
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
Code: f6 75 65 f0 ff 4b 24 0f 94 c1 84 c9 44 89 e2 74 0f 8b 43 20 8b 73 10 8d 48 01 89 73 24 89 4b 20 83 fa 04 74 23 f3 90 44 8b 63 20 <41> 39 d4 74 f0 41 83 fc 02 75 c2 fa 66 0f 1f 44 00 00 eb c4 66 
Complete report #511314
Frame # Function Binary Source or offset Line
1
migration_cpu_stop
vmlinux 0xffffffff810c211a
Build id: 3.10.0-514.el7.x86_64
0x2a
2
cpu_stopper_thread
vmlinux 0xffffffff811163c6
Build id: 3.10.0-514.el7.x86_64
0x96
3
? __schedule
vmlinux 0xffffffff8168b070
Build id: 3.10.0-514.el7.x86_64
0x3b0
4
smpboot_thread_fn
vmlinux 0xffffffff810b90af
Build id: 3.10.0-514.el7.x86_64
0x12f
5
? lg_double_unlock
vmlinux 0xffffffff810b8f80
Build id: 3.10.0-514.el7.x86_64
0x90
6
kthread
vmlinux 0xffffffff810b052f
Build id: 3.10.0-514.el7.x86_64
0xcf
7
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
8
ret_from_fork
vmlinux 0xffffffff81696518
Build id: 3.10.0-514.el7.x86_64
0x58
9
? kthread_create_on_node
vmlinux 0xffffffff810b0460
Build id: 3.10.0-514.el7.x86_64
0x140
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 503s! [migration/1:12]
Modules linked in: tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun ipt_REJECT nf_reject_ipv4 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6_tables iptable_filter intel_powerclamp ppdev sg iosf_mbi parport_pc parport i2c_piix4 i2c_core pcspkr nfsd auth_rpcgss nfs_acl lockd grace binfmt_misc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi 8021q garp stp llc mrp crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel scsi_transport_iscsi aesni_intel ahci lrw libahci ata_piix gf128mul glue_helper ablk_helper cryptd serio_raw e1000 libata
fjes video sunrpc dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 12 Comm: migration/1 Tainted: G             L ------------   3.10.0-514.el7.x86_64 #1
Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
task: ffff880184d7de20 ti: ffff880184d98000 task.ti: ffff880184d98000
RIP: 0010:[<ffffffff810c211a>]  [<ffffffff810c211a>] migration_cpu_stop+0x2a/0x30
RSP: 0018:ffff880184d9bdc0  EFLAGS: 00000286
RAX: 0000000000000001 RBX: ffff88021fd16c40 RCX: 000000000000003c
RDX: 000000000000000f RSI: 00000000000a0008 RDI: ffff8801f9fb6624
RBP: ffff880184d9bdc0 R08: ffffffff816afea0 R09: 0000000000000001
R10: 00000000000044b7 R11: 0000000000000001 R12: ffff880184d9bd50
R13: ffffffff8104e17c R14: ffff880184d9bd30 R15: ffffffff81057bc3
FS:  0000000000000000(0000) GS:ffff88021fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000448c90 CR3: 00000000a78a7000 CR4: 00000000000406e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
ffff880184d9be90 ffffffff811163c6 ffff880184d9bfd8 ffff88021fc8ff10
0000000000000000 0000000000000000 ffff88021fc96c40 ffff88020d6c0000
0000000000000000 ffff88021fc96c40 ffff88020d6c0000 ffff8801ddfbbe00
Call Trace:
[<ffffffff811163c6>] cpu_stopper_thread+0x96/0x170
[<ffffffff8168b070>] ? __schedule+0x3b0/0x990
[<ffffffff810b90af>] smpboot_thread_fn+0x12f/0x180
[<ffffffff810b8f80>] ? lg_double_unlock+0x90/0x90
[<ffffffff810b052f>] kthread+0xcf/0xe0
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
[<ffffffff81696518>] ret_from_fork+0x58/0x90
[<ffffffff810b0460>] ? kthread_create_on_node+0x140/0x140
Code: 00 0f 1f 44 00 00 55 48 89 e5 fa 66 0f 1f 44 00 00 8b 57 08 48 8b 3f 65 8b 34 25 1c a0 00 00 e8 7d fe ff ff fb 66 0f 1f 44 00 00 <31> c0 5d c3 66 90 0f 1f 44 00 00 55 48 89 e5 41 57 49 89 f7 41