Info

Permalink
Function
smp_call_function_many
First occurrence
2021-09-25
Last occurrence
2024-02-12
State
NEW
Components
kernel
Quality
-26

Statistics

Operating system Count
CentOS 7 51
Architecture Count
x86_64 51
Related packages Count
kernel 51
 0:3.10.0-693.el7 36
 0:3.10.0-862.el7 15

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #534612
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a6
Build id: 3.10.0-693.el7.x86_64
0x206
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
0xf4
7
? lockref_put_or_lock
vmlinux 0xffffffff81331f48
Build id: 3.10.0-693.el7.x86_64
0x48
8
? __vma_rb_erase
vmlinux 0xffffffff811b5a01
Build id: 3.10.0-693.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
0x292
10
vm_munmap
vmlinux 0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#14 stuck for 22s! [java:116136]
Modules linked in: uio_ivshmem(OE) gsch(OE) redirfs(OE) binfmt_misc nls_utf8 isofs xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter uio ppdev parport_pc sg i2c_piix4 pcspkr parport virtio_balloon joydev nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net virtio_console virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci floppy virtio_ring i2c_core serio_raw virtio dm_mirror dm_region_hash dm_log dm_mod [last unloaded: redirfs]
CPU: 14 PID: 116136 Comm: java Tainted: G           OEL ------------   3.10.0-693.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff880813ecbf40 ti: ffff880801ee8000 task.ti: ffff880801ee8000
RIP: 0010:[<ffffffff810f99a6>]  [<ffffffff810f99a6>] smp_call_function_many+0x206/0x260
RSP: 0018:ffff880801eebcf0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800008106e4a0 RCX: ffff88081ce1af70
RDX: 0000000000000000 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff880801eebd28 R08: ffff880174b8ec00 R09: ffffffff813227d9
R10: ffff88081d199c80 R11: ffffea00204df800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff880801eebca0 R15: 00007f0c340df000
FS:  00007f0c87fff700(0000) GS:ffff88081d180000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0c340df000 CR3: 000000080aae5000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000113ecbf40 ffff880819321c48 ffff880819321900 0000000000000000
ffffffffffffffff ffff880819321c48 ffff880034957290 ffff880801eebd78
ffffffff8106e668 ffff880819321900 0000000000000000 ffffffffffffffff
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff81331f48>] ? lockref_put_or_lock+0x48/0x80
[<ffffffff811b5a01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #254544
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
native_flush_tlb_others
vmlinux 0xffffffff8106e668
Build id: 3.10.0-693.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffff8106e738
Build id: 3.10.0-693.el7.x86_64
0x68
4
tlb_flush_mmu.part.63
vmlinux 0xffffffff811ad053
Build id: 3.10.0-693.el7.x86_64
0x33
5
tlb_finish_mmu
vmlinux 0xffffffff811ae375
Build id: 3.10.0-693.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffff811b5444
Build id: 3.10.0-693.el7.x86_64
0xf4
7
? lockref_put_or_lock
vmlinux 0xffffffff81331f48
Build id: 3.10.0-693.el7.x86_64
0x48
8
? __vma_rb_erase
vmlinux 0xffffffff811b5a01
Build id: 3.10.0-693.el7.x86_64
0x121
9
do_munmap
vmlinux 0xffffffff811b7a02
Build id: 3.10.0-693.el7.x86_64
0x292
10
vm_munmap
vmlinux 0xffffffff811b7c45
Build id: 3.10.0-693.el7.x86_64
0x65
11
SyS_munmap
vmlinux 0xffffffff811b8d52
Build id: 3.10.0-693.el7.x86_64
0x22
12
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 24s! [ftdc:1780]
Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock snd_seq_midi snd_seq_midi_event sb_edac edac_core iosf_mbi crc32_pclmul snd_ens1371 snd_rawmidi ghash_clmulni_intel snd_ac97_codec ac97_bus aesni_intel lrw snd_seq gf128mul glue_helper ablk_helper snd_seq_device cryptd snd_pcm ppdev nfit vmw_balloon sg snd_timer pcspkr snd joydev vmw_vmci soundcore libnvdimm i2c_piix4
shpchp parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod ata_generic crc_t10dif crct10dif_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi crct10dif_pclmul scsi_transport_spi crct10dif_common mptscsih crc32c_intel libata e1000 mptbase serio_raw i2c_core dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 1780 Comm: ftdc Tainted: G             L ------------   3.10.0-693.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/19/2017
task: ffff880232b95ee0 ti: ffff880221770000 task.ti: ffff880221770000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff880221773cf0  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000800008106e4a0 RCX: ffff88023569af98
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff880221773d28 R08: ffff88017cf35800 R09: ffffffff813227d9
R10: ffff880235719c80 R11: ffffea0008c85400 R12: 0000000000000286
R13: 0000000000000018 R14: ffff880221773c60 R15: 0000000000000286
FS:  00007f9ac5948700(0000) GS:ffff880235700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9ad1914000 CR3: 0000000035e88000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000132b95ee0 ffff880232b3a288 ffff880232b39f40 00007f9ad1914000
00007f9ad1915000 ffff880232b3a288 ffff880232247a28 ffff880221773d78
ffffffff8106e668 ffff880232b39f40 00007f9ad1914000 00007f9ad1915000
Call Trace:
[<ffffffff8106e668>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff8106e738>] flush_tlb_mm_range+0x68/0x140
[<ffffffff811ad053>] tlb_flush_mmu.part.63+0x33/0xc0
[<ffffffff811ae375>] tlb_finish_mmu+0x55/0x60
[<ffffffff811b5444>] unmap_region+0xf4/0x140
[<ffffffff81331f48>] ? lockref_put_or_lock+0x48/0x80
[<ffffffff811b5a01>] ? __vma_rb_erase+0x121/0x220
[<ffffffff811b7a02>] do_munmap+0x292/0x470
[<ffffffff811b7c45>] vm_munmap+0x65/0xb0
[<ffffffff811b8d52>] SyS_munmap+0x22/0x30
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #409300
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb2d07e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
native_flush_tlb_others
vmlinux 0xffffffffb2c77018
Build id: 3.10.0-862.el7.x86_64
0xb8
3
flush_tlb_mm_range
vmlinux 0xffffffffb2c77088
Build id: 3.10.0-862.el7.x86_64
0x68
4
tlb_flush_mmu.part.75
vmlinux 0xffffffffb2dc0497
Build id: 3.10.0-862.el7.x86_64
0x37
5
tlb_finish_mmu
vmlinux 0xffffffffb2dc1c75
Build id: 3.10.0-862.el7.x86_64
0x55
6
unmap_region
vmlinux 0xffffffffb2dca624
Build id: 3.10.0-862.el7.x86_64
0xf4
7
? list_del
vmlinux 0xffffffffb2f683ed
Build id: 3.10.0-862.el7.x86_64
0xd
8
? remove_wait_queue
vmlinux 0xffffffffb2cbba41
Build id: 3.10.0-862.el7.x86_64
0x31
9
? __vma_rb_erase
vmlinux 0xffffffffb2dcabe1
Build id: 3.10.0-862.el7.x86_64
0x121
10
do_munmap
vmlinux 0xffffffffb2dccbf5
Build id: 3.10.0-862.el7.x86_64
0x295
11
vm_munmap
vmlinux 0xffffffffb2dcce35
Build id: 3.10.0-862.el7.x86_64
0x65
12
SyS_munmap
vmlinux 0xffffffffb2dce052
Build id: 3.10.0-862.el7.x86_64
0x22
13
system_call_fastpath
vmlinux 0xffffffffb331f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [python2.7:55644]
Modules linked in: xt_iprange ip6table_filter ip6_tables iptable_filter binfmt_misc joydev bochs_drm ttm sb_edac drm_kms_helper iosf_mbi ppdev virtio_rng sg syscopyarea kvm_intel kvm sysfillrect sysimgblt irqbypass fb_sys_fops crc32_pclmul ghash_clmulni_intel i2c_piix4 drm aesni_intel lrw parport_pc i2c_core gf128mul parport glue_helper ablk_helper cryptd pcspkr ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_net virtio_console virtio_scsi crct10dif_pclmul crct10dif_common crc32c_intel ata_piix libata serio_raw virtio_pci virtio_ring virtio floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 55644 Comm: python2.7 Kdump: loaded Not tainted 3.10.0-862.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff99729e379fa0 ti: ffff99736a7cc000 task.ti: ffff99736a7cc000
RIP: 0010:[<ffffffffb2d07e3a>]  [<ffffffffb2d07e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff99736a7cfcb8  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 0000000000019680 RCX: ffff99737f69cf28
RDX: 0000000000000002 RSI: 00000000000000ff RDI: 0000000000000000
RBP: ffff99736a7cfcf0 R08: ffff9965ea6a2c00 R09: ffffffffb2f4a3d9
R10: ffff99737f71bb00 R11: fffffc19b895fe00 R12: ffff99736a7cfc68
R13: 00007f73a22de000 R14: 0000000000000001 R15: fffffc19b72a6d80
FS:  00007f738effd700(0000) GS:ffff99737f700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffb2dfe4080 CR3: 0000000e63366000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb2c77018>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb2c77088>] flush_tlb_mm_range+0x68/0x140
[<ffffffffb2dc0497>] tlb_flush_mmu.part.75+0x37/0xe0
[<ffffffffb2dc1c75>] tlb_finish_mmu+0x55/0x60
[<ffffffffb2dca624>] unmap_region+0xf4/0x140
[<ffffffffb2f683ed>] ? list_del+0xd/0x30
[<ffffffffb2cbba41>] ? remove_wait_queue+0x31/0x40
[<ffffffffb2dcabe1>] ? __vma_rb_erase+0x121/0x220
[<ffffffffb2dccbf5>] do_munmap+0x295/0x470
[<ffffffffb2dcce35>] vm_munmap+0x65/0xb0
[<ffffffffb2dce052>] SyS_munmap+0x22/0x30
[<ffffffffb331f7d5>] system_call_fastpath+0x1c/0x21
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 93 b3 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89