Info

Permalink
Function
smp_call_function_many
First occurrence
2023-03-27
Last occurrence
2026-06-17
State
NEW
Components
kernel
Quality
-31

Statistics

Operating system Count
CentOS 7 7
Architecture Count
x86_64 7
Related packages Count
kernel 7
 0:3.10.0-1160.el7 3
 0:3.10.0-1160.88.1.el7 2
 0:3.10.0-1160.119.1.el7.tuxcare.els18 2

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1415821
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff86f16c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffff86fc74a0
Build id: 3.10.0-1160.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff86f16d59
Build id: 3.10.0-1160.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff86fc4385
Build id: 3.10.0-1160.el7.x86_64
0xb5
5
__alloc_pages_slowpath
vmlinux 0xffffffff8757c6b2
Build id: 3.10.0-1160.el7.x86_64
0x510
6
__alloc_pages_nodemask
vmlinux 0xffffffff86fc8d76
Build id: 3.10.0-1160.el7.x86_64
0x436
7
alloc_pages_current
vmlinux 0xffffffff870189d8
Build id: 3.10.0-1160.el7.x86_64
0x98
8
alloc_skb_with_frags
vmlinux 0xffffffff8743e44d
Build id: 3.10.0-1160.el7.x86_64
0xcd
9
sock_alloc_send_pskb
vmlinux 0xffffffff87437fb9
Build id: 3.10.0-1160.el7.x86_64
0x1b9
10
unix_stream_sendmsg
vmlinux 0xffffffff8751409d
Build id: 3.10.0-1160.el7.x86_64
0x2ad
11
sock_sendmsg
vmlinux 0xffffffff874343a6
Build id: 3.10.0-1160.el7.x86_64
0xb6
12
? wake_up_q
vmlinux 0xffffffff86edaaab
Build id: 3.10.0-1160.el7.x86_64
0x5b
13
SYSC_sendto
vmlinux 0xffffffff87434ad1
Build id: 3.10.0-1160.el7.x86_64
0x121
14
SyS_sendto
vmlinux 0xffffffff874365ee
Build id: 3.10.0-1160.el7.x86_64
0xe
15
system_call_fastpath
vmlinux 0xffffffff87593f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [connection:16724]
Modules linked in: tcp_lp xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink cmac arc4 md4 nls_utf8 cifs ccm dns_resolver ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr joydev vmw_balloon sg i2c_piix4 parport_pc parport vmw_vmci ip_tables xfs libcrc32c
sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci drm ata_piix libahci nfit crct10dif_pclmul crct10dif_common crc32c_intel libnvdimm libata serio_raw vmw_pvscsi vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 4 PID: 16724 Comm: connection Kdump: loaded Tainted: G             L ------------   3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/28/2017
task: ffff89e409b8e300 ti: ffff89e3cfb88000 task.ti: ffff89e3cfb88000
RIP: 0010:[<ffffffff86f16c0a>]  [<ffffffff86f16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff89e3cfb8b8c0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 0008000086fc74a0 RCX: ffff8a023f220678
RDX: 0000000000000000 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff89e3cfb8b8f8 R08: ffff89e3c5b57400 R09: ffffffff87185f99
R10: ffff8a023f31f160 R11: fffff33d46cff800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff89e3cfb8b870 R15: 0000000001320122
FS:  00007fafe4188700(0000) GS:ffff8a023f300000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8a13267978 CR3: 0000001be1f20000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff86fc74a0>] ? drain_pages+0xb0/0xb0
[<ffffffff86f16d59>] on_each_cpu_mask+0x29/0x70
[<ffffffff86fc4385>] drain_all_pages+0xb5/0xc0
[<ffffffff8757c6b2>] __alloc_pages_slowpath+0x510/0x729
[<ffffffff86fc8d76>] __alloc_pages_nodemask+0x436/0x450
[<ffffffff870189d8>] alloc_pages_current+0x98/0x110
[<ffffffff8743e44d>] alloc_skb_with_frags+0xcd/0x1d0
[<ffffffff87437fb9>] sock_alloc_send_pskb+0x1b9/0x260
[<ffffffff8751409d>] unix_stream_sendmsg+0x2ad/0x430
[<ffffffff874343a6>] sock_sendmsg+0xb6/0xf0
[<ffffffff86edaaab>] ? wake_up_q+0x5b/0x80
[<ffffffff87434ad1>] SYSC_sendto+0x121/0x1c0
[<ffffffff874365ee>] SyS_sendto+0xe/0x10
[<ffffffff87593f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 b5 87 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1182402
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa4b1e2be
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0x20e
2
? drain_pages
vmlinux 0xffffffffa4bd2940
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffffa4b1e409
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffffa4bcf745
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffffa4bd472b
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0x8db
6
alloc_pages_current
vmlinux 0xffffffffa4c25af8
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0x98
7
alloc_skb_with_frags
vmlinux 0xffffffffa5068b3d
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0xcd
8
sock_alloc_send_pskb
vmlinux 0xffffffffa506236d
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0x1bd
9
unix_stream_sendmsg
vmlinux 0xffffffffa5142cbd
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0x2ad
10
sock_sendmsg
vmlinux 0xffffffffa505e4f6
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0xb6
11
? wp_page_copy.isra.73
vmlinux 0xffffffffa4bfb754
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0x374
12
? __mem_cgroup_commit_charge
vmlinux 0xffffffffa4c4bfc2
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0xe2
13
SYSC_sendto
vmlinux 0xffffffffa505ec51
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0x121
14
? __do_page_fault
vmlinux 0xffffffffa51bf7f8
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0x238
15
SyS_sendto
vmlinux 0xffffffffa50607ee
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0xe
16
system_call_fastpath
vmlinux 0xffffffffa51c539a
Build id: 3.10.0-1160.119.1.el7.tuxcare.els18.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [mysqld:20794]
Modules linked in: ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 ip6table_filter nf_log_ipv4 nf_log_common nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack libcrc32c ip6_tables tcp_diag inet_diag iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter loop sunrpc dm_mirror dm_region_hash dm_log dm_mod cirrus ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev drm_panel_orientation_quirks aesni_intel parport_pc virtio_balloon lrw joydev gf128mul parport glue_helper pcspkr ablk_helper i2c_piix4 cryptd ip_tables
ext4 mbcache jbd2 ata_generic pata_acpi virtio_net net_failover ata_piix virtio_blk failover crct10dif_pclmul crct10dif_common virtio_pci libata virtio_ring serio_raw crc32c_intel floppy virtio
CPU: 0 PID: 20794 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1160.119.1.el7.tuxcare.els18.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
task: ffff98d9bbddb180 ti: ffff98da01a10000 task.ti: ffff98da01a10000
RIP: 0010:[<ffffffffa4b1e2be>]  [<ffffffffa4b1e2be>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff98da01a13928  EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00000000000000fc RCX: ffff98db1fddfc10
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 23s! [mysqld:20795]
Modules linked in: ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 ip6table_filter nf_log_ipv4 nf_log_common nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack libcrc32c ip6_tables tcp_diag inet_diag iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter loop sunrpc dm_mirror dm_region_hash dm_log dm_mod cirrus ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev drm_panel_orientation_quirks aesni_intel parport_pc virtio_balloon lrw joydev gf128mul parport glue_helper pcspkr ablk_helper i2c_piix4 cryptd ip_tables
ext4 mbcache jbd2 ata_generic pata_acpi virtio_net net_failover ata_piix virtio_blk failover crct10dif_pclmul crct10dif_common virtio_pci libata virtio_ring serio_raw crc32c_intel floppy virtio
CPU: 6 PID: 20795 Comm: mysqld Kdump: loaded Tainted: G             L ------------   3.10.0-1160.119.1.el7.tuxcare.els18.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2007
task: ffff98da0215b180 ti: ffff98da01a70000 task.ti: ffff98da01a70000
RIP: 0010:[<ffffffffa4b1e2be>]  [<ffffffffa4b1e2be>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff98da01a73928  EFLAGS: 00000202
RAX: 0000000000000007 RBX: 00000000000000fc RCX: ffff98db1fde06c8
RDX: 0000000000000007 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff98da01a73960 R08: ffff98da7aeeac00 R09: ffffffffa4d9cb29
R10: ffff98db1fd9f160 R11: ffffe08248631200 R12: 0000000000000292
R13: ffff98da01a738d8 R14: 0000000001320122 R15: 00000000c68af90a
FS:  00007f5eb81d9700(0000) GS:ffff98db1fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5e4000d6a0 CR3: 000000021a748000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa4bd2940>] ? drain_pages+0xb0/0xb0
[<ffffffffa4b1e409>] on_each_cpu_mask+0x29/0x70
[<ffffffffa4bcf745>] drain_all_pages+0xb5/0xd0
[<ffffffffa4bd472b>] __alloc_pages_nodemask+0x8db/0xbf0
[<ffffffffa4c25af8>] alloc_pages_current+0x98/0x110
[<ffffffffa5068b3d>] alloc_skb_with_frags+0xcd/0x1e0
[<ffffffffa506236d>] sock_alloc_send_pskb+0x1bd/0x260
[<ffffffffa5142cbd>] unix_stream_sendmsg+0x2ad/0x430
[<ffffffffa505e4f6>] sock_sendmsg+0xb6/0x100
[<ffffffffa4bfb754>] ? wp_page_copy.isra.73+0x374/0x5d0
[<ffffffffa4c4bfc2>] ? __mem_cgroup_commit_charge+0xe2/0x2f0
[<ffffffffa505ec51>] SYSC_sendto+0x121/0x1d0
[<ffffffffa51bf7f8>] ? __do_page_fault+0x238/0x510
[<ffffffffa50607ee>] SyS_sendto+0xe/0x20
[<ffffffffa51c539a>] system_call_fastpath+0x25/0x2a
Code: 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 80 19 75 a5 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 dd 17 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #657299
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff81b1e1ca
Build id: 3.10.0-1160.88.1.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffff81bd2850
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff81b1e319
Build id: 3.10.0-1160.88.1.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff81bcf655
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffff81bd463b
Build id: 3.10.0-1160.88.1.el7.x86_64
0x8db
6
alloc_pages_current
vmlinux 0xffffffff81c259d8
Build id: 3.10.0-1160.88.1.el7.x86_64
0x98
7
alloc_skb_with_frags
vmlinux 0xffffffff82068b3d
Build id: 3.10.0-1160.88.1.el7.x86_64
0xcd
8
sock_alloc_send_pskb
vmlinux 0xffffffff8206236d
Build id: 3.10.0-1160.88.1.el7.x86_64
0x1bd
9
? put_prev_entity
vmlinux 0xffffffff81aea151
Build id: 3.10.0-1160.88.1.el7.x86_64
0x31
10
unix_stream_sendmsg
vmlinux 0xffffffff82142c3d
Build id: 3.10.0-1160.88.1.el7.x86_64
0x2ad
11
sock_sendmsg
vmlinux 0xffffffff8205e4f6
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb6
12
SYSC_sendto
vmlinux 0xffffffff8205ec51
Build id: 3.10.0-1160.88.1.el7.x86_64
0x121
13
SyS_sendto
vmlinux 0xffffffff820607ee
Build id: 3.10.0-1160.88.1.el7.x86_64
0xe
14
system_call_fastpath
vmlinux 0xffffffff821c539a
Build id: 3.10.0-1160.88.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#17 stuck for 22s! [Compositor:102449]
Modules linked in: nfsv3 xt_nat veth ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink xt_addrtype mfe_fileaccess_100714877(OE) rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache cbsensor_2_5_1230541(OE) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables mfe_aac_100714877(OE) ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ppdev joydev pcspkr sg vmw_balloon parport_pc parport vmw_vmci i2c_piix4 nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables
xfs libcrc32c sd_mod crc_t10dif crct10dif_generic crct10dif_common sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mptspi ata_piix drm scsi_transport_spi nfit mptscsih crc32c_intel libata libnvdimm mptbase serio_raw vmxnet3 drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse br_netfilter bridge stp llc overlay(T)
CPU: 17 PID: 102449 Comm: Compositor Tainted: G           OEL ------------ T 3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 05/28/2020
task: ffff9153c2fe8000 ti: ffff915889050000 task.ti: ffff915889050000
RIP: 0010:[<ffffffff81b1e1ca>]  [<ffffffff81b1e1ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff915889053928  EFLAGS: 00000202
RAX: 000000000000000c RBX: 00000000000000fc RCX: ffff915e13920858
RDX: 000000000000000c RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff915889053960 R08: ffff9154e469f000 R09: ffffffff81d9c669
R10: ffff915e13a5f160 R11: fffffd20c47fb800 R12: 0000000000000292
R13: ffff9158890538d8 R14: ffffffff81b3aec5 R15: 0000000000436f2f
FS:  00007f2eed7a8700(0000) GS:ffff915e13a40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000005540ad91000 CR3: 00000009bf184000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81bd2850>] ? drain_pages+0xb0/0xb0
[<ffffffff81b1e319>] on_each_cpu_mask+0x29/0x70
[<ffffffff81bcf655>] drain_all_pages+0xb5/0xd0
[<ffffffff81bd463b>] __alloc_pages_nodemask+0x8db/0xbf0
[<ffffffff81c259d8>] alloc_pages_current+0x98/0x110
[<ffffffff82068b3d>] alloc_skb_with_frags+0xcd/0x1e0
[<ffffffff8206236d>] sock_alloc_send_pskb+0x1bd/0x260
[<ffffffff81aea151>] ? put_prev_entity+0x31/0x400
[<ffffffff82142c3d>] unix_stream_sendmsg+0x2ad/0x430
[<ffffffff8205e4f6>] sock_sendmsg+0xb6/0x100
[<ffffffff8205ec51>] SYSC_sendto+0x121/0x1d0
[<ffffffff820607ee>] SyS_sendto+0xe/0x20
[<ffffffff821c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 7e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 75 82 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89