Info

Permalink
Function
smp_call_function_many
First occurrence
2021-11-13
Last occurrence
2026-07-05
State
NEW
Components
kernel
Quality
-57

Statistics

Operating system Count
CentOS 7 2
Architecture Count
x86_64 2
Related packages Count
kernel 2
 0:3.10.0-1160.el7 1
 0:3.10.0-1160.119.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

No reports in the last 20 weeks.

Report backtrace


Complete report #1417435
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa231e25a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffffa23d2910
Build id: 3.10.0-1160.119.1.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffffa231e3a9
Build id: 3.10.0-1160.119.1.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffffa23cf715
Build id: 3.10.0-1160.119.1.el7.x86_64
0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffffa23d46fb
Build id: 3.10.0-1160.119.1.el7.x86_64
0x8db
6
alloc_pages_current
vmlinux 0xffffffffa2425a98
Build id: 3.10.0-1160.119.1.el7.x86_64
0x98
7
kmalloc_order
vmlinux 0xffffffffa23f1868
Build id: 3.10.0-1160.119.1.el7.x86_64
0x18
8
kmalloc_order_trace
vmlinux 0xffffffffa2431836
Build id: 3.10.0-1160.119.1.el7.x86_64
0x26
9
__kmalloc
vmlinux 0xffffffffa24358c1
Build id: 3.10.0-1160.119.1.el7.x86_64
0x221
10
kmem_alloc
xfs 0xffffffffc03818c7
Build id: 3.10.0-1160.119.1.el7.x86_64
0x97
11
? avc_has_perm_flags
vmlinux 0xffffffffa251eedc
Build id: 3.10.0-1160.119.1.el7.x86_64
0xdc
12
kmem_alloc_large
xfs 0xffffffffc038197b
Build id: 3.10.0-1160.119.1.el7.x86_64
0x1b
13
xfs_get_acl
xfs 0xffffffffc039d232
Build id: 3.10.0-1160.119.1.el7.x86_64
0x162
14
xfs_generic_create
xfs 0xffffffffc0372523
Build id: 3.10.0-1160.119.1.el7.x86_64
0x1c3
15
xfs_vn_mknod
xfs 0xffffffffc0372634
Build id: 3.10.0-1160.119.1.el7.x86_64
0x14
16
xfs_vn_create
xfs 0xffffffffc0372673
Build id: 3.10.0-1160.119.1.el7.x86_64
0x13
17
vfs_create
vmlinux 0xffffffffa246968a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x15a
18
do_last
vmlinux 0xffffffffa246d8f5
Build id: 3.10.0-1160.119.1.el7.x86_64
0x11d5
19
? kmem_cache_alloc_trace
vmlinux 0xffffffffa2435f56
Build id: 3.10.0-1160.119.1.el7.x86_64
0x1d6
20
path_openat
vmlinux 0xffffffffa246db6d
Build id: 3.10.0-1160.119.1.el7.x86_64
0xcd
21
? do_read_fault.isra.63
vmlinux 0xffffffffa23fa8b9
Build id: 3.10.0-1160.119.1.el7.x86_64
0x139
22
do_filp_open
vmlinux 0xffffffffa246f52d
Build id: 3.10.0-1160.119.1.el7.x86_64
0x4d
23
? __alloc_fd
vmlinux 0xffffffffa247dd97
Build id: 3.10.0-1160.119.1.el7.x86_64
0x47
24
do_sys_open
vmlinux 0xffffffffa245abf4
Build id: 3.10.0-1160.119.1.el7.x86_64
0x124
25
SyS_open
vmlinux 0xffffffffa245ad0e
Build id: 3.10.0-1160.119.1.el7.x86_64
0x1e
26
system_call_fastpath
vmlinux 0xffffffffa29c539a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [chronyd:874]
Modules linked in: veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter binfmt_misc overlay(T) vmw_vsock_vmci_transport vsock sunrpc nfit libnvdimm coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd pcspkr sg vmw_balloon joydev parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix mptspi scsi_transport_spi
mptscsih libata crct10dif_pclmul mptbase crct10dif_common serio_raw vmxnet3 crc32c_intel drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 874 Comm: chronyd Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff99d575981080 ti: ffff99d3f9490000 task.ti: ffff99d3f9490000
RIP: 0010:[<ffffffffa231e25a>]  [<ffffffffa231e25a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff99d3f9493888  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff99d57fc20650
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff99d3f94938c0 R08: ffff99d4baefe000 R09: ffffffffa259c7f9
R10: ffff99d57fd9f160 R11: ffffe5e140756600 R12: 0000000000000292
R13: ffff99d3f9493838 R14: ffffffffa233af55 R15: 00000000fde6ef4a
FS:  00007f18edd0f740(0000) GS:ffff99d57fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f18ecd55fa8 CR3: 00000000b948e000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffa23d2910>] ? drain_pages+0xb0/0xb0
[<ffffffffa231e3a9>] on_each_cpu_mask+0x29/0x70
[<ffffffffa23cf715>] drain_all_pages+0xb5/0xd0
[<ffffffffa23d46fb>] __alloc_pages_nodemask+0x8db/0xbf0
[<ffffffffa2425a98>] alloc_pages_current+0x98/0x110
[<ffffffffa23f1868>] kmalloc_order+0x18/0x50
[<ffffffffa2431836>] kmalloc_order_trace+0x26/0xa0
[<ffffffffa24358c1>] __kmalloc+0x221/0x240
[<ffffffffc03818c7>] kmem_alloc+0x97/0x130 [xfs]
[<ffffffffa251eedc>] ? avc_has_perm_flags+0xdc/0x1c0
[<ffffffffc038197b>] kmem_alloc_large+0x1b/0x130 [xfs]
[<ffffffffc039d232>] xfs_get_acl+0x162/0x2a0 [xfs]
[<ffffffffc0372523>] xfs_generic_create+0x1c3/0x2a0 [xfs]
[<ffffffffc0372634>] xfs_vn_mknod+0x14/0x20 [xfs]
[<ffffffffc0372673>] xfs_vn_create+0x13/0x20 [xfs]
[<ffffffffa246968a>] vfs_create+0x15a/0x1e0
[<ffffffffa246d8f5>] do_last+0x11d5/0x1380
[<ffffffffa2435f56>] ? kmem_cache_alloc_trace+0x1d6/0x200
[<ffffffffa246db6d>] path_openat+0xcd/0x5a0
[<ffffffffa23fa8b9>] ? do_read_fault.isra.63+0x139/0x1b0
[<ffffffffa246f52d>] do_filp_open+0x4d/0xc0
[<ffffffffa247dd97>] ? __alloc_fd+0x47/0x180
[<ffffffffa245abf4>] do_sys_open+0x124/0x220
[<ffffffffa245ad0e>] SyS_open+0x1e/0x30
[<ffffffffa29c539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ae 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 19 f5 a2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 7d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #288838
Frame # Function Binary Source or offset Line
1
clear_page_c
vmlinux 0xffffffffb5d94727
Build id: 3.10.0-1160.el7.x86_64
0x7
2
? get_page_from_freelist
vmlinux 0xffffffffb5bc81ef
Build id: 3.10.0-1160.el7.x86_64
0x36f
3
? deadline_add_request
vmlinux 0xffffffffb5d77f8e
Build id: 3.10.0-1160.el7.x86_64
0x6e
4
? ___slab_alloc
vmlinux 0xffffffffb5c27439
Build id: 3.10.0-1160.el7.x86_64
0x229
5
__alloc_pages_nodemask
vmlinux 0xffffffffb5bc8aa6
Build id: 3.10.0-1160.el7.x86_64
0x166
6
? avc_alloc_node
vmlinux 0xffffffffb617ff76
Build id: 3.10.0-1160.el7.x86_64
0x24
7
alloc_pages_current
vmlinux 0xffffffffb5c189d8
Build id: 3.10.0-1160.el7.x86_64
0x98
8
kmalloc_order
vmlinux 0xffffffffb5be5678
Build id: 3.10.0-1160.el7.x86_64
0x18
9
kmalloc_order_trace
vmlinux 0xffffffffb5c243a6
Build id: 3.10.0-1160.el7.x86_64
0x26
10
__kmalloc
vmlinux 0xffffffffb5c28331
Build id: 3.10.0-1160.el7.x86_64
0x211
11
kmem_alloc
xfs 0xffffffffc03a6a67
Build id: 3.10.0-1160.el7.x86_64
0x97
12
? avc_has_perm_flags
vmlinux 0xffffffffb5d0b63c
Build id: 3.10.0-1160.el7.x86_64
0xdc
13
kmem_alloc_large
xfs 0xffffffffc03a6b1b
Build id: 3.10.0-1160.el7.x86_64
0x1b
14
xfs_get_acl
xfs 0xffffffffc03c1a12
Build id: 3.10.0-1160.el7.x86_64
0x162
15
xfs_generic_create
xfs 0xffffffffc0397aeb
Build id: 3.10.0-1160.el7.x86_64
0x1bb
16
xfs_vn_mknod
xfs 0xffffffffc0397c04
Build id: 3.10.0-1160.el7.x86_64
0x14
17
xfs_vn_create
xfs 0xffffffffc0397c43
Build id: 3.10.0-1160.el7.x86_64
0x13
18
vfs_create
vmlinux 0xffffffffb5c5acea
Build id: 3.10.0-1160.el7.x86_64
0x13a
19
do_last
vmlinux 0xffffffffb5c5e9a8
Build id: 3.10.0-1160.el7.x86_64
0x1168
20
? kmem_cache_alloc_trace
vmlinux 0xffffffffb5c289c6
Build id: 3.10.0-1160.el7.x86_64
0x1d6
21
path_openat
vmlinux 0xffffffffb5c5ec4d
Build id: 3.10.0-1160.el7.x86_64
0xcd
22
do_filp_open
vmlinux 0xffffffffb5c60e9d
Build id: 3.10.0-1160.el7.x86_64
0x4d
23
? __alloc_fd
vmlinux 0xffffffffb5c6ef97
Build id: 3.10.0-1160.el7.x86_64
0x47
24
do_sys_open
vmlinux 0xffffffffb5c4c9e4
Build id: 3.10.0-1160.el7.x86_64
0x124
25
SyS_open
vmlinux 0xffffffffb5c4cafe
Build id: 3.10.0-1160.el7.x86_64
0x1e
26
system_call_fastpath
vmlinux 0xffffffffb6193f92
Build id: 3.10.0-1160.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 36s! [cupsd:1095]
Modules linked in: rfcomm xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock bnep sunrpc snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec kvm_amd ac97_bus snd_seq kvm snd_seq_device snd_pcm irqbypass crc32_pclmul ppdev ghash_clmulni_intel snd_timer btusb btrtl btbcm btintel bluetooth aesni_intel snd lrw gf128mul glue_helper ablk_helper cryptd rfkill vmw_balloon pcspkr joydev soundcore sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic
vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix crct10dif_pclmul crct10dif_common crc32c_intel drm nfit libnvdimm libata mptspi serio_raw scsi_transport_spi mptscsih e1000 mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 0 PID: 1095 Comm: cupsd Not tainted 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/29/2019
task: ffff9a42f7700000 ti: ffff9a42f7398000 task.ti: ffff9a42f7398000
RIP: 0010:[<ffffffffb5d94727>]  [<ffffffffb5d94727>] clear_page_c+0x7/0x10
RSP: 0018:ffff9a42f739b898  EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffffffffb5da5bbd RCX: 0000000000000200
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffff9a42d0638000
RBP: ffff9a42f739b9a8 R08: ffffffffb648e477 R09: fffff44140419000
R10: ffff9a42fffd7f40 R11: 0000000000000000 R12: ffff9a42f739bfd8
R13: ffff9a42f739bfd8 R14: ffffffffb6186c8f R15: ffff9a42f739b870
FS:  00007faf91caa8c0(0000) GS:ffff9a42fc600000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffa0c4807d0 CR3: 0000000033438000 CR4: 00000000003407f0
Call Trace:
[<ffffffffb5bc81ef>] ? get_page_from_freelist+0x36f/0xac0
[<ffffffffb5d77f8e>] ? deadline_add_request+0x6e/0x90
[<ffffffffb5c27439>] ? ___slab_alloc+0x229/0x520
[<ffffffffb5bc8aa6>] __alloc_pages_nodemask+0x166/0x450
[<ffffffffb617ff76>] ? avc_alloc_node+0x24/0x123
[<ffffffffb5c189d8>] alloc_pages_current+0x98/0x110
[<ffffffffb5be5678>] kmalloc_order+0x18/0x40
[<ffffffffb5c243a6>] kmalloc_order_trace+0x26/0xa0
[<ffffffffb5c28331>] __kmalloc+0x211/0x230
[<ffffffffc03a6a67>] kmem_alloc+0x97/0x130 [xfs]
[<ffffffffb5d0b63c>] ? avc_has_perm_flags+0xdc/0x1c0
[<ffffffffc03a6b1b>] kmem_alloc_large+0x1b/0x120 [xfs]
[<ffffffffc03c1a12>] xfs_get_acl+0x162/0x2a0 [xfs]
[<ffffffffc0397aeb>] xfs_generic_create+0x1bb/0x2a0 [xfs]
[<ffffffffc0397c04>] xfs_vn_mknod+0x14/0x20 [xfs]
[<ffffffffc0397c43>] xfs_vn_create+0x13/0x20 [xfs]
[<ffffffffb5c5acea>] vfs_create+0x13a/0x1b0
[<ffffffffb5c5e9a8>] do_last+0x1168/0x1340
[<ffffffffb5c289c6>] ? kmem_cache_alloc_trace+0x1d6/0x200
[<ffffffffb5c5ec4d>] path_openat+0xcd/0x5a0
[<ffffffffb5c60e9d>] do_filp_open+0x4d/0xb0
[<ffffffffb5c6ef97>] ? __alloc_fd+0x47/0x170
[<ffffffffb5c4c9e4>] do_sys_open+0x124/0x220
[<ffffffffb5c4cafe>] SyS_open+0x1e/0x20
[<ffffffffb6193f92>] system_call_fastpath+0x25/0x2a
Code: 4c 29 ea 39 da 89 d1 7f c4 85 d2 7f 9d 89 d0 eb bc 0f 1f 00 e8 6b 68 d0 ff 90 90 90 90 90 90 90 90 90 90 90 b9 00 02 00 00 31 c0 <f3> 48 ab c3 0f 1f 44 00 00 b9 00 10 00 00 31 c0 f3 aa c3 66 0f