ABRT Analytics - Problem #829389 - kernel in smp_call_function

Info

Permalink
Function: smp_call_function_many
First occurrence: 2022-01-15
Last occurrence: 2022-05-08
State: NEW
Components: kernel
Quality: -21

Statistics

Operating system	Count
CentOS 7	10

Architecture	Count
x86_64	10


Related packages	Count
kernel	10
0:3.10.0-1160.119.1.el7	3
0:3.10.0-1160.6.1.el7	1
0:3.10.0-957.el7	1
0:3.10.0-1160.62.1.el7	1
0:3.10.0-1160.81.1.el7	1
0:3.10.0-1160.88.1.el7	1
0:3.10.0-862.6.3.el7	1
0:3.10.0-862.el7	1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace

Complete report #1422457

Frame #	Function	Binary	Source or offset	Line
1	smp_call_function_many	vmlinux	0xffffffff91507e3a Build id: 3.10.0-862.el7.x86_64	0x20a
2	? drain_pages	vmlinux	0xffffffff9159cb10 Build id: 3.10.0-862.el7.x86_64	0xb0
3	on_each_cpu_mask	vmlinux	0xffffffff91507f89 Build id: 3.10.0-862.el7.x86_64	0x29
4	drain_all_pages	vmlinux	0xffffffff915999f5 Build id: 3.10.0-862.el7.x86_64	0xb5
5	__alloc_pages_nodemask	vmlinux	0xffffffff9159e860 Build id: 3.10.0-862.el7.x86_64	0x8a0
6	copy_process	vmlinux	0xffffffff9148ed4d Build id: 3.10.0-862.el7.x86_64	0x1dd
7	do_fork	vmlinux	0xffffffff91490761 Build id: 3.10.0-862.el7.x86_64	0x91
8	? security_file_fcntl	vmlinux	0xffffffff916d0b0c Build id: 3.10.0-862.el7.x86_64	0x1c
9	? SyS_fcntl	vmlinux	0xffffffff9162e69f Build id: 3.10.0-862.el7.x86_64	0xef
10	SyS_clone	vmlinux	0xffffffff91490a76 Build id: 3.10.0-862.el7.x86_64	0x16
11	stub_clone	vmlinux	0xffffffff91b1fb94 Build id: 3.10.0-862.el7.x86_64	0x44
12	? system_call_fastpath	vmlinux	0xffffffff91b1f7d5 Build id: 3.10.0-862.el7.x86_64	0x1c

NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [systemd:1]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg nfit libnvdimm vmw_vmci i2c_piix4 parport_pc shpchp parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ahci libahci drm ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3
i2c_core vmw_pvscsi dm_mirror dm_region_hash dm_log dm_mod
CPU: 4 PID: 1 Comm: systemd Kdump: loaded Tainted: G        W    L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9161e94e0000 ti: ffff9161e947c000 task.ti: ffff9161e947c000
RIP: 0010:[<ffffffff91507e3a>]  [<ffffffff91507e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9161e947fc58  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9170bfc1cf28
RDX: 0000000000000000 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff9161e947fc90 R08: ffff9161e9517800 R09: ffffffff9174a3d9
R10: ffff9170bfd1bb00 R11: ffffd6e000f4e800 R12: 0000000000000292
R13: ffff9161e947fc08 R14: 0000000001320122 R15: 00000000b2cba826
FS:  00007f18e6abe940(0000) GS:ffff9170bfd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000820af98 CR3: 0000000ffe614000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff9159cb10>] ? drain_pages+0xb0/0xb0
[<ffffffff91507f89>] on_each_cpu_mask+0x29/0x70
[<ffffffff915999f5>] drain_all_pages+0xb5/0xc0
[<ffffffff9159e860>] __alloc_pages_nodemask+0x8a0/0xbb0
[<ffffffff9148ed4d>] copy_process+0x1dd/0x1a40
[<ffffffff91490761>] do_fork+0x91/0x320
[<ffffffff916d0b0c>] ? security_file_fcntl+0x1c/0x20
[<ffffffff9162e69f>] ? SyS_fcntl+0xef/0x690
[<ffffffff91490a76>] SyS_clone+0x16/0x20
[<ffffffff91b1fb94>] stub_clone+0x44/0x70
[<ffffffff91b1f7d5>] ? system_call_fastpath+0x1c/0x21
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 13 92 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89

Complete report #1411168

Frame #	Function	Binary	Source or offset	Line
1	smp_call_function_many	vmlinux	0xffffffff9f11e25a Build id: 3.10.0-1160.119.1.el7.x86_64	0x20a
2	? drain_pages	vmlinux	0xffffffff9f1d2910 Build id: 3.10.0-1160.119.1.el7.x86_64	0xb0
3	on_each_cpu_mask	vmlinux	0xffffffff9f11e3a9 Build id: 3.10.0-1160.119.1.el7.x86_64	0x29
4	drain_all_pages	vmlinux	0xffffffff9f1cf715 Build id: 3.10.0-1160.119.1.el7.x86_64	0xb5
5	__alloc_pages_nodemask	vmlinux	0xffffffff9f1d46fb Build id: 3.10.0-1160.119.1.el7.x86_64	0x8db
6	copy_process	vmlinux	0xffffffff9f09cd45 Build id: 3.10.0-1160.119.1.el7.x86_64	0x1e5
7	do_fork	vmlinux	0xffffffff9f09e791 Build id: 3.10.0-1160.119.1.el7.x86_64	0x91
8	? security_file_fcntl	vmlinux	0xffffffff9f31b50c Build id: 3.10.0-1160.119.1.el7.x86_64	0x1c
9	? SyS_fcntl	vmlinux	0xffffffff9f27045f Build id: 3.10.0-1160.119.1.el7.x86_64	0xef

NMI watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [systemd:1]
Modules linked in: iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_set binfmt_misc ext4 mbcache jbd2 ppdev sg parport_pc parport virtio_balloon joydev pcspkr i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom virtio_net virtio_console virtio_blk net_failover virtio_scsi failover ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci floppy virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 1 Comm: systemd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
task: ffff96dbb6320000 ti: ffff96dbb631c000 task.ti: ffff96dbb631c000
RIP: 0010:[<ffffffff9f11e25a>]  [<ffffffff9f11e25a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff96dbb631fc60  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 000800009f1d2910 RCX: ffff96de6ef1fc10
RDX: 0000000000000004 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff96dbb631fc98 R08: ffff96dbbfd6f400 R09: ffffffff9f39c7f9
R10: ffff96de6ee1f160 R11: ffffca821207ec00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff96dbb631fc10 R15: 0000000001320122
FS:  00007fc51dc7c940(0000) GS:ffff96de6ee00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f336d185024 CR3: 000000041e4bc000 CR4: 00000000000006f0
Call Trace:
[<ffffffff9f1d2910>] ? drain_pages+0xb0/0xb0
[<ffffffff9f11e3a9>] on_each_cpu_mask+0x29/0x70
[<ffffffff9f1cf715>] drain_all_pages+0xb5/0xd0
[<ffffffff9f1d46fb>] __alloc_pages_nodemask+0x8db/0xbf0
[<ffffffff9f09cd45>] copy_process+0x1e5/0x1a80
[<ffffffff9f09e791>] do_fork+0x91/0x340
[<ffffffff9f31b50c>] ? security_file_fcntl+0x1c/0x30
[<ffffffff9f27045f>] ? SyS_fcntl+0xef/0x6e0

Complete report #470284

Frame #	Function	Binary	Source or offset	Line
1	smp_call_function_many	vmlinux	0xffffffff9bd1722a Build id: 3.10.0-1160.62.1.el7.x86_64	0x20a
2	? drain_pages	vmlinux	0xffffffff9bdc78f0 Build id: 3.10.0-1160.62.1.el7.x86_64	0xb0
3	on_each_cpu_mask	vmlinux	0xffffffff9bd17379 Build id: 3.10.0-1160.62.1.el7.x86_64	0x29
4	drain_all_pages	vmlinux	0xffffffff9bdc47d5 Build id: 3.10.0-1160.62.1.el7.x86_64	0xb5
5	__alloc_pages_nodemask	vmlinux	0xffffffff9bdc965b Build id: 3.10.0-1160.62.1.el7.x86_64	0x8cb
6	copy_process	vmlinux	0xffffffff9bc989bd Build id: 3.10.0-1160.62.1.el7.x86_64	0x1dd
7	do_fork	vmlinux	0xffffffff9bc9a411 Build id: 3.10.0-1160.62.1.el7.x86_64	0x91
8	? security_file_fcntl	vmlinux	0xffffffff9bf0931c Build id: 3.10.0-1160.62.1.el7.x86_64	0x1c
9	? SyS_fcntl	vmlinux	0xffffffff9be62357 Build id: 3.10.0-1160.62.1.el7.x86_64	0xe7
10	SyS_clone	vmlinux	0xffffffff9bc9a736 Build id: 3.10.0-1160.62.1.el7.x86_64	0x16
11	stub_clone	vmlinux	0xffffffff9c39a374 Build id: 3.10.0-1160.62.1.el7.x86_64	0x44
12	? system_call_fastpath	vmlinux	0xffffffff9c399f92 Build id: 3.10.0-1160.62.1.el7.x86_64	0x25

NMI watchdog: BUG: soft lockup - CPU#4 stuck for 21s! [automount:21784]
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 21s! [systemd:1]
Modules linked in: nfsv3 nfs_acl tcp_lp rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev pcspkr sg joydev vmw_balloon vmw_vmci i2c_piix4 parport_pc parport binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx sd_mod crc_t10dif crct10dif_generic drm_kms_helper syscopyarea sysfillrect sysimgblt
fb_sys_fops ttm ahci nfit libahci libnvdimm drm ata_piix libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 5 PID: 1 Comm: systemd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8f9df3750000 ti: ffff8f9df36dc000 task.ti: ffff8f9df36dc000
RIP: 0010:[<ffffffff9bd1722a>]  [<ffffffff9bd1722a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8f9df36dfc60  EFLAGS: 00000202
RAX: 0000000000000001 RBX: 000800009bdc78f0 RCX: ffff8fbcffc606a0
RDX: 0000000000000001 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff8f9df36dfc98 R08: ffff8f9df37b3000 R09: ffffffff9bf869d9
R10: ffff8fbcffd5f160 R11: ffffee8d3e027400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff8f9df36dfc10 R15: 0000000001320122
FS:  00007fd48889c940(0000) GS:ffff8fbcffd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc8149dc5a0 CR3: 0000001fb7d78000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9bdc78f0>] ? drain_pages+0xb0/0xb0
[<ffffffff9bd17379>] on_each_cpu_mask+0x29/0x70
[<ffffffff9bdc47d5>] drain_all_pages+0xb5/0xc0
[<ffffffff9bdc965b>] __alloc_pages_nodemask+0x8cb/0xbe0
[<ffffffff9bc989bd>] copy_process+0x1dd/0x1a80
[<ffffffff9bc9a411>] do_fork+0x91/0x330
[<ffffffff9bf0931c>] ? security_file_fcntl+0x1c/0x20
[<ffffffff9be62357>] ? SyS_fcntl+0xe7/0x6d0
[<ffffffff9bc9a736>] SyS_clone+0x16/0x20
[<ffffffff9c39a374>] stub_clone+0x44/0x70
[<ffffffff9c399f92>] ? system_call_fastpath+0x25/0x2a
Code: 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 95 9c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89

Complete report #330350

Frame #	Function	Binary	Source or offset	Line
1	smp_call_function_many	vmlinux	0xffffffffa671173a Build id: 3.10.0-957.el7.x86_64	0x20a
2	? drain_pages	vmlinux	0xffffffffa67bf550 Build id: 3.10.0-957.el7.x86_64	0xb0
3	on_each_cpu_mask	vmlinux	0xffffffffa6711889 Build id: 3.10.0-957.el7.x86_64	0x29
4	drain_all_pages	vmlinux	0xffffffffa67bc435 Build id: 3.10.0-957.el7.x86_64	0xb5
5	__alloc_pages_slowpath	vmlinux	0xffffffffa6d5d228 Build id: 3.10.0-957.el7.x86_64	0x510
6	__alloc_pages_nodemask	vmlinux	0xffffffffa67c0e95 Build id: 3.10.0-957.el7.x86_64	0x405
7	copy_process	vmlinux	0xffffffffa6694cdd Build id: 3.10.0-957.el7.x86_64	0x1dd
8	do_fork	vmlinux	0xffffffffa66966f1 Build id: 3.10.0-957.el7.x86_64	0x91
9	? security_file_fcntl	vmlinux	0xffffffffa68f906c Build id: 3.10.0-957.el7.x86_64	0x1c
10	? SyS_fcntl	vmlinux	0xffffffffa6854c3f Build id: 3.10.0-957.el7.x86_64	0xef
11	SyS_clone	vmlinux	0xffffffffa6696a06 Build id: 3.10.0-957.el7.x86_64	0x16
12	stub_clone	vmlinux	0xffffffffa6d751a4 Build id: 3.10.0-957.el7.x86_64	0x44
13	? system_call_fastpath	vmlinux	0xffffffffa6d74ddb Build id: 3.10.0-957.el7.x86_64	0x22

NMI watchdog: BUG: soft lockup - CPU#3 stuck for 32s! [systemd:1]
Modules linked in: binfmt_misc ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic pata_acpi crct10dif_pclmul crct10dif_common
crc32c_intel serio_raw vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm vmxnet3 ata_piix libata drm_panel_orientation_quirks mptspi scsi_transport_spi mptscsih mptbase dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 1 Comm: systemd Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff97c3bce48000 ti: ffff97c3bcec0000 task.ti: ffff97c3bcec0000
RIP: 0010:[<ffffffffa671173a>]  [<ffffffffa671173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff97c3bcec3bf0  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000c0000a67bf550 RCX: ffff97c47fc20600
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff97c3bcec3c28 R08: ffff97c3bcf12400 R09: ffffffffa69755c9
R10: ffff97c47fd9f160 R11: ffffd7b4c899e900 R12: 0000000000000286
R13: 0000000000000286 R14: ffff97c3bcec3ba0 R15: 0000000001320122
FS:  00007f2160c9c940(0000) GS:ffff97c47fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f85afa33000 CR3: 00000000365b0000 CR4: 00000000001607e0
Call Trace:
[<ffffffffa67bf550>] ? drain_pages+0xb0/0xb0
[<ffffffffa6711889>] on_each_cpu_mask+0x29/0x70
[<ffffffffa67bc435>] drain_all_pages+0xb5/0xc0
[<ffffffffa6d5d228>] __alloc_pages_slowpath+0x510/0x724
[<ffffffffa67c0e95>] __alloc_pages_nodemask+0x405/0x420
[<ffffffffa6694cdd>] copy_process+0x1dd/0x1a40
[<ffffffffa66966f1>] do_fork+0x91/0x320
[<ffffffffa68f906c>] ? security_file_fcntl+0x1c/0x20
[<ffffffffa6854c3f>] ? SyS_fcntl+0xef/0x6f0
[<ffffffffa6696a06>] SyS_clone+0x16/0x20
[<ffffffffa6d751a4>] stub_clone+0x44/0x70
[<ffffffffa6d74ddb>] ? system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 34 a7 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89