Info

Permalink
Function
smp_call_function_many
First occurrence
2022-11-29
Last occurrence
2022-12-28
State
NEW
Components
kernel
Quality
-38

Statistics

Operating system Count
CentOS 7 8
Architecture Count
x86_64 8
Related packages Count
kernel 8
 0:3.10.0-1160.59.1.el7 2
 0:3.10.0-1160.el7 2
 0:3.10.0-1127.18.2.el7 2
 0:3.10.0-1160.119.1.el7 1
 0:3.10.0-1160.144.1.el7.tuxcare.els4 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

No reports in the last 20 days.

Report backtrace


Complete report #1418096
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa151e2ba
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffffa15d27f0
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffffa151e409
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffffa15cf5f5
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffffa15d45db
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x8db
6
alloc_pages_current
vmlinux 0xffffffffa1625c18
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x98
7
__page_cache_alloc
vmlinux 0xffffffffa15c8bf7
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x97
8
__do_page_cache_readahead
vmlinux 0xffffffffa15d6d8d
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0xfd
9
ra_submit
vmlinux 0xffffffffa15d74a1
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x21
10
filemap_fault
vmlinux 0xffffffffa15cbb15
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x105
11
ext4_filemap_fault
ext4 0xffffffffc02b1a06
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x36
12
__do_fault.isra.61
vmlinux 0xffffffffa15fa12a
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x8a
13
do_read_fault.isra.63
vmlinux 0xffffffffa15fa6dc
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x4c
14
handle_mm_fault
vmlinux 0xffffffffa1602113
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0xa33
15
__do_page_fault
vmlinux 0xffffffffa1bc07d3
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x213
16
trace_do_page_fault
vmlinux 0xffffffffa1bc0bb6
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x56
17
do_async_page_fault
vmlinux 0xffffffffa1bc0112
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x22
18
async_page_fault
vmlinux 0xffffffffa1bbc7e8
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [imunify-residen:7740]
Modules linked in: kcare(OE) tcp_diag inet_diag ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter bridge stp llc ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_ipv4 nf_log_common xt_TCPMSS nf_tables nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 overlay(T) loop binfmt_misc nfnetlink_log sunrpc dm_mirror dm_region_hash dm_log dm_mod virtio_gpu ttm kvm_amd drm_kms_helper kvm syscopyarea sysfillrect irqbypass sysimgblt crc32_pclmul fb_sys_fops ghash_clmulni_intel drm aesni_intel virtio_rng lrw drm_panel_orientation_quirks gf128mul glue_helper ablk_helper sg cryptd pcspkr virtio_balloon i2c_piix4 joydev ip_set_bitmap_port ip_set_hash_netport ip_set_hash_ip ip_set_hash_net xt_set ip_set ip6table_filter iptable_filter ip6_tables ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_net net_failover virtio_console virtio_scsi failover ata_piix crct10dif_pclmul libata crct10dif_common crc32c_intel virtio_pci virtio_ring serio_raw floppy virtio
CPU: 7 PID: 7740 Comm: imunify-residen Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64 #1
Hardware name: RDO OpenStack Compute, BIOS 1.16.3-5.el9.alma.1 11/05/2023
task: ffffa11fdb722100 ti: ffffa11f4bdc4000 task.ti: ffffa11f4bdc4000
RIP: 0010:[<ffffffffa151e2ba>]  [<ffffffffa151e2ba>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffffa11f4bdc7920  EFLAGS: 00000202
RAX: 000000000000000d RBX: 00080000a15d27f0 RCX: ffffa1206f560528
RDX: 000000000000000d RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffffa11f4bdc7958 R08: ffffa11db5d10000 R09: ffffffffa179cc89
R10: ffffa1206f3df160 R11: ffffdf5ec250d600 R12: 0000000000000286
R13: 0000000000000000 R14: ffffa11f4bdc7890 R15: 0000000000000286
FS:  00007f7644ff9700(0000) GS:ffffa1206f3c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7591f8d000 CR3: 00000000a8c28000 CR4: 0000000000760ee0
PKRU: 55555554
Call Trace:
 [<ffffffffa15d27f0>] ? drain_pages+0xb0/0xb0
 [<ffffffffa151e409>] on_each_cpu_mask+0x29/0x70
 [<ffffffffa15cf5f5>] drain_all_pages+0xb5/0xd0
 [<ffffffffa15d45db>] __alloc_pages_nodemask+0x8db/0xbf0
 [<ffffffffa1625c18>] alloc_pages_current+0x98/0x110
 [<ffffffffa15c8bf7>] __page_cache_alloc+0x97/0xb0
 [<ffffffffa15d6d8d>] __do_page_cache_readahead+0xfd/0x260
 [<ffffffffa15d74a1>] ra_submit+0x21/0x30
 [<ffffffffa15cbb15>] filemap_fault+0x105/0x420
 [<ffffffffc02b1a06>] ext4_filemap_fault+0x36/0x50 [ext4]
 [<ffffffffa15fa12a>] __do_fault.isra.61+0x8a/0x100
 [<ffffffffa15fa6dc>] do_read_fault.isra.63+0x4c/0x1b0
 [<ffffffffa1602113>] handle_mm_fault+0xa33/0x1190
 [<ffffffffa1bc07d3>] __do_page_fault+0x213/0x510
 [<ffffffffa1bc0bb6>] trace_do_page_fault+0x56/0x150
 [<ffffffffa1bc0112>] do_async_page_fault+0x22/0x100
 [<ffffffffa1bbc7e8>] async_page_fault+0x28/0x30
Code: 48 63 35 0e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 80 19 15 a2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 dd 17 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1158504
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa991e25a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffffa99d2910
Build id: 3.10.0-1160.119.1.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffffa991e3a9
Build id: 3.10.0-1160.119.1.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffffa99cf715
Build id: 3.10.0-1160.119.1.el7.x86_64
0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffffa99d46fb
Build id: 3.10.0-1160.119.1.el7.x86_64
0x8db
6
alloc_pages_current
vmlinux 0xffffffffa9a25a98
Build id: 3.10.0-1160.119.1.el7.x86_64
0x98
7
__page_cache_alloc
vmlinux 0xffffffffa99c8d17
Build id: 3.10.0-1160.119.1.el7.x86_64
0x97
8
__do_page_cache_readahead
vmlinux 0xffffffffa99d6ead
Build id: 3.10.0-1160.119.1.el7.x86_64
0xfd
9
ra_submit
vmlinux 0xffffffffa99d75c1
Build id: 3.10.0-1160.119.1.el7.x86_64
0x21
10
filemap_fault
vmlinux 0xffffffffa99cbc35
Build id: 3.10.0-1160.119.1.el7.x86_64
0x105
11
? account_entity_dequeue
vmlinux 0xffffffffa98e72ee
Build id: 3.10.0-1160.119.1.el7.x86_64
0xae
12
ext4_filemap_fault
ext4 0xffffffffc056ca06
Build id: 3.10.0-1160.119.1.el7.x86_64
0x36
13
__do_fault.isra.61
vmlinux 0xffffffffa99fa21a
Build id: 3.10.0-1160.119.1.el7.x86_64
0x8a
14
do_read_fault.isra.63
vmlinux 0xffffffffa99fa7cc
Build id: 3.10.0-1160.119.1.el7.x86_64
0x4c
15
? hrtimer_try_to_cancel
vmlinux 0xffffffffa98cfca9
Build id: 3.10.0-1160.119.1.el7.x86_64
0x29
16
handle_mm_fault
vmlinux 0xffffffffa9a021d3
Build id: 3.10.0-1160.119.1.el7.x86_64
0xa33
17
? hrtimer_get_res
vmlinux 0xffffffffa98cf340
Build id: 3.10.0-1160.119.1.el7.x86_64
0x50
18
__do_page_fault
vmlinux 0xffffffffa9fbf7d3
Build id: 3.10.0-1160.119.1.el7.x86_64
0x213
19
trace_do_page_fault
vmlinux 0xffffffffa9fbfbb6
Build id: 3.10.0-1160.119.1.el7.x86_64
0x56
20
do_async_page_fault
vmlinux 0xffffffffa9fbf112
Build id: 3.10.0-1160.119.1.el7.x86_64
0x22
21
async_page_fault
vmlinux 0xffffffffa9fbb7e8
Build id: 3.10.0-1160.119.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [qmgr:1511]
Modules linked in: binfmt_misc ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod squashfs loop edac_mce_amd sg pcspkr joydev virtio_balloon i2c_piix4 ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_common ata_generic pata_acpi virtio_net virtio_console net_failover virtio_scsi failover bochs_drm drm_kms_helper ata_piix syscopyarea sysfillrect sysimgblt fb_sys_fops ttm libata drm serio_raw virtio_pci virtio_ring virtio drm_panel_orientation_quirks floppy fuse
CPU: 1 PID: 1511 Comm: qmgr Kdump: loaded Tainted: G             L ------------   3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org 04/01/2014
task: ffff8e0173a95280 ti: ffff8e016a794000 task.ti: ffff8e016a794000
RIP: 0010:[<ffffffffa991e25a>]  [<ffffffffa991e25a>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8e016a797920  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000c0000a99d2910 RCX: ffff8e01ffc204d0
RDX: 0000000000000000 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8e016a797958 R08: ffff8e01f8491800 R09: ffffffffa9b9c7f9
R10: ffff8e01ffc9f160 R11: ffffe57682c9b000 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8e016a7978d0 R15: 0000000001320122
FS:  00007f3d65726880(0000) GS:ffff8e01ffc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055aa9ddb20e0 CR3: 00000000b1b22000 CR4: 00000000000007e0
Call Trace:
 [<ffffffffa99d2910>] ? drain_pages+0xb0/0xb0
 [<ffffffffa991e3a9>] on_each_cpu_mask+0x29/0x70
 [<ffffffffa99cf715>] drain_all_pages+0xb5/0xd0
 [<ffffffffa99d46fb>] __alloc_pages_nodemask+0x8db/0xbf0
 [<ffffffffa9a25a98>] alloc_pages_current+0x98/0x110
 [<ffffffffa99c8d17>] __page_cache_alloc+0x97/0xb0
 [<ffffffffa99d6ead>] __do_page_cache_readahead+0xfd/0x260
 [<ffffffffa99d75c1>] ra_submit+0x21/0x30
 [<ffffffffa99cbc35>] filemap_fault+0x105/0x420
 [<ffffffffa98e72ee>] ? account_entity_dequeue+0xae/0xd0
 [<ffffffffc056ca06>] ext4_filemap_fault+0x36/0x50 [ext4]
 [<ffffffffa99fa21a>] __do_fault.isra.61+0x8a/0x100
 [<ffffffffa99fa7cc>] do_read_fault.isra.63+0x4c/0x1b0
 [<ffffffffa98cfca9>] ? hrtimer_try_to_cancel+0x29/0x130
 [<ffffffffa9a021d3>] handle_mm_fault+0xa33/0x1190
 [<ffffffffa98cf340>] ? hrtimer_get_res+0x50/0x50
 [<ffffffffa9fbf7d3>] __do_page_fault+0x213/0x510
 [<ffffffffa9fbfbb6>] trace_do_page_fault+0x56/0x150
 [<ffffffffa9fbf112>] do_async_page_fault+0x22/0x100
 [<ffffffffa9fbb7e8>] async_page_fault+0x28/0x30
Code: 48 63 35 ae 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 19 55 aa f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 7d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1193625
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8e31761a
Build id: 3.10.0-1127.18.2.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffff8e3c7900
Build id: 3.10.0-1127.18.2.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff8e317769
Build id: 3.10.0-1127.18.2.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff8e3c4775
Build id: 3.10.0-1127.18.2.el7.x86_64
0xb5
5
__alloc_pages_slowpath
vmlinux 0xffffffff8e97b315
Build id: 3.10.0-1127.18.2.el7.x86_64
0x510
6
__alloc_pages_nodemask
vmlinux 0xffffffff8e3c91f6
Build id: 3.10.0-1127.18.2.el7.x86_64
0x436
7
alloc_pages_current
vmlinux 0xffffffff8e418ea8
Build id: 3.10.0-1127.18.2.el7.x86_64
0x98
8
__page_cache_alloc
vmlinux 0xffffffff8e3be427
Build id: 3.10.0-1127.18.2.el7.x86_64
0x97
9
__do_page_cache_readahead
vmlinux 0xffffffff8e3cb5fd
Build id: 3.10.0-1127.18.2.el7.x86_64
0xfd
10
ra_submit
vmlinux 0xffffffff8e3cbd01
Build id: 3.10.0-1127.18.2.el7.x86_64
0x21
11
filemap_fault
vmlinux 0xffffffff8e3c0e75
Build id: 3.10.0-1127.18.2.el7.x86_64
0x105
12
ext4_filemap_fault
ext4 0xffffffffc04d4706
Build id: 3.10.0-1127.18.2.el7.x86_64
0x36
13
__do_fault.isra.61
vmlinux 0xffffffff8e3edf6a
Build id: 3.10.0-1127.18.2.el7.x86_64
0x8a
14
do_read_fault.isra.63
vmlinux 0xffffffff8e3ee51c
Build id: 3.10.0-1127.18.2.el7.x86_64
0x4c
15
handle_mm_fault
vmlinux 0xffffffff8e3f5d80
Build id: 3.10.0-1127.18.2.el7.x86_64
0xa20
16
? mmput
vmlinux 0xffffffff8e298614
Build id: 3.10.0-1127.18.2.el7.x86_64
0x14
17
__do_page_fault
vmlinux 0xffffffff8e98d653
Build id: 3.10.0-1127.18.2.el7.x86_64
0x213
18
do_page_fault
vmlinux 0xffffffff8e98d975
Build id: 3.10.0-1127.18.2.el7.x86_64
0x35
19
? error_swapgs
vmlinux 0xffffffff8e989ac9
Build id: 3.10.0-1127.18.2.el7.x86_64
0xaa
20
page_fault
vmlinux 0xffffffff8e989778
Build id: 3.10.0-1127.18.2.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [gnome-shell:16403]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(E) falcon_kal(E) falcon_lsm_pinned_17405(E) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter falcon_lsm_pinned_17308(E) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg vmw_balloon ppdev pcspkr joydev parport_pc parport i2c_piix4 vmw_vmci sch_fq_codel ip_tables ext4 mbcache jbd2 vmwgfx drm_kms_helper syscopyarea ata_generic sysfillrect nfit pata_acpi sysimgblt
fb_sys_fops libnvdimm ttm ahci sd_mod ata_piix libahci crc_t10dif drm crct10dif_generic crct10dif_pclmul vmxnet3 crct10dif_common libata crc32c_intel serio_raw vmw_pvscsi drm_panel_orientation_quirks [last unloaded: falcon_kal]
CPU: 3 PID: 16403 Comm: gnome-shell Kdump: loaded Tainted: P            EL ------------   3.10.0-1127.18.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff8a4feae420e0 ti: ffff8a4de7a80000 task.ti: ffff8a4de7a80000
RIP: 0010:[<ffffffff8e31761a>]  [<ffffffff8e31761a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8a4de7a838e8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff8a4fffc20650
RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff8a4de7a83920 R08: ffff8a4d3a372800 R09: ffffffff8e585179
R10: ffff8a4fffcdf160 R11: fffffd9650269200 R12: 0000000000000292
R13: ffff8a4de7a83898 R14: 0000000001320122 R15: ffffffff8e42806a
FS:  00007f3be2505a80(0000) GS:ffff8a4fffcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3bd7bcb570 CR3: 0000000422398000 CR4: 00000000000607e0
Call Trace:
[<ffffffff8e3c7900>] ? drain_pages+0xb0/0xb0
[<ffffffff8e317769>] on_each_cpu_mask+0x29/0x70
[<ffffffff8e3c4775>] drain_all_pages+0xb5/0xc0
[<ffffffff8e97b315>] __alloc_pages_slowpath+0x510/0x729
[<ffffffff8e3c91f6>] __alloc_pages_nodemask+0x436/0x450
[<ffffffff8e418ea8>] alloc_pages_current+0x98/0x110
[<ffffffff8e3be427>] __page_cache_alloc+0x97/0xb0
[<ffffffff8e3cb5fd>] __do_page_cache_readahead+0xfd/0x260
[<ffffffff8e3cbd01>] ra_submit+0x21/0x30
[<ffffffff8e3c0e75>] filemap_fault+0x105/0x420
[<ffffffffc04d4706>] ext4_filemap_fault+0x36/0x50 [ext4]
[<ffffffff8e3edf6a>] __do_fault.isra.61+0x8a/0x100
[<ffffffff8e3ee51c>] do_read_fault.isra.63+0x4c/0x1b0
[<ffffffff8e3f5d80>] handle_mm_fault+0xa20/0xfb0
[<ffffffff8e298614>] ? mmput+0x14/0xf0
[<ffffffff8e98d653>] __do_page_fault+0x213/0x500
[<ffffffff8e98d975>] do_page_fault+0x35/0x90
[<ffffffff8e989ac9>] ? error_swapgs+0xaa/0xc0
[<ffffffff8e989778>] page_fault+0x28/0x30
Code: 48 63 35 ce 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 10 f5 8e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 7b c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #747249
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff83316c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffff833c74a0
Build id: 3.10.0-1160.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff83316d59
Build id: 3.10.0-1160.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff833c4385
Build id: 3.10.0-1160.el7.x86_64
0xb5
5
__alloc_pages_slowpath
vmlinux 0xffffffff8397c6b2
Build id: 3.10.0-1160.el7.x86_64
0x510
6
__alloc_pages_nodemask
vmlinux 0xffffffff833c8d76
Build id: 3.10.0-1160.el7.x86_64
0x436
7
alloc_pages_current
vmlinux 0xffffffff834189d8
Build id: 3.10.0-1160.el7.x86_64
0x98
8
__page_cache_alloc
vmlinux 0xffffffff833bdb47
Build id: 3.10.0-1160.el7.x86_64
0x97
9
__do_page_cache_readahead
vmlinux 0xffffffff833cb17d
Build id: 3.10.0-1160.el7.x86_64
0xfd
10
ra_submit
vmlinux 0xffffffff833cb881
Build id: 3.10.0-1160.el7.x86_64
0x21
11
filemap_fault
vmlinux 0xffffffff833c0975
Build id: 3.10.0-1160.el7.x86_64
0x105
12
ext4_filemap_fault
ext4 0xffffffffc08ae756
Build id: 3.10.0-1160.el7.x86_64
0x36
13
__do_fault.isra.61
vmlinux 0xffffffff833ede3a
Build id: 3.10.0-1160.el7.x86_64
0x8a
14
? __schedule
vmlinux 0xffffffff83986c8f
Build id: 3.10.0-1160.el7.x86_64
0x3af
15
do_read_fault.isra.63
vmlinux 0xffffffff833ee3ec
Build id: 3.10.0-1160.el7.x86_64
0x4c
16
handle_mm_fault
vmlinux 0xffffffff833f5c30
Build id: 3.10.0-1160.el7.x86_64
0xa20
17
? schedule_hrtimeout_range_clock
vmlinux 0xffffffff83986346
Build id: 3.10.0-1160.el7.x86_64
0xa6
18
? __enqueue_entity
vmlinux 0xffffffff832dfb48
Build id: 3.10.0-1160.el7.x86_64
0x78
19
__do_page_fault
vmlinux 0xffffffff8398e653
Build id: 3.10.0-1160.el7.x86_64
0x213
20
do_page_fault
vmlinux 0xffffffff8398e975
Build id: 3.10.0-1160.el7.x86_64
0x35
21
page_fault
vmlinux 0xffffffff8398a778
Build id: 3.10.0-1160.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 25s! [java:35912]
Modules linked in: tcp_lp veth xt_nat nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc overlay(T) vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event ext4 ppdev mbcache jbd2 snd_ens1371 snd_rawmidi crc32_pclmul snd_ac97_codec ghash_clmulni_intel vmw_balloon ac97_bus aesni_intel lrw gf128mul glue_helper ablk_helper cryptd snd_seq snd_seq_device joydev snd_pcm pcspkr sg snd_timer snd soundcore i2c_piix4 vmw_vmci parport_pc parport ip_tables xfs
libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper nfit crct10dif_pclmul crct10dif_common crc32c_intel syscopyarea libnvdimm sysfillrect sysimgblt fb_sys_fops ttm serio_raw drm ata_piix e1000 mptspi scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 3 PID: 35912 Comm: java Kdump: loaded Tainted: G        W    L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffffa1273c6a9080 ti: ffffa12180788000 task.ti: ffffa12180788000
RIP: 0010:[<ffffffff83316c0a>]  [<ffffffff83316c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa1218078b8e8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffffa1289d620650
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffffa1218078b920 R08: ffffa121f2f5c000 R09: ffffffff83585f99
R10: ffffa1289d6df160 R11: ffffed5189d82000 R12: 0000000000000292
R13: ffffa1218078b898 R14: 0000000001320122 R15: 00000000ddde7aa4
FS:  00007fefc8332700(0000) GS:ffffa1289d6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f8d3554c100 CR3: 000000010b512000 CR4: 0000000000760fe0
PKRU: 55555554
Call Trace:
[<ffffffff833c74a0>] ? drain_pages+0xb0/0xb0
[<ffffffff83316d59>] on_each_cpu_mask+0x29/0x70
[<ffffffff833c4385>] drain_all_pages+0xb5/0xc0
[<ffffffff8397c6b2>] __alloc_pages_slowpath+0x510/0x729
[<ffffffff833c8d76>] __alloc_pages_nodemask+0x436/0x450
[<ffffffff834189d8>] alloc_pages_current+0x98/0x110
[<ffffffff833bdb47>] __page_cache_alloc+0x97/0xb0
[<ffffffff833cb17d>] __do_page_cache_readahead+0xfd/0x260
[<ffffffff833cb881>] ra_submit+0x21/0x30
[<ffffffff833c0975>] filemap_fault+0x105/0x420
[<ffffffffc08ae756>] ext4_filemap_fault+0x36/0x50 [ext4]
[<ffffffff833ede3a>] __do_fault.isra.61+0x8a/0x100
[<ffffffff83986c8f>] ? __schedule+0x3af/0x860
[<ffffffff833ee3ec>] do_read_fault.isra.63+0x4c/0x1b0
[<ffffffff833f5c30>] handle_mm_fault+0xa20/0xfb0
[<ffffffff83986346>] ? schedule_hrtimeout_range_clock+0xa6/0x150
[<ffffffff832dfb48>] ? __enqueue_entity+0x78/0x80
[<ffffffff8398e653>] __do_page_fault+0x213/0x500
[<ffffffff8398e975>] do_page_fault+0x35/0x90
[<ffffffff8398a778>] page_fault+0x28/0x30
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 f5 83 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #561639
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff847170da
Build id: 3.10.0-1160.59.1.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffff847c78e0
Build id: 3.10.0-1160.59.1.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff84717229
Build id: 3.10.0-1160.59.1.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff847c47c5
Build id: 3.10.0-1160.59.1.el7.x86_64
0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffff847c964b
Build id: 3.10.0-1160.59.1.el7.x86_64
0x8cb
6
alloc_pages_current
vmlinux 0xffffffff848193b8
Build id: 3.10.0-1160.59.1.el7.x86_64
0x98
7
__page_cache_alloc
vmlinux 0xffffffff847be007
Build id: 3.10.0-1160.59.1.el7.x86_64
0x97
8
__do_page_cache_readahead
vmlinux 0xffffffff847cbd4d
Build id: 3.10.0-1160.59.1.el7.x86_64
0xfd
9
ra_submit
vmlinux 0xffffffff847cc451
Build id: 3.10.0-1160.59.1.el7.x86_64
0x21
10
filemap_fault
vmlinux 0xffffffff847c0e35
Build id: 3.10.0-1160.59.1.el7.x86_64
0x105
11
ext4_filemap_fault
ext4 0xffffffffc028b756
Build id: 3.10.0-1160.59.1.el7.x86_64
0x36
12
__do_fault.isra.61
vmlinux 0xffffffff847ee7aa
Build id: 3.10.0-1160.59.1.el7.x86_64
0x8a
13
do_read_fault.isra.63
vmlinux 0xffffffff847eed5c
Build id: 3.10.0-1160.59.1.el7.x86_64
0x4c
14
handle_mm_fault
vmlinux 0xffffffff847f65a0
Build id: 3.10.0-1160.59.1.el7.x86_64
0xa20
15
__do_page_fault
vmlinux 0xffffffff84d94653
Build id: 3.10.0-1160.59.1.el7.x86_64
0x213
16
do_page_fault
vmlinux 0xffffffff84d94975
Build id: 3.10.0-1160.59.1.el7.x86_64
0x35
17
? error_swapgs
vmlinux 0xffffffff84d90ac9
Build id: 3.10.0-1160.59.1.el7.x86_64
0xaa
18
page_fault
vmlinux 0xffffffff84d90778
Build id: 3.10.0-1160.59.1.el7.x86_64
0x28
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [kswapd0:47]
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [sh:18444]
Modules linked in: vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev vmw_vmci sg i2c_piix4 parport_pc parport sch_fq_codel auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 nfit vmwgfx libnvdimm sr_mod cdrom drm_kms_helper ata_generic pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm sd_mod crc_t10dif ata_piix crct10dif_generic libata crct10dif_pclmul vmxnet3 crct10dif_common serio_raw crc32c_intel vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 3 PID: 18444 Comm: sh Kdump: loaded Tainted: G             L ------------   3.10.0-1160.59.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff8fd22b308000 ti: ffff8fd226d04000 task.ti: ffff8fd226d04000
RIP: 0010:[<ffffffff847170da>]  [<ffffffff847170da>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff8fd226d07950  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00080000847c78e0 RCX: ffff8fd23fd20650
RDX: 0000000000000002 RSI: 0000000000000004 RDI: 0000000000000000
RBP: ffff8fd226d07988 R08: ffff8fcf76357000 R09: ffffffff84986c09
R10: ffff8fd23fd9f160 R11: ffffdfd84d1d0100 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8fd226d07900 R15: 0000000001320122
FS:  00007f9e7e77a740(0000) GS:ffff8fd23fd80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f9e7786560b CR3: 00000002e6052000 CR4: 00000000000607e0
Call Trace:
[<ffffffff847c78e0>] ? drain_pages+0xb0/0xb0
[<ffffffff84717229>] on_each_cpu_mask+0x29/0x70
[<ffffffff847c47c5>] drain_all_pages+0xb5/0xc0
[<ffffffff847c964b>] __alloc_pages_nodemask+0x8cb/0xbe0
[<ffffffff848193b8>] alloc_pages_current+0x98/0x110
[<ffffffff847be007>] __page_cache_alloc+0x97/0xb0
[<ffffffff847cbd4d>] __do_page_cache_readahead+0xfd/0x260
[<ffffffff847cc451>] ra_submit+0x21/0x30
[<ffffffff847c0e35>] filemap_fault+0x105/0x420
[<ffffffffc028b756>] ext4_filemap_fault+0x36/0x50 [ext4]
[<ffffffff847ee7aa>] __do_fault.isra.61+0x8a/0x100
[<ffffffff847eed5c>] do_read_fault.isra.63+0x4c/0x1b0
[<ffffffff847f65a0>] handle_mm_fault+0xa20/0xfb0
[<ffffffff84d94653>] __do_page_fault+0x213/0x500
[<ffffffff84d94975>] do_page_fault+0x35/0x90
[<ffffffff84d90ac9>] ? error_swapgs+0xaa/0xc0
[<ffffffff84d90778>] page_fault+0x28/0x30
Code: 48 63 35 ce 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 35 85 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 88 c4 00 eb b7 0f b6 4d cc 4c 89