Complete report #1418095
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffa151e2ba
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x20a
|
|
2
|
|
vmlinux |
0xffffffffa15d27f0
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0xb0
|
|
3
|
on_each_cpu_mask
|
vmlinux |
0xffffffffa151e409
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x29
|
|
4
|
drain_all_pages
|
vmlinux |
0xffffffffa15cf5f5
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0xb5
|
|
5
|
__alloc_pages_nodemask
|
vmlinux |
0xffffffffa15d45db
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x8db
|
|
6
|
alloc_pages_current
|
vmlinux |
0xffffffffa1625c18
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x98
|
|
7
|
__page_cache_alloc
|
vmlinux |
0xffffffffa15c8bf7
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x97
|
|
8
|
__do_page_cache_readahead
|
vmlinux |
0xffffffffa15d6d8d
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0xfd
|
|
9
|
ondemand_readahead
|
vmlinux |
0xffffffffa15d700f
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x11f
|
|
10
|
page_cache_sync_readahead
|
vmlinux |
0xffffffffa15d7468
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x68
|
|
11
|
do_generic_file_read.constprop.52
|
vmlinux |
0xffffffffa15c9fec
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0xdc
|
|
12
|
|
vmlinux |
0xffffffffa1680ea4
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x24
|
|
13
|
generic_file_aio_read2
|
vmlinux |
0xffffffffa15cb0a9
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x1b9
|
|
14
|
ext4_file_read
|
ext4 |
0xffffffffc02a3ea3
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x43
|
|
15
|
do_sync_read
|
vmlinux |
0xffffffffa165b323
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x93
|
|
16
|
vfs_read
|
vmlinux |
0xffffffffa165bd3f
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x9f
|
|
17
|
SyS_read
|
vmlinux |
0xffffffffa165cbb5
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x55
|
|
18
|
system_call_fastpath
|
vmlinux |
0xffffffffa1bc639f
Build id: 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#9 stuck for 23s! [php-fpm:8433]
Modules linked in: kcare(OE) tcp_diag inet_diag ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter bridge stp llc ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_ipv4 nf_log_common xt_TCPMSS nf_tables nf_nat_ftp xt_REDIRECT nf_nat_redirect xt_conntrack nf_conntrack_ftp xt_LOG xt_limit xt_multiport iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack libcrc32c iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 overlay(T) loop binfmt_misc nfnetlink_log sunrpc dm_mirror dm_region_hash dm_log dm_mod virtio_gpu ttm kvm_amd drm_kms_helper kvm syscopyarea sysfillrect irqbypass sysimgblt crc32_pclmul fb_sys_fops ghash_clmulni_intel drm aesni_intel virtio_rng lrw drm_panel_orientation_quirks gf128mul glue_helper ablk_helper sg cryptd pcspkr virtio_balloon i2c_piix4 joydev ip_set_bitmap_port ip_set_hash_netport ip_set_hash_ip ip_set_hash_net xt_set ip_set ip6table_filter iptable_filter ip6_tables ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi virtio_net net_failover virtio_console virtio_scsi failover ata_piix crct10dif_pclmul libata crct10dif_common crc32c_intel virtio_pci virtio_ring serio_raw floppy virtio
CPU: 9 PID: 8433 Comm: php-fpm Kdump: loaded Tainted: G OEL ------------ T 3.10.0-1160.144.1.el7.tuxcare.els4.x86_64 #1
Hardware name: RDO OpenStack Compute, BIOS 1.16.3-5.el9.alma.1 11/05/2023
task: ffffa11ce8cd9080 ti: ffffa11ef9a9c000 task.ti: ffffa11ef9a9c000
RIP: 0010:[<ffffffffa151e2ba>] [<ffffffffa151e2ba>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa11ef9a9f928 EFLAGS: 00000202
RAX: 000000000000000d RBX: 00000000000000fc RCX: ffffa1206f560578
RDX: 000000000000000d RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffffa11ef9a9f960 R08: ffffa11db5d10800 R09: ffffffffa179cc89
R10: ffffa1206f45f160 R11: ffffdf5ecdcbd600 R12: 0000000000000292
R13: ffffa11ef9a9f8d8 R14: 0000000001320122 R15: 00000000bfd4271b
FS: 00007fe24663c880(0000) GS:ffffa1206f440000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000005550b0 CR3: 000000042d566000 CR4: 0000000000760ee0
PKRU: 55555554
Call Trace:
[<ffffffffa15d27f0>] ? drain_pages+0xb0/0xb0
[<ffffffffa151e409>] on_each_cpu_mask+0x29/0x70
[<ffffffffa15cf5f5>] drain_all_pages+0xb5/0xd0
[<ffffffffa15d45db>] __alloc_pages_nodemask+0x8db/0xbf0
[<ffffffffa1625c18>] alloc_pages_current+0x98/0x110
[<ffffffffa15c8bf7>] __page_cache_alloc+0x97/0xb0
[<ffffffffa15d6d8d>] __do_page_cache_readahead+0xfd/0x260
[<ffffffffa15d700f>] ondemand_readahead+0x11f/0x240
[<ffffffffa15d7468>] page_cache_sync_readahead+0x68/0x80
[<ffffffffa15c9fec>] do_generic_file_read.constprop.52+0xdc/0x510
[<ffffffffa1680ea4>] ? mntput+0x24/0x40
[<ffffffffa15cb0a9>] generic_file_aio_read2+0x1b9/0x330
[<ffffffffc02a3ea3>] ext4_file_read+0x43/0x140 [ext4]
[<ffffffffa165b323>] do_sync_read+0x93/0xe0
[<ffffffffa165bd3f>] vfs_read+0x9f/0x170
[<ffffffffa165cbb5>] SyS_read+0x55/0xd0
[<ffffffffa1bc639f>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 0e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 80 19 15 a2 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 dd 17 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #403634
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb931722a
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x20a
|
|
2
|
|
vmlinux |
0xffffffffb93c78f0
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xb0
|
|
3
|
on_each_cpu_mask
|
vmlinux |
0xffffffffb9317379
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x29
|
|
4
|
drain_all_pages
|
vmlinux |
0xffffffffb93c47d5
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xb5
|
|
5
|
__alloc_pages_nodemask
|
vmlinux |
0xffffffffb93c965b
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x8cb
|
|
6
|
alloc_pages_current
|
vmlinux |
0xffffffffb94193a8
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x98
|
|
7
|
__page_cache_alloc
|
vmlinux |
0xffffffffb93be027
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x97
|
|
8
|
__do_page_cache_readahead
|
vmlinux |
0xffffffffb93cbd5d
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0xfd
|
|
9
|
ondemand_readahead
|
vmlinux |
0xffffffffb93cbfdf
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x11f
|
|
10
|
page_cache_async_readahead
|
vmlinux |
0xffffffffb93cc180
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x80
|
|
11
|
|
vmlinux |
0xffffffffb93bd93e
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x1e
|
|
12
|
do_generic_file_read.constprop.52
|
vmlinux |
0xffffffffb93bf6b1
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x3c1
|
|
13
|
generic_file_aio_read2
|
vmlinux |
0xffffffffb93c0425
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x1b5
|
|
14
|
|
vmlinux |
0xffffffffb946bf1f
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x5f
|
|
15
|
ext4_file_read
|
ext4 |
0xffffffffc0849dc3
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x43
|
|
16
|
do_sync_read
|
vmlinux |
0xffffffffb944d923
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x93
|
|
17
|
vfs_read
|
vmlinux |
0xffffffffb944e2ff
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x9f
|
|
18
|
SyS_read
|
vmlinux |
0xffffffffb944f145
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x55
|
|
19
|
system_call_fastpath
|
vmlinux |
0xffffffffb9999f92
Build id: 3.10.0-1160.62.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [tar:16432]
Modules linked in: tcp_lp uinput xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink arc4 md4 nls_utf8 cifs ccm dns_resolver sunrpc ext4 snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core mbcache jbd2 snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd i2c_piix4 ppdev soundcore virtio_balloon iosf_mbi parport_pc joydev parport pcspkr crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ip_tables xfs libcrc32c ata_generic pata_acpi qxl drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm virtio_net ata_piix
net_failover virtio_console virtio_blk failover drm libata serio_raw crct10dif_pclmul crct10dif_common crc32c_intel virtio_pci floppy virtio_ring virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 4 PID: 16432 Comm: tar Kdump: loaded Tainted: G L ------------ 3.10.0-1160.62.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff942c3fd38000 ti: ffff942a4bda4000 task.ti: ffff942a4bda4000
RIP: 0010:[<ffffffffb931722a>] [<ffffffffb931722a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff942a4bda7900 EFLAGS: 00000202
RAX: 0000000000000003 RBX: 000c0000b93c78f0 RCX: ffff942c7fce0398
RDX: 0000000000000003 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff942a4bda7938 R08: ffff942ab8321400 R09: ffffffffb95869d9
R10: ffff942c7fd1f160 R11: fffff08d0bf14400 R12: 0000000000000282
R13: 0000000000000282 R14: ffff942a4bda78b0 R15: 0000000001320122
FS: 00007f603676a840(0000) GS:ffff942c7fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f6036745000 CR3: 0000000103a38000 CR4: 00000000003606e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb93c78f0>] ? drain_pages+0xb0/0xb0
[<ffffffffb9317379>] on_each_cpu_mask+0x29/0x70
[<ffffffffb93c47d5>] drain_all_pages+0xb5/0xc0
[<ffffffffb93c965b>] __alloc_pages_nodemask+0x8cb/0xbe0
[<ffffffffb94193a8>] alloc_pages_current+0x98/0x110
[<ffffffffb93be027>] __page_cache_alloc+0x97/0xb0
[<ffffffffb93cbd5d>] __do_page_cache_readahead+0xfd/0x260
[<ffffffffb93cbfdf>] ondemand_readahead+0x11f/0x240
[<ffffffffb93cc180>] page_cache_async_readahead+0x80/0xa0
[<ffffffffb93bd93e>] ? __find_get_page+0x1e/0xa0
[<ffffffffb93bf6b1>] do_generic_file_read.constprop.52+0x3c1/0x510
[<ffffffffb93c0425>] generic_file_aio_read2+0x1b5/0x330
[<ffffffffb946bf1f>] ? file_update_time+0x5f/0xf0
[<ffffffffc0849dc3>] ext4_file_read+0x43/0x130 [ext4]
[<ffffffffb944d923>] do_sync_read+0x93/0xe0
[<ffffffffb944e2ff>] vfs_read+0x9f/0x170
[<ffffffffb944f145>] SyS_read+0x55/0xd0
[<ffffffffb9999f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 7e 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 18 f5 b9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 87 c4 00 eb b7 0f b6 4d cc 4c 89