Info

Permalink
Function
smp_call_function_many
First occurrence
2023-09-30
Last occurrence
2024-08-17
State
NEW
Components
kernel
Quality
-66

Statistics

Operating system Count
CentOS 7 360
Architecture Count
x86_64 360
Related packages Count
kernel 360
 0:3.10.0-1160.53.1.el7 360

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1274645
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff825170de
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x20e
2
? drain_pages
vmlinux 0xffffffff825c78e0
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff82517229
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x29
4
drain_all_pages
vmlinux 0xffffffff825c47c5
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffff825c964b
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x8cb
6
alloc_pages_current
vmlinux 0xffffffff826193b8
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x98
7
new_slab
vmlinux 0xffffffff82627aa3
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x393
8
___slab_alloc
vmlinux 0xffffffff82627fbc
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x3cc
9
? alloc_pages_current
vmlinux 0xffffffff826193b8
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x98
10
? nfs_readhdr_alloc
nfs 0xffffffffc096474a
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1a
11
? new_slab
vmlinux 0xffffffff82627843
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x133
12
? nfs_readhdr_alloc
nfs 0xffffffffc096474a
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1a
13
__slab_alloc
vmlinux 0xffffffff82b7fea5
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x40
14
kmem_cache_alloc
vmlinux 0xffffffff8262917b
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x19b
15
? nfs_readhdr_alloc
nfs 0xffffffffc096474a
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1a
16
nfs_readhdr_alloc
nfs 0xffffffffc096474a
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1a
17
nfs_generic_pg_pgios
nfs 0xffffffffc0963392
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x22
18
nfs_pageio_doio
nfs 0xffffffffc0962cbd
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x2d
19
__nfs_pageio_add_request
nfs 0xffffffffc09637aa
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xba
20
? nfs_get_lock_context
nfs 0xffffffffc0958fc1
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x51
21
nfs_pageio_add_request
nfs 0xffffffffc0964266
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x176
22
readpage_async_filler
nfs 0xffffffffc0964d0b
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xfb
23
read_cache_pages
vmlinux 0xffffffff825cc280
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xa0
24
? nfs_return_empty_page
nfs 0xffffffffc0964c10
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x110
25
nfs_readpages
nfs 0xffffffffc0965555
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x155
26
__do_page_cache_readahead
vmlinux 0xffffffff825cbe1f
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1cf
27
force_page_cache_readahead
vmlinux 0xffffffff825cc369
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x99
28
SyS_fadvise64_64
vmlinux 0xffffffff825c309c
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1ec
29
SyS_fadvise64
vmlinux 0xffffffff825c315e
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xe
30
system_call_fastpath
vmlinux 0xffffffff82b95f92
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [bareos-fd:49939]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter tun bridge stp llc nfsv3 nfs_acl nfs lockd grace fscache vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper ppdev cryptd vmw_balloon joydev pcspkr sg parport_pc parport i2c_piix4 vmw_vmci auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm crct10dif_pclmul crct10dif_common
ahci crc32c_intel drm ata_piix libahci serio_raw nfit libata libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 5 PID: 49939 Comm: bareos-fd Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff94f02d295280 ti: ffff94ee2d998000 task.ti: ffff94ee2d998000
RIP: 0010:[<ffffffff825170de>]  [<ffffffff825170de>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff94ee2d99b7d8  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 00000000000000fc RCX: ffff94f0337a06a0
RDX: 0000000000000006 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff94ee2d99b810 R08: ffff94eef8c6c000 R09: ffffffff82786bf9
R10: ffff94f03375f160 R11: fffffcf5c4083c00 R12: 0000000000000296
R13: ffff94ee2d99b788 R14: 0000000001320122 R15: 00000000114c65cc
FS:  00007faccf558700(0000) GS:ffff94f033740000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fb08ed07000 CR3: 00000000bb07a000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff825c78e0>] ? drain_pages+0xb0/0xb0
[<ffffffff82517229>] on_each_cpu_mask+0x29/0x70
[<ffffffff825c47c5>] drain_all_pages+0xb5/0xc0
[<ffffffff825c964b>] __alloc_pages_nodemask+0x8cb/0xbe0
[<ffffffff826193b8>] alloc_pages_current+0x98/0x110
[<ffffffff82627aa3>] new_slab+0x393/0x4e0
[<ffffffff82627fbc>] ___slab_alloc+0x3cc/0x520
[<ffffffff826193b8>] ? alloc_pages_current+0x98/0x110
[<ffffffffc096474a>] ? nfs_readhdr_alloc+0x1a/0x30 [nfs]
[<ffffffff82627843>] ? new_slab+0x133/0x4e0
[<ffffffffc096474a>] ? nfs_readhdr_alloc+0x1a/0x30 [nfs]
[<ffffffff82b7fea5>] __slab_alloc+0x40/0x5c
[<ffffffff8262917b>] kmem_cache_alloc+0x19b/0x1f0
[<ffffffffc096474a>] ? nfs_readhdr_alloc+0x1a/0x30 [nfs]
[<ffffffffc096474a>] nfs_readhdr_alloc+0x1a/0x30 [nfs]
[<ffffffffc0963392>] nfs_generic_pg_pgios+0x22/0xe0 [nfs]
[<ffffffffc0962cbd>] nfs_pageio_doio+0x2d/0x70 [nfs]
[<ffffffffc09637aa>] __nfs_pageio_add_request+0xba/0x450 [nfs]
[<ffffffffc0958fc1>] ? nfs_get_lock_context+0x51/0x100 [nfs]
[<ffffffffc0964266>] nfs_pageio_add_request+0x176/0x330 [nfs]
[<ffffffffc0964d0b>] readpage_async_filler+0xfb/0x1d0 [nfs]
[<ffffffff825cc280>] read_cache_pages+0xa0/0xf0
[<ffffffffc0964c10>] ? nfs_return_empty_page+0x110/0x110 [nfs]
[<ffffffffc0965555>] nfs_readpages+0x155/0x1f0 [nfs]
[<ffffffff825cbe1f>] __do_page_cache_readahead+0x1cf/0x260
[<ffffffff825cc369>] force_page_cache_readahead+0x99/0xe0
[<ffffffff825c309c>] SyS_fadvise64_64+0x1ec/0x2a0
[<ffffffff825c315e>] SyS_fadvise64+0xe/0x10
[<ffffffff82b95f92>] system_call_fastpath+0x25/0x2a
Code: 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 15 83 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #970326
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb49170da
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x20a
2
? drain_pages
vmlinux 0xffffffffb49c78e0
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xb0
3
on_each_cpu_mask
vmlinux 0xffffffffb4917229
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x29
4
drain_all_pages
vmlinux 0xffffffffb49c47c5
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffffb49c964b
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x8cb
6
alloc_pages_current
vmlinux 0xffffffffb4a193b8
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x98
7
new_slab
vmlinux 0xffffffffb4a27aa3
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x393
8
___slab_alloc
vmlinux 0xffffffffb4a27fbc
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x3cc
9
? nfs_generic_pgio
nfs 0xffffffffc070f289
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x249
10
? get_partial_node.isra.48
vmlinux 0xffffffffb4a265f5
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x95
11
? nfs_generic_pgio
nfs 0xffffffffc070f289
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x249
12
__slab_alloc
vmlinux 0xffffffffb4f7fea5
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x40
13
__kmalloc
vmlinux 0xffffffffb4a28cc0
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1c0
14
nfs_generic_pgio
nfs 0xffffffffc070f289
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x249
15
? memcg_check_events
vmlinux 0xffffffffb4a3d82e
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xbe
16
? kmem_cache_alloc
vmlinux 0xffffffffb4a291a2
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1c2
17
? nfs_readhdr_alloc
nfs 0xffffffffc071074a
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1a
18
nfs_generic_pg_pgios
nfs 0xffffffffc070f3d2
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x62
19
nfs_pageio_doio
nfs 0xffffffffc070ecbd
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x2d
20
__nfs_pageio_add_request
nfs 0xffffffffc070f7aa
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xba
21
? nfs_get_lock_context
nfs 0xffffffffc0704fc1
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x51
22
nfs_pageio_add_request
nfs 0xffffffffc0710266
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x176
23
readpage_async_filler
nfs 0xffffffffc0710d0b
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xfb
24
read_cache_pages
vmlinux 0xffffffffb49cc280
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xa0
25
? nfs_return_empty_page
nfs 0xffffffffc0710c10
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x110
26
nfs_readpages
nfs 0xffffffffc0711555
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x155
27
__do_page_cache_readahead
vmlinux 0xffffffffb49cbe1f
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1cf
28
force_page_cache_readahead
vmlinux 0xffffffffb49cc369
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x99
29
SyS_fadvise64_64
vmlinux 0xffffffffb49c309c
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1ec
30
SyS_fadvise64
vmlinux 0xffffffffb49c315e
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xe
31
system_call_fastpath
vmlinux 0xffffffffb4f95f92
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 28s! [bareos-fd:24019]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter tun bridge stp llc nfsv3 nfs_acl nfs lockd grace fscache vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw ppdev gf128mul glue_helper ablk_helper cryptd vmw_balloon joydev pcspkr sg i2c_piix4 vmw_vmci parport_pc parport auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm sd_mod nfit ahci crc_t10dif crct10dif_generic drm libnvdimm
libahci ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 3 PID: 24019 Comm: bareos-fd Kdump: loaded Tainted: G             L ------------   3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff91351fd8c200 ti: ffff9136450f4000 task.ti: ffff9136450f4000
RIP: 0010:[<ffffffffb49170da>]  [<ffffffffb49170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9136450f7758  EFLAGS: 00000202
RAX: 0000000000000004 RBX: 00000000000000fc RCX: ffff91377fd20650
RDX: 0000000000000004 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff9136450f7790 R08: ffff913639375800 R09: ffffffffb4b86bf9
R10: ffff91377fcdf160 R11: ffffedb70909b700 R12: 0000000000000292
R13: ffff9136450f7708 R14: 0000000001320122 R15: 000000001a611ef0
FS:  00007f0951fb4700(0000) GS:ffff91377fcc0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3c0f81ebb8 CR3: 00000002b3380000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffb49c78e0>] ? drain_pages+0xb0/0xb0
[<ffffffffb4917229>] on_each_cpu_mask+0x29/0x70
[<ffffffffb49c47c5>] drain_all_pages+0xb5/0xc0
[<ffffffffb49c964b>] __alloc_pages_nodemask+0x8cb/0xbe0
[<ffffffffb4a193b8>] alloc_pages_current+0x98/0x110
[<ffffffffb4a27aa3>] new_slab+0x393/0x4e0
[<ffffffffb4a27fbc>] ___slab_alloc+0x3cc/0x520
[<ffffffffc070f289>] ? nfs_generic_pgio+0x249/0x330 [nfs]
[<ffffffffb4a265f5>] ? get_partial_node.isra.48+0x95/0x1f0
[<ffffffffc070f289>] ? nfs_generic_pgio+0x249/0x330 [nfs]
[<ffffffffb4f7fea5>] __slab_alloc+0x40/0x5c
[<ffffffffb4a28cc0>] __kmalloc+0x1c0/0x230
[<ffffffffc070f289>] nfs_generic_pgio+0x249/0x330 [nfs]
[<ffffffffb4a3d82e>] ? memcg_check_events+0xbe/0x210
[<ffffffffb4a291a2>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc071074a>] ? nfs_readhdr_alloc+0x1a/0x30 [nfs]
[<ffffffffc070f3d2>] nfs_generic_pg_pgios+0x62/0xe0 [nfs]
[<ffffffffc070ecbd>] nfs_pageio_doio+0x2d/0x70 [nfs]
[<ffffffffc070f7aa>] __nfs_pageio_add_request+0xba/0x450 [nfs]
[<ffffffffc0704fc1>] ? nfs_get_lock_context+0x51/0x100 [nfs]
[<ffffffffc0710266>] nfs_pageio_add_request+0x176/0x330 [nfs]
[<ffffffffc0710d0b>] readpage_async_filler+0xfb/0x1d0 [nfs]
[<ffffffffb49cc280>] read_cache_pages+0xa0/0xf0
[<ffffffffc0710c10>] ? nfs_return_empty_page+0x110/0x110 [nfs]
[<ffffffffc0711555>] nfs_readpages+0x155/0x1f0 [nfs]
[<ffffffffb49cbe1f>] __do_page_cache_readahead+0x1cf/0x260
[<ffffffffb49cc369>] force_page_cache_readahead+0x99/0xe0
[<ffffffffb49c309c>] SyS_fadvise64_64+0x1ec/0x2a0
[<ffffffffb49c315e>] SyS_fadvise64+0xe/0x10
[<ffffffffb4f95f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 55 b5 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #864365
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb59170de
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x20e
2
? drain_pages
vmlinux 0xffffffffb59c78e0
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xb0
3
on_each_cpu_mask
vmlinux 0xffffffffb5917229
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x29
4
drain_all_pages
vmlinux 0xffffffffb59c47c5
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffffb59c964b
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x8cb
6
alloc_pages_current
vmlinux 0xffffffffb5a193b8
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x98
7
new_slab
vmlinux 0xffffffffb5a27aa3
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x393
8
___slab_alloc
vmlinux 0xffffffffb5a27fbc
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x3cc
9
? nfs_generic_pgio
nfs 0xffffffffc07ce289
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x249
10
? nfs_generic_pgio
nfs 0xffffffffc07ce289
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x249
11
__slab_alloc
vmlinux 0xffffffffb5f7fea5
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x40
12
__kmalloc
vmlinux 0xffffffffb5a28cc0
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1c0
13
nfs_generic_pgio
nfs 0xffffffffc07ce289
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x249
14
? memcg_check_events
vmlinux 0xffffffffb5a3d82e
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xbe
15
? kmem_cache_alloc
vmlinux 0xffffffffb5a291a2
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1c2
16
? nfs_readhdr_alloc
nfs 0xffffffffc07cf74a
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1a
17
nfs_generic_pg_pgios
nfs 0xffffffffc07ce3d2
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x62
18
nfs_pageio_doio
nfs 0xffffffffc07cdcbd
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x2d
19
__nfs_pageio_add_request
nfs 0xffffffffc07ce7aa
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xba
20
? nfs_get_lock_context
nfs 0xffffffffc07c3fc1
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x51
21
nfs_pageio_add_request
nfs 0xffffffffc07cf266
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x176
22
readpage_async_filler
nfs 0xffffffffc07cfd0b
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xfb
23
read_cache_pages
vmlinux 0xffffffffb59cc280
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xa0
24
? nfs_return_empty_page
nfs 0xffffffffc07cfc10
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x110
25
nfs_readpages
nfs 0xffffffffc07d0555
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x155
26
__do_page_cache_readahead
vmlinux 0xffffffffb59cbe1f
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1cf
27
force_page_cache_readahead
vmlinux 0xffffffffb59cc369
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x99
28
SyS_fadvise64_64
vmlinux 0xffffffffb59c309c
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1ec
29
SyS_fadvise64
vmlinux 0xffffffffb59c315e
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xe
30
system_call_fastpath
vmlinux 0xffffffffb5f95f92
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [bareos-fd:30860]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter nfsv3 nfs_acl nfs lockd grace fscache vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev sg joydev vmw_balloon pcspkr vmw_vmci parport_pc i2c_piix4 parport auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif syscopyarea sysfillrect crct10dif_generic sysimgblt fb_sys_fops ttm ahci libahci
drm crct10dif_pclmul ata_piix crct10dif_common nfit crc32c_intel serio_raw libata libnvdimm vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 1 PID: 30860 Comm: bareos-fd Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9f7cf269b180 ti: ffff9f7a4268c000 task.ti: ffff9f7a4268c000
RIP: 0010:[<ffffffffb59170de>]  [<ffffffffb59170de>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff9f7a4268f758  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff9f7cffca0600
RDX: 0000000000000002 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff9f7a4268f790 R08: ffff9f7bb9375000 R09: ffffffffb5b86bf9
R10: ffff9f7cffc5f160 R11: ffffe4d48020f900 R12: 0000000000000292
R13: ffff9f7a4268f708 R14: 0000000001320122 R15: 00000000f39bde72
FS:  00007fcf1f144700(0000) GS:ffff9f7cffc40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0e89ac9a6e CR3: 00000002b277e000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffffb59c78e0>] ? drain_pages+0xb0/0xb0
[<ffffffffb5917229>] on_each_cpu_mask+0x29/0x70
[<ffffffffb59c47c5>] drain_all_pages+0xb5/0xc0
[<ffffffffb59c964b>] __alloc_pages_nodemask+0x8cb/0xbe0
[<ffffffffb5a193b8>] alloc_pages_current+0x98/0x110
[<ffffffffb5a27aa3>] new_slab+0x393/0x4e0
[<ffffffffb5a27fbc>] ___slab_alloc+0x3cc/0x520
[<ffffffffc07ce289>] ? nfs_generic_pgio+0x249/0x330 [nfs]
[<ffffffffc07ce289>] ? nfs_generic_pgio+0x249/0x330 [nfs]
[<ffffffffb5f7fea5>] __slab_alloc+0x40/0x5c
[<ffffffffb5a28cc0>] __kmalloc+0x1c0/0x230
[<ffffffffc07ce289>] nfs_generic_pgio+0x249/0x330 [nfs]
[<ffffffffb5a3d82e>] ? memcg_check_events+0xbe/0x210
[<ffffffffb5a291a2>] ? kmem_cache_alloc+0x1c2/0x1f0
[<ffffffffc07cf74a>] ? nfs_readhdr_alloc+0x1a/0x30 [nfs]
[<ffffffffc07ce3d2>] nfs_generic_pg_pgios+0x62/0xe0 [nfs]
[<ffffffffc07cdcbd>] nfs_pageio_doio+0x2d/0x70 [nfs]
[<ffffffffc07ce7aa>] __nfs_pageio_add_request+0xba/0x450 [nfs]
[<ffffffffc07c3fc1>] ? nfs_get_lock_context+0x51/0x100 [nfs]
[<ffffffffc07cf266>] nfs_pageio_add_request+0x176/0x330 [nfs]
[<ffffffffc07cfd0b>] readpage_async_filler+0xfb/0x1d0 [nfs]
[<ffffffffb59cc280>] read_cache_pages+0xa0/0xf0
[<ffffffffc07cfc10>] ? nfs_return_empty_page+0x110/0x110 [nfs]
[<ffffffffc07d0555>] nfs_readpages+0x155/0x1f0 [nfs]
[<ffffffffb59cbe1f>] __do_page_cache_readahead+0x1cf/0x260
[<ffffffffb59cc369>] force_page_cache_readahead+0x99/0xe0
[<ffffffffb59c309c>] SyS_fadvise64_64+0x1ec/0x2a0
[<ffffffffb59c315e>] SyS_fadvise64+0xe/0x10
[<ffffffffb5f95f92>] system_call_fastpath+0x25/0x2a
Code: 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 55 b6 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee
Complete report #814235
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff92d170da
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x20a
2
? drain_pages
vmlinux 0xffffffff92dc78e0
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff92d17229
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x29
4
drain_all_pages
vmlinux 0xffffffff92dc47c5
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffff92dc964b
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x8cb
6
alloc_pages_current
vmlinux 0xffffffff92e193b8
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x98
7
new_slab
vmlinux 0xffffffff92e27aa3
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x393
8
___slab_alloc
vmlinux 0xffffffff92e27fbc
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x3cc
9
? nfs_readhdr_alloc
nfs 0xffffffffc08bc74a
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1a
10
? new_slab
vmlinux 0xffffffff92e27843
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x133
11
? nfs_readhdr_alloc
nfs 0xffffffffc08bc74a
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1a
12
__slab_alloc
vmlinux 0xffffffff9337fea5
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x40
13
kmem_cache_alloc
vmlinux 0xffffffff92e2917b
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x19b
14
? nfs_readhdr_alloc
nfs 0xffffffffc08bc74a
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1a
15
nfs_readhdr_alloc
nfs 0xffffffffc08bc74a
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1a
16
nfs_generic_pg_pgios
nfs 0xffffffffc08bb392
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x22
17
nfs_pageio_doio
nfs 0xffffffffc08bacbd
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x2d
18
__nfs_pageio_add_request
nfs 0xffffffffc08bb7aa
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xba
19
? nfs_get_lock_context
nfs 0xffffffffc08b0fc1
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x51
20
nfs_pageio_add_request
nfs 0xffffffffc08bc266
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x176
21
readpage_async_filler
nfs 0xffffffffc08bcd0b
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xfb
22
read_cache_pages
vmlinux 0xffffffff92dcc280
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xa0
23
? nfs_return_empty_page
nfs 0xffffffffc08bcc10
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x110
24
nfs_readpages
nfs 0xffffffffc08bd555
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x155
25
__do_page_cache_readahead
vmlinux 0xffffffff92dcbe1f
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1cf
26
force_page_cache_readahead
vmlinux 0xffffffff92dcc369
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x99
27
SyS_fadvise64_64
vmlinux 0xffffffff92dc309c
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x1ec
28
SyS_fadvise64
vmlinux 0xffffffff92dc315e
Build id: 3.10.0-1160.53.1.el7.x86_64 		0xe
29
system_call_fastpath
vmlinux 0xffffffff93395f92
Build id: 3.10.0-1160.53.1.el7.x86_64 		0x25 
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 26s! [kworker/2:1H:429]
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 26s! [bareos-fd:24366]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter devlink ip6_tables iptable_filter nfsv3 nfs_acl nfs lockd grace fscache vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul vmw_balloon glue_helper ablk_helper cryptd joydev pcspkr sg vmw_vmci parport_pc parport i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt sd_mod fb_sys_fops ttm crc_t10dif crct10dif_generic ahci ata_piix
libahci nfit crct10dif_pclmul crct10dif_common drm libnvdimm libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 5 PID: 24366 Comm: bareos-fd Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9d12315f0000 ti: ffff9d1084b68000 task.ti: ffff9d1084b68000
RIP: 0010:[<ffffffff92d170da>]  [<ffffffff92d170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9d1084b6b7d8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9d123fc206a0
RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff9d1084b6b810 R08: ffff9d10f9392000 R09: ffffffff92f86bf9
R10: ffff9d123fd5f160 R11: fffff349401a3800 R12: 0000000000000296
R13: ffff9d1084b6b788 R14: 0000000001320122 R15: 00000000c9fcacd2
FS:  00007f134c5c9700(0000) GS:ffff9d123fd40000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f23c0f87a18 CR3: 00000002a2474000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff92dc78e0>] ? drain_pages+0xb0/0xb0
[<ffffffff92d17229>] on_each_cpu_mask+0x29/0x70
[<ffffffff92dc47c5>] drain_all_pages+0xb5/0xc0
[<ffffffff92dc964b>] __alloc_pages_nodemask+0x8cb/0xbe0
[<ffffffff92e193b8>] alloc_pages_current+0x98/0x110
[<ffffffff92e27aa3>] new_slab+0x393/0x4e0
[<ffffffff92e27fbc>] ___slab_alloc+0x3cc/0x520
[<ffffffffc08bc74a>] ? nfs_readhdr_alloc+0x1a/0x30 [nfs]
[<ffffffff92e27843>] ? new_slab+0x133/0x4e0
[<ffffffffc08bc74a>] ? nfs_readhdr_alloc+0x1a/0x30 [nfs]
[<ffffffff9337fea5>] __slab_alloc+0x40/0x5c
[<ffffffff92e2917b>] kmem_cache_alloc+0x19b/0x1f0
[<ffffffffc08bc74a>] ? nfs_readhdr_alloc+0x1a/0x30 [nfs]
[<ffffffffc08bc74a>] nfs_readhdr_alloc+0x1a/0x30 [nfs]
[<ffffffffc08bb392>] nfs_generic_pg_pgios+0x22/0xe0 [nfs]
[<ffffffffc08bacbd>] nfs_pageio_doio+0x2d/0x70 [nfs]
[<ffffffffc08bb7aa>] __nfs_pageio_add_request+0xba/0x450 [nfs]
[<ffffffffc08b0fc1>] ? nfs_get_lock_context+0x51/0x100 [nfs]
[<ffffffffc08bc266>] nfs_pageio_add_request+0x176/0x330 [nfs]
[<ffffffffc08bcd0b>] readpage_async_filler+0xfb/0x1d0 [nfs]
[<ffffffff92dcc280>] read_cache_pages+0xa0/0xf0
[<ffffffffc08bcc10>] ? nfs_return_empty_page+0x110/0x110 [nfs]
[<ffffffffc08bd555>] nfs_readpages+0x155/0x1f0 [nfs]
[<ffffffff92dcbe1f>] __do_page_cache_readahead+0x1cf/0x260
[<ffffffff92dcc369>] force_page_cache_readahead+0x99/0xe0
[<ffffffff92dc309c>] SyS_fadvise64_64+0x1ec/0x2a0
[<ffffffff92dc315e>] SyS_fadvise64+0xe/0x10
[<ffffffff93395f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 95 93 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89