Info

Permalink
Function
smp_call_function_many
First occurrence
2021-02-11
Last occurrence
2026-03-24
State
NEW
Components
kernel
Quality
-38

Statistics

Operating system Count
CentOS 7 27
Architecture Count
x86_64 27
Related packages Count
kernel 27
 0:3.10.0-957.el7 9
 0:3.10.0-1062.1.2.el7 3
 0:3.10.0-693.el7 3
 0:3.10.0-1160.88.1.el7 3
 0:3.10.0-1160.53.1.el7 2
 0:3.10.0-862.el7 2
 0:3.10.0-1127.el7 1
 0:3.10.0-1160.25.1.el7 1
 0:3.10.0-1160.76.1.el7 1
 0:3.10.0-1160.el7 1
 0:3.10.0-693.11.1.el7 1

Packages names with count less than the total count of reports are most probably not the cause of the problem.

History

Daily:

Weekly:

Monthly:

Report backtrace


Complete report #1405599
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff81b1173e
Build id: 3.10.0-957.el7.x86_64
0x20e
2
? drain_pages
vmlinux 0xffffffff81bbf550
Build id: 3.10.0-957.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff81b11889
Build id: 3.10.0-957.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff81bbc435
Build id: 3.10.0-957.el7.x86_64
0xb5
5
__alloc_pages_slowpath
vmlinux 0xffffffff8215d228
Build id: 3.10.0-957.el7.x86_64
0x510
6
__alloc_pages_nodemask
vmlinux 0xffffffff81bc0e95
Build id: 3.10.0-957.el7.x86_64
0x405
7
new_slab
vmlinux 0xffffffff81c18ad1
Build id: 3.10.0-957.el7.x86_64
0x91
8
___slab_alloc
vmlinux 0xffffffff81c1a77c
Build id: 3.10.0-957.el7.x86_64
0x3ac
9
? alloc_rt_sched_group
vmlinux 0xffffffff81ae87bb
Build id: 3.10.0-957.el7.x86_64
0x10b
10
? alloc_rt_sched_group
vmlinux 0xffffffff81ae87bb
Build id: 3.10.0-957.el7.x86_64
0x10b
11
__slab_alloc
vmlinux 0xffffffff8215e58c
Build id: 3.10.0-957.el7.x86_64
0x40
12
kmem_cache_alloc_node_trace
vmlinux 0xffffffff81c1d04a
Build id: 3.10.0-957.el7.x86_64
0x9a
13
alloc_rt_sched_group
vmlinux 0xffffffff81ae87bb
Build id: 3.10.0-957.el7.x86_64
0x10b
14
sched_create_group
vmlinux 0xffffffff81ad919b
Build id: 3.10.0-957.el7.x86_64
0x6b
15
sched_autogroup_create_attach
vmlinux 0xffffffff81aecccf
Build id: 3.10.0-957.el7.x86_64
0x3f
16
sys_setsid
vmlinux 0xffffffff81ab48f3
Build id: 3.10.0-957.el7.x86_64
0x73
17
system_call_fastpath
vmlinux 0xffffffff82174ddb
Build id: 3.10.0-957.el7.x86_64
0x22
18
? system_call_after_swapgs
vmlinux 0xffffffff82174d21
Build id: 3.10.0-957.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#29 stuck for 22s! [tnslsnr:625211]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge devlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock ppdev vmw_balloon pcspkr joydev sg i2c_piix4 vmw_vmci parport_pc parport nfsd auth_rpcgss nfs_acl lockd grace binfmt_misc fuse ip_tables xfs libcrc32c ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic crct10dif_common vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci 8021q garp mrp ata_piix libahci stp serio_raw llc libata nfit vmxnet3 vmw_pvscsi libnvdimm drm_panel_orientation_quirks scsi_transport_iscsi sunrpc dm_mirror dm_region_hash
dm_log dm_mod
CPU: 29 PID: 625211 Comm: tnslsnr Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9a1449e84100 ti: ffff99eedc858000 task.ti: ffff99eedc858000
RIP: 0010:[<ffffffff81b1173e>]  [<ffffffff81b1173e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff99eedc85bab8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000000000001b700 RCX: ffff9a05e0820a70
RDX: 0000000000000000 RSI: 0000000000000030 RDI: 0000000000000000
RBP: ffff99eedc85baf0 R08: ffff9a069b681c00 R09: ffffffff81d755c9
R10: ffff9a1e1e15f160 R11: fffff55ac1dfce00 R12: ffff99eedc85ba68
R13: 00000000000004e5 R14: 0000000001320122 R15: 00000000f192b54f
FS:  00007effc460d180(0000) GS:ffff9a1e1e140000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007effbe0efcd0 CR3: 0000000079914000 CR4: 00000000000007e0
Call Trace:
[<ffffffff81bbf550>] ? drain_pages+0xb0/0xb0
[<ffffffff81b11889>] on_each_cpu_mask+0x29/0x70
[<ffffffff81bbc435>] drain_all_pages+0xb5/0xc0
[<ffffffff8215d228>] __alloc_pages_slowpath+0x510/0x724
[<ffffffff81bc0e95>] __alloc_pages_nodemask+0x405/0x420
[<ffffffff81c18ad1>] new_slab+0x91/0x390
[<ffffffff81c1a77c>] ___slab_alloc+0x3ac/0x4f0
[<ffffffff81ae87bb>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff81ae87bb>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff8215e58c>] __slab_alloc+0x40/0x5c
[<ffffffff81c1d04a>] kmem_cache_alloc_node_trace+0x9a/0x210
[<ffffffff81ae87bb>] alloc_rt_sched_group+0x10b/0x240
[<ffffffff81ad919b>] sched_create_group+0x6b/0x80
[<ffffffff81aecccf>] sched_autogroup_create_attach+0x3f/0x1a0
[<ffffffff81ab48f3>] sys_setsid+0x73/0xc0
[<ffffffff82174ddb>] system_call_fastpath+0x22/0x27
[<ffffffff82174d21>] ? system_call_after_swapgs+0xae/0x146
Code: 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 74 82 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #317859
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99a2
Build id: 3.10.0-693.el7.x86_64
0x202
2
? drain_pages
vmlinux 0xffffffff8118b530
Build id: 3.10.0-693.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff810f9ae9
Build id: 3.10.0-693.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff81188475
Build id: 3.10.0-693.el7.x86_64
0xb5
5
__alloc_pages_slowpath
vmlinux 0xffffffff8169fbc4
Build id: 3.10.0-693.el7.x86_64
0x510
6
__alloc_pages_nodemask
vmlinux 0xffffffff8118cd85
Build id: 3.10.0-693.el7.x86_64
0x405
7
new_slab
vmlinux 0xffffffff811dbc11
Build id: 3.10.0-693.el7.x86_64
0x91
8
___slab_alloc
vmlinux 0xffffffff811dd70c
Build id: 3.10.0-693.el7.x86_64
0x3ac
9
? __sb_end_write
vmlinux 0xffffffff81203bd1
Build id: 3.10.0-693.el7.x86_64
0x31
10
? alloc_rt_sched_group
vmlinux 0xffffffff810d5c3b
Build id: 3.10.0-693.el7.x86_64
0x10b
11
? filp_close
vmlinux 0xffffffff811fe33b
Build id: 3.10.0-693.el7.x86_64
0x5b
12
? alloc_rt_sched_group
vmlinux 0xffffffff810d5c3b
Build id: 3.10.0-693.el7.x86_64
0x10b
13
__slab_alloc
vmlinux 0xffffffff816a10ce
Build id: 3.10.0-693.el7.x86_64
0x40
14
kmem_cache_alloc_node_trace
vmlinux 0xffffffff811e02ba
Build id: 3.10.0-693.el7.x86_64
0x9a
15
alloc_rt_sched_group
vmlinux 0xffffffff810d5c3b
Build id: 3.10.0-693.el7.x86_64
0x10b
16
sched_create_group
vmlinux 0xffffffff810c6adb
Build id: 3.10.0-693.el7.x86_64
0x6b
17
sched_autogroup_create_attach
vmlinux 0xffffffff810d8bff
Build id: 3.10.0-693.el7.x86_64
0x3f
18
sys_setsid
vmlinux 0xffffffff810a36a3
Build id: 3.10.0-693.el7.x86_64
0x73
19
system_call_fastpath
vmlinux 0xffffffff816b4fc9
Build id: 3.10.0-693.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 23s! [ora_psp0_orclcd:10905]
Modules linked in: vmsec_nfq(OE) vmsecmod(OE) nf_conntrack_ipv6 nf_defrag_ipv6 nf_conntrack_ipv4 nf_defrag_ipv4 nfnetlink_queue tun ip_set nfnetlink bridge stp llc nf_conntrack ppdev parport_pc sg parport virtio_balloon i2c_piix4 joydev pcspkr nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_console virtio_scsi virtio_net virtio_blk cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm virtio_pci libata serio_raw virtio_ring i2c_core virtio floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: vmsec_nfq]
CPU: 4 PID: 10905 Comm: ora_psp0_orclcd Tainted: G           OEL ------------   3.10.0-693.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff8806d1ed9fa0 ti: ffff88000e2c0000 task.ti: ffff88000e2c0000
RIP: 0010:[<ffffffff810f99a2>]  [<ffffffff810f99a2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff88000e2c3ae8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff88081da1ade0
RDX: 0000000000000000 RSI: 0000000000000070 RDI: 0000000000000000
RBP: ffff88000e2c3b20 R08: ffff880174b4cc00 R09: ffffffff813227d9
R10: ffff88081db19c80 R11: ffffea0013ede000 R12: 0000000000000292
R13: ffff88000e2c3a98 R14: 0000000001320122 R15: 00000000a67a2f29
FS:  00007fdb112853c0(0000) GS:ffff88081db00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007ff92473e478 CR3: 000000010515d000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
0000000100000040 ffffffff81f404c0 ffffffff81f404c0 ffffffff8118b530
0000000000000000 0000000000000004 ffff88083ffc3000 ffff88000e2c3b50
ffffffff810f9ae9 000000000000000f 0000000000000070 0000000000000070
Call Trace:
[<ffffffff8118b530>] ? drain_pages+0xb0/0xb0
[<ffffffff810f9ae9>] on_each_cpu_mask+0x29/0x60
[<ffffffff81188475>] drain_all_pages+0xb5/0xc0
[<ffffffff8169fbc4>] __alloc_pages_slowpath+0x510/0x724
[<ffffffff8118cd85>] __alloc_pages_nodemask+0x405/0x420
[<ffffffff811dbc11>] new_slab+0x91/0x310
[<ffffffff811dd70c>] ___slab_alloc+0x3ac/0x4f0
[<ffffffff81203bd1>] ? __sb_end_write+0x31/0x60
[<ffffffff810d5c3b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff811fe33b>] ? filp_close+0x5b/0x80
[<ffffffff810d5c3b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff816a10ce>] __slab_alloc+0x40/0x5c
[<ffffffff811e02ba>] kmem_cache_alloc_node_trace+0x9a/0x210
[<ffffffff810d5c3b>] alloc_rt_sched_group+0x10b/0x240
[<ffffffff810c6adb>] sched_create_group+0x6b/0x80
[<ffffffff810d8bff>] sched_autogroup_create_attach+0x3f/0x1a0
[<ffffffff810a36a3>] sys_setsid+0x73/0xc0
[<ffffffff816b4fc9>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 26 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 f5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1351703
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9c51e1ca
Build id: 3.10.0-1160.88.1.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffff9c5d2850
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff9c51e319
Build id: 3.10.0-1160.88.1.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff9c5cf655
Build id: 3.10.0-1160.88.1.el7.x86_64
0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffff9c5d463b
Build id: 3.10.0-1160.88.1.el7.x86_64
0x8db
6
new_slab
vmlinux 0xffffffff9c634266
Build id: 3.10.0-1160.88.1.el7.x86_64
0xa6
7
___slab_alloc
vmlinux 0xffffffff9c634a6c
Build id: 3.10.0-1160.88.1.el7.x86_64
0x3cc
8
? alloc_rt_sched_group
vmlinux 0xffffffff9c4f3c4b
Build id: 3.10.0-1160.88.1.el7.x86_64
0x10b
9
? do_read_fault.isra.63
vmlinux 0xffffffff9c5fa7f9
Build id: 3.10.0-1160.88.1.el7.x86_64
0x139
10
? alloc_rt_sched_group
vmlinux 0xffffffff9c4f3c4b
Build id: 3.10.0-1160.88.1.el7.x86_64
0x10b
11
__slab_alloc
vmlinux 0xffffffff9cbae458
Build id: 3.10.0-1160.88.1.el7.x86_64
0x40
12
kmem_cache_alloc_node_trace
vmlinux 0xffffffff9c63545a
Build id: 3.10.0-1160.88.1.el7.x86_64
0x9a
13
alloc_rt_sched_group
vmlinux 0xffffffff9c4f3c4b
Build id: 3.10.0-1160.88.1.el7.x86_64
0x10b
14
sched_create_group
vmlinux 0xffffffff9c4e357b
Build id: 3.10.0-1160.88.1.el7.x86_64
0x6b
15
sched_autogroup_create_attach
vmlinux 0xffffffff9c4f84af
Build id: 3.10.0-1160.88.1.el7.x86_64
0x3f
16
sys_setsid
vmlinux 0xffffffff9c4bdb63
Build id: 3.10.0-1160.88.1.el7.x86_64
0x73
17
system_call_fastpath
vmlinux 0xffffffff9cbc539a
Build id: 3.10.0-1160.88.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [tnslsnr:4785]
Modules linked in: vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 nfit libnvdimm coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev vmw_balloon aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom ata_generic vmwgfx drm_kms_helper pata_acpi syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix scsi_transport_spi mptscsih libata mptbase crct10dif_pclmul crct10dif_common vmxnet3 crc32c_intel serio_raw drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 2 PID: 4785 Comm: tnslsnr Kdump: loaded Tainted: G             L ------------   3.10.0-1160.88.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff91a9b55c3180 ti: ffff91a9a2584000 task.ti: ffff91a9a2584000
RIP: 0010:[<ffffffff9c51e1ca>]  [<ffffffff9c51e1ca>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff91a9a2587b20  EFLAGS: 00000202
RAX: 0000000000000006 RBX: 000c00009c5d2850 RCX: ffff91abbbd20650
RDX: 0000000000000006 RSI: 0000000000000010 RDI: 0000000000000000
RBP: ffff91a9a2587b58 R08: ffff91aab8c80c00 R09: ffffffff9c79c669
R10: ffff91aabbd1f160 R11: ffffedc2c0872800 R12: 0000000000000286
R13: 0000000000000286 R14: ffff91a9a2587ad0 R15: 0000000001320122
FS:  00007fbb4b10b740(0000) GS:ffff91aabbd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fbb471e9cd0 CR3: 0000000023168000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9c5d2850>] ? drain_pages+0xb0/0xb0
[<ffffffff9c51e319>] on_each_cpu_mask+0x29/0x70
[<ffffffff9c5cf655>] drain_all_pages+0xb5/0xd0
[<ffffffff9c5d463b>] __alloc_pages_nodemask+0x8db/0xbf0
[<ffffffff9c634266>] new_slab+0xa6/0x4e0
[<ffffffff9c634a6c>] ___slab_alloc+0x3cc/0x520
[<ffffffff9c4f3c4b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff9c5fa7f9>] ? do_read_fault.isra.63+0x139/0x1b0
[<ffffffff9c4f3c4b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff9cbae458>] __slab_alloc+0x40/0x60
[<ffffffff9c63545a>] kmem_cache_alloc_node_trace+0x9a/0x220
[<ffffffff9c4f3c4b>] alloc_rt_sched_group+0x10b/0x240
[<ffffffff9c4e357b>] sched_create_group+0x6b/0x80
[<ffffffff9c4f84af>] sched_autogroup_create_attach+0x3f/0x1a0
[<ffffffff9c4bdb63>] sys_setsid+0x73/0xc0
[<ffffffff9cbc539a>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 7e 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 15 9d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 4d 18 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1280960
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffb9b07e3a
Build id: 3.10.0-862.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffffb9b9cb10
Build id: 3.10.0-862.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffffb9b07f89
Build id: 3.10.0-862.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffffb9b999f5
Build id: 3.10.0-862.el7.x86_64
0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffffb9b9e860
Build id: 3.10.0-862.el7.x86_64
0x8a0
6
new_slab
vmlinux 0xffffffffb9bf3941
Build id: 3.10.0-862.el7.x86_64
0x91
7
___slab_alloc
vmlinux 0xffffffffb9bf543c
Build id: 3.10.0-862.el7.x86_64
0x3ac
8
? ___slab_alloc
vmlinux 0xffffffffb9bf5299
Build id: 3.10.0-862.el7.x86_64
0x209
9
? handle_pte_fault
vmlinux 0xffffffffb9bc52dc
Build id: 3.10.0-862.el7.x86_64
0x2dc
10
? alloc_rt_sched_group
vmlinux 0xffffffffb9ae076b
Build id: 3.10.0-862.el7.x86_64
0x10b
11
? alloc_rt_sched_group
vmlinux 0xffffffffb9ae0692
Build id: 3.10.0-862.el7.x86_64
0x32
12
? alloc_rt_sched_group
vmlinux 0xffffffffb9ae076b
Build id: 3.10.0-862.el7.x86_64
0x10b
13
__slab_alloc
vmlinux 0xffffffffba10a9b0
Build id: 3.10.0-862.el7.x86_64
0x40
14
kmem_cache_alloc_node_trace
vmlinux 0xffffffffb9bf7ffa
Build id: 3.10.0-862.el7.x86_64
0x9a
15
alloc_rt_sched_group
vmlinux 0xffffffffb9ae076b
Build id: 3.10.0-862.el7.x86_64
0x10b
16
sched_create_group
vmlinux 0xffffffffb9ad134b
Build id: 3.10.0-862.el7.x86_64
0x6b
17
sched_autogroup_create_attach
vmlinux 0xffffffffb9ae3abf
Build id: 3.10.0-862.el7.x86_64
0x3f
18
sys_setsid
vmlinux 0xffffffffb9aadc23
Build id: 3.10.0-862.el7.x86_64
0x73
19
system_call_fastpath
vmlinux 0xffffffffba11f7d5
Build id: 3.10.0-862.el7.x86_64
0x1c
20
? system_call_after_swapgs
vmlinux 0xffffffffba11f721
Build id: 3.10.0-862.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 21s! [setsid:17488]
Modules linked in: tcp_lp fuse binfmt_misc af_packet_diag netlink_diag vsock_diag vsock sctp_diag sctp tcp_diag udp_diag inet_diag unix_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter devlink sunrpc ext4 mbcache jbd2 ppdev sg virtio_balloon joydev i2c_piix4 pcspkr parport_pc parport sch_fq_codel ip_tables xfs libcrc32c sr_mod cdrom virtio_blk virtio_console virtio_scsi virtio_net ata_generic pata_acpi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata virtio_pci serio_raw floppy i2c_core virtio_ring virtio dm_mirror dm_region_hash
dm_log dm_mod
CPU: 2 PID: 17488 Comm: setsid Kdump: loaded Tainted: G             L ------------   3.10.0-862.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff929a346d4f10 ti: ffff92992346c000 task.ti: ffff92992346c000
RIP: 0010:[<ffffffffb9b07e3a>]  [<ffffffffb9b07e3a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff92992346fb20  EFLAGS: 00000202
RAX: 0000000000000008 RBX: 00080000b9b9cb10 RCX: ffff92a0de01c4d8
RDX: 0000000000000008 RSI: 0000000000000060 RDI: 0000000000000000
RBP: ffff92992346fb58 R08: ffff929a3443c400 R09: ffffffffb9d4a3d9
R10: ffff92a0dde9bb00 R11: fffffc9900f9a400 R12: 0000000000000286
R13: 0000000000000286 R14: ffff92992346fad0 R15: 0000000001320122
FS:  00007f5d983ee740(0000) GS:ffff92a0dde80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5d97ee87d0 CR3: 000000005e2b8000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffffb9b9cb10>] ? drain_pages+0xb0/0xb0
[<ffffffffb9b07f89>] on_each_cpu_mask+0x29/0x70
[<ffffffffb9b999f5>] drain_all_pages+0xb5/0xc0
[<ffffffffb9b9e860>] __alloc_pages_nodemask+0x8a0/0xbb0
[<ffffffffb9bf3941>] new_slab+0x91/0x310
[<ffffffffb9bf543c>] ___slab_alloc+0x3ac/0x4f0
[<ffffffffb9bf5299>] ? ___slab_alloc+0x209/0x4f0
[<ffffffffb9bc52dc>] ? handle_pte_fault+0x2dc/0xc30
[<ffffffffb9ae076b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffffb9ae0692>] ? alloc_rt_sched_group+0x32/0x240
[<ffffffffb9ae076b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffffba10a9b0>] __slab_alloc+0x40/0x5c
[<ffffffffb9bf7ffa>] kmem_cache_alloc_node_trace+0x9a/0x210
[<ffffffffb9ae076b>] alloc_rt_sched_group+0x10b/0x240
[<ffffffffb9ad134b>] sched_create_group+0x6b/0x80
[<ffffffffb9ae3abf>] sched_autogroup_create_attach+0x3f/0x1a0
[<ffffffffb9aadc23>] sys_setsid+0x73/0xc0
[<ffffffffba11f7d5>] system_call_fastpath+0x1c/0x21
[<ffffffffba11f721>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 8e b6 c3 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 53 73 ba f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 5d b6 c3 00 eb b7 0f b6 4d cc 4c 89 
Complete report #1026673
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff810f99b2
Build id: 3.10.0-693.11.1.el7.x86_64
0x202
2
? drain_pages
vmlinux 0xffffffff8118b540
Build id: 3.10.0-693.11.1.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff810f9af9
Build id: 3.10.0-693.11.1.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff81188485
Build id: 3.10.0-693.11.1.el7.x86_64
0xb5
5
__alloc_pages_slowpath
vmlinux 0xffffffff8169fc94
Build id: 3.10.0-693.11.1.el7.x86_64
0x510
6
__alloc_pages_nodemask
vmlinux 0xffffffff8118cdb5
Build id: 3.10.0-693.11.1.el7.x86_64
0x405
7
new_slab
vmlinux 0xffffffff811dbba1
Build id: 3.10.0-693.11.1.el7.x86_64
0x91
8
___slab_alloc
vmlinux 0xffffffff811dd69c
Build id: 3.10.0-693.11.1.el7.x86_64
0x3ac
9
? alloc_rt_sched_group
vmlinux 0xffffffff810d5c4b
Build id: 3.10.0-693.11.1.el7.x86_64
0x10b
10
? unlock_page
vmlinux 0xffffffff8118296b
Build id: 3.10.0-693.11.1.el7.x86_64
0x2b
11
? do_read_fault.isra.44
vmlinux 0xffffffff811ad616
Build id: 3.10.0-693.11.1.el7.x86_64
0xe6
12
? alloc_rt_sched_group
vmlinux 0xffffffff810d5c4b
Build id: 3.10.0-693.11.1.el7.x86_64
0x10b
13
__slab_alloc
vmlinux 0xffffffff816a119e
Build id: 3.10.0-693.11.1.el7.x86_64
0x40
14
kmem_cache_alloc_node_trace
vmlinux 0xffffffff811e024a
Build id: 3.10.0-693.11.1.el7.x86_64
0x9a
15
alloc_rt_sched_group
vmlinux 0xffffffff810d5c4b
Build id: 3.10.0-693.11.1.el7.x86_64
0x10b
16
sched_create_group
vmlinux 0xffffffff810c6aeb
Build id: 3.10.0-693.11.1.el7.x86_64
0x6b
17
sched_autogroup_create_attach
vmlinux 0xffffffff810d8c0f
Build id: 3.10.0-693.11.1.el7.x86_64
0x3f
18
sys_setsid
vmlinux 0xffffffff810a36b3
Build id: 3.10.0-693.11.1.el7.x86_64
0x73
19
system_call_fastpath
vmlinux 0xffffffff816b5089
Build id: 3.10.0-693.11.1.el7.x86_64
0x16
NMI watchdog: BUG: soft lockup - CPU#17 stuck for 23s! [sshd:15772]
Modules linked in: tcp_diag inet_diag vmw_vsock_vmci_transport vsock sb_edac edac_core iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper nfit ppdev vmw_balloon cryptd pcspkr sg vmw_vmci joydev i2c_piix4 shpchp parport_pc libnvdimm parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm mptspi ata_piix scsi_transport_spi crct10dif_pclmul crct10dif_common libata crc32c_intel mptscsih e1000 serio_raw mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 17 PID: 15772 Comm: sshd Tainted: G             L ------------   3.10.0-693.11.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff880163fb6eb0 ti: ffff88010b800000 task.ti: ffff88010b800000
RIP: 0010:[<ffffffff810f99b2>]  [<ffffffff810f99b2>] smp_call_function_many+0x202/0x260
RSP: 0018:ffff88010b803ae8  EFLAGS: 00000202
RAX: 000000000000000b RBX: 0000000000017800 RCX: ffff88081fcdb210
RDX: 000000000000000b RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff88010b803b20 R08: ffff8808b4def000 R09: ffffffff81322729
R10: ffff88101f259c80 R11: ffffea0021515a00 R12: ffff88010b803a98
R13: ffff88017fc03500 R14: 0000000001320122 R15: ffffffff811e024a
FS:  00007f5db32ae8c0(0000) GS:ffff88101f240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f5db06cb310 CR3: 0000000208081000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Stack:
0000000100000040 ffffffff81f404c0 ffffffff81f404c0 ffffffff8118b540
0000000000000000 0000000000000011 ffff88083ffda000 ffff88010b803b50
ffffffff810f9af9 000000000000001f 0000000000000020 0000000000000020
Call Trace:
[<ffffffff8118b540>] ? drain_pages+0xb0/0xb0
[<ffffffff810f9af9>] on_each_cpu_mask+0x29/0x60
[<ffffffff81188485>] drain_all_pages+0xb5/0xc0
[<ffffffff8169fc94>] __alloc_pages_slowpath+0x510/0x724
[<ffffffff8118cdb5>] __alloc_pages_nodemask+0x405/0x420
[<ffffffff811dbba1>] new_slab+0x91/0x310
[<ffffffff811dd69c>] ___slab_alloc+0x3ac/0x4f0
[<ffffffff810d5c4b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff8118296b>] ? unlock_page+0x2b/0x30
[<ffffffff811ad616>] ? do_read_fault.isra.44+0xe6/0x130
[<ffffffff810d5c4b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff816a119e>] __slab_alloc+0x40/0x5c
[<ffffffff811e024a>] kmem_cache_alloc_node_trace+0x9a/0x210
[<ffffffff810d5c4b>] alloc_rt_sched_group+0x10b/0x240
[<ffffffff810c6aeb>] sched_create_group+0x6b/0x80
[<ffffffff810d8c0f>] sched_autogroup_create_attach+0x3f/0x1a0
[<ffffffff810a36b3>] sys_setsid+0x73/0xc0
[<ffffffff816b5089>] system_call_fastpath+0x16/0x1b
Code: 48 63 35 16 45 a2 00 89 c2 39 f0 0f 8d 85 fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 fd b0 81 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 e5 44 a2 00 eb b7 0f b6 4d cc 4c 89 
Complete report #935762
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9ad170de
Build id: 3.10.0-1160.53.1.el7.x86_64
0x20e
2
? drain_pages
vmlinux 0xffffffff9adc78e0
Build id: 3.10.0-1160.53.1.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff9ad17229
Build id: 3.10.0-1160.53.1.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff9adc47c5
Build id: 3.10.0-1160.53.1.el7.x86_64
0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffff9adc964b
Build id: 3.10.0-1160.53.1.el7.x86_64
0x8cb
6
new_slab
vmlinux 0xffffffff9ae277b6
Build id: 3.10.0-1160.53.1.el7.x86_64
0xa6
7
___slab_alloc
vmlinux 0xffffffff9ae27fbc
Build id: 3.10.0-1160.53.1.el7.x86_64
0x3cc
8
? do_wp_page
vmlinux 0xffffffff9adf202b
Build id: 3.10.0-1160.53.1.el7.x86_64
0xfb
9
? alloc_rt_sched_group
vmlinux 0xffffffff9aced71b
Build id: 3.10.0-1160.53.1.el7.x86_64
0x10b
10
? kmem_cache_alloc
vmlinux 0xffffffff9ae29015
Build id: 3.10.0-1160.53.1.el7.x86_64
0x35
11
? alloc_rt_sched_group
vmlinux 0xffffffff9aced71b
Build id: 3.10.0-1160.53.1.el7.x86_64
0x10b
12
__slab_alloc
vmlinux 0xffffffff9b37fea5
Build id: 3.10.0-1160.53.1.el7.x86_64
0x40
13
kmem_cache_alloc_node_trace
vmlinux 0xffffffff9ae2898a
Build id: 3.10.0-1160.53.1.el7.x86_64
0x9a
14
alloc_rt_sched_group
vmlinux 0xffffffff9aced71b
Build id: 3.10.0-1160.53.1.el7.x86_64
0x10b
15
sched_create_group
vmlinux 0xffffffff9acdd5cb
Build id: 3.10.0-1160.53.1.el7.x86_64
0x6b
16
sched_autogroup_create_attach
vmlinux 0xffffffff9acf1d8f
Build id: 3.10.0-1160.53.1.el7.x86_64
0x3f
17
sys_setsid
vmlinux 0xffffffff9acb8963
Build id: 3.10.0-1160.53.1.el7.x86_64
0x73
18
system_call_fastpath
vmlinux 0xffffffff9b395f92
Build id: 3.10.0-1160.53.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 26s! [kworker/3:2H:1690]
NMI watchdog: BUG: soft lockup - CPU#4 stuck for 26s! [crond:16854]
Modules linked in: xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter tun bridge stp llc nfsv3 nfs_acl nfs lockd grace fscache vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev joydev pcspkr sg vmw_balloon parport_pc parport vmw_vmci i2c_piix4 auth_rpcgss sunrpc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper nfit syscopyarea sysfillrect sysimgblt libnvdimm fb_sys_fops ttm ahci sd_mod libahci ata_piix crc_t10dif crct10dif_generic
drm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks fuse
CPU: 4 PID: 16854 Comm: crond Kdump: loaded Tainted: G        W    L ------------   3.10.0-1160.53.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff97a875e48000 ti: ffff97a85d70c000 task.ti: ffff97a85d70c000
RIP: 0010:[<ffffffff9ad170de>]  [<ffffffff9ad170de>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff97a85d70fb20  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000c00009adc78e0 RCX: ffff97aaffc20678
RDX: 0000000000000000 RSI: 0000000000000006 RDI: 0000000000000000
RBP: ffff97a85d70fb58 R08: ffff97a9b9391c00 R09: ffffffff9af86bf9
R10: ffff97aaffd1f160 R11: ffffe144805f3500 R12: 0000000000000286
R13: 0000000000000286 R14: ffff97a85d70fad0 R15: 0000000001320122
FS:  00007f8191ff0840(0000) GS:ffff97aaffd00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000563d1f6032f0 CR3: 000000011c050000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff9adc78e0>] ? drain_pages+0xb0/0xb0
[<ffffffff9ad17229>] on_each_cpu_mask+0x29/0x70
[<ffffffff9adc47c5>] drain_all_pages+0xb5/0xc0
[<ffffffff9adc964b>] __alloc_pages_nodemask+0x8cb/0xbe0
[<ffffffff9ae277b6>] new_slab+0xa6/0x4e0
[<ffffffff9ae27fbc>] ___slab_alloc+0x3cc/0x520
[<ffffffff9adf202b>] ? do_wp_page+0xfb/0x720
[<ffffffff9aced71b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff9ae29015>] ? kmem_cache_alloc+0x35/0x1f0
[<ffffffff9aced71b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff9b37fea5>] __slab_alloc+0x40/0x5c
[<ffffffff9ae2898a>] kmem_cache_alloc_node_trace+0x9a/0x210
[<ffffffff9aced71b>] alloc_rt_sched_group+0x10b/0x240
[<ffffffff9acdd5cb>] sched_create_group+0x6b/0x80
[<ffffffff9acf1d8f>] sched_autogroup_create_attach+0x3f/0x1a0
[<ffffffff9acb8963>] sys_setsid+0x73/0xc0
[<ffffffff9b395f92>] system_call_fastpath+0x25/0x2a
Code: 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 95 9b f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #773027
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8b51173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffff8b5bf550
Build id: 3.10.0-957.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff8b511889
Build id: 3.10.0-957.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff8b5bc435
Build id: 3.10.0-957.el7.x86_64
0xb5
5
__alloc_pages_slowpath
vmlinux 0xffffffff8bb5d228
Build id: 3.10.0-957.el7.x86_64
0x510
6
__alloc_pages_nodemask
vmlinux 0xffffffff8b5c0e95
Build id: 3.10.0-957.el7.x86_64
0x405
7
new_slab
vmlinux 0xffffffff8b618ad1
Build id: 3.10.0-957.el7.x86_64
0x91
8
___slab_alloc
vmlinux 0xffffffff8b61a77c
Build id: 3.10.0-957.el7.x86_64
0x3ac
9
? __switch_to
vmlinux 0xffffffff8b42a621
Build id: 3.10.0-957.el7.x86_64
0x151
10
? alloc_rt_sched_group
vmlinux 0xffffffff8b4e87bb
Build id: 3.10.0-957.el7.x86_64
0x10b
11
? alloc_rt_sched_group
vmlinux 0xffffffff8b4e87bb
Build id: 3.10.0-957.el7.x86_64
0x10b
12
__slab_alloc
vmlinux 0xffffffff8bb5e58c
Build id: 3.10.0-957.el7.x86_64
0x40
13
kmem_cache_alloc_node_trace
vmlinux 0xffffffff8b61d04a
Build id: 3.10.0-957.el7.x86_64
0x9a
14
alloc_rt_sched_group
vmlinux 0xffffffff8b4e87bb
Build id: 3.10.0-957.el7.x86_64
0x10b
15
sched_create_group
vmlinux 0xffffffff8b4d919b
Build id: 3.10.0-957.el7.x86_64
0x6b
16
sched_autogroup_create_attach
vmlinux 0xffffffff8b4ecccf
Build id: 3.10.0-957.el7.x86_64
0x3f
17
sys_setsid
vmlinux 0xffffffff8b4b48f3
Build id: 3.10.0-957.el7.x86_64
0x73
18
system_call_fastpath
vmlinux 0xffffffff8bb74ddb
Build id: 3.10.0-957.el7.x86_64
0x22
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [oracle:127399]
Modules linked in: nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache tcp_diag inet_diag xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg parport_pc parport vmw_vmci i2c_piix4 binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi sd_mod crc_t10dif crct10dif_generic vmwgfx crct10dif_pclmul drm_kms_helper crct10dif_common crc32c_intel syscopyarea sysfillrect
sysimgblt fb_sys_fops ttm serio_raw ahci drm ata_piix libahci libata vmxnet3 vmw_pvscsi drm_panel_orientation_quirks nfit libnvdimm dm_mirror dm_region_hash dm_log dm_mod
CPU: 13 PID: 127399 Comm: oracle Kdump: loaded Tainted: G             L ------------   3.10.0-957.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff9befee01d140 ti: ffff9bf242ea8000 task.ti: ffff9bf242ea8000
RIP: 0010:[<ffffffff8b51173a>]  [<ffffffff8b51173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9bf242eabab8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff9c116f620790
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff9bf242eabaf0 R08: ffff9bef8358e800 R09: ffffffff8b7755c9
R10: ffff9c116f95f160 R11: fffffe70593ce400 R12: 0000000000000296
R13: ffff9bf242eaba68 R14: 0000000001320122 R15: 0000000035011e73
FS:  00007fcd37615740(0000) GS:ffff9c116f940000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000000228a000 CR3: 00000005e6ad8000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff8b5bf550>] ? drain_pages+0xb0/0xb0
[<ffffffff8b511889>] on_each_cpu_mask+0x29/0x70
[<ffffffff8b5bc435>] drain_all_pages+0xb5/0xc0
[<ffffffff8bb5d228>] __alloc_pages_slowpath+0x510/0x724
[<ffffffff8b5c0e95>] __alloc_pages_nodemask+0x405/0x420
[<ffffffff8b618ad1>] new_slab+0x91/0x390
[<ffffffff8b61a77c>] ___slab_alloc+0x3ac/0x4f0
[<ffffffff8b42a621>] ? __switch_to+0x151/0x580
[<ffffffff8b4e87bb>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff8b4e87bb>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff8bb5e58c>] __slab_alloc+0x40/0x5c
[<ffffffff8b61d04a>] kmem_cache_alloc_node_trace+0x9a/0x210
[<ffffffff8b4e87bb>] alloc_rt_sched_group+0x10b/0x240
[<ffffffff8b4d919b>] sched_create_group+0x6b/0x80
[<ffffffff8b4ecccf>] sched_autogroup_create_attach+0x3f/0x1a0
[<ffffffff8b4b48f3>] sys_setsid+0x73/0xc0
[<ffffffff8bb74ddb>] system_call_fastpath+0x22/0x27
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 14 8c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #602601
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff96f1173a
Build id: 3.10.0-957.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffff96fbf550
Build id: 3.10.0-957.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff96f11889
Build id: 3.10.0-957.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff96fbc435
Build id: 3.10.0-957.el7.x86_64
0xb5
5
__alloc_pages_slowpath
vmlinux 0xffffffff9755d228
Build id: 3.10.0-957.el7.x86_64
0x510
6
__alloc_pages_nodemask
vmlinux 0xffffffff96fc0e95
Build id: 3.10.0-957.el7.x86_64
0x405
7
new_slab
vmlinux 0xffffffff97018ad1
Build id: 3.10.0-957.el7.x86_64
0x91
8
___slab_alloc
vmlinux 0xffffffff9701a77c
Build id: 3.10.0-957.el7.x86_64
0x3ac
9
? handle_pte_fault
vmlinux 0xffffffff96fe9134
Build id: 3.10.0-957.el7.x86_64
0x2f4
10
? alloc_rt_sched_group
vmlinux 0xffffffff96ee87bb
Build id: 3.10.0-957.el7.x86_64
0x10b
11
? ___slab_alloc
vmlinux 0xffffffff9701a5d9
Build id: 3.10.0-957.el7.x86_64
0x209
12
? sched_create_group
vmlinux 0xffffffff96ed9155
Build id: 3.10.0-957.el7.x86_64
0x25
13
? alloc_rt_sched_group
vmlinux 0xffffffff96ee87bb
Build id: 3.10.0-957.el7.x86_64
0x10b
14
__slab_alloc
vmlinux 0xffffffff9755e58c
Build id: 3.10.0-957.el7.x86_64
0x40
15
kmem_cache_alloc_node_trace
vmlinux 0xffffffff9701d04a
Build id: 3.10.0-957.el7.x86_64
0x9a
16
alloc_rt_sched_group
vmlinux 0xffffffff96ee87bb
Build id: 3.10.0-957.el7.x86_64
0x10b
17
sched_create_group
vmlinux 0xffffffff96ed919b
Build id: 3.10.0-957.el7.x86_64
0x6b
18
sched_autogroup_create_attach
vmlinux 0xffffffff96eecccf
Build id: 3.10.0-957.el7.x86_64
0x3f
19
sys_setsid
vmlinux 0xffffffff96eb48f3
Build id: 3.10.0-957.el7.x86_64
0x73
20
system_call_fastpath
vmlinux 0xffffffff97574ddb
Build id: 3.10.0-957.el7.x86_64
0x22
21
? system_call_after_swapgs
vmlinux 0xffffffff97574d21
Build id: 3.10.0-957.el7.x86_64
0xae
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [tnslsnr:65036]
Modules linked in: netfilter_edr(OE) tun devlink ip_set nfnetlink bridge stp llc sunrpc ppdev pcspkr joydev parport_pc parport i2c_piix4 sg virtio_balloon binfmt_misc xfs libcrc32c sr_mod cdrom virtio_blk virtio_console virtio_scsi virtio_net ata_generic pata_acpi serio_raw floppy cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix drm_panel_orientation_quirks libata virtio_pci virtio_ring virtio dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 2 PID: 65036 Comm: tnslsnr Tainted: G           OEL ------------   3.10.0-957.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffffa09581e95140 ti: ffffa09416e58000 task.ti: ffffa09416e58000
RIP: 0010:[<ffffffff96f1173a>]  [<ffffffff96f1173a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffffa09416e5bab8  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00000000000000fc RCX: ffffa09c1dedfba8
RDX: 0000000000000003 RSI: 0000000000000060 RDI: 0000000000000000
RBP: ffffa09416e5baf0 R08: ffffa0957443c400 R09: ffffffff971755c9
R10: ffffa09c1de9f160 R11: ffffe43b133d6a00 R12: 0000000000000296
R13: ffffa09416e5ba68 R14: 0000000001320122 R15: 000000002a9da678
FS:  00007f254811f180(0000) GS:ffffa09c1de80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2541c00b30 CR3: 000000071bf08000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff96fbf550>] ? drain_pages+0xb0/0xb0
[<ffffffff96f11889>] on_each_cpu_mask+0x29/0x70
[<ffffffff96fbc435>] drain_all_pages+0xb5/0xc0
[<ffffffff9755d228>] __alloc_pages_slowpath+0x510/0x724
[<ffffffff96fc0e95>] __alloc_pages_nodemask+0x405/0x420
[<ffffffff97018ad1>] new_slab+0x91/0x390
[<ffffffff9701a77c>] ___slab_alloc+0x3ac/0x4f0
[<ffffffff96fe9134>] ? handle_pte_fault+0x2f4/0xd10
[<ffffffff96ee87bb>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff9701a5d9>] ? ___slab_alloc+0x209/0x4f0
[<ffffffff96ed9155>] ? sched_create_group+0x25/0x80
[<ffffffff96ee87bb>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff9755e58c>] __slab_alloc+0x40/0x5c
[<ffffffff9701d04a>] kmem_cache_alloc_node_trace+0x9a/0x210
[<ffffffff96ee87bb>] alloc_rt_sched_group+0x10b/0x240
[<ffffffff96ed919b>] sched_create_group+0x6b/0x80
[<ffffffff96eecccf>] sched_autogroup_create_attach+0x3f/0x1a0
[<ffffffff96eb48f3>] sys_setsid+0x73/0xc0
[<ffffffff97574ddb>] system_call_fastpath+0x22/0x27
[<ffffffff97574d21>] ? system_call_after_swapgs+0xae/0x146
Code: 48 63 35 4e 83 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 b9 b4 97 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 1d 83 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #589197
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffa8b16c0a
Build id: 3.10.0-1160.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffffa8bc74a0
Build id: 3.10.0-1160.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffffa8b16d59
Build id: 3.10.0-1160.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffffa8bc4385
Build id: 3.10.0-1160.el7.x86_64
0xb5
5
__alloc_pages_slowpath
vmlinux 0xffffffffa917c6b2
Build id: 3.10.0-1160.el7.x86_64
0x510
6
__alloc_pages_nodemask
vmlinux 0xffffffffa8bc8d76
Build id: 3.10.0-1160.el7.x86_64
0x436
7
new_slab
vmlinux 0xffffffffa8c26dd6
Build id: 3.10.0-1160.el7.x86_64
0xa6
8
___slab_alloc
vmlinux 0xffffffffa8c275dc
Build id: 3.10.0-1160.el7.x86_64
0x3cc
9
? do_wp_page
vmlinux 0xffffffffa8bf16bb
Build id: 3.10.0-1160.el7.x86_64
0xfb
10
? alloc_rt_sched_group
vmlinux 0xffffffffa8aed2db
Build id: 3.10.0-1160.el7.x86_64
0x10b
11
? do_read_fault.isra.63
vmlinux 0xffffffffa8bee4d9
Build id: 3.10.0-1160.el7.x86_64
0x139
12
? alloc_rt_sched_group
vmlinux 0xffffffffa8aed2db
Build id: 3.10.0-1160.el7.x86_64
0x10b
13
__slab_alloc
vmlinux 0xffffffffa917dc41
Build id: 3.10.0-1160.el7.x86_64
0x40
14
kmem_cache_alloc_node_trace
vmlinux 0xffffffffa8c27faa
Build id: 3.10.0-1160.el7.x86_64
0x9a
15
alloc_rt_sched_group
vmlinux 0xffffffffa8aed2db
Build id: 3.10.0-1160.el7.x86_64
0x10b
16
sched_create_group
vmlinux 0xffffffffa8add44b
Build id: 3.10.0-1160.el7.x86_64
0x6b
17
sched_autogroup_create_attach
vmlinux 0xffffffffa8af195f
Build id: 3.10.0-1160.el7.x86_64
0x3f
18
sys_setsid
vmlinux 0xffffffffa8ab8723
Build id: 3.10.0-1160.el7.x86_64
0x73
19
tracesys
vmlinux 0xffffffffa9194226
Build id: 3.10.0-1160.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [postgres:32394]
Modules linked in: tcp_lp veth xt_nat nf_conntrack_netlink xt_addrtype br_netfilter xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink overlay(T) ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul snd_ens1371 ghash_clmulni_intel snd_rawmidi ppdev snd_ac97_codec ac97_bus aesni_intel
snd_seq lrw gf128mul snd_seq_device glue_helper ablk_helper vmw_balloon snd_pcm cryptd snd_timer joydev pcspkr snd soundcore i2c_piix4 sg vmw_vmci parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic sr_mod cdrom vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt ata_generic pata_acpi crct10dif_pclmul mptspi fb_sys_fops crct10dif_common ttm scsi_transport_spi drm ata_piix e1000 mptscsih crc32c_intel serio_raw mptbase nfit libnvdimm libata drm_panel_orientation_quirks fuse
CPU: 1 PID: 32394 Comm: postgres Kdump: loaded Tainted: G             L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 07/22/2020
task: ffff88968482b180 ti: ffff88958ba40000 task.ti: ffff88958ba40000
RIP: 0010:[<ffffffffa8b16c0a>]  [<ffffffffa8b16c0a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff88958ba43ab8  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff8897af2a0600
RDX: 0000000000000002 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff88958ba43af0 R08: ffff8896fe34b000 R09: ffffffffa8d85f99
R10: ffff8897af25f160 R11: ffffe442c407fa00 R12: 0000000000000296
R13: ffff88958ba43a68 R14: ffffffffa8b32fe5 R15: ffffffffa8c27faa
FS:  00007fe9e187f700(0000) GS:ffff8897af240000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000ce9600 CR3: 0000000120764000 CR4: 00000000003607e0
Call Trace:
[<ffffffffa8bc74a0>] ? drain_pages+0xb0/0xb0
[<ffffffffa8b16d59>] on_each_cpu_mask+0x29/0x70
[<ffffffffa8bc4385>] drain_all_pages+0xb5/0xc0
[<ffffffffa917c6b2>] __alloc_pages_slowpath+0x510/0x729
[<ffffffffa8bc8d76>] __alloc_pages_nodemask+0x436/0x450
[<ffffffffa8c26dd6>] new_slab+0xa6/0x4e0
[<ffffffffa8c275dc>] ___slab_alloc+0x3cc/0x520
[<ffffffffa8bf16bb>] ? do_wp_page+0xfb/0x720
[<ffffffffa8aed2db>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffffa8bee4d9>] ? do_read_fault.isra.63+0x139/0x1b0
[<ffffffffa8aed2db>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffffa917dc41>] __slab_alloc+0x40/0x5c
[<ffffffffa8c27faa>] kmem_cache_alloc_node_trace+0x9a/0x210
[<ffffffffa8aed2db>] alloc_rt_sched_group+0x10b/0x240
[<ffffffffa8add44b>] sched_create_group+0x6b/0x80
[<ffffffffa8af195f>] sched_autogroup_create_attach+0x3f/0x1a0
[<ffffffffa8ab8723>] sys_setsid+0x73/0xc0
[<ffffffffa9194226>] tracesys+0xa6/0xcc
Code: 48 63 35 1e 89 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 13 75 a9 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 ed 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #581555
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffffab51718a
Build id: 3.10.0-1160.76.1.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffffab5c7920
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffffab5172d9
Build id: 3.10.0-1160.76.1.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffffab5c4805
Build id: 3.10.0-1160.76.1.el7.x86_64
0xb5
5
__alloc_pages_nodemask
vmlinux 0xffffffffab5c968b
Build id: 3.10.0-1160.76.1.el7.x86_64
0x8cb
6
new_slab
vmlinux 0xffffffffab6277b6
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa6
7
___slab_alloc
vmlinux 0xffffffffab627fbc
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3cc
8
? unlock_page
vmlinux 0xffffffffab5be47b
Build id: 3.10.0-1160.76.1.el7.x86_64
0x2b
9
? alloc_rt_sched_group
vmlinux 0xffffffffab4ed81b
Build id: 3.10.0-1160.76.1.el7.x86_64
0x10b
10
? handle_mm_fault
vmlinux 0xffffffffab5f65d0
Build id: 3.10.0-1160.76.1.el7.x86_64
0xa20
11
? alloc_rt_sched_group
vmlinux 0xffffffffab4ed81b
Build id: 3.10.0-1160.76.1.el7.x86_64
0x10b
12
__slab_alloc
vmlinux 0xffffffffabb82ef5
Build id: 3.10.0-1160.76.1.el7.x86_64
0x40
13
kmem_cache_alloc_node_trace
vmlinux 0xffffffffab62898a
Build id: 3.10.0-1160.76.1.el7.x86_64
0x9a
14
alloc_rt_sched_group
vmlinux 0xffffffffab4ed81b
Build id: 3.10.0-1160.76.1.el7.x86_64
0x10b
15
sched_create_group
vmlinux 0xffffffffab4dd66b
Build id: 3.10.0-1160.76.1.el7.x86_64
0x6b
16
sched_autogroup_create_attach
vmlinux 0xffffffffab4f1e8f
Build id: 3.10.0-1160.76.1.el7.x86_64
0x3f
17
sys_setsid
vmlinux 0xffffffffab4b8a93
Build id: 3.10.0-1160.76.1.el7.x86_64
0x73
18
system_call_fastpath
vmlinux 0xffffffffabb99f92
Build id: 3.10.0-1160.76.1.el7.x86_64
0x25
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [sshd:93997]
Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter dm_mirror dm_region_hash dm_log dm_mod ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg parport_pc parport joydev virtio_balloon bochs_drm ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm i2c_piix4 drm_panel_orientation_quirks pcspkr ip_tables ext4 mbcache jbd2 sr_mod
cdrom ata_generic pata_acpi virtio_net net_failover failover virtio_blk crct10dif_pclmul crct10dif_common crc32c_intel ata_piix serio_raw libata virtio_pci virtio_ring virtio floppy
CPU: 3 PID: 93997 Comm: sshd Tainted: G             L ------------   3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-0-ga698c8995f-prebuilt.qemu.org 04/01/2014
task: ffff8ad3c7518000 ti: ffff8ad3ca4c0000 task.ti: ffff8ad3ca4c0000
RIP: 0010:[<ffffffffab51718a>]  [<ffffffffab51718a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8ad3ca4c3b20  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 000c0000ab5c7920 RCX: ffff8ad3fe6a04d8
RDX: 0000000000000002 RSI: 0000000000000064 RDI: 0000000000000000
RBP: ffff8ad3ca4c3b58 R08: ffff8ad3f7f53800 R09: ffffffffab7869f9
R10: ffff8ad3fe6df160 R11: ffffca5140d30a00 R12: 0000000000000286
R13: 0000000000000286 R14: ffff8ad3ca4c3ad0 R15: 0000000001320122
FS:  00007efdbea428c0(0000) GS:ffff8ad3fe6c0000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00000000d4a4b003 CR3: 0000000005ba4000 CR4: 00000000001607e0
Call Trace:
[<ffffffffab5c7920>] ? drain_pages+0xb0/0xb0
[<ffffffffab5172d9>] on_each_cpu_mask+0x29/0x70
[<ffffffffab5c4805>] drain_all_pages+0xb5/0xc0
[<ffffffffab5c968b>] __alloc_pages_nodemask+0x8cb/0xbe0
[<ffffffffab6277b6>] new_slab+0xa6/0x4e0
[<ffffffffab627fbc>] ___slab_alloc+0x3cc/0x520
[<ffffffffab5be47b>] ? unlock_page+0x2b/0x30
[<ffffffffab4ed81b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffffab5f65d0>] ? handle_mm_fault+0xa20/0xfb0
[<ffffffffab4ed81b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffffabb82ef5>] __slab_alloc+0x40/0x5c
[<ffffffffab62898a>] kmem_cache_alloc_node_trace+0x9a/0x210
[<ffffffffab4ed81b>] alloc_rt_sched_group+0x10b/0x240
[<ffffffffab4dd66b>] sched_create_group+0x6b/0x80
[<ffffffffab4f1e8f>] sched_autogroup_create_attach+0x3f/0x1a0
[<ffffffffab4b8a93>] sys_setsid+0x73/0xc0
[<ffffffffabb99f92>] system_call_fastpath+0x25/0x2a
Code: 48 63 35 be 88 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 00 19 15 ac f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 8d 88 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #418784
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff86116eea
Build id: 3.10.0-1160.25.1.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffff861c76c0
Build id: 3.10.0-1160.25.1.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff86117039
Build id: 3.10.0-1160.25.1.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff861c45a5
Build id: 3.10.0-1160.25.1.el7.x86_64
0xb5
5
__alloc_pages_slowpath
vmlinux 0xffffffff8677e48c
Build id: 3.10.0-1160.25.1.el7.x86_64
0x510
6
__alloc_pages_nodemask
vmlinux 0xffffffff861c8f96
Build id: 3.10.0-1160.25.1.el7.x86_64
0x436
7
new_slab
vmlinux 0xffffffff86227086
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa6
8
___slab_alloc
vmlinux 0xffffffff8622788c
Build id: 3.10.0-1160.25.1.el7.x86_64
0x3cc
9
? complete_walk
vmlinux 0xffffffff862594e0
Build id: 3.10.0-1160.25.1.el7.x86_64
0x50
10
? alloc_rt_sched_group
vmlinux 0xffffffff860ed5cb
Build id: 3.10.0-1160.25.1.el7.x86_64
0x10b
11
? handle_mm_fault
vmlinux 0xffffffff861f5f82
Build id: 3.10.0-1160.25.1.el7.x86_64
0xb22
12
? alloc_rt_sched_group
vmlinux 0xffffffff860ed5cb
Build id: 3.10.0-1160.25.1.el7.x86_64
0x10b
13
__slab_alloc
vmlinux 0xffffffff8677fa1b
Build id: 3.10.0-1160.25.1.el7.x86_64
0x40
14
kmem_cache_alloc_node_trace
vmlinux 0xffffffff8622825a
Build id: 3.10.0-1160.25.1.el7.x86_64
0x9a
15
alloc_rt_sched_group
vmlinux 0xffffffff860ed5cb
Build id: 3.10.0-1160.25.1.el7.x86_64
0x10b
16
sched_create_group
vmlinux 0xffffffff860dd74b
Build id: 3.10.0-1160.25.1.el7.x86_64
0x6b
17
sched_autogroup_create_attach
vmlinux 0xffffffff860f1c3f
Build id: 3.10.0-1160.25.1.el7.x86_64
0x3f
18
sys_setsid
vmlinux 0xffffffff860b88a3
Build id: 3.10.0-1160.25.1.el7.x86_64
0x73
19
system_call_fastpath
vmlinux 0xffffffff86795f92
Build id: 3.10.0-1160.25.1.el7.x86_64
0x25
20
? system_call_after_swapgs
vmlinux 0xffffffff86795ed5
Build id: 3.10.0-1160.25.1.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#13 stuck for 22s! [crond:122066]
Modules linked in: ip6table_nat ip6_tables rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache nfsd auth_rpcgss nfs_acl lockd grace sunrpc xt_physdev iptable_mangle xt_NFLOG nfnetlink_log xt_state vport_vxlan vxlan ip6_udp_tunnel udp_tunnel openvswitch nf_conntrack_ipv6 nf_nat_ipv6 nf_defrag_ipv6 ip_set_hash_net nf_tables ip_set_hash_ip xt_set ip_set_hash_ipportip ip_set_bitmap_port ip_set_hash_ipportnet ip_set_hash_ipport dummy ipt_REJECT nf_reject_ipv4 ip_set xt_comment xt_mark ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs binfmt_misc tcp_diag inet_diag xt_nat veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc ipmi_devintf ipmi_msghandler
overlay(T) uio gsch(OE) redirfs(OE) ext4 mbcache jbd2 ppdev joydev virtio_balloon sg i2c_piix4 pcspkr parport_pc parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover virtio_console failover virtio_blk virtio_scsi cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm virtio_pci libata virtio_ring floppy serio_raw virtio drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 13 PID: 122066 Comm: crond Kdump: loaded Tainted: G           OEL ------------ T 3.10.0-1160.25.1.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.1-0-g0551a4be2c-prebuilt.qemu-project.org 04/01/2014
task: ffff9e9b9b515280 ti: ffff9e9b997b0000 task.ti: ffff9e9b997b0000
RIP: 0010:[<ffffffff86116eea>]  [<ffffffff86116eea>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff9e9b997b3ab8  EFLAGS: 00000202
RAX: 0000000000000002 RBX: 00000000000000fc RCX: ffff9ea39cea05e0
RDX: 0000000000000002 RSI: 00000000000000a0 RDI: 0000000000000000
RBP: ffff9e9b997b3af0 R08: ffff9e9cf2356000 R09: ffffffff86386549
R10: ffff9ea39d15f160 R11: ffffd836c0a66800 R12: 0000000000000296
R13: 0000000000000018 R14: ffff9e9b997b3a28 R15: 0000000000000296
FS:  00007f7c4e0f1840(0000) GS:ffff9ea39d140000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055ed0f9872f0 CR3: 0000000009980000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff861c76c0>] ? drain_pages+0xb0/0xb0
[<ffffffff86117039>] on_each_cpu_mask+0x29/0x70
[<ffffffff861c45a5>] drain_all_pages+0xb5/0xc0
[<ffffffff8677e48c>] __alloc_pages_slowpath+0x510/0x729
[<ffffffff861c8f96>] __alloc_pages_nodemask+0x436/0x450
[<ffffffff86227086>] new_slab+0xa6/0x4e0
[<ffffffff8622788c>] ___slab_alloc+0x3cc/0x520
[<ffffffff862594e0>] ? complete_walk+0x50/0xe0
[<ffffffff860ed5cb>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff861f5f82>] ? handle_mm_fault+0xb22/0xfb0
[<ffffffff860ed5cb>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff8677fa1b>] __slab_alloc+0x40/0x5c
[<ffffffff8622825a>] kmem_cache_alloc_node_trace+0x9a/0x210
[<ffffffff860ed5cb>] alloc_rt_sched_group+0x10b/0x240
[<ffffffff860dd74b>] sched_create_group+0x6b/0x80
[<ffffffff860f1c3f>] sched_autogroup_create_attach+0x3f/0x1a0
[<ffffffff860b88a3>] sys_setsid+0x73/0xc0
[<ffffffff86795f92>] system_call_fastpath+0x25/0x2a
[<ffffffff86795ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 48 63 35 fe 87 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 a0 15 d5 86 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 cd 87 c4 00 eb b7 0f b6 4d cc 4c 89 
Complete report #179689
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff8cf1760e
Build id: 3.10.0-1127.el7.x86_64
0x20e
2
? drain_pages
vmlinux 0xffffffff8cfc7850
Build id: 3.10.0-1127.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff8cf17759
Build id: 3.10.0-1127.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff8cfc46c5
Build id: 3.10.0-1127.el7.x86_64
0xb5
5
__alloc_pages_slowpath
vmlinux 0xffffffff8d57b2f5
Build id: 3.10.0-1127.el7.x86_64
0x510
6
__alloc_pages_nodemask
vmlinux 0xffffffff8cfc9146
Build id: 3.10.0-1127.el7.x86_64
0x436
7
new_slab
vmlinux 0xffffffff8d026e06
Build id: 3.10.0-1127.el7.x86_64
0xa6
8
___slab_alloc
vmlinux 0xffffffff8d02760c
Build id: 3.10.0-1127.el7.x86_64
0x3cc
9
? ___slab_alloc
vmlinux 0xffffffff8d027469
Build id: 3.10.0-1127.el7.x86_64
0x229
10
? alloc_rt_sched_group
vmlinux 0xffffffff8ceedd6b
Build id: 3.10.0-1127.el7.x86_64
0x10b
11
? alloc_rt_sched_group
vmlinux 0xffffffff8ceedc92
Build id: 3.10.0-1127.el7.x86_64
0x32
12
? handle_mm_fault
vmlinux 0xffffffff8cff5d00
Build id: 3.10.0-1127.el7.x86_64
0xa20
13
? sched_create_group
vmlinux 0xffffffff8cedde55
Build id: 3.10.0-1127.el7.x86_64
0x25
14
? alloc_rt_sched_group
vmlinux 0xffffffff8ceedd6b
Build id: 3.10.0-1127.el7.x86_64
0x10b
15
__slab_alloc
vmlinux 0xffffffff8d57c884
Build id: 3.10.0-1127.el7.x86_64
0x40
16
kmem_cache_alloc_node_trace
vmlinux 0xffffffff8d027fda
Build id: 3.10.0-1127.el7.x86_64
0x9a
17
alloc_rt_sched_group
vmlinux 0xffffffff8ceedd6b
Build id: 3.10.0-1127.el7.x86_64
0x10b
18
sched_create_group
vmlinux 0xffffffff8cedde9b
Build id: 3.10.0-1127.el7.x86_64
0x6b
19
sched_autogroup_create_attach
vmlinux 0xffffffff8cef23ef
Build id: 3.10.0-1127.el7.x86_64
0x3f
20
sys_setsid
vmlinux 0xffffffff8ceb9193
Build id: 3.10.0-1127.el7.x86_64
0x73
21
system_call_fastpath
vmlinux 0xffffffff8d592ed2
Build id: 3.10.0-1127.el7.x86_64
0x25
22
? system_call_after_swapgs
vmlinux 0xffffffff8d592e15
Build id: 3.10.0-1127.el7.x86_64
0xa2
NMI watchdog: BUG: soft lockup - CPU#2 stuck for 22s! [sshd:3690]
Modules linked in: fuse nls_utf8 isofs ext4 mbcache jbd2 nfnetlink_queue nfnetlink_log nfnetlink bluetooth rfkill tcp_diag inet_diag binfmt_misc xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter sunrpc ppdev pcspkr joydev virtio_balloon sg parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi virtio_net net_failover virtio_scsi virtio_blk virtio_console failover cirrus drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata serio_raw virtio_pci virtio_ring floppy virtio drm_panel_orientation_quirks dm_mirror
dm_region_hash dm_log dm_mod
CPU: 2 PID: 3690 Comm: sshd Kdump: loaded Tainted: G             L ------------   3.10.0-1127.el7.x86_64 #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014
task: ffff90d5ef2241c0 ti: ffff90d402f08000 task.ti: ffff90d402f08000
RIP: 0010:[<ffffffff8cf1760e>]  [<ffffffff8cf1760e>] smp_call_function_many+0x20e/0x270
RSP: 0018:ffff90d402f0bab8  EFLAGS: 00000202
RAX: 0000000000000003 RBX: 00000000000000fc RCX: ffff90d5f62e0428
RDX: 0000000000000003 RSI: 0000000000000050 RDI: 0000000000000000
RBP: ffff90d402f0baf0 R08: ffff90d53cf47400 R09: ffffffff8d185099
R10: ffff90d5f629f160 R11: ffffe30a88c41c00 R12: 0000000000000296
R13: ffff90d402f0ba68 R14: 0000000001320122 R15: 000000009fea5e2a
FS:  00007f91e989f8c0(0000) GS:ffff90d5f6280000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f91e6a65b30 CR3: 00000000186d8000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff8cfc7850>] ? drain_pages+0xb0/0xb0
[<ffffffff8cf17759>] on_each_cpu_mask+0x29/0x70
[<ffffffff8cfc46c5>] drain_all_pages+0xb5/0xc0
[<ffffffff8d57b2f5>] __alloc_pages_slowpath+0x510/0x729
[<ffffffff8cfc9146>] __alloc_pages_nodemask+0x436/0x450
[<ffffffff8d026e06>] new_slab+0xa6/0x4e0
[<ffffffff8d02760c>] ___slab_alloc+0x3cc/0x520
[<ffffffff8d027469>] ? ___slab_alloc+0x229/0x520
[<ffffffff8ceedd6b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff8ceedc92>] ? alloc_rt_sched_group+0x32/0x240
[<ffffffff8cff5d00>] ? handle_mm_fault+0xa20/0xfb0
[<ffffffff8cedde55>] ? sched_create_group+0x25/0x80
[<ffffffff8ceedd6b>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff8d57c884>] __slab_alloc+0x40/0x5c
[<ffffffff8d027fda>] kmem_cache_alloc_node_trace+0x9a/0x210
[<ffffffff8ceedd6b>] alloc_rt_sched_group+0x10b/0x240
[<ffffffff8cedde9b>] sched_create_group+0x6b/0x80
[<ffffffff8cef23ef>] sched_autogroup_create_attach+0x3f/0x1a0
[<ffffffff8ceb9193>] sys_setsid+0x73/0xc0
[<ffffffff8d592ed2>] system_call_fastpath+0x25/0x2a
[<ffffffff8d592e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 7b c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 0f b5 8d f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 f6 41 20 01 <75> f8 48 63 35 ed 7a c4 00 eb b7 0f b6 4d cc 4c 89 f2 4c 89 ee 
Complete report #65646
Frame # Function Binary Source or offset Line
1
smp_call_function_many
vmlinux 0xffffffff9e9150ca
Build id: 3.10.0-1062.1.2.el7.x86_64
0x20a
2
? drain_pages
vmlinux 0xffffffff9e9c5210
Build id: 3.10.0-1062.1.2.el7.x86_64
0xb0
3
on_each_cpu_mask
vmlinux 0xffffffff9e915219
Build id: 3.10.0-1062.1.2.el7.x86_64
0x29
4
drain_all_pages
vmlinux 0xffffffff9e9c1fd5
Build id: 3.10.0-1062.1.2.el7.x86_64
0xb5
5
__alloc_pages_slowpath
vmlinux 0xffffffff9ef746b6
Build id: 3.10.0-1062.1.2.el7.x86_64
0x510
6
__alloc_pages_nodemask
vmlinux 0xffffffff9e9c6b84
Build id: 3.10.0-1062.1.2.el7.x86_64
0x404
7
new_slab
vmlinux 0xffffffff9ea1fc71
Build id: 3.10.0-1062.1.2.el7.x86_64
0x91
8
___slab_alloc
vmlinux 0xffffffff9ea2170c
Build id: 3.10.0-1062.1.2.el7.x86_64
0x3ac
9
? alloc_rt_sched_group
vmlinux 0xffffffff9e8ebfbb
Build id: 3.10.0-1062.1.2.el7.x86_64
0x10b
10
? __schedule
vmlinux 0xffffffff9ef7ec88
Build id: 3.10.0-1062.1.2.el7.x86_64
0x448
11
? alloc_rt_sched_group
vmlinux 0xffffffff9e8ebfbb
Build id: 3.10.0-1062.1.2.el7.x86_64
0x10b
12
__slab_alloc
vmlinux 0xffffffff9ef75a1e
Build id: 3.10.0-1062.1.2.el7.x86_64
0x40
13
kmem_cache_alloc_node_trace
vmlinux 0xffffffff9ea241ea
Build id: 3.10.0-1062.1.2.el7.x86_64
0x9a
14
alloc_rt_sched_group
vmlinux 0xffffffff9e8ebfbb
Build id: 3.10.0-1062.1.2.el7.x86_64
0x10b
15
sched_create_group
vmlinux 0xffffffff9e8dc6eb
Build id: 3.10.0-1062.1.2.el7.x86_64
0x6b
16
sched_autogroup_create_attach
vmlinux 0xffffffff9e8f04ff
Build id: 3.10.0-1062.1.2.el7.x86_64
0x3f
17
sys_setsid
vmlinux 0xffffffff9e8b7bd3
Build id: 3.10.0-1062.1.2.el7.x86_64
0x73
18
tracesys
vmlinux 0xffffffff9ef8d172
Build id: 3.10.0-1062.1.2.el7.x86_64
0xa6
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 23s! [postgres:58970]
Modules linked in: btrfs raid6_pq xor msdos fat ext4 mbcache jbd2 udp_diag tcp_diag inet_diag binfmt_misc tcp_lp fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun bridge stp llc devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ppdev vmw_balloon sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg i2c_piix4 vmw_vmci parport_pc
parport ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm crct10dif_pclmul crct10dif_common mptspi crc32c_intel scsi_transport_spi libata mptscsih serio_raw mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 58970 Comm: postgres Kdump: loaded Tainted: G        W    L ------------   3.10.0-1062.1.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff98750b418000 ti: ffff98759a01c000 task.ti: ffff98759a01c000
RIP: 0010:[<ffffffff9e9150ca>]  [<ffffffff9e9150ca>] smp_call_function_many+0x20a/0x270
RSP: 0000:ffff98759a01fab8  EFLAGS: 00000202
RAX: 0000000000000000 RBX: 00000000000000fc RCX: ffff987d1d620490
RDX: 0000000000000000 RSI: 0000000000000080 RDI: 0000000000000000
RBP: ffff98759a01faf0 R08: ffff98767451bc00 R09: ffffffff9eb7f629
R10: ffff987d1d65f160 R11: fffff8e61350a600 R12: 0000000000000296
R13: ffff98759a01fa68 R14: 0000000001320122 R15: ffffffff9ea241ea
FS:  00007f1f5d4ec8c0(0000) GS:ffff987d1d640000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f0c4d9fc000 CR3: 00000000531f8000 CR4: 00000000003607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff9e9c5210>] ? drain_pages+0xb0/0xb0
[<ffffffff9e915219>] on_each_cpu_mask+0x29/0x70
[<ffffffff9e9c1fd5>] drain_all_pages+0xb5/0xc0
[<ffffffff9ef746b6>] __alloc_pages_slowpath+0x510/0x724
[<ffffffff9e9c6b84>] __alloc_pages_nodemask+0x404/0x420
[<ffffffff9ea1fc71>] new_slab+0x91/0x390
[<ffffffff9ea2170c>] ___slab_alloc+0x3ac/0x4f0
[<ffffffff9e8ebfbb>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff9ef7ec88>] ? __schedule+0x448/0x9c0
[<ffffffff9e8ebfbb>] ? alloc_rt_sched_group+0x10b/0x240
[<ffffffff9ef75a1e>] __slab_alloc+0x40/0x5c
[<ffffffff9ea241ea>] kmem_cache_alloc_node_trace+0x9a/0x210
[<ffffffff9e8ebfbb>] alloc_rt_sched_group+0x10b/0x240
[<ffffffff9e8dc6eb>] sched_create_group+0x6b/0x80
[<ffffffff9e8f04ff>] sched_autogroup_create_attach+0x3f/0x1a0
[<ffffffff9e8b7bd3>] sys_setsid+0x73/0xc0
[<ffffffff9ef8d172>] tracesys+0xa6/0xcc
Code: 48 63 35 3e 68 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 e0 d7 54 9f f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 0d 68 c4 00 eb b7 0f b6 4d cc 4c 89