Complete report #1179343
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
free_pages_and_swap_cache
|
vmlinux |
0xffffffff91e09693
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x63
|
|
2
|
tlb_flush_mmu_free
|
vmlinux |
0xffffffff91def864
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x34
|
|
3
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff91def8e7
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x47
|
|
4
|
tlb_finish_mmu
|
vmlinux |
0xffffffff91defa03
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x23
|
|
5
|
unmap_region
|
vmlinux |
0xffffffff91df9214
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0xf4
|
|
6
|
do_munmap
|
vmlinux |
0xffffffff91dfb805
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x2a5
|
|
7
|
vm_munmap
|
vmlinux |
0xffffffff91dfba45
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x65
|
|
8
|
SyS_munmap
|
vmlinux |
0xffffffff91dfcce2
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x22
|
|
9
|
system_call_fastpath
|
vmlinux |
0xffffffff92392ed2
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [httpd:18112]
Modules linked in: fuse nls_utf8 cifs ccm dns_resolver binfmt_misc ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 vmw_vsock_vmci_transport vsock xt_conntrack nf_conntrack iptable_filter iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev parport_pc vmw_balloon pcspkr joydev parport sg vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix nfit crct10dif_pclmul crct10dif_common crc32c_intel libata mptspi libnvdimm scsi_transport_spi serio_raw mptscsih vmxnet3
mptbase drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 18112 Comm: httpd Kdump: loaded Tainted: G L ------------ 3.10.0-1127.10.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8dbd52f5e2a0 ti: ffff8dbb9bfa0000 task.ti: ffff8dbb9bfa0000
RIP: 0010:[<ffffffff91e09693>] [<ffffffff91e09693>] free_pages_and_swap_cache+0x63/0xd0
RSP: 0018:ffff8dbb9bfa3d30 EFLAGS: 00010246
RAX: 002fffff00080068 RBX: ffff8dbb00000000 RCX: 0000000000000030
RDX: fffff7abc47de740 RSI: ffff8dbdffc1fd38 RDI: 0000000000000297
RBP: ffff8dbb9bfa3d58 R08: fffff7abc5359860 R09: 0000000000000000
R10: ffff8dbdffbd4380 R11: 0000000000000002 R12: 0000000300000000
R13: ffffffff91dcd00e R14: ffff8dbb9bfa3d20 R15: ffff8dbdfffd8000
FS: 00007f36993bf880(0000) GS:ffff8dbdffc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f36993cd000 CR3: 00000003578a8000 CR4: 00000000007607f0
PKRU: 55555554
Call Trace:
[<ffffffff91def864>] tlb_flush_mmu_free+0x34/0x50
[<ffffffff91def8e7>] arch_tlb_finish_mmu+0x47/0x80
[<ffffffff91defa03>] tlb_finish_mmu+0x23/0x30
[<ffffffff91df9214>] unmap_region+0xf4/0x140
[<ffffffff91dfb805>] do_munmap+0x2a5/0x480
[<ffffffff91dfba45>] vm_munmap+0x65/0xb0
[<ffffffff91dfcce2>] SyS_munmap+0x22/0x30
[<ffffffff92392ed2>] system_call_fastpath+0x25/0x2a
Code: 00 00 45 0f 4e ef 31 db 45 85 ed 7f 14 eb 59 66 0f 1f 84 00 00 00 00 00 48 83 c3 01 41 39 dd 7e 47 4d 8b 24 de 41 f6 c4 01 75 ed <49> 8b 04 24 a9 00 00 01 00 74 e2 41 8b 44 24 18 85 c0 79 d9 f0
Complete report #602188
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_unlock_irqrestore
|
vmlinux |
0xffffffff9f7883c5
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x15
|
|
2
|
release_pages
|
vmlinux |
0xffffffff9f1ccff5
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x235
|
|
3
|
free_pages_and_swap_cache
|
vmlinux |
0xffffffff9f2096dd
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0xad
|
|
4
|
tlb_flush_mmu_free
|
vmlinux |
0xffffffff9f1ef864
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x34
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff9f1ef8e7
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x47
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff9f1efa03
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x23
|
|
7
|
unmap_region
|
vmlinux |
0xffffffff9f1f9214
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0xf4
|
|
8
|
do_munmap
|
vmlinux |
0xffffffff9f1fb805
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x2a5
|
|
9
|
vm_munmap
|
vmlinux |
0xffffffff9f1fba45
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x65
|
|
10
|
SyS_munmap
|
vmlinux |
0xffffffff9f1fcce2
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x22
|
|
11
|
system_call_fastpath
|
vmlinux |
0xffffffff9f792ed2
Build id: 3.10.0-1127.10.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 24s! [postmaster:7825]
ata1: lost interrupt (Status 0x58)
Modules linked in: vmw_vsock_vmci_transport vsock sb_edac coretemp iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon joydev pcspkr sg parport_pc i2c_piix4 vmw_vmci parport ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod crc_t10dif crct10dif_generic syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix crct10dif_pclmul crct10dif_common libata crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 3 PID: 7825 Comm: postmaster Tainted: G L ------------ 3.10.0-1127.10.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff9681345fd230 ti: ffff967fe2330000 task.ti: ffff967fe2330000
RIP: 0010:[<ffffffff9f7883c5>] [<ffffffff9f7883c5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff967fe2333c90 EFLAGS: 00000246
RAX: 000000000001fd4b RBX: 0000000000000010 RCX: ffffffffffffffb8
RDX: fffffcccdbff9760 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff967fe2333c90 R08: fffffcccdbff9760 R09: 0000000000000000
R10: ffff9687ffbb9940 R11: ffffffffffffffff R12: ffffffffffffff10
R13: 0000000000000297 R14: 0000000000000000 R15: 0000000000000012
FS: 00007fca4d4d9880(0000) GS:ffff9687ffcc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc898125ec6 CR3: 0000000418bdc000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9f1ccff5>] release_pages+0x235/0x430
[<ffffffff9f2096dd>] free_pages_and_swap_cache+0xad/0xd0
[<ffffffff9f1ef864>] tlb_flush_mmu_free+0x34/0x50
[<ffffffff9f1ef8e7>] arch_tlb_finish_mmu+0x47/0x80
[<ffffffff9f1efa03>] tlb_finish_mmu+0x23/0x30
[<ffffffff9f1f9214>] unmap_region+0xf4/0x140
[<ffffffff9f1fb805>] do_munmap+0x2a5/0x480
[<ffffffff9f1fba45>] vm_munmap+0x65/0xb0
[<ffffffff9f1fcce2>] SyS_munmap+0x22/0x30
[<ffffffff9f792ed2>] system_call_fastpath+0x25/0x2a
Code: 07 00 0f 1f 40 00 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 89 e5 c6 07 00 0f 1f 40 00 48 89 f7 57 9d <0f> 1f 44 00 00 5d c3 0f 1f 40 00 0f 1f 44 00 00 55 48 89 e5 48
Complete report #541888
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
free_pages_and_swap_cache
|
vmlinux |
0xffffffffaba09f43
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x63
|
|
2
|
tlb_flush_mmu_free
|
vmlinux |
0xffffffffab9f00f4
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x34
|
|
3
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffab9f0177
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x47
|
|
4
|
tlb_finish_mmu
|
vmlinux |
0xffffffffab9f0293
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x23
|
|
5
|
unmap_region
|
vmlinux |
0xffffffffab9f9a84
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xf4
|
|
6
|
do_munmap
|
vmlinux |
0xffffffffab9fc025
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x2a5
|
|
7
|
vm_munmap
|
vmlinux |
0xffffffffab9fc265
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x65
|
|
8
|
SyS_munmap
|
vmlinux |
0xffffffffab9fd562
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x22
|
|
9
|
system_call_fastpath
|
vmlinux |
0xffffffffabf99f92
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0x25
|
|
10
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffabf99ed5
Build id: 3.10.0-1160.76.1.el7.x86_64
|
0xa2
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [lsphp:8610]
Modules linked in: xt_time xt_conntrack tcp_diag inet_diag xt_owner ip6t_REJECT nf_reject_ipv6 ip6table_filter xt_nat xt_NFLOG xt_set xt_multiport ip_set_bitmap_port ip_set_list_set ip_set_hash_net kcare(OE) iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 ip6table_mangle ip6table_raw ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 nf_nat nf_conntrack ip6_tables iptable_mangle iptable_raw ipt_REJECT nf_reject_ipv4 iptable_filter nfnetlink_log ext4 mbcache jbd2 loop ip_set vmw_vsock_vmci_transport vsock ppdev crc32_pclmul vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper vmw_vmci cryptd joydev pcspkr sg i2c_piix4 parport_pc parport ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea
sysfillrect sysimgblt fb_sys_fops ttm drm ata_piix libata nfit mptspi scsi_transport_spi libnvdimm mptscsih mptbase crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 6 PID: 8610 Comm: lsphp Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.76.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9158757dc200 ti: ffff915535c9c000 task.ti: ffff915535c9c000
RIP: 0010:[<ffffffffaba09f43>] [<ffffffffaba09f43>] free_pages_and_swap_cache+0x63/0xd0
RSP: 0018:ffff915535c9fd30 EFLAGS: 00000246
RAX: 00afffff0000086c RBX: ffff915535c9fd78 RCX: ffff915535c9fcd0
RDX: 0000000000000000 RSI: 000000000000000e RDI: ffff9158418b8be0
RBP: ffff915535c9fd58 R08: 0000000000000000 R09: 0000000000000000
R10: ffff915fbff9e000 R11: 0000000000000000 R12: 00007fe800000000
R13: 0000000000000018 R14: ffff915535c9fca0 R15: 0000000000000246
FS: 00007fe9069db840(0000) GS:ffff915a5a900000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f7dbbba1000 CR3: 00000005a2fdc000 CR4: 00000000000407e0
Call Trace:
[<ffffffffab9f00f4>] tlb_flush_mmu_free+0x34/0x50
[<ffffffffab9f0177>] arch_tlb_finish_mmu+0x47/0x80
[<ffffffffab9f0293>] tlb_finish_mmu+0x23/0x30
[<ffffffffab9f9a84>] unmap_region+0xf4/0x140
[<ffffffffab9fc025>] do_munmap+0x2a5/0x480
[<ffffffffab9fc265>] vm_munmap+0x65/0xb0
[<ffffffffab9fd562>] SyS_munmap+0x22/0x30
[<ffffffffabf99f92>] system_call_fastpath+0x25/0x2a
[<ffffffffabf99ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 00 00 45 0f 4e ef 31 db 45 85 ed 7f 14 eb 59 66 0f 1f 84 00 00 00 00 00 48 83 c3 01 41 39 dd 7e 47 4d 8b 24 de 41 f6 c4 01 75 ed <49> 8b 04 24 a9 00 00 01 00 74 e2 41 8b 44 24 18 85 c0 79 d9 f0
Complete report #446598
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_unlock_irqrestore
|
vmlinux |
0xffffffffb918a3f5
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x15
|
|
2
|
release_pages
|
vmlinux |
0xffffffffb8bccd05
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x235
|
|
3
|
free_pages_and_swap_cache
|
vmlinux |
0xffffffffb8c0978d
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xad
|
|
4
|
tlb_flush_mmu_free
|
vmlinux |
0xffffffffb8bef8d4
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x34
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffb8bef957
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x47
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffb8befa73
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x23
|
|
7
|
unmap_region
|
vmlinux |
0xffffffffb8bf9264
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0xf4
|
|
8
|
do_munmap
|
vmlinux |
0xffffffffb8bfb855
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x2a5
|
|
9
|
vm_munmap
|
vmlinux |
0xffffffffb8bfba95
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x65
|
|
10
|
SyS_munmap
|
vmlinux |
0xffffffffb8bfcd92
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x22
|
|
11
|
system_call_fastpath
|
vmlinux |
0xffffffffb9194f92
Build id: 3.10.0-1160.15.2.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#52 stuck for 24s! [samtools:36094]
Modules linked in: veth ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_addrtype br_netfilter overlay(T) tcp_lp rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache uinput devlink binfmt_misc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_13501(E) snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hda_core snd_hwdep snd_seq snd_seq_device
snd_pcm snd_timer qxl ttm snd drm_kms_helper ppdev syscopyarea sysfillrect iTCO_wdt iTCO_vendor_support crc32_pclmul parport_pc ghash_clmulni_intel aesni_intel sysimgblt lrw gf128mul pcspkr fb_sys_fops glue_helper ablk_helper soundcore lpc_ich cryptd joydev parport i2c_i801 virtio_balloon virtio_rng drm drm_panel_orientation_quirks nfsd nfs_acl lockd auth_rpcgss grace sunrpc ip_tables xfs libcrc32c virtio_net virtio_console virtio_blk net_failover failover ahci libahci crct10dif_pclmul crct10dif_common virtio_pci crc32c_intel libata virtio_ring serio_raw virtio dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 52 PID: 36094 Comm: samtools Kdump: loaded Tainted: P EL ------------ T 3.10.0-1160.15.2.el7.x86_64 #1
Hardware name: Red Hat KVM/RHEL-AV, BIOS 1.13.0-2.module+el8.4.0+534+4680a14e 04/01/2014
task: ffff8f903f1ec200 ti: ffff8f954f2f4000 task.ti: ffff8f954f2f4000
RIP: 0010:[<ffffffffb918a3f5>] [<ffffffffb918a3f5>] _raw_spin_unlock_irqrestore+0x15/0x20
RSP: 0018:ffff8f954f2f7c90 EFLAGS: 00000246
RAX: 0000000000000000 RBX: ffff8f954f2f7c60 RCX: ffffffffffffff83
RDX: ffffdbb5299f6660 RSI: 0000000000000246 RDI: 0000000000000246
RBP: ffff8f954f2f7c90 R08: ffffdbb5299f6660 R09: 0000000000000000
R10: 0000000000000000 R11: ffffffffffffffff R12: ffffffffb8b1791b
R13: ffff8f954f2f7c10 R14: ffff9173bffd7540 R15: 0000000000000002
FS: 00007fbbd276f740(0000) GS:ffff916bec300000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd2d212eed0 CR3: 00000009197a4000 CR4: 0000000000760ee0
PKRU: 55555554
Call Trace:
[<ffffffffb8bccd05>] release_pages+0x235/0x430
[<ffffffffb8c0978d>] free_pages_and_swap_cache+0xad/0xd0
[<ffffffffb8bef8d4>] tlb_flush_mmu_free+0x34/0x50
[<ffffffffb8bef957>] arch_tlb_finish_mmu+0x47/0x80
[<ffffffffb8befa73>] tlb_finish_mmu+0x23/0x30
[<ffffffffb8bf9264>] unmap_region+0xf4/0x140
[<ffffffffb8bfb855>] do_munmap+0x2a5/0x480
[<ffffffffb8bfba95>] vm_munmap+0x65/0xb0
[<ffffffffb8bfcd92>] SyS_munmap+0x22/0x30
[<ffffffffb9194f92>] system_call_fastpath+0x25/0x2a
Code: 16 ce 98 ff 66 90 5d c3 0f 1f 40 00 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 e8 f6 cd 98 ff 66 90 48 89 f7 57 9d <66> 66 90 66 90 5d c3 0f 1f 40 00 66 66 66 66 90 55 48 89 e5 48