Complete report #1420873
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9b71e25a
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9b681fa8
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9b68201b
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff9b7fa0ce
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xae
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff9b7fbc4f
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff9b7fbd83
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x23
|
|
7
|
madvise_free_single_vma
|
vmlinux |
0xffffffff9b7f87b6
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x116
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff9b7f96f1
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x4d1
|
|
9
|
|
traps |
0xffffffffc09dd11d
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x1bed
|
|
10
|
tracesys
|
vmlinux |
0xffffffff9bdc562e
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#7 stuck for 23s! [jemalloc_bg_thd:28731]
Modules linked in: udp_diag traps(OE) tcp_diag inet_diag nfsv3 rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log dm_mod ppdev sb_edac vmw_balloon iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr
sg parport_pc parport vmw_vmci i2c_piix4 nfsd nfs_acl lockd auth_rpcgss grace sunrpc binfmt_misc ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper sd_mod syscopyarea sysfillrect sysimgblt crc_t10dif fb_sys_fops crct10dif_generic ttm drm ata_piix nfit libnvdimm libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks floppy fuse [last unloaded: traps]
CPU: 7 PID: 28731 Comm: jemalloc_bg_thd Kdump: loaded Tainted: G OE ------------ 3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff98c0ac2a1080 ti: ffff98c0aba48000 task.ti: ffff98c0aba48000
RIP: 0010:[<ffffffff9b71e25a>] [<ffffffff9b71e25a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff98c0aba4bc70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: 000800009b681b10 RCX: ffff98c0cf2206f0
RDX: 0000000000000000 RSI: 000000000000000c RDI: 0000000000000000
RBP: ffff98c0aba4bca8 R08: ffff98b621603000 R09: ffffffff9b99c7f9
R10: ffff98c0cf3df160 R11: ffffde926d34ae00 R12: 0000000000000282
R13: 0000000000000282 R14: ffff98c0aba4bc20 R15: 00007ff798fd6000
FS: 00007ff7c9eff700(0000) GS:ffff98c0cf3c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff792bd7000 CR3: 00000007089ae000 CR4: 00000000001607e0
Call Trace:
[<ffffffff9b681fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9b68201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9b7fa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffff9b7fbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9b7fbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffff9b7f87b6>] madvise_free_single_vma+0x116/0x180
[<ffffffff9b7f96f1>] SyS_madvise+0x4d1/0xac0
[<ffffffffc09dd11d>] ? probe_fini+0x1bed/0x1ee0 [traps]
[<ffffffff9bdc562e>] tracesys+0xa6/0xcc
Code: 48 63 35 ae 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 19 35 9c f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 7d 18 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1419619
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffff9e3170da
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffff9e27e258
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffff9e27e2cb
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffff9e3ee676
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0xa6
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffff9e3f013f
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffff9e3f0263
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffff9e3f372a
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffff9e3edc1d
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x49d
|
|
9
|
|
traps |
0xffffffffc07def7d
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x1bad
|
|
10
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff9e995ed5
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0xa2
|
|
11
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff9e995ec9
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x96
|
|
12
|
tracesys
|
vmlinux |
0xffffffff9e996226
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [ReportScheduler:4089]
Modules linked in: traps(OE) devlink cmac arc4 md4 nls_utf8 cifs ccm dns_resolver ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd ppdev vmw_balloon pcspkr joydev sg parport_pc vmw_vmci parport i2c_piix4 ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic ata_generic
pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm nfit drm ata_piix libnvdimm crct10dif_pclmul crct10dif_common crc32c_intel mptspi serio_raw libata scsi_transport_spi mptscsih mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 4089 Comm: ReportScheduler Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff92b45ff51080 ti: ffff92b4a3724000 task.ti: ffff92b4a3724000
RIP: 0010:[<ffffffff9e3170da>] [<ffffffff9e3170da>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff92b4a3727c70 EFLAGS: 00000202
RAX: 0000000000000002 RBX: ffffffff9e27ddd0 RCX: ffff92b4bfca0600
RDX: 0000000000000002 RSI: 0000000000000008 RDI: 0000000000000000
RBP: ffff92b4a3727ca8 R08: ffff92b0f2411800 R09: ffffffff9e586bf9
R10: ffff92b4bfc5f160 R11: fffff7e414a2d600 R12: ffff92b4a3727c40
R13: ffffffff9e586b4f R14: ffff92b4a3727c18 R15: ffff92b4bfc5acc0
FS: 00007f38dc1ff700(0000) GS:ffff92b4bfc40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f38b87f0020 CR3: 000000052ab60000 CR4: 00000000000407e0
Call Trace:
[<ffffffff9e27e258>] native_flush_tlb_others+0xb8/0xc0
[<ffffffff9e27e2cb>] flush_tlb_mm_range+0x6b/0x140
[<ffffffff9e3ee676>] tlb_flush_mmu_tlbonly+0xa6/0xc0
[<ffffffff9e3f013f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffff9e3f0263>] tlb_finish_mmu+0x23/0x30
[<ffffffff9e3f372a>] zap_page_range+0xea/0x130
[<ffffffff9e3edc1d>] SyS_madvise+0x49d/0xac0
[<ffffffffc07def7d>] ? probe_fini+0x1bad/0x1e90 [traps]
[<ffffffff9e995ed5>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff9e995ec9>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff9e996226>] tracesys+0xa6/0xcc
Code: 48 63 35 ce 85 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 60 15 f5 9e f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 9d 85 c4 00 eb b7 0f b6 4d cc 4c 89
Complete report #1146779
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
smp_call_function_many
|
vmlinux |
0xffffffffb6f1e25a
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x20a
|
|
2
|
native_flush_tlb_others
|
vmlinux |
0xffffffffb6e81fa8
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xb8
|
|
3
|
flush_tlb_mm_range
|
vmlinux |
0xffffffffb6e8201b
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x6b
|
|
4
|
tlb_flush_mmu_tlbonly
|
vmlinux |
0xffffffffb6ffa0ce
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xae
|
|
5
|
arch_tlb_finish_mmu
|
vmlinux |
0xffffffffb6ffbc4f
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x3f
|
|
6
|
tlb_finish_mmu
|
vmlinux |
0xffffffffb6ffbd83
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x23
|
|
7
|
zap_page_range
|
vmlinux |
0xffffffffb6fff2ca
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xea
|
|
8
|
SyS_madvise
|
vmlinux |
0xffffffffb6ff96bd
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0x49d
|
|
9
|
?
uid_tracker_process_syscall
|
traps |
0xffffffffc08d4234
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xb4
|
|
10
|
tracesys
|
vmlinux |
0xffffffffb75c562e
Build id: 3.10.0-1160.119.1.el7.x86_64
|
0xa6
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 23s! [SendReceiveJobs:6846]
Modules linked in: tcp_lp traps(OE) tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables cmac arc4 md4 nls_utf8 cifs ccm dns_resolver ipt_REJECT nf_reject_ipv4 xt_conntrack xt_multiport iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_CHECKSUM vmw_vsock_vmci_transport vsock iptable_mangle sunrpc xfs iosf_mbi libcrc32c crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul glue_helper vmw_balloon ablk_helper cryptd joydev vmw_vmci pcspkr sg i2c_piix4 parport_pc parport binfmt_misc ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul
crct10dif_common ata_piix mptspi crc32c_intel scsi_transport_spi libata mptscsih nfit serio_raw libnvdimm mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 3 PID: 6846 Comm: SendReceiveJobs Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.119.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 11/12/2020
task: ffff8b6137388000 ti: ffff8b61b9ab4000 task.ti: ffff8b61b9ab4000
RIP: 0010:[<ffffffffb6f1e25a>] [<ffffffffb6f1e25a>] smp_call_function_many+0x20a/0x270
RSP: 0018:ffff8b61b9ab7c70 EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffffb6e81b10 RCX: ffff8b6236e20650
RDX: 0000000000000000 RSI: 0000000000000020 RDI: 0000000000000000
RBP: ffff8b61b9ab7ca8 R08: ffff8b617afab400 R09: ffffffffb719c7f9
R10: ffff8b6236edf160 R11: ffffdb3a014dfc00 R12: 0000000000000286
R13: ffff8b61b9ab7c20 R14: ffff8b617ae08000 R15: 0000000000000001
FS: 00007fbbd5552700(0000) GS:ffff8b6236ec0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000234b3c000 CR4: 00000000000407e0
Call Trace:
[<ffffffffb6e81fa8>] native_flush_tlb_others+0xb8/0xc0
[<ffffffffb6e8201b>] flush_tlb_mm_range+0x6b/0x140
[<ffffffffb6ffa0ce>] tlb_flush_mmu_tlbonly+0xae/0xd0
[<ffffffffb6ffbc4f>] arch_tlb_finish_mmu+0x3f/0x80
[<ffffffffb6ffbd83>] tlb_finish_mmu+0x23/0x40
[<ffffffffb6fff2ca>] zap_page_range+0xea/0x140
[<ffffffffb6ff96bd>] SyS_madvise+0x49d/0xac0
[<ffffffffc08d4234>] ? uid_tracker_process_syscall+0xb4/0x1c0 [traps]
[<ffffffffb75c562e>] tracesys+0xa6/0xcc
Code: 48 63 35 ae 18 c4 00 89 c2 39 f0 0f 8d 7d fe ff ff 48 98 49 8b 0f 48 03 0c c5 c0 19 b5 b7 f6 41 20 01 74 cd 0f 1f 44 00 00 f3 90 <f6> 41 20 01 75 f8 48 63 35 7d 18 c4 00 eb b7 0f b6 4d cc 4c 89