Complete report #1423358
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
sync_inodes_sb
|
vmlinux |
0xffffffff8a27f494
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x194
|
|
2
|
|
vmlinux |
0xffffffff8a2840a0
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x70
|
|
3
|
sync_inodes_one_sb
|
vmlinux |
0xffffffff8a2840b9
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x19
|
|
4
|
iterate_supers
|
vmlinux |
0xffffffff8a252933
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xc3
|
|
5
|
sys_sync
|
vmlinux |
0xffffffff8a284394
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x44
|
|
6
|
system_call_fastpath
|
vmlinux |
0xffffffff8a795f92
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0x25
|
|
7
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff8a795ed5
Build id: 3.10.0-1160.49.1.el7.x86_64
|
0xa2
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [VeeamService se:26840]
Modules linked in: veeamsnap(OE) nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache vmw_vsock_vmci_transport vsock sunrpc squashfs loop sb_edac coretemp ppdev iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev pcspkr sg i2c_piix4 vmw_vmci parport_pc parport binfmt_misc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel e1000e syscopyarea sysfillrect serio_raw sysimgblt fb_sys_fops ttm drm ahci ptp libahci pps_core vmw_pvscsi ata_piix nfit libata libnvdimm drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 1 PID: 26840 Comm: VeeamService se Kdump: loaded Tainted: G OEL ------------ 3.10.0-1160.49.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9ed50dd86300 ti: ffff9ed59a560000 task.ti: ffff9ed59a560000
RIP: 0010:[<ffffffff8a27f494>] [<ffffffff8a27f494>] sync_inodes_sb+0x194/0x3d0
RSP: 0018:ffff9ed59a563e20 EFLAGS: 00010293
RAX: ffff9ed39d788dd8 RBX: 0000000000000001 RCX: ffff9ed59a563fd8
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9ed39d789158
RBP: ffff9ed59a563ed8 R08: ffff9ed59a563d60 R09: 0000000000000001
R10: 0000000000000001 R11: 7fffffffffffffff R12: 7fffffffffffffff
R13: ffff9ed59a563ed8 R14: ffffffff8a0bc4f1 R15: 00000000787bd378
FS: 00007faf3f7fe700(0000) GS:ffff9ed63fc80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00001e1a9b42d048 CR3: 000000042bc80000 CR4: 00000000000607e0
Call Trace:
[<ffffffff8a2840a0>] ? generic_write_sync+0x70/0x70
[<ffffffff8a2840b9>] sync_inodes_one_sb+0x19/0x20
[<ffffffff8a252933>] iterate_supers+0xc3/0x120
[<ffffffff8a284394>] sys_sync+0x44/0xb0
[<ffffffff8a795f92>] system_call_fastpath+0x25/0x2a
[<ffffffff8a795ed5>] ? system_call_after_swapgs+0xa2/0x13a
Code: 1f 84 00 00 00 00 00 48 89 df c6 07 00 66 66 66 90 49 8b 87 08 01 00 00 49 39 c5 4c 8d b8 f8 fe ff ff 74 93 49 8d 9f 88 00 00 00 <4d> 8b 67 30 48 89 df e8 e0 c5 50 00 41 f6 87 a0 00 00 00 38 75
Complete report #1412136
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
sync_inodes_sb
|
vmlinux |
0xffffffff9c87f49b
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x19b
|
|
2
|
|
vmlinux |
0xffffffff9c8840a0
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x70
|
|
3
|
sync_inodes_one_sb
|
vmlinux |
0xffffffff9c8840b9
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x19
|
|
4
|
iterate_supers
|
vmlinux |
0xffffffff9c852933
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0xc3
|
|
5
|
sys_sync
|
vmlinux |
0xffffffff9c884394
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x44
|
|
6
|
system_call_fastpath
|
vmlinux |
0xffffffff9cd95f92
Build id: 3.10.0-1160.45.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#12 stuck for 22s! [dsync:132327]
Modules linked in: ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp scsi_tgt i40iw rdma_ucm ib_iser libiscsi ib_umad ib_ipoib scsi_transport_iscsi mlx5_ib ib_uverbs rpcrdma rdma_cm iw_cm ib_cm ib_core rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache tcp_diag udp_diag inet_diag unix_diag af_packet_diag netlink_diag mmfs26(OE) mmfslinux(OE) tracedev(OE) bonding ip6table_filter ip6table_raw ip6_tables xt_comment xt_multiport xt_conntrack iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod iTCO_wdt iTCO_vendor_support skx_edac intel_powerclamp coretemp intel_rapl iosf_mbi kvm_intel kvm irqbypass crc32_pclmul ghash_clmulni_intel
aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr lpc_ich i2c_i801 joydev mei_me mei wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad binfmt_misc ip_tables xfs libcrc32c sd_mod crc_t10dif crct10dif_generic i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm mlx5_core i40e ahci drm mlxfw libahci devlink megaraid_sas crct10dif_pclmul ptp crct10dif_common libata crc32c_intel drm_panel_orientation_quirks pps_core nfit libnvdimm
CPU: 12 PID: 132327 Comm: dsync Kdump: loaded Tainted: G W OE ------------ 3.10.0-1160.45.1.el7.x86_64 #1
Hardware name: Supermicro SYS-1019P-WTR/X11SPW-TF, BIOS 3.2 10/17/2019
task: ffffa055c7983180 ti: ffffa01a260dc000 task.ti: ffffa01a260dc000
RIP: 0010:[<ffffffff9c87f49b>] [<ffffffff9c87f49b>] sync_inodes_sb+0x19b/0x3d0
RSP: 0018:ffffa01a260dfe20 EFLAGS: 00000202
RAX: ffffa034524b9330 RBX: 0000000000000002 RCX: 00000000c0000100
RDX: 0000000000000001 RSI: ffffa055c7983180 RDI: ffffa034524b92b0
RBP: ffffa01a260dfed8 R08: ffffa01a260dc000 R09: 0000000000000001
R10: 0000000000000002 R11: 0000000000000206 R12: ffffa01a260dfda0
R13: ffffa01a260dfed8 R14: 00000000ffffffff R15: ffffa055c7983180
FS: 00007fbb82e6f740(0000) GS:ffffa0723cb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f2d70cd7d70 CR3: 00000004fdc42000 CR4: 00000000007607e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
PKRU: 55555554
Call Trace:
[<ffffffff9c8840a0>] ? generic_write_sync+0x70/0x70
[<ffffffff9c8840b9>] sync_inodes_one_sb+0x19/0x20
[<ffffffff9c852933>] iterate_supers+0xc3/0x120
[<ffffffff9c884394>] sys_sync+0x44/0xb0
[<ffffffff9cd95f92>] system_call_fastpath+0x25/0x2a
Code: 48 89 df c6 07 00 0f 1f 40 00 49 8b 87 08 01 00 00 49 39 c5 4c 8d b8 f8 fe ff ff 74 93 49 8d 9f 88 00 00 00 4d 8b 67 30 48 89 df <e8> e0 c5 50 00 41 f6 87 a0 00 00 00 38 75 c6 49 83 7c 24 60 00
Complete report #1040680
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
sync_inodes_sb
|
vmlinux |
0xffffffff9447ec74
Build id: 3.10.0-1160.el7.x86_64
|
0x194
|
|
2
|
|
vmlinux |
0xffffffff94483880
Build id: 3.10.0-1160.el7.x86_64
|
0x70
|
|
3
|
sync_inodes_one_sb
|
vmlinux |
0xffffffff94483899
Build id: 3.10.0-1160.el7.x86_64
|
0x19
|
|
4
|
iterate_supers
|
vmlinux |
0xffffffff94451ef3
Build id: 3.10.0-1160.el7.x86_64
|
0xc3
|
|
5
|
sys_sync
|
vmlinux |
0xffffffff94483b74
Build id: 3.10.0-1160.el7.x86_64
|
0x44
|
|
6
|
system_call_fastpath
|
vmlinux |
0xffffffff94993f92
Build id: 3.10.0-1160.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#5 stuck for 22s! [systemd:31723]
Modules linked in: xt_nat veth nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables devlink ip6table_filter ip6_tables iptable_filter rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache overlay(T) vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 iosf_mbi crc32_pclmul ghash_clmulni_intel ppdev aesni_intel lrw gf128mul vmw_balloon glue_helper pcspkr ablk_helper joydev cryptd sg parport_pc parport vmw_vmci i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops
ttm drm nfit sd_mod libnvdimm ata_piix crc_t10dif crct10dif_generic libata crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod fuse
CPU: 5 PID: 31723 Comm: systemd Kdump: loaded Tainted: G W L ------------ T 3.10.0-1160.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/09/2019
task: ffff9e0f831de300 ti: ffff9e0a85200000 task.ti: ffff9e0a85200000
RIP: 0010:[<ffffffff9447ec74>] [<ffffffff9447ec74>] sync_inodes_sb+0x194/0x3d0
RSP: 0018:ffff9e0a85203e20 EFLAGS: 00010293
RAX: ffff9e0a377ba7a8 RBX: 0000000000000001 RCX: ffff9e0a85203fd8
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9e0a377b5c60
RBP: ffff9e0a85203ed8 R08: ffff9e0a85203d60 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000400 R12: 000000000001acc0
R13: ffff9e0a85203ed8 R14: ffff9e0a85203fd8 R15: ffff9e0a85203fd8
FS: 00007f3921119700(0000) GS:ffff9e0fffd40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005633f37df4e0 CR3: 0000000171754000 CR4: 00000000007607e0
PKRU: 55555554
Call Trace:
[<ffffffff94483880>] ? generic_write_sync+0x70/0x70
[<ffffffff94483899>] sync_inodes_one_sb+0x19/0x20
[<ffffffff94451ef3>] iterate_supers+0xc3/0x120
[<ffffffff94483b74>] sys_sync+0x44/0xb0
[<ffffffff94993f92>] system_call_fastpath+0x25/0x2a
Code: 1f 84 00 00 00 00 00 48 89 df c6 07 00 0f 1f 40 00 49 8b 87 08 01 00 00 49 39 c5 4c 8d b8 f8 fe ff ff 74 93 49 8d 9f 88 00 00 00 <4d> 8b 67 30 48 89 df e8 f0 ad 50 00 41 f6 87 a0 00 00 00 38 75
Complete report #534377
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_lock
|
vmlinux |
0xffffffff9c58ecec
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0xc
|
|
2
|
|
vmlinux |
0xffffffff9c07f4d0
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0x1a0
|
|
3
|
|
vmlinux |
0xffffffff9c0840b0
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0x70
|
|
4
|
sync_inodes_one_sb
|
vmlinux |
0xffffffff9c0840c9
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0x19
|
|
5
|
iterate_supers
|
vmlinux |
0xffffffff9c052793
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0xc3
|
|
6
|
sys_sync
|
vmlinux |
0xffffffff9c084374
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0x44
|
|
7
|
system_call_fastpath
|
vmlinux |
0xffffffff9c599f92
Build id: 3.10.0-1160.66.1.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#6 stuck for 22s! [sync:20097]
Modules linked in: falcon_lsm_serviceable(PE) falcon_nf_netcontain(PE) falcon_kal(E) falcon_lsm_pinned_14203(E) falcon_lsm_pinned_14108(E) vmw_vsock_vmci_transport vsock sunrpc btrfs raid6_pq sb_edac iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel xor aesni_intel vmw_balloon lrw gf128mul glue_helper ablk_helper cryptd joydev sg pcspkr i2c_piix4 vmw_vmci parport_pc parport binfmt_misc sch_fq_codel ip_tables xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix drm mptspi libata scsi_transport_spi crct10dif_pclmul crct10dif_common mptscsih crc32c_intel nfit mptbase serio_raw libnvdimm vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: falcon_kal]
CPU: 6 PID: 20097 Comm: sync Kdump: loaded Tainted: P E ------------ 3.10.0-1160.66.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffff9f986caa0000 ti: ffff9f99c6310000 task.ti: ffff9f99c6310000
RIP: 0010:[<ffffffff9c58ecec>] [<ffffffff9c58ecec>] _raw_spin_lock+0xc/0x30
RSP: 0018:ffff9f99c6313e18 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 7fffffffffffffff RCX: ffff9f99c6313fd8
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff9f9c02b8fe18
RBP: ffff9f99c6313ed8 R08: ffff9f99c6313d60 R09: 0000000000000001
R10: 0000000000000001 R11: ffff9f98237ad700 R12: ffffffff9bebc621
R13: 00000000bc69277b R14: ffffffff9bebc2ba R15: ffff9f99c6313da0
FS: 00007f8efd0f2740(0000) GS:ffff9fa0bfd80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd339452028 CR3: 000000050c730000 CR4: 00000000003607e0
Call Trace:
[<ffffffff9c07f4d0>] ? sync_inodes_sb+0x1a0/0x3d0
[<ffffffff9c0840b0>] ? generic_write_sync+0x70/0x70
[<ffffffff9c0840c9>] sync_inodes_one_sb+0x19/0x20
[<ffffffff9c052793>] iterate_supers+0xc3/0x120
[<ffffffff9c084374>] sys_sync+0x44/0xb0
[<ffffffff9c599f92>] system_call_fastpath+0x25/0x2a
Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 58 20 ff ff 5d
Complete report #518559
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_lock
|
vmlinux |
0xffffffffb876b46c
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xc
|
|
2
|
|
vmlinux |
0xffffffffb8271800
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x1a0
|
|
3
|
|
vmlinux |
0xffffffffb8276160
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x70
|
|
4
|
sync_inodes_one_sb
|
vmlinux |
0xffffffffb8276179
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x19
|
|
5
|
iterate_supers
|
vmlinux |
0xffffffffb8245b63
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xc3
|
|
6
|
sys_sync
|
vmlinux |
0xffffffffb8276454
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x44
|
|
7
|
system_call_fastpath
|
vmlinux |
0xffffffffb8775ddb
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x22
|
|
8
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffb8775d21
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xae
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [uvsh:5593]
Modules linked in: vmw_vsock_vmci_transport vsock dm_mirror dm_region_hash dm_log sb_edac iosf_mbi crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul dm_mod glue_helper ablk_helper ppdev cryptd vmw_balloon joydev pcspkr sg parport_pc vmw_vmci parport i2c_piix4 sch_fq_codel auth_rpcgss sunrpc ip_tables ext4 mbcache jbd2 sd_mod crc_t10dif crct10dif_generic sr_mod cdrom crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmxnet3 vmw_pvscsi ata_generic pata_acpi vmwgfx floppy drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm ata_piix nfit drm libata libnvdimm drm_panel_orientation_quirks
CPU: 1 PID: 5593 Comm: uvsh Kdump: loaded Not tainted 3.10.0-957.10.1.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 12/12/2018
task: ffffa0edb3f3d140 ti: ffffa0edac0cc000 task.ti: ffffa0edac0cc000
RIP: 0010:[<ffffffffb876b46c>] [<ffffffffb876b46c>] _raw_spin_lock+0xc/0x30
RSP: 0000:ffffa0edac0cfe18 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 7fffffffffffffff RCX: ffffa0edac0cffd8
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffa0ed972d5d90
RBP: ffffa0edac0cfed8 R08: ffffa0edac0cfd60 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffb80b8370
R13: 00000000fa212bd5 R14: ffffffffb80b7fba R15: ffffa0edac0cfda0
FS: 00007effa8ac0780(0000) GS:ffffa0ee3fd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ffe0f627fc4 CR3: 000000011fb92000 CR4: 00000000000607e0
Call Trace:
[<ffffffffb8271800>] ? sync_inodes_sb+0x1a0/0x3d0
[<ffffffffb8276160>] ? generic_write_sync+0x70/0x70
[<ffffffffb8276179>] sync_inodes_one_sb+0x19/0x20
[<ffffffffb8245b63>] iterate_supers+0xc3/0x120
[<ffffffffb8276454>] sys_sync+0x44/0xb0
[<ffffffffb8775ddb>] system_call_fastpath+0x22/0x27
[<ffffffffb8775d21>] ? system_call_after_swapgs+0xae/0x146
Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 66 66 66 66 90 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 40 1b ff ff 5d
Complete report #413235
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
sync_inodes_sb
|
vmlinux |
0xffffffffaa878c64
Build id: 3.10.0-1062.el7.x86_64
|
0x194
|
|
2
|
|
vmlinux |
0xffffffffaa87d870
Build id: 3.10.0-1062.el7.x86_64
|
0x70
|
|
3
|
sync_inodes_one_sb
|
vmlinux |
0xffffffffaa87d889
Build id: 3.10.0-1062.el7.x86_64
|
0x19
|
|
4
|
iterate_supers
|
vmlinux |
0xffffffffaa84cb13
Build id: 3.10.0-1062.el7.x86_64
|
0xc3
|
|
5
|
sys_sync
|
vmlinux |
0xffffffffaa87db64
Build id: 3.10.0-1062.el7.x86_64
|
0x44
|
|
6
|
system_call_fastpath
|
vmlinux |
0xffffffffaad8bede
Build id: 3.10.0-1062.el7.x86_64
|
0x25
|
|
7
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffaad8be21
Build id: 3.10.0-1062.el7.x86_64
|
0xae
|
NMI watchdog: BUG: soft lockup - CPU#14 stuck for 24s! [sync:39327]
Modules linked in: veth xt_nat xt_addrtype br_netfilter overlay(T) xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT nf_reject_ipv4 tun bridge stp llc ebtable_filter ebtables ip6table_filter ip6_tables devlink iptable_filter vmw_vsock_vmci_transport vsock sunrpc ext4 mbcache jbd2 vmw_balloon ppdev crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd sg pcspkr joydev parport_pc parport snd_seq_midi snd_seq_midi_event snd_ens1371 snd_rawmidi snd_ac97_codec ac97_bus snd_seq snd_seq_device snd_pcm snd_timer snd soundcore i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx
drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel mptspi ata_piix nfit scsi_transport_spi libnvdimm mptscsih libata e1000 serio_raw mptbase drm_panel_orientation_quirks dm_mirror dm_region_hash dm_log dm_mod
CPU: 14 PID: 39327 Comm: sync Kdump: loaded Tainted: G W L ------------ T 3.10.0-1062.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/13/2018
task: ffff94668aa7d230 ti: ffff94639c630000 task.ti: ffff94639c630000
RIP: 0010:[<ffffffffaa878c64>] [<ffffffffaa878c64>] sync_inodes_sb+0x194/0x3d0
RSP: 0018:ffff94639c633e20 EFLAGS: 00010216
RAX: ffff946392d109e8 RBX: 0000000000000001 RCX: 00000000c0000100
RDX: 0000000000000001 RSI: ffff94668aa7d230 RDI: ffff946715bba9a8
RBP: ffff94639c633ed8 R08: ffff94639c630000 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000001 R12: ffff94639c633fd8
R13: ffff94639c633ed8 R14: ffff94639c633fd8 R15: ffff94668aa7d230
FS: 00007f4502d16740(0000) GS:ffff946b9d980000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055615c6b53e8 CR3: 000000001c668000 CR4: 00000000000007e0
Call Trace:
[<ffffffffaa87d870>] ? generic_write_sync+0x70/0x70
[<ffffffffaa87d889>] sync_inodes_one_sb+0x19/0x20
[<ffffffffaa84cb13>] iterate_supers+0xc3/0x120
[<ffffffffaa87db64>] sys_sync+0x44/0xb0
[<ffffffffaad8bede>] system_call_fastpath+0x25/0x2a
[<ffffffffaad8be21>] ? system_call_after_swapgs+0xae/0x146
Code: 1f 84 00 00 00 00 00 48 89 df c6 07 00 66 66 66 90 49 8b 87 08 01 00 00 49 39 c5 4c 8d b8 f8 fe ff ff 74 93 49 8d 9f 88 00 00 00 <4d> 8b 67 30 48 89 df e8 40 8e 50 00 41 f6 87 a0 00 00 00 38 75
Complete report #229678
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_lock
|
vmlinux |
0xffffffffafd8baac
Build id: 3.10.0-1160.36.2.el7.x86_64
|
0xc
|
|
2
|
|
vmlinux |
0xffffffffaf87f450
Build id: 3.10.0-1160.36.2.el7.x86_64
|
0x1a0
|
|
3
|
|
vmlinux |
0xffffffffaf884050
Build id: 3.10.0-1160.36.2.el7.x86_64
|
0x70
|
|
4
|
sync_inodes_one_sb
|
vmlinux |
0xffffffffaf884069
Build id: 3.10.0-1160.36.2.el7.x86_64
|
0x19
|
|
5
|
iterate_supers
|
vmlinux |
0xffffffffaf8528e3
Build id: 3.10.0-1160.36.2.el7.x86_64
|
0xc3
|
|
6
|
sys_sync
|
vmlinux |
0xffffffffaf884344
Build id: 3.10.0-1160.36.2.el7.x86_64
|
0x44
|
|
7
|
system_call_fastpath
|
vmlinux |
0xffffffffafd95f92
Build id: 3.10.0-1160.36.2.el7.x86_64
|
0x25
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [systemd:37973]
Modules linked in: tcp_lp nls_utf8 isofs rfcomm xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack libcrc32c ip_set ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock bnep sunrpc dm_mirror dm_region_hash dm_log dm_mod snd_seq_midi snd_seq_midi_event iosf_mbi crc32_pclmul snd_ens1371 snd_rawmidi ppdev snd_ac97_codec ghash_clmulni_intel ac97_bus snd_seq snd_seq_device aesni_intel vmw_balloon
lrw gf128mul glue_helper ablk_helper snd_pcm cryptd btusb btrtl btbcm btintel bluetooth joydev pcspkr snd_timer sg rfkill snd soundcore vmw_vmci i2c_piix4 parport_pc parport ip_tables ext4 mbcache jbd2 sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx crct10dif_pclmul crct10dif_common drm_kms_helper crc32c_intel syscopyarea ahci sysfillrect sysimgblt fb_sys_fops ttm libahci nfit ata_piix serio_raw libnvdimm mptspi drm e1000 scsi_transport_spi libata mptscsih mptbase drm_panel_orientation_quirks fuse
CPU: 1 PID: 37973 Comm: systemd Kdump: loaded Not tainted 3.10.0-1160.36.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 02/27/2020
task: ffffa010ff87a100 ti: ffffa010ddc30000 task.ti: ffffa010ddc30000
RIP: 0010:[<ffffffffafd8baac>] [<ffffffffafd8baac>] _raw_spin_lock+0xc/0x30
RSP: 0018:ffffa010ddc33e18 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffffffffafd88e10 RCX: ffffa010ddc33fd8
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffffa010c7eefd58
RBP: ffffa010ddc33ed8 R08: ffffa010ddc33d60 R09: 0000000000000001
R10: 0000000000000001 R11: 00000000000001c5 R12: 0000000000000000
R13: ffffa010f5476400 R14: ffffa0113b65acc0 R15: ffffa010f5476400
FS: 00007fd7c571f700(0000) GS:ffffa0113b640000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000562901b1ae38 CR3: 0000000035786000 CR4: 00000000003607e0
Call Trace:
[<ffffffffaf87f450>] ? sync_inodes_sb+0x1a0/0x3d0
[<ffffffffaf884050>] ? generic_write_sync+0x70/0x70
[<ffffffffaf884069>] sync_inodes_one_sb+0x19/0x20
[<ffffffffaf8528e3>] iterate_supers+0xc3/0x120
[<ffffffffaf884344>] sys_sync+0x44/0xb0
[<ffffffffafd95f92>] system_call_fastpath+0x25/0x2a
Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 0f 1f 44 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 48 22 ff ff 5d
Complete report #17191
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
sync_inodes_sb
|
vmlinux |
0xffffffffaea7e1ea
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x1aa
|
|
2
|
|
vmlinux |
0xffffffffaea82de0
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x70
|
|
3
|
sync_inodes_one_sb
|
vmlinux |
0xffffffffaea82df9
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x19
|
|
4
|
iterate_supers
|
vmlinux |
0xffffffffaea51543
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0xc3
|
|
5
|
sys_sync
|
vmlinux |
0xffffffffaea830d4
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x44
|
|
6
|
system_call_fastpath
|
vmlinux |
0xffffffffaef92ed2
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0x25
|
|
7
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffaef92e15
Build id: 3.10.0-1127.8.2.el7.x86_64
|
0xa2
|
NMI watchdog: BUG: soft lockup - CPU#3 stuck for 22s! [sync:28683]
Modules linked in: ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables cmac arc4 md4 nls_utf8 cifs ccm dns_resolver nfsv3 nfs fscache ip_set_hash_ip ses enclosure scsi_transport_sas iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi bridge stp llc ip_set nfnetlink vmw_vsock_vmci_transport vsock crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd vmw_balloon pcspkr joydev sg parport_pc parport
vmw_vmci i2c_piix4 nfsd auth_rpcgss nfs_acl lockd grace sunrpc binfmt_misc xfs libcrc32c sr_mod sd_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect crct10dif_pclmul sysimgblt fb_sys_fops crct10dif_common ttm crc32c_intel serio_raw drm ata_piix nfit mptspi scsi_transport_spi libata mptscsih libnvdimm mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod [last unloaded: ip_tables]
CPU: 3 PID: 28683 Comm: sync Tainted: G L ------------ 3.10.0-1127.8.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 04/05/2016
task: ffff9410d36eb150 ti: ffff940d06eec000 task.ti: ffff940d06eec000
RIP: 0010:[<ffffffffaea7e1ea>] [<ffffffffaea7e1ea>] sync_inodes_sb+0x1aa/0x3d0
RSP: 0018:ffff940d06eefe20 EFLAGS: 00010246
RAX: 0000000000000000 RBX: 0000000000000001 RCX: ffff940d06eeffd8
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff940e04052f58
RBP: ffff940d06eefed8 R08: ffff940d06eefd60 R09: 0000000000000001
R10: 0000000000000001 R11: 0000000000000000 R12: 000000000001acc0
R13: ffff940d06eefed8 R14: ffff940d06eeffd8 R15: ffff940d06eeffd8
FS: 00007f9732e59740(0000) GS:ffff9412ffd80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f731baf4384 CR3: 00000002b6902000 CR4: 00000000000007e0
Call Trace:
[<ffffffffaea82de0>] ? generic_write_sync+0x70/0x70
[<ffffffffaea82df9>] sync_inodes_one_sb+0x19/0x20
[<ffffffffaea51543>] iterate_supers+0xc3/0x120
[<ffffffffaea830d4>] sys_sync+0x44/0xb0
[<ffffffffaef92ed2>] system_call_fastpath+0x25/0x2a
[<ffffffffaef92e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 00 00 49 39 c5 4c 8d b8 f8 fe ff ff 74 93 49 8d 9f 88 00 00 00 4d 8b 67 30 48 89 df e8 d0 a4 50 00 41 f6 87 a0 00 00 00 38 75 c6 <49> 83 7c 24 60 00 74 be 4c 89 ff e8 46 d7 fe ff 48 89 df c6 07
Complete report #5588
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
sync_inodes_sb
|
vmlinux |
0xffffffff8bc4a684
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x194
|
|
2
|
|
vmlinux |
0xffffffff8bc4ef60
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x70
|
|
3
|
sync_inodes_one_sb
|
vmlinux |
0xffffffff8bc4ef79
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x19
|
|
4
|
iterate_supers
|
vmlinux |
0xffffffff8bc1f3d3
Build id: 3.10.0-862.3.2.el7.x86_64
|
0xc3
|
|
5
|
sys_sync
|
vmlinux |
0xffffffff8bc4f254
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x44
|
|
6
|
system_call_fastpath
|
vmlinux |
0xffffffff8c12082f
Build id: 3.10.0-862.3.2.el7.x86_64
|
0x1c
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [dockerd:1534]
Modules linked in: veth xt_conntrack ipt_MASQUERADE nf_nat_masquerade_ipv4 nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack br_netfilter bridge stp llc dm_thin_pool dm_persistent_data dm_bio_prison dm_bufio loop binfmt_misc overlay sb_edac ext4 coretemp ppdev iosf_mbi mbcache crc32_pclmul jbd2 vmw_balloon ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr joydev sg parport_pc parport i2c_piix4 shpchp vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm crct10dif_pclmul crct10dif_common crc32c_intel serio_raw ata_piix mptspi scsi_transport_spi libata
mptscsih vmxnet3 mptbase i2c_core floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 1534 Comm: dockerd Kdump: loaded Tainted: G L ------------ 3.10.0-862.3.2.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/30/2014
task: ffff8d962c6c0fd0 ti: ffff8d9623e38000 task.ti: ffff8d9623e38000
RIP: 0010:[<ffffffff8bc4a684>] [<ffffffff8bc4a684>] sync_inodes_sb+0x194/0x3d0
RSP: 0018:ffff8d9623e3be20 EFLAGS: 00010206
RAX: ffff8d95727c4e70 RBX: 0000000000000000 RCX: ffff8d9623e3bfd8
RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8d95727c51f8
RBP: ffff8d9623e3bed8 R08: ffff8d9623e3bd60 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8d9623e3bfd8
R13: ffff8d9623e3bed8 R14: ffff8d9623e3bfd8 R15: ffff8d962c6c0fd0
FS: 00007fb10da7f700(0000) GS:ffff8d963fc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fab4dfc45f0 CR3: 000000022f304000 CR4: 00000000001607f0
Call Trace:
[<ffffffff8bc4ef60>] ? generic_write_sync+0x70/0x70
[<ffffffff8bc4ef79>] sync_inodes_one_sb+0x19/0x20
[<ffffffff8bc1f3d3>] iterate_supers+0xc3/0x120
[<ffffffff8bc4f254>] sys_sync+0x44/0xb0
[<ffffffff8c12082f>] system_call_fastpath+0x1c/0x21
Code: 1f 84 00 00 00 00 00 48 89 df c6 07 00 0f 1f 40 00 49 8b 87 08 01 00 00 49 39 c5 4c 8d b8 f8 fe ff ff 74 93 49 8d 9f 88 00 00 00 <4d> 8b 67 30 48 89 df e8 d0 c0 4c 00 41 f6 87 a0 00 00 00 38 75