Complete report #253617
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
_raw_spin_lock
|
vmlinux |
0xffffffff92f8869c
Build id: 3.10.0-1127.el7.x86_64
|
0xc
|
|
2
|
|
vmlinux |
0xffffffff92a692ca
Build id: 3.10.0-1127.el7.x86_64
|
0x7a
|
|
3
|
|
vmlinux |
0xffffffff92a62c10
Build id: 3.10.0-1127.el7.x86_64
|
0x130
|
|
4
|
d_lookup
|
vmlinux |
0xffffffff92a693da
Build id: 3.10.0-1127.el7.x86_64
|
0x2a
|
|
5
|
proc_fill_cache
|
vmlinux |
0xffffffff92acb0df
Build id: 3.10.0-1127.el7.x86_64
|
0x6f
|
|
6
|
|
vmlinux |
0xffffffff92acefc0
Build id: 3.10.0-1127.el7.x86_64
|
0x190
|
|
7
|
proc_readfd_common
|
vmlinux |
0xffffffff92acf1e5
Build id: 3.10.0-1127.el7.x86_64
|
0xf5
|
|
8
|
|
vmlinux |
0xffffffff92acefc0
Build id: 3.10.0-1127.el7.x86_64
|
0x190
|
|
9
|
|
vmlinux |
0xffffffff92a62c10
Build id: 3.10.0-1127.el7.x86_64
|
0x130
|
|
10
|
proc_readfd
|
vmlinux |
0xffffffff92acf2f5
Build id: 3.10.0-1127.el7.x86_64
|
0x15
|
|
11
|
iterate_dir
|
vmlinux |
0xffffffff92a62b77
Build id: 3.10.0-1127.el7.x86_64
|
0x97
|
|
12
|
SyS_getdents
|
vmlinux |
0xffffffff92a63092
Build id: 3.10.0-1127.el7.x86_64
|
0xa2
|
|
13
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff92f92e09
Build id: 3.10.0-1127.el7.x86_64
|
0x96
|
|
14
|
|
vmlinux |
0xffffffff92a62c10
Build id: 3.10.0-1127.el7.x86_64
|
0x130
|
|
15
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff92f92e09
Build id: 3.10.0-1127.el7.x86_64
|
0x96
|
|
16
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff92f92e15
Build id: 3.10.0-1127.el7.x86_64
|
0xa2
|
|
17
|
system_call_fastpath
|
vmlinux |
0xffffffff92f92ed2
Build id: 3.10.0-1127.el7.x86_64
|
0x25
|
|
18
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffff92f92e15
Build id: 3.10.0-1127.el7.x86_64
|
0xa2
|
NMI watchdog: BUG: soft lockup - CPU#0 stuck for 28s! [influx_sampler:7119]
Modules linked in: tcp_lp btrfs raid6_pq xor vfat fat tcp_diag inet_diag ext4 mbcache jbd2 fuse xt_CHECKSUM ipt_MASQUERADE nf_nat_masquerade_ipv4 tun devlink nf_conntrack_netbios_ns nf_conntrack_broadcast ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock sunrpc uas coretemp usb_storage crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul ppdev glue_helper ablk_helper cryptd vmw_balloon
sg pcspkr joydev vmw_vmci parport_pc parport i2c_piix4 ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm serio_raw crct10dif_pclmul crct10dif_common crc32c_intel drm mptspi ata_piix scsi_transport_spi mptscsih libata mptbase vmxnet3 drm_panel_orientation_quirks floppy dm_mirror dm_region_hash dm_log dm_mod
CPU: 0 PID: 7119 Comm: influx_sampler Kdump: loaded Tainted: G L ------------ 3.10.0-1127.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/17/2015
task: ffff9c859c208000 ti: ffff9c84d8f50000 task.ti: ffff9c84d8f50000
RIP: 0010:[<ffffffff92f8869c>] [<ffffffff92f8869c>] _raw_spin_lock+0xc/0x30
RSP: 0018:ffff9c84d8f53d38 EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff9c84d8f53ef4 RCX: 0000000000000013
RDX: 0000000000000001 RSI: ffff9c84d8f53dd8 RDI: ffff9c8512a0cd18
RBP: ffff9c84d8f53d80 R08: 0000000000000001 R09: ffffffff92acefc0
R10: 0000000000000000 R11: ffff9c8581fc13c0 R12: ffff9c84d8f53cb8
R13: ffffffff92a71054 R14: ffff9c84d8f53ca0 R15: ffff9c84d8f53db0
FS: 00007fd23c7e9700(0000) GS:ffff9c85bfc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd2164791f0 CR3: 000000000cf7e000 CR4: 00000000000007f0
Call Trace:
[<ffffffff92a692ca>] ? __d_lookup+0x7a/0x160
[<ffffffff92a62c10>] ? iterate_dir+0x130/0x130
[<ffffffff92a693da>] d_lookup+0x2a/0x50
[<ffffffff92acb0df>] proc_fill_cache+0x6f/0x180
[<ffffffff92acefc0>] ? tid_fd_revalidate+0x190/0x190
[<ffffffff92acf1e5>] proc_readfd_common+0xf5/0x1f0
[<ffffffff92acefc0>] ? tid_fd_revalidate+0x190/0x190
[<ffffffff92a62c10>] ? iterate_dir+0x130/0x130
[<ffffffff92acf2f5>] proc_readfd+0x15/0x20
[<ffffffff92a62b77>] iterate_dir+0x97/0x130
[<ffffffff92a63092>] SyS_getdents+0xa2/0x130
[<ffffffff92f92e09>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff92a62c10>] ? iterate_dir+0x130/0x130
[<ffffffff92f92e09>] ? system_call_after_swapgs+0x96/0x13a
[<ffffffff92f92e15>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff92f92ed2>] system_call_fastpath+0x25/0x2a
[<ffffffff92f92e15>] ? system_call_after_swapgs+0xa2/0x13a
Code: 5d c3 0f 1f 44 00 00 85 d2 74 e4 0f 1f 40 00 eb ed 66 0f 1f 44 00 00 b8 01 00 00 00 5d c3 90 66 66 66 66 90 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 55 89 c6 48 89 e5 e8 49 19 ff ff 5d
Complete report #14115
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
__d_lookup
|
vmlinux |
0xffffffffb0a5ca98
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x68
|
|
2
|
|
vmlinux |
0xffffffffb0a56c70
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x130
|
|
3
|
d_lookup
|
vmlinux |
0xffffffffb0a5cbba
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x2a
|
|
4
|
proc_fill_cache
|
vmlinux |
0xffffffffb0abd45f
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x6f
|
|
5
|
|
vmlinux |
0xffffffffb0ac1330
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x190
|
|
6
|
proc_readfd_common
|
vmlinux |
0xffffffffb0ac1555
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xf5
|
|
7
|
|
vmlinux |
0xffffffffb0ac1330
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x190
|
|
8
|
|
vmlinux |
0xffffffffb0a56c70
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x130
|
|
9
|
proc_readfd
|
vmlinux |
0xffffffffb0ac1665
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x15
|
|
10
|
iterate_dir
|
vmlinux |
0xffffffffb0a56bd7
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x97
|
|
11
|
SyS_getdents
|
vmlinux |
0xffffffffb0a570f2
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xa2
|
|
12
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffb0f75d15
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xa2
|
|
13
|
|
vmlinux |
0xffffffffb0a56c70
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x130
|
|
14
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffb0f75d15
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xa2
|
|
15
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffb0f75d21
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xae
|
|
16
|
system_call_fastpath
|
vmlinux |
0xffffffffb0f75ddb
Build id: 3.10.0-957.10.1.el7.x86_64
|
0x22
|
|
17
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffb0f75d21
Build id: 3.10.0-957.10.1.el7.x86_64
|
0xae
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [sshd:5502]
Modules linked in: binfmt_misc udp_diag tcp_diag inet_diag nfsv3 nfs_acl rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache ip6t_REJECT nf_reject_ipv6 nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack libcrc32c iptable_filter sunrpc dm_mirror dm_region_hash dm_log dm_mod iosf_mbi crc32_pclmul ppdev ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd joydev virtio_balloon pcspkr parport_pc parport i2c_piix4 ip_tables ext4 mbcache jbd2 virtio_blk virtio_net crct10dif_pclmul crct10dif_common crc32c_intel ata_generic pata_acpi serio_raw cirrus drm_kms_helper syscopyarea sysfillrect floppy sysimgblt fb_sys_fops ttm ata_piix drm libata virtio_pci virtio_ring
drm_panel_orientation_quirks virtio
CPU: 1 PID: 5502 Comm: sshd Tainted: G L ------------ 3.10.0-957.10.1.el7.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
task: ffff9296761a5140 ti: ffff92965e3c0000 task.ti: ffff92965e3c0000
RIP: 0010:[<ffffffffb0a5ca98>] [<ffffffffb0a5ca98>] __d_lookup+0x68/0x160
RSP: 0018:ffff92965e3c3d40 EFLAGS: 00010282
RAX: ffffb1d9c008fde8 RBX: 0000000100000001 RCX: 0000000000000012
RDX: ffffb1d9c0000000 RSI: ffff92965e3c3dd8 RDI: ffff929675424600
RBP: ffff92965e3c3d80 R08: 0000000000000001 R09: ffffffffb0ac1330
R10: 0000000000000000 R11: 0000000000000246 R12: ffff9296761a5140
R13: 0000000000000000 R14: ffffffffb0a32ea2 R15: ffff92965e3c3cf0
FS: 00007f19325b38c0(0000) GS:ffff9296bfd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000555edce4c1d8 CR3: 000000007b5f2000 CR4: 00000000000406e0
Call Trace:
[<ffffffffb0a56c70>] ? iterate_dir+0x130/0x130
[<ffffffffb0a5cbba>] d_lookup+0x2a/0x50
[<ffffffffb0abd45f>] proc_fill_cache+0x6f/0x180
[<ffffffffb0ac1330>] ? tid_fd_revalidate+0x190/0x190
[<ffffffffb0ac1555>] proc_readfd_common+0xf5/0x1f0
[<ffffffffb0ac1330>] ? tid_fd_revalidate+0x190/0x190
[<ffffffffb0a56c70>] ? iterate_dir+0x130/0x130
[<ffffffffb0ac1665>] proc_readfd+0x15/0x20
[<ffffffffb0a56bd7>] iterate_dir+0x97/0x130
[<ffffffffb0a570f2>] SyS_getdents+0xa2/0x130
[<ffffffffb0f75d15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffb0a56c70>] ? iterate_dir+0x130/0x130
[<ffffffffb0f75d15>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffb0f75d21>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffb0f75ddb>] system_call_fastpath+0x22/0x27
[<ffffffffb0f75d21>] ? system_call_after_swapgs+0xae/0x146
Code: 89 c2 d3 ea 01 d0 23 05 db 2f b0 00 48 8b 15 c8 2f b0 00 48 8d 04 c2 48 8b 18 48 83 e3 fe 75 0b eb 39 90 48 8b 1b 48 85 db 74 30 <44> 39 73 18 75 f2 4c 8d 7b 50 4c 89 ff e8 b6 e9 50 00 4c 39 63
Complete report #174701
| Frame # |
Function |
Binary |
Source or offset |
Line |
|
1
|
__d_lookup
|
vmlinux |
0xffffffffb4e35cb8
Build id: 3.10.0-862.el7.x86_64
|
0x68
|
|
2
|
|
vmlinux |
0xffffffffb4e30160
Build id: 3.10.0-862.el7.x86_64
|
0xf0
|
|
3
|
d_lookup
|
vmlinux |
0xffffffffb4e35dda
Build id: 3.10.0-862.el7.x86_64
|
0x2a
|
|
4
|
proc_fill_cache
|
vmlinux |
0xffffffffb4e9537f
Build id: 3.10.0-862.el7.x86_64
|
0x6f
|
|
5
|
|
vmlinux |
0xffffffffb4e99120
Build id: 3.10.0-862.el7.x86_64
|
0x190
|
|
6
|
proc_readfd_common
|
vmlinux |
0xffffffffb4e9933d
Build id: 3.10.0-862.el7.x86_64
|
0xed
|
|
7
|
|
vmlinux |
0xffffffffb4e99120
Build id: 3.10.0-862.el7.x86_64
|
0x190
|
|
8
|
|
vmlinux |
0xffffffffb4e30160
Build id: 3.10.0-862.el7.x86_64
|
0xf0
|
|
9
|
proc_readfd
|
vmlinux |
0xffffffffb4e99455
Build id: 3.10.0-862.el7.x86_64
|
0x15
|
|
10
|
iterate_dir
|
vmlinux |
0xffffffffb4e2fef7
Build id: 3.10.0-862.el7.x86_64
|
0x97
|
|
11
|
SyS_getdents64
|
vmlinux |
0xffffffffb4e304d1
Build id: 3.10.0-862.el7.x86_64
|
0xa1
|
|
12
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffb531f715
Build id: 3.10.0-862.el7.x86_64
|
0xa2
|
|
13
|
|
vmlinux |
0xffffffffb4e30160
Build id: 3.10.0-862.el7.x86_64
|
0xf0
|
|
14
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffb531f715
Build id: 3.10.0-862.el7.x86_64
|
0xa2
|
|
15
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffb531f721
Build id: 3.10.0-862.el7.x86_64
|
0xae
|
|
16
|
system_call_fastpath
|
vmlinux |
0xffffffffb531f7d5
Build id: 3.10.0-862.el7.x86_64
|
0x1c
|
|
17
|
?
system_call_after_swapgs
|
vmlinux |
0xffffffffb531f721
Build id: 3.10.0-862.el7.x86_64
|
0xae
|
NMI watchdog: BUG: soft lockup - CPU#1 stuck for 28s! [etcd:1367]
Modules linked in: veth vxlan ip6_udp_tunnel udp_tunnel rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace ip6table_nat nf_conntrack_ipv6 fscache nf_defrag_ipv6 nf_nat_ipv6 xt_set xt_mark ip_set_hash_ipportnet ip_set_bitmap_port ip_set_hash_ipportip ip_set_hash_ipport ip_set dummy xt_comment nf_conntrack_netlink nfnetlink xt_addrtype br_netfilter xt_CHECKSUM iptable_mangle ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_nat_ipv4 nf_nat xt_conntrack ipt_REJECT nf_reject_ipv4 overlay(T) tun bridge stp llc ebtable_filter ebtables nf_conntrack_ipv4 nf_defrag_ipv4 ip_vs_sh ip_vs_wrr ip6table_filter ip_vs_rr ip_vs ip6_tables nf_conntrack iptable_filter devlink vmw_vsock_vmci_transport vsock sunrpc crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper
ppdev cryptd sg vmw_balloon pcspkr parport_pc joydev parport i2c_piix4 vmw_vmci shpchp sch_fq_codel ip_tables xfs libcrc32c sd_mod sr_mod cdrom crc_t10dif crct10dif_generic ata_generic pata_acpi vmwgfx drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops ttm drm ahci ata_piix libahci crct10dif_pclmul crct10dif_common serio_raw mptspi libata crc32c_intel scsi_transport_spi vmxnet3 i2c_core mptscsih mptbase dm_mirror dm_region_hash dm_log dm_mod
CPU: 1 PID: 1367 Comm: etcd Kdump: loaded Tainted: G L ------------ T 3.10.0-862.el7.x86_64 #1
Hardware name: VMware, Inc. VMware Virtual Platform/440BX Desktop Reference Platform, BIOS 6.00 09/21/2015
task: ffff8f8158e33f40 ti: ffff8f7dfb89c000 task.ti: ffff8f7dfb89c000
RIP: 0010:[<ffffffffb4e35cb8>] [<ffffffffb4e35cb8>] __d_lookup+0x68/0x160
RSP: 0018:ffff8f7dfb89fd40 EFLAGS: 00010282
RAX: ffffb99800a5fd60 RBX: ffff8f7dfb89fef4 RCX: 0000000000000015
RDX: ffffb99800002000 RSI: ffff8f7dfb89fdd8 RDI: ffff8f816d2ed180
RBP: ffff8f7dfb89fd80 R08: 0000000000000001 R09: ffffffffb4e99120
R10: 0000000000000000 R11: ffff8f816d2ed840 R12: ffff8f7dfb89fcc8
R13: ffffffffb4e3d674 R14: ffff8f7dfb89fcb0 R15: ffff8f7dfb89fdb0
FS: 000000000142b430(0063) GS:ffff8f817fc40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000000c42018d010 CR3: 0000000035bbe000 CR4: 00000000000007e0
Call Trace:
[<ffffffffb4e30160>] ? filldir+0xf0/0xf0
[<ffffffffb4e35dda>] d_lookup+0x2a/0x50
[<ffffffffb4e9537f>] proc_fill_cache+0x6f/0x180
[<ffffffffb4e99120>] ? tid_fd_revalidate+0x190/0x190
[<ffffffffb4e9933d>] proc_readfd_common+0xed/0x1f0
[<ffffffffb4e99120>] ? tid_fd_revalidate+0x190/0x190
[<ffffffffb4e30160>] ? filldir+0xf0/0xf0
[<ffffffffb4e99455>] proc_readfd+0x15/0x20
[<ffffffffb4e2fef7>] iterate_dir+0x97/0x130
[<ffffffffb4e304d1>] SyS_getdents64+0xa1/0x140
[<ffffffffb531f715>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffb4e30160>] ? filldir+0xf0/0xf0
[<ffffffffb531f715>] ? system_call_after_swapgs+0xa2/0x146
[<ffffffffb531f721>] ? system_call_after_swapgs+0xae/0x146
[<ffffffffb531f7d5>] system_call_fastpath+0x1c/0x21
[<ffffffffb531f721>] ? system_call_after_swapgs+0xae/0x146
Code: 89 c2 d3 ea 01 d0 23 05 9b 37 b1 00 48 8b 15 88 37 b1 00 48 8d 04 c2 48 8b 18 48 83 e3 fe 75 0b eb 39 90 48 8b 1b 48 85 db 74 30 <44> 39 73 18 75 f2 4c 8d 7b 50 4c 89 ff e8 96 f9 4d 00 4c 39 63