Transcript PowerPoint 프레젠테이션

Nonlinear Dimensionality
Reduction Approach
(ISOMAP)
2006. 2. 28
Young Ki Baik
Computer Vision Lab.
Seoul National University
Nonlinear Dimensionality Reduction Approach (ISOMAP)
References
A global geometric framework for nonlinear dimensionality
reduction
J. B. Tenenbaum, V. De Silva, J. C. Langford (Science 2000)
LLE and Isomap Analysis of Spectra and Colour Images
Dejan Kulpinski (Thesis 1999)
Out-of-Sample Extensions for LLE, Isomap, MDS,
Eigenmaps, and Spectral Clustering
Yoshua Bengio et.al. (TR 2003)
Computer Vision Lab. SNU
Nonlinear Dimensionality Reduction Approach (ISOMAP)
Contents
Introduction
PCA and MDS
ISOMAP
Conclusion
Computer Vision Lab. SNU
Nonlinear Dimensionality Reduction Approach (ISOMAP)
Dimensionality Reduction
The goal
The meaningful low-dimensional structures hidden in their
high-dimensional observations.
Classical techniques
PCA (Principle Component Analysis)
– preserves the variance
MDS (MultiDimensional Scaling)
- preserves inter-point distance
ISOMAP
LLE (Locally Linear Embedding)
Computer Vision Lab. SNU
Nonlinear Dimensionality Reduction Approach (ISOMAP)
Linear Dimensionality Reduction
PCA
Finds a low-dimensional embedding of the data points that
best preserves their variance as measured in the highdimensional input space.
MDS
Finds an embedding that preserves the inter-point distances,
equivalent to PCA when the distances are Euclidean.
Computer Vision Lab. SNU
Nonlinear Dimensionality Reduction Approach (ISOMAP)
Linear Dimensionality Reduction
MDS
distances
dij
dij2  ( xi  x j )T ( xi  x j )
Relation
A   12 dij2
B  HAH, H is thecenteringmatrix
bij  ( xi  x)T ( x j  x)

 T
then B  (HX)(HX)  X X
T
Computer Vision Lab. SNU
Nonlinear Dimensionality Reduction Approach (ISOMAP)
Nonlinear Dimensionality Reduction
Many data sets contain essential nonlinear structures
that invisible to PCA and MDS
Resort to some nonlinear dimensionality reduction
approaches.
Computer Vision Lab. SNU
Nonlinear Dimensionality Reduction Approach (ISOMAP)
ISOMAP
Example of Non-linear structure (Swiss roll)
Only the geodesic distances reflect the true low-dimensional
geometry of the manifold.
ISOMAP (Isometric feature Mapping)
Preserves the intrinsic geometry of the data.
Uses the geodesic manifold distances between all pairs.
Computer Vision Lab. SNU
Nonlinear Dimensionality Reduction Approach (ISOMAP)
ISOMAP (Algorithm Description)
Step 1
Determining neighboring points within a fixed radius based on
the input space distance dX i, j  .
These neighborhood relations are represented as a weighted
graph G over the data points.
Step 2
Estimating the geodesic distances dG i, j  between all pairs of
points on the manifold by computing their shortest path
distances in the graph G.
Step 3
Constructing an embedding of the data in d-dimensional
Euclidean space Y that best preserves the manifold’s geometry.
Computer Vision Lab. SNU
Nonlinear Dimensionality Reduction Approach (ISOMAP)
ISOMAP (Algorithm Description)
Step 1
Determining neighboring points within a fixed radius based on
the input space distance dX i, j  .
# ε-radius
# K-nearest neighbors
K=4
ε
These neighborhood relations are represented as a weighted
graph G over the data points.
i
dX i, j 
j
dX i, k 
k
Computer Vision Lab. SNU
Nonlinear Dimensionality Reduction Approach (ISOMAP)
ISOMAP (Algorithm Description)
Step 2
Estimating the geodesic distances dG i, j  between all pairs of
points on the manifold by computing their shortest path
distances in the graph G.
Can be done using Floyd’s algorithm or Dijkstra’s algorithm
d G (i, j )  d (i, j ) neighborin g i, j
d G (i, j )  
othewise
i
dG i, k 
j
k
dG k , j 
for k  1,2,..., N
d G (i, j )  min{ d G (i, j ), d G (i, k )  d G (k , j )}
Computer Vision Lab. SNU
Nonlinear Dimensionality Reduction Approach (ISOMAP)
ISOMAP (Algorithm Description)
Step 3
Constructing an embedding of the data in d-dimensional
Euclidean space Y that best preserves the manifold’s geometry.
Minimize the cost function:
E   ( DG )   ( DY ) L2
where DY (i, j )  yi  y j
DG (i, j )  d G (i, j )
and
 ( D) 
1
2
( I  N1 ) D .2 ( I  N1 )
Solution: take top d
eigenvectors of the
matrix  ( DG )
Computer Vision Lab. SNU
Nonlinear Dimensionality Reduction Approach (ISOMAP)
Experimental results
# FACE
: face pose and illumination
# Hand writing
: bottom loop and top arch
MDS : open triangles
Isomap : filled circles
Computer Vision Lab. SNU
Nonlinear Dimensionality Reduction Approach (ISOMAP)
Discussion
Isomap handles non-linear manifold.
Isomap keeps the advantages of PCA and MDS.
Non-iterative procedure
Polynomial procedure
Guaranteed convergence
Isomap represents the global structure of a data set
within a single coordinate system.
Computer Vision Lab. SNU
Nonlinear Dimensionality Reduction Approach (ISOMAP)
Computer Vision Lab. SNU