Transcript Obtaining Shape from Scanning Electron Microscope Using

Obtaining Shape from
Scanning Electron Microscope
Using Hopfield Neural Network
Yuji Iwahori1, Haruki Kawanaka1,
Shinji Fukui2 and Kenji Funahashi1
1
Nagoya Institute of Technology, Japan
2 Aichi University of Education, Japan
1/20
Introduction
Shape from Scanning Electron
Microscope (SEM) images is
the recent topic in computer
vision.
The position of a light source
and a viewing point are the
same under the orthographic
projection.
The object stand is rotated to
some extent through the
observation.
2D Image of SEM
Recovering
3D Shape
2/20
Previous Approaches (1)
Photometric Stereo
Estimation using the temporal color space
use multiple images under the different light
source directions.
Linear Shape from Shading
Photometric Motion
the position of viewing point (camera) and light
source should be widely located
3/20
Previous Approaches (2)
Shape from Occluding Boundaries
is limited to a simply convex closed curved surface
Shape from Silhouette
uses multiple images through 360 degree rotation,
is also unavailable to object with local concave
shape
Surface Reflectance and Shape from Images
Using 90 degree rotation to get the feature
points
However the rotation angle is limited to SEM
4/20
New Proposed Approach
Uses optimization with two images
observed through the rotation of the
object stand
1.
2.
The appropriate initial vector is
determined using the Radial Basis
Function neural network (RBF-NN)
from two images during rotation.
The optimization is introduced using the
Hopfield like neural network (HF-NN).
5/20
Characteristics of SEM Image (1)
Orthographic projection
Rotation angle
 30    30
Reflectance property
s
R(i )  1  s  
cos i
 (1)
i : incident angle, < 70°
s ≈ 0.5

R(i) is normalized to the range of 0 and 1.
6/20
Characteristics of SEM Image (2)

z = F(x, y)
F : height distribution
p
z
z
, q
x
y
p, q : gradient parameters
 l = (0, 0, 1) : light source direction
 n  (nx , n y , nz ) : surface normal

 p,  q ,1
Cross Section of

(2)
Reflectance Map(q=0)
p2  q2 1
 cos i = n・l = nz
… (3)
From Eq.(1)(2)(3),R( p, q )   s  s p 2  q 2  1
7/20
Rotation Axis on Object Stand
Under the orthographic projection

the gradient of the rotation axis is the
same for both images observed during
rotation.
ex.
8/20
Estimation of Rotation Axis
1.
Assume A and B move A’
and B’ during the rotation
2.
Set A and A’ be the
same pixel
3.
Then rotation axis is
determined so that it
becomes perpendicular
to the line BB’ and passes
through the point A.
9/20
Shape Recovery from Two Images
Using Hopfield Neural Network
Hopfield Neural Network
(HF-NN)



the mutual connection network
the connection between the
neurons are the symmetric
HF-NN can be applied to solve
the optimization problem of the
energy function
m1
m2
m3
10/20
Energy Function to be Minimized
E  C1 E1  C2 E2  C3 E3
(p,q,z): unknown
2
2
2
2

 p   p   q   q  
 E1              dxdy variables
D

 x   y   x   y  

2


E

I
(
x
,
y
)

R
(
p
,
q
)
dxdy
C1, C2, C3 : the

2
D

2
regularization
2





z

z




parameters


E


p


q
dxdy




3




D
  y
 x
 
D : the target region of the

object
E1 : the smoothness constraint
E2 : the error of the observed image brightness I(x,y) and
the reflectance map R(p, q)
E3 : the error of the geometric relation for z and (p, q)
11/20
Initial Vector for Optimization (1)
Radial Basis Function Neural Network
(RBF-NN) is introduced to obtain the
approximation of gradient p, q
Assume the same pixel (x, y) during the
rotation.
I1(x, y)
RBF
nx
I2(x, y)
NN
nz
p
nx
nz
The integration of
along x direction
results in the height distribution.
12/20
Initial Vector for Optimization (2)
How to make dataset of RBF-NN
A sphere is used to make I1 and I2 using R(p,
q), where, R is
2
2
R ( p, q )   s  s p  q  1
since a sphere has the whole combination of
the surface gradient.
How to use learned RBF-NN
The corresponding point of the target object
is assumed to be the same during the rotation.
13/20
Updating Equation using HF-NN
E
z
E

 3
t
z
z
  2 z p  2 z q 
E3
 2C3  2   2  
z
x y
y 
 x
 E E E 
p
E

  1  2  3 
t
p
p
p 
 p
 2 p 2 p 
E1
 2C1  2  2 
p
y 
 x
E2
R( p, q)
 2C2 I ( x, y )  R( p, q) 
p
p
E3
 z

 2C3   p 
p
 x

 E E E 
q
E

  1  2  3 
t
q
q
q 
 q
  2q  2q 
E1
 2C1  2  2 
q
y 
 x
E2
R( p, q )
 2C2 I ( x, y )  R( p, q ) 
q
q
 z

E3
 2C3   q 
q
 y

The equation is iteratively used to optimize
the energy function, that is, each partial
difference becomes 0.
14/20
Iteration for Optimization
The optimization is applied to each of two
images repeatedly. The height z’with the
rotation angle
z( x, yis
) given
 x sin by
 z ( x, y) cos
Gradient are also calculated from the height
repeatedly during rotation.
C1 is gradually reduces
E  C1E1  C2 E2  C3 E3
constraint
E1 : the smoothness
Optimization is terminated the value of
energy function converges in comparison with
15/20
that of one step before.
Experiments (synthesis image)
Input Images
Rotation angle is 10°
Image size is 64×64 pixels
Rotation axis is along the center of
the image.RBF-NN
Initial Height
Theoretical Height
Maximum Height 10.37
Learning Data : 2000
Learning Epoch : 15
Recovered Height
MSE 1.8961
16/20
Experiments (SEM image)
Rotation angle taken is 10 °
Rotation axis is set from the known
feature points A and B
Input Images
Theoretical Height
Initial Height
Theoretical Depth 13.1031
Recovered Height Relaxation Method
MSE 3.8926
17/20
Experiments (SEM image)
Rotation angle taken is 10 °
Rotation axis is set from the known
feature points A and B
Input Images
Initial Height
Recovered Height
18/20
Conclusion
A new method is proposed to recover the
shape from SEM images.
HF-NN is introduced to solve the optimization
problem. The energy function is formulated
from two image during rotation.
The initial vector is obtained using RBF-NN.
19/20
Further works
Getting more accurate result using more
images
Treatment of the inter-reflection
Thank you
20/20