Transcript Lip Contour Tracking - University of California, San Diego

Lip Contour Tracking
Kevin Liu
Purpose
1
• Track the outer
contour of a person’s
lip while speaking
• Can be as a quality
control in speech
recognition, improve
accuracy (combine
with audio data)
Input/Output
• Input – Video/Image of a person speaking
• Output – Video/Image with contour of the lip and
key points clearly mapped
Video/
Image
Program
Lip Contour/
Key points
Lip Key Points
Approach
• User pick a “Starting Point”
• Use “Jumping Snake” to track the upper lip and
upper key points (P2, P3, P4)
• Analyze the vertical luminance along the center
and find lower key point (P6)
• Analyze luminance and Construct curve fitting to
find corners (P1, P5)
• Curve-fit the key points to map contour
• (For Video) use P3 from previous frame to do
“jumping snake”
Jumping Snake
1.
2.
3.
4.
5.
6.
Specify initial point
Pick 4 parameters
Take integral on the
image gradient along
various paths
Select the path with
maximum integral value
Self correct to get new
initial point
Repeat step 2 with new
initial point
Detect Lower Key Point
• Analyze the gradient value along the vertical
path through upper-center key point (P3)
Lower Key Point
• Black = Minima Peak
• White = Maxima Peak
Corner Key Points
• Find the path of minimum luminance value
• The corners must exist on the minimum path
1
Corner Key Points
• Construct various
curve fitting paths
connecting the snake
and the corner points
• Integrate the gradient
along various paths
and find the
maximum
• The corner that yields
the maximum value
will be desired corner
key point
1
Resulting Key Points
1
Curve Fitting to Key points
• The upper path is fitted with 4th order polynomial
• The lower path is fitted with 2nd order polynomial
1
Results
• See video “output.avi”
– False key points in P3, P6
• See video “output1.avi”
– Wrinkles will cause undesired snake result
Improvements
• Use key point tracking instead of “jumping
snake” after the first frame for more
accurate tracking and speed
• Use better color transformation to enhance
the gradient result
• Try the algorithm on different skin tones
• Handle imperfect mouth shapes
• Work with different orientation of the head