Transcript Document 7295856
Mobile Motion Tracking using Onboard Camera
Supervisor: Prof. LYU, Rung Tsong Michael Prepared by: Lam Man Kit Wong Yuk Man
Agenda
Motivation & Objective Previous Work Improvement & New components Sample Applications Conclusion Q&A
Motivation
Rapid increase in the use of camera phone Camera-phone can perform image processing tasks on the device itself Enhance human-computer interaction by mobile phone
Objective
Implement real-time motion tracking on Symbian phone, without requiring additional hardware Translational motion tracking Rotational motion tracking To develop a tracking engine for other mobile devices developers to use
Previous Work
A translational motion tracking engine Blocking matching • SEA (Successive Elimination Algorithm) • PPNM • PDE (Partial Distortion Elimination) • Adaptive Window Search • Spiral Scan Feature selection A translational motion tracking engine
Block Matching Algorithm
Block Matching Algorithm
Evaluate the "goodness" of a match by Sum of Absolute Difference
SAD
(
x
,
y
)
i
1
j N N
1 |
X
(
i
,
j
)
Y
(
i
,
j
) | Select the candidate block with the lowest error
Feature Selection
A good feature block: High variance -> complex block Not in a repeated pattern If a good feature block is found, the performance of the motion tracking is improved We have made improvements to our existing algorithm
Old Feature Selection
Divide the current frame into squares Apply Feature Selection Algorithm to each squares However, the best feature block may appear between two squares squares
New Feature Selection
More blocks are sampled Sample 26x26 variances of blocks with size 15x15 in a 54x54 window Search in spiral way and stop searching when selection criteria are matched Prefer to find a feature block in the center Prevent out of bound problem (block appear out of screen)
New Feature Selection
If a feature block is found on the edge, the following will happen: The tracking algorithm will fail to find the exact match Solution: Apply additional constraint
New Feature selection
In order to find a feature block that is not on the edge, we apply a checking algorithm Important difference in all directions => interest point
New Feature Selection
Now our newest feature selection algorithm becomes: Find a block with a large variance -> which indicate the complexity of the block Check if the block is on the edge or not by calculating the SAD between the candidate block and its 4 neighbors • If either one of the SAD is small -> the block is on edge -> reject • Else the block is not on edge
SSD is used instead of SAD
SSD = Sum of Squared Difference Last term, SAD is used as matching criteria SAD is commonly used because of its lower complexity (faster) But we chose to use SSD finally
SSD is used instead of SAD
Experiment shows that SSD has better performance in
accuracy
and Algorithm with SSD run as with SAD
fast
as that Reason (of why as fast as SAD): Elimination effects from SEA, PPNM & PDE become large when SSD is used Compensate the higher complexity of SSD
SSD is used instead of SAD
SEA and PPNM lower bound is adjusted using the inequality (12) proposed in the following paper J.J. Francis and G. de Jager. Motion Estimation (2002) A Sum Square Error based Successive Elimination Algorithm for Block
Experimental Result on Symbian phone
Frame rate Maximum frame rate supported by Nokia 6600 is about 14 frames/sec Running our algorithm (1/0.007 frames/sec) once only for each frame do not slow down the displaying frame rate, in other word, the frame rate can still be 14 frames/sec, our algorithm doesn ’ t lag the display
Rotation Tracking Engine
Observation and Motivation As we rotate the phone, the object can still be tracked correctly (center of object roughly equals center of green box)
Rotation Tracking Engine
Our approach If two blocks are tracked at the same time, angle of line connecting the two blocks reflect the tiling angle of the phone Two-block approach
Rotation Tracking Engine
Another approach Track one block by simple motion tracking engine, then find which rotation gives the best match Fail if tracking object is the same for different angle One-block approach
Rotation Tracking Engine
Modification of motion tracking algorithm to suit rotation tracking Linear Adaptive method used in translation motion tracking is not used here because phone ’ s motion can be both linear motion and circular motion
Rotation Tracking Engine
What to predict?
Given the coordinates of the tracking blocks in the last 2 previous frames Predict the coordinates of the blocks in the next frame Line L 1 Line L 2 Line L 3 θ θ A simplified mathematic problem (the following assumptions are approximately correct) All three lines pass through the circle ’ s center End points of the three lines lie on the circle
Rotation Tracking Engine
Solution Angle between L 2 and L 3 • θ = Tan -1 (slope(L 2 )) – = θ Tan -1 (slope(L 3 )) Coordinates of the next tracking block (x rotation matrix L1 , y of coordinates of the previous block with a L1 ) are calculated by multiplying the column matrix
x L
1
y L
1 cos sin sin cos
x L
2
y L
2
Rotation Tracking Engine
Solution The prediction of the position of the next tracking block should also take the translational movement into account Horizontal displacement = Tx • Tx = ( x L21 + x L22 – x L31 - x L32 )/2 Vertical displacement = Ty • Ty = ( y L21 + y L22 – y L31 - y L32 )/2 Coordinates of the next tracking block (x L1 , y L1 ) is calculated by
x L
1
y
1
L
1 cos sin 1 sin cos 1
Tx Ty
1
x L
2
y L
2 1
Rotation Tracking Engine
A simple drawing to show the detected rotation angle of the phone
Rotation Tracking Engine
Reducing error by using level Apply threshold on output, increase/decrease one level only when change is large To give less sensitive but more desirable output for game e.g. skiing game: skier face only to 7 directions Reduce difficulty, increase reliability A small rotation or a small detection error will not increase/decrease one level
Conditions to be a good background for both engines
Objective Max. performance measurement Increase usability Condition: Feature selection can always find a good feature point Within certain distance • No repeat pattern • Very distinct pattern Pattern is nearly the same when rotated. E.g. Circle which is good for rotation detection
Virtual Mouse
An application that make full use of the translational motion detection engine Remote control the mouse of PC by Symbian phone using motion tracking Advantages: It allows input in all directions It provides high levels of control Joystick can still be used as rough moving while camera input can be used as fine translation
Virtual Mouse
Server In server side (Windows), the Bluetooth is configured to provide RFComm Service and server regards the Bluetooth transmission port as Comm. Port Server receives message from that Comm. Port Call function in MouseAction.h to make the mouse move and trigger mouse click event
Virtual Mouse
Client Search for Bluetooth device nearby Connect to the selected Bluetooth device Every time motion tracking algorithm finishes, results are sent to server (12 times/sec) Joystick and keypad can also be used to control mouse in PC (Multi-button mouse)
Car Racing Game
A “ Car Racing Game cannot proceed Solutions: Double Buffering Direct Screen Access Single Thread ” is developed using the motion tracking engine.
Game lags. It is because: The engine takes time to find the motion vector CPU speed of the Symbian phones are low The game engine thread uses most of the CPU power while the thread for updating the screen
Double Buffered Area
Double-buffering – two complete color buffers One is displayed while the other is calculating next scene to be drawn, When the drawing of the next frame is completed, the content of the back buffer is copied to the front screen
Front Screen Frame 1
Computing Frame 2 Show Frame 2 Computing ……
Frame 2 Frame 2 ready Buffer 1 Buffer 2 copy
Direct Screen Access
In Symbian, traditional drawing requires the help of Window Server Overhead in context switching Lower speed Access the screen directly in 3 ways: Creating and using CfbsScreenDevice Accessing screen memory directly.
Using CdirectScreenAccess.
Using DSA can bypass the Window Server Get rid of context switching Faster
Traditional Graphic Programming
Device Screen and Keypad Key Presses Update Display Window Server RWsSession Application 1 RWsSession Application 1 RWsSession Application 1
Direct Screen Access
1.
Request to Window Server to work in Direct Screen Access mode Application 2. If successful, a region for drawing to is returned to the application.
Window Server
Single Thread
Prevent context switching No need to do synchronization
Car Racing Game
With the use of efficient graphic programming algorithms, the tracking engine will not affect the performance of the game It shows that we can use the engine to develop other applications without performance degradation
Skiing Game
Sample program from the book Developers ” “ Developing Series 60 Applications: A Guide for Symbian Os C++ Based on this game, we plug in our rotation tracking engine into it The game becomes more interactive Rotate the phone to control the angle of the skier
Skiing Game
Skiing Game
The process of using the engine is very simple Create an instance of the tracking engine Call the function of the engine to find the motion vector Use the motion vector for the game logic
Skiing Game
Demo
Experimental Result on Symbian phone
Testing Environment Platform • Symbian Phone Nokia 6600 Algorithm • Our final (hybrid-type) Algorithm (Block matching algorithm featured with Adaptive Window, Spiral Scan, SEA, PPNM and PDE method) Algorithm parameter • Block size = 17 x 17 (pixels) • Search window size = 16 x 16 (pixels) Number of block to track in each run of algorithm = 1 block ONLY
Experimental Result on Symbian phone
Testing Result (avg time to run) Final algorithm • 7ms Final algorithm without adaptive window method and SEA, PPNM method • 22ms Full Exhaustive Search algorithm (the most simplest one) • 55ms