#### Transcript Human pose recognition

1. Introduction 2. Article [1] Real Time Motion Capture Using a Single TOF Camera (2010) 3. Article [2] Real Time Human Pose Recognition In Parts Using a Single Depth Images(2011) Fig From [2] Why do we need this? Robotics Smart surveillance virtual reality motion analysis Gaming - Kinect Microsoft Xbox 360 console “You are the controller” Launched - 04/11/10 In the first 60 days on the market sold over 8M units! (Guinness world record) http://www.youtube.com/watch?v=p2qlHoxPioM mocap using markers – expensive Multi View camera systems – limited applicability. Monocular – simplified problems. Time Of Flight Camera. (TOF) Dense depth High frame rate (100 Hz) Robust to: Lighting shadows other problems. 2.1 previous work 2.2 What’s new? 2.3 Overview 2.4 results 2.5 limitations & future work 2.6 Evaluation Many many many articles… (Moeslund et al 2006–covered 350 articles…) (2006) (2006) (1998) TOF technology Propagating information up the kinematic chain. Probabilistic model using the unscented transform. Multiple GPUs. 1. Probabilistic Model 2. Algorithm Overview: Model Based Hill Climbing Search Evidence Propagation Full Algorithm 15 body parts DAG – Directed Acyclic Graph pose X { X i } N t t i 1 DBN– Dynamic Bayesian Network speed V t range scan z t dynamic Bayesian network (DBN) Assumptions V t | V t 1 ~ N (V t 1 , ) Use ray casting to evaluate P ( X t V t X t 1 ) 1 i zk i i distance from measurement. Goal: Find the most likely states, given previous frame MAP, i.e.: Xˆ t , Vˆt arg m ax X t ,V t log P ( z t | X t , V t ) log P ( X t , V t | Xˆ t 1 , Vˆt 1 ) Fig From [1] 1. Hill climbing search (HC) 2. Evidence Propagation –EP Grid around i i P (V t | V t 1 ) 0 .0 5m 0 .0 5m i Sample Vt evaluate likelihood choose best point! i Calculate X t from V t , Xˆ t 1 Coarse to fine Grids. Fig From [1] The good: Simple Fast run in parallel in GPUS The Bad: Local optimum Ridges, Plateau, Alleys Can lose track when motion is fast ,or occlusions occur. Also has 3 stages: 1. Body part detection (C. Plagemann et al 2010) 2. Probabilistic Inverse Kinematics 3. Data association and inference Bottom up approach: 1. Locate interest points with AGEX – Accumulative Geodesic Extrema. 2. Find orientation. 3. Classify the head, foots and hands using local shape descriptors. Fig From [3] Results: Fig From [3] p i { H ead , H ands , Legs }i 1 of X 5 pˆ j ( j 1, ..., N ) Assume Correspondence p i pˆ j Need new MAP conditioned on Xˆ t 1 , pˆ j . Problem – p i (V t , Xˆ t 1 , Vˆt 1 ) isn’t linear! Solution: Linearize with the unscented Kalman filter . Easy to determine P (V t | Vˆt 1 , Xˆ t 1 , pˆ j ) . X’>Xbest? {( p i , pˆ j )} Experiments: 28 real depth image sequences. Ground Truth - tracking markers. M avg || m i mˆ i || i 1 M , m i – real marker position mˆ i – estimated position avg 0.1m perfect tracks. avg 0.3 m fault tracking. Compared 3 algorithms: EP, HC, HC+EP . best – HC+EP, worse – EP. Runs close to real time. HC: 6 frames per second. HC+EP: 4-6 frames per second. Fig From [1] Extreme case – 27: Lose track HC HC+EP Fig From [1] Limitations: Manual Initialization. Tracking more than one person at a time. Using temporal data – consume more time, reinitialization problem. Future work: improving the speed. combining with color cameras fully automatic model initialization. Track more than 1 person. Well Written Self Contained Novel combination of existing parts New technology Achieving goals (real time) Missing examples on probabilistic model. Not clear how X0 is defined Extensively validated: Data set and code available not enough visual examples in article No comparison to different algorithms 2.1 previous work 2.2 What’s new? 2.3 Overview 2.4 results 2.5 limitations & future work 2.6 Evaluation Same as Article [1]. Using no temporal information – robust and fast (200 frames per second). Object recognition approach. per pixel classification. Large and highly varied training dataset . Fig From [2] 1. Database construction 2. Body part inference and joint proposals: Goals: computational efficiency and robustness Pose estimation is often overcome lack of training data… why??? Huge color and texture variability. Computer simulation don’t produce the range of volitional motions of a human subject. Fig From [2] Fig From [2] 1. Body part labeling 2. Depth image features 3. Randomized decision forests 4. Joint position proposals 31 body parts labeled . The problem now can be solved by an efficient classification algorithms. Fig From [2] Simple depth comparison features:(1) d I ( x ) – depth at pixel x in image I, offset ( u , v ) normalization - depth invariant. computational efficiency: no preprocessing. Fig From [2] How does it work? Pixel x Node = feature f and a threshold Classify pixel x: P (c | I , x ) 1 T T P (c | I , x ) t t 1 Fig From [2] Training Algorithm: 1M Images – 2000 pixels Per image ( , ) arg m ax G ( ) G ( ) H ( Q ) s ( l , r ) | Q s ( ) | |Q | H ( Q s ( )) *H-antropy Training 3 trees, depth 20, 1M images~ 1 day (1000 core cluster) 14 1M images*2000pixels*2000 f *50 = 2 10 com putations ... Trained tree: Fig From [2] Local mode finding approach based on mean shift with a weighted Gaussian kernel. Density estimator: N f c ( xˆ ) i 1 x xi w ic exp bc w ic P ( c | I , x i ) d I ( x i ) 2 2 Fig From [4] Experiments: 8800 frames of real depth images. 5000 synthetic depth images. Also evaluate Article [1] dataset. Measures : 1. Classification accuracy – confusion matrix. 2. joint accuracy –mean Average Precision (mAP) results within D=0.1m –TP. Fig From [2] high correlation between real and synthetic. Depth of tree – most effective Fig From [2] Comparing the algorithm on: real set (red) – mAP 0.731 ground truth set (blue) – mAP 0.914 mAP 0.984 – upper body Fig From [2] Comparing algorithm to ideal Nearest Neighbor matching, and realistic NN - Chamfer NN. Fig From [2] Comparison to Article[1]: Run on the same dataset Better results (even without temporal data) Runs 10x faster. Fig From [2] Full rotations and multiple people Right-left ambiguity mAP of 0.655 ( good for our uses) Result Video Fig From [2] Future work: better synthesis pipeline Is there efficient approach that directly regress joint positions? (already done in future work - Efficient offset regression of body joint positions) Well Written Self Contained Novel combination of existing parts New technology Achieving goals (real time) Extensively validated: Used in real console Many results graphs and examples (Another pdf of supplementary material) Broad comparison to other algorithms data set and code not available [1] Real Time Motion Capture Using a Single TOF Camera (V. Ganapathi et al. 2010) [2] Real Time Human Pose Recognition In Parts Using a Single Depth Images(Shotton et al. & Xbox Incubation 2011) [3] Real time identification and localization of body parts from depth images (C. Plagemann et al. 2010) [4] Computer Graphics course (046746), Technion.