Transcript Slides
Real-Time Hierarchical Scene Segmentation and Classification Andre Uckermann, Christof Elbrechter, Robert Haschke and Helge Ritter John Grossmann Real-Time Scene Segmentation/Classification • Purpose: Correctly segment various objects in a scene for the end goal of classifying or identifying each segmented object. • Important for robots to be able to classify objects in real-time. o Collision Detection o Don’t want to harm humans o Grasping correct objects o Pointing gestures to specific objects or areas Existing Work Classify based on features from existing Object Model library • Limited to existing object models Segment objects in a scene based on generic rules • Color, Smoothness, Convexity • Not good with ambiguous situations Use World knowledge and heuristic or learned segment relations • Predict connectivity of object surface patches based on learned relations (too slow) • Employ Heuristic rules to connect surface patches in a weighted graph Improving the Existing Work Create a full hierarchy of grouping hypotheses • Tree-like connectivity structure of the scene • Results in spatial scene analysis grouping spatially connected objects Higher Level Decision Process to Determine Optimal Grouping Hypothesis • Implement world knowledge • Change level of granularity on task specific manner Model-Free Segmentation Pre-Segmentation into Surface Patches • Separate scene surfaces into uniform regions 1. Three-stage smoothing (median, temporal, Gaussian) 2. Calculation of surface normals from cross products 3. Detection of edges from angle between normal and distance of adjacent points 4. Connected component analysis to assign unique IDs to all surface regions 5. Assignment of edge points to closest surface region Connectivity Graph Creation Criteria 1. Cut-Free Adjacency 2. Co-planarity 3. Similar Curvature of Surface Pairs Cut-Free Adjacency • Consider adjacent surfaces • If one surface cuts another surface they most likely are not part of the same object Co-Planarity Combine object parts separated by occlusion Match shapes of non-adjacent surface patches Co-Planar surfaces must span a common plane Check if both surfaces are separated by occlusion or by space Curvature Matching Also combine object parts based on occlusion Compare normalized curvature histograms: 𝑆 𝐴, 𝐵 = min(𝑎𝑖𝑗 , 𝑏𝑖𝑗 ) 𝑖𝑗 Check if each surface is separated due to occlusion Handling Remaining Edge Points • Some larger edge-point blobs were not assigned to a neighbored surface patch during pre-segmentation • Normally small, separate objects • Separate object “blobs” by a region growing algorithm • Consider maxima in depth image as “separation-clefts” • All new found edges added as nodes to connectivity graph https://www.youtube.com/watch?v=gI7c9RC7gKg Hierarchical Segmentation Takes connectivity graph created in previous step, and attempts to find hierarchical clustering of graph nodes. Groups of increasing connectivity strength 1. Assignment of Initial Edge Weights (Cutting cost) 2. Fine-Tuning Edge Weights Initializing Edge Weights Assign edge weights: node 𝑖. 𝑤𝑖𝑗 = 1 𝑛 to all edges originating from Average incoming and outgoing edge weights: 𝑊= 1 2 𝑊𝑖 + 𝑊𝑗 Weights are normalized to [0…1] Edge cut between strongly connected nodes goes to 1 Fine-Tuning Edge Weights • Assign weights to edges based on criteria they were added from 𝑤𝑎 > 𝑤𝑝 > 𝑤𝑐 > 𝑤𝑟 • Cut-free adjacency, co-planarity, curvature, remaining points • Multiply initial edge weights by these feature related weights Hierarchical Segmentation Tree Each least cut creates a new branch • Repeat until all edges cut Each leaf is a single surface The branching nodes are potential grouping hypotheses Hierarchical Classification • Fast NN-based classifier traverses tree from top down • Computes class probability distribution of the object hypothesis at each branching node • Peaks of classification confidence will indicate salient entities of objects or groupings of objects • Segmentation hierarchy efficiently connects: 1. Generic low-level regularities about structures governing segmentation (surfaces and edges) 2. Specialized, high level feature correlations indicative of objects and object parts or configurations (handle and mug, door and door knob) Nearest-Neighbor Classifier • Uses a bag of features model • For each feature type, there is a normalized distance function between a pair of features • Features which exactly match have 0 distance • The distance between feature vectors is the max of all individual feature distances • Adding new features and classes can be performed very fast • Main features used are a Size, Elongation and Color Feature: Size • Volume of the surface segment • Use PCA of the surface point cloud to get eigenvalues 𝑣= λ𝑖 𝑖 Distance function: 𝑑𝑣𝑜𝑙𝑢𝑚𝑒 (𝑣1 , 𝑣2 ) = |𝑣1 − 𝑣2 | max(𝑣1 ,𝑣2 ) Feature: Elongation Ratio between length and thickness of an object • Also computed from PCA eigenvalues 𝑒 = min λ1 λ2 , Use same distance function as size λ2 λ1 Feature: Color • Use normalized, non-uniformly binned hue/saturation histogram • Lightness value is ignored for robustness against changing light • Histograms are blurred by applying an 11x3 Gaussian filter • Distance function: 𝑑𝑐𝑜𝑙𝑜𝑟 𝐻1 , 𝐻2 = 1 2 ℎ,𝑠 | 𝐻1ℎ,𝑠 − 𝐻2ℎ,𝑠 | Computes the overlap between histograms Applying the Classifier https://www.youtube.com/watch?v=gI7c9RC7gKg