#### Transcript ppt - TAMU Computer Science Faculty Pages

Scale-Invariant Feature Transform (SIFT) Jinxiang Chai Review Image Processing - Median filtering - Bilateral filtering - Edge detection - Corner detection Review: Corner Detection 1. Compute image gradients 2. Construct the matrix from it and its neighborhood values 3. Determine the 2 eigenvalues λ(i.j)= [λ1, λ2]. 4. If both λ1 and λ2 are big, we have a corner C( i , j ) I x2 I x I y I I I x y 2 y The Orientation Field Corners are detected where both λ1 and λ2 are big Good Image Features • What are we looking for? – Strong features – Invariant to changes (affine and perspective/occlusion) – Solve the problem of correspondence • Locate an object in multiple images (i.e. in video) • Track the path of the object, infer 3D structures, object and camera movement, Scale Invariant Feature Transform (SIFT) • Choosing features that are invariant to image scaling and rotation • Also, partially invariant to changes in illumination and 3D camera viewpoint Invariance • • • • Illumination Scale Rotation Affine Required Readings • Object recognition from local scaleinvariant features [pdf link], ICCV 09 • David G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, 60, 2 (2004), pp. 91-110 Motivation for SIFT • Earlier Methods – Harris corner detector • Sensitive to changes in image scale • Finds locations in image with large gradients in two directions – No method was fully affine invariant • Although the SIFT approach is not fully invariant it allows for considerable affine change • SIFT also allows for changes in 3D viewpoint SIFT Algorithm Overview 1. 2. 3. 4. Scale-space extrema detection Keypoint localization Orientation Assignment Generation of keypoint descriptors. Scale Space • Different scales are appropriate for describing different objects in the image, and we may not know the correct scale/size ahead of time. Scale space (Cont.) • Looking for features (locations) that are stable (invariant) across all possible scale changes – use a continuous function of scale (scale space) • Which scale-space kernel will we use? – The Gaussian Function Scale-Space of Image • L(x, y,k ) G(x, y,k ) * I(x,y) G( x, y, k ) - variable-scale Gaussian I ( x, y ) - input image Scale-Space of Image • L(x, y,k ) G(x, y,k ) * I(x,y) G( x, y, k ) - variable-scale Gaussian I ( x, y ) - input image • To detect stable keypoint locations, find the scale-space extrema in difference-ofGaussian function D( x, y, ) L( x, y, k ) L( x, y, ) D( x, y, ) (G( x, y, k ) G( x, y, )) * I ( x, y) Scale-Space of Image • L(x, y,k ) G(x, y,k ) * I(x,y) G( x, y, k ) - variable-scale Gaussian I ( x, y ) - input image • To detect stable keypoint locations, find the scale-space extrema in difference-ofGaussian function D( x, y, ) L( x, y, k ) L( x, y, ) D( x, y, ) (G( x, y, k ) G( x, y, )) * I ( x, y) Scale-Space of Image • L(x, y,k ) G(x, y,k ) * I(x,y) G( x, y, k ) - variable-scale Gaussian I ( x, y ) - input image • To detect stable keypoint locations, find the scale-space extrema in difference-ofGaussian function D( x, y, ) L( x, y, k ) L( x, y, ) D( x, y, ) (G( x, y, k ) G( x, y, )) * I ( x, y) Look familiar? Scale-Space of Image • L(x, y,k ) G(x, y,k ) * I(x,y) G( x, y, k ) - variable-scale Gaussian I ( x, y ) - input image • To detect stable keypoint locations, find the scale-space extrema in difference-ofGaussian function D( x, y, ) L( x, y, k ) L( x, y, ) D( x, y, ) (G( x, y, k ) G( x, y, )) * I ( x, y) Look familiar? -bandpass filter! Difference of Gaussian 1. A = Convolve image with vertical and horizontal 1D Gaussians, σ=sqrt(2) 2. B = Convolve A with vertical and horizontal 1D Gaussians, σ=sqrt(2) 3. DOG (Difference of Gaussian) = A – B 4. So how to deal with different scales? Difference of Gaussian 1. A = Convolve image with vertical and horizontal 1D Gaussians, σ=sqrt(2) 2. B = Convolve A with vertical and horizontal 1D Gaussians, σ=sqrt(2) 3. DOG (Difference of Gaussian) = A – B 4. Downsample B with bilinear interpolation with pixel spacing of 1.5 (linear combination of 4 adjacent pixels) Difference of Gaussian Pyramid Blur A3-B3 B3 DOG3 A3 Downsample A2-B2 B2 Blur DOG2 A2 Input Image Downsample A1-B1 Blur B1 DOG1 Blur A1 Other issues • Initial smoothing ignores highest spatial frequencies of images Other issues • Initial smoothing ignores highest spatial frequencies of images - expand the input image by a factor of 2, using bilinear interpolation, prior to building the pyramid Other issues • Initial smoothing ignores highest spatial frequencies of images - expand the input image by a factor of 2, using bilinear interpolation, prior to building the pyramid • How to do downsampling with bilinear interpolations? Bilinear Filter Weighted sum of four neighboring pixels x u y v Bilinear Filter y (i,j) Sampling at S(x,y): (i,j+1) u x v (i+1,j) S(x,y) = a*b*S(i,j) (i+1,j+1) + a*(1-b)*S(i+1,j) + (1-a)*b*S(i,j+1) + (1-a)*(1-b)*S(i+1,j+1) Bilinear Filter y (i,j) Sampling at S(x,y): (i,j+1) u x v (i+1,j) S(x,y) = a*b*S(i,j) (i+1,j+1) + a*(1-b)*S(i+1,j) + (1-a)*b*S(i,j+1) + (1-a)*(1-b)*S(i+1,j+1) To optimize the above, do the following Si = S(i,j) + a*(S(i,j+1)-S(i)) Sj = S(i+1,j) + a*(S(i+1,j+1)-S(i+1,j)) S(x,y) = Si+b*(Sj-Si) Bilinear Filter y (i,j) (i,j+1) x (i+1,j) (i+1,j+1) Pyramid Example A3 A2 A1 DOG3 B3 B2 DOG3 B1 DOG1 Feature Detection • Find maxima and minima of scale space • For each point on a DOG level: – Compare to 8 neighbors at same level – If max/min, identify corresponding point at pyramid level below – Determine if the corresponding point is max/min of its 8 neighbors – If so, repeat at pyramid level above • Repeat for each DOG level • Those that remain are key points Identifying Max/Min DOG L+1 DOG L DOG L-1 Refining Key List: Illumination • For all levels, use the “A” smoothed image to compute – Gradient Magnitude • Threshold gradient magnitudes: – Remove all key points with MIJ less than 0.1 times the max gradient value • Motivation: Low contrast is generally less reliable than high for feature points Results: Eliminating Features • Removing features in low-contrast regions Results: Eliminating Features • Removing features in low-contrast regions Assigning Canonical Orientation • For each remaining key point: – Choose surrounding N x N window at DOG level it was detected DOG image Assigning Canonical Orientation • For all levels, use the “A” smoothed image to compute – Gradient Orientation + Gaussian Smoothed Image Gradient Orientation Gradient Magnitude Assigning Canonical Orientation • Gradient magnitude weighted by 2D Gaussian with σ of 3 times that of the current smoothing scale = * Gradient Magnitude 2D Gaussian Weighted Magnitude Assigning Canonical Orientation Weighted Magnitude Gradient Orientation Sum of Weighted Magnitudes • Accumulate in histogram based on orientation • Histogram has 36 bins with 10° increments Gradient Orientation Assigning Canonical Orientation Weighted Magnitude Gradient Orientation Sum of Weighted Magnitudes • Identify peak and assign orientation and sum of magnitude to key point Peak Gradient Orientation Eliminating edges • Difference-of-Gaussian function will be strong along edges – So how can we get rid of these edges? Eliminating edges • Difference-of-Gaussian function will be strong along edges – Similar to Harris corner detector T Dxx Dxy I I x x H H (i , j ) D D yy xy I y I y I x2 I x I y – We are not concerned about actual values of eigenvalue, just the ratio of the two Tr (H) 2 ( ) 2 (r ) 2 (r 1) 2 2 Det(H) r r I I I x y 2 y Eliminating edges • Difference-of-Gaussian function will be strong along edges – So how can we get rid of these edges? Local Image Description • SIFT keys each assigned: – Location – Scale (analogous to level it was detected) – Orientation (assigned in previous canonical orientation steps) • Now: Describe local image region invariant to the above transformations SIFT: Local Image Description • Needs to be invariant to changes in location, scale and rotation SIFT Key Example Local Image Description For each key point: • Identify 8x8 neighborhood (from DOG level it was detected) • Align orientation to xaxis Local Image Description 3. Calculate gradient magnitude and orientation map 4. Weight by Gaussian Local Image Description 5. Calculate histogram of each 4x4 region. 8 bins for gradient orientation. Tally weighted gradient magnitude. Local Image Description 6. This histogram array is the image descriptor. (Example here is vector, length 8*4=32. Best suggestion: 128 vector for 16x16 neighborhood) Applications: Image Matching • Find all key points identified in source and target image – Each key point will have 2d location, scale and orientation, as well as invariant descriptor vector • For each key point in source image, search corresponding SIFT features in target image. • Find the transformation between two images using epipolar geometry constraints or affine transformation. Image matching via SIFT featrues Feature detection Image matching via SIFT featrues • Image matching via nearest neighbor search - if the ratio of closest distance to 2nd closest distance greater than 0.8 then reject as a false match. • Remove outliers using epipolar line constraints. Image matching via SIFT featrues Summary • SIFT features are reasonably invariant to rotation, scaling, and illumination changes. • We can use them for image matching and object recognition among other things. • Efficient on-line matching and recognition can be performed in real time