Streaming Multigrid for Gradient-Domain Operations on Large Images Misha Kazhdan and Hugues Hoppe Johns Hopkins University Microsoft Research.
Download ReportTranscript Streaming Multigrid for Gradient-Domain Operations on Large Images Misha Kazhdan and Hugues Hoppe Johns Hopkins University Microsoft Research.
Streaming Multigrid for Gradient-Domain Operations on Large Images Misha Kazhdan and Hugues Hoppe Johns Hopkins University Microsoft Research Stitching Image Panoramas Gradient Domain Fix: Seams in the panorama Different exposures Set seam-crossing gradients to zero. Courtesy of Uyttendaele High Dynamic Range Compression GradientNoDomain Fix: exposes all the detail single image Amplify small gradients and dampen large ones. Short Exposure Medium Exposure Long Exposure Courtesy of Industrial Light & Magic Tone-Mapped Image Gradient-Domain Image Processing Many image processing operations are easier to implement in the gradient-domain. Image0 Grad0 Image1 Grad1 Imagen Gradn Gradient Domain Operator Grad Solver Image Gradient-Domain Image Processing Many image processing operations are easier to implement in the gradient-domain. Image0 Grad0 Image1 Grad1 Imagen Gradn Removing Lighting Effects Range Compression Image Compositing Image Stitching Image Authoring Gradient Gradient Domain Domain Operator Operator Grad Solver Image [Horn ’74, Weiss ’01, Finlayson ’02, Agrawal ’05] [Fattal ’02, Weyrich ’07] [Perez ’03, Agarwala ’04, Jia ’06] [Levin ’04, Agarwala ’07] [McCann ’08] Gradient-Domain Image Processing Many image processing operations are easier to implement in the gradient-domain. Image0 Grad0 Image1 Grad1 Imagen Gradn Removing Lighting Effects Range Compression Image Compositing Image Stitching Image Authoring Gradient Domain Operator Grad Solver Solver Image [Horn ’74, Weiss ’01, Finlayson ’02, Agrawal ’05] [Fattal ’02, Weyrich ’07] [Perez ’03, Agarwala ’04, Jia ’06] [Levin ’04, Agarwala ’07] [McCann ’08] Outline • Introduction • Solving for the Image – The Poisson Equation – Solving the Linear System • Choosing the System • Solving for Big Images • Results and Discussion The Poisson Equation Gradient-domain processing requires fitting an image to^local-difference constraints: averaged u g div(uu) div f (g) f=div(g) g u 11 Gauss-Seidel Solvers Gradient-domain processing requires fitting an image to^local-difference constraints. averaged u g u f Iterative Algorithm: For each pixel (i,j) 1. Assume the rest of the pixel values are correct 2. Update u[i][j] to satisfy the Laplacian: u[i][j] (Average of Neighbors) + f[i][j] Gauss-Seidel Solvers This quickly solves for the high frequencies, but the low frequencies converge slowly. Desired Solution 2 Iterations 2 Iteration Residual 10 10 Iterations Iteration Residual 100100 Iterations Iteration Residual Multigrid Solvers V-Cycle Algorithm: 1. 2. 3. 4. 5. Perform Compute Down-sample, Sum thethe a few the low-res residual solve, updates and high-res and image constraints on up-sample the results TheRefine process ishigh-res recursed for thehigh-res low-resimage problem. + Desired Best-Guess Iterated Best-Guess Iterated Residual Desired Solved Up-Sampled Outline • • • • • Introduction Solving for the Image Choosing the System Solving for Big Images Results and Discussion Choosing the System The choice depends on the interpretation of pixels as elements of a continuous function. th st nd-order 0 1 2rd -orderinterpretation interpretation 3 Pixel values Elements Choosing the System Implementing Up-/Down-Sampling: The nesting of the elements determines the transition between the levels of the solver. 0th-order elements 1 1 1st-order elements 1 1/2 2nd-order elements 3rd-order elements 3/4 3/4 1/2 1/4 1/4 1/2 1/8 3/4 1/2 1/8 Choosing the System Defining the Laplacian: The neighborhood of the Laplacian is defined by the element overlap. 0th-order elements 1st-order elements 2nd-order elements 3rd-order elements Undefined 1-Ring Average 2-Ring Average 3-Ring Average Second-Order Elements Which order element converges fastest to an accurate solution? 1st-order error Average Error 256 1st-Order 2nd-Order 3rd Order Original Image 2nd-order error Image Gradients 1 1/256 0 5 GS Updates 3rd-order error Second-Order Elements Spectral Analysis of Convergence: The eigenvalues of the multigrid operator determine how efficiently the residual is dampened. Two-Grid Spectra (k=5) G.S Update G.S Update DownSample UpSample G.S Update Two-Grid Operator Eigenvalue 0.4 1st-order 2nd-order 3rd-order 0.2 0.0 0 Eigenvalue Index 32 Outline • • • • Introduction Solving for the Image Choosing the System Solving for Big Images – Naïve Out-of-Core Multigrid – Temporal Blocking – Interleaved Multilevel Streaming • Results and Discussion Naïve Out-of-Core Multigrid Components of the solver: 1. Gauss-Seidel updates 2. Down-sampling 3. Up-sampling The multigrid solver can be streamed by advancing a fixedsize window. Local data access On Disk In Memory Naïve Out-of-Core Multigrid The large linear system can be solved by streaming each step of the V-cycle. Constraints G.S. Update k G.S. Update k Solution Residual UpSample DownSample G.S. Update k k G.S. Update Residual DownSample UpSample k Gauss-Seidel Updates Low-Res 2(k+2) Streaming Solve Passes Disk Streaming Temporal Blocking A pixel can be updated for the kth time only if all neighbor other pixels have been updated k-1 times. Temporal Blocking Window A pixel can be updated for the kth time only if all neighbor other pixels have been updated k-1 times. 2nd-order elements k=2 Gauss-Seidel Updates 0 2 2 1 0 1 0 2 0 1 1 0 2 0 1 1 0 1 0 0 1 0 0 0 Temporal Blocking A pixel can be updated for the kth time only if all neighbor other pixels have been updated k-1 times. Constraints G.S. Update G.S. Update Solution Residual UpSample DownSample G.S. Update G.S. Update Residual UpSample DownSample Low-Res Solve Disk Streaming Interleaved Multilevel Streaming piped into Processing at one level is buffered for the next. Interleaved Multilevel Streaming Residual piped into Processing at one level is buffered for the next. High-Resolution 02 2 1 0 1 02 2 0 1 1 02 2 0 1 1 02 2 0 1 1 0 2 0 1 1 0 2 0 1 1 0 2 0 1 1 0 2 0 1 1 0 2 0 1 1 0 1 0 0 1 0 0 0 Low-Resolution ¼ ¾ ¾ ¼ ¼ ¾ ¾ ¼ ¼ ¾ ¾ ¼ 0 1 0 0 0 3/4 3/4 1/4 1/4 2nd-order elements Interleaved Multilevel Streaming piped into Processing at one level is buffered for the next. Streaming Pass 1 Constraints Streaming Pass 2 G.S. Update G.S. Update Solution Residual UpSample DownSample Globality of the Poisson equation G.S. G.S. Update Update Two streaming passes is optimal Residual UpSample DownSample Low-Res Solve Memory Streaming Disk Streaming Streaming through the Gradient-Domain For gradient-domain operators that are local, processing only takes two streaming passes. +1 streaming pass for each additional V-cycle. u0 u1 g0 g1 … … … Op un g f Pass 1 Pass 2 Update Update Pass 3 u Update gn Update Update Memory Streaming Disk Streaming Solve Update Update Solve u Outline • • • • • Introduction Solving for the Image Choosing the System Solving for Big Images Results and Discussion Stitching Big Images Agarwala ’07: A high-res solution is only required near the seams, so solve over an adapted quadtree. Courtesy of Agarwala Stitching Results Rainier: 10,9602,096 Beynac: 6,0461,920 Edinburgh: 16,9502,956 Redrock: 19,5884,457 Dataset Pixels Images Error [0,256) Memory (MB) Time (seconds) Quadtree Streaming Quadtree Streaming Quadtree Streaming Beynac 12x106 3 0.01 0.01 16 Rainier 23x106 5 0.02 0.01 17 Edinburgh 50x106 25 0.01 123 203 122 79 Redrock 87x106 9 0.01 112 133 118 118 190 8 17 Courtesy of Agarwala and Curless 110 14 33 Tone-Mapping Results Bring out subtle details by amplifying small gradients and dampening large ones. Large Image Results St James: • Stitched from 643 photographs • Contains 3.3 billion (88,309 x 37,842) pixels Stitched Memory: 408 MB Time: 1:27:50 Max Error [0,256): 0.15 Tone-Mapped Memory: 224 MB Time: 45:10 Max Error [0,256): 0.5 Courtesy of Uyttendaele Large Image Results St James: • Peak Memory: 408 MB • Solver Time: 1:27:50 53GB in-core 1:19:02 I/O Time Total disk I/O: • Down-Sampling – Read in the gradient constraints: – Write out solution and Laplacian: 40 GB 53 GB • Up-Sampling – Read in solution and Laplacian: – Write out the image: 53 GB 20 GB Courtesy of Uyttendaele Summary Streaming multigrid for big image processing: – 2nd order elements Accurate solution in – Temporal blocking two streaming passes – Interleaved streaming Issues addressed in the paper: – Integration with finite-differences – Setting the average color – Non power-of-two images Issues for Future Work 1. Non-trivial boundaries and constraints? e.g. cut-and-paste, matting 2. Adaptive weighting? e.g. local tone adjustment 3. Extensions to higher dimensions? for incompressible fluid simulation 4. Less I/O in the streaming passes? on-the-fly compression 5. Really big images? parallelized and distributed solvers Acknowledgments Datasets: Aseem Agarwala Brian Curless Matt Uyttendaele Discussions and Ideas: Ketan Dalal Bill Bolosky CAVGRAPH Reviewers Rick Szeliski Ann Wolverton SIGGRAPH Reviewers Thank You! Code & Images @ http://www.cs.jhu.edu/~misha/Code/SMG General Temporal Blocking … k Updating the 3rd row from front, The row two behind that, The row two behind that, Window To perform k updates: k=3 Gauss-Seidel Updates 2 0 3 3 0 1 1 2 2 0 3 0 1 1 2 2 0 3 0 1 1 2 1 0 2 0 1 1 0 2 0 1 0 0 1 0 0 1 0 0 General Temporal Blocking with d-th order elements To perform k updates:^ Window k (d+1)-st Updating the 3rd row from front, d The row two behind that, d behind that, The row two … k=3 Gauss-Seidel Updates 3 23 23 12 12 01 01 0 0 Finite Differences to Finite Elements 1st-order Element Finite Differences Differentiation 0th-order Elements Coefficients of Derivative Elements The derivative of a d-th order element is the difference of elements of order d-1. Second-Order Elements Which order element converges fastest to an accurate solution? Average Error Average Error 256 1 1st-order error 256 1st-Order 2nd-Order 3rd-Order Original Image Image Gradients 1st-Order 2nd-Order nd 2 -order error 3rd-Order 1 1/256 1/256 1/256 0 5 10 GS Updates 15 20 0 3rd-order error1000 Operations