Transcript Gradient-Based Image Editing

Gradient-Based Image Editing
Poisson Image Editing
Patrick Perez, Michel Gangnet and Andrew Blake
Microsoft Research UK
Poisson Image Editing
Outline


Introduction
Poisson solution to guided interpolation
◦ Guided Interpolation
◦ Discrete Poisson solver

Seamless cloning
◦ Importing gradients
◦ Mixing gradients

Selection editing
◦
◦
◦
◦

Texture flattening
Local illumination change
Local color change
Seamless tiling
Conclusion
Poisson Image Editing
Introduction

A generic interpolation machinery based on
solving Poisson equations

A variety of novel tools for seamless editing of
image regions
◦ Import the source image region to a target region
◦ Modify appearance seamless in selected image region:
illumination, color, texture, etc.
Poisson Image Editing
Introduction

Image editing
◦ Global changes (color/intensity corrections, Filters,
deformations)
◦ Local changes confined to a selection
 Selected region
 Seamless
 Changes: slight distortion -> complete replacement
Classic tools:
Image filters
Cut-and-Paste
Seams: partly hidden by feathering
Poisson Image Editing
Introduction

Seamless editing and cloning of a selection
region

Mathematical tool
◦ Poisson partial differential equation
 Laplacian of unknown function over the domain of interest
◦ Dirichlet boundary conditions
 Known function value along the boundary
Poisson Image Editing
Boundary Conditions

Dirichlet boundary condition
◦ It specifies the values of a solution needs to take on
the boundary of the domain

T  f (t ) ,
at x  0 , t  0
T  T1 ,
T  T0 ,
at x  1 , t  0
at x  0 , t  0
Neumann boundary condition
◦ It specifies the values of the derivative of a solution is
to take on the boundary of the domain
T
 f (t ) , at t  0 ,
x
0  x 1
T
 0 , at x  1 ,
x
t0
Poisson Image Editing
Poisson Equation
Second-order variations extracted by
Laplacian operator are the most
significant perceptually
 Scalar function on a bounded domain is
uniquely defined by its values on the
boundary and its Laplacian in the interior

◦ Poisson equation therefore has a unique
solution
Poisson Image Editing
Poisson Equation

Solving Poisson equation == interpretation
as minimization problem
◦ Gradient of the function is the closest (in L2-norm)
to some prescribed vector field (Guidance vector
field) under given boundary conditions

Possible choice of guidance vector filed
◦ Source image
◦ Mixing suitably gradient of source and target
image
◦ Modify a region of the image
Poisson Image Editing
Related Work

Poisson equation has been used
extensively in computer vision
◦ Rescale gradient field of High Dynamic Range
(HDR) image
 Solving Poisson equation with Neumann boundary
◦ Edit image via a sparse set of edge elements
 Solving a Laplace equation with Dirichlet boundary
◦ Spot removing
 Replace the brightness by harmonic interpolation
(solving a Laplace equation)
Poisson Image Editing
Poisson Solution to Guided Interpolation

Guided Interpolation
S: Image domain
Closed
subset of S
Guidance vector field
Gradient field of a
source function
f*:Destination function
f: Unknown function
Poisson Image Editing
Guided Interpolation

Membrane interpolant
2
min  f with
f

f |  f * |
 . . 
gradient operator .   , 
 x y 
The minimizer must satisfy the associated Euler-Lagrange equation
f  0 over  with f |  f * |
 2. 2. 
Laplacian operator .   2 , 2 
 x y 
Poisson Image Editing
Guided Interpolation

Guidance filed: a vector field v
2
min  f  v with
f

f |  f * |
f  divv over  with f |  f * |
u v
divv 

x y
Divergence of v=(u,v)
The fundamental machinery of Poisson editing of color images:
independently solve the three color channels
Poisson Image Editing
Guided Interpolation
With a conservative vector field v (it is the
gradient of some gradient function)
 Correction function f  g  f
Poisson equation -> Laplace equation

f  divv over  with f |  f * |
f  0 over  with f |  ( f *  g ) |
Membrane interpolant
Poisson Image Editing
Discrete Poisson solver


Can be discretized and solved in a number of
ways
S,  become finite point sets defined on a
discrete grid
◦ For each pixel p in S, Np is its 4-connectd
neighbors in S
◦ The boundary   { p  S \  : N p  }
◦ Pixel pair
 p , q , q  N p
The task is to compute the pixel values over
the region
f |  { f , p  }

p
Poisson Image Editing
Discrete Poisson solver

Dirichlet boundary conditions defined on a
boundary of arbitrary shape, so discrete the
problem directly rather than the Poisson
equation
2
min  f  v with
f
min
f |

 p , q 

 f p  f q  v pq  with
2
f |  f * |
f p  f p* , for all p  
 pq
v pq  v 
  pq
 2 
Poisson Image Editing
Discrete Poisson solver

Solution: linear equations
for all p  ,

Np fp 

qN p 
fq 

qN p 
f q* 
v
qN p
pq
For all pixels p interior to  , no
boundary terms
Np fp 

qN p 
fq 
v
qN p
pq
A classical, sparse, symmetric, positive-definite system
Poisson Image Editing
Discrete Poisson solver

Arbitrary shape of boundary
iterative solvers

, use
◦ Gauss-Seidel iteration
◦ V-cycle multigrid
◦ Both are fast enough for interactive editing
Poisson Image Editing
Seamless Cloning

Importing gradients
◦ Gradient field directly from source image g
*

f

div
v
over

with
f
|

f
|
v  g

f  g over , with f |  f * |
for all p, q , v pq  g p  gq
Several strokes were used
to select source pieces
Poisson Image Editing
Seamless Cloning

Very
simple user
interaction
by drawing
outlines
Poisson Image Editing
Feature Exchange

Draw precise
boundary for
specific object
Poisson Image Editing
Seamless Cloning

Transfer only part of the source content
Poisson Image Editing
Inserting object with holes
Color-based cut-and-paste
Seamless cloning and destination averaged
Seamless cloning
Mixed seamless cloning
Mixing gradients

Linear combination of source and
destination gradient fields
v  g   f

Non-conservative guidance fields
*
*

f (x) if f (x)  g (x)
for all x  , v(x) = 
otherwise

 g (x)
*
*
*
*

f

f
if
f

f
 p
q
p
q  g p  gq
v pq = 
otherwise

 g p  gq
Poisson Image Editing
Mixing gradients

Transparent object
Poisson Image Editing
Mixing Graidents
Source/destination
Seamless cloning
mixed seamless cloning
Poisson Image Editing
Selection Editing
Modify one image in its gradient domain
 Texture flattening
 Spatially selective illumination changes

Non-linear modification of the original gradient field
Background and foreground color modification
 Seamless titling

Modify the original image, provide a new source
image or new boundary
Poisson Image Editing
Texture Flattening
for all x , v(x)=M (x)f * (x)

M: a binary mask tuned on at a few locations of
interest
 f p  fq
v pq  
 0

if an edge lies between p and q
otherwise
Preserve main structure
Poisson Image Editing
Local Illumination Changes

Modify the gradient of
original image then use
it as a new guided
vector field

 
v   f
f 

Modify locally the
apparent illumination
◦ Highlight under-exposed
foreground object
◦ Reduce specular
reflections
Poisson Image Editing
Local Color Changes

Given a source color image g
◦ The destination image f* is set to be the luminance
channel from g,
◦ The user selects a region containing the object, and
this may be some what bigger than the actual object
◦ Modify the colors of background or in the selected
region as new destination/source image
◦ The Poisson equation is solved in each color channel.
Poisson Image Editing
Local Color Changes
Original Image
Decolorization:
f* is the
luminance of g;
new source image by
multiplying the RGB
channels by 1.5, 0.5,
0.5
Poisson Image Editing
Seamless tiling


Seamless tiling by enforcing periodic boundary
conditions with Poisson solver
Use original image as source image, set identical
boundary from opposite sides of the original boundary
f
*
north
 f
*
south
 0.5( g north  g south )
Poisson Image Editing
Conclusion
A Generic framework of guided interpolation
 A series of tools to edit in a seamless and
effortless manner the contents of an image
selection
 The modification can be:

◦ Replacement by
◦ Mixing with other images
◦ Alternations of some aspects of the original image,
such as texture, illumination or color

No need for precise object delineation
Poisson Image Editing
Extensions to Poisson Editing
3D mesh editing
 Guided vector field
 Image matting
 Good boundary finding

Poisson Image Editing
Drag-and-Drop Pasting
http://www.cse.cuhk.edu.hk/~leojia/all_project_web
pages/ddp/drag-and-drop_pasting.html
Detail Preserving Shape
Deformation in Image Editing

http://graphics.cs.uiuc.edu/~huifang/def
ormation.htm
photomontage

http://grail.cs.washington.edu/projects/p
hotomontage/
Efficient Gradient Domain Editing

http://agarwala.org/efficient_gdc/#comp
arisons