Transcript Graphcut Textures:Image and Video Synthesis Using Graph Cuts

Graphcut Textures:Image and
Video Synthesis Using Graph
Cuts
Vivek Kwatra ,Arno Schodl ,Irfan Essa ,
Greg Turk ,Aaron Bobick
Outline

Texture Synthesis



Graphcut Texture





What is texture?
How to synthesis?
Main idea, Contribution
Patch placement and matching techniques
Patch fitting
Refinements and extensions
Video Textures
Texture

Generate a large image from smaller image
or longer video from smaller one

How to do it?


Copy patches (or pixels ) from input to output
Problems

Artifacts ( e.g., boundaries of patches)
Solution

Copy patches from input to output with overlap
Definitions


Where to position the input texture called Offset
Which part of the input texture to transfer called Seam
Cutting the graph
overlap
7
2
5
8
3
6
9
8
4
8
old
pixels
1
new
patch
8
corresponding
graph
8
new
patch
B
8
existing
pixels
A
cut
8
cut
Use Max Flow/ Min Cut algorithm to find the best cut where the cost between two
adjacent nodes s and t :
M(s,t,A,B) = ||A(s) – B(s)|| + || A(t) – B(t)||
A(s) and B(s) are pixel colors in position s in the old and new patch respectively
Cutting the graph
4
7
corresponding
graph
2
5
8
3
6
9
8
new
patch
1
8
cut
cut
8
8
existing
pixels
8
old
pixels
8
8
overlap
new
patch
old cut
1
5
9
13
cut
overlap
old
pixels
new cut
corresponding
graph
8
existing
pixels
2
6
10
14
3
7
11
15
4
8
12
16
new
patch
Seam nodes
Problem: What should we do if we lay down a patch in a spot that already has
existing seams? How should we take preexisting seams into account when making
a cut?
old cut
new cut
new cut
4
7
old
pixels
2
5
8
3
6
9
8
overlap
8
corresponding
graph
8
new
patch
B
8
existing
pixels
A
8
1
8
old cut
new
patch
Seam nodes
Solution: add a seam node between two pixels and connect the seam node with
an arc to the new patch. The weight of the arc will be the old cost between the
pixels.
old cut
new cut
new cut
1
5
8
8
S1
8
S2
corresponding
graph
old
pixels
8
new
patch
B
7
2
S3
overlap
8
8
existing
pixels
A
4
3
6
9
S4
Then add arcs from pixels to seam node with new costs
1
8
A
M(1,2, B, A)
S1
4
7
8
M(1,2, A, B)
8
old cut
B
new
patch
The Synthesis Process

Step1:Patch placement and Matching
( Choose candidate patches or offset)




Random Placement
Entire Patch Matching
Sub-patch Matching
Step2:Patch Fitting ( Choose optimal portion
or seam)


Only those pixels are copied that are chosen by
graph-cut algorithm
Cost of graph-cut is a measure of similarity
Stemp1. Patch Placement

Random Placement



Entire input image is translated to random location
in the output image
Good results for random textures
Patch Matching ( Entire or Sub )



Used when we already have some patches in the
output image (refinement).
Every seam has a cost (min graph cut cost)
Uses error region
Sub-Patch Matching
Entire Patch Matching
Error Region

Error = seam cost




Sum of costs along minimum cut patch
Choose a pixel with largest error
Select a region around that pixel, called error
region
Patch Matching ( entire or sub ) will select
those patches that completely cover our error
region
Patch Placement

Entire Patch Matching


Search for translated input versions and choose
that gives best match
Matching criteria:

Normalized SSD:

C(t): The smaller, the better (means similar)
Patch Placement


Compute cost for all possible offsets. Cost is
inversely proportional to similarity
Choose the cost that has the highest probability of
resulting in a similar region


A low value of k leads to picking of only those patch
locations that have a very good match with the output
whereas a high value of k leads to more random patch
selection.
Good results for structured and semi-structured texture
Patch Placement

Sub-Patch Matching



Pick a small sub-patch from output
Search for output-patch in input texture or look for
translations of input sub-patch
Matching criteria:


Use the same probability function
Best results for unstructured regions or video
textures ( fire, waves, smoke, etc.)
Patch Fitting

Make graph for overlap region




Every pixel is a node
Edge weights:
Associate weight with each edge
Find minimum graph cut
Refinements

Modified Matching cost function
Refinements

Search across all translations is costly (use
FFT)

speed up

150x100x30 video sequence

Search for a new patch reduces from around 10 minutes
(using naïve search) to 5 seconds (using FFT-based
search)
Extensions

Translation to Transformation


Rotation, scaling, affine or projection
Interactive merging and blending




Many source images
User specifies position & constraints pixels
Algorithm finds best seam
SIGGRAPH banner
Interactive merging and blending
Interactive merging and blending
Extensions


Goal is to loop the video forever
Video texture


One way is to find the pair of similar looking
frames and use them to repeat the video
Video synthesis using graphcut

Find time of transition on pixel-wise basis
Video Texture – GraphCut

Finding Seams for Video Transitions





Find good transition between pair of images
Take a window around transition (60 frames)
Construct the graph by connecting a pixel to its
neighbors in space and time
Min cut will give you time of transition on per pixel
basis
Use translation in time and space both
Video Synthesis
Video Synthesis

Temporal Constraints for Video

To loop the video, add k frames in start and end of
video (10 frames), constraint these frames to stay
the same, graph is generated and best seam is
found, then k frames are removed.
Video Synthesis






We fix the first k and last k frames of the output sequence
to be the same k frames of the input.
The pixels in these frames are now constrained to stay the
same.
This is ensured by adding links of infinite cost between
these pixels and the patches they are constrained to copy
from, during graph construction
Use graphcut to find the best seam given that these pixels
don’t change.
Once the output has been generated, we remove the first k
frames from it.
This ensures a video loop since the kth frame of the output
is the same as its last frame before this removal operation