Transcript Hybrid Texture Synthesis Andrew Nealen Marc Alexa Discrete Geometric Modeling Group (DGM) Technische Universität Darmstadt Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU.

Hybrid Texture Synthesis
Andrew Nealen
Marc Alexa
Discrete Geometric Modeling Group (DGM)
Technische Universität Darmstadt
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
2D Texture Synthesis
nxm Input Texture
NxM Output Texture
► The
goal: Synthesize an output texture which is
perceptually similar to the input texture. Also
ensure that the result contains sufficient variation.
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Existing
(and Impressive) Technology
• Pixel-Based:
─
─
─
─
Non-parametric Sampling [Efros and Leung 1999]
Tree-structured Vector Quantization [Wei and Levoy 2000]
Synthesizing Natural Textures [Ashikhmin 2001]
Image Analogies [Hertzmann et al. 2001]
• Patch-Based:
─ Patch-Based Sampling [Guo et al. 2001]
─ Image Quilting [Efros and Freeman 2001]
─ Hierarchical Pattern Mapping [Soler et al. 2002]
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Possible
•
•
•
•
Drawbacks ?
Loss of global structure
Loss of scale
Boundary mismatch
Blurring
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Possible
•
•
•
•
Drawbacks ?
Loss of global structure
Loss of scale
Boundary mismatch
Blurring
Wei/Levoy Algorithm:
Can occur when using structured
Textures with rich histograms.
This is due to the L2 norm.
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Possible
•
•
•
•
Drawbacks ?
Loss of global structure
Loss of scale
Boundary mismatch
Blurring
Patch-Based Sampling Algorithm:
Can occur when texture features are not
well-aligned across patch boundaries
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Our Inspiration
► Possible
•
•
•
•
Drawbacks ?
Loss of global structure
Loss of scale
Boundary mismatch
Blurring
► Is
it possible to reduce all deficiencies for an
even larger class of textures ?
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input
Output
Step n in Algorithm
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input
Output
Build L-shaped Neighborhood
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input
Output
Find Matching Neighborhood(s) in Input
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input
Output
Select Best Neighborhood from all Candidates
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input
Output
Copy Selected Pixel to Output
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input
Output
Repeat for all Pixels in this Row...
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
Input
Output
...and all other Output Rows
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Pixel-Based Texture Synthesis
►
►
►
►
Main difference between Efros/Leung (EL),
Wei/Levoy (WL) and Ashikhmin (AS): PerPixel Search Strategy.
Ashikhmin’s insight: “WL tends to produce
textures with smoothed out edges in some
cases.”
Ashikhmin’s solution: consider only
coherent pixels in the search, which
encourages region growing.
Problem: inconsistencies can still occur
for highly-structured textures.
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input
Output
Step n in Algorithm
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input
Output
Patch to be Synthesized in this Step
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input
Output
Build Neighborhood (Image Mask)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input
Output
Find Matching Neighborhood(s) in Input
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input
Output
Select Best Neighborhood from all Candidates
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input
Output
Copy Selected Patch to Output
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input
Output
Apply some Overlap Repair Strategy ...
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input
Output
... And we are done with this patch
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Input
Output
Repeat for all Patches in Output
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Generalization of Pixel-Based Synthesis
►
►
►
►
►
Main (visual) difference between Image
Quilting (IQ) and Patch-based Sampling
(PBS): Overlap Repair Strategy.
IQ performs a minimum-error-boundarycut (MEBC) through the error surface (least
cost path; Dijkstra).
PBS feathers the overlap region (alphablending).
Pro: patch-based algorithms are good at
maintaining global structure.
Problems: boundary mismatch (MEBC,
top) and blurring (PBS, bottom).
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Generalization of Pixel-Based Synthesis
►
►
►
►
►
Main (visual) difference between Image
Quilting (IQ) and Patch-based Sampling
(PBS): Overlap Repair Strategy.
IQ performs a minimum-error-boundarycut (MEBC) through the error surface (least
cost path; Dijkstra).
PBS feathers the overlap region (alphablending).
Pro: patch-based algorithms are good at
maintaining global structure.
Problems: boundary mismatch (MEBC,
top) and blurring (PBS, bottom).
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch-Based Texture Synthesis
Generalization of Pixel-Based Synthesis
►
►
►
►
►
Main (visual) difference between Image
Quilting (IQ) and Patch-based Sampling
(PBS): Overlap Repair Strategy.
IQ performs a minimum-error-boundarycut (MEBC) through the error surface (least
cost path; Dijkstra).
PBS feathers the overlap region (alphablending).
Pro: patch-based algorithms are good at
maintaining global structure.
Problems: boundary mismatch (MEBC,
top) and blurring (PBS, bottom).
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Hybrid Texture Synthesis (HTS)
Algorithm
► Hybrid,
two-fold approach to texture synthesis
• Adaptive patch sampling [Soler et al. 2002]
─ Start with a uniform quadrilateral grid of patches (the output)
─ Recursively split a patch if the best fit taken from the input
texture is not good enough (a user-defined tradeoff max)
• Overlap re-synthesis (novel overlap repair strategy)
─ Mark invalid pixels in the overlap region
─ Validity map is defined by the error surface E(x)=(xsrc – xdst)2
and a user-defined tolerance parameter max
─ Define an ordering for the invalid pixels and re-synthesize them
using a per-pixel strategy
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch Sampling
Principle
Synthesize White Patch
►
►
p > max  Split Patch
p < max
Essentially: Hierarchical Pattern Mapping applied to the
plane [Soler et al. 2002]
Analogy: Assembling a quadtree puzzle
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch Sampling
Trade-off
max = 0.01
max = 1.0
► Trade-off
max = 0.04
between preserving global structure and
avoiding detail artifacts
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch Sampling
Trade-off
max = 0.01
max = 1.0
► Trade-off
max = 0.04
between preserving global structure and
avoiding detail artifacts
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch Sampling
Trade-off
max = 0.01
max = 1.0
► Trade-off
max = 0.04
between preserving global structure and
avoiding detail artifacts
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Adaptive Patch Sampling
Trade-off
max = 0.01
max = 1.0
► Trade-off
max = 0.04
between preserving global structure and
avoiding detail artifacts
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good Patches
Synthesizing a single patch
Synthesize Black Patch i
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good Patches
Extracting the image mask
Synthesize Black Patch i
► Grow
patch by overlap (toroidally)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good Patches
Extracting the image mask
Synthesize Black Patch i
Image Mask (Ii)
► Extract
Binary Support (Ji)
image mask (Ii) and binary support (Ji), and
circularly shift to upper left corner
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good Patches
Computing the error image
Input Texture (T)
Image Mask (Ii)
► Compute
Binary Support (Ji)
error Ei(x0) between Ii and T for each
circular shift x0=(x,y) of the input texture T
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good Patches
Computing the error image
Input Texture (T)
1
Ei (x 0 ) 
κi
Error Image (Ei)
 J (x)w ( I
i
x
c
(x)  Ti ,c (x  x0 ))
2
i ,c
c
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good Patches
Selecting a Patch
Input Texture (T)
1
Ei (x 0 ) 
κi
Error Image (E)
 J (x)w ( I
i
x
c
Selected Patch (P)
(x)  Ti ,c (x  x0 ))
2
i ,c
c
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
In Search of Good Patches
Selecting a Patch
Input Texture (T)
Error Image (E)
► Note:
Selected Patch (P)
error image can be computed efficiently in the
Fourier domain. Complexity: O(n log n) per patch.
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Overlap error and pixel invalidation
Patch and Image Mask
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Selected Patch (P)
Overlap Re-synthesis
Overlap error and pixel invalidation
Image Mask (I)
Selected Patch (P)
Si (x)   J i (x)wc ( I i ,c (x)  Pi ,c (x))2
c
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Overlap error and pixel invalidation
Image Mask (I)
Error Surface (S)
Selected Patch (P)
Si (x)   J i (x)wc ( I i ,c (x)  Pi ,c (x))2
c
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Overlap error and pixel invalidation
Patch and Image Mask
Error Surface (S)
P with Invalid Pixels (blue)
user-defined max is the pixel error
tolerance. Setting to 1 results in pure feathering.
► Trade-off:
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Compositing
P OVER Ri-1
Intermediate Result
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Patch with Invalid Pixels
Overlap Re-synthesis
Compositing
Composited Result
►
►
Patch with Invalid Pixels
Simple scanline re-synthesis: possibly insufficient causal
neighborhood within valid pixel region
Solution: alternative ordering (Pixel Traversal Map)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Pixel Traversal Map (Ordering)
Composited Result
►
Pixel Traversal Map (M) Patch with Invalid Pixels
Pixel Traversal Map: repeated morphological dilation of
the binary support for valid pixels with a euclidian disk of
radius 1 (city-block distance transform).
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Pixel Traversal Map (Ordering)
Composited Result
► Pixel
Pixel Traversal Map (M) Patch with Invalid Pixels
Traversal Map: step 1
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Pixel Traversal Map (Ordering)
Composited Result
► Pixel
Pixel Traversal Map (M) Patch with Invalid Pixels
Traversal Map: step 2 ...
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Pixel Traversal Map (Ordering)
Composited Result
► Pixel
Pixel Traversal Map (M) Patch with Invalid Pixels
Traversal Map: ... step n
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Pixel Traversal Map (Ordering)
Composited Result
Pixel Traversal Map (M) Patch with Invalid Pixels
► Art
Restorer Analogy: stepwise restoration of the
hole from the boundary of the existing image.
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Per-Pixel Re-synthesis
Intermediate Result
► Synthesize
Pixel Traversal Map (M)
red pixel from valid (or already resynthesized) pixels
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Per-Pixel Re-synthesis
Image Mask (Ij)
Binary Support (Jj)
Intermediate Result
►
Extract image mask (Ij) and binary support (Jj) for bestpixel search in the input texture. Complexity: O(n log n) per
pixel.
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Per-Pixel Re-synthesis
Intermediate Result
Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Per-Pixel Re-synthesis
Intermediate Result
Shifted
Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Per-Pixel Re-synthesis
Intermediate Result
with Traversal Map
Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Per-Pixel Re-synthesis
Overlap Re-synthesis
Step 1
Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Per-Pixel Re-synthesis
Overlap Re-synthesis
Step 2
Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Per-Pixel Re-synthesis
Overlap Re-synthesis
Step 3
Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Overlap Re-synthesis
Per-Pixel Re-synthesis
Overlap Re-synthesis
Result
Pixel Traversal Map (M)
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Overlap Repair Comparisons
No Repair
IQ
PBS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
HTS
Results
Synthesis Comparisons
Input
IQ
Efros/Leung
Wei/Levoy
PBS
HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Results
Synthesis Comparisons
Input
IQ
Efros/Leung
Wei/Levoy
PBS
HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Results
Synthesis Comparisons
Input
IQ
Efros/Leung
Wei/Levoy
PBS
HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Results
Synthesis Comparisons
Input
IQ
Efros/Leung
Wei/Levoy
PBS
HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Results
Synthesis Comparisons
Input
IQ
Efros/Leung
Wei/Levoy
PBS
HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Results
Synthesis Comparisons
Input
IQ
Efros/Leung
Wei/Levoy
PBS
HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Results
Synthesis Comparisons
Input
IQ
Efros/Leung
Wei/Levoy
PBS
HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Results
Synthesis Comparisons
Input
Input
PBS
HTS
PBS
HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Results
Synthesis Comparisons
Input
Input
PBS
HTS
PBS
HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Results
Synthesis Comparisons
Input
Input
PBS
HTS
PBS
HTS
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Results
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Conclusions and Future Work
► Improve
Computational Complexity
• Pixel neighborhoods of patch and pixel stage are not
known a priori, so precomputation is not straightforward
• We can solve this in the pixel stage by employing kcoherence search [Tong et al. 2002] [Ashikhmin 2001]
► Improve
Error Metric
• Still using the L2 norm due to its simplicity
• Develop a metric which takes feature mismatch into
account
• Texton approach looks promising [Zhang et al. 2003]
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003
Questions ?
► Contact
Information
Andrew Nealen
[email protected]
Marc Alexa
[email protected]
http://www.dgm.informatik.tu-darmstadt.de
Matlab code and extended Thesis:
http://www.nealen.com
Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003