Transcript Multiscale Geometric Analysis

Multiscale
Geometric
Image
Analysis
Richard Baraniuk
Rice University
dsp.rice.edu
Joint work with
Hyeokho Choi
Justin Romberg
Mike Wakin
Besov
Bayes
Chomsky
Plato
Low-Level Image Structures
• Smooth/texture regions
• Edge singularities along smooth curves
(geometry)
geometry
texture
texture
Computational Harmonic Analysis
• Representation
coefficients basis, frame
Fourier sinusoids, Gabor functions,
wavelets, curvelets, Laplacian pyramid
Computational Harmonic Analysis
• Representation
coefficients basis, frame
• Analysis
study
• Approximation
through structure of
might extract features of interest
uses just a few terms
exploit sparsity of
Multiscale Image Analysis
• Analyze an image a multiple scales
• How?
Zoom out and record information lost in
wavelet coefficients
…
info 1
info 2
info 3
Wavelet-based Image Processing
• Standard 2-D tensor
product wavelet transform
Transform-domain Modeling
• Transform-domain modeling and processing
transform
coefficient
model
Transform-domain Modeling
• Transform-domain modeling and processing
transform
coefficient
model
vocabulary
Transform-domain Modeling
• Transform-domain modeling and processing
transform
vocabulary
coefficient
model
grammar
• Vocabulary + grammar capture image structure
• Challenging co-design problem
Nonlinear Image Modeling
• Natural images do not form a linear space!
+
=
• Form of the “set of natural images”?
Set of Natural Images
Small:
N x N sampled images comprise an
N2
extremely small subset of R
Set of Natural Images
Small:
N x N sampled images comprise an
N2
extremely small subset of R
Set of Natural Images
Small:
N x N sampled images comprise an
N2
extremely small subset of R
Complicated:
Manifold structure
2
RN
Wedge Manifold
Wedge Manifold
3 Haar wavelets
Wedge Manifold
3 Haar wavelets
Stochastic Projections
• Projection
= “shadow”
Higher-D Stochastic Projections
• Model (partial)
wavelet coefficient
joint statistics
• Shadow more
faithful to manifold
Manifold Projections …
Multiscale Grammars
for
Wavelet Transforms
Wavelet Statistical Properties
Marginal Distribution
0
• many
small
S
coefficients
• smooth
regions
Marginal Distribution
0
• a few
large
L
coefficients
• edge
regions
Wavelet Mixture Model
state: S or L
wc:
Gaussian with
S or L variance
Magnitude Correlations
• Persistence
of wc’s on
quadtree
• S
S
• L
L
Wavelet Hidden Markov Tree
state: S or L
wc:
Gaussian with
S or L variance
A
Wavelet Statistical Models
• Independent wc’s w/ generalized Gaussian marginal
– processing:
– application:
thresholding
denoising
(“coring”)
• Correlated wc magnitudes
–
–
–
–
zero tree image coder
[Shapiro]
estimation/quantization (EQ) model
[Ramchandran, Orchard]
JPEG2000 encoder
graphical models:
• Gaussian scale mixture
[Wainwright, Simoncelli]
• hidden Markov process on quadtree (HMT)
• processing:
tree-structured algorithms
EM algorithm, Viterbi algorithm, …
– applications:
denoising, compression, classification, …
Barbara
Barbara’s Books
DWT HMT
WT thresholding
noisy books
Denoising Barbara’s Books
Denoising Barb’s Books
HMT Image Segmentation
HMT Image Segmentation
Zero Tree Compression
• Idea: Prune wavelet subtrees in smooth regions
Z
– tree-structured
thresholding
– spend bits on
(large)
“significant” wc’s
– spend no bits on
(small) wc’s…
zerotree symbol
Z = {wc and all
decendants = 0}
New Multiscale Vocabularies
• Geometrical info
not explicit
• Modulations
around singularities
(geometry)
• Inefficient large number
of significant wc’s
cluster around
edge contours, no
matter how smooth
Wavelet Modulations
• Wavelets are poor edge detectors
• Severe modulation effects
Wavelet Modulations
• Wavelet wiggles…
so its inner product with a singularity wiggles
signal
scale
L
S
L
S
Wavelet Modulation Effects
• Wavelet transform
of edge
Wavelet Modulation Effects
• Wavelet transform
of edge
• Seek
amplitude/envelope
• To extract amplitude need coherent representation
1-D Complex Wavelets
[Grossman, Morlet, Lina, Abry, Flandrin, Mallat, Bernard, Kingsbury, Selesnick, Fernandes, …]
• real wavelet
even symmetry
imaginary wavelet
odd symmetry
• Hilbert transform pair
(complex Gabor atom)
• 2x redundant tight frame
• Alias-free; shift-invariant
• Coherent wavelet representation (magnitude/phase)
2-D Complex Wavelets
[Lina, Kingsbury, Selesnick, …]
• Even/odd real/imag symmetry
• Almost Hilbert transform pair
(complex Gabor atom)
• Almost shift invariant
• Compute using 1-D CWT
-75
real
imag
+75
+45
• 4x redundant tight frame
• 6 directional subbands
aligned along 6 1-D
manifold directions
• Magnitude/phase
+15
-15
-45
Wavelet Image Processing
Coherent Wavelet Processing
real
part
+i
imaginary
part
Coherent Wavelet Processing
|magnitude|
x exp(i phase)
Coherent Image Processing
magnitude
FFT
[Lina]
Coherent Image Processing
magnitude
FFT
[Lina]
phase
Coherent Image Processing
magnitude
FFT
CWT
[Lina]
phase
Coherent Wavelet Processing
feature
magnitude
phase
1 edge
L
coherent
“speckle”
> 1 edge
L
incoherent
smooth
S
undefined
Coherent Segmentation
feature
magnitude
phase
1 edge
L
coherent
“speckle”
> 1 edge
L
incoherent
smooth
S
undefined
Edge Geometry Extraction
• Extract 1-D edge geometry from piecewise smooth
image
r

wedgelet
• Goal:
Extract r and  locally
Edge Geometry Extraction
• CWT magnitude encodes angle 
• CWT phase encodes offset r

r

Edge Geometry Extraction in 2-D
original
estimated
Multiscale Edge Geometry Grammar
for Wavelet Transforms
• Inefficient large number
of significant WCs
cluster around
edge contours, no
matter how smooth
Cartoon Processing
2-D Dyadic Partitions for Cartoons
• C2 smooth regions
• separated by
edge discontinuities
along C2 curves
2-D Dyadic Partitions for Cartoons
• Multiscale analysis
• Partition;
not a basis/frame
• Zoom in by factor
of 2 each scale
2-D Dyadic Partition = Quadtree
• Multiscale analysis
• Partition;
not a basis/frame
• Zoom in by factor
of 2 each scale
• Each parent node
has 4 children at
next finer scale
Wedgelet Representation
[Donoho]
• Build a cartoon using wedgelets on dyadic squares
r
• Choose orientation (r,) from
finite dictionary (toroidal sampling)
• Quad-tree structure
deeper in tree
finer curve approximation

2-D Wedgelets
2-D Wedgelets
Wedgelet Representation
• Prune wedgelet quadtree to approximate local
geometry (adaptive)
• Decorate leaves with (r,) parameters
Wedgelet Inference
• Find representation / prune tree to balance a
fidelity vs. complexity trade-off
• For Comp(W) –
need a model for the
wedgelet representation
(quadtree + (r,))
• Donoho: Comp(W) = #leaves
#Leaves Complexity Penalty
• Accounts for wedgelet partition size,
but not wedgelet orientation
#leaves
#leaves
“smooth”
“simple”
“rough”
“complex”
Multiscale Geometry Model (MGM)
Multiscale Geometry Model (MGM)
• Decorate each tree node with orientation (r,)
and then model dependencies thru scale
• Insight:
Smooth curve
Geometric innovations small at fine scales
• Model:
Favor small innovations
over large innovations
(statistically)
Multiscale Geometry Model (MGM)
• Wavelet-like geometry model:
coarse-to-fine prediction
– model parent-to-child transitions of orientations
small
innovations
large
MGM
• Wavelet-like geometry model:
coarse-to-fine prediction
– model parent-to-child transitions of orientations
• Markov-1 statistical model
– state = (r,) orientation of wedgelet
– parent-to-child state transition matrix
MGM
• Markov-1 statistical model
• Joint wedgelet Markov probability model:
• Complexity = Shannon codelength =
= number of bits to encode
MGM and Edge Smoothness
“smooth”
“simple”
“rough”
“complex”
MGM Inference
• Find representation / prune tree to balance the
fidelity vs. complexity trade-off
• Efficient O(N) solution via dynamic programming
MGM Approximation
• Find representation / prune tree to balance the
fidelity vs. complexity trade-off
• Optimal L2 error decay rate for cartoons
single scale
multiscale
Wedgelet Coding of Cartoon Images
• Choosing wedgelets = rate-distortion optimization
Shannon code length
to encode
Joint Texture/Geometry Modeling
• Dictionary D = {wavelets} U {wedgelets}
• Representation tradeoff:
texture vs.
geometry
Zero Tree Compression
• Idea: Prune wavelet subtrees in smooth regions
Z
– tree-structured
thresholding
– spend bits on
(large)
“significant” wc’s
– spend no bits on
(small) wc’s…
zerotree symbol
Z = {wc and all
decendants = 0}
Zero Tree Compression
• Label pruned wavelet quadtree with 2 states
zero-tree
significant
- smooth region
(prune)
- edge/texture region (keep)
S
Z
smooth
smooth
smooth
not
smooth
Z
S
Z
Z: all wc’s below=0
S
S
S
S
Wedgelet Trees for Geometry
• Label pruned wavelet quadtree with 3 states
zero-tree
geometry
significant
- smooth region
- edge region
- texture region
(prune)
(prune)
(keep)
• Optimize placement of Z, G, S by dyn. programming
S
smooth
Z
smooth
Z
S
G
G: wc’s below
are wc’s of a
wedgelet tree
texture
S
S
S
S
geometry
Optimality of Wedgelets + Wavelets
• Theorem
For C2 / C2 images, optimal
asymptotic L2 error decay
and near-optimal rate-distortion
C2 contour (1-d)
C2 texture (2-d)
Practical Image Coder
• Wedgelet-SFQ (WSFQ) coder
builds on SFQ coder [Xiong, Ramchandran, Orchard]
WSFQ
Improvement over
JPEG-2000
PSNR dB
SFQ
bits per pixel
• At low bit rates, often significant improvement in visual quality
over SFQ and JPEG-2k (much sharper edges)
• Bonus:
WSFQ representation contains
explicit geometry information
Wet Paint Test Image
PSNR = 26.32dB @ 0.0103bpp
JPEG Compressed (DCT)
PSNR = 29.77dB @ 0.0103bpp
JPEG2000 Compressed (Wavelets)
PSNR = 30.19dB @ 0.0102bpp
WSFQ Compressed
WSFQ Contour Trees
original
SFQ
JPEG2000
WSFQ
Gain Over JPEG2000 – Cameraman
40% MSE improvement
WSFQ
SFQ
Extensions
S
zerotree
Z
S
S
W
S
S
S
S
wedgeprint
Z
W
D
S
S
S
S
Extensions
DCTprint
S
zerotree
Z
D
S
barprint
W
B
B
B
B
wedgeprint
Z
W
D
B
B
B
B
“Coifman’s
Dream”
Extensions
S
zerotree
Z
S
S
W
S
S
S
S
wedgeprint
Z
W
D
S
S
S
S
Extensions
DCTprint
S
zerotree
Z
D
S
barprint
W
B
B
B
B
wedgeprint
Z
W
D
B
B
B
B
“Coifman’s
Dream”
Extensions
DCTprint
S
zerotree
Z
D
S
barprint
W
V
B
B
B
wedgeprint
Z
W
D
Eeroprint
B
S
B
B
Bars / Ridges
16x16 image block
Multiple Wedgelet Coding
80 bits to jointly encode 5 wedgelets
“Barlet” Coding
(fat edgelet/beamlet)
22 bits to encode 1 barlet
Periodic Textures
DCTprint
PSNR
DCT
32x32 block = 1024 pixels
5dB
wavelets
# terms
DCTprint
PSNR
DCT
32x32 block = 1024 pixels
wavelets
# terms
4x fewer coefficients
Summary
• Wavelets and other bases are vocabularies
• Image structure induces coefficient grammar
• Statistical models
– extract higher-dimensional joint statistics via
quadtrees, wedgelets, wedgeprints, …
• New vocabularies
– curvelets, bandelets, complex wavelets, …
– random projections
[Candes, Romberg, Tao; Donoho]
• Grand challenge
– rather than low-dimensional projections, work on the
manifold
– potential for image processing, compression, ATR, …