Transcript Scratch détection et inpainting spatial/temporel

Film Restoration PhD presentation
Alasdair Newson
Tuesday 16 November 2010
Context/introduction
• PhD Subject : Robust and efficient tools for digital film restoration
• Initial directions/considerations :
1. Artifact detection
 Non-impulsive defects (line scratches ...)
2. Restoration of said defects
3. High definition material
• Project : CPW 1
• Administrative aspects (official)
 Technicolor PhD supervisor : Patrick PEREZ
 Academic PhD supervisor : Yann GOUSSEAU
2
4/8/2015
Contents
1. Scratch :
• Detection
• Tracking
2. Inpainting/Restoration
• Spatial
• Temporal
• Spatio-Temporel
3. Computational Issues
3
4/8/2015
Principal methods used
- Wavelet decomposition
- Hough Transform, morphological detectors
- Bayesian networks, Hidden Markov Models
- Harmonic equation modeling of scratches
- Neural networks
3 global method groups
- Kalman filtering (tracking)
• Graphic models (stats.)
• Image Proc. filters
• Movement estimation
- Optical flow
- Movement estimation
- Low and high frequency interpolation
(polynomial and Fourier series interpolation)
- Random Walks
4
4/8/2015
Detection/Tracking
Scratch detection
Methods
Papers
Comments
• Projection onto the x-axis
(summation along the y-axis)
Many ...
- A ubiquitous technique. Many detection
algorithms use this.
-Reduces complexity : because of
manipulation of a 1-D Signal.
- In practice only works for simple cases
(scratches not exactly vertical, do not
always cover the whole image)
• Hough Transform
- Detection and Removal of Line Scratches in
Degraded Motion Picture Restoration
(A.Kokaram)[1996]
- Structure detection
• Wavelet decomposition
(transform)
• Local maxima/minima
detection
-An OWE-based Algorithm for Line Scratches
Restoration in old Movies (J. Xu, J. Guan)
-Scratch Detection Supported by Coherency
Analysis of Motion Vector Fields Spatial Detection
(S. Müller, J. Bühler ...)[2009]
-
• 2° order differential equation
modeling of scratches
- Scratch detection via Underdamped Harmonic
Motion (V. Bruni, D. Vitulano)[ICPR 2002]
-Supposition :
- Scratch modelled by a sinc
function (side-lobes on scratches)
• Hidden Markov Model based
detection
- HMM based Archive Film Detection with Spatial
and Temporal Constraints (X. Wang, M.
Mirmehdi)[BMVC 2009]
- Train a HMM to determine whether a
pixel sequence is normal
6
4/8/2015
Scratch detection (continued)
7
Methods
Papers
Comments
• Detection based on energy of x-axis
projection
- Line scratch detection on Digital Images : an
Energy Based Model (D. Vitulano, V. Bruni, P.
Ciarlini)[WSCG 2002]
- Energy based on sum of absolute
values of difference between peak
and side-lobes.
- Use of Weber’s law.
• Bayesian framework
- On Missing Data Treatment for Degraded
Video and Film Archives : A Survey of a New
Bayesian Approach (A. Kokaram)[2004]
• Morphological filters (math.
morph.)
- Detection and Removal of Line Scratches in
Motion Picture Films (L. Joyeux, O. Buisson
...)[CVPR 1999]
• Neural network based detection
-An Automatic Film Line Scratch Removal
System based on Spatial Information (K. Kim,
E. Kim)
- Film Line Scratch Detection using Neural
Network and Morphological Filter (K. Kim, E.
Kim)[CIS_2008]
4/8/2015
- Morphological detector ...
Scratch tracking
Methods
• Kalman filter
8
4/8/2015
Papers
- Film Line Scratch Removal using Kalman
Filtering and Bayesian Restoration (L.
Joyeux, S. Boukir, B. Besserer) [2000]
Comments
- Supposes sinusoidal
movement of scratch (based
on mechanical observations)
Inpainting/Restoration
Inpainting/Restoration
Spatial
Methods
Papers
Comments
• Pixel probability distribution
• Adaptive sample image definition
- Missing Data Correction in Still Images and
Image sequences (R. Bornard, E. Lecan
...)[ACM 2002]
• Manipulation of Wavelet decomposition
- Fast Removal of Line Scratches in Old
Movies (V. Bruni, D. Vitulano, A.
Kokaram)[ICPR 2004]
- Only certain Wavelet subbands processed :
“approximation” and
“vertical” (?...)
• Low and high frequency interpolation
- Detection and Removal of Line Scratches
in Motion Picture Films (L. Joyeux, O.
Buisson ...)[CVPR 1999]
- Low frequency
interpolation via polynomial
interp.
- High frequency via Fourier
series.
10
4/8/2015
Inpainting/Restoration
Temporal
Methods
• Use of optical flow
11
4/8/2015
Papers
- Inpainting of Movies Using Optical Flow
(H. Grossauer)[2005]
Comments
- Optical flow to determine useful n-1
and n+1 images for copying pixels (spatial
inpainting used if temporal unsuccessful)
Inpainting/Restoration
Spatio-temporal
Methods
Papers
Comments
• Optimisation of a coherence function
- Space-Time Video Completion (Wexler,
Schechtman ...)[CVPR 2004]
- Space-Time patches used.
Spatio-temporal aliasing :
different size patches
• Backwards/forwards global (coarse) motion
detectors
- Missing Data Correction in Still Images and
Image sequences (R. Bornard, E. Lecan
...)[ACM 2002]
- If sample zones are large
enough, accurate motion
vectors not needed ...
• Random walks
- Archive Film Restoration based on
Spatiotemporal Random Walks (ECCV 2010)
- Random walks used to
determine the spatiotemporal exemplar region
12
4/8/2015
Computational issues
Computation Time
Detection
Method
SDIp
Morris95
Joyeux00 Kokaram04
RV07
Wang09
Seconds (per frame)
4
443
10
80
93
19
Image Size
480*356
480*356
720*576
480*356
480*356
480*356
Method
Bornard02
Drori03
Kokaram04
GD06
Patwardhan07
ECCV10
Seconds (per
frame)
10-1 ->100
~104
174
256
< 180
406
Image Size
700*466
384*256
480*360
480*360
640*480
480*360
Restoration
14
4/8/2015
Various ideas, notes and questions:
- Spatial seam-carving
- Multiple orientation of patches
- Scratches (hypotheses etc ...) :
- Do they have side-lobes ?
- Do they show periodic motion (sinusoidal, damped
sinusoid) ?
- Are they additive (ie. added to the original image
information, rather than destroying it completely) ?
- Weber’s law : determines the visibility/detectability of a
scratch with respect to the brightness of the rest of the
image
- Use of Wavelet domain : why ? Because of “colour
independent detection” ?
- Radon projections ?
- Temporal analysis DEFINATELY needed
Materials needed :
-LOTS of scratched footage (the higher definition the better
– to observe side-lobes)
- Matlab ...
15
4/8/2015
Questions (Yann questions) :
Technical questions :
• What exactly is the Wavelet domain ? (Mallat 1992)
• What is a film’s resolution (analog ... ?)
• Is Flicker restoration allowed (in my PhD) ?
• Flicker seems to be very present in old films
• It would seem that scratches are not completely “non-impulsive” (The Cabinet
of Dr. Caligari, Nosferatu ...)
16
4/8/2015
Bibliography
Detection
Title
17
Authors
Conference/date
HMM based Archive Film Defect Detection
with Spatial and Temporal Constraints
X. Wang, M. Mirmehdi
British Machine Vision Conf.
2009
Scratch Detection Supported by Coherency
Analysis of Motion Vector Fields
S. Müller, J. Bühler, S. Weitbruch, C.
Thebault, I. Doser, O. Neisse
ICIP 2009
Line Scratch Detection on Digital Images : An
Energy Based Model
D. Vitulano, V. Bruni, P. Ciarlini
WSCG 2002
Scratch Detection via Underdamped Harmonic
Motion
V. Bruni, D. Vitulano
ICPR 2002
A Generalised Model for Scratch Detection
V. Bruni, D. Vitulano
IEEE Transactions on Image
Processing 2004
Film Line Scratch Removal Using Kalman
Filtering and Bayesian Restoration
L. Joyeux, S. Boukir, B. Besserer
IEEE Workshop on Applications
of Computer Vision 2000
Detection and Removal of Line Scratches in
Motion Picture Films
L. Joyeux, O. Buisson, B. Besserer,
S. Boukir
CVPR 1999
An OWE (Over-Complete Wavelet Expansion)
based Algorithm for Line Scratches
Restoration in Old Movies
J. Xu, J.Guan, X. Wang, Z. Li
ICAS 2007
Automatic Film Line Scratch Removal System
based on Spatial Information
K. Kim, E. Kim
ISCE 2007
4/8/2015
Bibliography (continued)
Restoration
Title
18
Authors
Conference/date
Space-Time Video Completion
Y. Wexler, E. Schechtman, M. Irani
CVPR 2004
Removal of Line Artefacts for Digital
Dissemination of Archived Film and Video
A. Kokaram
ICMCS 1999
Inpainting of Movies Using Optical Flow
H. Grossauer
2005
Missing Data Correction in Still Images and
Image Sequences
R. Bornard, E. Lecan, L. Laborelli,
J-H. Chenot
ACM 2002
Fragment Based Image Completion
I. Drori, D. Cohen-Or, H. Yeshurun
Siggraph 2003
Video Inpainting Under Constrained Camera
Motion
K. Patwardhan, G. Sapiro, M.
Bertalmio
IEEE Transaction on Image
Processing 2007
Archive Film Restoration based on
Spatiotemporal Random Walks
Anonymous ECCV submission
ECCV 2010
Fast Removal of Line Scratches in Old Movies
V. Bruni, D. Vitulano
ICPR 2004
On Missing Data Treatment for Degraded Film
Video and Archives : a Survey and a New
Bayesian Approach
A. Kokaram
IEEE Transactions on Image
Processing 2004
Detection and removal of line scratches in
degraded motion picture sequences
A. Kokaram
Signal Processing VIII,1996
4/8/2015