Overview on Scalable Video Coding
Download
Report
Transcript Overview on Scalable Video Coding
Scalable Video Coding with
Wavelet-Based Approaches
Presenter: Mahin Torki
July 2008
ENSC 820 - Simon Fraser University
1
Paper Title: “State-of-the-Art and Trends in Scalable Video
Compression With Wavelet-Based Approaches”
Authors: Nicola Adami, Alberto Signoroni, Ricardo Leonardi
IEEE Transactions on Circuits and Systems for Video Technology,
Vol. 17, No. 9, September 2007
July 2008
ENSC 820 - Simon Fraser University
2
Outline
Motivation
Wavelet SVC (WSVC) Fundamentals
Coding Architectures for WSVC Systems
WSVC Reference Platform in MPEG
Comparison between WSVC and SVC
Conclusion
July 2008
ENSC 820 - Simon Fraser University
3
Motivation
Several working points corresponding to different
quality, picture size and frame rate in a unique bit
stream
Two types of SVC systems:
Hybrid schemes (used in all MPEG-x or H.26x
standards)
Spatio-temporal wavelet technologies
Main difference of SVC and transcoding systems
Low complexity
Do not require coding/decoding operations
Simple parsing operation on the coded bitstream
July 2008
ENSC 820 - Simon Fraser University
4
Motivation
Decode according to
required QoS or
available hardware resources.
Encode
once
July 2008
ENSC 820 - Simon Fraser University
5
A Typical SVC System
July 2008
ENSC 820 - Simon Fraser University
6
A possible structure of an SVC bitstream
July 2008
ENSC 820 - Simon Fraser University
7
Extracting a scaled bitstream
July 2008
ENSC 820 - Simon Fraser University
8
Tools Enabling Scalability
A multi-resolution signal decomposition inherently enables a low to high
resolution scalability by representing the signal in transformed domain
July 2008
ENSC 820 - Simon Fraser University
9
Tools Enabling Scalability
Inter-Scale Prediction (ISP)
The simplest way to represent a signal with two
resolutions
The signal x can be seen as a coarse resolution c and a
~
detailed signal d
Not critically sampled
Laplacian Pyramid
An iterated version of ISP
Results in a coarsest resolution signal c and a set of
~
details d (l ), l 1,...,n
July 2008
ENSC 820 - Simon Fraser University
10
Laplacian Pyramid
July 2008
ENSC 820 - Simon Fraser University
11
Spatial Scalability
Discrete Wavelet Transform (DWT)
Projects the signal in a set of multi-resolution
(MR) subspaces
Critically sampled
Generates a coarse signal and a set of details
For multi-dimensional signals like images
Separable pyramidal and DWT decompositions
July 2008
Separate filtering on rows and columns
ENSC 820 - Simon Fraser University
12
DWT Filter Bank
Implementing DWT by a two-channel filter
bank iterated on a dyadic tree path
July 2008
ENSC 820 - Simon Fraser University
13
2D-DWT Transform
2D Wavelet decomposition inherently provides spatial
scalability
Bit-plane
Coder
July 2008
ENSC 820 - Simon Fraser University
14
Spatial Scalability
Lifting scheme
July 2008
Alternative spatial domain processing
introduced by Sweldens
Generates a critically sampled (c,d)
representation of the signal x
ENSC 820 - Simon Fraser University
15
Lifting Scheme
Signal x is split in two polyphase components, even
and odd samples(each one half the original resolution)
Two components are correlated
A prediction can be performed
The subsampled signal x2i could contain a lot of
aliased components, so, it should be updated
Perfect reconstruction is guaranteed
Every DWT can be factorized in a chain of lifting steps
Has a fundamental role in MC Temporal Filtering
(MCTF)
July 2008
ENSC 820 - Simon Fraser University
16
Temporal Scalability
Motion Compensating Temporal Filter (MCTF)
July 2008
A key tool enabling temporal scalability while
exploiting temporal correlation
ENSC 820 - Simon Fraser University
17
MCTF implementation by Lifting steps
Index i has now a temporal meaning
P and U can be guided by motion information
July 2008
ENSC 820 - Simon Fraser University
18
MCTF implementation by Lifting steps
ME/MC implemented according to a certain motion
model
ME/MC usually generate a set of motion vector fields
mv(l,k)
mv(l,k) is estimation of the trajectory of the blocks
between the temporal frames, at spatial level l, involved
in the kth MCTF temporal decomposition level
With lifting structure, non-dyadic temporal decomposition
is possible
July 2008
Temporal scalability factors different from a power of two
ENSC 820 - Simon Fraser University
19
Some benefits of MCTF
By exploiting local adaptability of P and U
operators and using mv(l,k) information, MCTF
can handle:
July 2008
Handle occlusion and uncovered area problems
Blocking effects can be reduced by considering
adjacent blocks
When fractional pixel MVs are provided, the lifting
structure can be modified to implement the
necessary pixel interpolation
ENSC 820 - Simon Fraser University
20
MCTF
L0
L0
L0
L0
L0
L1
H1 L1
H1
1
L1 H
H2
L2
H3
July 2008
L0
H2
L0
L0
L0
L0
L0
L0
1
L1 H
L1
H1 L1
H1
H2
L2
L2
L3
ENSC 820 - Simon Fraser University
H3
21
Hybrid temporal and spatial scalability
video sequence
1st temporal level
H
2nd temporal level
LH
3rd temporal level
LLL
July 2008
LLH
ENSC 820 - Simon Fraser University
22
Quality Scalability
Wavelet-based image compression schemes, provide high R-D
performance with limited computational complexity
They do not interfere with spatial scalability requirements
High degree of quality scalability
Truncating the coded bitstream at arbitrary points
Most techniques are inspired from zero tree idea
Embedded Zero Tree Wavelet (EZTW) by Shapiro
SPIHT, reformulated EZTW by Said and Pearlman
Embedded Zero Block Coding (EZBC), with higher performance
Embedded Block Coding with Optimized Truncation (EBCOT)
Do not use zero tree idea
Adopted in JPEG2000
Combines layered block coding, block-based R-D optimizations,
and Context-based arithmetic coding
Good scalability and high coding efficiency
July 2008
ENSC 820 - Simon Fraser University
23
WSVC Notation
xS(n) (xT(m)): the original signal undergoes an n-level (m-
level) multi-resolutional spatial (temporal) Transform
S(n) (T(n))
The spatially transformed signal consist of the subband
set:
xS ( n) {xSc ( n) , xSd((nn)) ,...,xSd((1n)) }
l
k
xˆ
is the decoded version of the original signal x, at
given temporal resolution k and spatial resolution l at
reduced quality rate
July 2008
ENSC 820 - Simon Fraser University
24
Basic WSVC Architectures
T+2D
2D+T
Adaptive Architectures
Multiscale Pyramids
July 2008
ENSC 820 - Simon Fraser University
25
Basic WSVC Architectures
T+2D
Temporal transform is applied before spatial
Guarantees critically sampled subbands
Low spatial scalability performance
Full resolution motion vectors
July 2008
ENSC 820 - Simon Fraser University
26
Basic WSVC Architectures
2D+T
Spatial transform is applied before temporal
Often called In-band MCTF (IBMCTF)
Estimation of mv(l,k) is made independently on each spatial level
Leading to a structurally scalable motion representation
Spatial and temporal scalability are more decoupled
Lower coding efficiency especially at higher temporal resolutions
July 2008
ENSC 820 - Simon Fraser University
27
Basic WSVC Architectures
Adaptive Architectures
Combine the positive aspects of T+2D and 2D+T structures
Adaptive spatio-temporal decompositions optimized with
respect to suitable criteria
Content-adaptive 2D+T versus T+2D improves coding
performance
Multiscale Pyramids
Also called 2D+T+2D
Compensates the T+2D versus 2D+T drawbacks
Uses ISP to exploit the multiscale representation
redundancy
Disadvantage: over-complete transforms, which result in a
full size residual image
July 2008
ENSC 820 - Simon Fraser University
28
Pyramidal WSVC with pyramidal
decomposition before MCTF
July 2008
ENSC 820 - Simon Fraser University
29
Pyramidal WSVC with pyramidal
decomposition after MCTF
July 2008
ENSC 820 - Simon Fraser University
30
Spatio-Temporal prediction (STP)Tool Scheme
Promising WSVC architecture which presents
some similarities to the SVC standard
Adopted as a possible configuration of the
MPEG VidWav (Video Wavelet) reference
software
Based on a multiscale pyramid but differs in
the ISP mechanism
July 2008
ENSC 820 - Simon Fraser University
31
STP-Tool Scheme
July 2008
ENSC 820 - Simon Fraser University
32
Advantages of STP-Tool Scheme
Prediction is performed between two signals
which are likely to bear similar pattern in the
spatio-temporal domain
No need to perform any interpolation
Instead of full resolution residuals, the spatiotemporal subbands and residues are
produced for different resolutions
July 2008
ENSC 820 - Simon Fraser University
33
WSVC Reference Platform in MPEG
In 2004, the ISO/MPEG set up a formal evaluation of
SVC
Performance of H.264/AVC pyramid appeared the most
competitive
Later, MPEG and IEC/ITU-T jointly adopted JSVM (Joint
Scalable Video Coding)
As scalable reference model and software platform
Microsoft Research Asia (MRA) was selected as the
reference for wavelet technologies
The MPEG WSVC reference model and software
(RM/RS) is indicated as VidWav (Video Wavelet)
July 2008
ENSC 820 - Simon Fraser University
34
VidWav: General framework
July 2008
ENSC 820 - Simon Fraser University
35
VidWav: Main modules
Spatial Transform
with pre- and post-spatial decomposition, different SVC
configurations (T+2D, 2D+T, STP-Tool) can be implemented.
Temporal Transform
Framewise MC wavelet transform on a lifting structure
ME and Coding
MB-based motion model with H.264/AVC like partition patterns
Forward, backward or bidirectional motion model for each block
Entropy coding
3D extension of the EBCOT algorithm is used for entropy coding of
the resulted coeficients
July 2008
ENSC 820 - Simon Fraser University
36
VidWav STP-Tool Configuration
July 2008
ENSC 820 - Simon Fraser University
37
Comparison between WSVC and SVC
Single layer coding tools
Scalable coding tools
July 2008
ENSC 820 - Simon Fraser University
38
Comparison between WSVC and SVC
Single layer coding tools
VidWav uses a block-based motion model
Block mode types are similar to JSVM but no Intra-mode is
supported by VidWav
JSVM operates in a local manner
Divides frames into MB and treats MB separately in all
coding phases
VidWav operates with a global approach
Spatio-temporal transform applied to a group of frames
Unlike JSVM, single layer VidWav only supports open loop
encoding/decoding
In-loop deblocking filter in JSVM due to closed loop
encoding
July 2008
ENSC 820 - Simon Fraser University
39
Comparison between WSVC and SVC
Scalable coding tools
Spatial scalability in JSVM compared to VidWav in
STP-Tool configuration
July 2008
Block-based versus frame-based
Similar to JSVC, STP-Tool can use both closed and
open loop inter layer encoding
ENSC 820 - Simon Fraser University
40
Objective and Visual Result Comparisons
Fair objective comparison is impaired due to
Visually, the ref. seq. generated by wavelet
filters are more detailed, but sometimes have
spatial aliasing effects due to different down
sampling filters
Depending on the spatial down-sampling filter
used, reduced spatial resolution decoded seq.
differ even at full quality
PSNR is used as the performance criterion at
intermediate spatio-temporal resolution levels
July 2008
ENSC 820 - Simon Fraser University
41
Objective Comparison Results
July 2008
ENSC 820 - Simon Fraser University
42
Subjective Comparison Results
Visual tests conducted by ISO/MPEG
included 12 expert viewers
July 2008
On average JSVM 4.0 is superior
Marginal gains in SNR conditions
Superior gains in combined scalability settings
ENSC 820 - Simon Fraser University
43
Applications of WSVC
Based on a series of experiments:
July 2008
DCT-based technologies outperform waveletbased ones for relatively smooth signals and vice
versa
Eligible applications for WSVC are those that
produce or use High Definition/High Resolution
content
ENSC 820 - Simon Fraser University
44
Home distribution of HD video using
WSVC
July 2008
ENSC 820 - Simon Fraser University
45
New Application Potentials for WSVC
HD material storage and distribution
Use nondyadic wavelet decomposition to
support multiple HD formats to be used in
video surveillance and mobile video
efficient similarity search in large video
databases
Multiple descriptions coding
Space variant resolution adaptive decoding
July 2008
Only a certain region of the image is decoded
at high resolution
ENSC 820 - Simon Fraser University
46
Conclusion
Brief review of different tools used in WSVC
WSVC architectures are introduced
Comparison of WSVC with SVC
Potential applications for WSVC
July 2008
ENSC 820 - Simon Fraser University
47
Any questions?
Thank you!
July 2008
ENSC 820 - Simon Fraser University
48