Transcript Error-Resilient Coding and Decoding Strategies for Video

Error-Resilient Coding and Decoding
Strategies for Video Communication
Thomas Stockhammer and Waqar Zia
Presented by Li Ma
Background & Motivation
Video becoming more popular
Advances in bandwidth, capacity enhancements
Requirements:





data transmission rate
Real-time delivery of multimedia data
Limitation:


QoS available is not sufficient to guarantee error-free delivery
for all receivers
Motivation:


2
Provide means of dealing with various transmission
impairments
Content

Focus on MCP-coded video
Concentrate on tools and features integrated in standard
H.264/AVC
Focus on specific tools for improved error resilience

Other advanced error-resilience features not covered:





3
Multiple description coding
Distributed video coding
Combinations with network prioritization and FEC
Outline
Video Communication Systems
Error-Resilient Video Transmission
Resynchronization and Error Concealment
Error Mitigation
Summary





4
Video Communication Systems
End-to-End Video Transmission

5
Video Applications
wide variety of applications
Different bit rate ranges:




HDTV: 20 Mbit/s
MMS on cell: 20 Kbit/s
Different tolerable end-to-end delay


6
Conversational applications constraints: ≤ 200-250 ms
Transmission Impairments
Differences of errors


Wireless networks:


Fading and interference cause burst errors: multiple lost bits
IP network:

Congestion results in packets lost
Methods




7
Detect presence of errors
Intermediate protocol layers (UDP) could drop erroneous pkts
Video data pkts treated as lost if delayed more than threshold
Data Losses in MCP-Coded Video
Transmitted over error-prone channels



8
Error concealment
Error propagation / Spatiotemporal error propagation
Example of Error Propagation
Pt @ t=0 is lost.
Error propagation till t=8
Intra-coded image transmitted @ t=9



9

Therefore, when data units might get lost, a video coding
system should provide:



10
Means that allow completely avoiding transmission errors
Features that allow minimizing the visual effects of errors
Features to limit spatial and spatiotemporal error propagation
Outline





Video Communication Systems
Error-Resilient Video Transmission
Resynchronization and Error Concealment
Error Mitigation
Summary
11
Error-resilient Video Transmission

System Overview
12
Features

@Sender


MBs are grouped in data units and entropy coding used
Error Control before transmission over lossy channel




Forward Error Correction(FEC)
Backward Error Correction (BEC)
Prioritization Methods
Combinations of above
@Receiver





13
Erroneous and missing data detected and localized
Decoder gets correct data units or error indication
Error concealment applied at positions where no data received
Report loss of data units to encoder
Design Principles


error-resilience tools decrease compression efficiency
Main goal:



Shannon’s separation principle: compression separated with
transport
In low delay situations, error-free transport is impossible
System Design Principles





14
1. Loss correction below codec layer
2. Error detection
3. Prioritization methods
4. Error recovery and concealment
5. Encoder-decoder mismatch avoidance
Video Compression Tools Related to Error
Resilience







Slice Structured Coding
Flexible MB Ordering
Scalable Coding
Data Partitioning
Flexible Reference Frame
Intra Information Coding
Pictures Switching
15
Slice Structured Coding






Slices provide spatially distinct resynchronization points
within the data for a single frame
Several MBs grouped together: a slice header
Variable sized data units
Encoder can select the location of sync. Points
Motion vector prediction not allowed over boundaries
Encoder decides either:


16
Allocate fixed number of MB to one slice
Or fixed bits to one slice (matched to pkt size in network)
Flexible MB Ordering

Flexible Macroblock Ordering (FMO)





17
Allows mapping of MBs to Slice groups
A slice group may contain several slices
MBs can be transmitted in flexible and efficient way
Spatially collocated images areas can be interleaved in different
slices  greater probability of concealing lost MB
Protection: Can map ROI (region of interest) into a separated
slice group
Data Partitioning

Loss of some syntax elements of a bit stream results in
larger degradation of quality compared to others



E.g. Loss of motion vector
Data partitioning results in Graceful Degradation
Categorize syntax elements



18
Header information
Motion information
Texture information
Layout of compressed data
Without Data Partitioning:
With Data Partitioning:
2 additional sync. Points available
19
Data Partitioning (Cont.)

Unequal Error Protection (UEP)


20
Protect partitions of different importance
More important data offered more protection
Flexible Reference Frame



H.263 v.1 & MPEG-2 allow only a single reference frame
for predicting P frame and mostly 2 for B frame.
Possible to have significant statistical dependencies
between other pictures too
Use more frames than just the recent one as reference

Advantages:


21
Increased compression efficiency
Improved error resilience
Example of Flexible Reference Frame
22
Example of Flexible Reference Frame

Enable Subsequences

Use a subsequence of “anchor frames” at lower frame rate


Other frames inserted in between to achieve overall frame rate


E.g “ P’ ”
Error propagation:

23
E.g “ P ”
Only till next P received
Outline





Video Communication Systems
Error-Resilient Video Transmission
Resynchronization and Error Concealment
Error Mitigation
Summary
24
Resynchronization and Error Concealment

Video Packetization Modes

Without FMO(flexible macroblock ordering)


25
1. a constant number of MBs within one slice (arbitrary size)
2. the slice size bounded to some max bytes (arbitrary # of MBs)
Video Packetization Modes (Cont.)

With FMO (more flexible)




26
Slice interleaving
Dispersed MB allocation using checkerboard patterns
Subpictures within a picture
etc.
Error Concealment

Basic Idea




Decoder should generate a representation for lost area
Match as close as possible to the lost info
Within manageable complexity
Techniques




27
Spatial Error Concealment
Temporal Error Concealment
Hybrid Concealment
Other Techniques
Spatial Error Concealment

Based on assumption of continuity of natural scene
content in space
Use pixel values of surrounding available MBs
Estimate of lost pixel:

αβγ are weighing factors




Determine relative impact of vertical,
Horizontal, upper, lower…
Disadvantage

28
Blurred reconstruction
Temporal Error Concealment



Rely on the continuity of a video sequence in time
Use temporally neighboring areas to conceal lost regions
Previous Frame Concealment (PFC)



29
Use previous corresponding data to copy to current frame
Only good when little motion
Widely used due to simplicity
Hybrid Concealment

When only apply spatial concealment


When only use temporal error concealment


Concealed regions are significantly blurred
Significantly discontinuities in the concealed regions
Hybrid temporal-spatial technique applied

30
MB mode info of reliable and concealed neighbors decide
which concealment method to use
Hybrid (cont.)

For intra-coded images


For inter-coded images



Only use spatial concealment
Use temporal concealment when more than half of the
available neighbor MBs are inter-coded
Otherwise, use spatial concealment
Referred to as Adaptive temporal and spatial Error
Concealment (AEC)
31
Selected Results

Performance of different error concealment strategies
32
Selected Performance Results for Wireless


Low-delay and low-complexity requirements
Max allowed buffering at encoder limited to 250ms
33

Using a smaller slice size of 150 bytes is lower in PSNR
when error free

Because:



increased packetization overhead
prediction limitations on slice boundaries
Performs good when in lossy channel

34
Because the loss affects only a small area of the image for fixed
slice size
Outline





Video Communication Systems
Error-Resilient Video Transmission
Resynchronization and Error Concealment
Error Mitigation
Summary
35
Motivation


Error propagation is major problem over lossy channels
Encoder can change encoding behavior when he finds it’s
likely to be lossy or knows decoder suffering losses
36
Operational Encoder Control

Encoder appropriately select parameters





37
Motion vectors
MB modes
Quantization parameters
Reference frames
Spatial and temporal resolution
38
Conclusion

Bad decisions at the encoder can lead to




If no feedback is available


Poor results in coding efficiency
Poor in error resilience
Or both
an increased percentage of intra MBs performs best
If feedback available

39
Interactive Error Control is best
Outline





Video Communication Systems
Error-Resilient Video Transmission
Resynchronization and Error Concealment
Error Mitigation
Summary
40
Summary



Important to understand video can benefit significantly
when data delivered reliably
Introduced error-resilience tools and impact
For good overall performance, should take into account:



41
the selection of error-resilience tools
rate-distortion-optimized mode selection
the channel characteristics
42