Transcript TOPIC 2: GENERALIZATION APPROACHES FOR K

Error Concealment
CMPT-884
CMPT 820
October 19 2010
Jan 18, 2010
Presented by: Cameron Harvey
Internet Basics

The internet is not reliable as a
communication channel
 Packets
can become corrupted
 Packets can get lost
 Packets can have high latency
 Packets can arrive out of order
Encoder/decoder
MULTIMEDIA OVER IP AND WIRELESS NETWORKS, pg 16 FIGURE 2.4
Motion Compensated Prediction

Standard compression techniques use MCP to
estimate a macroblock based on previous or
future frames

Packet loss will affect not just the frame being
decoded, but many other frames as well
 Errors
are propagated temporally until a new Iframe is encountered
Slices

Slices
A
slice is composed of a series of macroblocks
 Slices are non-overlapping
 Slices are structure in such a way that motion
vectors are all contained within the slice
Slice Coding Schemes


Constant MBs/slice
Maximum bytes/slice
Macroblock Allocation Maps

FMO – Flexible Macro-block Ordering
 New
to the H.264/AVC standard is the ability to
define slice groups
 Groups can be further divided into separate slices
Flexible Macroblock Ordering



FMO provides a useful tool for error
concealment
We can use FMO to arrange for slice groups
in such a way that macroblocks are not
adjacent to macroblocks from the same slice
group
Slices are packetized separately and
transmitted
Some types of FMO patterns
Y Dhondt, P Lambert, Flexible Macroblock
Ordering – an error resilience tool in H.264/AVC,
Gent, Belgium, Dec 2004
FMO – Type 1

Slices are interleaved in
such a way that a
missing slice will be
surrounded by many
samples which are
decoded correctly
MULTIMEDIA OVER IP AND WIRELESS
NETWORKS, pg 31 FIGURE 2.12
Example

Suppose the packet containing
macroblocks of slice group #0 is lost
Example (2)

The missing macroblock
can be interpolated
using the data from the
surrounding
L
macroblocks
T
R
B

This is type of
concealment is called
Spatial Concealment
Spatial Error Concealment

The key assumption behind Spatial Error
Concealment is that image content is
continuous and has smooth texture
Spatial Error Concealment

The pixel, Xij, is
calculated as the
weighted average of
four pixel values on
the edge of the
macroblock
MULTIMEDIA OVER IP AND WIRELESS
NETWORKS, pg 33 FIGURE 2.13
Spatial Error Concealment (2)

The calculation for pixel (i,j) is


The values of
determine the
impact the border pixels will have and are
generally related to the distance from the pixel
under consideration
Spatial Error Concealment (3)

This technique will result in blurred areas as
the reconstructed macroblock is basically an
average of the surrounding macroblocks

Refinements of this technique can be used to
recover some texturing information
Spatial Error Concealment (4)
Picture 11 from a GOP – 5% packet loss
Error concealment without FMO
Error concealment using FMO
Y Dhondt, P Lambert, Flexible Macroblock Ordering – an error
resilience tool in H.264/AVC, Figure 2, Gent, Belgium, Dec 2004
Temporal Error Concealment


Temporal concealment operates with the
assumption that a video sequence is
continuous
In its simplest form, one can simply copy a
missing macroblock from the corresponding
macroblock in the previous frame. This is
called Previous Frame Concealment (PFC)
 Useful
if the scene is static
 Problematic if the scene changes rapidly as this
creates many artifacts
Temporal Error Concealment (2)

We can improve on PFC with an assumption of
uniform motion. With this assumption, one can
examine the motion vectors of the adjacent
macroblocks to infer the motion of the lost
macroblock

Many possibilities exist. A boundary matching
technique can be used to select from the many
possibilities
Boundary Matching Technique

The sum of the squares of
the differences between
border pixels of Top,
Right, and Bottom is
determined

The motion vector which
minimizes this sum is a
good candidate
MULTIMEDIA OVER IP AND
WIRELESS NETWORKS, pg 35
FIGURE 2.15
Other Temporal Concealment
Refinements

Overlapped Block Motion
 The



Use Median Motion Vector of T, R, and B
Sum of Absolute Difference (instead of square)
Estimate motion with the Top macroblock
 Use

average of three 16X16 pixel regions is used
PFC if top motion vector is unavailabel
Multi-hypothesis error concealment uses a
weighted average of multiple frames
Example

Click to see an example of Spatial-temporal
error concealment
 http://w3.antd.nist.gov/public_safety/concealment/
Project_conceal_files/ecdemo.shtml
Temporal Concealment over a
Scene Change

Packet loss over a scene change violates the
assumption of temporal continuity
 Scene
change detection can be used at the cost
of additional complexity
 Macroblock information over a scene change
exhibit distinct characteristics
Abrupt Scene Changes
S. Pei and Y Chou, Novel Error Concealment Method With Adaptive Prediction to the
Abrupt and Gradual Scene Changes, Proceedings of the 16 th International Conference on
Pattern Recognition (ICPR'02) Volume 3, Figure 2(a) Apr 2002
Abrupt Scene Changes (2)


Use Spatial Concealment on the P-frame
(Why?)
Use Temporal Error Concealment on the B
frames


Use the P or I frame from the same scene as
reference
For IBBP, all four frames must be processed
before concealment can occur
Gradual Scene Changes
S. Pei and Y Chou, Novel Error Concealment Method With Adaptive Prediction to the
Abrupt and Gradual Scene Changes, Proceedings of the 16 th International Conference on
Pattern Recognition (ICPR'02) Volume 3, Figure 2(b) Apr 2002
Gradual Scene Changes (2)



In the first sub-GOP (IBBP or PBBP), temporal
concealment can be used for the final P-frame
(Why?)
B frames are calculated as a weighted
average of the first P or I frame and the final Pframe
Across P frame in different sub-GOP,
interpolation is used aided by the large
number of intracoded macroblocks
Questions
He must be very ignorant
for he answers every
question he is asked.
Voltaire