The MPEG Standard
Download
Report
Transcript The MPEG Standard
The MPEG Standard
MPEG-1 (1992) actually a video player
plays out audio/video streams
same type of access as home VCR
MPEG-2 (1995) introduced for compression
and transmission of digital TV signals
still limited interactivity
MPEG-4 (1999) is completely different
high level of interactivity
MPEG-7 (2002) for the description of
metadata only
E.G.M. Petrakis
The MPEG Standard
1
MPEG-4
MPEG-4 addresses the need towards
Mixing of natural and synthetic audiovisual
information
High interactivity in the presentation of
multimedia content
Deployment of communication systems for realtime or broadcast delivery of coded data streams
A new approach for describing, coding and
presenting a scene
MPEG-4 combines different coding tools for
Audio/video
Synthetic objects and graphics
E.G.M. Petrakis
The MPEG Standard
2
MPEG-4 Objects
The audio/video components of MPEG-4
Objects are coded, transmitted separately
and composed at the decoder site
They can exist independently
Multiple objects can be grouped together
to form complex objects
Video and audio can be easily manipulated
Permits choosing appropriate coding tools
for audio, video and graphics objects
E.G.M. Petrakis
The MPEG Standard
3
MPEG-4 Object Based Coding
E.G.M. Petrakis
The MPEG Standard
4
MPEG-4 Coding
The scene is composed and rendered at the
sender site
video frames, audio are coded, multiplexed
and transmitted
tools for coding arbitrarily shaped objects
At the receiver the stream is demultiplexed
video and audio are decoded, composed,
synchronized and presented as defined at
the senders site
E.G.M. Petrakis
The MPEG Standard
5
Object Coding
Objects are described mathematically
(e.g. by their positions)
similarly for audio and graphics objects
an object need only be defined once
the viewer can change their position
transmit calculations to update the scene
at the receiver
this is a critical feature when the response
has to be fast and bit-rate is limited
E.G.M. Petrakis
The MPEG Standard
6
Binary Format for Scenes (BIFS)
MPEG-4’s language for describing and
dynamically changing a scene
Borrows concepts from VRML
Both define representations of the same data
VRML defines objects and actions in text
BIFS code is binary (10-15 times shorter)
Unlike VRML, MPEG-4 uses BIFS for realtime streaming: a scene can be built-up and
played on the fly
VRML
and BIFS evolve
consistently
E.G.M. Petrakis
The MPEG Standard
7
scene
graph
E.G.M. Petrakis
The MPEG Standard
8
The Scene Graph
Represents a scene as independent or
compound objects e.g.,
father and child
the audio track of his voice
floor and walls (sprites: for backgrounds)
the web site
the synthetic image of the furniture
a synthetic HDTV set playing a movie from
the families DVD library
E.G.M. Petrakis
The MPEG Standard
9
Elementary Streams (ES)
The scheme for preparing content for
transmission, storage and decoding
Objects are placed in ESs
Probably two or more ESs per object
A sound track or a video may have a single ES
Scalable objects way have one ES for basic
quality information + one or more
enhancement layers for improved quality (e.g.,
finer detail, faster motion)
ESs are split into packets and sent along with
timing information for proper synchronization
E.G.M. Petrakis
The MPEG Standard
10
Object Descriptors (OD)
MPEG-4s mechanism that informs the
system which ES belongs to a certain
object
OD contain Elementary Stream Descriptors
(ESD) which tell the system which
decoders to use
ODs are sent in their own stream which
allows them to be added or deleted as the
scene changes
E.G.M. Petrakis
The MPEG Standard
11
Profiles and Levels
MPEG-4 provides a set of tools for
coding multimedia contents
an application may use only subsets of
these tools
Profiles: MPEG-4s definitions of these
subsets for audio, visual, graphics
information
Levels: define the computational
complexity of the profile’s tool subset
Certain combinations of profiles fit well
E.G.M. Petrakis
The MPEG Standard
12
together
MPEG-4 Profiles
E.G.M. Petrakis
The MPEG Standard
13
MPEG-4 Visual Objects
Arbitrarily shaped objects are coded apart
from their background
Binary shape coding: a pixel is or is not part
of an object
simple, crude technique, suitable for low-bit rates,
suffers from aliasing
Alpha shape (gray scale) coding: each pixel is
assigned a value for its transparency
objects can be smoothly blended into a background
or with other objects
E.G.M. Petrakis
The MPEG Standard
14
Visual Objects
Rectangular natural images and scenes
are coded using MPEG-1, 2
Texture is coded separately by a DCT,
block based coding scheme or wavelets
E.g., weather reports: the weatherman’s
image seems to be standing in front of a
map which is actually generated
elsewhere
E.G.M. Petrakis
The MPEG Standard
15
Object Segmentation
MPEG does not specify how objects are
extracted
video object segmentation is difficult
e.g., record weatherman’s image in front of
a color background
MPEG-4 specifies decoding
implementation of encoding is left to the
industry to decide
E.G.M. Petrakis
The MPEG Standard
16
MPEG-4 Applications
MPEG-4 makes video possible even at
very low bit-rates (e.g., 10 kb/s)
mobile devices, internet
Scalable objects for low bit-rates
a base layer conveys all the information in
some basic quality
one of more enhancement layers can be
sent to get better quality
send only the most important objects
E.G.M. Petrakis
The MPEG Standard
17
Sprites
For coding unchanged backgrounds
The background is defined and coded
only once
Must be updated for each change (e.g.,
when the viewing angles changes)
The sprite is sent only once
New views are created by sending the
new positions
E.G.M. Petrakis
The MPEG Standard
18
Advanced Features
Map images into computer generated shapes
a 2D or 3D mesh may have an image mapped onto it
a few parameters to deform the mesh generate
the impression of a moving picture
rather than sending new images for each change,
send commands and parameters to the viewer
pre-defined faces are particularly interesting
meshes
the appearance of a face may be left to the
decoder (e.g., custom facial models can be
downloaded)
E.G.M. Petrakis
The MPEG Standard
19
MPEG-4 Faces
Images laid over a
wire-frame face
Send wire-frame plus
parameters
Image reconstruction
at receivers site
Speech is generated
from text in steps
with motions of the
mouth, eyes and lips
E.G.M. Petrakis
The MPEG Standard
20
MPEG-7
MPEG-7 (2002) focuses on description of
multimedia content
modalities: image, speech, video, graphics and their
combinations
MPEG-7 complements existing MPEG
standards and is applicable even to non-MPEG
formats (compressed or uncompressed)
MPEG-7 is driven by trends in technology,
market and user needs
Applications: VideoOnDemand,
NewsOnDemand, InteractiveTV, multimedia
information systems etc.
E.G.M. Petrakis
The MPEG Standard
21
Scope of the Standard
Provides the means for indexing,
searching, filtering and managing audiovisual content
broadcast media selection (e.g.,
personalized TV)
multimedia editing (e.g., personalized news
service)
MPEG-7 interoperable interface defines
syntax and semantics
tools may be designed for specific
modalities, aspects or applications
E.G.M. Petrakis
The MPEG Standard
22
Interoperable Services
and Applications
E.G.M. Petrakis
The MPEG Standard
23
MPEG-7 Main Tasks
Multimedia: generate customized
program guides or summaries of
broadcast audio-visual content
Archive: generate descriptions of audiovisual content (or elements)
Adaptation: filter and transform
multimedia streams in low bit-rate
environments (e.g., mobile users)
E.G.M. Petrakis
The MPEG Standard
24
MPEG-7 Specific Tasks
Music/audio: play a few notes and
return music with similar music/audio
Images/graphics: draw a sketch and
return images with similar graphics
Movement: describe movements and
return video clips with the specified
temporal and spatial relations
Scenario: describe actions and return
scenarios where similar actions take
place
E.G.M. Petrakis
The MPEG Standard
25
MPEG-7 Elements
1. Descriptors (D) : define syntax and
semantics of features of audio-visual
content
Application independent
Low level: shape, motion, color, camera
motion, harmonicity, timbre for audio ...
Semantic level: events, concepts ...
E.G.M. Petrakis
The MPEG Standard
26
MPEG-7 Elements (cont.d)
2. Description Schemes (DS): specify the
structure and semantics of the
relationships among the constituent
Ds or DSs e.g.,
Video DS specify syntax and semantics
for segment decomposition, attributes,
their relationships
DS related to creation, production, and
access of content (e.g., property rights,
parental rating, etc.)
E.G.M. Petrakis
The MPEG Standard
27
MPEG-7 Elements (cont.d)
3. Description Definition Language (DDL):
allows flexible definition of Ds and
DSs based on XML schema
Ds and DSs are application independent
DLLs to define specialized tools
E.G.M. Petrakis
The MPEG Standard
28
MPEG-7 Descriptions
MPEG-7 allows descriptions at different
levels of abstractions
low level features extracted automatically
semantic features with human interaction
or textual annotation
MPEG-7 does not specify how features
are extracted or used (e.g., filtering,
retrieval)
their representation must conform to the
MPEG-7 standard
E.G.M. Petrakis
The MPEG Standard
29
MPEG-7 Parts
Systems: specifies functionality at
system level
Preparation of descriptions for efficient
transport and storage
synchronization of content and descriptors
development of decoders
Description Definition Language (DDL):
language for specifying new Ds and DSs
extension of XML schema
E.G.M. Petrakis
The MPEG Standard
30
MPEG-7 Visual
Specifies a set of standardized visual
Ds and DSs
Color descriptors: color space, quantization
Texture descriptors: homogeneous texture,
texture browsing, edge histogram ...
Shape descriptors: for regions or contours
Motion descriptors: camera motion,
trajectories, motion activity ...
Face recognition
E.G.M. Petrakis
The MPEG Standard
31
MPEG-7 Audio
Specifies standardized audio
descriptors and descriptor schemes for
pure music, pure speech, sound effects,
soundtracks
silence descriptor
spoken content descriptors
sound effects descriptors
melody contour descriptors
E.G.M. Petrakis
The MPEG Standard
32
Multimedia Description Schemes
Specify a framework that allows
generic description of all kinds of
multimedia data
basic elements: data types, structures, Ds
content management: content from
several viewpoints (creation, usage etc.)
organization of content by collections,
classification
navigation and access
user interaction
E.G.M. Petrakis
The MPEG Standard
33
Multimedia Description Schemes
E.G.M. Petrakis
The MPEG Standard
34
MPEG-7 Reference Software
Reference implementation of the
relevant parts of the MPEG-7 standard
The focus is on creating bit-streams of
descriptors and description schemes (DDL
parser, DDL validation, multimedia
description schemes)
Some software for extracting descriptors
is also included (visual, audio descriptors)
E.G.M. Petrakis
The MPEG Standard
35
References
1.
2.
3.
4.
“MPEG-4 Multimedia for our Time” R. Koenen, IEEE
Spectrum, Feb. 1999, pp. 26-33
“Applying and Implementing the MPEG-4 Multimedia
Standard”, J. Kneip et.al. IEEE Micro, Nov-Dec
1999, pp. 64-74
“Overview of the MPEG-7 Standard”, S.-Fu Chang,
T. Sikora and A. Puri, IEEE Transactions on Circuits
and Systems for Video Technology, special issue on
MPEG-7, June 2001
“Everything You Wanted to Know about MPEG-7” F.
Nack and A.T. Lindsay, Part I, II, IEEE Multimedia,
Aug-Dec1999
E.G.M. Petrakis
The MPEG Standard
36