Transcript Document

Streaming
Multimedia
Metadata
Frank Nack
WWW9
Amsterdam
&
Jane Hunter
CWI, Amsterdam
DSTC, Uni. Of Qld
[email protected]
[email protected]
MPEG-7 System Architecture
MM Content
XML
Schema
Parser
Description
Generation
MPEG7
Description
Encoder
WWW9
Amsterdam
Description Definition
Language (DDL)
User or data
processing
system
data
Filter
Agents
Description Schemes
(DS)
Descriptors (D)
MPEG7
Coded
Description
Search /
Query
Engine
Decoder
Human
Or
System
MPEG-7 Applications
• Indexing and retrieval of audiovisual archives (pull)
• Broadcast media selection (push media filtering e.g.
news, radio, tv)
• Journalism, Entertainment
• Education
• Surveillance and remote sensing
• Telemedicine, Bio-medical applications
WWW9
Amsterdam
When Is Metadata Streaming Required
• Applications performing in real time
e.g. sports, news
• Interactive Applications
e.g. education, VR games, etc.
WWW9
Amsterdam
MPEG-7 Video Camera
?
WWW9
Amsterdam
Types of Metadata Required
•
•
•
•
•
•
•
WWW9
Amsterdam
Content Description
Rights
Formatting
Administrative (financial, usage, rating)
Delivery
Security
EPGs
Metadata
Actors
Content
Creator
• Original descriptive
information
• Creator IPR
Input
Provider
• Descriptive information
• Content Format
• Content referencing
• Copyright protections
• Payment models
Service
Provider
Network
Provider
• Advanced
Applications, EPG
• Content
management in PDR
•Service Mgt in PDR
•Location and Access
Criteria
•Payment Models
•Billing
•Security
PDR
• Copyright
management
• Storage
management
• Billing
Consumer
•Content mgt
(bookmarking etc)
•Descriptive information
•Payment
•User Preferences
•Usage History
Metadata and
Content Referencing
• Descriptive
information
• Applications
• EPG
• Copyright mgmt
• Service mgmt
• Content Referencing
• Security
Output
Target
Actors
Capturing
Related
Processes
Content Creation & Packaging
Delivery
• Copyright mgmt
• PDR Storage
management
• Payment models
• Descriptive
information
• Content Format
• Content mgmt
• Payment Models
• Usage History
• Content Reference
• Security
Local storing
Consuming
Storing
•Viewing
•Copying
METADATA FLOW IN THE TV-ANYTIME SYSTEM (TV037.DOC)
MPEG-7 Components
Description Definition
 extension
Language
Definition
Tags
DS1
D2
D7
DS2
Instantiation
D1
D3
D6
D10
D1
D4
D9
D8
D5
D2
Structuring
D6
DS4
D3
DS3
D4
D5
Descriptors:
Description Schemes
(Syntax & semantic
of feature representation)
WWW9
Amsterdam
<scene id=1>
<time> ....
<camera>..
<annotation
</scene>
Encoding
&
Delivery
1 10101 0
Description Definition Language (DDL)
Definition:
The Description Definition Language (DDL) is a
language that allows the creation of new Description
Schemes and, possibly, Descriptors. It also allows the
extension and modification of existing Description
Schemes.
(The MPEG-7 DDL is XML Schema Language with
some extensions.)
WWW9
Amsterdam
UML of MediaInformationDS
WWW9
Amsterdam
<complexType name="MediaInformation">
<element ref="MediaProfile" maxOccurs=“unbounded/>
<attribute name="id" type="ID"/>
</complexType>
<complexType name="MediaProfile">
<element ref="MediaIdentification"/>
<element ref="MediaFormat"/>
<element ref="MediaCoding" minOccurs="0" maxOccurs=“unbounded”/>
<element ref="MediaInstance" minOccurs="0" maxOccurs=“unbounded"/>
</complexType>
<complexType name="MediaFormat">
<element name="FileFormat" type="controlledTerm" minOccurs="0"/>
<element name="System" type="controlledTerm" minOccurs="0"/>
<element name="Medium" type="controlledTerm" minOccurs="0"/>
<element ref="FileSize" minOccurs="0"/>
<element ref="Duration" minOccurs="0"/>
<element ref="AudioChannels" minOccurs="0"/>
</complexType>
<complexType name="MediaCoding">
<element ref="FrameWidth" minOccurs="0” type="non-negative-integer"/>
<element ref="FrameHeight" minOccurs="0“ type="non-negative-integer"/>
<element ref="FrameRate" minOccurs="0"/>
<element name="CompressionFormat" type="controlledTerm" minOccurs="0"/>
</complexType>
Description
Definition:
An MPEG-7 Description consists of a DS (structure)
and the set of Descriptor Values (instantiations) that
describe the Data.
MPEG-7 descriptions are XML but are there any nonXML metadata standards?
WWW9
Amsterdam
<MediaInformation>
<MediaProfile>
<MediaIdentification>
<Identifier IdOrg=’MPEG’ IdName=’MPEG7ContentSet’>
mpeg7_content:news1</Identifier>
<Locator>
<MediaURL>file://D:/Mpeg7_17/news1.mpg</MediaURL>
</Locator>
</MediaIdentification>
<MediaFormat>
<FileFormat>MPEG-1</FileFormat>
<System>PAL</System>
<Medium>CD</Medium>
<Color>color</Color>
<Sound>mono</Sound>
<FileSize>666.478.608</FileSize>
<Duration><m>38</m><s>18</s></Duration>
<AudioChannels>1</AudioChannels>
</MediaFormat>
<MediaCoding>
<FrameWidth>352</FrameWidth>
<FrameHeight>288</FrameHeight>
<FrameRate>25</FrameRate>
<CompressionFormat>MPEG-1</CompressionFormat>
</MediaCoding>
</MediaProfile>
</MediaInformation>
Coded Description
Definition:
A Coded Description is a Description that has been
encoded to fulfil relevant requirements such as
compression efficiency, error resilience, random
access, etc.
WWW9
Amsterdam
Streaming MPEG-7
Two possible approaches
1 Multiplex descriptive metadata on top of video
stream - packets interleaved
• only one stream to parse
• don’t know what is lost - metadata or essence?
2 Separate streams - one for video content, another
for the XML metadata description(s)
• can prioritize streams, reserve bandwidth,
specify QoS
WWW9
Amsterdam
Delivery-ready Description
Description
Ref. To Schemas +
Instantiation
(Text or binary)
Stand-alone Elementary Streams (ESs)
Transport
Sync. & encapsulation
data for storage or
streaming
(MPEG-2, MPEG-4)
Delivery-ready Described Media
Multiplex
Delivery-ready Media
Media Data
Transport
Metadata
MPEG-1
MPEG-2
MPEG-4
RealMedia, QT, AVI??
Transport
Sync. & encapsulation
data for storage or
streaming
(MPEG-2, MPEG-4)
Stand-alone Media Stream
Metadata Categories
realtime vs mediatime
Specific metadata is application-dependent, however 4
relevant categories:
• Temporal delivery - metadata needs to be delivered at
a certain real time - before or synchronized with the
audiovisual delivery (content descriptions for filtering)
• Atemporal delivery - metadata can be delivered
anytime (before, during, after - the billing information?)
• Temporal meaning - metadata linked by time stamp to
video content e.g. keyframe.
• Atemporal meaning - e.g. title, director
WWW9
Amsterdam
MPEG-7 Systems Approach
Elementary Streams
• Separate metadata into separate streams according to type,
error sensitivity, temporal aspects
• Do we need to send schema definitions with the
descriptions?
• Store schema defns. on set top box - update periodically
by downloading from URLs
Access Units - MPEG-7 data packets
Synchronization layer - defines buffer model for MPEG-7
data plus interface to enable synchronous consumption
WWW9
Amsterdam
Description Layer
MPEG-7 Elementary Stream
Flattened and compressed
metadata description
Des_ES_ID
Description
AU Link
MPEG-7 AU
ES segmented into
packets with timing
Media Elementary
Stream
Media_ES_ID
Media AU
ES segmented into
packets with timing
Media Layer
Encoding Metadata for Streaming
Encoder has to:
• Flatten nested descriptions;
• Determine optimum packet size;
• Resend header/contextual info periodically to
cover errors, packet loss and mid-way entry;
• Add time stamps to support real time delivery and
synchronizn;
• Binary encode/compress packets;
WWW9
Amsterdam
Linking
Need to be able to link from MPEG-7 descriptions to
media or essence:
• MPEG-2 systems layer
• MPEG-4
• MPEG-4 file format on top of MPEG-2 or MPEG-4
WWW9
Amsterdam
Effect of Lossiness
Built-in error checking but...
• What happens to timestamps/links to essence in
the description when essence packets are lost?
Link from metadata points to nothing.
• What happens if description packets are lost?
Users miss lost ‘highlights’
WWW9
Amsterdam
Streaming Choices
• Define a transport-independent MPEG-7 content stream
and then define the carriage on transport mechanism
• MPEG-7 on MPEG-2 (transport stream private data)
• MPEG-7 on MPEG-4
• MPEG-7 on file (extension of MP4 file format)
• Or Use WAP/WML to define the transport-independent
MPEG-7 content stream
WWW9
Amsterdam
MPEG 1/2
Internal Metadata
•
•
•
•
•
•
•
Image Size
Frame Rate
Motion Vectors
User Data
Private Data Streams
Copyright Markers
Color Separation (RGB ->
YUV)
• Timing and Sync Information
WWW9
Amsterdam
MPEG 2
Meta Data
Program Specific Information
(PSI)
(ISO 13818-1/ITU-T Rec. H222.0, ETS
468)
• Program Association
Table (PAT)
• Program Map Table
(PMT)
• Conditional Access Table
(CAT)
• Network Information Table
(NIT)
WWW9
Amsterdam
Service Information (SI by DVB)
(ETS 468, TM 1324)
• Bouquet Assotiation Table
(BAT)
• Service Description Table
(SDT)
• Event Information Table
(EIT)
• Time and Date Table (TDT)
• Running Status Table (RST)
Video Text (ETS 472)
Closed Caption (TM 1398)
MPEG 4
Media Object Description
• Audio
• TextToSpeech
• Score Driven Synthesis: SAOL (Structured Audio Orchestra
Language)
• Video
• Parametric descriptions of
• a) a synthetic description of human face and body
• b) animation streams of the face and body
• Static and Dynamic Mesh Coding with texture mapping
• Texture Coding for View Dependent applications
• Shape and alpha coding
WWW9
Amsterdam
MPEG 4
Object Content Information
Data stream to send textual information
along with MPEG-4 content
Possibilities:
• classify content according to pre-defined tables
(defined outside of MPEG)
• giving unique labels to content
• storing camera parameters
WWW9
Amsterdam
MPEG 4
Scene Description
BIFS:
• Binary Format for Scenes
• to compose a set of objects into a scene
• coded and transmitted together with the media
objects
• follows a hierarchical structure
• Each node of the graph is a media object
• audiovisual objects have both a spatial and a
temporal extent
• parameters to the composition layer
(e.g. pitch of a sound, the color for a synthetic object)
WWW9
Amsterdam
MPEG 4
Scene Description II
BIFS: Scene Graph
WWW9
Amsterdam
MPEG 4
Stream Concept
WWW9
Amsterdam
MPEG 4
Scene Composition
WWW9
Amsterdam
WAP/WML
Wireless Application Protocol (WAP) - specifn. that
enables mobile users with wireless devices to easily access
and interact with information and services
Wireless Markup Language (WML) - designed to send
XML over low bandwidth with limited memory and CPU
• lightweight, subset of HDML (Hand-held Device Markup)
• specifies presentation/layout of text and images
• specifies user interaction
• linear, non-nested flat context
Binary XML Content Format Specification (WBML) compact binary representation of XML documents
WWW9
Amsterdam
References
MPEG-7
• http://www.cselt.it/mpeg/
• Document N3292 Systems Working Draft 0.3
MPEG-1, MPEG-2, MPEG-4
• http://www.cselt.it/mpeg/
WAP/WML
• http://www.oasis-open.org/cover/wap-wml.html
• http://www.wapforum.org/
• http://www.colorline.no/wap-faq/
WWW9
Amsterdam