MPEG-7

Hao Jiang Computer Science Department Boston College Oct. 30, 2007 CS335 Principles of Multimedia Systems

Introduction

 To manage large volume of multimedia data, we need schemes for – Media archiving, searching, browsing, filtering.

– Managing media delivery to diverse users.

– Controlling copyright and classification issues …  MPEG-7 (Multimedia Content Description Interface) is a standard by ISO/IEC to achieve these goals.

 The difference of MPEG-7 from previous standards – MPEG-7 defines the schemes to describe media data.

– MPEG-1,2 and 4 defines coding schemes to represent the media itself.

 MPEG-7 is another layer on top of multimedia data. CS335 Principles of Multimedia Systems

Multimedia Description

 A hierarchical description of a documentary movie about Tokyo.

s1 Fest 4 CS335 Principles of Multimedia Systems

Media Description (Another Example)

New York Christmas Rockefeller center Evening (x1, y1, x2, y2) Christmas tree Color histogram 1024x768 200kbytes JPEG Creator of the image Copyright CS335 Principles of Multimedia Systems

The Scope of MPEG-7

  MPEG-7 presents a standard interface for media description.

The standard includes: – The system: storing and transmitting mpeg-7 bit streams.

– Description Definition Language: XML. – Visual – the Description Tools dealing with (only) Visual descriptions.

– Audio – the Description Tools dealing with (only) Audio descriptions.

– Multimedia Description Schemes. – Reference software, conformance testing, profiles and levels, and schema definition. CS335 Principles of Multimedia Systems

Application of MPEG-7

CS335 Principles of Multimedia Systems

MPEG-7 Framework

CS335 Principles of Multimedia Systems

Compression Coding MPEG-1,-2,-4 Transmission Retrieval Streaming Searching Indexing Time Axis

Segment 1 Sub-segment 1 Sub-segment 2 Sub-segment 3 Sub-segment 4 segment 2 Segment 3 Segment 4 Segment 5 Segment 6 Segment 7

Segment Tree

Shot1 Shot2 Shot3

MPEG-7 Framework (cont)

Management Filtering



MPEG-7 Normative elements:

– Descriptors and Description Schemes – DDL for defining Description Schemes – Extensible for application domains

Acquisition Authoring Editing MPEG-7 Browsing Navigation Event Tree

• Introduction • Summary • Program logo • Studio • Overview • News Presenter  MPEG-7 Standard D D DS

description:

DS D DDL D • News Items • International • Clinton Case • Pope in Cuba • National • Twins • Closing • Sports Reference Region Motion CS335 Principles of Multimedia Systems DS D Application domain i.e., Medical imaging Remote sensing images Surveillance video Computer animations and graphics

Rich, highly granular multimedia content

– Video segments, moving regions, shots, frames, … – Audio-visual features: color, texture, shape, … – Semantics: people, events, objects, scenes, … Reference Region Motion Reference Region Motion

An Example MPEG-7 Media Description

 The following example gives an MPEG-7 description of a car that is depicted in an image:

Car

Four wheel motorized vehicle

image.jpg

CS335 Principles of Multimedia Systems

MPEG-7 annotation for 1 frame of the documentary movie.

CS335 Principles of Multimedia Systems

A XML Tutorial

  XML denotes “eXtensible Markup Language”.

XML “tags” data with “markup”s to facilitate data processing.

A declaration Flour Yeast Water Salt Mix all ingredients together. Knead thoroughly. Cover with a cloth, and leave for one hour in warm room. Knead again. Place in a bread baking tin. Cover with a cloth, and leave for one hour in warm room. Bake in the oven at 350° for 30 minutes. Root element elements CS335 Principles of Multimedia Systems

More about XML

     XML is a subset of SGML(Standard Generalized Markup Language).

There are many tools and libraries for parsing XML documents.

XML can be adapted to different situations by defining domain specific types.

XML can then be validated based on the data definitions.

To specify the types in XML, we can use DTD (Document Type Definition) or XML schema. CS335 Principles of Multimedia Systems

XML DTD

DTD is a simple way of defining types in XML.

John Smith.

1032 2001-07-13.

order.dtd

CS335 Principles of Multimedia Systems

XML Schema

XML schema is a more powerful way of defining data types.

Other indicators: all, choice, maxOccurs, minOccurs CS335 Principles of Multimedia Systems

A Complete Example (XML)

Hege Refsnes Cecilie Tove Refsnes Hege Stale Jim Borge Stale Refsnes CS335 Principles of Multimedia Systems

A Complete Example (XML Schema)

CS335 Principles of Multimedia Systems

MPEG-7 Schema

 An Example: ParametricMotionType CS335 Principles of Multimedia Systems

CS335 Principles of Multimedia Systems

Digital Media Respository Model Qu ery MPEG-7 Search Engi ne (X ML Metadata) Similarity Search Sema ntics Query Descriptions Descriptio ns IBM Content Manager (Library Server & Object S erver) MPEG-7 Search Network MPEG-7 Processing Pervasive Usage Environment Sounds like ...

Looks like ...

User CS335 Principles of Multimedia Systems

Descriptors

     Color – Color space, color quantization, dominant color, scalable color, color layout, … etc.

Texture Shape Motion – Object trajectory, camera motion, activity.

Localization CS335 Principles of Multimedia Systems

Description Schemes Examples

CS335 Principles of Multimedia Systems

DS Examples

CS335 Principles of Multimedia Systems

DS Examples

Video Summarization CS335 Principles of Multimedia Systems

DS Example

 “Variation” describes the variation of different AV contents. CS335 Principles of Multimedia Systems

Useful Links

   XML tutorial: http://www.w3schools.com/schema/ MPEG-7 schema search engine http://www.tom.comm.waseda.ac.jp/map7/ MPEG-7 overview http://www.chiariglione.org/mpeg/standards/mpeg 7/mpeg-7.htm

CS335 Principles of Multimedia Systems