Transcript Towards a Semantic Turn in Rich

Towards a Semantic Turn
in Rich-Media Analysis
11th International Conference
on Electronic Publishing
Vienna, Austria - June 15th, 2007
Georg Güntner, Tobias Bürger
{georg.guentner,tobias.buerger}
@salzburgresearch.at
Version 1.0 (June 14th, 2007)
© Salzburg Research, 2007
Salzburg Research Forschungsgesellschaft m.b.H.| Jakob-Haringer-Str. 5/III | A-5020 Salzburg
T +43.662.2288-200 | F +43.662.2288-222 | [email protected] | www.salzburgresearch.at
Overview
|
|
|
|
|
|
15.06.2007
Motivation for Leveraging Media Semantics
Knowledge Models and Media Semantics
Case study 1: Automatic Semantic Analysis
Smart Content Factory
Case study 2: Semi-automatic Semantic Analysis
LIVE – Live staging of media events
Discussion of Approaches
Conclusions and Recommendations
© Salzburg Research, 2007
2
Motivation for Leveraging
Media Semantics
15.06.2007
|
The market perspective:
| Corporate, community based & personal audiovisual
collections (archives, libraries): spanning from YouTube
to the big “National Libraries”
| Media producers: broadcasters, film producers, game
producers
| Learning applications, industrial applications
| „Embedded publishers“
| Media analysts
|
The „prosumer crisis“ as a driving factor:
| Consumers become producers
| Content production gets easier: The amount of
manageable assets explodes.
| Professional content production is expensive: re-use is
mandatory
| Solutions of the prosumer crisis depend heavily on
improvements of the search and retrieval process
© Salzburg Research, 2007
3
Domain Ontologies – an Example
Musical Terminology
(e.g. synonyms, classification)
Musical forms
(z.B. musical genres and
forms)
Musical facts
(Instances: e.g. works,
composers, titles)
Geographic names
(places, regions, districts,
countries, areas)
15.06.2007
© Salzburg Research, 2007
4
The Dynamont 3D-Matrix
Model
Scope
●
●
which parts of semantics are modeled?
how is the perspective of the ontology
onto the knowledge of the users?
Level of
Expressiveness
Model
Acceptance
●
●
15.06.2007
●
●
what are the user communities using the
ontology?
which communities accept the ontology?
© Salzburg Research, 2007
what kind of semantics is used?
what kind of semantics are required to
fulfill requirements?
DynamOnt
Austrian research project
in FIT-IT Semantic Systems
(2005 – 2007)
5
Knowledge Models
and Media Semantics
|
Glossary,
controlled vocabulary
|
|
Taxonomy
|
|
Typed associations, schema
based on a data model
Formal Ontology
|
15.06.2007
Taxonomy, enhanced by
associations,
e.g. “A is_related_to B”
Database schema
|
|
Controlled vocabulary,
hierarchically structured
Thesaurus
|
|
Finite number of terms,
uniquely defined
A (meta) model, describing
possible and true
predicates which are
assured by axioms
© Salzburg Research, 2007
© Alistar J. Miles. SKOS Core
Guidelines for Migration.
http://www.w3.org/
6
Approaches to Bridge the
“Semantic Gap”
|
Construction of meaning
Is an act of interpretation based on pre-existing knowledge
(the “context”)
| Can usually not be extracted by low-level feature analysis
 Semantic gap
|
 Solutions:
 Creation of training data sets
 Analysis of usage context
 Automatic, semi-automatic and manual annotation
 Combinations of above annotation approaches approaches
 Social Web approaches
15.06.2007
© Salzburg Research, 2007
7
Case Study 1
Smart Content Factory
|
Vision:
|
|
|
|
|
|
Development of a prototype of a
knowledge-based audiovisual
archive and navigation system for
TV, radio and online broadcasts
Increasing the utilization of
audiovisual content repositories
Duration: 10/2003 - 09/2006
Volume: 735.000 €
Efforts: 80 PM
Partners:
|
|
|
|
Salzburg Research (co-ordinator)
ORF
X-Art ProDivision
Joanneum Research
http://scf.salzburgresearch.at/
15.06.2007
© Salzburg Research, 2007
8
Smart Tracks
Metadata Tracks:
Clinton running
for senate
“Clinton spoke
with reporters”
Speech
Speaking
at rally
“Arrived in New York”
Applause
Improving
Medicare
User Annotation
“Older citizens
who have”
Speech to Text
Speech
Audio Classification
Hillary Clinton
Lynne Russell
>>Hillary Clinton Spoke with reporters when she arrived in New York about her
proposal for older citizens...
00:08:40:00
Speaker ID
Listen
Closed Caption
00:08:26:12
00:08:34:29
Special Report
Medicare Issues
On-Screen Text
Lynne Russell
Hillary Clinton
Face ID
Timecode
Read
Keyframes
Encoded Video
15.06.2007
© Salzburg Research, 2007
Time
Watch
© Virage
9
Location Based Navigation
(using Google Maps™)
15.06.2007
© Salzburg Research, 2007
10
Browsing by Category
15.06.2007
© Salzburg Research, 2007
11
Case Study 2
Live Staging of Media Events
|
Vision:
The central idea of Live is to create
|
|
|
Novel Content Production Methods
Tools for Interactive Digital Broadcasters
New ITV video formats and services
to stage Live Media Events such as
the 2008 Olympic Games.
|
|
|
|
Duration: 01/2006 - 09/2009
Volume: 11.3 mio. €
Efforts: 1165 PM (~100 PY)
Partners:
|
|
|
|
|
|
|
|
|
15.06.2007
Fraunhofer IAIS, Cologne (DE)
Academy of Media Arts, Cologne (DE)
ATOS Origin, Madrid (ES)
Austrian Broadcasting Corporation/ORF (AT)
Pixelpark (DE)
Salzburg Research (AT)
University of Applied Sciences, Cologne (DE)
University of Bradford (UK)
University of Ljubljana (SLO)
© Salzburg Research, 2007
12
Modelling the LIVE Staging Domain
•
Automatic annotators
• Human annotators
• Editor
•
Video
conductor
Event
Content
Intelligent
Content
Model
•
User
•
Staging
•
•
•
Production
Archive
15.06.2007
© Salzburg Research, 2007
Consumer
Content
formats
Stream profiles
Stream interrelation
Audience profile
13
LIVE System Components Related
to Media Semantics
15.06.2007
© Salzburg Research, 2007
14
Intelligent Content Models for
Leveraging Media Semantics
Essence
(videoclip)
(1) Resource Level
Media Asset
(metadata)
(2) Meta Level
Knowledge
(ontologies)
(3) Subject Matter Level
Model
15.06.2007
© Salzburg Research, 2007
15
Discussion of Approaches (1/2)
|
Enrichment of multimedia data
|
|
|
|
|
Combination of low-level feature extraction with
background knowledge
|
|
|
15.06.2007
Analysis of usage context
Analysis of different modalities
Human annotation
Use of Web2.0 approaches
Rich News combines text extraction with Google searches
Google extracts topics from the subtitle channel and links to
related web pages
In intranets: combination with production metadata
© Salzburg Research, 2007
16
Discussion of Approaches (2/2)
|
Using different modalities
|
|
|
|
|
|
Combination of media asset management approaches with
social software approaches
|
15.06.2007
MediaMill: e.g. when visual content is not reflected in closed
caption or in the audio track
MISTRAL: “Measurable intelligent and secure semantic
extraction and retrieval of multimedia data“; Austrian FIT-IT
project
Combination of media asset management approaches with
document management technologies
Virage, Autonomy, Convera, etc.
Google, Yahoo, etc.
Web 2.0 platforms (some of them recently acquired by the
big players in the media community)
© Salzburg Research, 2007
17
A Recipe for the Semantic Turn in
Rich-Media Analysis
|
|
Use existing content (URIs)
Define an appropriate content model
|
|
Integrate existing information sources
|
|
|
|
15.06.2007
Is social tagging an alternative?
Define an appropriate domain knowledge model
|
|
|
Production metadata (speakers, editing), archival
data, playout systems, EPGs
Consider creating links to other sources
Test metadata extraction tools
Consider metadata annotation
|
|
Is the content invariant to the semantic approach?
Must the structures change?
Scope, expressiveness, acceptance
Implement the semantic indexing process
Integrate the system in the production /
archiving workflow
© Salzburg Research, 2007
18
Conclusions & Recommendations
|
|
|
|
|
|
|
|
|
15.06.2007
Define the scope of your application!
Create persistent URIs (J. Hendler)
Include reliable information sources!
Leave data in place – rather integrate through
an RDF store! (J. Hendler, Semantics 2006)
Use intelligent content models support
semantic search!
Consider social software approaches where
applicable!
Train your staff!
Define the role of the archivist!
„Install“ a knowledge engineer!
© Salzburg Research, 2007
19
Thank you for your Attention!
|
Additional information
|
|
|
|
|
Salzburg Research: www.salzburgresearch.at
Salzburg NewMediaLab: www.newmedialab.at
Smart Content Factory: scf.salzburgresearch.at
IST-Project LIVE: www.ist-live.org
Contacts
|
Güntner Georg
T: +43.662.2288.401, M: +43.664.2807149
[email protected]
Salzburg Research
Project manager of Smart Content Factory
Scientific and technological coordinator of LIVE
15.06.2007
© Salzburg Research, 2007
Salzburg Research
Jakob Haringer Straße 5/III
A-5020 Salzburg
Austria/Österreich
Tel. +43.662.2288.400
Fax: +43.662.2288.222
20