Transcript PowerPoint 演示文稿

Challenges
in Multimedia Information
Retrieval & Filtering
Media Computing & Web Intelligence
薛向阳
[email protected]
复旦大学计算机科学与工程系
上海市智能信息处理重点实验室
Outline
Media Computing & Web Intelligence, Dept. of CSE, Fudan University



Potential Applications, Query
Examples & Achievements
Basic Concepts & Architectures
Key Techniques & Problems
2015/7/18
薛向阳 － 复旦大学计算机科学系
2
Many Potential Applications
Media Computing & Web Intelligence, Dept. of CSE, Fudan University






Broadcast media selection (e.g. radio channel, TV
channel)
Cultural services (e.g. history museums, art galleries)
Digital libraries (e.g. image catalogue, musical
dictionary, bio-medical imaging catalogues, film,
video and radio archives)
Journalism (e.g. searching speeches of a certain
politician using his name, his voice or his face)
Multimedia directory services (e.g. yellow pages,
Tourist information)
……
2015/7/18
薛向阳 － 复旦大学计算机科学系
3
Video Query Examples（TREC）
Media Computing & Web Intelligence, Dept. of CSE, Fudan University

a specific person


I want all the information you have on Ronald Reagan
a specific thing

2015/7/18
I'm interested in any material on Hoover Dam. I'm looking
for a picture of the OGO satellite
薛向阳 － 复旦大学计算机科学系
4
Informedia – CMU
Media Computing & Web Intelligence, Dept. of CSE, Fudan University

Establishment of large video libraries as a
searchable information resource




2015/7/18
Full content information retrieval in both spoken
language and video/image domains
Integration of speech, image and natural language
understanding for library creation and exploration
Fully automated transcriptions generated entirely
speech recognition or with closed captions
Information summaries at varying detail, both
visually and textually
薛向阳 － 复旦大学计算机科学系
5
CueVideo – IBM
Media Computing & Web Intelligence, Dept. of CSE, Fudan University

Developing fully automatic means for
indexing, hyper-linking and preparation of
media material for effective searching and
browsing by users



2015/7/18
Combines several automated indexing，searching
and browsing tools
Video analysis and summarization
Use of speech recognition for spoken media
retrieval
薛向阳 － 复旦大学计算机科学系
6
Outline
Media Computing & Web Intelligence, Dept. of CSE, Fudan University



Potential Applications, Query Examples
& Achievements
Basic Concepts & Architectures
Key Techniques & Problems
2015/7/18
薛向阳 － 复旦大学计算机科学系
7
An Instance of IR System
Media Computing & Web Intelligence, Dept. of CSE, Fudan University
Document
Corpus
Query
String
IR System
Ranked
Documents
2015/7/18
薛向阳 － 复旦大学计算机科学系
1. Doc1
2. Doc2
3. Doc3
.
.
8
Information Retrieval
Media Computing & Web Intelligence, Dept. of CSE, Fudan University

Information Retrieval (IR) Deals with:

Representation (or Modeling)
Storage
Organization
Access

of / to Information Items



2015/7/18
薛向阳 － 复旦大学计算机科学系
9
Architecture：IR
Media Computing & Web Intelligence, Dept. of CSE, Fudan University
Multi -Modal User Interface
offline
Relevance feedback
Representation ， Modeling
Description (MPEG -7/XML)
Query Processi
Multimedia
ng
Database
Organizing:
Searching
Index Structure
Ranking
2015/7/18
薛向阳 － 复旦大学计算机科学系
10
Information Filtering
Media Computing & Web Intelligence, Dept. of CSE, Fudan University

Generally, the goal of an Information Filtering
(IF) system is to sort through large volumes
of dynamically generated information and
present to the user those which are likely to
satisfy his or her information requirement
2015/7/18
薛向阳 － 复旦大学计算机科学系
11
Architecture：IF
Media Computing & Web Intelligence, Dept. of CSE, Fudan University
representation
Initialization
User Profile
Incoming Information…
Binary
…
Doc Source
Classifier
User
User profile
Utility
Accepted Docs
Functions
Accumulated
Docs
2015/7/18
Learning
薛向阳 － 复旦大学计算机科学系
Feedback
12
Applications using MPEG7
Media Computing & Web Intelligence, Dept. of CSE, Fudan University
2015/7/18
薛向阳 － 复旦大学计算机科学系
13
Comparison：IR & IF
Media Computing & Web Intelligence, Dept. of CSE, Fudan University

Information Retrieval



Information Filtering



User Information Needs or Query – Varying
Database or Collection – Static
User Information Needs or Profile – Static
Incoming Data – Varying
Common to both


2015/7/18
how to represent information
how to select relevant information
薛向阳 － 复旦大学计算机科学系
14
Outline
Media Computing & Web Intelligence, Dept. of CSE, Fudan University



Practical Applications, Query Examples
& Achievements
Basic Concepts & Architectures
Key Techniques & Problems
2015/7/18
薛向阳 － 复旦大学计算机科学系
15
Digital TV Program Filtering &
Searching System
Media Computing & Web Intelligence, Dept. of CSE, Fudan University
•Shot
•Scene
•Key-frame
•Face
•Speech
•Motion
•Ocr
•…
User
DVB-S
DVB-C
MPEG2 TS
XML
Template
Filtering
2015/7/18
Database: >2TB
薛向阳 － 复旦大学计算机科学系
Search Engine
16
Representation – extract low level
features
Media Computing & Web Intelligence, Dept. of CSE, Fudan University

Text Features


Image and Video Features


color, texture, shape, motion, …
Audio (Speech,Music) Features




stop word elimination, stemming, index term selection,
thesauri, word cut…
zero-crossing ratio, short time energy
Spectral, Spectral Flux, Spectral Centroid, LPC, MFCC
Pitch,Rhythm,Timbre,…
Requirements － Good Representation, Fast,
Automatic, Robust
2015/7/18
薛向阳 － 复旦大学计算机科学系
17
Representation – get high level
features
Media Computing & Web Intelligence, Dept. of CSE, Fudan University

Structured Video Analysis



Audio & Visual Object Recognition



Video – Scene – Shot – Key frame
Summaries at varying detail, both visually and
textually
Face,Character,Car,…
Word Spotting,Speech Recognition,Speaker,…
Problem - Low Precision, Infant, Inevitable
Incompleteness in the Representation,…
2015/7/18
薛向阳 － 复旦大学计算机科学系
18
metadata
author
producer
copyright
Media Computing & Web Intelligence, Dept. of CSE, Fudan University
semantics
objects
events
people
structure
regions
segments
features
colors
textures
shapes
2015/7/18
Audio-Visual Data
Searching and Filtering
薛向阳 － 复旦大学计算机科学系
19
Retrieval Model
Media Computing & Web Intelligence, Dept. of CSE, Fudan University






Boolean Model
Vector Model
Probabilistic Model
Fuzzy Set Model
Neural Network
……
2015/7/18
薛向阳 － 复旦大学计算机科学系
20
Storage & Organization
Media Computing & Web Intelligence, Dept. of CSE, Fudan University

Storage



Standardized Descriptors - MPEG-7
Management of XML Documents
Index Structures – For Fast Query



2015/7/18
Inverted File for Text
Index Structure for XML Documents
Index Structures for High Dimensional Vector
(Visual Features) - Dimensionality Curse
薛向阳 － 复旦大学计算机科学系
21
Curse of Dimensionality
Media Computing & Web Intelligence, Dept. of CSE, Fudan University
An Intuitive Explanation
D
Assume n-dimensional points distributed in
super-cubic. Selectivity can be computed：
r


n
n
 r 2 
超球容积
1
 n
P ( n) 
＝
超立方体容积   n
D

  1 
  2
 
（0,0）
r
（0,0）
2015/7/18
D
查询窗口容积  r 
P(n) 
＝ 
超立方体容积  D 
n
When n increasing, P(n) will go down to zero
exponentially. In order to find relevant points,
searching window should be enlarged!
薛向阳 － 复旦大学计算机科学系
22
Multi-Modal Interface － 1
Media Computing & Web Intelligence, Dept. of CSE, Fudan University

Input Information Needs








2015/7/18
Key Word,…
Example Image, Example Face, Example Video Clip,…
Speech, Humming,…
Relevance feedback
How to submit user’s query easily and friendly to IR system?
How can IR system understand user’s query intention?
People are unable to specify that which they don't know
There is inevitable uncertainty in the representation or
understanding of information problems
薛向阳 － 复旦大学计算机科学系
23
Multi-Modal Interface － 2
Media Computing & Web Intelligence, Dept. of CSE, Fudan University

Output Query Results


2015/7/18
Enable user to browse full content in hierarchy or
web
Visualization is important for presentation
薛向阳 － 复旦大学计算机科学系
24
How to Compute Relevance?
Media Computing & Web Intelligence, Dept. of CSE, Fudan University



Relevance is a dynamic and idiosyncratic
relationship between person and information
object
Information objects mean many different
things to different people (or the same
person at different times)
There is inherent uncertainty in the relevance
relationship
2015/7/18
薛向阳 － 复旦大学计算机科学系
25
Comparison：IR & DR
Media Computing & Web Intelligence, Dept. of CSE, Fudan University
DR（数据检索）
IR（信息检索）
Matching
Exact
Partial, Fuzzy
Inference
Deduction
Induction
Model
Deterministic
Probabilistic
Classification
Monothetic
Polythetic
Query Language
Artificial
Natural
Query
Specification
Complete
Incomplete
Items Wanted
Matching
Relevant
Error Response
Sensitive
Insensitive
2015/7/18
薛向阳 － 复旦大学计算机科学系
26
Conclusion
Media Computing & Web Intelligence, Dept. of CSE, Fudan University




Many types of data without strict structure in
huge multimedia database
Almost all algorithms of intelligent information
processing and recognition (audio & visual)
are necessary for better representation
Seeking good retrieval model may be key to
reduce gap between person and computer
Uncertain & chaotic task – unable to be
formulated
2015/7/18
薛向阳 － 复旦大学计算机科学系
27
Media Computing & Web Intelligence, Dept. of CSE, Fudan University
Q/A？
Thank You！
2015/7/18
薛向阳 － 复旦大学计算机科学系
28