Transcript Outline
Multimedia Information Systems
Vijay Atluri
atluri@andromeda
Office 200R Ackerson Hall
Phone: 973-353-1642
Office Hours: 2 hours after the class and by
appointment
1
What are the most known media?
Images
2-d
color images, gray scale medical images 2-d (X-ray) or 3-d
(MRI scans)
Captured images
Captured images need to be analyzed
Synthesized
images / visualizations (Artificially created)
Synthetic images do not need to be analyzed ( featured information is
already available).
Text
sequential
vs. hypertext
semistructured
Organized into chapters, paragraphs etc. or may be indexed using
HTML/SGML/XML
2
What are the most known media?
Video
video
clips, movies
captured
interactive
Audio
digitized
voice or music
1-d time-series
financial,
marketing, production time series data such as stock
prices, sales numbers
Handwritten
electronic
notes
Traditional data
Scientific data
collections
of sensor data, e.g.,
<x,y,z,t,pressure,temperature,resistivity…>
3
MM Applications
Travel Industry
intelligent
travel agent, multimedia presentation,
Entertainment Industry
Film
clip database, video-on-demand, pay-per-view, interactive
TV, in-flight entertainment, video game database, video dating
services
Users will be able to select using a mix of query/retrieval and
browsing capabilities
Education and training
classroom
without walls: distance learning, teleclassrooms,
interactive training, self education, employee reeducation
multimedia training is 40% more effective, retention rate is 30% higher,
learning curve is 30% shorter (study conducted by DoD)
approximately US spends $56.6 billion every year (Marketing research by
Training Magazine)
4
MM Applications
Expert Advice
Auto
repair, medical advice, ..
Home Shopping
multimedia
presentation of goods sold, sales information
Medical databases
Text and photograph archives
Digital Libraries
Office automation
Electronic encyclopedia
DNA databases
Geographic Information Systems
5
MMDBMS: Requirements
Ability to uniformly query multimedia data
query
and seamlessly integrate data contained in different
databases (that may possibly use different schema), flat files,
object-oriented databases, spatial databases, arbitrary legacy
sources
elicit the content of the media data (a challenge by itself) which is
highly dependent on the media type and storage format
merge results from different data sources and media types
Ability to retrieve media objects from a local storage
device in a smooth, jitter-free manner
considering
large storage requirements, highly compressed
format, secondary and tertiary storage devices, mix of storage
devices with different performance characteristics
6
MMDBMS: Requirements
Ability to develop a presentation in audiovisual media from
the answer generated by the query
Ability to deliver the presentation that satisfies various
quality of service requirements
synchronization,
fidelity, temporal constraints
does not suffer from jitter and hiccups
limited buffer availability and bandwidth (output devices may
reside at distributed network nodes)
7
What kind of queries can users ask?
Find all images which are created by J. Smith
Find all images with the same color, shape and texture
Find all images which look like this image
Find all images which look like this sketch
Find all images with the same color distribution like a
sunset photograph
Find all images which contains a part which looks like this
image or sketch
Find companies whose stock prices move similarly
Find other companies that have similar sales patterns with
our company
Find cases in the past that resemble last month’s sales
pattern of our product
8
What kind of queries can users ask?
Find past days in which the solar magnetic wind showed
patterns similar to today’s pattern
Find similar music scores or video clips
Find all images of “sunny days” (we are getting into
semantics)
Find all images which contain a car
Find all images which contain a car and a man who look
like this
Find all image pairs which contain similar objects. (data
mining)
9
Sample Multimedia Scenario (from the text)
Consider a police investigation of a large-scale drug
operation:
Video
data captured by surveillance cameras that record the
activities taking place at various locations
Audio data captured by legally authorized telephone wiretaps
Image data consisting of still photographs taken by investigators
Document data seized by the police when raiding one or more
places
Structured relational data containing background information,
bank records, etc., of the suspects involved
Geographic information systems data containing geographic data
relevant to the drug investigation being conducted
10
Example of Queries for the MM Scenario
A police officer, Tom, has a photograph in front of him. He
wants to find the identity of the person in picture.
Q1:
“Retrieve all images from the image library in which the
person appearing in the photograph appears”
Tom wants to examine pictures of a suspect Dick.
Q2:
“Retrieve all images from the image library in which Dick
appears”
Two types of queries
Image-based
Keyword-based
11
Example of Queries for the MM Scenario
Q1: input is an image, output is a ranked list of images that
are similar to the query image
Need
to know what
“similarity” means
“ranking” means
Need
to efficiently support these two operations
Q2: input is a keyword, output is an image whose name
attribute is Dick
Need
to know how to associate different attributes with images
Need to know how to effectively index and retrieve images based
on such attributes
12
Example of Queries for the MM Scenario
Tom is listening to an audio surveillance tape.It contains a
conversation between two individuals A and B
Q3:
Tom wants to review all audio logs that Dick participated
during some specified time period
Q4:
“Find the identity of B, given that A is Dick
“Find all audio tapes in which Dick was a participant”
Tom is browsing an archive of text documents (old
newspaper archives, police dept files on old unsolved
murder cases, witness statements)
Q5:
“Find all documents that deal with financial transactions with
ABC corporation”
Similar queries may be posed on video data
13
Example of Queries for the MM Scenario
MM Query
Q6:
“Find all individuals who have been photographed with Dick
and who have been convicted of attempted murder in North
America and who have recently had electronic fund transfers
made into their bank account from ABC Corp.”
Need to access heterogeneous database systems
Need to access several MM databases:
•
•
•
•
Mugshot database containing the pictures and names of individuals
Surveillance photograph database of still images
Surveillance video database
Image processing algorithms to determine who is present in which video or
still photograph
14
MM Research Issues
Queries
Need
a single language
with which MM data of different types can be accessed
with which one should be able to specify operations to combine different
media types (just like join, union, intersection, difference, Cartesian
product)
that must be able to access metadata as well as raw data
that must be able to merge, manipulate, and join together results from
different media sources
After
devising such a language, we need techniques to
optimize a single query
develop servers that can optimize processing of a set of queries
15
MM Research Issues
Content
What
is meant by content?
Under what conditions can it be described textually?
Under what conditions it must be described directly through the
original media type?
How can we extract the content of
an image, a video-clip, an audio-clip, a free/structured text document?
How
should we index the results of the extracted content?
What is retrieval similarity?
What algorithms can be used to efficiently retrieve media data on
the basis of similarity?
16
MM Research Issues
Storage
How
well disks, CD-ROM, tape systems and tape libraries work?
How do we design disk/CD-ROM/tape servers so as to optimally
satisfy different clients concurrently when they execute
playback, rewind, fast forward, pause, etc.
17
MM Research Issues
Presentation and Delivery
How
do we specify the content of MM presentations?
How do we specify the form (temporal/spatial layout, fidelity) of
this content?
How do we create a presentation schedule that satisfies these
temporal, spatial and fidelity requirements?
How can we deliver MM presentation to users when there is
a need to interact with other remote servers to assemble the presentation
a bound on the buffer, bandwidth, load, and other resources
a mismatch between the host server’s capabilities and the customer’s
machine capabilities, preferences, etc.?
How
can such presentations optimize Quality of Service?
18