Transcript Multimedia Data Stream Management System By David Kleinman

Multimedia Data Stream
Management System
By David Kleinman
Outline




Definition
Motivating Examples
Nine Requirements
Current Systems



Comparison
Brief Overview of current Stream Systems
Preview of My Project
What is it?



Stream of multimedia data from a source
(video camera)
Query stored in a system (This query may
itself change
Process high volumes of data in real-time
Motivating Examples

Security Surveillance






Crowd Security
Air Security
Burglary
Baby Sitting
Traffic Reports
Science


Animal behavior
Ocean
Reqirement #1 - Process
Quickly


Low latency
Messages Processed “In-Stream”


No Storage to perform operation
Active System

Avoid Polling
Requirement #2 – Query using
SigmaQL for Streams
(StreamSigmaQL)



Querying Mechanism
Based on SQL
Express Continuous Streams of Data

Window Construct




Time
Frames
Breakpoints
Merge Operator
Requirement # 3 –Handle
Imperfections




Data might be late delayed, missing, or out-of
sequence
Time out individual calculations or
computations
Challenges with Dealing with out-of-order
data
Mechanism for additional time
Requirement #4 – Generate
Predictable Outcomes



Generate deterministic and repeatable results
Time-ordered deterministic processing
throughout entire pipeline
Important for fault tolerance and recovery
Requirement #5 – Integrate
Stored and Streaming Data


Comparing present with past
Capability to efficiently store, access, and
modify state information
Requirement #6 – Guarantee
Data Safety


Must use a high-availability solution
Secondary System


Synchronizes with primary frequently
Takes over in case of failure
Requirement #7 – Partition and
Scale Automatically


Take advantage of distributed computing
Support multi-threading





Takes advantage of multi-processor
Avoids blocking
Load Balance across machines
Automatic process
Transparent
Requirement #8 – Process and
Respond Instantaneously



Needs to respond in real – time
Highly optimized, minimal overhead
execution path
All system components have high
performance
Requirement #9 - Adaptability





Change queries without restarting
Accept all different types of multimedia
streams
Allow for custom configuration
Work with different systems
API
DBMS

Widely used






Use SQL – but not equipped for Streams
Passive
Do not keep data moving
Difficult to handle out of order data
Difficulty with predictable out comes
Incur latency with seamless integration
Rule Engine




Example – Prolog
Active
Handle imperfections
Troubles with seamless integration
Stream Processing Engine



Handle all the requirements
Not specifically designed to handle
multimedia constraints
Not Specifically designed to handle streams
of multimedia
Chart
DBMS
Rule Engine
SPE
MSPE
Keep data moving
No
Yes
Yes
Yes
SigmaQL
No
No
No
Yes
Handle Imperfections
Difficult
Possible
Possible
Yes
Predictable outcome
Difficult
Possible
Possible
Yes
High availability
Possible
Possible
Possible
Yes
Stored and Streamed data
No
No
Yes
Yes
Distribution and scalability
Possible
POssible
Possible
Yes
Real time
Possible
Possible
Possible
Yes
Adaptability
Possible
Difficult
Possible
Yes
Aurora

DSMS developed at MIT and Brown
Aurora Query Network
QoS
..
.
QoS
QoS
..
.
Stream Management System

Developed at Stanford
Simple Query Plan
Q1
Q2

State3
⋈
State4
Scheduler
State1
Stream1
⋈
State2
Stream2
Stream3
NiagaraCQ




Developed at Wisconsin
First DSMS
Uses a grouping strategy
Not as complete as other two
System Architecture
TelegraphCQ





Developed at Berkeley
Stem – storage point
Eddy – route tuples
Good at handling multiple queries
Adaptive
Adaptivity (Telegraph)
Output
STeMs for join
Queues
R
EDDY
grouped
filter (R.A)
RxSxT
grouped
filter (S.B)
S
T
Input Streams
R



S
T
Runtime Adaptivity
Multi-query Optimization
Framework – implements arbitrary
schemes
My Project






Design a multimedia streaming database
Outline the specifications
The Scheduling algorithm
The query structure
The operators
Etc.