Technical University of Crete

Department of Electronic and Computer Engineering E.G.M. Petrakis Multimedia Data Management Euripides G.M. Petrakis http://www.intelligence.tuc.gr/~petrakis http://courses.ced.tuc.gr

Chania 2010 Introduction 1

Definition



Multimedia

: composite entities combining text, audio, images, video (bit-stream objects), graphics 

Multimedia Information Systems

: database systems that support all multimedia data types and handle very large volumes of information E.G.M. Petrakis Introduction 2

TEXT

 The most common type of information  The least space intensive data type  The form in which text is stored varies (plain ascii, word files, spreadsheets, annotations, database fields etc.)  Text fonts are becoming complex allowing special effects (color, shade, fill etc.) E.G.M. Petrakis Introduction 3

AUDIO

 Mbytes), presented as analog, digital or MIDI  Analog waveform : electrical signal,  amplitude specifies the loudness of the sound  speakers  Digital waveform audio : digital,  less sensitivity to noise and distortion  involves larger processing and storage capacities  Digital Audio Tape (DAT), Compact discs (CD)  WAV (Microsoft’s wave file format) E.G.M. Petrakis Introduction 4

MIDI (Musical Instrument Digital Interface) Furht et.al.96

    Commands that describe how the music should be played are stored (instead of sound) A music synthesizer generates sound Provides high data compression, Widely accepted E.G.M. Petrakis Introduction 5

IMAGES

 Digital images: sequences of pixels 

Pixels:

numbers interpreted to display intensity, color, contrast etc  Binary (0-1 values), gray-scale (8 bits/pixel), colour (3x8 values for RGB)  Space overhead depends on image type, resolution, compression scheme 

Image formats

: tiff, bmb, jpeg etc. E.G.M. Petrakis Introduction 6

Image Concepts and Structures



Binary images

: 1 bit/pixel  black & white photos, facsimile images 

Computer Graphics

: 4 bits/pixel 

Grayscale images

: 8 bits/sample 

Color images

: 16, 24 bits/pixel E.G.M. Petrakis Introduction 7

RGB Representation

 A color is produced by adding  red, green and blue  The straight line R=G=B specifies gray values ranging from black to white E.G.M. Petrakis Introduction 8

YUV Representation

 YUV describes the luminance and chrominance components of an image  1 luminance : gray scale version of an image  Y = 0.299R + 0.587G + 0.114B

 2 chrominance components:  U = 0.564(B - Y)  V = 0.713(R - Y) E.G.M. Petrakis Introduction 9

Conversions



Conversion

between RGB and YUV requires multiplication operations  an approximation:  Y = R/4 + G/2 +B/2, U=(B-Y)/2, V=(R-Y)/2  R = Y + 2V, G = Y – (U + V), B = Y + 2U 

YCbCr

is another color format  for compression  Cb = U/2 + 0.5, Cr = V/1.6 + 0.5

E.G.M. Petrakis Introduction 10

VIDEO

 The most space intensive data type  A sequence of frames  Realistic video playback, transmission, compression/decompression require transfer rates about 30frames/sec  Microsoft’s AVI and Apple’s Quicktime file formats integrate video and audio in the same presentation E.G.M. Petrakis Introduction 11

Audio-video Modes of Operation

 Can be either stored

or

used / transmitted live in real-time  Can be used interactively or non interactively E.G.M. Petrakis Introduction Furht et.al.96

12

GRAPHICS



Objects

described through their basic elements (e.g., 2D, 3D shapes)  these elements can have different sizes, position, orientation, surface, fill etc.  compact representations  generated and can be manipulated by design tools (e.g., CAD tools)  Their descriptions are stored in files E.G.M. Petrakis Introduction 13

2D and 3D graphics objects

Khoshafian Baker 96 E.G.M. Petrakis Introduction 14

MULTIMEDIA objects

 Text, audio, images, video, graphics are elements of complex multimedia objects  Various tools or applications integrate, process and combine multimedia  Applications : multimedia authoring applications that output documents and databases and end-user applications (e.g., video on demand)  Tools : for viewing, updating, querying (presentation viewers, browsers etc.) E.G.M. Petrakis Introduction 15

Multimedia Databases (MDB)

 Means stored information or database management systems (dbms) 

Multimedia dbms

(mdbms) integrate conventional database capabilities together with different technologies such as Hierarchical storage management (HSM) and Information retrieval (IR) E.G.M. Petrakis Introduction 16

Multimedia Technologies

 Technologies integrated within a mdbms    HSM IR support support (exact and approximate) Spatial data types and queries  Interactive querying, relevance feedback, refining  Automatic feature extraction  Automatic content retrieval and indexing  Query optimization E.G.M. Petrakis Introduction 17

Database capabilities

         Persistence : object persist through invocations Transactions : content is inserted, deleted, updated Concurrency control: transactions run concurrently Recovery : failed transactions are not propagated to the db Querying : content can be retrieved Versioning : access previous states of objects Integrity : transactions guarantee consistency of content Security : constraints for accessing/updating objects Performance : optimal data structures and programs E.G.M. Petrakis Introduction 18

Hierarchical Storage Management (HSM)

 Support storage of multimedia objects  On-line : on RAM, magnetic disk  Near-on-line : on optical storage  Off-line : on tapes, shelves  Each level has different  Performance : decreases from top to bottom  Capacity : increases from top to bottom  Cost : decreases from top to bottom E.G.M. Petrakis Introduction 19

Information Retrieval (IR) Capabilities

 Retrieval is the most common operation  Deletions and updates are less common  Exact match : search based on exact information  Inexact: search based on inexact information e.g., probabilistic  The results are ranked by order of relevance  Query refinement  Iterate over query results  Adjust weights of query terms or features  And finally resubmit queries E.G.M. Petrakis Introduction 20

MDBMS architecture

Khoshafian Baker 96 E.G.M. Petrakis Introduction 21

MDBMS Implementation

 Relies on 3 rd component    separate party vendors for each Relational dbms Text retrieval for typical records optical storage module  Audio/image/video retrieval  Feature extraction system  Multimedia object interface for text/audio/graphics/images/video system (e.g. Lucene ) system system E.G.M. Petrakis Introduction 22

Object-Oriented Multimedia Databases

 Better design, better suited for multimedia applications  Uniform handling of data and operations  Data types are objects with internal structures and operations that capture the behavior of objects (e.g., audio playback, video browsing)  OO dbms does not satisfy all MM requirements  Provides primitives for object handling  Multimedia components need to be implemented or integrated E.G.M. Petrakis Introduction 23

Multimedia Applications

 Multimedia Systems suggest a variety of applications  Multimedia conferencing  Multimedia on demand news on demand) (interactive TV,  See next page for more … E.G.M. Petrakis Introduction 24

Multimedia Applications

Furht et.al.96

E.G.M. Petrakis Introduction 25

Multimedia Conferencing (MC)

 Multimedia conferencing enable a number of participants to exchange multimedia information  Each participant has a workstation linked to other workstations over high-speed networks  Each participant can send or receive mm data and perform certain collaborative activities E.G.M. Petrakis Introduction 26

A video conference system

Furht et.al.96

 The biggest performance challenge occurs when the participants transmit voice and video  These are mixed together to form a composite stream consisting of video and voice streams E.G.M. Petrakis Introduction 27

Software Architecture

Furht et.al.96

E.G.M. Petrakis Introduction 28

Architectures

 Fully distributed : direct connections between the participants  Processing and mixing of media at every location  Shortest delay  The connections increase rapidly  Centralized (star) network : a central is connected to every participant  Processing and mixing at central node  The central node waits until all media is received before mixing and broadcasting E.G.M. Petrakis Introduction 29

Architectures (cont.)

 Double star network : a central node from one star network is connected to another central node of another star network  Hierarchical network hierarchically : intermediate nodes, root and leaves (participants) connected  intermediate nodes perform mixing and processing  the completely mixed data is sent to root who broadcasts directly to the leaves  reduces network traffic significantly E.G.M. Petrakis Introduction 30

Multimedia conferencing network architectures

Furht et.al.96

E.G.M. Petrakis Introduction 31

Video on Demand (VoD)

 Fast networks coupled with powerful computers and compression techniques will be capable of delivering stream data in real-time  On-demand multimedia services  interactive entertainment  video news distribution  video rental services  digital multimedia libraries E.G.M. Petrakis Introduction 32

Interactive Television (ITV)

 An ITV system must be capable of providing  basic TV  subscription TV  pay per view  video on demand  shopping  education  electronic newspaper  financial transactions  single-user and multi-user games E.G.M. Petrakis Introduction 33

This Course

  Emphasis on     Text, images, video Information retrieval & systems Data organization Web information systems   Semantic Web Video & MPEG standards No emphasis on    Architectures Specific applications (VoD, ITV,MC) Services E.G.M. Petrakis Introduction 34