Transcript Dali Virtual Machine.

Dali Virtual Machine
Wei Tsang Ooi
(With Brian, Sugata, Tibor, Steve, Haye, Matthew, Dmitriy)
Motivations
We know how to encode, transmit and
store multimedia data today
People start looking into ways to process
multimedia data
Video Editing
Concat, cross-fade, overlay, cut and paste
Gateway
Convert from one format to another
Change compression rate
input video
(format A)
output video
transcoder
format B
DBMS
“ Find all movies in the database that
contain this image “
Set-top Boxes
Subtitling and close captioning
Mixing of incoming movies
Lecture Browser
Match video to slide
Switch between video streams
Current Solutions
Black box C code (mpeg_play)
Hard to reuse/adapt/break apart
Unpredictable performance
Standard library (PPM, IJG)
Least common denominator (RGB for all)
Video frames --> RGB --> gray
History
Rivl -
high level scripting language for
multimedia processing
Still suffers from “black box” problem
Experience
Real cost of processing
Inherently complex operations
Projective transform
Layered operations
Video decoding
Simple operations but lots of data
Copy
Dali Architecture
User
Dali Code
Dali Compiler
Expert User
DVM Code
Dali Virtual
Machine
MMX
C
TriMedia
Today’s Talk
DVM Code
Dali Virtual
Machine
DVM Code : Design Goal
Small number of composable abstractions
Simple, predictable opcodes
High performance
Target for compiler
Easy to extend
Abstractions
Byte image
Bitstream
MPEG
JPEG
Bit masks
PCM
Abstractions
Byte image
Bitstream
MPEG
JPEG
Bit masks
PCM
Byte Image
Two dimensional array of byte
Can be either physical or virtual
physical
image
virtual
image
Byte Image Can Represent
Gray scale image
RGB image
YUV image
Alpha channel
Example Code : Fractal
image
newh
smaller
neww
smaller = byte_new (neww, newh);
byte_shrink_2x2(image,smaller);
dx
target
image
2dy
target = byte_clip(image,0,0,dx,2*dy);
byte_set(target,0);
target = byte_clip(image,3*dx,0,dx,2*dy);
byte_set(target,0);
target
image
dx
smaller
newh
neww
target = byte_clip(image, dx, 0, neww, newh);
byte_copy(smaller, target);
dx
image
2dy
newh
smaller
neww
target = byte_clip(image, 2*dx, 2*dy, neww, newh);
byte_copy(smaller, target);
target = byte_clip(image, 0, 2*dy, neww, newh);
byte_copy(smaller, target);
DVM Code
Dx
= 0.25 * byte_width(image);
dy
= 0.25 * byte_height(image);
neww = 0.5 * byte_width(image);
newh = 0.5 * byte_height(image);
smaller = byte_new (neww,newh);
for (i = 0; i < n; i++) {
byte_shrink_2x2(image,smaller);
Target = byte_clip(image,0,0,dx,2*dy);
byte_set(target,0);
target = byte_clip(image,3*dx,0,dx,2*dy);
byte_set(target,0);
target = byte_clip(image, dx, 0, neww, newh);
byte_copy(smaller, target);
target = byte_clip(image, 2*dx, 2*dy, neww, newh);
byte_copy(smaller, target);
target = byte_clip(image, 0, 2*dy, neww, newh);
byte_copy(smaller, target);
}
General Dali Strategies
Specific instruction
byte_shrink_2x2, byte_shrink_4x4, etc
Explicit memory allocation
byte_new
Reduce data
byte_clip
Composable abstraction
byte image
Performance
run fractal with n = 4 on 800x600 gray scale image
Dali VM
Rivl
Sparc 20
0.5 s
8.8 s
P2 266
0.3 s
4.0 s
about 21 frames/sec for 320x240 video
Abstractions
Byte image
Bitstream
MPEG
JPEG
Bit masks
PCM
MPEG
Getting important and pervasive
Complex format
Complex code (mpeg_play)
Most decoders provides GetNextFrame()
interface
Random access, direct transfer difficult
Abstraction for MPEG Video
seq
hdr
gop
hdr
gop
gop
hdr
gop
...
seq
end
Abstraction for MPEG Video
seq
hdr
pic
hdr
gop
hdr
pic
gop
pic
hdr
gop
hdr
pic
gop
...
...
seq
end
DVM Code Examples
Skip to the n-th frame
for (i = 0; i < n; i++) {
mpeg_pic_hdr_find(bs);
mpeg_pic_hdr_skip(bs);
}
DVM Code Examples
Skip to the n-th frame
for (i = 0; i < n; i++) {
mpeg_pic_hdr_find(bs);
mpeg_pic_hdr_skip(bs);
}
DVM Code Examples
Skip to the n-th frame
for (i = 0; i < n; i++) {
mpeg_pic_hdr_find(bs);
mpeg_pic_hdr_skip(bs);
}
DVM Code Examples
Skip to the n-th frame
for (i = 0; i < n; i++) {
mpeg_pic_hdr_find(bs);
mpeg_pic_hdr_skip(bs);
}
Concat 2 Video Sequences
inbs1
hdr
gop
inbs2
hdr
gop
outbs
hdr
gop
hdr
gop
DVM Code
Concat two sequences
While not end_of(inbs1) {
gop_hdr_dump(inbs1, outbs);
gop_dump(inbs1, outbs);
}
While not end_of(inbs2) {
gop_hdr_dump(inbs2, outbs);
gop_dump(inbs2, outbs);
}
Performance Analysis
Decode MPEG sequences to PPM
No output
Run on 3 different sequences
Performance
Rate (frame/sec)
psycho
Mpeg Play
Sparc 20
Dali VM
Sparc 20
27.3
27.9
8.8
9.9
7.8
9.1
(160x120)
bus
(320x240)
tennis
(320x240)
Performance
Rate (frame/sec)
psycho
Mpeg Play
Sparc 20
Dali VM
Sparc 20
Dali VM
P2 266
27.3
27.9
112.1
8.8
9.9
36.0
7.8
9.1
32.6
(160x120)
bus
(320x240)
tennis
(320x240)
MPEG Strategies
Expose structure
Gop header, gop instead of just frames
Break up layer
Parse, decode, color space conversions
Show Time !
Decode a MPEG sequence
For each frame scale it to “stamp” size
Create a contact sheet
Abstractions
Byte image
Bitstream
MPEG
JPEG
Bit masks
PCM
What Is Bitstream ?
Skip to the n-th frame
for (i = 0; i < n; i++) {
mpeg_pic_hdr_find(bs);
mpeg_pic_hdr_skip(bs);
}
Bitstream
chunk of memory
1011001
parser
Mode of Operations
File i/o
static memory
parser
fread/fwrite
Mode of Operations
Network
static memory
parser
socket
Mode of Operations
Memory map
entire file
parser
mpeg system stream
inbs
audio
video
audio
video
audio
video
mpeg system stream
inbs
audio
video
audio
video
filter
videobs
audio
video
mpeg system stream
inbs
audio
video
audio
video
filter
videobs
parser
audio
video
Bitstream Strategies
Predictable performance
Explicit I/O
Recap: Current Solutions
Black box C code (mpeg_play)
Hard to reuse/adapt/break apart
Unpredictable performance
Recap: Cost of Processing
Inherently complex operations
Layered operations
Simple operations but lots of data
How We Solve the Problems
Difficult to reuse/adapt/break apart
Minimum set of abstractions
Abstractions are simple
Simple reusable opcodes
Unpredictable performance
Explicit I/O
Explicit memory allocations
How We Solve the Problems
Inherently complex operations
Special case
Clipping and masking
Layered operations
Expose structure
Simple operations but lots of data
Special case
Clipping and masking
Work in Progress
Extending with new formats : JPEG, WAV,
AVI etc
TriMedia chip implementation
Application : lecture browser
Future Work
Multithreading
MMX implementation
Compiler