No Slide Title
Download
Report
Transcript No Slide Title
Multimedia Framework and
Libraries
Heejune AHN
Embedded Communications Laboratory
Seoul National Univ. of Technology
Fall 2011
Last updated 2011. 05. 23
Agenda
Intro to Multimedia Framework
FFMPEG Library
Khronos’s OpenMAX
MicroSoft’s DirectX
Heejune AHN: Image and Video Compression
p. 2
1. Multimedia framework
Why framework
Provides base structure for component-based
development
• It is the way many people work for a big system !
Multimedia system shares patterns.
While(!stop){
read;
decode;
render;
}
Framework
Framework
Famous multimedia framework
ffmpeg
Gstreamer (KDE)
Openmax (Khronos Group)
DirectX
Heejune AHN: Image and Video Compression
p. 3
2. FFMPEG
Ffmpeg (http://ffmpeg.org)
free (GNU LGPL license) multimedia library
Farbrice Bellard Since 2004, “ff” = Fast Forwards
Main part
• libavcodec : video/audio encoder and decoder
• libavformat : mux-demux and file format
• command line tools : ffmpeg (transcoder), ffplay, ffserver
Heejune AHN: Image and Video Compression
p. 4
2. FFMPEG : libavcodec structure
AVCodecContext
avcodec_register_all
avcodec_find_encoder
avcodec_alloc_context // new 와 같음
avcodec_ open
// 코덱을 연결하고 초기화함.
avcode_close
// 코덱 사용을 종료 함.
samples
AVFrame
buf (compressed video)
buf (compressed audio)
avcodec_encode_video
avcodec_encode_audio
buf (compressed video)
buf (compressed audio)
avcodec_decode_video
avcodec_decode_audio
samples
AVFrame
codec
name
type // A, V etc
id
// standard
init();
encode();
decode();
close();
AVCodec : 함수덩어리임.
Heejune AHN: Image and Video Compression
p. 5
2. FFMPEG : encoding outline
Encoding (libavcodec/apiexample.c)
avcodec_init( ); // libavcodec 을 초기화 한다.
avcodec_register_all( ); // 모든 코덱을 등록한다.
codec = avcodec_find_encoder(CODEC_ID_XXXX); // 해당하는 코덱을
먼저 찾는다. (코덱이 없으면 ctx 만들어봐야 소용없으므로)
ctx = avcodec_alloc_context(); // 코덱 객체를 생성한다
ctx-> // frame_rate, bir_rate, tolerance, qscale, picture format, picture
resolution 등 코딩 파라메터를 세팅
avcodec_open(ctx, codec); // ctx 에 codec 을 연결한다.
unsigned char *comp_buf[BUF_SIZE];
AVPicture *picture;
// set the data[4] and linesize[4] for YUV420P 포맷
avcodec_encode_video(ctx, comp_buf, size, picture);
avcodec_close(ctx); // 사용을 종료 ?
av_free(ctx); // 메모리 및 객체 제거
Heejune AHN: Image and Video Compression
p. 6
2. FFMPEG : decoding outline
Decoding (libavcodec/apiexample.c)
avcodec_init( ); // libavcodec 을 초기화 한다.
avcodec_register_all( ); // 모든 코덱을 등록한다.
codec = avcodec_find_decoder(CODEC_ID_XXXX); // 해당하는 코덱을
먼저 찾는다. (코덱이 없으면 ctx 만들어봐야 소용없으므로)
ctx = avcodec_alloc_context(); // 코덱 객체를 생성한다
ctx-> // frame_rate, bir_rate, tolerance, qscale, picture format, picture
resolution 등 코딩 파라메터를 세팅
avcodec_open(ctx, codec); // ctx 에 codec 을 연결한다.
AVPacket pkt; // 그리고 read 함 (e.g., av_read_frame())
AVFrame * frame;
// set the data[4] and linesize[4] for YUV420P 포맷
avcodec_decode_video(ctx, frame, &gotFrame, pkt.data, pkt.size);
if(gotFrame) display or process
avcodec_close(ctx); // 사용을 종료 ?
av_free(ctx); // 메모리 및 객체 제거
Heejune AHN: Image and Video Compression
p. 7
2. FFMPEG: libformat structure
AVFormatParameter
av_open_input_file
// new with input file
av_alloc_format_context // new wo input file
name
av_guess_format
// find appropriate avformat
read_probe
file
read_header
iformat
AVStream
AVPacket
read_packet
av_read_packet
read_seek2
AVCondecCtx *
read_timestamp
codec;
stream
read_play
s[ ]
read_pause
read_close
AVOutputFormat
AVInputFormat
AVPack
et
data
pts/dts
stream_i
d
oformat
av_write_header
av_write_frame
av_(interleaved)_write_frame
AVPacket
name
mime_type
write_header
write_packet
write_trailer
av_dump_format
Heejune AHN: Image and Video Compression
p. 8
2. FFMPEG : muxing outline
Encoding (libavformat/out-example.c)
1.
register mux/demux
2. create instances
av_register_all();
fmt = av_open_input_file(&ctx, filename, NULL, 0 NULL);
3. set up codecs for
each stream
st = ctx->streams[sidx];
codecctx = st->codec;
avcodec_find_decoder(CODEC_ID_XXX);
avcodec_open(ctx,.codec);
4. write stream packets
av_read_frame(ctx, &packet);
avcodec_decode_video(codecctx, &Frame, &gotframe, pkt.data, pkt.size);
5. destroy instance
(use the decoded frame )
av_close(ctx);
av_free(ctx);
Heejune AHN: Image and Video Compression
p. 9
2. FFMPEG : demuxing outline
demux (ffplayer.c)
1.
register mux/demux
av_register_all();
2. create instances
fmt = av_guess_format(fmtname, filename, mimetype);
ctx = avformat_alloc_context();
ctx->oformat = fmt;
3. create and
configure streams
st = av_new_stream(ctx, 0);
st-codec->xxx = xxx;
(codec_id, codec_type, bit_rate,
av_set_paramters(ctx, NULL);
width, height, timebase)
(prepare codecs if required)
4. write stream packets
5. destroy instance
av_write_header(ctx);
// get pkt (e.g. encoding a picture)
av_interleaved_write_frame(oc, &pkt);
av_write_trailer(oc);
av_close(ctx);
av_free(ctx);
Heejune AHN: Image and Video Compression
p. 10
4.
History
.
Khronos Group Inc. (famous for
openGL and openCL etc)
Release: DL 1.02 (2007), IL
1.1.2 (2008), AL 1.1 (2010)
Abstraction layer and
interface API
Portability for different HW and
application environ.
C-language, but with OOP
concept
Heejune AHN: Image and Video Compression
p. 11
OpenMAX AL, IL, DL
AL (API for application programmers)
IL (API for framework developers)
video/audio playback and recoding (play, stop, pause, record)
camera control, Image capture and display (capture and show)
Meta data extraction and insertion etc
Core framework
components : source, codec, sink, filter, splitter, mixer, etc
By codec venders, e.g. Packet Video
Bellagio Ref. Impl. (STMicrotech) on Linux
DL (API for component developers)
filter, transform, scaling, fft, etc
Domains (AC: audio codecs, IC:image codecs, IP:image processing,
SP: signal processing, VC:video codecs)
By HW venders, ARM, INTEL, maybe NVIDIA
Heejune AHN: Image and Video Compression
p. 12
OpenMAX IL
3 role players
IL Client : IL API user (commands, event callback)
IL Core : the IL engine (load, configure, connect components)
IL Components : function blocks (HW or SW)
APIs
Core API
Component API
Heejune AHN: Image and Video Compression
p. 13
Media graph
Core and Core API
• Component loading/unloading and building a Graph
Control & Data API
• Commands and Data Flow (not thru. Core for speed-up)
Example of media graph
Heejune AHN: Image and Video Compression
p. 14
OpenMAX IL: Action Flow
Component Life cycle
IL client control the state of component
dynamic/static resource
Has static resource
(2) OMX_Init
(5) Connect
(3) Load_Cmp
Has context
(1) register
(4) loading
Errornous
Has context
Heejune AHN: Image and Video Compression
p. 15
OpenMAX IL: Communication
Port
fmt : video/audio/img/other
output, Input
Communication Mechanisms
Non-tunneling, tunneling,
proprietary
Tunneling
• UseBuffer Call
• Supplier/non supplier port
• allocator/shared port
Flow control
• Out <=In :OMX_FillThisBuffer
• Out=>In : OMX_EmptyThisBuffer
Component Profile
• Base : For easy to implement
• Interoperability : tunneling support
Heejune AHN: Image and Video Compression
p. 16
OpenMAX IL: Thread Control
In-context vs out-of-context (i.e., sync. vs async.)
callback is used before after the client
multi-thread/process for multi-core and HW accelerator
Heejune AHN: Image and Video Compression
p. 17
OpenMAX IL: Integration
With Existing Media Framework
Gstreamer plugin
Heejune AHN: Image and Video Compression
Android PV OpenCore
p. 18
3. DirectShow
History
Video for Windows (VFW): 1991, Windows 3.1
ActiveMovie in1995 (code name Quartz), Windows 95, COM-based
DirectShow in1996, just renamed, in the DirectX family
DirectX 8.0 in 2000, DirectX 9.0 in 2002, DirectX 11.0 in 2009
Now included MS platform SDK (for download)
DShow
COM (C++) library and runtime for multi-media processing
filter-graph-manager and filters
streaming (data flow) process
Heejune AHN: Image and Video Compression
p. 19
DirectShow Architecture
Filters
A COM object with Interfaces (e.g. IBase_Filter )
Pin and connection
• input, output Pins, Connection between in and out pins
Type
• Source, transform (splitter, mixer), renderer filter
SourceFilter
TransformFilter
(Splitter)
TransformFilter
RendererFilter
RendererFilter
Heejune AHN: Image and Video Compression
p. 20
Graph Editor
Graph Editor
DShow built-in GUI filter graph IDE
Can build graphs and peep the filter from a running process.
Filters are logged in System Registry
Heejune AHN: Image and Video Compression
p. 21
Filter Graph Building
IGraphBuilder
create a filter instance (using CLSID), connect, and control
IMediaControl , IMediaEventEx, IMediaPoistion Inerfaces
In fact, they are different interface for one instance.
Connection
Requirement : Pin type (input and output) => media type (audio,
video, samples)=> transport mechanism (i.e., pull, push and buffer
allocation)
Intelligent or manual
• Filter Graph Manager can insert most suitable filters between two filters.
Thread
one thread for one filter graph
CAMthread : A filter can have its-own thread, by inheriting it.
E.g. stream is auto-threaded pin (run, stop, pause etc)
Heejune AHN: Image and Video Compression
p. 22
Comment for DS Developers
Must have Strong COM understanding
All detailed code is COM instance and interface handling.
Most Simple Code Snipset
Initialize the COM library.
• HRESULT hr = CoInitializeEx(NULL, COINIT_APARTMENTTHREADED);
Get a GraphBuilder
• IGraphBuilder *pGraph = NULL;
• hr = CoCreateInstance(CLSID_FilterGraph, NULL,
CLSCTX_INPROC_SERVER,IID_IGraphBuilder,(void**)&pGraph);
Get Control Interface
• hr = pGraph->QueryInterface(IID_IMediaEvent, (void **)&pEvent);
Render file OR
• hr = pGraph->RenderFile(wfilename, NULL));
Or Add Fitlers
• hr = pGraph->AddSourceFilter(wFileName, wFileName, &pInputFileFilter);
• hr = CoCreateInstance(CLSID_DSoundRender, NULL, CLSCTX_INPROC_SERVER,
IID_IBaseFilter, (void **)&pDSoundRenderer);
• hr = pGraph->AddFilter(pDSoundRenderer, L"Audio Renderer");
Heejune AHN: Image and Video Compression
p. 23
Most Simple Code Snipset (Cont’d)
Connect Pins
• pFileOut = GetPin(pInputFileFilter, PINDIR_OUTPUT);
• pWAVIn = GetPin(pDSoundRenderer, PINDIR_INPUT);
• hr = pGraph->Connect(pFileOut, pWAVIn);
Control
• hr = pControl->Run();
Release COM instances
• pFileOut->Release();
• pWAVIn->Release();
• pInputFileFilter->Release();
• pDSoundRenderer->Release();
• pControl->Release();
• pEvent->Release();
• pGraph->Release();
Com Release
• CoUninitialize();
Heejune AHN: Image and Video Compression
p. 24