Transcript DV/HDV Tape Drive Synchronization

3dtv.at
DV/HDV Tape Drive
Synchronization
Stereoscopic Displays and Applications Conference
29th – 31th January 2007
San Jose, United States
Overview

Introduction to the Stereoscopic Multiplexer



Simple tape drive synchronization



Concept and architecture
DirectShow capture graphs
Basic idea
Missync detection
Advanced tape synchronization concepts


Buffering data in system memory
Recognizing invalid scenes
Peter Wimmer - 3dtv.at
2
Stereoscopic Multiplexer Concept
Right stream
Left stream
R
L
R
L
Combined video stream
Peter Wimmer - 3dtv.at
3
DirectShow

DirectShow






Architecture to process media data on the Windows platform
Processing objects are called filters
Filters expose input and output pins
Filter pins can be connected to a directed graph
Each capture device is represented by a capture filter
Stereoscopic Multiplexer



Not a proprietary capture applications but …
… implemented as DirectShow capture filter
Can be used from any capture application
Peter Wimmer - 3dtv.at
4
DirectShow Capture Graph
Capture
Device

The capture application loads…



Sink
… the capture device filter
… the downstream filters
Example downstream filters are



… video renderers
… file writer
… network renderer
Peter Wimmer - 3dtv.at
5
Stereoscopic Multiplexer Capture Graph
Left
Capture
Device
3dtv.at
Stereoscopic
Multiplexer
Sink
Right
Capture
Device

The application loads…



… the Stereoscopic Multiplexer
… the downstream filters
The Stereoscopic Multiplexer loads…


… the ‘real’ physical capture drivers
… video decoders (if required)
Peter Wimmer - 3dtv.at
6
Stereoscopic Multiplexer Architecture
Application
User
Mode
Application
DirectShowFilter
Stereoscopic Multiplexer
WDM Video
Capture Filter
Kernel
Mode
Peter Wimmer - 3dtv.at
Miniport
Driver
WDM Video
Capture Filter
WDM Video
Capture Filter
Miniport
Driver
Miniport
Driver
7
Stereoscopic Video Recording

One camcorder + optical attachment




Half resolution per eye
Degraded image quality
Simple playback
Two camcorders + remote control



Full resolution
Optimum image quality
No immediate playback possible
Peter Wimmer - 3dtv.at
8
Dual Tape Playback

Problem



Different number of frames on the tapes
Usually happens when camcorders were turned off
Playing back dual camcorder recordings


Starting both tapes simultaneously  tapes drift apart
Transfer both tapes to harddisk  splitting scenes 
playing each scene separately  ok but inconvenient
Peter Wimmer - 3dtv.at
9
Tape Synchronization

Tape synchronization involves…



… detecting missync
… correcting missync
Detecting missync



Image analysis: tracking motion and parallax
 not reliable, computationally complex
Sound analysis: detecting corresponding events in the left
and right audio track
 does not work for silent scenes
Scene change detection: based on metadata (recording date
and time) in the compressed DV stream
 reliable, no computational overhead
Peter Wimmer - 3dtv.at
10
Basic Tape Synchronization

Wait for
scene change
1. Pause first tape
2. Wait for scene change
on second tape
3. Pause second tape
4. Step forward to
smallest common offset
from the scene changes
5. Send ‘play’ command
to both tapes
Yes
Scene change occurs
on the other tape within
a certain tolerance?
No
Resynchronize
Peter Wimmer - 3dtv.at
Resynchronize

A tolerance of one frame
is good tradeoff between
playback interruptions
and sync quality
11
Basic Tape Synchronization Graph
Tape control
Timecode
Recording date/time, timecode
Microsoft
DV Camera
and VCR
3dtv.at
Metadata
Reader
DV Video
Decoder
3dtv.at
Stereoscopic
Multiplexer
Microsoft
DV Camera
and VCR
3dtv.at
Metadata
Reader
Sink
DV Video
Decoder
Created by Stereoscopic Multiplexer
Peter Wimmer - 3dtv.at
12
Tape Synchronization Graph with Audio
3dtv.at
Audio Slaving
Filter
Media type: A/V interleaved
Microsoft
DV Camera
and VCR
3dtv.at
Metadata
Reader
Audio
Renderer
DV Splitter
DV Video
Decoder
3dtv.at
Stereoscopic
Multiplexer
Media type: Video only
Microsoft
DV Camera
and VCR


3dtv.at
Metadata
Reader
DV Video
Decoder
Audio data are sent to sound card
Audio data are not available for capturing
Peter Wimmer - 3dtv.at
13
HDV Support

HDV standards




Downconversion mode





HDV1: 720p (1280 x 720 pixels), used by JVC
HDV2: 1080i (1440 x 1080 pixels), used by Sony and Canon
MPEG-2 interframe compression
Standard definition playback  low system requirements
Firewire output of camcorders set to DV mode
Camcorders perform downconversion from HDV to DV
DV graphs still work
Native HDV mode



High definition playback  high system requirements
Metadata readers parse MPEG streams
Changes to the filter graph are necessary
Peter Wimmer - 3dtv.at
14
HDV Graph
Audio stream
Microsoft
AVC Tape
Subunit Device

3dtv.at
Audio Slaving
Filter
3dtv.at
Metadata
Reader
MPEG-2
Video Decoder
Audio
Renderer
MPEG-2
Demultiplexer
Video stream
Video aux stream
Microsoft
AVC Tape
Subunit Device
MPEG
Audio Decoder
MPEG-2
Demultiplexer
3dtv.at
Stereoscopic
Multiplexer
3dtv.at
Metadata
Reader
MPEG-2
Video Decoder
MPEG transport stream



PID 0810h: Elementary video stream
PID 0814h: Elementary audio stream
PID 0811h: Video auxiliary stream (contains metadata)
Peter Wimmer - 3dtv.at
15
Advanced Tape Synchronization

Problems with basic tape synchronization




Solution





Frequent playback interruptions
Frames are multiplexed in the order they arrive
Precision only approx. ±1 frame
Synchronize the multiplexer output, not the camcorders
Buffer data in system memory to delay the stream that is ahead
Resynchronize tapes only if buffer is full
Precise synchronization based on offset from scene change
Invalid scene recognition




Detect scenes that are present just on one of the tapes
Pause the other tape until matching scenes have been found
Implementation: Monitor difference of left and right recording date/time
Clock time might not be set accurately  do not monitor absolute time
Peter Wimmer - 3dtv.at
16
Invalid Scene Recognition
Left tape
Scene 1
05/03/2006
12:05
Scene 2
05/03/2006
12:09
5 min.
Right tape
Scene 1
05/03/2006
12:10
5 min.
Scene 2
05/03/2006
12:14
Scene 3
11/08/2006
8:40
1 min.
Scene 3
11/08/2006
8:41
Day 1


Scene 4
11/08/2006
8:45
8 min.
Scene 4
11/08/2006
8:53
Scene 5
11/08/2006
8:52
...
1 min.
Scene 5
11/08/2006
9:02
...
Day 2
Do not drop the first scene after startup
Do not drop the first scene after a recording break (it is likely that the
clocks have been adjusted or another pair of camcorders was used)
Peter Wimmer - 3dtv.at
17
Advanced Tape Synchronization Graph
3dtv.at
Audio Slaving
Filter
Microsoft
DV Camera
and VCR
3dtv.at
Media Sample
Buffer
3dtv.at
Metadata
Reader
Audio
Renderer
DV Splitter
DV Video
Decoder
3dtv.at
Stereoscopic
Multiplexer
Microsoft
DV Camera
and VCR


3dtv.at
Media Sample
Buffer
3dtv.at
Metadata
Reader
DV Video
Decoder
Media sample buffers operate on compressed streams
Buffering one second DV/HDV video requires 3.5 MB of memory
Peter Wimmer - 3dtv.at
18
Stereoscopic Multiplexer Versions

Version 0.5 (May 2006)


Basic DV tape
synchronization
Configuration tool
Peter Wimmer - 3dtv.at

Version 0.6 (Q1 2007)


Advanced DV/HDV tape
synchronization
HDV support (HDV2 only)
19
Thanks for Your Attention
 Q&A
Visit us at http://www.3dtv.at!
Peter Wimmer - 3dtv.at
20