Transcript Media Processing Workflow Alex Zambelli Media Technology Evangelist Developer & Platform Evangelism [email protected] Workshop Overview • Silverlight media capabilities • Smooth Streaming 101 – 15 min break •

Media Processing Workflow
Alex Zambelli
Media Technology Evangelist
Developer & Platform Evangelism
[email protected]
Workshop Overview
• Silverlight media capabilities
• Smooth Streaming 101
– 15 min break
• Live Smooth Streaming
• Workflow and architecture
– 15 min break
• Encoding for Smooth Streaming
A Brave New World
• Encoding 320x240 video on a laptop is “so
2002”
• HTTP adaptive streaming has entirely changed
the game in the past 2 years
• The future for Silverlight is all Smooth
Streaming, all HD
• Video quality is better than ever, but at the
price of complexity and bandwidth
• Don’t worry, it only gets easier from here
Silverlight Media Formats
• Windows Media
– ASF file format (*.asf, *.wmv, *.wma)
– VC-1 (WMV9), WMV8, WMV7 video codecs
– WMA Pro, WMA Standard, MP3 audio codecs
• MPEG-4
– MP4 file format (*.mp4, *.3gp, *.mov)
– H.264 video codec (Baseline, Main, High profiles)
– AAC audio codec (AAC-LC profile)
Silverlight Media Extensibility
• Smooth Streaming
– VC-1 and H.264 video
– WMA Pro, WMA Standard and AAC-LC audio
– Supported via Smooth Streaming Media Element
extension
• MediaStreamSource lets developers add
support for 3rd party codecs, formats and
delivery methods
Silverlight Media Protection
• Silverlight DRM, powered by PlayReady
– PlayReady AES-256 encryption
– Windows Media DRM encryption
• Silverlight 3
– VC-1 and WMA encryption only
– Live DRM (Direct License Acquisition) only
• Silverlight 4
– H.264 and AAC encryption
– Offline DRM
Silverlight Media Delivery
• Windows Media Services streaming
– Windows Server 2008 and 2003
• Unicast (WMS HTTP)
• Multicast
• HTTP progressive download
– Any HTTP 1.0/1.1 web server
• IIS Smooth Streaming
– Windows Server 2008 (IIS7)
Silverlight Media Framework
• No need to reinvent the wheel every time a
Silverlight video player is needed
• Silverlight Media Framework:
http://smf.codeplex.com
• All essential playback features plus advanced
Smooth Streaming features
– DVR, Fast Forward, Rewind, Replay, Slow Motion
– Markers, captions, dynamic ad insertion
– Rich analytics and logging
Media Delivery Methods
• Unicast
– Traditional Streaming
– Progressive Download
– HTTP-based Adaptive Streaming
• Multicast
– IP Multicast
Unicast
Origin
Server
Many one-to-one
streams
Requires more
bandwidth per user
May require more
servers
For private and
public networks
Multicast
Origin
Server
A single one-tomany stream
Uses bandwidth of
only one stream
Requires multicastenabled networks
Typically requires
fewer servers
Unicast Methods
04
06
08
Packet
Packet
Packet
Packet
Packet
Internet Streaming Challenge
• Traditional streaming designed for efficient
media delivery
• Works well in small networks but on the
Internet suffers from serious drawbacks:
– Doesn’t scale to today’s Internet audience
– Doesn’t take into account today’s Internet
structure and organization
– Designed for network conditions less volatile than
Internet traffic
HTTP Adaptive Streaming
• Hybrid media delivery method
– Acts like streaming but is in fact a series of short progressive
downloads
– Video and audio delivered as series of small files over HTTP
• Built for the Web
– Leverages existing HTTP caches
– Scales exceptionally well to meet high demand
• Resilient to network unpredictability
– Client can seamlessly switch video quality and bit rate based on
perceived network bandwidth and CPU resources
• Applicable to both on-demand and live delivery
Windows Server Media Delivery
IIS Media Services
• Traditional Streaming
• Unicast
• WMS RTSP
• WMS HTTP
• Multicast
• WMS Multicast
• Progressive Download
• Bit Rate Throttling
• HTTP Adaptive Streaming
• Smooth Streaming
Smooth Streaming
• Smooth Streaming
– Microsoft implementation of HTTP-based adaptive
streaming
– Best of both worlds
• Responsive and efficient like streaming
• Cheap and scalable like progressive download
– Superior user experience
• No buffering, no stutter
• Instant start, instant seek
• Seamless bit rate switching based on bandwidth and CPU
Multi Bit Rate Revisited
00:27.24
2.4 Mbps
1.5 Mbps
700 kbps
300 kbps
03:47.16
05:05.04
07:33.10
Adapting Bit Rate in Real-Time
00
02
04
06
08
00
02
04
06
08
Bit Rate
Heuristics
Smooth Streaming Design
• Smooth Streaming File Format based on MP4 (ISO
Base Media File Format)
• Video is encoded and stored on disk as one
contiguous MP4 file
– One file per bit rate
• Each video Group of Pictures (GOP) is stored in a
separate Movie Fragment box
– This allows accurate fragmentation at key frames
• Contiguous file is virtually split up into chunks when
responding to a client request
Smooth Streaming File Format
Movie Fragment Random Access
Offset (mfro)
Track Fragment Random Access
(tfra)
Movie
Fragment
Random
Access
(mfra)
Media Data (mdat)
(traf)
Track Fragment
(mfhd)
Movie Fragment Header
Media Data (mdat)
•Movie
Extends
Header
(mehd)
•Track
Extends
(trex)
Movie
Fragment
(moof)
(traf)
(mvhd)
• Track
Header
(tkhd)
• Media
(mdia)
Movie
Fragment
(moof)
Track Fragment
Movie
Header
Fragment
(mfhd)
Track
(trak)
Movie
Extends
(mvex)
Fragment
Movie Fragment Header
File Type (ftyp)
Movie Metadata
(moov)
On-Demand Presentations
• MP4 files containing video/audio/data fragments
– New file extensions instead of *.mp4
– *.ismv for files containing video and audio tracks, or only video tracks
– *.isma for files containing only audio tracks
• On-demand server manifest file
– File extension: *.ism
– Describes the relationships between media tracks, bitrates and files on disk
– Uses SMIL 2.0 XML format
• Client manifest file
– File extension: *.ismc
– Describes available streams, codecs, bitrates, resolutions, metadata
– Uses XML format
On Demand Server Manifest
On Demand Client Manifest
Client-Server Communication
• All requests to the server are formed as RESTful URLs
• Client always first requests the client manifest:
http://iis.net/video.ism/Manifest
• After parsing the manifest the client begins making requests
for fragments by indicating bit rates and offset times it
learned from the manifest:
http://iis.net/video.ism/QualityLevels(350000)/Fragments(video=40040000)
http://iis.net/video.ism/QualityLevels(48000)/Fragments(audio=62293333)
• Every valid HTTP request returns a file of MIME type
“video/mp4”
Serving Fragments On Demand
• IIS7 Smooth Streaming handler interprets the RESTful URL
request
• It maps the quality level to a physical ISMV file based on the
server manifest (*.ism) and calculates the fragment location
within the contiguous ISMV file based on the time offset
• Because the wire format (fragment) is just a subset of the disk
format (MP4), the extraction process is simple
– No re-muxing necessary
• Dynamic stream switching logic is fully implemented in
Silverlight application code – no server-side detection
Demo
• DEMO:
• Setting up On Demand publishing points
• Using Fiddler to monitor Smooth Streaming
traffic
Live Smooth Streaming
• Live publishing point identified by *.isml
extension
• Client-server communication very similar to
on-demand
• Server continually appends MP4 file with new
fragments from encoder and makes them
available to client
– This allows DVR-like seeking within the existing
archive
Encoder-Server Communication
• Encoder pushes fragmented MP4 stream to
server in body of a long-running HTTP POST
request
http://iis.net/live.isml/stream(720p)
• Encoder inserts Live Server Manifest Box into
start of MP4 stream
– Contains a SMIL-formatted Live Server Manifest
very similar to On-Demand Manifest
– Describes all tracks in that encoder’s stream
Encoder-Server Communication
• Each sent fragment contains a box with
absolute time and duration
• Timestamps are typically derived from the
input source Linear Time Code (LTC)
• If multiple encoders are used to encode the
same video source, it’s essential that a time
code generator is used to keep their LTC in
sync
Serving Live Fragments
• Server parses the encoder manifests and
starts collecting MP4 fragments
• Server builds a cumulative runtime index in
memory for all incoming fragments
• Server stores fragments into a temporary DVR
ISMV or a permanent ISMV archive
– When the pub point is shut down, server
generates ‘mfra’ index box for ISMV archive
Live Client Workflow
• Client requests the latest manifest
http://iis.net/live.isml/Manifest
• Server adds up all the encoder manifests and the
runtime fragment index and sends the latest
version to the client
• Every fragment contains information about the
next fragment
• No need to repeatedly download manifest
– Client builds its own cumulative runtime index based
on the lookahead info in the fragments
Demo
• DEMO:
• Basic Live Smooth Streaming workflow
Live Smooth Publishing Points
• ISML files are SMIL 2.0 formatted XML
configuration files
• Live publishing points can be created via the
IIS Manager UI or by manually creating ISML
files on server
• Both push and pull publishing models are
supported
– Live publishing points can also push to and pull
from other live publishing points
Live Publishing Point – IIS Mgr
Live Publishing Point - ISML
ISML Syntax
• sourceType
– Push: source streams will be pushed to pub point
– Pull: pub point will pull streams from listed URLs
• publishing
– Streams: pub point can serve streams to other live pub points
– Fragments: clients can request fragments
– Archives: archiving is enabled
• lookaheadChunks: number of fragments server will buffer
before making them available to clients
• manifestWindowLength: length of DVR moving window
• archivePath: local path where archives will be written
Ingests and Origins
Encoder
8 Mbps
Origin Server:
Delayed Stream
Origin Server:
Delayed Stream
Origin Server:
Ingest Server
Realtime Stream
Origin Server:
Realtime Stream
Origin Server:
VOD Archive
Origin Server:
VOD Archive
DEMO
• DEMO:
• Advanced Live Smooth Streaming workflow
Network Design Considerations
• Bandwidth, bandwidth, bandwidth
Encoder egress bit rate =
(All video and audio bit rates together)
x
(Number of ingest publishing points)
• Leave enough headroom for bit rate spikes and
bursts
– Play it: leave 50%-100% headroom
Building Smooth Streaming Origins
• Network
– Make sure NICs can handle the cumulative ingress
and egress bandwidth
• CPU and memory
– Follow typical server configuration guidelines
• Disk
– Live Smooth Streaming can require high
read/write rates
– 10,000 RPM disks are a good idea
Redundancy and Failover
• Encoders
– Active-active redundant encoder configuration not
supported
– Encoder hot swap is possible as long as encoder
doesn’t signal end-of-stream to server
– Configurable time-out periods can help survive
temporary loss of network connectivity
– Some encoders support pushing to multiple
publishing points concurrently, while others do
pub point roll over in case of failure
Redundancy and Failover
• Servers
– IIS is designed to discard duplicate fragments
– This allows ingest and origin servers to be set in
active-active configuration
Ingest server:
Primary
Encoder
Origin server
Ingest server:
Secondary
Redundancy and Failover
• Origins
– CDNs can dynamically change DNS to redirect
traffic to secondary origin when primary fails
Ingest server:
Primary
Origin server:
Primary
D
N
S
Encoder
Ingest server:
Secondary
Origin server:
Secondary
Video Encoding
Smooth Streaming Encoders
• Many vendors to choose from:
–
–
–
–
–
–
–
–
–
–
Anystream (Grab Networks)
Digital Rapids
Envivio
Inlet Technologies
Microsoft
Rhozet (Harmonic)
Telestream
VBrick
ViewCast
Winnov
H.264 or VC-1?
• Customers should choose whichever codec best fits their
encoding workflow and meets their quality requirements
• However, keep in mind:
– H.264 decoding is typically more CPU intensive than VC-1
decoding for the same resolution and frame rate
– In Silverlight 3: H.264 decoding requires about 15-25% more
CPU time than VC-1 decoding with similar content properties
• Good rule of thumb (for now):
– For HD video, use VC-1 to reach largest audience
– For SD video and smaller, use H.264 if quality gains are
noticeable
Encoding for Smooth Streaming
• Use VC-1 Advanced Profile or H.264 Main/High Profile
• 2 second key frame distance works very well
– Must be closed GOP
– Client heuristics work best with fixed length (non-adaptive) GOP
• Set video buffer size to 2x-3x key frame distance
• Use WMA Professional audio codec for best quality at low
bitrates
– Comparable to HE-AAC – excellent stereo quality even at 48
kbps
• Client heuristics are currently tuned for CBR content, so if
using VBR set the peak bit rate within 15% of average bit
rate for best playback
Multi Bit Rate Encoding Revisited
00:27.24
2.5 Mbps
1.5 Mbps
750 kbps
350 kbps
03:47.16
05:05.04
07:33.10
Variable Resolution
• Why do we vary encoded resolution together with bit rate?
• Encoding the same resolution at inappropriately low bit
rates introduces objectionable compression side effects
into the video: blockiness, twirling details, color smearing,
etc.
• By lowering the resolution proportionally to the bit rate we
maintain a consistent level of compression quality in
exchange for giving up some visual detail
• It’s a compromise: between two evils, customers prefer a
blurry picture over a blocky picture
VC-1: 640x360 at 1250 kbps
VC-1: 640x360 at 300 kbps
VC-1: 288x160 at 300 kbps
How Many Bit Rates Do I Need?
• You can use as few as you’d like – even just a
single bit rate
• Upper bound is typically set by the encoder
– Anything more than 12 video bit rates is probably
overkill
• 4 video bit rates for SD and 6 bit rates for 720p
HD are good starting points
• Silverlight client currently only supports 1
audio bit rate
Choosing the Quality Levels
• Smooth Streaming Multi Bit Rate Calculator
http://alexzambelli.com/WMV/MBRCalc.html
Choosing the Quality Levels
• Recommended minimum quality level:
320x176 at 350 kbps
• Never set the minimum quality level below
288x160 or 300 kbps
– Small video = Bad full screen experience
• Compression is most efficient when width and
height are multiples of 16
• One of the resolutions should match the
player video window size
NBC Sunday Night Football
Video:
VC-1 Advanced Profile
Video Bit Rate
Width
Height
FPS
3450
1280
720
29.97
2250
960
540
29.97
1500
720
404
29.97
950
544
304
29.97
600
400
224
29.97
350
288
160
29.97
Audio:
WMA 10 Professional
48 kbps 48 kHz 16-bit stereo
Configuring Live Encoders
• Live encoders have finite CPU resources
• Higher resolution = More CPU cycles
• Overloading a live video encoder will cause
poor image quality and dropped frames
• Most live encoders are bound by number of
CPU cores
Configuring Live Encoders
• Good rule of thumb for live VC-1 encoding:
For each resolution you’d like to use, count the
number of CPU cores needed per this table:
Min Resolution
Max Resolution
CPU Cores Required
16x16
512x288
1
544x304
704x400
2
736x416
1280x720
4
• If total number of needed cores exceeds the
number of CPU cores in your encoder, reduce the
number of quality levels until they fit
Distributed Live Encoding
• Some live encoders (Inlet Spinnaker) support
distributing bit rates of the same video across
multiple encoders
– Muse use LTC time code generator for sync
• Always attach audio stream to the video
stream with lowest bit rate
• Make sure encoder load is distributed evenly
between encoders
– Balance based on pixel count
CBS March Madness
Video Bit Rate
Width
Height
FPS
3450
1280
720
29.97
2400
960
540
29.97
1800
800
448
29.97
1050
544
304
29.97
650
400
224
29.97
350
288
160
29.97
Encoder A
Encoder B
288x160 + audio
400x224
800x448
544x304
960x540
1280x720
Live Encoding Tips & Tricks
• Most live Smooth Streaming encoders can’t handle
above 720p
• If source is 720p50 or 720p59.94, don’t forget to set
frame rate to half
• If source is 1080i, it’s faster to do single field resizing
(bobbing) instead of full frame deinterlacing
• If source is 480i or 576i, use full frame deinterlacing,
preferably motion adaptive
• Super Sampling scaling gives best quality
• Denoising filters can improve compressed quality
Expression Encoder 3: Smooth VC-1
• Required Settings:
–
–
–
–
Profile: VC-1 Advanced Profile
Adaptive GOP: Disabled
Closed GOP: Enabled
Output Mode: Elementary Stream
Sequence Header
– Create separate file per stream:
Enabled
• Recommended Settings:
–
–
–
–
–
–
Resize Mode: Stretch
Adaptive Dead Zone: Conservative
In-Loop: Enabled
Overlap: Enabled
B-Frame Number: 1
Search Range: Adaptive
Expression Encoder 3: Smooth H.264
• Required Settings:
– Profile: H.264 Main or Baseline
– Create separate file per stream:
Enabled
• Recommended Settings:
–
–
–
–
–
–
–
Profile: H.264 Main Profile
Resize Mode: Stretch
Number of Reference Frames: 4
B-Frame Number: 2
Search Range: 256
Entropy Mode: CABAC
In-Loop Filter: Enabled
DEMO
• DEMO:
• Configuring Inlet Spinnaker HD
The More You Know…
• Related MIX10 Sessions:
• Microsoft Silverlight "Media" : Moving at 60fps
– Tuesday 11:30 am
• Introducing the Silverlight Rough Cut Editor
– Tuesday 1:30 pm
• Smooth Streaming Live in HD: From Camera to Screen
– Tuesday 3:00 pm
• Smooth Streaming Live in HD: 2010 Olympic Winter
Games
– Tuesday 4:30 pm
That’s All!
• Please fill out the evaluation forms!
• Enjoy MIX!