1

Streaming Video Over the Internet

Andreas Panteli 03/05/2012 ECE 654 Advanced Computer Networks

Streaming Video Over The Internet 03 May 2012

Introduction

    Video principles  When an image appears on the retina it is retained for some millisecond before decaying  If a sequence of images is drawn line by line at 50 images/sec the eye does not notice that it is looking at discrete images Streaming Video   Video content need not be downloaded in full It is being played out while parts of the content are being received and decoded Real-time multimedia have timing constraints   Audio and video data must be played out continuously Bandwidth , delay and loss requirements No QoS guarantees to streaming video, from the current best-effort Internet!

2 03 May 2012 Streaming Video Over The Internet

An Architecture for Video Streaming

Streaming Server Storage Device

Raw Video Video Compression Raw Audio Audio Compression Compressed Video Compressed Audio Application-layer QoS Control Transport Protocol Video Decoder

Client/Receiver

Media Synchronization Audio Decoder Application-layer QoS Control Transport Protocol Internet (Continuous media distribution services)

Streaming Video Over The Internet 3 03 May 2012

Key Areas

 Video Compression  Application-Layer QoS Control  Continuous Media Distribution Services  Media Synchronization Mechanisms  Protocols for Streaming Media

4 Streaming Video Over The Internet 03 May 2012

Key Areas

     Video Compression  JPEG standard  MPEG standard Application-Layer QoS Control Continuous Media Distribution Services Media Synchronization Mechanisms Protocols for Streaming Media

5 Streaming Video Over The Internet 03 May 2012

Digital Video

     Digital video is represented as a sequent of frames, each consisting of a rectangular grid of picture elements (

pixels

) For color video, 8 bits for each of the RGB colors are being used  24 bits/pixel => 16 million colors To produce smooth motion, digital video, we must display at least 25 frames/sec, but to avoid flickering we need to repaint each frame on the screen twice Smoothness and flickering affects the bandwidth requirements  Consider the lower video configuration for computer monitor 1024x768 with 24 bits/pixel and 25 frames/sec  Need to be fed at 472 Mbps We need video compression !

6 03 May 2012 Streaming Video Over The Internet

Video Compression

  Two algorithms are needed 

Encoding

and

Decoding

Asymmetries of the algorithms  Acceptable for the encoding algorithm to be slow and require expensive HW, if the decoding algorithm is fast   Only for video on demand For real-time multimedia like video conferencing, slow encoding is unacceptable  Acceptable to have a slightly different video signal than the original, after encoding and decoding (

Lossy

 Lossy systems are important  system) Accepting a small amount of information loss can give a huge payoff in terms of the compression ratio possible

7 03 May 2012 Streaming Video Over The Internet

The JPEG Standard

 Compression algorithm for still images  Could be applied to each image in succession to achieve video compression     Compute the luminance

Y

   Y = 0.30R + 0.59G + 0.11B

I = 0.60R – 0.28G - 0.32B

and the two chrominances

I

Q = 0.21R – 0.52G + 0.31B

and

Q

Compress the two chrominances twice than the luminance  Lossy reduction, but the eye barely notices it since it responds to luminance more than to chrominance JPEG often produces 20:1 compression or better Decoding a JPEG image requires running the algorithm backwards

8 03 May 2012 Streaming Video Over The Internet

The MPEG Standard (1/2)

  Main algorithms for video compression and international standards since 1993 Can compress both audio and video  Three parts in MPEG-1    Audio Video System (integrates the other two)

9 Streaming Video Over The Internet 03 May 2012

The MPEG Standard (2/2)

    Two kinds of redundancies in videos  Spatial redundancy  Coding each frame separately with JPEG  Temporal redundancy  Consecutive frames are often almost identical MPEG-1 output consists of four kinds of frames    I (Intracoded) frames: Self-contained JPEG-encoded still pictures P (Predictive) frames: Block-by-Block difference with the last frame B (Bidirectional) frames: Differences between the last and the next frame  D (DC-coded) frames: Block averages used for fast-forward MPEG-1 can achieve a compression factor of 40 Decoding has about the same complexity as encoding

10 03 May 2012 Streaming Video Over The Internet

Key Areas

     Video Compression Application-Layer QoS Control  Congestion control  Error control Continuous Media Distribution Services Media Synchronization Mechanisms Protocols for Streaming Media

11 Streaming Video Over The Internet 03 May 2012

Application-Layer QoS Control



Avoids congestion and maximizes video quality in the presence to packet loss

Congestion Control

 Rate Control   Source-Based Rate Control Receiver-Based Rate Control  Hybrid-Rate Control  Rate Shaping  Frame-dropping filter    Layer-dropping filter Frequency filter Re quantization filter

Error Control

  FEC Delay-Constrained Retransmission   Error-Resilient Encoding Error-Concealment

12 03 May 2012 Streaming Video Over The Internet

Rate Control (1/2)

  Technique to determine the sending rate of video traffic based on the estimated available bandwidth of the network Source-based rate control  The sender is responsible for adapting the video transmission rate based on feedback from the receivers  Probe-based approach  Sending rate adjustment to maintain the packet loss ratio below a certain threshold  Model-based approach

13 03 May 2012 Streaming Video Over The Internet

Rate Control (2/2)

  Receiver-based rate control  The receivers regulate the receiving rate of video streams by adding/dropping channels  Probe-based approach  When no congestion is detected, join a channel  => increase of it’s receiving rate  When congestion is detected, drop a channel  => reduction of it’s receiving rate  Model-based approach   Explicit estimation for the available network bandwidth Also based on Equation (1) Hybrid rate Control  Combination of the above mentioned techniques

14 Streaming Video Over The Internet 03 May 2012

Rate Shaping

 Matches the rate of a pre-compressed video bit stream to the target rate constraint  A rate shaper (or filter) is required for source based rate  control Types of filters  Codec filter  Performs transcoding between different compression schemes  Frame-dropping filter  I, P ,B , D frames in MPEG  Frequency filter  Operates in the frequency domain (DCT coefficients)  Re-quantization filter  Re-quantizes the DCT coefficients with a larger quantization step

15 03 May 2012 Streaming Video Over The Internet

Error Control (1/2)

 Forward Error Correction (FEC)  Add redundant information so that original message can be reconstructed in the presence of packet loss  Channel coding     Stream is chopped into segments Each segment is packetized into

k

packets A block code generates an

n

-packet block, where

n

>

k

User only needs to receive any

k

packets out of

n

of the block  Source coding 

N

-th group of blocks contains redundant information of (

n-1

)th group of blocks  Joint source/channel coding  Optimal rate allocation between Channel and Source coding

16 03 May 2012 Streaming Video Over The Internet

Error Control (2/2)

 Delay-constraint Retransmission When the receiver detects the loss of packet N if (T

c + RTT + D s < T d (N))

send the request for packet N to the sender Where

T c

current time

RTT

estimated round-trip time

D s T d (N)

a slack term (e.g receiver’s decoding delay) time when packet N is scheduled for display   Error-resilient Encoding  Enhances robustness of compressed video to packet loss    A video is compressed into multiple streams (

descriptions

) Each description provides acceptable visual quality Combined descriptions provide a better visual quality Error Concealment  Performed by the receiver when packet loss has already occurred  Spatial interpolation  Missing pixel values are reconstructed using neighboring pixels  Temporal interpolation  Loss data is reconstructed using data from the previous frame

17 Streaming Video Over The Internet 03 May 2012

Key Areas

     Video Compression Application-Layer QoS Control Continuous Media Distribution Services  Network Filtering   Application-level Multicast Content Replication Media Synchronization Mechanisms Protocols for Streaming Media

18 Streaming Video Over The Internet 03 May 2012

Network Filtering

 Adequate support from the network is critical in order to provide quality multimedia presentations    Routers have no knowledge of the format of the media streams  Randomly discard packets Filters receive the client’s requests and adapt the stream sent by the server accordingly Typically, frame-dropping  filters are used Drop packets in a way that gracefully degrades the stream’s quality  Increase bandwidth efficiency by discarding late frames R: Router

19 03 May 2012 Streaming Video Over The Internet

Application-level Multicast

    Unicast Multicast Internet’s original design fails to effectively support streaming-media multicast IP multicast has barriers  Scalability, network management, support for higher layer functionality (e.g. error, flow and congestion control) Application-level multicast to the rescue  Builds a multicast service on top of the Internet Media bridge  Routing in the application layer  Each Media Bridge is interconnected with one or more neighboring Media Bridges through explicit configuration  Collectively employ a distributed application-level multicast routing algorthm

20 03 May 2012 Streaming Video Over The Internet

Content Replication



Advantages

    Increased network’s bandwidth efficiency Reduced load on streaming servers Reduced latency for clients Increased availability

Mirroring

 Scatter copies of the original multimedia files in different locations around the Internet  Clients retrieve data from nearest duplicate server

Caching

  Exploits temporal locality Cache retrieves data from server  Clients retrieve data from cache if available  Cache sharing and cache hierarchies

21 03 May 2012 Streaming Video Over The Internet

Key Areas

    Video Compression Application-Layer QoS Control Continuous Media Distribution Services Media Synchronization Mechanisms  Intra-stream Synchronization   Inter-stream Synchronization Inter-object Synchronization  Protocols for Streaming Media

22 Streaming Video Over The Internet 03 May 2012

Media Synchronization

   Multimedia applications  Various integrated media streams that must be presented in a synchronized fashion Media streams may lose synchronization after moving from the server to the client Three layers of synchronization  Intra-stream synchronization  Maintain continuity of logical data units  Inter-stream synchronization  Maintain temporal relationships among different continuous media  e.g. movements of the lips of a speaker do not correspond to the presented audio  Inter-object synchronization  Relation with time-independent data, such as text, still image and streams  e.g. during a presentation, audio is commenting one slide while another slide is presented

23 03 May 2012 Streaming Video Over The Internet

Media Synchronization Mechanisms

   Delay in the network is unpredictable due to the best-effort nature of the Internet Axes-based specifications, or time-stamping  At the source, a stream is time-stamped to keep temporal information within the stream and with respect to other streams Mechanisms implemented on the end systems  Preventive mechanisms   Minimize latencies and jitters i.e. disk-reading scheduling algorithms, network transport protocols, operating systems and synchronization schedulers  Corrective mechanisms  Recover synchronization in the presence of synchronization errors  i.e. Stream synchronization protocol (SSP)  Concept of ”intentional delay”

24 03 May 2012 Streaming Video Over The Internet

Key Areas

 Video Compression  Application-Layer QoS Control  Continuous Media Distribution Services  Media Synchronization Mechanisms  Protocols for Streaming Media   Transport Protocols Session Control Protocols

25 Streaming Video Over The Internet 03 May 2012

Protocols for Streaming Video

 Data Plane  Compressed video/audio is packetized at the real-time transport protocol (RTP) layer  RTP packetized streams are then passed to the UDP/TCP layer and IP layer  IP packets are transported over the Internet  Control Plane  Real-time control protocol (RTCP) and real-time streaming protocol (RTSP) packets are multiplexed at the UDP/TCP layer  IP packets are transported over the Internet

26 03 May 2012 Streaming Video Over The Internet

Transport Protocols

   UDP and TCP provide basic transport functions  UDP and TCP  Multiplexing, Error control (e.g. checksum)  TCP only:  Retransmission, Congestion control, Flow control  UDP is typically employed as the transport protocol for video streams   No packet delivery guarrantees Receiver needs to rely in upper layer to detect packet loss RTP and RTCP run on top of UDP/TCP 

RTP : does not guarantee QoS or reliable delivery

    Time stamping Sequence numbering Payload type identification Source identification

RTCP : designed to work in conjunction with RTP

     QoS feedback Participant identification Control packets scaling Inter-media synchronization Minimal session control information

27 03 May 2012 Streaming Video Over The Internet

Session Control Protocols

  RTSP (Real-Time Streaming Protocol)  Support VCR-like control operations  (stop, pause/resume, fast forward, fast backward)   Choosing delivery channel (UDT, multicast UDT, TCP) Establish and control streams of continuous audio and video media between the media servers and the clients 

Media retrieval:

ask server to setup a session to send the requested media data 

Adding media to an existing session:

server or client can notify each other about any additional media becoming available to the established session SIP (Session Initiation Protocol)   Similar to RTSP Support user mobility by proxying and redirecting request

28 03 May 2012 Streaming Video Over The Internet

References

    A. S. Tanenbaum, Computer Networks, 4th ed. New Jersey: Prentice Hall, 2002.

Dapeng Wu, Hou, Y.T., Wenwu Zhu, Ya-Qin Zhang, Peha, J.M., "Streaming video over the Internet: approaches and directions“,

Circuits and Systems for Video Technology, IEEE Transactions on

, vol.11, no.3, pp.282-300, Mar 2001 Dapeng Wu, Yiwei Thoms Hou, Ya-Qin Zhang, "Transporting real-time video over the Internet: challenges and approaches,"

Proceedings of the IEEE

pp.1855-1877, Dec 2000 , vol.88, no.12, Dapeng Wu and Y. Thomas Hou and Jason Yao and Y. Thomas and Hou Jason Yao and H. Jonathan Chao, “Real time Video over the Internet: A Big Picture”, IEEE NetWorld+Interop, 2000

29 03 May 2012 Streaming Video Over The Internet

30

Questions?

Thank you!

Streaming Video Over The Internet 03 May 2012