English - Leonardo Chiariglione
Download
Report
Transcript English - Leonardo Chiariglione
MPEG for the
past, present and future
of television
Leonardo Chiariglione
Keynote Speech at ATSC Annual Meeting
Washington, DC – 2012/05/08
MPEG for the past of
television
System
Transport Stream
DSM-CC
Timed metadata
Video
MPEG-2 High Profile
Audio
MPEG-1 Audio Layer II
MPEG-2 AAC
MPEG for the past, present and future of TV
2012/05/02
2
An assessment
MPEG has provided the means for the television
business on air, cable and satellite to migrate from the
analogue to the digital age
It has been a collective efforts
Representatives from the entire spectrum of the
broadcasting world provided their requirements
Packages of patent licences were created (outside of
MPEG)
Today there is virtually no broadcasting system that is
not based, at least partially, on MPEG standards
MPEG for the past, present and future of TV
2012/05/08
3
MPEG for the present of
television
Systems
Carriage of MPEG-4 on MPEG-2 TS
Dynamic Adaptive Streaming over HTTP (DASH)
Video
MPEG-4 Advanced Video Coding (AVC)
3D extensions
Audio
Various forms of Advanced Audio Coding (AAC)
Universal Speech and Audio (USAC)
MPEG for the past, present and future of TV
2012/05/08
4
An assessment
The deployment of 2nd generation digital broadcasting
systems continues successfully
Use of the AVC on MPEG-2 TS allows packing twice as
many programs as MPEG-2 Video in an analogue
channel
With IPTV the notion of broadcasting is expanding
MPEG is helping the broadcasting industry
To enrich the broadcasted user experience
Through standard technologies that serve multiple fields
MPEG for the past, present and future of TV
2012/05/08
5
MPEG for the future of
television
Well, that’s what I am supposed to talk about…
MPEG for the past, present and future of TV
2012/05/08
6
Broadcasting is a great
business to be in…
Broadcasting is a proven business with a constant
revenue flow
Many technologies appear that
Can help new competing businesses but
Can also be used to extend the broadcasting business
MPEG is a neutral body devoted to the development of
standard technologies for user communities to exploit
MPEG for the past, present and future of TV
2012/05/08
7
MPEG-H: a video standard for
future broadcasting
ISO/IEC 23008 High Efficiency Coding and Media
Delivery in Heterogeneous Environments
A suite of standards for media coding and delivery
Currently scheduled to appear in 3 parts
MPEG Media Transport
High Efficiency Video Coding
3D Audio
Timeline
Part
CD
DIS
FDIS
MMT
12/07
13/01
13/07
HEVC
12/02
12/07
13/01
3D Audio
14/04
14/10
15/01
MPEG for the past, present and future of TV
2012/05/08
8
Service composition in MMT
MMT Package
composition
information
friendly
with html5
<subtitle>
Bird is flying…
UD contents
3D Earth
Bird is flying…
legacy broadcasting
MPEG for the past, present and future of TV
future broadcasting
2012/05/02
9
Smart delivery in Clouds
video
file
Web
Clouds
pull
pull
MMT Encapsulator
-
Service Component List
Component location
Composition relationship
QoE requirement
How to deliver
How to consume
MMT Frame
- Configuration
- Adaptive delivery
- Sync, Delay, Loss control
2012/05/08
10
Mash-up service
Audio
Video1
Video2
widget
Web page
MMT
assets
Service 1 (MMT Package1)
MPEG for the past, present and future of TV
Service 2 (MMT Package2)
2012/05/08
11
Second screen management
MMT Package
composition
information
<device 1>
…
<device2>
…
Video content
on 2nd device
Additional Information with
Live Content (HTML)
MPEG for the past, present and future of TV
2012/05/08
12
MPEG-H part 2
High Efficiency Video Coding
Primary goal: ≥ 50% better compression than AVC
For large displays (e.g. 4kx2k)
For mobile (e.g. WVGA)
Developed jointly with ITU-T by Joint Collaborative
Team (JCT) on Video Coding
Technically in the same stream of past MPEG Video
Coding standards (AVC and prior standards)
Currently at Committee Draft level
MPEG for the past, present and future of TV
2012/05/08
13
HEVC vs AVC Subjective
Performance Measurements
Preliminary subjective tests carried out for
HEVC Committee Draft
Reasonably small and non-overlapping
confidence intervals
Average rate savings at same (interpolated)
MOS points
HEVC with 67% lower rate in class B (Full HD)
HEVC with 49% lower rate in class C (WVGA)
HEVC with 58% lower rate overall
MPEG for the past, present and future of TV
2012/05/08
14
A performance sample
MPEG for the past, present and future of TV
2012/05/08
15
For those technology aware…
Block-based/Variable block sizes
Block motion compensation
Fractional-pel motion vectors
Spatial intra prediction
Spatial transform of residual difference
Integer-based transform designs
Arithmetic or VLC-based entropy coding
In-loop filtering to form final decoded picture
MPEG for the past, present and future of TV
2012/05/08
16
Where we are (May 2012)
Activity: level of 1,000 documents per meeting crossed
Very diverse participation from industry/academia
Significant technical advance over prior standards
Challenge: computational/implementation complexity
Deliverables:
Spec
Reference software
Conformance
Currently only one profile
MPEG for the past, present and future of TV
2012/05/08
17
HEVC Scalability Extensions/1
Serve the needs of heterogeneous environment of
future digital TV distribution
Multiple devices: SHDTVs, set-top boxes, tablets, smart
phones, PCs etc.
Wide range of processing powers, display sizes and power
consumption needs
Multiple networks: Cable, Satellite, Terrestrial, CDN,
Cable Modem, xDSL, WiFi, 4G
Wide range of channel bandwidths and QoS
Multiple protocols: MPEG-2 TS, DASH, HLS, RTP, UDP
Multiple services: Broadcasting, On demand, Streaming,
Over The Top
MPEG for the past, present and future of TV
2012/05/08
18
HEVC Scalability Extensions/2
Options for backward compatible migration paths
From 720p60 to 1080p60
From1080p60 to 4kx2kx60
From AVC to HEVC
Time Line
May 2012 Preliminary Call for Proposals (public)
July 2012 Final Call for Proposals (public)
October 2012
Evaluation of proposals & start of collaborative design phase
Development of the time line associated with standardisation
phases (WD, CD/PDAM, FDIS/FDAM, DIS/AMD)
MPEG for the past, present and future of TV
2012/05/08
19
Service Transitions
1950s: Color TV
Analog, backward compatible
1990s: Digital TV
New infrastructure required
Transitions from SD to HD
2010s: 3D
Mixed results from services introduction is
Not a single format across all services
MPEG for the past, present and future of TV
2012/05/08
20
Current Status of 3D Video
Increasing investments in 3D production and services
Increasingly capable 3D displays in the market
(many competing and emerging technologies)
Market is primarily stereo
(mix of different formats being deployed right now)
Focus of current MPEG work: development of a new
3D format and associated compression techniques that
could facilitate generation of multiview output
MPEG for the past, present and future of TV
2012/05/08
21
Existing 3DV Coding Formats
Left
Right
Pack pixels from left and right views
into a single frame (loss of
resolution)
Leverage existing infrastructure and
equipment, only require additional
signaling to de-interleave frame
Embraced by broadcasters for first
phase of 3D broadcast services
Full-resolution coding of stereo and
MV video as extension of AVC
High coding efficiency achieved via
inter-view prediction techniques
2D compatibility supported
Adopted as format for 3D Blu-ray
Disc
Being considered for second phase
of broadcast standards
Left View
Right View
Effective prediction between views for high compression performance
MPEG for the past, present and future of TV
2012/05/08
22
Targets of Future 3DV Format
Stereoscopic displays
• Variable stereo baseline
• Adjust depth perception
Left
Right
Limited
Camera
Inputs
Data
Format
Data
Format
Constrained Rate
(based on distribution)
Auto-stereoscopic
N-view displays
MPEG for the past, present and future of TV
• Wide viewing angle
• Large number of
output views
2012/05/08
23
3DV Reference Framework
Limited # Video Inputs
(e.g., 2 or 3 views)
Depth
Estimation
Video/Depth
Encoder
Decoder and
View Synthesis
Larger #
Output Views
1010001010001
+
Binary Representation
& Reconstruction Process
MPEG for the past, present and future of TV
2012/05/08
24
Overview of Call For
Proposals
Test Material & Conditions
8 test sequences (1920x1080 & 1024x768)
4 target bit rates per sequence
2-view and 3-view test scenarios
AVC and HEVC compatible test categories
23 responses (12 AVC + 11 HEVC)
Evaluation in Nov 2011
Objective quality (PSNR of synthesized views)
Subjective assessment (stereo and auto-stereo)
MPEG for the past, present and future of TV
2012/05/08
25
3DV Testing Scenarios
2-view Configuration
Left
Depth
Estimation
3D Video
Codec
Multiview
Rendering
virtual
view
Output to
Stereo
Display
Right
3-view Configuration
Left
Output to
Auto-Stereo
N-view
Display
3D Video
Codec
Center
Right
Depth
Estimation
MPEG for the past, present and future of TV
Multiview
Rendering
virtual
views
2012/05/08
26
Tool Categories
Texture coding
Independent of depth
E.g., inter-view prediction of color view, inter-view prediction
of motion parameters and residual data
Using depth data
E.g., view synthesis prediction, motion prediction
Depth coding
Independent of texture
E.g., depth modeling modes, weighted prediction, reduced
resolution coding
Using texture data
E.g., motion parameter inheritance, intra prediction
MPEG for the past, present and future of TV
2012/05/08
27
3DV Current Status
Extensive subjective evaluation of
20+ proposals to the 3DV CfP
Considered both AVC and HEVC
compatible
Many new coding tools proposed
for MV texture and depth coding
Substantial rate savings compared
to capabilities of existing standards
AVC and HEVC extensions
standardisation under wayd
Add support for depth
New AVC-based coding tools
MV extensions of HEVC
Hybrid architectures
MPEG for the past, present and future of TV
2012/05/08
28
Hybrid Architectures
From a pure compression efficiency point of view, it is
always best to use the most advanced codec
However, when introducing new services, providers
must also consider capabilities of existing receivers
and transition plan
Use Case:
Many terrestrial broadcast systems based on MPEG-2
US cable systems based on mix of MPEG-2 and AVC
May not be easy to simply switch codecs in near term
MPEG for the past, present and future of TV
2012/05/08
29
Multiview Extensions of
HEVC
HEVC Stereo/MV extensions being considered
Extensions expected to be completed approximately 12
months after the base specification, i.e., Jan 2014
Reportedly simple extensions of HEVC can achieve 3040% bit rate reduction compared to HEVC simulcast
Further gains expected if block-level changes to codec
are considered
MPEG for the past, present and future of TV
2012/05/08
30
MPEG-H part 3: 3D Audio
Well, there is no Systems and Video without Audio…
MPEG for the past, present and future of TV
2012/05/08
31
3D Audio requirements/1
High quality: quality of decoded sound perceptually
transparent
Localization and Envelopment: Accurate sound
localization and very high sense of sound envelopment
within a targeted listening area (“sweet spot”)
Flexible Loudspeaker Placement: the transmitted
audio program to be mapped to a rendering setup in
which loudspeakers are in alternate locations
Interactivity: Interactive modification of the sound
scene rendered from the coded representation
MPEG for the past, present and future of TV
2012/05/08
32
3D Audio requirements/2
Rendering on setups with fewer loudspeakers
(including headphones): ability to derive a signal from
the transmitted program material for reproduction with
fewer loudspeakers
Audio/visual alignment & consistency: ability to
adapt the rendered acoustic scene to be consistent
with the visual
Efficiency for decoding on different setups: ability to
be rendered on all reproduction loudspeaker setups or
headphone
MPEG for the past, present and future of TV
2012/05/08
33
3D Audio requirements/3
Appropriate computational complexity and system
latency for the target application (Broadcasting, Spatial
two-(n-)way communication / telepresence
Transcoding for low bandwidth devices: A lower
bandwidth signal can be extracted from the original
program material
Backward compatibility: (e.g. to 5.1 channel
programs and decoders or transmission of HRTF
encoded signals)
MPEG for the past, present and future of TV
2012/05/08
34
Envisioned Architecture
22.2 Loudspeakers
Home
Theatre
Spatialized on Headphones
22.2 Channel
Audio Program
Audio Objects
3D Audio
Encoder
500 kb/s
bitstream
3D Audio
Encoder
In-Frame Speakers
Tablet
TV
Spatialized on Headphones
Audio Scene
Transcoder
Inputs
Audio Channels
Channel
–
“normal” content
Audio Objects
•
64 kb/s
bitstream
–
“helicopter overflight”
500 kb/s primary
channel
64 kb/s cellular channel
Smart
Phone
TV
Spatialized on
Headphones
•
Outputs
–
–
Loudspeakers
Headphones
•
–
Spatialized sound
In-frame loudspeakers
•
“Sound Bar”
Audio Scene
Ambisonics
MPEG for the past, present and future of TV
2012/05/08
35
Audio, part of
an Audio-visual Experience
What is new in video?
Larger, high-resolution displays
Greater sense of envelopment
Possibly closer viewing distance
More efficient compression
Wireless transmission to portable devices
Immersive and enveloping audio experience
MPEG for the past, present and future of TV
2012/05/08
36
Use Case 1
Home Theatre
Many loudspeakers
10.1
22.2
“3D” Experience
Height loudspeakers for greater envelopment
Sense that audio objects are near or distant
When near, they are consistent with video image
MPEG for the past, present and future of TV
2012/05/08
37
22.2 multichannel sound
system (NHK)
TpFC
TpFL
TpFR
Top layer
9 channels
TpC
TpSiL
TpSiR
TpBC
TpBR
TpBL
FLc
FRc
FC
FR
FL
Middle layer
10 channels
TV Screen
SiL
SiR
BtFC
BtFL
BtFR
LFE1
LFE2
BR
BL
BC
MPEG for the past, present and future of TV
Bottom layer
3 channels
LFE
2 channels
2012/05/08
38
Home Theater – Issues
Maintain compatibility with legacy systems: 5.1, stereo
Will content providers really step up to 22.2?
Will consumers adopt 22.2?: 2 front high
How to compress 22.2 for transmission?
MPEG AAC 64 kb/s/channel: -> 1.5 mb/s
How to render for legacy setups?
Relatively slow-growth market
MPEG for the past, present and future of TV
2012/05/08
39
Use Case 2
Personal TV
Small but high-resolution display
“super-tablet”
Local wireless communications
To cable or fiber home hub
Possible audio presentation
Headphones
Loudspeakers around perimeter of display
Issue: To what extent can the user get an enveloping
experience from only “front” loudspeakers?
MPEG for the past, present and future of TV
2012/05/08
40
Use Case 3
Mobile TV
Hand-held display: Smartphone
Headphone listening:
Stereo, perhaps with binauralisation
Fastest growing market: Quick rollout and adoption of
standard
What is new?
USAC for stereo; MPEG Surround for binauralisation
What is the “Wow” factor?
To spur adoption of new technology
MPEG for the past, present and future of TV
2012/05/08
41
User Experience
Spatialized on headphones
Home Theatre
Tablet TV
MPEG for the past, present and future of TV
2012/05/08
42
Flexible Rendering
Content providers create 22.2 program
Consumed on stereo, 5.1, 10.1, 22.2 layouts
Consumed on “wrong” layouts
Mis-positioned surrounds
Missing surrounds
Non-standard layouts
2 front high, 5.1 mid
MPEG for the past, present and future of TV
2012/05/08
43
3D Audio Test Platform
NHK “Loudspeaker
Array Frame” (LAF)
Suggestions for
Alternative platform
for assessing “Home
Theatre” use case
Model for assessing
“Personal TV” use
case
MPEG for the past, present and future of TV
2012/05/02
44
Summary
Provide compelling immersive audio experience for
audio/visual programs
Create content once, present on many different
loudspeaker layouts or on headphones
Provide high compression and high-quality audio
presentation
MPEG for the past, present and future of TV
2012/05/08
45
MPEG-V: Humans have
more than two senses
MPEG for the past, present and future of TV
2012/05/08
46
Augmented Reality
Application Format (ARAF)
Remote
Real World
Environment
Remote
Sensors &
Actuators
Local
Real World
Environment
Local
Sensors &
Actuators
MARBle
User
ARAF file
or stream
Media
Servers
MPEG for the past, present and future of TV
Service
Servers
2012/05/08
47
Augmentation Region in
ARAF
AR service
provider A
User A
Broadcaster
User B
Augmentation
Region
MPEG for the past, present and future of TV
AR service
provider B
2012/05/02
48
The past, the present
and the future, again
25 years ago MPEG selected the digital media way
bringing innovation and interoperability
…and thrived
This year the broadcasting industry has started a move
in future broadcast services with interoperability at its
core
…and it is bound to thrive
MPEG looks forward to continuing a cooperation with
the broadcasting industry providing the necessary
standard technologies
MPEG for the past, present and future of TV
2012/05/08
49
http://mpeg.chiariglione.org/
MPEG for the past, present and future of TV
2012/05/08
50