Transcript Technical Overview Open Call Information Teleconferences 16-17 May 2013 Thanos Voulodimos, ICCS/NTUA ([email protected]) EXPERIMEDIA Components  The functional pieces of software in EXPERIMEDIA have been grouped.

Technical Overview
Open Call Information Teleconferences
16-17 May 2013
Thanos Voulodimos, ICCS/NTUA
([email protected])
EXPERIMEDIA Components
 The functional pieces of software in
EXPERIMEDIA have been grouped into logical
“components”
 Components




Provide extension points
Interface to each other
Will interface to your technology
Can be adapted to fit your technology or have existing
parts replaced
 Experiments drive the evolution of the facility
 Pushing the components to provide new capabilities
2
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
Five EXPERIMEDIA Components





3
Experiment Content Component (ECC)
Social Content Component (SCC)
Audio Visual Content Component (AVCC)
Pervasive Content Component (PCC)
3D Content Component (3DCC)
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
Experiment Content Component (ECC)
 Experiment management system
 Support for automatic deployment
 Local machine or Cloud
 Automatic configuration available
 Web based dashboard:
 Experiment control via UI
 Live QoS/QoE/QoC metric visualisation
 Instrumentation API (Java, C#, Ruby, [C++ in development])
 Security modelling
4
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
Instrumentation & the ECC
Your technologies
Mobile apps
Metrics
FMI Content
Desktop apps
Metrics
Internet services
Metrics
AVCC
Metrics
PCC
Metrics
SCC
Metrics
3DCC
Metrics
EXPERIMEDIA baseline components
5
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
The ECC dashboard
 Web based management:
 Connections to instrumented systems
 Experimental process




6
Support for NAGIOS views
Live visualisation of in-coming data
Metric model & database persistence
Data export
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
Social Content Component (SCC)
 SCC publishes, gathers, manages and monitors data
generated on social networks during an experiment
 Provides access to different social networks in terms of
 Authentication
 Reading content
 Publishing content
 Communicates monitoring metrics to the ECC for
analyzing social network activities during the experiment
 Two sub-components:
 SocialIntegrator
 Social Analytics Dashboard (SAD)
7
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
SocialIntegrator API
 Java library for accessing different social networks
transparently in terms of:
 Content retrieval (posts, photos, comments, likes, etc)
 Publishing and modifying capabilities
 Extends SocialAuth Core and SocialAuth Android Java API
 Currently supports Facebook and Twitter
 To be extended for more social networks
 Supports monitoring of multiple QoE
metrics (fully integrated with ECC)
 Two applications developed for FHW:
 Android Social Integrator App (Visitor App)
 Web-based Social Integrator App (Expert App)
8
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
Social Analytics Dashboard
 Configurable, extensible web service for gathering
and analyzing data from social networks.
 Provides a plugin framework with supporting
database and scheduling services along with a
connection to the ECC for reporting metrics.
9
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
Audio Visual Content Component (AVCC)
Overview
Atos provides AVCC as
a service
AVCC
PRODUCER
10
CONSUMER
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
AVCC Services










11
Publish a live stream
Live recording
Play a video on demand
Play a live stream (SD/HD)
Publish live metadata streaming service
Adaptive live stream
Live Timeshift
Ingest
Professional editing/directing
UGC editing/directing
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
AVCC Service Combination Example
Publish an adaptive live stream + timeshift + playing
PRODUCER
(Encoder)
12
CONSUMER
(Player)
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
Industrial robust services:
SMART player @ Paralympic Games
87.395 XML,
565 Megabytes of
Metadata
1.9TB traffic
300.000 hours
130 Terabytes live content
170.000 unique users
2.400.000 sessions
179 countries.
7 users from Kyrgyzstan!
13.500 users watching the
ceremony with English
Audio and 870 users in
Spanish.
13
smart.paralympic.org
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
Pervasive Content Component (PCC)
Overview
Location tracking (Tracker)
Live games (Creator)
Augmented Reality Platform
14
POI Data
Management
QoE measurement
(Babylon)
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
PCC: Augmented Reality Platform
 Overlay camera with information




POI data for example
Show lots of information even on old HW
Customizable icons for POIs
Render icon size according to distance
 Interaction
 E.g. pop up info box links to detail view





15
Smooth out user/device movement (jitter)
Show POI according to orientation (compass)
Use data from location manager for positioning
Android
Maven deployment
Copyright JOANNEUM RESEARCH Forschungsgesellschaft mbH and other members of the EXPERIMEDIA consortium
Augmented Reality Viewer screenshot
16
Copyright JOANNEUM RESEARCH Forschungsgesellschaft mbH and other members of the EXPERIMEDIA consortium
PCC: The Creator
 Rapid prototyping support for game mechanics and game
structure
 Collaborative, accessible from any web browser
 Creator makes it easy to author interactive experiences in the
physical domain using a wide variety of sensors, services and
devices
 Creator acts as a content management system for required
assets and game objects. Users can edit the game in real time
and changes are propagated immediately.
 A rule engine allows the Creator to support custom game
logic. The rule engine is extendable using plugins and users
can change or update rules in real-time.
 Details: http://www.slideshare.net/kallep/pervasive-gamedevelopment-with-the-creator
17
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
PCC: Babylon
 Continuous in-game / in-experiment measurements of
subjective, qualitative QoE
 Complemented by objective, quantitative measures (data
logging), and...
 pre- and post-experiment subjective, qualitative measures
(questionnaires, interviews, focus groups).
 A server component gathers information and feedback
from participants in real-time. Data is visualized using
maps, graphs and timelines.
 Babylon supports integration into iOS and Android
mobile clients. The API is accessible over HTTP for
custom integrations.
 Babylon is integrated with the ECC.
18
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
Babylon server and client screenshots
19
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
PCC: Tracker
 Tracker is an open location-tracking server
which provides an open API for external
reports to post locations.
 Currently, both mobile clients and custom
hardware integrations exist.
20
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
3D Content Component
 3DCC supports experimenters in acquiring and
manipulating 3D information from depth
sensing devices.
 Various types of information can be acquired:
raw depth images, human skeleton, RGB
images registered to the depth images.
 3DCC comprises three different levels (low,
mid and high).
21
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
3DCC: Low level functionality
 Depth Acquisition:
 Provides the experimenter with the raw depth
information of a scene.
 Is the middle layer between the Kinect device and the
experimenter.
 Has simple and easy to use functions that provide the
experimenter with per pixel information about the depth
of the scene.
 Skeleton Acquisition:
 3DCC can provide robust skeleton extraction for up to 15
joints.
 RGB Acquisition:
 Images coming from Kinect, along with their registration
to depth pixel transformation, can be provided to the
experimenter
 Texturing of a post produced 3D model can be made
possible through this information.
22
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
3DCC: Mid level functionality
 Depth Enhancement:
 Several filtering algorithm are provided to smooth and de-noise the raw information
so that more accurate depth measurement can be made possible.
 Skeleton Enhancement:
 Jerky (noisy) skeleton joints are detected and tracked and therefore corrected through
a sophisticated tailored filtering framework to provide a more realistic skeleton.
 Biomechanical Measurements:
 Biomechanical measurements that are inferred from both depth and skeleton
information as angles between bones, human joints and calibrated objects’ velocities,
human body parts surface areas and calibrated objects’ surface area.
23
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
3DCC: High level functionality
 Avatar Creation:
 The 3DCC provides an avatar authoring tool so
that experimenters can create their own avatars
that can be easily integrated into a virtual world.
 Avatar Motion:
 The 3DCC can also
interactively move the
avatar using a Kinect alone.
24
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013
Thank you!
Key
Extension point
Extension point
Content interface
Configuration
Content Interface
ECC Interface
Sub-component
Monitoring
ECC Interface
Security
ECC Interface
Sub-component
Component name
26
Copyright University of Southampton IT Innovation Centre and other members of the EXPERIMEDIA consortium, 2012
Live streaming
Video Codec
H.264
VP6
SorensonSpark
Screen Share
Player/Device
Adobe Flash player
iOS based device
QuickTime player
Microsoft Silverlight
VideoLAN VLC player
Set-top boxes
3GPP devices
Adobe Flash player
Adobe Flash player
Adobe Flash player
Audio Codec
AAC
AAC LC, HE-AAC
MP3
Speex
Nelly Moser ASAO
Player/Device
Adobe Flash player
iOS based device
QuickTime player
Microsoft Silverlight
VideoLAN VLC player
Set-top boxes
3GPP devices
27
Player/Device
Adobe Flash player
iOS based device
QuickTime player
Microsoft Silverlight
VideoLAN VLC player
Set-top boxes
3GPP devices
Adobe Flash player
iOS based device
QuickTime player
Microsoft Silverlight
Adobe Flash player
iOS based device
QuickTime player
Microsoft Silverlight
Adobe Flash player
Adobe Flash player
Protocol (OUTPUT)
RTMP/RTMPE/RTMPT/RTMPTE/RTMPS
Flash HDS
Apple HLS
Apple HLS
Microsoft Smooth Streaming
RTSP/RTP
MPEG2-TS
RTSP/RTP
MPEG2-TS
RTSP/RTP
MPEG2-TS
AVCC modules
UGC
 Input Manager: manages the reception
of all live content
 Live Metadata Manager: manages the
reception of metadata and time stamps
 MoreVideo
 Video Analysis
 Device Adaptation Manager: adapts the
media content to different platforms
 Multi Quality Manager: coordinates the
mutiquality content
 Timeshift Manager: continuously records live streams for immediate playback on
deferred
 VoD Manager: records live content and delivers pre-recorded video and audio streams
 Ingest Manager: stores new content from a media file
 Media Distribution: produces all multiplexes media output of the distribution
28
VoD ingest process
AV content
upload
Content
description and
adaptation
instructions
Content
adaptation
Codec accepted
Output profiles supported*
Distribution
video/avi
video/mp4
Flash / Flex / HTML5 / Mobile native applications
video/mp4
video/ogg
PC / Android / iOS, etc…
video/webm
video/webm
video/ogg
audio/mpeg (mp3)
Progresive download, RTMP, HDS, HLS, Smooth
Streaming, RTSP/RTP, MPEG2-TS
video/quicktime
video/x-flv (adaptive)
Img/JPEG (thumbnail)
29
Content
publication
&
distribution
Content
description
and access
URLs
forward to
Exp. CMS
* The actual profiles can be optimised for each
experiment
AVCC Monitoring
 Focus on
Monitoring
 Easy to
deploy java
software Clients
AVCC Server
Client Count
I/O Bytes Total
I/O Bytes Rate
Publisher Count
Stream Count
Connection Count per Stream
Type
I/O Bytes Total per Stream
Type
I/O Bytes Rate per Stream Type
I/O Bytes Total per Stream
I/O Bytes Rate per Stream
30
AVCC
Monitoring control
signals
AVCC EM
Interface
RabbitMQ
Server
Playback client
Audio, Video and Data
Bitrate ([audio,video,data]
Audio, Video and Data Buffer
Length (bytes and seconds)
A/V Data Input Bitrate
Playback Rate
Dropped Frames
Network capacity
AVCC Server RTT (SRTT)
Last Metadata Received
Experiment
data
Publish client
FPS (from camera)
Publish/Playing time
A/V Activity Level (0-100)
Video Resolution
Key Frame Interval
ECC
AR Integration
 For Android Apps
 Android 2.3+
 Devices with camera
 Necessary permissions:
 Camera access
 Data retrieval from Internet
 If used: GPS, cell, wifi information for localization
 Maven deployment
31
Copyright JOANNEUM RESEARCH Forschungsgesellschaft mbH and other members of the EXPERIMEDIA consortium
Creator screenshot
32
Copyright ICCS/NTUA and other members of the EXPERIMEDIA consortium, 2013