Transcript IMSC mission

Integrated Media Systems Center
Ulrich Neumann, Director
Charles Lee Powell Professor of Engineering
Associate Professor, Computer Science
Isaac Maya, Ph.D., P.E.
Director, Industry and Technology Transfer Programs
IMSC’s Immersive Technologies
National Science Foundation Engineering Research Center
Integrated Media
Systems Center
a partnership among:
National Science Foundation
University of Southern California
School of Engineering
Annenberg Center for Communication
State of California
The City of Los Angeles
Industry Partners:
Computer Hardware and Software
Entertainment
Broadcasting
Telecommunications
Publishing
Aerospace
NSF
Downtown LA
Other Government Agencies:
DARPA, NASA, JPL, ONR, US Army
$10 M operating budget
IMSC Headquarters
National Science Foundation Engineering Research Center
Faculty and Academia







A top-notch array of investigators – 14 of whom (~ 50%) were sought specifically
because of IMSC

29 investigators working with 87 Phd, 26 MS, and 33 UG students
Investigators come primarily from EE and CS – others from Psychology, Industrial and
System Eng., School of Cinema/Television, Annenberg School for Comm., School of
Gerontology, Biomedical Engineering, and the Information Sciences Institute
Two IMSC investigators have PYI awards
Seven IMSC faculty have CAREER awards (3 CAREER awards this year)
Tomlinson Holman won a 2001 Technology Achievement Award from the Academy of
Motion Pictures
Robert Sholtz won a MILCOM Achievement Award for UWB research (only 4th in 20 yrs)
IMSC faculty published 53 peer-reviewed journal articles and 205 peer-reviewed
conference papers

articles appearing in print over a 12 month period in this past year (2001-2002)
National Science Foundation Engineering Research Center
Education
Programs created by IMSC

171 students graduated with IMSC providing funding, classes, and
research aspects of their education experience


81 with PHD, 77 with MS, and 13 with BS
IMSC created 3 MS programs enrolling 358 students

unique among U.S. universities

IMSC created 2 UG minor programs enrolling 207 students

IMSC gave research fellowships to 32 UG students

Created more than 20 courses for programs – e.g.:



Human Factors in Integrated Media Systems
Integrated Media Systems and Architectures
School of Engineering/Fine Arts (joint project course)
National Science Foundation Engineering Research Center
Industry Program

IMSC offers cost-effective research partnership at the 5-10 yr horizon

Pursue science and discovery of methods and algorithms – IP licenses

Develop technology-impact and proof-of-concept before “start up”

Prototype feasibility applications and testbeds

Provide partners a view to potential opportunities and threats

Strong support through six years

28 members this past year (2001-2002)
Recognized 5 members for consistent partnership over 6 yr life of center

HP, NCR, FXPAL, TRW, Lockheed Martin


Strong growth in international corporations and government programs

Taiwan (III, ITRI)

Korea (LG Electronics)

Japan (FXPAL, Hitachi, NTT DoCoMo, Toshiba)

Netherlands (Philips Research)

Italy (ST Microelectronics)

Patents and tech transfer

46 patents filed over the life of center

24 patents licensed and 39 software licenses
National Science Foundation Engineering Research Center
IMSC Research Members
Japan
NTT DoCoMo
III
ITRI
Taiwan
Microsoft
Intel
WA
NavTech
IL
Eastman Kodak
NY
DRS Technologies
Prentice Hall PTR
NJ
OR
See California Map
Collaborations with
Government Agencies
and Foundations:
NCR
Time Domain
US Army
DARPA
NASA
Toyota Foundation
National Science Foundation Engineering Research Center
OH
AL
IMSC Research Members
Los Angeles/San Diego
• AboveNet Communications
• Boeing
• HRL
• JPL
• Los Angeles Times
• NMUA
• Panoram Technologies
• TMH Corporation
• TRW
Silicon Valley
•ConceptLabs
•F-X Palo Alto Laboratory
•Geometrix
•Hewlett-Packard
•InterVideo
•Lockheed Martin
•Sun Microsystems
Silicon Valley
Hollywood
USC
IMSC
Los Angeles
National Science Foundation Engineering Research Center
IMSC Research Program -- Immersipresence
Motivated by Moore’s law: exponential performance growth...
enables advances towards the future of the internet, grid computing, nano-technology, ...
National Science Foundation Engineering Research Center
People are often the ultimate
purpose behind silicon systems
In pursuit of an experience, or
the performance of a task, or
understanding of information
Integrated Media System (IMS)
mediate the gap between silicon
and humans
An extension of the human senses,
actuators, or cognitive process
Our goal is to advance the science and engineering of “well designed” Integrated Media Systems that exploit human
information processing capabilities - providing effectiveness that we associate with “immersipresense”.
National Science Foundation Engineering Research Center
To better understand
immersipresence and help set
specific goals, we develop
visions of how IMS
technologies will affect people
and their activities
We examine four domains for IMS
impacts in the future
These four vision scenarios drive
the center’s program of six
research and engineering areas
National Science Foundation Engineering Research Center
UWB console captures 3D model
and haptics scan
PDA provides spatial array of
participants and sonic
translation of haptic trace
Immersive
workstation
coordinates
video,
avatars,
graphics,
audio, and
haptics to
facilitate
problem
resolution
Theory of Perceptual
and Cognitive
Pleasure
Immersive
Audio
User State Sensing
and Perceptual User
Interfaces
Facial
Gesture
Analysis and
Animation
Virtual Environments
for Performance
Testing and Training
Robust
Vision
Analysis
Immersidata
Content
Extraction &
Analysis
Multimedia
Networks,
Transmission,
and Communication
Ultra
Wideband
Wireless
Remote Media
Immersion
Software
Architecture for
Immersipresence
HAPTICS – devices
and translations
National Science Foundation Engineering Research Center
Immersive on-demand
streaming of audio/video
experiences - fusion of cinema
and internet
Virtual/video peer-to-peer
telepresence experiences for
multiple participants
Full body immersion
role-playing game
experiences with
realistic or synthetic
characters
Emotional Expressions
User-State Sensing
and Perceptual User
Interfaces
Immersive
Audio
Speech
Recognition
and
Synthesis
Facial
Gesture
Analysis and
Animation
Immersive
Media RealTime
Storage &
Retrieval
Multimedia
Networks,
Transmission,
and Communication
Ultra
Wideband
Wireless
Remote Media
Immersion
Software
Architecture for
Immersipresence
Remote Media
Immersion - fusion
of internet and
cinema
Robust
Vision
Analysis
Digital
Geometry
Processing
National Science Foundation Engineering Research Center
IMSC research program
Theory of Perceptual
and Cognitive
Pleasure
Emotional Expressions
User State Sensing
and Perceptual User
Interfaces
Virtual Environments
for Performance
Testing and Training
Immersive
audio
Information
Integration
Speech
Recognition
and
Synthesis
Immersidata
Content
Extraction &
Analysis
Facial
Gesture
Analysis
and
Animation
Robust
Vision
Analysis
Digital
Geometry
Processing
Customized
Querying &
Rendering
Multimedia
Networks,
Transmission,
and Communication
Ultra
Wideband
Wireless
Compression
Remote Media
Immersion
RMI - fusion of
internet and cinema
Software
Architecture for
Immersipresence
BioSIGHT - authoring
and assessment
I-NEWS - story
models and user
preferences
HAPTICS - devices
and translations
Immersive
Media RealTime
Storage &
Retrieval
National Science Foundation Engineering Research Center
Research Highlights
IMSC has produced ground breaking results and fundamental
research in:

immersive audio


streaming servers


computational framework for grouping based on tensor voting
graphics & animation


theoretical foundations for understanding benefits of media immersion
computer vision


leadership in technology, FCC regulation, and commercialization
perception & cognitive modeling


distributed and scalable architecture
UWB wireless


multichannel and HRTF approaches - holistic DSP approach
3D DSP mesh processing, facial expression analysis and animation, hair modeling
virtual reality -- chair of VR2003 in LA

applications to psychology (ADD diagnosis), haptics applications, and user studies
National Science Foundation Engineering Research Center
IMSC Immersipresence Vision:
Applications in Entertainment
USC Homecoming Game at the Los Angeles Coliseum
QuickTime™ and a
Cinepak decompressor
Animation
Here:
are needed
to see File
this Goes
picture.
PC – panoramic_animation.avi
MAC - panoramic_animation.mov
360º Camera
Panoramic Visual Immersion: Malibu Hills
National Science Foundation Engineering Research Center
IMSC Immersipresence Vision:
Medical Applications
Animation File Goes Here:
PC –
200_0003.AVI
QuickTime™ and a
Microsof t Video 1 decompressor
are needed to see this picture.
Evaluation of ADHD in Children
National Science Foundation Engineering Research Center
Animation and Modeling
of People for Immersipresence
Avatar: Digital Cloning
Video Files
Mac: face_1.mov
QuickTime™ and a
PC: Cinepak
avatars_doug.avi
decompressor
are needed to see this picture.
3D Visual Environment
Avatar simulates live video
National Science Foundation Engineering Research Center
Animation and Modeling:
Expression Cloning
QuickTime™ and a
None decompressor
are needed to see this picture.
QuickTime™ and a
None decompressor
are needed to see this picture.
Real Expression can be “cloned” to any model
National Science Foundation Engineering Research Center
Animation and Modeling:
Expression Analysis
QuickTime™ and a
None decompressor
are needed to see this picture.
Video Files
Mac: face_1.mov
PC: avatars_doug.avi
Real Expression can be “cloned” and analyzed
National Science Foundation Engineering Research Center
Digital Geometry Processing:
3D Modeling and Elasticity
Animation File Goes Here:
PC – QuickTime™
mat_1.avi and a
Cinepak decompressor
MAC
- mat_1.mov
are
needed
to see this picture.
Animation File Goes Here:
PC – mat_3.avi
MAC - QuickTime™
mat_3.mov and a
Cinepak decompressor
are needed to see this picture.

Animation File Goes Here:
QuickTime™
and a
PC
– mat_2.avi
Cinepak decompressor
MAC to
- mat_2.mov
are needed
see this picture.
Next wave of visual data: Better
understanding and manipulation
of geometric data
National Science Foundation Engineering Research Center
Mixed Reality and Visualization



Augmented Reality for Space Flight (NASA funded)
 Develop AR authoring tools for video-based training
 Anthony Majoros, Human Factors group, The Boeing
Corp.
4D Battlefield Visualization (MURI funded)
 Develop fusion of video, images, and 3D models
 Avideh Zakhor (Berkeley), Suresh Lodha (UC Santa
Cruz), Bill Ribarsky (Georgia Tech), Pramod Varshney
(Syracuse)
Wide Area AR Tracking (ONR funded)
 Novel sensors and fusion for tracking
position/orientation
National Science Foundation Engineering Research Center
Recognizing Emotions in Speech


Natural Human-computer interaction:

need for machines that can recognize “what, who and how”

Call center applications, Teleconferencing, Tutoring by machines,
Entertainment
Algorithms (ASRU’01, ICME’02, ICSLP’02)

Support Vector Machine, Linear discriminant and k-NN classifier for
acoustic information

Novel information-theoretic measure of emotional
salience in language (word sequences)

Acoustic and linguistic information fusion

Performance evaluated on real call center data
rather than feigned data from actors
Non-negative
Negative
(From the same person and dialog)
National Science Foundation Engineering Research Center
Immersive Audio 10.2
QuickTime™ and a
Cinepak decompressor
Video Files
are needed to see this picture.
Mac: audio_10.2_channel.mov
PC: audio_10.2_channel.avi
Immersive Audio Environment
3D Virtual audio field at the
location of the ears at all times
National Science Foundation Engineering Research Center
Haptics Research
QuickTime™ and a
Cinepak decompressor
are needed to see this picture.
Movie File:
PC: haptic_1.avi
MAC: haptic_1.mov
CyberGrasp





QuickTime™ and a
Cinepak decompressor
are needed to see this picture.
Movie File:
PC: haptic_2.avi
MAC: haptic_2.mov
PHANToM (3D Mouse)
Force control for haptics rendering
Collision detection
Compression of haptic data
Haptic playback and analysis
Application Research Project: Haptic Museum
National Science Foundation Engineering Research Center
Haptics Experiment
Mutual Touch
INTERNET
PHANToM
CyberGrasp



Virtual “touching” through the internet
Virtual “handshaking”
Distributed Haptic World
National Science Foundation Engineering Research Center
Media Communications
High Bandwidth Networks: synchronization, latency, bandwidth
Wireless Systems
Optical Communication
Compression
Encryption and Network Security
Photonic Networks
QoS Management of IP Networks
National Science Foundation Engineering Research Center
Managing Immersidata
Avatar
Haptic
Application
Control
Video
Immersidata
3D Model Immersive Audio



Text
Real-time acquisition and playback
Efficient query and analysis
Integration of disparate information sources
National Science Foundation Engineering Research Center
“User Centered Sciences” Research
Integrates Multiple Methodologies:








Human Factors
Human Computer Interaction
General Experimental Psychology
Usability Engineering
Communication
User-Centered Design
Cognitive Science
Ethnography
With the aim to improve:




Effectiveness
Efficiency
Enjoyment
Safety
National Science Foundation Engineering Research Center
Systems Engineering
Similarly, unique systems engineering and integration are
hallmarks of IMSC
RMI testbed for streaming media immersion
Showcase for IMSC research and integration
May 2002 demonstration (NY Times)
Flow Scheduling Framework of the Media Immersion
Environment - open source release
BioSIGHT authoring tools and web-based education
Testing and assessment in K-12 classrooms
National Science Foundation Engineering Research Center
Applications
RMI

A fusion of internet browsing with a theater-like immersive
experience



HD Video at up to 45 Mbits/sec
10.2 ch Immersive audio (16 Mbits/sec)
Steaming over the Internet on-demand to a mouse click
(20 min prog)

Immersed in a college
football game
Doctors assisting in a
remote procedure
RMI demonstrates IMSC innovations in several critical
technical areas:
Streaming
media
servers
Immersive
audio
capture
and
rendering
Network
protocols
for error
correction
Business people
negotiating like they
are in the same room
Synchronization
Students visiting an
aquarium a thousand
miles away
National Science Foundation Engineering Research Center
RMI Challenges

Immersive, high-quality video and
audio acquisition and rendering
 High Definition video 1080i and
720p (~40 Mb/s)
 10.2 channels of uncompressed
audio (12 Mb/s)

Storage and transmission of media
streams across networks
 Streaming media servers/clients
 Error correction via selective
retransmission
 Synchronization between
streams
 A/V, A/A, V/V
National Science Foundation Engineering Research Center
RMI Architecture
National Science Foundation Engineering Research Center
Storage, Streaming & Rendering

Focus:
End-to-end
streaming
architecture



Requirements:
A streaming
platform that can
scale and handle
synchronized, highbandwidth streams

Distributed Server
 Storage
 Scheduling
 Scalability
Clients
 Multi-stream
Synchronized
playback
Transmission
 Robust VBR
Flow control
[ACM Computer’02]
National Science Foundation Engineering Research Center
Scalability: Multi-Node, Multi-Disk

Data and control network traffic can be routed with
different logical topologies
 Yima-1: single data path
(high inter-node traffic)
 Yima-2: multiple data paths
(low inter-node traffic)
Yima-1
Yima-2
National Science Foundation Engineering Research Center
Robust Stream Delivery

Variable bit rate (VBR) media encoders allocate more bits
to complex scenes and less bits to simple ones

Smoothing of VBR media traffic has the following quality
benefits:
 Better resource utilization (less bursty)
 More streams with the same network capacity

Multi-Threshold Flow Control
(MTFC) algorithm objectives:
 Online operation
 Content independence
 Minimizing feedback
control signaling
 Rate smoothing
National Science Foundation Engineering Research Center
Premier
Demonstration May 9,
2002





Cross-country streaming from
ISI-East in Virginia
NY Times coverage in “Circuits”
section
NBC-TV and KTLA
http://imsc.usc.edu/rmi/
http://www.east.isi.edu/NGI-S/
National Science Foundation Engineering Research Center
Future Impact of RMI


High-quality audio/video experience streamed ondemand over the internet (live or recorded)
Everyone is a content producer, distributor, theater
 Currently only broadcast, cable, theater, or satellite
operators can distribute content…



high entry cost and regulation barriers
RMI technology removes distribution barriers
 CEO NVidia: “the end of TV as we know it”
 LIVE concerts, sporting events, alternative news
coverage, …
 RECORDED movie archives, news archives,
education programs, …
“More than TV” experience – powerful human motivation
 Playback quality and options are flexible – retain all
the capabilities of computing on the internet
National Science Foundation Engineering Research Center
Distributed Immersive Performance




Next generation of Remote Media Immersion (RMI)
Vision – interactive peer-to-peer streaming
 Create seamless immersive environment for
distributed high-quality audio/video interaction
Impact – widespread telepresence
 Distributed performances, meetings, social
gatherings, education,
Technical barriers
 Precise timing and synchronization of streams
 Low latency, error correction – high quality-of-service
network protocols
National Science Foundation Engineering Research Center
National Science Foundation Engineering Research Center
Summary


IMSC’s vision of Immersipresence and Integrated Media Systems aims
for fundamental breakthroughs in technology, industry partnerships and
education
IMSC is carrying out




Highly cross-disciplinary cutting edge research
Aggressive industry collaboration and technology transfer
Unique educational experience and teamwork skills for
students
IMSC is eager to expand its Collaborations
National Science Foundation Engineering Research Center
Acknowledgments
National Science Foundation
Engineering Research Center program
http://www.eng.nsf.gov/eec/erc.htm
IMSC Industry partners.....
More information about IMSC
http/:imsc.usc.edu
IMSC PARTNERS
Airborne 1 Corporation
Compression Science Corporation
ConceptLabs
Eastman Kodak
F-X Palo Alto Laboratory
Geometrix
Hewlett-Packard
HRL
IBM
Informix
Institute for Information Industry
Intel
InterVideo
JPL
Litton
Lockheed Martin
Los Angeles Times
Lucent Technologies/Bell Labs
NCR
Panoram Technologies
Philips Multimedia Center
Prentice Hall PTR
Raytheon
STMicroelectronics
Sun Microsystems
TMH Corporation
TRW