Transcript API’s for Digital Media

UI & Gaming
in Mobile Handsets
Yoshio Kuniyoshi
Vice President & CTO Acrodea Inc.
© Copyright Khronos Group, 2006 - Page 1
Acrodea Group Presentation
Corporate Profile:
Business Locations
Date of Establishment
Capital
End of Fiscal Year
Net Sales
Number of Employees
Overseas branch
5F, HAGIWARA BLDG.1,
1-3-1 Hiroo, Shibuya-ku,
Tokyo, 150-0012 Japan
TEL: +81-3-5798-4885, FAX: +81-3-5798-4886
July 12, 2004
591.20 million yen (As of June 30, 2006)
March 31
1,077.2 million yen (April 1, 2005 - March 31, 2006)
49 (As of As of June 30, 2006, excluding Directors)
Acrodea, Inc. Korea Branch
7F, Joong-A Building, 157-12,
Samseong-dong, Gangnam-gu,
Seoul, Korea, 135-880
TEL: +82-2-553-4955, FAX: +82-2-553-4979
Acrodea, Inc. Europe Branch
Keilasatama 3 Espoo, 02150 FINLAND
TEL: +358-9-2510-7000, FAX: +358-9-2510-7100
Acrodea, Inc. Beijing Representative Office
#1516 Block B Lucky Tower,
No.3 Dongsanhuan Beilu, Chaoyang District,
Beijing 100027, P.R.China
TEL: +86-10-6465-0568, FAX: +86-10-6465-6338
Major Customers
Samsung Electronics Co., LTD.
Gemini Mobile Technologies, KK
NTT DoCoMo, Inc.
SANYO Electric Company
InnoPath Software
Correspondent Bank
Mizuho Bank, Ltd.
Bank of Tokyo-Mitsubishi UFJ
Outline of the Presentation
• Background
- Recently, many handsets become equipped with GPU. However the purpose of GPU is mainly
for Gaming only. On the other hand, user interface is coming to need more and more graphics
processing power.
• VIVID UI
- As a state of the art UI platform for mobile devices, the features of VIVID UI is introduced to
describe how today’s UI platform for mobile handsets needs graphics power.
- Describe how current VIVID UI uses GPU.
• Roadmap for Accelerating User Interface
- Describe the ideal way of accelerating user interface.
- Describe problems of current GPU and its API implementation.
• Gaming
- As a state of the art Game platform for mobile devices, the features of X-Forge is described.
- Describe programmable shader support for X-Forge.
© Copyright Khronos Group, 2006 - Page 3
Introduction
• UI and Gaming trends in Mobile Handsets
- More and More Graphics Processing Power is Required
- Display Resolution Gets Higher
QCIF → QVGA → WQVGA → VGA
QCIF : VGA = 1 : 12
- Demands for Richer Media Support
Flash, SVG, PDF, 3D, Movie(MP4 → H.264 quality)
- Demands for Advanced 3D Technologies
3D Effects makes user interface become more attractive
Programmable shaders makes more realistic 3D games
- Recently, many handsets become equipped with GPU.
- Example
- UI Platform for PC
- MacOS X Quartz Extreme
- Windows Vista Avalon
- Handset
- 3D Menu (SHARP V602SH)
- VGA Display (SHARP 904SH)
- Flash Lite 1.1 and 2.0 (Nokia, Sony Ericsson, and Most of Japanese Phones for
au/DoCoMo/Vodafone)
- GPU Processors
NVIDIA GoForce, IMG PowerVR MBX, TOSHIBA T5G, ATI Imageon, …
All these GPUs will soon/already support programmable shaders and OpenVG.
© Copyright Khronos Group, 2006 - Page 4
Introduction
Next-generation UI architecture framework:
UI architecture based on scripting and OTA differential updating
© Copyright Khronos Group, 2006 - Page 5
VIVID UI Features
• Scripting Based UI Platform --UI is Data not Code
Easy to create arbitrary UI structure with rich visual expression through
scripting. No need to write native codes. This “UI is Data” concept enables
easy provisioning of UI content service by OTA.
• Development with GUI Authoring Tools
GUI based WYSIWYG operations result in automatic script generation.
• Separate Management of UI from Logic
Look and feel, and application logic managed separately.
Improves collaboration between designers and programmers.
• Provides of Standard UI Components
Provides buttons, labels, checkboxes, radioboxes, textfields and so on.
Other components can be easily added.
• Enables Any Kind of Media Composition
Designed as a container for any kind of graphics renderer. Various
materials including 2D images, Flash, 3D, and Movies can be freely
composed by alpha blending.
© Copyright Khronos Group, 2006 - Page 6
VIVID UI Requirements
Complete UI solution for today and tomorrows devices
•Supported platforms
– OS
– CPU
– TOOL
:
:
:
Brew / Linux / Symbian / RTOS
ARM7 30MHz or later
Windows / Macintosh (future plan)
• ROM / RAM size
– ROM
– RAM
: between 100KB and 500KB
(depends on components)
: between 200KB and more
(depends on contents)
© Copyright Khronos Group, 2006 - Page 7
VIVID UI Contents Development and
Distribution Flow
UI Concept
Planning
UI Materials
design
Authoring
with tools
VividUI Workshop
Terminal
embedding
Packaging
OTA delivery
Package all data such as XML
scripts (compiled form), and
resource files into one file
Placed in VividUI
distribution server
Handset with
VividUI engine
© Copyright Khronos Group, 2006 - Page 8
Achieving Diverse UI
Instead of embedding a fully designed UI, loading a UI platform
environment is a much more desirable direction for handset
manufacturers. Then subscribers can select their desired UI from among
a variety of options to suit their preferences.
By responding to diversified needs, a whole new “UI epoch” will begin.
© Copyright Khronos Group, 2006 - Page 9
VIVID UI Multi Media Support
2D, Flash, 3D, and Movies
3D Animated Menu
Icons
Flash Animation
2D Images
• VIVID UI can integrates any kind of
graphics renderer on a handset
including 2D images, Flash, 3D, and
Movies.
• Various materials including 2D images,
Flash, 3D, and Movies can be freely
composed by alpha blending.
• VIVID UI has built-in 2D renderer, 3D
scene graph engine based on
OpenGL|ES, and interface to Flash and
several movie decoders.
2D Menu on top of
movie.
© Copyright Khronos Group, 2006 - Page 10
VIVID UI Workshop
UI Design Tool
© Copyright Khronos Group, 2006 - Page 11
• VIVID UI Workshop is an authoring
tool to facilitate user interface
development with VIVID UI.
• This GUI tool enables WYSIWYG
based editing of UI structure. UI
designers can create UI by just
putting components on the design
pane of the tool then setting several
properties for each component and
the tool automatically generates
corresponding XML scripts on the
fly.
• UI designer can verify the behavior
and appearance of the UI content
with built-in emulator.
• Handset profile feature provides
easy adoption of different handset
models.
• Integrates all operations needed for
UI content creation.
Client Architecture
VU
app
VUL interpreter
VividUI API
Key events and
handset UI functions
such as screen
drawing are
accessible through
scripting also.
VividUI engine
Native Abstraction Layer
Handset
Platform
API
Flash
3D
OES
© Copyright Khronos Group, 2006 - Page 12
2D
SVG-T
Video
AMC
Audio
MP3
App
launcher
Yellow frame
indicates VIVID UI
Platform
Plug-in API
VU
app
Skins can be open
to the end users,
who can freely
customize
appearances.
User Customizable Skin
The abstraction layer
for various native
functions.
Folder Menu Script
VU API can be used
directly for UI app
implementation.
Camera Menu Script
Top Menu Script
VUL scripts are interpreted
and then drive the VividUI
engine.
Setting Menu Script
Plug-ins provide extensibility,
and are accessible through
scripting
Handset menus
developed by VUL
scripting
VU
plug-in
VU
plug-in
Scripts can launch
various embedded
programs.
UI Scripting Architecture
VUL* script 3 components
Contents
Structural scripts such as menus
Action scripts
Skins
• UI architecture through scripting
- VividUI builds a resource file UI composed of
XML files, simple scripting language, and
images.
• Separated UI components
- UI components are written as
Contents: UI structure
Skin: UI appearance
Locale: UI localization
Style scripts
Sound and image resources
• Modular structure
Locales
• Presentation script feature
Localized language data
*VividUI Scripting Language.
© Copyright Khronos Group, 2006 - Page 13
- Overlaying feature enables UI sharing and
componentizing
- Style sheet definitions enable pixel-level
precise control.
Embedded Advanced
Functions
• Embedded animation effect
- Can be used to impart attractive motion on
the UI.
• Push and pull communications
News and
real-time stock
prices through
“push” comm.
- Adapts to server “push” communications for
timely data delivery to handsets.
Innovative UI
possible with 3D
display
Animation
generates
motion effect
• Abstraction with rendering engines
- 3D/Movie/Flash can be accessed in concert
with native rendering engine by scripting.
• Abstraction with sound functions
VividUI Engine
- MP3 and other native sound functions can
be accessed by scripting.
• Abstraction with status displays
Unique design
and placement
of status
indicators
Introduction of
realistic sound
© Copyright Khronos Group, 2006 - Page 14
- Allows acquisition of status through
scripting (signal level, battery life)
VIVIDUI Other Features
• Support multi-task environment
- re-entrant thread safe architecture / passive library
• Effective compression for data
- LZ and Huffman based compression / optimization for re-use of resource files
• Lightweight alpha blending technology
- optimized for mobile environment / anti-aliasing feature
• Easy to expand UI components
- support 2D vector graphics (FLASH) / 3D graphics (OpenGL-ES)
- component plug-in API enables to add other UI components
- renderer plug-in API enables to support other graphics rendering engines
• Support hardware acceleration
- can be optimized for CPU – ARM9/ARM11 / HITACHI SH-Mobile
- can be optimized for GPU – OpenGL-ES compatible HW
• Easy to port
- simple porting layer (30 – 70 functions: depends on the requirements)
- customizable porting layer
• Support OTA differential update
- can be optimized for OTA differential technology
- can share porting layer with OTA differential technology
© Copyright Khronos Group, 2006 - Page 15
VIVID UI OES Implementation
• Current VIVID UI OpenGL ES Implementation
-
Supports OpenGL|ES 1.0 and 1.1
Full OpenGL|ES acceleration for built-in 2D renderer
Full OpenGL|ES acceleration for 3D scene graph engine
Movie is supported by software decoder
Flash Lite is supported (w/ Macromedia Flash Lite library)
© Copyright Khronos Group, 2006 - Page 16
VIVID UI Acceleration Roadmap
• The Ideal way of VIVID UI Accelerated Implementation
-
Supports OpenGL|ES 1.0 and higher, and/or OpenVG
Full OpenGL|ES and/or OpenVG acceleration for built-in 2D renderer
Full OpenGL|ES acceleration for 3D scene graph engine
Movie is supported by hardware decoder w/ OpenMAX
Accelerated version of Flash Lite is supported
• Problems of current GPU and its API implementation
- Requirements
- Multi context support with independent surface for multi processes.
- Multi context support with shared surface for single processes(single/multi threads).
- Shared surface support between OpenGL|ES, OpenVG, and OpenMAX.
- Resource management
- Fine grained power management
- Current GPU and API Implementation
- Some environments does not support multi context.
- Some environments only support multi context with independent surface, but no support for shared
surface.
- Some environments does not support resource management. E.g.: The upper limit of the texture area is
the same as GPU hardware limitation.
- No/Rare existing support for OpenMAX and OpenVG w/ shared surface support w/ OpenGL|ES.
- No support for fine grained power management from the application.
© Copyright Khronos Group, 2006 - Page 17
X-Forge - General Presentation
© Copyright Khronos Group, 2006 - Page 18
The X-Forge™ Tool Chain
1. 2D and 3D Content
CODE
2. Particle Effects
Executable
(EXE)
PC Emulator Version
CONTENT
Compressed File Library (CFL)
3. Music and Sound Effects
4. Scripts
Native Device Version
© Copyright Khronos Group, 2006 - Page 19
The X-Forge™ SDK
Your Game Content
Graphics
Audio
Scripts
Your Game Code
XForge™
Game
Engine
Game Graph UI Toolkit
Particle
Physics
framework FX system
Collisions
AI Framework
3D Audio
X-Forge Online
Application
framework
Game
Specific
Code
OS and Device Abstraction
X-Forge™
Core
Audio
Networking
Framework
Graphics
Controls
File I/O
Resources
UTF8 unicode
XML
SVG
Localization
tools
Debugging
tools
Mobile
Device
OS
Device Driver
Hardwar
e
OpenGL ES 1.x/2
CPU
3D Graphics Accelerator
© Copyright Khronos Group, 2006 - Page 20
Additional X-Forge™
Components
Documentation
Stand-alone Tools
1000+ page API
reference
Programmer
Tools
Artist
Tools
Tutorials & sample
code
Packager
Profile
Analyzer
3ds Max™
Exporter
Maya™
Exporter
Particle FX
Editor
Production & tools
Optimization
guidelines
Multi-platform Support
Acrodea works in close co-operation with all leading mobile technology companies in order to
support a wide variety of different mobile operating systems, processors, 3D graphics
accelerators and devices in X-Forge.
•
Symbian OS™ / Nokia Series 60
-
•
Symbian OS™ / UIQ
-
•
•
Microsoft Windows® CE.NET
-
•
Microsoft Smartphone
-
•
AlphaCell M5
Orange SPV™
Microsoft Pocket PC
-
Sony Ericsson P800
ARM-based devices
Palm OS (V5.x)
-
•
Nokia N-Gage™
Nokia 3650, 7650, 6600…
Siemens SX1
Samsung SGH-D700
Tapwave Zodiac™
Mobile Linux
-
X-Forge can also be ported to virtually any
additional operating system and hardware.
Sharp Zaurus™
• X-Forge is Nokia OK certified and optimized for the Nokia N-Gage™ game deck
• X-Forge has been optimized for leading processor architectures including Intel XScale™ / StrongARM™ and Texas Instruments OMAP™
• X-Forge supports various 2D and 3D graphics accelerators, including products from ATI, nVIDIA and Intel
• X-Forge is compatible with standard 3D rendering engines such as OpenGL ES™
© Copyright Khronos Group, 2006 - Page 21
Existing X-Forge™ Game Content
X-Forge™ Delivers Superior Quality
Red Faction™
Spy Hunter™
Resistance™
Stuntcar Extreme™
Super Monkey Ball™
Virtua Tennis™
Galactic Realms™
Gadarka’s Delusion
Geopod™
Men in Black II™: Alien Pursuit
MotoGP™
Stuntcar Extreme™
Legend of Zord™
Spy Hunter™
© Copyright Khronos Group, 2006 - Page 22
Gadarka’s Delusion
Geopod™
Unfinished Business™
Unfinished Business™
Firehammer™
Resistance™
Snowboarding Extreme
Red Faction™
Phantom Strike™
Snowboarding Extreme
Gadarka’s Delusion
Stuntcar Extreme™
Developer Network
This is a partial list of developers who have licensed / evaluated X-Forge™.
© Copyright Khronos Group, 2006 - Page 23
X-Forge Programmable Shader Support
• OES Version
- Support OpenGL|ES 2.0.
• Effects
- Supports “Effect” which consists of a shader, and accompanying data and rendering paths.
(i.e.: FX file of DirectX)
• Built-in Effects
- Some effects are built-in (or pre-installed) with X-Forge SDK. Effects for glow, normal map, environment
map, bump map will be prepared.
• Treat Effect as a part of a scene graph
- Assign effects to each model or the entire scene and then export.
- These effects will be automatically processed at runtime.
• Effect Editing Tool
- Effect editor will be provided.
• Support High Level API
- Provides high level API to manipulate effects from C++ source code.
© Copyright Khronos Group, 2006 - Page 24