Transcript QNX Corporate Sales Presentation

Presented by:
“Embedded Operating Systems:
The State of the Art”
Jeff Schaffer
Sr. Field Applications Engineer
QNX Software Systems
[email protected]
818-227-5105
QNX is a leading provider of real time
operating system (RTOS) software,
development tools, and services for
mission critical embedded applications.
1
Role of the Embedded OS
 Traditional
– Permit sharing of common resources of the computer
(disks, printers, CPU)
– Provide low-level control of I/O devices that may be
complex, time dependent, and non-portable
– Provide device-independent abstractions (e.g. files,
filenames, directories)
 Additional Roles
– Prevent common causes of system failure and
instability; minimize impact when they occur
– Extend system life cycles
– Isolate problems during development and at runtime
2
Architecture Comparison
REAL TIME EXECUTIVE
Advantage: single address space
Disadvantage: single address space,
different binary images
Failure: means reboot
MONOLITHIC KERNEL
Advantage: apps run in own memory space
Disadvantage: kernel not protected,
kernel testing
Failure: might mean reboot
TRUE MICROKERNEL
Advantage
Modules run in own memory space
Add/replace services on the fly
Reusable modules
Direct hardware access
Disadvantage: context switching
Failure: usually does not mean reboot
3
MicroKernel – Neutrino
Process
Manager
Flash
fsys
Audio
driver
TCP/IP
App
Microkernel
X86, PPC, MIPS, SH4,
ARM, StrongARM,
XScale
Java
Serial
driver
Http
server
Photon
GUI
• Philosophy: a trusted kernel running a system of untrusted software components
• Dynamic architecture makes hot-start and upgrades easy, even with drivers
• Processes communicate via messages or other methods, such as shared memory.
Permits loose inter-module coupling.
• Processes provide a reusable component model with well defined message interfaces
• No requirement for filesystem, GUI, etc.
4
Typical Forms of IPC
Mailboxes
Process 1
Kernel
Message
Queues
Process 1
Pipes
msg 4 msg 3 msg 2 msg 5
Process
address
map
Shared
Memory
Process 2
Process
address
map
Shared
memory
object
map
Process 2
map
5
Which Architecture for me?
 Depends on your application and processor!
 Simple apps (such as single control loops) generally
only need a real-time executive
 As system becomes more complex, typically need a
more complex operating system architecture
 Need to look at factors such as scalability and
reliability
 Do standards matter?
6
API’s
Two most common standards
Advantages of standards
Portability of code
Hiring of programmers
Do I need Real-Time?
Maybe ...
What is Real Time?
Less than 1 second response?
Less than 1 millisecond response?
Less than 1 microsecond response?
8
Real-Time
"A real-time system is one in which the
correctness of the computations not only
depends upon the logical correctness of the
computation but also upon the time at which
the result is produced. If the timing
constraints of the system are not met, system
failure is said to have occurred."
Donald Gillies (comp.realtime FAQ)
9
A Simple Example...
“it doesn’t do you any good if the
signal that cuts fuel to the jet engine
arrives a millisecond after the engine
has exploded”
Bill O. Gallmeister - POSIX.4 Programming for the Real World
10
“Hard” vs. “Soft” Real Time
 Hard
– absolute deadlines
– late responses cannot be tolerated and may
have a catastrophic effect on the system
– example: flight control
 Soft
ATM
– systems which have reduced constraints on
"lateness”; e.g. late responses may still have
some value
– still must operate very quickly and repeatably
– example: cardiac pacemaker
11
Real-time OS Requirements
 Operating system factors that permit real-time:
– Thread Scheduling
– Control of Priority Inversion
– Time Spent in Kernel
– Interrupt Processing
12
Factor #1: Scheduling
 Non real-time scheduling
– round-robin
– FIFO
– adaptive
 Real-time scheduling
– priority based
– sporadic
13
Factor #2: Priority Inversion
Information Bus
Manager
Communications Task
Meteorological Data
Gathering Task
Source: Embedded Systems Programming
Sequence:
1. Low priority task acquires bus mutex to transfer data
2. High priority task blocks until mutex released
3. Medium priority task pre-empts low priority task
4. Watchdog timer resets since Bus Manager has not run in some time
14
Factor #3: Kernel Time
 Kernel operations must be pre-emptible
– if they are not, an unknown amount of time can
be spent in the kernel performing an operation on
behalf of a user process
– can cause real-time process to miss deadline
 All kernels have some window (or multiple windows)
of time where pre-emption cannot occur
 Some operating systems attempt to provide realtime capability by adding “checkpoints” within the
kernel so they can be interrupted at these points
15
Example
int KER
Entry
a few opcodes
Interrupts off
Unlocked
A Kernel call is a
software interrupt
Kernel
Operation
which
may
include
message
pass
usecs
to
msecs
Pre-emptable
Locked
usecs
No pre-emption
Interrupts on
Unlocked
usecs
Pre-emptable
a few opcodes
Interrupts off
iret
Exit
16
Split Out Long Operations
Nto
Proc
Fork
Thread
Spawn
Sched
Exec
Sync
Mmap
Signal
Pathname
Message
Waitpid
Channel
UID/GID
Timer
Session
Clock
Debug
Intr
Process
Manager
Factor #4: Interrupts
This is broken down into the following areas:
 Method of handling the interrupt processing chain
 Handling of Nested Interrupts
18
Interrupt Processing Chain
INT x INT y
ISR
ISR
INT x
ISR
INT y
ISR
IST scheduled
whenever queue
emptied, nondeterministic
IST scheduled by
normal OS
scheduling,
deterministic
IST
IST
IST
IST
19
Can I Make Any
Conventional OS Real-Time
Conventional OS
Real-time kernel
 Method
– Add real-time layer below
conventional OS, running
conventional OS as a low
priority real-time process
– Add real-time layer to
hardware service layer
 Problems
– different API’s
– real-time layer proprietary
– existing OS apps not R/T
– poor communication
between operating systems
– loss of control issue
20
Title ofScalability
presentation
Title 2
21
Scaling Solution #1:
Single Board, Single Node
Bridge
Mem.
PCI
Bus
CPU
Peripherals
 The only scaling possible is a CPU replacement
22
Scaling Solution #2:
Single Board, Multiple Nodes
Bridge
Mem.
PCI
Bus
CPU
Peripherals
Node 1
Bridge
PCI
Bus
CPU
Node 2
Relatively simple to implement
Allows “scaling-on-demand”
Suitable if nodes have independent
“work”
Peripherals
Inter-node IPC slower than memory access
Complexity in maintaining global view of data
Difficult to break-up computationally-intensive
tasks
23
Scaling Solution #3:
Single Board, Multiple Processors
Bridge
CPU
0
CPU
1
Mem.
PCI
Bus
Peripherals
 Tightly-coupled symmetric multiprocessing (SMP)
 All processors have a symmetric and consistent
view of physical memory and peripherals
 Scales processing power
 Need software (RTOS) support
24
The SMP OS Dilemma
 SMP systems to date use desktop operating systems;
not responsive enough for real-time requirements
• Application servers
• Databases
• Web servers
 Typical real-time operating systems (home-built or
commercial), such as are commonly used in routers and
switches today, do not have SMP support
 SMP capable real-time operating systems run the CPU’s
as independent processors with independent operating
systems
25
SMP Support
 True (tightly coupled) SMP
support
 Only the kernel needs
SMP awareness
 Transparent to application
software and drivers identical binaries for UP
and SMP systems
 Automatic scheduling
across all CPU’s
26
QNX “True” SMP
 STATE_RUNNING thread on
each processor
Running
Thread
Process
CPU 0
Thread
Process
Ready queues
Priority
63
62
61
...
0
Thread
Thread
Thread
 Each thread can be locked to
a specific CPU by using a
processor affinity mask
 Scheduler remembers last
CPU thread ran on
Thread
CPU 1
Blocked states
 Priority-based ready queues
Thread
– Minimize thread migration
– Optimize cache usage
 Highest-priority READY
thread always immediately
scheduled
27
Why Is Cache Important?
 Cache efficiency is probably the single largest
determinant of performance on SMP
 Coherent view of physical memory is maintained
using cache snooping
 Cache snooping is done at the CPU bus level and
so operates at lower speeds than core
 Coherency is “invisible” to software
28
Performance Implications
 Snoop traffic expected on SMP
 Cache hits generally cause no bus transaction
 Multiple processors writing to same location
degrades performance (ping-pong effect)
 Performance degrades when large amount of data
modified on one processor and read on the other
 Sometimes it is better to have specific threads in a
process run on same CPU
29
Designing for SMP:
One Big task
• Will not work with SMP
Single thread
Giant App
30
Designing for SMP:
Single Threaded Tasks
Single thread
Single thread
App 1
App 2
• Works with SMP
• Process data can be shared with
shared memory
• Good concurrency, some
complexity
• IPC not usually as efficient as
memory sharing
31
Designing for SMP:
Scaling Software with Threads
Threads
Server
• Single copy server
• All process data is implicitly shared
and accessible
• Can achieve good concurrency with
less complexity
• POSIX synchronization used
• Mutexes
• Semaphores
• Condition variables
• Usually more efficient than
inter-process synchronization
Note: SMP finds concurrency problems fast!
32
Optimizing Compute-intensive
Applications
Worker thread
Worker thread
Main thread
 Pool of worker threads
 Dispatch “work” to worker
threads
 Scales very well with SMP
 The tricky part is “breaking
up” the problem
Threads
Application
33
Interrupt Handling
IST
ISR
ISR
CPU 0
IRQ 7
CPU 1
IRQ
7
8
9
10
IRQ 8
CPU
0
1
1
1
 Interrupt processed on
CPU that was targeted
 Can distribute load by
handling interrupts on
different processors
 Sometimes not the
optimal strategy due to
cache effects
IRQ 10
IRQ 9
34
Scaling Solution #4:
Multiple Processors/Nodes
Bridge
CPU
0
CPU
1
PCI
Bus
Peripherals
Node 1
Bridge
CPU
0
CPU
1
Mem.
PCI
Bus
Peripherals
Node 2
35
Example
Low-speed bus
High-speed interconnect
Chassis
Network
Network
Network
Line card
Network
Network
Network
Line card
...
36
The QNET MicroNetwork
Process
Manager
Flash
Fsys
QNET
App
CDROM
Fsys
Audio
TCP/IP
Photon
Messages flow transparently through QNET from
one message bus to another.
LAN or
Internet or
Backplane
QNET
All applications and servers become network
distributed without any special code.
Microkernel
App
QNX Qnet Manager
Control
card
Line
card
Line
card
 Extends message passing
across multiple QNX
microkernels
 Over anything with a packet
driver:
– Ethernet, RapidIO, 3GIO,
InfiniBand, Stargen, etc.
 Class of service
 Use symbolic prefixes to make
client code independent of
location of resource manager
38
QNET Class of Service
Control
card
Line
card
Line
card
One or multiple links can connect different nodes.
39
QNET: Load-Balanced
Distribution
Control
card
Line
card
Line
card
Data is sent out the link which will deliver it the fastest. This is based upon link
speed and queue length for each link.
40
QNET: Ordered Distribution
Control
card
Line
card
Line
card
Data is sent out a primary link. If it fails, data is diverted to a secondary link. The
primary link is probed and when it comes back online, data is diverted back to it.
41
QNET: Parallel Distribution
Control
card
Line
card
Line
card
Data is sent out both links at the same time. A failure on either of the links is
handled gracefully.
42
Designing for Networked SMP:
Single/Multi Threaded Tasks
Multiple threads
Single thread
App 1
App 2
• Different processes necessary for
different nodes
• Works with SMP
• Process data can be shared with
shared memory
• IPC for networked communication
43
Transparent Redirection
Client Node
/net/a/dev/service
A
Client
/service
B
/net/b/dev/service
• Simple link provides transparent redirection
• Process has to monitor status of link
• Switch over is not transparent to client
44
Transparent Redirection
Client Node
/net/a/dev/service
A
Client
Service
mgr
/dev/service
B
/net/b/dev/service
• Service manager acts as a proxy
• Monitors health of and/or load on services/nodes
• Switch over is transparent to client
45
Redundant Links
Client Node
/net/a/dev/service
A
Client
Service
mgr
/dev/service
B
/net/b/dev/service
• Requests serviced redundantly
• First/majority/best result
• Different implementations
46
FLASH
FSYS
Blue
Tooth
TCP/IP
App
App
Photon
Graphics
Browser
Audio
Photon
Graphics
Browser
Audio
Qnet
CDROM
FSYS
Qnet
Qnet
MOST BUS
FLASH
FSYS
TCP/IP
App
App
FLASH
FSYS
Photon
Graphics
Browser
Browser
Audio
Qnet
CDROM
FSYS
Qnet
Graphics
Qnet
MOST BUS
Blue
Tooth
FLASH
FSYS
Reliability
and Availability
Title
of presentation
Title 2
49
Why?
 Embedded systems are different!
 Failure in an embedded system can have severe
effects - like death …
“Pilots really hate to be told they have
to reboot their plane while in flight”
Walter Shawlee
50
Definitions
 MTBF: Mean Time Between Failure
– The average number of hours between failures
for a large number of components over a long
time. (e.g. MIL-HDBK-217)
 MTTR: Mean Time To Repair
– Total amount of time spent performing all
corrective maintenance repairs divided by the
number of repairs
 MTBI: Mean Time Between Interruptions.
– The average number of hours between failures
while a redundant component is down.
51
Defining HA
Reliability
Availability
5 Nines
Quantified by failure rate (MTBF)
Time to resume service after failure is MTTR
Allows for failure, with quick service restoration.
As MTTR  0, Availability  100%
< 5 minutes downtime / year (> 99.999% uptime)
Assume faults exist: design to contain, notify,
recover and restore rapidly
52
Costs speak for themselves
$68,372,928
annual losses
Annual Cost of Downtime
versus Availability
$6,837,293
99%
99.9%
$683,729
99.99%
$68,373
99.999%
annual availability
Source: Gartner Group ($13,000/minute Cross-industry Average)
53
Availability via Reliability
and Repair
 low MTTR -> high availability
– System is composed of reliable components,
that are protected from each other, and that
communicate ONLY through well known
interfaces.
 this leads to
– fault isolation
– speedy recovery
– reset a component not a board/system
– dynamic control
• stop/start
• upgrade
54
Software vs Hardware HA
Hardware HA
– utilizes redundancy of key components
• a single fault cannot cause all redundant
components to fail (No SPOF). e.g. mirrored
disks, multiple system boards, I/O cards
– Active/active, active/spare, active/standby
But that’s only part of the problem!!!
Software is a Significant Cause of Downtime
55
Comparison
Planned
Outage
30%
Operator
Error
15%
Environment
5%
Software
Fault
40%
Hardware
10%
56
High Level Look at a Core
Router/Switch
ON
ON
I
I
O
O
2
3
4
6
8
9
10
11
12
13
14
15
16
17
18
19
Filler
OCLD (1E)
Shelf Processor
OCLD (2E)
OCLD (3E)
OCLD (4E)
OCI (4A)
OCI (3A)
OCM (B)
OCM (A)
OCI (2A)
7
OCI (2B)
OCI (1A)
5
OCI (1B)
OCLD (4W)
OCLD (3W)
OCLD (2W)
OCLD (1W)
1
OCI (4B)
OFF
OFF
OCI (3B)
Maintenance Panel
20
Fiber Management Trough
Cooling Unit
Optical Multiplexer Tray (OMX)
One or more control elements
57
Handling Failures
ON
ON
I
I
O
O
2
3
4
6
8
9
10
11
12
13
14
15
16
17
19
Filler
OCLD (1E)
18
Shelf Processor
OCLD (2E)
OCLD (3E)
OCLD (4E)
OCI (4A)
OCI (3A)
OCM (B)
OCM (A)
OCI (2A)
7
OCI (2B)
OCI (1A)
5
OCI (1B)
OCLD (4W)
OCLD (3W)
OCLD (2W)
OCLD (1W)
1
OCI (4B)
OFF
OFF
OCI (3B)
Maintenance Panel
20
Fiber Management Trough
Cooling Unit
Optical Multiplexer Tray (OMX)
Isolate Fault to a Board
Switch to Backup
58
May not be in the Hardware
ON
ON
I
2
3
4
6
8
9
11
12
13
15
16
18
19
Filler
Shelf Processor
OCLD (2E)
17
OCLD (1E)
OCLD (3E)
OCI (4B)
14
OCLD (4E)
OCM (B)
10
OCI (4A)
OFF
OCI (3A)
OCM (A)
OCI (2A)
7
OCI (2B)
OCI (1A)
5
OCI (1B)
OCLD (4W)
OCLD (3W)
OCLD (2W)
OCLD (1W)
1
Isolate fault to a
SW component
I
O
O
OFF
OCI (3B)
Maintenance Panel
Application
20
Application
Fiber Management Trough
Cooling Unit
Application
Network
Manager
SNMP
Manager
Device
Manager
Route
Manager
TCP/IP
stack
Flash
Drivers
Optical Multiplexer Tray (OMX)
Application
RTOS
Hardware
59
Ideal: Identify and Fix
Faulty Software
Component
Application
Application
Application
Application
Network
Manager
SNMP
Manager
Device
Manager
Route
Manager
TCP/IP
stack
Flash
Drivers
•
•
•
•
•
Isolate and contain
Repair (e.g. restart)
Notify
Diagnose
Upgrade
RTOS
60
Component-level recovery
rarely done




Lack of suitable protection and isolation
Lack of modularity
Tight component coupling
Few dynamic capabilities
Software failures normally handled by:
 Hardware watchdogs
 Redundant boards
61
Repair Time
Board Replacement
Hours
Reboot
Minutes
Failover to Standby
Seconds
SW Component Restart
10’s Milliseconds
SW Failover
Milliseconds
62
High Availability Manager
Process Memory Violation
FLASH
FSYS
DISK
FSYS
Kernel notifies HA Manager
TCP/IP
Dump file for
post-mortem
analysis
Microkernel
HA
Manager
ATM
TCP/IP
HA Manager
restarts
service
63
Notification and Recovery
 HA Manager (HAM) monitors
components, sends notification
of component failure
HAM Checkpointed
State
HAM
Guardian
HAM
Stack
 Heart-beat services detect
component hangs
 Core file on crash can be
created for debugging and
analysis
Driver
App
 Checkpointing permits
recovering current state
Checkpointed
State
64
Recovery
• A second “shadow” server attaches to the same name
65
Recovery
• A second “shadow” server attaches to the same name
• If primary faults, new clients connect to shadow server
• Old clients can re-connect to shadow server.
66
Recovery
• Start a new “shadow” server
67
Service Upgrades
Client
Server
v 1.0
/dev/service
New
Client
Server
v 1.1
/dev/service
 New version of server attaches to same name
 New clients connect to new server
 Old server exits when all old clients have exited
68
QNX Momentics Tools
69
Design Goals
 Tools needed to be easy to learn
 Tools which could take advantage of QNX
 Tools which could integrate tools from other vendors,
company designed tools, and industry specific tools
and have them work with our tools and each other
 Tools needed to be customizable to the user or the
company
70
The Best Tools and the Best
RTOS
QNX® Momentics
Commandline
tools
BSPs
QNX® Neutrino® RTOS
Invoke command-line
tools
C/C++ code
developer
3rd-Party
Tools
Rational
DDKs
Source
debugger
Java code
developer
Profiler
Target
information
Memory
analysis
Ethernet, Serial,
JTAG, ROMulator
Flash
fsys
Virtio
Neutrino
runtime
…TBA
System
builder
Photon
app builder
IDE Workbench
(Eclipse framework)
Target
agent
TCP/IP
Microkernel
Java
Photon
microGUI
Http
server
Windows, Solaris, QNX Neutrino
XScale
71
QNX IDE: Standards based
IBM donated Framework
Java IDE
200 person-years of effort
Open Source
Consortium founding members
include
72
System Profiling
73
Systems Analysis Toolkit
System
Event
Log
System Events
• interrupts,
System Characterization
• scheduler,
• Performance analysis
• messages,
• Field diagnostic
• system calls
• Live or post-mortem
Application
Instrumented
MicroKernel
Device
Driver
Protocol
Trace
TCP/IP
Printer
Statistical &
Numerical
Analysis
Data display
74
Providing Technology for Today…
Architecture for Tomorrow
Irvine Office - 949-727-0444
David Weintraub - Regional Sales Manager
[email protected]
Woodland Hills Office - 818-227-5105
Jeff Schaffer - Sr. Field Applications Engineer
[email protected]