Transcript Performance Monitor for Multiple ORB Instances

Performance Monitor for Complex,
Distributed ORB Instances
Orrington United
Chen Chen, Kaijian Liu, Hui Long, Gabe Plank
Aug. 29, 2010
Problem: Assuming massive data flow and
complex ORB systems, how can problems be
identified and performance analyzed?
Scenarios:
• Consider: Millions of Strong Motion
measurements along a pipeline. One 3C
accelerometer every 10m for 1000km at 40 SPS,
equals 12Million samples per second!
OR
• Consider: Millions of data points per second from
mobile GPS devices.
2
Who Cares ?
Currently tools exist to monitor performance of
one ORB at a time. For a complex
organization, this would be useless:
Who Cares ?
• Tools also exist to monitor large networks but
these are complicated and time consuming to
implement.
• As such, a lightweight tool for basic
performance analysis would be useful.
4
Implementation Sketch
Implement ORB
performance algorithm
using Matlab Antelope
toolkit
Manage Antelope RT
Start and Stop.
Develop algorithms to detect
potential problems to system.
User interface
(Matlab/GUI)
5
User Interface
• A graphical tool to design synthetic ORB hierarchies and
read/write configurations
• Option to load a pre-specified configuration file, for existing
real systems (implies no data added).
• Two slider bars to control size/# of data packets.
• View adjustments for looking at subsets of ORBs.
• Per-ORB dynamic performance graphs.
• Whole-system statistics such as average performance.
6
Performance Testing
Algorithm
• Stream data to the system according to user inputs.
• For each ORB, determine CPU usage, Packet Latency,
Read/Write time, Disk Access.
• Multi-thread each performance query so that all data is
returned in real time.
• Data returned from threads added to a common data
structure or a list of structures accessible by graphing and
analysis routines.
7
Manage Antelope
1. Format Antelope rtexec files from Matlab, to configure run
time parameters such as ORB port.
2. Launch Antelope real time instances Using Matlab ‘system’
call.
3. Provide process ID storage mechanism for shutting down
the processes when Matlab application exits.
ORB ports
.......
Antelope
Matlab
GUI
Start/Stop
8
Detect potential problems
Instance
1
Instance 2
Instance 4
Instance 3
1. Infinite data loops.
2. Exceeding the buffer capacity and
potential data loss.
3. Hardware problems.
9
Discussion and Questions
1. How to determine & implement meaningful wholesystem statistics?
2. Possible automated runs with notification problems do Matlab users run Matlab-based applications as
services?
3. Are ORB type buffers scalable to handle massive,
distributed data loads?
10
Summary
1. Real time data sources are growing, and so is
telemetry bandwidth.
2. Datacenters should consider the potential need for
distributed processing nodes and the means to monitor
them.
3. This project attempts a first pass specification for a
Matlab-based, performance analysis and monitoring
system for high volume, distributed real time systems.
11