Concept - Alfresco » Login
Download
Report
Transcript Concept - Alfresco » Login
Karabo: The European XFEL
software framework
Design Concepts
Burkhard Heisen for CAS
December 2014
The star marks concepts, which are not yet implemented in the current release
Karabo: The European XFEL software framework
Functional requirements
2
A typical use case:
Control
drive hardware and
complex experiments
monitor variables &
trigger alarms
DAQ
data readout
online processing
quality monitoring
(vetoing)
allow some
control & show
hardware status
Accelerator Undulator Beam Transport
show online data
whilst running
DM
DAQ
DM
Control
SC
Tight integration of applications
storage of experiment
& control data
data access,
authentication
authorization etc.
setup computation &
show scientific results
SC
processing pipelines
distributed and GPU
computing
specific algorithms
(e.g. reconstruction)
Karabo: The European XFEL software framework
Functionality: What are we dealing with?
1.
Data containers (transport and storage through serialization)
2.
Data transport (communication patterns)
3.
Devices (distributed end points)
4.
States and state machines (when can what be called/assigned on the devices)
5.
Log Messages (active, passive, central, local)
6.
(Slow) Control-Data Logging
7.
(Fast) Data acquisition
8.
Time synchronization/tagging (time stamps, cycle ids, etc.)
9.
Real-time needs (where necessary)
10. Notifications and Alarms
11. Security (who’s allowed to do what from where?)
12. Statistics (control system itself, operation, …)
13. Processing workflows (parallelism, pipeline execution, provenance)
14. Clients / User interfaces (API, languages, macro writing, CLI, GUI)
15. Experiment, Run and Configuration management
16. Software management (coding, building, packaging, deployment, versioning, …)
3
Karabo: The European XFEL software framework
Data containers
Some special data containers are provided by Karabo and are exposed in the API
Hash
String-key, any-value associative container
Keeps insertion order (iteration possible), hash performance for random lookup
Provide (string-key, any-value) attributes per hash-key
Fully recursive structure (i.e. Hashes of Hashes)
Serialization: XML, Binary, HDF5
Usage: configuration, device-state cache, DB-interface, message protocol, etc.
Schema
Describes possible/allowed structures for the Hash. In analogy: Schema would
be for Hash, what an XSD document is for an XML file
Internally uses Hash
RawImageData
Specialized class for transporting image-like data
Easily convertible to numpy in Python and to CpuImage<T> in C++
Optimized serialization into HDF5
Internally uses Hash
4
Karabo: The European XFEL software framework
STATUS: Data containers
Recent changes
None
Future work
Improve Hash serialization implementation with respect to the XML format to allow
for slashes “/” in hash-keys
Find a proper data object (eventually plus some description) to exchange data with
the DAQ layer
Open issues
Understand the conceptual difference between using standardized objects vs.
generic container + description throughout the system (see also DAQ section)
5
Karabo: The European XFEL software framework
Data transport – Message broker based
6
Basic communication between objects is established via a central
message broker using a publish-subscribe pattern (topic based)
Each communicating object is an instance of the SignalSlotable class which connects
to a configurable broker (host/port/topic)
The SignalSlotable API allows to register regular functions of any signature (currently
up to 4 arguments) to be remotely callable (such a function is called: Slot)
Slots can be uniquely addressed by a pair of strings, the instanceId (string name of
the SignalSlotable object) and the functionName (string name of the function)
Slot registration can be done during construction or later at runtime without extra tools
Slot calls can be done cross-network, cross-operating-system and crosslanguage (currently C++ and Python)
The language’s native data types are directly supported as arguments
Additionally supported arguments are Karabo’s data objects (e.g. Hash and Schema)
Data packets are on the fly compressed/decompressed if reaching some size
threshold
New
Karabo: The European XFEL software framework
DETAIL: Data transport
Broker based communication API – Four Patterns
7
① Signals & Slots
SLOT ( function, [argTypes] )
SIGNAL ( funcName, [argTypes] )
connect ( signalInstanceId, signalFunc, slotInstanceName, slotFunc )
emit ( signalFunc, [args] )
SLOT(onFoo, int, std::string);
void onFoo(const int i, std::string& s) { }
SIGNAL(“foo”, int, std::string);
connect(“Device1”, “foo”, “Device2”, “onFoo”);
connect(“”, “foo”, “Device3”, “onGoo”);
connect(“”, “foo”, “Device4”, “onHoo”);
emit(“foo”, 42, “bar”);
SLOT(onGoo, int, std::string);
void onGoo(const int i) { }
Device2
Notify
Device1
Emit
Device3
Notify
Notify
Device4
SLOT(onHoo, int, std::string);
void onHoo(const int i, std::string& s) { }
Karabo: The European XFEL software framework
DETAIL: Data transport
Broker based communication API – Four Patterns
call(“Device2”, “onFoo”, “bar”);
8
SLOT(onFoo, std::string);
void onFoo(const std::string& s) { }
② Direct Call
call ( instanceId, funcName, [args] )
Call
Device1
Notify
Device2
③ Request / Reply
request ( instanceId, funcName, [reqArgs] ).timeout( msec ).receive( [repArgs] )
int number;
request(“Device2”, “onFoo”, 21).timeout(100).receive(number);
Request
SLOT(onFoo, int);
void onFoo(const int i) { reply( i + i ); }
Notify
Device2
Device1
Notify
Reply
Karabo: The European XFEL software framework
DETAIL: Data transport
Broker based communication API – Four Patterns
9
④ Asynchronous Request / Reply
requestNoWait ( req_instanceId, req_funcName, rec_instanceId, rec_funcName, [reqArgs] )
SLOT(onFoo, int);
void onFoo(const int i) { reply( i + i ); }
requestNoWait(“Device2”, “onFoo”, “”, “onBar”, 21);
Request
Notify
Device2
Device1
Notify
SLOT(onBar, int);
onBar(const int i) { … }
Reply
New
Karabo: The European XFEL software framework
STATUS: Broker communication
Recent changes
Fundamental change of how messages are consumed
Before: Each Slot presented an own consumer on the broker, forcing the broker
to route (using “Selectors”) messages by selecting on instanceId and
functionName. Larger installation caused huge number of consumer clients on
the broker and needed a thread per Slot on the device
Now: Each object is a consumer on the broker, routing is done only utilizing the
instanceId only. Slot selection is done on the client side. Using an own queuing
system on the SignalSlotable the number of threads used per instance is
decoupled from the number of slots (can be single threaded context)
Heartbeats get first priority (using own topic and by placing on front of queue)
Future work
Performance and scalability tests
Check whether trouble with heartbeats is finally solved
Open issues
10
Karabo: The European XFEL software framework
Data transport – P2P
Another fundamental communication pattern between objects is realized by
connecting so-called input and output channels to form a direct point-to-point
connection (shortcutting the broker)
Unlike slots (which are functions) input and output channels are named objects
with a read/write/update API
The SignalSlotable API allows to create one or more such channels per
SignalSlotable instance
Technically output channels are (multi-client capable) TCP servers, input channels
are clients
The connection between them is established by referring to instanceId and
channelName instead of host and port.
Host and port are transparently communicated during connection time using the
broker based communication
Channels are highly configurable and are intended to serve the need of flexible
streaming data pipeline setups
11
Karabo: The European XFEL software framework
DETAIL: Data transport
P2P communication
12
One channel is specific for one data object (e.g. Hash, Image, Byte-Array)
For input and output channels within the same application data exchange will
happen by handing over pointers in memory instead of transmitting via TCP
Users can register two function call-backs indicating availability of data (e.g.
onData) and (optionally) the end of the data stream (e.g. onEndOfStream) on the
input channel
Input channels configure whether they share the sent data with all input channels
connected to the same output or whether they receive a copy of each data token
Output channels may specify a special hostname (in case of multiple adapters) to
which the clients are routed to
New
Message Broker
P2P
Data
[…]
Karabo: The European XFEL software framework
STATUS: P2P communication
Recent changes
Added possibility to select the interface on which to communicate
Future work
More performance and scalability tests
Whilst reading is already asynchronous to the users-code execution (IO during
processing), for writing this is currently not true (was implemented but removed for
instability issues)
Asynchronous must again be implemented for performance improvement
Open issues
Think carefully whether this communication could also be used (performance issue)
for transporting fast DAQ data from Device to DAQ-Layer
13
Karabo: The European XFEL software framework
Devices (distributed end points)
The distributed end points follow the “Device Server Model”
Similar to: TANGO or DOOCS
End points are controllable objects managed by a device server
Instance of such an object is a Device, with a hierarchical name
Device classes can be loaded at runtime (plugins)
Devices inherit SignalSlotable and wrap the communication API into a simpler
subset
Actions pertaining to a device given by its properties, commands, and channels
i.e. get, set, monitor some property or execute some command
write/read some data to/from a channel and update when done
Properties, commands and channels are statically described (expectedParameters
function) and further described via attributes in the device class. This description is
saved in form of a Schema. Dynamic (runtime) extension (Schema injection) of
expectedParameters is possible.
Devices can be written in either C++ or Python
14
Karabo: The European XFEL software framework
DETAIL: Devices
Configuration - API
Any Device uses a standardized API to
describe itself. This information is
shipped as Schema object and used by
interested clients (GUI, CLI other
devices)
We distinguish between properties
and commands and associated
attributes, all of them can be
expressed within the expected
parameters function
No need for device developers to
validate any parameters. This is
internally done taking the
expectedParameters as white-list
Properties and commands can be
nested, such that hierarchical
groupings are possible
15
Class: MotorDevice
Property
static expectedParameters( Schema& s ) {
FLOAT_ELEMENT(s).key(“velocity”)
.description(“Velocity of the motor”)
.assignmentOptional().defaultValue(0.3)
.maxInc(10)
.minInc(0.01)
Attribute
.reconfigurable()
.allowedStates(“Idle”)
.commit();
INT32_ELEMENT(s).key(“currentPosition”)
.description = “Current position of the motor”
.readOnly()
.warnLow(10)
[…]
Command
SLOT_ELEMENT(s).key(“move”)
.description = “Will move motor to target position”
.allowedStates(“Idle”)
[…]
}
// Constructor with initial configuration
MotorDevice( const Hash& config ) { […] }
// Called at each (re-)configuration request
onReconfigure( const Hash& config ) { […] }
Karabo: The European XFEL software framework
DETAIL: Devices
Creating a new device
1.
Write a class (say: MyDevice) that derives
from Device
2.
libMy
Device.so
Compile it into a shared library (say
libMyDevice.so)
3.
16
Select a running Device-Server or start a
plugins
signalNewDeviceClassAvailable (.xsd)
new one
4.
Copy the libMyDevice.so to the plugins
folder of the Device-Server
5.
The Device-Server will emit a signal to the
broker that a new Device class is
GUI-Srv
available, it ships the expected
parameters as read from static context of
the MyDevice class
GUI
Karabo: The European XFEL software framework
DETAIL: Devices
Creating a new device
6.
Given the mask of possible parameters the
17
factory: create(“MyDevice”, xml)
user may fill a valid configuration and emit
an instantiate signal to the broker
7.
MyDevice
1
The configuration will be validated by the
plugins
Device factory and if valid, an instance of
MyDevice will be created
8.
The constructor of the device class will be
called and provided with the configuration
9.
signalInstantiate(“MyDevice”, xml)
The run method will be called which starts
the state-machine and finally blocks by
GUI-Srv
activating the event-loop
10.
The device will asynchronously listen to
allowed events (slots)
GUI
Karabo: The European XFEL software framework
Device “flavors”
Equipment
Control
e.g. motor, pump,
valve, sensor
Composite
Device
DAQ
Equipment
without Data
DAQ
Equipment
with Data
e.g. commercial camera
PCLayer
Node
Service
Device
e.g. digitizer, beam position
monitor, 2D-detectors
e.g. calibrationManager,
projectManager,
brokerMonitor
Workflow
Node
Karabo: The European XFEL software framework
DETAIL: Devices
Devices taking part in distributed system
Device Instance
HV
Device-Server
Application
19
Digitizer
Pump
Message Broker
(Event Loop)
Store
Camera
Disk
Storage
Load
Calibrate1
Simulate
Terminal(s)
Calibrate2
Logger
GUI
Server
GUI(s)
Karabo: The European XFEL software framework
STATUS: Devices
Recent changes
Future work
Best practices and all concepts for hierarchical device structures must be defined
The composed-in DeviceClient API needs more functionality to make composition
easier
Open issues
20
Karabo: The European XFEL software framework
States and state machines
Any property setting or command execution on a Device
can be restricted to a set of allowed states (using the
allowedStates attribute)
The state of a device can be changed by simply setting the
state property (string) to the desired value New
The GUI is state and allowed states aware and
enables/disables buttons and properties pro-actively
Devices may optionally implement a finite state machine
(FSM) following the UML standard
In this case an incoming slot call is not directly
implemented but triggers and event into the state
machine
User defined hooks are executed as consequence of a
start-to-finish event processing algorithm.
Possible hooks are: guard, src-state-on-exit, transitionaction, tgt-state-on-entry, on-state-action
New
21
Start Stop State Machine
Initialization
none
OK
Stopped
stop
start
Started
reset
errorFound
Error
Karabo: The European XFEL software framework
DETAIL: States and state machines
Finite state machines – There is a UML standard
State Machine: the life cycle of a thing. It is made of states, transitions and processes
incoming events.
State: a stage in the life cycle of a state machine. A state (like a submachine) can have
an entry and exit behaviors
Event: an incident provoking (or not) a reaction of the state machine
Transition: a specification of how a state machine reacts to an event. It specifies a
source state, the event triggering the transition, the target state (which will become the
newly active state if the transition is triggered), guard and actions
Action: an operation executed during the triggering of the transition
Guard: a boolean operation being able to prevent the triggering of a transition which
would otherwise fire
Transition Table: representation of a state machine. A state machine diagram is a
graphical, but incomplete representation of the same model. A transition table, on the
other hand, is a complete representation
22
Karabo: The European XFEL software framework
DETAIL: States
FSM implementation example in C++ (header only)
// Events
FSM_EVENT2(ErrorFoundEvent,
FSM_EVENT0(EndErrorEvent,
FSM_EVENT0(StartEvent,
FSM_EVENT0(StopEvent,
onException, string, string)
endErrorEvent)
slotMoveStartEvent)
slotStopEvent)
// States
FSM_STATE_EE(ErrorState, errorStateOnEntry, errorStateOnExit)
FSM_STATE_E(InitializationState, initializationStateOnEntry)
FSM_STATE_EE(StartedState, startedStateOnEntry, startedStateOnExit)
FSM_STATE_EE(StoppedState, stoppedStateOnEntry, stoppedStateOnExit)
// Transition Actions
FSM_ACTION0(StartAction, startAction)
FSM_ACTION0(StopAction, stopAction)
Regular callable function (triggers event)
Transition table element
Regular function hook (will be call-backed)
Transition table element
// AllOkState Machine
FSM_TABLE_BEGIN(AllOkStateTransitionTable)
//
SrcState
Event
TgtState
Action
Guard
Row< StartedState, StopEvent, StoppedState, StopAction, none >,
Row< StoppedState, StartEvent, StartedState, StartAction, none >
FSM_TABLE_END
FSM_STATE_MACHINE(AllOkState, AllOkStateTransitionTable, StoppedState, Self)
// StartStop Machine
FSM_TABLE_BEGIN(StartStopTransitionTable)
Row< InitializationState, none,
AllOkState, none,
none >,
Row< AllOkState,
ErrorFoundEvent, ErrorState, ErrorFoundAction, none >,
Row< ErrorState,
EndErrorEvent,
AllOkState, EndErrorAction,
none >
FSM_TABLE_END
KARABO_FSM_STATE_MACHINE(StartStopMachine, StartStopMachineTransitionTable, InitializationState, Self)
FSM_CREATE_MACHINE(StartStopMachine, m_fsm);
FSM_SET_CONTEXT_TOP(this, m_fsm)
FSM_SET_CONTEXT_SUB(this, m_fsm, AllOkState)
FSM_START_MACHINE(m_fsm)
23
Karabo: The European XFEL software framework
DETAIL: States
FSM implementation example in Python
# Events
FSM_EVENT2(self,
FSM_EVENT0(self,
FSM_EVENT0(self,
FSM_EVENT0(self,
‘ErrorFoundEvent’,
‘EndErrorEvent’,
‘StartEvent’,
‘StopEvent’,
# States
FSM_STATE_EE(‘ErrorState’,
FSM_STATE_E( ‘InitializationState’,
FSM_STATE_EE(‘StartedState’,
FSM_STATE_EE(‘StoppedState’,
‘onException’)
‘slotEndError’)
‘slotStart’)
‘slotStop’)
self.errorStateOnEntry,
self.errorStateOnExit )
self.initializationStateOnEntry
)
self.startedStateOnEntry, self.startedStateOnExit)
self.stoppedStateOnEntry, self.stoppedStateOnExit)
# Transition Actions
FSM_ACTION0(‘StartAction’, self.startAction)
FSM_ACTION0(‘StopAction’, self.stopAction)
# AllOkState Machine
allOkStt = [
# SrcState
Event
(‘StartedState’, ‘StartEvent’,
(‘StoppedState’, ‘StopEvent’,
]
FSM_STATE_MACHINE(‘AllOkState’,
TgtState
Action
Guard
‘StoppedState’, ‘StartAction’, ‘none’),
‘StartedState’, ‘StopAction’, ‘none’)
allOkStt, ‘InitializationState’)
# Top Machine
topStt = [
(‘InitializationState’, ‘none’,
‘AllOkState’, ‘none’, ‘none’),
(‘AllOkState’,
‘ErrorFoundEvent’, ‘ErrorState’, ‘none’, ‘none’),
(‘ErrorState’,
‘EndErrorEvent’,
‘AllOkState’, ‘none’, ‘none’)
]
FSM_STATE_MACHINE(‘StartStopDeviceMachine’, topStt, ‘AllOkState’)
self.fsm = FSM_CREATE_MACHINE(‘StartStopMachine’)
self.startStateMachine()
24
Karabo: The European XFEL software framework
STATUS: States
Recent changes
A hook for performing some (periodic) action whilst being in a state was added to
the FSM
A clean way of implementing devices without FSM is available (and is now
recommended)
Future work
In case of no FSM: Device-side validation of command executions and property
settings against allowed states attribute
Open issues
25
Karabo: The European XFEL software framework
Data logger
All property changes of all devices are archived centrally and in an eventdriven way
The archive can be used to debug the system at a later point
The data logger allows fast retrieval of two kinds of information:
Values of a property in a selected time range (feeding e.g. trend line plots in
GUI)
The full configuration of a device at a given time point
By default all devices and all their properties are logged. However, entire
devices or individual properties of those may be flagged to be excluded from
logging
Logging is done in a per-device fashion and for any device currently 3 append
able text files are generated:
*_configuration.txt: Stores all changes of the device properties
*_schema.txt: Stores all changes of the device schema
*_index.txt: Index file for speeding up queries
Changed
Karabo: The European XFEL software framework
Data logger
Any regular device has a
DataLogger_<deviceName> companion
DeviceB
DeviceA
A DataLoggerManager composite
device couples the life-time of the two
companions
DataLogger
Manager
DataLogger
DeviceA
DataLogger
DeviceB
Karabo: The European XFEL software framework
STATUS: Data Logger
Recent changes
A hook for performing some (periodic) action whilst being in a state was added to
the FSM
A clean way of implementing devices without FSM is available (and is now
recommended)
Future work
Further scaling will be done by running DataLoggers on several hosts (connected to
a shared file system) as configured via the DataLoggerManager
A second device will be implemented that reads the generated files and
asynchronously populates a RDBMS
Open issues
Is the data we log complete? Should not command executions also be part of the
logged data?
28
Karabo: The European XFEL software framework
Data acquisition
29
direct TCP
channels
via broker
Data aggregation,
integration &
dissemination
Multiple aggregator
instances to handle
all slow & fast data
Borrowed from Djelloul Boukhelef
Karabo: The European XFEL software framework
Concept thoughts: DAQ integration
DAQ
Equipment
without Data
Equipment
Control
via broker
Data aggregation,
integration &
dissemination
30
DAQ
Equipment direct TCP
with Data channels
Multiple aggregator
instances to handle
all slow & fast data
Aggregator
PCLayer
Node
Workflow
Node
Borrowed from Djelloul Boukhelef
Karabo: The European XFEL software framework
STATUS: Data Acquisition integration
Future work
Think about the best way how to transport the data (which is send by Karabo
devices) to the DAQ layer
Open questions
Requirements for sending data from Karabo devices to DAQ layer instead of
sending data between devices for workflow purposes are different
No “smartness” needed (like load balancing, multi-cast, etc.)
Writing to file can be done more generic, than further processing (what format
is the best)
Can we and should we try to use the same API and implementation for scientific
workflows and DAQ sinking?
Burkhard Heisen (WP76)
31
Karabo: The European XFEL software framework
Real time needs (where necessary)
32
Karabo itself does not provide real time
processes/communications
Motor1
Motor2
Pump1
Real time processes (if needed) must be
defined and executed in layers below
Karabo. Karabo devices will only
start/stop/monitor real time processes
Gather/Scatter
An example for a real-time system are the
TCP (own protocoll)
Ethercat based solutions from the company
Beckhoff which we can interface to
Beck
Com
Interlock/Supervisory code can be
PLCCPU
Ethercat
Motor2
Pump1
implemented at either PLC (realtime) and
Karabo
Burkhard Heisen (WP76)
Motor1
Karabo: The European XFEL software framework
Time synchronization (time stamps, cycle ids, etc.)
Concept: Any changed property will carry timing information as attribute(s)
Time information is assigned per property
Karabo’s timestamp consists of the following information:
Seconds since unix epoch, uint64
Fractional seconds (up to atto-second resolution), uint64
Train ID, uint64
Time information is assigned as early as possible (best: already on hardware) but
latest in the software device
On event-driven update, the device ships the property key, the property value and
associated time information as property attribute(s)
Real-time synchronization is not subject to Karabo
Correlation between control system (monitor) data and instrument data will be
done using the archived central DB information (or information previously
exported into HDF5 files)
Burkhard Heisen (WP76)
33
Karabo: The European XFEL software framework
DETAIL: Time synchronization
Distributed Train ID clock
Concept: A dedicated machine with a time receiver board
(h/w) distributes clocks on the Karabo level
Scenario 1: No time information from h/w
Example: commercial cameras
Timestamp is associated to the event-driven data in
the Karabo device
If clock signal is too late, the next trainId is calculated
(extrapolated) given the previous one and the interval
between trainId's
The interval is configurable on the Clock device and
must be stable within a run. Error is flagged if clock
tick is lost.
Scenario 2: Time information is already provided by h/w
34
creates
timestamp and
associates to
trainId
Device
signals:
1.
trainId
2.
epochTime
3.
interval
Clock
The timestamp can be taken from the h/w or the
device (configurable). The rest is the same as in
scenario 1.
Time receiver board
Burkhard Heisen (WP76)
Karabo: The European XFEL software framework
Central services - Name resolution/access
The only central service technically needed is the broker, others are optional
Start-up issues
Any object connecting to the same broker (host/port/topic) must have a unique ID
(string)
All communication objects will finally derive the SignalSlotable class which can be
instantiated with a given ID (configured) or generates one if no ID is provided
If no instance ID is provided the ID is auto-generated locally
Servers: hostname_Server_pid
Devices: hostname-pid_classId_counter
Any instance ID is validated (by request-response trial) prior startup to be unique in
the distributed system
Burkhard Heisen (WP76)
35
Karabo: The European XFEL software framework
DETAIL: Access levels
36
We will initially have five access levels (enums) with intrinsic ordering
ADMIN = 4
EXPERT = 3
OPERATOR = 2
USER = 1
OBSERVER = 0
Any Device can restrict access globally or on a per-parameter basis
Global restriction is enforced through the “visibility” property (base class)
Only if the requestor is of same or higher access level he can see/use the device
The “visibility” property is part of the topology info (seen immediately by clients)
Parameter restriction is enforced through the “requiredAccessLevel” schema-attribute
Parameter restriction typically is set programmatically but may be re-configured
at initialization time (or even runtime?)
The “visibility” property might be re-configured if the requestors access level is higher
than the associated “requiredAccessLevel” (should typically be ADMIN)
The default access level for settable properties and commands is USER
The default access level for read-only properties is OBSERVER
The default value for the visibility is OBSERVER
Burkhard Heisen (WP76)
Karabo: The European XFEL software framework
DETAIL: Access levels
A role is defined in the DB and consists of a default access level and a deviceinstance specific access list (overwriting the default level) which can be empty.
SPB_Operator
defaultAccessLevel => USER
accessList
SPB_* => OPERATOR
Undulator_GapMover_0 => OPERATOR
Global_Observer
defaultAccessLevel => OBSERVER
Global_Expert
defaultAccessLevel = EXPERT
After authentication the DB computes the user specific access levels considering
current time, current location and associated role. It then ships a default access and
an access level list back to the user.
If the authentication service (or DB) is not available, Karabo falls back to a
compiled default access level (in-house: OBSERVER, shipped-versions: ADMIN)
For a ADMIN user it might be possible to temporarily (per session) change the
access list of another user.
Burkhard Heisen (WP76)
37
Karabo: The European XFEL software framework
DETAIL: Security
38
Broker-Message
GUI or CLI
GUI-Srv
Header […]
__uid=42
__accessLevel=“admin”
Body […]
userId
sessionToken
defaultAccessLevel
accessList
username
password
provider
ownIP*
brokerHost*
brokerPort*
brokerTopic*
Locking:
if is locked:
if is __uid == owner then ok
Device
Access control:
if __accessLevel >= visibility:
if __accessLevel >= param.accessLevel then ok
Central DB
1.
2.
Burkhard Heisen (WP76)
Authorizes
Computes context based access levels
Karabo: The European XFEL software framework
Statistics (control system itself, operation, …)
Concept: Statistics will be collected by regular devices
OpenMQ implementation provides a wealth of statistics (e.g. messages in
system, average flow, number of consumers/producers, broker memory used…)
Have a (broker-)statistic device that does system calls to retrieve information
Similar idea for other statistical data
Burkhard Heisen (WP76)
39
Karabo: The European XFEL software framework
Logging (active, passive, central, local)
Concept: Categorized into the following classes
Active Logging Additional code (inserted by the developer) accompanying the
production/business code, which is intended to increase the verbosity of what is currently
happening.
Code Tracing Macro based, no overhead if disabled, for low-level purposes
Code Logging Conceptual analog to Log4j, network appender, remote and at runtime
priority (re-)configuration
Passive Logging Recording of activities in the distributed event-driven system. No extra
coding is required from developers, passive logging transparently records system relevant
events.
Broker-message logging Low-level debugging purpose, start/stop, not active during
production
Burkhard Heisen (WP76)
Transactional logging Archival of the full distributed state (see DataLogger)
40
Karabo: The European XFEL software framework
Project
41
The project is an organizational structure for logically related devices
The project does not describe:
Which device-server should run on what host
Which plugin is loaded to what device-server
The project acts on top of existing (running) device-servers and loaded plugins
It describes initial configurations, runtime configurations, macros, scenes, monitors
and resources for a set of logically connected devices
Example projects could be:
Detector_FXE
Laser_FXE
DAQ_FXE
Macros have an API to work with the project
Projects are associated to a user (can be a functional user)
The project itself is a set of files, it does not maintain a state (like “started” or “stopped”)
Karabo: The European XFEL software framework
Project
Centralized project storing via a Karabo service device “ProjectManager”
Implement in Python (code already exists in GUI code)
Analog to DataLoggerManager or CalibrationManager within Karabo Framework
Implement an output (loading project) and an input (saving project) channel
Allow multi-user (read) and single-user (write) access
42
Karabo: The European XFEL software framework
Detail: Project file organization
The project is saved as a zipped folder named <projectname>.krb
The folder contains a project.xml file with the following structure:
<project>
<devices>[…]</devices>
<macros>[…]</macros>
<scenes>[…]</scenes>
<monitors>[…]</monitors>
<resources>[…]</resources>
</project>
And sub-folders containing files which are referenced by the above mentioned
project.xml
Devices
Containing <device>.xml files
Macros
Containing <macro>.py files
Scenes
Containing <scene>.svg files
Resources
Containing any files (images, specific configurations, notes, etc.)
etc.
Burkhard Heisen (WP76)
43
Karabo: The European XFEL software framework
STATUS: Project
Recent changes
Logical grouping of devices of same class is possible
Groups allow multi-edit functionality (very useful for work-flow configurations)
Future work
Central project handling must be implemented
The Monitors section must be implemented
Monitors are a user defined collection of properties that will be associated to a experimental
run (or a control scan)
Open questions
Current idea is to introduce another top hierarchy level -> a project group
Groups should be logical associations and only point/link to the physical projects
A project group could reflect all settings an experiment needs by aggregating all specialists
projects (like laser, detector, daq, experiment) with the user project
Experiment configurations could be started by copying a (template) group and then modifying it
by the individual experts until the specified setup is reached
Still not completely clear whether this approach will cover all needs for experiment control
44
Karabo: The European XFEL software framework
Processing workflows (parallelism, pipeline
execution, provenance)
Concept: Devices as modules of a scientific workflow system
Configurable generic input/output channels on devices
One channel is specific for one data structure (e.g. Hash, Image, File, etc.)
New data structures can be “registered” and are immediately usable
Input channel configuration: copy of connected output’s data or share the data with
other input channels, minimum number of data needed
ComputeFsm as base class, developers just need to code the compute method
IO system is decoupled from processing system (process whilst transferring data)
Automatic (API transparent) data transfer optimization (pointer if local, TCP if remote)
Broker-based communication for workflow coordination and meta-data sharing
GUI integration to setup workflows graphically (drag-and-drop featured)
Workflows can be stored and shared (following the general rules of data privacy and
security) executed, paused and stepped
Parallel execution
Burkhard Heisen (WP76)
45
Karabo: The European XFEL software framework
DETAIL: Processing workflows
Parallelism and load-balancing by design
46
Devices within the same device-server:
Data will be transferred by handing over pointers
to corresponding memory locations
Multiple instances connected to one output
channel will run in parallel using CPU threads
Memory
CPU-threads
Devices in different device-servers:
Data will be transferred via TCP
Multiple instances connected to one output
channel will perform distributed computing
TCP
Distributed processing
Output channel technically is TCP server, inputs are clients
Data transfer model follows an event-driven poll architecture, leads to load-balancing
and maximum per module performance even on heterogeneous h/w
Configurable output channel behavior in case no input currently available: throw, queue,
wait, drop
Burkhard Heisen (WP76)
Karabo: The European XFEL software framework
DETAIL: Processing workflows
GPU enabled processing
47
Concept: GPU parallelization will happen within a compute execution
The data structures (e.g. image) are prepared for GPU parallelization
Karabo will detect whether a given hardware is capable for GPU computing at runtime,
if not falls back to corresponding CPU algorithm
Differences in runtime are balanced by the workflow system
CPU
IO whilst computing
Pixel parallel processing
(one GPU thread per pixel)
Notification about new data possible to obtain
GPU
Burkhard Heisen (WP76)
Karabo: The European XFEL software framework
Clients / User interfaces (API, languages, macro writing,
CLI, GUI)
Concept: Two UIs – graphical (GUI) and scriptable command line (CLI)
GUI
Have one multi-purpose GUI system satisfying all needs
See following slides for details
Non-GUI
We distinguish APIs for programmatically set up of control sequences (others call
those Macros) versus and API which allows interactive, commandline-based
control (IPython based)
The programmatic API exists for C++ and Python and features:
Querying of distributed system topology (hosts, device-servers, devices, their
properties/commands, etc.): getServers, getDevices, getClasses
instantiate, kill, set, execute (in “wait” or “noWait” fashion), get, monitorProperty,
monitorDevice
Both APIs are state and access-role aware, caching mechanisms provide proper
Schema and synchronous (poll-feel API) although always event-driven in the backend
The interactive API integrates auto-completion and improved interactive
functionality suited to iPython
Burkhard Heisen (WP76)
48
Karabo: The European XFEL software framework
GUI: What do we have to deal with?
Client-Server (network protocol, optimizations)
User management (login/logout, load/save settings, access role support)
Layout (panels, full screen, docking/undocking)
Navigation (devices, configurations, data, …)
Configuration (initialization vs. runtime, loading/saving, …)
Customization (widget galleries, custom GUI builder, composition, …)
Notification (about alarms, finished pipelines, …)
Log Inspection (filtering, configuration of log-levels, …)
Embedded scripting (iPython, macro recording/playing)
Online documentation (embedded wiki, bug-tracing, …)
Kerstin Weger (WP76)
49
Karabo: The European XFEL software framework
Client-Server (network protocol, optimizations)
Message
Broker
Concept: One server, many clients, TCP
Server knows what each client user sees (on a
device level) and optimizes traffic accordingly
Client-Server protocol is TCP, messages are
header/body style using Hash serialization (default
binary protocol)
Client side socket will be threaded to decouple from
main-event loop
Central DB
Master
GUI-Srv
On client start server provides current distributed
state utilizing the DB, later clients are updated
through the broker
Image data is pre-processed on server-side and
brought into QImage format before sending
I only see
device “A”
onChange
information only
related to “A”
GUI-Client
Kerstin Weger (WP76)
50
Karabo: The European XFEL software framework
User management (login/logout, load/save settings,
access role support)
51
Concept: User centralized, login mandatory
Login necessary to connect to system
Access role will be computed (context based)
User specific settings will be loaded from DB
View and control is adapted to access role
User or role specific configuration and wizards are
available
userId
accessRole
session
username
password
Central DB
1.
2.
Kerstin Weger (WP76)
Authorizes
Computes context based
access role
Karabo: The European XFEL software framework
Layout (panels, full screen, docking/undocking)
Six dock-able and slide-able (optionally tabbed) main panels
Panels are organized by functionality
Navigation
Custom composition area (sub-GUI building)
Configuration (non-tabbed, changes view based on selection elsewhere)
Documentation (linked and updated with current configuration view)
Logging
Notifications
Project
Panels and their tabs can be undocked (windows then belongs to OS’s window
manager) and made full-screen (distribution across several monitors possible)
GUI behaves natively under MacOSX, Linux and Windows
Kerstin Weger (WP76)
52
Karabo: The European XFEL software framework
Graphical interface - Overview
Live
Navigation
53
drag & drop
Configuration
Custom Scene
User centric and access-controlled setup (login at startup)
Dock-able and resizable multi-panel, all-in-one user interface
Live navigation showing all device-servers, plugins, and device instances
Automatically generated configuration panel, allowing to
read/write/execute
PowerPoint like, drag & droppable, tabbed custom scene
Project panel for persisting configurations, macros, scenes, resources, etc.
Centralized logging information, notification handling, documentation, etc.
Project
Burkhard Heisen (CAS Group)
Log Messages
Interactive Command Line
Documentation
Bug Reporting
Karabo: The European XFEL software framework
Navigation (devices, configurations, data, …)
Concept: Navigate device-servers, devices,
configurations, data(-files), etc.
Different views (tabs) on data
Hierarchical distributed system view
Device ownership centric (view compositions)
Hierarchical file view (e.g. HDF5)
Automatic (by access level) filtering of items
Auto select navigation item if context is selected
somewhere else in GUI
Kerstin Weger (WP76)
54
Karabo: The European XFEL software framework
Configuration (initialization vs. runtime, loading/saving, …)
Concept: Auto-generated default widgets for
configuring classes and instances
Widgets are generated from device information (.xsd
format)
2-column layout for class configuration (label,
initialization-value)
3-column layout (label, value-on-device, edit-value)
for instance configuration
Allows reading/writing properties (all data-types)
Allows executing commands (as buttons)
Is aware about device’s FSM, enables/disables
widgets accordingly
Is aware about access level, enables/disables
widgets accordingly
Single, selection and all apply capability
Kerstin Weger (WP76)
55
Karabo: The European XFEL software framework
Customization (widget galleries, custom GUI builder,
composition, …)
Concept: Combination of PowerPoint-like editor and online
properties/commands with changeable widget types
Tabbed, static panel (does not change on navigation)
Two modes: Pre-configuration (classes) and runtime configuration (instances)
Visual composition of properties/commands of any devices
Visual composition of devices (workflow layouting)
Data-type aware widget factory for properties/commands (edit/display)
PowerPoint-like tools for drawing, arranging, grouping, selecting, zooming of text,
shapes, pictures, etc.
Capability to save/load custom panels, open several simultaneously
Kerstin Weger (WP76)
56
Karabo: The European XFEL software framework
DETAIL: Customization
Property/Command composition
Display widget (Trend-Line)
Editable widget
drag & drop
Display widget
Kerstin Weger (WP76)
57
Karabo: The European XFEL software framework
DETAIL: Customization
Property/Command composition
Display widget
(Image View)
Display widget
(Histogram)
Kerstin Weger (WP76)
drag & drop
58
Karabo: The European XFEL software framework
DETAIL: Customization
Device (workflow) composition
Whole devices can be dragged
(from left side) as pipeline nodes
Dragging individual parameters
from right is still possible (e.g.
control parameters)
Devices can be grouped and
edited as group (connections
and configurations)
Distributed computing will
happen if different hosts are
involved
Display of per node or nodegroup utilization
Kerstin Weger (WP76)
drag & drop
59
Karabo: The European XFEL software framework
Macro editing and execution
Macro editing and execution from
within GUI possible
Macro parameters and functions
integrate automatically into
configuration panel
Macros are running within the
GUI’s event loop (direct widget
manipulation possible)
Macro API can be interactively
executed in embedded IPython
interpreter
Asynchronous operations use
Python 3’s coroutines and the
yield from keyword (extension
written allowing this for IPython)
60
Courtesy of M. Teichmann
Burkhard Heisen (WP76)
Karabo: The European XFEL software framework
Notification (about alarms, finished runs, …)
Concept: Single place for all system relevant notifications, will link-out to more
detailed information
Can be of arbitrary type, e.g.:
Finished experiment run/scan
Finished analysis job
Occurrences of errors, alarms
Update notifications, etc.
Intended to be conceptually similar to now-a-days smartphone notification bars
Visibility and/or acknowledgment of notifications may be user and/or access role
specific
May implement some configurable forwarding system (SMS, email, etc.)
Kerstin Weger (WP76)
61
Karabo: The European XFEL software framework
Log Inspection (filtering, configuration of log-levels, …)
Concept: Device’s network appenders provide active logging information which
can be inspected/filtered/exported
Tabular view
Filtering by: full-text, date/time, message type, description
Export logging data to file
Logging events are decoupled from main event loop (threading)
Uses Qt’s model/view with SQLite DB as model (MVC design)
Kerstin Weger (WP76)
62
Karabo: The European XFEL software framework
Online documentation (embedded wiki, bug-tracing, …)
63
Concept: Make the GUI a rich-client having embedded internet access. Use it
for web based device documentation, bug tracking, feature requests, etc.
Any device class will have an individual (standardized) wiki page. Pages are
automatically loaded (within the documentation panel) as soon as any
property/command/device is selected elsewhere in GUI (identical to configuration panel
behavior). Depending on access role, pages are immediately readable/editable.
Device wiki pages are also readable/editable via European XFEL’s document
management system (Alfresco) using standard browsers
For each property/command the coded attributes (e.g. description, units, min/max
values, etc.) is shown.
European XFEL’s bug tracking system will be integrated
Kerstin Weger (WP76)
Karabo: The European XFEL software framework
Software management (coding, building, packaging,
deployment, versioning, …)
Concept: Spiced up NetBeans-based build system, software-bundle approach
Clear splitting of Karabo-Framework (distributed system) from Karabo-Packages
(plugins, extensions)
Karabo-Framework (SVN: karabo/karaboFramework/trunk)
Coding done using NetBeans (for c++ and python), Makefile based
Contains: karabo-library (libkarabo.so), karabo-deviceserver, karabobrokermessagelogger, karabo-gui, and karabo-cli
Karabo-library already contains python bindings (i.e. can be imported into python)
Makefile target “package” creates self-extracting shell-script which can be installed
on a blank (supported) operating system and is immediately functional
Embedded unit-testing, graphically integrated into NetBeans (c++ and python)
Karabo-Packages (SVN: karabo/karaboPackages/category/packageName/trunk)
After installation of Karabo-Framework packages can be build
SVN checkout of a package to any location and immediate make possible
Everything needed to start a full distributed Karabo instance available in package
A tool for package development is provided (templates, auto svn integration, etc.)
Burkhard Heisen (WP76)
64
Karabo: The European XFEL software framework
Software management - Tools
Continuous integration system using
Jenkins (nightly builds on different
platforms)
Jenkins automatically runs all unit-tests
for each build and tests execution of
binaries
Redmine for project management
(features, bugs, releases, versioning
integration)
Installation through software bundle
approach (all dependencies are
shipped), user does not need to compile
nor install any system packages
Deployment system for distributed
device-servers and their plugins
Burkhard Heisen (CAS Group)
65
Karabo: The European XFEL software framework
DETAIL: Software management
The four audiences and their requirements
Framework Developer
Package Developer
Flexible access to the Karabo framework ($HOME/.karabo encodes default location)
Allow "one package - one software" project mode (each device project has its own versioning
cycle, individual Netbeans project)
Standards for in-house development or XFEL developers need to be fullfilled: use parametrized
templates provided, development under Netbeans, use SVN, final code review
Possibility to add further extern dependencies to the Karabo framework (see above)
System Integrator/Tester
SVN interaction, versioning, releases
Code development using Netbeans/Visual Studio
Addition of tests, easy addition of external dependencies
Tools for packaging the software into either binary + header or source bundles
Allow for being framework developer and package developer (see below) in one person at the
same time
Simple installation of Karabo framework and selected Karabo packages as binaries
Start broker, master, i.e. a full distributed system
Flexible setup of device-servers + plugins, allow hot-fixes, sanity checks
XFEL-User/Operator
Easy installation of pre-configured (binary framework + assortment of packages) karabo systems
Run system (GUI, CLI)
Burkhard Heisen (WP76)
66
Karabo: The European XFEL software framework
DETAIL: Software management
Unit-testing
C++
Burkhard Heisen (WP76)
67
Python
Karabo: The European XFEL software framework
DETAIL: Software management
Continuous integration
Continuous Integration is a software development practice where members of a team integrate
their work frequently, usually each person integrates at least daily - leading to multiple integrations per
day. Each integration is verified by an automated build (including test) to detect integration errors as
quickly as possible. [Wikipedia]
Required Features:
Support for different build systems and different OS
Automated builds – nightly builds
Continuous builds – on demand, triggered by SVN commit
Build matrix – different OS, compiler, compiler options
Web interface – configuration, results
Email notification
Build output logging – easy access to output of build errors
Reporting all changes from SVN since last successful build – easy trace of guilty developer
Plugin for any virtualization product (VirtualBox, VMWare, etc.)
Netbeans plugin for build triggering
Easy uploading of build results (installation packages) to web repository
CI systems on the market: Hudson, CruiseControl, buildbot, TeamCity, Jenkins …
Burkhard Heisen (WP76)
68
Karabo: The European XFEL software framework
DETAIL: Software management
Continuous integration
Burkhard Heisen (WP76)
69
Karabo: The European XFEL software framework
Conclusions
The distributed system is device-centric (not attribute-centric), devices inherently express
functionality for communication, configuration and flow control
The provided services focus on solving general problems like data-flow, configuration,
project-tracking, logging, parallelization, visualization, provenance
XFEL.EU software will be designed to allow simple integration of existing
algorithm/packages
The ultimate goal is to provide a homogenous software landscape to allow fast and
simple crosstalk between all computing enabled categories (Control, DAQ, Data
Management and Scientific Computing)
Burkhard Heisen (WP76)
70
Karabo: The European XFEL software framework
71
Thank you for your kind attention.
Burkhard Heisen (WP76)