Transcript ATLAS/LHCb GANGA DEVELOPMENT A. Soroko (Oxford), K. Harrison (Cambridge) Introduction Requirements Architecture and design Interfacing to the Grid Ganga prototyping.

ATLAS/LHCb GANGA
DEVELOPMENT
A. Soroko (Oxford), K. Harrison (Cambridge)
Introduction
Requirements
Architecture and design
Interfacing to the Grid
Ganga prototyping
Introduction
ATLAS and LHCb develop applications within a
common framework: Gaudi/Athena
Both collaborations aim to exploit potential of Grid for
large-scale, data-intensive distributed computing
Simplify management of analysis and production jobs for
end-user physicists by developing tool for accessing
Grid services with built-in knowledge of how
Gaudi/Athena works:
Gaudi/Athena and Grid Alliance (GANGA)
GridPP meeting, IC
16/09/2002
2
Introduction
GANGA: Gaudi/Athena and Grid Alliance

First ideas for GANGA were presented by P.Mato and C.Tull in summer 2001
An Atlas/LHCb design team is being set up, including US representatives
GUI

GANGA
JobOptions
Algorithms
Histograms
Monitoring
Results
Collective
&
Resource
Grid
Services
GAUDI Program
GridPP meeting, IC
16/09/2002
3
General
requirements for the
GUI
The user will interact with a single application
integrating all stages of job life-time.
He will be able to restore his workspace (list of files,
tools state, jobs in preparation) at the beginning of
each session.
The GUI will be similar to work with for both the Grid
and a local network
The user will be able to access the interface not only
from the computer with the Grid UI program running,
but also from a remote “thin” client.
GridPP meeting, IC
16/09/2002
4
Required functionality
Job preparation and configuration
Resource booking
Job submission

User can choose between Grid and local resource management system
Job monitoring and control

Can we use GRM and PROVE and/or modified Gaudi services?
GUI for the resource browsing




VO active services
CEs
SEs
Query existing files in the Grid
GUI for data management tools



e.g., Dataset registration to the Grid (used by Production Manager)
Copy file from a Computing Element to a Storage Element
Replication of files
GridPP meeting, IC
16/09/2002
5
Required functionality
Job preparation and configuration :


Determine job requirements in terms of software products needed:
executables, libraries, databases, etc.
Get access to the Job Configurations DB



Perform job configuration:






Common configurations could be stored in a database and retrieved using
high-level commands
User would have possibility of modifying settings and storing personalised
configurations in his/her own area
select algorithms to run and set properties
specify input event data
requested output, etc
Provide graphical tools for editing default Job Options files
Contact the Gaudi Bookkeeping Database and the Grid Replica
Catalogue to obtain the list of Logical File Names (LFNs) from highlevel physics selection criteria
Automated generation of JDL scripts for job submission
GridPP meeting, IC
16/09/2002
6
General Requirements
for the Architecture
GridPP meeting, IC
16/09/2002
Multilayered Grid
architecture
GANGA
GUI interface
Simplicity of
Application specific layer (Athena/Gaudi, …)
implementation
GRID middleware (EDG, PPDG,…)
Portability (platform
Underlying GRID services (GLOBUS toolkit)
independence)
OS and Network services
Rich functionality
Modularity, which allows
for Extensibility
Should provide interactivity
7
Design of GANGA
Two ways of implementation have been discussed at Cosener’s House
meeting:

Based on one of the general-purpose grid portals (not tied to a single
application/framework):





Alice Environment (AliEn)
Grid Enabled Web eNvironment for Site-Independent User Job Submission
(GENIUS)
Grid access portal for physics applications (Grappa)
Simulation for LHCb and its Integrated Control Environment (SLICE)
Based on the concept of Python bus (P. Mato):


use different modules whichever are required to provide full functionality of the
interface
use Python to glue this modules, i.e., allow interaction and communication
between them
Different existing implementations have been studied and outcome is
been summarized in the Technical Survey (K. Harrison and A. Soroko)
We have decided that a new development using Python software bus
is better suited to the aims of ATLAS and LHCb
GridPP meeting, IC
16/09/2002
8
Remote
user
(client)
GUI
Python Bus Design
PYTHON SW BUS
XML RPC
module
Local Job
DB
GANGA
Module
OS
Module
Server
XML RPC
server
Job
Configuration
DB
PYTHON SW BUS
PythonROOT
Athena\
GAUDI
LRMS
LAN/WAN
GaudiPython
Production
DB
GRID
Bookkeeping
DB
EDG UI
GridPP meeting, IC
16/09/2002
9
Interfacing to the Grid
GANGA functionality requires use of Grid tools in many cases:
Job preparation and
configuration
Contact RC to get list of LFNs relevant to that job
Contact RB to get list of suitable CE
Resource booking
Contact RB???
Job submission
Contact GSI and/or Myproxy server to get proxy
certificate
Contact RB to submit a job
Job monitoring and
control
Contact LB service to get logging information
Contact RB to get job status
Resource browsing
Contact Globus MDS (meta-computing directory
service) using LDAP search commands
Data management
Contact RB to get job output
Use GridFTP and globus-url-copy for file transfers
Contact RM to register and replicate files
GridPP meeting, IC
16/09/2002
10
Interfacing to the Grid
Job class
Job submission
service
GANGA
kernel module
Security service
Jobs registry
class
Job monitoring
service
Data
management
service
XML RPC
dg-job-list-match
grid-proxy-init
dg-job-status
edg-replica-manager
dg-job-submit
MyProxy ?
dg-job-get-logging-info
dg-job-get-output
dg-job-cancel
GSI ?
GRM/PROVE
globus-url-copy
GDMP?
EDG UI
GridPP meeting, IC
16/09/2002
11
Ganga Prototyping
Tree of user
jobs
Job options
for selected
job
Embedded
Python
interpreter
GridPP meeting, IC
16/09/2002
12
Ganga Prototyping
GUI is created using wxPython extension module
Access to the Gaudi Job Configuration DB is
implemented with the xmlrpclib module
User can browse and create Job Options files using this
DB
Serialization of objects (user jobs) is implemented with
the Python pickle module
Python interpreter is embedded into the GUI and allows
user to configure interface from the command line
GRID stuff is under development at the moment and is
oriented on EDG testbed 1.2
GridPP meeting, IC
16/09/2002
13
Conclusions
We intend to have our prototype interfaced to both production and
analysis Gaudi-aware jobs.
Many details of GANGA architecture are in the experimental stage
and require further approval from the Atlas/LHCb design team (e.g.
XML RPC layer between GANGA client and EDG UI).
Most likely many of GANGA modules will not be presented in the
prototype. For example, possible design of a module responsible for
the HTML-based access to the interface has not been developed yet.
Some questions remain related to the security issues, e.g., could we
have GSI compiled for different platforms ?
In order to implement interfacing to the Grid we need Python API to
the EDG UI. Should we create this API ourselves?
GridPP meeting, IC
16/09/2002
14