Transcript LCG Applications Area

LCG Applications Area
13th GridPP Meeting, Durham
4-6 July 2005
Pere Mato/CERN
Outline
Applications area scope and organization
 Requirements and Architecture
 Applications Area projects

–
–
–
–

SPI
ROOT
POOL
Simulation
Summary
06/07/05
LCG Applications Area Summary
2
Application Area Focus
Deliver the common physics applications software
 Organized to ensure focus on real experiment
needs

–
–
–
–
Experiment-driven requirements and monitoring
Architects in management and execution
Open information flow and decision making
Participation of LHC experiment and external
developers
– Frequent releases enabling iterative feedback

Success defined by experiment validation
– Integration, evaluation, successful deployment
06/07/05
LCG Applications Area Summary
3
Domain Decomposition
Simulation Program
Event
Reconstruction Program
Detector
Calibration
Engines
Generators
Framework
Algorithms
Experiment Frameworks
Persistency
DataBase
Batch
FileCatalog
Conditions
Interactive
Simulation
Data Management
Geometry
Histograms
Fitters
MathLibs
I/O
GUI
PluginMgr
Dictionary
Interpreter
Foundation
Utilities
OS binding
06/07/05
Analysis Program
NTuple
Distributed Analysis
Physics
2D Graphics
Collections
LCG Applications Area Summary
3D Graphics
Core
4
Principal Architecture Requirements











Long lifetime: support technology evolution
C++ today; support language evolution
Seamless distributed operation and usability off-network
Component modularity, public interfaces
Interchangeability of implementations
Integration into coherent framework and experiment
software
Design for end-user’s convenience more than the developer’s
Re-use existing implementations
Software quality at least as good as any LHC experiment
Meet performance, quality requirements of trigger/DAQ
software
Platforms: Linux/gcc, MacOSX/gcc, Windows/vc++
06/07/05
LCG Applications Area Summary
5
Applications Area Organization
PEB
SC2
LHCC
Alice
Workplans
Quartery Reports
Atlas
CMS
LHCb
Reviews
Resources
AA Manager
Architects Forum
Chairs
Decisions
Application Area Meeting
LCG AA Projects
PROJECT A
PROJECT B
WP1
WP1
WP2
WP2
PROJECT D
...
WP1
WP2
WP3
External Collaborations
06/07/05
Geant4
LCG Applications Area Summary
Fluka
EGEE
6
Current AA Projects

SPI – Software process infrastructure (A. Aimar)
– Software and development services: external libraries, savannah,
software distribution, support for build, test, QA, etc.

ROOT – Core Libraries and Services (R. Brun)
– Foundation class libraries, math libraries, framework services,
dictionaries, scripting, GUI, graphics, etc.

POOL – Persistency Framework (D. Duellmann)
– Storage manager, file catalogs, event collections, relational access
layer, conditions database, etc.

SIMU - Simulation project (G. Cosmo)
– Simulation framework, physics validation studies, MC event
generators, participation in Geant4, Fluka.
06/07/05
LCG Applications Area Summary
7
Changes in AA for LCG Phase 2
1.
2.
3.
4.
5.
SEAL and ROOT projects merge
Some redefinition of SPI role
Some adaptations of POOL required
PI discontinued and existing libraries absorbed
by client projects
SIMULATION project basically unchanged
06/07/05
LCG Applications Area Summary
8
Rational for the SEAL & ROOT merge

Optimization of resources
– Avoid duplication of developments
Better “coherency” vis-à-vis our clients, the LHC
experiments
 ROOT activity fully integrated in the LCG
organization

– Planning, milestones, reviews, resources, etc.

Ease long-term maintenance and evolution of a
single set of software products
– Thinking on the post-LCG era
06/07/05
LCG Applications Area Summary
9
SEAL and ROOT Merge

Internal AA review in April supported the merge
– “Ensure that the best part of the two projects is taken forward”

Details of the merge are being discussed following a
process defined by the AF
–
–
–
–

Breakdown into a number of topics
Proposals discussed with the experiments
Public presentations
Final decisions by the AF
Current status
– Dictionary plans approved
– MathCore and Vector libraries proposals approved
– First development release of ROOT including these new libraries
06/07/05
LCG Applications Area Summary
10
SEAL + ROOT Migration

Adiabatic changes towards experiments
– Experiments need to see libraries they use currently will evolve
from current usage today towards a unique set

Details are being planned in the AA Phase 2 plan document
– Currently in draft status
SEAL Libraries
SEAL Libraries
newROOT Libraries
ROOT Libraries
ROOT Libraries
2 deliverables
1 deliverable but
some duplication still
now
06/07/05
~August 2005
LCG Applications Area Summary
1 deliverable and
no duplication
time
~January 2006
11
Applications Area Projects
Software Process Infrastructure (SPI)

The AA projects share a single development infrastructure
provided by the SPI project
– Crucial for fostering homogeneity and avoiding duplications
– Users are the LCG AA projects, LHC experiments and others
external projects
– Provides a number of “roles” and “services”

Roles
–
–
–
–
Software librarian
QA and Policies
Testing frameworks
Configuration Management
06/07/05
Services
–
–
–
–
External Software service
Software distribution service
Savannah portal
Documentation and web site
LCG Applications Area Summary
13
Software Development Roles
Maintain distributions
Install and support
distribution tools
Maintain installations
Automate installations
Software
Distribution
Librarian
External
Libraries
Release
Manager
SPI Web
Workbook
Savannah maintenance
Web
services
Provide and maintain
CVS, build, tests
servers
OS, compiler support
Infrastructure support
06/07/05
QA
Manager
Developer
Developer
Developer
Project
LCG Applications Area Summary
Prepare configuration
Build releases on “all” platforms
Run and check “all” tests
Produce documentation
Produce distribution kits
Announce release
Ensures uniformity
Integrate contributions
Build release on few platforms
Run “all” unit tests
Run system and regression tests
Prepare release notes
Document
Design software
Write code
Write unit tests
Run unit tests
Debug
Write Documentation
14
External Libraries service


Install and upgrade, on the
LCG platforms, all external
software needed AA project
and LHC experiments
~80 packages following the
same structure
http://spi.cern.ch/extsoft
– <package>/<version>/<platform>

Platforms
– RH73, SCL3, Win32, OSX,…


500 installations on 100 GB
Ongoing work to automate as
much as possible
06/07/05
LCG Applications Area Summary
15
Savannah Project Portal


Using the open source
Savannah tool to provide
“portals” for all software
projects
Current activities
http://savannah.cern.ch/
– User support
– Functionality enhancements
– Maintenance of the service
and bug fixes
– Work/Merge with Savannah
open source
06/07/05
LCG Applications Area Summary
16
Software Distribution Service


Internal (AA) and external software
Handling the complex use case and requirements from
users
–
–
–
–
–

Binary distributions (supported/compatible platforms)
Source distributions (non-supported platforms)
Distributions for developers
Run-time distributions (batch farms)
Remote central installations
Towards a single source of information concerning
dependencies and configuration
– Tool neutral XML configuration file

Starting to use Pacman
– Pacman caches (binary and source) are becoming available
06/07/05
LCG Applications Area Summary
17
Testing and Quality Assurance


Testing is an integral part of the software
development process in the AA
Testing frameworks
– CppUnit, QmTest

Test support
– Policies, Tools, Documentation

Quality Assurance
– Codewizard for EGEE
– QA reports
– Test coverage
06/07/05
LCG Applications Area Summary
18
Core libraries and services (ROOT)


Provides basic functionality needed by any application
Evolution of the current ROOT+SEAL projects
– Adiabatic changes towards experiments

Current work packages
– BASE: Foundation and system classes, documentation and releases
– DICT: Reflexion system, meta classes, CINT and Python
interpreters
– I/O: Basic I/O, Trees, queries
– PROOF: parallel ROOT facility, xrootd
– MATH: Mathematical libraries, histogramming, fitting
– GUI: Graphical Uner interfaces and Object editors
– GRAPHICS: 2-D and 3-D graphics
– GEOM: Geometry system
06/07/05
LCG Applications Area Summary
19
Dictionary

Adding reflexion/introspection capabilities to C++
– Essential for I/O, distribution, interpreters, GUI, etc.

Towards a “single” reflexion system
– Updating ROOT/CINT to use the reflexion system Reflex (SEAL)
– Dictionary generation from C++ class descriptions (.h files)

CINT: C++ interpreter
– Reengineering the interpreter to handle uniformly byte code
generating and interpretation

PyROOT: interface of any C++ class to Python interpreter
– Generic Python binding to any C++ class having a dictionary
– Completing the C++ language constructs (namespace, templates, etc.)
06/07/05
LCG Applications Area Summary
20
Math Libraries

Development of a common and complete set of Mathematical
functionality
– Available for all programs (from interactive analysis to real-time software
triggers)

Developing the MathCore library (from the SEAL project)

MathMore library: larger set of utilities and functions

Linear Algebra

Minimization and Fitting
– Mathematical functions, utilities, numerical algorithms, physics vectors and
random number generators
– Standalone library, license free.
– Wrappers on top of GSL (Gnu Scientific Library)
– Evaluation of alternative solutions, recommendations.
– Re-engineering the current design in ROOT
– New C++ Minuit
– Incorporating RooFit package from BaBar
06/07/05
LCG Applications Area Summary
21
I/O




Development of the core I/O system, ROOT TTree and
TTree queries
Functionality developed in SEAL and POOL being
integrated
New optimizations in speed
New C++ constructs
– Better STL support, virtual inheritance, typedefs, etc.

Support for bitmap indices (improved selection
performance)
06/07/05
LCG Applications Area Summary
22
PROOF

The work on the Parallel ROOT Facility (PROOF) is
accelerating with several new developments
– From short blocking queries to long asynchronous queries in a
stateless client mode.
– Later reconnection of the client from a different session possible.
– Reusing the xrootd file server infrastructure.
– “zero-config” cluster setup by using the Apple Bonjour protocol
– Improving usability and friendless

Extensions to be able to run PROOF on the Grid
– Interfacing to the middleware to access file catalogs, job
schedulers and storage elements

Aiming for a demo with all the new features by September
06/07/05
LCG Applications Area Summary
23
GUI and Graphics

ROOT provides several implementations of the abstract interface
TVirtualX used for the GUI and 2-D graphics
– X11, native Win32, TrollTech Qt

Set of high level widgets
– canvas manager, object browser, TTree viewer


Object editors and GUI designer and code generator
2D Graphics
– Basic graphics like: lines, text, polygons, etc.
– Generation of various output formats: ps, eps, pdf, gif, jpeg, bmp
– Image processing classes

3D Graphics
– Abstract interface to a 3-D viewer
– Implementations: X3D and powerful Open GL taking advantage of modern
graphics cards
06/07/05
LCG Applications Area Summary
24
Persistency Frameworks (POOL)

The LCG persistency framework project consists of two parts
– Common project with CERN IT and strong experiment involvement

POOL
– Hybrid object persistency integration object streaming (using ROOT I/O
for event data) with Relational Database technology (for meta data)
– Established baseline for three LHC experiments
– Successfully integrated into the software frameworks of ATLAS, CMS
and LHCb
– Successfully deployed in three large scale data challenges

Conditions Database (COOL)
– Store, manage and distribute time varying data (detector conditions)
– Conditions DB was moved into the scope of the LCG project
» To consolidate different independent developments
and integrate with other LCG components (SEAL, POOL)
– Storage of complex objects via POOL into Root I/O and RDBMS backend
06/07/05
LCG Applications Area Summary
25
POOL Components

Storage Manager
– Streams transient C++ objects to/from
disk
– Resolves a logical object reference to a
physical object

POOL API
File Catalog
Storage
– Maintains consistent lists of accessible
FileCatalog
Collections
Service
files together with their unique
identifiers (FileID), which appear in the
XML
Explicit
object representation in the persistent ROOT I/O
Collection
Storage Svc
Catalog
space
– Resolves a logical file reference (FileID)
to a physical file
Relational
MySQL
Implicit
Collection
Storage Svc
Catalog
 Collections
– Provides the tools to manage potentially
large sets of objects stored via POOL
Grid Replica
Catalog
06/07/05
LCG Applications Area Summary
26
POOL deployment in the Grid

Coupling to Grid services

But Grid-decoupled modes also required by production use-cases
– In 2004 based on the EDG-RLS service using Oracle Application Server + DB
» Connects POOL to all LCG files
» Local Replica Catalog (LRC) for GUID <-> PFN mapping for all local files
» Replica Metadata Catalog (RMC) for file level meta-data and GUID <-> LFN
» Replica Location Index (RLI) to find files at remote sites (not deployed in LCG)
» Resulted in a single centralized catalog (scalability and availability concerns)
– New file catalog implementations released recently (version 2.1) implementing of the
POOL interface
» LFCCatalog (lfc), GliteCatalog (glite, Fireman), GTCatalog (globus toolkit)
– XML based Catalog
» typically used as local file by a single user/process at a time
» no need for network
» supports R/O operations via http; tested up to 50K entries
– Native MySQL Catalog
» Shared catalog e.g. in a production LAN
» handles multiple users and jobs (multi-threaded); tested up to 1M entries
06/07/05
LCG Applications Area Summary
27
Relational Abstraction Layer (RAL)

C++ API for SQL-free, technology
neutral access to relational data
– Inserting, deleting, updating and
retrieving rows



Support for bulk operations,
client side cashing and SQL
variable binding
RAL enforces “best practices” in
database programming
Strong links to providers of
database services

Available backends
–
–
–
–
Oracle
SQLite
MySQL
Authentication
services
OracleAccess
RelationalAccess
SQLiteAccess
ODBCAccess
AttributeList
AuthenticationService
06/07/05
LCG Applications Area Summary
28
Conditions Objects for LCG (COOL)




Conditions data: non-event data that vary with time and
exists in several versions
Produced from both online (slow control) and offline
(calibrations)
COOL implementation based on Oracle/MySQL using RAL
Payload defined
by end-users
– AtributeList
– External reference
(POOL token)

Production release
– Version 1.2
06/07/05
LCG Applications Area Summary
29
Simulation Project

Simulation framework
– Interface to multiple simulation engines (Geant4, Fluka) and geometry
models exchange

Geant4 team participating
– Aligned with and responding to needs from LHC experiments, physics
validation, simulation framework

Fluka team participating
– Framework integration, physics validation

Garfield team participating
– Garfield package integration and support in LCG - SPI

Simulation physics validation
– Assess adequacy of simulation and physics environment for LHC and
provide the feedback to drive needed improvements

Generator services
– MC generator libraries; common event files; validation/test suite;
development when needed (HepMC, etc.)
06/07/05
LCG Applications Area Summary
30
Simulation Framework


Provide flexible infrastructure for the development,
validation and usage of Monte Carlo simulation applications
Work packages
– GDML: Geometry description markup language
» GDML writer and readers exists for Geant4 and ROOT
– Geant4 Geometry persistency
» Saving/retrieving Geant4 geometries with ROOT I/O
– FLUGG: Calling Geant4 geometry from FLUKA
» example application exists (ATLAS Pixel)
– Python interface to Geant4
» Provide Python bindings to G4 classes (SEAL PyLCGDict)
» Steering Geant4 applications from Python scripts
– Monte Carlo truth handling
06/07/05
LCG Applications Area Summary
31
Physics Validation

Compare detector simulation engines (Geant4, FLUKA) with
experimental data to understand suitability for the LHC
experiments
– Collaboration and coordination between physics groups of LHC
experiments and developers of simulation codes

Ongoing work
– Re-visit requirements for the simulation packages by evaluating the
impact of simulation uncertainties on physics observables
– Validation of electromagnetic physics using test beam data and
simple benchmarks (thin targets setups)
– Validation of hadronic physics (calorimetry, inner detectors,
background radiation)
06/07/05
LCG Applications Area Summary
32
Geant4 and Fluka

Contribution to the support/maintenance and development
of Geant4 to server the needs of the LHC experiments
– Leading role in the development/maintenance of the geometry,
field and transportation module
– Release management and creation of release distributions
– Participation in the development of electromagnetic physics
packages
– Participation in the development of hadronic physics packages
– System integration, development of an acceptance test suit, etc.

The Fluka team participates as externals to the simulation
project
– Very beneficial for the Physics Validation and Framework subprojects
06/07/05
LCG Applications Area Summary
33
Garfield

Garfield is a specialized program for simulation of gaseous
detectors
– E.g. two- and three-dimensional wire chambers, TPCs, muti-wire
counters, etc.

Ongoing work
– Interfacing to Maxwell and FEMLAB packages for electric field
maps
– Diffusion modeling for strongly converging and diverging fields
– Electron transport properties in arbitrary gas mixtures obtained
with the Magbolz program
– Ionization simulation using the Heed program
06/07/05
LCG Applications Area Summary
34
Generator Services



The goal is to guarantee the MC generator support for the
LHC experiments
The project collaborates with the MC generator authors to
provide validated code for the theoretical and experimental
communities at LHC
On going work
– Generator library (GENSER). Central code repository and
installations for most popular MC generators
– Contribution to the definition of standards for generator interfaces
– Database of “certified” MC event files to be used for benchmarks,
comparisons and combinations
– Functional validation of MC generators
06/07/05
LCG Applications Area Summary
35
Concluding
AA Validation Highlights

POOL successfully used in large scale production in ATLAS, CMS,
LHCb data challenges in 2004
– ~400TB of POOL data produced
– Objective of a quickly-developed persistency hybrid leveraging ROOT I/O
and RDBMSes has been fulfilled

Geant4 firmly established as baseline simulation in successful ATLAS,
CMS, LHCb production
– EM & hadronic physics validated
– Highly stable: 1 G4-related crash per O(1M) events

SEAL components underpin POOL’s success, in particular the
dictionary system
– Now entering a second generation with Reflex

SPI’s Savannah project portal and external software service are
accepted standards inside and outside the project
06/07/05
LCG Applications Area Summary
37
LCG Phase 2 Planning

Started to plan second phase of Applications Area
– The major change for this new phase is the merge of
ROOT and SEAL projects
Internal AA Review: “Evolution plan technically
reasonable and supported by all experiments”
 Technical details of the plan are being discussed
(one topic at the time) by the projects and
experiments and approved by Architects Forum
 The planning document will be finished during this
month

06/07/05
LCG Applications Area Summary
38
Conclusion
AA is consolidating of a number of key products
 Establishing the level of long-term support that is
required for the products that are essential for
the experiments

– Minimizing duplication
– Re-using software and infrastructure across projects
– Easing maintenance of AA software at the end of the
LCG
 Stressing
development in the area of Physics
Analysis (local, distributed, etc.)
06/07/05
LCG Applications Area Summary
39