Transcript Interfacing Interactive
Interfacing Interactive Data Analysis Tools with the Grid: PPDG CS-11 Activity
Doug Olson, LBNL Joseph Perl, SLAC ACAT 2002, Moscow 24 June 2002
Contents
• Background on PPDG, CS-11 • Who is involved • Workshop last week (18,19 June) • Themes that emerged • Near-term goals • Longer term planning • Summary 24 June 2002 D. Olson, PPDG CS-11 for ACAT 2
The 3 US grid projects for HENP are PPDG, GriPhyN, iVDGL 24 June 2002 D. Olson, PPDG CS-11 for ACAT 3
PPDG CS-11
24 June 2002 D. Olson, PPDG CS-11 for ACAT 4
Background
• CS-11 long title: Interfacing and Integrating Interactive Data Analysis Tools with the Grid and Identifying Common Components and Services – Subtitle: Consider physicist sitting at home institution; “What does she need from grid to carry out physics analysis?” • CS-11 is: – Not new funding – New work area within PPDG mission of grid enabling end to-end physics applications for US HENP – Driven by experiments needs, middleware providers want to know if new/different grid services needed 24 June 2002 D. Olson, PPDG CS-11 for ACAT 5
Chip Watson David Liu Doug Olson Joseph Perl Conrad Steenberg Bing Zhu Tony Johnson Eric Hjort Iwona Sakrejda Arie Shoshani John Wu Alex Romosan Junmin Gu Alex Sim Koen Holtman
18,19 June workshop in Berkeley
Purpose: •Review experiments’ requirements •Overview of existing tools & technology •Discuss existing/planned activities •Identify opportunities for cooperative work on defining interfaces and prototype integration of analysis tools with common grid services.
Fons Rademakers Ruth Pordes Kurt Stockinger Jesus Marco Celso Martinez Rivero Massimiliano Turri Mark Donszelmann Maarten Ballintijn Stephen J. Gowdy David Alexander Chris Brock Paul Avery Wensheng Deng Erik Aslakson [email protected]
24 June 2002 D. Olson, PPDG CS-11 for ACAT 6
Review Use Cases for requirements 24 June 2002 D. Olson, PPDG CS-11 for ACAT 7
Review Tools & Technology
24 June 2002 D. Olson, PPDG CS-11 for ACAT 8
Abstract Interfaces for Data Analysis 24 June 2002 D. Olson, PPDG CS-11 for ACAT 9
Java Analysis Studio
24 June 2002 D. Olson, PPDG CS-11 for ACAT 10
PROOF
24 June 2002 D. Olson, PPDG CS-11 for ACAT 11
Clarens
24 June 2002 D. Olson, PPDG CS-11 for ACAT 12
Interactivity in a batched grid environment 24 June 2002 D. Olson, PPDG CS-11 for ACAT 13
MCAT – Metadata Catalog in SRB
24 June 2002 D. Olson, PPDG CS-11 for ACAT 14
SDM Center – bitmap index
24 June 2002 D. Olson, PPDG CS-11 for ACAT 15
Grid Architecture view
24 June 2002 D. Olson, PPDG CS-11 for ACAT 16
EDG testbed
24 June 2002 D. Olson, PPDG CS-11 for ACAT 17
Experiments thoughts, plans, activities • ATLAS – Python interface between Athena framework and grid services • CMS – Grid Analysis Environment (GAE) • Phobos (& ALICE) – PROOF-based analysis • Others (BaBar, Jlab, STAR) at meeting without presentations – Extraction model probably good for BaBar, Jlab – PROOF likely to work for STAR 24 June 2002 D. Olson, PPDG CS-11 for ACAT 18
ATLAS extraction view
24 June 2002 D. Olson, PPDG CS-11 for ACAT 19
24 June 2002 D. Olson, PPDG CS-11 for ACAT 20
CMS Analysis Scope
24 June 2002 D. Olson, PPDG CS-11 for ACAT 21
24 June 2002 D. Olson, PPDG CS-11 for ACAT 22
CMS – Clarens for interconnect (arrows) 24 June 2002 D. Olson, PPDG CS-11 for ACAT 23
PROOF & Grid
24 June 2002 D. Olson, PPDG CS-11 for ACAT 24
ALICE (by proxy)
24 June 2002 D. Olson, PPDG CS-11 for ACAT 25
Themes (or opinions)
• Varying degrees of depth to which grid penetrates interactive analysis: 1. Select data from grid and extract a local (non grid) copy (proceed with interactive analysis independent of grid) 2. Run analysis as grid batch jobs while having intermediate results returned for monitoring 3. Run analysis as grid jobs while having intermediate results returned and have a control channel to jobs to interrupt or guide processing 24 June 2002 D. Olson, PPDG CS-11 for ACAT 26
Sample Requirements
• Ability to select/extract data objects from grid at one level below event (raw, ESD, AOD, … components) – Do not need arbitrarily fine-grained objects from grid (hit, track, …) • User interface/interaction should be same with or without network connection – Similar to web browser cache, I.e., same tool, same URL • Ability to debug grid jobs • Distributed databases (metadata, calibration/conditions, …) • Working single sign-on and VO/group/user authorization • Estimate of time & resources to run an analysis • Laundry list of requirements being developed in use-cases document, not all shown here 24 June 2002 D. Olson, PPDG CS-11 for ACAT 27
Near-term goals
• Interest in common metadata catalog – ATLAS, CMS collaborating on GriPhyN Virtual Data Catalog, others welcome – What about SRB/MCAT, AliEn?
• Considering metadata catalog at event-component level • Considering AIDA, HepRep for results collection, extraction interface • Interest in PROOF-Grid • Interest in JAS-Grid • Finish use cases / requirements document • Example demos for SC2002 24 June 2002 D. Olson, PPDG CS-11 for ACAT 28
Longer-term planning
• Develop detailed workplan for Sept. 9 (US Physics Grid Projects week in San Diego) • Discuss interaction & cooperation with Crossgrid work on interactive analysis • Consider grid interface to PROOF and JAS as good test of common services 24 June 2002 D. Olson, PPDG CS-11 for ACAT 29
Summary
• Just beginning to consider grid for interactive data analysis.
• Aim at interfacing existing tools to grid services • To identify missing services and collaborate on defining/developing common services – HEP-specific metadata catalog – Interactive control/monitor interface?
• Identify a few common projects, possible candidates are: – PROOF + Grid – JAS + Grid – Event component level catalog – … (work in progress) • Want close ties with other grid effort on interactive analysis 24 June 2002 D. Olson, PPDG CS-11 for ACAT 30