Transcript ELSSISuite Services QIZHI ZHANG Argonne National Laboratory on behalf of the TAG developers group ATLAS Software and Computing Week, 4~8 April, 2011

ELSSISuite Services
QIZHI ZHANG
Argonne National Laboratory
on behalf of the TAG developers group
ATLAS Software and Computing Week, 4~8 April, 2011
TAGS
• TAGS are produced as files. These files are uploaded to a
database by the Tier0.
– General (with links to TAG monitoring):
https://atlas-tz-monitoring.cern.ch/
– TAG Task Lister (upload and postTagUpload tasks):
https://atlas-tz-monitoring.cern.ch/tagInterface.html
– TAG Dataset Lister:
https://atlas-tz-monitoring.cern.ch/tagLister.html
• Details tracked in the TASK database
– Data distribution, mapping to datasets, sites, etc.
– Service installations and usages
• Other metadata is also needed for a meaningful event
selection.
– COMA: triggers, periods, streams, etc.
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
BACKBONES—Infrastructure development
Responsible: Florbela Viegas, Elizabeth Gallas and Elisabeth Vinek
•Databases (Oracle)
Routine uploads of TAGs and other metadata info (TASK
and COMA [COditions Meta datA])
•Topology
CERN and several Tier-1 and Tier-2 sites (BNL, TRIUMF,
DESY, PIC, RAL, and …) are hosting different sets (or
subsets) of TAG data with or without overlaps and
replicates of CATALOG and COMA data
•Data Volume and Complexity
Grow/evolve and pose technical challenges on code
flexibility and scalability, affect all the services
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
3
TASK
COMA
Event
Metadata
Collection
Registry
Event-File
Map
Sites
File
Registry
Service
Registry
DDM
TASK
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
Runs
POOLTAG
COnditions
MetadatA
ELSSI Services Suite
• Web Tools
– iELSSI
– RunBrowser (used by iELSSI)
• Web Services
– GUID Lookup (used by Pathena)
– Trigger decoder (used by iELSSI, EventSelector in Athena)
– Token Extraction (used by iELSSI, Ganga)
– Skimming
– Lumirange calculation (used by iELSSI, being factorized)
–…
• https://twiki.cern.ch/twiki/bin/view/Atlas/ELSSIServicesSuite
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
iELSSI--NOT just a browser
1. Does provide browsing functions
 Metadata info on full streams
 Triggers/Runs
 Physics metadata from AOD
 Event flags, e.g. Higgs candidate
2. Performs statistics on these metadata
 Are there enough dileptons with 2 forward jets to do an analysis
requiring that?
 What is the overlap of the muon and egamma streams and what
are the properties of those events?
 What is the data reduction if I want to build a DAOD stream of
only the digamma events that pass triggers this and that?
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
6
iELSSI--NOT just a browser
Also provides an integrated, full-chain service by:
1. interconnecting with multiple DBs and sites (setting
the right parameters for db handshakes)
2. Integrating GRL, JO files with event selection/skim
3. building efficient queries (valid attributes, data
streams, triggers and runs/periods)
4. interfacing with ELSSISuite services
5. building inputs to Athena or DA tools such as Ganga or
Pathena.
6. Submitting grid jobs on selected events to produce
ntuples or skims of AOD, ESD or RAW.
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
7
Trigger decoder service
• Map a trigger name to the appropriate bit in the TAG
trigger words
• Provide the the SQL fragment
(e.g.,operation="bitand(EFPassedTrigMask0,
power(2,1))") for programs using the TAG database
• Clients:
– iELSSI when querying triggers on the database
– EventSelector in Athena--TAG file analyses to decode the
trigger words in the TAG files.
• triggerDecoderUrl:
https://atlas-tagservices.cern.ch/tagservices/triggerDecoder
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
Trigger decoder service
• Usage:
– From a browser:
triggerDecoderUrl/getDecodedTriggerInfo.php?trignms=EF
_e10_loose&runrange=162882&phys_raw=raw&filenam
e_tag=data10_7TeV&ami_tag=f287_m588_p250_p160
– From a command line or within a program:
curl -s -S --key server_hotkey --sslv3 --cert server_hotcert -k
--url
"triggerDecoderUrl/getDecodedTriggerInfo.php?trignms=E
F_e10_loose&runrange=162882&phys_raw=raw&filename
_tag=data10_7TeV&ami_tag=f287_m588_p250_p160"
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
Token Extraction service
• Constructs a ROOT collection of selected events which can then
be retrieved from the /afs space at CERN or directly from the
web.
• Times can vary between seconds and minutes depending on the
size of your selection (can be checked beforehand in iELSSI).
• A link to the results will be emailed to you or it will also appear
in iELSSI when the job completes. The email will contain all the
information on the inputs and outputs of the job which can be
browsed or wget-ed.
– Configuration (in xml)
– Log files
– Root Collection
• At the end of the Extract results, it gives instructions for running
the command line or portable client version of Extract.
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
Token Extraction service invoked by iELSSI
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
Skimming service
• https://twiki.cern.ch/twiki/bin/viewauth/Atlas/SkimmingService#Running_a_Manu
al_Skim
• Skim the events selected in your ROOT collection
• Supply with status updates as it progresses through skim
configuration, submission and completion (or failure).
• Create scripts and send them to the Ganga Service
• The skim does two jobs:
– ‘Tag Prepare’ job finds (using a guid lookup) the datasets the selected events
belong to.
– ‘Tag Run’ job. The chosen AOD Athena JO is then run on the LCG on these
datasets. The Tag run job is submitted to the Grid.
• Email notice with the name of the resulting dataset stored in dq2
(e.g.
user09.thomasdoherty.ganga.users.thomasdoherty.ganga.7.2010041
3/) will be made available with a dq2-get.
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
Skimming service invoked by iELSSI
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
TAGs for analysis?
• TAGs have proven extremely useful in commissioning and in early
collision data
• TAGs discovered and quantified the scope of lumi blocks being split
across multiple files in 2010 reprocessings
– Problematic for cross-section calculation
• TAGs discovered the event duplication issues coming from TDAQ in
early 2011 collision data
• If you have used event picking, you have used TAGs
• TAGs have identified anomalous and/or problematic events
• TAGS have been useful in understanding the overall cross-stream
statistics of our data
• TAGs provide a quick means of checking whether your analysis needs
to process the debug stream
• TAGs for analysis? Needs involvement of PAT group
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
SUMMARY
ELSSISuite services have been developed on
top of an integrated worldwide database
system to provide extended selection
possibilities
You are invited to more TAG related talks at
the DB session this afternoon
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
QIZHI ZHANG
15
Thank you!
Supporting slides or a live iELSSI demo
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
Period-based selection-1
1
2
3
4
5
1
2
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
17
period-based selection-2
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
18
Count per run/per trigger-1
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
19
Count per trigger-2
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
20
Stream mapped with runs
Without run range
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
Perform queries of different project-1
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
Perform queries of different project-2
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
Perform queries of different project-3
D
F
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
E
Synchronized editing-1
B
C
A
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
Synchronized editing-2
C
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
Download GUIDs into a text file-1
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011
Download GUIDs into a text file-2
Qizhi Zhang, Software and Computing Week, 4-8 April, 2011