Transcript No Slide Title
Pegasus -a framework for planning for execution in grids
Ewa Deelman [email protected]
USC Information Sciences Institute
Outline Pegasus overview Components used by Pegasus Deferred planning Portal Pegasus Acknowledgments: Carl Kesselman, Gaurang Mehta, Mei-Hui Su, Gurmeet Singh, Karan Vahi, Ewa Deelman
Ewa Deelman pegasus.isi.edu
2
Pegasus
Flexible framework, maps abstract workflows onto the Grid Possess well-defined APIs and clients for: – – –
Information gathering > Resource information > Replica query mechanism > Transformation catalog query mechanism Resource selection > Compute site selection > Replica selection Data transfer mechanism
Can support a variety of workflow executors
Ewa Deelman pegasus.isi.edu
3
Rls-client Replica Query and Registration Mechanism RLS Tc-client Transformation Catalog Mechanism (TC) File Database Genpoolconfig client Resource Information Catalog MDS File Replica Selection RLS Site Selector Random Round Robin Grasp Min-Min Prophesy Max-Min CPlanner (gencdag) Stork Gridlab transfer Data Transfer Mechanism Globus url-copy Transfer2 Multiple Transfer Existing Interfaces Interfaces in development Production Implementations Research Implementations
PEGASUS ENGINE
Pegasus command line clients Submit Writer Condor GridLab GRMS Stork Writer
Ewa Deelman 4
Pegasus: A particular configuration
Automatically locates physical locations for both components (transformations) and data – Use Globus RLS and the Transformation Catalog Finds appropriate resources to execute the jobs – Via Globus MDS Reuses existing data products where applicable – Possibly reduces the workflow Publishes newly derived data products – RLS, Chimera virtual data catalog
Ewa Deelman pegasus.isi.edu
5
Replica Location Service
Pegasus uses the RLS to find input data
RLI Computation LRC LRC LRC Pegasus uses the RLS to register new data products
Ewa Deelman pegasus.isi.edu
6
Use of MDS in Pegasus
MDS provides up-to-date Grid state information – – – – Total and idle job queues length on a pool of resources (condor) Total and available memory on the pool Disk space on the pools Number of jobs running on a job manager Can be used for resource discovery and selection – Developing various task to resource mapping heuristics (pluggable) Can be used to publish information necessary for replica selection – Developing replica selection components
Ewa Deelman pegasus.isi.edu
7
Abstract Dag Reduction
On applying the reduction algorithm additional jobs a,b,c are deleted Job a Job b Job c Job f Pegasus Queries the RLS and finds the data products of jobs d,e,f already materialized. Hence deletes those jobs Job d Job e Job g Job i Job h
KEY
The original node Pull transfer node Registration node Push transfer node
Implemented by Karan Vahi pegasus.isi.edu
Ewa Deelman 8
Pegasus adds transfer nodes for transferring the input files for the root nodes of the decomposed dag (job g) Pegasus schedules job g,h on pool X and job i on pool Y. Hence adding an interpool transfer node
Concrete Planner (1)
Job c Job a Job b Job d Job e Job g Job f Job h Pegasus adds replica nodes for each job that materializes data (g, h, i ). Job i These three nodes are for transferring the output files of the leaf job (f) to the output pool, since job f has been deleted by the Reduction Algorithm.
KEY
The original node Pull transfer node Registration node Push transfer node Node deleted by Reduction algo Inter-pool transfer node
Implemented by Karan Vahi 9 Ewa Deelman pegasus.isi.edu
Pegasus Components
Concrete Planner and Submit file generator (gencdag) –
The Concrete Planner of the VDS makes the logical to physical mapping of the DAX taking into account the pool where the jobs are to be executed (execution pool) and the final output location (output pool).
pegasus.isi.edu
10 Ewa Deelman
Pegasus Components (cont’d)
The following catalogs are looked up to make the translation – Transformation Catalog (tc.data) (also DB based) – Pool Config File – – Replica Location Services Monitoring and Discovery Services XML Pool Config generator (genpoolconfig) – The Pool Config generator queries the MDS as well as local pool config files to generate a XML pool config which is used by Pegasus. – MDS is preferred for generation pool configuration as it provides a much richer information about the pool including the queue statistics, available memory etc.
Ewa Deelman pegasus.isi.edu
11
Transformation Catalog
Pegasus needs to access a catalog to determine the pools where it can run a particular piece of code.
If a site does not have the executable, one should be able to ship the executable to the remote site.
– Newer version of Pegasus will prestage a statically linked executable Generic TC API for users to implement their own transformation catalog.
Current Implementations – File Based #poolname logical tr physical tr env isi preprocess /usr/vds/bin/preprocess VDS_HOME=/usr/vds/; – Database Based
Ewa Deelman pegasus.isi.edu
12
Pool Config
Pool Config is an XML file which contains information about various pools on which DAGs may execute.
Some of the information contained in the Pool Config file is – Specifies the various job-managers that are available on the pool for the different types of condor universes.
– – – – Specifies the GridFtp storage servers associated with each pool.
Specifies the Local Replica Catalogs where data residing in the pool has to be cataloged.
Contains profiles like environment hints which are common site-wide.
Contains the working and storage directories to be used on the pool.
Ewa Deelman pegasus.isi.edu
13
Properties
Properties file define and modify the behavior of Pegasus.
Properties set in the $VDS_HOME/properties can be overridden by defining them either in $HOME/.chimerarc or by giving them on the command line of any executable.
– eg. Gendax –Dvds.home=path to vds home…… Some examples follow but for more details please read the sample.properties file in $VDS_HOME/etc directory.
Basic Required Properties – vds.home : This is auto set by the clients from the environment variable $VDS_HOME – vds.properties : Path to the default properties file > Default : ${vds.home}/etc/properties
Ewa Deelman pegasus.isi.edu
16
Concrete Planner Gencdag
The Concrete planner takes the DAX produced by Chimera and converts into a set of condor dag and submit files. Usage : gencdag --dax [--dir
The Output pool is the pool where you want all the output products to be transferred to. If not specified the materialized data stays on the execution pool
Ewa Deelman pegasus.isi.edu
17
Full Ahead Planning
At the time of submission of the workflow, decisions are made as to where to schedule the jobs in the workflow.
Allows to perform certain optimizations by looking ahead for bottleneck jobs and then scheduling around them.
However, for large workflows the decision made at submission time may no longer be valid or optimum at the point the job is actually run.
Ewa Deelman pegasus.isi.edu
18
Deferred Planning
Delay the decision of mapping the job to the site as late as possible. Involves partitioning of the original dax into smaller daxes each of which refers to a partition on which Pegasus is run.
A Mega DAG is constructed. It ends up running Pegasus automatically on the partition daxes, as each partition is ready to run.
Ewa Deelman pegasus.isi.edu
19
Deferred Planning through Partitioning
A Particular Partitioning A variety of planning algorithms can be implemented
Ewa Deelman pegasus.isi.edu
PW A PW B PW C New Abstract Workflow
20
Pegasus(A) = Su(A) DAGMan(Su(A)) Pegasus(X): Pegasus generated the concrete workflow and the submit files for Partition X -- Su(X) DAGMan(Su(X): DAGMan executes the concrete workflow for X Pegasus(B) = Su(B) DAGMan(Su(B))
Mega DAG is created by Pegasus and then submitted to DAGMan
Ewa Deelman
Pegasus(C) = Su(C) DAGMan(Su(C))
pegasus.isi.edu
21
Create workflow partitions – partitiondax --dax ./blackdiamond.dax --dir dax Create the MegaDAG (creates the dagman submit files) – gencdag - Dvds.properties=~/conf/properties - pdax ./dax/blackdiamond.pdax --pools isi_condor --o isi_condor --dir ./dags/
Note the --pdax option instead of the normal --dax option.
submit the .dag file for the mega dag – condor_submit_dag black-diamond_0.dag
Ewa Deelman pegasus.isi.edu
22
More info
www.griphyn.org/chimera pegasus.isi.edu
Ewa Deelman pegasus.isi.edu
23