Transcript Title

Virtual Data Management for
Grid Computing
Michael Wilde
Argonne National Laboratory
[email protected]
Gaurang Mehta, Karan Vahi
Center for Grid Technologies
USC Information Sciences Institute
gmehta,vahi @isi.edu
Outline

The concept of Virtual Data

The Virtual Data System and Language

Tools and approach for Running on the Grid

Pegasus: Grid Workflow Planning

Virtual Data Applications on the Grid

Research issues
3
The Virtual Data Concept
Enhance scientific productivity through:


Discovery and application of datasets and
programs at petabyte scale
Enabling use of a worldwide data grid as a
scientific workstation
Virtual Data enables this approach by creating
datasets from workflow “recipes” and
recording their provenance.
4
Virtual Data Concept

Motivated by next generation data-intensive
applications
– Enormous quantities of data, petabyte-scale
– The need to discover, access, explore, and
analyze diverse distributed data sources
– The need to organize, archive, schedule, and
explain scientific workflows
– The need to share data products and the
resources needed to produce and store them
5
GriPhyN – The Grid Physics Network






NSF-funded IT research project
Supports the concept of Virtual Data, where data is
materialized on demand
Data can exist on some data resource and be
directly accessible
Data can exist only in a form of a recipe
The GriPhyN Virtual Data System can seamlessly
deliver the data to the user or application
regardless of the form in which the data exists
GriPhyN targets applications in high-energy
physics, gravitational-wave physics and astronomy
6
Virtual Data System Capabilities
Producing data from transformations with uniform,
precise data interface descriptions enables…


Discovery: finding and understanding datasets and
transformations
Workflow: structured paradigm for organizing,
locating, specifying, & producing scientific datasets
– Forming new workflow
– Building new workflow from existing patterns
– Managing change

Planning: automated to make the Grid transparent

Audit: explanation and validation via provenance
7
Virtual Data Example:
Galaxy Cluster Search
DAG
Sloan Data
Galaxy cluster
size distribution
100000
Number of Clusters
10000
1000
100
10
1
1
10
Number of Galaxies
100
Jim Annis, Steve Kent, Vijay Sehkri,
Fermilab, Michael Milligan, Yong Zhao,
8
University of Chicago
Virtual Data Application:
High Energy Physics
Data Analysis
mass = 200
decay = bb
mass = 200
mass = 200
decay = ZZ
mass = 200
decay = WW
stability = 3
mass = 200
decay = WW
mass = 200
decay = WW
stability = 1
mass = 200
event = 8
mass = 200
plot = 1
Work and slide by
Rick Cavanaugh and
Dimitri Bourilkov,
University of Florida
mass = 200
decay = WW
stability = 1
LowPt = 20
HighPt = 10000
mass = 200
decay = WW
event = 8
mass = 200
decay = WW
plot = 1
mass = 200
decay = WW
stability = 1
event = 8
mass = 200
decay = WW
stability = 1
plot = 1
9
Lifecycle of Virtual Data
I need to document
the devices and
methods used to
measure the data, and
keep track of the
analysis steps applied
to the data.
I have some subject
images, what analyses
are available? which
can be applied to this
format?
Describe
Discover
VDC
I want to apply an
image registration
program to thousands
of objects. If the
results already exist,
I’ll save weeks of
computation.
Reuse
Validate
I’ve come across some
interesting data, but I
need to understand the
nature of the analyses
applied when it was
constructed before I
can trust it for my
10
purposes.
A Virtual Data Grid
11
Virtual Data Scenario
psearch –t 10 …
file1
file8
simulate –t 10 …
file2
reformat –f fz …
file1
file1
File3,4,5
file7
conv –I esd –o aod
Update
workflow
following
changes
file6
summarize –t 10 …
Manage workflow;
Explain provenance, e.g. for file8:
psearch –t 10 –i file3 file4 file5 –o file8
summarize –t 10 –i file6 –o file7
reformat –f fz –i file2 –o file3 file4 file5
conv –l esd –o aod –i file 2 –o file6
simulate –t 10 –o file1 file2
On-demand
data
generation
12
VDL and Abstract Workflow
a
d1
b
VDL descriptions
b
d2
c
User request data file “c”
a
Abstract Workflow
d1
b
d2
c
13
Example Workflow Reduction

Original abstract workflow
a

b
d1
d2
c
If “b” already exists (as determined by query to
the RLS), the workflow can be reduced
b
d2
c
14
Mapping from abstract to concrete
b

d2
c
Query RLS, MDS, and TC, schedule
computation and data movement
Move b
from A
to B
Execute
d2 at B
Move c
from B
to U
Register
c in the
RLS
15
Workflow management in GriPhyN



Workflow Generation: how do you describe
the workflow (at various levels of
abstraction)? (Virtual Data Language and
catalog “VDC”))
Workflow Mapping/Refinement: how do
you map an abstract workflow
representation to an executable form?
(“Planners”: Pegasus)
Workflow Execution: how to you reliably
execute the workflow? (Condor’s DAGMan)
16
Executable Workflow Construction



VDL tools used to build an abstract workflow
based on VDL descriptions
Planners (e.g. Pegasus) take the abstract
workflow and produce an executable
workflow for the Grid or other environments
Workflow executors (“enactment engines”)
like Condor DAGMan execute the workflow
VDL
VDL Tools
and Catalog
Abstract
Worfklow
Execution
Planner
(e.g. Pegasus)
Concrete
Workflow
Workflow
Executor
(e.g. DAGman)
Jobs
17
Terms

Abstract Workflow (DAX)
– Expressed in terms of logical entities
– Specifies all logical files required to generate the
desired data product from scratch
– Dependencies between the jobs
– Analogous to build style dag

Concrete Workflow
– Expressed in terms of physical entities
– Specifies the location of the data and executables
– Analogous to a make style dag
18
Outline

The concept of Virtual Data

The Virtual Data System and Language

Tools and Approach for Running on the Grid

Pegasus: Grid Workflow Planning

Virtual Data Applications on the Grid

Research issues
19
VDL: Virtual Data Language
Describes Data Transformations

Transformation - “TR”
– Abstract template of program invocation
– Similar to "function definition"

Derivation – “DV”
– “Function call” to a Transformation
– Store past and future:
> A record of how data products were generated
> A recipe of how data products can be generated

Invocation
– Record of a Derivation execution
20
Example Transformation
TR t1( out a2, in a1, none pa = "500", none
env = "100000" ) {
argument = "-p "${pa};
$a1
argument = "-f "${a1};
argument = "-x –y";
t1
argument stdout = ${a2};
profile env.MAXMEM = ${env};
$a2
}
21
Example Derivations
DV d1->t1 (
env="20000", pa="600",
a2=@{out:run1.exp15.T1932.summary},
a1=@{in:run1.exp15.T1932.raw},
);
DV d2->t1 (
a1=@{in:run1.exp16.T1918.raw},
a2=@{out.run1.exp16.T1918.summary}
);
22
Workflow from File Dependencies
TR tr1(in a1, out a2) {
file1
argument stdin = ${a1};
argument stdout = ${a2}; }
x1
TR tr2(in a1, out a2) {
argument stdin = ${a1};
file2
argument stdout = ${a2}; }
DV x1->tr1(a1=@{in:file1}, a2=@{out:file2});
x2
DV x2->tr2(a1=@{in:file2}, a2=@{out:file3});
file3
23
Example Workflow
preprocess

Complex structure
– Fan-in
– Fan-out
findrange
– "left" and "right" can
run in parallel
findrange

Uses input file
– Register with RLS

analyze
Complex file
dependencies
– Glues workflow
24
Workflow step "preprocess"

TR preprocess turns f.a into f.b1 and f.b2
TR preprocess( output b[], input a ) {
argument = "-a top";
argument = " –i "${input:a};
argument = " –o " ${output:b};
}

Makes use of the "list" feature of VDL
– Generates 0..N output files.
– Number file files depend on the caller.
25
Workflow step "findrange"

Turns two inputs into one output
TR findrange( output b, input a1, input a2,
none name="findrange", none p="0.0" ) {
argument = "-a "${name};
argument = " –i " ${a1} " " ${a2};
argument = " –o " ${b};
argument = " –p " ${p};
}

Uses the default argument feature
26
Can also use list[] parameters
TR findrange( output b, input a[],
none name="findrange", none p="0.0" ) {
argument = "-a "${name};
argument = " –i " ${" "|a};
argument = " –o " ${b};
argument = " –p " ${p};
}
27
Workflow step "analyze"

Combines intermediary results
TR analyze( output b, input a[] ) {
argument = "-a bottom";
argument = " –i " ${a};
argument = " –o " ${b};
}
28
Complete VDL workflow

Generate appropriate derivations
DV top->preprocess( b=[ @{out:"f.b1"}, @{
out:"f.b2"} ], a=@{in:"f.a"} );
DV left->findrange( b=@{out:"f.c1"},
a2=@{in:"f.b2"}, a1=@{in:"f.b1"},
name="left", p="0.5" );
DV right->findrange( b=@{out:"f.c2"},
a2=@{in:"f.b2"}, a1=@{in:"f.b1"},
name="right" );
DV bottom->analyze( b=@{out:"f.d"}, a=[
@{in:"f.c1"}, @{in:"f.c2"} );
29
Compound Transformations

Using compound TR
– Permits composition of complex TRs from basic
ones
– Calls are independent
> unless linked through LFN
– A Call is effectively an anonymous derivation
> Late instantiation at workflow generation time
– Permits bundling of repetitive workflows
– Model: Function calls nested within a function
definition
30
Compound Transformations

(cont)
TR diamond bundles black-diamonds
TR diamond( out fd, io fc1, io fc2, io fb1, io fb2, in fa, p1,
p2 ) {
call preprocess( a=${fa}, b=[ ${out:fb1}, ${out:fb2} ]
);
call findrange( a1=${in:fb1}, a2=${in:fb2},
name="LEFT", p=${p1}, b=${out:fc1} );
call findrange( a1=${in:fb1}, a2=${in:fb2},
name="RIGHT", p=${p2}, b=${out:fc2} );
call analyze( a=[ ${in:fc1}, ${in:fc2} ], b=${fd} );
}
31
Compound Transformations

(cont)
Multiple DVs allow easy generator scripts:
DV d1->diamond( fd=@{out:"f.00005"},
fc1=@{io:"f.00004"}, fc2=@{io:"f.00003"},
fb1=@{io:"f.00002"}, fb2=@{io:"f.00001"},
fa=@{io:"f.00000"}, p2="100", p1="0" );
DV d2->diamond( fd=@{out:"f.0000B"},
fc1=@{io:"f.0000A"}, fc2=@{io:"f.00009"},
fb1=@{io:"f.00008"}, fb2=@{io:"f.00007"},
fa=@{io:"f.00006"}, p2="141.42135623731", p1="0" );
...
DV d70->diamond( fd=@{out:"f.001A3"},
fc1=@{io:"f.001A2"}, fc2=@{io:"f.001A1"},
fb1=@{io:"f.001A0"}, fb2=@{io:"f.0019F"},
fa=@{io:"f.0019E"}, p2="800", p1="18" );
32
Functional MRI Analysis
3a.h
3a.i
4a.h
4a.i
ref.h
ref.i
5a.h
5a.i
6a.h
align_warp/1
align_warp/3
align_warp/5
align_warp/7
3a.w
4a.w
5a.w
6a.w
reslice/2
reslice/4
reslice/6
reslice/8
3a.s.h
3a.s.i
4a.s.h
4a.s.i
5a.s.h
5a.s.i
6a.s.h
6a.i
6a.s.i
softmean/9
atlas.h
slicer/10
atlas.i
slicer/12
slicer/14
atlas_x.ppm
atlas_y.ppm
atlas_z.ppm
convert/11
convert/13
convert/15
atlas_x.jpg
atlas_y.jpg
atlas_z.jpg
33
fMRI Example: AIR Tools
TR air::align_warp( in reg_img, in reg_hdr, in sub_img,
in sub_hdr, m, out warp )
{
argument = ${reg_img};
argument = ${sub_img};
argument = ${warp};
argument = "-m " ${m};
argument = "-q";
}
TR air::reslice( in warp, sliced, out sliced_img, out
sliced_hdr )
{
argument = ${warp};
argument = ${sliced};
}
34
fMRI Example: AIR Tools
TR air::warp_n_slice(
in reg_img, in reg_hdr, in sub_img, in sub_hdr, m = "12",
io warp, sliced, out sliced_img, out sliced_hdr )
{
call air::align_warp( reg_img=${reg_img}, reg_hdr=${reg_hdr},
sub_img=${sub_img}, sub_hdr=${sub_hdr},
m=${m},
warp = ${out:warp} );
call air::reslice( warp=${in:warp}, sliced=${sliced},
sliced_img=${sliced_img}, sliced_hdr=${sliced_hdr} );
}
TR air::softmean( in sliced_img[], in sliced_hdr[], arg1 = "y",
arg2 = "null", atlas, out atlas_img, out atlas_hdr )
{
argument = ${atlas};
argument = ${arg1} " " ${arg2};
argument = ${sliced_img};
}
35
fMRI Example: AIR Tools
DV air::i3472_3->air::warp_n_slice(
reg_hdr = @{in:"3472-3_anonymized.hdr"},
reg_img = @{in:"3472-3_anonymized.img"},
sub_hdr = @{in:"3472-3_anonymized.hdr"},
sub_img = @{in:"3472-3_anonymized.img"},
warp = @{io:"3472-3_anonymized.warp"},
sliced = "3472-3_anonymized.sliced",
sliced_hdr = @{out:"3472-3_anonymized.sliced.hdr"},
sliced_img = @{out:"3472-3_anonymized.sliced.img"} );
…
DV air::i3472_6->air::warp_n_slice(
reg_hdr = @{in:"3472-3_anonymized.hdr"},
reg_img = @{in:"3472-3_anonymized.img"},
sub_hdr = @{in:"3472-6_anonymized.hdr"},
sub_img = @{in:"3472-6_anonymized.img"},
warp = @{io:"3472-6_anonymized.warp"},
sliced = "3472-6_anonymized.sliced",
sliced_hdr = @{out:"3472-6_anonymized.sliced.hdr"},
sliced_img = @{out:"3472-6_anonymized.sliced.img"} );
36
fMRI Example: AIR Tools
DV air::a3472_3->air::softmean(
sliced_img = [
@{in:"3472-3_anonymized.sliced.img"},
@{in:"3472-4_anonymized.sliced.img"},
@{in:"3472-5_anonymized.sliced.img"},
@{in:"3472-6_anonymized.sliced.img"} ],
sliced_hdr = [
@{in:"3472-3_anonymized.sliced.hdr"},
@{in:"3472-4_anonymized.sliced.hdr"},
@{in:"3472-5_anonymized.sliced.hdr"},
@{in:"3472-6_anonymized.sliced.hdr"} ],
atlas = "atlas",
atlas_img = @{out:"atlas.img"},
atlas_hdr = @{out:"atlas.hdr"}
);
37
Query Examples
Which TRs can process a "subject" image?

Q: xsearchvdc -q tr_meta dataType subject_image input

A: fMRIDC.AIR::align_warp
Which TRs can create an "ATLAS"?

Q: xsearchvdc -q tr_meta dataType atlas_image output

A: fMRIDC.AIR::softmean
Which TRs have output parameter "warp" and a parameter "options"

Q: xsearchvdc -q tr_para warp output options

A: fMRIDC.AIR::align_warp
Which DVs call TR "slicer"?

Q: xsearchvdc -q tr_dv slicer

A: fMRIDC.FSL::s3472_3_x->fMRIDC.FSL::slicer
fMRIDC.FSL::s3472_3_y->fMRIDC.FSL::slicer
fMRIDC.FSL::s3472_3_z->fMRIDC.FSL::slicer
38
Query Examples
List anonymized subject-images for young subjects. This query
searches for files based on their metadata.

Q: xsearchvdc -q lfn_meta
dataType subject_image
privacy anonymized
subjectType young

A: 3472-4_anonymized.img
For a specific patient image, 3472-3, show all DVs and files that
were derived from this image, directly or indirectly.

Q: xsearchvdc -q lfn_tree
3472-3_anonymized.img

A: 3472-3_anonymized.img
3472-3_anonymized.sliced.hdr
atlas.hdr
atlas.img
…
atlas_z.jpg
3472-3_anonymized.sliced.img
39
Virtual Provenance:
list of derivations and files
<job id="ID000001" namespace="Quarknet.HEPSRCH" name="ECalEnergySum" level="5"
dv-namespace="Quarknet.HEPSRCH" dv-name="run1aesum">
<argument><filename file="run1a.event"/> <filename file="run1a.esm"/></argument>
<uses file="run1a.esm" link="output" dontRegister="false" dontTransfer="false"/>
<uses file="run1a.event" link="input" dontRegister="false“ dontTransfer="false"/>
</job>
...
<job id="ID000014" namespace="Quarknet.HEPSRCH" name="ReconTotalEnergy" level="3"…
<argument><filename file="run1a.mis"/> <filename file="run1a.ecal"/> …
<uses file="run1a.muon" link="input" dontRegister="false" dontTransfer="false"/>
<uses file="run1a.total" link="output" dontRegister="false" dontTransfer="false"/>
<uses file="run1a.ecal" link="input" dontRegister="false" dontTransfer="false"/>
<uses file="run1a.hcal" link="input" dontRegister="false" dontTransfer="false"/>
<uses file="run1a.mis" link="input" dontRegister="false" dontTransfer="false"/>
</job>
<!--list of all files used -->
<filename file="ecal.pct" link="inout"/>
<filename file="electron10GeV.avg" link="inout"/>
<filename file="electron10GeV.sum" link="inout"/>
<filename file="hcal.pct" link="inout"/>
...
(excerpted for display)
40
Virtual Provenance in XML:
control flow graph
<child ref="ID000003">
<child ref="ID000004">
<child ref="ID000005">
<child ref="ID000009">
<child ref="ID000010">
<child ref="ID000012">
<child ref="ID000013">
<child ref="ID000014">
<parent
<parent
<parent
<parent
<parent
<parent
<parent
<parent
<parent
<parent
<parent
<parent
ref="ID000002"/>
</child>
ref="ID000003"/>
</child>
ref="ID000004"/>
ref="ID000001"/>...
ref="ID000008"/>
</child>
ref="ID000009"/>
ref="ID000006"/>...
ref="ID000011"/>
</child>
ref="ID000011"/>
</child>
ref="ID000010"/>
ref="ID000012"/>...
ref="ID000013"/>...</child>…
(excerpted for display…)
41
Invocation Provenance
Completion status and
resource usage
Attributes of executable
transformation
Attributes of input and output
files
42
Future: Provenance & Workflow
for Web Services
43
44
Outline

The concept of Virtual Data

The Virtual Data System and Language

Tools and Issues for Running on the Grid

Pegasus: Grid Workflow Planning

Virtual Data Applications on the Grid

Research issues
45
Outline

Introduction and the GriPhyN project

Chimera

Overview of Grid concepts and tools

Pegasus

Applications using Chimera and Pegasus

Research issues

Exercises
46
Motivation for Grids:
How do we solve problems?

Communities committed to common goals
– Virtual organizations

Teams with heterogeneous members & capabilities

Distributed geographically and politically
– No location/organization possesses all required skills
and resources

Adapt as a function of the situation
– Adjust membership, reallocate responsibilities,
renegotiate resources
47
The Grid Vision
“Resource sharing & coordinated problem solving in
dynamic, multi-institutional virtual organizations”
– On-demand, ubiquitous access to computing, data,
and services
– New capabilities constructed dynamically and
transparently from distributed services
48
The Grid

Emerging computational, networking, and storage
infrastructure
– Pervasive, uniform, and reliable access to remote
data, computational, sensor, and human resources

Enable new approaches to applications and
problem solving
– Remote resources the rule, not the exception

Challenges
–
–
–
–
Heterogeneous components
Component failures common
Different administrative domains
Local policies for security and resource usage
49
Data Grids for High Energy Physics
~PBytes/sec
Online System
~100 MBytes/sec
~20 TIPS
There are 100 “triggers” per second
Each triggered event is ~1 MByte in size
~622 Mbits/sec
or Air Freight (deprecated)
France Regional
Centre
SpecInt95 equivalents
Offline Processor Farm
There is a “bunch crossing” every 25 nsecs.
Tier 1
1 TIPS is approximately 25,000
Tier 0
Germany Regional
Centre
Italy Regional
Centre
~100 MBytes/sec
CERN Computer Centre
FermiLab ~4 TIPS
~622 Mbits/sec
Tier 2
~622 Mbits/sec
Institute
Institute Institute
~0.25TIPS
Physics data cache
Caltech
~1 TIPS
Institute
Tier2 Centre
Tier2 Centre
Tier2 Centre
Tier2 Centre
~1 TIPS ~1 TIPS ~1 TIPS ~1 TIPS
Physicists work on analysis “channels”.
Each institute will have ~10 physicists working on one or more
channels; data for these channels should be cached by the
institute server
~1 MBytes/sec
Tier 4
Physicist workstations
www.griphyn.org
Slide courtesy Harvey Newman, CalTech
www.ppdg.net
50
www.eu-datagrid.org
Grid Applications





Increasing in the level of complexity
Use of individual application components
Reuse of individual intermediate data products
Description of Data Products using Metadata
Attributes
Execution environment is complex and very
dynamic
– Resources come and go
– Data is replicated
– Components can be found at various locations or
staged in on demand

Separation between
– the application description
– the actual execution description
51
Grid3 – The Laboratory
Supported by the National
Science Foundation and the
Department of Energy.
52
Globus Monitoring and Discovery
Service (MDS)
The MDS architecture is a flexible hierarchy. There can be several levels of
GIISes; any GRIS can register with any GIIS; and any GIIS can register
with another, making this approach modular and extensible.
53
GridFTP


Data-intensive grid applications transfer and
replicate large data sets (terabytes, petabytes)
GridFTP Features:
–
–
–
–
–

Third party (client mediated) transfer
Parallel transfers
Striped transfers
TCP buffer optimizations
Grid security
Important feature is separation of control and data
channel
54
A Replica Location Service


A Replica Location Service (RLS) is a distributed
registry service that records the locations of data copies
and allows discovery of replicas
Maintains mappings between logical identifiers and
target names
– Physical targets: Map to exact locations of replicated data
– Logical targets: Map to another layer of logical names,
allowing storage systems to move data without informing
the RLS

RLS was designed and implemented in a collaboration
between the Globus project and the DataGrid project
55
Replica Location Indexes
RLI
LRC
RLI
LRC
LRC
LRC
LRC
Local Replica Catalogs
• LRCs contain consistent information about logical-totarget mappings on a site
•
RLIs nodes aggregate information about LRCs
• Soft state updates from LRCs to RLIs: relaxed
consistency of index information, used to rebuild index
after failures
•
Arbitrary levels of RLI hierarchy
56
Condor’s DAGMan

Developed at UW Madison (Livny)

Executes a concrete workflow

Makes sure the dependencies are followed

Executes the jobs specified in the workflow
– Execution
– Data movement
– Catalog updates

Provides a “rescue DAG” in case of failure
57
Outline

The concept of Virtual Data

The Virtual Data System and Language

Tools and Approach for Running on the Grid

Pegasus: Grid Workflow Planning

Virtual Data Applications on the Grid

Research issues
58
Pegasus Outline
– Introduction
– Pegasus and How it works
– Pegasus Components and Internals
– Deferred Planning
– Pegasus Portal
59
Introduction



Scientific applications in various domains
are compute and data intensive.
Not feasible to run on a single resource
(limits scalability).
Scientific application can be represented as
a workflow.

Use of individual application components.

Reuse data if possible (Virtual Data).
60
Application Development and Execution Process
Abstract
Workflow
Generation
FFT
Application
Component
Selection
ApplicationDomain
Specify a
Different
Workflow
Concrete
Workflow
Generation
FFT filea
Resource Selection
Data Replica Selection
Transformation Instance
Selection
Abstract
Workflow
Pick different Resources
transfer filea from host1://
home/filea
to host2://home/file1
/usr/local/bin/fft /home/file1
DataTransfer
Concrete
Workflow
host1
host2
host2
Retry
Data
Data
Execution
Environment
Failure Recovery
Method
61
Generating an Abstract Workflow
Available Information

–
–
Specification of component
capabilities
Ability to generate the desired data
products
FFT
ApplicationDomain
Select and configure application
components to form an abstract
workflow
–
–
–
assign input files that exist or that can
be generated by other application
components.
specify the order in which the
components must be executed
components and files are referred to by
their logical names
>
>
>
FFT filea
Abstract
Workflow
Logical transformation name
Logical file name
Both transformations and data can be
replicated
62
Generating a Concrete Workflow
Information

–
–
location of files and component
Instances
State of the Grid resources
FFT filea
Abstract
Workflow
Select specific

–
–
–
–
Resources
Files
Add jobs required to form a
concrete workflow that can be
executed in the Grid
environment
Each component in the
abstract workflow is turned
into an executable job
Move filea from host1://
home/filea
to host2://home/file1
/usr/local/bin/fft /home/file1
DataTransfer
Concrete
Workflow
Data Registration
63
Concrete Workflow Generator
Domain
Knowledge
Resource
Information
Location
Information
Plan submitted to the
grid
Concrete Workflow
Generator
Abstract Workflow
64
Pegasus

Pegasus - Planning for Execution on Grids

Planning framework

Take as input an abstract workflow
– Abstract Dag in XML (DAX)


Generates concrete workflow.
Submits the workflow to a workflow
executor (e.g. Condor Dagman).
65
Pegasus Outline
– Introduction
– Pegasus and How it works
– Pegasus Components and Internals
– Deferred Planning
– Pegasus Portal
66
jobmanager
Compute
Resource
Chimera
Information
Pegasus
DAGMan
MDS Index
RLI
LRC
MDS local
GridFTP Server
Storage System
Pool of Resources
Submit
Host
Pool of
Resources
Pool of
Resources
Transformation Catalog
Pool Configuration Info
Properties
Pool of
Resources
jobs
Pool of
Resources
Pool of
Resources
67
Abstract Workflow Reduction
Job a
Job b
Job c
Job f
Job d
Job e
Job g
KEY
The original node
Input transfer node
Job h
Registration node
Output transfer node
Job i
•
•
•
Node deleted by Reduction
algorithm
The output jobs for the Dag are all the leaf nodes i.e. f, h, i.
Each job requires 2 input files and generates 2 output files.
The user specifies the output location
68
Optimizing from the point
of view of Virtual Data
Job c
Job a
Job b
Job f
Job d
KEY
The original node
Job e
Input transfer node
Job g
Job h
Registration node
Output transfer node
Job i
Node deleted by Reduction
algorithm
• Jobs d, e, f have output files that have been found in the
Replica Location Service.
• Additional jobs are deleted.
• All jobs (a, b, c, d, e, f) are removed from the DAG.
69
Planner picks execution
and replica locations
Plans for staging data in
Job c
Job a
Job b
Job f
Job d
adding transfer
nodes for the input
files for the root
nodes
Job e
Job g
Job h
KEY
The original node
Job i
Input transfer node
Registration node
Output transfer node
Node deleted by Reduction
algorithm
70
Staging data out and registering
new derived products
Job c
Job a
Job b
Job f
Job d
Job e
Job g
Staging and
registering for
each job that
materializes
data (g, h, i ).
transferring the
output files of
the leaf job (f) to
the output
location
Job h
Job i
KEY
The original node
Input transfer node
Registration node
Output transfer node
Node deleted by Reduction
algorithm
71
The final, executable DAG
Input DAG
Job a
Job g
Job b
Job h
Job c
Job i
Job f
Job d
Job e
Job g
Job h
KEY
The original node
Input transfer node
Job i
Registration node
Output transfer node
72
Pegasus Outline
– Introduction
– Pegasus and How it works
– Pegasus Components and Internals
– Deferred Planning
– Pegasus Portal
73
Rls-client
Genpoolconfig
client
Tc-client
Replica Query
and Registration
Mechanism
RLS
Transformation
Catalog
Mechanism
(TC)
File
Resource
Information
Catalog
Rls-query-client
Replica
Selection
NonJav
aCallout
Site Selector
Database
MDS
CPlanner
(gencdag)
Min-Min
Random
RLS
Round
Robin
File
Group Grasp
MaxMin
PEGASUS
ENGINE
Stork
Globus-urlcopy
Data
Transfer
Mechanism
Multiple
Transfer
Existing Interfaces
Interfaces in development
Submit Writer
Gridlab
transfer
Transfer2
Production Implementations
Research Implementations
Condor
GridLab
GRMS
Stork
Writer
Pegasus command line clients
Pegasus Components
74
Replica Location Service

Pegasus uses the RLS to find input data
RLI
LRC
LRC
Computation
LRC
Pegasus uses the RLS to register new data products

75
Rls-client
Genpoolconfig
client
Tc-client
Replica Query
and Registration
Mechanism
RLS
Transformation
Catalog
Mechanism
(TC)
File
Resource
Information
Catalog
Rls-query-client
Replica
Selection
NonJav
aCallout
Site Selector
Database
MDS
CPlanner
(gencdag)
Min-Min
Random
RLS
Round
Robin
File
Group Grasp
MaxMin
PEGASUS
ENGINE
Stork
Globus-urlcopy
Data
Transfer
Mechanism
Multiple
Transfer
Existing Interfaces
Interfaces in development
Submit Writer
Gridlab
transfer
Transfer2
Production Implementations
Research Implementations
Condor
GridLab
GRMS
Stork
Writer
Pegasus command line clients
Pegasus Components
76
Transformation Catalog




Transformation Catalog keeps information about domain
transformations (executables).
It keeps information like the logical name of the
transformation, the resource on the which the transformation
is available and the URL for the physical transformation.
The new transformation catalog also allows to define different
types of transformation, Architecture, OS, OS-version and
glibc version of the transformation.
Profiles which allow a user to attach Environment variables,
hints, scheduler related information, job-manager
configurations
77
Transformation Catalog


A set of standard api's are available to perform various
operations on the transformation Catalog.
Various implementations are available for the
transformation catalog like
– Database :
>Useful for large Catalogs and supports fast adds, deletes
and queries.
>Standard database security used.
– File Format :
>Useful for small number of transformations and easy
editing.
> isi vds::fft:1.0 /opt/bin/fft INSTALLED INTEL32::LINUX
ENV::VDS_HOME=/opt/vds

Users can provide their own implementation.
78
Rls-client
Genpoolconfig
client
Tc-client
Replica Query
and Registration
Mechanism
RLS
Transformation
Catalog
Mechanism
(TC)
File
Resource
Information
Catalog
Rls-query-client
Replica
Selection
NonJav
aCallout
Site Selector
Database
MDS
CPlanner
(gencdag)
Min-Min
Random
RLS
Round
Robin
File
Group Grasp
MaxMin
PEGASUS
ENGINE
Stork
Globus-urlcopy
Data
Transfer
Mechanism
Multiple
Transfer
Existing Interfaces
Interfaces in development
Submit Writer
Gridlab
transfer
Transfer2
Production Implementations
Research Implementations
Condor
GridLab
GRMS
Stork
Writer
Pegasus command line clients
Pegasus Components
79
Resource Information Catalog
(Pool Config)


Pool Config is an XML file which contains information
about various pools/sites on which DAGs may
execute.
Some of the information contained in the Pool Config
file is
– Specifies the various job-managers that are available
on the pool for the different types of schedulers.
– Specifies the GridFtp storage servers associated with
each pool.
– Specifies the Local Replica Catalogs where data
residing in the pool has to be cataloged.
– Contains profiles like environment hints which are
common site-wide.
– Contains the working and storage directories to be
used on the pool.
80
Resource Information Catalog
Pool Config

Two Ways to construct the Pool Config File.
– Monitoring and Discovery Service
– Local Pool Config File (Text Based)

Client tool to generate Pool Config File
– The tool gen-poolconfig is used to query the
MDS and/or the local pool config file/s to
generate the XML Pool Config file.
81
Monitoring and Discovery Service

MDS provides up-to-date Grid state information
– Total and idle job queues length on a pool of
resources (condor)
– Total and available memory on the pool
– Disk space on the pools
– Number of jobs running on a job manager

Can be used for resource discovery and selection
– Developing various task to resource mapping
heuristics

Can be used to publish information necessary for
replica selection
– Publish gridftp transfer statistics
– Developing replica selection components
82
Rls-client
Genpoolconfig
client
Tc-client
Replica Query
and Registration
Mechanism
RLS
Transformation
Catalog
Mechanism
(TC)
File
Resource
Information
Catalog
Rls-query-client
Replica
Selection
NonJav
aCallout
Site Selector
Database
MDS
CPlanner
(gencdag)
Min-Min
Random
RLS
Round
Robin
File
Group Grasp
MaxMin
PEGASUS
ENGINE
Stork
Globus-urlcopy
Data
Transfer
Mechanism
Multiple
Transfer
Existing Interfaces
Interfaces in development
Submit Writer
Gridlab
transfer
Transfer2
Production Implementations
Research Implementations
Condor
GridLab
GRMS
Stork
Writer
Pegasus command line clients
Pegasus Components
83
Site Selector



Determines job to site mappings.
Selector Api provides easy pluggable
implementations.
Implementations provided
– Random
– RoundRobin
– Group
– NonJavaCallout

Research on advance algorithms for site selector in
progress
84
Rls-client
Genpoolconfig
client
Tc-client
Replica Query
and Registration
Mechanism
RLS
Transformation
Catalog
Mechanism
(TC)
File
Resource
Information
Catalog
Rls-query-client
Replica
Selection
NonJav
aCallout
Site Selector
Database
MDS
CPlanner
(gencdag)
Min-Min
Random
RLS
Round
Robin
File
Group Grasp
MaxMin
PEGASUS
ENGINE
Stork
Globus-urlcopy
Data
Transfer
Mechanism
Multiple
Transfer
Existing Interfaces
Interfaces in development
Submit Writer
Gridlab
transfer
Transfer2
Production Implementations
Research Implementations
Condor
GridLab
GRMS
Stork
Writer
Pegasus command line clients
Pegasus Components
85
Transfer Tools / Workflow Executors

Various transfer tools are used for staging data between grid sites.

Define strategies to reliably and efficiently transfer data

Current implementations
–
–
–
–


Globus url copy
Transfer
Transfer2 (T2)
Stork
Workflow executors take the concrete workflow and act as a
metascheduler for the Grid.
Current supported workflow executors
– Condor Dagman
– Gridlab GRMS
86
System Configuration - Properties


Properties file define and modify the behavior of Pegasus.
Properties set in the $VDS_HOME/etc/properties can be
overridden by defining them either in $HOME/.chimerarc or by
giving them on the command line of any executable.
– eg. Gendax –Dvds.home=path to vds home……


Some examples follow but for more details please read the
sample.properties file in $VDS_HOME/etc directory.
Basic Required Properties
– vds.home : This is auto set by the clients from the
environment variable $VDS_HOME
– vds.properties : Path to the default properties file
> Default : ${vds.home}/etc/properties
87
Pegasus Clients in the Virtual Data
System

gencdag
– Converts DAX to concrete Dag

genpoolconfig
– Generates a pool config file from MDS

tc-client
– Queries, adds, deletes transformations from the
Transformation Catalog

rls-client, rls-query-client
– Queries, adds, deletes entry from the RLS

partiondax
– Partitions the abstract Dax into smaller dax’s.
88
jobmanager
Compute
Resource
Chimera
Information
Pegasus
DAGMan
MDS Index
RLI
LRC
MDS local
GridFTP Server
Storage System
Pool of Resources
Submit
Host
Pool of
Resources
Pool of
Resources
Transformation Catalog
Pool Configuration Info
Properties
Pool of
Resources
jobs
Pool of
Resources
Pool of
Resources
89
Concrete Planner Gencdag




The Concrete planner takes the DAX produced by
Chimera and converts into a set of condor dag and
submit files.
You can specify more then one execution pools.
Execution will take place on the pools on which the
executable exists. If the executable exists on more
then one pool then the pool on which the executable
will run is selected depending on the site selector
used.
Output pool is the pool where you want all the output
products to be transferred to. If not specified the
materialized data stays on the execution pool
Authentication removes sites which are not accessible.
90
Pegasus Outline
– Introduction
– Pegasus and How it works
– Pegasus Components and Internals
– Deferred Planning
– Pegasus Portal
91
Abstract To Concrete Steps
Abstract
Workflow
MDS
(available
Resources)
Pool config
RLS
(available
data)
Check Resource
Access
Reduce the
Workflow
TC
MDS
Pool config
Perform Site
Selection
Site Selector
RLS
TC
Cluster
Individual Jobs
Add Transfer
Nodes
Fully
Instantiated
Workflow
Write Submit
Files
DAGMan/
Condog-G
file
Replica
Selector
Pegasus’ Logic
92
Original Pegasus configuration
Pegasus(Abstract
Workflow)
Concrete Worfklow
DAGMan(CW))
Original Abstract
Workflow
Original Pegasus
Configuration
Workflow Execution
Simple scheduling: random or round robin
using well-defined scheduling interfaces.
93
Deferred Planning through Partitioning
PW A
PW B
PW C
A Particular Partitioning
New Abstract
Workflow
A variety of partitioning algorithms can be implemented
94
Pegasus(A) = Su(A)
DAGMan(Su(A))
Pegasus(X): Pegasus
generated the concrete
workflow and the submit
files for Partition X -Su(X)
Pegasus(B) = Su(B)
DAGMan(Su(X): DAGMan
executes the concrete
workflow for X
Mega DAG is created
by Pegasus and then
submitted to DAGMan
DAGMan(Su(B))
Pegasus(C) = Su(C)
DAGMan(Su(C))
95
clients
clients
Information Gathering Interfaces
Resource Selection Interfaces
PEGASUS
ENGINE
CPlanner
(gencdag)
Data Transfer
Mechanism
Stork
Partitioner
BFS
Partitioner
Transfer2
Gridlab
transfer
Multiple
Transfer
Single
node
Partition Writer
MEGADAG
Generator
Submit Writer
Condor
Single
pass
Globusurl-copy
GridLab
GRMS
Stork
Writer
Multiple
pass
Mega DAG Pegasus
96
Re-planning
capabilities
Pegasus(X): Pegasus
generated the concrete
workflow and the submit
files for X = Submit(X)
Pegasus’ Log files
record sites considered
Pegasus(A) =
Submit(A)
DAGMan(Submit(A))
Retry Y times
Pegasus(A) =
Submit(A)
DAGMan(Submit(A))
DAGMan(Submit(X)):
DAGMan executes the
concrete workflow for X
Retry Y times
Pegasus(A) =
Submit(A)
DAGMan(Submit(A))
Retry Y times
97
Complex Replanning for Free (almost)
Pegasus(A) = Su(A)
Move f1
to R2
DAGMan(Su(A))
Retry Y times
Execute C at R2
failure
A
f1
f1
B
Move
f3 to R1
Move f2
to R1
Move
f3 to R1
Move f2
to R1
f1
C
Execute D at R1
Execute D at R1
D
f4
Original abstract
workflow partition
Move
f4 to
Output
location
Move
f4 to
Output
location
Pegasus mapping, f2
and f3 were found in
a replica catalog
Workflow submitted
to DAGMan
f1
B
f2
f3
f2
Move
f3 to R1
A
C
f2
D
Execute D at R1
f3
Move
f4 to
Output
location
f4
Pegasus is called
again with original
partition
New mapping, here
assuming R1 was
picked again
98
Planning & Scheduling Granularity

Partitioning
– Allows to set the granularity of planning ahead

Node aggregation
– Allows to combine nodes in the workflow and schedule
them as one unit (minimizes the scheduling overheads)
– May reduce the overheads of making scheduling and
planning decisions

Related but separate concepts
– Small jobs
> High-level of node aggregation
> Large partitions
– Very dynamic system
> Small partitions
99
Partitioned Workflow Processing

Create workflow partitions
– partition the abstract workflow into smaller
workflows.
– create the xml partition graph that lists out the
dependencies between partitions.

Create the MegaDAG (creates the dagman submit
files)
– transform the xml partition graph to it’s
corresponding condor representation.

Submit the MegaDAG
– Each job invokes Pegasus on a partition and then
submits the plan generated back to condor.
100
Future work for Pegasus

Staging in executables on demand

Expanding the scheduling plug-ins


Investigating various partitioning
approaches
Investigating reliability across partitions
101
Pegasus Outline
– Introduction
– Pegasus and How it works
– Pegasus Components and Internals
– Deferred Planning
– Pegasus Portal
102
Portal

Provides proxy based login

Authentication to the sites

Configuration for files

Metadata based workflow creation

Workflow Submission and Monitoring

User Notification
103
104
Simplified View of SC 2004 Portal
Portal Demonstration
105
Outline

The concept of Virtual Data

The Virtual Data System and Language

Tools and Issues for Running on the Grid

Pegasus: Grid Workflow Planning

Virtual Data Applications on the Grid

Research issues
106
Types of Applications

Gravitational Wave Physics

High Energy Physics

Astronomy

Earthquake Science

Computational Biology
107
LIGO Scientific Collaboration



Continuous gravitational waves are expected to be
produced by a variety of celestial objects
Only a small fraction of potential sources are known
Need to perform blind searches, scanning the regions of
the sky where we have no a priori information of the
presence of a source
– Wide area, wide frequency searches

Search is performed for potential sources of continuous
periodic waves near the Galactic Center and the galactic
core

Search for binary inspirals collapsing into black holes.

The search is very compute and data intensive
108
LSC Resources
Caltech Teragrid
Teragrid Resources
WISC
UWM
ANL-Teragrid
Caltech
Other Resources
NCSA-Teragrid
PSC-Teragrid
Cardiff
ISI
SC04
USC
SDSC-Teragrid
UTB
LSU
AEI
Additional resources used: Grid3 iVDGL resources
109
LIGO Acknowledgements







Patrick Brady, Scott Koranda, Duncan Brown, Stephen
Fairhurst University of Wisconsin Milwaukee, USA
Stuart Anderson, Kent Blackburn, Albert Lazzarini, Hari
Pulapaka, Teviet Creighton Caltech, USA
Gabriela Gonzalez, Louisiana State University
Many Others involved in the Testbed
www.ligo.caltech.edu
www.lsc-group.phys.uwm.edu/lscdatagrid/
LIGO Laboratory operates under NSF cooperative agreement
PHY-0107417
110

Montage (NASA and
NVO)
Montage
– Deliver science-grade
custom mosaics on
demand
– Produce mosaics from a
wide range of data
sources (possibly in
different spectra)
– User-specified
parameters of
projection, coordinates,
size, rotation and
spatial sampling.
Mosaic created by Pegasus based Montage from a run of
the M101 galaxy images on the Teragrid.
111
Small Montage Workflow
~1200 nodes
112
Montage Acknowledgments




Bruce Berriman, John Good, Anastasia Laity,
Caltech/IPAC
Joseph C. Jacob, Daniel S. Katz, JPL
http://montage.ipac. caltech.edu/
Testbed for Montage: Condor pools at USC/ISI, UW
Madison, and Teragrid resources at NCSA, Caltech,
and SDSC.
Montage is funded by the National Aeronautics and
Space Administration's Earth Science Technology
Office, Computational Technologies Project, under
Cooperative Agreement Number NCC5-626
between NASA and the California Institute of
Technology.
113
Other Applications
Southern California Earthquake Center
• Southern California Earthquake Center (SCEC), in collaboration with the USC
Information Sciences Institute, San Diego Supercomputer Center, the
Incorporated Research Institutions for Seismology, and the U.S. Geological
Survey, is developing the Southern California Earthquake Center Community
Modeling Environment (SCEC/CME).
•Create fully three-dimensional (3D) simulations of fault-system dynamics.
•Physics-based simulations can potentially provide enormous practical benefits
for assessing and mitigating earthquake risks through Seismic Hazard Analysis
(SHA).
•The SCEC/CME system is an integrated geophysical simulation modeling
framework that automates the process of selecting, configuring, and executing
models of earthquake systems.
Acknowledgments :
Philip Maechling and Vipin Gupta
University Of Southern California
Figure 1: Fréchet sensitivity Kernel showing travel path between a Yorba
Linda earthquake and the TriNet Station DLA.
114
Astronomy

Galaxy Morphology
(National Virtual
Observatory)
– Investigates the dynamical
state of galaxy clusters
– Explores galaxy evolution
inside the context of largescale structure.
– Uses galaxy morphologies
as a probe of the star
formation and stellar
distribution history of the
galaxies inside the clusters.
– Data intensive
computations involving
hundreds of galaxies in a
cluster
The x-ray emission is shown in blue, and the optical mission is in red. The colored dots are located at the
positions of the galaxies within the cluster; the dot color represents the value of the asymmetry index. Blue
dots represent the most asymmetric galaxies and are scattered throughout the image, while orange are the
most symmetric, indicative of elliptical galaxies, are concentrated more toward the center.
People involved: Gurmeet Singh, Mei-Hui Su, many others
115
Biology Applications
Tomography (NIH-funded
project)

Derivation of 3D structure from a
series of 2D electron microscopic
projection images,

Reconstruction and detailed
structural analysis
– complex structures like
synapses
– large structures like dendritic
spines.


Acquisition and generation of
huge amounts of data
Large amount of state-of-the-art
image processing required to
segment structures from
extraneous background.
Dendrite structure to be rendered by
Tomography
Work performed by Mei-Hui Su with Mark Ellisman, Steve Peltier, Abel Lin, Thomas
Molina (SDSC)
116
BLAST: set of sequence comparison algorithms that are used to
search sequence databases for optimal local alignments to a query
2 major runs were performed using
Chimera and Pegasus:
1) 60 genomes (4,000 sequences each),
In 24 hours processed Genomes selected
from DOE-sponsored sequencing
projects
67 CPU-days of processing time
delivered
~ 10,000 Grid jobs
>200,000 BLAST executions
50 GB of data generated
2) 450 genomes processed
Speedup of 5-20 times were achieved
because the compute nodes we used
efficiently by keeping the submission
of the jobs to the compute cluster
constant.
Lead by Veronika Nefedova (ANL) as part of the
Paci Data Quest Expedition program
117
Virtual Data Example:
Galaxy Cluster Search
DAG
Sloan Data
Galaxy cluster
size distribution
100000
Number of Clusters
10000
1000
100
10
1
1
10
Number of Galaxies
100
Jim Annis, Steve Kent, Vijay Sehkri,
Fermilab, Michael Milligan, Yong Zhao,
118
University of Chicago
Cluster Search
Workflow Graph
and Execution Trace
Workflow jobs vs time
119
Capone Grid Interactions
RLS
MonALISA
DonQuijote
ProdDB
MDS
GridCat
Monitoring
Windmill
SE
Capone
gsiftp
CE
WN
gatekeeper
Chimera
VDC
schedd
Pegasus
GridMgr
Condor-G
ATLAS Slides courtesy of Marco Mambelli, U Chicago ATLAS Team
120
ATLAS “Capone” Production Executor
RLS
MDS
GridCat
Monitoring
Reception
MDS
– Job received fromMonitoring
work distributor
RLS
GridCat
 Translation
MonALISA
– Un-marshalling, ATLAS
transformation SE
DonQuijote
 DAX generation
– VDL tools generate abstract DAG
 Input file
retrieval fromCapone
RLS catalog
ProdDB
Windmill
gsiftp
WN
CEPFN)
– Check RLS for input LFNs (retrieval
of
GUID,
gatekeeper
 Scheduling: CE and SE are chosen
Chimera
 Concrete DAG generation and submission
VDC
schedd files
– Pegasus creates Condor submit
Pegasus
– DAGMan invoked to manage
remote steps
GridMgr

MonALISA
DonQuijote
ProdDB
Windmill
SE
Capone
gsiftp
CE
WN
gatekeeper
Chimera
VDC
schedd
Pegasus
GridMgr
Condor-G
Condor-G
121

A job in Capone (2, execution)
Remote job running / status checking
– Stage-in of input files,
create POOL
MDS FileCatalog
Monitoring
– AthenaRLS
(ATLAS code) execution
GridCat

Remote Execution Check
MonALISA
DonQuijote
– Verification
SE
of output files and exit codes
– Recovery of metadata (GUID, MD5sum, exe attributes)
Stage Out: transfer from CE site to destination SE
ProdDB
 OutputWindmill
registration Capone
gsiftp

WN
CEin RLS
– Registration of the output LFN/PFN and metadata

gatekeeper
Finish
Chimera
– Job completed
successfully, communicates to Windmill
that jobs is ready for validation
VDC


schedd
Job status
is sent to Windmill during all the execution
Pegasus
GridMgr
Windmill/DQ validate & register
output in ProdDB
Condor-G
122
CPU-day
Ramp up ATLAS DC2
Mid July
Sep 10
123
Outline

The concept of Virtual Data

The Virtual Data System and Language

Tools and approach for Running on the Grid

Pegasus: Grid Workflow Planning

Virtual Data Applications on the Grid

Research issues
124
Research directions in
Virtual Data Grid Architecture
virtual
data
catalog
discovery
discovery
workflow
executor
(DAGman)
request planner
request executor
(Condor-G,
GRAM)
request
predictor
(Prophesy)
Grid Monitor
Grid
Operations
Data
Transport
storage
element
replica location
service
storage
element
da
ta
detector
storage
element
Data Grid
simulation
g
nin
workflow
planner
w
ra
n
tio
a
riv
de
virtual
data
catalog
simulation data
analysis
Researcher
virtual data
index
Storage
Resource
Mgmt
n
pla
Production
Manager
sharing
composition
Science
Review
virtual
data
catalog
Computing Grid
125
Research issues




Focus on data intensive science
Planning is necessary
Reaction to the environment is a must
(things go wrong, resources come up)
Iterative Workflow Execution:
– Workflow Planner
– Workload Manager

Abstract workflow
Planner
Concrete workflow
Planning decision points
– Workflow Delegation Time (eager)
– Activity Scheduling Time (deferred)
– Resource Availability Time (just in time)



Decision specification level
Reacting to the changing
environment and recovering from failures
How does the communication takes place?
Callbacks, workflow annotations etc…
info
Manager
Tasks
info
Resource Manager
Grid
126
Provenance Hyperlinks
Personal
VDS
DV
DS
TR
DV
DV
TR
DS
DS
TR
TR
Collaboration
VDS
DV
Group VDS
DV
DV
Personal
VDS
127
Goals for a Dataset Model
<FORM
<Title…>
/FORM>
File
Set of files
Relational query or
spreadsheet range
Object closure
XML Element
New user-defined Set of files with
dataset type:
relational index
Speculative model described in CIDR 2003 paper by Foster, Voeckler, Wilde and Zhao
128
Acknowledgements
GriPhyN, iVDGL, and QuarkNet
(in part) are supported by the
National Science Foundation
The Globus Alliance, PPDG, and QuarkNet are
supported in part by the US Department of
Energy, Office of Science; by the NASA
Information Power Grid program; and by IBM
129
Acknowledgements

Argonne and The University of Chicago:
Ian Foster, Jens Voeckler, Yong Zhao, Jin
Soon Chang, Luiz Meyer (UFRJ)
– www.griphyn.org/vds

The USC Information Sciences Institute:
Ewa Deelman, Carl Kesselman, Gurmeet
Singh, Mei-Hui Su
– http://pegasus.isi.edu

Special thanks to Jed Dobson, fMRI Data
Center, Dartmouth College
130
For further information

Chimera and Pegasus:
– www.griphyn.org/chimera
– http://pegasus.isi.edu

Workflow Management research group in
GGF:
– www.isi.edu/~deelman/wfm-rg

Demos on the SC floor
– Argonne National Laboratory and
University of Southern California Booths.
131