Transcript GAT Einf

GAT
Alexander Beck-Ratzka
OFFIS, 23.06.06
23.06.2006
GAT
1
GAT: Agenda
• Introduction
–
–
–
–
Why a new Grid-API?
What GAT offers, what not?
Examples
Architecture
• The single API groups
– File (-stream) management, logical file management,
– AdvertService management,
– Job management
• Adaptors
• Conclusion
23.06.2006
GAT
2
Why a new Grid-API?
GAT as a uniform API for the access to heterogeneous grid
technologies / grid middleware.
GAT is only a framework; the real grid operationen must be done
by adaptors. GAT offers the possibility for accessing adpaptors.
New grid technologies only needs to be coupled to GAT via
adaptors -> No changes in the source sode necessary, even not
If using new grid technologies.
GAT enables a simple access to grid technologies.
GAT uses the grid middleware, which is available at the moment,
where is should be used, and this by only typing a single statement
for GAT_JobSubmit or GATFileCopy...
23.06.2006
GAT
3
What GAT offers? (1)
GAT is not a new grid middleware.
GAT enables an easy to different grid infrastructures.
• GRAM
• Condor
• Unicore
• GridFTP
• ...
GAT is an OpenSource project.
23.06.2006
GAT
5
What GAT offers? (2)
• Applications call the GAT-API for grid operations.
Applications hav to be linked against the GAT.

• Applications are independend of the underlying grid infrastructure.
GAT Engine loads available adaptors at runtime

During a calls of the GAT-API the GAT engine decides, which adaptor
performs the grid operation.


In error case during a „grid operation“, another adaptor will be accessed.
• default adaptors offer local operations
grid applications can be compiled, linked and tested without available grid
services.

The same application can run in a „complete grid environment“: without a
new build.

23.06.2006
GAT
6
What GAT offers? (3)
The GAT-API doesn‘t change. Changes e.g. in Globus job submit
are included in the GAT Globus adaptor.
GAT offers reliability. If one grid service is not available, another
grid service is used.
GAT is much eaysier to install as Globus.
GAT offers grid with a minimum effort for the end user.
23.06.2006
GAT
7
What GAT not offers?
GAT doesn‘t replace any function of a grid middleware.
Without an adaptor for a grid middleware GAT is useless for this
middleware.
GAT has no resource broker.
23.06.2006
GAT
8
Examples (Java-GAT)
File operations
Job submit
23.06.2006
GAT
9
File copy: Java-GAT
import org.gridlab.gat.*;
import org.gridlab.gat.io.File;
public class RemoteCopy
{
public static void main(String[] args) throws Exception
{
GATContext context = new GATContext();
URI src = new URI(args[0]);
URI dest = new URI(args[1]);
File file = GAT.createFile(context, src); // create file object
file.copy(dest);
// and copy it
GAT.end();
}
}
23.06.2006
GAT
10
File Write (Java-GAT)
import java.io.PrintWriter;
import org.gridlab.gat.GAT.*;
import org.gridlab.gat.io.FileOutputStream;
public class FileStreamSimple {
public static void main(String[] args) {
GATContext context = new GATContext();
URI src = null;
FileOutputStream stream = null;
PrintWriter p;
src = new URI(args[0]);
stream = GAT.createFileOutputStream(context, null, src);
p = new PrintWriter(stream);
String toBeStreamed = "hello world\n";
p.println(toBeStreamed);
p.close();
GAT.end();
}
}
23.06.2006
GAT
11
Job Submit (Java-GAT)
public class SubmitJobToHost {
public static void main(String[] args) {
GATContext context = new GATContext();
Preferences prefs = new Preferences();
prefs.put("ResourceBroker.adaptor.name", "globus");
prefs.put("ResourceBroker.jobmanager", "sge");
SoftwareDescription sd = new SoftwareDescription();
sd.setLocation(exe);
sd.setStdout(outFile);
sd.setStderr(errFile);
sd.addPreStagedFile(pre1, pre1Dest);
Hashtable hardwareAttributes = new Hashtable();
hardwareAttributes.put("machine.node", "fs0.das2.cs.vu.nl");
URI exe = null;
File outFile = null;
File errFile = null;
File pre1 = null;
File pre1Dest = null;
ResourceDescription rd = new HardwareResourceDescription(
hardwareAttributes);
JobDescription jd = null;
ResourceBroker broker = null;
try {
exe = new URI("file:////bin/hostname");
try {
jd = new JobDescription(sd, rd);
broker = GAT.createResourceBroker(context, prefs);
} catch (Exception e) {
System.err.println("Could not create Job description: " + e);
System.exit(1);
}
outFile = GAT.createFile(context, prefs, new URI(
"any://fs0.das2.cs.vu.nl/out"));
errFile = GAT.createFile(context, prefs, new URI(
"any://fs0.das2.cs.vu.nl/err"));
pre1 = GAT.createFile(context, prefs, new URI(
"any://fs0.das2.cs.vu.nl//bin/echo"));
Job job = null;
pre1Dest = GAT.createFile(context, prefs, new URI(
"any://fs0.das2.cs.vu.nl//home/rob/my_temp_file"));
} catch (Exception e) {
System.err.println("error: " + e);
System.exit(1);
}
23.06.2006
try {
job = broker.submitJob(jd);
} catch (Exception e) {
System.err.println("submission failed: " + e);
e.printStackTrace();
System.exit(1);
}
GAT
12
GAT architecture
API thin layer; only a frame.
Adaptor implements “grid functionality” as
“ordered” by the API.
Engine mediates betwen API u. adaptor:
the adaptors are called at runtime
error tracing and “Fallback” (default local adaptors)
CPI for Implementation of single adaptors.
23.06.2006
GAT
13
GAT architecture
Application layer
User Space
Application
GAT layer
GAT API
GAT Engine
GAT Adapter
„Grid“ Space
GTK4
PBS
SGE
Globus 2/3.x
Unicore
23.06.2006
DRMAA
GAT
14
Implementation (Engine)
C version fully implemented
C++ wrapper fully implemented
Python wrapper fully implemented
JAVA version fully implemented
23.06.2006
GAT
15
Implementation (Adaptors)
C-GAT
Globus:
• gram, gridftp, RLS, gsiscp, gsissh
Unicore:
• Job Submit, Job Monitoring
DRMAA (Distributed Resource Management Application API)
SGE
(Sun Grid Engine)
PBS
(Portable Batch System)
23.06.2006
GAT
16
Implementation (Adaptors)
Java-GAT
Globus:
• With Java CoG Kit; Globus 3.y complete, GTK 4 up
to now without WebServices.
Unicore:
• In work
SGE
PBS
23.06.2006
GAT
17
Implementation (Adaptors)
Java-GAT and C-GAT
Local adpaptors:
• ssh, scp, ftp, sftp,
• File-Adapter: (cp, mv, read, write, etc...)
• Job-Adapter: fork, exec, auch über ssh...
23.06.2006
GAT
18
File.copy:
schematic run through
File.copy(dest)
FileCPI.copy(dest)
Adapter1 copy
Adapter2 copy
Adapter3 copy
23.06.2006
GAT
19
GAT API overview
File operations
(Monitoring and events)
Resources, jobs
(Information exchange)
(Utility classes: error handling, security,
preferences)
23.06.2006
GAT
20
File sub system
GATFile class
GATObject
GATFile
Create
Destroy
Copy
Move
Delete
IsReadable
IsWritable
GetLength
LastWriteTime
GetLocation
23.06.2006
GAT
22
File sub system
GATFileStream class
GATObject
GATFileStream
Create
Destroy
Read
Write
Seek
23.06.2006
GAT
23
File sub system
GATLogicalFile class
GATObject
GATLogicalFile
Create
Destroy
GetFiles
GetLocations
Remove
RemoveFile
AddFile
AddLocation
Replicate
23.06.2006
GAT
24
Advert package
Goal: Advertisement of important informations
at a central place, e.g. job finished, job rc...
Realized via an SQL adaptor in C-GAT.
23.06.2006
GAT
25
Advert package
GATObject
GATAdvertService
Add
Delete
AddMetadata
GetMetadata
GetAdertisable
Find
SetPWD
GetPWD
23.06.2006
GAT
26
Job management
classes
GATResourceBroker
GATJob
Unschedule
CheckPoint
CloneJob
Migrate
Stop
GetJobDescription
GetState
GetJobID
GetNativeID
ReserveResource
FindResources
SubmitJob
23.06.2006
GAT
27
Job management
available job status
GATJobState value
meaning
GATJobState Unknown
Status not retrievable
GATJobState Initial
At initialization
GATJobState Scheduled
e.g. queued
GATJobState Running
Job is executing
GATJobState Stopped
Job read or stopped
23.06.2006
GAT
28
Job management
structures
GATHardwareResourceDescription
GATSoftwareResourceDescription
Executable
Arguments
Stdin
Stdout
Stderr
Pre-Stage-Files
Post-Stage-Files
Machine type
Memory
CPU-Time
Nodes needed
GATJobDescription
23.06.2006
GAT
29
Job management
example (PBS adaptor)
GATHardwareResourceDescription
machine.queue
jobname
yeo
memory.size
file.size
cpu.walltime
cpu.nodes
23.06.2006
= destination (-q)
= jobname (-N)
= join (-j)
= mem (-l)
= file (-l)
= walltime (-l)
= nodes (-l)
GAT
30
Job management
example (PBS adaptor)
GATSoftwareResourceDescription
Stdin
= stdin (exec < input)
Stdout
= stdout (-o)
Stderr
= stderr (-e)
Executable = executable
Arguments = arglist[]
23.06.2006
GAT
31
Job management
example (PBS adaptor)
Realization in QSUB script
machine.queue
jobname
yeo
memory.size
file.size
cpu.walltime
cpu.nodes
= [email protected]
= TestJob
= eo
= 1G
= 2G
= 12:00:00
=8
stdin
stdout
stderr
executable
arglist[0]
arglist[1]
arglist[2]
= input
= out.testjob
= err.testjob
= /bin/prog
= arg1
= arg2
= lastarg
23.06.2006
#PBS -q [email protected]
#PBS -l walltime=12:00:00, \
-mem=1G,file=2G,[email protected]
#PBS -N testjob
#PBS -o out.testjob
#PBS -e err.testjob
#PBS -j eo
/bin/prog arg1 arg2 lastarg < input
.
.
GAT
32
Job management
lacks
Problem:
Defintions in HardwareResourceDescription
quite arbitrarily.
Solution:
Orientation at the JSDL standard.
23.06.2006
GAT
33
Job management
lacks
Problem:
To less JOB status.
Solution:
Orientation at DRMAA.
23.06.2006
GAT
34
GAT future
C-GAT is now in maintenance mode
Java-GAT is still further developed
SAGA (Simple API for Grid Applications) is the
upcoming GGF standard. First SAGA engine with
GAT wrapper and GTK4 adaptor in autumn 2006
23.06.2006
GAT
35
Usage example
ProC MPA Garching
23.06.2006
GAT
36
SAGA
To be integrated: GAT, Java-CoG, DRMAA, RealityGrid, JSDL,
GridRPC, OSGA-BES, GridCPR, gLite, HDF5
Developed by: GAT, RealityGrid UK Science, OMII Grid UK
Science, CCT Louisana, VU Netherlands, NAREGI Japan,
Globus/CoG, GGF DRMAA, GGF GridRPC
Important: Please remarks needs on adaptors!
23.06.2006
GAT
37
GAT users
C-GAT
Java-GAT
SuperScalar (Univ. of
Barcelona, ESP)
SURA-Grid (63 partners, US)
SCOOP project (LSU + 9
partners, US)
UCOMS project (LSU + 4
partners, US)
Cactus (LSU, US)
Clusterix Project (PSNC,
Univ. of Krakow, PL)
Amolf (Vl-e, NL)
Frank Seinstra (UvA, NL)
Triana group (Cardiff, UK)
23.06.2006
MPA in Garching
ProC-Workflows on the Grid
LSU in Baton Rouge
Chemistry project
AMOLF NL
Fourier Transform Mass
Spectrometry (FTMS) Analyse. FTMS
data are delivered to the Grid with
JavaGAT (ssh, sftp, gridftp).
Multimedian Project NL
Start of parallel Jobs
ZIB Berlin
INRIA Frankreich
GAT
38
Links for GAT
GAT overall:
http://www.gridlab.org/WorkPackages/wp-1/
GAT CVS:
cvs.gridlab.org
cvsroot:
pserver:[email protected]:/cvs/gridlab
Password:
anon
GAT sources:
wp-1/Codes
GAT documents: wp-1/Documents
Only GATEngine: wp-1/Codes/GATEngine
Download tarball:
http://www.gridlab.org/WorkPackages/wp-1/gatreleases.html
http://www.gridlab.org/WorkPackages/wp-1/adaptorreleases.html
German GAT Mailing-Liste: [email protected] (www.listserv.dfn.de)
23.06.2006
GAT
39