Transcript Prezentacja programu PowerPoint - PL-Grid

Scientific Workflow Support in the
PL-Grid Infrastructure with HyperFlow
Bartosz Baliś, Tomasz Bartyński, Kamil Figiela, Maciej Malawski,
Piotr Nowakowski, Maciej Pawlik, Paweł Suder, Bartosz Wilk, Marian Bubak
AGH University of Science and Technology
Department of Computer Science &
ACC CYFRONET AGH
KU KDM 2015, 12-13.03.2015
Motivation & objectives
Scientific workflow = scientific application as a graph of activities (multi-step ‘pipeline’)
Arguably the most widely adopted programming & execution model for scientific
applications
Most workflow systems are oriented on domain scientists
 Simplified programming approach (graphical graph composition)
 Limited programming capabilities
In many cases workflow developers are also skilled programmers
 They want workflow development experience similar to ‘regular’ programming
 They want capabilities of a mainstream programming language / ecosystem
HyperFlow: a lightweight workflow programming and execution environment deployed
as a service in PL-Grid
• Simple high-level workflow description + low-level programming capabilities for
advanced developers
• Skilled programmers can be as productive as in any mainstream programming
language
HyperFlow: overview
Web
Service
Local execution
function getPathWayByGene(ins, outs, config, cb) {
var geneId = ins.geneId.data[0],
url = ...
http({"timeout": 10000, "url": url },
function(error, response, body) {
...
cb(null, outs);
});
}
HyperFlow scientific workflow
service: PL-Grid deployment
App deployment
configuration
User files
Workflow implementation
p1
HyperFlow
enactment engine
p2
Job queue
(RabbitMQ)
p3
p2
Invocation
deployment
hflowc
client
Deployment
planner
Communication
Atmosphere
Service
Redis
Node
VM
VM
VM
p3
p1
VM
VM
App VM instances
Client machine
Data staging in / out
Running a workflow – simple command line client
hflowc run <workflow_dir>
HyperFlow runtime components and application
VMs are started on-demand in the cloud
Input and output files are automatically transferred
from user directory to VMs and back
<workflow_dir> contains:
• File workflow.json (workflow graph
• File workflow.cfg (workflow config)
• Optionally: file functions.js
(advanced workflow activities)
• Input files