SimCity Building Blocks at the DICE team

Marian Bubak, Bartosz Baliś, Marek Kasztelnik, Maciej Malawski, Piotr Nowakowski

{bubak,balis, malawski}@agh.edu.pl

, {m.kasztelnik, p.nowakowski}@cyfronet.pl

Department of Computer Science and ACC Cyfronet AGH Krakow, PL dice.cyfronet.pl

Example application: Flood threat assessment

• •

Scenario: levee-protected area endangered by flood due to high water levels The user selects an area for flood threat assessment

Flood threat assessment: our solution

Data sets

• • • • •

GIS data (levees, sensor locations) Sensor data Simulated sensor data Computation results Lots of metadata

(Virtual) Experiments

• • • Support for conducting virtual experiments: Flooding a reservoir protected by an artificial levee Observation of a levee during heavy rain Simulation of a flooding scenario • • • • Experiment lifecycle Creating a new experiment and defining its context Collecting information during the experiment Concluding the experiment Reusing experiment results in future experiments

HyperFlow: programming and execution of workflow-based scientific applications • • • • • Innovative programming approach and enactment engine for scientific workflows Combines declarative workflow description with low-level programming in JavaScript / node.js

for implementing workflow activities Simple and concise syntax + mainstream scripting language & runtime platform = increased programming productivity Based on a formal model of computation (Process Networks) Supports a rich set of complex workflow patterns Input sequence

6 5 4 3 2 1

num Process Sqr

sqr(x)

«dataflow» Consumes three numbers at a time

36 25 16 9 4 1

square :3 Output sequence Sum

sum(x 1 ..x

n )

«dataflow»

77 14

sum {

"processes" : [ { "name" : "Sqr" , "function" : "sqr" , "type" : "dataflow" , "parlevel" : 0 , // level of parallelism (unlimited) "ordering" : true , // ordering of results "ins" : [ "number" ], "outs" : [ "square" ] }, { "name" : "Sum" , "function" : "sum" , "ins" : [ "square:3" ], "outs" : [ "sum" ] } ],

}

"signals" : [ { "name" : "number" , "data" : [ 1 , 2 , 3 , 4 , 5 , 6 ] }, { "name" : "square" }, { "name" : "sum" } ], "ins" : [ "number" ], "outs" : [ "sum" ] } }

function sqr(ins, outs, config, cb) { var n = Number cb(null, outs); var sum= outs[ 0 0.0

; cb(null, outs); (ins.number.data[ outs.square.data = [n * n]; function sum(ins, outs, config, cb) { ].data = [ sum ]; 0 ]); ins.square.data.forEach(function (n) { sum += n; }); B. Baliś, Increasing Scientific Workflow Programming Productivity with HyperFlow. In Proceedings of the 9th Workshop on Workflows in Support of Large-Scale Science. 2015 (In Print).

SimCity Requirements for “CIS”

Start a simulation based on user input.

Let an automated component start workflows within CIS with new parameter sets, and receive results asynchronously.

All functions should be accessible from a user friendly interface that is not concerned with how something is computed but with what is computed.

SimCity Requirements for “CIS”

Start an external component (for parameter exploration) via an external (custom) user interface.

Ideally, parameter exploration is exposed as a web service, so different algorithms can be started from a custom web interface.

The parameter exploration algorithm should be modifiable or at least selectable by the user.

SimCity Requirements for “CIS”

Stage data to and from clusters or cloud infrastructure (virtual machines).

For each job, the input and output files should end up at the correct places.

Construct input files based on parameter sets.

TRANSIMS, for example, works with control files to determine the parameters, but also with data files with parameters. Both would have to be editable from CIS.

SimCity Requirements for “CIS”

Automatically schedule workflows on cluster or cloud infrastructure.

Preferably, the user should be able to select their own cluster.

SimCity Requirements for “CIS”

Provide feedback to a component based on user input.

Parameter exploration may be guided by the user. In this case, ideally, a web service is provided for real-time interaction with the parameter exploration component.

CIS gives feedback to a component based on sensor data, which may then start new workflows.

Hybrid cloud as a means of provisioning computing power for virtual experiments – the Atmosphere framework GUI host (provisions end-user features and access options) Cloud Management Portlets Provide GUI elements which enable service developers and end users to interact with the Atmosphere platform and create/deploy services on the available cloud resources Head Node Image store Worker Node Worker Node Worker Node Worker Node Worker Node Worker Node Worker Node Worker Node OpenStack cloud site at ACC CYFRONET AGH 96 CPU cores 184 GB RAM 4 TB storage private IP space Secure RESTful API (Cloud Facade) Atmosphere Core Services Host Atmosphere Core • • • • • • Authentication and authorization logic Communication with underlying computational clouds Launching and monitoring service instances Creating new service templates Billing and accounting Logging and administrative services Head Node Worker node w/large resource pool („fat node”) Image store Worker node w/large resource pool („fat node”) VPH-Share cloud site at UNIVIE 128 CPU cores 256 GB RAM 4 TB storage private IP space Atmosphere Registry (AIR) user accounts available cloud sites services and templates API host Worker Node Massive (functionally limitless) hardware resource pool Image store Worker Node public IP space Amazon Elastic Compute Cloud (EC2) – European availability zone

Atmosphere platform interfaces End user A full range of user-friendly GUIs is provided to enable service creation, instantiation and access. A comprehensive online user guide is also available.

The GUIs work by invoking a secure RESTful API which is exposed by the Atmosphere host. We refer to this API as the Cloud Facade.

Atmosphere Ruby on Rails controller layer (core Atmosphere logic) Application -- or - Any operation which can be performed using the GUI may also be invoked programmatically by tools acting on behalf of the platform user – this includes standalone applications and workflow management environments (which VPH-Share also provides).

Workflow environment Atmosphere Registry (AIR) Cloud sites All operations on cloud hardware are abstracted by the Atmosphere platform which exposes a unified RESTful API (with a suitable set of developer’s documentation available). For end users, the API is concealed by a layer of platform GUIs embedded in the VPH-Share portal and providing a user-friendly work environment - for domain scientists and service developers alike.

The API can also be directly invoked by external services as long as they possess the required security credentials (Atmosphere relies on the well-known OpenID authentication standard).

Shared and scalable services – smart utilization of hardware resources Atmosphere Cloud Platform • • Published services become visible to non-developers and can be instantiated using the Generic Invoker.

Developers are free to spawn „snapshot” images of their cloud services (e.g. for backup purposes) without exposing them to external users.

Developer Cloud Service Published Scientist • • A Shared service is backended by a single virtual machine which „mimics” multiple instances from the users’ point of view.

Shared services greatly conserve hardware resources and can be instantiated quickly.

Atmosphere Cloud Service Shared Cloud WN Shared VM Scientist Scientist Scientist Scientist • • When a Scalable service is overloaded with requests Atmosphere can spawn additional instances in the cloud to handle the additional load.

The process is transparent from the user’s perspective.

Atmosphere Cloud Service Scalable Cloud WN Separate VM Cloud WN Separate VM Scientist

More information about the hybrid computational cloud platform

• • A more detailed introduction to the Atmosphere cloud platform (including user manuals) can be found at https://vph.cyfronet.pl/tutorial The DIstributed Computing Environments (DICE) team homepage at http://dice.cyfronet.pl

cloud resource provisioning has information on projects which use Atmosphere for

Cost optimization of applications on clouds

• • • • • •

Task

Infrastructure model – Multiple compute and storage clouds – Heterogeneous instance types Layer 1 Layer 2 Application model – Bag of tasks – Multi-level workflows Layer 3 Layer 4 Modeling with AMPL (A Modeling Language for Mathematical Programming) and CMPL 3000 2500 Layer 5 Cost optimization under deadline constraints 2000 Mixed integer programming Bonmin, Cplex solvers 1500 1000 500 A B B B D E F C Multiple providers 1h 2.5 h 0.5 h 0.3 h 2 h Amazon's and private instances 6 h Input Output Application Storage m1.small

m1.large

t1.micro

m2.xlarge

Compute

Amazon

20000 tasks, 512 MiB input and 512 MiB output, task execution time 0.1h @ 1ccu machine Amazon S3 Rackspace Cloud Files Optimal Rackspace and private instances Rackspace instances private Compute

Private cloud

rs.1gb

rs.2gb

rs.4gb

rs.16gb

Compute

Rackspace

Storage 0 0 10 20 30 40 50 Time limit (hours) 60 70 80 90 100 M. Malawski, K. Figiela, J. Nabrzyski: Cost minimization for computational applications on hybrid cloud infrastructures, Future Generation Computer Systems, Volume 29, Issue 7, September 2013, Pages 1786-1794, ISSN 0167-739X, Workflows on Clouds. PPAM (1) 2013: 251-260 https://github.com/kfigiela/optimization-models http://dx.doi.org/10.1016/j.future.2013.01.004

Maciej Malawski, Kamil Figiela, Marian Bubak, Ewa Deelman, Jarek Nabrzyski: Cost Optimization of Execution of Multi-level Deadline-Constrained Scientific http://dx.doi.org/10.1007/978-3-642-55224-3_24

• • • •

Simulation and scheduling of large-scale scientific workflows on IaaS clouds

Large-scale scientific workflows from Pegasus workflow management system Workflows of 100,000 tasks Workflow ensembles: schedule as many workflows as possible within a budget and deadline Cloud infrastructure simulated using CloudSim M. Malawski, G. Juve, E. Deelman, J. Nabrzyski: Cost- and deadline-constrained provisioning for scientific workflow ensembles in IaaS clouds. SC 2012: 22 https://github.com/malawski/cloudworkflowsimulator Time 17

Cloud performance evaluation

• • • • • Performance of VM deployment times Virtualization overhead Evaluation of open source cloud stacks (Eucalyptus, OpenNebula, OpenStack) Survey of European public cloud providers Performance evaluation of top cloud providers (EC2, RackSpace, SoftLayer) A grant from Amazon has been obtained

IaaS Provider EEA Zoning jClouds API Support BLOB storage support Per hour instance billing API Access Published price VM Image Import / Export Relational DB support Score

14 15 16 17 18 19 20 21 22 23 24 25 26 1 2 3 4 5 6 7 8 9 10 11 12 13

Weight

Amazon AWS Rackspace SoftLayer CloudSigma ElasticHosts Serverlove GoGrid Terremark ecloud RimuHosting Stratogen Bluelock Fujitsu GCP BitRefinery BrightBox BT Global Services Carpathia Hosting City Cloud Claris Networks Codero CSC Datapipe e24cloud eApps FlexiScale Google GCE Green House Data 1 1 1 1 0 0 1 1 1 0 1 1 0

20

1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0

10

1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

20

1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 0 1 0 1 0 1 0 0 1 1 1

5

1 1 1 1 1 1 1 1 1 1 1 1 0 1 0 0 1 1 1 0 1 1 0 1 1 0

5

1 1 1 1 1 1 1 1 0 0 0 0 0 1 1 1 0 0 0 1 0 0 0 1 0 1

3

0 0 0 1 1 1 0 1 0 1 0 0 0 1 0 0 1 0 1 0 0 1 1 1 1 0

5

1 1 1 1 1 1 1 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0

2

1 1 0 0 0 0 0 0 1 0 0 0 1 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 Hosting.com HP Cloud IBM SmartCloud IIJ GIO iland cloud Internap Joyent LunaCloud Oktawave Openhosting.co.uk Openhosting.com OpSource ProfitBricks Qube ReliaCloud SaavisDirect SkaliCloud Teklinks Terremark vcloud 0 0 0 0 0 1 0 1 1 1 1 0 0 1 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1 0 1 0 0 0 0 0 1 0 0 0 1 0 1 1 0 1 1 0 0 1 1 0 1 1 1 1 1 0 1 1 1 0 0 1 1 0 1 0 0 1 0 1 0 1 1 1 0 0 1 1 1 1 0 1 1 0 1 1 1 0 1 1 1 1 1 1 1 1 1 1 0 1 1 0 1 0 0 1 0 1 0 1 1 0 0 1 0 0 0 0 1 1 0 0 1 1 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

46 Tier 3 0 0 0 0 1 0 0 0 Umbee

Evaluation of Cloud Providers for VPH Applications

, poster 48 49 VPS.net Windows Azure 1 1 0 0 0 1 0 1 1 1 1 1 0 0 0 1

0 0 0 0 0 0 0 0 0 0 0 0 0 0 13 12 8 5 5 0 0 27 27 25 18 18 18 15

DICE team - http://dice.cyfronet.pl

Main research interests:

• investigation of methods for building complex scientific collaborative applications and large-scale distributed computing infrastructures • elaboration of environments and tools for e-Science • development of knowledge-based approach to services, components, and their semantic composition and integration CrossGrid K-Wf Grid CoreGRID GREDIA ViroLab PLGrid series gSLM UrbanFlood MAPPER VPH-Share Collage ISMOP PaaSage 2002-2005 2004-2007 2004-2008 2006-2009 2006-2009 2009-2015 2010-2012 2009-2012 2010-2013 2011-2015 2011-2013 2013-2016 2013-2016 interactive compute- and data-intensive applications knowledge-based composition of grid workflow applications problem solving environments, programming models grid platform for media and banking applications GridSpace virtual laboratory advanced virtual laboratory, DataNet service level management for grids and clouds Common Information Space for Early Warning Systems computational strategies, software and services for distributed multiscale simulations federating cloud resources for development and execution of VPH compute- and data-intensive applications Executable Papers; 1st award in the Elsevier Grand Challenge competition at ICCS2011 cloud resource management and optimization, big data storage and analysis tools federating cloud resources, workflow composition, optimization of cloud resources, porting existing applications to cloud infrastructure