FutureGrid

Cyberinfrastructure for Computational Research

FutureGrid Goals

    An environment that can be used for middleware developers to develop and test middleware An environment that can instantiate a wide variety of computing environments (ranging from current TeraGrid software, VMs, cloud, grid systems other than TG) for development and testing of applications on experimental basis in a test bed, in replicable ways Education related to all of the above US Industry research related to the above

FutureGrid Components

• • • Compute – flexible deployment of images, VM's, clouds Storage – local storage sufficient to facilitate projects, Lustre for larger projects Network – Dedicated networks between FutureGrid sites, Impairment Device

System type # CPUs

Compute Hardware

# Cores TFLOPS Total RAM (GB) Secondary Storage (TB) Site Status Dynamically configurable systems

IBM iDataPlex Dell PowerEdge 256 192 1024 768 IBM iDataPlex IBM iDataPlex 168 168 672 672

Subtotal 784 3136

Systems not dynamically configurable

Cray XT5m Shared memory system TBD 168 40 672 480 IBM iDataPlex High Throughput Cluster

Subtotal

Total

64 192

464

1248

256 384

1792

4928

11 8 7 7

33

6 4 2 4

16

49

3072 1152 2016 2688

8928

1344 640 768 192

2944

11872

339* 30 120 96

585

339* 339* 1

1

586

IU TACC New System New System UC SDSC New System Existing System IU IU UF PU New System New System 4Q2010 New System Existing System

Storage Hardware

System Type Capacity (TB)

DDN 9550 (Data Capacitor) 339

File System

Lustre

Site

IU

Status

Existing System • • • DDN 6620 SunFire x4170 Dell MD3000 120 96 30 GPFS ZFS NFS UC SDSC TACC New System New System New System FutureGrid has a dedicated network (except to TACC) and a network fault and delay generator Experiments can be isolated by request Additional partner machines may run FutureGrid software and be supported (but allocated in specialized ways)

Logical Network Diagram

Network Impairments Device

• • • • • • • Spirent XGEM Network Impairments Simulator for jitter, errors, delay, etc Full Bidirectional 10G w/64 byte packets up to 15 seconds introduced delay (in 16ns increments) 0-100% introduced packet loss in .0001% increments Packet manipulation in first 2000 bytes up to 16k frame size TCL for scripting, HTML for manual configuration

08/25/10 http://futuregrid.org

• • • • •

FutureGrid Architecture

Open Architecture allows to configure resources based on images Managed images allows to create similar experiment environments Experiment management allows reproducible activities Through our modular design we allow different clouds and images to be “rained” upon hardware.

Will support deployment of preconfigured middleware including TeraGrid stack, Condor, BOINC, gLite, Unicore, Genesis II

Software Goals

• Open-source, integrated suite of software to – instantiate and execute grid and cloud experiments. – perform an experiment – collect the results – tools for instantiating a test environment • TORQUE, Moab, xCAT, bcfg, and Pegasus, Inca, ViNE, a number of other tools from our partners and the open source community • Portal for interacting with tools – Benchmarking 08/25/10 http://futuregrid.org

10

Draft GUI for FutureGrid Dynamic Provisioning

Command line

• • fg-deploy-image – – host name image name – start time – end time – label name fg-add – label name – framework hadoop – version 1.0

• Deploys an image on a host • Adds a feature to a deployed image 08/25/10 http://futuregrid.org

12

FG Stratosphere

• Objective – Higher than a particular cloud – Provides all mechanisms to provision a cloud on a given FG hardware – Allows the management of reproducible experiments – Allows monitoring of the environment and the results • • Risks – – Lots of software Possible multiple path to do the same thing Good news – We worked in a team, know about different solutions and have identified a very good plan – We can componentize Stratosphere 08/25/10 http://futuregrid.org

13

Dynamic Provisioning

     Change underlying system to support current user demands Linux, Windows, Xen/KVM, Nimbus, Eucalyptus Stateless images   Shorter boot times Easier to maintain Stateful installs  Windows Use Moab to trigger changes and xCAT to manage installs 08/25/10 http://futuregrid.org

14

Experiment Manager

• Objective – Manage the provisioning for reproducible experiments – Coordinate workflow of experiments – Share workflow and experiment images – Minimize space through reuse 08/25/10 http://futuregrid.org

15

• • • •

Getting started with FutureGrid

www.futuregrid.org

Account/Project request – Nimbus/Eucalyptus account for those systems, if needed Getting started information at www.futuregrid.org/tutorials Mail [email protected]

for any issues

Acknowledgements

• • • • • • • • FutureGrid http://www.futuregrid.org/ NSF Award OCI-0910812 NSF Solicitation 08-573 – http://www.nsf.gov/pubs/2008/nsf08573/nsf08573.htm

ViNe http://vine.acis.ufl.edu/ Nimbus http://www.nimbusproject.org/ Eucalyptus http://www.eucalyptus.com/ VAMPIR http://www.vampir.eu/ Pegasus http://pegasus.isi.edu/