Transcript Cloud Project – Calibration
Academic Compute Cloud Provisioning and Usage
Project
Peter Kunszt ETH/SystemsX.ch
2012, November 19 Bern
Motivation
Researchers often only want services, not products.
Services rely on • Infrastructure • Middleware • Application Software • Research Informatics ‘Glue’ 19 Nov. 2012
We ‘the supporters’ want to offer ‘Apps’
•
Maintainable services
•
Published, usable tools and software
•
Browsable published research data
SDCD Bern
Motivation: SystemsX.ch
Largest Swiss national research effort to date SCHWEIZERISCHE EIDGENOSSENSCHAFT CONFÉDÉRATION SUISSE CONFEDERAZIONE SVIZZERA CONFEDERAZIUN SVIZRA
19 Nov. 2012 SDCD Bern
Some numbers..
• Funded by the Swiss government with CHF 25Million/year for 2008-2011, 2012, 2013-2016 • 12 Swiss Universities and Research Institutions invest a matching 25 Million/y • Projects approved by the SNSF 19 Nov. 2012 • 14 large research projects (4-7MCHF) until 2012, 10 new starting 2013 (3MCHF) • 50+ PhD projects • 20+ interdisciplinary pilot projects • 1 strategic support project: SyBIT 2MCHF/y average SDCD Bern
SyBIT Project Motivation
SystemsX.ch will produce and analyze a large
amount of data
Strong need for coordination among data providers Strong need for common semantics and compatible service offerings Increased need for professionally supported tools and services 19 Nov. 2012 SDCD Bern
19 Nov. 2012
SyBIT provides support
Service Providers
Platforms IT Infrastructure Bioinformatics
IPP
PhosphoNetX LipidX MetaNetX PlantGrowth CellPlasticity LiverX CycliX Neurochoice WingX YeastX DynamiX CINA BattleX InfectX
IPHD SDCD Bern
19 Nov. 2012
SyBIT gives feedback
Service Providers
Platforms IT Infrastructure Bioinformatics
IPP
PhosphoNetX LipidX MetaNetX PlantGrowth CellPlasticity LiverX CycliX Neurochoice WingX YeastX DynamiX CINA BattleX InfectX
IPHD SDCD Bern
Motivation
Researchers often only want services, not products.
Services rely on • Infrastructure • Middleware • Application Software • Research Informatics ‘Glue’ 19 Nov. 2012
We ‘the supporters’ want to offer ‘Apps’
•
Maintainable services
•
Published, usable tools and software
•
Browsable published research data
SDCD Bern
Project Goals
• • • • How to extend current cluster services using cloud technology? Support new application models (MapReduce, specialized servers).
Test real applications.
Understand performance implications.
1. Define Service Models : How to move to cloud-like service orientation models.
2. Define Business Models : How to accommodate pay-per use, OpEx vs. CapEx, how to plan an academic private cloud, and how to use and offer public clouds 3. Run real applications: 19 Nov. 2012 Run a regular, a compute-intensive and a data-intensive application on the cloud.
SDCD Bern
Project Goals
• • • • How to extend current cluster services using cloud Provide input to the mid- and long Support new application models (MapReduce, specialized servers).
infrastructure at ETH and UZH.
Understand performance implications.
1. Define Service Models : How to move to cloud-like service Disseminate results in Switzerland broadly in academia and to interested cloud, and how to use and offer public clouds parties (Workshop at project end) and a data-intensive application on the cloud.
19 Nov. 2012 SDCD Bern
Cloud Attributes: When do we talk about a cloud
DEFINITION • •
Self-service, On-demand, Cost transparency
– Access to immediately available resources, paying for usage only. No long-term commitments. No up-front investments needed. Operational expenses only.
Elasticity, Multi-tenancy, Scalability
– Grow and shrink size of resource on request. Sharing with other users without impacting each other. Economies of scale.
19 Nov. 2012 SDCD Bern
Definitions
• • • Self-service: A consumer can unilaterally provision computing capabilities, such as server time and network storage, without requiring human interaction. On-demand: As needed, at the time when needed, automatic provisioning.
Cost Transparency: Accounting of actual usage transparent to user and service provider both, measured in corresponding terms (Hours CPU time, GB per Month, MB Transfer, etc) 19 Nov. 2012 SDCD Bern
Definitions
• • • Elastic: Capabilities can be elastically provisioned and released, in some cases automatically, to scale rapidly outward and inward commensurate with demand. Multi-tenant: The provider’s computing resources are pooled to serve multiple consumers, with resources dynamically assigned and reassigned according to consumer demand. Scalable: To the consumer, the capabilities available for provisioning often appear to be unlimited and can be appropriated in any quantity at any time. http://csrc.nist.gov/publications/nistpubs/800-145/SP800-145.pdf
19 Nov. 2012
HPC Pyramid
CSCS
Local Cluster (e.g. ETH Brutus)
Servers / Mini-clusters Laptop, Desktop, iPad
Number of users SDCD Bern
Relation to Cloud: As User (extension)
CSCS
Local Cluster (e.g. ETH Brutus)
Servers / Mini-clusters Laptop, Desktop, iPad
Number of users SDCD Bern
Cloud
Use Burst
19 Nov. 2012
• • •
Today, University Clusters do not make use of the Cloud:
Technical details to be investigated: – Bursting the cluster into the cloud • Networking?
• User Management?
• File System?
Cloud-compatible licenses for commercial products are often not available No billing mechanism to bill users of cluster for pay-per-use services 19 Nov. 2012 SDCD Bern
19 Nov. 2012
Relation to Cloud: As Provider
CSCS
Local Cluster (e.g. ETH Brutus)
Servers / Mini-clusters Laptop, Desktop, iPad
Number of users SDCD Bern
Account / charge usage
Cloud
Expose to
Not clear how to be a Cloud Provider with a University Cluster
• • • • • • Univ. cluster is not self-service Capital expenses, not just pay-per-use Long-term commitment Not extensible on-demand, not elastic Sharing with others only according to policies More stringent terms of use, needs account • We have examples to look at: – SDSC, Cornell, Oslo SDCD Bern
Infrastructure and Platform as a Service
Classic Approach Today IaaS .
From www.cloudadoption.org
PaaS
95%
time savings
Infrastructure
START
Platform
SaaS
Software
FINISH
Software & Apps run on platforms, NOT infrastructure
www.cloudadoption.org
CLIENTS
User Interface Machine Interface
Software
Components Services
Platform
Compute Network Storage
Infrastructure HARDWARE
Cloud Stack
DEFINITION Users or Portals. Can directly use each stack. SaaS = Software as a Service • Scientific / office / business / etc. Software as a service. Interactive or programmable.
PaaS = Platform as a Service • Programming and deployment frameworks. Integrated programmable high-level services for composition.
IaaS = Infrastructure as a Service • Virtual or hosted hardware: for HPC, compute, storage, network, specialized servers (memory, GPU, DB) Any kind of infrastructure for any of the stacks.
19 Nov. 2012
Who can makes use of what
IaaS User Portal SaaS PaaS Hardware • • • • Users may use any service Portals may use any service SaaS may or may not be built on top of PaaS or IaaS PaaS may or may not be built on top of IaaS SDCD Bern
Public, Private, Hybrid Clouds
DEFINITION Private Cloud Hybrid Cloud • • • • Own infrastructure only In-house or hosted • Private Cloud connected to Public Cloud Internal use or for sale • Remote cloud resources on-demand Full control on cloud stack, accounting, etc • Constraints on own cloud stack: needs to interoperate with public cloud Connect Public Cloud • Offered by partner organizations or cloud providers • Only operational expenses • No control on cloud stack, dependency on external partner 19 Nov. 2012 SDCD Bern
How to evolve the HPC Service..
• ..to be able to offer a Platform as a Service.
• ..to be able to make use of public clouds seamlessly (Hybrid model, CloudBursting) 19 Nov. 2012 SDCD Bern
Information Gathering
• We collected a lot of information and conducted a survey on existing solutions (mandate to CloudBroker) 19 Nov. 2012 SDCD Bern
Lots of Interactions
• • • With Cloud providers – IBM, Amazon, CloudSigma, HP, Google Software providers – VMWare, HP, Dell, OpenStack flavors (Piston, ..) Universities – SWITCH, ZHAW, SDSC, Cornell, Imperial College, U Oslo, Zaragoza 19 Nov. 2012 SDCD Bern
Choices
• • Commercial Cloud Appliance – Evaluate HP CloudSystem Matrix – Integrated hardware: HP blades and 3PAR storage – Runs with VMWare or Hyper-V – Complete management and end-user interfaces Build our own – 2 different systems (Dell based) – OpenStack: Several distributions to test – Special software: ScaleMP, cloud FS 19 Nov. 2012 SDCD Bern
Cloud Stack Comparison Matrix
OpenStack Distribution comparison
Public IaaS Comparison
Infrastructure 1
• ETH: HP CloudSystem Matrix Testbed – Operational as of THIS WEEK • • • 8 Intel, 8 AMD blades 128GB memory per blade 10TB storage 3PAR • • HP Matrix cloud software is fixed This is on RENT we have to give it back
Infrastructure 2
• • • • • ETH: Build our own from new components. – Standard cluster nodes x16, diskless – 128GB RAM on each node – Very fast storage (SSD based) for VM images Attach standard storage NAS from ETH Cloud Stack: – OpenStack – VMWare Being installed next monday This remains at ETH after the project
Infrastructure 3
• University of Zurich: Recycle existing components.
– Set of old cluster nodes, heterogeneous – Cloud filesystem using local node storage (technologies will be evaluated) • GlusterFS • Ceph 19 Nov. 2012 SDCD Bern
HPC + Cloud: On the same HW
…….
19 Nov. 2012 HPC CLUSTER SDCD Bern CLOUD HW
HPC + Cloud: On the same HW
Classic CLUSTER – Not Virtualized
• Can be heterogeneous HW • OS controlled by Admins • Scheduler for job submission • Applications compiled and installed • Shared FS 19 Nov. 2012 HPC CLUSTER SDCD Bern
CLOUD – Virtualized
• Hypervisor and Cloud Stack controlled by Admins • Template ‘Apps’ • Users can create new • Different kinds of storage • Different setups possible • Virtual SMP CLOUD HW
Storage
• • • • • Ceph, Gluster Mount REAL=non-virtual cluster FS (Lustre, GPFS) Mount NFS Object stores, e.g. SWIFT Different HW – Local Disks – iSCSI – Very fast SSD-based appliance over 10Gb or FC or IB (deduplication, compression) – for VMs and fast disk 19 Nov. 2012 SDCD Bern
Cloud HPC Use Cases to Test 1
• Extending the regular cluster into the cloud – Just run cluster node instances – Register back with cluster scheduler – Jobs can request these nodes explicitly – ALREADY tested using Amazon • Building a full virtualized cluster in our Cloud – Everything virtual: Cluster nodes, headnodes – Cluster FS : several options (see storage) – What do we learn? Reality check: HPC 19 Nov. 2012 performance SDCD Bern
Test Case 1 Software
• • • • Use regular cluster workloads, NOT data intensive Rosetta: structural biology GAMESS: molecular chemistry simulation SMSCG workloads (if we get there) 19 Nov. 2012 SDCD Bern
Cloud HPC Use Cases to Test 2
• • Hadoop cluster – Build the virtual cluster dedicated to Hadoop – HFS or Swift Commercial tool cluster: Matlab – Matlab ‘cluster’: allocate a few ‘fat’ VMs to Matlab – Let it run its internal clustering, expose to user 19 Nov. 2012 SDCD Bern
Test Case 2 Software
• • • A bit more data intensive Hadoop use cases – Proteomics: analysis of selected reaction monitoring data – Genomics: bowtie over hadoop (Crossbow) Matlab and R – Set up cluster matlab on regular cluster – On SMP’d nodes 19 Nov. 2012 SDCD Bern
Cloud HPC Use Cases to Test 3
• • Data intensive workflow – InfectX pipeline: Image analysis – several TB of small files – Many kinds of scripts, mostly Matlab – Same workflow can be submitted many times – Error prone!
OpenBIS on-demand workflow – Extend metadata catalog with some basic processing capabilities using remote resources – Streaming of data to perform some processing in the cloud
Business Models
• • • • Cannot charge at full cost if we want to be the service provider (competitive advantage) • Internal and external views Efficient, fair, feasible and generally accepted funding and charging model New opportunities should not require to change existing business procedures for existing infrastructure (evolution not revolution) Transparent Financial Accounting mechanism 19 Nov. 2012 SDCD Bern
Business Models
• • Several models are being worked out – Shareholder model – one-time fee for TFLOPS or TB – Subscription model – yearly fee – Pay-per-use model Self service options – Very detailed like Amazon – High-level ‘virtual cluster’ or PaaS – Top-level SaaS user gateways 19 Nov. 2012 SDCD Bern
Apr‘12 Jul‘12
Timeline
today Oct‘12 Jan‘13 Apr‘13 ETH Project Start SWITCH AAA Project Start Information Gathering Refinement of Targets delivered SWITCH AAA Project End Business Model Application Definition HP CloudSystem on lease ready return to HP ETH Self-built system call assembly ready from existing stuff UZH Self-built system Application testing
Output
• Workshop in April’13 to show results of project – To all Swiss research community – See you there!
• Input to ETH, UZH strategies for research infrastructure – Drive next procurement processes – Drive strategies for cooperation/outsourcing models – Drive new policy models for funding and sustainability 19 Nov. 2012 SDCD Bern