Distributed Virtual Computer (DVC):

Simplifying the Development of High-Performance Grid Applications Nut Taesombut and Andrew A. Chien Department of Computer Science and Engineering University of California, San Diego

Workshop on Grids and Advanced Networks (GAN’04) Chicago, Illinois April 22, 2004

Outline

• • • • •

Background and Motivation Distributed Virtual Computer (DVC) Example Application Related Work Summary and Future Work

4/26/2020 2

Emerging Opportunity of Lambda Grids

• •

Network Advances and Trends

–

DWDM optical paths (or lambdas) enable

–

Dedicated High Bandwidth

–

Dynamic Configuration Lambda Grid

–

Distributed, shared resources interconnected by plentiful lambdas

– – –

Configurable Connections and Capacity Deterministic Communication Performance Novel Communication Capabilities (e.g. optical multicast)

4/26/2020 3

OptIPuter Project

•

Opt IP uter – Large-scale Research Project on Impact of Lambdas on System Software and next-generation E-science

–

International Testbed for Experimentation (UCSD, UIC, UCI, Amsterdam, etc.)

–

Leading E-science Drivers (Neuroscience, Geophysical/Earth Sciences)

– –

3-D Data Analysis, Visualization and Collaboration Applications Data-intensive and Real-time, Distributed data sources/sinks

–

Wealth of Innovative System Software Research (protocols, DVC, storage, etc.) Smarr, Papadopoulos, Ellisman, Orcutt, Chien – UCSD DeFanti, Leigh - UIC

4 4/26/2020 http://www.optiputer.net

Motivation

•

Building Grid Applications is Difficult!

–

Applications must deal with complexity of resource environment

–

Resource Heterogeneity, Performance, Communication

– – –

Multi-Organization Security, Resource Management Shared and Untrusted Resource Environments Low-Quality Networks

•

Adding Low-level Management of Network Complicates the Task

– – –

No Uniform Interfaces (routers, switches, end nodes) Wildly Different Semantic Level (BELOW IP!) Novel Capabilities (e.g. multicast, RDMA, etc.)

•

Key Requirements

– –

A new abstraction which simplifies Grid environments A view which integrates communication as first class

4/26/2020 5

Example of Grid Complexity

http://dream.uci.edu:4010 http://intania.uic.edu:4566 Internet http://zebra.ucsd.edu:5220 10.1.3.34

10.1.2.61

10.1.2.68

• • •

Access to Resources Across Multiple Namespaces User-Controlled Configuration of Dynamic Lambda Network Heterogeneous Communication

4/26/2020 6

Distributed Virtual Computer (DVC)

• •

A Simple Execution Environment for Grid Applications

–

Set of LambdaGrid resources (connections, resources)

– –

Naming, access, and management services Transparently shared amongst Applications Simplify Use of Network and Grid Capabilities

–

Automate compute/data resource binding

– –

Automate dynamic λ-configuration; expose novel λ-capabilities Leverage existing Grid Technologies (Globus, NWS etc.) DVC

4/26/2020 7

DVC Design Principles

•

Separate Resource Config/Mgmt and Application Programming

–

Resource Environments Configured to Spec

–

Applications simply use them

•

Aggregate and Bind Grid Resources; Present as Workgroup

–

Central resource control

– – – –

Single namespace Unified resource access mechanisms Trusted and secure environment Controllable performance

•

Enable Collective Resource Views

– –

Unified naming structures (e.g. collective names) Collective properties (e.g. group communication, trust, access control)

4/26/2020 8

Example: Locally Simplified Grid Programming

DVC Domain comp1 comp2 http://dream.uic.edu:4010 http://intania.uic.edu:4566 Internet http://zebra.ucsd.edu:5220 10.1.3.34

10.1.2.61

10.1.2.68

comp3 str1

• • •

Simple View of DVC Single Control Domain Unified Naming Mechanism Uniform Use of Different Communication Mechanisms (e.g. protocols)

4/26/2020 9

How DVC’s Simplify Application Grid Programming

• • • •

Automate Resource Binding and Configuration

–

Reduce user interaction through resource broker and manager Unify Resource Naming and Access Mechanisms

–

Hide heterogeneity through simple abstractions Transparently Enable Security Protection

–

Implement cryptographic operations at the middleware layer Monitor and Control Resource and Communication Performance

–

Detect asynchronous events and notify application based on subscription

4/26/2020 10

Realizing Simplified Application Grid Programming

DVC • •

DVC Manager

– – – –

Single master controller Resource selector/negotiator/scheduler Trust mediator and security authority Synchronizer of global state information Ghost Managers

– – – –

Slave managers, each running at each bound resource Job process controller at remote resource Communication mediator Resource status monitor and reporter

4/26/2020 Data Flow Control Flow DVC Manager Ghost Manager 11

Example: Dynamic Configuration of Lambda Grid

•

Example Application:

–

BIRN (Biomedical Information Research Network) Duke NCMIR/UCSD Harvard BIRN DVC SDSC UNC

•

UCLA UCI DVC Advantages

– –

On-demand creation of a private Grid resource workgroup Transparent use of novel communication capabilities

–

high-speed multi-point communication

–

SAN-like storage access across geographically distributed resources

4/26/2020 12

Example: Dynamic Configuration of Lambda Grid

•

Sequence to Create a BIRN DVC 1.

2.

3.

4.

Create a resource configuration specification and send a request to bind resources Create resource groups (i.e. for collective data source and sink) Create multipoint-to-point and point-to-multipoint communication sessions Define the properties of communication sessions (e.g. security and communication mechanisms) NCMIR/UCSD Harvard Duke SDSC UCI UCLA UNC Physical-Level View of BIRN DVC

4/26/2020

ucsd harvard duke Grp1 GTP + enc + auth sdsc TCP + Optical Multicast unc uci uci Grp2 Application-Level View of BIRN DVC

13

Related Work

•

Abstractions of Distributed Resources

–

PVM [Geist94]

•

Grid Middleware

–

Globus System

•

Grid Programming Tools

– –

GridRPC [Nakada02], MPICH-G2 [Karonis03], Condor-G [Frey01] GrADS [Berman01], GridLab [Allen03],

•

Federation of Resources

–

Virtual Organization (VO) [Foster01]

•

Distributed Virtual Computer

–

Provide an Application-Focused Dynamic Resource Container

–

Dynamic resource configuration and sharing policies

4/26/2020 14

Summary and Future Work

•

Summary

–

DVC’s provide simple computing environments for Grid applications

– – –

Locally simplified resource workgroup DVC’s allow on-demand instantiation and dynamic configuration of Lambda-Grid DVC’s enable simple use of novel communication capabilities

•

Future Work

–

Develop the full implementation of the DVC Prototype

–

Implement as Web Services (i.e. WS-RF specification)

–

Deploy the prototype on the OptIPuter Testbed

–

Demonstrate with OptIPuter applications (Bioinformatics and Geophysical)

–

Explore other system technologies that can be integrated into the DVC framework

– –

Real-time System High-Performance Distributed File System

4/26/2020 15

Thank You

•

Questions and Remarks?

•

Contact Information:

–

Nut Taesombut ( [email protected]

)

•

OptIPuter Website:

–

www.optiputer.net

4/26/2020 16