PlanetLab – nurturing the next internet David Culler UC Berkeley

www.planet-lab.org

PlanetLab is …

http://www.planet-lab.org

• •

9/13/2003

A novel world-wide testbed 205 machines at 85 sites in 19 countries

– – –

Towards thousands Universities, Internet 2, co-location centers 230 research projects COE Dist. Alumnae - PlanetLab

... is many, many vantage points on the internet

• • •

Internet in the middle Close to you wherever you are A truly global network perspective 9/13/2003 COE Dist. Alumnae - PlanetLab

...is a powerful community of network systems researchers

Washington MIT Tom Anderson Steven Gribble David Wetherall Frans Kaashoek Hari Balakrishnan Robert Morris David Anderson Berkeley David Culler, Ion Stoica Joe Helerstein Eric Brewer John Kubiatowicz Intel Research Timothy Roscoe Brent Chun Sylvia Ratnasamy Gaetano Borriello Satya Mic Bowman Duke Amin Vadat Jeff Chase Princeton Larry Peterson Randy Wang Vivek Pai

9/13/2003 COE Dist. Alumnae - PlanetLab see http://www.cs.berkeley.edu/~culler/planetlab

Rice Peter Druschel Utah Jay Lepreau CMU Srini Seshan Hui Zhang UCSD Stefan Savage Columbia Andrew ICIR Campbell Scott Shenker Eddie Kohler

… is a novel academic / industry collaboration

• • • • •

Inspired by University research Seeded by Intel Architected and led by academic community Developed and maintained by combined effort Growing an industrial consortium

–

Hosted at Princeton with UCB and UWash 9/13/2003 COE Dist. Alumnae - PlanetLab

is a Confluence of Technologies

• • • • • • • • • • • Cluster-based scalable distribution, remote execution, management, monitoring tools –

UCB Millennium, OSCAR, ..., Utah Emulab, ...

CDNS and P2Ps –

Gnutella, Kazaa, ...

Proxies routine Virtual machines & Sandboxing –

VMWare, Janos, Denali,... web-host slices (EnSim)

Overlay networks becoming ubiquitous –

xBone, RON, Detour... Akamai, Digital Island, ....

Service Composition Frameworks –

yahoo, ninja, .net, websphere, Eliza

Established internet ‘crossroads’ – colos Web Services / Utility Computing Authentication infrastructure (grid) Packet processing (layer 7 switches, NATs, firewalls) Internet instrumentation

9/13/2003 COE Dist. Alumnae - PlanetLab

Is a novel system architecture

•

Distributed means of acquiring a slice of virtual machines spanning much of the planet 9/13/2003 COE Dist. Alumnae - PlanetLab

...is a rich research agenda

• Network measurement –

Scriptroute, PlanetProbe, I3, etc.

• Application-level multicast –

ESM, Scribe, TACT, etc.

• Distributed Hash Tables –

Chord, Tapestry, Pastry, Bamboo, etc.

• Wide-area distributed storage –

Oceanstore, SFS, CFS, Palimpsest, IBP

• Resource allocation –

Sharp, Slices, XenoCorp, Automated contracts

• Distributed query processing –

PIER, IrisLog, Sophia, etc.

• Content Dist. Networks –

CoDeeN, ESM, UltraPeer emulation, Gnutella mapping

• Management and Monitoring –

Ganglia, InfoSpect, Scout Monitor, BGP Sensors, etc.

• Overlay Networks –

RON, ROM++, ESM, XBone, ABone, etc.

• Virtualization and Isolation –

Xen, Denali, VServers, SILK, Mgmt VMs, etc.

• Router Design implications –

NetBind, Scout, NewArch, Icarus, etc.

• • Testbed Federation –

NetBed, RON, XenoServers Etc., etc., etc. 9/13/2003 COE Dist. Alumnae - PlanetLab

...is an incubator for the next generation of the internet

Underlay: the new thin waste?

routing, topology services sink down into the internet Internet “the next internet will be created as an overlay on the current one” 9/13/2003 COE Dist. Alumnae - PlanetLab

Where did it come from?

9/13/2003 COE Dist. Alumnae - PlanetLab

Innovator’s Dilemma

•

The Internet is an enormous success story

– – –

commercially impact on our daily lives global reach

•

Success has an unexpected cost:

ossification

–

difficult to deploy disruptive technologies

» »

address vulnerabilities introduce new capabilities 9/13/2003 COE Dist. Alumnae - PlanetLab

A new look at internet services

9/13/2003 COE Dist. Alumnae - PlanetLab

Planetary-Scale Services

• • •

Services and applications spread over the web

– –

Proximity => low latency, high bandwidth, predictable, reliable Perspective => adapt to load, delays, failures, $ on a global scale Content-distribution Networks and Peer-to-Peer sharing just the tip of the iceberg.

Academic Community developing the architectural building blocks to enable many kinds of distributed services

– – – – –

scalable translation, dist. storage, dist. events, instrumentation, management 9/13/2003 COE Dist. Alumnae - PlanetLab

Key Concept: Overlay networks

9/13/2003 COE Dist. Alumnae - PlanetLab

Overlay network routing

9/13/2003 COE Dist. Alumnae - PlanetLab

key missing element – real hands on experience

•

Researchers had no vehicle to try out their next

n

great ideas in this space

• • • •

Lot’s of simulations Lot’s of emulation on large clusters

–

emulab, millennium, modelnet Lot’s of folks calling their 17 friends before the next deadline

–

RON testbed but not the surprises and frustrations of experience at scale to drive innovation 9/13/2003 COE Dist. Alumnae - PlanetLab

Growing up quick

• • • • •

“Underground” meeting March 2002 Intel seeds effort

– –

First 100 nodes Operational support First node up July 2002 By SOSP (deadline March 2003) 25% of accepted papers refer to PlanetLab Each following conference has seen dramatic load

– –

OSDI NDSI COE Dist. Alumnae - PlanetLab 9/13/2003

Guidelines (1)

•

Thousand viewpoints on “the cloud” is what matters

– – –

not the thousand servers not the routers, per se not the pipes 9/13/2003 COE Dist. Alumnae - PlanetLab

Guidelines (2)

•

and you must have the vantage points of the crossroads

–

primarily co-location centers 9/13/2003 COE Dist. Alumnae - PlanetLab

Guidelines (3)

•

Each service needs an overlay covering many points

–

logically isolated

•

Many concurrent services and applications

– –

must be able to slice nodes = > VM per service service has a slice across large subset

•

Must be able to run each service / app over long period to build meaningful workload

–

traffic capture/generator must be part of facility

•

Consensus on “a node” more important than “which node” 9/13/2003 COE Dist. Alumnae - PlanetLab

Guidelines (4)

Management, Management, Management

• • • •

Test-lab as a whole must be up a lot

–

global remote administration and management

»

mission control

–

redundancy within Each service will require its own remote management capability Planetlab nodes cannot “bring down” their site

– – –

generally not on main forwarding path proxy path must be able to extend overlay out to user nodes?

Relationship to firewalls and proxies is key 9/13/2003 COE Dist. Alumnae - PlanetLab

Guidelines (5)

• •

Storage has to be a part of it

–

edge nodes have significant capacity Needs a basic well-managed capability

–

but growing to the seti@home some stage model should be considered at

–

may be essential for some services 9/13/2003 COE Dist. Alumnae - PlanetLab

Outcome

•

“Mirror of Dreams” project

• •

K.I.S.S.

– –

Building Blocks, not solutions no big standards, OGSA-like, meta-hyper-supercomputer Compromise

–

A basic working testbed in the hand is much better than “exactly my way” in the bush

•

“just give me a bunch of (virtual) machines spread around the planet,.. I’ll take it from there”

•

small distr. arch team, builders, users 9/13/2003 COE Dist. Alumnae - PlanetLab

UCB enabling technology

9/13/2003 COE Dist. Alumnae - PlanetLab

Tension of Dual Roles

• •

Research testbed

– – –

run fixed-scope experiments large set of geographically distributed machines diverse & realistic network conditions Deployment platform for novel services

– –

run continuously develop a user community that provides realistic workload 9/13/2003

design measure

COE Dist. Alumnae - PlanetLab

deploy

Architecture principles

• • • •

“Slices” as fundamental resource unit

–

distributed set of (virtual machine) resources

– –

a service runs in a slice resources allocated / limited per-slice (proc, bw, namespace) Distributed Resource Control

–

host controls node, service producer, service consumers Unbundled Management

–

provided by basic services (in slices)

–

instrumentation and monitoring a fundamental service Application-Centric Interfaces

–

evolve from what people actually use

•

Self-obsolescence

–

everything we build should eventually be replaced by the community

–

initial centralized services only bootstrap distributed ones COE Dist. Alumnae - PlanetLab 9/13/2003

Slice-ability

• • •

Each service runs in a slice of PlanetLab

– –

distributed set of resources (network of virtual machines) allows services to run continuously VM monitor on each node enforces slices

– –

limits fraction of node resources consumed limits portion of name spaces consumed Challenges

– – – –

global resource discovery allocation and management enforcing virtualization security COE Dist. Alumnae - PlanetLab 9/13/2003

Unbundled Management

• •

Partition management into orthogonal services

– – – – –

resource discovery monitoring system health topology management manage user accounts and credentials software distribution and updates Approach

– – –

management services run in their own slice allow competing alternatives engineer for innovation (define minimal interfaces) COE Dist. Alumnae - PlanetLab 9/13/2003

Distributed Resource Control

• •

At least two interested parties

–

service producers (researchers)

»

decide how their services are deployed over available nodes

–

service consumers (users)

»

decide what services run on their nodes At least two contributing factors

–

fair slice allocation policy

»

both local and global components (see above)

–

knowledge about node state

»

freshest at the node itself 9/13/2003 COE Dist. Alumnae - PlanetLab

Application-Centric Interfaces

• •

Inherent problems

– – –

stable platform versus research into platforms writing applications for temporary testbeds integrating testbeds with desktop machines Approach

– – –

adopt popular API (Linux) and evolve implementation eventually separate isolation and application interfaces provide generic “shim” library for desktops 9/13/2003 COE Dist. Alumnae - PlanetLab

Research Thrusts

9/13/2003 COE Dist. Alumnae - PlanetLab

Open Content Distribution Networks

9/13/2003 Codeen – Vivek Pai @ Princeton COE Dist. Alumnae - PlanetLab

Application-Level Multicast

9/13/2003 Druschel, Rice Srini, CMU COE Dist. Alumnae - PlanetLab

Global Objects

name address

• •

Global objects drawn from a large namespace

– –

128 – 256 bits Table would have more rows than atoms in universe Objects could be anywhere on the planet 9/13/2003 lookup COE Dist. Alumnae - PlanetLab

Distributed Hash Tables (DHT)

•

Combine lookup and routing by doing a series of

– –

small lookup => next hop route name address 9/13/2003 COE Dist. Alumnae - PlanetLab

DHT Design

• •

Locate object from large namespace anywhere within small factor (<2) of knowing its address Dozens of competing alternative based on different mathematical structures

–

CAN, Chord, Pastry, Tapestry, Plaxton, Viceroy, Kademlia, Skipnet, Symphony, Koorde, Apocrypha, Land, ORDI …

100 110 010 000 101 001 111 011 111 000

h = 2

000 001 010

h = 1

011 100 101 110 111 110 101 100 001 010 011

COE Dist. Alumnae - PlanetLab 9/13/2003

Empirical Comparison (Rhea, Roscoe,Kubi)

• •

79 PlanetLab nodes, 400 ids per node Performed by Tapesty side 9/13/2003 COE Dist. Alumnae - PlanetLab

Redefined set of critical issues

• • •

Not dilation once converged, but behavior under churn Convergence of approaches Convergence of interfaces 9/13/2003 COE Dist. Alumnae - PlanetLab

Ossified or fragile?

• •

One group forgot to turn off an experiment

–

after 2 weeks of router being pinged every 2 seconds, ISP contacted ISI and threatened to shut them down.

One group failed to initialize destination address and ports (and had many virtual nodes on each of many physical nodes)

– – – – –

worked OK when tested on a LAN trashed flow-caches in routers probably generated a lot of unreachable destination traffic triggered port-scan alarms at ISPs (port 0) n^2 probe packets trigger other alarms COE Dist. Alumnae - PlanetLab 9/13/2003

Distributed Storage

• • • • •

Phase 0 provides basic copy scripts

–

community calls for global nfs / afs !!!

Good spectrum of novel proposals Internet Backplane Protocol (Tenn)

– – –

basic transport and storage of variable sized blocks (in depots) intermittently available, untrusted, bounded duration do E2E redundancy, encryption, permanence Cooperative File System (MIT, UCB)

–

FS over DHASH (replicated blocks) over Chord

»

PAST distributes whole files over Pastry

–

distributed read-only file storage Ocean store (UCB)

–

versioned updates of private, durable storage over untrusted servers COE Dist. Alumnae - PlanetLab 9/13/2003

OceanStore (Kubiatowicz)

RAID distributed over the whole Internet

9/13/2003 COE Dist. Alumnae - PlanetLab

Dipping in to OceanStore Prototype

• •

Routine studies on thousands virtual nodes across a hundred planetlab sites Efficiency of dissemination tree

–

more replicas allows more of the bytes to move across fast links 9/13/2003 COE Dist. Alumnae - PlanetLab

Watching the internet in the middle

scp 4 MB to MIT, Rice, CIT confirm Padhye SIGCOMM98 83 machines, 11/1/02 Sean Rhea basis for DHT comparison 143 RON+PlanetLab Synthetic Coodinate c/o Frans Kaashoek COE Dist. Alumnae - PlanetLab i3 weather service

Towards an instrumentation service

•

Critical underlying issue

– –

All the design techniques are evaluated relative to the raw internet Sophisticated services observe and adapt to the internet

• • •

every overlay, DHT, and multicast is measuring the internet in the middle they do it in different ways they do different things with the data

•

Can this be abstracted into a customizable instrumentation service?

– – –

Share common underlying measurements Reduce ping, scp load Grow down into the infrastructure COE Dist. Alumnae - PlanetLab 9/13/2003

Internet Measurement

9/13/2003 COE Dist. Alumnae - PlanetLab

Representative Sample of the Internet?

9/13/2003 COE Dist. Alumnae - PlanetLab

Pier: Distributed Query Processing

Single Site Clusters Distributed 10’s – 100’s Database Community Internet Scale 1000’s – Millions Network Community •

Challenge: How to run DB style queries at Internet Scale!

9/13/2003 COE Dist. Alumnae - PlanetLab Hellerstein, Stoica, Shenker

Declarative Queries Query Plan Overlay Network Physical Network Network Monitoring Other User Apps Applications Query Optimizer Catalog Manager PIER Core Relational Execution Engine DHT Wrapper Overlay Routing DHT Storage Manager IP Network Network

Does This Work for Real?

Scale-up Performance (1MB source data/node) 9/13/2003 Real Network Simulation COE Dist. Alumnae - PlanetLab

the Gaetano advice

• •

for this to be successful, it will need the support of network and system administrators at all the sites...

it would be good to start by building tools that made their job easier 9/13/2003 COE Dist. Alumnae - PlanetLab

ScriptRoute (Spring, Wetherall, Anderson)

• • •

Traceroute provides a way to measure from you out 100s of traceroute servers have appeared to help debug connectivity problems

–

very limited functionality => provide simple, instrumentation sandbox at many sites in the internet

– –

TTL, MTU, BW, congestion, reordering safe interpreter + network guardian to limit impact

»

individual and aggregate limits COE Dist. Alumnae - PlanetLab 9/13/2003

Example: reverse trace

UW Google

•

underlying debate: open, unauthenticated, community measurement infrastructure vs closed, engineered service

•

see also Princeton BGP multilateration 9/13/2003 COE Dist. Alumnae - PlanetLab

Ossified or brittle?

• • •

Scriptroute set of several alarms Low bandwidth traffic to lots of ip addresses brought routers to a crawl Lots of small TTLs but not exactly Traceroute packets...

•

isp installed filter blocking subnet at Harvard and sent notice to network administrator without

human intervention

–

Is innovation still allowed?

COE Dist. Alumnae - PlanetLab 9/13/2003

NetBait Serendipity

• • • • • •

Brent Chun built a simple http server on port 80 to explain what planetlab was about and to direct inquiries to planet-lab.org

It also logged requests Sitting just outside the firewall of ~40 universities...

the worlds largest honey pot the number of worm probes from compromized machines was shocking imagine the the epidemiology

•

see netbait.planet-lab.org

COE Dist. Alumnae - PlanetLab 9/13/2003

One example

250 200 150 Code Red Nimda 100 50 0 1/ 5/ 20 1/ 03 10 /2 00 1/ 3 15 /2 00 1/ 3 20 /2 00 1/ 3 25 /2 00 1/ 3 30 /2 00 2/ 3 4/ 20 03 2/ 9/ 20 2/ 03 14 /2 00 2/ 3 19 /2 00 2/ 3 24 /2 00 3/ 3 1/ 20 03 3/ 6/ 20 3/ 03 11 /2 00 3/ 3 16 /2 00 3 • •

The monthly code-red cycle in the large?

What happened in March?

COE Dist. Alumnae - PlanetLab 9/13/2003

No, not Iraq

1400 1200 1000 800 Code Red Nimda Code Red II.F

600 400 200 0 3/ 1/ 20 03 3/ 2/ 20 03 3/ 3/ 20 03 3/ 4/ 20 03 3/ 5/ 20 03 3/ 6/ 20 03 3/ 7/ 20 03 3/ 8/ 20 03 3/ 9/ 20 3/ 03 10 /2 00 3/ 3 11 /2 00 3/ 3 12 /2 00 3/ 3 13 /2 00 3/ 3 14 /2 00 3/ 3 15 /2 00 3/ 3 16 /2 00 3/ 3 17 /2 00 3/ 3 18 /2 00 3/ 3 19 /2 00 3/ 3 20 /2 00 3 •

A new voracious worm appeared and displaced the older Code Red 9/13/2003 COE Dist. Alumnae - PlanetLab

Netbait view of March

9/13/2003 COE Dist. Alumnae - PlanetLab

But where is the real action

•

Management, Management, Management

•

Truly distributed resource allocation and management

–

Perhaps the first truly meaningful computational economy 9/13/2003 COE Dist. Alumnae - PlanetLab

What Planet-Lab is about?

• Create the open infrastructure for invention of the next generation of wide area (“planetary scale”) services –

post-cluster, post-yahoo, post-CDN, post-P2P, ...

• Potentially, the foundation on which the next Internet can emerge –

think beyond TCP/UDP/IP + DNS + BGP + OSPF... as to what the net provides

– –

building-blocks upon which services and applications will be based “the next internet will be created as an overlay in the current one” (NRC)

• A different kind of network testbed – – – –

not a collection of pipes and giga-pops not a distributed supercomputer geographically distributed network services alternative network architectures and protocols

• Focus and Mobilize the Network / Systems Research Community to define the emerging internet

9/13/2003

Current Institutions (partial)

Academia Sinica, Taiwan Boston University Caltech Carnegie Mellon University Chinese Univ of Hong Kong Columbia University Cornell University Datalogisk Institut Copenhagen Duke University Georgia Tech Harvard University HP Labs Intel Research Johns Hopkins Lancaster University Lawrence Berkeley Laboratory MIT Michigan State University National Tsing Hua Univ.

New York University Northwestern University Princeton University Purdue University Rensselaer Polytechnic Inst.

Rice University Rutgers University Stanford University Technische Universitat Berlin The Hebrew Univ of Jerusalem University College London University of Arizona University of Basel University of Bologna University of British Columbia UC Berkeley UCLA UC San Diego UC Santa Barbara University of Cambridge University of Canterbury University of Chicago University of Illinois University of Kansas University of Kentucky University of Maryland University of Massachusetts University of Michigan University of North Carolina University of Pennsylvania University of Rochester USC / ISI University of Technology Sydney University of Tennessee University of Texas University of Toronto University of Utah University of Virginia University of Washington University of Wisconsin Uppsala University, Sweden Washington University in St Louis Wayne State University

9/13/2003 COE Dist. Alumnae - PlanetLab

Thanks

9/13/2003 COE Dist. Alumnae - PlanetLab