Transcript Oceanstore
OceanStore: An Infrastructure for Global-Scale Persistent Storage John Kubiatowicz, David Bindel, Yan Chen, Steven Czerwinski, Patrick Eaton, Dennis Geels, Ramakrishna Gummadi, Sean Rhea, Hakim Weatherspoon, Westley Weimer, Chris Wells, Ben Zhao A few slides have been borrowed from the authors’ presentations Vision • What is Oceanstore? • “a utility infrastructure to span the globe and provide continuous access to persistent information” Source: Berkeley OceanStore Website Vision • What is Oceanstore? • “a utility infrastructure to span the globe and provide continuous access to persistent information” • data • all kinds of information • desktop, laptop, palmtop • cars, cellular phones, other devices • futuristic: embedded in environment Vision • What is Oceanstore? • “a utility infrastructure to span the globe and provide continuous access to persistent information” • persistence • devices can be rebooted, lost, replaced • reliable, durable data (“deep archival” will last forever) • Automatic maintenance Vision What is Oceanstore? • “a utility infrastructure to span the globe and provide continuous access to persistent information” • connectivity • even to tiniest devices, possibly intermittent • variable bandwidth, latency • availability • uniform access, comparable to LAN-based networked storage • fault-tolerant, DoS-tolerant Vision • what is oceanstore? • “a utility infrastructure to span the globe and provide continuous access to persistent information” • scale • geographically distributed • 1010 users • 1014 files / objects Questions about information: • Where is persistent information stored? • 20th-century tie between location and content outdated • In world-scale system, locality is key • How is it protected? • Can disgruntled employee of ISP sell your secrets? • Can’t trust anyone (how paranoid are you?) • Can we make it indestructible? • Want our data to survive “the big one”! • Highly resistant to hackers (denial of service) • Wide-scale disaster recovery • Is it hard to manage? • Worst failures are human-related • Want automatic (introspective) diagnosis and repair First Observation: Want Utility Infrastructure • Mark Weiser from Xerox: Transparent computing is the ultimate goal. Computers should disappear into the background • In the context of storage: • Don’t want to worry about backup • Don’t want to worry about obsolescence • Need lots of resources to make data secure and highly available, BUT don’t want to own them • Outsourcing of storage already becoming popular • Pay monthly fee and your “data is out there” Utility-based Infrastructure Canadian OceanStore Sprint AT&T Pac Bell IBM IBM • Service provided by confederation of companies • Monthly fee paid to one service provider • Companies buy and sell capacity from each other Target applications Email Group calendar, contacts Distributed design tools Computer Supported Cooperative Work Digital libraries Distributed/shared repositories Assumptions • Untrusted infrastructure • • • • A small number of servers may crash or leak information most of the servers functioning correctly financially “responsible party” of servers ensure integrity but only clients trusted with cleartext • Nomadic data • • • • • data divorced from location flows freely within the storage infrastructure promiscuous caching: “anywhere, anytime” location important for performance dynamic system tuning through introspection System overview • persistent object • GUID: 160-bit SHA-1 hash • secure identification – globally unique and unforgeable • 280 unique objects before collisions (birthday paradox) • floating object replicas: independent of location • encrypted data • read • try fast probabilistic replica search (Bloom filter) • fallback to slower deterministic search (Tapestry) • write • update with predicates [as in Bayou – what is Bayou?] • creates new version What is Bayou The Bayou System (Xerox PARC) is a platform of replicated, highly-available, variable-consistency, databases on which collaborative applications can be built. It caters to portable devices having intermittent connections. System overview • application interface • sessions: sequence of read/writes • session guarantees [Bayou] • loose consistency levels, ACID • active and archival forms • active: latest version, with update handle • archive: erasure coded read-only version • dynamic optimization • object location • degree of replication Tentative Updates: Epidemic Dissemination Committed Updates: Multicast Dissemination naming • self-certifying path names (Mazières) • object GUID = hash of owner key and readable name • create hierarchies using “directory” objects • read restriction • through client encryption of data • write restriction, access control • associate ACL lists with object, respected by servers addressing • address an object by its GUID • message: GUID, random number, small predicate • route to closest GUID replica matching predicate • combines data location and routing: • no central name service to attack • save one round-trip for location discovery • routing • fast, probabilistic search algorithm • slow, deterministic search algorithm routing • fast, probabilistic search algorithm • Bloom filter • probabilistic set membership test using bit vector • n-bit vector generated from n hashes of each set element • filter is union (OR) of all bit vectors • attenuated Bloom filter • array of d Bloom filters • i th Bloom filter is union of all <i -hop nodes • slow, deterministic algorithm • Tapestry addressing and routing deterministic probabilistic Attenuated Bloom Filter updates • Updates based on versioning and conflict resolution • i.e. no locking • update: actions with predicates • commit – apply action of first true predicate • abort – no true predicates • conflict resolution on encrypted data • possible predicates: • compare-version, compare-size, compare-block, search • possible actions: • replace-block, insert-block, delete-block, append archival • produced when objects idle • use erasure codes (redundant fragmentation) • simplest example: parity bit • need any (n-1) out of n fragments • interleaved Reed-Solomon codes, Tornado codes • fragmentation improves reliability • “deep archival storage” • sweeper processes ensure replication sustained over time • fragmentation improves performance Erasure Codes Simple parity bits, or generalized Reed-Solomon codes can be used to implement it. Floating Replica and Deep Archival Coding Full Copy Ver1: 0x34243 Ver2: 0x49873 Ver3: … Conflict Resolution Logs Floating Replica Full Copy Ver1: 0x34243 Ver2: 0x49873 Ver3: … Conflict Resolution Full Copy Ver1: 0x34243 Ver2: 0x49873 Ver3: … Conflict Resolution Logs Erasure-coded Fragments dynamic optimization (introspection) • observation modules • collect and summarize information • incrementally update system database • optimization modules • periodically process the observation database • cluster recognition: group related objects • replica management: maintain replica number and location • periodic migration: work-home-work-home… • maintenance: routing, dissemination, availability, durability