Distributed Shared Memory

Providing a shared-memory abstraction in a distributed memory system

Shared Memory Programming Model

• In a shared memory system cooperating processes/threads communicate by reading and writing shared memory. • OS provides system calls to allow separate processes to share memory and communicate. • Threads communicate through global memory in the process’s address space.

Message Passing Programming Model

• In distributed systems (no shared memory) – Applications may run on several processors & communication is based on message passing • Packages like MPI support message-based communication • RPC and client-server models provide a high-level interface that makes message passing resemble procedure calls.

Distributed Shared Memory Introduction

• Alternative: implement a software interface to let users access remote memory just like any other virtual memory reference. – i.e., remote memory references are transparent to the application, just like page faults are transparent in virtual memory. – In theory, it could be possible to write the statement “X = Y + Z;” and have it execute correctly even though X, Y, and Z are stored on separate computers.

Distributed Shared Memory Introduction

• Distributed Shared Memory (DSM) systems aim to provide this interface – it is still necessary to use message passing to transfer data from one memory to another, but DSM system handles it, not the application.

• Software DSM (as opposed to hardware implementations) is useful for clusters, grids and other loosely coupled systems • Originally proposed by Li (the Ivy System) in 1986.

How Does it Work?

• If an application generates an address that maps to local memory the reference is satisfied locally (normal virtual memory) • If the address refers to a remote machine location, the data is automatically moved to the local machine, where it can then be accessed normally.

• DSM can be implemented at the operating system level or with a library of functions that run at the user level (middleware).

Advantages – Big Picture

• DSM can be used to support data sharing between processes running on separate computers in a distributed system.

• DSM can (possibly) improve performance by speeding up data access.

Advantages

• Hide message passing from the application • Move data in large blocks that may be able to take advantage of locality • Port programs from shared memory multiprocessors with little or no change.

– Clusters of workstations are cheaper & more scalable than shared memory multiprocessors • May do away with the need for disk-based virtual memory. High-speed network transfers may be cheaper than disk reads.

– It may be possible to store everything in the combined memories of all the processors.

Implementation Issues

• Locating remote data • Protocol overhead, transmission delays • Concurrent access at multiple nodes may cause data consistency problems (~ caches in DFS) • Structure : should the DSM resemble ordinary VM (linear array) or should it be a collection of shared objects?

• Granularity: how much data is transferred in a single operation?

– If a multiple of the native page-size is used, the paging hardware can interact with the DSM system. –

False sharing

block sizes. is a problem, especially with larger

Two Approaches to DSM Unstructured

• Based on the native paging system.

• Combined memories of all processors are treated as if they are one shared memory.

• A process’s virtual memory pages could be stored on any machine.

• Page faults can be satisfied locally or remotely.

• Page sharing is supported, as it is in a shared memory processor.

Two Approaches to DSM Structured

• Unstructured DSM shares on a page-by page basis.

• In structured DSM, programmers designate certain data structures/objects as being “shared”; other data is private and managed locally – will not usually be stored at other sites.

Replication in DSM

• Replication of shared data promotes increased parallelism, fewer page faults, reduced network traffic, and in general is more efficient than non-replicated implementations.

• Main problem: preserving consistency when multiple copies are present.

– How/when do changes made at one node become visible on another node?

Consistency Semantics

• Hardware cache coherence promises

strict consistency

(UNIX semantics): a read returns the most recent write. – Multiprocessors maintain cache coherence by broadcasting writes to all processors, which can then either update or invalidate their local caches.

• Since software DSM can’t efficiently implement the atomic broadcasts needed to preserve strict consistency, other consistency models are needed.

Sequential Consistency - Review

• Formal Definition: The result of any execution is the same as if the operations of all processors were executed in some sequential order, and the operations of each individual processor appear in this sequence in the order specified by its program. • In other words, the instructions of all processes are interleaved in some sequential order.

Sequential Consistency

• For independent processes (no data sharing), this presents no problem. For critical sections, there is a possibility of race conditions.

• Users who wish to enforce a certain order of execution can use synchronization mechanisms, just as they would in a shared memory processor.

• Unstructured DSM usually enforces sequential consistency.

Consistency in Structured DSM

• Structured, or object-based DSM can use more efficient consistency semantics because it is easier to specify what is shared.

– Users only share designated objects or variables. Shared pages may have shared and unshared data on them.

– Users can identify points in the program where the data must be consistent.

Consistency in Structured DSM

• If shared-data accesses occur only inside critical sections, the DSM only needs to ensure that variables are consistent when a process enters a critical section (or when it exits) • We also assume that processes have a way to “lock” a critical section – either by a centralized or distributed mutual exclusion algorithm, for example.

Two Consistency Models

• •

Release consistency

: When a process exits a critical section, new values of the variables are propagated to all sites. No updates need to occur during the critical section, because no other process can see the data then.

Entry consistency

: When a process enters a critical section, it updates the values of the shared variables.

Summary

• Consistency is not an issue if pages/objects aren’t replicated • Also not an issue if only read-only pages are replicated.

• If read-write pages are replicated, consistency must be addressed.

• The issue: when/how to propagate updates to all sites that have replicas of the data.

Summary

• Sequential consistency sends each update to all sites in program order, although not necessarily immediately.

• Entry consistency propagates updates to a site S i when S i enters a critical section (the update must be requested).

• Release consistency propagates updates to all other sites when a process leaves its critical section.

• In each case “all sites” means “all sites that have a copy of the updated data”.

Evaluation

• A good idea, offered a plausible alternative to message passing models on clusters.

• Much research, not widely adopted.

• Recently, revived interest due to interest in grid computing.

Locating Pages

(As proposed by developers of the Ivy system) • Three approaches – Centralized manager – Fixed distributed manager – Dynamic distributed manager • This solution was designed for page based memory systems that use the read replicate approach. Structured, or object based systems can use a similar technique.

Centralized Manager

• A central manager has information about the owner of each page. • When a process faults, the local “mapping manager” (software that runs on each machine) contacts the central manager. • The manager is updated if page ownership changes. (The owner is usually the last process to have the page in write-mode).

Fixed Distributed Manager

• Each processor knows the owners of a subset of the pages; together, all processors know all owners • The function that determines who has ownership data about a page also determines how a faulting processor locates the owner. – Compare to Chord algorithm • Here, as in centralized approach, concurrent requests are serialized at the manager. (Access is FCFS)

Dynamic Distributed Manager

• Each processor tracks the ownership of pages currently in its page table.

• Page tables are augmented with a probable owner (

probowner

) field.

• Field contents may or may not be up-to date (in other words, treat as hints) • Page faults are directed to the processor in

probowner

field, which either satisfies the fault, or forwards request elsewhere.

The Ivy System

• Page based • Enforces multiple readers, single writer – supports “strict” consistency (within the limits mentioned earlier). All read-only pages are invalidated before a write operation is allowed to continue. • On any processor, a page in the system wide DSM can be either read-only, write, or nil (invalid: out of date or not present).

Real World Systems

Complete systems Add-ons to existing systems

TreadMarks

http://www.cs.rice.edu/~willy/TreadMarks/overview.html

• Supports parallel computing on networks of workstations and clusters. • Main feature: “provides a global shared address space across the different machines on a cluster.” • Contrast to packages such as PVM or MPI that provide a message passing interface machines. between • The shared memory interface lets programmers focus on algorithms instead of communication.

• Research project appears to have terminated.

Other Sources

• • • DSM 2006:The Sixth Annual Conference on Distributed Shared Memory .

Distributed Shared Memory in a Grid Environment , J.P. Ryan, B. A. Coghland,

Parallel Computing: Current and Future Issues in Parallel Computing

, 2006 First Annual Conference on Data Sharing, Consistency and DSM . 2007