Transcript CS 4xx – Distributed Systems

Topics in Distributed Systems
CS350, CS552,
CS580F
Topics
1.
2.
3.
4.
5.
6.
7.
Types of Distributed systems
Communications
Naming
Synchronization
Consistency & Replication
Fault Tolerance
Security
2
©2010 D. J. Foreman
Introduction
3
©2010 D. J. Foreman
Definition of a Distributed System
 A collection of functions implemented across a


set of independent devices that appears to its
users as a single coherent system.
Horizontal - all devices are peers
Vertical - there is a control hierarchy
 Note: Devices may be computers or ….
4
©2010 D. J. Foreman
Types of Distributed systems
 Client-Server
 Peer-peer
 Processor Pool
 Network Operating Systems
5
©2010 D. J. Foreman
Examples of Distributed Systems
 An "ATM machine"
 A remote file access mechanism
 A database
 A chat room
 A computing "grid" (like SETI)
6
©2010 D. J. Foreman
Design
 All “usually” have:



Multi-threaded/Multi-process operation
User interface (need not be graphical)
Point(s) of control (any system shuts-down all)
7
©2010 D. J. Foreman
Some Problems (topics)
 System Character codes (ASCII/EBCDIC)
 Number representation (big- vs. little-Endian)
Size of an "int" (16/32/64 - bit)
 Float
Shared Memory
Operational Compatibility
 Thread control
 System services
 Process handling
Pointers & References
Objects





8
©2010 D. J. Foreman
Hardware Systems
 Multiprocessors (modularized shared memory)



Bus traffic  cache  incoherence
Limited scalability (network cost/speed)
NUMA – Non-Uniform Memory Access

(some local, some shared via network)
 Homogeneous Multicomputers



Private memory + switching circuits
Bus-based connection: ethernet routing
Switch-based: routing via h/w net (mesh, n-cube)

e.g., Clusters of Workstations
 Heterogeneous Multicomputers

No commonality, no global view
9
©2010 D. J. Foreman
Software
 Tightly coupled (true "Distributed System")

Maintains global view
Multiprocessor (one O/S)
 Multicomputer (homogeneous vs. heterogeneous)

 Loosely coupled (NOS)



Each heterogeneous computer runs its own O/S
Not much transparency as seen by applications
BUT: Local services are shared on the network
10
©2010 D. J. Foreman
Some Classical Distributed Systems
(see list of papers)
 Locus – distributed UNIX
 Lightweight kernel systems w/UNIX emulation
Amoeba – groups of threads (processor pool)
 Mach – synch msg-based IPC, threads
Argus ('79) – "Guardians" for transactions
Clouds – object-based (like methods)
V-system (1984)– IPC via kernel (msgs only),
true multi-threading (unavailable in UNIX '84)
Cedar - µcoded VM, IDE, concurrent execution





11
©2010 D. J. Foreman
Classical Systems, continued
 XDFS - ca. 1977 - servers
 Cambridge File Server – ca.1982 - indexes on

servers used by OS to build its directory
Sun NFS – networked UNIX FS
12
©2010 D. J. Foreman
Open vs Closed Systems
 Open


Extensible
Services external to kernel
 Closed


Non-extensible
Services internal to kernel
 Services are:






File system
Process mgmt
Network communications
Names
Time
Security/authentication
13
©2010 D. J. Foreman
Atomicity

Semaphores



Unstructured – requires app to test/signal
Mutexes
Monitors

A programming construct
Shared variables
 Procedures



lock and unlock functions (may be kernel-provided)
Threads (Pthreads, POSIX threads)


Butenhof, "Programming with POSIX threads", AW
Lewis & Berg, "Multithreaded Programming with
Pthreads", PTR-PH
14
©2010 D. J. Foreman
Number Representations
 What is an "int"


16 bits ???
32 bits ???
 FP number storage


Many mechanisms, depending on h/w
Less portable than integers
15
©2010 D. J. Foreman
An example using int's
 32-bit compiler, treats "int" as 32 bits

int x;
x=1; // x is 00000000 00000000 00000000 00000001
Y(x); // all 32 bits ---> 16-bit target system
Target of call compiled on 16-bit system
function Y (int val) // "int" is 16 bits
{print val;} // prints 0 ---- WHY???? ---
 'Y' only gets the 1st 16 bits of x

because it doesn't KNOW there are more to get !!! --WHY??? --
 Same problem for 32-bit 
64-bit calls and
whatever comes next. Remember Y2K!
16
©2010 D. J. Foreman
Operational Compatibility
 Thread controls


Pthreads on Windows (compatible w/POSIX)
POSIX threads




seen on Linux & Solaris
MS threads
Solaris threads
others (OS/2, MVS)
 System services (commands, signals, file


manipulation)
Process handling (creation, deletion)
Communication (IPC: msgs vs pipes)
17
©2010 D. J. Foreman
Memory
18
©2010 D. J. Foreman
Memory Problems
a.
b.
Little Endian (Intel): lowest # byte = highest value
Big Endian (Sun): highest # byte = highest value

Receiver must know data types
24
5
5*2
Original
(Sun)
as received
(Windows)
improper
correction
19
©2010 D. J. Foreman
Shared Memory
 Inhibits distribution (coherency lost, cost to build)
 When NOT shared, problems arise:



Consistency
Access (speed, mechanisms)
Compatibility (endian) etc.
 Atomicity

Program level
 Compare & Swap
 Test & OP (add, subtract, set-bit)
 Kernel/library supported

Semaphores, Mutexes, Condition variables
20
©2010 D. J. Foreman
Distributed Memory
 Partial solutions for heterogeneous systems



Small local (non-shared) memory
Bus and caches to a main store
Bus and caches to each processor
 Full solutions for multiprocessors

Interconnection (h/w switching) networks


Limited scalability
VERY expensive, but VERY fast
21
©2010 D. J. Foreman
2
Communications
22
©2010 D. J. Foreman
Primitives - 1
 Reliable – e.g.; TCP



Ack
Ordered
Guaranteed delivery
 Unreliable – e.g.; UDP




No ack
No guarantee
No ordering
(bottles in the ocean)
23
©2010 D. J. Foreman
Primitives - 2
 Blocking/non-blocking
 Buffered/unbuffered
 Distinguished messages



Send_get (sender is blocked)
Get_request
Send_reply (original sender is unblocked)
24
©2010 D. J. Foreman
Methods
 Shared variables - coherence??
 Message passing

RPC – Remote Procedure Call



RMI – Remote Method Invocation (also Remote Object
Invocation)




Very good for Client-server
Hides message passing
Like RPC, but with system-wide objects
Java server implements a Security Manager & registry
 Very much like DCE
MOM – Message Oriented Middleware
Streams (TCP/IP)

Needed for continuous information flow (video, audio)
25
©2010 D. J. Foreman
RPC Protocols
T0
Requestor
Receiver (server)
Request
Ack
Reply
Ack
Request
Ack
Reply
Request
Reply
Tn
26
©2010 D. J. Foreman
Marshaling
 Heterogeneous systems


No guarantee of data conformity
On guarantee of directionality (endian-ness)
 Data must be converted
 Data format must be standardized
 Communication types


Virtual circuits –reliable, flow controlled
Datagrams unreliable, no TCP “connect”
27
©2010 D. J. Foreman
RPC Steps
2
1. Client procedure calls client stub in normal way
2. Client stub builds message, calls local OS
3. Client's OS sends message to remote OS
4. Remote OS gives message to server stub
5. Server stub unpacks parameters, calls server
6. Server does work, returns result to the stub
7. Server stub packs it in message, calls local OS
8. Server's OS sends message to client's OS
9. Client's OS gives message to client stub
10.Stub unpacks result, returns to client
28
©2010 D. J. Foreman
Passing Value Parameters
a)
b)
c)
2
Original message on the PC
The message after receipt on a SPARC WS
The message after being inverted. Numbers in
dotted-boxes indicate the address of each byte
29
©2010 D. J. Foreman
Berkeley Sockets (1)
2
Socket
Create an endpoint from a connection
Bind
Assign a "handle" to the socket
Listen
Signal availability for connection requests
Accept
Allow a specific connection request to an ethereal
port, selected by the O/S, freeing the "listen" port
Send
Output data
Receive
Input data
Close
Delete the connection
Basic TCP/IP 'commands'
30
©2010 D. J. Foreman
Berkeley Sockets: Client-Server
2
Server
Socket
Bind
Listen
Accept
Read
Write
Close
t0
Client
Socket
Connect
Write
Read
Close
 Connection-oriented communication pattern using sockets.
 Note the ordering of Reads & Writes vs time.
31
©2010 D. J. Foreman
Berkeley Sockets: Peer-Peer
2
Peer 1
Socket, Bind,
Listen, Accept
Connect
Read
Write
Peer 2
Socket, Bind,
Listen, Accept

Connect
Read
Write
Read & Write are independent threads
32
©2010 D. J. Foreman
Berkeley Sockets: Producer-Consumer
2
Consumer
Socket
Bind
Listen
Accept
Read
Close
Producer
Socket, Bind

Connect
Write
Close
Note the one-way nature of communication
33
©2010 D. J. Foreman
4
Naming
34
©2010 D. J. Foreman
What is a name?
 Basic objects



File
Value
Program
 Data structures across a network


How to reference
How to modify
 Types


Human readable
Machine readable
35
©2010 D. J. Foreman
5
Synchronization
36
©2010 D. J. Foreman
Mechanisms
 Explicit signals

Semaphores, conditions, events
 Test and Set shared variables

Consider chip cycle vs. bus cycle
37
©2010 D. J. Foreman
Methods
 Semaphores
 Conditional critical sections
 Messages
 Event counts and sequencers
 Monitors
 Modules, common procedures and entries
 Path expressions
 Managers
 I/O commands
38
©2010 D. J. Foreman
Problems
 Race conditions - cause loss of data
 Action sequences - hierarchy to be maintained
 Data protection
39
©2010 D. J. Foreman
6
Consistency and
Replication
40
©2010 D. J. Foreman
World view
 Users must see same data at all times
 Updates must be controlled
 Rollbacks must maintain consistency
41
©2010 D. J. Foreman
7
Fault Tolerance
42
©2010 D. J. Foreman
Partitioning
 Problem: when is the system partitioned?



Response time?
Keep-alive?
Other?
 Solutions




Rollback?
Re-start?
Cut-off?
Other?
43
©2010 D. J. Foreman
8
Security
44
©2010 D. J. Foreman
Whose data is it?
 Must protect data and programs




Viruses
Worms
Errors
Theft
45
©2010 D. J. Foreman
8
Virtualization
46
©2010 D. J. Foreman
Problems
 Hardware differences on ends of real systems
 Time delays cannot be simulated
47
©2010 D. J. Foreman
8
Papers
48
©2010 D. J. Foreman
Introductory Papers
 The papers listed in the following section,
while dated in the early 1970s and 80's, form
some of the foundations of contemporary
implementations and research.
49
©2010 D. J. Foreman
Basic Concepts
 Synchronizing Resources, G. Andrews, 1981
 Specification of Synchronizing Processes, K.


Ramanathan, 1983
Concepts and Notations for Concurrent
Programming, G. Andrews, 1983
Distributed Operating Systems, A.
Tanenbaum, 1985
50
©2010 D. J. Foreman
Programming & Concept Papers
 The Temporal Semantics of Concurrent




Programs, A. Pneuli, 1981
Time, Clocks, and the Ordering of Events in a
D.S., L. Lamport, 1978
Determining the Last Process to Fail, D.
Skeen, 1985
Correctness Proofs of D. Termination
Algorithms, K. Apt, 1986
Monitors: An OS Structuring Concept, C.A.R.
Hoare, 1974
51
©2010 D. J. Foreman
Actual System Papers
 An Introduction to the V System, E. Berglund,


1986
The Design of the Saguaro D.O.S., G.
Andrews,1987
XMS: A Rendezvous-based D. S. Software
Architecture, N. Gammage, 1985
52
©2010 D. J. Foreman