Transcript Slide 1

Communication Issues
1
Synchronization and
Communication
• The correct behaviour of a concurrent program depends on
synchronization and communication between its processes.
• Synchronization: the satisfaction of constraints on the
interleaving of the actions of processes (e.g. an action by one
process only occurring after an action by another).
• Communication: the passing of information from one process to
another.
– Concepts are linked since communication requires
synchronization, and synchronization can be considered as
contentless communication.
– Data communication is usually based upon either shared
variables or message passing.
2
Communication primitives
• Communication primitives are the high
level construct with which programs use
the underlying communication network
• They play a significant role in the effective
usage of distributed systems
• The communication primitives influence a
programmers choice of algorithms
– Performance of the programs
3
Basic primitives
1. Send
2. Receive
• Send has two parameters
– Message and a destination
•
Receive has two parameters
– A source and a buffer
4
Buffered Message Passing
1. From user buffer to kernel buffer
2. From the kernel on the sending computer
to the kernel buffer on the receiving
computer
3. Finally from the buffer on the receiving
computer to a user buffer
5
Non blocking primitives
• The send primitive returns the control to the user
process as soon as the message Is copied from
the user buffer onto the kernel buffer
• The corresponding receive primitive signals its
intention to receive a message and provides a
buffer to copy the message
• The receiving process may either periodically
check for the arrival of a message or to be
signaled by the kernel upon arrival of a message
6
Advantage of Non blocking
primitives
7
Advantage of Non blocking
primitives
• Programs have maximum flexibility to
perform computation and communication
in any order they want
8
Disadvantage of Non blocking
primitives
9
Disadvantage of Non blocking
primitives
• Programming is tricky and difficult
• Programs may become time-dependent
where problems( or system states) are
irreproducible, making the programs very
difficult to debug
10
Unbuffered Message Passing
•
•
•
From one user buffer to another user buffer
directly
Program using send should avoid reusing the
buffer until the message has been transmitted
For large systems a combination of unbuffered
and non blocking semantics allows almost
complete overlap between the communication
and the on going computational activity in the
user program
11
Non blocking Message Passing
•
•
•
•
•
A natural use of non blocking communication
occurs in producer-consumer relationships
The consumer process can issue a non
blocking receive
If a message is present, the consumer process
reads it, otherwise it performs some other
computation.
The producer can issue non blocking sends
If a send fails for any reason (e.g. the buffer is
full), it can be retried later
12
Blocking primitives
• The send primitive does not return the control to the user
program until
– the message has been sent ( an unreliable blocking primitive) or
– an acknowledgement has been received ( a reliable blocking
primitive)
• In both case, user buffer can be reused as soon as the
control is returned to the user program
• The corresponding receive primitive does not return
control until a message is copied to the user buffer
• A reliable receive primitive automatically sends
acknowledgement
• An unreliable receive primitive does not send an
acknowledgement
13
Advantage of blocking primitives
14
Advantage of blocking primitives
• Behavior of the program is predictable
– Programming is easy
15
Disadvantage of blocking primitives
16
Disadvantage of blocking primitives
• Lack of flexibility in programming
• The absence of concurrency between
computation and communication
17
Synchronous primitive
• Send is blocked until a corresponding receive is
primitive is executed at the receiver end
• This is also called rendezvous
• A blocking synchronous primitive can be
extended to an unblocking synchronous
primitive by first copying the message to a buffer
at the sending side and then allowing the
process to perform other computational activity
except another send
18
How to convert a blocking Synchronous
primitive to unblocking Synchronous
primitive?
19
How to convert a blocking Synchronous
primitive to unblocking Synchronous
primitive?
• A blocking synchronous primitive can be
extended to an unblocking synchronous
primitive by
– first copying the message to a buffer at the sending
side and
– then allowing the process to perform other
computational activity except another send
20
Asynchronous primitive
• The messages are buffered with the
asynchronous primitives
• A send primitive does not block even if
there is no corresponding execution of a
receive primitive
• The corresponding receive primitive can
either be a blocking or a non blocking
primitive
21
Disadvantage of Asynchronous
primitive
22
Disadvantage of Asynchronous
primitive
• One disadvantage of buffering messages
is that it is more complex,
– as it involves creating , managing and
destroying buffers
• Another issue is what to do with the
messages that are meant for processes
that have already died
23
Message-Based Communication and
Synchronization
•
•
Use of a single construct for both synchronization and communication
Three issues:
– the model of synchronization
– the method of process naming
– the message structure
Process P1
Process P2
receive message
send message
time
time
24
Process Synchronization
•
•
Variations in the process synchronization model arise from the semantics
of the send operation
Asynchronous (or no-wait) (e.g. POSIX)
– Requires buffer space. What happens when the buffer is full?
Process P1
Process P2
send message
message
receive message
time
time
25
Process Synchronization
•
Synchronous (e.g. CSP, occam2)
– No buffer space required
– Known as a rendezvous
Process P1
Process P2
send message
blocked
time
M
receive message
time
26
Process Synchronization
•
•
Remote invocation (e.g. Ada)
– Known as an extended rendezvous
Analogy:
– The posting of a letter is an asynchronous send
– A telephone is a better analogy for synchronous communication
Process P1
Process P2
send message
M
receive message
blocked
reply
time
time
27
Asynchronous and Synchronous Sends
28
Asynchronous and Synchronous Sends
•
Asynchronous communication can implement synchronous communication:
P1
P2
asyn_send (M)
wait (M)
wait (ack)
asyn_send (ack)
29
Asynchronous and Synchronous Sends
•
•
Asynchronous communication can implement synchronous communication:
P1
P2
asyn_send (M)
wait (M)
wait (ack)
asyn_send (ack)
Two synchronous communications can be used to construct a remote
invocation:
P1
P2
syn_send (message)
wait (message)
wait (reply)
...
construct reply
...
syn_send (reply)
30
Disadvantages of Asynchronous Send
31
Disadvantages of Asynchronous Send
• Potentially infinite buffers are needed to store unread
messages
• Asynchronous communication is out-of-date; most sends
are programmed to expect an acknowledgement
• More communications are needed with the asynchronous
model, hence programs are more complex
• It is more difficult to prove the correctness of the complete
system
• Where asynchronous communication is desired with
synchronised message passing then buffer processes can
easily be constructed; however, this is not without cost
32
Process Naming
• Two distinct sub-issues
– direction versus indirection
– symmetry
• With direct naming, the sender explicitly names the receiver:
send <message> to <process-name>
• With indirect naming, the sender names an intermediate entity
(e.g. a channel, mailbox, link or pipe):
send <message> to <mailbox>
• With a mailbox, message passing can still be synchronous
• Direct naming has the advantage of simplicity, whilst indirect
naming aids the decomposition of the software; a mailbox can be
seen as an interface between parts of the program
33
Process Naming
• A naming scheme is symmetric if both sender and receiver name each other
(directly or indirectly)
send <message> to <process-name>
wait <message> from <process-name>
send <message> to <mailbox>
wait <message> from <mailbox>
• It is asymmetric if the receiver names no specific source but accepts
messages from any process (or mailbox)
wait <message>
• Asymmetric naming fits the client-server paradigm
• With indirect the intermediary could have:
– a many-to-one structure – a many-to-many structure
– a one-to-one structure
– a one-to-many
34
Message Structure
• A language usually allows any data object of any
defined type (predefined or user) to be transmitted in
a message
• Need to convert to a standard format for transmission
across a network in a heterogeneous environment
• OS allow only arrays of bytes to be sent
35
Message Characteristics
Principal Operations
Anonymity
• send
• anonymous message passing
• receive
• non-anonymous message passing
Synchronization
Receipt of Messages
• Synchronous
• Unconditional
• Asynchronous
• Selective
• Rendevous
Multiplicity
• one-one
• many-one
• many-many
• Selective
36
Selective waiting
• So far, the receiver of a message must wait until the
specified process, or mailbox, delivers the
communication
• A receiver process may actually wish to wait for any
one of a number of processes to call it
• Server processes receive request messages from a
number of clients; the order in which the clients call
being unknown to the servers
• To facilitate this common program structure, receiver
processes are allowed to wait selectively for a
number of possible messages
37
Non-determinism and Selective Waiting
• Concurrent languages make few assumptions about
the execution order of processes
• A scheduler is assumed to schedule processes nondeterministically
• Consider a process P that will execute a selective wait
construct upon which processes S and T could call
38
Non-determinism and Selective
Waiting
• P runs first; it is blocked on the select. S (or T) then runs and rendezvous with
P
• S (or T) runs, blocks on the call to P; P runs and executes the select; a
rendezvous takes place with S (or T)
• S (or T) runs first and blocks on the call to P; T (or S) now runs and is also
blocked on P. Finally P runs and executes the select on which T and S are
waiting
• The three possible interleavings lead to P having none, one or two calls
outstanding on the selective wait
• If P, S and T can execute in any order then, in latter case, P should be able to
choose to rendezvous with S or T — it will not affect the programs
correctness
39
Non-determinism and Selective
Waiting
• A similar argument applies to any queue that a
synchronisation primitive defines
• Non-deterministic scheduling implies all queues
should release processes in a non-deterministic
order
• Semaphore queues are often defined in this way;
entry queues and monitor queues are specified to
be FIFO
• The rationale here is that FIFO queues prohibit
starvation but if the scheduler is non-deterministic
then starvation can occur anyway!
40
POSIX Message Queues
• POSIX supports asynchronous, indirect message passing through
the notion of message queues
• A message queue can have many readers and many writers
• Priority may be associated with the queue
• Intended for communication between processes (not threads)
• Message queues have attributes which indicate their maximum
size, the size of each message, the number of messages currently
queued etc.
• An attribute object is used to set the queue attributes when the
queue is created
41
POSIX Message Queues
• Message queues are given a name when they are created
• To gain access to the queue, requires an mq_open name
• mq_open is used to both create and open an already existing
queue (also mq_close and mq_unlink)
• Sending and receiving messages is done via mq_send and
mq_receive
• Data is read/written from/to a character buffer.
• If the buffer is full or empty, the sending/receiving process is
blocked unless the attribute O_NONBLOCK has been set for the
queue (in which case an error return is given)
• If senders and receivers are waiting when a message queue
becomes unblocked, it is not specified which one is woken up
unless the priority scheduling option is specified
42
POSIX Message Queues
• A process can also indicate that a signal should be sent to it
when an empty queue receives a message and there are no
waiting receivers
• In this way, a process can continue executing whilst waiting for
messages to arrive or one or more message queues
• It is also possible for a process to wait for a signal to arrive; this
allows the equivalent of selective waiting to be implemented
• If the process is multi-threaded, each thread is considered to be a
potential sender/receiver in its own right
43
Remote Procedure Call (RPC)
Remote Procedure Call is a procedure P that
caller process C gets server process S to execute
as if C had executed P in C's own address space
RPCs support
distributed computing at higher level than sockets
architecture/OS neutral passing of simple & complex data types
common application needs like name resolution, security etc.
caller process
Call procedure
and wait for reply
server process
Receive request and
start procedure
execution
Procedure P
executes
Resume
execution
Send reply and wait
for the next request
44
Remote Procedure Call (RPC)
RPC Standards
There are at least 3 widely used forms of RPC
Open Network Computing version of RPC
Distributed Computing Environment version of RPC
Microsoft's COM/DCOM proprietary standard
ONC RPC specified in RFC 1831
is widely available RPC standard on UNIX and other OSs
has been developed by Sun Microsystems
Distributed Computing Environment
is widely available RPC standard on many OSs
supports RPCs, directory of RPC servers & security services
COM is proprietary, adaptive extension of DCE standard
Java RMI
45
Event Ordering
X,Y, Z and A on a mailing list
1. X sends a message with the subject Meeting
2. Y and Z reply by sending a, Message with subject Re: Meeting
In real-time,
X’s message was sent first
Y reads and reply
Z reads both X and Y’s reply and reply again
But due to independent delay A might see
Item
from
Subject
23
Z
Re: Meeting
24
X
Meeting
25
Y
Re: Meeting
46
Event Ordering
If the clocks of X, Y, Z’s computer synchronized, then each message would carry
time on the local computer’s clock
For example, messages m1, m2, and m3 and
time t1, t2, and t3 where
t1,<t2<t3
Since clocks can not be synchronized the perfectly logical clock was introduced
by Lamport
Logically a message was received after it was sent
X sends m1 before Y receives m1;
Y send m2 before X receives m2
Replies are sent after receiving messages
Y receives m1 before sending m2
Logical time takes the idea further by assigning a number to each event
corresponding to its logical ordering.
Figure shows numbers 1 to 4 on the events at X and Y.
47
Real-time ordering of events
s end
X
receive
1
m1
2
Y
receive
4
s end
3
m2
receive
Physic al
ti me
receive
s end
Z
receive
receive
m3
A
t1
t2
m1
m2
receive receive receive
t3
Instructor’s Guide for Coulouris, Dollimore and Kindberg Distributed Systems: Concepts and Design Edn. 3
© Addison-Wesley Publishers 2000
48
The happened-before relation
The happened-before relation (denoted by ) is defined as
follows:
Rule 1 : If a and b are events in the same process and a was executed
before b, then a  b.
Rule 2 : If a is the event of sending a message by one process and b
is the event of receiving that message by another process, then a  b.
Rule 3 : If a  b and b  c, then a  c.
Relationship between two events.
Two events a and b are causally related if a  b or b  a.
Two distinct events a and b are said to be concurrent if a  b
and b  a (denoted as a b).
49
A time-space representation
A time-space view of a distributed
system.
Rule 1:
a0  a1  a2  a3
b0  b1  b2  b3
c0  c1  c2  c3
Rule 2:
a0  b3
b1  a3; b2  c1; b0  c2
50
States
State model:
A process executes three types of events: internal actions,
send actions, and receive actions.
Global state:
A collection of local states and the state of all
the communication channels.
Process
Process
receive
send
Process
Process
System structure from logical point of view.
51
References
George Coulouris, Jean Dollimore and Tim Kindberg. Distributed Systems: Concepts and
Design (Edition 3 ). Addison-Wesley 2001 http://www.cdk3.net/
Andrew S. Tanenbaum, Maarten van Steen. Distributed Systems: Principles and Paradigms.
Prentice-Hall 2002.
http://www.cs.vu.nl/~ast/books/ds1/
P. K. Sinha, P.K. "Distributed Operating Systems, Concepts and Design", IEEE Press, 1993
Sape J. Mullender, editor. Distributed Systems, 2nd edition, ACM Press, 1993
http://wwwhome.cs.utwente.nl/~sape/gos0102.html
Gregory R. Andrews. Foundations of Multithreaded, Parallel, and Distributed
Programming, Addison Wesley, 2000
http://www.cs.arizona.edu/people/greg/
J. Magee & J. Kramer. Concurrency. State Models and Java Programs. John Wiley, 1999
http://www-dse.doc.ic.ac.uk/concurrency/
52