Transcript Chapter 4 -- Gadi Taubenfeld

Synchronization Algorithms
and Concurrent Programming
Gadi Taubenfeld
Chapter 4
Blocking and Non-blocking Synchronization
Version: June 2014
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
1
Synchronization Algorithms
and Concurrent Programming
ISBN: 0131972596, 1st edition
A note on the use of these power-point slides:
I am making these slides freely available to all (faculty, students, readers).
They are in PowerPoint form so you can add, modify, and delete slides and slide
content to suit your needs. They obviously represent a lot of work on my part.
In return for use, I only ask the following:
 That you mention their source, after all, I would like people to use my book!
 That you note that they are adapted from (or perhaps identical to)
my slides, and note my copyright of this material.
Thanks and enjoy!
Gadi Taubenfeld
All material copyright 2014
Gadi Taubenfeld, All Rights Reserved
To get the most updated version of these slides go to:
http://www.faculty.idc.ac.il/gadi/book.htm
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
2
Chapter 4
Blocking and Non-blocking
Synchronization
4.1 Synchronization Primitives
4.2 Collision Avoidance using Test-and-set Bits
4.3 The Ticket Algorithm using RMW Objects
4.4 Local Spinning using Strong Primitives
4.5 Concurrent Data Structures
-- Progress Conditions
-- Consistency Conditions
-- Non-blocking Queue
-- Memory Barriers
These topics are
covered in this
presentation
4.6 Semaphores
4.7 Monitors
4.8 Fairness of Shared Objects
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
3
Concurrent Data Structures
Using locks
P1
P2
Without locks
P3
data structure
Chapter 4
P1
P2
P3
data structure
structure
data
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
4
Concurrent Data Structures
Using locks
P1
P2
Without locks
P3
data structure
P1
P2
P3
data structure
structure
data
• simple programming model
• resilient to failures, etc.
• false conflicts
• often complex
• fault-free solutions only
• memory consuming
• sequential bottleneck
• sometime -- weak progress cond.
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
5
Progress Conditions
Section 4.5.1
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
6
Progress Conditions
Using locks
P1
P2
lock-free
P1
P3
data structure
deadlock
(livelock)
freedom
Chapter 4
starvation
freedom
P2
P3
data structure
structure
data
FIFO
(+ df )
obstruction
freedom
non-blocking
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
wait
freedom
7
Obstruction-freedom
P1
P2
P3
P4
Done
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
8
Non-blocking
P1
P2
P3
P4
Done
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
9
Wait-freedom
P1
Chapter 4
P2
P3
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
P4
10
Wait-freedom
Chapter 4
P1
P2
P3
P4
Done
Done
Done
Done
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
11
Lock-free Data Structures
Obstruction-freedom
• too weak progress condition
• not complex
Chapter 4
Non-blocking
Wait-freedom
• strong enough
• strong/desirable
• not so complex
• complex/less efficient
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
12
Consistency Conditions
How do we define the correctness of a concurrent object ?
 Linearizability
 Sequential Consistency
 …
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
13
Why are consistency conditions important
 Formal specification of concurrent objects
 Help to understand possible behaviors of an implementation
of a concurrent object and to reason about correctness
 Clarifies what optimizations are acceptable
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
14
Example: Concurrent Queue
Chapter 4
q.enq(
)
q.deq(
)
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
15
Sequential Execution
Is this execution reasonable?
q.enq
P1
P2
Yes
q.deq
q.enq
q.deq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
16
Concurrent Execution
Is this execution reasonable?
q.enq
P1
P2
???
q.deq
q.enq
q.deq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
17
Concurrent Execution
Is this execution reasonable?
q.enq
P1
P2
Yes
q.deq
q.enq
q.deq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
18
Concurrent Execution
Is this execution reasonable?
q.enq
P1
P2
Yes
q.deq
q.enq
q.deq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
19
Sequential Specification of an object
Queue: In sequential executions, the enqueue operation
inserts a value to the queue and the dequeue operation
returns and deletes the oldest value in the queue.
Sequential
specification
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
21
Linearizability
for an object
Each concurrent execution is
“equivalent’’ to an execution in
the sequential specification
concurrent
executions
Chapter 4
Each sequential execution
is included in the
sequential specification
Sequential
executions
All executions
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
22
Linearizability
for an object
 Need to preserve real time order
 Each operation should appear to “take effect” instantaneously
at some moment between its invocation and response
 Object is correct if this “sequential” behavior is correct
q.enq
P1
P2
q.deq
q.enq
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
q.deq
23
Linearizability
 Permits programmers to specify and reason about concurrent
objects using known techniques from the sequential domain
 provides the illusion that each operation applied by concurrent
processes takes effect instantaneously between its invocation
and its response
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
24
Terminology
Invocation
Linearization
Point
Response
q.enq
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
25
Linearizability
Is this execution linearizable?
q.enq
P1
P2
Yes
q.deq
q.enq
q.deq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
26
Linearizability
Is this execution linearizable?
q.enq
P1
P2
Yes X 2
q.deq
q.enq
q.deq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
27
Linearizability
Is this execution linearizable?
P1
q.enq
P2
No, BUT …
q.deq
q.enq
q.deq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
28
Linearizability
Is this execution linearizable?
P1
q.enq
P2
No, BUT …
q.deq
q.enq
q.deq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
29
Linearizability
Sequential Consistency
Is this execution linearizable?
P1
q.enq
P2
No, BUT …
q.deq
q.enq
q.deq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
30
Sequential Consistency
 No need to preserve real time order
– Cannot re-order operations done by the same thread
– Can re-order non-overlapping operations done by different
threads
 Each operation should “take effect” instantaneously
between invocation and response events
 Object is correct if this “sequential” behavior is
correct
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
31
Sequential Consistency
Is this execution sequential consistent? Yes
P1
q.enq
P2
q.deq
q.enq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
32
Sequential Consistency
Is this execution sequential consistent? Yes
Its also linearizable!
p.enq
P1
P2
p.enq
p.deq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
33
Sequential Consistency
Is this execution sequential consistent?
p.enq
P1
P2
p.enq
p.deq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
34
Sequential Consistency
Is this execution sequential consistent? No !!!
P1
q.enq
P2
p.enq
p.enq
q.deq
q.enq
p.deq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
35
Sequential Consistency
Is this execution sequential consistent? No !!!
P1
q.enq
P2
p.enq
p.enq
q.deq
q.enq
p.deq
time
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
36
Compositionality
Theorem: Sequential consistency is not compositional.
Proof: The example from the previous slide.
Theorem: Linearizability is compositional.
Proof: ...
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
37
A Non-blocking Concurrent Queue Algorithm
Section 4.5.2
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
Implementing a Non-blocking Queue
enqueue
Chapter 4
dequeue
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
39
Implementing a Non-blocking Queue
Empty queue
tail
head
Dummy Node
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
40
Enqueue
lnode
Chapter 4
tail
head
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
41
2 concurrent enqueue operations
What can go wrong when using registers?
tail
Chapter 4
head
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
42
Dequeue
tail
Chapter 4
head
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
43
2 concurrent dequeue operations
What can go wrong when using registers?
tail
Chapter 4
head
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
44
Enqueue + Dequeue
(Non-empty queue)
tail
Chapter 4
head
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
45
Enqueue + Dequeue
(Empty queue)
tail
Chapter 4
head
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
46
Enqueue + Dequeue
(Empty queue)
tail
head
Lost tail !!!
The dequeuer must advance tail before redirecting head
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
47
Impossibility Result
Theorem: There is a wait-free consensus algorithm for 2
processes, using queues and atomic registers.
Theorem: There is NO wait-free (or non-blocking) consensus
algorithm for 2 processes, using only atomic registers.
There is NO wait-free or non-blocking implementation of
a queue for two processes or more, from atomic registers.
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
48
Compare & swap (CAS)
shared
register
old
new
CAS ( A, B, C )
if A=B then A:=C; return(true)
else return(false)
Supported by Sun, Intel, AMD, …
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
49
CAS: The ABA problem
shared
register
old
new
local:= A
CAS ( A, B, C )
if A=B then A:=C
CAS (A, local, 100)
local:
value
tag
A:
value
tag
CAS (A, local, <100,local.tag+1> )
In the following, we will ignore the ABA problem and assume that it
Synchronization Algorithms
and Concurrent Programming
Chapter
4
50
is resolved
using additional
tag fields.
Gadi Taubenfeld © 2014
Section 4.5
A Non-blocking Queue
Chapter 4

The algorithm is due to M. M. Michael and M.
L. Scott (1996).

The algorithm is included in the Standard Java
Concurrency Package.

Every process must be prepared to encounter
a half-finished enqueue operation, and help to
finish it.

Invariant: The tail refers to either the last
node or to the node just before the last node.
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
51
lnode
Enqueue
tail
head
ltail
true
false
done
repeat
ltail  tail
lnext
lnext  ltail.next
if ltail = tail then
if lnext = NULL then
if CAS (ltail.next, lnext, lnode) then done  true fi
else CAS (tail, ltail, lnext) fi fi
until done = true
CAS (tail, ltail, lnode)
Chapter 4
try to link lnode to
the end
queue
enqueue failed:
tryof
tothe
swing
tail to
a node
that
inserted
enqueue
done:
trywas
to swing
tailby
to some
other process
the inserted
node.
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
52
lnode
Enqueue
tail
head
ltail
true
false
done
repeat
ltail  tail
lnext
lnext  ltail.next
if ltail = tail then
if lnext = NULL then
if CAS (ltail.next, lnext, node) then done  true fi
else CAS (tail, ltail, lnext) fi fi
until done = true
CAS (tail, ltail, lnode)
Chapter 4
Question: Would the enqueue
operation be correct if the last
line is omitted?
Answer: Yes, but…
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
53
lnode
Enqueue
tail
head
ltail
true
false
done
repeat
ltail  tail
lnext
lnext  ltail.next
if ltail = tail then
if lnext = NULL then
Question: Would the enqueue
operation be correct if this line
is omitted?
if CAS (ltail.next, lnext, node) then done  true fi
else CAS (tail, ltail, lnext) fi fi
until done = true
CAS (tail, ltail, lnode)
Chapter 4
No, when it is assumed that cells can
be released and then reallocated.
Otherwise, yes.
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
54
tail
Dequeue
repeat
head
lvalue
lhead  head
ltail  tail
lnext  lhead.next
lnext
true
false
done
ltail
lhead
if lhead = head then
if lhead = ltail then
/* empty or tail behind?
if lnext = NULL then return(NULL) fi ;
/* empty
CAS (tail, ltail, lnext)
/* try to advance tail
else lvalue  lnext.value
/* no need to deal with tail
if CAS (head, lhead, lnext) then done  true fi fi fi
until done = true
free(lhead) ; return(lvalue)
Synchronization Algorithms and Concurrent Programming
Chapter 4
Gadi Taubenfeld © 2014
55
tail
Dequeue
repeat
head
lvalue
lhead  head
ltail  tail
lnext  lhead.next
lnext
true
false
done
ltail
lhead
Question: Would the dequeue operation
be correct if this line is omitted?
if lhead = head then
if lhead = ltail then
/* empty or tail behind?
if lnext = NULL then return(NULL) fi ;
/* empty
CAS (tail, ltail, lnext)
/* try to advance tail
else lvalue  lnext.value
/* no need to deal with tail
if CAS (head, lhead, lnext) then done  true fi fi fi
until done = true
free(lhead) ; return(lvalue)
Synchronization Algorithms and Concurrent Programming
Chapter 4
Gadi Taubenfeld © 2014
56
A note about
Memory Barriers
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
Example: Using Flags
x and y : atomic bits, initially 0
x
0
y
0
Process A
Process B
write.x(1)
write.y(1)
read.y
read.x
Q: Is it possible that both
processes read the value 0 ?
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
58
Example: Using Flags
x and y : atomic bits, initially 0
x
0
y
0
Process A
Process B
write.x(1)
write.y(1)
read.y
read.x
Fact: Many hardware architectures do
not support sequential consistency
because they think it is too strong 
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
59
Example: Using Flags
x and y : atomic bits, initially 0
x
0
y
0
Process A
Process B
write.x(1)
write.y(1)
read.y
read.x
Solution: Memory barriers
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
60
Binary Semaphores
Section 4.6
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
Section 4.6
Binary Semaphores
Operations
down(S)
up(S)
1
Types
unfair semaphore
weak semaphore
strong semaphore
down(S)
 if S > 0, then S = 0 otherwise,
the process is blocked until the
value becomes greater than 0.
 Testing and decrementing the
semaphore are executed
atomically without interruption.
up(S)

Chapter 4
S=1
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
62
Semaphores
Deadlock-free Mutual exclusion Algorithm
down(S)
critical section
up(S)
Chapter 4
1
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
63
Semaphores
Constant Space Starvation-free Algorithm
Weak semaphores
queue
S.0 1
10
S.1 1
Shared bits
queue 0
S.0 10
1
1
S.1 0
empty false
local bits
doorkeeper
critical
section
Chapter 4
empty
myqueue
true
false
otherqueue
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
64
Semaphores
Constant Space Starvation-free Algorithm
Weak semaphores
queue
S.0 1
1
S.1 1
Shared bits
queue 0
S.0 0
1
S.1 0
empty false
local bits
critical
section
Chapter 4
empty
myqueue
true
false
otherqueue
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
65
Semaphores
Constant Space Starvation-free Algorithm
Weak semaphores
queue
S.0 1
1
S.1 1
Shared bits
queue 0
S.0 1
S.1 0
empty false
local bits
critical
section
Chapter 4
empty
myqueue
true
false
otherqueue
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
66
Semaphores
Constant Space Starvation-free Algorithm
Weak semaphores
queue
S.0 1
1
S.1 1
Shared bits
queue 0
S.0 10
1
S.1 0
empty false
local bits
critical
section
Chapter 4
empty
myqueue
true
false
otherqueue
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
67
Semaphores
Constant Space Starvation-free Algorithm
Weak semaphores
queue
S.0 1
1
S.1 1
Shared bits
queue 0
S.0 1
S.1 0
empty false
local bits
critical
section
Chapter 4
empty
myqueue
true
false
otherqueue
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
68
Semaphores
Constant Space Starvation-free Algorithm
Weak semaphores
queue
S.0 1
1
S.1 1
Shared bits
queue 0
S.0 10
1
S.1 0
empty false
local bits
critical
section
Chapter 4
empty
myqueue
true
false
otherqueue
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
69
Semaphores
Constant Space Starvation-free Algorithm
Weak semaphores
queue
S.0 1
1
S.1 1
Shared bits
queue 0
S.0 0
1
S.1 10
empty false
local bits
critical
section
Chapter 4
empty
myqueue
true
otherqueue
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
70
Semaphores
Constant Space Starvation-free Algorithm
myeuque = queue
down (S.myqueue)
if queue = myqueue then
otherqueue = 1 – myqueue
down(S.otherqueue)
queue = otherqueue
repeat
empty = true
up(S.myqueue)
down(S.myqueue)
until empty
critical section
up(S.otherqueue)
else empty = false
critical section fi
up(S.myqueue)
Chapter 4
Weak semaphores
S.0 1
S.1 1
Shared bits
queue 0
empty false
local bits
myqueue
otherqueue
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
71
Semaphores
Constant Space Starvation-free Algorithm
Question:
Is it possible
Yes
that some process will
be in its CS, while no
process is a doorkeeper?
queue
10
S.0 10
1
S.1 1
empty
critical
section
Chapter 4
true
false
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
72
Properties of the Algorithm
 Satisfies mutex & deadlock-freedom.
 Satisfies 2-bounded-waiting: A process can enter its
CS ahead of anoter only twice.
 There is no need to know the # of processes
 System response time is a constant
 Two weak semaphores and two atomic bits are used.
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
73
The End
Chapter 4
Synchronization Algorithms and Concurrent Programming
Gadi Taubenfeld © 2014
74