Transcript Pregel: A System for Large

Pregel:
A System for Large-Scale Graph Processing
Grzegorz Malewicz, Matthew H. Austern, Aart J. C. Bik,
James C. Dehnert, Ilan Horn, Naty Leiser, and Grzegorz Czajkwoski
Google, Inc.
SIGMOD ’10
7 July 2010
Taewhi Lee
Outline
 Introduction
 Computation Model
 Writing a Pregel Program
 System Implementation
 Experiments
 Conclusion & Future Work
2
Outline
 Introduction
 Computation Model
 Writing a Pregel Program
 System Implementation
 Experiments
 Conclusion & Future Work
3
Introduction (1/2)
Source: SIGMETRICS ’09 Tutorial – MapReduce: The Programming Model and Practice, by Jerry Zhao
4
Introduction (2/2)
 Many practical computing problems concern large graphs
Large graph data
Graph algorithms
Web graph
Transportation routes
Citation relationships
Social networks
PageRank
Shortest path
Connected components
Clustering techniques
 MapReduce is ill-suited for graph processing
– Many iterations are needed for parallel graph processing
– Materializations of intermediate results at every MapReduce iteration
harm performance
5
Single Source Shortest Path (SSSP)
 Problem
– Find shortest path from a source node to all target nodes
 Solution
– Single processor machine: Dijkstra’s algorithm
6
Example: SSSP – Dijkstra’s Algorithm
1


10
2
0
9
3
5
4
6
7

2
7

Example: SSSP – Dijkstra’s Algorithm
1
10

10
2
0
9
3
5
4
6
7
5
2
8

Example: SSSP – Dijkstra’s Algorithm
1
8
14
10
2
0
9
3
5
4
6
7
5
2
9
7
Example: SSSP – Dijkstra’s Algorithm
1
8
13
10
2
0
9
3
5
4
6
7
5
2
10
7
Example: SSSP – Dijkstra’s Algorithm
1
8
9
10
2
0
9
3
5
4
6
7
5
2
11
7
Example: SSSP – Dijkstra’s Algorithm
1
8
9
10
2
0
9
3
5
4
6
7
5
2
12
7
Single Source Shortest Path (SSSP)
 Problem
– Find shortest path from a source node to all target nodes
 Solution
– Single processor machine: Dijkstra’s algorithm
– MapReduce/Pregel: parallel breadth-first search (BFS)
13
MapReduce Execution Overview
14
Example: SSSP – Parallel BFS in MapReduce
 Adjacency matrix
A
A
B
C
10
B
D
E
1
2
4
D
3
7
9
2
C
1

5
C
E
B
A

10
0
2
9
3
4
6
6
5
 Adjacency List
7

A: (B, 10), (D, 5)
B: (C, 1), (D, 2)
D
C: (E, 4)
D: (B, 3), (C, 9), (E, 2)
E: (A, 7), (C, 6)
15
2

E
Example: SSSP – Parallel BFS in MapReduce
 Map input: <node ID, <dist, adj list>>
B
<A, <0, <(B, 10), (D, 5)>>>

<B, <inf, <(C, 1), (D, 2)>>>
<C, <inf, <(E, 4)>>>
<D, <inf, <(B, 3), (C, 9), (E, 2)>>>
<E, <inf, <(A, 7), (C, 6)>>>
A
2

9
3
5
4
6
7

<B, 10> <D, 5>
<A, <0, <(B, 10), (D, 5)>>>
<C, inf> <D, inf>
<B, <inf, <(C, 1), (D, 2)>>>
<E, inf>
<C, <inf, <(E, 4)>>>
<B, inf> <C, inf> <E, inf>
<D, <inf, <(B, 3), (C, 9), (E, 2)>>>
<A, inf> <C, inf>
<E, <inf, <(A, 7), (C, 6)>>>
16
1
10
0
 Map output: <dest node ID, dist>
C
D
2

E
Flushed to local disk!!
Example: SSSP – Parallel BFS in MapReduce
 Reduce input: <node ID, dist>
B
<A, <0, <(B, 10), (D, 5)>>>
C
1

<A, inf>
<B, <inf, <(C, 1), (D, 2)>>>
A
<B, 10> <B, inf>
0
<C, <inf, <(E, 4)>>>
10
2
9
3
5
<C, inf> <C, inf> <C, inf>
<D, 5> <D, inf>
D
<E, <inf, <(A, 7), (C, 6)>>>
<E, inf> <E, inf>
17
4
6
7

<D, <inf, <(B, 3), (C, 9), (E, 2)>>>

2

E
Example: SSSP – Parallel BFS in MapReduce
 Reduce input: <node ID, dist>
B
<A, <0, <(B, 10), (D, 5)>>>
C
1

<A, inf>
<B, <inf, <(C, 1), (D, 2)>>>
A
<B, 10> <B, inf>
0
<C, <inf, <(E, 4)>>>
10
2
9
3
5
<C, inf> <C, inf> <C, inf>
<D, 5> <D, inf>
D
<E, <inf, <(A, 7), (C, 6)>>>
<E, inf> <E, inf>
18
4
6
7

<D, <inf, <(B, 3), (C, 9), (E, 2)>>>

2

E
Example: SSSP – Parallel BFS in MapReduce
 Reduce output: <node ID, <dist, adj list>>
= Map input for next iteration
<A, <0, <(B, 10), (D, 5)>>>
Flushed to DFS!!
<B, <10, <(C, 1), (D, 2)>>>
A
<C, <inf, <(E, 4)>>>
<D, <5, <(B, 3), (C, 9), (E, 2)>>>
 Map output: <dest node ID, dist>
1
10
2

9
3
5
<B, 10> <D, 5>
<A, <0, <(B, 10), (D, 5)>>>
<C, 11> <D, 12>
<B, <10, <(C, 1), (D, 2)>>>
<E, inf>
<C, <inf, <(E, 4)>>>
<B, 8> <C, 14> <E, 7>
<D, <5, <(B, 3), (C, 9), (E, 2)>>>
<A, inf> <C, inf>
<E, <inf, <(A, 7), (C, 6)>>>
19
C
10
0
<E, <inf, <(A, 7), (C, 6)>>>
B
4
6
7
5
D
2

E
Flushed to local disk!!
Example: SSSP – Parallel BFS in MapReduce
 Reduce input: <node ID, dist>
B
<A, <0, <(B, 10), (D, 5)>>>
C
1
10
<A, inf>
<B, <10, <(C, 1), (D, 2)>>>
A
<B, 10> <B, 8>
0
<C, <inf, <(E, 4)>>>
10
2
9
3
5
<C, 11> <C, 14> <C, inf>
<D, 5> <D, 12>
D
<E, <inf, <(A, 7), (C, 6)>>>
<E, inf> <E, 7>
20
4
6
7
5
<D, <5, <(B, 3), (C, 9), (E, 2)>>>

2

E
Example: SSSP – Parallel BFS in MapReduce
 Reduce input: <node ID, dist>
B
<A, <0, <(B, 10), (D, 5)>>>
C
1
10
<A, inf>
<B, <10, <(C, 1), (D, 2)>>>
A
<B, 10> <B, 8>
0
<C, <inf, <(E, 4)>>>
10
2
9
3
5
<C, 11> <C, 14> <C, inf>
<D, 5> <D, 12>
D
<E, <inf, <(A, 7), (C, 6)>>>
<E, inf> <E, 7>
21
4
6
7
5
<D, <5, <(B, 3), (C, 9), (E, 2)>>>

2

E
Example: SSSP – Parallel BFS in MapReduce
 Reduce output: <node ID, <dist, adj list>>
= Map input for next iteration
<A, <0, <(B, 10), (D, 5)>>>
Flushed to DFS!!
<B, <8, <(C, 1), (D, 2)>>>
<C, <11, <(E, 4)>>>
<D, <5, <(B, 3), (C, 9), (E, 2)>>>
A
C
1
8
2
9
3
5
D
22
4
6
7
5
… the rest omitted …
11
10
0
<E, <7, <(A, 7), (C, 6)>>>
B
2
7
E
Outline
 Introduction
 Computation Model
 Writing a Pregel Program
 System Implementation
 Experiments
 Conclusion & Future Work
23
Computation Model (1/3)
Input
Supersteps
(a sequence of iterations)
Output
24
Computation Model (2/3)
 “Think like a vertex”
 Inspired by Valiant’s Bulk Synchronous Parallel model (1990)
Source: http://en.wikipedia.org/wiki/Bulk_synchronous_parallel
25
Computation Model (3/3)
 Superstep: the vertices compute in parallel
– Each vertex
 Receives messages sent in the previous superstep
 Executes the same user-defined function
 Modifies its value or that of its outgoing edges
 Sends messages to other vertices (to be received in the next superstep)
 Mutates the topology of the graph
 Votes to halt if it has no further work to do
– Termination condition
 All vertices are simultaneously inactive
 There are no messages in transit
26
Example: SSSP – Parallel BFS in Pregel
1


10
2
0
9
3
5
4
6
7

2
27

Example: SSSP – Parallel BFS in Pregel
10

2
9
3

5
5



10
0

1

4
7

2
28

6



Example: SSSP – Parallel BFS in Pregel
1
10

10
2
0
9
3
5
4
6
7
5
2
29

Example: SSSP – Parallel BFS in Pregel
2
5

14
8
10
0
11
1
10
9
3
12
4
6
7
5
2
30
7

Example: SSSP – Parallel BFS in Pregel
1
8
11
10
2
0
9
3
5
4
6
7
5
2
31
7
Example: SSSP – Parallel BFS in Pregel
9
1
8
11
10
0
14
13
2
9
3
5
4
7
5
2
32
6
15
7
Example: SSSP – Parallel BFS in Pregel
1
8
9
10
2
0
9
3
5
4
6
7
5
2
33
7
Example: SSSP – Parallel BFS in Pregel
1
8
9
10
2
0
9
3
5
4
7
5
2
34
6
13
7
Example: SSSP – Parallel BFS in Pregel
1
8
9
10
2
0
9
3
5
4
6
7
5
2
35
7
Differences from MapReduce
 Graph algorithms can be written as a series of chained
MapReduce invocation
 Pregel
– Keeps vertices & edges on the machine that performs computation
– Uses network transfers only for messages
 MapReduce
– Passes the entire state of the graph from one stage to the next
– Needs to coordinate the steps of a chained MapReduce
36
Outline
 Introduction
 Computation Model
 Writing a Pregel Program
 System Implementation
 Experiments
 Conclusion & Future Work
37
C++ API
 Writing a Pregel program
– Subclassing the predefined Vertex class
Override this!
in msgs
out msg
38
Example: Vertex Class for SSSP
39
Outline
 Introduction
 Computation Model
 Writing a Pregel Program
 System Implementation
 Experiments
 Conclusion & Future Work
40
System Architecture
 Pregel system also uses the master/worker model
– Master
 Maintains worker
 Recovers faults of workers
 Provides Web-UI monitoring tool of job progress
– Worker
 Processes its task
 Communicates with the other workers
 Persistent data is stored as files on a distributed storage system
(such as GFS or BigTable)
 Temporary data is stored on local disk
41
Execution of a Pregel Program
1. Many copies of the program begin executing on a cluster of machines
2. The master assigns a partition of the input to each worker
– Each worker loads the vertices and marks them as active
3. The master instructs each worker to perform a superstep
– Each worker loops through its active vertices & computes for each vertex
– Messages are sent asynchronously, but are delivered before the end of the
superstep
– This step is repeated as long as any vertices are active, or any messages are
in transit
4. After the computation halts, the master may instruct each worker to
save its portion of the graph
42
Fault Tolerance
 Checkpointing
– The master periodically instructs the workers to save the state of their
partitions to persistent storage
 e.g., Vertex values, edge values, incoming messages
 Failure detection
– Using regular “ping” messages
 Recovery
– The master reassigns graph partitions to the currently available workers
– The workers all reload their partition state from most recent available
checkpoint
43
Outline
 Introduction
 Computation Model
 Writing a Pregel Program
 System Implementation
 Experiments
 Conclusion & Future Work
44
Experiments
 Environment
– H/W: A cluster of 300 multicore commodity PCs
– Data: binary trees, log-normal random graphs (general graphs)
 Naïve SSSP implementation
– The weight of all edges = 1
– No checkpointing
45
Experiments
 SSSP – 1 billion vertex binary tree: varying # of worker tasks
46
Experiments
 SSSP – binary trees: varying graph sizes on 800 worker tasks
47
Experiments
 SSSP – Random graphs: varying graph sizes on 800 worker tasks
48
Outline
 Introduction
 Computation Model
 Writing a Pregel Program
 System Implementation
 Experiments
 Conclusion & Future Work
49
Conclusion & Future Work
 Pregel is a scalable and fault-tolerant platform with an API that is
sufficiently flexible to express arbitrary graph algorithms
 Future work
– Relaxing the synchronicity of the model
 Not to wait for slower workers at inter-superstep barriers
– Assigning vertices to machines to minimize inter-machine communication
– Caring dense graphs in which most vertices send messages to most other
vertices
50
Thank You!
Any questions or comments?