Transcript Scalable Distributed Data Structures State-of-the-art

SDDS-2000 : A Prototype System
for Scalable Distributed Data
Structures on Windows 2000
Witold Litwin
[email protected]
1
Plan


What are SDDSs ?
Where are we in 2002 ?
–
–
–
–



LH* : Scalable Distributed Hash Partitioning
Scalable Distributed Range Partitioning RP*
High-Availability : LH*RS & RP*RS
DBMS coupling : AMOS-SDDS & SD-AMOS
Architecture of SDDS-2000
Experimental performance results
Conclusion
– Future work
2
What is an SDDS

A new type of data structure
– Specifically for multicomputers

Designed for data intensive files :
– horizontal scalability to very large sizes
» larger than any single-site file
– parallel and distributed processing
» especially in (distributed) RAM
– Record access time better than for any disk file
– 100-300 s usually under Win 2000 (100 Mb/s net, 700
MHZ CPU, 100B – 1 KB records)
– access by multiple autonomous clients
3
Killer apps

Any traditional application using a large hash or Btree or k-d file
– Access time to an RAM SDDS record is 100 faster than to
a disk record






Network storage servers (SNA and NAS)
DBMSs
WEB servers
Video servers
Real-time systems
High Perf. Comp.
4
Multicomputers

A collection of loosely coupled computers
– common and/or preexisting hardware
– share nothing architecture
– message passing through high-speed net (Mb/s)
 Network multicomputers
– use general purpose nets
» LANs: Ethernet, Token Ring, Fast Ethernet, SCI, FDDI...
» WANs: ATM...

Switched multicomputers
– use a bus, or a switch
» e.g., IBM-SP2, Parsytec
5
Typical Network Multicomputer
Client
Server
Network
segments
6
Why multicomputers ?

Potentially unbeatable price-performance ratio
– Much cheaper and more powerful than supercomputers
» 1500 WSs at HPL with 500+ GB of RAM & TBs of disks


Potential computing power
– file size
– access and processing time
– throughput
For more pro & cons :
–
–
–
–
Bill Gates at Microsoft Scalability Day
NOW project (UC Berkeley)
Tanenbaum: "Distributed Operating Systems", Prentice Hall, 1995
www.microoft.com White Papers from Business Syst. Div.
7
Why SDDSs
Multicomputers need data structures and
file systems
 Trivial extensions of traditional structures
are not best






hot-spots
scalability
parallel queries
distributed and autonomous clients
distributed RAM & distance to data
8
Distance to data
(Jim Gray)
10 msec
local disk
100 sec
distant RAM
(Ethernet)
1 sec
distant RAM
(gigabit net)
100 ns
RAM
9
Distance to Data
(Jim Gray)
Moon
10 ms
local disk
8d
100 s
distant RAM
(Ethernet)
2h
10 s
distant RAM
(gigabit net)
10 m
1 s
RAM
1m
10
Scalability Dimensions
(Client view)
Scale-up
Operation time
(distant RAM)
Sub-linear (usuel)
Linear (idéal)
# operation / s
Data size
# servers and # clients
11
Scalability Dimensions
(Client view, SDDS specific)
Operation time
Tapes, jukebox…
Sub-linear (usuel)
Scale-up
ClusterComputer
Disk & Cache
Cache & Disk
Linear
Multicomputer &
SDDS
Single comp.
Local RAM
Data size
Cluster-Computer- = fix # of servers
12
Scalability Dimensions
(Client view)
Speed-up
# operations/ s
Linear (ideal)
Sub-linear (usuel)
# servers
13
What is an SDDS ?
Queries
come from multiple autonomous
clients
Data are on servers
Data are structured
records with keys  objects with OIDs
 more semantics than in Unix flat-file model
 abstraction most popular with applications
 parallel scans & function shipping
 Overflowing
servers split into new servers
14
An SDDS
growth through splits under inserts
Servers
Clients
15
An SDDS
growth through splits under inserts
Servers
Clients
16
An SDDS
growth through splits under inserts
Servers
Clients
17
An SDDS
growth through splits under inserts
Servers
Clients
18
SDDS : Addressing Principles

SDDS Clients
 Are not informed about the splits.
 Do not access any centralized directory for record
address computations
 Have each a less or more adequate private image of the
actual file structure
 Can make addressing errors
 Sending queries or records to incorrect servers
 Searching for a record that was moved elsewhere
by splits
 Sending a record that should be elsewhere for the
same reason
19
SDDS
: Addressing
Principles
What
is an SDDS
?
Servers are able to forward the queries to the
correct address
– perhaps in several messages
Servers may send Image Adjustment
Messages

» Clients do not make same error twice
 Servers supports parallel scans

Sent out by multicast or unicast
With deterministic or probabilistic termination
See the SDDS talk & papers for more
– ceria.dauphine.fr/witold.html

Or the LH* ACM-TODS paper (Dec. 96)
20
An SDDS
Client Access
Clients
21
An SDDS
Client Access
Clients
22
An SDDS
Client Access
Clients
23
An SDDS
Client Access
Clients
24
An SDDS
Client Access
Clients
25
Known SDDSs
DS
SDDS
Classics
(1993)
Hash
LH*
DDH
Breitbart & al
m-d trees
1-d tree
Disk
k-RP*
dPi-tree
Nardelli-tree
RP*
Kroll & Widmayer
Breitbart & Vingralek
SDLSA
LH*m, LH*g
LH*SA
LH*RS
Security
http://ceria.dauphine/SDDS-bibliograhie.html
LH*s
26
LH* (A classic)

Scalable distributed hash partitioning
– Transparent for the applications
» Unlike the current static schemes (i.e. DB2)
– Generalizes the LH addressing schema
» used in many products
– SQL-Server, IIS, MsExchange, Frontpage, Netscape suites,
Berkeley DB Library, LH-Server, Unify...


Typical load factor 70 - 90 %
In practice, at most 2 forwarding messages
– regardless of the size of the file



In general, 1 message/insert and 2 messages/search on the
average
4 messages in the worst case
Several variants are known
– LH*LH is most studied
27
Overview of LH

Extensible hash algorithm
– Widely used, e.g.,
» Netscape browser (100M copies)
» LH-Server by AR (700K copies sold)
» MS Frontpage, Exchange, IIS




– tought in most DB and DS classes
– address space expands
» to avoid overflows & access performance
deterioration
the file has buckets with capacity b >> 1
Hash by division hi : c -> c mod 2i N provides the address
h (c) of key c.
Buckets split through the replacement of hi with h i+1 ; i =
0,1,..
On the average, b/2 keys move towards new bucket
28
Overview of LH




Basically, a split occurs when some bucket m
overflows
One splits bucket n, pointed by pointer n.
– usually m n
n évolue : 0, 0,1, 0,1,..,2, 0,1..,3, 0,..,7, 0,..,2i N, 0..
One consequence => no index
– characteristic of other EH schemes
29
LH File Evolution
N=1
b=4
i=0
35
12
7
15
24
h0 : c -> 20
0
h0 ; n = 0
30
LH File Evolution
N=1
b=4
i=0
35
12
7
15
24
h1 : c -> 21
0
h1 ; n = 0
31
LH File Evolution
12
24
35
7
15
0
1
N=1
b=4
i=1
h1 : c -> 21
h1 ; n = 0
32
LH File Evolution
32
58
12
24
21
11
35
7
15
0
1
h1
h1
N=1
b=4
i=1
h1 : c -> 21
33
LH File Evolution
N=1
b=4
i=1
32
12
24
21
11
35
7
15
58
0
1
2
h2
h1
h2
h2 : c -> 22
34
LH File Evolution
32
12
24
33
21
11
35
7
15
58
0
1
2
h2
h1
h2
N=1
b=4
i=1
h2 : c -> 22
35
LH File Evolution
32
12
24
33
21
58
11
35
7
15
0
1
2
3
h2
h2
h2
h2
N=1
b=4
i=1
h2 : c -> 22
36
LH File Evolution
32
12
24
33
21
58
11
35
7
15
0
1
2
3
h2
h2
h2
h2
N=1
b=4
i= 2
h2 : c -> 22
37
LH File Evolution

Etc
– One starts h3 then h4 ...

The file can expand as much as needed
– without too many overflows ever
38
Addressing Algorithm
a <- h (i, c)
if n = 0 then exit
else
if a < n then a <- h (i+1, c) ;
end
39
LH*

Property of LH :
– Given j = i or j = i + 1, key c is in bucket m iff
hj (c) = m ; j = i or j = i + 1
» Verify yourself

Ideas for LH* :
–
–
–
–
LH addresing rule = global rule for LH* file
every bucket at a server
bucket level j in the header
Check the LH property when the key comes form a
client
40
LH* : file structure
servers
j=4
j=4
j=3
0
1
2
j=3
7
j=4
8
j=4
9
n=2;i=3
n' = 0, i' = 0
n' = 3, i' = 2
Client
Client
Coordinator
41
LH* : file structure
servers
j=4
j=4
j=3
0
1
2
j=3
7
j=4
8
j=4
9
n=2;i=3
n' = 0, i' = 0
n' = 3, i' = 2
Client
Client
Coordinator
42
LH* : split
servers
j=4
j=4
j=3
0
1
2
j=3
7
j=4
8
j=4
9
n=2;i=3
n' = 0, i' = 0
n' = 3, i' = 2
Client
Client
Coordinator
43
LH* : split
servers
j=4
j=4
j=3
0
1
2
j=3
7
j=4
8
j=4
9
n=2;i=3
n' = 0, i' = 0
n' = 3, i' = 2
Client
Client
Coordinator
44
LH* : split
servers
j=4
j=4
j=4
0
1
2
j=3
7
j=4
8
j=4
9
j=4
10
n=3;i=3
n' = 0, i' = 0
n' = 3, i' = 2
Client
Client
Coordinator
45
LH* Addressing Schema

Client
– computes the LH address m of c using its image,
– send c to bucket m

Server
– Server a getting key c, a = m in particular, computes :
a' := hj (c) ;
if a' = a then accept c ;
else a'' := hj - 1 (c) ;
if a'' > a and a'' < a' then a' := a'' ;
send c to bucket a' ;
46
LH* Addressing Schema

Client
– computes the LH address m of c using its image,
– send c to bucket m

Server
– Server a getting key c, a = m in particular, computes :
a' := hj (c) ;
if a' = a then accept c ;
else a'' := hj - 1 (c) ;
Simple ?
if a'' > a and a'' < a' then a' := a'' ;
send c to bucket a' ;

See [LNS93] for the (long) proof
47
Client Image Adjustement

The IAM consists of address a where the client sent c and
of j (a)
– i' is presumed i in client's image.
– n' is preumed value of pointer n in client's image.
– initially, i' = n' = 0.
if j > i' then i' := j - 1, n' := a +1 ;
if n' 2^i' then n' = 0, i' := i' +1 ;

The algo. garantees that client image is within the file
[LNS93]
– if there is no file contractions (merge)
48
LH* : addressing
servers
j=4
j=4
j=4
0
1
2
j=3
7
j=4
8
j=4
9
j=4
10
15
n=3;i=3
n' = 0, i' = 0
n' = 3, i' = 2
Client
Client
Coordinator
49
LH* : addressing
servers
15
j=4
j=4
j=4
0
1
2
j=3
7
j=4
8
j=4
9
j=4
10
n=3;i=3
n' = 0, i' = 0
n' = 3, i' = 2
Client
Client
Coordinator
50
LH* : addressing
servers
15
j=4
j=4
j=4
0
1
2
j=3
7
j=4
n' = 1, i' = 3
n' = 3, i' = 2
Client
Client
j=4
8
j=4
9
j=4
10
n=3;i=3
Coordinator
51
LH* : addressing
servers
j=4
j=4
j=4
0
1
2
j=3
7
j=4
8
j=4
9
j=4
10
9
n=3;i=3
n' = 0, i' = 0
n' = 3, i' = 2
Client
Client
Coordinator
52
LH* : addressing
servers
j=4
j=4
j=4
0
1
2
j=3
7
j=4
8
j=4
9
j=4
10
9
n=3;i=3
n' = 0, i' = 0
n' = 3, i' = 2
Client
Client
Coordinator
53
LH* : addressing
servers
9
j=4
j=4
j=4
0
1
2
j=3
7
j=4
8
j=4
9
j=4
10
n=3;i=3
n' = 0, i' = 0
n' = 3, i' = 2
Client
Client
Coordinator
54
LH* : addressing
servers
9
j=4
j=4
j=4
0
1
2
j=3
7
j=4
n' = 1, i' = 3
n' = 3, i' = 2
Client
Client
j=4
8
j=4
9
j=4
10
n=3;i=3
Coordinator
55
Result

The distributed file can grow to even whole
Internet so that :
– every insert and search are done in four
messages (IAM included)
– in general an insert is done in one message
and search in two messages
– proof in [LNS 93]
56
10,000 inserts
Global cost
Client's cost
57
58
59
Inserts by two clients
60
Parallel Queries

A query Q for all buckets of file F with
independent local executions
– every buckets should get Q exactly once

The basis for function shipping

– fundamental for high-perf. DBMS appl.
Send Mode :
– multicast
» not always possible or convenient
– unicast
» client may not know all the servers
» severs have to forward the query
– how ??
Q
Image
File
61
LH* Algorithm for Parallel
Queries
(unicast)

Client sends Q to every bucket a in the image
The message with Q has the message level j' :

– initialy j' = i' if n' i' else j' = i' + 1
– bucket a (of level j ) copies Q to all its children using
the alg. :
while j' < j do
j' := j' + 1
forward (Q, j' ) to bucket a + 2 j' - 1 ;
endwhile
Prove it !

62
Termination of Parallel
Query
(multicast or unicast)


How client C knows that last reply came ?
Deterministic Solution (expensive)
– Every bucket sends its j, m and selected records if any
» m is its (logical) address
– The client terminates when it has every m fullfiling the
condition ;
» m = 0,1..., 2 i + n where
– i = min (j) and n = min (m) where j = i
i+1
i
i+1
n
63
Termination of Parallel
Query
(multicast or unicast)


Probabilistic Termination ( may need less messaging)
– all and only buckets with selected records reply
– after each reply C reinitialises a time-out T
– C terminates when T expires
Practical choice of T is network and query dependent
– ex. 5 times Ethernet everage retry time
» 1-2 msec ?
– experiments needed

Which termination is finally more useful in practice ?
– an open problem
64
LH* variants


With/without load (factor) control
With/without the (split) coordinator
– the former one was discussed
– the latter one is a token-passing schema
» bucket with the token is next to split
– if an insert occurs, and file overload is guessed
– several algs. for the decision
– use cascading splits

See the talk on LH* at http://ceria.dauphine.fr/
65
RP* schemes

Produce scalable distributed 1-d ordered files
– for range search
Each bucket (server) has the unique range of
keys it may contain
 Ranges partition the key space
 Ranges evolve dynamically through splits

– Transparently for the application

Use RAM m-ary trees at each server
– Like B-trees
– Optimized for the RP* split efficiency
66
Current PDBMS technology
(Ex. Non-Stop SQL)




Static Range Partitioning
Done manually by DBA
Requires goods skills
Not scalable
67
RP* schemes
RP*S
+ servers index optional multicast
RP*C
+ client index
RP*N
No index
limited multicast
all multicast
Fig. 1 RP* design trade-offs
68
High-availability SDDS schemes

Data remain available despite :
– any single server failure & most of two server failures
– or any up to n-server failure
– and some catastrophic failures

n scales with the file size
– To offset the reliability decline which would otherwise
occur

Three principles for high-availability SDDS
schemes are currently known
– mirroring (LH*m)
– striping (LH*s)
– grouping (LH*g, LH*sa, LH*rs, RP* rs)

Realize different performance trade-offs
69
LH*RS : Record Groups

LH* records
RS
– LH* data records & parity records


Records with same rank r in the bucket group form a
record group
Each record group gets n parity records
– Computed using Reed-Salomon erasure correction codes
» Additions ans multiplications in Galois Fields
» See the Sigmod 2000 paper on the Web site for details


r is the common key of these records
Each group supports unavailability of up to n of its
members
70
LH*RS Record Groups
Data records
Parity records
71
LH*RS : Parity Management



An insert of data record with rank r creates or,
usually, updates parity records r
An update of data record with rank r updates
parity records r
A split recreates parity records
– Data record usually change the rank after the split
72
LH*RS Scalable availability


Create 1 parity bucket per group until M = 2i buckets
Then, at each split,
1
– add 2nd parity bucket to each existing group
– create 2 parity buckets for new groups until 2i buckets
2

etc.
73
LH*RS Scalable availability
74
LH*RS Scalable availability
75
LH*RS Scalable availability
76
LH*RS Scalable availability
77
LH*RS Scalable availability
78
SDDS-2000 : global architecture
79
SDDS-2000 : global architecture
Applications
Applications
Applications
etc
SDDS-2000 Client
SDDS-2000 Client
SDDS-2000 Client
SDDS-2000 Client
UDP & TCP
Network
SDDS-2000 Server
SDDS-2000 Server
SDDS-2000 Server
SDDS Data
server
SDDS Data
server
SDDS Data
server
80
SDDS-2000: Client Architecture (RP*c)

2 Modules
Send
Server
...
Server
Network
(TCP/IP, UDP)
Module
Receive
Multithread
Module
Send
Request
Architecture
Client
Flow control
Manager
SendRequest
ReceiveRequest
Key IP Add.
IP Add.
AnalyzeResponse1..4
…
Flow
Images
control
Queues
Response
Request
ReturnResponse
Client
…
Update Analyze Response
1 … 4
Images
GetRequest
Synchronization
Receive
Response
Get
Request
Client
Return
Response
Id_Req Id_App
…
…
Send Module
Requests Journal
Receive Module
SDDS Applications Interface
Application 1
...
Application N
81
SDDS-2000: Server Architecture (RP*c)
 Multithread architecture
Server
Main memory
RP* Buckets
 Synchronization queues
 Listen Thread for incoming
requests
...
BAT
 SendAck Thread for flow
control
RP* Functions :
Insert, Search, Update, Delete,
Forward, Splite.
 Work Threads for
 request processing
 response sendout
 request forwarding
 UDP for shorter messages
(< 64K)
Execution
Results
Results
Request Analyze
W.Thread 1

W.Thread N

...
Response
Response
Ack
queue
 TCP/IP for longer data
exchanges
SendAck
Several buckets of different
SDDS files
Network
(TCP/IP, UDP)
Requests
queue
ListenThread
Client
...
Client
82
AMOS-SDDS Architecture

For database queries
– Especially parallel scans

Couples SDDS-2000 and Amos II :
– RAM OR-DBMS.
– AMOSQL declarative query language
» can be embedded into C and Lisp.
– call-level interface (callin )
– external procedures (functions) (callout)

See the AMOS-II talk & papers for more
http://www.dis.uu.se/~udbl/
83
AMOS-SDDS Architecture



SDDS is used as the distributed RAM storage
manager.
– RP* scheme for the scalable distributed range
partitioning.
– Like in a B-tree, records are lexicographically
ordered according to their keys.
– Supports efficiently the range queries.
Amos II provide a fast SDDS-based RAM ORDBMS.
The callout capability realizes the AMOSQL objectrelational capability usually called external or foreign
functions.
84
AMOS-SDDS Architecture
SDDS
Application
AMOS-SDDS
Application
AMOS II
AMOS-SDDS Server
AMOS-SDDS Server
AMOS II
AMOS II
Callout
Callout
Callout
Callin
SDDS Client
SDDS Server
Callin
SDDS Server
AMOS-SDDS Client
Network
AMOS-SDDS Architecture
85
AMOS-SDDS Architecture
process-AMOS(query)
Client AMOS II
Ship(query)
Results
Server AMOS II
external functions
Results(scan)
Send-AMOS(query)
Receive-AMOS(query)
Receive-AMOS(Buffer)
Send-AMOS(Buffer)
SDDS bucket
Send-SDDS(query)
SDDS
Client
Receive-SDDS(Buffer)
AMOS-SDDS Client
Receive-SDDS(query)
Process-SDDS(query)
Send-SDDS(Buffer)
SDDS Server
AMOS-SDDS Server
Network
AMOS-SDDS scalable distributed query processing
86
AMOS-SDDS
Server Query Processing

E-strategy
– Data stay external to AMOS
» within the SDDS bucket
– Custom foreign functions perform the query

I-strategy
– Data are on-the-fly imported into AMOS-II
» Perhaps with the local index creation
– Good for joins
– AMOS performs the query

Which strategy is preferable ?
– Good question
87
SD-AMOS
Server storage manager is a full scale DBMS
– AMOS-II in our case since it is a RAM DBMS
– Could be any DBMS


SDDS-2000 provides the scalable distributed partitioning
schema
Server DBMS performs the splits
– When et How ???

Client manages scalable query decomposition &
execution
– Easier said than done

The whole system generalizes the PDBMS technology
– Static partitioning only
88
Scalability Analysis

Theoretical
– To validate an SDDS
» See the papers
– To get an idea of system performance
» Limited validity

Experimental
– More accurate validation of design issues
– Practical necessity
– Costs orders of magnitude more of time and
money
89
Experimental Configuration
6 machines : 700 Mhz P3
 100 Mb/s Ethernet
 150 byte records

90
8
7
6
5
4
3
2
1
0
LH* Scalability is confirmed
sans éclatements
with splits
0
1
2
3
4
5
6
Temps (ms)
Temps (s)
LH* file creation
de case
# Nombre
of buckets
sans éclatement
with splits
éclatement
wavec
/ splits
Temps (s)
8
0,5
0,4
0,3
0,2
0,1
0
0
6
éclatements
w /avec
splits
5000
10000
15000
20000
du fichier (enregistrements)
#Taille
of inserts
4
2
0
0
5000
10000
15000
20000
du fichier (enregistrements)
#Taille
of inserts
Ph. D Thesis of F. Bennour, 2000
Bucket size b = 5.000
Flow control On
Performance bound by the
client processing speed
91
LH* Key search
image client
client non-actualisée
Actual
clientactualisée
image New image
client image
& IAMs
Temps (ms)
0,5
0,4
0,3
0,2
2
0,1
0
5000
3
10000
4
5
15000
20000
TailleSize
du fichier
(enregistrements)
File
in records
and # servers
Performance bound by the client processing speed
92
LH* Experimental Performance
RS
(Preliminary results)
Insert time during the file creation
2,5000
2,0000
1,5000
1,0000
0,5000
25
00
0
22
50
0
20
00
0
17
50
0
15
00
0
12
50
0
75
00
10
00
0
k=0
k=1
k=2
50
00
0,0000
25
00
Temps d'insertion d'un
enreg
(moving average)
Nbre d'enregistrements
insérés
Number of records
93
LH* Experimental Performance
RS
(Preliminary results)
Temps d'insertion
File Creation Time
30,000
25,000
20,000
15,000
10,000
5,000
0,000
k=0
k=1
k=2
2500 5000 7500 10000 12500 15000 17500 20000 22500 25000
Number of records
Nbre d'enregistrements
insérés
94
LH* Experimental Performance
RS
(Preliminary results)

Normal key search
– Unaffected by the parity calculus
» 0.3 ms per key search

Degraded key search
– About 2 ms for the application
» 1.1 ms (k =4) for the record recovery
» 1 ms for the client time-out and the coordinator action

Bucket recovery at the spare
– 0.3 ms per record (k =4)
95
LH* Experimental Performance
RS
(Preliminary results)
Insert time during the file creation
2,5000
2,0000
1,5000
1,0000
0,5000
25
00
0
22
50
0
20
00
0
17
50
0
15
00
0
12
50
0
75
00
10
00
0
k=0
k=1
k=2
50
00
0,0000
25
00
Temps d'insertion d'un
enreg
(moving average)
Nbre d'enregistrements
insérés
Number of records
96
LH* Experimental Performance
RS
(Preliminary results)
Temps d'insertion
File Creation Time
30,000
25,000
20,000
15,000
10,000
5,000
0,000
k=0
k=1
k=2
2500 5000 7500 10000 12500 15000 17500 20000 22500 25000
Number of records
Nbre d'enregistrements
insérés
97
LH* Experimental Performance
RS
(Preliminary results)

Normal key search
– Unaffected by the parity calculus
» 0.3 ms per key search

Degraded key search
– About 2 ms for the application
» 1.1 ms (k =4) for the record recovery
» 1 ms for the client time-out and the coordinator action

Bucket recovery at the spare
– 0.3 ms per record (k =4)
98
Performance Analysis (RP*)
Experimental Environment
 Six Pentium III 700 MHz
o Windows 2000
– 128 MB RAM extended later to 256 MB RAM
– 100 Mb/s Ethernet
 Messages
– 180 bytes : 80 for the header, 100 for the record
– Keys are random integers within some interval
– Flow Control sliding window of 10 messages
 Index
–Capacity of an internal node : 80 index elements
–Capacity of a leaf : 100 records
99
Performance Analysis
File Creation
 Bucket capacity : 50.000 records
 150.000 random inserts by a single client
 With flow control (FC) or without
80000
70000
T ime (ms)
60000
Time (ms)
50000
40000
30000
20000
10000
0
1.000
0.900
0.800
0.700
0.600
0.500
0.400
0.300
0.200
0.100
0.000
0
0
50000
100000
Number of records
150000
50000
100000
Number of records
Rp*c/ Without FC
RP*c/ With FC
RP*c without FC
RP*c with FC
RP*n/ With FC
RP*n/ Without FC
RP*n with FC
RP*n without FC
File creation time
Average insert time
150000
100
Discussion


Creation time is almost linearly scalable
Flow control is quite expensive
– Losses without were negligible

Both schemes perform almost equally well
– RP*C slightly better
» As one could expect


Insert time 30 faster than for a disk file
Insert time appears bound by the client speed
101
Performance Analysis
File Creation
 File created by 120.000 random inserts by 2 clients
45000
0.450
60000
40000
35000
0.400
0.350
50000
30000
0.300
40000
25000
20000
0.250
0.200
15000
0.150
10000
5000
0.100
0.050
10000
0
0.000
150000
0
0
50000
100000
Number of records
Time (ms)
Time (ms)
 Without flow control
30000
20000
0
50000
100000
150000
Number of servers
RP*c to. time / 2 clients
RP*c / 1 client
RP*n to. time / 2 clients
RP*n / 1 client
RP*c / Time per record
RP*c to. time / 2 clients
RP*n/ Time per record
RP*n to. time / 2 clients
File creation by two clients : total time and
per insert
200000
Comparative file creation time by one or two
102
clients
Discussion
Performance improves
 Insert times appear bound by a server speed
 More clients would not improve
performance of a server

103
Performance Analysis
3500
0.14
3000
0.12
2500
0.1
2000
0.08
1500
0.06
1000
0.04
500
0.02
10000
90000
80000
70000
60000
50000
0
40000
0
30000
Time/Record
0.137
0.088
0.065
0.057
0.052
0.047
0.045
0.043
0.040
0.037
0.16
20000
Time
1372
1763
1952
2294
2594
2824
3165
3465
3595
3666
4000
10000
b
10000
20000
30000
40000
50000
60000
70000
80000
90000
100000
Split time (ms)
Split Time
Bucket size
Split time
Time per Record
Split times versus bucket capacity
104
Discusion
About linear scalability in function of
bucket size
 Larger buckets are more efficient
 Splitting is very efficient

– Reaching as little as 40 s per record
105
Performance Analysis
Key Search
 A single client sends 100.000 successful random search requests
 Flow control : the client sends at most 10 requests without reply
40000
30000
25000
Time (ms)
T ime (ms)
35000
20000
15000
10000
5000
0
0
1
2
3
4
5
Number of servers
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0
0
1
2
3
4
5
Number of servers
RP*c/ With FC
RP*c/ Without FC
RP*c/ With FC
RP*c/ Without FC
RP*n/ With FC
RP*n/ Without FC
RP*n/ With FC
RP*n/ Without FC
Total search time
Search time per record
107
Discussion

Single search time about 30 times faster
than for a disk file
– 350 s per search

Search throughput more than 65 times faster
than that of a disk file
– 145 s per search
 RP*N appears again surprisingly efficient with
respect RP*c for more buckets
108
Performance Analysis
Range Query
 Deterministic termination
4000
0.04
3500
0.035
3000
0.03
2500
0.025
Time (ms)
T ime (ms)
 Parallel scan of the entire file with all the 100.000 records sent to the client
2000
1500
1000
0.02
0.015
0.01
500
0.005
0
0
1
2
3
Number of servers
Range query total time
4
5
0
0
1
2
3
4
5
Number of servers
Range query time per record
109
Discussion

Range search appears also very efficient
– Reaching 10 s per record delivered

More servers should further improve the
efficiency
– Curves do not become flat yet
110
Range Query Parallel Execution Strategies
Study of MM. Tsangou (Master Th.) & Prof. Samba (U. Dakar)
Scénari o 1
Scénari o 2
Scénari o 3
6000
Sc. 3 ; 1 server at the time
Sc. 1,2 ; all servers together
Time (ms)
Response
Temps de réponse
(ms)
5000
Sc. 1 ; single connection request per server
4000
3000
2000
1000
0
1
10
20
30
40
50
60
70
80
90
100
120
140
# servers
Nom bre de serveurs
111
File Size Limits


Bucket capacity : 751K records, 196 MB
Number of inserts : 3M
Flow control (FC) is necessary to limit the input
queue at each server
File creation by a single client - file size : 3,000,000 records
1600000
1400000
1200000
Time (ms)

1000000
RP*c/ With FC
800000
RP*n/ With FC
600000
400000
200000
0
0
500000
1000000
1500000
2000000
2500000
3000000
3500000
Number of records
112
File Size Limits



Bucket capacity : 751K records, 196 MB
Number of inserts : 3M
GA : Global Average; MA : Moving Average
Insert time by a single client - file size : 3,000,000 records
0,8
0,7
Time (ms)
0,6
RP*c with FC / GA
0,5
RP*c with FC / MA
0,4
RP*n with FC / GA
0,3
RP*n with FC / MA
0,2
0,1
0
0
500000
1000000
1500000
2000000
2500000
3000000
3500000
Number of records
113
AMOS-SDDS

Benchmark data
–
–
–
–

Table Pers (SS#, Name, City)
Size 20.000 to 300.000 tuples
50 Cities
Random distribution
Benchmark queries
– Join : SS# and Name of persons in the same city
» Nested loop or Local index
– Count & Join : Count couples in the same city
» To determine the result transfer time to the client
– Count (*) Pers, Max (SS#) from Pers

Measures
– Scale-up & Speed-up
– Comparison to AMOS-II alone
115
Elapsed time(in ms)
Join : best time per tuple
80
60
40
14.4
20
2.4
0
1
2
E-Strategy
I-Strategy without index
I-Strategy with index
20.000 tuples in Pers
3,990,070 tuples produced
1.6
3
4
5
Number of servers
AMOS-II alone
13.5 Nested loop
2.25 Index lookup
116
Join & Count: best time per tuple
Elapsed time(in ms)
I-strategy
8
6
4
2
1.8
1.0
0
1
2
Query2 without index
Query2 with index
20.000 tuples in Pers
3,990,070 tuples produced
3
4
5
Number of servers
AMOS-II alone
13.5 Nested loop
2.25 Index lookup
117
Join : Speed-up
140
120
100
80
60
40
20
0
0.007
0.006
0.005
0.004
0.003
0.002
0.001
0
1
2
3
4
5
10
Number of servers
20.000 tuples in Pers
3,990,070 tuples produced
15
Time per tuple (in s)
Elapsed time (in s)
I-strategy
Without index
Extrapolation
without index
With index
Extrapolation
with index
AMOS-II alone
13.5 Nested loop
2.25 Index lookup
118
Elapsed time(in ms)
Count : Speed-up
2000
12
10
8
6
4
2
0
1500
1000
500
341
0
1
2
3
4
Count with
importation (left
timescale)
External count (right
timescale)
5
Number of servers
100.000 tuples in Pers
E-strategy wins
AMOS-II alone: 280 ms
119
Join Scale-up Performance
The file scales to 300.000 tuples
 Spreading from 1 to 15 AMOS-SDDS Servers

– Transparently for the application !

3 servers per machine
– The poor men configuration has only 5 server
machines

Results are extrapolated to 1 server per
machine
– Basically, the CPU component of the elapsed time
time is divided by 3
120
Join : Elapsed Time Scale-up
Elapsed time (s)
12000
10000
8000
6000
4000
2000
0
20 000
60 000
1
3
100 000 160 000 200 000 240 000 300 000
AMOS-II
AMOS-SDDS join
AMOS-SDDS join w. count
5
8
10
12
15
File size:
# tuples
#AMOS-SDDS servers
AMOS-SDDS: I-Strategy with Index Lookup Join
121
Join : Elapsed Time Scale-up
File size
20 000
60 000
100 000
160 000
200 000
240 000
300 000
# SDDS servers
1
3
5
8
10
12
15
Result size
3 990 070
35 970 600
99 951 670
255 924 270
399 906 670
575 889 600
899 865 000
Actual Join Time (s)
61
301
684
1817
2555
3901
5564
Join & Count (s)
51
185
335
986
1270
2022
2624
Join w. extrp. (s)
61
301
684
1324
1920
2553
3815
Join & Count.w. extrp. (s)
51
185
335
498
640
681
881
AMOS-II (s)
46
430
1201
3106
4824
6979
10 933
Index creation
3
29
87
230
360
520
819
time (s)
122
Time per tuple (ms)
Join : Time per Tuple Scale-up
40
35
30
25
20
15
10
5
0
20 000
1
60 000 100 000 160 000 200 000 240 000 300 000
3
5
8
AMOS-II
AMOS-SDDS join
AMOS-SDDS join w. count
Join w. Count flat !
10
12
15
File size:
# tuples
#AMOS-SDDS servers
Better scalability than any
current P-DBMS could provide
123
(ms)
Insert
TempsTime
moyen(ms)
SD-AMOS : File Creation
0.20
0.15
0.10
0.05
0.00
1
2
2
2000
4000
6000
Global
MG Avg.
Moving
Avg.
MC
b = 4.000
3
4
4
4
5
8000 10000 12000 14000 15800
# servers
# serveurs
# inserts
#enreg.
Flat & unexpectedly fast insert time
124
(ms)
Timemoyen(ms)
InsertTemps
SD-AMOS : Large File Creation
0.20
0.15
0.10
0.05
0.00
1
20000
Global Avg.
MG
Moving Avg.
MC
b = 40.000
2
2
40000
60000
3
4
4
4
5
80000 100000 120000 140000 159000
# serveurs
# servers
# enreg.
# inserts
Flat & fast insert time remains
125
SD-AMOS : Large File Search
(time per record)
Time (ms)
0.20
0.15
Range query
0.10
Bulk search
Individual search
0.05
0.00
1
2
3
4
5
#serveurs
File of 300K Records
Client max process. speed is reached
126
SD-AMOS : Very Large File Creation
Times (ms)
Bucket size 750 K records
Max file size 3M records
Record size : 100 B
1.20
1.00
0.80
0.60
0.40
0.20
0.00
1 1 1 1 1 1 1 2 2 2 2 2 2 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 4 5
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
# servers
G.A.T
# tuples x 100K
M.A.T
127
Conclusion
 SDDS-2000 : a prototype SDDS manager for
Windows multicomputer
 Several variants of LH* and RP*
 High-availability
 Scalable distributed database query processing
 AMOS-SDDS & SD-AMOS
128
Conclusion
 Experimental performance of SDDS schemes
appears in line with the expectations
Record search & insert times in the range of a
fraction of a millisecond
About 30 to 100 times faster than a disk file access
performance
 About ideal (linear) scalability
 Including the query processing
 Results prove the overall efficiency of SDDS-2000
system
129
Current & Future Work


SDDS-2000 Implementation
High-Availability through RS-Codes
– CERIA & U. Santa Clara (Prof. Th. Schwarz)

Disk-based High-Availability SDDSs
– CERIA & IBM-Almaden (J. Menon)

Parallel Queries
– U. Dakar (Prof. S. Ndiaye) & CERIA

Concurrency & Transactions
– U. Dakar (Prof. T. Seck)


Overall Performance Analysis
SD-AMOS & SD-DBMS in general
– CERIA & U. Uppsala (Prof. T. Risch) & U. Dakar


SD-SQL-Server ?
Extent dependent basically on available funding
130
Credits

SDDS-2000 Implementation
– CERIA Ph. D Students
» F. Bennour (Now Post-Doc) (SDDS-2000 ; LH*)
» A. Wan Diene, Y. Ndiaye (SDDS-2000 RP* & AMOS-SDDS &
SD-AMOS)
» R. Moussa (RS-Parity Subsystem)
– Master Student Thesis
» At CERIA and coop. Universities
– See Ceria Web page ceria.dauphine.fr

Partial Support for SDDS-2000 research
– HPL, IBM Research, Microsoft Research
131
Problems & Exercices
 Install SDDS-2000 and experiment with the interactive appl. Comment the





experience on a few pages.
Create your favorite appl. Using the .h interfaces provided with the package.
Comment on on a few pages on LH*rs goal and way to work, on the basis of the
Sigmod paper and the WDAS-2002 paper by Rim Moussa.
Comment on a few pages on the strategies for a scalable distributed hash joins
according to D. Schneider & al paper. Your own ?
Can you propose how to deal with Theta-joins ?
You should split a table under one SQL Server into two tables on two SQL
Servers. You wish to generate RP* like range partitioning on the key attribute(s).
You wish to use as much as possible standard SQL queries
– How can you find the record with median (middle) key ?
– What are the catalog tables where you can find the table size, the associated check
constraints, indexes, triggers, stored procedures… to move to the new node
– How will you move the records that should leave to the new node.



Idem for a stored funciton under Amos
Idem for Oracle
Idem for DB2
132
END
Thank you for your attention
Witold Litwin
[email protected]
[email protected]
133
134