Transcript Microbenchmarking IHPCL clusters Neil Bright 3/26/2001 File I/O vs. mmap A fd = open() A A read(fd, buffer, size) A A x = buffer[4] A close(fd) A A A fd = open() buffer = mmap(fd, size) close(fd) x =

Microbenchmarking IHPCL clusters
Neil Bright 3/26/2001
File I/O vs. mmap
A
fd = open()
A
A
read(fd, buffer, size)
A
A
x = buffer[4]
A
close(fd)
A
A
A
fd = open()
buffer = mmap(fd,
size)
close(fd)
x = buffer[4]
munmap(buffer, size)
File I/O vs. mmap (cont.)
In microse conds
10K File Create 10K File delete Mmap latency
Solaris 8
1,476.60
1,030.00
20,043.60
Linux 2.2.16-3smp
21.90
1.00
24,299.20
Linux 2.4.3-pre4-smp-pae
22.50
3.00
2,070.20
In M B/se c
Solaris 8
Linux 2.2.16-3smp
Linux 2.4.3-pre4-smp-pae
File reread
Mmap reread
84.40
240.80
178.00
240.00
166.30
239.00
Memory Latency (nanoseconds)
250
225
200
175
150
D anish
125
B eetle
w s330
100
Jedi
75
50
25
0
L1 $
L2 $
M ain m em ory
Memory Bandwidth (MB/sec)
1400
1340
1200
1000
D a n is h
800
7 0 5 .3
B e e tle
ws330
600
Jedi
400
240
191
200
0
Read
W rite
A
PCI 2.0 33Mhz, 32 bit
A
132 Mbytes/sec
PCI 2.0 66Mhz, 64 bit
A
A
528 Mbytes/sec
A
SCSI 160
A
160 Mbytes/sec
Quantum Atlas 10KII SCSI160
A
60 Mbytes/sec
Gigabit Ethernet
A
A
A
125 Mbytes/sec
Network Hardware
A
A
A
TCP Bandwidth
UDP Bandwidth
TCP Latency
A
A
A
2.2 Myrinet
2.2 Gigabit Ethernet
2.4 Gigabit Ethernet
TCP Bandwidth
Throughput
(Mb/sec)
Linux 2.2.16-3smp gm 1.2.3
Linux 2.2.16-3smp e1000 3.0.7
Linux 2.4.3-pre4-smp-pae e1000 3.0.7
256.09
212.23
273.70
CPU utilization
Sender
Receiver
48.37
28.20
2.09
54.33
27.46
0.35
Confidence Intervals
Throughput Local CPU Remote CPU
Linux 2.2.16-3smp gm 1.2.3
Linux 2.2.16-3smp e1000 3.0.7
Linux 2.4.3-pre4-smp-pae e1000 3.0.7
8,192 byte messages
57,344 byte buffers
0.00%
0.00%
3.50%
0.00%
0.00%
50.60%
0.00%
0.00%
537.30%
UDP Bandwidth
Lin u x 2 .2 .1 6 -3 sm p g m 1 .2 .3
Lin u x 2 .2 .1 6 -3 sm p e 1 0 0 0 3 .0 .7
Lin u x 2 .4 .3 -p re 4 -sm p -p a e e 1 0 0 0 3 .0 .7
Lin u x 2 .2 .1 6 -3 sm p g m 1 .2 .3
Lin u x 2 .2 .1 6 -3 sm p e 1 0 0 0 3 .0 .7
Lin u x 2 .4 .3 -p re 4 -sm p -p a e e 1 0 0 0 3 .0 .7
1,472 b yte m e ssag e s
T h ro u g h p u t (M b /se c )
Se n d e r
Re c e ive r
247.90
247.90
676.40
526.10
690.00
199.80
CPU u tilizatio n
Se n d e r
Re c e ive r
18.57
55.34
4.96
60.89
0.40
0.84
Co n fid e n c e In te rvals
T h ro u g h p u t Lo c al CPU Re m o te CPU
0.00%
0.00%
0.00%
7.60%
10.30%
4.90%
156.00%
378.40%
508.80%
TCP Latency
Linux 2.2.16-3smp gm 1.2.3
Linux 2.2.16-3smp e 1000 3.0.7
Linux 2.4.3-pre 4-smp-pae e 1000 3.0.7
T rans/sec
3,663.81
3,207.67
4,160.90
64 byte m essages
Impact of driver on performance
Throughput CPU utilization
(Mb/sec) Sender Receiver
Linux 2.2.16-3smp e1000 2.0.6
Linux 2.2.16-3smp e1000 2.5.14
Linux 2.2.16-3smp e1000 3.0.7
8,192 byte messages
57,344 byte buffers
167.67
186.55
212.23
35.81
31.70
28.20
51.19
48.30
27.46