Transcript No Slide Title

Analyzing NFS Client
Performance with IOzone
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
Don Capps
1
Tom McNeal
Performance Architect
Independent Consultant
HP
TMCN Consulting
[email protected]
[email protected]
NFS Industry Conference
Page 1 of
Benchmark Overview
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
Characteristics
of IOzone
Activities
NFS Industry Conference
Page 2 of
Load Generation
• File System I/O requests
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
–
File sizes vary from 64K to 512M
–
Record sizes vary from 4K to 16M
–
Each increase doubles previous size
–
Large file system calls supported
• System variants supported
–
Memory mapped files
–
fread(), fwrite()
–
pread(), pwrite()
NFS Industry Conference
Page 3 of
Sequential Reads/Writes
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
• Reads & Rereads
• Writes & Rewrites
• Backwards sequential read
• “Stride” read
–
Uses constant intervals for sequential
reads from beginning to end
NFS Industry Conference
Page 4 of
Other Reads/Writes
• Randomized Reads/Writes
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
• Record Rewrite (from offset 0)
• fread() – Reads and Rereads
–
Serialized, Buffered & Blocked IO
• fwrite() – Writes and Rewrites
–
Serialized, Buffered & Blocked IO
NFS Industry Conference
Page 5 of
Recommended Variants
for NFS Clients
./iozone –azc –U /mnt/testdir –f /mnt/testdir/testfile
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
• All tests, all record sizes
• Commit time included in
measurements
• IO targeted at mounted file
–
Unmount clears out caches between tests
–
Target file specified in mounted directory
NFS Industry Conference
Page 6 of
Benchmark Results
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
Characteristics
of IOzone
Reports
NFS Industry Conference
Page 7 of
Graphical Reports
./iozone –R -b exceloutput.xls > logfile
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
• Generate Excel output text
• Named file has graphs and data
–
3D Surface Charts for all tests
–
Includes text output used for graphs
• Standard output sent to log file
–
Generally useful for debugging problems
NFS Industry Conference
Page 8 of
Surface Plot Graphs
Write performance
CPU Cache effect
Memory /
buffer cache
170000-180000
160000-170000
Kbytes/sec
150000-160000
170000
140000-150000
160000
130000-140000
150000
120000-130000
140000
110000-120000
130000
100000-110000
120000
90000-100000
110000
80000-90000
100000
70000-80000
90000
60000-70000
Physical disk I/O
80000
40000-50000
60000
30000-40000
50000
40000
20000-30000
10000-20000
30000
16384
20000
10000
256
NFS Industry Conference
4
1048576
524288
262144
131072
65536
32768
16384
8192
32
4096
2048
1024
512
Record size in Kbytes
256
128
0
Not measured
0-10000
2048
File size in Kbytes
October 22-23, 2002
50000-60000
70000
64
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
180000
Page 9 of
Surface Plot Graphs II
Read Performance
CPU Cache effect
Memory /
buffer cache
420000
October 22-23, 2002
390000
360000-390000
360000
330000-360000
300000-330000
330000
270000-300000
300000
240000-270000
270000
210000-240000
240000
180000-210000
150000-180000
Kbytes/sec 210000
120000-150000
180000
Physical
disk I/O
150000
90000-120000
60000-90000
30000-60000
120000
0-30000
90000
60000
30000
16384
Record size in Kbytes
4
1048576
524288
262144
131072
65536
16384
8192
2048
32
32768
File size in Kbytes
4096
512
256
1024
Not measured
128
2048
256
0
64
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
390000-420000
NFS Industry Conference
Page 10 of
System & Network Variants
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
Environmental
Control and
Interactions
NFS Industry Conference
Page 11 of
System Level Variations
-p, -P #, -l n, -S size, -L size
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
• SMP Issues
–
Processor cache purges
–
Processor affinity (for a given # of cpus)
–
Lower bound of number of cpus
• Cache Management
–
CPU Cache size
–
CPU Cache line size
NFS Industry Conference
Page 12 of
System parameters
• Client BIOD Daemons
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
• Server NFSD Daemons
• Number of file system nodes
–
rnode/inode/vnode/file handles
• Directory Name Lookup Cache
• Network buffer sizes
NFS Industry Conference
Page 13 of
File System Variations
-o, -W, -e, -g #
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
• O_SYNC file option for all tests
• File locking required for all IO
• Flush timings included
–
fsync() and fflush()
• Large file offsets
–
File system calls determined at make time
–
Alternate max file size may be specified
NFS Industry Conference
Page 14 of
File System Variations II
-B, -D, -G, -H n, -k n
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
• Memory mapped file IO
–
mmap() interface
–
MS_ASYNC or MS_SYNC usage available
• Posix asynchronous IO
NFS Industry Conference
Page 15 of
Network Variations
• UDP/TCP Protocol
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
• Client transfer sizes
• Network speed, duplex settings
–
Autonegotiation is often “interesting”
• IP issues
–
Jumbo frames with gigabit ethernet
–
Stream heads, Socket buffer sizes
NFS Industry Conference
Page 16 of
Clustered Clients
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
Managing
and Measuring
a Cluster
NFS Industry Conference
Page 17 of
Client Specification
-+m filename
• Clients specified in a file
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
• Clients must be accessible
–
Remote shells enabled through .rhosts
• DNS ®
• IOzone revision 3.128 or later
• Stonewalling helpful (removed by –x)
NFS Industry Conference
Page 18 of
“Stonewalling”
• Client tests initiated in tandem
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
–
All clients kept equally busy
• When one finishes, they all finish
–
Tests halted when the first client completes
• Emulates high performance parallel
processing clusters
–
Beowulf clusters at LLNL, PNNL, Los
Alamos
NFS Industry Conference
Page 19 of
Summary
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
Examples
and
References
NFS Industry Conference
Page 20 of
Summary
• NFS Client measurement standard
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
October 22-23, 2002
./iozone –azcR –U /mnt/testdir –f /mnt/testdir/testfile \
-b exceloutput.xls > logfile
• Gather standard data first
–
What is right for your client?
• Review Variations and Features
–
Review System, FS, and Network setup
–
Start tuning, playing, tuning, playing….
NFS Industry Conference
Page 21 of
Rewrite Graph
Re-write performance
Memory /
buffer cache
CPU Cache effect
450000
October 22-23, 2002
390000-420000
390000
360000-390000
360000
330000-360000
300000-330000
330000
270000-300000
300000
Kbytes/sec
240000-270000
270000
210000-240000
240000
180000-210000
150000-180000
210000
Physical disk I/O
180000
60000-90000
120000
30000-60000
90000
16384
60000
0-30000
2048
30000
256
NFS Industry Conference
4
1048576
524288
262144
65536
32768
16384
131072
File size in Kbytes
8192
32
4096
2048
1024
512
Record size in Kbytes
256
128
0
Not measured
120000-150000
90000-120000
150000
64
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
420000-450000
420000
Page 22 of
Reread Graph
Re-read performance
CPU Cache effect
750000
Memory /
buffer cache
700000
October 22-23, 2002
650000-700000
600000
600000-650000
550000-600000
550000
500000-550000
500000
Kbytes/sec
450000-500000
450000
400000-450000
400000
350000-400000
300000-350000
350000
Physical disk I/O
300000
150000-200000
200000
100000-150000
150000
16384
100000
2048
50000
50000-100000
0-50000
256
NFS Industry Conference
4
1048576
524288
262144
65536
32768
16384
131072
File size in Kbytes
8192
32
4096
2048
1024
512
Record size in Kbytes
256
128
0
Not measured
250000-300000
200000-250000
250000
64
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
700000-750000
650000
Page 23 of
Random Read Graph
Random Read Performance
CPU Cache effect
Memory /
buffer cache
600000
October 22-23, 2002
560000-600000
520000
520000-560000
480000
480000-520000
440000
440000-480000
400000-440000
400000
360000-400000
360000
Kbytes/sec
320000-360000
320000
280000-320000
280000
Physical disk I/O
240000-280000
240000
200000-240000
200000
160000-200000
120000-160000
160000
80000-120000
120000
16384
80000
2048
40000
0-40000
NFS Industry Conference
4
1048576
524288
262144
65536
32768
16384
131072
File size in Kbytes
8192
32
4096
2048
1024
512
128
Record size in Kbytes
256
Not measured
40000-80000
256
0
64
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
560000
Page 24 of
References
• http://www.iozone.org
• http://www.iozone.org/src/current
N I C
F N O
S D N
U F
S E
T R
R E
Y N
C
E
–
• http://www.sourceforge.net/projects/
nfstestmatrix
–
Includes functional tests, destructive tests,
and benchmarks for Linux systems
• http://www.mclx.com/orph
–
October 22-23, 2002
Contains 8K vs. 32K Transfer Size graphs
Linux Server performance review (late 2001)
NFS Industry Conference
Page 25 of