Transcript Lecture 6: Vector - Texas A&M University

I/O Devices & RAID
Motivation: Who Cares About I/O?
• CPU Performance: 60% per year
• I/O system performance limited by mechanical delays
(disk I/O)
< 10% per year (IO per sec)
• Amdahl's Law: system speed-up limited by the slowest
part!
10% IO & 10x CPU => 5x Performance (lose 50%)
10% IO & 100x CPU => 10x Performance (lose 90%)
• I/O bottleneck:
Diminishing fraction of time in CPU
Diminishing value of faster CPUs
Big Picture: Who cares about CPUs?
• Why still important to keep CPUs busy vs. IO
devices ("CPU time"), as CPUs not costly?
– Moore's Law leads to both large, fast CPUs but also to
very small, cheap CPUs
– 2002 Hypothesis: 2 GHz PC is fast enough for MS Office?
– PC industry slowdown since PCs fast enough except for
games, new apps?
• People care more about about storing information
and communicating information than calculating
– "Information Technology" vs. "Computer Science"
– 1960s and 1980s: Computing Revolution
– 1990s and 2000s: Information Age
I/O Systems
Processor
interrupts
Cache
Memory - I/O Bus
Main
Memory
I/O
Controller
Disk
Disk
I/O
Controller
I/O
Controller
Graphics
Network
Storage Technology Drivers
• Driven by the prevailing computing paradigm
– 1950s: migration from batch to on-line processing
– 1990s: migration to ubiquitous computing
» computers in phones, books, cars, video cameras, …
» nationwide fiber optical network with wireless tails
• Effects on storage industry:
– Embedded storage
» smaller, cheaper, more reliable, lower power
– Data utilities
» high capacity, hierarchically managed storage
Disk Device Terminology
Arm Head
Actuator
Inner Outer
Sector
Track Track
Platter
• Several platters, with information recorded magnetically on both
surfaces (usually)
• Bits recorded in tracks, which are in turn divided into sectors (e.g., 512
Bytes)
• Actuator moves head (end of arm,1/surface) over track (“seek”), select
surface, wait for sector rotate under head, then read or write
–
“Cylinder”: all tracks under heads
Disk Device Performance
Outer
Track
Platter
Inner Sector
Head Arm Controller
Spindle
Track
Actuator
• Disk Latency = Seek Time + Rotation Time + Transfer
Time + Controller Overhead
• Seek Time? depends no. tracks move arm, seek speed of disk
• Rotation Time? depends on speed disk rotates, how far sector is
from head
• Transfer Time? depends on data rate (bandwidth) of disk (bit
density), size of request
Disk Device Performance
• Average distance sector from head?
• 1/2 time of a rotation
–
–
–
–
10000 Revolutions Per Minute  166.67 Rev/s
1 revolution = 1/ 166.67 sec  6.00 ms
1/2 rotation (revolution)  3.00 ms
This assumes only one arm
• Average no. tracks to move arm?
– Sum all possible seek distances
from all possible tracks / # possible
» Assumes average seek distance is random
– Disk industry standard benchmark
Data Rate: Inner vs. Outer Tracks
• To keep things simple, originally kept same number
of sectors per track
– Since outer track longer, fewer bits per inch (BPI)
• Competition  decided to keep BPI the same for all
tracks (“constant bit density”)
 More capacity per disk
 More of sectors per track towards edge
 Since disk spins at constant speed,
outer tracks have faster data rate
• Bandwidth of outer track is 1.7x inner track!
– Inner track highest density, outer track lowest, so not really
constant
– 2.1x length of track outer / inner, 1.7x bits outer / inner
Devices: Magnetic Disks
• Purpose:
Track
Sector
– Long-term, nonvolatile storage
– Large, inexpensive, slow level
in the storage hierarchy
• Characteristics:
Cylinder
– Seek Time (~8 ms avg)
»
»
•
Transfer rate
–
–
10-40 MB/s
Blocks
• Capacity
–
–
Head
positional latency
rotational latency
Platter
7200 RPM = 120 RPS => 8 ms per rev
ave rot. latency = 4 ms
128 sectors per track => 0.25 ms per sector
1 KB per sector => 16 MB / s
Response time
Gigabytes
= Queue + Controller + Seek + Rot + Xfer
Quadruples every 2 years
Service time
(aerodynamics)
Disk Performance Model /Trends
• Capacity
+ 100%/year (2x / 1.0 yrs)
• Transfer rate (BW)
+ 40%/year (2x / 2.0 yrs)
• Rotation + Seek time
– 8%/ year (1/2 in 10 yrs)
• MB/$
> 100%/year (2x / 1.0 yrs)
Fewer chips + areal density
State of the Art: Barracuda 180
Track
Sector
Cylinder
Track Arm
Platter
Head
Buffer
Latency =
Queuing Time +
Controller time +
per access
Seek Time +
+
Rotation Time +
per byte
Size / Bandwidth
{
source: www.seagate.com
–181.6 GB, 3.5 inch
disk
–12 platters, 24
surfaces
–24,247 cylinders
–7,200 RPM (4.2 ms
avg. latency)
–7.4/8.2 ms avg. seek
(R/W)
–64 to 35 MB/s
(internal)
–0.1 ms controller time
–10.3 watts (idle)
Simple Disk Performance Example
(will fix later)
• Calculate time to read 64 KB (128 sectors) for
Barracuda 180 X using advertised performance;
sector is on outer track
Disk latency = average seek time + average rotational
delay + transfer time + controller overhead
= 7.4 ms + 0.5 * 1/(7200 RPM)
+ 64 KB / (65 MB/s) + 0.1 ms
= 7.4 ms + 0.5 /(7200 RPM/(60000ms/M))
+ 64 KB / (65 KB/ms) + 0.1 ms
= 7.4 + 4.2 + 1.0 + 0.1 ms = 12.7 ms
Area Density
• Bits recorded along a track
– Metric is Bits Per Inch (BPI)
• Number of tracks per surface
– Metric is Tracks Per Inch (TPI)
• Disk Designs Brag about bit density per unit area
– Metric is Bits Per Square Inch
– Called Areal Density
– Areal Density = BPI x TPI
• Note disk people still using English units - even
measure head flying height in microinches
Areal Density
Year
Areal Density
1.7
1979
7.7
1989
63
1997
3090
2000
17100
100000
10000
1000
Areal Density
1973
100
10
1
1970
1980
1990
Year
– Areal Density = BPI x TPI
– Change slope 30%/yr to 60%/yr about 1991
2000
MBits per Square Inch:
DRAM as % of Disk over time
9 v. 22 Mb/si
50%
40%
30%
20%
470 v. 3000 Mb/in2
10%
0%
0.2 v. 1.7 Mb/in2
1974 1980 1986 1992 1998 2000
source: New York Times, 2/23/98, page C3,
“Makers of disk drives crowd even mroe data into even smaller spaces”
Historical Perspective
• 1956 IBM Ramac — early 1970s Winchester
– Developed for mainframe computers, proprietary interfaces
– Steady shrink in form factor: 27 in. to 14 in
• Form factor and capacity drives market, more than
performance
• 1970s: Mainframes  14” diameter disks
• 1980s: Minicomputers,Servers  8”, 5.25” diameter
• PCs, workstations Late 1980s/Early 1990s:
– Mass market disk drives become a reality
» industry standards: SCSI, IPI, IDE
– Pizzabox PCs  3.5 inch disks
– Laptops, notebooks  2.5 inch disks
– PDAs didn’t use disks,
so 1.8 inch diameter disks didn’t make it
• 2000s:
– 1 inch for cameras (yes), PDAs? cell phones?
1 inch disk drive!
• 2000 IBM MicroDrive:
– 1.7” x 1.4” x 0.2”
– 1 GB, 3600 RPM,
5 MB/s, 15 ms seek
– Digital cameras, PDAs?
• 2006 MicroDrive?
– 9 GB, 50 MB/s!
– Assuming past trends continue
– Assuming it finds a niche
in a successful product
– Big concern will be power
Disk Characteristics in 2000
Seagate
IBM
IBM 1GB
Cheetah
Travelstar
Microdrive
ST173404LC 32GH DJSA - DSCM-11000
Ultra160 SCSI 232 ATA-4
Disk diameter
(inches)
Formatted data
capacity (GB)
Cylinders
3.5
2.5
1.0
73.4
32.0
1.0
14,100
21,664
7,167
Disks
12
4
1
Recording
Surfaces (Heads)
Bytes per sector
24
8
2
512 to 4096
512
512
~ 424
~ 360
~ 140
6.0
14.0
15.2
Avg Sectors per
track (512 byte)
Max. areal
density(Gbit/sq.in.)
$828
$447
$435
Disk Characteristics in 2000
Seagate
IBM
IBM 1GB
Cheetah
Travelstar
Microdrive
ST173404LC 32GH DJSA - DSCM-11000
Ultra160 SCSI 232 ATA-4
Rotation speed
(RPM)
Avg. seek ms
(read/write)
Minimum seek
ms (read/write)
Max. seek ms
Data transfer
rate MB/second
Link speed to
buffer MB/s
Power
idle/operating
Watts
10033
5411
3600
5.6/6.2
12.0
12.0
0.6/0.9
2.5
1.0
14.0/15.0
23.0
19.0
27 to 40
11 to 21
2.6 to 4.2
160
67
13
16.4 / 23.5
2.0 / 2.6
0.5 / 0.8
Disk Characteristics in 2000
Seagate
IBM
IBM 1GB
Cheetah
Travelstar
Microdrive
ST173404LC 32GH DJSA - DSCM-11000
Ultra160 SCSI 232 ATA-4
Buffer size in MB
4.0
2.0
0.125
Size: height x
width x depth
inches
Weight pounds
1.6 x 4.0 x
5.8
2.00
0.5 x 2.7 x 0.2 x 1.4 x
3.9
1.7
0.34
0.035
Rated MTTF in
powered-on hours
1,200,000
% of POH per
month
% of POH
seeking, reading,
writing
100%
(300,000?) (20K/5 yr
life?)
45%
20%
90%
20%
20%
Disk Characteristics in 2000
Seagate
IBM Travelstar
Cheetah
32GH DJSA ST173404LC
232 ATA-4
Ultra160 SCSI
IBM 1GB Microdrive
DSCM-11000
Load/Unload
cycles (disk
powered on/off)
Nonrecoverable
read errors per
bits read
Seek errors
250 per year
300,000
300,000
<1 per 1015
< 1 per 1013
< 1 per 1013
<1 per 107
not available
not available
Shock tolerance:
Operating, Not
operating
Vibration
tolerance:
Operating, Not
operating (sine
swept, 0 to peak)
10 G, 175 G 150 G, 700 G
175 G, 1500 G
5-400 Hz @ 5-500 Hz @ 5-500 Hz @ 1G, 10
0.5G, 22-400 1.0G, 2.5-500
500 Hz @ 5G
Hz @ 2.0G Hz @ 5.0G
Fallacy: Use Data Sheet
“Average Seek” Time
• Manufacturers needed standard for fair
comparison (“benchmark”)
– Calculate all seeks from all tracks, divide by
number of seeks => “average”
• Real average would be based on how data
laid out on disk, where seek in real
applications, then measure performance
– Usually, tend to seek to tracks nearby, not to
random track
• Rule of Thumb: observed average seek time
is typically about 1/4 to 1/3 of quoted seek
time (i.e., 3X-4X faster)
– Barracuda 180 X avg. seek: 7.4 ms
2.5 ms
Fallacy: Use Data Sheet
Transfer Rate
• Manufacturers quote the speed of the data
rate off the surface of the disk
• Sectors contain an error detection and
correction field (can be 20% of sector size)
plus sector number as well as data
• There are gaps between sectors on track
• Rule of Thumb: disks deliver about 3/4 of
internal media rate (1.3X slower) for data
• For example, Barracuda 180X quotes
64 to 35 MB/sec internal media rate
– 47 to 26 MB/sec external data rate (74%)
Disk Performance Example
• Calculate time to read 64 KB for UltraStar 72 again,
this time using 1/3 quoted seek time, 3/4 of internal
outer track bandwidth; (12.7 ms before)
Disk latency = average seek time + average rotational
delay + transfer time + controller overhead
= (0.33 * 7.4 ms) + 0.5 * 1/(7200 RPM)
+ 64 KB / (0.75 * 65 MB/s) + 0.1 ms
= 2.5 ms + 0.5 /(7200 RPM/(60000ms/M))
+ 64 KB / (47 KB/ms) + 0.1 ms
= 2.5 + 4.2 + 1.4 + 0.1 ms = 8.2 ms (64% of 12.7)
Future Disk Size and Performance
• Continued advance in capacity (60%/yr) and
bandwidth (40%/yr)
• Slow improvement in seek, rotation (8%/yr)
• Time to read whole disk
Year
Sequentially
Randomly
(1 sector/seek)
1990
4 minutes
6 hours
2000
12 minutes
1 week(!)
• 3.5” form factor make sense in 5 years?
– What is capacity, bandwidth, seek time, RPM?
– Assume today 80 GB, 30 MB/sec, 6 ms, 10000 RPM
– May want higher bandwidth => more smaller disks
Tape vs. Disk
• Longitudinal tape uses same technology as
hard disk; tracks its density improvements
• Disk head flies above surface, tape head lies on surface
• Disk fixed, tape removable
• Inherent cost-performance based on geometries:
fixed rotating platters with gaps
(random access, limited area, 1 media / reader)
vs.
removable long strips wound on spool
(sequential access, "unlimited" length, multiple / reader)
• Helical Scan (VCR, Camcoder, DAT)
Spins head at angle to tape to improve density
Current Drawbacks to Tape
• Tape wear out:
– Helical 100s of passes to 1000s for longitudinal
• Heads wear out:
– 2000 hours for helical
• Both must be accounted for in economic /
reliability model
• Tape stretches => bits stretch
• Readers must be compatible with multiple
generations of media
• Long rewind, eject, load, spin-up times;
not inherent, just no need in marketplace
• Designed for archival storage
Library vs. Storage
• Getting books today as quaint as the way I
learned to program
– punch cards, batch processing
– wander thru shelves, anticipatory purchasing
•
•
•
•
•
Cost $1 per book to check out
$30 to create a catalog entry
30% of all books never checked out
Write-only journals?
Digital library can transform campuses
Use Arrays of Small Disks?
•Katz and Patterson asked in 1987:
•Can smaller disks be used to close gap in
performance between disks and CPUs?
Conventional:
4 disk
3.5” 5.25”
designs
Low End
Disk Array:
1 disk design
3.5”
10”
14”
High End
Advantages of Small
Formfactor Disk Drives
Low $/MB
High MB/volume
High MB/W
Low $/Actuator
Cost and Environmental Efficiencies
Replace Small Number of Large Disks
with Large Number of Small Disks!
(1988 Disks)
IBM 3390K IBM 3.5" 0061
x70
20 GB
23 GB
Capacity
320 MB
97 cu. ft.
11 cu. ft. 9X
Volume
0.1 cu. ft.
3 KW
1 KW 3X
Power
11 W
15 MB/s
120 MB/s 8X
Data Rate
1.5 MB/s
600 I/Os/s
3900 IOs/s 6X
I/O Rate
55 I/Os/s
250 KHrs
??? Hrs
MTTF
50 KHrs
$250K
$150K
Cost
$2K
Disk Arrays have potential for large data and
I/O rates, high MB per cu. ft., high MB per KW,
but what about reliability?
Array Reliability
• Reliability of N disks = Reliability of 1 Disk ÷ N
50,000 Hours ÷ 70 disks = 700 hours
Disk system MTTF: Drops from 6 years to 1 month!
• Arrays (without redundancy) too unreliable to be useful!
Hot spares support reconstruction in parallel with
access: very high media availability can be achieved
Redundant Arrays of
(Inexpensive) Disks (RAID)
• Files are "striped" across multiple disks
– 1st sector on disk 1, 2nd sector on disk 2, …
– Very high bandwidth
• Redundancy yields high data availability
– Availability: service still provided to user, even if some
components failed
• Disks will still fail
• Contents reconstructed from data redundantly stored in
the array
 Capacity penalty to store redundant info
 Bandwidth penalty to update redundant info
Redundant Arrays of Inexpensive Disks
RAID 1: Disk Mirroring/Shadowing
recovery
group
• Each disk is fully duplicated onto its “mirror”
Very high availability can be achieved
• Bandwidth sacrifice on write:
Logical write = two physical writes
• Reads may be optimized
• Most expensive solution: 100% capacity overhead
• (RAID 2 not interesting, so skip)
Redundant Array of Inexpensive
Disks RAID 3: Parity Disk
10010011
11001101
10010011
...
logical record
1
1
0
1
Striped physical
1
0
records
0
0
P contains sum of
0
1
other disks per stripe 0
1
mod 2 (“parity”)
1
0
If disk fails, subtract 1
1
P from sum of other
disks to find missing information
P
1
0
1
0
0
0
1
1
1
1
0
0
1
1
0
1
RAID 3
• Sum computed across recovery group to
protect against hard disk failures, stored in P
disk
• Logically, a single high capacity, high transfer
rate disk: good for large transfers
• Wider arrays reduce capacity costs (less
parity overhead), but decreases availability
(more chance of two disk failures)
• 33% capacity cost for parity in this
configuration
Inspiration for RAID 4
• RAID 3 relies on parity disk to discover errors
on Read
• But every sector has an error detection field
– Using powerful Reed-Solomon burst error
correcting codes
• Rely on error detection field to catch errors
on read, not on the parity disk
• Allows independent reads to different disks
simultaneously
– Increases bandwidth or throughput
RAID 4: High I/O Rate Parity
Insides of
5 disks
Example:
small read
D0 & D5,
large write
D12-D15
D0
D1
D2
D3
P
D4
D5
D6
D7
P
D8
D9
D10
D11
P
D12
D13
D14
D15
P
D16
D17
D18
D19
P
D20
D21
D22
D23
P
.
.
.
.
Columns
.
.
.
.
.
.
.
.
.
.
Disk
.
Increasing
Logical
Disk
Address
Stripe
Inspiration for RAID 5
• RAID 4 works well for small reads
• Small writes (write to one disk):
– Option 1: read other data disks, create new sum
and write to Parity Disk
– Option 2: since P has old sum, compare old data to
new data, add the difference to P
• Small writes are limited by Parity Disk: Write
to D0, D5 both also write to P disk
D0
D1
D2
D3
P
D4
D5
D6
D7
P
RAID 5: High I/O Rate Interleaved Parity
Independent
writes
possible
because of
interleaved
parity
Example:
write to
D0, D5
uses disks
0, 1, 3, 4
D0
D1
D2
D3
P
D4
D5
D6
P
D7
D8
D9
P
D10
D11
D12
P
D13
D14
D15
P
D16
D17
D18
D19
D20
D21
D22
D23
P
.
.
.
.
.
.
.
.
.
.
Disk Columns
.
.
.
.
.
Increasing
Logical
Disk
Addresses
Problems of Disk Arrays:
Small Writes
RAID-5: Small Write Algorithm
1 Logical Write = 2 Physical Reads + 2 Physical Writes
D0'
new
data
D0
D1
D2
D3
old
data (1. Read)
P
old
(2. Read)
parity
+ XOR
+ XOR
(3. Write)
D0'
D1
(4. Write)
D2
D3
P'
System Availability: Orthogonal RAIDs
Array
Controller
String
Controller
. . .
String
Controller
. . .
String
Controller
. . .
String
Controller
. . .
String
Controller
. . .
String
Controller
. . .
Data Recovery Group: unit of data redundancy
Redundant Support Components: fans, power supplies, controller, cables
End to End Data Integrity: internal parity protected data paths
System-Level Availability
host
host
Fully dual redundant
I/O Controller
Array Controller
I/O Controller
Array Controller
...
...
...
...
Goal: No Single
Points of
Failure
...
Recovery
Group
.
.
.
with duplicated paths, higher performance can be
obtained when there are no failures
RAID Summary: Goal was performance,
popularity due to reliability of storage
• Disk Mirroring, Shadowing (RAID 1)
Each disk is fully duplicated onto its "shadow"
Logical write = two physical writes
100% capacity overhead
• Parity Data Bandwidth Array (RAID 3)
Parity computed horizontally
Logically a single high data bw disk
• High I/O Rate Parity Array (RAID 5)
Interleaved parity blocks
Independent reads and writes
Logical write = 2 reads + 2 writes
1
0
0
1
0
0
1
1
1
0
0
1
0
0
1
1
1
0
0
1
0
0
1
1
1
1
0
0
1
1
0
1
1
0
0
1
0
0
1
1
0
0
1
1
0
0
1
0
Queuing Theory and I/O
Benchmarks
Review: Disk I/O Performance
300
Metrics:
Response Time
Throughput
Response
Time (ms)
200
100
0
100%
0%
Throughput
(% total BW)
Queue
Proc
IOC
Device
Response time = Queue + Device Service time
Introduction to
Queuing Theory
Arrivals
Departures
• More interested in long term, steady state than in
startup => Arrivals = Departures
• Little’s Law:
Mean number tasks in system = arrival rate x mean
response time
– Observed by many, Little was first to prove
• Applies to any system in equilibrium,
as long as nothing in black box
is creating or destroying tasks
A Little Queuing Theory:
Notation
System
Queue
Proc
server
IOC
Device
• Queuing models assume state of equilibrium:
input rate = output rate
• Notation:
r
Tser
u
Tq
Tsys
Lq
Lsys
average number of arriving customers/second
average time to service a customer (traditionally µ = 1/ Tser )
server utilization (0..1): u = r x Tser (or u = r / Tser )
average time/customer in queue
average time/customer in system: Tsys = Tq + Tser
average length of queue: Lq = r x Tq
average length of system: Lsys = r x Tsys
• Little’s Law: Lengthserver = rate x Timeserver
(Mean number customers = arrival rate x mean service time)
A Little Queuing Theory
System
Queue
Proc
server
IOC
Device
• Service time completions vs. waiting time for a busy
server: randomly arriving event joins a queue of arbitrary
length when server is busy,
otherwise serviced immediately
– Unlimited length queues key simplification
• A single server queue: combination of a servicing facility
that accommodates 1 customer at a time (server) +
waiting area (queue): together called a system
• Server spends a variable amount of time with customers;
how do you characterize variability?
– Distribution of a random variable: histogram? curve?
A Little Queuing Theory
System
Queue
Proc
server
IOC
Device
• Server spends variable amount of time with customers
– Weighted mean m1 = (f1 x T1 + f2 x T2 +...+ fn x Tn)/F (F=f1 + f2...)
– variance = (f1 x T12 + f2 x T22 +...+ fn x Tn2)/F – m12
» Must keep track of unit of measure (100 ms2 vs. 0.1 s2 )
Avg.
– Squared coefficient of variance: C2 = variance/m12
» Unitless measure (100 ms2 vs. 0.1 s2)
•
Exponential distribution C2 = 1 : most short relative to average, few others long; 90%
< 2.3 x average, 63% < average
•
Hypoexponential distribution C2 < 1 : most close to average,
C2=0.5 => 90% < 2.0 x average, only 57% < average
•
Hyperexponential distribution C2 > 1 : further from average
C2 =2.0 => 90% < 2.8 x average, 69% < average
A Little Queuing Theory:
Variable Service Time
System
Queue
Proc
server
IOC
Device
• Server spends a variable amount of time with customers
– Weighted mean m1 = (f1xT1 + f2xT2 +...+ fnXTn)/F (F=f1+f2+...)
• Usually pick C = 1.0 for simplicity
• Another useful value is average time
must wait for server to complete task: m1(z)
– Not just 1/2 x m1 because doesn’t capture variance
– Can derive m1(z) = 1/2 x m1 x (1 + C2)
– No variance => C2 = 0 => m1(z) = 1/2 x m1
A Little Queuing Theory:
Average Wait Time
• Calculating average wait time in queue Tq
– If something at server, it takes to complete on average m1(z)
– Chance server is busy = u; average delay is u x m1(z)
– All customers in line must complete; each avg Tser
Tq = u x m1(z) + Lq x Ts er= 1/2 x u x Tser x (1 + C) + Lq x Ts er
Tq = 1/2 x u x Ts er x (1 + C) + r x Tq x Ts er
Tq = 1/2 x u x Ts er x (1 + C) + u x Tq
Tq x (1 – u) = Ts er x u x (1 + C) /2
Tq = Ts er x u x (1 + C) / (2 x (1 – u))
• Notation:
r
Tser
u
Tq
Lq
average number of arriving customers/second
average time to service a customer
server utilization (0..1): u = r x Tser
average time/customer in queue
average length of queue:Lq= r x Tq
A Little Queuing Theory:
M/G/1 and M/M/1
• Assumptions so far:
– System in equilibrium, number sources of requests unlimited
– Time between two successive arrivals in line are exponentially
distrib.
– Server can start on next customer immediately after prior
finishes
– No limit to the queue: works First-In-First-Out "discipline"
– Afterward, all customers in line must complete; each avg Tser
• Described “memoryless” or Markovian request arrival
(M for C=1 exponentially random), General service
distribution (no restrictions), 1 server: M/G/1 queue
• When Service times have C = 1, M/M/1 queue
Tq = Tser x u x (1 + C) /(2 x (1 – u)) = Tser x u / (1 – u)
Tser
u
Tq
average time to service a customer
server utilization (0..1): u = r x Tser
average time/customer in queue
A Little Queuing Theory:
An Example
• processor sends 10 x 8KB disk I/Os per second, requests
& service exponentially distrib., avg. disk service = 20 ms
• On average, how utilized is the disk?
– What is the number of requests in the queue?
– What is the average time spent in the queue?
– What is the average response time for a disk request?
• Notation:
r
Tser
u
Tq
Tsys
Lq
Lsys
average number of arriving customers/second = 10
average time to service a customer = 20 ms (0.02s)
server utilization (0..1): u = r x Tser= 10/s x .02s = 0.2
average time/customer in queue = Tser x u / (1 – u)
= 20 x 0.2/(1-0.2) = 20 x 0.25 = 5 ms (0 .005s)
average time/customer in system: Tsys =Tq +Tser= 25 ms
average length of queue:Lq= r x Tq
= 10/s x .005s = 0.05 requests in queue
average # tasks in system: Lsys = r x Tsys = 10/s x .025s = 0.25
A Little Queuing Theory:
Another Example
• processor sends 20 x 8KB disk I/Os per sec, requests &
service exponentially distrib., avg. disk service = 12 ms
• On average, how utilized is the disk?
– What is the number of requests in the queue?
– What is the average time a spent in the queue?
– What is the average response time for a disk request?
• Notation:
r
Tser
u
Tq
Tsys
Lq
Lsys
average number of arriving customers/second= 20
average time to service a customer= 12 ms
server utilization (0..1): u = r x Tser=
/s x . s =
average time/customer in queue = Ts er x u / (1 – u)
=
x
/(
)=
x
=
ms
average time/customer in system: Tsys =Tq +Tser= 16 ms
average length of queue:Lq= r x Tq
=
/s x
s=
requests in queue
average # tasks in system : Lsys = r x Tsys =
/s x s =
Summary: A Little Queuing
Theory
System
Queue
Proc
server
IOC
Device
• Queuing models assume state of equilibrium:
input rate = output rate
• Notation:
r
Tser
u
Tq
Tsys
Lq
Lsys
average number of arriving customers/second
average time to service a customer (tradtionally µ = 1/ Tser )
server utilization (0..1): u = r x Tser
average time/customer in queue
average time/customer in system: Tsys = Tq + Tser
average length of queue: Lq = r x Tq
average length of system : Lsys = r x Tsys
• Little’s Law: Lengthsystem = rate x Timesystem
(Mean number customers = arrival rate x mean service time)
I/O Benchmarks
• For better or worse, benchmarks shape a field
– Processor benchmarks classically aimed at response time
for fixed sized problem
– I/O benchmarks typically measure throughput, possibly with
upper limit on response times (or 90% of response times)
• What if fix problem size, given 60%/year increase in
DRAM capacity?
Benchmark
I/OStones
Andrew
Size of Data
% Time I/O
Year
1 MB
4.5 MB
26%
4%
1990
1988
– Not much time in I/O
– Not measuring disk (or even main memory)
I/O Benchmarks: Transaction Processing
• Transaction Processing (TP) (or On-line TP=OLTP)
– Changes to a large body of shared information from many terminals,
with the TP system guaranteeing proper behavior on a failure
– If a bank’s computer fails when a customer withdraws money, the TP
system would guarantee that the account is debited if the customer
received the money and that the account is unchanged if the money
was not received
– Airline reservation systems & banks use TP
• Atomic transactions makes this work
• Each transaction => 2 to 10 disk I/Os & 5,000 and 20,000 CPU
instructions per disk I/O
– Efficiency of TP SW & avoiding disks accesses by keeping information
in main memory
• Classic metric is Transactions Per Second (TPS)
– Under what workload? how machine configured?
I/O Benchmarks: Transaction Processing
• Early 1980s great interest in OLTP
– Expecting demand for high TPS (e.g., ATM machines, credit cards)
– Tandem’s success implied medium range OLTP expands
– Each vendor picked own conditions for TPS claims, report only CPU
times with widely different I/O
– Conflicting claims led to disbelief of all benchmarks=> chaos
• 1984 Jim Gray of Tandem distributed paper to Tandem
employees and 19 in other industries to propose standard
benchmark
• Published “A measure of transaction processing power,”
Datamation, 1985 by Anonymous et. al
– To indicate that this was effort of large group
– To avoid delays of legal department of each author’s firm
– Still get mail at Tandem to author
I/O Benchmarks: TP1 by Anon et. al
• DebitCredit Scalability: size of account, branch, teller,
history function of throughput
TPS Number of ATMs Account-file size
10
1,000
0.1 GB
100
10,000
1.0 GB
1,000
100,000
10.0 GB
10,000
1,000,000
100.0 GB
– Each input TPS =>100,000 account records, 10 branches, 100
ATMs
– Accounts must grow since a person is not likely to use the bank
more frequently just because the bank has a faster computer!
• Response time: 95% transactions take Š 1 second
• Configuration control: just report price (initial purchase
price + 5 year maintenance = cost of ownership)
• By publishing, in public domain
I/O Benchmarks: TP1 by Anon et. al
• Problems
– Often ignored the user network to terminals
– Used transaction generator with no think time; made sense for database
vendors, but not what customer would see
• Solution: Hire auditor to certify results
– Auditors soon saw many variations of ways to trick system
• Proposed minimum compliance list (13 pages); still,
DEC tried IBM test on different machine with poorer results than
claimed by auditor
• Created Transaction Processing Performance Council in 1988:
founders were CDC, DEC, ICL, Pyramid, Stratus, Sybase, Tandem,
and Wang; ~40 companies today
• Led to TPC standard benchmarks in 1990, www.tpc.org
Unusual Characteristics of TPC
• Price is included in the benchmarks
– cost of HW, SW, and 5-year maintenance agreements included => priceperformance as well as performance
• The data set generally must scale in size as the throughput increases
– trying to model real systems, demand on system and size of the data
stored in it increase together
• The benchmark results are audited
– Must be approved by certified TPC auditor, who enforces TPC rules =>
only fair results are submitted
• Throughput is the performance metric but response times are limited
– eg, TPC-C: 90% transaction response times < 5 seconds
• An independent organization maintains the benchmarks
– COO ballots on changes, meetings, to settle disputes...
TPC Benchmark History/Status
Benchmark
Data Size (GB) Performance
Metric
0.1 to 10
transactions/
sec
0.1 to 10
transactions
per second
100 to 3000
new order
(min.07 * tpm) trans/min.
100, 300, 1000 queries/hour
A: Debit Credit
(retired)
B: Batch Debit Credit
(retired)
C: Complex Query
OLTP
D: Decision Support
(retired)
H: Ad hoc decision
100, 300, 1000
support
R: Business reporting
1000
decision support
W: Transactional web
~ 50, 500
benchmark
1st
Results
Jul-90
Jul-91
Sep-92
Dec-95
queries/hour Oct-99
queries/hour Aug-99
web interactions/sec
Jul-00
•
•
•
•
•
•
I/O Benchmarks: TPC-W
Transactional
Web
Benchmark
Represent any business (retail store, software distribution, airline
reservation, ...) that markets and sells over the Internet/ Intranet
Measure systems supporting users browsing, ordering, and
conducting transaction oriented business activities.
Security (including user authentication and data encryption) and
dynamic page generation are important
Before: processing of customer order by terminal operator working
on LAN connected to database system
Today: customer accesses company site over Internet connection,
browses both static and dynamically generated Web pages, and
searches the database for product or customer information.
Customer also initiate, finalize & check on product orders &
deliveries
Started 1/97; hoped to release Fall, 1998?Jul 2000!
1998 TPC-C Performance
tpm(c)
Rank
1 IBM
2
HP
3
Sun
4
HP
5Fujitsu
6
Sun
7Digital
8
SGI
9 IBM
10Digital
Config
RS/6000 SP (12 node x 8-way)
HP 9000 V2250 (16-way)
Ultra E6000 c/s (2 node x 22-way)
HP 9000 V2200 (16-way)
GRANPOWER 7000 Model 800
Ultra E6000 c/s (24-way)
AlphaS8400 (4 node x 8-way)
Origin2000 Server c/s (28-way)
AS/400e Server (12-way)
AlphaS8400 5/625 (10-way)
tpmC
$/tpmC
57,053.80
$147.40
52,117.80
$81.17
51,871.62
$134.46
39,469.47
$94.18
34,116.93 $57,883.00
31,147.04
$108.90
30,390.00
$305.00
25,309.20
$139.04
25,149.75
$128.00
24,537.00
$110.48
• Notes: 7 SMPs , 3 clusters of SMPs,
• avg 30 CPUs/system
Database
Oracle8 8.0.4
Sybase ASE
Oracle8 8.0.3
Sybase ASE
Oracle8
Oracle8 8.0.3
Oracle7 V7.3
INFORMIX
DB2
Sybase SQL
1998 TPC-C Price/Performance
$/tpm(c)
Rank
1
2
3
4
5
6
7
8
9
10
Acer
Dell
Compaq
ALR
HP
Fujitsu
Fujitsu
Unisys
Compaq
Unisys
Config
$/tpmC
AcerAltos 19000Pro4
$27.25
PowerEdge 6100 c/s
$29.55
ProLiant 5500 c/s
$33.37
Revolution 6x6 c/s
$35.44
NetServer LX Pro
$35.82
teamserver M796i
$37.62
GRANPOWER 5000 Model 670
$37.62
Aquanta HS/6 c/s
$37.96
ProLiant 7000 c/s
$39.25
Aquanta HS/6 c/s
$39.39
tpmC
11,072.07
10,984.07
10,526.90
13,089.30
10,505.97
13,391.13
13,391.13
13,089.30
11,055.70
12,026.07
• Notes: all Microsoft SQL Server Database
• All uniprocessors?
Database
M/S SQL 6.5
M/S SQL 6.5
M/S SQL 6.5
M/S SQL 6.5
M/S SQL 6.5
M/S SQL 6.5
M/S SQL 6.5
M/S SQL 6.5
M/S SQL 6.5
M/S SQL 6.5
2001 TPC-C Performance
Results
RankCompany System
1 Compaq ProLiant 8500-700-192P
2 IBM
Netfinity 8500R c/s
tpmC
$/tpmC CPUs Database Software
505,303 $ 19.80
192 SQL Server 2000
440,880 $ 32.28
128 DB2 UDB 7.1
3 Compaq
ProLiant 8500-X700-96P
4 Compaq
5 Bull
6 IBM
ProLiant 8500-X550-96P
229,914 $ 23.08
Escala EPC2450
220,807 $ 43.31
IBM eServer pSeries 680 Model 7017-S85
220,807 $ 43.30
96 SQL Server 2000
24 Oracle 8i Enterprise
24 Oracle 8i Enterprise
7 HP
HP 9000 Superdome Enterprise Server
197,024 $ 66.27
48 Oracle8 Enterprise E
8 Fujitsu
9 Compaq
PRIMEPOWER 2000 c/s w/32 Front-Ends
183,771 $ 56.16
ProLiant 8500-X700-64P
179,658 $ 19.75
64 SymfoWARE Serve
64 SQL Server 2000
IBM eServer iSeries 840-2420-001
163,776 $ 58.88
24 DB2 for AS/400 V4
10 IBM
262,244 $ 20.24
96 SQL Server 2000
• Notes: 4 SMPs, 6 clusters of SMPs: 76 CPUs/system
• 3 years => Peak Performance 8.9x, 2x/yr
2001 TPC-C Price Performance
Results
Company
1 Compaq
2 Dell
3 Dell
4 Dell
5 Dell
6 HP
7 Compaq
8 HP
9 Compaq
10 Compaq
System
ProLiant ML-570-6/700-3P
PowerEdge 6450/3P
PowerEdge 6400/3P
PowerEdge 6400
PowerEdge 6450
NetServer LH 6000
ProLiant ML-570-6/700
HP NetServer LXr 8500
ProLiant 8500-6/700-4
ProLiant 8500-550-6P
tpmC
20,207
24,925
24,925
30,231
30,231
33,136
32,328
43,047
34,600
33,617
$/tpmC CPUsDatabase Software
$ 9.51 3 SQL Server 2000
$ 9.90 3 SQL Server 2000
$ 9.91 3 SQL Server 2000
$ 11.07 4 SQL Server 2000
$ 11.08 4 SQL Server 2000
$ 11.85 6 SQL Server Enterprise
$ 12.49 4 SQL Server 2000
$ 12.76 8 SQL Server 2000
$ 12.89 4 SQL Server 2000
$ 12.91 6 SQL Server Enterprise
• Notes: All small SMPs, all running M/S SQL server
• 3 years => Cost Performance 2.9x, 1.4x/yr
SPEC SFS/LADDIS
• 1993 Attempt by NFS companies to agree on standard
benchmark: Legato, Auspex, Data General, DEC,
Interphase, Sun. Like NFSstones but
–
–
–
–
–
–
–
Run on multiple clients & networks (to prevent bottlenecks)
Same caching policy in all clients
Reads: 85% full block & 15% partial blocks
Writes: 50% full block & 50% partial blocks
Average response time: 50 ms
Scaling: for every 100 NFS ops/sec, increase capacity 1GB
Results: plot of server load (throughput) vs. response time &
number of users
» Assumes: 1 user => 10 NFS ops/sec
1998 Example SPEC SFS Result: DEC
Alpha
Avg. NSF Resp. Time
• 200 MHz 21064: 8KI + 8KD + 2MB L2; 512 MB; 1
Gigaswitch
• DEC OSF1 v2.0
• 4 FDDI networks; 32 NFS Daemons, 24 GB file size
• 88 Disks, 16 controllers, 84 file systems
50
40
4817
30
20
10
0
0
10 00
20 00
30 00
40 00
NFS Throughput (nfs ops/sec )
50 00
SPEC sfs97 for EMC Celera NFS servers
2, 4, 8, 14 CPUs; 67, 133, 265, 433 disks
15,700, 32,000, 61,800 104,600 ops/sec
2 CPU s
4 C PUs
8 CPUs
1 4 CPU s
10
9
8
7
6
5
4
3
Ove ra ll response tim e (m s)
2
1
0
0
20000
40 000
60000
SPEC sfs 97.v3 Ops/se c
8 0000
10000 0
1 20000
SPEC WEB99
• Simulates accesses to web service provider, supports home
pages for several organizations. File sizes:
–
–
–
–
less than 1 KB, representing an small icon: 35% of activity
1 to 10 KB: 50% of activity
10 to 100 KB: 14% of activity
100 KB to 1 MB: a large document and image,1% of activity
• Workload simulates dynamic operations: rotating
advertisements on a web page, customized web page creation,
and user registration
• Workload gradually increased until server software is saturated
with hits and response time degrades significantly
SPEC WEB99 for Dells in 2000
System Name
PowerEdge 2400/667
PowerEdge 2400/667
PowerEdge 4400/800
PowerEdge 4400/800
PowerEdge 6400/700
PowerEdge 6400/700
ResultHTTP Version/OS
CPUsCPU typeDRAM
732 IIS 5.0/Windows 2000
1 667 MHz2 Pentium
GB
1270 TUX 1.0/Red Hat Linux 6.21 667 MHz2 Pentium
GB
1060 IIS 5.0/Windows 2000
2 800 MHz4 Pentium
GB
2200 TUX 1.0/Red Hat Linux 6.22 800 MHz4 Pentium
GB
1598 IIS 5.0/Windows 2000
4 700 MHz8 Pentium
GB
4200 TUX 1.0/Red Hat Linux 6.24 700 MHz8 Pentium
GB
• Each uses 5 9GB, 10,000 RPM disks except the 5th system, which
had 7 disks, and the first 4 have 0.25 MB of L2 cache while the last 2
have 2 MB of L2 cache
• Appears that the large amount of DRAM is used as a large file cache
to reduce disk I/O, so not really an I/O benchmark
Summary: I/O Benchmarks
• Scaling to track technological change
• TPC: price performance as nomalizing
configuration feature
• Throughput with restricted response time is
normal measure
• Benchmarks to measure Availability,
Maintainability?
Summary: A Little Queuing Theory
System
Queue
Proc
server
IOC
Device
• Queuing models assume state of equilibrium:
input rate = output rate
• Notation:
r
Tser
u
Tq
Tsys
Lq
Lsys
average number of arriving customers/second
average time to service a customer (tradtionally µ = 1/ Tser )
server utilization (0..1): u = r x Tser
average time/customer in queue
average time/customer in system: Tsys = Tq + Tser
average length of queue: Lq = r x Tq
average length of system : Lsys = r x Tsys
• Little’s Law: Lengthsystem = rate x Timesystem
(Mean number customers = arrival rate x mean service time)