iSCSI Storage

Dennis Martin Demartek May 2007 RMWTUG Meetings

Introduction

♦ Dennis Martin, President, Demartek ♦ 27 years in I.T.

– Software Engineer – Software Engineering Manager – Data Center Manager – Marketing – Industry Analyst – Mainframe, UNIX, Windows ♦ Founded RMWNTUG in Denver in 1994 ♦ Microsoft Storage MVP 2005, 2006, 2007 © 2007 Demartek 2

Pre-Presentation Survey

♦ How many are using iSCSI storage today?

♦ How many are planning to use iSCSI storage?

♦ How many have heard of iSCSI but want to know more?

© 2007 Demartek 3

iSCSI Storage – Agenda

♦ Storage: DAS, NAS and SAN ♦ iSCSI Concepts ♦ Network Technology Items ♦ Storage Technology Items ♦ Solutions Available Today ♦ Microsoft iSCSI Best Practices ♦ Performance ♦ Futures ♦ Recommendations ♦ Reference links © 2007 Demartek 4

Direct Access Storage (DAS)

♦ Storage is directly connected to CPU ♦ Short distance between CPU and storage (inches or small number of meters) ♦ Storage is “owned” by one host computer only ♦ Limited number of storage devices can be connected ♦ Common interfaces are ATA/IDE, SCSI, SATA, SAS © 2007 Demartek 5

DAS Diagram

CPU Inside the Host Computer Cabinet Storage Interface Memory © 2007 Demartek Storage Device Due to cable length limitations, the storage devices often reside in the same cabinet as the CPU, or in a separate enclosure physically near the CPU cabinet.

Host CPU performs “block” I/O directly to device.

© 2007 Demartek 6

Network Attached Storage (NAS)

♦ One host server shares its resources with other clients on the network ♦ Clients make requests by filename ♦ NAS server has its own storage, clients only see “share” names ♦ Large number of “shares” can be created ♦ NAS server can be a long distance from clients ♦ Common protocols are NFS and CIFS © 2007 Demartek 7

NAS Diagram

Network clients Local Area Network (LAN) Network Attached Storage (NAS) Diagram NAS Server © 2007 Demartek NAS Private Storage Network clients request files from file server (NAS device) over the LAN.

NAS device receives file requests from network clients and translates those requests to “block” I/O commands to its own private storage. It then formats its data into a “file” format and responds to the network clients.

© 2007 Demartek 8

Storage Area Network (SAN)

♦ Infrastructure allows computers to appear to have direct connection to storage ♦ Storage can be great distance from computers requiring access ♦ Clients make “block” I/O requests ♦ Can be millions of storage devices in a SAN ♦ Two technologies: Fibre Channel and Ethernet © 2007 Demartek 9

SAN Diagram

Storage Area Network (SAN) Diagram Network clients Local Area Network (LAN) Application Servers Storage Area Network (SAN) Storage Devices © 2007 Demartek Storage traffic for SAN storage generally flows over a separate network from the LAN traffic.

© 2007 Demartek 10

SAN Components

♦ Server ♦ SAN Interface Card ♦ SAN Switch ♦ Storage Device (with Interface) © 2007 Demartek 11

SAN Component Diagram

Storage Area Network (SAN) Component Diagram Application Server Interface card Switch Storage Device © 2007 Demartek © 2007 Demartek 12

Unified Storage

♦ Combination of NAS and SAN into single, integrated storage solution ♦ Often includes: – Fibre Channel SAN technology – iSCSI SAN technology – NAS technology © 2007 Demartek 13

iSCSI SAN Components

iSCSI SAN Components Application Server Ethernet Switch Ethernet adapter Ethernet cabling iSCSI Target Storage Device © 2007 Demartek For iSCSI today, Ethernet components should be 1-Gigabit or better.

Cabling and interfaces can be copper or fiber-optic.

Ethernet adapters can be on the motherboard or in a PCI/PCI-X/PCI-E slot.

© 2007 Demartek 14

iSCSI Initiators

♦ Same basic function as SCSI initiators ♦ Available in software or hardware adapters ♦ Microsoft provides free software initiator – Free download for : Windows Server 2003, Windows XP Pro SP1, Windows 2000 SP3 – Included in Control Panel in Windows Vista, Longhorn Server – Available in 32-bit and 64-bit ♦ Third-parties sell iSCSI Host Bus Adapters (HBA) ♦ Initiators have iSCSI Qualified Name (IQN) © 2007 Demartek 15

iSCSI Targets

♦ Same basic function as SCSI targets ♦ Available in software or hardware ♦ Wide variety of hardware targets available today that can fit price, capacity or performance needs – SATA disk arrays – SAS disk arrays – Fibre Channel disk arrays ♦ Targets have iSCSI Qualified Name (IQN) © 2007 Demartek 16

1

Microsoft iSCSI Initiator Install

3 2 MPIO not checked by default © 2007 Demartek 17 4

Microsoft iSCSI Initiator

Two target portals visible to this initiator Three targets (volumes) visible to this initiator Notice the IQNs © 2007 Demartek 18

Multi-path I/O (MPIO)

♦ MPIO is supported for iSCSI ♦ Same MPIO as Fiber Channel ♦ Allows multiple network connections from the same host server to function as data paths to the target ♦ Different load balance policies are available © 2007 Demartek 19

iSCSI Security

♦ iSCSI security should be considered in addition to other network security that may be implemented ♦ Challenge Handshake Authentication Protocol (CHAP) – Predefined shared secret between initiators and targets – Only allows logon if the other party knows the secret – Can be one-way or mutual authentication ♦ IPSec – If enabled, all data transfer packets are encrypted and authenticated © 2007 Demartek 20

Network Technology Items

♦ These affect iSCSI

AND

your regular network ♦ Server-class NICs ♦ Scalable Networking Pack ♦ Receive-side Scaling ♦ TCP Offload Engines (TOE) ♦ Switches ♦ Jumbo Frames © 2007 Demartek 21

Server-class NICs

♦ There’s a big difference between a low-cost desktop NIC and a server-class NIC – Server-class NICs typically have some degree of offload function such as

TCP Checksum Offload

,

Large Send Offload

and other similar functions – Server-class NICs have equal or better performance with lower CPU utilization on the server – Server-class NICs are more expensive than low-cost NICs © 2007 Demartek 22

Scalable Networking Pack

♦ Microsoft Scalable Networking Pack (SNP) – Free download for Windows 2003 Server and Windows XP 64-bit – Included in Windows Server 2003 R2 SP2 ♦ Provides network acceleration and compatibility with hardware-based offload technologies – Receive-side Scaling – TCP Offload Engines – NetDMA ♦ Improves all network traffic, not just iSCSI © 2007 Demartek 23

Receive-side Scaling

* Also known as “Scalable I/O” on Linux systems ♦ Especially important with multi-core and multi processor systems ♦ Without SNP and Receive-side Scaling, all incoming network traffic interrupts are routed to exactly one core ♦ With SNP and Receive-side Scaling, incoming network traffic interrupts are distributed across the cores and processors ♦ Requires a NIC that supports Receive-side Scaling ♦ Provides very good performance © 2007 Demartek 24

Receive-side Scaling

© 2007 Demartek 25

TCP Offload Engine (TOE)

♦ TCP Offload NIC handles all TCP/IP processing ♦ Reduces CPU utilization on host server ♦ Provides overall best performance ♦ More expensive than server-class NICs with Receive-side Scaling © 2007 Demartek 26

CPU Utilization Graphs

Sequential Read 50 40 30 20 100 90 80 70 60 NIC-LOW NIC-SVR NIC-TOE

Queue depth = 4 Paths = 2

10 0 512B 1K 2K 4K 8K 16K 32K Block Size 64K 128K 256K 512K 1M

Percent CPU Utilization

Sequential Write 50 40 30 20 100 90 80 70 60 NIC-LOW NIC-SVR NIC-TOE

Queue depth = 4 Paths = 2

10 0 512B 1K 2K 4K 8K 16K 32K Block Size 64K 128K 256K 512K 1M

Affect of NIC type on CPU Utilization © 2007 Demartek 27

Switches, Jumbo Frames

♦ Switches used for iSCSI should be Gigabit or better ♦ When jumbo frames are enabled, must be enabled in the switch and all NICs connected to that switch, including iSCSI initiators and targets – Make sure all components are using the same jumbo frame size © 2007 Demartek 28

Storage Technology Items

♦ RAID type ♦ Stripe sizes ♦ Disk drive type ♦ MPIO in the target ♦ Caching © 2007 Demartek 29

RAID Definitions

♦ RAID – Redundant Array of Inexpensive Disks – RAID 0: Interleaving or “striping” data across two or more disks – RAID 1: Disk mirroring – same data written on two different disks – RAID 5: Data striping with parity across multiple disks – RAID 6: Data striping with double parity across multiple disks – RAID 1+0 or RAID 10: combine RAID 1 and RAID 0 • Two variations: stripe first or mirror first © 2007 Demartek 30

RAID 0 A C B D

RAID Types

Data: “ABCD” RAID 1 ABCD ABCD A C RAID 10 B D A C B D A D B RAID 5 C P1 A D B RAID 6 C P1 © 2007 Demartek P2 31

Stripe Sizes

♦ Size of data block that is striped across disk drives in a RAID group ♦ I/O of larger than the stripe size causes the target to perform extra work to complete the I/O © 2007 Demartek 32

Disk Drive Type

♦ Enterprise class (SAS, SCSI, Fibre Channel) – Designed to be operated 7 x 24 x 365 – Dual internal processors (data transfer, physical motion) ♦ Desktop class (SATA, ATA/IDE) – Largest capacities, best price/capacity ♦ Notebook class – Low power consumption, slower ♦ Consumer class – Very low power consumption and weight © 2007 Demartek 33

Device Seek time Xfer rate (MB/s)* RPM (K) Capacities Processors Cmd. Queuing Power need

Drive Type Chart

Enterprise 3 – 5 ms 70 – 120 10, 15 Large 2 TCQ High Desktop 8 – 10 ms 60 – 80 5.4, 7.2, 10 Very large 1 NCQ Medium Notebook 10 – 12 ms 30 – 60 4.2, 5.4, 7.2

Medium 1 Low Consumer ~12 ms 10 – 30 3.6, 4.2

Small 1 Very low Warranty 5 years 3 – 5 years 1 – 3 years 1 year * Maximum device transfer rate from drive surface to buffer. Sustained rates are lower. This is not the same as the interface transfer rate.

© 2007 Demartek 34

Caching

♦ Different iSCSI target solutions use different caching – Some are based on Windows Server platform – Some are based on Linux platform – Some use proprietary platform © 2007 Demartek 35

Solutions Available Today

♦ Most of the major disk storage vendors provide iSCSI solutions ♦ Several iSCSI-only vendors are producing very good solutions ♦ Wide variety of price, capacity, performance and advanced feature selection is available ♦ Generally speaking, iSCSI storage vendors have done a good job of making tasks such as LUN provisioning very easy to accomplish © 2007 Demartek 36

Microsoft iSCSI Best Practices

♦ Use true server-class NIC ♦ Separate iSCSI traffic from all other network traffic ♦ Use enterprise-class managed switches © 2007 Demartek 37

Performance

♦ I have personally used iSCSI target solutions that: – Achieve greater than 30,000 IOPs (with two paths) – Achieve greater than 200 MB/sec (with two paths) ♦ You can have great performance with iSCSI © 2007 Demartek 38

Futures

♦ 10-Gigabit Ethernet – Currently used primarily in clustering and network trunking applications – Expect prices to drop, especially as PCI-Express becomes the dominant bus type in servers ♦ Storage Management Software – Expect iSCSI target vendors to make more information available to centralized storage management software solutions ♦ Microsoft iSCSI Deployment Paper – soon!

© 2007 Demartek 39

Recommendations

♦ Make sure you have Scalable Networking Pack installed and use either Receive-side Scaling capable NICs or TCP Offload Engines ♦ Take advantage of MPIO © 2007 Demartek 40

References

♦ ♦ ♦ ♦ ♦ Microsoft Storage www.microsoft.com/storage/ Microsoft iSCSI Storage www.microsoft.com/WindowsServer2003/technologies/storage/iscs i/default.mspx

Microsoft Storage Technical Articles and White Papers www.microsoft.com/windowsserversystem/storage/indextecharticle.

mspx Microsoft Scalable Networking Pack www.microsoft.com/technet/network/snp/default.mspx

Microsoft Exchange Solution Reviewed Program – Storage technet.microsoft.com/en-us/exchange/bb412165.aspx

© 2007 Demartek 41

Post-Presentation Survey

♦ How many would be interested in a day-long, vendor-neutral, storage class?

♦ What topics would be interesting to you in such a class?

© 2007 Demartek 42

Thank You

Dennis Martin Demartek (303) 940-7575 Dennis @ demartek.com

www.demartek.com

© 2007 Demartek 43