11/05/07

TDC 564 Local Area Networks

Lecture 8 IP-based Storage Area Network

TDC564-08 1

11/05/07

Course Outline

   IP over FC (RFC 2625) IP-SAN  iSCCI (RFC 3720) IP and FC-SAN Interworking  FC Encapsulation (RFC 3643)  FCIP (RFC 3821) – FC over IP  iFCP (RFC 4172) TDC564-08 2

RFC 2625 – IP and ARP over FC

   FC supports multiple higher layer protocols, and SCSI is the most widely used one.

Needs/Applications of IP over FC:  Access data in SAN from IP-based servers  interworking between NAS and SAN RFC 2625 addresses two issues.

  A scheme to encapsulate IP and ARP packets inside the FC frame (as the FC payload) A procedure to resolve the address mapping 11/05/07 TDC564-08 3

IP over FC (RFC 2625) App-1: accessing SAN from IP-based servers

11/05/07 Data IP L2 PHY SAN FC/IP Gateway IP L2 PHY IP RFC 2625 FC FC SAN TDC564-08 Data IP RFC 2625 FC FC-based Storage Device 4

Data IP L2 PHY 11/05/07

IP over FC (RFC 2625) (App-2: interworking between SAN and NAS)

SAN FC/IP Gateway IP L2 PHY IP RFC 2625 FC FC SAN FC/IP Gateway IP RFC 2625 FC IP L2 PHY NAS-based Storage Device Data IP L2 PHY TDC564-08 5

FC Address Resolution

  Three steps schemes  ARP - IP to MAC   MAC => World-Wide Port Name (WW_PN) FC-ARP - WW_PN => Port_ID Optional Header is used for Network Header by RFC 2625 D_NAA (4) Dst. Network Address (60) S_NAA (4) Src. Network Address (60) Format of Network Header 16 bytes NAA: Network Address Authority unused (12 bits) IEEE 802.3 MAC Address (48 bits) Format of Network Address (WW_PN) 11/05/07 TDC564-08 6

RFC 2625: IP and ARP over FC (1) ARP Request (broadcast) - Sender HW (MAC) Address = HW(A) - Sender IP Address = IP(A) - Target HW (MAC) Address = ??

- Target IP Address IP(B) A

FC SAN FC switch 11/05/07

B C D

TDC564-08 7

RFC 2625: IP and ARP over FC (2) ARP Reply (unicast): - Sender HW (MAC) Address = HW(B) - Sender IP Address = IP(B) - Target HW (MAC) Address = HW(A) - Target IP Address IP(A) A

11/05/07

B

TDC564-08

C D

8

RFC 2625: IP and ARP over FC (3) FC-ARP Request: - Requester N_Port_Name (WW_PN) - Requester N_Port_ID - Responder N_Port_Name (WW_PN) - Responder N_Port_ID = ??

A B

11/05/07

C

unused (12 bits) IEEE 802.3 MAC Address (48 bits) WW_PN TDC564-08

D

9

RFC 2625: IP and ARP over FC (4)

FC-ARP Reply: - Requester N_Port_Name - Requester N_Port_ID - Responder N_Port_Name - Responder N_Port_ID A

11/05/07

B

TDC564-08

C D

10

Advantages of IP for SAN

11/05/07 TDC564-08 11

IP Network Capabilities

11/05/07 TDC564-08 12

IP-SAN Protocols

11/05/07 TDC564-08 13

IP - SAN

IP 11/05/07 FC -SAN IP TDC564-08 Storage Devices 14

IP - Storage Area Network (SAN)

    IP storage networking – carrying storage traffic over IP Uses TCP, a reliable transport for delivery Can be used for local data center and long haul applications Two primary IETF protocols/standards: 

iSCSI

– Internet SCSI – allows block storage to be accessed over a TCP/IP network as though it were locally attached

IP TCP iSCSI SCSI Data



FCIP

– Fibre-Channel-over-IP – used to tunnel Fibre Channel frames over TCP/IP connections

IP TCP FCIP FC SCSI Data

11/05/07 TDC564-08 15

Internet SCSI (iSCSI)

     iSCSI is a proposed industry standard that allows SCSI block I/O protocols (commands, sequences, and attributes) to be sent over a network using the popular TCP/IP protocol. A way to access storage across an IP network as though it was locally attached.

Transports SCSI protocol commands and data across an IP network Cisco and IBM co-authored original iSCSI protocol draft iSCSI Protocol is a standard maintained by the IETF  IP Storage (IPS) Working Group  RFC 3720 11/05/07 TDC564-08 16

iSCSI

11/05/07 TDC564-08 17

iSCSI over TCP/IP

SCSI Command Command Data Command Data TCP Header TCP Payload 11/05/07 TDC564-08 18

iSCSI Applications

   Connectivity : iSCSI can be used for DAS or SAN connections. iSCSI capable devices could be placed on an existing LAN (shared with other applications) in a similar way to NAS devices.

iSCSI capable devices could be attached to a LAN which is dedicated to storage I/O (in other words an IP SAN), or even on a LAN connected to only one processor (like a DAS). 11/05/07 TDC564-08 19

Software iSCSI

Data SCSI iSCSI TCP IP Driver Ethernet PHY 11/05/07

iSCSI Protocol Stack iSCSI TCP Offload Engine

Data SCSI

iSCSI Host Bus Adapter (HBA)

Data SCSI Host Processing iSCSI Driver TCP IP Ethernet PHY TDC564-08 Driver iSCSI TCP IP Ethernet PHY Host Bus Adapter (HBA) Adapter Card 20

iSCSI Performance Consideration

11/05/07 TDC564-08 21

iSCSI Benefit

11/05/07 TDC564-08 22

Host Bus Adapter (Fibre Channel)

QLogic PCI-EXPRESS 4Gb 1PT Fibre Host Bus Adapter – STD FW • PCI-Express • • 1 Port (Fibre Channel) • Data Rate: 4Gb • Standard Firmware Price Quote: $951 Source: http://www.cdw.com

Host Bus Adapter (iSCSI)

10/100/1000BaseT PCI The QLE4060C iSCSI Host Bus Adapter (HBA) provides connectivity to SANs over Ethernet and TCP/IP network infrastructures for PCI Express bus-based servers. Using a widely deployed and familiar networking technology, iSCSI can lower the total cost of ownership to better manage and support storage networking.

Source: http://www.cdw.org

Price quote: $815 11/05/07 TDC564-08 24

11/05/07

iSCSI Storage

Promise Technology VTrak M-Class iSCSI RAID Price quote: $4,929 Source: //www.cdw.com

Network: Gigabit Ethernet (2) Protocol: TCP/IP, iSCSI Capacity: 16-drive bay (500G – 1TB per drive) TDC564-08 25

Data SCSI iSCSI TCP IP Layer2 PHY Server

iSCSI Application

IP iSCSI TCP IP Layer2 PHY SCSI FCP FC 0-2 iSCSI Gateway Data SCSI FCP FC 0-2 Storage Device Cisco iSCSI Configuration Example 26 11/05/07 TDC564-08

Multi-TCP Sessions

11/05/07 TDC564-08 27

HBA: Multi-Path I/O

11/05/07 TDC564-08 LUN: Logical [Storage] Units 28

HBA iSCSI vs. Software iSCSI

    No performance advantage of using iSCSI HBA  Throughput  Round trip time Significantly higher cost of iSCSI HBA  $800 vs. $100 (Server NIC) What are the advantages of iSCSI HBA?

 Answer: CPU Utilization (off-load) Reference: http://download.intel.com/network/connectivity/products/iSCSIcomp0407.pdf

11/05/07 TDC564-08 29

11/05/07

Is IP-SAN similar to NAS?

What are the advantages, if any, of IP-SAN vs. NAS?

TDC564-08 30

Performance Analysis: iSCSI vs. NAS (software based – no HBA)

11/05/07 iSCSI NFS http://technomagesinc.com/pdf/ip_paper.pdf

TDC564-08 31

Sequential Read/Write Tests

Conclusion: 1. Comparable performance in character read/write and block write 2. Significant advantage of iSCSI in block read (20-25%) 11/05/07 TDC564-08 32

Small Files Read/Write

11/05/07 TDC564-08 33

IOGen Test (Emulation of Database)

11/05/07 TDC564-08 34

FC-SAN vs. iSCSI

    Since the iSCSI appliance attaches to the existing Ethernet network, NAS and iSCSI are very similar in network architecture  However, the performance would be significantly different.

Both iSCSI and SAN use Block I/O to transport data, whereas NAS uses File I/O. SAN offers better performance (c.f. NAS), but is more expensive and requires a higher skill set to implement. iSCSI and NAS offer better pricing and skills may already be in place to implement them. Both SAN and iSCSI offer the performance benefit of Block I/O.

11/05/07 TDC564-08 35

Case Study: FC-SAN vs. iSCSI

11/05/07 http://www.promise.com/marketing/whitepaper/file/2_MClass iSCSI%20SCSI%20and%20Fibre-10-07-05.pdf

TDC564-08 36

11/05/07  4G bps TDC564-08 37

FC over IP

• The encapsulation of Fibre Channel frames into IP packets and

tunneled through an existing TCP/IP network infrastructure, in order to connect remote islands of FC-based SANs

FC SAN IP 11/05/07

IP TCP FCIP FC SCSI Data

TDC564-08 FC SAN 38

FC Frame Encapsulation (RFC 3643)

 Describes the common Fibre Channel (FC) frame encapsulation format and a procedure for the measurement and calculation of frame transit time through the IP network Encapsulation Header SOF FC Frame Body EOF 11/05/07 TDC564-08 39

FC Encapsulation Header

11/05/07 Q: Why is time stamp needed in the FC encapsulation header?

TDC564-08 40

FC Encapsulation

11/05/07 TDC564-08 41

FCIP (RFC 3821)

Fibre Channel Over TCP/IP (FCIP) describes mechanisms that allow the interconnection of islands of Fibre Channel storage area networks over IP-based networks to form a unified storage area network in a single Fibre Channel fabric 11/05/07 TDC564-08 42

FCIP Protocol Stacks

11/05/07 TDC564-08 43

SAN

FC over IP

IP SAN Data SCSI FCP FC 0-2 an IP tunnel for FC-based SAN Application: interconnect SAN over IP-WAN.

FCP FC 0-2 FCIP TCP IP L2 PHY IP L2 PHY FCIP TCP IP L2 PHY FCP FC 0-2 Data SCSI FCP FC 0-2 11/05/07 TDC564-08 44

FCIP Configuration Example

11/05/07 TDC564-08 45

Internet Fiber Channel Protocol (iFCP)

 iFCP – RFC 4172 (Sept. 2005)  iFCP is a protocol that uses the FCP over IP networks to interconnect FC-based storage devices.

 It is considered a gateway protocol, while FCIP is considered a tunnel protocol.

11/05/07 TDC564-08 46

iFCP (RFC 4172)

   iFCP specifies an architecture and a gateway-to-gateway protocol for the implementation of fibre channel fabric functionality over an IP network. This functionality is provided through TCP protocols for fibre channel frame transport and the distributed fabric services specified by the fibre channel standards. The architecture enables internetworking of fibre channel devices through gateway-accessed regions with the fault isolation properties of autonomous systems and the scalability of the IP network 11/05/07 TDC564-08 47

iFCP Network Example

11/05/07 TDC564-08 48

Data SCSI FCP FC 0-2 F-port

iFCP (RFC 4172)

F-port IP iFCP GW iFCP GW Gateway approach Application: interconnect SAN over IP-WAN.

FCP FC 0-2 iFCP TCP IP L2 PHY IP L2 PHY iFCP FCP FC 0-2 TCP IP L2 PHY Data SCSI FCP FC 0-2

iFCP

11/05/07 TDC564-08 50

FCIP vs. iFCP

http://whatis.techtarget.com/magPrintFriendly/0,293813,sid35_gci1096482,00.html

11/05/07 TDC564-08 51

FCIP vs. iFCP

11/05/07 TDC564-08

McDATA

52

SAN Performance Comparison

11/05/07 http://www.netapp.com/library/tr/3423.pdf

TDC564-08 53

11/05/07

Summary

     Needs for large storage – continual growth  10 9 (G) => 10 12 (T) => 10 15 (P) => 10 18 (E) …….

From dedicated solution to network-based solution  DAS => NAS => SAN => IP-SAN Convergence of SAN and IP-LAN/WAN  It is an IP world!

SCSI is the protocol for block data transmission   SCSI over FC SCSI over IP (iSCSI) FC and IP interworking protocols   IP over FC FC over IP (FCIP) and iFCP TDC564-08 54