Transcript Outlook to PCI Express

Who We Are?
Detector Building Group of
KFKI-RMKI
(Research Institute for Particle and Nuclear Physics),
Budapest, HUNGARY
Our Results
Previous projects:
 Designing Fibre Channel test equipments (1994-1998)
 Portable, 266 Mb/s Fibre Channel Tester
 Fibre Channel Preprocessor for Logic Analyzers
 Designing high-speed data transmission interfaces
for CERN detectors (1996 - ...)




Protocol design and verification
FC, GbE, physical layer components
Hardware design
Software: linux drivers, test programs, program library
S-LINK (CERN)
 S-LINK interface cards for CERN (mostly used at ATLAS)
 simple, unidirectional link interface
 first successful version was designed by KFKI-RMKI
 newer versions, now: 2.5 Gb/s
 Also used at MPI, Garching and at several HEP and other
scientific institutes all over the world
ALICE Detector Data Link (DDL)
DDL
 Detector Data Link (DDL)
for the ALICE detector at CERN
 2.5 Gb/s, duplex link
 advanced featues
 Test devices for DDL
(Front-end emulator, DDL link emulator, etc.)
 PCI Read-Out Receiver Card
(RORC) for DDL
 1st version: 33 MHz, 32-bit PCI card
 2nd version: 66 MHz, 64-bit PCI
with 2 integrated DDL interfaces
DDL Interfaces
DIU -RORC Interface
FEE - SIU Interface
Front-end Bus
32 bit
Front-end
Electronics T
A
P
4
1
DDL
Source
Interface
Unit
(SIU)
DDL
Destination
Interface
optical cable, max 350 m
Unit
(DIU)
JTAG BST lines
DDL
32 bit
32 bit
Read-out
Receiver
Card
(RORC)
66 MHz
64-bit
PCI
DDL Features
Interface:
 Full duplex 32-bit data path on the destination interface (DIU card)
 Half duplex 32-bit data path on the source interface (SIU card)
 Full duplex flow control (XON/XOFF)
 Interface clock up to 66 MHz (easy integration with PCI 66)
 264 MB/s peak data rate, 240 MB/s sustained bandwidth (max.)
Implementation:
 Duplex LC optical link up to 300 m
 2x FC or 2x GbE physical layer components
 Small Form Factor Pluggable (SFP) optical transceivers
 Bit error rate < 10 -12
 Robust error detection: very low undetected bit error rate < 10-40
 Automatic link synchronization and management
Extras:
 Stand-by support (low power consumption)
 In-system reconfiguration / Remote system upgrade
 Monitoring of the aging of laser diode of optical transceivers
 JTAG Boundary Scan Test interface for the Front-End electronics
Outlook to PCI Express
PCI Express (formerly 3GIO)
a „third generation” high performance I/O bus
(1st generation: ISA, EISA, VESA, 2nd generation: PCI, PCI-X)
PCI evolution (PCI, PCI-X, PCI-Express)
PCI Express is software compatible to PCI and PCI-X
PCI Evolution
 PCI
 „multi-drop” parallel bus
 conventional PCI: 33 MHz, 32-bit, max 4-5 card slots per bus
 newer versions: 66 MHz, 64-bit, 1 (max 2) card slots per bus (!)
 PCI-X
 parallel bus, backward compatible (hw and sw)
 66 MHz , 32/64-bit, max 4-5 card slots per bus
 133 MHz, 32/64-bit, max 1-2 card slots per bus
 266, 532 MHz versions: max 1 card slot per bus (!)
Buses can be bridged to each other (complex, expensive)
 PCI-XP
 It is still a local /IO bus, a „PCI bus”, but the connections
between devices are serial, point-to-point interconnections
 devices are interconnected via switch(es)
 large number of devices can be interconnected
 highly scalable, hot-plug, hot-swap, QoS, etc.
Limitations of PCI and PCI-X
 Conventional 33 MHz PCI system
 low bandwith to nowaday’s needs
 66 MHz, 133 MHz (PCI-X), ...
 only few devices can be interconnected (only 1 or two) on a single bus
because of the strict electrical load and timing constraints
 Further limitations of PCI architecture
 inefficient solutions in:
 data transfer cycles (wait states)
 accessing of system memory
 interrupt handling
 error handling
PCI Express
 However, the basic problems of a parallel bus system
(electrical load and timing constraints, lack of hotpluggability, lack of scalability, etc.) can be solved
only by a complete redesign of the architecture.
 This resulted in PCI Express.
The PCI Express Link
 Serial point-to-point interconnect between two devices
a PCI Express link
Device A
Device B
(e.g. a switch)
 1x, 2x, 4x, 8x, 12x, 16x, 32x type links
 1, 2, 4, ... 32 bidirectional signal pairs (lanes)
 1 lane: 2.5 Gb/s now, up to 10 Gb/s later
 Low voltage, differential signaling (LVDS)
 AC coupled
 Data is encoded: 8B/10B
1 to 32 lanes
Bandwith
 Scalable: more lanes /link: higher bandwith
 2.5 Gb/s per lane, 8B/10B encoding
 Simultaneous traffic in both directions
 1x type link
 500 MByte/s aggregate bandwith
 250 MByte/s per direction
 32x type link:
 16 GByte/s aggregate bandwith
 (8 GByte/s per direction)
Backward compatibility
 Supports familiar PCI transactions such as memory




read/write, IO read/write and configuration read/write
Same memory, IO and configuration address space
as in PCI and PCI-X
Existing OSs and driver software will run in a
PCI Express system without any modifications
Supports chip-to-chip interconnect and board-to-board
interconnect via cards and connectors similar to the present
PCI systems
PCI Express motherboard will have a similar form factor to
existing ATX motherboards in PCs
Improvements, Benefits

Fast, highly scalable, serial point-to-point interconnect between two devices


bytes striped accross the lanes  more lanes per link: faster transmission
Packet-based communication protocol



Packets are transmitted serially
CRC embedded in each packet (auto retry: link level error correction)
Buffer-to buffer (link level) flow control  no need the packet retry

Message Signaled Interrupt (MSI) architecture

No side-band signals


Multiple devices are interconnected via switches


hot-plug, error handling, interrupt signaling and else are accopmlished in-band
Large number of devices can be connected together in a system
Much fewer pins per device package

Reduces chip and board design cost and design complexity

Reconfigurable, hot-plug, hot-swap, improved power management, etc.

Quality of Service (QoS) features: Traffic Classes, Virtual Channels

Configuration address space of devices is extended from 256B to 4KB

This needs new software
Outside the box?
 Initial focus of usage:
inside the box
 It is expected that later it will
also be used outside the box
for I/O expansion
PCI-XP links
Add-in
card
optical
cable
PCI-XP links
Add-in
card
switch
switch
connector
PCI-XP link
Mechanical Form Factors
 Connector and daughter card form factors are under
specification
 The main type of connectors is very similar to the
present PCI card edge connector
 A 1x type card can be inserted in a x4 type slot, a.s.o.
 Like with PCI, there will be other form factors
 Server I/O module
 Mini PCI Express card and connector (e.g. for notebooks)
 mezzanine type card
 NEWCARD (will replace CardBus PC card), etc.
Compete or complete?
 PCI Express will coexist with PCI / PCI-X in the same system
 PCI Express intends to replace AGP (graphics card if),
but will not replace Serial ATA (hard disk, CD, etc)
 HyperTransport
 onboard chip-to-chip interconnect
 said to be complementary to PCI Express
 RapidIO
 as an onboard chip-to-chip interconnect: complementary to PCI-Express
 as a system interconnect: a competitive technology
 1394b, USB2.0, Fibre Channel, Gb Ethernet, Infinband, etc.
 PCI Express can be a system interconnect that bridges these technologies
 Will it be?