Transcript No Slide Title

A
Smart Port Card
(SPC and SPC-II)
Tutorial
--Hardware
John DeHart
Washington University
[email protected]
http://www.arl.wustl.edu/~jdd
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
1
Motivation
• Active Networking
• Network Probe
• High performance router architectures
– PC as router is VERY limited
– (Gigabit/s + Processing) on each port
– MSR: Multi-Service multiport Router
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
2
The Original Smart Port Card
• Hardware:
– SPC as a PC
• How do they each boot?
– SPC Hardware Components
• What roles do they play?
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
3
Typical Pentium PC
Addr/Data
Ctrl
CPU/Memory Bus
Ctrl
CPU
NorthBridge
DRAM
Addr/Data/Ctrl
PCI Bus
INIT
NMI
Intr
Cache
SouthBridge (PIIX3)
(PIC, PIT, …)
PCI
Devices
ISA Bus
June 16, 2002
SPC Tutorial
Super-IO
BIOS
RTC
Uarts
Kbd/Mse
Floppy
Parallel
...
ISA
Devices
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
4
Typical Pentium PC: CPU/Memory Bus
Addr/Data
Ctrl
CPU/Memory Bus
Ctrl
CPU
NorthBridge
DRAM
Addr/Data/Ctrl
PCI Bus
INIT
NMI
Intr
Cache
SouthBridge (PIIX3)
(PIC, PIT, …)
PCI
Devices
ISA Bus
June 16, 2002
SPC Tutorial
Super-IO
BIOS
RTC
Uarts
Kbd/Mse
Floppy
Parallel
...
ISA
Devices
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
5
Typical Pentium PC: PCI Bus
Addr/Data
Ctrl
CPU/Memory Bus
Ctrl
CPU
NorthBridge
DRAM
Addr/Data/Ctrl
PCI Bus
INIT
NMI
Intr
Cache
SouthBridge (PIIX3)
(PIC, PIT, …)
PCI
Devices
ISA Bus
June 16, 2002
SPC Tutorial
Super-IO
BIOS
RTC
Uarts
Kbd/Mse
Floppy
Parallel
...
ISA
Devices
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
6
Typical Pentium PC: ISA Bus
Addr/Data
Ctrl
CPU/Memory Bus
Ctrl
CPU
NorthBridge
DRAM
Addr/Data/Ctrl
PCI Bus
INIT
NMI
Intr
Cache
SouthBridge (PIIX3)
(PIC, PIT, …)
PCI
Devices
ISA Bus
June 16, 2002
SPC Tutorial
Super-IO
BIOS
RTC
Uarts
Kbd/Mse
Floppy
Parallel
...
ISA
Devices
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
7
How NetBSD Boots on a PC
Components:
–
–
–
–
–
Pentium
Boot ROM (part of BIOS in modern systems?)
BIOS
Bootloader
Kernel
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
8
Sketch of How a PC Boots
• Pentium after Reset:
– fetches its first instruction from location 0xFFFFFFF0
• Boot Code must be located at 0xFFFFFFF0
• Boot Code jumps to BIOS located in ROM
– Boot Code may actually be part of the BIOS...
• BIOS copies itself into memory (Shadow)
• BIOS remaps memory
– future accesses to BIOS addresses go to memory instead of ROM.
• BIOS performs system configuration (some proprietary)
–
–
–
–
Motherboard Details
Pentium Details
NB/SB Chipset Details
Device configuration: IRQs, Memory maps, ...
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
9
How a PC Boots (continued…)
• BIOS loads bootloader into memory (from disk…)
• BIOS jumps to bootloader
• Bootloader performs some more configuration:
– Pentium control registers
– Cache configuration
– Memory/Page model
• Bootloader determines what to run next.
• Bootloader may have to do some device configuration.
– e.g. to get OS from a disk.
• Bootloader loads OS kernel into memory
• Bootloader jumps to start of OS kernel
• Kernel does some OS-specific configuration:
–
–
–
–
for NetBSD look in: sys/arch/i386/i386/locore.s
Determines what CPU it has (“cpuid” instruction)
Paging
Virtual Memory
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
10
Typical Pentium PC (Again…)
Addr/Data
Ctrl
CPU/Memory Bus
Ctrl
CPU
NorthBridge
DRAM
Addr/Data/Ctrl
PCI Bus
INIT
NMI
Intr
Cache
SouthBridge (PIIX3)
(PIC, PIT, …)
PCI
Devices
ISA Bus
June 16, 2002
SPC Tutorial
Super-IO
BIOS
RTC
Uarts
Kbd/Mse
Floppy
Parallel
...
ISA
Devices
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
11
What SPC Needs
Addr/Data
Ctrl
CPU/Memory Bus
Ctrl
CPU
NorthBridge
DRAM
Addr/Data/Ctrl
PCI Bus
INIT
NMI
Intr
Cache
APIC
SouthBridge
(PIC, PIT, …)
RTC
Uarts
BIOS
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
12
What SPC Needs: CPU/Memory Bus
Addr/Data
Ctrl
CPU/Memory Bus
Ctrl
CPU
NorthBridge
DRAM
Addr/Data/Ctrl
PCI Bus
INIT
NMI
Intr
Cache
APIC
SouthBridge
(PIC, PIT, …)
RTC
Uarts
BIOS
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
13
What SPC Needs: PCI Bus
Addr/Data
Ctrl
CPU/Memory Bus
Ctrl
CPU
NorthBridge
DRAM
Addr/Data/Ctrl
PCI Bus
INIT
NMI
Intr
Cache
APIC
SouthBridge
(PIC, PIT, …)
RTC
Uarts
BIOS
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
14
What SPC Needs: ISA Bus?
Addr/Data
Ctrl
CPU/Memory Bus
Ctrl
CPU
NorthBridge
DRAM
Addr/Data/Ctrl
PCI Bus
INIT
NMI
Intr
Cache
APIC
SouthBridge
(PIC, PIT, …)
RTC
Uarts
BIOS
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
15
SPC Architecture
Addr/Data
Ctrl
Ctrl
Cache
NorthBridge
CPU
DRAM
Addr/Data/Ctrl
Intel Embedded Module
INIT
NMI
Intr
PCI Bus
RTC’
UART1
PIC
APIC
PIT
UART1 Interface
BIOS ROM
Link Interface
UART2
UART2 Interface
System FPGA
Switch Interface
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
16
SPC Components
• APIC
• Pentium Embedded Module
– 166 MHz MMX Pentium Processor
• L1 Cache: 16KB Data, 16KB Code
– L2 cache: 512 KB
– NorthBridge - 33 MHz, 32 bit PCI Bus
• System FPGA
– Xilinx XC4020XLA-1 FPGA
• Memory
– EDO DRAM: 64MB (Max for current design)
– SO DIMM
• Switch and Link Interfaces – Each 1 Gb Utopia
• UART supports two Serial Ports:
• NetBSD system console
• TTY port
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
17
System FPGA
• Coded in VHDL
• PCI slave device
• Replaces some of the PIIX3 (south bridge)
• Replaces some of the BIOS
• Replaces some of the Super IO Chip
• Provides reset capability
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
18
System FPGA: PIIX3 Functionality
• Programmable Interrupt Controller (PIC)
– Four Interrupts supported and statically assigned:
•
•
•
•
PIT (IRQ 0)
APIC (IRQ 5)
COM1 (IRQ 4)
COM2 (IRQ 3)
– Static fully-nested interrupt priority structure.
– Specific End of Interrupt is the only EOI mode
supported
• Programmable Interval Timer (PIT)
– generates a clock interrupt for NetBSD every ~10ms
• Reset - covered in a later slide
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
19
System FPGA: BIOS Functionality
• Interrupt functionality replaced by static values
• Simple 16 word by 32-bit “ROM”
– implements loop waiting for location 0xFFE00 to
change value
– then jumps to boot loader code
• Does NOT perform configuration of Northbridge
– This will be done by the boot loader
• Does NOT perform PCI configuration of APIC
– This will be done by the APIC Driver
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
20
System FPGA: Super IO Chip Functionality
• UART Interface
– Two Serial lines supported
– Fixed IRQs
• Real Time Clock
–
–
–
–
only the register accesses of the RTC are supported
no interrupts supported
i.e. supported only so NetBSD didn’t need to change
i.e. no alarms will be generated
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
21
System FPGA: Reset
• SPC needs a reset before every download:
– switch reset:
• causes SPC to be reset
• causes all connections in switch to be lost
– System FPGA reset
• causes SPC to be reset
• has no effect on the switch
• Normal SouthBridge reset:
– I/O Register: 0xCF9
– Hard Reset: assert CPURST, PCIRST#, and RSTDRV
• write 0xCF9 0x02 (00000010b)
• write 0xCF9 0x06 (00000110b)
– Soft Reset: assert INIT
bits
• write 0xCF9 0x00 (00000000b)
• write 0xCF9 0x04 (00000100b)
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
22
System FPGA: Reset
• SPC Reset:
– a sequence of two writes to memory addresses
– APIC Control cells can write to
• memory addresses
• configuration registers
• NOT I/O Registers! Argh...
– To mimic the reset structure of the SB we use:
• 0xFFFFFFF0
• 0xFFFFFFF4
– Hard Reset (all we really care about)
• write 0xFFFFFFF0 0x02 (00000010b)
• write 0xFFFFFFF4 0x06 (00000110b)
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
bits
23
References: SPC-I (on Kits Web Pages)
• Intel Embedded Module:
– Data Sheet
– Design Guide
• 430HX Chipset
– NorthBridge
– SouthBridge
• System FPGA
• Memory
• Mobile Pentium with MMX
– Software Developer Manuals 1,2,3
– Datasheet
• APIC
• Cache
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
24
SPC-II
• Motivation
–
–
–
–
Faster Processor
More Memory
Faster Memory Bus
Simpler
• Real BIOS
• No System FPGA to build a fake Southbridge
– Interchangeable modules
• 700 MHz already available
• Other low-power modules also available
• You can buy new/faster CPU modules if you want…
• More details to be given by Dave Richard on
Tuesday at the Workshop
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
25
SPC-II Architecture
AGP Bus
DRAM
Video
Video DB15
IDE Bus
P-III
CPU
NorthBridge
SouthBridge
FLASH DISK
I/O
Keyboard
Mouse
Serial Ports
BIOS
ETX Embedded Module
ISA Bus
PCI Bus
APIC
Switch Interface
June 16, 2002
FPGA
SPC Tutorial
Link Interface
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
26
SPC-II Architecture
IPP
OPP
SWITCH
BIOS
ISA
Devices
32 bit
ISA Bus
16 bit
Port 1
APIC
Port 0
FPGA
South
Bridge
PCI
BUS
256 MB
SDRAM
Intel BX
North Bridge
16/32 bit 16 bit SPC-II
500/700 MHz
Pentium-III
TI
100 MHz Memory Bus
FPX
Super-IO
256 KB L2 Cache
Link
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
27
SPC-II
• Components
– Intel Module
•
•
•
•
•
•
•
–
–
–
–
–
500 MHz Pentium III
256 KB L2 Cache
Northbridge and Southbridge
33 MHz, 32 bit PCI Bus
100 MHz memory bus
BIOS
Video, Mouse, Keyboard
APIC
FPGA: for ATM level routing
Extra PCI Slot: Debugging possibilities.
256 MB of Memory
32 MB IDE DOM (Disk On Module)
• right now holds binary boot code
• could later hold actual kernel
• larger ones are possible
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
28
Booting an SPC-II
• IDE DOM contains boot code
– modified Stage2 from AAL5_download
• configured to boot in place of a kernel
– No special BIOS tricks to play
• contains APIC code to put kernel into memory
• kernel downloaded via AAL5 frames
– uses Stage3 from AAL5_download
• kernel booted when download complete
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
29
SPC II FPGA Architecture
PCI Bus Port
APIC
Port 0
Port 1
OSC
Reset
VPI[0]=1
VCI = 38
VPI[0]=1
VPI[0]=0
64<=VCI<=127???
LC
June 16, 2002
SPC-II FPGA
SPC Tutorial
FPX
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
Switch
30
SPC-II References and more info
• APIC References the same as before
• Other references to come soon
• Dave Richard will talk more about SPC-II on
Wednesday at the Workshop
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
31
• The rest of the slides are a Gallery of SPC Photos
June 16, 2002
SPC Tutorial
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
32