Transcript NetFPGA Tutorial Slides

NetFPGA Hands-on Training
Day 1
Presented by:
Adam Covington
(Stanford University)
Indiana University, Bloomington
June 18 - 19, 2012
http://NetFPGA.org
Hands-on Training – June 18-19, 2012
1
Tutorial Outline
•
Background
– Introduction
– The NetFPGA Platform
•
The Stanford Base Reference Router
– Motivation: Basic IP review
– Example: Reference Router running on the NetFPGA
•
Infrastructure
– Tree
– Build System
– Scripts
•
The Life of a Packet Through the NetFPGA
– Hardware Datapath
– Interface to software: Exceptions and Host I/O
•
Implementation
– Module Template
– User Data Path
– Write Crypto NIC using a static key
•
Simulation and Debug
– Write and Run Simulations for Crypto NIC
•
Concluding Remarks
Hands-on Training – June 18-19, 2012
2
Section I: Motivation
Hands-on Training – June 18-19, 2012
3
NetFPGA = Networked FPGA
A line-rate, flexible, open networking
platform for teaching and research
Hands-on Training – June 18-19, 2012
4
NetFPGA consists of…
Four elements:
• NetFPGA board
NetFPGA 1G Board
• Tools + reference designs
• Contributed projects
• Community
NetFPGA 10G Board
Hands-on Training – June 18-19, 2012
5
NetFPGA Board Comparison
NetFPGA 1G
NetFPGA 10G
4 x 1Gbps Ethernet Ports
4 x 10Gbps Ethernet Ports
4.5 MB ZBT SRAM
64 MB DDR2 SDRAM
27 MB QDRII-SRAM
288 MB RLDRAM-II
PCI
PCI Express x8
Virtex II-Pro 50
Virtex 5 TX240T
Hands-on Training – June 18-19, 2012
6
NetFPGA board
Networking
Software
running on a
standard PC
CPU
Memory
PCI
A hardware
accelerator
built with a Field
Programmable
Gate Array
driving Gigabit
network links
PC with NetFPGA
1GE
FPGA
1GE
1GE
Memory
Hands-on Training – June 18-19, 2012
1GE
NetFPGA Board
7
Tools + Reference Designs
Tools:
• Compile designs
• Verify designs
• Interact with hardware
Reference designs:
• Router (HW)
• Switch (HW)
• Network Interface Card (HW)
• Router Kit (SW)
• SCONE (SW)
Hands-on Training – June 18-19, 2012
8
Contributed Projects
Project
OpenFlow switch
Contributor
Stanford University
Packet generator
NetFlow Probe
NetThreads
Stanford University
Brno University
University of Toronto
zFilter (Sp)router
Traffic Monitor
DFA
Ericsson
University of Catania
UMass Lowell
More projects:
http://netfpga.org/foswiki/NetFPGA/OneGig/ProjectTable
Hands-on Training – June 18-19, 2012
9
Community
Wiki
• Documentation
– User’s Guide
– Developer’s Guide
• Encourage users to contribute
Forums
• Support by users for users
• Active community - 10s-100s of posts/week
Hands-on Training – June 18-19, 2012
10
International Community
Over 1,000 users, using 2,000 cards at
150 universities in 40 countries
Hands-on Training – June 18-19, 2012
11
NetFPGA’s Defining Characteristics
• Line-Rate
– Processes back-to-back packets
• Without dropping packets
• At full rate of Gigabit Ethernet Links
– Operating on packet headers
• For switching, routing, and firewall rules
– And packet payloads
• For content processing and intrusion prevention
• Open-source Hardware
– Similar to open-source software
• Full source code available
• BSD-Style License
– But harder, because
• Hardware modules must meeting timing
• Verilog & VHDL Components have more complex interfaces
• Hardware designers need high confidence in specification of modules
Hands-on Training – June 18-19, 2012
12
Test-Driven Design
• Regression tests
– Have repeatable results
– Define the supported features
– Provide clear expectation on functionality
• Example: Internet Router
–
–
–
–
–
Drops packets with bad IP checksum
Performs Longest Prefix Matching on destination address
Forwards IPv4 packets of length 64-1500 bytes
Generates ICMP message for packets with TTL <= 1
Defines how packets with IP options or non IPv4
… and dozens more …
Every feature is defined by a regression test
Hands-on Training – June 18-19, 2012
13
Who, How, Why
Who uses the NetFPGA?
–
–
–
Teachers
Students
Researchers
How do they use the NetFPGA?
–
–
To run the Router Kit
To build modular reference designs
•
•
•
IPv4 router
4-port NIC
Ethernet switch, …
Why do they use the NetFPGA?
–
–
To measure performance of Internet systems
To prototype new networking systems
Hands-on Training – June 18-19, 2012
14
Summer Camp Objectives
• Overall picture of NetFPGA
• How reference designs work
• How you can work on a project
– NetFPGA Design Flow
– Directory Structure, library modules and projects
– How to utilize contributed projects
• Interface/Registers
– How to verify a design (Simulation and Regression
Tests)
– Things to do when you get stuck
AND… You can build your own projects!
Hands-on Training – June 18-19, 2012
15
Section II: Network review
Hands-on Training – June 18-19, 2012 16
16
STAN FO R D U N IVE R S ITY
Internet Protocol (IP)
Data to be
transmitted:
IP packets:
Ethernet
Frames:
Data
…
IP
Hdr
Data
IP
Hdr
Data
Eth IP
Hdr Hdr
Data
Eth IP
Hdr Hdr
Data
…
17
STAN FO R D U N IVE R S ITY
Hands-on Training – June 18-19, 2012 17
IP
Hdr
Data
Eth IP
Hdr Hdr
Data
Internet Protocol (IP)
Data
…
1
4
Ver
16
HLen
T.Service
20 bytes
Fragment ID
TTL
Protocol
IP
Hdr
Data
32
Total Packet Length
Flags
Fragment Offset
Header Checksum
Source Address
Destination Address
Options (if any)
Hands-on Training – June 18-19, 2012 18
18
STAN FO R D U N IVE R S ITY
Basic operation of an IP router
R3
A
R1
R4
B
C
D
E
R2
Destination
Next Hop
D
R3
E
R3
F
R5
Hands-on Training – June 18-19, 2012 19
19
R5
F
STAN FO R D U N IVE R S ITY
Basic operation of an IP router
R3
A
R1
R4
B
C
D
E
R2
Hands-on Training – June 18-19, 2012 20
R5
20
F
STAN FO R D U N IVE R S ITY
Forwarding tables
IP address
32 bits wide → ~ 4 billion unique address
Naïve approach:
One entry per address
Entry
Destination
Port
1
2
⋮
232
0.0.0.0
0.0.0.1
⋮
255.255.255.255
1
2
⋮
12
~ 4 billion entries
Improved approach:
Group entries to reduce table size
Entry
Destination
Port
1
2
⋮
50
0.0.0.0 – 127.255.255.255
128.0.0.1 – 128.255.255.255
⋮
248.0.0.0 – 255.255.255.255
1
2
⋮
12
Hands-on Training – June 18-19, 2012 21
21
STAN FO R D U N IVE R S ITY
IP addresses as a line
Your computer
My computer
Stanford
Berkeley
North America
Asia
232-1
0
All IP addresses
Entry
Destination
Port
1
2
3
4
5
Stanford
Berkeley
North America
Asia
Everywhere (default)
1
2
3
4
5
Hands-on Training – June 18-19, 2012 22
22
STAN FO R D U N IVE R S ITY
Longest Prefix Match (LPM)
Entry
Destination
Port
1
2
3
4
5
Stanford
Berkeley
North America
Asia
Everywhere (default)
1
2
3
4
5
Matching entries:
• Stanford
• North America
• Everywhere
To:
Stanford
Universities
Continents
Planet
Most specific
Data
Hands-on Training – June 18-19, 2012 23
23
STAN FO R D U N IVE R S ITY
Longest Prefix Match (LPM)
Entry
Destination
Port
1
2
3
4
5
Stanford
Berkeley
North America
Asia
Everywhere (default)
1
2
3
4
5
Matching entries:
• North America
• Everywhere
To:
Canada
Universities
Continents
Planet
Most specific
Data
Hands-on Training – June 18-19, 2012 24
24
STAN FO R D U N IVE R S ITY
Implementing Longest Prefix Match
Entry
Destination
Port
1
2
3
4
5
Stanford
Berkeley
North America
Asia
Everywhere (default)
1
2
3
4
5
Hands-on Training – June 18-19, 2012 25
25
Searching
Most specific
FOUND
Least specific
STAN FO R D U N IVE R S ITY
Basic components of an IP router
Software
Management
& CLI
Routing
Protocols
Routing
Table
Hardware
Forwarding
Switching Queuing
Table
Hands-on Training – June 18-19, 2012 26
26
Control Plane
Data Plane
per-packet
processing
STAN FO R D U N IVE R S ITY
IP router components in NetFPGA
Linux
SCONE
Management
& CLI
Routing
Protocols
Routing
Table
OR
Routing
Protocols
Routing
Table
Software
Management
& CLI
Router Kit
Forwarding
Table
Input
Arbiter
Switching
Hands-on Training – June 18-19, 2012 27
Output
Queues
Queuing
27
Hardware
Output Port
Lookup
STAN FO R D U N IVE R S ITY
Section III: Example
Hands-on Training – June 18-19, 2012 28
28
Operational IPv4 router
Java GUI
Software
SCONE
Management
& CLI
Routing
Protocols
Routing
Table
Hardware
Reference router
Forwarding
Switching Queuing
Table
Hands-on Training – June 18-19, 2012 29
29
Control Plane
Data Plane
per-packet
processing
Streaming video
Hands-on Training – June 18-19, 2012 30
30
Streaming video
PC & NetFPGA
(NetFPGA in PC)
NetFPGA running
reference router
Hands-on Training – June 18-19, 2012 31
31
Streaming video
Video streaming over shortest path
Video
client
Video
server
Hands-on Training – June 18-19, 2012 32
32
Streaming video
Link breaks
Video
client
Video
server
Hands-on Training – June 18-19, 2012 33
33
Streaming video
.1.1
.1.2
.4.1
.7.1
.10.1
.13.1
.4.2
.7.2
.10.2
.13.2
.3.1
.30.2
.6.2
.3.2
.9.2
.6.1
.12.2
.9.1
.30.1
.12.1
.23.1
.27.2
.28.2
.28.1
.24.2
.27.1
.25.2
.21.2
.24.1
.25.1
Hands-on Training – June 18-19, 2012 34
.22.2
.22.1
34
.21.1
.19.2
.19.1
.16.2
.15.2
.15.1
.18.2
.16.1
.18.1
Observing the routing tables
Columns:
• Subnet address
• Subnet mask
• Next hop IP
• Output ports
Hands-on Training – June 18-19, 2012 35
35
Demo
Hands-on Training – June 18-19, 2012 36
36
Review
NetFPGA as IPv4 router:
•Reference hardware + SCONE software
•Routing protocol discovers topology
Demo:
•Ring topology
•Traffic flows over shortest path
•Broken link: automatically route around
failure
Hands-on Training – June 18-19, 2012 37
37
Section IV: Infrastructure
Hands-on Training – June 18-19, 2012 38
38
Infrastructure
• Tree structure
• NetFPGA package contents
– Reusable Verilog modules
– Verification infrastructure
– Build infrastructure
– Utilities
– Software libraries
Hands-on Training – June 18-19, 2012 39
39
Tree Structure (1)
netfpga
bin
(scripts for running simulations and setting up the
environment)
bitfiles
lib
(contains the bitfiles for all projects that have been
synthesized)
(shared Verilog modules, libraries needed for
simulation/synthesis/design)
projects
(user projects, including reference designs)
Hands-on Training – June 18-19, 2012 40
40
Tree Structure (2)
lib
C (common software and code for reference designs)
java (contains software for the graphical user interface)
Makefiles (makefiles for simulation and synthesis)
(libraries to interact with reference designs, create
Perl5 test data, and manage simulations/regression tests)
python (common libraries to aid in regression tests)
(utility scripts – less commonly used than those in
the bin directory)
scripts
verilog (modules that can be reused in designs)
Hands-on Training – June 18-19, 2012 41
41
Tree Structure (3)
projects/crypto_nic
doc (project specific documentation)
(XML files defining project and any local modules,
include auto-generated Verilog register defines)
lib (C/Perl defines for registers)
Verilog code used for synthesis and
src (non-library
simulation)
sw (software elements of the project)
.xco files to generate cores,
synth (project-specific
Makefile to implement the design)
test (simulation and hardware tests)
Hands-on Training – June 18-19, 2012 42
42
NetFPGA package contents
• Projects:
– HW: router, switch, NIC, buffer sizing router
– SW: router kit, SCONE
• Reusable Verilog modules
• Verification infrastructure:
– simulate full board with PCI + physical interfaces
– run tests against hardware
– test data generation libraries (eg. packets)
• Build infrastructure
• Utilities:
– register I/O, packaging, …
• Software libraries
Hands-on Training – June 18-19, 2012 43
43
Reusable Verilog modules
Category
Modules
I/O interfaces
Ethernet MAC
CPU DMA queues
CPU register queues
MDIO
PCI
Output queues
SRAM-based
DRAM-based
BRAM-based
Output port lookup
Router (CAM-based)
Learning switch (CAM-based)
NIC
Hardwired
Memory interfaces
SRAM
DRAM
Miscellaneous
FIFOs
Generic register module
Rate limiter
Hands-on Training – June 18-19, 2012 44
44
Verification Infrastructure
• Simulation: nf_test.py sim
– allows testing before synthesis
– catches many bugs
• Hardware tests: nf_test.py hw
– test synthesized hardware
• Test data generation libraries:
– easily create test data:
– many standard packet formats supported out of
the box
– easily add support for custom formats
Hands-on Training – June 18-19, 2012 45
45
Build infrastructure
• Register system:
– allocates memory to modules
– generates “include” files for various languages
• Build/synthesis:
– required shared modules documented XML
(shared with register system)
– shared modules pulled in during synthesis
– resultant bitfile checked for timing errors
Hands-on Training – June 18-19, 2012 46
46
Utilities
•
•
•
•
Bitfile download: nf_download
Register I/O: regread, regwrite
Device querying: nf_info
SRAM dumping: lib/scripts/sram_dump
Hands-on Training – June 18-19, 2012 47
47
Software libraries
• Libraries for interfacing with NetFPGA:
– C, Perl, Java, Python support
Hands-on Training – June 18-19, 2012 48
48
Section V: Life of a Packet
Hands-on Training – June 18-19, 2012
49
Reference Router Pipeline
• Five stages
MAC
RxQ
– Input
– Input arbitration
– Routing decision and
packet modification
– Output queuing
– Output
• Packet-based
module interface
• Pluggable design
MAC
TxQ
Hands-on Training – June 18-19, 2012
CPU
RxQ
MAC
RxQ
CPU
RxQ
MAC
RxQ
CPU
RxQ
MAC
RxQ
CPU
RxQ
Input Arbiter
Output Port Lookup
Output Queues
CPU
TxQ
MAC
TxQ
50
CPU
TxQ
MAC
TxQ
CPU
TxQ
MAC
TxQ
CPU
TxQ
Full System Components
Software
nf2c0
nf2c1
nf2c2
nf2c3
ioctl
PCI Bus
CPU CPU
CPU
CPU
CPU
CPU
RxQ
TxQ
CPURxQ
CPU TxQ
RxQ
TxQ
RxQ TxQ
NetFPGA
user data path
MAC MAC
MAC
MAC
MAC
MAC
TxQMAC
RxQ
MAC
TxQ
RxQ
TxQ
RxQ
TxQ RxQ
Ethernet
Hands-on Training – June 18-19, 2012
51
nf2_reg_grp
Life of a Packet through the Hardware
192.168.1.x
port0
Hands-on Training – June 18-19, 2012
port2
52
192.168.2.y
Inter-Module Communication
Using “Module Headers”:
Ctrl Word
(8 bits)
Data Word
(64 bits)
x
Module Hdr
…
…
y
Last Module Hdr
0
Eth Hdr
0
IP Hdr
0
0x10
…
Last word of packet
Hands-on Training – June 18-19, 2012
Contain information
such as packet
length, input port,
output port, …
53
Inter-Module Communication
data
Module
i
Module
i+1
ctrl
wr
rdy
Hands-on Training – June 18-19, 2012
54
MAC Rx Queue
MAC Rx
Queue
Hands-on Training – June 18-19, 2012
55
Rx Queue
0xff
0
0
Pkt length,
input port = 0
Eth Hdr:
Dst MAC = port 0,
Ethertype = IP
IP Hdr:
IP Dst: 192.168.2.3,
TTL: 64, Csum:0x3ab4
Rx
Queue
0
Hands-on Training – June 18-19, 2012
Data
56
Input Arbiter
Rx
Q7
Pkt
…
Rx
Q1
Input
Arbiter
Pkt
Rx
Q0
Pkt
Hands-on Training – June 18-19, 2012
57
Output Port Lookup
Output
Port
Lookup
Hands-on Training – June 18-19, 2012
58
Output Port Lookup
5- Add output
port header
1- Check input
port matches
Dst MAC
2- Check TTL,
checksum
0xff
3- Lookup
next hop IP &
output port
(LPM)
0
4- Lookup
next hop MAC
address (ARP)
0
0
Pkt length,
input port = 0
output port = 4
Output
Port
Lookup
EthHdr:
MAC
=0
EthHdr:
Dst Dst
MAC
= nextHop
Src
MAC
= x, 4,
Src
MAC
= port
Ethertype = IP
IP Hdr:
IP Dst: 192.168.2.3,
TTL: 64,
63, Csum:0x3ab4
Csum:0x3ac2
Hands-on Training – June 18-19, 2012
Data
59
6- Modify MAC
Dst and Src
addresses
7-Decrement
TTL and
update
checksum
Output Queues
OQ0
Output
Queues
OQ4
OQ7
Hands-on Training – June 18-19, 2012
60
MAC Tx Queue
MAC Tx
Queue
Hands-on Training – June 18-19, 2012
61
MAC Tx Queue
Pkt length,
0xff
input port = 0
output port = 4
EthHdr: Dst MAC = nextHop
0
Src MAC = port 4,
Ethertype = IP
IP Hdr:
0
IP Dst: 192.168.2.3,
TTL: 64,
63, Csum:0x3ab4
Csum:0x3ac2
MAC Tx
Queue
0
Hands-on Training – June 18-19, 2012
Data
62
Exception Packets
• Example: TTL = 0 or TTL = 1
• Packet has to be sent to the CPU
• Host generates an ICMP packet response
• Difference starts at the Output Port Lookup
Hands-on Training – June 18-19, 2012
63
Exception Packet Path
Software
nf2c0
nf2c1
nf2c2
nf2c3
CPU CPU
RxQ TxQ
CPU CPU
RxQ TxQ
ioctl
PCI Bus
CPU CPU
RxQ TxQ
CPU CPU
RxQ TxQ
NetFPGA
nf2_reg_grp
user data path
MAC MAC
TxQ RxQ
MAC MAC
TxQ RxQ
MAC MAC
TxQ RxQ
Ethernet
Hands-on Training – June 18-19, 2012
64
MAC MAC
TxQ RxQ
Output Port Lookup
1- Check input
port matches
Dst MAC
2- Check TTL,
checksum –
EXCEPTION!
0xff
0
3- Add output
port module
0
Pkt length,
input port = 0
output port = 1
EthHdr: Dst MAC = 0,
Src MAC = x,
Ethertype = IP
IP Hdr:
IP Dst: 192.168.2.3,
TTL: 1, Csum:0x3ab4
Output
Port
Lookup
0
Hands-on Training – June 18-19, 2012
Data
65
Output Queues
OQ0
OQ1
Output
Queues
OQ2
OQ7
Hands-on Training – June 18-19, 2012
66
CPU Tx Queue
CPU Tx
Queue
Hands-on Training – June 18-19, 2012
67
CPU Tx Queue
0xff
0
0
Pkt length,
input port = 0
output port = 1
EthHdr: Dst MAC = 0,
Src MAC = x,
Ethertype = IP
IP Hdr:
IP Dst: 192.168.2.3,
TTL: 1, Csum:0x3ab4
CPU Tx
Queue
0
Hands-on Training – June 18-19, 2012
Data
68
ICMP Packet
• Packet arrives at the CPU Rx Queue from
the PCI Bus
• Same path as a packet from the MAC until
it reaches the Output Port Lookup (OPL)
• The OPL module sees the packet is from
the CPU Rx Queue 1 and sets the output
port directly to 0
• The packet continues on the same path as
the non-exception packet to the Output
Queues and then MAC Tx queue 0
Hands-on Training – June 18-19, 2012
69
ICMP Packet Path
Software
nf2c0
nf2c1
nf2c2
nf2c3
CPU CPU
RxQ TxQ
CPU CPU
RxQ TxQ
ioctl
PCI Bus
CPU CPU
RxQ TxQ
CPU CPU
RxQ TxQ
NetFPGA
nf2_reg_grp
user data path
MAC MAC
TxQ RxQ
MAC MAC
TxQ RxQ
MAC MAC
TxQ RxQ
Ethernet
Hands-on Training – June 18-19, 2012
70
MAC MAC
TxQ RxQ
NetFPGA-Host Interaction
• Linux driver interfaces with hardware
– Packet interface via standard Linux network
stack
– Register reads/writes via ioctl system call
with wrapper functions:
• readReg(nf2device *dev, int address, unsigned *rd_data);
• writeReg(nf2device *dev, int address, unsigned *wr_data);
eg:
readReg(&nf2, OQ_NUM_PKTS_STORED_0, &val);
Hands-on Training – June 18-19, 2012
71
NetFPGA-Host Interaction
NetFPGA to host packet transfer
1. Packet arrives –
forwarding table
sends to CPU queue
Hands-on Training – June 18-19, 2012
PCI Bus
2. Interrupt
notifies
driver of
packet
arrival
3. Driver sets up
and initiates
DMA transfer
72
NetFPGA-Host Interaction
NetFPGA to host packet transfer (cont.)
PCI Bus
4. NetFPGA
transfers
packet via
DMA
5. Interrupt
signals
completion
of DMA
6. Driver passes packet to
network stack
Hands-on Training – June 18-19, 2012
73
NetFPGA-Host Interaction
Host to NetFPGA packet transfers
PCI Bus
2. Driver sets up
and initiates
DMA transfer
3. Interrupt
signals
completion
of DMA
1. Software sends packet
via network sockets
Packet delivered to driver
Hands-on Training – June 18-19, 2012
74
NetFPGA-Host Interaction
Register access
PCI Bus
2. Driver
performs
PCI memory
read/write
1. Software makes ioctl
call on network socket
ioctl passed to driver
Hands-on Training – June 18-19, 2012
75
NetFPGA-Host Interaction
• Packet transfers shown using DMA
interface
• Alternative: use programmed IO to transfer
packets via register reads/writes
– slower but eliminates the need to deal with
network sockets
Hands-on Training – June 18-19, 2012
76
Section VI: Example Project
Hands-on Training – June 18-19, 2012
77
Project: Cryptographic NIC
Implement a network interface card (NIC)
that encrypts upon transmission and
decrypts upon reception
Hands-on Training – June 18-19, 2012
78
Cryptography
XOR function
A
B
A^B
0
0
0
0
1
1
1
0
1
1
1
0
XORing a
value with
itself always
yields 0
XOR written as: ^ ⊻ ⨁
XOR is commutative: (A ^ B) ^ C = A ^ (B ^ C)
Hands-on Training – June 18-19, 2012
79
Cryptography (cont.)
Simple cryptography:
– Generate a secret key
– Encrypt the message by XORing the message and key
– Decrypt the ciphertext by XORing with the key
Explanation:
(M ^ K) ^ K = M ^ (K ^ K)
= M^0
= M
Hands-on Training – June 18-19, 2012
80
Commutativity
A^A=0
Cryptography (cont.)
Example:
Message: 00111011
Key: 10110001
Message ^ Key: 10001010
Key: 10110001
Message ^ Key ^ Key: 00111011
Hands-on Training – June 18-19, 2012
81
Cryptography (cont.)
Idea: Implement simple cryptography using XOR
– 32-bit key
– Encrypt every word in payload with key
Header
Payload
⨁
Key
Key
Key
Key
Key
Note: XORing with a one-time pad of the same length of the message is
secure/uncrackable. See: http://en.wikipedia.org/wiki/One-time_pad
Hands-on Training – June 18-19, 2012
82
Section VII: Implementation
Hands-on Training – June 18-19, 2012
83
Getting started with a new project (1)
• Projects:
– Each design represented by a project
– Location: netfpga/projects/<proj_name>
• netfpga/projects/crypto_nic
– Consists of:
•
•
•
•
•
Verilog source
Simulation tests
Hardware tests
Libraries
Optional software
Hands-on Training – June 18-19, 2012
84
Getting started with a new project (2)
– Normally:
• copy an existing project as the starting point
– Today:
• pre-created project
– Missing from pre-created project:
•
•
•
•
Verilog files (with crypto implementation)
Simulation tests
Hardware tests
Custom software
Hands-on Training – June 18-19, 2012
85
Getting started with a new project (3)
Typically implement
functionality in one or
more modules inside
the user data path
MAC
RxQ
CPU
RxQ
MAC
RxQ
CPU
RxQ
MAC
RxQ
CPU
RxQ
MAC
RxQ
CPU
RxQ
CPU
TxQ
MAC
TxQ
CPU
TxQ
Input Arbiter
Output Port Lookup
User data
path
Crypto
Output Queues
Crypto module
to encrypt and
decrypt packets
Hands-on Training – June 18-19, 2012
MAC
TxQ
CPU
TxQ
86
MAC
TxQ
CPU
TxQ
MAC
TxQ
Getting started with a new project (4)
– Shared modules included from netfpga/lib/verilog
• Generic modules that are re-used in multiple projects
• Specify shared modules in project’s include/project.xml
– Local src modules override shared modules
– crypto_nic:
Local
Shared
user_data_path.v
crypto.v
Everything else
Hands-on Training – June 18-19, 2012
87
Exploring project.xml (1)
• Location: project/<proj_name>/include
<?xml version="1.0" encoding="UTF-8"?>
<nf:project …>
<nf:name>Crypto NIC</nf:name>
Short name
Description
<nf:description>NIC with basic crypto support</nf:description>
<nf:version_major>0</nf:version_major>
<nf:version_minor>1</nf:version_minor>
<nf:version_revision>0</nf:version_revision>
Version information
• indicate bitfile version
<nf:dev_id>0</nf:dev_id>
Unique ID to identify project
See: http://netfpga.org/foswiki/bin/view/NetFPGA/OneGig/DeviceIDList
Hands-on Training – June 18-19, 2012
88
Exploring project.xml (2)
<nf:use_modules>
core/io_queues/cpu_dma_queue
core/io_queues/ethernet_mac
core/input_arbiter/rr_input_arbiter
Shared modules to
core/nf2/generic_top
load from lib/verilog
core/nf2/reference_core
core/output_port_lookup/nic
core/output_queues/sram_rr_output_queues
core/sram_arbiter/sram_weighted_rr
core/user_data_path/reference_user_data_path
core/io/mdio
core/cpci_bus
core/dma
core/user_data_path/udp_reg_master
core/io_queues/add_rm_hdr
core/strip_headers/keep_length
core/utils/generic_regs
core/utils
</nf:use_modules>
Hands-on Training – June 18-19, 2012
89
Exploring project.xml (3)
<nf:memalloc layout="reference">
<nf:group name="core1">
<nf:instance name="device_id" />
<nf:instance name="dma" base="0x0500000"/>
<nf:instance name="mdio" />
<nf:instance name="nf2_mac_grp" count="4" />
<nf:instance name="cpu_dma_queue" count="4" />
</nf:group>
<nf:group name="udp">
Specify where to
<nf:instance name="in_arb" />
instantiate modules, the
<nf:instance name="crypto" />
<nf:instance name="strip_headers" /> number of instances,
<nf:instance name="output_queues" /> and the memory
</nf:group>
addresses to use
</nf:memalloc>
</nf:project>
Hands-on Training – June 18-19, 2012
90
Getting started with a new project (5)
Tasks:
Set the project that we’ll be working with:
1.
Add the following lines to the end of ~/.bashrc:
export NF_DESIGN_DIR=$NF_ROOT/projects/crypto_nic
export
PERL5LIB=$NF_ROOT/lib/Perl5:$NF_DESIGN_DIR/lib/Per
l5
2.
Type: source ~/.bashrc
Copy reference files as starting points:
3.
Copy the following files from netfpga/lib/verilog/core into
netfpga/projects/crpyto_nic/src
user_data_path/reference_user_data_path/src/user_data_path.v
module_template/src/module_template.v
Hands-on Training – June 18-19, 2012
91
Getting started with a new project (6)
Create crypto.v from module_template.v:
1.
2.
Rename the local module_template.v to crypto.v
Change the module name inside crypto.v (first noncomment line of the file)
3.
Add the crypto module to the user data path
Hands-on Training – June 18-19, 2012
92
user_data_path.v (1)
module user_data_path
#(
parameter DATA_WIDTH = 64,
...
)
(
...
)
Module port declaration
//------------------ Internal parameters ----------------------...
//----------------- Input arbiter wires/regs ------------------...
Hands-on Training – June 18-19, 2012
93
user_data_path.v (2)
//-------------- output port lut wires/regs -------------------wire [CTRL_WIDTH-1:0]
op_lut_in_ctrl;
wire [DATA_WIDTH-1:0]
op_lut_in_data;
wire
op_lut_in_wr;
wire
op_lut_in_rdy;
...
Wire declarations for the
output port lookup module.
Duplicate this section, and
replace op_lut with crypto
//------- output queues wires/regs -----...
Hands-on Training – June 18-19, 2012
94
user_data_path.v (3)
//--------- Connect the data path ----------input_arbiter #(
...
Module instantiations.
) input_arbiter (
...
)
1.Duplicate the output_port_lookup
output_port_lookup #(
...
) output_port_lookup (
...
)
...
instantiation
2.Rename to crypto
3.Remove all parameters (inside the
first set or parentheses)
4.In the output_port_lookup
instantiation, replace oq_ with
crypto_
5.In the crypto instantiation, replace
op_lut_ with crypto_
We’ve inserted the new module into
the pipeline
Hands-on Training – June 18-19, 2012
95
Getting started with a new project (7)
Run a simulation to verify changes:
1.
nf_test.py sim --major nic --minor short
Now we can implement the crypto functionality
Hands-on Training – June 18-19, 2012
96
Implementing the Crypto Module (1)
• What do we want to encrypt?
– IP payload only
• Plaintext IP header allows routing
• Content is hidden
– Encrypt bytes 35 onward
• Bytes 1-14 – Ethernet header
• Bytes 15-34 – IPv4 header (assume no options)
– Assume all packets are IPv4 for simplicity
Hands-on Training – June 18-19, 2012
97
Implementing the Crypto Module (2)
• State machine (draw on next page):
– Module headers on each packet
– Datapath 64-bits wide
• 34 / 8 is not an integer! 
• Inside the crypto module
Hands-on Training – June 18-19, 2012
98
Crypto Module State Diagram
Hint: We suggest 4 states (or 3 if you’re feeling adventurous)
Skip
Module
Headers
Hands-on Training – June 18-19, 2012
99
Implementing the Crypto Module (3)
Implement your state machine inside crypto.v
–
Use a static key initially
Suggested sequence of steps:
1.
Create a static key value
•
2.
Constants can be declared in the module with localparam:
localparam MY_EXAMPLE = 32’h01234567;
Implement your state machine without modifying the
packet
Update your state machine to modify the packet by
XORing the key and the payload
3.
•
Use two copies of the key to create a 64-bit value to XOR
with data words
Hands-on Training – June 18-19, 2012
100
module_template.v (1)
module module_template
#(
Module
parameter DATA_WIDTH = 64,
parameter CTRL_WIDTH = DATA_WIDTH/8,
parameter UDP_REG_SRC_WIDTH = 2
)
(
...
)
port declaration
//----------------------- Signals---------------------------...
//------------------ Local assignments ----------------------...
Hands-on Training – June 18-19, 2012
101
module_template.v (2)
//------------------------- Modules-------------------------------
Packet data dumped in
a FIFO. Allows some
“decoupling” between
input and output.
fallthrough_small_fifo #(
.WIDTH(CTRL_WIDTH+DATA_WIDTH),
.MAX_DEPTH_BITS(2)
) input_fifo (
.din
({in_ctrl, in_data}),
// Data in
.wr_en
(in_wr),
// Write enable
.rd_en
(in_fifo_rd_en),
// Read the next word
.dout
({in_fifo_ctrl, in_fifo_data}),
.full
(),
.nearly_full
(in_fifo_nearly_full),
.prog_full
(),
.empty
(in_fifo_empty),
.reset
(reset),
.clk
(clk)
);
Hands-on Training – June 18-19, 2012
102
module_template.v (3)
generic_regs
#(
.UDP_REG_SRC_WIDTH
.TAG
.REG_ADDR_WIDTH
.NUM_COUNTERS
.NUM_SOFTWARE_REGS
.NUM_HARDWARE_REGS
) module_regs (
...
);
(UDP_REG_SRC_WIDTH),
(0),
(1),
Generic registers.
(0),
(0),
Ignore for now – we’ll
(0)
Hands-on Training – June 18-19, 2012
explore this later
103
module_template.v (4)
//------------------------- Logic-------------------------------
Combinational logic to
read data from the FIFO.
(Data is output to
output ports.)
always @(*) begin
// Default values
out_wr_int = 0;
in_fifo_rd_en = 0;
if (!in_fifo_empty && out_rdy) begin
out_wr_int = 1;
in_fifo_rd_en = 1;
end
end
Hands-on Training – June 18-19, 2012
104
You’ll want to add your
state in this section.
Inter-module Communication
data
`
Module
i
ctrl
wr
Module
i+1
rdy
Hands-on Training – June 18-19, 2012
105
Implementing the Crypto Module (3)
Implement your state machine inside crypto.v
–
Use a static key initially
Suggested sequence of steps:
1.
Create a static key value
•
2.
Constants can be declared in the module with localparam:
localparam MY_EXAMPLE = 32’h01234567;
Implement your state machine without modifying the
packet
Update your state machine to modify the packet by
XORing the key and the payload
3.
•
Use two copies of the key to create a 64-bit value to XOR
with data words
Hands-on Training – June 18-19, 2012
106
Section VIII: Simulation and Debug
Hands-on Training – June 18-19, 2012
107
Testing: Simulation (1)
• Simulation allows testing without requiring
lengthy synthesis process
• NetFPGA simulation environment allows:
– Send/receive packets
• Physical ports and CPU
– Read/write registers
– Verify results
• Simulations run in ModelSim/VCS/ISim
Hands-on Training – June 18-19, 2012
108
Testing: Simulation (2)
• Simulations located in project/test
• Multiple simulations per project
– Test different features
• Example:
– crypto_nic/test/both_nic_short
• Send one packet from CPU, expect packet out
physical port
• Send one packet in physical port, expect packet to
CPU
Note: This test will not work once your crypto
module is implemented!
Hands-on Training – June 18-19, 2012
109
Testing: Simulation (3)
Useful functions:
Register access:
nftest_regwrite(addr, value)
nftest_regread_expect(addr, expect)
Packet generation:
make_IP_pkt(…) – see documentation
encrypt_pkt(key, pkt)
decrypt_pkt(key, pkt)
Packet transmission/reception:
nftest_send_phy(interface, pkt)
nftest_expect_phy(interface, pkt)
nftest_send_dma(interface, pkt)
nftest_expect_dma(interface, pkt)
Hands-on Training – June 18-19, 2012
110
Testing: Simulation (4)
Task:
Implement tests for encryption and decryption
Modify the following tests:
netfpga/projects/crypto_nic/test/both_crypto_encrypt/run.py
netfpga/projects/crypto_nic/test/both_crypto_decrypt/run.py
Look at both_nic_short as an example of creating IP packets and
sending/receiving them
Hands-on Training – June 18-19, 2012
111
Running Simulations
• Use command nf_test.py
– Required Parameter
• sim or hw (right now only use sim)
– Optional parameters
• --major <major_name>
• --minor <minor_name>
• --gui (starts the default viewing environment)
both_crypto_encrypt
major
minor
Hands-on Training – June 18-19, 2012
112
Running Simulations
Non-GUI execution example:
# 10756.00ns testbench.host32.service_interrupt: Info: Interrupt signaled
#
10935
Host read 0x00000044 with cmd 0x6: Disconnect with Data,
#
10995
CPCI Interrupt: DMA ingress xfer complete
#
11175
Host read 0x00000148 with cmd 0x6: Disconnect with Data,
#
11415
Host read 0x00000150 with cmd 0x6: Disconnect with Data,
# 11475.00ns testbench.host32.service_interrupt: Info: DMA ingress transfer complete.
#
11655
Host read 0x00000040 with cmd 0x6: Disconnect with Data,
# Timecheck: 13645.00ns
#
20100 Simulation has reached finish time - ending.
# ** Note: $finish
: /home/summercamp/netfpga/lib/verilog/core/testbench/target32.v
#
Time: 20100 ns Iteration: 0 Instance: /testbench/target32
--- Simulation is complete. Validating the output.
Comparing simulation output for port 1 ...
Port 1 matches [1 packets]
Comparing simulation output for port 2 ...
Port 2 matches [0 packets]
--- Test PASSED (test_nic_short)
Test test_nic_short passed!
------------SUMMARY--------------PASSING TESTS:
test_nic_short
FAILING TESTS:
TOTAL: 1 PASS: 1 FAIL: 0
Hands-on Training – June 18-19, 2012
113
Running Simulations
GUI execution example:
Waveforms
Modules
Transcript
/command entry
Signals in
selected module
Hands-on Training – June 18-19, 2012
114
Running Simulations
GUI execution example (cont)
Try the following:
nf_test.py sim --major crypto --minor encrypt –gui
In the transcript window of the GUI:
do wave.do
run 10us
You should see waveforms of packets going in and
coming out of the crypto module
Hands-on Training – June 18-19, 2012
115
Running Simulations
• When running ModelSim interactively:
– Click "no" when simulator prompts to finish
– Changes to code can be recompiled without
quitting ModelSim:
• bash# cd /tmp/$(whoami)/test/<projname>;
make model_sim
• VSIM 5> restart -f; run -a
– Ensure $NF_DESIGN_DIR is correct
Hands-on Training – June 18-19, 2012
116
Section IX: Conclusion
Hands-on Training – June 18-19, 2012
117
Acknowledgments
NetFPGA Team at University of Cambridge (Past and Present):
Andrew Moore, David Miller, Martin Zadnik, Muhammad Shahbaz
NetFPGA Team at Stanford University (Past and Present):
Nick McKeown, Glen Gibb, Jad Naous, David Erickson,
G. Adam Covington, John W. Lockwood, Jianying Luo, Brandon Heller,
Paul Hartke, Neda Beheshti, Sara Bolouki, James Zeng,
Jonathan Ellithorpe, Sachidanandan Sambandan, Eric Lo,
Sam D’Amico
All Community members (including but not limited to):
Paul Rodman, Kumar Sanghvi, Wojciech A. Koszek,
Yahsar Ganjali, Martin Labrecque, Jeff Shafer,
Eric Keller , Tatsuya Yabe, Bilal Anwer,
Yashar Ganjali, Martin Labrecque
Ram Subramanian, Kees Vissers, Michaela Blott, Shep Siegel
Hands-on Training – June 18-19, 2012
118
Special thanks to our Partners:
Ram Subramanian, Patrick Lysaght, Veena Kumar, Paul Hartke,
Anna Acevedo
Xilinx University Program (XUP)
Other NetFPGA Tutorials Presented At:
SIGMETRICS
See: http://NetFPGA.org/tutorials/
Hands-on Training – June 18-19, 2012
119
Thanks to our Sponsors:
• Support for the NetFPGA project has been provided
by the following companies and institutions
Disclaimer: Any opinions, findings, conclusions, or recommendations expressed in these
materials do not necessarily reflect the views of the National Science Foundation or of
any other sponsors supporting this project.
Hands-on Training – June 18-19, 2012
120