NetBSD Kernel Topics:

IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous

January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 1

• IP – Understand the structure of IP processing in NetBSD – Understand the IP packet format – Become familiar with IP packet field access » accessing src and dst addresses, etc… • mbufs – Understand the mbuf structure – Become familiar with mbuf access routines » packet size » data location • Stevens TCP/IP books – impress upon you the importance of these ref. books January 9, 2001

Goals Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 2

What we are NOT going to talk about

• IPv6 – Everything we talk about will be in terms of IPv4 – Router Plugins also supports IPv6 – The code can be confusing » use IPv6 structures since they are superset of IPv4 • IP Options • Fast Forward Path • Details of interaction between device drivers and IP – actually, we’ll talk about some of this… • Packet Scheduler • Routing protocols • • daemons: routed, rsvpd, … January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 3

NetBSD Kernel Topics:

IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous

January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 4

Vocabulary: Terms Used in Stevens

• message: Transport protocol data given to IP • datagram: message + IP header • fragment: if datagram is too large for network: IP splits it. Each fragment contains its own IP header • packet: fragment or datagram small enough for network • frame: packet + data-link layer header January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 5

Protocol messages, IP Packets, Fragments, …

IP UDP

message

Application data

datagram fragment 1

IP UDP application data

packet

IP Application data

fragment 2

IP Application data

fragment 3

link IP UDP application data link IP Application data

frame

link IP Application data

Derived from Figure 8.7 from Wright/Stevens, Vol. 2

January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 6

IP Packet Format

0 Vers HLen TOS Total length ID Flags Fragment Offset TTL Protocol

. . .

Header Checksum Source Address Destination Address Source Port Destination Port Packet Payload 8 16 24 31 IP Header (with no Options fields) Transport Protocol (e.g. TCP, UDP) Header January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 7

IPv4 Forwarding in the NetBSD Kernel (APIC) N E T W O R K T R A N S P O R T

tcp/udp_input ip6_input ipintr socket layer ip6_forward ip_forward tcp/udp_output ip6_output ip_output ipintrq:

L I N K D A T A

atmc_input apic_isr apic_intr atmc_output apic_devoutput PS_Q packet_scheduler

U P P E R

apic_start

L O W E R

January 9, 2001 input packets interrupts output packets

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 8

User IP in the NetBSD Kernel

User Program

N E T W O R K T R A N S P O R T

socket layer Waits in Socket tcp/udp_input Buffer for User Program to pick up tcp/udp_output ipintr ip_forward Software Interrupt ip_output ipintrq:

L I N K D A T A

atmc_input apic_isr apic_intr atmc_output apic_devoutput PS_Q packet_scheduler

U P P E R

apic_start

L O W E R

January 9, 2001 input packets interrupts output packets

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 9

IP Packet Handling in the CB Kernel

User Space Kernel SW Interrupt HW Interrupt

D A T A L I N K N E T W O R K T R N S P O R T

tcp/udp_input socket layer

Here are the rest of the gates?

tcp/udp_output ip6_input ip6_forward ip6_output ipintr ip_forward ip_output ipintrq: atmc_input apic_isr apic_intr atmc_output apic_devoutput

sgate dgate

PS_Q packet_scheduler

U P P E R

apic_start

L O W E R

January 9, 2001 input packets interrupts output packets

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 10

Important Source Files (usr/src/sys/netinet/ip*)

• ip.h

– struct ip { } – #define’s • ip_input.c

– ipintr() (This is ip_input routine) – ip_forward() • ip_output.c

– ip_output() January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 11

IP Packet Handling in CB Kernel with APIC

• Device Driver -- Device Dependent –

apic_intr

() » read INTR_ACK: 0 => not for us, return »

apic_isr

() –

apic_isr

() » read NOTIFY_LIST, next rcv channel needing attention • Process descriptors for RCV channel – swap words in each received mbuf (APIC BUG) –

link mbufs until we find end of frame

– verify CRC (just check flag set by APIC) –

atmc_input

(packet) – repeat ipintr • repeat ipintrq: atmc_input ip_forward ip_output atmc_output apic_isr apic_intr apic_devoutput January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 12

IP Packet Handling in CB Kernel (continued)

• Device Driver - Dev. Independent

atmc_input

(): – LLC/SNAP processing (if needed) » extract type (IP/RATM/…) – AAL5 processing » extract length » strip trailer –

aiu_getafix

(packet) (flow table will be discussed later) »

aiu_getafix stores a ptr to FTE in packet

–

Enqueue in IP Queue

January 9, 2001 ipintr ipintrq: atmc_input ip_forward ip_output atmc_output apic_isr apic_intr apic_devoutput

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 13

IP Packet Handling in CB Kernel (continued)

• IP Input --

ipintr

(): – Get next packet from IP Queue – Do some basic checks, header, length, checksum… – process IP options –

aiu_dgate(&m,2)

Router Plugins Dynamic Gate #2 – if packet is not for us and we can forward it: » forward packet

ip_forward

() -- upcoming slide...

» return –

aiu_dgate(&m,4)

Router Plugins Dynamic Gate #4 – protocol specific input routine e.g.

tcp_input

() ipintr ipintrq: atmc_input ip_forward ip_output atmc_output apic_isr apic_intr apic_devoutput January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 14

IP Packet Handling in CB Kernel (continued)

• IP Forwarding --

ip_forward

(): – – – decide if we need to send any redirects to sender –

aiu_dgate(&m,5)

Router Plugins Dynamic Gate #5

route lookup

ip_output

() Update for Fred’s new Gate also check details of if/when route lookup is done January 9, 2001 ipintr ipintrq: atmc_input ip_forward ip_output atmc_output apic_isr apic_intr apic_devoutput

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 15

IP Packet Handling in CB Kernel (continued)

• IP Output --

ip_output

() –

aiu_dgate(&m,3)

– send on interface Router Plugins Dynamic Gate #3 – if (no route yet) ip_forward ip_output ipintr »

get route

ipintrq: atmc_input atmc_output – check for special processing apic_isr apic_intr apic_devoutput » ANEP Options •

anep_output

() – if DAN then

afd_handle_dan_packet

() :

ACTIVE PROCESSING

January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 16

IP Packet Handling in CB Kernel (continued)

• Device Driver Device Indep.--

atmc_output

(): – LLC/SNAP processing – AAL5 processing (if needed) – if packet_scheduling: enqueue for PS – if !packet_scheduling: send to devoutput() • Device Driver Device Dep. --

apic_devoutput

() –

configure an APIC descriptor for each mbuf packet in

– Resume APIC TX channel ip_forward ip_output ipintr ipintrq: atmc_input atmc_output apic_isr apic_intr apic_devoutput January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 17

Note about Addresses

• The address structures in the Crossbow Kernel are IPv6 – IPv6 address structures are a superset of IPv4 – IPv6 address are 128 bits • We will be using IPv4 addresses – e.g. 192.168.5.2

– IPv4 addresses are 32 bits • Notation for using IPv4 address in IPv6: – Use double colon before address: ::192.168.5.2

– Double colon tells the utilities to set everything to the left to 0’s January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 18

NetBSD Kernel Topics

IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous

January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 19

mbuf{} m_next m_nextpkt m_len m_data m_type m_flags

0-108 MT_xxx 0

The mbuf Data Structure

mbuf{}

0-100 MT_xxx

M_PKTHDR

mbuf{}

208-2048 MT_xxx

M_EXT

2048

mbuf{}

208-2048 MT_xxx

M_PKTHDR | M_EXT

2048

m_pkthdr.len

m_pkthdr.rcvif

m_ext.ext_buf

m_ext.ext_free

m_ext.ext_size

January 9, 2001 Normal mbuf 2048-byte cluster 2048-byte cluster Normal mbuf with Packet Hdr Cluster mbuf

Router Plugins (Crossbow)

Cluster mbuf with Packet Hdr Washington WASHINGTON UNIVERSITY IN ST LOUIS 20

• Highlight: (add Stevens’ like figures…) – m_next vs m_nextpkt (figure 2.2 page 35) » m_next: points to next mbuf in THIS packet » m_nextpkt: points to the first mbuf of the NEXT packet – m_len vs. m_pkthdr.len

– m_ext.ext_buf vs. m_data – Which mbuf structure we use with APIC • Example: – do m_pullup – get data pointer

Sorry: This was just one of my notes slides that should have been removed

– print out src addr, dst addr and protocol fields – get Transport header pointer – print out dst port January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 21

The mbuf Data Structure

m_next m_nextpkt m_len m_data m_type m_flags mbuf{}

0

mbuf{} mbuf{} mbuf{}

0-100 208-2048 208-2048 MT_DATA MT_xxx

M_PKTHDR

MT_xxx

M_EXT

MT_xxx

M_PKTHDR | M_EXT

Sorry: This was just one of my notes slides that should have been removed

2048 2048

before printing

m_pkthdr.len

m_ext.ext_size

2048-byte cluster 2048-byte cluster January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 22

The mbuf Data Structure (example)

head of queue tail of queue mbuf{} m_next m_nextpkt m_len m_data next mbuf in chain

42

m_type

MT_DATA

m_flags m_pkthdr.len

m_pkthdr.rcvif

M_PKTHDR 192 NULL

mbuf packet header mbuf{} next mbuf in chain m_next m_nextpkt m_len m_data

NULL 100

m_type

MT_DATA

m_flags m_pkthdr.len

0

m_pkthdr.rcvif

mbuf{} m_next m_nextpkt m_len m_data m_type m_flags 50 bytes of data

NULL NULL 50 MT_DATA 0

100 bytes of data Ethernet header, IP header, UDP header next packet in queue January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 23

tail of queue

The mbuf Data Structure (example, continued)

mbuf{} m_next m_nextpkt m_len m_data next mbuf in chain

NULL 54

m_type

MT_HEADER

m_flags m_pkthdr.len

m_pkthdr.rcvif

M_PKTHDR 1514 NULL

mbuf packet header mbuf{} m_next m_nextpkt m_len m_data

NULL NULL 1460

m_type

MT_DATA

m_flags m_pkthdr.len

M_EXT

m_pkthdr.rcvif

Ethernet header, IP header, TCP header m_ext.ext_buf

m_ext.ext_free

m_ext.ext_size

2048 2048 –byte cluster 1460 bytes of data

January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 24

NetBSD Kernel Topics

IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous

January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 25

Loadable Kernel Modules

• Mechanism in NetBSD to dynamically load code into running kernel • NetBSD System Utilities: – modload(8) – modunload(8) – modstat(8) • Module Types supported: – System Call modules » When unloaded, returns to original system call » Any system call can be replace » Take care when replacing ioctl(2) since LKM uses it to load/unload modules!!

– Virtual File System modules – Device Driver modules: Block and character device drivers – Execution Interpreters: For binaries not normally usable by OS – Miscellaneous modules » No defined interfaces » Up to user/developer to provide hooks to get to the code » This is what Router Plugins uses January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 26

Loadable Kernel Modules: modload

• Open /dev/lkm – ioctl’s will be performed on the open file descriptor • Prelink module, open it and calculate size info – system(“ld -A /netbsd -e _ -o -T ”); • Reserve Kernel memory: – ioctl(lkm_fd, LMRESERV, size_info) – returns kernel load address • Relink at kernel load address • Open relinked module and load it into kernel – ioctl(lkm_fd, LMLOADBUF, ...) • Adjusting symbol table entry pointers and load it.

• sync(); • Call the module’s entry function to test it.

• Post-install if called for...

– ioctl(lkm_fd, LMSTAT, ...) January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 27

NetBSD Kernel Topics

IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous

January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 28

Interrupts

• 8 levels – spl0 – splsoftclock – splnet – spltty – splbio – splimp – splclock – splhigh normal low-priority clock processing network protocol processing terminal I/O disk and tape I/O network device I/O high-priority clock processing all interrupts blocked • Typical usage: int i; i = splimp(); [do something at device I/O level] splx(i); January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 29

NetBSD Kernel Topics

IP Processing mbuf structure Loadable Kernel Modules Interrupts Miscellaneous

January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 30

Other kernel details you must know

• No floating point • No #include in header files • More caveats and info from Fred later… January 9, 2001

Router Plugins (Crossbow)

Washington WASHINGTON UNIVERSITY IN ST LOUIS 31