Transcript Managing SPP Resources: System Resource Manager (SRM) Fred Kuhns Applied Research Laboratory

Managing SPP Resources:
System Resource Manager (SRM)
Fred Kuhns
[email protected]
Applied Research Laboratory
Washington University in St. Louis
Common Definitions and Types
• For all commands the message context ID is used to
identify the context within which a command is to
be executed.
– The special value of 0 (cid = 0) indicates a privileged
operation performed by the substrate.
– Otherwise the context ID is an identifier indicating a user
specific context in which the command is to be interpreted
and executing. In most cases this is the Global Fastpath
ID.
• See presentation titled Types.ppt
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
2
NATD Interface
• [egress_map, ingress_map]
get_sched_map(ipAddr_t link, hwaddr_t board)
– Message context ID = 0.
sched_map_t {
qmID_t qmID;
// 2-bit QM identifier
schID_t schID; // 3-bit qm relative scheduler identifier
qid_t qid;
// 16-bit queue identifier
statID_t sindx; // Stats Index
vlanID_t vid;
// VLAN identifier
}
Return:
egress_map: sched_map_t
ingress_map: sched_map_t
Arguments:
link: Interface IP address (aka external IP)
board: Board/Device MAC Address
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
3
Reserving Fastpath Resources
See Types.ppt for type definition and mapping to substrate API, message context ID = 0.
Reserve resources for 0..N fastpaths (on NPEs) and 0..1 GPEs
• retCode_t make_reservation(plabID, rsvRecord_t)
Update an existing reservation, note because all operations need to be idempotent
the update argument includes all required resources.
• retCode_t update_reservation(plabID_t, rsvRecord_t)
Cancel any reservation overlapping date_t
• retCode_t cancel_reservation(plabID_t, date_t)
Return any reservation (belonging to slice) that overlaps with date_t
• rsvRecord_t? get_reservation(plabID_t, date_t)
Return a list of all reservations associated with calling slice
• rsvRecord_t* get_reservations(plabID_t)
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
4
Allocating Fast-path Instance
•
fastpath_t alloc_fastpath(plabID_t, copt, bwspec, rcnts, mem)
– Message context ID = 0
plabID_t : GPE/PlanetLab slice identifier
uint16_t copt : code option type. {IPv4=1,I3=2}
struct bwspec {bw_t bw-firm, bw-soft;}
struct rcnts {cnt_t fltrs, queues, buffers, stats;}
struct mem
{cnt_t sram, dram;} // Requested size of SRAM and DRAM in Bytes
struct fastpath_t {
fpid_t fpid;
// fastpath id assigned by srm, used as context ID in future messages
vlan_t vlan;
// VLAN id to associate with fp’s logical interface
ipAddr_t fpIP; // IP address of logical fp’s interface on GPE
ipAddr_t npeIP; // IP address for fp’s data interface on NPE
ipAddr_t fpNM; // Netmask of fp’s logical interface
ipAddr_t scdIP; // IP address for fp’s assigned SCD
ipPort_t scdPort;
// UDP Port number for fp’s assigned SCD
hwaddr_t npeMAC;}
// NPE MAC address
– bw-firm: Expresses the required aggregate processing resources in terms of BW, units are Kbps.
– bw-soft: This is just a hint as to the expected maximum processing resources required. Resources are not actually allocated to the
fastpath, rather the system uses this number for load balancing best-effort fastpaths across the available NPEs..
•
Actions required to allocate code option instance and resources:
–
Select NPE
•
•
–
Either allocates requested resources or returns error
•
•
–
–
Load balance across available NPEs
Must have sufficient resources (SRAM, TCAM space, queues, etc)
Keeps memory map of SRAM so can perform allocation, though the absolute starting address is not required.
If compaction is necessary then must communicate with SCD directly.
Allocate VLAN and configure switch.
Send commands to SCD on NPE
•
•
•
•
set_fastpath(fpid, copt, VLAN, TParams, Mem[])
set_sched_params(sid(gpe) GPE_MIID, GPE_BW_MAX*, GPE_BW_MIN)
set_fpmi_bw(fpid, sid(gpe), GPE_MIID, GPE_BW_MIN)
set_encap_gpe(fpid, gpeIP(fpid), npeIP(fpid))
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
5
Releasing a fast-path instance
• retCode_t free_fastpath()
– Message context ID = Global fastpath ID (fpid)
– Send command to SCD on NPE
• set_sched_params(sid(gpe), GPE_MIID, GPE_BW_MAX*,
GPE_BW_MIN)
• retCode_t set_fpmi_bw(fpid, sid(gpe), GPE_MIID, 0)
• set_encap_gpe(fpid, 0, 0)
• rem_fastpath(fpid)
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
6
Instantiate Fastpath and/or Slowpath reservations
• All resource allocation methods operate on the currently active reservation
(i.e. must be used during the interval of time when a reservation is valid).
– If free is not called then all resources are automatically freed when reservation
expires.
allocate any GPE/Slice resources reserved in the current interval
• retCode_t alloc_plspec(plabID_t)
instantiate the “current” fastpath reservation with name fpName
• fpInfo_t alloc_fpspec(plabID_t, name_t fpName=“”)
– if only one fastpath then the fpName parameter can be omitted.
free any GPE specific resources currently allocated to slice.
• retCode_t free_plspec(plabID_t)
Destroys fastpath instance, releasing meta-interfaces and NPE resources.
• retCode_t free_fpspec()
– Message context ID set to the Global fastpath ID
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
7
Managing Interfaces
Reserve Aggregate Interface BW for slice (traffic to/from GPE).
• retcode resrv_slice_ifbw(plabID, bw, ifn)
Release slice interface BW reservation for GPE
• retcode reles_slice_ifbw(plabID, bw, ifn)
Reserve aggregate Interface BW for indicated fastpath
• retcode resrv_fpath_ifbw(bw, ifn)
– message context ID = fpid
Release Aggregate Interface BW allocation for indicated fastpath
• retcode reles_fpath_ifbw(bw, ifn)
– message context ID = fpid
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
8
System Interfaces, Addresses and Ports
Get list of available interfaces
• if_list get_interfaces(plabID)
– Message context ID = 0.
– if_list = {ifAttr_t, ...}
ifAttr_t {
ifn_t
ifn;
// Interface number
ifType_t type;
// {Multi-Access = 0, Point-to-Point = 1}
ipAddr_t ipaddr;
// Interface/Link IP address
ipAddr_t linkBW;
// Raw (Gross) interface (link) BW
bw_t
availBW;
// Available (Net) interface (link) BW
}
– if type = point-to-point then may call get_ifpeer()
Map interface IP address to interface number.
• ifn_t get_ifn(plabID_t, ipAddr_t)
Get updated interface attributes
• ifAttr_t get_ifattrs(plabID_t, ifn_t)
Get the IP address of point-to-point link’s peer node (SPP peer).
• ipAddr_t get_ifpeer(plabID_t, ifn_t)
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
9
Managing Externally Visible Address for GPE Traffic
•
•
Message context ID set to 0, plabID used to identify Slice making request
Managing bandwidth:
– SRM: verifies slice meta-interfaces and queues do not exceed interface allocations
– SRM: Notifies SCD of per Scheduler slice limits
Allocate a local endpoint for slice (linecard directs matching traffic to the corresponding
GPE)
• epInfo_t alloc_endpoint(plabID_t, epInfo_t)
– Message context ID = 0.
Release allocated endpoint (linecard filters removed and address added bac kto
available pool of addresses tuples).
• retCode_t free_endpoint(plabID_t, epoint_t)
– Message context ID = 0
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
10
Internal Interfaces for NAT
map{bw_t
ipAddr_t
ipPort_t
ipProto_t
dw1_t
bw;
ipaddr;
port;
proto;
smac};
# Kbps
# local interface IP address
# if TCP/UDP and 0 then allocate
# TCP/UDP,ICMP
# low order 8 bits of source mac
• {fid, port} alloc_epmap(map)
– Message context ID = 0.
– Allocate endpoint and add LC TCAM filter to send pkts to
GPE.
• retCode_t free_epmap(fid)
– Message context ID = 0.
– Allocate endpoint and add LC TCAM filter to send pkts to
GPE.
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
11
Allocating Meta-Interfaces (tunnel endpoint)
•
Common Types
ep {uint32_t bw, uint32_t ipaddr; uint16_t port; uint8_t proto}
dw2_t mi: Meta-interface number for an endpoint. Interface numbers start at 0.
•
•
[mi, ep] alloc_tunnel(bw, ipaddr, tunnel_type, params) Not implemented
[mi, ep] alloc_udp_tunnel(bw, ipaddr, port)
Message context Id = fpid
uint8_t tunnel_type : must be one of {UDP = 0}
uint32_t bw : (Minimum guaranteed) BW assigned to endpoint, Kbps
uint32_t ipaddr: External IP address of endpoint/tunnel. There is a 1-to-1 relationship between links and ip addresses.
uint16_t port: External UDP port number
– Actions required to allocate code option instance and resources:
•
•
•
•
–
Send command to SCD on Line Card Ingress
•
•
•
•
•
–
set_sched_params(sid(link), ifn(link), BW*, BWmin)
set_queue_sched(qid, sid(link))
set_queue_params(qid, threshold, bw)
Send command to SCD on NPE
•
•
•
set_sched_params(sid(fabric,NPE), ifn(fabric), BW*, BWmin)
set_queue_sched(qid, sid(fabric,NPE))
set_queue_params(qid, threshold, bw)
write_fltr(dbid_ingress, fid, key_ingress, mask, result_ingress)
write_fltr(dbid_egress, fid, key_egress, mask, result_egress)
Send commands to Line Card Egress
•
•
•
–
verify requested address is available, assign port if necessary (port == 0)
Allocate new meta-interface ID
allocate and assign filter id for ingress and egress TCAM
Lookup existing or allocate qid for ingress and/or egress
set_sched_params(sid(link, miid), ifn(fabric), BW*, BWmin)
set_fpmi_bw(fpid, sid(link, miid), miid, bw*)
ep get_endpoint(mi)
Message context Id = fpid
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
12
Releasing Meta-Interfaces
•
•
Common Types
ep {uint32_t bw, uint32_t ipaddr; uint16_t port; uint8_t proto}
uint_t mi: Meta-interface number for an endpoint. Interface numbers start at 0.
retCode_t free_tunnel(ipaddr, tunnel_type, params)
retCode_t free_udp_tunnel(ipaddr, port)
–
Message context Id = fpid.
uint32_t ipaddr: External IP address of endpoint/tunnel
uint16_t port: External UDP port number
–
Actions required to allocate code option instance and resources:
– Send command to SCD on Line Card Ingress
•
•
•
•
set_sched_params(sid(fabric,NPE), ifn(fabric), BW*, BWmin)
set_queue_params(qid, threshold, bw)
rem_fltr_byfid(dbid_ingress, fid)
rem_fltr.fid(dbid_egress)
– Send commands to Line Card Egress
• set_sched_params(sid(link), ifn(link), BW*, BWmin)
• set_queue_params(qid, threshold, bw)
– Send command to SCD on NPE
• set_sched_params(sid(link, miid), ifn(fabric), BW*, BWmin)
• set_fpmi_bw(fpid, sid(link, miid), miid, bw*)
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
13
Allocating Meta-Interfaces (tunnel endpoint)
--- Simplified interface, see steps from deprecated slides --Allocate local Address for Slice’s fastpath (MI) and forward to NPE, will also reserve
sufficient aggregate BW on corresponding internal and external interfaces.
• [miID_t, epInfo_t]
alloc_udp_tunnel(bw_t, ipAddr_t, ipPort_t)
– Message context ID set to the global fastpath ID
–
–
–
–
It is an error if the sum of requested BW exceeds the reserved fastpath BW.
Adds one end of a UDP tunnel, treated as a meta-interface for a fast path instance.
if port = 0 then one is allocated from the free pool
Returns the meta-interface number representing the new endpoint
Release BW and Address associated with corresponding meta-interface
• retCode_t free_udp_tunnel(ipAddr_t, ipPort_t)
Map meta-interface ID to corresponding externally visible address tuple
• epInfo_t get_endpoint(miID_t)
– mi must be a valid meta-interface ID returned from a call to alloc_udp_tunnel
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
14
Classes and Tables
Per Slice
Tables
gpe
board id
vlan
BW
plabID
NPE (allocated)
sram {start,size} #flts
board ID
#Qs
BW
free {...}
resvMap
#Stats
meta-ifaces
mi:endpoint
...
plab sliceID
xsid
VLAN maps
range:{start,end}
...
...
NPE Table
Interfaces
System
ifn:{type,ipaddr,linkBW,availBW}
id:{addr,BW/Port,copts,fltrs,sram,Qs}
tables
endpoint (port) maps
availMap usedMaps
controlIP GPE
BWmaps
xsidMap
Slices
endpoint references
servMap
resvMap
xsid
Slice Table
plabID
Mappings:
plabID to Entry
xsid to Enry
Entry: {plabID, xsid, gpe, npe, MI}
GPE: {GPE object reference, BW,
EPs, ??}
NPE: {NPE object reference, vlan
bw, sram, fltrs, qs, stats}
MI: {EP object reference, ID}
Fred Kuhns - 5/27/2016
Washington
WASHINGTON UNIVERSITY IN ST LOUIS
15