Transcript Requirements for Broadband Wholesaling

Bulk Data Retrieval
Implemented as MIB extensions
Bryan Levin
[email protected]
David Battle
[email protected]
draft-ietf-eos-snmp-bulkdata-01.txt
MPLS, 18-mar-2002
Problem Statement
• SNMP/UDP datagrams are not efficient for Bulk Data
– UDP has limited payload size
• SNMP/TCP never gained mass acceptance
– Double ACK’ing is inefficient
• SNMP lacks native ability for atomic snapshots
– get-bulk is great but still not atomic
• No SNMP way to sync shapshot to many NMS’s
– Each NMS’s polled data would be different (time skew)
Goals
• Define semantics for building synthetic tables
– Based on real table “slices”
• Define semantics for atomic table snapshots
– Snapshot happens entirely on local agent
• Define semantics for bulk file transfers
– Using existing encryption and compression
• Define data file format to be as simple as possible
– ASCII flatfiles are very easy to write and read
• Allow remote control of locally queued snapshots
Proposed Solution Based on Existing Ideas
• Many vendors use “SNMP-triggered” file transfers
– Understood & accepted method of sending large datasets.
• Additional features added
–
–
–
–
–
–
Atomic snapshot semantic
Data slicing creates synthetic tables
Compression/Encryption
File redundancy
Both Push and Pull models supported
Local snapshot queue management
• Not addressed
– Row-based filtering/suppression
Benefits
• Leverages existing File Transfer Protocols
– FTP, SCP, HTTP are well debugged and deployed.
– Minimal engineering needs be done
• Lower risk since less new code has to be developed.
• Quick interoperability with minimal fuss
– Many fileservers already have ftpd and sshd configured.
• Reduced NMS polling & interactions
– The NMS for initial Bulk Definition (via an SNMP SET).
– File transfer reliable & on time (agent self-triggering).
Synthetic Tables
• Similar to D. Perkin’s getcols & Nalin Pai’s idea
– Allow the user to define a table in terms of columns from existing
(agent) tables
– Once a synthetic table has been defined, it can be reused without the
need to redefine it in every incoming PDU
– This works for tables that are logical ‘augmentations’ of each other
(classic example: ifTable and ifXtable)
Setup Procedure Overview
1) Read agent’s snapshot capabilities (scalars)
2) Define a synthetic table (SliceTable entries)
– List of column OIDs from real tables
3) Define fileserver table entries
– Where and how to get the snapshot to the server
4) Request a snapshot to a local file
– Refer to an entry in (2) and an entry in (3)
5) Monitor results table
– Check status, abort, retry, delete-row
MIB Scalars
• NMS should check these before initiating SETs
• Describes the agent’s snapshot mib capabilities
– acFileEncoding (xml, ascii)
– acFileCompression (none, bzip, gzip)
– acXferProtocol (cp, ftp, scp)
MIB Table #1 Slice Table
• Describe a subset of columns from real tables
– sliceIndex, sliceSubId (major, minor instance fields)
– sliceColumnOID (column to include in synthetic table definition)
– sliceSnmpContext (community string to select mgmt context)
– sliceColumnDisplayHint (printf-style %s, %d strings)
– sliceAdminString (for NMS use)
– sliceEntryStatus (RowStatus)
MIB Table #2 Xfer Table
• Describe Where and How it gets there
– xferIndex, xferSubId (instance fields)
– xferHostIpType, xferHostIpAddr (network address of fileserver)
– xferProtocol (cp, ftp, scp)
– xferWriteControl (failIfExists, ovewrite)
– xferFilePath (path on fileserver to save files)
– xferAuthUsername, xferAuthPassword (remote authentication)
– xferAdminString (for NMS use)
– xferEntryStatus (RowStatus)
MIB Table #3 Snapshot Table
• Request a snapshot to local non-volatile storage
– index, sliceIndex, xferId (instance fields)
– snapshotSnapFileName (name used to save snapshot, local+remote)
– snapshotFileEncoding (ascii, xml)
– snapshotFileCompression (gzip, bzip, none)
– snapshotStartTime, snapshotCompletionTime (timestamps)
– snapshotFileSize (of actual snapshot file)
– snapshotState (status of local snapshot event)
– snapshotAdminString (for NMS use)
– snapshotEntryStatus (RowStatus)
MIB Table #4 XferControl Table
• Monitor & Manage queued transfer requests
– xferCtlIndex, sliceIndex, xferIndex, xferSubId (instance fields)
– xferCtlStartTime (timestamp at begin of file transfer)
– xferCtlCompletionTime (timestamp at ending of file transfer)
– xferCtlPercentXferred (amount of file that has transferred so far)
– xferCtlStatus (misc status + abort, retry, delete)
sliceTable
xferTable
snapshotTable
xferControlTable
Result of Snapshot
# 992796710, index, ifHCInOctets, ifInErrors, ifHCOutOctets, ifOutErrors
1,
9279383,
25,
8443465,
2
5,
2794335,
0,
1646610,
0
10,
733837,
1,
731334,
28
14,
2793883,
22,
2789734,
397
23,
794335,
6,
564610,
19
cp
ftp
scp
localhost:/pub/stats/
192.168.1.1:/home/abc/
192.168.1.2:/home/abc/
Iftable_030102.gz
Iftable_030102.gz
Iftable_030102.gz
Server-side Bulk Table Format
• CSV’s are natural for application import
• File contents are viewable by standard text editors
# timestamp, column, column,…
inst,val,val,val,…
inst,val,val,val,…
…
• Empty cells are indicated by “null” values (,,)
• Quotes are allowed, escaped by reverse slash “\”
Bulk Table File Examples
ifTable
# 992796710, ifDescr, ifOperStatus, ifLastChange, ifHCInOctets, ifHCOutOctets
1, fxp0, up(1), 992793883, 2443465, 24435775
5, ge-6/1/0.0, up(1), 992794335, 16496610, 14634989
10, fddi-0, down(2), 992733837, 2342334, 2863772
14, fxp0.0, up(1), 992793883, 38734, 39773
23, so-7/0/0, up(1), 992794335, 16496610, 14634989
ipCidrRouteTable
# 992796710, ipCidrRouteIfIndex, ipCidrRouteType, ipCidrRouteProto
0.0.0.0.0.0.0.0.0.192.168.4.254 14, remote(4), netmgt(3)
192.168.1.0.255.255.255.0.0.192.168.4.254, 5, remote(4), netmgt(3)
192.168.9.195.255.255.255.255.0.0.0.0.0, 23 reject(2), ospf(13)
Summary
• Low impact to existing management infrastructure
• File compression keeps xfer overhead low
• Datasets as ASCII comma-delimited fields
• Rendundant fileservers (multiple file copies) allowed
• Local atomic snapshot semantic
• Data slices from similarly-instanced real tables
• Timing (snapshot, upload) determined by agent
• Leverages other MIBS (eg, DISMAN)