Performance Debugging Measurements Overview

Matt Zekauskas, [email protected] GNEW 2004

2004-Mar-15

Really

Measurement project collaboration in the US that I am aware of Two recent events • Performance (Debugging) Measurement Architecture Workshop (December) • Ongoing work by US Federal R&E nets + Abilene to cooperate (Joint Engineering Team) – workshop upcoming in April 2004-Mar-15

2

Performance Debugging Measurements

More Specifically

Working with DANTE, TF-NGN, UCL, GGF, NLANR (DAST on Advisor; MNA on AMP/PMA), and knowing a bunch of more basic research projects… widen the conversation on how we can work more cooperatively?

Is there an overall plan/architecture?

What’s common, what’s missing?

2004-Mar-15

3

Performance Debugging Measurements

P(d)MA 2003

Performance (Debugging) Measurement Architecture Workshop 2003 1.75 day workshop last December Bring together a bunch of NSF-funded projects Including a related one: end-to-end middleware diagnostics (start with authorization and authentication, but can expand) And some network-specific [BB, Campus] 2004-Mar-15

4

Performance Debugging Measurements

Not a new idea

“Challenge: evolve the field of Internet measurement from a set of disjoint, independent activities of network researchers, operators, and users toward an integrated service of the network that can support diagnosis of problems in real time.” – kc claffy, ed. “Measurement Roadmap” from the NREN/NASA workshop in 2000 on gigabit networking 2004-Mar-15

5

Performance Debugging Measurements

Our leverage

Projects specifically about the integration R&E backbone networks are interested in supporting A few big demanding users (e.g. HENP) with diverse endpoints Performance Debugging Measurements 2004-Mar-15

6

Existing Architectures

Grid Measurement piPEs TF-NGN Performance Monitoring Group Internet2 Middleware End-to-End Diagnostics Performance Debugging Measurements 2004-Mar-15

7

Internet2 Middleware E2E

Federated Security – Shibboleth; Authenticate locally, Authorize globally Based on trust of institutional authentication Pass minimum amount of information necessary If failure • What piece of the distributed infrastructure is to blame?

• Want to debug without releasing private information Diagnostic backplane with common event record 2004-Mar-15

8

Performance Debugging Measurements

Current Projects

CalNGI / NPACI • Accessible to researchers NIMI • Authentication focus • Experiments SLAC • pinger, iepm-bw • netflow {local} NLANR DAST: Advisor NLANR MNA • AMP, PMA {oc192 soon} piPEs, as implemented TeraGrid (SAMI) Esnet Perf. Monitring LBL: NTAF, SCNM IU: tools for campuses • Flowscan, gcube, L2TP Mathis: NPAD 2004-Mar-15

9

Performance Debugging Measurements

Backbones

Abilene Observatory, piPEs in Abilene • Ongoing measurements, openly accessable; on demand with piPEs for throughput (bwctl) or latency (owamp) TeraGrid • Ongoing measurements ESnet • On-demand for ESnet sites (Ongoing work on GEANT) 2004-Mar-15

10

Performance Debugging Measurements

Wide-ranging Projects

pinger, iepm-bw • Ongoing active AMP • Ongoing active NIMI • Active, but focussed on experiments (pipEs) • Ongoing active Performance Debugging Measurements 2004-Mar-15

11

Drawing conclusions

Advisor (based on iperf, Web100) • Intelligence at ends Rich Carlson’s NDT • Intelligence at middle (packet streams, Web100 at Server) piPEs provide data to others, Rich Carlson is working on the piPEs analysis component 2004-Mar-15

12

Performance Debugging Measurements

New tools

Most of the projects that integrate use existing tools • Ping, tracroute, iperf (or similar) • SNMP (some) Some new tools arriving • Capacity discovery, lower-overhead throughput testing • Web100-style host instrumentation 2004-Mar-15

13

Performance Debugging Measurements

Network Path & Application Diagnostic

NPAD – New tool(set) under development at PSC, Matt Mathis • Capitalize on Web100, but look for flaws in link layers –Tweak TCP on short runs, detect problems that would show up on long runs [drops, reordering]: hits network harder than UDP tests, but doesn’t drive into congestion –Find bugs in campus infrastructure • Way to bench-test applications via packet reflection 2004-Mar-15

14

Performance Debugging Measurements

Plug: Abilene Observatory

Publish measurement data • Some we collect for operations • Some we collect for research The ability for research projects to add their equipment, or run on our platform [limited] • Peer reviewed • Why? Passive, collocation makes analysis easier • AMP, PMA, Planetlab [prefer run own code here] 2004-Mar-15

15

Performance Debugging Measurements

Measurement Data

One way latency, jitter, loss [owamp] • IPv4 and IPv6 Regular TCP/UDP throughput tests – ~1 Gbps • IPv4 and IPv6; On-demand available [bwctl] SNMP (NOC) [octets, packets, errors; collected frequently] • NOC working on SNMP proxy “Netflow” (ITEC Ohio) [anon. by 0-ing last 11 bits] Multicast beacon with historical data Routing data (BGP & IGP) [IGP under development] • Looking at Zebra + mods, Japanese routing research is driver 2004-Mar-15

16

Performance Debugging Measurements

“Visual Backbone”

XML access of 1/hr snapshots • Lots of show commands • Show config, isis, bgp, interface, … • Multicast stuff • http://loadrunner.uits.iu.edu/~gcbrowni/Abilene/ • MonaLisa uses this data! http://vinci.cacr.caltech.edu:8080/ Router proxy: ask questions of router: safely, rate controlled 2004-Mar-15

17

Performance Debugging Measurements

My view of results

People want to work together, but they also want autonomy (don’t constrain possible solutions by architecture) Want ways to share results, enable cross-framework testing (but not forced) Tools folks would like to use existing infrastructures Performance Debugging Measurements 2004-Mar-15

18

My view of results

Common Themes • Discovery (domains, boxes along path) • Interfaces (requests, response) • Authorization, Authentication • Calibration, Meta-data • Infra. Operational issues • Algebra for metrics • How know physical topology [ not common, but big discussion!] 2004-Mar-15

19

Performance Debugging Measurements

Wrappers Wrappers History Real-Time On Demand

DISPLAY / ANALYSIS

Interrupt Discovery Test R/R Result R/R Test Availability Scheduling AAA Who are you?

What do you want to do?

Framework

Optional: Database Cache Requestor Scheduler On Demand Continuous Passive

TOOLS Policy

(MDI)

Schedule

(PMC)

Invoke

(PMP) Tool Wrapper Library: + CMD + Output + Logs (publish to encourage commonality

Reference for Workshop

http://e2epi.internet2.edu/WK03/ But… we are still working on a report.

Performance Debugging Measurements 2004-Mar-15

21

Summary

Taste of what’s going on (US-centrically) R&E networks are planning or implementing measurement frameworks Looking to interoperate New tools for segments, correlation on horizion You are an important customer, friendly partner 2004-Mar-15

22

Performance Debugging Measurements

Summary

Lots of details (and hard problems!) left Interoperable federated(?) authentication Finding the right points An algebra for measurements What can we do for “lightpaths” (in part, build measurements/reporting into layer 2 and expose them) (in part, endpoint cooperation) 2004-Mar-15

23

Performance Debugging Measurements

Acknowledgements

NSF • This material is based in part on work supported by the National Science Foundation under Grant No. ANI-0314723. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation (NSF).

San Diego Supercomputer Center CAIDA Internet2, especially the End-to-End Performance Initiative All the participants!

2004-Mar-15

24

Performance Debugging Measurements

www.internet2.edu

Abilene URLs

http://abilene.internet2.edu/observatory • Pointers to all measurements/sites/projects http://www.abilene.iu.edu/ • NOC home page. Weathermap, Proxy, SNMP measurements http://netflow.internet2.edu/weekly/ • Summarized flow data http://www.itec.oar.net/abilene-netflow/ • “Raw” – matrices; (Anon) feeds available on request 2004-Mar-15

26

Performance Debugging Measurements