Transcript No Slide Title

Theoretical foundations and
challenges
John Doyle
Control and Dynamical System
Bioengineering
Electrical Engineering
Caltech
NSF reviewer concerns
• Narrow focus on TCP/AQM
– WAN in Lab absolutely critical for FAST
– Other Caltech projects: existing & new
– External projects, e.g., in Emulab community, HENP, Grid
communities
• WAN in Lab valuable for theory research on entire
protocol stack, not just TCP/AQM.
NSF review criteria
• Intellectual merit
– Theory, implementation, experiment, deployment must
inform and influence each other intimately
– Experimental testbed tied to rich theoretical research
program
• Broader impacts
– Internet as a complex system
– New community of network researchers
Broader impact (cont.)
• Integration of research & education
– Proposed new course on unified theory of complex
systems (Doyle, CDS and BioEng)
– Shortcourse (with Walter Willinger) at 2003 Sigcomm
– Other shortcourses in 2003
• Diversity (Doyle’s group)
–
–
–
–
–
2 of 4 postdocs are women
6 of 11 graduate students are women
Wide ethnic and racial diversity
Incoming 2003 class even more diverse
Nearly all co-authored papers have female co-author(s)
Spectrum of tools
log(cost)
HENP
Abilene
CalREN
WAIL
PlanetLab
CAIRN
NLR
?
DummyNet
EmuLab
ModelNet
NS
WAIL
SSFNet
QualNet
JavaSim
Mathis formula
Optimization
Linear model
Nonlinear model
Stochastic model
log(abstraction)
live nk
WANiLab
emulation
simulation
math
…we use them all
errors in
inference
log(cost)
HENP
Abilene
CalREN
WAIL
PlanetLab
CAIRN
NLR
?
DummyNet
EmuLab
ModelNet
NS
WAIL
SSFNet
QualNet
JavaSim
Theory
Mathis formula
Optimization
Linear model
Nonlinear model
Stochastic model
log(abstraction)
live nk
WANiLab
emulation
simulation
math
errors in
modeling
Compute
Communications and computing
Compute
Act
Sense
Physical world &
Environment
Computation
Devices
Devices
Control
Dynamical Systems
From
• Software to/from human
• Human in the loop
Compute
To
• Software to Software
• Full automation
• Integrated control,
comms, computing
• Closer to physical
substrate
Computation
• New capabilities & robustness
• New fragilities & vulnerabilities
Devices
Devices
Control
Dynamical Systems
Two great abstractions of the 20th Century
1. Separate systems from physical substrate
2. Separate systems engineering into control,
communications, and computing
–
–
•
•
Theory
Applications
Facilitated massive, wildly successful, and explosive
growth in both technology and mathematical
theory…
…but creating a new Tower of Babel where even the
experts do not read papers or understand systems
outside their subspecialty.
Biology and advanced technology
• Biology
– Integrates control, communications, computing
– Into distributed control systems
– Built at the molecular level
• Advanced technologies will increasingly do the same
• We need new theory and math, plus unprecedented
connection between systems and devices
• Two challenges for greater integration:
– Unified theory of systems
– Multiscale: from devices to systems
Math
Biology
Advanced
Technology
A coherent foundation is emerging.
Math
Biology
Advanced
Technology
Complementary ways to tell a “convincing” story:
1. Prove theorems
2. Give lots of examples
Focus on familiar
technology:
Internet
Biology
Math
Internet
Technology
Compute
With math and
biology in the
background…
Undecidable
NP
coNP
P
coNP
NP
P
coNP
Hard
Problems
Economics
Algorithms
Controls
NP
Communications
Dynamical Systems
Physics
P
coNP
Hard
Problems
Economics
Biology?
Algorithms
Internet
NP
Controls
Communications
Dynamical Systems
Physics
P
coNP
Hard
Problems
Unified
Economics
Theory
Algorithms
Internet
NP
Controls
Communications
Dynamical Systems
Physics
P
coNP
Hard
Problems
Unified
Economics
Theory
Algorithms
Internet
NP
Controls
Communications
Dynamical Systems
Physics
P
The Internet hourglass
Applications
Web
FTP
Mail
News
Video
Audio
ping
napster
Transport protocols
TCP SCTP UDP
ICMP
IP
Ethernet 802.11
Power lines ATM
Optical
Link technologies
Satellite Bluetooth
The Internet hourglass
Applications
Web
FTP
Mail
TCP
News
Video
Audio
ping
napster
Everything
Transport protocols
on IP
SCTP
UDP
ICMP
IP
Ethernet 802.11
IP on
Power lines ATM Optical
everything
Satellite Bluetooth
Link technologies
From Hari Balakrishnan
The Internet hourglass
Applications
Web
FTP
Mail
News
Video
Audio
ping
napster
Everything
Transport protocols
on IP biological and
Note: All advanced
TCP
SCTP
UDP
ICMP
technological networks are
organized with nested “hourglass”
IP
protocol architectures.
Ethernet 802.11
IP on
Power lines ATM Optical
everything
Satellite Bluetooth
Link technologies
From Hari Balakrishnan
Internet has robustness and fragility
• Can tolerate many orders of magnitude changes in
number of components and network configuration
• Robust to loss of routers and attacks on routers
• Vulnerable to denial of service attacks which use
the network against itself by flooding
Routers
Hosts
Links
Sources
Links
Sources
Network protocols.
HTTP
Files
TCP
IP
packets
packets
packets
packets
packets
packets
Links
Sources
Network protocols.
HTTP
Hidden from the user
Sources
Network protocols.
Vertical decomposition
Protocol Stack
HTTP
TCP
IP
Routing
Provisioning
Network protocols.
HTTP
TCP
IP
Horizontal decomposition
Each level is decentralized and asynchronous
Routing
Provisioning
Vertical decomposition
• These decompositions are essential features for
HTTP
robust networks,
but…
•…“break” standard mathematical theories from
engineering
TCP and science.
• Coherent, complete theory is missing but
possible. First cut nearly done.
IP
• New insight and promising new protocols with
some provable robustness characteristics.
•Beginnings of deep theory of complex networks.
Horizontal decomposition
Routing
Provisioning
web traffic
Web/internet traffic
Is streamed out
on the net.
Web
servers
Creating
internet traffic
Web
client
web traffic
Is streamed
out on the net.
Web
servers
Creating
internet traffic
Let’s look at
some web traffic
Web
client
6
Data
compression
WWW files
5
Mbytes
(Crovella)
4
Cumulative
Frequency
(Huffman)
3
Forest fires
1000 km2
2
(Malamud)
1
0
-1
-6
-5
Decimated data
Log (base 10)
-4
-3
-2
-1
0
1
Size of events
2
6
Web files
5
Codewords
4
Cumulative
Frequency
-1
3
Fires
2
-1/2
1
0
-1
-6
-5
-4
-3
-2
-1
0
1
Size of events
Log (base 10)
2
6
5
 power laws
4
Cumulative
Frequency
Data
compression
WWW files
Mbytes
exponential
-1
3
Forest fires
1000 km2
2
-1/2
1
0
-1
-6
-5
-4
-3
-2
-1
0
1
Size of events
2
6
Data
compression
WWW files
Mbytes
5
exponential
4
Cumulative
Frequency
3
Forest fires
1000 km2
2
All events are
close in size.
1
0
-1
-6
-5
-4
-3
-2
-1
0
1
Size of events
2
6
5
4
Cumulative
Frequency
Data
compression
WWW files
Mbytes
-1
3
Forest fires
Most2events
1000 km2
are small
1
0
-1/2
But the large
events are huge
-1
-6
-5
-4
-3
-2
-1
0
1
Size of events
2
Router
queues
Mice
Sources
Network
Elephants
Router
queues
Mice
Delay sensitive
Sources
Network
Bandwidth
sensitive
Elephants
Router
queues
Mice
Delay sensitive
Why?
Sources
Network
Bandwidth
sensitive
Elephants
Web
layout
Optimize 0-dimensional
cuts in a 1-dimensional
document
# links = # files
Probability of user access to
a simple website
links
• Navigate with feedback
• Limit # clicks= constrain
depth
• Minimize average file size
• Give intuitive cartoon
explanation
Probability of user access
Wasteful
Probability of user access
Hard to navigate.
Probability of user access
•
•
•
•
Optimal layout
Heavy tailed distributions
Mice and elephants
Far greater generality than
hinted at here
Delay
sensitiv
e
Probability of user access
Mice
Bandwidth
sensitive
Elephants
Generalized “coding” theory
Shannon
Web layout
• Minimize avg file transfer
• No feedback
• Discrete (0-d) topology
• Minimize avg file transfer
• Feedback
• 1-d topology
Web
Data compression
Feedback
• Transfer happens once
• User navigates web
• Feedback:
– Click, download, look
– Repeat
Web
Data compression
Topology
• Discrete topology
• Source unordered
• Content determines
topology
• Information is connected
• Trees are 1-d
• Hyperlinking makes d < 1
Web
Data compression
Generalized “coding” theory
• Optimizing d-1 dimensional cuts in d dimensional
spaces…
• To minimize average size of files
• Models of greatly varying detail and sophistication all
give a consistent story.
• The simplest models have almost trivial mathematics.
• Power laws have   1/d.
Web
Data compression
Data
6
DC
5
WWW
4
3
2
1
0
-1
-6
-5
-4
-3
-2
-1
0
1
2
Data + Model/Theory
6
DC
5
WWW
4
3
2
1
0
-1
-6
-5
-4
-3
-2
-1
0
1
2
Data + Model/Theory
6
  1
5
WWW
  
DC
4
3
x0
2
1
0
  x  x0    l  x0 
P( X  x | l )  



-1

 -5 x0 -4 l  x0 -3
-6
1
1

 1
dim 1



-2
0 xl
-1
0
1
2
l
Data + Model/Theory
6
DC
5
WWW
4
3
2
Unified “source coding” theory:
1
1. Data compression (Shannon)
0
2. Web
layout
3. Other
network
applications
-1
-6
-5
-4
-3
-2
-1
0
1
2
How general is this
mice/elephant picture?
•
•
•
•
•
•
•
Selecting and reading books
Selecting and reading magazine articles
Selecting and viewing television
Deciding what movie to go to
Deciding where to go on vacation
Deciding which meetings and classes to attend
Etc….
How general is this
mice/elephant picture?
• Deciding who to have conversations and
relationships with?
• All levels of biological control systems?
• All levels of hierarchical engineering control?
• Information exchange in well-organized
institutions?
Typical web traffic
Heavy tailed
web traffic
 > 1.0
log(freq
> size)
p  s-
Is streamed out
on the net.
Web
servers
Creating fractal
Gaussian internet
traffic (Willinger,…)
log(file size)
3 
H
2
Fat tail
web traffic
time
Is streamed
onto the
Internet
creating long-range
correlations with
3 
H
2
web traffic
Is streamed
out on the net.
Web
servers
Creating
internet traffic
Let’s look at some
internet traffic
Web
client
Internet traffic
Links
Notices of the AMS, September 1998
Poisson
Measured
Internet traffic
Standard Poisson
models don’t capture
long-range correlations.
(Tn X )i 
1
1/ 
n
(i 1) n1
Xj
j in
 1
“bursty” on all
time scales
Fractal
Measured
Fractional Gaussian
(fractal) noise models
measurements well.
Hurst parameter H is an
aggregate measure of
long-range correlations.
(Power laws in
correlations, as opposed
to exponentials.)
“bursty” on all
time scales
What is the
sophisticated
“null” hypothesis?
Internet traffic
Links
That is, what does
the physics
literature claim?
At each link
The SOC
(Self-Organized
Criticality) view

kKl
xkl  cl
flow  capacity
Links
At each link
Average
Queue

kKl
xkl  cl
flow  capacity
Links
Flow
capacity
“phase
transition”
• Lattice without congestion control (?!?)
• “Critical” phase transition at max capacity
• At criticality: self-similar fluctuations, long
tailed queues and latencies, 1/f time series, etc
Average
Queue
Flow
capacity
• Alternative “edge of chaos” models
• Self-similarity due to chaos and independent of
higher-layer characteristics
Why SOC/EOC/… models fail
• No “critical” traffic rate
• Self-similar scaling at all different rates
• TCP can be unstable and perhaps chaotic, but does
not generate self-similar scaling
• Self-similar scaling occurs in all forms of traffic
(TCP and nonTCP)
• Measured traffic is not consistent with these
models
• Fractal and scale-free topology models are equally
specious (for different reasons)
A network based explanation
• Underlying cause: If connections arrive randomly
(in time) and if their size (# packets) have high
variability (i.e. are heavy-tailed with infinite
variance) then the aggregate traffic is perforce
self-similar
• Evidence
– Coherent and mathematically rigorous framework
– Alternative measurements (e.g. TCP connections, IP
flows)
– Alternative analysis (e.g. heavy-tailed property)
Heavy tails in networks?
Heavy tails and divergent length scales are
everywhere in networks.
There is a large literature since 1994:
Leland, Taqqu, Willinger, Wilson
Paxson, Floyd
Crovella, Bestavros
Harchol-Balter,…
Typical web traffic
Heavy tailed
web traffic
 > 1.0
log(freq
> size)
p  s-
Is streamed out
on the net.
Web
servers
Piece of a consistent,
rigorous theory with
supporting
measurements
log(file size)
3 
H
2
Router
queues
Mice
Sources
Network
Elephants
Router
queues
Mice
Delay sensitive
Sources
Network
Bandwidth
sensitive
Elephants
Router
queues
Mice
Delay sensitive
Control
Sources
Network
Bandwidth
sensitive
Elephants
TCP/AQM
flows:
packets
Source.
dxk


 F  xk ,  pl 
dt
lL


Network


dpl
l
 G  pl ,  xk 
dt
 kKl 
Link.
“prices:”
drops or
marks
Horizontal decomposition
Each level is decentralized and asynchronous
Routers


dpl
l
 G  pl ,  xk 
dt
 kKl 
dxk


 F  xk ,  pl 
dt
 lL 
Hosts
Separation theories
• Shannon: Source and channel coding can be done
independently, and under suitable assumption are
asymptotically optimal (as file size  ∞)
• TCP/AQM: Decentralized, asynchronous control
at host and router converges to near-optimal
utilitization
• Application/TCP/IP/Layout: Can these be done
independently and be asymptotically optimal in
any meaningful sense?
Application/TCP/IP/Layout Separation?
• Independent: Each layer can be designed and
implemented decentralized and asynchronously,
assuming only that the other layers are done well,
but without relying on the details.
• Asymptotically : In the limit of large network,
bandwidth, number of users, etc, can get nearly…
• Optimal: “pareto” type, no users performance can
be improved without either additional expense or
degradation of another user’s performance?
• Application versus TCP separation is possible.
• Application/TCP versus IP separation is not
possible for arbitrary network geometries
Elephants
Conjectures
Source
mice
Destination
Bad network
provisioning?
Elephants
Conjectures
Source
mice
Destination
Vertical decomposition
• Bare sketch of a very incomplete but
nevertheless promising “Internet theory”
HTTP
• Must go beyond either control or information
theories alone
• Aim TCP
to explain, understand, and help design
both
IP protocol stack &
– vertical
– horizontal asynchronous and decentralized
decomposition
Horizontal decomposition
Routing
Provisioning
“Minimal” toy topology
“Core” network
Customers
Regional
POPs
Simplifying assumptions:
• Physical and IP connectivity are
identical (No level 2 or MPLS)
• Minimal geometry (= topology
plus link speeds and locations)
• Aim to capture essence of
“topology”
• Add complexity back in later
Line thickness
roughly represents
link bandwidth.
“Minimal” toy topology
“Core” network
Customers
Regional
POPs
Router bandwidth
is constrained.
 BW 
log 
 high speed
 link 
Technically
feasible
Technically
infeasible
high connectivity
log  #links 
Customers
with a
variety of
local
connectivity
speeds.
high
connectivity
Gateway routers
high
connectivity
high
speed
 BW 
log 

link


high
speed
Technically
infeasible
high
connectivity
high speed
high connectivity
log  #links 
log  rank 
1
2
9
3
4
10
1
9
1
10
log  #links 
9
6
5
log  rank 
Total # of nodes = 56
1
2
56
9
  slope
 1.34
9
4
10
1
3
1
10
log  #links 
9
6
5
We are interested in
distributions of core
and gateways so will
largely ignore local
connectivity for now
Local
56
“Core”
?
?
10
9
log(rank)
6
5
4
3
2
1
1
log  #links 
1
10
Gateways
log  rank 
“Scaling” (Mandelbrot)
or “Power law”
56
  slope
 1.34
(Very roughly yes.
And power laws per
se are not important,
but heavy tails are.)
10
1
Do real networks
have power law
connectivity?
1
10
log  #links 
Varied
customer
demand
1
2
9
3
9
4
9
Conjecture
log  rank 
6
56
5
 BW 
log 

 link 
high
speed
 Power law
connectivity
10
+ Bandwidth
constraints
1
high connectivity
1 log  #links  10
log  #links 
What is the “null”
hypothesis?
Varied
Interpret statistics
as
customer
1
2
a probabilistic
model.
demand
9
3
9
4
 BW 
log 

 link 
high
speed
9
log  rank 
6
56
 Power law
connectivity
5
Eliminate “design”
constraints.
10
+ Bandwidth
constraints
1
high connectivity
1 log  #links  10
log  #links 
Total # of nodes = 56
Total # of links = 72
Compute the degree distribution for random
graph with 56 nodes and 72 links. This has
some probability distribution (needs to be looked
up).
Probability that there exists a node with
degree ≥ 20 is vanishingly small.
Total # of nodes = 56
Total # of links = 72
Therefore,
vanishingly
unlikely to
have this
distribution in
a purely
random
graph.
1
10
Degree
distribution for
random graph
with 56 nodes
and 72 links????
0
10
-1
10
-2
10
Probability exists
node with degree
≥ 20 is less than
1e-12????
-3
10
Can reject
simplest null
hypothesis.
-4
10
10
0
10
1
A more sophisticated
null hypothesis.
• Assume this “scaling”
degree distribution but
otherwise random
• Find “typical” or
“generic” cases
• Standard statistical
physics approach to
“complex systems”
• Yields “scale-free”
networks
log  rank 
56
 Power law
connectivity
10
1
1 log  #links  10
log  rank 
56
Note: local
connections
not shown
10
These are extremely
different, but have the same
connectivity distribution.
1
1
log  #links 
10
• Scale-rich
• Highly structured
• Self-dissimilar
• Efficient
• Robust
• Designed
• Scale-free
• Unstructured
• Self-similar
• Wasteful
• Fragile
• “Emergent”
Scale-rich
• Each level has very
different characteristics
• Pieces need not
resemble the whole
Scale-free
• Each level is similar
• Pieces resemble the
whole
• Delete the highly
connected node
• Only local loss
• Delete the highly
connected node
• Global loss
• Remaining parts are still
scale-free
• Delete the worst case
node
• Relatively local loss
• The Internet does have
huge fragilities
• Denial of service, DNS,
BGP, etc
• The one fragility it
doesn’t have is deletion
of routers
• These are completely opposite extremes
• Scale-rich, Highly structured, Self-dissimilar,
Efficient, Robust, Designed
• Vs. Scale-free, Unstructured, Self-similar, Wasteful,
Fragile, “Emergent”
• This is very different from
the real Internet
• More importantly, it is very
different from what any
Internet could possibly be
• Of course, no one would
seriously propose that the
Internet really does look like
this…
• Surely this is just a purely
hypothetical “null hypothesis”
against which to compare
more sensible explanations.
• Right?
• Scale-free
• Unstructured
• Self-similar
• Wasteful
• Fragile
• “Emergent”
• Typical of physics-based
approaches to complex
networks
• This approach dominates
mainstream science
literature
• Assume statistics
describe an otherwise
generic configuration
• Similar to edge-of-chaos,
“order for free,” criticality,
order-disorder transitions,
emergence, …
• Essentially the same
arguments and results hold
for biological networks
• Scale-free
• Unstructured
• Self-similar
• Wasteful
• Fragile
• “Emergent”
log  rank 
56
“Scaling”
10
1
“Scale-free”
1
log  #links 
10
“Scale-rich”
• Power laws (scaling) are
consistent with either scale-rich
or scale-free
• Scale-free is a natural and
sophisticated “null hypothesis”
and is clearly refuted
Varied
customer
demand
1
2
9
3
9
4
9
Designed, Scale-rich,
Highly structured, Selfdissimilar, Efficient,
Robust
6
5
56
log  rank 
 BW 
log 

 link 
+ Bandwidth
constraints
10
high connectivity
1
 Power law
connectivity
high
speed
1 log  #links  10
More realism
• Redundancy can easily give
additional robustness
• Level 2 (ATM, Ethernet) and
MPLS increase IP versus physical
connectivity
• Real networks are obviously
much more complicated
• Does this capture the essence
of the degree distribution?
• What is a simple way to quantify
efficiency and robustness?
• Could customer demand plus
bandwidth constraints give useful
way to generate “realistic”
geometries (= topology plus
locations plus bandwidths)?
ASx
ASx
AS
graph
ASy
ASy
• Every level is much more
complicated.
Message: Theory
should help explain
what is observed
and suggest what
is necessary vs
what is accident.
log  rank 
56
10
Power
laws
1
1
log  #links 
high
speed
10
“Scale-rich”
Constraints
“Scale-free”
high connectivity
•
•
For both metabolism and the Internet graphs we want to
have a picture that looks something like this.
Here bad performance means wasteful and fragile.
Likelihood
High
Low
Scale-free
Empty
Possible, but
to be avoided
“optimal”
Good
Bad
Performance
•
But also we want to make the point that this is a very
general picture, and applies to edge-of-chaos, criticality, and
self-organized criticality as well.
Likelihood
High
Low
EOC/SOC
Empty
“optimal”
Good
Bad
Performance
Is there a nearly
“universal”
mechanism that
explains many
of the specific
instances of
power laws?
6
5
3
FF
2
-1
-6
10
Power
laws
log  #links 
WWW
4
0
56
1
DC
1
log  rank 
1
Data + Model/Theory
10
-5
-4
-3
-2
-1
0
1
2
Reduce
waste &
fragility
Constraints
Somethin
g else gets
big
HOT
Claimed examples of EOC/SOC, fractal and
scale-free networks and related mechanisms
•
•
•
•
•
•
•
Internet traffic and topology
Biological and ecological networks
Evolution and extinction
Earthquakes and forest fires
yield
Finance and economics
Social and political systems
There is an enormous literature in “premier”
journals.
None of these claims hold up under careful scrutiny.
• Ubiquity of power laws
• Coherent structures in shear
flow turbulence
• Macro dissipation and
irreversibility vs. micro
reversibility.
• Quantum entanglement,
measurement, and the
QM/Classical transition
• Growing group of physicists
and experimentalists are
enabling this effort (Carlson,
Mabuchi, Doherty, Gharib,…)
Multiscale physics
Rich new “unifying
theory” of complex
control, communication,
and computing systems
is resolving persistent
mysteries at the
foundations of physics
NSF reviewer concerns
• Narrow focus on TCP/AQM
– WAN in Lab absolutely critical for FAST
– Other Caltech projects: existing & new
– External projects, e.g., in Emulab community, HENP, Grid
communities
• WAN in Lab valuable for theory research on entire
protocol stack, not just TCP/AQM.
NSF review criteria
• Intellectual merit
– Theory, implementation, experiment, deployment must
inform and influence each other intimately
– Experimental testbed tied to rich theoretical research
program
• Broader impacts
– Internet as a complex system
– New community of network researchers
Broader impact (cont.)
• Integration of research & education
– Proposed new course on unified theory of complex
systems (Doyle, CDS and BioEng)
– Shortcourse (with Walter Willinger) at 2003 Sigcomm
– Other shortcourses in 2003
• Diversity (Doyle’s group)
–
–
–
–
–
2 of 4 postdocs are women
6 of 11 graduate students are women
Wide ethnic and racial diversity
Incoming 2003 class even more diverse
Nearly all co-authored papers have female co-author(s)