Transcript On the Accuracy of MANET Simulators

On the Accuracy of MANET
Simulators
David Cavin
Yoav Sasson
& André Schiper
Presented by
Michael W. Totaro
Mobile Computing and Wireless Systems
(MoCWiS) group
UL Lafayette - CACS
Topics








Overview
Introduction
Related Work
Flooding Algorithm
The Simulators
Simulations
Conclusions
Q&A
Overview



The simulation phase of MANET applications
or protocol deployment requires meaningful
simulation results
The model on which the simulator is based
should match as closely as possible to reality
Simulation results of a straightforward
algorithm using several popular simulators
are presented, whereby significant
divergences exist between the simulators
Introduction

Context
–
–
–
Interest in MANETs (Mobile Ad-hoc Networks)
requires adaptation of solutions from the
traditional wired networks to the wireless
environment
Simulation is a tool that can often help to improve
or validate protocols
Generally speaking, all simulators provide a
complete toolkit to developers that facilitates
metrics collection and various wireless network
diagnostics
Introduction (2)

Accuracy of simulation results
–
–
–
Popular simulators such as NS-2, OPNET
Modeler, and GloMoSim provide advanced
simulation environments to test and debug
networking protocols, including wireless
applications
It is essential that the simulated behaviors
match as closely as possible the reality
This latter requirement assumes that several
issues are sufficiently addressed
Introduction (3)

Accuracy of simulation results (cont’d)
–
First Issue



Application is likely to rely on components such as a
collision detection module, as well as radio propagation
or MAC protocols
Correct definitions of these components in the simulator
is critical
Typically, the algorithm being evaluated is modeled in
detail; however, cross-layer interactions are very rarely
taken into account
Introduction (4)

Accuracy of simulation results (cont’d)
–
Second Issue


Simulation parameters and the environment (e.g.,
mobility schemes, power ranges, connectivity) must be
realistic
Incorrect initial conditions may lead to unexpected
results that are not realizable in a real network
Introduction (5)

Accuracy of simulation results (cont’d)
–
Focus of research


The research presented in this paper shows the results
of a set of measures collected during the simulation of a
flooding algorithm on three different simulators: OPNET,
NS-2, and GloMoSim
Special attention was given to setting the same
parameters and considering the same scenarios in each
simulator; nevertheless, very different results—barely
compatible—were collected
Related Work


The research literature offers an abundance of
papers on the efficiency of wireless algorithms
comparing relative performances of each by means
of simulation
Few of these papers, however, focus on possible
divergences that may occur between simulators,
probably because the researchers work with only a
single simulator—one with which they are most
familiar—and thus do not expect nor anticipate
significant differences among simulators
Related Work (2)

The physical layer and the important
parameters that influence its behavior have
been modeled in NS-2 and OPNET
–
Results suggest that the configuration affects
seriously the absolute performance of a protocol,
and can even change the relative ranking among
protocols for the same scenario
Related Work (3)

The effect of detail in MANET simulations
has been studied
–
Appropriate levels of detail in simulation models
for radio propagation and energy consumption
remain questionable


Simulations that are too detailed may not be easily
adapted to expeditiously explore alternatives
Conversely, simulations that lack detail can lead to
misleading or incorrect results
Flooding Algorithm

Introduction
–
–
A frequently used operation to spread information
to the whole network is the broadcast of
messages
The performance of the broadcast is likely to
affect the global efficiency of any protocol using it;
hence, the broadcast should be implemented in
the most efficient way
Flooding Algorithm (2)

Introduction
–
Simulations



Peer-to-peer wireless network, roughly 50 nodes
randomly placed on a 1km x 1km area
Ad-hoc mode, without any central access point
(infrastructureless)
Every node (peer) has the same possibilities and
functionalities
Flooding Algorithm (3)

Flooding
–
Flooding a message over the network is a simple
way to implement broadcast



Node initiates a broadcast
Message is transmitted to its neighborhood (i.e., all
nodes within the sender’s transmission range)
When the message is received by a recipient for the first
time, the recipient re-broadcasts it
Flooding Algorithm (4)
Flooding example
Flooding Algorithm (5)


Drawback of Flooding → overhead of flooded
messages in the network
Under ideal conditions (i.e., all nodes received the
broadcast) in a network of N nodes, a single
broadcast will generate exactly N copies of itself
–
–
Likely to increase probability of collisions
Most nodes will receive the same message several times,
thus keeping the shared medium unnecessarily busy
Flooding Algorithm (6)
Architecture
Assume that
every
message
has unique
ID
Algorithm protocol
stack
The Simulators

Introduction
–
–
The way a new algorithm is integrated can be
considerably different from one simulator to
another
A summary of the different implementation
approaches for each simulator is presented, along
with particular requirements and challenges
The Simulators (2)

OPNET Modeler
–
–
Can simulate many kinds of wired networks, and a 802.11
compliant MAC layer implementation is also provided
Phases of OPNET deployment process
1.
2.
3.
Choose and configure node models to use in simulations—for
example, a wireless node, a workstation, a router, and so on
Build and organize network by connecting the different
entities
Select the statistics you wish to collect during simulations
The Simulators (3)

OPNET Modeler (cont’d)
–
In this experiment, the authors created a new node model
which encapsulates 802.11 MAC layer of OPNET, as well
as an application process that implements the flooding
algorithm



Flooding algorithm process model is described as a state
machine, whereby each state has code that is executed upon
state activation
A transition that links two states is followed whenever a
certain condition carried by the transition is true
Difficulty with OPNET is actually building the state machine
for each level of the protocol stack
The Simulators (4)

NS-2
–
–
–
Discrete event network simulator that supports both wired
and wireless networks, including most MANET routing
protocols as well as an 802.11 MAC layer implementation
Source code is split between C++ for its core engine, and
OTcl, an object-oriented version of PCL for configuration
and simulation scripts
Implementation and simulation steps
1.
2.
3.
4.
Implement the protocol by adding a combination of C++ and
OTcl code to NS-2’s source base
Describe the simulation in an OTcl script
Run the simulation
Analyze the generated trace file
The Simulators (5)

NS-2 (cont’d)
–
In this experiment, the authors adapted the
implementation of flooding provided in NS-2


An Agent (which, in NS-2, represents an endpoint
where packets are constructed, processed, or
consumed) was implemented at the Application layer
for the broadcast source, and the simulation trace was
collected at the MAC layer
Major challenges with NS-2 include: a substantial
learning curve; difficult debugging; a large memory
footprint; and, a lack of scalability
The Simulators (6)

GloMoSim
–
–
–
Scalable simulation environment for wireless and wired
networks, developed initially at UCLA Computing
Laboratory
Provides various applications (CBR, ftp, telnet), transport
protocols (tcp, udp), routing protocols (AODV, flooding),
and mobility schemes (random waypoint, random drunken)
User must define specific scenarios in text configuration
files


app file—contains description of traffic to generate (e.g., app
type, bit rate, and so on)
Config file—contains description of other (remaining)
parameters
The Simulators (3)

GloMoSim (cont’d)
–
–
Statistics collected can be either textual or
graphical
According to the authors, compared to OPNET,
GloMoSim’s architecture is much less flexible
Simulations
Static parameters
Common constant parameters
of the simulations
Varying parameters
Varying parameters that
describe the behavior of
an ad-hoc network and
that can be set in a
controlled way
Simulations (2)

Metrics
–
First metric gives information about the time needed to flood a
message

–
Second metric measures the general efficiency of the algorithm

–
Time delay: For a node n, this is the average time needed for one
packet to reach n
Success rate: For a node n, this is the difference between the
expected and the actual number of messages received at n
Third metric stores the overhead of messages that are
unnecessarily flooded in the network

Overhead: For a node n, this is the sum of duplicated packets
received by n
Simulations (3)

Results
–
–
–
Only the most striking graphs are provided in the
paper
Several scenarios are defined by varying one or
more parameters from the previous table labeled
“Varying parameters”
For each scenario, the set of varied parameters is
given in the table just above the graph
Simulations (4)
Success rate vs. Power range


This scenario depicts
a critical factor that
influences the
success rate in
MANETs: the
effective transmission
range
OPNET
NS-2
GloMoSim
Notice the apparent
differences in trend
between the
simulators
Success rate vs. Power range
Simulations (5)
Success rate vs. Mobility


This scenario
evaluates the effects
of node mobility on the
flooding’s ability to
deliver packets reliably
GloMoSim
OPNET
NS-2
Again, we see a
significant difference in
success rate
Success rate vs. Mobility
Simulations (6)
Overhead vs. Mobility


This scenario presents
the average overhead of
messages flooded in the
network for a single
simulation run
This metric is related to
the mean number of
reachable neighbors
(that is, within
transmission range
OPNET
NS-2
GloMoSim
Overhead vs. Mobility
Simulations (7)
Time delay vs. Mobility

The final scenario
compares average time
delay needed to flood a
message throughout
the whole network
OPNET

This metric increases
with the number of hops
from source to
destination and also
whenever collisions
occur
GloMoSim
NS-2
Time delay vs. Mobility
Simulations (8)

Analysis and interpretation
–
–
Simulation results of the flooding algorithm demonstrate that
modeling of the MAC protocol and of the physical layer can
lead to different results, depending upon the simulator
Possible reasons



Differing physical layer implementations
Implementation of a new protocol is itself difficult to transpose
from one simulator to another
Given that successive releases provide bug fixes, it is
reasonable to assume that MANET simulators still contain
errors or incompatibilities to IEEE 802.11 standard
Conclusions (Authors)


Instead of simulations, a more realistic scheme
might entail a hybrid approach in which only the
lowest layers—MAC and physical—and the mobility
model are simulated and all the upper layers (from
transport to application) are executed on a cluster of
machines
There is an important lack of real experiments the
prove the feasibility of wireless protocols
Questions?
?