Transcript bas2003 3870

Modeling Adaptive Robot Swarms
Not an Inverse Problem
Kristina Lerman
Aram Galstyan
USC Information Sciences Institute
Why Analyze Adaptive Systems?

To predict


Behavior of the system under new conditions
Aspects of the system not studied experimentally
 Large numbers of robots
 Dynamic environments

To control

Find parameters that optimize system performance
 Optimal number of robots for the task


Find parameters that prevent instabilities, etc.
To understand


How individual robot characteristics affect collective behavior
How system size affects performance of the system
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Modeling and Analysis Tools

Microscopic (discrete) models
describe/model individual robot’s behavior
 Equations of motion approach
 Explicitly account for all interactions
 Solve to obtain trajectories
 E.g., pheromone-based trail formation

(Schweitzer et al., 1997)
Microscopic simulations
Abstract individual robot properties
 Probabilistic models (Martinoli et al., 1999)
 Robot’s actions modeled as series of stochastic events
 E.g., collaboration in robots (Ijspert et al., 2001)
 Others: cellular automata, molecular dynamics, etc.
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Modeling and Analysis Tools (cont)

Macroscopic (continuous) models
describe collective behavior of groups of robots
 Finite difference equations
 Collaboration in robots
 Synchronous

(Martinoli & Easton, 2003)
Rate Equations
 Continuous limit of finite difference equations
 Alternatively, derived from Stochastic Master Equation
 Collaboration (Lerman et al., 2001) and foraging (Lerman &
Galstyan, 2002) in robots
 Asynchronous
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Rate Equations

We will use macroscopic models – Rate Equations
– to study robots



Rate Equations describe average collective
behavior, not a specific experiment


More computationally efficient than microscopic models
Directly describe experimental observations
describe results averaged over many experiments
They are usually phenomenological


Not derived from first principles
Are easier to construct because you don’t have to explicitly
model many individual details
 Microscopic details appear only in parameters of models

Can be derived from stochastic microscopic description
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
What We Can Model Now

Types of robots modeled

Reactive robots
 Perception and action are tightly coupled
 No memory or use of historical information
 Can use timers to trigger actions

Adaptive robots
 Change their behavior in response to environmental changes or
actions of other robots

These are simple robots



No intentionality
No abstract representations
Distributed system

collective behavior arises out of local interactions among robots
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Why Amenable to Analysis
Individual robots can be modeled as stochastic
processes
Individual robot’s behavior subject to
 External forces


Noise



fluctuations and random events
Other robots with complex trajectories


may not be anticipated
Can’t predict which robots will interact
Errors in sensors and actuators
Randomness programmed into controllers

e.g., avoidance
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Our Results

A framework for modeling collective behavior in
reactive and adaptive robot systems



Derived equations from theory of stochastic processes
“Recipe” for creating application-specific models from individual
robot controllers
Models of distributed robot systems

Collaborative stick-pulling
 Qualitative agreement with experimental/simulations results
 Analytic form for critical parameters

Foraging
 Quantitative agreement with simulations results
 Optimal group size and its dependence on individual robot
parameter

Dynamic task allocation
 Individual robot transition rules for stable steady state distribution
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Adaptation in Robot Swarms
Adaptation is crucial for robotic swarms
functioning in unstructured dynamic
environments
Adaptation allows robots
 to change their behavior in response to
environmental changes or actions of other
robots
 to improve the overall performance of the
system
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Memory as a Mechanism for Adaptation
Memory-based adaptation



Robot makes observations; stores them in memory
Uses observations to estimate the state of the
environment and other robots
Modifies its behavior accordingly
Adaptive robots using memory of length m
to make decisions about future actions
can be represented as a generalized
Markov process of order m
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Adaptive Robots as Markov Processes


Individual robot probability distribution
 p(n,t) = probability robot is in state n at time t
Generalized Markov property: robot’s state at
time t+Dt depends on its state at time t, and
history of past m-1 states
 h={(n1,t), (n2,t-Dt),…, (nm,t-(m-1)Dt)}
p(n, t  Dt | h)   p(n, t  Dt | n, t ; h) p(n, t | h)
n

Change in probability density
Dp(n, t )  p(n, t  Dt )  p(n, t )
  [ p(n, t  Dt | h)  p(n, t | h)] p(h)
USC Information Sciences Institute
October 4, 2003
h
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Master Equation for Adaptive Systems

In the continuum limit, Dt  0
dp (n, t )
 W (n | n, t ; h) p(n, t | h) p(h)
dt
h n
 W (n | n, t ; h) p(n, t | h) p(h)
h n
with transition rates
p (n, t  Dt | n, t ; h)
W (n | n, t ; h)  lim
Dt
Dt 0
Similar to the Stochastic Master Equation that
describes evolution of physical systems
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
From One to Many

Collective robot state
If robots are independent and indistinguishable
 Collective configuration described by n={N1,…, NL}
 Nj = number of robots in state j
 P(n,t) = probability system is in configuration n at t

Master Equation for P(n,t) specifies how the entire
system evolves in time
dP(n, t )
   W (n | n, t ; h) P (n, t | h) P (h)
h n
dt
   W (n | n, t ; h) P (n, t | h) P (h)
h n

However, ME is often difficult to formulate and solve – it is
difficult to define correct probability distribution for a system
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Rate Equation
Instead, we work with the Rate Equation
 Derived from the Master Equation


“First moment” of the ME
Describes how the average number of robots in
state k changes in time
d N n (t )
  W (n | n; t )
n
dt
h
N n   W (n | n; t )
n
h
Nn
where <…>h denotes average over histories


No need to know exact probability distributions
Used in Ecology, Population Dynamics
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Dynamic Task Allocation
Jones & Mataric
USC Information Sciences Institute
October 4, 2003

Task

Goal

Mechanism
Robots are allocated to collect
red or green pucks
Dynamically achieve an
appropriate division of
labor
• Robot makes local
observations and adds
them to memory
• Each robot estimates the
proportion of pucks and
robots in the environment
(from memory), and
switches state accordingly
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Modeling Adaptive Task Allocation
dN R (t )
 a R (t , h) N G (t )  a G (t , h) N R (t )
dt
dM R (t )
  b R M R (t ) N R  m R  0
dt





NR|G(t) number of robots in Red|Green state
MR|G(t) number of Red|Green tasks
aR|G(t,h) rate robots switch to Red|Green state
bR|G rate robots complete Red|Green tasks
mR|G rate at which new Red|Green tasks are added
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Transition Rates

Transition rates depend on
nR,obs=NR,obs/(NR,obs+NG,obs) : observed density of Red robots
mR,obs= MR,obs/(MR,obs+MG,obs) : observed density of Red pucks

Mathematical form of transition rates


Steady-state (SS):
 dnR/dt=0=aR nG-aGnR
 Desired SS solution: nR,SS= mR
Intuitive guess: aR=f(nR-mR)
 Will lead to desired SS solution only when f(nR-mR)=0

Informed by analysis:
aR=(1- mR)f(nR-mR)
 guarantees desired steady-state is satisfied non-trivially
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Mathematical Form for f
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Intuitive vs Informed Transition Rates
Simulations results with different transition rates
Intuitive (power f ):
aR=f(nR-mR)
Informed (power f ):
aR=(1- mR)f(nR-mR)
time
USC Information Sciences Institute
October 4, 2003
Courtesy of C. Jones
ISI
Modeling Adaptive Robot Swarms
K. Lerman
History and Transition Rates
…
h
h
Robot’s nRn,obs
h
Rn,obs
h
memory mR,obs
hR ,obs
mR (t  3D )
nR (t  D )
mR (t  D )
…
…
…
nR (t  3D )
t
n1R,1obs nR0 ,0obs
n obs nR,0obs
1 Rn,1R
R ,obs
mR,1obs,obsmR0 ,n0obs
mR,1obs mR,0obs
mR,obs mR,obs
mR ,obs
h
mR ,obs
All
robots
t-D
…
t-|h|D
nR (t )
mR (t )
a R (t , h)  (1  mR (t , h)) f (nR (h)  mR (t , h))
1 h1
nR (t , h)   nR (t  iD )
h i 0
1 h1
mR (t , h)   mR (t  iD )
h i 0
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Time Delay Equations
dnR (t )
 a R (t , h)nG (t )  a G (t , h)nR (t )
dt
 1 h1

a R (t , h)  (1  mR ) f   nR (t  iD)  mR 
 h i 0



1 h1
a G (t , h)  mR f  mR   nR (t  iD) 
h i 0



Initial conditions
nR (t  0)  1, nG (t  0)  0
mR (t )  mR
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Dynamics of Red Robots
Linear f



USC Information Sciences Institute
October 4, 2003
Solutions show
oscillations
characteristic of
delay equations
Solutions
eventually relax to
puck distribution
Magnitude of
oscillations and
relaxation time
depend on size of
history window
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Solutions for Different f Same h
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Jones & Mataric
Conclusions

Created a model of collective dynamics based on
theory of stochastic processes



Applied formalism to distributed robotic systems




Reactive robots
Adaptive robots
Collaborative stick-pulling
Foraging
Dynamic task allocation
Results



Theoretical predictions agree at least qualitatively with results of
experiments and simulations
Analytic results not obtainable by other methods
Insights into robot design (form for transition rates)
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Future Work

Beyond the Rate Equation

Take into account fluctuations
 Noisy observations

Formulate and solve the collective Master Equation
 Appropriate form for probability distribution function for dynamic
task allocation application

More realistic models



Don’t coarse-grain behaviors
Automatic model construction
Other systems – new challenges


Self-reconfigurable robots
Nano-robots
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Roadmap to Theory

Starting with an individual robot


Derive stochastic Master Equation
ME describes how robot’s state changes in time
 State=action or behavior the robot is executing

Make transition to a multi-robot system

Derive collective Master Equation
describes how configuration of the system evolves in time


ME is often difficult to formulate and solve
Instead, work with the Rate Equation
 “Mean” or “First Moment” of the ME

Practical “recipe” for constructing the Rate Equation
from individual robot controller
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Representation of Reactive Robots

Finite state automata used to represent
individual reactive robots (Arbib et al., 1981)



State = behavior; transitions between states
Example: simplified foraging diagram
Collective behavior is captured by the same FSA


Each robot in exactly one of finite number of states
State = number of robots executing that behavior
start
searching
Reach
home
homing
USC Information Sciences Institute
October 4, 2003
Gripper
closed
Puck
detected
pickup
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Coarse-graining
search
Avoid
obstacle
search
Detect
object
search
Avoid
obstacle
• Coarse-graining reduces the complexity of the model
• Helps construct a minimal model that explains experiments
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
A “Recipe” for Rate Equations
start
searching
homing
pickup
dN s
  f s ( E , N s ) N s  f h ( h , N ) N h
dt
dN p

f s ( E , N s ) N s  f p ( p ) N p
dt
Nh  N  Ns  N p
Initial conditions: Ns(t=0)=N, Nh(0)=0, Np(0)=0
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman
Transition Rates

Transition is triggered

By a stimulus
 Obstacle, another robot in a particular state, location (e.g., home)

By a timer
 Turn in a random direction for x seconds

Calculating transition rates

Calculated under assumptions
 Triggers are uniformly distributed in space
 Robots encounter triggers randomly

Estimated from data by
 Calibration
 Run experiment or simulation for a single robot in an arbitrarily
complex environment and measure relevant parameters
 Fitting
 Fit the model to the data
USC Information Sciences Institute
October 4, 2003
ISI
Modeling Adaptive Robot Swarms
K. Lerman