Transcript Information in Wargaming Concept

Modeling the effects of International Interventions
with Nexus Network Learner
Dr. Deborah Duong
Agent Based Learning Systems
AAAI12 Spring Symposium, Stanford University
Purpose and Agenda
Purpose: To show the benefits of Symbolic Interactionist
Simulation for the Simulation of Social Aggregation
Agenda
• Coevolution and Symbolic Interactionist
Simulation
• The Sociological Dynamical System
Simulation
• SISTER: Symbolic Interactionist Simulation
of Trade and Emergent Roles
• Nexus Network Learner
• Modeling Corruption
Social Impact Model
2
Coevolving Agents
• Genetic Algorithms, Neural Networks, or Reinforcement Learning
Algorithms in Agents can co-evolve
• Agents learn to optimize a function in an environment composed
mostly of agents also learning to optimize a function.
• Moving fitness landscapes
– Agents apply selective pressure upon each other.
– Selective pressure causes the optimal way to achieve the goal to
change over time.
• Evolutionary Stable Strategies
– Maynard-Smith’s theory that a co-evolutionary system
converges when no species (or inducing agent) can make a
change that will make it better off (Nash Equilibrium)
• Agents Differentiate into a System
– Species (or inducing agents) can be both cooperative and
competitive
Social Impact Model
3
Symbolic Interactionist Simulation
• Coevolutionary reinforcement learning algorithm.
• Autonomous agents each have an inductive mechanism.
– For example, an entire Genetic Algorithm or Neural
Network.
– Agents only experience through their senses (no direct
knowledge transfers from other agent minds).
• Agents choose to interact with other agents based on
signs that the other agents display.
– Agents induce both the signs to display and the signs that
they read.
• Social system emerges
– Signs come to mean behaviors.
– Behaviors interlock into a system of expectations.
– Value function becomes reward function as society
evolves (Adam Smith’s invisible hand).
Social Impact Model
4
The First Symbolic Interactionist Simulation: The
Sociological Dynamical System Simulation
• A System of IAC Networks as the Basis for Self Organization
in a Sociological Dynamical System Simulation
- Duong, 1991; Duong and Reilly, Behavioral Science 1995.
• Employer agents hire from a pool of employee agents, periodically laying
off employees.
• Employees who are less talented are laid off in greater proportions than
those who are more talented.
• Employees seek employment.
• Employees can choose to wear three different kinds of signs
– One of two signs they can not change. This is their “skin color”.
– One of three signs they have to pay for with money from
employment. This is their “suit”.
– One of three signs they can change arbitrarily. This is their “fad”.
• Employees induce what sign they should wear based on what has gotten
them employed in the past.
Social Impact Model
5
The Sociological Dynamical System Simulation
(SDSS)
• Employees have a hidden, unchanging, talent level that
employers can not see until after the employee is employed.
• Employers seek talented employees.
• Employers hire based on the signs the employees wear,
inducing how much talent they have from their signs.
• The two “races” of employees are equally talented, but the
employers do not know this.
• Employers and Employee Agents both have the same kind of
induction mechanism.
– Each Agent has their own Interactive Activation and
Competition (IAC) Neural Network to induce the signs they
read and display.
Social Impact Model
6
Interactive Activation and Competition
• The IAC is a Constraint
Satisfaction (Hopfield) Neural
Network.
• Nodes within each pool are
mutually inhibited.
• A central pool contains memory
instances.
• Each instance has positive
connections to its features.
• To guess the talent of an
applicant, the employer “turns
on” the applicants features and
sees which talent node turns
on.
• Simulates schema, or mental
groupings of features that go
together
• An Employer’s IAC
Social Impact Model
7
SDSS: Emergent Phenomena
• Status Symbols
– Employee agents learn to buy expensive suits and Employer
agents learn to seek expensive suits.
– Because the less talented get laid off more, they have less
money, and talented employees learn to differentiate
themselves.
• Racism and Social Class
– One race gets into a vicious cycle: because of schema, by
accident one race gets associated with less talent (even though
they are equally talented).
– Many talented in a race could not afford suits because they were
never given opportunity.
– One race would have less money than the others as a result
• Meaning attributed to Fads
Social Impact Model
8
SISTER: Symbolic Interactionist Simulation of
Trade and Emergent Roles
“SISTER: A Symbolic Interactionist Simulation of Trade and Emergent
Roles” - Duong 1995, Duong and Grefenstette, JASSS 1/2005.
• Trade is good for “farmer” agents.
– Agents need each of four food groups, and as much as they
can get of each.
– Agents can make more food if they concentrate their efforts on
fewer of them.
• Agents have efforts to spend on making or trading food as
they wish.
• Agents can trade if they have corresponding trade plans.
• Agents have a sign to display to attract trade.
• To learn to trade, agents induce what signs to display and
what signs to trade with based on whatever gets them the
most of each food in the four food groups at the end of the
day.
Social Impact Model
9
Coevolving Genetic Algorithms
• Each agent has an entire
genetic algorithm that
tells it:
– Where to place efforts
– What sign to trade with,
what and how much to
trade
– What sign to display
Social Impact Model
10
SISTER: Emergent Phenomena
• Division of Labor
• As agents learn to trade,
their utility increases, and
their sign comes to mean
a role
• Price
• Goods become valued at
standard ratios
• Money
• In a third of the runs, one
good is traded for the
purpose of trading again
• Different types of stores
• Central bargain stores
and local convenience
stores
Social Impact Model
11
SISTER: Results
Social Impact Model
12
Nexus
• SISTER is a “theoretical” symbolic interactionist
simulation
– SISTER embodies the formation of social patterns of
behavior
– Its scenarios are not realistic
• Nexus is a “data-driven” symbolic interactionist
simulation
– In order to do analysis, we must start from a scenario in
the real world.
– Being realistic and theoretically correct at the same time
is difficult.
– Nexus attempts to mirror the virtuous and vicious cycles
of the real world that created its input data.
Social Impact Model
13
What is Nexus Network Learner ?
• One of the two Nexus Cognitive Agent models that Debbie
Duong wrote at the OSD/CAPE/Simulation Analysis Center.
– Nexus Network Learner models Social Role Networks
– Nexus Schema Learner models Cognitive Dissonance
• A Simulation of Social Role Networks in which Agents learn:
– To choose role partners to perform transactions with:
- Choice based on signs, social markers and communications
on past transaction behaviors.
– Transaction behaviors and signs.
- Choice based on signs and social markers.
– Based on Cultural Values.
• Social markers, roles, transaction behaviors, signs, rolebased communications and cultural values are all input to
the program.
– Population data determines the initial population tendencies.
– Utility and motivation determines how they change.
14
How Does Nexus Network Learner Work?
• Artificial Intelligence Technologies represent Cognition.
– Rule Based.
- An ontology of roles with crisp rules for roles.
– Represents general social structures, that can be used in many
scenarios.
– Defines utility of transactions.
– Machine Learning.
- Bayesian networks initialize social markers , signs/transaction
behaviors, and role choice behaviors.
- The Bayesian Optimization Algorithm (BOA) changes those
behaviors based on the utility of transactions.
– BOA can be seeded with initialization data and injected data.
– A form of Evolutionary Computation using reinforcement
Learning optimizes (satisfices) utility.
– As conditionals change, the equilibrium point moves (in accord
with the New Institutional Economics).
15
What Happens in Nexus Network Learner?
• Individual Agents Choose Network Partners.
– Ontology tells who may choose and how many.
- Example: an “Employer” may choose an average of 5 employees with a standard
deviation of three.
– Bayesian network tells how the choices are ranked.
– Passive role may have an option to reject offer.
- Example: an “Employee” may reject an employer because a role relation has told
her he steals paychecks.
– Ontology may include a chance occurrence of natural attrition.
• Individual Agents engage in transactions.
– Account distributions send funds through networks according to rules in
ontology and transaction behaviors in Bayesian networks.
– Probability of observing, reporting, and knowing about behaviors are role-based.
– Agents may go to jail, and not be allowed to participate in transactions for a time.
• Every N cycles, they judge their learned strategies by utility based on
transactions that their valued role partners engaged in.
– Ontology determines culturally valued individuals and transactions.
– After testing all strategies agents recombine them.
16
Performing Tests with Nexus Network Learner
• A wide variety of tests relevant to Irregular Warfare (IW) may be
performed.
• For example, new network formations and behaviors may be tested
based on many different things…
– The effect of different utility functions.
- For example, make agents care only for self rather than larger social
network.
– The effect of different penalties.
- For example, a penalty attribute that encodes different jail terms or
different chances of getting caught.
– The effect of different exogenous resources.
- For example, test resource rents or foreign aid.
– The effect of different abilities to observe.
- For example, the effect of a media agent.
– The effect of removing different agents.
- For example, measure how long it takes to replace a terrorist leader
• Monte Carlo methods reveal if new structures are the result of different
CONOPS.
– Bayesian Networks make Nexus Stochastic
17
How Nexus Agents Learn
• As each agent learns, all the agents coevolve, making them very
adaptive.
– Every agent has its own private learning algorithm.
– Their behaviors effect the larger social structure and the larger
social structure effects their behaviors.
- Micro-Macro Integration is modeled.
– They can adapt to data from other simulations and to initial
country data as well.
• The learning algorithm in each agent makes the adaptation to
data flexible.
– BOA (Bayesian Optimization Algorithm) can start learning from
initial data.
- In the calibration phase. agents to adapt to initial data, so that
they generate it though their perceptions and motivations.
- Thus they “explain” the data, going from correlation to cause.
– This greater ability to ingest data also allows them to meld with
other simulations in a composition.
- Together, composed simulations create a coherent picture of
the social environment.
- Conflicts are resolved through mutual adaptation.
18
Use Case: Modeling Corruption with
Nexus Network Learner
Social Impact Model
19
Assumptions
•
•
•
•
A role perspective is appropriate for examining corruption
– With Nexus, we may explore how the patron-client role relationships in traditional
African societies interact with the bureaucratic relationships made necessary by
globalization
People adjust their behaviors based not only on policies but on other peoples reactions to
policies
– With Nexus, we can explore how agents adjust their behaviors to meet their cultural
goals, given that other agents are doing the same thing
Corruption is a social process, a vicious cycle
– People typically do not participate because they like it, but because they feel they
have to
– People take into account both legal penalties and social penalties in adjusting their
behavior
– With Nexus, we can explore the effects of legal penalties on eight specific corrupt
behaviors, and how they interact with social penalties
The ability to hide what you are doing (bounded knowledge) is important
– The chances that one person will know about another’s corrupt behaviors is based
on the role relation between the two
– The more people know about a corrupt behavior, the more likely the perpetrator is to
get caught
– With Nexus, we can explore the effects of transparency programs on the chances of
getting caught for a corrupt behavior, based on social relations
Perspective Orientation
• The foundation for the Nexus Network
Learner is built upon a rich literature in
social constructivism and social
emergence where methodological
individualism (only thing that matters
is an individual) is rejected.
• Nexus embraces the study of both
individuals and institutions:
endogenously (within the model)
modeling the institution-individual
linkage simultaneously with the
individual-institution links.
Institutions
Transform
Pressure
Individuals
[1] For an in-depth view see Bourdieu, P. (1977). Outline of a Theory of Practice. (R Nice, Trans.).
Cambridge: Cambridge University Press. (Original work published in 1972) and Giddens, A. (1984). The
Constitution of Society. Berkeley: University of California Press. Another potential source is Sawyer, RK.
(2005). Social Emergence: Societies as complex systems. Cambridge: Cambridge University Press.
Interpretive Social Science Used in Nexus
• From economics: The New Institutional Economics (NIE) (North)
– Institutions (Social and Legal Norms and Rules) underlie economic
activity and constitute economic incentive structures
– Institutions come from the efforts of agents to understand their
environment, so as to reduce uncertainty, given their limited perception
– When some uncertainties are reduced, others arise, causing economic
change
– To find the leverages to corruption, NIE would look at actor’s definition of
their environment, and how this changes incentives and thus institutions
• From sociology: Symbolic Interactionism (Mead)
– Roles and Role Relations (such as in trade roles and trade relations) are
learned, created during the display and interpretation of signs (such as
gender, ethnicity, and other demographic characteristics)
– Institutions (social and legal norms and rules) are commonly accepted
interpretations of symbols, that start out as a subjective perception and
engrained in society as an objective rule
[1] See http://coase.org/niereadinglist.htm for an extended reading list
[2] See Duong, Deborah Vakas, “The Generative Power of Signs: Tags for Cultural. Reproduction” Handbook of
Research on Agent-Based Societies: Social and Cultural Interactions, Goran Trajkovsky and Samuel Collins,
eds., 2008. http://www.scs.gmu.edu/~dduong/GenerativePowerOfSigns.pdf and also Blumer, Herbert (1969).
Symbolic Interactionism: Perspective and Method. Berkeley: University of California Press.
[3] Duong, Deborah Vakas and John Grefenstette. “SISTER: A Symbolic Interactionist Simulation of Trade and
Emergent Roles”. Journal of Artificial Societies and Social Simulation, January 2005.
http://jasss.soc.surrey.ac.uk/8/1/1.html.
Conceptual Model
• Nexus is a model of corruption based on the theory that corruption is a
result of globalization.
• Many social scientists assert that corruption is the result of conflict
between the roles and role relations of the kin network and the trade and
bureaucratic networks, separate social structures with their own
institutions forced together because of globalization.
Strategy (Behavior)
Kinship
Network
Trade
Network
Bureaucratic
Network
Transaction-based Utility
Incentives
[1] Smith, Daniel Jordan. 2007. A Culture of Corruption. Princeton: Princeton University Press.
Nexus Main Components
• Nexus models individuals and
their interactions.
• Individuals have various roles on
the three different networks and
dynamically interact with other
agents through these roles.
– Retailer – Customer
– Government Employer – Government
Employee
– Head Of Household– Dependent
• Role Networks are Input to Nexus
• Only commodity is Money
– Agents pass money to other agents’
accounts
– External support is in the form of
injections of funds to certain individuals
(that have certain roles)
– Utility of agents (their “happiness”) is
raised when they spend the money on
things they need
Institutions
Emerged Learned Attributes
External Control
User-defined Policies
Role Interactions
Kinship
Network
Trade
Network
Bureaucratic
Network
Individual
Determined Traits
Situational Traits
Behavioral Traits
Transaction-based Utility
Experience
Cognition
External
Support
Nexus Main Components: Individuals
• They want their kin to be happy, and can think about how to adjust their behaviors
towards that goal, based on experience of what met that goal in the past
• They have the demographic characteristics, both determined and situational, of the
modeled country
– Determined Traits: Gender, Ethnicity, Age, etc.
– Behavioral Traits: Tendency to Steal or Bribe, based on other traits and on
learning during the run
– Situational Traits: Employment, Are they under penalty, etc.
• They actively seek role relationships, following socio-cultural rules about who
proposes the relationship, what sort of person is chosen
– For example, a husband chooses a wife, or an employer chooses an employee
– They judge others based on characteristics they can see or they have heard
rumors about
• There role responsibilities include the distribution of funds to accounts they are
responsible for
• They are happy when funds flow through certain accounts, for example, from the
household budget to the grocery store income.
• They have differing length legal penalties, as well as social stigma
Nexus Main Components: Role and Role Relations
• There are eight types of corruption relations possible in the three networks
(example actions provided):
–
–
–
–
–
–
–
–
Nepotism: Hiring Kin/ Trade Network
Commission for Illicit Services: Bribing/Government Network
Misappropriation: Stealing/ Trade or Gov Network
Rig Election: Elected Officials bribing for Employment
Gratuity: Bribing/ Trade Network
Unwarranted Payment: Accepting Bribes/Government Network
Levy Toll Sidelining: Stealing/Government Network
Scam: Stealing From Customer in Trade Sector.
• There are many other types of role and role relations (64) in the model:
– Each role has a corresponding role
– Roles are dynamic (such as an agent can move from a government employee to unemployed)
Bureaucratic Network
Gov Employee Attempts to Bribe Gov Employer
Accepts/Rejects
Bribe
Observer
Nexus Main Components: Kinship Network
Active Roles
Corresponding Passive Role
Father
Child
Head Of Household
Home Receiver
Husband
Wife
• Three main active roles, from
which 14 more are derived
• Derived Roles are used to model
Residence
– Utility (Satisfaction)
calculated based on
Residence in Anthropology
– Matrilineal, Patrilineal of
Neolocal
• Support account goes from
Provider to Dependent
Derived Roles
Brother
Dependent (Provider)
MaternalAunt
MaternalCousin
MaternalGrandparent
Mother
Parent
PaternalCousin
PaternalGrandparent
PaternalUncle
Sibling
Sister
Spouse
Provider (Dependent)
Nexus Main Components: Trade Network
Active Roles
Corresponding Passive Role
HeadOfCorporation
CorporateReceiver
Customer
Retailer
Employer
Employee
Purchaser
Vendor
Service Providee
Service Provider
• Derived Roles are ten different income levels
• Accounts include personal salary, employee salaries, money for
office purchases
Nexus Main Components:
Bureaucratic Network
Active Roles
Corresponding Passive Role
Taxpayer
Taxman
GovernmentEmployer
GovernmentEmployee
GovernmentPurchaser
GovernmentVendor
HeadOfGovernment
GovernmentReceiver
Service Providee
Service Provider
• Derived Roles are ten different pay grades
• Accounts include corporate and income taxes, government salaries,
government office money, government money for purchases
Lets talk about the Agents
• Agents are able to learn and adapt to new role behaviors through the use of
evolutionary computation techniques of artificial intelligence, also known as
genetic algorithms. (cognitive agents)
• They learn other behaviors based on utility.
– Utility is in the trade interactions (transaction-based utility) of the themselves and the kin that an
agent cares about.
– Agents learn how to navigate their environment according to their individual traits and experience
through their own Bayesian Optimization Algorithm.
– Agents learn the type of persons to include in their social network, including kinship, ethnicity,
and bribing behavior.
– They also learn whether to divert funds across networks through bribing and stealing. Agents
have money in accounts (which is a situational attribute)
• Corruption behavior changes through synchronous individual interaction, driven
by new incentive structures created from government policies, agent’s reactions to
those policies, and agent interaction.
Traits
Determined
Situational
Behavior
Utility
Cognitive
Mechanism
Experience
Behavior
Computational Model
• Nexus was created with the REPAST
Simphony, a free and open source
agent-based modeling toolkit
• REPAST makes use of the social
network software Jung, shown below
– Dots along the circle are agents’
– Different colored arrows
represent relations of different
networks
- Bureaucratic is Blue, Trade
is Green, Kin is Red
• All the analysis functions of Jung and
data mining software Weka are
available for Nexus
– Weka displays number of agents
that did each type
– Jung can describe characteristics
of the network like centrality,
reach, etc.
- With Jung, you can tell who
are the important actors
So what happens first? (Initialization)
Demographic Data
• Demographic Data Input
Application of Bayesian
Network Algorithm
Initial Population
Representation
– The input to Nexus are the demographic characteristics for an entire population.
- Physical characteristics, social categories, and behavioral traits that are based on these
physical characteristics and social categories.
- Variables we are trying to explain are used to calibrate the simulation in the beginning.
- Example: We know that a subset of the population in this region who have characteristics
A and B, have a greater propensity of corruption than you would expect by chance.
• Application of Bayesian Network Algorithm (Data Interpretation)
• Initial Population Representation
– Describes characteristics that agents cannot change, for example, social markers such as
ethnicity or gender. (Determined Traits)
– Describes characteristics that agents can change on an individual basis during the simulation, for
example, behavioral characteristics, such as bribing or stealing, or preferences for choices of
others in social networks. (Situational and Behavioral Traits)
– Describes demographic characteristics which individual agents do not learn, but are rather the
output of the computations made during the simulation, such as unemployment statistics.
(Aggregated)
Application of Bayesian Network Algorithm
• A Bayesian network is a graphical model that encodes probabilistic relationships
among variables of interest. When used in conjunction with statistical techniques,
the graphical model has several advantages for data analysis:
– Because the model encodes dependencies among all variables, it readily handles situations
where some data entries are missing.
– A Bayesian network can be used to learn causal relationships, and hence can be used to gain
understanding about a problem domain and to predict the consequences of intervention.
– Because the model has both a causal and probabilistic semantics, it is an ideal representation for
combining prior knowledge (which often comes in causal form) and data.
– Bayesian statistical methods in conjunction with Bayesian networks offer an efficient and
principled approach for avoiding the over-fitting of data.
– Methods to construct Bayesian networks include using prior knowledge and implementing the
Bayesian statistical methods for using data to improve the models. This includes both
parameters and the structure of a Bayesian network, and techniques for learning with incomplete
data.
Real World Data
Example: % of Ethnic Population in a
Region
Bayesian Logic
Initial Population
Representation
Observed Correlations
Example: We know bribing employers is
common practice in Ethnic Group A
Source: Neural Network Learning and Expert Systems, Stephen I Gallant, MIT Press, Cambridge, MA,
1993
How does the Cognitive Mechanism work?
• In this model, the cognitive mechanism is the Bayesian optimization algorithm
(BOA).
• The output of this algorithm is an agent’s strategy (how to distribute funds to
maximize utility – network choices and behaviors)
• Recall at model initialization, the raw data is converted by a Bayesian network
into model input. Among other things, this process generates the individual
agent traits.
– Determined (or fixed traits) are set per agent for the simulation run.
– There are also two categories of behavioral traits (or learned behaviors):
- Learned network choice behaviors (Initialized Random)
– You choose a network partner when you choose a wife or
employee
– You may choose a wife or employee based on ethnicity,
– You may choose an employee based on whether he bribes you
– You may reject an employer if your kin tells you he steals
- Simple learned behaviors (set by Bayesian network during
initialization)
– You may learn not to steal even if you started out that way, if it
harms your dependents
So what happens when you hit go? (Simulation)
• Remember that Agents are initialized with money in their accounts.
– When the money reaches a certain threshold they will distribute it.
- Distribution includes trade, for example, you distribute your money to a
grocery store, and gain utility for yourself in doing so
– Exogenous funds are then pumped in on a regular basis (if there
are any like diamond revenue)
- Wages, Taxes, Accounts Receivable are internal
• All those with active roles seek relationships. (an active govt employee
will seek a new govt employer)
– Both active and passive agents have thresholds for initiating or
accepting a partnership.
– There are distance thresholds associated with the role choice, that
make them have to match by a certain amount or no partner will be
chosen.
• As required, agents update their strategies (based on experience, traits,
and evaluation of past strategies) to maximize utility.
Experiment
Social Impact Model
36
Experiment: Stuck in Stealing Mode
• Comparison of the evolution of a society which initiates in a strong
vicious cycle of stealing to one with more moderate levels of stealing.
• If they started out stealing excessively, they never learned not to, never
attempted to find service providers who wouldn’t steal from them
• If the stealing is in more moderate amounts, agents learn to find service
providers that do not steal from them within two years
• Agents in a stealing vicious cycle never use bribing to accomplish
goals
• Agents with moderate stealing used bribes, but after fifteen years,
employers and service providers stopped bribing
• Convergence occurs at the fifteen year mark
• After 15 years in both the excessive stealing scenar-io and the
‘normal’ stealing scenario, we see ho-mogeneous responses
across strategies. For in-stance, agents generally provided the
same re-sponse for a particular parameter (say ‘Bribe-ForServices’)
after 15 years as opposed to more heterogeneous responses for
the same parameter after two years.
• Implication: Diversity of Behavior is needed for flexibility
37
Summary
Social Impact Model
38
Symbolic Interactionist Simulation
• Symbolic Interactionist Simulation is a form of Reinforcement
Learning by Coevolution.
• Agents learn associated rules in the form of actions to take with
other agents based on signs displayed and read.
• Symbolic Interactionist Simulation can be Theoretical (SISTER)
or Data-Driven (Nexus).
• Symbolic Interactionist Simulation can model the motivation
based vicious and virtuous cycles of behavior that determine
social structure
39
Questions and Comments
POC: Deborah Duong
[email protected]
Social Impact Model
40