Transcript Human Systems Dynamics Theory Applied to Evaluation Practice

Human Systems Dynamics Theory
Applied to Evaluation Practice
American Evaluation Association
2008
Beverly Parsons, Ph.D.
InSites
[email protected]
Meg Hargreaves, Ph.D.
Mathematica Policy Research, Inc.
[email protected]
1
2
3
4
Introduction to a Systems
Perspective In Evaluation
This section presents:
•
•
•
•
•
System definitions
System features
System characteristics
Types of systems
Examples of types
5
Systems Definitions
Multiple definitions:
• A group of interacting, interrelated, or
interdependent parts forming a complex
whole
• A configuration of parts joined together
by a web of relationships
• The parts form a whole, which is greater
than the sum of its parts
6
System Features
Systems are as much an “idea” about the
real world as a physical description of it:
• Boundaries define who or what lies inside
or outside the system
• Differences among the parts influence the
system’s dynamics
• Relationships among parts, between parts
and whole, and between whole and its
environment, are key focus of systems
7
System Characteristics
Common patterns, behaviors, and properties:
• Patterns – unorganized, organized, or organic
(self-organized)
• Behaviors – random, simple, complicated, or
complex adaptive; linear or nonlinear
• Properties – independent, interrelated, or
interdependent relationships
• Scale – small to large, self-similarity across
levels (fractals)
8
System Types
Systems can be grouped by their level of
complexity or organization:
• Random (no system) - unorganized
• Simple system - organized
• Complicated system – organized
• Complex adaptive system – organic
9
Random (Unorganized)
• Random, chaotic activity – no pattern
• Independent, unconnected parts
• No cause-effect relationships –
constant chaos and surprise
• Turbulence - no equilibrium
• Random parts without a system
• No leadership - people react blindly
• Unknowable
10
Random System Examples
• War zone: Civilians caught in crossfire,
random flight to escape conflict
• Natural disaster: At landfall or in the
eye of the storm, residents react
instinctively to events
• Leadership transitions: During changes
in administration old patterns are
suspended before new patterns are
established
11
Simple System (Organized)
•
•
•
•
•
•
•
Stable, static pattern
Parts connected in linear ways
Predictable cause-effect relationships
Set equilibrium
System reducible to parts and replicated
Directive leadership - designed change
Known knowns – answers are evident
12
Simple System Examples
• Baking a cake: Follow a recipe to assemble
and combine ingredients into a batter that
is baked at a pre-set temperature with
predictable results
• Flu shot clinics: Nurses use consistent
procedures to administer the same shots to
each person, following a set protocol in
assembly-line fashion
13
Complicated (Organized)
•
•
•
•
•
•
•
Dynamic pattern of feedback loops
Many interrelated parts across subsystems, levels
Complex, nonlinear cause-effect relationships
Feedback can stabilize equilibrium – thermostat
System can be reduced to parts and replicated
Multiple answers – investigate options
Unknowns become known through expert analysis
at multiple levels
14
Complicated System
Examples
• Space Shuttle Challenger disintegrated (1986)
when O-ring failure caused a rocket booster
breach, creating flare that damaged external fuel
tank, spilling fuel that exploded
• In large healthcare institutions, human behaviors
are part of wider systems of causality, in which
medical errors can occur in organizational and
policy contexts that result in long (36-hour) shifts,
large caseloads, and strained staff relations
15
Complex Adaptive System
(Organic)
• Dynamical patterns – parts adapting to each
other and to environment as a whole
• Parts are massively entangled, interdependent
• Parts self-organize, learn, coevolve organically
• Equilibrium in flux - sensitive to initial
conditions
• System not replicable, can’t repeat past
• Emergent change – manage conditions of
organic development and experimentation
• Unknown unknowns – trial and error
16
Complex Adaptive System
Examples
• Economic system – interactions of
homeowners, mortgage lenders, stock
market traders, investors, federal banking
institutions, and worried consumers are
coevolving into global crisis and recession,
despite governments’ interventions
• User networks (Diabetes, AA) facilitate
exchange of information and advice on care
for chronic conditions among participants,
learning from each other
17
Background about
Systems Theories
This section presents:
• General systems theory
• Cybernetics – systems dynamics
• Complex adaptive systems
• Implications for evaluation
18
General Systems Theory
• Holistic change ideas – ancient Greeks
• General systems theory - von Bertalanffy
(1930’s); earliest work by Bogdanov (1910)
• Open systems – nonrandom elements
organized into interacting, interrelated
components that seek to survive through
interactions with environment
• Each system level nested in higher level
(cells, organisms, families, organizations,
communities, societies)
19
Implications for Evaluation
• The whole can enable/constrain parts and
the parts can contribute to and/or challenge
stability of the whole
• Because open systems are structured in
hierarchies; useful to look one level above
and one level below the ‘system in focus’
• Evaluate system viability – does system have
both the parts and the information and
decision flows among the parts that are
needed to survive?
20
Cybernetics and
System Dynamics
• System dynamics founded by Forrester at MIT
(1950’s) for electrical engineering
• Method for calculating and modeling how
many circular, interlocking, sometimes timedelayed relationships among parts are
important in shaping system-wide behavior
• Through negative feedback, adjustments
made to keep system in balance; positive
feedback used to move system in same
direction, moving out of balance
21
Implications for Evaluation
• Assess influence of feedback loops on
behavior of system’s parts and on whole
• Behavior of whole not only explained by
behavior of parts (e.g. medical errors)
• Feedback loops undermine sustainability of
public interventions (policy resistance)
• Evaluators cannot step outside social and
ecological systems to observe (not valueneutral); self-reflection needed
22
Complex Adaptive Systems
• Key CAS writers – Weaver (1948), Simon
(1962), Prigogine (1987), Stacey (1997, 2007),
Zimmerman et al (2001), Eoyang (2006)
• CAS – many semi-independent and diverse
agents, who are free to act in unpredictable
ways, continually interact with each other,
adapting to each other and to environment as
a whole, creating system-wide patterns
• Key concepts – emergence, organic selforganization, co-evolution, simple rules
23
Implications for Evaluation
• Currently relevant evaluation criteria and
measures may need to be updated as new
conditions emerge
• Measure frequently for emerging patterns
• Avoid grand modeling projects for
prediction; use smaller projects for
ongoing experimentation and learning
• Also visualize system interactions as
networks; look outside nested levels for
system patterns, drivers, and constraints
• Ask what, so what, now what?
24
Three Dynamics of a Social
System and its Context
far from
agreement
C O N T E X T
Unorganized
dynamics
Agreement
organic
dynamics
(random
unpatterned
seemingly
chaotic)
(emerging patterns
coherent but not predictable)
close to
agreement
Organized dy
namics
(predictable
orderly
controlled)
far from
certainty
close to
certainty
Certainty
25
Match of Evaluation Designs to Dynamics
of Social Systems and Their Context
far from
agreement
CON T E X T
Exploratory
Design
Initiative Renewal
Design
unorganized dynamic
Organic
Design
close to
agreement
organic dynamic
Predictive
Design
organized dynamic
far from
certainty
close to
certainty
Certainty
26
Understanding Organic
Dynamics (Activity)
• Divide into triads
• Selects one other triad member (doesn’t tell)
and uninvolved person in refreshment area
• Stay at least two feet apart and equidistant from
the other two
• Do this for about 1-2 minutes while trying to
reach refreshments
• Reflect on experience
27
Case Study Introduction
Do the preconference professional
development offerings contribute to
effective evaluation-related work of
association members? If so, how?
28
Unorganized System
Dynamics
What is happening?
What boundaries, differences,
similarities, and relationships might shape
how the offerings contribute to
participants’ evaluation-related work?
29
Organized System Dynamics
Do participants receive high-quality
content that is relevant to their
evaluation-related work and is delivered
through high-quality instructional methods?
30
Organized System Dynamics
How do the format and content of the
session support or hinder participants in
understanding and using the session to
apply the learning from the session to their
evaluation work?
31
Organic System Dynamics
What patterns among participants
(including the session facilitators) before
and during the session are likely to affect
the participants’ understanding and
application of the learning to their
evaluation-related work?
32
Patterns
33
Patterns
34
Patterns
35
Patterns
36
Patterns
Centers for Medicare & Medicaid Services.
(2008). System and Impact Research and
Technical Assistance for CMS FY2005,
FY2006, and FY2007 RCSC Grants (2008).
[Annual Report]. Cambridge, MA: Abt
Associates, Inc. (p. 10)
37
Patterns
Centers for Medicare & Medicaid Services.
(2008). System and Impact Research and
Technical Assistance for CMS FY2005,
FY2006, and FY2007 RCSC Grants (2008).
[Annual Report]. Cambridge, MA: Abt
Associates, Inc. (p. 27)
38
Patterns
Centers for Medicare & Medicaid
Services. (2008). System and Impact
Research and Technical Assistance
for CMS FY2005, FY2006, and
FY2007 RCSC Grants (2008).
[Annual Report]. Cambridge, MA:
Abt Associates, Inc. (p.42)
39
Patterns
Centers for Medicare & Medicaid Services.
(2008). System and Impact Research and
Technical Assistance for CMS FY2005,
FY2006, and FY2007 RCSC Grants (2008).
[Annual Report]. Cambridge, MA: Abt
Associates, Inc. (p. 78)
40
Patterns
41
Fractals:
Patterns, Patterns Everywhere
In nature . . .
• Mathematical construct of iterating nonlinear equation
and plotting on complex number plane—Mandelbrot Set
• Similar shapes at all scales—Broccoli
• Biological scaling gives coherence in widely diverse
entities—Oak tree
• Scale-free networks
42
Fractals:
Patterns, Patterns
Everywhere
• Recognizing patterns is critical:
similarities, differences, and relationships
that have meaning across space and time
• Basic values or simple rules generate diverse, but selfsimilar behavior across scales
• Naming and telling stories about dynamics in a system help
reinforce and shape fractal patterns
43
Fractals
44
Looking at the Dynamics
as a Whole
• What is the overall picture of system dynamics
affecting how the preconference professional
development offerings contribute to effective
evaluation-related activities of AEA members?
• Given the findings from the three system
dynamics within the preconference session, how
might the preconference professional
development process and offerings be modified
to contribute more substantially to the quality of
AEA members’ evaluation-related work?
45