Transcript Document

Evaluating Complex System
Interventions
Evaluation 2009
Professional Development Workshop
Beverly Parsons and Meg Hargreaves
1
What is a System?
A group of interacting, interrelated, and
interdependent elements forming a
complex whole
 A configuration of parts connected and
joined together by a web of relationships
 The whole is different from, and greater
than, the sum of its parts

2
Parts of an Elephant
3
Systems Thinking
A way of understanding reality that
emphasizes the relationships among a
system’s parts, rather than the parts
themselves.
 Concerned about interrelationships
among parts and their relationship to a
functioning whole
 Sees underlying patterns and structures

4
Foundations of Systems Theory
Cybernetics: system feedback,
information; differences (that make a
difference); human – machine analogy;
inclusion of the observer and the
observed in the system
 General systems theory: open systems;
system integrity; nested system
hierarchy, boundaries, webs, emergence
(sum greater than parts)

5
Systems Theories
Soft and critical systems: human systems
- multiple perspectives, power issues,
intractable problems without simple
solutions
 Systems dynamics: systems have
reinforcing and balancing feedback loops,
circularity, system archetypes, mental
models, unintended consequences

6
More Systems Theories
Complexity theory: complex adaptive
systems; semi-independent, interacting
agents; self-organization; emergence;
nonlinearity; co-evolution; past is
irreversible; future is unpredictable
 Learning systems: the way that people
learn and the systems in which they learn

7
System Boundaries

Shows what is inside and outside of the
system
Geographical (location)
 Organization (department, unit or function)
 Physical (money, material, information)
 Conceptual (goals, mission, purpose, rules)
 Intangibles (perceptions, awareness, models)
 Natural or man-made

8
System Relationships (Interconnections)

Connections and exchanges among
system parts, parts and the whole, and
the whole and its environment
Flows of information
 Flows of funding
 Client referrals
 Collaborative partnerships
 Family, community, and social networks

9
System Perspectives
Stakeholders’ worldviews and purposes
 System agents who have different
perspectives may pursue different
purposes within a given situation
 Patterns of (mis)alignment of purposes
and processes within and across system
levels

10
System Change
System differences generate creative
tension or energy within a system
 Positive or negative, energy provides
potential for system change
 System change: shifts in patterns
(similarities and differences) of system
relationships, boundaries, focus, timing,
events and behaviors over time and space

11
System Dynamics
Random (unorganized)
 Organized (simple or complicated)
 Adaptive (organic, self-organizing)
 All three system dynamics can be present
in a complex situation

12
Random System Attributes
Random activity – no pattern
 Unconnected collection of parts
 No cause-effect relationships
 Turbulence – no stability or equilibrium
 Answers are unknowable
 No purpose or direction – people react
blindly in a war zone or natural disaster

13
Random System: Hurricane Katrina
14
Organized (Simple) System Attributes
Stable, static pattern
 Parts tightly connected machines
 Predictable cause-effect relationships
 System can be reduced to parts and
processes and replicated
 Directive leadership, designed change
 Answers are knowable, with recipes or
prescriptions for action

15
Single Organized System:
Ring-Around the Rosie
16
Simple Organized System:
Riding a Bicycle
17
Organized (Complicated) System
Attributes
Dynamic patterns of feedback loops with
many interrelated parts within and across
subsystem levels
 Recursive, non-linear cause-effect
relationships; reinforcing and balancing
feedback loops maintain equilibrium
 Expert analysis can identify causal loops,
deep structural causes to actions

18
Insider Trading: A Tangled Web of
Tips and Trades
19
Adaptive (Complex) System Attributes
Dynamical patterns – parts adapting, coevolving with each other and environment
 Parts are massively entangled and
interdependent; nested webs, networks
 Parts self-organize, learn, and change
 Equilibrium in flux, sensitive to initial
conditions; system change emerges
through interactions among parts

20
Ecological View of an Elephant
21
Complex Interdependencies
22
Alignment of Context, Program, and
Evaluation Dynamics
Context can be random, organized,
adaptive, or combination of dynamics
 Program design uses random, organized
(entity-based), or adaptive (paradigmbased) or a combination of dynamics
 Evaluation design (content and process)
can be entity-focused (organized),
paradigm-focused (adaptive) or a
combination of both

23
System Dynamics of
Family Nutrition
24
Simple Organized Dynamics of
Family Nutrition
Context: hungry family
 Intervention: buy ingredients, bake a
cake, serve family at dinner
 Evaluation: quality of cake, family
satisfaction

25
Complicated Organized Dynamics of
Family Nutrition
Context: hungry family with different
tastes and preferences
 Intervention: ask for family preferences,
create optional dishes, serve family
multiple dishes at dinner
 Evaluation: quality and variety of dinner
options, matching of dishes to tastes

26
Complex, Adaptive Dynamics of
Family Nutrition
Context: hungry family with different
tastes, schedules, and cooking ability
 Intervention: Buy and store meal options,
make dishes for non-cooks, agree on
dinner schedule, adapt shopping patterns
to use of food and supplies
 Evaluation: trends, patterns of food use,
meals, family nutrition, overall health

27
System Dynamics of H1N1 Flu
28
Simple Organized Dynamics of H1N1 Flu
Context – everyone should be protected
through vaccination
 Program design – universal flu shot clinics
 Evaluation design - How many clinics
were conducted, how many people were
vaccinated, how many people contracted
the H1N1 flu virus

29
Complicated Organized Dynamics
of H1N1 Flu



Context – people are at different risk levels for
contracting the H1N1 flu
Program design – allocate, administer flu shots
by risk level, triage patients by level of risk
Evaluation design - What proportion of people
with high/medium/low risk receive the vaccine?
What proportion of people at each risk level
contract the H1N1 flu? How many deaths and
hospitalizations are avoided as result of shots?
30
Complex Adaptive Dynamics of H1N1 Flu



Context – Timing of two interacting epidemics
(H1N1 and seasonal flu) is ahead of current
vaccine production
Program design – Multi-level intervention:
national media messages, provider triage by risk,
populations self-organize multiple responses
Evaluation design – What are changing patterns
of twin epidemics? How are governments,
providers, populations reacting and interacting in
response to situation? Population health impacts?
31
System Dynamics Discussion
What are the situations’ boundaries,
focus, interconnections, perspectives,
power, timing, and dynamics?
 What are the risks of not understanding
the system attributes and dynamics?
 What are the benefits of understanding
the system attributes and dynamics?

32
System Dynamics of Child Abuse
Prevention – Home Visiting
33
U.S. Child Abuse and Neglect
Trends
34
Context, Program Design of Child Abuse
Prevention
Context: Many programs exist but child
abuse and neglect rates are increasing
 Program design: AFC funding for 17 grants
for the adaptation, implementation, spread,
and sustainability of evidence-based home
visiting programs through infrastructure
development and system change

35
Evaluation Design of Evidence-based
Home Visiting Initiative
Program evaluation – tracking of cross-site
cost, implementation, fidelity, and child and
family outcomes of 17 EBHV programs
 System evaluation – tracking of cross-site
and grantee-specific system infrastructure,
theories of action, measures of system
change, partner collaboration and network
analysis; system unit of analysis

36
Open Space Technology:
System Dynamics Exercise
What are the dynamics (i.e., the nature
and balance of types of system
dynamics) of the situation as a whole?
 What are the system dynamics of the
intervention?
 What are the implications for the
evaluation design and process?

37
Three Dynamics of a Social
System and its Context
38
Match of Evaluation Designs to Dynamics
of Social Systems and Their Context
Initiative Renewal
Design
Exploratory
Design
Organic
Design
Predictive
Design
39
Complex Adaptive Systems and Adaptive
(Self-organizing) Dynamics
1.
2.
3.
4.
Self-organizing/adaptive/organic
Sensitivity to initial conditions
Emergence
Macro pattern
40
Complex Adaptive Systems and Adaptive
(Self-organizing) Dynamics (cont.)
5.
6.
7.
Feedback
Co-evolution
Pattern formation and points of
influence
41
Implications for Evaluation and Action
1.
2.
3.
4.
Small differences can create large
effects.
The past influences but does not predict
the future.
Many points of influence exist.
Boundaries, differences, and
relationships are levers of influence
toward a purpose.
42
Implications for Evaluation and Action
5.
6.
7.
8.
Simple rules underlie patterns.
Pattern-based feedback and actions are
iterative.
Tensions are not resolved.
Patterns are outcomes.
43
Four Stages of Evaluation
Design
Evaluation
Shape
Practice
Collect
Data
Make
Meaning from
Data
44
45
Example: LEAP
Learning through Engineering
Design and Practice
46
Example: LEAP Research Design

Quasi-experimental design embedded in curriculum
development process

Pre-post assessments of

Content knowledge

Perceptions of engineers at work

Tinkering

Self-efficacy

Engineering notebooks

Career behaviors survey
47
External Evaluation Design
The external evaluation focused on:
 Confirmation of effectiveness
 Scale-up
 Sustainability
48
Conceptual Shifts
The fundamental conceptual shift in this project was from:
 teacher-directed de-contextualized learning to studentengaged project-based learning
 fixed skills and knowledge as learning outcomes to the
desired outcomes being that students are actively
engaged; develop the capacity to explore and figure
things out; and act like an engineer.
49
Confirmation of Effectiveness

Knowledge and skills related to project topics
and STEM concepts

Enjoyment and pride in project work

Development of teamwork, collaboration and
workplace skills

Interest in STEM courses and pursuit of
STEM career and educational pathways
50
Scale-Up
Scale-up involved two tracks:
 Greater use of the curriculum within the
school system
 Transfer of the curriculum to Boys and
Girls Clubs
51
Sustainability
Program sustainability
 Sustainability of learning of participating
students
 Sustainability of collaborations
 Sustainability of teaching capacity

52
Program Sustainability






Maintain relationships with the district
Professional development for teachers
Shape research related to the project
Explore ways to continue project at 9th grade
Track STEM course selection of project students in high
school
Collaborate on additional community dissemination and
funding
53
Sustainability of Learning of Students
Classroom
 Extracurricular activities
 Career-related activities
 Focused attention through high school
 Continued involvement of university
faculty and students
 Continued contact with science center

54
Sustainability of Collaborations

Use current collaborations to spur others
over time
55
Sustainability of Teaching Capacity

Build capacity through formal and
informal professional development
approaches
Training for new teachers
 Coaching by master teachers
 Summer professional development activities
 Technology enhanced training

56
Example 2
Communities of Learning,
Inquiry, and Practice
(CLIPs)
(video at www.insites.org/clip)
57
Example 3
Strengthening Families
58
Strengthening Families
Protective Factors
Parental resilience
 Social connections
 Concrete support in time of need
 Knowledge of parenting and child
development
 Social and emotional competence of child

59
60
World Café Exercise
What data gathering and/or analysis tools
have you found helpful in gaining a
deeper understanding of complex systems
or interventions?
 What practices help you develop your
capacity to recognize patterns?
 Do certain practices seem more related to
finding surface patterns and others more
related to finding deep patterns?

61
Contact Information

Beverly Parsons, Ph.D.
[email protected]
(360) 638-1442

Meg Hargreaves, Ph.D.
[email protected]
(617) 301-8994
62