Transcript Slide 1

Evaluating the Complex:
Getting to Maybe
Michael Quinn Patton
Oslo, Norway
29 May, 2008
Michael Quinn Patton
May, 2008
Evolving Understandings
I keep changing what I said. Any person
who is intellectually alive changes his
ideas. If anyone at a university is
teaching the same thing they were
teaching five years ago, either the field
is dead, or they haven’t been thinking.
Noam Chomsky
“The Professor Provaocateur,” The New York Times Magazine, Nov. 2, 2003: 13.
Michael Quinn Patton
May, 2008
Interpretive Frameworks
• May 2003 Harvard Business Review "The
High Cost of Accuracy." Kathleen Sutcliffe
and Klaus Weber.
They concluded that "the way senior
executives interpret their business
environment is more important for
performance than how accurately they
know their environment."
Michael Quinn Patton
May, 2008
They further concluded that it is a waste of
resources to spend a lot of money
increasing the marginal accuracy of data
available to senior executives compared to
the value of enhancing their capacity to
interpret whatever data they have.
Executives were more limited by a lack of
capacity to make sense of data than by
inadequate or inaccurate data.
In essence, they found that interpretive
capacity, or "mind-sets," distinguish highperformance more than data quality and
Michael Quinn Patton
accuracy.
May, 2008
Evaluation’s Traditional
Interpretive Framework
In the beginning…
Michael Quinn Patton
May, 2008
Original Primary Options
Formative
and
Summative
Evaluation
(Mid-term and End-of-Project Reviews)
Michael Quinn Patton
May, 2008
Evidence-based Practice
Evaluation grew up in the “projects”
testing models under a theory of
change that pilot testing would lead to
proven models that could be
disseminated and taken to scale:
The search for best practices
and evidenced-based practices
Michael Quinn Patton
May, 2008
Fundamental Issue:
How the World Is Changed
Top-down dissemination of
“proven models”
versus
Bottoms-up adaptive
management
Michael Quinn Patton
May, 2008
Models vs. Principles
Identifying proven principles for
adaptive management
(bottoms-up approach)
versus
Identifying and disseminating
proven models
(top down approach)
Michael Quinn Patton
May, 2008
Conditions that challenge traditional
model-testing evaluation
•
•
•
•
•
•
High innovation
Development
High uncertainty
Dynamic
Emergent
Systems Change
Michael Quinn Patton
May, 2008
Adaptive
Management
Mintzberg on Strategy
Two types of strategy: Intended & Emergent
Unrealized
Strategy
Intended
Strategy
Deliberate
Strategy
Realized
Emergent
Strategy
Strategy
Michael Quinn Patton
May, 2008
Re-conceptualizing Use
• Use is a process not a event
• Use involves an interaction not just
a report
• Use involves training for use not
just delivery of results
Michael Quinn Patton
May, 2008
Some premises:
• Evaluation is part of initial program design, including
conceptualizing the theory of change
• Evaluator’s role is to help users clarify their purpose,
hoped-for results, and change model.
• Evaluators can/should offer conceptual and
methodological options.
• Evaluators can help by questioning assumptions.
• Evaluators can play a key role in facilitating
evaluative thinking all along the way.
• Interpretative dialogue is critical.
• Designs can be emergent and flexible.
Michael Quinn Patton
May, 2008
Three ways of conceptualizing
and mapping theories of change
 Linear Newtonian causality
 Interdependent systems
relationships
 Complex nonlinear dynamics
Michael Quinn Patton
May, 2008
Linear Logic Model
INPUTS (people, materials)
ACTIVITIES (processes) 
OUTPUTS 
OUTCOMES 
CHANGES IN PEOPLES LIVES 
IMPACTS 
CHANGES IN COMMUNITIES
Michael Quinn Patton
May, 2008
Pushing Force
(Non-directional)
(Inertia)
a.k.a. “Cost of
Change”
Dissatisfaction with
the
Status Quo
Resistance to
Change
Pulling Force
(Directional)
a.k.a. “Desirability of
the end state”
Compelling Vision
Michael Quinn Patton
May, 2008
Causal Diagram of Beckhard’s
Change Formula
Pushing Force
(Non-directional)
(Inertia)
a.k.a. “Cost of
Change”
Dissatisfaction with
the
Status Quo
Resistance to
Change
Pulling Force
(Directional)
a.k.a. “Desirability of
the end state”
Compelling Vision
Believability
Michael Quinn Patton
May, 2008
First Steps
Feedback Systems Logic Model
What inputs need to go into the
process to make the product
that produces the desired
result?
What steps need to be taken to
create the product that achieves
the desired result?
What features / characteristics should the product
have?
Process
Structure
What is the desired result?
What should customer
experience?
OUTCOMES
Feedback into process
Key Processes & Functions
Inputs
Staff Resources
Financial resources
Internal Standards
External Requirements
and Information
Equipment/Materials
Inputs organized and utilized
Procedures
Steps
Key processes
Measure Variability
Assess Process Control
Assess fidelity to planned
procedures
Assess impact of variation
Evaluate opportunity to raise the
bar
Cause
Output / Product
Essential Attributes
Attributes required to meet of exceed
customer needs:
"Do the Right Thing"
Efficacy
Appropriate
Characteristics to meet or exceed
customer wants and expectations
of excellence
"Do the Right Thing Well":
Efficiency
Dignity and Respect
Effectiveness
Timeliness
Reduce Waste
Safety
Continuity
Availability
Planning
Michael Quinn Patton
Implementation
May, 2008
Customer
Outcomes
&
Satisfaction
 Measure
Effectiveness
 Measure
Satisfaction
 Inform
Improvement
needs
Effect
Division for Heart Disease and Stroke Prevention Evaluation Planning Logic Model
Planning
Division Leadership
that provides
sufficient:
· Infrastructure
· Policies
· Strategic Planning
Activities
Workforce that is:
· Diverse
· Skilled
Translation,
Dissemination
Effective:
· Management
· Coordination
· Staff
development
Increased advocacy
and “activated
constituency”
Disparities
Enhanced external
application of Division
goals and strategies
Surveillance
Research
Division workplace
that:
· Offers a healthy
work environment
· Recognizes
excellence
· Provides quality
training and
management
· Includes effective
systems,
procedures, and
communication
(Goal 5)
Evaluation
Program
Translation
and dissemination
of the current
knowledge base,
and identification
of ways to improve
that knowledge
base
Enhanced ability
of programs to apply
findings to improve
public health
Increased focus
on heart disease
and stroke
prevention efforts
by states and
partners,
especially with
regard to
disparities
Enhanced
competency of public
health workforce
Increased adoption,
reach, implementation,
and sustainability of
recommended public
health strategies to
achieve strategic plan
goals:
· Prevent risk factors for
heart disease and
stroke (Goal 1)
· Increase detection
and treatment of risk
factors (Goal 2)
· Increase early
identification and
treatment of heart
attacks and strokes
(Goal 3)
· Prevent recurring
cardiovascular events
(Goal 4)
Enhanced integration
among chronic
disease programs
External
Internal
Michael Quinn Patton
May, 2008
Reduced risk
factors
Increased
knowledge of
signs and
symptoms
Improved quality
of care
Improved
emergency
response
Policy
Communication
Impact
Adoption, Practice, Sustainability
Engaged network of
states and partners
Leadership
Collaboration
Resources that are:
· Available
· Timely
WHY
WHAT
HOW
Reduced
morbidity and
mortality of heart
disease and stroke
Reduced levels of
disparities in heart
disease and stroke
Eliminated
preventable
strokes and risks
Reduced
economic impact
of heart disease
and stroke
Step Two: THEN A MIRACLE OCCURS
“I think you should be more explicit here in step two.”
Michael Quinn Patton
May, 2008
Systems
• Parts are interdependent such a
change in one part changes all parts
• The whole is greater than the sum of
the parts
• Focus on interconnected relationships
• Systems are made up of sub-systems
and function within larger systems
Michael Quinn Patton
May, 2008
Systems Concepts in Evaluation –
An Expert Anthology. 2006.
Bob Williams and Iraj Imam
AEA Monograph,
EdgePress/AEA Point Reyes CA.
Michael Quinn Patton
May, 2008
Michael Quinn Patton
May, 2008
Understanding the
Elephant
from
a Systems Perspective
Michael Quinn Patton
May, 2008
The relationship between what
goes in and what comes out
What
conceptual
framework
informs
front-end
evaluation
work?
Michael Quinn Patton
May, 2008
Teen Pregnancy Program
Example
Michael Quinn Patton
May, 2008
Logic Model for Pregnant Teens Program
1. Program reaches out to pregnant teens
2. Pregnant teens enter and attend the program (participation)
3. Teens learn prenatal nutrition and self-care (increased
knowledge)
4. Teens develop commitment to take care of themselves
and their babies (attitude change)
5. Teens adopt healthy behaviors: no smoking, no drinking,
attend prenatal clinic, eat properly (behavior change)
6. Teens have healthy babies (desired outcome)
Michael Quinn Patton
May, 2008
Systems web showing possible influence
linkages to a pregnant teenager
Prenatal
program
staff
Teachers/
other
adults
Child's
father &
peers
Her
parents &
other family
members
Young
pregnant
woman's
attitudes &
behaviors
Her peer
group
Michael Quinn Patton
May, 2008
Program systems web showing possible institutional
influences affecting pregnant teenagers:
Other Systems
-- welfare
-- legal
-- nutrition
programs
-- transportation
-- child
protection
-- media messages
Context factors
-- politics
-- economic
incentives
-- social norms
-- culture
-- music
Prenatal
program
SCHOOL
SYSTEM
Prenatal
Clinic and
Hospital
Outreach
Young
pregnant
women's
attitudes &
behaviors
Church
Michael Quinn Patton
May, 2008
Youth
Culture
Other
communitybased youth
programs
Using Different System Lenses to
Understand a “particular” System
Biologic System
• Emergence
• Coordination/synergy
• Structure, Process, Pattern
• Vitality
Economic System
Political System
 Inputs/Outputs
 Cost/Waste/Value/Benefits
 Customers/Suppliers
• Power
• Governance
• Citizenship
• Equity
Sociologic System
• Relationships
• Conversations
• Interdependence
• Loose-tight coupling
• Meaning/sense
SYSTEM
DIMENSIONS
Mechanical / Physical System
• Flow
• Temporal Sequencing
• Spatial Proximities
• Logistics
• Information
Psychological System
•Organizing
•Forces Field
Michael
Quinn Patton
•Ecological/Behaviour
May, 2008
Settings
Anthropologic
System
• Values
• Culture/Milieu
Information System
•Access
•Speed
•Fidelity/utility
•Privacy/security
•Storage
Map Systems as Webs
Source: Digital Capital: Harnessing the Power of Business Webs,
By Don Tapscott, David Ticoll and Alex Lowy
5
Michael Quinn Patton
May, 2008
Complex Dynamic Systems
Configuration
Michael Quinn Patton
May, 2008
HIV/AIDS Example
• Hits every system: health, family,
social, religious, economic, political,
community, international
• Requires multiple interventions on
multiple fronts in all subsystems
simultaneously
• Resulting reactions, interactions,
consequences dynamic, unpredictable,
emergent, and ever changing
Michael Quinn Patton
May, 2008
Challenges:
Situation
Recognition
and
Appropriate
Evaluation
Designs
Michael Quinn Patton
May, 2008
The nature of
EXPERTISE
Michael Quinn Patton
May, 2008
Contingency-based
Evaluation
• Situational analysis & responsiveness
• Context sensitivity
• Clarify and focus on intended users:
stakeholder analysis
• Clarify and focus on intended uses
• Methodological appropriateness
• Criteria for evaluating the evaluation:
credibility, meaningfulness
Michael Quinn Patton
May, 2008
Seeing Through A
Complexity Lens
“You don't see something until you have the
right metaphor to let
you perceive it”. Thomas Kuhn
Michael Quinn Patton
May, 2008
Complex Nonlinear Dynamics
• Nonlinear: Small actions can have large
reactions. “The Butterfly Wings Metaphor”
• Emergent: Self-organizing, Attractors
• Dynamic: Interactions within, between,
and among subsystems and parts within
systems can volatile, changing
• Getting to Maybe: Uncertainty,
unpredictable, uncontrollable
Michael Quinn Patton
May, 2008
SNOWDEN’s
CYNAFIN FRAMEWORK
Simple, Complicated,
Complex, Chaotic
and Disordered
Behaviours
Michael Quinn Patton
May,
TEI 2008
2008
Linear contextualisation: 1
unorder order
Most
chaotic
chaotic
Most
ordered
complex
complicated
Michael Quinn Patton
May,
TEI 2008
2008
simple
Conceptual Options
•Simple
•Complicated
•Complex
Michael Quinn Patton
May, 2008
Types of Community Issues
Close to
The Stacey Matrix
Close to
Certainty
Michael Quinn Patton
May, 2008
Far from
Close to
Simple
Simple
Plan, control
Close to
Certainty
Michael Quinn Patton
May, 2008
Far from
Simple (Known)
Inter-relationships between elements :
Tight, centralised connections.
Anyone can see the things the way they are.
Very simple linear cause and effect.
Everyone knows the right answer within the
current context (which of course may
not be self-evident or known to others –
hence importance of context).
Michael Quinn Patton
May,
TEI 2008
2008
Simple
Complicated Complex
Following a Recipe A Rocket to the Moon
The recipe is essential
Recipes are tested to
assure replicability of
later efforts
No particular
expertise; knowing how
to cook increases
success
Recipes produce
standard products
Certainty of same
results every time
Michael Quinn Patton
May, 2008
Raising a Child
Complicated (Knowable)
Inter-relationships between elements:
Relationships are looser but still
clustered around a central core.
Cause and effect is non-linear.
Relationships able to be modelled
and able to predicted.
An expert would know the right
answer(s)
Michael Quinn Patton
May,
TEI 2008
2008
Close to
Technically Complicated
Simple
Plan, control
Close to
Technically Complicated
Experiment, coordinate expertise
Certainty
Michael Quinn Patton
May, 2008
Far from
Simple
Following a Recipe
The recipe is essential
Recipes are tested to
assure replicability of
later efforts
No particular
expertise; knowing how
to cook increases
success
Recipes produce
standard products
Certainty of same
results every time
Complicated Complex
A Rocket to the Moon
• Formulae are
critical and
necessary
• Sending one rocket
increases
assurance that next
will be ok
• High level of
expertise in many
specialized fields +
coordination
• Rockets similar in
critical ways
• High degree of
certainty of
outcome
Michael Quinn Patton
May, 2008
Raising a
Child
Socially Complicated
Socially
Complicated
Close to
Build relationships,
create common
ground
Simple
Technically Complicated
Plan, control
Experiment, coordinate expertise
Close to
Certainty
Michael Quinn Patton
May, 2008
Far from
Socially complicated
Implementing human rights
agreements, like gender equity or
outlawing child labor
Environmental Initiatives



Many different and competing
stakeholders
Diverse vested interests
High stakes
Michael Quinn Patton
May, 2008
Socially complicated
situations
pose the challenge
of coordinating and
integrating
many players
Michael Quinn Patton
May, 2008
Stakeholder Mapping
High Interest/
Low Power
THE INVOLVED
THE CROWD
Low interest/
Low Power
High Interest/
High Power
THE PLAYERS
CONTEXT SETTERS
Low Interest/
High Power
Michael Quinn Patton
May, 2008
Complex
Inter-relationships between elements:
Centre is loosely connected to network.
Cause effect difficult to understand
in current setting.
Situation understandable only in retrospect.
Not predictable.
Michael Quinn Patton
May,
TEI 2008
2008
Know When Your Challenges Are In the
Zone of Complexity
Socially
Complicated
Close to
Build
relationships,
create common
ground
Simple
Plan, control
Close to
Zone of
Complexity
Systems Thinking
Relationship Building
Collaboration
Good Enough Vision
Chunking Around Drivers
Minimum Specifications
Multiple Actions
Adaptability & Organic
Technically Complicated
Experiment, coordinate expertise
Certainty
Michael Quinn Patton
May, 2008
Far from
Simple
Following a Recipe
The recipe is essential
Recipes are tested to
assure replicability of
later efforts
No particular
expertise; knowing how
to cook increases
success
Recipes produce
standard products
Certainty of same
results every time
Complicated Complex
A Rocket to the Moon
Raising a
Child
• Formulae have only
a limited application
Sending one rocket
increases
assurance that next • Raising one child
gives no assurance
will be ok
of success with the
next
High level of expertise • Expertise can help
in many specialized
but is not sufficient;
fields +
relationships are
coordination
key
Rockets similar in
critical ways
• Every child is
unique
High degree of
certainty of
outcome
Michael Quinn Patton
• Uncertainty of
outcome remains
May, 2008
Chaotic (Unordered)
Cause and effect unknowable,
unattributable even in retrospect.
No right answer(s)
Michael Quinn Patton
May,
TEI 2008
2008
Know When Your Challenges Are In the
Zone of Complexity
Chaos
Massive Avoidance
Socially
Complicated
Close to
Build
relationships,
create common
ground
Simple
Plan, control
Close to
Zone of
Complexity
Technically Complicated
Experiment, coordinate expertise
Certainty
Michael Quinn Patton
May, 2008
Far from
Simple
Following a Recipe
Complicated Complex
A Rocket to the Moon
Raising a Child
•
Formulae are critical •
and necessary
Formulae have only a
limited application
Recipes are tested to
assure replicability of
later efforts
•
Sending one rocket
increases assurance
that next will be ok
•
Raising one child
gives no assurance of
success with the next
No particular
expertise; knowing how
to cook increases
success
•
High level of
expertise in many
specialized fields +
coordination
•
Expertise can help
but is not sufficient;
Recipe notes the
quantity and nature of
“parts” needed
•
Separate into parts
and then coordinate
•
Can’t separate parts
from the whole
•
Rockets similar in
critical ways
•
Every child is unique
•
Uncertainty of
outcome remains
The recipe is essential
Recipes produce
standard products
Certainty of same
results every time
•
High degree of
certainty of outcome
Michael Quinn Patton
May, 2008
relationships are
key
Michael Quinn Patton
May,
TEI 2008
2008
Global Economic Complexity
Arthur Greenspan, Final speech to world’s
Central Bankers, Jackson Hole, Wyoming
August 26, 2005
Michael Quinn Patton
May, 2008
• “In the absence of a single variable, or at
most a few, that can serve as a reliable
guide, policymakers have been forced to
fall back on an approach that entails the
interpretation of the full range of economic
and financial data.”
Michael Quinn Patton
May, 2008
• “Despite extensive efforts to capture and
quantify what we perceive as the key
macroeconomic relationships, our
knowledge about many critical linkages is
far from complete and, in all likelihood, will
remain so. Every model, no matter how
detailed or how well conceived, designed,
and implemented, is a vastly simplified
representation of the world, with all of the
intricacies we experience on a day-to-day
basis.”
Michael Quinn Patton
May, 2008
• “We all temper the outputs of our models
and test their results against the ongoing
evaluations of a whole array of
observations that we do not capture in
either the data input or the structure of our
models. We are particularly sensitive to
observations that appear inconsistent with
the causal relationships of our formal
models.”
Michael Quinn Patton
May, 2008
Oct 8, 2007 on The Daily Show
“I was telling my colleagues the other day...I’d
been dealing with these big mathematical
models for forecasting the economy, and I’m
looking at what’s going on the last few weeks
and I say, “Y’know, if I could figure out a way to
determine whether or not people are more
fearful, or changing to euphoric... I don’t need
any of this other stuff. I could forecast the
economy better than any way I know. The
trouble is, we can’t figure that out. I’ve been in
the forecasting business for 50 years, and I’m no
better than I ever was, and nobody else is
either.” Alan Greenspan.
Michael Quinn Patton
May, 2008
The Age of Turbulence: Adventures in a
New World, 2007, by Alan Greenspan
Michael Quinn Patton
May, 2008
Two evaluation locations
for
Evaluating the Complex:
Prospective
and
Retrospective
Michael Quinn Patton
May, 2008
Prospective Example
The McGill-McConnell Leadership
Program Example
Simple elements
Complicated elements
Complex elements
Michael Quinn Patton
May, 2008
Simple outcomes
• Increase knowledge and skills of
participants
Evaluation: Pre-post data and
documentation of learning
Michael Quinn Patton
May, 2008
Complicated Impacts
• Change participants’ organizations
Evaluation:
Case studies
of
organizational change
Michael Quinn Patton
May, 2008
Complex Vision
• Infuse energy into the moribund notfor-profit (voluntary) sector
• Make the sector more dynamic
• Create network of leaders who
actively engage in change
Michael Quinn Patton
May, 2008
Evaluating the Complex
• Real time follow-up of network
connections and actions
• Follow-up is an intervention
• Rapid feedback of findings permits
infusion of resources in support of
emergent outcomes
Michael Quinn Patton
May, 2008
Process Use
Infusing evaluative thinking as a
primary type of process use.
Capacity-building as an
evaluation focus of
process use.
Michael Quinn Patton
May, 2008
Complex
Interdependencies
Michael Quinn Patton
May, 2008
Insert action into the
system
Michael Quinn Patton
May, 2008
EMERGENCE
Michael Quinn Patton
May, 2008
Retrospective Example
Advocacy Evaluation
Final Push Campaign
to overthrow the
Juvenile Death Penalty
Michael Quinn Patton
May, 2008
In late 2003 several petitions on behalf of
juvenile offenders facing the death penalty
were filed with the U.S. Supreme Court.
On January 26, 2004, the Court granted
certiorari in Roper v. Simmons and the
case was argued before the Court on
October 13, 2004. The decision was
announced March 1, 2005. The Court
ruled 5-4 that capital punishment for
juveniles was unconstitutional.
Michael Quinn Patton
May, 2008
In the brief window of time between when
the Court agreed to hear the case and the
case was argued, roughly nine months, a
coordinated campaign was organized and
funded aimed at overturning the juvenile
death penalty. Organizing, public
education, networking & communications
continued through to the Court's ruling in
March, 2005.
Michael Quinn Patton
May, 2008
Retrospective Evaluation
To what extent, if at all, was the Court’s
decision influenced by the campaign?
Modus Operandi
or
General Elimination Method
Michael Quinn Patton
May, 2008
1. Strong high capacity coalitions. Working through coalitions is a common centerpiece
of advocacy strategy.
2. Strong national-state-grassroots coordination. Effective policy change coalitions in
the United States have to be able to work bottoms-up and top-down, with national
campaigns supporting and coordinating state and grassroots efforts, while state efforts
infuse national campaigns with local knowledge and grassroots energy. Strengthening
strong national-state coordination is part of coalition development and field building.
3. Disciplined and focused messages with effective communications. Effective
communications must occur within movements (message discipline) and to target
audiences (focused messaging). Strengthening communications has been a key a key
component of advocacy coalition building.
4. Solid research and knowledge base. The content of effective messages must be based
on solid research and timely knowledge. In the knowledge age, policy coalitions must be
able to marry their values with relevant research and real time data about dynamic
policy environment.
5. Timely, opportunistic lobbying and judicial engagement. The evaluation findings
emphasize that effective lobbying requires connections, skill, flexibility, coordination, and
strategy.
6. Collaborating funders engaged in strategic funding. Effective funding involves not
only financial support, but infusion of expertise and strategy as part of field building.
Michael Quinn Patton
May, 2008
Targeted State
Campaigns &
Grassroots
Organizing
Effective Litigation
and Amicus
Briefs
Omnibus
Coordinated
Integrated
Strategy and
Implementation
United
Coalition
Partners
Credible, Useful
Up-to-date Research
Focused
Communications
Campaign
Knowledgeable
Funders
Michael Quinn Patton
May, 2008
Overall Lesson Learned for Effective Advocacy
In essence, strong national/state/grassroots coordination
depends on having a high capacity coalition. A solid knowledge
and research base contributes to a focused message and
effective communications. Message discipline depends on a
strong coalition and national-state coordination, as does timely
and opportunistic lobbying and judicial engagement. To build
and sustain a high capacity coalition, funders must use their
resources and knowledge to collaborate around shared
strategies. These factors in combination and mutual
reinforcement strengthen advocacy efforts. In classic
systems framing, the whole is greater than the sum of
parts, and the optimal functioning of each part is
dependent on the optimal integration and integrated
functioning of the whole.
Michael Quinn Patton
May, 2008
SIX INTERCONNECTED FACTORS,
DYNAMICALLY INTERACTING,
THAT STRENGTHEN ADVOCACY
Strong
High Capacity
Coalitions
Timely,
Opportunistic
Lobbying &
Judicial
Engagement
EFFECTIVE
ADVOCACY
Solid
Knowledge &
Research
Base
Collaborating
Funders/
Strategic
Funding
Michael Quinn Patton
May, 2008
Strong
National/
Grassroots
Coordination
Disciplined
Focused
Message/
Effective
Communications
Strong, high
capacity
coalition
Targeted
timely
lobbying
National
state
grassroots
coordinatio
n
Relevant
research
Disciplined
focused
message
Collaborating
funders/
strategic
funding
Michael Quinn Patton
May, 2008
The
interdependent
system of
factors that
contribute to
effective
advocacy and
change
Dealing with the Unexpected
and Unpredicted
Michael Quinn Patton
May, 2008
Michael Quinn Patton
May, 2008
Photo by
Lynsey Gornick
Creative Challenge
Situational adaptability:
 Contingency-based evaluation
 Appropriateness
--Using standard forms of evaluation
and
-- Going beyond standard forms when
appropriate and useful
Michael Quinn Patton
May, 2008
Being Open:
How hard is this to do?
My Colorado experience….
Michael Quinn Patton
May, 2008
“The range of what we think and do
is limited by what we fail to notice.
And because we fail to notice that
we fail to notice, there is little we
can do to change, until we notice
how failing to notice shapes our
thoughts and deeds.”
Scottish psychiatrist, R. Laing
Michael Quinn Patton
May, 2008
Paradigms and Lenses
• The importance of interpretive
frameworks
• Complexity as an interpretive
framework
Michael Quinn Patton
May, 2008
Getting to Maybe:
How the World Is
Changed? 2006
Frances Westley, Brenda
Zimmerman, Michael Q. Patton
Random House Canada,
Michael Quinn Patton
May, 2008
Complex Situations
• Highly emergent (difficult to
plan and predict)
• Highly dynamic, rapidly
changing
• Relationships are
interdependent and non-linear
rather than simple and linear
(cause-effect)
Michael Quinn Patton
May, 2008
?
Close to
Zone of
Complexity
Close to
Certainty
Michael Quinn Patton
May, 2008
Far from
Close to
Close to
Certainty
Michael Quinn Patton
May, 2008
Far from
Close to
Close to
Certainty
Michael Quinn Patton
May, 2008
Far from
Contingency-based
Developmental
Evaluation
Michael Quinn Patton
May, 2008
Improvement
versus
Development
Michael Quinn Patton
May, 2008
Beyond
just
Summative
and
Formative
Michael Quinn Patton
May, 2008
Beyond
Static Accountability
Models
Michael Quinn Patton
May, 2008
Example of an emergent option:
Developmental
Evaluation
Michael Quinn Patton
May, 2008
DEVELOPMENTAL EVALUATION
DEFINED
Evaluation processes, including asking evaluative
questions and applying evaluation logic, to support
program, product, staff and/or organizational
development. The evaluator is part of a team whose
members collaborate to conceptualize, design and test
new approaches in a long-term, on-going process of
continuous improvement, adaptation and intentional
change. The evaluator's primary function in the team
is to elucidate team discussions with evaluative
questions, data and logic, and facilitate data-based
decision-making in the developmental process.
Michael Quinn Patton
May, 2008
Other names
 Real time evaluation
 Emergent evaluation
 Action evaluation
 Adaptive evaluation
Michael Quinn Patton
May, 2008
CONTRASTS
Traditional
evaluations…
• Testing models
Complexity-based,
Developmental
Evaluation…
• Supporting
innovation and
adaptation
Michael Quinn Patton
May, 2008
Traditional
Evaluation…
• Render
definitive
judgments of
success or
failure
Developmental
Evaluation…
• Provide feedback,
generate learnings,
support direction
or affirm changes
in direction in real
time
Michael Quinn Patton
May, 2008
Developmental
Evaluation…
Traditional
Evaluation…
• Render definitive judgments of
success or failure
• Provide feedback, generate
learnings, support direction or
affirm changes in direction
• Measure success
• Develop new
against
measures and
predetermined goals
monitoring
mechanisms as
goals emerge &
evolve
Michael Quinn Patton
May, 2008
Traditional
Evaluation…
• Evaluator external,
independent,
objective
Developmental
Evaluation…
• Evaluator part of a
team, a facilitator
and learning coach
bringing evaluative
thinking to the table,
supportive of the
organization’s goals
Michael Quinn Patton
May, 2008
Traditional
Evaluation…
• Evaluator
determines the
design based on the
evaluator’s
perspective about
what is important.
The evaluator
controls the
evaluation.
Developmental
Evaluation…
• Evaluator
collaborates with
those engaged in
the change effort to
design an
evaluation process
that matches
philosophically and
organizationally.
Michael Quinn Patton
May, 2008
Traditional
Evaluation…
• Design the
evaluation based on
linear cause-effect
logic models
Developmental
Evaluation…
• Design the
evaluation to
capture system
dynamics,
interdependencies,
and emergent
interconnections
Michael Quinn Patton
May, 2008
Traditional
Evaluation…
• Aim to produce
generalizable
findings across time
& space
.
Developmental
Evaluation…
• Aim to produce
context-specific
understandings that
inform ongoing
innovation
Michael Quinn Patton
May, 2008
Traditional
Evaluation…
• Accountability
focused on and
directed to external
authorities and
funders.
Developmental
Evaluation…
• Accountability
centered on the
innovators’ deep
sense of
fundamental values
and commitments –
and learning.
Michael Quinn Patton
May, 2008
Traditional
Evaluation…
Developmental
Evaluation…
• Accountability to
control and locate
blame for failures
• Learning to respond
to lack of control
and stay in touch
with what’s
unfolding
• And thereby
respond
strategically
Michael Quinn Patton
May, 2008
Traditional
Evaluation…
• Evaluation often a
compliance function
delegated down in
the organization
Developmental
Evaluation…
• Evaluation a
leadership function:
Reality-testing,
results-focused,
learning-oriented
leadership
Michael Quinn Patton
May, 2008
Traditional
Evaluation…
• Evaluation
engenders
fear of failure.
Developmental
Evaluation…
• Evaluation supports
hunger for learning.
Michael Quinn Patton
May, 2008
Conditions
•
•
•
•
•
High innovation
Development
High uncertainty
Dynamic
Emergent
Michael Quinn Patton
May, 2008
SenseMaker software
• Dave Snowden, Founder of Cognitive
Edge, former Director of Knowledge
Management at IBM
• SenseMaker can code and map 95,000
stories in 24 hours
• See the world as others see it; anti-terror
applications.
• See the quantitative patterns in the metadata with qualitative context and meaning
Michael Quinn Patton
May, 2008
PANARCHY MODEL
Michael Quinn Patton
May, 2008
Routine Change
Conservation
K
Growth
r
Michael Quinn Patton
May, 2008
Turbulent Change
Backloop
Reorganization

Michael Quinn Patton
May, 2008
Release

Adaptive Cycle
Conservation
K
Release

Reorganization

Backloop
Growth
r
Michael Quinn Patton
May, 2008
Exploration phase
Mature product
and scaling up
Stored
System Capacity
Building
Leadership/initiative
Phases of Technological and Social Innovation
Little CAPITAL STORED
Much
The birth, growth, destruction and renewal of a forest
1
Creative
Destruction
Weak
CONNECTEDNESS
Strong
Little CAPITAL STORED
Much
The birth, growth, destruction and renewal of a forest
2
Renewal/Explorati
Renewal/Expl
onoration
1
Creative
Destruction
Weak
CONNECTEDNESS
Strong
Little CAPITAL STORED Much
The birth, growth, destruction and renewal of a forest
3
Exploitation
Weak
CONNECTEDNESS
Strong
Much
The birth, growth, destruction and renewal of a forest
4
3
Exploitation
Little
CAPITAL STORED
Conservation
Weak
CONNECTEDNESS
Strong
DEVEOPMENTAL
EVALUATION
SUMMATIVE
Stored
FORMATIVE
HARVESTING
LESSONS
Phases of Technological & Social Innovation
Taking Emergence Seriously
• Beyond “unanticipated
consequences” to genuine
openness
Michael Quinn Patton
May, 2008
SURPRISES
The psychologist Baruch Fischhoff wrote:
"The occurrence of an event increases its
reconstructed probability"—in other words,
surprises are psychologically untenable in
some ways, and we reshape our memories
and expectations until we believe that the
surprising event was, in fact, likely.
Michael Quinn Patton
May, 2008
The Epistemology of Surprise
MALCOLM GLADWELL
We hate surprises. We try to erase them from our
memory. This is part of what keeps us sane.
If, after all, we were always fully aware of the possibility
of completely unpredictable events, would we be able to
walk out the front door in the morning? Would we ever
invest in the stock market? Would we have children?
Generally speaking, people who have an accurate
mental picture of why and how things happen tend to
occupy mental hospitals—or, at the very least, a
psychiatrist's office….
Michael Quinn Patton
May, 2008
Information is like a stream
teeming with fish,
and if you stick out a net
you'll collect something—
but to decide what
information is
consequential.
How does one do that?
Michael Quinn Patton
May, 2008
At the heart is the process of distinguishing signal from
noise.
How is that done?
I have no idea, nor does anyone, I think, who isn't a
seasoned analyst. Pattern recognition is something that
comes only with experience. It's a matter of intuition, as
much as anything.
People always want to reduce this sort of thing to a
formula, or a system, and I'm not sure you can do that. I
suspect that there are some artificial-intelligence systems
that can help to sort through certain kinds of data.
But that could only be a first cut, and eventually human
judgment has to be involved.
Michael Quinn Patton
May, 2008
What role can evaluation play
with complex dynamic
innovations?
Michael Quinn Patton
May, 2008
Challenge:
Matching the evaluation
process and design to the
nature of the situation:
Contingency-based
Evaluation
Michael Quinn Patton
May, 2008
References
Getting to Maybe: How the World Is Changed?
Frances Westley, Brenda Zimmerman, Michael
Q. Patton, Random House Canada, 2006
Utilization-Focused Evaluation, 4th ed.,
Michael Quinn Patton, Sage, 2008.
Michael Quinn Patton
May, 2008