Transcript Making Change Happen: how do you know?

Kerry McCarthy
November 2012
Measuring & capturing the story
of your impact: where do I start?
It can seem complicated
It’s not that complicated
• What difference are we making?
• How do we know?
• Who cares?
Also interested in
• What is making the difference?
But it’s not that simple either
• It’s all in the mind
• It’s part of everything else
• No replacement for thinking, data, analysis
Start where you are…
1.
2.
3.
4.
5.
6.
7.
8.
What stage of development are we at?
What do I want to know, for what purpose, for what
audiences?
When do I need to know this by?
What evidence already exists that is relevant to what I
am doing?
What data do we currently have or are we likely to have
in the near future?
What resources do I have – money, people’s time and
expertise?
Who else wants to know about this? Can we work
together? Would our objectives be properly aligned or
would it over-complicate things?
What can I do myself? What skills and support do I
need to access?
And then go a bit deeper
• What do we want to measure?
– Theory of change
• How do we measure?
– Evaluation frameworks
– Qualitative & quantitative
• Quality & credibility
What do we want to measure?
Theory of Change
Theory of change: what is it?
It’s about being
more specific than
a general idea or
vision.
Articulation of the
often tacit
assumptions and
understanding about
how people expect
something to make
a difference.
Description of how resources
(inputs) are used for different
activities (outputs) and the
difference these activities make
for the people or organisations
involved (outcomes), and
whether these differences
would have occurred anyway
(impact).
Theory of change: why should I do it?
• It’s part of everything else…strategy, being
heard, business case
• Test the assumptions in your thinking,
highlight problems & gaps
• Plan your approach to data & evaluation
• Develop a shared narrative
• Record the trajectory of your plans as they
evolve
Inputs
Activities
Outputs
Outcomes
Impact
(money,
people,
expertise)
(develop an
App to
support YP
into ETE)
(no. of YP
using the
App)
(no. of YP for
who using
the App leads
to ETE)
(higher
employment,
educational
attainment ..)
Theory of change: how do I do it?
Keep it:
•
•
•
•
Simple
Relevant
Visual
Up to date
Theory of change: when should I do it?
• Surface assumptions about impact from early
on
• Update your theory of change with your team
at sensible intervals
• Re-visit your theory of change after evaluation
results
How do we measure?
Evaluation frameworks
Qualitative & quantitative
Evaluation framework
Aim or
vision
Specific Outcome Methods
outcome indicator (collecting
Who &
how
data,
(will data &
information) information
be
collected)
Analyse,
translate
& use
What counts?
• What other people say counts?
• What is useful?
• What is possible?
Three different answers?
Quality & credibility
Quality & credibility
• Not just about experimental & quantitative
methods
• Validity of design & conduct is different to
quality of reporting
• What is realistic – access, timeframe,
resources etc.
Quality in quantitative research example
• Quantitative research
– Defensible approach / design
– Rigour in conduct (data collection & analysis)
– Appropriate & convincing sampling strategy
Ref: from Dr Pamela Campanelli (2012) Q2 Training materials for ‘Quality in Social Research’
Quality in qualitative research example
1.
2.
3.
4.
5.
Are the aims of the study clearly stated?
Is a qualitative approach appropriate to the aims of the study?
Are the criteria for selecting the sample clearly explained?
Are the characteristics of the achieved sample adequately described?
Are the methods of data collection used appropriate for the aims of the
study?
6. Were efforts made to minimise the impact of the research process on
the study findings?
7. Was the collections of data systematic and comprehensive?
8. Were efforts made to assess reliability and validity?
9. Was the analysis of the data systematic?
10. Are the interpretations clearly presented and adequately supported by
the evidence?
Ref: Boulton M & Fitzpatrick R (97) Evaluating qualitative research’. Evidence Based Health Policy and
Management, 1(4): 83:85
End where you started…
1.
2.
3.
4.
5.
6.
7.
8.
What stage of development are we at?
What do I want to know, for what purpose, for what
audiences?
When do I need to know this by?
What evidence already exists that is relevant to what I
am doing?
What data do we currently have or are we likely to have
in the near future?
What resources do I have – money, people’s time and
expertise?
Who else wants to know about this? Can we work
together? Would our objectives be properly aligned or
would it over-complicate things?
What can I do myself? What skills and support do I need
to access?
Resources: theory of change
•
Funnell, S.C & Rogers, P.J. (2011) Purposeful Program Theory: Effective Use of
Theories of Change and Logic Models London: John Wiley & Sons
• Theory of Change community and resources (some free and some fee-charging)
http://www.theoryofchange.org/
•
Anderson A.A (2010) The Community Builder’s Approach to Theory of Change: a
practical guide to theory development New York: Aspen Institute
http://www.aspeninstitute.org/sites/default/files/content/docs/rcc/rcccommbuilde
rsapproach.pdf
•
Clappison A (2011) ‘Theories of change: an expanding resource list’ Research to
Action 11 May 2011, accessed16 August 2012
http://www.researchtoaction.org/2011/05/theory-of-change-useful-resources/
•
http://www.socialvelocity.net/tools/store/theory-of-change/
Resources: sources of free support
• Social Impact Analysts Association mentoring scheme
http://www.siaassociation.org/resources/siaamentoring-scheme/
• Pro Bono Economics
http://www.probonoeconomics.com/
• Project Oracle (matching researchers / interns with
London youth projects)
http://www.project-oracle.com/
Kerry McCarthy
e: [email protected]
t: @kclarity
www.linkedin.com/in/kerrymcc