Transcript Mountains or Molehills?

Mountains or Molehills?
Reducing the perceived
burden of monitoring and
evaluating outcomes
Rebecca
Colclough
Dr Mark
Llewellyn
Workshop Aim
To provide participants with useful
background information and a practical step
-by-step guide to planning your evaluation
approach and developing a simple and
effective outcome monitoring questionnaire
for your project
STEP 1: Digging Around
• Use simple Google searches to do some
research into your area of interest
• What evaluation tools and techniques are
similar organisations using?
• Some common systems are available including
Outcome Stars and Quality of Life Indicators:
check if these could be appropriate
• Take stock each time you start a new project
• Talk to other people doing similar work
STEP 2: Set your Outcomes
“Outcomes are the difference your project
can make to your community or
beneficiaries”
Big Lottery
SMART Outcomes
Outcomes need to be SMART before you
can begin to work out how to measure
whether you have achieved them or not!
•
•
•
•
•
Specific
Measurable
Achievable
Realistic
Time-bound
–
–
–
–
–
–
–
Significant
Meaningful
Attainable
Relevant
Resourced
Results-based
Timely
Example 1
OUTCOME
• Increased confidence
SMART OUTCOME
• Older people who participate in the programme
will evidence an increase in their confidence
to speak up for themselves, which will be
shown immediately following their participation,
and will still be evident 6 months later
Example 2
OUTCOME
•
Increased use of recreational facilities
SMART OUTCOME
•
The number of people using the recreational facilities
will be evidenced to have increased by 10% in the first
year of the project and by 20% by the end of the
project
Or
•
The number of people who report using more than one
of the recreational facilities at the Centre will have
increased from 15% to 20% in the first year of the
project and will reach 25% by the end of the project
STEP 3: Evaluation Questions
• Turn your SMART outcomes into
evaluation questions
• Remember, evaluation is COMPARATIVE
by nature
past with present
project 1 with project 2
method a) with method b)
before and after
GOOD evaluation questions
• Easy to understand
• Written in plain language (no jargon or
abbreviations)
• Not too long
• Only ask one thing per question
• Don’t ask ‘leading’ questions
• Offer mutually exclusive and exhaustive
categories
• Can be closed or open-ended
• Can use recognised ‘scales’
Ordinal Scales (LIKERT)
Excellent
Very good
Fair
Poor
Strongly
agree
Agree
Neither
agree nor
disagree
Disagree
Strongly
disagree
Always
Often
Sometimes
Rarely
Never
None
Very mild
Mild
Moderate
Severe
Completely
satisfied
Very
satisfied
Somewhat
satisfied
Somewhat
dissatisfied
Very
dissatisfied
Completely
dissatisfied
Developing Evaluation
Questions
SMART Outcome:
Older people who participate in the
programme will evidence an increase in
their confidence to speak up for
themselves, which will be shown
immediately following their participation, and
will still be evident 6 months later
…continued
Evaluation Metric:
What % of beneficiaries report an
improvement in their confidence to speak up
for themselves?
…continued
Evaluation Question:
To make this evaluation question comparative you
must ask it at least twice: BEFORE and AFTER your
project intervention (in this case the question must
also be asked 6 months later as well)
STEP 4:
Develop your Evaluation Plan
Develop an evaluation plan which details all
of your intended project outcomes and
evaluation questions, and sets out how you
will collect the data to monitor your
achievement of these, alongside wider
service monitoring and evaluation
Evaluation Plan: Contents (i)
• Data reporting deadlines
• Service monitoring and evaluation
questions (RBA ‘How much are we doing’
and ‘How well are we doing it’?)
• SMART outcomes (RBA ‘Is anyone better
off’?)
• Outcome evaluation questions
Evaluation Plan: Contents (ii)
• Use of monitoring and evaluation data
To meet funding requirements?
To influence social policy?
To gain further project funding?
To scope a particular issue or problem?
• Evaluation participants (who you will
collect the data from)?
Beneficiaries?
Volunteers?
Stakeholders?
Evaluation Plan: Contents (iii)
• The type of data you will collect
• The data collection methods you will
use
Evaluation Plan: Contents (iv)
Analysis
How you will analyse your results
Reporting
What kind of reports you will need to produce from your
results
Dissemination
How your findings will be shared to have the
greatest impact
Unfortunately there won’t be time today to go into these areas in
more detail
Quantitative / Qualitative
Quantitative data is essentially
NUMBERS
Qualitative data is essentially
WORDS
Quantitative
• Learning little from
many: can make
generalisations
• Standardised
responses
• Non-deliberative
• Numerical
• Based on sample
sizes
Qualitative
• Learning a lot from a
few: hard to make
generalisations
• Open responses;
conversational and
flexible
• Deliberative
• Non-numerical; ideas;
beliefs; meaning
• Based on experience
What kinds of data can be
collected?
“Everything that can be counted
does not necessarily count;
everything that counts cannot
necessarily be counted”
Albert Einstein
Data Collection Methods
Different types of data require different
data collection methods to be used
Qualitative
• Interview
• Focus group
• Digital story
Quantitative
• Survey
• Routine monitoring
data collection (stats)
Ensuring Quality Data
• Making sure you collect the right data at
the right time and in a standardised way is
essential to making sure the data you
collect, analyse and share is good quality
• Different ‘methods’ use different ‘tools’ to
do this, such as a questionnaire, a
structured interview schedule or a focus
group discussion guide
What is an Outcome TOOL?
An outcome tool is something designed to
ensure you can assess or describe change in a
consistent and standardised way. This
will produce data which is of a better quality
Good quality data makes it possible to draw
together information on a number of beneficiaries
and give an overview of the change achieved for a
whole project or group of projects
Developing an Outcome Tool
for your Project
Measuring outcomes for beneficiaries
Simple and Effective
• Make the TOOL as useful as possible
• Simple design which is easy to follow
• Gather data which answers your
evaluation questions
• Use recognised SCALES where
appropriate to ensure quality
• Think about when and how the TOOL is
used
Service Quality
Take the opportunity to
assess the QUALITY of
your service along with
beneficiary outcomes
Free Text Space
• It is always useful to include free text
space for beneficiaries to say more about
something
• Whilst not everyone will complete free text
space, those who do will often have
something constructive or positive to say
• You can get great quotes for future use!
Capture project ‘champions’
• Make sure there is space for beneficiaries
to fill in their contact details if they wish
• Take the opportunity to ask if they would
mind being interviewed or contributing
further to your project evaluation
• User voice is exceptionally powerful so
don’t risk losing it!
Simple Structure
• Order the questions well according to any
natural timelines
• Move from familiar to unfamiliar
• Don’t ask sensitive questions near the
beginning
• Limit the number of questions
• End with easy questions
• Make the completion instructions clear
Data Collection Points
“Timing has a lot to do with the outcome of a
rain dance”
Cowboy Proverb
Data Collection Points
• Baseline
• Progress
• End of Intervention
• Impact
You must ask exactly the same questions each
time when trying to evaluate outcomes. Service
monitoring and quality questions can be one-off
Completion Method
•
•
•
•
•
Face to face
By post
By the client alone or with support
By a proxy (for people with dementia)
By a volunteer
Standardising your data collection method
helps ensure the data gathered is high quality
Useful Resources
www.ces-vol.org.uk
http://www.biglotteryfund.org.uk/er_eval_explaining_the_difference.pdf
Link to a document called ‘Explaining the difference your project makes’
http://www.evaluationtrust.org/evaluation/evaluate
http://www.worldbank.org/ieg/ipdet/presentation/M_05-Pr.pdf
Power point module on writing good evaluation questions
http://www.keene.edu/crc/forms/designingsurveysthatcount.pdf
Slides on good practice in writing evaluation questions
http://www.homelessoutcomes.org.uk/resources/1/Guide%20to%20Outcomes
%20Tools%20Second%20Edition.pdf
Really useful publication describing outcome tools (written for the homelessness sector
but more widely applicable)