Transcript Extension RME - Western Extension Risk Management Education

Evaluation Best
Practices
Results Based & Logic Model Programs
The Extension Risk Management Education Grants Program Team
Bob Craven, Director – Digital Center
Brad Lubben, Director – North Central Region
Jo Ann Warner – Assistant Director – Western Region
Kevin Klair – Digital Center
With help from:
Kynda Curtis, Utah State University
Gary Hachfeld, University of Minnesota Extension
Our Mission
“Helping America’s farmers
and ranchers manage risk”
Extension Risk Management
Education Program
 University of Delaware – Northeast Region
 University of Arkansas – Southern Region
 University of Nebraska - North Central Region
 Washington State University Extension – Western Region
 University of Minnesota – Digital Center
 USDA, National Institute of Food and Agriculture
Regional Centers
 North Central Center
University of Nebraska
 Northeast Center University of
Delaware Cooperative
Extension
 Southern Center University of
Arkansas Division of
Agriculture
 Western Center Washington
State University Extension
Return on Investment
• The return on investment of Extension RME
awarded grant dollars are the risk
management improvements that
agricultural producers make as result of a
project.
• The risk management results/outcomes that
agricultural producers achieve “are” the
measure of success of any project we fund.
Process Focused on Results/Outcomes
• The results based process, including the grant
selection, reporting, and management system,
– increases the probability that funded programs
will improve the risk management skills of the
farmers & ranchers who participate,
– is designed to improve, over time, the capacity
and quality of the programs we fund, and
– increases the opportunities to improve the
economic viability of the farm and ranch family
participants.
Evaluation Best Practices
Keep the focus on:
 Producer Results/Outcomes
 Project Team’s Accountability to
Measuring/Verifying Producer Results
 Commitment to helping producers reach
their risk management goals
Ability to Measure Outcomes
• Improve effectiveness by looking at your project
from start to finish in terms of results for
participants.
• Demonstrate participant achievement of risk
management tools and strategies by posting on the
public reporting system (final reports on
ExtensionRME.org & the Ag Risk Education Library)
• Move the bar higher for the effectiveness of risk
management education
Beginning with the “end” in mind
• Begin with the end in mind
– Determine first what the result/outcome is that
the participants will achieve.
– Design the program to lead the participants to
those results/outcomes.
• Do not wait till the end of the project to
determine what the outcomes might be.
Outcomes
- Will Depend Upon:
- Knowledge of your target audience
- Why ag producers will choose to participate
- Action and learning processes (project steps)
that help producers achieve long term outcomes
- Program resources including the team,
collaborators and expert presenters sufficient to
lead participants to the desired outcomes.
Outcomes –
A Systems Approach
The Proposal as a working document
i.
Application Stage
ii. Program Delivery
iii. Reporting
Outcomes & the Logic Model
Identifying and Articulating
Program Outcomes
Participant Outcomes
“Begin with the end in mind”
Or
“plan backward, implement
forward”
Jon Newkirk, Director, WSU Western Center
Beverly Anderson Parsons, WKKF Cluster Evaluator
An Outcome Defined….
Change in:
Knowledge
Actions or
Conditions
Outcomes Defined….
Changes in knowledge…
When project participants
1. Learn or become aware of new fundamental
or applied knowledge which could include:
a) Methods and techniques
b) Policy Knowledge
c) Improved skills such as decision making, life
skills or quality of life choices
Outcomes Defined….
Change in Actions…
When project participants
1. Act upon what they have learned such as:
a) Implementing methods and techniques
b) Changing a practice
c) Adoption of improved skills such as decision
making, life skills or quality of life choices
d) Direct application of knowledge gained
Outcomes Defined….
Change in Conditions…
A socio/economic condition is positively
changed because of participant actions
1. Information resources that improve
community and/or farming infrastructure
2. Technology transfer
3. Improved farm income for target audience
4. Decrease in farm foreclosures
A Logic Model can help you if you begin with
the end in mind….
A Logic Model can help to identify your
outcomes (changes in knowledge,
actions or conditions) and the
activities and partners that will
support the accomplishment of
outcomes for project participants
Logic Model as a Tool to Construct Realistic
Outcomes

First – Identify the situation(s) that your project will help to
address (farm foreclosures, high price of inputs, lack of record
keeping skills etc.) and consider if what you are proposing will
be well received by the target audience you have identified
(Producer Demand).
1. Outcomes – The identification of changes in knowledge,
actions, or conditions for program participants
2.
Outputs – (activities – curriculum development, workshops,
trainings etc.) that will support and lead to the achievement of
participant outcomes
3.
Inputs – Resources that go into your program in the way of
project team, partners, collaborators, who provide expertise,
research, money etc.
Logic Model Beginning With the End in Mind
OUTCOMES
OUTPUTS
INPUTS
(Identify
Measurable
Risk
Management
Results)
(Lead producer
participants to the
desired risk
management
results)
(Resources that
will strengthen
producer
achievement of
risk management
results)
Logic Model Implementing
Forward
INPUTS
OUTCOMES
OUTPUTS
Producer Results Outcomes Impacts
(Learning – Action – Condition )
A Hierarchy of Actions and Decisions
that Range from Improved
Understanding to Incorporating
Beneficial Risk Management Practices
Producer Results/Outcomes – an example of
progressive steps…
↓ Understand or have improved understanding of the risk issue,
tool or need to take action.
↓ Decide to take an action. Producer commitment to write a
business or marketing plan, set up a record keeping system,
conduct a risk analysis, or initiate other steps that will lead to
improved risk management strategies
↓ Develop business tools to strengthen economic capacity, such
as writing a marketing plan, business plan, farm transfer plan,
or performing a risk assessment etc.
 Implement or take an action that improves or optimizes their
profitability– implementing a marketing plan, using a record
keeping system etc.
In Summary…
Results/Outcomes
– Measure of success for the participant
• Ability to change/incorporate/implement/adopt risk
management practices as a result of participation in your
project.
– Achievable and measureable within a reasonable
timeframe during and following training.
– Sequential
• Identifies changes in condition or practice you seek for
the participant
• Specifies the degree of change required for participant
success
• Stated in a measurable way knowing methods you will
use to verify changes in condition, practice etc.
Optimizing Producers’ Chances for Success

First – Identify the situation(s) that your project will help to
address (farm foreclosures, high price of inputs, lack of record
keeping skills etc.) and consider if what you are proposing will
be well received by the target audience you have identified
(Producer Demand).

Then…Identify the Results/Outcomes – changes in
knowledge, actions, or conditions for program
participants.

Describe what outputs (activities – curriculum development,
workshops, trainings etc.) will support and lead to the
achievement of participant outcomes.


Create a realistic timetable for your actions (Project Steps)
Describe inputs – resources that go into your program in the
way of project team, partners, collaborators, who provide
expertise, research, money etc.
Logic Model Beginning With the End in Mind
OUTCOMES
OUTPUTS
INPUTS
(Identify
Measurable
Risk
Management
Results)
(Lead producer
participants to the
desired risk
management
results)
(Resources that
will strengthen
producer
achievement of
risk management
results)
Logic Model Implementing
Forward
INPUTS
OUTCOMES
OUTPUTS
Using the Logic Model to Identify
Results/Outcomes & Create a Plan of
Work (Project Steps)
Planning Backwards – Beginning with the End in Mind
Results/Outcomes
Outcomes
Activities/Tasks
Outputs
Responsible Entities & Roles
Inputs
Implementing Forward
Responsible Entities & Roles
Activities/Tasks
Inputs
Outputs
Results/Outcomes
Outcomes
Post/Short Term
Follow-up
Long Term or 3-6
Month Follow-up
Project Steps
Production Risk Management
Outputs or Outcomes?
• A series of insurance workshops will
be delivered to producers
• A crop/livestock insurance spreadsheet
will be developed
• Producers will increase purchases of
crop/livestock insurance
• Producers will evaluate crop/livestock
insurance decisions
Production Risk Management
Outputs or Outcomes?
• A series of insurance workshops will
be delivered to producers - output
• A crop/livestock insurance spreadsheet
will be developed - output
• Producers will increase purchases of
crop/livestock insurance – w. outcome
• Producers will evaluate crop/livestock
insurance decisions – s. outcome
Marketing Risk Management
Outputs or Outcomes?
• Producers will attend a marketing
workshop
• A marketing curriculum will be
written
• Producers will implement a
marketing plan
• Producers will write a marketing
plan
Marketing Risk Management
Outputs or Outcomes?
• Producers will attend a marketing
workshop – output
• A marketing curriculum will be
written - output
• Producers will implement a
marketing plan – s. outcome
• Producers will write a marketing
plan – w. outcome
Financial Risk Management
Outputs or Outcomes?
• Producers will calculate
financial ratios for case
farms/ranches
• Producers will understand
and interpret financial
ratios
• Producers will evaluate the
financial ratios for their
farm/ranch
Financial Risk Management
Outputs or Outcomes?
• Producers will calculate
financial ratios for case
farms/ranches - output
• Producers will understand
and interpret financial
ratios – w. outcome
• Producers will evaluate the
financial ratios for their
farm/ranch – s. outcome
Legal Risk Management
Outputs or Outcomes?
• Producers will attend a farm
program educational
meeting
• Producers will successfully
analyze farm program signup decisions
• Farm program fact sheets
and calculators will be
produced
Legal Risk Management
Outputs or Outcomes?
• Producers will attend a farm
program educational
meeting - output
• Producers will successfully
analyze farm program signup decisions – s. outcome
• Farm program fact sheets
and calculators will be
produced - output
Human Resources Risk
Management
Outputs or Outcomes?
• Succession planning materials
will be published
• A series of programs will discuss
the components of a farm
succession plan
• Producers will develop a
succession plan
• Producers will implement a
succession plan
Human Resources Risk
Management
Outputs or Outcomes?
• Succession planning materials
will be published - output
• A series of programs will discuss
the components of a farm
succession plan - output
• Producers will develop a
succession plan – w. outcome
• Producers will implement a
succession plan – s. outcome
Exercise – Writing Effective
Outcomes
1. Identify the risk situation(s) and the risk
topic areas that you want your project to
address
2. Build a set of measurable risk
management results/outcomes
Include both short and long term outcomes (results
that will happen after the project – within 3 months
to a year)
3. Determine if the outcomes are a good fit
for the target audience
Target Audience
 Are Outcomes suitable for audience being
served?
Realistic goals that can be achieved within specified
timeframe?
Linguistically and Culturally Appropriate?
Learning Styles and Environment?
 Reliability of Information
Science based content?
Appropriate for outcomes being proposed?
 How well do you know your target audience?
Relationship Capital?
Evaluation Methodology
Measuring Outcomes
Evaluation Mechanics
 Protocols for Administering Evaluations
– Preparing Workshop Facilitators
• Evaluation Procedures
Audience Preparation
 Methodologies
– Pre/Post/Follow-Up
– Ease of design and implementation
• Methods suitable to audience
• Framing questions appropriately to know if outcomes have been
achieved.
• Questions designed to measure outcomes should flow/follow from
the hierarchy of results/outcomes.
• Qualitative and Quantitative
– Open and close ended questions
 Gathering informal data
- Producer Stories and Testimonials
Evaluation Mechanics
 Tabulating the results
Who will collect the data?
How will data be collected?
When will data be collected?
What will be done with the compiled
data?
These questions need to be addressed before
administering your evaluations and are an integral part
of program design!
Evaluation Mechanics
Suitable Question Formats
– Multiple Choice – Yes/No
– Rating scales
– Open ended
Simple for respondent to understand
Easier to analyze for project teams
with less evaluation experience
Evaluation Mechanics
Multiple Choice – Yes/No
1. Assign a numeric value to each question
response
 For example code “yes” responses as “1” and
“no” responses as “2”)
2. Use the “count” function in Excel to
analyze the responses

Pie Charts can be used to graphically depict
responses
Evaluation Mechanics
Rating Scales
–
Allow respondents to indicate level of
agreement with a statement or to rate
level of importance.
Straightforward and easy to understand
Easier to analyze than other question
formats (because responses are already
in numeric form, it is not necessary to
recode them)
Shows how respondents feel on average
about the question being asked.
Evaluation Mechanics
Open ended questions
– Invite more detail from the respondent
– Require a little more effort from
respondent
Can identify challenges/barriers to
progress
Provide a complete story or testimonial
– Analysis may require identifying
common themes
Evaluation Mechanics
– Add yes’s and no’s
– Add up ranking #’s
– Provide an average of answers, rankings,
or range
– Organize comments into similar patterns
– Report results should be structured to
show that participants goals/outcomes
are being achieved.
• Numbers first, percentages second
Keep it simple!
NIFA Metrics for Measuring
Outcomes
Testing your methods
- Will your evaluation tool(s) clearly
demonstrate progress participants are
making towards achieving the outcomes?
-
Changes in Knowledge?
Changes in Attitude?
Changes in Behavior/Approach?
Changes in Economic or Social
Circumstances?
Pre Course Evaluation
Purpose
1. Benchmark info on knowledge levels
2. Information on participant background,
past experience, attitudes and needs for
program managers and instructors
Combining Pre/Post into Post
• “Please rate your knowledge PRIOR to
attending today’s workshop”
• “Please rate your knowledge AFTER
attending this workshop”
• “Please rate to what extent you have
the knowledge, tools and resources
needed to …”
• “I better understand”
Likert Scale or Similar Method
Post Course Evaluation
Purpose:
Measures
Knowledge Gain
Attitude Change
Actions participants expected to take as a
result of attending course
Post/Short Term
Follow-up
Follow-Up Evaluation
Purpose:
To see what actions participants have
taken as a result of attending your
course and how these actions have led
to economic and other benefits.
Long Term or 3-6
Month Follow-up
3 – 6 Month Follow-Up
3 – 6 Month Follow-Up Cont.
Ease of Design
 Do evaluation questions flow logically from
sequentially stated outcomes?
 Asked in a manner to obtain specific information
on level of accomplishment or barriers
encountered?
 Build on progress/commitments made from short
term or post evaluations?
 Usability of information?
Ease of Design
 Avoid “respondent fatigue”?
 Use definitive language to obtain accurate
responses?
 Avoid words like “generally” and “usually” – i.e. rather than
“Do you generally follow a business plan?”, “Do you follow
a business plan?”
Keep it short and simple!
Exercise – Writing an Effective
Evaluation
Post – Short Term
Follow-Up – Long Term
Within 3 months to a year
Evaluation Tools
Variety of Methods:
– Pre/Post Tests
• Audience Response Systems (Clickers)
– Follow-up Evaluations (3 Months – Year)
• Mail or Internet Survey
–
–
–
–
Telephone Interview
Face-to-face interviews (in-person survey)
Dot survey
Informal Interview
Evaluation Tools
Considerations:
Time Frame
Budget
Audience characteristics
Results/Outcome Based
Reporting
• Documents:
– Producer Progress towards achieving
goals (short and long term)
– Producer Stories and Testimonials
– Unexpected results/outcomes
• Provides Access To:
– RME Curriculum and resource materials
• Training methodologies
• Evaluation Tools
Integrating Results/Outcomes
into Project Reporting
 Summarizing Outcomes
 Project Overview
 Actual Results/Outcomes
 Comments/Stories/Examples of Success
 Input into other reports
Impactful Reporting
• Builds capacity for educators who work directly
with farmers and ranchers through…
• Access to risk management curriculum and resources
• Access to training methodologies
• Access to successful evaluation methods and tools
• Creates transparency
• Improves accountability
• Documents in a database format risk management
outcomes/impacts for farmers and ranchers.
• Available to federal and state agencies, government offices such as
the GAO and OMB, institutions and other funding partners.
• Keeps moving the bar higher for the effectiveness
of risk management education.
•
Evaluation Best Practices
Measuring results/outcomes successfully:
• Choose methods that balance best practices with
available resources
– Combination of direct and indirect measures
• Direct measures (assignments, action plans etc.) yield stronger
evidence of change in knowledge and beneficial practices, but are
more time intensive and may take more expertise to evaluate.
• Indirect measures can give instant feedback but may not provide real
evidence of knowledge gained.
• Qualitative measures (like phone interviews or focus groups) that
may be more resource intensive could be accomplished through a
representative sample.
Evaluation Best Practices
Measuring results/outcomes successfully
• A randomly sampled group, representing approximately 5
percent of participants submitting a post-meeting evaluation,
will be surveyed by phone. Participants will be asked whether
or not they have completed and implemented their
farm/ranch transition and personal estate plan. They will be
asked what challenges and obstacles they encountered during
the process, how the program assisted them in the process
and how useful the resource workbook was to them.
Evaluation Best Practices
Measuring results/outcomes successfully
– Direct Interactions with Producers can increase the chance
that participants will implement beneficial risk
management practices.
– Good follow-up can lead to collegial relationships that may
bring results beyond the initial results/outcomes that were
initially projected (Unexpected results).
– Good program design allows for learning and application
to take place in linked stages within a timeframe that
encourages the implementation/adoption of longer term
results/outcomes.
Evaluation Best Practices
• Interactive discussion on best
practices
– Successes
– Pitfalls
*Handout
In Summary
If the “Return on Investment” of grant
dollars awarded are the risk
management improvements that
producers will make…
Then measuring producer results
(outcomes) becomes an integral
component of program planning
and design, not an afterthought!
Thank You!
Bob Craven, Director – Digital Center, University of Minnesota
[email protected]
Brad Lubben, Director – North Central Region, University of Nebraska-Lincoln
[email protected]
Jo Ann Warner , Assistant Director – Western Region, Washington State University
[email protected]
Kevin Klair – Digital Center, University of Minnesota
[email protected]
Kynda Curtis, Utah State University
[email protected]
Gary Hachfeld, University of Minnesota Extension
[email protected]
Results/Outcome Resources
The Rensselaerville Institute:
http://whatworks.org/displaycommon.cfm?an=1&subarticlenbr=266
“Outcome Funding: A New Approach to Targeted Grantmaking”
Based on the premise that grant makers are really investors in human gain, Outcome Funding addresses the key
questions that the funder as investor should ask: What am I buying? What is the probability I will get it? Outcome
Funding begins with an abrupt and timely challenge to the standard wisdom of the Request for Proposal process.
The authors make the point that very little of the traditional proposal content addresses results, favoring
explanations of process instead. This book is not only persuasive on the need for change; it is clear and practical
on how to achieve it.
Logic Model Templates:
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodelworksheets.html
Developing a Basic Logic Model For Your Program:
http://www.publichealth.arizona.edu/chwtoolkit/pdfs/logicmod/chapter2.pdf
I have become convinced that it makes a considerable difference if you do the outcomes before planning the
activities. I definitely advocate doing the outcomes first! I find that people come up with much more effective
activities when they do. Use the motto, “plan backward, implement forward.” Beverly Anderson Parsons, WKKF
Cluster Evaluator