Transcript NSSI Outcomes Presentation - National Strawberry Sustainability

National Strawberry
Sustainability Initiative
Reporting Project Outcomes
Part 1
Thinking about Your
Project Reporting
Results Based Program:
Outcomes & Impacts
Your project was selected because it was determined to have
the potential to make significant impact on sustainable
strawberry production.
• NSSI – a results based program; the grant selection, reporting,
and management
The Goal: To increase the probability that funded programs
will improve the skills, practices and technology of the
strawberry growers and industry that participate.
• Key components (Logic Model based):
•
•
•
•
Outputs
Outcomes
Impacts
Evaluation
Beginning with the “end” in mind
Projects should Begin with the End in Mind
• Determine first what the outcome is that the
participants will achieve – or you want them to
achieve.
• Design the program (outputs) to lead the participants
to those results/outcomes.
• “How will you get there?”
• Evaluation – how will you measure the change?
Logic Model: Beginning With the End in Mind
OUTCOMES
OUTPUTS
INPUTS
(Identify
Measurable
Technology/
Practice Results)
(Lead project
participants to the
desired results)
(Resources that will
strengthen
participant
achievement of
results)
Logic Model Implementing
Forward
INPUTS
OUTPUTS
OUTCOMES
From: Western Risk Management Education Center, http://westrme.wsu.edu/award-management/
Logic Model as a Tool to Construct Realistic Outcomes
First:
Identify the situation(s) that your project will help to address (pest
management, input reduction, food safety etc.) and consider if what you are
proposing will be well received by the target audience you have identified
(Producer/Industry Demand).
Then:
1.
2.
3.
Outcomes – Identify the changes in knowledge, actions, or
conditions/system you would like program participants to realize
Outputs – Identify activities (workshops, on-farm trials,
demonstrations, etc.) that will support and lead to participant
outcomes
Inputs – Resources that go into your program in the way of project
team, partners, collaborators, who provide expertise, research, money
etc.
An Outcome Defined….
Progressive Change in:
Knowledge*
Actions, Attitudes*
Condition, System, Environment
*Useful Tool: Search Bloom’s Taxonomy of Action Verbs
Or See:
http://www.clemson.edu/assessment/assessmentpractices/referencematerials/documents/Blooms%20Taxonomy%20Action%20Verbs.pdf
Outcomes Defined….
Changes in knowledge…
When project participants
Learn or become aware of new
fundamental or applied knowledge which
could include:
1. Practices and methods
2. Technology
3. Improved skills such as decision making,
evaluating
Outcomes Defined….
Change in Actions…
When project participants
Act upon what they have learned such as:
1. Implementing new practices, methods or
technology
2. Changing or modifying a practice
3. Adoption of improved skills
4. Direct application of knowledge gained
Outcomes Defined….
Change in Conditions, System, Environment…
A condition is changed because of
industry/growers wide-spread actions
1. Increased number of strawberry producers in
a state/region leading to increased
production and availability.
2. Increased local/regional strawberry
production or season extension to increase
availability to consumers.
3. Adoption of practices that leads to a
reduction in greenhouse gas emissions.
Etc.
Outcomes
Will Depend Upon:
- Knowledge of your target audience
- Why industry/producers will choose to participate
- Action and learning processes (project steps) that
help industry/producers achieve long term
outcomes
- Program resources including the team,
collaborators and expert presenters sufficient to
lead participants to the desired outcomes.
Producer Results or Outcomes
an example of progressive steps…
↓ Understand or have improved understanding of the issue,
the practice/technology/methods or the need to take action.
↓ Decide to take an action.
- Producer makes a commitment to write a business or marketing plan,
set up a record keeping system, or initiate other steps that will lead to
adoption
↓ Develop –customize a system or plan
↓ Implement or take an action that improves farm or business
i.e. – implementing a food safety plan, using a new
technology, etc.
In Summary…
Outcomes
• Measure of success for the participant
• Ability to change/incorporate/implement/adopt practices,
technology or skill, etc. as a result of participation in your project.
• Achievable and measureable within a reasonable
timeframe
• Sequential
• Identifies changes in condition or practice you seek for
the participant
• Specifies the degree of change required for participant
success
• Stated in a measurable way knowing methods you will
use to verify changes in practices, condition, etc.
Part 2
NSSI Reporting
NSSI REPORTING QUESTIONS
strawberry.uark.edu
NSSI Reporting: Outputs
“List your outputs in relation to the objectives of the
proposal. Include activities such as presentations
(professional, academic, outreach, stakeholders,
consumers, etc.), publications (professional, academic,
outreach, etc.), workshops, websites, blogs, etc. Include
target audience and number of people.”
What activities did you conduct to move your participants
towards project outcomes?
NSSI Reporting: Outcomes
“List the outcomes of your project, i.e. what change in
knowledge, attitude or actions did your outputs
accomplish in the short to intermediate time frame.
Include who was reached, how many were reached and
how outcomes were measured.”
• What did your participants/target audience do with the
project outputs?
• Increase knowledge? Initiate change in thought
processes? Change behaviors/practices?
• How did you measure this?
NSSI Reporting: Impact
“List the actual or anticipated impact of your project, i.e.
the big-picture effect of your project on the
industry/growers/region etc. (changes in condition,
environment, society) over the longer-term. Include how
impact were or will be measured.”
True impacts are harder to measure and may not be
evident until late in the project or after the project has
been completed.
Other NSSI Reporting Questions
• Summary of your project
• Objectives of your project
• Media contacts
• Internet media
• Problems delays or changes in plans
• Next accomplishments
• Attachments/Uploads
Underperformance
• Performance will be evaluated based upon your timely
and thorough reporting
• Underperforming projects risk termination.
• Performance metrics include;
• Timely activities as proposed
• Accomplishments towards your project goal
• Timely and appropriate use of funds
• Termination
• Unsatisfactory project performance will receive notice
• Notice will include corrective actions and reporting
• Project have one reporting period to correct
• Reminder: Project Deadline is June 30, 2014
NSSI Reporting and
Submitted Financials
• Project Progress Reports should be in
alignment with your Project Financial
Reports.
• The progress, outputs, and outcomes of
your project will be compared to what
funding has been used and its purpose.
EVALUATION
Evaluation
How will you measure the effect of your outputs
to ensure they are creating suitable/appropriate
participant change or any change at all?
• Protocols
• Methodologies
• Data collection
Measuring Outcomes
• Begin with the end in mind
• Begin with determining:
• The results that will be accomplished
• How they will be evaluated
• Design evaluation when developing project
• Key tip
• For each proposed outcome you should be able
to write an evaluation question
• Do this when you develop your funding proposal
• Validates that proposed results are actual results
• Helps design the evaluation from the beginning
NIFA Metrics for Measuring
Outcomes
Test your methods
 Will your evaluation tool(s) clearly
demonstrate progress participants are making
towards achieving the outcomes?
-
Changes in Knowledge?
Changes in Attitude?
Changes in Behavior/Approach?
Changes in Condition or the System?
How can you evaluate your outcomes?
1. Identify measurable expected outcomes.
2. Define criteria for success.
3. Develop evaluation tools to measure results.
– End-of-training evaluations
– Post-training evaluations
4. Evaluation distribution.
Defining Criteria for Success
Consider
• What will you actually be measuring?
• What is the standard for success?
• How many participants need to achieve ‘success’
in order to be successful?
• When should participants achieve results?
Evaluation Mechanics
Suitable Question Formats
– Multiple Choice – Yes/No
– Rating scales
– Open ended
 Simple for respondent to understand
 Easy to analyze for project teams with less evaluation experience
Variety of Methods:
• Pre/Post Tests
• Audience Response Systems (interactive polling Clickers or apps)
• Follow-up Evaluations (3 Months – Year)
• Mail or Internet Survey
•
•
•
•
Telephone Interview
Face-to-face interviews (in-person survey)
Dot survey
Informal Interview
Evaluation: Pre and Post Test
Pre-test Purpose
 Benchmark info on knowledge levels
 Information on participant background, past
experience, attitudes and needs for program
managers and instructors
Post-test Purpose
Measures
Knowledge Gain
Attitude Change
Actions participants expected to take as a result of
participating in workshop/demonstration/project.
Combining Pre/Post into Post
• “Please rate your knowledge PRIOR to attending today’s
workshop”
• “Please rate your knowledge AFTER attending this
workshop”
• “After attending today’s workshop, answer the following
questions:
• Please rate to what extent you have the knowledge, tools
and resources needed to …
• I better understand…”
 Likert Scale or Similar Method
Follow-Up Evaluation
Purpose:
To see what actions participants have
taken as a result of participating in your
project and how these actions have led to
benefits for their farm/business.
Evaluation Best Practices
Measuring results/outcomes successfully:
• Choose methods that balance best practices
with available resources
Combination of direct and indirect measures
• Direct measures (assignments, action plans etc.) yield stronger
evidence of change in knowledge and beneficial practices, but are
more time intensive and may take more expertise to evaluate.
• Indirect measures can give instant feedback but may not provide
real evidence of knowledge gained.
• Qualitative measures (like phone interviews or focus groups) that
may be more resource intensive could be accomplished through a
representative sample.
Evaluation Best Practices
Measuring results/outcomes successfully:
• Direct Interactions with Producers can increase the chance that
participants will implement beneficial practices/technology etc.
• Good follow-up can lead to collegial relationships that may bring
results beyond the initial results/outcomes that were initially
projected (Unexpected results).
• Good program design allows for learning and application to take
place in linked stages within a timeframe that encourages the
implementation/adoption of longer term results/outcomes.
When writing the evaluation questions, think about:
• How the information gathered will be used.
• The way you ask a question will influence how participants respond.
• When you ask a question will influence how participants respond.
Results/Outcome Resources
• The Rensselaerville Institute:
http://whatworks.org/displaycommon.cfm?an=1&subarticlenbr=266
“Outcome Funding: A New Approach to Targeted Grant making”
Based on the premise that grant makers are really investors in human gain, Outcome Funding addresses the key questions that the funder as
investor should ask: What am I buying? What is the probability I will get it? Outcome Funding begins with an abrupt and timely challenge to
the standard wisdom of the Request for Proposal process. The authors make the point that very little of the traditional proposal content
addresses results, favoring explanations of process instead. This book is not only persuasive on the need for change; it is clear and practical
on how to achieve it.
• Developing a Basic Logic Model For Your Program:
http://www.publichealth.arizona.edu/chwtoolkit/pdfs/logicmod/chapter2.pdf
I have become convinced that it makes a considerable difference if you do the outcomes before planning the activities. I definitely advocate
doing the outcomes first! I find that people come up with much more effective activities when they do. Use the motto, “plan backward,
implement forward.” Beverly Anderson Parsons, WKKF Cluster Evaluator
http://www.uwex.edu/ces/pdande/evaluation/evallogicmodelworksheets.html
• Ties to the Logic Model.
http://extension.oregonstate.edu/sites/default/files/for-employees/employee-resources/soarsmodules/logic_model/index.htm\
• Measuring Impacts for Sustainable Agriculture Programming
http://agsci.oregonstate.edu/sites/default/files/understanding_program_development_and_evaluation.pdf
Citations
• Information in these slides was borrowed and adapted from
Western Risk Management Education Center
• http://westrme.wsu.edu/award-management/
• How Will Your Verify? – Measuring Producer Results
• Extension Risk Management Education
• http://www.extensionrme.org/
• Oregon State University, Writing Powerful Impacts
• http://agsci.oregonstate.edu/research/writingimpacts
National Strawberry
Sustainability Initiative
Reporting Project Outcomes
[email protected]