A guide to outcomes based reporting

Presented by: United Way of Tompkins County ( www.uwtc.org

) & Johnson Board Fellows - Frank Aburto and David Andros May 1, 2020

2

The current economic environment, increased community needs, and a crowded nonprofit sector are pressuring organizations to become more outcomes based

Competitive pressures

 Increasing community needs unmet by government services  Decreasing federal, state, and local funding  Increased number of nonprofit organizations

Challenges

 More focus on the impact nonprofits deliver to their clients  Resources (staff, funding, etc.) limit nonprofits’ ability to conduct long term program tracking

The program outcome model depicts the relationship between inputs, activities, outputs, and outcomes

Inputs Activities Outputs Outcomes Definition

▪

Example

Resources dedicated to or consumed by the program (Examples: money, staff time, facilities, equipment) ▪

Inputs for a parent education class include:

− The hours of staff time spent designing and − delivering the program Constraints on the program (Examples: laws, regulations, requirements for receipt of funding ▪ ▪ − − What the program does with the inputs to fulfill its mission

Parent education class activities include:

Classes on proper children’s nutrition, well-being, and development for parents A class on dealing with children approaching teenage years ▪ The direct products of program activities (Examples: number of classes taught, number of participants) ▪ − −

Parent education outputs include:

Number of parents taught Number of class hours provided ▪ Benefits or changes for recipients during and/or after participating in program activities (Examples: behavior, skills, knowledge, attitudes, values, condition ) ▪ − −

Parent education outcomes are:

Parents now have more knowledge of how to properly care for and support the development of new-born babies Parents utilize all available resources in Ithaca for health care Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), 1-9, 18. 3

In addition to determining outcomes, organizations must establish outcome targets and use indicators to track their progress

Outcomes

▪ Benefits or changes for recipients during and/or after participating in program activities ▪ Examples: behavior, skills, knowledge, attitudes, values, condition

Outcome indicators

▪ Observable and measurable “milestones” toward an outcome target ▪ Example: number and percent of teens that quit smoking directly after the program

Outcome targets

▪ The number and percent of participants that you want to achieve the outcome ▪ Example: an outcome goal of 500 teens (10% of teens in Ithaca) who quit smoking over the next year Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), 61. 4

The three levels of outcomes, a series, each contribute to a benefit or change that leads to the ultimate program outcome

Long-term outcomes Intermediate outcomes Initial outcomes

▪ Most closely related to and influenced by a program’s outputs ▪ Changes in participants’ knowledge, attitude, or skills ▪ Link a program’s initial outcomes to the longer term outcomes ▪ Changes in behavior that result from participants’ new knowledge, attitude, or skills ▪ Represent meaningful change for participants, often in their condition or status ▪ The most removed benefits a program can reasonably expect to influence Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), 32. 5

Outcome based reporting is a continuous process that allows organizations to demonstrate achievement of goals to donors while improving quality of service

1 2 3 Define service goals and specify an outcome indicator

 Specify an outcome indicator: − Identify what will indicate achievement of the goal (Example: improve reading comprehension) − Determining the statistic(s) used to calculate the level of achievement (Example: child improved his test score from a 1 to a 5)

Collect outcome data

 Things to consider: − Is there at least one indicator for each outcome?

− Does the indicator measure some important aspect of the outcome that no other indicator measures?

− Is the wording of each indicator sufficiently specific?

•

Capture less concrete outcomes through creative methods (Example: a survey given at the start and end of a program) Measure performance

 Data provides context in two ways: − Number of program participants achieving the goal − Percentage of program participants achieving the goal

4 Report outcomes and adjust program if necessary

  Share the measured performance with funders Review internally, align performance with service goal(s)

Repeating steps 1 through 4 will increase the accuracy of outcomes, while improving quality of service

Source: Based on “Eight Steps to Success” in United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996). 6

7

Step 1: Define service goals and specify an outcome indicator

Defining service goals

▪ Choose a program with a reasonably clear set of clients and clear methods for providing service

Specifying outcome indicators

▪ Identify outcome indicators for: − Initial: 0-6 months − Intermediate: 3-9 months − Long-term: 6-12 months ▪ In cases with less concrete outcomes, learn as you apply your outcomes evaluation plan ▪ Consider words like: “enhanced…,”“increased…,” or “altered…” ▪ Think about how the indicators for each time period are connected ▪ If this is your first time determining indicators do not seek initial perfection, goals can be general in initial stages and become more specific as the program develops

Example: An organization that focuses on reducing hunger would define their goal as – providing a feeding program that will give children access to quality food in an adequate quantity. An outcome indicator would be “altered” pattern of behavior such as increased amount of fruits and vegetables consumed per day or better eating habits at a young age.

Step 2: Collect outcome data

Questionnaires or surveys Purpose

▪ To quickly and easily collect data in a non threatening manner

Participant interviews Observation

▪ To fully understand a client’s impression and experience ▪ To gather accurate information about how a program operates

Advantages

▪ ▪ ▪ Inexpensive for any size group Easily comparable Sample questions already exist

Challenges

▪ ▪ ▪ May not receive careful feedback Wording of questions can create bias Does not always capture the full story ▪ ▪ Gathers a fuller range of information Can help to develop client relationships ▪ ▪ Time intensive Can be difficult to analyze and compare ▪ ▪ ▪ Adaptive Accurate Collected in real time ▪ ▪ ▪ Time intensive Can influence the way the program operates Difficult to interpret results Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), 81-88. 8

9

Step 3: Measure performance

Dealing with numbers Analyzing comments

▪ ▪ Tabulate the information (Example: Sum the ratings) Consider computing a mean, median, high, and low for each question ▪ ▪ ▪ Read all data Organize comments into similar categories (Examples: concerns, suggestions, strengths) Attempt to identify patterns, associations, and causal relationships

Example: The following individuals completed the basic computer skills course. At the beginning and at the end of the course they were asked to provide, on a scale of 1 to 5, their comfort level with computers.

Name Before After

Lisa Frank John Jeff Susan 1 2 2 1 3 3 5 4 5 5 Average improvement of 2.6

Median improvement of 2 With an overall 44% increase in computer skills

Step 4: Report outcomes and adjust program if necessary

Reporting evaluations

▪ ▪ Align the level and scope of information in report with intended recipients (Example: funders) Ensure employees have a chance to review and discuss the report before distribution  

Example of Evaluation Report Contents

 Executive Summary – one-page, concise overview of findings and recommendations  Program background – describe the service, community need, overall goal  List – outcomes, outcome targets, outcome indicators  Define the data collection methodology: − − Types of data/information collected How data/information were collected − − − How data/information were analyzed Limitations of the evaluation Interpretations and Conclusions Recommendations - regarding the decisions that must be made about the service Appendices: − − Data: numbers, observations, testimonials Logic model Source: Carter McNamara, Field Guide to Nonprofit Program Design, Marketing and Evaluation (Authenticity Consulting, LLC.).

10

A useful framework - the “Is there a positive impact” question is a simple way to evaluate a program’s outcomes

How much did we do?

# customers served (by customer characteristics) # activities (by type of activity)

How well did we do it?

% common measures (e.g., customer satisfaction) % activity-specific measures (e.g., % of clients completing the program)

Is there a positive impact?

# skills/knowledge # attitude/opinion # behavior % skills/knowledge % attitude/opinion % behavior # circumstance Source: Robert M. Penna, The nonprofit outcomes toolbox (New Jersey: John Wiley & Sons, Inc., 2011), 63.

11 % circumstance

12

Questions

Key definitions

▪ Inputs: Resources a program uses to achieve program objectives (Examples: staff/personnel, teaching aids, classroom time, donated food or funds) ▪ Activities: Implementation of the inputs to move towards the goals, what a program does with its inputs, (Examples: classes, sessions, feedings or distributing clothing) ▪ Outputs: Products of a program’s activities - raw data collected at the end of the activities (Examples: number of people trained/helped, meals provided or classes taught) ▪ Outcomes: The key piece of information to report, this is the discernable difference the program created for its clients (Examples: skills/knowledge gained (as measured by an exit survey), attitudes/behaviors changed, and improved living conditions) ▪ Outcome Targets: Numerical objectives for a program’s level of achievement on its outcomes (Intended number and percentage of clients targeted for a particular outcome in the next reporting period) ▪ Outcome Indicators: The specific items of information that track a program’s success on outcomes (Examples: gaining incremental skills/knowledge or small changes in behavior during the program or shortly after) ▪ Measure: Quantifying the outcome by using a baseline (before picture) to determine the performance you are achieving from your service (e.g., the % in skills/knowledge from the beginning to the end of the program) Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), xv. 13

Program logic model worksheet

Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), 44.

14

Outcome measurement framework

Source: United Way of America, Measuring Program Outcomes: A Practical Approach (United Way of America, 1996), 60.

15