LOGIC MODELS AND OUTCOME MEASUREMENT

United Way Community Investment Process Training

Workshop Overview

I.

Welcome and Introductions II. Basics of Outcome Measurement III. Developing an Effective Logic Model IV. Evaluating your Logic Model V. Wrap-Up

What are your other expectations?

Logic Models

What difference are you making? How do you know it? What is the value of your program?

Theory of Change Program Matrix Logical Framework

Logic Model Conceptual Chain

5 • Goal • Theory • Target Population

Outputs Short-term Outcomes Mid-term Outcomes Long-term Outcomes Activities Inputs

What Exactly is An Outcome?

Short-Term

Increased Knowledge

Mid-Term

Change in Behavior

Long-Term

Change in Condition

What is Outcome Measurement?

Outcome measurement is the regular, systematic measuring of progress toward intended outcomes.

• Internal • External 10

Measuring Program Outcomes: Eight Steps to Success

8. Use Your Outcome Findings 7. Improve Your System 6. Analyze and Report Your findings 5. Try Out your Outcome Measurement System 4. Prepare to Collect Data on Your Indicators 3. Specify Indicators for Your Outcomes 2. Choose the Outcomes You Want to Measure 1. Get Ready 11

12

What happens with data that we are measuring?

Logic Model Process Questions

• What outcome do you want to achieve?

• What activities will it take to get you there?

• What resources (inputs) will you need to complete the activities?

• How will you know when you have achieved your outcome? (How will you measure?)

Logic Model Foundation

GOAL

Describe the Goal of your program: The goal should be client focused - broad statement of intended change which identifies the target population.

 Broad statement of desired condition  Client focused  Describe intended change of target population

Example: Working poor individuals and families living in the Roanoke Valley will attain gainful employment and become self-sufficient.

14

Logic Model Foundation

TARGET POPULATION

Describe the population of clients that will be served.  Age group of program targets  Description of population (i.e. gender, race, disability, socio economic status, etc)  Unique characteristics of the target population (i.e. needs, condition, barriers, etc)   Geographic location or community of focus of program targets May include a profile of a ‘typical’ client 15

Inputs • Human • Financial • Organizational • Community •

If I have access to these resources, I can conduct….

Activities • Processes • Events • Programs •

If we conduct these activities, we should see….

Outputs • Amount of Service delivered • Number of Product •

If the target population receives these services and products, they should demonstrate…

Outcomes • Indicates a measurable change to the target • Increased behavior • Increased knowledge • Increased skills •

If these benefits are realized, then we have IMPACT!

Logic Model Components

INPUTS

Inputs are resources needed for operation of the program, such as:  Staff positions & staff time  Volunteers & volunteers time  Facilities  Equipment and supplies  Funding and funding sources  Collaborative partners 17

Logic Model Components

ACTIVITIES

Activities, processes or events undertaken with the inputs  Services a program provides for its participants to fulfill its purpose  No quantities (percentages, numbers of participants, numbers served, etc.) •

Examples: Feed and shelter homeless families

•

Provide job training

•

Educate the public about signs of child abuse

•

Counsel pregnant women

•

Create mentoring relationships for youth

18

Logic Model Components

OUTPUTS

The direct results of program strategies, activities, processes, and/or events.

 Usually described in terms of size and scope of the services or products delivered or produced by the program.

 Outputs indicate whether or not a program was delivered to the intended audiences at the intended “dose.”

Example: the number of classes taught, meetings held, materials distributed, counseling hours, program participation rates, or total service delivery hours.

19

Logic Model Components

OUTCOMES

Outcomes are specific statements of the desired changes in the lives of a particular group (the target population).  All of the outcomes relate to the clients listed in the

population target

 All of the outcomes relate to the program’s components listed in the

strategies

 All of the outcomes state

desirable benefits significant changes

in or to be achieved by the target population  All of the outcomes are

stated positively appropriate

for the target population and are  All of the outcomes follow a (if….then)

logical progression

20

Logic Model Components

INDICATORS

Indicators track a program’s success on outcomes.

 All of the indicators are

clearly linked

to and

measure progress

toward the associated outcome.

 Each indicator is stated with a

target number and target percentage

of the total population that is expected to achieve the outcome.

 All the indicators are based on the

target population

.

 All the indicators are

observable

will

measure some aspect

of the outcomes they are attached to.

21

Logic Model Components

OUTCOMES vs. INDICATORS Outcome:

Participating students succeed in school.

Indicator:

The # and % of participating students who pass all their core subjects.

Outcome:

Job training graduates become gainfully employed.

Indicator:

The # and % of graduates who secure full-time employment.

Outcome:

Parents read to their preschoolers more often

Indicator:

The # and % of parents who read to their preschoolers everyday for a minimum of 30 minutes per day.

22

TEST YOUR UNDERSTANDING!

Build the logic model for your program

What are Outcomes?

Outcomes are specific statements of the desired changes in the lives of a particular group (the target population).  All of the outcomes relate to

the clients

  All of the outcomes flow logically from the

strategies

All of the outcomes state

significant changes

in or

desirable benefits

to be achieved by the target population  All of the outcomes are

stated positively

and are

appropriate

for the target population  All of the outcomes follow a

logical progression

(if….then) 24

Client-focused Outcomes vs. Process-focused Outcomes

OUTCOMES: Should be client-focused.



Client-focused Outcomes

Seniors increase their involvement in social activities  to improve family functioning   towards physical developments on regarding family health, economics, and technology   membership of committed members and poison safety attitudes and behaviors. 

Process-focused Outcomes

To make social activities available to senior citizens  counseling  disadvantaged children  classes   and alcohol 25

Outputs vs. Outcomes

Program Example

Comprehensive child care Treatment for youth abusing drugs and alcohol Congregate meals for seniors

Outputs (amount of work) might be … Intended outcomes (results of the work) might be …

• Number of children registered • Number of days of care • Number of counseling sessions conducted • Number of youth completing treatment • Number of seniors enrolled •Number of meals served • Children exhibit age appropriate social skills • Children are ready for kindergarten • Participating youth know the risks of substance abuse • Participants remain clean and sober for 6 months • Seniors eat at least one nutritious meal each day • Seniors interact socially with their peers

26

Writing Effective Outcome Statements

Some tips:

• Specifically

states what will be true

of the clients/participants as the result of your program’s influence on their lives • Outcomes

focus on the participants/beneficiaries

, not on the program • Statement contains

only one intended change

benefit or • Statement is very intentional (

specific

) in the desired change or benefit, and not simply that something is now improved, increased, decreased, enhanced, or expanded 27

Developing Your Outcomes

Questions to ask yourself

• What do we want to be true of participants because of their involvement with our program? … What do we want to be able to say about them?

• If we succeed with a participant (or don’t), what has changed (or hasn’t)?

• If we carry out these strategies, then what do participants believe, know, have, or do as a result? … And what benefit or change flows from that?

28

What are Indicators?

INDICATORS

Indicators track a program’s success on outcomes.

 All of the indicators are

clearly linked

to and

measure progress

toward the associated outcome.

 Each indicator is stated with a

target number and target percentage

of the total population that is expected to achieve the outcome.

 All the indicators are based on the

target population

.

 All the indicators are

observable

will

measure some aspect

of the outcomes they are attached to.

 Indicators are SMART: Specific, Measurable, Attainable, Relevant, and Time Bound 29

Outcomes & Associated Indicators

Outcome:

Benefits for participants due to their involvement with a program.

Indicator:

The specific information collected to track a program’s success on outcomes.

e.g., The number and percent of …

30

Outcomes & Associated Indicators

Outcome:

Participating students succeed in school.

Indicator 1:

70 of the participating students (78%) will pass all their core subjects.

Indicator 2:

60 of the participating students (67%) will advance to the next grade level.

Outcome:

Job training graduates become gainfully employed.

Indicator 1:

52 of the graduates (69%) will secure full-time employment.

Indicator2:

40 of the employed graduates (77%) will remain employed for at least six months.

Outcome:

Parents read to their preschoolers more often

Indicator:

280 parents (80%) will read to their preschoolers everyday for a minimum of 30 minutes per day during the Fall 2007 semester.

31

Common Misconstructions when Writing Indicators

 Vague Indicators

e.g. Job Promotions 50 clients (64%) will receive job promotions within 3-months of completing the course.

 Not Written Measurably

e.g. Able to converse better 42 ESL graduates (91%) will demonstrate improved English language conversation skills by graduation.

32

Common Misconstructions when Writing Indicators

 Listing Measurement Documents

e.g. Attendance Forms 33 students (67%) will have perfect attendance during the Fall 2007 semester.

 Listing Measurement Plans

e.g. Review of Test Scores 75 students (85%) will pass all of their core subject final exams.

33

Common Misconstructions when Writing Indicators

 Listing Outputs or Strategies (process-focused)

e.g. Clients attend all 10 workshops 45 clients (74%) who attended all 10 work shops will demonstrate improved credit score ratings.

e.g. Six trainings are offered during the year 123 training participants (90%) will show increased knowledge of the subject based on pre/post testing.

34

Common Misconstructions when Writing Indicators

 Listing Outcomes

e.g. Clients change behaviors and make healthier choices 58 clients (89%) show improved nutritional intake at the 3-month re-assessment

 Not Associated to Outcome

e.g. Homework assignment completed (measuring self confidence) 12 students (72%) who complete the leadership training course will self-report improved self confidence.

35

Selecting Outcomes You Want to Measure

Pick the fewest number of outcomes that, as a group, will yield information for three key purposes …

1. Assure that you are achieving meaningful benefits for the intended beneficiaries

– Which outcomes are the most important for program participants to achieve?

2. Communicate the value of your program to key audiences

Which outcomes do key audiences, including funders, care most about? Which best tell your story?

–

3. Guide program managers and staff in increasing the effectiveness of your program

– Which outcomes will be the most helpful in identifying where the program is, or is not, being successful?

36

Evaluating Your Logic Model for Improvements

Evaluating your Logic Model

Ask the following general questions:

Are the outcomes really outcomes?

Do they describe changes or benefits for program clients/participants?

Is the logic logical?

Does it make sense that the strategies will lead to the initial outcomes, which will lead to the next outcomes, etc.?

Is the long-term outcome meaningful for program participants?

Have clients/participants experienced a change or benefit that makes a real difference in their lives?

Is the long-term outcome reasonable?

Is reasonable to think that the program actually can influence this outcome to a reasonable degree?

Is the long-term outcome plausible?

Will clients actually achieve this outcome?