Welcome to the Allegany Franciscan Ministries

2014 Major Grant Evaluation & Success Measures Webinar We will begin PROMPTLY at 11:00 a.m.

~~ Please MUTE your phone ~~ Please do not use a headset as they cause feedback that is disruptive to others on the call.

April 29 th , 2014

Reflection

Blessed are they who give without expecting even thanks in return, for they shall be abundantly rewarded.

Blessed are they who translate every good thing they know into action, for ever higher truths shall be revealed unto them.

Blessed are they who do God's will without asking to see results, for great shall be their recompense.

Blessed are they who love and trust their fellow beings, for they shall reach the good in people and receive a loving response.

Blessed are they who after dedicating their lives and thereby receiving a blessing, have the courage and faith to surmount the difficulties of the path ahead, for they shall receive a second blessing.

Adapted from from the book "Peace Pilgrim"

Agenda

 Welcome and Reflection  The Application  Evaluation – our philosophy  Success Measures & Outcomes  Definitions  Examples  Resources  Q & A

Presenters

Jessica Gonzalez

, Program Officer

Erin Baird

, Director, Grants and Technology

Shelley Robertson,

Evaluation Consultant

Our goals for this webinar

 To help you and your staff team…  Understand our evaluation philosophy, process & expectations.

  Better understand the difference between program outcomes & indicators; and the difference between those outcomes that are measureable & meaningful and those that are not.

Begin to conceptualize program success measures before your initial meetings with the Program Officer.

 Have the tools and knowledge to submit a successful application.

2014 Major Grants

 In each region, 4 to 6 organizations are likely to be funded.  Complete Major Grant Applications are due by May 29, 2014 at 12 pm (noon).

 Site Visits will occur in late July and early August.

 Board will finalize decisions in September.  Grant agreements will then be developed and signed; initial payments will be made between. November and January, depending on the project.

Major Grant Timeline

Hillsborough, Pinellas, Palm Beach, Martin & St. Lucie Counties Notes LOI Due Notification of our decision; decline or invite to submit full application Major Grant Full Application Technical Assistance Webinar Deadline to submit full application Site Visits & Meetings Decisions Announced Grant Period Begins

March 20, 2014 By April 25 Online application must be submitted by 12 pm (noon).

We will e-mail and send a letter to you.

April 30 May 29, 2014 July & August late September November 1, 2014 or later For those selected to submit a full application; please hold date (date subject to change).

Online application must be submitted by 12 pm (noon). We will work with applicants between September & December to finalize budgets & success measures.

2014 Allegany Franciscan Ministries Strategies

 Community education and outreach designed to increase knowledge and support behavioral changes.  Healthcare services and basic needs.

 Increased capacity and coordination of the healthcare delivery system.

 Advocacy and community efforts that educate, engage, raise awareness, or mobilize communities.

The Application – How to Apply

 Read the entire 2014 Major Grant Application Instructions document.

 Go to our web site, www.afmfl.org.

 Click on the “Current Partner” button.

 Log on to the online application system.

 Confirm or edit all contact information.

 Under “LOI” you will see “Application”.

 Click on “Edit Application”.

The Online Application – Some Basics

 Complete your responses in Word© and then paste into the appropriate field.

 Pay attention to character limits. Document will not save if you are over the character count.

 Don’t use acronyms or jargon.

 Save your work frequently.

 When uploading documents, click “save as draft” to finalize each upload.

 Before submitting, “Print Packet” and read/have someone else read.

The Application

The Application

Budget

Evaluation Philosophy

As a good steward, Allegany Franciscan Ministries is dedicated to assuring that our investments move us toward our mission. Our evaluation process is clear, simple, and focused on outcomes. The evaluation process is respectful of organizations with different size, sophistication, values, beliefs, and traditions. It balances qualitative and quantitative methods to inform and guide our decisions while improving

services provided by our grant partners.

Getting to measures

 An organization should articulate how the program leads to long-term change for people.

 Success measures document progress being made towards that long-term change.

Organization Name: Year: From (month/year):

Allegany Franciscan Ministries SUCCESS MEASURES FORM

Project Title: To (month/year): Outcomes

Please state one outcome

per row. Participants increase knowledge about how to cook healthy meals.

Coalition members will increase their advocacy activities Indicators

What is the specific statistic you will track to determine your success on this outcome? What data tool will you use to collect this data?

Activities

What are the key activities that will lead to each outcome?

85 percent of participants are able to identify four healthy cooking methods and up to 10 healthy food choices.

90 percent of members will take at least two advocacy actions during the six months after training.

Provide 10 hours of training to 20 participants in 5 two hour sessions.

Recruit and train a coalition of up to 20 advocates. Measurement plan

What is your measurement plan? For example, how often will you administer your data tool? How will you assess whether you were successful?

We will administer a paper pre and post test to participants. Members will report actions at monthly coalition meetings; staff will document efforts.

complete one page – and one page only - per year of grant

Definitions



Outcomes:

improvement and/or change in knowledge, skills, attitudes, or behavior. Outcomes may also refer to desired changes in a community in response to community-level and system-wide activities.



Indicators:

the specific items used to track a program’s success. Indicators describe observable, measureable characteristics or changes. An indicator should include the specific characteristic measured, what data tool is used, when it is measured, and who collects the data.

Definitions



Activities:

what a program does with its resources to fulfill its mission.



Measurement plan:

states any sampling strategies, how the data is entered, how you will analyze the characteristics to assess whether you were successful, and how data reported. For example, what counts as an increase?

Program outcomes

Program outcomes state the changes in clients’ knowledge, skill, attitude, or behavior.

Example: Program outcomes

 Clients increase their knowledge about how to reduce risk behaviors. 

An indicator could be that 80% of clients increase their knowledge about what are risk behaviors, consequences of risk behaviors, and how to reduce risk behaviors as measured by comparing post test to pretest scores on the curriculum-embedded tool.

 Clients increase their skills in preparing healthy food.  Clients demonstrate improved confidence toward changing behavior.  Clients experience decreased health risk behaviors.

Organizational capacity outcomes

Organizational outcomes state changes in capacity areas such as revenue, fiscal management, board development, operations, etc.

 The organization should be able to explain – although probably not measure – how the change in capacity will impact those they serve, such as improved service quality or more clients served.

Example: Organizational capacity outcomes

 The organization increases the capacity of its board of directors. 

An indicator would be that the organization meets current needs on 10 benchmarks on the board development section of the Capacity Benchmarking tool (see resources).

 The organization increases its fundraising capacity. 

An indicator could be that the organization achieves benchmarks on a capacity tool or that the organization (a) develops at least two new revenue streams, (b) reduces expenses by 5 percent in comparing 2010 to 2009, and (c) that 90% of staff report that the new system saves them time.

Choosing outcomes: meaningful and measurable

 What we want to see:  Fewer youth exhibit at risk behaviors due to undiagnosed mental health problems.

 What we don’t want to see:  Fewer youth abuse substances and show increased stability and life skill.

Choosing outcomes: meaningful and measurable

 What we want to see:  75% of recipients increase their intake of healthy food daily.

 What we don’t want to see:  75% of recipients report eating more fruits, vegetables and protein after starting our food delivery program.

Choosing outcomes: meaningful and measurable

 What we want to see:  “Organization A” will provide primary medical services to individuals and families with no health insurance.

 What we don’t want to see:  Provide safety net medical services for individual and families without health insurance or who cannot afford or are not eligible for conventional health care insurance.

Creating Outcome Statements:

The Outcome Formula • Step one: describe your client (i.e., age, gender, status) • Examples: at-risk middle school youth,

adults with mental illness, frail seniors, Hispanic teens, adults with disabilities, low income parents

25

Creating Outcome Statements: The Outcome Formula • Step two: identify the issue or topic you are addressing.

• Examples: healthy lifestyle choices,

employment, substance abuse, grades, sports, self-esteem, coping skills, health status.

26

Creating Outcome Statements: The Outcome Formula

Client Verb Adjective Examples: 1. At -risk middle school youth will decrease behaviors such as participating in gangs, getting in fights, and using drugs. 2. Hispanic teens will increase skills in job seeking. Will Increase Decrease Maintain Type Knowledge about Skill(s) in Attitude about Demonstrate Behaviors such as Example/topic 27

Outputs vs. Outcomes

 Outputs answer the question “how much” .

 Outputs include number of clients, service units, hours, and appointments.  Outcomes are client-focused.

 Outcomes relate to knowledge, skills, attitudes, or behaviors.

Indicators

Characteristics:  Observable and measurable.

 Client-focused.

 Unambiguous.

 State a time frame.

 Match.

 Understandable.

29

The Indicator Formula

Tells what you will measure (see table below), how, when, and who is measuring.

For this type of outcome Tell us….

Examples: Knowledge Skill Attitude What three main things the clients learn What, specifically, they can demonstrate What they will or will not feel Behavior What they will or will not do 30

Indicator Examples

Tool: • Pick one: survey, pretest and posttest, posttest only, interview, case records, observation form, or official statistics. Tell us the name of the tool and who created it.

• Examples: curriculum pretest and posttest, online survey created by USF,

case records kept by the agency, report cards from the school, ADL checklist provided by the State Department of Elder Affairs, survey provided by DJJ.

Target: • Put the percentage of your clients that you think will achieve the outcome. The target is NOT the client population. • Examples: 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90% .

31

The Indicator Formula

How examples: • how the tool is administered. When examples: • at the end of the program, beginning, every 30

days, 90 days after completing the program, etc.

Who examples: • program staff, non-program staff, outside agency

personnel, other agencies.

32

Indicator Examples

Knowledge: 

[Clients] will learn three elements of sportsmanship as measured by an end of the program posttest administered by staff.

Skill: 

[Clients] will demonstrate that they can prepare a resume, find an open job through the paper or online, and answer interview questions as measured by non-program staff observation during the job fair at the end of the program.

Attitude: 

[Clients] will state that they feel hopeful about a positive future as demonstrated by exit interviews conducted by program staff prior to leaving the shelter.

Behavior: 

[Clients] will not be adjudicated for gang activity, fights, or drugs while they are in the program as measured by DJJ statistics collected by the School Resource Officer.

33

Making It Measurable

 Direction of change.

 Area of change.

 Target population.

 Degree of change.  Time frame.

 As measured by  Baseline.

 Example: To decrease the percentage of 18 to 20 year old college students reporting binge drinking from 13 percent to 12 percent by 6/30/2015 as measured by the CORE survey.

Hints



Outcomes:

State one outcome per row. 

Indicators and measurement:

What is the specific statistic will you track to determine your success on this outcome? What data tool will you use to collect this data?



Activities:

What are the key activities that will lead to each outcome?



Measurement plan:

What is your measurement plan? For example, how often will you administer your data tool? How will you assess whether you were successful?

Evaluation Resources

In addition, Dr. Shelley Robertson is available at no charge to you for up to two hours. Dr. Robertson can assist you in identifying which of these resources might be most helpful for your specific project. She can also help as you develop or refine the outcomes to be tracked and reported, and determine appropriate indicators and measurement tools.

Next steps

 Log in to the grant administration system and begin the application.

 Contact Jessica with application questions earlier rather than later.

 Complete the Full Application by May 29, 2014, 12 pm (noon).

 Communicate with your team and discuss possible outcomes with your team.

Contact Information

 For all questions and technical support: contact Jessica Gonzalez at 727-507-9668 or [email protected]

 For help with outcomes and success measures, first contact Jessica Gonzalez and then our evaluation consultant, Shelley Robertson, 239-633-3241, [email protected]

Questions?