Title I School Improvement 1003(a) Grant Writing Workshop Building a Logic Model With SMART Objectives Anduamlak Meharie Office of Research.
Download ReportTranscript Title I School Improvement 1003(a) Grant Writing Workshop Building a Logic Model With SMART Objectives Anduamlak Meharie Office of Research.
Title I School Improvement 1003(a) Grant Writing Workshop
Building a Logic Model With SMART Objectives
Anduamlak Meharie Office of Research
Logic Model
“A logic model is a systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan, and the changes or results you hope to achieve”
W.K. Kellogg Foundation Logic Model Development Guide.
“A logic model’s purpose is to communicate the underlying ‘theory’ or set of assumptions or hypotheses that program proponents have about why the program will work, or about why it is a good solution to an identified problem” http://www.insites.org/documents/logmod.htm
• • •
Logic Model: Purpose
Design and planning – To clearly communicate the central goal of an initiative, why it is important, and what will result from it. – To select specific and measurable objectives expected to lead to the desired result.
– To identify activities and resources needed to meet selected objectives. Implementation – To identify and collect data need to effectively monitor initiative activities and improve programming. – To create a common reference point for everyone involved in the initiative.
Evaluation – To help determine if activities are being implemented as planned. – To help determine if the initiative is making progress. – And, ultimately, to determine if the initiative has led to the desired result.
Logic Models: Components
• • • • • • •
Goal(s) Objective(s) Activities Input/Resources Outputs Outcome(s)/Impact(s) Indicator(s)
Logic Model: Goals vs. Objectives
• Goal: A broad statement of intended outcomes for a program, line of business or an organization.
– Goals are typically intended to guide the formation of (more specific) objectives that can be linked to goals. ( McDavid and Howthrorn, 2006) •
Example
–
To decrease the dropout rate of at-risk students in grades 9-12 at Washington High School.
Logic Model: Goals vs. Objectives
• • Objective: A statement of intended outcomes that is focused and time-specific (that is, achievable in a specified time frame). (McDavid and Howthrorn, 2006)
Example
–
By September 2013, at least 80% of all at-risk students in grades 9-12 will be assigned an adult advocate who will be responsible for meeting the student at least twice a week to address academic and social needs, and communicate with families.
–
By May 2014, all core curriculum teachers at Washington High School will receive evidence- based professional development on (1) how to help struggling students, (2) differentiated instructional methods, and (3) managing student behavior.
S.M.A.R.T. Objective
• Specific • Measurable • Attainable • Result-oriented and relevant • Time-bound
Specific Objective
• Is your objective Specific? • If you have a specific objective, you should be able to answer the following six questions: – – What: What do you want to accomplish?
Why: What are the reasons, purpose or benefits of accomplishing the objective?
– – Who: Who is involved? Who are the stakeholders?
Where: Where is it going to happen?
– Which: Which attributes are important? (Requirements and constraints; risk and protective factors).
Measurable Objective
• Is your objective measurable? • If your objectives are specific and measurable, you should be able to establish concrete criteria for measuring progress toward the attainment of each.
– – How will you know when it is accomplished? What indicators will you look for to measure progress and success? – Are data to measure your progress and success readily available? Or do you need to develop new measures, e.g., surveys, focus groups, etc…?
Attainable Objective
• Is your objective attainable? • Given resources available to you, the objective must be realistic and attainable. – Do you have sufficient financial capacity to accomplish your objective?
– Do you have personnel with sufficient abilities and skills to accomplish your objective (including evaluation)?
– Do you have sufficient time to accomplish your objective?
– What other types of resources do you need to attain your objective (i.e., technology, space, equipment, etc…)?
Result-Oriented and Relevant
Objective
• Is your objective result-oriented and relevant? • Ultimately, your objectives should help you achieve your major project goal(s). – Is it worthwhile? – – Is this the right time? Does this align with other efforts/needs?
Time-Bound Objective
• Is your objective time-bound? • Objectives should have starting points, ending points, and fixed durations.
– – When will you achieve this objective?
When will you undertake activities to achieve our objective? – When can you expect to see some short-term outcomes?
•
S.M.A.R.T. Objective: Example
Goal 1: Within two years after the coaching program for at-risk youth at Washington High School is established, the dropout rate of high school students will fall below 1% at grades 10, 11, & 12. • Objective 1.1: As a result of the program attendance rates at Washington High School will be at 100%.
Objective Specific?
As a result of the program attendance rates at Benedictine High School will be at 100%.
Attendance rates at Benedictine High School will exceed 97% one year after the apprenticeship/ coaching program is established.
Attendance rates for at-risk youth at Benedictine High School at grades 10, 11, & 12 will exceed 97% one year after the apprenticeship/ coaching program is established.
Yes No, revise (X) Yes No, revise (X) Yes (X) No, revise
Measurable?
Yes (X) No, revise Yes (X) No, revise Yes (X) No, revise
Is the objective… Attainable?
Yes
Result-oriented/ Relevant?
Yes (X) No, revise (X) Yes (X) No, revise Yes (X) No, revise No, revise Yes (X) No, revise Yes (X) No, revise
Time-Bound?
Yes No, revise (X) Yes (X) No, revise Yes (X) No, revise
S.M.A.R.T. Objective: Example
• Objective 1.1: As a result of the program attendance rates at Benedictine High School will be at 100%.
• Objective 1.1: Attendance rates for at-risk youth at Benedictine High School at grades 10, 11, & 12 will exceed 97% one year after the apprenticeship/ coaching program is established.
S.M.A.R.T. Objective: Exercise
1. Choose one of the objectives listed on your grant application. 2. Using the worksheet provided, examine your objective to ensure that it is S.M.A.R.T. with respect to the major goal(s) of your project. 3. Revise as needed until the objective meets all five (S.M.A.R.T.) criteria.
S.M.A.R.T. Objective
Worksheet Goal: Objective: Objective
Specific?
Yes No, revise Yes No, revise Yes No, revise Yes No, revise Measurable?
Yes No, revise Yes No, revise Yes No, revise Yes No, revise
Is the objective…
Attainable?
Yes No, revise Result-oriented/ Relevant?
Yes No, revise Yes No, revise Yes No, revise Yes No, revise Yes No, revise Yes No, revise Yes No, revise Time-Bound?
Yes No, revise Yes No, revise Yes No, revise Yes No, revise
Logic Models: Components
• • • • • • •
Goal(s) Objective(s) Activities Input/Resources Outputs Outcome(s)/Impact(s) Indicator(s)
Logic Models: Components
• • • • • Activities: the actions that are needed to meet your objectives.
Resources: inputs needed design and implement activities.
Outputs: the tangible and direct results of activities.
Outcome(s): specific, attainable and measurable changes that are likely to occur as a result of activities. Indicator(s): data to monitor implementation, and measure progress as well as success of initiative.
1.2
1.3
Logic Model
Goal 1: Objectives
(Specific, Measurable, Achievable, Relevant, and Time-bound)
1.1
1.1.1
Activities
(what we do to achieve objectives/goals)
Inputs/Resources
(human, financial, technology, etc… resources needed for activities)
Outputs Outcomes Indicator/Data Source
(tangible and direct products of activities that lead to desired outcomes) (short-, medium-, and long- term desired results of activities) (indicators we look for to measure implementation, progress and success)
1.1.1 1.1.1 1.1.2 1.1.3
1.2.1 1.2.2 1.2.3
1.3.1 1.3.2 1.3.3
1.1.2 1.1.3
1.2.1 1.2.2 1.2.3
1.3.1 1.3.2 1.3.3
1.1.2
1.1.3
1.2.1 1.2.2 1.2.3
1.3.1 1.3.2 1.3.3
Logic Model: Example
Goal 1: To decrease the dropout rate of at-risk students in grades 9-12 at Washington High School. Objectives 1.1 By September 2013, at least 80% of all at-risk students in grades 9-12 will be assigned an adult advocate who will be responsible for meeting the student at least twice a week to address academic and social needs, and communicate with families. Activities
1.1.1 Identify at-risk students 1.1.2 Identify, recruit, and train adult advocates 1.1.3 Clearly define the roles and responsibilities of all stakeholders (guiding document) 1.1.4 Assign 10 or less at risk students per advocate
Inputs/Resources Outputs Outcomes Indicator/Data Source
-School staff, community members, and/or social workers to work as advocates -Monetary compensation 1.1.1 Number of at-risk students identified by grade level 1.1.2 Number of advocates; list of 1.1.1 By June 2013, at-risk students in grades 9-12 will indicate a greater sense of school membership and involvement.
-Student survey to measure effect on sense of school membership and involvement (maybe pre/post). for trainers and advocates. -Cost of training facility, printed materials, meals for participants. training dates, locations, and participants; 1.1.2 By June 2013, at-risk students in grades 9-12 will have -Student disciplinary and attendance record reduced risky behaviors and 1.1.3 A document that elucidates the roles and absentee rate. -Student academic record. -Staff to document, monitor and evaluate implementation and progress.
responsibilities adult advocates, students, school staff, parents/guardians, and other stakeholders 1.1.4 List of advocates and their assigned at risk students; number and average duration of meetings with each 1.1.3 By June 2014, at-risk improved grades. -List of all at-risk students students in grades 9-12 will have and those who participated in the adult
Ultimate outcome (Impact): The
sum total of all objectives- will result in a decrease in dropout rate of at-risk students in grades 9-12. advocate program to ascertain the % of at-risk students reached. -Participant questionnaire about the quality and usefulness of training and guiding document. student; number and average duration of meetings with parents of at-risk students -Student and parent survey to gather feedback about the quality and usefulness of the adult advocate program.
Logic Model: Exercise
• • • •
Insert the objective you developed earlier in the logic model template.
Identify activities and inputs/resources needed to achieve the objective. Explicitly state expected results of activities in terms of outputs and outcomes. Identify indicators/data sources that will allow you to document and monitor implementation as well as measure progress and success.
1.2
1.3
Logic Model
Goal 1: Objectives
(Specific, Measurable, Achievable, Relevant, and Time-bound)
1.1
1.1.1
Activities
(what we do to achieve objectives/goals)
Inputs/Resources
(human, financial, technology, etc… resources needed for activities)
Outputs Outcomes Indicator/Data Source
(tangible and direct products of activities that lead to desired outcomes) (short-, medium-, and long- term desired results of activities) (indicators we look for to measure implementation, progress and success)
1.1.1 1.1.1 1.1.2 1.1.3
1.2.1 1.2.2 1.2.3
1.3.1 1.3.2 1.3.3
1.1.2 1.1.3
1.2.1 1.2.2 1.2.3
1.3.1 1.3.2 1.3.3
1.1.2
1.1.3
1.2.1 1.2.2 1.2.3
1.3.1 1.3.2 1.3.3
Evaluation
• It involves looking at your program and asking – – – – Are we doing what we said we were going to do?
Is what we’re doing working?
How do we know it’s working?
Did we accomplish what we set out to accomplish?
• It is part of what you put in place beforehand to help you run your programs. – It should be about looking ahead not backwards.
• It is part of the continuum of planning (rolling evaluation).
Evaluation
It is a systematic process of asking questions, collecting information, and using the answers to • • • • • Set realistic goals and fine tune strategic decisions.
Measure progress and identify areas of improvement.
Identify training and technical assistance needs.
Be accountable and be credible to your constituents.
Motivate by providing documentation of your achievements.
• • • Guide budget and resource allocation.
Generate support for programs.
Be an efficient organization that works toward its mission.
Formative vs. Summative Evaluation
• • • • • Formative evaluation (implementation and progress) How well were activities implemented (fidelity)? Was the target audience reached? What did they think about the activity? How are resources allocated and used for activities?
What were other products of those activities. How are preliminary evaluation findings used to improve implementation of activities?
What obstacles were encountered and how were they overcome?
Formative vs. Summative Evaluation
• • • • • Summative evaluation (Outcome) Are the initiative goal(s) and objectives being achieved? How fully are they being achieved?
Do activities have beneficial effects on at-risk students? If so, what are they?
Do activities have any adverse side effects on the at risk students? If so, what are they?
Are some at-risk students affected more by the services than others?
Can the results be traced to the initiative itself? (Is a control group possible?)What other factors may have caused the results?
Evaluation: Planning
• • • • • • What do you really want to know and for what purpose?
What evidence or indicators do you need to answer your questions?
Who’s in charge of evaluation? (roles and responsibilities) What time frame are you evaluating? How much time is allocated to conducting evaluation and when?
What is the budget for evaluation?
Commit the plan to paper!!!!
Evaluation Plan Matrix
Objective Source of Data Responsible Person (s) Evaluation Timeline
Instrument development Data collection Data analysis Reporting
Next Steps
• • • • • Examine your remaining objectives and ensure that they are all S.M.A.R.T.!
Insert each objective in the logic model template Identify activities and inputs/resources needed to meet objectives. Explicitly state expected results of activities in terms of outputs and outcomes. Identify indicators/data sources that will allow you to document and monitor implementation as well as measure progress and success.
Next Steps
• Look at each of your completed SMART objectives and identify data sources, time lines, and assign responsible parties for collecting, analyzing, and reporting data about your project.
– If necessary, assign responsibility for the design of new measures (for example, surveys). Define a time line for completion, administration, and analysis.
• As you assign responsibility, remember, evaluation is a team effort, not just one person’s job.
• Put evaluation on your upcoming meeting agendas. Always take some time to talk about where you are in your project, what data you have collected, what it is telling you and what you still need to know.
Thank You Anduamlak Meharie Coordinator – Research and Evaluation WVDE Office of Research
http://wvde.state.wv.us/research