Karla A. Henderson, Ph.D. Professor, North Carolina State University [email protected] Framework Evaluation Inventory (handout) Needs Assessment Regarding Program Evaluation (Types of Needs Explained)
Download ReportTranscript Karla A. Henderson, Ph.D. Professor, North Carolina State University [email protected] Framework Evaluation Inventory (handout) Needs Assessment Regarding Program Evaluation (Types of Needs Explained)
Karla A. Henderson, Ph.D. Professor, North Carolina State University [email protected] Framework Evaluation Inventory (handout) Needs Assessment Regarding Program Evaluation (Types of Needs Explained) Small Group Needs Identification (6 people per group) Importance (Very, Moderately, Slightly, Not) 5-10 topics to Discuss Together What is Evaluation? Evaluation= systematic collection and analysis of data to address criteria to make judgments about the worth or improvement of something; making decisions based on identified criteria and supporting evidence Assessment= the examination of some type of need that provides a foundation for further planning Evaluation is sometimes like solving a mystery Types of ProgramEvaluation Macro Micro (System (Activity or Evaluation) Event Evaluation) Formal (Systematic) Evaluation MACRO and MICRO approaches----- Provides rigor Systematic gathering (procedures and methods) of evidence Leads to decisions and action Criteria Evidence Judgment Assessments Why assess? eventually use the input to design programs and objectives generates new program ideas gives constituents a say helps you be responsive What is a need? A want? An intention? Assessments determine all three of these so YOU can figure out how to promote what you are doing Types of needs (Caution!) Normative needs- what should be available Felt needs- what individuals believe they would like Expressed needs- needs fulfilled through participation 1. Why do Systematic Evaluation Potential Purposes: Efficiency-How are we doing? (Management /Process Focused) Effectiveness- What difference do our efforts make? (Impact/Outcome Focused) Accountability Era—DO YOU AGREE? What gets measured gets done If you don’t measure results, you can’t tell success from failure If you can’t see success, you can’t reward it If you can’t reward success, you’re probably rewarding failure Reinventing Government, Osborne and Gaebler, 1992 University of Wisconsin-Extension, Program Development and Evaluation Accountability Era If you can’t see success, you can’t learn from it If you can’t recognize failure, you can’t correct it. If you can demonstrate results, you can win public support. Reinventing Government, Osborne and Gaebler, 1992 University of Wisconsin-Extension, Program Development and Evaluation Evaluation Process: “Begin with the end in mind.” Covey (1990), 7 Habits of Highly Effective People 2. How to Think Like an Evaluator A thinking process used by evaluators: (from Richard Krueger) Reflecting Develop a theory of action. A logical sequence that results in change. Begin with what is supposed to happen--the results. Listening Share the theory of action with others. Measuring Determine your measurement strategies--how you're going to look at the program Adding value to the program What can evaluation do to contribute to the program? How can evaluation make the program better, more enjoyable, focused on results, accountable, satisfying to participants and educators, etc? Ways of Thinking: Goal Based Thinkers - "We look for goals" Audit Thinkers -"We investigate and find out what's wrong" Utilization Focused Thinkers - "We make evaluation useful" Empowerment Focused Thinkers - "We empower local people" Positivistic Thinkers - "We are scientists" Number Thinkers - "We count--and we do it well“ Qualitative Thinkers - "We tell stories" (Richard Krueger) Practical tips for successful evaluation: (Richard Krueger) Involve others Utilization, impact and believability emerge from involving colleagues and clientele. If you want the information used then involve others! Ask yourself: Do I have a program? and, Is it worthy of evaluation? Consider your purpose for evaluating---- (see earlier slide) Consider who wants the evaluation-Who requested it? Use a variety of evaluation methods when possible. Keep costs low by: Sampling strategically Keep interest high by adding payoff to the participant. Start with goals, but don't be unduly limited by goals Consider "early evaluation" Design the evaluation carefully. The evaluation should: Enhance the program Yield information beneficial to stakeholders Conserve resources 3. Differences Between Assessment, Evaluation, and (Action) Research Evaluation= systematic collection and analysis of data to address criteria to make judgments about the worth or improvement of something; making decisions based on identified criteria and supporting evidence Assessment= the examination of some type of need that provides a foundation for further planning Action Research-Evaluation leading to decisions/changes 4. Steps Involved in Evaluation Process Steps Problem, Idea Identified Problem Statement/Purpose Determined Instrument/Method Chosen Data Sources Data Collection Data Analysis Conclusions/Recommendations Evaluation Process: “Begin with the end in mind.” Covey (1990), 7 Habits of Highly Effective People 5. What Should be Evaluated Areas to Evaluate Personnel Places Policies Programs Participant Outcomes Potential Purposes: Efficiency-How are we doing? (Mgmt/Process Focused) Effectiveness- What difference do our efforts make? (Impact/Outcome Focused) Levels of Evaluation: END RESULTS (Impact) PRACTICE CHANGE (Outcomes) KASA CHANGE (Knowledge, attitudes, skills, and aspirations)(Outcomes) REACTIONS (SATISFACTION) (Outputs) PEOPLE INVOLVEMENT (Outputs) ACTIVITIES (Outputs) INPUTS--RESOURCES What is sampling? A population is the theoretically specified aggregation of study elements. A sample represents or is representative of a population Types of Sampling Probability Non-Probability Theoretical Probability Probability sampling Samples are selected in accord with probability theory, typically involving some random selection mechanism. Random Stratified Random Systematic Cluster Representativeness Quality of a sample having the same distribution of characteristics as the population from which it was selected. Nonprobability Technique in which samples are selected in a way that is not suggested by probability theory. Purposive Convenience Quota Expert Snowball 6. Who Should Conduct Evaluations WHO DOES THE EVALUATIONS? Internal You! Staff Agency Evaluation Personnel External Consultants University Students! Regardless—YOU have to know your purpose, goals, and appropriate methods! 7. When Should Evaluataions Be Done Timing Assessments (planning) – find out where to begin based on what you know Formative (process)- concerned with efficiency and effectiveness Summative (product) – overall performance Approaches to Needs Assessments Literature/Professional Development Advisory Groups Structured Interviews (individual and focus groups) Surveys Formative Evaluation Evaluation in Process Allows Changes to be Made Immediately Most often Focused on Inputs and Outputs Summative Evaluation At the END of something “What was?” Recommendations for the Future 8. What Tools are Available for Evaluations Data Collection Methods MAJOR ONES: Questionnaires/Surveys Interviews (Individual and Focus Groups) (Pros and Cons) Other Methods: Systematic Observation Checklists Field Observations Unobtrusive Measures Physical Evidence Archives Covert Observations Visual Analyses Experimental Designs Case Studies 9. What “Cutting Edge” Strategies Exist for Evaluation To consider (see below for further info) Logic Models—”outcome focused” Trends Analysis Benchmarking Proragis (NRPA) 10. CAPRA Standards related to Evaluation CAPRA Standards 10.1 Systematic Evaluation Program There shall be a systematic evaluation plan to assess outcomes and the operational deficiency and effectiveness of the agency. 10.2 Demonstration Projects and Action Research There shall be at least one experimental or demonstration project or involvement in some aspect of research, as related to any part of parks and recreation operations, each year. CAPRA Standards 10.3 Evaluation Personnel There shall be personnel either on staff or a consultant with expertise to direct the technical evaluation/research process. 10.4 Employee Education There shall be an in-service education program for professional employees to enable them to carry out quality evaluations. 11. Evaluating Inputs, Outputs, Outcomes, and Impacts Evaluation Approaches—KEY POINTS! Multiple LEVELS of evaluations: Inputs (costs, personnel, etc.) Outputs Activities People involvement Reactions Outcomes KASA-knowledge, attitudes, skills, aspirations Behavior CHANGE Impacts Long-term Benefits 12. Using Goals and Objectives as Basis for Evaluation What are the goals of the program? What do we expect to happen? What do we want participants to do, gain, learn? BEGIN WITH THE END IN MIND!! Goals and Objectives Goals Broad, long-range statements that define the programs/services that are going to be provided Objectives Specific statements (about the attainable parts of the goal) that are measurable and have some dimension of time. Objectives Specific Measurable Must be attainable and reality-based!!! Relevant Must be some way to determine whether or not the desired results have been achieved Achievable Must be clear and concrete Must be useful; must have worth to your organization Time-limited/Time connected Must specify a time frame for accomplishment Adapted from Edginton, Hanson, & Edginton, 1980 13. Using Logic Models as Basis for Evaluation Logic Model (Preview) Goal StrategiesActivities Short-term Outcomes Long-term Outcomes Data Sources & Performance Measures Moving Toward Success: Framework for After-School Programs, 2005 A logic model is… A HOT TOPIC in (Program) Evaluation A depiction of a program showing what the program will do and what it is to accomplish. A series of “if-then” relationships that, if implemented as intended, lead to the desired outcomes The core of program planning and evaluation University of Wisconsin-Extension, Program Development and Evaluation Simplest Form of a Logic Model INPUTS OUTPUTS OUTCOMES 54 Everyday example H E A D A C H E Situation Get pills Take pills Feel better INPUTS OUTPUTS OUTCOMES 55 Where are you going? How will you get there? What will show that you’ve arrived? “If you don’t know where you are going, how are you gonna’ know when you get there?” Yogi Berra 56 How will activities lead to desired outcomes? A series of if-then relationships Tutoring Program Example IF We invest time and money then IF We can provide afterschool programs to 50 children then IF Students will have opportuniti es for active recreation then IF They will learn and improve their recreation skills then IF They will be more active University of Wisconsin-Extension, Program Development and Evaluation then They will become fit and develop life long skills 57 Logical chain of connections showing what A program is to accomplish INPUTS OUTPUTS Program investments Activities Participation What we invest What we do Who we reach OUTCOMES Short Medium Longterm What results 58 OUTPUTS What we do ACTIVITIES •Train, teach •Deliver services •Develop products and resources •Network with others •Build partnerships •Assess •Facilitate •Work with the media •… Who we reach PARTICIPATION •Participants •Clients •Customers •Agencies •Decision makers •Policy makers University of Wisconsin-Extension, Program Development and Evaluation Satisfaction 59 OUTCOMES What results for individuals, families, communities..… SHORT MEDIUM LONG-TERM Learning Action Conditions Changes in Changes in Changes in • Awareness • Knowledge • Attitudes • Skills • Opinion • Aspirations • Motivation • Behavioral intent •Behavior •Decision-making •Policies •Social action Conditions Social (well-being) Health Economic Civic Environmental CHAIN OF OUTCOMES Limitations Logic Model… Doesn’t address: Are we doing the right thing? University of Wisconsin-Extension, Program Development and Evaluation 14. Using Benchmarking as a Basis for Evaluation Benchmarking Agencies or Services compared to other agencies or services to determine a relative value Set goals against a “standard” of the best Benchmarks are the statistics; Benchmarking is the process Information examples: budgets, salaries, fees charged, number of participants, number of staff Decide who you want to compare, design a questionnaire, analyze, and write report 15. Using NRPA’s PRORAGIS Tool for Evaluation 16. Doing Cost/Benefit and Economic Impacts Economic Impact Analyses Determining the dollar value a program or service has on the economy The net change of money in a community from the spending of visitors—only visitors are part of the analyses Cost-Benefit Analyses Focus on Monetary Forms Sometimes Difficult to Do Social Benefits may outweigh Monetary Benefits 17. Using Trends Analysis for Evaluation Trends analysis Examining the future by looking at the past Measuring data over time Secret is good record keeping 18. Designing Valid, Reliable, and Useable Questionnaires Kinds of Questions Open-ended Close-ended Fixed alternatives Likert Scale Semantic differential Ranking Partially close-ended Examples Open-ended: “What did you like best about day camp?” Close-ended, fixed alternatives: “What did you like best about day camp this summer?” Counselors Arts Snacks Swimming Sports Another Example Close-ended, Likert: “What did you think about each of these parts of day camp?” Counselors Arts & Crafts Snacks Swimming Poor Fair Good Great 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 Wording Advise One idea per question Clear, brief, and simple Avoid leading questions Avoid estimates if possible Use words familiar to respondent Avoid fancy words and jargon Be clear about meanings of words Avoid negative questions Do a pilot study State alternatives precisely-mutually exclusive responses Use stages if needed but give GOOD directions Format and Layout Design For Good Response Give clear directions Start with something easy & familiar (NOT demos though) Have white space & easy font to read Colored paper if easy to read—font size appropriate for audience Have easy to follow directions for staged questions Have “professional look” Front page should include title, date or time or year, perhaps a graphic Keep the length manageable for the info desired Anchor all numbered responses—e.g., 5=strongly agree, 4=agree etc. NO TYPOGRAPHICAL ERRORS PILOT TESTING Is the survey valid? How much time does it take to complete? Do the respondents feel comfortable answering the questions? Is the wording of the survey clear? Are the answer choices compatible with the respondents’ experience in the matter? 19. Using Individual Interviews Types of Interviewing Personal (in-depth) Telephone Part of Field Observation Focus Groups (Group Interviews) Interviews Approaches: Structured, close-ended Standardized, open-ended Interview guide (semi-structured) Informal conversational (unstructured) Asking Open-Ended Questions NEVER allow a person to say “yes” or “no” Purpose is to get people to talk Start with noncontroversial questions Demographics at end/don’t ask if you know (e.g., gender) Avoid “WHY” Use probes and follow-ups (“Tell me more about…?” Use listening skills Maintain control of the interview Share yourself with the interviewee Structuring Interviews Purpose of interviews Individual or group If group- number of persons involved Structured or unstructured? Audio-taping/Note-taking Facilitator/Recorder Roles Rapport-building Timing Things to remember Pay attention to the content of your questions (keep your focus) Give thought to setting up the interview Pay attention to how you plan to record the data Take notes-on-notes Conduct interviewer training Go over possible problem areas Training for Interviewers Discussion of general guidelines and procedures. Specify how to handle difficult or confusing situations. Conduct demonstration interviews. Conduct “real” interviews. Guidelines for Structured Survey Interviewing Dress in a similar manner to the people who will be interviewed. Study and become familiar with the questions. Follow question wording exactly (if it is quantitative). Record responses exactly. Probe for responses when necessary. Telephone Surveys Advantages: Money and time. Control over data collection. Disadvantages: Surveys that are really ad campaigns. Answering machines. 20. Conducting Focus Groups Focus Group A group of people are brought together in a room to engage in guided discussion of a topic. Pros and cons Pros Socially oriented Flexible High face validity Speedy Low in cost? Cons Researcher has less control Data may be difficult to analyze Need good moderator Must do more than one group-great differences between groups Groups difficult to assemble Must have conducive environment What about group interviews? Focus groups encourage ideas by interacting with other individuals Pay attention to the same issues in a personal interview Have a clear purpose for the group meeting (criteria) Have clear structured, open-ended questions Points for Conducting a Focus Group Be a good moderator Set ground rules The beginning of the focus group process is crucial to its success Keep the process focused and flowing Probe where needed Be sure to thank participants Other Focus Group Tips Usually 5-10 individuals Key is to get people to talk to each other Usually two moderators (one to direct discussion and another to take care of technical aspects) Use incentives to get people to particiapte (maybe) Pilot test just like ALL surveys USE OPEN-ENDED questions Be familiar with group process—getting people to talk,not letting some take over Ground rules-speaking one at a time, confidentiality, breaks taken, positive and negative Summary Focus groups encourage ideas by interacting with other individuals Pay attention to the same issues in a personal interview Have a clear purpose for the group meeting (criteria) Have clear structured, open-ended questions 21. Using Observational Tools Types of Data Qualitative Quantitative Qualitative Observations Takes a great deal of field time Need in-depth notes on your observations Use of key informants is often helpful Criteria often change and become re-defined Tips on (Field) Observations Only unscientific if inappropriate techniques are used Highly reliable and valid IF systematically done Time-consuming Anecdotal records or critical incidents describe FACTS of situations Field notes are collected-MANY, MANY, MANY Researcher is data collection instrument SEPARATE facts from interpretations Notetaking Record as soon as possible Take notes in detail-running descriptions, conversations, incidents Make copies of notes ASAP Indicate whose language-yours, quotes, paraphrases Rule of thumb-several (5-6 single spaced typed pages for an hour of observation) Can use tape-recorder and transcription Record ALSO what you do not understand MONITOR yourself as a data collector Length of time spent depends on research question and theoretical saturation “IF it is not written down, it never happened!” Dimensions of Approaches Full participant observer-> Outside observer Overt observer-> Covert observer Single observation-> Long-term multiple obs. Narrow focus-> Broad focus Quantitative: Duration Sampling Checklist Participant: _________ Activity: _____________ Time: _____ to ________ Behavior Duration Total On task (appropriate) 2 sec., 3 sec., 19 sec 24 sec Approp. Conversation (w/ peers) Walks around room Argues w/ leader Argues w/ peers 14 sec., 29 sec 112, 143 sec, 157 sec 0 sec 43 sec 43 sec 412 sec 0 sec 43 sec Frequency Sampling Checklist Client: _________ Activity: _____________ Time: _____ to ________ Behavior Frequency of Behavior Initiates a conversation Shifts attention to new project Asked question by peer // / //// Responds appropriately when asked question by peer /// Rating Scales Behavior Excellent Average Poor* How well does the client respond to questions? How appropriate does client interact with peers? *Excellent= top third Average= middle third Poor= lowest third 22. Using On-line Tools (SurveyMonkey etc.) Question Structures-SurveyMonkey Question 1: Have you used a swimming pool in the last year? Yes No Question 2: How many times in the last year have you gone to a public swimming pool? Never Once Twice More than 3 times Question 3: How many times have you gone to a public swimming pool this past year? (Please write in an estimate number) __________ On-Line Considerations--Pros Same strengths as a paper version •Better at addressing sensitive issues •Cost efficient •Faster delivery •Design options •Dynamic •Ability to track •Quick response time •Easier to use for skip logic On-Line Considerations--Cons Spam/Privacy concerns •Technical issues •Submitting multiple submissions •No interviewer present to clarify questions or issues New Technologies and Survey Research CAPI - computer assisted personal interviewing. CASI - computer assisted self interviewing. CSAQ - computerized self-administered questionnaires. TDE - touchtone data entry. VR - voice recognition. 23. “Fun” Strategies for Doing Evaluations Importance-Performance Eval How important is something? How well did the organization perform it? Matrix of: Keep up the good work Possible Overkill Needs Work Low Priority Other Ideas: Use Grading Scale A, B , C, D, F Use Pie Chart and distribute money for preferences Use Happy Faces Computer Assisted Others… Measuring Physical Activity SOPLAY or SOPARC Pedometers or Accelerometers 24. Selecting Evaluation Instruments Selecting an Evaluation/Measurement Tool Is it reliable and valid? Does it measure what you want? Appropriate for the participants? Reasonable to administer and in your price range? Directions clear, concise, unambiguous? Easy to analyze? Is it the best way to measure the objectives? Is the activity reaction form put together by Cary, NC the one you should use? 25. Contracting with Outside Consultants Outside: You have to know what you want done You have to be able to evaluate results and make decisions You probably need some amount of financial resources Could save you a lot of time if you do it right 26. Developing Your Own Evaluation Tool Steps: Define problem/level of evaluation Determine contents (including criteria) and broad questions Identify and categorize respondents Develop items, structure format Write directions Ensure response Purpose of Data Collection What do you want to know? Who has the information? What is the best approach (based on purpose, time, money, resources, your expertise) to use? How will you use the results? Are you interested in outputs and/or outcomes? Kinds of Information Sought Behavior info Knowledge info Attitudes/beliefs/values info Demographic info **pay attention to relationship of respondent to the question (their past, present, future) 27. Ethical Considerations related to Evaluation Issues Ethical Political Legal Moral Evaluator must be Competent Knowledge about area to be researched Knows about evaluation design Knows wide range of methods Knows how to analyze and interpret data and relate them back to conclusions/recommendations Developing Competencies – con’t Knows how to use the results Understands how to handle political, legal, and ethical concerns encountered Must have certain personal qualities: trustworthy, strives for improvement, responsive to sensitive issues “Doing the Right Thing” Political Issues (science is touched by politics but goes on anyway; social change is always political; values matter) Supports/refutes views and values Personal contacts Value-laden definitions Controversial findings Pressures to produce certain findings Know the organization’s position, don’t go beyond the data in conclusions, have clear purpose for research “Doing the Right Thing” Legal Issues Not many legal concerns except around illegal behaviors Moral Unintentional mistakes made by bias or mistake Cultural & procedural biases Letting bias, prejudice, friendships influence outcomes Dealing with negative findings Taking too long to get the results out Be prepared to recognize the possibility of statistical errors and know how to explain them “Doing the Right Thing” Voluntary Participation No Harm Anonymity and Confidentiality Issues of Deception Analysis and Reporting Ethics Principles Be open with people Don’t promise more than you can do Protect the rights and privacy of your subjects Guard against coercion Get written consent & Board approval Guard against harm Let participants know the results Anonymity The researcher cannot identify a given response with a given respondent. Confidentiality Researcher can identify a given person's responses but promises not to do so publicly. Ethical Considerations Avoid bias (settings, questions, populations) Consider question content Protect confidentiality (if appropriate) Manage data/records Report results of evaluation Respect/report different opinions 28. Devising an Evaluation Plan for an Agency Purposeful Planning for Evaluation Schedule time on your yearly calendar Involve planning Board, committees, colleagues, staff, volunteers, etc. Identify what you hope to achieve (desired outcomes) Identify your goals and objectives Purposeful Planning for Evaluation Identify the methods you will use to measure whether or not your program met objectives Identify how often evaluation will be undertaken Identify WHO will do the evaluation and how they will get the appropriate resources Formal (Systematic) Evaluation--Review MACRO and MICRO approaches----- Provides rigor Systematic gathering (procedures and methods) of evidence Leads to decisions and action Criteria Evidence Judgment 29. How to Use Information from Evaluation Projects Using Data-KEY ADMONISHMENTS If you collect the data, you need to analyze the data! Record the results of your data collection Compare your findings to your original goals and objectives Share with appropriate people Incorporate into your planning cycle