Transcript Slide 1
Introduction to Evaluation and Assessments Presented by Kathleen (Kat) Miller, Senior Consultant with Booz Allen Hamilton 4 August, 2007 SNDT Women's University Evaluation Model – Donald Kirkpatrick In 1959, Donald Kirkpatrick developed what has become one of the most popular models for evaluating education programs. Kirkpatrick's system has four levels of evaluation. 2 SNDT Women's University Evaluation Purpose of Evaluation “The reason for evaluating it to determine the effectiveness of a training program.” © Donald L. Kirkpatrick, “Evaluating Training Programs: The Four Levels,” 3 SNDT Women's University Main Reasons for Evaluation Determine how to improve future programs Determine whether a program should be continued or dropped. Justify the existence of the training department 4 SNDT Women's University Kirkpatrick’s Level 1 - Reaction Measure of “Customer Satisfaction” Measures the emotional reaction of participants after a learning experience. – It attempts to answer questions regarding the participants' perceptions: – Did they like it? – Was the material relevant to their work? – Did it meet their expectations? – How should the learning experience be improved? SNDT Women's University 5 Why Measure Reaction? It provides valuable feedback and suggestions It makes the trainees feel that their input is important It is easy to do effectively A positive reaction promotes motivation! 6 SNDT Women's University Guidelines for Evaluating “Reaction” 1. Determine exactly what you want to find out 2. Design a form the will quantify reaction 3. Encourage written comments and suggestions 4. Get a 100 percent immediate response 5. Get honest responses 6. Develop acceptable standards 7. Measure reactions against standards and that action 8. Communicate (as appropriate) 7 SNDT Women's University Kirkpatrick’s Level 2 - Learning Measures whether or not a student achieved the learning objectives during the learning event or during a series of such events. Before Training For example: we might test for written communication skills after instruction to determine if the learner is now qualified for the job. After Training SNDT Women's University 8 Why Measure “Learning?” Kirkpatrick defines learning as, “the extent to which participants change attitudes, increase knowledge and/or skill as a result of attending a program.” Was knowledge increased? Were skills developed? Were attitudes changed? 9 SNDT Women's University Guidelines for Evaluating “Learning” 1. Use a control group if possible 2. Use pre/post tests – Written for knowledge measurement – Performance for skills 3. Get 100 percent response 4. Use results to take appropriate action 10 SNDT Women's University Kirkpatrick’s Level 3 - Behavior Measures whether learners are able to apply their new knowledge and skills to their job. – Are they using those skills on the job? – Are there other issues that are stopping them from being successful on the job? – Has their behavior changed? Information for Level 3 evaluations is generally gathered via surveys and personal interviews. 11 SNDT Women's University Four Conditions for Behavior to Change The person must have the desire to change The person must know what to do and how to do it The person must work in the right climate The person must be rewarded for changing 12 SNDT Women's University Guidelines for Evaluating “Behavior” Use a control group if possible Allow time for the change in behavior to take place Evaluate before and after the program Survey or interview the trainee, as well as someone who observes their behavior Get 100 percent response Repeat the evaluation at appropriate times Consider cost versus benefits 13 SNDT Women's University Kirkpatrick’s Level 4 - Results Increased Production Improved Quality Decreased Costs Reduction in Accidents Increased Sales Reduced Turnovers Higher Profits 14 SNDT Women's University Challenges to Evaluating “Results” Some are intangible Results may not occur for a long time Many other factors impact results Such is hard to control all those wild cats! The trainer cannot control all factors 15 SNDT Women's University Guidelines for Evaluating Results Use a control group if possible Allow time for results to be achieved Measure both before and after Repeat the measurement at appropriate times Consider cost versus behavior Be satisfied with evidence, if proof is not possible 16 SNDT Women's University Assessments What is an assessment ? “Any systematic method of obtaining evidence from posing questions to draw inferences about the knowledge, skills, attitudes, and other characteristics of people for a specific purpose” © Copyright Questionmark Corporation 2004 17 SNDT Women's University Types of Assessments Exam Test Quiz Survey 18 SNDT Women's University Uses of Assessments Diagnostic Formative Needs Reaction Summative 19 SNDT Women's University Diagnostic Assessments Typically used in pre-learning Identity skill gaps and needs Identity student preferences 20 SNDT Women's University Formative Assessment Provide feedback for the learners Search and retrieval May contribute to overall grades 21 SNDT Women's University Needs Assessments Assess knowledge, skills, abilities, and attitudes Serve as a gap analysis Determine what should be taught based upon needs 22 SNDT Women's University Reaction Assessments Students’ opinions are collected Kirkpatrick's Level 1 Instructor can alter the learning experience for the future 23 SNDT Women's University Summative Assessments Summary of knowledge/skill of the learner Provides a quantitative grade May result in certification 24 SNDT Women's University Stakes of an Assessment Assessment “stakes” are the consequences to the candidate Low Medium High 25 SNDT Women's University Comparison Assessment Type Assessment Use Assessment Stakes Exam Summative Medium, High Test Diagnostic Low, Medium Quiz Formative Low Survey Needs, Reaction, Diagnostic Low 26 SNDT Women's University Summary Kirkpatrick’s Evaluation Model Types of Assessments Uses for Assessments Assessment Stakes 27 SNDT Women's University Thank You for Your Time! Goodbye from your new “virtual” friend, Kat Miller Contact me at: [email protected] 28 SNDT Women's University