Evaluating the Effectiveness of Service Learning Susan Agre-Kippenhan, Portland State University Professor, Art Department Why evaluate? Assessment as a continual Service Learning improvement tool Clarification of critical.
Download ReportTranscript Evaluating the Effectiveness of Service Learning Susan Agre-Kippenhan, Portland State University Professor, Art Department Why evaluate? Assessment as a continual Service Learning improvement tool Clarification of critical.
Evaluating the Effectiveness of Service Learning Susan Agre-Kippenhan, Portland State University Professor, Art Department Why evaluate? Assessment as a continual Service Learning improvement tool Clarification of critical learning objectives, ability to focus on Service Learning specifics Yields useful information for programs, faculty, students Evidence and support claims SL Outcomes: Service Learning Claims: Understanding of course content Theory to practice Understanding of Community Community related skills Civic Engagement Diversity MORE? Student Learning Focus Value of authentic work samples Moving from self-report Moving from less integrated forms of assessment (i.e. observation, focus group) Continual Improvement Feedback to actual courses, programs Applicable/actionable assessment Can be course/assignment based Can serve other programmatic goals How do we know what works? A working model for assessment based on the premise that there are: Key questions to ask Logical steps to follow Usable data Transforms student learning and community experiences Assessment Cycle Model 1 2 3 Develop Learning Objectives Identify Work Samples Locate Critical Junctures 4 5 6 Determine Assessment Criteria/Quality Test Validity /Reliability Disseminate Revise/Revisit Assessment Cycle Model Identify Work Samples 1 Develop Learning Objectives 2 Locate Critical Junctures 3 6 Disseminate Revise/Revisit 4 5 Determine Assessment Criteria/Quality Test Validity /Reliability Assessment Cycle Model 1 What am I trying to assess? Student Learning Outcomes in the context of: Mission Priorities Program Goals Action: Define Student Learning Outcome Assessment Cycle Model 2 What are appropriate work samples? Qualitative vs. Quantitative Examples: Portfolios, Papers (writing samples, essays, reflective papers, editorials, journals…) Performances, Artwork, Lab reports Action: Identify Student Work Samples Assessment Cycle Model 3 When should I assess? Critical Junctures Entering students Passages Culminating Action: Identify critical junctures Assessment Cycle Model 4 What am I looking for in student work? Rubric Development: Scoring Criteria What indicators will I use to demonstrate desired student learning? Examples: Bloom's taxonomy Action: Define scoring criteria Assessment Cycle Model 4 How do I evaluate the range of student work? Rubric Development: Performance standards How many levels will my rubric have and how will I define performance at each level? Specific clear descriptors Qualitative differences Common errors at each level Action: Define performance levels Sample Rubric Work Sample Scoring Criteria 1 Scoring Criteria 2 Scoring Criteria 3 Performance Level 1Low Performance Level 2Medium Performance Level 3High Assessment Cycle Model 5 Once I have a rubric how do I know it's valid? Evaluate the rubric against the learning objectives, learning opportunities etc… Is there clear alignment? How do I know if the rubric is reliable? Pilot test with multiple evaluators Discuss how well the rubric works Compare scores for consistency Determine procedures for inconsistent scores Assessment Cycle Model 6 What's next? Revise and revisit - learning outcomes, learning opportunities, assignments, critical junctures, rubrics, program, course offerings/order, curriculum, pedagogy… Action: Back to the cycle SL Case Study University Studies - General Education 4 Year integrated program Interdisciplinary program Critical Features Clearly defined goals Communication, Critical thinking, Diversity, Civic Responsibility Common Pedagogy Required Courses SL Case Study 1 Student Learning Outcomes PSU Mission - Let Knowledge Serve the City University Studies Goal - Civic Responsibility Action: Define Student Learning Outcome Appreciation and Understanding of Civic Responsibility SL Case Study 2 Work samples Paper Action: Identify Student Work Samples Common assignment given in each class SL Case Study 3 Critical Junctures Senior Capstone required course Culminating group experience for major and general education Application of expertise to a real project in the community Action: Identify critical junctures Pilot in Senior Capstone Classes SL Case Study Common Assignment “Please write a 3-5-page reflective analysis summarizing the ways you have gained experience with the University Studies goal of Civic Responsibility. Use specific examples from your discussions and course readings, or in completing your project.” SL Case Study 4 Scoring Criteria and Performance Levels Based on Bloom's taxonomy Action: Define scoring criteria/levels Civic Responsibility Stages SL Case Study 5 Valid Rubric PSU Mission - Let Knowledge Serve the City University Studies Goal - Civic Responsibility How do I know if the rubric is reliable? Pilot test with multiple evaluators Compare scores for consistency. Discuss how well the rubric works. SL Case Study 6 Revisiting Process Suggestions for improvements Revise assignment for clarity Revise rubrics for specificity Clarify pedagogy Confirmed Program goals addressed in class Basis of assignment is valid Keep assessment in Capstone SL Assessment in practice on student learning objectives What are appropriate work samples? Are there existing critical junctures? How do we evaluate quality? How can this improve our practice? Focus Lessons Learned Connect to goals and mission Knowledge of the process that is transferable Assessment as a tool for partnerships Applicable/actionable assessment For More Information Susan Agre-Kippenhan [email protected] 503-725-8506