Transcript Slide 1
Assessment and Development: The Keys to Quality DLA2007 Distance Learning Administration Conference 8:00 AM on Monday, June 25, 2007 Questions… What is quality? How can an institution improve its effectiveness? What areas can the institution focus on to make improvements and exceed quality standards? How are training and evaluation involved in the focus on quality? Copyright 2007 © Lycurgus Group, Inc. Agenda Brief introduction about who we are Quality in distance learning Why focus on training and evaluation to promote quality distance education How we can help Copyright 2007 © Lycurgus Group, Inc. The Lycurgus Group, Inc. An organizational effectiveness and development consulting firm that uses scientific and research approaches to assist organizations in the areas of organizational assessment, change, development and online learning. Copyright 2007 © Lycurgus Group, Inc. Distance Learning The demand for quality online education is high with the increased number of programs and schools offering online classes. The options now available to students are extensive. What is quality criteria varies among different distance learning organizations The key is to establish quality criteria that are linked to best practices, accreditation standards and the organization’s mission and vision How effective the organization is in meeting these quality standards needs to be evaluated Training should be offered in areas that are in need of development Copyright 2007 © Lycurgus Group, Inc. How do you define quality? Can mean different things to different schools/organizations Often defined as effectiveness of programs and/or satisfaction with learning experience (classes/programs) Copyright 2007 © Lycurgus Group, Inc. Evaluation Why Evaluate? Great vehicle to communicate quality standards Evaluation is required for satisfaction, retention and accreditation reasons. Determination if course objectives are being met Provides the means of celebrating what is going well Identification of changes that need to be made with courses and programs Copyright 2007 © Lycurgus Group, Inc. Traditional Evaluation Methods Formative=Ongoing Institutional Effectiveness Formative evaluations can be accomplished through: This can be accomplished by obtaining feedback from students and faculty on their satisfaction with online learning. Improvement of programs Evaluate strengths/weaknesses and areas of concern Summative=Completion Course Effectiveness Synthesis of data to make decisions on course improvements Summative evaluations It is important to be able to improve the course design to facilitate learning in a more effective way. Evaluate course objectives, experience and facilitation style Improvement of programs Copyright 2007 © Lycurgus Group, Inc. Proactive Evaluation Proactive evaluation is the foundation for the methodology that will be proposed Thus, an evaluation system that provides a comprehensive and systematic view of institutional effectiveness supports this model and includes feedback from multiple sources. Copyright 2007 © Lycurgus Group, Inc. Quality Framework for Evaluation Quality framework: enables institutions to set goals, measure progress in reaching goals and implement required changes The focus on quality and best practices are the basis of our evaluation system. Applied to each individual institution uniquely by identifying needs/goals Following organizations identified quality standards: American Council of Education, American Distance Education Consortium, Sloan-C, Institute for Higher Education Copyright 2007 © Lycurgus Group, Inc. Lycurgus Evaluation Assessment Process The LEAP System™ The LEAP System™ is a systematic approach in assessment, change implementation and evaluation. Assessments measure learning outcomes and institutional effectiveness. Data is gathered and used to identify planned change efforts and needed improvements. All key stakeholders are included in the feedback process. (students, faculty and staff) The evaluation system is linked to measuring quality standards and best practices in distance education. Key factors are based on research leveraging organizational effectiveness and best practices in distance education. Identify strengths and areas of improvement. Copyright 2007 © Lycurgus Group, Inc. Evaluation System: Based on Quality standards in Online Education Learning Effectiveness Main Measure: End of Course Survey Also, can use focus groups and interviews Student Satisfaction Student Survey on Institutional Effectiveness Faculty Satisfaction Faculty Survey of online learning experience/Ins tructional Effectiveness Copyright 2007 © Lycurgus Group, Inc. Cost Effectiveness *Goal is to cut or n control costs *Reduce teacher turnover, lower training costs and improve technologies *Increase retention *Number of students obtaining degrees -Good to develop dashboard of results as a communication tool -important to utilize internal and external benchmarks Survey Assessments The nucleus of the system LEAP System™ Faculty/Staff & Student Effectiveness Survey & End of course Survey Web-Based Platform – Quickly gather critical data about your organization. Detailed reports based on Data – Have the knowledge to make crucial decisions. Recommendations made on Key Areas – Know how to make improvements. Web-based Custom Dashboards – Know critical information about your organization anytime and anywhere in the world. Copyright 2007 © Lycurgus Group, Inc. Factors to Evaluate End of Course Survey The factors that should be evaluated in a comprehensive system include: Course content Course outcomes/objectives Technology Resources/Student support Interactivity (with peers and instructor) Feedback Barriers to Class learning engagement/Motivation Instructional approaches/Addresses different learning styles General Satisfaction Most valuable aspects of course Areas of improvement Copyright 2007 © Lycurgus Group, Inc. Factors to Evaluate Faculty Survey Course Feedback Evaluation of learning/university Faculty Concerns What is going well What can be improved upon Experience with Students Incentives/Development opportunities Faculty Time/Workload Copyright 2007 © Lycurgus Group, Inc. Factors to Evaluate Institutional Effectiveness University commitment to students and quality education Fairness in Evaluation/Adequacy of Feedback Support/Resources Received Instructor/Student interaction Overall experience with classes and university services Copyright 2007 © Lycurgus Group, Inc. Copyright 2007 © Lycurgus Group, Inc. Leap System™ Steps Step 1: Determine goals, objectives and standards of excellence. Step 2: Identify internal benchmarks. (These benchmarks are used to compare to future evaluations and to compare with other schools.) Step 3: Collect data from all stakeholders. Assessment includes surveys as well as interviews and focus groups when applicable. Step 4: Report results and make recommendations. Step 5: Identify top areas for planned improvements. Step 6: Implement changes. Step 7: Reevaluate. Copyright 2007 © Lycurgus Group, Inc. Goals/Objectives It is important to determine: Why you are doing the evaluation? What do you hope to accomplish? What is success? Student performance, satisfaction, retention What will you do with the data? Who do you want to share the results with? Copyright 2007 © Lycurgus Group, Inc. Identify Internal Benchmarks Important to identify key internal measures Such as student and faculty satisfaction and retention Will there be longitudinal comparisons made? Comparisons conducted within classes/departments/curriculum? What are internal messages of success? Or causes for alarm? Also, valuable to identify comparisons against established external benchmarks when applicable Copyright 2007 © Lycurgus Group, Inc. Data Collection Options Identify who will be involved in data collection process: Staff, faculty and/or students These are key stakeholders Also, graduates can provide informative data Identify best methods of data collection Different surveys, focus groups, or interviews, document evaluation Copyright 2007 © Lycurgus Group, Inc. Survey Assessment Benefits Can involve many individuals in the feedback process in a short timeframe Able to get detailed feedback Ease in administration Utilize web to contact participants Ensures confidentiality of feedback obtained Copyright 2007 © Lycurgus Group, Inc. Online Surveys Surveys delivered via email Ease in administration and analysis Branching of data is not apparent to participants Web-based deployment Ensures confidentiality Provides quick turnaround of information Copyright 2007 © Lycurgus Group, Inc. Survey Items These factors should then be broken into specify survey items that measure each dimension. Also, different surveys should be developed for student, staff and instructor feedback. Copyright 2007 © Lycurgus Group, Inc. Other Important Data Retention Data Feedback from graduates/# of students getting jobs in field Document examination (handbooks, orientation, strategy, mission/vision) History of changes made to courses Copyright 2007 © Lycurgus Group, Inc. Report Results/Make Recommendations Different levels of reports can be provided that will allow for different levels of action planning Strengths can be identified Can see overall picture of institutional performance Copyright 2007 © Lycurgus Group, Inc. Planned Improvements: Identify Top Areas Top areas of concern can be discovered Focus of top action plans Use data to identify planned improvements, timeline and who will be involved in making needed changes Copyright 2007 © Lycurgus Group, Inc. Implement Changes Utilizing an organizational effectiveness approach Identified from all data & participants Examples of changes made working with other schools include: Implementation of professional development activities Construction of new student orientation materials Changes to online courses/programs Copyright 2007 © Lycurgus Group, Inc. Evaluation Revisited Important to evaluate to see if improvements have occurred, if new changes need to be made or if new issues have developed Also, important to continually celebrate accomplishments! Copyright 2007 © Lycurgus Group, Inc. Training Training and Quality DeSimone (2006) indicated how often the only type of training offered to instructors is technical training. Educators need more! A training/development program that is focused on quality often involves creating an ongoing interactive learning environment where ideas can be exchanged and this develops a community where resources can be shared. Copyright 2007 © Lycurgus Group, Inc. Training Options to Promote Instructor Satisfaction Initial Training Involves Faculty Orientation to the online learning environment. Usually conducted with new online instructors to a school Peer Support Good for seasoned instructor. Gain continuous feedback from colleagues Mentoring/ Coaching •Good for both seasoned and new instructor •Working with someone who has a lot of experience helpful when starting first class Copyright 2007 © Lycurgus Group, Inc. Professional Development •Good for both seasoned and new instructor •Develop new skills sets or discuss timely issues Training for Online Instructors Use the LCAQ (Lycurgus Competency Assessment Questionnaire) to identify training needs A competency-based assessment instrument developed to identify training needs of distance educators. Klein et.al. (2006) stated “competencies state quality standards for what is usually the single most important influence in instructor-led instruction.” (p. 196) Competencies are important in providing a foundation to an ongoing training program Can determine what king if developmental opportunities should be the focus of training for distance educators. SMEs rate areas in regards to importance to job and level of knowledge. High importance + low knowledge = Training areas Copyright 2007 © Lycurgus Group, Inc. Domain areas in LCAQ Specific competencies are assessed using the LCAQ that are connected with nine different domain areas. 9 Domains/48 Competencies Research on distance educator competency requirement used to develop LCAQ Ongoing validation of instrument Instructional Design Facilitation (includes motivating students) Multimedia Admin/Time Evaluation management (provide feedback) Policies/ Laws/Ethics Communication Copyright 2007 © Lycurgus Group, Inc. Technical Knowledge Development of community Knowledge & Skills of Distance Educators Create an Online Course Identify online voice Develop course materials Describe available resources (plagiarism detection, writing, publisher, internet &library) Total Client Classroom Development Solution Management/Online Learning Format Total Litigation Policies/Procedures Solution Time Management Assessment of Learning & Evaluation Copyright and confidentiality FERPA ADA/Accommodations Plagiarism Copyright 2007 © Lycurgus Group, Inc. Platform Functionality & Software Learn Software Programs Additional Tools to Use Training Modules Training modules available for different competency areas. Can complete on own time and/or participate in real-time training opportunities Provides an opportunity to develop a learning community All training available online Key is to develop training based on your needs that is ongoing to support and develop faculty Topics for 2007 are focused on resources, instructional design, multimedia options, facilitation and workload management. Copyright 2007 © Lycurgus Group, Inc. Summary The focus on quality in distance education is essential There are ways to stay competitive, which will enhance the marketability of programs and also attract and retain quality distance education instructors Assessment and Development are two keys to quality distance education Copyright 2007 © Lycurgus Group, Inc. What you can expect from us We will... Design tailored programs to measure and manage Student, Faculty and Staff experience and effectiveness Utilize surveys, focus groups, interviews and other data as part of assessment Develop, deliver and evaluate faculty training and development programs Conduct training needs assessment Conduct industry studies/identify benchmarks Determine instructional design needs Partner with your organization to achieve distance education excellence! Copyright 2007 © Lycurgus Group, Inc. Lycurgus Group Lycurgus Group Consulting Survey Assessment & Research Faculty Training & Development LEAP System Organizational Metrics Benchmarking Change Management Needs Assessment (LCAQ) Development & Delivery of Training Evaluation E-learning Instructional Design Course Delivery Evaluation Copyright 2007 © Lycurgus Group, Inc. Q&A or to discuss how we can help you! We can present solutions for your most difficult problems. www.lycurgusgroup.com (View presentation here) [email protected] [email protected] [email protected] Phone: (310) 543-0909 Fax: (310) 543-0050 Copyright 2007 © Lycurgus Group, Inc. References Aamodt, M.G. (2004). Applied industrial/organizational psychology. 4th edition. Belmont, CA: Wadsworth, Thomson Learning. Arnone, M. (2002, March 4) Online Education Must Capitalize on Students' Unique Approaches to Learning, Scholar Says (Electronic version). The Chronicle of Higher Education. Retrieved July 7, 2004, from http://chronicle.com/free/2002/03/2002030401u.htm. Cheung, D. (1998) Developing a student evaluation instrument for distance teaching. Distance Education, 19 (1), 23-43. Covington, D., Petherbridge, D. & Warren, S.E. (Spring 2005). Best Practices: A Triangulated Support Approach in Transitioning Faculty to Online Teaching, Online Journal of Distance Learning Administration, 8 (1), Retrieved February 25 2006 from http://www.westga.edu/~distance/ojdla/spring81/covington81.htm. DeSimone, C. (2006). Preparing our teachers for distance education. Heldref Publications. DeVellis, R. F. (2003). Scale development: Theory and applications. Thousand Oaks, California: Sage Publications. Copyright 2007 © Lycurgus Group, Inc. References Dominguez, P.S. & Ridley, D. (Sept. 1999) Reassessing the assessment of distance education courses. T.H.E. Journal, 27 (2), 70-74. Kaufman, R., Guerra, I., Platt, W.A. (2006). Practical evaluation for educators. Thousand Oaks, California: Sage Publications. Klein, J.D., Spector, J. M., Grabowski, B. & de la Teja, I. (2006). Instructor competencies: Standards for face-to-face, online and blended settings. The Quarterly Review of Distance Education, 7 (2), 195-199. Ko, S. & Rossen, S. (2004). Teaching online: A practical guide. 2nd edition. New York: Houghton Mifflin Company. Lockee, B., Moore, M., & Burton, J. (2002). Measuring success: Evaluation strategies for distance education. Educause Quarterly, 1, 20-26. Palloff, R. M. & Pratt, K. (2003). The virtual student: A profile and guide to working with online learners. San Francisco, California: Jossey-Bass. Pitton, D. E. (2006). Mentoring novice teachers. 2nd edition. Thousand Oaks, CA: Sage Publications. Copyright 2007 © Lycurgus Group, Inc. References Reisetter, M. & Boris, G. (Winter 2004). What works: Student perceptions of effective elements in online learning. Quarterly Review of Distance Education, 5 (4), 277-293. Schankman, L.H. (2004). Holistic Evaluation of an Academic Online Program. Retrieved May 3, 2006, from http://library.mansfield.edu/larry/evalplan.pdf Sims, R., Dobbs, G., & Hand, T. (Oct. 2002) Enhancing quality in online learning: Scaffolding planning and design through proactive evaluation, Distance Education, 23 (2), 135-149. Stewart, B.L., Waight, C.L., Norwood, M.M., & Ezell, S.D. (Summer 2004) Formative and summative evaluation of online courses. Quarterly Review of Distance Education, 5 (2), 101-110. Strong, J. H. ( 2006). Evaluating teaching. Thousand Oaks, California: Sage Publications. Wilson, C. (Fall 1998). Concerns of instructors delivering distance learning via the WWW, Online Journal of Distance Learning Administration, 1 (3). Retrieved June 2, 2004 from http://www.westga.edu/~distance/wilson13.html. Copyright 2007 © Lycurgus Group, Inc.