Transcript Slide 1
University-Wide Assessment ePortfolio: From Planning to Implementation and Beyond Debra Sowell, Ph.D. Dean of Instruction Oral Roberts University Oral Roberts University Christian, non-denominational 5,200 students served 65 undergraduate majors 10 masters degree programs 2 doctoral degree programs Students from 50 states & 66 countries Ethnic minorities = 26% African American = 18% ORU Mission It is the mission of Oral Roberts University, in its commitment to the historic Christian faith, and to the University’s founding vision, to assist students in a quest for knowledge of and relationship to God, humanity, and the universe. Dedicated to the realization of Truth and the achievement of one’s potential life capacity, the University seeks to educate the whole person in spirit, mind, and body, thereby preparing its graduates to be spiritually alive, intellectually alert, physically disciplined, and socially adept. Fulfilling the Mission General Education Curriculum Major Curricula Co-Curricular Activities Graduate School Curricula 2002: Buy-In to New Paradigm Input to output Product of ORU is a changed life Teacher-centered to student-centered Emphasis on engaged learning Teacher as mentor: Not only “sage on the stage,” but also “guide on the side.” Note: Technology was not part of this paradigm. A purposeful educational program starts at the endpoint, with the desired characteristics of an institution’s graduates, and asks the faculty to reason backwards from outcomes to the implementation of an intentionally designed curriculum to cultivate the desired qualities. In such a purposeful program, faculty members are expected to employ teaching practices that advance the desired learning outcomes; to assess how well the curriculum and instruction succeed; and then to make adjustments for still greater success. “Taking Responsibility for the Quality of the Baccalaureate Degree,” AAC&U Report, 2004. Learning Process Mission Outcomes with Performance Indicators Curriculum Assessment Pedagogy Adapted from Greater Expectations Process of AACU. New Outcomes (See Pink Handout) Sixteen proficiencies/capacities to document progress with ORU’s four outcomes were developed. Learning Process Mission Outcomes with Performance Indicators Curriculum Assessment Pedagogy Adapted from Greater Expectations Process of AACU. Course Application (See Green Handout) Faculty designate how significantly every course contributes to each of ORU’s 16 proficiencies/capacities. Learning Process Mission Outcomes with Performance Indicators Curriculum Assessment Pedagogy Adapted from Greater Expectations Process of AACU. A purposeful educational program starts at the endpoint, with the desired characteristics of an institution’s graduates, and asks the faculty to reason backwards from outcomes to the implementation of an intentionally designed curriculum to cultivate the desired qualities. In such a purposeful program, faculty members are expected to employ teaching practices that advance the desired learning outcomes; to assess how well the curriculum and instruction succeed; and then to make adjustments for still greater success. “Taking Responsibility for the Quality of the Baccalaureate Degree,” AAC&U Report, 2004. Learning Process Mission Outcomes with Performance Indicators Curriculum Assessment Pedagogy Adapted from Greater Expectations Process of AACU. Student Learning Assessment Although it may seem to be the most obvious way to assess the quality of undergraduate education, the use of direct measures of student learning is uncommon. “Looking Where the Light is Better,” 2002 Winter/Spring Peer Review, Marc Chun Critical Thinking Two of the following required artifacts: BIO 101L/Principles of Bio. Lab Sunflower germination lab report BIO 111L/Introductory Bio. I Lab Yeast experiment lab report BIO 112L/Introductory Bio. II Lab Wetlands report BIO 310L/Microbiology Lab Unknown bacteria lab report CHE 101L/Principles of Chemistry Meat analysis lab abstract Laboratory CHE 111L/General Chem. I Lab Dimensional analysis lab problem set CHE 112L/General Chem. II Lab Kinetics lab abstract PSC 101L/Principles of Physical Science Laboratory PSC 201L/Principles of Earth Science Laboratory Acceleration of gravity measurement lab report Relative humidity lab report Meat Analysis Lab Abstract (CHE 101L/Principles of Chemistry Laboratory) (2A) Outcome Intellectually Alert Proficiency Critical Thinking (2A) Definition The ability to integrate knowledge in order to identify and weigh possible responses to different situations, and to process information—both analytically and critically—so as to determine the validity of different, competing claims Content Purpose: To identify one of the four types of meat as the healthiest and the most costeffective (high quality but low cost). Hypothesis: Before determining the results, which one of the four meats do you think will be the healthiest and the most cost-effective? Write a summary (abstract) of the Meat Analysis experiment. Include brief descriptions of the following: 1. The purpose and the three factors we are determining in the experiment. 2. Your hypothesis and the reasons for your hypothesis. 3. How we experimentally measured the three factors and how we separated the compounds. 4. The experimental results from the different types of meat and how we used those results to determine which is the healthiest and most cost-effective. 5. How the results addressed your hypothesis. Include which meat was the healthiest and most cost-effective and how what we determined affects you as a consumer. Format Use between 250 and 300 words (do a word count). This should be a double spaced Word document submitted by ePortfolio to your lab instructor. Grade The grade on this abstract will be included in the “exercises” portion of the grade and is worth 10 points for completion. It will be evaluated for ePortfolio according to the rubric on the next page, but the points for the lab class are solely dependent on assignment completion. Meat Analysis Lab Abstract (CHE 101L/Principles of Chemistry Laboratory) (2A) Criteria Purpose or goal Exemplary Identifies clearly and addresses consistently a realistic and significant purpose or goal Hypothesis Identifies and Identifies an addresses important consistently an hypothesis important hypothesis Identifies a hypothesis Evidence Presents clearly and accurately all of the necessary data Presents clearly and accurately most of the necessary data Presents Presents Does not accurately some of inaccurately some present data the necessary data data or irrelevant data Conceptual understanding Responds clearly and effectively using appropriate and sufficient course content and outside sources Presents and justifies crucial and consistent assumptions Responds clearly using appropriate and sufficient course content Responds using appropriate and sufficient course content Presents and Presents justifies consistent consistent assumptions assumptions Presents Does not extraneous and/or present contradictory assumptions assumptions Draws clear and valid conclusions or inferences supported by content Draws valid conclusions or inferences supported by content Draws inaccurate Does not draw conclusions or conclusions or inferences inferences Assumptions Inferences Competent Acceptable Identifies clearly a Identifies a realistic and reasonable significant purpose purpose or goal or goal Draws valid conclusions or inferences Unacceptable Identifies an unclear and/or unrealistic purpose or goal Not Attempted Does not identify a purpose or goal Identifies an insignificant hypothesis Does not identify a hypothesis Responds using Does not insufficient and/or respond using inappropriate course content course content Definition The ePortfolio is an assessment tool that documents progress toward fulfilling the University's mission, which focuses upon educating students as whole persons by preparing them to achieve the ORU Student Learning Outcomes. Definition Artifacts and evaluations in the ePortfolio provide feedback for the students' reflection to improve their performance and demonstrated growth. Students are evaluated on uploaded artifacts. Honor Code Reflection Paper 2004-2005 ePortfolio Performance Levels by Rubric Exemplary Competent Acceptable Unacceptable Not Attempted TOTAL 188 271 101 13 13 586 PERCENT 32% 46% 17% 2% 2% 100% Honor Code Paper Competent or Higher = 78% N Honor Code Reflection Paper 2004-2005 ePortfolio Performance Levels by Sex Male PERCENT Exemplary Competent Acceptable Unacceptable Not Attempted N 30% 45% 18% 4% 2% 100% Male Competent or Higher = 75% Female PERCENT Exemplary Competent Acceptable Unacceptable Not Attempted N 32% 46% 18% 2% 2% 100% Female Competent or Higher = 78% Honor Code Paper (GEN 099/ePortfolio: Journey to a Transformed Life) (1D) Exemplary Competent Acceptable Unacceptable Not Attempted Identification of ethical dilemmas 32% 47% 15% 3% 3% Use of the Bible in handling ethical dilemmas 28% 31% 34% 2% 5% 586 Knowledge of university Honor Code 26% 56% 15% 3% 0% 586 Use of university Honor Code in handling ethical dilemmas 29% 48% 17% 2% 4% 586 Ability to make morally correct choices 44% 44% 10% 2% 0% 586 Use of personal ethical/moral code 43% 44% 11% 2% 0% 586 Analysis of ethical dilemma 4% 3% 21% 49% 23% 586 Criteria N 586 ePortfolios Each undergraduate student constructs at least two ePortfolios: General Education Major Others (professional, personal, fun, informative) Each graduate student constructs at least one ePortfolio Graduate Others (professional, personal, fun, informative) ePortfolio Implementation Time-Line • 2001 -- School of Education using the LiveText technology • 2002-2003 -- Education changed to Chalk and Wire technology • May 2003 -- Decision was made to launch a university-wide ePortfolio system • 2003-2004 -- School of Nursing and the Engineering Department initiated ePortfolio programs ePortfolio Implementation Time-Line • 2003-2004 — All schools and academic departments of the University, along with General Education, Spiritual Life, Student Affairs, and the College and Career Guidance Center, engaged in hundreds of discussions regarding which assignments in which courses should become artifacts and constructed rubrics for those artifacts. Faculty/staff training began in earnest. ePortfolio Implementation Time-Line • 2004-2005 — Seven departments in the School of Arts and Sciences, the undergraduate School of Business, and the graduate School of Theology, along with General Education (supported by Spiritual Life, Student Affairs, and the College and Career Guidance Center) launched ePortfolio, training the students and continuing faculty/staff training. ePortfolio Implementation Time-Line • 2005-2006 — The remaining four departments in the School of Arts and Sciences launched ePortfolio during the fall semester. The last unit to come on-line will be the graduate School of Business in the spring semester. ePortfolio Implementation Time-Line Fall 2003 Education Spring 2004 Fall 2004 Spring 2005 Fall 2005 Engineering, Physics Communication Arts Chemistry Biology Nursing (Juniors) Computer Science, Mathematics Graduate Theology Behavioral Science Nursing (All) UG Theology HPER General Education UG Business Art Modern Language Music LifeLong Education Spring 2006 Graduate Business Challenges Implementing and coordinating such a major and pervasive program at all levels of the University including academic departments, co-curricular areas, administration, staff, and students. Training faculty, staff, and students to effectively use the technology. Challenges Tracking the progress of students in submitting the appropriate artifacts correctly, and faculty in evaluating and assessing student artifacts in a knowledgeable and timely manner. Restructuring deadlines and plans due to technology issues at the end of the semester. Designing and redesigning the GEN 099—“ePortfolio: Journey to a Transformed Life” class to better enable all students to succeed at ePortfolio. Benefits of ePortfolio ORU can document that it is accomplishing its mission by producing and proving that it is delivering whole-person graduates. More and better writing assignments and more service learning projects have become a part of the curriculum. Benefits of ePortfolio Students know what to aim for and receive guidance on how they can continue to improve. Students become aware that learning transcends individual courses by experiencing specific competencies assigned and measured across the curriculum. Benefits of ePortfolio The mission of the University is continually before the students as they prepare artifacts and examine rubrics; thus, students are provided with a complete picture of Oral Roberts University’s outcomes and proficiencies. Benefits of ePortfolio ePortfolio artifacts provide direct evidence of learning, and rubrics provide direct assessment of learning. Benefits of ePortfolio This common assessment system increases communication and coordination of faculty and staff across departments. Data can be aggregated and disaggregated for decision makers seeking to improve student, department, school, and university performance, and to prepare for accreditation. Benefits of ePortfolio The general education program has become more integrated. Gaps and inconsistencies are revealed in the curriculum, and strengths are highlighted. Benefits of ePortfolio ePortfolio heightens student interest because it is technology-related and studentmaintained. Benefits of ePortfolio Students are guided within a learnercentered environment. Graduating students can develop an additional professional ePortfolio that could be used when seeking employment. Benefits of ePortfolio Co-curricular activities (RA, Chaplain, Missions, Community Outreach, Music Ministries, Chapel) are integrated with academics. Benefits of ePortfolio Potential donors to the university can grasp the value of ePortfolio and choose to contribute. ORU is becoming a model for universitywide implementation of ePortfolio. Benefits Specific to Faculty Development Faculty Development Components Re-examination of University Mission Statement to develop a framework for learner-centered paradigm Faculty Development Components Comprehensive study of the general education curriculum, including pedagogy and assessment Faculty Development Components Revisiting old ideas and approaching them from a fresh perspective: Assessment Rubrics Validity Curriculum Mapping Faculty Development Components Enhanced technology training Faculty Development Components Cross-disciplinary activities Interaction with co-curricular areas (whole person approach) Overall Benefit ePortfolio helps enable Oral Roberts University to produce transformed students—student lives are changed, and students are prepared to fulfill their destinies and change the world. Taking the Idea Home Identify how your campus documents the ability of graduates to think critically. Identify a specific assignment that could serve as a rubric-evaluated artifact to assess critical thinking. Choose 4 criteria from the rubric. Discuss how the criteria will help you evaluate the assignment. ORU’s Portfolio Handbooks and Resource Information (copyrighted) www.oru.edu Click “Academics” Click “ePortfolio”