Transcript FAU’s Response to FL State Requirements for Academic
CS-90 Strategies for Aligning Assessment for Program Improvement with Accountability for Individual Student Learning
Thomas P. Pusateri, Ph.D.
Florida Atlantic University [email protected]
Concurrent session 110th Annual Meeting of the Commission on Colleges Southern Association of Colleges and Schools Atlanta Hyatt Regency Hotel December 5, 2005, 3:45-4:45 PM
1
http://iea.fau.edu
2
3
4
Background Information
• Increased emphasis on identifying and assessing core student learning outcomes in higher education (e.g.,
Measuring Up 2004
).
• The Florida Board of Governors included student
achievement as 1 of 7 of its accountability measures.
• A Board of Governors’ resolution (April 22, 2004) directed university personnel to develop Academic Learning Compacts (ALCs) and related processes.
5
What are ALCs?
“University personnel will provide students and prospective students with Academic Learning Compacts, which include concise statements of what active and successful participants in the joint teaching-learning process will know and be able to do, expressed in terms of the core student learning outcomes embodied in the requirements for each baccalaureate degree. The Compacts also will list the types of assessments used in the program. Compacts provided to students must be written in a user-friendly, jargon-free format.” Policy Guideline 04.08.26
State University System (of Florida) Division of Colleges and Universities
Florida Atlantic University’s is realigning its assessment system
to accommodate Academic Learning Compacts.
6
BoG Policy Guideline
University personnel will develop Academic Learning Compacts that: I.a. Identify, at a minimum, the expected core
student learning
outcomes for program graduates in the areas of
i. content/discipline knowledge and skills ;
;
* ; * these skills may be defined at either the university level or the program level
FAU’s Realignment
FAU’s programs identify one or more of each of the following outcomes:
– Vocabulary, Theories – Research skills – Technical skills *
* Written–Oral–Other–Team
Analytical – Creative – Practical FAU’s Writing Across the Curriculum requirement 7
BoG Policy Guideline
I.b. Identify corresponding assessments used to determine how well student learning matches those articulated expectations.
FAU’s Realignment
Departments assess all students through one or more of the following:
Where/When are students assessed?
How are students held to performance standards?
– Senior thesis – Senior project/portfolio – Field placement
8
BoG Policy Guideline
I.b. Identify corresponding assessments used to determine how well student learning matches those articulated expectations.
Where/When are students assessed?
How are students held to performance standards?
FAU’s Realignment
Departments establish standards through one or more approaches:
– Common syllabi – Common expectations – Common assignments
– Faculty define levels of student performance – Periodic calibration
Setting cut-off scores on required exams
9
BoG Policy Guideline
2. University personnel will
develop the evaluation
systems (including external validations) necessary to corroborate that the assessments referenced above measure student achievement against the expected core learning outcomes and results will be used to improve student achievement and program effectiveness.
FAU’s Realignment
FAU’s departments rely on one or more systems:
– Juried performances – 2+ faculty members – Non-faculty raters
Collection and periodic review of student work
Comparisons to norms on standardized tests
10
BoG Policy Guideline
[U]niversity personnel shall develop an action plan
that will be approved by their university’s board
of trustees and submitted to the Division of Colleges and Universities. The plan shall include a. a proposed timeline for developing policies and implementing procedures to capture each element outlined above, as well as in making Academic Learning Compacts readily available to students.
FAU’s Realignment
FAU’s action plan
• Briefings for Board of Trustees and University Undergraduate Programs Committee (Fall 04) • Consultations with academic departments (04-05) • Progress report to Faculty Senate Chair (Mar 05) • Board of Trustees - Workshop (Apr 05) - Approves timeline (May 05) • Faculty Senate approves ALC Guidelines (Oct 05) • Board of Trustees approves ALC Policies (Nov 05) • ALCs will be made available to students (Dec 05) 11
12
13
BoG Policy Guideline
REVIEW OF ALCs
University personnel are expected to demonstrate how information from the periodic review of student learning outcomes, as well as from the evaluation of corresponding assessment mechanisms, has been used to improve student achievement and program effectiveness.
FAU’s Realignment
Departments will report their evaluation (assessment) strategies by February 1, 2006.
14
Reporting samples from core required courses or capstones on the distribution of scores obtained on embedded questions, key assignments, rubrics, etc.
15
Assessing multiple raters
16
Assessing multiple raters
17
FAU’S DEFINITIONS OF CONTENT KNOWLEDGE IN THE DISCIPLINE
Declarative knowledge: Students will demonstrate knowledge of the vocabulary, history, theories or concepts specific to the discipline. Declarative knowledge is usually assessed via in-class or standardized tests, typically in an objective (multiple-choice; short answer) format. Procedural knowledge: Research skills: Students will demonstrate knowledge of the procedures involved in discipline-specific research (e.g., idea generation, literature review, data collection, reporting). These skills are usually measured via the evaluation of a research project, thesis, or dissertation.
Procedural knowledge: Technical skills: Student will demonstrate technical skills related to the discipline (e.g., developing a business report or lesson plan, mastering a musical instrument, learning how to use statistical software). This type of procedural knowledge may be assessed via an in-class project (e.g., case study, paper), observation (e.g., juried performance), or portfolio. 18
FAU’S DEFINITIONS OF COMMUNICATION SKILLS
Written communication: Students will produce writing that is grammatically correct, well-organized, and properly formatted for the purpose of the assignment and the discipline. This is usually assessed via individual paper assignments where instructors assess the quality of written expression.
Oral communication: Students will prepare and deliver informative and/or persuasive oral presentations that attend to the audience and are well-organized. This is usually assessed via individual oral presentations in a classroom setting, although sometimes instructors and/or supervisors may assess oral communication across time (e.g., during classroom discussions or at an internship site). Other forms of communication: Students will prepare and present information and/or persuasive material using media that may or may not include written or oral communication (e.g., a musical or theatrical performance; a visual, audiovisual, or graphical product). This is usually assessed via classroom projects, juried performances, and/or portfolios.
Team/Collaborative communication: Students will demonstrate team-oriented, collaborative skills in which they contribute to group products. This is usually assessed via group papers or presentations and may involve peer-assessment (e.g., group members assess each other) and/or assessment by audience members (e.g., an instructor, supervisor, or peer).
19
BoG Policy Guideline
b. A description of how university personnel will certify that each baccalaureate graduate has completed a program with clearly articulated core student learning expectations… and that corresponding robust and effective assessment mechanisms have been used to ensure that graduates have met the criteria of the Compacts.
FAU’s Realignment
FAU will certify students through the awarding of a baccalaureate degree from programs that have articulated ALCs and that require students to achieve acceptable performance standards prior to graduation via completion of one or more of the following: (a) senior capstone course, (b) exit examination (c) a series of core required courses.
20
FAU’S DEFINITIONS OF CRITICAL THINKING SKILLS
*
Analytical skills: Students will analyze, evaluate, compare/contrast or judge discipline-specific theories, issues, events, or other content. This is usually assessed through examining the quality of argument in a student essay, oral presentation, or formal report. Creative skills: Students will create a product by synthesizing knowledge from a discipline (e.g., create a new piece of art, interpret a piece of music in a personal and appropriate way, develop a new theory or research proposal). This is usually assessed by an instructor or supervisor examining a student product for the quality of its synthesis of current knowledge into a new product.
Practical skills: Students will put into practice their knowledge and skills within a discipline (e.g., developing a business proposal or lesson plan, writing a grant). This is assessed ideally by examining an authentic product the student employs in an appropriate setting (e.g., a student teacher delivering a lesson, an intern proposing a plan to a supervisor); it may also be assessed via in-class projects in which students propose practical solutions based on their understanding of the discipline (e.g., case study analysis).
* Based on Robert Sternberg’s Theory of Successful Intelligence 21
EXAMPLES OF ALCs
CONTENT KNOWLEDGE
PSYCHOLOGY (Terms, theories): Graduates in Psychology will understand basic concepts, theories, and experimental findings in four core areas of psychology (cognitive, developmental, social, and psychobiology). HISTORY (Research & Technical skills): Students will demonstrate the ability to conduct archival research utilizing both primary and secondary sources. They will develop procedural and technical skills involved in historical research such as library research, oral history, book review writing, and proper citations. FINANCE (Technical skills): Students will develop proficiency in the use of Microsoft Excel to solve financial problems.
22
EXAMPLES OF ALCs
COMMUNICATION SKILLS
GEOGRAPHY: Graduates in Geography will be able to produce writing that is grammatically correct and well-organized and to deliver clear and well-organized oral presentations. Both written and oral communications should illustrate clear communication skills utilizing graphic tools in geographic information systems, remote sensing and other techniques. REAL ESTATE: Students will demonstrate their abilities to report the results from their feasibility analyses both orally and in written form in a clear and grammatically sound manner. Students will also demonstrate skills in discussing the analyses presented by their peers. TEACHER EDUCATION: Students will demonstrate the ability to communicate effectively verbally and in writing (EAP 2, Indicator 2.2) by successfully developing and implementing a lesson plan. 23
EXAMPLES OF ALCs
CRITICAL THINKING SKILLS
ENGLISH: Students will analyze/interpret a complex literary work, demonstrating the capacity to formulate and develop in a formal essay a significant topic in depth and in an independent and even original way. Essays are expected to exhibit mature creative and analytical thinking, and an ability to create a mature critical argument quickly with no external assistance.
COMPUTER ANIMATION IN ARTS: Students will demonstrate proficiency in understanding and apply the stages of problem solving skills (e.g. research, thumbnail, roughs, comprehensive and mechanical preparation) as applied to 3D computer arts. ELECTRICAL ENGINEERING: Design and conduct scientific and engineering experiments including analysis and interpretation of data. Deliver engineering results that meet performance standards for cost, safety, and quality. Make and defend ethical judgments in keeping with professional standards.
24
CORE REQUIRED COURSES
PSYCHOLOGY: For each of the courses listed below, faculty teaching that course will develop a master syllabus which will include student learning outcomes to be addressed in all sections of the course. Faculty will share strategies for developing course assignments (e.g., exams, exercises, term papers) to assess student attainment of the learning outcomes. The courses are: EXP 3505: Cognition, DEP 3054: Psychology of Human Development, SOP 3004: Social Psychology, and PSB 3002: Biological Bases of Behavior.
MUSIC: All students must complete four specific courses in the history and literature of music: MUH 3013 (Introduction to Music History), MUH 4211 (Music of Western Civilization I), MUH 4212 (Music of Western Civilization II), and MUH 4371 (20th Century Music History).
25
CAPSTONE EXPERIENCE
SOCIAL WORK: All BSW students are required to take SOW 4510 (Field Education in Social Work) , which includes a classroom field seminar plus 400 agency-based hours supervised by a professional social worker. This experience integrates classroom knowledge from the Human Behavior in the Social Environment (HBSE) sequence, practice sequence, research sequence, and policy sequence; students use this knowledge base to develop skills to intervene with various clients. HONORS COLLEGE: During the senior year, students will enroll in thesis research and thesis writing and will produce a draft honors thesis to be evaluated. To prepare for writing a senior thesis, students will complete (1) courses in their concentration aimed at providing the content knowledge needed to write a thesis, (2) writing courses aimed at providing competence in effective written communication, and (3) courses in the core and concentration designed to build critical thinking skills. 26
STANDARDIZED TEST
NURSING: Approximately three months prior to graduation, all seniors will take the Health Education Systems, Inc. (HESI) examination . This test is a valid predictor of performance on the National Licensing Examination (NCLEX). Students must achieve a score of 850 or better on the HESI to graduate. Students who do not receive a passing score are provided with advising and a remediation plan and the opportunity to re-take the exam; they will not graduate until their score is 850 or better. OCEAN ENGINEERING: One of the requirements for graduation is that a student must take the nationally-administered Fundamentals of Engineering (FE) examination networks and strength of materials. . The examination tests one's knowledge of basic and fundamental engineering subjects: statics, dynamics, thermodynamics, fluid mechanics, computer programming, engineering economics, electrical 27
STANDARDIZING COURSES
PSYCHOLOGY:
Faculty teaching core courses will develop a master syllabus which will include student learning outcomes to be addressed in all sections of the course exercises, term papers) . Faculty will share strategies for developing course assignments (e.g., exams, to assess student attainment of the learning outcomes. ACCOUNTING: Faculty members teaching the undergraduate accounting core courses: ACG 3131 (Financial Reporting), ACG 3341 (Managerial Decision Making and Accounting)., and TAX 4001 (Federal Taxation I) are required to teach a set number of chapters from a common text selected by departmental decision. Faculty members teaching a particular course will develop and periodically review a common list of educational objectives and the topics and critical thinking skills appropriate to these objectives to teach in each course.
28
GRADING RUBRICS
ENGLISH: Essays in all 3000- and 4000-level English courses will be assessed holistically according to a rubric developed and shared by faculty and communicated to students.
SUPERIOR
Addresses the question or prompt fully Demonstrates substantial comprehension of relevant material Shows substantial depth, complexity, and creativity of thought Demonstrates clear and coherent organization Develops arguments fully with ample supporting detail Demonstrates superior control of diction, syntactic variety, and transitions
STRONG
Addresses the question or prompt substantively, yet not fully Demonstrates comprehension of relevant material Shows depth, complexity, and creativity of thought Demonstrates clear organization Develops arguments with supporting detail Demonstrates control of diction, syntactic variety, and transition 29
(continued)
COMPETENT
Adequately addresses the question or prompt Demonstrates adequate understanding of relevant material Shows clarity of thought but may treat the topic simplistically or repetitively Demonstrates adequate organization Develops arguments adequately, with some detail Demonstrates adequate facility with syntax, mechanics, and usage but contains some errors
INADEQUATE
May distort or neglect parts of the question or prompt, and/or Fails to comprehend relevant material Lacks clarity of thought; demonstrates confused/simplistic thinking, and/or Lacks adequate organization, and/or Fails to provide adequate or appropriate details to support generalizations, or may provide details without generalization, and/or Demonstrates significant errors in language, syntax, or mechanics 30
Honors with distinction: appropriate to the area of study. The thesis demonstrates a profound knowledge of the fundamental concepts in the subject area. The thesis shows mastery of grammar, syntax, structure and style: it is lucid, well-organized, and stylistically elegant. The thesis engages in sophisticated critical thinking and rigorously considers alternate resolutions of a problem in an exemplary fashion or otherwise shows sophisticated critical thinking in an exemplary manner
Honors:
The thesis shows a thorough knowledge of the fundamental concepts in the subject area. The thesis shows effective command of grammar, syntax, structure and style: it is clearly written and sensibly organized. The thesis engages in sophisticated critical thinking and explores alternate approaches or resolutions of a problem effectively or otherwise shows sophisticated critical thinking in a manner appropriate to the area of study.
Unacceptable:
problem.
The thesis does not show competent knowledge of the fundamental concepts in the subject area. The thesis does not show mastery of grammar, syntax, structure, or style. The thesis does not competently engage in critical thinking or explore alternate approaches or resolutions of a 31
CUT-OFF SCORE ON STANDARDIZED EXAM
NURSING: Approximately three months prior to graduation, all seniors will take the Health Education Systems, Inc. (HESI) examination. This test is a valid predictor of performance on the National Licensing Examination (NCLEX). Students must achieve a score of 850 or better on the HESI to graduate.
Students who do not receive a passing score are provided with advising and a remediation plan and the opportunity to re-take the exam; they will not graduate until their score is 850 or better.
COMPARISON TO NORMS ON STANDARDIZED TESTS
32
MULTIPLE RATERS
MUSIC
CONTENT KNOWLEDGE (Technical skills) and COMMUNICATION and CRITICAL THINKING (Creative skills): Graduates will have ability to perform a cross section of the music from the complete repertoire of the performance medium and exhibit the appropriate technical skills for artistic self-expression. Students in the Bachelors Music program will be appropriate faculty at jury performances each semester also required to successfully complete a senior recital, which must include representative examples from the appropriate repertoire for the performance medium. scheduled.
evaluated by the . They are A pre-hearing evaluated by three faculty members must be approved before the recital is 33
SAMPLING
LANGUAGES & LINGUISTICS: Will randomly sample the performance of students writing term papers for program-specific 3000- and 4000-level courses to compare demonstrated research skills for these with the research-skill performance of students taking and passing FOL 3880.
Will randomly sample the accuracy of linguistic structures and attention to linguistic nuance in the essays of students enrolled in selected 3000- and 4000-level courses to compare rankings of student performance in linguistics-specific courses with rankings of students’ linguistic awareness as evidenced in other courses. JEWISH STUDIES: Periodically selected exams and papers will be read by members of the Jewish Studies Executive Committee to ensure that the criteria of the Academic Learning Compacts are being met by individual students.
34
National Center for Public Policy and Higher Education “Measuring Up” 2004 report
Florida Board of Governors
Accountability Measures
1. Number/Percent of students from underserved populations enrolled/graduating 2. 4- and 6-year graduation rates 3. Access to and production of degrees 4. Meeting statewide workforce needs 5. Research expenditures from external sources 6. World-class academic research programs 7. Student learning: Academic Learning Compacts
36
37