Assessing and Assuring Graduate Learning Outcomes A PRACTICAL WORKSHOP ON ASSESSMENT WITH ALVERNO COLLEGE SYDNEY 28 - 29 MAY 2013

Collaborative national project supported by ALTC / OLT with strategic priority funding › What are the assessment tasks in a range of disciplines that generate convincing evidence of achievement of graduate learning outcomes?

› What are the assurance process trusted by disciplines in relation to those assessment tasks and judgements?

Situational analysis, literature review, expert reference group, institutional visits, interviews with 48 academics 28 universities in 7 disciplines Ten key issues papers, endnote library, submissions to government discussion papers, research report – 5710 unique page views…and counting. Full story at: http://www.itl.usyd.edu.au/projects/aaglo/ 2

SOME AAGLO HEADLINES 1. THE NATURE OF GRADUATE LEARNING OUTCOMES 1. Coherent assessment of program level graduate learning outcomes requires coherent institutional and discipline statements of outcomes - but what that statement ‘is’ matters.

› Not

what

skills are included – but the

nature

of the skills, and what that says in terms of traditional or contemporary conceptions of knowledge.

Some of the ways we have articulated GLOs are limiting - taking us backwards to outdated expectations of the goals of higher education – not forwards to the contemporary outcomes and purposes we espouse.

3

SOME AAGLO HEADLINES 2. ASSESSMENT CHOICES 2. What we assess and how we assess it shapes but:

curriculum

and

teaching

as much as it does learning; we do not have to assess everything that is taught › Assessment prioritizes teaching of some GLO and ignores others › Communication skills – privileged › Information literacy - privileged › Research and inquiry – (less) privileged › Ethical social professional understandings - neglected › Personal intellectual autonomy - neglected Consequences for students 4

SOME AAGLO HEADLINES 3. ASSESSMENT DESIGN 3. Core features of effective assessment practices identified by respondents › interconnected multi-component › authentic, relevant › standards-based with effective communication of criteria › involve multiple decision makers – including students › …..coupled with effective assurance processes around the quality of these tasks and the judgments made 5

SOME AAGLO HEADLINES 4. QUALITY ASSURANCE OF ASSESSMENT 4. Variety of assessment quality assurance strategies › Quality of task: Pre-implementation (policies, curriculum approval processes for new and revised tasks) Post- implementation (evaluation by students, examiners, peers, curriculum audit & review) › Quality of judgment: Pre-judgement - (calibration) Post-judgement – (consensus moderation) › Confidence? Combination of effective task and assurance was less common than it could be Evidence of ‘assurance’ engaging staff and leading to ‘enhancement’ was rare 6

FUTURE RECOMMENDATIONS ›

Recommendation 1:

That Universities review their choice of assessment tasks in programs and units of study to ensure these include design features that provide convincing evidence of achievement of GLOs.

›

Recommendation 2

: That universities provide professional learning opportunities and support for academics to develop effective discipline based assessment for the assurance of GLOs 3,4,5,6,…..

›

Recommendation 7

: That the OLT continue to support initiatives that engage academics in productive, evidence-based dialogue around the issue…… 7

Thank you to the OLT for their wonderful support in bringing you Alverno College!

Introducing Kathy Lake - Alverno Jeana Abromeit - Alverno Clair Hughes – AAGLO

From Principles to Practice

Jeana Abromeit

Associate Vice President for Academic Affairs, Professor of Sociology

Kathy Lake

Associate Vice President for Academic Affairs, Professor of Education

Alverno College

Outcomes for Session

 To have a sense of each other’s educational principles and their meaning for current and future practices -- both unique and shared  To have an initial understanding of Alverno’s educational program to start a meaningful conversation  To explore principles for design of practice: curriculum, teaching, learning, and assessment

Shared Values and Principles

Our Goal: Reach agreement on a set of important underlying principles of higher education

Individual and Group Activity

Principles I hold:

Discuss your list with others at your table.

Principles which our group holds:

Discuss shared principles with all participants

In 1970, one of the questions the President asked each department to address:  What are you teaching that is so important that students cannot afford to pass up courses in your department?

  Which led to questions like: How do you want your students to be able to think as a result of studying your discipline?

What do you want your students to be able to do as a result of studying your discipline?

In 1973, new curriculum initiated based on eight institutional abilities or outcomes



Graduation requirements based on demonstration of outcomes rather than distribution requirements.

PRINCIPLE

Education GOES BEYOND Knowing to being able to

DO WHAT ONE KNOWS

PRACTICE

 Performance in the context of

content

PRINCIPLE

Educators are RESPONSIBLE for making learning more available by ARTICULATING OUTCOMES and making them PUBLIC

PRACTICE

 Effective Communication  Analysis  Problem Solving  Valuing in Decision Making  Social Interaction  Developing a Global Perspective  Effective Citizenship  Aesthetic Engagement

     

Abilities that

Involve the whole person Are teachable Can be assessed Transfer across settings Are continually re-evaluated and re-defined

PRINCIPLE

Abilities need to be defined in a way that our teaching of them can be DEVELOPMENTAL

The Alverno Impact

  As you watch the clips of student performance, how have students changed?

Students are being assessed for communication ability. What other abilities are also demonstrated?

PRACTICE

Analysis

 Level 1 – Show observational skills  Level 2 – Draw reasonable inferences  Level 3 – Perceive and make relationships  Level 4 – Analyze structure and organization  Level 5 – Employ frameworks from major and support disciplines in order to analyze  Level 6 – Independently employ frameworks

PRINCIPLE

Designing for learning involves

INTEGRATING abilities with

disciplines and across disciplines

PRACTICE



Coherent curricular design – Sociology example



Fusion of content and abilities

 Self Assessment and Feedback

PRACTICE in AUSTRALIA



Draft Standards for Learning Outcomes from Australian Higher Education Standards Panel: Standards 3, 4 & 6

PRACTICE

Examples from Outcomes for Chemistry

1.

Communicates effectively, using language, concepts, and models of chemistry 2.

Uses methodology of chemistry to define and solve problems individually and collaboratively

PRACTICE

Examples of Outcomes for Business and Management

1.

Uses discipline models and theories to analyze interdependence among systems, organizations, individuals, and events (Systems Thinking and Analysis) 2.

3.

Applies business and management principles to develop and deliver quality products or services (Enterprising and Problem Solving) Uses team and organizational skills to work effectively with diverse individuals, teams, and organizational units to meet stakeholder and organizational goals (Interacting and Leading)

PRINCIPLE

Abilities must be carefully

IDENTIFIED and COMPARED

to what CONTEMPORARY

LIFE requires

PRACTICE



Ability departments



Scheduled Institutes



Ongoing revisions of outcomes

PRINCIPLE

INTEGRAL

to LEARNING is ASSESSMENT

Student Assessment-as-Learning Institutional and Program Assessment

Student Assessment-as-Learning

 A process in operation at Alverno College, integral to learning, that involves observation and judgment of each student’s performance on the basis of explicit criteria, with self assessment and resulting feedback to the student.

 It serves to confirm student achievement and provide feedback to the student for the improvement of learning and to the instructor for the improvement of teaching.

Student Assessment-as-Learning Essential to learning and assessment are:

   

Public, Expected Outcomes/Criteria Performance Feedback Self Assessment

Institutional and Program Assessment

 Processes that yield patterns of student and alumna learning, development, and performance on a range of educational outcomes. They provide meaningful feedback to faculty, staff, and various publics for improvement, shared learning, and demonstrated effectiveness.

 These processes ensure comparisons to standards (faculty, disciplinary, professional, accrediting, certifying), and enable evidence-based judgments of how students and alumnae benefit from the curriculum and college culture.

PRINCIPLE

The effectiveness of assessment for everyone involved depends on the existence of a total dynamic system that contributes to the coherence and continuous improvement of the curriculum

PRACTICE



Matrix structure



Discipline departments



Office of Research and Evaluation

STUDENT LEARNING EXPECTED OUTCOMES (INSTITUTION: PROGRAM/COURSE: COURSE/UNIT) INSTITUTIONAL/ PROGRAM ASSESSMENT ASSESSMENT TEACHING EDUCATIONAL PRINCIPLES

e e

as . sess ( ses’) v.t.

[late ME assess (us) ptp. of assidere (ad + sedere)] to sit down beside

Goals For This Session

Consider the relationships among institutional outcomes, program outcomes, course outcomes, and criteria.

Definition of Outcomes

Outcomes are a set of statements that describe what a student can do with what s/he knows as a result of a set of learning experiences.

Characteristics of Outcomes

 Generic abilities, not tasks  Multidimensional and holistic  Cognitive, affective and kinesthetic dimensions  Transferable

Sources of Outcomes

      Institutional mission Accreditation standards Professional requirements Needs of society Nature of Discipline Aspirations of learners

Increasing Degrees of Specificity

Institutional Outcomes Major Outcomes & Accreditation Standards Course Outcomes Assessment of Course Outcomes Criteria (more specific than Outcomes) Judgment of Student Performance Using Criteria

Benefits of Using Outcomes

    Provide direction for learning Provide continuity Provide departments with standards to judge university level learning Assure accountability by making teaching/learning public and explicit

More Benefits of Using Outcomes

  Help students transfer learning because learning experiences are embedded in multiple performance contexts Foster cross-discipline communication: creates a community of faculty with common goals

Continuum of Increasing Specificity

  Outcomes become more specific and contexualized the closer they come to the student and his or her performance Criteria are outcomes at their most specific and most contextualized

Connections to the Disciplines

  Outcomes help students translate experience into learning Outcomes help facilitate efficient, coherent, consistent assessment procedures

Examples of Outcomes for English

1.

2.

Communicates an understanding of literary criticism, questions its assumptions, and uses its frameworks to analyze and evaluate works. Reads and interprets diverse cultural expressions in works of literature, film, and other media.

Examples of Outcomes for Computing and Information Technology

1.

2.

Effectively uses computing frameworks, both independently and collaboratively, to analyze, develop, implement, and evaluate solutions to diverse problems. Appling the principles of usability to the design and implementation of components, products, and systems.

Guidelines for Developing Course Outcomes

   What do you want students to be able to do with what they know?

How does your course assist students to be effective?

What abilities do students need to do what you want them to do in the course/at the end of the course?

Guidelines for Developing Course Outcomes

   What processes are involved in using the knowledge students have gained?

How is student performance evaluated with regard to both knowledge and application in this course?

Why should a student not graduate without having had your course?

Establishing Context

    How does this course relate to the major/program, general education and institution?

Who are the students in this course?

What are the abilities to be developed?

What are the key concepts to be learned?

Connections between Outcomes and Assessment

How can we take outcomes and turn them into something concrete and observable that students could be expected to do?

Create learning and assessment experiences and develop criteria,

all of which are explicitly linked to student learning outcomes.

Criteria -- A Definition

   Indicators of an integration of ability and knowledge as seen in performance Breaking open an outcome into specifics C riteria need to be specific enough for a student at a given level to understand and use, without being so specific that the performance is limited to following directions

Guidelines For Criteria

    Are they more specific than the outcome?

Are they something you could observe?

Would they fit various situations?

Together would they give you a picture of the outcome?

Criteria

   Articulate specific criteria for your assessment by which you would judge student performances How will you establish standards that the student must meet?

Students will use the same criteria to guide their work and for self assessment

Questions that guide decisions about the assignment/assessment

  Does the situation bring the student as close as possible to a situation within which she will use the outcomes outside of the classroom?

Will it give the student an opportunity to demonstrate sufficient indicators of the ability?

A course example

Integrated Literacy Curriculum

Institutional Outcomes Analysis Levels 3 & 4

Uses disciplinary concepts and frameworks with growing understanding 3. Perceives and makes relationships 4. Analyzes structure and organization Based on analysis of student work, draw connections between criteria and performance.

Institutional Outcomes Problem Solving Levels 3 & 4

Takes thoughtful responsibility for process and proposed solutions to problems 3. Performs all phases or steps within a disciplinary problem solving process, including evaluation and real or simulated implementation 4. Independently analyzes, selects, uses, and evaluates various approaches to develop solutions Identifies and designs instruction appropriate to students’ stages of development

Education Major Outcomes

 Diagnosis: Relating observed behaviors to relevant frameworks in order to determine and implement learning prescriptions  Conceptualization: Integrating content knowledge with educational frameworks and a broadly-based understanding of the liberal arts in order to plan and implement instruction

Course outcomes include:  Analyzes and applies learning theory in designing and implementing literacy instruction  Assesses literacy development of intermediate students and prescribes appropriate teaching strategies

Assessment

Students develop and use a rubric to assess actual intermediate student writing samples. Based on their analysis, they plan appropriate teaching strategies and collaborate in formulating a teaching plan.

Criteria, Based on Education Outcomes

 Develops and utilizes a rubric, based on developmental stages and literacy patterns, to determine learner’s strengths and needs in writing  Prescribes effective strategies to meet needs of grades 4-6 learner  Integrates understanding of balanced writing approaches when planning instruction  Integrates literacy theory and characteristics of fiction and non-fiction genres in planning literacy and content lessons

A course example

Understanding the Young Child

Institutional Outcomes Problem Solving Levels 3 & 4

Takes thoughtful responsibility for process and proposed solutions to problems 3.

4.

Performs all phases or steps within a disciplinary problem solving process, including evaluation and real or simulated implementation Independently analyzes, selects, uses, and evaluates various approaches to develop solutions Plans lessons based on developmental frameworks and observations of children’s behaviors Integrates theories of development and teaching, as appropriate for young children Implements all three aspects of developmentally appropriate practices

Education Major Outcomes

 Conceptualization-integrate knowledge of a discipline with educational frameworks in the context of a broadly-based understanding of the liberal arts  Diagnosis-relate observed behaviors to relevant frameworks in order to determine and implement plans that will meet students’ needs and lead them to the next level of development

Course Outcomes Include:  Develop the ability to observe and explain the developmental characteristics of the young child from a variety of theoretical frameworks  Develop the ability to articulate the effectiveness of those frameworks as a means of analyzing developmental maturity in young children

 Demonstrate developmental understanding of young children by making concrete connections to classroom practice and evaluating those decisions based on developmental frameworks (Developed and assessed through interviews with early childhood professionals, research presentation, and intentional teaching experience.)

Assessment

Students are assigned a three, four or five year old child to follow over the semester. The child will be enrolled at the Alverno College Child Care Center where the observations will take place. The purpose of these observations is to develop your ability to carefully watch, record, and interpret what children do. This ability is critical to effective teaching. As we discuss various areas of development, you will reflect on the information from your observations. Each observation will also offer an opportunity to learn or practice a new skill useful in documenting development.

Criteria, Based on Education Outcomes

  Analysis of decisions that went into preparing for the observation are based on developmental frameworks  Record of observation is sufficiently objective and detailed to portray the child’s activities (includes language, interactions with peers, adults, and classroom materials, physical movement, play behaviors) Evidence is provided to accurately support conclusions and reflections  Observations are analyzed in relation to frameworks introduced through class readings and discussions

Learning-Focused Course Planning

     How will course content offer opportunities for students to develop these learning outcomes?

What assignments will be meaningful and contribute to the development of the learning outcomes?

What type of assessment stimulus or mode of performance best elicits this learning outcome?

What criteria are needed?

What kinds of feedback will I give based on criteria?

Connecting Student Learning Outcomes to Teaching, Assessment, Curriculum

• Identify Pertinent Institutional Outcomes • Create Program Outcomes • Create and Break Open Course Outcomes • Plan Course Including Student Practice • Design Performance Prompt • Develop Criteria to Judge Student Performance • Develop Self Assessment Prompt • Plan for Feedback • Design Program Evaluation

Implications for Improvement

How might you analyze students’ performances and your feedback to improve the assessment? Your teaching? Your program? University curriculum? To review your outcomes?

Curriculum-Embedded Assessment

Assessment integral to ongoing instruction that also supports assessment at the level of the program and institution  Fosters and evaluates each individual student’s learning in terms of outcomes  Fits with existing roles, responsibilities, and expertise of faculty in carrying out curriculum evaluation and development  Allows faculty to collectively review patterns of student performance as they occur in the curriculum

Curriculum-Embedded Assessment

  Gives students information about: • • How well they are meeting course and program outcomes How they can improve Gives faculty information about: • Previous courses where outcomes were taught and assessed • The current course • The overall program

Faculty Roles

  Collectively review how students perform on assessments that are integral to teaching and learning in terms of outcomes Focus on key assessments in a program that illustrate development of outcomes

FOR LEARNER INSTITUTIONAL OUTCOMES (core abilities) Institutional Criteria PROGRAM OUTCOMES COURSE OUTCOMES OR COMPETENCIES ASSESSMENT OUTCOMES Program or Major Criteria Course Criteria Assessment Criteria Behaviors as Evidence MOST GENERIC CONTINUUM OF INCREASING SPECIFICATION OF ABILITY IN CONTEXT MOST SPECIFIC