Transcript Assurance of Learning: Why, what, who, when and how?

Assurance of Learning:
Why, what, who, when and how?
Marta Colón de Toro, SPHR
Assessment Coordinator
College of Business Administration
UPR-Mayagüez
Today’s Learning Goals
After completing this workshop you should be
able to:
1.
2.
3.
4.
5.
6.
Explain the importance and purpose of assessing student
learning outcomes.
Recognize AACSB’s standards on assessment.
Describe the process of assessment and the challenges it
represents.
Develop a plan to conduct classroom assessment.
Identify and apply appropriate instruments to directly assess
student learning in the classroom.
Create and maintain a course portfolio to document your
classroom assessment.
2
A Transformation Process
Input
Incoming
Student
• Demographics
• Academics
• Experience
• Talents
• Attitudes /Values
Teaching-Learning
Curriculum
Seminars and Competitions
Internship/Coop
Student Associations
Extracurricular Experiences
Output
Completing
Student
• Knowledge
•Skills
•Abilities
•Experience
•Attitudes and Values
3
Effective curricula is…
…pertinent
…coherent
…strategic
…dynamic
…assessable
…intentionally designed!
4
Objective #1

Explain the importance and purpose of
assessing student learning outcomes
5
Objective #1
Have you ever asked yourself…?
Are my students learning?
How much are they learning?
How well are they learning?
What affects their learning?
6
Objective #1
Assessment
…is a way of describing student
learning to an identified audience
for clearly- articulated reasons.
…produces information useful for
communication and decisionmaking.
7
Objective #1
Purpose of Assessment
Assessment
Intended Outcomes
Actual Outcomes
1. Act ethically
1.
2. Quantitative Analysis
2.
3. Apply models
3.
4. Decision making skills
4.
5. Communication skills
5.
Reveals the
Gap
8
Objective #1
For example…UPR-Utuado
Misión
El Programa de Contabilidad de la Universidad de Puerto
Rico en Utuado, reconoce que tiene ante si un reto y
compromiso con la comunidad altamente industrializada y
comercializada de nuestros tiempos. Su objetivo es
capacitar al estudiantado con las herramientas necesarias:
destrezas, conocimientos, aptitudes y actitudes que
respondan a la demanda y expectativas del mundo laboral
y en los principios técnicos necesarios en la Contabilidad
General, en el análisis de información de transacciones
financieras en forma sistemática y organizada.
La finalidad de esta tecnología es desarrollar el potencial
individual de estudiantes para que puedan competir en
una economía global, ampliar sus oportunidades de
empleo y proveer a la comunidad de personal técnico
capacitado.
9
Objective #1
Another example…Arecibo
MISIÓN
El programa de Administración de Empresas tiene como
misión lograr la formación humana, ética, intelectual y
profesional del estudiante para que provea un servicio
de excelencia en la administración de las empresas
privadas y públicas, y en la comunidad. Además, el
estudiante desarrollará las competencias necesarias
para desempeñar posiciones de liderato y enfrentar retos
con la disciplina y el grado de flexibilidad que exige el
ambiente cambiante empresarial, y así contribuye a su
éxito profesional.
10
Objective #1
Another example…Arecibo
Para lograr su misión, el Departamento tiene como metas:
 Familiarizar al estudiante con los conceptos, teorías, y técnicas
administrativas y fiscales de las empresas privadas y públicas.
 Capacitar al estudiante para recopilar datos y analizar y
presentar información de su área para que se pueda utilizar en el
proceso decisional.
 Fomentar en el estudiante la comprensión de su responsabilidad
como ciudadano en el desarrollo de la organización, la economía,
la sociedad y la cultura del país.
 Capacitar al estudiante con las destrezas intelectuales, técnicas y
profesionales que le permitan integrarse exitosamente a la
constante innovación en la empresa y la sociedad.
 Desarrollar en el estudiante la agudeza de percepción para
reconocer los cambios del ambiente externo e integrar las
competencias que faciliten identificar mecanismos
administrativos que permitan la adaptación exitosa de la
empresa.
11
Objective #1
Assess Learning to:
1.
2.
3.
Provide feedback and guidance to
individual students.
Assist the school and faculty members
to improve courses and programs.
Assure external constituents that the
institution meets its academic goals.
AACSB
12
Objective #1
Levels of Assessment
Institutional Learning Goals
Program Learning Goals
Course Goals
Lesson
Objectives
13
Objective #1
When to Assess

Entry




Midpoint


Measure progress / added value
Exit


Demographics
Academic achievement
Previous Business KSAOs
Measure attainment of expected learning outcomes
Follow-up



Further education
Workforce
Personal goals
14
Objective #1
Effective assessment…

Is goal directed


Uses multiple measures


Tests, essays, projects, transcripts, surveys
Show student development overtime


Ethics, writing, critical thinking, teamwork
Entry, midpoint, exit, follow-up
Uses multiple types of information

Demographics, behavioral, perceptual
15
Objective #1
Challenges of Assessment
Motivating faculty, staff, students,
administration
 Needs realistic goal setting
 Data management
 Assuring right purpose
 Requires taking action
 Could be costly

16
Objective #2

Recognize AACSB’s standards on
assessment
17
AACSB
Objective #2
Assurance of Learning Standards

AACSB only requires assessment at the
program level.

Other levels of assessment bring support

Assessments using samples are allowed.
 Assessment must use direct measures of
achievement




Course-embedded exercises
Course-embedded examinations
Stand-alone examinations
Indirect measures may be used to supplement
direct measures of achievement.
18
AACSB’s
Objective #2
Learning Expectations at the Bachelor’s Level




Educate students in a broad range of knowledge and
skills as a basis for careers in business.
Build on the students’ pre-collegiate educations to
prepare them to enter and sustain careers in the
business world and contribute positively in the larger
society.
Provide knowledge and skills for successful performance
in a complex environment requiring intellectual ability to
organize work, make and communicate sound decisions,
and react successfully to unanticipated events.
Develop learning abilities suitable to continue higherlevel intellectual development.
19
Objective #2
Standard 15

The school uses well documented, systematic
processes to develop, monitor, evaluate, and
revise the substance and delivery of the curricula
of degree programs and to assess the impact of
the curricula on learning.
 Curriculum management includes inputs from all
appropriate constituencies which may include
faculty, staff, administrators, students, faculty from
non-business disciplines, alumni, and the business
community served by the school.
20
Objective #2
Standard 16

Adapting expectations to the school’s
mission and cultural circumstances, the
school specifies learning goals and
demonstrates achievement of learning
goals for key general, managementspecific, and/or appropriate disciplinespecific knowledge and skills that its
students achieve in each
undergraduate degree program.
21
Objective #2
Standard 17

The bachelor’s or undergraduate level
degree programs must provide sufficient
time, content coverage, student effort,
and student-faculty interaction to assure
that the learning goals are accomplished.
22
Objective #2
Program-level learning goals

General, not specific.







general knowledge and skills
management-specific learning goals
State the broad educational expectations for each degree
program.
Specify the intellectual and behavioral competencies a
program is intended to instill.
Clarify how we intend for graduates to be different as a
result of their completion of the program.
Need to be defined operationally in order to be
assessable.
Normally, four to ten learning goals will be specified for
each degree program.
23
Objective #3
•
Describe the process of assessment.
24
Objective #3
The Assessment Process
External Reality
Vision/Mission
Set Goals and
Design Curriculum
Improve
Continuously
Enable Learning
Assess
Learning
25
Objective #3
The Assessment Process
1.
Establish learning goals



2.
Identify/design enabling experiences

3.
4.
5.
6.
Consult appropriate constituencies
Outcomes-based
Specify level of learning
Courses and Extracurricular activities
Select instruments and indicators of goal
attainment
Assess
Interpret and report results
Identify strategies for change
26
Objective #3
For example…
1.
2.
3.
4.
5.
6.
7.
Establish Learning goals: :
“The student will be able to discuss the importance of assessing student
learning.”
Identify or design enabling activities:
 Brief presentation on topic

Brainstorming

Q&A
Assurance of Learning Standards
Select:

Indicator: “Participant will present 2 arguments pro/con of assessment w/o
consulting notes.”

Instrument: Focused Listing
Assess:

List 2 arguments in favor and 2 arguments against doing assessment.
Measure, interpret and report results

_________________________
Introduce change
Revise plan
27
Objective #4
•
Develop a plan to conduct classroom
assessment.
28
Objective #4
Where to start?
PLAN!
Step#1 - Revisit your course learning goals
(CLGs). (Also known as: learning outcomes, course objectives)
A. Definition
“…types of executions that the students can perform after
being taught in a way that they show that they have
learned what was expected.” Norman Gronlund
B. Purpose
a. Express expectations of what is to be learned in a
course
b. Provide a context for what will be learned
29
Objective #4
Learning Outcomes…
C. Should answer five questions:
1. Are they aligned with the Program Learning
Goals?
2. What type of learning do they address?
3. Do they express the appropriate learning level?
 Express the “ultimate goal”
4. Are they outcomes-based? Use action verbs!!

List what will the student will be able to do as a result of
learning.
5. Are they student-centered?
6. Do they address one element of learning?
30
Ejercicio
Al finalizar el curso, los(as) estudiantes
podrán:
Manejar la calculadora para calcular datos
específicos como la media y la
desviación …

Solucionar problemas estadísticos
utilizando la calculadora científica.
31
Objective #4
Hierarchy of Learning Goals
Institutional Goals
“…Apply scientific inquiry methods.”
College Goals
“…Think logically and critically.”
Program Goals
“…basic and applied research.”
Course Goals
“…apply data collection techniques.”
Session Objectives
“…list steps in the scientific research process.”
32
Objective #4
How to align goals
Complete the following for each course you teach: ADMI 3000
INSTITUTIONAL
LEARNING
GOALS
PROGRAM
LEARNING GOALS
(Those that apply)
COURSE
LEARNING
GOALS
Communication
Skills
Interpersonal
Skills
Required
team project
…
Information
Technology Skills
Assignments using
Peachtree
…
Ethical
Behavior
…
…
Research
skills
…
33
Objective #4
Program Goal Matrix
Goal #2 - “The students will use the appropriate
software packages and hardware as a working tool
in the daily operations of a business”.
Course/Experience
Level of Learning
Application of Learning
ADMI 3007 - Intro. to Data
Processing
Application
Business related problems
ESTA 3001- 3002 - Business
Statistics
Application
Business related problems
CONT 3005-3006 - Elementary
Accounting
Application
Accounting problems
Internship/COOP
Application
Job environment
Computer-assisted courses
Application
Business problems
Assigned projects in several
courses.
Application
34
Objective #4
Types of Learning

Cognitive: mental skills (Knowledge)


Affective: growth in feelings or
emotional areas (Attitude)


“…enumerate three major EEO laws…”
“…will value diversity…”
Psychomotor: manual or physical skills
(Skills)

“… correctly use the keyboard…”
35
Objective #4
You may want to use:
Compare &
contrast,
critique, justify

Evaluate
Adapt, combine,
compare, contrast,
design, generate

Bloom’s
Taxonomy
Synthesize
(Cognitive Domain)
Correlate, diagram, distinguish,
outline, infer

Analyze
Determine, develop, compute, utilize,
conduct

Apply
Classify, explain, discuss, give
examples, summarize

Comprehend
Know

Define, describe, list, reproduce, enumerate
36
Objective #4
Dealing with “…understand”…

Always ask yourself…
How do you know the student understands?
“When he/she:

Gives examples of …
 Determines the “correct” method to use…
 Discusses pros and cons…
 Identifies elements in a given case…etc.”

37
Objective #4
Using Measurable Action Verbs
analyze
compute
compare
appreciate
demonstrate direct
discuss
display
identify
infer
justify
list
grasp
report
state
synthesize
classify
contrast
derive
evaluate
integrate
understand
respond
name
collaborate
define
designate
know
interpret
organize
solicit
explain
38
Objective #4
Course goals help…

The Professor



Design relevant
classroom material
Design relevant
assignments
Design
tests/projects

The Student




Clarify personal
goals
Measure success
Reduce anxiety
Improve studying
effectiveness
39
Objective #4
Right focus! The student’s learning.
Focused on Teaching

Example:
 “Expose the student to
recent research in
organizational behavior”.
Focuses on the
professor’s effort,
intent
or
Focused on Learning

Focuses on the
student’s achievement
Example:
 “After completing this
course the student should
be able to evaluate recent
research in organizational
behavior”.
40
Objective #4
Where to start? (cont.)
Step #2 - Identify/design activities to enable
students to achieve each learning goal:
Learning Goals
1.
2.
3.
4.
5.
Teaching Methodologies






Lecture
Case analysis and discussion
Oral presentations
Essays/Reports
Team Projects
Experiential Learning
Extracurricular Activities


Seminars/forums
Field trips
41
Objective #4
Complete the course matrix
PROGRAM
LEARNING GOALS
(Indicate only those
that apply)
COURSE
LEARNING
GOALS
Interpersonal Skills
“…able to work in
teams”
Information
Technology Skills
“…complete
accounting cycle
using Peachtree”
Ethical Behavior
“…identify ethical
dilemmas in
marketing”
Research skills
“conduct a
research study in
accounting
theory”
ACADEMIC
ACTIVITIES
(Methodology)
ASSESSMENT
INSTRUMENTS
(Rubrics, pre/post tests…)
Required team
project
Assignments
using
Peachtree
42
Objective #5

Identify and apply appropriate
instruments to directly assess student
learning at nd classroom levels.
43
Objective #5
Where to start? (cont.)
Step# 3 - Select or design classroom
assessment instruments (CATs) and
indicators
“Classroom assessment is a simple
method faculty can use to collect
feedback on how well their students are
learning what they are being taught”

Angelo, T.A (1991)
44
Objective #5
Classroom Assessment Assumptions:

Quality of student learning is directly, although
not exclusively, related to the quality of teaching.
 Learning is goal oriented.
 Students need to receive appropriate and
focused feedback early and often
 Classroom Assessment provides systematic
inquiry and intellectual challenge.
 Classroom Assessment does not require
specialized training
 With Classroom Assessment efforts, faculty (and
students) enhance learning and personal
satisfaction.
45
Objective #5
Classroom Assessment Techniques
Learner-Centered
 Teacher-Directed
 Mutually Beneficial
 Formative
 Context-Specific
 Ongoing
 Rooted in Good Teaching Practice

46
Objective #5
Three step process
STEP 3
RESPOND
STEP 1
PLAN
STEP 2
IMPLEMENT
47
Objective #5
Classroom Assessment Techniques
Direct Measures










1 Minute paper
Muddiest Point
One-sentence summary
Direct paraphrasing
Application cards
Empty outlines
Pro-con grids
Class journals/Portfolios
What’s the principle?
Rubrics
Indirect Measures
Student evaluation of courses
 Alumni Feedback
 Internship/Coop

48
Objective #5
CATs
MINUTE
 Ask
PAPER
them…
The
most important thing
learned
What remains unanswered
 Used
in lecture/discussion
courses
49
Objective #5
CATs
MUDDIEST
 Ask
POINT
them…
Identify
what is least clear to
you or the most confusing point
 Used
for large, lower level
courses
50
Objective #5
CATs
ONE
SENTENCE
SUMMARY
 Ask
them…
Summarize
today’s session.
51
Objective #5
CATs
DIRECTED
PARAPHRASING
 Ask
them
Explain
______ in your own
words.
52
Objective #5
CATs
APPLICATION
 Ask
CARDS
them
Connect
today’s topic with…
Mention two possible
applications of today’s topic.
53
Objective #5
REMEMBER…
 CATs
are suggestions to be
adapted, not models to be
adopted
 Don’t
ask if you don’t really want
to know or are not going to do
anything about it.
54
Objective #5
Examples of Measures of Achievement
Example 1

School A has defined a learning goal in ethical reasoning
for each of its four undergraduate majors. Student achievement
on this goal is relevant to demonstrating satisfaction of Standard
16. The school’s faculty has defined the goal.
Learning Goal
 “Each student can recognize and analyze ethical problems and
choose and defend resolutions for practical situations that occur
in accounting, human resource management, and marketing.”
Demonstration of Achievement
 The school uses course-embedded exercises in three required
introductory-level courses. Faculty in the three disciplines have
developed different methods for instructing and assessing
55
achievement toward this learning goal.
Objective #5
Examples of Measures of Achievement
In Accounting

A two-week module near the end of the introductory
course is devoted to “Ethical standards and fraud in
accounting.”
 A topic outline has been developed by faculty
members to structure an exam on the materials of
this module, and a standard set of expectations has
been created for grading the exam.
 In addition to this exam’s contribution to the course
grade, it provides a pass/fail indication on the
learning goal.
56
Objective #5
Examples of Measures of Achievement
In Human Resource Management

Students must provide four written analyses of
problem situations during the course. On three of
these analyses (on the topics of selection, reward
systems, and job design), students are asked to
respond to ethical issues.
 A standard scoring key on the ethical component
provides evaluation toward the course grade and a
pass/fail indication on the learning goal.
57
Objective #5
Examples of Measures of Achievement
In Marketing

Each student must compose a term paper analyzing
a current national or international marketing
campaign.
 The analysis must include a specified set of
components, and ethical issues that have been
presented in lectures are among the required
components.
 In addition to the overall grade of the paper, each
student receives a pass/fail indicator on the ethics
component.
58
Objective #5
Examples of Measures of Achievement
In addition to reporting course grades

Each instructor of these three courses provides a
checklist of all of those students who
successfully completed the ethics expectation.
 This information is a part of each student’s
record and all three parts of the learning goal
must be achieved before graduation.
 Students who fail the ethics evaluation while
passing the course repeat the evaluation
exercise or ethics module until they are
successful.
59
Step #4 Conduct Assessment
60
Objective #6

Create and maintain a course portfolio to
document your classroom assessment.
61
Step #5 Document results with course
portfolios
“ The portfolio is to teaching what
lists of publications, grants, and
honors are to research and
scholarship.”
Peter Seldin
The Teaching Portfolio (Anker, 1991)
62
Objective #6
Uses of Course Portfolios

Document existing courses in order to:
Constitute an official record of courses syllabi
 Facilitate the assessment of our students’
learning outcomes
 Provide quantitative and qualitative information
for continuous improvement
 Serve as reference for professors that will
teach the course in the future
 Render accountability to stakeholders

63
Objective #6
Proposed Content
I. Course Syllabus (In compliance with Certification #130 – UPR-BT)
II. Educational Goals Addressed in the Course
1. Institutional Learning Goals Matrix
2. Program Learning Goals Matrix
3. Concentration Learning Goals Matrix
III. Classroom Assessment Techniques – For each assessment instrument
used please include the following documentation:
1. Brief description of each technique used
2. Graded Sample Instrument (High, average and low grades)
3. Outcomes
4. Analysis of Outcomes
5. Lessons Learned
6. Recommendations to improve teaching/learning
7. Specific Action(s) Taken or To Be Taken
IV. Evaluation Instruments (Tests, Quizzes, Projects and Special Activities) For each evaluation instrument used include the following documentation:
(Same as above)
V. Recommendations and Proposed Action Plan
64
Objective #6
Documenting CATs
CAT’s

Diagnostic Tests
 Pre-Post Test
 Muddiest Point
 Minute Paper
 Application Cards
 Direct Paraphrasing
 Course Final General
Assessment
Documentation







Brief Description or
Sample copy
Sample of Grades
(H,M,L)
Sample Answers
Analysis of Outcomes
Conclusions
Recommendations
Actions Taken
65
Objective #6
Analysis of Outcomes

Quantitative

Descriptive statistics



Average, Median, Mode, Standard deviation
Question by question analysis
Qualitative

Look for
Misconceptions
 Attitudes/prejudices

66
Objective #6
Conclusions/Recommendations
Summarize lessons learned
 List actions to be taken for improvement
Step #6 - TAKE ACTION!!!

Next time you teach the course read the list
and apply recommendations
Refer pertinent recommendations to
appropriate forum
67
Stages of Grieving





Denial
Anger
Bargaining
Depression
Acceptance
On Death and Dying
by Elisabeth Kubler-Ross
68
Thank You!
[email protected]
69