Transcript AACSB Assurance of Learning Standards

McCoy College of Business
Administration (MBA Program)






Why Assessment?
We give grades don’t we?
What is an Assessment Program?
What can we do?
How do we do it?
Special thanks to Dr. Doug Eder and Dr. Kathryn Martell for much of the content
in this presentation.
The First and Only Goal:
To teach for
long-term
retention of
information
and
application to
new situations
--after Halpern & Hakel
Deep (expert) learning is a
curriculum function.
It is not a course function.
Universities
are systems
Direct vs. Indirect Assessment


Direct assessment acquires evidence
about student learning and the learning
environment: Exams, projects, logs,
portfolios, observations....
Indirect assessment acquires evidence
about how students feel about learning
and their learning environment:
Surveys, questionnaires, interviews,
focus groups, reflective essays....
Whatever Assessment Is...
...It’s Multiple Measures
Over Time.
Why Assessment Won’t Go Away
•
•
•
•
Limits on money
Shifts in public view of Higher Education
State laws and education standards
Accountability to accrediting agencies to
maintain operating credentials
*
*
*
*
*
Result: We must demonstrate quality of outcome
and return on investment.
Through assessment educators meet
responsibilities to themselves, to students, and
to the public.
What is “Program-Level” assessment
as specified by AACSB?

The college or school must specify program-level
learning goals for each separate degree program.
Generally, such goals are anticipated for each degree,
not for separate majors or concentrations within a
degree. For example, the McCoy College offers a
Bachelor of Business Administration (BBA) degree
with defined majors in accounting, CIS, economics,
finance, management, and marketing. A set of
program-level learning goals for the BBA degree must
be provided and assessed; goals for each major will
not be required for AACSB program review
accreditation. The MBA program has program-level
learning goals reflecting graduate-level learning
outcomes.
Strongest Links:
The Scholarship of Teaching






Clear Goals
Adequate Preparation
Appropriate Methods
Significant Results
Effective Presentation
Reflective Critique
Assessment vs. Evaluation


Assessment focuses on the
student and the learning
environment.
Evaluation focuses on the
professor and the teaching
performance.
We Give Grades, Don’t We?
Aren’t grades
(by themselves)
enough?
NO!
When used alone, grades are no
longer regarded as adequate
indicators of student learning.


Professor autonomy: Grades in one
course or section may be recorded
using a different standard than
grades in another.
General accountability: Much of the
current public annoyance with higher
education comes from a lack of skill
and content mastery by students who
have received good grades.
Grades may reflect many things besides
student mastery of course objectives:
Verbal ability
Participation
Cooperation
Extra credit
Attendance
Effort
Criterion Performance vs. Value Added
Good Assessment Asks
Good Questions




Do we have a curriculum or a set of common courses?
Which of these do our students experience?
What do we want our students “to be” or “to have” when
they have completed the program? What are our
curricular goals?
What do our students “do” to demonstrate
accomplishment of curricular goals (i.e. course
objectives)? How much and what kinds of writing?
What kinds of math? Oral Presentations? Analysis and
Reports? Are these relevant to our curricular goals?
How do we demonstrate to ourselves that students are
achieving or have achieved curricular goals?
Graduate Common Core
Courses

1.
2.
3.
4.

5.




6.
7.

8.

MGT 5314 Organizational Behavior and Theory
ECO 5316 Managerial Economics
MKT 5321 Marketing Management
QMST 5334 Advanced Statistical Methods for
Business
ACC 5361 Financial and Managerial Reporting
and Analysis
FIN 5387 Managerial Finance
CIS 5354 Decision Support Models in
Management
MGT. 5313 Administrative Policy
GRADUATE LEARNING GOALS (MBA)
Upon completion of the Master of Business Administration (MBA), a
student will be able to:






1. Integrate knowledge of fundamental business disciplines to
effectively manage domestic and global organizations in a
dynamic environment.
2. Integrate appropriate information technologies for
managing business data for decision making, enhancing
productivity, and communicating with others.
3. Evaluate analytical skills and critical thinking processes
used in business decision making.
4. Evaluate the issues associated with ethical leadership and
conducting business in an ethical, legal, and socially
responsible manner.
5. Demonstrate the ability to communicate effectively, both
orally and in writing, in new and unfamiliar circumstances.
6. Apply the skills needed to effectively lead and contribute to
dynamic workgroups.
An Assessment Question:
How Do You Know...
...that students walk out your door
looking like you want them to?
What behaviors have they
exhibited or products have they
produced? What are the indicators
for your goals?
Goals and Objectives
for Students


Goals express what we want our
students to be.
Objectives describe what we want
our students to do.
.....
• Objectives are indicators of goals.
Graduate Faculty Teaching
Common Core Courses Should:



Talk to each other (if multiple
faculty teach the course) . . .
Agree on specific objectives for the
course . . .
Make sure course objectives are
relevant to Program-Level goals . .
To be assessment friendly,
objectives should:



Focus on students
Make the learning goals visible (serve as
indicators)
Describe behaviors or products (doing,
making) that can be captured by
assignments.
Bloom’s Taxonomy






LEVEL
SOME COGNITIVE BEHAVIORS
Evaluation
Appraisal of an Analysis
or Synthesis
Assembly of Application
Disassembly of Application
Use of Understanding
Management of Knowledge
Memorization of facts, language,
concepts, principles, theories
Synthesis
Analysis
Application
Understanding
Knowledge
The Next Step . . .

Graduate Faculty Should Agree on
How Best to Measure Student
Performance on each Objective
Contributing to Program-Level
Goals.
Some Assessment Ways and Means












Assessment days or centers
Case studies
Classroom assessments
Completion and retention
studies
Content analyses
Debates
Direct observations
Focus groups
Graduate success
Internships and service
learning
Interviews (including
videotapes)
Exams for certification and
licensure









Matrices
Performances
Portfolios of several kinds
Projects (Primary Trait
Analysis)
Questionnaires and
surveys (Direct and
telephone; employer,
alumni, and student
attitude and satisfaction)
Reflective essays
Study and activity logs
Tests
(Locally-developed and
standardized)
Transcript analyses
Another Step . . .




Depending on the method (ways and means)
used to assess a course objective, develop a
rubric or a method of measurement.
Evaluate student performance on the specific
objective on three levels: Exceeds
Expectation; Meets Expectation; Does Not
Meet Expectation.
Report your findings.
Discuss the result(s), determine any actions
to take (if necessary).
Example Rubric for MGT. 4335
Base:
100%-90%
All sections were thoroughly and insightfully completed. All narratives, profiles
and relevant sections were factually accurate and documented. Lots of facts
and data to support your claims. You clearly demonstrate a superior level of
analytical reasoning and critical thinking based on your analysis. The paper was
well-written and obviously put together with care (page breaks, layout, etc are
appropriately placed, the paper is grammatically correct, and the level of writing
is appropriate to an undergraduate senior in the McCoy College).
89%-80%
All or most sections were adequately addressed and factually accurate. All
sections included and reasonably prepared. Most sections include data to
support assertions and are appropriately documented. A large amount of
analytical reasoning and critical thinking was evident. Writing and layout may
contain a few errors, but most of the document is relatively clean.
79%-70%
Some (few) sections were rather shallow and superficial, lacking any significant
insight, and/or a few are missing in the document. Some to little data was
provided to support claims, and documentation is either incorrect or missing.
Some of the sections may contain factually inaccurate data and the layout may
have been sloppy and/or done without care. Writing may be fair to poor
containing several errors and style is poor or inappropriate for a formal
document.
69% or less
The paper was extremely poor, and/or some sections were missing. The
sections were VERY superficial and little documentation was provided, and/or
the sections deviate from the required format. Errors/inconsistencies are
evident. Writing contains several to many errors and grammar is poor. In short,
it appeared that the paper was written ‘at the last minute’ and didn’t reflect much
thought.
Responsibilities of Core Course
Coordinators





1. completing the Course Alignment Matrix which
specifies which program-level learning goals are covered
in a specific core course,
2. developing course learning objectives that lead to
fulfillment of program-level learning goals,
3. collecting data from faculty teaching course sections
to support or measure student learning of program-level
learning goals (including a representative sample of
measurement devices/techniques/assignments to be
maintained in the office of the core course coordinator),
4. reporting collective results on program-level learning
goals to the McCoy College Assessment Committee, and
5. discussing and acting on specific recommendations
made by the college assessment and curriculum
committees.
McCoy College of Business Administration
Graduate (MBA) Learning Goals Matrix
Course:
Core Course Coordinator:
Faculty member(s):
Date Form Completed:
Goal
1. Integrate knowledge of fundamental
business disciplines to effectively
manage domestic and global
organizations in a dynamic environment.
1a. Integrate knowledge of
fundamental business disciplines to
effectively manage domestic
organizations in a dynamic
environment.
1b. Integrate knowledge of
fundamental business disciplines to
effectively manage global
organizations in a dynamic
environment
Covered
Assessed
Method of
assessment
McCoy College Graduate (MBA) Program Goals Coverage
Fall 2005
Goals
1. Integrate knowledge of fundamental business
disciplines to effectively manage domestic and
global organizations in a dynamic environment.
1a. Integrate knowledge of fundamental business
disciplines to effectively manage domestic
organizations in a dynamic environment.
1b. Integrate knowledge of fundamental
business disciplines to effectively manage global
organizations in a dynamic environment.
2. Integrate appropriate information technologies for
managing business data for decision-making,
enhancing productivity, and communicating with
others.
2a. Integrate appropriate information technologies
for managing business data for decision-making.
MGT 5314
Cov.
Assd.
ECO 5316
Cov.
Assd.
MKT 5321
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
2b. Integrate appropriate information technologies
for managing business data for enhancing
productivity.
2c. Integrate appropriate information technologies
for managing business data for communicating
with others.
3. Evaluate analytical skills and critical thinking
processes used in business decision-making.
3a. Evaluate analytical skills used in business
Cov.
decision-making.
3b. Evaluate critical thinking processes used in Cov.
business decision-making.
4. Evaluate the issues associated with ethical
leadership and conducting business in an ethical,
legal, and socially responsible manner.
4a.Evaluate the issues associated with ethical
Cov.
leadership.
4b. Evaluate the issues associated with
Cov.
conducting business in an ethical, legal, and
socially responsible manner.
5. Demonstrate the ability to communicate
effectively, both orally and in writing, in new and
unfamiliar circumstances.
5a. Demonstrate the ability to effectively
Cov.
communicate orally in new and unfamiliar
circumstances.
5b. Demonstrate the ability to effectively
Cov.
communicate in writing in new and unfamiliar
circumstances.
6. Apply the skills needed to effectively lead and
contribute to dymanic workgroups.
6a. Apply the skills needed to effectively lead
Cov.
dynamic workgroups.
6b. Apply the skills needed to effectively
contribute to dynamic workgroups.
Cov.
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
Cov.
Cov.
QMST 5334
Cov.
Assd.
ACC 5361
Cov.
Assd.
FIN 5387
Cov.
Assd.
CIS 5354
Cov.
Assd.
MGT 5313
Cov.
Assd.
Cov.
Cov.
Cov.
Assd
Cov.
Assd
Cov.
Assd.
Assd.
Assd.
Cov.
Assd.
TOTALS:
Cov.
Assd.
7
6
3
2
4
4
17
12
Cov.
Assd.
Cov.
Assd.
Cov.
7
5
Cov.
Assd.
Cov.
Assd.
Cov.
5
3
Cov.
Assd.
Cov.
Assd.
Cov.
5
4
16
14
Assd.
Cov.
Assd.
Cov.
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
8
7
Assd.
Cov.
Assd.
Cov.
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
8
7
14
7
Assd.
Assd.
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
Assd.
Cov.
Cov.
Assd.
Cov.
Cov.
Assd.
Cov.
6
3
Cov.
Cov.
Assd.
Cov.
Cov.
Assd.
Cov.
8
4
13
10
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
Assd.
Cov.
Assd.
Cov.
Assd.
Cov.
Assd.
Cov.
Cov.
Assd.
6
4
Cov.
Cov.
Assd.
7
6
7
7
Cov.
Assd.
Cov.
Assd.
3
3
Cov.
Assd.
Cov.
Assd.
4
4
What documentation must be
retained by the Core Course
Coordinators?

Core course coordinators should retain copies of
instruments used for direct assessment of student
learning, such as assignments, written products, test
questions, rubrics, scoring grids, etc. The course
coordinators also must retain sample student
products relating to the instruments used. Aggregate
results of direct assessments for various programlevel learning goals gathered and/or submitted to the
assessment committee should be retained by the
course coordinator by semester or year for review.
Minutes of core course faculty meetings should be
retained with special emphasis given to items
concerning improvements in instruments used to
assess student learning and acknowledgement and
review of data.
Must all students be
assessed?

For the purpose of meeting AACSB-International
standards relating to Assurance of Learning (16, 18
and 20), sampling may be utilized as long as it is
representative. Faculty teaching college core courses
for various programs, as scholars in their respective
fields and under the leadership of the Core Course
Coordinator, have the responsibility for determining
appropriate representative sample sizes. Graduate
core course coordinators may use census data from
graduate core courses for assessment purposes.
Should college core course faculty and
coordinators gather data and submit
results every year?

Program-level learning goals for undergraduate (BBA)
and graduate (MBA) programs are reviewed on a
scheduled rotational basis. At least two program
goals are reviewed each year, and college core
courses covering that particular program goal are
required to submit the results of course-embedded
assessment to the assessment committee during the
scheduled review year. Since the McCoy College
views assessment as a continuous process, core
course coordinators and core course faculty are
encouraged, but not required, to gather data each
semester for the reporting of assessment results.
Example Undergraduate Goal Rotation










Program-Level Goal Rotation Sequence:
Academic Year 2005-2006:
Goal 1 – Conceptualize a complex issue into a coherent,
persuasive written or oral statement.
Goal 5 – Understand the importance of group dynamics in
achieving organizational goals and use the skills needed for
effective teamwork.
Academic Year 2006-2007:
Goal 2 – Use critical thinking skills to evaluate information, solve
problems, and make sound decisions.
Goal 3 – Use information technology skills in decision-making.
Academic Year 2007-2008:
Goal 4 – Apply general concepts of ethical behavior in dealing
with stakeholders.
Goal 6 – Understand the importance of culture and diversity.
We have never been
asked...
...to assess
everything all at
once.
Who sets performance standards? How should
Core Course Coordinators report student
performance on various college-level learning
goals?

For each program-level learning goal assessed in a
college core course, the faculty will determine their
minimum expectation or standard. For reporting data
on any specific program-level learning goal, core course
coordinators should aggregate data and report the
results by three levels: Exceeds Expectations; Meets
Expectations; or Does Not Meet Expectations. In order
to preserve the integrity of the assessment process,
core course coordinators should release aggregate data
by course (all sections) and not individual results by the
professor teaching the core course. Program
Assessment in the McCoy College is not the evaluation
of faculty performance in the classroom; it concerns the
aggregate level of student learning of various programlevel goals.
Example Report on Program Goal
Enclosed in this folder are the results of Goal 1a & 1b assessment for the Fall 2005
semester. I have included a packet of evaluation sheets; each student received a sheet
based on their performance on the individual case analysis. For McCoy College Learning
Goal 1a, the specific items are item 8 – Bibliography & Appendices, and item 9 –
Writing/Format. Greater weight was given to item 9 although item 8 was included in the
assessment. The bottom of each sheet indicates whether the student exceeded
expectations, met expectations, or did not meet expectations. A copy of the grading
rubric is included in this packet.
For Goal 1b, a single sheet included shows how each individual performed on their oral
communications segment of a case analysis. Last names have been removed although
first names remain for my control purposes.
Fall 2005 Goal 1a Results:
Exceeds Expectations:
5 students or 19.23%
Meets Expectations:
18 students or 69.23%
Does not meet Expectations: 3 students or 11.54%
Fall 2005 Goal 1b Results:
Exceeds Expectations:
5 students or 19.23%
Meets Expectations:
19 students or 73.08%
Does not meet Expectations: 2 students or 7.69%
An Important Lesson
from the Farm
A pig doesn’t get
any fatter
merely by
weighing it.
How many students must meet or
exceed the performance standard on
a college-level program goal?
There is no prescribed percentage of students that must
meet the standard articulated by faculty. AACSB and other
constituent groups will look at whether and how these
data are used for continuous improvement purposes.
Thus, a poor showing of student mastery of a given
program-level learning goal would only be a concern if the
curriculum was not subsequently modified to improve
student skills/performance of that learning goal. A second
emphasis in the Assurance of Learning Standards is that
program-level learning goals are used to communicate
competencies of graduates to students, employers and
other constituents. Thus, faculty have the responsibility to
develop and modify curricula to insure that program-level
goals are achieved by most graduates and not just a
portion.
What Assessment Asks of You!





One day of initial development time to determine course
objectives and agreeing on a grading rubric for specific course
objectives.
Two to three hours per semester (depending on the goal
rotation) to extract student performance on course objectives
leading to the fulfillment of program-level goals.
Writing a short report (1 page) concerning student
performance.
Maintaining examples of student work, grading rubrics or other
examples demonstrating student achievement of course
objectives.
Talking to other faculty about continuously improving classroom
processes and products to improve student learning of program
level goals over time.
Closing Thought
The enemy of the good
is the perfect.