Transcript AACSB Assurance of Learning Standards

McCoy College of Business
Administration (BBA Assessment)






Why Assessment?
We give grades don’t we?
What is an Assessment Program?
What can we do?
How do we do it?
Special thanks to Dr. Doug Eder and Dr. Kathryn Martell for much of the content
in this presentation.
The First and Only Goal:
To teach for
long-term
retention of
information
and
application to
new situations
--after Halpern & Hakel
Deep (expert) learning is a
curriculum function.
It is not a course function.
Universities
are systems
Direct vs. Indirect Assessment


Direct assessment acquires evidence
about student learning and the learning
environment: Exams, projects, logs,
portfolios, observations....
Indirect assessment acquires evidence
about how students feel about learning
and their learning environment:
Surveys, questionnaires, interviews,
focus groups, reflective essays....
Whatever Assessment Is...
...It’s Multiple Measures
Over Time.
Why Assessment Won’t Go Away
•
•
•
•
Limits on money
Shifts in public view of Higher Education
State laws and education standards
Accountability to accrediting agencies to
maintain operating credentials
*
*
*
*
*
Result: We must demonstrate quality of outcome
and return on investment.
Through assessment educators meet
responsibilities to themselves, to students, and
to the public.
What is “Program-Level” assessment
as specified by AACSB?

The college or school must specify program-level
learning goals for each separate degree program.
Generally, such goals are anticipated for each degree,
not for separate majors or concentrations within a
degree. For example, the McCoy College offers a
Bachelor of Business Administration (BBA) degree
with defined majors in accounting, CIS, economics,
finance, management, and marketing. A set of
program-level learning goals for the BBA degree must
be provided and assessed; goals for each major will
not be required for AACSB program review
accreditation. The MBA program has program-level
learning goals reflecting graduate-level learning
outcomes.
Strongest Links:
The Scholarship of Teaching






Clear Goals
Adequate Preparation
Appropriate Methods
Significant Results
Effective Presentation
Reflective Critique
Assessment vs. Evaluation


Assessment focuses on the
student and the learning
environment.
Evaluation focuses on the
professor and the teaching
performance.
We Give Grades, Don’t We?
Aren’t grades
(by themselves)
enough?
NO!
When used alone, grades are no
longer regarded as adequate
indicators of student learning.


Professor autonomy: Grades in one
course or section may be recorded
using a different standard than
grades in another.
General accountability: Much of the
current public annoyance with higher
education comes from a lack of skill
and content mastery by students who
have received good grades.
Grades may reflect many things besides
student mastery of course objectives:
Verbal ability
Participation
Cooperation
Extra credit
Attendance
Effort
Criterion Performance vs. Value Added
Good Assessment Asks
Good Questions




Do we have a curriculum or a set of common courses?
Which of these do our students experience?
What do we want our students “to be” or “to have” when
they have completed the program? What are our
curricular goals?
What do our students “do” to demonstrate
accomplishment of curricular goals (i.e. course
objectives)? How much and what kinds of writing?
What kinds of math? Oral Presentations? Analysis and
Reports? Are these relevant to our curricular goals?
How do we demonstrate to ourselves that students are
achieving or have achieved curricular goals?
BBA Common Core Courses













1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
CIS 1323 Introduction to Microcomputer Applications. (3)
QMST 2333 Business Statistics (3)
ACC 2361 Introduction to Financial Accounting (3)
ACC 2362 Introduction to Managerial Accounting (3)
ECO 2314 Principles of Microeconomics (3)
ECO 2315 Principles of Macroeconomics (3)
BLAW 2361 Legal Environment of Business (3)
MGT 3353 Business Communication (3)
MGT 3303 Management of Organizations (3)
MKT 3343 Principles of Marketing (3)
CIS 3380 Personal Productivity and Information
Technology (3)
12. FIN 3312 Business Finance (3)
13. MGT 4335 Strategic Management and Business Policy (3)
Bachelor of Business Administration
LEARNING GOALS (BBA)
Upon completion of the BBA, a student will be able to:






1. Conceptualize a complex issue into a coherent,
persuasive written or oral statement.
2. Use critical thinking skills to evaluate information,
solve problems, and make sound decisions.
3. Use information technology skills in decisionmaking.
4. Apply general concepts of ethical behavior in
dealing with stakeholders.
5. Understand the importance of group dynamics in
achieving organizational goals and use the skills
needed for effective teamwork.
6. Understand the importance of culture and
diversity.
An Assessment Question:
How Do You Know...
...that students walk out your door
looking like you want them to?
What behaviors have they
exhibited or products have they
produced? What are the indicators
for your goals?
Goals and Objectives
for Students


Goals express what we want our
students to be.
Objectives describe what we want
our students to do.
.....
• Objectives are indicators of goals.
Faculty Teaching Common
Core Courses Should:



Talk to each other (if multiple
faculty teach the course) . . .
Agree on specific objectives for the
course . . .
Make sure course objectives are
relevant to Program-Level goals . .
To be assessment friendly,
objectives should:



Focus on students
Make the learning goals visible (serve as
indicators)
Describe behaviors or products (doing,
making) that can be captured by
assignments.
Bloom’s Taxonomy






LEVEL
SOME COGNITIVE BEHAVIORS
Evaluation
Appraisal of an Analysis
or Synthesis
Assembly of Application
Disassembly of Application
Use of Understanding
Management of Knowledge
Memorization of facts, language,
concepts, principles, theories
Synthesis
Analysis
Application
Understanding
Knowledge
The Next Step . . .

Faculty Should Agree on How
Best to Measure Student
Performance on each Objective
Contributing to Program-Level
Goals.
Some Assessment Ways and Means












Assessment days or centers
Case studies
Classroom assessments
Completion and retention
studies
Content analyses
Debates
Direct observations
Focus groups
Graduate success
Internships and service
learning
Interviews (including
videotapes)
Exams for certification and
licensure









Matrices
Performances
Portfolios of several kinds
Projects (Primary Trait
Analysis)
Questionnaires and
surveys (Direct and
telephone; employer,
alumni, and student
attitude and satisfaction)
Reflective essays
Study and activity logs
Tests
(Locally-developed and
standardized)
Transcript analyses
Another Step . . .




Depending on the method (ways and means)
used to assess a course objective, develop a
rubric or a method of measurement.
Evaluate student performance on the specific
objective on three levels: Exceeds
Expectation; Meets Expectation; Does Not
Meet Expectation.
Report your findings.
Discuss the result(s), determine any actions
to take (if necessary).
Example Rubric for MGT. 4335
Base:
100%-90%
All sections were thoroughly and insightfully completed. All narratives, profiles
and relevant sections were factually accurate and documented. Lots of facts
and data to support your claims. You clearly demonstrate a superior level of
analytical reasoning and critical thinking based on your analysis. The paper was
well-written and obviously put together with care (page breaks, layout, etc are
appropriately placed, the paper is grammatically correct, and the level of writing
is appropriate to an undergraduate senior in the McCoy College).
89%-80%
All or most sections were adequately addressed and factually accurate. All
sections included and reasonably prepared. Most sections include data to
support assertions and are appropriately documented. A large amount of
analytical reasoning and critical thinking was evident. Writing and layout may
contain a few errors, but most of the document is relatively clean.
79%-70%
Some (few) sections were rather shallow and superficial, lacking any significant
insight, and/or a few are missing in the document. Some to little data was
provided to support claims, and documentation is either incorrect or missing.
Some of the sections may contain factually inaccurate data and the layout may
have been sloppy and/or done without care. Writing may be fair to poor
containing several errors and style is poor or inappropriate for a formal
document.
69% or less
The paper was extremely poor, and/or some sections were missing. The
sections were VERY superficial and little documentation was provided, and/or
the sections deviate from the required format. Errors/inconsistencies are
evident. Writing contains several to many errors and grammar is poor. In short,
it appeared that the paper was written ‘at the last minute’ and didn’t reflect much
thought.
Responsibilities of Core Course
Coordinators





1. completing the Course Alignment Matrix which
specifies which program-level learning goals are covered
in a specific core course,
2. developing course learning objectives that lead to
fulfillment of program-level learning goals,
3. collecting data from faculty teaching course sections
to support or measure student learning of program-level
learning goals (including a representative sample of
measurement devices/techniques/assignments to be
maintained in the office of the core course coordinator),
4. reporting collective results on program-level learning
goals to the McCoy College Assessment Committee, and
5. discussing and acting on specific recommendations
made by the college assessment and curriculum
committees.
McCoy College of Business
Undergraduate Learning Goals
Matrix Fall 2005-Spring 2006
Course:
Faculty member:
Date:
Goal
1. Conceptualize a complex issue into a
coherent, persuasive written or oral statement.
1a. Develop well-written reports, memos,
and letters that explain findings, organize
ideas into a coherent train of thought, and
justify a conclusion or recommendation.
1b. Make effective oral presentations that
explain findings, organize ideas into a
coherent train of thought, and justify a
conclusion or recommendation.
2. Acquire and use analytical tools and skills
for evaluating information, solving problems,
and making sound decisions.
2a. Synthesize and evaluate the relevance
of data.
2b. Demonstrate critical and reflective
thinking skills in exploring new questions,
analyzing complex issues from multiple
perspectives, and arriving at reasoned
conclusions.
Covered Assessed
Method of
assessment
McCoy College Undergraduate (BBA) Program Goals Coverage
Fall 2005
Goals
CIS 1323
QMST 2333 Acc. 2361
Acc. 2362
Eco 2314
Eco 2315
Blaw 2361 Mgt. 3353
Mgt. 3303
Mkt. 3343
CIS 3380
Fin. 3312
Mgt. 4335
Totals
Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd.
1. Conceptualize a complex issue into a
coherent, persuasive written or oral
statement.
1a. Develop well-written reports, memos,
and letters that explain findings, organize
ideas into a coherent train of thought,
and justify a conclusion or
recommendation.
1b. Make effective oral presentations
that explain findings, organize ideas into
a coherent train of thought, and justify a
conclusion or recommendation.
2. Acquire and use analytical tools and
skills for evaluating information, solving
problems, and making sound decisions.
2a. Synthesize and evaluate the
relevance of data.
2b. Demonstrate critical and reflective
thinking skills in exploring new
questions, analyzing complex issues
from multiple perspectives, and arriving
at reasoned conclusions.
3. Acquire and use information technology
skills in decision-making.
3a. Learn to use Information Technology
(IT).
3b. Apply IT in analysis and
communication.
4. Recognize and apply general concepts
of ethical behavior in dealing with
stakeholders.
4a. Respect the ethical foundations of
diverse populations.
4b. Recognize an ethical dilemma and
resolve it using one of several evaluative
frameworks.
5. Understand the importance of group
dynamics in achieving organizational goals
and acquire and use the skills needed for
effective teamwork.
5a. Understand the importance of group
dynamics in achieving organizational
goals.
5b. Acquire and use the skills needed to
work productively in teams.
6. Understand the importance of culture
and diversity.
6a. Comprehend the nature and
significance of differences in individuals
and groups based on their culture,
ethnicity, religion, and other background
factors.
6b. Understand the impact of globalism
and multiculturalism.
Cov. Assd. Cov. Assd.
Cov.
Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd.
Cov. Assd.
4
4
Cov. Assd.
Cov. Assd.
2
2
17
17
Cov. Assd. Cov. Assd. Cov. Assd.
10
9
Cov. Assd.
7
8
Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd.
Assd.
Cov. Assd.
10
10
Cov. Assd. Cov. Assd. Cov. Assd.
4
4
Cov. Assd.
6
6
7
6
5
4
2
2
5
6
3
3
2
3
Cov. Assd.
Cov. Assd. Cov.
Cov. Assd. Cov. Assd. Cov. Assd.
Cov. Assd. Cov. Assd.
Cov. Assd. Cov. Assd. Cov. Assd.
Cov. Assd. Cov. Assd.
Cov. Assd.
6
Cov. Assd.
Cov. Assd. Cov. Assd.
Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd.
6
Assd.
9
7
Cov. Assd. Cov. Assd. Cov. Assd. Cov
4
3
Cov. Assd. Cov. Assd. Cov. Assd. Cov.
5
4
What documentation must be
retained by the Core Course
Coordinators?

Core course coordinators should retain copies of
instruments used for direct assessment of student
learning, such as assignments, written products, test
questions, rubrics, scoring grids, etc. The course
coordinators also must retain sample student
products relating to the instruments used. Aggregate
results of direct assessments for various programlevel learning goals gathered and/or submitted to the
assessment committee should be retained by the
course coordinator by semester or year for review.
Minutes of core course faculty meetings should be
retained with special emphasis given to items
concerning improvements in instruments used to
assess student learning and acknowledgement and
review of data.
Must all students be
assessed?

For the purpose of meeting AACSB-International
standards relating to Assurance of Learning (16, 18
and 20), sampling may be utilized as long as it is
representative. Faculty teaching college core courses
for various programs, as scholars in their respective
fields and under the leadership of the Core Course
Coordinator, have the responsibility for determining
appropriate representative sample sizes. Graduate
core course coordinators may use census data from
graduate core courses for assessment purposes.
Should college core course faculty and
coordinators gather data and submit
results every year?

Program-level learning goals for undergraduate (BBA)
and graduate (MBA) programs are reviewed on a
scheduled rotational basis. At least two program
goals are reviewed each year, and college core
courses covering that particular program goal are
required to submit the results of course-embedded
assessment to the assessment committee during the
scheduled review year. Since the McCoy College
views assessment as a continuous process, core
course coordinators and core course faculty are
encouraged, but not required, to gather data each
semester for the reporting of assessment results.
Undergraduate Goal Rotation










Program-Level Goal Rotation Sequence:
Academic Year 2005-2006:
Goal 1 – Conceptualize a complex issue into a coherent,
persuasive written or oral statement.
Goal 5 – Understand the importance of group dynamics in
achieving organizational goals and use the skills needed for
effective teamwork.
Academic Year 2006-2007:
Goal 2 – Use critical thinking skills to evaluate information, solve
problems, and make sound decisions.
Goal 3 – Use information technology skills in decision-making.
Academic Year 2007-2008:
Goal 4 – Apply general concepts of ethical behavior in dealing
with stakeholders.
Goal 6 – Understand the importance of culture and diversity.
We have never been
asked...
...to assess
everything all at
once.
Who sets performance standards? How should
Core Course Coordinators report student
performance on various college-level learning
goals?

For each program-level learning goal assessed in a
college core course, the faculty will determine their
minimum expectation or standard. For reporting data
on any specific program-level learning goal, core course
coordinators should aggregate data and report the
results by three levels: Exceeds Expectations; Meets
Expectations; or Does Not Meet Expectations. In order
to preserve the integrity of the assessment process,
core course coordinators should release aggregate data
by course (all sections) and not individual results by the
professor teaching the core course. Program
Assessment in the McCoy College is not the evaluation
of faculty performance in the classroom; it concerns the
aggregate level of student learning of various programlevel goals.
Example Report on Program Goal
Enclosed in this folder are the results of Goal 1a & 1b assessment for the Fall 2005
semester. I have included a packet of evaluation sheets; each student received a sheet
based on their performance on the individual case analysis. For McCoy College Learning
Goal 1a, the specific items are item 8 – Bibliography & Appendices, and item 9 –
Writing/Format. Greater weight was given to item 9 although item 8 was included in the
assessment. The bottom of each sheet indicates whether the student exceeded
expectations, met expectations, or did not meet expectations. A copy of the grading
rubric is included in this packet.
For Goal 1b, a single sheet included shows how each individual performed on their oral
communications segment of a case analysis. Last names have been removed although
first names remain for my control purposes.
Fall 2005 Goal 1a Results:
Exceeds Expectations:
5 students or 19.23%
Meets Expectations:
18 students or 69.23%
Does not meet Expectations: 3 students or 11.54%
Fall 2005 Goal 1b Results:
Exceeds Expectations:
5 students or 19.23%
Meets Expectations:
19 students or 73.08%
Does not meet Expectations: 2 students or 7.69%
An Important Lesson
from the Farm
A pig doesn’t get
any fatter
merely by
weighing it.
How many students must meet or
exceed the performance standard on
a college-level program goal?
There is no prescribed percentage of students that must
meet the standard articulated by faculty. AACSB and other
constituent groups will look at whether and how these
data are used for continuous improvement purposes.
Thus, a poor showing of student mastery of a given
program-level learning goal would only be a concern if the
curriculum was not subsequently modified to improve
student skills/performance of that learning goal. A second
emphasis in the Assurance of Learning Standards is that
program-level learning goals are used to communicate
competencies of graduates to students, employers and
other constituents. Thus, faculty have the responsibility to
develop and modify curricula to insure that program-level
goals are achieved by most graduates and not just a
portion.
What Assessment Asks of You!





One day of initial development time to determine course
objectives and agreeing on a grading rubric for specific course
objectives.
Two to three hours per semester (depending on the goal
rotation) to extract student performance on course objectives
leading to the fulfillment of program-level goals.
Writing a short report (1 page) concerning student
performance.
Maintaining examples of student work, grading rubrics or other
examples demonstrating student achievement of course
objectives.
Talking to other faculty about continuously improving classroom
processes and products to improve student learning of program
level goals over time.
Closing Thought
The enemy of the good
is the perfect.