Transcript CREATE Conference - Southern Business Deans

2004 Southern Business Administration Association Meeting,
Myrtle Beach, South Carolina
The UCF College of Business Approach to
Assessing Student Learning
Taylor Ellis, Ph.D.
Associate Dean
Patrick Moskal, Ph.D.
Director, Testing & Evaluation
University of Central Florida
The UCF College of Business Approach to
Assessing Student Learning
Presentation Outline
1. UCF background.
2. Overview of the assessment process in the College of
Business Administration.
3. Development of student learning outcomes.
4. Description and role of our Testing Lab in the
assessment process.
The UCF College of Business Approach to
Assessing Student Learning
UCF Background
 Large metropolitan university with over 44,000 students.
 College of Business Administration (CBA) offers 7
undergraduate, 10 Masters, and 5 Doctoral degree
programs.
 ~140 full-time and 60 part-time faculty in CBA.
 ~8,000 students enrolled in CBA each year.
The UCF College of Business Approach to
Assessing Student Learning
Issues Faced in UCF CBA
 Number of students.
 Required courses in core.
 Pre-requisites in majors.
 Computer Testing Lab.
The UCF College of Business Approach to
Assessing Student Learning
University
Assessment
Committee (UAC)
CB
Ar
CBA Assessment Process
ep
CBA Departmental
Review Committee
(Dept. Chairs)
re s
en
tat
ive
to
U
AC
(
CBA’s DRC Chair
CBA Teaching &
Learning Committees
(Dept. level)
Develop learning
objectives,
outcomes, and
performance
measures
UA
CC
pp
Su
s
o rt
L
T&
co
e
itte
m
m
lo
eve
d
s
ha
i r)
’s
LO
pS
Associate Dean of
Undergraduate
Programs
Director of Testing &
Evaluation
The UCF College of Business Approach to
Assessing Student Learning
CBA Assessment Process
(continued)
Develop learning
objectives,
outcomes, and
performance
measures
(department level)
Paper-based
assessments (e.g.,
written exams,
papers,
presentations,
portfolios,
standardized exams
Associate Dean of
Undergraduate
Programs
Supports T & L committees develop SLO’s
Director of Testing &
Evaluation
Computer-based
assessment in
Testing Lab
Student Records
(University Database)
CBA Accreditation
Database
Reports
(to CBA, UCF,
Accreditating
Agencies, and State
of Florida)
Data Analyses
The UCF College of Business Approach to
Assessing Student Learning
Assessment Process
CBA Departmental Review Committee (DRC)
 DRC consists of CBA Department Chairs.
 DRC Chair participates on the UAC and reports CBA
assessment results.
 Supported by Director of Testing and Evaluation.
The UCF College of Business Approach to
Assessing Student Learning
Assessment Process
CBA Teaching and Learning Committees
 Selected for each degree program area.
 Establish student learning outcomes (SLOs) and ensure
data collection.
 Participate in an annual review of assessment results,
and plans for upcoming year.
The UCF College of Business Approach to
Assessing Student Learning
Assessment Process
 Each degree program will develop 4-10 general learning
objectives (expectations).
 Each objective will be evaluated by multiple outcome
measures (exam questions or assessment types).
 Summative scales will be developed to measure most
learning outcomes.
 Allows for evaluation in multiple classes.
 Allows for pinpointing specific weak areas.
The UCF College of Business Approach to
Assessing Student Learning
Assessment Process
Degree Program Learning Expectations
Expectation 1
Class 1
Affective 1
Questions & Beliefs 1-?
Collaboration score
Expectation 2
Class 2
Class 3, etc.
Cognitive 1
Questions 1-?
Expectation 3, etc.
Behavior 1
Projects
Questions & Projects 1-?
The UCF College of Business Approach to
Assessing Student Learning
Assessment Goals
 Establish student learning outcomes for each degree
program, based on the CBA Mission.
 Acquire data that provide direction for program
improvement and meet both AACSB and regional
accreditation requirements.
 Develop comprehensive data collection and reporting
system.
 Collect multiple measures over time (data triangulation).
 Utilize existing data where possible.
 Cause minimal intrusion on normal faculty workload.
The UCF College of Business Approach to
Assessing Student Learning
Why Do We Need Student Learning Outcomes?
 Provide feedback to students about their progress.
 Communicate instructional goals to stakeholders (e.g.,
trustees, community, university administration, faculty,
accrediting agencies, parents).
 Provide the means for assessing student learning.
The UCF College of Business Approach to
Assessing Student Learning
Learning Objectives
Begin by developing a list of goals and objectives,
such as the following “essential competencies” for all
UCF Business graduates.
BE2010 Competencies:
Teamwork
Communication
Creative Thinking
Adapting to change
Diversity
Ethics
The UCF College of Business Approach to
Assessing Student Learning
Learning Objectives
From our goals, we develop specific learning objectives. An
objective is defined as a quantifiable and/or observable
achievement accomplished under specifiable conditions.
 How do we develop objectives? Start by answering
basic questions like:
 What do we want our students to know when they
finish our course or program?
 What should our students be able to do when they
finish our course or program?
The UCF College of Business Approach to
Assessing Student Learning
Objectives Vs. Outcomes
 In general, an objective is what we hope that we will
see, and an outcome is what we did see.
 Learning objectives define the outcomes of
instruction in clear terms.
 Outcomes must be both observable and measurable.
 All objectives/outcomes should include three primary
components: the action or behavior to be measured,
conditions of learning, and standards of performance.
The UCF College of Business Approach to
Assessing Student Learning
You Want SMART Objectives
 Specific (Clear and definable terms describing the
knowledge, skills, and attitudes)
 Measurable (Reliable data that can be assessed in
multiple ways)
 Attainable (Feasible to be accurate; need not be perfect)
 Realistic (Expected standards must be practical)
 Time-bound (When will results will be observed?)
The UCF College of Business Approach to
Assessing Student Learning
Types and levels of student learning outcomes
 Cognitive: intellectual skills.
 Knowledge, comprehension, application, analysis,
synthesis, evaluation (from Bloom, 1956).
 Affective: what a student believes (values).
 Behavioral: psychomotor skills.
The UCF College of Business Approach to
Assessing Student Learning
General Knowledge and Skills Examples












Communication
Organizational
Quantitative
Critical Thinking
Interpersonal Skills and Awareness
Group Dynamics
Motivational and Leadership
Ability to Learn and Train Others
Ability to Manage Information Technology
Global Awareness
Understanding Issues of Diversity
Ethical Awareness
The UCF College of Business Approach to
Assessing Student Learning
CBA Assessment Types
 Course embedded assessments of individual and group
knowledge and skills (e.g., exams, projects).
 Program level exams (e.g., doctoral candidacy exams).
 Certification and standardized exams.
 Direct observation.
 Teamwork.
 Student portfolios.
 Internships and service learning.
 Student records.
 College, AACSB, and employer surveys.
 Placement/employer data.
CBA Accreditation Database Development Plan
Student Records
(Peoplesoft)
Faculty Information
credentials, course syllabi
Exam Records
(InQsit & WebCT):
selected test questions,
student responses,
course and student info,
applicable student
learning outcomes
Other Assessment Records:
assessment type (e.g.,
portfolio, oral presentation,
written communication,
collaborative project),
student score on
assessment) course and
student info, applicable
student learning outcomes
CBA Accreditation
Database
Reports
Data Analyses
Learning Outcome
Records:
student learning outcomes,
outcome classification
(Bloom), department and
course info
The UCF College of Business Approach to
Assessing Student Learning
CBA Assessment Database
 Maintain results for each student.
 Prefer automated input from electronic testing.
 Allow input from additional sources.
 Allow data analyses.
 Create reports specific to each degree program
and for diverse stakeholders.
The UCF College of Business Approach to
Assessing Student Learning
CBA Testing Laboratory
 140 networked PCs
 Positioned to maintain exam security.
 Netsupport School Pro software to manage PCs
(e.g., monitor, block unapproved access).
 WebCT and InQsit1 packages available for exams.
 Online student exam scheduling.
1
Developed at Ball State University.
The UCF College of Business Approach to
Assessing Student Learning
CBA Testing Laboratory
The UCF College of Business Approach to
Assessing Student Learning
CBA Testing Laboratory
 Over 7,500 exams given in fall 2003.
 Over 12,500 exams given in spring 2004.
 Goal is to automatically extract specified
program evaluation and accreditation data
from course exams.
The UCF College of Business Approach to
Assessing Student Learning
Testing Software Features
 InQsit: Easy to use and generate tests (Word, text
files, Publisher’s test banks).
 WebCT: Complete course management system.
 Test Question Types: T/F, multiple choice, multiple
response, short answer, matching, identification of
objects on chart/photo (InQsit), limited essay grading
(InQsit), ungraded essay, survey.
The UCF College of Business Approach to
Assessing Student Learning
Testing Software Features (continued)






Random selection of questions from a question pool.
Randomization of questions and answer choices.
Adaptive scoring.
Use of audio and video in tests.
Start and stop times may be set by date and time.
Analysis of Results: automatic student grading and item
analysis.
The UCF College of Business Approach to
Assessing Student Learning
Testing Lab Components Under Development
 Lab entrance will have student ID reader. Students
swipe card or enter PID for computer verification.
 Proctor software will allow the faculty/proctor PC to
display:
Student name and photo.
Test the student is authorized to take.
Computer assigned for exam.
Seat reservation duration.
The UCF College of Business Approach to
Assessing Student Learning
Questions?
The UCF College of Business Approach to
Assessing Student Learning
For more information, contact:
Patrick J. Moskal, Ph.D.
University of Central Florida
College of Business Administration
Undergraduate Student Services, BA-II, Room 101G
Orlando, Florida 32816
407-823-6059
[email protected]