Transcript ASSESSING THE IMPACT OF MANAGEMENT ON THE …

Evaluating and Assessing Adult Student Learning
Outcomes: A Quantitative Methodology
Presented at the International Assembly for Collegiate Business
Education (IACBE) Conference
Portland, OR
October 18-19, 2012
Gary F. Keller, Ph.D.
College of Business
Eastern Oregon University, LaGrande, OR
[email protected]
Introduction
 Quantifying the effect that a teacher has on the learning outcomes of
their students is a continuous and contentious issue in higher
education.
 Over the past decade, upgraded professional accreditation
requirements, reinvigorated regional accreditation associations,
government mandates and increasing competition from for-profit
educational institutions has forced changes in the evaluation process.
 Unlike scholarship, artistic productions, scientific experiments, and
etc. which results in a tangible product the aftermath of teaching is
contained in the minds of the learner and may or may not manifest
itself for years after the final lecture or test is administered (Hooper &
Page, 1986).
Purpose of the Study
 The issue addressed in this quantitative study was the inability of
standard post course evaluations of university level faculty teaching
effectiveness to accurately and reliably measure changes in student
learning outcomes.
 A secondary goal of the research was to study the sluggish feedback
faculty members receive (Wetzstein and Broder 1985) from
institutional sources that inhibits faculty from utilizing data to make
adjustments rapidly.
Location of Study: Macro View
Milwaukee,
Wisconsin
Location of the Study: Micro View
City of Milwaukee
Some Facts About Milwaukee, WI
 Milwaukee, Wisconsin, is one of the 25 largest cities in the United States with a
population of nearly 600,000 people.
 According to the 2010 Census, 44.8% of the population was White (37.0% non-
Hispanic white), 40.0% was Black or African American, 0.8% American Indian and
Alaska Native, 3.5% Asian, 3.4% from two or more races. 17.3% of Milwaukee's
population was of Hispanic, Latino, or Spanish origin (they may be of any race). The
metropolitan area was cited as being the most segregated in the U.S. in a Jet Magazine
article in 2002.The source of this information was a segregation index developed in the
mid 1950s and used since 1964. In 2003, a non-peer reviewed study was conducted by
hired researchers at the University of Wisconsin–Milwaukee which claimed that
Milwaukee is not "hypersegregated" and instead ranks as the 43rd most integrated city
in America
 Milwaukee is home to ten Fortune 1000 company headquarters, including such household
names as Johnson Controls, Northwestern Mutual, and Harley-Davidson.
 Milwaukee experiences all four seasons and rapidly changing weather. The "lake effect"
of Lake Michigan makes for cooler summers, warmer winters (than surrounding areas
further from the lake), and plenty of snow.
http://www.citytowninfo.com/places/wisconsin/milwaukee
Literature Search
Author (date)
Key Finding
Hooper and Page (1986)
Correlations found with extraneous variables - majors rating courses higher than nonmajors, students taking required courses gave lower ratings than those who selected them;
experienced students giving higher ratings than freshmen and sophomores and
experienced faculty being rated higher than beginning instructors.
Madu & Kuei, 1993
Flaws/Fallacies of Standard Evaluations - a) the student is completely unbiased; b) the
student can discern stellar instructional practices; c) the student has preconceived grade
expectations; d) the lack of student knowledge of course outcomes and e) the absence of
standardized tests to evaluate the quality of students from each class.
Hill and Herche (2001)
Students should be viewed as products rather than customers. By embracing this paradigm
faculty members would be attempting to serve a different “customer” - the employer.
Robbins (2001)
To improve and grade faculty teaching effectiveness – self-observation.
Pritchard, Saccucci, and Potter
(2010)
To demonstrate teaching effectiveness resulting data should be analyzed frequently and
more than one measure of teaching effectiveness should be included.
Jalbert, Jalbert and Furumo (2011)
Graduates from AACSB accredited colleges did not outperform other CEOs.
Methodology
 In this research project, a pre/post evaluation of learning
assessment was developed and tested with a sample of 72
adult students enrolled in an accelerated degree program at
a large urban university.
 Eight randomly selected courses (7 MBA and one
Bachelors).
 The type of data collected was quantitative and acquired
from a 10 question pre /post survey administered during
the first and last night of the subject's course. Respondents
were asked to answer questions by selecting a response from
a 10 point Likert type scale.
 All institutional research protocols followed.
Methodology
An example of a question and response is provided below.
Please rank your reply (circle one number) on a 1 (indicates a low level of
satisfaction or skills) to 10 (indicates a very high level of satisfaction or
skills).
1. I have some knowledge or experience in the content area of the course.
(if so please briefly state your professional or academic experience related to the
course)
1
2
Low
3
4
5 6 7 8 9 10
High
Hypothesis
 H1o. There is no statistically significant difference in the
learning outcomes of adult students enrolled in the course
due to the teaching practices of the faculty member.
 H1a. There is a statistically significant difference in the
learning outcomes of adult students enrolled in the course
due to the teaching practices of the faculty member.
Methodology
1.
2.
3.
4.
5.
Data was analyzed using a t-Test: Paired Two Sample for Means
to ascertain the significance of each respondent’s pre/posttests.
On an individual class basis, each respondent’s pre/posttests
were tested to determine the significance level of changes in
learning outcomes.
The significance levels for the pretest and posttest were then
complied and compared to one another to determine the level
of significance for the entire class.
Each class’s total significance level score was recorded in a
master file. The final significance level for the study was
determined by averaging all of the course significant level
scores.
A .05 level of significance was used to determine significance.
Findings
This research study consisted of adult learners in
accelerated undergraduate and graduate program. In
this research project, a pre/post evaluation of learning
assessment was developed and tested with a sample of
72 adult students enrolled in an accelerated degree
program at a large urban university.
This metropolitan area is composed of a diverse
population.
The response rate to survey was nearly 100%.
Key Findings
The t-Test: Paired Two Sample for Means was employed to
compare the averages of the subjects’ pre/posttests – example
is the result of one course average significant level.
t-Test: Paired Two Sample for Means
Mean
Variance
Observations
Pearson Correlation
Hypothesized Mean Difference
df
t Stat
P(T<=t) one-tail
t Critical one-tail
P(T<=t) two-tail
t Critical two-tail
(Pre –Course) Variable 1
6.701818182
3.958736364
11
0.620150786
0
10
-3.865049055
0.001567174
1.812461123
0.003134347
2.228138852
(Post-Course) Variable 2
8.559090909
0.831369091
11
Key Findings
The decision whether to accept or reject the null hypothesis
was determined by averaging the P one-tail scores of every
class.
A .05 level of significance was used to define significance. The
results for the null hypothesis are shown in Slide 13.
Key Findings
Group -Adult Learners
Pre Ave.
Post Ave.
P(T<=t) one-tail sig.
Year
CMB 510 MSM 2015
6.75
8.55
0.0015
2011
MGT 545 MSM 4028
7.1
8.35
0.0052
2012
MGT 584 MSM-4027
6.76
8.42
0.0114
2012
CMB 544, MBA 4028
7.69
8.79
0.004
2012
CMB 544, MBA 4029
7.8
9.08
0.0028
2012
MGT 460, BSBA 4031
8.42
9.44
0.008
2012
MGT 534, MSM 4034
8.32
9.11
0.0003
2012
MGT 545. MSM 2011
7.9
8.54
0.0013
2012
7.95
8.78
0.004
Average
Key Findings
 The null research hypothesis was not accepted in
regard to changes in student learning outcomes as a
consequence of the faculty member’s teaching practices.
 The .004 level of significance indicated that there is a
99.996% chance of a type 1 error if the null hypothesis was
accepted.
 The alternative hypothesis of this study was accepted
that there is a statistically significant difference in the learning
outcomes of adult students enrolled in the course due to the
teaching practices of the faculty member.
Summary, Conclusions And Recommendations
1. This survey and methodology should motivate others to test it and begin a discussion
2.
3.
4.
5.
6.
of the larger issues.
For example, a comparison between adult learners and traditional aged students
would be of interest to ascertain if these two groups learn differently and therefore
need faculty members to modify or specialize in teaching these groups.
Other interesting questions include do males and females learn differently; do
different ethnic groups learn differently than the majority population; do specific
academic disciplines learn differently than others; do on-line students learn
differently than face to face students?
A final question is how should student evaluation surveys and their results be used
within an academic institution’s promotion, institutional effectiveness and
accreditation processes?
To a degree, the standardized test instruments administered regularly are useful
methods to accumulate (like an autopsy) what transpired across the curriculum
during a given semester. For example students in CMB 510 (Table 1) rated this
faculty member a 5 (on a 1 (low) to 5 (high) scale for the key question on the
institution’s standard end of course survey: “I was satisfied with the performance of
the faculty member for this course” (See Appendix B for complete post course
survey results). A perfect or high score is always welcomed by a faculty member and
impressive for promotion. However, this score does not indicate if any measureable
learning took place.
There is room for both types of analysis.
Questions and Discussion