Transcript Using Multiple Methods to Assess the Impact of

Multiple Methods for
Assessing Learning
Community Outcomes
Maureen Pettitt, Ph.D.,
Skagit Valley College
Shanda Diehl,
Spokane Falls Community College
AIR San Diego
May 2005
Session Overview




Learning Communities at SVC &
SFCC
 Rationale
 Organization
LC Assessment at SVC
LC Assessment at SFCC
Conclusions
Collaborative Courses at SVC




Collaborative courses are a means of
delivering instruction and fostering student
learning; they do not have associated
credit requirements
Learning Communities (and English links)
are required for the transfer degree
Collaborative courses are options for the
technical arts degree.
60+ Learning Communities are offered
each year at the college.
Rationale for Interdisciplinary
Learning at SVC


Advantages (greater retention, student
involvement, etc.) were known, but not part of
the initial rationale for requirements.
A response to curricular issues: Faculty felt that
students
 did not see connections between and among
disciplines, and
 needed to engage subjects more fully, to see
education as a dynamic and interconnected
process of exploration and discovery
Multiple Assessment Methods
3. CCSSE
2. Student
Writing
1. Student
Satisfaction
Survey
1. Locally-Developed Student
Satisfaction Survey




Faculty-developed two-item survey
All responses on a scale from “strongly
disagree” to “strongly agree”
Current N for LCs= 1364
Faculty are provided their course results
and comments, plus the cumulative for
the quarter for all courses
“This learning community has been a
valuable learning experience”
60
50
40
49.1
34.5
30
20
10.7
10
0
3.7
Strongly
Agree
Agree
Neutral
Disagree
2.0
Strongly
Disgree
“It was more valuable to take these
classes together than separately”
40
35
30
25
20
15
10
5
0
35.0
34.0
16.7
11.0
3.3
Strongly
Agree
Agree
Neutral
Disagree
Strongly
Disgree
Student Comments



LC: “This class has been fun and not
sucky at all. I think I have learned a
lot.”
DE LC: “Doing both [topics] in depth
was overwhelming at times.”
DE LC: “Art History rocks!”
Student Comments

“If they had been
separate, I would
have known the
what and where, but
not the why, and the
why is always the
most important
question.”
2. Student Writing


Review of Learning Outcomes
Assessment Plans (LOAPS) indicated
that the overarching Gen Ed outcomes
were not addressed in any obvious way
Faculty/IR team developed two-year
research project to assess student
attainment of overarching General
Education learning outcomes
Study Goals
Assess students' ability to:
 Apply a variety of concepts/texts/contexts
and perspectives to solving problems and
thinking about issues.
 Connect one’s own life experience, ideas, and
abilities with those that others bring.
 Understand and value the learning process
for oneself and for others.
 Write, speak, read, and listen effectively.
 Demonstrate critical thinking skills.
Study Goals
 Assess learning outcomes by modality

determine "if" and "how" this learning
occurs in the context of "where" (i.e.,
interdisciplinary courses, stand-alone
courses, distance education courses)
Approach




Identified courses being taught in Learning
Communities, stand-alone, and DE that
could be “matched” over a two-year period
Faculty agreed to participate and give
students course credit
Reviewed work done at SVC with Bill
Moore (MID), Alverno College interviews,
etc.
Developed a set of questions for students
beginning-, mid-, and end-course
Faculty Participation



Winter 2002: Developed 20-item
list -- “Learning That We Value” –
collapsed to 5 items
Spring 2002: Analyzed student
writing from Winter quarter to
“pilot” the framework
Summer 2002 & 2003: Stipends to
analyze student writing; discuss
and report results
Beginning of Course Questions
1.
2.
3.
What are your learning expectations
for this course?
How will you know that your learning
expectations are being met?
What value, if any, do you expect this
learning will have for you?
Mid-Course Questions
1.
2.
3.
Are your learning expectations being
met?
Have you learned things that you hadn't
anticipated? If so, please describe.
Do you find your learning experience in
this class is any different from high
school or other community colleges that
you know about?
End of Course Questions
Have your learning expectations been
met?
2. Have you learned things that you hadn't
anticipated? If so, please describe.
3. What do you think are the most important
aspects of your experiences in this course
that account for your learning?
4. What have you learned in this course that
will matter to you five years from now?
1.
Findings



Student writing elicited adequate depth and
breadth of comments that demonstrated
 application of a variety of concepts, texts,
contexts to solving problems/thinking about
issues
 understanding/valuing learning process
Fewer, but adequate, comments demonstrating
self-other connection and critical thinking
Least useful for demonstrating writing, speaking,
reading and listening skills
Students in Learning
Communities


“After taking this course I feel that I can
make connections to various things,
such as history, influences, people, and
culture. This course taught me the
value of making connections and things
from my own perspective.”
“I think I will be more likely to make
connections between subjects, both
similarities and differences.”
Students in Learning
Communities


“I do believe that I may think in a
way of applying what I am learning
to something else.”
“The link between history and music
and the connection to the present
day versus the past was important
writing. The weekly seminar papers
really forced me to look at the world
differently.”
Students in Learning
Communities

“By combining course topics you
get the ‘bigger picture’ and are able
to sort of apply what we are
learning better. By applying a
subject or topic to another subject
or topic you have to comprehend
what you are learning and apply it
to other things.”
Comparisons

Comparing student responses in the
stand-alone courses with students in
collaborative courses—same courses
with the same instructor(s)-students in stand-alone courses:


were less likely to write about these
connections or about learning, and
tended to focus on personal growth,
liking faculty, etc.
Lots more to do….

Qualitative
assessments of this
kind are very
ambitious projects-review, coding and
analysis very time
consuming
3. Community College Survey
of Student Engagement
CCSSE Factors Analyzed Based on
Learning Community Attendance





Active & Collaborative Learning
Student Effort
Student-Faculty Interaction
College Contribution to Knowledge,
Skills & Personal Development
Mental Activities
Mental Activities



Similar in structure to Bloom’s
Taxonomy
Prompt: “During the current school year,
to what extent has your coursework
emphasized the following mental
activities…”
Response Options: Range from 1 “Very
Little” to 4 “Very Much”
Groupings
 SVC 1 = SVC students who have already taken a
learning community or linked course
 SVC 2 = SVC students who have not taken, but are
planning on taking a learning community or linked
course
 SVC 3 = SVC students who have not taken and do not
plan on taking a learning community or linked course
 Consortium = all students in the Northwest consortium
of colleges, excluding those from Skagit Valley College
and Douglas College
 All = all students who participated in the CCSSE Survey,
based on 93 colleges
Memorizing
”memorizing
facts, ideas, or
methods from
your courses
and readings so
you can repeat
them in pretty
much the same
form.”
3.00
2.90
2.80
2.80
2.83
2.80
2.72 2.71
2.70
2.60
2.50
Bloom's Taxonomy: Memorizing
SVC 1
SVC 2
SVC 3
Consortium
All
Analyzing
“mental
activities:
analyzing the
basic
elements of
an idea,
experience,
or theory.”
3.00
3.04
2.92
2.90
2.80
2.92
2.79
2.79
2.70
2.60
2.50
Bloom's Taxonomy: Analyzing
SVC 1
SVC 2
SVC 3
Consortium
All
Synthesizing
“synthesizing
and
organizing
ideas,
information,
or
experiences
in new ways.”
3.00
2.90
2.86
2.80
2.80
2.78
2.70
2.67
2.62
2.60
2.50
Bloom's Taxonomy: Synthesizing
SVC 1
SVC 2
SVC 3
Consortium
All
Making Judgments
“making
judgments
about the
value or
soundness
of
information,
arguments,
or
methods.”
2.80
2.74
2.70
2.67
2.60
2.60
2.50
2.49
2.52
2.40
2.30
Bloom's Taxonomy: Making
judgments
SVC 1
SVC 2
SVC 3
Consortium
All
Applying Theories or Concepts
“applying
theories or
concepts to
practical
problems
or in new
situations.”
3.00
2.90
2.80
2.77
2.70
2.71
2.68
2.60
2.60
2.63
2.50
Bloom's Taxonomy: Applying
Theories
SVC 1
SVC 2
SVC 3
Consortium
All
Using Info for a New Skill
3.00
“using
information
you have
read or
heard to
perform a
new skill.”
2.90
2.80
2.81
2.78
2.75
2.70
2.73
2.75
2.60
2.50
Bloom's Taxonomy: Using information
SVC 1
SVC 2
SVC 3
Consortium
All
Students who had taken Learning
Communities were:

Significantly more likely to have:




Discussed grades with an instructor
Worked with classmates outside of class to
prepare class assignments
Had discussions, conversations, and contacts
that encourage multiculturalism and global
awareness.
Used advising and computer labs
SFCC Learning Communities



Learning community is a package of
interdisciplinary courses where students
and faculty are consistently involved
Learning community provides strong
network of relationships, both
intellectual and emotional
Purpose: Improve Student Success in
Community College System
Benefits and Costs

Benefits





Improve Student Outcomes – congruent with
college mission
See interconnectedness of disciplines
Establish relationships with students and faculty
Student-centered
Costs


Higher expense
Efficiency measures
Methodology



Conducted Study in Fall 2003
Cohorts: 2000-01, 2001-02, and
2002-03 full-time, day-time, academic
degree-seeking freshmen taking statesupported classes
Separated cohorts into two groups:
Learning Community participation and
NO Learning Community participation
Methodology (continued)

Compared between the two groups





Cumulative Grade Point Average (T-Test)
Percent of Credits Earned versus Attempted (TTest)
Retention – came back the subsequent academic
year (Chi-Square)
Analyzed each cohort separately
Supplemented Analysis with Student
Satisfaction Surveys of Learning Community
Participation
Findings
Average
GPA
2000 – 2001
Learning Community
n=210
No Learning
Community n=994
2001 – 2002
Learning Community
n=199
No Learning
Community n=1,081
2002 – 2003
Learning Community
n=172
No Learning
Community n= 1,151
* Differences were statistically significant
Percent
Credits
Earned Versus
Attempted
2.64*
78%*
Came Back
Subsequent
Academic
Year
72%*
2.43*
70%*
63%*
2.42
72%*
65%
2.44
68%*
61%
2.44
68%
62%
2.48
72%
62%
Results Ambiguous



Participation in learning communities
make a difference in 2000-01, make a
slight difference in 2001-02 but no
difference at all in 2002-03
Study was replicated for late 90s
cohorts and results were consistent with
the 2000-01 results
What does this mean?
Thought Process


Fall 2003: Hypothesis- not enough time
had elapsed to get full cumulative GPA,
credits earned versus attempted and
only Fall 2003 was considered as the
time to come back for the 2002-03
cohort.
Study put on hold.
Spring 2005


Replicated the study, reran all cohort
years, and added 2003-04 cohort using
the same criteria.
Found that 2003-04 showed same
results as 2002-03. No difference was
detected in student outcomes by
learning community participation in the
2003-04 freshman cohort.
Revised Study


Is there a change in the learning community
offerings that have led to these findings?
Yes! The change in learning community
offerings was offering more developmental
education (remedial classes) as a part of the
learning community. The theory was that
students who form strong intellectual and
emotional connections would do better.



Writing
Reading
Math
Changes in Learning
Communities
2000-01 Freshmen
Cohort
2001-02 Freshmen
Cohort
2002-03 Freshmen
Cohort
2003-04 Freshmen
Cohort
Took
Learning
Developmental Community and
Education in
Took
Learning
Developmental
Community
Education Outside
of Learning
Community
11%
59%
College Level and
Learning
Community
Participation
30%
18%
50%
32%
26%
49%
25%
28%
36%
36%
So . . . 5 Groups Now





Group 1: Took developmental education in learning
community as a freshman
Group 2: Took developmental education AND had no
participation in a learning community as a freshman
Group 3: College Level students (NO developmental
education) AND participated in a learning community
as a freshman
Group 4: College Level students (NO developmental
education) AND did NOT participate in a learning
community as a freshman
Group 5: Participated in a Learning Community AND
took developmental education NOT in a learning
community as a freshman
Methodology



Analysis of individual cohort years
ANOVA tests were used to determine
differences between the 5 groups in
cumulative GPA and percent credits
earned versus credits attempted
Chi-Square Analysis was used to detect
differences in retention
Average
GPA*
Group 1:
Percent
Credits
Earned
Versus
Attempted*
2.17
66%
Group 2:
Took developmental
education AND had NO participation
in a learning community as a
freshman
2.46
70%
60%
Group 3:
College Level
students (NO developmental
education) AND participated in a
learning community as a freshman
2.47
75%
42%
Group 4:
College Level students
(NO developmental education) AND
did NOT participate in a learning
community as a freshman
2.60
76%
46%
Group 5:
2.54
74%
66%
Took developmental
education in learning community
as a freshman
Participated in a
learning community AND took
developmental education NOT in a
learning community as a freshman
* Differences are statistically significant
Came Back
Subsequent
Academic
Year*
56%
Results

Developmental Education seems to be the key
factor:


Of all developmental education freshmen included in
the study, those who take their developmental
education courses outside of the learning community
AND participate in a learning community have the
best outcomes
Students in learning communities with the best
outcomes are those students who participate in
learning communities but take developmental
education courses outside learning community
What Does This Mean?




SFCC purpose of learning communities is to
improve student success in community
college system
Yet, we find that this is not the case, due in
part to changes in learning communities to
incorporate developmental education
offerings
These findings will change the content and
themes of learning communities at SFCC
Re-examine delivery of developmental
education courses
SVC and SFCC



The goal and purpose of learning
communities shape the evaluation and
assessment methodology.
The study findings shape the delivery of
learning communities at each college.
Different approaches yield different
results.

Use the
results to
improve
programs
and
student
success