Transcript Assessing Integrated Learning

A Collaborative Study on the
Preparation of Community College
Students for the Four-Year
Experience,
With a Focus on Integrated Learning
Maureen Pettitt, Ph.D.
Skagit Valley College
Beth Hartsoch, M.A.
Western Washington University
2008 AIR Forum
Seattle, WA
Session Overview



Measuring Integrated Learning – Learning
Community Outcomes
Background: Learning Communities at SVC
Measurement Strategies

Established Assessment Approaches


New Assessment Approaches




Student Writing, Course Satisfaction Survey & CCSSE
Academic performance indicators
Western Educational Longitudinal Study (WELS)
Survey Focus Groups
Findings
Measuring Learning Community
Outcomes
Questions:
 How can we measure the impact of
learning communities at the 2-year
and 4-year level?

What do we gain by using multiple
assessment approaches?
Background: Integrated Learning
at Skagit Valley College


Interdisciplinary courses are a means of
delivering instruction and fostering student
learning
A response to curricular issues: Faculty felt
that students
did not see connections between and among
disciplines, and
 needed to engage subjects more fully, to see
education as a dynamic and interconnected
process of exploration and discovery

Typical Learning Communities

Fully Collaborative
Feast or Famine (Nutrition and Sociology)
 Sex.comm (Human Sexuality and Mass
Communication)
 Stating the Matter (Chemistry and English
Composition)


Developmental (federated, co-enrollment)

Reading Between the Numbers (developmental Math and
Reading)
Typical Learning Communities

Federated
Celluloid Science - science majors enroll in an
introductory film course and one of the courses
required for their major , with the explicit purpose
of exploring how films portray scientists, scientific
practices, and concepts
 This, That, and the Other - students co-enroll in a
research paper course and one of several social
science courses, with the goal of researching topics
specific to their field of study

Past Assessment Approaches
CCSSE
Student
Writing
Student
Satisfaction
Survey
1. Student Writing Study

Faculty/IR team developed two-year research
project to assess student attainment of
overarching General Education learning
outcomes using student writing
Student Writing Study: Method




Identified courses being taught in Learning
Communities, stand-alone, and DE that could
be “matched” over a two-year period
Faculty agreed to participate and give students
course credit
Developed a set of questions based on Gen
Ed goals and faculty input
Students surveyed beginning-, mid-, and endcourse
Student Writing Study: Findings


Students’ responses regarding valuing learning,
connecting classroom learning with the world,
and connecting with other were fairly similar in
both stand-alone and LC courses.
Students in LC courses were far more likely to
cite applying or appreciating a variety of
perspectives to problem-solving (55% versus
15%)
Student Writing Study: Findings

Comparing student responses in the standalone courses with students in collaborative
courses—same courses with the same
instructor(s):
Students in stand-alone courses were less likely
to write about these connections or about
learning, and
 tended to focus on personal growth, liking
faculty, etc.

Student Writing Study:
Student Comments

“After taking this course I feel that I can make
connections to various things, such as history,
influences, people, and culture. This course
taught me the value of making connections and
things from my own perspective.”
2. Quarterly Surveys


Short survey administered by staff
member in all Learning Communities
Scale from “strongly disagree” to
“strongly agree”
Faculty are provided their course results
and comments, plus the cumulative for
the quarter for all courses
Learning Community
Survey Questions
1.
"This learning community has been a
valuable educational experience."
2.
"I believe it was probably more
valuable to have taken these classes
together than it would have been to
take them separately."
Mean Responses to Value Questions
(N = 5156)
4.1
4
3.9
3.8
3.7
3.6
3.5
3.4
3.3
4.1
3.9
3.8
3.6
Valuable
More Value Together
LC
Comp LC
Student Comments



“This class has been fun and not sucky at all. I
think I have learned a lot.”
“I did not learn too much other than how to
explore more in depth, how to write more.”
“If they had been separate, I would have known
the what and where, but not the why, and the why
is always the most important question.”
3. Community College Survey of
Student Engagement



Administered the CCSSE in 2003, 2005,
and 2007
Went to this survey because it most closely
matched the institution’s focus on student
learning and engagement
The LC question was a bonus extra!
CCSSE LC Question



Stem: “Which of the following have you done, are
you doing, or do you plan to do while attending this
college?”
Category: “Organized learning communities (linked
courses/study groups led by faculty or counselors)”
Response Categories:



I have done
I plan to do
I have not done nor plan to do
Method


Used T-test for independent samples to
examine differences in effort and engagement
between students who had taken Learning
Communities and those who had not done
nor planned to do.
Results are only for the latest survey
administration (Spring 2007)
Results
Prompt: “In your experiences at this college during the
current school year, about how often have you done each of
the following?”
Responses range from 1 (“Never”) to 4 (“Very Often”)
Learning Community
Variable
Asked questions in class
or contributed to class
discussions
Made a class
presentation
Have
Taken
(n = 173 )
Not
Taken
(n = 232)
Sig.
3.00
3.00
1.000
2.43
2.27
.069
Learning Community
Variable
Prepared two or more drafts of a
paper or assignment before turning
it in
Worked on a paper or project that
required integrating ideas or
information from various sources
Have Taken Not Taken
(n = 173 )
(n = 232)
2.96
2.45
Sig.
.000
.000
3.14
2.75
Worked with other students on
projects during class
2.88
2.69
.021
Worked with classmates outside of
class to prepare class assignments
2.49
2.10
.000
2.87
2.69
.038
Discussed ideas from your readings
or classes with others outside of
class (students, family members,
co-workers, etc.)
Learning Community
Variable
Have Taken Not Taken
(n = 173 )
(n = 232)
Sig.
Worked harder than you thought
you could to meet an instructor’s
standards or expectations
2.78
2.52
.003
Used email to communicate with an
instructor
3.10
2.67
.000
Discussed grades or assignments
with an instructor
2.76
2.58
.039
Discussed ideas from your readings
or classes with instructors outside
of class
2.11
1.88
.012
Worked with instructors on
activities other than coursework
1.74
1.45
.000
Convergence of SVC Data

Findings from LC surveys, student writing and
the CCSSE strongly suggest that taking
Learning Communities results in higher levels
of
effort and engagement
 collaborate with peers
 interaction with faculty

What we didn’t have….
…Perceptions or performance of students
after they transferred from Skagit
Collaboration with Western Washington
University’s Office of Survey Research
Outcomes/Data Sources
Better Academic
Performance
Academic Performance Indicators
(GPA, credits earned,
persistence, etc.)
Engagement with faculty
and peers
WELS Student Engagement –
Previous College
Making connections among
disciplines
Academic preparation;
expected and perceived
difficulty
Focus Groups
Easier/Better Transition to
4-yr
WELS Expected Difficulty &
Perceived Difficulty
WELS Info/Tech Literacy
WELS Study Skills
WELS Integration into College
Focus Groups
Notes

Quality and type of experience is unknown, and we
know from quarterly surveys that this varies widely



linked course vs. fully integrated
Instructor investment and experience
Students who took no LCs could be a special case that
is not accounted for with available controls

Those with schedule constraints such as employment or
family may be less likely to take LCs
Notes

When looking at significance, we used a very liberal p
value of 0.1 to select outcomes.




Since this is an ongoing study used to improve programs, we
want not only evidence of effects, but also suggestions for
further investigation.
In the case of WELS outcomes, our sample is small when
broken down by number of LC experiences.
There are no substantial consequences for being wrong.
WWU will conduct another WELS transfer baseline in 2008,
and we may use that opportunity to gather additional
information about some of these items.
WWU Academic Performance
Indicators

Outcomes





Controls



Fall 1: credits enrolled, credits earned, GPA
Winter 1: retention
Year 1: GPA, mean credits earned/quarter
Fall 2: retention
Community college, sex, race/ethnic, age, transfer credits,
transfer GPA, associates degree, WWU full-time status
Variables in outcomes list used as controls when appropriate
Independent variables

1 LC, 2 LCs, 3 LCs. No LCs is excluded category.
WWU Academic Performance
Indicators
Fall 1 Significant Effects
Dependent
N
Mean
Independent
β
p
Credits enrolled - FT
197*
13.61
3 LCs
0.226
0.073
Credits earned
217
12.1
2 LCs
-0.14
0.069
209**
3.04
2 LCs
-0.17
0.023
GPA
* FT students only (registered for 12 or more credits in fall 1)
** Some GPAs missing because students took all courses pass/fail
WWU Academic Performance
Indicators

No significant differences for:
Fall - Winter 1 retention
 Year 1 major declared
 Year 1 credits earned
 Year 1 GPA

Western Educational Longitudinal
Study (WELS)

Fall 2005 Transfer Cohort



1051 students – 44 SVC respondents & 38 Seattle District
respondents
Survey items related to students’ expectations, preWWU experiences, college preparedness, etc.
Three separate surveys of incoming transfer students
administered on-line, with phone follow-up



Prior to starting classes - baseline
After the first quarter - transitions
After finishing three quarters & yearly follow-ups (not
included in this analyses)
WELS Baseline
Academic Preparation
Dependent
N
Study Skills
79
Info/Tech Literacy - Index
79
Plan/Implement/Organize
Research
79
Independent
β
p
1 LC
0.406
0.07
2 LCs
0.327
0.073
2 LCs
0.318
0.067
1 LC
0.379
0.078
2 LCs
0.503
0.005
3 LCs
0.536
0.018
WELS Baseline
Expected Difficulty
Negative beta indicates less difficulty expected
Dependent
N
Independent
β
p
Managing time effectively
77
3 LCs
0.461
0.072
1 LC
-0.491
0.028
3 LCs
-0.492
0.039
2 LCs
-0.361
0.054
1 LC
-0.533
0.014
2 LCs
-0.34
0.048
3 LCs
-0.599
0.01
Written & Oral Presentation Index
Writing Papers
Participating in Discussion
Groups
77
77
77
WELS Transitions
Perceived Difficulty – Fall 1
Negative beta indicates less difficulty perceived
Dependent
N
Independent
β
p
Participating in discussion groups
50
1 LC
-0.37
0.097
1 LC
-0.444
0.043
3 LCs
-0.473
0.072
1 LC
-0.395
0.072
Academic Performance - Index
Writing papers
50
58
Results – WELS
No significant differences for:
 Baseline:
Academic, community and campus engagement
 Expected integration into college
 Using computers for academic work
 Using library resources for research
 Preparing for & presenting work in front of a class


Transitions:

Time and task management index
Student Engagement – Previous College
Percent responding that they did this 4 or more times
No LC
(n = 9)
1 LC
(n = 27)
2 LC
(n = 14)
3+ LC
(n= 31)
Talked to a professor outside
of class
77.8%
74.1%
50.0%
90.3%
Personally contributed to a
classroom discussion
88.9%
85.2%
100.0%
90.3%
Volunteered on or off campus
outside of coursework
44.4%
14.8%
50.0%
41.9%
Sought out a job for credit
rather than pay
0.0%
0.0%
0.0%
0.0%
Did community service work
as part of a course
0.0%
7.4%
0.0%
12.9%
Participated in a club or
activity
22.2%
18.5%
21.4%
22.6%
Focus Groups
Making connections among disciplines…
Focus Groups: Design



WWU generated list of SVC transfers
SVC did transcript analysis to identify
student course taking, particularly Learning
Communities
Identified two groups for interviews:
1) a mix of none, one, or two or more LCs
taken at SVC
2) two or more LCs taken at SVC
Questions for Both Focus Groups




What aspects of your learning experiences at Skagit Valley
College have helped you in your courses at Western
Washington?
If you could change anything about your learning
experiences at Skagit Valley College so that you would be
better prepared to succeed at Western, what would that be?
Can you talk a little bit about how Learning Community
courses have helped or hindered your academic progress at
Western?
If you took an English course that was linked with another
course, can you describe how that learning experience has
been useful or not in your studies at Western?
Additional Group-Specific Questions

For the first group:


If you have taken more than one Learning
Community or English Link, how did taking a
second or third learning experience of this kind
impact the development of your skills or
knowledge?
For the second group:

How did taking the second or third learning
experience of this kind impact the development of
your skills or knowledge?
Focus Group Responses
We summarized the most commonly cited ways
in which students reported benefiting from
their LC experiences.
In most cases there were students who
disagreed with these ideas, and did not feel
they benefited in this way.
Our purpose is not to quantify the responses,
but to report the types of benefits reported
by students.
Helpful in Transition to WWU







Interdisciplinary learning
Analytical/critical thinking
Research and writing
Group work
One on one interaction with instructor
Course content in specific courses
Additional helpful aspects noted: completion of GURs,
knowing people who have attended Western, and the
financial benefit of attending SVC
Interdisciplinary Learning


The interdisciplinary aspect of learning
communities helps to make classes more
interesting and fun at SVC.
Exposure to interdisciplinary learning at SVC
helps students to feel more confident in their
classes at Western.

This is supported by WELS outcomes on expected
and perceived difficulty.
Interdisciplinary Learning
Students say they have learned to make
connections and to apply what they’ve learned
to another topic or to the real world.
Writing

Most students say English LCs were useful in their
studies at SVC and in transferring to Western—
challenged students to think and write analytically,
outside of the box.


This is supported by WELS outcomes for expected
and perceived difficulty with writing.
English was more interesting because it was linked
to a content area which provided something to
write about.
Research and Writing

Research and writing experiences in English
links and learning communities at SVC
helped to prepare students for research and
writing assignments at Western.

This is supported by WELS outcomes for
planning, implementing and organizing research.
Analytical/Critical Thinking

The emphasis on critical thinking at SVC
helps students in their classes at Western
because they learned how to analyze
information, make connections, and
synthesize major ideas.
Group Work

Group work in links and learning
communities helped to prepare students for
group work assignments at Western.

This is supported by WELS outcomes for
expected and perceived difficulty with group
work.
Faculty Coordination


Coordination between instructors is critical.
Absent this coordination, linked course
experiences are no different from stand-alone
courses.
Co-teaching enhances the learning community
experience further.
WWU Findings
Fostering Student Learning



Taking Learning Communities does not result in
significantly higher academic performance
Students who took LCs feel better prepared
academically for their WWU experience, especially
with regard to writing, research, and group discussion.
Students who took LCs perceive less difficulty at
WWU after their first quarter
Engaging with college, peers and faculty

Students taking Learning Communities did not report
significantly higher engagement
Questions – Future Research



Do students who took LCs take more challenging
classes?
Do the “transfer – in” credits and experiences (to
SVC) play a role in the differences? Is there a
way to quantify those experiences?
The differences between number of Learning
Communities taken and student engagement
warrants some additional study, particularly in
light of other results
The advantage of using multiple
methods….
…you get to do a
“reality check” on
your data
interpretation;
multiple views
generate better
understanding and
suggest new
inquiries
Thanks for coming!