Transcript Assessing Integrated Learning

Assessing Integrated Learning
Maureen Pettitt, Ph.D.
Skagit Valley College
Beth Hartsoch, M.A.
Western Washington University
PNAIRP – Oct ‘07
Session Overview


Learning Communities at SVC
Established Approaches


SVC/WWU Project



Student Writing, Course Satisfaction Survey
& CCSSE
WELS Survey & Focus Groups
Themes from the Data
New Studies in Progress
Interdisciplinary Learning at SVC


Interdisciplinary courses are a means of
delivering instruction and fostering student
learning
A response to curricular issues: Faculty felt
that students


did not see connections between and among
disciplines, and
needed to engage subjects more fully, to see
education as a dynamic and interconnected
process of exploration and discovery
Past Approaches:
Multiple Assessment Methods
CCSSE
Student
Writing
Student
Satisfaction
Survey
1. Student Writing Study

Faculty/IR team developed two-year
research project to assess student
attainment of overarching General
Education learning outcomes
Student Writing Study: Goals
Assess students' ability to:




Apply a variety of concepts/texts/contexts
and perspectives to solving problems,
thinking about issues
Connect one’s own life experience, ideas
and abilities with those that others bring
Understand and value the learning
process for oneself and for others
Connect to external, lifelong social issues
Student Writing Study: Method




Identified courses being taught in Learning
Communities, stand-alone, and DE that could
be “matched” over a two-year period
Faculty agreed to participate and give
students course credit
Developed a set of questions based on Gen
Ed goals and faculty input
Students surveyed beginning-, mid-, and endcourse
End of Course Questions
Have your learning expectations been met?
2. Have you learned things that you hadn't
anticipated? If so, please describe.
3. What do you think are the most important
aspects of your experiences in this course
that account for your learning?
4. What have you learned in this course that
will matter to you five years from now?
1.
Student Writing Study: Findings


Students’ responses regarding valuing
learning, connecting classroom learning
with the world, and connecting with other
were fairly similar in both stand-alone
and LC courses.
Students in LC courses were far more
likely to cite applying or appreciating a
variety of perspectives to problem-solving
(55% versus 15%)
Student Writing Study:
Student Comments

“After taking this course I feel that I can
make connections to various things, such
as history, influences, people, and culture.
This course taught me the value of
making connections and things from my
own perspective.”
Student Writing Study:
Student Comments

“By combining course topics you get
the ‘bigger picture’ and are able to
sort of apply what we are learning
better. By applying a subject or topic
to another subject or topic you have
to comprehend what you are
learning and apply it to other things.”
Student Writing Study:
Comparisons

Comparing student responses in the
stand-alone courses with students in
collaborative courses—same courses
with the same instructor(s):


Students in stand-alone courses were less
likely to write about these connections or
about learning, and
tended to focus on personal growth, liking
faculty, etc.
2. Quarterly Surveys




Faculty-developed pen-and-paper survey
Scale from “strongly disagree” to “strongly
agree”
Faculty are provided their course results
and comments, plus the cumulative for the
quarter for all courses
Several years ago, we created a version
that could be scanned
Learning Community
Survey Questions
1. "This learning community has been
a valuable educational
experience."
2. "I believe it was probably more
valuable to have taken these
classes together than it would
have been to take them
separately."
Mean Responses to Value Questions
(N = 5156)
4.1
4
3.9
3.8
3.7
3.6
3.5
3.4
3.3
4.1
3.9
3.8
3.6
Valuable
More Value Together
LC
Comp LC
Linked & Federated Composition LCs
(N = 2780)
4.0
3.8
4.0
3.6
3.4
3.6
3.7
3.4
3.2
3.0
Valuable
Linked
More Value Together
Federated
Student Comments



LC: “This class has been fun and not sucky at
all. I think I have learned a lot.”
Link: “I did not learn too much other than
how to explore more in depth, how to write
more.”
LC: “If they had been separate, I would have
known the what and where, but not the why,
and the why is always the most important
question.”
3. Community College Survey of
Student Engagement



Administered the CCSSE in 2003,
2005, and 2007
Went to this survey because it most
closely matched the institution’s focus
on student learning and engagement
The LC question was a bonus extra!
CCSSE LC Question



Stem: “Which of the following have you
done, are you doing, or do you plan to do
while attending this college?”
Category: “Organized learning communities
(linked courses/study groups led by faculty or
counselors)”
Response Categories:



I have done
I plan to do
I have not done nor plan to do
Method


Used T-test for independent samples to
examine differences in effort and
engagement between students who had
taken Learning Communities and those
who had not done nor planned to do.
Results are only for the latest survey
administration (Spring 2007)
Findings
Prompt: “In your experiences at this college during the current school
year, about how often have you done each of the following?”
Responses range from 1 (“Never”) to 4 (“Very Often”)
Learning Community
Variable
Asked questions in class
or contributed to class
discussions
Made a class
presentation
Have
Taken
(n = 173 )
Not
Taken
(n = 232)
Sig.
3.00
3.00
1.000
2.43
2.27
.069
Learning Community
Variable
Prepared two or more drafts of a
paper or assignment before
turning it in
Worked on a paper or project that
required integrating ideas or
information from various sources
Have Taken Not Taken
(n = 173 ) (n = 232)
2.96
2.45
Sig.
.000
.000
3.14
2.75
Worked with other students on
projects during class
2.88
2.69
.021
Worked with classmates outside
of class to prepare class
assignments
2.49
2.10
.000
2.87
2.69
.038
Discussed ideas from your
readings or classes with others
outside of class (students, family
members, co-workers, etc.)
Learning Community
Variable
Have Taken Not Taken
(n = 173 ) (n = 232)
Sig.
Worked harder than you thought
you could to meet an instructor’s
standards or expectations
2.78
2.52
.003
Used email to communicate with
an instructor
3.10
2.67
.000
Discussed grades or assignments
with an instructor
2.76
2.58
.039
Discussed ideas from your
readings or classes with
instructors outside of class
2.11
1.88
.012
Worked with instructors on
activities other than coursework
1.74
1.45
.000
The advantage of using
multiple methods….

…you get
to do a
“Reality
Check” on
a periodic
basis
What we didn’t have….
Perceptions of students after they
transferred from Skagit
Collaboration with Western
Washington University’s Office of
Survey Research


WELS (Western Educational
Longitudinal Study) Data
Focus Groups
WELS Transfer Study



Process began in 2001; baseline data Fall 2003
Survey items related to students’ expectations,
pre-WWU experiences, college preparedness,
etc.
Three separate surveys of incoming transfer
students administered on-line, with phone
follow-up



Prior to starting classes
After the first quarter
After finishing three quarters
Results for SVC/Non-SVC Transfers
Percent responding
“5 or more times”
Not SVC
SVC
(n = 507 )
(n = 45)
Talked to a professor outside of class
54.6%
66.7%
Personally contributed to a classroom
discussion
76.3%
80.0%
Volunteered on or off campus outside of
coursework
20.1%
33.3%
Participated in a club or activity
16.8%
24.4%
Wrote a substantial scientific or other
research paper of 5 pages or more
17.9%
17.8%
Wrote any other type of substantial paper of
5 pages or more
25.6%
4.4%
Focus Groups


WWU generated list; SVC did transcript
analysis to identify student course
taking, esp. Learning Communities and
English Links
Identified two groups for interviews:
1) a mix of none, one, or two or more LCs
taken at SVC
2) two or more LCs taken at SVC
Questions for all SVC transfers




What aspects of your learning experiences at Skagit Valley
College have helped you in your courses at Western
Washington?
If you could change anything about your learning
experiences at Skagit Valley College so that you would be
better prepared to succeed at Western, what would that
be?
Can you talk a little bit about how Learning Community
courses have helped or hindered your academic progress
at Western?
If you took an English course that was linked with another
course, can you describe how that learning experience has
been useful or not in your studies at Western?
Additional Group Questions

For the first group:


Of you who have taken more than one Learning
Community or English Link, how did taking a
second or third learning experience of this kind
impact the development of your skills or
knowledge?
For the second group:

How did taking the second or third learning
experience of this kind impact the development of
your skills or knowledge?
Helpful in Transition to WWU


Students say the following aspects have
helped: interdisciplinary learning,
analytical/critical thinking, research and
writing, group work, one on one interaction
with instructors, and course content in
specific courses
Additional helpful aspects noted:
completion of GURs, knowing people who
have attended Western, and the financial
benefit of attending SVC
Interdisciplinary Learning
 Students say that the interdisciplinary aspect of
learning communities helps to make classes
more interesting and fun at SVC.
 Exposure to interdisciplinary learning at SVC
helps students to feel more confident in their
classes at Western. Students say they have
learned to make connections and to apply what
they’ve learned to another topic or to the real
world.
English Links


Most students say English links were useful
in their studies at SVC and in transferring to
Western—because English links challenged
students to think and write analytically,
outside of the box.
Students say English was more interesting
because it was linked to a content area
which provided something to write about.
They felt better prepared for researching
and writing analytical papers at Western.
English Links

A few students say their English link
was not useful in their studies at
SVC. They say the focus was more
on grammar and punctuation, as it
was in high school.
Research and Writing


While some students say that Skagit
prepared them to think critically and write
analytical papers (discussed earlier), other
students say they were not prepared for
writing analytical papers at Western.
Research and writing experiences in English
links and learning communities at SVC helped
to prepare students for research and writing
assignments at Western.
Other Student Outcomes

Analytical/critical thinking


The emphasis on critical thinking at SVC helps
students in their classes at Western because they
learned how to analyze information, make
connections, and synthesize major ideas.
Group work

Group work in links and learning communities
helped to prepare students for group work
assignments at Western.
Challenge & Workload

An interesting side note is students’ perceptions
about the challenge and workload involved in
learning communities and links:


Some students say that learning communities and
links are more challenging because of the level of
analyzing involved. For some students, there is also a
greater workload than expected.
Other students say there is not a greater workload.
Faculty Coordination


When instructors do not coordinate well
together, the classes seem separate—not
linked. Students find that the lack of
coordination between some instructors means
lack of integration between courses that are
supposed to be linked.
Learning communities work best for students
when the instructors coordinate well together,
and particularly when instructors are in the
classroom teaching together.
Changes Students Would Make


Students say the biggest issue for them is
academic advising—because advising directly
impacts students’ learning experiences at SVC
as well as their preparation for Western.
Students also say they would finish a DTA to
avoid difficulties in transferring, make sure
they had learning experiences in a large
lecture class, and do something about
disruptive students.
Future WELS Analyses

Comparisons




SVC degree vs. no degree (or ranked by
number of LCs taken)
SVC vs. other CCs
LC comparison between CCs
SVC vs. other CC vs. non-transfer WWU
third-year students
Future WELS Analyses

Surveys

Baseline – administered prior to start of Fall qtr


Transitions – administered at the end of Fall qtr


Focus = first quarter WWU experiences
Spring Follow-Up – administered at the end of third
quarter


Focus = experiences at previous college & expectations at
WWU
Focus = first year experiences at WWU
Analyses

Descriptives and regression
GPA Analysis – LCs Taken
3.5
3
2.5
2
No LC
One LC
Two+ LCs
1.5
1
0.5
0
Entering GPA
QPA after 2 Qtrs
There were no significant differences between groups on either
variable.
GPA Analysis - Degree
3.5
3
2.5
2
No SVC Degree
SVC Degree
1.5
1
0.5
0
Entering GPA
GPA after 2 Qtrs
There were no significant difference between groups on entering
GPA but there was for GPA after two quarters (p=.000)
N = 60
Entering
GPA
GPA
After 2
Qtrs
Sig.
No
3.15
2.54
.000
Yes
3.19
3.14
None
3.32
2.43
One
2.87
2.77
Two or more
3.24
2.98
Degree Completed
LCs Completed
.362
Convergence of the Data

Findings from surveys, focus groups, and
student writing strongly suggest that
taking Learning Communities results in
higher levels of



effort and engagement
collaborate with peers
interaction with faculty
Convergence of the Data


The WWU data about transfer students
also suggest that students who complete
Learning Communities at Skagit or the
degree perform better than those who do
not
However, we may need to look at the
kinds and level of writing students are
required to do in their courses
It’s all about perceptions…
Latest SVC Efforts….

Learning Communities Assessment Project
(Washington Center for the Improvement
of Undergraduate Education)



Protocol developed at Harvard (Project Zero)
Rubric being tested
Developmental Learning Communities
Project

SVC project to research student success based
on developmental course delivery methods,
including counseling-enhanced
Thanks for coming!