Transcript The Catalyst E-Portfolio: Using Assessment to Shape Enterprise-Wide, Learner-Centered Technologies Tom Lewis, Director, Ed-Tech Development Group Educational Partnerships & Learning Technologies Scott Macklin, Director Program for Educational Transformation through.

The Catalyst E-Portfolio:
Using Assessment to Shape
Enterprise-Wide, Learner-Centered
Technologies
Tom Lewis, Director,
Ed-Tech Development Group
Educational Partnerships & Learning Technologies
Scott Macklin, Director
Program for Educational Transformation
through Technology
How do you scale beyond
the early adopters to
reach wary adopters and
encourage innovation?
PETTT Educational Technology Surveys
Faculty, Spring 2001
2000 respondents
Students, Fall 2002
1000 respondents
Technology Adoption: Student Driven
•20% of college students began using a
computer between 5 and 8 years old.
•By the time they are 16-18, all had used
a computer.
•80% of UW undergraduates use a
computer daily.
Students…
•72% of students think all courses
should have a Web site
•Students particularly want online:
course syllabi (81%)
grading criteria (79%)
problem sets or exercises (78.6%)
links to electronic course reserves
(72%)
•7% of students stated that educational
technology is used consistently through
UW
“I’m graduating now and I’m pretty upset
with how little I know about technology, I
really wish that I was more confident and
knew more after four years of my education at
a university. A lot of it is probably
independent work that you can do to learn it
but we’re busy enough with jobs and school
that I just really haven’t felt like I had the
time to actually independently learn it…it
would have been really helpful to have not
been expected to know it but to have actually
been taught it.”
-Undergraduate Student
Which of the following concern faculty when
using information technology to do university
work?
60
40
30
20
10
of
La
ck
dw
ar
e
Ha
r
ni
ng
ea
r
St
ud
en
tl
nt
ive
In
ce
Sk
ill
of
La
ck
of
Ti
m
s
e
0
La
ck
Percent
50
Scaling Innovation with the
4-Tiered Catalyst Initiative
Lesson Learned:
Scale support through
partnerships, making
disparate resources and
innovative practices
visible and available to all.
http://catalyst.washington.edu
The Catalyst Initiative
On an average day in Autumn 2002,
instructors viewed 5000 pages of
Catalyst content every day.
8557 instructors have created 18,592
implementations of Catalyst Web Tools.
Catalyst Web Tools were used for 66,000
learning activities during Autumn 2002.
Catalyst Web
Site
Participatory Design &
Catalyst Web Tools
Lesson Learned:
Create tools that meet pedagogical
needs expressed by faculty and
constantly engage with faculty to
make sure the tools remain current
and easy to use.
Collaborative Development
Assessing what the educator wants is
central:
• Get feedback at every step of design.
• Process is more important than
product.
• Direct communication between
developers and educator.
• Make feedback and interaction easy.
• Value educator’s ideas and time.
SimpleSite
• Create Course
Web pages
without HTML or
an editor
• Publish Web
pages without
FTP
• Keeps usage
statistics for you
VirtualCase
• For case
studies and
problem-based
learning
activities
• Contains a
message board
and
collaborative
file sharing
• Allows for
group decision
making
Partnerships as a support strategy
to bring easy to use, flexible &
campus-wide support.
Lesson Learned:
Make sure that all the units that
should be shaping technology
innovation are shaping innovation.
Portfolio NEW!
Resources
• Catalyst group (under Vice-Provost of
Educational Partnerships & Learning
Technologies) supplied main funding
• Student Technology Fee committee gave
financial support.
• Program for Educational Transformation
through Technology provided financial and
assessment support.
• Computing & Communications furnished
server space and maintenance
• Office of Undergraduate Education offered
the Freshman Interest Group program as a
test-bed
Project Assumptions
• Tool cannot be built without input from
students, instructors, and academic and
career advisers
• Tool should be flexible—students should be
able to use it for classes, extra-curricular
activities, job searches, or personal
reflection
• Tool should be useful for students in all
colleges and schools
How did we build it?
•
•
•
•
Asked everyone on campus who has ever
uttered, or is likely to utter the word
“Portfolio”: 200 +
Looked at other universities’ tools and
commercial products
Read scholarly articles on the uses of
portfolios in education
Conducted focus groups and usability tests
with UW instructors and students
Basic Decisions:
• Students should be able to collect anything
they want to and have exclusive access to
those collections
• Students should be able to publish many
different portfolios of their work on the Web
• Instructors and advisors should be able to
easily lead students through the process of
building certain types of portfolios, but
students still own these portfolios
What is a portfolio tool?
• There is a distinction between collecting
artifacts, reflecting upon them, and
thoughtfully presenting them in a portfolio
• Students will be collecting artifacts
throughout their academic careers and
should have exclusive access to their
collections
• Using portfolios for reflection is a process
that is important in all phases of an
academic career, not just at the end of
senior year
What is a portfolio tool (cont.)?
• Instructors, advisers, and peers should
collaborate with students to help them
present artifacts in a reflective way
• Students should be able to present a variety
of portfolios for different objectives:
completing coursework or demonstrating
academic, personal, or career development
Instructors need help in
effectively integrating
technologies into
teaching.
Lesson learned:
Need research and assessment to
inform the development of learning
technologies and teaching strategies.
The process
1.
2.
3.
4.
Perform studies in real
settings to investigate what
is learned, who learns, and
how it happens,
which in turn inform the
design of appropriate tools.
These new tools are
integrated into educational
settings,
in which new studies are
conducted to understand
how these new tools have
affected learning and
learners.
The FIG Program and Portfolios:
• Pilot project in Autumn 2001 with three
Freshman Interest Group (FIG) sections
(~60 students).
• Development of the Catalyst Portfolio
Tool in 2001-2002.
• Introduction of the tool into the FIG
program in Autumn 2002, with a study by
PETTT researchers.
Our Goals for Autumn 2002:
• Introduce the concept of portfolios to
3,200 freshman students.
• Within the goals of the existing University
Community class, encourage reflective
learning.
• Learn lessons that can guide future
portfolio projects and inform future
revisions of the tool.
What was our experiment?
Collect data on the use of the tool by FIG
leaders and students through:
• Surveys
• Focus groups
• Classroom observation
• Examination of student portfolios
Study design:
• 162 Freshman Interest Groups
• 162 FIG leaders, each with a class of 1025 students
• Special group: 20 FIG leaders, paid to
give regular, individualized feedback to
students
Study questions:
1) What method(s) of instruction help
students feel most competent using
Portfolio?
2) How can Portfolio be best integrated
into the GENST 199 curriculum?
3) What instructional supports (in-class
discussion, feedback, prompts within
tool) might best support reflective
thinking?
Data collection:
1)
2)
3)
4)
First day classroom observations
Student surveys
FIG leader surveys
Focus groups with students and FIG
leaders
5) Classroom observations & student
interviews
6) Review of student portfolios
Three themes from preliminary analysis:
• Students found Portfolio easy and convenient to
use. However, they largely perceived answering
the questions as busywork.
• FIG leaders' attitude toward the tool and the work
affected students' response to the tool and to
class.
• Feedback was important--Students wanted to
know that someone was reading what they wrote.
These are issues to consider as we head towards a
redesign and make recommendation for how to
use Portfolio. NOT reasons to abandon the
project.
Quotes…
“I feel it’s really fun to me, I mean maybe
it’s weird but I think it’s almost like a
penpal letter kind of to our FIG leader
and we can answer the questions and
sometimes they’re really in-depth and we
actually have to think about what we
think about life and on these certain
subjects, we have to think about what
our beliefs are, and I think that’s helped
me a lot to really figure out what I
believe and why I believe it, and apply
that to the discussion that we’ve had in
class and stuff.”
-FIG Student
“It’s nice when they return it and they
add comments so you know that they’re
actually looking at it.” (student)
-FIG Student
“If the FIG leaders did give feedback… it
does kind of make it more interesting and
you do feel like your FIG leader is paying
a little bit of attention to you. It might
even create discussions that you could
have in your FIG class”.
-FIG Student
“[Portfolio and the feedback function]
helps you know more about your
students, share and get ideas”.
-FIG leader
“Because I comment so much, they
[students] see me as a friend and come
talk to me. I learned a lot from them.”
-FIG leader
Building Catalyst through
Co-Branding
Lesson Learned:
Renew and refresh anytime-anywhere
technology support resources by cobranding and co-developing them
with campus teaching practitioners.
Students learn through portfolios
when they…
• assemble a collection of relevant artifacts
• select items from that collection for a
specific purpose
• reflect on the selected items and/or
process of selection
• design the presentation of their portfolio
for a particular audience/purpose
• receive feedback on the quality of their
selection, reflection, and/or design
Considerations for teaching with
portfolios…
•
•
•
•
specific purpose/audience
assignments to produce relevant artifacts
guidelines for item selection
guidelines for and examples of “good”
reflection
• guidelines for and examples of “good”
design
• time for feedback on quality of selection,
reflection, and/or design
More considerations for teaching…
Evaluation
• individual items?
• whole?
• reflection?
• establish clear criteria & SHARE with
students ahead of time.
• will the evaluation count toward the grade?
• if students don’t have opportunity to learn
from feedback, a grade isn’t a measure of
what they learned—only innate ability.
http://catalyst.washington.edu
Tom Lewis
Scott Macklin
<<[email protected]>>
<<[email protected]>>
http://depts.washington.edu/pettt