The Catalyst Initiative: Using Assessment to Drive Innovation in Teaching & learning with Technology Tom Lewis, Director Ed-Tech Development Group Educational Partnerships & Learning Technologies Scott Macklin,
Download ReportTranscript The Catalyst Initiative: Using Assessment to Drive Innovation in Teaching & learning with Technology Tom Lewis, Director Ed-Tech Development Group Educational Partnerships & Learning Technologies Scott Macklin,
The Catalyst Initiative: Using Assessment to Drive Innovation in Teaching & learning with Technology Tom Lewis, Director Ed-Tech Development Group Educational Partnerships & Learning Technologies Scott Macklin, Director Program for Educational Transformation through Technology
How do you scale beyond the early adopters to reach wary adopters and encourage innovation?
Scaling Innovation with the 4-Tiered Catalyst Initiative
Lesson Learned:
Scale support through partnerships, making disparate resources and innovative practices visible and available to all.
http://catalyst.washington.edu
The Catalyst Initiative On an average day in Autumn 2002, instructors viewed 5000 pages of Catalyst content every day.
8557 instructors have created 18,592 implementations of Catalyst Web Tools.
Catalyst Web Tools were used for 66,000 learning activities during Autumn 2002.
Catalyst Web Site
Catalyst Profiles . . . Stories of Innovation
Teaching Guides start with your teaching goals
Action Plans help you use technology
The How-To section has detailed instructions for specific tasks
The learning section provides information about workshops and clinics
Participatory Design & Catalyst Web Tools
Lesson Learned:
Create tools that meet pedagogical needs expressed by faculty and constantly engage with faculty to make sure the tools remain current and easy to use.
Collaborative Development Assessing what the educator wants is central:
• Get feedback at every step of design. • Process is more important than
product.
• Direct communication between
developers and educator.
• Make feedback and interaction easy. • Value educator’s ideas and time.
SimpleSite
• Create Course
Web pages without HTML or an editor
• Publish Web
pages without FTP
• Keeps usage
statistics for you
WebQ
• Create online
surveys, quizzes & questionnaires
• Download
data, or look at it on the Web
• Many security
options for password protecting your data
• Approved by
UW Human Subjects Division
Peer Review
• • • • •
Create a forum for to review each others’ documents Encourage group projects with the new group features Supports any kind of Web content: (text, graphics, sounds, movies) Comment on sentences, paragraphs or entire document Complete participant reports accessible to instructor
UMail
• An anonymous
email feedback tool
• Messages
come to your regular email inbox
E-Submit
• Students can
turn in homework online
• Saves time and
frustration with email attachments
• Students receive
a digital “receipt”
EPost
• Create an
online discussion board
• Many sort
options: by thread, subject, author, and date
• Generates
“participation reports”
• • • • •
QuickPoll Create 1-question surveys for your Web site Results tabulated automatically Numeric and graphic displays Many submission controls available Can be linked to EPost
Partnerships as a support strategy to bring easy to use, flexible & campus-wide support.
Lesson Learned:
Make sure that all the units that should be shaping technology innovation are shaping innovation.
Portfolio
NEW!
Resources
• Catalyst group (under Vice-Provost of
Educational Partnerships & Learning Technologies) supplied main funding
• Student Technology Fee committee gave
financial support.
• Program for Educational Transformation
through Technology provided financial and assessment support.
• Computing & Communications furnished
server space and maintenance
• Office of Undergraduate Education offered
the Freshman Interest Group program as a test-bed
Project Assumptions
• Tool cannot be built without input from
students, instructors, and academic and career advisers
• Tool should be flexible—students should be
able to use it for classes, extra-curricular activities, job searches, or personal reflection
• Tool should be useful for students in all
colleges and schools
How did we build it?
• • • •
Asked everyone on campus who has ever uttered, or is likely to utter the word “Portfolio”: 200 + Looked at other universities’ tools and commercial products Read scholarly articles on the uses of portfolios in education Conducted focus groups and usability tests with UW instructors and students
Basic Decisions:
• Students should be able to collect anything
they want to and have exclusive access to those collections
• Students should be able to publish many
different portfolios of their work on the Web
• Instructors and advisors should be able to
easily lead students through the process of building certain types of portfolios, but students still own these portfolios
What is a portfolio tool?
• There is a distinction between collecting
artifacts, reflecting upon them, and thoughtfully presenting them in a portfolio
• Students will be collecting artifacts
throughout their academic careers and should have exclusive access to their collections
• Using portfolios for reflection is a process
that is important in all phases of an academic career, not just at the end of senior year
What is a portfolio tool (cont.)?
• Instructors, advisers, and peers should
collaborate with students to help them present artifacts in a reflective way
• Students should be able to present a variety
of portfolios for different objectives: completing coursework or demonstrating academic, personal, or career development
Instructors need help in effectively integrating technologies into teaching.
Lesson learned: Need research and assessment to inform the development of learning technologies and teching strategies.
The process 1. which in turn inform the design of appropriate tools.
2. These new tools are integrated into educational settings, 3. in which new studies are conducted to understand how these new tools have affected learning and learners.
4. Perform studies in real settings to investigate what is learned, who learns, and how it happens,
The FIG Program and Portfolios:
• Pilot project in Autumn 2001 with three
Freshman Interest Group (FIG) sections (~60 students).
• Development of the Catalyst Portfolio
Tool in 2001-2002.
• Introduction of the tool into the FIG
program in Autumn 2002, with a study by PETTT researchers.
Our Goals for Autumn 2002:
• Introduce the concept of portfolios to
3,200 freshman students.
• Within the goals of the existing University
Community class, encourage reflective learning.
• Learn lessons that can guide future
portfolio projects and inform future revisions of the tool.
What was our experiment?
Collect data on the use of the tool by FIG leaders and students through:
• Surveys • Focus groups • Classroom observation • Examination of student portfolios
Study design:
• 162 Freshman Interest Groups • 162 FIG leaders, each with a class of 10-
25 students
• Special group: 20 FIG leaders, paid to
give regular, individualized feedback to students
Study questions: 1) What method(s) of instruction help students feel most competent using Portfolio?
2) How can Portfolio be best integrated into the GENST 199 curriculum?
3) What instructional supports (in-class discussion, feedback, prompts within tool) might best support reflective thinking?
Data collection: 1) First day classroom observations 2) Student surveys 3) FIG leader surveys 4) Focus groups with students and FIG leaders 5) Classroom observations & student interviews 6) Review of student portfolios
Three themes from preliminary analysis:
• Students found Portfolio easy and convenient to
use. However, they largely perceived answering the questions as busywork.
• FIG leaders' attitude toward the tool and the work
affected students' response to the tool and to class.
• Feedback was important--Students wanted to
know that someone was reading what they wrote. These are issues to consider as we head towards a redesign project.
and make recommendation for how to use Portfolio. NOT reasons to abandon the
Quotes… “I feel it’s really fun to me, I mean maybe it’s weird but I think it’s almost like a penpal letter kind of to our FIG leader and we can answer the questions and sometimes they’re really in-depth and we actually have to think about what we think about life and on these certain subjects, we have to think about what our beliefs are, and I think that’s helped me a lot to really figure out what I believe and why I believe it, and apply that to the discussion that we’ve had in class and stuff.”
-FIG Student
“It’s nice when they return it and they add comments so you know that they’re actually looking at it.” (student)
-FIG Student
“If the FIG leaders did give feedback… it does kind of make it more interesting and you do feel like your FIG leader is paying a little bit of attention to you. It might even create discussions that you could have in your FIG class”.
-FIG Student
“[Portfolio and the feedback function] helps you know more about your students, share and get ideas”.
-FIG leader
“Because I comment so much, they [students] see me as a friend and come talk to me. I learned a lot from them.”
-FIG leader
Building Catalyst through Co-Branding
Lesson Learned:
Renew and refresh anytime-anywhere technology support resources by co branding and co-developing them with campus teaching practitioners.
Students learn through portfolios when they…
• assemble a collection of relevant artifacts • select items from that collection for a
specific purpose
• reflect on the selected items and/or
process of selection
• design the presentation of their portfolio
for a particular audience/purpose
• receive feedback on the quality of their
selection, reflection, and/or design
Considerations for teaching with portfolios…
• specific purpose/audience • assignments to produce relevant artifacts • guidelines for item selection • guidelines for and examples of “good”
reflection
• guidelines for and examples of “good”
design
• time for feedback on quality of selection,
reflection, and/or design
More considerations for teaching… Evaluation
• individual items? • whole? • reflection? • establish clear criteria & SHARE with
students ahead of time.
• will the evaluation count toward the grade? • if students don’t have opportunity to learn
from feedback, a grade isn’t a measure of what they learned—only innate ability.
Extending Innovation through Assessment
Lesson Learned:
Reach out to instructors and students in as many ways and as often as possible.
PETTT Educational Technology Surveys Faculty, Spring 2001 2000 respondents Students, Fall 2002 2000 respondents
Technology Adoption: Student Driven
•20% of college students began using a
computer between 5 and 8 years old.
•By the time they are 16-18, all had used
a computer.
•80% of UW undergraduates use a
computer daily.
Students…
•72% of students think all courses
should have a Web site
•Students particularly want online:
course syllabi (81%)
grading criteria (79%)
problem sets or exercises (78.6%)
links to electronic course reserves (72%)
•7% of students stated that educational
technology is used consistently through UW
“I’m graduating now and I’m pretty upset with how little I know about technology, I really wish that I was more confident and knew more after four years of my education at a university. A lot of it is probably independent work that you can do to learn it but we’re busy enough with jobs and school that I just really haven’t felt like I had the time to actually independently learn it…it would have been really helpful to have not been expected to know it but to have actually been taught it.”
-Undergraduate Student
Which of the following concern faculty when using information technology to do university work?
60 50 40 30 20 10 0 La ck of Ti me La ck of Sk ills La ck of In ce nti ve Stu de nt lea rni ng Ha rdw are
Top 5 desired faculty uses for the Web Percent 90 80 70 60 50 40 30 20 37 33 21 41 35 24 21 33 10 0 Mo un t c ou rse m ate ria Ac ls ce ss re se Co rve nd Ac s uc ce t re ss se m arc ate h ria ls arc Su hiv bm es it f ina l g rad es 3 50 Already use Very much Somewhat
Faculty preferred methods for learning 80 60 % selecting 40 20 0 73.4
67.1
Colleagues, friends Local tech support Books, journals, tutorials 51 46.5
38.6
Exploring, experimenting Workshops
www.catalyst.washington.edu
[email protected]
Tom Lewis Scott Macklin << [email protected]>> <