Transcript Calibrated Peer Review (CPRTM)

Calibrated Peer Review (CPR)TM
at Texas A&M University
by Dr. Wendy Keeney-Kennicutt
Associate Director, First Year Chemistry Program
CPR Master Administrator, TAMU
[email protected]
HISTORY AT TAMU
cpr.tamu.edu

2002 - CPRTM was introduced to First Year
Chemistry in an NSF-sponsored MultiInitiative Dissemination spring workshop and a
CPR summer workshop. We started in the fall.

2003 - To avoid FERPA* issues & because of a
joint NSF CPR grant with UCLA and our Center
for Teaching Excellence, CPR was housed on a
secure TAMU server.

TM
2003 - I volunteered to be the CPR master
administrator because of experience.
*Family Educational Rights and Privacy Act (1974) – protects the privacy of
student education records
USE OF CPRTM AT TAMU
the last 3 years, the use of CPRTM at
TAMU been used by approximately
 In





10,000 undergraduate and graduate
students doing
320 new assignments in
200 courses spread over
30 majors in
7 colleges.
MAJORS USING CPRTM










Accounting
Ag. Economics
Animal Science
Archeology
Biochemistry
Biology
Botany
Bus. Admin.
Chemistry
English










Ed. Psychology
Film
French
Kinesiology
Learn. Comm.
Math
Microbiology
Nutrition
Physics
Poultry Science
 Psychology
 Secondary Ed.
 Vet Integr. Bio. Sci.
 Wildlife & Fish. Sci.
 Zoology
MISSING:
Engineering
•Starting F06
Geosciences
How Did CPRTM Become
So Popular?
(1) Creation of “W” Courses

Our Faculty Senate decided that for graduation, there
would be a writing-intensive (W) discipline-specific
course graduation requirement.
 The Core Curriculum Review Committee report entitled
"Educational Leadership at the Beginning of the 21st
Century" (March 28, 2000, as amended and approved by
the Faculty Senate, May 8, 2000), established this
requirement. The first writing-intensive course graduation
requirement went into effect in the Fall 2004 catalog. The
requirement for a second course is scheduled to be in
place in the Fall 2006 catalog.
How Did CPRTM Become
So Popular?
(2) CPR was recognized as good pedagogy
campus-wide

I co-ran a CPR workshop for TAMU and local community
college faculty, supported by TAMU College of Science
mini-grant in Summer 2003 and invited everyone.

The Director of the TAMU Writing Center participated,
recognized its effective pedagogy and gave CPR her
blessing for use in “W” courses.

By Fall 2003, we had CPRTM on our server, TAMU had
quit using SSN as SIDs and we were set to go.
How Did CPRTM Become
So Popular?
(3) Continued Support for our TAMU users:





A good website: cpr.tamu.edu with extended FAQs
Regular workshops supported by WALS*, Center for
Teaching Excellence (CTE), Information Technology
Services, The Writing Center, and the Office of the VicePresident for Information Technology
A local users group which has started to meet regularly
2 months summer salary for the TAMU CPR master
administrator; two other administrators are paid through
their position at CTE
Access to the actual program to solve problems
*WALS = Writing for Assessment and Learning
in the Natural and Mathematical Sciences (the UCLA/TAMU NSF project)
What are the issues?

TAMU has its own CPR webpage, so students
occasionally log on to the wrong one.

TAMU is disconnected from the UCLA server, so we
don’t have access to the library assignments. This will
be changing under the upgraded system.

Note: We have found that there is more student buyin when instructors write their own assignments:

The assignments reflect the instructor’s wording and tone

CPR fits into the course flow better

The instructor is much more familiar with the program after
they write assignments.
So what happened to CPR in
Chemistry?





In Fall 2002 and Spring 2003, we incorporated CPR in all
the Chemistry 101/102 classes
~ 2600 students per semester
Individual instructors (besides myself) had minimal
training in CPR and were not involved
Assignments were taken directly from the library
 4 assignments, dropping 1 to allow for absences
 I and a staff member managed the problems list, all
the grades, and student questions
So what do you think was the outcome? I kept track
using SALG.
Student Assessment of Learning Gains
www.wcer.wisc.edu/salgains/instructor/
First Year’s Results
from My Class
Percent of Students Who
Agree
SALG* Results on How Students View CPR
(*Student Assessment of Learning Gains)
50
Term - Chem Class
40
30
F'02-102
20
S'03-101
10
0
CPR is
Enjoyable
Helps us
Learn
Chemistry
Helps
Improve
Writing
Helps
Improve
Critiquing
Others
Should Do
CPR
This was not good…… Everyone but me stopped doing CPR.
I knew that scientific writing/critiquing were invaluable skills, so I
kept trying new approaches to improve student attitude.
Overview of Study
(manuscript in preparation)

I collected SALG data for 7 semesters from my students
(1515 total; 94-98% of classes participated), as I continued to
make CPR more palatable
 Likert scale (1=strongly disagree…….5=strongly agree)
•
•
•
•

I enjoyed doing the CPR assignments
The CPR assignments helped me learn some chemistry
The CPR assignments helped me improve my writing skills
The CPR assignments helped me learn to critique my own
writing and that of others
Yes/No + explanation
• Do you think that future classes should do CPR?

I had both quantitative and qualitative data for analysis
Improvements Made Over Time










Prepared more thorough instructions and a supportive website
Wrote most of my own assignments
Became more proactive at listening to students & adjusting grades
when appropriate
Told my students upfront in the syllabus that the class was a
writing-intensive class and included my teaching philosophy
Gave a “CPR lab holiday” – since I did 7 labs rather than 10
Invited students to let me review their essays before submission
Increased importance of the text entry from 20% to 30%
Increased CPR’s worth from 3-5% to 12% of class grade
Took classroom time to demonstrate CPR
Used Bloom’s Taxonomy to show importance of critiquing
Why is writing
important? Evaluation
Combination of
information to form a
unique
product; requires
creativity and originality
Use of information
to solve problems;
transfer of abstract
or theoretical ideas
to practical
situations.
Restatement in your
own words;
paraphrase; summary
Synthesis
Analysis
Application
Interpretation
Translation
Bloom’s Taxonomy – categorizing
level of abstraction of questions
Recall
Judgment: the ability to
make decisions and
support
views; requires
understanding of values
Identification of
component parts;
determination of
arrangement,
logic, semantics
Identification of
connections and
relationships
Verbatim information;
memorization with no
evidence of
understanding
Improvements Made Over Time












Prepared more thorough instructions and a supportive website
Wrote most of my own assignments
Became more proactive & adjusted grades when appropriate
Told my students upfront in the syllabus that the class was a
writing-intensive class and included my teaching philosophy
Gave a “CPR lab holiday” – since my class did 7 labs, not 10
Invited students to let me review their essays before submission
Increased importance of the text entry from 20% to 30%
Increased CPR’s worth from 5% to 12% of class grade
Related that CPR is a grade in its own right & helps poor test-takers
Took classroom time to demonstrate CPR
Used Bloom’s Taxonomy to show importance of critiquing
Emphasized that most students are novice reviewers and that I
would gladly look at anyone’s grade
Quantitative Results
Percent of Students Who
Agree
SALG* Results on How Students View CPR
(*Student Assessment of Learning Gains)
F'02-102
S'03-101
S'04-101
F'04-102
S'05-101
F'05-102
S'06-101
80
70
60
50
40
30
20
10
0
CPR is Enjoyable
Helps us Learn
Chemistry
Helps Improve Writing Helps Improve Critiquing Others Should Do CPR
Survey Questions
What I saw was a significant increase in student acceptance
and understanding of CPR over time.
Percent of Students
Relationship Over Time between Student
Experience with CPR and Their Promotion of CPR
for Future Classes
60.0
50.0
40.0
30.0
20.0
10.0
0.0
Group 1 (%)
Group 2 (%)
Group 3 (%)
Group 4 (%)
Fall Spring Spring Fall Spring Fall Spring
2002 2003 2004 2004 2005 2005 2006
Semester
Group 1:
Group 2:
Group 3:
Group 4:
Students with negative CPR experience, wanted future classes to do CPR
Students with negative CPR experience, did not want others to do CPR
Students with positive CPR experience, wanted others to do CPR
Students with positive CPR experience, did not want others to do CPR
Qualitative Results
“Do you think that future classes should do CPR? Explain.”

Over 7 semesters, there were






550 totally positive responses
515 totally negative responses
174 mixed responses
25 neutral responses
Total: 1264 responses
The qualitative part of this study gave invaluable
insight into student attitude about CPR and how
it changed as I made changes in presentation,
student support and grade intervention.
Qualitative Results – Negative Comments

On writing in a chemistry class:


On the peer review process:


“I have never viewed chemistry as being a subject where you
write things;” “We could take English to learn how to write
correctly;” “I didn’t understand why writing a paper and
grading other students papers had anything to do with
chemistry.”
“They ask you to grade the essays, but then your opinion of
how that person did would be wrong. I just don’t see how
your opinion could be wrong.”
Other:

Too time consuming, waste of time, not related to the subject;
it harmed their grade; was worse than lab; their peers lacked
motivation; added to stress
Qualitative Results – Positive Comments

On writing in a chemistry class:



“Calibrated Peer Reviews forces the student to look into the
topic way more than what he or she would do out of a
textbook. I know the CPR has tremendously helped me
understand each topic better although I didn’t exactly enjoy it.”
“The CPR really helped me understand the topics. It
reinforced the material by forcing me to teach myself and
explain it to others through writing. It was very helpful.”
“I think the first one is bad because you don’t really know
what you’re doing and how to approach the whole thing, but
after doing it you realize that you are learning the subject
because you had to write a paragraph on it. It was a big help
whether people will admit it or not.”
Qualitative Results – More Positive Comments

Overall:



“Although CPR was one of my least favorite things to do in
this class, I think the good in it outweighs the bad. I think that
especially in the science fields, students don’t have to do a lot
of writing and so they don’t develop communication skills that
they will need later on in life. I think communication is very
important and it is something that you just have to work on. I
think students will look back and wish they would have done
more stuff like CPR.”
“It seems like a pain at the time, but I can already see how
much I learned from it. Please continue to do it, it helps more
than people realize.”
Other positives:

Helped link chemistry to real life and their professional future,
developed time management and research skills
Qualitative Results – Overview
Percentage (%)
Percentage of Positive Comments per Semester
70.0
60.0
50.0
40.0
30.0
20.0
10.0
0.0
Fall 2002
Spring
2003
Spring
2004
Fall 2004
Semester
Spring
2005
Fall 2005
Spring
2006
Final Comments

CPR is an example of the move from teachercentered learning to student-centered learning
 If you add writing, peer review and technology to
the mix, should I have been surprised at the level
of student angst?
 Students have a more positive experience when
the instructor





actively promotes & demonstrates CPR in the classroom,
makes the assignments a significant part of grade,
is involved in writing assignments,
gives personalized support
regrades when warranted
Thanks to my co-authors:
Dr. Nancy Simpson,
Director,
Center for Teaching Excellence
Ms. A. Baris Gunersel,
Ph.D. Student, Educational Psychology
Graduate Research Assistant
Center for Teaching Excellence