Transcript here [.25 MB]

Tracking Transformed Courses:
Impacts of Tutorials,
Instructor, Text, …
SJP Summer '05
Tutorials at CU
• We’ve used “Tutorials” (a research-based
replacement for recitations) in Phys 1110 and
1120 twice each in the last 2 years.
• All four times have been very successful
(data to follow, more if you’re interested!)
=> we are motivated to continue
• Following is some evidence to make the case
that we should support this teaching
innovation.
Resources
• Tutorials carry some space, material,
and personnel needs. The latter are
what need ongoing support
• Need “Learning Assistants” (LA’s) - we
hire undergrad STEM majors interested
in teaching, who have excelled in the
course previously.
Courses studied
Recitation
Homework
Text:
Tutorials
Tutorials
CAPA
CAPA
HRW
Knight
Tutorials
Tutorials
Knight workbook
Trad recitations
CAPA
CAPA
MP
MP
• Phys 1120
– Fa 04
– Sp 05
• Phys 1110
–
–
–
–
Fa 03
Sp 04
Fa 04
Sp 05
HRW
HRW
Knight
Knight
Phys 1120:
Call them Tut1 (Fa04) and Tut2 (Sp05)
Attempt at replication.
Main differences: Tut2 has…
• different instructor
• different semester
• follows up an 1110 taught without Tutorials
• different textbook
• no long answer on CAPA or exams
Summary (up front!)
• Despite many other changes in course
elements, we replicated Tutorials and
basic course structure.
• Result: nearly identical learning gains,
very high by national standards. (The
final score matches what our junior
physics majors get on this hard exam!)
1120 BEMA pretest
(start of term, before instruction)
% of students
BEMA (matched) (CU scoring) Fa04
20
18
16
14
12
10
8
6
4
2
0
0
6
12
18
24
30
36
42
48
55
61
67
73
79
85
91
97
Score (%) (CU scoring)
PreF04
BEMA = “Brief E&M Assessment”, validated research-based
survey of Conceptual elements of E&M (kind of like the “FCI”)
Blue data above is F04 (N=319) Pretest ave 26%
1120 BEMA pretest
(comparing different semesters)
% of students
BEMA (matched) (CU scoring)
Compare Fa04 and Sp05
20
18
16
14
12
10
8
6
4
2
0
0
6
12
18
24
30
36
42
48
55
61
67
73
79
85
91
97
Score (%) (CU scoring)
PreF04
PreS05
F04 (N=319) Pretest ave 26%
S05 (N=232): 27%
The two semester’s pretests match closely - no obvious
different in starting population’s content knowledge.
1120 BEMA post
(how did they do at end of course?)
% of students
BEMA (matched) (CU scoring)
Compare Fa04 and Sp05
20
18
16
14
12
10
8
6
4
2
0
0
6
12
18
24
30
36
42
48
55
61
67
73
79
85
91
97
Score (%) (CU scoring)
PostF04
F04 (N=319) 26% -> 59%,
g(ave, F04) = .44+/- .01
PostS05
S05 (N=232) 27% -> 59%
g(ave, S05)=.43+/- .01
Posttest results are identical (an impressive replication)
High by nat’l standards (typical trad courses, post score = 30-40% !)
1120 mini-summary
Despite different instructors, text, exam
structures, semesters…
Two tutorial implementations: no sig diff
on BEMA, CLASS (an attitude/beliefs survey), or
common exam questions overall.
Final scores on nationally validated survey very
high in both classes.
Overall neutral results on surveys of student
satisfaction. (No disasters, but still room to
improve)
1120 “replication study” shows that
Tutorials are effective, and can be
transferred between instructors.
Next: 1110 study, comparing Tutorials with
other forms of recitations…
Phys 1110: Distinguishing features
• 1: “Tut” (Fa03/Sp04) Tutorials + CAPA
• 2: “Workbook” (Fa04) No Tutorials, but used
small groups/Knight workbook for 1st half of
semester (also Mastering Physics, a fancier
hw system)
• 3: “(More) Trad” (Sp05) Reverted back to
mostly traditional recit’s (Otherwise, same
text and hw as #2, same course structure,
lead instructor had team taught it in Fa04)
(different instructors, semesters …)
1110 summary - up front!
• “Tutorial-based” courses consistently show
stronger results on all measures of learning
gains and attitude surveys
• Middle course (“2”) used research-based text
+ hw system, clickers, small-group interactive
recitations (but not Tutorials). It fared well, but
consistently lower on all measures. (Despite
significantly stronger conceptual pretest)
• Last course (“3”) replicated “2” with one major
change: traditional recitations. Learning and
attitude shifts are worst for this course.
Pre/post FMCE (Sp04)
# of students
60
Pre
50
Post
40
30
20
10
0
0
12
24
36 48
61
Score (%)
73
85
97
Phys 1110 normalized learning gains
(on nationally validated exam “FMCE”)
dis tribution of g: c ompa re d
Tutorials
30
% of students
25
20
SP04
Middle
ground
FA04
SP05
15
10
Trad rec.
5
0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
normalized gain
gain(1)= .66 +/-.02 g(2)= .585 +/-.02 g(3)= .45 +/-.02
Course (1) with Tutorials has highest gains.
Course (2) is 8 points (several sigma’s) worse
Course (3) (trad recit.) => significantly lower gains. (but still, double nat’l standards!)
Impact on different pretest populations:
"low starters" pretest <=12.5%
nor malized gain for low pretest
0.6
0.5
Tutorials
Trad rec.
g
0.4
0.3
0.2
0.1
0
S04 (2 3%)
F04 (20%)
S05 (2 2%)
Semester (%
of c
lass in
(% of
class
in this
this
pool)
prete st <= 12.5%
Course
(1)
(2)
(3)
(Tutorial course (1) is significantly more effective for “at risk”
students than traditional-recitation course (3))
Impact on different pretest populations:
"high starters" 50<pre<93%
normaliz ed gain for high pretest
0.9
0.8
Tutorials
0.7
Trad rec.
0.6
g
0.5
0.4
0.3
0.2
0.1
0
S04 (13 %)
F0 4 (22%)
S05 (14 %)
Semester(%
(%
clas sininthis
thispool)
ofofclass
pretes t >50%
Course
(1)
(2)
(3)
=> Even the best students at the start (high pretest) benefit more
from Tutorials (or group activities) than from the trad recitation
1110 Summary
• Compare Tut-based with “workbook/small group”
recitation (1 to 2)
 measurable diffs in FMCE, common exam q’s,
CLASS
• Compare Tut-based with “more trad rec” (1 to 3),
 significant diffs in FMCE, exams, and CLASS
• Tutorials are only one effect. (Instructors, course
structure surely play a big role)
• But in 1120, changing instructors (and text) did NOT
harm the course when the Tutorials were retained.
Bottom line
• Not a controlled experiment (!) but data show Tutorials
are a successful, valuable, and productive course
element.
• They come at a cost (and benefit)
• Cost: $1500/LA/semester * (6-8 LA’s per course)
• Need one hr/week training session (TA’s too!)
• Add’tal benefits: Cadre of undergrads becoming LA’s,
some moving on to K-12 certification. (+ their learning
gains after LA’ing are through the roof)
• Grad TA’s exposed to research based learning env’t.
Recommendation
• We should continue implementing
Tutorials (and collecting this kind of
data)
• To run Tutorials, we need support for
LA’s, and the training infrastructure that
goes along (=> a faculty or experienced
grad student assigned to teach the
TA’s/LA’s)