Transcript Document

Towards Collaborative
Learning @ Scale
Marti A. Hearst
UC Berkeley
Joint work with Bjorn Hartmann, Armando Fox, Derrick Coetzee, Taek Lim
Sponsored in part by a Google Social Interactions Grant
20 million minds foundation
MOOC Drawbacks
 Retention
 Learning (?)
 Isolation (?)
Collaborative Learning
“Quick Thinks”
Structured Groups
Active & Peer Learning:
The Evidence (Large
Courses)

Pausing frequently during lecture for 2 minute
discussions leads to better comprehension
grade points higher)


[Ruhl et al, Jrnl Teacher Ed. 1987]
A meta-analysis over 60 physics courses and 6,500
students found improvements of almost 2 std.dev.


(1-2
[Hake, Am. J. Physics, 1998]
Controlled experiment with > 500 physics students
found improved attendance, engagement, and more
than twice the learning.
 [Deslauries et al., Science 2011]
Active & Peer Learning:
The Evidence (Large
Courses)
Even if no one in the group knows
the answer, discussing improves
results (genetics)
[Smith et al, Science 323, Jan 2, 2009]
Peer Learning Example

From Deslauries et al:





Pre-class reading assignments and quizzes
(CQ) In-class clicker questions with studentstudent discussion
(GT) Small-group active learning tasks

Turn in individual written response
(IF) Targeted in-class instructor feedback
Typical schedule for 50-min class:




CQ1, 2 min; IF, 4 min.
CQ2, 2 min; IF, 4 min; CQ2 (continued), 3 min; IF, 5 min;
Revote CQ2, 1 min.
CQ3, 3 min; IF, 6 min.
GT1, 6 min; IF with a demonstration, 6 min; GT1 (continued),
4 min; and IF, 3 min.
Results for Controlled
Experiment
From Deslauries et al., for a one-week intervention
Peer Learning (Smaller
Classes)
Peer Learning Core
Ideas
 Students learn better by explaining to others
 Extended group work must be structured
 Must promote both:
 Positive Interdependence
 Individual Accountability
 Group makeup:
 Best if heterogeneous
 Groups can change frequently
IN-PERSON COURSE: APPLIED
NLP
IN-PERSON COURSE: APPLIED
NLP
IN-PERSON COURSE: APPLIED
NLP
After 4 Weeks
After 12 Weeks
WHAT CAN BE IMPROVED?
More short assignments!
PROJECT GOAL:
MOOCS + PEER LEARNING
How to do it?
First Step: Try MTurk
 Hypothesis:
 People in groups will get answers right more
often than those working alone
 Expectations:
 The chats will be on topic
 People will try to solve the problems
First Step: Try MTurk
 Issues?
 How to motivate the workers?
 How to coordinate the workers?
 What kinds of questions to use?
 How to structure the conversation?
How To Motivate?
 Experimental Manipulation:
 If entire group gets the right answer,
everyone gets a bonus
 Control Group:
 No mention of a bonus (no incentive for
helping others)
MOOC Arrival Times,
First Question, First
Lecture
MOOC Arrival Times,
Last Question, Last
Lecture
Question Type:
GMAT Critical
Reasoning
System Workflow
Real Time Crowdsourcing: Lasecki, et al, CSCW 2013, Bernstein et al, UIST 2011
Interaction:
Small-Group Chat
 CMC Literature suggests the affordances
are appropriate
 Video on next slide
Experimental Setup
 226 worker sessions lasting on average
12.8 minutes.

(15.0 minutes excluding solo workers), with 169
solo workers, 25 discussions of size 2, and 73
discussions of size 3.
 Each session consisted of 2 questions.
2 minutes alone, 5 minutes in discussion, 20 seconds
for final answer choice
 56% of the 452 attempts to answer
questions were answered correctly.
Results
 All hypotheses confirmed
 Engaging in discussion leads to more correct
answers.

The bonus incentive leads to more correct
changed answers.
 The participants have substantive discussions.
 Of interest, but not a result:

More discussion is correlated with more correct answers
Results






138 workers (61%) kept their original choices
unchanged on both questions
74 (33%) changed one answer after the discussion
14 (6%) changed both.
50% of workers who changed their answers
improved their score
18% lowered their score;
86% of workers who changed both answers
improved their score.
Results

Engaging in Discussion Leads to More Correct Answers

The mean percentage of correct responses is higher in chatrooms with more than one student (Fisher’s
exact test, p < 0:01).
Results

Bonus Incentive Leads to More Correct Answers:


In the control condition, participants changed 33 out of 121 (27%) In the
bonus condition they changed 44 out of 139 answers (32%). No
significant difference (Fisher’s exact test, two-tailed p = 0.50 ).
However, among the changed answers, 14 answers (12%) changed
from incorrect to correct in the control condition, while 31 (22%)
changed from incorrect to correct in the bonus condition, a significant
difference (Fisher’s exact test, two-tailed p < 0.04 )
Results

Participants have Substantive Discussions



3 independent raters, Scale of 1 to 4
73 of 98 discussions (74%) were rated 4 by all raters
80 (82%) had a median rating of 4. (Spearman’s rho=0.65)
Next Steps
 Put this into MOOCs!
 We have an experiment underway right
now.
Other MOOC
Projects
 Forum Usage
 Role of Instructor
 Untangling Correlation from Causation
 MOOC Instructor Dashboards
Thank you!
Marti A. Hearst
UC Berkeley
Joint work with Bjorn Hartmann, Armando Fox, Derrick Coetzee, Taek Lim
Sponsored in part by a Google Social Interactions Grant