Transcript Livenotes A System for Cooperative and Augmented Note-Taking in Lectures Matthew Kam, Jingtao Wang, Alastair Iles, Eric Tse, Jane Chiu, Daniel Glaser, Orna Tarshish.

Livenotes
A System for Cooperative and
Augmented Note-Taking in Lectures
Matthew Kam, Jingtao Wang, Alastair Iles, Eric Tse,
Jane Chiu, Daniel Glaser, Orna Tarshish and John Canny
University of California, Berkeley, USA
Video
Outline






Motivation
Solution
Experiment
Results
Implications
Conclusion
Motivation: Problem Statement

Constructivism



Learners are not blank slates that teachers write on
Learners need to actively construct their own
understanding and knowledge
But large lecture classes are not conducive for
active learning



Passive mode of oral dissemination
Lack of interactivity among students
Lack of interactivity with instructor
Motivation: Precedents

Face-to-Face Tutored Video Instruction (TVI),
Distributed TVI (Gibbons, Stanford)




Group review of pre-recorded lectures
Regular pauses for small-group discussion
Students using DTVI received grades 0.5 std dev
higher than non-TVI students (Smith et al. 1999)
Peer Instruction (Mazur, Harvard)


Lecture pauses for small-group discussion with
neighbors
Improvements in conceptual understanding and
problem-solving (Crouch and Mazur 2001)
Motivation: Small-Group,
Cooperative Learning

More than 375 research studies since 1898
(Johnson and Johnson 1989)

Cooperative group learning results in greater




Efforts to achieve
Higher-level reasoning
Transfer from original context to new situations
Generation of new ideas and solutions
Motivation:
Background Lecture Notes

(Hartley 1978, Kiewra et al. 1988)
Experiments on note-taking that compared
students annotating over:
1)
2)
3)

Complete lecture notes provided by instructor, vs.
Skeletal (i.e. partial) notes, vs.
No background notes
Results: students were found to achieve
maximum retention with skeletal notes
Livenotes Recap



Both a technology and educational practice
Large lecture classes
Small-group discussions in ongoing lecture

Cooperative note-taking:
Combines real-time note-taking with discussion

Augmented note-taking:
Skeletal slides for students to annotate over
Related Systems

No interaction between students



No real-time interaction between students, i.e.
sharing of notes takes place after lecture


Classroom Presenter (Washington)
StuPad, eClass (Georgia Tech)
NotePals (Berkeley)
Limited real-time interaction between students

OneNote (Microsoft)
Outline






Motivation
Solution
Experiment
Results
Implications
Conclusion
Livenotes Evolution

2000: Implemented in Java, for WinCE Clios

Late 2000 to early 2003: 5 small-scale
deployments using Clios, laptops and
Tablet PCs

Spring 2003: Medium-scale experiment in
undergraduate class using Tablet PCs

Since 2003: Ported to Microsoft .NET
Livenotes User Interface
Group
awareness
(e.g. each
user’s page
number)
Unique
user
colors
Pen and keyboard input
Import
background
slides
Client-Server Topology
Group 1:
Server

802.11b networking

Large class broken
down into many small
groups (3-7 students)

One Tablet per group
is set to server mode
Other members’
Tablets connect
wirelessly to group’s
server
Clients
…
Group n:
Server

Clients
Outline






Motivation
Solution
Experiment
Results
Implications
Conclusion
Hypotheses
Cooperative note-taking:
Shared whiteboard interface enhances
learning through cooperative note-taking and
discussion
Augmented note-taking:
Background slides enhances learning by
augmenting student note-taking
Experiment

Spring 2003 undergraduate
HCI class

21 volunteers, randomly
partitioned into


Cooperative note-takers
Individual note-takers
(control group)

4 weeks (7 lectures)

Preloaded skeletal PowerPoint slides
Previous Observation

From 5 previous deployments,
we learned that



Graduate students engaged
spontaneously in group
discussions
Undergraduates were not used
to discussing lecture material
with one another
For this experiment (with undergraduates), we held
short, live group discussions in the classroom
Data Collection


Short quizzes (4 lectures)
Survey questionnaires




First week of deployment (~38% response rate)
End of semester (~29%)
Qualitative interviews (3 users)
Transcripts of students’ notes (~1581 pages)
Quantitative Analysis

Unit of analysis: mark



Spatio-temporally
contiguous segment of
user input
E.g.: “This lecture is very
interesting”
Quantitative handcoding of ~1581 pages
Taxonomy of Marks

Note-taking:
someone taking notes on lecture

Commentary:
someone making a statement

Question:
someone soliciting a response

Answer:
response to a question, clarification

Reinforcement: contribution to an existing thread
Outline






Motivation
Solution
Experiment
Results
Implications
Conclusion
Cooperative Note-Taking:
Richer Notes

Cooperative note-taking group engaged in more
than twice as much activity as individual note-taker
Cooperative Note-Taking:
Richer Notes

Almost one quarter of marks made by cooperative
note-takers were attributed to group interaction
Student Learning

Survey question: “How did Livenotes, if at all,
assist your learning in lecture?”

Early survey after 2 sessions with Livenotes:
75% of respondents self-reported affirmatively

Survey after semester (i.e. 7 lecture sessions):
83% of respondents self-reported affirmatively
Cooperative Note-Taking:
Taking Turns to Take Notes

66% of survey respondents agreed that
cooperative note-taking is more useful

“Someone else might note something that I
missed or hadn’t realized.”

“I liked how note-taking became a cooperative
effort … someone can take over if another user
is still inputing some notes, but the prof [had]
moved on already.”
Cooperative Note-Taking:
Paying Greater Attention

36% of students who self-reported learning
benefit explicitly attributed that to social
aspect of cooperative note-taking:
“Helped me to focus more in lecture. Often I
fall asleep/lose attention in lecture. Having
group members to respond to kept me better
on track.”
Cooperative Note-Taking:
Dual Conversations

Need to keep up with both lecture and ontablet conversation:
“It is helpful to be able to discuss questions.
However, this does take attention away from
the lecture if you are focusing on
answering/asking a question.”
Cooperative Note-Taking:
Decreasing Distraction

Is “running Livenotes during class distracting?”
(1 = extremely distracting,
5 = not distracting at all)
 Survey after two lectures: 2.6 out of 5
 Survey after deployment ended: 3.83 out of 5

From student notes, “playful” behavior were
observed to disappear almost completely after 2
lectures
Cooperative Note-Taking:
Unanswered Group Questions

Students did not have time to answer some
questions because they needed to keep up
with lecturer

Some questions were unanswered because
no group member knew the answer
Cooperative Note-Taking:
Interaction During Pauses

Group interaction during pauses in lecture accounted
for over half of group activity
Redeeming PowerPoint

Criticisms leveled at Microsoft PowerPoint



The Cognitive Style of PowerPoint (Edward Tufte)
Death by PowerPoint webpage
I  Powerpoint (David Byrne, Talking Heads)

Too boring, passive, does not promote active
engagement with material

But students commented that augmented
note-taking is “like having a conversation with
the professor”
Augmented Note-Taking:
Observed Behaviors






Elaborated on bullets
Appended bullets to list
Concurred and disagreed with bullet
Noted gist of HCI principles
Noted advantage and disadvantage of HCI
technique
Answered questions in bullets
Augmented Note-Taking:
Elaborated on Bullets
Augmented Note-Taking:
Appended Bullets to List
Augmented Note-Taking:
Answered Questions in Bullet
Augmented Note-Taking:
Answered Questions in Bullet

Students responded to questions in bullets
even when when they were not cooperative
note-takers

Each group responded to 35% of the questions

Each question received a response from 36% of
the groups
Augmented Note-Taking:
Student Learning

Several high-quality notes in both individual and
cooperative note-taking groups resulted from
students “working off” bullets


Possibly due to bullets focusing student attention to
relevant portions of lecture
A larger proportion (55%) of students who selfreported learning benefit attributed it to augmented
note-taking, compared to cooperative note-taking.

Half of this sub-group attributed that to having slides
at hand to annotate over
Quiz Scores
Quiz 1
Quiz 2
Quiz 3
Quiz 4
C [1]
I [2]
C
I
C
I
C
I
Mean Score
[3]
63.9
66.0
71.2
75.0
58.5
67.4
53.2
55.7
Std. Dev.
9.2
7.3
12.1
n/a
9.5
11.5
8.6
24.4
8
4
9
1
9
4
7
4
Sample size
P-value
0.702
0.771
0.168
0.810
[1] Cooperative note-takers.
[2] Individual note-takers.
[3] Quiz scores presented in this table are normalized on a scale of 100.


No statistical significance
But sample size was too small due to poor
attendance at end of semester
Outline






Motivation
Solution
Experiment
Results
Implications
Conclusion
Student-Instructor Interaction

To help instructor assess student
learning, we deployed feedback
feature in last two sessions

Students provide instructor with
real-time, anonymous lecture
feedback

Recently allowed students to alert
instructor that they have questions
Recommendation:
Background Slides as Scaffold

Bullets are a lightweight means for lecturer to
engage actively with students during class




Posing questions
Counter-intuitive bullets
Provocative statements
Direct student attention to critical parts of
lecture

E.g. prompts such as “Pros?” and “Cons?” with
blank spaces for students to fill in
Outline






Motivation
Solution
Experiment
Results
Implications
Conclusion
Conclusion

Cooperative note-taking





Richer variety of notes, higher-order thinking
More than twice as much notes as individuals
Members took turns to take notes
Students kept awake to interact with group
Augmented note-taking



Observed dialogue with bullets
Reflected higher-order thinking
High-quality notes resulted from “working off”
bullets
Acknowledgement




Corporation for Education Network Initiatives in
California
Microsoft Research
National Science Foundation
Qualcomm

Volunteers from Computer Science 160, Spring
2003

Public domain source code by James R. Weeks
Questions?
Livenotes can be downloaded from:
http://www.cs.berkeley.edu/~mattkam/livenotes
Matthew Kam, Ph.D. student
Department of Electrical Engineering and Computer Sciences,
and Berkeley Institute of Design
University of California at Berkeley, USA
[email protected]