Transcript Document

Scientifically Informed Digital
Learning Interventions
One Example: The Open Learning
Initiative at Carnegie Mellon
Financial and Intellectual Support:
• The William and Flora Hewlett Foundation
• The National Science Foundation
• A.W. Mellon Foundation
• Carnegie Mellon University
The Challenge
• To learn ways to design and build fully
web-based courses which by rigorous
assessments are proven to be as good or
better than traditional teaching methods
• Why?
– Increasing access
– Improving effectiveness
– Providing flexibility for faculty and students
– Containing costs
A Flaw and an Opportunity
• Current structure of higher education
presents substantial roadblocks to the
application of proven results and
methodologies from the learning
sciences
• eLearning interventions, developed by
teams rather than individuals, are more
conducive to making the practice of
education more scientific and effective
OLI Guiding Assumptions
• Digital learning interventions can make a
significant difference learning outcomes
• Designs grounded in contemporary learning
theory and scientific evaluation have the best
chance of achieving that goal
• A possible, acceptable outcome of the OLI
efforts is failure or mixed failures and
successes – we are doing “action research,”
not promoting eLearning for its own sake
OLI Guiding Assumptions
• Formative assessment will be a major
feature (and a major component of the
cost) of the designs and iterative
improvements of the courses
• IT staff working with faculty is too limited
a partnership – learning scientists, HCI
experts, and assessment experts must
be part of design, development,
production and iterative improvement
Open Learning Initiative Courses
•
•
•
•
•
•
•
•
•
•
•
Statistics
Modern Biology
Chemistry
French
Engineering Statics
Causal and Statistical Reasoning
Economics
Logic and Proofs
Physics
Empirical Research Methods
Computational Discrete Mathematics
Try it Yourself
• http://www.cmu.edu/oli
• Don’t expect an “OCW
experience”…this project has a different
set of goals than OCW
• “Clicking around” will be unsatisfying:
these interventions are designed to
support a novice learner in acquiring
knowledge working on their own
Key Elements in OLI Courses
• Theory Based: Course and individual lesson
designs based on current theories in the
learning sciences
• Feedback Loops: Courses record student
activity for robust feedback mechanisms
• Diversity of Perspectives, Roles and
Contexts: Courses developed and deployed
by teams that include faculty content experts,
learning scientists, software engineers
Theory Based: Build on Prior/
Informal Knowledge
Theory Based: Provide Immediate
Feedback in the Problem Solving
Context
Theory Based: Promote Authenticity,
Flexibility & Applicability
• Learning environments with ambiguous problems that
require flexible application of procedural knowledge
Feedback Loops in Learning
Evaluation
• Chemistry: Post-test scores by treatment group show
significant positive correlation for the OLI treatment.
Most significant indicator was time spent in Virtual
Lab Activities – made all other variables drop out.
• Biology: End of the 3rd week showed an advantage
for the OLI section. There was a positive and
significant association between students’ time spent
working on particular activities and performance on
quiz questions testing the corresponding topics even
after total time with OLI has been regressed out
Evaluation
• Statistics 1st Study:
Evaluation
 CAOS Sample:
n
Average % correct
Pre
488
43.3
Post
488
51.2
Increase: 7.9%
[t(487) = 13.8, p <.001]
 CMU OLI Course Sample:
n
Average % correct
Pre
24
55.8
Post
24
66.5
Increase: 11.7%
[t(23) = 4.7, p <.001]
Evaluation
Measured learning
Outcome
% correct CAOS % correct CMU
Pre
Post
Pre
Post
Box plots provide accurate
estimates of % data above
& below only for quartiles
22.2
22.2
22.2
50.0
Correctly estimate and
compare SD’s for different
histograms.
31.5
41.8
51.9
46.4
46.4
49.4
66.7
59.3
48.1
83.3
75.0
70.8
49.6
47.4
70.4
70.8
Correlation does not imply
causation
Calculating appropriate
conditional probabilities
given table of data
Accelerated Learning Study
• Taught Carnegie Mellon Introductory
Statistics course in a blended mode (one in
class meeting per week) in half a semester
• The OLI Statistics course was the “textbook”
• OLI course provided the professor immediate
feedback on students’ performance
• We compared learning outcomes in the two
different treatments
Accelerated Learning Study
• OLI students significantly outperformed
Traditional “control” students on the CAOS
post-test.
CA OS P ost-test
1
0.9
Proportion Correct
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
OLI
TRAD
Accelerated Learning Study
• OLI students showed significantly greater gains (pre
to post) than the Traditional “control” students on
the CAOS test.
0.25
Gain Score
0.2
0.15
0.1
0.05
0
OLI
TRAD
Student Satisfaction
– End of course survey for online section:
• All students reported at an increase in their
interest in statistics.
• 75% Definitely Recommend
25% Probably Recommend
0% Probably not Recommend
0% Definitely not Recommend
Feedback Loop – Current
Research
Instructors can use
such data to adjust
their teaching to
students’ needs.
Learning Curve Analysis on
Stoichiometry Data
The Vision – Digital Dashboard
for Teaching and Learning:
• Instructor assigns students to work
through online instruction
• System collects data as students work
• System automatically analyzes and
organizes the data to present instructor
with the students’ current “learning
state”
• Instructor reviews this data summary
and adapts instruction accordingly
The Anticipated Benefits
• Instructors get a window onto students’
progress
• They can adapt their teaching
accordingly
• Students get better feedback to monitor
and adjust their learning
• Strengthens the student-instructor
connection
Core OLI Community
•
•
•
•
•
•
•
Faculty Content Experts
Learning Scientists
Human Computer Interaction
Software Engineers
Evaluation/Assessment Specialists
Learners
A community of scholars from diverse
disciplines who are committed to improving
quality and access to instruction. The
collaborative nature of the OLI course design
process inspired participating faculty to
rethink their approach to classroom teaching.
“Improvement in post-secondary education will
require converting teaching from a ‘solo sport’
to a community-based research activity”
Herbert Simon
www.cmu.edu/oli
[email protected]
[email protected]
(Candace Thille – Director)