LearnLab: Bridging the Gap Between Learning Science and Educational Practice

Ken Koedinger Human-Computer Interaction & Psychology, CMU PI & CMU Director of LearnLab 1

Real World Impact of Cognitive Science

Algebra Cognitive Tutor

• Based on ACT-R theory & cognitive models of student learning • Used in 3000 schools 600,000 students • Spin-off: Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

2

Cognitive Tutors: Interactive Support for Learning by Doing Authentic problems Feedback

within

complex solutions Personalized instruction Progress… Challenging questions … individualization 3

Success ingredients

• AI technology • Cognitive Task Analysis • Principles of instruction & experimental methods • Fast development & use-driven iteration 4

Cognitive Task Analysis: What is hard for Algebra students?

Story Problem As a waiter, Ted gets $6 per hour. One night he made $66 in tips and earned a total of $81.90. How many hours did Ted work? Word Problem Starting with some number, if I multiply it by 6 and then add 66, I get 81.90. What number did I start with?

Equation x * 6 + 66 = 81.90

Data contradicts common beliefs of researchers and teachers

Koedinger & Nathan (2004). The real story behind story problems: Effects of representations on quantitative reasoning. The Journal of the Learning Sciences.

Expert Blind Spot!

100 90 80 70 60 50 40 30 20 10 0

% Correctly ranking equations as

Elementary Teachers

hardest

Middle School Teachers High School Teachers Nathan & Koedinger (2000). An investigation of teachers’ beliefs of students’ algebra development.

Cognition and Instruction.

6

Cognitive Tutor Algebra course yields significantly better learning

Course includes text, tutor, teacher professional development ~11 of 14 full-year controlled studies demonstrate significantly better student learning Koedinger, Anderson, Hadley, & Mark (1997). Intelligent tutoring goes to school in the big city.

7

Success? Yes Done? No!

Why not?

• Student achievement still not ideal • Field study results are imperfect • Many design decisions with no research base • Use deployed technology to collect

data, make discoveries, & continually improve

8

PSLC Vision

• Why?

Chasm between science & ed practice • Purpose : Identify the conditions

that cause robust student learning

– Educational technology as instrument – Science-practice collaboration structure • Core Funding: 2004-2014 9

Do you know what you know?

What we know about our own learning What we do

not

know You can’t design for what you don’t know!

10

Transforming Education R&D

Ed tech + wide use = “Basic research at scale”

Algebra Cognitive Tutor

+ =

Chemistry Virtual Lab English Grammar Tutor

• Fundamentally transform – Applied research in education – Generation of practice relevant learning theory

Educational Games

11

Ed Tech => Data => Better learning

LearnLab Course Committees LearnLab Thrusts

How you can benefit from LearnLab

• Research – General principles to improve learning • Methods – Cognitive task analysis, in vivo studies • Technology tools • People – Masters students & projects 13

What instructional strategies work best?

• More assistance vs. more challenge – Basics vs. understanding – Education wars in reading, math, science… • Research on many dimensions – Massed vs. distributed (Pashler) – Study vs. test (Roediger) – Examples vs. problem solving (Sweller,Renkl) – Direct instruction vs. discovery learning (Klahr) – Re-explain vs. ask for explanation (Chi, Renkl) – Immediate vs. delayed (Anderson vs. Bjork) – Concrete vs. abstract (Pavio vs. Kaminski) – … Koedinger & Aleven (2007). Exploring the assistance dilemma in experiments with Cognitive Tutors.

Ed Psych Review.

14

Knowledge-Learning-Instruction (KLI) Framework:

What conditions cause robust learning Koedinger et al. (2012). The Knowledge-Learning Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning.

Cognitive Science

.

LearnLab research thrusts address KLI elements • Cognitive Factors – Charles Perfetti, David Klahr • Metacognition & Motivation – Vincent Aleven, Tim Nokes-Malach • Social Communication – Lauren Resnick, Carolyn Rose • Computational Modeling & Data Mining – Geoff Gordon, Ken Koedinger 15

Results of ~200 in vivo experiments => Optimal instruction depends on knowledge goals 16

Cognitive Task Analysis using DataShop’s learning curve tools Without decomposition, using just a single “Geometry” KC, no smooth learning curve.

But with decomposition, 12 KCs for area concepts, a smoother learning curve.

Upshot

: Can automate analysis & produce better student models 17

How you can benefit from LearnLab

• Research – General principles to improve learning • Methods – Cognitive task analysis, in vivo studies • Technologies – Tutor authoring – Language processing – Educational Data Mining • People: Masters students & projects 18

Questions?

19

Question for you

What do you need in a learning science professional?

20

21

Extra slides

22

Cognitive Tutor Technology • Cognitive Model : A system that can solve problems in the various ways students can 3(2x - 5) = 9 If goal is solve a(bx+c) = d Then rewrite as abx + ac = d If goal is solve a(bx+c) = d Then rewrite as abx + c = d If goal is solve a(bx+c) = d Then rewrite as bx+c = d/a 6x - 15 = 9 2x - 5 = 3 6x - 5 = 9 • Model Tracing : Follows student through their individual approach to a problem -> context-sensitive instruction 23

Cognitive Tutor Technology • Cognitive Model : A system that can solve problems in the various ways students can 3(2x - 5) = 9 If goal is solve a(bx+c) = d Then rewrite as abx + ac = d Hint message: “Distribute a across the parentheses.” Known? = 85% chance Known? = 45% If goal is solve a(bx+c) = d Then rewrite as abx + c = d Bug message: “You need to multiply c by a also.” 6x - 15 = 9 2x - 5 = 3 6x - 5 = 9 • Model Tracing : Follows student through their individual approach to a problem -> context-sensitive instruction • Knowledge Tracing : Assesses student's knowledge growth -> individualized activity selection and pacing 24

Cognitive Task Analysis Improves Instruction

• Studies: Traditional instruction vs. CTA-based – Med school catheter insertion (Velmahos et al., 2004) – Radar system troubleshooting (Schaafstal et al., 2000) – Spreadsheet use (Merrill, 2002) • Lee (2004) meta-analysis: 1.7 effect size!

25

Learning Curves

26

Inspect curves for individual knowledge components (KCs)

Many curves show a reasonable decline Some do not => Opportunity to improve model!

27

DataShop’s “leaderboard” ranks alternative models 100s of datasets from ed tech in math, science, & language Best model finds 18 components of knowledge (KCs) that best predict transfer 28 28

Data from a variety of educational technologies & domains

Statistics Online Course English Article Tutor Numberline Game Algebra Cognitive Tutor

29

Model discovery across domains Variety of domains & technologies 11 of 11 improved models Koedinger, McLaughlin, & Stamper (2012). Automated student model improvement. In Proceedings of Educational Data Mining. [Conference best paper.] 30

Data reveals students’ achievement & motivations

We have used it to • Predict future state test scores as well or better than the tests themselves • Assess dispositions like work ethic • Assess motivation & engagement • Assess & improve learning skills like help seeking … 31

LearnLab courses at K12 & College Sites

• 6 + cyber-enabled courses: Chemistry, Physics, Algebra, Geometry, Chinese, English • Data collection – Students do home/lab work on tutors, vlab, OLI, … – Log data, questionnaires, tests  DataShop

REAP vocabulary tutor

Researchers

Learn Lab

Schools

Chemistry virtual lab Physics intelligent tutor

32

Bridging methodology: in vivo experiments

Setting Lab experiment Lab

In Vivo Experiment

School Design Research School Control condition Focus on principle vs. on solution (Change N things) Yes Scientific Principle Yes Scientific Principle No Instr. Solution Cost/Duration $/Short $$/Medium $$ /Long Randomzd Field Trial School Yes Instr. Solution $$$$/Long 33

Knowledge Components

• Definition: An acquired unit of cognitive function or structure that can be inferred from performance on a set of related tasks • Includes: – skills, concepts, schemas, metacognitive strategies, malleable habits of mind, thinking & learning skills • May also include: – malleable motivational beliefs & dispositions • Does not include: – fixed cognitive architecture, transient states of cognition or affect • Components of

“intellectual plasticity”

Koedinger et al. (2012). The Knowledge-Learning Instruction (KLI) framework: Bridging the science 34

Cognitive Science

.

General knowledge components, sense-making, motivation, social intelligence

Possible domain-general KCs • Metacognitive strategy – Novice KC: If I’m studying an example, try to remember each step – Desired KC: If I’m studying an example, try to explain how each step follows from the previous • Motivational belief – Novice: I am no good at math – Desired: I can get better at math by studying & practicing • Social communicative strategy – Novice: If an authority makes a claim, it is true – Desired: If considering a claim, look for evidence for & against it 35

What is Robust Learning?

• Achieved through: – Conceptual understanding & sense-making skills – Refinement of initial understanding – Development of procedural fluency with basic skills • Measured by: – Transfer to novel tasks – Retention over the long term, and/or – Acceleration of future learning 36

KLI summary

• Learning occurs in components (KCs) • KCs vary in kind/cmplxty – Require different kinds of learning mechanisms • Optimal instructional choices are dependent on KC complexity

Intelligence does not improve generically

Koedinger et al. (2012). The Knowledge-Learning-Instruction (KLI) framework: Bridging the science-practice chasm to enhance robust student learning.

Cognitive Science

.

37

Conclusions

• Learning & education are complex systems • Lots of work for learning science! • Use ed tech for “basic research at scale” => Bridge science-practice chasm 38