Model-based Inquiry: Epistemology, Modeling Skills, Assessment, & Research Janice Gobert The Concord Consortium mac.concord.org

mtv.concord.org Based on work from 1) Making Thinking Visible (NSF #9980600) and 2) Modeling Across the Curriculum (IERI #0115699) . All opinions expressed are those of the author and do not necessarily reflect the views of the granting agencies.

How are you defining “scientific practice” in your design and empirical work?

The Scientific Practice is Modeling, this includes model-based reasoning, model based inquiry, etc.

• • • • MBL is a theory of science learning that integrates research in cognitive psychology and science education (Gobert & Buckley, 2000).

Its tenets are that understanding requires the construction of mental models and all subsequent problem-solving, inferencing, or reasoning are done by means of manipulating or ‘running’ these mental models (Johnson-Laird, 1983).

Model-based reasoning also involves the testing, reinforcement, revision, or rejection of mental models.

and subsequent Modeling research at the Concord Consortium organizes learning activities, assessment, and research around model-based learning.

MBR Involves both internal and external models

cognitive processes act on mental model Mental model External Models, i.e., hypermodels Assumes students’ epistemologies influences model-based reasoning; Gobert & Discenna, 1997; Gobert & Pallant, 2004).

Other research literature….

• • • • • •

In addition to students’ pre-instruction models in designing the unit, we (J. Gobert, Jim Slotta, Amy Pallant) drew on current findings from: causal models

(White, 1993; Schauble et al, 1991; Raghavan & Glaser, 1995),

model-based teaching and learning

(Gilbert, S., 1991; Gilbert, J. 1993);

model revising

(Clement, 1989; 1993; Stewart & Hafner, 1991);

diagram generation and comprehension

(Gobert, 1994; Gobert & Frederiksen, 1988; Kindfield, 1993; Larkin & Simon, 1987; Lowe, 1989; 1993),

the integration of text and diagrams

(Hegarty & Just, 1993), and

text comprehension

(van Dijk & Kintsch, 1983; Kintsch, 1998).

How is it being supported?

from Making Thinking Visible Project (mtv.concord.org) • • • • • •

Scaffold drawing of their own models of plate tectonics phenomena based on progressive model-building principles (model pieces acquisition).

Scaffold on-line “field trip” to explore differences between the East and West coast in terms of earthquakes, volcanoes, mountains (beginning with the most salient differences to support knowledge building around the driving question; model-pieces acquisition).

Posing a question about their current model (to model pieces integration and model-building).

Learn about location of earth’s plates (to scaffold relationship between plate boundaries anf plate tectonic phenomena as model pieces integration).

Reify important spatial and dynamic knowledge (model pieces integration) about transform, divergent, collisional, and convergent boundaries .

Learn about causal mechanisms involved in plate tectonics, i.e., convection & subduction (scaffolded by reflection activities to integrate spatial, causal, dynamic, and temporal aspects of the domain- model pieces integration).

Pedagogical support (cont’d) from Making Thinking Visible Project (mtv.concord.org) • • •

Students are scaffolded to critique learning partners’ models using prompts in WISE (reconstruct, reify, & reflect). Prompts include: 1. Are the most important features in terms of what causes this geologic process depicted in this model? 2. Would this model be useful to teach someone who had never studied this geologic process before? 3. What important features are included in this model? Explain why you gave the model this rating.

4. What do you think should be added to this model in order to make it better for someone who had never studied this geologic process before?

Reflect on how their model was changed and what it now helps explain ( reconstruct, reify, & reflect) , e.g.,: prompts include:

  “

I changed my original model of.... because it did not explain or include....” “My model is now more useful for someone to learn from because it now includes….” Transfer what they have learned in the unit to answer intriguing points ( reconstruct, reify, & reflect ):

 

Why are there mountains on the East coast when there is no plate boundary there?

How will the coast of California look in the future?

How do you know when you see it?

• Examples to follow….

Comments on example 1….

• Original model- focus on crustal layer, no causal mechanisms for what causes mountain formation.

• W. coast partners’ critique requested labels.

• Revised model-includes labels, and a cut away view of the interior of the earth which includes convection in the mantle.

Comments on example 2… • Original model- cross section, no causal mechanisms for what causes mountain formation.

• W. coast partners’ critique requested information about direction of plate movement.

• Revised model-includes a cross section with plate movement, added the mantle as an interior layer.

Does it effect students’ epistemologies? * • Data to follow….

* Acknowledging the problem with assessing epistemologies with surveys.

WISE Period 1 - sig. Epistemological gains

16 14 12 10 8 6 4 2 0

Inte ra ction Bar Plot for modelgain Effe ct: Ca tegory for modelgain * teacher

preMto t pos tMtot Cell A S T

Fisher's PLSD for mode lgain Effe ct: te ache r Signific ance Level: 5 %

Mean Di ff.

Crit. Di ff.

A, S A, T S, T 1.0 12 .51 1 -.50 2 1.5 71 1.5 43 1.7 39 P-Val ue .20 47 .51 39 .56 92

ANOVA Table for mode lgain

tea che r Subj ect(Gro up) Categ ory for mode lgai n Categ ory for mode lgai n * te ach er Categ ory for mode lgai n * Sub ject(Group) DF 2 61 1 2 61 Sum of Sq uares 22.442

988 .926

115 .697

83.882

439 .837

Mean Square 11.221

16.212

115 .697

41.941

7.2 10 F-Va lue .69 2 P-Val ue .50 44 16.046

5.8 17 .00 02 .00 49 Lam bda 1.3 84 Power .15 7 16.046

11.633

.98 7 .86 6

WISE Period 2 - sig. Epistemological gains

14 12 10 8 6 4 2 0

Inte ra ction Bar Plot for modelgain Effe ct: Ca tegory for modelgain * teacher

preMto t pos tMtot Cell A S T

Fisher's PLSD for mode lgain Effe ct: te ache r Signific ance Level: 5 %

Mean Di ff.

Crit. Di ff.

A, S A, T S, T .06 4 -.28 9 -.35 3 1.6 32 1.6 32 1.8 27 P-Val ue .93 80 .72 68 .70 28

ANOVA Table for mode lgain

tea che r Subj ect(Gro up) Categ ory for mode lgai n Categ ory for mode lgai n * te ach er Categ ory for mode lgai n * Sub ject(Group) DF 2 59 1 2 59 Sum of Sq uares 2.3 35 874 .504

311 .401

56.782

448 .710

Mean Square 1.1 67 14.822

311 .401

28.391

7.6 05 F-Va lue .07 9 P-Val ue .92 44 40.945

3.7 33 <.0 001 .02 97 Lam bda .15 8 Power .06 1 40.945

7.4 66 1.0 00 .65 9

WISE Period 3 - sig. Epistemological gains

14 12 10 8 6 4 2 0

Inte ra ction Bar Plot for modelc hange Effe ct: Ca tegory for modelc hange * tea cher

preMto t pos tMtot Cell

ANOVA Table for mode lcha nge

tea che r Subj ect(Gro up) Categ ory for mode lch ange Categ ory for mode lch ange * tea che r Categ ory for mode lch ange * Subj ect(Gro up) A S T

Fisher's PLSD for mode lcha nge Effe ct: te ache r Signific ance Level: 5 %

A, S A, T S, T Mean Di ff.

-.80 9 .83 3 1.6 42 Crit. Di ff.

1.6 84 1.6 54 1.8 76 P-Val ue .34 37 .32 07 .08 57 DF 2 62 1 2 62 Sum of Sq uares 47.195

102 1.13 2 366 .531

106 .362

414 .665

Mean Square 23.597

16.470

366 .531

53.181

6.6 88 F-Va lue 1.4 33 P-Val ue .24 64 54.803

7.9 52 <.0 001 .00 08 Lam bda 2.8 66 Power .28 5 54.803

15.903

1.0 00 .95 8

WISE Period 4 - sig. Epistemological gains

14 12 10 8 6 4 2 0

Inte ra ction Bar Plot for modelc hange Effe ct: Ca tegory for modelc hange * tea cher

preMto t pos tMtot Cell

ANOVA Table for mode lcha nge

tea che r Subj ect(Gro up) Categ ory for mode lch ange Categ ory for mode lch ange * tea che r Categ ory for mode lch ange * Subj ect(Gro up) A S T

Fisher's PLSD for mode lcha nge Effe ct: te ache r Signific ance Level: 5 %

A, S A, T S, T Mean Di ff.

-.07 3 -1.5 89 -1.5 16 Crit. Di ff.

1.3 92 1.3 67 1.5 51 P-Val ue .91 80 .02 31 .05 52 DF 2 62 1 2 62 Sum of Sq uares 63.678

703 .214

190 .437

65.833

330 .098

Mean Square 31.839

11.342

190 .437

32.917

5.3 24 F-Va lue 2.8 07 P-Val ue .06 81 35.768

6.1 82 <.0 001 .00 36 Lam bda 5.6 14 Power .52 3 35.768

12.365

1.0 00 .88 9 S

WISE Period 5 - sig. Epistemological gains

14 12 10 8 6 4 2 0

Inte ra ction Bar Plot for modelc hange Effe ct: Ca tegory for modelc hange * tea cher

preMto t pos tMtot Cell A S T

Fisher's PLSD for mode lcha nge Effe ct: te ache r Signific ance Level: 5 %

A, S A, T S, T Mean Di ff.

.70 1 -.53 1 -1.2 32 Crit. Di ff.

1.6 31 1.7 58 1.9 09 P-Val ue .39 70 .55 10 .20 40

ANOVA Table for mode lcha nge

tea che r Subj ect(Gro up) Categ ory for mode lch ange Categ ory for mode lch ange * tea che r Categ ory for mode lch ange * Subj ect(Gro up) DF 2 60 1 2 60 Sum of Sq uares 26.202

936 .016

444 .676

90.227

353 .325

Mean Square 13.101

15.600

444 .676

45.113

5.8 89 F-Va lue .84 0 P-Val ue .43 68 75.513

7.6 61 <.0 001 .00 11 Lam bda 1.6 80 Power .18 1 75.513

15.322

1.0 00 .95 0

Modeling Across the Curriculum Team

Principal & Co-Principal Investigators

Paul Horwitz, Concord Consortium, Principal Investigator Janice Gobert, Concord Consortium, Co-PI & Research Director Robert Tinker, Concord Consortium, Co-PI Uri Wilensky, Northwestern University, Co-PI

Other senior personnel

Barbara Buckley, Concord Consortium Ken Bell, Concord Consortium Trudi Lord, Concord Consortium Chris Dede, Harvard University Sharona Levy, University of Haifa Jaclyn Scobo (intern), Northeastern University mac.concord.org; IERI #0115699 www.concord.org

http://ccl.northwestern.edu

Design of Activities, Scaffolding, & Research are based on… • • • Model-based learning (Gobert & Buckley, 2000) as well as other literature….

cognitive and perceptual affordances of learning with technology-based representations (Gobert, 2005; Larkin & Simon, 1987) progressive model-building (White & Frederiksen, 1990; Raghavan & Glaser, 1995) students’ difficulties in learning with models (Sweller, et al, 1990; Gobert, 1994; Lowe, 1989; Head, 1984).

• • Thus , scaffolding is designed to… guide search, supports perceptual cues, and inference-making from perceptual cues (Larkin & Simon, 1987). elicit prior knowledge, support integration with new knowledge, and support reification & reflection of knowledge.

• • Theory driving our analyses is based on… expert problem-solving for estimating solutions (Paige & Simon, 1966) experts vs. novices search and knowledge acquisition strategies (Gobert, 1994, 1999; Thorndyke & Stasz,1980).

Model-Based Learning in situ

Intrinsic Learner Factors Epistemology of models

(SUMS, Treagust et al, 2002)

….because students’ epistemologies influence both knowledge integration (Songer & Linn, 1991) and model based reasoning (Gobert & Discenna, 1997), prior knowledge new information model formation model use model reinforcement model revision model rejection Interacting with

understanding reasoning generating

model evaluation + Metacognitive

Selecting Directing Monitoring

Classroom Factors Implementation of MAC activity use

(logged)

Teacher practices

(reported via Classroom Communique)

Intrinsic Teacher Factors Hypermodels *

simulations diagrams explanations instructions data tables graphs

Epistemology of models

(adapted from Grosslight et al, 1991)

Phenomena

experiences experiments

Teaching experience Background

(adapted from Fishman, 1999)

What is the model for the pedagogical support of the practice? What kinds of designs put this model into effect?

Scaffolds from the MAC project include:

• • • • •

Representational Competence:

view and understand a representation or representational features of the domain.

Model pieces acquisition:

understand & reason with pieces of models (spatial, causal, functional, temporal).

Model pieces integration:

combine model components in order to come to a deeper understanding of how they work together as a causal system.

Model based reasoning:

reasoning with models or pieces of models.

Reconstruct, Reify, & Reflect:

reify knowledge and transfer it to another context or level of understanding.

Technology & Affordances for Research & Assessment with Models

Technology: Log files on students’ interactions with models capturing students’… – – – – Data on duration and sequence Actions and choices with models What info or help they seek Responses to questions Affordances Implementation data -- which activities were used, pattern of use (consecutive or intermittent days) at classroom level & student level.

Finer-grained log data can be used for – Measure of students’ systematicity and inquiry skills.

– Test for interactions with prior knowledge & epistemology.

Embedded & Performance Assessments with models & questions… – Generate profile for students at “pivotal” points in curriculum – Responses to questions These data are used to derive student reports….

¯ ¯ ¯ Formative assessments Summative assessments Performance assessments

Drill down to performance assessment with logs

Currently we are focusing on log files as indices of: 1) Domain specific model based reasoning by investigating “hot spots”

2) D

omain- General Inquiry skills (DoGI spots, similar to NSES inquiry strands).

3) – – – This allows us to assess inquiry development both within (hot spots) and across domains (DoGI spots). assess transfer from one domain to another assess how a student’s inquiry skills are progressing “independent” of content learning. Since our activities are enacted over multiple days and in three domains, we avoid the problems faced by earlier studies of inquiry in which there were not enough data to get at students’ inquiry skills (Shavelson et al, 1999).

Inquiry “Hot Spots”

Tasks or parts of tasks that contain multiple components of model-based inquiry these, by definition, require deep reasoning.

• • • • • MAC supports 5 strands of model-based inquiry. tasks.

These are more specific than the NSES (1996) inquiry standards which were are not specific to current technology-based learning nor are the NSES strands specific to modeling

Representational Competence:

view and understand a representation or features of the domain.

Model pieces acquisition:

understand & reason with pieces of models (spatial, causal, functional, temporal).

Model pieces integration:

combine model components in order to come to a deeper understanding of how they work together as a causal system.

Model based reasoning:

reasoning with models or pieces of models.

Reconstruct, Reify, & Reflect:

reify knowledge and transfer it to another context or level of understanding.

Fine-grained analysis, one hot spot at a time, is necessary in order for us to code the various process variables we plan to aggregate and focus on.

Hot spot from Collisions task 5: Student sets mass of two balls • The challenge: adjust the masses of the two balls to make the orange ball move as fast as possible after the collision.

Strategies for Inquiry Preliminary analysis based on human coding identified 2 different inquiry patterns:

1.

2.

haphazard systematic (Also, there are students who got it correct on first trial, sometimes with explicit test).

These are consistent with literature:

~ experts vs. novices search and knowledge acquisition strategies (Thorndyke & Stasz,1980; Gobert, 1994, 1999).

~ expert problem-solving for estimating solutions (Paige & Simon, 1966) .

Examples …

Haphazard Strategy- this student obtained the correct answer (11.0; 1.0) on trials 2,10,(& 15) but did not know it!

Student 12116 made 15 trials: Blue Ball 11.0

11.0

11.0

11.0

1.0

1.0

8.0

11.0

11.0

11.0

11.0

3.0

1.0

1.0

11.0

Orange ball 11.0

1.0

3.0

4.0 1.0 11.0 7.0 2.0 11.0 1.0

5.0

5.0

5.0

8.0

1.0

Systematic Strategy, e.g.,vary one ball at a time (a good strategy in the absence of prior knowledge).

Student 18115 had a plan: Blue Ball 11.0

5.0

10.0

11.0

Orange ball 11.0

11.0

11.0

1.0

Another Hot Spot from Dynamica: “What settings cause the blue ball to stop when it collides with the orange ball?” • Track students’ iterations of this as index of systematicity in inquiry.

Input sliders Numerical data from run Constructed text response

TL3 time

2.5

TL3 RdTsk

73 2.9

2.7

2 34 130 13 CC’s approach for Task 3 Additional Categories for coding & 4 students’

T3 trials T3 values T3 Vx

2 2.0 v 5.0 5.0 v 5.0 -1.7, 0.0, 8 2.0 v 5.0, 4.0 v 11.0, 1.0 v 4.0, 11.0 v 11.0, 6.0 v 7.0, 5.0 v 7.0, 3.0 v 7.0, 7.0 v 7.0, -1.71, -1.87, -2.4, 0.0, 0.31, 0.67, 1.6, 0.0,

T3 #rtPr

1 2

T3 success

1 1 data.

Q10A

that they must have have equal masses they have to both have to be the same size

T3 %vary1

1

T3 %rpt

0

T3 #eqPr

1

T3 #extrem Pr

0

T3 %clg

1

T3 %frg

0

T3 %goal Flips T3 CAT

0 B1 0.43

0 2 0 0.29 0.71

0.43 B2 1 10 5.0 v 5.0, 0.0, 2.0 v 5.0, 1.0 v 5.0, 11.0 v 5.0, 8.0 v 5.0, 8.0 v 11.0, 7.0 v 11.0, 6.0 v 11.0, 7.0 v 10.0, 11.0 v 10.0, 10.0 v 10.0, -1.71, -2.67, 1.5, 0.92, 0.63, 0.89, 1.18, 0.71, 0.19, 0.0, 1 1 match the orange 0 0 1 0 0 0 0 A 1 1 The mass must be almost as big as the other ball 0.89

0 1 0 0.67 0.33

0.33 B2 Additional categories (in addition to CMU) are % of trials in which ~ set the masses as equal ~ set the masses as extremes ~ closer the the goal, further from the goal, ~ goal flips.

Hot Spot from BioLogica: Monohybrid (Task 3): produce only 2-legged offspring Arrow tool Cross tool Snip tool Chromosome tool

Requires changing both Legs alleles of one parent

The task Dragon genome chart Punnett square pad

Monohybrid Task 3 Subtasks & Data collected

• Predict whether a pair of dragons can have only 2-legged offspring – Multiple choice question • Describe the necessary parental genotypes.

– Full text response • Change alleles of one parent to homozygous recessive • Cross parents – Success = making right cross – Number of crosses made – List of crosses made

Data Monohybrid performances

• • Student performance is scored by computer based on – Prediction – Success – Number of attempts – Whether they repeated any crosses (an indication of haphazard behavior).

Performances can be grouped into – Systematic & correct – Systematic & incorrect – Haphazard & correct – Haphazard & incorrect

Systematic vs. haphazard performance and Pre/Post gains

Dependent Var iable: Total Score Post T3CATSYS2NUM Haphazard Systematic Mean 19.980

22.868

Std. Error .740

.651

95% Confidence Interval Lower Bound Upper Bound 18.518

21.582

21.442

24.154

Data based on 649 students in 10 member schools; (54.2%) in ‘regular’ classes.

ANCOVA with pre test score as covariate indicates pre test is significant (p ≤ .001), as is Table below indicates that Pre-test cova riate is significant, as is the two-catego ry p redictor variable (S, H). Toge ther, the cova riate and this variable accoun t for 25.2% of the var ia nce in the variance in the Post-test scores.

Post-test scores.

irrespective of whether they succeeded at the inquiry task.

Thus the systematic inquiry is facilitating knowledge building (as measured by the post test).

Overview of Data Analysis with Hot spots

• • We are aggregating hot spots and testing their relationship: conceptual learning measurements, i.e., pre-post content tests measures of students’ epistemologies of models and views of science since students’ epistemologies influence learning (Songer & Linn, 1991; Gobert & Discenna, 1997).

• • With these data, we can: track students’ systematicity in learning with models as one important facet of inquiry skills and conceptual learning. To us, inquiry skills co-evolve with content learning but each can be measured separately (sort of).

test for development of inquiry strategies across time and across domains ~ complicated by task difficulty increasing over time ~ complicated by the co-evolution of the development between domain-knowledge and inquiry strategies ~ complicated by the likelihood that students build knowledge in small, conceptual pieces, I.e., about acceleration or velocity).

In the future, using log files we seek to identify at risk students- i.e., students whose inquiry strategies are buggy.

Domain-General Inquiry Spots (“DoGI” spots)

1- Making predictions with models 2- Interpreting data from a representation (i.e., model/graph, pedigree, etc). 3- Making explanations (about models, etc) 4- Mathematizing with models- Filling in an equation/solve an equation; reasoning with an equation.

5- Designing and/or conducting an experiment with models.

Thus, if a student can do these types of tasks, they are doing model-based inquiry.