Patterns in Education: Linking Theory to Practice Theodore Frick Department of Instructional Systems Technology School of Education Indiana University Bloomington Oct.
Download ReportTranscript Patterns in Education: Linking Theory to Practice Theodore Frick Department of Instructional Systems Technology School of Education Indiana University Bloomington Oct.
Patterns in Education: Linking Theory to Practice Theodore Frick Department of Instructional Systems Technology School of Education Indiana University Bloomington Oct. 13, 2006 Patterns in Education, AECT 2006 1 Overview of APT&C Analysis of Patterns in Time and Configuration: APT&C Fundamental change in perspective for measurement and analysis Bridges quantitative and qualitative paradigms APT for temporal patterns (both joint and sequential occurrences of events) APC for structural patterns (configurations) Oct. 13, 2006 Patterns in Education, AECT 2006 2 Overview cont’d APT&C based on mathematical theories and general systems theory Value of APT&C is that results can be directly related to practice Through APT&C we have new ways of conducting educational research Oct. 13, 2006 Patterns in Education, AECT 2006 3 Outline of this presentation The dilemma: qualitative vs. quantitative methodologies Three examples of empirical studies that used APT&C: Academic learning time (APT joint occurrences) Patterns of mode errors in human-computer interfaces (APT sequential occurrences) Student autonomy structures in a Montessori classroom (APC patterns of student choice of work and guidance of learning) Oct. 13, 2006 Patterns in Education, AECT 2006 4 Quantitative vs. Qualitative Paradigms Research methods in 20th century were largely quantitative. Qualitative and mixed methods are gaining more use in research during past two decades. Main problems: Quantitative methods seldom yield significant results that can be directly linked to educational practice (due to large within-group variances in experiments or treatments) Qualitative methods can provide good insights into practice, but conclusions are often restricted (low generalizability due to sampling strategy, and may or may not transfer to similar situations) Oct. 13, 2006 Patterns in Education, AECT 2006 5 Three Empirical Studies to Illustrate Value of APT&C Academic learning time of mildly handicapped children (Frick, 1990) Patterns of mode errors in humancomputer interfaces (An, 2003) Student autonomy structures in a Montessori classroom (Koh, 2006) Oct. 13, 2006 Patterns in Education, AECT 2006 6 Study # 1: Academic Learning Time Study 25 systems observed in central and southern Indiana Tracked 25 target students in academic activities over several months for 8 -10 hours each Trained observers coded types of academic learning contexts, task difficulty and task success Observers also coded student and instructor behaviors in math and reading (about 500 time samples at one-minute intervals for each target student) Nearly 15,000 time moments sampled overall. Oct. 13, 2006 Patterns in Education, AECT 2006 7 What observers coded in math and reading activities each minute Types of student engagement: written, oral, and covert on-task; off-task behaviors (later recoded as engagement, EN, and nonengagement, NE) Types of instructor behaviors: structuring, explaining, demonstrating, questioning, feedback (later recoded as direct instruction, DI), and monitoring academic seatwork (nondirect instruction, ND). Observer comments to elaborate what was happening Oct. 13, 2006 Patterns in Education, AECT 2006 8 Observer coding form Oct. 13, 2006 Patterns in Education, AECT 2006 9 Codes for target student moves Oct. 13, 2006 Patterns in Education, AECT 2006 10 Codes for instructor moves and focus Oct. 13, 2006 Patterns in Education, AECT 2006 11 Standard analysis: columns 1 and 2: independent measures of DI and of EN were correlated (n = 25) p(DI) p(EN) p(DI & EN) p(DI & NE) p(ND & EN) p(ND & NE) p(EN|DI) p(EN|ND) 0.50 0.39 0.27 0.34 0.48 0.40 0.44 0.36 Etc. Mean (SD) 0.432 (0.144) 0.80 0.49 0.56 0.69 0.73 0.75 0.84 0.75 Etc. Mean (SD) 0.741 (0.101) 0.46 0.37 0.26 0.34 0.47 0.39 0.40 0.33 0.04 0.02 0.01 0.00 0.01 0.01 0.04 0.03 0.34 0.12 0.30 0.35 0.25 0.35 0.44 0.42 0.16 0.49 0.43 0.31 0.26 0.25 0.11 0.22 0.92 0.95 0.97 1.00 0.98 0.98 0.91 0.92 0.67 0.20 0.41 0.53 0.49 0.59 0.80 0.65 Mean (SD) 0.416 (0.139) Mean (SD) 0.015 (0.010) Mean (SD) 0.324 (0.114) Mean (SD) 0.243 (0.104) Mean (SD) 0.967 (0.029) Mean (SD) 0.573 (0.142) Oct. 13, 2006 Patterns in Education, AECT 2006 12 Linear Models Approach Linear models approach (quantitative method): Relates independent measures through a mathematical function Treats deviation from model as error variance Oct. 13, 2006 Patterns in Education, AECT 2006 13 Linear Models Approach cont’d Oct. 13, 2006 Patterns in Education, AECT 2006 14 Linear models results: Means and standard deviations Mean p(DI) = 0.432 s.d. = 0.144 Mean p(EN) = 0.741 s.d. = 0.101 Regression equation EN = 0.57 + 0.40DI R2 = 0.33 DI “explains” 33 percent of the variance in student engagement; 67 percent unexplained Oct. 13, 2006 Patterns in Education, AECT 2006 15 Analysis of Patterns in Time APT measures a relation directly by counting occurrences of when a temporal pattern is true or false in observational data Probability of joint or sequential occurrence can be estimated for a pattern from the counts Oct. 13, 2006 Patterns in Education, AECT 2006 16 APT Results for same 25 systems: includes measures of joint and conditional occurrences p(DI) p(EN) p(DI & EN) p(DI & NE) p(ND & EN) p(ND & NE) p(EN|DI) p(EN|ND) 0.50 0.39 0.27 0.34 0.48 0.40 0.44 0.36 Etc. Mean (SD) 0.432 (0.144) 0.80 0.49 0.56 0.69 0.73 0.75 0.84 0.75 Etc. Mean (SD) 0.741 (0.101) 0.46 0.37 0.26 0.34 0.47 0.39 0.40 0.33 Etc. Mean (SD) 0.416 (0.139) 0.04 0.02 0.01 0.00 0.01 0.01 0.04 0.03 Etc. Mean (SD) 0.015 (0.010) 0.34 0.12 0.30 0.35 0.25 0.35 0.44 0.42 Etc. Mean (SD) 0.324 (0.114) 0.16 0.49 0.43 0.31 0.26 0.25 0.11 0.22 Etc. Mean (SD) 0.243 (0.104) 0.92 0.95 0.97 1.00 0.98 0.98 0.91 0.92 Etc. Mean (SD) 0.967 (0.029) 0.67 0.20 0.41 0.53 0.49 0.59 0.80 0.65 Etc. Mean (SD) 0.573 (0.142) Oct. 13, 2006 Patterns in Education, AECT 2006 17 APT Results Means and standard deviations for the relations Mean p(EN | DI) = 0.967 s.d. = 0.029 Mean p(EN | ND) = 0.573 s.d. = 0.142 When direct instruction is occurring, students are highly engaged. When non-direct instruction is occurring they are less engaged. Students were 13 times more likely to be off-task during non-direct instruction compared with direct instruction: (1 - 0.573) / (1 – 0.967) = 12.94. Oct. 13, 2006 Patterns in Education, AECT 2006 18 APT: joint occurrence calculation example Time Instr. Eng. 1:00 DI EN 1:01 DI NE 1:02 DI EN 1:03 ND NE Oct. 13, 2006 p(DI) = ¾ = 0.75 p(ND) = ¼ = 0.25 p(EN) = ½ = 0.50 p(NE) = ½ = 0.50 p(DI & EN) = 2/4 = 0.50 p(DI & NE) = ¼ = 0.25 p(ND & EN) = 0/4 = 0.0 p(ND & NE) = ¼ = 0.25 p(EN|DI) = 2/3 = 0.67 p(EN|ND) = 0/1 = 0.00 Patterns in Education, AECT 2006 19 LMA vs. APT Linear models relate the independent measures by a function for a line: e.g., EN = 0.57 + 0.40DI APT measures the relation in terms of joint, conditional, or sequential occurrence: e.g., p (EN|DI) = 0.967 e.g., p (EN|ND) = 0.573 DI = direct instruction, EN = student engagement, ND = non-direct instruction Oct. 13, 2006 Patterns in Education, AECT 2006 20 Study #2: Patterns of Mode Errors in HCI Software mode: when the same action results in two or more outcomes (Raskin, 2000). E.g., In one context, pressing the ‘d’ key results in the letter ‘d’ echoed on the screen In another context, pressing the ‘d’ key results in deleting a file. Mode errors by humans can cause serious problems: Destruction of important work Decreased productivity Not able to complete tasks Modes occur in almost all modern human-computer interfaces (e.g., OS 10, Windows XP, Word, Photoshop, etc.) Oct. 13, 2006 Patterns in Education, AECT 2006 21 An (2003) study of mode errors Mixed methods approach (usability evaluation, qualitative and quantitative) 16 college students performed eight computer tasks with three modern GUI interfaces (word processor, address book, image editor). Participants were videotaped, and stimulatedrecall interviews were conducted immediately afterwards to clarify why certain actions were taken, when viewing their videos. Oct. 13, 2006 Patterns in Education, AECT 2006 22 An (2003) study of mode errors (cont’d) Over 280 problematic actions were observed, and 52 were problems due to mode errors 52/280 = .19, or roughly 1 out of 5 problems were due to software modes Three general patterns (conditions) of mode errors emerged from qualitative analyses: Type A: Right action, wrong result Type B: It isn’t there where I need it Type C: It isn’t there at all Oct. 13, 2006 Patterns in Education, AECT 2006 23 An (2003) study of mode errors (cont’d) Source of error analysis revealed that mode errors appeared to result from 8 types of design incongruity: Unaffordance Invisibility Misled expectation Unmet expectation Mismatched expectation Inconsistency Unmemorability Over-automation Oct. 13, 2006 Patterns in Education, AECT 2006 24 An (2003) study of mode errors (cont’d) Consequences of mode errors: Can’t find hidden function Can’t find unavailable function False success Stuck performance Inhibited performance Inefficient performance Oct. 13, 2006 Patterns in Education, AECT 2006 25 APT: analysis of sequential patterns of mode errors, sources and consequences Relative Frequency Likelihood (p) 34 out of 52 0.65 AND IF source of mode error IS unaffordance, THEN consequence IS can’t find hidden function OR false success? 15 out of 34 0.44 10 out of 15 0.67 b) AND IF source of mode error IS invisibility, THEN consequence IS stuck performance? 6 out of 34 5 out of 6 0.18 0.83 c) AND IF source of mode error IS misled expectation, THEN consequence IS false success? 7 out of 34 6 out of 7 0.21 0.86 Query Type A 1 a) IF type of mode error IS right action, wrong result, Oct. 13, 2006 Patterns in Education, AECT 2006 26 APT: analysis of sequential patterns of mode errors, sources and consequences Relative Frequency Likelihood (p) 8 out of 52 0.15 AND IF source of mode error IS mismatched expectation, THEN consequence IS can’t find hidden function? 8 out of 8 8 out of 8 1.00 1.00 Query Relative Frequency Likelihood (p) 10 out of 52 0.19 Query Type B 2 a) Type C 3 IF type of mode error IS it isn’t there where I need it, IF type of mode error IS it isn’t there at all, a) AND IF source of mode error IS unmet expectation, THEN consequence IS can’t find unavailable function? 10 out of 10 10 out of 10 1.00 1.00 b) AND IF source of mode error IS unaffordance, THEN IF source of mode error IS unmet expectation, THEN consequence IS can’t find unavailable function? 3 out of 10 3 out of 3 3 out of 3 0.30 1.00 1.00 Oct. 13, 2006 Patterns in Education, AECT 2006 27 APT: analysis of sequential patterns of mode errors, sources and consequences APT results have practical implications E.g., if the mode error is ‘right action, wrong result’ and if the source of the error is unaffordance (function not obvious), then 67 percent of the time users could not find a hidden function or thought they did the task correctly when in fact they had not (false success). Oct. 13, 2006 Patterns in Education, AECT 2006 28 APT Methodology: sequential occurrence When one event precedes another, and when observers code the order in which events occur: APT can estimate the probability of the consequent following the antecedent event. APT can estimate likelihoods of sequences longer than two (unlike Markov chains). APT can estimate both joint and sequential event occurrences in complex combinations. Oct. 13, 2006 Patterns in Education, AECT 2006 29 APT Coding (temporal configuration) Clock Time 9:01 9:02 9:03 9:04 9:05 9:06 9:07 9:08 9:09 9:10 9:11 9:12 9:13 Oct. 13, 2006 Target Student Mona Instruction Direct Student Engagement Off-task On-task Off-task On-task Non-Direct Off-task Null Null Null Patterns in Education, AECT 2006 30 APT Classifications and Categories Each column is a classification Classifications co-exist in time Categories of events within a classification cannot co-exist in time (since they are mutually exclusive, by definition) An observer codes event changes within each classification in the order that they occur. Date/time is always a classification and is recorded whenever there is an event change. Oct. 13, 2006 Patterns in Education, AECT 2006 31 Example of sequential coding with three classifications Clock Time 9:01 9:02 9:03 9:04 9:05 9:06 9:07 9:08 9:09 9:10 9:11 9:12 9:13 Oct. 13, 2006 Target Student Mona Instruction Direct Student Engagement Off-task On-task Off-task On-task Non-Direct Off-task Null Null Null Patterns in Education, AECT 2006 32 APT Query: IF target student IS Mona? Clock Time 9:01 9:02 9:03 9:04 9:05 9:06 9:07 9:08 9:09 9:10 9:11 9:12 9:13 Oct. 13, 2006 Target Student Mona Instruction Direct Student Engagement Off-task On-task Off-task On-task Non-Direct Off-task Null Null Patterns in Education, AECT 2006 Null 33 APT Query and Results Query IF target student IS Mona? Results Cumulative duration = (9:13 – 9:01) = 12 minutes Cumulative frequency = 1 event Likelihood = 1 out of 1 relevant event changes = 1.00 Proportion time = 12 minutes out of 12 = 1.00 Oct. 13, 2006 Patterns in Education, AECT 2006 34 APT Query: IF target student is Mona AND instruction is direct? Clock Time 9:01 9:02 9:03 9:04 9:05 9:06 9:07 9:08 9:09 9:10 9:11 9:12 9:13 Oct. 13, 2006 Target Student Mona Instruction Direct Student Engagement Off-task On-task Off-task On-task Non-Direct Off-task Null Null Null Patterns in Education, AECT 2006 35 APT Query Results Query IF target student IS Mona AND instruction IS direct? Results Cumulative duration = (9:08 – 9:01) = 7 minutes Cumulative frequency = 1 event Likelihood = 1 out of 2 relevant event changes = 0.50 Proportion time = 7 minutes out of 12 = 0.583 Oct. 13, 2006 Patterns in Education, AECT 2006 36 APT Query: IF target student IS Mona AND instruction IS direct, THEN student engagement IS on-task? Clock Time 9:01 9:02 9:03 9:04 9:05 9:06 9:07 9:08 9:09 9:10 9:11 9:12 9:13 Oct. 13, 2006 Target Student Mona Instruction Direct Student Engagement Off-task On-task Off-task On-task Non-Direct Off-task Null Null Patterns in Education, AECT 2006 Null 37 APT Query Results Query IF target student IS Mona AND instruction IS direct, THEN student engagement IS on-task? Results Cumulative duration = (9:06 – 9:03) + (9:08 – 9:07) = 4 minutes Cumulative frequency = 2 Likelihood = 2 out of 4 = 0.50 Proportion time = 4 minutes out of 6 = 0.667 Oct. 13, 2006 Patterns in Education, AECT 2006 38 APT Query Syntax Oct. 13, 2006 Patterns in Education, AECT 2006 39 APT Syntax (cont’d) Oct. 13, 2006 Patterns in Education, AECT 2006 40 APT Syntax (cont’d) Oct. 13, 2006 Patterns in Education, AECT 2006 41 APT Query Syntax Thus, simple to very complex temporal patterns can be specified within APT queries. Joint and/or sequential occurrences of events can be specified. Results include frequency counts, likelihood estimates, durations and proportions of total time. Oct. 13, 2006 Patterns in Education, AECT 2006 42 Theoretical Foundations of APT Mathematical theory Information theory Set theory Probability theory Classifications (more than one, non-exclusive) Categories within each classification must be mutually exclusive and exhaustive General systems theory SIGGS Theory Model Oct. 13, 2006 Patterns in Education, AECT 2006 43 Advantages of APT APT brings theoretical rigor to pattern identification in qualitative research. APT measures relations not possible in quantitative methods such as the linear models approach. APT requires a different kind of conceptual framework for measurement and analysis than those for qualitative and quantitative approaches. Oct. 13, 2006 Patterns in Education, AECT 2006 44 APC: Analysis of Patterns in Configuration Thompson (2005) realized that APT could be extended to measure and analyze structure of systems. Structure pertains to relationships among parts. Oct. 13, 2006 Patterns in Education, AECT 2006 45 Familiar Patterns: Structural Geographical relation: Bloomington is located in southern Indiana on the North American continent. Bloomington is south of Indianapolis. Organizational relation: Gerardo Gonzalez is University Dean of the School of Education who directs and supervises: Oct. 13, 2006 Peter Kloosterman, Executive Associate Dean, SoE, IUB campus Khaula Murtahda, Executive Associate Dean, SoE, IUPUI campus Patterns in Education, AECT 2006 46 Familiar Patterns: Structural Familial relation: Philip and Irma Frick are the parents of Theodore Frick William and Helen Brophy are the parents of Kathleen Brophy Instructional relation: During fall semester, 2005,T. Frick was the R690 instructor of: Oct. 13, 2006 Andrew, Omer, Shyamasri, Nichole, Jamison, Sunnie, Emmanuel, Uvsh, Chris, Theano Patterns in Education, AECT 2006 47 A pattern is a relation General form of a relation: Oct. 13, 2006 Patterns in Education, AECT 2006 48 Temporal & Structural Patterns & Logical Relations Temporal Patterns Structural Patterns or Configurations A precedes B A co-occurs with B A affect relation B Logical Relations A implies B A is equivalent to B Oct. 13, 2006 Patterns in Education, AECT 2006 49 Affect relation: guides research of Faculty Person 2 Faculty Person 1 Student 3 Student 1 Student 2 Student 4 Student 5 Old IST Ph.D. structure Oct. 13, 2006 Patterns in Education, AECT 2006 50 Affect relation: guides research of Faculty Person 2 Faculty Person 1 Student 3 Student 1 Student 2 Student 4 Student 5 New IST Ph.D. structure Oct. 13, 2006 Patterns in Education, AECT 2006 51 APC allows us to measure structural properties of di-graphs Property Count Value Active Dependence 1.00 paths 5.97 Centrality 4.00 paths 23.89 Compactness 9.00 paths 53.76 Complete Connectedness 0.00 paths 0.00 Complexness 5.00 paths 5.00 Etc. Etc. Etc. Oct. 13, 2006 Patterns in Education, AECT 2006 52 Study #3: Autonomy structures in a Montessori classroom (Koh, 2006) Case study to explore Montessori classroom structures that support student autonomy Observed on 10 occasions for about an hour at different times of morning session (1 head teacher, 2 assistant teachers, 28 students ages 10-12) Ethnographic approach initially Oct. 13, 2006 Patterns in Education, AECT 2006 53 Koh (2006) study cont’d Class activities were built around two different activity structures: Head problems Morning work period Koh was interested in two kinds of affect relations: s chooses work y y guides learning of s Oct. 13, 2006 Patterns in Education, AECT 2006 54 Koh (2006) study cont’d Digraphs were drawn for affect relation structures during Head Problems and during Morning Work Period APC software was used to calculate structure measures of these digraphs (Frick & Thompson, 2006) Oct. 13, 2006 Patterns in Education, AECT 2006 55 Koh (2006) study cont’d Structures measured: Active dependence Centrality Complexity Independence Interdependence Complete connectivity Oct. 13, 2006 Patterns in Education, AECT 2006 56 Active Dependence: definition and measure Active dependent-component partition, ADS, =df a partition, Y = (VGO,RGA), characterized by initiating component affect-relations. S =df Y | vi,vjY(V )rd(I)(e)Y(R)[e = (vi,vj) rd(I)(e) = 1] AD M : Active dependent-component partition measure, M (ADS), =df a measure of initiating affect-relations. M (ADS) =df [(i=1,…,n[j=1,…,mdI(j)(v) log2|Ai|]) n] 100 Oct. 13, 2006 Patterns in Education, AECT 2006 57 APC Results from Koh (2006) study Morning Work Period Head Problems Property Value 60 50 40 30 20 10 0 Active dependence Centrality Complexity Independence Interdependence Complete connectivity Structural Property Oct. 13, 2006 Patterns in Education, AECT 2006 58 APC Results from Koh (2006) study (cont’d) Active dependence higher in Head Problems vs. Morning Work Period Centrality higher in Head Problems vs. Morning Work Period Interdependence lower in Head Problems vs. Morning Work Period Complexity lower in Head Problems vs. Morning Work Period Oct. 13, 2006 Patterns in Education, AECT 2006 59 APC Results from Koh (2006) study (cont’d) The structure of the Morning Work Period supported student autonomy During the Morning Work period there was: Less active dependence No centrality Greater complexity Greater interdependence Oct. 13, 2006 Patterns in Education, AECT 2006 60 APC Results from Koh (2006) study (cont’d) The 3 teachers’ responses to the Problems in Schools Questionnaire (SDT, 2006) showed them to be “highly autonomy supportive”. Student responses to the Academic SelfRegulation Questionnaire (SDT, 2006) indicated a greater tendency to undertake learning activities because they perceived some personal value and identification with the learning goals, rather than because they felt compelled by external factors. Oct. 13, 2006 Patterns in Education, AECT 2006 61 APC Results from Koh (2006) study (cont’d) The structural configuration of the Morning Work Period, where students chose learning activities and worked at their own pace is characteristic of Montessori classrooms. The structural configuration of the Head Problems activity chosen by the head teacher with all students working on the same problems, is more typical of traditional K-12 classrooms in the U.S. APC allowed analysis and comparison of structural properties of those two configurations of affect relations. Oct. 13, 2006 Patterns in Education, AECT 2006 62 Summary APT allows measurement and analysis of temporal properties Joint occurrences Sequential occurrences Combinations of joint and sequential occurrences Oct. 13, 2006 Patterns in Education, AECT 2006 63 APT: joint occurrence example Oct. 13, 2006 Time Instr. Eng. 1:00 DI EN 1:01 DI NE 1:02 DI EN 1:03 ND NE Patterns in Education, AECT 2006 64 APT joint and sequential occurrence example Clock Time 9:01 9:02 9:03 9:04 9:05 9:06 9:07 9:08 9:09 9:10 9:11 9:12 9:13 Oct. 13, 2006 Target Student Mona Instruction Direct Student Engagement Off-task On-task Off-task On-task Non-Direct Off-task Null Null Patterns in Education, AECT 2006 Null 65 Summary APC allows measurement and analysis of structural properties Oct. 13, 2006 Patterns in Education, AECT 2006 66 APC allows measures of structural properties of an affect relation (e.g., guides research of) Faculty Person 2 Faculty Person 1 Student 3 Student 1 Student 2 Student 4 Student 5 New IST Ph.D. structure Oct. 13, 2006 Patterns in Education, AECT 2006 67 APC property measures and values Property Count Value Active Dependence 1.00 paths 5.97 Centrality 4.00 paths 23.89 Compactness 9.00 paths 53.76 Complete Connectedness 0.00 paths 0.00 Complexness 5.00 paths 5.00 Etc. Etc. Etc. Oct. 13, 2006 Patterns in Education, AECT 2006 68 Summary: APT&C Analysis of Patterns in Time and Configuration permits measurement and analysis of human learning and work environments. The value of APT&C methodology was illustrated by clear results from three empirical studies. These results have direct implications for practice. APT&C is a way to link theory to practice. Software is under development to do APT&C. Oct. 13, 2006 Patterns in Education, AECT 2006 69 Questions For more information on APT&C: http://www.indiana.edu/~aptfrick Oct. 13, 2006 Patterns in Education, AECT 2006 70