Using concept map approaches to communicate and present knowledge University of Oulu, Finland EDTECH A41857 (1 credit) – Challenges, Problems, & Future of EdTech Wednesday.
Download ReportTranscript Using concept map approaches to communicate and present knowledge University of Oulu, Finland EDTECH A41857 (1 credit) – Challenges, Problems, & Future of EdTech Wednesday.
Using concept map approaches to communicate and present knowledge University of Oulu, Finland EDTECH A41857 (1 credit) – Challenges, Problems, & Future of EdTech Wednesday March 30, 2005 Dr. Roy Clariana Penn State University email: [email protected] home: www.personal.psu.edu/rbc4 "First we build the tools, then they build us!" -- Marshall McLuhan 1 goals Your take aways: • Some experiences with collaborative concept • • mapping, mindmapping Some understanding of how/why it works Some examples that you could implement on Monday morning in your classroom or in you research Your Digital Portfolio for future reference and for sharing 2 1 credit option Digital portfolio – Formulate as a group a digital portfolio of mindmapping, which you may utilize in the future in your studies or work. • For teachers, provide specific examples for • using mind mapping in your classroom For researchers, provide specific examples for using mind mapping in your research 3 2 credit option? Digital Portfolio plus a White paper – a 5-10 page (double-spaced) persuasive review of some aspect of mind mapping, i.e., scripting MM in CSCL, MM as an artifact, etc. [Based on your intuition, describe how a MM can work, this is your first iteration of a “solution”. The White papers is a “soft sell” for your “solution” that describes the problem (90% of the document) and then states clearly how your solution solves the problem (10%). Avoid straw man arguments.] 4 If you are interested… Manuscript for presentation – I hope that we can publish this experience, i.e., based on several projects we will do, together we formulate questions, collect and analyze data, write… (this will likely go beyond the workshop time frame and is also more open-ended) For example: How does interaction develop/evolve in online collaborative mind mapping? What scripts can support online collaborative mind mapping? 5 Agenda for today Welcome and introductions all around Q&A Brief overview of concept maps Intro to Cmap tools software Brainstorm activity (group roles) Set up Project 1 (see handout) Set up Project 2 (see handout) Does anyone have any student essays that we can use in Project 3 on Monday? Click here for projects handout 6 Some foundation stuff some terminology Vygotsky Concept map – diagrams indicating interrelationships among concepts and representing conceptual frameworks within a specific domain of knowledge (vanBoxtel) contrast Concept map – a visual set of nodes and arcs (a network representation) that embodies the relationships among the set of concepts. Also called knowledge maps, mindmaps, semantic maps (Turns, et al.). Nodes – terms/complexes/concepts (usually nouns, things, examples, ideas, categories, people, locations…) Links (arcs) – lines connecting nodes, usually labeled with a relationship term (usually verbs) Propositions – node-link-node combinations, also called “soup” (ketti) by IHMC Turns, Atman, & Adams, 2000 7 Mindmaps vs. concept maps Bahr (2004) using concept maps to teach English to German students 8 sama työtapa liian pitkään vaihto kyllin usein demot, konkreettiset esimerkit tekeminen pelkkä kalvoshow työtavat vs. pelkkä kuunteleminen oppilaiden erot huomaa Mindmap of “group” knowledge hitaat nopeat erot tukiopetus. lisätehtäviä (Anni, Anna, Paula, Esa, ja Herkko), konkretisoi ! apu yllätä ! luokkakohtaiset source hallway erot opettajan kikkoja is the second floor oppettajan vaikutusmahdollisuudet muista huumori ! haasta, kysele ! oma tarina elävöittää kytke oppilaan arkeen ! liikuta oppilas ylös penkistä ikäluokka vaikuttaa vilkas luokka hiljainen luokka ei palautetta opettajalle erityisen paljon kikkoja näennäinen keskittyminen ? 9 Mindmaps vs. concept maps My question is, do concept maps or do mindmaps fit better with the Finnish language? 10 Tools to support mapping Yellow stickies!! Pencil and paper may be best for your classroom Software – PowerPoint is pretty good Inspiration is good but expensive CMAP tool is free, but your tech person will have to agree to support it At least 22 other tools are available, some free some not 11 Other concept map automatic scoring approaches CMap tools (IHMC) that we will use today C-TOOLS – Luckie (PI), University of Michigan NSF grant available: http://ctools.msu.edu/ctools/index.html TPL-KATS – University of Central Florida (e.g., Hoeft, Jentsch, Harper, Evans, Bowers, & Salas, 1990). TPLKATS: concept map: a computerized knowledge assessment tool. Computers in Human Behavior, 19 (6), 653-657. SEMNET – http://www.semanticresearch.com/about/ CMAT – Arneson & Lagowski, University of Texas, http://chemed.cm.utexas.edu Plus 22 other non-scoring map tools, Inspiration, Kidspiration 12 Some previous uses of mapping Usually involve individuals working alone, and involve text in some way Some collaborative strategies have been used Lets look at a few… 13 Using a student mindmap to “capture” a text (note taking) Mindmap notes Textbook text Text text text text text text text text text text text memo text text text Examples? student 14 Using a student mindmap to “capture” research on a topic text Mindmap notes Text text text text text tex Text text Text text text text tex text text Text text textt text text textt text memo text text www text Examples? video video student 15 Then using the mindmap to write an essay Mindmap notes essay text memo Text text text text text text text text text text text text text text Examples? student 16 Using a researcher drawn mindmap to “capture” an interview transcript Interview 1 Interview 1 text Text text text text text text text text text text text attribute theory note issue memo text text text The capability and experience of the person coding the text is critical… coder 17 Using a group drawn mindmap to “capture” an interview Interview 1 text text text text Qs The capability and experience of the person coding the text is critical… interviewer 18 Example of dyad collaboration Note the attentional effects of the artifact (not online) Mindmap artefact Verbal discussion (taped) Observations: On task Abstract talk 3-propositions/min Question Answer Criticize Conflict Elaboration Co-construction text Analyze the discussion text text Blah blah blah blah Blah blah text Blah blah blah blah Blah blah The incredible value of talk! Hannah Yergin Problem: Sometimes unscientific notions are ingrained Inferred: Active use of prior knowledge Acknowledged problems Look for meaningful relations Negotiation Shared objects play an important role in negotiation and co-construction van Boxtel, van der Linden, Roelofs, & Erkens (2002) 19 Chiu et al. example of an online collaboration Mindmap artefact Mindmap session lasted 80 minutes. 3 x 12 online groups, communicate by chat, 745 messages were exchanged (avg. of 62 per group). text text text Online chat H: WE should … J: Did you see… Y: Yeah, but … Etc. Etc. Jari The ‘other 2 members used chat to “advise” text creates Hannah (lead) Only the lead could alter the mindmap Researchers Analyzed the chat text And the mindmap Yergin p.22, Chiu, Huang, & Chang (2000) 20 Project 1 and 2 We will experiment with two online collaboration approaches Project 1 is a synchronous concept map collaboration using Cmap tools software Project 2 is an asynchronous concept map collaboration using PowerPoint software and email But next, we will try brainstorming with Cmap tools to become familiar with the tools and process before setting up Project 1 Click here for projects handout 21 Mindmap activity… First Mind map CSCL roles… Starter: You work as a discussion moderator. Your assignment is to engage your group members to the discussion by asking questions and commenting. And if the wrapper makes small summaries during discussion you can utilize his or her work to raise new questions. Active participation in the discussions is essential. Wrapper: Your assignment is to sum up the discussion. If you think it is easier you can summarize frequently and weave ideas together. For example, if five participants of your group are having a discussion about collaborative and co-operative learning you can summarize their main points during the discussion. An alternative way is to sum up the discussions in the end of article-videoclip task (and the last course assignment). Please overview your group's discussions and make a brief summary of the main topics. Active participation in the discussions is essential. Group member: Your assignment is to participate actively into discussions by asking questions making comments and stating arguments. You are expected to be a critical inquirer. Evaluator (an optional role): You are required to evaluate your group's work during the course. Please focus on the group interaction and group dynamics, for example how the starters, wrappers and group members performed during the discussions and last course assignment. The tutors inform you when to perform evaluations. Notice that you are also a deputy starter and a deputy wrapper if the originally named persons are not available. If you are called to work as a starter or wrapper please see the instructions given above. The role of evaluators are used only if you have not had a role of starter or wrapper during this course. 22 Cluster analysis Brainstorming Sorting (corpus list) (move like terms closer) Merging & Pruning (combine like terms, delete or move unlike terms, synthesize terms) Build consensus! Naming Clusters (name the categories/themes) and if necessary Sorting Clusters (move like clusters closer) E-document (to save/print) Naming broad themes (name the cluster of clusters) 23 Brainstorm, then make the map Open IHMC Cmap tools Fill in personal information on first use (I’ll tell you what to type in here) Click Other Places Open brainstorm file Click collaborate icon if necessary Type in your first name Collaborate 24 Now go back and Mindmap activity… add Small Group Roles Group Task Roles Initiator-contributor. Proposes new ideas or approaches to group problem solving; may suggest a different approach to procedure or organizing the problem-solving task Information seeker. Asks for clarification of suggestions; also asks for facts or other information that may help the group deal with the issues at hand Opinion seeker. Asks for clarification of the values and opinions expressed by other group members Information giver. Provides facts, examples, statistics, and other evidence that pertains to the problem the group is attempting to solve Opinion giver. Offers beliefs or opinions about the ideas under discussion Elaborator. Provides examples based on his or her experience or the experience of others that help to show how an idea or suggestion would work if the group accepted a particular course of action Coordinator. Tries to clarify and note relationships among the ideas and suggestions that have been provided by others Etc.. 25 Project 1 – Cmap tools synchronous collaboration Set day and time to join online ……. (see the Project handout) 26 Project 1 IHMC Public Cmaps conv v2 on Jan 22 2004 27 Project 1 Oulu EDTECH Public 28 Project 2 – Overview of “Pass the soup” PowerPoint file Email to Email to Email to Email to (see the Project handout) 29 Project 2 – “Pass the soup” PowerPoint file Slide 1 – mindmap is developed bit-by-bit here by the group by adding only 3 to 5 elements and then emailing it to the next person on the list 1. 2. 3. 4. Bob – [email protected] Mary – [email protected] Tiina – [email protected] Etc. Instructions: Add 3 or 4 components, pass to the next person… B: I decided to add blah and blah because I am interested in artifacts M: I deleted Bob’s blah because it is stupid, and then added blah T: I linked blah and blah etc… Slide 2 – numbered list of names of group members with email address, other instructions Slide 3, 4, etc. – comments about changes that you want to make, suggestions, etc. 30 How to use ALA-Reader Monday, April 4, 2005 31 Agenda for today Debrief “pass the soup” activity, and come up with a better Finnish name for it Q&A Brief overview of my concept map assessment research ALA-Reader demo (English language essays) Set up Project 3 for Finnish (see handout) How can we find Finnish essays for use in Project 3? 32 Final map for Project 2: Team 1 Why don’t we read from computer screens concept map? computer to communicate, paper to study easy to underline and write notes on paper paper more portable computer to store, paper to read with paper, easier to multi-task several paper pages simultanosly viewed and compared working options/possiblities and requirements easier to make good-looking slides and copies about drafts with computer Click her to See progression Of this map comp screen is smaller than paper appearance Computer screens Poor screen resolution (96 dpi) computers require paper has better contrast manual dexterity of child and adult feelings/perceptions computers require constant updates computers skills paper has weight, texture, and feel Group: Tanja, Henna & Roy familiarity 33 Final map for Project 2: Team 2 underline reliable but heavy to travel with reliable luminosity doesn’t need any hardware resolution eyes gets mixed up archive Paper /Hard copy paper easy to read for eyes Click her to See progression of this map headache Why don’t we read text from computers? shoulder problems never seen one personal preference e-book electronic documents easy to use light to carry/ travel technical problems multimedia copyright own comments possibility to store ergonomy screen size able to write notes decision to print e-text easy to copy/paste Group: Maria, Paivi & Roy etext feels ephemeral amount of text can make paper copies my own 1 page or less 2-10 pages different versions book feels comfortable connections print to paper screen 34 Debriefing What happened? What worked? What did not work? What would you do differently next time? If you like, write this up as a team for your final paper. 35 My research interests prototypes Mind map assessment – automatic scoring software tool called ALA-Mapper http://www.personal.psu.edu/rbc4/ala.htm Essay assessment – automatic scoring software tool called ALA-Reader http://www.personal.psu.edu/rbc4/score.htm for Latent Semantic Analysis (LSA) see: http://www.personal.psu.edu/rbc4/frame.htm 36 Novak Novak says “Concept maps were first developed in our research program in 1972 as a way to represent changes in children’s understanding of science concepts over the 12-year span of schooling. We were using modified Piagetian clinical interviews to assess changes in their knowledge over time, but we found the interview transcripts were too difficult to analyze for changes in specific aspects of the children’s knowledge. Instead we prepared concept maps from the interviews.” From: http://wwwcsi.unian.it/educa/mappeconc/jdn_an2.html 37 First uses… to represent knowledge in a visual format The primary parts of the system are the heart, blood tissue cells, within and the vessels. bodyThe by approximately human heart, a9pump, pints of is made bloodof The through cardiac human 100,000 muscle circulatory miles system of vessels is a transportation The Cardiac system. primary muscles Nutrients partshave of and thea system unique oxygenfeature are arethe carried heart, of to living forming blood tissue connections cells, within and the vessels. between bodyThe bytwo approximately human adjacent heart, cardiac a9pump, pints of cells. is made This bloodallows ofthrough cardiac the100,000 muscle miles cells toofcontract vessels powerfully The Cardiac primary andmuscles quickly partshave involuntarily of thea system unique feature are the heart, of Theforming brain blood isconnections cells, unableand to vessels. increase between The ortwo decrease human adjacent heart, the cardiac a pump, heart's cells. isbeating made This allows of cardiac the muscle cells to contract powerfully The heart Cardiac isand comprised muscles quickly of have involuntarily foura unique chambers; feature two of upper Theforming chambers brain isconnections unable called to atriums, increase between andortwo decrease lower adjacent the cardiac chambers heart's cells.beating called This ventricles allows the muscle cells to contract The blood powerfully The heart flowsisthrough and comprised quickly the right of involuntarily four side chambers; to the lungs two where upper The it picks chambers brainupisoxygen. unable called to atriums, The increase blood and then ortwo decrease returns lower the to thechambers right. heart's Next, beating called it flows ventricles into the left where it I xxxx The blood The heart flowsisthrough comprised the right of four side chambers; to the lungs two where upper it picks chambers up oxygen. called atriums, The blood and then tworeturns lower to thechambers right. Next, called it flows ventricles into the left where it I xxxx The blood flows through the right side to the lungs where it picks up oxygen. The blood then returns to the right. Next, it flows into the left where it I xxxx Novak interview data Was science content knowledge right ventricle left atrium pulmonary vein pulmonary artery lungs remove oxygenate CO2 blood Mind Map 38 Finnish research with concept maps… Mainly for knowledge representation for instructional use but also for representing the structure of a curriculum and for group communication Pasi Eronen, Jussi Nuutinenn and Erkki Sutinen, (http://www.cs.joensuu.fi/pages/avt/concept.htm), Joensuu (computer science) Mauri Ählberg, Helsinki (education) and Erkki Rautama (computer science) University of Art and Design, Helsinki (http://www2.uiah.fi/~araike/papers/articles/CinemaSense_Collaborativ e_Cinemastudies_DeafWay2002.htm) (see also: Future Learning Environment 3) Text graphs (Helsinki): http://www.cs.hut.fi/Research/TextGraph/ Kari Lehtonen, Helsinki Polytechnic, concept maps as a portfolio component (http://cs.stadia.fi/~lehtonen/DPF/dpf-berlin-02muotoiltu.doc) Also School astronomy and Vocational Training and Education 4th IEEE International Conference on Advanced Learning Technologies Joensuu, Finland, August 30 - September 1, 2004 39 Concept map for assessment: score validity??? oxygenate pulmonary CO2 lungs vein artery ventricle blood atrium left atrium pulmonary vein lungs remove oxygenate CO2 blood Concept maps contains propositions These propositions scores are generally considered to be valid and reliable measures of science content knowledge organization (Ruiz-Primo, Schultz, Li, Shavelson, CREST in California. . .). 40 e.g.,… Rye and Rubba (2002) reported that traditional concept map scores were related to California Achievement total test scores (r = 0.73). (Note that Crocker and Algina say that validation coefficients rarely exceed r=0.50.) Concept maps (cognitive maps, concept maps) may be an appropriate approach for assessing structural knowledge (Jonassen, Beissner, & Yacci, 1993). For example, concept maps have been used to visualize the change from novice to expert. 41 Scoring Concept Maps Traditionally, concept maps are scored by teachers or trained raters using scoring rubrics (e.g., Lomask’s rubric) Although this marking approach is time consuming and fairly subjective, map scores usually correlate well with more traditional measures of science content knowledge (multiple choice, fill-in-the blank, and essays) Complex scoring rubrics decrease the concept map score reliability (so keep scoring simple) 42 Scoring Concept Maps C3 describes our automatic system for scoring concept maps: collect –>convert –> compare 1. 2. 3. Collect raw map data Convert raw data into a mathematical network representation Compare the mathematical network representation of two maps (e.g., student to teacher, student to expert, student to student) 43 1. Collect raw data What raw data can a computer “extract” from a concept map? Term counts – in open-ended maps, count required terms included Propositions – a link connecting two terms and a link label Associations – geometric distance between pairs of terms. Small values indicate stronger relationship. 44 (n2-n)/2 pair-wise comparisons Link and distance data Link Array left atrium right ventricle to the pulmonary vein moves through pulmonary artery a b c d e f g left atrium lungs oxygenate pulmonary artery pulmonary vein deoxgenate right ventricle b c d e f g 1 1 1 1 0 0 0 0 0 0 0 1 0 0 0 - d e f g passes into Distance Array to the lungs deoxygenated a 0 0 0 1 0 0 oxygenated a b c d e f g left atrium lungs oxygenate pulmonary artery pulmonary vein deoxgenate right ventricle Most approaches use only link label information, usually called “propositions”. a 120 150 108 73 156 66 b 36 84 102 42 102 c 120 114 138 54 84 144 138 42 114 120 - 45 Link and distance Link data (propositions) – are the common way to compare/assess concept maps Distance data – not common, based on James Deese’s (1965) ideas on the structure of association in language and thought, cardsorting task approaches (Vygotsky in Luria, 1979, Miller, 1969), Kintsch and Landauer’s ideas on representing text structure, and neural network methods (Elman, e.g., 1995) 46 Using our Finnish Mind Map example Borrowed from Anni, Anna, Paula, Esa, ja Herkko Found in the hallway on the second floor See next slide 47 sama työtapa liian pitkään vaihto kyllin usein demot, konkreettiset esimerkit tekeminen pelkkä kalvoshow työtavat huomaa erot nopeat lisätehtäviä konkretisoi ! luokkakohtaiset erot yllätä ! opettajan oma tarina elävöittää pelkkä kuunteleminen oppilaiden erot oppettajan vaikutusmahdollisuudet muista huumori ! vs. kikkoja kytke oppilaan arkeen ! liikuta oppilas ylös penkistä haasta, kysele ! ikäluokka vaikuttaa tukiopetus. apu vilkas luokka hiljainen luokka ei palautetta opettajalle hitaat erityisen paljon kikkoja näennäinen keskittyminen ? 48 hiljainen luokka huomaa erot kikkoja luokkakohtaiset erot oppettajan vaikutus-mahdollisuudet oppilaiden erot työtavat vilkas luokka vältä Link array hiljainen luokka huomaa erot kikkoja luokkakohtaiset erot oppettajan vaikutus-mahdollisuudet oppilaiden erot työtavat vilkas luokka vältä -0 0 1 0 0 0 0 0 -0 1 1 1 0 0 0 -0 1 0 0 0 0 -0 0 0 1 0 -0 1 0 1 -0 0 0 -0 0 -0 -- Distance array hiljainen luokka huomaa erot kikkoja luokkakohtaiset erot oppettajan vaikutus-mahdollisuudet oppilaiden erot työtavat vilkas luokka vältä -127 245 79 214 161 234 73 302 -199 52 122 91 111 117 207 Collect Mind Map raw data -225 -100 164 -290 93 205 -175 164 76 166 -288 68 232 105 227 -114 252 88 282 122 320 -- 9 main terms selected here (ALA-Mapper max=30) 49 Selecting terms Selecting important terms (and their synonyms) is a critical step (for example, singular value decomposition in LSA derives terms). We use an expert(s) to determine terms. Goldsmith, Johnson, and Acton (1991) 50 predictive validity of PFNets directly relates to the number of terms used 0,80 predictive validity 0,70 0,60 0,50 0,40 0,30 So, perhaps the predictive validity of Concept Maps (and essays) directly relates to the number of terms used 0,20 0,10 0,00 0 10 20 30 Number of terms Goldsmith, Johnson, and Acton (1991) 51 2. Convert raw data into scores Currently, we use a data reduction and comparison approach called Pathfinder network representation (PFNet, Schanveldt, 1990). Our future research will consider additional approaches, such as MDS and data-mining. http://interlinkinc.net/Pathfinder.html PFNets describe the least weighted path to connect the terms Scores are established by comparing the participant’s PFNet to a referent (expert) PFNet, and calculating the number of common links (the intersection) Visual example 52 Finnish example: PFNet for distance data hiljainen luokka vilkas luokka luokkakohtaiset erot oppilaiden erot huomaa erot työtavat oppettajan vaikutusmahdollisuudet vältä kikkoja PFNet for distance data 53 Compare student to expert referent hiljainen luokka O vilkas luokka vilkas luokka hiljainen luokka luokkakohtaiset erot oppilaiden erot huomaa erot luokkakohtaiset erot oppilaiden erot huomaa erot 6 of 8 common links O työtavat oppettajan vaikutusmahdollisuudet oppettajan vaikutusmahdollisuudet työtavat kikkoja vältä Expert Referent PFNet vältä kikkoja Student PFNet 54 #1st Poindexter and Clariana Participants – 23 undergraduate students in intro EdPsyc course (Penn State Erie) Food rewards for participation Setup – complete a demographic survey and how to make a concept map lesson Text based lesson interventions – instructional text on the “heart” with either proposition specific or relational lesson approach Poindexter, M. T., & Clariana, R. B. (in press). The influence of relational and proposition-specific processing on structural knowledge and traditional learning outcomes. International Journal of Instructional Media, 33 (2), in press. link to doc file 55 Treatments Relational condition, participants were required to “unscramble” sentences (following Einstein, McDaniel, Bowers, & Stevens, 1984) in one paragraph in each of the five sections or about 20% of the total text content Proposition-specific condition (following Hamilton, 1985), participants answered three or four adjunct constructed response questions (taken nearly verbatim from the text) provided at the end of each of the five sections, for a total of 17 questions covering about 20% of the total text content (no feedback was provided). 56 Posttests Concept map (use 26 terms provided) Multiple-choice tests (Dwyer, 1976) • Link-based common scores • Distance-based common scores • Identification (20) • Terminology (20) • Comprehension (20) 57 Means and sd Treatments Posttests Map-link COMP Map-prop 7.3 14.1 (5.4) (4.6) control ID 15.1 (4.4) TERM 12.3 (4.6) Map-dist Map-assoc 9.0 (3.6) propositionspecific 16.3 (5.6) 14.6 (5.7) 13.8 (3.7) 16.5 (8.3) 11.5 (3.4) relational 17.0 (2.6) 12.7 (3.5) 12.4 (3.0) 13.9 (9.4) 10.7 (4.6) 58 Analysis MANOVA (relational, proposition-specific, and control) and five dependent variables including ID, TERM, COMP, Map-prop, and Map-assoc. COMP was significance, F = 5.25, MSe = 17.836, p = 0.015, none of the other dependent variables were significance. Follow-up Scheffé tests revealed that the proposition-specific group’s COMP mean was significantly greater than the control group’s COMP mean (see previous Table). 59 Correlations ID TERM COMP Map-prop Map-link Map-distance Map-assoc ID -0.71 0.50 0.56 0.45 TERM COMP Map-link Prop -0.74 0.77 0.69 -0.53 0.71 -0.73 All sig. at p<.05 Compare to Taricani & Clariana next 60 Taricani and Clariana – Replication of Poindexter and Clariana Term Comp Link data 0.78 0.54 Distance data 0.48 0.61 Taricani, E. M. & Clariana, R. B. (in press). A technique for automatically scoring open-ended concept maps. Educational Technology Research and Development, 53 (4), in press. 61 Compare these two . . . Taricani & Clariana Term Comp Link data 0.78 0.54 Distance data 0.48 0.61 Poindexter & Clariana Term Comp Link data 0.77 0.53 Distance data 0.69 0.71 62 # 2nd Clariana, Koul, & Salehi Participants – A group of 24 practicing teachers enrolled in CI 400 Lesson intervention – while researching online, completed concept maps in pairs (newsprint & yellow stickies) to describe the structure and function of the heart and then individually wrote essays on this topic from their maps. Clariana, R. B., Koul, R., & Salehi, R. (in press). The criterion related validity of a computer-based approach for scoring concept maps. International Journal of Instructional Media, 33 (3), in press. 63 Posttests Essays Multiple-raters using holistic rubric Computer-derived LSA Essay scores (http://www.personal.psu.edu/rbc4/frame.htm) Concept Maps Multiple-raters using Lomask’s rubric ALA-Mapper PFNet link and distance agreement with an expert 64 Correlation matrix Human Map Essay LSA Link data Distance data Map 1 0.49 0.31 0.36 0.60 Computer Essay LSA Link 1 0.73 0.76 0.77 1 0.83 0.71 1 0.82 1 p < .05 shown in boldface type. Many investigators have noted the close relationship between maps and essays. 65 Overview: Tools to score Essays ETS – PEG (Project Essay Grade), e-rater, Criterion and other products… http://www.ets.org/research/erater.html Walter Kintsch (and Landau) at CU-Boulder – Latent semantic analysis, many uses, i.e., score online training for the Army http://lsa.colorado.edu/ Vantage Learning essay scoring products http://www.vantagelearning.com/ ALA-Reader: http://www.personal.psu.edu/rbc4/score.htm 66 ALA-Reader Text PFNet … an electrical signal starts the heartbeat, by causing the atrium to contract. The blood then flows through the pulmonary valve into the pulmonary artery and then into the lungs. Once inside the lungs, the blood gives up the carbon dioxide (cleansed) and receives oxygen. This oxygenated blood … Link array atrium contract P valve P artery lungs cleansed oxygenated 67 # 3rd Clariana & Koul Participants – Again, a group of 24 practicing teachers enrolled in CI 400 Lesson – while researching the topic “the structure and function of the heart” online, students completed concept maps using Inspiration software and later wrote an essay on this topic from their maps. Clariana, R.B., & Koul, R. (2004). A computer-based approach for translating text into concept map-like representations. In A.J.Canas, J.D.Novak, and F.M.Gonzales, Eds., Concept maps: theory, methodology, technology, vol. 2, in the Proceedings of the First International Conference on Concept Mapping, Pamplona, Spain, Sep 14-17, pp.131-134. http://cmc.ihmc.us/papers/cmc2004-045.pdf 68 Posttests Essays Multiple-raters using holistic rubric Computer-derived LSA Essay scores (http://www.personal.psu.edu/rbc4/frame.htm) Concept Maps Multiple-raters using Lomask’s rubric ALA-Mapper PFNet link and distance agreement with an expert ALA-Reader PFNet link scores (from 1 to 5) (so far, only looked at essay scores) 69 ALA-Rater PFNet scores The scores for each text and rater-pair are shown ordered from best to worst. ALA-Reader scores were moderately related to the combined text score, Pearson r = 0.69, and ranked 5th overall. 70 Comments and Questions ?? 71 Demo ALA-Reader Download ALA-Reader.exe Create terms file (can include 2 synonyms) Create 2 expert baseline reference texts called expert1.txt and expert2.txt (i.e., Instructor, best student) Use it (type in the students essay file name) Files created • • Summary file called report.txt Multiple *.prx files (PRX folder) Available at: www.personal.psu.edu/rbc4 72 Other methods for eliciting and representing knowledge structure Monday, April 11, 2005 73 agenda Today is a hands-on demonstration day Brief overview of the ideas SPSS for representing Pathfinder KU-Mapper My intent, you will know enough to begin to use these approaches 74 Eliciting structural knowledge oxygenate pulmonary CO2 lungs vein artery ventricle blood atrium left atrium pulmonary vein lungs remove oxygenate CO2 blood Every method for eliciting knowledge should be viewed as “sampling” Caution, never forget the likely effects of contiguity (time, space, etc.) dominating over semantics (meaning) 75 Dave’s ideas Knowledge elicitation Knowledge representation Jonassen, Beissner, & Yacci (1993), page 22 Knowledge comparison 76 Dave’s ideas word associations semantic proximity similarity ratings card sort Knowledge elicitation relatedness coefficients ordered recall graph building quantitative graph comparisons free recall additive trees hierarchical clustering C of PFNets qualitative graph comparisons Knowledge comparison Knowledge representation Trees scaling solutions expert/ novice Dimensional Networks MDS – multidimensional scaling ordered trees minimum spanning trees link weighted Jonassen, Beissner, & Yacci (1993), page 22 Pathfinder nets principal components cluster analysis 77 Eliciting structural knowledge Vygotsky (in Luria, 1979); Miller (1969) cardsorting approaches Deese’s (1965) ideas on the structure of association in language and thought Kintsch and Landauer’s ideas on representing text structure, and latent semantic analysis Recent neural network representations (e.g., Elman, 1995) 78 Analyzing Deese free association data with MDS Hands-on with MDS in SPSS • A good description of MDS: • http://www.statsoft.com/textbook/stmulsca.html (Aside: a good description of Factor analysis: http://www.statsoft.com/textbook/stfacan.html ) Hands-on with Pathfinder KNOT 79 bug flower yellow fly bird wing insect moth Deese, free recall data (p.56) moth 100 12 12 12 11 1 0 4 insect 12 100 9 9 17 1 1 33 wing 12 9 100 44 19 0 0 3 are shown bird 12100 participants 9 44 100 21 a 1list of 0 3 time, and fly 11related 17 words, 19 one 21 at a100 1 1 8 asked to free recall a related term yellow 1 1 0 1 1 100 7 0 flower 0 1 0 0 1 7 100 2 bug 4 33 3 3 8 0 2 100 cocoon 11 10 2 2 6 0 0 7 color 0 1 0 1 1 17 3 0 Full array (n * n): 19 x 19 = 361 blue 0 1 0 1 2 23 7 0 Half array ((n – n)/2): ((19 x 19) –19 )/2 = 171 bees 2 in language 3 and thought. 10 Baltimore, 10 MD: John 6 Hopkins 2 Press, page 2 56 805 Deese, J. (1965). The structure of associations 2 moth insect wing bird fly yellow flower bug cocoon color blue bees summer sunshine garden sky nature spring butterfly butterfly spring nature sky garden sunshine summer bees blue color cocoon bug flower yellow fly bird wing insect moth Deese, free recall data (p.56) 100 12 12 12 11 1 0 4 11 0 0 2 2 5 1 1 1 1 15 12 100 9 9 17 1 1 33 10 1 1 3 0 0 0 0 1 0 12 12 9 100 44 19 0 0 3 2 0 0 10 0 0 0 0 3 0 13 12 9 44 100 21 1 0 3 2 1 1 10 0 1 0 1 5 0 12 11 17 19 21 100 1 1 8 6 1 2 6 0 3 0 2 4 0 11 1 1 0 1 1 100 7 0 0 17 23 2 2 7 5 2 4 3 5 0 1 0 0 1 7 100 2 0 3 7 2 1 6 18 2 6 2 6 4 33 3 3 8 0 2 100 7 0 0 5 0 0 0 0 2 0 4 11 10 2 2 6 0 0 7 100 0 0 4 1 1 1 0 2 0 22 0 1 0 1 1 17 3 0 0 100 32 0 0 2 0 8 0 0 0 0 1 0 1 2 23 7 0 0 32 100 1 2 4 4 46 3 2 2 2 3 10 10 6 2 2 5 4 0 1 100 1 2 3 0 4 2 7 2 0 0 0 0 2 1 0 1 0 2 1 100 5 2 0 1 10 0 5 0 0 1 3 7 6 0 1 2 4 2 5 100 2 3 2 15 4 1 0 0 0 0 5 18 0 1 0 4 3 2 2 100 0 4 4 2 1 0 0 1 2 2 2 0 0 8 46 0 0 3 0 100 0 1 0 1 1 3 5 4 4 6 2 2 0 3 4 1 2 4 0 100 2 3 1 0 0 0 0 3 2 0 0 0 2 2 10 15 4 1 2 100 2 15 12 13 12 11 5 6 4 22 0 2 7 0 4 2 0 3 2 100 Full array (n * n): 19 x 19 = 361 Half array ((n2 – n)/2): ((19 x 19) –19 )/2 = 171 Deese, J. (1965). The structure of associations in language and thought. Baltimore, MD: John Hopkins Press, page 56 81 Using MDS in SPSS Start SPSS and open the deese.sav file Under Analyze, select Scale, then select Multidimensional Scaling (ALSCAL)… Move Variable from left to right Create distances from data Model Next page Options 82 Select all of these 83 Multi-dimensional scaling (MDS) of Deese data Derived Stimulus Configuration Euclidean distance model 1,5 spring garden 1,0 summer sunshine nature bees Dimension 2 flower cocoon 0,5 butterfl 0,0 moth -0,5 yellow blue -1,0 fly insect color sky wing bird bug -1,5 -2 -1 0 1 Dimension 1 84 Side issue, the MDS obtains alternate (e.g., enantiomorphic) visual representations Helsinki Oulu Both are “correct”. Is this map correct? Tampere Pori Jyväsklyä Jyväsklyä Pori Tampere Helsinki Oulu geographic data, for example, may be oriented in different ways 85 Derived Stimulus Configuration How good is the representation? Euclidean distance model many dimensions (as many as 19) reduced to 2 dimensions Check the “stress” value to estimate how strained the results are 1,5 spring garden Dimension 2 1,0 summer sunshine nature bees flower cocoon 0,5 butterfl 0,0 moth -0,5 yellow blue -1,0 fly insect color sky wing bird bug -1,5 -2 -1 0 1 Dimension An algorithmic, power, approach rather than based on a model so 1 no assumptions about data structure are required… 86 Side trip Wordnet: http://wordnet.princeton.edu/ http://wordnet.princeton.edu/cgi-bin/webwn What is the Visual Thesaurus? – The Visual Thesaurus offers stunning visual displays of the English language. Looking up a word creates an interactive visual map with your word in the center of the display, connected to related words and meanings. Type “bird” in at: http://www.visualthesaurus.com/trialover.jsp 87 Pathfinder Network (PFNet) analysis Pathfinder is a mathematical approach for representing and comparing networks, see: http://interlinkinc.net/index.html Pathfinder data reduction is based on the least weighted path between nodes (terms), so for example, Deese’s 171 data points become 18 data points. Only the salient or important data is retained. Pathfinder PFNet uses, for example: • • • Library reference analysis Measuring Team Knowledge (Nancy J. Cooke) next slide Use google to see many more 88 Pathfinder for cognitive task analysis Shope, DeJoode, Cooke, and Pedersen (2004) 89 PFNet of same data Now let’s try Pathfinder analysis of the same Deese data set… Find the pfnet folder Double-click to run PCKNOT.bat (notice the bat extension, see next slide below) We will do it together 90 Select the right PCKNOT file 91 PFNet of Deese data sky summer blue spring sunshine color garden yellow flower nature butterfly cocoon moth wing bird bees fly insect bug 92 Derived Stimulus Configuration MDS and PFNet of Deese data Euclidean distance model 1,5 sky summer spring blue color garden 1,0 garden sunshine yellow flower butterfly cocoon wing bird fly nature bees cocoon nature 0,5 butterfl 0,0 moth moth -0,5 bees yellow blue -1,0 fly insect color sky insect bug sunshine flower Dimension 2 spring summer wing bird bug -1,5 Pathfinder KNOT PFNet -2 -1 0 SPSS 1MDS Dimension 1 93 MDS and PFNet data reduction MDS uses all of the data points to reduce the dimensions in the representation, and so may be improperly driven by noise in the data or by unimportant data points Pathfinder uses only the most important data 94 Transition to your real life example Finally, you will collect *real* data (using my KU-Mapper software) and analyze it with Pathfinder KNOT 95 KU-Mapper Your data, determine 15 important terms in your research area (Finnish and English), create a “terms.txt” file with the 15 terms Run KU-Mapper (do all 3 tasks: pairwise, list-wise, and card sort) Use KNOT to analyze and compare all three prx files Download KU-Mapper from: http://www.personal.psu.edu/rbc4/KUmapper.htm 96 Debrief your data activity What happened? What worked? What did not work? What would you do differently next time? If you like as your final paper, describe how you might use this approach. 97 Final thoughts… I enjoyed working with you If you want a credit, • Email to let me know this • Then be sure to send me you paper via email as soon as possible 98 Stop here 99 Amount of collaboration Possible research question on optimal scripts: Under- vs. over-scripting CSCL linear S-curve Amount of scripting Amount of scripting J-curve Amount of scripting Some possibilities 100 generative learning strategies + ++ Generative learning (Jonassen, 1988) recall - repetition, rehearsal, review, mnemonics integration - learner paraphrases, generates questions, generates examples organization - learner analyzes key ideas by creating headings, underlining keywords, outlining, categorizing (i.e., invent table categories, populate a table with existing ideas) elaboration - generate mental images, create physical diagrams, sentence elaboration (i.e., invent stuff to fill cells in a table) 101 I just "think" systemically and "ndimensionally" on paper, with imagery… My essential skill is simply--If you can explain it to me, I can draw a picture of it. It doesn't matter if it's something totally new to me, if you can make a coherent explanation, and let me understand it. I can "visualize it" and make a picture that shows you what you said. This is why I work in aerospace. I'm able to sit down with SME's (Subject Matter Experts-in any discipline), let them do a "data-dump" and put a sketch in their hand at the end of the conversation that "say's it all". This skill is vital to helping disparate technical types talk to each other (communication across cultural barrier of the "dialect" of the various technical disciplines). It also provides a way for ideas to get from that rough-semi coherent stage and into a practical and "do-able" condition. For example, One day I found myself working a Kelly Temp job for a bunch of Boeing System Analysts doing a JAD (joint application development) project to design a computing architecture for a new tooling system for the 777. The first drawing came by accident, started a huge argument, and eventually (2 weeks later) resolved in a group wide "a-hah"... that put everyone on the same wavelengthallowing the new system to be built a lot more "right" than usual, quicker than usual. From: http://visual.wiki.taoriver.net/moin.cgi/MichaelErickson 102