Transcript Document

Dimensions of
Media Object Compehensibility
Lawrie Hunter
Kochi University of Technology
http://www.core.kochi-tech.ac.jp/hunter/
Dimensions Island
of of Shikoku
Media Object Compehensibility
KUT
Lawrie Hunter
Kochi University of Technology
http://www.core.kochi-tech.ac.jp/hunter/
Niigata
Osaka
Kochi
A pattern language for MMC
Source of insight: language / language learning
Second language (L2) learning: a cognitive process?
Comprehension of partially acquired L2:
revealing of the nature of text/media.
Language learning issues are germane to MMC.
A pattern language for MMC
Background
work towards a human-communication
paradigm for the guidance of machines
Intervention
author's structural model of content
information (for second language
learning materials)
Frame
the new multidisciplinary approach
of machine-mediated communication
Discussion
of parameters of difficulty
Objective
development of a pattern language for
that multidisciplinary approach to MMC
Focus
factors influencing the difficulty of
comprehension of media objects
Question
how media objects carry information.
Ground:
related issues in
second language learning materials
Exemplars
‘considerate text’
‘considerate multimedia’
Generating parameters of difficulty in
media object comprehension
Tempering: questions of
significance and applicability for
machine automation.
L2 learning materials
The creation of second language (L2) learning materials demands
document transparency:
1. document purpose
2. document content
3. target behavior
4. target lexical items
must all be more immediately
apparent to the learner than
in the case of materials for
L1 medium learning scenarios
L2 learning materials
The creation of second language (L2) learning materials demands
document transparency:
1. document purpose
2. document content
3. target behavior
4. target lexical items
work towards transparency
is informed by
difficulty-related issues
difficulty-related issues
inform
human interaction with info media
Earlier work: an EAP tool
David Kolb* re using hypertext to present scholarly text:
"...the easiest ways of making a complex argument available in HT
tend to move the text toward linear structures
that do not take full advantage of the possibilities of linked text."
*in a talk to KMI at the Open University
Earlier work: an EAP tool
David Kolb* re using hypertext to present scholarly text:
"...the easiest ways of making a complex argument available in HT
tend to move the text toward linear structures
that do not take full advantage of the possibilities of linked text."
"...what the HT can do is present the argument,
but also use linkage and juxtaposition
to make the reader’s engagement with the argument
more creative, self-conscious, and self-critical."
*in a talk to KMI at the Open University
Earlier work: an EAP tool
Lawrie Hunter
re using hypertext to present technical L2 text:
For the L2 reader,
engagement can only be enhanced
if the rhetorical and information structures are articulated.
Earlier work: an EAP tool
Lawrie Hunter
re using hypertext to present technical L2 text:
For the L2 reader,
engagement can only be enhanced
if the rhetorical and information structures are articulated.
What the HT can do for the NNR/W is
tp present simultaneously the various faces of a research paper:
the rhetorical moves;
the bits of structured information;
the text;
necessary glosses.
* NNR/W EAP = non-native reader/writer of English for Academic Purposes
Earlier work: an EAP tool
Lawrie Hunter
re using hypertext to present technical L2 text:
For the L2 reader,
engagement can only be enhanced
if the rhetorical and information structures are articulated.
What the HT can do for the NNR/W is
to present simultaneously the various faces of a research paper:
the rhetorical moves;
the bits of structured information;
the text;
necessary glosses.
And if the NNR/Ws design their personal interface,
a negotiated pattern language of NNR/W EAP* will emerge.
* NNR/W EAP = non-native reader/writer of English for Academic Purposes
Arguably important direction
"Tomorrow's literacies...
need to be process and systems literacies.”
-John Thackara,
In the Bubble: Designing in a complex world.
MIT Press 2005.
Structural view of writing
Train of argument
This is the
domain of
structures
up here.
Rhetorical
structures
Extension
Functional
structures
Knowledge
structures
Cohesion
devices
This is the
domain of
texture
down here.
Diversions
Grammar
(sentence surface structure)
Background
Rhetorical structure theory,
systemic functional linguistics and
knowledge structure mapping
form a hierarchy of structures,
whereas grammar and sentence diagrams
reflect rules for texture management.
Structural view of writing
Train of argument
This is the
domain of
structures
up here.
False
hierarchy:
the train
stops here.
Rhetorical
structures
Extension
Functional
structures
Knowledge
structures
Cohesion
devices
This is the
domain of
texture
down here.
Diversions
Grammar
(sentence surface structure)
Background
Rhetorical structure theory,
systemic functional linguistics and
knowledge structure mapping
form a hierarchy of structures,
whereas grammar and sentence diagrams
reflect rules for texture management.
L2 reader needs analysis
Knowledge
Language skills
Facilities
Niche
grammar structures
Argument
sequencing
Concordance &
collocation resource
Niche
rhetorical structures
Info-structured
sentence generation
Bank of model
research papers
(annotated*)
General register
repertoires
(distinguishing
formal academic from
informal academic)
Mimicry of model
language
Research Paper
text structure and
information structure
*c.f. Brown and Brown’s ‘annotation’
L2 reader wants analysis
In a technical hypertext, L2 reader/writers want*:
1. Glossing (of 'difficult' terms and phrases)
2. Moves indicator
3. Lexia position indicator
4. PDF-drawer-like phrase recurrence tab
5. Register converter
(e.g. research paper <=> presentation script)
6. Information structure maps for atomic utterances
7. Overall argument map on every lexia
(similar to Horn's argument maps
or Rhetorical Structure Analysis?)
*Based on a survey of 22 PhD engineering students
Technical hypertext design:
NEEDS
A pattern language?
WANTS
www.patternlanguage.com
Technical hypertext design:
NEEDS
A pattern language?
WANTS
www.patternlanguage.com
…The language, and the processes which stem from it,
merely release the fundamental order which is native to us.
They do not teach us, they only remind us of what we know
already, and of what we shall discover time and time again,
when we give up our ideas and opinions, and do exactly
what emerges from ourselves.
-Christopher Alexander,
The Timeless Way of Building
Do humans have a
GRAPHIC THOUGHT FACILITY?
<
$$$
!
The knowledge structure map is a matrix (confluence) for the
situated learner* and the situated mentor to confirm context
and the nature of "stolen property."**
*Jean Lave
**Duguid and Brown
Hunter’s knowledge structure map links
Classification
Description
Degree
comparison
<
Attribute
comparison
big
Contrast
!
Sequence
Cause-effect
2005 project: design level
EEAP* students: HT designs
for the analysis of technical academic papers.
*EEAP = Engineering English for Academic Purposes,
a subset of EAP,
which is a subset of ESP (English for Specific Purposes)
Hunter L. (2005) Technical Hypertext Accessibility: Information Structures and Rhetorical Framing.
Presentation at HyperText 2005, Salzburg.
http://www.lawriehunter.com/presns/%20HT05poster0818.htm
TEXT
STRUCTURE
RHETORICAL
MOVES
Introduction
Common
knowledge
Background
Question
Methods and
materials
Results
Conclusion
INFO
STRUCTURE
Describe
Cite
Report
Explain
Compare
Claim
Sequence
Question
Qualify
Evaluate
Observations
INFOMAP(s)
Decide
Infer
Project
Classify
UTTERANCE(s)
In general, power plants boil some
liquid to make steam, which rotates
turbines, which generate electricity.
Power plants boil a liquid to
produce steam, which is used to
rotate turbines, which in turn
generate electricity.
Cause-effect
Contrast
TEXT
STRUCTURE
RHETORICAL
MOVES
Introduction
Common
knowledge
Background
Question
Methods and
materials
Results
Conclusion
INFO
STRUCTURE
Describe
Cite
Report
Explain
Compare
Claim
Sequence
Question
Qualify
Evaluate
Observations
INFOMAP(s)
Decide
Infer
Project
Classify
UTTERANCE(s)
Traditional power plants use fossil
fuel heat or heat from nuclear
fission to boil water and produce
steam at 500°C.
Older type power plants boil water
with heat from fossil fuel
combustion or nuclear fission to
produce steam with a temperature
of 500°C.
Cause-effect
Contrast
TEXT
STRUCTURE
RHETORICAL
MOVES
Introduction
Common
knowledge
Background
Question
Methods and
materials
Results
Conclusion
INFO
STRUCTURE
Describe
Cite
Report
Explain
Compare
Claim
Sequence
Question
Qualify
Evaluate
Observations
INFOMAP(s)
Decide
Infer
Project
Classify
UTTERANCE(s)
OTEC power plants use seawater
heat to boil ammonia and produce
steam at 20°C.
OTEC type power plants boil
ammonia with the heat of the sea to
produce steam with a temperature
of 20°C.
Cause-effect
Contrast
TEXT
STRUCTURE
RHETORICAL
MOVES
Introduction
Common
knowledge
Background
INFOMAP(s)
Describe
Cite
Report
Explain
Compare
Claim
Sequence
Classify
Question
Methods and
materials
Question
Qualify
Results
Evaluate
Observations
Conclusion
INFO
STRUCTURE
Decide
Infer
Project
UTTERANCE(s)
Traditional power plants use fossil fuel
heat or heat from nuclear fission to boil
water and produce steam at 500°C,
whereas OTEC type power plants boil
ammonia using the heat of the sea to
produce steam with a temperature of
20°C.
Older type power plants boil water with
heat from fossil fuel combustion or
nuclear fission to produce steam with a
temperature of 500°C, while OTEC
power plants use seawater heat to boil
ammonia and produce steam at 20°C.
Cause-effect
Contrast
Obstacle in 2005 project
Massive diversity in learner perception of knowledge structures.
Obstacle in 2005 project
Massive diversity in learner perception of knowledge structures.
Structural view of writing
Train of argument
Diversions
Rhetorical
structures
Extension
This is the
domain of
structures
up here.
Functional
structures
Knowledge
structures
Cohesion
devices
This is the
domain of
texture
down here.
Grammar
(sentence surface structure)
Background
False
hierarchy:
the train
stops here.
Rhetorical structure theory,
systemic functional linguistics and
knowledge structure mapping
form a hierarchy of structures,
whereas grammar and sentence diagrams
reflect rules for texture management.
Structural view of writing
Grammar
staging
Information
orchestration
Rhetoric, flow
Prescriptive order
charts (linear);
sentence diagrams
Knowledge
structure maps
Topic/stress and
subject-verb
distance
gizmos
Paragraph level
Readability charts
Knowledge
structure maps
Old/new and
topic/stress
gizmos
Document level
Readability
outlines
Knowledge
structure maps
Old/new and
topic/stress
gizmos
Sentence level
2006~ new layer: READABILITY
The missing link in
technical academic
writing:
Gopen’s readability
-subject-verb distance
-topic position / stress position
-old/new information placement
Background: readability work
In the design of traditional high-text language learning materials,
readability is a prominent concern.
Reading difficulty has for some time been seen as depending on
-word length
-sentence length
-text length
-number of sentences per paragraph
-vocabulary ‘difficulty’
More recent work has extended this list to include
-subject-verb distance
-adherence to old/new position conventions
-topic position/stress position conventions
Treated extensively in
Hunter L. (1998) Text Nouveau: Visible Structure in Text Presentation. Computer Assisted Language Learning 11(4) pp. 363-379.
Background: MM readability
Chun, D. M. and Plass, J. L. 1997.
Research on text comprehension in multimedia environments.
Language learning and technology 1(1): 60-81.
Treated extensively in
Hunter L. (1998) Text Nouveau: Visible Structure in Text Presentation. Computer Assisted Language Learning 11(4) pp. 363-379.
2006~ new layer: READABILITY
P age
Hunter’s new
TAW syllabus:
assume grammar
1
Readability
and cohesion
Topic / stress positions
Old / new information
Subject-verb separation
Logic gaps
Ambiguity
2
Usage
Dictionaries, guides, corpus and concordance
3
Registers
Formal academic
Informal academic
Casual
4
Abstracts and
introductions
The structure of a paper
Outlining
Summarizing
5
Organization
of
information
Situation-problem-solution-evaluation
General-Specific
6
Information
structures,
information
mapping
Description
Classification
Comparison, includingpie and bar graphs
Sequence, including line and bar graphs
Cause-Effect
Inference (deduction/induction)
P ro andCon
7
Rhetoric vs.
information
Backgroundinformation /new content
8
English
models
9
Data
commentaries
10
Appendix:
language
features
The Style Dossier: model language selection / evaluation
Mimicry skills
P lagiarism avoidance
TAW-related grammar points
Usage points
2006~ new layer: READABILITY
P age
Textural
Grammar
Register
1
Readability
and cohesion
Topic / stress positions
Old / new information
Subject-verb separation
Logic gaps
Ambiguity
2
Usage
Dictionaries, guides, corpus and concordance
3
4
5
Cohesion
Structural
Lexical patterns
Registers
Formal academic
Informal academic
Casual
Abstracts and
introductions
The structure of a paper
Outlining
Summarizing
Organization
of
information
Situation-problem-solution-evaluation
General-Specific
Information
structures,
information
mapping
Description
Classification
Comparison, includingpie and bar graphs
Sequence, including line and bar graphs
Cause-Effect
Inference (deduction/induction)
P ro andCon
Rhetoric vs.
information
Backgroundinformation /new content
Knowledge structures
Coherence/readability
6
Functional grammar
Information organization
7
Rhetorical device
Rhetorical structure
8
English
models
9
Data
commentaries
10
Appendix:
language
features
The Style Dossier: model language selection / evaluation
Mimicry skills
P lagiarism avoidance
TAW-related grammar points
Usage points
Readability
The creation of second language (L2) learning materials demands
appropriate readability.
1. understandable by the learner
2. ‘stretching’ learner knowledge/skill
3. contextualized to support stretching
4. orchestrated with degrees of scaffolding
Considerate text
One inroad to readability is
considerate text:
Original framing:
･well-written,
･well-organized, and
･signals the organization of its thought to the reader
Considerate text
One inroad to readability is
considerate text:
Original framing:
･well-written,
･well-organized, and
･signals the organization of its thought to the reader
More recent takes:
-glossing
-phrase boundary marking
-de-idiomatizing
-the Plain English movement
-graphic organizers
-text nouveau
Text nouveau is still text
Text comprehension in multimedia environments is
a rich variant, BUT :
Chun, D. M. and Plass, J. L. 1997.
Research on text comprehension in multimedia environments.
Language learning and technology 1(1): 60-81.
Text comprehension in multimedia environments is
a rich variant, BUT :
Text nouveau is still text
Chun, D. M. and Plass, J. L. 1997.
Research on text comprehension in multimedia environments.
Language learning and technology 1(1): 60-81.
Sharing considerate text
L2 learning objects: welcome to the TagTower of Babel!
KUT English
is a Moodle department.
Appropriateness of learning materials/tasks is very complex.
Tagging of these materials & tasks is daunting.
Sharing considerate text
Appropriateness of learning materials/tasks is very complex.
Tagging of these materials & tasks is daunting.
Fortunately, as David Weinberger points out*,
there is a huge amount of metadata out there,
but this allows multiple simultaneous organizations of content.
*June 12, 2007 interview with IT Conversations
http://www.itconversations.com/shows/detail1838.html
Weinberger books
The Cluetrain Manifesto
Small Pieces Loosely Joined
Everything is Miscellaneous
Considerate multimedia?
Considerate text in the context of M3C suggests
the notion of considerate multimedia
Tentative definition:
considerate multimodal objects are
those which contain few
non-essential obstacles to their comprehension.
Considerate multimedia?
Considerate text in the context of M3C suggests
the notion of considerate multimedia
Tentative definition:
considerate multimodal objects are
those which contain few
non-essential obstacles to their comprehension.
Tentative definition 2:
considerate multimodal objects are
those which are tagged
for various forms of comprehension difficulty.
Considerate multimedia?
Multimedia comprehensibility?
“Considerate multimedia” confronts vastly more complexity
than considerate text
Alternative approach: create a set of
parameters for multimedia comprehensibility
One approach to comprehensibility:
explore obstacles to comprehensibility,
as has been done in readability work.
Multimodal equivalent of readability
In the domain of multimodal computer-mediated communication,
the question of readability translates as ease of comprehension:
How easy is it for a human
to extract all the information
contained in a multimodal media object (MMO)?
Parameters of media object function
To measure the ease of extraction of
all the INTENDED information contained in a MMO,
we need a characterization of the difficulty of extraction:
Tentative
list of sources of information extraction difficulty,
for simplicity’s sake limited here to
text objects
graphic objects
speech objects
video objects
and combinations thereof.
This is a tentative, exploratory framing of MMO comprehensibility,
Parameters of media object function
Learning object tag ‘shopping list’
Parameter
Instance/unit
concept density
exophoric references per paragraph/page/frame
metaphor density
metaphors per scene/argument/minute
phoneme density
phonemes per unbroken utterance* (e.g. Italian speech)
phonemes per inhalation
phonemes per word
phonemes per minute
mathematical symbol density
numerals per page
numerals per sentence
formulae per sentence/paragraph
formulae per argument
noise density
superfluous signals per utterance, e.g. "...in 1960, oh, sorry, I meant to say in 1960...)
readability stoppages** per sentence
asides per sentence/message
cognitive dissonances per utterance
facial expression/statement conflicts
reference transparency
anchoring devices per lexia
anchoring devices per reference
channel-channel synchronicity
number of channel-channel synchronicities
number of channel-channel asynchronicities
message-message agreement
number of message-message agreements
number of message-message dischords
*utterance: minimal spoken, written or graphical communication unit
Parameters of media object function
Concept density
Concept density* in text space
Concept density* in aural time
Concept density* in video space
*concept density =
number of exophoric references
per sentence/minute/frame
Parameters of media object function
Metaphor density
Metaphors per sentence.
Metaphors per argument.
Metaphors per minute.
Idioms per sentence.
Idioms per argument.
Idioms per minute.
Parameters of media object function
Phoneme density
Phonemes
Phonemes
Phonemes
Phonemes
per
per
per
per
unbroken utterance*.
exhalation.
word.
minute.
*e.g. Italian speech.
Parameters of media object function
‘Noise’ density
Superfluous signals/utterance
Readability ‘stoppages’ per sentence
Asides per sentence/message
Cognitive dissonances per utterance
Facial expression-statement conflicts
Parameters of media object function
Symbol density
Numerals per page.
Numerals per sentence.
Formulae per sentence/paragraph.
Formulae per rhetorical move.
Parameters of media object function
Channel imperfections
Imperfect audio channel
Imperfect text channel
Imperfect visual channel
Parameters of media object function
Channel imperfections
Finding:
humans retain more info
from video with an
imperfect audio channel
Nass and Brave, Wired for speech
Reeves and Nass, The media equation
Parameters of media object function
Channel-channel discord
Parameters of media object function
Channel-channel synchronicity
Parameters of media object function
Message-message harmony
Message-message agreements
Message-message discords
manga vs. knowledge structure maps
Both involve reduced text density and spatial highlighting of text,
and suggest the question of a 'graphic thought facility' in humans.
manga
knowledge structure maps
Low phoneme density
Low phoneme density
Isolated conversational text chunks: X
idioms per sentence.
Isolated descriptive text chunks:
0 idioms per sentence
X metaphors per utterance
0 metaphors per utterance
X idioms per utterance
0 idioms per utterance
Graphical situating: narrative/mood
Graphical situating:
symbolized relations to other text chunks
To illustrate the use of the parameter approach, here is a comparison of two relatively similar
types of media objects, manga and knowledge structure maps. Both involve reduced text
density and spatial highlighting of text, and suggest the question of a 'graphic thought facility' in
humans.
Generating parameters of difficulty in
media object comprehension
Tempering: questions of significance
and applicability for machine automation
Once a comprehensive set of parameters of MMO comprehensibility has been
developed, questions of application will arise.
How can (should?) these parameters be situated among larger semantic
frameworks?
Which of these parameters are relevant to the development of machinemediated communication?
How can they be operationalized in computable form?
Daunting: ontology-based readability
Work on ontology-based research writing * :
reforming how scientific research is written/read.
EXPO* and the Robot Scientist
Does the ontology EXPO
feed back
from a machine interface with a body of knowledge/practice
to a solidification of human interface with that body of knowledge/practice?
EXPO: An Ontology of Scientific Research. Ross D. King & Larisa N. Soldatova
http://www-tsujii.is.s.u-tokyo.ac.jp/jw-tmnlpo/RossKing.pdf
Daunting: ontology-based readability
Work on ontology-based research writing * :
reforming how scientific research is written/read.
“Use of Natural Language is a great hindrance
when using computers to store and analyse data
hence the growing importance of text-mining.
We argue that the content of scientific papers
should increasingly be expressed in formal languages.
Is writing a scientific paper closer to
writing poetry or a computer program?”
EXPO: An Ontology of Scientific Research. Ross D. King & Larisa N. Soldatova
http://www-tsujii.is.s.u-tokyo.ac.jp/jw-tmnlpo/RossKing.pdf
Daunting: ontology-based readability
Work on ontology-based research writing * :
reforming how scientific research is written/read.
Can humans now experience knowledge differently,
thanks to machine interface work,
i.e. through a formal language imposed for the machine’s sake?
Will this reform how we read? how we think?
EXPO: An Ontology of Scientific Research. Ross D. King & Larisa N. Soldatova
http://www-tsujii.is.s.u-tokyo.ac.jp/jw-tmnlpo/RossKing.pdf
References
[1] Elsayed, A. (2007) Machine-mediated communication: the technology. 6th IEEE International
Conference on Advanced Learning Technologies, ICALT 2006, 5-7 July 2006, Kerkrade, The
Netherlands.
[2] Hunter, L. (2005) Technical hypertext accessibility: information structures and rhetorical framing.
Proceedings of the sixteenth ACM conference on Hypertext and hypermedia, Salzburg, Austria.
[3] Kalyuga, S. (2006) Instructing and testing advanced learners: A cognitive approach. Nova Science
Publishers.
[4] Mann, B. (1999) An introduction to rhetorical structure theory (RST).
http://www.sil.org/mannb/rst/rintro99.htm
[5] Mohan, B.A.M. (1986) Language and content. Reading, MASS: Addison-Wesley.
[6] Nass, C. and S. Brave. (2005) Wired for speech: How voice activates and advances the humancomputer relationship. MIT Press.
Chun, D. M. and Plass, J. L. 1997. Research on text comprehension in multimedia environments.
Language learning and technology 1(1): 60-81.
Grow, G. (1996) Serving the strategic reader: cognitive reading theory and its implications for the
teaching of writing. Viewed June 30, 2007 at
http://www.longleaf.net/ggrow/StrategicReader/index.html
Goldman, S.R., & Rakestraw, J.A. (2000). Structural aspects of constructing meaning from text. In M.L.
Kamil, P. B. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. II, pp.
311-335). Mahwah, NJ: Erlbaum.
The Plain English movement http://www.plainenglish.co.uk/index.htm (de-idiomatizing)
References 2
Research via ontologies
Ian Horrocks http://www.cs.man.ac.uk/~horrocks/
EXPO Ontology of scientific experiments http://expo.sourceforge.net/
Soldatova L.N., Clare A., Sparkes A. and King, R.D. (2006) An ontology for a Robot Scientist.
Bioinformatics (Special issue ISMB) (in press).
Soldatova, LN & King, RD. (2006) An Ontology of Scientific Experiments. Journal of the Royal Society
Interface (in press).
EXPO: An Ontology of Scientific Research by Ross D. King & Larisa N. Soldatova, Department of
Computer Science, University of Wales, Aberystwyth.
Hunter
Hunter L. (2005) Technical Hypertext Accessibility: Information Structures and Rhetorical Framing.
Presentation at HyperText 2005, Salzburg.
http://www.lawriehunter.com/presns/%20HT05poster0818.htm
Text Nouveau: Visible Structure in Text Presentation. Computer Assisted Language Learning 11(4) pp.
363-379. (text nouveau)
WordbyWord http://www.core.kochi-tech.ac.jp/hunter/WordByWord/index.html (text nouveau)
Text usability for non-native readers of English. Ueta, R, Hunter, L. & Ren, X. Proceedings, Information
Processing Society of Japan, Vol. 2003.7. Pp. 199-200. (phrase boundary marking)
Thank you
for your kind attention.
Don’t hesitate to write to me.
Lawrie Hunter
Kochi University of Technology
http://www.core.kochi-tech.ac.jp/hunter