Transcript How to Measure Information

How to Measure Information:
- from Action to Cognition Matthias Rauterberg
Department of Industrial Design
Technical University Eindhoven
2004
2/35
Possible interpretations of ‘information'
1.) ‘information' as a message
(syntax)
2.) ‘information' as the meaning of a message
(semantic)
3.) ‘information' as the effect of a message
(pragmatic)
4.) ‘information' as a process
5.) ‘information' as knowledge
(e.g. database)
6.) ‘information' as an entity of the world (next to energy and matter)
Ref: Folberth, O. & Hackl, C. (1986, eds.) Der Informationsbegriff in Technik und Wissenschaft. München: Oldenbourg.
© M. Rauterberg, 2004
3/35
“Information” for learning systems
before reception
after reception
author
DoF of the decision
content of the decision
HARTLEY 1928
uncertainty
SHANNON 1949
uncertainty
BRILLOUIN 1964
potential information
ZUCKER 1974
entropy
TOPSØE 1974
certainty
© M. Rauterberg, 2004
information
actual information
amount of information
4/35
Incongruity and learning
complexity
situation-1
learning
positive
incongruity
mental model
negative
incongruity
situation-2
context
human
incongruity = complexity
context – complexity
human
Ref: Rauterberg, M. (1995). About a framework for information and information processing of learning systems. In: E. Falkenberg, W. Hesse &
A. Olive (eds.), Information System Concepts--Towards a consolidation of views (IFIP Working Group 8.1, pp. 54-69). London: Chapman&Hall.
© M. Rauterberg, 2004
5/35
The complete human action cycle
task(s)
goal-, sub-goal-setting
planning of execution
feedback
control of
action
© M. Rauterberg, 2004
selection of means
mental operation
not direct
observable
physical operation
direct observable
6/35
The basic idea
Any human task solving process can be described in a finite
state-transition chain, if the task can be described in an ‘action
space’, specified by a finite set of States and Transitions.
s0
s1
d
s2
a
s3
s3
_
© M. Rauterberg, 2004
F3
s3
s3
s3
_
s3
_
F2
TAB
s3
CR
s3
TAB
s1
F9
s0
h
State description:
Action description:
s0 : main menu
s1 : modul "data"
s2 : routine "browse"
s3 : "wrong input" state
_ : ascii key "BLANK"
a : ascii key "a"
d : ascii key "d"
h : ascii key "h"
CR: carriage return
F2: function key "2"
F9: function key "9"
TAB:tabulator key
7/35
The folding operation in Petri nets
s0
d
s1
s1
h
s0
s1
a
s2
s2
F3
s3
s0
h
d
s1
s3
F9
s1
s3
CR
s3
s3
_
s3
TAB
s3
F2
s3
s3
s3
elementary processes
© M. Rauterberg, 2004
a
folding
F9
_
TAB
s2
F3
s3
CR
F2
Petri-Net
8/35
Task description
In the experiment all 12 users had to play the role of a camping place manager. This manager uses a database
system with a data base consisting of three data files: PLACE, GROUP, and ADDRESS. All users had to solve the
following four different tasks operating the database system:
Task 1: "How many data records are in the file ADDRESS, in the file PLACE, and in the file GROUP? Find
out, please."
The user has to activate a specific menu option ("Datafile" in module "Info" of the menu interface) and to read the
file size (solutions: PLACE = 17 data records, GROUP = 27 data records, ADDRESS = 280 data records).
Task 2: "Delete only the last data record of the file ADDRESS, the file PLACE, and the file GROUP (sorted
by the attribute 'namekey')."
The user has to open (sorted according to the given attribute), select and delete the last data record (file: PLACE,
GROUP, ADDRESS).
Task 3: "Search and select the data record with the namekey 'D..8000C O M' in the file ADDRESS, and
show the content of all attributes of this data record on the screen. Correct this data record for the
following attributes: State: Germany, Place number: 07. Remarks: Database system dealer can give a
demonstration."
The user must select a certain data record (file: ADDRESS), update the data record with regard to the three
attributes: State, Place number, Remarks.
Task 4: "Define a filter for the file PLACE with the following condition: all holidaymakers arrived on date
02/07/87. Apply this filter to the file PLACE, and show the content of all selected data records in the mask
browsing mode on the screen."
The user must define a filter for the attribute "arrival date", apply the filter to the data file PLACE, and display the
content of each data record found on the screen.
© M. Rauterberg, 2004
9/35
System description
The dialog system was the relational data base system (ADIMENS version 2.21)
with a character oriented user interface (CUI) running on standard IBM PC's
with standard keyboard.
The whole dialog structure is strictly hierarchical organized with three levels:
(1)
the main menu has 7 dialog operations (ordinary ASCII characters chosen from a menu) to go
down to 7 different modules, and 5 function keys with specific semantics;
(2)
at the module level each module has exactly 4 different dialog operations to change to
routines and on average 4.1 (±1.7; range: 0-5) function keys with specific semantics;
(3)
at the routine level the user has only on average 3.7 (±2.9; range: 0-10) different function
keys to control the dialog (additionally all ASCII keys and the 4 cursor keys are usable).
The number of all ordinary dialog contexts (main menu, modules, routines) is
1+7*4=29.
But to describe the complete dialog structure with all help, error and additional
dialog states we needed at least 144 different system states.
To change from one state to the other the system offers overall 358 different
dialog operations (= transitions).
© M. Rauterberg, 2004
10/35
Observable data of human behavior
task
description
result
goal-, sub-goal setting
goal
action planning
evaluation
and control
selection of means
mental execution
system
state
state
© M. Rauterberg, 2004
physical execution
transition
selected
action
11/35
Example of a task solving process
G_2
Mai n menu
marked
M_3
Star t menu
Automatic
tr ansi tion
... continues
... continues
... continues
... continues
1
Mai n menu.
F fi le
Mai n menu
i
Mai n menu
d
Info
Automatic
tr ansi tion
User
key pr ess
User
key pr ess
User
key pr ess
User
key pr ess
M_22
M_22
M_22
M_11
d
Info.fil e
Info.screen2
Info.screen1
DB content
display
DB content
display
M_22
M_22
Info.fil e
Info.screen3
DB content
display
M_22
Info.screen2
Info.screen1
DB content
display
DB content
display
M_22
BL
F_10
Info.fil e
Info.screen1
stopped
Info.screen1
Info
Automatic
tr ansi tion
User
key pr ess
M_11
d
Info.screen3
DB content
display
Info
Automatic
tr ansi tion
User
key pr ess
M_22
M_22
Info.screen2
Info.screen1
DB content
display
User
key pr ess
User
key pr ess
DB content
display
DB content
display
M_11
h
h
G_2
F_10
Info
Info.screen3
Automatic
tr ansi tion
© M. Rauterberg, 2004
F_3
Mai n menu
User
key pr ess
Star t menu
User
key pr ess
Star t menu
Automatic
tr ansi tion
Star t menu
User
key pr ess
12/35
How to extract the user’s mental model?
s0
main menu level
h
d
unknown structure
(e.g., mental model)
structure
as a
Petri net
s1
module level
?
b
F9
CR
TAB
routine level
s2
F3
s3
_
1
2
s0
observable
process
s1
d
s2
b
s3
s3
_
CR
F3
s3
_
s3
F2
TAB
s3
_
FOLDING
s3
s3
s3
© M. Rauterberg, 2004
F2
TAB
s1
F9
s0
h
13/35
How to measure complexity?
In Computer Science...
•
•
•
•
(e.g., Solomonoff-Kolmogorov-Chaitin)
algorithmic information
computational universality
computational time/space
according McCabe in graph theory
In Physics...
•
•
•
•
•
•
thermodynamics potentials
long-range order
long-range mutual information
self-similar structures
thermodynamic depth
logical depth
In Psychology...
•
•
•
© M. Rauterberg, 2004
properties of objects
properties of attributes
properties of cognitive structure
(e.g. valence)
(e.g. cardinality)
(e.g. centrality)
14/35
Net complexity metrics
Stevens, Myers and Constantine (1974):
T = number of transitions
S = number of states
simple Petri net:
McCabe (1976):
[with P=1]
State-1
Kornwachs (1987):
Transition-1
Transition-2
State-2
Validation study:
Ccycle from McCabe outperforms all other metrics!
Ref: Rauterberg, M. (1992). A method of a quantitative measurement of cognitive complexity. In: G. van der Veer, M. Tauber, S. Bagnara
& M. Antalovits (eds.), Human-Computer Interaction: Tasks and Organisation--ECCE'92 (pp. 295-307). Roma: CUD.
© M. Rauterberg, 2004
15/35
The program structure of AMME
transformation
to a syntactical
correct logfile
interactive
dialog
system
automatic
recorded
process
Automatic
Mental
Model
Evaluation
USER
system
description
"defaultp.ps"
"*.str"
"*.log"
the analyzing program AMME
"*.net"
v2132 13
S'initial_state' 15@-12
390@330 S 0 0
Tnil nil 480@420
S 1 2 nil 0
cS CSCS cSsS SSRS rS
5 tftt 10 ft
"*.ptf"
0
1
2
3
0
0
1
0
0
1
1
0
0
1
2
0
1
0
0
3
0
0
1
1
"*.mkv"
0
1
2
3
0
0
3
0
0
1
1
0
0
2
2
0
1
0
0
3
0
0
1
7
state transition net adjacency matrix frequency matrix
• simulation
• task-subtask
Petri net simulator
PACE
© M. Rauterberg, 2004
• similarity
• learning
• MDS
Path finder
KNOT
• distances
• personal styles
• MDS
Markov analyzer
SEQUENZ
"*.ps"
"*.pro"
ascii text
graphic
outputfile in outputfile with
PostScript
quantitative
format
measures
• interface
design
• deadlocks
any Postscript
interpreter
• complexity
• routine
any text
processor
In an overview Ivory and
Hearst (2001) compared 132
usability evaluation and
modeling methods worldwide;
19 different modeling methods
are based on logfile analysis:
“AMME is the only surveyed
approach that constructs a
WIMP simulation model (Petri
net) directly from usage data”
(Ivory and Hearst 2001, p. 499).
Therefore they conclude,
“AMME appears to be the most
effective method, since it is
based on actual usage” (2001,
p. 502).
16/35
Behavioral Complexity (BC) according McCabe (1976)
BCcycle = T – S + 1
Box Plot
Box Plot
40
40
B Cc y c l e
*
Experiment:
N=6 novices; N=6 experts
4 tasks with a database
Metric BC=Ccycle
*
30
30
*
*
*
20
20
10
10
0
1
2
beginners - experts
SOURCE
0
Ref: Rauterberg, M. (1993). AMME: an
Automatic Mental Model Evaluation to
analyze user behaviour traced in a finite,
discrete state space. Ergonomics, vol.
36(11), pp. 1369-1380.
1
SUM-OF-SQUARES
2
3
task no.
DF
4
MEAN-SQUARE
F-RATIO
P
experience
275.521
1
275.521
10.337
0.00
tasks
259.563
3
86.521
3.246
0.03
25.729
3
8.576
0.322
0.81
1066.167
40
26.654
exp.
x
ERROR
© M. Rauterberg, 2004
tasks
17/35
From analyzing to modeling
Adding g oal
setting
structure
Device model
Human mental model
Reconstructed
mental task model
3
?
F_ 10
MsDO S
Observation
of human
behaviour
G_ 2 M_ 3 h
St ar t
menu
F_ 3 1
Main
menu
i h
Main menu
F-f ile
F_ 10
Adding seq uential
and temporal
information
2
Folding
MsDO S
G_ 2 M_ 3
St ar t m enu
h
F_ 3
i
i
1
h
Main menu
F-f ile
Main menu
Model
execution
4
1
5
G_ 2
M_ 3
F_ 3
G_ 2
Ms DOS
Ma in men u
Sta rt men u
Au toma ti c
tra ns i ti on
Au toma ti c
tra ns i ti on
Ms DOS
Us er
ke y p re ss
Au toma ti c
tra ns i ti on
M_ 3
Sta rt men u
Ma in men u
original behavioural seq uence
Us er
ke y p re ss
Au toma ti c
tra ns i ti on
simulated behavioural sequence
6
Validation of the functional equivalence,
computed by the similarity ratio (SR)
© M. Rauterberg, 2004
F_ 3
18/35
Two different mental models
Model-1
Model-4
Ref: Rauterberg, M. (1995). From novice to expert decision behaviour: a qualitative modelling approach with Petri nets. In: Y. Anzai, K. Ogawa & H.
Mori (eds.), Symbiosis of Human and Artifact: Human and Social Aspects of Human-Computer Interaction--HCI'95 (Vol. 20B, pp. 449-454). Elsevier.
© M. Rauterberg, 2004
19/35
The Similarity Ratio: SR
  N
SR  1   R
N org
sim
t 1
org ,t
 R sim, t 
 max  R
N sim  1




org

N * 100%

2
org
Legend: R is the absolute rank position in the original or simulated process
Ref: Rauterberg, M. (1995). From novice to expert decision behaviour: a qualitative modelling approach with Petri nets. In: Y. Anzai, K.
Ogawa & H. Mori (eds.), Symbiosis of Human and Artifact: Human and Social Aspects of Human-Computer Interaction--HCI'95
(Advances in Human Factors/Ergonomics, Vol. 20B, pp. 449-454). Amsterdam: Elsevier.
© M. Rauterberg, 2004
20/35
Simulation Results: Model-1
Simulated logfiles with Model-1
or iginal
SR
43%
40%
77%
76%
10%
10%
79%
67%
10%
10%
10%
83%
d
d
d
d
d
d
d
d
d
d
d
d
a
a
a
a
h
h
a
a
h
h
h
a
F3
F3
F3
F3
F3
F3
F3
space
TAB
TAB
space
TAB
space
F9
space
F9
space
CR
CR
space
a
TAB
h
F2
space
CR
CR
F3
F2
F2
F9
TAB
space
TAB
TAB
CR
a
space
F9
CR
CR
F9
F3
F2
h
space
space
h
TAB
F9
space
F9
F9
h
F9
h
a
...
© M. Rauterberg, 2004
h
21/35
Simulation Results: Model-4
Simulated logfiles with Model-4
or iginal
SR
95%
94%
94%
d
d
d
a
a
F3
96%
94%
96%
94%
94%
94%
94%
94%
96%
d
d
d
d
d
d
d
d
d
d
a
a
a
a
a
a
a
a
a
a
a
F3
F3
F3
F3
F3
F3
F3
F3
F3
F3
F3
F3
space
space
space
space
space
space
space
space
space
space
space
space
space
space
space
TAB
space
space
TAB
space
TAB
space
TAB
TAB
TAB
space
TAB
space
space
TAB
TAB
space
TAB
space
space
space
space
space
TAB
F2
TAB
TAB
TAB
space
TAB
TAB
TAB
TAB
space
space
TAB
space
TAB
TAB
space
space
TAB
space
space
space
TAB
TAB
TAB
space
TAB
CR
F2
F2
F2
F2
F2
F2
F2
F2
F2
F2
F2
F2
space
CR
CR
CR
CR
CR
CR
CR
CR
CR
CR
CR
CR
F9
F9
F9
F9
F9
F9
F9
F9
F9
F9
F9
F9
F9
h
h
h
h
h
h
h
h
h
h
h
h
h
© M. Rauterberg, 2004
96%
22/35
Approach-2: event-driven goal setting
Model-2 (first part): Event-driven goal setting strategy
Cognitive level
level
F_10
Start menu
h
F_3
Main menu
1
i
h
Main menu
F-file
Ref: Rauterberg, M., Fjeld, M. & Schluep S. (1997). Parallel or event-driven goal setting mechanism in Petri net based models of expert
decision behaviour. In: S. Bagnara, E. Hollnagel, M. Mariani & L. Norros (eds.), Time and Space in Process Control--CSAPC'97 (Sixth
European Conference on Cognitive Science Approaches to Process Control, pp. 98-102). Roma: CNR.
© M. Rauterberg, 2004
Action level
Ms DOS
G_2 M_3
Goal
System levelinstanciation
[Remark:
Approach-1 is called ‘pure’ action driven model
without the ‘goal selection’ level and only feedback
between ‘action’ and ‘goal instanciation’]
23/35
Approach-3: parallel goal setting without feedback
Model-3 (first part): Parallel goal setting strategy
C ognitive level
System level
Goal
instanciation
level
F_10
© M. Rauterberg, 2004
Start menu
h
F_3
Main menu
1
i
Main menu
F- file
h
Action level
Ms D OS
G_2 M_3
24/35
Approach-4: parallel goal setting with feedback
Model-4 (first part): Parallel goal setting with feedback
C ognitive level
System level
Goal
instanciation
level
Feedback
level
Ms D OS
© M. Rauterberg, 2004
G_2 M_3
Start menu
h
F_3
Main menu
1
i
Main menu
F- file
h
Action level
F_10
25/35
Results for the 4 different modeling approaches
The model complexity (Ccycle) and similarity ratio (SR) of the
approaches-1, -2, -3 and -4 [std:=standard deviation].
approach
no. 1
Ccycle:
(mean ± std):
Ccycle:
(min…max.):
SR
(mean % ± std):
SR
(min…max. %):
# simulated
sequences
© M. Rauterberg, 2004
approach
no. 2
approach
no. 3
approach
no. 4
13 ± 5
43 ± 17
57 ± 25
101 ± 43
6…18
22…68
30…97
55…170
41 ± 28
66 ± 21
88 ± 11
100 ± 0
3…79
36…98
67…100
100…100
5*6=30
5*6=30
5*6=30
5*6=30
26/35
Overview over additional measures
Measuring ‘Complexity’
(C)
Ccycle = #T – #S+P
with #S =< #T and P=1
C'cycle = #F–(#T + #S)+P
with #S > #T and P=1
Measuring 'Routine‘ (R)
R = #UT /
#DT
Measuring 'Personality Style‘
(PS)
PS = #TST /
#DS
© M. Rauterberg, 2004
T = number of transitions
S = number of states
F = number of connectors
TST = task solving time
UT = all used transitions
DT = all different transitions
DS = all different states
27/35
The common assumption of the AI community
observable
behaviour
learning
mental
model
Ref: Rauterberg, M. (1996). About faults, errors, and other dangerous things. In: C. Ntuen & E. Park (eds.), Human Interaction with
Complex Systems: Conceptual Principles and Design Practice (pp. 291-305). Norwell: Kluwer.
© M. Rauterberg, 2004
28/35
The reality: look, what we found!
exp ert
advanced
beginner
complex
s ys tem
Experiment:
6 novices and 6 real experts
4 tasks with a database
Metric BC=Ccycle
SC
interaction
BC
operator
learning
time
CC
We found a negative correlation between
Behavior-Complexity BC and [assumed] Cognitive-Complexity CC
© M. Rauterberg, 2004
29/35
How to interpret this negative correlation?
observable
behaviour
learning
mental
model
© M. Rauterberg, 2004
30/35
Mental knowledge structures: a metaphor
"dale“: knowledge about successful behavior
s0
“hill“: knowledge about unsuccessful behavior
h
d
s1
CR
b
F9
F2
s2
F3
3D picture
s3
TAB
_
© M. Rauterberg, 2004
This new view would have major
impact e.g. on training procedures
of operators of complex systems!
31/35
Learning: the traditional understanding
before learning phase
after learning phase
Learning is seen as ‘digging dales’!
© M. Rauterberg, 2004
32/35
Decision and action: a new view
Mental decision making for concrete actions is like rolling a ball between hills,
consisting of two kinds of knowledge: successful and unsuccessful!
© M. Rauterberg, 2004
33/35
Learning and experience
task-1
task-1'
task-1''
task complexity
time-1
time-2
time-3
time
Ref: Rauterberg, M. & Aeppli, R. (1995). Learning in man-machine systems: the measurement of behavioural and cognitive complexity.
In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics--SMC'95 (Vol. 5, IEEE Catalog Number
95CH3576-7, pp. 4685-4690). Piscataway: Institute of Electrical and Electronics Engineers.
© M. Rauterberg, 2004
34/35
The learning experiment
N=6 beginners (all male, average age of 25 ± 3 years)
Task solving time
Behavioral complexity
Time structure and knowledge structure are different!
© M. Rauterberg, 2004
35/35
Conclusions
• A valid metric for task complexity based on
task structure allows an objective comparison
• Automatic analysis for unconstrained task
solving behavior allows analysis with applied
statistics
• This new analysis and modeling approach
with AMME leads to new insights…
© M. Rauterberg, 2004
36/35
Thank you for your attention.
© M. Rauterberg, 2004