Transcript Studie Materiaal Oefeningen

Measuring User Satisfaction
through Experiments
B. de Vries
Technische Universiteit
Eindhoven
Domotica
Wildgroei
Toekomst
droom
Losse
deelmarkt
en
Computable: 5 November 2004
Technische Universiteit
Eindhoven
Innovatie
•
•
•
•
•
•
Juiste product ?
Juiste doelgroep ?
Juiste distributie ?
Juiste tijd ?
Juiste marketing ?
…
Technische Universiteit
Eindhoven
Experiments
• Observational
• Experimental
Technische Universiteit
Eindhoven
Characteristics
Empirical: Gather evidence through
observation and measurement
that can be replicated by others
• Measurement
• Replicability
• Objectivity
Technische Universiteit
Eindhoven
Variables
• Independent: Cause
• Dependent: Effect
Technische Universiteit
Eindhoven
Scientific research
• Validity: Are you measuring what you
claim to measure
( measuring the right thing)
• Reliability: The ability to produce the
same results under the same condition
(Measuring thing right)
• Error: The difference between our
measurements and the value of the
construct we are measuring
Technische Universiteit
Eindhoven
Validity
Internal validity problems
• Group threats, regression to the mean,
time threats, history, maturation,
instrumental change, differential
mortality, reactive and experimenter
effects
External validity problem
• Over-use pf special participants group,
restricted number of participants
Technische Universiteit
Eindhoven
Between groups
Random
allocation
Treatment
(experimental gp.)
Measurement
No Treatment
(control gp.)
Measurement
Technische Universiteit
Eindhoven
Measuring User satisfaction
• Virtual Reality
• Bayesian Belief Networks
Technische Universiteit
Eindhoven
Desk-Cave
Desk-Cave
Set-UP
• 2 synchronized
PC’s with dual
monitor output
• 4 LCD Projectors
Technische Universiteit
Eindhoven
Features
• 1 : 1 Scale
• 3DS import
• Immersion
Technische Universiteit
Eindhoven
Bayes Theorem
P( B | A) P( A)
P( A | B) 
P( B)
From: Evaluation and Decision (7M834)
Technische Universiteit
Eindhoven
Bayesian Belief Network
Train
Strike
Martin
Late
Norma
n
Late
Technische Universiteit
Eindhoven
Node Probability Table
Train
Strike
Norman
late
True
False
True
0.8
0.1
False
0.2
0.9
Train
Strike
Norma
n
Late
Martin
Late
Technische Universiteit
Eindhoven
NPT’s
Train Strike
Train strike
Martin late
True
False
True
0.1
True
0.6
0.5
False
0.9
False
0.4
0.5
Technische Universiteit
Eindhoven
Analyzing a BBN
Marginal probability
p(Norman late) = p(Norman late | train strike) * p(train strike) +
p(Norman late | no train strike) * p(no train strike)
= (0.8 * 0.1) + (0.1 * 0.9) = 0.17
Conditional probability
p(Train|Norman late) = (p(Norman late|train strike) * p(train strike))/
P(Norman late)
= (0.8 * 0.1) / 0.17 = 0.47
Technische Universiteit
Eindhoven
Measuring User Satisfaction Using Virtual Reality
and Bayesian Belief Networks.
Maciej A. Orzechowski
01.11.2004
Technische Universiteit
Eindhoven
Motivations, aims
Current techniques for measuring user preferences (CA, MM,
interview) are artificial, lengthy or expensive.
For good results we need to get the respondents more
involved in the measurement.
Can Virtual Reality (VR) improve the quality of measuring
preferences: more involved and higher reliability?
The aim of this project was to develop and test an interactive
VR tool for measuring housing preferences.
Technische Universiteit
Eindhoven
Solution strategy
Interactive Virtual Environment (iVE).
Modification of a design.
Translation of applied modifications into choices.
Entering this information into a Bayesian Belief Network.
Checking the consistency (if necessary prompting for
verification).
Learning (updating) the preference network.
Technische Universiteit
Eindhoven
VR System
MuseV3 – a Virtual Reality application with
functionality of a simple CAD system.
Two categories of modifications:
Structural modifications (change layout).
Textural modifications (change visual
impression).
Technische Universiteit
Eindhoven
Structural Modifications
Change of internal and external dwelling’s layout.
The most important for estimating user preferences.
Include following commands: create/resize space; insert
openings.
Direct impact on overall costs of the dwelling.
Technische Universiteit
Eindhoven
MuseV3 in Desktop CAVE
Technische Universiteit
Eindhoven
Bayesian Belief Network
Non-obtrusive interactive method to collect housing preferences.
Potential advantages
Interaction with the model during the time of preferences
estimation.
Incremental learning.
Possibility to assess:
 where the knowledge about preferences is most uncertain.
 consistency of measurements.
Technische Universiteit
Eindhoven
Bayesian Belief Network cont.
A Bayesian Belief Network (BBN) captures believed relations
(which may be uncertain, stochastic, or imprecise) between
variables, which are relevant to some problem.
Family Situation
Lounge Ext
(β1)
Garage Ext
(β2)
Extra
Kitchen
(β3)
Age
2 Bedrooms
(β4)
First Floor
Ext (β5)
Dormer
Window (β6)
Price (γ)
Choice of
Lounge Ext
Choice of
Garage Ext
Choice of
Extra
Kitchen
Choice of
2 Bedrooms
Choice of
Choice of
First Floor
Dormer
Technische Universiteit
Ext
Window
Eindhoven
CPT calculation
set B, G CPT to uniform
probability
calculate utility for each
Choice state, according to
each combination of states of
nodes B, G (eq. 2)
G-CPT
State 1 PG1
State 2 PG2
State 3 PG3
State 4 PG4
State 5 PG5
B-CPT
State 1 PB1
State 2 PB2
State 3 PB3
B
based on utilities, calculate
probability for each Choice
state, for each combination of
nodes B, G (eq. 3)
For each Choice node
G
Choice
Choice-CPT
B-State 1 G-State 1
B-State 2 G-State 1
B-State 3 G-State 1
B-State 1 G-State 2
…
…
Choice-State 1
P 11
P 21
P 31
P 41
…
Technische Universiteit
Choice-State 2
P 12
P 22
P 32
P 42
…
Eindhoven
Learning process
new
respondent
make
design
choices
ultimate
design
solution
N
Y
set choice
to selected
state
update CPT's of nodes B, G
Technische Universiteit
Eindhoven
Convergence
Technische Universiteit
Eindhoven
Utility Convergence
Technische Universiteit
Eindhoven
Experiment
1600 letters -> 100 answers -> 64 respondents.
Respondents were people searching for a house or who just bought one.
4 kinds of 2 types of tasks (2 traditional, 2 based on MuseV3):
 CA: Verbal Description Only (VDO) Multimedia Presentation (MM).
 BBN: Preset Options (PO) Free Modification (FM).
Each respondent completed both types of tasks.
Technische Universiteit
Eindhoven
Experiment Types – cont.
VR Experiment
CA Experiment
Type
Free Modification
Preset Options
Multimedia
Presentation
Verbal
Description
Software
(Mean of
presentation)
MuseV3 FM
MuseV3 OE
MuseV3 SC
Web Pages
Collection
Method
Interaction with 3D
environment
Interaction with 3D
environment
Questionnaire
Questionnaire
Task
Modification of
architectural design
Respond to pre
designed options
Choice from
between three
design
alternatives
Choice from
between three
design
alternatives
Interactivity
with 3D model
Restrained to
design constrains
Finishes and
furniture
Walk Through
N/a
Feedback from
the system
yes
yes
none
none
Estimation
method
Belief Network
Belief Network
MNL Model
MNL Model
Technische Universiteit
Eindhoven
Analysis
Estimation of separate models for each task.
Test for order effect.
Comparison of CA and BBN models in terms of:
 Internal validity.
 Predictive validity.
Questionnaire.
Technische Universiteit
Eindhoven
Internal Validity CA vs. BBN
Roughly similar between CA and BBN.
Estimated utilities are not identical but strongly correlated.
The difference in scale suggests that the BBN has a lower
error variance.
The task order effect suggests VR pre-learning improves
the validity.
Technische Universiteit
Eindhoven
External Validity CA vs. BBN
Models based on BBN on average predicted correctly
69% of the choices.
Models based on CA on average predicted correctly 56%
of the choices.
High increase in CA model performance when task is
preceded by VR task:
 for VDO from 56% to 62%.
 for MM from 32% to 73%.
Technische Universiteit
Eindhoven
Observed-Predicted
Technische Universiteit
Eindhoven
Conclusions
The results support the potential of the suggested approach.
The results suggests higher involvement of respondents.
This approach is non-obtrusive compared to different
preference measurement techniques.
The system (tool) can be used to:
 To assist individual users in creating their own design.
 To derive market potential of housing designs at
aggregate level.
Technische Universiteit
Eindhoven
Domotica applications
•
•
•
•
Alarmering: inbraak, zorg, brand
Autom. Verlichting
Autom. Zonwering
…
Technische Universiteit
Eindhoven