Transcript Document 7432524

JEM: A Motivational Model of
Evaluation for Information
Environments
Jeff Stanton, Ping Zhang, Gisela von Dran
Syracuse University
Information Environments (IE)
 An Information Environment (IE) is an artifact that encompasses
information content and organization or structure in support of
access to information content (Davern, Te’eni, Moon 02)
 Examples are
o Websites, Web pages, databases, palm PDAs (info & access), smartphones (address book, access to other sources), kiosks (interface),
handheld GPS mapping systems (map, info, control), Information Systems
 Excludes
o The technical infrastructure (e.g. hardware, supporting software, network,
connection, etc.) of an artifact
o Other artifacts that may have an user interface but not for information
seeking purposes (dishwasher, a car’s dashboard)
 Supports goal-driven (information seeking) activities
12/14/2002
HCI/MIS'02 Workshop
2
Problems w/ Existing Studies
 Many studies lack theoretical foundations for user
evaluation of IEs
 For those that are theoretically based (80+ empirical
studies in IS literature),
o Most are generally about beliefs and attitude changes,and
their impact on subsequent intention and use
o Few on the affect changes of user experience & evaluation
12/14/2002
HCI/MIS'02 Workshop
3
Individual Level IS Evaluation:
Dominating Theories, Constructs, & Emphases
Antecedents
Internal
Antecedents
External Antecedents
System
factors
Task
factors
Individual Evaluative Process
User
Trait
Social
& Org.
Factors
User
Tech
factors
Norms
System Beliefs
Outcomes
Attitude
Subj.
norms
Behavior
Beh.
Intent
Attitude
PU
Beh.
Intent
Satisfa
Use
PEOU
Demo
P. Fit
Perform
P.
Hedonic
Values
DOI, 1962
TRA/TPB,1975
SCT, 1977
TAM, 1989
ETAM
IS Succ, 1992
TTF, 1995
# of empirical
studies using
the specified
variables
12/14/2002
*****
*****
*****
*****
*****
*****
****
***
*****
*****
*****
*****
***
*****
*****
*****
*****
*****
*
*****
*****
*****
***
*****
*****
*****
*****
*****
**
*****
***
*****
*****
*****
*****
*****
*****
*****
*****
****
HCI/MIS'02 Workshop
*****
*****
*****
*****
*****
*****
*****
*****
*****
*****
***
*****
***
*****
*****
*****
*****
*****
*****
*****
**
*****
*****
*****
*****
**
*****
*****
*****
*****
*****
*****
*****
*****
*****
*****
*
4
Objectives of the Study
 To understand the affective experience a user has when using
and evaluating an IE.
 To synthesize current theories from behavioral science literature
into a workable process-oriented framework of user evaluation
 To use the framework to gain a better understanding of processes
underlying user evaluation of an artifact as well as the likely future
behavior toward that artifact
 After testing and pruning, to use the framework as a guide to
evaluation and assessment for IEs
 To use good, theory-driven evaluation tools to provide guidance to
IE designers about how to improve the experience of users
12/14/2002
HCI/MIS'02 Workshop
5
Basic Working Assumptions
 A given user’s evaluation of an IE is a function of both:
o The artifact itself, and
o The person who is using it.
 A user’s ultimate judgments about an IE accrue neither
solely from “person” characteristics nor solely from the
characteristics of the IE but from the intersection of the
two that occurs as the user uses the artifact to conduct
and possibly complete some goal oriented tasks.
12/14/2002
HCI/MIS'02 Workshop
6
JEM = Joint Evaluation Model
 Jointly scaling artifact and person
 Artifact brings observable features to the interaction
o Static: visible without interaction (color, layout of a web page)
o Dynamic: aspects of performance (loading time, search
results, operation of controls)
 Person brings both stable and transient characteristics to
the interaction
o Stable: general +/- outlook (affectivity), prior experience with
and knowledge of similar artifacts
o Transient: specific goals for using the artifact, mood (state
affect)
12/14/2002
HCI/MIS'02 Workshop
7
Our Strategy for Framework Synthesis
 Try to shift the focus from attitudes (cold) to emotions
(warm) by integrating Regulatory Focus Theory
 Include the notion of discrepancy or matching or
confirmation of expectations and reality by incorporating
elements of Expectation-Confirmation Theory
 Describe a process model that provides details about
what happens during the actual process of evaluation
(rather than simply post-evaluation)
12/14/2002
HCI/MIS'02 Workshop
8
Existing Theories Part I:
Regulatory Focus Theory (RFT)
 Higgins 1997, 1998; Brockner & Higgins 2001 developed an
emotion-centered motivation theory that overarches several
popular precursor theories:
o Goal setting theory, expectancy-valence theory, behavioral decision theory
 RFT posits two “self-regulatory” (internal) systems through which
individuals exert control over their emotional state via behavior
o Promotion focused: Exert effort towards achieving an ideal goal or standard
• Cheerful when goal achieved, dejected when goal not achieved
o Prevention focused: Exert effort towards vigilance and loss avoidance
• Quiescent when goal achieved, agitated when goal not achieved
 Individual difference + situational component
o Stable trait: Predisposition toward promotion or prevention focus
o Situational factors: Shift regulatory focus depending upon whether one
perceives situationally specific outcomes as potential gains or losses
 Extensive psychological literature supporting theory predictions
12/14/2002
HCI/MIS'02 Workshop
9
Existing Theories Part II:
Expectation-Confirmation Theory
 Expectation confirmation/disconfirmation from the
customer satisfaction literature on consumer satisfaction &
post-purchase behavior
o E.g., Cardozo (1965), Anderson (1973), Oliver (1980, 93),
Dabholkar (2000)
 Bhattacherjee 01: IS Continuance
 Key constructs: initial expectation, initial consumption/use
and perceptions about performance, confirmation,
satisfaction, continuing intention.
 Consistent with Fishbein/Ajzen Theory of Reasoned Action
with the addition of an “attitude discrepancy” construct
12/14/2002
HCI/MIS'02 Workshop
10
Pre-Evaluation Phase
Similar Artifacts
Artifact
expectation
Evaluation Phase
This Artifact
Circumplex
Goal
Cognition
Quiescence
Goal
Progress
12/14/2002
Approach
Intentions
Artifact
Confirmation
Regulatory
Focus
State & Trait
Affect
Post-Evaluation Phase
Elation
Evaluation
Affect
Change
HCI/MIS'02 Workshop
Avoidance
Intentions
11
Pre-Evaluation Phase
 Conditions that exist prior to the user’s experience with
the artifact:
o Prior experience with similar artifacts
o Regulatory focus: Whether one’s purpose in use of the artifact
is pursuit of a gain or avoidance of a loss.
o Goal cognition: Specifically what the user hopes to achieve in
using the artifact and something about his/her strategy or plan
for achieving it.
o Affect (Emotion): The user’s emotional and mood status just
prior to the use of the artifact (important to establish a baseline
for assessing subsequent change)
12/14/2002
HCI/MIS'02 Workshop
12
Evaluation Phase
 What occurs during the time the user uses the artifact
 Three processes (with iterations):
o The user, guided by previous experience with similar artifacts, makes
sense of the artifact by understanding its features, then compare the
perceived performance with expectation
o Meanwhile, the user attempts to pursue a particular goal, thus cognitively
evaluating whether s/he is achieving that goal
o The user’s hedonic (emotional or affective) state changes as a result of
making progress toward that goal and also a result of making sense of the
artifact
 We suggest that reaching a goal or making progress toward a
goal has a positive influence on hedonic state as does
recognizing the presence of a desired feature that would facilitate
goal progress
12/14/2002
HCI/MIS'02 Workshop
13
Circumplex Affect
 Following recent psychological research on the structure
of affect (e.g., Watson and Tellegen)
o Also consistent with RFT
 Affect/Emotion has two distinct and orthogonal
dimensions to it:
o Elation: The positive/negative dimension, where one falls on
the cheerfulness-dejectedness dimension
o Quiescence: The activated/inactivated dimension, where one
falls on the quiescence-agitation dimension
12/14/2002
HCI/MIS'02 Workshop
14
Post-Evaluation Phase
 The user reaches a final hedonic state as a result of his
or her experience with the artifact
 A memory of this hedonic state will guide his/her future
behavioral intentions and thereby influence subsequent
behavior
 Because RFT and Circumplex describe two distinct
emotive conditions, the behavioral intentions may be
separable into:
o Approach intentions
o Avoidance intentions
12/14/2002
HCI/MIS'02 Workshop
15
Examples Using Websites
Person A
Person B
Artifact expectation
Normal search engines
Normal info intensive sites
Regulatory focus
Pursue a gain
Avoid a loss
Goal
Find best airfare for vacation
Find info on breast cancer
Affect: trait
Positive
Positive
Affect: state
Happy, calm
Sad, agitated
Artifact confirmation
Below expectation
Equal to or higher than expectation
Goal progress
Not be able to find a decent fare
during a reasonable time
A comprehensive coverage
Affect change
Yes
Yes
Elation affect
Dejected
Sad
Quiescence affect
Agitated
Calm
Intention
Weak approach & strong
avoidance: user will actively seek
for alternatives and criticize it
when prompted HCI/MIS'02 Workshop
Strong approach and weak
avoidance: user will actively and
frequently reuse or recommend it if
opportunities arise
12/14/2002
16
Examples (cont’d): Affect Changes
Person A
Bad Travel Site
Person B
Good Breast Cancer Site
Happy
Happy
Elation
Elation
Start
End
End
Start
Sad
Sad
Agitated
12/14/2002
Quiescence
Calm
Agitated
HCI/MIS'02 Workshop
Quiescence
Calm
17
Preliminary Propositions (1/2)
 A user, dependent upon the goal (or mood at the time of evaluation)
may evaluate the same IE differently at different times.
 The user’s hedonic state can be improved during the evaluation
process by facilitating goal achievement (or matching the artifact
toward the expectation).
 Positive changes in Elation component of the hedonic state will
increase the likelihood of subsequence approach behavior; may not
strongly affect subsequent avoidance behavior.
 Changes in Quiescence (Activation) component of the hedonic
state toward the agitated end will increase the likelihood of
subsequent avoidance behavior, but will not strongly affect
subsequent approach behavior.
12/14/2002
HCI/MIS'02 Workshop
18
Preliminary Propositions (2/2)
Positive Elation
Strong approach and
strong avoidance: user will
High
Activation actively and frequently
reuse or recommend IE in
an effort to avoid lost
opportunities.
Strong approach and
weak avoidance
Low
Activation intentions: user may use,
revisit, or recommend IE if
opportunity arises.
12/14/2002
HCI/MIS'02 Workshop
Negative Elation
Weak approach and strong
avoidance: user will actively
seek alternatives and criticize
IE in order to help prevent
own or others’ losses due to
wasted time or frustration.
Weak approach and weak
avoidance: user will forget
about IE or criticize it if
prompted.
19
Pilot Study
 Designed to explore the salience of various IE features:
o E.g., how important were the navigational features versus the
credibility of the information versus the layout, and so forth
 ~150 working people in Spring 2002; a pre-measure and
a post-measure of mood using Watson and Tellegen’s
Positive Affect, Negative Affect Survey (PANAS)
 We found that a brief use of a web-based encyclopedia
showed changes in both positive and negative affect
 Additionally we found that both “person variance” and
“artifact variance” influenced affect
12/14/2002
HCI/MIS'02 Workshop
20
Potential Contributions
 With more testing, we may have a process model that
elucidates the processes underlying user evaluation of
an artifact as well as the likely future behavior toward
that artifact
 The model focuses on the hedonic experience a user
has during goal driven use of an IE.
 The model tries to clarify constructs and their
relationships, as well as provide a model-driven basis for
measurement
12/14/2002
HCI/MIS'02 Workshop
21
Limitations
 A preliminary conceptual model: needs empirical support
 Quite complex: needs trimming in order to include just
the most important mechanisms and constructs
 Did not specifically consider repeated use of the same IE
and the evaluation of it
 Does not reflect the social context of artifact evaluation
 May have no applicability to evaluation of other kinds of
products or user interfaces
12/14/2002
HCI/MIS'02 Workshop
23
Future Research







Operationalize the JEM model
Develop research propositions
Develop specific hypotheses for empirical testing
Complete a preliminary verification
Revise the model as needed
Confirm the revised model
Publish validated assessment tools
12/14/2002
HCI/MIS'02 Workshop
24