Transcript End-User Perceptions of Formal and Informal Representations of Web Sites Jason Hong

End-User Perceptions of Formal
and Informal Representations of
Web Sites
Jason Hong
Francis Li
James Lin
SIMS 271
December 15, 2000
Overview





Motivation
Hypotheses
Experiment Design
Quantitative and Qualitative Results
Conclusion
Design Representations

Differing levels of detail
Informal (low detail)
Formal (high detail)
What Designers Report

Formal representations encourage looking at
details inappropriate at early stages
–
–
–
–

Fonts
Colors
Graphics
Pixel-level layout
People looking at formal
representations often
think the visual design is done
What Designers Report (cont’d)

Informal representations encourage looking at
issues appropriate for early-stage design
–
–
–
Information architecture
Navigation flow
Overall page layout
Hypotheses

Between informal and formal prototypes
–
People will judge the following attributes differently:




professional
finished
functional
likely to change
Hypotheses (cont’d)

Between informal and formal prototypes
–
People will rank the importance of visual vs. nonvisual design issues differently


visual issues: font, color, graphics
non-visual issues: navigation bars, overall page layout,
organization of information
Experiment Design

2×2 within-subjects
Betweensubjects
Within-subjects

Formal
Informal
Total
Site A
4
4
8
Site B
4
4
8
Total
8
8
16
Each participant evaluated formal for one site,
informal for other site
Experiment Design

For formal representations
–
–
–
–
Downloaded two actual web sites
Pared down product list
Greeked some text out
Hardwired shopping cart data
Experiment Design

For informal representations
–
–
–
–
Made electronic representations in DENIM instead
of using paper
Exported DENIM pages to GIF + image maps
Touched up sketches to improve handwriting
Added HTML text fields by hand
Experiment Design

Participants asked to do 5 tasks per web site
–
–
–
–
–
Find a specific piece of information
Find another specific piece of information
Find information deeper in site
Add two items to shopping cart
Purchase items in shopping cart
Experiment Design

After each web site
–
Participants were asked overall impressions of site
(scale: 0–10)

–
professional, finished, functional, likely to change
Participants ranked ten possible ways of improving
web site


6 were non-visual improvements
4 were visual improvements
Are the two web sites the same?

Significant differences found
(p < .05, 2 tailed independent t-test)
–
between formal sites




attractiveness
quality
security
likelihood of changing
–
between informal sites





ease of use
efficiency
complexity
attractiveness
usefulness
Analysis

Merged site data
–
–

Assume sites are similar
Within-subjects analysis
Evaluated each site separately (peggyli, casa)
–
Between-subjects analysis
Quantitative Results
Formal
Informal
p
(2-tailed
paired ttest)
Professional
8.4
3.6
< 0.01
Finished
Functional
Likely to change
8.0
8.1
4.4
2.4
6.5
8.7
< 0.01
< 0.05
< 0.01

Within-subjects, site data merged
–

Similar results for separate, between subjects analysis
This supports our hypothesis
Quantitative Results

To analyze ranking data:
–
–
–
1-10 rank was divided into 3 bins: top 3, middle 4,
bottom 3
each rank was recoded into midpoint of each bin
(2, 5.5, 8)
average rank for visual issues, non-visual issues
were calculated
Non-Visual
Visual
More descriptive text in links
Improved navigation bars
Clearer indicators of what can be
clicked on
Clearer scheme for organizing
pages on the web site
6
4
9
Better page layout
More streamlined shopping cart
and checkout
More emphasis on web site brand
5
10
Better use of fonts
Better use of colors
Better use of graphics
8
7
2
1
3
Non-Visual
Visual
More descriptive text in links
Improved navigation bars
Clearer indicators of what can be
clicked on
Clearer scheme for organizing
pages on the web site
6 5.5
4 5.5
9 9
Better page layout
More streamlined shopping cart
and checkout
More emphasis on web site brand
5 5.5
10 9
Better use of fonts
Better use of colors
Better use of graphics
8 9
7
2
1
3
5.5
2
2
2
Non-Visual
Visual
More descriptive text in links
Improved navigation bars
Clearer indicators of what can be
clicked on
Clearer scheme for organizing
pages on the web site
6 5.5
4 2
9 9
Better page layout
More streamlined shopping cart
and checkout
More emphasis on web site brand
5 5.5
10 9
Better use of fonts
Better use of colors
Better use of graphics
8 9
7
2
1
3
5.5
2
2
2
7.25
2.875
Participant Informal
Formal
Non-Visual Visual
1
2
3
4
5
Mean
7.25
2.875
Non-Visual
Visual
Participant Informal
1
2
3
4
5
Mean
Formal
Non-Visual Visual
Non-Visual
Visual
7.25
6.667
3.75
2.875
Participant Informal
Formal
Non-Visual Visual
Non-Visual
Visual
1
2
3
7.25
6.083
7.25
2.875
4.625
2.875
6.667
6.083
6.083
3.75
4.625
4.625
4
5
Mean
5.5
3.75
5.5
8.125
3.75
4.917
8.125
6.375
Participant Informal
Formal
Non-Visual Visual
Non-Visual
Visual
1
2
3
7.25
6.083
7.25
2.875
4.625
2.875
6.667
6.083
6.083
3.75
4.625
4.625
4
5
Mean
5.5
3.75
5.967
5.5
8.125
4.8
3.75
4.917
5.5
8.125
6.375
5.5
Quantitative Results

Assume web sites are same  within-subjects
Formal
Informal
p
(2-tailed
t-test)
Visual
5.9
4.8
< 0.005
Non-visual
5.2
5.9
< 0.005
Quantitative Results



Assume web sites are different 
between-subjects
All rankings averaged around 5
Rankings of visual vs. non-visual issues were
not significantly different for either web site
Quantitative Results

Informal representations
–

Formal representations
–

visual issues ranked higher
non-visual issues ranked higher
This does not support our hypothesis
–
–
electronic medium may raise expectations
quality of sketches may affect perception
Qualitative Results

Many of the same usability problems were
found in both formal and informal
–

Ex. Add to Shopping Cart / Registration problem
Informal prototypes are sufficient for taskbased usability evaluation while also faster and
easier to create
Future Work



Study earlier stages of design cycle
Task-based vs non-task-based evaluation
Investigate effect of medium on perception
–
–
–
–
paper sketches
scanned paper sketches
electronic sketches
printed electronic sketches
Conclusions

Informal representations look:
–
–
–
–


less professional
less finished
less functional
more likely to change
Visual improvements are more likely to be
ranked important for informal representations
Needs more work!
Extra Slides
Web Site Design Process
Experiment Design

After each task, participants were asked how
well site supported task (scale: 0–10)
–
–
–
–
easy
efficient
clear
helpful