Comparative Usability Study Reporting of Results

Robert Murphy

Study Overview

• Comparative usability study between SHOP.COM and Amazon / Shopping.com with a concentration on search UI elements and use of search functionality

Participants

• 18 Shoppers – 16 women – 2 men – 2 – age 20-29 – 4 – age 30-39 – 7 – age 40-49 – 5 – age 50-59 – 6 w/SHOP.COM experienced – 12 w/out SHOP.COM experience – 13 shoppers mentioned “Amazon” as a commonly used ecommerce site

Study Overview

• • • • •

Methodology

Each shopper performed 2 - 4 unaided searches on each website beginning from the home page Total of 90 searches performed (search and navigation) – 25 on SHOP.COM and Shopping.com

– 20 on SHOP.COM and Amazon – SHOP.COM was always presented first Categories of searches included: – jewelry, shoes, sporting goods, clothing, electronics, auto supplies, books & video Questions rating their experience and preference between sites were asked after performing searches and after using both sites Various additional pages were visited, involving tasks in both navigation and general site usage

2 1.5

1 0.5

0 5 4.5

4 3.5

3 2.5

Study Overview Comparison to previous study

Overall Site Satisfaction Rating

2.8

4.3

• Generally, improved response over similar study conducted April, 2006 – 87% success rate over 60% – Overall satisfaction rating 4.3 over 2.8

April, 2006

Date of Study

Dec, 2006 • Several favorable responses from shoppers: – – –

I had a good experience You guys have a great product I like the 1-cart concept

Overall Findings SHOP.COM

Overall Ratings for SHOP.COM

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 87% 70% Successfully Found Product 60% Searches Returning Relevant Results Searches with Acceptable Number of Results • • • Shoppers successfully found the product they were looking for 87% of the time Shoppers felt the results were relevant in 70% of the searches Shoppers felt an acceptable number of results were returned in 60% of the searches

Use of Navigational Elements SHOP.COM

Percentage Usage of Navigational Elements

35% 30% 25% 20% 15% 10% 5% 0% Search Again 9 Shoppers Left Nav Links 4 Shoppers Search Within 7 Shoppers Additional (Next) Pages 6 Shoppers

Navigational Elements

Sort Results 9 Shoppers • Breakdown of total usage of navigational elements: – Performing a search again – Use of Left Navigation links – Using Search Within function – Clicking to view the “next” page of results – Sorting of results

Use of Navigational Elements SHOP.COM

Percentage of Use of Navigational Elements by SHOP.COM Experience

50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% Search Again Left Nav Links Search Within Additional (Next) Pages Sort Results No SHOP.COM Experience SHOP.COM

Experience • Breakdown of total usage of navigation functions by experience on SHOP.COM

– Inexperienced shoppers relied primarily on searching again to refine their results – Experienced shoppers utilized searching again, Search Within and (primarily) Left Nav links – Experienced shoppers did not need to go from page to page of results

Comparative Analysis Successful Searches

Percentage of Successful Searches

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% SHOP.COM

Shopping.com

SHOP.COM

Amazon.com

• Q:

“Do you feel you successfully found the product you were looking for?”

• Percentage of searches where shopper successfully found the product they were looking for

Comparative Analysis Relevancy of Results

100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%

Percentage of Relevant Results

SHOP.COM

Shopping.com

SHOP.COM

Amazon.com

• Q:

“When you were searching or navigating through the results, was the relevancy of the products displayed what you expected?”

• • • Percentage of searches in which shopper felt their search returned relevant results Areas identified for improvement on SHOP.COM: – shoes, books, sporting equip, clothing Underperforming category on Amazon: – household

Comparative Analysis Quantity of Results

Quantity of Results

80% 70% 60% 50% 40% 30% 20% 10% 0% SHOP.COM

Shopping.com

SHOP.COM

Amazon.com

• Q:

“Were the amount of results initially returned: too few, too many or an acceptable amount?”

Too Few Just Right Too Many • • Shopper’s perception of the amount of results returned for their search

NOTE

: Amazon only returns the first 48 products (3 pages of results) and does not display the number of results for their “all department” search

Comparative Analysis Ease of Navigation

Preference of Ease of Navigation

70% 60% 50% 40% 30% 20% 10% 0% S H O P .C

O M S ho pp in g.

co m S am e S H O P .C

O M A m az on • Q:

“Which site was easier to navigate around?”

S am e • Shopper’s impression of which site was easier to navigate around in

Comparative Analysis Overall Layout

Preference of Overall Layout

70% 60% 50% 40% 30% 20% 10% 0% S H O P .C

O M S ho pp in g.

co m S am e S H O P .C

O M A m az on S am e • Q:

“In general, which site’s overall layout do you prefer?”

• Shopper’s preference of overall layout of the sites

Comparative Analysis Site Preference

Preference of Overall Usage

70% 60% 50% 40% 30% 20% 10% 0% S H O P .C

O M S ho pp in g.

co m S am e S H O P .C

O M • Q:

“Overall, which site do you prefer?”

A m az on S am e • Shopper’s overall preference of sites – 6 of 6 shoppers preferring Amazon, listed Amazon as a commonly used site – 2 of 3 shoppers preferring SHOP.COM, listed Amazon as a commonly used site

Qualitative Comparative Feedback Shopping.com (vs. SHOP.COM)

Pros

• Price links on top of Left Navigation • Cleaner UI • Easier to read • Less cluttered

Cons

• Fixed width • Not enough products above the fold • Too far to scroll (defaults to 30 products per page) • Some text too small • Sort was difficult to locate

Qualitative Comparative Feedback Amazon (vs. SHOP.COM)

Pros

• Easier to read • Not as busy • Better use of spacing; more open feeling • Minimum use of boxes, lines • Better color usage – organic, subtle colors • Price closer to photos • Attributes more visible • Better logic in attribute (Left Nav) hierarchy • Friendlier; more inviting, personal, shopping-like

Cons

• Only 3 pages of results (misleading) • Too stretched out • Too open • No descriptions • In-line department links caused confusion • Too much information in the lower left navigation bar

Qualitative Comparative Feedback SHOP.COM

Pros

• Search within • Colors OK; easy on the eyes • More products above the fold • Easy to narrow-by • Clean •

NOTE

Row view was primary format of search results

Cons

• Fonts too small • Overwhelming amount of info; too much to read • Too crowded • Too grid-like; too many boxes and lines • Too top-heavy; red search too overwhelming • Column of red buttons too much attention • No color harmony; too much contrast • Grey too dull • Cold, clinical, techy, utilitarian

Use of Navigational Elements SHOP.COM

• • • • 9 of 18 shoppers missed the “Next Page” links at the top of the page Multiple shoppers complained about the size of the text and links in the grey bar Shoppers complained The “Sort By” links and text seemed to be a lot of links and information Breadcrumb text seemed easy enough for shoppers to indicate quantity of results and location within department hierarchy

Search Results Messaging Live Site

• 6 of 8 shoppers missed the message: “There are 0 matches…” • All shoppers missed the message: “Or see matches in all department…” (has since changed to non-italized) • 5 out of 7 shoppers missed the message: “Here are the closest matches…” – When pointed out, this particular message was understood

Navigation Elements Live Site

• Very little use of Left Navigation links and Search Within function for first-time shoppers • 1 shopper using Left Navigation links, assumed they expanded results • 3 additional shoppers were unsure whether links would expand or narrow

Navigation Elements Mockup

• • • • • • • •

Changes Made Based on Shopper Feedback:

Search within moved to left nav 2 sets of labels added to Left Navigation bar Vertical line added to Left Navigation bar Sort removed for “All Department” search “Showing 1-24…” enlarged and moved out of gray bar Box removed from photos Vertical line next to price column removed Descriptions shortened

Navigation Elements Mockup

• • • •

Results of Usage:

Significant increase in usage of Left Nav and Search Within All but 2 shoppers preferred the new “Search Within” position All shoppers preferred this mockup to original – Better; Cleaner; Less Cluttered All but 1 shopper preferred photos without the outline

Result Listings

• 14 out of 18 shoppers prefer to have the price closer to the product name/photo • 13 out of 18 shoppers indicated they liked having the description

Attribute Quantities

• 12 out of 18 shoppers preferred to have the quantities displayed with the attribute links • 6 out of 18 shoppers indicated that large quantities of results may discourage them from clicking the links (supports findings in previous study)

Single Product Search Results

• No shopper show any confusion or surprise when 1 product search result or navigation link returned a product page • Supports current functionality (recent change to live site)

Search-Specific Recommendations

• • • •

Overall

Explore options to enhance visibility of Search Messaging Increase font size and visibility of “Showing 1-24 of 86” Increase font size of “Next Page” links Explore dropping “Next Page” links on top of results • • • • •

Left Nav Bar

Keep “Search Within” at top of Left Nav bar Use 2 Left Nav headers (“Search Within” and “Narrow Results By”) Maintain visibility of Left Nav headings/labels (as per peach colored background) Increase font size of attribute listing Continue to restrict showing quantity of results until shopper narrows their results – need to establish rule – is currently inconsistent • •

Left Nav Bar

(con’t) Show price attributes high in left nav column Remove ‘Showing # and counts’ from long list of attribute listings (i.e. brand listing) • • • • • •

Display of Results

Minimize usage of lines Remove boxes around photos Shorten descriptions Provide detailed/important information (i.e. books: author, publication date, …) Explore moving price closer to photo and product information Explore dropping or changing color of “view details” buttons

Home Page Initial Reaction

• 14 out of 18 shoppers (78%) had initial positive reactions to this page • Half the shoppers thought the “Featured Store” logos were link-offs

Home Page Verbal Feedback

• • • • • •

Positive Feedback:

Like store listing on right Like the colors Broad selection of products Clean, wide variety of products Nice variety of things Looks pretty good • • • • • •

Negative Feedback:

Animated graphic distracting Lack of focus Not enough access points, few top-level departments displayed Crowded and busy, a lot of things going on A lot of information to comprehend Things too close together

Home Page Ratings of Visual Measures

Rating of Visual Measures on Home Page

3.00

2.80

2.60

2.40

2.20

2.00

1.80

1.60

1.40

1.20

1.00

Professional Looking Easy to begin Shopping Attractiveness Organization • Q sample:

“Do you consider this page attractive, unattractive, or neither?”

• • As reflected in Shopper Feedback, Organization of the Home Page was lowest scoring measure 3 point scale

Merchant Home Page

• • • • 8 out of 10 shoppers expected to see more products 6 out of 10 shoppers thought only 3 products were available from the merchant displayed 4 Shoppers thought the strike-through pricing meant the product was unavailable for sale (missing explanation on search results also) 2 of 3 shoppers navigating in the catalog expressed confusion: – Different UI – Looking for “narrow by” functionality

Multi-Merchant Product Page

• • • • 10 out of 18 shoppers initially thought the product review stars indicated “5 Star Rating” (not grayed out = active) 9 out of 18 shoppers could not find the Zip Code / shipping calculator (some clicked “View shipping info”) No shopper saw the need for pricing at the top and at the merchant level Confusion with red price (at merchant level) - not indicated as “Sale”

On-Line Shopping

• Concept of on-line shopping as “fun”:

– Unexpected finds – Variety of products

• Things that would most influence your decision to return to an ecommerce site:

– Ease of use – Shipping charges – Product pricing – Order fulfillment – Variety of products – Visual appeal of site