Transcript rc-murphy.com
Comparative Usability Study Reporting of Results
Robert Murphy
Study Overview
• Comparative usability study between SHOP.COM and Amazon / Shopping.com with a concentration on search UI elements and use of search functionality
Participants
• 18 Shoppers – 16 women – 2 men – 2 – age 20-29 – 4 – age 30-39 – 7 – age 40-49 – 5 – age 50-59 – 6 w/SHOP.COM experienced – 12 w/out SHOP.COM experience – 13 shoppers mentioned “Amazon” as a commonly used ecommerce site
Study Overview
• • • • •
Methodology
Each shopper performed 2 - 4 unaided searches on each website beginning from the home page Total of 90 searches performed (search and navigation) – 25 on SHOP.COM and Shopping.com
– 20 on SHOP.COM and Amazon – SHOP.COM was always presented first Categories of searches included: – jewelry, shoes, sporting goods, clothing, electronics, auto supplies, books & video Questions rating their experience and preference between sites were asked after performing searches and after using both sites Various additional pages were visited, involving tasks in both navigation and general site usage
2 1.5
1 0.5
0 5 4.5
4 3.5
3 2.5
Study Overview Comparison to previous study
Overall Site Satisfaction Rating
2.8
4.3
• Generally, improved response over similar study conducted April, 2006 – 87% success rate over 60% – Overall satisfaction rating 4.3 over 2.8
April, 2006
Date of Study
Dec, 2006 • Several favorable responses from shoppers: – – –
I had a good experience You guys have a great product I like the 1-cart concept
Overall Findings SHOP.COM
Overall Ratings for SHOP.COM
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 87% 70% Successfully Found Product 60% Searches Returning Relevant Results Searches with Acceptable Number of Results • • • Shoppers successfully found the product they were looking for 87% of the time Shoppers felt the results were relevant in 70% of the searches Shoppers felt an acceptable number of results were returned in 60% of the searches
Use of Navigational Elements SHOP.COM
Percentage Usage of Navigational Elements
35% 30% 25% 20% 15% 10% 5% 0% Search Again 9 Shoppers Left Nav Links 4 Shoppers Search Within 7 Shoppers Additional (Next) Pages 6 Shoppers
Navigational Elements
Sort Results 9 Shoppers • Breakdown of total usage of navigational elements: – Performing a search again – Use of Left Navigation links – Using Search Within function – Clicking to view the “next” page of results – Sorting of results
Use of Navigational Elements SHOP.COM
Percentage of Use of Navigational Elements by SHOP.COM Experience
50% 45% 40% 35% 30% 25% 20% 15% 10% 5% 0% Search Again Left Nav Links Search Within Additional (Next) Pages Sort Results No SHOP.COM Experience SHOP.COM
Experience • Breakdown of total usage of navigation functions by experience on SHOP.COM
– Inexperienced shoppers relied primarily on searching again to refine their results – Experienced shoppers utilized searching again, Search Within and (primarily) Left Nav links – Experienced shoppers did not need to go from page to page of results
Comparative Analysis Successful Searches
Percentage of Successful Searches
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% SHOP.COM
Shopping.com
SHOP.COM
Amazon.com
• Q:
“Do you feel you successfully found the product you were looking for?”
• Percentage of searches where shopper successfully found the product they were looking for
Comparative Analysis Relevancy of Results
100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0%
Percentage of Relevant Results
SHOP.COM
Shopping.com
SHOP.COM
Amazon.com
• Q:
“When you were searching or navigating through the results, was the relevancy of the products displayed what you expected?”
• • • Percentage of searches in which shopper felt their search returned relevant results Areas identified for improvement on SHOP.COM: – shoes, books, sporting equip, clothing Underperforming category on Amazon: – household
Comparative Analysis Quantity of Results
Quantity of Results
80% 70% 60% 50% 40% 30% 20% 10% 0% SHOP.COM
Shopping.com
SHOP.COM
Amazon.com
• Q:
“Were the amount of results initially returned: too few, too many or an acceptable amount?”
Too Few Just Right Too Many • • Shopper’s perception of the amount of results returned for their search
NOTE
: Amazon only returns the first 48 products (3 pages of results) and does not display the number of results for their “all department” search
Comparative Analysis Ease of Navigation
Preference of Ease of Navigation
70% 60% 50% 40% 30% 20% 10% 0% S H O P .C
O M S ho pp in g.
co m S am e S H O P .C
O M A m az on • Q:
“Which site was easier to navigate around?”
S am e • Shopper’s impression of which site was easier to navigate around in
Comparative Analysis Overall Layout
Preference of Overall Layout
70% 60% 50% 40% 30% 20% 10% 0% S H O P .C
O M S ho pp in g.
co m S am e S H O P .C
O M A m az on S am e • Q:
“In general, which site’s overall layout do you prefer?”
• Shopper’s preference of overall layout of the sites
Comparative Analysis Site Preference
Preference of Overall Usage
70% 60% 50% 40% 30% 20% 10% 0% S H O P .C
O M S ho pp in g.
co m S am e S H O P .C
O M • Q:
“Overall, which site do you prefer?”
A m az on S am e • Shopper’s overall preference of sites – 6 of 6 shoppers preferring Amazon, listed Amazon as a commonly used site – 2 of 3 shoppers preferring SHOP.COM, listed Amazon as a commonly used site
Qualitative Comparative Feedback Shopping.com (vs. SHOP.COM)
Pros
• Price links on top of Left Navigation • Cleaner UI • Easier to read • Less cluttered
Cons
• Fixed width • Not enough products above the fold • Too far to scroll (defaults to 30 products per page) • Some text too small • Sort was difficult to locate
Qualitative Comparative Feedback Amazon (vs. SHOP.COM)
Pros
• Easier to read • Not as busy • Better use of spacing; more open feeling • Minimum use of boxes, lines • Better color usage – organic, subtle colors • Price closer to photos • Attributes more visible • Better logic in attribute (Left Nav) hierarchy • Friendlier; more inviting, personal, shopping-like
Cons
• Only 3 pages of results (misleading) • Too stretched out • Too open • No descriptions • In-line department links caused confusion • Too much information in the lower left navigation bar
Qualitative Comparative Feedback SHOP.COM
Pros
• Search within • Colors OK; easy on the eyes • More products above the fold • Easy to narrow-by • Clean •
NOTE
Row view was primary format of search results
Cons
• Fonts too small • Overwhelming amount of info; too much to read • Too crowded • Too grid-like; too many boxes and lines • Too top-heavy; red search too overwhelming • Column of red buttons too much attention • No color harmony; too much contrast • Grey too dull • Cold, clinical, techy, utilitarian
Use of Navigational Elements SHOP.COM
• • • • 9 of 18 shoppers missed the “Next Page” links at the top of the page Multiple shoppers complained about the size of the text and links in the grey bar Shoppers complained The “Sort By” links and text seemed to be a lot of links and information Breadcrumb text seemed easy enough for shoppers to indicate quantity of results and location within department hierarchy
Search Results Messaging Live Site
• 6 of 8 shoppers missed the message: “There are 0 matches…” • All shoppers missed the message: “Or see matches in all department…” (has since changed to non-italized) • 5 out of 7 shoppers missed the message: “Here are the closest matches…” – When pointed out, this particular message was understood
Navigation Elements Live Site
• Very little use of Left Navigation links and Search Within function for first-time shoppers • 1 shopper using Left Navigation links, assumed they expanded results • 3 additional shoppers were unsure whether links would expand or narrow
Navigation Elements Mockup
• • • • • • • •
Changes Made Based on Shopper Feedback:
Search within moved to left nav 2 sets of labels added to Left Navigation bar Vertical line added to Left Navigation bar Sort removed for “All Department” search “Showing 1-24…” enlarged and moved out of gray bar Box removed from photos Vertical line next to price column removed Descriptions shortened
Navigation Elements Mockup
• • • •
Results of Usage:
Significant increase in usage of Left Nav and Search Within All but 2 shoppers preferred the new “Search Within” position All shoppers preferred this mockup to original – Better; Cleaner; Less Cluttered All but 1 shopper preferred photos without the outline
Result Listings
• 14 out of 18 shoppers prefer to have the price closer to the product name/photo • 13 out of 18 shoppers indicated they liked having the description
Attribute Quantities
• 12 out of 18 shoppers preferred to have the quantities displayed with the attribute links • 6 out of 18 shoppers indicated that large quantities of results may discourage them from clicking the links (supports findings in previous study)
Single Product Search Results
• No shopper show any confusion or surprise when 1 product search result or navigation link returned a product page • Supports current functionality (recent change to live site)
Search-Specific Recommendations
• • • •
Overall
Explore options to enhance visibility of Search Messaging Increase font size and visibility of “Showing 1-24 of 86” Increase font size of “Next Page” links Explore dropping “Next Page” links on top of results • • • • •
Left Nav Bar
Keep “Search Within” at top of Left Nav bar Use 2 Left Nav headers (“Search Within” and “Narrow Results By”) Maintain visibility of Left Nav headings/labels (as per peach colored background) Increase font size of attribute listing Continue to restrict showing quantity of results until shopper narrows their results – need to establish rule – is currently inconsistent • •
Left Nav Bar
(con’t) Show price attributes high in left nav column Remove ‘Showing # and counts’ from long list of attribute listings (i.e. brand listing) • • • • • •
Display of Results
Minimize usage of lines Remove boxes around photos Shorten descriptions Provide detailed/important information (i.e. books: author, publication date, …) Explore moving price closer to photo and product information Explore dropping or changing color of “view details” buttons
Home Page Initial Reaction
• 14 out of 18 shoppers (78%) had initial positive reactions to this page • Half the shoppers thought the “Featured Store” logos were link-offs
Home Page Verbal Feedback
• • • • • •
Positive Feedback:
Like store listing on right Like the colors Broad selection of products Clean, wide variety of products Nice variety of things Looks pretty good • • • • • •
Negative Feedback:
Animated graphic distracting Lack of focus Not enough access points, few top-level departments displayed Crowded and busy, a lot of things going on A lot of information to comprehend Things too close together
Home Page Ratings of Visual Measures
Rating of Visual Measures on Home Page
3.00
2.80
2.60
2.40
2.20
2.00
1.80
1.60
1.40
1.20
1.00
Professional Looking Easy to begin Shopping Attractiveness Organization • Q sample:
“Do you consider this page attractive, unattractive, or neither?”
• • As reflected in Shopper Feedback, Organization of the Home Page was lowest scoring measure 3 point scale
Merchant Home Page
• • • • 8 out of 10 shoppers expected to see more products 6 out of 10 shoppers thought only 3 products were available from the merchant displayed 4 Shoppers thought the strike-through pricing meant the product was unavailable for sale (missing explanation on search results also) 2 of 3 shoppers navigating in the catalog expressed confusion: – Different UI – Looking for “narrow by” functionality
Multi-Merchant Product Page
• • • • 10 out of 18 shoppers initially thought the product review stars indicated “5 Star Rating” (not grayed out = active) 9 out of 18 shoppers could not find the Zip Code / shipping calculator (some clicked “View shipping info”) No shopper saw the need for pricing at the top and at the merchant level Confusion with red price (at merchant level) - not indicated as “Sale”
On-Line Shopping
• Concept of on-line shopping as “fun”:
– Unexpected finds – Variety of products
• Things that would most influence your decision to return to an ecommerce site:
– Ease of use – Shipping charges – Product pricing – Order fulfillment – Variety of products – Visual appeal of site