Transcript Tools for decision making and problem solving: Choice (e.g., Comparison shopping, recommender systems)

Am Introduction to
Recommendation Systems
Hasan Davulcu
CIPS, ASU
Recommendation Systems
U: C X S  R
C: profile: age, gender, income
S: title, director, actor, year, genre
Recommendations:
Recommender Systems
Content Based
Limitations
Too Similar !
 New user problem

User - Collaborative
Methods

U(C,S) is
estimated based
on utilities U(Cj,S)
by those users Cj
who are “similar”
to user C.
Limitations
New Item Problem
 Sparsity

Amazon’s Item-to-Item
Comparing Human
Recommenders to Online
Systems
Rashmi Sinha &
Kirsten Swearingen
SIMS, UC Berkeley
Recommender Systems are
technological proxy for a social process
Which one
should I read?
Recommendations
from friends
Recommendations
from Online
Systems
I know what you’ll read next summer
(Amazon, Barnes&Noble)

what movies you should watch…
(Reel, RatingZone, Amazon)

what music you should listen to…
(CDNow, Mubu, Gigabeat)

what websites you should visit
(Alexa)

what jokes you will like (Jester)

& who you should date (Yenta)
Method Philosophy: Testing & Analysis as
part of the Iterative Design Process
Design
Evaluate
Use both quantitative
& qualitative methods
Analyze
Slide adapted from James Landay
Generate Design
Recommendations
Taking a closer look at the
Recommendation Process
Input
User incurs cost in using
system:
Time,
Effort, Privacy Issues
Receives
Recommendation
Cost in reviewing
recommendations
Judges if he/she
will sample
recommendation
Benefit only if recommended item
appeals
Amazon’s Recommendation Process

Input: One artist/author name
Search using
Recommendations


Output: List of Recommendations
Explore / Refine Recommendations
Book Recommendation Site: Sleeper


Input: Ratings of 10 books for all users
Use of continuous Rating Bar
(System designed by Ken Goldberg)
Sleeper: Output


Output: List of items with brief
information about each item
Degree of confidence in prediction
What convinces a user to
sample the recommendation

Judging recommendations:


Trust in a Recommender System:


What is a good recommendation from the
user’s perspective?
What factors lead to trust in a system?
System Transparency:

Do users need to know why an item was
recommended?
Study of RS has focused mostly on
Collaborative Filtering Algorithms
Social
Recommendations
Input from user
Output
(Recommendations)
Collaborative
Filtering Algorithms
Beyond “Algorithms Only” : An HCI
Perspective on Recommender Systems

Comparing the Social Recommendation
Process to Online Recommender Systems

Understanding the factors that go into an
effective recommendation (by studying
users interaction with 6 online RS)
The Human vs. Recommenders
Death Match
Book Systems
Amazon Books
Rating Zone
Sleeper
Movie Systems
Amazon Movies
Movie Critic
Reel
Method
19

For each of 3 online systems:





participants, age:18 to 34 years
Registered at site
Rated items
Reviewed and evaluated recommendation set
Completed questionnaire
Also reviewed and evaluated sets of
recommendations from 3 friends each
Results
Defining Types of
Recommendations
GOOD:
User likes
USEFUL
Not yet read/viewed
Good Recs. (Precision)
•% items user felt
interested in
Useful Recs.
•Subset of Good
Recs.
•User felt interested
in and had not read /
viewed yet
Comparing Human Recommenders to RS:
“Good” and “Useful” Recommendations
% Good Recommendations
% Useful Recommendations
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0
Amazon Sleeper Rating Friends
(10)
Zone (8) (9)
(15)
Amazon
(15)
Reel
Movie
Friends
(5-10) Critic (20)
(9)
Movies
Books
Ave. Std. Error
(x)
No. of Recommendations
RS Average
However users like online RS
Number preferring system
Do you prefer recommendations from friends or online
systems?
System
Friends
7.00
6.00
5.00
4.00
3.00
2.00
1.00
0.00
Books
Movies
This result was supported by post test interviews.
Why systems over friends?

“Suggested a number of things I hadn’t heard of,
interesting matches.”

“It was like going to Cody’s—looking at that
table up front for new and interesting books.”

“Systems can pull from a large database—no
one person knows about all the movies I might
like.”
Items users had “Heard of” before
% Heard of
70
60
50
40
30
20
10
0
Amazon
S leeper RatingZone Friends
Books
Amazon MovieCritic
Reel
Friends
Movies
Friends recommended mostly “old” previously
experienced items
What systems did users prefer?
Yes
Would you use system again?
No
Amazon
Sleeper
0.8
Average Rating
0.6
0.4
0.2
0
-0.2
-0.4
-0.6
Books


RatingZone
Amazon
Reel
MovieCritic
Movies
Sleeper and Amazon books average highest ratings
Split opinions on Reel, MovieCritic
Why did some systems…

Provide useful recommendations but
leave users unsatisfied?

RatingZone, MovieCritic & Reel
Possible Reasons
What predicts overall usefulness of a System?
0.6
Correlation
0.5
All
correlations
are significant
at .05
0.4
0.3
0.2
0.1
0
Good Rec. Useful Rec.



Trust
Adequate
Generating
Item
Rec.
Description
Ease of
Use
Previously Enjoyed Items are important: We term these
Trust-Generating Items
Adequate Item Description & Ease of Use are important
Missing from List: Time to Receive Recommendations &
No. of Items to Rate not important!
A Question of Trust…
GOOD:
User likes
USEFUL
Not yet read/viewed
TRUST-GENERATING
Previously read/viewed
Post Test Interviews showed
that users “trust” systems if
they have already sampled
some recommendations
•Positive Experiences lead to
“trust
•Negative Experiences with
Recommended Items lead to
mistrust of system
A Question of Trust …
Number of Trust Generating Items
Number of Items
6.00
5.00
4.00
3.00
2.00
1.00
.00
Amazon
Books
Sleeper
RatingZone
Amazon
Reel
M ovieCritic
Movies
Difference between Amazon and Sleeper highlights the fact that
there are different kinds of good Recommender Systems
Adequate Item Description: The
RatingZone Story
% Useful Recs.
% Useful For Both Versions of RatingZOne
45
40
35
30
25
20
15
0 % of Version 1
and 60% of
Version 2 users
found item
description
adequate
10
5
0
Version 1: Without Description
Version 2: With Description
An adequate item description, and links to other sources
about item was a crucial factor in users being convinced by a
recommendation.
System Transparency

Why was this item recommended?

Do users understand why an item was recommended
% Good Recommendations
Effect of System Transparency on Recommendation
60
50
40
30
20
10
0
System Reasoning was
Transparent
System Reasoning Not
Transparent
Users mentioned this factor in post test interviews
Discussion &
Design Recommendations
Design Recommendations:
Justification

Justify your Recommendations




Adequate Item Information: Providing enough detail about
item for user to make choice
System Transparency: Generate (at least some)
recommendations which are clearly linked to the rated
items
Explanation: Provide an Explanation, why the item was
recommended.
Community Ratings: Provide link to ratings / reviews by
other users. If possible, present numerical summary of
ratings.
Design Recommendations:Accuracy
vs. Less Input
Don’t
sacrifice accuracy for the sake of generating
quick recommendations. Users don’t mind rating
more items to receive quality recommendations.
A
possible way to achieve this: have multilevel
recommendations. Users can initially use the system by
providing one rating, and are offered subsequent
opportunities to refine recommendation
One
needs a happy medium between too little input
(leading to low accuracy) and too much input (leading to
user impatience)
Design Recommendations: New
Unexpected Items

Users like Rec. Systems as they provide
information about new, unexpected items.


List of recommended items should include new
items which the user might not find out in any
other way.
List could also include some unexpected items
(e.g., from other topics / genres) which the user
might not have thought of themselves.
Design Recommendations: Trust
Generating Items
 Users (especially first time users) need to
develop trust in the system.
 Trust in system is enhanced by the presence of
items that the user has already enjoyed.
 Generating some very popular (which have
probably been experienced previously) in the
initial recommendation set might be one way to
achieve this.
Design Recommendations: Mix of
Items
 Systems need to provide a mix of different kinds of
items to cater to different users:
 Trust Generating Items: A few very popular ones, which
the system has high confidence in
 Unexpected Items: Some unexpected items, whose
purpose is to allow users to broaden horizons.
 Transparent Items: At least some items for which the user
can see the clear link between the items he /she rated and
the recommendation.
 New Items: Some items which are new.
Question: Should these be presented as a sorted list /
unsorted list/ different categories of recommendations?
Design Recommendations: Continuous
Scales for Input
 Allow users to provide ratings on a
continuous scale.
 One of the reasons users liked Sleeper was
because it allowed them to rate on a continuous
scale. Users did not like binary scales.
Limitations of Study

Simulated first-time visit, did not allow system to
learn user preferences over time

Source of recommendations known to subjects—
might have biased towards friends

Fairly homogenous group of subjects, no novice
users
Future Plans: Second Generation
Music Recommender Systems
•Have evolved beyond previous systems
•Use a variety of sophisticated algorithms to map users
preferences over music domain
•Require a lot more input from the user
•Users can sample recommendations during the study!
MusicBudha (Mubu.com): Exploring Genres
Mubu.com: Exploring Jazz Styles
Mubu.com: Rating Samples
Mubu.com: Recommendations as Audio Samples
The Turing Test for Music
Recommender Systems


Compare systems, friends and experts
Anonymize the source of recommendation
Study
Design
Friends
Music Experts
Online RS
Goal: Compare recommendations by online RS, Experts
(who have same information as RS) & Friends
In conclusion…
So far we have heard what this study tells us about
Recommender Systems
But what (if anything) does it have
to say about human nature?
Recommender Systems tantalize us with the idea
that we are not as unique and unpredictable as we
think we are.
Study results show that Recommender Systems
do not know us better than our friends!
But …
11 out of 19 users preferred Recommender
Systems over Friends Recommendations!
Ultimately, we all want to be tables in
a database!
Email: [email protected]
Web address: http://sims.berkeley.edu/~sinha