Transcript ppt

Multi-Click Search Behavior
Chia-Jung Lee1, Jaime Teevan2,
and Sebastian de la Chica2
1 University of Massachusetts, 2 Microsoft
teevan children
Jaime Teevan, Ph.D.
research.microsoft.com/en-us/um/people/teevan/publications
Jaime Teevan, Ph.D. Researcher studying information retrieval and human computer
interaction at Microsoft Research.
Jaime Teevan (Microsoft Research)
research.microsoft.com/en-us/um/people/teevan
Jaime Teevan, Ph.D. Researcher studying information retrieval and human computer
interaction at Microsoft Research.
Teevan.com | Restoration, Preservation, Remodeling …
www.teevan.com
Voted San Francisco’s favorite small business. Call us at 415-474-TVAN. Building
Restoration, Preservation, Friends and Community.
Jaime Teevan – Wikipedia, the free encyclopedia
en.wikipedia.org/wiki/Jaime_Teevan
Jaime Teevan is a researcher at Microsoft Research and an affiliate professor at the
University of Washington. She was named a Technology Review (TR35) 2009 Young ...
Inside Microsoft Research – Home – TechNet Blog
blogs.technet.com/b/inside_microsoft_research/rss.aspx
Jaime Teevan, it seems, can do it all. Since joining Microsoft Research in 2006, ...
Researcher Teevan Wins Borg Early Career Award. Posted by Rob Knies.
teevan children
Jaime Teevan, Ph.D.
research.microsoft.com/en-us/um/people/teevan/publications
Jaime Teevan, Ph.D. Researcher studying information retrieval and human computer
interaction at Microsoft Research.
For kids, by a kid – Bellevue Reporter
59%
www.bellevuereporter.com/community/207896681.html
Griffin's mother, Jaime Teevan, said she first learned about Griffin's book when he
was three months in to writing it. At the time, Griffin's teacher, ...
Jaime Teevan – Wikipedia, the free encyclopedia
en.wikipedia.org/wiki/Jaime_Teevan
Jaime Teevan is a researcher at Microsoft Research and an affiliate professor at the
University of Washington. She was named a Technology Review (TR35) 2009 Young ...
Slow Searching: Attending a Conference with a Baby
slowsearching.blogspot.com/.../attending-conference...with-baby.html
May 03, 2013 · Below are some things that I found made attending a conference
alone with a baby a little easier for me. ... Jaime Teevan Pacific Northwest, …
Inside Microsoft Research – Home – TechNet Blog
blogs.technet.com/b/inside_microsoft_research/rss.aspx
Jaime Teevan, it seems, can do it all. Since joining Microsoft Research in 2006, ...
Researcher Teevan Wins Borg Early Career Award. Posted by Rob Knies.
teevan children
Jaime Teevan, Ph.D.
research.microsoft.com/en-us/um/people/teevan/publications
Jaime Teevan, Ph.D. Researcher studying information retrieval and human computer
interaction at Microsoft Research.
For kids, by a kid – Bellevue Reporter
www.bellevuereporter.com/community/207896681.html
Griffin's mother, Jaime Teevan, said she first learned about Griffin's book when he
was three months in to writing it. At the time, Griffin's teacher, ...
Jaime Teevan – Wikipedia, the free encyclopedia
Build a Picture of
Multi-Click Behavior
en.wikipedia.org/wiki/Jaime_Teevan
Jaime Teevan is a researcher at Microsoft Research and an affiliate professor at the
University of Washington. She was named a Technology Review (TR35) 2009 Young ...
Slow Searching: Attending a Conference with a Baby
slowsearching.blogspot.com/.../attending-conference...with-baby.html
May 03, 2013 · Below are some things that I found made attending a conference
alone with a baby a little easier for me. ... Jaime Teevan Pacific Northwest, …
Inside Microsoft Research – Home – TechNet Blog
blogs.technet.com/b/inside_microsoft_research/rss.aspx
Jaime Teevan, it seems, can do it all. Since joining Microsoft Research in 2006, ...
Researcher Teevan Wins Borg Early Career Award. Posted by Rob Knies.
teevan children
Jaime Teevan, Ph.D.
teevan children
Jaime Teevan, Ph.D.
research.microsoft.com/en-us/um/people/teevan/publications research.microsoft.com/en-us/um/people/teevan/public
Jaime Teevan, Ph.D. Researcher studying information retrieval and
Jaime
human
Teevan,
computer
Ph.D. Researcher studying information retr
interaction at Microsoft Research.
interaction at Microsoft Research.
Jaime Teevan (Microsoft Research)
For kids, by a kid – Bellevue Reporter
research.microsoft.com/en-us/um/people/teevan
www.bellevuereporter.com/community/207896681.html
Jaime Teevan, Ph.D. Researcher studying information retrieval and
Griffin's
humanmother,
computer
Jaime Teevan, said she first learned abou
interaction at Microsoft Research.
was three months in to writing it. At the time, Griffin's tea
Teevan.com | Restoration, Preservation, Remodeling … Jaime Teevan – Wikipedia, the free encyclopedia
www.teevan.com
en.wikipedia.org/wiki/Jaime_Teevan
Voted San Francisco’s favorite small business. Call us at 415-474-TVAN.
Jaime Teevan
Buildingis a researcher at Microsoft Research and a
Restoration, Preservation, Friends and Community.
University of Washington. She was named a Technology R
Jaime Teevan – Wikipedia, the free encyclopedia
Slow Searching: Attending a Conference with a Bab
en.wikipedia.org/wiki/Jaime_Teevan
slowsearching.blogspot.com/.../attending-conference...w
Jaime Teevan is a researcher at Microsoft Research and an affiliate
May
professor
03, 2013
at· the
Below are some things that I found made
University of Washington. She was named a Technology Review (TR35)
alone with
2009aYoung
baby a...little easier for me. ... Jaime Teevan P
Inside Microsoft Research – Home – TechNet Blog
Inside Microsoft Research – Home – TechNet Blog
blogs.technet.com/b/inside_microsoft_research/rss.aspx
blogs.technet.com/b/inside_microsoft_research/rss.aspx
Jaime Teevan, it seems, can do it all. Since joining Microsoft Research
JaimeinTeevan,
2006, ...it seems, can do it all. Since joining Microso
Researcher Teevan Wins Borg Early Career Award. Posted by RobResearcher
Knies.
Teevan Wins Borg Early Career Award. Posted
Problem Specification
𝑇 = < 𝑄, 𝑆1 , 𝐶𝑆1 , 𝛿𝑆1→𝑆2 >
User-Centric Factors
System-Centric Factors
• User issues query 𝑄
• User views results 𝑆1
• User clicks result 𝐶𝑆1
• Search results
change from 𝑆1 to 𝑆2
1. Which factors from 𝑇 impact user interaction with 𝑆2
and how? Which lead to the most user satisfaction?
2. How do changes to the result page (𝛿𝑆1→𝑆2) impact users?
Do changes improve or degrade user experience?
Dataset
• Two months of log data from Bing
• Identify queries with a click and a return
– Filtered for bots, spam, outliers
– Adult and navigational queries removed
Queries
8,863,684
Clicks
17,154,920
Users
1,658,931
• Know results shown before and after click
Problem Specification
𝑇 = < 𝑄, 𝑆1 , 𝐶𝑆1 , 𝛿𝑆1→𝑆2 >
User-Centric Factors
• User issues query 𝑄
• User views results 𝑆1
• User clicks result 𝐶𝑆1
1. Which factors from 𝑇 impact user interaction with 𝑆2
and how? Which lead to the most user satisfaction?
2. How do changes to the result page (𝛿𝑆1→𝑆2) impact users?
Do changes improve or degrade user experience?
Measures of Search Success
• Number of clicks
– 𝑝(𝐴𝑏𝑎𝑛𝑑𝑜𝑛) =
#({𝑞𝑢𝑒𝑟𝑖𝑒𝑠, 𝑎𝑏𝑎𝑛𝑑𝑜𝑛})
#( 𝑞𝑢𝑒𝑟𝑖𝑒𝑠 )
• Satisfaction with click
– 𝑅𝑆𝐴𝑇 =
#({𝑞𝑢𝑒𝑟𝑖𝑒𝑠, 𝑅𝑒𝑡𝑢𝑟𝑛𝑆𝐴𝑇 })
#({𝑞𝑢𝑒𝑟𝑖𝑒𝑠,𝑅𝑒𝑡𝑢𝑟𝑛𝑁𝑆𝐴𝑇 })
• Click position
– Position change: Up, Down, Stay
–𝑝 𝐶=𝑐 =
• Time to click
#({𝑞𝑢𝑒𝑟𝑖𝑒𝑠, 𝑐})
#({𝑞𝑢𝑒𝑟𝑖𝑒𝑠})
,𝑐 ∈ 𝐶
Summary of Post-Click Behavior
• Confirm trend to move down the page
– Top results revisited for later clicks
– Lower initial clicks mean larger subsequent jumps
• Satisfied search behavior different
– Slower, less likely to identify new content to visit
– More likely to be satisfied when they do
• Searchers learn about the search results
– Second click faster than the first
Problem Specification
𝑇 = < 𝑄, 𝑆1 , 𝐶𝑆1 , 𝛿𝑆1→𝑆2 >
User-Centric Factors
System-Centric Factors
• User issues query 𝑄
• User views results 𝑆1
• User clicks result 𝐶𝑆1
• Search results
change from 𝑆1 to 𝑆2
1. Which factors from 𝑇 impact user interaction with 𝑆2
and how? Which lead to the most user satisfaction?
2. How do changes to the result page (𝛿𝑆1→𝑆2) impact users?
Do changes improve or degrade user experience?
Measures of Result Change
• Clicked result changes
• Other results change
– Move up or down
– Stay in the same place
– Disappear entirely
– Above or below
– Assume clicked result
stays the same
Jaime Teevan – Wikipedia, the free encyclopedia
en.wikipedia.org/wiki/Jaime_Teevan
Jaime Teevan is a researcher at Microsoft Research and an
affiliate professor at the University of Washington. She...
Jaime Teevan – Wikipedia, the free encyclopedia
Jaime Teevan – Wikipedia, the free encyclopedia
en.wikipedia.org/wiki/Jaime_Teevan
Jaime Teevan is a researcher at Microsoft Research and an
affiliate professor at the University of Washington. She...
en.wikipedia.org/wiki/Jaime_Teevan
Jaime Teevan is a researcher at Microsoft Research and an
affiliate professor at the University of Washington. She...
Jaime Teevan – Wikipedia, the free encyclopedia
en.wikipedia.org/wiki/Jaime_Teevan
Jaime Teevan is a researcher at Microsoft Research and an
affiliate professor at the University of Washington. She...
Change = Slower Interaction
• Every type of change delays second click
Time to click S2 (secs)
9
5.5
Down
Gone
Stay
Up
2
0
4
8
12
16
Time to click S1 (secs)
• Change to click: Up > Down > Gone
• Changes around click: Above > Below
20
Change = More Abandonment
• Change to click
– Down > Stay > Up > Gone
• Changes around click
Up
0.432
Stay
0.425
Down
0.375
Gone
0.492
– Changes above increases abandonment
– Changes below decreases it
Changes
Static
Above
0.414
0.396
Below
0.423
0.431
Change = Increased Satisfaction
• Change to click
– Unsatisfied initially
• Gone > Down > Stay > Up
– Satisfied initially
NSAT
SAT
Up
2.00
4.65
Stay
2.08
4.78
Down
2.20
4.75
Gone
2.31
4.61
• Stay > Down > Up > Gone
• Changes around click
– Always benefit NSAT users
– Best below the click for
satisfied users
NSAT
SAT
Changes
Static
Above
2.30
4.93
2.21
4.93
Below
2.09
4.79
1.99
4.61
Post-Click Behavior with Change
• Changes appear to attract attention
• Correlated with less, slower interaction
– Second click slower following change
– People abandon more following most changes
↑Clicked result disappears, results above change
↓Clicked result moves down, results below change
• Users more satisfied when they do interact
– Particularly for changes below click
– Satisfied users tolerate less change
QUESTIONS?
Clicks Progress Down Page
Pos1
0.52
Pos2
Pos3
Probability
0.39
Pos4
Pos5
0.26
Pos6
Pos7
0.13
Pos8
Pos9
0
C1
C2
C3
C4
C5
C6
Click Position
C7
C8
C9
C10
Pos1
0
Initial Click Position Matters
• Initial click can be on a top result or low
• When the initial click is low
– Searchers tend to make larger jumps
– More likely to move up the list than down
– Search is more likely to be abandoned
0.55
Prob of Abandon
0.43
0.45
0.39
0.39
0.40
0.41
0.51
0.54
0.48
0.44
0.46
0.35
0
2
4
6
Initial Click Position
8
10
Second Click Faster
Time to Click S2 (secs)
9
Particularly
after NSAT
All
C1:SAT
C1:NSAT
2
0
4
8
12
Time to Click S1 (secs)
16
20
Clicks Oriented Around Initial Click
• Click position much more likely to move
upwards if initial click moves upwards
S1: dog breed
1
2
3
4
5
6
7
8
www.nextdaypets.com/directory/breeds
www.dogbreederdirectory.com/breeders...
www.justdogbreeds.com/dog-breeds.html
www.justdogbreeds.com/dog-breeds.html
en.wikipedia.org/wiki/List_of_dog_breeds
www.nctoybreeds.com/
en.wikipedia.org/wiki/Breed_of_dog
www.democraticunderground.com/discus...
www.dogbreedinfo.com/abc.htm
Clicks Oriented Around Initial Click
• Click position much more likely to move
upwards if initial click moves upwards
1
2
3
4
5
6
7
8
9
10
11
12
S1: dog breed
S2: dog breed
www.nextdaypets.com/directory/breeds
www.dogbreederdirectory.com/breeders...
www.justdogbreeds.com/dog-breeds.html
en.wikipedia.org/wiki/List_of_dog_breeds
www.nctoybreeds.com/
en.wikipedia.org/wiki/Breed_of_dog
www.democraticunderground.com/discus...
www.dogbreedinfo.com/abc.htm
www.nextdaypets.com/directory/breeds
www.justdogbreeds.com/dog-breeds.html
www.dogbreederdirectory.com/breeders...
en.wikipedia.org/wiki/List_of_dog_breeds
www.nctoybreeds.com/
en.wikipedia.org/wiki/Breed_of_dog
www.democraticunderground.com/discuss...
www.justdogbreeds.com/
www.dogbreedinfo.com/abc.htm
www.akc.org/breeds/
www.dogbreedinfo.com/index.htm
www.terrificpets.com/dog_breeds/
Clicks Oriented Around Initial Click
• Click position much more likely to move
upwards if initial click moves upwards
• Click position much more likely to move
upwards if above results change