Transcript When Do Noisy Votes Reveal the Truth? Ioannis Caragiannis1 Ariel D. Procaccia2 Nisarg Shah2 ( speaker ) University of Patras & CTI 2 Carnegie Mellon University.

When Do Noisy Votes
Reveal the Truth?
Ioannis Caragiannis1
Ariel D. Procaccia2
Nisarg Shah2 ( speaker )
1
University of Patras & CTI
2 Carnegie Mellon University
What? Why?
• What?




Alternatives to be compared
True order (unknown ground truth)
Noisy estimates (votes) drawn from
some distribution around it
Q: How many votes are needed to
accurately find the true order?
• Why?


Practical motivation
Theoretical motivation
b>a>c>d
a>c>b>d
a>b>c>d
a>b>d>c
Alternatives
a, b, c, d
2
Practical Motivation
1. Human Computation


EteRNA, Foldit,
Crowdsourcing …
How many users/workers
are required?
2. Judgement Aggregation


Jury system, experts ranking
restaurants, …
How many experts are
required?
Theoretical Motivation
• Maximum Likelihood Estimator
(MLE) View: Is a given voting rule
the MLE for any noise model?
Voting
Rules
Noise
Models
• Problems



Only 1 MLE/noise model
Strange noise models
Noise model is usually unknown
• Our Contribution



MLE is too stringent!
Just want low sample complexity
Family of reasonable noise models
4
Boring Stuff!
• Voting rule (𝑟)



Input  several rankings of alternatives
Social choice function (traditionally) : Output  a winning alternative
Social welfare function (this work) : Output  a ranking of alternatives
• Noise model over rankings (𝑀)


For every ground truth 𝜎 ∗ and every ranking σ  Pr 𝜎 𝜎 ∗
Mallows’ model : Pr 𝜎 𝜎 ∗ ∝ exp{−𝜆 𝑑𝐾 𝜎, 𝜎 ∗ }

𝑑𝐾 = Kendall-Tau distance = #pairwise comparisons two rankings disagree on
• Sample complexity of rule 𝑟 for model 𝑀 and accuracy 𝜀

Smallest 𝑛  For every σ*, Pr𝑀 [𝑟(𝑛 i.i.d. 𝑠𝑎𝑚𝑝𝑙𝑒𝑠) = 𝜎 ∗ ] ≥ 1 − 𝜀
5
Sample Complexity for
Mallows’ Model
• Kemeny rule (+ any tie-breaking) = MLE
Maximize
𝑖
exp{−𝜆 𝑑𝐾 𝜎𝑖 , 𝜎 } = Minimize
𝑖
𝑑𝐾 σ𝑖 , σ
• Theorem: Kemeny rule + uniformly random tie-breaking =
optimal sample complexity for Mallows’ model, any accuracy.

Subtlety: MLE does not always imply optimal sample complexity!
• So, are the other voting rules really bad for Mallows’ model?

No.
6
PM-c and PD-c Rules
• Pairwise Majority Consistent Rules (PM-c)


Must match the pairwise majority graph whenever it is acyclic
Condorcet consistency for social welfare functions
𝑎 ≻𝑏≻𝑐≻𝑑
𝑏≻𝑐≻𝑑≻𝑎
𝑎≻𝑏≻𝑐≻𝑑
𝑑≻𝑏≻𝑎≻𝑐
𝑎≻𝑑≻𝑏≻𝑐
a
c
𝑎≻𝑏≻𝑐≻𝑑
b
d
7
PM-c and PD-c Rules
• PD-c rules  similar, but focus on positions of alternatives
PD-c
PM-c
KM
SL
PSR
SC
BL
CP
RP
8
The Big Picture
 PM-c O(log m) (m = #alternatives)
 Any voting rule Ω(log m)
Exponential
Polynomial
PM-c
Logarithmic
 Kemeny rule +
uniform tie breaking
 Optimal sample
complexity
Many scoring rules
 Plurality, veto
 Strictly exponential
9
Take-Away - I
 Given any fixed noise model, sample
complexity is a clear and useful
criterion for selecting voting rules
• Hey, what happened to the noise model being unknown?
10
Generalization
• Stronger need  Unknown noise model
Working well on a family of reasonable noise models
• Problems
1.
2.
What is reasonable?
HUGE sample complexity for near-extreme parameter values!
• Relaxation  Accuracy in the Limit
Ground truth with probability 1 given infinitely many samples
• Novel axiomatic property
11
Accuracy in the Limit
Monotonicity is reasonable,
but why Kendall-Tau distance?
Voting Rules
Noise models for which they are accurate in the limit
PM-c + PD-c
Mallows’ model
(probability decreases exponentially in the KT distance)
PM-c + PD-c
All KT-monotonic noise models
(probability decreases monotonically in the KT distance)
PM-c
All d-monotonic iff d = Majority Concentric (MC)
PD-c
All d-monotonic iff d = Position Concentric (PC)
PM-c + PD-c
All d-monotonic iff d = both MC and PC
12
Take-Away - II
 Robustness  accuracy in the limit over a family of
reasonable noise models
 d-monotonic noise models  reasonable


If you believe in PM-c and PD-c rules  look for distances that are
both MC and PC Kendall-Tau, footrule, maximum displacement
Cayley distance and Hamming distance are neither MC nor PC


Even the most popular rule – plurality – is not accurate in the limit for any
monotonic noise model over either distance !
Lose just too much information for the true ranking to be recovered
13
Distances over Rankings
• MC (Majority-Concentric) Distance


Ranking 𝜎 ∗ , distance 𝑘  𝐵𝑘 (𝜎 ∗ ) = 𝜎 𝑑 𝜎, 𝜎 ∗ ≤ 𝑘}
For every pairwise comparison, a (weak) majority of rankings in every
𝐵𝑘 (𝜎 ∗ ) must agree with 𝜎 ∗
σ*
𝑎 ≻ 𝑏?
σ*
𝑐 ≻ 𝑑?
14
Discussion
1. The stringent MLE requirement  sample complexity

Connections to axiomatic and distance rationalizability views?
2. Noise model unknown  d-monotonic noise models


Some distances over rankings are better suited for voting than
others (e.g., MC and PC distances)
An extensive study of the applicability of various distance metrics in
social choice
3. Practical applications  Extension to voting with partial
information - pairwise comparisons, partial orders, top-𝑘 lists
15