Transcript PPT
Theory of LSH
Distance Measures
LS Families of Hash Functions
S-Curves
1
Distance Measures
Generalized LSH is based on some
kind of “distance” between points.
Similar points are “close.”
Two major classes of distance
measure:
1. Euclidean
2. Non-Euclidean
2
Euclidean Vs. Non-Euclidean
A Euclidean space has some number of
real-valued dimensions and “dense” points.
There is a notion of “average” of two points.
A Euclidean distance is based on the locations
of points in such a space.
A Non-Euclidean distance is based on
properties of points, but not their
“location” in a space.
3
Axioms of a Distance Measure
d is a distance measure if it is a
function from pairs of points to real
numbers such that:
1.
2.
3.
4.
d(x,y)
d(x,y)
d(x,y)
d(x,y)
>
=
=
<
0.
0 iff x = y.
d(y,x).
d(x,z) + d(z,y) (triangle
inequality ).
4
Some Euclidean Distances
L2 norm : d(x,y) = square root of the
sum of the squares of the differences
between x and y in each dimension.
The most common notion of “distance.”
L1 norm : sum of the differences in
each dimension.
Manhattan distance = distance if you had
to travel along coordinates only.
5
Examples of Euclidean Distances
b = (9,8)
L2-norm:
dist(x,y) =
(42+32)
=5
5
4
a = (5,5)
3
L1-norm:
dist(x,y) =
4+3 = 7
6
Another Euclidean Distance
L∞ norm : d(x,y) = the maximum of
the differences between x and y in
any dimension.
Note: the maximum is the limit as n
goes to ∞ of the Ln norm: what you
get by taking the n th power of the
differences, summing and taking the
n th root.
7
Non-Euclidean Distances
Jaccard distance for sets = 1 minus
Jaccard similarity.
Cosine distance = angle between vectors
from the origin to the points in question.
Edit distance = number of inserts and
deletes to change one string into another.
Hamming Distance = number of positions
in which bit vectors differ.
8
Jaccard Distance for Sets
(Bit-Vectors)
Example: p1 = 10111; p2 = 10011.
Size of intersection = 3; size of union =
4, Jaccard similarity (not distance) =
3/4.
d(x,y) = 1 – (Jaccard similarity) = 1/4.
9
Why J.D. Is a Distance Measure
d(x,x) = 0 because xx = xx.
d(x,y) = d(y,x) because union and
intersection are symmetric.
d(x,y) > 0 because |xy| < |xy|.
d(x,y) < d(x,z) + d(z,y) trickier – next
slide.
10
Triangle Inequality for J.D.
1 - |x z| + 1 - |y z| > 1 - |x y|
|x z|
|y z|
|x y|
Remember: |a b|/|a b| = probability
that minhash(a) = minhash(b).
Thus, 1 - |a b|/|a b| = probability
that minhash(a) minhash(b).
11
Triangle Inequality – (2)
Claim: prob[minhash(x) minhash(y)] <
prob[minhash(x) minhash(z)] +
prob[minhash(z) minhash(y)]
Proof: whenever minhash(x) minhash(y), at
least one of minhash(x) minhash(z) and
minhash(z) minhash(y) must be true.
12
Cosine Distance
Think of a point as a vector from the
origin (0,0,…,0) to its location.
Two points’ vectors make an angle,
whose cosine is the normalized dotproduct of the vectors: p1.p2/|p2||p1|.
Example: p1 = 00111; p2 = 10011.
p1.p2 = 2; |p1| = |p2| = 3.
cos() = 2/3; is about 48 degrees.
13
Cosine-Measure Diagram
p1
p1.p2
|p2|
d (p1, p2) =
p2
= arccos(p1.p2/|p2||p1|)
14
Why C.D. Is a Distance Measure
d(x,x) = 0 because arccos(1) = 0.
d(x,y) = d(y,x) by symmetry.
d(x,y) > 0 because angles are chosen
to be in the range 0 to 180 degrees.
Triangle inequality: physical reasoning.
If I rotate an angle from x to z and
then from z to y, I can’t rotate less
than from x to y.
15
Edit Distance
The edit distance of two strings is the
number of inserts and deletes of
characters needed to turn one into the
other. Equivalently:
d(x,y) = |x| + |y| - 2|LCS(x,y)|.
LCS = longest common subsequence = any
longest string obtained both by deleting from
x and deleting from y.
16
Example: LCS
x = abcde ; y = bcduve.
Turn x into y by deleting a, then
inserting u and v after d.
Edit distance = 3.
Or, LCS(x,y) = bcde.
Note: |x| + |y| - 2|LCS(x,y)| =
5 + 6 –2*4 = 3 = edit distance.
17
Why Edit Distance Is a
Distance Measure
d(x,x) = 0 because 0 edits suffice.
d(x,y) = d(y,x) because insert/delete
are inverses of each other.
d(x,y) > 0: no notion of negative edits.
Triangle inequality: changing x to z
and then to y is one way to change x
to y.
18
Variant Edit Distances
Allow insert, delete, and mutate.
Change one character into another.
Minimum number of inserts, deletes, and
mutates also forms a distance measure.
Ditto for any set of operations on strings.
Example: substring reversal OK for DNA
sequences
19
Hamming Distance
Hamming distance is the number of
positions in which bit-vectors differ.
Example: p1 = 10101; p2 = 10011.
d(p1, p2) = 2 because the bit-vectors
differ in the 3rd and 4th positions.
20
Why Hamming Distance Is a
Distance Measure
d(x,x) = 0 since no positions differ.
d(x,y) = d(y,x) by symmetry of
“different from.”
d(x,y) > 0 since strings cannot differ in
a negative number of positions.
Triangle inequality: changing x to z
and then to y is one way to change x
to y.
21
Families of Hash Functions
1. A “hash function” is any function that
takes two elements and says whether
or not they are “equal” (really, are
candidates for similarity checking).
Shorthand: h(x) = h(y) means “h says x
and y are equal.”
2. A family of hash functions is any set
of functions as in (1).
22
LS Families of Hash Functions
Suppose we have a space S of points
with a distance measure d.
A family H of hash functions is said to
be (d1,d2,p1,p2)-sensitive if for any x
and y in S :
1. If d(x,y) < d1, then prob. over all h in H,
that h(x) = h(y) is at least p1.
2. If d(x,y) > d2, then prob. over all h in H,
that h(x) = h(y) is at most p2.
23
LS Families: Illustration
High
probability;
at least p1
Low
probability;
at most p2
???
d1
d2
24
Example: LS Family
Let S = sets, d = Jaccard distance, H is
formed from the minhash functions for
all permutations.
Then Prob[h(x)=h(y)] = 1-d(x,y).
Restates theorem about Jaccard similarity
and minhashing in terms of Jaccard
distance.
25
Example: LS Family – (2)
Claim: H is a (1/3, 2/3, 2/3, 1/3)sensitive family for S and d.
If distance < 1/3
(so similarity > 2/3)
Then probability
that minhash values
agree is > 2/3
26
Comments
1. For Jaccard similarity, minhashing
gives us a (d1,d2,(1-d1),(1-d2))sensitive family for any d1 < d2.
2. The theory leaves unknown what
happens to pairs that are at distance
between d1 and d2.
Consequence: no guarantees about
fraction of false positives in that range.
27
Amplifying a LS-Family
The “bands” technique we learned for
signature matrices carries over to this
more general setting.
Goal: the “S-curve” effect seen there.
AND construction like “rows in a band.”
OR construction like “many bands.”
28
AND of Hash Functions
Given family H, construct family H’
consisting of r functions from H.
For h = [h1,…,hr] in H’, h(x)=h(y) if and
only if hi(x)=hi(y) for all i.
Theorem: If H is (d1,d2,p1,p2)-sensitive,
then H’ is (d1,d2,(p1)r,(p2)r)-sensitive.
Proof: Use fact that hi ’s are independent.
29
OR of Hash Functions
Given family H, construct family H’
consisting of b functions from H.
For h = [h1,…,hb] in H’, h(x)=h(y) if and
only if hi(x)=hi(y) for some i.
Theorem: If H is (d1,d2,p1,p2)-sensitive,
then H’ is (d1,d2,1-(1-p1)b,1-(1-p2)b)sensitive.
30
Effect of AND and OR Constructions
AND makes all probabilities shrink, but
by choosing r correctly, we can make
the lower probability approach 0 while
the higher does not.
OR makes all probabilities grow, but by
choosing b correctly, we can make the
upper probability approach 1 while the
lower does not.
31
Composing Constructions
As for the signature matrix, we can use
the AND construction followed by the
OR construction.
Or vice-versa.
Or any sequence of AND’s and OR’s
alternating.
32
AND-OR Composition
Each of the two probabilities p is
transformed into 1-(1-pr)b.
The “S-curve” studied before.
Example: Take H and construct H’ by
the AND construction with r = 4. Then,
from H’, construct H’’ by the OR
construction with b = 4.
33
Table for Function
p
.2
.3
.4
.5
.6
.7
.8
.9
1-(1-p4)4
.0064
.0320
.0985
.2275
.4260
.6666
.8785
.9860
4
4
1-(1-p )
Example: Transforms a
(.2,.8,.8,.2)-sensitive
family into a
(.2,.8,.8785,.0064)sensitive family.
34
OR-AND Composition
Each of the two probabilities p is
transformed into (1-(1-p)b)r.
The same S-curve, mirrored horizontally
and vertically.
Example: Take H and construct H’ by
the OR construction with b = 4. Then,
from H’, construct H’’ by the AND
construction with r = 4.
35
Table for Function
p
.1
.2
.3
.4
.5
.6
.7
.8
(1-(1-p)4)4
.0140
.1215
.3334
.5740
.7725
.9015
.9680
.9936
4
4
(1-(1-p) )
Example:Transforms a
(.2,.8,.8,.2)-sensitive
family into a
(.2,.8,.9936,.1215)sensitive family.
36
Cascading Constructions
Example: Apply the (4,4) OR-AND
construction followed by the (4,4) ANDOR construction.
Transforms a (.2,.8,.8,.2)-sensitive
family into a (.2,.8,.9999996,.0008715)sensitive family.
Note this family uses 256 of the original
hash functions.
37
General Use of S-Curves
For each S-curve 1-(1-pr)b, there is a
threshhold t, for which 1-(1-tr)b = t.
Above t, high probabilities are
increased; below t, they are decreased.
You improve the sensitivity as long as
the low probability is less than t, and
the high probability is greater than t.
Iterate as you like.
38
Use of S-Curves – (2)
Thus, we can pick any two distances x < y,
start with a (x, y, (1-x), (1-y))-sensitive
family, and apply constructions to produce
a (x, y, p, q)-sensitive family, where p is
almost 1 and q is almost 0.
The closer to 0 and 1 we get, the more
hash functions must be used.
39
LSH for Cosine Distance
For cosine distance, there is a technique
analogous to minhashing for generating
a (d1,d2,(1-d1/180),(1-d2/180))- sensitive
family for any d1 and d2.
Called random hyperplanes.
40
Random Hyperplanes
Pick a random vector v, which
determines a hash function hv with two
buckets.
hv(x) = +1 if v.x > 0; = -1 if v.x < 0.
LS-family H = set of all functions
derived from any vector.
Claim: Prob[h(x)=h(y)] = 1 – (angle
between x and y divided by 180).
41
Proof of Claim
Look in the
plane of x
and y.
v
x
θ
Hyperplanes
for which
h(x) = h(y)
Hyperplanes
(normal to v )
for which h(x)
<> h(y)
y
Prob[Red case]
= θ/180
42
Signatures for Cosine Distance
Pick some number of vectors, and hash
your data for each vector.
The result is a signature (sketch ) of +1’s
and –1’s that can be used for LSH like the
minhash signatures for Jaccard distance.
But you don’t have to think this way.
The existence of the LS-family is
sufficient for amplification by AND/OR.
43
Simplification
We need not pick from among all
possible vectors v to form a component
of a sketch.
It suffices to consider only vectors v
consisting of +1 and –1 components.
44
LSH for Euclidean Distance
Simple idea: hash functions correspond
to lines.
Partition the line into buckets of size a.
Hash each point to the bucket
containing its projection onto the line.
Nearby points are always close; distant
points are rarely in same bucket.
45
Projection of Points
If d >> a, θ must
be close to 90o
for there to be
any chance points
go to the same
bucket.
Points at
distance d
θ
d cos θ
Bucket
width a
If d << a, then
the chance the
points are in the
same bucket is
at least 1 – d /a.
Randomly
chosen
line
46
An LS-Family for Euclidean Distance
If points are distance > 2a apart, then
60 < θ < 90 for there to be a chance
that the points go in the same bucket.
I.e., at most 1/3 probability.
If points are distance < a/2, then there
is at least ½ chance they share a bucket.
Yields a (a/2, 2a, 1/2, 1/3)-sensitive
family of hash functions.
47
Fixup: Euclidean Distance
For previous distance measures, we
could start with an (x, y, p, q)-sensitive
family for any x < y, and drive p and q
to 1 and 0 by AND/OR constructions.
Here, we seem to need y > 4x.
48
Fixup – (2)
But as long as x < y, the probability of
points at distance x falling in the same
bucket is greater than the probability of
points at distance y doing so.
Thus, the hash family formed by
projecting onto lines is an (x, y, p, q)sensitive family for some p > q.
Then, amplify by AND/OR constructions.
49