Intelligent Text Processing

lecture 3

Word distribution laws.

Word-based indexing

Szymon Grabowski

[email protected]

http://szgrabowski.kis.p.lodz.pl/IPT08/ Łódź, 2008

Zipf’s law (Zipf, 1935, 1949) [ http://en.wikipedia.org/wiki/Zipf's_law, http://ciir.cs.umass.edu/cmpsci646/Slides/ir08%20compression.pdf ] word-rank  word-freq  constant That is, a few most frequent words cover a relatively large part of the text, while the majority of words in the given text’s vocabulary occur only once or twice.

More formally, the frequency of any word is (approximately) inversely proportional to its rank in the frequency table. Zipf’s law is empirical!

Example from the Brown Corpus (slightly over 10 6 words): “the” is the most freq. word with ~7% (69971) of all word occs. The next word, “of”, has ~3.5% occs (36411), followed by “and” with less than 3% occs (28852). Only 135 items are needed to account for half the Brown Corpus.

2

Does Wikipedia confirm Zipf’s law?

[ http://en.wikipedia.org/wiki/Zipf's_law ] Word freq in Wikipedia, Nov 2006, log-log plot. x is a word rank, y is the total # of occs.

Zipf's law roughly corresponds to the green (1/x) line.

3

Let’s check it in Python...

Dickens collection: http://sun.aei.polsl.pl/~sdeor/stat.php?s=Corpus_dickens&u=corpus/dickens.bz2

distinct words: 28347 5 top freq words: [('the', 80103), ('and', 63971), ('to', 47413), ('of', 46825), ('a', 37182)] 4

Lots of words with only a few occurrences, (

dickens

example continued) there are 9423 words with freq 1 there are 3735 words with freq 2 there are 2309 words with freq 3 there are 1518 words with freq 4 5

Brown corpus statistics [ http://ciir.cs.umass.edu/cmpsci646/Slides/ir08%20compression.pdf ] 6

Heaps’ law (Heaps, 1978) [ http://en.wikipedia.org/wiki/Heaps'_law ] Another empirical law. It tells how vocabulary size

V

grows with growing text size

n

(expressed in words):

V

(

n

) =

K · n β ,

where

K

is typically around 10..100 (for English) and

β

between 0.4 and 0.6.

Roughly speaking, the vocabulary (# of distinct words) grows proportially to the square root of the text length .

7

Musings on Heaps’ law The number of ‘words’ grows without a limit...

How is it possible?

Because in new documents new words tend to occur: e.g. (human, geographical etc.) names. But also typos!

On the other hand, the dictionary size grows significantly slower than the text itself.

I.e. it doesn’t pay much to represent the dictionary succinctly (with compression) – dictionary compression ideas which slow down the access should be avoided.

8

Inverted indexes Almost all real-world text indexes are inverted indexes (in one form or another).

An inverted index (Salton and McGill, 1983) stores words and their occurrence lists in the text database/collection.

The occurrence lists may store exact word positions ( inverted list ) or just a block or a document ( inverted file ).

Storing exact word positions (inverted list) enables for faster search and facilitates some kinds of queries (compared to the inverted file), but requires (much) more storage.

9

Inverted file, example [ http://en.wikipedia.org/wiki/Inverted_index ] We have three texts (documents): T0 = "it is what it is" T1 = "what is it" T2 = "it is a banana" We built the vocabulary and the associated document index lists: Let’s search for

what

,

is

and

it

.

That is, we want to obtain the references to all the documents which contain those three words (at least once each, and in arbitrary positions!).

The answer: 10

How to build an inverted index [ http://nlp.stanford.edu/IR-book/pdf/02voc.pdf ] Step 1 – obvious.

Step 2 – split the text into tokens (roughly: words).

Step 3 – reduce the amount of unique tokens (e.g. eliminate capitalized words, plural nouns).

Step 4 – build the dictionary structure and occurrence lists.

11

Tokenization [ http://nlp.stanford.edu/IR-book/pdf/02voc.pdf ]

A token is an instance of a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing.

How to tokenize? Not so easy as it first seems...

See the following sent. and possible tokenizations of some excerpts from it.

12

Tokenization, cont’d [ http://nlp.stanford.edu/IR-book/pdf/02voc.pdf ] Should we accept tokens like

C++

,

C#

,

.NET

,

DC-9

,

M*A*S*H

?

And if

M*A*S*H

is ‘legal’ (single token), then why not

2*3*5

?

Is a dot (.) a punctuation mark (sentence terminator)?

Usually yes (of course), but think about emails, URLs and IPs...

13

Tokenization, cont’d [ http://nlp.stanford.edu/IR-book/pdf/02voc.pdf ] Is a space always a reasonable token delimiter?

Is it better to perceive

New York

(or

San Francisco

,

Tomaszów Mazowiecki

...) as one token or two?

Similarly with foreign phrases (

cafe latte

,

en face

).

Splitting on spaces may bring bad retrieval results: search for

York University

will mainly fetch documents related to

New York University

.

14

Stop words [ http://nlp.stanford.edu/IR-book/pdf/02voc.pdf ] Stop words are very frequent words which carry little information (e.g. pronouns, articles). For example: Consider an inverted list (i.e. exact positions for all word occurrences are kept in the index).

If we have the stop list (i.e. discard them during indexing), the index will get smaller, say by 20 –30% (note it has to do with Zipf’s law).

Danger of using stop words: some meaningful queries may consist of stop words exclusively:

The Who

,

to be or not to be

...

The modern trend is not to use a stop list at all.

15

Glimpse – a classic inverted file index [ Manber & Wu, 1993 ] Assumption: the text to index is a single unstructure file (perhaps a concatenation of documents).

It is divided into 256 blocks of approximately the same size.

Each entry in the index is a word and the list of blocks in which it occurs, in ascending order. Each block number takes 1 byte (why?).

16

Block-addressing (e.g. Glimpse) index, general scheme [ fig. 3 from Navarro et al., “Adding compression...”, IR, 2000 ] 17

How to search in Glimpse?

Two basic queries: • keyword search – the user specifies one or more words and requires a list of all documents (or blocks) in which those words occur in any positions , • phrase search – the user specifies two or more words and requires a list of all documents (or blocks) in which those words occur as a phrase, i.e. one-by-one.

Why phrase search is important?

Imagine keyword search for +new +Scotland and phrase search for “new Scotland”. What’s the difference?

18

How to search in Glimpse, cont’d (fig. from http://nlp.stanford.edu/IR-book/pdf/01bool.pdf ) The key operation is to intersect several block lists .

Imagine the query has 4 words, and their corresponding lists have length: 10, 120, 5, 30. How do you perform the intersection?

It’s best to start with two shortest lists, i.e. of length 5 and 10 in our example. The intersection output will have length at most 5 (but usually less, even 0, when we just stop!).

Then we intersect the obtained list with the list of length 30, and finally with the longest list.

No matter the intersection order, the same result but (vastly) different speeds!

19

How to search in Glimpse, cont’d We have obtained the intersection of all lists; what then?

Depends on the query: for the keyword query, we’re done (can retrieve now the found blocks / docs).

For the phrase query , we have yet to scan the resulting blocks / documents and check if and where the phrase occurs in them.

To this end, we can use any fast exact string matching alg, e.g. Boyer –Moore–Horspool.

Conclusion: the smaller the resulting list, the faster the phrase query is handled.

20

Approximate matching with Glimpse Imagine we want to find occurrences of a given phrase but with up to

k

(Levenshtein) errors. How to do it?

Assume for example

k

= 2 and the phrase

grey cat

.

The phrase has two words, so there are the following error per word possibilities: 0 + 0, 0 + 1, 0 + 2, 1 + 0, 1 + 1, 2 + 0.

21

Approximate matching with Glimpse, cont’d E.g. 0 + 2 means here that the first word (

grey

) must be exactly matched, but the second with 2 errors (e.g.

rats

).

So, the approximate query

grey cat

translates to many exact queries (many of them rather silly...):

grey cat

,

gray cat

,

grey rat

,

great cat

,

grey colt

,

gore cat

,

grey date

... All those possibilities are obtained from traversing the vocabulary structure (e.g., a trie). Another option is on-line approx matching over the vocabulary represented as plain text (concatenation of words) – see fig. at the next slide.

22

Approx matching with Glimpse, query example with a single word

x

(fig. from Baeza-Yates & Navarro,

Fast Approximate String Matching in a Dictionary

, SPIRE’98) 23

Block boundaries problem If the Glimpse blocks never cross document boundaries (i.e., are natural), we don’t have this problem...

But if the block boundaries are artificial , then we may be unlucky and have one of our keywords at the very end of a block B

j

and the next keyword at the beginning of B

j

+1 . How not to miss an occurrence ?

There is a simple solution : block may overlap a little. E.g. the last 30 words of each block are repeated at the beginning of the next block. Assuming the phrase length / keyword set has no more than 30 words, we are safe.

But we may then need to scan more blocks than necessary (why?).

24

Glimpse issues and limitations The authors claim their index takes only 2 –4 % of the original text size.

But it can work to text collections to about 200 MB only; then it starts to degenerate, i.e., the block lists tend to be long and many queries and handled not much faster than using online seach (without any index).  How can we help it?

Overall idea is fine, so we must take care of details.

One major idea is to apply compression to the index...

25

(Glimpse-like) index compression The purpose of data compression in inverted indexes is not only to save space (storage).

It is also to make queries faster!

(One big reason is less I/O, e.g. one disk access where without compression we’d need two accesses.

Another reason is more cache-friendly memory access.) 26

Compression opportunities (Navarro et al., 2000) • Text (partitioned into blocks) may be compressed on word level (faster text search in the last stage).

• Long lists may be represented as their complements, i.e. the numbers of the block in which a given word does NOT occur.

• Lists store increasing numbers, so the gaps (differences) between them may be encoded, e.g.

2, 15, 23, 24, 100  2, 13, 8, 1, 76 (smaller numbers).

• The resulting gaps may be statistically encoded (e.g. with some Elias code; next slides...).

27

Compression paradigm: modeling + coding Modeling: the way we look at input data. They can be perceived as individual (context-free) 1-byte characters, or pairs of bytes, or triples etc. We can look for matching sequences in the past buffer (bounded or unbounded, sliding or not), the minimum match length can be set to some value, etc. etc. We can apply lossy or lossless transforms (DCT in JPEG, Burrows –Wheeler transform), etc.

Modeling is difficult.

Sometimes more art than science. Often data-specific.

Coding : what we do with the data transformed in the modeling phase.

28

Intro to coding theory A uniquely decodable code : if any concatenation of its codewords can be uniquely parsed.

A prefix code : no codeword is a proper prefix of any other codeword. Also called an instantaneous code .

Trivially, any prefix code is uniquely decodable.

(But not vice versa!) 29

Average codeword length

Let an alphabet have

s

symbols, with probabilities

p

0

,

p

1

, ...,

p s

–1

.

Let’s have a (uniquely decodable) code

C

= [

c

0

,

c

1

, ...,

c s

–1

].

The avg codeword length for a given probability distribution is defined as

So this is a weighted average length over individual codewords. More frequent symbols have a stronger influence.

30

Entropy Entropy is the average information in a symbol.

Or: the lower bound on the average number (may be fractional) of bits needed to encode an input symbol. Higher entropy = less compressible data.

What is “the entropy of data” is a vague issue.

We measure the entropy always according to a given model (e.g. context-free aka order-0 statistics).

(

S

Shannon’s entropy formula – the “source” emitting “messages” / symbols) 31

Redundancy Simply speaking, redudancy is the excess in the representation of data.

Redundant

data means: compressible data. A redundant code is a non-optimal (or: far from optimal) code.

A code redundancy (for a given prob. distribution):

L

(

C

,

p

0

,

p

1

R

(

C

,

p

0

,

p

1

, ...,

p s

–1 ) =

, ...,

p s

–1

)

–

H

(

p

0

,

p

1

, ...,

p s

–1 ) 

0.

The redundancy of a code is the avg excess (over the entropy) per symbol.

Can’t be below 0, of course.

32

Basic codes. Unary code Extremely simple though. Application: very skew distribution (expected for a given problem).

33

Basic codes. Elias gamma code Still simple, and usually much better than the unary code.

34

Elias gamma code in Glimpse (example) an examplary occurrence list: [2, 4, 5, 6, 9, 11, 40, 42, 43, 94, 96, 120, 133, 134, 151, 203] list of deltas (differences): [2, 2, 1, 1, 3, 2, 29, 2, 1, 51, 2, 24, 13, 1, 17, 52] list of deltas minus 1 (as zero was previously impossible, except perhaps on the 1st position): [2, 1, 0, 0, 2, 1, 28, 1, 0, 50, 1, 23, 12, 0, 16, 51] no compression (one item – one byte): 16 * 8 = 128 bits with Elias coding: 78 bits 101 100 0 0 101 100 111101101 100 0 11111010011 100 111101000 1110101 0 111100001 11111010100 35

Python code for the previous example occ_list = [2, 4, 5, 6, 9, 11, 40, 42, 43, 94, 96, 120, 133, 134, 151, 203] import math delta_list = [occ_list[0]] for i in range(1, len(occ_list)): delta_list += [occ_list[i]-occ_list[i-1]-1] print occ_list print delta_list total_len = 0 total_seq = "" for x in delta_list: z = int(math.log(x+1, 2)) code1 = "1" * z + "0" code2 = bin(x+1-2**z)[2:].zfill(z) if z >= 1 else "" total_seq += code1 + code2 + " " total_len += len(code1 + code2) print "no compression:", len(occ_list)*8, "bits" print "with Elias gamma coding:", total_len, "bits" print total_seq v2.6 needed as the function bin() is used 36

Huffman coding (1952) – basic facts Elias codes assume that we know (roughly) the symbol distribution before encoding. What if we guessed it badly...?

If we know the symbol distribution, we may construct an optimal code, more precisely, optimal among the uniquely decodable codes having a codebook.

It is called Huffman coding.

Example. Symbol frequencies above, final Huffman tree on the right. 37

Huffman coding (1952) – basic facts, cont’d Its redundancy is always less than 1 bit / symbol (but may be arbitrarily close).

In most practical applications (data not very skew) Huffman code avg length only 1-3% worse than entropy.

E.g. the order-0 entropy of book1 (English 19th century novel, plain text) from the Calgary corpus: 4.567 bpc.

Huffman avg length: 4.569 bpc.

38

Why not using only Huffman for encoding occurrence lists?

39