Transcript Chinese WordSketch Online, corpus-based summaries of word usage

Chinese
WordSketch
Online, corpus-based
summaries of word usage
1
Participants
•
•
•
•
•
•
Adam Kilgarriff, Lexical Computing, UK
David Tugwell, Tech University Budapest
Pavel Rychly, Brno University
Simon Smith, 銘傳大學 (中研院)
黃居仁, 中研院
巫宜靜, 清華大學 (中研院)
2
Facing the problem: lexical choice
• “You shall know a word by the company it keeps”
(Firth, 1957)
• The meaning of face depends on the collocation
(詞語搭配)
– 學漢語的外國人要面對詞語選擇的問題
– 許多種動物正在面臨絕種
• Similarly with save
– Save money
– Save life
– Save a seat for me
3
Look in a dictionary? A corpus?
• Some modern English dictionaries give
some collocation (詞語搭配) information
– Chinese dictionaries give very limited help
• Since the 1980s, corpus KWIC (KeyWord
In Context) concordances have been
available
4
Pre-computer
corpus!
Oxford English
Dictionary:
• 20 million
index cards
5
KWIC Concordance
arity, which will be used to take a
from outside. You are invited to a
tion, we believe politicians of all
ould be reaching agreement with all
lack people. I have certainly been
. These should be discussed by both
presents They had hosted a cocktail
akes. By midnight the end-of-course
e should be a right for the injured
by the Safran Peoples ' Liberation
s. Ahead I could see the rest of my
cial ethic. The two main political
ritish successes in Perth The small
to help control. One member of the
rket society fashion magazine. The
security and secrecy than any Tory
party of under-privileged childr
party and after a couple of drink
parties will listen to our views.
parties concerned, as to which
party to one or two discussions a
parties before entering into the
party at Kensington palace, for
party is in full swing, but mos
party to terminate the contract.
Party. This presents the powerfu
party plodding towards the final
parties - the Tories and the Libe
party of British players competin
party went to summon the rescue t
party was held at his flat which
Party Conference : it seems that
6
The coloured pens method
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
arity, which will be used to take a
from outside. You are invited to a
tion, we believe politicians of all
ould be reaching agreement with all
lack people. I have certainly been
. These should be discussed by both
presents They had hosted a cocktail
akes. By midnight the end-of-course
e should be a right for the injured
by the Safran Peoples ' Liberation
s. Ahead I could see the rest of my
cial ethic. The two main political
ritish successes in Perth The small
to help control. One member of the
rket society fashion magazine. The
security and secrecy than any Tory
1 political association
2 social event
3 group of people
party of under-privileged children to D
party and after a couple of drinks you d
parties will listen to our views. &equo
parties concerned, as to which events,
party to one or two discussions amongst
parties before entering into the relatio
party at Kensington palace, for example
party is in full swing, but most cadet
party to terminate the contract. A mana
Party. This presents the powerful neigh
party plodding towards the final slope t
parties - the Tories and the Liberals party of British players competing in th
party went to summon the rescue team and
party was held at his flat which was a l
Party Conference : it seems that bootleg
4 person in an agreement/dispute
5 to be party to something...
7
Limitation of KWIC analysis
• As corpora get bigger: too much data
– 50 lines for a word: read all
– 500 lines: could read all, takes a long time
– 5000 lines: no
• Instead, create a statistical summary of
word usage
– Show most salient 最有顯著性 collocates
(Mutual Information)
8
Mutual Information
• Church and Hanks 1989
• MI: How much more often does a word
pair occur, than one might expect by
chance:
I ( x; y)  log
P( x | y )
P( x)
9
Collocation listing
For right collocates of save (>5 hits)
word
f(x+y)
f(y)
word
f(x+y)
f(y)
forests
6
170
life
36
4875
$1.2
6
180
dollars
8
1668
lives
37
1697
costs
7
1719
enormous
6
301
thousands
6
1481
annually
7
447
face
9
2590
jobs
20
2001
estimated
6
2387
money
64
6776
your
7
3141
10
Limitations of collocation listing
• Some items are not genuine collocates
– yours appears only because it is adjacent to save
• The collocates can belong to any part of speech
– It would better if they were classified into POS
– and the role they play in the sentence
• Thus,
– for arrest in “The police were quick to arrest a number of
suspects on the spot”
• We would like to see
–
–
–
–
Keyword: arrest
Subject: police
Object: suspect(s)
Modifier: on the spot
11
Wordsketch
• Attempts to meet these requirements
• A corpus-derived one-page summary of a
word’s grammatical and collocational
behaviour
• Implemented for English and Czech
• Chinese and Irish implementations in
progress
12
The corpus: Chinese Gigaword
• A Linguistic Data Consortium corpus
– Very large: over 1 billion characters
– Compiled by David Graff & Ke Chen in 2003
– Minimally tagged
• 286 newswire stories, half from each of:
– CNA Taiwan (740 million traditional characters)
– Xinhua PRC (380 million simplified characters)
• Corpus was segmented and tagged using
Academia Sinica tools
13
http://corpora.fi.muni.cz/chinese/
•
•
•
•
•
•
逮捕
教
學習
銀行
捉
http://corpora.fi.muni.cz/chinese/
14
15
16
17
18
19
20
Functions
• KWIC concordance
– Sorting, filtering etc
• Word sketch
• Automatic thesaurus
• Sketch difference
– discriminate near-synonyms
• In development
– key words in a subcorpus / text type
– how word varies with text type
21
22
Grammar writing
• Uses CQL (Corpus query language)
– Christ and Schulze, U. Stuttgart, 1994
• defining an object:
v (adj|n|det|num|adv)* n
rewriting in CQL with BNC/CLAWS-5 tags
[tag="VV.*"] [tag="(A[JTV]|D|O).*"]* [tag="NN.*"]
23
Further work
• Improve grammatical relations, especially
sentence objects, to account for
– topicalization (啤酒,葡萄酒,他都愛喝)
– 把 fronting (請把啤酒喝完)
• Create “Dr Eye” style interface, to show
common collocations online, in a text
24
English version available
• For personal use
– www.sketchengine.co.uk
• 歡迎註冊及多善加利用!
25