Transcript スライド 1
Linguistically-motivated Tree-based Probabilistic Phrase Alignment Toshiaki Nakazawa, Sadao Kurohashi (Kyoto University) Outline Background Tree-based Probabilistic Phrase Alignment Model Model Training Symmetrization Algorithm Experiments Conclusions 2 7/18/2015 Background Many of state-of-the-art SMT systems are based on “word-based” alignment results Phrase-based SMT [Koehn et al., 2003] Hierarchical Phrase-based SMT [Chiang, 2005] and so on Some of them incorporate syntactic information “after” word-based alignment [Quirk et al., 2005], [Galley et al., 2006] and so on Is it enough? Is it able to achieve “practical” translation quality? 3 7/18/2015 Background (cont.) Word-based alignment model works well for structurally similar language pairs It is not effective for language pairs with great difference in linguistic structure such as Japanese and English SOV versus SVO For such language pair, syntactic information is necessary even during alignment process 4 7/18/2015 Related Work Syntactic tree-based model [Yamada and Knight, 2001], [Gildea, 2003], ITG by Wu Incorporating some operations which control subtrees (re-order, insert, delete, clone) to reproduce the opposite tree structure Our model does not require any operations Our model utilizes dependency trees Dependency tree-based model 5 [Cherry and Lin, 2003] Word-to-word, and one-to-one alignment Our model makes phrase-to-phrase alignment, and can make many-to-many links 7/18/2015 Features of Proposed Tree-based Probabilistic Phrase Alignment Model Generation model similar to IBM models Using phrase dependency structures “phrase” means a linguistic phrase (cf. phrase-based SMT) Phrase to phrase alignment model Each phrase (node) consists of basically 1 content word and 0 or more function words Source side content words can be aligned to content words of target side only (same for function words) Generation starts from the root node and end up with one of leaf nodes (cf. IBM model is from first word to last word) 6 7/18/2015 Outline Background Tree-based Probabilistic Phrase Alignment Model Model Training Symmetrization Algorithm Experiments Conclusions 7 7/18/2015 Dependency Analysis of Sentences Source プロピレングリコールは血中グルコース インスリンを上昇させ、血中NEFA濃度 を減少させる Target Propylene glycol increases in blood glucose and insulin and decreases in NEFA concentration in the blood Word order Root node Root node 8 Head node 7/18/2015 Head node IBM Model v.s Tree-based Model IBM Model [Brown et al., 93] f : source sentence e : target sentence a : alignment : parameters aˆ arg max p( f | e, a) p(a | e) a S ˆ arg max p( f s | es , a) p(a | es ) s 1 a Tree-based Model aˆ arg max p(T f | Te , a) p(a | Te ) Tf : source tree Te : target tree a S ˆ arg max p(T f ,s | Te,s , a) p(a | Te,s ) 9 s 1 a 7/18/2015 Model Decomposition: Lexicon Probability p(Tf | Te , a) aˆ arg max p(T f | Te , a) p(a | Te ) a Suppose T f consists of J nodes and Te consists of I nodes Phrase J p(T f | Te , a) p( f j | ea j ) translation probability j 1 p( f j | ea j ) is calculated as a product of two probabilities p( f j | ea j ) pcont. ( f j | ea j ) p func. ( f j | ea j ) Ex) 濃度 を - in concentration pcont. (濃度| concentration) p func. (を| in) 上昇 さ せ - increase pcont. (上昇| increase) p func. (さ せ| EMPTY) 10 7/18/2015 Model Decomposition: Alignment Probability p(a | Te ) aˆ arg max p(T f | Te , a) p(a | Te ) a Define the parent node of f j as f j p(a | Te ) is decomposed as a product of target Dependency side dependency relation probability relation conditioned on source side relation probability J p(a | Te ) p(rel (ea j , eaj ) | rel ( f j , f j)) j 1 If the parent node has been aligned to NULL, f j indicates the grandparent of f j , and this continues until f j has been aligned to other than NULL p(a | Te ) models a tree-based reordering 11 7/18/2015 Outline Background Tree-based Probabilistic Phrase Alignment Model Model Training Symmetrization Algorithm Experiments Conclusions 12 7/18/2015 Model Training The proposed model is trained by EM algorithm First, phrase translation probability is learned (Model 1) Model 1 can be efficiently learned without approximation (cf. IBM model 1 and 2) Next, dependency relation probability is learned (Model 2) with probabilities learned in Model 1 as initial parameters 13 Model 2 needs some approximation (cf. IBM model 3 or greater), we use beam-search algorithm 7/18/2015 Model 1 Each phrase in source side f j (1 j J ) can correspond to an arbitrary phrase in target side ei (1 i I ) or NULL phrase (e0 ) a A probability of one possible alignment is: J p(a, T f | Te ) pcont. ( f j | ea j ) p func. ( f j | ea j ) j 1 Then, tree translation probability is: p(T f | Te ) p(a, T f | Te ) a Efficiently calculated as: J p( f a 14 j 1 J j I | ea j ) p( f j | ea j ) j 1 i 0 7/18/2015 Model 2 (imaginary ROOT node) Root node of a sentence is supposed to depend on the imaginary ROOT node, which works as a Start-Of-Sentence (SOS) in word-based model The ROOT node in source tree always corresponds to that of target tree 事例 を 通して 援助 の 視点 に 必要な ポイント を 確認 した ROOT 15 necessary the point through the case in the viewpoint of the assist was confirmed ROOT 7/18/2015 Model 2 (beam-search algorithm) It is impossible to enumerate all the possible alignment Consider only a subset of “good-looking” alignments using beam-search algorithm Ex) beam-width = 4 事例 を 通して 援助 の 視点 に 必要な ポイント を 確認 した necessary the point through the case in the viewpoint of the assist was confirmed NULL 16 7/18/2015 Model 2 (beam-search algorithm) 事例 を 通して 援助 の necessary 事例 を 通して 援助 の the point 視点 に necessary the point 視点 に through the case through the case 必要な in the viewpoint 必要な in the viewpoint ポイント を of the assist ポイント を of the assist 確認 した was confirmed 確認 した was confirmed NULL 事例 を 通して 援助 の NULL necessary 事例 を 通して 援助 の the point 視点 に necessary the point 視点 に through the case through the case 必要な in the viewpoint 必要な in the viewpoint ポイント を of the assist ポイント を of the assist 確認 した was confirmed NULL 17 確認 した was confirmed NULL 7/18/2015 Model 2 (parameter notations) Dependency relation rel ( P1 , P2 ) between two phrases P1 and P2 is defined as a path from P2 to P1 using the following notations: P c “c-” if P1 is a pre-child of P2 1 P2 “c+” if P1 is a post-child of P2 “p-” if P2 is a post-child of P1 P2 P1 c+ p- P1 “p+” if P2 is a pre-child of P1 P2 P2 p+ P1 “INCL” if P1 and P2 are same phrase “ROOT” if P2 is an imaginary ROOT node 18 “NULL” if P1 is aligned to NULL P1 ROOT P2 ROOT 7/18/2015 Model 2 (parameter notations, cont.) In a case where P1 and P2 are two or more nodes distant from each other, the relation is described by combining the notations Ex) p- P2 c- P1 c- P1 c+ c+ c-;c+ 19 P2 p-;c+;c- 7/18/2015 Dependency Relation Probability Examples 事例 を 通して 援助 の necessary 事例 を 通して 援助 の the point 視点 に necessary the point 視点 に through the case through the case 必要な in the viewpoint 必要な in the viewpoint ポイント を of the assist ポイント を of the assist 確認 した was confirmed 確認 した was confirmed NULL NULL p(ROOT | ROOT ) p(c- | c-) 事例 を 通して 援助 の necessary p(ROOT ;c- | ROOT ) p(p-| c-) 事例 を 通して 援助 の the point 視点 に necessary the point 視点 に through the case through the case 必要な in the viewpoint 必要な in the viewpoint ポイント を of the assist ポイント を of the assist 確認 した was confirmed NULL p(ROOT | ROOT ) p(c-;c | c-) 20 確認 した was confirmed NULL p(NULL| ROOT ) p(ROOT | ROOT ;c-) 7/18/2015 Example 事例 を 通して 援助 の 視点 に 必要な ポイント を 確認 した ROOT necessary the point through the case in the viewpoint of the assist was confirmed ROOT p(a, Tf | Te ) p(確認| was confirmed) p(した | EMPT Y) p(ROOT| ROOT ) p(事例| case) p(を 通して | through) p(c-;c | c-) p(ポイント | point) p(を | EMPT Y) p(c- | c-) p(必要な | necessary) p(EMPT Y| EMPT Y) p(c- | c-) p(視点 | viewpoint) p(に | in) p(c | c-) p(援助| assist) p(の | of) p(c | c-) 21 7/18/2015 Outline Background Tree-based Probabilistic Phrase Alignment Model Model Training Symmetrization Algorithm Experiments Conclusions 22 7/18/2015 Symmetrization Algorithm Since our model is directed, we run the model bi-directionally and symmetrize two alignment results heuristically Symmetrization algorithm is similar to [Koehn et al. 2003], which uses 1-best GIZA++ word alignment result of each direction Our algorithm exploits n-best alignment results of each direction Three steps: 23 Superimposition Growing Handling isolations 7/18/2015 Symmetrization Algorithm 1. Superimposition Source to Target 5-best 5 2 10 Target to Source 5-best 10 10 5 3 5 1 5 7 3 10 5 9 7 7 1 ・ ・ ・ 24 ・ ・ ・ 7/18/2015 Symmetrization Algorithm 1. Superimposition (cont.) Definitive alignment points are adopted The points which don’t have same or higher scored point in their same row or column Conflicting points are discarded 25 The points which is in the same row or column of the adopted point and is not contiguous to the adopted point on tree 5 2 10 10 10 5 3 5 1 5 7 3 10 5 7 7 1 7/18/2015 9 Symmetrization Algorithm 2. Growing Adopt contiguous points to adopted points in both source and target tree In descending order of the score From top to bottom From left to right Discard conflicting points 26 5 2 10 10 10 3 5 7 10 5 7 7 The points which have adopted point both in the same row and column 7/18/2015 9 Symmetrization Algorithm 3. Handling Isolation Adopt points which are not aligned to any phrase in both source and target language 5 10 10 10 3 5 7 10 9 7 7 27 7/18/2015 Alignment Experiment Training corpus Gold-standard alignment Manually annotated 100 sentence pairs among the training corpus Sure (S) alignment only [Och and Ney, 2003] Evaluation unit Japanese-English paper abstract corpus provided by JST which consists of about 1M parallel sentences Morpheme-based for Japanese Word-based for English Iterations 28 5 iterations for Model 1, and 5 iterations for Model 2 7/18/2015 Alignment Experiment (cont.) Comparative experiment (word-base alignment) GIZA++ and various symmetrization heuristics [Koehn et al., 2007] Default settings for GIZA++ Use original forms of words for both Japanese and English 29 7/18/2015 Results proposed GIZA++ 30 Precision Recall F-measure 1-best-intersection 90.92 41.69 57.17 1-best-grow 83.30 54.33 65.76 3-best-grow 81.21 56.52 66.65 5-best-grow 80.59 57.33 67.00 intersection 88.14 40.18 55.20 grow 83.50 49.65 62.27 grow-final 67.19 56.91 61.63 grow-final-and 78.00 52.93 63.06 grow-diag 77.34 53.18 63.03 grow-diag-final 67.24 56.63 61.48 grow-diag-final-and 74.95 54.26 62.95 7/18/2015 Example of Alignment Improvement Proposed model 31 Word-base alignment 7/18/2015 Translation Experiments Training corpus Test corpus 500 paper abstract sentences Decoder Same to alignment experiments Moses [Koehn et al., 2007] Use default options except for phrase table limit (20 -> 10) and distortion limit (6 -> -1) No minimum error rate training Evaluation 33 BLEU No punctuations and case-insensitive 7/18/2015 Results proposed GIZA++ Rec F BLEU 1-best-intersection 90.92 41.69 57.17 12.73 5-best-grow 80.59 57.33 67.00 15.40 intersection 88.14 40.18 55.20 16.35 grow-diag 77.34 53.18 63.03 17.89 grow-diag-final-and 74.95 54.26 62.95 17.76 Definition of function words is improper Pre Articles? Auxiliary verbs? … Tree-based decoder is necessary BLEU is essentially insensitive to syntactic structure 34 Translation quality potentially improved 7/18/2015 Potentially Improved Example Input: これ は ＬＢ 膜 の 厚み が アビジン を 吸着 する こと で 増 加 した こと に よる 。 Proposed (30.13): this is due to the increase in the thickness of the lb film avidin adsorb GIZA++ (33.78): the thickness of the lb film avidin to adsorption increased by it Reference: this was due to increased thickness of the lb film by adsorbing avidin 35 7/18/2015 Conclusion Tree-based probabilistic phrase alignment model using dependency tree structures N-best symmetrization algorithm Achieve high alignment accuracy compared to word-based models Phrase translation probability Dependency relation probability Syntactic information is useful during alignment process BUT: Unable to improve the BLEU scores of translation 36 7/18/2015 Future Work More flexible model Integrate parsing probabilities into the model Content words sometimes correspond to function words and vice versa Parsing errors easily lead to alignment errors By integrating parsing probabilities, parsing results and alignment can be revised complementary More syntactical information 37 Use POS or phrase category into the model 7/18/2015 Thank You! 38 7/18/2015