Transcript Document

Error Analysis of Rule-based Machine Translation Outputs
A Case Study on English – Persian MT System
‫تجزیه و تحلیل خطا از حکومت مبتنی بر خروجی دستگاه ترجمه؟‬
MT ‫ ف ارسی سیستم های‬- ‫مطالعه موردی در زبان انگلیسی‬
Zahra Pourniksefat
Islamic Azad University – Science & Research Branch
Agenda
Introduction
 Machine Translation Overview
 Evaluation of MT systems
Methods & Materials
Error Categories & Description
Results & Discussion
Machine Translation Overview
Definition: The term Machine Translation (MT) is used for translating text or
speech from one natural language to another by using computers and software.
• Systran: MT is much faster than human translators because it is much cheaper
and has a better memory than human translators.
• Shahba 2002 believed that “It’s better to spend our time on the actual act of
translation rather than typing the English text or scanning it for the MT to
translate. Efforts in MT are by themselves valuable as they at least satisfy one of
the needs of human beings: need for innovation and discovery”
•
MT is more economic on time and money, but it is less accurate than human
translators (Frederking, 2004).
Why MT matters?
According to Hatim and Munday it’s an important topic - socially, politically, commercially,
scientifically, and intellectually or philosophically (2004)
•
The social or political importance of MT arises from the socio- political importance of translation in
communities where more than one language is generally spoken. So translation is necessary for
communication- for ordinary human interaction, and for gathering the information one needs to play
a full part in society.
•
The commercial importance of MT is a result of related factors. First, translation itself is
commercially important. Second, translation is expensive.
•
Scientifically, MT is interesting, because it is an obvious application and testing ground for many
ideas in Computer Science, Artificial Intelligence, and Linguistics.
•
Philosophically , MT is interesting, because it represents an attempt to automate an activity that can
require the full range of human knowledge.
Some Misconceptions about MT
MT is a waste of time because you will never make a machine that can translate
Shakespeare. This criticism that MT systems cannot translate Shakespeare is a bit like
the criticism of industrial robots for not being able to dance.(Hatim and Munday, 2004)
•
First, translating literature requires special literary skills – it is not the kind of
thing that the average professional translators normally attempt
•
Second, literary translation is a small proportion of the translation that has to be
done.
•
Finally, one may wonder who would ever want to translate Shakespeare by
machine – it’s a job that human translators find challenging and rewarding, and
it’s not a job that MT systems have been designed for.
Approaches to MT
•
Direct Machine Translation Approach
The first developed MT systems where a word–for–word translation from the source language to the target
language is performed.
•
Transfer Machine Translation Approach
1.
The analysis stage that is the direct strategy which takes benefits of a dictionary in source language to
demonstrate the source language from linguistic point of view.
2.
The transfer stage varies the outputs of the analysis stage to produce structural and linguistic equivalents
between the two languages.
3.
The generation stage is the third stage in which a target language dictionary is applied to result the target
language document on the basis of linguistic information. (Steiner, 1988)
•
Interlingua Machine Translation Approach

First the source text meaning is decoded

Second the resulted meaning is re-encoded in the target language
Approaches to MT cont’d.
• Rule-based Machine Translation Approach
It operates on the linguistic data on source and target languages fundamentally
taken from bilingual dictionaries and the basic semantic, morphological, and
syntactic grammar of the individual language (Gelbukh, 2011).
Minimally, to get a Persian translation of English sentence one needs:
1.
A dictionary that will map each English word to an appropriate Persian
word.
2.
Rules representing regular English sentence structure
3.
Rules representing regular Persian sentence structure
4.
And finally, we need rules according to which one can relate these two
structures together.
Approaches to MT cont’d.
•
Statistical Machine Translation Approach
This system uses a corpus or database as a translated example for analyzing and decoding
source language. In comparison with the machine translation of about three decades ago,
Google Translate as an example of more contemporary automated engine for the task of
translation has taken a giant leap. However, it is still too imperfect. (Nierenberg, 1998)
•
Hybrid Machine Translation Approach
1.
Rules post-processed by statistics in which translation are practiced on the pivot of
rule-based engine. Next statistics are applied to correct the output.
2.
Statistics guided by rules in which rules have an important role to pre-process date
to quite the statistical representation to normalize. This approach is powerful,
flexible and under more control when it’s translating.
Evaluation of MT Systems
• Human translation assessment (Secară 2005; Williams 2001) has been
moving from microtextual, word- or sentence-level error analysis methods
toward more macrotextual methods focused on the function, purpose and
effect of the text. At the same time, machine translation assessment has
mainly been microtextual and focused on the aspects of accuracy and
fluency.
• Hovy (2002) discussed the complexity of MT evaluation, and stressed the
importance of adjusting evaluation to the purpose and context of the
translation.
Evaluation of MT Systems cont’d.
Mary A. Flangan Believed that Machine translation quality can be difficult to
quantify for a number of reasons:
1)
A text can have several different translations, all of which are correct.
2)
Defining the boundaries of errors in MT output is often difficult. Errors sometimes
involve only single words, but more often involve phrases, discontinuous
expressions, word order or relationships across sentence boundaries. Therefore,
simply counting the number of wrong words in the translation is not meaningful.
3)
One error can lead to another. For example, if the part of speech of a word is
identified incorrectly by the MT software, the entire analysis of the sentence may be
affected, creating a chain of errors.
4)
The cause of errors in MT output is not always apparent. The evaluator usually does
not have access to a trace of the software's tests and actions. Thus it can be difficult
to identify what went wrong in the translation of a sentence.
Evaluation of MT Systems cont’d.
Types of Evaluation
 Automatic Evaluation
the Word Error Rate (WER), the Position independent word Error Rate (PER), the
BLEU (Papineni et al., 2002) and the NIST (Doddington, 2002) where the MT
output is compared to one or more human reference translations.
 Human Evaluation
Due to the complexity of natural language, manual evaluation seems more reliable
1.
Three passages were selected and translated by Rule-based MT Systems
and compared with one Statistical MT System and Human translator
2.
Error categories were derived after the analysis of each text
Methods & Materials
•
•
•
•
•
Three passages were translated by two different MT systems and also a human
translator.
From each text type a passage of approximately 400 words was taken from story,
user guide and magazine.
The rule-based MT – Arya TM– system was designed based on thousands of lexical
and grammatical rules.
The statistical system, Google Translate by Google Inc., is based on the use of large
monolingual and parallel corpora for translation.
The unit of analysis was set to a sentence level because it’s the largest unit which
can be easily recognized in MT systems and ST sentence can be clearly
corresponded to its TT pairs.
Table of Source Text Passages for Analysis
Number of Words
Number of Sentences
398
13
390
16
415
15
Short Story
The Lottery
User Guide
Microsoft Access 2012
Magazine
Academic article
Error Categories & Descriptions
• For English-to-Persian Rule- based MT systems the following categories were
derived
Errors Category
Syntactic
Word
Order
Missing
Words
Punctuation
Parts
of
Speech
Semantic
Conjugation
Unknown
Words
Incorrect
Words
Polysemy
Idiomatic
Expressions
Error Categories & Descriptions cont’d.
Description of Error Categories:
•
Syntactic Errors: Those errors that are related to the grammar of the language such as parts
of speech or conjugation
Word order that means sentence elements ordered incorrectly
Example: Commands generally take the form of buttons and lists. (User Guide)
. ‫دستور ها بطور کلی فرم شاگرد می گیرد و فهرست ها‬
Arya Translation System
.‫دستورات به طور کلی به شکل دکمه ها و لیست‬
Google Translate
Missing words: incorrect elision of some words
Example: This requires better data collection and analysis tools for studying outcomes and
consistent use of these tools across individual studies. (Magazine)
Arya Translation System
‫این مجموعه اطالعات بهتر نیاز های و ابزار ها تحلیل برای مطالعه می‬
. ‫کن حاصل ها و سازگار استفاده می کند‬
Google Translate
‫این امر مستلزم جمع آوری داده ها بهتر و با استفاده از ابزار تجزیه و‬
‫تحلیل برای بررسی نتایج و استفاده مداوم از این ابزار در سراسر مطالعات‬
.‫فردی‬
Error Categories & Descriptions cont’d.
Unknown words: word not in a dictionary
Example: The women, wearing faded house dresses and sweaters, came shortly after their
menfolk.( Story)
Arya Translation System
‫ به زودی پس‬، ‫ خسته کننده لباس ها خانه و ژاکت ها محو کردند‬, ‫زن ها‬
‫ شان آمدند‬menfolk ‫از‬
Google Translate
‫ در آمد مدت کوتاهی پس از‬،‫ پوشیدن لباس و ژاکت پژمرده خانه‬،‫زنان‬
.‫خود را‬menfolk
Punctuation: incorrect punctuation
Example: The children assembled first, of course. (Story)
Arya Translations
Google Translations
. ‫ البته جمع کردند‬, ‫بچه ها اول‬
.‫ البته‬،‫کودکان مونتاژ اول‬
Error Categories & Descriptions cont’d.
Parts of speech: errors in identifying pars of speech such as noun or verb
Example: If you decrease the width of the ribbon, small button labels disappear. (User Guide)
Arya Translation System
Google Translate
‫ دکمه کوچک ناپدید برچسب می زند‬، ‫اگر شما پهنا نوار کاهشبیابید‬
‫ برچسب ها دکمه کوچک ناپدید می شوند‬،‫اگر عرض نوار شما را کاهش دهد‬
Conjugation: incorrectly formed verb or wrong tense
Example: Soon the women, standing by their husbands, began to call to their children, and the
children came reluctantly, having to be called four or five times.
Arya Translations
‫ به شروع کردن صدا به بچه ها‬, ‫ حمایت می کن شوهر های شان‬, ‫بزودی زن ها‬
. ‫ می دارد که اشد صدا زده چهار یا پنج دوره‬، ‫ و بچه ها با اکراه آمدند‬, ‫شان‬
Google Translations
‫ و بچه ها به‬،‫ شروع به تماس به فرزندان خود‬،‫ شوهران خود ایستاده‬،‫به زودی زنان‬
.‫ به نام چهار یا پنج بار‬،‫اکراه‬
Error Categories & Descriptions cont’d.
•
Semantic Errors: Those errors that are related to the meaning such as incorrect meaning
of words or expressions which caused the incorrect meaning of the whole sentence.
Incorrect word: completely incorrect meaning
Polysemy: incorrect selection of the meaning of the words with more than one meaning
Example: The people of the village began to gather in the square, between the post office and the bank,
around ten o'clock.
Arya Translations
Google Translations
‫ در‬, ‫مردم روستا شروع کردند که جمع شوند در مربع‬
‫ حدود َده ساعت‬، ‫میان پستخانه و بانک‬
‫ بین اداره‬،‫مردم روستا در میدان شروع به جمع آوری‬
‫ ساعت حدود ده‬،‫پست و بانک‬
.
Style and idiomatic expression : incorrect translation of multi-word expression
Example: They greeted one another and exchanged bits of gossip as they went to join their husbands.
Arya Translations
‫آنها سالم همدیگر و ذره غیبت معاوضه کردند آنها رفتند که وصل‬
. ‫کنند شوهر های شان‬
Google Translations
‫آنها استقبال یکدیگر و رد و بدل بیت از شایعات بی اساس را به‬
.‫عنوان آنها را برای پیوستن به شوهر خود رفت‬
Results & Discussions
Syntactic Category
Missing
Words
Table of Syntactic Errors
Word Order
RBMT
SMT
Human
Unknown
Words
Punctuation
Parts of
Speech
Conjugation
Word Order
Missing Words
Unknown
Words
Punctuation
Parts of Speech
Conjugation
Story
12
6
3
12
8
9
User Guide
17
5
1
10
5
13
Magazine
14
4
10
7
15
Story
11
7
3
12
7
8
User Guide
17
5
1
9
7
11
Magazine
15
2
5
9
6
14
Story
1
2
0
3
1
3
User Guide
0
0
2
1
0
1
Magazine
2
1
1
1
2
2
Results & Discussions contd.
Semantic Category
Incorrect Lexicon
Polysemy
Table of Semantic Errors
Incorrect Lexicon
RBMT
SMT
Human
Idiomatic Expression
Polysemy
Idiomatic Expression
Story
10
7
12
User Guide
7
9
5
Magazine
7
11
9
Story
8
8
9
User Guide
3
7
3
Magazine
5
13
8
Story
0
0
2
User Guide
1
0
0
Magazine
0
1
1
Results & Discussions cont’d.
•
Both systems made the least errors with the simpler sentences and the most ones with the
compound- complex sentences, as well as lexically or structurally ambiguous texts. This is
because ambiguous source texts with different contents can correspond with more than one
representation.
•
For the rule-based system, the most typical errors are in conjugation, word order and also in
rendering polysemous words and idiomatic expressions. For the statistical system the most
common error is in conjugating and determining the tense. However, it has also some
problems in translating words with multiple meaning and idiomatic expression.
•
To see whether machine translation accuracy is affected by text-type three different genres
were analyzed thoroughly. And for the different text types, the rule- based system had
similar amounts of syntactic and semantic errors in each text.
Future!
• Evaluating MT quality is necessarily a subjective process because it involves
human judgments.
• Determining the best category for an error in MT output is not easy because we
have to place them on how they are realized rather than the cause of errors and
many machine translated sentences contained multiple, linked errors.
• Future work will therefore be focused on the cause of errors and ranking error
categories. The error categories presented here is flexible, allowing for the
deletion or addition of more categories.