Transcript Leaving our mark(ing) with computer

Nadaclair.com: Cut
L2Marking Time in Half (and
maybe they’ll learn at the same
time!)
Terry Nadasdi, Stéfan Sinclair, Denis
Cousineau
Preliminary remarks
 My focus is error identification
 This is an activity on which teachers spend
countless hours and dollars
 Is it helpful?
 Probably
 This study is indirectly interested in this
 Who is quantitatively better and by how
much?
Overview of talk
a) discussion of previous “research”
b) discussion of methodology (who
are the humans, what is the
computer?)
c) presentation of quantitative results
d) discussion of results
e) conclusions about which is better
(and what that means)
Advantages
a) some studies found checkers
helpful for SLA since they find a
large number of errors (e.g.
Burston, 2001)
b) one study suggests that they are
good because they make learners
reflect on language (e.g. Durel,
2003)
Disadvantages
- Too many false positives
- No L1 interface
- Not designed for L2 learners
- Not pedagogical in orientation
Research problems
a) Small databases
b) Unrealistic representation of
human correction.
c) Criticisms have usually been
based on invented sentences, not
on empirical data.
Our study
Who is better?
Humans or …. Computers?
Our Data
- 30 random texts submitted to
BonPatron.com
- 1000 - 1500 characters (roughly 250
words),
- Anglophones, residing in Canada (outside
of Quebec)
The humans
- Two University Professors
- Over 20 years experience marking
FSL texts
- PhD in French linguistics
- PhD in French literature
Error Legend
sp: Spelling (any word caught by spell check);
only one error per word coded
agdn: Determiner/Noun agreement error, e.g.: “tes
ami”
agan: Adjective/Noun agreement error, e.g.:
“premier chose”; past participle agreement, e.g.:
“la chose que j’ai remarqué
agsv: Subject/Verb agreement error, e.g.:
mw: Missing word, e.g.: “Je veux _ il mange”.
wf: Wrong word form (e.g. infinitive instead of
past participle, etc.); generally any time a French
word form (found in the dictionary), was used in
place of another word form, e.g. il a manger
instead of il a mange
Error Legend (Con’t)
la: Lexical (anglicism), e.g. “avoir un bon temps”
wc: Word choice (e.g. avoir instead of être; wrong
preposition)
p: Punctuation (capitalization, missing space,
missing comma, etc.)
wo: Word order (syntax problems), e.g.: je veux la
el: Elision, e.g.: “je aime”
Example of human correction
Le*agdn premier*agan chose que nous avons remarqué*agan a
été*wf le temps*la que tu te chouches*sp. Plutôt de*wc 11h 30, tu
devrais te couches*wf à 10h. Ce changement est le plus
importante*agan de tout. Si tu te chouches*sp à 10h, tu peux te
léver*sp à une heure matinale. Le*agdn deuxième chose *mw nous
avons remaqué*sp*agan a été*wf que tu prendsd*sp l’*p autobus
pour arriver à l'école très vite. Nous pensons que tu devrais
marcher*la à l'école. Puis tu auras marcher*wf pour*la
un*pkilometre*sp, tu seras en bonne forme. Probablement que tu
n'auras pas le temps *mw récontrer*sp tes amis avant le début des
cours *pmais ce n'est pas très improtante*sp*agan …
The computer: BonPatron!
Designed to address many of the
criticisms previously raised in
software reviews
Basic Stats
Daily page views: 120,000
Daily users: 30, 000
Countries: 183
Who: school kids, university students,
private industry, sword makers,
roller-skaters, judo enthusiasts, selfmutilators, horse lovers, etc.
Let’s have a look at this
BonPatron of yours:
www.pro.bonpatron.com
1) Il a reste
Process-writing approach
Table 1 (human results)
Error type
Super Human
grammar
845
Punctuation
172
Spelling
206
Elision
15
Total
1238
Computer1
Computer2
Table 2 (computer1)
Error type
Super Human
Computer1
grammar
845
564 (68%)
Punctuation
172
240 (140%)
Spelling
206
203 (99%)
Elision
15
14 (93%)
Total
1238
1021 (82%)
Computer2
Table 3 (computer2)
Error type
Super Human
Computer1
Computer2
grammar
845
564 (68%)
729 (86%)
Punctuation
172
240 (140%)
251 (146%)
Spelling
206
203 (99%)
230 (112%)
Elision
15
14 (93%)
17 (113%)
Total
1238
1021 (82%)
1227 (99%)
Table4 (grammar subcategories)
Error code
Super human
Computer1
Computer2
Adj./Nous agr.
102
62 (61%)
89 (87%)
Det/Noun agr.
116
96 (83%)
113 (97%)
Subj./verb agr.
91
77 (85%)
89 (98%)
Word choice
179
103 (58%)
138 (77%)
Word order
16
7 (44%)
11 (69%)
Missing word
47
25 (53%)
33 (70%)
Lexical anglicism
60
43 (72%)
57 (95%)
Word form
234
151 (65%)
199 (85%)
Total
845
564 (68%)
729 (86%)
Table 5: results excluding space before punctuation mark:
Error type
Super Human
Computer1
Computer 2
grammar
845
564 (68%)
729 (86%)
Punctuation
91
108 (119%)
119 (131%)
Spelling
206
203 (99%)
230 (112%)
Elision
15
14 (93%)
17 (113%)
totals
1157
889 (77%)
1095 (95%)
Discussion
a) what does human correction
really look like?
b) do students learn from human
correction?
2) “Enfin, exhaustif dans son repérage … [la médiation humaine] … est
pratiquement infaillible même s’il peut parfois être influencé par des
facteurs psychologiques (être tolérant ou agacé, se lasser ou même
s’ennuyer) ou physiologiques (se fatiguer)” (Cordier-Gauther & Dion,
2003).
Perfect in it’s ability to identify errors, human correction is
almost infallible, even it can occasionally be influenced by
psychological factors (to be accepting, or bothered, lose
interest or even become bored or physiological (become
tired).
In the context of the student – teacher
marking experience, certain humans
are at best slightly better at error
identification, but computers may
well be better at error remediation.
Computer advantages
a) Tireless (and always available)
b) Impartial
c) Real-time feedback
d) Explicit corrective feedback (with
an example)
Vast majority of research shows
the positive effect of corrective
feedback on L2 written production
- Ferris (1995, 1997, 1999, 2003, 2006)
- Chandler (2000, 2003)
- Bitchener (2006, 2007)
- Bitchener, Young & Cameron (2003)
I am in grade 7 and My French has improved because of BonPatron, I used
to get c-- or c--- even F now i get c+ and sometimes c, and I hope to get a B
some day! BonPatron you've changed my Life ! Geneviève
Mes notes de Français (je prends un cours d'histoire en immersion au niveau
10e année) étaient en train de baisser, mais avec l'aide de votre site mon
prof a noté beaucoup d'amélioration. Merci! Hannah Jackson
A colleague of mine, gave the BonPatron site as a tool for her advanced
translation class. She says she has never received assignments for that
class with so few agreement errors! Cheers and many thanks to the both of
you... Dana Paramskas, French Professor, University of Guelph
Thank you so much BonPatron you have helped my mark go up at least 20
percent now I have a happy B+ My teacher was really smart to tell you this
site! Sarah.
I use your website all the time. It is so awesome. I actually learn a lot of
great things and I like seeing that my writing is improving because I get
less and less little yellow and red marks. Keep it up, it's really great! Sarah
Concluding remarks
- Previous accounts of human marking have
not been realistic
- Our study has shown that even when
compared with super human correction,
BonPatron.com fares very well
(conservatively, it catches more than 70% of
errors)
- It is likely that the feedback BonPatron
provides is more beneficial for learning the
rules of written French
Thank you!