Transcript Reflections in the mirror: the contribution of self and

GALA 14th INTERNATIONAL
CONFERENCE
Advances in Research on Language Acquisition and
Teaching
Thessaloniki, 14-16 December 2007
“ Reflections in the mirror:
the contribution of self and peer assessment in
the teaching of speaking skills”
AEGINITOU V.,
NTELIOU E.,
VLAHOYANNI N.
CHAROKOPIO UNIVERSITY
ATHENS
Overview
1. Introduction
1.1. Peer assessment: Benefits
1.2. Self-assessment
2. Our study
2.1 Background
2.2 Context and purpose
2.3 Methodology
2.4 Discussion and results
2.4.1 Students’ presentations
2.4.2 Questionnaires
2.4.3 Tutorials
3. Implications – Conclusions
4. Bibliography
1.1 Peer assessment: Benefits
a. development of professional skills
b. students’ involvement in the learning
process
c. better rapport between speaker and
audience
d. increased objectivity of results
Boud & Holmes, 1995; Stefani, 1998; Lejk et al, 1999; Magin & Helmore, 2001; Falchikov,
1986, 1995; Magin & Churches, 1989; Mockford, 1994; Lynch, 1988
1.2 Self assessment: Benefits
a.
b.
c.
d.
e.
f.
Monitoring of learning and progress
Setting goals for the future
Encouraging responsibility for learning
Promoting critical thinking
Constructing and reconstructing knowledge
Bridging the gap between high and low
achievers
(Carr, 2002; Harlen & Winter, 2004)
2. Our study
2.1 Background: Pilot study 2005
• Subjects: EAP/ESP under graduates
• Purpose
2.2 Context and purpose
Research Questions
1. Is there a significant level of agreement
between the tutors’ and the students’
assessment of oral presentation skills?
2. Is peer evaluation motivating and useful?
3. To what extent is self assessment enhanced by
peer assessment?
2.3 Methodology
• prior training
- presentation of their own strengths and weaknesses
- assessment checklists (different for
tutors and students),
- audio-taped sample presentations
• Students’ presentations & Questionnaire completion
• Tutorials
• Statistical tools: (SPSS-Matlab)
Cohen’s Kappa statistic, Spearman correlation, Mc NemarBowker test
Assessment criteria
• CONTENT: content relevant to title / clear central idea /
topic well supported / proper use of sources
• ORGANIZATION: clear introduction / main points
coherently stated / main points cohesively stated /
relevant conclusion
• LANGUAGE: accurate and clear/ voc. appropriate to topic
/ technical vocabulary clearly explained / use of
transitions / comprehensible pronunciation
• PRESENTATION TECHNIQUES: speed / loudness of
voice / eye contact
• VISUAL AIDS: clarity / length
Adapted from: Rignall, M. and Fourneaux, C. 1997. Speaking (English for Academic
Studies Series). UK: Prentice Hall.
2.4. Discussion and results
2.4.1. Analysis of students’ presentations
Intermediate level
Kappaweighted pvalue
Variables
Kappa
kappa p-value
Kappa weighted
Topic support
0.483
0.001
0.503
0.014
Cohesion
0.637
0.000
0.637
0.000
Clarity of
visual aids
0.543
0.000
0.550
0.017
Speed
0.787
0.000
Loudness
0.653
0.033
Eye contact
0.386
0.001
0.479
0.004
Kappaweighted pvalue
Variables
Kappa
kappa p-value
Kappa weighted
Content
relevant to
title
-0.031
0.517
-0.031
0.512
Sources
-0.021
0.591
0.105
0.265
Technical
vocabulary
0.108
0.182
0.164
0.1921
Advanced Level
Kappaweighted pvalue
Variables
Kappa
kappa p-value
Kappa weighted
Topic support
0.380
0.006
0.353
0.049
Clarity of
visual aids
0.540
0.000
0.580
0.001
Speed
0.382
0.004
Kappaweighted pvalue
Variables
Kappa
kappa p-value
Kappa weighted
Sources
0.146
0.131
0.293
0.063
Cohesion
-0.019
0.546
0.031
0.454
Technical
vocabulary
0.160
0.092
0.133
0.227
Loudness
0.079
0.408
Eye contact
0.192
0.072
0.289
0.066
Problematic variables
Intermediate Level
Advanced Level
1. Use of sources
1. Use of sources
2. Technical vocabulary
2. Technical vocabulary
3. Content relevant to
title
3. Cohesion
4. Eye contact
5. Loudness
Sources (Intermediate)
Symmetric Measures
Asymp.
Std. Error
Value
a
Approx. T
b
Approx. Sig.
Ordinal by
Ordinal
Kendall's tau-b
,456
,128
3,349
,001
Spearman Correlation
,488
,137
3,112
,004c
Interval by Interval
Pearson's R
,436
,127
2,700
,011c
N of Valid Cases
33
a. Not as suming the null hypothesis.
b. Using the asymptotic standard error ass uming the null hypothesis.
c. Based on normal approximation.
Sources (Intermediate)
Sources (students) * Sources (professors) Crosstabulation
Count
Sources (professors)
No
Sources
(s tudents)
No
Quite
Yes
Total
Quite
Yes
Total
3
0
0
3
13
2
1
16
5
7
2
14
21
9
3
33
Sources (Intermediate)
Chi-Square Tests
Value
McNemar-Bowker Test
N of Valid Cases
Asymp. Sig.
(2-sided)
df
22,500
33
3
,000
Sources (Advanced)
Symmetric Measures
Asymp.
Std. Error
Value
a
Approx. T
b
Approx. Sig.
Ordinal by
Ordinal
Kendall's tau-b
,475
,127
3,534
,000
Spearman Correlation
,530
,137
3,477
,002c
Interval by Interval
Pearson's R
,534
,116
3,520
,001c
N of Valid Cases
33
a. Not as suming the null hypothesis.
b. Using the asymptotic standard error ass uming the null hypothesis.
c. Based on normal approximation.
Sources (Advanced)
Sources (students) * Sources (professors) Crosstabulation
Count
Sources (professors)
No
Quite
Yes
Total
Sources
No
5
0
0
5
(students)
Quite
5
5
3
13
Yes
1
10
4
15
11
15
7
33
Total
Sources (Advanced)
Chi-Square Tests
Value
McNemar-Bowker Test
N of Valid Cases
Asymp. Sig.
(2-sided)
df
9,769
33
3
,021
Technical vocabulary (Intermediate)
Symmetric Measures
Asymp.
Std. Error
Value
a
Approx. T
b
Approx. Sig.
Ordinal by
Ordinal
Kendall's tau-b
,326
,152
2,135
,033
Spearman Correlation
,345
,161
2,045
,049c
Interval by Interval
Pearson's R
,347
,161
2,062
,048c
N of Valid Cases
33
a. Not as suming the null hypothesis.
b. Using the asymptotic standard error ass uming the null hypothesis.
c. Based on normal approximation.
Technical vocabulary (Intermediate)
Technical Vocabulary (students) * Technical Vocabulary (professors)
Crosstabulation
Count
Technical Vocabulary
(students)
Total
Quite
Yes
Technical Vocabulary (professors)
No
Quite
Yes
10
5
3
4
4
7
14
9
10
Total
18
15
33
Technical vocabulary (Advanced)
Symmetric Measures
Asymp.
Std. Error
Value
a
Approx. T
b
Approx. Sig.
Ordinal by
Ordinal
Kendall's tau-b
,145
,167
,863
,388
Spearman Correlation
,155
,179
,873
,389c
Interval by Interval
Pearson's R
,152
,186
,859
,397c
N of Valid Cases
33
a. Not as suming the null hypothesis.
b. Using the asymptotic standard error ass uming the null hypothesis.
c. Based on normal approximation.
Content relevant to title (Intermediate)
Symmetric Measures
Asymp.
Std. Error
Value
Ordinal by Ordinal
a
Approx. T
b
Approx. Sig.
Kendall's tau-b
-,031
,022
-,730
,466
Spearman Correlation
-,031
,022
-,174
,863c
Interval by Interval
Pearson's R
-,031
,022
-,174
,863c
Measure of Agreement
Kappa
-,031
,022
-,180
,858
N of Valid Cases
33
a. Not as suming the null hypothesis.
b. Using the asymptotic standard error ass uming the null hypothesis.
c. Based on normal approximation.
Content relevant to title (Advanced)
Content r elevant to topic (students) * Content re levant
to topic (professors) Cr ossta bula tion
Count
Content
relevant to
topic
(professors)
Yes
Content relevant to
topic (students)
Total
Yes
Total
33
33
33
33
Cohesion (Advanced)
Symmetric Measures
Asymp.
Std. Error
Value
a
Approx. T
b
Approx. Sig.
Ordinal by
Ordinal
Kendall's tau-b
,085
,167
,501
,617
Spearman Correlation
,089
,176
,496
,623c
Interval by Interval
Pearson's R
,119
,161
,670
,508c
N of Valid Cases
33
a. Not as suming the null hypothesis.
b. Using the asymptotic standard error ass uming the null hypothesis.
c. Based on normal approximation.
Loudness (Advanced)
Symmetric Measures
Asymp.
Std. Error
Value
a
Approx. T
b
Approx. Sig.
Ordinal by
Ordinal
Kendall's tau-b
,194
,234
,761
,447
Spearman Correlation
,196
,236
1,110
,275c
Interval by Interval
Pearson's R
,117
,193
,656
,517c
N of Valid Cases
33
a. Not as suming the null hypothesis.
b. Using the asymptotic standard error ass uming the null hypothesis.
c. Based on normal approximation.
Eye contact (Advanced)
Symmetric Measures
Asymp.
Std. Error
Value
a
Approx. T
b
Approx. Sig.
Ordinal by
Ordinal
Kendall's tau-b
,458
,124
3,477
,001
Spearman Correlation
,496
,134
3,184
,003c
Interval by Interval
Pearson's R
,496
,123
3,176
,003c
N of Valid Cases
33
a. Not as suming the null hypothesis.
b. Using the asymptotic standard error ass uming the null hypothesis.
c. Based on normal approximation.
Eye contact (Advanced)
Eye contact (students) * Eye contact (professors) Crosstabulation
Count
Eye contact
(students)
Total
No
Quite
Yes
Eye contact (professors)
No
Quite
Yes
4
0
7
7
2
7
13
14
Total
0
2
4
6
4
16
13
33
Eye contact (Advanced)
Chi-Square Tests
Value
McNemar-Bowker Test
N of Valid Cases
Asymp. Sig.
(2-sided)
df
11,778
33
3
,008
2.4.2. Questionnaire analysis
1. While listening to the presentations of your
classmates, have you learned anything new on the
topics under discussion?
12,12%
1,52%
Q1
18,18%
No
Little
Quite
A lot
Definitely ye s
Pies show counts
33,33%
34,85%
2. Has the organization of the presentations helped
you in the way you will organize your future
presentations?
Q2
9,09%
Little
19,70%
Quite
A lot
Definitely yes
24,24%
46,97%
Pies show counts
3. Have you learned useful words / expressions in
your subject area?
6,06%
7,58%
Q3
No
Little
Quite
19,70%
24,24%
A lot
Definitely yes
Pies show counts
42,42%
4. Were the visual aids helpful in your future selection
of relevant graphics?
1,52%
18,18%
Q4
13,64%
No
Little
Quite
A lot
Definitely yes
Pies show counts
34,85%
31,82%
5. Did you find evaluating your classmates interesting?
10,61%
3,03%
Q5
No
12,12%
Little
Quite
A lot
Definitely yes
Pies show counts
36,36%
37,88%
2.4.3. Tutorials
Questions asked in the tutorials
1. Were you satisfied with your presentation?
2. In which aspects of your presentation you
feel you need more practice? Why?
3. In which aspects of your presentation you
feel you performed well and you would
not change?
4. Is it easy for you to assess yourself?
Students’ comments
“I did not do any presentations at school. I am
not quite sure what I have to do”.
“This is not my job. The teacher should do
that”.
“Before the training I did not know how to
assess myself or my classmates. Now I
think I can”.
Analysis of students' comments
• Weaknesses more easily identified than
strengths
• Participation in self-assessment procedures
can facilitate better judgement on
performance levels
• More realistic goals are set for future
presentations
Bachman & Palmer 1989; Ready-Morfitt, 1991; Dickinson, 1987; Oscarson, 1997
3. Implications
• Number of subjects
• Absence from training session
• Future design of self-assessment practice
3. Conclusions
• Prior training positively modified the results
• The beneficial combination of peer and selfassessment processes
• Two problematic areas: technical
vocabulary and reference to sources
• Self-reflective practices should be
introduced in the early stages of instruction
• Future research
4. Bibliography
•
•
•
•
•
•
•
•
Altman DG. 1991. Practical Statistics for Medical Research. London: Chapman & Hall.
Bland JM & Altman DG. 1986. “Statistical methods for assessing agreement between
two methods of clinical measurement”. Lancett 32, pp. 307-10.
Boud, D. & Holmes, H. 1995. “Peer and self marking in a large technical subject”. In: D.
Boud (Ed.) Enhancing learning through self assessment, London, Kogan, 63-78.
Brindley, G. 2001. ‘Assessment’. In Carter, R. & Nunan, D. (Eds.). The Cambridge
Guide to Teaching English to Speakers of Other Languages. Cambridge: CUP. pp. 137143.
Carr, S.C. 2002. “Self-evaluation: involving students in their own learning”. Reading
and Writing Quarterly, 18, pp.195-199.
Cohen, JA. 1968. Weighted Kappa: nominal scale agreement with provision for scaled
disagreement or partial credit. Psychological Bulletin 70, pp. 213-20.
Falchikov, N. 1986. “Product comparisons and process benefits of collaborative peer
group and self assessments”. Assessment and evaluation in Higher education, 11, 146166
Falchikov, N. 1995. Peer feedback marking: developing peer assessment, Innovations in
Education and Training International, 32, 175-187.
4. Bibliography
•
•
•
•
•
•
•
•
•
•
•
Fleis JL & Cohen JA. 1973. “The equivalence of weighted Kappa and the intraclass correlation
coefficient as measures of reliability”. Educational Psychology Measurements 33, pp. 613-9.
Harlen, W. & Winter, J. 2004. “The development for assessment for learning: learning from the
case of science and mathematics”. Language Testing, 21(3), pp.390-408.
Hughes, I.E. & Large, B.J. 1993. ‘Staff and peer-group assessment of oral communication skills’.
Studies in Higher Education, 18(3), 379-385.
Landis, JR, Kock, GG. 1977. “The measurement of observer agreement for categorical data”.
Biometrics 33, pp.159-74
Lejk et al. 1999. “Group assessment in systems analysis and design : a comparison of the
performance of streamed and mixed-ability groups”. Assessment and evaluation in Higher
education, 24, 5-14.
Lynch, T. 1988. Peer evaluation in practice, in: A. Brookes and P. Grundy (Eds.) Individualisation
and autonomy in language learning. ELT Documents 131
Magin, D. & Churches, A. 1989. “Using self and peer assessment in teaching design”, Proceedings,
World conference on Engineering Education for Advancing Technology, Institution of Engineers,
Australia, 89/1, 640-644
Magin, D. & Helmore, P. 2001. “Peer and Teacher Assessments of Oral Presentation Skills: how
reliable are they?”. Studies in Higher Education, 26/3, 287-298.
Mockford, C. 1994. “The use of peer group review in the assessment of project work in higher
education”, Mentoring and Tutoring, 2, 45-52.
Rignall, M. & Fourneaux, C. 1997. Speaking (English for Academic Studies Series). UK: Prentice Hall.
Stefani, L. 1998. “Assessment in partnership with learners”, Assessment and evaluation in Higher
MERRY CHRISTMAS