Transcript Vocabulary Progression Test

Vocabulary Progression Test
Nuzhat Jafari PHD student, CRELLA
Under supervision of
Professor Dr. Stephen Bax
Dr. Fumiyo Nakatsuhara
Dr. Sathena Chan
Overview
• Brief summary of my study
• Brief summary of Vocabulary Progression Test (VPT)
17/07/2015
N Jafari 0809237
2
Brief background and rationale
• Language learners use more strategies in vocabulary
learning than in any other linguistic competences
(Schmitt, 1997).
• Higher –achievement learners use more strategies than
lower-achievement peers (Ahmad,1989; Schmitt,1997;
Gu, 1994; Fan, 2003).
17/07/2015
N Jafari 0809237
3
Research gap
• “The research which has been done on vocabulary learning
strategies has tended to deal with individual or small numbers of
strategies, with very few studies looking at the group as a whole”
(Schmitt, 1997, p.199).
• No work is done on the Vocabulary Learning Strategies &
factors
To partially fill such a gap, this study is designed to
evaluate vocabulary learning strategies and other factors
in vocabulary acquisition in EFL context.
17/07/2015
N Jafari 0809237
4
Research questions
• What is the role of vocabulary learning strategies and other
factors in successful and unsuccessful learning of vocabulary?
RQ1: What is/are the overall pattern(s) of vocabulary learning
strategies and other factors adopted by Pakistani tertiary
students to acquire English vocabulary?
RQ2: What is the relationship between vocabulary learning
strategies and successful/unsuccessful vocabulary learning in
the Pakistani university context?
RQ3: What other factors influence vocabulary learning in this
context?
17/07/2015
N Jafari 0809237
5
Research design and methodology
• Longitudinal research
• Mixed methods approach
• Phases of study
• Research instruments
• Data analysis
17/07/2015
N Jafari 0809237
6
Participants
• Total participants : 356
• Intermediate level students
• Chartered Accountant students
17/07/2015
N Jafari 0809237
7
Data Collection
• January 2013
• Pre-VLT
• Pre-VPT
• Pre-SDS (four weeks)
• Class observations
• Field notes
17/07/2015
• January 2014
• Post-VLT
• Post-VPT
• Post-SDS (four weeks)
• Class observations
• Field notes
• Questionnaire
• Interviews
N Jafari 0809237
8
Specimen Item (VPT)
Write a sentence for each of the following words
(prepositions, idioms and words) to show that
you know what the word means and how it is used.
You may choose a different form of the word if you wish.
Fall back
Hanker after
Hostile to
Make away with
17/07/2015
N Jafari 0809237
9
Trait and Method of VPT
• “In language testing we are concerned with the extent to which
a test can be shown to produce scores that are an accurate
reflection of candidate’s ability in a particular area. It demands
an understanding of both trait and method. Trait is with the
underlying constructs/abilities we wish to measure in students,
the what of language testing. Method deals with the how, the
instrument we develop to provide us with the information about
these construct(s)” (Weir, 2005, p.1).
17/07/2015
N Jafari 0809237
10
Stages of VPT construction (Hughes, 1989)
1. Statement of the problem
2. Writing specifications for VPT
17/07/2015
N Jafari 0809237
11
Statement of the problem
• What kind of test is it?
• What is its precise purpose?
• What abilities are to be tested?
• How much time can be provided to perform the task?
(Hughes, 1989, p.48)
17/07/2015
N Jafari 0809237
12
 Aims of VPT
• To test vocabulary gain in 48 weeks
• To assess the aspects of word knowledge
addressed in this study
oProductive Knowledge
17/07/2015
N Jafari 0809237
13
 Suitability of VPT
• Fits in Pakistani background
• Participants of this study were aware of such sort of
tests
• Being used in Pakistani EFL context.
• Quick to take, easy to mark, easy to interpret
17/07/2015
N Jafari 0809237
14
Writing specifications for VPT
• Content
• Sampling
• Format & timing
• Scoring procedure
• Pretesting (piloting)
(Hughes, 1989, p.49)
17/07/2015
N Jafari 0809237
15
VPT Test Construction
• Adopted from previous studies (Paribakht & Wesche, 1996,
1997; Joe, 1995, 1998; McNeill, 1996; Read, 2000; Nation,
2001).
• Consultation
 class teachers & course managers
 past papers, syllabus
 reviewed related literature
• Content
 course books & Vocabulary lists
 sampling vigorously
17/07/2015
N Jafari 0809237
16
Need of Marking Scheme for VPT
• Practical to use and reflects the aspects of word
knowledge that need to be assessed (Read, 2000).
• The validity of the test also depends on how the items
are marked.
• Initial marking scheme.
• Mark each sentence out of 3.
• No marks
• One mark for the meaning of the word
• One mark for its grammar
• One mark for appropriate use in context ( Read, 2000)
17/07/2015
N Jafari 0809237
17
Expert focus group discussion
• Discussed and revised with group of CRELLA staff
experts
• Marked real sentence examples according to new
marking scheme
• After two weeks times I emailed finalised marking
scheme with few sentences from real data for marking
and feed back
• Compared results from each marker
• Utilised with their feed back
• Finalised the new marking scheme
17/07/2015
N Jafari 0809237
18
Vocabul
ary Item
Example sentences from real data
Fall After heavy blows our forces was forced to fall
back back
Ali Fall back when Farhan need his help.
After a heavy blow, our cavalry was forced
to fall back.
I fall back on my brother’s quarrel.
He fall back from the upper post to the
lower post.
The enemy fell back into a defensive
position.
The family has no savings to fall back on.
Mark/ feedback
2
1 – capitalisation, third person singular -s
2
0 – can’t guess what he/she wanted to say
1 – third person singular –s
2
2 –different meaning (i.e. rely on) from the previous
examples (i.e. retreat/move back)
2 – same as above comment
I fell back on skills I had learned years ago
when I had to earn a living for myself.
Imran falls back on Tahir’s protest against 0? Not sure what this student wanted to say…Imran
relies on Tahir in protesting?
17/07/2015
N Jafari 0809237
19
government.
Example sentences from real data
He did not work hard and cut a sorry figure
in the final exam.
Do not meddles with Mr Mehboob, he is very
dangerous.
He has a great power of fight hostile to Ali.
17/07/2015
N Jafari 0809237
Total
Mark
2
1
0
20
Finalised Marking scheme
Marking Scheme
2
Demonstrates a full understanding
and accurate grammatical use.
1
Demonstrates full/partial
a) full meaning+ partial grammatical accuracy understanding of meaning and/or
b) partial meaning+ full grammatical accuracy partial/ full grammatical accuracy.
c) partial meaning+ partial grammar
d) partial meaning
0
Fails to demonstrate any
understanding
17/07/2015
N Jafari 0809237
21
VPT an instrument used in main study
• How it was conducted
oTime
oPhases
oParticipants
17/07/2015
N Jafari 0809237
22
Markers’ training system by Harrison (1993).
• “The marker needs considerable practice if he is to apply it
consistently and, if more than one marker is involved ”
(Harrison, 1993, p.112).
• Discussion about what each of categories of marking scheme
means with samples
• Few sample sentences to mark
• All re-marked again after two weeks
• Emailed the same sentences to expert focus group to see
consistency between actual marker and the expert group.
17/07/2015
N Jafari 0809237
23
VPT: Inter-rater reliability
Table1: VPT: Inter-rater reliability
N
Pre-VPT
Post-VPT
356
356
Pearson
Correlation
r
.99
.99
sig
.000
.000
• Pre-vpt marker 1 (M=5.97), Std.Deviation (5.948)
• Pre vpt marker2 (M=5.87), Std.Deviation (5.913)
• Post-vpt marker 1(M=20.38), Std.Deviation (11.837)
• Pos-vpt marker 2 (M=20.12), Std.Deviation (11.687)
17/07/2015
N Jafari 0809237
24
A paired-samples t-test
Table 2: A paired samples t-test
Pre-vpt
Mean N
SD
Mean of t
difference
5.92
5.92
14.328
356
Post-vpt 20.24 356
df
Sig.(2tailed)
-28.245 355 .000
11.74
Pre & Post VPT t-test results:
• Pre-VPT (M=5.92), Post-VPT (M=20.24)
• The mean increase 14.32
17/07/2015
N Jafari 0809237
25
Limitations
• Only two markers marked VPT though independently
• Advanced learners can compose acceptable sentences without
having a good understanding of what the target word means (McNeill,
1996).
• Reproduce memorised sentence from book (McNeill, 1996).
• Used in previous studies (self report questionnaire):
oThe vocabulary knowledge scale (Paribakht & Wesche, 1996,
1997; Joe, 1995, 1998; McNeill, 1996)
17/07/2015
N Jafari 0809237
26
“This can allow the learners to demonstrate
several aspects of their vocabulary ability:
whether they understand the meaning of the
target word; whether they know the word
functions grammatically within a sentence and
more generally, whether they use the word
‘productively’ in their writing” (Read, 2000, p.175).
17/07/2015
N Jafari 0809237
27
Development
• Reliability analysis VLT, VPT, Questionnaire
• A paired-samples t-test to examine if there is any
difference between pre and post tests.
• Factor analysis questionnaire
• Correlation questionnaire-vocabulary test results
• Qualitative analysis Interviews, SDS, Observations and
personal diary
17/07/2015
N Jafari 0809237
28
Hughes, A. (1989) Testing for Language Teachers. Cambridge: CUP
Harrison, A (1983) A Language testing handbook. London: Macmillan Press
Joe, A. (1995) Text-based tasks and incidental vocabulary learning. Second Language Research 11, 149-158.
Joe, A. (1998) What effects do text-based tasks promoting generation have on incidental vocabulary acquisition? Applied
Linguistics 19, 357-377.
Nation, I.S.P. (2001) Learning Vocabulary in Another Language. Cambridge: CUP
Oxford, R.L. (1990) Language Learning Strategies: What Every Teacher Should Know. Boston: Newbury House.
Paribakht, T. S. and M. Wesche. (1993) Reading comprehension and second language development in a comprehensionbased ESL program. TESL Canada Journal 11, 9-29.
Paribakht, T. S. and M. Wesche. (1998) Vocabulary enhancement activites reading for meaning in second language
vocabulary acquisition. In Coady and Huckin (eds.), (1997), pp.177-200.
Read, J. (2000) Assessing Vocabulary. Cambridge: CUP
17/07/2015
N Jafari 0809237
29
Schmitt, N. (2000) Vocabulary in language teaching. Cambridge: CUP.
Schmitt, N. (2010) Researching Vocabulary. GB: Palgrave Macmillan.
Schmitt, N., and Schmitt, D.R. (1993) Identifying and assessing vocabulary learning
strategies. Thai TESOL Bulletin 5(4), 27-33.
Schmitt, N.(1997) Vocabulary learning strategies. In Schmitt, N., and McCarthy, M. (eds)
Vocabulary : Description, Acquisition, and Pedagogy. Cambridge: CUP, pp.199-227.
Weir, C.J. (2005) Language Testing and Validation. An Evidence-Based Approach. GB:
Palgrave Machmillan.
Zhang, B., and Li, C. (2011) Classification of L2 vocabulary learning strategies: evidence
from exploratory and confirmatory factor analyses. RELC, 42 (2), 141-154.
17/07/2015
N Jafari 0809237
30
[email protected]
Thank you very much
17/07/2015
N Jafari 0809237
31