Transcript University of Bristol 14th of March, 2008 Presentation

Advisory Group Meeting, 6th
November 2009
Student Performance in National Examinations (SPINE): the
dynamics of language in school achievement
(ESRC/DfID RES-167-25-0263)
www.bristol.ac.uk/spine
2
Research Team Changes
• Bristol:
– Dr Neil Ingram (Biology)
– Dr Federica Olivero (Maths)
• Zanzibar:
– Dr Haji Mwevura (overseas: post-doctoral
attachment); Shumbana Said (overseas:
doctoral studies)
– Mohamed Abeid Mbarouk (Maths & Science)
RESEARCH UPDATE AND FINDINGS
• Study 2 : System wide examination study
• Study 4: Classroom study: focus on Teacher
Assessment Issues
• Study 5: Examination achievement: focus on the
learners
• Update on data collection for Process study (5.2) and
Product study (5.3)
• Dissemination: Conferences, Publications
4
STUDY 2: ENG and MATH
Total: 64.717, school=29.18%,
pupil=70.82%
ENGLISH explains (90.464.717)/90.4=28.41% of the
maths total variance
Total=90.4, School=30.75%,
Pupil=69.25%
5
STUDY 2: ENG and BIO
Total=65.646, school=18.17%,
pupil=81.83%
ENGLISH alone explains
(114.968-65.646)/114.968=42.90%
of the total variance in BIOLOGY
Total=114.968, school=18.79%,
pupil=81.21%
6
STUDY 2: English and CHEM
Total=158.281, school=23.06%,
pupil=76.94%
ENGLISH explains (275.993158.281)/275.993=42.65% of the
total CHEM variance
Total=275.993, school=22.05%,
pupil=77.95%
7
STUDY 2: Summary of multilevel models
•
ENGLISH is a significant and substantial
predictor of student performance in MATH,
BIO & CHEM.
•
School-level variances – all models (using
ENGLISH) as the single explanatory variable
demonstrated a substantial proportion of
the variance attributable to school factors
Study 4:
Focus on Teacher Assessment
8
In Form II, Teacher Assessment (TA) was
implemented based on the following assessment
procedures:
• Class work
• Oral questions
• Homework
• Weekly tests
• Terminal tests
•
Study 4:
Focus on Teacher Assessment
• Understanding of what assessment is to be done
• Awareness of guidelines
• Percentage of marks that go to the final
examination
• Sharing what teachers know and practice about
assessment guidelines among teachers within and
across schools
• Level of monitoring of Teacher Assessment by
head teachers.
Study 2
Study 4:
Focus on Teacher Assessment
However: significant variability among teachers:
• the types of specific activities that constitute teacher
assessment
• the number of activities from which they take marks for
each month
• decisions on how to handle student absenteeism
• how to get 10% of marks for each month; some picked
the one that students did better on, some picked
randomly, and some put the average of all marks
11
Study 5.1: What are the four things that
man could resemble whales?
45 students took this item:
• 35.6% = no answer
• 26.7% = wrong answer
• 28.9% = partially correct answer
•
8.8% = correct answer
12
Study 5.1: Interview: D1 who didn’t answer
Q3 explains
“because I did not understand by this this …
resemble” (lines 115-117)
Int:
“If I tell you that resemble means ‘to look like’
… can you do the question now?
D1: “Yes”
Int:
OK so what’s the answer?
D1: “Man … is warm blooded … and whales also
… whales have lungs and man also have lungs
…” (122-133)
D1:
Study 5 - Biology:
responses to original item
13
•
Give a brief explanation on what will happen on the following: A
locust not suffocating when its head is immersed in water while the
remaining part of the body is outside.
• No answer
=
67.4%
• Wrong answer
=
21.7%
• Partially correct answer =
6.5%
• Correct answer
4.4%
=
14
Biology: on locusts
15
Biology: on locusts
16
Biology: on locusts
Questions:
a) In which picture do you think the locust will/may
die?
b) Why do you think it will/may die?
17
Original item modified:
• Greater contextualisation
• Simplification of instruction
• 2 structured parts: A & B
• Visual clues to support information retrieval
• Rephrasing of the item
• Altering item layout
18
Results
• Original item
– Only 32.6% of students wrote an answer
– Just under 11% gave a partially correct or
correct answer
• Modified item
– 100% responded to this item
– 42% gave a partially correct answer to Part A
– 53% gave a partially correct answer to Part B
19
Changes in student response
OR
Sc
Written responses on
Modified Item
MOD
Score
Comment
(Neil Ingram:
biologist)
H3 0
In picture A the locust may/will
die
I think it will/may dies because
the locust get its breathing by
using its body
More
than 0
Understands that
locusts breathe using
the body
C2 1
I think it will/may die because
the boy is dipping the locust in
the water to all the bodies with
its trachea that used to
respiration as a respiratory
surface of a locust
more
than 0
Getting very close to
a very complex
answer “I think a very
able pupil indeed”.
The right answer but
with great difficulty in
expressing this in
English
20
Study 5: Language Factors in Exam
Performance
• Considerable challenges faced by students in
understanding English lexis/vocabulary
• Piloted an ‘off the shelf’ vocabulary test (receptive)
• Developed a measure of vocabulary knowledge based on
corpora (an innovation in the design) developed from all
past examination papers & current textbooks (Maths,
Chemistry, and Biology)
• Piloted (Sept09), revised administered (Nov09)
• To our knowledge this is the first time a special purpose
receptive vocabulary measure has been developed for
English L2 school-age learners in sub-Saharan Africa.
21
Study 5.2: Focus on Process – CH & MT
• Aims: to capture in-depth insights on how
students:
– process and produce their answers to
examination questions
– use both English and Kiswahili in so doing
– use English and Kiswahili in class
• Stimulus
– video clips from Study 4
– exemplar examination items
22
Study 5.3: Focus on Product (achievement)
• Aim: to probe further the complex relationships between
the use and effects of learning and assessment through
English.
• developed exemplar examination items for Maths,
Chemistry and Biology
• 3 different forms of the examination for c. 1000 Form II
learners:
– English only
– Kiswahili only
– bilingual: Kiswahili and English.
23
Study 5.3:Focus on Product (achievement)
• Receptive vocabulary measure
• Student questionnaire (capture variables e.g.
English language exposure, SES data)
• Head Teacher questionnaire (capture variables
e.g. teacher qualifications matched with subjects
they teach)
24
Progress of SPINE
• anticipated all data collection activities will be
completed on schedule as well as the bulk of the
analyses – end July 2010
• refinement of analyses, writing & publication, &
ongoing dissemination of research findings will be
ongoing post July 2010.
25
Next Steps
• Data analysis ongoing & preparation of project
outputs
• Dissemination in Zanzibar & regional engagement:
practitioners, academics, NGO, policy makers
• SPINE International Symposium, Zanzibar 2011
• SPINE ‘mark 2’: what do we - SUZA/MoEVT - want
to do next?