Transcript from Average GCSE

Advanced Users –
MidYIS, Yellis & ALIS
Durham 2013
Understanding the Students
Introduction to the Test Data
Underlying Principle
If we measure a student’s ability we can determine ‘typical
progress’ for the individual and use this to inform likely
outcomes and against which to measure performance of
individuals and groups.
Q. How does this work ?
Q. How do we measure and interpret ‘ability’ ?
Q. How do we interpret the data fairly and reliably ?
Measuring and Interpreting
Ability
Options
1) Use Pre-existing qualification data
Post-16 – Average GCSE
2) Use Baseline Test
Post-16 and Pre-16 – Computer Adaptive Baseline
Test
Note: Issues regarding use of CABT alongside Average GCSE at Post-16 will be
examined later in the day with predictive information.
Adaptive approach
Low
Average
High
Baseline Test Standardisation
•
Test scores are standardised; Mean = 100, SD = 15
Standardised
Score
National
Percentage
Comment
>130
Top 2.5%
Traditional classification of ‘mentally gifted’
>120
Top 10%
>100
Top 50%
<80
Bottom 10%
<70
Bottom 2.5% Potential special educational needs ??
Stanine
1
2
4%
3
4
6
7
8
9
7% 12% 17% 20% 17% 12% 7%
A
v
e
r
a
g
e
Below Average
Band D
25%
SEN ??
50
5
60
70
80
Percentiles:
1
5
10
Above Average
Band C Band B
25%
25%
Band A
25%
G & T ??
90
100
110
120
Standardised Test Score
20
30 40 50 60 70
4%
80
90
130
95
140
99
150
Cohort Ability
Intake Profiles
Intake Profiles
Intake Profiles (Historical)
Student Ability
IPRs
Individual Pupil Record Sheets (IPRs)
Look for sections that
are inconsistent
Two students
Same Ability
Different Profiles
General IPR Patterns
•
•
•
Pupils with high scores across all components
Pupils with low scores across all components
Pupils with significant differences between one or two components
•
Vocab lower than others
•
Vocab higher than others
•
Maths higher than others
•
Maths lower than others
•
Non-Verbal higher than others
•
Non-Verbal lower than others
•
Low Skills
•
High Skills
www.cem.org/midyisiprbooklet
Vocab significantly lower than other sections
•
•
•
English Second Language ?
Understanding language used in learning and assessment ?
Language enrichment ?
Vocab significantly higher than other sections
•
•
•
Good Communicator ?
Work in class may not be to this standard
→ Weak Non-verbal
→ Weak Maths
→ Weak Skills (speed of working ?)
Many benefit from verbal descriptors ?
Maths significantly higher than other sections
•
•
•
•
Strong Maths ability
Not 100% curriculum free
May depend on prior teaching effectiveness
Far East influence ?
Maths significantly lower than other sections
•
•
•
Implications not just for maths but other numerate or databased subjects
General poor numeracy ?
Remedial Maths ?
Non-Verbal significantly higher than other sections
•
•
•
•
Good spatial and non-verbal ability
May have high specific skills
Low Vocab, Maths & Skills may indicate has difficulty
communicating
Frustration ?
Non-Verbal significantly lower than other sections
•
•
•
•
Difficulty understanding diagrams or graphical instructions ?
Verbal explanation ?
Physical demonstration ?
Physical Models ?
Low Skills Scores
•
•
•
•
•
Skills = Proof Reading and Perceptual Speed & Accuracy
Speed of Working
Work well in class / homework but underachieve in exams ?
Problems checking work or decoding questions ?
Low Skills + Low Vocab
→ Poor written work in class (unable to work quickly)
→ Dyslexia ? Further specialist assessment required
High Skills Scores
•
•
•
•
•
Skills = Proof Reading and Perceptual Speed & Accuracy
Can work quickly and accurately
Difficulty communicating and expressing ideas ?
May perform poorly in areas using numeracy skills and subjects
needing 3D visualisation and spatial concepts ?
May struggle in most areas of curriculum
Working with Individual
Pupil Records (IPRs)
Objectives
• To gain understanding of interpreting IPRs
• To share strategies for supporting individual pupils
Strategy
• To look first at interpretation of MidYIS IPRs
• Exercises with MidYIS IPRs
• Use generic patterns to apply to exercises on
IPRs with ALIS, Yellis and INSIGHT though there
are slight differences
What does the MidYIS test measure?
• Vocabulary
– Most culturally linked. Affects all subjects but most important in
English, History and some Foreign Languages. Measures fluency
rather than knowledge.
• Maths
– The Maths score is well correlated with most subjects but is
particularly important when predicting Maths, Statistics, ICT, Design
& Technology and Economics.
• Non-verbal
– Tests 3D visualisation, spatial aptitude, pattern recognition and
logical thinking. Important when predicting Maths, Science, Design &
Technology, Geography, Art and Drama.
• Skills
– Tests proof reading skills (SPG) and perceptual speed and accuracy
(e.g. matching symbols under time pressure). Measures fluency and
speed necessary in exams and in the work place. Relies on a pupil’s
scanning and skimming skills.
Using MidYIS IPRs to Inform Teaching and Learning
The IPR on its own simply tells us about the relative performances
of the pupil on the separate sections of the test, where the pupil
is strong, where performance has been significantly above or
below national averages or where the pupil has significantly
outperformed in one section or another.
It is when the IPR is placed in the hands of a teacher who knows that pupil
that it becomes a powerful tool.
It is what teachers know about individual pupils: what has
happened in the past, how they respond to
given situations and how they work in the teacher’s
specific subject that inform the interpretation of the IPR.
If the IPR data from MidYIS, the teacher’s personal and
subject specific knowledge and experiences regarding
the pupil can be shared, then there becomes a much more
powerful instrument for supporting pupils’ learning needs.
Examples of Individual Pupil Profiles
For each example look at
– the information contained in the graph
– the issues that may arise for this pupil in your
subject
– strategies you could employ to support that pupil
(either for the whole class or for that specific
individual)
Student A
Strategies for Student A
• Word banks for each topic
• Practise writing with words rather than symbols e.g.
To find the common denominator, first of all you ...
• Discussion groups (although ensure pupils with low
vocabulary scores do not all congregate)
• Wider reading
• Visits/trips etc. to enrich language and cultural
experience
Student B
Strategies for Student B
• May struggle to understand diagrams – use spoken
and written explanations, paired work or group work
to interpret
• Physical/practical/kinesthetic explanations may
help (e.g. modelling solar system with clay/string or
demonstrating distance between planets on football
pitch etc.)
• Use drama/active methods to demonstrate difficult
concepts
Student C
Strategies for Student C
• Pupil may seem more able than is the case, e.g.
‘talks a good talk’
• Allow paired work or group discussion to
communicate answers orally
• Describe maths problems
• Encourage leadership roles as well as
debates/drama
• Support with scaffolding/writing frames etc.
Student D
Strategies for Student D
Analysis
A pupil like this may:
• struggle to proof read his work, therefore achieve a lower grade
than he seems capable of
• struggle to interpret or understand exam questions
• either work slowly with more accuracy OR work quickly with less
accuracy – result is similar i.e. lower test score than expected
Strategies:
• allow extra time
• practise timing e.g. clock on IWB
• use a range of question words to develop ability to understand
instructions
• develop proof reading technique e.g. spotting comon errors
• consider further testing for dyslexia
Some pupil data MIDYIS
2009
Vocabulary
Maths
Non-Verbal
Skills
Overall
Score
Band
Score
Band
Score
Band
Score
Band
Score
Band
Pupil 01
122
A
125
A
116
A
107
B
126
A
Pupil 02
105
B
110
A
127
A
95
C
108
B
Pupil 03
105
B
93
C
110
B
89
D
99
C
Pupil 04
91
C
116
A
130
A
115
A
103
B
Pupil 05
111
A
144
A
122
A
103
B
129
A
Pupil 06
107
B
112
A
85
D
97
C
109
B
Pupil 07
115
A
106
B
100
C
86
D
112
A
Pupil 08
141
A
137
A
132
A
135
A
143
A
Pupil 09
104
B
92
C
105
B
109
B
98
C
Pupil 10
99
C
119
A
114
A
99
C
109
B
Pupil 11
108
B
126
A
130
A
140
A
118
A
Pupil 12
106
B
123
A
120
A
105
B
116
A
Pupil 13
103
B
96
C
103
B
104
B
99
C
Pupil 14
108
B
110
B
112
A
108
B
110
A
Pupil 15
95
C
104
B
103
B
122
A
99
C
The class from Waterloo Road
A useful quick reference for staff
A Selection Of MidYIS Scores For ‘Waterloo Road’
Why would this be a very challenging class to teach?
Vocabulary
Surname
Maths
Non Verbal
Skills
MidYIS Score
St. Score
Band
St. Score
Band
St. Score
Band
St. Score
Band
St. Score
Band
Sex
A
F
81
D
110
B
108
B
112
A
94
C
B
F
128
A
107
B
105
B
94
C
120
A
C
M
106
B
121
A
103
B
90
D
114
A
D
F
107
B
84
D
96
C
107
B
96
C
E
M
96
C
90
D
130
A
91
C
92
C
F
F
86
D
86
D
120
A
74
D
84
D
G
F
100
B
115
A
80
D
103
B
108
B
H
F
121
A
96
C
114
A
86
D
111
A
I
M
92
C
100
C
96
C
123
A
95
C
J
M
100
C
105
B
100
C
99
C
102
B
K
M
128
A
132
A
114
A
131
A
133
A
L
M
76
D
70
D
74
D
73
D
71
D
What do I need to know/do to teach this (difficult) class of twelve pupils?
These are real anonymous scores from a number of schools around the UK
IPR Patterns – A Summary
•
•
Vocabulary scores significantly lower than other component scores
Second language? Deprived areas? Difficulty accessing curriculum.? Targeted help does work. Seen in
nearly all schools. Worth further diagnosis. Could potentially affect performance in all subjects.
•
•
Vocabulary scores significantly higher than other component scores
Good communicators. Get on. Put Maths problems in words?
•
•
Mathematics significantly higher than other scores
From Far East? Done entrance tests? Primary experience?
•
•
Mathematics significantly lower than other scores
Primary experience. Use words and diagrams? Sometimes difficult to change attitude… Difficulties with
logical thinking and skills such as sequencing.
•
•
Low Mathematics scores with high Non-verbal Scores
Use diagrams. Confidence building often needed.
•
Pupils with non-verbal scores different from others (High) Frustration? Behaviour problems? Don’t
do as well as good communicators or numerate pupils? Good at 3D and 3D to 2D visualisation and
spatial awareness. Good at extracting information from visual images.
•
Pupils with non verbal scores different from others (Low) - Peak at GCSE? A level ?
•
Pupils with low Skills scores - Exams a difficulty after good coursework? Suggests slow speed of
processing.
•
High Skills Scores - Do well in exams compared with classwork?
•
The Average Pupil - They do exist!
•
High scores throughout - Above a score of 130 puts the pupil in the top 2% nationally
•
Low scores throughout - Below a score of 70 puts the pupil in the bottom 2% nationally
Interpreting IPRs Exercises
Have a look at the IPRs on the following pages.
These show examples for Yellis (Year 10) and
ALIS (Year 12) as well as MidYIS.
What do the scores suggest about the students
and how would you use this information to aid
the teaching and learning process for each of
them?
1
2
3
4
5
6
7
Proof-Reading 88
PSA 108
8
Yellis
Case Study 1
You are given data relating to an institution where students completed the ALIS computer
adaptive test. They are chosen because they show significant differences between the
various parts of the test. Remember scores are standardised around 100.
Name
Overall
Vocab
Maths
Non Verbal
Average
A Level subjects chosen
St.Score
Band
St.Score
Band
St.Score
Band
St.Score
Band
GCSE
A
78
D
49
D
99
B
92
C
na
Biology, Maths, Business, Art
B
94
C
115
A
85
D
104
B
na
Biology, Business, Psychology, English
C
88
D
97
C
85
D
104
B
5.6
History, Psychology, English, Media
D
101
B
107
B
97
C
80
D
5.9
Business, History, English, Drama
E
104
B
87
D
112
A
116
A
na
Biology, Physics, Maths, Business
F
81
D
47
D
103
B
111
B
na
Maths, Further Maths, Business
G
93
C
113
A
84
D
113
A
na
Biology, Business, French, Geography
H
97
C
111
A
89
D
99
C
7
Art, English, Psychology, Religious St.
I
87
D
68
D
100
B
109
B
5.4
Maths, Geography, French, Music
J
105
B
67
D
124
A
85
D
6.1
Maths, Further Maths, Psychology,
Economics
K
96
C
71
D
110
A
97
C
na
Biology, Maths, Art, English
L
92
C
60
D
111
A
97
C
na
Maths, History Religious St., English
a) Are there any apparent mismatches between the subjects being followed and this data?
b) What support can be given to those students who have weaknesses in Vocabulary or
Mathematics?
c) How might predictions made for these students be tempered in the light of the inconsistencies
in the test components and missing average GCSE points scores?
Case Study 2
What are the strengths and weaknesses of this A/AS level student?
To use the IPR (Individual pupil record) familiarise yourself with the terms
standard score, band, stanine, percentile and confidence band.
a) Which AS/A level subjects might be
avoided?
b) This student chose English, Film
Studies, Music Technology and
Psychology.
Is this a good choice?
Do you foresee any problems?
INSIGHT Pupil IPR
Speed Reading
Text Comprehension
Passage Comprehension
Overall Reading
Number & Algebra
Handling Data
Space, Shape & Measures
Overall Mathematics
Biology
Chemistry
Physics
Overall Science
Vocabulary
Non Verbal
Skills
Overall Ability
Comments?
Standard
KS3
Score
Equivalent
Band
Stanine
Percentile
B
B
6
5
69
60
107
104
6c
5a
C
5
45
98
5b
B
5
59
103
5a
D
3
19
87
4a
B
D
6
4
61
23
104
89
6a
5c
C
4
31
93
5b
A
A
A
7
9
8
82
96
89
114
127
118
6a
7a
7c
A
8
93
122
7b
B
B
A
6
6
8
65
70
92
106
108
121
C
4
36
94
Overall Ability
Skills
Non Verbal
40
Maths
60
Vocabulary
Overall Science
Physics
Chemistry
Biology
Overall Mathematics
Space, Shape & Measures
Handling Data
Number & Algebra
Overall Reading
Passage Comprehension
Text Comprehension
Speed Reading
Standardised Scores
Standardised Scores With 95% Confidence Band
160
140
120
100
80
Looking Forwards
Introduction to ‘Predictions’
Theory
How CEM ‘Predictions’ are made…
Subject
SubjectXX
Result
A*
A* / A
A
B
C C
D
E
‘Ability’
(Baseline)
Some Subjects are More Equal than Others….
A-Levels
140
A*
A
Grade
120
B
100
>1 grade
Photography
Sociology
English Lit
Psychology
Maths
Physics
Latin
80
C
60
D
E
40
5
C
6
B
A7
Average GCSE
A*8
Some Subjects are More Equal than Others …
Performance varies between subjects, thus analysing and
predicting each subject individually is essential.
e.g. Student with Average GCSE = 6.0
Subject Choices
Predicted Grades
Maths, Physics,
Chemistry, Biology
C, C/D, C/D, C/D
Sociology, RS, Drama,
Media
B, B/C, B/C, B/C
Some Subjects are More Equal than Others …
GCSE
A*
A
GCSE Grades
B
C
1 grade
D
E
F
Test Score
Art & Design
Biology
Chemistry
Economics
English
French
Geography
German
History
Ict
Mathematics
Media Studies
Music
Physical Education
Physics
Religious Studies
Science (Double)
Spanish
Feedback
Predictions – MidYIS example
5.0
C
B
6.0
4.4
C/D
4.8
C
3.9
D
Similar spreadsheets available from Yellis, INSIGHT
Adjusting Predictions in MidYIS / Yellis / INSIGHT
0.5
6.3
6.9
5.8
6.1
5.5
Chances Graphs
Individual Chances Graph for Student no.5 - GCSE English
MidYIS Score 82 MidYIS Band D
40
33
35
Prediction/expected
grade: 3.8 grade D
Percent
30
23
25
23
Most likely grade
20
15
10
10
5
2
5
3
1
0
A
A*
0
U
G
F
E
D
Grade
C
B
Post-16 : CABT vs Average GCSE
Average GCSE correlates very well to A-level / IB etc,
but by itself is not sufficient….
• What is a GCSE ?
• Students without GCSE ?
• Years out between GCSE & A-level ?
• Reliability of GCSE ?
• Prior Value-Added ?
The Effect of Prior Value Added
Beyond Expectation
In line with Expectation
Below Expectation
+ve Value-Added
0 Value-Added
-ve Value-Added
Average GCSE = 6
Average GCSE = 6
Do these 3 students all have the same ability ?
Average GCSE = 6
Rationale for CABT in addition to GCSE
•
•
•
•
•
Do students with the same GCSE score from feeder schools
with differing value-added have the same ability ?
How can you tell if a student has underachieved at GCSE and
thus can you maximise their potential ?
Has a student got very good GCSE scores through the school
effort rather than their ability alone ?
How will this affect expectation of attainment in the Sixth
Form ?
Can you add value at every Key Stage ?
Baseline testing provides a measure of ability that (to a large
extent) is independent of the effect of prior treatment.
‘Predictions’
Predictions Based
on GCSE
Probability of
achieving
each grade
Expected Grade
Predictions Based
on Baseline Test
Which predicted
grades are the most
appropriate for this
student ?
Step 1
Adjusting Predictions in ALIS (Paris Software)
75th
Percentile
Prior
Value-Added
Working with ‘Predictions’
(Average performance by similar pupils in previous years)
Objectives
• To gain understanding of the interpretation of ‘predictions’
• Remembering that they are not really PREDICTIONS but
part of a ‘chances scenario’
• Using chances to explore the setting of targets
• Discussion of monitoring performance against targets
A
6.8 7.2
2
F
7E 120
A
3
F
7C 110
4
F
7J
5
M
7D
1
7.2
7.5
7.0
6.3
6.7 6.4 6.1
6.4
6.5
6.3
B
5.8
6.2 5.7 5.4
5.5
5.6
5.5
101
B
5.4 5.8
5.1 4.8
4.9
4.9
4.9
82
D
4.5 4.9
3.8 3.5
3.4
3.2
3.5
M
7E 131
A
2
F
7E 120
A
3
F
7C 110
B
B
B
4
F
7J
101
B
B/C
B
C
5
M
7D
82
D
C/D
C
D
A
A
English
7.1 6.8
Biology
Science
Art & Design
7E 131
Mathematics
MidYIS Band
M
History
Form
1
French
Sex
Concentrate
on student 4
student no.
WHY ARE THE
SUBJECT
PREDICTIONS
DIFFERENT?
MidYIS Score
Point and grade ‘predictions’ to GCSE
A
A/B A/B A/B
A
B
A A*/A
A
A/B A/B A/B
B/C B/C B/C B/C B/C
C
C
D/E D/E
C
C
E
D/E
Individual Chances Graph for Student 4- GCSE English
MidYIS Score 101 MidYIS Band B
Teacher's Adjustment : 0 grades / levels / points
45
39
40
35
Percent
30
Prediction/expected grade:
5.1 grade C
27
25
20
19
Most likely grade
15
9
10
5
5
0
0
U
G
1
1
0
F
E
D
C
B
A
Grade
What are the chances
a) of getting a grade C or above ?
b) of not getting a C ?
A*
Yellis predictive data: baseline score 103 (55%)
Yellis
Average
Subject
Predicted
Business Studies
5.4 (B/C)
English
5.7 (B/C)
French
5.4 (B/C)
Geography
5.6 (B/C)
Mathematics
5.7 (B/C)
Physical Education
5.7 (B/C)
Science: GCSE
5.6 (B/C)
Science: GCSE Additional 5.6 (B/C)
SC Religious Studies
5.3 (B/C)
Yellis
Average
Subject
Predicted
Business Studies
5.6 (B/C)*
English
5.9 (B)*
French
5.6 (B/C)*
Geography
5.8 (B)*
Mathematics
5.9 (B)*
Physical Education
6.0 (B)*
Science: GCSE
5.8 (B)*
Science: GCSE Additional 5.8 (B)*
SC Religious Studies
5.6 (B/C)*
Weighted Average
Weighted Average
Comment?
5.6 (B/C)
Yellis
Average
Subject
Predicted
Business Studies
5.4 (B/C)
English
5.7 (B/C)*
French
5.4 (B/C)
Geography
5.8 (B)*
Mathematics
5.9 (B)*
Physical Education
6.7 (A/B)*
Science: GCSE
6.3 (A/B)*
Science: GCSE Additional 5.6 (B/C)
SC Religious Studies
5.7 (B/C)*
Weighted Average
5.8 (B)
5.8 (B)
Chances graphs MidYIS and Yellis
Situation
You are a tutor to a Year 10 pupil and you wish to help him/her to set target
grades. Here is a chances graph based on the pupil’s Year 7 MidYIS test (114)
and one based on the Year 10 Yellis test (58%)
MidYIS Chances Graph
This graph is based on the pupil’s exact MidYIS score, adjusted to include
the school’s previous value-added performance.
Yellis Chances Graph
This graph is based on one ability band and has no
value-added adjustment.
a) What do the graphs tell you about this pupil’s GCSE chances in this subject
(Maths)?
b) What could account for the differences between the two graphs and are these
important?
IMPORTANT FOR STAFF AND STUDENTS TO UNDERSTAND THE DIFFERENCE
Fixed Mindset:
[My intelligence is fixed and tests tell me how clever I am.]
This graph tells me I’m going to get a B, but I thought I was going to get an A. I’m
obviously not as clever as I hoped I was and so the A and A* grades I’ve got for my
work so far can’t really be true.
Growth Mindset:
[My intelligence can develop and tests tell me how far I have got.]
This tells me that most people with the same MidYIS score as me achieved a B last
year, but I think I have a good chance of an A and I know that my work has been
about that level so far so I must be doing well. What do I need to do to be one of the
10% who gets an A*?
How was this information produced?
The MidYIS graphs are produced using the predictions spreadsheet. Select the pupil(s) and subject(s) to display or print using the GCSE
Pupil Summary 1 tab. Adjustments for value-added can be made for individual subjects on the GCSE Preds tab.
The Yellis graphs for all GCSE subjects (showing all four ability bands) can be downloaded from the Secondary+ website.
Commentary
From MidYIS - The most likely grade is a B (35%) but remember there is a 65%
(100-65) chance of getting a different grade but also a 75% (35+30+10) chance
of the top three grades.
From Yellis - The most likely grade appears to be a C but remember that the
band has been decided over a range, not for the individual student and this
pupils score is near the top of that range, 58 compared with 60.8. It has also
not been adjusted for this school’s prior value added in the past.
In an interview with the student one has to use your professional judgement
about that student, taking everything into account. Certainly the Yellis chart
warns against complacency, but if the school has a strong value added history
it is better to rely in this case on the MidYIS chart for negotiating a target. Grade
A is a fair aspirational target for the student but accountability for a teacher
cannot fairly be judged by not achieving this grade with this student. Even a
very good teacher may only achieve B or C with this student.
Can the aspirational target set for the student be the same as that used for staff
accountability purposes? There is a trap here.
ALIS
You are the subject teacher and are discussing possible A2 target grades
with individual students. You are about to talk to Jonathan who achieved
an average GCSE score of 6.22. This gives a statistical
prediction=28.35x6.22-99.57= 77 UCAS points using the regression
formula at A2 for this subject (Grade C at A2). Assume that the computer
adaptive baseline test confirms this prediction. Chances graphs for this
subject are shown showing the percentage of students with similar profiles
achieving the various grades.
Individual chances graph for Jonathan
a) Why are these two chances graphs different?
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
(b) ‘Most candidates with Jonathan’s GCSE background score achieved a C in my
subject last year so Jonathan’s target grade should be a C’.
What are the weaknesses of this statement?
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------(c) What other factors should be taken into consideration apart from chances graph
data, when determining a target grade?
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The difference in the chances graphs is that one of them provides for a range of
GCSE scores whilst the other is linked to Jonathan’s individual average GCSE
score of 6.22. The strength of the chances graph is that it shows more than a bald
prediction.
True, most students starting from an average GCSE score like Jonathan did
achieve a C grade at A2 in examinations for this subject. However the probability of
a B grade is also high since his score was not at the bottom of this range. This
might be reflected too if the department also has a history of high prior value
added. The converse is also true with a D grade probability warning against
complacency. Students are not robots who will always fit with statistics so it is
dangerous to make sweeping statements based on one set of results.
As well as looking at the prediction you should use the chances graph as a starting
point, with your professional judgement taking into account factors such as his and
the department’s previous performance in the subject, his attitude to work, what he
is likely to achieve based on your own experience. You might want to start with the
most popular outcome grade C and use your judgement to decide how far up (or
down!) to go. He may be a very committed student and if the department has
achieved high value added in the past, an A/B grade may be more appropriate
though A* looks unlikely. If you are using aspirational targets for psychological
reasons with students then A may be appropriate even though it less probable than
B/C.
Key Questions for Intelligent Target Setting
• What type of valid and reliable predictive data
should be used to set the targets?
• Should students be involved as part of the
process (ownership, empowerment etc.)?
• Should parents be informed of the process and
outcome?
Key points to consider might include:
• Where has the data come from?
• What (reliable and relevant) data should we
use?
• Enabling colleagues to trust the data: Training
(staff)
• Communication with parents and students
• Challenging, NOT Demoralising, students…….
• Storage and retrieval of data
• Consistency of understanding what the data
means and does not mean
• The process of setting targets is crucial…….
There is wide-ranging practice using CEM data to set student,
department and institution targets.
Increasingly sophisticated methods are used by schools and colleges.
The simplest model is to use the student grade predictions. These
then become the targets against which student progress and
achievement can be monitored.
Theoretically, if these targets were to be met, residuals would be zero so
overall progress would be average.
The school/college would be at the 50th percentile.
More challenging targets would be those based on the basis of history.
For example. Where is the school/college now? Where is your subject
now?
If your subject value added history shows that performance is in the
upper quartile it may be sensible to adjust targets. This may have the
effect of raising point predictions between 0.2-0.5 of a grade.
This would be a useful starting point, but it would not be advisable to use
the predictions for below average subjects, which might lead to
continuing under-achievement.
Yellis Predictions For Modelling
FOUR approaches
• YELLIS GCSE Predictions
• YELLIS GCSE Predictions + say 0.5 a grade
• Prior value added analysis based on 3 year VA per
department
•
75th percentile (upper quartile) analysis
Subject
Art & Design
Business Studies
Design & Technology
Drama
English
English Literature
French
Geography
German
History
Home Economics
ICT
Maths
Music
Physical Education
Religious Studies
Double Science
Welsh
Number of
Students
Percentage
of A* to C
Grades
Percentage
of A* to G
Grades
68
64
103
27
181
15
53
84
7
49
48
71
180
12
72
37
180
177
84
48
63
85
64
60
64
63
71
67
48
68
54
67
65
70
52
72
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
School Average GCSE score:
Average
Grade
5.2
4.3
4.7
5.3
4.8
4.6
4.9
4.8
5.1
5.1
4.5
4.9
4.5
5.2
4.9
5.2
4.4
5.1
(C)
(C/D)
(C/D)
(B/C)
(C)
(C/D)
(C)
(C)
(C)
(C)
(C/D)
(C)
(C/D)
(C)
(C)
(C)
(C/D)
(C)
4.7 (C/D)
Counted Performance Statistics (Based on Subject Choice Predictions)
5 or more A* to C Grades:
106
1 or more A* to C Grades:
141
5 or more A* to G Grades:
181
1 or more A* to G Grades:
181
58%
77%
99%
99%
5 or more A* to C Grades inc Maths and English:
2 or more A* to C Grades - Sciences:
1 or more A* to C Grades - Modern Foreign Language:
54%
51%
20%
98
93
36
The underlying predictions summarised here are based on expectations for an average school achieving
zero value added results. Appropriate care should be taken in interpreting them within your school.
Please note that the cut-off points for grade C and grade G have been set at 4.5 and 0.5 respectively.
Due to the sensitive nature of the cut off points, predictions may vary for your school if the cut off points
could be altered.
(*Predictions Adjusted for Positive Prior Value-added Performance)
Subject
Art & Design
Business Studies
Design & Technology
Drama
English
English Literature
French
Geography
German
History
Home Economics
ICT
Maths
Music
Physical Education
Religious Studies
Double Science
Welsh
Number of
Students
Percentage
of A* to C
Grades
Percentage
of A* to G
Grades
68
64
103
27
181
15
53
84
7
49
48
71
180
12
72
37
180
177
84
48
87
100
69
67
96
73
86
67
79
96
57
92
65
70
59
86
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
School Average GCSE score:
Average
Grade
5.2
4.3
5.3
6.0
4.9
4.9
6.4
5.2
5.6
5.1
5.2
5.7
4.6
5.7
4.9
5.3
4.7
5.5
(C)
(C/D)
(B/C)*
(B)*
(C)*
(C)*
(A/B)*
(C)*
(B/C)*
(C)
(C)*
(B/C)*
(C/D)*
(B/C)*
(C)
(B/C)*
(C/D)*
(B/C)*
5.1 (C)
Counted Performance Statistics (Based on Subject Choice Predictions)
5 or more A* to C Grades:
125
1 or more A* to C Grades:
162
5 or more A* to G Grades:
181
1 or more A* to G Grades:
181
69%
89%
99%
99%
5 or more A* to C Grades inc Maths and English:
2 or more A* to C Grades - Sciences:
1 or more A* to C Grades - Modern Foreign Language:
56% *
58% *
30% *
102
106
54
*
*
*
*
(*Predictions Adjusted for 75th Percentile)
Subject
Art & Design
Business Studies
Design & Technology
Drama
English
English Literature
French
Geography
German
History
Home Economics
ICT
Maths
Music
Physical Education
Religious Studies
Double Science
Welsh
Number of
Students
Percentage
of A* to C
Grades
Percentage
of A* to G
Grades
68
64
103
27
181
15
53
84
7
49
48
71
180
12
72
37
180
177
97
63
73
96
70
67
74
70
71
84
63
77
61
83
72
81
59
82
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
100
School Average GCSE score:
Average
Grade
5.5
4.6
5.0
5.5
5.0
4.9
5.1
5.1
5.4
5.4
4.8
5.2
4.8
5.5
5.2
5.5
4.7
5.4
(B/C)*
(C/D)*
(C)*
(B/C)*
(C)*
(C)*
(C)*
(C)*
(B/C)*
(B/C)*
(C)*
(C)*
(C)*
(B/C)*
(C)*
(B/C)*
(C/D)*
(B/C)*
5.0 (C)
Counted Performance Statistics (Based on Subject Choice Predictions)
5 or more A* to C Grades:
123
1 or more A* to C Grades:
162
5 or more A* to G Grades:
181
1 or more A* to G Grades:
181
68%
89%
99%
99%
5 or more A* to C Grades inc Maths and English:
2 or more A* to C Grades - Sciences:
1 or more A* to C Grades - Modern Foreign Language:
60% *
58% *
23% *
109
106
41
*
*
*
*
Case Study
Here is the Individual Pupil Record
from the ALIS computer adaptive
test taken in Year 12 for a current
Year 13 student.
This student had a high positive
value added in every GCSE subject
as measured using MidYIS as a
baseline.
( Average GCSE score 7.44)
On the next page are her A level
predictions and chances graphs.
Why are the predictions different?
Are the chances graphs useful
here?
Predictions and chances graphs
Using PARIS software and
tweaking the predictions for
prior value added by these
subjects, then from a GCSE
baseline A*s are predicted in
three of the four.
If we did the same for the
adaptive test baseline solid Bs
might be predicted in all three.
It is also worth looking at
the value added at GCSE.
See commentary
Commentary
The differences in prediction from the GCSE baseline and the computer adaptive
test for some students are interesting and these can be in either direction. Here
there has been a very large value added at GCSE which may or may not be
sustainable at A level. This student’s history is shown below:
GCSE PREDICTIONS MIDYIS ALL
GSCE Grade predictions
GCSE ACHIEVED
VALUE ADDED RAW
Drama
5.8
BA*
2.2
English
6
B
A*
2
GCSE PREDICTIONS MIDYIS IND.
GSCE Grade predictions
GCSE ACHIEVED
VALUE ADDED RAW IND
6.6
A/B
A*
1.4
6.8
AA*
1.2
English Lit German
6
5.5
B
B/C
A
A
1
1.5
6.7
AA
0.3
6.5
A/B
A
0.5
Latin
6.5
A/B
A*
1.5
Maths
6
B
A*
2
Music
5.9
BA
1.1
6.9
AA*
1.1
6.9
AA*
1.1
6.9
AA
0.1
Science
5.8
BA
1.2
6.9
AA
0.1
from year7 data
from year9 data
Average GCSE score =7.44
The value added here at GCSE is between 1 and 2 grades (for all institution data at year
7) and significantly positive for subjects (for the Independent school data from year 9).
Actually if we measure this student’s value added from an average GCSE score of 7.44
next year, it does not tell the whole story. We need to look as well at the value added from
the computer adaptive test too.
The chances graphs should be used with extreme caution here and the growth mindset is
vital if used with students.
Case study : setting departmental targets
•
•
•
•
Uses valid and reliable data e.g. chances graphs
Involves sharing data with the students
Gives ownership of the learning to the student
Enables a shared responsibility between student,
parent(s)/guardian, and the teacher
• Encourages professional judgement
• Leads to the teachers working smarter and not harder
• Leads to students being challenged and not ‘over
supported’, thus becoming independent learners…
DEPARTMENT:
GCSE ANALYSIS
year
2006
2007
2008
2009
no. of
pupils
66
88
92
108
av. Std.
raw resid. Resid
0.8
0.8
1.1
0.7
0.6
0.5
0.8
0.6
n.b. A raw residual of 1.0 is equivalent to one grade.
TARGETS FOR 2011, using CEM predictive data and dept's prior value-added
The target grade has a prior value-added of 0.8
1
2
3
4
5
6
7
8
9
10
12
M
F
M
F
M
F
M
M
F
M
M
prediction
5.4
3.8
3.6
4.2
5.7
6.5
7.0
3.8
4.2
5.9
3.8
pred
grade
(B/C)
(D)
(D/E)
(D)
(B/C)
(A/B)
(A)
(D)
(D)
(B)
(D)
target
6.2
4.6
4.4
5.0
6.5
7.3
7.8
4.6
5.0
6.7
4.6
target
grade
B
C
D
C
B
A
A*
C
C
A
C
dept adj
grade
A
C
D
D
B
A*
A*
C
C
B
D
etc.
Student no.1 GCSE Geography
Individual Chances Graph for student A- GCSE English
MidYIS Score 105 MidYIS Band B
Teacher's Adjustment : 0 grades / levels / points
45
40
36
35
Percent
30
32
Prediction/expected grade: 5.4
grade B/C
25
20
Most likely grade
14
15
14
10
5
0
0
0
U
G
F
3
2
0
E
D
Grade
C
B
A
A*
Student no.1 GCSE Geography
Individual Chances Graph for Student A- GCSE English
MidYIS Score 105 MidYIS Band B
Teacher's Adjustment : 0.8 grades / levels / points
45
40
36
35
Percent
30
25
Prediction/expected grade: 6.2
grade B
Most likely grade
32
20
20
15
9
10
4
5
0
0
0
0
U
G
F
E
0
D
Grade
C
B
A
A*
Results
13
19
23
21
10
6
COMMENTS?
Monitoring Student Progress
Monitoring students’ work against target grades is established practice in
schools and colleges, and there are many diverse monitoring systems in place
Simple monitoring systems can be very effective
Current student achievement compared to the target grade done at
predetermined regular intervals to coincide with, for example internal
assessments/examinations
Designated staff having an overview of each student’s achievements across
subjects
All parents being informed of progress compared to targets
Review of progress between parents and staff
Subject progress being monitored by a member of the management team in
conjunction with the head of subject/department
A tracking system to show progress over time for subjects and students
Monitoring Progress:
Schools and departments use various monitoring systems
for comparing present progress with either the target grade
or in some cases the minimum acceptable grade or basic
suggested grade.
Six examples from schools are shown.
If you were Polly Bolton’s Form Teacher, how would you
approach a discussion with her parents at a Parents’
Evening? Should parents be told the baseline scores?
Polly Bolton
B
SECURE
5
ORG
MC
E
B
UNLIKELY
3
Science
CPa
E
C
UNLIKELY
Science Additional
CPa
D
C
French
CK
C
History
KM
RS
CG
CB
Maths
B
SECURE
5
C
B
LIKELY
5
HW
D
B
POSSIBLE
4
4
ORG
D
C
POSSIBLE
4
ORG
LIKELY
4
ORG
B
C
LIKELY
4
ORG
B
LIKELY
4
C
B
LIKELY
5
A
A
SECURE
5
B
A
SECURE
5
D
B
POSSIBLE
4
C
B
POSSIBLE
4
KEY - target is:
SECURE
LIKELY
POSSIBLE
UNLIKELY
effort: 5 excellent - 4 good - 3 satisfactory - 2 poor - 1 very poor
concern: WW working well - ATT attitude - BEH behaviour
TEN attendance - PUN punctuality - HW homework
CON confidence - ORG organisation - EAL language
concern
B
English Literature
B
is:
effort
ORG
target
grade
5
year 10
exam
LIKELY
concern
concern
B
CB
is:
effort
effort
C
English
target
grade
is:
Teach
Subject
SUMMER
current
grade
target
grade
SPRING
current
grade
AUTUMN
Subjects
Tracking at departmental level for one student
Student: Peter Hendry
test: Geol
Time Scale
target
grade
test essay:
radiometric
dating
test: dating
homework
rock cycle
pract: rock
textures
2006-8
test:
igneous
rocks
15/09/2006 22/09/2006 06/10/2006 20/10/2006 06/11/2006 21/11/2006
97%
A
84%
68%
B
C
Department: Geology
57%
54%
D
E
U
50%
SURNAME
Briggs
Fletcher
Green
Havard
etc
punctuality
meeting deadlines
D
B
A
A
1
2
1
3
2
2
1
3
1
2
2
4
C
B
B
B
1
2
2
4
1
1
2
2
1
1
2
2
meeting deadlines
punctuality
DEC
effort
OCT
current level
effort
C
B
B
A
current level
C
A
C
A
meeting deadlines
Alice
Kevin
Felicity
Michael
punctuality
yr 12
effort
BIOLOGY
current level
FORENAME
negociated target
grade
subject:
initial target grade
Traditional mark book approach
07-08
MAR
Targets
for learning…. reporting to pupils
-7.07013
0.59938
A
B
C
D
E
F
g
h
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y
Z
ZA
ZB
MidYis Score Test Score
80
96
95
119
111
84
67
88
118
91
120
108
115
87
117
105
98
69
69
115
118
109
123
89
115
76
90
97
33
63
80
80
73
45
45
63
50
60
50
35
35
58
83
45
73
5
30
70
50
45
60
30
65
10
55
70
41.12001 49.34402
50.17065 60.20478
49.87096 59.84515
64.1362 76.96344
59.46104 71.35324
43.33772 52.00526
33.02838 39.63406
100 55.02614
45.85511
63.83651 76.60381
90 56.96813
47.47344
64.79552 77.75462
80 69.12355
57.60296
62.09831 74.51797
45.31567
70 54.37881
62.99738 75.59685
55.80482
60 66.96578
51.54922 61.85907
34.4669
50 41.36028
34.10727 40.92872
61.91849
40 74.30219
63.71663 76.45996
58.32222
30 69.98666
66.47378 79.76854
46.03493
20 55.24191
61.55887 73.87064
38.48274
10 46.17929
46.57437 55.88924
50.88990 61.06789
-7.07013 -8.484156
60
-7.07013 -8.484156
-7.07013 -8.484156
-7.07013 -8.484156
-7.07013 -8.484156
-7.07013 -8.484156
Test Score
Name
32.89601
40.13652
39.89677
51.30896
47.56883
34.67017
26.42271
36.68409
51.06921
37.97875
51.83641
46.08237
49.67865
36.25254
50.3979
44.64386
41.23938
27.57352
27.28581
49.5348
50.97331
46.65777
53.17903
36.82794
49.24709
30.78619
37.2595
40.71192
-5.656104
70
-5.656104
-5.656104
-5.656104
-5.656104
-5.656104
Astronomy 7N
MidYis Test Review
80
90
100
MidYis Score
110
120
130
J
M 97.3
C
F
71.8
101 A
MIDYIS ON ENTRY
99 B
132 131 127 105
94 5
4
5
-2.2 6
5
6
6
-2.5 5
6
6
5
-3
101
86 6
4
5
-0.1 5
4
3
4
-2 5
5
5
4
-1.8
83 116
KEY STAGE 3 STATUTORY TEACHER ASSESSMENT
94
92 113
SOSCA STANDARDISED SCORES
98
96
83 102
98
90 103
87
97
95
98
83
95
88
SOSCA Maths
SOSCA St.ScoreSPACE.Maths
SOSCA St.ScoreNUMBERMaths
SOSCA St.ScoreH.DATAMaths
SOSCA St. Score Physics
SOSCA St. Score Chemistry
SOSCA St. Score Biology
SOSCA (STA.) Reading
St. res. MIDYIS- KS3 Sc
SC TA Science Subject Wa
PE TA Phys Ed Subject Wa
MU TA Music Subject Wa
MF TA MFL Subject Wa
St. res. MIDYIS- KS3 Ma
MA TA Maths Subject Wa
IC TA Inf Tech Sub Wa
HI TA History Subject Wa
GE TA Geography Sub Wa
St. res. MIDYIS- KS3 En
EN TA English Subject Wa
DA TA Des and Tech Sub Wa
AR TA Art Subject Wa
MidYIS Skills Standardise
MidYIS Non Verbal Standar
MidYIS Vocabulary Standar
MidYIS Maths Standardised
MidYIS Overall Standardis
MidYIS Overall Band Year
LONDON READING
% Attendance Y10
Gender
Surname Forename
Not a label for life
... just another piece of information
• The Chances graphs show that, from almost any baseline
score, students come up with almost any grade - - - there
are just different probabilities for each grade depending
on the baseline score
• In working with students these graphs are more useful than
a single predicted or target grade
• Chances graphs show what can be achieved:
– By students of similar ability
– By students with lower baseline scores
Student 1
Student 2
Student 3
Student 4
Student 4 - IPR
Performance Monitoring
Introduction to Value-Added
Theory
How CEM ‘Value-Added’ is calculated…
Result
Subject X
-ve VA
+ve VA
Residuals
VA
Ability
(Baseline)
Burning Question :
What is my Value-Added Score ?
Better Question :
Is it Important ?
Key Value Added Charts
1) SPC (Statistical Process Control) chart
VA Score
Performance above expectation
Good Practice to Share ?
Performance in line
with expectation
Performance below expectation
Problem with Teaching & Learning ?
2000
2001
2002
2003
2004
2005
Year
2006
2007
2008
2009
2010
4
3
0.0
0.1
0.0
0.1
0.1
-0.3
-0.3
0.7
0.4
0.2
0.0
0.0
-0.4
-0.3
-1
0.2
0.2
0.5
1.0
0.5
1
0.8
1.1
2
0
-2
-2.9
-3
-4
Short Course Religious
Studies
Spanish
Science
Religious Studies
Physics
Physical Education
Music
Mathematics
History
German
Geography
French
English Literature
English
Drama
Design & Technology
Chemistry
Business Studies
Biology
Art & Design
Additional Science
Additional Applied Science
Average Standardised Residual
2) Subject Bar Chart
Average Standardised Residuals by Subject
3) Scatter Plot
Religious Studies
Scatter Plot Example 1
A2 – English Literature
General Underachievement ?
Scatter Plot Example 2
A2 – English Literature
Too many U’s ?
Other things to look for…
Why did these students do so badly ?
Why did this student do so well ?
How did they do in their other subjects ?
Post-16 : Impact of Baseline
Choice on Value-Added
GCSE as
Baseline
Same School - Spot the Difference ?
Test as
Baseline
Does the Type of School
make a Difference ?
Comparison to all schools
Comparison to Independent Schools Only
Comparison to FE Colleges Only
Comparison to all schools
Questions:
→
→
How does the unit of comparison used
affect the Value-Added data and what
implications does this have on your
understanding of performance ?
Does this have implications for Self
Evaluation ?
Using Value-Added Data
Necessary knowledge base to use CEM systems to their potential
1.
•
•
•
•
•
The forms of Value Added Data:
scatter graphs
raw and standardised residuals
SPC charts
tables of data
use of PARIS for further analyses (e.g. by gender, teaching group)
2.
•
•
•
Predictive Data:
point and grade predictions
importance of chances graphs
availability of different predictive data
3. Baseline Data
• band profile graphs
• IPRs
• average GCSE score
• computer adaptive tests
4. Attitudinal Data
If you have the tools you can use them to do these:
•
Make curriculum changes
•
Adjust staffing structure and cater for student needs
•
Self-evaluation procedures including the analysis of examination results using
value added data
•
The target setting process
•
School and department development plans…….
•
Improve your monitoring and reporting procedures
•
Provide information to governors
•
Have conversations with feeder primary schools
•
Etc. etc.
Below are the value added charts from Yellis to GCSE for two contrasting
institutions. Which subjects are outside the confidence limits in a ‘negative value
added’ way? There must be questions to ask regarding teaching and learning?
Which subjects are outside the confidence limits in a ‘positive value added’ way?
Any result within the outer shaded area
decreases the probability that the value
added result is down to chance. The
probability here is about 1 in 20.
Outside the 99.7% confidence limit
chance is less than 3 in a 1000.
GCSE value added A challenging school
GCSE value added A successful school
2.0
3.0
0.3
0.4
Double Science
Welsh
0.0
-0.1
-0.1
-0.2
Religious Studies
-1.0
-2.0
-3.0
-4.0
SC ICT
Voc Health & Social Care
Voc Applied Science
SC History
SC Religious Studies
SC Geography
SC Art & Design
Science: GCSE Additional
Science: GCSE
Religious Studies
Music
Media Studies
Mathematics
History
Geography
French
English Literature
English
Design & Technology
-4.0
0.2
Physical Education
-3.0
0.1
0.0
0.9
0.8
0.6
Music
-2.0
0.0
0.5
0.3
ICT
-1.3
-0.5
-0.9
0.4
0.2
Maths
-0.5
0.4
Home Economics
-0.5
-1.1
1.0
History
-0.3
German
-0.5
-0.1
Geography
-0.6
-1.0
-0.1
-0.3
French
-0.2
1.4
0.9
English
0.0
2.0
English Literature
0.3
0.1
Drama
0.1
Design & Technology
0.1
Art & Design
0.4
0.2
Business Studies
1.0
Average Standardised Residual
3.0
4.0
Art & Design
Average Standardised Residual
4.0
Here is a value-added subject report from a recent examination session at
a school
G
C
S
E
s
c
o
r
e
MidYIS score
Write the equivalent GCSE grades next to the points scores.
Compare the value-added performance of candidates scoring A*, B, and D grades.
Which result would cause you to ask questions?
Compare the data for Student A and Student B
Student A
Sex
Female
Weight
1
1
1
1
1
1
1
2
1
Subject
Drama
English
English Literature
French
Geography
Maths
Physical Education
Double Science
Welsh
Weighted Average
Student B
Sex
Male
Weight
1
1
1
1
1
1
1
2
1
Subject
Business Studies
Drama
English
English Literature
Geography
Maths
Physical Education
Double Science
Welsh
Weighted Average
Band
B
Maths
46
YELLIS
Predicted
5.6 (B/C)
5.4 (B/C)
5.4 (B/C)
4.9 (C)
5.3 (B/C)
5.2 (C)
5.4 (B/C)
5.1 (C)
5.6 (B/C)
5.3
(B/C)
Band
A
7.4
(A*/A)
Maths
55
YELLIS Predicted
5.7 (B/C)
6.1 (B)
6.0 (B)
6.0 (B)
6.0 (B)
6.0 (B)
6.0 (B)
5.8 (B)
6.1 (B)
6.0
Achieved
Grade
8 (A*)
7 (A)
7 (A)
8 (A*)
8 (A*)
7 (A)
6 (B)
8 (A*)
7 (A)
(B)
Achieved Grade
4 (D)
6 (B)
5 (C)
5 (C)
3 (E)
6 (B)
2 (F)
5 (C)
5 (C)
4.6
(C/D)
Vocab
61
Pattern
53
Raw
Residual
2.4
1.6
1.6
3.1
2.7
1.8
0.6
2.9
1.4
Standardised
Residual
1.8
1.6
1.4
2.5
2.2
1.7
0.4
2.6
1.0
2.1
1.8
YELLIS
54
Vocab
71
Pattern
76
Raw
Residual
-1.7
-0.1
-1.0
-1.0
-3.0
0.0
-4.0
-0.8
-1.1
Standardised
Residual
-1.4
0.0
-1.0
-0.9
-2.4
0.0
-3.1
-0.7
-0.8
-1.4
-1.1
YELLIS
63
Find students A and B on each of the scatter graphs English and Maths
Scatter graph English
A*
8
A
7
C
5
D
4
E
3
F
2
Scatter graph Maths
G
1
U
0
A*
8
0
10
20
30
40
50
60
70
80
90
100
YELLIS Test Score (% )
A
7
B
6
GCSE Grade
GCSE Grade
B
6
C
5
D
4
E
3
F
2
Is there anything to learn from
these scatter graphs?
G
1
U
0
0
10
20
30
40
50
60
YELLIS Test Score (% )
70
80
90
100
Common Scenario…
Jane has completed her 6th form studies and a review has been
received for her by the college after A level results. Choose one
subject at a time and look carefully at what happened in that subject
both from a baseline of average GCSE grades and from a baseline of
the computer adaptive test.
This student has been placed in different bands, band B from average GCSE score and
band C from the computer adaptive test. This sometimes happens. It may be that the
student had an off day when she did the computer adaptive test or it may be that there
could have been a lot of ‘spoon feeding’ at GCSE. Jane may do better at coursework!
Even though we may not know the cause it can act as a warning when analysing results
though the predictions are not wildly out.
Profile Sheet: Jane (from Average GCSE)
Year: 2007
DOB: 01/06/89
(Average GCSE = 6.00 (Band B))
FINAL RESULTS
PREDICTIONS
Review: Final_Result
Review Review
Subject
Points Grade
(A1) Health & Social Care30.00
D
(A2) Religious Studies
80.00
C
(A2) English Literature
80.00
C
(A2) Drama & Theatre St100.00 B
Average Average
Points Grade Residual Std. Residual
39.64
C
-9.64
-0.66
89.84
B/C
-9.84
-0.52
87.29
B/C
-7.29
-0.40
91.84
B/C
8.16
0.46
STANDARDISED
RAW
Chances Graphs - Band B from average GCSE
Individual Chances Graphs for Jane from average GCSE score
Profile Sheet: Jane (from Computer adaptive test)
Year: 2007
DOB: 01/06/89
(Online adaptive test = 0.11 (Band C))
FINAL RESULTS
PREDICTIONS
Review
Subject
Points Grade
(A1)Health & Social Care 30.00
D
(A2) Religious Studies
80.00
C
(A2) English Literature
80.00
C
(A2) Drama & Theatre St 100.00 B
Average Average
Points
Grade Residual St
25.97
D/E
4.03
0.24
79.01
C
0.99
0.04
74.3
C/D
5.7
0.26
85.16
B/C
14.84
0.70
RAW
STANDARDISED
a) Did Jane reach her potential in all subjects?
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------b) Jane had been set aspirational targets prior to AS and A level by her teachers
as below:
Health and Social Care C
Drama
B
Religious Studies
B
English
B
Were these reasonable target grades?
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
c) Why should these grades not be used for accountability of her teachers?
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Commentary
a) Jane certainly reached her potential in Drama and Theatre Studies
with positive standardised residuals by both methods. On the basis
of the computer adaptive test she broadly reached potential in all
subjects. On the basis of average GCSE two A level subjects were
broadly down about half a grade and she dropped a grade in the AS.
b) Hopefully you agree these were reasonable target grades.
Remember we don’t know the student, but use the chances graphs
and if these were aspirational grades for the student then
accountability of a department’s staff on that basis is not appropriate,
but it certainly is on the basis of a whole class’s average
standardised residuals, particularly over a number of years.
YELLIS ATTITUDINAL
You are looking at the attitudinal feedback from Year 10 over time:
Do you notice a pattern between the four charts? There was an initiative in
the school that contributed, but was it sustainable?
Yellis Further Comparison charts for English and Maths
What concerns you about these charts? Can you suggest which is the stronger
department?
Departmental analysis (based on average GCSE score baseline)
SUBJECT A (A LEVEL)
Exam Year
-1.0
-2.0
Exam Year
2011
-1.0
2010
0.0
2009
0.0
2008
1.0
2007
1.0
2006
2.0
2005
-2.0
Final_Result
2.0
2004
-2.0
2011
-1.0
2010
-1.0
2009
0.0
2008
0.0
2007
1.0
2006
1.0
2005
2.0
2004
2.0
AS LEVEL
2003
Final_Result
Average Standardised Residual
A LEVEL
2003
Average Standardised Residual
There may be lots of reasons for changes in performance but here one factor is
known by the school. The subject teacher for subject A goes on long sick leave for
one of the autumn terms. When do you think this happened? To help you the AS
chart is shown below for this same subject. It may or may not be relevant.
-2.0
This school has a 5-year development plan which includes as one of its goals:
“To help students prepare for university and the world of work by developing
independent learning skills, the ability to reflect and to learn from others and to
maximise the benefits to learning offered by emerging technologies.”
The graphs on the next page reflect students’ perceptions of the style of
learning that has been adopted in their A-level classes in two broadly similar
subjects.
a) If the students’ perceptions are an accurate reflection of what takes place in
the classroom, which subject seems more on board with the school’s
development plan?
b) How would these perceptions inform the Senior Management Team’s
evaluation of progress with its 5-year plan if the subject achieving significantly
better value-added results was
i) Subject 1?
ii) Subject 2?
Subject 1
Subject 2
Case study A: ALIS value-added data
Many sets of VAD are available!
From average GCSE baseline:
• all ALIS cohort
• type of Institution
• syllabus
Also the same from the baseline test
SPC Chart with confidence limits: WHOLE SCHOOL
All ALIS Cohort
Syllabus
Institution
Using PARIS software: Baseline Test
Whole School
From your perspective, which set of VAD would you use for the
different user groups? (Governors, HoDs, Media, Parents, SMT/SLT...)
•
USE ONE YEAR’S DATA WITH CAUTION!
•
Better to use three years’ data, as patterns over time
are more significant.
Using data to inform leadership decisions
Some key questions:
1. Which data do I need AND which data do I not
need? (e.g. MidYIS cohort or Independent
Sector)
2. What does the data mean and what does the
data not mean? (e.g. staff INSET and support)
3. Who is the data for?
4. Storage, retrieval and use of data (e.g. selfevaluation and preparing for Inspection)
The use of this data needs to allow us to do our
best to help every student to at least achieve if
not exceed their potential.
It may challenge
•
•
•
•
The culture of ‘my’ school/college
Accountability policy
Expectations
Staff training in use of data and ability to cope with data
(data overload)
• Integrating the data into school procedures, storage,
retrieval, distribution and access
• Roles and Responsibilities
Who should data be
shared with?
Colleagues
A. Subject Teachers
B. Heads of Department
C. Pastoral Staff
D. Managers
Subject Teachers/HODs
1. This will be interpreted as a personalised
prediction
2. The data doesn’t work for this particular student
3. You’re raising false expectation – he’ll never
get that result
4. You’re making us accountable for guaranteeing
particular grades – when the pupils don’t get
them we’ll get sacked and the school will get
sued
Subject Teachers/HODs
Remind them that:
1. Baseline data can give useful information about a
pupil’s strengths and weaknesses which can assist
teaching and learning
2. “Predictions” are not a substitute for their professional
judgement
Reassure them that:
1. It is not a “witch hunt”
2. Value added data is used to assess pupil performance
not teacher performance!
Pupils
1. Make sure they know why they are taking the
test
2. Make sure they take it seriously
3. Make sure they don’t deliberately mess it up in
order to lower their BSGs!
4. Be prepared to look for clear anomalies and retest if necessary
5. Explain the chances graphs to them clearly
Parents
1. Make sure they know why the pupils are taking
the test
2. Explain the results to them
3. Explain lots of times that the chances graphs
and BSGs do NOT give personalised
predictions
4. Ensure that they receive good quality feedback
from staff when ambers or reds are awarded
5. Encourage them to ask lots of questions
YOUR QUESTIONS
Thank You
Robert Clark
([email protected])
Neil Defty
([email protected])