TARGET SETTING AND MONITORING

APRIL 2011

Bristol Conference Geoff Davies

How can the CEM help target setting and monitoring?

Questions – What is our students’ potential and are they fulfilling it?

– How does our School/College compare to similar institutions?

– What are appropriate targets for our School/College? – Who plays the tune!!

– How do we monitor within a supportive culture?

TOXINS

Ideas being rejected or stolen Constant Carping Criticisms Being Ignored Being Judged Being Over Directed Not being listened to Being misunderstood

NUTRIENTS

Being valued Being Encouraged Being Noticed Being Trusted Being listened to Being Respected Our experience is that the CEM centre systems provide NUTRIENTS It is a highly respected Independent Research Base The largest provider of performance indicators in the world NO political agenda

THIS CEM stuff IS THE BEST THING SINCE SLICED BREAD!

A RIGHT TO BE SUSPICIOUS! HEADTEACHER

A PLEA

DON’T LET THERE BE A LONE RANGER!

ALL need to be involved in the process Having one ‘expert’ is unhealthy and can be dangerous Most teachers know their students’ strengths and weaknesses. Professional judgement is still of the utmost importance but there are surprises. Using data as a tool to help improve your student’s achievement cannot be a one-person job.

The processes of discussion about students between teachers based on an objective data source have proved to be important in improving outcomes .

Necessary knowledge base to use CEM systems to their potential 1. The forms of Value Added Data : • scatter graphs • raw and standardised residuals • SPC charts • tables of data • use of PARIS for further analyses (e.g. by gender, teaching group) 2. Predictive Data : • point and grade predictions • importance of chances graphs • availability of different predictive data 3. Baseline Data • band profile graphs • IPRs • Average GCSE score • Computer adaptive tests

If you have the tools you can use them to do these • Make curriculum changes • Adjust staffing structure and cater for student needs • self-evaluation procedures including the analysis of examination results using value added data •

the target setting process

• school and department development plans…….

•

Improve your monitoring procedures

THE TARGET SETTING PROCESS

You know what Research Tells Us

• Goals must be specific • Goals must be challenging • Need for goal commitment • Need for feedback

Research Does Not Say

• Who should set the targets.

• Possible levels of achievement!

• If it works in education • If it can be made to work in very complex tasks.

What are the targets for?

Are you aiming the targets!

OR Do the targets belong to and are being aimed at by all?

THE MOST IMPORTANT WORD IN EDUCATION???

CULTURE

TARGET SETTING

‘Intelligent’ Target Setting involves:

• Using reliable predictive data and chances graphs • Dialogue between the users: teachers,

parents

students? (empowering, ownership, and taking and responsibility) •

The use of professional judgement……..

• • • • •

Setting Targets: why?

A process between

student and staff

of setting a challenging and realistic goal The subsequent monitoring of student progress Incorporating target data into department self evaluation And

NEVER

because it has to be done!

And NEVER to use aspirational targets for accountability.

There is wide-ranging practice using CEM data to set student, department and institution targets. Increasingly sophisticated methods are used by schools and colleges.

The simplest model is to use

the student grade predictions

. These then become the targets against which student progress and achievement can be monitored. Theoretically, if these targets were to be met, residuals would be zero so overall progress would be average. The school/college would be at the 50th percentile.

More challenging targets would be those based on the basis of history. For example. Where is the school/college now? Where is your subject now?

If your subject value added history shows that performance is in the upper quartile it may be sensible to adjust targets. This may have the effect of raising point predictions between 0.2-0.5 of a grade. This would be a useful starting point, but it would not be advisable to use the predictions for below average subjects, which might lead to continuing under achievement.

However, these examples do not necessarily give ownership of the data to staff or students, so may not be effective in practice.

Involving teachers

as part of the process should encourage more responsibility to achieve the targets.

For example, staff receive the statistically generated targets. Using these as a starting point, they could adjust using their professional judgement and knowledge of the students. This process may have to be monitored initially. Targets are purely established as part of the internal monitoring system and do not become part of the value added data.

The targets set in this way only become aspirational for the student if they are involved in the process

Involving students

should result in the student taking more responsibility for their learning.. This would give ownership of the adjusted predictions to the department. The subject teachers could then discuss these predictions with the students to finalise the target grade . Referring to this part of the discussion could be a powerful motivator to encourage the student, as well as reminding them what could happen if they do not work/take responsibility. Students are not robots who will always fit with statistics so it is dangerous to make sweeping statements based on one set of results.

A suggested pincer attack

CULTURE IS EVERYTHING IT’S NOT WHAT YOU DO BUT THE WAY THAT YOU DO IT •

ASPIRATIONAL

TARGETS FOR STUDENTS (VERY HIGH EXPECTATIONS IN NEGOTIATION WITH SUBJECT TEACHERS)

MINIMUM ACCEPTABLE

GRADES CULTURE FOR ALL WORK •

HISTORIC

DATA INFORMING DEPARTMENTAL TARGETS (

See next slide

) •

CLASS TARGETS

JUDGEMENT SET BY TEACHER/

MENTOR

USING DATA AND PROFESSIONAL •

HEAD’S/PRINCIPAL’S TARGETS

(INFORMED FROM ABOVE BUT NOT SIMPLY AGGREGATED) •

PUBLISHED TARGETS

(INFORMED FROM ABOVE BUT NOT SIMPLY AGGREGATED) •

OTHER EXTERNAL TARGETS

(

AN ENGLISH DISEASE THAT NEEDS TREATMENT!

)

Setting departmental targets :

one suggested approach • Discuss previous value added data with each HoD • Start with an agreed REALISTIC representative figure based previous years (3 ideally) of value added data • add to each pupil prediction, and convert to grade (i.e. in built value added) • By discussion between teachers and students and using professional judgement, AND THE CHANCES GRAPHS, adjust target grade • calculate the department’s target grades from the addition of individual pupil’s targets Whatever you do respect the professional judgement of the vast majority of teachers

Paris97.xls

Subject

Art & Design Business Studies Design & Technology Drama English English Literature French Geography German History Home Economics ICT Maths Music Physical Education Religious Studies Double Science Welsh

Number of Students Percentage of A* to C Grades Percentage of A* to G Grades

68 64 103 27 181 15 53 84 7 49 48 71 180 12 72 37 180 177 67 65 70 52 72 84 48 63 85 64 60 64 63 71 67 48 68 54 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100

Average Grade

5.2 (C) 4.3 (C/D) 4.7 (C/D) 5.3 (B/C) 4.8 (C) 4.6 (C/D) 4.9 (C) 4.8 (C) 5.1 (C) 5.1 (C) 4.5 (C/D) 4.9 (C) 4.5 (C/D) 5.2 (C) 4.9 (C) 5.2 (C) 4.4 (C/D) 5.1 (C) School Average GCSE score: 4.7 (C/D)

Counted Performance Statistics (Based on Subject Choice Predictions)

5 or more A* to C Grades: 106 1 or more A* to C Grades: 5 or more A* to G Grades: 1 or more A* to G Grades: 141 181 181 58% 77% 99% 99% 5 or more A* to C Grades inc Maths and English: 2 or more A* to C Grades - Sciences: 1 or more A* to C Grades - Modern Foreign Language: 98 93 36 54% 51% 20% The underlying predictions summarised here are based on expectations for an average school achieving zero value added results. Appropriate care should be taken in interpreting them within your school. Please note that the cut-off points for grade C and grade G have been set at 4.5 and 0.5 respectively. Due to the sensitive nature of the cut off points, predictions may vary for your school if the cut off points could be altered.

Subject

Art & Design Business Studies Design & Technology Drama English English Literature French Geography German History Home Economics ICT Maths Music Physical Education Religious Studies Double Science Welsh (*Predictions Adjusted for Positive Prior Value-added Performance)

Number of Students Percentage of A* to C Grades Percentage of A* to G Grades

68 64 103 27 181 15 53 84 7 49 48 71 180 12 72 37 180 177 84 48 87 100 69 67 96 73 86 67 79 96 57 92 65 70 59 86 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100

Average Grade

5.2 (C) 4.3 (C/D) 5.3 (B/C)* 6.0 (B)* 4.9 (C)* 4.9 (C)* 6.4 (A/B)* 5.2 (C)* 5.6 (B/C)* 5.1 (C) 5.2 (C)* 5.7 (B/C)* 4.6 (C/D)* 5.7 (B/C)* 4.9 (C) 5.3 (B/C)* 4.7 (C/D)* 5.5 (B/C)* School Average GCSE score: 5 or more A* to C Grades inc Maths and English: 2 or more A* to C Grades - Sciences: 1 or more A* to C Grades - Modern Foreign Language: 5.1 (C)

Counted Performance Statistics (Based on Subject Choice Predictions)

5 or more A* to C Grades: 1 or more A* to C Grades: 125 162 5 or more A* to G Grades: 1 or more A* to G Grades: 181 181 69% * 89% * 99% * 99% * 102 106 54 56% * 58% * 30% *

Subject

Art & Design Business Studies Design & Technology Drama English English Literature French Geography German History Home Economics ICT Maths Music Physical Education Religious Studies Double Science Welsh (*Predictions Adjusted for 75th Percentile)

Number of Students Percentage of A* to C Grades Percentage of A* to G Grades

68 64 103 27 181 15 53 84 7 49 48 71 180 12 72 37 180 177 97 63 73 96 70 67 74 70 71 84 63 77 61 83 72 81 59 82 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100

Average Grade

5.5 (B/C)* 4.6 (C/D)* 5.0 (C)* 5.5 (B/C)* 5.0 (C)* 4.9 (C)* 5.1 (C)* 5.1 (C)* 5.4 (B/C)* 5.4 (B/C)* 4.8 (C)* 5.2 (C)* 4.8 (C)* 5.5 (B/C)* 5.2 (C)* 5.5 (B/C)* 4.7 (C/D)* 5.4 (B/C)* School Average GCSE score: 5 or more A* to C Grades inc Maths and English: 2 or more A* to C Grades - Sciences: 1 or more A* to C Grades - Modern Foreign Language: 5.0 (C)

Counted Performance Statistics (Based on Subject Choice Predictions)

5 or more A* to C Grades: 1 or more A* to C Grades: 123 162 5 or more A* to G Grades: 1 or more A* to G Grades: 181 181 68% * 89% * 99% * 99% * 109 106 41 60% * 58% * 23% *

Summary

Successful target setting should be a simple and transparent process. The outcomes of this process would support the monitoring of student progress over the duration of the course and provide a realistic but challenging goal that students can take responsibility for to achieve their potential

MONITORING PUPIL PROGRESS

• You need monitoring and tracking procedures appropriate to your culture • based on the agreed targets • reporting to parents/guardians • and linked to appropriate intervention, support and so on

Monitoring Student Progress

Monitoring students’ work against target grades is established practice in schools and colleges, and there are many diverse monitoring systems in place. Simple monitoring systems can be very effective

Current student achievement compared to the target grade

done at predetermined regular intervals to coincide with, for example internal assessments/examinations

Designated staff having an overview

of each student’s achievements across subjects All

parents being informed of progress

compared to targets

Review of progress between parents and staff Subject progress being monitored

by a member of the management team in conjunction with the head of subject/department

A tracking system

to show progress over time for subjects and students

J

M 97.3

MIDYIS ON ENTRY 101 A 132 131 127 105 KEY STAGE 3 STATUTORY TEACHER ASSESSMENT 94 5 4 5 -2.2 6 5 6 6 -2.5 5 6 6 5 -3 SOSCA STANDARDISED SCORES 92 113 98 90 103 97 95 98

C

F 71.8

99 B 101 83 116 94 86 6 4 5 -0.1 5 4 3 4 -2 5 5 5 4 -1.8

96 98 83 102 87 83 95 88 Pupil Tracking

Tracking at departmental level for one student

Student: Peter Hendry Department: Geology test: Geol Time Scale test essay: radiometric dating test: dating homework rock cycle pract: rock textures test: igneous rocks target grade

15/09/2006 22/09/2006 06/10/2006 20/10/2006 06/11/2006 21/11/2006

A 97% 84% B 68% C 57% 54% D 50% E U 2006-8

Traditional mark book approach subject: BIOLOGY yr 12 07-08 OCT DEC MAR SURNAME FORENAME Briggs Fletcher Green Havard etc Alice Kevin Felicity Michael C A C A C B B A D B A A 1 2 1 3 2 2 1 3 1 2 2 4 C B B B 1 2 2 4 1 1 2 2 1 1 2 2

Name

A K L M N B C D E F g h I J X Y Z ZA ZB T U V W O P Q R S

MidYis Score Test Score

80 69 115 118 109 123 89 115 76 90 97 96 95 119 111 84 67 88 118 91 120 108 115 87 117 105 98 69 33 63 80 80 73 45 45 63 50 60 50 35 35 58 30 65 10 55 70 83 45 73 5 30 70 50 45 60 0.59938

-7.07013

41.12001 49.34402 32.89601

50.17065 60.20478 40.13652

49.87096 59.84515 39.89677

64.1362 76.96344 51.30896

59.46104 71.35324 47.56883

43.33772 52.00526 34.67017

33.02838 39.63406 26.42271

Astronomy 7N

MidYis Test Review

63.83651 76.60381 51.06921

64.79552 77.75462 51.83641

62.09831 74.51797 49.67865

62.99738 75.59685

51.54922 61.85907 41.23938

34.10727 40.92872 27.28581

49.5348

63.71663 76.45996 50.97331

66.47378 79.76854 53.17903

61.55887 73.87064 49.24709

46.57437 55.88924

50.3979

37.2595

-7.07013 -8.484156 -5.656104

60 70 -7.07013 -8.484156 -5.656104

-7.07013 -8.484156 -5.656104

-7.07013 -8.484156 -5.656104

-7.07013 -8.484156 -5.656104

80 90 100

MidYis Score

110 OTHER IDEAS 120 130

MONITORING YOUR SCHOOL OVER TIME INFORMS SELF EVALUATION SELF EVALUATION DRIVES DEVELOPMENT PLANNING

Alis Value-Added

MONITORING MIDYIS YEAR 7 TO SOSCA MATHS SCORE YEAR 9 Surname

A B C D E

Forename Sex MidYIS Test Score Predicted SOSCA Score Actual SOSCA Score Raw Residual Standardised Residual

F 99 F 105 M 102 F 72 F 152 93 97 95 73 134 96 97 86 73 121 3 -1 -10 1 -13 0.4

-0.1

-1.1

0.1

-1.5

On-line attitudinal surveys

• MidYIS, SOSCA, INSIGHT Yellis and Alis • Developed to meet new needs • Can be more easily adapted to meet new developments • The Parental Questionnaire • The Event mapper

Evidence from Pupils (Alis)

Extended Comparison Graphs The following list gives the titles of Comparison Graphs currently provided by the Extended Yellis questionnaire: •Alienation Indicators •Attitudes to Design & Technology •Attitudes to English •Attitudes to a Foreign Language •Attitudes to Mathematics •Attitudes to Science •Career-Relevant Activities Experienced •Sources of Careers Information Found Useful •Preferences for Kinds of Work •Items in the Freedom from Free Scale •Design and Technology Homework •English Homework •Foreign Language Homework •Mathematics Homework •Science Homework •Career Choice Motivators •Parental Involvement: Mothers •Parental Involvement: Fathers •Influences on Staying-On •Traumatic Events Experienced by Pupils •Places Where Pupils Feel Unsafe •Work Place Preferences •Cigarettes, Alcohol and Drugs Your data is above the Yellis average Your data is below the Yellis average Your data is about the same as the Yellis average Your data is about the same as the Yellis average

SUMMARY

1. SUMMATIVE VALUEADDED MONITORING AT THE END OF THE COURSE 2. FORMATIVE VALUE ADDED MONITORING DURING THE COURSE 3. FORMATIVELY WITH STUDENTS TO SET TARGET GRADES

‘When working with individual students the predictions themselves are of lesser importance than the formative process of working with students to motivate them to focus on raising achievement.’

PROXIMAL FACTORS WORK BEST Prof Peter Tymms

SOME TRAPS TO AVOID

Marksheet Name : SUBJECT REVIEW Marksheet Group : 11S1 Export Date : 04/10/2005

CLASS REVIEW a b c d e f g h j I k l m n o p q r s t u v w x y z ab ac ad ae af 90019 90090 90045 90063 90166 90123 90129 90146 90047 90115 90004 90164 90099 90011 90112 90058 90150 90127 90030 90050 90016 90174 91165 90109 90138 90122 90009 90169 90153 90010 90154 6 7 6 7 6 7 6 6 7 7 6 7 7 7 7 6 7 6 6 7 6 7 7 6 7 6 7 7 7 7 7

Total Mean

62 A 63 A 64 A 48 B 70 A 47 C 59 B 62 A 67 A 46 C 65 A 70 A 61 A 66 A 70 A 72 A 52 B 58 B 71 A 69 A 74 A 62 A 63 A 47 C 60 A 60 A 79 A 56 B 64 A 61 A A B A B A C B A A B A A A A A A B B A A A B B B A A A B B C B B B B B A B A B * B A A A B B A B B A B B B B A B B B * A * B 1323 31 42.68

6.00 1868 30 62.27

B 109 30 3.63

B 105 30 3.5

B 201 31 6.48

B 156 30 5.2 B B C A C B B A C B A B A A A C B A A A B B B B B C B B A* B B A A A A A A A B B A B A A B B B C B C A B A* B A* A* B B A B 1.20

1.00

0.50

0.40

-0.40

0.20

0.20

0.10

1.40

1.30

1.30

-0.20

0.70

1.00

0.30

0.10

0.10

0.40

-0.40

0.50

1.40

0.20

1.70

1.50

1.90

1.50

1.30

0.80

0.50

0.40

B 12 2 6 B 190 29 6.55

20.90

30 0.70

BEWARE PITFALLS INTERPRETATION

SUBJECT M

Sex

M M F F F F M M F F M M

Score (Band)

53 (B) 38 (C) 36 (D) 48 (C) 52 (B) 65 (A) 70 (A) 38 (C) 40 (C) 70 (A) 44 (C) 56 (B)

Predicted Grade

5.4 (B/C) 4.5 (C/D) 4.4 (C/D) 5.1 (C) 5.3 (B/C) 6.1 (B) 6.4 (A/B) 4.5 (C/D) 4.6 (C/D) 6.4 (A/B) 4.8 (C) 5.6 (B/C)

5.3 (B/C) Achieved Grade

6 (B) 3 (E) 3 (E) 5 (C) 6 (B) 7 (A) 3 (E) 4 (D) 5 (C) 7 (A) 6 (B) 5 (C)

5.0 (C) Raw Standard ised Residual

0.6

-1.5

-1.4

Residual

0.5

-1.1

-1.0

-0.1

0.7

0.9

-3.4

-0.5

0.4

0.6

1.2

-0.6

-0.3

0.9

-0.4

-0.2

-0.1

0.5

0.7

-2.5

-0.4

0.3

0.4

REVISED

0.5

-1.1

-1.0

-0.1

0.5

0.7

-0.4

0.3

0.4

0.9

-0.4

0.0

RESPECT HIGH EXPECTATIONS FROM TEACHERS BUT DON’T USE ASPIRATIONAL TARGETS SET FOR PSYCHOLOGICAL REASONS TO MOTIVATE STUDENTS FOR THE ACCOUNTABILITY OF TEACHERS OTHERWISE THEIR EXPECTATIONS MAY NOT BE AS HIGH NEXT TIME

TARGET SETTING AND MONITORING

BRISTOL CONFERENCE APRIL 2011

Acknowledgments are made to numerous colleagues across the UK who have shared their experiences of using CEM centre systems.

Particularly thanks to Professor Peter Tymms, Peter Hendry, CEM centre staff and Dyffryn Taf School