Transcript Slide 1

Data-driven School
Improvement through the
VCE Data Service
Presented by
Philip Holmes-Smith
School Research Evaluation and Measurement Services
What is the VCE Data Service?
VCEDS
• is a confidential online service provided by
the VCAA to all VCE schools through the
VASS system;
• includes an archival database holding a
snapshot of your school’s VCE data for each
year from 1998 onwards;
• allows you to input your own school data to
integrate with the data held by the VCAA.
The Origins of VCEDS
• VCEDS was developed in 2002 following an
extensive R & D trial with 50 schools since the
late 1990s (the VCE Data Project).
• VCEDS was fully implemented in January 2003,
when the 2002 VCE data were loaded into the
system.
• Subsequent improvements have been made to
the data Service in successive years
What’s available?
You can
• analyse graded assessment results, GAT
scores, and ENTER Subject scores as well as
VCAA Study Scores;
• generate reports for your “Assessing
School” students, as well as your “Home
School” students;
• look at patterns of results for individual
students;
• export data for further analyses.
Who has access to VCEDS?
• VCEDS was designed to be used by the senior
leadership team in each school.
• Each VASS coordinator has access to VCEDS through the
opening menu of the VASS system (look under “School
Administration”).
• The VASS coordinator can grant access to other users
with the approval of the Principal.
• These users have access to VCEDS only; not the rest of
the VASS system.
• Each user has his/her own security device, which
cannot be shared.
Comparisons you can make
External to your school:
Your school results can be compared with
results for
• All schools
• Like schools.
• Schools in your sector.
• Like schools in your sector.
Like School Groups (for Government Schools only)
Comparisons you can make
Internal to your school:
Your school results can be compared
•
•
•
•
•
by gender,
by class grouping,
by year level
by year (1998 - present),
by any subgroups in the school for which you can
provide data (e.g. campus, language background, year
level).
VCE Data Service: Menu Structure
VCE Data Service: Menu Structure
VCE Data Service: Menu Structure
VCE Data Service: Menu Structure
VCE Data Service: Menu Structure
VCE Data Service: Menu Structure
Report 1
There are 30 students in this Year Level.
Therefore:
–
–
50% (or 15 students) are above the median.
50% (or 15 students are below the median.
–
50% (or 15 students) are inside the “box”. Half
of these (7-8 students) are above the median
and half (7-8 students) are below the median.
–
–
–
–
10% (or 3 students) are at or below the 10th
percentile “whisker”.
10% (or 3 students are at or above the 90th
percentile “whisker”.
15% (or 4-5 students) are spread between the
25th down to the 10th percentile.
15% (or 4-5 students) are spread between the
75th up to the 90th percentile.
3 students on or above the
90th percentile. (Note, all we
know is at least one student
scored on the 90th percentile.)
4-5 students between the
75th up to the 90th
percentile.
7-8 spread above the median.
15 students inside the box.
7-8 bunched below the median.
15 students below median
•
15 students above median
Interpreting “box and whisker” graphs
4-5 students between the 25th
down to the 10th percentile.
3 students on or below the 10th
percentile. (Note, all we know
is at least one student scored
on the 10th percentile.)
Report 5
Report 7
Report 8: Example 1
Report 8: Example 2
Report 8: Example 3
Report 8: Example 3 (cont.)
Report 8: Example 3 (cont.)
Report 8:
Example 4
Report 9
Mathematical Methods : (VCAA Study Score x Time)
zTest College Home School Data
Report 15: Example 1
Report 15: Example 2
Report 16
Biology: GAT and Study Score Distributions x Time
zTest College Home School Data
Relationship between GAT scores and
VCE Study Scores
• The last slide shows that for Biology, the
higher the ability of the group (as
measured by the GAT) the higher the VCE
study score
• The VCE Data Project established that
overall, the GAT is a very good predictor of
VCE study scores
Example: Relationship between GAT (Comm) scores
and English Study Scores
50
45
40
English Study Score
35
30
25
Pred Score = 13.431 + 0.5204 x GAT
20
15
10
5
0
0
5
10
15
20
25
GAT (Communication)
30
35
40
45
50
Predicting VCE Study Scores
The VCE Data Project established clearly that,
other things being equal, you can expect
higher levels of performance from a
student who
• has greater ability;
• is in a class with more able students;
• is female.
Adjusted Estimates
VCEDS calculates what each of your students could have been
expected to get on the basis of
• their ability, as indicated by their three GAT component
scores;
• the average GAT performance of the subject group in their
school; and
• their gender.
It then compares your students’ results to those that would
have been expected.
The difference is called an adjusted score
Aggregated across students, it is called an adjusted estimate
for the group.
Example using just the GAT (Comm) Score to predict English
study scores
50
45
40
English Study Score
35
30
25
Pred Score = 13.431 + 0.5204 x GAT
20
15
Students who scored 25 on the GAT would
be predicted to score 26 for their VCE
English study score
10
5
0
0
5
10
15
20
25
GAT (Communication)
30
35
40
45
50
Example (cont)
50
However, this student actually scored 35
for English which is 9 study score points
higher than we would have predicted
45
40
English Study Score
35
30
25
Pred Score = 13.431 + 0.5204 x GAT
20
15
On the other hand, this student actually
scored 18 for English which is 8 study
score points lower than we would have
predicted
10
5
0
0
5
10
15
20
25
GAT (Communication)
30
35
40
45
50
How does VCEDS report Adjusted
Estimates?
• If students performed precisely as predicted in a given
study taking into account their abilities and their
gender composition, then the group will have an
adjusted estimate of zero.
• If the students perform less well than predicted, the
group will have a negative adjusted estimate.
• If students perform better than predicted, the group
will have a positive adjusted estimate.
What sort on numbers do the
Adjusted Estimates produce?
• The scale on which adjusted estimates are
reported is the same as that for Study Scores.
• An adjusted estimate of +2.5 would mean that
the student group has achieved, on average, 2.5
Study Score points higher than expected.
• An adjusted estimate of -1.7 would mean that the
student group has achieved, on average, 1.7
Study Score points lower than expected.
Why the GAT?
• The GAT measures the general knowledge
and skills that students have built up during
their school years.
• It provides an indication of how wellequipped students are to undertake VCE
studies.
• Data show it to be a good predictor of VCE
performance and provides a good indicator
of where students would be in their VCE
studies, “other things being equal”.
Reporting Constraints
VCEDS does not report:
• Study score statistics and assessment grades where
studies have less than 10 students
• Ability and Gender Adjusted estimates where there is:
–
–
–
–
less than 400 total enrolments with complete data
less than 20 schools with enrolments
less than 5 students in the school
insufficient correlation between study and GAT scores
Reporting adjusted estimates
All adjusted estimates are associated with a
degree of uncertainty. This uncertainty can
be quantified and a confidence interval can
be constructed around each estimate.
Confidence Intervals
Confidence intervals are displayed by
means of ‘error bars’ around the
estimates. You can be confident that
two adjusted estimates differ only if their
error bars do not overlap.
Confidence Intervals- Example
Estimates
Error Bars
Report 10
Report 11
Report 12
Mathematical Methods (Adjusted) x Time
zTest College
VCE study scores for the five Math Method
classes
Report 13
GAT (MST) scores vs. study scores for the five
Math Method classes
CLASS
C1
C2
C3
C4
C5
Total
GENDER
Males
Female
Total
Males
Female
Total
Males
Female
Total
Males
Female
Total
Males
Female
Total
Males
Female
Total
GATMST
33.5
35.2
34.3
35.3
37.4
35.9
20.2
22.7
21.9
31.4
34.0
32.0
34.7
26.5
31.4
32.2
31.1
31.8
SCORE
27.6
29.4
28.4
31.6
34.4
32.4
28.4
24.0
26.3
24.7
26.4
25.0
31.3
19.5
26.6
28.1
27.6
28.0
N
12
11
23
18
7
25
13
12
25
20
5
25
3
2
5
66
37
103
Report 17
Report 18
Student Results across Studies zStudent 14, zTest
Confidential Report
2005 zTest College Home School Data
A strategy for approaching VCEDS
1. Look at the overall performance for the year.
2. Look at the results, study by study.
3. Drill down within studies, comparing performance of
subgroups within your school and trends over time.
4. Look at trends in GAT results within studies and over
time.
5. Look at adjusted estimates, study by study.
6. Drill down within studies, comparing adjusted scores
of subgroups within your school.
7. Seek a deeper understanding by examining patterns
of individual performance.
Exporting data
Exporting data (Cont)
Exporting data (Cont)
Defining school-level variables