Transcript Reporting of results in HKDSE Examination

Understanding the Work of
the HKEAA:
Mark Processing and Grading
Mark processing
Some important steps
completion of marking:
after

Equating of marks of optional
parts / papers

Moderation of SBA
submitted by schools
marks
2
Graded Approach for
English Language Papers 1 & 3
Why do we need a graded
approach?
 Adopted in Paper 1 (Reading) and
Paper 3 (Listening and Integrated
Skills)
 Compulsory Part A and either Part B1
(easier) or Part B2 (more difficult)
 Cater to candidates with a wide range
of ability
4
What are the advantages?
 No labeling effect
 No need to extend examination time or
reduce coverage of the curriculum
 Candidates need not indicate choice at
registration so possible to catch up
before the examination
 Statistical
equating to ensure that
candidates receive comparable scores
on the same scale whether they opt for
the easier or more difficult part
5
Why is there a capping policy?
 Candidates attempting Parts A and B2
will be able to attain the highest level of
performance (Levels 5 and 5* / 5**)
 Candidates who attempt Parts A and B1
will only be able to attain up to Level 4 in
the relevant component
 Reason for capping policy is that items in
B1 do not assess skills covered by Level 5
descriptors
6
Why is equating necessary?

Easier to get high marks in Part B1 than
in Part B2 because the items are easier

1 mark in B1 is not equal to 1 mark in B2

Need to convert the marks to the same
scale

Use the marks of Part A as the mediator
because all candidates do Part A
7
Example
 100 candidates got 36 marks in Part A
 50 of them did B1 and 50 did B2
 The 50 B1 candidates got 40 marks in B1
 The 50 B2 candidates got 32 marks in B2
 40 marks in B1 = 32 marks in B2
8
What is equipercentile equating?
 Large candidature for English Language
(e.g. ~ 80,000 in 2013)
 All candidates do Part A and can be rank
ordered based on their marks in Part A
 B1 marks are statistically adjusted for
difficulty to the same level as B2 marks
using the percentile ranks of candidates’
performance in Part A and Part B1/B2
9
Equipercentile method of
equating
B2
candidates’
ranking in
Part B2
B2
candidates’
ranking in
Part A
B1
candidates’
ranking in
Part A
B1
candidates’
ranking in
Part B1
50th%
40th%
40th%
40
50th%
36
36
32
10
Equipercentile equating
http://www.hkeaa.edu.hk/DocLibrary/Media/Le
aflets/HKDSE_SRR_A4_Booklet_Jun2011.p
df
11
Is there a fixed conversion
factor?
 The examination papers for each year
will be different from the previous
year’s
 Therefore no fixed conversion factor
 A conversion table will be produced
every year for that particular
examination paper based on the
performance of the candidates in Parts
A, B1 and B2
12
How to choose between
B1 and B2?
Levels 4-5
Levels 3-4
Levels 1-3
Parts A & B2
(more difficult part)
Parts A & B1
or
Parts A and B2
Parts A and B1
(easier part)
13
How is the paper mark calculated?
 For Part B1 candidates:
Part A marks + Part B2 equivalent
marks
 For Part B2 candidates:
Part A marks + Part B2 marks
 Therefore,
choosing B1 or B2
makes no difference for mid-range
candidates
14
Moderation of SBA Marks
Some queries/concerns
 Why moderate schools’ SBA marks?
 Is it fair to moderate students’ SBA
marks based on exam results?
16
Why moderation?
 Teachers know their students well and thus
are best placed to judge their relative
performance within a school
 However, they are not necessarily aware of
the standards of performance across all
schools
 Moderation
ensures the consistency of
assessment standards across schools in
order to achieve across-school comparability
17
What are the principles of
SBA moderation?
 Accept teacher judgments of relative
student performance within a school
(internal standardisation, student
queries)
 Moderation is done on school basis
(not on individual basis) to standardise
across schools
18
Methods of SBA moderation
A. Statistical

The distribution of a school’s SBA
marks for a subject is re-shaped taking
into consideration school’s pattern of
mark distribution for that subject’s
examination papers
B. Expert judgment

A team of experienced, subject experts
reviews samples of schools’ SBA work
for a subject
19
Moderation methods
Expert Judgment
Moderation
Statistical Moderation
for subjects with a small
candidature or that involve
outcomes that are very different
from those assessed through the
public examination
for subjects in which the SBA and
the public examination share a
substantial portion of common
assessment objectives
Design and Applied Technology
English Language, Chinese Language
Liberal Studies,
History, Chinese History
Biology, Chemistry, Physics
Science (Combined
Science/Integrated Science)
Information and Communication
Technology
Visual Arts
20
Features of statistical
moderation
 Internal
changed
ranking
would
not
be
 The statistical moderation method
would
consider
the
empirical
correlation
between
examination
results and SBA results
 Eliminate the effect of strictness
and leniency
21
Features of statistical
moderation
 Sample review to help identify outlier
schools
 For outlier schools, adjustment will be
carried out, if necessary, by considering
a number of factors, such as moderated
marks based on examination results and
sample review findings so as to reflect
the actual SBA performance
22
Expert judgment
moderation
 Review of samples of students’ work
 Determination of school performance
level on SBA and individual students’
moderated scores
 Detection
of cases with extreme
adjustment and follow-up actions
23
Review of samples of
students’ work

Stratified random sampling within
each school

District Coordinators / External
Assessors review samples with
reference
to
the
stipulated
assessment criteria to determine
the group performance level

More samples from a school may be
collected if necessary
24
Result of moderation in
2012 HKDSE
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
2009CE
2010CE
2012DSE
10.0%
0.0%
much lower
lower
slightly
lower
as
expected
slightly
higher
higher
much higher
Mean SBA Adjustment
25
Result of moderation in
2012 HKDSE
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
2009CE
2010CE
2012DSE
10.0%
0.0%
much lower
lower
slightly
lower
as
expected
slightly
higher
higher
much higher
Mean SBA Adjustment
26
Result of moderation in
2012 HKDSE
 In general, most teachers are aware of
the standards and their SBA marks need
very little adjustment
 Small deviation from the standard is
expected due to normal variation of
experience and subjective judgment: no
systemic bias observed
 Moderation
provides
assurance
of
fairness to students, parents, and
schools
27
Reference
 HKDSE Examination: Moderation of
School-based Assessment Scores
http://www.hkeaa.edu.hk/DocLibrary/SBA/H
KDSE/HKDSE-SBA-ModerationBooklet_r.pdf
28
Standards-referenced
Reporting
Some queries/concerns
 Why the grade awards are different
across subjects?
 Is this unfair to my subject?
30
出席
人數
No. Sat
考生考獲各等級的百分率
Percentage of levels awarded
中文作答
Chinese
Version
%
5**
5*+
5+
4+
3+
2+
1+
U
7 918
61.6
0.7
2.9
7.2
25.2
50.9
77.5
93.0
7.0
69 721
88.0
0.8
3.0
7.7
35.7
72.5
90.8
98.9
1.1
451
-
2.7
10.4
25.9
57.6
80.5
92.5
96.9
3.1
69
299
49.5
1.1
4.8
12.1
34.1
57.1
79.7
91.7
8.3
延伸部分（微積分與統計）
Extended Part
(Calculus and Statistics)
7 347
27.1
1.4
5.9
15.1
43.4
69.7
86.9
95.1
4.9
延伸部分（代數與微積分）
Extended Part
(Algebra and Calculus)
8 091
18.1
3.0
12.0
31.4
56.1
79.8
92.5
98.0
2.0
音樂 Music
232
8.6
1.7
7.8
19.0
60.8
85.8
97.8
99.6
0.4
體育 Physical Education
681
93.5
0.1
0.4
1.6
9.3
37.7
63.9
88.7
11.3
15 233
31.4
2.4
9.6
23.8
48.8
73.0
88.8
97.0
3.0
科目
Subject
資訊及通訊科技
Information and Communication
Technology
通識教育 Liberal Studies
英語文學 Literature in English
必修部分
Compulsory Part
數學
Mathematics
物理 Physics
Standards-referenced reporting
system 24 NSS Subjects


Descriptors and
exemplars to indicate
standards required for
each level
Facilitate users,
including tertiary
institutions and
employers, to set
appropriate entrance/
job requirements
32
3
Standards-referenced reporting
system 24 NSS Subjects
 Adopted
to report candidates'
assessment results in all 24 HKDSE
Category A subjects
 SRR
adopted for HKCE English
Language and Chinese Language since
2007
33
Grading in the HKDSE
Cut scores and grade levels determined so
that each level represents a fixed
standard of achievement from year to year
 Methodology combines psychometric
measurement and expert judgment
 Psychometric measurements:

i) Monitoring tests for the core
subjects
ii) Group ability index (GAI) for
elective subjects
34
Grading in the HKDSE
Expert judgments made independently by
experienced examiners and experts based
on the actual performance of candidates on
the exams, using the level descriptors and
library scripts for reference
 Psychometric cut score recommendations
are derived for the reference of the
expert panel
 No pre-set quota for each level /subject

35
Standards maintenance in 2013
Four core subjects (CEML)
Additional reference made to:
 statistical
data from the monitoring tests
which were administered to a
representative sample of 2012 and 2013
DSE school candidates
36
Using monitoring tests to
maintain standards
Statistically-generated cut scores for
reference of Expert Panel
37
Group Ability Index (GAI)
for elective subjects
 Different Category A elective subjects may attract
students of different abilities
 Compute an indicator (index) to reflect overall
performance in core subjects (ability) of all candidates
taking the subject (group), taking into account how the
performance in the elective correlates with each of the
core subjects

Provided as supplementary statistics, in addition to the
level descriptors, library scripts and live scripts, to the
subject experts for their expert judgment. Final
decision based on performance as reflected in the
candidates’ scripts
38
Group Ability Index
The GAI of Subject X for a certain level (e.g.
Level 3) or above is defined as:
 nC
1
nE
nM
nL 

  100%
P
bC
 bE
 bM
 bL

bC  bE  bM  bL  NC
NE
NM
NL 
b
n/N
Coefficients obtained by
Proportion of candidates in a
regressing scores of Subject X on
scores of CEML
group attained a particular level (e.g.
Level 3) for a core subject
39
Grading Methodology and
Procedures
Who are involved in grading
 Each subject sets up its expert panel
which comprises:
the chief examiner(s) of different
papers,
 assistant examiners/markers, if
necessary;
 SBA supervisors/chief project
examiner, if applicable, and
 subject manager

41
Overview of grading procedures
1. Familiarisation with standards
2. Sample script selection
3. Standardisation
4. Post-marking exercise
5. Expert Panel Meeting
6. Internal Meeting
7. Public Examinations Board Meeting
42
4. Post-marking exercise
1. Determine preliminary cut scores ranges
for all levels for each exam paper (incl.
SBA) with reference to:





the difficulty of live question paper and
requirements of marking scheme
inter-paper correlations, mean and standard
deviation of current year’s papers in
comparison to previous years
item statistics, if available
impression of candidates’ performance when
marking live scripts or reviewing students’
SBA work
level descriptors and borderline performance
samples in the previous exams
43
5. Expert panel meeting
 Objective:
 Apply expert judgment to arrive at a
set of recommended cut scores for
different levels for the subject and its
components, if applicable
 Make reference to marked live scripts
falling within / around the
recommended cut score ranges and
exam statistics
44
Expert panel meeting
(cont’d)
 Also refer to cut scores suggested
by statistical means:
 for
core subjects, the suggested cut
scores are derived from the results of
the monitoring tests (2013 data in
comparison with the 2012 data)
 for elective subjects, the suggested cut
scores are derived from GAI
45
出席
人數
No. Sat
考生考獲各等級的百分率
Percentage of levels awarded
中文作答
Chinese
Version
%
5**
5*+
5+
4+
3+
2+
1+
U
7 918
61.6
0.7
2.9
7.2
25.2
50.9
77.5
93.0
7.0
69 721
88.0
0.8
3.0
7.7
35.7
72.5
90.8
98.9
1.1
451
-
2.7
10.4
25.9
57.6
80.5
92.5
96.9
3.1
69
299
49.5
1.1
4.8
12.1
34.1
57.1
79.7
91.7
8.3
延伸部分（微積分與統計）
Extended Part
(Calculus and Statistics)
7 347
27.1
1.4
5.9
15.1
43.4
69.7
86.9
95.1
4.9
延伸部分（代數與微積分）
Extended Part
(Algebra and Calculus)
8 091
18.1
3.0
12.0
31.4
56.1
79.8
92.5
98.0
2.0
音樂 Music
232
8.6
1.7
7.8
19.0
60.8
85.8
97.8
99.6
0.4
體育 Physical Education
681
93.5
0.1
0.4
1.6
9.3
37.7
63.9
88.7
11.3
15 233
31.4
2.4
9.6
23.8
48.8
73.0
88.8
97.0
3.0
科目
Subject
資訊及通訊科技
Information and Communication
Technology
通識教育 Liberal Studies
英語文學 Literature in English
必修部分
Compulsory Part
數學
Mathematics
物理 Physics
Reference
 Grading Procedures and Standards-
reference Reporting in the HKDSE
Examination
http://www.hkeaa.edu.hk/DocLibrary/Media/Leaflets/HKD
SE_SRR_A4_Booklet_Jun2011.pdf
47
Reporting of Results
Reporting of HKDSE results
 For Category A subjects, results
reported mostly on subject basis
 Component results reported for:
 Chinese
Language
 English Language
 Combined Science
49
Reasons for component
reporting
 Allow end users to understand the
achievement of candidates in
 different
areas of the subject e.g. reading,
writing, listening and speaking skills in
language subjects
 different half subjects in Combined
Science
 For selection for studies or employment
50
FAQ on component result
and subject result
 Why is my subject level lower than
the average of individual component
levels?
51
The facts
 Subject level is based on the overall
subject mark which is the
combination of all component marks
 NOT a simple averaging of component
levels achieved
52
The facts
 Each level covers a range of marks; a
candidate may have a score close to
the lower/higher end of the range
 Components of a subject carry
different weightings; the marks for
each component go through a
weighting
procedure
before
combining them to arrive at the
subject mark
53
Example:
HKDSE English Language
 Comprises four components:
 Component 1: Reading (20%)
 Component 2: Writing (25%)
 Component 3: Listening and Integrated
Skills (30%)
 Component 4: Speaking + SBA (25%)
54
Level
achieved
Student A
Student B
5
4
3
*

Reading
(20%)
Writing
(25%)
Listening
(30%)
Speaking
(25%)
Subject
5
5
4
5
4
5
3
5
4
5

*

*

*
*

*

2
1
55
Thank You
56