Transcript Comments on Analysis submitted by Chairman-CBSE

Comments on Analysis submitted
by Chairman-CBSE
Debasis Sengupta
Indian Statistical Institute
April 14, 2013
General remarks
•
•
•
•
It is good to see some concrete analysis.
Summary of findings was missing.
I have summarized as I have understood.
I do not have access to the 516,416 sets of matched
data (about 48.6% of AIEEE 2012 candidates).
• I have done additional analysis from what I could get
from the presentation.
• I used same definitions and notations.
• No contradiction with findings of Normalisation
Committee.
A major issue: How the two
procedures alter current scenario
• The NIT’s are concerned with top 3% of rank list.
• Chairman-CBSE has computed, for each state,
No. of candidatesfromthestateabove95th percentile
T otalno.candidatesappearingfromthestate
as a percentage.
• ‘%age under procedure 1’ minus ‘%age under current
scenario’ is called P1 – P0.
• P2 – P0 is similarly defined.
• Comparative plot is reproduced here from page 6 of
circulated presentation.
Plot of P1–P0 and P2–P0 for 25 boards
8%
P1-P0
6%
P2-P0
4%
2%
0%
-2%
-4%
•
Main conclusion: Maximum deviation is only about 2%.
•
No problem (on this aspect) with either procedure.
•
Larger deviations for P1 – P0 is found in other percentile ranges, which do not matter to NIT’s.
•
Board scores often have less than 0.5 correlation with AIEEE scores (see page 21 of report).
A procedure giving 40% weight on board scores is expected to produce some changes.
•
This fact was known to the normalisation committee from earlier analysis of over 2 lakh
matched data (subset of present data), and included in the report (pages 22-23, F12-F21).
Another percentage
• Consider, for each state
No. of candidatesabove95thpercentilefrom the state
T otalno.candidatesabove95thpercentile
•
•
•
•
expressed as a percentage.
This gives the composition of the top 5% students.
This is a matter or direct relevance to NIT’s.
P1 – P0 and P2 – P0 can be defined as before.
Comparative plot is shown next.
Change in representation of boards in
top 5%, for the two procedures
4%
P1-P0
P2-P0
2%
UP
UK
TN
RJ
PB
NL
NIOS
MZ
MR
MP
MN
MG
KL
KK
JK
JH
HP
GT
GO
CISC
CG
BR
CBSE
-2%
AP
AM
0%
-4%
•
Main conclusion: CBSE gains from Procedure 2; AP board and CBSE lose from Procedure 1.
•
Under Procedure 1,
– AP board share declines from 20.5% to 17.5%; CBSE share declines from 61.7% to 59.4%.
– Both boards have above-par representation even after decline,
as 10.8% of candidates in matched data set are from AP board; 39.6% from CBSE.
•
Under Procedure 2, rise of CBSE share is from 61.7% to 65.7%, which is further above-par.
Tip of the iceberg
• This finding (increase in CBSE’s already large share
among top 5%) is not new.
• Same phenomenon was observed from simulation study
based on the 2 lakh matched cases available earlier.
• The present finding (though based on more data) is not
necessarily the ultimate one, as the matched cases have
different composition than the ‘set of all candidates’.
• In AIEEE 2012, 30% students were from CBSE; in
matched part of data, 40% were from CBSE.
• Simulations indicate more dramatic increase (through
Procedure 2) in CBSE’s share in top 1%, 0.1% etc.
Simulation conditions
• 10% Random samples chosen from AIEEE
2012 scores data.
• Presumed correlation between AIEEE and
board scores:
– Observed correlations for AS, CBSE, JK, MR, MZ,
UK (available till Feb’13)
– 0.4 for other boards (observed value for most
boards).
• 100 separate simulation runs.
Actual performance of CBSE students
in AIEEE 2012
• 30.1% of all AIEEE 2012 candidates were CBSE students.
• However, this groups constitutes
–
–
–
–
66 of the top 100 AIEEE rank holders,
608 of the top 1,000 AIEEE rank holders,
5,977 of the top 10,000 AIEEE rank holders,
54,430 of the top 100,000 AIEEE rank holders.
• Slide 12 of Chairman-CBSE reports the corresponding values
for the matched data set, which show similar trend. (Figure in
that slide is correct, the numbers listed below are swapped.)
Share of boards in different percentile
ranges of AIEEE 2012
100%
90%
80%
70%
CBSE
60%
MR
50%
AP
BR
40%
RJ
30%
UP
All other
20%
10%
0%
0-10
10-20
20-30
30-40
40-50
50-60
60-70
70-80
80-90 90 to 99 99 to 99.9 to above
99.9 99.99 99.99
Rest of the iceberg: How Procedure 1
or 2 would change the shares
AIEEE 2012 – Current procedure
100%
80%
60%
40%
100%
CBSE
0%
80%
CBSE
MR
60%
AP
40%
AP
20%
BR
BR
20%
AIEEE 2012 - Procedure 1
RJ
MR
RJ
0%
UP
UP
All other
All other
100%
• Procedure 1 reduces marginally the
dominance of CBSE students in
merit list ,
• Procedure 2 increases drastically
that dominance.
AIEEE 2012 - Procedure 2
80%
UP
60%
CBSE
40%
MR
20%
AP
0%
BR
RJ
All other
…and the numbers behind the charts
Current scenario: share of boards in different percentile ranges of AIEEE 2012
10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100
17%
20%
22%
26%
28%
33%
39%
46%
54%
17%
15%
14%
12%
11%
9%
8%
6%
5%
8%
8%
8%
8%
8%
9%
9%
10%
13%
9%
9%
8%
8%
7%
6%
5%
4%
3%
49%
48%
48%
46%
45%
42%
38%
33%
25%
Board
CBSE
MR
AP
BR
All other
0-10
14%
17%
7%
10%
51%
Board
CBSE
MR
AP
BR
All other
Procedure 1: Share of boards in different percentile ranges of AIEEE 2012
0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100
15%
18%
21%
24%
26%
29%
33%
38%
44%
53%
17%
16%
14%
13%
12%
11%
10%
8%
7%
5%
7%
8%
8%
8%
8%
8%
9%
9%
10%
13%
10%
9%
9%
8%
8%
7%
6%
6%
5%
3%
51%
49%
48%
47%
46%
44%
42%
39%
35%
26%
Board
CBSE
MR
AP
BR
All other
Procedure 2: Share of boards in different percentile ranges of AIEEE 2012
0-10 10-20 20-30 30-40 40-50 50-60 60-70 70-80 80-90 90-100
11%
14%
17%
21%
25%
29%
35%
42%
48%
59%
20%
18%
16%
14%
12%
11%
9%
7%
5%
4%
6%
6%
7%
8%
8%
9%
9%
10%
11%
13%
11%
10%
10%
9%
8%
7%
6%
5%
4%
2%
53%
51%
50%
49%
47%
44%
41%
36%
31%
22%
Overall
30%
11%
9%
7%
43%
Findings from simulation
Present scenario
Procedure 1
Procedure 2
Top 0.01%
65.1%
61.9%
82.7%
Top 0.1%
60.5%
60.4%
80.1%
Top 1%
60.0%
60.0%
69.6%
Top 10%
54.4%
53.4%
58.8%
•
Procedure 1 produces a marginal reduction in CBSE’s share at the top; 2% reduction
at the top 0.01%.
•
Procedure 2 brings a great windfall to CBSE students.
•
CBSE students gain more and more from Procedure 2 as one goes up the merit list
– share at top 1% increases by 10% ,
– share at top 0.1% increases by 20%.
•
Table given in Slide 16 of Chairman-CBSE is in line with this finding. (Slide 15
erroneously shows CBSE’s share in top 0.01% under Procedure 1 as 88.2%. This
goes against common sense and also against the trend of the adjacent column.)
How does Procedure 2 dramatically increase
CBSE’s share of top-rankers?
• It does so by practically turning the board score into another
AIEEE score.
• The composite score for each board has same mean as AIEEE
scores from that board, but has much smaller spread.
• The set of scores from each board get more concentrated
around their respective means.
• It is as if every student gets a second chance to perform at the
same mean level.
• As a result, top CBSE students gain conspicuously.
Why does Procedure 1 have only a mild effect
on CBSE’s share of top-rankers?
• For CBSE, the students who do well in AIEEE are generally
those who do well in board exam.
– This may be guessed from the higher rank correlation (see page
21 of report) of CBSE and AIEEE scores.
– They mostly hold on to their position in merit list.
• For boards with less correlation with AIEEE, this is not
the case.
– Many AIEEE top-rankers from other boards move towards
middle in terms of composite score.
– If their board exams had been better aligned with AIEEE, they
would have gained more from Procedure 1.
Will smaller boards gain unfairly from
Procedure 1?
•
Consider the percentage of candidates from a board making it to top 5%.
– Plot in Slide 6 and Table in Slide 8 of Chairman-CBSE shows that the maximum change
effected by Procedure 1 is 2%.
– Table in slide 7 of Chairman-CBSE shows that these percentages are below 5% (parscore) for all small boards under Procedure 1.
(In fact, all boards except AP, CBSE, CISC and KK have below par representation.)
– If boards with less than 1000 matched samples are regarded as ‘small’ and the rest are
‘large’, the difference in mean changes in the two groups are statistically insignificant
at the 5% level.
•
Consider the percentage of the top 5% candidates coming from a board.
– Slide 5 of this presentation shows that the changes effected by Procedure 1 are well
below 2% for all boards except AP and CBSE.
– ‘Small’ and ‘large’ boards have statistically insignificant difference of mean change.
•
Answer to the question is NO.
Fate of students with common scores
• In 2012, five students (4 from CBSE, 1 from Maharashtra board) were tied at
– AIEEE aggregate marks of 130 ,
– board percentile of 93.1.
• According to Procedure 1, they shared rank 20,352.
• According to Procedure 2
– CBSE students had shared rank 18,077,
– Maharashtra board students had rank 34,175.
• The differential treatment is entirely due to relatively poor performance of
other students of Maharashtra board in AIEEE.
• This goes against the principle of fairness.
• Chairman-CBSE has tried to give a counter-example (Slide 11), involving 19
matched scores tied exactly as above. It is claimed that Procedure 2 would
preserve the equivalence but Procedure 1 would not. This is impossible, as
equal percentiles are treated equally by Procedure 1.
Assumptions
• Chairman-CBSE mentions twice in his email message: Procedure 1 assumes
same merit distribution across boards, Procedure 2 does not.
• This is a half truth. The other half is: Procedure 2 assumes that the disparity of
merit distributions across boards can be precisely measured (and hence
adjusted for) by the AIEEE/JEE-Main score.
• In reality, AIEEE/JEE-Main performance disparity across boards can result from
–
–
–
–
–
Difference in merit/ability distributions,
Difference of board examination pattern with AIEEE (rank correlations indicate this),
Lack of instruction in English and Hindi (only available languages for AIEEE/JEE-Main),
Less access to coaching,
Load of an extra subject in board (for some boards).
• Adjusting only for one factor is impossible – in presence of confounding factors.
• Disparity may worsen after adjustment.
• AIEEE performance patterns are VERY disparate. If that is used to calibrate, 80%
of MH board would be deemed equivalent to 50% of CBSE. The implied
assumption of LARGE merit disparity among large boards is risky.
How students with same AIEEE score
move under two procedures
•
•
•
•
•
•
•
Slide 10 of Chairman-CBSE shows the movements of ranks of candidates tied at AIEEE
aggregate marks 130 (rank 27493).
Claim : Procedure 1 shows greater movement of rank than Procedure 2.
Fact : Barring stray cases, the range of movements differ only minimally.
In Slide 10, a large movement of rank of a UP candidate through Procedure 1 has been
marked with an arrow. Greater movement can be seen in the case of some CBSE
candidates for Procedure 2. The other marked case is not too far.
The table given in Slide 13 gives an overall picture of rank movements. However, it is
hard to draw conclusions from a large table. The normalization committee had looked
at scatter plots of the ranks for a pictorial overview (Figure 3.5, page 23). The scatter for
Procedure 2 was found to be marginally less dispersed.
As explained in the report, this may be attributed to the board-specific distortions
inflicted on board percentiles by Procedure 2, which bring them closer to AIEEE score.
The low rank correlation between AIEEE percentiles (restricted to boards) and board
percentiles, mentioned in page 21 of the report, imply that large movements in some
cases are normal – if board percentiles are indeed given 40% importance.
In summary…
• Barring some errors (e.g., in slides 11, 12 and
15), the computations reported by ChairmanCBSE make sense.
• These computations corroborate the findings
of the normalisation committee led by Prof.
S.K. Joshi.