Transcript Educational production functions

Approaches to evidence based
policy making in education
Professor Geeta Kingdon
Congratulations CBSE

CAER commitment to evidence based decision-making

Centre will help to improve learning by promoting reliable
and valid assessments

also help to create capacity do robust evaluations

Thus bring assessment-led reform in Indian education

Splendid example of a PPP

Talk is on evidence based policy, focus on macro

Current landscape

Pitiably poor learning levels (PISA, EI, NAS, ASER, NFHS,
SchoolTELLS; NAS showed that <50% of class V kids could
answer “how much greater is 555 than 198”? )

Measuring learning can be disheartening, when much effort &
large resources expended (SSA reforms, RTE, NCF, 6 Pay
Comm, TET, NCFTE, NCSL) & still outcomes are poor; denial

Encouraging that Planning Comm’s 12th FYP (GOI, 2013, p49)
“will place the greatest emphasis on improving learning
outcomes at all levels”
Steps for evidence based decision-making

Obtain the evidence (e.g. measure learning levels)

Evidence has to be credible (e.g. measure learning with validity
and reliability, with proper sampling)

Embrace & analyse results to reach diagnoses

Formulating policies to take remedial action

Pilot test new policies, to see their impacts on learning

Scale up policies that have strong impact at lowest cost
Evidence from learning assessments –
can empower parents

An imp potential use of evidence from A – is parental info

Has potential to increase school accountability

World university rankings, but not school rankings

Fear they may reflect socio-econ background but there are
ways of reducing that

In India a debate needed
Evidence from assessment of teachers

One potential reason for poor learning - teachers
themselves lack competence

Teachers rarely tested in large scale way

Evidence from T tests in India showed poor competence

Led to the decision to bring in the TET
SchoolTELLS survey
Assessment tasks for teachers aligned with standard teaching tasks that teachers
in primary school would be required to do in the classroom routinely.
Language tool helped to understand teacher’s ability in the following:



Do you know: e.g. meaning of difficult words in a grade 4 level text
Can you explain : e.g. explain difficult words in simple language or summarize
a Std 4 story text effectively
Can you spot common mistakes: e.g spelling and grammar mistakes
Maths tool also helped to understand the teacher’s ability in the following :



Do you know: e.g. solve problems Std 4 or 5 level
Can you explain: e.g. explain problem solving in simple steps
Can you spot and analyze common mistakes : e.g in arithmetic operations
The teacher tests were graded by SCERT Bihar staff
From Std 3 onwards, the
vocabulary in language
textbooks becomes
difficult. So, teachers
need to be able to explain
difficult words in simple
language.
TASK 1: VOCABULARY RELATED TASKS
Grading done on 3 criteria:
 Was the word meaning “meaningful” ?
 Was language used “easy to understand” ?
 Were there any spelling mistakes ?
TASK 2 : SUMMARIZING TEXTS
TASK 3: SPOTTING SPELLING & GRAMMATICAL MISTAKES
Deficits in teachers literacy skills

43% of word meanings correct; 57% wrong

45% of summaries were meaningful; 55% wrong

40% teachers did not have spelling mistakes (in a 2sentence write up)

35% had 1-2 mistakes; 25% had >=3 mistakes

Only 50% of teachers could spot >3 mistakes in a write up
in which we had deliberately introduced 6 mistakes.
TASK 6 : SPOTTING & UNDERSTANDING
COMMON MISTAKES IN ARITHMETIC
Samples of children’s work shown. Teachers asked to choose…
Deficits in teachers’ numeracy skills

78% could spot correct one when presented
a sample of three simple division sums

24.5% could do a percentage sum; 75% not

27.9% could do an area sum; 72% not

About 20% said they never had problems in
addressing the maths queries of their pupils
How this evidence and TET evidence helps



School-TELLS (2008) & ‘Inside Classrooms’ study (2011)
highlighted deficits in T knowledge & ability to teach
This evidence contributed to decision to introduce the
TET
TET objective to vet applicants, ensure competence
 Abysmal pass rate - 0.4 to 3.7% pass rate
Evidence in this T assessment is
extremely valuable

The evidence in this T assessment is extremely valuable
It helps to identify the training needs of teachers.

It can inform policy makers who decide training curricula

But have states used this evidence in this way? States
need the desire to analyse; make use of this evidence; a
transparent approach

Congratulate the CAER for analysing C-TET – this is
evidence based policy making (the policy maker CBSE
sought evidence – gave data)
Is this evidence relevant only for govt and
rural private schools?


Is this problem of low cognitive skills of Ts confined to
government primary teachers ?
Clearly more generic problem –


Even highly paid govt T have major deficits in skills
TET evidence

Private schools cannot be complacent on this

Testing T can help to assess the training needs of each T
The importance of evidence





There are many initiatives to improve education
NFE
TLC
SK
DPEP
(1982)
(1988)
(1987)
(1993)
OBB
MDM
LJ
SSA
(1986)
(1982)
(1988)
(2003)

Aadhar, ABL, MLE, Nallikali, Nai Disha, Read India, RTE

Have these programs had impact? – little evidence to judge
Efforts to improve more successful, when based on evidence
Culture of seeking evidence

Why base decisions on evidence?

While poor quality schooling does not threaten lives, it
seriously affects people’s quality of life, and even longevity.

In medicine, its unthinkable without thorough testing by
experts, & the use of most robust, expensive randomised
control trials; But in Education, Ministers freely make policies
without consulting evidence / experts

In good educ systems, and ideally, policy will be not made on
supposition, ideology or political expediency. Govt seeks
evidence
UK policy makers’ use of evidence from
impact evaluations

Early intervention - quality & effectiveness of pre-school experience
in securing better long term outcomes was used to justify more
investment (Sammons et al, 2006)

Class size - evidence of no stat difference was used to justify not
extending the policy into later years (Blatchford et al, 2002)

Formative assessment - effect sizes in attainment (alongside pupil &
teacher perspectives) led to inclusion in national policy (Wiliam et al,
2004)

Education Maintenance Allowance - evidence from pilot study of
post 16 retention led to national roll out, though long-term
sustainability was worse in pilot areas (Middleton et al, 2005)
Policies could be better, if evidence based

The narrative in the 6th Pay Comm for across-the-board doubling
teacher salaries (without increased accountability) was : it will
motivate teachers. Was this effective? Did it raise teacher effort? No
one checked. [next slide]

A state govt recently announced it would regularise 176,000 para
teachers, in the name of quality of education; it did not look at the
relative effectiveness of regular and para teachers (3 papers)

Under RTE it has been made mandatory for teachers to have B.Ed.
Certification, reduced PTR to 30, and many inputs mandatory; where
is the evidence for this? no pilot testing

If there were garnering of evidence on the impact of policies, then
ineffective policies could be weeded out
SchoolTELLS survey (2008)
Higher resources, lower effort
UP
Reg.
Para
Priv.
12,017
(27,000)
3,000
(3,500)
940
(1400)
Absence rate
24.6
12.0
17.4
% time teaching
75.3
83.3
89.0
Salary/month, 2008
th
(today after 6 Pay Comm.)
Structure of accountability matters more than resources
What kind of evidence?

What kind of evidence is useful / acceptable?

Distinguishing correlation and causation

The importance of methodology

Using the force of the federal chequebook to nudge
researchers to use robust methods capable of yielding
causal inferences.
Interest in evidence of impact

Field of medicine long interested in evidence of impact

Recent upsurge of interest in impact evaluation in many fields

nutrition, labour, governance, rural development, education, poverty

‘International Initiative for Impact Evaluation’ (3ie) established 2008

Network of Networks on Impact Evaluation (NONIE), comprised of:




the OECD/DAC Evaluation Network (DACEN),
the UN Evaluation Group (UNEG),
the Evaluation Cooperation Group (ECG), and
the International Organization for Cooperation in Evaluation (IOCE)

International Development Evaluations Association (IDEAS)

Development IMpact Evaluation (DIME)

Spanish Impact Evaluation Fund (SIEF)
What is impact evaluation?

Measuring outcomes (eg learning) is difficult

But measuring outcomes is NOT impact evaluation

measurement of net effects of a program on the
outcomes of interest (e.g. on learning)

In IE, it is important to appreciate the difference
between correlation and causation
Correlation vs Causation

As ice cream sales increase, number of drowning
deaths increases sharply


Sleeping with one's shoes on is strongly correlated
with waking up with a headache


Therefore, ice cream causes drowning
Therefore, sleeping with one's shoes on causes headache.
Taller people have higher earnings

Therefore, higher height causes higher earnings
Salary and Height
Salary and Height?
Spurious Relationships


This relationship is said to be “spurious”
When we did the bivariate relationship, we said,
Height

Salary
In reality, things look like this:
Height
Salary
Gender
Think of
Sarva Shiksha
Abhiyan
and its impact
Correlation vs Causation

In achievement equation with
class-size, the coeff on class-size
var is -0.46, i.e. an increase in
class-size by 10 is associated
with a 4.6 point lower ach mark
Can we say from this that:
 class-size is negatively related
to achievement?
 reducing class-size will lead
student achievement to rise?
 what is the diff between these?
Class size graph
mark

80
70
60
50
40
30
20
10
0
y = -0.4635x + 76.437
0

what else determines ach?

where is that included in the
way the reg. equation is written
20
40
class size
60
80
Quantitative impact evaluation
methods





OLS production function (not causal relation)
Instrumental Variables
Panel data
Randomised experiment
Quasi-experimental



Propensity score matching method
Each method has strengths & drawbacks
RCT, PSM, Panel, IV
Evidence on impact of
policy interventions

Performance related pay






Duflo and Hanna, RCT
Muralidharan/Sundararaman, RCT
Contract teachers

Atherton & Kingdon, panel data
Muralidharan et al, RCT
Goyal & Pandey, OLS, school FE
DPEP



Jalan and Glinskaya, PSM
Schmid, IV
Private schools



Mid Day Meal


Desai et al, panel
French/Kingdon, panel
Afridi, 2010, panel
Union membership

Kingdon Teal, panel
Greater emphasis on evidence

NCERT evaluated impact of programs in 4 states, under TCF

Data becoming available – EI, ASER, DISE, SEMIS, NAS

What’s imp is the quality of the data / studies, i.e. robustness of
methods/ designs; degree to which they tease out causal effect

Long term investment needed in capacity dev

MHRD / JRM approval for estb of National Assessment &
Evaluation Centre
Some important considerations

Even evidence showing reliably what works is insufficient

Policy makers need to:
 seek, read such evidence
 discern good/bad evidence
 act on evidence, i.e. make evidence-based policy

Freedom from political interference

Independence from funding body, hence PPP better
Good evidence does not always permit
good decisions

Even when we have good evidence a policy has capacity to
substantially improve outcomes, there can be powerful
political economy barriers to the implementation of policies.

E.g. Duflo and Hanna (Rajasthan) say : “Although this study
suggests that a system of automatic monitoring with enforcement by
physically remote agents who are prepared to enforce the rules is
technically feasible and indeed provides better incentives for teachers,
a later effort to introduce this system with higher-skilled, higher-status,
and more politically powerful health-care workers ran into strong
political obstacles (Banerjee et al. 2007b).” Another e.g. para T in UP

We need to understand the political economy constraints, and
how can they be eased
Thanks
VOCABULARY TASKS : DO TEACHERS KNOW WORD MEANINGS ?
Four difficult words are given. Please write their meaning using simple words
TASK WITH
DIFFICULT
WORDS
Bihar govt.
schools
UP govt.
schools
Bihar/
UP
ALL
All
Reg.
Para
05
Para
06
Reg.
Para
Private
8.8
10.8
10.2
5.0
8.3
12.5
9.4
Wrong meaning
35.5
33.4
35.4
27.2
29.3
32.0
32.2
Partial meaning
12.4
14.8
17.3
16.9
13.5
15.1
14.9
Full meaning
42.4
41.0
37.1
50.8
48.9
40.4
43.4
Word meaning
Not attempted
SUMMARIZING TEXTS : EXAMPLES FROM TEACHERS
SUMMARIZING TEXTS : CAN TEACHERS SUMMARIZE ?
TASK WITH
PASSAGE
Bihar govt.
schools
UP govt.
schools
Bihar/
UP
ALL
Private
All
Reg.
Para
05
Para
06
Reg.
Para
Not attempted
3.5
2.4
5.6
3.3
2.6
8.6
4.4
Irrelevant/wrong
25.4
27.0
40.5
28.6
40.2
37.9
33.0
Partially meaningful
25.4
15.9
22.5
9.9
16.2
16.4
17.9
Fully meaningful
45.6
54.8
31.5
58.2
41.0
37.1
44.7
Gave meaningful
summary?
SUMMARIZING TEXTS : CAN TEACHER SUMMARIZE
USING SIMPLE LANGUAGE
TASK WITH
PASSAGE
Of those writing
meaningful summary
Bihar govt.
schools
Reg.
Para
05
UP govt.
schools
Para
06
Reg.
Para
Bihar/
UP
Private
ALL
All
Summary easy to
understand
Easy to Understand
71.2
69.6
78.6
79.3
81.3
73.8
75.0
No spelling error
32.0
29.1
28.2
54.7
48.2
46.7
39.6
1-2 Spelling errors
44.3
38.5
25.6
30.2
35.5
34.4
35.3
>=3 spelling errors
23.7
32.5
46.2
15.1
16.4
18.9
25.1
Are there any spelling
errors?
TASK 4 : PERCENTAGE PROBLEM : Findings
Bihar
UP
All
Reg.
Para
05
Para
06
Priv.
Reg.
Para
Priv.
Not attempted
14.4
12.0
26.4
37.0
16.7
23.5
28.6
20.6
Incomplete
32.7
48.8
46.2
25.9
40.0
40.0
54.6
42.6
Wrong steps
5.8
6.4
5.5
11.1
10.0
3.5
1.3
5.7
Correct steps
wrong answer
3.9
6.4
3.3
3.7
4.4
7.0
1.3
4.6
Correct answer
no steps
0.0
1.6
3.3
0.0
1.1
4.4
2.6
2.1
43.3
24.8
15.4
22.2
27.8
21.7
11.7
24.5
TASK: PERCENTAGE
WORD PROBLEM
Solved correctly
TASK 5 : ARITHMETIC : AREA PROBLEM
Bihar
AREA PROBLEM
TASK
UP
All
Reg.
Para Para
05 06-07
Priv.
Reg.
Para
Priv.
No attempt
27.9
28.8
38.5
51.9
30.0
48.7
41.6
36.6
Incomplete
19.2
25.6
26.4
7.4
18.9
19.1
26.0
21.8
Wrong steps & A
5.8
4.0
1.1
3.7
7.8
3.5
2.6
4.1
Correct steps, wrong A
3.9
3.2
8.8
0.0
4.4
1.7
5.2
4.1
Only correct A, no steps
4.8
5.6
3.3
0.0
8.9
4.4
9.1
5.5
38.5
32.8
22.0
37.0
30.0
22.6
15.6
27.9
Solved correctly
80% primary school teachers have difficulties
in teaching maths
% teachers who agree with the statement
“Sometime I have difficulties in addressing mathematical queries and
problems of my students”
BIHAR
Fully
agree
UP
Parti Some
Dis
ally
what agree
agree agree
Fully
agree
Parti Some
Dis
ally
what agree
agree agree
Govt. school teachers
24.5
11.0
46.8
17.7
15.2
18.3
43.1
22.3
Private school teachers
16.7
12.5
45.8
25.0
16.9
18.5
36.9
27.7
Only about 20% of govt. school teachers believe they don’t face problems.
About 80% admit to have difficulties sometimes.
This suggests possible interest in in-service training to upgrade maths skills