Transcript Alliance talk

Cycles of Improvement and
Implementation of Reforms
Knowledge Alliance
August 3, 2010
Marshall S. Smith
One moment for Game Changers
• Technology (for Access and Quality) : 1). Courses (AP,
Language): 2). Science laboratories: 3). Games and Stories and
simulations for Social Science, humanities: 4). Music learning
and practicing and creating: 5). Reading (lots of practice: 6).
Special Education.
• Continuous Process Improvement.* Get better and better.
• Common standards, curricula, assessment. A big step. Raise
all boats. *
• Aggressive Early language development: One of three
change keys to closing the gap!! Technology is the second,
human support is the third.
Two Short Stories
• Singapore teachers evaluation system
• But before being formally evaluated…
• Black and Wiliam: Formative Assessment
• Cycle of Improvement:
Singapore and B&W
Execute
immediately
Assess
Evaluate
Personalize
Plan
Engage
participant
REPEAT RAPIDLY
Execute
immediately
Assess
Evaluate
Personalize
Plan
Engage
participant
Lots of examples of methodologies
with rapid feedback loops
• Plan-do-study-Act (IHI): multiple cycles, differentiation.
• Developmental Evaluation: Complexity orientation. Social
Systems as CAS – complex adaptive systems: Double Loop
Learning: find cause, not solve problem. Involve participant.
Continual learning. Evaluator is part of a team whose members
collaborate to conceptualize, design and test new approaches in
a long-term, on-going process of continuous improvement,
adaptation, and intentional change. Michael Q Patton
• Realistic Evaluation: Theory based – causal mechanisms.
External validity. Not “what works” but What works for whom
and under what circumstances?” Causal mechanisms interacting
with context! Continuous exploration. Involve participant.
Fine wine in a new bottle? At the
least, good company!
•
Karl Popper: Piece meal social engineering. The only course for the social sciences
is to forget all about the verbal fireworks and to tackle the practical problems of
our time with the help of the theoretical methods which are fundamentally the
same in all sciences. I mean the methods of trial and error, of inventing hypotheses
which can be practically tested, and of submitting them to practical tests. A social
technology is needed whose results can be tested by piecemeal social engineering.
(Popper 1945, p.222)
•
Donald Campbell: (T)he experimenting society is a process utopia, not a utopian
social structure per se. It seeks to implement that recommendation of Popper’s,
“A social technology is needed whose results can be tested by piecemeal social
engineering.’ (in Campbell and Russo 1999).
•
Reinventing Research Capacity-- rapid cycle development Bryk and Gomez
Are all of these methodologies derivations of
Continuous Process Improvement (CPI)?
•
•
•
•
Examples of CPI: Long Beach, Austin, Singapore
Process really matters!!
“Problem solving puts out fires – CPI gets at causes.”
Characteristics in school systems:
•
–
–
–
All about execution!!
longevity important
Feedback loops: cycle of improvement
Data systems necessary - not just on student achievement
- link to back office systems.
– Intervention (rapid). Need leader prepared to act.
– Kaizen – all functions: everyone engaged
– Accountability and transparency
More on Continuous Improvement
• Sometimes metrics wrong
• Sometimes goals wrong
• Ability to change course rapidly – reset goals and strategies.
New knowledge.
• Supports Innovation? – necessary for successful innovation?
• Supports implementation of new policies?
• Is good implementation a derivative of continuous
improvement?
• Is CPI aspirin?
What is happening?
• Iterative models replace straight-line models.
• Complexity / chaos replace a priori predictability.
• Improvement and understanding replace simple
measurement of effect size in one setting.
• Participants role adaptive rather than passive.
• Personalization / differentiation replace 1 size fits all.
• Theory plays large role – identifies causal drivers and
hypotheses– guides understanding of role of context.
What about Implementation?
• Pressman and Wildavsky (1973: xiii-xv), "Policies imply
theories... Policies become programs when, by
authoritative action, the initial conditions are
created…Implementation, then, is the ability to forge
subsequent links in the causal chain so as to obtain the
desired result.
• The limitations of the 'classical' model, however, began
to be highlighted in the post World War-II period as it
became apparent that public policy worked less as an
efficient and orderly machine and more as a process of
"muddling through" (Lindblom, 1979).
•
http://webcache.googleusercontent.com/search?q=cache:http://unpan1.un.org/intradoc/groups/public/documents/AAPAM
/UNPAN025756.pdf
Other Thoughts about implementation
• What types of challenges: Adaptive v technical!! Adaptive
policies that involve human beings – single human beings
complex – 30 students and one teacher incredibly complex.
• McLaughlin: we need a third generation of implementation
analysts—integrating the macro world of policymakers with
the micro world of individual implementers.
• Implementation is 90% of the ball game:
Let’s take a simple case:
Pakistan: Implementation of National Policy
• Pakistan: A little background.
– Surrounded by Iran, Afghanistan, China, India, Arabian Sea
– 180 million people: by 2045, 350 million, the youngest
nation
– Continually at war internally – continual threat externally
– Over 50% of GDP to military
– Relatively very low literacy rate
– Official language Urdu, spoken by less than 10%, English
language of elite; many local languages.
Pakistan Education System
• A few facts:
–
–
–
–
–
–
–
–
–
Average years of schooling: 3.9 for people age 15 +
Approx 250,000 schools, 35-40% private schools.
Federal system: 4 provinces+: districts inside provinces:
Federal money goes to provinces as part of a block grant
Education Spending somewhere between 2.1 and 2.9% of
GDP -- 4.5% is neighborhood average
Public school Teachers effectively tenured on first day of
teaching.
Infrastructure very weak on all counts.
2009 Legislative plan -- quite reasonable
2006 National curriculum standards. – also reasonable.
Challenge: To implement 2009 legislative plan
and 2006 curriculum frameworks
• Pakistan-UK task force (PETF): Prime Ministers: 10/09
• Provinces buy in – early skepticism from NGOs, other donors,
some bureaucracies. Karachi.
• Fifth Meeting early July.
• Implementation: Federal to Provincial: Provincial to District:
District to School and Community.
• Focus of PETF: Federal to Provincial: creating capacity for
Provincial to District and District to school: differentiated
strategies and processes
• Barber: Delivery is: goal + implementation + outcome. Use
delivery methodology.
The Task Force Scorecard
▪
The Scorecard is a tool to monitor implementation of the National Education
Policy
▪
The Scorecard shows:
– The main actions which need to be taken by each province to implement the
Policy (on the left hand side of the Scorecard)
– The current state of implementation of each action in each province
▪
The Scorecard will be updated every two months in consultation with the
provinces and the Ministry of Education. Updating the scorecard should be
simple and fast, and will not require detailed analysis.
▪
The Task Force scorecard is designed to:
– Clarify exactly what needs to be done to implement the National Policy
– Monitor progress during implementation
– Increase accountability for progress
Standards
Monitoring
Empowerment
Teaching
Public-Private
Partnership
Capacity
Facilities
▪ Curriculum standards for student learning and progress
▪ Standards for school performance
▪ Standards for school district performance
▪ Provincial assessments of students in primary school
▪ Provincial assessments of students in secondary school
▪ Objective monitoring of school performance
▪ Objective monitoring of school district performance
▪ Management and publishing of performance data
▪ Clear expectations for parents of schools
▪ Accessible means for parents to give feedback on schools
▪ Provision of all students with good textbooks
▪ Provision of all teachers with good teacher guides
▪ Provision of good training for every teacher
▪ Merit-based selection of teachers
▪ 10% of education budget to low-cost non-government schools
▪ 1% of education budget to NGO support of government schools
▪ Capacity development of school heads
▪ Capacity development for district education managers
▪ Capacity development for provincial leaders
▪ Provision of basic facilities in every school
Balochistan
NWFP
Sindh
Punjab
Pakistan Education Scorecard (Provinces)
Levels
0
1
2
Scoring system
Criteria for grading
This item is not implemented in accordance
with the scorecard criteria, and there is no
plan to implement/ improve it
This item is not implemented in accordance
with the scorecard criteria, but there is a plan
to deliver it
Implementation of this item is underway, but
is still in early phases
▪
▪
There is a plan in place with all of the following:
▪ Sufficient detail to guide implementation
▪ A timeline with milestones identified
▪ Named leaders with their precise roles and
responsibilities
▪
▪
▪
▪
3
4
Implementation of this item is between 20%
and 80% complete
Implementation of this item is complete
The item is not implemented; or implementation is
significantly different from the scorecard criteria (e.g.
assessments are in place, but they test only a small
sample of students)
There is no plan; or the plan is incomplete
The item has an approved budget
A full-time team is working to implement this item
Significant implementation activities are underway
(e.g. recruitment of staff, writing of standards,
development of systems)
Rollout or launch of the item has not yet begun
▪
▪
▪
Program is operational but is not yet at scale
Program can demonstrate impact at the school level
Rollout, launch, or mass communication of this item is
underway (e.g. testing is underway; standards are
being communicated)
▪
This item has been largely implemented in line with
the scorecard criteria and covers at least 80% of the
target school population
The focus has now shifted to improvement and
refinement
▪
How the Score card works
1. Curriculum standards for student learning and progress
Yes
No
a
Are there curriculum standards stating what students should learn in each subject?
o
Go to b
o
Go to e
b
Are the curriculum standards sufficiently detailed for teachers to understand what
students are expected to achieve?
o
Go to c
o
Go to e
c
Are the curriculum standards written clearly enough for teachers to understand them?
o
Go to d
o
Go to e
d
Have standards been widely communicated (does every school have a copy)?
o
Score 4
o
Score 3
e
Is there a plan to create standards which will achieve (a-d) above?
o
Go to f
o
Score 0
f
Is the plan detailed enough to guide implementation and has it been communicated?
o
Go to g
o
Score 0
g
Does the plan include a clear timeline and milestones?
o
Go to h
o
Score 0
h
Does the plan define clear responsibilities for each item and are these understood?
o
Go to i
o
Score 0
i
Has implementation of the plan been started?
o
Go to j
o
Score 1
j
Is the budget approved and funds available?
o
Go to k
o
Score 1
k
Is there a full time team working on implementation?
o
Go to l
o
Score 1
l
Is writing of the standards underway?
o
Score 2
o
Score 1
Curriculum standards for student learning and progress
The Work
• Secretaries of Provinces and their staffs – key players.
• In March and April: baseline evaluation agreed by Secretaries
with PETF Secretariat
• Key items selected to work on: Analysis of resources, time, …
needed: Specific plans and strategies developed and actions
initiated at Provincial, district and school levels.
• Evaluation process iterated—evidence of change developed
and assessed.
• PETF working on aligning donors.
• Low Hanging Fruit for provinces – some gains by July.
• Provinces need to push down implementation.
Standards
Monitoring
Empowerment
Teaching
Public-Private
Partnership
Capacity
21
Facilities
▪ Curriculum standards for student learning and progress
▪ Standards for school performance
▪ Standards for school district performance
▪ Provincial assessments of students in primary school
▪ Provincial assessments of students in secondary school
▪ Objective monitoring of school performance
▪ Objective monitoring of school district performance
▪ Management and publishing of performance data
▪ Clear expectations for parents of schools
▪ Accessible means for parents to give feedback on schools
▪ Provision of all students with textbooks
▪ Implementation of the 2006 curriculum
▪ Provision of all teachers with good teacher guides
▪ Provision of good training for every teacher
▪ Merit-based selection of teachers
▪ 10% of education budget to low-cost non-government schools
▪ 1% of education budget to NGO support of government schools
▪ Capacity development of school heads
▪ Capacity development for district education managers
▪ Capacity development for provincial leaders
▪ Provision of basic facilities in every school
Balochistan
Khyber
Pakhtunkhwa
Sindh
Provinces, 30 June 2010
Punjab
PETF Scorecard
Status of System Reform in the US
• By 9/30/10 thirteen to fifteen states will begin to implement their RTT
plans around the 4 pillars.
• Another 20-25 states will not win grants but will have reasonably good
plans, partners, timelines, commitments etc.
• Close to 40 states with serious plans will face serious transition issues
as well as straight implementation problems for to make new systems
work.
• What do the states do? Is there a systematic approach? How might
the R&D system support/help all 35-40 states that are committed to
change? National to state: State to Local: Local to Schools.
• ED planning to form a unit to support and track delivery. This is a new,
critical role. From program monitoring to supporting system change.
7/18/2015
22
Implementing state plans around four pillars!
Think Systemically and by pillar simultaneously
• Common standards, far better assessments, potentially
aligned with powerful curricula.
• Powerful teaching (through improving teacher
performance) (human capital, social capital, supporting
materials including curricula). TIF
• Data systems: Spanning – Preschool – Gr 16: Lead to
Continuous improvement/learning systems approaches
to management and delivery. Link in teachers. SLDS
• Focus on improving education for students in lowest
performing schools (5%, 5 thousand schools). SIP
Why Common Standards?
•
Against: Why not stay the same?
– All the eggs in one basket - what if the quality is weak?
– Lose competition among states.
– Creates standardization -- destroys innovation and imagination.
– My state already has great standards. Loss of ownership
– Fear of politicization
•
For: If change the following can follow.
– High Quality by expert judgment, built on recent research, benchmarked.
– Efficiency/quality in developing aligned assessments, professional
development, evaluations, curriculum , textbooks: costs can be pooled
– Education marketplace rationalized. Innovation should be stimulated.
– Comparisons in achievement simplified
– Creation of national networks of teachers – lesson planning on steroids.
– Curriculum and curriculum materials can be truly aligned – can have multiple
curricula – embedded formative assessments.
7/18/2015
24
Hypothetical next phase
• ED organizes: Procurements change; Tracks
progress/ transparency; shares ideas
• States engage; create delivery units.
• Existing and new NGOs focus on state reform
implementation. NGOs work together.
• States work together on curricula–share costs.
Who creates?
• R&D and innovation thrive in new markets.
National Challenge
• How can we make the next phase happen?
• How can the Knowledge Alliance support and
help lead a common, very difficult effort?
• Where does research, development.
evaluation, and innovation fit in?
• What are the next steps?