Education in Conflict Affected Fragile States: Global Evaluation 2008-2010

Nepal Presented by Dr Dragana Sretenov, Learning and Impact Assessment Adviser, Save the Children UK April 2009

Presentation Outline

1.

2.

3.

4.

Global Terms of Reference Methodology Findings in 4 Countries Data Use Plan Note: Dr.Ruth Naylor is the consultant - Global Researcher leading this research.

Angola

Global Terms of Reference Research Questions & Country Focus 1) How have Save the Children’s project level interventions contributed to quality primary education for children affected by conflict?

2) Which project level interventions have had what impact on the education quality of children affected by conflict?

Afghanistan: Child & Parent Participation Nepal: Schools as Zones of Peace Southern Sudan: Teacher Training Angola: Teacher Professionalism

Global Terms of Reference

Objectives

– – – Contribute to technical knowledge about what works and why Contribute to internal and external documentation used in advocacy, communications, and fundraising Contribute lessons learned into the internal organizational and operational decision making process

Audience

– Donors* – – – External Communities of Practice* Governments & International Bodies Headquarters policy advisors and country office programmers

Timing

– 2008 Midterm (formative) 2010 Final (summative)

Global Terms of Reference In Country Methodology • Design: in-country participatory evaluation design and implementation. Country specific • Key Informants: students, child clubs, teachers, PTAs, partner organizations, parents • Data Collection: focus groups, lesson observations, interviews, learning assessment, secondary data • Sample: 6 schools per district, 3-5 districts per country

Global Terms of Reference Participatory Design Agenda 1.

2.

3.

4.

5.

6.

7.

8.

9.

Identify Research Team Identify Advisory Committee Narrow scope: effective, unique, innovative Select intervention package Map intervention program theory Identify indicators for program theoretical framework Identify informants Develop tools Pilot & Capacity Building

Global Terms of Reference Tools and Sample Data Collection Plan (Angola)

Name of tool School data collection Lesson observation Learning Assessment Student exercise Director interview Teacher interview Supervisor interview CPEE focus group CCPC focus group description

Objective data about the school based on observation and school records Observation checklist Oral questions to one child at a time Group discussions on what students like and dislike about their teachers Structured interview

Respondent group

School director

Zaire

Headmaster can assist,/ check documents/ school observation 7 schools One for a lower grade (1-3) lesson one for a higher grade (4-6) 3 girls and 3 boys chosen at random from grade 3 per school 4 girls and 4 boys from oldest class available 14 42 2 7 14 Structured section followed by in depth questions Structured interview 2 teachers per school and one in depth supervisors Focus group/ interview Focus group during longer school visits

Uige

30 2 5 10

Kwanza Sul

5 school s 10 6 ALP 6 36 2 Not applicable 6-12 Approx4 3 Appro x 4 2 3 Not applicable Focus group/ interview Focus group during longer school visits 3 1 (Not applica ble?) Not applicable

Total

18 8 5 12 36 4 30 102 6

Global Terms of Reference

Questions?

Afghanistan Nepal Southern Sudan Angola

Nepal: Schools as Zones of Peace

Program Theory: Schools as Zones of Peace

Save the Children Partners

Child Club Village Child Protection Committee School Management Committee • Bring together community members and political groups • Develop, agree on and display

codes of conduct

Items on codes of conduct: no armed activities in school; no weapons in the school; no use of children in political activities; no political meetings or graffiti in school; no arrest, abduction and torture of children and teachers; no school closure for meetings; no corporal punishment; no cattle grazing; no alcohol or smoking in school.

Nepal: Successes of SZOP

•

Increased sense of security in schools “

Though some fear is still there with ongoing conflict, we feel safer than earlier and also do not feel fear while sending our children to school.” Parent, Mahottari • • • • •

In 2007 project schools on average were open 12 days more than comparison schools (of 220 days) Informants at over half of project schools reported improved student and teacher attendance Evidence of beating was observed in one project school only

“Instead of beating children, teachers now support and counsel them.” Parent, Palpa

11 out of 16 project schools had child representatives in the SMC 12 out of 16 project schools had developed thier own codes of conduct

Nepal: Challenges

• • • Monitoring systems for SZOP not well established Difficult to establish SZOP in contexts with limited experience of community participation in school development Ensuring that SZOP is accepted and respected by political parties and armed groups where conflict is ongoing

Methodological Challenges Nepal

• • • • • • Partners implementing at the school level need to be involved in the design workshop Sampling plans must adapt to local conditions (weather, migration, and conflict) Time for staff capacity building should be built into piloting.

More emphasis needed during design on participatory research methods for working with children School level data from districts to be collected prior to researcher arrival in country and made available at workshop Checklist sent to country offices giving requirements in terms of what needs to be done before, during and after the visit, who should be involved and for how long

Afghanistan: Child and Adult Participation

Afghanistan: Child and Adult Participation for Quality Education • • •

Children’s participation in school

Student Councils Student representation in PTA Student led education survey • • •

Children’s participation in learning

Child-to-child health education Training teachers in participatory methodology Resource centres, mobile libraries •

Parents’ participation in school

PTA training and support

Afghanistan: Successes

Student group and PTA success stories

• Follow up on absent students • • • • Recruitment of new teachers Construction/maintenance of classrooms, boundary walls, latrines and school water supplies Reduction of corporal punishment Children learn about health through teaching it Training provided by Save the Children was more effective when it was followed up by support to carry out a concrete action

Afghanistan: Challenges

Participation of girls and women

• In two of the mixed project schools visited there were no females in the Student Councils • Only five of the 12 project schools visited had mothers in their PTAs, and three of these were girls’ schools • 12% of teachers in formal schools are female PTA training: spot the mothers

Methodological Challenges Afghanistan

• • • • • • Translation of reading assessment to local language/dialect is critical in cases where instruction is done in local language (e.g., Paghman ALC). Also, be careful about key words on assessment. Difficult to record all information. Need to have group discussion immediately following visit. Random selection of children for assessment can be difficult to orchestrate.

Security situation may cause last minute challenges / changes in plans Full course of data collection is very time consuming and can be logistically challenging, with short hours and unpredictable constraints.

Very difficult to avoid the bias of the most remote schools being underrepresented for practical reasons.

Global Evaluation: Angola

1) How have Save the Children’s project level interventions contributed to quality primary education for children affected by conflict?

2) Which project level interventions have had what impact on the education quality of children affected by conflict?

Afghanistan: Child & Parent Participation Nepal: Schools as Zones of Peace Southern Sudan: Teacher Training Angola: Teacher Professionalism

Angola: Supporting Teacher Professionalism

Putting our knowledge to use

Angola: Teacher Professionalism

Provincial Department of Education (DPE) – main partner

School directors Supervisors Other teachers

Teachers

Save the Children Parents Students

Angola: Successes

• • • • 93% of the teachers in schools visited had some training. Most teachers had been observed by supervisors and school directors Half of teachers observed had a lesson plan Respondents reported Improved teacher conduct (reduction in drunkenness and exchanging grades for money)

Angola: Challenges

• • • • Only ¼ of grade 3 students tested were able to read well enough for simple comprehension There was no significant difference found between the teaching of trained teachers and that of untrained teachers 7 cases of teacher misconduct identified (in 14 schools + 6 ALCs) 12 out of 16 student focus groups reported physical and humiliating punishment in schools

Angola: Student Learning Findings

• • • Only ¼ of grade 3 students tested were able to read well enough for simple comprehension Over 50% of students tested were unable to read a single word Most children tested could solve simple addition and subtraction problems, including word problems

Angola: Teaching and Learning

THANK YOU!

Methodological Challenges Angola

• • • Context of very rapid economic and social change including major changes in education (new school buildings, distribution of books, mass teacher training programmes). School level data on dropout and changes in enrolment is highly influenced by migration (urbanisation) and construction of new schools, and hence does not reflect quality (or access gains).

Evaluators measurement of teaching quality is restricted by low exposure to good teaching.

Southern Sudan: Teacher Training

Southern Sudan: Teacher Training

• • • • • Phase training In-service training Women into Teaching Intensive English Course Accelerated Learning Programme teacher training

Southern Sudan: Successes

According to respondents, teachers trained by Save the Children: • know how to structure lessons • adapt lessons to the learner • have increased confidence • have improved English skills • use less corporal punishment Teachers with training from Save the Children scored significantly higher in lesson observations than those without.

Southern Sudan: Challenges

• • • • • No formal follow up of the training 60% of enrolled students absent from lessons observed. Only 10% of teachers in the evaluation counties were female Only 27% of students in the evaluation schools were female.

Dropout rate for girls was 16%, and for boys 5%

Southern Sudan: Learning findings

• • • • • 19% of grade 3 students tested could not read a single word Boys score higher then girls in reading, comprehension and mathematics Multivariate analyses of the learning assessment results indicated no significant difference between the reading and comprehension scores of students from project and comparison schools In mathematics students from comparison schools scored significantly less Teacher training level did not appear to make a difference in the learning outcomes

Methodological Challenges Southern Sudan

• • • • • • There is a need for a higher level of involvement from the country office team during the evaluation design training.

Due to geographical and funding reasons the design workshop had to be separated from the training workshop for data collection, and the workshops were organised in 2 distant part of the country, which limited the possibility of the head office staff to take part in the data collection workshop.

The difficulty in undertaking the education research in Southern Sudan is also related to inaccessibility of schools in some parts of the country during the rainy session when schools are fully running. Project schools in each county were to be selected at random but flooding in two of the counties restricted the number of accessible project schools, so there was a bias towards the more accessible schools.

Comparison schools were chosen to match the project schools in terms of their distance from the nearest town as there was little other data available to base the selection on.

The enumerators had limited knowledge of teaching processes and the classroom observation tool had to be modified and simplified. The originally developed classroom observation tool was used by the international and local researcher only.

There is a need to enable greater child participation in the data review process, which proved to be very productive in other countries.

Moving towards data use

• From our evaluation we know: In Angola, around half of the children tested (project and comparison) could not read a word in a simple Portuguese text In Nepal, a quarter of the children tested could not read a single word in a simple Nepali text In Southern Sudan, of children tested, girls’ reading accuracy was 30% lower than boys (43% compared to 73%) In Afghanistan, 40% of children tested in SC supported schools could read and comprehend a simple text, whereas 1/32 students tested in comparison schools could

So what do we do now?

Evaluation Findings Data Use Plan

• • • • •

Global Evaluation Executive Summary (4pgs) Global Evaluation Midterm Report (20 pgs) External sharing of lessons learned, e.g. GMR, AEA, Education in Emergency Conference Follow up action plan at country office & headquarters Linking with other evaluations on related topics

Education in Conflict Affected Fragile States: Global Evaluation 2008-2010

Question s?

Nepal

Education in CAFS: Global Evaluation 2008-2010 1) How have Save the Children’s project level interventions contributed to quality primary education for children affected by conflict?

2) Which project level interventions have had what impact on the education quality of children affected by conflict?

Afghanistan Nepal For more information: Southern Sudan Angola – SC Denmark Gretehe Markuson [email protected]

Afganistán – SC US Rita Oswald Cristiano [email protected]

Nepal – SC Norway Inguun Nakkim Angola [email protected]

Southern Sudan – SC UK Dragana Sretenov [email protected]

Cynthia Koons, M&E Manager, Inernational Save the Children Alliance cynthia@save-children alliance.org