Who We Are

• Naomi Hupert and Tomoe Kanaya • Education Development Center/Center for Children and Technology--a non-profit education research institute. www.edc.org/CCT

Some things we’ll touch on in this presentation

• Why we think talking about Reading First evaluation is important • What we’ve learned (informally) about states’ Reading First evaluations • Some interesting challenges we’ve encountered in planning a Reading First Evaluation • Some early trends we are seeing in our evaluation work

Federal guidance for Reading First evaluation*

• 1. Implementation Evidence • 2. Achievement Gains • 3. Program Effectiveness • 4. Reducing Students Reading Below Grade Level Statewide * From Guidance for the Reading First Program, US Department of Education, April 2002

What we’ve been hearing about Reading First and evaluation:

Through informal conversations with evaluators, technical assistance providers, RF directors, and others we’ve collected the following information: – States are postponing their evaluation activities until their programs are underway – States are finding they are not equipped to collect and analyze the information necessary for their evaluations – Many states intend to identify outside evaluators – While most states’ RF

proposals

indicate implementing summative evaluations only, our

conversations

indicate states are interested in formative evaluation as well

A few things to consider when putting an evaluation plan into action:

• What approach best meets the needs of the implementation plan: summative, formative and summative?

• What kinds of data are available? (and will you need to supplement with additional data collection activities?) • What is the data collection and reporting capacity of your participating schools and districts?

Summative Evaluation

• Focuses on program impact • Measures success based on criteria set forth by the Federal Government and a state’s Reading First proposal • Keep in mind: “impact” can mean more than change in student achievement, it can include change in classroom practice, change in teacher knowledge, change in school and district approach to reading

Formative Evaluation

• Primary goal is to support and inform program growth, development and improvement • Provides ongoing feedback loop from schools to state Reading First program staff • Focuses on implementation • Requires a team effort--evaluators and program staff and participants form partnership

Where to find Data:

• Some possible sources of data to inform the evaluation process: – Literacy assessment tools – State developed early screening or assessment tools – Publisher generated measures of student literacy skill – Student data collected by school, district, state: • Attendance • Library use • Referral to Special Education • English Language Learner status in relation to literacy acquisition • Student mobility

Other sources of data:

• Interviews with relevant educators: – Teachers – Principals – Superintendents – Coach/professional development staff – Literacy specialists • Observations of relevant activities: – Professional development events – Classroom activities – Coaching

Data Collection:

• Options for data collection assistance • In school: – Reading first coordinators – Teachers – School-level administration – District-level administration • Non-school: – Outside evaluator – Graduate students/program collaboration w/preservice – Parents/school volunteers

Some interesting challenges we’ve encountered in planning a Reading First evaluation:

The challenges can be grouped into two categories: 1. The data analysis perspective 2. Other things…

Our Goal: To monitor students’ progress over time

ACCURATELY

1. We need a large number of students (sample size).

2. We need to ‘measure’ as many contextual variables as possible 3. We need the ability to follow individual students over time 4. We need to be patient

1. Sample Size

• When the sample size is small, the results may be due to just a few unique individuals.

• The more students we have, the more confident we can be that the results are real and ‘generalizable’.

2. Contextual variables

• Students’ improvement may be due to various contexts that surround the child, rather than the Reading First initiative. • Therefore, it is important to ‘measure’ and account for all potential contextual variables that could impact literacy.

– Examples: • Student-teacher ratio • Exact age of student • Baseline performance of student • Other initiatives that may support literacy skill development

3. Following Students

• Measuring classroom/school level performance is not enough • It is difficult to determine improving if we cannot follow individual students at each testing

how

students are • We cannot determine the nature of the improvement (e.g., which contextual variables matter for which particular students) without following individual students over time

Example:

Classroom A and B both improved 10% after 1 year in Reading First Initial Conclusion: Classroom A and B performed equally well Conclusion after following students: • In Classroom A all girls improved 20% while boys had 0% improvement • In Classroom B, every student improved 10%

4. Patience, Patience, Patience!

• Changes and improvements do not happen overnight—or over 1 testing cycle • Small changes at an early age can eventually turn into large differences later in life

Evaluation planning: other things to keep in mind

• Student mobility • Issues around student and teacher privacy • Issues around data collection and data entry

A few more things:

• too many interviews, questionnaires, and surveys • students assessed too often • who administers assessments to students • the challenge of creating an evaluation design that is rigorous yet compatible with a Reading First program design

More things…

• developing partnerships with teachers and schools • making evaluation “safe” • the formative loop

Our work in New Mexico

• Year 1: 32 schools in 10 districts (Approx. 30 schools will be added in year 2 and 30 more in year 3) • Year 1: 491 K - 3rd grade teachers administered DIBELS assessments during November, 2003 and February, 2004 – 7097 students were assessed in November, 2003 – 7586 students were assessed in February, 2004 – 4924 students for whom we have complete and matched data from both assessment periods

Assessment analysis plan

• Fall, winter and spring assessments • First assessment may be inaccurate/unreliable • Look at movement of individual students from one support recommendation level to another – (ie: from Benchmark to Strategic Support) • Examine demographic data in relation to student assessment findings • Match DIBELS findings with school and/or district program and implementation

Assessment: Some trends

• Nearly all districts have implemented some kind of professional development to support DIBELS and TPRI – Administering the DIBELS and TPRI was nearly universal by first assessment period – Using the Palm Pilot to administer the DIBELS and TPRI was adopted by nearly every RF teacher – Using the DIBELS and TPRI data in some way to inform classroom instruction, or to raise questions about past practice, is rapidly spreading to all RF schools

Assessment: Trends

• Professional development appears to be more targeted to teacher need in schools with a dedicated instructional or literacy coach, or with a geographically close cadre member.

Assessment: Trends

• Teachers are changing what they are doing in classrooms • Teachers are looking at their students’ assessment data • Teachers are making instructional decisions based on assessment data • Professional Development providers (cadre members, coaches, or staff developers) are tailoring their presentations to address teachers’ questions about using assessment data

Professional Development

The majority of Reading First teachers had access to a wide range of professional development activities provided by: • The state • District level staff • School level staff • Outside Consultants/Experts in Literacy • Publishers/Vendors

Professional Development: Trends

• In some schools/districts the systematic use of assessments, and the expectation that assessment data will guide instruction has begun to guide professional development • Teachers are asking for more information about how to teach those students most in need of support • Schools are reaching out for help in addressing the needs of their most struggling readers

Professional Development: Implications

• Interest in cross-school and cross-district collaboration and information sharing • Interest among Principals for meeting with other Principals • Interest among Reading First coordinators in regional meetings to share with other schools different approaches to Reading First • Interest among sites in increased offerings of Professional Development from State, particularly among smaller districts, and preferably offered regionally

Role of NM Cadre Member (Lit. expert assigned to NMRF school) : Some Trends • Geographic proximity greatly improved the significance of a coach/cadre member’s role in a school • Schools with an active cadre member/coach appeared to be more likely to discuss use of assessment data to inform classroom practice

School Level Leadership • Principals: express high levels of support, but report lack of adequate knowledge about the program • Planning time: The majority of schools have instituted new or extended planning times for Reading First teachers, either school-wide, or across grade levels (depending on size of school) • Study groups: Many schools have instituted a range of activities that engage teachers in discussion about key issues of reading first

School Level Leadership: Professional development on classroom level • A combination of: – 1) support from principal, – 2) time to plan, and – 3) time to engage in pedagogical discussion support an atmosphere of ongoing professional development that focuses on supporting students’ reading development

Issues particular to states with large numbers of very small school districts

• Equity in program supports • Funding issues in relation to school size, location and resource availability • Access to professional development opportunities, resources, information, and exposure to new ideas

Some Technical Assistance resources for Reading First: • NCREL http://www.ncrel.org/rf/ • SEDL http://www.sedl.org/readingfirst/welcome.html

• RMC Research Corporation: 1-800-788-1887

Education Development Center, Inc. Center for Children and Technology • To learn more about our organization, go to our web site: – www.edc.org/CCT/ • Contact information: – Naomi Hupert [email protected] 707-829-8532