Systematic Methods of Evaluation and Monitoring

Early Reading First Conference Wild Horse Pass Chandler, Arizona October, 2004

• • • • • •

ACE Partners

Gadsden Elementary School District- Lead Fiscal Agency Cocopah Indian Tribe Head Start Somerton Elementary School District Head Starts WACOG identified Head Starts Arizona State University- Lead Curriculum and Alignment Southwest Institute- Lead Community Development and Evaluation

Faculty and Staff

• • • • • • Olivia Zepeda- Project Director James Christie- Co-Proj Director Jay Blanchard- Co-Proj Director Tanis Bryan- Co-Proj Director Karen Burstein- Evaluator Robert Bernhart, Business Director • • • • • • • • • Mentor Elizabeth “Bibi” Garza Mentor Lolinda Lee- Mentor Roselia Ramirez- Mentor 5 Parent Liaisons 16 Head Start Teachers 19 Instructional Aids Technology staff from ASU

• • • • • • •

Academic Centers of Excellence ACE*

Rosalia Ramirez, Teacher Mentor, WACOG Marla Chamberlain, Teacher Mentor, Gadsden Schools Brenda Casillas, Teacher, WACOG Connie Felix, Teacher, WACOG Tanis Bryan, Karen Burstein, SWI James Christie, Jay Blanchard, ASU Cevriye Ergul, Pen-Chiang Chao, ASU/SWI • *ACE is a collaborative project between Gadsden School District, WACOG HeadStart, Somerton School District, Cocopah Indian Reservation, Southwest Institute for Families and Children with Special Needs (SWI), and Arizona State University

Evaluation and Monitoring

• Standardized, Norm Referenced – Twice annually – All participants and a matched control group – Measure the progress of the PROJECT • Curriculum-Based Measurement – Weekly – A small group of children form each class based on standardized test results and teacher report – Monitor and adjust instruction for children

STANDARDIZED ASSESSMENTS Individually or group administered Compares individual to group Used for program evaluation and planning CURRICULUM BASED ASSESSMENT Individually administered Compares individual to curriculum Used for individual evaluation and instructional planning

EFFECTIVE PROGRAM EVALUATION AND MONITORING CONSITS OF A BALANCE BETWEEN STANDARDIZED NORM REFERENCED AND CURRICULUM-BASED ASSESSMENTS

Standardized Testing

ACE

3

Assessment

• Initial assessment: We want to know what children know at the beginning of the project in the areas of: – Oral Language- understanding of language and spoken vocabulary – Phonologic Awareness- recognizes/ demonstrates that language is made up of individual sounds – Alphabet Knowledge- demonstrates the relationship between sounds and letters – Concepts of Print- demonstrates how to use printed materials, e.g., top of page, bottom, beginning, end, word v. letter

• • • • Standardized assessments are professionally developed tests administered under standard conditions, producing scores that can be used to evaluate programs. The type of standardized test required by ERF is designed to determine whether groups of children within a program are progressing or meeting the standards of learning as compared to children not engaged in ERF programs. Scores of standardized tests also indicate how children within each tested program in each site performed, how groups performed, and how groups of children across the state performed. NCLB requires that scores for schools and districts be disaggregated so that the performance of children from different subgroups can be examined. Standardized tests aligned with state standards are essential for administrators to determine whether schools/sites are meeting their goals under NCLB.

– Standardized testing achieves standardization by norming practices, machine scoring of multiple-choice questions, precise instructions for administration, and standard formats for tests and recording of responses. The results can then be used to draw inferences about the state of cohorts or individuals as compared to an established standard.

Standardization Procedures

• • • • • • Pilot testing Controlled environment (no distractions) Trained observers, evaluators Written administration instructions Guidelines for motivation of children/ question responses Cross-check procedures for scoring and data input

Assessing Measures

• • • Make sure that the tests measure what you want them to measure (content validity) Choose measures that have been validated on populations of children who have like characteristics of age, language, cognitive ability, and regional background Train assessors to administer tests and compare the results of different assessors so that controls are in place to eliminate the effects of these different examiners.

ACE Measures

Individually Administered, Twice Annually, Approximate Time 25 minutes per child.

• • • • • • IDEA Individual Proficiency Test- Oral Language, English Proficiency Peabody Picture Vocabulary Test- III- Receptive Vocabulary Predictive Assessment of Reading- Vocabulary, Phonemic Awareness, Rapid Object Naming (Spanish and English) Head Start Letter Naming Get Ready to Read- Vocabulary, Concepts of Print, Letter Naming, Phonemic Awareness Get It, Got It, Go! (Baby DIBLES) Vocabulary, Phonemic Awareness

Norm Referenced Tests

• • Standardized tests, the results of which compare individual scores to those of a norm, or group average. The average is typically gathered from individuals of like age, Results are reported in scores such as (per)centiles, stanines, normal curve equivalencies,

Standardization is of little importance if the results of assessment are to be used in isolation from all other factors. In other words, if the purpose is simply to learn about the state of a single subject/child, a unique assessment (CBM) might be devised to furnish the information desired. However, if the assessment is to be used for the purpose of comparison, generalization, or decision making, then standardization is essential.

Monitoring Children’s Learning

• • A major responsibility of educators is to teach children academic skills.

Teaching includes: – Monitoring whether children are learning what is being taught and – Making changes in instruction when children are not making adequate progress.

Testing to Monitor Children’s Learning

Testing is one way of monitoring what children are learning.

– When professionals and the public think about testing, they tend to think in terms of standardized test scores.

– “No Child Left Behind” includes extensive mandates to use standardized tests to hold schools accountable for gains in children’s academic performance.

Limitations of Standardized Tests

• • The tests typically used: • Norms are based on a national average • Lack overlap with local curriculum • Given infrequently Not sensitive to short-term academic gains Do not provide teachers with the information they need to make classroom instructional decisions

Curriculum Based Measurement (CBM)

• • Teachers need-and value- information that is directly applicable to their day to-day practice and linked to specific instructional objectives and learning concepts.

An alternative to standardized tests is Curriculum Based Measurement (CBM)

CBM

• • • • CBM is a method of monitoring student educational progress using materials taken from the child's classroom curriculum. CBM tests are constructed from the lessons taught.

CBM is Fast! (2 minutes, weekly) CBM is Inexpensive and Easy!

CBM

• CBM allows teachers to – Continuously measure their students' growth in performance, – Determine if their students' are learning at the expected rate CBM provides data for teachers to evaluate their instructional strategies when students fail to demonstrate expected growth.

• • • • • • • • • Advantages of CBM over other testing methods Based on classroom curriculum Individual referenced Peer referenced Provides teachers with information to make instructional change decisions Allows for direct and continuous monitoring of student achievement related to expected Highly sensitive to student growth Time efficient Cost effective Produces results that are easy to understand

Purpose of CBM in the ACE

• • • • weekly CBM in order to: Monitor children’s acquisition of receptive and expressive vocabulary, letter recognition, and phonemic awareness skills.

Monitor how well children with special needs and at-risk are doing relative to higher achieving children. Make adaptations in instruction based on

CBM Procedures

• Steps to CBM: • • 1. Identify children (each ACE classroom is monitoring 4 special education (referred but did not qualify for special education), 1 typical achiever, and 1 “super star”).

The “super star” child defines what is possible; the average achieving child’ s “sets the standard” for what is reasonable.

Teachers selections of average and super star achievers will be compared with the results of standardized tests

CBM Procedures

2. Develop CBM: Collect samples from each week’s lessons (e.g., receptive vocabulary, expressive vocabulary, alliteration). 3. Alliteration items are on cards for easier administration. 4. Prepare a separate score sheet for each child. The score sheet includes the vocabulary and letters being tested as well the materials needed to assess (e.g., posters, small books).

Sample Score Sheet

Curriculum-Based Measurement Procedures 5.

6.

7.

8.

Administer CBM weekly. Teachers select a time that allow them to be consistent each week in administering the CBM.

Ideally, teachers should use the same wording, speed, and wait time for each child. In reality, there are time differences based on the child’s needs. Record number of items done correctly on score sheet.

Aim Line

Graph the results for the children in each classroom.

The average achieving child’s performance defines the “aim line” In the ACE project, we are defining the aim line as the expected performance based on the typically developing child’s performance.

Curriculum-Based Measurement (Cont.) 8.

Develop a line for each child on a graph

Interpreting the Results

Decision Rule 1: If three consecutive data points are below the aim line (See sample graph) for a child, teacher asks, why. • • • upsetting events in the home?

illness? absences? • • behavior difficulties?

CBM items too difficult?

Interpreting the Results

Interpreting Results

Instructional changes for Decision Rule 1: I.

Change the classroom environment II.

• • • • • • • change the student's seating change time of instruction or testing Modify Instruction change the type of instruction (small group or one-on-one instruction) Give more time for guided and independent practice of skills taught Reteach when necessary Review skills Simplify directions.

Interpreting the Results

Decision Rule 2: If the child's performance is not consistently above or below the aim line (i.e., more than three consecutive data points), don't make any instructional changes (See the sample graph for decision rule 2).

Administration of Curriculum-based measurement (Cont.)

Interpreting the Results

Decision Rule 3: If the child's performance is above the aim line for three consecutive data points, consider enhancing and enriching the instructional program (See the sample graph for decision rule 3).

Interpreting the Results

Using the Results

Continue to measure and monitor Regularly inform parents about their child's progress toward the goals. – – – A written explanation of individual child’s graph might be sent home. The written note may include an explanation of child’s progress on the graph.

Note positive changes in the child’s progress on the graph Arrange conferences for parents who might need some assistance in understanding the concept and may need an explanation of the CBM.

Data from ACE CBM— Feb to May, 2004

CBM Mean Percentage of Correct Response for the Three Groups

Typical At Risk Special Needs

(n = 25) (n = 7) (n = 21) Letter Identification Unit 1 81.00 (30.4) 59.53 (34.5) 50.20 (37.4) Letter Identification Unit 2 72.92 (30.4) 61.63 (27.3) 51.15 (34.8) Letter Identification Unit 3 76.08 (28.7) 50.00 (32.3) 60.04 (27.2) Show Me Unit 1 73.29 (14.9) 48.19 (14.3) 69.23 (24.0) Show Me Unit 2 78.83 (13.4) 56.70 (19.4) 70.66 (25.5) Show Me Unit 3 82.37 (13.5) 46.67 (27.1) 68.64 (21.7) Tell Me Unit 1 59.38 (20.3) 47.56 (17.1) 50.71 (26.0) Tell Me Unit 2 59.83 (17.6) 47.23 (15.1) 52.69 (25.7) Tell Me Unit 3 75.07 (19.0) 57.51 (14.2) 57.48 (25.0) Alliteration 70.31 (27.1) 69.96 (21.7) 50.49 (29.6)

CBM Mean Percentage of Correct Response for the Three Groups 50 40 30 20 10 0 150 140 130 120 110 100 90 80 70 60 S

Letter Identification Show Me Tell Me Alliteration

Unit 1 Unit 2 Unit 3 Unit 1 Unit 2 Unit 3 Unit 1 Unit 2 Unit 3 Unit 1 Unit

An Example of CBM Performance: Typical vs. Special Needs 160 150 140 130 120 110 100 90

Letter Identification Show Me Tell Me Alliteration

80 70 60 50 40 30 20 10 0 U ni t1 U _W ni t1 ee U ni k _W 1 ee U ni k _W ee U 2 _W ni k 3 ee U ni k _W t1 4 ee U ni k _W 1 ee U _W ni k U 2 ee ni k _W 3 ee U ni k _W ee t1 4 U k _W ni t1 1 U ee ni k _W t1 2 ee k _W 3 ee U ni k t1 4 U _W ni ee U _W ni k 1 ee U ni k _W 2 ee U k _W ni t1 3 ee U ni k _W 4 ee U _W ni k U 1 ee _W ni k 2 ee U _W ni k 3 ee t1 U k _W ni t1 4 U ee _W ni k t1 U 1 ee _W ni k t1 2 ee k _W 3 ee U ni k t1 4 U _W ni ee U k _W ni 1 ee U ni k _W t1 2 ee U ni k _W 3 ee U ni k _W U 4 ee ni k _W 1 ee U _W ni k ee t1 2 U k _W ni t1 3 U ee ni k _W t1 U 4 ee ni k _W t1 1 ee U ni k _W t1 2 ee k _W 3 ee U ni k t1 U 4 _W ni ee U k _W ni t1 1 ee U ni k _W 2 ee k _W 3 ee k 4 Unit / Week

Comparisons Between the Three Groups Statistically Significant Group Differences (ANOVA): Letter Identification Unit 1: F (2, 50) = 4.87, p = .012

Letter Identification Unit 3: F (2, 53) = 3.25, p = .047

Show Me Unit 1: F (2, 50) = 4.80, p = .012 Show Me Unit 2: F (2, 53) = 3.73, p = .031

Show Me Unit 3: F (2, 53) = 10.61, p < .001

Tell Me Unit 3: F (2, 53) = 4.83, p = 0.12

Alliteration: F (2, 53) = 3.43, p = 0.40

Significant Post Hoc Comparisons: Letter Identification Unit 1: Typical (M = 81.0) > Special Needs (M = 50.2), p = .009

Show Me Unit 1: Typical (M = 73.3) > At Risk (M = 48.2), p = .009

Show Me Unit 2: Typical (M = 78.8) > At Risk (M = 56.7), p = .028

Show Me Unit 3: Typical (M = 82.4) > At Risk (M = 46.7), p < .001 Typical (M = 82.4) > Special Needs (M = 60.6), p = 039 Special Needs (M = 60.6) > At Risk (M = 46.7), p = .027

Tell Me Unit 3: Typical (M = 75.1) > Special Needs (M = 57.5), p = .015

Alliteration: Typical (M = 70.3) > Special Needs (M = 50.5), p = .040

Comparisons Within the Three Groups

Typical Children (n = 25): Letter Identification:

No significant changes over time

Show Me:

Statistically significant, Wilks’ Λ = .71, F (2, 23) = 4.75, p = .019, multivariate η 2 = .29 (effect size) Children’s “Show Me” performance improved over time Unit 3 (M = 82.4) > Unit 2 (M = 78.8) > Unit 1 (M = 73.3)

Tell Me:

Statistically significant, Wilks’ Λ = .54, F (2, 23) = 9.92, p = .001, multivariate η 2 = .46 (effect size) Children’s “Tell Me” performance improved over time Unit 3 (M = 75.1) > Unit 2 (M = 59.8) > Unit 1 (M = 53.4)

At-Risk Children (n = 7):

No significant changes

Children with special needs (n = 21):

No significant changes