Transcript Challenging Education to Meet the Changing Face of Indiana

The Indiana Acuity Efficacy Study:
Year 1 Results and Implications
Terry Spradlin
June 24, 2009
CCSSO National Conference on Student Assessment
About the Center for
Evaluation & Education Policy
•The Center for Evaluation & Education Policy (CEEP) is a
client-focused, self-funded research center associated
with the School of Education at Indiana University
•CEEP promotes and supports rigorous program
evaluation and nonpartisan policy research primarily,
but not exclusively, for education, human service and
non-profit organizations
•In the area of K-12 education policy, CEEP’s mission is to
help inform, influence and shape sound policy through
effective, nonpartisan research and analysis
2
CEEP Associates focus their broad spectrum of
experience and capabilities to produce high impact
within the following "Areas of Excellence":
•Educational Evaluation
oEarly Childhood Education Evaluation
oLiteracy Evaluation
oMath, Science and Technology Evaluation
•Education Policy Research & Technical Assistance
•Health, Human Services & Community Development
Evaluation
3
Table of Contents
I.
Indiana’s Comprehensive Assessment Plan
II. 2008-09 Testing Schedule (formative and summative)
III. Objectives of Efficacy Study in Indiana
IV. Study Design and Methods
V. Year 1 Findings (from Qualitative Analyses)
VI. Future Study Components
4
I. Indiana’s Comprehensive
Assessment Plan
• Adopted by the Indiana State Board of Education on November 1, 2006
• Plan called for moving the Indiana Statewide Testing for Educational
Progress-Plus (ISTEP+) from Fall to Spring and the implementation of
formative/diagnostic assessments
• Features implemented during the 2008-09 school year:
o Wireless Generation’s mClass Reading 3D and Math (Grades K-2 formative)
o CTB/McGraw-Hill’s Acuity Assessment Program (Grades 3-8 formative)
o Phase-out of the Graduation Qualifying Exam (GQE)
• Class of 2011 last to be required to pass GQE
• To be replaced with end-of-course assessments in core subject areas
o Move ISTEP+ from Fall to Spring
• (Students in grades 3-10 were tested twice during the 2008-09 school
year)
5
II. 2008-09 Fall Testing Schedule
(Formative and Summative)
6
II. 2008-09 Spring Testing Schedule
(Formative and Summative)
7
III. Objectives of Efficacy Study in Indiana
• Objectives of CEEP Study:
o Evaluate the effects of CTB/McGraw-Hill’s Acuity
Assessment Program, a formative assessment system, on
instructional practice and student achievement during
the initial/pilot year of implementation in 510 schools
o Information intended to inform CTB and the IDOE about
the kind of support needed to make the implementation
of Acuity most effective during subsequent school years
8
IV. Study Design and Methods
Qualitative Methods
•
Fall statewide online survey of administrators and testing coordinators from Acuity and non-Acuity
schools to address:
o
Why schools did or did not participate in Acuity, their interest in participating in the Acuity Program
in future school years, and their present use of other formative assessment systems
o
Administrators and testing coordinators’ expectations of their school’s participation in Acuity and its
potential impact on instruction and student achievement outcomes, and the levels of their initial
training and preparedness
o
The survey window was open from September 2 to September 19, 2008
o
Respondents included superintendents, principals, and testing coordinators
• 237 respondents: 70% of school districts in Indiana were represented by at least one survey
respondent
• 99 respondents were from schools using Acuity, while the remainder were from non-Acuity
schools
9
IV. Study Design and Methods (cont.)
Qualitative Methods
•
Spring statewide online survey of Acuity schools to measure attitudes and
perceptions of teachers and administrators on the benefits and outcomes of their
participation in the Acuity Assessment Program during the 2008-09 school year
o The survey window was open from April 20 to May 8, 2009
o Respondents included teachers, testing coordinators, and principals
• 731 respondents: 87% of school corporations participating in Acuity in
Indiana were represented by at least one survey respondent
10
IV. Study Design and Methods (cont.)
Qualitative Methods (cont.)
•
Intensive case study on the use of the Predictive and Diagnostic assessment results to:
1) inform and enhance instruction
2) drive improvement in student achievement outcomes as measured by the summative
assessment
o The Project Team identified 12 school corporations for the case study that were
reflective of Indiana’s student population based on prior ISTEP+ performance, free and
reduced-price meal program eligibility data, school size, and local type (urban,
suburban, and rural)
o Qualitative procedures including a questionnaire, focus group sessions, and onsite visits
were used to inform the case study regarding the extent to which these corporations
have implemented the Acuity Program and how they utilized Acuity data to inform or
alter instruction
11
Quantitative Methods
•
Completion of a Comparison-Group Study using regression and other statistical techniques
to analyze quantitative data collected from the Acuity predictive and diagnostic assessments
as well as from Indiana’s ISTEP+ summative assessment
o These analyses will be used to assess the degree to which use of the predictive and
diagnostic assessments are associated with increased achievement on the staterequired summative assessments in mathematics and English/language arts
o These analyses will account for the varying level of participation by pilot schools
selecting among predictive and diagnostic assessments that best meet the needs of
their students
o Comparison schools will be matched with Acuity schools using prior ISTEP+
performance, free and reduced-price meal program data, school size, and local type
(urban, suburban, and rural)
12
V. Year 1 Findings (from Qualitative Analyses)
A. The End-of-Year survey and comparisons to the Fall survey
B. Case Study Site Visits
o Pressing Issues
o Positive Educator Feedback
o Educator Recommendations and Considerations
C. Teacher and Student Impressions of Acuity (Video)
13
A. End-of-Year survey and
comparisons to Fall survey
14
Details of End-of-Year Survey
Spring (End-of-Year) statewide online survey
• 731 respondents included 460 (63%) teachers, 119 (16%) principals, 77 (11%) testing
coordinators, and 75 (10%) other school personnel
16-question survey examining:
• Educator opinions regarding Acuity Assessment Program content, technology/user
experience, professional development, and customer support after use of the system for a
full school year
•
In addition, the survey was intended to obtain suggestions for improvement of the program
and to gauge views regarding the impact of the program on classroom instruction, general
student achievement, and student achievement on ISTEP+
•
Some questions from the Spring survey were comparable to questions asked in the Fall
survey, so comparisons were also made between educator opinions at the beginning of the
school year and in the Spring
15
Survey Results: Overall Educator
Opinions of Formative Assessment
16
Frequency of Formative Assessment Use
•
Question 3 of the Spring survey
(“How often did you access the
Acuity Assessment System?”) is
comparable to Question 5 of the
Fall survey: “How often do you
plan to use the Acuity Assessment
System?”
Once/week
Multiple
times/week
Once/month
•
Respondents most commonly
used the program only during
administration windows.
However, respondents used the
program once/week, multiple
times/week, and once/month
more than predicted
Only during
administration
windows
Other (Varied)
Total Number of
Respondents
Percent of
Respondents in
Fall Survey
Percent of
Respondents in
Spring Survey
3.1
14.2
3.1
13.7
4.1
15.9
53.1
39.8
36.6
16.4
98
724
17
Other Frequency of Use Issues
• Most Acuity schools (70%) used the online tools exclusively in their
administration of the Acuity Assessment Program and another 22% use
online and paper/pencil tests.
• Educators are not using all components of the system as often as
necessary to maximize the use and benefits of the system (e.g., only 42%
of respondents indicated the use of the Instructional Resources and only
16% indicated the use of the Item Bank)
• Approximately one-third of respondents indicated that they were still not
comfortable using class roster and student reports
18
Perceived Impact on Classroom
Instruction
•
•
•
A majority of respondents (51%) indicated that participation in the Acuity
Assessment Program helped to somewhat improve classroom instruction
32% felt participation had no impact on instruction
14% indicated that participation led to a decreased quality of instruction (due to
the multiple assessments scheduled during the school year, both formative and
summative, and the “interruption” to teaching time)
Greatly decreased quality of instruction
Somewhat decreased quality of instruction
No impact on instruction
Somewhat improved quality of instruction
Greatly improved quality of instruction
Total Number of Respondents
Percent of Respondents in Fall
Survey
Percent of Respondents in Spring
Survey
1.0
2.8
0
11.4
5.1
32.3
68.7
51.0
25.3
2.5
99
718
19
Perceived Impact on Student
Achievement
•
•
A plurality (47%) of respondents indicated that participation in the Acuity
Assessment Program during the 2008-09 school year had no impact on student
achievement; 43% of respondents indicated that participation led to somewhat
improved student achievement outcomes
The opinions about the impact of Acuity on student achievement were not as
favorable as the very optimistic outlook expressed by educators participating in
the Fall survey
Percent of Respondents in Fall
Survey
Percent of Respondents in Spring
Survey
0
1.1
Somewhat decreased achievement
1.0
8.0
No impact on achievement
4.0
46.5
Somewhat improved achievement
67.7
42.7
Greatly improved achievement
27.3
1.7
Total Number of Respondents
99
723
Greatly decreased achievement
20
Perceived Impact on Student
Performance on ISTEP+
•
•
•
A plurality (49%) of respondents indicated that they believe participation in Acuity
will lead to improved student performance on the spring 2009 ISTEP+ (note:
Spring ISTEP+ had not been fully administered or scored at the time of the survey)
42% stated that participation would have no impact on student ISTEP+ scores
Like with student achievement in general, the opinions about the impact of Acuity
on student performance on ISTEP+ were not as favorable as the very optimistic
outlook expressed by educators participating in the Fall survey
Percent of Respondents in Fall
Survey
Percent of Respondents in Spring
Survey
0
1.0
Somewhat decreased performance
1.0
4.9
No impact on performance
8.1
41.8
Somewhat improved performance
73.7
48.5
Greatly improved performance
17
3.8
Total Number of Respondents
99
717
Greatly decreased performance
21
Educator’s Views on Impact of Acuity on
Instruction and/or Student Achievement
(Q 11) Why do you think classroom instruction and/or student achievement declined,
improved, or did not change? Of the 568 written responses:
•
For those who believed Acuity led to improvement in classroom instruction
and/or student achievement, the largest number of respondents (99) cited the
reason “Teachers are better able to target teaching material based on
demonstrated student needs”
•
For those who believed Acuity led to no change in classroom instruction and/or
student achievement, the largest number of respondents (66) cited the reason
“Because the test is in its first year, its impact is yet to be determined”
•
For those who believed Acuity led to declined classroom instruction and/or
student achievement, the largest number of respondents (112) indicated the
volume of assessment during the school year was overwhelming for teachers and
students
22
(Q 13) What was the most helpful component
of the Acuity Assessment Program?
• The largest number of
respondents (93 out of 503)
indicated the reports were
the most helpful component
of the Acuity Assessment
Program
• 77 respondents indicated
the Instructional Resources
were the most helpful
component
Most Helpful Program Components
Reports
Instructional Resources
Provided data to drive
instruction
Immediate feedback
Custom tests
ISTEP+ preparation
Ability to assign customized,
individual assignments
Alignment with state
standards
Provided data and resources
that were useful for
remediation
Total Number
of Respondents
Frequency
93
Percent
18.5
77
15.3
60
11.9
58
38
35
11.5
7.6
7.0
27
5.4
24
4.8
23
4.6
503
23
(Q 14) How can CTB/McGraw-Hill enhance the Acuity
Assessment Program for future assessment
administrations?
•
•
The largest number of
respondents (80 out of 417)
indicated they would like the
program to be more userfriendly. (Make program use less
time-consuming and enhance
features to assign skills and
Instructional Resources to
students and create custom
tests)
75 respondents indicated they
would like the alignment of
questions with Indiana Academic
Standards and their school’s
curriculum to be improved
Suggested Program Improvements
Make the program more userfriendly
Improve Alignment of Questions
Improve the layout and formatting
of the assessments
Provide additional training
Improve reporting features to make
them easier to use and understand
ISTEP+ preparation
Improve the Instructional
Resources
Provide a Larger Bank of Questions
and More Questions Per
Standard/Sub skill
Increase Instructions and Audio For
Students with Accommodations
Total Number
of Respondents
Frequency
Percent
80
19.2
75
18.0
66
15.8
58
11.5
30
7.2
35
7.0
27
6.5
19
4.6
11
2.6
417
24
B. Case Studies
Site Visit Suggestions and Implications
25
Pressing Issues








Provide more professional development
Simplify and improve reports
Ensure Item Bank includes sufficient number of items for each standard
Make Instructional Resources more engaging and automate the
instructional resource assignment process
Adequate computer access in schools must be addressed
Align Diagnostic assessments with school curriculum and pacing guides
Improve developmental appropriateness of assessments
Provide clear guidance on the accommodations for students with special
needs
26
Positive Educator Feedback
General Teacher Feedback:
 The program is very user-friendly overall and the immediate scoring is one of
the program’s strengths
 Students can easily log on, study, or take tests independently
 Student practice (using Instructional Resource activities) are easy to set up
and teachers like the ability to have students practice skills often
 Teachers are able to give students individualized practice. (High and low level
students’ needs are met)
 Teachers said they liked the online features of the instant reports
27
Positive Educator Feedback (cont.)
Miscellaneous
 Taking the tests online has reportedly been enjoyable for students and has improved many students’
computer skills
 Most administrators indicated they would like to continue using formative assessment in future
years and said it could replace many of the other assessments their students currently take
 Teachers reportedly like that the questions are tied to Indiana Academic Standards, are aligned with
ISTEP+, and provide standardized tests that can be used with Response to Intervention (RTI) for
continuous progress monitoring
 School personnel like that the program makes them look closely at their curriculum and ensure that
it is standard-driven and appropriate
 The program provides teachers and administrators with information about skills they need to
address both on a classroom level and school-wide. Many school personnel like that the tool can
help school personnel drill down to the standard and indicator levels to identify student needs
 The Instructional Resources reportedly provide a great resource for remediation efforts
28
Educator Recommendations &
Considerations
•
•
•
Reports
o Teachers commented that the reports could be simplified to better meet their needs as classroom
teachers
o School personnel would also like to have access to a parent report and a website that provides links
to educational websites parents could access at home to remediate their children’s skill deficits
Information on Teaching Strategies
o Teachers would like information on teaching and learning strategies they can use to address the
specific problems identified by the assessments
o Due to issues with adequate computer access in some schools, some teachers would like some
suggestions/strategies for remediation that do not require the students to use computers
Tips From Educators
o Students are given their score after taking the assessment but they don’t necessarily know what that
score means. Some teachers found it beneficial to write down students’ scores and provide feedback
immediately when they return to class
o Some schools require every teacher to complete an “Acuity Data Analysis Organizer,” which is a form
created to facilitate teacher involvement in the program. The form requires teachers to look at the
assessment report for each class and list all standards in the appropriate category based upon
percentage. Teachers are then asked to describe their plan of action: “What do you plan on doing
with this class to improve on Critical Need Skills?”
29
C. Teacher and Student Impressions of the Acuity
Assessment Program (Video)
30
V. Year 1 Summary of Findings
 Educators like the technology platform of the system and tests
 No dispute about the relevancy of the formative assessments with state
academic standards and the summative accountability test
 Some concern about alignment of particular formative assessments with
the curriculum pacing guides of schools
 Teachers want more training on using student and class reports to inform
and improve instruction
 Educators offered many suggestions for modification to Acuity; CTB
appears to be listening
31
V. Year 1 Summary of Findings (cont.)
 Educators expressed high levels of satisfaction with customer service and
attentiveness by CTB
 Overall, some evidence participation is impacting instruction, though not
compelling – “just get it over with” mentality pervasive in some schools
 Likely to change moving forward as user experience and training increases
 Educators generally recognized the benefit of the formative assessment
system, but impact on instruction and student achievement outcomes yet
to be fully determined.
32
VI. Future Study Components
• Quantitative analysis of Indiana Acuity data
o What is the association between Acuity participation by
Indiana schools and ISTEP performance?
o How does participation in the Acuity program associate
with differential gains on ISTEP scores between schools
that do and do not participate in Acuity?
33
VI. Future Study Components (cont.)
• Quantitative analysis of Indiana Acuity data
o Regression models will be specified to determine association between
Acuity predictive components and ISTEP math and language arts for
each participating grade
o Variables and covariates include:
• School size (large, small)
• School location (urban, suburban, rural)
• Socio-economic status (high, medium, low)
• Program participation
• Student scores prior to Acuity participation
34
CEEP Contact Information
Terry E. Spradlin, MPA
Associate Director for Education Policy
1900 East Tenth Street
Bloomington, Indiana 47406-7512
812-855-4438
Fax: 812-856-5890
http://ceep.indiana.edu
35