Transcript Data Presentation - Missouri Charter Public School Association

An Overview of the Types of
Performance Data Used to Describe
Missouri Schools
A Close Look at the Indices that Governing Board
Members Should Use to Evaluate Charter School Quality
Sharon Ford Schattgen, Ph.D.
November 7, 2012
Presentation Prepared for the Missouri Charter School Association’s
Seminar for Governing Board Members
Types of Performance Data Used to
Evaluate School Quality
 Missouri Assessment Program Data





MAP Grade-Level Assessment & End-of-Course Assessment Proficiency Rates
TerraNova Median National Percentile Ranks
MAP Scale Scores Averages, Arrayed by Cohort
MAP Index Scores
MAP Growth Estimates
 Other Summative Standardized Assessment Data
 Additional Standardized Achievement Test Scores (e.g., Stanford, ITBS, etc.)
 College Readiness/Placement Data (e.g., ACT/SAT Scores, Advanced Placement
Scores)
 Local Interim Assessment Data (e.g., NWEA Assessment Data)
 Additional Performance Measures
 Attendance, Dropout, & Graduation Rates
 Post-Secondary Success Data
 Accountability Data
 Missouri School Improvement Program Annual Performance Report (MSIP APR)
 Information Associated with Special Designations
2
School X Demographic Data
Enrollment/S
chool Year
2007-08
2008-09
2009-10
Total
285
350
434
504
575
Black
270
(94.7%)
332
(94.9%)
407
(93.8%)
472
(93.7%)
540
(93.9%)
2
(.7%)
2
(.6%)
2
(.5%)
3
(.6%)
Not
reported
White
13
(4.6%)
16
(4.6%)
25
(5.8%)
28
(5.6%)
31
(5.4%)
Free/Reduce
d Price Lunch
227
(85%)
321
(96.1%)
393
(92%)
467
(92.3%)
514
(89.4%)
Hispanic
2010-11
2011-12
Free/Reduced Price Lunch Percentages for Corresponding Years:
Comparison District = 72.3%, 68.7%, 83.8%, 85.7%, 87.4%; MO = 42.1%, 43.7%, 46.9%, 47.8%,
3
Note 1: Disaggregation of data by
“subgroup” is required!
One of the goals of the No Child Left Behind Act is to
eliminate discrepancies in achievement across
“subgroups” of students, which means Governing
Board members should examine data that are
disaggregated to allow you to see any “gaps” in
performance. In other words, you need to review data
that are disaggregated by racial or ethnic group,
free/reduced-price lunch status, special-education
status, and limited English proficiency status, and,
when appropriate, gender.
4
Note 2: Most of the data Governing
Board members will receive will not be
presented in its original format. Rather,
it will likely be summarized and
reformatted for the purposes of
creating an annual report.
School personnel access their data from the DESE
MCDS Portal, and test publishers also provide
electronic and print reports.
School Report Cards are posted at
www.dese.mo.gov
5
Note 3: We assess achievement for specific
purposes; here are 5 such purposes.
1. Provide summary information about what students know
and can do relative to course objectives or content/process
standards
2. Yield information on an interim basis to inform policymaker
and/or educator decisions at classroom, school, or district
level
3. Collect student-centered information to guide minute-byminute and/or day-to-day instruction
4. Place students in specific programs/services
5. Identify students who need further evaluation
Purposes 3, 4, and 5 (listed in green, italic type) yield data that are not
appropriate for inclusion in a report to a Governing Board. Data collected for
6
these purposes have utility only for educators.
Perie, Marion, & Gong, 2009
Interim (instructional, evaluative,
predictive) aka “common” or
“benchmark”
Formative (minute by minute and/or day-to-day
integrated into lesson) aka “classroom”
Frequency of administration
Increasing
Summative
Scope (curriculum focus) & duration of
cycle (time frame)
Tiers of Achievement
Assessment
A Reminder: Types of Performance Data
Used to Evaluate School Quality
 Missouri Assessment Program Data





MAP Grade-Level Assessment & End-of-Course Assessment Proficiency Rates
TerraNova Median National Percentile Ranks
MAP Scale Scores Averages, Arrayed by Cohort
MAP Index Scores
MAP Growth Estimates
 Other Summative Standardized Assessment Data
 Additional Standardized Achievement Test Scores (e.g., Stanford, ITBS, etc.)
 College Readiness/Placement Data (e.g., ACT/SAT Scores, Advanced Placement
Scores)
 Local Interim Assessment Data (e.g., NWEA Assessment Data)
 Additional Performance Measures
 Attendance, Dropout, & Graduation Rates
 Post-Secondary Success Data
 Accountability Data
 Missouri School Improvement Program Annual Performance Report (MSIP APR)
 Information Associated with Special Designations
8
Missouri Assessment Program (MAP)
Grade-Level & End-of-Course (EOC) Assessment Proficiency Rates
“MAP Proficiency Rate” is the percentage of students scoring in the top two MAP
achievement levels—the Proficient Level and the Advanced Level.
Following a brief introduction to the MAP, there is a sample line graph showing trends in
MAP Grade-Level Proficiency Rates, a sample bar graph showing disaggregated GradeLevel Proficiency Rates for one year, and a sample table showing trends in End-of-Course
Proficiency Rates.
ASSESSMENT DATA
9
Missouri Assessment Program (MAP)
• MAP includes required consecutive grade-level (3-8) assessments in
Communication Arts and Mathematics and grade-span (5 and 8)
assessments in Science.
• MAP also includes end-of-course (EOC) exams, including English II, Algebra
I, and Biology I. See DESE website for testing requirements.
• MAP results are analyzed and reported in a variety of ways in order to
inform policy and practice and to satisfy federal and state accountability
requirements.
•
•
•
•
Performance on each MAP assessment is reported in terms of four achievement levels:
Below Basic, Basic, Proficient, and Advanced.
The TerraNova tests, which yield National Percentile Ranks, are components of the MAP
grade-level assessments.
MAP results are also reported using Scale Scores (calculated by test publisher).
DESE also calculates MAP Index Scores to use in the MO School Improvement Program
(state accountability system).
10
MAP Mathematics Proficiency Rates
Grade 7
School X
Comparison
Missouri
70
60.1
60
55.2
56.4
52.5
49.8
50
37.9
40
34.6
32.1
27.8
30
29.3
27.7
20
16.5
10
24.3
25.1
2011
2012
14.5
0
2008
2009
2010
11
2009 Proficiency Rates for Black Students and White Students: MAP Communication Arts
Grade-Level and End-of-Course Assessments
Columbia Public Schools and Missouri
District Black
District White
State Black
State White
100.0
90.0
85.6
Percent Proficient/Advanced
80.0
77.2
70.0
66.2
60.6
60.0
57.6
60.0
56.5
54.9
50.0
56.0
53.4
51.9
50.1
64.7
53.4
51.2
46.1
40.0
28.0
30.0
21.8
20.0
28.0
26.3
25.2
21.7
28.6
26.9
25.0
21.9
19.9
16.4
10.0
0.0
Grade 3
Grade 4
Grade 5
Grade 6
Grade 7
Grade 8
Eng II EOC
12
MAP End-of-Course (EOC) Exam
Proficiency Rates
Algebra I
2009
2010
2011
2012
22.2%
32.8%
62.3%
59.9%
N=72
N=49
N=50
N=50
Comparison
School
22.7%
26.3%
41.7%
44.8%
Missouri
52.6%
57.3%
59.7%
56.7%
School X
13
TerraNova Median National Percentile Ranks
TerraNova Tests are embedded within the MAP Grade-Level Assessments. Each
TerraNova yields a National Percentile Rank, which allows us to compare the performance
of our students to that of their national grade-level peers. To report group performance,
we calculate the Median National Percentile Rank.
TerraNova Median National Percentile Ranks for multiple years are shown on the
following slide.
ASSESSMENT DATA
14
TerraNova Median National Percentile Ranks
School X Communication Arts
School X Mathematics
2008 2009 2010 2011 2012
2008
2009
2010 2011 2012
Grade 6
41
36
33
33
43
Grade 6
28
21
27
22
32
Grade 7
43
39
44
43
41
Grade 7
29
33
29
31
31
Grade 8
34
36
48
44
37
Grade 8
49
45
56
29
36
School X Science (Grade 8)
2008: 39
2009: 30
2010: 37
2011: 36
2012: 37
NATIONAL Terra Nova Median Percentile Rank = 50; Range = 1 to 99
15
MAP Scale Score Averages, Arrayed by Cohort
When arrayed by cohort, MAP Scale Score Averages show the change over time for a
given group of students as they move through the grade span (e.g., sixth grade to
seventh grade; seventh grade to eighth grade).
This type of analysis, shown in the following table, indicates, in a very general way,
whether there is student growth in achievement.
ASSESSMENT DATA
16
MAP Scale Score Averages for Grade Cohorts
School X
Communication Arts
Grade 6
Grade 7
Grade 8
2008
650
651
674
2009
644
653
675
2010
652
655
680
2011
646
654
676
2012
653
656
671
MAP Communication Arts Scale Scores range from 455 to 875 (grades 3-8).
17
MAP Index Scores
Each year, DESE calculates a MAP Index Score for each content area in each grade span.
The Index is calculated by multiplying the percentage of reportable students scoring
within a given achievement level by a constant: 6 x % in Below Basic; 7 x % in Basic; 8 x
% in Proficient; and 9 x % in Advanced. The resultant products are then summed to
produce the Index, which ranges from 600-900. MAP Index Scores, arrayed over time,
are used in the Missouri School Improvement Program as an indicator of whether there
is longitudinal improvement in performance.
The following table presents an example of MAP Index Scores over time.
ASSESSMENT DATA
18
School Y
MAP Index Scores
Grade
Level
2008
2009
2010
2011
2012
Grades 35 Math
711.5
725.0
735.0
739.4
742.6
Status
730.7
Source: Understanding Your Annual Performance Report, 2012-13. Missouri Department of Elementary &
Secondary Education
19
MAP Growth Estimates
MAP Growth Estimates are calculated using complex statistical formulae, which utilize
each student’s MAP Scale Scores over time—for example, from sixth grade to seventh
grade, and from seventh grade to eighth grade. The Department of Elementary and
Secondary Education first reported MAP growth data in 2011
The next slide presents an example of MAP growth data.
ASSESSMENT DATA
20
MAP Value-Added Growth Estimates
Mathematics
2011
School X
2012
Between
0.095 and 0.16
School X
Significant
Between
0.082 and 0.25
Significant
State Range: -3 to +3
State Average = 0
*Growth estimates were calculated using longitudinal MAP grade-level assessment data.
21
Other Summative Standardized Assessment Data
Students may take other summative standardized assessments, in addition to the MAP—
assessments such as the Stanford Achievement Test or the Iowa Test of Basic Skills or
ACT’s EXPLORE or PLAN. If so, these tests will likely be administered in the fall or the
spring of the year. Results are used to inform instructional decisions.
A subsequent slide presents an example of data from PLAN, a standardized test that may
be administered to 10th-grade students.
ASSESSMENT DATA
22
Data from Other Standardized
Achievement Tests
• In addition to the MAP, students may take
another summative assessment, such as
Stanford Achievement Test, Iowa Test of Basic
Skills, EXPLORE, or PLAN.
• Scores from these tests could be reported in
terms of:
– Median Percentile Rank
– Percent of Students Scoring At or Above Grade
– Scale Score Average
23
PLAN Scale Score Averages
Columbia Public Schools and Nation
2005-06 through 2008-09
Scale: 1-32
In 2008-09, 67% of CPS students
earned Composite Scores at or above
the national average.
Scale Score Average
31
26
21
16
11
6
1
English
Mathematics
Reading
Science
Composite
District 2005-06
18.6
18.7
18.2
19.7
18.9
Nation 2005-06
16.1
16.3
15.8
17.4
16.5
District 2006-07
18.6
19.2
18.5
19.7
19.1
District 2007-08
17.8
19.0
18.0
19.9
18.8
District 2008-09
18.2
19.2
18.0
19.1
18.8
Nation 2007-09
16.9
17.4
16.9
18.2
17.5
24
College Readiness/Placement Data
The ACT is a set of curriculum-referenced tests that measure students’ readiness for
college. ACT Scale Scores range from 1 to 36, and the 2012 national average is 21.1. A
student whose Scale Score in a particular content area meets or exceeds the
corresponding ACT College Readiness Benchmark has a high probability of being
successful in an entry-level, credit-bearing college course in that subject.
Scores from Advanced Placement Exams are used to award college credit or advanced
placement in college coursework. Advanced Placement Exam cores range from 1 to 5.
The following tables present example ACT data and Advanced Placement data.
ASSESSMENT DATA
25
ACT Scale Score Averages
Class of 2012
School X
English
Math
Reading
Science
Composite
14
15.9
16.3
16
15.8
Comparison
District
Data not available
16.5
Missouri
21.4
21.1
21.9
21.5
21.6
National
20.5
21.1
21.3
20.9
21.1
26
ACT Data: Class of 2012
Percentage of Test-Takers
Scoring at or Above National
Average Composite Scale
Score of 21
School X
20.0%
Comparison
District
18.2%
Percentage of 2012 graduates
taking ACT:
School X = 85%
Comparison District = 70%
MO = 67-75%
US = 52%
Percentage of Test-Takers Scoring at
or Above College Readiness
Benchmark
School X
English:
Math:
Reading:
Science:
All 4:
18%
13%
15%
12%
12%
Missouri
English:
Math:
Reading:
Science:
All 4:
73%
46%
56%
33%
27%
27
Average Scores on Selected Advanced Placement Examinations, 2009
(Exams Most Frequently Taken by District's Students)
District
State
Nation & Beyond
3.
74
3.
14
64
2.
84
2.
83
2.
2.
67
72
2.
2.
88
3.
21
99
2.
2.
92
88
2.
3.
31
3.
54
22
3.
2.
76
3.0
3.
48
3.5
2.
92
Average Score
4.0
3.
74
3.
93
4.
07
4.5
4.
02
4.
38
5.0
71
2.
2.5
2.0
1.5
1.0
Lang &
Comp
Psychology
US History Calculus AB
CPS n=189
CPS n=181
CPS n=143
CPS n=133
World
History
Lit & Comp
Statistics
Biology
CPS n=120
CPS n=96
CSP n=74
CPS n=54
28
NWEA Assessment Indices
Some charter schools periodically administer classroom-based assessments (such as those
published by NWEA, the Northwest Evaluation Association) in order to monitor student
progress and to adjust instruction based on student needs.
Pre- and post-test NWEA assessment indices for the 2011-12 school year are presented on
the following table; however, these data must be interpreted with care because we have
no evidence of their technical characteristics (e.g., reliability and validity). However, such
data could be appropriate for inclusion in a report to a board because they have the
potential to provide information about student learning.
ASSESSMENT DATA
29
Example of Local Interim
Assessment: NWEA Indices
Reading
Math
Grade 6
Percent At/Above Grade Level, Fall 2011
42%
18%
Percent At/Above Grade Level, Spring 2012
44%
46%
Percent At/Above Grade Level, Fall 2011
15%
20%
Percent At/Above Grade Level, Spring 2012
40%
54%
Percent At/Above Grade Level, Fall 2011
16%
25%
Percent At/Above Grade Level, Spring 2012
53%
42%
Percent At/Above Grade Level, Fall 2011
25%
36%
Percent At/Above Grade Level, Spring 2012
66%
54%
Grade 7
Grade 8
Grade 9
30
Attendance, Dropout, & Graduation Rates; Post-Secondary Success Data
Attendance rates, dropout rates, and graduation rates, along with data about postsecondary success, are shown on the following four slides. These measures of
performance complement achievement data.
ADDITIONAL PERFORMANCE
MEASURES
31
Attendance Rate
2008
2009
2010
2011
2012
Example
School
91.2%
92.6%
93.4%
94.7%
93.9%
Comparison
District
88.6%
91.5%
92.0%
92.5%
93.1%
Missouri
94.0%
94.4%
94.2%
94.4%
94.7%
32
Dropout Rate
2008
Example
School
Comparison
District
Missouri
2009
2010
2011
2012
0%
1.5%
1.6%
3.4%
7.8%
13.9%
13.8%
17.5%
23.6%
19.5%
3.5%
3.5%
3.2%
3.4%
3.4%
33
Graduation Rate
Graduation Rate
Academic Year: 2011-2012
Example School
88.7%
Comparison District
74.5%
Missouri
87.7%
Provide trend data when possible.
34
Graduation Rate
Post-Secondary Placement Rate
• Over 85% of School Y’s 2010 graduates
enrolled in post-secondary institutions,
entered the job market in a field related to
their career-education coursework, or joined
the military.
35
Missouri School Improvement Program: Annual Performance Report
The MSIP APR presents evaluative data about districts and charter schools in relation to
14 accountability standards. Schools are held accountable for only those standards
pertaining to its grade span. In accordance with established criteria, the Missouri
Department of Elementary & Secondary Education can issue designations based on a
school’s or a district’s performance.
The following two tables summarize School X’s and a comparison district’s 2012 MSIP
APRs and provide data supporting a specific designation—in this case as a “focus school.”
ACCOUNTABILITY DATA
36
MSIP Standard/Indicator
School X
Comparison District
9.1.3. MAP Grades 6-8 Math
Not Met
Not Met
9.1.4 MAP Grades 6-8 Comm Arts
Not Met
Not Met
9.1.5 EOC Algebra I Math
Not Met
Met
9.1.6 EOC English II Comm Arts
Not Met
Not Met
[Bonus MAP Achievement]
Not Met
Not Met
[MAP Grade 8 Science]
Not Met
Not Met
9.3 ACT
Not Met
Not Met
9.4.1 Advanced Courses
Not Met
Met
9.4.2 Career Education Courses
Not Met
Met
9.4.3 College Placement
Not Met
Met
9.4.4 Career Education Placement
Not Met
Met
9.5 Graduation Rate
Met
Met
9.6 Attendance Rate
Met
Met
9.7 Subgroup Achievement
Not Met
Not Met
37
School X “Focus School” Indices
Percentage of Students Scoring Proficient or Advanced on MAP Assessments
(i.e., MAP Proficiency Rate)
2010
2011
2012
Three-Year
Cumulative
Student Gap
Group,
English/Langua
ge Arts
27.5%
22.8%
20.9%
24.0%
Student Gap
Group,
Mathematics
29.0%
26.4%
29.7%
28.3%
Combined English/Language Arts & Mathematics Proficiency Rate = 26.2%
Targets: Communication Arts = 44.21%; Math = 44.78%
38
And, by the way . . .
The tests, they are a changin’
• In 2014-15, new state assessments will become
operational in Missouri—assessments that are
referenced to the Common Core State Standards
and are designed to measure students’ readiness
for college and careers.
• Two consortia are developing such assessments:
* Smarter Balanced
* PARCC
39
In closing . . . .
Questions & Comments . . . .
For further information, go to these websites.
MAP Assessments & MSIP:
www.dese.mo.gov
Assessment Consortia:
www.smarterbalanced.org
www.parcconline.org
40