Transcript Faculty Presentation 2

PPT Presentation Template:
This PPT includes all slides to present
a district or building level overview of
PVAAS. This was used with a districtwide K-12 faculty group in a small
school district, where one grade level
represented one building versus
multiple buildings on a district level
report.
PVAAS:
Evaluating Growth,
Projecting Performance
Today’s Session


Concept of PVAAS
Overview of Reporting
•
•
•
•


Math and Reading, Grades 4-8
Math and Reading, Grade 11
Science, Grades 4, 8 and 11
Writing, Grades 5, 8 and 11
School District Data
Questions and Answers
What Is
PVAAS?
PVAAS IS:
• Analysis of existing assessment data
– Longitudinal PSSA scores in available
content areas
• Math, Reading, Science, Writing
– Provides reliable estimates
• A measure of a school’s effectiveness
• Critical Relationship:
Growth Data vs. Achievement Data
PVAAS:
•
Is not another test
•
Is not a comparison of single test
scores
•
Critical concept:
Does not provide a measure of
growth for an individual student
Current Data Collection
Approaches
High Achievement
Average Achievement
Low Achievement
Achievement is based
on each student’s ending point.
PVAAS Adds Another Variable
To Achievement…Progress
Below
Growth
Standard
At or Above
Growth
Standard
Progress is based
on each student’s starting point.
Viewing Achievement and
Progress Together
Scatter Plots:
Grades 4-8
Reading and Math
The Scatter Plot
Coordinate System
Vertical Axis
PSSA Percent
Proficient or
Advanced
Achievement and Growth - 4th Math
% Proficient or Advanced
AYP Percent
Proficient Target
Growth Standard
Boundary
One Year
Growth
56
0
-20
0
PVAAS Growth Measure
20
Horizontal Axis
PVAAS Growth
Measure
Scatter Plot Quadrants
and Their Meanings
Quad 2
Adequate Achievement
Negative Growth
Achievement and Growth - 4th Math
% Proficient or Advanced
Quad 3
Below Adequate
Achievement
Negative Growth
Quad 1
Adequate Achievement
Positive Growth
56
Quad 4
Below Adequate
Achievement
Positive Growth
0
-20
0
PVAAS Growth Measure
20
SD Scatter Plots
Grade 4
Reading
Math
Insert District Scatter Plots Here
SD Scatter Plots
Grade 5
Reading
Math
Insert District Scatter Plots Here
SD Scatter Plots
Grade 6
Reading
Math
Insert District Scatter Plots Here
SD Scatter Plots
Grade 7
Reading
Math
Insert District Scatter Plots Here
SD Scatter Plots
Grade 8
Reading
Math
Insert District Scatter Plots Here
Viewing Achievement and
Progress Together
Scatter Plots:
Grade 11
Reading and Math
Grade 11 Scatter Plots
The PSSA-PVAAS Scatter Plots visually display the
following data from the statewide distribution of PSSA
results for the most current school year.

Achievement
• The mean predicted performance of the cohort.
• The actual mean performance of all students in the state is
displayed for reference

Growth
• School Effect ≈ Observed Mean – Predicted Mean
• Standardized School Effect = School Effect relative to its
error – displays degree of confidence
The following two slides explain the meaning of each
of the regions of an 11th grade scatter plot.
The Scatter Plot
Coordinate System for Grade 11
Vertical Axis
Mean
Predicted
Scaled Score
2010 Mean
Scaled Score
Growth Descriptor
Boundaries (±2SEs)
Horizontal Axis
Growth Measure in
Standard Errors
Scatter Plot Regions
and Their Meanings for Grade 11
SD Scatter Plots
Grade 11
Reading
Math
Insert District Scatter Plots Here
The TWO Types of
PVAAS Information
Looking
Forward/Planning…
Looking Back/Evaluation…
PVAAS Projection Reports
For Individual Students
and Cohorts of Students
Value-added Growth Reports For
Cohorts of Students
Today
NO individual measure of student growth.
PVAAS
Reporting
Student Projections
Wouldn’t it be great to
know the likelihood that a
student will be proficient
on a future PSSA?
Student Projection Methodology



A separate longitudinal modeling process
focused on the performance scores of
individual students.
The model uses all of the available data (both
math and reading) to project future PSSA
performances in Math, Reading, Science and
Writing.
Projections can be calculated for an individual
student or groups of students.
How do I know that the projection
is accurate?

The projection is precise, and is created using a model
that has been reviewed and approved by four different
peer review panels and the GAO (US Government
Accountability Office).

The most recent 2008 growth model proposal to USDOE
includes information regarding the statistical model and
projection reliability study.
(http://www.ed.gov/admins/lead/account/growthmodel/pa/index.html)

Recent studies have confirmed that the PVAAS
projections (even as far as 3 years into the future) are
more reliable at looking at the future performance of a
student than the most recent PSSA score.
PVAAS Student Projection Report
PVAAS Projections for
Groups of Students
Student Search
Identified Students
Meeting Search Requirements
PVAAS Reporting
Projections to Basic, Proficient, and Advanced
Math and Reading:
 3  4 or 5
 4  5 or 6
 5  6 or 7
 6  7 or 8
78
 8  11
Grade Last Tested
Writing:
 35
45
58
68
78
 8  11
Grade Last Tested
Science:
34
 4  NA
58
68
78
 8  11
Grade Last Tested
Projection Summary Reports



What is it?
• Report that summarizes the numbers and
percents of students in likelihood ranges of
performing at the proficient level on a future
PSSA exam.
How might a school use this report?
• Intervention Planning
• Resource Allocation
• Strategic Planning
• School Improvement Planning
Cautions
• This report provides another indicator about
likelihood of future performance.
School (Single Grade)
Projection Summary Report
Value-Added
Wouldn’t it be great to know if
your students made one year’s
growth for the year of schooling
they experienced last year?
In Math and Reading (4-8 and 11)?
In Science (4, 8 and 11)?
In Writing (5, 8 and 11)?
PVAAS Growth Methods
Testing
Subjects – Grades
Methodology
PSSA in
consecutive
years
Math & Reading – 4-8
Growth
Standard
PSSA not in
consecutive years
Writing – 5, 8, 11
Science – 4, 8, 11
Math & Reading – 11
Predictive
Value-added Measure of Growth
Math and Reading (Grades 4-8)



Comparison to a State Growth Standard
The Growth Standard specifies the minimal
acceptable academic gain from grade to
grade for a cohort of students.
The use of a Growth Standard creates the
possibility that ALL schools can demonstrate
appropriate growth.
PVAAS Value-Added
Growth Descriptors – Grade 4 – 8
Math and Reading
The PA Growth Standard specifies the minimal acceptable
academic gain from grade to grade for a cohort of students. The
Growth Standard is met when the students in a school or district on
average maintain their relative position when compared to the 2006
statewide distribution of student scores (2006 is the base year).
Estimated mean NCE gain equal or greater than zero. There is
evidence that the school has met or exceeded the Growth
Standard. All schools can achieve this rating.
Estimated mean NCE gain less than zero by one standard error or
less. There is slight evidence that the school has not met the
Growth Standard.
Estimated mean NCE gain less than zero by between one and two
standard errors. There is moderate evidence that the school has
not met the Growth Standard.
Estimated mean NCE gain more than two standard errors less than
zero. There is significant evidence that the school has not met the
Growth Standard.
PVAAS School Report
Grades 4 through 8 - Math and Reading
What Value-added Gain Means!
Cohort’s
Achievement
7th grade
66.6
Cohort’s
Achievement
6th grade
60.7
0
10
20
30
40
50
60
70
80
90
100
NCE Scale
State Mean = 50
PVAAS Gain ≈ Change in position from 6th to 7th grade
≈ 66.6 – 60.7 ≈ 5.9 NCE points
PVAAS Growth Methods
Testing
Subjects – Grades
Methodology
PSSA in
consecutive years
Math & Reading – 4-8
Growth
Standard
PSSA not in
consecutive
years
Writing – 5, 8, 11
Science – 4, 8, 11
Math & Reading – 11
Predictive
Value-added Measure of Cohort Growth
Math and Reading (Grade 11)
Science (Grades 4, 8, and 11)
Writing (Grades 5, 8, and 11)
Based on a predictive methodology…




A predicted score is calculated for each
student;
Mean of predicted scores is calculated for
students in a cohort;
Mean of observed scores of students in the
cohort;
School Effect is the difference between the
Observed and the Predicted Scores.
PVAAS Predictive Analysis
1. We collect all of the individual student data that is
available for a minimum of two years.
2. All of these data are used to develop individual
student profiles.
3. PVAAS then calculates a Predicted Performance
Value based on the student profile for each
student.
Year 1
PSSA
Year 2
PSSA
Year 3
PSSA (11th)
7th
8th
Predicted
Actual
Reading
1280
1300
1290
??
Math
1330
1305
1320
??
PVAAS Ratings of School Effect
Remember:
School Effect  Mean of Student Scores – Mean of Predicted Scores
So, if the students perform as expected, School Effect  0.
The ratings are in terms of the expected value of 0.
Not Detectably
Below
Above
Different
-2 SEs
0
+2 SEs
Standard Error (SE) is a measure of variation in the prediction.
Writing, Science & Grade 11
Reading/Math Growth Descriptors
Progress is defined by the district or school
effect, which is a measure of the difference
between observed performance and predicted
performance.
Progress is significantly above the progress of
the average district in the state.
Progress is not detectably different from the
progress of the average district in the state.
Progress is significantly below the progress of
the average district in the state.
PVAAS School Report
Grades 11- Math and Reading
Grades 4, 8 & 11 – Science
Grades 5, 8 & 11 – Writing
Summary of
PVAAS Growth Methods
Testing
Subjects – Grades
Methodology
PSSA in
consecutive years
Math & Reading – 4-8
Growth
Standard
PSSA not in
consecutive years
Writing – 5, 8, 11
Science – 4, 8, 11
Math & Reading – 11
Predictive
Summary:
One Year’s Growth
Growth Standard Method
Math and Reading – Grades 4 through 8
Cohort maintains its achievement position.
Predictive Method
Math & Reading – Grade 11
Writing – Grades 5, 8, 11 Science – Grades 4, 8, 11
Cohort’s actual performance is as expected.
PVAAS Performance Diagnostic Report
The Key is the RED Whisker!
Growth Standard Methodology
What the whiskers tell us…
Exceeded Growth Standard;
More than One Year’s Growth
Green
Met the Growth Standard;
Made One Year’s Growth
Yellow
Growth
Standard
(One Year’s
Growth)
Below Growth Standard;
Less than One Year’s Growth
Rose
Predictive Methodology
What the whiskers tell us…
ABOVE Expectations -Observed Score Was Greater
Than Predicted Score
NDD – No Detectable Difference
Between Observed Score and
Predicted Score
Observed –
Predicted
(One Year’s
Growth)
BELOW Expectations -Observed Score Was Less
Than Predicted Score
Performance Diagnostic Report
Greater Detail – based on % of students
Click on
Hot Keys
for lists of
students.
Student Lists from Performance
Diagnostic Reports
Your District
and School
Data
PA’s Intent for PVAAS



Another Tool for Continuous
Improvement
Measure of Growth to Add to
Achievement/Status Measures
Not to Be Used in Isolation
Questions and Feedback
Questions:
PVAAS Materials or
Statewide Implementation
[email protected]
717-606-1911
PVAAS Report Web Site
https://pvaas.sas.com
www.pde.state.pa.us
333 Market Street
Harrisburg, PA 17126