Transcript www.wiu.k12.pa.us

PVAAS Overview:
Evaluating Growth,
Projecting Performance
PVAAS Statewide Core Team
Fall 2011
August 2011
1
Partners & Teams for Session
Activities


The activities throughout the session
will be completed in pairs or small
groups, while others will be done
independently.
Take a few minutes now to divide
yourselves into:
• Partners (groups of 2), AND
• Small groups (groups of 4-6).
August 2011
2
Session Workbook

Workbook contains:
• Key notes from PowerPoint
• Session activities

Yellow boxes on PPT slides indicate
the page number(s) you should be
referencing in your workbook.
August 2011
3
WB
P. 1
Goal of Session
By the end of the session, participants will be able
to answer the following questions:
•
•
•
•
August 2011
What is PVAAS? How does PVAAS measure
growth?
What does PVAAS tell us about growth in our
district and school?
What does PVAAS tell us about how students are
projected to perform on a future PSSA?
Which reports are most meaningful for school
level planning, grade level planning and student
level planning?
4
WB
P. 1








August 2011
Agenda
Pre-test
What is PVAAS?
PVAAS Value-added Reports
Going Deeper with Value-added
PVAAS Projections
Projection Summaries
PVAAS Help Menus and Podcasts
Post-test
5
WB
P. 2
PVAAS Overview Pre-test!
Complete
Independently
August 2011
6
WB
P. 3
What Is
PVAAS?
August 2011
7
WB
P. 3
PVAAS IS:
Analysis of existing assessment data

• Longitudinal PSSA scores in available
content areas

Math, Reading, Science, Writing
• Provides reliable estimates
A measure of a school’s effectiveness
and growth over time.


Critical Relationship:
Growth Data vs. Achievement Data
August 2011
8
WB
P. 3
PVAAS:

Is not another test

Critical concept:
Is not a comparison of single test
scores
August 2011
9
WB
P. 3
Review of Literature
Is there a relationship between:

Demographics vs. Achievement?

Demographics vs. Growth?
Demographics are inherently
controlled for in PVAAS by the
use of all longitudinal data on all
students.
August 2011
10
WB
P. 4
Summary and Reflections
Work with a partner to
complete the activity
August 2011
11
WB
P. 4
Summary and Reflections
Answers
August 2011
12
Achievement and Growth
Together!
August 2011
13
WB
P. 5
Current Data Collection
Approaches
High Achievement
Average Achievement
Low Achievement
August 2011
Achievement is based
on each student’s ending point.
14
WB
P. 5
PVAAS Adds Another Variable To
Achievement…GROWTH!
Below Standard
for PA
Academic Growth
Above Standard
for PA
Academic Growth
Progress is based
on each student’s starting point.
August 2011
15
WB
P. 5
Take notes on page 5 of your workbook
using the information on the next slides
August 2011
16
WB
P. 5
The Scatter Plot
Coordinate System
Vertical Axis
PSSA Percent
Proficient or
Advanced
Standard for PA
Academic Growth
Boundary
AYP Percent
Proficient Target
Horizontal Axis
PVAAS Growth
Index
August 2011
17
WB
P. 5
Scatter Plot Quadrants
and Their Meanings
Quad 2 - Slipping
Adequate
Achievement
Negative Growth
Quad 3 Underperforming
Below Adequate
Achievement
Negative Growth
August 2011
Quad 1 Excelling
Adequate
Achievement
Positive Growth
Quad 4 Improving
Below Adequate
Achievement
Positive Growth
18
WB
P. 6



August 2011
Interpreting Scatter Plots
Work with your partner to review either
scatter plot A or scatter plot B.
Fill-in strengths and weaknesses of the
scatter plot (A or B) on page 8 of your
workbook. Remember, each “dot”
represents a school!
We will discuss as a group…“What can you
conclude by looking at both?”
19
WB
P. 6
Example: District Scatter Plot
A
August 2011
Note: Each dot on the scatter plot represents a school from this district.
20
WB
P. 6
Example: District Scatter Plot:
Same Schools, Different Subject
B
August 2011
Note: Each dot on the scatter plot represents a school from this district.
21
WB
P. 7
NEW! Web-Based Scatter
Plots
Page 7 in your workbook.
August 2011
22
WB
P. 7
August 2011
Web-based Scatter Plots
23
WB
P. 7
Web-based Scatter Plots!
Users may choose to plot different variables
with achievement and/or growth!
This includes demographics such as:
•
•
•
•
August 2011
% Tested Economically Disadvantaged
% Tested Limited English Proficiency
% Tested Special Education
% Tested Minority
24
WB
P. 7
August 2011
Example
25
WB
P. 8
Summary and Reflections
Work independently to complete the Summary and
Reflections page in your workbook.
August 2011
26
Two Types of PVAAS
Information
August 2011
27
WB
P. 9
TWO Types of PVAAS Information
Looking
Forward/Planning…
Looking Back/Evaluation…
PVAAS Projection Reports
For Individual Students
and Groups of Students
Value-added Growth Reports For
Groups of Students
Today
August 2011
28
WB
P. 9
PVAAS Value-Added
Reporting
How is growth measured in PVAAS?
August 2011
29
WB
P. 9
Value-added
Wouldn’t it be great
to know if your district/school
met the standard for PA
Academic Growth for
students tested in grades 4-8?
And in grades 9-11?
August 2011
30
WB
P. 9
PVAAS Growth Methodologies
Testing
Subjects – Grades Methodologies
PSSA in
consecutive
years
Mathematics 4-8
Reading 4-8
Growth Standard
PSSA not in
consecutive
years
Writing – 5, 8, 11
Science – 4, 8, 11
Math & Reading – 11
Predictive
August 2011
31
WB
P. 10
August 2011
Growth Charts
32
WB
P. 10
Growth Charts
Vertical Axis
Length (cm)
Percentile Bands
3rd-15th-50th-85th-97th
Horizontal Axis
Age (months - years
August 2011
33
WB
P. 10
August 2011
Growth Charts
34
WB
P. 10
Concept of Growth
Growth Standard Methodology
Grades 4–8, Math and Reading
Complete the activity on page 10 in your
workbook.
August 2011
35
WB
PP. 10 & 11
August 2011
Growth Chart Activity
36
WB
P. 11
One example of a
growth spurt from 6
months to 18 months
One example of less than
typical growth from 6
months to 18 months
X
X
X
August 2011
X
37
A Challenge to Comparing
Scores Across Years

The means and standard deviations
of PSSA exams change every year.
• A score of 1300 can be above the state
mean one year and below it in another.


Solutions – rescaling using a Normal
Curve Equivalent Scale (NCE).
Rescales all tests to have a state
mean of 50 and standard deviation
of 21.06.
August 2011
38
Example of Rescaling
Grades 4-8: Math & Reading
1000 1100 1200 1300 1400 1500
Scaled Score Units
0 10 20 30 40 50 60 70 80 90 100
NCE Units
August 2011
39
WB
P. 12
Growth Standard Methodology
Grades 4 through 8 – Reading & Math
A group of students makes one year’s growth when…
4th NCE Distribution 2006
The group maintains
their relative
achievement level
from one year to the
next in relation to
the statewide
distribution (2006
baseline).
August 2011
Note: A dot represents a school.
Group’s Position – Last Year
5th NCE Distribution 2006
Group’s Position – This Year
40
WB
P. 12
Complete the calculation in
your workbook using the
information on the next slide.
August 2011
41
WB
P. 12
What Value-Added Growth Means!
Growth Standard Methodology
Grades 4-8, Reading and Math
Achievement
6th grade
52.7
Achievement
5th grade
47.6
0
10
20
30
40
50
60
70
80
90
100
NCE Scale
State Mean = 50
PVAAS Growth ≈ Change in position from 5th to 6th grade
August 2011
≈ 52.7 – 47.6 ≈ 5.1 NCE units
42
WB
P. 13
Complete
“Check for Understanding”
August 2011
Complete independently
or with your partner
43
WB
P. 13
Example of Growth Measure=0 for
a low-achieving district/school
August 2011
Example of Growth Measure=0 for
a high-achieving district/school
44
WB
P. 13
Example of negative growth
for a district/school
August 2011
45
WB
P. 14
August 2011
Answer the questions on page
14 with your partner.
46
WB
P. 14
Value-added Measure of Growth
Grades 4-8 Reading and Math



Comparison to a State Growth Standard
The Growth Standard specifies the
minimal acceptable academic gain from
grade to grade for a group of students.
The use of a Growth Standard creates
the possibility that ALL schools can
demonstrate appropriate growth.
August 2011
47
WB
P. 15

NEW! Legends and Color Codes
Alignment of colors and interpretative legends
across all Value-Added Reports (all subjects and
grade levels)
• New 5 color-coding scheme
• All colors now have the same interpretation and
meaning
• Use of the data becomes easier as there is no
confusion about what the colors indicate

New 5 Color Codes
• Allow for more differentiation among schools
making a year’s worth of academic growth or
more
• Allow for more differentiation in Science and
Writing reporting, as well as grades 9-11 Reading
and Math reporting
August 2011
48
WB
P. 15
August 2011
49
WB
P. 15
August 2011
NEW! PVAAS Value-added
Growth Descriptors
Grade 4-8, Math and Reading
DB
Significant evidence that the district/school
exceeded the standard for PA Academic Growth
LB
Moderate evidence that the district/school
exceeded the standard for PA Academic Growth
G
Evidence that the district/school met the standard
for PA Academic Growth
Y
Moderate evidence that the district/school did not
meet the standard for PA Academic Growth
R
Significant evidence that the district/school did not
meet the standard for PA Academic Growth
50
WB
P. 16
August 2011
Complete the boxes on page 16.
51
WB
P. 16
August 2011
PVAAS School Report
Grades 4 through 8
52
WB
P. 16
Your Turn!
Interpret this report on page 16.
August 2011
53
WB
P. 17
Check for Understanding
In Your Own Words
Describe the growth of
students tested in 5th
grade math in 2011.
What does the 3-year
growth history of 5th
grade math indicate?
August 2011
What does the 3-year
achievement estimate
history indicate for this
most recent group of
students in 5th grade?
54
WB
P. 18
Concept of Growth:
Predictive Methodology
Writing and Science
Grade 11 - Math and Reading
Complete the activity on workbook page 18.
August 2011
55
WB
P. 18
Predictive Methodology
Writing, Science, and Grade 11 Reading & Math
The mean observed score from the actual test is not
significantly different from the mean predicted score
for the group of students investigated. The mean
predicted score is calculated based on all Reading
and Math data in each student’s record in the group.
Mean Observed Score ≈ Mean Predicted Score
Mean Observed Score
Mean Predicted Score
August 2011
± error
56
WB
P. 19
PVAAS Value-added Reports
Writing, Science & Grade 11 Math, Reading
Predictive Methodology
• A predicted score for each student;
• Mean of predicted scores for the group of students;
• Mean of observed scores for the group of students.
Growth Measure ≈ Observed Mean - Predicted Mean
August 2011
57
NEW! PVAAS Value-added
Growth Descriptors
Grade 9-11, Math and Reading
Science and Writing
WB
P. 19
Note that these
Significant evidence that the district/school
are now the same
DB
exceeded the standard for PA Academic Growth
growth color and
descriptors asModerate evidence that the district/school
GradesLB
4-8 Math
exceeded the standard for PA Academic Growth
and Reading!!
Evidence that the district/school met the standard
G
for PA Academic Growth
August 2011
Y
Moderate evidence that the district/school did not
meet the standard for PA Academic Growth
R
Significant evidence that the district/school did not
meet the standard for PA Academic Growth
58
WB
P. 20
August 2011
Answer the questions in the
boxes on page 20 using the
information on the next slide
59
WB
P. 20
Science Value-added Report
August 2011
60
WB
P. 21
Independent Practice
Report A
Report B
August 2011
61
WB
P. 22
Summary and Reflections
Work independently to answer the
questions on page 22.
August 2011
62
WB
P. 23
Growth Standard Methodology
Math & Reading – Grades 4 through 8
Definition:
A group meets the standard for PA
Academic Growth when they maintain
their relative achievement level from
one year to the next in relation to the
statewide distribution (2006 baseline).
WB
P. 23
Predictive Methodology
Math & Reading – Grades 9-11
Writing – Grades 5, 8, & 9-11
Science – Grades 4, 8, & 9-11
Definition:
A group meets the standard for PA Academic
Growth when their actual achievement meets
their predicted achievement - based on the
average schooling experience.
Going Deeper…
Performance Diagnostic Report
WB
P. 24
Fill-in the appropriate information on page 24 using
the information on the next three slides.
August 2011
65
WB
P. 24
Performance Diagnostic Report
Blue Bar –
Current
Year
Gold Bar –
Previous
Years
August 2011
Missing Bar –
Insufficient
Number of
Students
Whisker – Margin
of Error on
Growth Value
66
WB
P. 24
Complete the balloons for the
picture at the bottom of page 24.
August 2011
67
WB
P. 24
PVAAS Performance Diagnostic
Growth Descriptors - Interpretation
What the whiskers tell us…
Exceeded the standard for
PA Academic Growth
Blue
Met the standard for PA
Academic Growth
Green
Zero (0) Line
Growth
August 2011
Did not meet the standard
for PA Academic Growth
Pink
68
WB
P. 24
Performance Diagnostic Report
August 2011
69
WB
P. 25
Performance Diagnostic
Use the information on the next two slides
to complete the boxes on page 25.
August 2011
70
WB
P. 25
Link to all
students
reported for
this subject.
August 2011
Links to
students in
selected
predicted
PSSA
performance
category
71
WB
P. 25
Student Lists from Performance
Diagnostic Reports
Current PSSA
Performance
Category for each
student
August 2011
72
WB
P. 26
Use the information on the next slide
to complete the boxes on page 26.
August 2011
73
WB
P. 26
Patterns of Growth
A
C
B
August 2011
74
WB
P. 26
August 2011
Check for Understanding
75
WB
P. 27
August 2011
Performance Diagnostic
Subgroup Reports
76
WB
P. 27
August 2011
Performance Diagnostic
Subgroup Report
77
WB
P. 27
Best to Compare Subgroup
to Entire Grade Level
Same School – Same Grade – Same Subject
Entire grade level
August 2011
Subgroup
78
WB
P. 28
Summary and Reflections
Discuss with a partner. List conclusions you can make
about the growth of different groups of students in 8th
grade Reading.
Entire Grade Level
Economically Disadvantaged
August 2011
Special Education
79
PVAAS Projections
Individual Students and
Groups of Students
August 2011
80
WB
P. 29
Student Projections
Wouldn’t it be great
to know the likelihood
that a student will
be proficient on a
future PSSA?
August 2011
81
WB
P. 29

Student Projection
Methodology
Is this the same methodology as the one used to
estimate growth?
• No, the projection methodology is a separate modeling
process that focuses on individual students.

What data are used in this methodology?
• ALL available longitudinal data in both Reading and Math
are used in projection calculations!

What students have projections? In what subjects?
• Projections are available for all students who have a
minimum of 2 years of historical data. The next slide details
the grade levels and subjects in which projections are
reported.
August 2011
82
WB
P. 29
PVAAS Fall 2011 Reporting:
Projections to Basic, Proficient, Advanced
Math and Reading:
 3  4 or 5
 4  5 or 6
 5  6 or 7
 6  7 or 8
 7  8
 8  11
Writing:
 3  5
 4  5
 5  8
 6  8
 7  8
 8  11
Science:
 3  4
 4  NA
 5  8
 6  8
 7  8
 8  11
Grade PSSA
Grade PSSA
Grade PSSA
Last Taken
Last Taken
Last Taken
August 2011
83
WB
PP. 29-30
Information to complete boxes on pages
29 & 30 in your workbook will be
discussed on the next two slides.
August 2011
84
WB
P. 29
August 2011
PVAAS Student Report
85
WB
P. 30
August 2011
PVAAS Student Projection Report
86
WB
PP. 31 & 32
Independent Practice
Report A
Report B
August 2011
87
WB
P. 33
PVAAS Projections for
Groups of Students
Student Search
August 2011
88
WB
P. 33
August 2011
Identified Students
Meeting Search Requirements
89
Quality of Projections for
Decision Making



The projection is precise, and is created using a model that
has been reviewed and approved by four different peer
review panels and the GAO (US Government Accountability
Office).
The most recent 2008 growth model proposal to USDOE
includes information regarding the statistical model and
projection reliability study.
(http://www.ed.gov/admins/lead/account/growthmodel/pa/
index.html)
Recent studies have confirmed that the PVAAS projections
(even as far as 3 years into the future) are more reliable at
looking at the future performance of a student than the
most recent PSSA score.
August 2011
90
WB
P. 34
Summary and Reflections
Discuss with a partner and answer the questions
for the reports displayed on page 34.
August 2011
91
WB
P. 35
Projection Summary Reports

What are they?
• This is a report that summarizes the numbers and
percentages of students in various likelihood ranges of
performing at a proficient level on a future PSSA exam.

How might a school use these reports?
•
•
•
•

Intervention Planning
Resource Allocation
Strategic Planning
School Improvement Planning
Cautions
• This report provides ONE indicator about likelihood of
future performance.
• Additional data should be used with the projection for
August 2011
better decision-making.
92
WB
P. 35
August 2011
School (Single Grade) Projection
Summary Report
93
WB
P. 36
District (Single Grade) Projection
Summary Report
August 2011
94
WB
P. 36
Summary and Reflections
Discuss with a partner and answer the
questions on page 36 of your workbook.
August 2011
95
TWO Types of PVAAS Information
Looking
Forward/Planning…
PVAAS Projection Reports
For Individual Students
and Groups of Students
Looking Back/Evaluation…
Value-added Growth Reports For
Groups of Students
Today
August 2011
96
WB
P. 37
PVAAS Help Menus
August 2011
97
WB
P. 37
PVAAS Help Menus
August 2011
98
WB
P. 38
PVAAS Podcasts on iTunes U
August 2011
99
August 2011
100
Fall Professional
Development Opportunities
Check out what is being offered this school
year by visiting the PVAAS login page!
WB
P. 38
August 2011
PA’s Intent for PVAAS

Another Tool for Continuous
Improvement

Measure of Growth to Add to
Status/Achievement Measures

Not to Be Used in Isolation
102
WB
P. 39
PVAAS Overview Post-test!
August 2011
103
WB
P. 39
Discussion of Post-Test

As a small group, review questions 1-10 on
the Post-test.

Most importantly, report out and discuss
question 11 as a small group – 3 ways in
which you intend to use PVAAS.
August 2011
104
WB
P. 40
Questions:
PVAAS Materials or
Statewide Implementation
[email protected]
717-606-1911
PVAAS Report Web Site
https://pvaas.sas.com
August 2011
105
www.pde.state.pa.us
333 Market Street
Harrisburg, PA 17126
August 2011
106