Transcript 2011 STAR Post-Test Workshop Slides
2011 STAR Interpreting and Using Results
August 10, 2011 Webcast Webcast starts at 9:00 a.m.
Objectives
• • • • •
Workshop participants will be able to: Describe the purposes of STAR reports Interpret STAR results Explain key statistics Compare and contrast types of reports Identify proper uses of reports
2 August 2011 Post-Test Workshop
Agenda
• • • • • • •
What’s New?
Results and Statistical Analysis Using Results Summary and Internet Reports Data CDs Individual Student Reports Early Assessment Program
3 August 2011 Post-Test Workshop
4 • •
What’s New in 2011 California Modified Assessment (CMA) reports
–
Scale score, performance levels for
ELA grade 9
Life Science, grade 10
Algebra I
–
Percent correct for
ELA, grades 10 and 11
Geometry Writing component of grade 4 CST and CMA ELA restored
Manual (M) 2 August 2011 Post-Test Workshop
Quiz Question 1 Which of these tests had scale scores reported for the first time in 2011?
A. CST for World History B. CAPA for Science C. CMA for Algebra I D. CMA for Geometry
5 August 2011 Post-Test Workshop
Quiz Question 1 Which of these tests had scale scores reported for the first time in 2011?
C. CMA for Algebra I
6 August 2011 Post-Test Workshop
• • •
Results: Purposes of STAR Reports Report progress toward proficiency on the state’s academic content standards Notify where improvement needed
– –
To help students’ achievement To improve educational programs Provide data for state and federal accountability programs
M 3 7 August 2011 Post-Test Workshop
8
Results: Performance Levels
•
State goal: All students score at proficient or higher
•
350 or higher scale score
– –
All CST CMA for ELA in grades 3 –9, math in grades 3 –7 and Algebra I, and science in grades 5, 8, and 10
–
STS for RLA and math in grades 2 –7, excluding Algebra I
•
CAPA proficient: 35 or higher scale score
August 2011 Post-Test Workshop M 8−9
9 • • • • •
Results: Other Performance Levels Advanced Basic cut score
–
CST: 300
–
CMA: 300 for ELA in grades 3 –9, math in grades 3 –7, Algebra I, science in grades 5, 8, and 10
–
STS: 300 for RLA and math in grades 2 –7 (excluding Algebra I)
–
CAPA: 30 Below basic Far below basic For each testing program, cut points vary for advanced and below basic by
–
Subject
–
Grade
M 8−9; Appendix B August 2011 Post-Test Workshop
• • •
Results: Scale Scores
M 7−11
Scale scores allow the same score to mean the same thing across test versions within grade and content area Accounts for differences in difficulty Scale score ranges by program:
–
CST, CMA (ELA grades 3 –9, mathematics 3 –7, Algebra I, science in grades 5, 8, and 10), STS (grades 2 –7, excluding Algebra I): 150 –600 for each grade and subject
–
CAPA: 15 –60 for each level and subject
10 August 2011 Post-Test Workshop
Results: Equating
• • •
Psychometric procedure Adjusts for test difficulty Additional information in the technical report on the CDE Web site
M 7 11 August 2011 Post-Test Workshop
•
Results: Reporting Clusters (Content Areas)
M 9−11; Appendix A
Three to six clusters for each subject
• •
May be useful as indicators of individual or group strengths and weaknesses But . . . Reporting clusters should be interpreted with caution
12 August 2011 Post-Test Workshop
• • •
Results: Cluster Percent Correct Available for 2011
M 9−11; Appendix A
CST: all CMA:
–
Grades 3 –9 ELA, grades 3–7 math, Algebra I, grades 5, 8, and 10 science STS: all
13 August 2011 Post-Test Workshop
• • •
Results: Reporting Cluster Cautions Based on small numbers of items; therefore, may not be reliable or generalizable
M 9−11; Appendix A
NOT equated from year to year Should not compare reporting cluster percent correct from year to year
14 August 2011 Post-Test Workshop
•
Interpreting Reporting Clusters or Content Areas in the Same Year
M 8−12; Appendix A
Compare to percent-correct range of proficient students statewide
15 August 2011 Post-Test Workshop
2011 CST Reporting Clusters: Number of Questions and Average Percent Correct
M Appendix A
2011 Post-Test Guide, Appendices A and C as posted on startest.org will be finalized with complete data on August 15.
16 August 2011 Post-Test Workshop
Examples —Interpreting Reporting Clusters for the CST for Geometry
M 11 My Class 2010 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Just Proficient Students 2010 Just Advanced Students 2010 Logic/Geom Proofs Volume & Area Angle Relationships Trigonometry My Class 2011 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Just Proficient Students 2011 Logic/Geom Proofs Volume & Area Just Advanced Students 2011 Angle Relationships Trigonometry 17 August 2011 Post-Test Workshop
Quiz Question 2 What is a scale score?
A. Percent correct of all questions B. Mean percent correct of all questions C.
An adjustment of this year’s and last year’s raw scores to show changes D. An adjustment of the raw score to account for differences in difficulty
18 August 2011 Post-Test Workshop
Quiz Question 2 What is a scale score?
D. An adjustment of the raw score to account for differences in difficulty
19 August 2011 Post-Test Workshop
• • •
Using Results
M 3
For instructional decisions with other data Used in Academic Performance Index (API) calculations: CSTs, CAPA, CMA (only grades 3 –9, Algebra I, Life Science) Used in adequate yearly progress (AYP) calculations, ELA and mathematics:
–
Grades 2 –8, CSTs
–
Grades 3 –8 and Algebra I, CMA
–
Grades 2 –8 and 10, CAPA
20 August 2011 Post-Test Workshop
Year-to-Year Comparisons Do Compare CSTs:
M
Same Grade and Same Content Area
12−15 • •
Mean scale score
–
Same content and grade, varying years Percent in each performance level
–
Same content by grade across years
e.g., 2010 grade 10 ELA with 2011 grade 10 ELA
21 August 2011 Post-Test Workshop
•
Year-to-Year Comparisons Do Compare CSTs: Percent Proficient and Advanced
M 12−15
Percentage of students scoring at PROFICIENT and above
–
For a given grade and subject, e.g., Percent proficient and above for grade 3 math in 2010 and 2011
–
For a given subject and aggregated grades, e.g., Percent proficient and above for grades 2 – 6 mathematics in 2010 and 2011
–
Across grades and a subject, e.g., Percent proficient and above in all courses and all grades
22 August 2011 Post-Test Workshop
23 • • • • •
Year-to-Year Comparisons DON’T Compare
M 12−15
Individual scale scores or statistics based on scale scores for different grades or content areas
–
Subjects by grade are independently scaled
–
Different content standards are measured in different grades Cohorts across grades Across tests CMA grades 9−11, STS grades 8−11 to previous years
–
CMA grades 6−8, STS grades 5−7, only compare 2011 to 2010 CAPA, CMA grades 3−5, STS grades 2−4, DON’T compare to years before 2009
August 2011 Post-Test Workshop
Example —Using CST Results to Compare Grade Results from Year to Year
M 14
Grade Grade 2 Grade 3 Grade 4 Grade 5 Grade 6 All Grades 2010 CST for ELA % Prof or Above 31% 33% 29% 34% 31% 32% 2011 CST for ELA % Prof or Above 35% 33% 31% 32% 32% 33% Difference 4% 0% 2% –2% 1% 1%
24 August 2011 Post-Test Workshop
25
Quiz Question 3 What is the best comparison for CST scores of students within a middle school?
A. 2010 mean scale scores for ELA of a cohort of grade 7 students with 2011 scale scores for ELA of the same students in grade 8 B. 2010 mean scale scores for ELA for grade 8 students with 2011 mean scale scores for ELA for grade 8 students C. 2010 mean percent correct scores for ELA with 2011 percent correct scores for ELA for the same students in grade 8 D. 2010 mean percent correct scores for ELA for grade 8 students with 2011 mean percent correct for ELA for grade 8 students
August 2011 Post-Test Workshop
Quiz Question 3 What is the best comparison for CST scores of students within a middle school?
B. 2010 mean scale scores for ELA for grade 8 students with 2011 mean scale scores for ELA for grade 8 students
26 August 2011 Post-Test Workshop
Quiz Question 4 Which is the best comparison of cluster scores for a single student? Compare . . .
A. To proficient students statewide B. One cluster to another, same year C. The same cluster to the same cluster, different years D. To the average percent correct of all students in a class
27 August 2011 Post-Test Workshop
Quiz Question 4 Which is the best comparison of cluster scores for a single student? Compare… A. To proficient students statewide
28 August 2011 Post-Test Workshop
29 • •
Aggregate (Summary) Reports
M 18−22
What are they?
– –
Student Master List Summary Student Master List Summary End-of Course (EOC)
– –
Subgroup Summary Back of reports: guide to abbreviations, score codes Report Emphasis: CSTs
– –
Criterion-referenced tests Progress is measured in percent of students scoring proficient and advanced
August 2011 Post-Test Workshop
• • • • •
Student Master List Summary By grade or EOC subject CSTs, CMA, CAPA, and STS Lists subjects or EOC grades % and # at each performance level
M M 19–20, 29−35
Mean scale score
–
CST
–
CAPA
–
CMA ELA in grades 3 –9, math in grades 3–7, Algebra I, and science in grades 5, 8, and 10
–
STS RLA and math in grades 2 –7 [not Algebra I])
30 August 2011 Post-Test Workshop
Student Master List Summary (cont.)
•
Reporting cluster: mean percent correct
– –
CST CMA ELA in grades 3 –9, math in grades 3 –7, Algebra I, and science in grades 5, 8, and 10
–
STS
31 August 2011 Post-Test Workshop
Student Master List Summary Grade 7 Example
M 35 32 August 2011 Post-Test Workshop
Student Master List Summary Basic Statistics
M 29−35 33 August 2011 Post-Test Workshop
• •
Who Counts? Number Enrolled Total CST/CMA and CAPA multiple choice answer documents submitted as scorable
M 31
Minus
–
Documents marked as “Student enrolled after the first day of testing and was given this test”
34 August 2011 Post-Test Workshop
Who Counts? Number Tested
M 31 35 • • •
All CST, CMA, CAPA, STS answer documents with one or more answers Plus
–
Z = Tested but marked no answers Not included
–
A = Students absent
–
E = Not tested due to significant medical emergency
– –
P = Parent/guardian exemptions T = Enrolled first day, not tested, tested at previous school
– –
Students with inconsistent grades Non –English learners who took the STS
August 2011 Post-Test Workshop
• • 36
Who Counts? Number and Percent Valid Scores Number Valid Scores
–
For the subject, number of students tested at grade level who received a score for the test (that is, a scale score or percent correct) Percent Valid Scores
–
For the subject, number of valid scores divided by the number of students tested
–
Not included:
Incomplete tests
Modified tests Non –English learners who took the STS Unknown EOC mathematics (except grade 7 mathematics) or science tests
Inconsistent grades
August 2011 Post-Test Workshop M 31
37 August 2011
Who Counts? Number Tested with Scores
•
with modifications, that receive a score
M 31
All tests taken, including those taken
•
Not included:
–
Incomplete tests
–
Non –English learners who took the STS
–
Unknown EOC mathematics (except grade 7 mathematics) or science tests
–
Inconsistent grades
Post-Test Workshop
Student Master List Summary Performance Levels
M 32 38 August 2011 Post-Test Workshop
• • • • •
Who Counts? Performance Levels
M 32
All CSTs, CAPA CMA grades 3 –9 ELA, grades 2–7 math, Algebra I, and grades 5, 8, and 10 science STS grades 2 –7 RLA and math (excluding Algebra I) Advanced, proficient, basic, below basic
–
All valid scores falling in the performance level Far below basic
–
All valid scores falling in the performance level
–
CSTs taken with modifications (in aggregate reporting and accountability only)
39 August 2011 Post-Test Workshop
• •
Who Counts?
Mean Scale Scores Average of valid scale scores Can be used to compare results for the same content/grade across years
M 32 40 August 2011 Post-Test Workshop
Student Master List Summary: Reporting Clusters
M 32 41
Compare to: Average percent correct range for students statewide who scored proficient on the total test (See the Post-Test Guide, Appendix A.)
August 2011 Post-Test Workshop
Student Master List Summary: Writing
M 33
I B = Blank C = Copied prompt = Illegible L = Language other than English
42 August 2011
R = Refusal T = Off Topic W = Wrong prompt (prompt from an earlier administration)
Post-Test Workshop
43 • • • • • •
Subgroup Summary: CSTs, CMA, CAPA, and STS
M 42−58
Disability status
–
Based on disability status for CST, CMA, STS
–
CAPA: each disability type Economic status
–
Based on NSLP eligibility or parent education level Gender English proficiency Ethnicity Ethnicity for Economic Status (only for CSTs; CMA in grades 3 –9 ELA; grades 3–7 math, Algebra I, and grades 5, 8, and 10 science; and CAPA)
August 2011 Post-Test Workshop
Subgroup Summary: Ethnicity for Economic Status
M 54 −58
Ethnicity for Economically Disadvantaged CST Grade 6 Example
44 August 2011 Post-Test Workshop
Subgroup Summary: Ethnicity for Economic Status
M 54 −58
Example: Economically disadvantaged for each ethnicity
45 August 2011 Post-Test Workshop
Subgroup Summary: Ethnicity for Economic Status
M 54-58 46 August 2011 Post-Test Workshop
Break — 10 minutes
48 • • • • • •
Internet Reports
M 101−111
Summaries based on same data as paper reports: CSTs, CMA, CAPA, STS Available to the public online for school, district, county, and state “Students with Scores” = number tested with scores CST Summaries of % Advanced and Proficient More subgroups than paper reports
– –
Parent education Special program participation Access from http://star.cde.ca.gov/
August 2010 Post-Test Workshop
Internet Demonstration
49 August 2011 Post-Test Workshop
Internet Reports: CST Sample
M 102 50 August 2011 Post-Test Workshop
Internet Reports: CST Summary Sample Only available for CSTs
M 103 51 August 2011 Post-Test Workshop
•
Other Internet Reports CMA
(M 104−105) –
Same as CST —CMA grades 3−9 ELA, grades 3 –7 math, Algebra I, grades 5, 8, and 10 science
52 • –
Average percent (%) correct —CMA grades 10 –11 ELA, Geometry CAPA
(M 106−109) –
State level: same as CST; separate Level I
–
County, district, school
Mean scale score
Percent proficient or above
August 2011 Post-Test Workshop
•
Other Internet Reports (cont.) STS
(M 110−111) –
Same as CST —STS grades 2−7 RLA and math (except Algebra I)
–
Average percent (%) correct —STS grades 8−11 RLA and Algebra I and Geometry
53 August 2011 Post-Test Workshop
Quiz Question 5 Which subgroup can only be accessed from the Internet?
A. Parent Education Level B. CAPA by individual disability status C. Ethnicity for Economic Status D. English Proficiency
54 August 2011 Post-Test Workshop
Quiz Question 5 Which subgroup can only be accessed from the Internet?
A. Parent Education Level
55 August 2011 Post-Test Workshop
56 • • •
Data CDs What are they?
–
Lists of information from answer documents and scores of every student in the district
–
In .txt format: wraps What are they used for?
–
Searching for specific data
– –
Creating unique reports Verifying paper reports What else is needed?
– – –
Text editor or Desktop application or Student Information System
August 2011 Post-Test Workshop
View of Data
•
As .txt, word wrap on
•
With text editor, word wrap off
57 August 2011 Post-Test Workshop
•
Organization of Data
•
Two files:
–
Demographics, special conditions, and test scores
–
Accommodations, modifications, English Learners, and irregularities Data Layout = guide to location of data on files
– – –
Position Number of characters Whether numeric or alpha
58 August 2011 Post-Test Workshop
Data Layout Sample
59 August 2011 Post-Test Workshop
• • •
Individual Reports
M
STAR Student Record Label
–
Adhesive label to affix to student’s permanent school record
23−27
STAR Student Master List
–
Alphabetical list of students and their scores
–
Tests listed in order within grade
CSTs
CMA
CAPA STS STAR Student Report: individual’s scores
–
2 two-sided color copies for each test
–
For parents/guardians, school
60 August 2011 Post-Test Workshop
Student Record Label Grade 10 Sample: Student Name and Identification
M 59−60 61 August 2011 Post-Test Workshop
Student Record Label CST/CMA Grade 10 Example
M 59−60 62 August 2011 Post-Test Workshop
Student Master List CST/CMA Grade 3 Example
M 61−65 63 August 2011 Post-Test Workshop
Student Report CST Grade 11 Example
M 66−97 64 August 2011 Post-Test Workshop
Student Report CST Grade 11 Example
M 75−76 65 August 2011 Post-Test Workshop
Student Report CST Grade 11 Example
M 76
Student’s name on back
66
♦ = Percent correct obtained by the student on the reporting cluster/content area ▬ = Average percent-correct range on the
August 2011
reporting cluster for students statewide who scored proficient on the total test scores
Post-Test Workshop
Explain to Parents Scale Score
Average % correct cluster score x 600
M 7 • •
Reporting clusters not comparable
– – –
Different difficulty Varying number of questions Average % correct of clusters
% correct on total test (or total number of test items) Scale scores
–
Use conversion tables resulting from statistical procedures
–
Equating allows scores to have similar meaning (e.g., 350 = lowest score for CSTs proficient)
67 August 2011 Post-Test Workshop
• •
Other Student Reports CMA
(M 77−84) –
Performance levels and scale scores reported for grades 3 science
–9 ELA, grades 3–7 math, Algebra I, and grades 5, 8, and 10 Back, cluster reporting
–
Percent correct for grades 10 and 11 ELA and Geometry
Back, “About the CMA” CAPA
(M 85−89) –
Back, “About the CAPA”
68 August 2011 Post-Test Workshop
•
Other Student Reports (cont.) STS
(M 90−97) – –
In Spanish
–
Performance levels and scale scores, grades 2 –7 RLA and math (except for Algebra I)
Back, cluster percent correct Percent correct for entire test, grades 8 –11 RLA and Algebra I and Geometry
Back, cluster percent correct
Back, how to use report
69 August 2011 Post-Test Workshop
Unmatched Reports Writing CST, CMA —Grs 4, 7 CST, CMA —Grs 8, 11
•
Only multiple-choice
•
Only writing
•
Students receive 2 reports if writing score not matched to multiple-choice score
• •
Only CMA and no CST Grades 8 and 11 require CST for History –Social Science
70 August 2011 Post-Test Workshop
71
Summary
•
Do’s and Don’ts
–
Do compare mean scale score, percent at performance levels within same grade, same content area
–
Do compare cluster scores to the corresponding proficient range provided for this year
–
Do not compare cluster scores to each other within test, nor across years, grades, content area
–
Do not compare mean scale scores across grades, content area
• •
Summary and Internet reports Data CDs
•
Individual reports
August 2011 Post-Test Workshop
Upcoming Dates Event Post-Test Guide posted on startest.org
State Superintendent’s public announcement Appendixes A & C Post-Test Guide posted on startest.org
Web Reports available Research files available Post-Test Guide distributed Date July 28 August 15 After August 15 August 15 August 15 Last weeks of August
72 August 2011 Post-Test Workshop
For more information see:
• • • • •
http://www.cde.ca.gov/ta/tg/sr/ http://www.star.cde.ca.gov/ http://www.startest.org/ STAR Technical Assistance Center
–
800-955-2954 CDE Accountability
– –
916-319-0863
73 August 2011 Post-Test Workshop
74
To Contact the CDE
•
CDE STAR office: 916-445-8765
–
English –Language Arts tests, including writing: Jamie Contreras, [email protected], 916-319-0353
–
Mathematics tests: Jane Liang, [email protected], 916-322-1854
–
Science tests: Blessing Mupanduki, [email protected], 916-319-0349
–
CST History –Social Science, CAPA, CMA: Don Killmer, [email protected], 916-319-0350
August 2011 Post-Test Workshop
Early Assessment Program Results
August 10, 2011 Dusty Fox
Early Assessment Program (EAP)
• EAP results are being accepted at participating California Community Colleges (CCC).
– Approximately 55 CCC campuses are accepting EAP results – List of participating colleges is available on the CCC website at
www.CollegeEAP.org.
Check the site often for an updated list of campuses.
76 PowerPoint slides - www.CollegeEap.org, Educators, About EAP .
August 2011 Post-Test Workshop
Early Assessment Program (EAP)
• An EAP informational web site has been developed.
www. CollegeEAP.org
.
– The site provides EAP information for both CSU and CCC – It is a resource for Students, Parents, and Educators 77 PowerPoint slides - www.CollegeEap.org, Educators, About EAP .
August 2011 Post-Test Workshop
www.collegeEAP.org
78 August 2011 Post-Test Workshop
Early Assessment Program (EAP)
•
Reporting Results
– Student Statuses are on the STAR Report – Ready for CSU and participating CCC college-level mathematics courses – Conditional • This status in conjunction with a grade of “C” or better in an approved math course or a supervised e-learning program can be exempt from the ELM for CSU or other math placement tests at participating California Community Colleges (CCC) • A list of courses and e-learning programs can be found at www.csumathsuccess.org
79 PowerPoint slides - www.CollegeEap.org, Educators, About EAP .
August 2011 Post-Test Workshop
Early Assessment Program (EAP)
–
Sample EAP Box on the STAR Student Report
The EAP box can be found on page 78 of the Post-Test Guide 80 PowerPoint slides - www.CollegeEap.org, Educators, About EAP .
August 2011 Post-Test Workshop
Early Assessment Program (EAP)
• Web Sites – www.CollegeEAP.org
Links to CSU and CCC informational sites for Students, Parents, and Educators – www.CalState.edu/EAP Provides a list of EAP Coordinators, EAP test blueprints, and informational materials – www.CCCCO.edu/EAP Provides a list of participating Community Colleges 81 PowerPoint slides - www.CollegeEap.org, Educators, About EAP .
August 2011 Post-Test Workshop