Transcript Using Assessment Program Resources to Shape Effective ELA

Using Assessment Program
Resources to Shape Effective
ELA Instruction
Amy F. Radikas, Literacy Specialist
Renee A. Savoie, NAEP State Coordinator
Connecticut Assessment Forum III
Crowne Plaza, Cromwell
August 13, 2012
Session Overview
 Key
shifts in ELA/Literacy
 NAEP
and PISA basics
 Evidence
of the shifts illustrated
through NAEP & PISA
 Discussion
CCSS Requires Three Shifts
in ELA/Literacy
#1
Building knowledge through
content-rich nonfiction
#2
Reading, writing and speaking grounded
in evidence from text, both literary
and informational
#3
Regular practice with complex texts
and its academic language
achievethecore.org
ACT Report
In 2006, ACT released a
report called Reading
Between the Lines. The
findings suggested that
the ability to read
complex texts is the
“clearest differentiator
between those ready for
college-level reading and
those not.”
ACT: Reading Between the Lines
The most important implication of this study:
“What students could read, in terms of its
complexity, was at least as important as
what they could do with what they read.”
CCSS Appendix A. p. 2
What is NAEP?
National Assessment of Educational Progress
 Nation’s Report Card
 Established by Congress in 1969 to answer
questions about student academic
performance
 The Nation’s ONLY monitor of what students
know and can do

◦ “common yardstick”

“Survey” design
◦ National, regional and state-level results
Overall NAEP Reading Results for
Connecticut 2003-2011
YEAR
2003
2005
2007
2009
2011
GRADE 4
% OF
AVG.
STUDENTS
SCALE
AT/ABOVE
SCORE
PROFICIENT
228
43
226
38
227
41
229
42
227
42
GRADE 8
% OF
AVG.
STUDENTS
SCALE
AT/ABOVE
SCORE
PROFICIENT
267*
37*
264*
34*
267*
37*
272*
43X
275X
45X
* indicates a statistically significant difference when compared to performance in 2011.
Grade 4 Reading in 2011:
Average Scale Score Comparisons
Higher avg. ss. than CT
Avg. ss. not significantly
different from CT
Lower avg. ss. than CT
Grade
8 Reading
in in
2011:
Grade
8 Reading
2011:
Average
Scale
Score
Comparisons
Average
Scale
Score
Comparisons
Higher avg. ss. than CT
Avg. ss. not significantly
different from CT
Lower avg. ss. than CT
What is PISA?
Click to view YouTube video about PISA

Program for International Student Assessment
(sponsored by the OECD)

Administered every 3 years since 2000

Assesses 15-year-old students

Measures application of knowledge and skills to
problems in a real-life context
PISA aims to answer…

Are students well prepared to meet the
challenges of the future?

Are they able to analyze, reason, and
communicate their ideas effectively?

Do they have the capacity to continue
learning throughout life?
Who Participates in PISA?
OECD countries (34)
Non-OECD education systems (33)
PISA 2009 Reading Literacy Results
Data for 65 education systems
(34 OECD and 31 non-OECD)
• U.S. not measurably different
from OECD average
• U.S. scored below 9 education
systems
• U.S. scored above 39
education systems
13
-40
Ireland
Argentina
Sweden
Czech Republic
Australia
Spain
Finland
Canada
Thailand
France
New Zealand
Iceland
United States
Romania
Japan
Russian Federation
Norway
Denmark
Italy
Bulgaria
Belgium
OECD average
Mexico
Switzerland
Hong Kong-China
Greece
Germany
Hungary
Korea
Brazil
Liechtenstein
Portugal
Poland
Israel
Latvia
Indonesia
Albania
Chile
Peru
Difference in score points
PISA Reading Literacy Trends
50
40
36
30
Higher score in 2009
20
10
1
-10
-20
-30
-20 -19
-13-13
3
Education systems
9
6 8
22
19 21
17
16
13 14 15
40
43
26
31
0
-2 -1 -1 -1
-5 -3 -2 -2 -2
-7
-8
-12 -11 -10 -9 -9
Higher score in 2000
-31
-50
PISA and NAEP: Key Differences
 PISA
includes a considerable amount
of noncontinuous text.
 NAEP measures “meaning vocabulary.”
 NAEP Grade 8 and 12 passages are
notably longer than PISA passages.
 In terms of readability and grade level,
PISA is closer to Grade 12 NAEP.
The Origins of CCSS
CCSS
International
Assessments
NAEP
CCSS Requires Three Shifts
in ELA/Literacy
#1—Building knowledge through
content-rich nonfiction
#2—Reading, writing and speaking grounded
in evidence from text, both literary
and informational
#3—Regular practice with complex texts
and its academic language
achievethecore.org
Informational Text
• Students are required to read very little
informational text in elementary and middle
school.
• Informational text is more difficult for
students to comprehend than narrative
text.
• Informational text makes up the vast
majority of required reading in
college/workplace.
NAEP Reading: Passage Types by Grade
Grade
Literary
Informational
4
50%
50%
8
45%
55%
12
30%
70%
CCSS ELA p.5
In the Early Years
Children must:
Develop strong foundational cognitive
skills (literacy/communication).
Grade Two
Informational
texts use clear and
consistent formats, use of simple
headings to organize information
into categories, illustrations
extend the meaning, and simple
graphics support understanding
of content
Elfrieda H. Hiebert, Fundamentals of Literacy Instruction and Assessment, Pre-K–6
Edited by Martha C. Hougen, Ph.D., & Susan M. Smart
An Example from NAEP

NAEP 2011 Grade 4 Informational Passage:
Marian’s Revolution by Sudipta Bardhan-Quallen
Available through the NAEP Questions Tool
One of 10 Released Items
Why is "A Voice for Civil Rights" a
good heading for the section that
follows it on page 3? Use information
from the article to support your
answer.
Student Responses:
Full Comprehension
Student Responses:
Partial Comprehension
Student Responses:
Little or No Comprehension
Item-Level Performance Data
Little/No
Avg.
SS
NP
CT
Partial
%
Avg.
SS
205
44
207
34
Full
%
Avg.
SS
234
40
239
49
NP= National Public
‡ Reporting standards not met.
Omitted
%
Avg.
SS
%
245
10
191
5
249
10
‡
5
CCSS Requires Three Shifts
in ELA/Literacy
#1—Building knowledge through
content-rich nonfiction
#2—Reading, writing and speaking grounded
in evidence from text, both literary
and informational
#3—Regular practice with complex texts
and its academic language
achievethecore.org
Evidence to Support
Text Dependent Questions
• Can be literal (checking for understanding)
but must also involve analysis, synthesis,
evaluation.
• Focus on word, sentence, and paragraph, as
well as larger ideas, themes, or events.
• Focus on difficult portions of text in order
to enhance reading proficiency.
• Can also include prompts for writing
(DOK) and discussion questions.
achievethecore.org
Bloom’s Taxonomy [1956 ] &
Bloom’s Cognitive Process Dimensions [2005]
Knowledge -- Define, duplicate, label,
list, name, order, recognize, relate, recall
Remember Retrieve knowledge from
long-term memory, recognize, recall,
locate, identify
Comprehension -- Classify, describe,
discuss, explain, express, identify,
indicate, locate, recognize, report,
review, select, translate
Understand -- Construct meaning,
clarify, paraphrase, represent, translate,
illustrate, give examples, classify,
categorize, summarize, generalize,
predict…
Application -- Apply, choose,
demonstrate, dramatize, employ,
illustrate, interpret, practice, write
Apply -- Carry out or use a procedure in
a given situation; carry out or use /apply
Analysis -- Analyze, appraise, explain
calculate, categorize, compare, criticize,
discriminate, examine
Analyze -- Break into constituent parts,
Synthesis -- Rearrange, assemble,
collect, compose, create, design,
develop, formulate, manage, write
Evaluate -- Make judgments based on
criteria, check, detect
inconsistencies/fallacies, critique
Evaluation -- Appraise, argue, assess,
choose, compare, defend, estimate,
explain, judge, predict, rate, core, select,
support, value
Create -- Put elements together to form
a coherent whole, reorganize elements
into new patterns/ structures
to an unfamiliar task
determine how parts relate
Karin Hess
Webb’s Depth-of-Knowledge Levels

DOK-1 – Recall & Reproduction - Recall of a fact, term, principle,
concept, or perform a routine procedure

DOK-2 - Basic Application of Skills/Concepts - Use of
information, conceptual knowledge, select appropriate procedures for
a task, two or more steps with decision points along the way, routine
problems, organize/display data, interpret/use simple graphs

DOK-3 - Strategic Thinking - Requires reasoning, developing a plan
or sequence of steps to approach problem; requires some decision
making and justification; abstract, complex, or non-routine; often more
than one possible answer

DOK-4 - Extended Thinking - An investigation or application to
real world; requires time to research, problem solve, and process
multiple conditions of the problem or task; non-routine manipulations,
across disciplines/content areas/multiple sources
Karin Hess
The Hess Cognitive Rigor Matrix integrates
Bloom + Webb
Different models used to describe
cognitive rigor:

Bloom – What type of thinking (verbs)
is needed to complete a task?

Webb – How deeply do you have to
understand the content to successfully
interact with it? How complex is the
content?
Karin Hess
The Hess Cognitive Rigor Matrix Applies Webb’s
DOK to Bloom’s Cognitive Process Dimensions
Depth +
Thinking
Level 1
Recall &
Reproduction
Remember
- Recall, locate basic
facts, details, events
Understand
Level 2
Skills &
Concepts
Level 3
- Select appropriate
words to use when
intended meaning is
clearly evident
- Specify, explain
relationships
- summarize
– identify main ideas
- Explain, generalize, or
connect ideas using
supporting evidence
(quote, example…)
- Explain how concepts
or ideas specifically
relate to other content
domains or concepts
Apply
- Use language structure
(pre/suffix) or word
relationships
(synonym/antonym) to
determine meaning
– Use context to identify
meaning of word
- Obtain and interpret
information using text
features
- Use concepts to solve
non-routine problems
- Devise an approach
among many
alternatives to research
a novel problem
Analyze
- Identify whether
information is contained
in a graph, table, text
feature, etc.
– Compare literary
elements, terms, facts,
events
– analyze format,
organization, & text
structures
- Analyze or interpret
author’s craft (literary
devices, viewpoint, or
potential bias) to
critique a text
– Analyze multiple
sources
- Analyze
complex/abstract
themes
– Cite evidence and
develop a logical
argument for
conjectures
- Evaluate relevancy,
accuracy, &
completeness of
information
- Synthesize information
within one source or
text
- Synthesize information
across multiple sources
or texts
Evaluate
Create
- Brainstorm ideas about
a topic
- Generate conjectures
based on observations
or prior knowledge
Strategic Thinking/
Reasoning
Level 4
Extended
Thinking
DOK is about complexity—not difficulty!

The intended student learning outcome determines the
DOK level. What mental processing must occur?

While verbs may appear to point to a DOK level, it is what
comes after the verb that is the best indicator of the
rigor/DOK level.
◦ Describe the process of photosynthesis.
◦ Describe how the two political parties are alike and
different.
◦ Describe the most significant effect of WWII on the
nations of Europe.
Karin Hess
Let’s practice
Your class has just read some version of
Little Red Riding Hood.
What is a basic comprehension question
you might ask?
What is a more rigorous question
you might ask?
Karin Hess
Depth +
Thinking
Level 1
Recall &
Reproduction
Remember
What color was
Red’s cape?
Who is this story
about?
Level 2
Skills & Concepts
Level 3
Strategic Thinking/
Reasoning
Understand
Identify words/phrases
that helped you to
know the sequence of
events in the story.
Apply
Analyze
Is this a realistic or
fantasy story?
Is this a realistic or
fantasy story? Justify
your interpretation
using text evidence.
Evaluate
Create
Write a telephone
conversation
between Red and
her mother to
explain the wolf
incident.
Level 4
Extended
Thinking
How can we apply these
ideas back in our schools
AND CLASSROOMS?
Guiding Questions
• What skills & concepts are most
important?
• Is the intended rigor of skills/concepts
reflected in materials used (e.g., texts,
tasks)
• Is there a range of DOK (rigor) within the
learning activities/lesson?
9-10
6-8
4-5
2-3
K-1
Standards Two through Nine
achievethecore.org
Increasing Range and Complexity
11CCR
Standard Ten
Standard One
Increased Ability to Use Text Evidence
Bands
Bands
11CCR
9-10
6-8
4-5
2-3
K-1
39
PISA Reading Unit 3: Graffiti
See Page 13
Question 4: Graffiti (p. 17)

Full Credit Guidelines: Explain opinion with
reference to style or form of one or both
letters. Refers to criteria such as structure of
argument, cogency of argument, strategies for
persuading readers, etc.

Example: “Helga’s. She gave you lots of
different points to consider and she
mentioned the environmental damage that
graffiti artists do which I think is very
important.”
Question 4:
International Performance Data
Overall Percent Correct
Japan
75
>
Canada
63
>
United Kingdom
62
>
New Zealand
58
>
Country average vs.
OECD average
Finland
58
>
Higher
>
Australia
54
=
Not Different
=
OECD Average
53
Korea, Republic of
52
=
Lower
<
Ireland
51
=
Poland
50
=
Norway
48
=
United States
46
<
Mexico
45
<
CCSS Requires Three Shifts
in ELA/Literacy
#1—Building knowledge through
content-rich nonfiction
#2—Reading, writing and speaking grounded
in evidence from text, both literary
and informational
#3—Regular practice with complex texts
and its academic language
achievethecore.org
Academic Vocabulary
• Words that add to students’ language ability
(e.g., maintain, fortunate, required, tend,
contrast/compare, insisted, summarize)

Words that are needed in a content area
(e.g., isotope, peninsula, photosynthesis,
cubism, isosceles triangle)
More Examples from NAEP

NAEP 2009 Grade 12 Literary Passage:
The Open Window by Saki
Available through the NAEP Questions Tool
NAEP’s “Meaning Vocabulary”
On page 3, the narrator says Framton has the delusion
that total strangers are interested in his nervous condition.
This means Framton
A. has a common illness related to his nerves.
B. is worried about what people think of him.
C. dreams that he is constantly being watched.
D. has a mistaken idea about how others view him.
A
Avg.
SS
NP
CT
B
%
Avg.
SS
269
14
‡
9
C
%
Avg.
SS
279
28
289
23
D*
%
Avg.
SS
%
265
11
304
46
‡
10
307
56
Grade 8 Example
On page 2, when the author of the biographical sketch says
that E. B. White's essays "appealed to an urbane crowd," he is
describing the audience for White's writing as
A. having sophisticated tastes.
B. leading busy lives.
C. being politically conservative.
D. having a good sense of humor.
A*
Avg.
SS
NP
CT
B
%
Avg.
SS
269
36
281
42
C
%
Avg.
SS
271
25
278
29
D
%
Avg.
SS
%
245
13
256
25
‡
12
266
17
The Origins of CCSS
CCSS
International
Assessments
NAEP
Discussion
•
•
•
•
•
What are the opportunities and challenges
related to the shifts?
How can districts support teachers in making
these important shifts?
Revisit your definition of rigor – has it changed? In
what ways?
What existing curriculum, assessment materials,
or activities will you examine for a range of
cognitive rigor?
What types of supports and resources can CSDE
provide?
Related Resources
Many papers and presentations available at
www.nciea.org or contact Karin Hess [email protected]

Hess, K. (2004). “Applying Webb’s Depth-of-Knowledge
(DOK) Levels in social studies and science” [online]:
http://www.nciea.org/publications/DOKsocialstudies_KH08.pdf
http://www.nciea.org/publications/DOKscience_KH11.pdf

Little Red Riding Hood handout
http://www.nciea.org/publications/Little%20Red%20Ridin
g%20Hood-CRM_KH11.pdf

Achievethecore.org—Student Achievement Partners
provides modules for professional development and
other valuable CCSS resources.
NAEP Questions Tool
Online access to 2000+ released NAEP
items in a variety of subjects over many
years
 Teachers use the NQT …

◦ to see models of well-designed items measuring the
same content taught in their classrooms;
◦ as a starting point when building common
assessments; and
◦ to reflect on the performance of the students in their
classrooms in comparison to students statewide and
across the nation.
http://nces.ed.gov/nationsreportcard/itmrls/
PISA Resources
PISA in Focus:
monthly policy-oriented notes
PISA Released Items from
2000, 2003 and 2006
PISA Assessment Framework
Please contact us to request
further information.
Renee Savoie
[email protected]
phone: 860-713-6858
Amy Radikas
[email protected]
phone: 860-713-6762