Transcript Slide 1

Adolescent Literacy Assessment:
Observing and Coaching Evidence-Based
Instruction in Discipline-Specific Classrooms
Spotlight on Assessment, Evaluation, and Research
Joshua Wilson
Yan Wei
Michael Faggella-Luby, PhD
3/26/2013
Neag School
of Education
Overview
I. Setting the Stage
II. Content & Pedagogy of Adolescent
Literacy
III. Research on Observational Assessments
IV. The Observation Tool
V. Suggestions and Lessons Learned
7/20/2015
Neag School
2
of Education
I. Setting the Stage
Neag School
of Education
Who is “at-risk”?
Students who are successful during their
9th grade year are 3.5 times more likely
to graduate
• One F decreases likelihood of graduating from
83% to 60%
• 2 Fs decreases likelihood to 44%
• 3 Fs decreases likelihood to 31%
Allensworth, E., & Easton, J. (2005). The on-track indicator as a predictor of high
school graduation. Chicago: Consortium of Chicago School Research
7/20/2015
Neag School
4
of Education
7/20/2015
Neag School
5
of Education
Here in CT
 “At least 72 percent of [students] attending
community colleges require remedial or
developmental math or English; for the
Connecticut state universities — Central,
Western, Southern and Eastern — the figure is
 … 65 percent.”
Megan, Kathleen (10/27/2010). “Many state high school graduates attending public
colleges unprepared.” Hartford Courant.
7/20/2015
Neag School
6
of Education
What impacts student outcomes?
PROPORTION OF VARIANCE IN STUDENT GAIN
SCORES-- READING, MATH (Gr.4,8,10)-- EXPLAINED
BY LEVEL--PROSPECTS STUDY
STUDENTS
28% R
CLASS
Slide From R. Elmore
SCHOOLS
60%
READING
12% R
52-72%
10-30% M
MATH
ROWAN, ET AL., “. . .PROSPECTS.
. .” TEACHERS COLLEGE
RECORD( 2005).
19% M
What do observation tools show
about typical practice?
or
How can we explain these outcomes?
Neag School
of Education
Schumaker et al., 2002
1. Lecture/read
2. Give directions
3. Listening
4. Ask question
5. Monitor
6. Model
7. Verbal rehearsal
8. Simple enhancer
9. Advance organizer
10. Role Play
11. Content Enhancement (complex)
12. Elaborated Feedback
13. Write on board
14. Describe skill/strategy
1
2
3
4
5
6
7
8
9
10
11
12
13
14
Schumaker et al., 2002
II. Content & Pedagogy of
Adolescent Literacy
Neag School
of Education
SRBI/RTI Framework
Tier 3:
5%
15%
Tier 1:
Comprehensive &
Coordinated
Instruction
for All Students
7/20/2015
Specialized,
Individualized
Intervention for Students
at High Risk
Tier 2:
Supplemental
Instruction for Students
at Some Risk
80% of
Students
Neag School
12
of Education
Tiered System of
Instruction
Alterable Components of Instruction
 Content
– Components of Reading and Writing
– Content-specific instructional methods
– Programs/Materials
 Pedagogy
–
–
–
–
–
7/20/2015
Explicitness
Intensity and Pacing
Interactiveness
Grouping
Interventionist Expertise
Neag School
13
of Education
Critical Content:
What to Teach
1. Background
Knowledge
2. Knowledge of Text
Structure
3. Cognitive Strategies
4. Motivation
5. Word Study
6. Writing
References
Biancarosa & Snow, 2004
Ehren, 2005
Faggella-Luby & Deshler, 2008
Gersten et al., 2001
Graham & Hebert, 2010
McCabe, 2009
Torgesen et al., 2007
Neag School
of Education
1. Background Knowledge
•
World
Concepts
Vocabulary
•
Example: Social Studies Classroom
Vocabulary
– Democracy, communism,
filibuster, gridlock
– Academic or Signal Language
– Passage-specific vocabulary
Concepts
– Representative government vs.
communism
– Free-market economy vs,
socialism
• World Knowledge
– Travel
– Experience with an adult voting
– Reading/Listening/Viewing
political advertisements or
debates
CCSS: R1-R3; L4-L6
Neag School
of Education
2. Knowledge of Text/Discourse
Structures
 Narrative/Expository
organizational
patterns
 Student awareness
 Strategic use
Examples
• Narrative Text Structure
– Sequential
– In medias res
• Expository Text Structure (Essay)
– Persuasive
– Comparison-Contrast
– Description
– Cause-Effect
– Sequential
– Problem-Solution
– Lab Report
– Journal Article
– Policy
CCSS: R4-R6
Neag School
of Education
3. Cognitive Strategies
 Goal-specific
 Packaging
 Monitoring and
repair
Examples
• Goal-Specific Strategies
–
–
–
–
questioning
summarization
prediction
inferencing
• Packages of Strategies
– peer assisted learning (e.g.,
CWPT)
– Reciprocal Teaching
• Self-Monitoring/Repair
– Collaborative Strategic
Reading
CCSS: R7-9; R10
Neag School
of Education
4. Motivation
Examples
 Engagement
 Self-efficacy
 Task persistence
• Engagement
– Provide rationale for learning/academic tasks
– Co-Construct meaning
– Use first-glance influence
• Self-efficacy
– Students set goals, develop a plan, self-monitor,
and reflect
– Timely teacher feedback
– Engineer success
• Task Persistence
– Balance of challenging and independent material
– Student choice of reading material
CCSS: R10
Neag School
of Education
5. Word Study
 Multi-syllabic
words
 Difficult often
content specific
syntax
 Unique semantic
constructions
 Fluency
•
•
•
•
Examples
Multi-Syllabic Words
 Onomatopoeia
 Stoichiometry
 Pythagoras/Pythagorean
Difficult Often Content Specific Syntax
• Word problem in math
• Elizabethan English
• Legislative language
Unique Semantic Constructions
• ‘Republican’ in Federalist papers vs.
present day
• Use semantic map line labels
• Pre-teach key terms and
explanations
Fluency
• Reading Shakespeare for prosody
CCSS: L4, RF3-RF4
Neag School
of Education
6. Writing
Examples
 Writing Fluency skills • Writing Fluency Skills
– Teach spelling and word-study skills
•
Spelling
together
•
Sentence construction – Use text models to illustrate
sentence fluency and word choice
•
Word choice
• Writing Process
 Writing Process
 Strategies to
compose genrespecific text
– Planning and Revision
– Summarization
• Genre-specific Composition
– Link strategies for identifying and
utilizing text-structure with
composition strategies
CCSS: L1-L2; W4-W6; W1-W3
Neag School
of Education
K.L.P.s - Content
(key learning points)
 What do I think is most important about what
we’ve covered?
 How can I use this information?
7/20/2015
Neag School
21
of Education
How Teachers Teach
Struggling Readers
A significant difference between
struggling readers and their typically
achieving peers is not what they are
taught about literacy, but how they
are taught!
7/20/2015
Neag School
22
of Education
Pedagogy That
Builds Literacy Skills
• Instructional Core
– Daily review
– Introduce lesson
objectives
– Present new content
– Guided practice
– Independent Practice
– Evaluation
• Formative
• Summative
• Priority Instruction
– Small groups*
– Strategy Cueing and
Questioning
– Advance Organizer
– Distributed Practice
• Corrective and
elaborative feedback
References: Faggella-Luby & Deshler, 2008;
Rosenshine,1995; Swanson, 1999; Swanson &
Hoskyn, 2001
Neag School
of Education
Instructional Core – Teach
Intentionally
 Introduce Lesson Objectives
• Explain how the strategy will improve learning
• Clarify expectations for learning and independent use
 Present New Content
• Share important strategy components or steps
• Share the sequence of strategy steps as necessary
 Guided Practice
• Present a verbal think-aloud of how to use strategy
• Identify obstacles and explain how to overcome them
• Complete at least one successful application of the strategy
7/20/2015
Neag School
24
of Education
Instructional Core – Teach
Intentionally (cont.)
 Independent Practice
• Prompt student involvement to get them actively thinking about the
strategy
• Practice with appropriate examples to allow for successful student
practice
• Create opportunities for teacher questioning and student selfquestioning
 Evaluation
• Formative: determine specific instructional needs prior to instruction,
frequently check for understanding
• Summative: design summative assessments to assess student mastery
of learning objectives
7/20/2015
Neag School
25
of Education
Gradual Release and
Intensity
7/20/2015
Neag School
26
of Education
K.L.P.s - Pedagogy
(key learning points)
 What do I think is most important about what
we’ve covered?
 How can I use this information?
7/20/2015
Neag School
27
of Education
III. Research on Observational
Assessments
Neag School
of Education
Recent Research on Observational
Tools in Secondary Classrooms
An electronic search was conducted for studies published
between 1990 to 2012 from the ERIC, PsycINFO and PQDT. To be
included in this summary, a study was required to meet the
following criteria:
 An observation study of literacy instruction in ELA classes or in
discipline-specific classrooms.
 Pertaining to secondary level (9th-12th grade).
 Conducting planning and/or observational tools in tier I
instruction.
 Designed to observe teaching behaviors and/or teaching
instructions in the study.
7/20/2015
Neag School
29
of Education
Recent Research on Observational
Tools in Secondary Classrooms
Summarized research from the following elements:
 Audience
 Person observed
 Intended purpose: improve instruction; motivate teachers;
make principals accessible to students and teachers;
 Observation categories
 Method for use: length of observation, tools to observe,
recommended use
 Observation tools
7/20/2015
Neag School
30
of Education
Observation Tools for Teaching Behaviors in
Secondary Classrooms
Observational
Tool
Audience
Person
Observed
Intended
Purpose
Cornett, J. (2010)
Two
Trained
observers
T and S
(special
educator &
General
educator)
• To understand
the typical
instruction in
high school
classrooms
involving coteaching for
students with
disabilities.
Classroom
Observation
Sheet
7/20/2015
Observation
Categories
Four foci:
•Student
engagement
•Learning
arrangement
•Transition time
•Instructional
activity
Method for Use
• Length: 60
minutes
• Tool: data
collection sheet,
pen, digital
countdown clock
• Recommended
Use:
Using 30-second
Momentary Time
Sampling (MTS) to
code the sheet.
Neag School
31
of Education
Classroom Observation
Sheet
7/20/2015
Neag School
32
of Education
Observation Tools for Teaching Behaviors in
Secondary Classrooms
Observational
Tool
Rissman,
Miller, &
Torgesen,
(2009).
Adolescent
Literacy WalkThrough for
Principals
(ALWP
template)
7/20/2015
Audience
Perso
n
Obser
ved
Intended Purpose
Observation
Categories
Method for Use
Principals &
Administrators
T and
S
• To help principals
monitor and support
adolescent literacy
instruction and
student achievement
in their schools more
effectively.
• To provide guidelines
for improving schoolwide reading
instruction.
• Coaching focus
instead of judging
focus.
• To recognize
effective academic
literacy instruction.
Research-based academic
literacy instruction in
grades
6–12 classrooms.
Content area and
Reading intervention:
•Vocabulary/concept
•Background knowledge
•Comprehension strategy
instruction
•Discussion of reading
content
•Motivation
• Length: 3-5 minutes
• Tool: Template,
pen/pencil
• Recommended use:
The principal visits
classrooms
frequently to look at
instructional
practices and
student learning with
the tool.
The principal shares
feedback with
individual teachers
and results are also
used to plan
professional
development.
Neag School
33
of Education
ALWP Template
7/20/2015
Neag School
34
of Education
Observation Tools for Teaching Behaviors in
Secondary Classroom
Observational
Tool
Vogt, M.E.
(1991).
Checklist for
Integrated
Reading/Langua
ge Arts
Instruction
7/20/2015
Audience
Pers
on
Obse
rved
Intended Purpose
Observation
Categories
Method for Use
Supervisors
Administrators
T
• To provide a
checklist for
supervisors and
administrators to
observe reading and
writing lessons.
• Guideline for
developing an
integrated
reading/language
arts program.
Checklist includes:
• Reading
• Writing
• Listening
• Speaking
• General
• Length: not
specified
• Tools: checklist
• Recommended
use:
Make preobservation and
follow-up
conferences
between teachers,
principals and
supervisors to
facilitate the
professional
growth of
teachers.
Neag School
35
of Education
Checklist for Integrated
Reading/Language Arts Instruction
7/20/2015
Neag School
36
of Education
Observation Tools for Teaching Behaviors in
Secondary Classroom
Observational
Tool
William et al.,
(2004)
Writing
Observation
Framework (WOF)
7/20/2015
Audience
Person
Observed
Intended Purpose
Observation
Categories
Method for Use
Principals/
Supervisors/
Peers
T
• To monitor and improve
writing instruction.
• To ensure fair and
substantive evaluation
of teaching instruction
in writing.
• Can be used for peermentoring.
• Applicable for in-depth
self-evaluation with
instructional practices in
writing.
• Classroom
climate
• Prewriting
• Drafting
• Conferencing
• Revising
• Editing
• Skill/strategy
instruction
• Assessment
• Student practice.
• Length: whole
period
• Tools: WOF
observational tool
• Recommended
use:
Coding O, C, R, N for
each items: O
(Observed), C
(Commendation), R
(Recommendation),
N (Not applicable)
Make preobservation and
follow-up
conferences.
Provide feedback to
individual teachers.
Neag School
37
of Education
The Writing Observation
Framework
7/20/2015
Neag School
38
of Education
In Summary
Reasons to share:
 Observe literacy practices in four instructional
settings:
• co-teaching classroom, discipline-specific classroom,
reading/language arts program, and writing classroom.
 Address the components of evidence-based literacy
instruction in tier I settings, including content of
instruction and teaching pedagogy.
 Demonstrate different types of observational formats
(checklists, questions, reflections).
7/20/2015
Neag School
39
of Education
Strengths and
Limitations All Together
Strengths:
 Increase communication and collaboration among teachers, peers , and
administrators through providing frequent and timely feedback at
pre/post-observational conferences.
 Teachers gaining new ideas and perspectives about teaching from
colleagues.
 Provide guidelines for developing school-wide literacy instruction(e.g.,
Vogt, 1991).
Limitations:
 Possible bias relating to the observer's own beliefs about teaching.
 Many elements to observe at once.
 Limited observer reliability, variable observation times (short to long).
 No studies embedding CCSS into instructional observation.
7/20/2015
Neag School
40
of Education
IV. The Observation Tool
Putting Adolescent Literacy Content
& Pedagogy Together
Neag School
of Education
The Observational Tool
 Audience: school administrators, researchers,
teachers, and/or peers
 Person observed: a content-area teacher
 Settings: secondary classroom in tier I instruction
 The intended purpose: coaching, planning, and
professional feedback
 Length of the observation: not specified
7/20/2015
Neag School
42
of Education
The Tool Page 1
The Tool Page 2
Implementing the Tool
 Before:
• Review the protocol outlines and clarify operational definitions of each items
before observation.
• Clearly orient the observational goals, share lesson plan, and confirm the
observation and meeting time with the teacher.
 During:
• Observer visits the classroom for a specified length of time (longer and more
frequent are best).
• Sit in a location to minimize student distraction.
• Complete Observation Tool by recording Observed Data & Observer
Comments
• Give to teacher at end to complete Teacher Reflection prior to follow-up
7/20/2015
Neag School
45
of Education
Implementing the Tool
 After:
• Teacher completes reflection
• Conduct a follow-up discussion between teacher and
observer after observation
• Share summary, feedback on positive comments, and
areas of concern.
• Based on reflections, co-construct action-oriented goals
and future action plans jointly.
7/20/2015
Neag School
46
of Education
“We can have 30 years of research on
instruction, but if it doesn’t get
implemented by teachers, it’s not going
to make a big difference.
What we want to know is what’s the
most efficient way to translate those
ideas into practice in the classroom.”
–Jim Knight, KUCRL
Coaching &
Professional Development
The Observation Tool measures the behavior,
but how do we change the behavior?
 Use Tool to identify key areas of professional
development
 Less top down and more practice driven
 Use Partnership Learning approach to PD
7/20/2015
Neag School
48
of Education
Teacher Planning
Questions - Content
 Background Knowledge
• Have I determined what relevant prior knowledge is necessary for
understanding the academic task?
• Have I analyzed the academic tasks for any vocabulary that may be
unfamiliar to my students?
 Text Structure
• Have I selected readings that represent clear examples of a variety of
text structures?
 Cognitive Strategies
• Have I determined what strategies my students need to help them
accomplish the academic task?
• Have I selected a small number of powerful strategies for students to
learn to mastery levels?
7/20/2015
Neag School
49
of Education
Teacher Planning
Questions - Content
 Motivation
• Am I prepared to answer the ‘So What?’ question?
• What types of in-class support will I provide to promote student reading persistence?
 Word Study
• Does my lesson focus on common and/or difficult root words and affixes?
• Have I chosen texts for fluency practice that support content area learning?
 Writing
• How can my instruction emphasize specific writing skills during reading tasks?
• How can I use text models to instruct genre-specific composing?
7/20/2015
Neag School
50
of Education
Teacher Planning
Questions - Pedagogy
 Will I begin my lesson with a short review of previous learning?
 What grouping strategies will my lesson include?
 How can I break down new skills into small steps to facilitate
student learning?
 What are the critical behaviors and thinking skills that should
be explicitly modeled?
 Have I provided ample opportunities to practice new skills in
multiple modes of response?
 Have I selected assessment methods and clear criteria for
triggering instructional adjustments?
7/20/2015
Neag School
51
of Education
V. Suggestions and Lessons
Learned
Neag School
of Education
Practical Suggestions:
Observing
 Prioritize checklist items at a time—don’t look for too much!
 Plan to visit several times—don’t use a single data point!
 Plan to spend longer time at a class- don’t stop by and move forward
immediately
 Operationalize definitions of terms—don’t make anyone guess!
 Provide frequent and timely feedback—don’t make them wait too
long!
 Practice! Practice! Practice!
7/20/2015
Neag School
53
of Education
Practical Suggestions:
Teaching
 Plan to use the tool as a guide—sample from the
tool categories as needed!
 Embrace a cooperative process of professional
improvement—grow with feedback and reflection!
 Remember this tool can empower—improved
practice is more than evaluation!
 Acknowledge positives—grow with observed
challenges!
7/20/2015
Neag School
54
of Education
More Suggestions






Consider transition time
Target the lesson objectives
Notice teacher/student engagement in instruction
Verbal praise should specifically refer to the task
Pay attention to time allocation, especially reviewing.
Address the assessment to measure lesson
objectives
7/20/2015
Neag School
55
of Education
In Summary
Clarify the lesson objectives
Increase the frequency
Extend the observation time
Provide timely
feedback
after
observation
Deepen the observational tools
7/20/2015
Neag School
56
of Education
Questions?
7/20/2015
Neag School
57
of Education
Thank You!
For Follow-Up or More Information:
[email protected]
[email protected] [email protected]
7/20/2015
Neag School
58
of Education