Transcript Issues in Evaluation

The FIE = Data-based Problem
Solving
Texas Educational Diagnostician
Association (TEDA)
Annual Conference
April 8, 2011
PIEMS October 2010 Snapshot
Primary Disability
Category
Number
Primary Disability
Category
Number
AI
7,569
MR
30, 490
AU
29,536
NEC
3,776
DB
128
OHI
54,436
DD
446
OI
4,287
ED
28,939
SI
88,958
LD
187,296
TBI
1,336
MD
5,576
VI
3,819
TOTAL = 446,592
Diagnostic Conclusions
• The determination of a disability category,
(especially MR and LD) must be made through the
use of professional judgment, including
consideration of multiple information/data
sources to support the eligibility determination.
(TEA, §89.1040 Eligibility Criteria, Frequently
Asked Questions)
• Information/data sources = statewide assessment
results, formal evaluation test scores, RtI progress
monitoring data, informal data (e.g., work
samples, interviews, rating scales), anecdotal
reports
Data-based Problem Solving
• In order to improve the diagnostic process, need to
approach diagnosis and the FIE as
• Problem Identification & Analysis – process of
assessment and evaluation for identifying and
understanding the causal and maintaining
variables of the problem
Problem-Solving:
Identification and Analysis
• Problem Identification – define referral problem in
specific, operational terms; only then can problem can
be objectively assessed; think of problem as
discordance between expected/desired levels of
performance and current performance
• Problem Analysis – why is the problem occurring;
functional assessment of academic and/or behavioral
problems; problem analysis is the link between a wellspecified problem and a problem solution
Data-Based Problem-Solving: Analysis
• In Analysis stage –
– Gather Data and then form hypotheses
– Hypotheses must be systematically tested, rejected or
verified not just formulated
– Systematic hypothesis testing must use low inference
approaches (high inference approaches usually rely on
theoretical within-person constructs that require
assumptions that can’t be verified; e.g., observe a
student and say he has low self-esteem)
– Use analytic versus standard battery approach
Clinical Judgment
• A special type of judgment rooted in a high
level of clinical expertise and experience –
emerges directly from extensive data;
based on clinician’s training, experience, and
knowledge of person and environment
• Characterized by being:
– Systematic (organized, sequential, logical)
– Formal (explicit and reasoned)
– Transparent (apparent and communicated
clearly)
4 Clinical Judgment Strategies
•
•
•
•
Clarify and state precisely the question set
Conduct/gather a thorough history
Perform assessments
Synthesize the data
Thinking Errors to Avoid
• Affective error – your “feelings”
– e.g., incorrect stereotypes, misplaced empathy,
or what you wish were true
• Anchoring Error – first bit of information
“anchors” your mind on an incorrect decision
• Availability – happened recently or most
dramatically
• Blind obedience - what authority said
Thinking Errors to Avoid
• Commission bias – Do something, anything
• Confirmation bias – find what you expect
• Diagnosis momentum
– Piling on after an initial diagnosis
• Framing effects Mistakenly influenced by context
• Premature closure – deciding too soon
• Representativeness error – what is typically
true
John O. Willis, 2010
• Complex and difficult problems require carefully
planned, very thorough, goal-directed comprehensive
evaluations that will guide the examinees, parents,
teachers, and other professionals in fully understanding
the student’s strengths and needs as a basis for
planning interventions that will ameliorate the
problems and lead to improved functioning
• p. xxi, Forward to Mather & Jaffe (Eds.) (2010)
Comprehensive Evaluations: Case Reports for
Psychologists, Diagnosticians, and Special Educators
Time Consuming Process
• P-Time: Planning Time – once you have reviewed
records need to plan assessment
• E-Time: Administration and Scoring of instruments,
including conducting interviews, observations and
other informal evaluation activities
• T-Time: Thinking Time – you must build this in to your
evaluation
• I-Time: Interpretation/Analysis/Synthesis of Data
• T-Time: again! Prove hypotheses
• R-Time: Report-writing time
Determining the “Appropriateness” of the FIE (Etscheidt,
2003)
• “Adequacy” - Compliance with IDEA evaluation
requirements
– Use a variety of assessment tools
– Use technically sound instruments
– Utilize non-discriminatory procedures
– Administer standardized tests validated for
the specific purpose for which they are used
– Administered by trained and knowledgeable
personnel in accordance with instructions
provided by the test producer
Determining the “Appropriateness” of the FIE (Etscheidt,
2003)
• “Scope” of the evaluation
– Must be assessed in all areas of suspected
disability
– Should not use a limited selection of assessment
tools
– If there are reasons to suspect additional areas
requiring assessment, then the scope of the
evaluation must be broadened to include
assessment in those areas
– Must be comprehensive enough to address
related services as well
Determining the “Appropriateness” of the FIE
• “Utility” - Provision of sufficient information to develop
the IEP
– Must determine/identify the educational needs of the
child
– Must discuss the impact of the child’s disability on
educational performance
– Must lead to the development of goals and objectives
• Reference: Etscheidt, S. (2003). Ascertaining the
Adequacy, Scope and Utility of District Evaluations.
Exceptional Children, 69(2), 227-247.
Do’s and Do not’s
Do
Don’t
Have a MDT
Be the only evaluator
Follow the RIOT and ICEL Model; Be
thorough
Refer to “audit file” vs. conducting
review; conduct a standard battery
only approach
Select most appropriate methods
Select instruments based on
and instruments; Keep up with
convenience; use outdated
current literature and “best practice” instruments; rely on information or
procedures that are not current
Administer procedures in
standardized manner; Explain any
violations
Provide scores for tests not designed
for the age of the student
Score correctly; Write all responses
verbatim
Make errors; just put score versus
what the student answered/did
Do’s and Do not’s
Do
Don’t
Verify hypotheses and validate
Leave “danglers” or just form
hypotheses without proving them
Write about student and not test;
give examples of student’s
performance
Write about the test more than about
the student
Link conclusions and
recommendations to data; make
research-based recommendations
Generate a conclusion without an
adequate explanation; Make generic
recommendations that are too
broad/could apply to all students
Proof FIE report for errors; Make
sure report is finalized prior to
giving it to parents
Give out “drafts”
Keep the data from which you
based decisions
Destroy test protocols
Anatomy of a Paragraph
Identify cluster or index
The Basic Reading Skills cluster measures single
word decoding skills.
Describe task
On the LW ID test the student reads from a list of
real words; on the WA test the student is
presented with a list of nonsense words to read.
Explain student
Jenny performed poorly on both tasks. She used a
performance with examples whole word approach to decode unfamiliar words
(e.g., sentence=certainty; moustache=muscle), and
when attempting to apply phonics to break down
the word, could not do so correctly (e.g., bounties=
bount teece). On nonsense words, Jenny had
difficulty with short vowel sounds (e.g., nan=nane).
This pattern indicates that Jenny has not acquired
phonemic awareness skills consistent with age
expectations. …
Typical Order for FIE
• Reason for Referral
– Description of why the student in being referred
for the FIE. Usually ends with a standard
statement such as:
• The purpose of this FIE is to (a) determine the presence
or absence of a disability condition(s), (b) identify and
describe the student’s strengths/assets and
weaknesses/needs, and (c) provide data that will assist
the ARD committee in its deliberations regarding
educational programming.
Typical Order for FIE
• Sociological including Educational History/
Educational Progression
• Review of Previous Evaluations (if applicable)
• Review of ARD or 504 documents including
previous interventions and services
At this point in the report, the chronology and background of student is clear
and narrative takes you to current status. Also you have identified any
cultural or experiential factors which may have an impact on decision
making.
Typical Order for FIE
• Speech/Language/Communication (Articulation,
Voice, Fluency, Intelligibility, Single-words to Language Use
to higher level Language skills including pragmatics if
applicable; Communication Samples)
• Physical/Developmental/Motor (Developmental
Progression, Medical History, Vision, Hearing, Sensory if
applicable, Medications, Motor skills - fine and gross; OT, PT,
Audiological, etc. would go here if applicable)
At this point in the report, we have data regarding language/communication,
developmental, and medical status. Depending on the student, Physical could
come before Language, especially if student has a particular condition that will
impact your selection of instruments and interpretation of the actual test results.
Typical Order for FIE
• Intellectual/Cognitive (thorough evaluation of
cognitive processes related to referral questions – e.g., if
considering MR/ID focus on broad or general processes; for
LD, careful selection of narrow abilities under these broad
areas; developmental assessment would be here if
applicable; also other processes not only on IQ measures
need to be considered – e.g., phonological processing,
executive function, attention, memory)
• Adaptive Behavior (can be informal or formal; need
formal measures for MR/ID and AU; consider issues of
informants and rating scales versus interview methods)
Sample intro statements for IQ &AB
• The following measures assess cognitive processing. IQ does
not equate with innate or intellectual capacity but rather
evaluates the various processes by which students learn. [If
all processes are intact, learning should be intact.
Conversely, if certain processes are deficient then there will
be an impact (constraint) on learning.]
• Adaptive behavior refers to the degree to which the
student is personally and socially self-sufficient. This level of
independence is based on age expectations and is
performance and not only ability based (e.g., a student
may possess the ability to perform a skill but does not do
so at an independent level, thus self-sufficiency is adversely
affected).
What is cognitive processing?
• Cognitive Processes – higher level brain
processes necessary for learning; processes
involved in perception, thinking, reasoning,
problem solving, learning, storing and
retrieving information (Dehn, 2005)
• LD model – postulates that LDs are associated
with cognitive processing deficits that
significantly impair learning
Processing – Disability & Intervention
• Processing strengths and weaknesses
associated with diagnosing LD, but
• Also important and associated with other
disabilities
• If learning depends on processing, then having
information about a student’s strengths and
weaknesses can assist in design of
interventions, selection of accommodations,
and determination of modifications
Main Processing Components to
Assess
Attention processing
Processing speed
Auditory processing
Short-term memory
Executive processing
Simultaneous processing
Fluid reasoning
Successive processing
Long-term retrieval
Visual Processing
Phonemic awareness
Working Memory
Planning
Typical Order for FIE
• Emotional/Behavioral/Social [discipline history,
attention and task orientation, social relationships with
peers and teachers, relating behavioral characteristics to
developmental level, etc.; should include subheading for
FBA and any specialized assessments if applicable (e.g.,
social skills)]
If no significant emotional/behavioral issues, then this section is usually
before IQ or Achievement. If there are significant emotional/behavioral
difficulties, then this section usually follows IQ and Ach.
Typical Order for FIE
• Academic Achievement/Educational
Performance Levels [4 types of data: Informal (e.g.,
interviews, grades), Criterion-referenced (e.g., benchmarks,
TAKS), Curriculum-based (e.g., universal screening, RtI
progress monitoring), Norm-referenced (standardized
measures of academic achievement); important here is
comparison of student to peers (e.g., rank) and expected
criterion (e.g., DRA level should be 28, student is 10)]
Depending on nature of referral and how you want data presented, the
Academic section can come before the IQ section.
Typical Order for FIE
• Assistive Technology [must be “required”, thus must
assess for this (e.g., writing sample with and without use of
computer)]
AT needs should be discussed given physical, cognitive, linguistic, and
academic functioning levels; on most reports the AT section is at the end
Vocational/Transition [when applicable, prevocational
and vocational skills should be addressed; functional
vocational evaluation would go here if applicable;
measurement of interests and aptitudes; should also conduct
skills analysis (e.g., cognitive, academic, behavioral,
etc.)with transition goals and needs)]
Typical Order for FIE
At this point in the FIE, we now have all the information to make decisions
regarding diagnosis/classification and educational needs
• Conclusions
• Recommendations (all needs and related services must
be addressed, e.g., AU supplement)
– Lead-in statement (next slide)
– Usually have categories for this section such as Educational
Programming, Academic Skills, Behavior, Social, Transition,
etc.)
• Assurances and MDT Signatures
Recommendations: Lead Statement
• This multidisciplinary evaluation is considered a
valid representation of ____’s current levels of
functioning in the areas assessed. The purpose of
the FIE is to identify the presence or absence of a
disability condition and provide information
regarding the student’s needs within the
educational setting. All determinations pertaining
to eligibility, educational planning/programming,
services to be provided, and placements remain
the responsibility of the ARD committee.
XBA is Commonplace – Acknowledge the
Procedure in Your Report
• The results presented in this report were compiled from
tests that do not share a common norm group; however,
test results have been interpreted following the crossbattery approach and integrated with data from other
sources including educational records, parent/teacher
interviews, behavioral observations, work samples, and
other test findings to ensure ecological validity.
Standardization was followed for all test administrations.
No single test or procedure was used as the sole criterion
for classification, eligibility or educational planning. Unless
otherwise noted, the results of this evaluation are
considered a reliable and valid estimate of [Student’s
Name] demonstrated skills and abilities at this time.
Adapted from D. Miller (2010)
References
• Essentials of Psychological Assessment Series. Hoboken, New
Jersey: John Wiley & Sons.
– Dehn, M.J. (2006). Essentials of Processing Assessment.
– Flanagan, D.P. & Alfonso, V.C. (2011) Essentials of Specific
Learning Disability Identification
– Flanagan, D.P., Ortiz, S.O., & Alfonso, V.C. (2007). Essentials of
Cross-Battery Assessment, 2nd Ed.
• Intellectual Disability: Definition, Classification, and Systems of Supports.
11th Edition of the AAIDD Definition Manual. (2010). Washington, DC:
American Association on Intellectual and Developmental Disabilities.
– Chapter 8 – Role of Clinical Judgment in Diagnosis, Classification, and
Development of Systems Supports
• Other references noted on slides