Transcript Universal Screening for Social

Universal Screening
for Social-Emotional and
Behavioral Difficulties
Delaware PBS Project
April 23, 2008
Kathleen Minke
Workshop Overview



Background/Rationale for universal
screening
Methods for your consideration
Linking identified students to
services
What Is Universal Screening?
 Purpose:
to identify youth who have
high risk for developing behavioral or
mental health problems
 Conducted on a schoolwide basis
 Typically involves several levels of
assessment to avoid over- or underidentification of students
Universal Screening for What?

Behavior problems?
 Social-emotional

problems?
Mental illness? Mental health?
Why Should Schools Be
Concerned With This?
 Fits

with the three-tiered model
Fits with RTI
Why Should Schools Be
Concerned With This?
 Children
with better social-emotional
and behavioral skills do better
academically
 Children’s academic and social
competencies influence each other
– When you improve children’s social
competence you also make it more likely
that they will improve academically
 Students
are more likely to use
services that are offered at school
Do Students Need Support?
2005: Around 2.2 million adolescents (1217) reported a major depressive episode;
nearly 60% received no treatment
 Students with significant social-emotional
and behavioral needs drop out at a rate
twice that of other students
 In a given year, around 20% of children
and adolescents experience symptoms of
mental health problems

Facts about Delaware’s Youth
 16.6%
carried a weapon in the prior
month; 5.4% carried a gun
 4.6% did not go to school because
they felt unsafe
 6.2% were threatened or injured on
school grounds in prior 12 months
 7.1% attempted suicide; 12.7%
seriously considered suicide
The Good News…
Prevention Works
Greenberg et al. (2003):
 Programs that developed students’ skills
and promoted positive school climates
demonstrated improved interpersonal
relationships, improved academic
achievement, and reductions in problem
behaviors (e.g., truancy, substance use,
violence) among students (Catalono et al, 2002)
 Primary prevention programs in school
settings can enhance interpersonal
competencies and prevent externalizing
and internalizing problems (Durlak & Wells,
1997)
“There is a solid and growing
empirical base indicating that welldesigned, well-implemented, schoolbased prevention and youth
development programming can
positively influence a diverse array
of social, health, and academic
outcomes.”
Greenberg et al. (2003)
Is My School Ready to Do This?
The school faculty and staff are
committed to improving social-emotional
and behavioral health among students.
The school and district are committed to
“following the data.”
Faculty understand and accept that
current referral and support structures in
the school may need to change.
The community supports screening for
social-emotional and behavioral concerns.
Is My School Ready to Do This?
The schoolwide program is working well
for 80-85% of students.
The targeted program is working well and
the problem-solving process at this level is
not overwhelmed.
 In-school resources are available to
provide interventions.
Effective connections have been
established with community support
services.
Academic and Behavioral Systems Work Together
Intensive Interventions
•Individual Students
•Assessment-based, function based
•Intense, durable procedures
Targeted Interventions
•Some students (at-risk)
•Function-based
•Skill development and/or
relationship development
•Individual or group
•High efficiency
•Rapid response
School-wide Interventions
•All settings, all students
•Preventive, proactive
All interventions are based on
function and progress monitoring
data
5-10%
10-15%
•Intensive academic support
•School based adult mentors
•Intensive social skills training
•Individualized function based behavior
support plans
•Parent training and collaboration
•Multi-agency collaboration (wrap around)
•Alternatives to suspension and expulsion
•More detailed social skills training and
support
•Self-management programs
•School-based adult mentors (check-in)
•Increased academic support & practice
•Alternatives to school suspension
•Effective academic support
•Teaching social skills
• Character development
•Teaching school-wide expectations
•Active supervision in common areas
•Positive reinforcement (tangible and
intangible) for all
•Firm fair, corrective discipline
•Family-school collaboration efforts
•Effective classroom management
Multiple Gating Procedures

Gate 1: cast a wide net
– Teacher report or nomination
– Brief parent report
– Brief self-report
Remember that measurement
class?
True Positives
False Positives
Identified as at risk
who actually are
at risk
False Negatives
Identified as at risk
but who are
not at risk
True Negatives
Identified as not at
risk but who are
actually at risk
Identified as not at
risk who are actually
not at risk
Remember that measurement
class?

Sensitivity:
– Of those actually at risk, what
proportion is identified?

Specificity:
– Of those actually not at risk, what
proportion is identified?

Positive predictive value
– Of those identified as at risk, what
proportion is correctly identified?
Multiple Gating Procedures

Gate 1: cast a wide net
– Teacher report or nomination
– Brief parent report
– Brief self-report

Gate 2: refine the “catch”
– Ratings and/or rankings
Multiple Gating Procedures

Gate 1: cast a wide net
– Teacher report or nomination
– Brief parent report
– Brief self-report

Gate 2: refine the “catch”
– Ratings and/or rankings

Gate 3: identify those most at risk
– Observations, interviews, other assessments
Then What?

Individualized planning

Intervention

Progress monitoring

Fading of intervention
Universal = All?

All kindergartners

All transfers into school

All students during transition year to
middle school or high school
When to Screen?



If using parent report, at beginning
of school year is ok
If using teacher report, wait at least
two months
Self-report?
What about Informed Consent?

Passive vs. active consent

The trade-off
What About Informed Consent?
Hatch Amendment (1974)









No student shall be required, as part of any applicable program,
to submit to a survey, analysis, or evaluation that reveals
information concerning—
(1) political affiliations;
(2) mental and psychological problems potentially embarrassing
to the student or his family;
(3) sex behavior and attitudes;
(4) illegal, anti-social, self-incriminating and demeaning behavior;
(5) critical appraisals of other individuals with whom respondents
have close family relationships;
(6) legally recognized privileged or analogous relationships, such
as those of lawyers, physicians, and ministers; or
(7) income (other than that required by law to determine
eligibility for participation in a program or for receiving financial
assistance under such program),
without the prior consent of the student (if the student is an adult
or emancipated minor), or in the case of an unemancipated minor,
without the prior written consent of the parent.
What About Informed Consent?
 Broad
interpretations of the Hatch
Amendment (1974); parents seek to
restrict activities related to:
– Values clarification; moral dilemmas
– Religious and moral standards
– Death education; Sex education
– Drug and alcohol use
– Nuclear policy and globalism
– Witchcraft, occult, supernatural
– Evolution
What About Informed Consent?

“Psychological and psychiatric
treatment that is designed to affect
behavioral, emotional, or attitudinal
characteristics of an individual or
designed to elicit information about
attitudes, habits, traits, opinions,
beliefs or feelings of an individual or
group.”
What about Informed Consent?

Passive consent probably ok for teacher
nominations or rankings at Gate 1 and 2;
obtain active consent for Gate 3

Almost always obtain active consent when
student or parent report is used

Get consent and screen when parents are
physically present at school

Don’t forget the lawyers….
Assessing Technical Adequacy

Just because it is published…

School or district level evaluation…
Except for a few statistics,
everything I learned in graduate
school turned out to be wrong.
- Jack Bardon
Statistics
 Please
remember to
answer in the form
of a question…or
whatever….
 And,
as always, no
wagering.
Variance is a measure of
a. the amount of spread in a
set of scores.
b. the most frequently occurring
score.
c. the average score.
d. the difficulty level of scores.
The statistic that tells you the
strength and direction of the
relationship between two
variables.
True or false:
If you know that there is an r of
.85 between measures of these
two variables, you can state
with confidence that parent
involvement in schooling results
in students’ high grades.
Which of the following is a more
impressive result in a study?
A. “the correlation between the
variables was significant at the
.0001 level.”
B. “An effect size of 1.2 was
observed in favor of the
experimental group.”
A self-concept scale was
normed in 1968 on 200 children
who lived near a university in
Colorado. Name at least 3
problems with using this
measure in Delaware.
A T score has a mean of _____
and a SD of _____ .
a. 50; 10
b. 50; 50
c. 100; 15
d. 0; 2
Cronbach’s coefficient alpha is a
measure of
a. construct validity.
b. internal consistency
reliability.
c. discriminant validity.
d. test-retest reliability.
True or False: A low test-rest
reliability coefficient always
means that the test is poorly
constructed.
__________ validity is the
extent to which the items in a
measure represent the domain
being assessed.
A. Construct
B. Face
C. Content
D. Concurrent
Technical Adequacy Summary








Representative norms
Internal consistency reliability
Test-retest reliability
Inter-rater reliability
Predictive validity
Concurrent validity
Construct validity
Content validity
Additional Factors to Consider





Acceptability
Appropriateness
Feasibility
Adaptability
Utility

See Glover and Albers (2007)
Utility


Are the outcomes useful for guiding
intervention?
Does this process improve student
outcomes?
Cautions

Build support for the program
– Provide clearly written, family-friendly
information that outlines the benefits of
screening
– Provide prompt answers and additional
information to any parent expressing
concern
– Provide information about screening
outcomes (and effectiveness of
programming)
Cautions
 Gain
student assent
– Even when parents have provided
permission, students must have the
freedom to opt out without penalty
– Provide student-friendly information
about the screening
– Encourage family decision-making
Cautions
 Protect
student and family privacy
– Educate teachers about confidentiality
– Guard against labeling
 Screening
must lead to effective
intervention!
Some Examples

Published methods
– SSBD
– BASC-2/BESS

Build-Your-Own-Screening
– Nomination
– Delaware Behavior Checklist
– Parent or Youth Self-Report
Systematic Screening for Behavior
Disorders (SSBD)
 Walker
and Severson (1992)
 Published by Sopris West
 Cost: $120 (plus $14 per pack of
forms)
 K-6
 Identifies externalizing and
internalizing disorders
Systematic Screening for Behavior
Disorders (SSBD)
Based on teacher judgment which is
reliable and valid but underutilized
 Simple procedures at Gates 1 and 2 that:
– Take full advantage of teachers’
judgments
– Require attention to the full range of
potentially problematic behaviors
(internalizing and externalizing)
– Require systematic attention to all
students in a class

Systematic Screening for Behavior
Disorders (SSBD)
 Procedures
at Gate 3 that:
–Address the two major adjustment
areas for children
Teacher-related
classroom adjustment
Peer-related social adjustment
Systematic Screening for Behavior
Disorders (SSBD)
 Don’t
conduct screening until at least
one month into school year; end of
October preferred
 Don’t rank any student that the
teacher has known for less than one
month
 Screen again in February to pick up
new students, missed students, and
those whose behavior has changed
Systematic Screening for Behavior
Disorders (SSBD)
 Gate
1 – Step 1:
 Gate
1 – Step 2:
 Gate
1 – Step 3:
– Study definitions and examples of
externalizing and internalizing behavior
problems
– Using class roster, select 10 students for
EACH of the externalizing and
internalizing groups
– Rank order students in each group from
most to least according to degree of
externalizing or internalizing behavior
Systematic Screening for Behavior
Disorders (SSBD)
 Gate
2
–Working only with the top three
students from each list
–Complete two rating scales on all
six children
Systematic Screening for Behavior
Disorders (SSBD)
 Gate
2:
– Critical Events Index
 Steals,
trantrums, damages property, etc.
– Combined Frequency Index
5
point Likert scale (never to frequently)
 12 adaptive (e.g., considerate of others’
feelings, cooperates)
 12 maladaptive (e.g., behaves
inappropriately when corrected, manipulates
others to get his/her own way)
Systematic Screening for Behavior
Disorders (SSBD)
 Gates
1 and 2 are designed to be
completed in a single one hour
faculty meeting.
–Allows oral review of definitions
–Allows questions to be answered
 When
students are selected for
Gate 3, probably seek
permission to continue
Systematic Screening for Behavior
Disorders (SSBD)
 Decision
rules to pass to Gate 3
–For each “externalizing” child:
If
CEI ≥ 5, child passes to Gate 3
If CEI is < 5 but >0, check CFI
– If adaptive score is > 30, stop.
– If adaptive score is ≤ 30, examine
maladaptive score.
– If maladaptive score is < 35, stop.
– If maladaptive score is ≥ 35, child passes
to Gate 3.
Systematic Screening for Behavior
Disorders (SSBD)
 Gate
3 – systematic observation
–Classroom and playground
Two
observations in each setting
At least 15 minutes per observation
Includes observation of normative
peer (optional)
–Conducted by a trained observer
other than the child’s classroom
teacher
Systematic Screening for Behavior
Disorders (SSBD)
 Academic
Engaged Time (AET)
–Attending to materials and task
–Making appropriate responses
–Asking for assistance appropriately
–Duration recording
Systematic Screening for Behavior
Disorders (SSBD)
 Peer
Social Behavior Observation
–Social engagement
–Participation
–Parallel play
–Alone
–(No Code)
–Interval recording
Systematic Screening for Behavior
Disorders (SSBD)
 Decision
criteria for Gate 3:
–Internalizers:
AET
≤ 45% OR
% Alone + % Parallel Play
– 40% or more (Grades 1-3)
– 35% or more (Grades 4-6)
–Externalizers:
AET
≤ 35% OR
Total Negative behavior ≥ 12%
Systematic Screening for Behavior
Disorders (SSBD)
 Around
17% of students
nominated as externalizing pass
Gates 2 and 3 (0% of controls)
 Around
11% of students
nominated as internalizing pass
Gates 2 and 3 (0.5% of controls)
Systematic Screening for Behavior
Disorders (SSBD)
 Technical
data
–Gate 1: Test-retest (one month)
averages .88 for externalizing and
.74 for internalizing
–Gate 2: CEI r = .81; CFI r = .90
(adaptive); .87 (maladaptive)
–Gate 3: About 90% of students
correctly classified as externalizing,
internalizing, or control
Systematic Screening for Behavior
Disorders (SSBD)
 Technical
data
–Large normative sample (4,463
students at Stage II and 1,275 students
at Stage III)
– From 8 states with west overrepresented
– Predictive validity could be higher (52%
of internalizers and 69% of externalizers
were ranked in top 3 the following year)
Jefferson Parish Public Schools
Screening 2007-2008
Students
Total # screened
Total # nominated
Average # nominated
per school
Total # “AT-RISK”
Average # “AT-RISK”
per school
Morgan-D’atrio, et al. (2008)
#
16,634
%
38% of JPPSS
students
3,521
21.2% of all students
screened
66
22%
Range: 5.1% - 36.8%
1299
36.9% of nominated /
7.8% of total population
25
8.0%
Range: 2.0% - 21.7%
Jefferson Parish Public Schools
Screening 2007-2008
% “AT-RISK”
by Gender:
% “AT-RISK” by
SSBD Dimension:
% “AT-RISK”
by Race:
*(N = 1,299)
71.7%*
Male
28.3%*
Female
66.6%*
Externalizing
33.4%*
Internalizing
60.6%*
29.3%*
African
White
American
Morgan-D’atrio, et al. (2008)
6.2%* 3.9%*
Hispanic Other
Systematic Screening for Behavior
Disorders (SSBD)
 Conclusions:
– Worth considering
– Use caution with norms
– Conduct local evaluation
BASC-2 Behavioral and Emotional
Screening System (BESS)
 Purpose:
– To provide a standardized, efficient, and
effective way to identify behavioral and
emotional strengths and weaknesses in
children and adolescents from PK-high
school.
– Intended for prevention and early
intervention
BASC-2/BESS
Kamphaus and Reynolds (2007)
 Published by Pearson
 Cost: $60 (manual); $589 (scoring
software); $98 (pack of 100 forms)
 Preschool (age 3) to Grade 12; separate
forms depending on age and informant
 Teacher, parent, self-report forms
(beginning grade 3)
 Global score to indicate risk for problems

BASC-2/BESS
 Designed
for use in a multiple gating
procedure
 Parent and student forms are
available in Spanish
 Reading levels at about 6th grade for
parents and 2nd grade for students
 Includes a validity check for
respondents
 Software available for easy scoring
BASC-2/BESS
 Similar
procedures
– Conduct assessment using teachers as
informants only after at least one month
of daily contact
– Ask parent who is most familiar with the
child
– Self report beginning in grade 3
– Active parental consent for parent and
self-report versions; teacher version?
BASC-2/BESS
 Who
to screen?
– Transitional years
– New students
– Schools with high numbers of behavioral
problems (for prevention at early
grades)
BASC-2/BESS
 Gate
1: BESS administration
– 25-30 items (depending on form)
– 4 point Likert scale
 Never,
sometimes, often almost always
– About 5-10 minutes to administer
– Scan or hand-key forms
– ASSIST Software
BASC-2/BESS
 Total
Score for each student
– T score in one of three categories
 Normal
risk (T ≤ 60)
 Elevated risk (T = 61 to 70)
 Extremely elevated risk (T ≥ 71)
– User can set level of risk for moving to
next gate
BASC-2/BESS
 Gate
2: Administer BASC-2 for all
those identified as falling above
accepted level of risk
 All those above pre-determined
elevation levels for internalizing,
externalizing, total problems, etc.
Further
evaluation
And/or
Intervention
BASC-2/BESS
 Gate
3: “comprehensive diagnostic
assessment” leading to placement
decision or differential diagnosis.
BASC-2/BESS
 Technical
data
– Nationally representative sample
(matched to US population for race,
region, and SES) of 12,350
– High levels of reliability:
Test-retest (interval not reported):
.80-.91
Internal consistency: .90-.96
Inter-rater: .71-.83
BASC-2/BESS
 Technical
data
– Good positive predictive values for total
problems (.73-.82) and externalizing
(.61-.76)
– Low levels for internalizing at preschool
age (.30-.47)
– Moderate levels for internalizing at
child/adolescent (above .5)
BASC-2/BESS
 Conclusions:
– Excellent national norms, co-normed
with instrument already in use in many
places
– Too soon to tell if predictive validity
estimates will hold up
– Expense of scoring software a drawback
– Offers flexibility in setting levels of risk
– Allows large scale, district wide data
collection and reporting
– Not good for progress monitoring
Build-Your-Own-Screening
 Simple
nomination and individualized
follow-up
– Clearly define “at risk” for purposes of
nomination
– Teachers nominate X students meeting the
definition (Gate 1)
– Select a more detailed rating scale for
nominated students that matches the
definition; set criterion for risk (Gate 2)
– Observations & interviews as needed to
determine intervention steps (Gate 3)
Build-Your-Own-Screening
 Simple
nomination and individualized
follow-up
– Clearly define “at risk” for purposes of
nomination
– Teachers nominate X students meeting the
definition (Gate 1)
– Select a more detailed rating scale for
nominated students that matches the
definition; set criterion for risk (Gate 2)
– Observations & interviews as needed to
determine intervention steps (Gate 3)
Build-Your-Own-Screening
 Simple
nomination and individualized
follow-up
– Clearly define “at risk” for purposes of
nomination
– Teachers nominate X students meeting the
definition (Gate 1)
– Select a more detailed rating scale for
nominated students that matches the
definition; set criterion for risk (Gate 2)
– Observations & interviews as needed to
determine intervention steps (Gate 3)
Build-Your-Own-Screening
 Simple
nomination and individualized
follow-up
– Clearly define “at risk” for purposes of
nomination
– Teachers nominate X students meeting the
definition (Gate 1)
– Select a more detailed rating scale for
nominated students that matches the
definition; set criterion for risk (Gate 2)
– Observations & interviews as needed to
determine intervention steps (Gate 3)
Build-Your-Own-Screening
 Delaware
Behavior Checklist
– Brief (20 items) rating of all students in
classroom
– 3 point Likert scale (not a problem,
sometimes a problem, often a problem)
– Produces 5 scale scores and a total:
 Aggression
 Classroom
disruption
 Depression/anxiety
 Peer relations
 Academic problems
Build-Your-Own-Screening
 Delaware
Behavior Checklist
– Currently, norms for Christina School
District only
– Use as Gate 1 screening; follow up
those scoring 1.5 standard deviations
above the mean for either total or
individual scale scores
– Gates 2 and 3 as above
Build-Your-Own-Screening
 Parent
report at Gate 1
– Pediatric Symptom Checklist-17 (Jellinek,
Murphy, & Burns, 1986)
 Ages
6-12
 17 items
 3 point Likert scale (never, sometimes,
often)
 Yields internalizing, externalizing, and total
scores
 Available on line:
http://ocs.ccri.ws/psc-17.asp
Build-Your-Own-Screening
 Youth
self-report at Gate 1
– Use active consent
– Broad-band measures:
 BASC-2
Self-report
 Youth Self Report (Achenbach scales)
– Narrow-band measures:
 Multi-dimensional
Anxiety Scale
 Reynolds Adolescent Depression Scale
 Children’s Depression Inventory
 Personal Experiences Screening
Questionnaire (substance use)
No matter what the method…
 Consider
issues of consent
 Make
sure students identified as high
risk receive intervention
 Conduct
program evaluation
– See www.casel.org/assessment
Linking to Intervention
 Use
what you have
 Monitor
what you do
Tools for progress monitoring:
http://www.jimwrightonline.com/
php/tbrc/tbrc.php
Linking to Intervention
 Use
what you have
 Monitor
 Refer
what you do
as needed
Linking to Intervention
Keys to making a “good” referral:

Involve family early and often
– Describe what has been learned
– ASK family for their views and ideas about
what, if anything, should be done

Brainstorm options whenever possible
– More than one agency?
– Community resources?

Assist the family with the appointment
– Call while they are with you
– Follow up
 Read
more about it….
 Questions
or support?
– http://www.udel.edu/cds/pbs/
– [email protected]