Transcript Document

And the Survey Says….
Putting YOUR Healthy Youth Survey
Results to Work
Spring 2003
Purpose:
Help people understand & use
their HYS reports
Welcome!
• Introduction of
presenters
• How many agencies are
here today?
• Review Agenda
• What we will have
accomplished by the end
of the day?
Welcoming picture drawn
by a Healthy Youth
Ice breaker
Ice Breaker
• In small groups, each person shares
– Name
– Where you work
– Share one interesting or obscure fact about yourself
Background of HYS 2002
History of WA survey efforts
Content of HYS 2002
Sources of HYS 2002 items
Data processing and quality control steps
Past Washington Surveys:
Primary Content
‘88 ‘89 ‘90 ‘91 ‘92 ‘93 ‘94 ‘95 ‘96 ‘97 ‘98 ‘99 ‘00 ‘01 ‘02
Health-risk Focused
Combination
Risk & Protective Factor Focused
Joint Survey Planning Committee
•
•
•
•
•
•
Office of the Superintendent of Public Instruction
Division of Alcohol & Substance Abuse, DSHS
Department of Health
Office of Community Development
Governor’s Family Policy Council
RMC Research (survey contractor)
HYS 2002: Collaborative Effort of
OSPI, DOH, DSHS, and CTED
• Simple random sample of schools recruited at state level
– County samples drawn as appropriate
•
•
•
•
Survey consistently administered in the Fall of even years
Survey given to 6th, 8th, 10th, and 12th graders
Survey booklets will have one-page tear-off answer sheet
Survey for 8th, 10th, and 12th graders will use 2 form
“interleaved” administration
HYS 2002: Participation
• The Healthy Youth Survey 2002 was completed…
By 137,335 students…
In 752 schools…
In 203 school districts…
In 39 counties.
Risk & Protective Factor Data
Consistent with CTC/MTF Survey
Development led by DASA/DSHS & OSPI
WSSAHB Questionnaire
• Demographics
• Alcohol, Tobacco, other drugs
• School risk/protective factors
• Community risk/protective factors
• Peer-Individual-Family risk/ protective
factors
Youth Health Behavior Data
Consistent with National YRBS & YTS
Development led by DOH
YRBS Questionnaire
• Demographics
• Alcohol, Tobacco, Other drugs
• Nutrition
• Physical Activity
• Safety Behaviors (Helmets,
Seatbelts)
• Mental Health/Depression/Suicide
• Additional Tobacco Indicators
Getting the Best of Both
Form A
WSSAHBlike
Survey
Core
Form B
YRBS-like
6th Grade Survey
One Form
Survey
Core
+
Mutually
Agreed
Items
Core Survey Items
•
•
•
•
•
Demographics
Alcohol, Tobacco and Other Drug Use Items
Key Violence-Related Items
School-Specific Asset Items
Depression
Sources of HYS 2002 Items
•
•
•
•
•
Monitoring the Future (MTF)
Youth Risk Behavior Survey (YRBS)
Communities that Care (CTC)
Youth Tobacco Survey (YTS)
PRIDE Survey
Data Processing and Quality Control
• Administration procedures
• Data cleaning procedures
Statistical Issues
Validity and reliability
Confidence intervals
Comparing state and local results
Generalizability
Validity and Reliability
• Definitions
• How we assure validity: Items from established
instruments, validity checks
• How we assure reliability: Standardized
administration procedures
Confidence Intervals
•
•
•
•
What are they?
How to interpret the numbers
How to interpret the graphs
Practice activity (Ex. 1–2)
What is a Confidence Interval?
• The reported value is probably a little different
than the “true” value for all your students
• We are “95% confident” that the “true” value is
within the +/- range, called the “confidence
interval”
Interpreting CIs Numerically
Smoked cigarettes: 9.2% (± 1.1%)
9.2 – 1.1 = 8.1
9.2 + 1.1 = 10.3
Between 8.1% and 10.3% smoked cigarettes
Interpreting CIs Graphically
Smoked
cigarettes
0
5
10
Percent of students
15
20
Practice Activity
• Exercises 1 and 2
Why Are Confidence Intervals
Different Sizes?
•
•
•
•
Number of students
Inherent variability
Level of confidence (All 95% for HYS)
Sampling design
If the data are valid, why do we need
confidence intervals?
• Confidence intervals account for variability
among students, NOT validity of the data
• Variability is inherent in any population worth
studying
• Variability causes uncertainty in the results
• “Doubt is not a pleasant condition, but certainty is
absurd” – Voltaire
Comparing State and Local Results
• Looking at differences numerically
• Looking at differences graphically
• Practice activity (Ex. 3–4)
Significant Differences: Numerically (Ex. 1)
• Smoked cigarettes in the state: 9.2% (± 1.1%)
Between 8.1% and 10.3%
• Smoked cigarettes at my school: 14.1% (± 2.2%)
Between 11.9% to 16.3%
• Conclusion: Difference IS statistically significant
Significant Differences: Numerically (Ex. 2)
• Smoked cigarettes in the state: 9.2% (± 1.1%)
Between 8.1% and 10.3%
• Smoked cigarettes at my school: 14.1% (± 9.6%)
Between 4.5% to 23.7%
• Conclusion: Difference is NOT statistically significant
Significant Differences: Graphically (Ex. 1)
Local
State
Smoked
cigarettes
0
5
10
15
Percent of students
20
25
Significant Differences: Graphically (Ex. 2)
Local
State
Smoked
cigarettes
0
5
10
15
Percent of students
20
25
Practice Activity
• Exercises 3 and 4
Generalizability
• What is generalizability?
• To whom can we generalize results?
• State & County sample
• Non-sampled counties, districts, and ESDs
• Schools
• Why 70% participation is required
Quotations From Teachers Who
Administered the Survey
• About 25 students missed part of the session
because of club photos
• Absent students are taking a PSAT test
• Many students had too much homework. Chose to
do that rather than take the survey.
• Seven students came too late to take the survey
• The perfectionists, poor direction followers and
slow readers could not finish
Challenges to Generalizability
• Brainstorm challenges to generalizability
– School level
– ESD, County, or District level
• How do these challenges affect data
interpretation?
What if I don’t want to generalize?
• Confidence intervals matter if you want to
generalize to a larger population
• You can describe the students surveyed, in that
moment, without confidence intervals:
– “Elmo Elementary School students who took the survey
said…”
Final Note About Significance
• Even if a difference is statistically significant, it
might not be practically significant
• Example
– State alcohol use: 12.8 (±0.2) %
– Local alcohol use: 14.4 (±1.2) %
Statistically significant difference, but should it influence
program planning?
Comparing data over time
Things to think about
• Did the questions change?
• Were the challenges to generalizability similar?
• Is there a reason to think that things would have
changed? (a prevention program)
Surveys over time
• WSSAHB 2000 reports did not report confidence
intervals – but the variability was still there!
• Future HYS reports will have confidence
intervals, so 2002 vs. 2004 comparisons will be
easier
What should I know about comparing
2000 to 2002 data?
• If you think you have comparable data, then:
– If the 2000 percentage is INSIDE the 2002 confidence
interval, then there is not a “statistically significant”
difference
– If the 2000 percentage is OUTSIDE the 2002
confidence interval, then you can’t be sure if the
difference is “statistically significant” but the further
away it is the more likely that a difference is significant
– If you really, really need to know whether it’s
significant, you can contact DOH for a specialized data
run
What if I am in a small school, and we
have huge confidence intervals?
• Having a confidence interval protects you (and
your program) from appearing to be ineffective
when just a few students can make big changes
• Consider the input of teachers & staff from smallschool environments when interpreting data – the
data should be used to complement what they
already know about their kids
Break
Local Report Overview
Guide to the Local Report
FAQ
How to use your local report
Local Report Contents
• Introduction and Overview
– Key to the Notes
– Highlights of the Local Results
• Graphical summary of selected results
• Selected results disaggregated by gender
• Responses to all items
Guide to the Local Report, cont.
Healthy Youth Survey 2002
Survey Results
Sample Middle School, Grade 8
Number of students surveyed:
64
Number of valid responses:
59
The number of students who
submitted survey forms
The impact of adolescent health risk behaviors remains a primary concern of
citizens throughout the country. Many health problems experienced by adolescents
are caused by a very few preventable behaviors….
Guide to the Local Report, cont.
Healthy Youth Survey 2002
Survey Results
Sample Middle School, Grade 8
Number of students surveyed:
64
Number of valid responses:
59
The number of forms that were
considered usable.
The impact of adolescent health risk behaviors remains a primary concern of
citizens throughout the country. Many health problems experienced by adolescents
are caused by a very few preventable behaviors….
Guide to the Local Report, cont.
Selected Results Disaggregated by Gender
25. During the past 30 days, on how many
days did you smoke cigarettes?
Local
State
(n = 59)
(n = 4,960)
Female Male Female Male
None
70.4% 93.8% 89.9% 84.9%
1 or more
29.6
6.2
10.1
15.1
Note. Results are suppressed if any cell represents fewer than 15 students;
p = .021 from a chi-square test.
To protect anonymity when data are broken
down into identifiable groups.
Guide to the Local Report, cont.
Item Results
42.
How old were you the first time you smoked a whole
cigarette?
a. Never have
b. 10 or younger
c. 11
d. 12
Number of students who
e. 13
responded to this item
f. 14
g. 15
h. 16
i. 17 or older
Local
(n = 20)
60.0% (± 21.5%)
10.0
(± 13.1)
15.0
(±15.6)
5.0
(± 9.6)
10.0
(± 13.1)
0.0
(± 0.0)
0.0
(± 0.0)
0.0
(± 0.0)
0.0
(± 0.0)
State
(n =4,687)
74.7% (± 1.2%)
6.4 (± 0.7)
6.5 (± 0.7)
6.8 (± 0.7)
4.8 (± 0.6)
0.8 (± 0.3)
0.0 (± 0.0)
0.0 (± 0.0)
0.1 (± 0.1)
[Item 43 appears only on the elementary version of the survey.]
45.
Do you think young people risk harming themselves if they
smoke from 1–5 cigarettes per day?B (one to five cigarettes
a day?) C
a. Definitely no
b. Probably no
c. Probably yes
d. Definitely yes
Local
(n = 8)
12.5%
12.5
0.0
75.0
(± 22.9%)
(± 22.9)
(± 0.0)
(± 30.0)
State
(n =4,877)
4.9%
11.5
38.7
44.9
(± 0.6%)
(± 0.9)
(± 1.4)
(± 1.4)
Where Do Find Information About . . . ?
•
•
•
•
•
•
General Information (e.g. demographics)
Alcohol, Tobacco, and Other Drug Use
Other Health Concerns (e.g. health/safety)
School climate (e.g., bullying)
Quality of Life (e.g., student outlook)
Risk and Protective Factors:
– Scale Results
– Item Results
Frequently Asked Questions
• Review handout
• Note about optional items
How to use your local report
• Example
• Practice Activity
Local Report Example
168. If you wanted to get some
marijuana, how easy would
it be for you to get some?
a. Very hard
b. Sort of hard
c. Sort of easy
d. Very easy
Local
(n = 84)
52.4% (± 10.7%)
19.1 (± 8.4)
15.5 (± 7.7)
13.1 (± 7.2)
State
(n = 3,738)
56.6% (± 2.7%)
16.9 (± 1.5)
13.9 (± 1.3)
12.6 (± 1.4)
State Agencies Using the HYS Data
Federal Initiatives
• Drug-Free Schools and Communities Act
• Department of Education Safe and Drug-Free
Schools Program
• Principles of Effectiveness
• Healthy People 2010: National Health Promotion
and Disease Prevention Objectives
• National Drug Control Strategy
• No Child Left Behind
Washington State Initiatives
•
•
•
•
•
Washington Education Reform Act
Omnibus Alcohol and Controlled Substances Act
Violence Reduction Programs Act
Department of Health Priority Health Goals
Governor's Council on Substance Abuse
Why Do You Keep Changing Your
Terminology?
Or…When is an Impact an Outcome?
Putting Your Data to Work
Program Planning and Evaluation
What’s the Problem?
•
What is “a problem”?
– Getting worse
– Worse than average
– Just plain bad
What’s the Problem?
• Combine HYS data with other information,
including knowledge of community stakeholders
– For example, HYS data may show that the highest rates
of tobacco use are in a particular area of the county,
where police know there is a problem with retailers
who provide tobacco to minors
Exercise 1: What’s the Problem?
Lunch
What’s the Priority?
•
•
•
HYS data helped to identify a set of problems
Need to prioritize among them
Identify available resources
– Who is already working on the issue, or who has a
stake in the issue
– Gaps in services or programs
– Modify existing programs, instead of creating all
new ones
– Figure out what the community is “ready” for
What’s the Priority?
• Target Efforts
– Describe a target audience (grade level, gender,
universal or selective approaches)
– Write “SMART” goals and objectives to get FOCUSED
and CLEAR with partners, and evaluation
• Specific
• Measurable
• Achievable
• Relevant
• Time-bound
Setting a Good Objective
• You have 1,000 students
• Current smoking prevalence is 25% (250 kids)
• SMART Objective: By 2004, we will implement a
comprehensive tobacco prevention program and
reduce current smoking among our school’s
students by 10%
• This translates into about 25 fewer smokers
• This is a change from 25% to 22.5%
Keeping it Real
• Will your program reach enough kids to affect the
overall prevalence?
• Remember the difference between a “percentage
point” goal and a “percent change” goal!
Population Group Changes - Gender
Current Smoking Trends, by Gender
Current Smokers
40%
2000
20%
2002
0%
girls
boys
6th
girls
boys
girls
8th
boys
10th
Grade
Source: WSSAHB 2000, HYS 2002, state sample data
girls
boys
12th
Exercise 2: What’s the Priority?
What Should I Do About It?
• Find “Best Practice” programs
–
–
–
–
Search for proven activities
The “Community Guide to Preventive Services”
CSAP “Science-based Prevention Programs”
Agency recommendations
• for example, DOH recommends conducting a
retailer compliance program + public awareness
about sources of tobacco
How Do I Know If It Worked?
Evaluate
• Identify and collect process indicators, outcomes,
and impacts relative to your program
– Process measures may include compliance checks &
compliance rates
– Outcomes may include perceived availability of
tobacco & ‘usual sources’ of tobacco
– Impacts may include lifetime and current use of
tobacco
Evaluation made easy
•
•
•
•
What do I do when I go to work every day?
….and then what happens?
….and then what happens?
….so that the WHY of going to work every day
happens.
Exercise 3: How Do I Know If It
Worked?
Just in case you need more money…
• Healthy Youth Survey data can help
– Provide justification of need for funding
– Provide means for evaluating funded activities
• Find out about grant scoring
• You will need to justify that HYS data are
– Valid
– Reliable
Exercise 4: Grant Writing
Next Steps
Communicating With Your Data
Communications
•
•
•
•
Nuts and bolts
Communicating about your program
Communicating to the media
Using data to make change
The Nuts and Bolts
• Round percents to whole numbers
– 17.0% to 17.4% becomes 17%
– 17.5% to 17.9% becomes 18%
• Use language to acknowledge that these percents
are estimates
• “About 17%”
• “14 to 20 percent”
• “plus or minus three percent”
Different Ways to Say It
Instead of saying:
• “17.2% +/- 3.2% youth said that they ‘mostly’ or
‘definitely’ felt unsafe at school”
You could say:
• “About 17% of our youth feel unsafe in school”
• “About one in five youth feel unsafe in school”
Remember you can present it positively:
• “About 83% of our youth feel SAFE in school”
• “About four out of five youth feel SAFE in school”
Additional Tips
• If you had a low response rate for your survey, it
would be best to say:
– “Elmo Elementary School students who took the survey
said…”
Instead of
– “Elmo Elementary School students said…”
Communicating About Your Program
What data are appropriate for each type of
communication?
• Your co-workers
• Your neighbor at a cocktail party
• Your grandmother
Communicating to the Media
•
What’s your SOCO? (Single Overriding
Communication Objective)
–
–
•
For broadcast media: 10-12 word “sound bite”
For print media: 1-3 line quotation
Message map
Single Overriding Communication Objective
Detail 1
Detail 3
Detail 3
Fact
Fact
Fact
Fact
Fact
Fact
Fact
Fact
Fact
For Example: Tobacco Program Results
• SOCO: Our program has been successful in
reducing youth smoking
– Detail 1: Fewer youth are currently smoking than prior
to the program
• Fact 1: 53% fewer 6th graders
– Detail 2: Fewer youth have ever tried smoking
• Fact: 29% fewer 10th graders have ever smoked a
whole cigarette
– Detail 3: Washington is doing better than the nation
• Fact: there was a 22% decrease in the nation for
current smoking, so Washington’s decline is greater
Potential Pitfalls
• When communicating with data, it’s easy to make
these mistakes:
– Include information that’s not totally relevant to your
SOCO
– Include information that is relevant, but goes beyond
what you NEED (too much detail)
– Speculate beyond what you have in the data
– Treat non-significant or unimportant differences as
significant
– Forget to double-check your results and/or the math
Keep “on message”
•
The answer to every question is your SOCO.
“That’s an interesting point, but what
I’d really like people to know is,
[ my SOCO]”
Message Mapping Exercise
In small groups
• Choose 1 topic of interest from HYS
• Develop SOCO and 3 supporting details
• Use HYS local report to find 3 supporting facts for
each detail
• Message map template in handouts
Using Data to Make Change
• The message should tell your audience what you
want them to do:
–
–
–
–
Do you want them to be outraged? And then what…
Do you want them to call a policymaker?
Do you want them to change their beliefs or behaviors?
Do you want them to support your program?
Choose and Describe Your Audience
• Potential Audiences
–
–
–
–
–
–
–
–
Co-workers
Area residents
Elected officials
Civic organizations
Health care providers
Media
Regulatory agencies
Activists
• Demographics
• Characteristics
–
–
–
–
–
–
–
–
–
Concerns
Attitudes
Levels of interests
Levels of involvement
Histories
Levels of knowledge
Opinions
Reasons for interest
Types of involvement
How Will You Reach Your Audience?
Area residents
• Community meetings
• Newspaper articles and
ads
• Radio and TV talk shows
• Fliers
• Direct Mailings
Elected officials
• Frequent phone calls
• Fact sheets
• Personal visits
• Invitations to
community meetings
• News releases
Cardinal Rules of Communication
with Data
1. Accept and involve the public as a partner
–
–
–
They’re going to comment on your stuff anyway
Know ahead of time what public opinion is around your topic
Anticipate (and prepare for) the “tough questions & smart
answers” – including criticism of the survey itself
2. Plan carefully and evaluate your efforts
•
•
•
Was your SOCO ‘heard’?
Did you reach your target audience?
Did they know what you wanted them to do?
3. Be honest, frank, and open
•
Be prepared to talk about the limitations of your data
4. Work with other credible sources
•
•
Have someone critique your findings or interpretation
…and double-check your math
Cardinal Rules of Communicating
with Data (cont.)
5.
Meet the needs of the media
•
•
•
6.
Meet their deadlines
Help them understand what the data mean
Help them understand how the survey works, if needed
Speak clearly and with compassion
•
•
7.
“data are just people with the tears wiped away”
Practice saying it out loud (numbers can trip your tongue)
Use AV or graphics with caution
•
•
•
Sometimes a graph hurts more than helps you
Visual aids might distract from your message
Don’t try to exaggerate with graphics
Thank You!!
Sun setting on training session
More questions about the training?
• Department of Health
Lauren Jenks
360-236-3567
[email protected]
• OSPI
Bob McArdle
360-725-6046
[email protected]
Mona Johnson
360-725-6044
[email protected]
• DASA, DSHS
Linda Becker
360-407-0640
[email protected]
• RMC Research Corporation
Eric Einspruch
Gwen Hyatt
1-800-788-1887
[email protected]
http://www.rmccorp.com/project/hys02.html
The Joint Survey Planning Committee
OSPI
• Mona Johnson
• Bob McArdle
• Martin Mueller
• Pam Tollefsen
OCD
• Susie Roberts
Family Policy Council
• Bill Hall
DASA
• Linda Becker
• Steve Smothers
DOH
• Lillian Bensley
• Julia Dilley
• Lauren Jenks
• Susan Richardson
• Judy Schoder
• Juliet VanEenwyk