Transcript Document
Healthy Youth Survey Using your Data Workshop Spring 2009 Welcome! Review agenda and objectives for the day Is there anything else you hope to get out of this training? Logistics Break Workbook follows this powerpoint Do you have a local report??? 2 20 Years of Student Surveying!!!! What were you doing 20 years ago? Share with your neighbor. 3 Group Discussion: How have you used HYS results? Who do you plan on using your results with? 1 New Ways to Use Your Results With schools, communities, and youth Using Healthy Youth Survey Data with School Administrators 6 Points to Include….. Connections to School Risk and Protective Basics Link Student Behavior/Attitudes with Academics Overview HYS Example of Action Resources 7 Why is linking academics and child well-being important? “What we do in the name of health, safety, and well-being are linked with teaching and learning. Teaching and learning can’t take place if students aren’t healthy, aren’t physically and mentally fit, or aren’t safe.” ~ William Modzeleski, Director, Safe and Drug Free Schools Program, U.S. Department of Education 8 Make the Connection ~ Student Behaviors & Academic Success Early use of alcohol and cigarettes = lower WASL scores More disruptive/aggressive behavior = lower WASL scores More alcohol and drug use = less likelihood of passing WASL Higher bonding to school = higher WASL scores & better grades Better social skills = higher WASL scores & better grades 9 Taking Action… some examples Share data with parents/students Implement targeted or school-wide initiatives to address areas of interest Set school improvement goals (or strategies embedded within goals) related to risk and protective factors 10 Regional Prevention Contacts http://www.k12.wa.us/PreventionIntervention/ CoordinatorDirectory.aspx Curriculum Professional Development Technical Assistance Data Support 11 Community Mobilization Using HYS data to help change community perspectives about important issues among kids 1. Bring the community together to discuss how to reduce underage drinking 2. Motivate community to talk with children about 13 not using alcohol before age 21. 14 15 Keys to success Well organized and strong coalition Close collaboration with the school district Involvement of the kids They planned a great meeting, and fed everybody! Resulted in a new parent network 16 Sharing HYS with Youth To get their perspective on what’s important to them and their ideas for improving their community. Example: Wahkiakum’s Healthy Youth Summits In 2005 and 2007, Wahkiakum held Healthy Youth Summits: A community-wide group of youth and community/ school adult leaders met Youth and adults shared their interpretations of HYS 18 data In the morning, the YOUTH…. Reviewed HYS data and discussed: If they believed the results Reasons for their results What needed to change in their community Prioritized issues and recommendations for adult 19 leaders In the afternoon, the ADULTS…. Reviewed the data & what youth said Discussed priorities and their recommendations Began to think about coming year & what could be done to address youth issues 20 Keys for a successful summit Create a comfortable and confidential environment for youth Assure youth that they will be taken seriously - they are the experts and their opinions are highly valued Develop concrete ideas for addressing youth issues Recognize that everything doesn’t have to be solved immediately Appreciate the importance of starting this type of dialog between youth and adults 21 Replicating something similar There are tools available to it easy to talk about HYS data: Powerpoint slides of results (sent by RMC, see page 15 in your workbook) AskHYS.net topical fact sheets This could be done with in many ways: With a school or community youth group Within any geographic location – a county, a city, a school district, or a school It could even be done in a single classroom 22 2 Understanding Your Local Report and Results 2008 administration Local report overview Understanding your results HYS 2008 administration A collaborative effort among the Joint Survey Planning Committee, made up of 6 state agencies and the survey contractor: Office of the Superintendent of Public Instruction Department of Health Department of Social & Health Services/ Department of Alcohol & Substance Abuse Department of Community, Trade and Economic Development Family Policy Council Liquor Control Board 24 2008 HYS participation The 2008 Healthy Youth Survey was completed… By 211,244 students… In all 39 counties. In 247 school districts… In 1,097 schools… 25 Local Report Overview Types of reports Your report Summary of selected results Selected results by gender Item groupings Risk and protective factors List of core items & item index Types of reports Individual School reports Districts, counties & ESDs Report of Results o >=70% participation rate Report of Participating Schools o 40-69% participation rate 27 Your report Healthy Youth Survey 2008 Survey Results Sample Middle School, Grade 8 Number of students surveyed: Number of valid responses: Estimate of enrolled students: Survey participation rate: 173 165 206 80% The impact of adolescent health risk behaviors remains a primary concern of citizens throughout the country…. 28 Your report, cont. Key to the Notes A = wording on Form A B = wording on Form B C = wording on Form C † = optional item 29 Summary of selected results Reports include same questions and graphs as in 2002 through 2006 reports Substance Use and Weapon Carrying in the Past 30 Days Local State Smoked cigarettes Drank alcohol Used marijuana or hashish Carried a weapon at school 0 10 20 30 40 50 60 70 80 90 100 Percent of students 30 Cross-tab results by gender Same as in 2002 through 2006 reports Results suppressed to protect anonymity During the past l2 months, did you ever feel so sad or hopeless almost every day for two weeks or more in a row that you stopped doing some usual activities? (See item 122.) No Yes Local State (n = 164) (n = 8,295) Female Male Female Male 62.8% 76.7% 69.6% 80.9% 37.2 23.3 30.4 19.1 Note. p = .04 from a chi-square test. If p value < 0.05 the difference between local males and local females is statistically significant Do you have asterisks instead of numbers for gender? 31 Item groupings General Information ATOD Use Other Health Concerns School Climate Quality of Life Risk and Protective Factor Items 32 Item examples (Grade 10) Does anyone who lives with you now smoke cigarettes? a. No b. Yes Local (n = 291) 56.4% (± 5.7%) 43.6 (± 5.7) State (n = 3,074) 64.6% (± 3.1%) 35.4 (± 3.1) 134. Does your school provide a counselor, intervention specialist, or other school staff member for students to discuss problems with alcohol, tobacco, or other drugs? a. No b. Yes c. I’m not sure Local (n = 593) 11.5% (± 2.6%) 58.7 (± 4.0) 29.9 (± 3.7) State (n = 6,525) 8.4% (± 1.0%) 60.1 (± 3.9) 31.4 (± 3.2) 53. 33 Why is item n < overall n? Not a core item (Grades 8, 10, and 12) Optional: look for † symbol Students did not reach item near end of survey Students skipped or did not mark clearly 34 Why asterisks instead of numbers for frequencies? No students responded (n=0) Suppressed to protect anonymity Overweight (Item 65) Results by Gender (cell n<10) 35 Risk and Protective Factors Framework and Reporting Schedule Scale Results Community Risk Factors Low Neighborhood Attachment (Items 149–151) Local (n = 311) 46.6% (± 5.5%) (n = 308) State (n = 3,463) 44.8% (± 2.5%) (n = 3,441) Graphs Individual Item Results 36 Last page List of Core Items Secondary only Helps interpret n’s Item Index 37 Fundamentals for Understanding Your Results Validity and reliability Generalizability Confidence intervals Comparing state and local results Comparing data over time Reliability Reliability: Does the survey consistently produce the same results under the same conditions? How we assure reliability: Standardized administration procedures 39 Validity Does the item measure what it is intended to measure? How we assure validity: Items from established instruments, validity checks 40 Generalizability What is generalizability? Why 70% participation is important Challenges to generalizability School level ESD, County, or District level How do these challenges affect data interpretation? 41 Do I have to generalize? Yes, if you want to apply the results to a larger population. 8th graders in our district said…. Yes, if you want to compare to others or results over time No, if you want to just describe the students surveyed, in that moment, without confidence intervals: Students at our school who took the survey said…. 42 Confidence Intervals What are they? How to interpret the numbers How to interpret the graphs Comparing State and Local Results Practice Activity # 1 What is a confidence interval? The reported value is unlikely to be exactly the same as the “true” value for all your students We are “95% confident” that the “true” value is within the +/- range, called the “confidence interval” 44 Why do we need confidence intervals? Confidence intervals account for variability among students, NOT validity of the data Variability is inherent in any population worth studying Variability causes uncertainty in the results “Doubt is not a pleasant condition, but certainty is absurd” – Voltaire They help us compare our results to others and to ourselves over time 45 What do confidence intervals look like? Smoked cigarettes Smoked cigarettes (Grade 10, 2008): 14.4% (± 1.6%) 14.4 – 1.6 = 12.8%, 14.4 + 1.6 = 16.0% Between 12.8% and 16.0% smoked cigarettes 1 0% 5% 10% 15% 20% Percent of Students who Smoked 25% 30% 46 Why are confidence intervals different sizes? Number of students Inherent variability Level of confidence (All 95% for HYS) Sampling design 47 Significant differences Smoked cigarettes in the state: 14.4% (± 1.6%) Between 12.8% and 16.0% Smoked cigarettes at my school: 20.0% (± 2.0%) Between 18.0% to 22.0% Conclusion: Difference IS statistically significant (confidence intervals do not overlap) 48 Significant differences State School Smoked Cigarettes 0 5 10 15 Percent of Students Who Smoked 20 25 30 49 Non-significant differences Smoked cigarettes in the state: 14.4% (± 1.6%) Between 12.8% and 16.0% Smoked cigarettes at my school: 20.0% (± 10%) Between 10.0% to 30.0% Conclusion: Difference is NOT statistically significant (at least 1 confidence interval overlaps the other point estimate) 50 Non-significant differences State School Smoked Cigarettes 0 5 10 15 Percent of Students Who Smoked 20 25 30 51 Significance inconclusive Smoked cigarettes in the state: 14.4% (± 1.6%) Between 12.8% and 16.0% Smoked cigarettes at my school: 20.0% (± 5.0%) Between 15.0% to 25.0% Conclusion: Inconclusive, more testing required (confidence intervals overlap each other but not the point estimates) 52 Significance inconclusive State School Smoked Cigarettes 0 5 10 15 Percent of Students Who Smoked 20 25 30 53 Exercise 1:Comparing State and Local Results Directions: Page 38 in your workbook Question 1 o Determine if the difference between the local result and state result provided statistically significant Question 2-4 o Calculate the highest and lowest percentages o Determine if the difference between your local result and state result is statistically significant o What it mean about your local students? 54 Example – question #1 Example: What do the statistics suggest are the lowest and highest percentages of students who said “Yes” to the question “Has a doctor or nurse ever told you that you have asthma?” Example local estimate: Actual range 28.0% ± 4.0% Example state estimate: 24.0% to 32.0% Actual range (lowest) (highest) 20.8% ±1.5% 19.3% to 22.3% (lowest) (highest) Are the state and local rates different? 55 Example – question #1 graphically 24.0% Local to 32.0% State 19.3% to 22.3% 0% 5% 10% 15% 20% 25% 30% 35% Percent Ever had Asthma 56 What if I am in a small school, and we have large confidence intervals? Having a confidence interval protects you (and your program) from appearing to be ineffective when just a few students can make big changes Consider the input of teachers & staff from small-school environments when interpreting data – the data should be used to complement what they already know about their kids 57 New Tool: Using Excel and CIs to perform a statistical test Conduct a statistical test using this Excel file to determine if two groups are significantly different. Use this method if there are 30 or more respondents for the item for both the local and state results. http://www.hys.wa.gov/Reporting/Default.aspx 58 Using Excel and CIs to perform a statistical test (cont’d.) Input Section Percent Plus or minus Local Results 14.4 20.0 1.6 5.0 State Results 14.4 20.0 1.6 5.0 The p-value is less than 0.05, so the local result is significantly higher compared to the state result. Output section p-value: 0.0365494 Calculations pooled standard error Z-statistic 3 # 2.6784 -2.0908 59 Combining item responses 53. During the past 30 days, on how many days did you use tobacco (cigarettes, cigars, or chew/dip) on school property? a. 0 days b. 1 – 2 days c. 3 – 9 days d. 10 – 29 days e. All 30 days To determine the result for “Any days” Subtract the “0 days” result from 100% 100.0% - 97.0% = 3.0% Local (n=200) 97.0% (± 2.0%) 0.2 (± 1.5) 0.6 (± 0.8) 1.6 (± 1.0) 0.6 (± 0.4) State (n = 4,243) 91.1% (± 1.4%) 3.8 (± 0.8) 2.2 (± 0.4) 1.3 (± 0.4) 1.6 (± 0.4) To determine the confidence interval for “Any days”. Use the same confidence interval as the “0 days" response. (± 2.0%) 60 Combining item responses cont. 52. Do you think that rules about not using tobacco at your school are usually enforced? a. Definitely no b. Probably no c. Probably yes d. Definitely yes To determine the result for “YES” Add the “Probably yes” and “Definitely yes” responses together 29.5% + 59.0% = 88.5% Local (n=200) 5.0% (± 2.3%) 6.5 (± 4.3) 29.5 (± 6.6) 59.0 (± 5.0) State (n = 4,216) 15.3% (± 1.8%) 22.7 (± 2.2) 42.6 (± 2.5) 19.4 (± 2.0) When you combine 4 response options down to 2 groups like this, you cannot determine the confidence interval 61 Comparing data over time Things to think about Did the questions change? o http://www.hys.wa.gov/Reporting/Default.aspx Were the challenges to generalizability similar? Is there a reason to think that things would have changed? (a prevention program) 62 Final notes about significance Even if a difference is statistically significant, it might not be practically significant For example: State use of some drug: 12.8% (±0.2) Local use of that drug: 14.4% (±1.2) The difference is statistically significant, but is it different enough to influence program planning? CIs are quick and available, but there are more precise tests to determine significance. EXCEL 63 Don’t worry, help is available from many sources: Local Health Department Local ESD JSPC agencies Web materials 64 3 Exploring AskHYS.net Introduction to the website Topic specific survey fact sheets Q x Q data query analysis system Welcome to AskHYS.net 66 Log on page You don’t need to “Log On” for State, County and ESD HYS results Only need to “Log On” if you have permission to access school building and district HYS results. 67 Log in site Need permission? Click here to find out more. 68 How to get access 69 Results on AskHYS.net 70 Available results? 71 Available results Survey Year 2008 STATE Washington EDUCATIONAL SERVICE DISTRICT ESD 101 ESD 105 ESD 112 ESD 113 ESD 114 Puget Sound ESD 121 ESD 123 Northeast ESD 171 No results available Northwest ESD 189 COUNTY Adams Asotin Benton Chelan Clallam Clark Columbia Cowlitz Douglas Ferry Franklin Garfield Grade 6 Yes Grade 8 Yes Grade 10 Yes Grade 12 Yes Grade 6 Yes Yes Yes Yes Yes Yes Yes Yes Yes Grade 8 Yes Yes Yes Yes Yes Yes Yes Yes Yes Grade 10 Yes Yes Yes Yes Yes Yes Yes Yes Yes Grade 12 Yes Yes Yes Yes Yes Yes Yes Yes Yes Grade 6 Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Grade 8 Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Yes Grade 10 Yes Yes Yes Yes <40% Yes Yes Yes Yes Yes Yes Yes Grade 12 Yes Yes Yes Yes <40% Yes Yes Yes Yes Too few 72schools Yes Yes Fact sheets on AskHYS.net 73 Check the topic(s) & grade(s) For more information about the HYS questions that are used in the fact sheets – click the question mark. X 74 Select: geography, year & gender *If you have permission to access school district and building fact sheets drop menus will appear. 75 76 Information and background Healthy Youth Survey Fact Sheets Unintentional Injury for Washington State Year: 2008 Grade: 8 Gender: Both Number of Students Surveyed: 8,730 Background: Unintentional injury is the leading cause of death among youth. Motor vehicle crashes are the most common cause. Unintentional injuries are caused by behaviors such as not wearing a seatbelt, drinking and driving, riding with a drunk driver, riding a bike or motorcycle without a helmet, or not wearing a life jacket. Most unintentional injuries are not accidents because they can be prevented. Alcohol and other substances impair one’s judgment and may contribute to injuries and even death. For More Information and Resources: Visit the Washington State Injury and Violence Prevention Website at: www.doh.wa.gov/hsqa/emstrauma/injury 77 Current measure chart 2008 Unintentional Injury Grade 8 100% % of Students 80% 59% 60% 40% 19% 20% 4% 21% 4% 0% Seatbelt use (never/rarely) Rode with drinking driver Drove after drinking Bicycle Boaters riders wearing life wearing vests helmets (never/rarely) (never/rarely) th In 2008, 19% of 8 grade students in our state rode with a drinking driver in the past month. 78 State Comparison Chart Rode with Drinking Driver Compared to the State, All Grades, 2008 Local State % of Students 30% * Significantly lower local rate for 6th grade. 20% 10% * Significantly higher local rate for 12th graders. 0% Prevalence 6th 8th 10th 12th 6th 8th 10th 12th 15% ±3 26% ±2 28% ±2 19% ±1 24% ±2 23% ±2 Local 12% ±2 State 22% ±1 * * 79 Trend chart Unintentional Injury Trends, Grade 8 Rode with a drinking driver Drove after drinking Bicycle riders wearing helmets (never/rarely) Boaters wearing life vests (never/rarely) 100% % of Students 80% 60% 40% 20% 0% 2002 Prevalence 2004 2006 2008 NO Significant changes change from 2006 to 2008 2002 2004 2006 2008 Rode with drinking driver 20% ±2 18% ±2 17% ±2 19% ±1 Drove after drinking 5% ±1 4% ±1 4% ±1 4% ±1 Bicycle riders wearing helmets (n/r) 58% ±5 56% ±4 55% ±4 59% ±5 Boaters wearing life vests (n/r) 25% ±2 23% ±2 22% ±2 21% ±2 80 Academic Achievement 2008 Statewide Academic Grades in School by Rode with Drinking Driver, Grade 8 % Mostly getting C's, D's or F's 50% Statewide, youth who rode with a drinking driver are more likely to get lower grades in school (C’s, D’s or F’s) compared those who did not. 40% 30% 20% 10% 0% Rode w ith a drinking driver Did not ride 81 Analysis with Q x Q on AskHYS.net 82 Running frequencies and cross-tabs Frequencies: Use to get results for a single variable Cross-tabs: Use to get results for a variable crossed with another variable. Even gender, is a cross-tab Minimum respondents per cell required State analysis must have 5 or more Sub-state analysis must have 10 or more Questions must be on the same survey form or a “core” question E.g., can’t cross cigar smoking (only form B) with meth use (only form A) 83 Select: year, grade, gender & location 84 Starting your analysis 85 Select variable in dropdown menu 86 Select your variable Drag your variable into the “Drop first variable (row) here…” box 87 Drag your variable into 1st box Response options are collapsed. For Complete response options, click “Surveyed”. 88 For frequency results 89 Your results Current alcohol drinking for Washington State 12th graders is 40.8% of ±2.5% 90 For crosstab results Drag your SECOND variable into the “Drop second variable (column) here…” box 91 Your crosstab results Do these results look right? Race / Ethnicity Current Alcohol Drinking 92 Switch your variables around 93 Your new crosstab results Do these results look better now? 94 Selecting a sub-state location with same variables 95 Error message if cell size is too small 96 Error message if variables are not on the same form, or core 97 Right click on variable for more information or to remove it from your analysis Don’t use this – it might work later… 98 www.AskHYS.net 99 4 Communicating Your HYS Results How to talk about your results Simplify when talking about numbers Don’t say: “17.4% plus or minus 3.1% of students said…” Round whole numbers Use language to convey that these percents are estimates “About 17%” “About 17 % plus or minus three percent” “Between “14 to 21 percent” 101 Use the correct terminology How does your response rate effect how you talk about your results?: 70% Participation: “About 17% of 8th graders said……” 45% Participation: “About 17% of the 8th graders in our community who took the survey said……” 102 Consider different ways to say it… You could say: “About 17% of our youth feel unsafe in school” “About one in six youth feel unsafe in school” Remember you can present it positively: “About 83% of our youth feel SAFE in school” “About five out of six youth feel SAFE in school” 103 Communication planning What’s your SOCO? (Single Overriding Communication Objective) The “so what” of your message Meaningful to your target audience Tells them what you want them to learn or do 104 What does a SOCO look like? For broadcast media: 10-12 word “sound bite” For print media: 1-3 line quotation For real people: people just won’t remember more than 3-4 ideas 105 SOCO example “Community leaders should speak out in support of Safe & Drug Free Schools Programs” A SOCO needs to be supported by: Details: “Healthy kids learn better Provide logical justification These are “reasons why” The details need to be proven by: Facts: HYS data show strong associations between substance abuse and lower academic achievement These are the evidence for your details Use your data here 106 Message map Use your data here Single Overriding Communication Objective Detail 1 Detail 2 Detail 3 Fact Fact Fact Fact Fact Fact Fact Fact Fact 107 Example: Tobacco program results Single Overriding Communications Objective (SOCO): Our program has been successful in reducing youth smoking, but there are still challenges ahead. [and we still need funding for it] Detail 1 Washington has a comprehensive youth tobacco prevention program Detail 2 Fewer youth are smoking than prior to the program in WA Detail 3 Youth are still at risk for using tobacco. Continued work is necessary to keep rates low Fact 1 The program reaches youth at home, in their community and at school in all areas of the state Fact 1 Overall, current youth smoking rates have dropped by 50% Fact 1 The tobacco industry spends $180 million promoting the use of tobacco products in WA each year Fact 2 The program is based on CDC best practices Fact 2 There are 65,000 fewer youth smokers Fact 2 45 kids start smoking every day in WA Fact 3 Comprehensive youth programs have been effective in other states like Oregon, Florida and Massachusetts Fact 3 Declines in smoking rates have stalled in the past three years. Fact 3 The use of alternative tobacco products such as cigars, flavored cigarettes and cloves has been increasing and needs to be investigated 108 Exercise 2: Using your data to communicate Pick one scenario (A,B,C) on page 64 Work in a small group Use the data in the grid on page 65 Create a SOCO - page 66 Share 109 Exercise 2: Discussion Some SOCOs that we came up with: “The school board needs to take action to prioritize alcohol prevention programs for our middle school children” “Every parent in our community should talk to their kids about drinking and driving” “Funding this program will reduce the excess danger that children in our community face from alcohol” 110 Exercise 2: Discussion, cont. Details (supported by facts in the data) Alcohol use in 8th graders is higher for our children compared to the state Both driving while drinking and riding with a driver who had been drinking are significantly higher for 10th and 12th graders compared to the state Our 8th, 10th and 12th graders are more likely to report their parents did not talk to them about the dangers of drinking alcohol. Our 10th and 12th graders are less likely than children statewide to believe that regular alcohol 111 use is dangerous Potential traps…. Including non-relevant information or providing too much detail (be brief!) Speculating beyond the data or treating differences as significant when they are not Forgetting to double-check your numbers Using too many graphics, or graphics that distort the data Talking about numbers instead of the people they represent 112 Helpful tips… Make your first words count Have someone you trust critique your message for understandability and accuracy Practice saying everything, including the numbers, out loud Practice saying “I don’t know, but I can get back to you about that” Be prepared to discuss the limitations of your data Speak with clarity and compassion 113 Keep “on message” The answer to every question is your SOCO. “That’s an interesting point, but what I’d really like people to know is, [my SOCO]” 114 Remember……. Data are only ONE piece of the puzzle. DATA But sometimes: “Without data, you’re just another person with an opinion” - unknown 115 Final thoughts Don’t flip through your HYS report to find the story, find your story and use data to support it (most of the time) If your story is “we have important data to describe our kids” (overview) then use the local slide sets provided to you Share your HYS results! 116 Why share your results? Awareness of data availability Gain community, school, and youth perspectives Planning: Change perceptions of issues Make informed decisions Needs assessment Evaluation: Establish baselines Monitor and evaluating progress But watch out for unreasonable expectations 117 Obtaining more information Survey participation information and training materials www.hys.wa.gov/ State and county level results, state level online data query, and survey background information https://fortress.wa.gov/doh/hys/ Preformatted fact sheets with HYS results http://www.AskHYS.net 118 Questions about this training? DASA, DSHS, Linda Becker: 360-725-3705, [email protected] Department of Health, Kevin Beck: 360-236-3492, [email protected] OSPI: Dixie Grunenfelder: 360-725-6045; [email protected] CTED, Ramona Leber: 360-725-3033 [email protected] RMC Research Corporation, Eric Einspruch: 1-800-788-1887, www.rmccorp.com 119 Thanks! from the Joint Survey Planning Committee OSPI: Jeff Soder, Dixie Grunenfelder DOH: Kevin Beck, Diane Pilkey, Susan Richardson, Lillian Bensley, Juliet VanEenwyk CTED: Ramona Leber Family Policy Council: Bill Hall DSHS, DASA: Steve Smothers, Linda Becker Liquor Control Board: Tony Masias RMC Research Corporation: Eric Einspruch, Jennifer Lembach Don’t forget to fill out an evaluation! 120