Transcript SCPSC 2013 Presentation at ASCA (Hatch

SHIFTS IN BELIEFS USING DATA COMPARISON
FINDINGS OF SCHOOL COUNSELOR BELIEFS ABOUT
ASCA NATIONAL MODEL SCHOOL COUNSELING
PROGRAM COMPONENTS USING THE SCPCS
Trish Hatch, Ph.D.
San Diego State University
Timothy Poynton, Ed.D.
Suffolk University
OVERVIEW OF SESSION
• Activity: What do you think?
• Have school counselors’ beliefs shifted since the introduction
of the ASCA Model?
• How have beliefs shifted for use of data, accountability and
reporting results?
• Which areas are positive?
• Has this affected school counselor behavior?
• What work is still needed?
• Sharing ideas for supporting school counselors who may not
yet believe in the value of data and accountability
SCPSC
• Hatch and Chen-Hayes (2008) published article about school
counselors’ beliefs regarding the ASCA National Model School
Counseling Program Components using the School Counselor
Program Component Scale (SCPCS).
• 2002 Survey administered to over 1,200 ASCA members
• Established psychometric properties of the SCPCS
• Collected national baseline data on school counselor beliefs
about certain program components in the ASCA National
Model prior to its release in 2003.
ACTIVITY #1
WHAT DO YOU PREDICT?
• To what extent do you think school counselor beliefs shifted on
the SCPCS items since the introduction of the ASCA National
Model?
• Note rankings from the 2002 survey are activities that align
with the ASCA National Model.
• They are listed from most important to least important
• What shifts do you predict are significant?
2009 SCPSC INSTRUMENT
• The SCPCS was modified slightly for use in the
2009 study, based on feedback provided by
Professional School Counseling reviewers
• The original 5 point scale, which ranged from 1 to
5 with named anchors at 1 (very important), 3
(moderately important) and 5 (not important) was
modified to a 3 point scale using only the named
anchors of the original scale.
USEABLE SURVEYS
• The initial email invitation yielded 285 usable surveys
(46%)
• Remaining 332 participants completed the survey after
the reminder email was sent.
• The online survey was a multi-page survey which
contained a page with informed consent
SURVEY PARTICIPANTS
•
•
•
•
•
•
•
617 participants
27% (n = 165) elementary school counselors
21% (n = 130) middle/junior high school counselors
32% (n = 196) were high school counselors, 12%
(n = 72) worked at multiple levels
4% (n = 27) school counseling program supervisors
Twenty-seven respondents (4%) did not answer this
question.
DEMOGRAPHICS
• 83% Female (n = 512)
• 15% Male (n = 95) 2% (n = 10) did not answer
•
•
•
•
•
•
•
86% percent of the respondents White (n = 512)
4% identified as Black (n = 23)
4% identified as Hispanic (n = 24)
2% identified as Asian or Pacific Islander (n = 12)
2% identified as multiracial/multiethnic (n = 13)
1% identified as some other race/ethnicity (n = 6)
10 respondents (2%) did not answer this question
MORE DEMOGRAPHICS
• Forty-six percent of the respondents worked in suburban
settings (n = 286), 28% in rural settings (n = 171)
• 24% in urban settings (n = 147)
• 13 respondents (2%) did not answer this question
• Average amount of experience working as school
counselor reported by respondents was 8.1 years (SD = 7.1).
Q #1 FACTOR ANALYSIS
• To assess the possible impact of changing the scale on the
factor structure, reliability, and validity of the SCPCS,
several analyses were performed in a manner identical to
those reported by Hatch and Chen-Hayes (2008).
• Specifically, internal consistency estimates and a principal
components analysis (PCA) were calculated in a manner
identical to the initial SCPCS study.
FACTOR ANALYSIS
• Compared Factor Structures 2002-2009
• 18 of 19 SCPSC items loaded on same factors
• Previous item not retained in FA in 2002 (#19) loaded on
administrator support.
• Eigenvalues of three of the four factors in the 2009 data
were greater than 1.00
• The amount of variance each factor explained differed from
the 2002 data
SUMMARY RESULTS
• Despite the differences between the two studies (scale change
and participation due to on-line administration), analyses
revealed four distinct and internally consistent factors
evidencing construct validity.
• The factor structure of the SCPCS remained intact in the
second study.
• Internal consistency estimates exceeded acceptable
requirements for the scale and each of the four subscales.
• Thus, the four subscales were found to be consistent and
reliable.
EIGENVALUES
•
•
•
•
•
9.94 (Use of Data for Accountability, 52.60%)
1.35 (Use of Data for Program Planning, 7.10%)
1.14 (Administrator Support, 5.99%)
.89 (Mission, Goals, and Competencies, 4.71%)
Majority of the variance in the 2002 data was explained by
the Use of Data for Program Planning factor (43.50%)
• Majority of the variance in the 2009 data was explained by
the Use of Data for Accountability factor (52.60%).
INTERNAL CONSISTENCY
(CRONBACH’S ALPHA)
•
•
•
•
•
Use of Data for Program Planning = .87 (.82);
Use of Data for Accountability = .91 (.80);
Administrator Support = .85 (.78);
Mission, Goals, and Competencies = .84 (.86)
All subscales evidenced acceptable reliability
characteristics using the commonly accepted criterion
provided by Nunnally (1978).
ANOVA (SUBSCALE MEANS)
•
•
•
•
•
•
•
•
Statistically significant, F (3, 606) = 27.78, p < .001, η2 = .12, a small effect size.
Post-hoc tests with a Bonferroni correction revealed differences among all
possible pairwise comparisons:
Mission, Goals, and Competencies (M = 1.32, SD = .41)
Use of Data for Program Planning (M =1.45 , SD = .46 )
Use of Data for Accountability (M = 1.37, SD = .46)
Subscales all significant at the p < .001 level.
Observed differences between the Administrator Support (M = 1.41, SD = .44)
and Use of Data for Program Planning subscales was significant (p = .033)
Difference between the Administrator Support and Use of Data for
Accountability did not reach statistical significance (p = .074).
Q #2 RANK ORDER
• To facilitate analysis of changes in perceived importance of
the items over time, individual items were rank-ordered
based on observed mean scores.
• The use of rank-ordered lists of the 2002 and 2009 data
facilitates comparison between the two administrations in
spite of the change in the scale
• Allows conclusions to be drawn about the perceived
importance of each item relative to all other SCPCS items.
RANK ORDER (PERCEIVED IMPORTANCE)
• To assess the perceived importance of each individual
item in a comprehensive school counseling program,
descriptive statistics (mean and standard deviation)
were calculated.
• The SCPCS item means and standard deviations are
presented in rank order in Table 2, along with the rank
of the item based on the 2002 data.
CHANGES IN RANKS OF BELIEFS OVER TIME
GREATEST RANK ORDER SHIFTS
(TOWARD VERY IMPORTANT)
• 13th  6th use of data to measure the outcome results
of the school counseling program
• 17th 10th using various data student data to identify
gaps
• 10th 5th use of data to demonstrate the impact of
the school counseling program on student
success in school
DISCUSSION
POSSIBLE EXPLANATIONS
• Are you SURPRISED by the findings?
• What do YOU think contributed to these findings?
• If not what expected, what factors contributed to your
differing perspective?
DISCUSSION
• Notable shift in Use of Data for Program Planning factor
accounting for the majority of the variance in 2002 to
the Use of Data for Accountability factor accounting for
the majority in 2009.
• Consistent with the 2002 findings, the Use of Data for
Program Planning item “included five items related to
using data to target interventions and identify program
foci” (Hatch & Chen-Hayes, 2008, p. 38).
TOP RANKED:
DEVELOPING GOALS FOR PROGRAM
• Setting goals within the school counseling program is a
required component of the Recognized ASCA Model
Program (RAMP) (ASCA, 2012a)
• SMART goals are included in ASCA’s Making Data Work
(Young & Kaffenberger, 2009).
• Setting program goals has now been added as a new
topic in the foundation section of the ASCA National
Model (2012b).
SIX TOP RANKED ITEMS (2009)
CONSISTENT WITH ASCA MODEL
1. Develop goals for the counseling program
2. Write a mission statement or philosophy
3. Utilize school-wide and student data to design new counseling
activities
4. Identify specific student competencies to which the school counseling
program, curriculum or activities contribute or align
5. Use data to demonstrate the impact of the school counseling program
on student success in school
6. Use data to measure the outcome results of the school
counseling program
GREATEST POSITIVE RANK ORDER
SHIFTS WHAT DOES IT MEAN?
• Have school counselors begun to prioritize the value,
importance, and necessity of using data in their school
counseling programs?
• Provides positive feedback for ASCA and national
leaders in school counseling who have lamented the
paucity of outcome research and the lack of school
counselor engagement in data driven activities and
accountability (it’s a start)
GREATEST POSITIVE RANK ORDER
SHIFTS WHAT DOES IT MEAN?
• Two items moved up 7 RANKINGS!
o Data to identify gaps
o Data to measure impact
• Shift aligns with consistent requests for school counselors
to do more than use data to design and prepare program
activities .
• School counselors have been encouraged to improve
efficiency and effectiveness and to promote the schools
counselors’ value as worth its cost or resource
RANK ORDER FINDINGS ALIGN WITH
NATIONAL REFORM EFFORTS
• Aligns with work of Education Trust Transforming School
Counseling Initiative (TSCI)
• Central tenets of ASCA National Model (Closing the Gap
Action Plans)
• The College Board’s National Office for School Counselor
Advocacy (NOSCA), TSCI
• National Association of College Admissions Counselors
(NACAC)
ASCA 3RD EDITION (161 PAGES)
•
•
•
•
•
Data mentioned 124 times
Gaps is mentioned 60 times.
Results (124 mentions)
Accountability (48 mentions)
ASCA Model incorporates language from ethical guidelines
(ASCA, 2010) stressing the responsibility of school counselors
to use equity-based data to identify, address, and resolve
attainment, achievement, and opportunity gaps.
pretty irrefutable
FUTURE RESEARCH
• School counselors who responded ASCA members who
have received firm directive from the professional
association to answer the question: “How are students
are different as a result of what school counselors do?”
• Future research with a more diverse sample is needed
to understand more about the beliefs of non-ASCA
member school counselors regarding data for planning
and accountability.
GREATEST NEGATIVE RANK ORDER
SHIFTS
• 8th  14th Delineating non-school counselors related
activities
• 12th  18th Adhering to non-school counseling
activities
• Is there a relationship between school counselor beliefs and
behaviors regarding accountability and non-school counselor
activities
• Are school counselors who produce student and program
results less likely to be asked to perform non-school
counselor activities?
CONSULTING “MORE” WITH ADMIN.
SHIFTED -4 FROM 3RD TO 7TH
• Are SC’s more competent and confident with regard to
improving their school counseling programs and thus,
believe there is less need to consult with
administrators?
• Have they already begun to consult “more” regularly
with administrators and thus, do not necessarily need
to consult even “more”?
PRESENTING RESULTS STILL LOW
• Consistently low ranking on both surveys in relationship to other
items (16th & 15th)
• Despite strong and consistent messages have been sent for school
counselors to share their results with stakeholders (Hatch, 2008;
Dimmitt, et al., 2007).
• Decisions on the employment of counselors can be based on the
results they produce (Duarte & Hatch, in review; Fladager, 2012).
• Further research is needed to understand factors contributing to
lack of perceived importance in relationship to other items.
LIMITATIONS
• Not longitudinal
• Email to on-line not paper pencil
• Problems with email:
o Did they ever get it? Sent to Junk?
o Or not respond?
RESEARCH NEEDED ON…
•
•
•
•
•
83% of the respondents in both surveys identified as female.
2009 slightly more diverse (86% white compared to 92% in 2002).
Largest increase Hispanic/Latino: doubled 2% in 2002 to 4% in 2009.
African American respondents increased minimally from 3% to 4%.
Compared to College Board Study: 78% female, 10% African
American, and 15% percent Hispanic or Latino (Bruce & Bridgeland,
2012).
• Future research is encouraged to compare national school counselors’
demographics to membership in professional associations.
AND FINALLY…
• 3rd edition ASCA Model reflects current practice
• School counselors and Common Core
• Future research needed to access shifts
2002  2009  2016
What do YOU think the NEXT a shifts will be?
SHARING IDEAS
• Share ideas for supporting school counselors who may
not yet believe in the value of data and accountability
• Ideas to support their future growth in this area…
PRIMARY REFERENCE
(EXTENSIVE LIST AVAILABLE UPON REQUEST)
•
Hatch, T., & Chen-Hayes, S. F. (2008). School counselor beliefs about
ASCA National Model school counseling program components using
the SCPCS. Professional School Counseling, 12(1), 34-42.
•
Hatch, T. & Poynton, T. & Perusse, R. (in review). Shifts in beliefs
using data Comparison: Findings of School Counselor Beliefs about
ASCA National Model School Counseling Program Components
Using the SCPCS
THANK YOU
• Tim Poynton [email protected]
• Trish Hatch [email protected] (until July 1, 2013)
• Contact Tim if you have questions (Thanks Tim!) 