Transcript Document

1
ACCOUNTABILITY: ASSESSING
NEEDS, DETERMINING OUTCOMES,
AND EVALUATING PROGRAMS
Chapter 5
Chapter 5
2
ACCOUNTABILITY
•
•
Accountability involves responsibility for professional actions. In the
context of a school counseling program, this may involve:
1) Identifying and collaborating with stakeholder groups
2) Collecting data and assessing needs of students, staff, and
community
3) Setting goals and establishing objectives based on data
4) Implementing effective interventions
5) Measuring the outcomes of interventions
6) Using results for program improvement
7) Sharing results with major stakeholders
Numerous contemporary factors have called for greater accountability
from professional school counselors and an increased focus on
assessment.
Chapter 5
3
5 QUESTIONS TO ADDRESS ACCOUNTABILITY IN THE
SCHOOL COUNSELING PROGRAM
Is a comprehensive, standards-based program in place?
(program evaluation or audit)
What are the needs of the school’s student population when
compared to these standards? (needs assessment)
What services were implemented to address the identified
needs and standards? (service assessment)
What was the result of the implemented services? (results or
outcomes studies)
How well is the professional school counselor performing?
(performance evaluation/appraisal)
Chapter 5
ADVANTAGES & DISADVANTAGES
OF ACCOUNTABILITY STUDIES
4
ADVANTAGES
• Data is better than perception when
guiding decision-making.
• Accountability helps demonstrate
necessity, efficiency, and effectiveness
of program.
• Studies can help identify professional
development needs.
• Professional school counselors can
network to share program results.
• Conducting accountability studies is a
professional responsibility and
demonstrates commitment to
improvement.
Chapter 5
DISADVANTAGES
• Outcome measures take some training and skill
to develop.
• Quality outcome research and evaluation
takes time and resources.
• Many do not understand the nature and
purpose of accountability because of
misperceptions or previous “bad” experiences.
• Data is sometimes “over interpreted” or given
undue meaning.
• Comprehensive evaluations are seldom
conducted, leaving the “big picture”
incomplete.
5
ACCOUNTABILITY: EVALUATING PROGRAMS AND
ASSESSING OUTCOMES
• It is a professional and ethical responsibility for
professional school counselors to evaluate their
program to ensure that the services provided to
students, school personnel, and parents are of high
quality and effective in meeting intended needs.
• Without accountability data to back up service
provision, school counseling services could be
among the first “nonessential services” to go during
budget cutbacks.
Chapter 5
6
NEEDS ASSESSMENT
• Purposes of needs assessment:
• Helps professional school counselors understand the
needs of various subpopulations of a school
community.
• Helps establish the priorities that guide the
construction of a comprehensive developmental
school counseling program.
• Assesses not only what currently is, but what should
be.
Chapter 5
7
DATA DRIVEN NEEDS ASSESSMENT
• Data-driven decision-making deals with real needs and impact,
and begins with an analysis of school-based performance data.
• Aggregated data: results are lumped together to show total
grade level or school-wide (average) results.
• Helpful in understanding how the average students perform
in a given class, grade, or school.
• Tells very little about the diversity of learner performance or
needs.
• Tells nothing about how various subgroups or subpopulations
performed.
Chapter 5
8
DATA DRIVEN NEEDS ASSESSMENT
• Disaggregated data: results are broken down by
subpopulations so that performance differences between
and among groups can be analyzed.
• Analysis usually involves intergroup differences (male vs.
female, race, ethnicity)
• Helps to provide hard evidence of gaps in student
performance, rather than relying on perceptions.
• Provides direction for the types of strategies and
interventions needed to close these achievement gaps.
Chapter 5
INTERPRETING NORM-REFERENCED SCORES:
THE BASICS
9
• Percentile rank:
• Most easily understood by visualizing a line up of 100
individuals all with certain characteristics in common, with the
1st student in the line as the lowest performing student and the
100th student in the line the highest performing.
• A student’s place indicates their relative standing compared
to other students the same age across the country (national
percentile rank).
• Quartile:
• Interpretive statistic which divides the percentile rank
distribution into 4 segments.
• Q1 = 0-25, Q2 = 26-50, Q3 = 51-75, Q4 = 76-100
Chapter 5
INTERPRETING NORM-REFERENCED
SCORES: THE BASICS
10
• Stanine:
• Short for standard nine
• Divides a normal
distribution into nine
segments, the 1st represents
the lowest level of
performance and the 9th
represents the highest level
of performance
• Unlike quartiles, stanines
represent one-half
standard deviation units
and may be comprised of
varying percentages of the
population
Chapter 5
11
PERCEPTIONS-BASED NEEDS ASSESSMENTS
• Professional school counselors are often interested in
what teachers, parents, and students perceive as
primary needs to be addressed in a developmental
way.
Chapter 5
12
FREQUENCY OF CONDUCTING A NEEDS
ASSESSMENT
• It is best to follow a continuous cycle of assessing
programmatic needs.
• Breaking down needs into areas of academic, career, and
personal-social development can guide the rotation of
needs assessments.
• Assessing needs is part of a much bigger endeavor:
implementing curricular changes to continuously improve
the comprehensive developmental counseling program.
Chapter 5
13
POPULATIONS TO BE ASSESSED
• It is most practical and efficient to seek out those stakeholders who are
informed and likely to respond, such as teachers, administrators, students,
and parents.
• Although more difficult to obtain large responses, valuable information
can be garnered from community organizations, local businesses, and
the general public as well.
• When considering the return rate, a higher return rate leads to a lower
sampling error, which leads to greater confidence in the accuracy of the
results.
• The highest priority needs should be those agreed to by all or most
populations assessed.
Chapter 5
DESIGN ISSUES IN AN EFFICIENT NEEDS
ASSESSMENT
14
•
•
Stone and Bradley (1994) recommended seven methods for
determining needs:
1) Questionnaires and inventories
2) Analysis of records
3) Personal interviews
4) Counseling statistics
5) Classroom visits
6) Use of outside consultants
7) Systematic evaluation of the guidance program
What is most important is that the needs assessment uses objective
methods for data gathering and analysis.
Chapter 5
STEPS TO FORM AN EFFICIENT NEEDS
ASSESSMENT
15
Decide what you need to know.
Decide on the best approach to derive what you need to know.
Develop the needs assessment instrument or method.
Enlist the support of colleagues and a few individuals from the
target groups to review and try out items for understanding.
Implement the final version on the target groups.
Tabulate, analyze, and interpret the results.
Translate the results into programmatic goals and objectives.
Chapter 5
DESIGN ISSUES IN AN EFFICIENT NEEDS
ASSESSMENT
16
• It is important to take the population into consideration when designing
a scale:
• For teachers—should ask for grade level, number of students in the
class, etc.
• For parents—should ask for the names of the parent’s children in
case responses require contact by the counselor.
• For students—should ask for student’s grade and homeroom
teacher’s name.
• Questions or response stems should be short, to the point, and easily
understood.
• Reading level of the items should be appropriate for the target
audience.
Chapter 5
RESPONSE FORMATS
17
•
•
•
If the purpose is to determine the importance or frequency of a
potential problem, it is generally best to use a multipoint scale with 3-5
choices such as:
1) Rarely
2) Sometimes
3) Frequently
4) Most of the time
5) Almost always
It is important to provide a clear description of each choice.
Be wary of response choices such as:
• “Never” and “always,” as they are rarely ever the case.
• “Yes” and “no”, as they force an all or nothing response.
Chapter 5
DESIGN ISSUES IN AN EFFICIENT NEEDS
ASSESSMENT
18
• It is important to include a frequency count in a needs
assessment in order for professional school counselors to not
only assess the importance of an issue, but also determine
how many, and which students are in need of a specific
service.
• Computing the results of a needs assessment is probably
best accomplished by assigning a number value to each
response category and averaging all responses for a given
item.
Chapter 5
19
CONVERTING NEEDS TO PROGRAM GOALS AND
OBJECTIVES
• If the needs assessment was designed correctly, translating the results
into goals and learning objectives is relatively easy.
• Step 1: Prioritize the needs in order of importance and their
relationship to existing components of the program.
• Step 2: Match up or translate the needs into goals included in the
national standards.
• Step 3: Operationalize the goals through development of learning
objectives.
Chapter 5
20
PROGRAM (PROCESS) ASSESSMENT
• Program Assessment: a systematic examination of a
program’s content.
• Program evaluation asks 2 questions:
• Is there written program documentation?
• Is the written program being appropriately
implemented?
Chapter 5
21
PROGRAM ASSESSMENT
• A program audit should be conducted at the end of the year
and reports should address:
• Program strengths
• Areas in need of improvement
• Long and short-term goals
• ASCA suggested that program criteria be evaluated on the
following response choices:
• None (not in place)
• In Progress (begun, but not completed)
• Completed (but perhaps not implemented)
• Implemented (fully implemented)
• Not Applicable (when the criteria do not apply)
Chapter 5
22
SERVICE ASSESSMENT
• Service assessments are sometimes requested by guidance supervisors
and demanded by superintendents and school boards to document
how counselors are spending their time.
• Two types are commonly used:
• Event-topic counts
• Involve documenting each time an individual is contacted or
provided with a counseling service and the nature of the topic
addressed.
• Time logs
• Involve documenting and categorizing activities for every minute
of the workday.
• Unfortunately, service assessments give no information about the quality
or effectiveness of counselor interventions.
Chapter 5
23
RESULTS OR OUTCOMES EVALUATION
• Answers the question, “How are students different as a
result of the program?” (Johnson & Johnson, 2003)
• Assessment is not a discrete component, but rather an
integrated part of a continuous process for program
improvement.
• The evaluation cycle:
• QuestionsEvidenceInterpretationConclusions
Change
Chapter 5
24
RESULTS OR OUTCOMES EVALUATION
• Assessment of school counseling
programs IS:
• a way to answer important
program-related questions.
• the responsibility of an
accountable professional
school counselor.
• a cooperative endeavor
with other school counseling
program advisory committee
(SCPAC) members and
stakeholders.
• ongoing and evolving, a
means to a better end better education for all
students.
Chapter 5
• Assessment of school counseling
programs IS NOT:
• the evaluation of an
individual professional school
counselor.
• an assault on the
professional school
counselor’s freedom.
• a mandate for standardized
tests or curricula.
• all figured out.
25
IMPORTANT ASSESSMENT TERMS
• Evaluation: the measurement of worth; indicates that a
judgment will be made regarding the effectiveness of a
program.
• Evidence: any data that will help make judgments or
decisions; can be quantitative or qualitative.
• Formative evaluation: evaluative feedback that occurs during
the implementation of a program; allows corrective action to
occur if a program is shown to be off course.
• Summative evaluation: feedback collected at a specific
endpoint in an evaluation process.
Chapter 5
26
IMPORTANT ASSESSMENT TERMS
• Stakeholder: anyone involved in or potentially benefiting from
the school counseling program.
• Baseline: any data gathered to establish a starting point.
• Inputs: any resources that go into a program.
• Pretest: a measure administered before a program is
implemented.
• Posttest: a measure administered after the program or
intervention has been completed.
Chapter 5
27
SOURCES OF EVIDENCE
People
• Students
• Teachers
• Staff
• Administration
• Parents
• Employers
• Graduates
• Community resource
agents
Chapter 5
Products
• Portfolios
• Performances
• Use of rating from external
examiners
• Observations
• Local tests
• Purchased tests
• Student self-assessments
• Surveys
• Interviews
• Focus groups
• Student work
PRACTICAL PROGRAM EVALUATION
CONSIDERATIONS
• To be of practical value, assessment must be connected to real program concerns as
well as the core values of the school or program.
• Suggestions:
• Avoid overwhelming the data collectors.
• Focus on only one or several important questions at a time.
• Always select measures that will yield reliable and valid scores for the purposes
under study.
• Involve the relevant stakeholders.
• Use a variety of approaches.
• Do not reinvent the wheel—use what you are already doing to generate useful
data.
• Don’t be afraid to call upon outside experts to consult on the development and
evaluation of a program.
• Start small and build upon what is found to work.
• Remember: different can still be effective.
Chapter 5
28
29
AGGREGATED OUTCOMES
Objective 1
Outcome 1
Objective 2
Objective 3
Outcome 2
Objective 4
Objective 5
Outcome 3
Objective 6
Chapter 5
Standard 1
• Aggregation can be a
valuable tool in evaluating
how well school counseling
programs meet higher level
standards, such as the ASCA
National Standards.
• Curriculum begins with a
statement of standards which
are further described through
a series of outcomes, which
are broken down into
measurable objectives.
30
DESIGNING OUTCOME STUDIES
• In most instances, professional school counselors should measure
outcomes or results by designing a research-type study.
• Answering the following questions can help the professional school
counselor determine which research design to use:
• Has the treatment already been implemented?
• Can I randomly assign participants to treatment conditions?
• Can I conduct (one or several) pretests, posttests, or both?
• True experimental designs involve randomization of participants and
allow causative conclusions to be reached.
• Professional school counselors need to select or design an outcome
measure with sufficient sensitivity and direct applicability to the issue
being studied in order to receive significant results.
Chapter 5
31
COMMON DESIGNS USED FOR
OUTCOMES RESEARCH
• Pretest-posttest single group design
• Case study
• Static-group comparison
• Two-sample pretest-posttest design
• Nonequivalent control group design
• Time series design
• Randomized pretest-posttest control group design
• Randomized post-test only control group design
Chapter 5
32
PRINCIPLES TO CONSIDER WHEN DEVELOPING AN
OUTCOME MEASURE FROM SCRATCH
• Use simple language.
• Ask only about things that the
respondent can be expected
to know.
• Make the question specific.
• Define terms that are in any
way unclear.
• Avoid yes-no questions.
• Avoid double negatives.
• Don’t ask double-barreled
questions.
Chapter 5
• Use wording that has been adopted
in the field.
• Include enough information to jog
people’s memories.
• Look for second-hand opinions only
when first hand information is
unavailable.
• Be sensitive to cultural differences.
• Learn how to deal with difficult
respondent groups.
33
COMMON DATA COLLECTION
METHODS
• Interviews
• Observations
• Written questionnaires, surveys and rating scales
• Program records and schedules
• Standardized and educator-made tests
• Academic performance indicators
• Products and portfolios
Chapter 5
34
ACTION RESEARCH
• Action research allows professional school counselors to focus on
changing social, ecological, or client conditions in particular
situations by creating a study and intervention to explore and solve
a particular problem, usually in the client’s environment.
• Involves the analysis of data to improve practice and solve
practical problems.
• Advantages to action research: It requires minimal training, helps
develop effective, practice-based solutions for practical problems,
and creates a collaborative atmosphere where professionals work
together to address and improve conditions affecting students.
Chapter 5
35
ACTION RESEARCH
• Steps to developing an action research plan:
• Identify and clarify the research problem
• Gather data (should be gathered using a specific or
particular group of individuals)
• Analyze and interpret data
• Create an action plan (should support and validate
gathered data)
• Evaluate and reflect (critically and periodically
evaluate the plan’s effectiveness)
Chapter 5
36
REPORTING THE RESULTS
• It is important that professional school counselors involve the school
counseling program advisory committee (SCPAC) in every step of
the report writing process.
• A comprehensive report may be helpful for analysis by the SCPAC,
however a 1-2 page executive summary should also be prepared
for administrators, system administrators, and the school
community.
• After results have been reviewed by the SCPAC, professional school
counselors, and administration, it is important to release the results
of the program outcomes to relevant stakeholder groups at regular
intervals.
Chapter 5
37
REPORTING THE RESULTS
• Possible ways to disseminate the school counseling program results:
• Written report
• Verbal presentation
• Multimedia presentation
• Journal articles
• Web page
• Videotape
• Posters
• E-mail
• Text messaging
• Newspaper article
Chapter 5
38
PERFORMANCE APPRAISAL
• The effectiveness of a school counseling program often
relies on the competence and efficiency of its
implementers.
• The services provided by, and responsibilities of, the
professional school counselor are the most complex of any
school employee.
• It is best practice to have counselor supervisors coordinate
in the appraisal of counselor performance.
• Criteria for any evaluation always should be available well
in advance.
Chapter 5
39
PERFORMANCE APPRAISAL
• Most professional school counselor performance
appraisals are comprised of a rating system that
aggregates or averages responses across a variety of
categories or work-related responsibilities.
• The primary focus of the appraisal system should be to
enhance the quality and competence of all those
evaluated.
• It is important that professional school counselors be
held to performance standards such as those provided
by the ASCA National Model (2005).
Chapter 5