Transcript Raising the Profile of Career Guidance: The Educational

Evaluation as a Subversive Activity
(Demonstrating the Value of Career Services)
Kris Magnusson
Dean of Education
Simon Fraser University
Bryan Hiebert
Educational Psychology & Leadership Studies
University of Victoria
Overview
Background information
2. Evaluation approach
3. Implementation planning
4. Some sample tools
1.
2
Background and
Rationale
3
4
A Challenge by Canadian Policy
Makers:
“You haven’t made the case
for the impact and value
of career development services!”
A research team formed in 2004 to follow-up
The Canadian Research Working Group for EvidenceBased Practice in Career Development
10 researchers from 7 universities & 1 foundation
5
Why “Subversive”?

The goals of funders of service do not always
align with the goals of service providers
Funders may not recognize legitimate and useful
results of service (e.g., progress towards a goal)

Practitioners often believe that they are too
busy, and that “evaluation” is too complex
 Evaluation is rarely planned at the DESIGN
stage, where it can be most effective; it is
usually “bolted on” to a program
6
How to address the problem
We need an approach that is:
1. Comprehensive enough to include what is needed
2. Simple enough for people to use; and
3. Incorporates evaluation into standard practice
7
What Gets “Subverted”?

Changed attitudes from funders
 Changed practice from service providers
 More evidence for the field to justify the
power and efficacy of career interventions
8
General Approach to
Evaluation
Showing worth, documenting impact,
relating success stories ….
9
Evidence-Based Outcome-Focused Practice
Input  Process  Outcome
Need to link process with outcome
10
Evidence-Based Outcome-Focused Practice
Input

Process

Outcome
Indicators of client change
1. Learning outcomes
• Knowledge and skills linked to intervention
2. Personal attribute outcomes
• Changes in attitudes
• Intrapersonal variables
(self-esteem, motivation, independence)
3. Impact outcomes
• Impact of #1 & #2 on client’s life,
e.g., employment status, enrolled in training
• Societal, economic, relational impact
11
Outcomes of Counselling
1. Client learning outcomes
 Knowledge
 Skills
+
Precursors
Personal Attributes
2. Impact on client’s life
 Client presenting problem
 Economic factors
 Third party factors
12
Precursors
Intervene between learning outcomes
& impact outcomes
 Attitude
 Motivation
 Self-esteem
 Stress
 Internal locus of control
 Belief that change is possible
13
What outcomes are you achieving that are
going unreported or unmeasured?
 Client empowerment
 Client skill development
 personal self-management skills
 Client increased self-esteem
 Client changes in attitudes
 about their future
 about the nature of the workforce




Client knowledge gains
Financial independence
Creation of support networks
More opportunities for clients
These are
legitimate
areas for
intervention
14
Evidence-based Outcome-focused Practice
Input

Process
 Outcome
Activities that link to outputs or deliverables
Generic interventions
 Working alliance, microskills, etc.
Specific interventions
1. Interventions used by service providers
 Skills used by service providers
 Home practice completed by students
2. Programs offered by schools or agencies
3. Involvement by 3rd parties
4. Quality of service indicators
 Stakeholder satisfaction, including students 15
Evidence-based Outcome-focused Practice
Input 
Process
 Outcome
Resources available
1. Staff
 Number of staff, level of training, type of training
2. Funding
 Budget
3. Service guidelines
 Agency mandate
4. Facilities
5. Infrastructure
6. Community resources
16
Inputs
•
•
•
•
•
Number of staff
Staff level of training
% or staff time on
various tasks
Staff: client ratio
Client case load
•
•
•
•
•
Program dollars
Client volumes
Types of client
problems
Adherence to
mandate
Cost-effectiveness
Be realistic about what you can deliver,
given the level of resources you can commit
17
Outcome Focused Evidence-Based Practice
Quality Improvement
Input
Resources

Process
Activities
Counsellor
&
Client

Outcome
Client change
• Knowledge
• Skill
• Attribute
• impact
18
Outcome Focused Evidence-Based Practice
Dynamic and Interactive
Resources
Counselling,
Linking process to outcome
Outcome
Process
19
Intervention Planning & Intervention Evaluation
Intervention Planning Framework
Context:
Client
Needs

1
Client Outcomes
Client
Counsellor
Client
• Knowledge



Goals
Strategy
Strategy
• Skills
• Attributes
• Impact
3
2
20
20
Outcome Focused Evidence-Based Practice
RESOURCES AVAILABLE
Activities that link to outcomes or
deliverables
Generic interventions
Working alliance, microskills, etc.
Specific interventions
Client
 Context
 Needs
 Goals
OUTCOMES
 Context: Structure of
opportunity
 Staff: Number of staff, level
of training, type of training
 Funding: Budget
 Service guidelines
 Facilities
 Infrastructure
 Community resources
Strategies linked to specific client problems
(stress, grief, depression, career, etc.)
Client home practice
Other
Programs & Workshops
Facilitation guides
Intervention manuals
External Referral
Indicators of client (Learner) change
1.Learning outcomes
Changes in knowledge and skills linked to the program or intervention used
Progress Indicators
End Result Indicators
2.Personal attribute outcomes
Changes in intrapersonal variables e.g., attitudes, self-esteem, motivation, etc.
Progress Indicators
End Result Indicators
3.Impact Outcomes
Changes in the client’s life resulting from application of learning
21
Quality Service Delivery
Accessibility
1.
•
•
•
•
•
2.
Regular hours
Extended hours
Physical accessibility
Resources in alternate format
Ease of access, who can access
5.
Need to negotiate
these with funders
Service delivery
• Client volumes
• Client presenting problems
• Number of sessions
6.
Responsiveness
• Respect from staff
• Courteous service
• Clear communication
System requirements
• Adherence to mandate
• Completion of paper work
Service standards
• Staff credentials,
competencies, resources
Timeliness
• % calls answered by 3rd ring
• Wait time for appointment
• Wait time in waiting room
3.
4.
7.
Overall satisfaction
• % rating service good or
excellent
• % referrals from other clients
22
Negotiated outcomes



If you are lucky, funders might identify personal
attributes [client motivation, improved job satisfaction,
increased self-confidence] or knowledge, or skills, as
accountability indicators
BUT more likely funders will identify impact outcomes
[employment status, enrolment in training, reduced # of
sick days, increased productivity, etc.] or inputs [client
flow, accessibility, timeliness of paper work, etc.]
So service providers need to identify the knowledge,
skills, personal attributes that will produce the impacts
and negotiate these as accountability indicators
Be careful what you promise to deliver
BUT
DELIVER WHAT YOU PROMISE
23
Outcome Focused Evidence-Based Practice
Input  Process  Outcome
Need to link process with outcome
1. What will I do?
2. What are the expected client changes?
 What do I expect clients to learn?
 What sorts of personal attributes do I want my
clients to acquire?
 What will be the impact on their lives?
3. How will I tell?
Professional Practitioner
24
Model Summary:
Integrating Evaluation into Practice
1.
Evaluation is part of service delivery
Not bolted on the side of service
2.
Service = Intervention + Outcome
25
Implementation
Planning
26
Integrating Evaluation into Practice
Understand Service Foundations &Client Context
2. Describe Desired Outcomes
3. Describe Core Activities
4. Select Measures and Scales
5. Collect Evidence
6. Work with the Data
7. Report Results & Market your Services
1.
27
Understanding Service Foundations &
Client Context
What factors outside of the intervention might
have an impact on the results?
 Nature of services
 Context of service
Name of Program or Service:
Context
Social
Cultural
Political
Economic
Spiritual
Gender
Other
Factors that may Affect Results
Describing Desired Outcomes

What do we want to achieve?
 Type of client change being sought?
Be careful to avoid attempting to do too much
in too short a time frame
29
Describe Core Activities

What do we (service provider and client)
need to do to achieve the outcomes?
 Link services to outcomes
Outcomes
Service-outcome planning grid
Learning
Career Decision Making
Processes
Work-specific skills enhancement
Work search
Job maintenance
Career-related personal
development
Other
Attributes Impact
Other
Select Measures and Scales

What will be the indicators of success?
 Process factors
How well did the service provider deliver the service
as intended?
How well did the participants follow the program as
intended?

Outcome factors
Indicators of progress + ultimate indicators of success
 Knowledge
 Skills
 Personal attributes
 Impacts
31
Collect Evidence

How do we collect evidence most efficiently?
Process assessments should not take more than 5
minutes to complete
 Outcome assessments should not take more than
10 - 15 minutes to complete
32
Work with the Data

How do I make sense of the data I have
collected?
Frequency counts and percentages
Mean scores
Measures of association
In most cases,
sophisticated statistics are not necessary
33
Report Results & Market your Services

How do we use the data to convince others?
Work from the macro to the micro
Demonstrate movement in your results
Ensure that decision-makers actually
see the results of your work
Tell the people who need to know
in language that they can understand
34
Points to Ponder

Consider these Assessment Alternatives
 For group (workshop) interventions
 For individual interventions

Not saying these are better (or worse) than what
you are currently doing
 Only encouraging you to consider other alternatives
35
Examples of Subversive
Tools and Approaches
36
Assessment as Decision Making (vs. Judgement)
Please use a two-step process
1. Would you say that your level of mastery of
the attribute under considerations is
unacceptable
00
1.
1
acceptable
2
3
44
Then assign the appropriate rating





0 = really quite poor
1 = just about OK, but not quite
2 = OK, but just barely
4 = really very good
3 = in between barely OK and really good
37
Points To Ponder
Trustworthiness …
Inter-rater reliability
 Would 2 different people give the same ratings?
Intra-rater reliability
 Would the same person give the same rating on 2
different occasions?
More choice alternatives
is not necessarily better
38
Problem with skill self-assessment

Participants asked to rate their skill
(or knowledge) before and after a program
 Often, pre-workshop scores are high
and post-workshop scores are lower
 People find out as a result of the workshop that they knew
less than they thought or had less skill than they thought
 Based on the new awareness, post-scores are lower
People don’t know what they don’t know
 How can we get around this problem?

39
Assessing Learning & Attribute Outcomes
Post-Pre Assessment

We would like you to compare yourself now and before
the workshop. Knowing what you know now, how
would you rate yourself before the workshop, and how
would you rate yourself now?
 Please use a two-step process:
 Decide whether the characteristic in question is
acceptable or unacceptable, then
 assign the appropriate rating
unacceptable
00
1
acceptable
2
3
44
40
Orientation Workshop: Summative Evaluation
Regarding the Primary Objectives of
this workshop, and knowing what you
know now, how would you rate
yourself before the workshop, and
how would you rate yourself now?
Before
After
Unacceptable
Unacceptable
Acceptable
Acceptable
0
1
2
3
4
ave
0
1 2
3
4
ave
1 Clear understanding of basic
career development theory
6
5
11
3
1
1.5
0
0
0
10
16
3.6
2 Knowledge about the factors that
contribute to (or interfere with)
people’s career development
4
10
6
5
1
1.6
0
0
1
6
19
3.6
7
11
3
4
1
1.3
0
0
2
6
18
3.6
4 Tools for demonstrating the value
of careers guidance & counselling
8
5
4
5
1
1.4
0
0
1
5
17
3.3
Awareness of the importance of
career-life planning in TVET
6
5
6
5
1
1.7
0
0
0
3
20
3.4
8
9
4
4
1
1.4
0
0
2
6
18
3.5
3 Knowledge regarding basic skills
used in career-life planning
5
Repertoire of practical tools and
6 approaches for facilitating career
development
Orientation Workshop:
Summative Evaluation Results

156 ratings (6 questions times 26 people):
 84 (54%) ratings were unacceptable before the workshop
 0 ratings were unacceptable after the workshop
 6 (4%) ratings were excellent before the workshop
 108 (69%) ratings were excellent after the workshop

Mean scores before and after the workshop
 Before, all were unacceptable (<2)
 After, all were more than minimally acceptable (>3)
42
Orientation Workshop: Formative Evaluation
For each component of the workshop listed
below, please assess how useful
that component was for you.
Unacceptable
Acceptable
0
1
2
3
4
Ave
1.
General Model: Road Map
--
--
1
10
15
3.5
2.
Exploring the Context
--
--
2
9
15
3.4
3.
Factors Influencing Career Plans
--
--
--
8
18
3.7
4.
Personal Career Line
--
--
1
12
13
3.5
5.
Clarifying Roles (advising, guidance, counselling)
--
--
--
5
21
3.8
6.
Assets and Resources
--
--
3
8
15
3.5
7.
Skill Framework for service providers
--
--
2
6
18
3.6
8.
Group process strategies
--
--
--
5
16
3.6
9.
Skill Practice
--
--
--
9
17
3.7
10. Demonstrating value (evaluation)
--
--
--
8
16
3.7
11. Infrastructure
--
--
3
13
10
3.3
12. Action planning
--
--
--
7
19
3.7
Process Results & Formative Feedback
Completion Rating
Usefulness Rating
Activity
UR
Mean
0
1
2
0
1
2
3
4
Big Picture
0
2
27
0
0
2
15
11
3.3
De-Motivators
0
3
26
0
0
1
16
11
3.4
Career Options
1
10
18
0
1
4
17
6
3.0
Core Motivators
0
2
27
0
0
1
11
16
3.5
Reality Check
1
8
20
0
1
1
16
9
3.2
Framework
1
4
22
0
0
5
7
15
3.4
Mean
Completion = 1.80
Usefulness = 3.30
Big Picture, Core Motivators, De-Motivators were most engaging & most useful.
Possible Career Options and Reality Check were least engaging & least44
useful,
perhaps because participants has done similar exercises before.
Post-Pre Assessment:
Bridging Health Care with Self-Care
Knowledge of stress and stress control
2. Knowledge of how to manage personal change
3. Knowledge of nutrition and nutrition control
-------------1. Level of stress
2. Level of nutrition (high=healthy)
3. Level of fitness
4. Confidence in ability to manage personal change
1.
Presenting the results
45
Participant Self-Assessed Change
Stress & Stress Control
Knowledge
Level
Response category
Start
Finish
Start
Finish
very low
17%
0%
4%
9%
somewhat low
30%
0%
4%
35%
neither high nor low
26%
0%
13%
26%
somewhat high
21%
70%
39%
26%
very high
4%
30%
39%
4%
46
Post-Pre Results
Bridging Health Care with Self-Care
Stress & Stress Control
Start of Program
End of Program
Vertical axis indicates % of participants
47
Informal Evaluation
of
Headache, Pain,
and
Related Affective States
48
Self-Monitoring Headache
0 - No headache
1 - Low level,
only enters awareness when you think about it
2 - Aware of headache most of the time,
but it can be ignored at times
3 - Painful headache, but still able to continue job
4 - Severe headache,
difficult to concentrate with demanding tasks
5 - Intense incapacitating headache
49
Headache Monitoring Grid
Level
5
4
3
2
1
0
06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 01 02 03 04 05
(time of day)
Before treatment
After treatment
50
Headache: Indicators of Success
Measure
Beginning
End
4.1
0.6
Number of intense (level 4 or 5 )
headaches
3
0
Hours of intense (level 4 or 5 )
headache
13
0
Hours headache free
0
10
Average headache level
51
Self-Monitoring Stress
0 - No stress
1 - Low level, only enters awareness when
you think about it
2 - Aware of stress level most of the time, but
still able to stay focused on job
3 - Very stressful, beginning to interfere with
what you are doing
4 - Severe stress, creates major interference
with what you are doing
5 - The most stress you ever feel
52
Self-Monitoring Confidence
0 - No confidence at all in your ability to handle the
situation
1 - Low confidence, but you might get lucky and it
will turn out OK
2 - Sort of confident, you think you might be able to
handle it, but probably not
3 - Pretty confident, you think you have a chance
of this working out OK
4 - Very confident, you’ll probably be able to pull
this off
5 - Completely confident about what you are doing
53
Self-Monitoring Job Satisfaction
0 - No satisfaction at all with your job
1 - Somewhat satisfied, there are some
23-
45-
things about your job that are OK
Satisfied enough – as jobs go, this one
is as good as any
Mostly satisfied, occasionally there are
things about your job that you actually
like
Very satisfied, but there are occasional
things bother you
Completely satisfied with your job,
feeling a sense of self-fulfillment
54
Self-Monitoring Motivation
5 - Very motivated-it's the most important thing for
me to do today
4 - Very motivated, but something might come up
to interfere
3 - Quite a bit motivated, I think I will end up
doing some of it
2 - I sort of care and I might get around to doing
some work today
1 - If I run out of things to do, I'll try looking in
some of my books
0 - I couldn't care less if I ever did my school work
55
Monitoring Your Self-Talk
Con
 
Pro
  





56
Changing Your Self-Talk
Con

Pro








57
Where is This
Leading?
58
To demonstrate value, we need to develop
Culture of evaluation:
We need to reach the state where
 Identification of outcomes is an integrated part of
providing services


Without efficacy data, counselling services are vulnerable
It is in our best interest to gather evidence attesting to the value
of the services we provide
 Measuring and reporting outcomes is integrated into
practice
 Reporting outcomes is a policy priority
 Outcome assessment is a prominent part of practitioner
education
This needs to be a priority in all sectors
59
Professional Identity:
What we do defines who we are

Most practitioners define their job as delivering
services
So … they do not evaluate
the impact of their services on clients

What is counselling all about?
 Needs to include BOTH process and outcome
What will I do to facilitate client change?
+
How well is it working?
60
Multiple Effects of
Relevant Evaluation Practices
1.
On client
 Increased motivation
 Increased awareness of effectiveness
 How am I doing at this?
2.
On counsellor




3.
Increased effectiveness
Increased confidence
Increased motivation
Increased job satisfaction
On agency
 Value of services
61
Don’t worry about getting it right, just start
and improve it as you use it
Small steps are OK
2. Several small steps = one BIG STEP
3. Share your success stories
1.
 with the people who need to hear them,
 in language they can understand
Be persistent
5. Build support for yourself
4.
62
Don’t Ever
Give Up
63
Questions?
Comments?
Kris Magnusson
Bryan Hiebert
Simon Fraser University
University of Victoria
[email protected]
[email protected]
Evaluation as a Subversive Activity
(Demonstrating the Value of Career Services)