Transcript Have Your Say: Making the Case for Career Development!

How Career Development
Services Are Assessed!
The Canadian research group on measuring
the impact of career development services
(Robert Baudoin, Lynne Bezanson, Bill Borgen, Liette
Goyer, Bryan Hiebert, Vivian Lalande, Kris
Magnusson, Guylaine Michaud, Céline Renald, and
Michel Turcotte)
***Names listed in alphabetical order***
Why this interest?
• International symposia in 1999 and 2001
• The OCED study on career development, 2002
-2004
• The International Centre for Career
Development and Public Policy, 2003
• Canada-wide symposium on career
development, continuous education and human
resource development, 2003
Canadian group - impact of services 05
2
What do we mean by career
development?
• The aim of high-calibre career development
services is to help individuals to:
• become informed about the various training and
work options;
• choose the options that correspond best to their
interests, values, aptitudes and beliefs as they
make occupational choices and identify their
aspirations in an uncertain and changing
environment.
Canadian group - impact of services 05
3
What career development services
contribute
• High-calibre services that can:
• contribute to an individual’s well-being and
overall development.
• stimulate motivation for studies and work
• help maintain good mental health for work and
studies
• create a sound foundation for building effective
skill development and training systems.
Canadian group - impact of services 05
4
Benefits of career development
for society
• Spin-off benefits may include:
• Fewer learners operating in “trial and error” mode
• Higher success rates in programs, courses, and
obtaining certificates and degrees
• Less time on social assistance
• Lower healthcare costs
• Fewer social drop-outs
• Happier employees, less sick leave, higher retention
rates, greater involvement in training and learning
Canadian group - impact of services 05
5
Why is assessment important?
• To obtain funding
• To have a clearer idea of actual
results as opposed to expected results
• To request the investment needed to
achieve the expected results.
Canadian group - impact of services 05
6
Lack of solid data
• At this time, we unfortunately don’t have
much information on how career
development services are assessed.
• Before suggesting ways of solving the
problem, it is crucial to know more about
the current state of assessment practice and
its impact.
Canadian group - impact of services 05
7
Responses to the need
for solid data
• Fall 2004: online survey:
• How do service providers assess the impact of
their services?
• Our goal: To better understand the state of
current practice in Canada with respect to
documenting the results of career development
activities and services.
Canadian group - impact of services 05
8
The purpose of the survey:
4 main questions
1. How important is it to assess the impact of
career development services?
2. How can the impact of these services be
determined?
3. What are the results from such services?
4. How can these results be measured?
Canadian group - impact of services 05
9
Data collection
•
•
•
•
Questionnaire for agencies (online)
Questionnaire for practitioners (online)
Questionnaires in both official languages
Telephone interviews with employers and
policymakers
Canadian group - impact of services 05
10
What we mean by “results”
• A reaction or specific product that results from an
activity
• A result can be one or other of the following :
• Changes in the client’s skills: demonstrated by
changes in the client’s knowledge, abilities or
characteristics
• Changes in the client’s situation: for example, job
situation, educational situation, etc.
• General changes for clients and the community:
• Changes in terms of financial situation, social integration,
impact on families, community economic development, etc.
Canadian group - impact of services 05
11
What we mean by “activities”
Any activity carried out with the intention of
helping clients to make changes to their
individual situations – in other words, any
action that produces a result for a client or a
group of clients.
Canadian group - impact of services 05
12
Survey sample
• 173 agencies; 214 practitioners; 9 policymakers and 7
employers
• The types of service offered include all customary services
– individual interviews, group sessions, assessment, LMI
and employment development
• Target age-group: mostly young people and adult jobseekers, followed by young people at the post-secondary
level or finishing high school
• Type of agency: mostly NGOs (50.3%), followed by noneducational provincial organizations (23.3%)
• Agency size: 41% with over 10 employees; 20% with 6-10
employees; and 39% with fewer than 6 employees
Canadian group - impact of services 05
13
What importance is put on measuring the results
or effectiveness of the services provided?
97.6% of the agencies responded that it was
“quite important” (24.4%) or “very
important” (73.2%)
84% of the agencies report on the impact or
effectiveness of their services.
Canadian group - impact of services 05
14
There is a significant relationship between the type
of service provider and the importance placed on
measuring impact and determining its actual value
• Agencies: Primary and secondary schools
are less inclined to respond “very
important” and post-secondary institutions
are less inclined to respond “quite
important.”
• Practitioners: Practitioners in educational
institutions are less inclined to respond
“very important,” and the NGOs are more
inclined to respond “very important.”
Canadian group - impact of services 05
15
Agencies and practitioners: What are the 3
most important results you report on?
1. Changes in the client’s job or educational
situation;**
2. Development of skills; financial independence,
self-confidence; **
3.
4.
5.
6.
7.
and, to a lesser extent:
Number of clients served
Client satisfaction
Programs completed
Service delivery
Cost/benefit ratio
Canadian group - impact of services 05
16
Agencies and practitioners: Types of
evidence for results
• Assessment/follow-up reports: Reports from client or a
partner; investigations, interviews
• Frequencies: number of clients served/month; number of
clients finding jobs; number of action plans; number of
clients completing programs
To a lesser extent:
•
•
•
•
Observation of changes in the client
Cost/benefit analysis
No difference between agency and practitioner responses
33% of the practitioners did not respond
Canadian group - impact of services 05
17
Comments received
• Assessment is difficult
• Certain results can’t be measured
• Too much importance put on employment
statistics – to the detriment of other types of
results
• There is need for research on career
development services and on specific
populations
Canadian group - impact of services 05
18
What results do you obtain that are not
measured or reported?
• Developing the client’s ability to act
• Skill development (e.g., self-confidence, independence,
changed attitudes towards their future and their
perception of the labour market)
• Financial independence
• Creation of support networks
• More job opportunities and other kinds of opportunities
for clients
To a lesser extent: benefits for the community; client
satisfaction; increase in demand for political lobbying
(agency); program implementation and development
(agency); repeat clients (practitioner); professional
development (practitioner)
Canadian group - impact of services 05
19
What proves that you have achieved these
results?
• Anecdotal evidence
• Verbal reports from clients
• Verbal reports from employers
• Observations
• Observed changes in the client
• Observed changes in the client’s attitudes
• Recurring theme in the data: belief that several of
the results observed or reported are either not
quantifiable or are very difficult to measure (and,
as a result, are not taken seriously)
Canadian group - impact of services 05
20
What difficulties do you experience in proving or
measuring results?
• Complexity
• Difficulty in determining the results of the services
provided
• Belief that certain results cannot be measured
• Assessment considered to be of little importance
• Lack of allotted resources (money and time)
• Little recognition of the importance of collecting solid
data
• Lack of training in evaluation methods
• Difficulty in obtaining feedback from clients
• Loss of contact with clients; clients don’t want to give
feedback
Canadian group - impact of services 05
21
Difficulties (cont’d)
• Lack of uniformity or agreement between the
agencies and the funders on the type of results to
be collected
• Little linkage between service delivery and
assessment (agencies)
• Absence of assessment protocols or formal
processes for completing effectiveness evaluations
• Lack of experience in, and limited access to,
models of best practices (practitioners).
• Rapidity of changes (practitioners)
Canadian group - impact of services 05
22
What things would you like to report on, if
given the opportunity?
• The social impact of services on individuals,
families, the community, and employers
• Development of the ability to act, and client
growth and satisfaction
• Changes in behaviour and attitudes
• Quality of services
• Components of the process; time needed to achieve the
goals
• Longitudinal impact
• Link between local and national results
Canadian group - impact of services 05
23
The other side of the coin
What do policymakers and employers
think?
What do policymakers want?
• Client results
• e.g., client satisfaction, intelligent decisions, and client’s ability to identify
career opportunities.
• Indicators of client results
• e.g., number of jobs obtained, number of successful transitions
• Services that make a difference in client’s lives; the change
would not have occurred without the service provided
• Value added of the services in terms of return on
investment or economic improvement
• Longitudinal studies to make comparisons between those
who receive services and those who don’t
Canadian group - impact of services 05
25
Other information that
policymakers are interested in
• How services are delivered
• Number of clients served; number of places
offered in programs, client assessment
• Level of service provided and degree of
correspondence between level of service
and the client’s expressed need
• Monitoring and assessing contracts
Canadian group - impact of services 05
26
What do employers want?
• Employees who are well-trained,
competent, committed, motivated and
satisfied
• Increased productivity, less employee
turnover, skill-based job descriptions,
improved internal employee mobility
• Feedback from employees and service
providers on what worked and what didn’t
Canadian group - impact of services 05
27
What policymakers and employers
agree on
• Need to collect solid data linking results and
services
• Need to develop better methods for
assessing results.
Canadian group - impact of services 05
28
Next steps
• Develop detailed impact assessment models
• Develop clear, valid and reliable tools
• Create an analytical framework for
comparing data (meta-analysis)
• Create a mechanism for disseminating and
promoting service impact assessment
information.
Canadian group - impact of services 05
29
Thank you!
The Canadian research group on
measuring the impact of career
development services