Transcript Slide 1

ASSESSING THE IMPACT OF LMI:
PRELIMINARY RESULTS OF PHASE TWO
Research Team
Canadian Research Working Group in Evidence-Based
Practice (CRWG),
Canadian Career Development Foundation (CCDF)
In partnership with
New Brunswick Post-Secondary Education & Labour,
Saskatchewan Advanced Education, Employment &
Immigration
1
Overview
•
Provide background on the project
•
Share results
•
Hear the perspectives of the research
partners
•
Consider next research steps
2
Our Research was an Important
Step Forward
• Most LMI research focuses on usability of products
–
Readability
–
Accuracy of information
–
Easy to access
–
Amount of use
–
Most research is with students; very little with adults
• Several questions remain unanswered
–
How do people use LMI?
–
What prompts them to make an action plan
+ implement it?
–
What (if any) assistance would be helpful?
3
•
There is very little literature about the
direct contribution of LMI to
employment/career decision-making
•
This research project directly
addressed this question
Savard & Michaud, The Impact of LMI on Career Decision-Making
Process: Literature Review, FLMM, 2005
4
Research Questions
• If client needs are assessed and clients are
given LMI consistent with their needs,
•
• To what extent does assistance by a service
provider enhance their effective use of LMI?
OR
• To what extent is independent self-help a
sufficient process for clients to use LMI
effectively?
5
Research Questions
 What type of assistance in finding and
using LMI, provided to clients with jobsearch and decision-making needs,
leads to what kinds of outcomes?
 How do clients process the LMI they
access and how do clients use that
information to create a plan for action?
6
Our Approach was Unique
• We used real frontline counsellors with real “adult”
clients in their customary settings – not counselling
interns and students in an academic setting.
• We based our experimental design process on the
existing service delivery processes used by the
participating agencies so the processes could be
incorporated with little effort if research results
proved positive.
• We delivered interventions that isolated the effects of
LMI on decision-making – separating it from other
interventions such as career counselling or job search
workshops that usually subsume LMI.
7
Method
• We reviewed current practices to
determine:
– The current process for identifying
client service needs
– “Favourite” LMI resources
– Access to the Resource Centre
8
Favourite LMI Resources
• We prepared “guided” LMI
packages
– Career Decision Making:
• Know yourself
• Know the Labour Market
• Put it all Together
– Job Search:
•
•
•
•
Check for “Fit”
Get Ready
Search for Work
Get a Job
9
E
M
P
L
O
Y
A
B
I
L
I
T
Y
D
I
M
E
N
S
I
O
N
S
CCDF, 2010
10
Checklist for Counsellor
STEPS
STEP 2: Gather Employability Information
 Gather data on job readiness issues (financial needs, personal needs,
motivation, ability to keep a job)
 Gather employability information relevant to Career Decision Making
o Client’s self-knowledge (abilities, interests, values, personal
characteristics)
o Client’s knowledge of career options
o Client’s ability to research options
o Client’s knowledge of local labour market opportunities
o Client’s career vision/employment goal
 Gather employability information relevant to Job Search
o Clarity of client’s employment goal
o Client’s need for appropriate job search tools (resume, references,
professional pitch)
o Client’s ability to search for work (using networks, job interview skills)
o Client’s ability to assess employment possibilities
 Verify client perspective
 Obtain agreement on identified needs
Not
done
Sort of
done
Done










































11
Another protocol to follow
STEP 2: Provide information and advice by using some of the interventions listed below:
(It is not expected that you will do all of these with every client, but we want to know what
you did cover with each client. Check off as many as apply)
1st AIS Did you:
1. help clients find a resource to link their interests and/or skills to job/career

requirements
2. help clients link their interests and/or skills to job/career requirements

3. help clients find meaningful Labour Market Information

4. help clients interpret/understand Labour Market Information

5. help clients find any information that applies to their own career/job search
(companies, employers, job fairs, job banks, openings/closings, industry

information, job descriptions, wages, etc.)
6. help clients interpret any information so that it meant something to their own

career/job search

7. help clients find resources on job search methods (resume, interview skills, etc.)



8. give clients advice on job search methods (resume, interview skills, etc.)
9. help clients to find decision-making tools
10. help clients to use decision-making tools
12
Even the Client had a Checklist
• From the list of LMI resources they had
in their LMI package, they had to check
which they used and how many times.
Example:
– Career Cruising
– Job Bank
– Job Futures
• And give names of any other sources
they used
13
Intervention
All participants in the study:
 Received a needs assessment interview &
completed an initial, pre-program survey
 Received an LMI package specific to their
identified employability need
 Were randomly assigned to either a selfdirected intervention delivery method or an
assisted self-directed intervention delivery
method.
 Were given an orientation to the Resource
Centre which they could freely use on their
own
 Completed a post-pre survey
 Received a cash honorarium and certificate of
participation
14
Intervention (cont)
In addition, the assisted self-help clients
received:
 Two additional AIS interviews (20-30 minutes)
focused on helping them understand, interpret and
apply the LMI to their own situations and /or
access additional LMI
15
Experimental Conditions
• Intervention = a Career Decision-Making
or a Job Search LMI Package
• Delivery =
Independent Self-directed
Assisted Self-directed
Resource Centre only plus
independent research
Resource Centre and
independent research
PLUS 2 follow-up
Information and Advice
Interviews with a counsellor
• In two provinces: New Brunswick and
Saskatchewan
16
Intervention
Research Design
Job Search
CDM
Time 2:
After
Time 1:
Before
Delivery
Sample
• 8 employment centres in two provinces,
Saskatchewan and New Brunswick.
• 169 participants began the study
» 13 discontinued participation
» 5 submitted incomplete data where one
or more of the survey forms was missing
• 151 provided complete data
18
Sample by Province
Province
Saskatchewan
Intervention Type
Delivery
Total
Independent
Assisted
CDM
20
15
35
JS
23
25
48
Province Total
43
40
83
CDM
28
15
43
12
13
25
40
28
68
New Brunswick JS
Province total
Total for 2 provinces
151
There were no differences in pretest scores between provinces
• Therefore we combined provinces for data analysis
• This Increased the cell size + therefore increased the statistical power
19
Sample Continued
 74 males and 77 females
 Age: 19 - 62 years (Mean age = 44 years)
 149 were Canadian citizens or landed
immigrants (legally entitled to work in
Canada)
 118 had not previously participated in
employment services programs
 35 were working full or part-time
 6 said their work was a good fit for them
 115 participants were not employed
20
Employment History
• In the past 5 years
• 30 of them had been in 5 or more jobs
• 22 had been in 4 jobs
• 25 had been in 3 jobs
• 46 had been in 2 jobs
• 24 had been in the same job for the past
5 years
21
Unemployment History
• In the past 5 years
• 15 had been unemployed for 36-60 months
• 15 had been unemployed for 18-30 months
• 15 had been unemployed for 12-16 months
• 19 had been unemployed for 6-11 months
• 26 had been unemployed for 1-5 months
• 42 had not been unemployed over the past
5 years
22
Approach to Evaluation
Outcome-Focused, Evidence-Based
Practice
Input  Process  Outcome
Framework developed by CWRG
23
Outcome-Focused Evidence-Based
Practice
Input

Process

Outcome
Indicators of client change
1.Learning outcomes
• Knowledge and skills about
using LMI
2.Personal attribute outcomes
• Changes in attitudes,
confidence, optimism, etc.
3.Impact outcomes
•employment status
24
Outcome-Focused Evidence-Based
Practice
Input

Process

Outcome
Activities that link to outputs or outcomes
1. Tailored LMI Packages
2. Protocol for assisted self-help
3. Counsellor check lists
4. Client checklists
25
Outcome-Focused Evidence-Based
Practice
Input 
1.
2.
3.
4.
Process
 Outcome
Resources available
Staff time
Resource Centres
Self-selected on-line resources
Needs determination protocol
26
What Did We Measure?
The Dependent Measures for the data analysis were:
•
General ability to use LMI
•
Knowledge
» Clear vision of what I want in my career future
» Knowledge of print and online resources
•
Skill
» Have effective strategies for keeping myself motivated
» Have a realistic action plan
•
Personal Attributes
• Optimism about what lies ahead re meeting my career
goals
• Confidence in my ability to manage future career
transitions
27
How Did We Measure?
Problems with standard Pre-Post
Design
• Pre: Before they experience a program,
participants are asked to rate their skill (or
knowledge)
– Often, pre-program scores are higher because
people don’t know what they don’t know
• Post: After experiencing a program,
participants are also asked to rate their skill
– Often post-program scores are lower because
people have found out that they knew less than
they thought or had less skill than they thought
• So, a comparison of difference in scores may
not reflect the true difference.
28
How can we get around this
problem?
Use a Post-Pre Assessment AFTER the
program
At the end of the intervention, we asked
the participants to compare themselves
now and before the intervention:
“Knowing what you know now, how would
you rate yourself before the research
session, and how would you rate yourself
now?”
29
Descriptive Results
• Looking at the 14 items in the survey instrument:
– Before the intervention: between 30% and 58% of the
respondents indicated that their level of competence on that
item was “Not OK”;
– After the intervention: 1% – 9% indicated that their level of
competence on that item was “Not OK”.
• Mean Scores:
– Before the intervention: All responses (1 exception) were in
the “Not OK” range;
– After the intervention: All responses were greater than
minimally OK (mean score 3 or greater) (2 exceptions where
the mean scores were 2.19 and 2.96)
30
Descriptive Results (cont)
• Before the intervention: 45% (n=946) of
responses were “not OK” (0 or 1);
• After the intervention: 5% (n=95) of the
responses were “not OK”.
31
Descriptive Results (cont)
• Before the intervention: 5% (n=108) of the
responses were “Exceptional” (4);
• After the intervention: 39% (n=825) of the
responses were “Exceptional”.
• The amount of change was similar across all
three dimensions of the survey: knowledge,
skills, and personal attributes.
• All 3 dimensions demonstrated about the same
amount of change.
32
Descriptive Results (cont)
• Of particular note are items that suggest
increased ability to self-manage their careers,
such as:
– A clear understanding of what I need to do to move
forward in my career.
– A clear vision of what I want in my career future.
– Knowledge of print and online resources that help
me to research career/employment options.
33
Descriptive Results (cont)
• More items that suggest increased ability to selfmanage their careers:
– The ability to access career resources that can help
me implement my career vision.
– Effective strategies for keeping myself motivated to
achieve my career/employment goals.
– A realistic action plan (or schedule) summarizing
the main career/employment-related activities I
want to pursue and the processes I am engaging in.
– Confidence in your ability to manage future career
transitions.
– Confidence in my ability to research career,
employment, and training options that are available
34
Overall Ability to Use LMI
For group as a whole:
• significant increase in
skills for using LMI
• neither intervention was
more conducive to one
manner of delivery
compared to the other
• Both CDM and JS groups had significant increases across time
• Change in CDM group was significantly larger than in JS group
• Participants in the JS group had higher scores than participants in the
CDM group, likely indicating that JS participants were more familiar
with using LMI before the project began.
• Participants receiving assistance demonstrated greater change in skill
at using LMI than did those in the independent mode
35
Knowledge on How to Use LMI
For group as a whole:
• significant increase in
knowledge about how to
use LMI
• neither intervention was
more conducive to one
manner of delivery
compared to the other
• Both CDM and JS groups had significant increases across time
• Change in CDM group was significantly larger than in JS group
• Participants in the JS group had higher scores than participants in the
CDM group, likely indicating that JS participants were more familiar with
using LMI before the project began.
• No difference in knowledge gain between those receiving assistance and
those in the independent mode (p = .09)
36
Skills in Using LMI
For group as a whole:
• significant increase in
skills for using LMI
• neither intervention was
more conducive to one
manner of delivery
compared to the other
• Both CDM and JS groups had significant increases across time
• Change in CDM group was significantly larger than in JS group
• Participants in the JS group had higher scores than participants in the
CDM group, likely indicating that JS participants were more familiar
with using LMI before the project began.
• Participants receiving assistance demonstrated greater change in skill at
using LMI than did those in the independent mode
37
Personal Attributes Related to using LMI
For group as a whole:
• significant increase in
personal attributes
related to using LMI
• neither intervention
was more conducive to
one manner of delivery
compared to the other
• Both CDM and JS groups had significant increases across time
• Change in CDM group was significantly larger than in JS group
• Participants in the JS group had higher scores than participants in the
CDM group, but the gap was less at the end of the study (i.e., CDM
group reported greater change).
• Participants receiving assistance reported higher scores than did those in
the independent mode
38
Sample of learning outcomes
Regarding the Primary Objectives, and
knowing what you know now,
how would you rate yourself before the workshop, and how would you rate
yourself now?
Before
After
Ave
Ave
1. A clear understanding of what I need to do to move forward in my career.
21 57
48
16
1.57 1
3
28 66 53
3.11
2. A clear vision of what I want in my career future.
24 37
40
39 11 1.84 1
7
29 63 51
3.03
3. Reviewed my past work, education and experience so that I know what
skills and strengths I have.
11 33
57
35 14 2.05 2
3
21 68 56
3.15
4. A list of possible options that may fit with what I want in my career future.
23 49
47
23
9
1.64 1
4
25 66 54
3.12
5. Knowledge of print and online resources that help me to research
career/employment options.
32 55
42
16
5
1.38 1
1
21 53 74
3.32
6. Confidence that career-related employment opportunities actually exist
that fit with what I want in my career future.
19 41
58
25
8
1.75 1 10 27 58 55
3.03
•
•
•
•
•
•
9
On these 6 items (K = 1, 2, 3, 5; S = 4; PA = 6)
Pre: 402 Unacceptable Ratings – Post: 35 Unacceptable Ratings
Unacceptable Ratings decreased from 45% to 4%
Pre: 56 Exceptional Ratings – Post: 343 Exceptional Ratings
Exceptional Ratings increased from 6% to 38% of the participants
Pre: 80% of means were Not OK – Post: all means were more than min OK
39
Attribution for Change
To what extent would you say that any changes in the ratings on
the previous pages are a result of your participation in this
research project, and to what extent were they a function of
other factors in your life?
mostly
other
factors
somewhat
other
factors
uncertain
somewhat
this program
mostly
this
program
CDMIndependent
0
2
5
14
26
CDM-Assisted
0
1
1
8
20
JS-Independent
3
0
6
14
12
JS-Assisted
0
0
3
11
22
Total
3
3
15
47
80
Program
40
Attribution for Change
• To what extent would you say that any
changes in the ratings for the before and
after survey questions are a result of
your participation in this research
project, and to what extent were they a
function of other factors in your life?
Of 148 respondents:
– 47 said “somewhat this program”
– 80 said “mostly this program”
41
Impact Outcomes-1
Employment status
If you answered yes to the above question,
to what extent does this work fit with your career vision?
42
Impact Outcomes-2
Action plan
43
Impact Outcomes-3
Desire help in creating an action plan
44
Other Noteworthy Results
•
No significant gender differences
– Women and men responded equally well to all
intervention-delivery combinations
•
Other analyses in progress
–
–
–
–
Work history
Unemployment history
Age
Post-intervention interviews with selected
clients (n=64) 1week and 4 months post the
study
45
Other Noteworthy Results (cont)
•
We also are analyzing the process data
– The process checklists will provide evidence
that the counsellors followed the game plan and
that the clients were engaged with the LMI
material
– This will permit us to say that the program
(Intervention + Delivery) is responsible for
the change
46
LMI: General Summary of Results
 All intervention-delivery combinations
produced significant change in:
•
•
•
•
General ability to access and use LMI
Knowledge about how to use LMI
Skills for using LMI and taking action
Personal attributes, e.g., optimism, confidence,
and by inference, motivation
 Assisted use produced greater change across
time than independent use
 Over 80% of clients attribute change to the
program and not other factors
47
What have we learned?
Perspectives of the Research
Partners
48
What have we learned?
(for clients in this study)
• “Guided” LMI, (i.e., LMI embedded in a learning
process) results in knowledge and skill acquisition
as well as the capacity for self-management
• LMI appropriate for a client’s specific need
(opposed to general LMI) appears to support
engagement and action
• A little professional support can go a long way for
many clients!
• Structure, a roadmap, and timelines appear to
motivate action and a sense of progress
• Giving clients hands-on tools appears to motivate
more than money!
49
What helped make the project work
•
•
•
•
•
The leadership of the provinces was
exceptionally strong
The service providers caught the spirit of the
research and went with it, even though it was
not an exact fit for them
The front-end screening (service needs
interview) was well done, and permitted a
strong match between client need and
intervention focus
The tailored intervention packages were superb
The attention to detail in data collection was
very well done
50
What needs to be done next?
A Sample…
• This research was in English only; French to
come (hopefully!)
• There are 5 employability dimensions; only 2
have “guided” LMI
• We do not know what the impact of “guided”
LMI within an employment counselling process
would be; nor do we know the impact of
“guided” LMI packages self-selected by clients
(no assessment) – these could be hugely
informative for service delivery
51
Thank you Research Partners,
and
Thank you for your attention
today!