Transcript www.ncrcadvocates.org

WorkKeys Innovations:
A Holistic Solution for
WIRED West Michigan
Steve Robbins, AVP, Applied
Research, ACT, Inc.
Overview
• Why we should care about combining
cognitive- and personality-based
measures
• National Career Readiness Certificate
“Plus”
• What we know from workforce and
educational literatures
• Differential test strategies along the
entire continuum of employment
2
2
Why Now?
• Market Need: SHRM National Study
• Body of research informs how to
optimize Personal Skill Assessments
• ACT strength in cognitive & noncognitive assessment (John Holland, VP
Research in 1960’s)
• Solution-focused approach
3
3
Businesses want integration of
Cognitive & Personality Constructs
SHRM: Applied Skills and Basic Knowledge:
Combining and Ranking
For new entrants with a two-year college/technical school diploma,
applied skills are four of the top five “very important” skills in
combined ranking with basic knowledge and skills.
Rank
Skill
%
1
Professionalism/Work Ethic
83.4
2
Teamwork/Collaboration
82.7
3
Oral Communications
82.0
4
Critical Thinking/Problem Solving 72.7
5
Reading Comprehension
71.6
6
Written Communications
71.5
7
English Language
70.6
8
Ethics/Social Responsibility
70.6
Casner-Lotto, J. & Barrington, L. (2006)
4
4
Solutions are Need Driven:
Continuum of Employment
Purpose
Needs
Purpose
Needs
PreSelection
-Screen people in most cost
effective way
-Find honest/dependable
employees
-Save time in the screening
process
Coaching/
Development
-Identify other jobs that an
employee can fit
-Develop employees for
future company needs
-Employees identify areas of
improvement
Recruitment
-Identify people who fit the
work environment
-Identify people with skills
that match the job
Succession/
Leadership
Planning
-Identify candidates for toplevel positions
-Develop employees for
future needs of company
-Retain top performers
Selection
-Select employees with skills
that best fit the job
-Save time in selection
process
-Select people in most costeffective way
-Find honest/dependable
employees
-Certifying employees
Training/
Development
-Identifying work readiness
-Identify basic workplace
skill levels
-Educating about career
planning
-Job Placement
5
5
WorkKeys Assessment Solutions:
Pyramid for Success
• Ensure work and training readiness
(WorkKeys Foundational Skills)
• Provide employers another source of
information for selection (“Performance”)
• Give a “snap shot” of strengths and
areas of improvement across key
response tendencies or domains
(“Talent”)
• Promote career exploration of job fit
(“Fit”)
6
6
Pyramid for Success
FIT
•Match individual interests/values to
work environment
•Enhance job persistence &
satisfaction
•Develop Talent pool to meet needs
TALENT
•Benchmarking for selection
•Coaching & Development
•Compound Indices: Sales, Managerial,
Leadership, Safety
PERFORMANCE
•General Work Performance: Productivity, Absenteeism,
Complaints about conduct
•Safety and Risk Reduction
FOUNDATIONAL SKILLS
•Job Analysis – identifies the skills and skill levels needed to be
successful on the job
•Assessments – show the current skill levels of an individual
•Training – helps individuals and employers correct skill gaps
7
7
National Career Readiness
Certificate
Currently, the WorkKeys system assesses foundational skills,
such as reading and locating information, using a three-step
process of job profiling, assessment, and skill-gap training.
6
5
5
4
4
3
3
Skill
Gap
4
Benchmark Levels
Required
3
Worker’s Skill
Level
2
1
Applied Math
Reading
Locating Information
8
8
National Career Readiness
Certificate “Plus”
• A flexible solution to meet state and
system needs
• Georgia example
• Talent for coaching & development
• “Fit” for career exploration
• Alterable variables allow for
intervention & guidance
9
9
Cognitive Ability Tests
“Rule”
• General vs. specific test effect sizes
Training Outcomes Work Performance
GMA
.54 (.55*)
.62**
Math
.48 (.48*)
.52**
Reading
.44 (.40*)
.35**
* Brown, Le, & Schmidt (2006) **Salgado et al. (2003)
• Adverse impact issues can be ameliorated
with specific tests:
– Task analysis
– Job-specific tests
– Combinational use of cognitive and noncognitive tests
10
10
Personality Tests
“Add Value”
• Validity Estimates:
Training
Outcomes
Work
Performance
Old
Validity
New1
Validity
Old
Validity
New1
Validity
GMA
.526
.668
.507
.652
Conscientiousness
.169
.176
.217
.230
Emotional Stability
.098
.103
.112
.121
1 From
Schmidt et al. (2007) using indirect range restriction
11
11
What are We Talking About?
Personality Test Constructs
• The Big Five:
Emotional Stability, Extraversion,
Agreeableness, Conscientiousness,
Openness to Experience
• Specific or Facet-level Traits:
Carefulness, Discipline, Influence, Order,
Sociability, Drive, Creativity
• Compound Traits:
Service Orientation, Integrity, Managerial
Potential, Teamwork
12
12
Personality Test Issues
• Selection vs. Coaching and Development as
Differential Applications
• Selection Approach
– Recommend multiple hurdles and/or top-down approach
• Adverse Impact Limited
• Differential Outcomes (Rotundo & Sackett, 2002)
– Task Performance (Technical Core)
– Organizational Citizenship Behavior (Contextual
Performance)
– Counterproductive Work Behavior
• Add Satisfaction/Tenure as another Key
Outcome
13
13
Combining Personality &
Cognitive Ability Tests
• Level of correlations are low:
GMA x C = .02
Math x C = -.15
Reading x C = -.05
GMA x ES = .17
Math x ES = .17
Reading x ES = .11
(Ackerman & Heggestad, 1997)
14
14
Combining Personality &
Cognitive Ability Tests
Creating opportunity for incremental validity
especially as criteria vary
Correlations between general cognitive ability and personality
tests and measures of job performance in Project A
Cog
Pers
Both
Criteria
.63
.26
.67
Core technical proficiency
.65
.25
.70
General Soldiering Proficiency
.31
.33
.44
Effort and Leadership
.16
.32
.37
Personal Discipline
.20
.37
.42
Physical fitness and military bearing
(McHenry, Hough, Toquam, Hanson, & Ashworth, 1990)
15
15
Educational Examples
Retention Behavior
R1
Cumulative R1, 2
ACT
.176
.176
Motivation
.146
.227
Social Engagement
.097
.200
Personality Combined
.155
.232
1 Correlation of fitted probabilities and dichotomous retention outcome
2 SRI scale scores and ACT Composite score used as predictors
16
16
Educational Examples
Academic Performance
R1
Cumulative R1, 2
ACT
.464
.464
Motivation
.297
.545
Self Regulation
.196
.467
Personality Combined
.315
.553
1 Multiple R from linear regression model
2 SRI scale scores and ACT Composite score used as predictors
17
17
ACT: Foundational and Personal
Skills Assessments measure
Different Applied Job Skills:
Communication
Problem
Solving
Interpersonal
Personal
Teamwork
Performance
Business Writing
Applied
Technology
Listening
Applied
Mathematics
Talent
Reading for
Information
Locating
Information
Fit
Writing
Observation
18
18
“If they staged a slowdown, how would we know?”
Harvard Business Review. March 2007. p. 90
19
19
Page 1 of 2
Employer Report
Performance Assessment
Examinee:
Alvin C. Tracey
Examinee ID:
XXXXX7890
Report for: Abbatoir Industries
Site: Iowa City, IA
Test Date: 4/12/07
WorkKeys Performance Assessment measures personal behaviors and attitudes critical to workplace success. A high score on the
Performance Index indicates higher likelihood of having successful work attitudes and engaging in safety behaviors.
WorkKeys Performance Assessment Profile
Performance
Score Report
Percentile Rank: Approximate percent at or below score
0
PERFORMANCE
10
20
30
40
50
60
70
80
90
100
64
General Work Attitudes 50
Risk Reduction
89
What This Means:
Percentile
64
Scale Definitions
Performance Index -- The overall rating for an examinee based on the combination of General Work Attitudes
and Risk Reduction scores.
A moderate Performance Index (combination of General Work Attitudes and Risk Reduction) suggests a candidate may be
moderately desirable. Individuals with similar scores are likely to perform at a moderate level of productivity and work safety.
See below for additional interpretive information about this person’s general work attitudes and risk reduction tendencies.
Highly desirable level of expected performance
Moderately desirable level of expected performance
Less desirable level of expected performance
50
89
General Work Attitudes—Positive and productive attitudes toward work tasks, coworkers, the organization itself, and
other work-related behaviors.
A candidate with a moderate General Work Attitudes score may tend to be an employee who:
 May be disagreeable with coworkers or supervisors under stressful circumstances
 Is usually conscientious about completing work on time
 Is usually honest with coworkers or supervisors
 Will make appropriate use of company assets under most circumstances
Risk Reduction—Tendency to avoid engaging in risky behaviors, such as noncompliance with safety rules and
conflict with supervisors and coworkers.
A candidate with a high Risk Reduction score may be an employee who:
 Consistently follows safety rules and procedures
 Is consistently alert to job risks
 Is very unlikely to engage in inappropriate interpersonal behaviors such as, aggression or hostility
20
20
“It was about here, wasn’t it, Ed, when you came on board as
sales manager?”
Harvard Business Review. March 2007. p. 90
21
21
Page 1 of 3
Employer Report
Report for: Abbatoir Industries
Talent Assessment
Examinee:
Alvin C. Tracey
Examinee ID:
XXXXX7890
Site: Iowa City, IA
Test Date: 4/12/07
WorkKeys Talent Assessment measures personal and workplace behaviors and attitudes. This report is designed to help identify examinee
strengths and weaknesses in order to ensure success in the workplace.
WORKEYS TALENT ASSESSMENT PROFILE
Percentile Rank: Approximate percent at or below score
0 10
Carefulness
90
Cooperation
25
Creativity
35
Discipline
85
Drive
65
Goodwill
20
Influence
40
Optimism
50
Order
90
Savvy
50
Sociability
50
Stability
90
20
30
40
50
60
70
80
90
100
Talent
Score
Report
Capitalize on Individual Strengths
Percentile
90
Scale Definitions
Carefulness: Tendency to think and plan carefully before acting.
This individual’s responses suggest that he or she is cautious, deliberate, and pays close attention to detail in the workplace.
Responders at this score level tend to think carefully before acting or speaking. They always consider the consequences of their
actions, and their decisions are usually well thought-out.
90
Order: Tendency to be well organized as well as keeping the work area neat and tidy.
90
Stability: Tendency to maintain composure and rationality in situations of actual or perceived stress.
85
Discipline: Tendency to begin tasks and complete them without becoming distracted or bored.
The examinee’s responses suggest that he or she is well-organized, and consistently keeps physical surroundings neat and tidy.
Individuals who respond at this score level are always methodical in their manner and maintain a structured professional
environment.
This individual’s responses suggest that he or she maintains his or her composure even when faced with highly stressful
situations. Individuals who respond at this score level tend to remain calm and even-tempered in their conduct, and they feel
confident in their ability to handle the pressure and stress of working under deadlines..
The examinee’s responses suggest that he or she commits to work duties until they are complete. Individuals who respond at
this score level take responsibility and can always be relied upon to get their work done on time. They are not easily distracted,
and always persist through challenges until the task is done.
22
22
23
23
1 of 3
Employer Report
Report for: Abbatoir Industries
Fit Assessment
Examinee:
Alvin C. Tracey
Examinee ID:
XXXXX7890
Site: Iowa City, IA
Test Date: 4/12/07
WorkKeys Fit Assessment measures the fit between an examinee's work-relevant interests and values and the
characteristics of occupations. Worker satisfaction and commitment are associated with greater fit. Examinee-specified
primary occupation is in BOLD.
Fit Index for Occupations Specified by Examinee
Percentile Rank: Approximate percent at or below score
0
11-3040.00
Human Resources Managers
56
11-3042.00
Training and Development Managers
61
11-3049.99
Human Resources Managers, All Other
54
11-3061.00
Purchasing Managers
56
11-3071.00
Transportation, Storage, and Distribution Managers
86
10
20
30
40
50
60
70
80
90
100
FIT
Score
Report
Fit scores are based on the examinee's responses to the Interest Inventory and Work Values Inventory.
Fit Index for Related Occupations
Occupations specified by the examinee, as well as related occupations, are ranked by Fit Index. Examinee-specified
occupations are in BOLD.
Code
11-3071.00
11-3011.00
11-2022.00
11-3071.01
11-3071.00
11-3042.02
11-3040.00
11-3061.00
11-3031.02
11-3041.00
11-3049.99
11-3031.00
11-2011.00
11-2021.00
11-2031.00
11-3031.01
11-3051.00
11-3021.00
11-9011.02
11-9011.01
Title
Transportation, Storage, and Distribution Managers
Administrative Services Managers
Sales Managers
Transportation Managers
Storage and Distribution Managers
Training and Development Managers
Human Resources Managers
Purchasing Managers
Financial Managers, Branch or Department
Compensation and Benefits Managers
Human Resources Managers, All Other
Financial Managers
Advertising and Promotions Managers
Marketing Managers
Public Relations Managers
Treasurers and Controllers
Industrial Production Managers
Computer and Information Systems Managers
Crop and Livestock Managers
Nursery and Greenhouse Managers
Fit Index
86
80
79
77
69
61
56
56
55
54
54
53
51
50
50
42
29
27
23
14
Level of Fit
High
High
High
High
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Moderate
Low
Low
24
24
25
25
Selection Solutions
• Reducing Risk
– Task Competence through WorkKeys and job
profiling
– General Work & Safety
• Increasing Tenure
– Task Competence through WorkKeys and job
profiling
– Fit
• Getting the Right Person
– Talent Benchmarking
– Past Work Performance Record
26
26
Coaching & Development
Solutions
• Career Exploration
– Task Competency
– Fit
• Leadership Development
– Talent
– Fit
• Teamwork
– Talent
27
27
Return on Investment Approximations
under Various Scenarios
Selection %
Candidate
Success %
Selected
Success %
25%
54%
10%
50%
25%
80%
45%
25%
50%
72%
25%
36%
50%
64%
50%
Cost per
failure
$5,000
$13,000
$10,000
$27,500
$5,000
$13,500
$10,000
$28,500
$5,000
$23,500
$10,000
$48,500
$5,000
$26,000
$10,000
$53,500
$5,000
$26,000
$10,000
$53,500
$5,000
$33,500
$10,000
$68,500
Notes: Selection % = the percentage of the candidate pool selected for
hire, Candidate Success % = the percentage of the candidate pool that
would be successful if hired, Selected Success % = the percentage of the
selected candidate pool that will be successful, Cost per failure = the
average cost for each unsuccessful employee relative to each successful
employee, ROI per 100 candidates = the average return on investment for
the selection procedure assuming a $15 fee per candidate.
ROI per 100
candidates
28
28
Final Thoughts
• Adverse Impact may be reduced when
combining tests
– Still recommend “multiple hurdles” approach
• Incremental Validity Research
underway:
– Combination of Task & Personality
measures
– Differential work outcomes
– ROI x Solution
29
29
References
•
Ackerman, P. L., & Heggestad, E. D. (1997). Intelligence, personality, and interests: Evidence
for overlapping traits. Psychological Bulletin, 121, 219-245.
•
Brown, K. G., Le, H., & Schmidt, F. L. (2006). Specific aptitude theory revisited: Is there
incremental validity for training performance? International Journal of Selection and Assessment,
14, 87-100.
•
Casner-Lotto, J. & Barrington, L. (2006). Are they really ready to Work? Society for Human
Resource Management. http://www.shrm.org/hrresources/surveys_published
•
McHenry, J. J., Hough, L. M., Toquam, J. L., Hanson, M. A., & Ashworth, S. (1990). Project A
validity results: The relationship between predictor and criterion domains. Personnel Psychology,
43, 335-354.
•
Robbins, S., Allen, J., Casillas, A., Peterson, C., & Le, H. (2006). Unraveling the differential
effects of motivational and skills, social, and self-management measures from traditional
predictors of college outcomes. Journal of Educational Psychology, 98, 598-616.
•
Rotundo, M., & Sackett, P. R. (2002). The relative importance of task, citizenship, and
counterproductive performance to global ratings of job performance: A policy-capturing
approach. Journal of Applied Psychology, 87, 66-80.
•
Salgado, J. F., Anderson, N., Moscoso, S., Bertua, C., & de Fruyt, F. (2003). International
validity generalization of GMA and cognitive abilities: A European community meta-analysis.
Personnel Psychology, 56, 573-605.
•
Schmidt, F. L., & Hunter, J. E. (1998). The validity and utility of selection methods in personnel
psychology: Practical and theoretical implications of 85 years of research findings. Psychological
Bulletin, 124, 262-274.
•
Schmidt, F. L., Shaffer, J., & Oh. I. (2007). Reassessing the Relative Importance of Cognitive
Ability and Personality in Job Performance and Training Performance: Some Surprising New
Research Findings. Paper presented at the 2007 ATP conference, Palm Springs, CA. Feb. 6.
30
30
Incorporating Foundational
and Soft Skill Assessments
For questions regarding this presentation or for further
information contact:
Steve Robbins at 319-337-1227 or [email protected],
Gary Nolan at 319-337-1526 or [email protected],
31
31