Transcript Firefighter Selection, Inc.

Fire & Police Selection, Inc.
Dan Biddle, Ph.D.
Stacy L. Bell, M.S.
August 26, 2011
A National Fire Chief Survey Reveals the Key
Ingredients to Include in Your
Fire Department Test
Contact Information
Dan A. Biddle, Ph.D.
CEO, Biddle Consulting Group, Inc./Fire & Police Selection, Inc.
Stacy L. Bell, M.S.
Executive Vice President, Fire & Police Selection, Inc.
193 Blue Ravine, Ste. 270
Folsom, CA 95630
1-888-990-3473
www.FPSI.com | www.biddle.com
Email: [email protected] | [email protected]
Copyright © 2011 Fire & Police Selection, Inc.
Biddle Consulting Group, Inc.
•
•
•
•
•
•
Assisted hundreds of clients with cases involving Equal Employment Opportunity
(EEO) / Affirmative Action (AA) (both plaintiff and defense)
EEO Litigation Support / OFCCP (federal contracting) Audit Support
Compensation Analyses / Test Development and Validation
Published: Adverse Impact and Test Validation, 2nd Ed., as a practical guide
for HR professionals
Editor & Publisher: EEO Insight an industry e-Journal
Creator and publisher of a variety of productivity
Software/Web Tools:
– OPAC® (Administrative Skills Testing)
– CritiCall® (9-1-1 Dispatcher Testing)
– AutoAAP™ (Affirmative Action Software and Services)
Industry
– C4™ (Contact Center Employee Testing)
Leader
– Encounter™ (Video Situational Judgment Test)
– Adverse Impact Toolkit™ (free online at www.disparateimpact.com)
– AutoGOJA® (Automated Guidelines Oriented Job Analysis®)
Copyright © 2011 Fire & Police Selection, Inc.
Fire & Police Selection, Inc.
•
•
•
•
•
•
•
Over 80 Physical Ability Test installations at fire
departments across the U.S.
Over 800 personnel and fire and police clients across the
U.S. and Canada
Entry-level written, physical, interview assessments,
manipulative skills tests, cognitive ability tests, and
personality tests available for selection of public safety
personnel
Post-Academy tests
Fire Promotional tests and assessment exercises
Online applicant testing
Custom job analysis and validation services
Copyright © 2011 Fire & Police Selection, Inc.
Presentation Overview
•
•
•
•
•
Personnel Testing Overview
What is Validation and Why Do I Need It?
Test Usage
The National Fire Select Test—A Case
Study
Benefits to a Balanced Testing Process
Copyright © 2011 Fire & Police Selection, Inc.
Approach to Test Validation
The Quickly Changing Climate of
Personnel Testing…
• EEOC
• DOJ
• DOL (OFCCP—Office of Federal
Contract Compliance Programs)
• State EEO Offices of Fair Employment
• Private Plaintiff Groups
Copyright © 2011 Fire & Police Selection, Inc.
Adverse Impact & Test Validation
Uniform Guidelines & Civil Rights Act of 1991
Amends Section 703 of the 1964 Civil Rights Act (Title VII)
(k)(1)(A). An unlawful employment practice based on
disparate impact is established under this title only if:
•
•
A(i) a complaining party demonstrates that a respondent uses a
particular employment practice that causes a disparate impact on the
basis of race, color, religion, sex, or national origin, and the respondent
fails to demonstrate that the challenged practice is job-related for the
position in question and consistent with business necessity; OR,
A(ii) the complaining party makes the demonstration described in
subparagraph (C) with respect to an alternate employment practice,
and the respondent refuses to adopt such alternative employment
practice.
Copyright © 2011 Fire & Police Selection, Inc.
How Can Testing Practices be Challenged?
Title VII Disparate Impact Discrimination
Flowchart
Practice,
Procedure,
or Test (PPT)
Adverse
Impact?
“OR”
YES
END
Is the PPT
Valid?
YES
Alternative
Employment
Practice?
NO
Defendant Prevails
YES
Plaintiff Prevails
NO
NO
Plaintiff
Prevails
Copyright © 2011 Fire & Police Selection, Inc.
Validation Standards
Court Precedence
• Title VII: Uniform Guidelines on Employee Selection Procedures
(1978): EEOC, DOJ, CSB, DOL
• Principles for the Validation and Use of Personnel Selection
Procedures (2003)
•
• Division 14 of the American Psychological Association (the Society
for Industrial and Organizational Psychology, or SIOP)
•
Standards for Educational and Psychological Testing, published
by:
• American Educational Research Association
• American Psychological Association, and
• National Council on Measurement in Education
Copyright © 2011 Fire & Police Selection, Inc.
What is Validation?
Traditionally, it is making sure a
selection procedure (i.e., test) measures
what it is designed to measure
• In a legal realm, a selection procedure is
valid if it can be proven by an employer
that it is “…job related for the position in
question and consistent with business
necessity.”
•
Copyright © 2011 Fire & Police Selection, Inc.
Great Job
Performance
Copyright © 2011 Fire & Police Selection, Inc.
Reliable… but not Valid
Consistently
hiring the
wrong people
Copyright © 2011 Fire & Police Selection, Inc.
Neither Reliable or Valid
Hit or
Miss…
Copyright © 2011 Fire & Police Selection, Inc.
Reliable and Valid!
Reduced
Turnover &
Absenteeism
Better
Productivity
More
Satisfied
Clients
Less Time
to Train
Employee
Satisfaction
Increased
Less Liability
Copyright © 2011 Fire & Police Selection, Inc.
Uniform Guidelines Style
Validation…
Content Validity
• Section14C
• Criterion Validity
• Section 14B
• Construct Validity
• Section 14D
•
Copyright © 2011 Fire & Police Selection, Inc.
Content Validity Essential #3: Connecting
the Test to the Job
Job
Duties
Other KSAOs
Operationally
defined KSAOs
Selection
Devices
(e.g., application
form, tests,
interviews)
Copyright © 2011 Fire & Police Selection, Inc.
Content Valid!
Test Usage



Pass/Fail Cutoffs:
– “Normal Expectations of Acceptable Proficiency in the
Workplace” (Guidelines, 5H)
– Modified Angoff (U.S. v. South Carolina, USSC)
Banding:
– Substantially Equally Qualified Applicants
– Statistically Driven (use Std. Error of Difference)
Ranking:
– Is there adequate score dispersion?

– Does the test have high reliability? (e.g., >.85)
– Is the KSA performance differentiating?
Weighted/combined with other tests
– How are the weights related to the job
– Do they come from the job analysis or SME ratings?
Copyright © 2011 Fire & Police Selection, Inc.
Using Test Scores
Applicant
Tom
Stacy
Bob
Frank
Julie
Rozanne
Mark
Luke
Henry
Paul
Peter
Rebecca
Alyssa
Matthew
John
Annette
Ray
Thomas
Julissa
Score
100
100
100
100
99
99
98
98
97
97
96
96
95
94
93
93
92
91
90
• Ranking assumes one applicant is
reliably more qualified than the other
• Banding considers the unreliability of
the test battery and “ties” applicants
• Pass/fail cutoffs treat all applicants as
either “qualified” or “not qualified”
Fire & Police Selection, Inc. 2007
www.fpsi.com
18
Pass/Fail Cutoffs
•
Some major considerations:
• NOT ADVISED:
• Using an arbitrary cutoff (e.g., 70%)
• Using applicant scores to benchmark (e.g., setting cutoff scores at
mean-SD of applicant scores)
• ADVISED:
• Consider “Normal expectations of acceptable proficiency in the
workplace” (Guidelines, 5H)
• Usually requires SME-level data or ratings
• # Applicants that can be feasibly processed
• CONSIDER:
• Is the test supported by content validity information or criterionrelated information?
• How critical are the KSAs measured?
• Does the test measure “baseline” or “differentiating” KSAs?
• How would current incumbents perform on this test?
Copyright © 2011 Fire & Police Selection, Inc.
Comparison Between “Test Use”
Options
Factor
Ranking
Banding
Pass/Fail
Cutoffs
Validation
Requirements
High
Moderate
Low
Adverse Impact
High
Moderate
Low
Defensibility
Low
High
High
Litigation "Red Flag"
High
Moderate
Low
Utility
High
Moderate
Low
Cost
Low
Moderate
High
Applicant Flow
Restrictive/Controllable
Moderate/Controllable
High
Development Time
Low
Moderate
High
Reliability
Requirements
High
Moderate
Low
# Item Requirements
High
Moderate
Low
Copyright © 2011 Fire & Police Selection, Inc.
The Value of a Standardized Written Test
•
Candidates bounce from one department to the next taking
entry-level firefighter tests that measure different skills and
abilities when in fact the job requirements rarely differ from
department to department.
•
There is value to a standardized national firefighter written
test similar to the concept of the standardized physical
ability test (e.g., C-PAT).
•
How can we measure critical skills and abilities that are
standard entry-level requirements across fire departments in
a way that is job-related and fair to candidates?
Copyright © 2011 Fire & Police Selection, Inc.
Selecting the Ideal Firefighter
Mechanical Aptitudes
Critical Skills &
Abilities
Test Map
“A”
Reading
Test “B”
Reading Ability
Mathematical Ability
Writing Ability
Interpersonal Skills
NFST™
Decision
Making
Test
“C”
Spatial Abilities
Copyright © 2011 Fire & Police Selection, Inc.
Development & Validation of the
National Fire Select Test™
•
Leveraged cumulative firefighter job analysis data from the
last decade to identify which skills and abilities are
consistently rated as being “critical to successful firefighter
job performance.”
•
Administered the “Buy the Perfect Firefighter” survey to
build the test content and test weights.
•
Used Subject-Matter Experts (SMEs) and job performance
ratings to identify a job-related cutoff.
Copyright © 2011 Fire & Police Selection, Inc.
Development & Validation of the
National Fire Select Test™
•
FPSI sent surveys to over 1,000 chief officers from
fire departments across the U.S. who were asked to
weigh the importance of key critical skills and
abilities required for successful entry-level firefighter
job performance.
•
Respondents were asked to evaluate both cognitive
(e.g., “Academic Success”) and non-cognitive (e.g.,
“Personal Characteristics”) skills and abilities.
Copyright © 2011 Fire & Police Selection, Inc.
National Survey—Part I
Buy the “Perfect Firefighter”
There are two parts to this survey. The first section asks you to spend
$100.00 to “buy” the perfect Entry-Level Firefighter by spending
this money within two major categories: Cognitive/Academic and
Personal Characteristics. The second section asks you to spend $100
within both of these categories.
•
Section One: Distribute $100 Between Two Qualification Areas
Cognitive/Academic Skills/Abilities (see detailed list below) ____%
Personal Characteristics (see detailed list below)
____%
Copyright © 2011 Fire & Police Selection, Inc.
National Survey—Part II
$ to
Spend
Cognitive/Academic Skill/Ability
Skill/Ability
Description
Math
Basic mathematics to add, subtract, multiply, divide.
Reading
Ability to read, retain, recall, and comprehend written material.
Verbal Communication
Verbally communicate in English.
Writing
Writing ability including spelling, grammar, sentence structure.
Map Reading
Ability to read maps and mapped information.
Problem-Solving
Problem solving and analysis
Strategic Decision-Making
Decision-making ability to carry out assignments.
Mechanical Ability
Sufficient to operate, use, and/or repair equipment.
Total must sum to $100
Copyright © 2011 Fire & Police Selection, Inc.
National Survey—Part II
$ to Spend
Personal Characteristics
Personal Characteristic
Description
Teamwork
Sufficient to work and live harmoniously with co-workers and to
consistently follow-through on routine duties
Working Under Stress
Ability to perform duties under stressful emergency situations.
Allegiance/Loyalty
Have allegiance to department’s core values and mission.
Truthfulness/Integrity
demonstrate truthfulness/integrity to self and others even in the
face of severe consequences.
Public Relations
Willing to be resourceful, goal-oriented, and proactive in bringing
tasks to completion.
Emotional Stability
Willing to remain calm under pressure or heavy workloads, not
easily given to hostility, anxiousness, or vulnerability.
Sensitivity
Ability to maintain respect for other’s emotional needs following a
crisis.
Proactive/Goal-Oriented
Represent public relations and public perception of department’s
image.
Thoroughness/Attention to Detail
Thoroughness and attention to detail in performing all duties.
Following Orders
Receives and responds to orders or instructions from superiors.
Total must sum to $100
Copyright © 2011 Fire & Police Selection, Inc.
U.S. Fire Department Survey
Results
National Fire Select Test™
Personal
Characteristics
55%
Cognitive/Academic
Skills & Abilities
45%
Personal Characteristics Component
Cognitive/Academic Component
Mechanical
Ability
14%
Following Orders
11%
Mathematical
Reasoning
15%
Teamwork
12%
Thoroughness/
Attention
To Detail
9%
Strategic
DecisionMaking
14%
Reading
Ability
18%
Working
Under Stress
10%
Public
Relations
9%
Allegiance/
Loyalty
10%
Sensitivity
8%
Problem
Solving
14%
Writing Ability
15%
Map Reading
10%
Emotional
Stability
10%
Copyright © 2011 Fire & Police Selection, Inc.
Proactive/
Goal-Oriented
8%
Truthfulness/
Integrity
13%
U.S. Fire Department Survey
Results
Test Section
# of Items
Cognitive/
Academic Skills
Personal
Characteristics
55
Weighted
Value
45%
26
43%
Work Styles
Inventory
85
12%
Copyright © 2011 Fire & Police Selection, Inc.
Cutoff Score Determination


12 Fire Chiefs reviewed the academic (cognitive) test items and were asked to identify the
difficulty level. The difficulty levels of the remaining cognitive and non-cognitive test items,
which were validated using criterion-related validity, were calculated from the original validation
studies.
These score bands were derived using the composite Standard Error of Difference (SED) from
the validation sample scores (N = 728). The composite SED is calculated by taking the
composite SEM and multiplying it by the square root of 2. Band A was calculated by taking the
highest score received on the NFST validation administration and reducing it by one composite
SED. Band B was calculated by taking the next highest score below the lowest score in Band A
and reducing it by one composite SED. This process was continued until the composite adjusted
critical score was reached (e.g., a 70.704% on the test). The SED was used to create these
score bands because it effectively provides meaningful score breaks that represent different
levels of the attributes measured. If ranking is used for the NFST, FPSI recommends utilizing
the statistically derived bands shown in Table 1 (rather than top-down ranking).
Band
Highest Score
Lowest Score
A
100%
86.246%
B
86.245%
80.297%
C
80.296%
74.336
D
74.335%
70.704%
Copyright © 2011 Fire & Police Selection, Inc.
Failure to Use a “Balanced Test”
•
•
•
•
•
No validity
Academy failure rates increase and/or job
performance suffers
Less minority representation (i.e., adverse
impact) in the final eligibility list
Might hire mostly “book smart” but not
“people smart” firefighters
Greater potential for lawsuits which almost
always results in lowering standards
Copyright © 2011 Fire & Police Selection, Inc.
Fire & Police Selection, Inc.
Contact us at:
Mailing Address
• 193 Blue Ravine Rd. Suite 270
• Folsom, CA. 95630
• Website
• www.FPSI.com
• www.NationalFireSelect.com
•
•
Telephone
• 916.294.4242 x. 245
•
E-Mail
• [email protected]
Copyright © 2011 Fire & Police Selection, Inc.