Transcript COMPETENCY MODELING - Personnel Testing Council of

Examination Cycle Time
Management
Personnel Testing Council of Southern California
Luncheon Topic, August 13, 2014
Rod Freudenberg, Ph.D.
The Selection Funnel
Recruitment
Start
Screen Apps
Interview
Cycle Time
Test
Ranking
Final Selection
Verifications
Hire
2
Finish
Exam Cycle Time:
Why does it matter?
3

The longer a vacancy exists the longer some of the work of
the organization is not getting done or is being done by
people taking on larger workloads.

Attrition of the strongest candidates because of loss of
interest, pursuit of other employment options, negative
image of agency.

Analyst exam overload – the exam requests keep coming so
the longer it takes, more exams need to be juggled
simultaneously (death spiral)
It’s a Process (right to left)
Client
Organizational
Effectiveness
List Quality
Maximization
List
Predictability
Variability
Reduction
Client
Satisfaction
Cycle Time
Minimization
Cycle Time
Predictability
Feedback
4
Defined and
managed
process
Process
Requirements
Process Maturity Stages (CMM)
Enhancing
Ad-hoc
Defined
and
Repeatable
Enabled
and
Instituted
Measured
and
Predictable
Reducing cycle time requires controlling the process.
5
Outline
6
1.
Defining the Process
2.
Cycle-Time and Process Efficiency
3.
Cycle-Time Analysis
4.
Cycle Time Management Strategies
Defining the Process:
Makes it “Repeatable”
Role 1
Process Step
1
Role 2
Process Step
2
Role 3
Role 4
Role N
7
Decision
No
Yes
Process Step
3
Process Step
2.5
Process Step
4
Exam Process Map
8
Research Phase
2. RESEARCH
Client
Conference
Client
Competency
Modeling
Analyst
see
Competency
Modeling
detailed
process
Director’s
Office
9
Review analysis
plan
ID SMEs for
developing the
exam
exam design
summary from
competency
analysis system
approved
Update exam
design in
tracking system
GATE 3
Is the exam plan sound?
GATE 2
Is the plan for analysis sound?
Specialist
Team Lead
(see Competency
Modeling detailed
process)
Review exam
history and
develop analysis
plan
Exam plan:
Competencies,
Methods and
Weights
approved
Design review
Recruitment Phase
10
Exam Process
4. APPLICATION REVIEW
Requalifying information
– allow 6 days via “backdoor”
Client
Analyst
Specialist
Recruiter
Monitor
Applications in
NEOGOV
Prior to Close of
Filing, review
applications with
client, extend
filing and revise
recruitment if
needed
CLOSE Filing
Check Bars
in NEOGOV,
DQ where
indicated
Paper Apps:
Retrieve, clockin, manual entry
into NEOGOV
Print and mail
reject notice
Consult and plan
outreach, media,
and budget
See bar sub-process
Team Lead
Director’s
Office
11
Qualify
Applications in
NEOGOV
(consult with
Client where
necessary)
Bar Records
Send DQ
Emails
Exam First Part
5. EXAMINATION, FIRST PART (e.g., T&E or Written) (See detailed process for examination parts.)
Client
Analyst
Prepare TPI
and Materials
for Packet
Prepare
invitation
email/letter
Review and
verify
Assistant
Team Lead
Director’s
Office
12
Review Plan
at Team
Meeting
GATE 5a
Are schedule/technical risks controlled?
Specialist
TPI: Test Preparation
Instructions
Send emails
Schedule
Location,
Dates, Raters
Finalize and
Prepare Packets
Send letters
approved
Finalize Material
Administer
Assessment
(see Proctoring
detailed process)
Scoring First Part
6. SCORING AND PASS POINT SETTING (e.g. Written Test)
Client
Review comment
sheets, proctor
notes, item
analysis.
Analyst
Specialist
Assistant
Scoring Formula
in NEOGOV -Parts and
weights
Scan/Enter in NEOGOV
produce item analysis, sort
candidate comment forms.
Consult w/ SME
on questionable
items
Complete Pass
Point cover
sheet" and
recommend
pass point
Qualify Apps
if not done
previously
Determine
pass/fail
results in
Neogov
Scoring changes
if needed
Send fail
notices
Dismantle
Packet.
Ensure return of
all material
Team Lead
Director’s
Office
13
Set pass point
Receive
Protests (see
appeal
process)
Other Parts
7. EXAM PREPARATION - SUBSEQUENT PARTS (e.g., Interview)
Client
Analyst
Review and
verify
Assistant
Team Lead
Director’s
Office
14
Review Plan
at Team
Meeting
GATE 5a
Are schedule/technical risks controlled?
Specialist
Prepare notice of
results of first
part and next
part invitations
Prepare TPI
and Materials
for Packet
Send emails
Print
Applications.
Finalize and
Prepare Packets
Schedule
Location,
Dates, Raters
Send letters
approved
Finalize Material
Administer
Assessment
(see Proctoring
detailed process)
Final Scoring and List
Check list:
Minimum Qualifications verified
Prior terminations/involuntary separations
Arrest/Conviction status screened
Right-to-work status screened
Employment Bars screened
Diploma, Degree, Transcripts, License, Certificate imaged
Scoring parts and weights verified
Pass points and rescaling verified
Composite based on all rescaled or all percentage scoring
Seniority points added for dual
Veteran points added for entry-open
Eligibility dates verified (merged, 6 mo., 12 mo.)
Selective certification designated and verified
8. FINALIZING SCORING AND LIST PRODUCTION
Client
Analyst
Enter
withholds in
NEOGOV
Collect Confidentiality
Forms, Note those who do
not have essential
documents. Materials
accounting.
Enter scores
into NEOGOV
and Produce
Distribution
Calculate
total score,
add
seniority
points,
calculate
final score
Team Lead
Director’s
Office
15
Set final pass
point
Complete
Eligibility
Checklist
Prepare
final
eligiblity
report
GATE 6
Is the eligibility list ready to promulgate?
Specialist
Update cut-off
score
considerations
form
Determine or
confirm
withholds
Sign-off
Final Report
Promulgation and Review
9. PROMULGATION AND REVIEW
Client
Prepare final
results notices
Analyst
Specialist
Assistant
Send emails
Prepare PC
Agenda
Assist candidates in
understanding
scoring
Send letters
Pull Exam
Folders /Results
for Candidate
Review Session
Candidate Review.
Monitor Review and
provide appeals
forms
Assist
Appeals
Investigation
Schedule review
location and
date
Team Lead
Director’s
Office
16
Receive
protests (see
appeals
process)
Close-out
10. CLOSE OUT PHASE
Client
Update and
Return Files
Electronic
files
transfered to
K:Exam Files
Test
stats & item
changes in
master copies
Complete
Exam Close
Checklist
File Exam
Materials.
Retain for 5
years
Yes
Specialist
Assistant
Competency Modeling
Examination Design
Conference notes
End notes
Team Lead
Director’s
Office
17
Job Analysis
and Exam Plan
File
Determine appeals
disposition, enter
in tracking system
Put on List and
appeals on PC
Agenda
Commissioners
deny appeals?
Accept List?
GATE 7
Is everything properly documented and filed?
Analyst
No
Close exam
in tracking
system
Corrective
Action
Cycle Time Targets
What’s the least time it can take?
WRITTEN
10
INTERVIEW
5
LIST
25 days
(40)
WRITTEN
10
PERFORMANCE
10
INTERVIEW
TE
10
INTERVIEW
5
LIST
20 days
(35)
INTERVIEW
5
LIST
15 days
(30)
10
10
BULLETIN
15
CLOSE OF
FILING
5
10
18
5
LIST
35 days
(50)
CYCLE-TIME MANAGEMENT
19
Exam Cycle Time:
What is it?
20
•
The amount of time in working days
to complete of the process
•
Broken down to key stages of the
process.
Time is a Resource
Resource Requirement = Minimum cycle time to
complete one thing TIMES the number of things in
process
10 actual work days * 10 exams in process
= 100 total days to complete.
Process Efficiency = Resources Allocated / Resource
Requirement
75 days available / 100 days to complete = .75
Actual Cycle time = Resource Requirement / PE
100/75 = 133 days
21
PE and Work in Process
Given finite resources:
PE
1.00
.75
.50
.25
.05
Exams
1
22
2
3
4
5
6
7
8
9 10 11 12 13 14 15
Cycle time and Process Efficiency
Reducing Total Cycle Time involves:



Reducing the number of things in process (fewer exams,
putting exams in queue)
Increasing process capacity (more staff/resources)
Increase process efficiency
Increasing Process Efficiency involves:







23
Standardization of materials and procedures
Leveraging technology and automation
Eliminating steps
Eliminating rework
Eliminating false starts
Eliminating wait-states*
Maybe even working harder or faster
Exam Cycle Time Measurement

Mean and Standard Deviation for each exam phase days
in process and overall
• Prep/Development time
• Recruitment time
• Exam administration time

24
Percentage of exams exceeding “control limits” for each
phase
Regular Exams
Cycle Time Graph (2012-13 actual)
Average of ExamDays
80
Average of RecruitDays
70
Average of PrepDays
12
60
16
14
50
11
40
11
30
10
10
16
15
12
10
15
23
19
22
16
9
18
16
20
10
23
25
22
0
12/07'
8
11
12/08'
12/09'
Prep M=22, SD=20!
25
14
39
17
12/10'
26
25
25
18
12/11'
12/12'
13/02'
Recruit PE = .83
13/03'
13/04'
21
13/05'
17
13/06'
Exam PE = 2.08!
Analysis of Variance

Structural: Exam Types
Open, Promotional, Continuous
 Entry, Professional, Management, Executive
 Complexity (number of exam parts)


Situational: Priority


Logistical:


SME time, Raters, Facilities, etc.
Performance:

26
Urgent, Expedited, Regular, Special
Individual
Time/Process Analysis
Value-added: Time that is spend directly
contributing to the quality of the product
 Business Value-added: Time that is required by
law or regulation (recruitment, notices, reviews,
protests, approvals, documentation, etc.)
 Non-Value-added: Everything else!

• What can be eliminated? Consolidated? Run
concurrently? Automated?
• Where can you squeeze out the dead air?
27
Cycle Time Management
28

Set baseline for “intervals that matter”
 Mean and Standard Deviation
 Start-to-Open, Open-to-Close, Close-to-List

Set control limit (CL) for example:
 Fixed target (theoretic “ideal”)
 One SD >mean
 Top 15% outliers

Identify cases that exceed the CL
 Elevate attention/intervention for these
 Determine root-cause
 Get outliers back on track

Systemically address high-leverage root causes
Variability Reduction and
Cycle Time Reduction
Statistical Process Control (SPC)
Cycle Time
29
Cycle-Time Management Practices
Outlier Review

Track projects weekly – monitor time

30
Interval times displayed in status reports in Exam
Tracking

Escalate when exam exceeds an interval CL (StartOpen, Open-Close, Close-List)

Is there a problem? If so, how can it be addressed?

Follow through

Elevate if challenging or persistent (Director to Director)
Systemic Root Causes Need
Systemic Solutions

Transcend Space and Time:
•
•
•
•
31
Virtual SME
On-line/electronic, auto-scoring T&Es
Video-based interviews and assessments
Un-proctored Internet Testing (UIT)

Establish partnering relationships at executive levels for
rater-sharing (supply-chain management!)

Examination planning template –set reasonable
expectations

Testing room, interview room, proctor supply and
availability
Cycle-Time Management Practices
Consulting Skills







32
Identifying the “customer(s)”
Scoping and prioritize the project
Setting overall target dates for exam milestones
Clarifying roles, responsibilities, and resources (SMEs
and raters)
Asking permission to press for action
Communicating status throughout project – keeping the
project visible and urgent
Pressing for action (tough love)
Consulting Skills
Exam Start-up (triage)

Priority Options (from start-up)
•
•
•
•

Priority Implications
•
•
•
•
•
33
Urgent: 25 days,
Expedited: 40 days
Regular: 60 days
Unclassified
Faster exams require more concentrated effort
Faster exams require fewer parts
Faster exams require fewer assembled parts
Faster exams require no wait time
Impact of a poor hire versus impact of unfilled positions?
Consulting Skills
Key Levers

Contact requesting manager within one working day of receipt of
request
• Take lead in creating urgency

Put “in process” only after the exam is prioritized and timeline set

Open recruitment only after first exam part is ready.
• “Ready” includes establishing availability of administration and scoring
resources for a specific date.
• Raters are lined up.

Use “On-hold” status and “Suspended” status (take out of process)
• Initiate status discussion with customer at second instance of delay or
unavailability
• Cancel exam if in suspended status for more than 1 month. Restart when
time and resources permit

Reprioritize if customer requirements change
• e.g., move from expedited to regular; regular to complex
34
Organization Climate and Culture
Process improvement and cycle-time management
are functions of “employee engagement.”

Involves discretionary effort, creativity, and caring for the
product, service, and customer

Involves shared ownership of the data
• Must be shared, transparent, credible, and “safe”

35
Involves shared ownership of goals, problems, and
successes