Transcript Leveraging Technology in the Selection Process

Leveraging Technology
in the Selection Process
Alfredo Rocha and Eric Rowen
Personnel Commission
Los Angeles County Office of Education
Presented to: PTC-SC, February 25, 2015
Current Process Improvements
1. Interview Rater Briefing Video
• Standardizes briefing process
• Length: 12 minutes
Current Process Improvements
1. Interview Rater Briefing Video
• Standardizes briefing process
• Length: 12 minutes
2. Video-Rated Interviews (VRI)
• Reduces length of interview schedule
• Increases control and standardization
3. Written Exam Pass Point Excel Workbook (RQS)
4. Rater App
RQS Version 1
Written Exam Pass Point Excel Workbook
• Automated statistical information
− Measures of central tendency
− Measures of variability
− Percentile scores
− Absolute floor
− Correlation coefficients
− Data-driven histograms
• Pass Point Survival tool
• Adverse Impact Analysis tool
RQS Version 1
Disclaimer
• This workbook is provided as a professional courtesy
• Any use in an operational context is solely the
responsibility of the user agency
• Unknown “bugs” or deficiencies may exist or develop
with continued use
• LACOE makes no performance assurance or guarantee
for this Excel-based tool
RQS Version 1
Candidate Results
RQS Version 1
RQS Version 1
Candidate 1
16
17
12
45
Candidate 2
12
14
14
15
13
14
19
18
19
18
6
19
17
18
16
32
52
49
52
47
3
4
5
6
RQS Version 1
Test Characteristics
RQS Version 1
Test Characteristics
20
15
5
20
RQS Version 1
Test Characteristics
Section/Test Reliability
• KR-20
• Cronbach’s Alpha
RQS Version 1
Test Characteristics
Section/Test Reliability
• KR-20
• Cronbach’s Alpha
RQS Version 1
Exam Statistics
RQS Version 1
Exam Statistics
Math
Reading
Safety
Overall
RQS Version 1
Exam Statistics
Math
Reading
Safety
Overall
RQS Version 1
Exam Statistics
Math
Reading
Safety
Overall
RQS Version 1
Exam Statistics
Math
Reading
Safety
Absolute floor can rule out someone
passing by just guessing!
Overall
RQS Version 1
Correlations
RQS Version 1
Correlation Coefficients
RQS Version 1
Correlation Coefficients
Relationship Strength
0 - 0.29 = None – Weak
0.3 - 0.49 = Moderate
0.5 - 0.69 = Strong
0.7 - 1 = Very strong – Perfect
RQS Version 1
Look at the distribution
of scores to
z
establish norm-referenced pass points.
Bimodal
Distribution
Normal
Distribution
Skewed
Distribution
Data-Driven Histograms
RQS Version 1
Can use SMEs to establish
criterion-referenced pass points!
Angoff Method
RQS Version 1
Pass Point Survival
RQS Version 1
Pass Point Survival
Math
Reading
Safety
Overall
RQS Version 1
Pass Point Survival
RQS Version 1
Pass Point Survival
RQS Version 1
Pass Point Survival
RQS Version 1
Adverse Impact Analysis
RQS Version 1
Adverse Impact Analysis
Example #1:
Out of 26 males
taking an exam, 16
pass.
Out of 19 females,
14 pass.
RQS Version 1
Adverse Impact Analysis
Example #1:
Out of 26 males
taking an exam, 16
pass.
Out of 19 females,
14 pass.
Example #2:
White: 4 pass, 2 fail
Black: 2 pass, 10 fail
Hispanic: 3 pass, 8 fail
Asian: 8 pass, 3 fail
Native American:
2 pass, 4 fail
RQS Version 1
Adverse Impact Analysis
What if another
group had a
higher selection
ratio?
Example #2:
White: 4 pass, 2 fail
Black: 2 pass, 10 fail
Hispanic: 3 pass, 8 fail
Asian: 8 pass, 3 fail
Native American:
2 pass, 4 fail
Instead of 8
passing,
only 5 had
passed?
RQS Version 1
Adverse Impact Analysis
What if another
group had a
higher selection
ratio?
Example #2:
White: 4 pass, 2 fail
Black: 2 pass, 10 fail
Hispanic: 3 pass, 8 fail
Asian: 8 pass, 3 fail
Native American:
2 pass, 4 fail
This results in
different
calculations
RQS Version 1
Limitations
• Not all encompassing…yet
• Allows for only 5 test sections and 999 candidates
• May require scrolling to view RQS sections
• Data-driven histograms may not work well with some
types of tests
• Possible issues with workbook after continued use
(Keeping a master copy is necessary!)
• All features may not be supported by previous versions
of Excel
User feedback is welcome!
Rater App
Created by Eric Rowen
& LACOE PC staff, 2014
• Microsoft Access database application
• Used with any competency-based examination
(T&E, Performance, Writing Projects, Interviews, etc.)
• Goal: Simple, yet powerful tool to collect & store ratings
Microsoft Access Basics
Initial Challenges
• Learning how
Access works
• Finding relationships
between Access
objects and how
data links together
• Invaluable resources
– GCFLearnFree.org
– Microsoft Support
– MSDN
– Stack Overflow
Image Credit to GCF LearnFree.org
– YouTube
Microsoft Access Basics (cont.)
• Most difficult parts: setting table relationships and
queries for the data
Image Credit to GCF LearnFree.org
Database Design
Development Goals
• Minimize preparation and output time
– Also remove potential transfer errors from the process
• Increased access to information for Analysts
• Reduce paper use in administration and archiving
Application Limitations
• Limited to use with LACOE rating procedures
– Comp Model 3G, Rating Scale, Weighted Sum Scores , NEOGOV
• Designed for use on our servers & hardware
• NOT available to share (yet) – contact our director 
The Rater App Demonstration
LACOE Experience Takeaways
• Consistently positive feedback
• Trouble with user error and network connections
– Many issues should be fixed with new hardware
• Requires agency-wide training (analysts & specialists)
• Continuous improvement is essential
– ALWAYS adding new features based on user feedback
– Application bugs are inevitable
• Multiple administrators if possible
Thank you!
Questions?
Connect with us on
!