Transcript Reliability Definition: The stability or consistency of a test. Assumption: True score = obtained score +/- error. ~ Domain Sampling Model ~ Item Domain Test.

Reliability
Definition: The stability or consistency of a test.
Assumption: True score = obtained score +/- error.
~ Domain Sampling Model ~
Item Domain
Test
Test-retest method
[error is due to changes occurring due to the passage of time]
Test 1
-------------
r
Test 1
-------------
Issues:
•
Length of time between test administrations if crucial (generally, the
longer the interval, the lower the reliability)
• Memory
• Stability of the construct being assessed
• Speed tests, sensory discrimination, psychomotor tests (possible fatigue
factor)
Parallel/Alternate Forms
[error due to test content and perhaps passage of time]
Test 1
-------------
r
Test 2
-------------
Issues:
• Need same number & type of items on each test
• Item difficulty must be the same on each test
• Variability of scores must be the same on each test
Two types:
1) Immediate (back-to-back administrations)
2) Delayed (a time interval between administrations)
KR-20 and Coefficient Alpha [error due to item similarity]
• KR-20 is used with scales that have right & wrong
responses (e.g., achievement tests)
• Alpha is used for scales that have a range of response
options where there are no right or wrong responses (e.g.,
7-point Likert-type scales
KR-20
Rtt = k
k–1
% of those getting
the item correct
Σ pi (1 – pi)
σy
2
# of items variance of test scores
variance of scores
on each item
Alpha
ά = k
1 – Σ σ i2
k–1
# of items
σ y2
variance of
test scores
Factors Affecting Reliability
1) Variability of scores (generally, the more variability, the
higher the reliability)
2) Number of items (the more questions, the higher the
reliability)
3) Item difficulty (moderately difficult items lead to higher
reliability, e.g., p-value of .40 to .60)
4) Homogeneity/similarity of item content (e.g., item x total
score correlation; the more homogeneity, the higher the
reliability)
5) Scale format/number of response options (the more options,
the higher the reliability)
Types
ofof
Validity
Types
Validity
Content Validity [the extent to which test
items represent the domain]
a) Subject Matter Expert Opinions (e.g.,
CVR statistic)
b) Internal consistency reliability
c) Correlation with other similar tests
Types of Validity (cont.)
Criterion-related Validity
Concurrent
Correlation between test scores
and performance scores
collected at the same time (e.g.,
correlating test scores with
existing performance scores of
employees)
• Motivation level
• Guessing, Faking
• Job experience factor
• Range restriction issue on
performance scores
Predictive
Correlation between test
scores and performance
scores after some time
interval has passed (e.g.,
correlating test scores of
applicants and subsequent
performance scores collected
6 months to a year later)
• Range restriction issue on
performance scores
• Time, cost, & pragmatic
concerns
Types of Selection Tests
• Application Blanks (Weighted Application Blanks,
Biographical Information Blanks)
• Honesty (Integrity) Tests, Drug Tests
• Interviews (e.g., Situational)
• Aptitude Tests (e.g., Mechanical, Clerical)
• Work Samples
• Assessment Centers (Situational Exercises)
Application Blanks
• Content of items (use of job analysis)
• Number of application blanks (one for each position or job category)
• Legal issues
• Image of organization (e.g., format, recruitment issue, perceived fairness
of questions)
• Accuracy of data
Education (e.g., 21%), salary (22%), job title (24%), years worked (29%)
College students willing to include one lie on AB (95%); 45% had done so
Frequency of Common Inappropriate Application Blank Questions
Worded
Appropriately
Not Asked
Item
Not
Appropriate
Past salary
98.9%
0%
1.1%
Minimum acceptable salary
72.7%
0%
27.2%
Reference source
59.1%
0%
40.9%
Age
54.5%
37.5%
8.0%
relatives
50.0%
10.2%
39.8%
Conviction records
43.2%
28.4%
28.4%
Health
40.9%
2.3%
56.8%
Military service
30.7%
30.7%
38.6%
Marital status
27.3%
0%
72.7%
25.0%
43.2%
31.8%
residence
23.9%
0%
76.1%
Physical description, photo
19.3%
0%
80.7%
Rent or own car or home
18.2%
0%
81.8%
Handicap
17.0%
6.8%
76.2%
Organizational membership
15.9%
21.6%
12.5%
Work schedule
13.6%
63.6%
22.7%
Information about
Who to notify in case
of emergency
Length of time in
~ Honesty
Testing ~
Types of Questions:
1)
Frequency and extent of theft (e.g., What percentage of people take more than $1.00
per week from their employer?)
2)
Punitiveness toward theft (e.g., should a person be fired if caught stealing $5.00?)
3) Thoughts about theft (e.g., Have you ever thought about taking company merchandise
without actually taking any?)
4)
Perceived ease of theft (e.g., How easy would it be for a dishonest person to steal
from an employer?)
5) Likelihood of detection (e.g., What percent of employee thieves are ever caught?
Validity Issues:
a) Correlations with polygraph results
b) Future behavior (e.g., # days with cash shortage, discharges)
c) Admissions of past theft
d) Shrinkage reduction
e) Contrasted groups (e.g., scores by criminals vs. general population scores)
Drug Testing
• What does a positive drug test score indicate?
Some Issues:
• Those being tested
Applicants, employees or both
• Testing procedure
Random or for cause
• Type of company
Public or private
• Type of test
Sensitivity, Cross-reactivity
(TLC, EIA/RIA, GCMS)
• Type of job
Safety concerns or not
~ Reference Checks ~
(Exceptionally common technique; e.g., 95% usage by organizations)
In-Person (e.g., interview)
• Costly, time consuming
• Used in jobs that involve the concern for risks (e.g., security, $)
• Can elicit different types of information (differences
between in-person and written reference information)
Mail (or e-mail)
• Low return rate (e.g., 56 – 64%)
• Standardized questions, format
• Written record of responses
• Ensure confidentiality of responses (signed statement by applicant)
~ Telephone Checks ~
(More frequently used than written references)
• Allows follow-up or clarification of answers given
• Less resistance to giving certain types of information can be collected
• Quick process
• Important data can be gleaned from various verbal cues (e.g., pauses,
hesitations, voice inflections, voice level, intonations)
• Relatively high return rate
• Better responsiveness, more interactive nature of the method
• More confidence in the identity of responder
Sample Biographical Information Blank Items
• During high school, how many times did you make the honor roll?
• How much freedom or independence did your parents allow you in grade school?
• How important did your favorite high school teachers stress discipline in the
classroom?
• How many times did you change schools before you were sixteen years old?
• Compared to other people in high school, how many friends did you have?
• How old were you when you spent your first week (or more) away from your
parents?
• How bothered are you if you a job is left undone?
• How often do you read craft and mechanics magazines?
• How quickly do you normally work?
• How well do you feel you can understand the feelings of others?
• How well do you tolerate performing routine tasks?
~ Employment Interview ~
• Frequently used to make selection decisions (over 90%)
• Social exchange (interpersonal) process
• Search for information
COMMON PROBLEMS WITH THE “TRADITIONAL” INTERVIEW
•Variety of Interviewer Biases
* 1st Impressions
* Expectancy Effect
* Contrast Effect
* Stereotype Matching
•Different Questions Asked to Applicants
(Lack of standardization)
•Disagreement on the Desirability of Interview Responses
•Little Formal Interviewer Training
•Subjective (or no) Scoring System
•Interview Conducted and Scored by One Person
•Poor Reliability, Validity, and Job Relevancy (Open to Legal Challenge)
Overview of Situational Interview Process
1)
Perform a Job Analysis Using the Critical Incident Technique
2)
Place Critical Incidents into Relevant Job Dimensions (e.g., Safety, Responsibility,
Interpersonal Skills)
3)
Reword Critical Incidents Into Question Format
Incident: The employee was married for a year and a half and used any excuse to
stay home. One day the employee’s children got colds and no one was around to care
for them. So, the employee didn’t show up for work and didn’t phone in.
Question: Your two teenage children are home in bed sick with colds. No friends or
relatives are available to watch them. Your shift starts in two hours. What would you
do in this situation?
4)
Decide on the desirability of responses [Think of how good, average, and mediocre workers
would have answered such a question]
_____
1
Stay home
_____
_____
_____
3
Phone in &
explain the problem
_____
5
Go in, they just
have colds
5) Conduct interviews in groups of two or more. Each interviewer scores applicant
independently. A single score is given after group discussion
Differential Aptitude Test (DAT)
Verbal Reasoning
..… is to water as eat is to …..
..... is to night as breakfast is to …..
A.
B.
C.
D.
E.
A.
B.
C.
D.
E.
continue ----- drive
foot ----- enemy
drink ----- food
girl ----- industry
drink ----- enemy
….. is to one as second is to …..
A.
B.
C.
D.
E.
two ----- middle
first ----- fire
queen ----- hill
first ----- two
rain ----- fire
supper ----- corner
gentle ----- morning
door ----- corner
flow ----- enjoy
supper ----- morning
Differential Aptitude Test (DAT)
Numerical Ability
Add
30
20
A. 8
B. 15
C. 16
D. 26
N. none of these
Add
13
12
A. 14
B. 16
C. 25
D. 59
N. none of these
Differential Aptitude Test (DAT)
Abstract Reasoning
PROBLEM FIGURES
ANSWER FIGURES
A
B
C
D
E
Differential Aptitude Test (DAT)
Abstract Reasoning (cont.)
PROBLEM FIGURES
ANSWER FIGURES
A
B
C
D
E
Differential Aptitude Test (DAT)
Mechanical Reasoning
A
B
Which weighs more?
(If equal, mark C.)
Differential Aptitude Test (DAT)
Space Relations
A
B
C
D
Differential Aptitude Test (DAT)
Spelling
A) man
B) Gurl
C) Catt
D) dog
Language Usage
1) I just / left / my friends / house.
A
B
C
D
2) Ain’t we / going to / the office / next week?
A
B
C
D
3) I went / to a ball / game with / Jane.
A
B
C
D
Differential Aptitude Test (DAT)
Clerical Speed and Accuracy
V. AB AC AD AE AF
W. aA aB
BA Ba Bb
X. A7 7A
B7 7B AB
Y. Aa Ba
bA BA bB
Z. 3A 3B
33
B3 BB
~ Work Sample Tests ~
(performing a piece, or sample task, of the job)
~ Assessment Center Process ~
Candidates Participate in
Situational Exercises
Sample Group Exercises
Leaderless Group Discussion
Business Game
Sample Individual Exercises
Interview Simulation
Scheduling Exercise
In-Basket
Observed, discussed, and
scored on various
dimensions (e.g.,
communication,
decision making,
planning/organizational
skills) by trained raters
• Overall scores
computed and
ranked for
personnel
decisions
• Indiviudaul
dimension scores
used for
developmental
purposes
Types of Selection Tests (cont).
Ability Tests
Sensory (e.g.,
hearing, vision)
Motor (e.g.,
dexterity,
strength, agility)
ADA concerns
• Reasonable accommodation
• Essential job duties
Cognitive (e.g.,
Intelligence)
Wonderlic Personnel Test
~ “Big 5” Personality Factors ~
• Extraversion --- Outgoing, sociable
• Neuroticism (Emotional Stability): Depressed, anxious,
worrisome, etc.
• Agreeableness: Flexible, forgiving
• Conscientiousness: Careful, thorough, persevering
• Openness to Experience: Curious, imaginative
 Overall, conscientiousness and extraversion are best
predictors of managerial performance across jobs
 Personality measures add to prediction above and beyond
other commonly used measure such as cognitive ability
Self-Report Inventories
~ Myers-Briggs Type Indicator ~
Introversion
Extraversion
(Internally motivated; enjoys spending time
prefers one-to-one communication)
(Action-oriented; motivated enjoys alone;
by outside world and social interactions)
Sensing
Intuitive
(Desires concrete information; practical;
in orientation)
(Imaginative, creative factual and detailed
improvises comfortable with ambiguity;
focuses on contexts and connections)
Thinking
(Searches for facts and logic in a decision situation
Focus on tasks and work to be accomplished
Easily able to provide an objective and critical analysis)
Judging
Feeling
(Sensitive to other’s needs, Seeks
consensus, does not like conflict)
Perceiving
Plans before acting, breaks tasks down
Is comfortable to acts without planning,
into subsets, uses deadlines to accomplish tasks
Enjoys working on multiple tasks at the
same time, enjoys flexibility in work.
Sample Myers-Briggs Items
•
•
•
•
When you go somewhere for a day, would you rather:
a)
plan what you will do and when, or
b)
just go
In a large group, do you more often:
a)
introduce yourself, or
b)
get introduced
Is it harder for you to adapt to:
a)
routine, or
b)
constant change
Do you think it is a worse fault to be
a)
unsympathetic
b)
unreasonable
Literal ______________________ Figurative
Forgive _____________________ Tolerate
Impulse _____________________ Decision
Thinking --- Feeling
Very Clear Clear
30
25
Moderate Slight
20
15
10
5
T
Slight Moderate
5
10
Clear
15
20
Very Clear
25
30
F
Holly (7)
Steve (10)
Paul (2)
Bill (5)
Sally (2)
Frank (1)
Betty (19)
Thinking
• Looks at the logical consequences
of actions;
• Examines pros and cons of
approaches objectively;
• Energized by critique and analysis;
• Focuses on tasks
• Desires to find a standard to apply
in all situations; reasonable and fair
Feeling
• Focuses on how others may be affected by
decisions by mentally placing themselves into
situations;
• Guided by personal values;
• Energized by appreciating and supporting
others;
• Focuses on interactions
• Strives for harmony and positive interactions
Types of Selection Tests (cont).
• Personality Inventories
Self-reports -(e.g., Minnesota Multiphasic
Personality Inventory (MMPI),
California Psychological Inventory
(CPI), Myers-Briggs Type
Indicator (MBTI), Personality
Inventory (HPI), NEO PI-R
NEO (assesses the 5-Factor
Model consisting of: Neuroticism,
Extraversion Openness,
Agreeableness Conscientiousness)
Projective Techniques -(e.g., Thematic
Apperception Test (TAT),
Rorschach Inkblot Test
(RIT)
Rorschach Inkblot Test
Ten cards which bilateral and symmetrical inkblots
Scoring --Location: the part of the blot used (e.g., use of the
whole blot, common or unusual detail)
Determinants: form, color, shading, and movement)
Content (human figures, animal figures, anatomical
diagrams, inanimate objects
Thematic Apperception Test
TAT --- 31 pictures that depict a variety of social and interpersonal
situations. Participants are requested to write or tell a story about each
picture to the examiner (e.g., what happened, what Ten pictures are
gender-specific; the others can be used with either sex.
Use: To uncover internal conflicts, dominant drives, interests, and
motives. Specific motives include the need for achievement, need for
power, the need for intimacy, and problem-solving abilities.