Transcript Personality Retest Effects: Guilt as a Mechanism for
Personality Retest Effects: Guilt as a Mechanism for Managing Response Distortion
Jill E. Ellingson, Eric D. Heggestad, and Erin E. Coyne
October 13 – 14, 2006 ETS Technical Advisory Group Meeting
Current Retesting Policy
• Applicants allowed to voluntarily retake assessment after period of time if displeased with outcome – Applicant elects to retake the assessment • Common in organizations which use assessment tools for hiring • Most often used in conjunction with cognitively loaded assessments
Personality Assessment Retesting
• Organization directs certain applicants whose responses are likely distorted to retake the personality assessment • Responses deemed distorted on basis of embedded intentional distortion scale – Flags extreme response profiles • Applicants informed that responses were flagged as suspect • Hiring decisions made using retested scores
Key Questions
• Does retesting flagged applicants lower previously inflated personality scale scores?
• What psychological mechanism operates within applicants to help explain why they would adjust their responses?
Scale Score Changes
• Flagged applicants have positively biased score profiles • Retest effect evident in degree to which second assessment scores are lower • Preliminary research suggests that scores may be lowered up to 0.7 standard deviation units (Ellingson & Heggestad, 2003)
Hypothesis 1:
Retesting flagged individuals will result in decreased personality scale scores in the second assessment relative to the first assessment.
Role of Guilt: Appraisal Theory
Event
Evaluation Factors
Relevance?
Congruence?
Associated values?
Accountability?
Coping potential?
Emotion Behavior
Role of Guilt: Applicant Appraisal
Told to retest
Evaluation Factors
•Personally relevant •Incongruent •Violates personal standards •Personally accountable •Coping potential?
Hypothesis 2:
Retesting flagged individuals will result in increased feelings of guilt in the second assessment relative to the first assessment.
Guilt Behavior
Role of Guilt: Applicant Appraisal
Told to retest
Evaluation Factors
•Personally relevant •Incongruent •Violates personal standards •Personally accountable •Make reparation Guilt
Hypothesis 3:
The level of guilt reported by flagged individuals in the second assessment will moderate the degree to which personality scale scores change.
Respond honestly
Sample and Measures
• 288 undergraduate students • Measures: – NEO-Five Factor Inventory (NEO-FFI) – Balanced Inventory of Desirable Responding Impression Management scale (BIDR-IM) – Personal Feelings Questionnaire-2 Guilt scale (PFQ2-G) – Positive and Negative Affect Schedule-Expanded (PANAS-X) – Test of Self-Conscious Affect-3 (TOSCA3)
Procedure
All participants:
1. Completed the TOSCA3 2. Completed NEO-FFI and BIDR-IM under motivating instructions 3. Completed the PFQ2-G and PANAS-X regarding feelings had while taking the personality measure Sorted participants into 3 groups based on BIDR-IM score
Low Control Group
•Low BIDR-IM score
High Control Group
•High BIDR-IM score Retested for neutral reason
Flagged Retest Group
•High BIDR-IM score Told responses were suspect and unusable Asked to retest
Effect Sizes
NEO-FFI Extraversion Conscientiousness Agreeableness Openness Emotional Stability Average
d
Low Control 0.36
0.04
0.12
0.04
-0.10
0.09
High Control 0.29
0.22
0.26
0.10
0.05
0.18
Flagged Retest 0.45
0.59
0.56
0.28
0.27
0.43
Impression Management State Guilt -0.03
0.20
-0.13
0.33
0.54
0.07
Positive values indicate that Time 1 score was larger than Time 2 score.
Repeated-measures MANCOVA: Personality Scales
Within-subjects factor (Time): Time 1, Time 2 Between-subjects factor (Condition): Low Control, High Control, Flagged Retest Covariate: Trait Guilt Pillai’s Trace
F df p
Source
Within-subjects effects
Time Time x Trait Guilt Time x Condition
Between-subjects effects
Trait Guilt Condition 0.018
0.010
.170
0.093
0.255
1.045
.577
5.232
5.739
8.229
(5, 280) (5, 280) (10, 562) (5, 280) (10, 562) 0.392
0.718
0.000
0.000
0.000
η 2 0.018
0.010
0.085
0.093
0.128
Repeated-measures ANCOVA: Personality Scales
Within-subjects factor (Time): Time 1, Time 2 Between-subjects factor (Condition): Low Control, High Control, Flagged Retest Covariate: Trait Guilt NEO-FFI Extraversion Conscientiousness Agreeableness Openness Emotional Stability
F
Time η 2 2.32
0.008
Within-subjects effects Time x Guilt
F
η 2 Time x Condition
F
η 2 0.06
0.000
2.20
0.015
0.95
1.30
2.62
0.003
0.005
0.009
0.01
0.01
1.01
0.000
0.000
0.004
16.78* 7.84* 6.61* 0.106
0.052
0.044
2.45
0.009
1.55
0.005
12.59* 0.081
Between-subjects effects Trait Guilt
F
η 2
F
Condition η 2 1.10
0.004
0.75
0.005
4.02* 1.24
8.21* 0.014
0.004
0.028
17.25* 33.07* 0.38
0.108
0.189
0.003
6.26* 0.022
7.94* 0.053
*
p
< .05
Repeated-measures ANCOVA: Agreeableness Interaction
68 66 64 62 60 58 56 54 52 50 48 High Control Flagged Retest Low Control
Tim e 1 Tim e 2
Repeated-measures ANCOVA: State Guilt
Within-subjects factor (Time): Time 1, Time 2 Between-subjects factor (Condition): Low Control, High Control, Flagged Retest Covariate: Trait Guilt Pillai’s Trace
F df p
Source
Within-subjects effects
Time Time x Trait Guilt Time x Condition
Between-subjects effects
Trait Guilt Condition 0.004
0.002
0.004
81.353
244.106
1.274
.466
.537
4.500
6.752
(1, 283) (1, 283) (2, 283) (1, 283) (2, 283) 0.260
0.495
0.585
0.035
0.001
η 2 0.004
0.002
0.004
0.016
0.046
Moderated Regressions: Understanding Score Change
Predictors Step 1 Time 1 Score Step 2 Time 2 State Guilt Low Control Condition Step 3 High Control Condition Guilt x Low Control Guilt x High Control Extraversion .838* -.083
.097
.087
.074
.077
Full Model Predicting Time 2 Personality Scale Scores Standardized Beta Coefficients and Variance Explained Conscientiousness .888* -.195* .176* .143* .104* .123* Agreeableness .829* -.291* .064
.101* .225* .098* Openness .910* -.138* .106* .085* .135* .058† Emotional Stability .850* .141* -.163* -.114* -.072
-.077† R-squared Δ R-squared .688
.004
.730
.010* .726
.018* .826
.007* .722
.004
* p
< .05
†
p
< .10
Moderated Regressions: Conscientiousness Interaction
Moderating Effect of State Guilt on Time 2 Conscientiousness Scores
0.3
0.2
0.1
0 -0.1
-0.2
-0.3
Low Control High Control Flagged Retest
Low State Guilt High State Guilt
Conclusion
• Retesting flagged applicants will result in a set of personality scale scores that are less positively inflated • The appraisal profile of guilt helps explain this effect – Flagged applicants who feel guilty as a result of being retested decrease their scores in response.