Transcript Psychometrics Made Easy

Psychometrics Made Easy
Certification Exam Preparation
January 15, 2015
F. Sam Griggs, Jr., CCP, Cost Engineering Consultant
Introduction
What Is It?
How Did We Get Here?
What Is Its Language?
What is the Process?
How Can You Pass the Exam?
Your Technical Paper
Psychometrics
Technique
Theory
Measurement
Psychometrics
Statistical
Research
Effect on
Measurement
Objective
Measurement
of Skills and
Knowledge
Psychometrics
Key Concepts
are Reliability
& Validity
How Did We Get Here?
8 Hour Hand
Written Exam
Computerized
Testing
Statistical
Data
Psychometrics
Review
Removed
Difficult
Questions Most
Got Wrong
Removed
Easy
Questions
Most Got
Right
Grading:
 To determine a passing grade of any test, it must be justified
with empirical data.
 It should be legally defensible.
 Meet the Standards for Educational and Psychological
Testing.
Glossary of Terms
 Alpha Coefficient
 Angoff Method
Distractor
Item Difficulty
Item Distraction
Cutscore
 Key
Glossary of Terms
 Mean
 Mean P
Mean Rpbis
P-value
Point Biserial Correlation (Rbpis)
Standard Deviation
 Standard Error of Measurement (SEM)
Glossary of Terms
 Reliability
 Validity
Test Bias
Negative
Correlation
SME
William H. Angoff
Distinguished research scientist at
ETS for more than 40 years.
Angoff Method: One of his many
contributions to psychometrics.
Author of “Scales, Norms, and
Equivalent Scores”.
Angoff Method:
Most widely
used
Relies
on SME
Goal
• Determines the passing % or cutscore.
• SME are subject matter experts who examine each question.
• Predict how many minimally qualified candidates would answer
correctly.
• The average of the SME’s predictions becomes its predicted
difficulty.
• The sum of the predicted difficulty values averaged is the
recommended cut score
• Rate each test item
• To determine whether a minimally qualified candidate
would answer the item correctly or incorrectly.
Mechanics of
Psychometrics
Reliability
Validity
Statistics
Reliability
 A Reliable Measure: Measures test items consistently
across time, individuals, and situations.
 Reliability is necessary, but not sufficient for validity.
Alpha
Coefficient
Test
Retest
Validity
 Valid Measure: Measures what it is intended to
measure.
Sample question: What is the capital of Texas?
A valid answer might be Houston.
This question is a reliable measure but would not be a
valid measure for the CCP exam.
A valid measure for a CCP exam must measure only CCP components such as
Cost Estimating, Scheduling, Project Management, etc.
Negative Correlation
 This Negative number indicates that those who did well on the overall
exam actually performed very poorly on this particular item..
Sometimes it indicates that the answer might be keyed incorrectly.
Sample question: What is the capital of Texas?
A valid answer was thought to be “Houston”
and was erroneously marked as correct
answer.
In this case, some candidates who passed the overall
exam did not get this question correct – as marked.
They selected the distractor “Austin”.
This statistic shows that a dis-proportionate number of successful candidates
missed this question. So, those questions with Negative Correlation are
reviewed very closely. Sometimes, candidates “over-think” the question.
Test Bias
 A form of systematic (non-random) error which leads to examinees
from one demographic group having an advantage over
examinees from another demographic group.
Test bias may cause differences in average scores across
demographic groups.
But differences in group scores are not
sufficient evidence that Test Bias is present.
Example: USA uses different system of measurement
than Europe. They use metric system so they may
have problems with our questions re measurement.
AACE Certification Board has committee working on problem – Goal is to
reduce Test Bias.
Statistics
 Both Reliability & Validity can be assessed Statistically.
Statistical Assessments:
P-Value (Pass Rate or Item Difficulty)
Point Biserial Correlation (Rbpis)
Negative Correlation
Alpha Correlation
Easy Test Question
 90% tested answer correctly.
Sample Question:
What is the Capital of Florida?
a. Miami
b. Tallahassee
c. Jacksonville
d. Orlando
Easy Test Question
Sample Question:
What is the Capital of Florida?
a. Miami
b.Tallahassee
c. Jacksonville
d. Orlando
Hard Test Question
 50% tested answer incorrectly.
Sample Question:
What is the Capital of Michigan?
a. Detroit
b. Ann Arbor
c. Lansing
d. Kalamazoo
Alternative is to change
distractor(s)
Sample Question:
What is the Capital of Michigan?
a. Detroit – change to say, Boston
b. Ann Arbor – change to say, Cloquet
c. Lansing
d. Kalamazoo
Hard Test Question
 50% tested answer incorrectly but what if we changed
distractor answers?
Sample Question:
What is the Capital of Michigan?
a. Boston
b. Cloquet
c. Lansing
d. Kalamazoo
Hard Test Question
Sample Question:
What is the Capital of Michigan?
a. Detroit
b. Ann Arbor
c. Lansing
d. Kalamazoo
Sample of Math Question
 Less than 60% tested answer incorrectly.
Sample Math Question:
What is $5,100 invested at 12%
compounded yearly, at end of 6 years?
a. $5,712
b. $8,988
c. $10,066
d. $98,950
Solution to Math Problem
 Solution : There are a couple of different ways to work this.
 Start
Begin
Rate
End
 Year 1
$5,100
1.12
$5,712
 Year 2
$5,712
1.12
$6,397
 Year 3
$6,397
1.12
$7,165
 Year 4
$7,165
1.12
$8,025
 Year 5
$8,025
1.12
$8,988
 Year 6
$8,988
1.12
$10,066
Sample Math Problem Tips for Solution
 There are a couple of different ways to work this example.
 Solution : Work up a Table shown on previous slide.
 Solution : Remember the formulas. There should be some basic
formulas you want to remember.
 Solution : Remember the ‘Rule of 72”. To determine the amount of
time to double an investment, divide 72 by the interest rate. In this
case, 72/12 = 6 years. The $5k should double and maybe answer is
approx $10k.
 Get rid of obviously wrong answers. These would be answer a and d.
Answer a is too small and d is too high. That narrows it down to 2
plausible answers. If you had to guess, odds are 50% instead of 25%.
Certification Board
Review:
Steps in Making the Question Easier or Harder
 Review the question to confirm that the marked answer
is correct.
 Change the wording of the question.
 Modify the distractor to add or decrease the difficulty.
 Make no changes.
 Retire the question.
Top Tips to Get a Great Score
• Understand the purpose of the exam.
Purpose
• Know what to expect in the timed testing
environment.
Expect
• A positive attitude is a most.
Attitude
Top Tips to Get a Great Score
Practice
• Especially necessary for anyone who has never taken a Psychometric Exam.
• Find websites & books that have practice test.
• Focus on tests that specialize in your area(s) of weakness.
• Test yourself working against the clock.
Time
Yourself
Broaden
Sample Test
• Don’t rely only on the study guides.
• Use AACE Recommended Practices (RP) on their web page.
• Many of the new questions on exams come from the new RPs.
Top Tips to Get a Great Score
Refresh Math
Skills
• Fact 1:
• Fact 2:
AACE has Math Questions that other exams do not.
Cost Engineers are expected to know the Rate of Return and other Math
problems.
• Make sure you get a good night’s sleep before the exam so you are fully alert.
Get a Good
Night’s Sleep
• Carefully read or listen to the instructions on taking the exam.
Follow
Instructions
Top Tips to Get a Great Score
Answering Questions
• Trust yourself…Usually your first answer is correct.
• Answer easy questions first, skip those you get stuck on.
• At the end, return to the questions you skipped.
• Do not leave any questions unanswered. Eliminate all obviously wrong choices.
• Last resort…Guess if you need to as you have a 25% chance of getting it right.
• Complete the Feedback Section at the end of the exam.
• Your comments or concerns help the Reviewers modify and improve future exams.
Complete the Feedback
Section on Exam
Don’t Give Up
• Keep trying until you Pass!
• Identify what areas that kept you from passing.
• Study these problematic areas to improve your skills.
Conclusion:
Learn the
Language
Use
Angoff
Method
Implement
Processes
Achieve
Certification
Study &
Prepare for
Exam
Technical Paper
Important information you should know before writing your paper
Must Submit Your Technical Paper Before You Can Take the Exam
Graded Independently of the Certification Exam
CCP’s Need Passing Grades on the Certification Exam and the
Technical Paper for Certification
Know the Basic Guidelines used to determine whether your
Technical Paper is acceptable
Tips for a
Winning
Paper
Make sure your paper Includes:
Table of Contents
1
2
Abstract
3
4
5
Introduction: Purpose & Background
Basis & Analysis of Your Data
References
Tips for a
Winning
Paper
Make sure your paper Includes:
Graphs or Tables to Support Premise
6
7
Organization & Clarity
8
9
10
Logical Conclusion
Do Word Count: Minimum 2,500 Req.
Extra Points for Appendix
Questions??