Transcript WELCOME TO ITEM REVIEW - ASQ Section 0912 - Louisville

Pete Cotterill
ASQ CQIA Certification Chair
ASQ Certification Process
An Overview of the ASQ Certification
Program: From Concept; through
Development; to Execution.
What is Certification?
Certification is formal
recognition by ASQ that an
individual has demonstrated a
proficiency within, and a
comprehension of, a specified
body of knowledge. It is peer
recognition, not registration or
licensure.
- Source: ASQ Website
ASQ Certifications: Some are lifetime while
others require renewal every three years.
•
•
•
•
•
•
•
•
•
Quality Engineer – CQE
Reliability Engineer – CRE
Six Sigma Black Belt – SSBB
Manager of Quality/Organizational Excellence – CMQ/QE
Biomedical Auditor – CBA
Certified HACCP Auditor – CHA
Calibration Technician – CCT
Quality Auditor – CQA
Software Quality Engineer – CSQE
•
•
•
•
•
Six Sigma Green Belt – SSGB
Quality Process Analyst – CQPA
Quality Technician – CQT
Quality Inspector – CQI
Quality Improvement Associate – CQIA
Life-time
Qualification to Sit for Exams
Education and Work Experience
• CMQ/QE: Ten years of Education and/or Work Experience (5
years of Work Exp must be in a decision making role)
• CRE/CQE/CSQE/CQA: Eight years of Education and/or Work
Experience (3 years of W.E. must be in a decision making role)
• CBA/CHA: Five years of Education and/or Work Experience
(1 year of Work Exp must be in a decision making role)
• SSBB: 2 completed projects or one completed project and 3 years
Work Exp
•
•
•
•
•
SSGB: Three years of Work Exp
CQI : Two years of Work Exp and a degree or 5 Years Work Exp
CCT: Five years of Education and/or Work Exp
CQT: Four years of Education and/or Work Exp
CQIA/CQPA: Two years Work Exp or a degree
How Are Tests Developed?
STEPS IN TEST DEVELOPMENT
New Cert.
approved by
ASQ BOD
Job Analysis
Workshop
5-year Review
of Existing
Program
Job Analysis
Survey
Test Specifications
Workshop
Item-Writing
Workshop
Enough
Items in
Pool?
No
Yes
Item Pool
Maintenance
Needed?
Item-Review
Workshop
IPM
Workshop
Review
Camera-Ready
Version
Administer Exam
Exam-Review
Workshop
Item Analysis
& Scoring Changes
No
Yes
Yes
Cut-Score Study
New
BOK?
No
Score Exams and
Notify Candidates
STEPS IN TEST DEVELOPMENT
New Cert.
approved by
ASQ BOD
Job Analysis
Workshop
5-year Review
of Existing
Program
Job Analysis
Survey
Test Specifications
Workshop
Item-Writing
Workshop
Enough
Items in
Pool?
No
Yes
Workshops conducted
with between 12 and 20 ASQ
volunteers
Item Pool
Maintenance
Needed?
Item-Review
Workshop
IPM
Workshop
Review
Camera-Ready
Version
Administer Exam
Exam-Review
Workshop
Item Analysis
& Scoring Changes
No
Yes
Yes
Cut-Score Study
New
BOK?
No
Score Exams and
Notify Candidates
What makes a good test?
Tests and the component Questions, need to
be:
• Valid
• Reliable
• Fair
ASSESSMENT CONCERNS:
VALIDITY
VALIDITY:
Evidence that the test is measuring what it says it will
measure.
Processes that provide test validity evidence
include:
Job Analysis 
Test Specifications 
Body of Knowledge (BOK) 
Test Question Classification 
Reference Text & Rationale
ASSESSMENT CONCERNS:
RELIABILITY
Is the behavior being measured consistently?
Consistency is based on
Test Length
Consistency in Test Administration
Clarity of Questions *
Appropriateness of Questions *
* Major Focus of ASQ volunteer Workshops
EQUATERS
Equating =
Fairness to Candidates
Equaters defined:
They are a set of 25-30 items selected from a
previous administration of the exam and placed in
the newly created exam in same position as
previously used. This set of items represent a
“Mini-test” in terms of BOK coverage and
difficulty.
9. What
yield
for ais normally
Whichisofthe
thepercent
following
items
NOT part of
inspection
planning?
distributed process in which the item length
specification is 5.750  0.005, is 5.754, and the
(A) Characteristics
to inspect
standard
deviation is 0.002
?
(B)
Sampling
plans
(A) 19.15%
(C) 30.85%
Cost of inspection
(B)
(D)
Calibration status
(C) 50.00%
(D) 69.15%
10. In
preparation
for construction
of a
The
extent to which
an instrument
replicates
result
when measurements
cause
andits
effect
diagram,
it is important toare
taken repeatedly on the same unit is called
13. The accuracy of a profilometer is verified with
(A) fingernail comparators
(B) surface plates
(C) calibration patches
(D) optical flats
14. The
a material to resist
Whatability
is theofrecommended
minimum number of
mechanical
penetration
a test ofthe limits for a
subgroups necessary
toiscalculate
control chart?
(A) surface finish
(B)
(A) hardness
10
(C)
tensile strength
(B) 15
(D)
(C) porosity
25
(D) 35
OCTOBER 2003
EXAM
(A)
charts for each source
(A) plot
real separate
bias
(B)
only on what makes things go
(B) focus
precision
(C) wrong
accuracy
(C)
record
everything people suggest
(D) true value
(D) validate possible root causes
15. Which of 2003
the following measures
is a
types of micrometers
is
Equated to MARCH
sufficient
statistic for
the parameter
?
best for detecting
external
out-of-roundness
on a
11. Inspection accuracy increases with
(A) reductions in defect rates
(B) repeated inspections
(C) additional product complexity
(D) order of sample selection
12. Which of the following describes
tools wouldthe
be best
for
measuring
projected
surface
intersections?
Deming method for continuous
improvement?
(A) Cost
Optical
(A)
of comparator
quality analysis
(B)
Thread
micrometer
(B) Process map
(C) Tree
Profilometer
(C)
diagram
(D)
Protractor
(D) Plan-do-check-act cycle
cylindrical part?
(A) Median
(B)
(A) Mid-range
Blade
(C)
Mean
(B) Three-point
(D)
(C) Mode
V-anvil
(D) Depth
16. Which of the following tools can only be used
to collect attributes data?
(A) Dial indicator
(B) Ring gage
(C) Micrometer
(D) Steel scale
Mar 2003 Exam
1. 9999.1254
2. 9999.5412
3. 9999.6578
4. 9999.0324
5. 9999.0154
October 2003
6. 9999.0111
Equated to
7. 9999.4521
March 2003
8. 9999.3256
9. 9999.4514
10. 9999.0547
11. 9999.0978
12. 9999.0877
13. 9999.0364
14. 9999.0254
15. 9999.0574
16. 9999.6574
Oct 2003 Exam
1. 9999.4567
2. 9999.1571
3. 9999.6578
4. 9999.1865
5. 9999.1954
6. 9999.0998
7. 9999.4521
8. 9999.1244
9. 9999.4514
10. 9999.9145
11. 9999.8748
12. 9999.1754
13. 9999.0364
14. 9999.1654
15. 9999.7457
16. 9999.4214
October 2003
Equated to
March 2003
Oct 2003
Mar
2003Exam
Exam
1. 9999.4567
9999.1254
2. 9999.1571
9999.5412
3. 9999.6578
4. 9999.1865
9999.0324
5. 9999.1954
9999.0154
6. 9999.0998
9999.0111
7. 9999.4521
8. 9999.1244
9999.3256
9. 9999.4514
10. 9999.9145
9999.0547
11. 9999.8748
9999.0978
12. 9999.1754
9999.0877
13. 9999.0364
14. 9999.1654
9999.0254
15. 9999.7457
9999.0574
16. 9999.4214
9999.6574
EQUATERS Continued…
100 ITEM TEST WITH 25 EQUATERS
MARCH 2003 ADMIN
Total Exam
Mean score = 80
OCT 2003 ADMIN
Total Exam
Mean score = 70
Is there a difference in the tests
or a difference in the groups?
Mini Exam (equaters)
Mini Exam (equaters)
Mean score = 20
Mean score = 20
EQUATERS
 It is VERY important that Equaters not be
revised or moved within the exam.
 Think of it as an experiment in which this is the
one variable that can be truly controlled.
ITEM INFORMATION
NUMBER 8000.0554
REV: 001
ANS: A
ST: T-tried, A-accept, H-hold, R-raw, D-delete
CLASSIFICATION: 1 A 1
TYPE: NG RN MT RO
EXAM: 0/0/00
EDIT:1/8/00
EQUATER: 2 or 1-Equater, 0-non-Equater
AUTHOR: GRIFFITH, GARY K.
REF: QUALITY HANDBOOK ED. 5 1998 PP. 2.1-2.2
All Items are Coded to the lowest
level in the BOK
IV. AUDIT REPORTING (10 questions)
A. Review and Finalize Audit Results
B. Written Report Format and Content
1. Audit details
2. Compliance
3. System effectiveness
4. Conclusions to be reported
5. Request for corrective actions
Items are classified to the lowest sub-topic level, e.g., 4B3
If there is no sub-topic level, a zero is used, e.g., 4A0
Tested Item and its Stats
NUMBER: 8000.0273 REV: 001 ACC:
OVLP: 0000 ANS: D ST: T L4X: 2
CLASSIFICATION: 2 C 1 TYPE:
EXAM: 12/04/04 EDIT: 01/20/04
AUTHOR: Scholtes, Joiner and Streibel
REF: The Team Handbook ed 2 p 4-37
The Test Question (The Stem)
a)
Distracter
b)
Distracter
c)
Distracter
d)
Key
PERCENT RESPONDING CORRECT BY QUINTILE
MATRIX OF RESPONSES BY QUINTILE
12/04/04 [001]
A B C
D
E BLANK
5TH:
*
5TH:
5
9
0 90
0
0
4TH:
*
4TH:
3 13
2 76
0
0
3RD:
*
3RD: 5 20
2 77
0
0
2ND:
*
2ND: 22 26
3 42
0
0
1ST:
*
1ST: 31 29 10 22
0
3
+----+----+----+----+----+----+----+----+----+----+
PROB: 0.13 0.20 0.03 [0.63] 0.00 0.01
0 10 20 30 40 50 60 70 80 90 100
DSCR:
0.65
Stats on an “Easy” question
PERCENT RESPONDING CORRECT BY QUINTILE
MATRIX OF RESPONSES BY QUINTILE
12/06/03 [000]
A B C
D
E BLANK
5TH:
*
5TH: 81
0
0
0
0
0
4TH:
*
4TH: 79
1
2
0
0
0
3RD:
*
3RD: 72
1
0
1
0
1
2ND:
*
2ND: 73
3
3
0
0
0
1ST:
*
1ST: 68
5
4
1
0
0
+----+----+----+----+----+----+----+----+----+----+ PROB: [0.94] 0.03 0.02 0.01 0.00 0.00
0 10 20 30 40 50 60 70 80 90 100
DSCR: 0.13
Stats on an “Hard” question
PERCENT RESPONDING CORRECT BY QUINTILE
06/05/99 [000]
5TH:
*
5TH:
4TH:
*
4TH:
3RD:
*
3RD:
2ND:
*
2ND:
1ST:
*
1ST:
+----+----+----+----+----+----+----+----+----+----+ PROB:
0 10 20 30 40 50 60 70 80 90 100 DSCR:
MATRIX OF RESPONSES BY QUINTILE
A
B
C
D E BLANK
79
97 16 218
0
1
99 116 22 104
0
1
123 126 30 118
0
1
118 87
32 100
0
5
103 115 39 110
0
3
0.28 0.29 0.07 [0.35] 0.00 0.01
0.23
Stats on an question with “Issues”
PERCENT RESPONDING CORRECT BY QUINTILE
MATRIX OF RESPONSES BY QUINTILE
12/04/04 [000]
A
B
C
D
E BLANK
5TH:
*
5TH:
46
41
17
0
0
0
4TH:
*
4TH:
33
44
16
1
0
0
3RD:
*
3RD: 36
41
23
4
0
0
2ND:
*
2ND:
41
36
13
3
0
0
1ST:
*
1ST:
43
23
22
6
0
1
+----+----+----+----+----+----+----+----+----+----+ PROB: [0.41] 0.38 0.19 0.03 0.00 0.00
0 10 20 30 40 50 60 70 80 90 100
DSCR: -0.02
Questions?
Contact Information
Pete Cotterill
[email protected]