Transcript Document

Chapter 4A
Validity and Test
Development
Basic Concepts of Validity
• Validity must be built into the test from the
outset rather than being limited to the final
stages of test development.
• Validity defines the meaning of test scores.
• Test validation is a developmental process
that begins with test construction and
continues indefinitely.
Validity: A Definition
• Standards for Educational and
Psychological Testing: A test is valid to
extent that inferences made from it are
appropriate, meaningful, and useful.
• Traditionally, the different ways of
accumulating validity evidence have been
grouped into three categories: Content
validity, Criterion-related validity, Construct
validity.
Content Validity
• In theory, content validity is really nothing
more than a sampling issue.
• If the sample (specific items on the test) is
representative of the population (all
possible items), then the test possesses
content validity.
• Content validity is a useful concept when a
great deal is known about the variable that
the researcher wisher to measure.
Content Validity
• Content validity is more difficult to assure
when the test measures an ill-defined trait.
• Quantification of Content Validity:
determining the overall content validity of a
test from the judgments of experts.
• However, it cannot identify nonexistent
items that should be added to a test to
help make the pool of questions more
representative of the intended domain.
Content Validity
• Face validity is not really a form of validity
at all
• A test has face validity if it looks validity to
test users, examiners, and especially the
examinees.
• Face validity is really a matter of social
acceptability and not a technical form of
validity in the same category as content,
criterion-related, or construct validity.
Criterion-Related Validity
• Criterion-related validity is demonstrated when a
test is shown to be effective in estimating an
examinee’s performance on some outcome
measure (criterion).
• In concurrent validity, the criterion measures are
obtained at approximately the same time as the
test scores.
• In predictive validity, the criterion measures are
obtained in the future, usually months or years
after the test scores are obtained.
Characteristics of a Good criterion
• A criterion is any outcome measure
against which a test is validated.
• A good criterion must also be reliable,
approximate, and free of contamination
from the test itself.
• The theoretical upper limit of the validity
coefficient is constrained by the reliability
of both the test and the criterion.
rxy  (rxx )  (ryy )
Characteristics of a Good criterion
• Criterion contamination
• A criterion must also be free of contamination
from the test itself.
• Criterion contamination is also possible when
the criterion consists of ratings from experts.
• When validating a test against a criterion of
expert ratings, the test scores must be held in
strict confidence until the ratings have been
collected.
Concurrent Validity
• Correlations between a new test and
existing tests are often cited as evidence
of concurrent validity. But two conditions
must be satisfied:
• 1. the criterion (existing) tests must have
been validated through correlations with
appropriate nontest behavioral data.
• 2.the instrument being validated must
measure the same construct as the
criterion tests.
Predictive Validity
• Validity coefficient and the standard error
of the estimate:
SEest  SDy 1  r
2
xy
Construct Validity
• A construct is a theoretical, intangible quality or
trait in which individuals differ.
• All psychological constructs possess two
characteristics in common:
• 1.There is no single external referent sufficient to
validate the existence of the construct; that is ,
the construct cannot be operationally defined.
• 2.Nontheless, a network of interlocking
suppositions can be derived from existing theory
about the construct.
Approaches to Construct Validity
•
•
•
•
•
•
•
Test Homogeneity
Appropriate Developmental Changes
Theory-Consistent Group Differences
Theory-Consistent Intervention Effects
Convergent and Discriminant Validation
Factor Analysis
Classification Accuracy
Extravalidity Concerns and the
Widening Scope of Test Validity
• Psychologists confirm that the decision to
use a test involves social, legal, and
political considerations that extend far
beyond the traditional questions of
technical validity.
• Unintended side effects of Testing
The Widening Scope of Test
Validity
• A general definition:
• A test is valid if it measures “what it
purports to measure.”
• A recent claim:
• A test is valid if it serves the purpose for
which it is used. Test validity, then, is an
overall evaluative judgment of the
adequacy and appropriateness of
inferences and actions that flow from test
scores.
Structural Equation Model
• Website:
• http://www.ssicentral.com/lisrel/downloads.
html