Transcript Applied Research Methods for Counselors

Appraisal and Its Application to
Counseling
COUN 550
Saint Joseph College
For Week # 4
Copyright © 2004 by R. Halstead. All rights reserved.
Class Topics
Validity
Administering Tests & Communicating Results
Scoring & Interpreting Tests
Norms
Standard Scores
I was just looking at the
tree across the street. The
distance variance of the
fallen leaves from the
trunk of the tree looks like
they are normally
distributed. Seeing that
phenomenon is like looking
into the eyes of Dog!!
Test Validity
Validity - a means of expressing the degree to
which a certain inference drawn from a test is
appropriate and meaningful.
Stated more simply, validity helps us answer the
following question: Does a test measure what it
purports to measure?
The Multidimensional Nature of
Validity
Validity - a means of expressing the degree to
which a certain inference drawn from a test is
appropriate and meaningful.
There are two different types of inferences drawn
from tests:
Descriptive inferences and Predictive inferences
Because validity, in a sense, is multidimensional in
nature, one must use a variety of methods to
establish if a test is valid.
Categories of Validity
Most test manuals refer to three categories of
validity
Content Validity
Criterion Validity
Construct Validity
Categories of Validity
Content Validity - The items of a test represent
that which they are supposed to measure.
Content Validity is often supported by expert
judgment.
Face Validity is also a term used in some test
manuals to express that a panel of judges has held
that test appears to measure what it purports to
measure.
Categories of Validity
Criterion-Related Validity - A test score is related
to outcome criteria.
There are two elements addressed when considering
the Criterion Validity
Concurrent Validity - the test score agrees (concurs)
with other valid measures of the same construct.
Predictive Validity - the test score is able to accurately
predict performance within some domain that it
purports to measure.
Criterion-Related Validity
Concurrent Validity
Concurrent Validity is established through the use
of correlation.
“The scores on the Aggressiveness Scale correlated .70
with the rating of teachers of students in their
classes.”
“The scores on the Beck Hopelessness Scale correlated
.81 with the Beck Depression Inventory.”
Criterion-Related Validity
Predictive Validity
If a test is used to estimate criterion scores at some
point in the future, predictive validity must be
established.
Predictive Validity - the test score is able to
accurately predict performance within some
domain that is purports to measure.
The SAT score, for example, that one establishes in
high school is used to predict performance level during
one’s first year performance level in college.
Criterion-Related Validity
Predictive Validity
Some Examples:
The SAT Verbal scores have been shown to
correlate (.40) with first year students’ grade
point average at the end of the first semester at
SJC.
The Spatial Relations Scale correlated .70 with
success in the mental fabricating training for
technical high school students.
Criterion-Related Validity
Predictive Validity
Aptitude and intelligence tests offer evidence
regarding predictive validity.
When one looks at these tests it is a good idea to
examine the utility of predictive tests.
Below is a model for considering predictive tests
False Positive
Positive
Negative
False Negative
Criterion-Related Validity
Concurrent and Predictive Validity
Concurrent and Predictive Validity
To establish either concurrent or predictive
validity it is critically important to make
certain that valid, reliable, and relevant
measures of the criterion are used.
Construct Validity
Construct - is a mental construction of some
grouping of variables or behaviors (e.g. anxiety,
locus of control, cognitive ability).
Construct Validity - establishes that a test score
expresses and accurate measure the construct in
question.
Construct Validity
Because constructs are mental constructions we
must derive Construct Validity
This is done by examining sets of evidence to build
a case for establishing the test's validity.
Convergent Validity - high positive correlation with
other measures of the same construct
Discriminate Validity - low correlation with other
measures of a different construct
Multitrait-multimethod Approach
to Construct Validity
Multitrait-multimethod Approach
1) compute r for same measure of same trait same
method
2) compute r for same measure of same trait different
method
3) compute r for measure of different trait same method
4) compute r for same measure of different trait
different method
Multitrait-multimethod Approach to
Construct Validity - An Example
Lets suppose we want to establish construct validity
for a new depression scale
1) compute r for measure with one that measures the
same trait - Beck Depression Scale
2) compute r for measure with a different measurement
method same trait - Clinical Interview
3) compute r for measure with one that measures a
different trait - Snyder’s Hope Scale
4) compute r for measure with a different trait (Hope)
and different method - Clinical Interview
Awareness and Orientation
Codes of ethical standards remind test
administrators of their responsibility for the
orientation of takers.
Orientation should describe the purpose of
testing, content areas measured, method of
administration as well as reporting and use of
scores
7 topics to cover in orientation
Purpose
Criteria used for selecting the test
Conditions under which test to be taken
Range of skills or domains to be measured
Administrative procedures and concerns
Types of questions on test and overview
Type of scoring, method, + schedule for reporting
Administering the Test
Important to deliver instructions in test manual
as they are stated. Follow the sequence and
timing of the instructions so that your client gets
that same information as those in the group on
which the test was that normed . Goal for the
results to give a valid picture of attributes
measured. Examiner should record critical
Incidents that deviate from normal conditions.
Posttesting Procedures
Recording Test Behavior – checklists/behaviors
Major Issues and Problems in Test Administration
awareness/orientation phase very important in
eliminating probs
Examiner and Bias – communication skills/attitudes +
expectations/gender/competence/test ethics =
examiner bias
Feedback on Test and Test Administration – Ask test
taker.. how would you rate physical environment,
performance, comfort with the administration,
time allotted adequate, orientation helpful, fair
test?
Communicating Results to Clients, Parents, and
Professionals
Guidelines & Standards for Communicating Test
Results
Know test manual, limits of test, informed
consent procedures, protect rights
Postinterview opportunity to deal w/
interpretation and its use in planning and
decision making
Methods for Reporting Test Results
5 major methods
Individual Sessions: discussion
Group Sessions
Written reports
Interactive Approaches
Video Approaches: current trend
Areas of Consideration
Acceptance: goal of feedback session is acceptance of
test results and incorporate info into decision
making
Readiness of client: critical factor in acceptance =
client readiness
Negative results
Flat Profiles: look at other things, interests, values,
goals, etc.
Motivation and Attitude: more significant when
motivated
Scoring Tests
Primary models for scoring tests
Cumulative – Number of items endorsed
Class – Serves to categorize or describe the person
Ipsative – How a person performed on a set of variables
Alternate & Authentic Assessment
Holistic Scoring –individual judgment with model answers
Interpreting Tests
Criterion-Referenced Tests describes the
specific types of skills, tasks, or knowledge
that the test taker can demonstrate.
Norm-Referenced Tests compare each person’s
score with that of the norm established for that
test.
Norms
What establishes a norm?
Answer: Norms are based on the occurrence of the
majority, or bulk, of values within a distribution
obtained from some defined sample of
individuals (Think of the Normal or Bell Curve).
Norms are used to give information about
performance relative to what has been observed
within a sample.
Age-Related Norms
Some normative groups have been established for
particular ages.
Once age-related norms have been established
tracking is possible.
Tracking is a process by which, with some level of
confidence, one can expect to see specific
characteristics in a sample of interest. Examples:
Developmental activities of infants, Grade based reading
levels, Age based capacity for task of increasing
cognitive complexity
Within-Group Norms
Establishing norms is very easy to do.
It is important to keep mindful that the science of
testing is not perfect.
One must be cautious, therefore, about assuming
that a test is a valid measure for everyone taking
it.
If a test provides an accurate measure for only
some members of a population accuracy of
within-group norms must be questioned.
Percentile, Percentile Ranks,
Quartiles, and Deciles
Percentiles - divide the total frequency for a set of
observations into hundredths
Percentile Rank - establishes that point below which a
certain percent of scores fall
Quartile - divide the total frequency for a set of
observations into quarters
Q1 = 25% Q2 = 50% Q3 = 75% Q4 = 100%
Deciles - divide the total frequency for a set of
observations into tenths
Standard Scores
One of the problems with establishing a mean and
standard deviation for a specific distribution is the
limitations to the meaning of such measures.
Standard scores are a means of presenting the
relative position of an individual on a test that are
tied to the normal curve.
There are two major forms of standard scores that
you will encounter.
The Z Score and the T Score
Z Scores
x-X
Z =
s
Z Scores are based on an mean of 0 and a standard
deviation of 1.
Z Score Table - Appendix 1 - Salkind
48 - 42.6
Z=
= 1.5
3.6
Z Scores and the Normal Curve
Z Score Table - Appendix 1 in Salkind
Z Scores give us a opportunity to establish a better
understanding of where individual scores occur
relative to standardized norms.
T Score
T Scores - are standard scores using a fixed mean
and standard deviation in units that eliminate the
need for decimals and signs. On many tests the
arbitrary mean or fixed mean is 50 and the
arbitrary standard deviation is 10.
T = s(z) + { X}
T = 10(-.50) + 50
T = - 5 + 50
T = 45