Transcript Approaches to Psychological Measurement

Measurements and
Validity
Julia Braverman, PhD
Division on Addictions
Types of measures
ANXIETY??
Michael
John
Types of measurement
1.
2.
3.
Objective/Physiological measures
•
•
Bodily activity, nervous system.
Response time
Observational measures
•
Direct observing participants.
Self-report
•
Participants provide information about
themselves.
Converging operations
Using several measurement approaches
to measure a particular variable
Basics of psychometrics: How to
build a trait/state assessment
measure?

Concept
• Affect
• E.g. I feel sad
• Behavior
• E.g. I cannot sleep, I cry a lot
• Cognition
• E.g. I think about suicide.

Question format (Likert scale, yes/no,
reverse scale)
Measure quality
1.
2.
Reliability
Validity
Reliability

The degree of consistency between
observations made by the same
measurement tool.
Measurement Error.
No measure is perfect.

Observed score =
True score + Measurement error.

True score – is the score that the
participant would have obtained if our
measure were perfect.
Sources of measurement errors
1.
2.
3.
4.
5.
Transient states
•
Mood, health, anxiety
Stable attributes
•
Suspicious participant may distort their answers
Situational factors
•
Weather outside, baseball game.
Characteristics of the measure
•
E.g. instruction ambiguity
Actual mistakes
Assessing reliability
1.
Test-retest reliability
•
•
•
Measuring the same thing twice.
Reliability = correlation ( r) between
results of the first and the second
measurements.
High reliability > .70
Assessing reliability
1.
Test-retest reliability
•
Problems
• Memory
• Experience
Assessing reliability

Interitem Reliability - Measure of
consistency among the items on a scale.
1. Item-total correlation
For each item how it is correlated with the sum of
other items. >. 30
2. Split-item reliability
Divide the items on the scale into 2 sets and test
the correlation (instead of test-retest).
3. Cronbach’s alpha coefficient
Average of all possible split-half reliabilities.
Benevolent sexism scale:
1 (disagree) – 7 (agree)
1.
2.
3.
4.
5.
6.
Women should be cherished and protected
by men.
Women, compared to men, tend to have a
superior moral sensibility.
Men should be willing to sacrifice their own
well-being in order to provide financially for
the women in their lives.
Many women have a quality of purity that few
men possess.
A good woman should be set on a pedestal
by her man.
Men are complete without women.
Made-up table of item-total
correlations
Item #
r
1
.7
2
.5
3
.9
4
.6
5
.7
6
.4
Made-up table of item-total
correlations
Chronbach α = .85
Item #
r
Chronbach α
(without the item)
1
.7
.8
2
.5
.8
3
.9
.7
4
.6
.8
5
.7
.8
6
.4
.9
Assessing reliability

Interrater reliability – consistency
between two or more raters or judges
who observe the same behavior.
•
High reliability > .70
Increasing the Reliability
Measures
1.
2.
3.
4.
Standardize administration of the measure
•
Same test conditions
Clarify instructions and questions.
•
•
To reduce ambiguity and misinterpretations.
Pretest questionnaires if possible.
Train observers.
•
To increase interrater reliability.
Minimize error in coding data.
Validity

If the measurement actually measures
what it is supposed to measure
•
•
Different from reliability
Same measure maybe valid for one purpose
and invalid for another one.
Assessing validity
1.
Face validity – if a measure appears
to be valid.
•
•
•
Does not mean actual validity.
E.g. SAT reading comprehension test
• Does it measure reading comprehension or
common sense? (Katz et al., 1990)
Affect motivation to participate?
Assessing validity
2.
Construct validity
 Relation to other measures.


Convergent validity

High correlation with conceptually relevant
measures.
Discriminate validity
• Low correlation with conceptually unrelated
constructs
Assessing validity
3.
Criterion-Related validity – the
correlation between the measure and
some current behavior.
•
•
E.g. IQ and GPA
Doctor’s productivity
• Peer evaluation
• Patient evaluation
Assessing validity
3.
Predictive validity – the ability of a
measure to predict a certain
behavior/situation in a future.
•
•
•
E.g. SAT and GPA or GPA and after-college
salary.
Doctor’s productivity
?
Reliability and Validity


If reliable
•
May be valid or not.
If not reliable
•
Not valid
Threats to measurement validity

Using non-validated measures

Solution
• Validate the measure
• Use pre-validated measures
Threats to measurement validity

Loose connection between theory and
method.
• Disagreement between conceptional and
operational definitions.
• E.g. putting more pepper as a measurement of
aggression?
• Solution
• Validate your measure with previous
measurements
Threats to measurement validity

Social desirability (evaluation apprehension) –
Desire to look “normal” or to be judged
favorably by another person (including the
experimenter).

Solutions
•
•
Anonymity
Ask indirect questions
• “How many drinks an average college student have
during a party?”
Threats to measurement validity

Yes-bias
Extreme-score bias

Solution

• Reverse score.
• Z-transformation within an individual.
Threats to measurement validity


Testing effects
•
Most participants perform better on a test of
personality/behavior/IQ measure the second time they
take it.
Reasons
•
•
•
•
Learning (e.g. IQ test)
Practice (e.g. physical skills)
Learn the test goal (e.g. personality test)
Attitude polarization
• Thinking about their attitudes
Threats to measurement validity


Testing effects
Solutions
• Control group
• No pretest
• Long waiting period
Validity of experiment

Internal validity
• Extent to which a study provides evidence of
a cause-effect relationship between the
variables.

External validity
• The ability to generalize results of the
experiment.
Internal validity

3 conditions to determine causality

Low internal validity – the conclusion that
A affects B is wrong.
• Covariation
• Temporal sequence
• No confounds
Threats to internal validity

Role demands – participants’
expectations to what an experiment
requires them to do
• Good-subject tendency
• E.g. hypnosis and antisocial acts
• Participants reactance
• E.g. What is the weather today?
Threats to internal validity

Role demands

Solution
• Cover story
• E.g. Independent studies
• Add non-relevant tasks, items (For
measurements)
Threats to internal validity

Experimenter bias

Solution
• E.g. Gratitude study
• Double-blind
Threats to internal validity

Hawthorne effect – Increases in
productivity that occur when participants
know they are being studied.
• Workers responded to any change in working
conditions by working harder than usual.

Solution
• Control group
Common Threats to Internal
Validity of Quasi-experiments








History
•
Something occurred between the pretest and posttest.
Maturation
•
Normal time changes
Regression to the mean
•
If extreme scored Ss. were selected.
Pretest sensitization
•
Pretest affects the posttest results
Selection bias
•
Comparison groups differed from the beginning
Local history
Contemporary history
Attrition/mortality
•
•
Only most motivated participants stay
Only participants who experience less adverse effects of treatment stay
External validity

How well the findings of an experiment
generalize to other situations or
populations.
Threats to external validity

Other subjects

Other times
Other settings

• Sampling/selection bias
Threats to external validity

Sampling bias
• Motivated volunteers
• Those available (at home, have phone)
Threats to external validity

Other setting
• Artificial experimental environment
External validity


External validity - the ability to generalize
results of the experiment.
Tight control - highly specific and artificial
situation -> less external validity.
Internal validity
External validity

You are a researcher. In your experiment, you
assign the first 20 people in your study to the
experimental condition and the second 20 people to
your control condition. This could pose a
threat to:
•
•
•
•
Internal validity
Reliability
External validity
Construct validity

Saying that some measure is ________
definitely means it is also __________.
• valid, reliable
• reliable, valid
• nominal, numerical
• observational, self-report
• none of the above

An experimenter wants to examine if a new behavioral
intervention program increases compliance among
hypertension patients. For this purpose she recruits
hypertension patients with low medication compliance
and tests their compliance before and after the
intervention. What are the potential threats to internal
validity:
•
•
•
•
•
Regression to the mean
Maturation
History
Pretest sensitization
All of the above
Find a threat/threats to internal
validity

The Alzheimers Center wants to evaluate the
effectiveness of their support groups for caregivers of
individuals with Alzheimers Disease. The caregivers are
given the choice when they first come to the center as to
whether they want to join these support groups. The
center gives a stress measure to the caregivers that
attend these weekly meetings, once they have attended
meetings for three months. They also administer the
same stress measure to the caregivers who have not
attended the support groups, as a control group. Both
groups of caregivers are married to the person with
Alzheimers disease and both groups have been involved
with the center for the same length of time
Any questions?