Slide 1

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 2

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 3

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 4

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 5

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 6

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 7

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 8

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 9

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 10

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 11

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 12

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 13

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 14

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 15

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 16

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 17

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 18

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 19

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 20

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 21

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 22

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 23

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 24

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 25

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 26

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 27

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 28

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 29

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 30

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 31

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 32

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 33

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 34

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 35

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)


Slide 36

Data and the Nature of
Measurement
Graziano and Raulin
Research Methods: Chapter 4
This multimedia product and its contents are protected under copyright law. The following
are prohibited by law: (1) Any public performance or display, including transmission of any
image over a network; (2) Preparation of any derivative work, including the extraction, in
whole or in part, of any images; (3) Any rental, lease, or lending of the program.

Copyright © Allyn & Bacon (2010)

Research Variables


Variable: Any characteristic that can take
on more than one value
– Examples: speed, level of hostility, accuracy of
feedback, reaction time



Research is the study of the relationship
among variables
– Therefore, there must be at least two variables
in a research study (or there is no relationship to
study)

Copyright © Allyn & Bacon (2010)

Measuring Variables


Measurement: Assigning numbers
to indicate the level of a variable
– Sometimes the number assignment is
intuitive (e.g., time measured in seconds)
– Sometimes it is more arbitrary (e.g., 1 for
male and 2 for female)

Copyright © Allyn & Bacon (2010)

Scales of Measurement




Based on how closely the measurement
scale matches the real number system
Scales of Measurement (Stevens, 1946,
1957)
–
–
–
–

Nominal
Ordinal
Interval
Ratio
Copyright © Allyn & Bacon (2010)

Properties of the Abstract
Number System








Identity: Each number has a particular
meaning.
Magnitude: Numbers have an inherent
order from smaller to larger.
Equal intervals: The difference between
units is the same anywhere on the scale.
True zero: Zero indicates there is no
amount of the variable.
Copyright © Allyn & Bacon (2010)

Nominal Scales


Naming scale
– Each number reflects a category
– Examples: diagnostic categories, political
affiliations




Produces nominal or categorical data
Mathematical properties
– Identity

Copyright © Allyn & Bacon (2010)

Ordinal Scales


Scale indicating rank order
– Reflects the order, but not the amount
Examples: order of finish in a race, class
rankings




Produces ordered data
Mathematical properties
– Identity
– Magnitude
Copyright © Allyn & Bacon (2010)

Interval Scales


Scale with equal intervals
– The scale indicates amount, but with no zero
point
– Examples: temperature on the Celsius scale,
most psychological tests




Produces score data
Mathematical properties
– Identity
– Magnitude
– Equal intervals
Copyright © Allyn & Bacon (2010)

Ratio Scales


Scale that fits the number system well
– Includes equal intervals and a true zero
– Examples: time, distance, frequency




Produces score data
Mathematical properties
–
–
–
–

Identity
Magnitude
Equal intervals
True zero
Copyright © Allyn & Bacon (2010)

Psychological Tests


Most psychological test fall somewhere
between an ordinal and a ratio scale
– Ordinal in that the distance between
scores may not be equal
– Ratio in that one can view the test score
as the number of correct items



Norm is to assume such tests
represent an interval scale
Copyright © Allyn & Bacon (2010)

Measurement Error




Decreases the accuracy of
measurement
Possible sources of measurement error
– Response set biases
– Inconsistent measurement procedures
– Sloppy procedures
– Unreliable measures
Copyright © Allyn & Bacon (2010)

Operational Definitions




Specific procedures for measuring
and/or manipulating a variable
Every variable should be operationally
defined
The more careful and complete the
operational definition, and the more
faithfully we follow it, the more precise
the measurement of the variable
Copyright © Allyn & Bacon (2010)

Reliability



The consistency of measurement
Consistency can be conceptualized in
different ways
– Therefore, there are different types of reliability



Usually measured with a correlation
– Covered in Chapter 5
– Sensitive to the consistency of rank orderings of
participants

Copyright © Allyn & Bacon (2010)

Types of Reliability






Interrater reliability: degree of
agreement between two independent raters
Test-retest reliability: degree of
consistency over time
Internal consistency reliability: degree
to which the items of a measure all measure
the same thing

Copyright © Allyn & Bacon (2010)

Perfect Reliability




Reliability is a
measure of
consistency.
Perfect reliability
means that the
scores are perfectly
consistent.

Copyright © Allyn & Bacon (2010)

Good Reliability




Reliability
deteriorates when
the consistency is
lost.
Here the rank
orderings are still
reasonably
consistent.
Copyright © Allyn & Bacon (2010)

Fair Reliability


As reliability
deteriorates further,
the rank orderings
from the two
testings are less
similar.

Copyright © Allyn & Bacon (2010)

Poor Reliability


When you reach
the point where the
rank orderings have
no relationship to
one another, your
reliability (i.e.,
consistency) is
poor.
Copyright © Allyn & Bacon (2010)

Effective Range




The range in which a measure gives an
accurate indication of the level of the
variable
Critical to select measures with an effective
range appropriate for
– Your sample
– Your study
– Example: Using a calculus test to measure math
skills in second graders would NOT work
Copyright © Allyn & Bacon (2010)

Scale Attenuation Effects


Results from a restriction of the range
of the measure
– Therefore, people above or below the
effective range are not measured
accurately



Two types of scale attenuation effects
– Floor effects
– Ceiling effects
Copyright © Allyn & Bacon (2010)

Illustrating These Effects


Situation



– Assume that we are
measuring the
weight of 9 men
using a standard
bathroom scale

Floor Effect
– The needle is stuck
so that it never
reads below 175



Ceiling Effect
– The needle is stuck
so that it never
reads above 200

Copyright © Allyn & Bacon (2010)

Weights for 9 Men
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Floor Effect




Range on the bottom of the scale is
insufficient to measure people who score
near the bottom
If the scale needle was stuck so that it
never read below 175 pounds
– There would be a floor at 175



Error due to this floor effect is shown in
dark blue in the next slide
– Weights recorded for these people are too high
Copyright © Allyn & Bacon (2010)

Floor Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Ceiling Effect




The range on the top of the scale is
insufficient to measure people who score
near the top
If the scale could not read above 200
pounds
– There would be a ceiling at 200 (everyone above
200 would be reported as weighing 200)



Error due to this ceiling effect is shown in
dark blue in the next slide
Copyright © Allyn & Bacon (2010)

Ceiling Effect
300

Weight

250
200
150
100
50
0
1

2

3

4

5

6

Individuals
Copyright © Allyn & Bacon (2010)

7

8

9

Source of These Effects


Result of psychological scale with an
insufficient range to measure the
range of performance in the sample
– Example: a measure of memory that is
either too easy or too hard

Copyright © Allyn & Bacon (2010)

Validity




A scale is valid if it measures what it is
supposed to measure
Validity also refers to how well a scale
predicts other variables
– Example: An IQ test is likely to be a valid
predictor of grades in school.
– When used this way, the scale is called the
predictor measure and the measure predicted is
called the criterion.
Copyright © Allyn & Bacon (2010)

Measuring Validity


Validity is usually measured with a
correlation
– The correlation is between



The measure and
A SPECIFIED criterion measure

– Always list the criterion when reporting the level
of validity


For example, the IQ test is a valid predictor of school
grades, but not a valid predictor of athletic ability.

Copyright © Allyn & Bacon (2010)

Perfect Validity




Validity is the
degree to which
one measure
predicts another.
With perfect
validity, the rank
orderings on the
predictor and
criterion measures
are identical.
Copyright © Allyn & Bacon (2010)

Good Validity


If the rank
orderings from the
predictor and
criterion measures
are similar, you
have good validity.

Copyright © Allyn & Bacon (2010)

Fair Validity


But the less the
rank orderings from
the predictor and
criterion measures
agree, the lower
the validity.

Copyright © Allyn & Bacon (2010)

Poor Validity


When you reach
the point that the
rank orderings of
predictor and
criterion measures
are unrelated, you
have essentially
zero validity.
Copyright © Allyn & Bacon (2010)

Objective Measurement


The hallmark of science
– Objective measures produce the same
result no matter who does the measuring
– Therefore, scientific principles will apply
no matter who tests these principles



Objective measures reduce biases that
could distort results (see Chapters 8 and 9)
Copyright © Allyn & Bacon (2010)

Ethical Principles




Previous discussion focused on
protecting participants
Other ethical obligations
– Present data accurately
– Interpret data fairly
– Never participate in selective withholding
of research data
Copyright © Allyn & Bacon (2010)

Summary






Measuring variables is central to research
Several scales of measurement exist
Reduce measurement error with carefully
developed operational definitions
Enhance reliability and validity of your
measures
The goal is to produce objective, accurate
measures of your variables
Copyright © Allyn & Bacon (2010)