Transcript Improving and Assessing the Quality of Behavioral Measurement

Chapter 5:
Improving and Assessing the
Quality of Behavioral
Measurement
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Indicators of Trustworthy
Measurement
• Validity
– Directly measures a socially significant
behavior
– Measures a dimension of the behavior
relevant to the question
– Ensures the data are representative
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Indicators of Trustworthy
Measurement
• Accuracy
– Observed values match the true values of
an event
• Reliability
– Measurement yields the same values
across repeated measurement of the same
event
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Threats to Measurement
Validity
• Indirect measurement
– Measuring a behavior other than the
behavior of interest
– Requires inferences be made about the
relationship between those behaviors
– Must provide evidence that the behavior
measured is directly related to behavior of
interest
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Threats to Measurement
Validity
• Measuring a dimension that is irrelevant
or ill suited to the reason for measuring
behavior
• Measurement artifacts
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Measurement artifacts
• Misleading data that result from the way
behavior is measured:
– Discontinuous measurement
– Poorly scheduled observations
– Insensitive or limiting measurement scales
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Threats to Measurement
Accuracy and Reliability
• Human error
• Poorly designed measurement systems
– Cumbersome
– Difficult to use
– Complex
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Threats to Measurement
Accuracy and Reliability
• Inadequate observer training
– Explicit and systematic
– Careful selection
– Train to competency standard
– On-going training to minimize observer drift
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Threats to Measurement
Accuracy and Reliability
• Unintended influences on observers
– Observer expectations of what the data
should look like
– Observer reactivity when she/he is aware
that others are evaluating the data
– Measurement bias
– Feedback to observers about how their
data relates to the goals of intervention
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Assessing the Accuracy and
Reliability of Behavioral
Measurement
• First, design a good measurement
system
• Second, train observers carefully
• Third, evaluate extent to which data are
accurate and reliable
– Measure the measurement system
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Assessing the Accuracy of
Measurement
• Accuracy means the observed values
match the true values of an event
• No one wants to base research
conclusions or treatment decisions on
faulty data
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Assessing the Accuracy of
Measurement
Four purposes of accuracy assessment:
1. Determine if data are good enough to
make decisions
2. Discovery and correction of measurement
errors
3. Reveal consistent patterns of
measurement error
4. Assure consumers that data are accurate
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Accuracy Assessment
Procedures
• Measurement is accurate when
observed values match true values
– Accuracy determined by calculating
correspondence of each data point with its
true value
– Process for determining true value must
differ from measurement procedures
– Accuracy assessment should be reported
in research
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Assessing the Reliability of
Measurement
• Measurement is reliable when it yields
the same values across repeated
measures of the same event
– Not the same as accuracy
– Reliable application of measurement
system is important
– Requires permanent products for remeasurement
– Low reliability signals suspect data
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Using Interobserver Agreement
to Assess Behavioral
Measurement
• The degree to which two or more
independent observers report the same
values for the same events
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Benefits of Interobserver
Agreement (IOA)
• Determine competence of new
observers
• Detect observer drift
• Judge clarity of definitions and system
• Increase believability of data
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Requisites for IOA
• Observers must:
– Use the same observation code and
measurement system
– Observe and measure the same
participants and events
– Observe and record independently of one
another
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Methods for Calculating IOA
• Percentage of agreement is most
common
• Event Recording methods compare:
– Total count recorded by each observer
– Mean count-per-interval
– Exact count-per-interval
– Trial-by-trial
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Methods for Calculating IOA
• Timing recording methods:
– Total duration IOA
– Mean duration-per-occurrence IOA
• Latency-per-response
• Mean IRT-per-response
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Methods for Calculating IOA
• Interval recording and Time sampling:
– Interval-by-interval IOA (Point by point)
– Scored-interval IOA
– Unscored-interval IOA
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Considerations in IOA
• During each condition and phase of a
study
• Distributed across days of the week,
time of day, settings, observers
• Minimum of 20% of sessions, preferably
25-30%
• More frequent with complex systems
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Considerations in IOA
• Obtain and report IOA at the same
levels at which researchers will report
and discuss in study results
– For each behavior
– For each participant
– In each phase of intervention or baseline
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Considerations in IOA
• More conservative methods should be
used
• Methods that will overestimate actual
agreement should be avoided
• If in doubt, can report more than one
calculation
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Considerations in IOA
• Believability of data increases as
agreement approaches 100%
• History of using 80% agreement as
acceptable benchmark
– Depends upon the complexity of the
measurement system
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Considerations in IOA
• Reporting IOA
– Narrative form
– Table
– Graphs
• In all formats, report how, when, and
how often IOA was assessed
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved
Assessing the Quality of
Measurement
• Indicators of the quality of data include:
– IOA
– Accuracy
– Reliability
• Can report multiple indices to assess
data quality
Cooper, Heron, and Heward
Applied Behavior Analysis, Second Edition
Copyright © 2007 by Pearson Education, Inc.
All rights reserved