Transcript UCF Academy

Session 3:
Framework for a Coding Guide
October 27, 2009
2:00-3:30 PM (EDT)
Instructor: Chad Nye
1
Life’s Elixir of
Youth, Beauty,
Wisdom &
Painless Coding
Dr. Chad’s Wonder Drug
2

A wholesome blend of natural extracts of
thirty different herbs and root vegetables
that promotes and revitalizes the capacity
for learning and memory, strengthens the
nerves, and effectively wards off
cataleptic neuroplexy AND coding
anxiety.
3
Inclusion and Exclusion Criteria

Reduce Reviewer Bias: Criteria are determined
before beginning Information Retrieval

‘Standardize’ the process and resulting decisions
4
Inclusion and Exclusion Criteria:
Potential Categories

Study Design

Population

Intervention

Outcomes

Measures

Publication Sources
5
Coding for Fun and Profit
Dr. Chad’s Wonder Drug taken 2 times a
day will result in the complete life
experience of the life well lived!!!
6
Coding is a 2 Step Process

Stage 1 – Title and Abstract


Eliminate studies that are clearly inappropriate
Used a limited set of criteria
7
Inclusion Criteria
Abstract & Title Stage

Studies will meet at least one of the following criteria:
1. Participant

The participant sample of study must include only adults 18
years or older, with a diagnosis of Autism Spectrum Disorder.
2. Intervention

The focus of the study intervention must center on the topic of
employment.
8
Inclusion Criteria
Abstract & Title Stage
Studies can be advanced to
the next stage if:
1.
They meet the criteria OR
2.
If these criteria are not clear from the
title or abstract
9
Coding is a 2 Step Process

Stage 1– Title and Abstract



Eliminate studies that are clearly inappropriate
Used a limited set of criteria
Stage 2– Full Text


Identify the studies that will be included in the review itself
Use full set of criteria
10
Inclusion Criteria
Full Text Stage
Research Design
Studies use an experimental (RCT), quasiexperimental design (QED), and single subject
experimental design (SSED)
Participants
Individuals with an autism spectrum disorder who
are 18 years of age or older and not enrolled in
a secondary school program (e.g., high school)
11
Inclusion Criteria
Full Text Stage
a. Research Design
b. Participants
c. Intervention
The treatment must be directed towards
the provision of vocationally oriented
services intended to produce
employment outcomes for individuals
with autism spectrum disorders.
12
Inclusion Criteria
Full Text Stage
a. Research Design
b. Participants
c. Intervention
d. Outcome Measures

The study must report subsequent attainment of an
employment placement and provide specific data about
the duration and/or retention of that placement.
Employment will be considered to encompass full or
part-time (25% time or more) placements. Interventions
of any length duration will be included provided adequate
13
description is provided.
Inclusion Criteria
Full Text Stage
a. Research Design
b. Participants
c. Intervention
d. Outcome Measures
e. Publication Status

Eligible studies may be either
published or unpublished, written in
any language, and conducted in any
country.
14
Inclusion/Exclusion Decision Tree
Potentially relevant
reports identified
(n=583)
Reports excluded by
title and abstract review
(n=523)
Reports progressing to
next level
(n=60)
Reports used to code
studies for inclusion
(n=20)
Reports excluded at
Phase II screening
(n=40)
15
Study Coding*
Purpose of Coding
1. Identify all those variables that you believe
might make difference in the observed results.
2. Organize those individual variables into
categories of characteristics in order to
evaluate the quality of the study
3. Try to determine what is really important about
the study
4. Quantify both text and data for analysis
16
Coding Form
5. Guide for locating and recording the who,
what, where, how, when, and how much
of a study (aka-variables)
6. A record of the ‘interpretation’ of the
variables that can be independently
replicated (aka-a monitor on the
accuracy of judgments)
7. Provides the basis for the data
arrangement at the analysis stage
17
Study Coding
Categories of Study Characteristics
Design
Participant
Treatment
Outcome
18
Design Characteristics
Type
of Design
 RCT
 QED
 Pre-Experimental
 Single Subject
 Blinding
 Attrition
 Sampling Method
19
Participant Characteristics
Information is collected for BOTH
Experimental AND Control groups.
Number of Participants
 Age
 Gender
 Severity
 Language
 Setting

20
Treatment Characteristics
Type of Tx Method
 Length of Tx Program
 Tx Mode: Group-Individual
 Number of Tx Sessions
 Length of Tx Sessions
 Dosage

21
Outcome Characteristics





Outcome Variable(s)
Outcome Measure(s)
Assessor of Outcome
Assessor Qualifications
Time Post Tx to Outcome Measure
 Immediate
 Follow-Up
22
CodeBook

A dictionary of all the terms used in the
Coding Form

A guide for all coders to follow to reduce
error and bias

A reference source for accurate replication
23
A. Publication Source
1. Journal
2. Dissertation
3.Thesis
4. Unpublished Paper
5. Book 6. Other: ________
B. Subject Characteristics (page(s): ____________)
Group Pretest Posttest Attrition
Age
Grade(s)
(n)
(n)
(n)
(n)
(n)
Males
(n)
Tx:
_22___ _22__
______ ______ ______
_____
Cp:
______ _____
______
______ ______
_____
Co:
_22___ __18_
__ 4___
______ ______
_____
24
C. Socioeconomic Status (page(s):___________
1. Low
2. Middle
3. Upper
4. Low-Middle 5. Middle Upper 6. Labeled Mixed
7. Unclear
8. ___________
9. NR
Comments:__________________________________
25
D. Sample Source (page(s)_________)
1. Public School
2. Secular Private School
3. Religious Private School
4. Other:_________________
5. NR
Comments
26
E. Study Setting (page(s) ______)
1. Urban
2. Suburban
3. Rural
Geographic Setting: ______________________
Comments:
27
F. Author’s Labeling of Participants’ Achievement Levels
(page _____)
1. Average Achieving 2. High Achieving 3. Underachieving
4. At-Risk
5. Unclear
6. Other:___________________
7. NR
G. Race/Ethnicity (page ______)
1. African American 2. American Indian
4. Hispanic/Latino
3. Asian
5. White
6. Other: ____________________
7. NR
28
H. Intervention Characteristics
1. Type of intervention (page __________________)
2. Total length of intervention program: __________ or NR
(page _____)
3. Length of time PER intervention session:________ or NR
(page _____)
4. Number of sessions: ________ per ________
(page _____)
or NR
5. Describe the intervention(s) intended for delivery to the study
groups including how and when it was
administered.______________________________________
29
6. Fidelity of implementation data reported? (page(s)______)
0=No
1=Yes
If yes, describe the intervention(s) intended for delivery to
the study groups including how and when it was
administered:
________________________________________________
________________________________________________
________________________________________________
________________________________________________
7. Intervention implemented as described? (page(s) _____)
1=No
2=Yes 3= Somewhat 9=NR
30
I. Outcome Measure(s)
Type of Measure
Instrument
Variable(s) Who Administered the
Measurement Instrument?
1. Norm Referenced
Test (page____)
_________ a. Parent
b. Teacher
_________ c. Researcher d. Other
_________ e. NR
2. Criterion
Referenced Test
(page____)
_________ a. Parent
b. Teacher
_________ c. Researcher d. Other
________ e. NR
4. Rating/Survey
(page____)
_________ a. Parent
b. Teacher
_________ c. Researcher d. Other
e. NR
_________ a. Parent
b. Teacher
_________ c. Researcher d. Other
_________ e. NR
5. Class Grade
(page____)
31
J. Design Characteristics
Recruitment Pool (page_____)
1. Referral 2. Criterion 3. Norm Reference Test
4. Existing Group 5. Cohort 6. Volunteer
7. Other ___________________ 99. NR
Research Design Type (pg____)
1. Group (Cluster, Place-based) Randomized Design
2. Individual Randomized Design
3. Quasi-Experiment: Interrupted Time Series Design
4. Quasi-Experiment: Regression Discontinuity
5. Quasi-Experiment: Nonequivalent Comparison Group Design
6. Single Subject Design
7. Other Designs: _____________________________________
32
K. Assignment Procedures (If applicable, otherwise skip to
section I)
Subject Assignment (page(s) _____)
1. Individual Random
2. Group Random
3. Individual Match Random
4. Quasi-random
33
2. Method of Random Allocation (page(s) _____)
Method of Random Allocation
Who Implemented the
Random Allocation?
1. Random Numbers (page____) a. Parent
b. Teacher
c. Researcher d. Other
e. NR
2. Coin Flip (page____)
a. Parent
b. Teacher
c. Researcher d. Other
e. NR
4.Other:_______(page____)
a. Parent
b. Teacher
c. Researcher d. Other
e. NR
34
Was concealment of allocation used as part of the
randomization procedures?
1=no 2= yes 3=NR
a. If yes, were the allocation procedures concealed from
any of the following people?
1. Researcher (pg _____)
2. Participant (pg _____)
3. Intervener
(pg _____)
4. Assessor
(pg _____)
35
I. Quasi-Experimental Assignment Procedures (If
applicable, otherwise skip to section J)
1. Was any of the following research design techniques used
to equate groups prior to the intervention?
•
•
•
•
a. Matching on pretest proxy (pg _____)
b. Matching on a pretest
(pg _____)
c. Matching on other variables (pg _____)
d. No Matching
(pg _____)
2. Was statistical equating used prior to the intervention?
1=no
2=yes 3.NR
Comments:________________________________________
36
J. Effect Size Characteristics
Comparisons: Group 1: _TX___vs. Group 2:_Ctl___
Outcomes
_Reading___ __________ ___________
Groups
__1__ _2_
__1__ __2__ __1_ __2__
Mean
_22__ __18_
_____ _____ ____ _____
SD
_3.55 _4.22_
N at Posttest _____ _____
_____ _____ ____ _____
_____ _____ ____ _____
F value
Chi-square
_____ _____ _____ _____ _____ ____
_____ _____ ___ _____ _____ _____ ____
p value
t value
__________
__________
_____
_____
_____
_____
_____ _____
_____ _____37
Method of Analysis (pg _____)
a. Intention to Treat:
b. Treated Participants Only
38
Excerpt from
Master Coding Guide
for Study Coding
------------------------------------Code the Fantuzzo Study
39
Data Extraction Practice
40