Developing Evidence-Based Products Using the Systematic Review Process

Session 3/Unit 9: Inclusion and Exclusion Criteria

Screening and Coding Studies Julia Lavenberg October 30, 2007 NCDDR training course for NIDRR grantees

Outline

    Study selection vs. data extraction Requirements for each Exploration of screening and coding via a worked example Application of inclusion and exclusion criteria and codes to planned reviews 2

Study Selection vs. Data Extraction

  The process by which one chooses studies for inclusion in a systematic review.

Accomplished by instituting specific and detailed inclusion and exclusion criteria.

  The process by which one locates and transcribes information from a primary study.

Accomplished by applying specific and detailed criteria to the information in a primary study.

3

Eligibility criteria

   Provide readers with an idea of the research domain of interest Aid the systematic reviewer in applying consistent and objective standards throughout the selection process Clearly circumscribe the review 4

Inclusion and Exclusion Criteria: Categories       Publication types Study design Population Intervention Outcomes Measures 5

Publications

  Identify the types of reports to be included Determine any geographic or linguistic limitations “Studies eligible for this review may be published or unpublished reports (e.g., dissertations theses, government reports, school district reports, etc.) of school-based interventions conducted in any country and reported in any language.” (Lavenberg, 2007) 6

Study design

   Specify the research designs to be included Also identify the research designs to be excluded Address the rationale for inclusion and exclusion   Consider theoretical framework Consider available evidence 7

Population

   Indicate the desired target population Stipulate any required characteristics Additionally, distinguish characteristics that would make the target population ineligible “Studies of interventions that target children and youth who are enrolled in kindergarten through grade 12 (or the international equivalents) at public, private, parochial, or alternative schools and are between the ages of 4 and 20 years will be included in this review. Persons identified as attending ‘preschool’ or ‘college’ will not be included, even if they are within the acceptable age range.” (Lavenberg, in process) 8

Intervention

   Detail the required characteristics of the intervention Provide definitions List examples of what would be included and what would be excluded 9

Outcomes

 Indicate outcome variables of interest “At least one quantitative measure of aggressive behavior must be reported in each study….[citations to literature here]… Therefore, both physical and verbal aggressive behaviors will be considered acceptable outcomes.” (Lavenberg, in process) 10

Measures

 Address acceptable measures “Standardized measures of aggressive behavior (e.g., Child Behavior Checklist – Teacher Report Form….) and unstandardized measures with adequate face validity (e.g., local administrative records…) will both be considered acceptable forms of reporting aggressive behavior. Measures reported in the Buros Institute of Mental Measures Yearbook…will be considered standardized measures; all others will be considered unstandardized measures.” (Lavenberg, in process) 11

Example: Application to a review with > 900 records retrieved

(Lavenberg, 2007)  Two phase screening process developed  Phase I – title and abstract  Goal: differentiate potential studies from clearly inappropriate studies   Evaluate each study for topical relevance, setting, participants, and general study design Phase II – method section   Goal: create the pool of eligible studies Evaluate each study for intervention, design details, and outcome 12

Selection process for including reports Potentially relevant reports identified (n>900) Reports retrieved for further review (n=155) Reports progressing to next level (n=109) Reports used to code studies for inclusion (n=50) Reports excluded by title and abstract review (n=750) Reports excluded at Phase I screening (n=46) Reports excluded at Phase II screening (n=59) 13

Data Extraction (i.e., “coding”)

   Initiated after identifying appropriate pool of studies for the systematic review Accomplished by means of a coding form and codebook Goal: ensure reliable and orderly extraction of information from each study report 14

Forms…

    May be either paper-based or computer based Are best completed by trained coders Should have items organized such that the order reflects the manner in which information is presented in the study report Contain the same categories of information as inclusion and exclusion criteria 15

Coding categories

 That is, the data extracted will address:       Publication type Design Participants Interventions Outcomes Measures 16

BUT:

. . . in coding, much more detail is required. The data that is extracted from each study report is the basis of all subsequent analyses.

17

Overarching principle:

 Conserve as much of the original information as possible.

 Fewer coder judgments = fewer errors 18

Publication characteristics

    Document publication year Distinguish between types Create mutually exclusive categories Must assign numerical values to these categorical variables 19

Study design and methodological characteristics

    Consider identifying the recruitment pool to provide additional context Describe the unit of assignment (e.g., individual, family, school, clinic, workplace) Address the mechanism of allocation variables, etc.) (e.g., random assignment, matching on pretest measures of outcome List unit of analysis  Note whether unit of assignment matches unit of analysis 20

Participant characteristics

      Age  May need to capture data in multiple ways (e.g., school grade, chronological age, mental age, age range) Gender Race/ethnicity Socioeconomic status Specific characteristics relevant to review topic (e.g., official diagnoses, developmental categories, baseline level of behavior) Co-morbidity 21

Intervention characteristics

  Describe important aspects of the intervention  Characteristics or features     Total length of intervention, number of sessions, length of each session Method of delivery Personnel implementing intervention Fidelity of implementation Identify comparison group conditions 22

Outcome and measure characteristics

     Identify outcome variables of interest  Allow for and document variations Identify outcome informant or source of information  Determine if informant was blind to assignment Identify personnel administering measure Specify outcome measures used Indicate time of outcome assessment 23

Effect size characteristics

 Extract information required to calculate an effect size      Sample size (intervention and control groups)  Pretest and posttest  Attrition Means, standard deviations d index F value Others: Chi-square, p value, t value, etc.

24

Worked example #1

Approaches to Parent Involvement for Improving the Academic Performance of Elementary School Age Children Nye, Turner, & Schwartz (2006) http://www.campbellcollaboration.org/ doc-pdf/Nye_PI_Review.pdf

25

Definition of parent involvement

 “…the parent has a direct interaction with the child in either the delivery or monitoring of the program or intervention.” (page 11) 26

Inclusion criteria

  Design: limited to randomized controlled trials (RCTs) Rationale: “…in order to provide the least biased estimate of the effect of parent involvement on student achievement and to control for variation in threats to internal validity.” (pg. 12) 27

Inclusion criteria,

continued  Participants: described in detail in the coding categories section of the review (pg. 13)      Age Grade Gender SES Ethnicity 28

Inclusion criteria,

continued  Interventions: “…studies reported the following characteristics of the intervention program: 1) Parent involvement with their child in academic support activities outside of school (e.g., reading or completing supplemental math problems with the child), and 2) Parent involvement [as defined earlier] for a minimum of 20 days” (pg. 11) 29

Inclusion criteria,

continued  Outcomes: “Included studies reported the following outcomes on children’s academic performance in: reading, mathematics, spelling, writing, language arts, or science. (pgs. 11-12) 30

Inclusion criteria,

continued  Measures: not specified as an inclusion criteria in Nye, et al.

 Again, specification of this characteristic will depend on the topic and decisions made by the review team  e.g., depression as measured by Beck inventory 31

Nye, Turner, & Schwartz Coding categories and coding sheet

(unique elements)   Report characteristics Subject characteristics  Treatment and comparison groups separately addressed    Sample source – types of schools Classroom assignments – high achieving, underachieving, average, academically at-risk Developmental categories – emotionally disabled, physically disabled, hearing impaired, speech/language impaired, etc. http://www.ncddr.org/training/ 1NyeTurnerSchwartz_PI_CodingForm_2006.doc

32

Coding categories,

continued    Intervention characteristics    Coder description required, in addition to numerical data (e.g., total length of intervention program, length of time per day of PI activity, etc.) Fidelity of implementation Outcome measures – norm referenced vs criterion referenced vs rating scale vs survey Design characteristics   Recruitment pool Specific method of randomization and identification of who implemented the process Effect size characteristics 33

Review topics

   Weight loss intervention to reduce physical disability in older adults Improving the ambulatory transition of pre- and post-adolescents with cerebral palsy Improving cultural competency to increase the employment of persons with disabilities 34

Review topics, continued

  Use of assistive technology and employment supports in the employment of working age people with disabilities Health promotion programs that improve primary access for adults with disability 35

Contact information

Julia Lavenberg Email: [email protected]

36

SUPPLEMENTAL SLIDES 37

Worked example #2

Effects of School-Based Cognitive Behavioral Anger Interventions: A Meta-Analysis (Lavenberg, 2007) 38

Primary Research Question:

Are school-based cognitive-behavioral anger interventions effective in reducing child and adolescent aggressive behavior in the school setting?

39

Secondary Research Questions…

 Involved looking at whether the following were related to the effect size:      Methodological issues (e.g., random assignment) Setting  Geographic location (urban, suburban, rural) Delivery personnel  Teachers vs. counselors Stand-alone vs. embedded anger interventions Participant characteristics (e.g., gender, age, baseline level of aggressive behavior) 40

Define and operationalize critical elements:

     Aggression Anger Cognitive-behavioral   Theory Intervention Anger management intervention School-based intervention 41

Screening Forms and Manual

 Copy sent to Joann Starks for posting http://www.ncddr.org/training/ 3Lavenberg_2007_Screening_Coding_FormsManuals.doc

42

Coding Form and Manual

 Copy sent to Joann Starks for posting http://www.ncddr.org/training/ 3Lavenberg_2007_Screening_Coding_FormsManuals.doc

43