Evaluating methodological quality in the criminal justice system literature

Dr Amanda Perry

Centre for Criminal Justice Economics and Psychology, University of York.

Overview of the session:

•Background to RCT in criminal justice •Quality assessment •The Maryland Scientific Methods Scale (SMS) •Problems associated with the SMS Where next…….

Background

Few RCT conducted in the UK Systematic review revealed 125 conducted between 1957-2005 (Farrington & Welsh, 2005) Concurrent findings from UK systematic review between 1990-2002 (Perry, McDougall & Farrington, 2005)

Resistance to RCT…

•Historical resistance •Ethical and moral reasons •Practical difficulties

BUT………... New era

Campbell Collaboration (2000) Department of Health/Home Office (2000)

New age of RCT…

Feasibility studies

(Farrington & Joliffe, 2002) (Farrington et al., 2002) Matrix (2006) – Criteria for assessing feasibility

Current RCTs

•Evaluation of Cognitive behavioural skills programmes ( McDougall, Bowles, Perry & Clarbour, ongoing).

•Evaluation of Restorative Justice Programmes (Strang & Sherman, 2006)

Quality assessment…

•Reviews of quality assessment tools: The medical field •Moher et al. (1995) identified 25 scales devised up to 1993.

•Juni et al (1999) compared 25 scales for purpose of inter-rater reliability (r=.72).

Quality assessment…

Reviews of quality assessment tools: The

Social Sciences

•Gibbs (1989) – social work •STROBE: STrengthening the Reporting of OBservational studies in Epidemiology http://www.strobe-statement.org/index.html

•Maryland Scientific Methods Scale – Criminal

Justice

(Sherman et al., 2002)

Quality assessment…

•The Maryland Scientific Methods Scale (SMS) (Sherman et al., 2002, based on work by Cook & Campbell, 1979)

Purpose of the SMS:

• Simple (measuring internal validity) • • To provide policy makers with information about the evidence Aim to classify all programmes into 1 of 4 categories

The SMS quality assessment..

Five point scale: Rating 1-5 Rating 1: Correlational study Rating 2: Pre and post test study Rating 3: Observational cohort with comparable group Rating 4: Quasi-experimental/controlled trial Rating 5: Randomised controlled trial

The SMS quality assessment..

Statistical conclusion validity

•Was the statistical analysis appropriate?

•Did the study have low statistical power to detect effects because of small samples?

•Was there a low response rate or differential attrition?

Construct validity

•What was the reliability and validity of measurement of the outcome?

Evidence and use of the SMS:

What Works : Evidence from 2 or more studies reporting positive results scoring 3 or above on the SMS showing statistical significance and desirable effects and the preponderance of all available evidence showing effectiveness.

What Doesn’t Work : Evidence from 2 or more studies reporting negative results scoring 3 or above on the SMS…

Evidence and use of the SMS:

What ‘s Promising : Reporting evidence from 1 study reporting positive results scoring 3 or above on the SMS….

What Unknown : Evidence from 1 study with a negative or inconclusive result scoring 3 or above on the SMS…..

Use of the SMS:

‘What works – an example’ CCTV in car parks (evidence from Welsh and Farrington, 2002) Street lighting (evidence from Painter & Farrington, 1997; 1999b;2001b; Farrington & Welsh, 2002) Burglary reduction schemes (evidence from Ekblom, 1996a; 1996b)

Limitations of the SMS…

•Not fully assessing all threats to the validity of a study •Categorising study designs •Does not take into consideration at What cost?

Limitations of the SMS…

•Is designed to apply equally to all experimental units •Does not embrace all study designs •Method of drawing conclusions on what works based on statistical significance rather than effect size

Improving the SMS….

•Farrington (2003) •Based on five key criteria: •Internal validity •Descriptive validity •Statistical conclusion/validity •Construct validity •External validity

Improving the SMS …

Information for policy makers 3 five point scales

•Design

(internal validity)

•Execution

(construct validity/statistical conclusion validity /sampling elements of external validity)

•Reporting of the trial

Beyond the SMS…..

•Assessing the cost of an intervention •Adequate and standardised follow-up periods for outcome measures (e.g., reconviction rates) •Encouraging journal editors to use a new scale/standard similar to CONSORT statement

Beyond the SMS…..

•Development of a specific quality assessment tool evaluating RCT in the criminal justice •To incorporate all elements of validity •To include an assessment of cost/cost effectiveness/cost-benefits of an intervention •To include guidance on adequate and standardised follow-up periods for outcome measures (e.g., reconviction rates)

SUMMARY

•RCT rarely used in UK criminal justice system •Use of the SMS as quality measure •Room for improvement •Development of a new scale….

CONTACT:

Amanda Perry Centre for Criminal Justice Economics and Psychology University of York Email: [email protected]