Transcript Evidence-Based Software Engineering

Evidence-Based
Software Engineering
and Systematic
Reviews
Barbara Kitchenham
1
Agenda
The evidence-based paradigm
 Evidence-Based Software
Engineering (EBSE)
 Systematic Reviews

2
The Evidence-Based Paradigm
Evidence-based medicine has changed
research practices


Medical researchers found
• Failure to organise existing medical research cost
lives
• Clinical judgement of experts worse than systematic
reviews
Evidence-based paradigm adopted by many
other disciplines providing service to public




3
Social policy
Education
Psychiatry
Goal of EBSE


EBM: Integration of best research evidence with
clinical expertise and patient values
EBSE: Adapted from Evidence-Based Medicine


Anticipated benefits





4
To provide the means by which current best evidence from
research can be integrated with practical experience and
human values in the decision making process regarding the
development and maintenance of software
Common goals for research groups
Help for practitioners adopting new technologies
Means to improve dependability
Increase acceptability of software-intensive systems
Input to certification process
Practicing EBSE
1.
2.
3.
4.
5.
5
Convert information need into
answerable question
Track down best evidence
Critically appraise evidence
Integrate critical appraisal with SE
expertise and stakeholder
requirements
Evaluate and improve above steps
Systematic Reviews - 1/2

A systematic review is


Systematic reviews aim to synthesise
existing research



6
An overview of research studies that uses
explicit and reproducible methods
Fairly (without bias)
Rigorously (according to a defined
procedure)
Openly (ensuring that the review procedure
is visible to other researchers)
Systematic Reviews – 2/2

Support Evidence-based paradigm

Start from a well-defined question
• Step 1

Define a repeatable strategy for searching
the literature
• Step 2

Critically assess relevant literature
• Step 3

Synthesise literature
• Step 4 (but only partially)
7
Advantages

Provide information about effects of a
phenomenon across wide range of settings


Essential for SE where we have sampling
problems
Consistent results provide evidence that
phenomena are
• Robust
• Transferable

Inconsistent results
• Allow sources of variation to be studied

8
Meta-analysis possible for quantitative
studies
Anticipated Benefits

Create a firm foundation for future research
• Position your own research in the context of existing
research



Close areas where no further research is
necessary
Uncover areas where research is necessary
Help the development of new theories



9
Identify common underlying trends
Identify explanations for conflicting results
Should be a standard research methodology
Disadvantages
Require more effort than informal
reviews
 Difficult for lone researchers


Standards require two researchers
• Minimising individual bias

10
Incompatible with requirements for
short papers
Value of Systematic Reviews

Can contradict “common knowledge”

Jørgensen and Moløkken reviewed surveys
of project overruns
• Standish CHAOS report is out of step with other
research
• May have used inappropriate methodology

Jørgensen reviewed evidence about expert
opinion estimates
• No consistent support for view that models are
better than human estimators
11
Systematic Review Process
Develop Review Protocol
Plan Review
Validate Review Protocol
Identify Relevant Research
Select Primary Studies
Conduct Review
Assess Study Quality
Extract Required Data
Synthesise Data
Write Review Report
Document Review
Validate Report
12
Developing the Protocol

Review protocol


Specifies methods to be used for a
systematic review
Predefined protocol
• Reduces researcher bias by reducing opportunity
for
• Selection of papers driven by researcher
expectations
• Changing the research question to fit the results of
the searches

13
Good practice for any empirical study
Protocol Contents -1/2

Background


Rationale for survey
Research question
Critical to define this before starting
the research
 Strategy used to search for primary
sources

• Individual studies of the phenomenon of
interest
14
Protocol Contents – 2/2

Strategy to find primary studies



Search terms, resources, databases, journals,
conferences
Procedures for storing references
How publication bias will be handled
• Grey literature
• Direct approach to active researchers

How completeness will be determined
• Useful to have the baseline paper to set start date

Selection Strategy

Inclusion/exclusion criteria
• Handling multiple papers on one experiment
15
Protocol Contents- 2/3

Quality assessment criteria


Criteria used to evaluate quality of primary sources
Data extraction



What data will be extracted from each primary source
How to handle missing information
How data reliability will be addressed
• Usually multiple reviewers


Procedures for data synthesis



16
Where data will be stored
Formats for summarising data
Measures and analysis if meta-analysis is proposed
Should tested during protocol construction
Research Question – 1/2

Question types for EBSE


Assessing the effect of an SE technology
Assessing the frequency or rate of a project
development factor
• E.g. Rate of project failures



Possible to have more general questions for
other purposes

17
Identifying cost and risk factors
Identifying impact of technology on reliability,
performance, cost
Review of research in software engineering
(Glass, et al., 2002)
Research Question – 2/2

Question structure

Population
• People, projects types, applications types affected
by the intervention

Intervention
• Software method, tool, procedure

Outcomes
• Impact of technology in terms relevant to
practitioners
• Cost, quality, time to market

Experimental designs
• Any constraints on type of primary studies to be
included
18
Next steps are easy!?

Conduct the review

Enact the protocol
• Expect further iterations of
• Search strategy
• Selection criteria
• Data extraction


Document the Review

19
Record any deviations from protocol
Using procedures defined in protocol
Conclusions

Evidence-based approach
Revolutionised medicine
 May be relevant to SE


Systematic reviews
Support the evidence-based approach
 Valuable as a research tool

• Even if we don’t accept EBSE
20
References
Australian National Health and Medical Research Council. How to review the
evidence: systematic identification and review of the scientific literature, 2000.
IBSN 186-4960329 .
Australian National Health and Medical Research Council. How to use the
evidence: assessment and application of scientific evidence. February 2000,
ISBN 0 642 43295 2.
Cochrane Collaboration. Cochrane Reviewers’ Handbook. Version 4.2.1.
December 2003.
Glass, R.L., Vessey, I., Ramesh, V. Research in software engineering: an
analysis of the literature. IST 44, 2002, pp491-506
Magne Jørgensen and Kjetil Moløkken. How large are Software Cost Overruns?
Critical Comments on the Standish Group’s CHAOS Reports,
http://www.simula.no/publication_one.php?publication_id=711, 2004.
Magne Jørgensen. A Review of Studies on Expert Estimation of Software
Development Effort. Journal Systems and Software, Vol 70, Issues 1-2, 2004,
pp 37-60.
21
References
Khan, Khalid, S., ter Riet, Gerben., Glanville, Julia., Sowden,
Amanda, J. and Kleijnen, Jo. (eds) Undertaking Systematic Review
of Research on Effectiveness. CRD’s Guidance for those Carrying
Out or Commissioning Reviews. CRD Report Number 4 (2nd
Edition), NHS Centre for Reviews and Dissemination, University of
York, IBSN 1 900640 20 1, March 2001.
Kitchenham, Barbara. Procedures for Performing Systematic
Reviews, Joint Technical Rreport, Keele University TR/SE-0401 and
NICTA 0400011T.1, July 2004.
Pai, Madhukar, McCullovch, Michael, Gorman, Jennifer D., Pai,
Nitika, Enanoria, Wayne, Kennedy, Gail, Tharyan, Prathap, Colford,
John M. Jnr. Systematic reviews and meta-analysis: An illustrated,
step-by-step guide. The National medical Journal of India, 17(2)
2004, pp 86-95.
Sackett, D.L., Straus, S.E., Richardson, W.S., Rosenberg, W., and
Haynes, R.B. Evidence-Based Medicine: How to Practice and Teach
EBM, Second Edition, Churchill Livingstone: Edinburgh, 2000.
22