Transcript Scientific and Program Advisory Committee

Evidence Check:
Title page
Commissioning better research syntheses
Sally Redman,
Danielle Campbell,
Gai Moore
Sax Institute Mission
To improve health and increase the quality and
performance of health services and programs
by:
 building excellent policy and practice
focused research and
 increasing the impact of this research on
health policy, programs and services
Is research used to inform policy
in NSW?
• Policy Maker Survey - 38 senior policy
makers from the NSW Department of Health
and Area Health Services
• Researcher Survey - 41 senior researchers
from 29 Universities and research centres
across NSW
Proportion perceiving a HIGH need to improve
the use of research in policy
90
80
70
60
%
50
40
30
20
10
0
Policy makers
Researchers
Source: Policy Maker Survey and Researcher Survey
Perceived facilitators of evidence in policy
NSW policy makers
 Better access to researchers
 Better access to research/reviews
 Policy-relevant research
Innvaer et al, 2002
(Review of 24 interview studies of health policy makers)
 Personal contact with researchers
 Timely relevance
 Inclusion of summaries & recommendations
Increasing the impact of research:
Some goals of the Sax Institute
 Increase research in areas where NSW Health
has knowledge priorities
 Give policy makers more timely and
effective access to research findings
 Bring policy makers and researchers closer
together to more easily understand each others
expertise and priorities
 Identify research findings that are of potential
importance to policy and assist researchers to
package and communicate their findings in
ways that can influence the policy agenda.
Ability to find research to inform
policy in last 12 months
 68% found brief research summaries difficult
or impossible to find
 84% found more systematic reviews difficult
or impossible to find
 Difficulty in finding research summaries
nominated as one of greatest barriers to
using evidence in policy
Policy Maker Survey
What goes wrong?
Policy makers…
Reviews are:
 Too slow
 Too focused on researcher’s interests rather than
policy question
 Too time consuming to commission
 Too long with key messages buried
 Focused on need for more research rather than
best summary of current position
Policy makers’ feedback about
research reviews
“I don’t think we use reviews much at all –
people don’t even think of it”
“A couple of times either I didn’t specify clearly
enough what I wanted or the specs changed
and what they produced...was awkward for the
Department”
“Also there’s the timing issue, you don’t want to
have to put a lot of effort into a tendering
process that won’t get you a product when you
really need it”
What goes wrong?
Researchers..
Commissioned reviews:
 Are often buried and cannot be published
 Are frustrating because policy makers do not
know what they want and change their mind
 Compete with research through urgent tender
requests
Prop’n of researchers perceiving that little value is
placed on efforts to get research into policy
90
80
70
60
50
%
40
30
20
10
0
When funding bodies assess track
record
When universities consider
employment/promotion
Source: Researcher Survey
Evidence Check
To enable policy makers to more readily
access high quality reviews of research
suited to their needs
Will also:
• Encourage researchers to undertake reviews
for government
• Result in greater knowledge of research
expertise in NSW among policy makers
Evidence Check
Identification
of expertise
Review
Commissioning
Tool
RADAR
Partnership and
production
guidelines
Knowledge
Brokers
Preferred Provider
Rules
Diagnosis
Evidence Check: Diagnosis
Evidence Check Commissioning Tool:
Series of structured questions to assist policy
makers to analyse and make explicit their
needs
Knowledge brokers:
Individuals with experience in policy AND
research - trained and supported to work
between researchers and policy makers
Commissioning Tool
1.
Background and purpose of the policy
• How will the review be used, at what stage in the
policy process, and by whom?
• What is the political context?
• What risks are associated with the policy issue?
2.
Questions to guide the review
• What are the specific questions that need to be
answered (define: intervention, target population
and outcome)?
Commissioning Tool
3.
Timeframe and funding
• Is an answer required immediately, in the short
term, or in the long term?
• What funds are available?
4.
Depth of analysis of evidence
• Systematic review or a summary of key findings?
• Are strength and recency of evidence important?
5.
Format of final product
• How long should the report be?
• What level of language is appropriate?
Case Study: Advanced Care Planning
Evidence need initially
defined by policy makers
Review questions agreed
after meeting with broker
Evidence to support a policy approach to comprehensive advanced
care planning (ACP) for end-of-life choices
Is there evidence of
interventions that
support a communitywide approach to
ACP for end-of-life
care?
What types of
interventions that
support a communitywide approach to
ACP have been
implemented and
proven effective?
Is there evidence of
interventions for other
sensitive issues that
could provide lessons
to support a
community-wide
approach to ACP?
Findings from scoping
exercise by broker
A recent meta-analysis provides a basis for identifying key features of
interventions, and other reviews of ACP and palliative care may
support these findings. Reviews of approaches to other issues (eg
domestic violence) may also describe relevant strategies
Recommendation
Commission researcher to: identify relevant high-quality metaanalyses and reviews; identify key features of effective interventions
in community settings; and develop recommendations for a policy and
program approach in NSW
Case Study: Type 2 Diabetes Prevention
Evidence need initially
defined by policy makers
Review questions agreed
after meeting with broker
Evidence to support the development of strategies to reduce the
incidence of type 2 diabetes in CALD communities
Are there CALD
groups in NSW with
increased incidence
of type 2 diabetes
or increased
prevalence of risk
factors?
What evidence
exists for interventions
What are the factors
that are likely to be
that may contribute to
effective in reducing
higher risk levels in
type 2 diabetes risk
specific CALD groups?
factors in specific
CALD groups?
Findings from scoping
exercise by broker
Risk factor prevalence and disease incidence data is available from
government bodies and diabetes associations. Data on factors that
influence risk and incidence in specific CALD groups can be derived
from these sources and the peer reviewed literature
Recommendation
Commission researcher to undertake a review and critical analysis of
epidemiological data on risk factor and disease prevalence, and
evidence on the most effective prevention approaches for specific
CALD groups
Evidence Check
Identification
of expertise
Review
Commissioning
Tool
RADAR
Partnership and
production
guidelines
Knowledge
Brokers
Preferred Provider
Rules
Diagnosis
Evidence Check: Identifying expertise
RADAR: Database
38 research centres and Universities that are
members of the Sax Institute used to generate a
database of expertise
Selecting reviewers: Preferred provider
Competitive tender avoided by decision rules for
selecting individuals with best expertise based on
discipline base (eg epidemiology, health economics), content
knowledge (i.e. diseases, settings, interventions, and populations),
and review experience
Prop’n of researchers interested in commissioned
research reviews
90
80
70
60
50
%
40
30
20
10
0
Funded by government
High interest in
High interest in joining
to conduct review in
conducting
register of reviewers
last 2 years
commissioned reviews
Source: Researcher survey
Evidence Check
Identification
of expertise
Review
Commissioning
Tool
RADAR
Partnership and
production
guidelines
Knowledge
Brokers
Preferred Provider
Rules
Diagnosis
Evidence check: the review
 Range of review types depending on policy issue
(eg systematic review, review of evaluations, ‘headline’ review)
 Guidelines for reviewers
(eg CHSRF 1:3:25 format)
 Translation process by knowledge broker
Feedback from the pilot study
 Provided a valuable framework for critical
thinking and discussion among policy makers
 Helped policy makers move from a general
issue to specific questions
 Helped policy makers anticipate the potential
challenges and risks that could result in a
product that does not meet policy needs
Outcomes so far
 7 reviews using Evidence Check
commissioned so far by NSW Health in broad
range of areas: e.g. health workforce, diabetes
prevention, HIV services, obesity, population health priorities
 Rapid increase in demand over the past few
months
 2 groups that used Evidence Check during
pilot/early stages have returned with new
project requests
Case Study: Health Workforce
Evidence need initially
defined by policy makers
Evidence to support improved workforce planning and education
for future health needs especially overseas trained doctors
Review questions agreed
after meeting with broker
What are the
characteristics of
newly appointed nonspecialist doctors
trained overseas
or in Australia?
Findings from scoping
exercise by broker
Little evidence exists to clarify: the extent to which NSW doctors are
supported and trained; community perceptions of types of doctors;
barriers and facilitators of effective workforce participation; and flow-on
effects of new training/recruitment methods.
Recommendation
What skills and
knowledge gaps can
be identified in newly
appointed doctors
trained overseas
and in Australia ?
How are these
doctors perceived by
the community and
the health system?
Commission researcher to provide an objective, descriptive and
analytical profile of doctors practicing in NSW who have been trained
in Australian medical schools or overseas.