Transcript OPERATIONS RESEARCH What, Why, When, How

How to Do Operational
Research
George Schmid, M.D., M.Sc.
World Health Organization/Centers for
Disease Control and Prevention
[email protected]
•
•
•
•
Format
Informal
Practical—if you want to hear about modeling (a
legitimate, important part of OR), run out the door now
The session is for you, and us
You will forget 90% of what you hear today. So, we've
given you materials on four CD-ROMs:
– Designing HIV/AIDS Intervention Studies (and other materials)
(Pop Council);
– Training course materials in operations research (Pop Council)
– Framework for Operations and Implementation Research in Health
and Disease Control Programmes (and other materials) (Global
Fund and WHO)
– Other materials (note the FHI guide to qualitative research)
Doing Operations and
Implementation Research
Research Techniques
•
Systematic data collection
•
Qualitative techniques
•
Quantitative techniques
•
Surveys, experiments, focus groups….
The OR Process
•
•
•
•
•
Problem identification
Solution generation
Solution testing
Results dissemination
Results utilization
OR Requires Collaboration
• Managers:
responsible for decision parameters and
desired outcome
• Researchers:
responsible for recommending and
implementing research techniques
Taking My Problem Into
Operational Research
Q1.What Proportion of Individuals
Potentially Helped by Bed Nets are
Using Them?
1.
2.
3.
4.
2%
7%
14%
24%
Morel CM et al. BMJ 2005
Taking My Problem Into
Operational Research
First, You Must Recognize a Problem
• Wanting to know
• Wanting to act
• Having the ability to act
Q2. Who Typically Identifies Problems
for OR? (more than one answer
possible)
1.
2.
3.
4.
5.
The patient
The programme staff
The programme manager
The District manager
National staff
Zambia Example
• Problem: TB patients are not being
tested for HIV
• Observation: "TB Corners" where TB
patients are seen are in open hallways
• Intervention: Provide counselling with
privacy
Some issues simply need some common
sense, and initiative
Identifying Problems
• The role of anyone who works in the
programme!
• But, primarily, the role of the manager
• Identifying problems requires a team
approach
– Managers must be present in their programmes
– Managers must have good relationships with
their staff
You've Identified a Potential Problem
• Patients may not be taking their ART
• People may not be using condoms
• Patients may not be returning for
follow-up visits
• Your health care workers seem to
have high rates of absenteeism
• ….
The OR Process
1.
2.
3.
4.
5.
Problem identification
Solution generation
Solution testing
Results dissemination
Results utilization
Diagnostic step
Intervention step
There are Lots of Ways to Approach
OR
• Some are simple, others more formal
Simple? Complex?
• Attendance at the ART clinic is low
– Patients do not have transport
• Injection drug users are not using
clean needles
– Clean needles are not available
There are Lots of Ways to Approach
OR
• Some are simple, others more formal
• Flexibility needed and desired, to
meet the objectives of what you need
to do
• But, let's talk about more formal
approaches now….
The Menu of the WHO/GF OR Book
1.
2.
3.
4.
5.
6.
7.
Research team
Determine Issues
Develop proposal
Ethical clearance
Funding
Budget
Capacity building
8.
9.
10.
11.
12.
13.
14.
15.
16.
Monitor project
Pre-test
Quality control
Stakeholder
discussions
Dissemination plan
Disseminate results
Document changes
Monitor changes
Consider further
improvement
WHO (TDR)/GF (Jane Kengeya-Kayondo, George Shakarishvili, Serge Xueref)
and Bill Brieger and Amy Ellis of Johns Hopkins School of Public Health
Three Phases to OR
1. Planning
2. Implementation
3. Follow Through
Planning
1. Organize the Research Group
• What disciplines do you need?
– Epidemiology, behavioural science…?
– Statistics?
– Economics (It is difficult to retroactively do the economics
part of a study)?
•
•
•
•
Structure
Assign tasks, with deadlines
Meet regularly
Give credit!!!!!!
– Who had the research idea?
– Builds teamwork among departments/persons and avoids
hard feelings
– Pays off in the future
Planning
2. Determine the research questions
(and objectives)
• Be clear, be specific
• Research question—What do you want to know?
– Observation
• "Our health care workers seem to have high rates of
absenteeism"
– Research question
• "What is the rate of absenteeism among our nurses in the
Kiev city hospital, and the reasons for it?"
• Objective
– Not: To study adherence
– But: To determine rates of adherence among attendees at the
XX hospital and reasons for adherence and nonadherence
Planning
3. Develop a research
proposal/protocol
•
A document that details:
1. What the research is about and why it
is important
2. How the study will be conducted
3. How the results will be used
Planning
3. Develop a research
proposal/protocol—"standard parts"
•
•
Research objectives and questions
Background
–
–
•
•
•
•
•
Literature review
Local context
Research team
Methods (study design, study population,
sampling details, how data will be collected, data
management, quality assurance)
Data collection instruments
Plans for use of data
Budget
Planning
3. Develop a research protocol
• Does every "formal" OR project need
a research protocol?
– Yes
– Makes you clearly state what you will do
(and helps to improve your thinking)
– Gives you the plan that you will follow, so
everyone knows what will be done, and
should be done
Q3. Does Every OR Proposal need
to go through Ethical Review?
1. Yes
2. No
Planning
4. Obtain ethical clearance
Q4. What is the most common thing
wrong with consent forms?
1. Do not contain the contact
information for the local ethics board
2. Cannot be understood by potential
participants
3. Do not mention the word "research"
4. All risks are not revealed
5. Give the patient too much information
Planning
5. Identify funding
• A variety of sources
• Requests for small amounts of money are
almost always preferred
• Follow closely the guidelines of funding
agencies
• Ensure projects are well-written
• Having money from one agency for a
project enhances the ability to get further
funds from any agency, i.e., "a proven track
record"
Planning
6. Establish a budget
• Should be comprehensive, even if not
a large budget—ensure you have
costed everything, including
dissemination costs
• A good budget is a sign of careful
planning
• ? Include a financial officer
Planning
7. Technical capability/capacity
building
• Ensure that staff know their roles and
responsibilities
• Ensure that staff have the training and
capability to perform their jobs
– Short term training (courses, TA)
– Long term training (building capacity)
– Help staff gain skills to advance
Implementation
8. Quality Assurance of the project
• There must be a protocol!
• There almost always is a work plan, that includes:
– All activities to be implemented, before and during the
study
– Starting date, completion date, intermediate deadlines
– Persons who are responsible for each activity
– Milestones for each activity
• An advisory group that meets regularly may assist
in maintaining quality and the work plan
• Monitoring should be part of regular team meetings
• A quality assurance plan should be part of the
protocol
If you do not perform the study exactly as you
said you would, you cannot trust the results
Implementation
9. Pre-test all research materials
• Valid? (Do data collection instruments
capture the desired information reliably?)
• Reliable? (Do data collection instruments
capture the desired information
consistently?)
Implementation
9. Pre-test all research materials
• Questionnaires
– Is the wording clear? Do people understand what
you want? Do the questions ask what they are
supposed to?
– Test in 5-20 persons
• Focus groups
– Are the guides clear?
– Hold one or two focus groups.
• Consent forms
– Do people understand them? Language level?
(have your child read it) (sometimes, to check
understanding during study, develop a series of 5
or so questions—patient must answer 4 or 5 to
enter study)
Dissemination
10. Establish and maintain data
management and QA
• Forms should be easy to use
• Observe data collectors regularly,
supervisors review all data collection
instruments daily for completeness and
accuracy, have regular meetings of the data
collectors (daily, likely)
• Data entry into computers preferably as
study is ongoing so errors by data
collectors can be quickly caught and fixed
• Double-entry of data?
Dissemination
11. Explore with stakeholders
interpretation of results
• Review preliminary results with the
team early, e.g., within a week after
data entry is complete, to identify
reasons for unusual findings
• Subsequently, review data with the
team and stakeholders, e.g.,
community leaders or project advisory
group, to identify key findings and
interpretations
Follow-through
12 & 13. Develop dissemination plan
and disseminate the results
• Audience #1 ("internal"). Those who can benefit
from, and act on, the information locally
– Local meetings
– Meetings with decision makers
• Audience #2 ("external"). Those who can adapt
the information for their own context, or use it for
further research
– National or international meetings
– Journals
• Create the right documents/strategy/medium for
the right audience
Q5. Who Need Not Be Part of the
Pre-Test Evaluation of Materials?
1. Potential participants
2. Staff who use the materials, e.g.,
interviewers, focus group leaders
3. Data management staff
4. The members of the research team
developing the materials
5. All must be
Dissemination
14. Document changes as a result of
the research
• Were the results implemented? If not,
why not?
Dissemination
15. Monitor changes in the revised
program
• If the findings were valid, and
appropriately implemented, then
program changes should occur when
the results are implemented
• More of a monitoring (and evaluation)
function than OR, but gets into the
area of Translation Research….
Dissemination
16. Consider ways of further
improving the program
Q6. Which Format of Information
Sharing Do Decision-makers Like
Least (Industrialized Country)
1.
2.
3.
4.
5.
Internet/e-mail
Meetings/conferences
Colleagues
Journal articles
Short summaries
Dobbins M et al. Implementation Sci 2007
Q7. Who Typically Acts on Identified
Problems?
1.
2.
3.
4.
5.
6.
The patient
The programme staff
The programme manager
The District manager
The National staff
Too few people act
Q1.What Proportion of Individuals
Potentially Helped by Bed Nets are
Using Them?
1.
2.
3.
4.
2%
7%
14%
24%
Morel CM et al. BMJ 2005
Q2. Who Typically Identifies Problems
for OR? (more than one answer
possible)
1.
2.
3.
4.
5.
The patient
The programme staff
The programme manager
The District manager
National staff
Q3. Does Every OR Proposal need
to go through Ethical Review?
1. Yes
2. No
Q4. What is the most common thing
wrong with consent forms?
1. Do not contain the contact
information for the local ethics board
2. Cannot be understood by potential
participants
3. Do not mention the word "research"
4. All risks are not revealed
5. Give the patient too much information
Q5. Who Need Not Be Part of the
Pre-Test Evaluation of Materials?
1. Potential participants
2. Staff who use the materials, e.g.,
interviewers, focus group leaders
3. Data management staff
4. The members of the research team
developing the materials
5. All must be
Q6. Which Format of Information
Sharing Do Decision-makers Like
Least (Industrialized Country)
1.
2.
3.
4.
5.
Internet/e-mail
Meetings/conferences
Colleagues
Journal articles
Short summaries
Dobbins M et al. Implementation Sci 2007
Q7. Who Typically Acts on Identified
Problems?
1.
2.
3.
4.
5.
6.
The patient
The programme staff
The programme manager
The District manager
The National staff
Too few people act