Transcript Writing a proposal - research design and methods

UNC ECHO, HBCU Health Promotion Alliance,
NC A&T School of Agriculture Workshop on
Grantwriting for Health Disparities Research
Victor J. Schoenbach, Ph.D.
Director, Minority Health Project
Department of Epidemiology
UNC-Chapel Hill School of Public Health
www.unc.edu/~vschoenb/
This presentation can be downloaded from:
www.minority.unc.edu/present/2004/
12/4/2004
1
An honest politician
“That lowdown scoundrel deserves to be
kicked to death by a jackass, and I'm just the
one to do it” – Congressional candidate in
Texas.
Portions drawn from
“A Grant Writing Primer” by Hinda Zlotnik, Ph.D.
MARC/MBRS Program, NIGMS
brin.hpcf.upr.edu/symposium/How%20to%20write%20successful%20proposals.ppt
“Writing a grant proposal”, by Malcolm Gordon, Ph.D.,
National Institute of Mental Health, 1999
www.charityadvantage.com/iaswr/images/grantproposals.pdf
Both downloaded 11/26/2004
3
Proposal components
• Abstract
• Budget, Biographical Sketches, Resources
• Research Plan
– A. Specific Aims (and hypotheses)
– B. Background and Significance
– C. Preliminary Studies / Progress Report (and
background of investigators)
– D. Research Design and Methods
– E. Human Subjects Research
– G. Literature cited
11/29/2004
PHS 398 Instructions
4
Follow instructions
“Read and follow the instructions
carefully to avoid delays,
misunderstandings and possible
return of the application. Adherence
to font and margin requirements is
necessary for several reasons.…”
“Use English and avoid jargon.”
11/29/2004
PHS 398 Instructions
5
Don’t believe everything
“The PHS estimates that it will
take approximately 40 hours to
complete this application for a
regular research project grant.”
11/29/2004
PHS 398 Instructions
6
Proposal components
• Abstract
• Budget, Biographical Sketches, Resources
• Research Plan
– A. Specific Aims (and hypotheses) [1 page]
– B. Background and Significance
– C. Preliminary Studies / Progress Report (and
background of investigators)
– D. Research Design and Methods
– E. Human Subjects Research
– G. Literature cited
11/29/2004
PHS 398 Instructions
7
a. Specific Aims
(outline labeling is inconsistent)
The framework for the entire
proposal, including the research
plan.
11/29/2004
PHS 398 Instructions
8
Requirements for a Good
Specific Aims Section
• Brief introduction including the long
range goal of the project
• The goals of this particular application
• The central hypothesis to be examined
• Rationale for the project
• Specific aims
• Anticipated results
Hinda Zlotnik, Ph.D.
9
Rationale
• This is the underlying reason for the
studies you propose.
• The rationale must be relevant to the
problem that has been presented in the
introduction.
Hinda Zlotnik, Ph.D.
10
A dubious rationale
“It isn't pollution that's harming the
environment. It's the impurities in our air
and water that are doing it.” – Al Gore,
Vice President
Specific Aims
• Statements of (measurable) end results, not
means to an end.
• Brief, focused, and limited in scope.
• Each aim flows logically into the next aim.
• Each should be briefly expanded upon.
• Be realistic: do not overestimate your
capabilities for completing the work in the
time requested.
From Hinda Zlotnik, Ph.D.
12
Proposal components
• Abstract
• Budget, Biographical Sketches, Resources
• Research Plan
– A. Specific Aims (and hypotheses)
– B. Background and Significance [2-3 pages]
– C. Preliminary Studies / Progress Report (and
background of investigators)
– D. Research Design and Methods
– E. Human Subjects Research
– G. Literature cited
11/29/2004
PHS 398 Instructions
13
b. Background and Significance
• Sketch the background, critically evaluate
existing knowledge, and specifically identify
gaps.
• State concisely the importance and health
relevance by relating the specific aims to broad,
long-term objectives.
• State how scientific knowledge or clinical practice
will be advanced. Describe the effect on the
concepts, methods, technologies, treatments,
services or preventative interventions that drive
11/29/2004
Adapted from the PHS 398
14
this field.
Instructions
The Review Process
• Lynchpin of the NIH review process is the
initial review group (IRG).
• Helpful to know the psychology and
sociology of review committees – many
agendas.
• Helpful to know who is on the review
committee, what is their expertise, what
they have published.
11/30/2004
V. Schoenbach
15
The Review Process - 2
Purchasing metaphor
• Want to buy, have money to spend
• Want to be an informed consumer
• Don’t want to be duped
• Want value for money
11/30/2004
V. Schoenbach
16
The Review Process - 2
Marketing metaphor:
• Stimulate interest, be in fashion
• Convey information
• Build positive regard
• Avoid unrealistic expectations
• Package attractively
11/30/2004
V. Schoenbach
17
Proposals not scored or downgraded
• “Fatally flawed” methodology or infeasible
• “Incomplete” – missing key detail
• Serious human subjects concern
• Likely not to provide any new or important
substantive knowledge
July 1999
Malcolm Gordon, Ph.D.
18
Would you fund this advance?
“If somebody has a bad heart, they can
plug this jack in at night as they go to bed
and it will monitor their heart throughout
the night. And the next morning, when
they wake up dead, there'll be a record.”
– Mark S. Fowler, FCC Chairman
Proposal components
• Abstract
• Budget, Biographical Sketches, Resources
• Research Plan
– A. Specific Aims (and hypotheses)
– B. Background and Significance
– C. Preliminary Studies / Progress Report
(and background of investigators) [8-10 pgs]
– D. Research Design and Methods
– E. Human Subjects Research
– G. Literature cited
11/29/2004
PHS 398 Instructions
20
c. Preliminary Studies/Progress Report
“Preliminary Studies. For new applications, use
this section to provide an account of the
principal investigator/program director’s
preliminary studies pertinent to this application,
including his/her preliminary experience with
and outreach to the proposed racial/ethnic group
members. This information will also help to
establish the experience and competence of the
investigator to pursue the proposed project.”
[pilot studies]
11/29/2004
PHS 398 Instructions
21
Preliminary Studies
“… results obtained from prior research that
are relevant to the proposed research,
including results that bear on the feasibility
of the proposed research, on the adequacy
of sample sizes you will have access to, and
on the likelihood that the major hypotheses
of the study will be supported.”
Malcolm Gordon, Ph.D.
22
Preliminary Studies – Example
“A recently published examination of racial differences in prostate cancer
treatment outcomes sets the stage for successful completion of the
proposed project. The study was conducted by the investigators, who have
extensive experience in population based case-control studies of cancer
etiology and in case-control studies of cancer screening.
“Godley PA, Schenck AP, Amamoo MA, Schoenbach VJ, Peacock S,
Manning M, et al. Racial differences in mortality among Medicare
recipients after treatment for localized prostate cancer. J Natl Cancer Inst
2003;95(22):1702-10.
We investigated whether racial disparities in outcomes of clinically
localized prostate cancer vary by treatment. Merged SEER and Medicare
files provided data (age, race, stage, grade, census tract socioeconomic
status, and date of death) on 5,747 black and 38,242 white patients with
clinically localized prostate cancer, aged 65 to 84, first diagnosed between
1986 and 1996 in five SEER sites. . . .”
From Paul Godley, M.D., Ph.D.
23
Preliminary Studies
• Describe published studies in limited detail
and include the most important figures
and/or tables.
• Describe unpublished studies in more
complete detail, including newer data.
• Do not duplicate the preliminary studies
with the proposed studies.
Hinda Zlotnik, Ph.D.
24
Preliminary Studies
• Include the results of your recent work that
have direct relevance to the studies
proposed in your grant application.
• Exclude any studies in which the
relationship to your proposed study is not
relevant.
Hinda Zlotnik, Ph.D.
25
Editorial Considerations for
Preliminary Studies
• Supporting data should be placed as close as
possible to where it is referenced in the text
of the application.
• All figures/graphs and tables should be
clear and legible.
• Methodology should be placed in the
figure/table legends, not in the text.
Hinda Zlotnik, Ph.D.
26
Technical Considerations for
Preliminary Studies
• Graphs should be uncomplicated ... the
simpler, the better.
• Each table or figure should be designed to
convey a single point or idea.
• Extraneous or irrelevant data should be
avoided.
Hinda Zlotnik, Ph.D.
27
Proposal components
• Abstract
• Budget, Biographical Sketches, Resources
• Research Plan
– A. Specific Aims (and hypotheses)
– B. Background and Significance
– C. Preliminary Studies / Progress Report (and
background of investigators)
– D. Research Design and Methods [8-14 pgs]
– E. Human Subjects Research
– G. Literature cited
11/29/2004
PHS 398 Instructions
28
d. Research Design and Methods
As described in the PHS 398 instructions:
• Conceptual or clinical framework, procedures,
and analyses to accomplish the specific aims
• How data will be collected, analyzed,
interpreted, and shared
• Novel concepts, approaches, methodologies,
tools, or technologies
11/26/2004
from PHS 398 instructions
29
d. Research Design and Methods
(more from the PHS 398 instructions)
• Discuss potential difficulties and
limitations and alternative approaches
• Tentative sequence or timetable
• Hazards to personnel and precautions
to be exercised
11/26/2004
from PHS 398 instructions
30
THE RESEARCH DESIGN
The “heart and soul” of the application.
Should state precisely:
• What you propose to do
• How you plan to do it
• What the result will mean in terms of the
overall project
• What pitfalls you might encounter
• Alternative approaches to cope with
anticipated problems
Hinda Zlotnik, Ph.D.
31
Research Design:
Organizational Structure
Restate each specific aim and for
each, provide:
1. Introduction
2. Methods of approach
3. Anticipated findings or results
4. Potential pitfalls/alternative approaches
Hinda Zlotnik, Ph.D.
32
Research Design (continued-1)
1. INTRODUCTION
Each section of the research design
should restate the hypothesis to be
tested, the rationale for the study,
overall approaches to be taken, and the
anticipated results.
Hinda Zlotnik, Ph.D.
33
Research Design (continued-2)
2. METHODS OF APPROACH
• Separate sections based on the specific
aims should be used to develop each of
the planned set of studies.
• Avoid emphasis on routine methods
• Use detailed methods only for unfamiliar
technology
Hinda Zlotnik, Ph.D.
34
Research Design (continued-3)
•
•
•
•
3. ANTICIPATED RESULTS
Summarize your results.
Emphasize only the most important results.
Do not overinflate or overinterpret the results.
If limitations exist, they should not be ignored,
but should be discussed in a positive manner.
Hinda Zlotnik, Ph.D.
35
Research Design (continued-4)
4. POTENTIAL PROBLEMS
• Anticipate potential problems, and discuss
them but do not overemphasize them.
• Offer alternative strategies.
• Reconcile the results of differing
approaches.
Hinda Zlotnik, Ph.D.
36
Research Design – Example
“The cases for the proposed study will be males with a diagnosis of
prostate cancer who have died in the years 1997 to 2000. A merged
SEER/Medicare dataset, Patient Entitlement and Diagnosis Summary File
(PEDSF) for years 1997 to 2000, will be used to identify the cases. The
PSA screening exposure will be documented for each case from 1988,
when PSA screening became widely available, to the date of diagnosis of
the case. Since prostate cancer can be a relatively slow growing cancer, by
limiting the study to most recent cases available in the SEER/Medicare
dataset we are allowing as much as 12 years for patients who were not
screened or patients who were unsuccessfully screened to become fatal
cases. Ideally, cases in a case-control study of screening efficacy should be
those who have suffered an adverse outcome of the condition that
screening is intended to identify or prevent (41). Five SEER regions
(Atlanta, Connecticut, Detroit, San Francisco and Seattle) were selected
because they provided a substantial number of African American cases for
analyses and included data collected since 1986.”
From Paul Godley, M.D., Ph.D.
37
Research Design – Example (cont’d)
“The following potential cases will be excluded from the dataset:
• Subjects with a diagnosis of a non-prostate cancer before or after the
prostate cancer diagnosis
• Subjects with prostatic intraepithelial neoplasia as the only prostate
cancer diagnosis
• Subjects with a missing month of diagnosis
• Subject who were diagnosed with prostate cancer at death
• Subjects without Medicare coverage during the study period
• No age restrictions will be used.
An analysis of the 1996 PEDSF file reveals that 924 deaths among prostate
cancer cases meet these criteria. Approximately 4,000 cases can be
expected over the proposed 4-year study period. Of the 924 deaths among
prostate cancer patients, 739 were Caucasian and 185 were African
American.”
From Paul Godley, M.D., Ph.D.
38
Research Design – Example (cont’d)
“Exposure Assessment
The proposed study will compare the screening exposure of deceased cases
to the screening exposure of living controls. This approach has been used
previously in case-control studies of cancer screening, and the use of
deceased cases not be a source of addition bias (55, 56).
Screening PSA determinations were not reimbursed by Medicare until
2000 (57). However, PSA tests were frequently charged to Medicare prior
to 2000 and appear in Medicare administrative data files. Diagnostic PSA
testing was covered by Medicare so some of the PSA tests appearing in
Medicare files may be from screening and not diagnostic patient
encounters. The high prevalence of PSA testing supports the probability
that PSA tests were predominantly screening and not diagnostic in intent.
Freeman et al. (57) found that 37% of men in 1998 SEER/Medicare files
had a PSA test. Among 1996 controls matched to the 924 cases described
above, 65% of men had PSA tests billed to Medicare between 1988 and
1996. It is unlikely that the majority of men ages 65 years and over had
PSA testing solely for diagnostic purposes.”
From Paul Godley, M.D., Ph.D.
39
Research Design – Example (cont’d)
“DATA ANALYSIS
The analysis of the primary specific aim will examine exposure to PSA testing
only, since PSA screening has been widely used since 1988, is the most
sensitive screening test for prostate cancer, and has been documented in
Medicare claims files since 1996. Our analysis will use conditional logistic
regression, with case-control status as the dependent variable and PSA testing
as the independent variables. We will assess the impact of the following
potential confounding variables: race, age, SEER site, income and education
(57). In this context a confounding variable is a determinant (or surrogate for a
determinant) of prostate cancer death and is also associated with the
probability of undergoing prostate cancer screening.
Since the analysis will evaluate fatal prostate cancer as the endpoint,
comparison of screening between cases and controls is only appropriate for the
period prior to the diagnosis of prostate cancer. Therefore, using an approach
similar to that of Friedman (19), we will estimate odds ratios for “screened
with PSA since 1988”. Since this approach is only appropriate for cases
diagnosed after 1988, any case diagnosed prior to that time will be excluded.
We will also examine shorter intervals that may correspond to estimates of the
detectable preclinical phase, as described by Weiss et al. (58).”
From Paul Godley, M.D., Ph.D.
40
Research Design (continued-5)
In general a review committee will be concerned
with four aspects of the proposal:
1. its importance
2. its feasibility
3. the technical merits of the methodology
4. its innovativeness
July 1999
Malcolm Gordon, Ph.D.
41
Research Design (continued-6)
Regarding technical merit, the review
committee considers:
(1) conceptual framework guiding the research
(2) measurement of key variables
(3) sampling plan of the research
(4) procedures for data collection
(5) data analysis plan for the research.
July 1999
Malcolm Gordon, Ph.D.
42
Most Common Reasons for Failure
•
•
•
•
•
Lack of a good original idea.
Unimportant or unresponsive problem.
Unacceptable rationale.
Lack of expertise, experience, or resources.
Superficial or unfocused research design.
Hinda Zlotnik, Ph.D.
43
Why you want to write with precision
"Your food stamps will be stopped
effective March 1992 because we
received notice that you passed away.
May God bless you. You may reapply if
there is a change in your circumstances."
– Department of Social Services,
Greenville, South Carolina
Most Common Reasons for Failure
(more)
• Unrealistic amount of work proposed.
• Uncertain outcomes and/or lack of future
directions.
• Disorganized/poorly written application.
• Lack of progress.
Hinda Zlotnik, Ph.D.
45
Practical suggestions for how to proceed
• Serve as a reviewer.
• Form a team, identify a leader, set a schedule
• Find out who will be reviewing the proposal
and with what criteria
• Find a model proposal to emulate
• Make an outline of the proposal in a word
processing document, add notes, write text
• Review and revise, review and revise, …
11/30/2004
V. Schoenbach
46
Practical suggestions - 2
Outline of Research Design and Methods section:
• Overview
• Design
• Subjects (eligibility, recruitment, consent)
• Data collection (measures, procedures, lab analyses)
• Pretesting, piloting
• Data analysis (variables, approach), sample size
• Quality control and data management
• Timetable and project management
11/29/2004
V. Schoenbach
47
The Research Plan: Summary of
General Considerations - 1
• Read and follow instructions carefully.
• Specific aims are statements of end results. They
are measurable statements, not means to an end.
• Make the logic very clear and write a focused
research plan.
• Don’t ramble--give sufficient background to
make the significance of the proposed research
very clear.
• Frame questions in terms of testable hypotheses.
Hinda Zlotnik, Ph.D.
48
The Research Plan: Summary of
General Considerations - 2
• Clearly delineate the problem and the purpose of
the research.
• Emphasize the specific advantages of doing the
research as you propose.
• Consider alternative strategies – there is concern
when only a single approach is given or a single
uncertain result serves as the basis for the project.
• Include preliminary data if possible.
• Don’t overinterpret or misinterpret data.
Hinda Zlotnik, Ph.D.
49
The Research Plan: Summary of
General Considerations - 3
• Don’t detail routine methods, but include
comments that can convey your expertise to the
reader.
• List relevant intellectual and physical resources
available to you.
• Consider and discuss all variables that will
impact the outcome. That is, which ones need
to be controlled, which ones don’t, and how this
will affect the outcome.
Hinda Zlotnik, Ph.D.
50
The Research Plan: Summary of
General Considerations - 4
• Discuss the limitations of your conclusions, if any.
• Prioritize the tasks proposed and provide time
estimates.
• Use shorter words rather than longer ones unless
they are technical.
• Use short sentences and avoid jargon.
• Make sure the proposal flows logically from
section to section, i.e. methods proposed = logical
extension of the specific aims.
Hinda Zlotnik, Ph.D.
51
Resources
• Succinct overview: “Proposal Writing: The Business
of Science”, by Wendy Sanders. The Whitaker
Foundation, 2004, www.whitaker.org/sanders.html
• Quick Guide for Grant Applications
(deainfo.nci.nih.gov/extra/extdocs/gntapp.htm)
• Guide: “Writing a grant proposal”, Malcolm
Gordon, Ph.D., National Institute of Mental Health,
July 1999
www.charityadvantage.com/iaswr/images/grantproposals.pdf
11/26/2004
V. Schoenbach
52
Resources
• www.training.nih.gov/careers/careercenter/grants.html
• “Getting started – useful links”
(grants.nih.gov/grants/useful_links.htm)
• Current NIH application forms
(grants.nih.gov/grants/forms.htm)
• fdncenter.org/learn/faqs/html/proposal_writing.html
• “Proposal writing short course”
(www.fdncenter.org/learn/shortcourse/prop1.html) –
available in Spanish and French
11/26/2004
V. Schoenbach
53
Resources
Books:
• Designing clinical research: an epidemiologic
perspective, Stephen B. Hulley and Steven R.
Cummings and others. Baltimore, Williams and
Wilkins, 1988
• How to Write a Successful Research Grant
Application: A Guide for Social and Behavioral
Scientists, by Willo Pequegnat, Ellen Stover. Plenum
Publishing Corporation (September 1, 1995,
Paperback: 252 pages, ISBN: 030644965X)
11/26/2004
V. Schoenbach
54
Resources
• AAPM&R-RPC: Resident’s Research Packet
American Academy of Physical Medicine and
Rehabilitation. Finding a Mentor, Funding,
Searching the Literature, Research Design, Research
Proposals, The IRB, Statistics, Presenting Your Work
www.aapmr.org/resident/resrchpac/respacka.htm
• Sigma Xi Grant Writing Course Series, 5-day
intensive course presented by the Grantsmanship
Center, Inc. (TGCI) for Sigma Xi [tuition: $1,000]
www.sigmaxi.org/meetings/grant/index.shtml
[email protected]
11/26/2004
V. Schoenbach
55
Resources
My home page www.unc.edu/~vschoenb/ has these
links:
• epid160.epidemiology.us – Introduction to
epidemiology (includes on-line recorded lectures)
• www.epidemiolog.net – Epidemiology learning
resources and “The Evolving Text” (“el Texto en
Desarrollo”)
• www.minority.unc.edu – Minority Health Project,
includes slide presentations, webcasts, readings
11/29/2004
V. Schoenbach
56
“If we don't succeed, we run the risk
of failure.” – Bill Clinton