Transcript Document 7603682

Evaluating Internet Based
Cessation Programs II:
Minnesota’s QUITPLAN.COMsm
Intensive Follow-up Study (IFUS)
Sharrilyn Evered, Ph.D., Jessie Saul, Ph.D.,
Annette Kavanaugh, MS, Michael Luxenberg,
Ph.D., Nathan Cobb, MD, Randi Lachter,
MPH, Lawrence An, MD, Barbara Schillo,
Ph.D., and Ann Wendling, MD MPH
Evaluation Team
• Minnesota Partnership for Action Against
Tobacco (MPAAT)
• Jessie Saul, PhD, Barbara A. Schillo, PhD, Ann
Wendling, MD, Randi Lachter, MPH
• Professional Data Analysts, Inc (PDA)
• Sharrilyn Evered, PhD, Michael Luxenberg, PhD,
Annette Kavanaugh, MS
• QuitNet.com
• Nathan Cobb, MD; Pat Milner, Dave Atkins, JD
• Consultants
• Lawrence An, MD, University of Minnesota
Background
• MPAAT launched quitplan.com (fall ‘03)
• Online six-month follow-up survey
obtained low response rate (~10%)
• Quitters are more likely to respond,
producing an inflated cessation rate
• Intention-to-treat assumes all nonrespondents still smoking underestimates cessation rate
Low Response Rates Produce Big
Gaps between Completer and ITT
100%
Standard Follow-Up Survey
80%
60%
40%
52.1%
44.4%
36.7%
20%
0%
Completer (N=162)
5.8%
4.7%
3.6%
Intention-to-Treat (N=1532)
Six-Month 7-Day Point Prevalence
Evaluation Context
• Contracted with PDA to design an
evaluation that would produce a valid
cessation rate (i.e., high response rate)
• Targets: 70% consent rate and 70%
response rate
• Barriers: distrust, reconnecting 6
months post-registration
• Adopted “cafeteria” approach
Evaluation Design
• Mixed mode survey with incentive
• Obtained consent at registration
• Mailed “pre-notification” letter, requesting
updated contact information
• Emailed live link to take survey (two
reminders: 3 & 7 days)
• After 12 days, phoned non-respondents
(up to 25 attempts)
• Respondents mailed $10 thank you check
Pre-IFUS (8/1/03 2/1/04) (N=3,743)
IFUS (2/2/04 4/13/04) (N=1,294)
Post-IFUS (4/14/04 8/28/04) (N=1,319)
Invited (N=1,139)*
Consented (N=685)
60.1%
Responded (N=535)
78.1%
Online (N=283)
52.8%
Declined (N=454)
39.9%
Did Not Respond (N=150)
21.9%
Phone (N=252)
47.1%
* Excludes 9 people who did not permit us to use their responses for research purposes and 7 people who were ineligible for the study (under 18 or
non-MN resident)
High Response Rate Closes the
Gap between Completer and ITT
IFUS
100%
80%
60%
40%
20%
23.4%
20.0%
16.6%
18.3%
15.6%
12.9%
0%
Completer (N=535)
Intention-to-Treat (N=685)
Six-Month 7-Day Point Prevalence
Design Tips
• Contact info update card: $253 in
postage; 19 people updated their info;
18 completed the survey (3.4%)
• Calling 25 times: 90% in 10, 99% in 15
• SurveyMonkey.com: low cost, control
Key to Our Success
• Getting commitment to participate in a
scientific study at the outset
• Provide an incentive
• Pre-notification letter
• Highly trained and invested interviewers
• Careful wording of messages is crucial
How Well do the IFUS
Respondents Generalize to Other
quitplan.com Registrants?
Two Sets of Comparisons to
Assess Bias
Groups Compared for Response
Bias: “IFUS-era”
Invited (N=1,139)
Consented (N=685)
60.1%
Responded (N=535)
78.1%
Online (N=283)
52.8%
Declined (N=454)
39.9%
Did Not Respond (N=150)
21.9%
Phone (N=252)
47.1%
First Comparison: “IFUS-era”
• Logistic regression to predict response
status (respondent vs. non-respondent)
• sex, education level, race/ethnicity
• smoking intensity, time to first cigarette,
stage of readiness, quit attempt past
year, use of quit aids, where heard
about website
• website features visited, postregistration login
Statistically Significant
Relationships: “IFUS-Era”
• The following groups were more likely to
be an IFUS respondent:
• women
• those with some education post HS
• used the nicotine lozenge in the year
prior to registration
• used more of the site’s features (i.e,
set a quit date, posted to a forum)
• logged in after registration
Groups Compared for Response
Bias: “Pre/Post IFUS”
Pre-IFUS (7/31/03 2/1/04) (N=3,743)
IFUS (2/2/04 4/13/04) (N=1,294)
Post-IFUS (4/14/04 8/28/04) (N=1,319)
Invited (N=1,139)*
Consented (N=685)
60.1%
Responded (N=535)
78.1%
Online (N=283)
52.8%
Declined (N=454)
39.9%
Did Not Respond (N=150)
21.9%
Phone (N=252)
47.1%
Significant Relationships:
“Pre/Post IFUS”
Overrepresented in IFUS Underrepresented
• women
• tried to quit at
least once in the
• more educated
past year
• used lozenge past year
• heard about the site on • heard about the
site from
TV or from healthcare
newspaper
or
provider
magazine
• Logged in after
registration
Summary of Response Bias
Analysis
• Bivariate analyses revealed seasonal
variation can account for some
relationships
• Differences between IFUS respondents
and others (not attributable to seasonal
variation): education, gender, site usage
Great Response Rate and a
Representative Sample!
But Did We Sacrifice Internal
Validity?
Groups Compared for Mode Bias:
“Online vs. Phone”
Pre-IFUS (8/1/03 2/1/04) (N=3,743)
IFUS (2/2/04 4/13/04) (N=1,294)
Post-IFUS (4/14/04 8/28/04) (N=1,319)
Invited (N=1,139)
Consented (N=685)
60.1%
Responded (N=535)
78.1%
Online (N=283)
52.8%
Declined (N=454)
39.9%
Did Not Respond (N=150)
21.9%
Phone (N=252)
47.1%
Analysis
• Compared respondents on:
• missing data
• sex, education level, race/ethnicity
• smoking intensity, time to first cigarette,
stage of readiness, quit attempt past
year, use of quit aids, where heard
about website
• website features used, post-registration
login
• cessation and satisfaction outcomes
Results: Demographics, Clinical
Characteristics, and Site Use
• Percent of cases with complete data on
every variable did not vary by mode
• These groups were overrepresented
among online respondents
• women
• those in action or maintenance stage
at registration
• those who used the site more
Results: Cessation Outcomes
Online Phone
Quit for at least 30 days
any time in past six mos
31%
31%
Abstinent past 7 days
21%
18%
Abstinent past 30 days
17%
12%
No significant differences by mode
Results: Attitude Items
The information and advice on
quitplan.com…
Can be trusted
Is easy to understand
Encouraged me to quit
Gave me a new way to think about
quitting
Was presented in a respectful manner
Percent Negative
Responses
Online
Phone
How easy or diffcult is it to use
quitplan.com?
How satisfied are you with the
information and services?
Would you recommend quitplan.com
to a friend?
* Indicates significant difference by mode
4.1
7.3
17.6
19.8
0.4*
2.5
26.3*
17.3
4.8
0.0*
10.7
11.5
20.3
24.2
3.9
7.8*
Mode Bias: Summary
• Cessation outcomes do not vary by
mode
• Some attitude items vary, but not in a
consistent direction
• Mode confounded with an early
response, so results are difficult to
interpret
Conclusions
• Mode bias suggests attitude questions
may be less reliable with this design
• Mixed mode design with incentive can
produce a high response rate
• High response rate generates a more
representative sample and closes the
gap between completer and ITT rates
• Web-based cessation programs can be
meaningfully evaluated
For more information…
• About the design, contact me at:
[email protected]
651-662-9383
• About the results, contact Jessie Saul
at:
[email protected]
952-767-1415