Transcript Presenting Health Statistics on the Web

Presenting Health
Statistics on the Web
Bill Killam, MA CHFP
User-Centered Design www.user-centereddesign.com
Acknowledgments
We would like to thank Dr. Holly Massett of the
National Cancer Institute’s Office of Market
Research and Evaluation, our contracting officer,
for funding this project.
We would also like to thank Dr. Paul Han, also of
the National Cancer Institute, for his desire to
explore the effects of uncertainty and
randomness on perception, which allowed us to
pursue these topics.
User-Centered Design www.user-centereddesign.com
Project Background
The National Cancer Institute (NCI) in its effort
to disseminate information about cancer, uses
both observed data and models to develop
statistical data about cancer.
NCI has developed three mathematical
models for estimating a persons risk for specific
cancers including breast, melanoma, and
colorectal cancer.
The colorectal cancer risk assessment tool
(CCRAT) came under scrutiny by the press.
User-Centered Design www.user-centereddesign.com
Our Goal
The project was initiated to address a concern about
the CCRAT that was unrelated to its output. (An issue
related to the scope of the tool’s data set.)
However, given the opportunity for some redesign,
we were able to address other design concerns
including issues of data entry and output display.
One of the NCI leads was specifically interested in
how uncertainty and randomness affected peoples’
understanding and acceptance of statistical
estimates.
User-Centered Design www.user-centereddesign.com
Guidelines
Nine guidelines for presenting health related
statistics (specifically for risk) had previously
been identified:
1. Provide absolute and comparative risk information.
2. Specify the duration of risk.
3. Provide risk reduction strategies.
4. Address numeracy issues by providing risk both percentage and
frequency formats.
5. Describe the risk using words and numbers.
6. Include a visual display of risk.
7. Acknowledge that the risk estimate contains an element of uncertainty.
8. Compare cancer risk to the risk of other hazards.
9. Frame the risk in positive and negative terms.
User-Centered Design www.user-centereddesign.com
The Original Deign
The original design of the CCRAT met only two of the 9 guidelines. It showed
absolute and relative risk and it indicated the duration of the risk. (Risk reduction
info was also available, but on a separate site and not integrated into the tool.)
User-Centered Design www.user-centereddesign.com
The “Easy” Stuff
Addressing some of the guidelines was not a
significant design issue.
Text changes in the current output page could
address numeracy issues by describing risk using
words and numbers.
Text changes in the current output page could
also address numeracy issues by presenting risk in
both percentage and frequency formats.
User-Centered Design www.user-centereddesign.com
Guidelines That Were Not Addressed
Other guidelines could also be addressed with text
changes in the output page:
1. Framing the risk in both positive and negative terms.
2. Comparing cancer risk to the risk of other hazards.
However, these two guidelines were considered less
critical and there was no agreement in the need to
follow them, so it was decided that these guidelines
would not be accommodated.
User-Centered Design www.user-centereddesign.com
The Not-So-Easy To Address Guidelines
The guideline to provide a visual display of risk has been
previously researched. And there were some examples
already online.
• Icon arrays have been recommended by several
studies, particularly those with human icons (Ancker et
al, 2011 ; Garcia-Retamero et al, 2010).
• Bar chart graphs had been recommended as more
useful under certain conditions (McCaffery et al, 2012)
and less useful under other conditions (Zikmund-Fisher
et al, 2008).
User-Centered Design www.user-centereddesign.com
Uncertainty
The value of reflecting uncertainty in calculations has
also been researched (Han et al, 2006; NadavGreenberg & Joslyn, 2009).
In addition, some research showed that overly precise
estimates, with or without uncertainty levels, may be
considered less believable or credible and are less
available for recall (Brasse et al, 202, Han, P. K., 2009,
Lipkus, 2007, Witteman et al, 2011, Zikmund-Fisher,
2013).
However, specific recommendations for how to
address uncertainty in visual displays are a bit less clear.
User-Centered Design www.user-centereddesign.com
Uncertainty (concluded)
We had done some prior work
exploring a visual design for
showing confidence intervals
as floating bars on an
otherwise standard bar chart
(by replacing the error bars
typical of a bar chart with a
rectangle and removing the
background).
This approach had tested well and was used in another tool, but
the users of this tool are physicians who generally have higher
numeracy scores and have been exposed to concepts of
confidence intervals and statistical error.
User-Centered Design www.user-centereddesign.com
Alternatives Evaluated
We tried several approaches to add uncertainty to icon arrays. Some were
determined to be completely unusable. We did test one concept with users
where the lower CI value remain constant and the CI range dynamically filled in
and was removed.
Though the array was understood after an explanation was given, no one in
testing picked it up on its meaning on their own. (There was also an
interesting misinterpretation that may be present in all icon arrays.)
User-Centered Design www.user-centereddesign.com
Alternatives Evaluated (cont.)
We tried similar approaches to add uncertainty to bar graphs.
Participants had more difficulty understanding uncertainty in this concept than in the
icon array – even after an explanation was provided.
User-Centered Design www.user-centereddesign.com
Alternatives Evaluated (concluded)
Finally we tried an alternate approach similar to the bar graph shown to physicians.
This approach tested very well, particularly when we added a visual indicator of decaying
confidence. However, it was particularly interesting that the placement of the anchor
points on the display was critical for participants to obtain a correct interpretation.
User-Centered Design www.user-centereddesign.com
Risk Output Display
The final output display for risk level was
arranged vertically. The average was added
onto the graph in a different style to help
distinguish it. The scale was set to a constant
value of 0 to 100% regardless of the risk level
to allow the same design to be used and
compared across all risk types (and to avoid
the infamous “Gee Whiz” graph – an issue
we had observed in a number of other
projects and other designs). To address
issues of screen size, a magnifier was used
to view the risk detail while still maintaining its
context on the scale.
User-Centered Design www.user-centereddesign.com
Ensuring Attention to the Details
Because of the large amount of information on the display, animation was used to direct
the user’s attention to each data element.
First, the scale was drawn, followed by the appearance of the average risk. The
person’s calculated risk was then added. Finally, the magnifier appeared showing
the risk in a more readable display.
User-Centered Design www.user-centereddesign.com
Supporting Text for Risk
Text was created to support the graphic. The text represented risk in two
text formats – percentage and frequency out of 100. All calculated values
were rounded to an integer level. A relative reference was made to the
average risk.
Different versions of the text for explaining uncertainty were evaluated.
“Statistical calculations are not exact. We can only say that we are 95% sure the actual
percentage of people who will get the cancer is somewhere in the range shown above.”
“Calculations for the number of people who will get the cancer are not exact. However, the
best estimate is that the value is somewhere in the range shown above. This also means that
there is a possibility that the value is higher or lower than the range shown.”
“Estimates are not exact. Your risk for developing colorectal cancer during your lifetime is most likely in
the range of X%-X% (or somewhere between X and X out of every 100 people), but may be higher or
lower. This means your risk is [higher/lower/the same as] than the average risk for all white males over
the age of 55 - which is approximately X%.”
User-Centered Design www.user-centereddesign.com
The Combined Risk Display
Testing demonstrated that the text
and the visual display were
completely redundant. In other
words, participants appeared able
to understand all of the data from
either the text or the graph.
Note: We added bolding to the text
to support skimming.
User-Centered Design www.user-centereddesign.com
Randomness (Chance)
The issue of randomness was a specific interest of
one of the stakeholders of the project.
The incorrect interpretation of the icon array
showing uncertainly suggested observers may
not be conceptualizing the randomness element
of the calculated risk.
Data on how to display of the concept of
randomness was the least well-researched issue.
User-Centered Design www.user-centereddesign.com
Conceptualizing Randomness
To address the concept of randomness, we returned to the icon array.
We developed an animated icon array to show the concept of randomness. In this array, a random
number of icons matching the calculated risk value are displayed. Then, after a short delay, the
first set of icons fade out and a different random set are displayed. The uncertainty of the risk
value is represented as well – the number of icons displayed any given moment is a random
number within the confidence interval.
User-Centered Design www.user-centereddesign.com
Supporting Randomness Text
Several versions of the text for explaining randomness
were also evaluated. This proved to be one of the most
difficult aspects of this project.
“The value shown describes what happens to a group of people: It does not tell what
will happen to a specific individual.”
“The calculation above is true for a large group of people: It cannot be applied to a specific
individual. Think about tossing a coin. Statistics tells us that out of 100 times a coin is tossed
it will land "heads" about 50 times. However, if you looked at one example of the coin from
that group of tosses, it would be impossible to predict which way that particular toss had
landed.”
“We can calculate that a certain number out of 100 similar people will get a type of cancer,
but we cannot tell which of the 100 people will get the cancer.”
“Though we estimate that somewhere between 5 and 13 out of every 100 people will get
[xxx] cancer, we can't tell for any one person if they will get it or not.”
User-Centered Design www.user-centereddesign.com
The Combined Randomness Display
The selected text was added below the graph.
However, unlike the display of the risk value, the text and the visual display describing randomness
performed differently. Participants initially looked at the icon array and did not understand it. Then they
read the text and the icon array made sense. We considered removing the icon array but participants
emphatically stated it was valuable and didn’t mind having to read the text to understand it.
User-Centered Design www.user-centereddesign.com
The Final Layout
In the final layout, users were
able to select different time
frames and see what factors
determined their risk.
Finally, risk mitigation factors
were added to the primary
display. Selecting an item on
the right side produced a
modified version of the
original estimate with the
original estimate shown as a
shadow display for direct
comparison.
User-Centered Design www.user-centereddesign.com
Risks Below 1%
Addressing low levels of risk was
an exception to some of our own
design guidelines. We displayed
risk as a decimal value on the risk
display, but described it in terms of
rate out of 1000 instead of rate out
of 100. This meant that risk values
below 1% were simply lowered on
the scale. The magnifier still
provided the data at a readable
level while maintaining a
conceptual view of absolute risk.
User-Centered Design www.user-centereddesign.com
Randomness Display at Values below 1%
However, we modified the
randomness display to be an
array of 1000.
User-Centered Design www.user-centereddesign.com
Summary
The final version was implemented and went live
almost exactly as designed and evaluated.
The NCI elected to remove the risk mitigation
factors from the display since the statisticians
were not comfortable predicting the data based
on changes in behavior. The risk mitigation
factors were replaced with basic information on
what increases or decreases a person’s risk.
User-Centered Design www.user-centereddesign.com
Summary (concluded)
Of the three model-based risk calculators currently
available from the NCI, this design is currently being
used only for the colorectal cancer risk assessment
tool (CCRAT).
However, additional research is ongoing to
evaluate the effects of randomness and
uncertainty using the CCRAT design.
User-Centered Design www.user-centereddesign.com
Bibliography
Akl EA, Oxman AD, Herrin J. (2011). Using alternative statistical formats for presenting risks and risk reductions. Cochrane
Database Syst Rev. 2011(3):CD006776.
Brasse G., Cosmides L., Tooby J. (1998). Individuation, counting, and statistical inference: the role of frequency and wholeobject representations in judgment under uncertainty. Journal of Experimental Psychology. 127(1):3–21.
Edwards A, Elwyn G, Gwyn R. General practice registrar responses to the use of different risk communication tools in
simulated consultations: a focus group study. BMJ Sep 18;319(7212):749-752 [FREE Full text] [Medline: 10488001]
Fagerlin A, Ubel P.A., Smith D.M., Zikmund-Fisher BJ. (2007). Making numbers matter: present and future research in risk
communication. Am J Health Behavior; 31(Suppl 1):S47 S56. [Medline: 17931136]
Forrow L., Taylor W.C., Arnold R.M. (1992). Absolutely relative: how research results are summarized can affect treatment
decisions. Am J Med, Feb; 92(2):121-124. [Medline: 1543193] [doi: 10.1016/0002-9343(92)90100-P]
Garcia-Retamero, R., gale sic, M. & Gigerenzer, G. (2010). Do Icon Arrays Help Reduce Denominator Neglect? Medical
Decision Making. 30:672-684.
Han, P. K., Lehman, T. C., Massett, H., & Freeman, A. N., (2011). Communication of uncertainty regarding individualized
cancer risk estimates: effects and influential factors. Medical Decision Making. 31(2):354-366.
Han P.K., Klein W.M., Killam B., Lehman T., Massett H., Freedman AN. Representing randomness in the communication of
individualized cancer risk estimates: Effects on cancer risk perceptions, worry, and subjective uncertainty about risk.
Patient Education and Counseling. Mar 3 2011.
Han, P. K., Moser, R. P., & Klein, W. M. (2006). Perceived ambiguity about cancer prevention recommendations: relationship to
perceptions of cancer preventability, risk, and worry. Journal of Health Communication, 11 (Suppl. 1), 51-69.
Kahneman, D., & Tversky, A. (1982). Variants of uncertainty. Cognition, 11, 143-157.
User-Centered Design www.user-centereddesign.com
Bibliography (concluded)
Lipkus I.M. (2007). Numeric, verbal, and visual formats of conveying health risks: suggested best practices and future recommendations.
Medical Decision Making. 27(5):696–713.
Lipkus I.M. (2007). Numeric, verbal, and visual formats of conveying health risks: suggested best practices and future recommendations.
Medical Decision Making. Sep-Oct 2007; 27(5):696-713.
Lipkus I.M., Hollands J.G. (1999). The visual communication of risk. Journal of the National Cancer Institute Monograph. (25):149–163.
McCaffery, K. J., Dixon, A., Hayen, A., Jansen, J., Smith, S., & Simpson, J. M. (2012). The Influence of Graphic Display Format on the
Interpretations of Quantitative Risk Information among Adults with Lower Education and Literacy: A Randomized Experimental Study.
Medical Decision Making. 27(32):532-544.
Trevena L, Zikmund-Fisher B, Edwards A, Gaissmaier W, Galesic M, Han P, King J, Lawson M, Linder S, Lipkus I, (2012). Ozanne E, Peters E,
Timmermans D, Woloshin S. 2012. Presenting Probabilities. In Volk R & Llewellyn-Thomas H (eds). Update of the International Patient Decision
Aids Standards (IPDAS) Collaboration’s Background Document. Chapter C. http://ipdas.ohri.ca/resources.html
Visschers V.H., Meertens R.M., Passchier W.W., de Vries N.N. (2009). Probability information in risk communication: a review of the research
literature. Risk Anal. Feb; 29(2):267-287.
Waters, E. A., Sullivan, H. W., Nelson, W., & Hesse, B. W. (2009). What is my cancer risk? How internet-based cancer risk assessment tools
communicate individualized risk estimates to the public: Content analysis. Journal of Medical Internet Research, 11(3), e33.
Waters E.A, Weinstein N.D., Colditz G.A., Emmons K. (2006) Formats for improving risk communication in medical tradeoff decisions. J Health
Communications. Mar;11(2):167-182. [Medline: 16537286] [doi: 10.1080/10810730500526695].
Wills CE, Holmes-Rovner M. (2003). Patient comprehension of information for shared treatment decision making: state of the art and future
directions. Patient Educ Couns. Jul;50(3):285-290. [Medline: 12900101] [doi: 10.1016/S0738-3991(03)00051-X]
Witteman, H. O., Zikmund-Fisher, B. J., Waters, E. A., Gavaruzzi, T., & Fagerlin, A. (2011). Risk Estimates From an Online Risk Calculator Are More
Believable and Recalled Better When Expressed as Integers. Journal of Medical Internet Research. 13(3):e54
Yamagishi K. (1997). When a 12.86% mortality risk is more dangerous than a 24.14%: implications for risk communication. Applied Cognitive
Psychology. 11(6):495–506.
Zickerman -Fisher, B. J., Fagerlin, A., & Ubel, P. A. (2010). Improving understanding of adjuvant therapy options by using simpler risk graphics.
Cancer. 113(12):3382-90.
User-Centered Design www.user-centereddesign.com