Transcript Optimising the utility of the NeLH VBL for Musculoskeletal

Optimising the Utility of the NeLH VBL for
Musculoskeletal Diseases—Technical
Considerations
Dr. Maged N. Kamel Boulos
Centre for Measurement and Information in Medicine,
City University, London
E-mail: [email protected]
27 November 2002
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Introduction
• A major goal for any online clinical information
service should be to maximise return on
investment (human and other resources, time and
money used to develop and maintain the service).
• Return on investment depends on (target) users’
perceived usefulness (or utility) of the service. A
highly useful service is a successful service.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Suggested Recipe for Success
• Maximising utility (usefulness) through usability (userservice interface issues).
• Maximising utility by ensuring high relevance of VBL
content returned to users in response to their queries (i.e.,
minimal noise and silence).
• Maximising utility by embedding contextual
knowledge and clinical practice guidelines
directly into users’ workflow, e.g., patient
records (direct problem-to-knowledge linking).
• Maximising utility by maintaining high quality/
currency of VBL content.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Usability - 1
Goal: A Powerful Easy-to-use Service
• Usability: The ease with which a user can learn to
operate, prepare inputs for, and interpret outputs of a
service or component of it (IEEE).
• Usability: The effectiveness, efficiency, and satisfaction
with which specified users achieve specified goals in
particular environments (ISO).
• Low service usability might mask useful service features
resulting in decreased users’ perceived usefulness (or
utility) of the service.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Usability - 2
Some Design Guidelines
• Clean, elegant and consistent screens that avoid
detail overload.
• Easy-to-learn/ easy-to-use functions and userservice interfaces that apply the famous principle
of “recognition not recall” (Nielsen, 1994).
• Prevent user errors and provide error correction/
tolerance (e.g., spelling mistakes/ variants in usertyped query strings).
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Usability - 3
Error
Correction/
Tolerance,
e.g.,
ostoarthritis

osteoarthritis
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Usability - 4
Clean, Elegant
Consistent
Screens that are
Easy to
Understand and
Easy to Use
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Usability - 5
Clean, Elegant
Consistent
Screens that are
Easy to
Understand and
Easy to Use
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Usability - 6
Clean, Elegant
Consistent
Screens that are
Easy to
Understand and
Easy to Use
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Usability - 7
Clean, Elegant
Consistent
Screens that are
Easy to
Understand and
Easy to Use
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Usability - 8
Recognition Not
Recall: A
Comparison of Two
Graphical (Visual)
Interfaces
Screenshot of parts of
HealthCyberMap and
Visual Net navigational
maps for “heart diseases”.
Notice the difference in
map iconicity between
HealthCyberMap and
Visual Net approaches,
and the map clutter
resulting from Visual
Net’s way of representing
each resource directly on
the map using a distinct
point symbol.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Relevance - 1
Locating the Needle in the Haystack
• The concept of “relevance” is a fundamental
concept of information retrieval.
• The design and evaluation of search engines/
information retrieval services should be based on
relevance metrics:
– recall (silence); and
– precision (noise).
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Relevance - 2
• The measurement of precision requires a qualified
individual, or group of individuals, to inspect the output
from a search and to sort the output into two groups of
resources—relevant and not relevant (precision = number
of relevant resources in query result/total number of
resources, relevant + irrelevant, in query result).
• The measurement of recall requires that the individual or
group of individuals also have access to the complete set
of resources that was searched (recall = number of
relevant resources in query result/number of relevant
resources in the queried resource pool).
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Relevance - 3
Problems with Free-text, Word-/ Phrase-based Search Engines
• The sought page might be using a different term
(synonym) that points to the same concept. “Myocardial
infarction” and “coronary thrombosis” cannot be
matched, although they are the same.
• Spelling mistakes and variants are considered as different
terms. For example, “psoriasis” (correct spelling) and
“psoriaisis” (typographical error) cannot be matched.
Similarly, “anaemia” (correct UK spelling) and “anemia”
(correct US spelling) cannot be matched.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Relevance - 4
Problems with Free-text, Word-/ Phrase-based Search Engines
• Search engines cannot process clinical documents intelligently and
are unaware of the actual context and content meaning of different
Web resources. For example, searching for resources on “psoriasis”
will retrieve all the documents containing this word, but many of
these resources might not be relevant (“psoriasis” was just
mentioned by the way in these documents, e.g., under a “See also”
heading, and is not their actual topic).
• If you are searching for a particular finding using plain free text
search, you might also pick documents where the clinician/ author
explicitly negates the presence of the finding (e.g., “no dysuria” will
be picked if you are looking for “dysuria”).
• Non-textual resources (images, audio, video) cannot be properly
indexed and retrieved.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Relevance - 5
A Proposed Solution
Metadata PLUS “intelligent” inference/ reasoning
with both metadata and user queries.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Relevance - 6
Metadata—Always Read the Label!
•
Metadata are information about information (e.g., author, subject, type,
format, etc. of NeLH resources/ documents). It can be compared to the
labels you put on your baggage (for unambiguous identification).
• Besides collecting metadata describing information resources, two
other types of metadata should be gathered:
– user profiles (information about user, including user’s location profile which
directly affects user’s health/ health information needs—e.g., location-specific
disease rates, guidelines and healthcare services); and
– device descriptions (information about the connection/ device/ browser that the
user is using to access the service).
• An ideal service should be able to reason with all three types of
metadata to personalise and optimise a Web user’s experience (serve
suitable content in a suitable form and format).
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Relevance - 7
Resource Metadata—Problems and Limitations
• Metadata can greatly enhance information retrieval, but this
depends on the quality of the metadata we are using and on our
resource indexing granularity [do we treat big sites, collections
and lengthy documents (e.g., a clinical guideline) as a single
resource or index individual pages, smaller collections and
subsections from these sites and documents that cover
individual (specific) topics as distinct resources].
• Using keywords in resource metadata to describe the content of
a resource is not the optimal solution. The user might not know
which keywords or terms were used to index the resource and
can thus miss a relevant resource when performing a search.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Relevance - 8
Resource Metadata—Problems and Limitations
• A further improvement would be to use a thesaurus to care for
more synonyms or force the user to select keywords from a
predefined collection of terms we are using in our metadata.
But, thesauri also have their own shortcomings. They do not
allow users to ask questions like “Get pages describing the
complications of diabetes mellitus” and retrieve relevant
pages, say on “peripheral neuropathy”. This is due to the fact
that a thesaurus cannot know the semantic relationship
between “diabetes mellitus” and “peripheral neuropathy”
(“peripheral neuropathy” is not a synonym or variant of
“diabetes mellitus”).
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Relevance - 9
Resource Metadata—Problems and Limitations
• A coding scheme (controlled vocabulary or concept- and
knowledge-based terminology) would take thesauri one
step further by offering a collection of terms along with a
hierarchy that tells us which terms are kinds of one
another (and other relationships). We can use the
hierarchy to ask more general questions. Queries for
“endocrine disorders” can now pick up “diabetes
mellitus” and “Grave’s disease”. Unfortunately, (older/
first-generation) coding schemes still have their
drawbacks.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Relevance - 10
Resource Metadata—Problems and Limitations
• A term has to be introduced for anything that we want to represent
(enumerative approach). This can lead to an unmanageable number of
terms (and many concepts will still be missing at the end, irrespective of
how hard we try).
• Adding compositional (concepts can be constructed from primitive
building blocks, governed by validation rules) and multiple parentage
capabilities to coding schemes can solve this. For example, “surgical
operation” acts as parent for “transplant operation” and “kidney
operation”, which are both in turn parents of “kidney transplant”. A
search for either “transplant operation” or “kidney operation” would find
“kidney transplant”, thanks to multiple parentage.
•
This is the approach adopted in SNOMED-CT (Systematised Nomenclature of Medicine–Clinical
Terms, released in February 2002—<http://www.snomed.org>) for example.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Relevance - 11
Resource Metadata—”Intelligent” Inference/ Reasoning
• It’s not practical or computationally efficient to encode (using
concepts from a suitable terminology) all semantic
relationships and other possibilities of related topics in an
information resource or a metadata record of it (especially
given the fact that resource indexing is still a largely manual
process).
• The ideal system should be able, given just the main (most
specific) concept code(s) of a resource topic, to automatically
infer all allowed textual synonyms/ descriptions (even in
multiple languages) for this topic, as well as the codes of any
other relevant topics related to this resource via semantic
relationships.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Relevance - 12
Resource Metadata—”Intelligent” Inference/ Reasoning
• A terminology server (a terminology inference engine shared by
many applications like NeLH, EPR, etc.) allows, given a
terminology concept, the retrieval of synonyms and related
broader/ narrower concepts (parent, cousin, uncle, sibling and child
concepts) from the underlying clinical terminology.
• Ideally, a terminology server should support concept mapping,
which involves processing free text queries to identify
corresponding terms from a controlled vocabulary. This relieves
users from any restrictions while ensuring accurate results and can
also support spelling variants and, if necessary, multiple languages
(SNOMED-CT concepts are language neutral and so can serve
different natural languages).
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Relevance -13
HealthCyberMap (HCM) Proposed Architecture: Explicit concepts in resource metadata map onto a
domain ontology (a clinical terminology) allowing a semantic search engine to infer implicit meanings
(synonyms and semantic relationships) not directly mentioned in either the resource or its metadata.
Similarly, user queries would map to the same ontology allowing the search engine to infer the implicit
semantics of user queries and use them to optimise retrieval.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Problem-to-Knowledge Linking - 1
The library goes to clinicians instead of clinicians going to the library
• Problem-to-knowledge linking aims at providing
contextually appropriate medical knowledge in
the right place and at the right time by embedding
contextual knowledge and clinical practice
guidelines directly into users’ workflow, e.g., at
the point of care in electronic patient records
(EPR).
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Problem-to-Knowledge Linking - 2
The library goes to clinicians instead of clinicians going to the library
N
e
L
H
• Clinical codes provide a common backbone language
(ontology) for proper communication between the EPR
and online health/ clinical information services like
NeLH.
• The actual success of problem-to-knowledge linking will
depend on the quality and granularity of the metadata it
uses, the topical coverage and quality of resources it
points to, and the use of a suitable concept qualifier
mechanism to maximise contextual relevance (by better
S
N
O
M
E
D
E
P
R
defining a topic, narrowing retrieval, or expressing a certain
aspect of a main concept).
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Problem-to-Knowledge Linking - 3
Beyond static knowledge: Self-executing guidelines that work for you!
• Computer-interpretable guidelines (CIG) aims at delivering patient-specific
recommendations that are integrated with electronic patient records at point of
care, i.e., integrated into workflow.
• CIG are used to generate automated reminders/ alerts; in decision support and
task management; to perform retrospective analysis to test if patients were
treated appropriately; to check order entry appropriateness, referral criteria; for
background monitoring, execution of care plans and quality review.
• Individual coded patient data from the electronic patient record are matched to
coded guideline terms and flowchart; the recommendations in guidelines are
matched to actions in an order entry system or for prescription printing.
• The ultimate goal is to fully apply guidelines to clinical practice, and
continually evaluate their application and modify/ refine guidelines accordingly
(protocol-guided care).
Knowledge  Guidelines  Workflow (Practice)
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Quality and Currency - 1
• Health and medical Web resources are not all written by qualified,
unbiased professionals, hence the need for rigorous quality
benchmarking.
• Currency (up-to-dateness) is one aspect of information quality.
• A quality metadata element should be introduced to store a
resource’s level of evidence (whether it is an official guideline,
systematic review, randomised controlled trial—RCT, other peerreviewed study, official critically appraised topic—CAT, or expert
opinion), and any other relevant information regarding its
compliance to a recognised code of ethics (e.g., Health On the
Net—<http://www.hon.ch/>) or quality seal, and whether it has
been published by a trusted publisher or listed in trusted directory.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Quality and Currency - 2
•
We need to define a consistent way for storing this quality
information and all its facets to ensure reliable processing of it later
on, especially if NeLH metadata is to be made available for use by
other external services as well.
• Emerging quality “standards” like HIDDEL (Health Information
Disclosure, Description and Evaluation Language—
<http://www.medcertain.org/english/about_us/overview.htm>) and European
Commission Guidelines (eEurope 2002/ eHealth Quality Criteria
for Health Related Websites
<http://europa.eu.int/information_society/eeurope/ehealth/quality/draft_guidelines/index_en.htm>)
must be also taken into consideration.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Quality and Currency - 3
A Plethora of Kitemarks
• New: MedCIRCLE—Collaboration for Internet Rating,
Certification, Labeling and Evaluation of Health Information
(<http://www.medcircle.org/>). It is the follow-up project to
MedCERTAIN and implements the HIDDEL vocabulary.
• Recommended review of the different quality benchmarking
tools and checklists for medical/ health Web resources: Kamel
Boulos MN, Roudsari AV, Gordon C, Muir Gray JA. The Use of
Quality Benchmarking in Assessing Web Resources for the
Dermatology Virtual Branch Library of the National electronic
Library for Health (NeLH). J Med Internet Res. 2001;3(1):e5.
Available from: <http://www.jmir.org/2001/1/e5/>.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Quality and Currency - 4
Managing Web Resources for Persistent Access
• The success of a distributed information system such as the Web
for research depends on the long-term consistency of the interlinks between online resources and the persistence of the links that
are provided in the catalogues, indexes and listings of resource
discovery services.
• The National Library of Australia has recently published a
document titled “Managing Web Resources for Persistent Access”
on how to overcome the deadly broken link message “HTTP 404
Not Found”.
National Library of Australia. Managing Web Resources for Persistent Access. Available
from: <http://www.nla.gov.au/guidelines/2000/persistence.html> and
<http://www.nla.gov.au/guidelines/2000/webresources.html>.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Quality and Currency - 5
Managing Web Resources for Persistent Access
• What NeLH can do is to regularly run a link checker on its
database of resource addresses to identify links that are no more
functional (and possibly exclude/ delete them). Replacing these
links with more current, functional versions still requires manual
intervention.
• NeLH should also perform regular quality checks on listed
resources (in case they have changed or the information they
contain is no more valid). This is another manual task for humans
to complete, as it depends on their discernment capabilities.
• Some of the National Library of Australia guidelines also apply to
NeLH as a service Web site, so that links to it from other sites
around the Web and within it between its different components
remain valid.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos
Conclusion and Recommendations
•
Learn who are your target users and what are their needs. Evaluate
your service continuously.
(Extremely useful: NIH Web Site Evaluation and Performance
Measures Toolkit—<http://irm.cit.nih.gov/itmra/weptest/acknow.htm>).
• Collaborate with relevant stakeholders, e.g., other NeLH VBLs,
EPR and Standards Bodies.
• A highly useful service is a successful service. The following four
ingredients are key to high service usefulness:
–
–
–
–
usability;
relevance;
integrability into work practices; and
quality and maintainability.
Concepts from HealthCyberMap—<http://healthcybermap.semanticweb.org>
MN Kamel Boulos