Transcript pptx - BioASQ

BioASQ
A challenge on large-scale biomedical
semantic indexing and question answering
www.bioasq.org
George Paliouras
National Center for Scientific Research ‘Demokritos’
Intelligent Information Management
Targeted Competition Framework
ICT-2011.4.4(d)
GeorgePaliouras,
Paliouras,May
May2014
2014
George
www.bioasq.org
Biomedical articles per year
George Paliouras, May 2014
www.bioasq.org
2/43
Questions of biomedical experts
Yes/No question
“Are there any DNMT3 proteins present in plants?”
Exact Answer
“Yes”
Ideal Answer
“Yes. The plant DOMAINS REARRANGED METHYLTRANSFERASE2 (DRM2) is a homolog of
the mammalian de novo methyltransferase DNMT3. DRM2 contains a novel arrangement
of the motifs required for DNA methyltransferase catalytic activity.”
George Paliouras, May 2014
www.bioasq.org
3/43
Questions of biomedical experts
Factoid question
“What is the methyl donor of DNA (cytosine-5)-methyltransferases?”
Exact Answer
“S-adenosyl-L-methionine”
Ideal Answer
“S-adenosyl-L-methionine (AdoMet, SAM) is the methyl donor of DNA (cytosine-5)methyltransferases. DNA (cytosine-5)-methyltransferases catalyze the transfer of a methyl
group from S-adenosyl-L-methionine to the C-5 position of cytosine residues in DNA.”
George Paliouras, May 2014
www.bioasq.org
4/43
Questions of biomedical experts (III)
List question
“Which species may be used for the biotechnological production of
itaconic acid?”
Exact Answer
“Aspergillus terreus”, “Aspergillus niger”, “Ustilago maydis”
Ideal Answer
“In 1955, the production of itaconic acid was firstly described for Ustilago maydis. Some
Aspergillus species, like A. itaconicus and A. terreus, show the ability to synthesize this
organic acid and A. terreus can secrete significant amounts to the media. Itaconic acid is
mainly supplied by biotechnological processes with the fungus Aspergillus terreus. Cloning
of the cadA gene into the citric acid producing fungus A. niger showed that it is possible to
produce itaconic acid also in a different host organism.”
George Paliouras, May 2014
www.bioasq.org
5/43
Questions of biomedical experts (III)
Summary question
“How do histone methyltransferases cause histone modification?”
Exact Answer
-
Ideal Answer
“Histone methyltransferases (HMTs) are responsible for the site-specific addition of covalent
modifications on the histone tails, which serve as markers for the recruitment of chromatin
organization complexes. There are two major types of HMTs: histone-lysine NMethyltransferases and histone-arginine N-methyltransferases. The former methylate specific
lysine (K) residues such as 4, 9, 27, 36, and 79 on histone H3 and residue 20 on histone H4. The
latter methylate arginine (R) residues such as 2, 8, 17, and 26 on histone H3 and residue 3 on
histone H4. Depending on what residue is modified and the degree of methylation (mono-, diand tri-methylation), lysine methylation of histones is linked to either transcriptionally active
or silent chromatin.”
George Paliouras, May 2014
www.bioasq.org
6/43
George Paliouras, May 2014
www.bioasq.org
7/43
Finding relevant snippets
George Paliouras, May 2014
www.bioasq.org
8/43
Not only texts: ontologies, linked data, …
George Paliouras, May 2014
www.bioasq.org
9/43
George Paliouras, May 2014
www.bioasq.org
10/43
Information from structured data
List question
“Which forms of cancer is the Tpl2 gene associated with?”
Related RDF triple
Subject: http://www4.wiwiss.fu-berlin.de/diseasome/resource/diseases/3003 (lung cancer)
Predicate: http://www4.wiwiss.fu-berlin.de/diseasome/resource/diseasome/associatedGene
Object: http://www4.wiwiss.fu-berlin.de/diseasome/resource/genes/TPL2"
Related concepts
http://www.disease-ontology.org/api/metadata/DOID:162 (cancer)
http://www.uniprot.org/uniprot/M3K8_RAT (TPL2 synonym)
George Paliouras, May 2014
www.bioasq.org
11/43
George Paliouras, May 2014
www.bioasq.org
12/43
BioASQ Vision
• Make sure this knowledge is used to the benefit of
patients
• Need to make it accessible to biomedical experts
• Search is not effective enough
• Push research in automated answering of
questions
• A challenge for such systems can achieve a
multiplying effect
George Paliouras, May 2014
www.bioasq.org
13/43
What is BioASQ?
A challenge funded by the European Union (FP7).
Task a: Hierarchical text classification
• Organizers distribute new unclassified PubMed articles.
• Participants assign MeSH terms to the articles.
• Evaluation based on annotations of PubMed curators.
Task b: IR, QA, summarization, …
• Organizers distribute English biomedical questions.
• Participants provide: relevant articles, snippets,
concepts, triples, “exact” answers, “ideal” answers.
• Evaluation: both automatic (GMAP, MRR, ROUGE etc.)
and manual (by biomedical experts).
George Paliouras, May 2014
www.bioasq.org
14/43
Task a
The challenge
George Paliouras, May 2014
www.bioasq.org
Task b
15/43
George Paliouras, May 2014
www.bioasq.org
16/43
Behind the scenes
George Paliouras, May 2014
www.bioasq.org
17/43
BioASQ Platform
George Paliouras, May 2014
www.bioasq.org
18/43
Datasets
Task a
1st challenge
2nd challenge
Training 10,876,004
12,628,968
Test
71950
83490
Task b
1st challenge
2nd challenge
Training
29
310
Test
281
500
Task b data contain gold articles, snippets, concepts,
triples, “exact” and “ideal” answers prepared by
biomedical experts from around Europe.
George Paliouras, May 2014
www.bioasq.org
19/43
Data sources
They include both text
and structured info.
►
PubMed abstracts,
PubMed Central articles,
MeSH.
► Gene Ontology, UniProt,
Jochem, Disease
Ontology.
George Paliouras, May 2014
www.bioasq.org
20/43
Annotation: questions and queries
George Paliouras, May 2014
www.bioasq.org
21/43
Annotation: snippets
George Paliouras, May 2014
www.bioasq.org
22/43
Annotation: answers
George Paliouras, May 2014
www.bioasq.org
23/43
Assessment: relevance of material
George Paliouras, May 2014
www.bioasq.org
24/43
Assessment: information in answers
George Paliouras, May 2014
www.bioasq.org
25/43
BioASQ social network
George Paliouras, May 2014
www.bioasq.org
26/43
Oracle
George Paliouras, May 2014
www.bioasq.org
27/43
Oracle
George Paliouras, May 2014
www.bioasq.org
28/43
Two cycles
2013 Schedule
March 2013
June 2013
August 2013
September 2013
2014 Schedule
February 2014
March 2014
May 2014
September 2014
The official challenge is over, but…
► Task a continues to run each week .
► An oracle for task b will be available soon.
► Oracles will remain available.
► Third cycle is being designed …
George Paliouras, May 2014
www.bioasq.org
29/43
Challenge participants so far
George Paliouras, May 2014
www.bioasq.org
30/43
Challenge participants in each cycle
George Paliouras, May 2014
www.bioasq.org
31/43
Evaluation measures
Task a: Hierarchical text classification
Flat measures for multi-label classification: Accuracy, MiF, MaF, EBF
Hierarchical measures: LCA-F (new), HF
Task b: IR, QA, summarization, …
Phase A:
standard IR measures, mean precision, mean recall, mean Fmeasure, MAP (used for winners selection), G-MAP
Phase B:
‘Exact answers’ (based on type): accuracy (yes/no), strict/lenient
accuracy, MRR (factoid), mean F-measure (list)
‘Ideal answers’: manual scores from the experts {Readability,
Repetition, Information Precision and Recall}, plus ROUGE
George Paliouras, May 2014
www.bioasq.org
32/43
First year technology/results overview
• Task 1a
–
–
–
–
Mainly SVMs and learning-to-rank.
Mostly flat classification, ignoring class taxonomy.
Mediocre results by hierarchical methods.
One of the systems outperformed NLM’s system.
• Task 1b
– Phase A (retrieve relevant documents, concepts, snippets, triples):
low performance (compared to baselines).
– Phase B (formulate ‘exact’ and ‘ideal’ answers): poor performance
for ‘exact’ answers (except for yes/no questions); high
performance for ‘ideal’ answers (paragraph-sized summaries), but
starting with gold documents, snippets etc.
• Large scope for improvements, esp. in Task 1b.
George Paliouras, May 2014
www.bioasq.org
33/43
“Exact” answer results (batch 2/3)
George Paliouras, May 2014
www.bioasq.org
34/43
“Ideal” answer results (batch 2/3)
George Paliouras, May 2014
www.bioasq.org
35/43
Results – task a – flat measures
George Paliouras, May 2014
www.bioasq.org
36/43
Results – task a – hierarchical
George Paliouras, May 2014
www.bioasq.org
37/43
First challenge prizes
George Paliouras, May 2014
www.bioasq.org
38/43
Sustainability
Making the challenge viable, at very low cost,
after the end of the project
•
•
•
•
•
BioASQ Oracle
Software release and installation instructions
Benchmark datasets
BioASQ social network
Involvement of the biomedical community in the
process
• Attracting sponsors for prizes
George Paliouras, May 2014
www.bioasq.org
39/43
Project Consortium
1. National Centre for Scientific Research “Demokritos” NSCR “D” (EL)
2. Transinsight GmbH – TI (D)
3. Universite Joseph Fourier- UJF (F)
4. University Leipzig - ULEI (D)
5. Universite Pierre et Marie Curie Paris 6 – UPMC (F)
6. Athens University of Economics and Business –
Research Centre – AUEB-RC (EL)
George Paliouras, May 2014
www.bioasq.org
40/43
Project Consortium
George Paliouras, May 2014
www.bioasq.org
41/43
Get in touch!
BioASQ workshop @CLEF (Sheffield, Sept 14)
Visit www.bioasq.org
Follow @BioASQ
George Paliouras, May 2014
www.bioasq.org
42/43
Useful Links
• BioASQ Annotation & assessment tools:
– http://at.bioasq.org/
– http://assess.bioasq.org/
– https://github.com/AKSW/BioASQ-AT
• BioASQ social network:
– http://sn.bioasq.org/
– https://github.com/AKSW/BioASQ-SN
• BioASQ platform:
– http://bioasq.lip6.fr/
• BioASQ Oracles:
– http://bioasq.lip6.fr/oracle/
A. Kosmopoulos, I. Partalas, E. Gaussier, G. Paliouras, I. Androutsopoulos, Evaluation
Measures for Hierarchical Classification: a unified view and novel approaches. Data
Mining and Knowledge Discovery (To appear)
George Paliouras, May 2014
www.bioasq.org
43/43