Guillaume Cabanac [email protected] March 28th, 2012 Musings at the Crossroads of Digital Libraries, Information Retrieval, and Scientometrics http://bit.ly/rguCabanac2012 Musings at the Crossroads of DL, IR, and SCIM Guillaume.
Download ReportTranscript Guillaume Cabanac [email protected] March 28th, 2012 Musings at the Crossroads of Digital Libraries, Information Retrieval, and Scientometrics http://bit.ly/rguCabanac2012 Musings at the Crossroads of DL, IR, and SCIM Guillaume.
Guillaume Cabanac [email protected] March 28th, 2012 Musings at the Crossroads of Digital Libraries, Information Retrieval, and Scientometrics http://bit.ly/rguCabanac2012 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Outline of these Musings Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators Scientometrics Recommendation based on topics and social clues Landscape of research in Information Systems The submission-date bias in peer-reviewed conferences 2 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Outline of these Musings Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators Scientometrics Recommendation based on topics and social clues Landscape of research in Information Systems The submission-date bias in peer-reviewed conferences 3 Musings at the Crossroads of DL, IR, and SCIM DL IR Guillaume Cabanac Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity SCIM Question DL-1 How to transpose paper-based annotations into digital documents? Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Collective annotation: Perspectives for information retrieval improvement.” RIAO’07 : Proceedings of the 8th conference on Information Retrieval and its Applications, pages 529–548. CID, may 2007. 4 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac From Individual Paper-based Annotation … 1541 1630 Annotated bible (Lortsch, 1910) 1790 1830 Fermat’s last theorem Annotations from Blake, Keats… (Kleiner, 2000) (Jackson, 2001) 1881 1998 Les Misérables US students Victor Hugo (Marshall, 1998) Characteristics of paper annotation Secular activity: older than 4 centuries Numerous applicative contexts: theology, science, literature … Personal use: “active reading” (Adler & van Doren, 1972) Collective use: review process, opinion exchange … 5 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac … to Collective Digital Annotations hardcopy Hard to share ‘lost’ author 87% (Ovsiannikov et al., 1999) Web servers reader 13% > 20 annotation systems (Cabanac et al., 2005) ComMentor … iMarkup … Annotation server 1993 a discussion thread Yawas … Amaya … 2005 6 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Digital Document Annotation: Examples W3C Annotea / Amaya (Kahan et al., 2002) a reader’s comment discussion thread Arakne, featuring “fluid annotations” (Bouvin et al., 2002) 7 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Collective Annotations Reviewed 64 systems designed during 1989–2008 Collective Annotation Objective data Subjective information Owner, creation date Anchoring point within the document. Granularity: all doc, words… Comments, various marks: stars, underlined text… Annotation types: support/refutation, question… Visibility: public, private, group… Purpose-oriented annotation categories Annotation remark Personal Annotation Space Annotation reminder Annotation argumentation 8 Musings at the Crossroads of DL, IR, and SCIM DL IR Guillaume Cabanac Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity SCIM Question DL-2 How to measure the social validity of a statement according to the argumentative discussion it sparked off? Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Social validation of collective annotations : Definition and experiment.” Journal of the American Society for Information Science and Technology, 61(2):271–287, feb. 2010, Wiley. DOI:10.1002/asi.21255 9 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Social Validation of Argumentative Debates Scalability issue Which annotations should I read? Social validation = degree of consensus of the group Social Validation 10 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Social Validation of Argumentative Debates Informing readers about how validated each annotation is Before After Annotation magma Filtered display 11 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Social Validation Algorithms Overview A A B B case 3 case 4 case 2 case 1 validity –1 socially refuted 0 socially neutral 1 socially confirmed Two proposed algorithms Empirical Recursive Scoring Algorithm (Cabanac et al., 2005) Bipolar Argumentation Framework Extension based on Artificial Intelligence research works (Cayrol & Lagasquie-Schiex, 2005) 12 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Social Validation Algorithm Example Computing the social validity of a debated annotation 13 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Validation with a User-study Aim: social validation vs human perception of consensus Design Corpus: 13 discussion threads = 222 annotations + answers Task of a participant Label opinion type Infer overall opinion Volunteer subjects 119 53 14 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Experimenting the Social Validation of Debates Q1 Do people agree when labeling opinions? Kappa coefficient (Fleiss, 1971; Fleiss et al., 2003) Inter-rater agreement among n > 2 raters Weak agreement, with variability subjective task agreement Value of Kappa Fair to good Poor Debate Id 15 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Experimenting the Social Validation of Debates Q2 How well SV approximates HP? HP = Human Perception of consensus SV = Social Validation algorithm 1. Test whether PH and VS are different (p < 0.05) Student’s paired t-test: (p = 0,20) > (a = 0,05) Density y = p(HP – SV) Density example: HP = SV for 24 % of all cases HP – SV 2. Correlate HP et SV Pearson’s coefficient of correlation r r(HP, SV) = 0.48 shows a weak correlation 16 Musings at the Crossroads of DL, IR, and SCIM DL IR Guillaume Cabanac Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity SCIM Question DL-3 How to harness a quiescent capital present in any community: its documents? Guillaume Cabanac, Max Chevalier, Claude Chrisment, Christine Julien. “Organization of digital resources as an original facet for exploring the quiescent information capital of a community.” International Journal on Digital Libraries, 11(4):239–261, dec. 2010, Springer. DOI:10.1007/s00799-011-0076-6 17 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Documents as a Quiescent Wealth Personal Documents Filtered, validated, organized information… … relevant to activities in the organization Paradox: profitable, but under-exploited Reason 1 – folders and files are private Reason 2 – manual sharing Reason 3 – automated sharing Consequences People resort to resources available outside of the community Weak ROI why would we have to look outside when it’s already there? 18 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac How to Benefit from Documents in a Community? Mapping the documents of the community SOM [Kohonen, 2001] Umap [Triviumsoft] TreeMap [Fekete & Plaisant, 2001]… Limitations Find the documents with same topics as D Find documents that colleagues use with D concept of usage: grouping documents ⇆ keeping stuff in common 19 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac How to Benefit from Documents in a Community? Organization-based similarities inter-folder inter-document inter-user 20 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac How to Help People to Discover/Find/Use Documents? Purpose: Offering a global view of … people and their documents community Requirement: non-intrusiveness and confidentiality Operational needs Find documents Based on document contents Based on document usage/organization With related materials With complementary materials Seeking people ⇆ seeking documents Managerial needs Visualize the global/individual activity Work position required documents 21 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Proposed System: Static Aspect 4 views = {documents, people} {group, unit} 1. Group of documents Main topics Usage groups 2. A single document Who to liaise with? What to read? 3. Group of people Community of interest Community of use 4. A single people Interests Similar users (potential help) 22 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Outline of these Musings Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators Scientometrics Recommendation based on topics and social clues Landscape of research in Information Systems The submission-date bias in peer-reviewed conferences 23 Musings at the Crossroads of DL, IR, and SCIM DL IR Guillaume Cabanac Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators SCIM Question IR-1 Is document tie-breaking affecting the evaluation of Information Retrieval systems? Guillaume Cabanac, Gilles Hubert, Mohand Boughanem, Claude Chrisment. “Tie-breaking Bias : Effect of an Uncontrolled Parameter on Information Retrieval Evaluation.” M. Agosti, N. Ferro, C. Peters, M. de Rijke, and A. F. Smeaton (Eds.) CLEF’10 : Proceedings of the 1st Conference on Multilingual and Multimodal Information Access Evaluation, volume 6360 de LNCS, pages 112–123. Springer, sep. 2010. DOI:10.1007/978-3-642-15998-5_13 24 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Measuring the Effectiveness of IR systems User-centered vs. System-focused Evaluation campaigns 1958 1992 1999 2001 … [Spärck Jones & Willett, 1997] Cranfield, UK TREC (Text Retrieval Conference), USA NTCIR (NII Test Collection for IR Systems), Japan CLEF (Cross-Language Evaluation Forum), Europe “Cranfield” methodology Task Test collection Corpus Topics Qrels Measures : MAP, P@X ... using trec_eval [Voorhees, 2007] 25 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Runs are Reordered Prior to Their Evaluation Qrels = qid, iter, docno, rel Run = qid, iter, docno, rank, sim, run_id ( , 0.8), ( , 0.8), ( , 0.5) Reordering by trec_eval qid asc, sim desc, docno desc ( , 0.8), ( , 0.8), ( , 0.5) Effectiveness measure = f (intrinsic_quality, MAP, P@X, MRR… ) 26 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Consequences of Run Reordering Measures of effectiveness for an IRS s RR(s,t) P(s,t,d) AP(s,t) MAP(s) 1/rank of the 1st relevant document, for topic t precision at document d, for topic t average precision for topic t mean average precision Sensitive to document rank Tie-breaking bias Ellen Chris Is the Wall Street Journal collection more relevant than Associated Press? Problem 1 Problem 2 comparing 2 systems comparing 2 topics AP(s1, t) vs. AP(s2, t) AP(s, t1) vs. AP(s, t2) 27 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac What we Learnt: Beware of Tie-breaking for AP Poor effect on MAP, larger effect on AP Measure bounds APRealistic APConventionnal APOptimistic padre1, adhoc’94 Failure analysis for the ranking process Error bar = element of chance potential for improvement 28 Musings at the Crossroads of DL, IR, and SCIM DL IR Guillaume Cabanac Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators SCIM Question IR-2 How to retrieve documents matching keywords and spatiotemporal constraints? Damien Palacio, Guillaume Cabanac, Christian Sallaberry, Gilles Hubert. “On the evaluation of geographic information retrieval systems: Evaluation framework and case study.” International Journal on Digital Libraries, 11(2):91–109, june 2010, Springer. DOI:10.1007/s00799-011-0070-z 29 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Geographic Information Retrieval Query = “Road trip around Aberdeen summer 1982” Search engines Topic term {road, trip, Aberdeen, summer} Geographic spatial {AberdeenCity, AberdeenCounty…} temporal [21-JUN-1982 .. 22-SEP-1982] term {road, trip, Aberdeen, summer} 1/6 queries = geographic queries Excite (Sanderson et al., 2004) AOL (Gan et al., 2008) Yahoo! (Jones et al., 2008) Current issue worth studying 30 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac The Internals of a Geographic IR System 3 dimensions to process Topical, spatial, temporal 1 index per dimension Topic Spatial Temporal bag of words, stemming, weighting, comparing with VSM… spatial entity detection, spatial relation resolution… temporal entity detection… Query processing with sequential filtering e.g., priority to theme, then filtering according to other dimensions Issue: effectiveness of GIRSs vs state-of-the-art IRSs? Hypothesis: GIRSs better than state-of-the-art IRSs 31 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Case Study: the PIV GIR System Indexing: one index per dimension Topical = Terrier IRS Spatial = tiling Temporal = tiling Retrieval Identification of the 3 dimensions in the query Routing towards each index Combination of results with CombMNZ [Fox & Shaw, 1993; Lee 1997] 32 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Case Study: the PIV GIR System Principle of CombMNZ and Borda Count 33 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Case Study: the PIV GIR System Gain in effectiveness 34 Musings at the Crossroads of DL, IR, and SCIM DL IR Guillaume Cabanac Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators SCIM Question IR-3 Do operators in search queries improve the effectiveness of search results? Gilles Hubert, Guillaume Cabanac, Christian Sallaberry, Damien Palacio. “Query Operators Shown Beneficial for Improving Search Results.” S. Gradmann, F. Borri, C. Meghini, H. Schuldt (Eds.) TPDL’11 : Proceedings of the 1st International Conference on Theory and Practice of Digital Libraries, volume 6966 de LNCS, pages 118–129. Springer, sep. 2011. DOI:10.1007/978-3-642-24469-8_14. 35 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Search Engines Offer Query Operators Information need “I’m looking for research projects funded in the DL domain” Regular query Query with operators Various Operators Quotation marks, Must appear (+), boosting operator (^), Boolean operators, proximity operators… 36 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Our Research Questions Q = Do query operators lead to improved search results? Q1 = Maximum gain in effectiveness when enriching a query with operators? Q2 = Do users succeed in formulating better queries involving operators? 37 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Our Methodology in a Nutshell V3 V2 V1: Query variant with operators Regular query . VN V4 . . 38 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Effectiveness of Query Operators TREC-7 per Topic Analysis: Boxplots ‘+’ and ‘^’ 39 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Effectiveness of Query Operators Per Topic Analysis: Box plot 0.4 AP (Average Precision) Query variant highest AP 0.3 AP of TREC’s regular query 0.2 0.1 Query variant lowest AP 32 Topics 40 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Effectiveness of Query Operators TREC-7 Per Topic Analysis ‘+’ and ‘^’ MAP = 0.1554 MAP ┬ = 0.2099 +35.1% 41 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Outline of these Musings Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators Scientometrics Recommendation based on topics and social clues Landscape of research in Information Systems The submission-date bias in peer-reviewed conferences 42 Musings at the Crossroads of DL, IR, and SCIM DL IR Guillaume Cabanac Scientometrics Recommendation based on topics and social clues Landscape of research in Information Systems The submission-date bias in peer-reviewed conferences SCIM Question SCIM-1 How to recommend researchers according to their research topics and social clues? Guillaume Cabanac. “Accuracy of inter-researcher similarity measures based on topical and social clues.” Scientometrics, 87(3):597–620, june 2011, Springer. DOI:10.1007/s11192-011-0358-1 43 Musings at the Crossroads of DL, IR, and SCIM Recommendation of Literature Principle: mining the preferences of researchers those who liked this paper also liked… Snowball effect / fad Innovation? Relevance of theme? ???? Cognitive filtering (McNee et al., 2006) Collaborative filtering Guillaume Cabanac Principle: mining the contents of articles profile of resources (researcher, articles) citation graph Hybrid approach 44 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Foundations: Similarity Measures Under Study Model Coauthors Venues graph authors auteurs graph authors conferences / journals Social similarities Inverse degree of separation Strength of the tie Shared conferences length of the shortest path number of shortest paths number of shared conference editions Thematic similarity Cosine on Vector Space Model di = (wi1, … , win) built on titles (doc / researcher) 45 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Computing Similarities with Social Clues Task of literature review Requirement topical relevance Preference social proximity (meetings, project…) re-rank topical results with social clues Combination with CombMNZ (Fox & Shaw, 1993) Degree of separation Strength of ties Shared conferences CombMNZ Social list Topical list CombMNZ TS list Final result: list of recommended researchers 46 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Evaluation Design Comparison of recommendations and researchers’ perception Q1 : Effectiveness of topical (only) recommendations? Q2 : Gain due to integrating social clues? IR experiments: Cranfield paradigm (TREC…) Does the search engine retrieve relevant documents? Doc relevant? corpus relevance judgments {0, 1} binary [0, N] gradual search engine x assessor topic trec_eval Effectiveness measures Mean Average Precision Normalized Discounted Cumulative Gain topic S1 S2 1 0.5687 0.6521 … … … 50 0.7124 0.7512 avg 0.6421 0.7215 improvement +12.3 % significativity p < 0.05 (paired t-test) 47 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Evaluating Recommendations Adaptation of the Cranfield paradigm (TREC…) corpus Is the search engine rec. sys. Retrieving relevant documents researchers? doc relevant ? recommender system search engine x Top 25 assessor researcher topic name of a researcher « With whom would you like to chat for improving your research? » trec_eval Effectiveness measures Mean Average Precision Normalized Discounted Cumulative Gain topical #subjects relevance judgments {0, 1} binary [0, N] gradual topical + social topic S1 S2 1 0.5687 0.6521 … … … 50 0.7124 0.7512 avg 0.6421 0.7215 improvement +12.3 % significativity p < 0.05 (paired t-test) 48 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Experiment Features Data Subjects dblp.xml (713 MB = 1.3M publications for 811,787 researchers) 90 researchers-contacts contacted by mail 74 researchers began to fill the questionnaire. 71 completed it Interface for assessing recommendations 49 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Experiments: Profile of the Participants Experience of the 71 subjects Number of participants Mdn = 13 years 74 Seniority Productivity of the 71 subjects Mdn = 15 publications Number of participants Number of publications 50 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Empirical Validation of our Hypothesis Strong baseline effective approach based on VSM Topical Thématique Topical + social Thématique + Social 1 +8,49 % +10,39 % +7,03 % +6,50 % +10,22 % global < 15 publis >= 15 publis < 13 ans years >= 13 ans years NDCG 0,9 0,8 0,7 0,6 0,5 productivity +8.49 % = experience significant improvement (p < 0.05 ; n = 70) of topical recommendations by social clues 51 Musings at the Crossroads of DL, IR, and SCIM DL IR Guillaume Cabanac Scientometrics Recommendation based on topics and social clues Landscape of research in Information Systems The submission-date bias in peer-reviewed conferences SCIM Question SCIM-2 What is the landscape of research in Information Systems from the perspective of gatekeepers? Guillaume Cabanac. “Shaping the landscape of research in Information Systems from the perspective of editorial boards : A scientometric study of 77 leading journals.” Journal of the American Society for Information Science and Technology, 63, to appear in 2012, Wiley. DOI:10.1002/asi.22609 52 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Landscape of Research in Information Systems The gatekeepers of science 53 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Landscape of Research in Information Systems The 77 core peer-reviewed IS journals in the WoS 54 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Landscape of Research in Information Systems Exploratory data analysis 55 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Landscape of Research in Information Systems Exploratory data analysis 56 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Landscape of Research in Information Systems Topical map of the IS field 57 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Landscape of Research in Information Systems Most influential gatekeepers 58 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Landscape of Research in Information Systems Number of gatekeepers per country 59 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Landscape of Research in Information Systems Geographic and gender diversity 60 Musings at the Crossroads of DL, IR, and SCIM DL IR Guillaume Cabanac Scientometrics Recommendation based on topics and social clues Landscape of research in Information Systems The submission-date bias in peer-reviewed conferences SCIM Question SCIM-3 What if submission date influenced the acceptance of conference papers? Guillaume Cabanac. “What if submission date influenced the acceptance of conference papers?” Submitted to the Journal of the American Society for Information Science and Technology, Wiley. 61 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Conferences Affected by a Submission-Date bias? Peer-review 62 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac The Submission-Date bias Dataset from the ConfMaster conference management system 63 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac The Submission-Date bias Influence of submission date on bids 64 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac The Submission-Date bias Influence of submission date on average marks 65 Musings at the Crossroads of DL, IR, and SCIM Guillaume Cabanac Conclusion Digital Libraries Collective annotations Social validation of discussion threads Organization-based document similarity Information Retrieval The tie-breaking bias in IR evaluation Geographic IR Effectiveness of query operators Scientometrics Recommendation based on topics and social clues Landscape of research in Information Systems The submission-date bias in peer-reviewed conferences 66 Thank you http://www.irit.fr/~Guillaume.Cabanac Twitter: @tafanor